VG and Aftenposten scrapers return no articles after site redesign #11

New issue

Closed

opened 2026-06-15 21:45:33 +02:00 by arne · 0 comments

arne commented

2026-06-15 21:45:33 +02:00

Owner

Both vg.no and aftenposten.no migrated to the same Schibsted frontend with build-hashed CSS-module class names. The homepage scrapers' selectors no longer match anything:

VG looks for article.article-extract and h2.headline — both gone.
Aftenposten looks for article.teaser[data-pulse-url] and data-pulse-teaser-title — the attributes were removed.

Both scrapers hit their no articles found guard. Article-page content extraction (.article-body p / article p + og: meta tags) is unaffected.

Fix: replace both homepage parsers with a shared Schibsted parser keyed on stable hooks (a[data-content-type="article"] for links, h2[aria-label] for titles), host-filtered per site. Refresh test fixtures.

Both vg.no and aftenposten.no migrated to the same Schibsted frontend with build-hashed CSS-module class names. The homepage scrapers' selectors no longer match anything: - VG looks for `article.article-extract` and `h2.headline` — both gone. - Aftenposten looks for `article.teaser[data-pulse-url]` and `data-pulse-teaser-title` — the attributes were removed. Both scrapers hit their `no articles found` guard. Article-page content extraction (`.article-body p` / `article p` + og: meta tags) is unaffected. Fix: replace both homepage parsers with a shared Schibsted parser keyed on stable hooks (`a[data-content-type="article"]` for links, `h2[aria-label]` for titles), host-filtered per site. Refresh test fixtures.

arne referenced this issue from a commit

2026-06-15 21:49:47 +02:00

Fix VG and Aftenposten scrapers for Schibsted redesign

arne referenced this issue from a pull request that will close it,

2026-06-15 21:50:04 +02:00

Fix VG and Aftenposten scrapers for Schibsted redesign #12

arne closed this issue

2026-06-15 21:50:14 +02:00