arne/news
0
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity

Fix VG and Aftenposten scrapers for Schibsted redesign #12

Merged
arne merged 1 commit from orbit-11-fix-schibsted-scrapers into main 2026-06-15 21:50:14 +02:00
Conversation 0 Commits 1 Files changed 6 +524 -6236
arne commented 2026-06-15 21:50:04 +02:00
Owner
Copy link

Both vg.no and aftenposten.no migrated to the same Schibsted frontend with build-hashed CSS-module class names, so the homepage scrapers' selectors matched nothing and both returned no articles found.

Change

  • New shared parser scrapers/schibsted.go keyed on stable hooks: a[data-content-type="article"] anchors for links and the teaser heading for titles.
  • Links are host-filtered per site (drops cross-promoted e24 content and video/event teasers) and deduplicated by URL with tracking params stripped.
  • Titles read the visible heading with <br> turned into spaces — this reconstructs hero headlines split across styled spans (otherwise Drittleidødtid) and avoids the Premium, kicker Schibsted folds into aria-labels.
  • vg.go and aftenposten.go now delegate to the shared parser.
  • Refreshed VG and Aftenposten homepage fixtures to the current markup.

Article-page content extraction (.article-body p / article p + og: meta) was verified still working and is unchanged.

Verification

  • go test ./scrapers/... passes; fixtures yield 19 (VG) and 14 (AP) articles.
  • Live run against the real sites: VG 20 articles, Aftenposten 14, with clean titles.

Note: four pre-existing failures in the news package (db/server desk tests) are unrelated to this change and fail on main as well.

Closes #11

Both vg.no and aftenposten.no migrated to the same Schibsted frontend with build-hashed CSS-module class names, so the homepage scrapers' selectors matched nothing and both returned `no articles found`. ## Change - New shared parser `scrapers/schibsted.go` keyed on stable hooks: `a[data-content-type="article"]` anchors for links and the teaser heading for titles. - Links are host-filtered per site (drops cross-promoted e24 content and video/event teasers) and deduplicated by URL with tracking params stripped. - Titles read the visible heading with `<br>` turned into spaces — this reconstructs hero headlines split across styled spans (otherwise `Drittleidødtid`) and avoids the `Premium,` kicker Schibsted folds into aria-labels. - `vg.go` and `aftenposten.go` now delegate to the shared parser. - Refreshed VG and Aftenposten homepage fixtures to the current markup. Article-page content extraction (`.article-body p` / `article p` + og: meta) was verified still working and is unchanged. ## Verification - `go test ./scrapers/...` passes; fixtures yield 19 (VG) and 14 (AP) articles. - Live run against the real sites: VG 20 articles, Aftenposten 14, with clean titles. Note: four pre-existing failures in the `news` package (db/server desk tests) are unrelated to this change and fail on `main` as well. Closes #11
Both sites moved to the same Schibsted frontend with build-hashed
CSS-module class names, so the old selectors (article.article-extract,
h2.headline, article.teaser[data-pulse-url]) matched nothing and both
scrapers returned "no articles found".

Replace both homepage parsers with a shared Schibsted parser keyed on
stable hooks: a[data-content-type="article"] anchors for links, the
teaser heading for titles. Links are host-filtered per site (dropping
cross-promoted e24 content and video/event teasers) and deduplicated by
URL with tracking params stripped. Titles read the visible heading with
<br> turned into spaces, which reconstructs hero headlines split across
styled spans and avoids the "Premium," kicker folded into aria-labels.

Refresh the VG and Aftenposten homepage fixtures to the current markup.

Closes #11

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
arne merged commit 720ae75de0 into main 2026-06-15 21:50:14 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
arne/news!12
No description provided.