Kobo sync: fix the download, restore series, clean up #5
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "kobo-sync-fixes"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
When we last left this branch the Kobo was happily syncing metadata from books.fismen.no but quietly refusing to download any of the actual epubs — the device would fetch each book's metadata, think about it for half a second, and then move on without ever asking for the file. We'd been chasing half a dozen increasingly unlikely suspects for this (URL length, publisher sentinels, publication-date mapping, DRM fields, content-access endpoints) and had checkpointed the branch at
5882803so we could start fresh.It turned out to be a one-line shape mismatch. Nickel parses
/v1/library/{id}/metadataas a JSON array and reads index[0]; we were sending the bare metadata object, so the indexed read came up empty andDownloadContentFileCommandsilently finished without ever issuing the GET toDownloadUrls[0].Url. calibre-web already wraps that endpoint in a one-element list — we just missed it because a dict-vs-dict diff between the two bodies stripped the outer array and showed zero key differences. Matching calibre-web fixes downloads end-to-end, confirmed on the device via thenc 192.168.86.39 5001Qt debug stream:ResumingDownloader::start→ResumingDownloader::finishedon/api/kobo/{token}/download/{id}/kepubwithin 700 ms of the metadata fetch.With the real root cause known, the branch also rolls back the pile of symptom-chasing workarounds from the earlier checkpoint:
NumberandNumberFloatas JSON numbers,Idas a UUIDv3 in the DNS namespace keyed on the series name).2020-01-01stand-in.content_access_bookstub are deleted. Nickel only ever calls the one URL we emit inDownloadUrls, and it works.mimicCalibreWebSyncdiagnostic, a now-unusedslugifyhelper, and several comments blaming the wrong fields are gone.docs/kobo-api-findings.mdis updated with the rediscovery trap so the next session doesn't fall into the same hole: the "What doesn't work" section becomes "The metadata-shape gotcha," and the status table shows downloads working.Also adds a new
README.md(the repo didn't have one) with the minimum env vars needed to run the server and the one-lineKobo eReader.confsetup procedure for pointing a stock-firmware device at books — no SSH, no factory reset, no firmware modification.Net diff: ~271 lines deleted, ~200 added, most of the additions being the new README. The actual behaviour change is ~10 lines in
internal/kobo/metadata.goandsync.go.Test plan
nc :5001Qt debug stream.go build ./... && go vet ./... && go test ./internal/kobo/...all green.api_endpointfrom the README intoKobo eReader.conf, reboot, confirm first-run sync pulls and downloads the whole shelf cleanly.Intended to be squash-merged; the individual commits on the branch are a breadcrumb trail, not a curated history.
🤖 Generated with Claude Code
Nickel post-sync calls /v1/user/profile, /v1/user/loyalty/benefits, /v1/user/wishlist, /v1/deals and (on collection change) /v1/library/tags/ {id}/items/delete. Unhandled, these fell through to the web catch-all and 303'd to /auth/login — which Nickel reads as HTML where it expects JSON and bails with "sync failed". Also make LibraryDelete idempotent: stale entitlement IDs the device remembers from a previous server (e.g. grimmory's integer IDs) now resolve to 200 instead of 404.Aligns our wire format and handler surface with the two reference implementations the Libra Colour is known to work with: - Use the book's integer primary key as EntitlementId / RevisionId / CrossRevisionId / WorkId / CoverImageId, matching grimmory. Routes now accept {id} and resolve via GetBookByID. - Always emit a BookEntitlement.ActivePeriod block — Nickel treats a missing ActivePeriod as "entitlement not active". - Expand /v1/initialization to the full Kobo store resource set (with our local overrides for sync / metadata / covers / analytics etc.), so Nickel doesn't clear URL templates on each refresh. - Emit Format="EPUB" first in DownloadUrls and drop DrmType entirely, matching calibre-web (which works with the same device family). - Add PublicationDate, drop Slug, default Categories to the store's generic genre UUID, default Language to BCP-47 "en". - Always emit Series (with empty strings when unset). - Register POST+GET catch-all /api/kobo/{token}/v1/ that returns empty JSON so unimplemented store endpoints don't fall through to the web UI's OIDC redirect and break sync. - Stub handlers for /v1/user/profile, /v1/user/loyalty/benefits, /v1/deals, /v1/user/wishlist and /v1/library/tags/{id}/items/delete. - LibraryDelete is idempotent on unknown / malformed IDs. - Auto-supersede drained sync sessions when shelf set changes. Result: sync completes cleanly, covers load, entitlements land in KoboReader.sqlite's content table. Downloads still do not trigger.- EntitlementId / RevisionId / CrossRevisionId / WorkId / CoverImageId are all back to book.UUID. Integer IDs were an accidental grimmory copycat; calibre-web proves UUIDs work against the Libra Colour. - lookupBookByPathID accepts both UUID and integer on path routes so Nickel and internal callers both work. - DownloadUrls emit a single Format="KEPUB" entry. The file we serve is a plain EPUB; Nickel reads it either way, but it won't queue a download unless the label is KEPUB. - Download URL changed to {base}/download/{id}/kepub to match the known-working calibre-web pattern. - Timestamps drop the 7-digit fractional second — plain ISO-8601 seconds like calibre-web. - Initial ReadingState.CurrentBookmark has only LastModified (no ProgressPercent, no Location sub-object). - ContributorRoles populated with {Name: author} entries. - ActivePeriod.From = current sync time, not book.AddedAt. Sync, metadata and covers all land cleanly in KoboReader.sqlite; downloads still do not trigger against books.fismen.no even though the identical device downloads successfully from calibre-web. The remaining delta must be something subtler — MimeType coercion in KoboReader.sqlite, Monetization flag, or a per-device Nickel cache.Branch checkpoint ahead of a fresh debugging session. All the progress in this commit is verified against the device's nc:5001 debug stream, either eliminating a failure mode or shaping the wire format to match calibre-web/grimmory. Only the final step — Nickel actually GETting the download URL — remains broken. Response-shape parity with the working references: - Content-Type: application/json; charset=utf-8 on /sync AND /library/{id}/metadata. Go's net/http goes chunked without an explicit Content-Length, which Nickel's Qt stack dislikes, so set Content-Length on both endpoints. - Metadata endpoint now also emits a Set-Cookie to match calibre-web's pattern. - Description is emitted as `null` (pointer) rather than omitted. - Publisher.Name is a *string pointer so `null` is emittable when no real publisher is known. - Series is omitted entirely when the book has no series. - ContributorRoles is built from the author list. - PublicationDate is currently a fixed past date (2020-01-01). - Format=KEPUB, Platform=Generic, DrmType dropped. - DownloadUrls.Url uses the api_endpoint-prefixed path /api/kobo/{token}/download/{id}/kepub so Nickel's prefix check passes. The integer book ID keeps the URL short (~95 chars). Download handler mimics calibre-web/grimmory response shape: application/octet-stream, Content-Disposition filename="{title}.kepub", x-kobo-apitoken header. Routing: - New alias GET /api/kobo/{token}/download/{id}/kepub matching calibre-web's path shape. - New alias GET /api/kobo/{token}/v1/user/library/books/{id} in case Nickel derives the download URL from the library_book resource template. - New GET /api/kobo/{token}/v1/products/books/{id}/access handler returning the kobo-book-downloader-style {ContentKeys:[], ContentUrls:[{...}]} object. - lookupBookByPathID accepts both UUID and integer path values. Init response: full Kobo store resource dict with our local overrides. Feature flags (audiobooks, dropbox, googledrive, nativeborrow) all forced to False to match calibre-web. Sync session lifecycle: previousBooks is only loaded when the request carries a sync token, so a fresh (no-token) sync computes its diff against an empty previous set and actually emits NewEntitlements instead of "everything already delivered". LibraryDelete is idempotent on unknown IDs. Plus docs/kobo-api-findings.md — a full writeup of the protocol, every knob we found, the nc:5001 debug channel, the Nickel command pipeline, and everything we've tried for the download bug.The "silent drop after /metadata" bug wasn't any of the fields we were chasing — it was the top-level shape of the /v1/library/{id}/metadata response. Nickel's DownloadContentFileCommand parses that endpoint as a JSON array and reads index [0]; we were emitting the bare metadata object, so the index read turned up nothing and the command finished without issuing the GET to DownloadUrls[0].Url. calibre-web's cps/kobo.py already wrapped the value in a one-element list; matching that is a one-liner in internal/kobo/metadata.go. Confirmed end-to-end against the Libra Colour: packetdump.debug shows ResumingDownloader::start → ::finished on /api/kobo/{token}/download/{id}/kepub within ~700ms of DownloadContentFileCommand executing. With the real root cause known, revert the pile of symptom-chasing workarounds from the previous checkpoint: - Restore series metadata. Shape matches calibre-web exactly (Name, Number + NumberFloat as JSON numbers, Id as a uuid v3 derived from the series name in the DNS namespace). Previous code forcibly omitted Series while blaming it for the download failure. - Series.Number is now float64, not string, to match calibre-web's wire. - PublicationDate goes from a hardcoded "2020-01-01" back to the book's AddedAt. We don't track real pubdates, but AddedAt is always a plausible past instant. - Drop mimicCalibreWebSync (140-line byte-for-byte calibre-web clone we were using to isolate the bug). - Drop slugify, unused once the Series id moved to uuid v3. - Drop three dead download-route aliases: /v1/books/{id}/download, /kobo/{token}/download/{id}/kepub, /v1/user/library/books/{id}. Nickel only uses the one URL we emit in DownloadUrls, so only /api/kobo/{token}/download/{id}/kepub stays. - Drop ContentAccessBook and its /v1/products/books/{id}/access route. calibre-web doesn't implement it either; Nickel doesn't need it once /metadata parses correctly. Promote github.com/google/uuid from an indirect to a direct dependency (uuid.NewMD5 for deterministic series ids). Updated docs/kobo-api-findings.md: the status table shows downloads working, the rediscovery-trap section about /metadata being an array replaces the old "same shape as a sync envelope" lie, and the "What doesn't work" section is rewritten as "The metadata-shape gotcha" with a list of the things we tried before noticing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>