arne/books
0
0
Fork 0
Code Issues 3 Pull requests Projects Releases 5 Packages Wiki Activity

Kobo annotations research spike (yes-feasible, w/ docs corrections) #24

Merged
arne merged 19 commits from kobo-annotations-spike into main 2026-04-20 21:08:59 +02:00
Conversation 0 Commits 19 Files changed 8 +2249 -17
arne commented 2026-04-20 21:05:44 +02:00
Owner
Copy link

Research spike into whether Kobo annotation sync is reachable via the /api/kobo/* sync API. Yes, feasible — wire shapes captured for the three observed reading-services endpoints. Follow-up implementation spec is a separate sub-project; auth model tracked in #23.

Summary

  • Spec: docs/superpowers/specs/2026-04-20-kobo-annotations-spike-design.md
  • Plan: docs/superpowers/plans/2026-04-20-kobo-annotations-spike.md
  • Findings: docs/kobo-annotations-spike-findings.md — methodology + results + wire-shape tables + recommendation.

Key discoveries

  • Annotation sync runs in Nickel when the device is signed in to a Kobo cloud account. Traffic goes to reading_services_host (default readingservices.kobo.com) with a Kobo-cloud-issued Bearer JWT.
  • Three endpoints observed + characterised: POST /api/v3/content/checkforchanges, PATCH /api/v3/content/{bookID}/annotations, GET /api/UserStorage/Metadata.
  • Nickel treats reading_services_host as host-only (path component discarded), so requests arrive at server root — not under api_endpoint's prefix.
  • nc :5001 was the tool that unstuck the spike after passive catch-all + probe stubs both yielded null results.

Docs corrections (spin-off)

Two claims in docs/kobo-api.md that the spike proved wrong, now corrected in the same PR:

  1. Automatic host rewriting is storeapi-specific, not universal.
  2. Init write-back is selective, not universal — only a subset of Resources keys get persisted.

Plus a new Chapter 3 subsection documenting reading_services_host's host-only quirk.

Infrastructure aftermath

  • Permanent: low-noise slog.Info("unknown kobo endpoint", ...) drift detector in the catch-all.
  • Removed at teardown: transcript recorder, probe handlers, spike env vars, reading-services + notebooks init overrides, broadened catch-alls.

Test plan

  • Read the findings doc; confirm wire shapes match your mental model.
  • Read the docs corrections in docs/kobo-api.md (diff against the merged docs/kobo-api from #22).
  • Verify go test ./... green on this branch.
  • Verify abase server runs the teardown build cleanly (no transcript infrastructure, no spike env vars). Already deployed.
  • Confirm the device's normal library sync still works end-to-end — spike cleanup should be invisible to Nickel.

Follow-ups

  • Issue #23 — auth model (TOFU on kobo_user_id) for reading-services endpoints.
  • Implementation spec for annotation sync — separate sub-project, separate branch.
Research spike into whether Kobo annotation sync is reachable via the `/api/kobo/*` sync API. **Yes, feasible** — wire shapes captured for the three observed reading-services endpoints. Follow-up implementation spec is a separate sub-project; auth model tracked in #23. ## Summary - Spec: `docs/superpowers/specs/2026-04-20-kobo-annotations-spike-design.md` - Plan: `docs/superpowers/plans/2026-04-20-kobo-annotations-spike.md` - **Findings:** `docs/kobo-annotations-spike-findings.md` — methodology + results + wire-shape tables + recommendation. ## Key discoveries - Annotation sync runs in Nickel when the device is signed in to a Kobo cloud account. Traffic goes to `reading_services_host` (default `readingservices.kobo.com`) with a Kobo-cloud-issued Bearer JWT. - Three endpoints observed + characterised: `POST /api/v3/content/checkforchanges`, `PATCH /api/v3/content/{bookID}/annotations`, `GET /api/UserStorage/Metadata`. - Nickel treats `reading_services_host` as host-only (path component discarded), so requests arrive at server root — not under `api_endpoint`'s prefix. - `nc :5001` was the tool that unstuck the spike after passive catch-all + probe stubs both yielded null results. ## Docs corrections (spin-off) Two claims in `docs/kobo-api.md` that the spike proved wrong, now corrected in the same PR: 1. Automatic host rewriting is **storeapi-specific**, not universal. 2. Init write-back is **selective**, not universal — only a subset of Resources keys get persisted. Plus a new Chapter 3 subsection documenting `reading_services_host`'s host-only quirk. ## Infrastructure aftermath - Permanent: low-noise `slog.Info("unknown kobo endpoint", ...)` drift detector in the catch-all. - Removed at teardown: transcript recorder, probe handlers, spike env vars, reading-services + notebooks init overrides, broadened catch-alls. ## Test plan - [ ] Read the findings doc; confirm wire shapes match your mental model. - [ ] Read the docs corrections in `docs/kobo-api.md` (diff against the merged `docs/kobo-api` from #22). - [ ] Verify `go test ./...` green on this branch. - [ ] Verify abase server runs the teardown build cleanly (no transcript infrastructure, no spike env vars). Already deployed. - [ ] Confirm the device's normal library sync still works end-to-end — spike cleanup should be invisible to Nickel. ## Follow-ups - Issue #23 — auth model (TOFU on `kobo_user_id`) for reading-services endpoints. - Implementation spec for annotation sync — separate sub-project, separate branch.
arne added 19 commits 2026-04-20 21:05:44 +02:00 
Two-phase investigation: passive catch-all logging, then active probing
of calibre-web/grimmory-derived candidate endpoints. Deliverables: a
findings report, any discovered endpoints documented in kobo-api.md,
and a permanent low-noise catch-all log line.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Task-by-task plan covering code scaffolding (catch-all log line,
transcript recorder, probe stubs), deploy, two-phase observation
protocol, findings writeup, and teardown.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Books migrated to books.a.bas.es in the books container on host abase.
Binary path is /usr/local/bin/books; data dir is /var/lib/bases/books/data.
Deploy sequence now uses stop → push → start rather than restart-in-place.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When BOOKS_KOBO_ANNOTATION_PROBES=1, the /v1/initialization response
rewrites the "notebooks" template onto our server so any Nickel
annotation code paths route through our catch-all for observation.
Removed at spike teardown along with probes.go.
Documents what the spike discovered: annotation sync via Kobo API is
feasible when the device is signed in to Kobo cloud. Wire shapes for
the three observed endpoints (checkforchanges, UserStorage/Metadata,
content/{id}/annotations) plus three protocol quirks to fold back
into docs/kobo-api.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Annotations research spike produced evidence that two load-bearing
claims in the reference doc were wrong:

1. Automatic host rewriting is specific to storeapi.kobo.com, not
   universal. Other Kobo hosts (readingservices, oauth, authorize,
   ereaderfiles, etc.) stay on their literal destinations unless
   explicitly overridden via /v1/initialization.

2. Init write-back is selective. image_* and reading_services_host
   are persisted to Kobo eReader.conf; library_* overrides are not
   (they still work via api_endpoint + hardcoded paths).

Also added a new Chapter 3 subsection documenting the quirk that
reading_services_host is treated as host-only — overriding it to a
server with a base path results in requests at server root, not
under the base. Relevant to follow-up annotations implementation
spec.
Remove spike-only files and env vars:
- internal/kobo/transcript.go + tests
- internal/kobo/probes.go + tests (including root-level reading-services
  handlers, notebooks override, broadened catch-all)
- BOOKS_KOBO_TRANSCRIPT_ENABLED / BOOKS_KOBO_ANNOTATION_PROBES env vars
- reading_services_host / notebooks init overrides
- TranscriptRecorder wiring through kobo.New + main.go
- StartNewSession call in LibrarySync

Kept permanently:
- slog.Info("unknown kobo endpoint", ...) in catch-all — low-noise
  drift detector.

Spike findings in docs/kobo-annotations-spike-findings.md. Follow-up
annotation implementation spec tracked via issue #23 (auth model) and
the findings doc's recommendations.
arne merged commit 60fcfdb3e7 into main 2026-04-20 21:08:59 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
arne/books!24
No description provided.