Foundation: importer pipeline + storage #1

Merged

arne merged 15 commits from foundation into master

2026-04-10 20:32:45 +02:00

arne commented

2026-04-10 20:25:45 +02:00

Owner

First of four plans implementing the books server design at docs/superpowers/specs/2026-04-10-books-server-design.md. Delivers the foundation: a books Go binary that watches an inbox directory, parses dropped epubs, persists metadata in SQLite, and files them into a managed library tree under <author>/<title>.epub. No HTTP, no auth, no Kobo — those come in plans 2–4.

What's in here

internal/config/ — env-based config loader (BOOKS_DATA_DIR plus three path overrides)
internal/store/ — SQLite store with embedded migrations. Full 11-table schema applied up front (most tables empty until later plans). Typed CRUD for books and authors. MaxOpenConns(1) so the foreign_keys pragma actually applies to every query.
internal/library/ — sanitised path components (handles /, \, :, control chars, whitespace, unicode), Place/WriteCover/Remove with os.Rename + EXDEV-style copy fallback. Atomic via .tmp rename.
internal/epub/ — pure functional OPF parser. Bytes in via io.ReaderAt, Metadata struct + cover bytes out. Supports both EPUB2 (<meta name=\"cover\">) and EPUB3 (properties=\"cover-image\") cover declarations.
internal/importer/ — orchestrates parse → upsert → place → set file path. Duplicate detection by file_hash runs before upsert so re-dropped files are deleted, not re-placed. Failed imports quarantined to inbox/.failed/<name> with sidecar .err.
internal/inbox/ — startup scanner (filepath.WalkDir, skips .failed/ and hidden) + fsnotify watcher with size-stability debouncing.
main.go — wires everything, signal handling, slog logging.

Out of scope (plans 2–4)

Web UI, OIDC, sessions
Device tokens, Kobo sync API
Deployment script, systemd unit, Caddy config

Test plan

go test ./... and go test -race ./... clean across all packages
Manual smoke: drop a non-zip → moves to .failed/ with sidecar .err, server stays up
Manual smoke: drop a constructed epub ("The Smoke Test" / "Anne Author") → lands at library/Anne Author/The Smoke Test.epub

Known follow-ups (deferred to plan 2)

N+1 in ListBooks flagged with TODO(plan-2)
No tests for internal/inbox/watcher.go (fsnotify timing is fiddly)
PRAGMA foreign_keys=1 is on but the junction tables lack REFERENCES clauses
Orphan rows in authors when a book's author list is replaced

Plan: docs/superpowers/plans/2026-04-10-foundation.md.

First of four plans implementing the books server design at `docs/superpowers/specs/2026-04-10-books-server-design.md`. Delivers the foundation: a `books` Go binary that watches an inbox directory, parses dropped epubs, persists metadata in SQLite, and files them into a managed library tree under `<author>/<title>.epub`. No HTTP, no auth, no Kobo — those come in plans 2–4. ## What's in here - `internal/config/` — env-based config loader (`BOOKS_DATA_DIR` plus three path overrides) - `internal/store/` — SQLite store with embedded migrations. Full 11-table schema applied up front (most tables empty until later plans). Typed CRUD for books and authors. `MaxOpenConns(1)` so the `foreign_keys` pragma actually applies to every query. - `internal/library/` — sanitised path components (handles `/`, `\`, `:`, control chars, whitespace, unicode), `Place`/`WriteCover`/`Remove` with `os.Rename` + EXDEV-style copy fallback. Atomic via `.tmp` rename. - `internal/epub/` — pure functional OPF parser. Bytes in via `io.ReaderAt`, `Metadata` struct + cover bytes out. Supports both EPUB2 (`<meta name=\"cover\">`) and EPUB3 (`properties=\"cover-image\"`) cover declarations. - `internal/importer/` — orchestrates parse → upsert → place → set file path. Duplicate detection by `file_hash` runs *before* upsert so re-dropped files are deleted, not re-placed. Failed imports quarantined to `inbox/.failed/<name>` with sidecar `.err`. - `internal/inbox/` — startup scanner (`filepath.WalkDir`, skips `.failed/` and hidden) + fsnotify watcher with size-stability debouncing. - `main.go` — wires everything, signal handling, slog logging. ## Out of scope (plans 2–4) - Web UI, OIDC, sessions - Device tokens, Kobo sync API - Deployment script, systemd unit, Caddy config ## Test plan - [x] `go test ./...` and `go test -race ./...` clean across all packages - [x] Manual smoke: drop a non-zip → moves to `.failed/` with sidecar `.err`, server stays up - [x] Manual smoke: drop a constructed epub (\"The Smoke Test\" / \"Anne Author\") → lands at `library/Anne Author/The Smoke Test.epub` ## Known follow-ups (deferred to plan 2) - N+1 in `ListBooks` flagged with `TODO(plan-2)` - No tests for `internal/inbox/watcher.go` (fsnotify timing is fiddly) - `PRAGMA foreign_keys=1` is on but the junction tables lack `REFERENCES` clauses - Orphan rows in `authors` when a book's author list is replaced Plan: `docs/superpowers/plans/2026-04-10-foundation.md`.

arne added 15 commits

2026-04-10 20:25:45 +02:00