From 9699ca330802843ea259ccf6fc4c023be69aabc1 Mon Sep 17 00:00:00 2001 From: Chmouel Boudjnah Date: Mon, 7 Sep 2026 09:05:29 +0200 Subject: [PATCH] docs: write down how a fresh device starts up The batch resolve route joins docs/integrating.md and the OpenAPI description, and ADR-0035 records why a route for many books exists when one for a single book already does. Alongside it, the sequence a device follows on its first day is now spelled out: name everything, ask where each book stands through /v1/heads, then sync normally. Every client wants that order and none of it was written down, which is part of why the first attempt at it took several refreshes to settle. --- docs/adr/0003-catalog-work-identity.md | 4 +- .../adr/0035-naming-a-shelf-is-one-request.md | 125 ++++++++++++++++++ docs/adr/README.md | 1 + docs/integrating.md | 87 +++++++++++- docs/openapi.yaml | 105 +++++++++++++++ 5 files changed, 320 insertions(+), 2 deletions(-) create mode 100644 docs/adr/0035-naming-a-shelf-is-one-request.md diff --git a/docs/adr/0003-catalog-work-identity.md b/docs/adr/0003-catalog-work-identity.md index ebad9db..0d5ce15 100644 --- a/docs/adr/0003-catalog-work-identity.md +++ b/docs/adr/0003-catalog-work-identity.md @@ -90,7 +90,9 @@ identity. and tenant-isolation/concurrency tests. **Implemented.** 2. Add an explicit catalog-book resolve operation for sync-capable clients. **Implemented** as `POST /v1/books/{id}/resolve`, which requires both - `library-read` and `sync`. + `library-read` and `sync`. [ADR-0035](0035-naming-a-shelf-is-one-request.md) + adds `POST /v1/books/resolve` for a whole shelf at once; it goes + through the same resolver and decides nothing differently. 3. Add maintenance backfill and mapping repair for split/merge. **Implemented.** Split and merge repair cover unavailable current files, and `liseur-sync admin backfill-works ` pre-resolves a user's whole diff --git a/docs/adr/0035-naming-a-shelf-is-one-request.md b/docs/adr/0035-naming-a-shelf-is-one-request.md new file mode 100644 index 0000000..eaa4c35 --- /dev/null +++ b/docs/adr/0035-naming-a-shelf-is-one-request.md @@ -0,0 +1,125 @@ +# ADR-0035: Naming a shelf is one request + +- **Status:** Accepted +- **Date:** 2026-09-07 +- **Depends on:** [ADR-0003](0003-catalog-work-identity.md) +- **Amends:** [ADR-0003](0003-catalog-work-identity.md), which gave + catalog resolution one route for one book. It gets a second shape for + many, deciding nothing differently. + +## Context + +Positions and sessions attach to a work, never to a catalog book, so a +book with no work on this server can neither send a position nor receive +one. `POST /v1/books/{id}/resolve` (ADR-0003) gives a book its work, one +book per request. + +That is the right shape for the case it was designed for: a reader +downloads a book and the client names it. It is the wrong shape for the +other case, which is every device's first day. A phone signing in to an +existing account has a name here for *none* of its books, so a library of +four hundred needs four hundred requests before anything syncs at all. + +Clients therefore ration it, and rationing has a cost the ration was +never meant to pay. Liseur named twenty-five books a run, and a run +happened when the reader pulled to refresh. Each refresh named the next +twenty-five, fetched their history, and put it on the shelf — so the +shelf arrived in batches over several refreshes, each batch landing on +top of the last. That is [chmouel/liseur#180], and while the ordering +half of it was the client's own bug, the batching half is this route's +shape. + +A client cannot fix it by asking faster. Four hundred requests is four +hundred round trips against a server that may be a Raspberry Pi behind a +domestic uplink, and the reader is watching. + +## Decision + +`POST /v1/books/resolve` resolves up to 500 books in one request. + +**It decides nothing that the single route does not.** Both go through +`resolveBookWork`, which is the whole of what resolving means: the same +evidence read off the catalog, the same `DecideWorkResolution`, the same +per-user mapping. A batch route that could reach a different answer would +be a second protocol wearing the first one's name. + +**Each book is resolved in its own transaction, and takes its own +answer.** One book the server cannot place settles nothing about the +others. This is the load-bearing decision and it is what makes the route +worth having: a client naming four hundred books must not lose the three +hundred and ninety-eight that worked because two did not. + +**So the response is `200` with per-item results, not a status code.** +There is no single status that honestly describes five hundred answers. +`error` is the per-item spelling of what the single route answers with — +`not_found` for its `404`, `ambiguous` (with the works) for its `409` — +and a doubtful title/author match is *not* an error: it comes back with +`confidence: "low"` and nothing stored, exactly as it does one at a time. + +**A failure that is the server's, rather than a book's, fails the +request.** A database that has gone away would otherwise be reported as a +permanent refusal of each book in turn, which a client cannot tell from +"this book will never resolve" — and would file away as a settled +answer. Only `ErrNotFound` and `ErrConflict` become items. + +**The same scope pair, `library-read` + `sync`.** The reasoning is +ADR-0003's unchanged: it reads the catalog and writes the caller's work +graph. A batch route with a weaker gate would be a way around the check +the single route exists to enforce. + +**`confirmed` applies to the whole batch**, and the documentation says to +send it only when there is no reader to ask. Accepting a title/author +guess is a question about one book; answering it five hundred at a time +is how two different books quietly become one reading history, which is +what ADR-0003 exists to prevent. The field is kept rather than dropped +because a migration tool has nobody to ask and knows what it is holding. + +**500 ids**, refused above that with `batch_too_large` and the limit, the +shape `POST /v1/ops` and `POST /v1/sessions` already use and clients +already know how to cut a request against. The bound is not about the +body — a list of ids is small — but about the work behind it, which is +five hundred short transactions. + +### What is *not* here + +`GET /v1/heads` already answers "where does each of my works stand" for a +whole account, and a freshly named library is exactly that question. It +needs no server change; it needed a client that knew to ask it. The +sequence — batch resolve, then heads, then the ordinary delta feed — is +now written down in `docs/integrating.md`, because every client will want +it and none of it was spelled out. + +The cursor does not move for a heads seed. `snapshot_seq` belongs to the +resync protocol, and the rule that a cursor advances only in the +transaction that stores the page it covers is unchanged. + +## Consequences + +- A fresh device names its whole catalog in one request and learns where + every book stands in one more. Two round trips instead of four hundred. +- A client that predates the route, or a server that does, keeps working: + the batch answers `404` and the per-book route is still there. Clients + are told to remember the probe rather than repeat it. +- The single route stays. It is the right shape for a book the reader + just downloaded, and it is the only place a doubtful match can be + confirmed for one book. +- Per-item errors mean a client must read the results rather than the + status. The alternative — a batch that is all-or-nothing — was + rejected: one unplaceable book in a library of four hundred is + ordinary, and it must not be able to stop the other three hundred and + ninety-nine. + +## Acceptance criteria + +- A batch of good ids names each book, once, with the catalog's own + evidence, and replays to the same works without creating more. +- A mixed batch returns the good books' works alongside `not_found` and + `ambiguous` for the ones that failed, and neither refusal leaves + anything in the work graph. +- A doubtful match returns `confidence: "low"` and stores no mapping. +- `library-read`-only and `sync`-only tokens are both refused, and a + refused batch maps nothing. +- Over 500 ids answers `batch_too_large` with the limit; empty and + malformed bodies are `400`, never `5xx`. + +[chmouel/liseur#180]: https://github.com/chmouel/liseur/issues/180 diff --git a/docs/adr/README.md b/docs/adr/README.md index 56ef42e..a72b6f2 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -59,6 +59,7 @@ loose ends that must come before any of it, are in | [0032](0032-reader-pages-are-readium-positions.md) | The reader's page is the app's page | Later | Accepted; implemented | Pages counted as Readium positions so the browser and the app name the same page; display only, no server or API change; three recorded patches to the vendored engine; amends [0031](0031-web-reader-footer.md) and [0012](0012-reader-engine-foliate.md) | | [0033](0033-a-device-outlives-its-token.md) | A device outlives its token | MVP | Accepted; implemented | `POST /v1/tokens` inherits a `device_id` the account already carried, `400 unknown_device` otherwise; `account_id` on `GET /v1/token`; the store keeps comparing devices; amends [0016](0016-token-self-introspection.md) | | [0034](0034-live-notifications-say-only-that-something-changed.md) | A live notification says only that something changed | Client | Accepted | `GET /v1/events` as topic-only SSE, published post-commit by the store, authorized per topic; correctness stays in the existing feeds; amends [0007](0007-web-reader.md) | +| [0035](0035-naming-a-shelf-is-one-request.md) | Naming a shelf is one request | Client | Accepted; implemented | `POST /v1/books/resolve` for up to 500 books, one transaction and one answer each, `200` with per-item `not_found`/`ambiguous`; same `library-read` + `sync` pair; amends [0003](0003-catalog-work-identity.md) | ## Convention diff --git a/docs/integrating.md b/docs/integrating.md index 1d3a8ac..ae646a0 100644 --- a/docs/integrating.md +++ b/docs/integrating.md @@ -1161,6 +1161,91 @@ This route needs both `library-read` and `sync`, the only one that does: it reads the catalog and writes your work graph. A catalog-only token gets `403`, and so does a sync-only one. +#### Naming a whole shelf at once + +One request per book is fine for a book the reader has just downloaded. +It is not fine for a device that has just signed in, which has a name +here for none of its books — and a book with no work can neither send a +position nor receive one, so nothing syncs until they all have names. + +`POST /v1/books/resolve` does the same join for up to 500 books: + +```json +POST /v1/books/resolve +{"book_ids": ["…", "…", "…"]} +``` + +```json +{"results": [ + {"book_id": "…", "work_id": "…", "confidence": "high", "created": true, + "identifiers": [{"kind": "sha256", "value": "…"}]}, + {"book_id": "…", "error": "ambiguous", "works": ["…", "…"]}, + {"book_id": "…", "error": "not_found"} +]} +``` + +Results come back in the order you asked for them. + +Each book is resolved on its own, so one book the server cannot place +settles nothing about the others: it takes its refusal in its own entry +and the rest of the batch stands. That is why the answer is `200` with +per-item results rather than a status code — refusing the batch would +throw away the four hundred and ninety-eight that worked. + +`error` is the per-item spelling of what the single route answers with: +`not_found` for its `404`, and `ambiguous`, with the works, or +`conflict` for its two `409`s. `conflict` means the work graph moved +under the write; that book alone is worth sending again, since the next +attempt sees the graph that displaced it. The other two are settled +answers. + +A doubtful match is *not* an error — it comes back with +`confidence: "low"` and nothing stored, exactly as it does one at a +time. + +Over 500 ids the whole request is refused with `code: batch_too_large` +and the `limit` it will take, so cut the list down and send it again. An +empty `book_ids` is a `400`. + +`confirmed` applies to every book in the batch. Send it only if you have +no reader to ask: accepting a title/author guess is a question about one +book, and the single-book route is where that answer belongs. + +A server that predates this route answers `404` or `405`. Fall back to +`POST /v1/books/{id}/resolve` per book, and remember the answer rather +than probing on every run. + +### Bootstrapping a fresh device + +A device signing in to an existing account has to do three things before +its shelf looks right, and the order matters. + +1. **Name everything.** Batch-resolve the catalog books as above. + Anything the reader side-loaded resolves through `POST + /v1/works/resolve` with identifiers you compute locally. + +2. **Ask where each book stands.** Everything the account did before + this device had a name for a book happened *behind* your cursor, so + the delta feed will never mention it. `GET /v1/heads` answers for the + whole account at once — the newest op per work — which is exactly the + question a device with a freshly named library is asking. Do not + advance your cursor from it: `snapshot_seq` belongs to the resync + protocol, and a seed is not a pull. + + `/v1/heads` carries only the newest record per work, so if one comes + back in a shape you cannot read, ask `GET /v1/works/{id}/positions` + for that work alone rather than treating it as "nothing here": an + older op behind it may still hold a position you can use. + +3. **Sync normally.** From here `GET /v1/changes?since=…` is enough, and + the cursor advances in the transaction that stores the page it + covers. + +Steps 1 and 2 are two requests for a whole library. Do them in one go +rather than a few books at a time: a reader who sees the shelf fill in +batches over several refreshes will refresh again, and every batch that +lands stamps its arrival on books that were read weeks ago. + ## KOReader through OPDS KOReader can browse the catalog with no plugin at all. Add an OPDS @@ -1211,7 +1296,7 @@ round-trips verbatim to kosync pulls. bug, report it. A refusal a client can recover from also carries a machine-readable `code`, currently only `unknown_work` (see [Pushing](#pushing)). -- Batch limits: 500 ops, 1000 sessions per request, 1 +- Batch limits: 500 ops, 1000 sessions, 500 books per batch resolve, 1 MiB body, 16 KiB per `locator`. - Auth endpoints are rate-limited per IP (429 + `Retry-After`). diff --git a/docs/openapi.yaml b/docs/openapi.yaml index 9829c8f..95bb3ef 100644 --- a/docs/openapi.yaml +++ b/docs/openapi.yaml @@ -1955,6 +1955,73 @@ paths: error: { type: string } works: { type: array, items: { type: string } } + /v1/books/resolve: + post: + tags: [library, sync] + summary: Join many catalog books to your own works + security: [{ deviceToken: [] }] # requires BOTH library-read and sync + description: | + The same join as `POST /v1/books/{id}/resolve`, for a shelf at a + time. It exists for the moment a device signs in to an account and + has a name here for none of its books: a book with no work can + neither send a position nor receive one, so asking one request per + book is what made a fresh library arrive in batches. + + Each book is resolved in its own transaction and takes its own + answer, so one book the server cannot place settles nothing about + the others. The response is therefore `200` with per-item results + rather than a status code: no single status could describe five + hundred answers, and refusing the batch would throw away the ones + that worked. + + Results come back in the order the ids were sent. A refused one + carries `error`: `not_found`, `ambiguous` with the works, or + `conflict` when the work graph moved under the write — the only + one of the three worth sending again. + + `confirmed` applies to the whole batch and should be sent only by a + caller with no reader to ask. Accepting a doubtful title/author + match is a question about one book, and the single-book route is + where the answer belongs. + requestBody: + required: true + content: + application/json: + schema: + type: object + required: [book_ids] + properties: + book_ids: + type: array + minItems: 1 + maxItems: 500 + items: { type: string, format: uuid } + confirmed: + type: boolean + description: Accept low-confidence title/author matches. + responses: + "200": + description: One result per requested book, in the order asked for + content: + application/json: + schema: + type: object + required: [results] + properties: + results: + type: array + items: { $ref: "#/components/schemas/CatalogResolutionItem" } + "400": + description: | + Malformed or empty body, or more than 500 ids — in which case + the answer carries `code: batch_too_large` and the `limit` the + server will take. + content: + application/json: + schema: { $ref: "#/components/schemas/BatchItemError" } + "403": { $ref: "#/components/responses/Error", description: "The token lacks library-read or sync" } + "413": { $ref: "#/components/responses/Error", description: "Request body too large" } + /v1/books/{id}/download: get: tags: [library] @@ -2395,6 +2462,44 @@ components: kind: { type: string, enum: [sha256, partial-md5, source, dc, ta] } value: { type: string } + CatalogResolutionItem: + type: object + description: | + One book's answer within a batch resolve. Either it resolved, and + carries the same fields `CatalogResolution` does, or it did not, + and carries the reason instead. + + A doubtful match is not a refusal: the work comes back with + `confidence: low` and nothing is stored, exactly as on the + single-book route. + required: [book_id] + properties: + book_id: { type: string } + work_id: { type: string } + confidence: { type: string, enum: [high, low] } + created: { type: boolean } + identifiers: + type: array + description: The catalog's evidence, strongest first. + items: + type: object + required: [kind, value] + properties: + kind: { type: string, enum: [sha256, partial-md5, source, dc, ta] } + value: { type: string } + error: + type: string + enum: [not_found, ambiguous, conflict] + description: | + Why this book alone was refused. `not_found` is the batch + spelling of the single route's `404`, `ambiguous` of its + `409`. `conflict` means the work graph moved under the write + and the book is worth asking about again. + works: + type: array + description: With `ambiguous`, the works the identifiers landed between. + items: { type: string } + CatalogBook: type: object description: |