feat(local): transcribe recordings as sources in the ingest worker

Step 5 of the local speech recognition proposal, backend half. A recording
uploaded as a source becomes a timestamped transcript in its content, so
chunking, search, citations and the agent take it unchanged, without one long
file holding every other upload behind it on the one-thread ingest queue.

- Upload: .wav .mp3 .flac .ogg .oga with server-owned audio/* types, checked
  by their bytes (RIFF/WAVE, ID3 or an MPEG frame, fLaC, Ogg with a Vorbis
  first packet). Ogg Opus is rejected per file: "Opus support comes in a
  later update". can_hold_figures is false for audio/*, so Docling never
  reads a recording.
- worker/audio_decode/: the worker's own binary, started as
  `worker --decode-audio <in> <out>` ahead of the consumer's imports, checks
  the disk from the file's header length, streams it through pyminiaudio
  (MIT) to 16 kHz mono s16 and exits when its parent does. A malformed file
  kills the child, not the consumer. miniaudio joins pyproject, uv.lock and
  worker.spec (with its licence metadata).
- worker/ingestion/audio/, reached from parsing before Docling: readiness
  through the API's status route, the decode, Silero over the file a minute at
  a time (the API's reader, now able to stream), spans padded and joined,
  windows of at most 28 s cut at the longest pause or a run's quietest
  100 ms, each window sent to /internal/transcription/windows with retries and
  waits, windows.jsonl and progress.json as the checkpoint, a progress event
  per window, then the header and [hh:mm:ss] paragraphs, segments.json and
  transcription {language, duration_s, model} in the row's metadata.
- Yielding: after a window or a 503 waiting, once a minute has passed and a
  source that is not a recording is pending, the job raises JobYieldedError;
  the pipeline sets the row back to pending (jobs.pause_job) and queues it at
  PRIORITY_BULK. Recordings never yield to each other.
- No speech model: the job fails without retry and marks
  transcription.awaiting; choosing a speech model, by the route or an install
  with select, queues every waiting recording again. Refusals (no room, no
  speech, no audio, undecodable, memory short for 10 minutes) are
  CancelExecution(retry=False); every other retry resumes from windows.jsonl
  when the model and language intent match.
- DocumentRead gains transcription {language, progress, awaiting} for audio
  rows, progress read from progress.json; InternalEvent and the SSE frame
  gain an optional transient progress {done_s, total_s}.

Measured here: an 8.6-minute file from pending to ready in 60 s at 5.83% word
error; an hour of MP3 decodes in 3.6 s; miniaudio's linear resampler read the
44.1 kHz copy at 5.91% against scipy's 5.83%, under the proposal's 2% relative,
so it stays. Docs: documents.md gains Recordings; data-model.md, overview.md,
selection.md and packaging.md follow; the proposal records the two step 5
measurements.

.github/workflows/build-audiocpp.yml

@@ -22,9 +22,12 @@ on:
       - surfsense_local/backend/modules/llm/catalog/local/engines/audiocpp/**
       - surfsense_local/backend/modules/llm/catalog/local/manifest/models.json
       - surfsense_local/backend/modules/llm/transcription/**
+      - surfsense_local/backend/worker/ingestion/audio/**
+      - surfsense_local/backend/worker/audio_decode/**
       - surfsense_local/backend/tests/packaging/test_audiocpp_voicing.py
       - surfsense_local/backend/tests/packaging/test_audiocpp_transcription.py
       - surfsense_local/backend/tests/packaging/test_audiocpp_transcription_config.py
+      - surfsense_local/backend/tests/packaging/test_recording_transcription.py
       - .github/workflows/build-audiocpp.yml
 
 permissions:
@@ -167,8 +170,9 @@ jobs:
 
       # Kokoro voices sentences in six languages and the transcription server,
       # started as Electron's second sidecar from the transcription.json the app
-      # writes, writes them down through the app's client. The test checks the
-      # file against the manifest's hash; Silero's check runs once it is staged.
+      # writes, writes them down through the app's client; then a recording goes
+      # through ingest, the API and that server. The test checks the file
+      # against the manifest's hash; Silero's check runs once it is staged.
       - name: Gate on speech recognition through the app
         working-directory: surfsense_local/backend
         env:
@@ -182,7 +186,7 @@ jobs:
           fi
           cp "$RECOGNISER" "$SURFSENSE_TEST_AUDIO_MODELS/"
           uv run scripts/fetch_silero_vad.py models
-          uv run pytest tests/packaging/test_audiocpp_transcription_config.py tests/packaging/test_audiocpp_transcription.py -m packaging
+          uv run pytest tests/packaging/test_audiocpp_transcription_config.py tests/packaging/test_audiocpp_transcription.py tests/packaging/test_recording_transcription.py -m packaging
 
       # An artifact drops symlinks and file modes; a tar keeps both.
       - name: Pack

.github/workflows/release-local.yml

@@ -173,8 +173,9 @@ jobs:
           "$WORKER" --check-vision-runtime
 
       # A script is not analysed when the worker is frozen, so a library the
-      # spec leaves out shows only when a script runs in the shipped worker.
-      - name: Run document scripts in the frozen worker
+      # spec leaves out shows only when a script runs in the shipped worker;
+      # the same holds for the decode child's miniaudio.
+      - name: Run document scripts and decode a recording in the frozen worker
         working-directory: surfsense_local/backend
         # addopts pins -m 'not packaging'; the later -m on the command line wins.
         run: |

docs/architecture/data-model.md

@@ -62,7 +62,7 @@ The API seeds one workspace, "My Workspace", at startup when none exists. Deleti
 | `content` | the markdown body: extracted text, the note itself, or an artifact's body |
 | `content_hash` | SHA-256 of a file's bytes, equal to `dedup_key` for an uploaded or imported file; `0025` copied it from `dedup_key`; null for notes and artifacts |
 | `dedup_key` | SHA-256 of an uploaded or imported file's bytes; null for notes and artifacts |
-| `document_metadata` | JSON: `mime_type`, `size_bytes`, `suffix` for files; import adds `folder_path`, `source` and the hosted ids |
+| `document_metadata` | JSON: `mime_type`, `size_bytes`, `suffix` for files; import adds `folder_path`, `source` and the hosted ids; a recording's ingest adds `transcription`, `{language, duration_s, model}` once transcribed or `{awaiting: true}` while it waits for a speech model ([`documents.md`](documents.md#recordings)) |
 | `embedding_index_id` | foreign key to `embedding_indexes`: the index its chunks were embedded into; null until its first ingest |
 | `folder_id` | foreign key to `folders`, `ON DELETE SET NULL`, from `0025`: every `FILE` and `NOTE` is filed; null is an unfiled artifact, or a source the backstop unfiled, which scopes and the tree treat as the top of the Library |
 | `created_at`, `updated_at` | |
@@ -184,7 +184,10 @@ One row per run of a plugin's action. What the run produced is not here: the plu
 └── data/
     └── workspaces/<workspace_id>/
         ├── documents/<document_id>/
-        │   └── <file name>       an uploaded or imported file, under its own name
+        │   ├── <file name>       an uploaded or imported file, under its own name
+        │   ├── figures/          the pictures ingest kept, and figures.json
+        │   └── transcript/       a recording's: segments.json once transcribed;
+        │       └── work/         the decoded audio, plan and windows a job resumes from
         ├── chats/<thread_id>/
         │   └── <sha256>.<ext>    an image a turn carried, normalised to PNG or JPEG
         ├── artifacts/<artifact_id>/

docs/architecture/documents.md

@@ -1,8 +1,8 @@
 # Documents and ingest
 
-A workspace holds a library of sources: uploaded files, notes written in the app, and the bodies of Studio artifacts. Every source ends up the same way, as a markdown body on its `documents` row cut into passages that are embedded and indexed, so chat and search treat them alike. The API only stores bytes and enqueues a job; parsing, chunking and embedding happen in the ingest worker, one document at a time, and a document is `ready` only once it is searchable. Ingest also keeps a file's pictures beside its original, as [figures](#figures) the agent can place in a document. Studio indexes an artifact's body itself, in its own worker, with the same chunking and embedding code.
+A workspace holds a library of sources: uploaded files, notes written in the app, and the bodies of Studio artifacts. Every source ends up the same way, as a markdown body on its `documents` row cut into passages that are embedded and indexed, so chat and search treat them alike. The API only stores bytes and enqueues a job; parsing, chunking and embedding happen in the ingest worker, one document at a time, and a document is `ready` only once it is searchable. Ingest also keeps a file's pictures beside its original, as [figures](#figures) the agent can place in a document, and turns a recording into a timestamped transcript through the speech model ([Recordings](#recordings)). Studio indexes an artifact's body itself, in its own worker, with the same chunking and embedding code.
 
-**Code:** [`modules/workspaces/`](../../surfsense_local/backend/modules/workspaces/), [`modules/documents/`](../../surfsense_local/backend/modules/documents/), [`worker/ingestion/`](../../surfsense_local/backend/worker/ingestion/), [`worker/jobs.py`](../../surfsense_local/backend/worker/jobs.py), [`shared/queue.py`](../../surfsense_local/backend/shared/queue.py), [`frontend/src/features/sources/`](../../surfsense_local/frontend/src/features/sources/)
+**Code:** [`modules/workspaces/`](../../surfsense_local/backend/modules/workspaces/), [`modules/documents/`](../../surfsense_local/backend/modules/documents/), [`worker/ingestion/`](../../surfsense_local/backend/worker/ingestion/), [`worker/audio_decode/`](../../surfsense_local/backend/worker/audio_decode/), [`worker/jobs.py`](../../surfsense_local/backend/worker/jobs.py), [`shared/queue.py`](../../surfsense_local/backend/shared/queue.py), [`frontend/src/features/sources/`](../../surfsense_local/frontend/src/features/sources/)
 **Decisions:** [ADR 0003](../adr/0003-artifacts-as-documents.md), [ADR 0007](../adr/0007-bundled-embeddings.md), [ADR 0036](../adr/0036-the-index-records-its-embedder.md), [ADR 0008](../adr/0008-two-job-queues.md), [ADR 0009](../adr/0009-freshness-by-invalidation.md)
 
 ## Workspaces
@@ -43,7 +43,8 @@ Chunks have no router of their own.
 
 - Filters are `?document_type=` and `?status=`, both repeatable, and `?folder_id=`. Paging is `?limit=` (default 50, at most 200) and `?offset=`; the sources panel reads pages of 200 until one comes short, so it holds every source, not the newest 50. Rows come newest first, by `created_at` and then id, since one upload batch shares a second, so a fresh upload leads the sources list while it is still indexing.
 - `ARTIFACT` rows are included and the caller filters them out. The sources panel asks for `document_type=FILE&document_type=NOTE`, so Studio output does not look like something the user uploaded.
-- A row carries `id`, `title`, `document_type`, `status`, `error_message`, `folder_id` and timestamps, but no `content`. The list is polled while ingest runs, and a body per row would ride along on every poll; `GET .../documents/{document_id}` returns one row with it.
+- A row carries `id`, `title`, `document_type`, `mime_type`, `status`, `error_message`, `folder_id`, timestamps and `transcription`, but no `content`. The list is polled while ingest runs, and a body per row would ride along on every poll; `GET .../documents/{document_id}` returns one row with it.
+- `transcription` is null for everything but a recording (`mime_type` `audio/*`), which carries `{language, progress, awaiting}`: the language its transcript was heard in once it is done; `{done_s, total_s}` read from the job's `progress.json` while it is `pending` or `processing`, so a reload or a late listener sees how far it has come; and whether it failed only for want of a speech model ([Recordings](#recordings)).
 
 ## What is editable
 
@@ -70,14 +71,14 @@ The sources panel shows the Library as a tree ([`features/sources/tree/`](../../
 
 - The request is multipart, one `files` part per file. Each file streams to a temporary file inside the workspace directory in 1 MB reads, hashed with SHA-256 on the way past and never held whole in memory. A file over 500 MB is refused with `413`. The temporary file sits on the same filesystem as its destination, because a rename is only atomic within one filesystem.
 - Only a plain extension, a dot and 1 to 16 letters or digits, is validated from the client's filename. The file is stored alone in a directory built from row ids, `data/workspaces/<workspace>/documents/<document>/`, under the client's filename made safe for every platform ([`original_file.py`](../../surfsense_local/backend/modules/documents/original_file.py)): the last path part only, with [pathvalidate](https://github.com/thombashi/pathvalidate) removing characters no filesystem accepts and suffixing Windows reserved names, leading dots dropped so the file is never hidden, cut before the extension so the whole name fits in 120 bytes, which keeps a typical Windows path under its 260-character limit, and ending in the validated extension lowercased. A name with nothing left becomes `untitled`, and one that would be `extracted.md` becomes `extracted (1).md` (see [The original file](#the-original-file)). The name reaches the disk; the path never does.
-- Accepted extensions are `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.html`, `.htm`, `.csv`, `.md`, `.markdown`, `.txt`, `.text`, `.png`, `.jpg`, `.jpeg`, `.tif`, `.tiff`, `.bmp` and `.webp`. The bytes must match the extension: the PDF signature; an OOXML zip holding its main part, with at most 10,000 entries and 2 GB unpacked; UTF-8 text with no NUL and few control bytes; or an image's magic number. The MIME type stored is the server's, never the client's. The frontend mirrors the list to filter its file picker and to skip unsupported files, dropped ones included, before uploading.
+- Accepted extensions are `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.html`, `.htm`, `.csv`, `.md`, `.markdown`, `.txt`, `.text`, `.png`, `.jpg`, `.jpeg`, `.tif`, `.tiff`, `.bmp` and `.webp`, and the recordings `.wav`, `.mp3`, `.flac`, `.ogg` and `.oga`. The bytes must match the extension: the PDF signature; an OOXML zip holding its main part, with at most 10,000 entries and 2 GB unpacked; UTF-8 text with no NUL and few control bytes; an image's magic number; or a recording's: RIFF with a WAVE form, an ID3 tag or an MPEG audio frame header (not AAC's), `fLaC`, or an Ogg page whose first packet is Vorbis's. An Ogg Opus file is rejected on its own with "Opus support comes in a later update", since the decoder reads Vorbis only. The MIME type stored is the server's, never the client's: `audio/wav`, `audio/mpeg`, `audio/flac` and `audio/ogg` for recordings. The frontend mirrors the list to filter its file picker and to skip unsupported files, dropped ones included, before uploading.
 - The sources panel takes files two ways: its **+** menu, and files dropped onto the panel, which shows "Drop files to add them as sources" while files are dragged over it ([`use-sources-drop.ts`](../../surfsense_local/frontend/src/features/sources/use-sources-drop.ts)). Both go through the same upload. Dropped files and rows dragged between folders go through [Pragmatic drag and drop](https://atlassian.design/components/pragmatic-drag-and-drop), which also scrolls the list while a drag nears its top or bottom, and draws a dragged row as a small chip. The panel takes no drop while an upload runs, as uploading is disabled then, and a drag carrying no files, such as selected text, does not light it up. A file dropped anywhere else in the window is refused ([`file-drop-guard.ts`](../../surfsense_local/frontend/src/app/file-drop-guard.ts)): unhandled, Chromium opens it in the window, and in Electron that replaces the app, since the main process lets `file:` navigation through.
 - A whole folder comes in two ways: **+ › Upload folder…**, an `<input webkitdirectory>` whose files carry `webkitRelativePath`, and a dropped folder, which the panel walks with `webkitGetAsEntry()` ([`dropped-files.ts`](../../surfsense_local/frontend/src/features/sources/folder-upload/dropped-files.ts)). Either sends each file's path under the folder as `relative_paths`, and a drop on a folder row, or on a source inside one, adds `folder_id`. Before sending, the client skips paths through `.git`, `node_modules`, `__pycache__`, `.DS_Store`, `.obsidian` and `.trash`, sets aside unsupported files and any file over 500 MB, which it lists as rejected rather than letting one fail its request with `413`, and sends the rest in requests of at most 50 files or 200 MB ([`upload-plan.ts`](../../surfsense_local/frontend/src/features/sources/folder-upload/upload-plan.ts)).
 - Files go into the `folder_id` form field's folder, else the top of the Library. `relative_paths`, a JSON list index for index with `files`, carries an added folder's tree: each file lands in the folders its path names below the target (`Research/2024/a.pdf` makes `Research` and `2024`), made as needed with `ensure_folder_path`. Levels past the depth cap are joined with " / " into the deepest level there is room for, so a deep tree arrives whole. A list of the wrong length is `422`; a folder from another workspace is `404`.
 - `dedup_key` is the SHA-256 of the bytes and is unique per folder. Keying on the bytes rather than the filename means the same report saved twice in one folder is one document, while two different files both called `report.pdf` are two. A file whose bytes sit in another folder is created in the new one too, so a copied folder has no holes, and copying it again adds only what is new.
 - A batch is split, not rejected, except that a file over 500 MB fails the whole request with 413 and nothing in it is created. The response lists `created`, `duplicates` (each with the existing document's id and its `folder_id`) and `rejected` (each with a reason), so a dropped folder holding one known file keeps the rest.
 - An upload of more than 20 files enqueues at `PRIORITY_BULK` (10); every other ingest runs at `PRIORITY_INTERACTIVE` (100), so a note written during a long folder copy is read first ([`tasks.py`](../../surfsense_local/backend/modules/documents/tasks.py)).
-- Accepted files become `FILE` rows in `pending`, with `mime_type`, `size_bytes` and `suffix` in `document_metadata`. The files move into place, the transaction commits, and only then is one ingest job enqueued per file, because the worker is another process and would look for rows this request had not yet written.
+- Accepted files become `FILE` rows in `pending`, with `mime_type`, `size_bytes` and `suffix` in `document_metadata`. A recording is accepted with or without a speech model chosen; with none, its ingest waits for one ([Recordings](#recordings)). The 500 MB cap holds for recordings too, with no cap on length: at 64 kbps that is about 18 hours. The files move into place, the transaction commits, and only then is one ingest job enqueued per file, because the worker is another process and would look for rows this request had not yet written.
 
 ## Folders
 
@@ -109,7 +110,7 @@ Sources sit in folders of the workspace's Library ([data model](data-model.md#so
 
 ## The original file
 
-No filename is stored in a row: the document's directory is the record. The original is the one file in it, so not the `figures/` folder, that is not hidden, not `Thumbs.db` or `desktop.ini`, and not `extracted.md`. Any other count, an empty or absent directory included, means the file is missing. Directories written before uploads kept their names hold `original.<ext>` beside an `extracted.md` nothing reads. They are read as they are and deliberately never rewritten: nothing in the index depends on a file's name, so renaming would only change what the file manager shows, and a bulk rename can fail part way on a file that is open or already deleted. Because `extracted.md` is never taken for the original, a legacy directory whose original was deleted reads as missing rather than serving Docling's text as the user's file.
+No filename is stored in a row: the document's directory is the record. The original is the one file in it, so not the `figures/` or a recording's `transcript/` folder, that is not hidden, not `Thumbs.db` or `desktop.ini`, and not `extracted.md`. Any other count, an empty or absent directory included, means the file is missing. Directories written before uploads kept their names hold `original.<ext>` beside an `extracted.md` nothing reads. They are read as they are and deliberately never rewritten: nothing in the index depends on a file's name, so renaming would only change what the file manager shows, and a bulk rename can fail part way on a file that is open or already deleted. Because `extracted.md` is never taken for the original, a legacy directory whose original was deleted reads as missing rather than serving Docling's text as the user's file.
 
 `GET .../original` serves the stored bytes with the document's title as the filename and the stored MIME type, always as an attachment: an HTML or SVG file rendered inline would run its script against the app's origin. It answers `404` when there is no file behind the document, as for a note.
 
@@ -120,13 +121,13 @@ The desktop app previews an original in place of the left sidebar when its MIME
 `ingest_document(document_id)` is declared in [`modules/documents/tasks.py`](../../surfsense_local/backend/modules/documents/tasks.py), because Huey binds a task to its queue when it is decorated and the API is what enqueues it. The task body imports the pipeline lazily and Docling is imported only when the converter is first built, so the API never loads Docling or torch. Each job opens its own database engine.
 
 1. **Start.** `begin_job` marks the row `processing` unless it was cancelled, and the worker notifies the API. The job then reads the active embedding index; with none, because onboarding has not chosen an embedder, it fails with `no embedding model has been chosen yet`.
-2. **Parse** ([`parsing.py`](../../surfsense_local/backend/worker/ingestion/parsing.py)). A `NOTE` or `ARTIFACT` already has its markdown in `content`. A `FILE` with a text extension (`.md`, `.markdown`, `.txt`, `.text`) is read as UTF-8; anything else goes to Docling. A `.pptx` with no slides, such as a template, skips Docling, which refuses a deck with no pages: its text is `PowerPoint template with no slides.`, its slide size and its slide layout names in order, one line each, read with python-pptx ([`slideless_deck.py`](../../surfsense_local/backend/worker/ingestion/slideless_deck.py)), so search finds it and the agent can start a deck from it as a template ([studio](studio.md)). A blank Word or Excel file goes to Docling, which reads it as empty, and is ready with no text. OCR and table structure are switched on, so a scanned PDF is readable and a table survives as a table. RapidOCR's per-line orientation classifier is off: on noisy scans it turned upright lines 180 degrees and they read back as garbage, and dropping it cut the character error rate on generated scans about eightfold. An image is first turned upright by its EXIF orientation, which Docling ignores, and given a DPI that reads it as one A4 page, never finer than the 216 dpi OCR renders at nor coarser than its own DPI ([`image_page.py`](../../surfsense_local/backend/worker/ingestion/image_page.py)); with none, Docling takes it as 72 dpi and OCR upsampled a photo threefold. Images and PDFs share one set of options, so they share one pipeline and its models. Inference gets one thread per physical core, at most 8, as audio.cpp does, instead of Docling's 4: on an 8-core CPU that parsed generated scans, photos and PDFs 1.4 times faster. The converter is built once per process and its models load on the first conversion, so the first document a worker parses pays for the import and the load. `HF_HOME` defaults to the models directory, because Docling would otherwise write weights into `site-packages`, which is read-only in a frozen bundle. When the bundled parser pack is present Docling uses it with RapidOCR, and `HF_HUB_OFFLINE=1` is set before Docling is imported, because huggingface_hub reads it only then, so a PDF or an image converts with networking off. The markdown is kept only in `content`. Beside the original, ingest writes only the pictures Docling found, as [figures](#figures).
+2. **Parse** ([`parsing.py`](../../surfsense_local/backend/worker/ingestion/parsing.py)). A `NOTE` or `ARTIFACT` already has its markdown in `content`. A recording, a `FILE` whose `mime_type` is `audio/*`, is transcribed before Docling is reached, which must never read one ([Recordings](#recordings)). A `FILE` with a text extension (`.md`, `.markdown`, `.txt`, `.text`) is read as UTF-8; anything else goes to Docling. A `.pptx` with no slides, such as a template, skips Docling, which refuses a deck with no pages: its text is `PowerPoint template with no slides.`, its slide size and its slide layout names in order, one line each, read with python-pptx ([`slideless_deck.py`](../../surfsense_local/backend/worker/ingestion/slideless_deck.py)), so search finds it and the agent can start a deck from it as a template ([studio](studio.md)). A blank Word or Excel file goes to Docling, which reads it as empty, and is ready with no text. OCR and table structure are switched on, so a scanned PDF is readable and a table survives as a table. RapidOCR's per-line orientation classifier is off: on noisy scans it turned upright lines 180 degrees and they read back as garbage, and dropping it cut the character error rate on generated scans about eightfold. An image is first turned upright by its EXIF orientation, which Docling ignores, and given a DPI that reads it as one A4 page, never finer than the 216 dpi OCR renders at nor coarser than its own DPI ([`image_page.py`](../../surfsense_local/backend/worker/ingestion/image_page.py)); with none, Docling takes it as 72 dpi and OCR upsampled a photo threefold. Images and PDFs share one set of options, so they share one pipeline and its models. Inference gets one thread per physical core, at most 8, as audio.cpp does, instead of Docling's 4: on an 8-core CPU that parsed generated scans, photos and PDFs 1.4 times faster. The converter is built once per process and its models load on the first conversion, so the first document a worker parses pays for the import and the load. `HF_HOME` defaults to the models directory, because Docling would otherwise write weights into `site-packages`, which is read-only in a frozen bundle. When the bundled parser pack is present Docling uses it with RapidOCR, and `HF_HUB_OFFLINE=1` is set before Docling is imported, because huggingface_hub reads it only then, so a PDF or an image converts with networking off. The markdown is kept only in `content`. Beside the original, ingest writes only the pictures Docling found, as [figures](#figures).
 3. **Chunk** ([`chunking.py`](../../surfsense_local/backend/worker/ingestion/chunking.py)). Chonkie's `RecursiveChunker` splits at the coarsest boundary that fits: a heading, a paragraph, a line, a sentence, a word, then a bare split so an over-long line still ends. Passages are at most 480 tokens by bge-small's tokenizer, under its 512-token limit with room for the two tokens it adds. That tokenizer is pinned to the chunker rather than taken from the active embedder, so changing the embedder would not re-cut documents and move every chunk id. Pieces under 24 characters are merged into a neighbour while splitting, so only a document shorter than that yields a shorter passage. The tokenizer is passed as an object, because one named by a string would reach the network even offline. Each passage records the lines its first and last characters fall on.
 4. **Embed** ([`encoder.py`](../../surfsense_local/backend/modules/embedding/encoder.py)). The active index's model, as its spec says, on onnxruntime's CPU provider: the document prefix, tokenize (truncated at the spec's `max_tokens`), pool, normalise, in batches of 32. For bge-small that is no prefix, 512 tokens, CLS pooling and L2 normalisation. No network and no model server. A vector whose width is not the spec's is refused rather than stored.
 5. **Index** ([`indexing.py`](../../surfsense_local/backend/worker/ingestion/indexing.py)). The document's old chunks are deleted, which clears both index tables through the delete trigger, and the new chunks are inserted with their float32 embeddings, followed by one row per chunk in the active index's vector table, and the document records that index in `embedding_index_id`. The keyword index follows by trigger; the vector cannot, because only ingest holds it. Editing a note therefore stops its old text being findable.
 6. **Finish.** `finish_job` writes `ready` unless a cancel won the race, and beside it only what the job produced: a file's extracted markdown on `documents.content`. A note's content is the user's and is only read, and ingest never writes `title`, so a rename or a note edit committed while the job ran stands, and the ingest that edit enqueued indexes the new text.
 
-Cancellation is checked after parsing and after embedding. On any other failure the worker rolls back and re-reads the row. If the document was deleted or cancelled meanwhile it stops; otherwise it writes `failed` with `error_message` set to the exception type and message, cut to 500 characters, notifies, and re-raises so Huey runs the job again, up to two more times. A file whose bytes Docling cannot load (a `ConversionError` caused by a `DocumentLoadError`, such as a damaged Office file) is not run again, because the same bytes are refused the same way: parsing raises `UnreadableFileError` and the task turns it into Huey's `CancelExecution(retry=False)`. Docling raises `ConversionError` too for a file it cannot open yet, such as one another program holds, and for a model that fails on every page, such as running out of memory; those are tried again. A later success clears the message. The sources panel shows it and offers Retry.
+Cancellation is checked after parsing and after embedding, and between a recording's windows. A recording that gives the queue up rolls back and goes back to `pending` (`pause_job`, unless a cancel won), queued again at `PRIORITY_BULK`: neither a failure nor a retry. On any other failure the worker rolls back and re-reads the row. If the document was deleted or cancelled meanwhile it stops; otherwise it writes `failed` with `error_message` set to the exception type and message, cut to 500 characters (a recording's failures are sentences written for the person, stored as they are), notifies, and re-raises so Huey runs the job again, up to two more times. A file whose bytes Docling cannot load (a `ConversionError` caused by a `DocumentLoadError`, such as a damaged Office file) is not run again, because the same bytes are refused the same way: parsing raises `UnreadableFileError` and the task turns it into Huey's `CancelExecution(retry=False)`. Docling raises `ConversionError` too for a file it cannot open yet, such as one another program holds, and for a model that fails on every page, such as running out of memory; those are tried again. A recording that would fail the same way again is not run again either ([Recordings](#recordings)). A later success clears the message. The sources panel shows it and offers Retry.
 
 ## Figures
 
@@ -143,7 +144,7 @@ Ingest keeps the pictures in each source, so the agent can place a source's char
   - **Word and PowerPoint:** the pixels the file embeds. Docling links no caption for them; PowerPoint gives the slide as the page, Word gives none.
   - **An image source** is its own one figure, turned upright by its EXIF orientation.
   - Pictures in page headers and footers count, since a letterhead logo is the picture a user asks for. A picture repeated with the same pixels is kept once. A picture that refers to a path or a URL rather than carrying its pixels is skipped, so an uploaded page cannot pull an image from the machine into the library. Every figure is a PNG in RGB, or RGBA when it has transparency, which python-docx and ReportLab both read.
-  - A text upload (`.md`, `.txt`, `.csv`, `.html`) keeps none.
+  - A text upload (`.md`, `.txt`, `.csv`, `.html`) keeps none, and neither does a recording: the figures-only pass would hand it to Docling.
 - **Never fails an ingest.** A failure while keeping figures is logged and recorded as `figures.json` with no figures and an `error`, and the text is indexed as usual.
 - **Names.** A figure is `<document id>-<n>`, built from the document when it is listed, never stored ([`source_figures/`](../../surfsense_local/backend/modules/documents/source_figures/)). `list_figures()` answers a source's figures from its `figures.json`; `figure_file()` resolves a name to its PNG only for a source in the caller's workspace.
 - **The figures-only pass.** A source ingested before figures were kept has no `figures.json`. The first `list_figures()` on it, when it is a ready `FILE`, queues `extract_figures(document_id)` on the ingest queue, once, and answers that its figures are being extracted; one still pending or processing gets them from its ingest. The pass reads the original with Docling again, without holding a transaction, and writes the figures, leaving the text and chunks alone ([`figures/backfill.py`](../../surfsense_local/backend/worker/ingestion/figures/backfill.py)). It is not retried, since its failure is recorded in the index.
@@ -162,7 +163,49 @@ Ingest keeps the pictures in each source, so the agent can place a source's char
 - The queue survives restarts. Electron starts the workers beside the API and only the API migrates, so each consumer waits until the database is at the latest revision before taking a job ([`wait_for_schema.py`](../../surfsense_local/backend/worker/wait_for_schema.py)); a job queued before an update would otherwise run against the old schema and fail. A job carries only its task's name, so each consumer imports every task before it starts (`import_tasks()`), and [`tests/integration/test_registration.py`](../../surfsense_local/backend/tests/integration/test_registration.py) fails when that list falls behind the task modules on disk.
 - Jobs are enqueued by upload, note creation, a note content edit and retry, and by import ([`import.md`](import.md)). The queue takes the highest priority first: interactive work at 100, a bulk copy and an import at 10. The figures-only pass, `extract_figures`, shares the queue ([Figures](#figures)).
 - A source the app quit in the middle of ingesting is failed with `interrupted when the app closed` when the ingest worker next starts, before it takes a job ([`interrupted_documents.py`](../../surfsense_local/backend/worker/interrupted_documents.py)). Huey drops a job as it starts it, so nothing else would end it, and Retry accepts only a failed or cancelled document. A source still `pending` is still queued and is left alone.
-- Each transition the ingest worker makes, to `processing`, `ready` or `failed`, sends a `documents` event keyed by document id, and so does each change the API makes itself: a note written, an upload, a rename or edit, a retry, a cancel, and a delete, whose status is `deleted`. The sources panel reloads its list on each event and whenever a dropped stream is back, so a note a plugin writes shows as it is written; it still refetches every 10 seconds while any row is `pending` or `processing`, in case a notice was lost ([`overview.md`](overview.md#freshness)).
+- A recording's job gives the queue up between windows once it has run a minute and a source that is not a recording is pending ([Recordings](#recordings)), so a long file never holds every upload and note behind it.
+- Each transition the ingest worker makes, to `processing`, `ready` or `failed`, sends a `documents` event keyed by document id, and a recording's job sends one per transcribed window carrying a transient `progress` `{done_s, total_s}`, never stored, and so does each change the API makes itself: a note written, an upload, a rename or edit, a retry, a cancel, and a delete, whose status is `deleted`. The sources panel reloads its list on each event and whenever a dropped stream is back, so a note a plugin writes shows as it is written; it still refetches every 10 seconds while any row is `pending` or `processing`, in case a notice was lost ([`overview.md`](overview.md#freshness)).
+
+## Recordings
+
+A recording becomes a source through the speech model chosen for `speech_to_text` ([selection](local-models/selection.md)), on the second audio.cpp server, which the worker reaches only through the API's gate ([admission](local-models/admission.md#speech-recognition)). The transcript is the recording's `content`, so chunking, embedding, search, citations and the agent's `sources/` mirror take it unchanged ([`worker/ingestion/audio/`](../../surfsense_local/backend/worker/ingestion/audio/), [proposal](../proposals/local-speech-to-text.md#audio-files-as-sources)).
+
+Everything the job keeps lives under the document's `transcript/` folder, since the original must stay the only file beside it ([`recordings/layout.py`](../../surfsense_local/backend/modules/documents/recordings/layout.py)): `transcript/work/` holds `audio16k.raw`, `plan.json`, `windows.jsonl` and `progress.json`, and `transcript/segments.json` is written when the transcript is done.
+
+1. **Ready?** The job asks `GET /llm/transcription/status`. With no speech model it stops before decoding ([below](#no-speech-model-yet)). With the server unavailable, as in the seconds after a model is chosen or a crash, it asks again every 5 s for up to 2 minutes, giving the queue up as a window does, then fails with "The speech recognition engine stopped." and Huey tries again.
+2. **Decode** ([`worker/audio_decode/`](../../surfsense_local/backend/worker/audio_decode/)). The worker's own binary, started as `worker --decode-audio <in> <out>` ahead of the consumer's imports, as `--run-document-script` is, reads the file's length from its headers and checks the data volume for room at 115.2 MB an hour plus 64 MiB, then streams it through pyminiaudio (MIT; WAV, MP3, FLAC, Ogg Vorbis) to 16 kHz mono 16-bit PCM in `audio16k.raw.part`, renamed to `audio16k.raw` once every sample is written. It prints its outcome on one line and exits when the worker's end of its stdin closes. A malformed file kills the child, not the consumer, and the job checks for a cancel every second while it waits. An hour of MP3 decodes in 3.6 s. miniaudio's linear resampler cost 0.08 points of word error over scipy's `resample_poly` on an 8.6-minute 44.1 kHz file (5.91% against 5.83%, under the proposal's 2% relative), so it stays.
+3. **Speech** ([`speech_regions.py`](../../surfsense_local/backend/worker/ingestion/audio/speech_regions.py)). Silero VAD, through the API's own reader ([`voice_activity/silero.py`](../../surfsense_local/backend/modules/llm/transcription/voice_activity/silero.py)) fed a minute at a time off disk, finds speech by its rule (0.5 threshold, at least 250 ms); each span is padded by 200 ms and joined to the next across gaps under 300 ms. It ran at 207 times real time on one thread. Without a staged Silero, or should it fail, all of the recording counts as speech, as a dictated phrase does.
+4. **Windows** ([`windows.py`](../../surfsense_local/backend/worker/ingestion/audio/windows.py)). Speech packs into windows of at most 28 s, which bounds a request's memory, ending at the longest pause once a window holds 20 s, the one nearest 24 s among equals. A single run past 28 s is cut at its quietest 100 ms, by loudness, between 20 and 28 s. Silence between windows is never sent, and every window keeps its place in the recording. The plan goes to `plan.json` once; a job that finds it skips the speech pass, and decodes again only while windows remain to send.
+5. **The loop** ([`window_loop.py`](../../surfsense_local/backend/worker/ingestion/audio/window_loop.py)). For each window not yet in `windows.jsonl`: check for cancel, send it as an in-memory WAV to `POST /internal/transcription/windows` (`last` when it is the final window and no other recording is pending or processing), append `{n, start_s, end_s, text, language, model_id, intent}`, write `progress.json` and send the window's `progress` event. A dropped connection or `503 transcription_unavailable` is asked again after 1, 2, 4, 8 and 16 s, then fails with "The speech recognition engine stopped." `503 waiting` is asked again after the cancel and yield checks: a hold or a dictation has no bound, and memory waits are summed in `progress.json` across yields, failing past 10 minutes with the API's sentence. A window answered by another model than the job began with starts the recording over, so one model writes the whole transcript.
+6. **Finish.** The transcript becomes `content` through `finish_job`. `transcript/segments.json` keeps `{model_id, language, duration_s, windows: [{line, start_s, end_s, language}]}`, the line being where the window's paragraph is, counted from 1 as chunks count theirs. The row's metadata gains `transcription: {language, duration_s, model}`, written with `merge_metadata`, which reads and writes the one key in a transaction so every other key stands. `transcript/work/` is deleted.
+
+On the measuring machine ([proposal](../proposals/local-speech-to-text.md#measurements)), an 8.6-minute 16 kHz file went from `pending` to `ready` in 60 s in 22 windows, at 5.83% word error.
+
+### The transcript
+
+One header line, then one paragraph per window that holds text, each starting with its time ([`transcript_text.py`](../../surfsense_local/backend/worker/ingestion/audio/transcript_text.py)):
+
+```markdown
+Transcript of meeting.mp3 · 42:10 · Japanese (detected) · Nemotron 3.5 ASR Streaming 0.6B. [hh:mm:ss] marks where each passage starts.
+
+[00:00:00] …
+
+[00:00:26] …
+```
+
+The chunker splits at blank lines, so while a window's paragraph stays under a chunk's 480 tokens, every chunk opens with its time. Measured on Kokoro's sentences at its pace, a 28 s window runs from 78 tokens in English to 214 in Hindi. The header names the language the user set, or else the one most windows were heard in, then the one heard longest, as "Japanese (detected)"; one the model may detect but is not listed for adds "lower accuracy with this model", and none at all reads "language not identified". The model is named as Settings names it, from its curated entry.
+
+### Yielding
+
+The ingest queue has one thread, and an hour at the measured 11.5 times real time holds it for some 5 minutes. After each window but the last, and after each `503 waiting`, once 60 s have passed in this run and a `FILE` or `NOTE` that is not a recording is pending, the job raises `JobYieldedError`, which `pipeline._ingest` catches before its failure handler: the row goes back to `pending` unless cancelled, and the job is queued again at `PRIORITY_BULK`. Recordings never give way to each other. A yielded recording survives a restart, since the startup sweep fails only `processing` rows, and resumes from its checkpoint with its decoded audio.
+
+### Cancel and retry
+
+Cancel is checked between windows without holding the write lock, and lands within one window; the window in flight finishes. A yielded job's queued copy is revoked as any is. No speech model, a memory wait past its limit, no room on disk ("Transcribing needs about 2.5 GB of free disk space."), no speech ("No speech was found in this recording.", from Silero or from a model that wrote nothing), no audio ("This file has no audio.") and a file the decoder cannot read ("This recording could not be decoded.") fail without retry: `RecordingRefusedError` becomes `CancelExecution(retry=False)` in [`tasks.py`](../../surfsense_local/backend/modules/documents/tasks.py). Anything else uses Huey's two retries, and every retry, Huey's or the user's, resumes from `windows.jsonl` when the model id and the language intent match; another model or language starts over. A run that ends the job, a cancel included, deletes `audio16k.raw` and `progress.json` and keeps the plan and the windows.
+
+### No speech model yet
+
+The upload is accepted. Finding no speech model, the job fails without retry with "Waiting for a speech model." and records `transcription: {awaiting: true}` in the row's metadata, which the next run removes as it starts; a model deleted between windows does the same and keeps the windows written. Choosing a speech model, by `PUT /llm/selection/speech_to_text` or an install with `select: true`, sets every failed row with the flag back to `pending`, clears the flag and its message, queues each at `PRIORITY_BULK` and sends each workspace a `documents` event ([`recordings/awaiting.py`](../../surfsense_local/backend/modules/documents/recordings/awaiting.py)), so nothing is uploaded twice. A row's `transcription.awaiting` reads true only while it is `failed`.
 
 ## Known gaps
 
@@ -173,4 +216,6 @@ Ingest keeps the pictures in each source, so the agent can place a source's char
 - There is no multi-select with Shift or Ctrl, so Move to… and Delete act on one row at a time.
 - Parsing runs on the CPU. Windows and Linux ship CPU-only torch and no CUDA payload ([ADR 0012](../adr/0012-vulkan-only-gpu-backend.md)).
 - A picture in a Word or PowerPoint file is kept with no caption, even under a paragraph in Word's Caption style, because Docling links none; only a PDF's figures carry theirs.
+- Recordings take WAV, MP3, FLAC and Ogg Vorbis only; Opus, M4A, AAC and audio-only WebM wait for a decode-only LGPL FFmpeg ([proposal](../proposals/local-speech-to-text.md#order-of-work), step 9).
+- A recording cannot be played, nor a citation of one opened at its time; `GET .../original` does not commit before streaming, which a player would need ([proposal](../proposals/local-speech-to-text.md#order-of-work), step 10).
 - OCR reads one script family per converter: PP-OCRv6 covers Latin, Chinese and Japanese, so scanned Korean, Cyrillic, Devanagari or Arabic text is misread.

docs/architecture/local-models/selection.md

@@ -25,7 +25,7 @@ type is the slot.
 | `image_edit` | `sdcpp`, a model whose entry names `edit` | `openai_compatible`, with a `connection_id` | nothing yet |
 | `audio_gen` | `audiocpp`, the bundled audio.cpp server | `openai_compatible`, with a `connection_id` | Studio's `podcast` |
 | `video_gen` | `sdcpp`, a model whose entry has a `video` block | `openai_compatible`, with a `connection_id` | nothing yet |
-| `speech_to_text` | `audiocpp`, a recogniser, run on a second audio.cpp server | none: local only | the transcription routes, which no screen or job calls yet ([admission](admission.md#speech-recognition)) |
+| `speech_to_text` | `audiocpp`, a recogniser, run on a second audio.cpp server | none: local only | the transcription routes ([admission](admission.md#speech-recognition)), which a recording's ingest calls ([documents](../documents.md#recordings)) |
 
 A type no feature reads can still be chosen; the feature that first reads one
 brings the client that calls it.
@@ -91,6 +91,10 @@ and choosing a model is when the user has said they are about to use it
 ([`runtime.md`](runtime.md#warming-listed-is-not-loaded)). Any other provider
 loads nothing.
 
+Choosing a `speech_to_text` model, through the route or an install with
+`select: true`, queues again every recording that failed for want of one, at
+`PRIORITY_BULK` ([documents](../documents.md#no-speech-model-yet)).
+
 Installing with `select: true` goes through the same `choose_model()`
 ([`catalog.md`](catalog.md)) and fills the row's own first slot unless the
 request names another, so Nemotron's install fills `speech_to_text` though

docs/architecture/overview.md

@@ -71,7 +71,7 @@ The rules that keep the processes out of each other's way:
 - Workers call `POST /internal/events` after each status change ([`worker/notify.py`](../../surfsense_local/backend/worker/notify.py)): ingest sends a `documents` event keyed by document id, Studio an `artifacts` event keyed by artifact id. The notice is best-effort with a 2-second timeout; losing one costs a live update, never the job.
 - The API announces each reply, chat or agent, starting, changing state and ending on the same stream, as a `chat-runs` event keyed by thread id with status `running`, `queued`, `needs-approval` or `done` ([chat](chat.md#runs)).
 - The API notifies of its own document changes on the same stream, straight to the broker: a note written, an upload, a rename or edit, a retry, a cancel, and a delete, whose status is `deleted` ([`api/notify.py`](../../surfsense_local/backend/api/notify.py)). So a change reaches an open window whoever made it: the window itself, a plugin, or a second window.
-- The API fans each notice out on `GET /workspaces/{id}/events` ([`modules/events/`](../../surfsense_local/backend/modules/events/)) as a named SSE event whose data is `{"ids": [...], "status": "..."}`. The stream opens with a `: connected` comment and sends `: ping` after 15 idle seconds. The broker is an in-memory map, which holds because one uvicorn process serves the app.
+- The API fans each notice out on `GET /workspaces/{id}/events` ([`modules/events/`](../../surfsense_local/backend/modules/events/)) as a named SSE event whose data is `{"ids": [...], "status": "..."}`, with a transient `progress` `{done_s, total_s}` on a recording's event for each transcribed window ([documents](documents.md#recordings)). The stream opens with a `: connected` comment and sends `: ping` after 15 idle seconds. The broker is an in-memory map, which holds because one uvicorn process serves the app.
 - The frontend holds one subscription per workspace, shared by the lists that listen ([`features/workspaces/workspace-changes.ts`](../../surfsense_local/frontend/src/features/workspaces/workspace-changes.ts)), and closes it with the last of them. The sources panel reloads its list on each `documents` event and the Studio artifact list on each `artifacts` event; both reload each time a dropped stream is back, for what changed while nothing listened. Because a notice can be lost, each list also refetches every 10 seconds while any of its rows is `pending` or `processing`, and stops when none is: a lost notice leaves a row stale for at most that long, and nothing is requested on a timer while nothing is in flight.
 
 ## Data directory

docs/architecture/packaging.md

@@ -148,15 +148,16 @@ All nine carry the `packaging` marker, which `pyproject.toml` excludes by defaul
 |---|---|
 | `test_frozen_boot.py` | a minimal frozen entry, `sys.frozen` true, migrates a real database from the bundled revisions, loads `vec0` and round-trips a vector through `chunk_vectors` |
 | `test_real_binaries.py` | the real API binary passes its retrieval import check and answers `/health`; the real worker passes its vision import check and both queue consumers stay up; both binaries ship the remote manifest, the worker ships the local one, and the frozen API serves a curated row |
-| `test_frozen_document_scripts.py` | a frozen worker, started by the runner in script mode as a packaged app starts it, places a matplotlib chart in a Word file with no display, saves charts as SVG and PDF and a `PdfPages` PDF, and writes a deck with a native chart and workbooks with xlsxwriter and openpyxl; an analysis script reads a workbook with pandas, prints a sum and keeps a table and a chart; it freezes the worker from `worker.spec` itself, or uses the one `SURFSENSE_TEST_FROZEN_WORKER` names |
+| `test_frozen_document_scripts.py` | a frozen worker, started by the runner in script mode as a packaged app starts it, places a matplotlib chart in a Word file with no display, saves charts as SVG and PDF and a `PdfPages` PDF, and writes a deck with a native chart and workbooks with xlsxwriter and openpyxl; an analysis script reads a workbook with pandas, prints a sum and keeps a table and a chart; the decode child, `worker --decode-audio`, reads an MP3 to 16 kHz mono with miniaudio, whose cffi backend is loaded by name; it freezes the worker from `worker.spec` itself, or uses the one `SURFSENSE_TEST_FROZEN_WORKER` names |
 | `test_spec_data_files.py` | every literal `datas` path in the specs exists, so a renamed file cannot ship missing, and both `api.spec` and `worker.spec` bundle the local model manifest |
 | `test_license_key.py` | the compiled license keys exclude the fixture key |
 | `test_audiocpp_voicing.py` | the staged audio.cpp server, started with the sidecar's flags from the `server.json` the app writes, voices two turns of each curated model through the app's adapter, at the sample rate its entry names, then holds no model loaded; it runs only when `SURFSENSE_TEST_AUDIO_MODELS` names a folder of the pinned files, each checked against its sha256, and then fails without a staged server; `build-audiocpp.yml` runs it on both builds |
 | `test_audiocpp_transcription_config.py` | the staged server, started as the transcription sidecar from the `transcription.json` the app writes, loads the curated recogniser and takes its request options |
 | `test_audiocpp_transcription.py` | Kokoro, on the voices' server, says one sentence each in English, Spanish, French, Hindi, Brazilian Portuguese and Chinese, and the transcription server writes each down through the app's client, tags and cleanup included, within 20% word error (character error for Chinese) and in the language voiced, which a Latin-script sentence Nemotron left untagged may leave unnamed; it records the server's peak memory, gives the model back by id, and, with Silero staged, checks the detector hears every sentence; it runs only when `SURFSENSE_TEST_AUDIO_MODELS` holds Kokoro's and the recogniser's pinned files |
+| `test_recording_transcription.py` | a recording of three sentences Kokoro voiced in English, Spanish and Portuguese goes through ingest end to end: the decode child, the real Silero, the app's API on a real port with the curated recogniser chosen, and the staged transcription server; the transcript has its header and two windows, each at its time, with words from every sentence, and `segments.json` points at both paragraphs; it runs only when `SURFSENSE_TEST_AUDIO_MODELS` holds the recogniser's pinned file |
 | `test_sdcpp_generation.py` | the staged sd.cpp server, started with the sidecar's flags on Stable Diffusion 1.5 as the engine reads an installed build, makes one image of a red apple through the app's image client, and the reply is checked for what such a picture has, not for the apple itself: it decodes, is the 512 pixels square the entry asks for, has more than a thousand colours, is red on average, and its middle is redder than its border, which neither a flat frame nor noise is; it runs at 4 steps and on seed 42, sd-server's default written down, the two flags the app does not pass (and `--backend cpu` on the hosted macOS runner, whose virtual graphics card Metal cannot use, so Metal itself is not exercised in CI), and only when `SURFSENSE_TEST_IMAGE_MODELS` names a folder holding the pinned file, checked against its sha256, and then fails without a staged server |
 
-Six run in CI: `test_license_key.py` inside the release workflow, `test_frozen_document_scripts.py` there too, on each OS against the worker the job just froze, the three audio.cpp tests inside `build-audiocpp.yml`, and `test_sdcpp_generation.py` inside `build-sdcpp.yml`. Windows' `sd-server` is a pinned download the release job stages, so that gate does not cover it.
+Seven run in CI: `test_license_key.py` inside the release workflow, `test_frozen_document_scripts.py` there too, on each OS against the worker the job just froze, the four audio.cpp tests inside `build-audiocpp.yml`, and `test_sdcpp_generation.py` inside `build-sdcpp.yml`. Windows' `sd-server` is a pinned download the release job stages, so that gate does not cover it.
 
 ## Known gaps
 

docs/proposals/local-speech-to-text.md

@@ -423,8 +423,8 @@ Each is a measurement with its rule already set, and the step that takes it:
 - **Whisper large-v3-turbo on long files** in transcribe.cpp, which only short clips have measured (1.43×): the number that decides whether the fallback can carry the long tail (step 11).
 - **Language flips in files (step 5).** How often windows of a one-language recording are detected as another. Above 1 window in 20, a file locks to the majority of its first three speech windows.
 - **Hallucination (step 5).** An hour of meeting audio and 10 minutes of music and noise, with and without the Silero gate and the compression-ratio drop. A guard that drops real speech from the FLEURS subset is removed.
-- **Transcript chunks (step 5).** Tokens per 28 s window in each listed language under the chunker's tokenizer. A language whose windows pass 480 tokens gets shorter windows, so every chunk keeps its time.
-- **Decoding (step 5).** pyminiaudio's time for an hour of MP3, and error with its linear resampler against scipy's `resample_poly`, already bundled. A loss above 2% relative moves resampling to scipy.
+- **Transcript chunks (step 5).** Tokens per 28 s window in each listed language under the chunker's tokenizer. A language whose windows pass 480 tokens gets shorter windows, so every chunk keeps its time. Measured in step 5 on Kokoro's seven sentences at its pace: 78 tokens in English, 152 to 161 in es, fr, it and pt, 117 in zh and 214 in hi, so no language needs shorter windows yet.
+- **Decoding (step 5).** pyminiaudio's time for an hour of MP3, and error with its linear resampler against scipy's `resample_poly`, already bundled. A loss above 2% relative moves resampling to scipy. Measured in step 5: an hour of 128 kbps MP3 decoded in 3.6 s, and the 8.6-minute file at 44.1 kHz stereo, cut into the same windows, read at 5.91% word error through miniaudio and 5.83% through `resample_poly`, as at 16 kHz: 1.4% relative, so miniaudio resamples.
 - **Yielding (step 5, before release).** A PDF uploaded midway through an hour's transcription starts within 90 s. Chat's tokens per second while a file transcribes, on a CPU-only laptop: a slowdown above 30% pauses file windows while a local reply generates, too.
 - **Capture in packaged builds (steps 6 and 7).** The 16 kHz context and the worklet over `file://` on Windows, macOS and Linux, and the macOS prompt with the new entitlement. A platform that refuses 16 kHz resamples per phrase.
 - **Language ID on short phrases (step 7).** Auto-detection on 3 to 8 s clips in the listed languages. Below 90%, the set-up popover asks for the language before the first recording.

surfsense_local/backend/bundling/worker.spec

@@ -13,6 +13,7 @@ from PyInstaller.utils.hooks import (
     collect_all,
     collect_data_files,
     collect_dynamic_libs,
+    copy_metadata,
 )
 
 sys.path.insert(0, SPECPATH)
@@ -84,6 +85,13 @@ binaries += collect_dynamic_libs("tokenizers")
 datas += collect_data_files("worker.studio", includes=["**/*.md"])
 datas += collect_data_files("modules.chat", includes=["prompts/*.md"])
 
+# The decode child (worker.py --decode-audio) reads recordings with miniaudio,
+# whose compiled half is a cffi extension: the C code imports _cffi_backend,
+# which the analyser cannot see. Its metadata carries the MIT notice, which
+# asks to travel with every copy.
+hiddenimports += ["miniaudio", "_miniaudio", "_cffi_backend"]
+datas += copy_metadata("miniaudio")
+
 # Huey resolves a task by its name, so the module that registers it must be in.
 hiddenimports += [
     "modules.documents.tasks",

surfsense_local/backend/modules/documents/recordings/__init__.py

@@ -0,0 +1,3 @@
+"""A recording as a source, as both the API and the ingest worker see it: where
+its transcript and the job's work files live, how far the job has come, and
+what the job keeps in the row's metadata."""

surfsense_local/backend/modules/documents/recordings/awaiting.py

@@ -0,0 +1,59 @@
+"""Recordings that failed for want of a speech model, queued again once one is
+chosen, so nothing has to be uploaded twice."""
+
+from collections import defaultdict
+from dataclasses import dataclass
+
+from sqlalchemy import select
+from sqlalchemy.orm import Session
+
+from api.notify import notify_changed
+from modules.documents.models import Document, DocumentStatus
+from modules.documents.recordings.metadata import AWAITING, KEY
+from modules.documents.tasks import PRIORITY_BULK, ingest_document
+from modules.events.broker import EventBroker
+from modules.events.schemas import EventKind
+
+__all__ = ["RequeuedRecording", "mark_awaiting_pending", "queue_requeued"]
+
+
+@dataclass(frozen=True)
+class RequeuedRecording:
+    workspace_id: int
+    document_id: int
+
+
+def mark_awaiting_pending(session: Session) -> list[RequeuedRecording]:
+    """Every waiting recording back to pending, its flag gone. Commit before
+    `queue_requeued`: the worker is another process."""
+    awaiting = session.scalars(
+        select(Document).where(
+            Document.status == DocumentStatus.FAILED,
+            Document.document_metadata[KEY][AWAITING].as_boolean().is_(True),
+        )
+    ).all()
+    for document in awaiting:
+        document.status = DocumentStatus.PENDING
+        document.error_message = None
+        # Reassigned, not mutated: a JSON column tracks assignment only.
+        document.document_metadata = {
+            name: value
+            for name, value in (document.document_metadata or {}).items()
+            if name != KEY
+        }
+    session.flush()
+    return [RequeuedRecording(d.workspace_id, d.id) for d in awaiting]
+
+
+def queue_requeued(
+    requeued: list[RequeuedRecording], broker: EventBroker | None
+) -> None:
+    """Behind interactive work, as a bulk copy is; each workspace told at once."""
+    by_workspace: dict[int, list[int]] = defaultdict(list)
+    for recording in requeued:
+        ingest_document(recording.document_id, priority=PRIORITY_BULK)
+        by_workspace[recording.workspace_id].append(recording.document_id)
+    if broker is None:
+        return
+    for workspace_id, ids in by_workspace.items():
+        notify_changed(broker, workspace_id, EventKind.DOCUMENTS, ids, "pending")

surfsense_local/backend/modules/documents/recordings/kind.py

@@ -0,0 +1,7 @@
+"""Which sources are recordings: the server-owned MIME type an upload was given."""
+
+RECORDING_MIME_PREFIX = "audio/"
+
+
+def is_recording(mime_type: str | None) -> bool:
+    return mime_type is not None and mime_type.startswith(RECORDING_MIME_PREFIX)

surfsense_local/backend/modules/documents/recordings/layout.py

@@ -0,0 +1,48 @@
+"""Where a recording's transcript files live: under transcript/ in its document
+folder, since the original must stay the only file there (original_file.py).
+
+transcript/work/ holds what a job resumes from; transcript/segments.json is
+written when the transcript is done, for seeking later.
+"""
+
+from dataclasses import dataclass
+from pathlib import Path
+
+TRANSCRIPT_FOLDER = "transcript"
+WORK_FOLDER = "work"
+SEGMENTS_NAME = "segments.json"
+
+
+@dataclass(frozen=True)
+class WorkFiles:
+    folder: Path
+
+    @property
+    def audio(self) -> Path:
+        """The recording decoded to 16 kHz mono 16-bit PCM, about 115 MB an
+        hour; written as a `.part` beside it until the decoder is done."""
+        return self.folder / "audio16k.raw"
+
+    @property
+    def plan(self) -> Path:
+        return self.folder / "plan.json"
+
+    @property
+    def windows(self) -> Path:
+        return self.folder / "windows.jsonl"
+
+    @property
+    def progress(self) -> Path:
+        return self.folder / "progress.json"
+
+
+def transcript_dir(document_dir: Path) -> Path:
+    return document_dir / TRANSCRIPT_FOLDER
+
+
+def work_files(document_dir: Path) -> WorkFiles:
+    return WorkFiles(transcript_dir(document_dir) / WORK_FOLDER)
+
+
+def segments_file(document_dir: Path) -> Path:
+    return transcript_dir(document_dir) / SEGMENTS_NAME

surfsense_local/backend/modules/documents/recordings/metadata.py

@@ -0,0 +1,17 @@
+"""What a recording's job keeps under `transcription` in the row's metadata:
+`{language, duration_s, model}` once transcribed, or `{awaiting: true}` while
+it waits for a speech model to be chosen."""
+
+from typing import Any
+
+KEY = "transcription"
+AWAITING = "awaiting"
+
+
+def transcription_entry(metadata: dict[str, Any] | None) -> dict[str, Any]:
+    entry = (metadata or {}).get(KEY)
+    return entry if isinstance(entry, dict) else {}
+
+
+def is_awaiting(metadata: dict[str, Any] | None) -> bool:
+    return transcription_entry(metadata).get(AWAITING) is True

surfsense_local/backend/modules/documents/recordings/progress_file.py

@@ -0,0 +1,54 @@
+"""How far a recording's transcription has come, kept beside its checkpoint so a
+reload, a late listener or the job after a yield reads it too."""
+
+import json
+import logging
+import time
+from dataclasses import asdict, dataclass
+from pathlib import Path
+
+__all__ = ["TranscriptionProgress", "read_progress", "write_progress"]
+
+logger = logging.getLogger(__name__)
+
+# Windows refuses to replace a file another process has open; the API reads
+# this one for a moment at a time.
+_REPLACE_ATTEMPTS = 5
+_REPLACE_PAUSE_SECONDS = 0.02
+
+
+@dataclass(frozen=True)
+class TranscriptionProgress:
+    # Where in the recording the last finished window ends.
+    done_s: float
+    total_s: float
+    # Time spent waiting for memory, summed across yields: the job gives up
+    # past its limit.
+    memory_wait_s: float = 0.0
+
+
+def read_progress(path: Path) -> TranscriptionProgress | None:
+    """None when there is no complete file to read."""
+    try:
+        kept = json.loads(path.read_text(encoding="utf-8"))
+        return TranscriptionProgress(
+            float(kept["done_s"]),
+            float(kept["total_s"]),
+            float(kept.get("memory_wait_s", 0.0)),
+        )
+    except (OSError, ValueError, TypeError, KeyError):
+        return None
+
+
+def write_progress(path: Path, progress: TranscriptionProgress) -> None:
+    """Replaced whole, so a reader never sees half of it. Raises OSError only
+    for a full disk; a reader holding the file costs this update alone."""
+    partial = path.with_name(f"{path.name}.partial")
+    partial.write_text(json.dumps(asdict(progress)), encoding="utf-8")
+    for _ in range(_REPLACE_ATTEMPTS):
+        try:
+            partial.replace(path)
+            return
+        except PermissionError:
+            time.sleep(_REPLACE_PAUSE_SECONDS)
+    logger.info("transcription: %s is held open; this update is skipped", path)

surfsense_local/backend/modules/documents/recordings/transcription_read.py

@@ -0,0 +1,45 @@
+"""A recording's transcription as its row reports it: the language heard, how
+far a queued or running job has come, and whether it waits for a speech model."""
+
+from pydantic import BaseModel
+
+from modules.documents.models import Document, DocumentStatus
+from modules.documents.recordings.kind import is_recording
+from modules.documents.recordings.layout import work_files
+from modules.documents.recordings.metadata import is_awaiting, transcription_entry
+from modules.documents.recordings.progress_file import read_progress
+from modules.events.schemas import Progress
+from shared.config import get_storage_settings
+
+__all__ = ["TranscriptionRead", "transcription_read"]
+
+_UNDER_WAY = (DocumentStatus.PENDING, DocumentStatus.PROCESSING)
+
+
+class TranscriptionRead(BaseModel):
+    # The language most windows were heard in, once transcribed.
+    language: str | None
+    # A yielded recording is pending with part of it done.
+    progress: Progress | None
+    # It failed only for want of a speech model; choosing one queues it again.
+    awaiting: bool
+
+
+def transcription_read(document: Document) -> TranscriptionRead | None:
+    """None for anything that is not a recording."""
+    if not is_recording(document.mime_type):
+        return None
+    entry = transcription_entry(document.document_metadata)
+    language = entry.get("language")
+    return TranscriptionRead(
+        language=language if isinstance(language, str) else None,
+        progress=_progress(document) if document.status in _UNDER_WAY else None,
+        awaiting=document.status is DocumentStatus.FAILED
+        and is_awaiting(document.document_metadata),
+    )
+
+
+def _progress(document: Document) -> Progress | None:
+    folder = get_storage_settings().document_dir(document.workspace_id, document.id)
+    kept = read_progress(work_files(folder).progress)
+    return None if kept is None else Progress(done_s=kept.done_s, total_s=kept.total_s)

surfsense_local/backend/modules/documents/schemas.py

@@ -1,15 +1,21 @@
 from datetime import datetime
-from typing import Annotated
+from typing import Annotated, Any, Self
 
 from pydantic import (
     BaseModel,
     ConfigDict,
     JsonValue,
+    ModelWrapValidatorHandler,
     PositiveInt,
     StringConstraints,
+    model_validator,
 )
 
-from modules.documents.models import DocumentStatus, DocumentType
+from modules.documents.models import Document, DocumentStatus, DocumentType
+from modules.documents.recordings.transcription_read import (
+    TranscriptionRead,
+    transcription_read,
+)
 
 DocumentTitle = Annotated[
     str, StringConstraints(strip_whitespace=True, min_length=1, max_length=500)
@@ -51,6 +57,19 @@ class DocumentRead(BaseModel):
     folder_id: int | None
     created_at: datetime
     updated_at: datetime
+    # Only for a recording: null for every other source.
+    transcription: TranscriptionRead | None = None
+
+    @model_validator(mode="wrap")
+    @classmethod
+    def _with_transcription(
+        cls, value: Any, handler: ModelWrapValidatorHandler[Self]
+    ) -> Self:
+        # Read off the row and the job's progress file, never stored as a column.
+        read = handler(value)
+        if isinstance(value, Document):
+            read.transcription = transcription_read(value)
+        return read
 
 
 class DocumentDetail(DocumentRead):

surfsense_local/backend/modules/documents/source_figures/layout.py

@@ -24,9 +24,10 @@ def figure_png(folder: Path, n: int) -> Path:
 
 
 def can_hold_figures(original: Path) -> bool:
-    """Text uploads carry no pictures; Docling drops an HTML page's linked ones."""
+    """Text uploads carry no pictures; Docling drops an HTML page's linked ones.
+    A recording has none, and the figures pass would hand it to Docling."""
     mime = UPLOAD_MIME_BY_SUFFIX.get(original.suffix.lower(), "")
-    return not mime.startswith("text/")
+    return not mime.startswith(("text/", "audio/"))
 
 
 def read_index(folder: Path) -> list[dict[str, Any]] | None:

surfsense_local/backend/modules/documents/storage.py

@@ -38,9 +38,21 @@ UPLOAD_MIME_BY_SUFFIX = {
     ".tiff": "image/tiff",
     ".bmp": "image/bmp",
     ".webp": "image/webp",
+    # Recordings the worker's decoder reads; transcribed, never parsed.
+    ".wav": "audio/wav",
+    ".mp3": "audio/mpeg",
+    ".flac": "audio/flac",
+    ".ogg": "audio/ogg",
+    ".oga": "audio/ogg",
 }
 SUPPORTED_UPLOAD_SUFFIXES = frozenset(UPLOAD_MIME_BY_SUFFIX)
 
+_AUDIO_SUFFIXES = frozenset({".wav", ".mp3", ".flac", ".ogg", ".oga"})
+# An Ogg file's first packet names its codec.
+_VORBIS_HEADER = b"\x01vorbis"
+_OPUS_HEADER = b"OpusHead"
+OPUS_LATER = "Opus support comes in a later update"
+
 _OOXML_MEMBER_BY_SUFFIX = {
     ".docx": "word/document.xml",
     ".pptx": "ppt/presentation.xml",
@@ -58,6 +70,8 @@ def validate_upload(path: Path, suffix: str) -> str:
         valid = _is_utf8_text(path)
     elif suffix in {".png", ".jpg", ".jpeg", ".tif", ".tiff", ".bmp", ".webp"}:
         valid = _is_image(path, suffix)
+    elif suffix in _AUDIO_SUFFIXES:
+        valid = _is_audio(path, suffix)
     else:
         raise ValueError(f"{suffix or 'extensionless files'} are not supported")
 
@@ -119,6 +133,40 @@ def _is_image(path: Path, suffix: str) -> bool:
     return header.startswith(b"RIFF") and header[8:12] == b"WEBP"
 
 
+def _is_audio(path: Path, suffix: str) -> bool:
+    """Raises for Ogg Opus, which the decoder cannot read yet, with the reason."""
+    with path.open("rb") as source:
+        header = source.read(512)
+    if suffix == ".wav":
+        return header.startswith(b"RIFF") and header[8:12] == b"WAVE"
+    if suffix == ".mp3":
+        return header.startswith(b"ID3") or _is_mpeg_audio_frame(header)
+    if suffix == ".flac":
+        return header.startswith(b"fLaC")
+    packet = _first_ogg_packet(header)
+    if packet.startswith(_OPUS_HEADER):
+        raise ValueError(OPUS_LATER)
+    return packet.startswith(_VORBIS_HEADER)
+
+
+def _is_mpeg_audio_frame(header: bytes) -> bool:
+    # Eleven sync bits, a known MPEG version and a layer: ADTS AAC's layer is 0.
+    return (
+        len(header) >= 2
+        and header[0] == 0xFF
+        and header[1] & 0xE0 == 0xE0
+        and (header[1] >> 3) & 0b11 != 0b01
+        and (header[1] >> 1) & 0b11 != 0b00
+    )
+
+
+def _first_ogg_packet(header: bytes) -> bytes:
+    # A page is 27 bytes, then one length byte per segment, then the packet.
+    if not header.startswith(b"OggS") or len(header) < 27:
+        return b""
+    return header[27 + header[26] :]
+
+
 def title_of(upload: UploadFile) -> str:
     name = Path(upload.filename or "").name.strip()
     return name[:500] or "untitled"

surfsense_local/backend/modules/documents/tasks.py

@@ -17,11 +17,12 @@ def ingest_document(document_id: int) -> None:
     """Parse, chunk, embed and index one document."""
     # Lazy: the body pulls in Docling and torch, which the API never needs.
     from worker.ingestion import run
+    from worker.ingestion.audio.failures import RecordingRefusedError
     from worker.ingestion.parsing import UnreadableFileError
 
     try:
         run(document_id)
-    except UnreadableFileError as refused:
+    except (UnreadableFileError, RecordingRefusedError) as refused:
         # Already failed with its reason; a retry would only refuse it again.
         raise CancelExecution(retry=False) from refused
 

surfsense_local/backend/modules/events/broker.py

@@ -34,5 +34,7 @@ class EventBroker:
         # Handed to the loop, not put directly: a `def` route publishes from a
         # threadpool thread, and asyncio queues are not thread-safe.
         event = {"kind": notice.kind, "ids": notice.ids, "status": notice.status}
+        if notice.progress is not None:
+            event["progress"] = notice.progress.model_dump()
         for queue in tuple(self._subscribers.get(notice.workspace_id, ())):
             self._loop.call_soon_threadsafe(queue.put_nowait, event)

surfsense_local/backend/modules/events/router.py

@@ -67,5 +67,8 @@ async def publish_event(payload: InternalEvent, broker: EventBrokerDep) -> Respo
 
 def _event(event: dict) -> bytes:
     """One named SSE event: an `event:` name the client listens for, then its JSON."""
-    data = json.dumps({"ids": event["ids"], "status": event["status"]})
+    frame = {"ids": event["ids"], "status": event["status"]}
+    if "progress" in event:
+        frame["progress"] = event["progress"]
+    data = json.dumps(frame)
     return f"event: {event['kind']}\ndata: {data}\n\n".encode()

surfsense_local/backend/modules/events/schemas.py

@@ -12,6 +12,14 @@ class EventKind(enum.StrEnum):
     FOLDERS = "folders"
 
 
+class Progress(BaseModel):
+    """How far a row's long job has come, in seconds of its media: a
+    recording's transcription."""
+
+    done_s: float
+    total_s: float
+
+
 class InternalEvent(BaseModel):
     """A notice that some rows changed, fanned out as one SSE event.
 
@@ -22,3 +30,6 @@ class InternalEvent(BaseModel):
     kind: EventKind
     ids: list[int]
     status: str
+    # Sent once per step of a long job and never stored, like chat's
+    # prompt-progress frame; a late listener reads the row instead.
+    progress: Progress | None = None

surfsense_local/backend/modules/llm/catalog/local/install_jobs/router.py

@@ -76,6 +76,7 @@ async def start_install(
             model_type=payload.model_type,
             session_factory=session_factory,
             slots=model_types,
+            broker=request.app.state.broker,
         ),
     )
     return _read(job)

surfsense_local/backend/modules/llm/catalog/local/install_jobs/steps.py

@@ -6,6 +6,7 @@ from collections.abc import AsyncIterator, Callable, Sequence
 import httpx
 from sqlalchemy.orm import Session
 
+from modules.events.broker import EventBroker
 from modules.llm.catalog.local.engines.engine import InstallStep
 from modules.llm.catalog.local.install.codes import InstallCode
 from modules.llm.catalog.local.install.plan import InstallPlan, InstallRefusedError
@@ -49,6 +50,7 @@ async def install_steps(
     model_type: ModelType | None,
     session_factory: Callable[[], Session],
     slots: Sequence[ModelType] = (),
+    broker: EventBroker | None = None,
 ) -> AsyncIterator[dict]:
     """Run inside the install lock; the job reports the first `starting`.
 
@@ -138,6 +140,7 @@ async def install_steps(
                 model_type or next(iter(slots), engine.model_types[0]),
                 engine.provider,
                 checked.model_id,
+                broker=broker,
             )
             selection = SelectionRead.model_validate(chosen).model_dump(mode="json")
     yield {

surfsense_local/backend/modules/llm/router.py

@@ -9,6 +9,7 @@ from api.dependencies import SessionDep, transact
 from modules.artifacts.local_image_demand import local_image_demand
 from modules.documents.models import Document, DocumentStatus, DocumentType
 from modules.embedding.active import active_index
+from modules.events.dependencies import EventBrokerDep
 from modules.llm.activity import (
     ModelBusyError,
     ModelFileHeldError,
@@ -336,6 +337,7 @@ async def set_selection(
     payload: SelectionWrite,
     session: SessionDep,
     background: BackgroundTasks,
+    broker: EventBrokerDep,
 ) -> SelectionRead:
     chosen = await choose_model(
         session,
@@ -344,6 +346,7 @@ async def set_selection(
         payload.name,
         connection_id=payload.connection_id,
         allow_unlisted=payload.allow_unlisted,
+        broker=broker,
     )
     # After the response, never during it: the load blocks until the model is
     # resident, which is the tens of seconds this exists to move somewhere the

surfsense_local/backend/modules/llm/selection.py

@@ -5,7 +5,9 @@ from fastapi import HTTPException, status
 from sqlalchemy.orm import Session
 
 from api.dependencies import transact
+from modules.documents.recordings.awaiting import mark_awaiting_pending, queue_requeued
 from modules.embedding.choose import lock_chosen
+from modules.events.broker import EventBroker
 from modules.llm.capability.modes import carried_to_next_model
 from modules.llm.catalog.local.dependencies import get_local_catalog
 from modules.llm.connections.listing import connection_models
@@ -31,7 +33,9 @@ async def choose_model(
     *,
     connection_id: int | None = None,
     allow_unlisted: bool = False,
+    broker: EventBroker | None = None,
 ) -> SelectedModel:
+    """`broker` tells open windows of recordings a speech model queues again."""
     model_name = model_name.strip()
     if not model_name:
         raise HTTPException(
@@ -65,6 +69,9 @@ async def choose_model(
         model_name,
         fingerprint,
     )
+    if model_type is ModelType.SPEECH_TO_TEXT:
+        # Recordings uploaded before any speech model waited for this one.
+        queue_requeued(await transact(session, mark_awaiting_pending), broker)
     logger.info(
         "llm: %s model %s/%s gets the %s prompt (params_b=%s vendor=%s line=%s)",
         model_type.value,

surfsense_local/backend/modules/llm/transcription/voice_activity/silero.py

@@ -6,31 +6,46 @@ from pathlib import Path
 
 import numpy as np
 
-__all__ = ["FRAME_SECONDS", "frame_probabilities"]
+__all__ = ["FRAME_BYTES", "FRAME_SECONDS", "SileroReader", "frame_probabilities"]
 
 _RATE = 16000
 _FRAME = 512
 # Silero v5 and later read each frame with the 64 samples before it.
 _CONTEXT = 64
 FRAME_SECONDS = _FRAME / _RATE
+FRAME_BYTES = _FRAME * 2
+
+
+class SileroReader:
+    """One stream of 16 kHz mono s16 PCM read piece by piece, as a recording
+    is read off disk: each piece continues where the last ended."""
+
+    def __init__(self, model: Path) -> None:
+        self._session = _session(str(model))
+        self._state = np.zeros((2, 1, 128), dtype=np.float32)
+        self._context = np.zeros((1, _CONTEXT), dtype=np.float32)
+        self._rate = np.array(_RATE, dtype=np.int64)
+
+    def read(self, pcm: bytes) -> list[float]:
+        """Every piece but the last holds whole frames (FRAME_BYTES each); the
+        last frame is padded with silence."""
+        samples = np.frombuffer(pcm, dtype="<i2").astype(np.float32) / 32768
+        probabilities = []
+        for start in range(0, len(samples), _FRAME):
+            frame = samples[start : start + _FRAME]
+            frame = np.pad(frame, (0, _FRAME - len(frame)))[None, :]
+            both = np.concatenate([self._context, frame], axis=1)
+            out, self._state = self._session.run(
+                None, {"input": both, "state": self._state, "sr": self._rate}
+            )
+            self._context = both[:, -_CONTEXT:]
+            probabilities.append(float(out[0, 0]))
+        return probabilities
 
 
 def frame_probabilities(pcm: bytes, model: Path) -> list[float]:
     """`pcm` is 16 kHz mono s16; the last frame is padded with silence."""
-    session = _session(str(model))
-    samples = np.frombuffer(pcm, dtype="<i2").astype(np.float32) / 32768
-    state = np.zeros((2, 1, 128), dtype=np.float32)
-    context = np.zeros((1, _CONTEXT), dtype=np.float32)
-    rate = np.array(_RATE, dtype=np.int64)
-    probabilities = []
-    for start in range(0, len(samples), _FRAME):
-        frame = samples[start : start + _FRAME]
-        frame = np.pad(frame, (0, _FRAME - len(frame)))[None, :]
-        both = np.concatenate([context, frame], axis=1)
-        out, state = session.run(None, {"input": both, "state": state, "sr": rate})
-        context = both[:, -_CONTEXT:]
-        probabilities.append(float(out[0, 0]))
-    return probabilities
+    return SileroReader(model).read(pcm)
 
 
 @lru_cache(maxsize=1)

surfsense_local/backend/pyproject.toml

@@ -21,6 +21,10 @@ dependencies = [
     # Charts in document scripts (worker/document_script/), which draw one and
     # place the PNG, and in Studio's Markdown builders' chart blocks.
     "matplotlib>=3.11.2",
+    # Decodes a recording (WAV, MP3, FLAC, Ogg Vorbis) to 16 kHz mono in the
+    # worker's decode child (worker/audio_decode). MIT, with its decoders'
+    # public-domain or MIT-0 code; no FFmpeg.
+    "miniaudio>=1.71",
     "onnxruntime>=1.20",
     # Reads back the workbook a document script wrote
     # (worker/studio/script_document/workbook_summary.py). Docling brings it

surfsense_local/backend/tests/integration/documents/test_source_figures.py

@@ -130,7 +130,8 @@ def test_a_source_still_being_ingested_waits_for_its_own_ingest(
 def test_sources_that_hold_no_figures_list_none(
     session: Session, workspace: Workspace
 ) -> None:
-    """Notes, Studio outputs, plain text and failed sources have nothing to extract."""
+    """Notes, Studio outputs, plain text, recordings and failed sources have
+    nothing to extract: Docling must never read a recording."""
     note = Document(
         workspace_id=workspace.id,
         title="Ideas",
@@ -141,11 +142,12 @@ def test_sources_that_hold_no_figures_list_none(
     session.add(note)
     session.commit()
     text = add_source(session, workspace, filename="notes.md")
+    recording = add_source(session, workspace, filename="standup.mp3")
     failed = add_source(
         session, workspace, filename="broken.pdf", status=DocumentStatus.FAILED
     )
 
-    for source in (note, text, failed):
+    for source in (note, text, recording, failed):
         assert list_figures(session, workspace.id, source.id) == []
     assert queued() == []
 

surfsense_local/backend/tests/integration/documents/test_upload.py

@@ -7,6 +7,12 @@ from httpx import AsyncClient
 from sqlalchemy import Engine, text
 
 from modules.documents import storage
+from modules.documents.recordings.layout import work_files
+from modules.documents.recordings.progress_file import (
+    TranscriptionProgress,
+    write_progress,
+)
+from shared.config import get_storage_settings
 from shared.queue import ingest_queue
 
 pytestmark = pytest.mark.integration
@@ -151,6 +157,61 @@ async def test_unsupported_and_mismatched_files_are_rejected_per_file(
     assert len(ingest_queue.pending()) == 1
 
 
+async def test_a_recording_is_queued_and_an_opus_voice_note_is_turned_away(
+    client: AsyncClient, workspace_id: int
+) -> None:
+    """MP3, FLAC and Ogg Vorbis are transcribed; Opus waits for its decoder,
+    and the person reads why beside that one file."""
+    tones = Path(__file__).resolve().parents[2] / "fixtures" / "audio"
+    response = await client.post(
+        f"/workspaces/{workspace_id}/documents/upload",
+        files=[
+            ("files", ("standup.mp3", (tones / "tone.mp3").read_bytes(), "audio/x")),
+            ("files", ("voice.ogg", (tones / "tone-opus.ogg").read_bytes(), "")),
+            ("files", ("notes.txt", b"research notes", "text/plain")),
+        ],
+    )
+
+    assert response.status_code == 201
+    recording, notes = response.json()["created"]
+    assert (recording["mime_type"], recording["transcription"]) == (
+        "audio/mpeg",
+        {"language": None, "progress": None, "awaiting": False},
+    )
+    assert notes["transcription"] is None
+    assert response.json()["rejected"] == [
+        {"filename": "voice.ogg", "reason": "Opus support comes in a later update"}
+    ]
+    assert len(ingest_queue.pending()) == 2
+
+
+async def test_a_queued_recording_reports_how_far_its_transcription_came(
+    client: AsyncClient, workspace_id: int
+) -> None:
+    """Read from the job's progress file, so a reload sees what the events said."""
+    tones = Path(__file__).resolve().parents[2] / "fixtures" / "audio"
+    uploaded = await client.post(
+        f"/workspaces/{workspace_id}/documents/upload",
+        files={"files": ("lecture.flac", (tones / "tone.flac").read_bytes(), "")},
+    )
+    recording = uploaded.json()["created"][0]["id"]
+    folder = get_storage_settings().document_dir(workspace_id, recording)
+    progress = work_files(folder).progress
+    progress.parent.mkdir(parents=True)
+    write_progress(progress, TranscriptionProgress(done_s=1392.5, total_s=3480.0))
+
+    listed = await client.get(f"/workspaces/{workspace_id}/documents")
+    read = await client.get(f"/workspaces/{workspace_id}/documents/{recording}")
+
+    expected = {
+        "language": None,
+        "progress": {"done_s": 1392.5, "total_s": 3480.0},
+        "awaiting": False,
+    }
+    assert listed.json()[0]["transcription"] == expected
+    assert read.json()["transcription"] == expected
+
+
 async def test_server_records_verified_mime_not_the_clients_claim(
     client: AsyncClient, workspace_id: int
 ) -> None:

surfsense_local/backend/tests/integration/events/test_events.py

@@ -56,6 +56,40 @@ async def test_an_internal_event_reaches_a_subscriber(base_url: str) -> None:
             }
 
 
+async def test_a_transcribed_window_reaches_a_subscriber_with_its_progress(
+    base_url: str,
+) -> None:
+    """A recording's row moves its bar on each window without a refetch."""
+    async with AsyncClient(base_url=base_url, timeout=5) as client:
+        workspace_id = (await client.post("/workspaces", json={"name": "w"})).json()[
+            "id"
+        ]
+
+        async def window_done() -> None:
+            posted = await client.post(
+                "/internal/events",
+                json={
+                    "workspace_id": workspace_id,
+                    "kind": "documents",
+                    "ids": [7],
+                    "status": "processing",
+                    "progress": {"done_s": 52.5, "total_s": 3480.0},
+                },
+            )
+            assert posted.status_code == 202
+
+        event = await _event_after(client, workspace_id, window_done)
+
+        assert event == (
+            "documents",
+            {
+                "ids": [7],
+                "status": "processing",
+                "progress": {"done_s": 52.5, "total_s": 3480.0},
+            },
+        )
+
+
 async def test_an_unknown_workspace_stream_is_a_404(base_url: str) -> None:
     """The stream validates the workspace like every other workspace-scoped route."""
     async with AsyncClient(base_url=base_url, timeout=5) as client:

surfsense_local/backend/tests/integration/llm/test_speech_model_requeues_recordings.py

@@ -0,0 +1,134 @@
+"""Recordings uploaded before any speech model wait for one, and choosing one
+queues every one of them again, so nothing is uploaded twice."""
+
+import pytest
+from httpx import AsyncClient
+from sqlalchemy import Engine
+
+from modules.documents.models import Document, DocumentStatus, DocumentType
+from modules.documents.tasks import PRIORITY_BULK
+from modules.workspaces.models import Workspace
+from shared.db import create_session_factory
+from shared.queue import ingest_queue
+from tests.integration.llm.installs import install_curated
+
+pytestmark = [pytest.mark.integration, pytest.mark.asyncio]
+
+NEMOTRON = "nemotron-3.5-asr-streaming-0.6b"
+MODEL = "nemotron-3.5-asr-streaming-0.6b-q8_0"
+WAITING = "Waiting for a speech model."
+
+
+def add_sources(engine: Engine) -> dict[str, int]:
+    """Two workspaces' recordings waiting for a model, a recording that failed
+    otherwise, and a failed PDF."""
+    with create_session_factory(engine)() as session:
+        first, second = Workspace(name="Standups"), Workspace(name="Interviews")
+        session.add_all([first, second])
+        session.flush()
+
+        def failed(
+            workspace: Workspace, title: str, metadata: dict, reason: str
+        ) -> Document:
+            document = Document(
+                workspace_id=workspace.id,
+                title=title,
+                document_type=DocumentType.FILE,
+                status=DocumentStatus.FAILED,
+                error_message=reason,
+                document_metadata=metadata,
+            )
+            session.add(document)
+            return document
+
+        audio = {"mime_type": "audio/mpeg", "suffix": ".mp3"}
+        rows = {
+            "monday": failed(
+                first,
+                "monday.mp3",
+                {**audio, "transcription": {"awaiting": True}},
+                WAITING,
+            ),
+            "guest": failed(
+                second,
+                "guest.mp3",
+                {**audio, "transcription": {"awaiting": True}},
+                WAITING,
+            ),
+            "silent": failed(
+                first, "silent.mp3", audio, "No speech was found in this recording."
+            ),
+            "report": failed(
+                first, "report.pdf", {"mime_type": "application/pdf"}, "boom"
+            ),
+        }
+        session.commit()
+        return {name: row.id for name, row in rows.items()}
+
+
+def read(engine: Engine, document_id: int) -> Document:
+    """The row as it is now, outside any session."""
+    with create_session_factory(engine)() as session:
+        document = session.get(Document, document_id)
+        session.expunge(document)
+        session.commit()
+        return document
+
+
+def queued() -> list[tuple[int, int]]:
+    """Each queued ingest's document and priority."""
+    return sorted(
+        (task.args[0], task.priority)
+        for task in ingest_queue.pending()
+        if task.name == "ingest_document"
+    )
+
+
+async def test_choosing_a_speech_model_queues_every_waiting_recording(
+    client: AsyncClient, engine: Engine, audio_dir, fake_hub
+) -> None:
+    """Behind interactive work, in every workspace; nothing else is retried."""
+    ids = add_sources(engine)
+    await install_curated(client, NEMOTRON, "Q8_0")
+
+    chosen = await client.put(
+        "/llm/selection/speech_to_text", json={"provider": "audiocpp", "name": MODEL}
+    )
+
+    assert chosen.status_code == 200, chosen.text
+    assert queued() == sorted(
+        [(ids["monday"], PRIORITY_BULK), (ids["guest"], PRIORITY_BULK)]
+    )
+    monday = read(engine, ids["monday"])
+    assert (monday.status, monday.error_message) == (DocumentStatus.PENDING, None)
+    assert "transcription" not in monday.document_metadata
+    assert read(engine, ids["silent"]).status is DocumentStatus.FAILED
+    assert read(engine, ids["report"]).status is DocumentStatus.FAILED
+
+
+async def test_a_download_that_chooses_the_model_queues_them_too(
+    client: AsyncClient, engine: Engine, audio_dir, fake_hub
+) -> None:
+    """The set-up popover's Download and set up selects as it installs."""
+    ids = add_sources(engine)
+
+    await install_curated(client, NEMOTRON, "Q8_0", select=True)
+
+    assert [document_id for document_id, _ in queued()] == sorted(
+        [ids["monday"], ids["guest"]]
+    )
+
+
+async def test_choosing_another_kind_of_model_queues_nothing(
+    client: AsyncClient, engine: Engine, audio_dir, fake_hub, bundled_voice
+) -> None:
+    """Only a speech model is what they wait for."""
+    add_sources(engine)
+
+    chosen = await client.put(
+        "/llm/selection/audio_gen",
+        json={"provider": "audiocpp", "name": "kokoro-82m-q8_0"},
+    )
+
+    assert chosen.status_code == 200, chosen.text
+    assert queued() == []

surfsense_local/backend/tests/integration/worker/conftest.py

@@ -5,6 +5,11 @@ import pytest
 from modules.llm.profile import Fingerprint, Tier
 from modules.llm.resolution import ResolvedGeneration, ResolvedImageGeneration
 
+# The API's speech routes, as a recording's ingest job reaches them.
+from tests.integration.worker.speech_api_stand_in import (
+    speech_api,  # noqa: F401 - a fixture
+)
+
 
 @pytest.fixture
 def stub_model(monkeypatch: pytest.MonkeyPatch) -> None:

surfsense_local/backend/tests/integration/worker/speech_api_stand_in.py

@@ -0,0 +1,130 @@
+"""The API as a recording's ingest job reaches it: the transcription status, the
+windows route, and the change notices it posts. Each test scripts the answers."""
+
+import io
+import json
+import threading
+import wave
+from collections.abc import Callable, Iterator
+from dataclasses import dataclass, field
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+import pytest
+
+MODEL = "nemotron-3.5-asr-streaming-0.6b-q8_0"
+
+
+@dataclass
+class SentWindow:
+    seconds: float
+    rate: int
+    channels: int
+    last: bool
+
+
+@dataclass
+class SpeechApi:
+    status: dict = field(
+        default_factory=lambda: {
+            "ready": True,
+            "reason": None,
+            "model": MODEL,
+            "language_intent": "auto",
+            "languages": ["en", "es", "ja"],
+        }
+    )
+    # Status and body for each window asked, in order; past them, `written`.
+    replies: list[tuple[int, dict]] = field(default_factory=list)
+    language: str | None = "en"
+    windows: list[SentWindow] = field(default_factory=list)
+    events: list[dict] = field(default_factory=list)
+    # Runs as window n is being answered, as a person's click lands meanwhile.
+    during_window: Callable[[int], None] | None = None
+
+    def written(self, n: int) -> dict:
+        return {
+            "text": f"Words of window {n}.",
+            "language": self.language,
+            "model_id": MODEL,
+        }
+
+    @staticmethod
+    def waiting(reason: str, message: str = "Waiting") -> tuple[int, dict]:
+        return 503, {
+            "detail": {"code": "waiting", "reason": reason, "message": message}
+        }
+
+    @staticmethod
+    def refused(status: int, code: str) -> tuple[int, dict]:
+        return status, {"detail": {"code": code, "message": code}}
+
+
+class _Handler(BaseHTTPRequestHandler):
+    def do_GET(self) -> None:
+        api: SpeechApi = self.server.stand_in  # type: ignore[attr-defined]
+        if self.path == "/llm/transcription/status":
+            self._json(200, api.status)
+        else:
+            self.send_error(404)
+
+    def do_POST(self) -> None:
+        api: SpeechApi = self.server.stand_in  # type: ignore[attr-defined]
+        body = self.rfile.read(int(self.headers["Content-Length"]))
+        if self.path == "/internal/events":
+            api.events.append(json.loads(body))
+            self._json(202, {})
+        elif self.path == "/internal/transcription/windows":
+            n = len(api.windows)
+            api.windows.append(_window(self.headers["Content-Type"], body))
+            if api.during_window is not None:
+                api.during_window(n)
+            if n < len(api.replies):
+                self._json(*api.replies[n])
+            else:
+                self._json(200, api.written(n))
+        else:
+            self.send_error(404)
+
+    def _json(self, status: int, payload: dict) -> None:
+        body = json.dumps(payload).encode()
+        self.send_response(status)
+        self.send_header("Content-Type", "application/json")
+        self.send_header("Content-Length", str(len(body)))
+        self.end_headers()
+        self.wfile.write(body)
+
+    def log_message(self, *args: object) -> None:
+        """Keep the request log out of the test output."""
+
+
+def _window(content_type: str, body: bytes) -> SentWindow:
+    boundary = content_type.split("boundary=")[1].encode()
+    last = False
+    clip = (0.0, 0, 0)
+    for part in body.split(b"--" + boundary)[1:-1]:
+        head, _, value = part.partition(b"\r\n\r\n")
+        value = value.removesuffix(b"\r\n")
+        if b"filename=" in head:
+            with wave.open(io.BytesIO(value)) as audio:
+                clip = (
+                    audio.getnframes() / audio.getframerate(),
+                    audio.getframerate(),
+                    audio.getnchannels(),
+                )
+        elif b'name="last"' in head:
+            last = value == b"true"
+    return SentWindow(*clip, last=last)
+
+
+@pytest.fixture
+def speech_api(monkeypatch: pytest.MonkeyPatch) -> Iterator[SpeechApi]:
+    """A stand-in on a real port, which the worker is pointed at as Electron does."""
+    stand_in = SpeechApi()
+    http = ThreadingHTTPServer(("127.0.0.1", 0), _Handler)
+    http.stand_in = stand_in  # type: ignore[attr-defined]
+    threading.Thread(target=http.serve_forever, daemon=True).start()
+    monkeypatch.setenv("SURFSENSE_LOCAL_HOST", "127.0.0.1")
+    monkeypatch.setenv("SURFSENSE_LOCAL_PORT", str(http.server_port))
+    yield stand_in
+    http.shutdown()
+    http.server_close()

surfsense_local/backend/tests/integration/worker/test_decode_child.py

@@ -0,0 +1,97 @@
+"""The decode child as the ingest worker runs it: worker.py --decode-audio, a
+process of its own whose death never reaches the consumer."""
+
+import io
+import struct
+import sys
+import wave
+from pathlib import Path
+
+import pytest
+
+from worker.audio_decode import run as decode_run
+from worker.audio_decode.run import decode_in_child
+
+pytestmark = pytest.mark.integration
+
+TONES = Path(__file__).resolve().parents[2] / "fixtures" / "audio"
+
+
+class CancelError(Exception):
+    """What a job's cancel check raises."""
+
+
+def nothing() -> None:
+    """A job nobody cancels."""
+
+
+def long_wav(seconds: float) -> bytes:
+    """Enough 44.1 kHz stereo audio for the decode to outlast a poll."""
+    frame = struct.pack("<hh", 3000, -3000)
+    out = io.BytesIO()
+    with wave.open(out, "wb") as clip:
+        clip.setnchannels(2)
+        clip.setsampwidth(2)
+        clip.setframerate(44100)
+        clip.writeframes(frame * int(seconds * 44100))
+    return out.getvalue()
+
+
+def test_the_child_leaves_the_whole_recording_at_16_khz(tmp_path: Path) -> None:
+    """Named `out` only once every sample is written."""
+    out = tmp_path / "audio16k.raw"
+
+    outcome = decode_in_child(TONES / "tone.ogg", out, between_polls=nothing)
+
+    assert outcome.kind == "decoded"
+    assert out.stat().st_size == pytest.approx(16000 * 2, rel=0.05)
+    assert sorted(p.name for p in tmp_path.iterdir()) == ["audio16k.raw"]
+
+
+def test_a_file_the_child_refuses_says_why(tmp_path: Path) -> None:
+    """Undecodable bytes come back as an outcome, not a crash of the worker."""
+    source = tmp_path / "broken.flac"
+    source.write_bytes(b"fLaC" + b"\xff" * 512)
+    out = tmp_path / "audio16k.raw"
+
+    outcome = decode_in_child(source, out, between_polls=nothing)
+
+    assert outcome.kind == "undecodable"
+    assert not out.exists()
+
+
+def test_a_decoder_that_dies_is_an_undecodable_file(
+    tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+    """A malformed file can take the decoder down with it; the job reads that
+    as bytes it cannot read, and nothing is left half written."""
+    out = tmp_path / "audio16k.raw"
+    (tmp_path / "audio16k.raw.part").write_bytes(b"\0" * 64)
+    monkeypatch.setattr(
+        decode_run,
+        "_command",
+        lambda _source, _out: [sys.executable, "-c", "import os; os._exit(70)"],
+    )
+
+    outcome = decode_in_child(TONES / "tone.mp3", out, between_polls=nothing)
+
+    assert outcome.kind == "undecodable"
+    assert list(tmp_path.iterdir()) == []
+
+
+def test_a_cancel_stops_the_child_and_leaves_nothing(
+    tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+    """The job's cancel check runs while the child decodes, and wins."""
+    monkeypatch.setattr(decode_run, "POLL_SECONDS", 0.01)
+    source = tmp_path / "long.wav"
+    source.write_bytes(long_wav(120))
+    out = tmp_path / "audio16k.raw"
+
+    def cancelled() -> None:
+        raise CancelError
+
+    with pytest.raises(CancelError):
+        decode_in_child(source, out, between_polls=cancelled)
+
+    assert sorted(p.name for p in tmp_path.iterdir()) == ["long.wav"]

surfsense_local/backend/tests/integration/worker/test_recordings.py

@@ -0,0 +1,519 @@
+"""A recording through ingest: decoded in a child, cut into windows, each window
+written down through a stand-in of the API's routes, and the transcript
+indexed. Without a staged Silero all of a recording counts as speech, so the
+plan here is cut by loudness alone: the sentences' pause makes two windows."""
+
+import io
+import json
+import wave
+from collections.abc import Iterator
+from pathlib import Path
+
+import pytest
+from huey.exceptions import CancelExecution
+from sqlalchemy import Engine, text
+from sqlalchemy.orm import Session
+
+from modules.documents.models import Document, DocumentStatus, DocumentType
+from modules.documents.recordings.layout import segments_file, work_files
+from modules.documents.schemas import DocumentRead
+from modules.documents.tasks import PRIORITY_BULK, ingest_document
+from modules.workspaces.models import Workspace
+from shared.config import get_storage_settings
+from shared.db import create_session_factory
+from shared.queue import ingest_queue
+from tests.integration.worker.speech_api_stand_in import MODEL, SpeechApi
+from worker.audio_decode.outcome import DecodeOutcome
+from worker.ingestion import run
+from worker.ingestion.audio import queue_turn, readiness, transcript_job, window_loop
+
+pytestmark = pytest.mark.integration
+
+# Three sentences voiced by Kokoro-82M, 3 s of silence between them: 36.65 s.
+SPEECH = Path(__file__).resolve().parents[2] / "fixtures" / "audio" / "speech.ogg"
+HEADER = (
+    "Transcript of standup.ogg · 0:36 · English (detected) · "
+    "Nemotron 3.5 ASR Streaming 0.6B. [hh:mm:ss] marks where each passage starts."
+)
+
+
+@pytest.fixture
+def session(engine: Engine) -> Iterator[Session]:
+    """A session on the migrated database ingest opens again by path."""
+    with create_session_factory(engine)() as opened:
+        yield opened
+
+
+@pytest.fixture(autouse=True)
+def quick_retries(monkeypatch: pytest.MonkeyPatch) -> None:
+    """The waits between a dropped server's retries, cut for the test."""
+    monkeypatch.setattr(window_loop, "RETRY_DELAYS", (0, 0, 0, 0, 0))
+    monkeypatch.setattr(readiness, "POLL_SECONDS", 0.0)
+
+
+def add_recording(
+    session: Session,
+    name: str = "standup.ogg",
+    content: bytes | None = None,
+    status: DocumentStatus = DocumentStatus.PENDING,
+) -> Document:
+    """A recording as the upload route leaves it: a pending row, its file alone
+    in its folder."""
+    workspace = session.scalar(text("SELECT id FROM workspaces LIMIT 1"))
+    if workspace is None:
+        created = Workspace(name="Standups")
+        session.add(created)
+        session.flush()
+        workspace = created.id
+    suffix = Path(name).suffix
+    mime = {".ogg": "audio/ogg", ".wav": "audio/wav", ".mp3": "audio/mpeg"}[suffix]
+    data = SPEECH.read_bytes() if content is None else content
+    document = Document(
+        workspace_id=workspace,
+        title=name,
+        document_type=DocumentType.FILE,
+        status=status,
+        document_metadata={
+            "mime_type": mime,
+            "size_bytes": len(data),
+            "suffix": suffix,
+        },
+    )
+    session.add(document)
+    session.commit()
+    folder = folder_of(document)
+    folder.mkdir(parents=True)
+    (folder / name).write_bytes(data)
+    return document
+
+
+def folder_of(document: Document) -> Path:
+    """The recording's own folder, where its original and transcript/ live."""
+    return get_storage_settings().document_dir(document.workspace_id, document.id)
+
+
+def silent_wav(seconds: float) -> bytes:
+    """Seconds of digital silence, as a 16 kHz memo."""
+    out = io.BytesIO()
+    with wave.open(out, "wb") as clip:
+        clip.setnchannels(1)
+        clip.setsampwidth(2)
+        clip.setframerate(16000)
+        clip.writeframes(b"\0\0" * int(seconds * 16000))
+    return out.getvalue()
+
+
+def checkpoint(document: Document) -> list[dict]:
+    """The windows a run kept for the next."""
+    path = work_files(folder_of(document)).windows
+    return [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines()]
+
+
+def refused(document_id: int) -> None:
+    """Ingest as Huey runs it, for a recording that is not tried again."""
+    with pytest.raises(CancelExecution):
+        ingest_document.call_local(document_id)
+
+
+# Transcribed
+
+
+def test_a_recording_becomes_a_timestamped_transcript(
+    session: Session, stub_model: None, speech_api: SpeechApi
+) -> None:
+    """One header line, a paragraph per window from its time, and searchable."""
+    recording = add_recording(session)
+
+    run(recording.id)
+
+    session.expire_all()
+    assert recording.status is DocumentStatus.READY
+    paragraphs = recording.content.split("\n\n")
+    assert paragraphs[0] == HEADER
+    assert paragraphs[1] == "[00:00:00] Words of window 0."
+    assert paragraphs[2].startswith("[00:00:2")
+    assert paragraphs[2].endswith("Words of window 1.\n")
+    assert session.scalar(text("SELECT count(*) FROM chunks")) >= 1
+
+
+def test_each_window_is_16_khz_mono_and_at_most_28_seconds(
+    session: Session, stub_model: None, speech_api: SpeechApi
+) -> None:
+    """What the server reads; only the last tells the API the model may go."""
+    run(add_recording(session).id)
+
+    assert len(speech_api.windows) == 2
+    assert all(w.seconds <= 28.0 for w in speech_api.windows)
+    assert {(w.rate, w.channels) for w in speech_api.windows} == {(16000, 1)}
+    assert [w.last for w in speech_api.windows] == [False, True]
+    assert sum(w.seconds for w in speech_api.windows) == pytest.approx(36.65, abs=0.1)
+
+
+def test_a_finished_recording_keeps_its_language_length_model_and_segments(
+    session: Session, stub_model: None, speech_api: SpeechApi
+) -> None:
+    """The row says what was heard; segments.json where; work/ is gone and the
+    original is still the only file beside the transcript."""
+    recording = add_recording(session)
+
+    run(recording.id)
+
+    session.expire_all()
+    assert recording.document_metadata["transcription"] == {
+        "language": "en",
+        "duration_s": pytest.approx(36.65, abs=0.01),
+        "model": MODEL,
+    }
+    assert DocumentRead.model_validate(recording).transcription.model_dump() == {
+        "language": "en",
+        "progress": None,
+        "awaiting": False,
+    }
+    segments = json.loads(segments_file(folder_of(recording)).read_text("utf-8"))
+    assert segments["model_id"] == MODEL
+    assert [w["line"] for w in segments["windows"]] == [3, 5]
+    assert not work_files(folder_of(recording)).folder.exists()
+    assert [p.name for p in folder_of(recording).iterdir() if p.is_file()] == [
+        "standup.ogg"
+    ]
+
+
+def test_each_window_moves_the_rows_progress(
+    session: Session, stub_model: None, speech_api: SpeechApi
+) -> None:
+    """A notice per window carries how far it has come, the last one all of it."""
+    recording = add_recording(session)
+
+    run(recording.id)
+
+    progress = [e["progress"] for e in speech_api.events if "progress" in e]
+    assert len(progress) == 2
+    assert progress[0]["done_s"] < progress[1]["done_s"]
+    assert progress[1] == {
+        "done_s": pytest.approx(36.65, abs=0.01),
+        "total_s": pytest.approx(36.65, abs=0.01),
+    }
+    assert all(e["ids"] == [recording.id] for e in speech_api.events)
+
+
+def test_a_language_the_user_set_heads_the_transcript_without_detected(
+    session: Session, stub_model: None, speech_api: SpeechApi
+) -> None:
+    """The intent is never rewritten from a detection."""
+    speech_api.status["language_intent"] = "ja"
+    speech_api.language = "en"
+    recording = add_recording(session)
+
+    run(recording.id)
+
+    session.expire_all()
+    assert " · Japanese · " in recording.content.split("\n", 1)[0]
+    assert recording.document_metadata["transcription"]["language"] == "ja"
+
+
+# Waiting for a speech model
+
+
+def test_with_no_speech_model_the_recording_waits_for_one(
+    session: Session, stub_model: None, speech_api: SpeechApi
+) -> None:
+    """Not an error the person must fix by uploading again: choosing a model
+    queues it. Nothing is decoded before a model is there."""
+    speech_api.status = {
+        "ready": False,
+        "reason": "needs_speech_model",
+        "model": None,
+        "language_intent": "auto",
+        "languages": [],
+    }
+    recording = add_recording(session)
+
+    refused(recording.id)
+
+    session.expire_all()
+    assert recording.status is DocumentStatus.FAILED
+    assert recording.error_message == "Waiting for a speech model."
+    assert DocumentRead.model_validate(recording).transcription.awaiting is True
+    assert speech_api.windows == []
+    assert not work_files(folder_of(recording)).audio.exists()
+
+
+def test_a_model_deleted_between_windows_waits_and_resumes_where_it_stopped(
+    session: Session, stub_model: None, speech_api: SpeechApi
+) -> None:
+    """The windows written stay for when a model is chosen again."""
+    speech_api.replies = [
+        (200, speech_api.written(0)),
+        SpeechApi.refused(409, "needs_speech_model"),
+    ]
+    recording = add_recording(session)
+
+    refused(recording.id)
+
+    session.expire_all()
+    assert DocumentRead.model_validate(recording).transcription.awaiting is True
+    assert [w["n"] for w in checkpoint(recording)] == [0]
+
+    speech_api.replies = []
+    speech_api.windows.clear()
+    session.commit()  # the reads above hold the write lock
+    run(recording.id)
+
+    session.expire_all()
+    assert recording.status is DocumentStatus.READY
+    assert len(speech_api.windows) == 1
+    assert "awaiting" not in recording.document_metadata["transcription"]
+
+
+# Yielding
+
+
+def test_after_a_minute_a_recording_gives_way_to_a_waiting_note(
+    session: Session,
+    stub_model: None,
+    speech_api: SpeechApi,
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """Back to pending and queued behind it, with what it has done kept: a
+    note added midway is read within a window."""
+    monkeypatch.setattr(queue_turn, "YIELD_AFTER_SECONDS", 0.0)
+    recording = add_recording(session)
+    session.add(
+        Document(
+            workspace_id=recording.workspace_id,
+            title="Note",
+            document_type=DocumentType.NOTE,
+            content="text",
+        )
+    )
+    session.commit()
+
+    run(recording.id)
+
+    session.expire_all()
+    assert recording.status is DocumentStatus.PENDING
+    assert [(t.name, t.args, t.priority) for t in ingest_queue.pending()] == [
+        ("ingest_document", (recording.id,), PRIORITY_BULK)
+    ]
+    progress = DocumentRead.model_validate(recording).transcription.progress
+    assert 0 < progress.done_s < progress.total_s
+    assert len(speech_api.windows) == 1
+    session.commit()  # the reads above hold the write lock
+
+    run(recording.id)
+
+    session.expire_all()
+    assert recording.status is DocumentStatus.READY
+    assert len(speech_api.windows) == 2
+
+
+def test_recordings_never_give_way_to_each_other(
+    session: Session,
+    stub_model: None,
+    speech_api: SpeechApi,
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """Two queued recordings would only trade places; the speech model stays
+    for the next one."""
+    monkeypatch.setattr(queue_turn, "YIELD_AFTER_SECONDS", 0.0)
+    recording = add_recording(session)
+    add_recording(session, name="retro.ogg")
+
+    run(recording.id)
+
+    session.expire_all()
+    assert recording.status is DocumentStatus.READY
+    assert [w.last for w in speech_api.windows] == [False, False]
+
+
+def test_a_cancel_during_a_window_stops_the_recording_after_it(
+    session: Session, stub_model: None, speech_api: SpeechApi
+) -> None:
+    """The window in flight finishes; no other is sent, and the decoded audio
+    goes while the windows written stay for a retry."""
+    recording = add_recording(session)
+
+    def cancel(n: int) -> None:
+        with create_session_factory(session.get_bind())() as api:
+            api.get(Document, recording.id).status = DocumentStatus.CANCELLED
+            api.commit()
+
+    speech_api.during_window = cancel
+
+    run(recording.id)
+
+    session.expire_all()
+    assert recording.status is DocumentStatus.CANCELLED
+    assert len(speech_api.windows) == 1
+    assert [w["n"] for w in checkpoint(recording)] == [0]
+    assert not work_files(folder_of(recording)).audio.exists()
+
+
+# Failures
+
+
+def test_a_retry_carries_on_from_the_windows_already_written(
+    session: Session, stub_model: None, speech_api: SpeechApi
+) -> None:
+    """A server down past its retries fails the run, and Huey's next run sends
+    only what is left."""
+    speech_api.replies = [(200, speech_api.written(0))] + [
+        SpeechApi.refused(503, "transcription_unavailable")
+    ] * 6
+    recording = add_recording(session)
+
+    with pytest.raises(Exception, match="The speech recognition engine stopped"):
+        ingest_document.call_local(recording.id)
+
+    session.expire_all()
+    assert recording.status is DocumentStatus.FAILED
+    assert recording.error_message == "The speech recognition engine stopped."
+
+    speech_api.replies = []
+    speech_api.windows.clear()
+    session.commit()  # the reads above hold the write lock
+    run(recording.id)
+
+    session.expire_all()
+    assert recording.status is DocumentStatus.READY
+    assert len(speech_api.windows) == 1
+
+
+def test_a_restarting_server_or_a_dictation_only_delays_a_window(
+    session: Session, stub_model: None, speech_api: SpeechApi
+) -> None:
+    """A dropped connection is asked again, and a dictation's turn waited out."""
+    speech_api.replies = [
+        SpeechApi.refused(503, "transcription_unavailable"),
+        SpeechApi.waiting("dictation"),
+        SpeechApi.waiting("hold"),
+    ]
+    recording = add_recording(session)
+
+    run(recording.id)
+
+    session.expire_all()
+    assert recording.status is DocumentStatus.READY
+    assert len(speech_api.windows) == 5
+
+
+def test_ten_minutes_short_of_memory_fails_with_the_apis_sentence(
+    session: Session,
+    stub_model: None,
+    speech_api: SpeechApi,
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """Summed across yields; the person reads what to free."""
+    monkeypatch.setattr(window_loop, "MEMORY_WAIT_LIMIT_S", 0.0)
+    sentence = "Transcribing needs about 2.9 GB free; this computer has 1.8 GB."
+    speech_api.replies = [SpeechApi.waiting("memory", sentence)]
+    recording = add_recording(session)
+
+    refused(recording.id)
+
+    session.expire_all()
+    assert recording.error_message == sentence
+
+
+def test_a_server_down_for_two_minutes_fails_and_is_tried_again(
+    session: Session,
+    stub_model: None,
+    speech_api: SpeechApi,
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """In the seconds after a choice or a crash it is waited for, not longer."""
+    monkeypatch.setattr(readiness, "GIVE_UP_SECONDS", 0.0)
+    speech_api.status = {
+        **speech_api.status,
+        "ready": False,
+        "reason": "transcription_unavailable",
+    }
+    recording = add_recording(session)
+
+    with pytest.raises(Exception, match="The speech recognition engine stopped"):
+        ingest_document.call_local(recording.id)
+
+    session.expire_all()
+    assert recording.status is DocumentStatus.FAILED
+
+
+def test_a_recording_with_no_speech_fails_without_a_retry(
+    session: Session, stub_model: None, speech_api: SpeechApi, real_silero_vad
+) -> None:
+    """Silence is not a transcript; nothing reaches the model."""
+    recording = add_recording(session, name="silence.wav", content=silent_wav(40))
+
+    refused(recording.id)
+
+    session.expire_all()
+    assert recording.error_message == "No speech was found in this recording."
+    assert speech_api.windows == []
+
+
+def test_a_file_with_no_audio_fails_without_a_retry(
+    session: Session, stub_model: None, speech_api: SpeechApi
+) -> None:
+    """A header and no samples."""
+    recording = add_recording(session, name="empty.wav", content=silent_wav(0))
+
+    refused(recording.id)
+
+    session.expire_all()
+    assert recording.error_message == "This file has no audio."
+
+
+def test_a_file_the_decoder_cannot_read_fails_without_a_retry(
+    session: Session, stub_model: None, speech_api: SpeechApi
+) -> None:
+    """The same bytes would fail the same way; the worker lives on."""
+    recording = add_recording(
+        session, name="broken.mp3", content=b"ID3\x04" + bytes(range(256)) * 8
+    )
+
+    refused(recording.id)
+
+    session.expire_all()
+    assert recording.error_message == "This recording could not be decoded."
+
+
+def test_a_disk_without_room_fails_with_what_it_needs(
+    session: Session,
+    stub_model: None,
+    speech_api: SpeechApi,
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """Checked by the decoder from the file's own length before it writes."""
+    monkeypatch.setattr(
+        transcript_job,
+        "decode_in_child",
+        lambda *_a, **_k: DecodeOutcome("no_room", needed_bytes=2_460_000_000),
+    )
+    recording = add_recording(session)
+
+    refused(recording.id)
+
+    session.expire_all()
+    assert recording.error_message == (
+        "Transcribing needs about 2.5 GB of free disk space."
+    )
+
+
+def test_another_model_chosen_midway_writes_the_whole_transcript(
+    session: Session, stub_model: None, speech_api: SpeechApi
+) -> None:
+    """One model's text is never stitched to another's."""
+    other = {"text": "Other words.", "language": "en", "model_id": "other-model"}
+    speech_api.replies = [
+        (200, speech_api.written(0)),
+        (200, other),
+        (200, other),
+        (200, other),
+    ]
+    recording = add_recording(session)
+
+    run(recording.id)
+
+    session.expire_all()
+    assert recording.document_metadata["transcription"]["model"] == "other-model"
+    assert "Words of window 0." not in recording.content
+    assert len(speech_api.windows) == 4

surfsense_local/backend/tests/packaging/test_frozen_document_scripts.py

@@ -1,7 +1,9 @@
-"""The frozen worker runs document scripts with every library a script is told it has.
+"""The frozen worker runs document scripts with every library a script is told it has,
+and decodes a recording in its decode child.
 
 A script is not analysed when the worker is frozen, so whatever it imports, or
-a library imports for it by name, ships only if the spec says so.
+a library imports for it by name, ships only if the spec says so; the decode
+child's miniaudio loads a cffi backend by name too.
 
 SURFSENSE_TEST_FROZEN_WORKER names a worker already frozen, as the release
 workflow's; without it the test freezes one from the spec.
@@ -18,6 +20,7 @@ import openpyxl
 import pypdfium2
 import pytest
 
+from worker.audio_decode.run import decode_in_child
 from worker.document_script.analysis_run import AnalysisInput, run_analysis_script
 from worker.document_script.run import run_document_script
 
@@ -179,3 +182,15 @@ def test_a_frozen_worker_analyses_a_workbook_with_pandas(tmp_path: Path) -> None
     assert run.ok, run.traceback_tail
     assert run.stdout == "12\n"
     assert run.kept == ("stock.csv", "stock.png")
+
+
+@pytest.mark.usefixtures("as_frozen")
+def test_a_frozen_worker_decodes_a_recording_in_its_child(tmp_path: Path) -> None:
+    """worker --decode-audio reads an MP3 to 16 kHz mono, as ingest starts it."""
+    tone = BACKEND / "tests" / "fixtures" / "audio" / "tone.mp3"
+    out = tmp_path / "audio16k.raw"
+
+    outcome = decode_in_child(tone, out, between_polls=lambda: None)
+
+    assert outcome.kind == "decoded", outcome
+    assert out.stat().st_size == pytest.approx(16000 * 2, rel=0.05)

surfsense_local/backend/tests/packaging/test_recording_transcription.py

@@ -0,0 +1,244 @@
+"""A recording through ingest end to end: the decode child, the real Silero, the
+app's API on a real port, and the staged transcription server writing down three
+sentences voiced by Kokoro-82M in English, Spanish and Portuguese, 3 s apart.
+
+Opt-in, like the other packaging tests, and skipped unless
+SURFSENSE_TEST_AUDIO_MODELS names a folder holding the curated recogniser's
+default file under its published name, checked against the manifest's sha256.
+Once it does, a missing server (`pnpm build:audiocpp`, or
+SURFSENSE_TEST_AUDIOCPP_DIR) fails the run; Silero comes from
+`scripts/fetch_silero_vad.py`.
+"""
+
+import contextlib
+import hashlib
+import json
+import os
+import shutil
+import socket
+import subprocess
+import sys
+import threading
+import time
+from collections.abc import Iterator
+from pathlib import Path
+from typing import Any
+
+import httpx
+import pytest
+import uvicorn
+from sqlalchemy import Engine
+
+from api.main import create_app
+from modules.documents.models import Document, DocumentStatus, DocumentType
+from modules.documents.recordings.layout import segments_file
+from modules.llm.catalog.local.dependencies import get_local_catalog
+from modules.llm.catalog.local.engines.audiocpp.engine import AudioCppEngine
+from modules.llm.catalog.local.installs import InstalledBuild, record_install
+from modules.llm.catalog.local.manifest import load_local_manifest
+from modules.workspaces.models import Workspace
+from shared.config import get_llm_settings, get_storage_settings
+from shared.db import create_session_factory
+from worker.ingestion import run
+
+pytestmark = pytest.mark.packaging
+
+BACKEND = Path(__file__).resolve().parents[2]
+STAGED = Path(
+    os.environ.get(
+        "SURFSENSE_TEST_AUDIOCPP_DIR", BACKEND.parent / "electron" / "audiocpp"
+    )
+)
+MODELS_DIR = os.environ.get("SURFSENSE_TEST_AUDIO_MODELS")
+SERVER = STAGED / (
+    "audiocpp_server.exe" if sys.platform == "win32" else "audiocpp_server"
+)
+SPEECH = BACKEND / "tests" / "fixtures" / "audio" / "speech.ogg"
+
+(RECOGNISER,) = [m for m in load_local_manifest().models if m.transcription]
+PINNED = RECOGNISER.builds[0].files[0]
+NAME = PINNED.path.rsplit("/", 1)[-1]
+MODEL = NAME.removesuffix(".gguf")
+
+if MODELS_DIR is None or not (Path(MODELS_DIR) / NAME).is_file():
+    pytest.skip(f"needs {NAME} in SURFSENSE_TEST_AUDIO_MODELS", allow_module_level=True)
+
+# A word or two from each sentence, as Nemotron writes them.
+HEARD = ("quarterly report", "informe trimestral", "relatório trimestral")
+
+
+def _sha256(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as handle:
+        for block in iter(lambda: handle.read(1 << 20), b""):
+            digest.update(block)
+    return digest.hexdigest()
+
+
+def _free_port() -> int:
+    with socket.socket() as probe:
+        probe.bind(("127.0.0.1", 0))
+        return probe.getsockname()[1]
+
+
+@pytest.fixture
+def audio_folder(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
+    """The recogniser installed as a download leaves it; the engine's startup
+    writes the transcription server's config from that."""
+    audio = tmp_path / "audio"
+    audio.mkdir()
+    source = Path(MODELS_DIR) / NAME  # type: ignore[arg-type]
+    assert _sha256(source) == PINNED.sha256, f"{NAME} is not the pinned file"
+    try:
+        (audio / NAME).symlink_to(source)
+    except OSError:  # Windows without the symlink privilege
+        shutil.copyfile(source, audio / NAME)
+    record_install(
+        audio,
+        InstalledBuild(
+            model_id=MODEL,
+            repo=PINNED.repo,
+            revision=PINNED.revision,
+            quantization=RECOGNISER.builds[0].quantization,
+            weights=(NAME,),
+        ),
+    )
+    AudioCppEngine(audio, [RECOGNISER]).on_startup()
+    monkeypatch.setattr(get_llm_settings(), "audio_models_dir", audio)
+    get_local_catalog.cache_clear()
+    yield audio
+    get_local_catalog.cache_clear()
+
+
+@pytest.fixture
+def transcription_server(
+    audio_folder: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> Iterator[str]:
+    """The server as Electron's transcription sidecar starts it, told to the API."""
+    assert SERVER.exists(), f"no staged audio.cpp server at {SERVER}"
+    port = _free_port()
+    threads = str(max(1, min(8, (os.cpu_count() or 2) // 2)))
+    with (tmp_path / "audiocpp.log").open("wb") as log:
+        process = subprocess.Popen(
+            [
+                str(SERVER),
+                *("--config", str(audio_folder / "transcription.json")),
+                *("--host", "127.0.0.1", "--port", str(port)),
+                *("--backend", "cpu", "--threads", threads),
+                *("--max-loaded-models", "1", "--idle-unload-ms", "300000"),
+                *("--min-free-memory-mb", "1024", "--no-ui"),
+            ],
+            cwd=STAGED,
+            stdout=log,
+            stderr=subprocess.STDOUT,
+        )
+        url = f"http://127.0.0.1:{port}"
+        try:
+            deadline = time.monotonic() + 60
+            while time.monotonic() < deadline:
+                with contextlib.suppress(httpx.HTTPError):
+                    if httpx.get(f"{url}/health").status_code == 200:
+                        break
+                time.sleep(0.5)
+            else:
+                pytest.fail("audio.cpp's server never answered /health")
+            monkeypatch.setattr(get_llm_settings(), "transcription_base_url", url)
+            yield url
+        finally:
+            process.terminate()
+            process.wait(timeout=30)
+
+
+@pytest.fixture
+def api(
+    transcription_server: str, engine: Engine, monkeypatch: pytest.MonkeyPatch
+) -> Iterator[str]:
+    """The app's API on a real port, where the worker is told it is."""
+    port = _free_port()
+    server = uvicorn.Server(
+        uvicorn.Config(create_app(), host="127.0.0.1", port=port, log_level="warning")
+    )
+    thread = threading.Thread(target=server.run, daemon=True)
+    thread.start()
+    deadline = time.monotonic() + 30
+    while not server.started:
+        assert time.monotonic() < deadline, "the API did not start"
+        time.sleep(0.05)
+    monkeypatch.setenv("SURFSENSE_LOCAL_HOST", "127.0.0.1")
+    monkeypatch.setenv("SURFSENSE_LOCAL_PORT", str(port))
+    url = f"http://127.0.0.1:{port}"
+    chosen = httpx.put(
+        f"{url}/llm/selection/speech_to_text",
+        json={"provider": "audiocpp", "name": MODEL},
+        timeout=30,
+    )
+    assert chosen.status_code == 200, chosen.text
+    yield url
+    server.should_exit = True
+    thread.join(timeout=10)
+
+
+@pytest.fixture
+def stub_embedder(monkeypatch: pytest.MonkeyPatch) -> None:
+    """The transcript is what is under test, not the embedder."""
+
+    def embed(spec: Any, texts: list[str], _purpose: Any) -> list[list[float]]:
+        return [[float(len(text) % 97)] * spec.dimension for text in texts]
+
+    monkeypatch.setattr("modules.embedding.encoder.embed", embed)
+    monkeypatch.setattr(
+        "worker.ingestion.chunking._default_tokenizer", lambda: "character"
+    )
+
+
+def test_a_recording_becomes_a_transcript_through_the_real_server(
+    api: str, engine: Engine, real_silero_vad, stub_embedder, record_property
+) -> None:
+    """Two windows, cut at a pause; every sentence written down at its time."""
+    with create_session_factory(engine)() as session:
+        workspace = Workspace(name="Recordings")
+        session.add(workspace)
+        session.flush()
+        recording = Document(
+            workspace_id=workspace.id,
+            title="standup.ogg",
+            document_type=DocumentType.FILE,
+            document_metadata={"mime_type": "audio/ogg", "suffix": ".ogg"},
+        )
+        session.add(recording)
+        session.commit()
+        folder = get_storage_settings().document_dir(workspace.id, recording.id)
+        folder.mkdir(parents=True)
+        shutil.copyfile(SPEECH, folder / "standup.ogg")
+        document_id = recording.id
+
+    started = time.monotonic()
+    run(document_id)
+    seconds = time.monotonic() - started
+
+    with create_session_factory(engine)() as session:
+        recording = session.get(Document, document_id)
+        assert recording.status is DocumentStatus.READY, recording.error_message
+        content = recording.content
+        metadata = recording.document_metadata["transcription"]
+        session.commit()
+    record_property("seconds", round(seconds, 1))
+    # Escaped: a Windows console cannot print every script.
+    sys.stdout.write(f"\n{seconds:.1f} s\n{content!a}\n")
+    paragraphs = content.split("\n\n")
+    assert paragraphs[0].startswith("Transcript of standup.ogg · 0:36 · ")
+    assert paragraphs[1].startswith("[00:00:00] ")
+    assert paragraphs[2].startswith("[00:00:2")
+    for words in HEARD:
+        assert words in content.lower(), words
+    assert metadata["model"] == MODEL
+    assert metadata["language"] in {"en", "es", "pt"}
+    segments = json.loads(segments_file(folder).read_text(encoding="utf-8"))
+    assert [w["line"] for w in segments["windows"]] == [3, 5]
+
+
+def test_the_api_status_reads_ready_once_the_model_is_chosen(api: str) -> None:
+    """What the job asks first, before decoding anything."""
+    status = httpx.get(f"{api}/llm/transcription/status", timeout=10).json()
+
+    assert (status["ready"], status["model"]) == (True, MODEL)

surfsense_local/backend/tests/unit/documents/test_upload_validation.py

@@ -35,6 +35,11 @@ def test_the_public_allowlist_is_deliberately_small() -> None:
         ".tiff",
         ".bmp",
         ".webp",
+        ".wav",
+        ".mp3",
+        ".flac",
+        ".ogg",
+        ".oga",
     } == SUPPORTED_UPLOAD_SUFFIXES
 
 
@@ -94,3 +99,80 @@ def test_ooxml_container_must_match_its_extension(
     wrong_suffix = ".pptx" if suffix == ".docx" else ".docx"
     with pytest.raises(ValueError, match="file contents do not match"):
         validate_upload(upload, wrong_suffix)
+
+
+# Made with ffmpeg from a 440 Hz tone, stereo at 44.1 kHz, under a second long.
+AUDIO = Path(__file__).resolve().parents[2] / "fixtures" / "audio"
+
+
+def _wav_header() -> bytes:
+    return b"RIFF\x24\x00\x00\x00WAVEfmt "
+
+
+@pytest.mark.parametrize(
+    ("suffix", "fixture", "mime"),
+    [
+        (".mp3", "tone.mp3", "audio/mpeg"),
+        (".flac", "tone.flac", "audio/flac"),
+        (".ogg", "tone.ogg", "audio/ogg"),
+        (".oga", "tone.ogg", "audio/ogg"),
+    ],
+)
+def test_recordings_are_accepted_by_their_bytes(
+    tmp_path: Path, suffix: str, fixture: str, mime: str
+) -> None:
+    """MP3, FLAC and Ogg Vorbis, as the decoder reads them."""
+    upload = tmp_path / f"meeting{suffix}"
+    upload.write_bytes((AUDIO / fixture).read_bytes())
+
+    assert validate_upload(upload, suffix) == mime
+
+
+def test_a_wave_file_is_a_recording(tmp_path: Path) -> None:
+    """RIFF with a WAVE form, not a WebP's."""
+    upload = tmp_path / "memo.wav"
+    upload.write_bytes(_wav_header() + b"\x00" * 64)
+
+    assert validate_upload(upload, ".wav") == "audio/wav"
+
+
+def test_an_mp3_without_a_tag_starts_with_a_frame(tmp_path: Path) -> None:
+    """Not every encoder writes ID3; an MPEG audio frame header is enough."""
+    upload = tmp_path / "bare.mp3"
+    upload.write_bytes(b"\xff\xfb\x90\x64" + b"\x00" * 64)
+
+    assert validate_upload(upload, ".mp3") == "audio/mpeg"
+
+
+@pytest.mark.parametrize(
+    ("suffix", "content"),
+    [
+        (".wav", b"RIFF\x04\x00\x00\x00WEBPVP8 "),
+        (".mp3", b"%PDF-1.7"),
+        # AAC in ADTS shares the sync word, and the decoder cannot read it.
+        (".mp3", b"\xff\xf1\x50\x80" + b"\x00" * 16),
+        (".flac", b"ID3\x04" + b"\x00" * 16),
+        (".ogg", b"OggS" + b"\x00" * 64),
+    ],
+)
+def test_bytes_that_are_not_the_recording_named_are_refused(
+    tmp_path: Path, suffix: str, content: bytes
+) -> None:
+    """A renamed file never reaches the decoder."""
+    upload = tmp_path / f"renamed{suffix}"
+    upload.write_bytes(content)
+
+    with pytest.raises(ValueError, match="file contents do not match"):
+        validate_upload(upload, suffix)
+
+
+@pytest.mark.parametrize("suffix", [".ogg", ".oga"])
+def test_ogg_opus_is_refused_until_its_decoder_ships(
+    tmp_path: Path, suffix: str
+) -> None:
+    """The decoder reads Vorbis only; the person learns why, file by file."""
+    upload = tmp_path / f"voice-note{suffix}"
+    upload.write_bytes((AUDIO / "tone-opus.ogg").read_bytes())
+
+    with pytest.raises(ValueError, match=r"^Opus support comes in a later update$"):
+        validate_upload(upload, suffix)

surfsense_local/backend/tests/unit/llm/catalog/local/test_install_steps.py

@@ -179,7 +179,7 @@ async def test_an_install_asked_to_select_says_so_before_it_ends(
 ) -> None:
     """Selecting is a step of its own, and the install ends with what it chose."""
 
-    async def choose(session, model_type, provider, name):
+    async def choose(session, model_type, provider, name, *, broker=None):
         return SimpleNamespace(
             model_type=model_type,
             provider=provider,

surfsense_local/backend/tests/unit/worker/audio/test_checkpoint.py

@@ -0,0 +1,52 @@
+"""The windows a job keeps as it goes, and which of them a later run resumes."""
+
+from pathlib import Path
+
+import pytest
+
+from worker.ingestion.audio.checkpoint import (
+    DoneWindow,
+    append_window,
+    resumable_windows,
+)
+
+pytestmark = pytest.mark.unit
+
+
+def done(n: int, model_id: str = "m", intent: str = "auto") -> DoneWindow:
+    """Window n as a run kept it."""
+    return DoneWindow(n, n * 30.0, n * 30.0 + 24, f"window {n}", "en", model_id, intent)
+
+
+def test_windows_written_by_the_same_model_and_language_resume(tmp_path: Path) -> None:
+    """A yield or a retry picks up where the last run stopped."""
+    path = tmp_path / "windows.jsonl"
+    append_window(path, done(0))
+    append_window(path, done(1))
+
+    assert sorted(resumable_windows(path, "m", "auto")) == [0, 1]
+
+
+@pytest.mark.parametrize(("model_id", "intent"), [("other", "auto"), ("m", "ja")])
+def test_another_model_or_language_starts_over(
+    tmp_path: Path, model_id: str, intent: str
+) -> None:
+    """Text from one model is not stitched to another's."""
+    path = tmp_path / "windows.jsonl"
+    append_window(path, done(0))
+
+    assert resumable_windows(path, model_id, intent) == {}
+    assert not path.exists()
+
+
+def test_a_line_torn_by_a_crash_is_left_out_and_writing_goes_on(
+    tmp_path: Path,
+) -> None:
+    """The app quit while a line was being written: that window is done again."""
+    path = tmp_path / "windows.jsonl"
+    append_window(path, done(0))
+    with path.open("ab") as checkpoint:
+        checkpoint.write(b'{"n": 1, "start_s": 30')
+    append_window(path, done(2))
+
+    assert sorted(resumable_windows(path, "m", "auto")) == [0, 2]

surfsense_local/backend/tests/unit/worker/audio/test_speech_regions.py

@@ -0,0 +1,72 @@
+"""Where a decoded recording holds speech, as the window planner reads it: the
+real Silero over the file a minute at a time, and all of it when Silero is not
+there."""
+
+from pathlib import Path
+
+import pytest
+
+from worker.audio_decode.decode import decode_to_pcm
+from worker.ingestion.audio.speech_regions import hear_recording
+
+pytestmark = pytest.mark.unit
+
+# Three sentences voiced by Kokoro-82M through the app's podcast server, in
+# English, Spanish and Portuguese, 3 s of silence between them: 36.65 s.
+SPEECH = Path(__file__).resolve().parents[3] / "fixtures" / "audio" / "speech.ogg"
+SILENCES = (14.0, 26.3)
+
+
+def nothing() -> None:
+    """A job nobody cancels."""
+
+
+@pytest.fixture
+def decoded(tmp_path: Path) -> Path:
+    """The sentences as the decode child leaves them."""
+    out = tmp_path / "audio16k.raw"
+    assert decode_to_pcm(SPEECH, out).kind == "decoded"
+    return out
+
+
+def test_each_sentence_is_speech_and_the_pauses_are_not(
+    decoded: Path, real_silero_vad
+) -> None:
+    """The silence between sentences falls between regions, never inside one."""
+    heard = hear_recording(decoded, between_reads=nothing)
+
+    assert heard.seconds == pytest.approx(36.65, abs=0.01)
+    assert not any(r.start_s < at < r.end_s for r in heard.regions for at in SILENCES)
+    assert sum(r.end_s - r.start_s for r in heard.regions) > 25.0
+
+
+def test_a_silent_recording_holds_no_speech(tmp_path: Path, real_silero_vad) -> None:
+    """What fails the recording with "No speech was found"."""
+    silent = tmp_path / "audio16k.raw"
+    silent.write_bytes(b"\0\0" * 16000 * 90)
+
+    assert hear_recording(silent, between_reads=nothing).regions == []
+
+
+def test_without_silero_all_of_it_is_speech(
+    decoded: Path, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+    """A recording is never lost for want of the detector."""
+    monkeypatch.setenv("SURFSENSE_LOCAL_MODELS_DIR", str(tmp_path / "no-models"))
+
+    heard = hear_recording(decoded, between_reads=nothing)
+
+    assert [(r.start_s, r.end_s) for r in heard.regions] == [(0.0, heard.seconds)]
+
+
+def test_the_job_is_asked_between_reads(decoded: Path, tmp_path: Path) -> None:
+    """A cancel lands while a long file is still being listened to."""
+
+    class CancelError(Exception):
+        pass
+
+    def cancelled() -> None:
+        raise CancelError
+
+    with pytest.raises(CancelError):
+        hear_recording(decoded, between_reads=cancelled)

surfsense_local/backend/tests/unit/worker/audio/test_transcript_text.py

@@ -0,0 +1,142 @@
+"""The transcript a recording's content holds: one header line, then a paragraph
+per window starting with its time, so every chunk the chunker cuts at a blank
+line starts with when it was said."""
+
+import pytest
+
+from modules.llm.catalog.local.manifest import load_local_manifest
+from worker.ingestion.audio.language_names import english_name
+from worker.ingestion.audio.model_name import model_name
+from worker.ingestion.audio.transcript_text import (
+    HeardWindow,
+    header_language,
+    paragraph_lines,
+    transcript_markdown,
+)
+
+pytestmark = pytest.mark.unit
+
+LISTED = ("en", "es", "ja")
+GUIDE = "[hh:mm:ss] marks where each passage starts."
+
+
+def heard(start_s: float, text: str, language: str | None) -> HeardWindow:
+    """A 20 s window written down."""
+    return HeardWindow(
+        start_s=start_s, end_s=start_s + 20, text=text, language=language
+    )
+
+
+def test_each_window_is_a_paragraph_under_one_header() -> None:
+    """The header names the file, its length, the language and the model."""
+    text = transcript_markdown(
+        title="meeting.mp3",
+        seconds=2530.4,
+        windows=[heard(0.4, "Hello all.", "ja"), heard(26.0, "Next item.", "ja")],
+        language="Japanese (detected)",
+        model="Nemotron 3.5 ASR Streaming 0.6B",
+    )
+
+    assert text == (
+        "Transcript of meeting.mp3 · 42:10 · Japanese (detected) · "
+        f"Nemotron 3.5 ASR Streaming 0.6B. {GUIDE}\n\n"
+        "[00:00:00] Hello all.\n\n"
+        "[00:00:26] Next item.\n"
+    )
+
+
+def test_hours_show_in_the_length_and_every_time() -> None:
+    """A long recording's places stay exact past the first hour."""
+    text = transcript_markdown(
+        title="lecture.flac",
+        seconds=3723.0,
+        windows=[heard(3661.9, "Questions?", "en")],
+        language="English",
+        model="m",
+    )
+
+    assert text.startswith("Transcript of lecture.flac · 1:02:03 · English · m.")
+    assert text.endswith("[01:01:01] Questions?\n")
+
+
+def test_a_window_with_nothing_written_has_no_paragraph() -> None:
+    """A cough the detector let through leaves no empty timestamp."""
+    text = transcript_markdown(
+        title="a.wav",
+        seconds=60,
+        windows=[heard(0, "", None), heard(30, "Right.", "en")],
+        language="English (detected)",
+        model="m",
+    )
+
+    assert text.count("[00:") == 1
+
+
+@pytest.mark.parametrize(
+    ("pinned", "languages", "label"),
+    [
+        # Set by the user: what they said they speak.
+        ("ja", ["en", "en"], "Japanese"),
+        (None, ["ja", "en", "ja"], "Japanese (detected)"),
+        # Detected, but not one the model is listed for.
+        (None, ["sv", "sv"], "Swedish (detected) · lower accuracy with this model"),
+        (None, [None, None], "language not identified"),
+    ],
+)
+def test_the_header_names_the_language_most_windows_were_heard_in(
+    pinned: str | None, languages: list[str | None], label: str
+) -> None:
+    """Each window keeps its own; the header takes the majority."""
+    windows = [heard(n * 30, "words", code) for n, code in enumerate(languages)]
+
+    assert header_language(windows, pinned, LISTED) == label
+
+
+def test_a_tie_goes_to_the_language_heard_for_longer() -> None:
+    """Two windows each: the longer speech names the recording."""
+    windows = [
+        heard(0, "a", "es"),
+        heard(30, "b", "en"),
+        heard(60, "c", "es"),
+        HeardWindow(start_s=90, end_s=118, text="d", language="en"),
+    ]
+
+    assert header_language(windows, None, LISTED) == "English (detected)"
+
+
+def test_every_language_a_recogniser_may_answer_in_has_a_name() -> None:
+    """A curated model's listed and detected languages all read in the header."""
+    codes = {
+        code
+        for model in load_local_manifest().models
+        if model.transcription is not None
+        for code in (*model.transcription.languages, *model.transcription.detected_only)
+    }
+
+    assert codes
+    assert all(english_name(code) != code for code in codes)
+
+
+def test_the_model_is_named_as_settings_shows_it() -> None:
+    """The server knows it by its file; the header by the curated entry."""
+    assert model_name("nemotron-3.5-asr-streaming-0.6b-q8_0") == (
+        "Nemotron 3.5 ASR Streaming 0.6B"
+    )
+    assert model_name("someone-elses-model") == "someone-elses-model"
+
+
+def test_each_paragraph_is_one_line_where_its_segment_says() -> None:
+    """segments.json points at the line a chunk records, for seeking later."""
+    windows = [
+        heard(0, "First\nline.", "en"),
+        heard(30, "", None),
+        heard(60, "Second.", "en"),
+    ]
+    text = transcript_markdown(
+        title="a.wav", seconds=90, windows=windows, language="English", model="m"
+    )
+
+    lines = text.splitlines()
+    assert paragraph_lines(windows) == [3, None, 5]
+    assert lines[2] == "[00:00:00] First line."
+    assert lines[4] == "[00:01:00] Second."

surfsense_local/backend/tests/unit/worker/audio/test_windows.py

@@ -0,0 +1,99 @@
+"""Speech cut into the windows the recogniser is sent: at most 28 s each, cut at
+pauses, silence between them never sent, and each window at its own place in
+the recording so the transcript keeps its times."""
+
+import pytest
+
+from modules.llm.transcription.voice_activity.speech_spans import SpeechSpan
+from worker.ingestion.audio.speech_regions import padded
+from worker.ingestion.audio.windows import Window, plan_windows
+
+pytestmark = pytest.mark.unit
+
+FRAME = 0.032
+
+
+def steady(seconds: float) -> list[float]:
+    """One loudness for every frame: nowhere quieter than anywhere else."""
+    return [1.0] * int(seconds / FRAME + 1)
+
+
+def test_speech_that_fits_is_one_window() -> None:
+    """A short memo goes in one request, its pause included."""
+    regions = [SpeechSpan(1.0, 5.0), SpeechSpan(6.0, 10.0)]
+
+    assert plan_windows(regions, steady(11), FRAME) == [Window(1.0, 10.0)]
+
+
+def test_silence_between_windows_is_never_sent() -> None:
+    """Two remarks a minute and a half apart: the quiet between them stays home."""
+    regions = [SpeechSpan(2.0, 6.0), SpeechSpan(100.0, 104.0)]
+
+    assert plan_windows(regions, steady(105), FRAME) == [
+        Window(2.0, 6.0),
+        Window(100.0, 104.0),
+    ]
+
+
+def test_a_window_ends_at_the_longest_pause_past_20_seconds() -> None:
+    """Of the places it could end between 20 and 28 s, the clearest break."""
+    regions = [
+        SpeechSpan(0.0, 10.0),
+        SpeechSpan(10.4, 20.5),
+        # 1 s pause: the cut.
+        SpeechSpan(21.5, 24.0),
+        SpeechSpan(24.3, 27.0),
+        SpeechSpan(27.5, 35.0),
+    ]
+
+    assert plan_windows(regions, steady(36), FRAME) == [
+        Window(0.0, 20.5),
+        Window(21.5, 35.0),
+    ]
+
+
+def test_no_window_is_longer_than_28_seconds() -> None:
+    """28 s bounds a request's memory, whatever the speaker does."""
+    regions = [SpeechSpan(n * 4.0, n * 4.0 + 3.0) for n in range(30)]
+
+    windows = plan_windows(regions, steady(121), FRAME)
+
+    assert all(w.end_s - w.start_s <= 28.0 for w in windows)
+    assert windows[0].start_s == 0.0
+    assert windows[-1].end_s == 119.0
+    # Each window starts where speech does: nothing from a pause before it.
+    starts = {r.start_s for r in regions}
+    assert all(w.start_s in starts for w in windows)
+
+
+def test_a_long_run_without_a_pause_is_cut_where_it_is_quietest() -> None:
+    """A lecturer who never stops is cut in the quietest 100 ms between 20 and
+    28 s, and the next window starts right there."""
+    levels = steady(51)
+    quiet = int(23.0 / FRAME)
+    levels[quiet : quiet + 3] = [0.0, 0.0, 0.0]
+
+    first, *rest = plan_windows([SpeechSpan(0.0, 50.0)], levels, FRAME)
+
+    assert first.start_s == 0.0
+    assert first.end_s == pytest.approx((quiet + 1.5) * FRAME)
+    assert rest[0].start_s == first.end_s
+    assert rest[-1].end_s == 50.0
+    assert all(w.end_s - w.start_s <= 28.0 for w in rest)
+
+
+def test_speech_is_padded_by_200_ms_within_the_recording() -> None:
+    """The first and last syllables are not clipped, nor run off either end."""
+    assert padded([SpeechSpan(0.1, 1.0), SpeechSpan(5.0, 9.95)], 10.0) == [
+        SpeechSpan(0.0, pytest.approx(1.2)),
+        SpeechSpan(pytest.approx(4.8), 10.0),
+    ]
+
+
+def test_spans_under_300_ms_apart_once_padded_are_one() -> None:
+    """A breath does not split a sentence into two regions."""
+    joined = padded([SpeechSpan(1.0, 2.0), SpeechSpan(2.6, 3.0)], 10.0)
+    apart = padded([SpeechSpan(1.0, 2.0), SpeechSpan(2.8, 3.0)], 10.0)
+
+    assert joined == [SpeechSpan(pytest.approx(0.8), pytest.approx(3.2))]
+    assert len(apart) == 2

surfsense_local/backend/tests/unit/worker/audio_decode/test_decode.py

@@ -0,0 +1,116 @@
+"""A recording read into 16 kHz mono 16-bit PCM, as the decode child does it,
+and what it reports when it cannot: no room on disk, no audio, bytes the
+decoder cannot read."""
+
+import io
+import math
+import shutil
+import struct
+import wave
+from collections import namedtuple
+from pathlib import Path
+
+import pytest
+
+from worker.audio_decode.decode import decode_to_pcm
+from worker.audio_decode.room import needed_bytes
+
+pytestmark = pytest.mark.unit
+
+# Made with ffmpeg from a 440 Hz tone, stereo at 44.1 kHz.
+AUDIO = Path(__file__).resolve().parents[3] / "fixtures" / "audio"
+RATE = 16000
+
+
+def tone_wav(seconds: float, rate: int = 44100, channels: int = 2) -> bytes:
+    """A 440 Hz tone as a WAV file, at a rate the recogniser does not take."""
+    frames = b"".join(
+        struct.pack("<h", int(8000 * math.sin(2 * math.pi * 440 * n / rate))) * channels
+        for n in range(int(seconds * rate))
+    )
+    out = io.BytesIO()
+    with wave.open(out, "wb") as clip:
+        clip.setnchannels(channels)
+        clip.setsampwidth(2)
+        clip.setframerate(rate)
+        clip.writeframes(frames)
+    return out.getvalue()
+
+
+def samples(path: Path) -> list[int]:
+    """The decoded file's samples."""
+    data = path.read_bytes()
+    return list(struct.unpack(f"<{len(data) // 2}h", data))
+
+
+@pytest.mark.parametrize(
+    ("name", "seconds"),
+    [("tone.mp3", 1.0), ("tone.flac", 0.5), ("tone.ogg", 1.0)],
+)
+def test_each_format_becomes_16_khz_mono(
+    tmp_path: Path, name: str, seconds: float
+) -> None:
+    """MP3, FLAC and Ogg Vorbis at 44.1 kHz stereo, down to what the server reads."""
+    out = tmp_path / "audio16k.raw"
+
+    outcome = decode_to_pcm(AUDIO / name, out)
+
+    assert outcome.kind == "decoded"
+    assert outcome.seconds == pytest.approx(seconds, rel=0.05)
+    decoded = samples(out)
+    assert len(decoded) == pytest.approx(seconds * RATE, rel=0.05)
+    assert max(abs(s) for s in decoded) > 1000
+
+
+def test_a_wave_file_at_another_rate_is_resampled(tmp_path: Path) -> None:
+    """A 44.1 kHz stereo memo keeps its length at 16 kHz."""
+    source = tmp_path / "memo.wav"
+    source.write_bytes(tone_wav(2.0))
+    out = tmp_path / "audio16k.raw"
+
+    outcome = decode_to_pcm(source, out)
+
+    assert (outcome.kind, len(samples(out))) == ("decoded", 2 * RATE)
+
+
+def test_a_file_with_no_samples_has_no_audio(tmp_path: Path) -> None:
+    """A header and nothing after it: nothing to transcribe, and no file left."""
+    source = tmp_path / "empty.wav"
+    source.write_bytes(tone_wav(0.0))
+    out = tmp_path / "audio16k.raw"
+
+    outcome = decode_to_pcm(source, out)
+
+    assert outcome.kind == "no_audio"
+    assert list(tmp_path.iterdir()) == [source]
+
+
+def test_bytes_the_decoder_cannot_read_are_undecodable(tmp_path: Path) -> None:
+    """Refused for good: the same bytes fail the same way again."""
+    source = tmp_path / "broken.mp3"
+    source.write_bytes(b"ID3\x04" + bytes(range(256)) * 8)
+    out = tmp_path / "audio16k.raw"
+
+    outcome = decode_to_pcm(source, out)
+
+    assert outcome.kind == "undecodable"
+    assert list(tmp_path.iterdir()) == [source]
+
+
+def test_a_disk_without_room_for_the_decoded_audio_is_refused_first(
+    tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+    """Checked from the file's own length before a byte is written."""
+    usage = namedtuple("usage", "total used free")
+    monkeypatch.setattr(shutil, "disk_usage", lambda _path: usage(1, 1, 10_000))
+    out = tmp_path / "audio16k.raw"
+
+    outcome = decode_to_pcm(AUDIO / "tone.mp3", out)
+
+    assert (outcome.kind, outcome.needed_bytes) == ("no_room", needed_bytes(1.0))
+    assert not out.exists()
+
+
+def test_an_hour_needs_its_pcm_and_some_room_to_spare() -> None:
+    """115.2 MB an hour at 16 kHz 16-bit mono, plus 64 MiB."""
+    assert needed_bytes(3600) == 115_200_000 + 64 * 2**20

surfsense_local/backend/tests/unit/worker/test_notify.py

@@ -3,6 +3,7 @@
 import pytest
 
 from modules.documents.models import Document, DocumentStatus, DocumentType
+from modules.events.schemas import Progress
 from worker import notify
 from worker.notify import notify_document_updates
 
@@ -58,3 +59,20 @@ def test_it_is_a_no_op_without_a_port(monkeypatch: pytest.MonkeyPatch) -> None:
     notify_document_updates(_document())
 
     assert posted is False
+
+
+def test_a_transcribed_window_carries_its_progress(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """The recording's bar moves on the event alone, never stored."""
+    monkeypatch.setenv("SURFSENSE_LOCAL_PORT", "8123")
+    sent: dict = {}
+
+    def fake_post(url: str, *, json: dict, timeout: float) -> None:
+        sent.update(json)
+
+    monkeypatch.setattr(notify.httpx, "post", fake_post)
+
+    notify_document_updates(_document(), Progress(done_s=56.0, total_s=3480.0))
+
+    assert sent["progress"] == {"done_s": 56.0, "total_s": 3480.0}

surfsense_local/backend/uv.lock

@@ -1616,6 +1616,35 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl", hash = "sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8", size = 9979, upload-time = "2022-08-14T12:40:09.779Z" },
 ]
 
+[[package]]
+name = "miniaudio"
+version = "1.71"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "cffi" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/d8/d5/e5439dc08561f73656bfeb3340fc64ab63163e101426593d8fb9a025ff1e/miniaudio-1.71.tar.gz", hash = "sha256:ff51e2887bb673e2e757752b586b3dc924d59aa5fbcae9bbc45f4a111bd3262b", size = 1116480, upload-time = "2026-04-29T21:20:38.182Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/39/d3/71124f5abbcdcae62e040f58d3dca3bd3d90fd01faa7bb276b112387b47a/miniaudio-1.71-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:62db602651bc20a2698f36a0d356d7217ed6f4f917550c7ffb3705c8e8be90cf", size = 377186, upload-time = "2026-04-29T21:20:16.713Z" },
+    { url = "https://files.pythonhosted.org/packages/f7/ac/30a324f758bed1b193e017ec25183cfb10a79e549656331f5d068a2d343a/miniaudio-1.71-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:8fc1a4f084cc1b4b25c567d22f54d1e46bfa505c17ed777c8b198e5c53d0f785", size = 351485, upload-time = "2026-04-29T21:20:17.761Z" },
+    { url = "https://files.pythonhosted.org/packages/fa/62/ae884a9d3b2ebec9c2ed1db857593e74bff45b70c4ab17fccc11db31cbeb/miniaudio-1.71-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:19be6f0a1e601c2237433e579734cfaf6469191b224c20c9e5f73c32ef9ee2b9", size = 643556, upload-time = "2026-04-29T21:20:18.87Z" },
+    { url = "https://files.pythonhosted.org/packages/a5/39/84fc665e2ea8f9f1301b6370226e3a24f08d5ad5d97143b69b4ae7ac260a/miniaudio-1.71-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:e6287f15caa808a88aad0700a182bec1ff6d98769717425adf9ebf41259d1936", size = 645176, upload-time = "2026-04-29T21:20:20.194Z" },
+    { url = "https://files.pythonhosted.org/packages/81/b8/37d9f67d4511da29bdb82b6c73a3ef6f4ebf2fbd30f9524f1e4a84d6f033/miniaudio-1.71-cp312-cp312-win32.whl", hash = "sha256:ab100e5240b104b5326e4ec1be07b6ae461f7d3d4d7a694857fd2f0493d210f9", size = 235095, upload-time = "2026-04-29T21:20:21.653Z" },
+    { url = "https://files.pythonhosted.org/packages/fd/cf/c1a19e6800e725b6e2b4576407620a798d69e3528ebdea9aea84d69d7088/miniaudio-1.71-cp312-cp312-win_amd64.whl", hash = "sha256:f4a44b70b66628b0c307e40ae0ae857695978cae18462179b806d8edc807d416", size = 274252, upload-time = "2026-04-29T21:20:22.824Z" },
+    { url = "https://files.pythonhosted.org/packages/3a/85/44545f767ec21142ffed5f9108406d11dc8a19aafed9bd57621a0892bb60/miniaudio-1.71-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:61b86f26d653040db32d9d15b05446321dd10e45beba25b44f841e26935213d5", size = 377184, upload-time = "2026-04-29T21:20:24.109Z" },
+    { url = "https://files.pythonhosted.org/packages/bd/d1/071a560000c8ce903dc919968ecce40fbe7a73213ac399051b887184f8a3/miniaudio-1.71-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:d9dc15eff711bcfc62a9d05e0c78e4bc34821a455595e049629f2fea7491a523", size = 351488, upload-time = "2026-04-29T21:20:25.183Z" },
+    { url = "https://files.pythonhosted.org/packages/46/24/5873a569451cae5686fb656ebd78ffe0b5eebe48ca21ef61e227d237d20a/miniaudio-1.71-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:12bc33e7e61072b4b541c14e10ef76119d5643e6bbb98e2dec0c0738889438fb", size = 643552, upload-time = "2026-04-29T21:20:26.211Z" },
+    { url = "https://files.pythonhosted.org/packages/90/9b/25785525e6b5ff9afd7f4c4279215dc09c3317ea4d837275b1ae17912b36/miniaudio-1.71-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:70fa2ea5353e6919aca59b8c5768144af009d18c3bca251749d66fb497424563", size = 645171, upload-time = "2026-04-29T21:20:27.494Z" },
+    { url = "https://files.pythonhosted.org/packages/b1/6d/cbfd55fdc40256231f7b0c861e2bf79cc289bfbbe5e15869317944e6d673/miniaudio-1.71-cp313-cp313-win32.whl", hash = "sha256:1bf93aeede652926f27f430f0fd69ef0cf8a949c07b537d6a2f295602c747037", size = 235088, upload-time = "2026-04-29T21:20:29.031Z" },
+    { url = "https://files.pythonhosted.org/packages/8d/8d/d5059c04b247b1079c0e48914a9ec20352910a3b9373060fb258dfd194ab/miniaudio-1.71-cp313-cp313-win_amd64.whl", hash = "sha256:4c849ccb1349f7b3553a77a66fe7e972315185f5c4c44a0bbda7ebcdd224db37", size = 274247, upload-time = "2026-04-29T21:20:29.96Z" },
+    { url = "https://files.pythonhosted.org/packages/16/e7/b3e0df641d2d5283446d7960fc407195ba722b9a0789bb0a1429bf9ee855/miniaudio-1.71-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:3ef441d139264f8a5dcb9aa6fcd0b1e1e69f58715baae416ff33f045ffba6ad5", size = 378418, upload-time = "2026-04-29T21:20:31.341Z" },
+    { url = "https://files.pythonhosted.org/packages/66/ea/f5940232d0c83777562e802f376a841046d78e94753450fb9a6685a44190/miniaudio-1.71-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:84139a10ef172acd762ccf120142877b037a1aaf71def99d2c75f66329f89d8b", size = 351473, upload-time = "2026-04-29T21:20:32.464Z" },
+    { url = "https://files.pythonhosted.org/packages/fd/a6/6b5ae21b74fe70da935de389e01b3cce86c790ca4083f6a63c3ee922673e/miniaudio-1.71-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8a28ff4ad23e55bbde8808ce525d3bb7d249d7612f77646b30e06fc6b7a778ac", size = 643683, upload-time = "2026-04-29T21:20:33.598Z" },
+    { url = "https://files.pythonhosted.org/packages/4b/43/ef851e2e1d9dfde2b97cc053f0d79c6612088b27044698ed5d8c687f05de/miniaudio-1.71-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:33986d5d725ebcbc253551e7358689bc81b19b6950b33cec8e8c1142ca4fc0a9", size = 645203, upload-time = "2026-04-29T21:20:34.713Z" },
+    { url = "https://files.pythonhosted.org/packages/1b/4a/0da61fea8b8469d51b77d43846572ef9255d54c9b6b65a552446bbd55f90/miniaudio-1.71-cp314-cp314-win32.whl", hash = "sha256:3bbeb1e068fe42475e017e8150e9e345182b583d0dd4d9e77ffa20c39935d9ec", size = 241126, upload-time = "2026-04-29T21:20:35.975Z" },
+    { url = "https://files.pythonhosted.org/packages/dd/d0/ad7bfa63e1baacd2d4803ee04862bb06fcfbbb34a340bf2bf3979e0068dc/miniaudio-1.71-cp314-cp314-win_amd64.whl", hash = "sha256:154b085dd914a0e79e3d93160e1a07aacb27d66c65f9ef6a0d87c1a194f32c04", size = 281740, upload-time = "2026-04-29T21:20:37.07Z" },
+]
+
 [[package]]
 name = "mpire"
 version = "2.10.2"
@@ -3094,6 +3123,7 @@ dependencies = [
     { name = "markdown-it-py" },
     { name = "matplotlib" },
     { name = "mcp" },
+    { name = "miniaudio" },
     { name = "onnxruntime" },
     { name = "openpyxl" },
     { name = "pandas" },
@@ -3139,6 +3169,7 @@ requires-dist = [
     { name = "markdown-it-py", specifier = ">=4.0" },
     { name = "matplotlib", specifier = ">=3.11.2" },
     { name = "mcp", specifier = ">=2.3,<2.4" },
+    { name = "miniaudio", specifier = ">=1.71" },
     { name = "onnxruntime", specifier = ">=1.20" },
     { name = "openpyxl", specifier = ">=3.1.5" },
     { name = "pandas", specifier = ">=3.0" },

surfsense_local/backend/worker.py

@@ -24,6 +24,11 @@ if __name__ == "__main__":
         from worker.document_script.child import main
 
         main(sys.argv[2])
+    elif len(sys.argv) == 4 and sys.argv[1] == "--decode-audio":
+        # Ahead of the consumer's imports too: a decode needs only miniaudio.
+        from worker.audio_decode.child import main
+
+        main(sys.argv[2], sys.argv[3])
     elif len(sys.argv) == 2:
         from worker.consumer import consume
 

surfsense_local/backend/worker/audio_decode/__init__.py

@@ -0,0 +1,3 @@
+"""A recording decoded to 16 kHz mono 16-bit PCM in a child process, worker.py's
+--decode-audio mode, so a malformed file kills the child and never the ingest
+consumer. pyminiaudio (MIT) reads WAV, MP3, FLAC and Ogg Vorbis."""

surfsense_local/backend/worker/audio_decode/child.py

@@ -0,0 +1,34 @@
+"""The decode process: worker.py's --decode-audio mode. It loads no queue,
+database or settings, prints its outcome on one line and exits 0; anything
+else is a decoder that died."""
+
+import os
+import sys
+import threading
+from pathlib import Path
+
+from worker.audio_decode.decode import decode_to_pcm
+
+# Exit status of a child whose worker went away mid-decode.
+ORPHANED = 3
+
+
+def main(source: str, out: str) -> None:
+    _exit_with_the_worker()
+    outcome = decode_to_pcm(Path(source), Path(out))
+    sys.stdout.write(outcome.line() + "\n")
+    sys.stdout.flush()
+
+
+def _exit_with_the_worker() -> None:
+    """The worker holds the other end of stdin, so stdin closing means it is
+    gone: a decode of a long file must not outlive it."""
+
+    def exit_at_eof() -> None:
+        # The raw descriptor: a thread blocked in sys.stdin holds its lock, and
+        # the interpreter aborts at exit when it cannot take it.
+        while os.read(sys.stdin.fileno(), 4096):
+            pass
+        os._exit(ORPHANED)
+
+    threading.Thread(target=exit_at_eof, daemon=True).start()

surfsense_local/backend/worker/audio_decode/decode.py

@@ -0,0 +1,70 @@
+"""The child's work: the recording's length from its headers, the disk checked
+for room, then its samples streamed to `<out>.part`, renamed to `out` once
+every one is written, so a file named `out` is always whole."""
+
+import errno
+import shutil
+import sys
+from pathlib import Path
+
+import miniaudio
+
+from worker.audio_decode.outcome import DecodeOutcome
+from worker.audio_decode.room import needed_bytes
+
+__all__ = ["decode_to_pcm"]
+
+RATE = 16000
+# Four seconds a read: memory stays flat for an 18-hour file.
+_FRAMES_PER_READ = 4 * RATE
+
+
+def decode_to_pcm(source: Path, out: Path) -> DecodeOutcome:
+    """Little-endian signed 16-bit mono at 16 kHz, through miniaudio's own
+    resampler and channel mix."""
+    part = out.with_name(f"{out.name}.part")
+    try:
+        needed = needed_bytes(_header_seconds(source))
+    except miniaudio.MiniaudioError as error:
+        return DecodeOutcome("undecodable", detail=str(error))
+    if shutil.disk_usage(out.parent).free < needed:
+        return DecodeOutcome("no_room", needed_bytes=needed)
+    try:
+        frames = _stream(source, part)
+    except miniaudio.MiniaudioError as error:
+        part.unlink(missing_ok=True)
+        return DecodeOutcome("undecodable", detail=str(error))
+    except OSError as error:
+        part.unlink(missing_ok=True)
+        if error.errno == errno.ENOSPC:
+            return DecodeOutcome("no_room", needed_bytes=needed)
+        raise
+    if frames == 0:
+        part.unlink()
+        return DecodeOutcome("no_audio")
+    part.replace(out)
+    return DecodeOutcome("decoded", seconds=frames / RATE)
+
+
+def _header_seconds(source: Path) -> float:
+    # miniaudio picks the reader by extension and knows .oga by no name.
+    if source.suffix.lower() == ".oga":
+        return miniaudio.vorbis_get_file_info(str(source)).duration
+    return miniaudio.get_file_info(str(source)).duration
+
+
+def _stream(source: Path, part: Path) -> int:
+    frames = 0
+    with part.open("wb") as written:
+        for chunk in miniaudio.stream_file(
+            str(source),
+            output_format=miniaudio.SampleFormat.SIGNED16,
+            nchannels=1,
+            sample_rate=RATE,
+            frames_to_read=_FRAMES_PER_READ,
+        ):
+            if sys.byteorder == "big":
+                chunk.byteswap()
+            written.write(chunk.tobytes())
+            frames += len(chunk)
+    return frames

surfsense_local/backend/worker/audio_decode/outcome.py

@@ -0,0 +1,28 @@
+"""What one decode came to, as the child prints it on one line for its parent."""
+
+import json
+from dataclasses import asdict, dataclass
+from typing import Literal
+
+__all__ = ["DecodeOutcome", "OutcomeKind"]
+
+# `no_room` and `no_audio` and `undecodable` would end the same way again.
+OutcomeKind = Literal["decoded", "no_room", "no_audio", "undecodable"]
+
+
+@dataclass(frozen=True)
+class DecodeOutcome:
+    kind: OutcomeKind
+    # The decoded length, once decoded.
+    seconds: float = 0.0
+    # For `no_room`: what the decoded audio and its checkpoints take.
+    needed_bytes: int = 0
+    # For `undecodable`: the decoder's own words, for the log.
+    detail: str = ""
+
+    def line(self) -> str:
+        return json.dumps(asdict(self))
+
+    @classmethod
+    def from_line(cls, line: str) -> "DecodeOutcome":
+        return cls(**json.loads(line))

surfsense_local/backend/worker/audio_decode/room.py

@@ -0,0 +1,11 @@
+"""The disk a decoded recording takes: 16 kHz of 16-bit mono samples, 115.2 MB
+an hour, and room to spare for the checkpoints beside it."""
+
+import math
+
+BYTES_PER_SECOND = 16000 * 2
+SPARE_BYTES = 64 * 2**20
+
+
+def needed_bytes(seconds: float) -> int:
+    return math.ceil(seconds * BYTES_PER_SECOND) + SPARE_BYTES

surfsense_local/backend/worker/audio_decode/run.py

@@ -0,0 +1,111 @@
+"""Decode a recording in the worker's own binary, started as a child, and wait
+for it while the job keeps checking whether it was cancelled."""
+
+import logging
+import subprocess
+import sys
+import threading
+from collections.abc import Callable
+from pathlib import Path
+from typing import IO
+
+from worker.audio_decode.outcome import DecodeOutcome
+
+__all__ = ["decode_in_child"]
+
+logger = logging.getLogger(__name__)
+
+POLL_SECONDS = 1.0
+# How long its last words are still read once it has exited.
+_LAST_OUTPUT_SECONDS = 5.0
+# What of a dead decoder's output reaches the log.
+_TAIL_CHARS = 2000
+_WORKER_ENTRY = Path(__file__).resolve().parents[2] / "worker.py"
+# A packaged worker may have no console; the child must not open one.
+_NO_WINDOW = subprocess.CREATE_NO_WINDOW if sys.platform == "win32" else 0
+
+
+def decode_in_child(
+    source: Path, out: Path, *, between_polls: Callable[[], None]
+) -> DecodeOutcome:
+    """`out` holds the whole recording once this answers `decoded`, and is
+    absent otherwise. `between_polls` runs about every second while the child
+    works: whatever it raises stops the child and is raised here."""
+    process = subprocess.Popen(
+        _command(source, out),
+        stdin=subprocess.PIPE,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.STDOUT,
+        creationflags=_NO_WINDOW,
+    )
+    printed = _Printed(process.stdout)
+    try:
+        exit_code = _wait(process, between_polls)
+    except BaseException:
+        process.kill()
+        process.wait()
+        _remove(out)
+        raise
+    finally:
+        # The child exits when this closes, should it still be running.
+        process.stdin.close()
+    said = printed.text()
+    outcome = _outcome(said)
+    if exit_code == 0 and outcome is not None:
+        return outcome
+    _remove(out)
+    logger.warning(
+        "transcription: the decoder exited with %s: %s", exit_code, said[-_TAIL_CHARS:]
+    )
+    return DecodeOutcome("undecodable", detail=f"the decoder exited with {exit_code}")
+
+
+def _command(source: Path, out: Path) -> list[str]:
+    """The worker in decode mode: its own binary when packaged, else worker.py."""
+    if getattr(sys, "frozen", False):
+        return [sys.executable, "--decode-audio", str(source), str(out)]
+    return [
+        sys.executable,
+        str(_WORKER_ENTRY),
+        "--decode-audio",
+        str(source),
+        str(out),
+    ]
+
+
+def _wait(process: subprocess.Popen, between_polls: Callable[[], None]) -> int:
+    while True:
+        try:
+            return process.wait(timeout=POLL_SECONDS)
+        except subprocess.TimeoutExpired:
+            between_polls()
+
+
+def _outcome(said: str) -> DecodeOutcome | None:
+    lines = said.strip().splitlines()
+    try:
+        return DecodeOutcome.from_line(lines[-1]) if lines else None
+    except (ValueError, TypeError):
+        return None
+
+
+def _remove(out: Path) -> None:
+    out.unlink(missing_ok=True)
+    out.with_name(f"{out.name}.part").unlink(missing_ok=True)
+
+
+class _Printed:
+    """Read on a thread, so a decoder that prints much never blocks on the pipe."""
+
+    def __init__(self, pipe: IO[bytes]) -> None:
+        self._said = b""
+        self._thread = threading.Thread(target=self._read, args=(pipe,), daemon=True)
+        self._thread.start()
+
+    def _read(self, pipe: IO[bytes]) -> None:
+        with pipe:
+            self._said = pipe.read()
+
+    def text(self) -> str:
+        self._thread.join(_LAST_OUTPUT_SECONDS)
+        return self._said.decode(errors="replace")

surfsense_local/backend/worker/ingestion/audio/__init__.py

@@ -0,0 +1,3 @@
+"""A recording as a source: decoded in a child, cut at its pauses into windows,
+each written down through the API, and kept as a timestamped transcript. The
+job resumes from its checkpoint after a yield, a retry or a restart."""

surfsense_local/backend/worker/ingestion/audio/checkpoint.py

@@ -0,0 +1,54 @@
+"""The windows a recording's job has transcribed, one JSON line each, appended
+as each comes back: a yield, a retry or a restart carries on from here, so a
+long file is never transcribed twice by the same model for the same language."""
+
+import json
+from dataclasses import asdict, dataclass
+from pathlib import Path
+
+__all__ = ["DoneWindow", "append_window", "resumable_windows"]
+
+
+@dataclass(frozen=True)
+class DoneWindow:
+    n: int
+    start_s: float
+    end_s: float
+    text: str
+    language: str | None
+    # What wrote it: a checkpoint from another model or language is not resumed.
+    model_id: str
+    intent: str
+
+
+def resumable_windows(path: Path, model_id: str, intent: str) -> dict[int, DoneWindow]:
+    """The windows done by this model for this language, by number. Any other
+    checkpoint is removed: the recording starts over."""
+    try:
+        lines = path.read_text(encoding="utf-8").splitlines()
+    except FileNotFoundError:
+        return {}
+    done: dict[int, DoneWindow] = {}
+    for line in lines:
+        try:
+            window = DoneWindow(**json.loads(line))
+        except (ValueError, TypeError):
+            continue  # a line the last run was writing when it stopped
+        if (window.model_id, window.intent) != (model_id, intent):
+            path.unlink()
+            return {}
+        done[window.n] = window
+    return done
+
+
+def append_window(path: Path, window: DoneWindow) -> None:
+    """Raises OSError when the disk is full."""
+    line = json.dumps(asdict(window), ensure_ascii=False).encode() + b"\n"
+    with path.open("a+b") as checkpoint:
+        size = checkpoint.seek(0, 2)
+        if size:
+            checkpoint.seek(size - 1)
+            if checkpoint.read(1) != b"\n":
+                # A line torn when the last run stopped stays a line of its own.
+                line = b"\n" + line
+        checkpoint.write(line)

surfsense_local/backend/worker/ingestion/audio/failures.py

@@ -0,0 +1,38 @@
+"""Why a recording failed, in the sentence its row shows as written."""
+
+__all__ = [
+    "ENGINE_STOPPED",
+    "NO_AUDIO",
+    "NO_SPEECH",
+    "UNDECODABLE",
+    "AwaitingSpeechModelError",
+    "RecordingFailedError",
+    "RecordingRefusedError",
+    "no_room",
+]
+
+NO_SPEECH = "No speech was found in this recording."
+NO_AUDIO = "This file has no audio."
+UNDECODABLE = "This recording could not be decoded."
+ENGINE_STOPPED = "The speech recognition engine stopped."
+
+
+class RecordingFailedError(Exception):
+    """The message is a sentence for the person; Huey tries again."""
+
+
+class RecordingRefusedError(RecordingFailedError):
+    """The same recording would fail the same way again: no retry."""
+
+
+class AwaitingSpeechModelError(RecordingRefusedError):
+    """No speech model is chosen: choosing one queues the recording again."""
+
+    def __init__(self) -> None:
+        super().__init__("Waiting for a speech model.")
+
+
+def no_room(needed_bytes: int) -> RecordingRefusedError:
+    return RecordingRefusedError(
+        f"Transcribing needs about {needed_bytes / 1e9:.1f} GB of free disk space."
+    )

surfsense_local/backend/worker/ingestion/audio/language_names.py

@@ -0,0 +1,49 @@
+"""Languages by their English names, for the transcript's header: the codes a
+curated recogniser lists or may detect. The screen names them itself, in the
+interface language."""
+
+__all__ = ["english_name"]
+
+_NAMES = {
+    "ar": "Arabic",
+    "bg": "Bulgarian",
+    "cs": "Czech",
+    "da": "Danish",
+    "de": "German",
+    "el": "Greek",
+    "en": "English",
+    "es": "Spanish",
+    "et": "Estonian",
+    "fi": "Finnish",
+    "fr": "French",
+    "he": "Hebrew",
+    "hi": "Hindi",
+    "hr": "Croatian",
+    "hu": "Hungarian",
+    "it": "Italian",
+    "ja": "Japanese",
+    "ko": "Korean",
+    "lt": "Lithuanian",
+    "lv": "Latvian",
+    "mt": "Maltese",
+    "nb": "Norwegian Bokmål",
+    "nl": "Dutch",
+    "nn": "Norwegian Nynorsk",
+    "pl": "Polish",
+    "pt": "Portuguese",
+    "ro": "Romanian",
+    "ru": "Russian",
+    "sk": "Slovak",
+    "sl": "Slovenian",
+    "sv": "Swedish",
+    "th": "Thai",
+    "tr": "Turkish",
+    "uk": "Ukrainian",
+    "vi": "Vietnamese",
+    "zh": "Chinese",
+}
+
+
+def english_name(code: str) -> str:
+    """The code itself for one no curated model answers in."""
+    return _NAMES.get(code, code)

surfsense_local/backend/worker/ingestion/audio/model_name.py

@@ -0,0 +1,25 @@
+"""The name a transcript's header gives the model that wrote it, as Settings
+shows it: the curated entry whose file the server knows the model by."""
+
+from functools import cache
+from pathlib import PurePosixPath
+
+from modules.llm.catalog.local.manifest import load_local_manifest
+
+__all__ = ["model_name"]
+
+
+def model_name(model_id: str) -> str:
+    """The id itself for a model no curated entry names."""
+    return _names().get(model_id, model_id)
+
+
+@cache
+def _names() -> dict[str, str]:
+    return {
+        PurePosixPath(file.path).stem: model.name
+        for model in load_local_manifest().models
+        if model.transcription is not None
+        for build in model.builds
+        for file in build.files
+    }

surfsense_local/backend/worker/ingestion/audio/plan_file.py

@@ -0,0 +1,49 @@
+"""The windows a recording is cut into, written once the speech pass ends: a
+job that finds it skips the pass, and one that does not decodes again."""
+
+import json
+from dataclasses import dataclass
+from pathlib import Path
+
+from worker.ingestion.audio.windows import Window
+
+__all__ = ["Plan", "read_plan", "write_plan"]
+
+_VERSION = 1
+
+
+@dataclass(frozen=True)
+class Plan:
+    # The decoded recording's length.
+    seconds: float
+    windows: list[Window]
+
+
+def write_plan(path: Path, plan: Plan) -> None:
+    """Replaced whole, so a reader never sees half a plan."""
+    partial = path.with_name(f"{path.name}.partial")
+    partial.write_text(
+        json.dumps(
+            {
+                "version": _VERSION,
+                "seconds": plan.seconds,
+                "windows": [[w.start_s, w.end_s] for w in plan.windows],
+            }
+        ),
+        encoding="utf-8",
+    )
+    partial.replace(path)
+
+
+def read_plan(path: Path) -> Plan | None:
+    """None when there is no complete plan of this version."""
+    try:
+        kept = json.loads(path.read_text(encoding="utf-8"))
+        if kept["version"] != _VERSION:
+            return None
+        return Plan(
+            float(kept["seconds"]),
+            [Window(float(start), float(end)) for start, end in kept["windows"]],
+        )
+    except (OSError, ValueError, TypeError, KeyError):
+        return None

surfsense_local/backend/worker/ingestion/audio/queue_turn.py

@@ -0,0 +1,35 @@
+"""A recording's turn on the one-thread ingest queue. An hour at 11.5x real time
+holds it for some 5 minutes, so once a minute has passed and another source
+waits, the job gives the queue up and carries on later from its checkpoint: a
+PDF added midway starts within about a minute."""
+
+import time
+
+from sqlalchemy.orm import Session
+
+from modules.documents.models import Document
+from worker.ingestion.audio.waiting_work import other_sources_pending
+from worker.jobs import JobYieldedError, raise_if_cancelled
+
+__all__ = ["QueueTurn"]
+
+YIELD_AFTER_SECONDS = 60.0
+
+
+class QueueTurn:
+    def __init__(self, session: Session, document: Document) -> None:
+        self._session = session
+        self._document = document
+        self._started = time.monotonic()
+
+    def check_cancelled(self) -> None:
+        """Raises JobCancelledError; holds no write lock past the check."""
+        raise_if_cancelled(self._session, self._document)
+
+    def give_way(self) -> None:
+        """Raises JobYieldedError once this run has had its minute and a source
+        that is not a recording waits."""
+        if time.monotonic() - self._started < YIELD_AFTER_SECONDS:
+            return
+        if other_sources_pending(self._session):
+            raise JobYieldedError

surfsense_local/backend/worker/ingestion/audio/readiness.py

@@ -0,0 +1,42 @@
+"""Before anything is decoded: is a speech model chosen, and is its server up?
+In the seconds after a model is chosen, or after a crash, it is not yet."""
+
+import time
+
+from worker.ingestion.audio.failures import (
+    ENGINE_STOPPED,
+    AwaitingSpeechModelError,
+    RecordingFailedError,
+)
+from worker.ingestion.audio.queue_turn import QueueTurn
+from worker.ingestion.audio.transcription_api import (
+    SpeechStatus,
+    TranscriptionApi,
+    UnavailableError,
+)
+
+__all__ = ["speech_recognition_ready"]
+
+POLL_SECONDS = 5.0
+GIVE_UP_SECONDS = 120.0
+
+
+def speech_recognition_ready(api: TranscriptionApi, turn: QueueTurn) -> SpeechStatus:
+    """Raises AwaitingSpeechModelError with none chosen, and a failure Huey
+    retries when the server stays down for two minutes. Gives the queue up
+    while it waits, as a window does."""
+    deadline = time.monotonic() + GIVE_UP_SECONDS
+    while True:
+        try:
+            status = api.status()
+        except UnavailableError:
+            status = None
+        if status is not None and status.ready:
+            return status
+        if status is not None and status.reason == "needs_speech_model":
+            raise AwaitingSpeechModelError
+        if time.monotonic() >= deadline:
+            raise RecordingFailedError(ENGINE_STOPPED)
+        turn.check_cancelled()
+        turn.give_way()
+        time.sleep(POLL_SECONDS)

surfsense_local/backend/worker/ingestion/audio/segments_file.py

@@ -0,0 +1,45 @@
+"""transcript/segments.json: where each paragraph of a finished transcript sits
+in the recording, so a citation of a recording can seek to it later."""
+
+import json
+from collections.abc import Sequence
+from pathlib import Path
+
+from worker.ingestion.audio.transcript_text import HeardWindow, paragraph_lines
+
+__all__ = ["write_segments"]
+
+
+def write_segments(
+    path: Path,
+    *,
+    model_id: str,
+    language: str | None,
+    seconds: float,
+    windows: Sequence[HeardWindow],
+) -> None:
+    """Raises OSError when the disk is full."""
+    segments = [
+        {
+            "line": line,
+            "start_s": window.start_s,
+            "end_s": window.end_s,
+            "language": window.language,
+        }
+        for window, line in zip(windows, paragraph_lines(windows), strict=True)
+        if line is not None
+    ]
+    partial = path.with_name(f"{path.name}.partial")
+    partial.write_text(
+        json.dumps(
+            {
+                "model_id": model_id,
+                "language": language,
+                "duration_s": seconds,
+                "windows": segments,
+            },
+            ensure_ascii=False,
+        ),
+        encoding="utf-8",
+    )
+    partial.replace(path)

surfsense_local/backend/worker/ingestion/audio/speech_regions.py

@@ -0,0 +1,89 @@
+"""Where a recording holds speech: Silero over the decoded audio, read off disk a
+minute at a time, each span padded by 200 ms and joined across gaps under
+300 ms. Each frame's loudness is kept too, for cutting a run of speech that
+outlasts a window."""
+
+import logging
+from array import array
+from collections.abc import Callable, Sequence
+from dataclasses import dataclass
+from pathlib import Path
+
+import numpy as np
+
+from modules.llm.transcription.voice_activity.model_file import silero_vad_file
+from modules.llm.transcription.voice_activity.silero import (
+    FRAME_BYTES,
+    FRAME_SECONDS,
+    SileroReader,
+)
+from modules.llm.transcription.voice_activity.speech_spans import (
+    SpeechSpan,
+    spans_in,
+)
+
+__all__ = ["HeardRecording", "hear_recording", "padded"]
+
+logger = logging.getLogger(__name__)
+
+_PAD_S = 0.2
+_JOIN_UNDER_S = 0.3
+_FRAMES_PER_READ = 1875  # a minute
+_BYTES_PER_SECOND = 32000
+
+
+@dataclass(frozen=True)
+class HeardRecording:
+    regions: list[SpeechSpan]
+    # Each frame's loudness: the RMS of its samples.
+    levels: Sequence[float]
+    frame_s: float
+    seconds: float
+
+
+def hear_recording(audio: Path, *, between_reads: Callable[[], None]) -> HeardRecording:
+    """`audio` is 16 kHz mono s16. Without Silero, or should it fail, all of it
+    counts as speech, as a phrase does in the API: a recording is never lost
+    for want of the detector. `between_reads` is the job's cancel check."""
+    model = silero_vad_file()
+    reader = SileroReader(model) if model.is_file() else None
+    if reader is None:
+        logger.warning("transcription: no Silero VAD at %s; all is speech", model)
+    probabilities: array = array("f")
+    levels: array = array("f")
+    size = 0
+    with audio.open("rb") as recording:
+        while piece := recording.read(FRAME_BYTES * _FRAMES_PER_READ):
+            size += len(piece)
+            levels.extend(_loudness(piece))
+            if reader is not None:
+                try:
+                    probabilities.extend(reader.read(piece))
+                except Exception:
+                    logger.warning("transcription: Silero failed", exc_info=True)
+                    reader = None
+            between_reads()
+    seconds = size / _BYTES_PER_SECOND
+    spans = [SpeechSpan(0.0, seconds)] if reader is None else spans_in(probabilities)
+    return HeardRecording(padded(spans, seconds), levels, FRAME_SECONDS, seconds)
+
+
+def padded(spans: Sequence[SpeechSpan], seconds: float) -> list[SpeechSpan]:
+    """Each span widened by 200 ms within the recording, then joined to the
+    one before when less than 300 ms apart."""
+    regions: list[SpeechSpan] = []
+    for span in spans:
+        start = max(0.0, span.start_s - _PAD_S)
+        end = min(seconds, span.end_s + _PAD_S)
+        if regions and start - regions[-1].end_s < _JOIN_UNDER_S:
+            regions[-1] = SpeechSpan(regions[-1].start_s, max(end, regions[-1].end_s))
+        else:
+            regions.append(SpeechSpan(start, end))
+    return regions
+
+
+def _loudness(piece: bytes) -> np.ndarray:
+    samples = np.frombuffer(piece, dtype="<i2").astype(np.float32) / 32768
+    frame = FRAME_BYTES // 2
+    samples = np.pad(samples, (0, -len(samples) % frame)).reshape(-1, frame)
+    return np.sqrt(np.mean(samples * samples, axis=1))

surfsense_local/backend/worker/ingestion/audio/transcript_job.py

@@ -0,0 +1,211 @@
+"""A recording's transcript, as ingest's parse step: is speech recognition ready,
+the file decoded in a child, where it holds speech, the windows that speech is
+cut into, each window written down through the API, and the transcript.
+
+Everything it keeps lives under the document's transcript/ folder; work/ holds
+what a yield, a retry or a restart resumes from, and goes once the transcript
+is done.
+"""
+
+import errno
+import logging
+import shutil
+from collections.abc import Iterator
+from contextlib import contextmanager
+from pathlib import Path
+
+from sqlalchemy.orm import Session, object_session
+
+from modules.documents.models import Document
+from modules.documents.recordings.layout import WorkFiles, segments_file, work_files
+from modules.documents.recordings.metadata import AWAITING, KEY, is_awaiting
+from modules.llm.model_route.api_address import api_url
+from shared.config import get_storage_settings
+from worker.audio_decode.room import needed_bytes
+from worker.audio_decode.run import decode_in_child
+from worker.ingestion.audio.checkpoint import DoneWindow, resumable_windows
+from worker.ingestion.audio.failures import (
+    NO_AUDIO,
+    NO_SPEECH,
+    UNDECODABLE,
+    AwaitingSpeechModelError,
+    RecordingRefusedError,
+    no_room,
+)
+from worker.ingestion.audio.model_name import model_name
+from worker.ingestion.audio.plan_file import Plan, read_plan, write_plan
+from worker.ingestion.audio.queue_turn import QueueTurn
+from worker.ingestion.audio.readiness import speech_recognition_ready
+from worker.ingestion.audio.segments_file import write_segments
+from worker.ingestion.audio.speech_regions import hear_recording
+from worker.ingestion.audio.transcript_text import (
+    HeardWindow,
+    header_language,
+    majority_language,
+    transcript_markdown,
+)
+from worker.ingestion.audio.transcription_api import SpeechStatus, TranscriptionApi
+from worker.ingestion.audio.window_loop import (
+    ModelChangedError,
+    RecordingWindows,
+    transcribe_windows,
+)
+from worker.ingestion.audio.windows import plan_windows
+from worker.jobs import JobYieldedError, merge_metadata
+
+__all__ = ["recording_markdown"]
+
+logger = logging.getLogger(__name__)
+
+
+def recording_markdown(document: Document, original: Path) -> str:
+    """Raises JobYieldedError to give the queue up, AwaitingSpeechModelError
+    with no speech model, RecordingRefusedError for a recording that would fail
+    the same way again, and RecordingFailedError for one worth another try."""
+    # The ingest job's own session: cancel is checked, and metadata kept, on it.
+    session = object_session(document)
+    assert session is not None, "a recording is transcribed inside its ingest job"
+    folder = get_storage_settings().document_dir(document.workspace_id, document.id)
+    files = work_files(folder)
+    turn = QueueTurn(session, document)
+    if is_awaiting(document.document_metadata):
+        merge_metadata(session, document, KEY, None)
+    try:
+        api = TranscriptionApi(api_url())
+        status = speech_recognition_ready(api, turn)
+        files.folder.mkdir(parents=True, exist_ok=True)
+        return _transcribe(session, document, original, api, turn, folder, status)
+    except JobYieldedError:
+        raise
+    except AwaitingSpeechModelError:
+        merge_metadata(session, document, KEY, {AWAITING: True})
+        _end_of_run(files)
+        raise
+    except BaseException:
+        _end_of_run(files)
+        raise
+
+
+def _transcribe(
+    session: Session,
+    document: Document,
+    original: Path,
+    api: TranscriptionApi,
+    turn: QueueTurn,
+    folder: Path,
+    status: SpeechStatus,
+) -> str:
+    assert status.model is not None, "a ready status names its model"
+    files = work_files(folder)
+    model_id, intent = status.model, status.language_intent
+    while True:
+        plan, done = _planned(original, files, turn, model_id, intent)
+        with _full_disk_refused(plan.seconds):
+            try:
+                done = transcribe_windows(
+                    RecordingWindows(
+                        session, document, api, turn, files, plan, model_id, intent
+                    ),
+                    done,
+                )
+            except ModelChangedError as changed:
+                files.windows.unlink(missing_ok=True)
+                model_id = changed.model_id
+                continue
+            return _finished(session, document, folder, plan, done, model_id, status)
+
+
+def _planned(
+    original: Path, files: WorkFiles, turn: QueueTurn, model_id: str, intent: str
+) -> tuple[Plan, dict[int, DoneWindow]]:
+    """The windows, and those done already by this model for this language.
+    The audio is decoded again only while windows remain to send."""
+    plan = read_plan(files.plan)
+    if plan is None:
+        # A checkpoint of some other cut of the recording.
+        files.windows.unlink(missing_ok=True)
+        files.progress.unlink(missing_ok=True)
+    done = resumable_windows(files.windows, model_id, intent)
+    if plan is not None and len(done) == len(plan.windows):
+        return plan, done
+    if not files.audio.is_file():
+        _decode(original, files.audio, turn)
+    if plan is None:
+        heard = hear_recording(files.audio, between_reads=turn.check_cancelled)
+        if not heard.regions:
+            raise RecordingRefusedError(NO_SPEECH)
+        plan = Plan(
+            heard.seconds, plan_windows(heard.regions, heard.levels, heard.frame_s)
+        )
+        with _full_disk_refused(plan.seconds):
+            write_plan(files.plan, plan)
+    return plan, done
+
+
+def _decode(original: Path, audio: Path, turn: QueueTurn) -> None:
+    outcome = decode_in_child(original, audio, between_polls=turn.check_cancelled)
+    if outcome.kind == "no_room":
+        raise no_room(outcome.needed_bytes)
+    if outcome.kind == "no_audio":
+        raise RecordingRefusedError(NO_AUDIO)
+    if outcome.kind == "undecodable":
+        logger.info("transcription: %s is undecodable: %s", original, outcome.detail)
+        raise RecordingRefusedError(UNDECODABLE)
+
+
+def _finished(
+    session: Session,
+    document: Document,
+    folder: Path,
+    plan: Plan,
+    done: dict[int, DoneWindow],
+    model_id: str,
+    status: SpeechStatus,
+) -> str:
+    pinned = status.pinned_language
+    heard = [
+        HeardWindow(w.start_s, w.end_s, w.text, w.language)
+        for _, w in sorted(done.items())
+    ]
+    if not any(window.text for window in heard):
+        raise RecordingRefusedError(NO_SPEECH)
+    language = pinned or majority_language(heard)
+    markdown = transcript_markdown(
+        title=document.title,
+        seconds=plan.seconds,
+        windows=heard,
+        language=header_language(heard, pinned, status.languages),
+        model=model_name(model_id),
+    )
+    write_segments(
+        segments_file(folder),
+        model_id=model_id,
+        language=language,
+        seconds=plan.seconds,
+        windows=heard,
+    )
+    merge_metadata(
+        session,
+        document,
+        KEY,
+        {"language": language, "duration_s": round(plan.seconds, 3), "model": model_id},
+    )
+    shutil.rmtree(work_files(folder).folder, ignore_errors=True)
+    return markdown
+
+
+def _end_of_run(files: WorkFiles) -> None:
+    """A run that ends the job keeps its windows and plan for the next one, not
+    the decoded audio, which decoding again costs seconds to make."""
+    files.audio.unlink(missing_ok=True)
+    files.progress.unlink(missing_ok=True)
+
+
+@contextmanager
+def _full_disk_refused(seconds: float) -> Iterator[None]:
+    try:
+        yield
+    except OSError as error:
+        if error.errno == errno.ENOSPC:
+            raise no_room(needed_bytes(seconds)) from error
+        raise

surfsense_local/backend/worker/ingestion/audio/transcript_text.py

@@ -0,0 +1,106 @@
+"""A recording's transcript as its content: one header line, then a paragraph
+per window starting with its time. The chunker splits at blank lines, so while
+a window's paragraph stays under a chunk, every chunk opens with when it was
+said, and the agent can quote times. CJK text has no `. ` to split at either."""
+
+from collections import Counter
+from collections.abc import Collection, Sequence
+from dataclasses import dataclass
+
+from worker.ingestion.audio.language_names import english_name
+
+__all__ = [
+    "HeardWindow",
+    "header_language",
+    "majority_language",
+    "paragraph_lines",
+    "transcript_markdown",
+]
+
+_GUIDE = "[hh:mm:ss] marks where each passage starts."
+_NOT_IDENTIFIED = "language not identified"
+_LOWER_ACCURACY = "lower accuracy with this model"
+
+
+@dataclass(frozen=True)
+class HeardWindow:
+    start_s: float
+    end_s: float
+    text: str
+    # A language code, or None when nothing said which.
+    language: str | None
+
+
+def transcript_markdown(
+    *,
+    title: str,
+    seconds: float,
+    windows: Sequence[HeardWindow],
+    language: str,
+    model: str,
+) -> str:
+    header = (
+        f"Transcript of {title} · {_length(seconds)} · {language} · {model}. {_GUIDE}"
+    )
+    paragraphs = [
+        f"[{_timestamp(w.start_s)}] {' '.join(w.text.split())}"
+        for w in windows
+        if w.text
+    ]
+    return "\n\n".join([header, *paragraphs]) + "\n"
+
+
+def paragraph_lines(windows: Sequence[HeardWindow]) -> list[int | None]:
+    """The line each window's paragraph is on, counted from 1 as chunks count
+    theirs; None for a window with nothing written."""
+    lines: list[int | None] = []
+    line = 3  # the header, then a blank line
+    for window in windows:
+        if window.text:
+            lines.append(line)
+            line += 2
+        else:
+            lines.append(None)
+    return lines
+
+
+def header_language(
+    windows: Sequence[HeardWindow], pinned: str | None, listed: Collection[str]
+) -> str:
+    """The language the user set, else the one most windows were heard in."""
+    if pinned is not None:
+        return english_name(pinned)
+    code = majority_language(windows)
+    if code is None:
+        return _NOT_IDENTIFIED
+    if code in listed:
+        return f"{english_name(code)} (detected)"
+    return f"{english_name(code)} (detected) · {_LOWER_ACCURACY}"
+
+
+def majority_language(windows: Sequence[HeardWindow]) -> str | None:
+    """Most windows with text, then most seconds of them."""
+    counts: Counter[str] = Counter()
+    seconds: Counter[str] = Counter()
+    for window in windows:
+        if window.text and window.language is not None:
+            counts[window.language] += 1
+            seconds[window.language] += window.end_s - window.start_s
+    if not counts:
+        return None
+    return max(counts, key=lambda code: (counts[code], seconds[code]))
+
+
+def _length(seconds: float) -> str:
+    hours, minutes, rest = _clock(seconds)
+    return f"{hours}:{minutes:02}:{rest:02}" if hours else f"{minutes}:{rest:02}"
+
+
+def _timestamp(seconds: float) -> str:
+    hours, minutes, rest = _clock(seconds)
+    return f"{hours:02}:{minutes:02}:{rest:02}"
+
+
+def _clock(seconds: float) -> tuple[int, int, int]:
+    whole = int(seconds)
+    return whole // 3600, whole // 60 % 60, whole % 60

surfsense_local/backend/worker/ingestion/audio/transcription_api.py

@@ -0,0 +1,126 @@
+"""The two API routes a recording's job calls: whether speech recognition can
+run, and one window at a time. Workers reach the transcription server only
+through the API, whose gate orders every request (ADR 0055)."""
+
+from dataclasses import dataclass
+
+import httpx
+
+__all__ = [
+    "NeedsSpeechModelError",
+    "SpeechStatus",
+    "TranscriptionApi",
+    "UnavailableError",
+    "WaitingError",
+    "WindowText",
+]
+
+# The route waits up to 30 s for its turn, then gives the server
+# max(120 s, 20 x the audio), as the API's client does.
+_TURN_WAIT_S = 30.0
+_SLACK_S = 30.0
+# The status route's word for "let the model detect it".
+_DETECT = "auto"
+
+
+@dataclass(frozen=True)
+class SpeechStatus:
+    ready: bool
+    # `needs_speech_model`, `transcription_unavailable`, or None when ready.
+    reason: str | None
+    model: str | None
+    # `auto`, or the code the user set.
+    language_intent: str
+    # The codes the model is listed for.
+    languages: tuple[str, ...]
+
+    @property
+    def pinned_language(self) -> str | None:
+        """The language the user set, or None when the model detects it."""
+        return None if self.language_intent == _DETECT else self.language_intent
+
+
+@dataclass(frozen=True)
+class WindowText:
+    text: str
+    language: str | None
+    model_id: str
+
+
+class NeedsSpeechModelError(Exception):
+    """None is chosen, or it was deleted since the job began."""
+
+
+class UnavailableError(Exception):
+    """The server or the API is restarting or stopped; asked again later."""
+
+
+class WaitingError(Exception):
+    """Not this window's turn yet: someone dictates, the text runtime is held,
+    memory is short, or another window is in flight."""
+
+    def __init__(self, reason: str, message: str) -> None:
+        super().__init__(message)
+        self.reason = reason
+
+
+class TranscriptionApi:
+    def __init__(
+        self, base_url: str, *, transport: httpx.BaseTransport | None = None
+    ) -> None:
+        self._base_url = base_url.rstrip("/")
+        self._transport = transport
+
+    def status(self) -> SpeechStatus:
+        reply = self._request("GET", "/llm/transcription/status", timeout=10.0)
+        reply.raise_for_status()
+        body = reply.json()
+        return SpeechStatus(
+            ready=body["ready"],
+            reason=body["reason"],
+            model=body["model"],
+            language_intent=body["language_intent"],
+            languages=tuple(body["languages"]),
+        )
+
+    def transcribe(self, wav: bytes, *, seconds: float, last: bool) -> WindowText:
+        """`last`: no other recording waits, so the model can go once done."""
+        reply = self._request(
+            "POST",
+            "/internal/transcription/windows",
+            timeout=_TURN_WAIT_S + max(120.0, 20 * seconds) + _SLACK_S,
+            files={"audio": ("window.wav", wav, "audio/wav")},
+            data={"last": "true" if last else "false"},
+        )
+        if reply.status_code in (409, 503):
+            _raise_refusal(reply)
+        reply.raise_for_status()
+        body = reply.json()
+        return WindowText(body["text"], body["language"], body["model_id"])
+
+    def _request(
+        self, method: str, path: str, *, timeout: float, **content: object
+    ) -> httpx.Response:
+        try:
+            with httpx.Client(
+                timeout=httpx.Timeout(timeout, connect=5.0), transport=self._transport
+            ) as client:
+                return client.request(method, f"{self._base_url}{path}", **content)
+        except httpx.TransportError as error:
+            raise UnavailableError(str(error)) from error
+
+
+def _raise_refusal(reply: httpx.Response) -> None:
+    try:
+        detail = reply.json()["detail"]
+        code = detail["code"]
+    except (ValueError, KeyError, TypeError):
+        reply.raise_for_status()
+        return
+    if code == "needs_speech_model":
+        raise NeedsSpeechModelError(detail.get("message", code))
+    if code == "waiting":
+        raise WaitingError(detail.get("reason", "busy"), detail.get("message", code))
+    if code == "transcription_unavailable":
+        raise UnavailableError(detail.get("message", code))
+    reply.raise_for_status()

surfsense_local/backend/worker/ingestion/audio/waiting_work.py

@@ -0,0 +1,48 @@
+"""What else waits on the ingest queue, as a recording's job decides whether to
+give the queue up and whether its last window is the server's last."""
+
+from sqlalchemy import exists, func, or_, select
+from sqlalchemy.orm import Session
+
+from modules.documents.models import Document, DocumentStatus, DocumentType
+from modules.documents.recordings.kind import RECORDING_MIME_PREFIX
+
+__all__ = ["another_recording_queued", "other_sources_pending"]
+
+_MIME = func.coalesce(Document.document_metadata["mime_type"].as_string(), "")
+_IS_RECORDING = (Document.document_type == DocumentType.FILE) & _MIME.startswith(
+    RECORDING_MIME_PREFIX
+)
+
+
+def other_sources_pending(session: Session) -> bool:
+    """A file or note that is not a recording waits: recordings never give the
+    queue up to each other."""
+    waiting = session.scalar(
+        select(
+            exists().where(
+                Document.status == DocumentStatus.PENDING,
+                Document.document_type.in_([DocumentType.FILE, DocumentType.NOTE]),
+                or_(Document.document_type == DocumentType.NOTE, ~_IS_RECORDING),
+            )
+        )
+    )
+    session.commit()  # the read took the write lock
+    return bool(waiting)
+
+
+def another_recording_queued(session: Session, document_id: int) -> bool:
+    """Another recording will want the speech model once this one is done."""
+    queued = session.scalar(
+        select(
+            exists().where(
+                Document.id != document_id,
+                Document.status.in_(
+                    [DocumentStatus.PENDING, DocumentStatus.PROCESSING]
+                ),
+                _IS_RECORDING,
+            )
+        )
+    )
+    session.commit()
+    return bool(queued)

surfsense_local/backend/worker/ingestion/audio/window_loop.py

@@ -0,0 +1,169 @@
+"""Every window the checkpoint lacks, sent to the API in turn: checked for cancel
+first, appended to the checkpoint once written down, its progress kept and
+announced, and the queue given up between windows when other sources wait."""
+
+import time
+from collections.abc import Iterator
+from dataclasses import dataclass, replace
+from pathlib import Path
+
+from sqlalchemy.orm import Session
+
+from modules.documents.models import Document
+from modules.documents.recordings.layout import WorkFiles
+from modules.documents.recordings.progress_file import (
+    TranscriptionProgress,
+    read_progress,
+    write_progress,
+)
+from modules.events.schemas import Progress
+from modules.llm.transcription.audio_checks import SAMPLE_RATE, wav_of
+from worker.ingestion.audio.checkpoint import DoneWindow, append_window
+from worker.ingestion.audio.failures import (
+    ENGINE_STOPPED,
+    AwaitingSpeechModelError,
+    RecordingFailedError,
+    RecordingRefusedError,
+)
+from worker.ingestion.audio.plan_file import Plan
+from worker.ingestion.audio.queue_turn import QueueTurn
+from worker.ingestion.audio.transcription_api import (
+    NeedsSpeechModelError,
+    TranscriptionApi,
+    UnavailableError,
+    WaitingError,
+    WindowText,
+)
+from worker.ingestion.audio.waiting_work import another_recording_queued
+from worker.notify import notify_document_updates
+
+__all__ = ["ModelChangedError", "RecordingWindows", "transcribe_windows"]
+
+# A server restart takes seconds; one that is down past these is a failure.
+RETRY_DELAYS = (1, 2, 4, 8, 16)
+# Memory waits are summed across yields; past this the recording fails.
+MEMORY_WAIT_LIMIT_S = 600.0
+
+
+class ModelChangedError(Exception):
+    """Another speech model was chosen while this recording was transcribed:
+    it starts over, so one model writes the whole transcript."""
+
+    def __init__(self, model_id: str) -> None:
+        super().__init__(model_id)
+        self.model_id = model_id
+
+
+@dataclass(frozen=True)
+class RecordingWindows:
+    session: Session
+    document: Document
+    api: TranscriptionApi
+    turn: QueueTurn
+    files: WorkFiles
+    plan: Plan
+    model_id: str
+    intent: str
+
+
+def transcribe_windows(
+    job: RecordingWindows, done: dict[int, DoneWindow]
+) -> dict[int, DoneWindow]:
+    """Raises ModelChangedError, AwaitingSpeechModelError, a refusal once memory
+    has been short for 10 minutes in all, and JobYieldedError or
+    JobCancelledError from the turn."""
+    final = len(job.plan.windows) - 1
+    progress = _progress_so_far(job, done)
+    for n, window in enumerate(job.plan.windows):
+        if n in done:
+            continue
+        job.turn.check_cancelled()
+        last = n == final and not another_recording_queued(job.session, job.document.id)
+        pcm = _samples(job.files.audio, window.start_s, window.end_s)
+        written, progress = _transcribed(job, pcm, last, progress)
+        if written.model_id != job.model_id:
+            raise ModelChangedError(written.model_id)
+        done[n] = DoneWindow(
+            n,
+            window.start_s,
+            window.end_s,
+            written.text,
+            written.language,
+            job.model_id,
+            job.intent,
+        )
+        append_window(job.files.windows, done[n])
+        progress = replace(progress, done_s=window.end_s)
+        write_progress(job.files.progress, progress)
+        notify_document_updates(
+            job.document, Progress(done_s=progress.done_s, total_s=progress.total_s)
+        )
+        if n != final:
+            job.turn.give_way()
+    return done
+
+
+def _transcribed(
+    job: RecordingWindows,
+    pcm: bytes,
+    last: bool,
+    progress: TranscriptionProgress,
+) -> tuple[WindowText, TranscriptionProgress]:
+    seconds = len(pcm) / 2 / SAMPLE_RATE
+    retries = iter(RETRY_DELAYS)
+    while True:
+        asked = time.monotonic()
+        try:
+            return job.api.transcribe(wav_of(pcm), seconds=seconds, last=last), progress
+        except WaitingError as waiting:
+            if waiting.reason == "memory":
+                progress = _waited_for_memory(job, progress, asked, waiting)
+        except UnavailableError:
+            _pause_or_stop(retries)
+        except NeedsSpeechModelError as error:
+            raise AwaitingSpeechModelError from error
+        job.turn.check_cancelled()
+        job.turn.give_way()
+
+
+def _waited_for_memory(
+    job: RecordingWindows,
+    progress: TranscriptionProgress,
+    asked: float,
+    waiting: WaitingError,
+) -> TranscriptionProgress:
+    waited = progress.memory_wait_s + time.monotonic() - asked
+    progress = replace(progress, memory_wait_s=waited)
+    write_progress(job.files.progress, progress)
+    if waited > MEMORY_WAIT_LIMIT_S:
+        # The API's sentence: "Transcribing needs about 2.9 GB free; ...".
+        raise RecordingRefusedError(str(waiting))
+    return progress
+
+
+def _pause_or_stop(retries: Iterator[int]) -> None:
+    delay = next(retries, None)
+    if delay is None:
+        raise RecordingFailedError(ENGINE_STOPPED)
+    time.sleep(delay)
+
+
+def _progress_so_far(
+    job: RecordingWindows, done: dict[int, DoneWindow]
+) -> TranscriptionProgress:
+    """From the checkpoint, and the memory waits of the runs this one carries
+    on from after a yield."""
+    kept = read_progress(job.files.progress)
+    return TranscriptionProgress(
+        max((window.end_s for window in done.values()), default=0.0),
+        job.plan.seconds,
+        0.0 if kept is None else kept.memory_wait_s,
+    )
+
+
+def _samples(audio: Path, start_s: float, end_s: float) -> bytes:
+    first = round(start_s * SAMPLE_RATE)
+    count = round(end_s * SAMPLE_RATE) - first
+    with audio.open("rb") as recording:
+        recording.seek(first * 2)
+        return recording.read(count * 2)

surfsense_local/backend/worker/ingestion/audio/windows.py

@@ -0,0 +1,86 @@
+"""Speech packed into the windows the recogniser is sent. 28 s bounds a request's
+memory: Nemotron's offline mode runs one encoder graph over the whole request,
+and over 28 s windows it peaked at 1,370 MiB. Windows aim at 24 s and end at
+the longest pause; silence between them is never sent, and each keeps its
+place in the recording, so timestamps survive."""
+
+from collections.abc import Sequence
+from dataclasses import dataclass
+
+from modules.llm.transcription.voice_activity.speech_spans import SpeechSpan
+
+__all__ = ["MAX_WINDOW_S", "Window", "plan_windows"]
+
+MAX_WINDOW_S = 28.0
+_AIM_S = 24.0
+# A window may end at a pause once it holds this much.
+_SHORTEST_CUT_S = 20.0
+_QUIET_S = 0.1
+
+
+@dataclass(frozen=True)
+class Window:
+    start_s: float
+    end_s: float
+
+
+def plan_windows(
+    regions: Sequence[SpeechSpan], levels: Sequence[float], frame_s: float
+) -> list[Window]:
+    """`regions` in order and apart; `levels` each frame's loudness, read when
+    one run of speech outlasts a window and must be cut inside."""
+    queue = list(regions)
+    windows: list[Window] = []
+    at = 0
+    while at < len(queue):
+        start = queue[at].start_s
+        if queue[at].end_s - start > MAX_WINDOW_S:
+            cut = _quietest(levels, frame_s, start + _SHORTEST_CUT_S, start)
+            windows.append(Window(start, cut))
+            queue[at] = SpeechSpan(cut, queue[at].end_s)
+            continue
+        last = at
+        while last + 1 < len(queue) and queue[last + 1].end_s - start <= MAX_WINDOW_S:
+            last += 1
+        if last + 1 == len(queue):
+            windows.append(Window(start, queue[last].end_s))
+            break
+        end = _end_at_a_pause(queue, at, last, start)
+        windows.append(Window(start, queue[end].end_s))
+        at = end + 1
+    return windows
+
+
+def _end_at_a_pause(
+    queue: Sequence[SpeechSpan], first: int, last: int, start: float
+) -> int:
+    """The region to end after: the longest pause once 20 s are held, the one
+    nearest 24 s among equals; else every region that fits."""
+    long_enough = [
+        n for n in range(first, last + 1) if queue[n].end_s - start >= _SHORTEST_CUT_S
+    ]
+    if not long_enough:
+        return last
+    return max(
+        long_enough,
+        key=lambda n: (
+            queue[n + 1].start_s - queue[n].end_s,
+            -abs(queue[n].end_s - start - _AIM_S),
+        ),
+    )
+
+
+def _quietest(
+    levels: Sequence[float], frame_s: float, earliest: float, start: float
+) -> float:
+    """The middle of the quietest 100 ms between `earliest` and the window's
+    end, in time; the aim when the levels do not reach that far."""
+    span = max(1, round(_QUIET_S / frame_s))
+    first = int(earliest / frame_s) + 1
+    last = int((start + MAX_WINDOW_S) / frame_s) - span
+    if last >= len(levels) - span:
+        last = len(levels) - span
+    if last < first:
+        return start + _AIM_S
+    quietest = min(range(first, last + 1), key=lambda f: sum(levels[f : f + span]))
+    return (quietest + span / 2) * frame_s

surfsense_local/backend/worker/ingestion/parsing.py

@@ -7,7 +7,9 @@ import psutil
 
 from modules.documents.models import Document, DocumentType
 from modules.documents.original_file import original_path
+from modules.documents.recordings.kind import is_recording
 from shared.config import get_storage_settings
+from worker.ingestion.audio.transcript_job import recording_markdown
 from worker.ingestion.figures.store import keep_figures
 from worker.ingestion.image_page import IMAGE_SUFFIXES, as_page
 from worker.ingestion.parser_pack import missing_parser_folders, parser_dir
@@ -30,6 +32,9 @@ def markdown_for(document: Document) -> str:
     if path is None:
         raise FileNotFoundError("the uploaded file is no longer on disk")
 
+    # Before Docling, which must never read a recording.
+    if is_recording(document.mime_type):
+        return recording_markdown(document, path)
     return _markdown_from(path)
 
 

surfsense_local/backend/worker/ingestion/pipeline.py

@@ -3,11 +3,20 @@ import logging
 from sqlalchemy.orm import Session
 
 from modules.documents.models import Document, DocumentStatus, DocumentType
+from modules.documents.tasks import PRIORITY_BULK, ingest_document
 from modules.embedding.active import require_active_index
 from shared.config import get_storage_settings
 from shared.db import create_db_engine, create_session_factory
 from worker.ingestion import chunking, indexing, parsing
-from worker.jobs import JobCancelledError, begin_job, finish_job, raise_if_cancelled
+from worker.ingestion.audio.failures import RecordingFailedError
+from worker.jobs import (
+    JobCancelledError,
+    JobYieldedError,
+    begin_job,
+    finish_job,
+    pause_job,
+    raise_if_cancelled,
+)
 from worker.notify import notify_document_updates
 
 logger = logging.getLogger(__name__)
@@ -57,6 +66,12 @@ def _ingest(session: Session, document: Document) -> None:
         notify_document_updates(document)
     except JobCancelledError:
         session.rollback()
+    except JobYieldedError:
+        # A recording gives the queue up and finishes later from its checkpoint.
+        session.rollback()
+        if pause_job(session, document):
+            ingest_document(document.id, priority=PRIORITY_BULK)
+            notify_document_updates(document)
     except Exception as failure:
         session.rollback()
         document = session.get(Document, document.id)
@@ -66,8 +81,15 @@ def _ingest(session: Session, document: Document) -> None:
             session,
             document,
             DocumentStatus.FAILED,
-            f"{type(failure).__name__}: {failure}"[:MESSAGE_CHARS],
+            _reason(failure)[:MESSAGE_CHARS],
         ):
             return
         notify_document_updates(document)
         raise  # Huey retries; a later success clears the message.
+
+
+def _reason(failure: Exception) -> str:
+    # A recording's failures are sentences written for the person.
+    if isinstance(failure, RecordingFailedError):
+        return str(failure)
+    return f"{type(failure).__name__}: {failure}"

surfsense_local/backend/worker/jobs.py

@@ -1,3 +1,5 @@
+from typing import Any
+
 from sqlalchemy import update
 from sqlalchemy.orm import Session
 
@@ -11,6 +13,11 @@ class JobCancelledError(Exception):
     """The user stopped this job; the worker must exit without a Huey retry."""
 
 
+class JobYieldedError(Exception):
+    """The job gave the queue to other work and will finish later, from its
+    checkpoint: not a failure, and no Huey retry."""
+
+
 def cancel_ingest_job(session: Session, document: Document) -> bool:
     """Revoke the ingest task. False if nothing was running."""
     if not _mark_cancelled(session, document):
@@ -54,6 +61,49 @@ def begin_job(session: Session, document: Document) -> bool:
     return True
 
 
+def pause_job(session: Session, document: Document) -> bool:
+    """Back to pending, so the job can queue again behind other work. False
+    when a cancel won; the startup sweep leaves a pending row alone."""
+    result = session.execute(
+        update(Document)
+        .where(
+            Document.id == document.id,
+            Document.status == DocumentStatus.PROCESSING,
+        )
+        .values(status=DocumentStatus.PENDING)
+    )
+    session.commit()
+    if result.rowcount == 0:
+        return False
+    session.refresh(document)
+    session.commit()
+    return True
+
+
+def merge_metadata(
+    session: Session, document: Document, key: str, value: Any | None
+) -> None:
+    """Set one key of the row's metadata, or remove it for None. Read and
+    written in one transaction, so every other key stays as last written."""
+    session.commit()
+    session.refresh(document, attribute_names=["document_metadata"])
+    metadata = {
+        name: kept
+        for name, kept in (document.document_metadata or {}).items()
+        if name != key
+    }
+    if value is not None:
+        metadata[key] = value
+    session.execute(
+        update(Document)
+        .where(Document.id == document.id)
+        .values(document_metadata=metadata)
+    )
+    session.commit()
+    session.refresh(document, attribute_names=["document_metadata"])
+    session.commit()
+
+
 def raise_if_cancelled(session: Session, document: Document) -> None:
     session.commit()
     session.refresh(document, attribute_names=["status"])

surfsense_local/backend/worker/notify.py

@@ -5,14 +5,23 @@ import httpx
 
 from modules.artifacts.models import Artifact
 from modules.documents.models import Document
-from modules.events.schemas import EventKind
+from modules.events.schemas import EventKind, Progress
 
 logger = logging.getLogger(__name__)
 
 
-def notify_document_updates(document: Document) -> None:
-    """Tell the API a document's row changed, so it can push the change to the UI."""
-    _notify(document.workspace_id, EventKind.DOCUMENTS, document.id, document.status.value)
+def notify_document_updates(
+    document: Document, progress: Progress | None = None
+) -> None:
+    """Tell the API a document's row changed, so it can push the change to the UI.
+    `progress`: a recording's, once per transcribed window."""
+    _notify(
+        document.workspace_id,
+        EventKind.DOCUMENTS,
+        document.id,
+        document.status.value,
+        progress,
+    )
 
 
 def notify_artifact_updates(artifact: Artifact) -> None:
@@ -25,7 +34,13 @@ def notify_artifact_updates(artifact: Artifact) -> None:
     )
 
 
-def _notify(workspace_id: int, kind: EventKind, row_id: int, status: str) -> None:
+def _notify(
+    workspace_id: int,
+    kind: EventKind,
+    row_id: int,
+    status: str,
+    progress: Progress | None = None,
+) -> None:
     """Fire the loopback change notice, best-effort.
 
     Worker and API share the database, so a dropped notice costs the client only
@@ -37,16 +52,21 @@ def _notify(workspace_id: int, kind: EventKind, row_id: int, status: str) -> Non
         return
 
     host = os.environ.get("SURFSENSE_LOCAL_HOST", "127.0.0.1")
+    notice: dict = {
+        "workspace_id": workspace_id,
+        "kind": kind.value,
+        "ids": [row_id],
+        "status": status,
+    }
+    if progress is not None:
+        notice["progress"] = progress.model_dump()
     try:
         httpx.post(
             f"http://{host}:{port}/internal/events",
-            json={
-                "workspace_id": workspace_id,
-                "kind": kind.value,
-                "ids": [row_id],
-                "status": status,
-            },
+            json=notice,
             timeout=2.0,
         )
     except Exception as error:
-        logger.warning("could not notify the API of %s %s: %s", kind.value, row_id, error)
+        logger.warning(
+            "could not notify the API of %s %s: %s", kind.value, row_id, error
+        )