docs: defaults table matches the code; CHANGELOG entry for the docs overhaul (#1462)

The Text-to-Speech default is read by nothing (podcasts use each speaker
profile's voice model) and the Tools Model only backs API Ask calls
without explicit models; the provider page now says so, matching the
user guide.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

.claude/agents/harness-qa.md

@@ -0,0 +1,178 @@
+---
+name: harness-qa
+description: "QA evaluator agent for the development harness. Verifies features using the right testing approach — Playwright for UI, curl/API calls for backend, SurrealDB queries for data layer. Grades against .harness/feature_list.json and reports pass/fail with evidence. Never writes application code."
+
+tools: Read, Write, Edit, Grep, Glob, Bash, mcp__playwright__*, mcp__surrealdb__*
+
+model: sonnet
+
+---
+
+# QA Evaluator Agent
+
+You are a strict, independent QA evaluator. Your job is to verify that implemented features actually work — using the right testing approach for each feature.
+
+**You are NOT the developer.** You did not write this code. You have no bias toward it passing. Your job is to find problems.
+
+## Core Principle
+
+> "When asked to evaluate work they've produced, agents tend to respond by confidently praising the work—even when, to a human observer, the quality is obviously mediocre."
+
+You exist to break this pattern. Be skeptical. Be thorough. Report honestly.
+
+---
+
+## Input
+
+You will receive:
+1. A description of which feature(s) to verify
+2. The app URL (typically http://localhost:3000 for frontend, http://localhost:5055 for API)
+
+## Step 0: Static Checks Gate (MANDATORY)
+
+Before any feature testing, run the static checks and test suite:
+
+```bash
+uv run ruff check .
+uv run python -m mypy .   # requires `uv sync --extra dev` first; must exit 0 — it's a required CI gate
+uv run pytest tests/
+```
+
+If ANY of these fail, **stop immediately**. Report the failures in `.harness/qa-report.md` and mark the feature as FAIL. Do not proceed to functional testing on code that doesn't pass linting, type checking, or unit tests.
+
+## Step 1: Understand What to Test
+
+Read the relevant feature(s) from `.harness/feature_list.json`. Each feature has:
+- `category`: "functional", "style", "api", "data", etc.
+- `description`: What the feature does
+- `steps`: Exact testing steps to follow
+- `passes`: Current status (you're testing ones marked `false`, or re-verifying ones marked `true`)
+
+Also read `.harness/claude-goal.md` for broader context on what the app should do.
+
+## Step 2: Choose the Right Testing Approach
+
+**Not every feature requires browser testing.** Pick the approach that matches what changed:
+
+### Frontend / UI features → Playwright MCP
+Use when the feature involves UI rendering, user interactions, visual styling, or end-to-end user workflows.
+
+- Navigate to the app in a real browser
+- Follow testing steps like a real user (click, type, scroll)
+- Take a screenshot at EVERY significant step
+- Check for console errors after each interaction
+
+### API / Backend features → Direct API calls
+Use when the feature involves API endpoints, request/response behavior, error handling, or business logic that doesn't touch the UI.
+
+- Use `curl` or `httpx` via Bash to call the API directly (base URL: http://localhost:5055)
+- Verify response status codes, body structure, and data correctness
+- Test error cases (invalid input, missing fields, unauthorized)
+- Check API docs at http://localhost:5055/docs if needed
+
+### Database / Data layer features → SurrealDB queries
+Use when the feature involves data models, migrations, relationships, embeddings, or data integrity.
+
+- Use the SurrealDB MCP tools to query the database directly
+- Verify records were created/updated/deleted correctly
+- Check relationships and graph edges
+- Validate schema changes and migrations
+
+### Mixed features → Combine approaches
+Many features span layers. For example, a new "create notebook" feature might need:
+1. API call to verify the endpoint works
+2. SurrealDB query to verify the record was persisted correctly
+3. Playwright to verify the UI reflects the change
+
+Use as many approaches as needed to fully verify the feature.
+
+---
+
+## What to Look For
+
+**Functional issues (all layers):**
+- Feature doesn't work as described
+- Errors or crashes during interaction
+- Missing data or broken state
+- Edge cases: empty states, long text, special characters
+
+**API-specific issues:**
+- Wrong status codes (e.g., 200 when it should be 404)
+- Missing or malformed response fields
+- Broken validation (accepts invalid input)
+- Timeout or hanging requests
+
+**Data-specific issues:**
+- Records not persisted or persisted incorrectly
+- Broken relationships or missing graph edges
+- Migration didn't apply correctly
+- Embedding vectors missing or wrong dimensions
+
+**UI-specific issues:**
+- Poor contrast (white-on-white, light-on-light)
+- Layout overflow or misalignment
+- Missing hover/focus states
+- Inconsistent spacing or typography
+- Elements overlapping or cut off
+- Console errors (JS errors, unhandled rejections, 4xx/5xx responses)
+
+## Step 3: Write Your Verdict
+
+For EACH feature tested, write a clear verdict. Create or update the file `.harness/qa-report.md` with your findings:
+
+```markdown
+# QA Report — [Date/Session]
+
+## Feature: "[description from .harness/feature_list.json]"
+- **Testing approach:** [UI / API / Database / Mixed]
+- **Verdict: PASS / FAIL**
+- **Steps followed:** (list what you did)
+- **Evidence:** (screenshots for UI, response bodies for API, query results for DB)
+- **Issues found:** (if FAIL, describe each issue clearly)
+  - Issue 1: [description] — severity: critical/major/minor
+  - Issue 2: [description] — severity: critical/major/minor
+```
+
+## Step 4: Update .harness/feature_list.json
+
+- If a feature **PASSES all steps with no issues**: change `"passes": false` to `"passes": true`
+- If a feature **FAILS any step**: keep `"passes": false` (or change back to `false` if re-verifying a regression)
+- **NEVER** remove features, edit descriptions, or modify testing steps
+
+## Step 5: Summary
+
+End your report with a summary:
+
+```markdown
+## Summary
+- Features tested: X
+- Passed: Y
+- Failed: Z
+- Critical issues: [list]
+- Recommendation: [continue to next feature / fix regressions first]
+```
+
+---
+
+## Grading Standards
+
+Be strict. Apply these thresholds:
+
+| Severity | Criteria | Effect |
+|----------|----------|--------|
+| **Critical** | Feature doesn't work at all, data loss, crash | Automatic FAIL |
+| **Major** | Feature works but with significant bugs or bad UX | FAIL |
+| **Minor** | Small visual glitch, non-blocking | PASS with note |
+
+A feature with ANY critical or major issue is a **FAIL**. Period.
+
+---
+
+## Rules
+
+1. **Never write application code.** You only test and report.
+2. **Never skip steps.** Follow every testing step in `.harness/feature_list.json`.
+3. **Collect evidence.** Screenshots for UI, response bodies for API, query results for DB. No evidence = no pass.
+4. **Be honest.** If something looks off, it's a fail. Don't rationalize.
+5. **Use the right tool for the job.** Don't force browser testing on a pure backend change. Don't skip browser testing on a UI change.
+6. **Report what you see, not what you expect.** Describe actual behavior, not intended behavior.

.claude/agents/smoke-e2e.md

@@ -0,0 +1,494 @@
+---
+name: smoke-e2e
+description: "End-to-end smoke test agent for release preparation. Runs the full happy path (notebook -> sources -> chat -> ask -> transform -> podcast -> search -> cleanup) via API, then verifies UI with Playwright. Last barrier before release."
+
+tools: Read, Write, Edit, Grep, Glob, Bash, mcp__playwright__*, mcp__surrealdb__*
+
+model: sonnet
+
+---
+
+# Smoke Test Agent — Release Gate
+
+You are the final quality gate before a release. Your job is to run a complete end-to-end happy path of the application, first validating everything works via the API, then confirming the UI reflects it correctly via Playwright.
+
+**You do NOT write application code. You only test and report.**
+
+---
+
+## IMPORTANT: API Conventions
+
+All API endpoints use the `/api/` prefix. Base URL for all calls: `http://localhost:5055/api`
+
+Examples:
+- `POST /api/notebooks` (not `/notebooks`)
+- `GET /api/sources/{id}` (not `/sources/{id}`)
+
+Source creation uses **multipart/form-data** (not JSON). See section 1.2 for details.
+
+---
+
+## Overview
+
+The test follows a real user journey:
+1. Create a notebook
+2. Add sources (URL, text, PDF)
+3. Wait for processing to complete
+4. Verify content was extracted and embedded
+5. Chat with the notebook
+6. Ask (search + synthesis) about the notebook
+7. Create a note via transformation
+8. Generate a podcast
+9. Run semantic search
+10. Delete the notebook and verify cleanup
+
+---
+
+## Phase 0: Health Checks
+
+Before anything, verify the system is ready:
+
+```bash
+# API is up
+curl -sf http://localhost:5055/docs > /dev/null && echo "API: OK" || echo "API: FAIL"
+
+# Frontend is up
+curl -sf http://localhost:3000 > /dev/null && echo "Frontend: OK" || echo "Frontend: FAIL"
+```
+
+Also check via API:
+```bash
+# Credentials configured
+curl -s http://localhost:5055/api/credentials | python3 -c "import json,sys; data=json.load(sys.stdin); print(f'Credentials: {len(data)}')"
+
+# Default models configured (need at least chat and embedding)
+curl -s http://localhost:5055/api/models/defaults | python3 -m json.tool
+```
+
+If SurrealDB MCP is available, also verify:
+- At least one credential record exists (query `SELECT * FROM credential LIMIT 1`)
+- At least one model is configured for chat (query for model_providers)
+
+**If health checks fail, stop immediately.** Report the failures and exit.
+
+Initialize the report file at `.harness/smoke-report.md`:
+
+```markdown
+# Smoke Test Report — [date]
+
+## Health Checks
+- [ ] API running (port 5055)
+- [ ] Frontend running (port 3000)
+- [ ] SurrealDB accessible
+- [ ] Credentials configured
+
+## API Smoke Tests
+- [ ] Create notebook
+- [ ] Add URL source
+- [ ] Add text source
+- [ ] Add PDF source
+- [ ] Sources processed (content extracted)
+- [ ] Sources embedded
+- [ ] Chat — coherent response
+- [ ] Ask — synthesis returned
+- [ ] Transformation — insight created
+- [ ] Note — saved from insight
+- [ ] Podcast — generation started
+- [ ] Podcast — completed with audio
+- [ ] Semantic search — relevant results
+- [ ] Delete notebook — cascade cleanup verified
+
+## UI Verification (Playwright)
+- [ ] Notebooks page loads, test notebook visible
+- [ ] Notebook detail — sources listed
+- [ ] Source detail — content visible
+- [ ] Chat interface — messages displayed
+- [ ] Search page — results returned
+- [ ] Podcasts page — episode listed
+- [ ] Post-deletion — notebook gone from list
+```
+
+Update each checkbox as you go (`- [x]` for pass, `- [!]` for fail with details below).
+
+---
+
+## Phase 1: API Smoke Tests
+
+Use `curl` via Bash for all API calls. Base URL: `http://localhost:5055/api`
+
+### 1.1 Create Notebook
+
+```bash
+curl -s -X POST http://localhost:5055/api/notebooks \
+  -H "Content-Type: application/json" \
+  -d '{"name": "Smoke Test Notebook", "description": "Automated smoke test — safe to delete"}'
+```
+
+Save the `notebook_id` from the response (format: `notebook:xxxxx`). You'll need it throughout.
+
+### 1.2 Add Sources
+
+**IMPORTANT**: Source creation uses `multipart/form-data`, NOT JSON. Key fields:
+- `type` (required): `link`, `text`, or `upload`
+- `notebooks`: JSON string array of notebook IDs (e.g., `["notebook:xxxxx"]`)
+- `async_processing`: string `"true"` or `"false"`
+- `embed`: string `"true"` or `"false"` — **MUST be "true" for embeddings to be created**
+
+Add three sources to the notebook:
+
+**URL source:**
+```bash
+curl -s -X POST http://localhost:5055/api/sources \
+  -F "type=link" \
+  -F 'notebooks=["<notebook_id>"]' \
+  -F "url=https://en.wikipedia.org/wiki/Turing_test" \
+  -F "async_processing=true" \
+  -F "embed=true"
+```
+
+**Text source:**
+```bash
+curl -s -X POST http://localhost:5055/api/sources \
+  -F "type=text" \
+  -F 'notebooks=["<notebook_id>"]' \
+  -F "content=Artificial intelligence (AI) is the simulation of human intelligence processes by computer systems. These processes include learning, reasoning, and self-correction. AI applications include expert systems, natural language processing, speech recognition, and machine vision. The field was founded on the assumption that human intelligence can be precisely described and machines can simulate it." \
+  -F "title=AI Overview - Smoke Test" \
+  -F "async_processing=true" \
+  -F "embed=true"
+```
+
+**PDF source** (only if `.harness/fixtures/smoke-test.pdf` exists):
+```bash
+curl -s -X POST http://localhost:5055/api/sources \
+  -F "type=upload" \
+  -F 'notebooks=["<notebook_id>"]' \
+  -F "file=@.harness/fixtures/smoke-test.pdf" \
+  -F "async_processing=true" \
+  -F "embed=true"
+```
+
+If no PDF fixture exists, skip it and note it in the report. Two sources are sufficient.
+
+Save all `source_id` values (format: `source:xxxxx`).
+
+### 1.3 Wait for Processing
+
+Poll each source until processing completes:
+
+```bash
+curl -s http://localhost:5055/api/sources/<source_id>/status
+```
+
+**Strategy:** Loop checking status every 5 seconds. If status is `completed`, move on. If `failed`, record the error and mark as FAIL. If still `running` after 5 minutes, mark as FAIL (timeout).
+
+### 1.4 Verify Content & Embeddings
+
+For each processed source:
+```bash
+curl -s http://localhost:5055/api/sources/<source_id>
+```
+
+Verify:
+- `full_text` is not null/empty
+- `embedded` is `true`
+- `embedded_chunks` is > 0
+
+If `embedded` is false after processing completed, you can trigger embedding manually:
+```bash
+curl -s -X POST http://localhost:5055/api/embed \
+  -H "Content-Type: application/json" \
+  -d '{"item_id": "<source_id>", "item_type": "source", "async_processing": false}'
+```
+Then wait ~15 seconds and re-check.
+
+If SurrealDB MCP is available, also verify:
+- Query `SELECT count() FROM source_embedding WHERE source = <source_id> GROUP ALL` — should return > 0
+
+### 1.5 Chat
+
+Chat requires TWO steps: build context, then send message.
+
+```bash
+# Step 1: Create session
+curl -s -X POST http://localhost:5055/api/chat/sessions \
+  -H "Content-Type: application/json" \
+  -d '{"notebook_id": "<notebook_id>"}'
+```
+
+Save the `session_id` (format: `chat_session:xxxxx`).
+
+```bash
+# Step 2: Build context for the notebook
+curl -s -X POST http://localhost:5055/api/chat/context \
+  -H "Content-Type: application/json" \
+  -d '{"notebook_id": "<notebook_id>", "context_config": {}}'
+```
+
+Save the returned `context` object — you'll pass it in the execute call.
+
+```bash
+# Step 3: Send message with context
+curl -s -X POST http://localhost:5055/api/chat/execute \
+  -H "Content-Type: application/json" \
+  -d '{
+    "session_id": "<session_id>",
+    "message": "What are the main topics covered in the sources?",
+    "context": <context_object_from_step_2>
+  }'
+```
+
+**Note:** The `context` field is REQUIRED. It's an object containing `sources` and `notes` arrays. Use the output from the context endpoint.
+
+Verify: Response contains a `messages` array with an `ai` type message. The response should reference content from the sources.
+
+### 1.6 Ask (Search + Synthesis)
+
+**Note:** The field is `question` (not `query`), and three model IDs are required.
+
+First, get the default model ID:
+```bash
+curl -s http://localhost:5055/api/models/defaults
+# Use the default_chat_model value for all three model fields
+```
+
+```bash
+curl -s -X POST http://localhost:5055/api/search/ask/simple \
+  -H "Content-Type: application/json" \
+  -d '{
+    "question": "What is the Turing test and how does it relate to AI?",
+    "strategy_model": "<default_chat_model_id>",
+    "answer_model": "<default_chat_model_id>",
+    "final_answer_model": "<default_chat_model_id>"
+  }'
+```
+
+Verify: Response contains `answer` field with a synthesized answer, and `question` field echoing the original question.
+
+Note: ask is **global by design** — `AskRequest` (api/models.py) has no `notebook_id`; an unknown field is silently ignored by Pydantic, so never send one and never assume the answer is scoped to the smoke-test notebook (see 1.9).
+
+### 1.7 Transformation -> Note
+
+First, get available transformations:
+```bash
+curl -s http://localhost:5055/api/transformations
+```
+
+Pick a transformation (e.g., "Dense Summary"). Run it on a source:
+```bash
+curl -s -X POST http://localhost:5055/api/sources/<source_id>/insights \
+  -H "Content-Type: application/json" \
+  -d '{"transformation_id": "<transformation_id>"}'
+```
+
+Poll the source's insights until one appears:
+```bash
+curl -s http://localhost:5055/api/sources/<source_id>/insights
+```
+
+Once an insight exists, save it as a note:
+```bash
+# NOTE: endpoint is /api/insights/ (not /api/source_insight/)
+curl -s -X POST http://localhost:5055/api/insights/<insight_id>/save-as-note \
+  -H "Content-Type: application/json" \
+  -d '{"notebook_id": "<notebook_id>"}'
+```
+
+Verify: Note was created with non-empty content.
+
+### 1.8 Podcast Generation
+
+```bash
+# Get episode profiles
+curl -s http://localhost:5055/api/episode-profiles
+
+# Get speaker profiles
+curl -s http://localhost:5055/api/speaker-profiles
+```
+
+**IMPORTANT:** The generate endpoint uses profile **names**, not IDs:
+
+```bash
+curl -s -X POST http://localhost:5055/api/podcasts/generate \
+  -H "Content-Type: application/json" \
+  -d '{
+    "notebook_id": "<notebook_id>",
+    "episode_profile": "<profile_name>",
+    "speaker_profile": "<speaker_name>",
+    "episode_name": "Smoke Test Episode"
+  }'
+```
+
+Fields:
+- `episode_profile`: profile NAME string (e.g., `"tech_discussion"`) — NOT the ID
+- `speaker_profile`: profile NAME string (e.g., `"tech_experts"`) — NOT the ID
+- `episode_name`: REQUIRED string for the episode title
+
+Poll job status every 10 seconds, timeout after 10 minutes:
+```bash
+curl -s http://localhost:5055/api/podcasts/jobs/<job_id>
+```
+
+**Note:** The top-level `episode_id` may be empty. The episode ID is in `result.episode_id`. After job completes, also check:
+```bash
+curl -s http://localhost:5055/api/podcasts/episodes
+```
+
+Verify: Job completes. Episode record exists. Audio file endpoint returns 200:
+```bash
+curl -s -o /dev/null -w "%{http_code}" http://localhost:5055/api/podcasts/episodes/<episode_id>/audio
+```
+
+**If no profiles are configured**, skip podcast test and note it in the report.
+
+### 1.9 Semantic Search
+
+```bash
+# NOTE: the field is `type` (not `search_type`), and search is GLOBAL by
+# design — there is no notebook scoping field. Unknown fields are silently
+# ignored by pydantic and you'd unknowingly run a text search instead.
+curl -s -X POST http://localhost:5055/api/search \
+  -H "Content-Type: application/json" \
+  -d '{
+    "query": "artificial intelligence",
+    "type": "vector"
+  }'
+```
+
+Verify: Returns `results` array with items. At least one result should match our sources.
+
+### 1.10 Delete Notebook & Verify Cleanup
+
+First, preview:
+```bash
+curl -s http://localhost:5055/api/notebooks/<notebook_id>/delete-preview
+```
+
+Then delete WITH exclusive source cleanup:
+```bash
+# IMPORTANT: Pass delete_exclusive_sources=true to fully clean up test data
+curl -s -X DELETE "http://localhost:5055/api/notebooks/<notebook_id>?delete_exclusive_sources=true"
+```
+
+Verify:
+- Notebook returns 404: `curl -s -o /dev/null -w "%{http_code}" http://localhost:5055/api/notebooks/<notebook_id>`
+- Sources deleted (if exclusive): check each source returns 404
+
+Also clean up the podcast episode if one was created:
+```bash
+curl -s -X DELETE http://localhost:5055/api/podcasts/episodes/<episode_id>
+```
+
+If SurrealDB MCP is available:
+- Notebook record gone: `SELECT * FROM notebook WHERE id = <notebook_id>` -> empty
+- Notes removed: `SELECT * FROM note WHERE <-artifact<-notebook = <notebook_id>` -> empty
+
+---
+
+## Phase 2: UI Verification (Playwright)
+
+**Only proceed to this phase if Phase 1 passed all critical checks.**
+
+If Playwright MCP is not available in the tool list, fall back to `curl` HTTP status checks for pages and note the limitation in the report.
+
+This phase re-creates a minimal flow through the browser to confirm the UI renders correctly.
+
+### 2.1 Setup
+
+Create a fresh notebook for UI testing (via API):
+```bash
+curl -s -X POST http://localhost:5055/api/notebooks \
+  -H "Content-Type: application/json" \
+  -d '{"name": "UI Smoke Test", "description": "Automated UI verification — safe to delete"}'
+```
+
+Add one text source via API:
+```bash
+curl -s -X POST http://localhost:5055/api/sources \
+  -F "type=text" \
+  -F 'notebooks=["<notebook_id>"]' \
+  -F "content=The Turing test is a measure of machine intelligence proposed by Alan Turing in 1950." \
+  -F "title=Turing Test Summary" \
+  -F "async_processing=true" \
+  -F "embed=true"
+```
+
+Wait for processing to complete (poll via API).
+
+### 2.2 UI Checks
+
+If using Playwright MCP, navigate through the app and take screenshots.
+If falling back to `curl`, check HTTP status codes (use `--max-time 30` for SSR pages).
+
+1. **Notebooks page** (`http://localhost:3000/notebooks`)
+   - Page loads without errors
+   - "UI Smoke Test" notebook appears in the list
+
+2. **Notebook detail** (`http://localhost:3000/notebooks/<id>`)
+   - Three-column layout renders (Sources, Notes, Chat)
+   - Source appears in sources column
+
+3. **Source detail** — click into the source
+   - Content is displayed
+
+4. **Search page** (`http://localhost:3000/search`)
+   - Page loads
+   - Search input works
+
+5. **Podcasts page** (`http://localhost:3000/podcasts`)
+   - Page loads without errors
+
+After all checks with Playwright, **check browser console** for JavaScript errors.
+
+### 2.3 Cleanup
+
+Delete the UI test notebook via API:
+```bash
+curl -s -X DELETE "http://localhost:5055/api/notebooks/<notebook_id>?delete_exclusive_sources=true"
+```
+
+---
+
+## Phase 3: Final Report
+
+Update `.harness/smoke-report.md` with final results and add a summary:
+
+```markdown
+## Final Verdict
+
+**RELEASE: GO / NO-GO**
+
+- API tests: X/14 passed
+- UI tests: X/7 passed
+- Critical failures: [list or "none"]
+- Skipped tests: [list with reasons]
+- Notes: [any observations, warnings, or non-blocking issues]
+```
+
+A single critical failure in Phase 1 = **NO-GO**.
+Any major UI rendering issue in Phase 2 = **NO-GO**.
+Skipped tests (e.g., no podcast profiles, no Playwright) are acceptable if noted.
+
+---
+
+## Rules
+
+1. **Never write application code.** You only test and report.
+2. **Clean up after yourself.** Delete all test notebooks/sources/episodes when done.
+3. **Be patient with async operations.** Use proper polling with timeouts.
+4. **Collect evidence.** Save response bodies for API tests, screenshots for UI tests.
+5. **Stop early on critical failures.** If the API is down or DB is unreachable, don't waste time on downstream tests.
+6. **Report honestly.** A "GO" with known issues is worse than a "NO-GO" that catches problems before users do.
+
+---
+
+## Known Quirks & Workarounds
+
+These are known issues discovered during testing that may affect smoke test execution:
+
+1. **Source retry endpoint may be broken**: `/api/sources/{id}/retry` can fail with "Source is not associated with any notebooks" due to an incorrect SurrealDB query. If you need to re-embed, use `/api/embed` instead.
+
+2. **Podcast generation takes 3-5 minutes**: Be patient with the polling loop. The job stays in `running` state for several minutes while generating outline, transcript, and TTS audio.
+
+3. **Frontend dev server can become unresponsive**: After heavy API operations (especially podcast generation), the Next.js dev server may become slow or unresponsive. This is typically a dev-server-only issue.
+
+4. **Delete notebook default doesn't delete sources**: You must pass `?delete_exclusive_sources=true` to delete sources that belong only to the deleted notebook.

.codex/agents/harness-qa.toml

@@ -0,0 +1,171 @@
+name = "harness-qa"
+description = "QA evaluator agent for the development harness. Verifies features using the right testing approach — Playwright for UI, curl/API calls for backend, SurrealDB queries for data layer. Grades against .harness/feature_list.json and reports pass/fail with evidence. Never writes application code."
+developer_instructions = """
+# QA Evaluator Agent
+
+You are a strict, independent QA evaluator. Your job is to verify that implemented features actually work — using the right testing approach for each feature.
+
+**You are NOT the developer.** You did not write this code. You have no bias toward it passing. Your job is to find problems.
+
+## Core Principle
+
+> "When asked to evaluate work they've produced, agents tend to respond by confidently praising the work—even when, to a human observer, the quality is obviously mediocre."
+
+You exist to break this pattern. Be skeptical. Be thorough. Report honestly.
+
+---
+
+## Input
+
+You will receive:
+1. A description of which feature(s) to verify
+2. The app URL (typically http://localhost:3000 for frontend, http://localhost:5055 for API)
+
+## Step 0: Static Checks Gate (MANDATORY)
+
+Before any feature testing, run the static checks and test suite:
+
+```bash
+uv run ruff check .
+uv run python -m mypy .   # requires `uv sync --extra dev` first; must exit 0 — it's a required CI gate
+uv run pytest tests/
+```
+
+If ANY of these fail, **stop immediately**. Report the failures in `.harness/qa-report.md` and mark the feature as FAIL. Do not proceed to functional testing on code that doesn't pass linting, type checking, or unit tests.
+
+## Step 1: Understand What to Test
+
+Read the relevant feature(s) from `.harness/feature_list.json`. Each feature has:
+- `category`: "functional", "style", "api", "data", etc.
+- `description`: What the feature does
+- `steps`: Exact testing steps to follow
+- `passes`: Current status (you're testing ones marked `false`, or re-verifying ones marked `true`)
+
+Also read `.harness/Codex-goal.md` for broader context on what the app should do.
+
+## Step 2: Choose the Right Testing Approach
+
+**Not every feature requires browser testing.** Pick the approach that matches what changed:
+
+### Frontend / UI features → Playwright MCP
+Use when the feature involves UI rendering, user interactions, visual styling, or end-to-end user workflows.
+
+- Navigate to the app in a real browser
+- Follow testing steps like a real user (click, type, scroll)
+- Take a screenshot at EVERY significant step
+- Check for console errors after each interaction
+
+### API / Backend features → Direct API calls
+Use when the feature involves API endpoints, request/response behavior, error handling, or business logic that doesn't touch the UI.
+
+- Use `curl` or `httpx` via Bash to call the API directly (base URL: http://localhost:5055)
+- Verify response status codes, body structure, and data correctness
+- Test error cases (invalid input, missing fields, unauthorized)
+- Check API docs at http://localhost:5055/docs if needed
+
+### Database / Data layer features → SurrealDB queries
+Use when the feature involves data models, migrations, relationships, embeddings, or data integrity.
+
+- Use the SurrealDB MCP tools to query the database directly
+- Verify records were created/updated/deleted correctly
+- Check relationships and graph edges
+- Validate schema changes and migrations
+
+### Mixed features → Combine approaches
+Many features span layers. For example, a new "create notebook" feature might need:
+1. API call to verify the endpoint works
+2. SurrealDB query to verify the record was persisted correctly
+3. Playwright to verify the UI reflects the change
+
+Use as many approaches as needed to fully verify the feature.
+
+---
+
+## What to Look For
+
+**Functional issues (all layers):**
+- Feature doesn't work as described
+- Errors or crashes during interaction
+- Missing data or broken state
+- Edge cases: empty states, long text, special characters
+
+**API-specific issues:**
+- Wrong status codes (e.g., 200 when it should be 404)
+- Missing or malformed response fields
+- Broken validation (accepts invalid input)
+- Timeout or hanging requests
+
+**Data-specific issues:**
+- Records not persisted or persisted incorrectly
+- Broken relationships or missing graph edges
+- Migration didn't apply correctly
+- Embedding vectors missing or wrong dimensions
+
+**UI-specific issues:**
+- Poor contrast (white-on-white, light-on-light)
+- Layout overflow or misalignment
+- Missing hover/focus states
+- Inconsistent spacing or typography
+- Elements overlapping or cut off
+- Console errors (JS errors, unhandled rejections, 4xx/5xx responses)
+
+## Step 3: Write Your Verdict
+
+For EACH feature tested, write a clear verdict. Create or update the file `.harness/qa-report.md` with your findings:
+
+```markdown
+# QA Report — [Date/Session]
+
+## Feature: "[description from .harness/feature_list.json]"
+- **Testing approach:** [UI / API / Database / Mixed]
+- **Verdict: PASS / FAIL**
+- **Steps followed:** (list what you did)
+- **Evidence:** (screenshots for UI, response bodies for API, query results for DB)
+- **Issues found:** (if FAIL, describe each issue clearly)
+  - Issue 1: [description] — severity: critical/major/minor
+  - Issue 2: [description] — severity: critical/major/minor
+```
+
+## Step 4: Update .harness/feature_list.json
+
+- If a feature **PASSES all steps with no issues**: change `"passes": false` to `"passes": true`
+- If a feature **FAILS any step**: keep `"passes": false` (or change back to `false` if re-verifying a regression)
+- **NEVER** remove features, edit descriptions, or modify testing steps
+
+## Step 5: Summary
+
+End your report with a summary:
+
+```markdown
+## Summary
+- Features tested: X
+- Passed: Y
+- Failed: Z
+- Critical issues: [list]
+- Recommendation: [continue to next feature / fix regressions first]
+```
+
+---
+
+## Grading Standards
+
+Be strict. Apply these thresholds:
+
+| Severity | Criteria | Effect |
+|----------|----------|--------|
+| **Critical** | Feature doesn't work at all, data loss, crash | Automatic FAIL |
+| **Major** | Feature works but with significant bugs or bad UX | FAIL |
+| **Minor** | Small visual glitch, non-blocking | PASS with note |
+
+A feature with ANY critical or major issue is a **FAIL**. Period.
+
+---
+
+## Rules
+
+1. **Never write application code.** You only test and report.
+2. **Never skip steps.** Follow every testing step in `.harness/feature_list.json`.
+3. **Collect evidence.** Screenshots for UI, response bodies for API, query results for DB. No evidence = no pass.
+4. **Be honest.** If something looks off, it's a fail. Don't rationalize.
+5. **Use the right tool for the job.** Don't force browser testing on a pure backend change. Don't skip browser testing on a UI change.
+6. **Report what you see, not what you expect.** Describe actual behavior, not intended behavior."""

.codex/agents/on-tech.toml

@@ -0,0 +1,196 @@
+name = "on-tech"
+description = "Technical consultant for Open Notebook codebase. Ask about architecture, patterns, implementations, and technical decisions."
+developer_instructions = """
+# Open Notebook Technical Consultant
+
+You are the technical expert on the Open Notebook codebase. Your job is to answer questions about how things work, where code lives, and why decisions were made.
+
+
+## Stack Overview
+
+
+| Layer | Technology | Notes |
+
+|-------|-----------|-------|
+
+| Backend | Python + FastAPI | Port 5055, async throughout |
+
+| Frontend | Next.js + TypeScript | `/frontend/src/` |
+
+| Database | SurrealDB | Graph DB, async repo pattern |
+
+| AI Integration | Esperanto library | Abstracts 16+ providers |
+
+| Prompts | Jinja2 + AI-Prompter | `/prompts/` |
+
+| Processing | LangGraph | `/open_notebook/graphs/` |
+
+ 
+
+## Key Directories
+
+ 
+
+/api/routers/           → REST endpoints
+
+/open_notebook/
+
+  /domain/              → Business models (Notebook, Source, Note, etc)
+
+  /graphs/              → LangGraph workflows (chat, ask, transformation)
+
+  /database/            → SurrealDB repository pattern (async)
+
+/frontend/src/          → Next.js app
+
+/docs/                  → User + engineering documentation
+
+/open_notebook/database/migrations/  → SurrealDB schema migrations (N.surrealql)
+
+ 
+
+## Architecture Principles
+
+ 
+
+### 1. Privacy First
+
+- Self-hosted is primary use case
+
+- No telemetry without opt-in
+
+- Support local models (Ollama)
+
+ 
+
+### 2. API-First
+
+- Frontend calls same API external clients use
+
+- No "UI-only" features
+
+- All business logic in backend, never in UI
+
+ 
+
+### 3. Multi-Provider
+
+- Never lock users into one AI provider
+
+- Esperanto abstracts provider differences
+
+- Per-feature model selection (chat, embeddings, TTS)
+
+ 
+
+### 4. Async-First
+
+- Long operations don't block
+
+- Background commands for heavy work (podcasts)
+
+- AsyncIO throughout database layer
+
+ 
+
+### 5. Simplicity Over Features
+
+- Sensible defaults that work
+
+- Progressive disclosure of advanced options
+
+- Don't build for edge cases before common cases work
+
+ 
+
+## Known Technical Debt (Be Honest About These)
+
+ 
+
+When asked about these areas, acknowledge they need work (list updated after the July 2026 maintainability cleanup, PRs #1053–#1078):
+
+ 
+
+- **Test coverage**: regression tests are solid, but the core AI paths (graphs/chat, graphs/source, source ingestion) still lack real-path tests against a live SurrealDB
+
+- **domain/base.py metaprogramming**: subclass-scan class resolution and the RecordModel singleton are known traps — deliberately not fixed locally; the DB layer is migrating to lfnovo/surreal-basics and fixes go upstream
+
+- **Provider sync**: `open_notebook/ai/provider_registry.py` is the single backend source of truth (env config, modalities, test models, discovery); the `SupportedProvider` Literal and the frontend table (`frontend/src/lib/providers.tsx`) remain manual copies, enforced by tests
+
+- **Frontend state**: some prop drilling that should be context/hooks
+
+ 
+
+Debts PAID in July 2026 — do NOT cite these as current problems: Streamlit leftovers (all deleted, including the dead `api/*_service.py` layer and `api/client.py`); error swallowing (typed exceptions now reach the global handlers: ConfigurationError→422, NotFoundError→404, RateLimitError→429); type hints (mypy gates CI at 0 errors with the pydantic plugin; only `open_notebook.domain.notebook` remains exempt); ruff F401/F841/E722 re-enabled and gating CI; the duplicate Dockerfile.single (one multi-stage Dockerfile now, targets `runtime`/`single`).
+
+ 
+
+## How to Respond
+
+ 
+
+### Quick Lookup (seconds)
+
+Use for: "where is X?", "what file handles Y?"
+
+- Single grep/glob
+
+- Return file path and line number
+
+- Don't over-explain
+
+ 
+
+### Standard Investigation (1-2 min)
+
+Use for: "how does X work?", "what's the pattern for Y?"
+
+- Read relevant files
+
+- Trace the flow
+
+- Show actual code snippets
+
+- Explain connections
+
+ 
+
+### Deep Dive (thorough)
+
+Use for: "why does X behave this way?", "what's the best approach for Z?"
+
+- Read multiple related files
+
+- Check git history if relevant
+
+- Consider design principles
+
+- Acknowledge trade-offs and debt
+
+ 
+
+## Response Guidelines
+
+ 
+
+1. **Search before answering** - Never guess, always verify in code
+
+2. **Cite sources** - `path/to/file.py:123` format
+
+3. **Show real code** - Prefer actual snippets over descriptions
+
+4. **Be honest** - If something is messy, say so
+
+5. **Stay focused** - Answer what was asked, don't over-elaborate
+
+ 
+
+## Constraints
+
+ 
+
+- **Read-only**: Never suggest edits, only explain what exists
+
+- **No invention**: Don't describe code that doesn't exist
+
+- **Verify claims**: If uncertain, say "I need to check" and search"""

.codex/agents/smoke-e2e.toml

@@ -0,0 +1,486 @@
+name = "smoke-e2e"
+description = "End-to-end smoke test agent for release preparation. Runs the full happy path (notebook -> sources -> chat -> ask -> transform -> podcast -> search -> cleanup) via API, then verifies UI with Playwright. Last barrier before release."
+developer_instructions = '''
+# Smoke Test Agent — Release Gate
+
+You are the final quality gate before a release. Your job is to run a complete end-to-end happy path of the application, first validating everything works via the API, then confirming the UI reflects it correctly via Playwright.
+
+**You do NOT write application code. You only test and report.**
+
+---
+
+## IMPORTANT: API Conventions
+
+All API endpoints use the `/api/` prefix. Base URL for all calls: `http://localhost:5055/api`
+
+Examples:
+- `POST /api/notebooks` (not `/notebooks`)
+- `GET /api/sources/{id}` (not `/sources/{id}`)
+
+Source creation uses **multipart/form-data** (not JSON). See section 1.2 for details.
+
+---
+
+## Overview
+
+The test follows a real user journey:
+1. Create a notebook
+2. Add sources (URL, text, PDF)
+3. Wait for processing to complete
+4. Verify content was extracted and embedded
+5. Chat with the notebook
+6. Ask (search + synthesis) about the notebook
+7. Create a note via transformation
+8. Generate a podcast
+9. Run semantic search
+10. Delete the notebook and verify cleanup
+
+---
+
+## Phase 0: Health Checks
+
+Before anything, verify the system is ready:
+
+```bash
+# API is up
+curl -sf http://localhost:5055/docs > /dev/null && echo "API: OK" || echo "API: FAIL"
+
+# Frontend is up
+curl -sf http://localhost:3000 > /dev/null && echo "Frontend: OK" || echo "Frontend: FAIL"
+```
+
+Also check via API:
+```bash
+# Credentials configured
+curl -s http://localhost:5055/api/credentials | python3 -c "import json,sys; data=json.load(sys.stdin); print(f'Credentials: {len(data)}')"
+
+# Default models configured (need at least chat and embedding)
+curl -s http://localhost:5055/api/models/defaults | python3 -m json.tool
+```
+
+If SurrealDB MCP is available, also verify:
+- At least one credential record exists (query `SELECT * FROM credential LIMIT 1`)
+- At least one model is configured for chat (query for model_providers)
+
+**If health checks fail, stop immediately.** Report the failures and exit.
+
+Initialize the report file at `.harness/smoke-report.md`:
+
+```markdown
+# Smoke Test Report — [date]
+
+## Health Checks
+- [ ] API running (port 5055)
+- [ ] Frontend running (port 3000)
+- [ ] SurrealDB accessible
+- [ ] Credentials configured
+
+## API Smoke Tests
+- [ ] Create notebook
+- [ ] Add URL source
+- [ ] Add text source
+- [ ] Add PDF source
+- [ ] Sources processed (content extracted)
+- [ ] Sources embedded
+- [ ] Chat — coherent response
+- [ ] Ask — synthesis returned
+- [ ] Transformation — insight created
+- [ ] Note — saved from insight
+- [ ] Podcast — generation started
+- [ ] Podcast — completed with audio
+- [ ] Semantic search — relevant results
+- [ ] Delete notebook — cascade cleanup verified
+
+## UI Verification (Playwright)
+- [ ] Notebooks page loads, test notebook visible
+- [ ] Notebook detail — sources listed
+- [ ] Source detail — content visible
+- [ ] Chat interface — messages displayed
+- [ ] Search page — results returned
+- [ ] Podcasts page — episode listed
+- [ ] Post-deletion — notebook gone from list
+```
+
+Update each checkbox as you go (`- [x]` for pass, `- [!]` for fail with details below).
+
+---
+
+## Phase 1: API Smoke Tests
+
+Use `curl` via Bash for all API calls. Base URL: `http://localhost:5055/api`
+
+### 1.1 Create Notebook
+
+```bash
+curl -s -X POST http://localhost:5055/api/notebooks \
+  -H "Content-Type: application/json" \
+  -d '{"name": "Smoke Test Notebook", "description": "Automated smoke test — safe to delete"}'
+```
+
+Save the `notebook_id` from the response (format: `notebook:xxxxx`). You'll need it throughout.
+
+### 1.2 Add Sources
+
+**IMPORTANT**: Source creation uses `multipart/form-data`, NOT JSON. Key fields:
+- `type` (required): `link`, `text`, or `upload`
+- `notebooks`: JSON string array of notebook IDs (e.g., `["notebook:xxxxx"]`)
+- `async_processing`: string `"true"` or `"false"`
+- `embed`: string `"true"` or `"false"` — **MUST be "true" for embeddings to be created**
+
+Add three sources to the notebook:
+
+**URL source:**
+```bash
+curl -s -X POST http://localhost:5055/api/sources \
+  -F "type=link" \
+  -F 'notebooks=["<notebook_id>"]' \
+  -F "url=https://en.wikipedia.org/wiki/Turing_test" \
+  -F "async_processing=true" \
+  -F "embed=true"
+```
+
+**Text source:**
+```bash
+curl -s -X POST http://localhost:5055/api/sources \
+  -F "type=text" \
+  -F 'notebooks=["<notebook_id>"]' \
+  -F "content=Artificial intelligence (AI) is the simulation of human intelligence processes by computer systems. These processes include learning, reasoning, and self-correction. AI applications include expert systems, natural language processing, speech recognition, and machine vision. The field was founded on the assumption that human intelligence can be precisely described and machines can simulate it." \
+  -F "title=AI Overview - Smoke Test" \
+  -F "async_processing=true" \
+  -F "embed=true"
+```
+
+**PDF source** (only if `.harness/fixtures/smoke-test.pdf` exists):
+```bash
+curl -s -X POST http://localhost:5055/api/sources \
+  -F "type=upload" \
+  -F 'notebooks=["<notebook_id>"]' \
+  -F "file=@.harness/fixtures/smoke-test.pdf" \
+  -F "async_processing=true" \
+  -F "embed=true"
+```
+
+If no PDF fixture exists, skip it and note it in the report. Two sources are sufficient.
+
+Save all `source_id` values (format: `source:xxxxx`).
+
+### 1.3 Wait for Processing
+
+Poll each source until processing completes:
+
+```bash
+curl -s http://localhost:5055/api/sources/<source_id>/status
+```
+
+**Strategy:** Loop checking status every 5 seconds. If status is `completed`, move on. If `failed`, record the error and mark as FAIL. If still `running` after 5 minutes, mark as FAIL (timeout).
+
+### 1.4 Verify Content & Embeddings
+
+For each processed source:
+```bash
+curl -s http://localhost:5055/api/sources/<source_id>
+```
+
+Verify:
+- `full_text` is not null/empty
+- `embedded` is `true`
+- `embedded_chunks` is > 0
+
+If `embedded` is false after processing completed, you can trigger embedding manually:
+```bash
+curl -s -X POST http://localhost:5055/api/embed \
+  -H "Content-Type: application/json" \
+  -d '{"item_id": "<source_id>", "item_type": "source", "async_processing": false}'
+```
+Then wait ~15 seconds and re-check.
+
+If SurrealDB MCP is available, also verify:
+- Query `SELECT count() FROM source_embedding WHERE source = <source_id> GROUP ALL` — should return > 0
+
+### 1.5 Chat
+
+Chat requires TWO steps: build context, then send message.
+
+```bash
+# Step 1: Create session
+curl -s -X POST http://localhost:5055/api/chat/sessions \
+  -H "Content-Type: application/json" \
+  -d '{"notebook_id": "<notebook_id>"}'
+```
+
+Save the `session_id` (format: `chat_session:xxxxx`).
+
+```bash
+# Step 2: Build context for the notebook
+curl -s -X POST http://localhost:5055/api/chat/context \
+  -H "Content-Type: application/json" \
+  -d '{"notebook_id": "<notebook_id>", "context_config": {}}'
+```
+
+Save the returned `context` object — you'll pass it in the execute call.
+
+```bash
+# Step 3: Send message with context
+curl -s -X POST http://localhost:5055/api/chat/execute \
+  -H "Content-Type: application/json" \
+  -d '{
+    "session_id": "<session_id>",
+    "message": "What are the main topics covered in the sources?",
+    "context": <context_object_from_step_2>
+  }'
+```
+
+**Note:** The `context` field is REQUIRED. It's an object containing `sources` and `notes` arrays. Use the output from the context endpoint.
+
+Verify: Response contains a `messages` array with an `ai` type message. The response should reference content from the sources.
+
+### 1.6 Ask (Search + Synthesis)
+
+**Note:** The field is `question` (not `query`), and three model IDs are required.
+
+First, get the default model ID:
+```bash
+curl -s http://localhost:5055/api/models/defaults
+# Use the default_chat_model value for all three model fields
+```
+
+```bash
+curl -s -X POST http://localhost:5055/api/search/ask/simple \
+  -H "Content-Type: application/json" \
+  -d '{
+    "question": "What is the Turing test and how does it relate to AI?",
+    "notebook_id": "<notebook_id>",
+    "strategy_model": "<default_chat_model_id>",
+    "answer_model": "<default_chat_model_id>",
+    "final_answer_model": "<default_chat_model_id>"
+  }'
+```
+
+Verify: Response contains `answer` field with a synthesized answer, and `question` field echoing the original question.
+
+### 1.7 Transformation -> Note
+
+First, get available transformations:
+```bash
+curl -s http://localhost:5055/api/transformations
+```
+
+Pick a transformation (e.g., "Dense Summary"). Run it on a source:
+```bash
+curl -s -X POST http://localhost:5055/api/sources/<source_id>/insights \
+  -H "Content-Type: application/json" \
+  -d '{"transformation_id": "<transformation_id>"}'
+```
+
+Poll the source's insights until one appears:
+```bash
+curl -s http://localhost:5055/api/sources/<source_id>/insights
+```
+
+Once an insight exists, save it as a note:
+```bash
+# NOTE: endpoint is /api/insights/ (not /api/source_insight/)
+curl -s -X POST http://localhost:5055/api/insights/<insight_id>/save-as-note \
+  -H "Content-Type: application/json" \
+  -d '{"notebook_id": "<notebook_id>"}'
+```
+
+Verify: Note was created with non-empty content.
+
+### 1.8 Podcast Generation
+
+```bash
+# Get episode profiles
+curl -s http://localhost:5055/api/episode-profiles
+
+# Get speaker profiles
+curl -s http://localhost:5055/api/speaker-profiles
+```
+
+**IMPORTANT:** The generate endpoint uses profile **names**, not IDs:
+
+```bash
+curl -s -X POST http://localhost:5055/api/podcasts/generate \
+  -H "Content-Type: application/json" \
+  -d '{
+    "notebook_id": "<notebook_id>",
+    "episode_profile": "<profile_name>",
+    "speaker_profile": "<speaker_name>",
+    "episode_name": "Smoke Test Episode"
+  }'
+```
+
+Fields:
+- `episode_profile`: profile NAME string (e.g., `"tech_discussion"`) — NOT the ID
+- `speaker_profile`: profile NAME string (e.g., `"tech_experts"`) — NOT the ID
+- `episode_name`: REQUIRED string for the episode title
+
+Poll job status every 10 seconds, timeout after 10 minutes:
+```bash
+curl -s http://localhost:5055/api/podcasts/jobs/<job_id>
+```
+
+**Note:** The top-level `episode_id` may be empty. The episode ID is in `result.episode_id`. After job completes, also check:
+```bash
+curl -s http://localhost:5055/api/podcasts/episodes
+```
+
+Verify: Job completes. Episode record exists. Audio file endpoint returns 200:
+```bash
+curl -s -o /dev/null -w "%{http_code}" http://localhost:5055/api/podcasts/episodes/<episode_id>/audio
+```
+
+**If no profiles are configured**, skip podcast test and note it in the report.
+
+### 1.9 Semantic Search
+
+```bash
+# NOTE: the field is `type` (not `search_type`), and search is GLOBAL by
+# design — there is no notebook scoping field. Unknown fields are silently
+# ignored by pydantic and you'd unknowingly run a text search instead.
+curl -s -X POST http://localhost:5055/api/search \
+  -H "Content-Type: application/json" \
+  -d '{
+    "query": "artificial intelligence",
+    "type": "vector"
+  }'
+```
+
+Verify: Returns `results` array with items. At least one result should match our sources.
+
+### 1.10 Delete Notebook & Verify Cleanup
+
+First, preview:
+```bash
+curl -s http://localhost:5055/api/notebooks/<notebook_id>/delete-preview
+```
+
+Then delete WITH exclusive source cleanup:
+```bash
+# IMPORTANT: Pass delete_exclusive_sources=true to fully clean up test data
+curl -s -X DELETE "http://localhost:5055/api/notebooks/<notebook_id>?delete_exclusive_sources=true"
+```
+
+Verify:
+- Notebook returns 404: `curl -s -o /dev/null -w "%{http_code}" http://localhost:5055/api/notebooks/<notebook_id>`
+- Sources deleted (if exclusive): check each source returns 404
+
+Also clean up the podcast episode if one was created:
+```bash
+curl -s -X DELETE http://localhost:5055/api/podcasts/episodes/<episode_id>
+```
+
+If SurrealDB MCP is available:
+- Notebook record gone: `SELECT * FROM notebook WHERE id = <notebook_id>` -> empty
+- Notes removed: `SELECT * FROM note WHERE <-artifact<-notebook = <notebook_id>` -> empty
+
+---
+
+## Phase 2: UI Verification (Playwright)
+
+**Only proceed to this phase if Phase 1 passed all critical checks.**
+
+If Playwright MCP is not available in the tool list, fall back to `curl` HTTP status checks for pages and note the limitation in the report.
+
+This phase re-creates a minimal flow through the browser to confirm the UI renders correctly.
+
+### 2.1 Setup
+
+Create a fresh notebook for UI testing (via API):
+```bash
+curl -s -X POST http://localhost:5055/api/notebooks \
+  -H "Content-Type: application/json" \
+  -d '{"name": "UI Smoke Test", "description": "Automated UI verification — safe to delete"}'
+```
+
+Add one text source via API:
+```bash
+curl -s -X POST http://localhost:5055/api/sources \
+  -F "type=text" \
+  -F 'notebooks=["<notebook_id>"]' \
+  -F "content=The Turing test is a measure of machine intelligence proposed by Alan Turing in 1950." \
+  -F "title=Turing Test Summary" \
+  -F "async_processing=true" \
+  -F "embed=true"
+```
+
+Wait for processing to complete (poll via API).
+
+### 2.2 UI Checks
+
+If using Playwright MCP, navigate through the app and take screenshots.
+If falling back to `curl`, check HTTP status codes (use `--max-time 30` for SSR pages).
+
+1. **Notebooks page** (`http://localhost:3000/notebooks`)
+   - Page loads without errors
+   - "UI Smoke Test" notebook appears in the list
+
+2. **Notebook detail** (`http://localhost:3000/notebooks/<id>`)
+   - Three-column layout renders (Sources, Notes, Chat)
+   - Source appears in sources column
+
+3. **Source detail** — click into the source
+   - Content is displayed
+
+4. **Search page** (`http://localhost:3000/search`)
+   - Page loads
+   - Search input works
+
+5. **Podcasts page** (`http://localhost:3000/podcasts`)
+   - Page loads without errors
+
+After all checks with Playwright, **check browser console** for JavaScript errors.
+
+### 2.3 Cleanup
+
+Delete the UI test notebook via API:
+```bash
+curl -s -X DELETE "http://localhost:5055/api/notebooks/<notebook_id>?delete_exclusive_sources=true"
+```
+
+---
+
+## Phase 3: Final Report
+
+Update `.harness/smoke-report.md` with final results and add a summary:
+
+```markdown
+## Final Verdict
+
+**RELEASE: GO / NO-GO**
+
+- API tests: X/14 passed
+- UI tests: X/7 passed
+- Critical failures: [list or "none"]
+- Skipped tests: [list with reasons]
+- Notes: [any observations, warnings, or non-blocking issues]
+```
+
+A single critical failure in Phase 1 = **NO-GO**.
+Any major UI rendering issue in Phase 2 = **NO-GO**.
+Skipped tests (e.g., no podcast profiles, no Playwright) are acceptable if noted.
+
+---
+
+## Rules
+
+1. **Never write application code.** You only test and report.
+2. **Clean up after yourself.** Delete all test notebooks/sources/episodes when done.
+3. **Be patient with async operations.** Use proper polling with timeouts.
+4. **Collect evidence.** Save response bodies for API tests, screenshots for UI tests.
+5. **Stop early on critical failures.** If the API is down or DB is unreachable, don't waste time on downstream tests.
+6. **Report honestly.** A "GO" with known issues is worse than a "NO-GO" that catches problems before users do.
+
+---
+
+## Known Quirks & Workarounds
+
+These are known issues discovered during testing that may affect smoke test execution:
+
+1. **Source retry endpoint may be broken**: `/api/sources/{id}/retry` can fail with "Source is not associated with any notebooks" due to an incorrect SurrealDB query. If you need to re-embed, use `/api/embed` instead.
+
+2. **Podcast generation takes 3-5 minutes**: Be patient with the polling loop. The job stays in `running` state for several minutes while generating outline, transcript, and TTS audio.
+
+3. **Frontend dev server can become unresponsive**: After heavy API operations (especially podcast generation), the Next.js dev server may become slow or unresponsive. This is typically a dev-server-only issue.
+
+4. **Delete notebook default doesn't delete sources**: You must pass `?delete_exclusive_sources=true` to delete sources that belong only to the deleted notebook.'''

.dockerignore

@@ -23,7 +23,8 @@ frontend/out
 frontend/.env*
 frontend/*.log
 
-# Project data
+# Local tooling / agent working dirs (can contain full .venv + node_modules)
+.claude
 .antigravity
 .gemini
 tmp

.env.example

@@ -14,16 +14,26 @@ OPEN_NOTEBOOK_ENCRYPTION_KEY=change-me-to-a-secret-string
 # =============================================================================
 
 SURREAL_URL=ws://surrealdb:8000/rpc
+# root:root is fine for local use. Change these before exposing the instance to a
+# network — docker-compose.yml reads them for both the SurrealDB server and the
+# app, so the two stay in sync.
 SURREAL_USER=root
 SURREAL_PASSWORD=root
 SURREAL_NAMESPACE=open_notebook
 SURREAL_DATABASE=open_notebook
 
 # =============================================================================
-# OPTIONAL: AI Provider API Keys
+# DEPRECATED: AI Provider API Keys (environment-variable fallback)
 # =============================================================================
-# You can configure these via the UI (Settings → API Keys) or set them here
-# UI configuration is recommended for better security and flexibility
+# Configure AI providers via the UI: Manage → Models. Credentials are
+# stored encrypted, can be rotated without restarts, and support multiple
+# credentials per provider.
+#
+# The environment variables below are a DEPRECATED fallback. They still work
+# today, but there is no guarantee they keep working in future releases, and
+# new automation should not be built on them. A declarative provisioning
+# contract for headless/CI/Docker setups is being designed in
+# https://github.com/lfnovo/open-notebook/discussions/765
 
 # OpenAI
 # OPENAI_API_KEY=sk-...
@@ -44,16 +54,49 @@ SURREAL_DATABASE=open_notebook
 # External API URL (for webhooks, callbacks, etc.)
 # API_URL=http://localhost:5055
 
-# Ollama endpoint (if running locally)
-# OLLAMA_BASE_URL=http://ollama:11434
+# Ollama endpoint — DEPRECATED fallback, configure via Manage → Models instead
+# OLLAMA_API_BASE=http://localhost:11434
 
-# Content processing
-# CHUNK_SIZE=1500
-# CHUNK_OVERLAP=150
+# oMLX (Apple Silicon) — DEPRECATED fallback, configure via Manage → Models instead
+# Port 11435 avoids clashing with SurrealDB on 8000
+# OMLX_API_BASE=http://localhost:11435/v1
+# OMLX_API_KEY=  # optional; only if oMLX was started with --api-key
+
+# Content processing: embedding chunk size, in tokens
+# OPEN_NOTEBOOK_CHUNK_SIZE=400
+# OPEN_NOTEBOOK_CHUNK_OVERLAP=60  # default: 15% of the chunk size
+
+# Remote Crawl4AI server for the crawl4ai URL engine (no local install needed).
+# Crawl4AI Docker >= 0.9.0 requires a bearer token for external connections.
+# CRAWL4AI_API_URL=http://crawl4ai:11235
+# CRAWL4AI_API_TOKEN=your-crawl4ai-token
+
+# Max background tasks the worker runs concurrently (default 5). Set to 1 for
+# sequential processing on single-GPU / local-LLM setups to avoid rate limits.
+# NOTE: consumed at worker launch, not by the app — for `make`/dev-init export
+# it in your shell; for Docker set it under `environment:` in docker-compose.yml.
+# OPEN_NOTEBOOK_WORKER_MAX_TASKS=1
+
+# Maximum upload/request body size in MB (default: 100). Raise this if you
+# need to upload larger audio/video files. A fronting reverse proxy's own
+# limit (e.g. nginx client_max_body_size) still applies and should be raised
+# to match.
+# OPEN_NOTEBOOK_MAX_UPLOAD_SIZE_MB=100
 
 # Security
-# BASIC_AUTH_USERNAME=admin
-# BASIC_AUTH_PASSWORD=secret
+# Password to protect this Open Notebook instance. Supports Docker secrets
+# via OPEN_NOTEBOOK_PASSWORD_FILE. Auth is fully disabled if unset.
+# OPEN_NOTEBOOK_PASSWORD=secret
+
+# Network / Proxy (corporate / firewalled environments)
+# Route outbound HTTP(S) through a proxy. NO_PROXY MUST list the internal
+# SurrealDB hosts (host.docker.internal, surrealdb) — otherwise the DB
+# websocket gets tunnelled through the proxy and the worker/API fail to start
+# with HTTP 403. The app also injects these hosts automatically as a safety
+# net, but keep them here so your config is explicit.
+# HTTP_PROXY=http://proxy.corp.com:8080
+# HTTPS_PROXY=http://proxy.corp.com:8080
+# NO_PROXY=localhost,127.0.0.1,host.docker.internal,surrealdb,.local
 
 # For more configuration options, see:
 # https://github.com/lfnovo/open-notebook/blob/main/docs/5-CONFIGURATION/environment-reference.md

.gitattributes

@@ -0,0 +1,2 @@
+# Ensure shell scripts always use LF so they run in Linux containers (e.g. Docker)
+*.sh text eol=lf

.github/DISCUSSION_TEMPLATE/ideas.yml

@@ -0,0 +1,73 @@
+title: "[Idea]: "
+labels: ["needs-triage"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for helping shape Open Notebook.
+
+        Ideas, feature requests, behavior changes, product/design/architecture proposals, and contribution proposals start here so the community can explore the problem before it becomes an implementation commitment. Reproducible bugs belong in [Issues](https://github.com/lfnovo/open-notebook/issues/new/choose).
+
+        You do not need to know how to implement the idea. Start with the outcome you need and the context behind it.
+
+  - type: textarea
+    id: goal
+    attributes:
+      label: What are you trying to do?
+      description: Describe the goal or workflow in your own words.
+      placeholder: "I am trying to..."
+    validations:
+      required: true
+
+  - type: textarea
+    id: problem
+    attributes:
+      label: What feels difficult or missing today?
+      description: Tell us what happens now and where the current experience falls short.
+      placeholder: "Today I have to... which makes it difficult to..."
+    validations:
+      required: true
+
+  - type: textarea
+    id: outcome
+    attributes:
+      label: What outcome would help?
+      description: Describe what success would look like without needing to prescribe the implementation.
+      placeholder: "A good outcome would let me..."
+    validations:
+      required: true
+
+  - type: textarea
+    id: workaround
+    attributes:
+      label: How do you handle this today?
+      description: Share any workaround, alternative tool, or process you currently use.
+    validations:
+      required: false
+
+  - type: textarea
+    id: context
+    attributes:
+      label: Additional context, examples, or possible directions
+      description: Screenshots, references, constraints, and implementation ideas are welcome, but optional.
+    validations:
+      required: false
+
+  - type: checkboxes
+    id: participation
+    attributes:
+      label: How would you like to participate?
+      description: This is not a commitment; it helps us know whom to involve if the idea moves forward.
+      options:
+        - label: I can add use cases or join the product/design discussion.
+        - label: I can help explore the technical approach.
+        - label: I can help implement an approved work item.
+        - label: I can test or validate a proposed solution.
+
+  - type: checkboxes
+    id: searched
+    attributes:
+      label: Before posting
+      options:
+        - label: I searched existing Discussions and Issues for related proposals.
+          required: true

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -8,7 +8,18 @@ body:
       value: |
         Thanks for reporting a bug! Please fill out the information below to help us understand and fix the issue.
 
-        **Note**: If you're having installation or setup issues, please use the "Installation Issue" template instead.
+        Issues are for **reproducible bugs** and maintainer-created work items. Feature requests, ideas, behavior changes, product/design/architecture proposals, and contribution proposals belong in [Discussions](https://github.com/lfnovo/open-notebook/discussions/new?category=ideas).
+
+        **Installation or setup problem?** Use the "Installation Issue" template instead.
+
+  - type: checkboxes
+    id: bug-confirmation
+    attributes:
+      label: Is this a reproducible bug?
+      description: Ideas and feature requests belong in Discussions and may be converted there if opened as Issues.
+      options:
+        - label: I confirm this is a reproducible bug, not a feature request, idea, question, or contribution proposal.
+          required: true
 
   - type: textarea
     id: what-happened

.github/ISSUE_TEMPLATE/config.yml

@@ -1,8 +1,14 @@
 blank_issues_enabled: false
 contact_links:
+  - name: 💡 Feature Requests and Ideas
+    url: https://github.com/lfnovo/open-notebook/discussions/new?category=ideas
+    about: Discuss features, product direction, design, architecture, and contribution proposals before they become work items
+  - name: ❓ Questions and Community Support
+    url: https://github.com/lfnovo/open-notebook/discussions/new?category=q-a
+    about: Ask questions and get help from maintainers and the community
   - name: 💬 Discord Community
     url: https://discord.gg/37XJPXfz2w
-    about: Get help from the community and share ideas
+    about: Get real-time help and talk with the community
   - name: 🤖 Installation Assistant (ChatGPT)
     url: https://chatgpt.com/g/g-68776e2765b48191bd1bae3f30212631-open-notebook-installation-assistant
     about: CustomGPT that knows all our docs. Really useful. Try it.

.github/ISSUE_TEMPLATE/feature_request.yml

@@ -1,65 +0,0 @@
-name: ✨ Feature Suggestion
-description: Suggest a new feature or improvement for Open Notebook
-title: "[Feature]: "
-labels: ["enhancement", "needs-triage"]
-body:
-  - type: markdown
-    attributes:
-      value: |
-        Thanks for taking the time to suggest a feature! Your ideas help make Open Notebook better for everyone.
-
-  - type: textarea
-    id: feature-description
-    attributes:
-      label: Feature Description
-      description: What feature would you like to see added or improved?
-      placeholder: "I would like to be able to..."
-    validations:
-      required: true
-
-  - type: textarea
-    id: why-helpful
-    attributes:
-      label: Why would this be helpful?
-      description: Explain how this feature would benefit you and other users
-      placeholder: "This would help because..."
-    validations:
-      required: true
-
-  - type: textarea
-    id: proposed-solution
-    attributes:
-      label: Proposed Solution (Optional)
-      description: If you have ideas on how to implement this feature, please share them
-      placeholder: "This could be implemented by..."
-    validations:
-      required: false
-
-  - type: textarea
-    id: additional-context
-    attributes:
-      label: Additional Context
-      description: Any other context, screenshots, or examples that might be helpful
-      placeholder: "For example, other tools do this by..."
-    validations:
-      required: false
-
-  - type: checkboxes
-    id: willing-to-contribute
-    attributes:
-      label: Contribution
-      description: Would you like to work on implementing this feature?
-      options:
-        - label: I am a developer and would like to work on implementing this feature (pending maintainer approval)
-          required: false
-
-  - type: markdown
-    attributes:
-      value: |
-        ---
-        **Next Steps:**
-        1. A maintainer will review your feature request
-        2. If approved and you checked the box above, the issue will be assigned to you
-        3. Please wait for assignment before starting development
-        4. See our [Contributing Guide](https://github.com/lfnovo/open-notebook/blob/main/CONTRIBUTING.md) for more details
-

.github/RELEASE_PROCESS.md

@@ -0,0 +1,200 @@
+# Release Process
+
+Open Notebook uses a flow-driven release process. Work moves from `ready`
+issues into pull requests, pull requests merge to `main`, and maintainers cut a
+version when the branch has enough validated change to ship.
+
+This document covers both the **mechanics** (how to cut, build and publish) and
+the **confidence process** (how we know a release is good before users get it).
+It was redesigned during the v1.11.0 release ([ADR-005](../docs/7-DEVELOPMENT/decisions/ADR-005-release-confidence-process.md)).
+
+## Release Model
+
+- Patch releases ship backwards-compatible fixes.
+- Minor releases ship backwards-compatible features and improvements.
+- Major releases are planned with a milestone when they include breaking
+  changes or migrations that need user coordination.
+- Use the `in-dev-build` label for changes available in development images and
+  `released` for shipped work. (The `released` label was recreated during
+  v1.12.0 — it had been dropped from the curated label taxonomy while this
+  document still required it. If a label this document references is missing,
+  recreate it rather than skipping the step.)
+
+## Normal Flow
+
+1. Triage issues into `ready` once the scope and design are clear.
+2. Implement each change in a focused pull request linked to the approved issue.
+3. Merge the pull request after review and required checks pass.
+4. Let the development build publish the `v1-dev` image from `main`.
+5. Cut a stable release when `main` has a coherent set of changes ready for
+   users — following the confidence process below.
+
+## The Confidence Process
+
+Releases keep getting bigger; ad-hoc verification does not scale. Before
+cutting, run this sequence:
+
+### 0. Changelog audit
+
+Diff `git log <last-tag>..main` against the `[Unreleased]` section of the
+CHANGELOG. Every merged PR must be represented (entries reference the issue
+number when one exists, the PR number otherwise). The changelog is the input
+for both the test plan and the release notes — close the gaps first, via PR.
+
+### 1. Risk-based test matrix
+
+Build a matrix from the actual release diff: each change → what it can break
+and for whom → which bucket tests it. Pay special attention to
+**"does the protection break legitimate use?"** for security changes (e.g. an
+SSRF guard vs. self-hosted Ollama on localhost) and to anything a reverse
+proxy, an upgrade, or a big upload would exercise.
+
+Buckets:
+
+- **A — automated, high confidence, run now**: full backend suite, frontend
+  lint/tests/production build, the smoke-e2e agent (full API happy path + UI
+  verification), targeted regression probes for the release's specific risks,
+  dependency audit.
+- **B — automatable with investment**: decide per item whether to build the
+  muscle now (it compounds: the image gate below started as a bucket-B item)
+  or verify manually this once.
+- **C — needs the release owner**: real provider credentials, real TTS podcast
+  generation, visual/UX judgment, and the final check of the pushed image.
+
+### 2. The image gate — test the artifact, not the repo
+
+A green suite on `main` is not a working image. Run:
+
+```bash
+make docker-build-local          # builds <version> + local tags
+make release-test TAG=<new> OLD_TAG=<previous>
+```
+
+This runs two scenarios against real containers (`scripts/release-test/`):
+
+- **Fresh install**: empty DB → migrations on boot → in-image worker processes
+  a source → API/frontend/nginx-proxied checks.
+- **Upgrade**: boot the *published* previous image, seed data, swap to the new
+  image on the same volume → migrations apply, data survives.
+
+Caveat: `docker-build-local` tags with the current `pyproject.toml` version —
+`docker pull` the genuine previous tag before the upgrade test so you are not
+comparing the new build against itself.
+
+### 3. Fix loop with a re-test policy
+
+Findings become focused PRs through the normal review flow. After each merge:
+the cheap suite always re-runs; smoke/image gates re-run only if the fix
+touches what they cover; manual verification is not repeated unless the fix
+touches what was manually verified. Pre-existing bugs found along the way that
+are not release regressions become backlog issues instead of scope creep.
+
+## Cutting A Stable Release
+
+1. Confirm `main` is green and the confidence process above has run.
+2. Open the **cut PR**: bump `pyproject.toml`, date the `[Unreleased]` section
+   as `[<version>] - <date>`.
+3. After merge: `make tag`.
+4. Build and push version images **via CI** (it holds the registry
+   credentials): trigger the *Build and Release* workflow with
+   `push_latest=false`. Local `make docker-push` also works but requires
+   `docker login` on both registries.
+5. **Verify the pushed image** (bucket C, final gate): run it locally with
+   `make release-stack TAG=<version> [DUMP=<dev-data-dump>]` — a browsable,
+   isolated stack, optionally with a copy of real data — and walk the core
+   flows in the browser.
+6. Publish the GitHub release. A non-prerelease publication triggers the
+   workflow again and pushes the `v1-latest` tags automatically.
+7. Verify the `v1-latest` manifests on Docker Hub and GHCR (both arches, both
+   variants), and mark shipped issues with `released`.
+
+## Communication
+
+Release notes follow this structure (see v1.11.0 as the reference):
+
+1. One-line verdict + upgrade recommendation.
+2. Sections: Security, Features, Performance, Notable fixes.
+3. **Behavior changes for self-hosters** — anything that can require a config
+   tweak on upgrade gets an explicit callout.
+4. **Thanks** — credit every contributor by handle with what they shipped
+   (collect via `git log <last-tag>..<tag>` + `gh pr view` for handles), plus
+   the issue reporters collectively. Never skip this section.
+
+Announce on Discord after `v1-latest` is live.
+
+## Retro
+
+Close every release by asking: what should improve in this process? Apply the
+accepted improvements immediately — update this document, the scripts under
+`scripts/release-test/`, and the decision log while the context is fresh.
+
+## Docker Image Publishing (reference)
+
+| Command | What it does | Updates latest? |
+|---------|--------------|-----------------|
+| `make docker-build-local` | Build for current platform only (tags `<version>` + `local`) | No registry push |
+| CI *Build and Release* (`push_latest=false`) | Push version tags via CI credentials | ❌ No |
+| GitHub release published (non-prerelease) | CI pushes version + `v1-latest` | ✅ Yes |
+| `make docker-push` / `docker-push-latest` | Local equivalents (need `docker login`) | ❌ / ✅ |
+| `make tag` | Create and push a git tag matching `pyproject.toml` | — |
+
+- **Platforms:** `linux/amd64`, `linux/arm64`
+- **Registries:** Docker Hub + GitHub Container Registry
+- **Image variants:** regular + single-container (`-single`). Both are built
+  from the same `Dockerfile`: regular is the default/`runtime` target, single
+  is `--target single`
+- **Version source:** `pyproject.toml`
+- Build issues: `docker builder prune`, then `make docker-buildx-reset`
+
+## Known Gotchas
+
+- **Never leave the version bump uncommitted on a branch.** Editing
+  `pyproject.toml` / `CHANGELOG.md` for the cut and then switching branches
+  carries those unstaged changes along, and the next `git add -A` sweeps the
+  bump into an unrelated fix PR — the version silently ships inside a `fix(...)`
+  commit. The cut is always the **last** step, on its own branch, committed
+  immediately; if you must build an image early (which needs the bumped
+  version), do it on a throwaway branch and `git stash`/discard before moving
+  on (v1.14.0 lesson).
+- **A fix that lands after the cut requires a full re-cut, not a tag nudge.**
+  If the release was tagged but not yet published (no GitHub release, no
+  `v1-latest`) and a blocker is found in bucket C, the tag must move to the new
+  commit AND the version images must be rebuilt — a stale tag or stale registry
+  image will otherwise be what publication promotes to `v1-latest`. The exact
+  sequence is in `.maintainer/release/runbook.md` → "Re-cut after a post-tag fix" (v1.14.0 lesson).
+- **RC stack on non-default ports needs `API_URL`** or the browser talks to
+  `host:5055` — on a dev machine that is the development API (data crossover).
+  `rc-stack.sh` sets it; remember this for any custom setup.
+- **Containerized app + host services**: credentials pointing at local
+  services (Ollama, LM Studio) need `http://host.docker.internal:<port>`.
+- **SurrealDB import**: `OVERWRITE` goes after the type keyword
+  (`DEFINE FIELD OVERWRITE …`), and the exporter can leak a log line into the
+  dump — `rc-stack.sh` handles both.
+- **Multiple local SurrealDB instances**: check which one the dev `.env`
+  actually points at (`SURREAL_URL`) before exporting data.
+- **Dev-machine ports may belong to other projects**: check who owns
+  3000/5055/8000 (`lsof -nP -iTCP:<port> -sTCP:LISTEN` + the process cwd)
+  before starting or killing anything. The frontend runs fine on an alternate
+  port for smoke testing (`PORT=3001 npm run dev`) — pass the URL to the
+  smoke agent.
+- **Manual error-path checklist items must be validated against the code
+  first**: some "missing configuration" scenarios are deliberate fallbacks,
+  not errors (e.g. transformation and tools defaults fall back to the chat
+  default). Confirm the expected behavior in the provisioning code before
+  putting "should show an error" on the bucket-C checklist.
+- **The test suite runs against the live dev database** when a developer
+  `.env` is loaded. During bucket A, snapshot record counts per table before
+  and after the suite (e.g. credentials count) — a diff means a test is
+  leaking writes (this caught 48 leaked `Test` credentials in v1.12.0).
+- **A local `docker-build-local` tag shadows the pushed image.** Both are
+  `lfnovo/open_notebook:<ver>`, so Phase 6 could verify your own local build
+  instead of the registry artifact. `rc-stack.sh up` now `docker pull`s the tag
+  by default; if you boot the image any other way, pull first (v1.13.0 lesson).
+- **Judge opt-in runtime gating on a clean image, not the dev venv.** A dev
+  venv may have `crawl4ai`/`docling` installed out-of-band (not via the opt-in
+  flag), so `GET /api/capabilities` reports them available and the UI enables
+  the engines — which does NOT reflect the lean default image. Verify the
+  gating on the RC stack (fresh pushed image) where the runtimes are genuinely
+  absent until enabled. To exercise the real install path with your data, use
+  `make release-stack TAG=<ver> DUMP=<dump>` plus `rc-stack.sh ... --with-runtimes`
+  (sets `OPEN_NOTEBOOK_ENABLE_DOCLING`/`_CRAWL4AI`; first boot is slow) (v1.13.0 lesson).

.github/dependabot.yml

@@ -0,0 +1,81 @@
+version: 2
+updates:
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+    cooldown:
+      default-days: 7
+    groups:
+      all-actions:
+        patterns:
+          - "*"
+      actions-security:
+        applies-to: security-updates
+        patterns:
+          - "*"
+
+  - package-ecosystem: "uv"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+    cooldown:
+      default-days: 7
+    groups:
+      all-python:
+        patterns:
+          - "*"
+      python-security:
+        applies-to: security-updates
+        patterns:
+          - "*"
+
+  - package-ecosystem: "npm"
+    directory: "/frontend"
+    schedule:
+      interval: "weekly"
+    cooldown:
+      default-days: 7
+    groups:
+      # Minor and patch updates travel together; majors arrive as their own PRs
+      # so one breaking package can't block the rest (#1371, #1410 failed whole).
+      all-npm:
+        patterns:
+          - "*"
+        update-types:
+          - "minor"
+          - "patch"
+      npm-security:
+        applies-to: security-updates
+        patterns:
+          - "*"
+    ignore:
+      # The direct katex dependency only provides katex.min.css; rendering is
+      # done by rehype-katex with its own katex (^0.16). The stylesheet must
+      # match the renderer's version, so katex moves when rehype-katex does.
+      - dependency-name: "katex"
+        versions: [">=0.17.0"]
+
+  - package-ecosystem: "docker"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+    cooldown:
+      default-days: 7
+    ignore:
+      # Base-image majors are deliberate migrations, not routine bumps:
+      # Node moves between LTS lines on purpose (#1381, #1406), and a SurrealDB
+      # major changes the on-disk format of the -single image's database — v3
+      # is tracked in #378 and needs the upgrade gate (make release-test).
+      - dependency-name: "node"
+        update-types: ["version-update:semver-major"]
+      - dependency-name: "surrealdb/surrealdb"
+        update-types: ["version-update:semver-major"]
+    groups:
+      all-docker:
+        patterns:
+          - "*"
+      docker-security:
+        applies-to: security-updates
+        patterns:
+          - "*"

.github/pull_request_template.md

@@ -4,9 +4,12 @@
 
 ## Related Issue
 
-<!-- This PR should be linked to an approved issue. If not, please create an issue first. -->
+<!-- Non-trivial PRs (features, architecture changes) must link an approved Issue.
+     Small obvious fixes (typo, docs, tiny bug) don't need one — write "N/A (small fix)" below.
+     Sizeable change without an approved Issue? Mark this PR as draft. Start a Discussion for an
+     idea/feature or open an Issue for a reproducible bug, then wait for the work to be approved. -->
 
-Fixes #<!-- issue number -->
+Fixes #<!-- issue number, or "N/A (small fix)" -->
 
 ## Type of Change
 
@@ -27,7 +30,7 @@ Fixes #<!-- issue number -->
 - [ ] Tested locally with Docker
 - [ ] Tested locally with development setup
 - [ ] Added new unit tests
-- [ ] Existing tests pass (`uv run pytest`)
+- [ ] Existing tests pass (`uv run pytest tests/`, and `npm run test` in `frontend/` if the frontend changed)
 - [ ] Manual testing performed (describe below)
 
 **Test Details:**
@@ -37,12 +40,12 @@ Fixes #<!-- issue number -->
 
 <!-- This section helps ensure your PR aligns with our project vision -->
 
-**Which design principles does this PR support?** (See [DESIGN_PRINCIPLES.md](../DESIGN_PRINCIPLES.md))
+**Which design principles does this PR support?** (See [VISION.md](https://github.com/lfnovo/open-notebook/blob/main/VISION.md))
 
 - [ ] Privacy First
 - [ ] Simplicity Over Features
 - [ ] API-First Architecture
-- [ ] Multi-Provider Flexibility
+- [ ] Provider-Agnostic Core
 - [ ] Extensibility Through Standards
 - [ ] Async-First for Performance
 
@@ -65,24 +68,26 @@ Fixes #<!-- issue number -->
 ### Testing
 - [ ] I have added tests that prove my fix is effective or that my feature works
 - [ ] New and existing unit tests pass locally with my changes
-- [ ] I ran linting: `make ruff` or `ruff check . --fix`
-- [ ] I ran type checking: `make lint` or `uv run python -m mypy .`
+- [ ] I ran linting: `uv run ruff check .`
+- [ ] I ran formatting: `uv run ruff format .` (CI fails if `ruff format --check .` finds changes)
+- [ ] I ran type checking: `uv run python -m mypy .`
+- [ ] Frontend changes: `npm run lint`, `npm run test` and `npm run build` pass (run inside `frontend/`)
 
 ### Documentation
 - [ ] I have updated the relevant documentation in `/docs` (if applicable)
 - [ ] I have added/updated docstrings for new/modified functions
-- [ ] I have updated the API documentation (if API changes were made)
+- [ ] I have added a CHANGELOG entry under `[Unreleased]` in the section for its type (if user-visible)
 - [ ] I have added comments to complex logic
 
 ### Database Changes
-- [ ] I have created migration scripts for any database schema changes (in `/migrations`)
+- [ ] I have created migration scripts for any database schema changes (in `open_notebook/database/migrations/`, registered in `async_migrate.py`)
 - [ ] Migration includes both up and down scripts
 - [ ] Migration has been tested locally
 
 ### Breaking Changes
 - [ ] This PR includes breaking changes
 - [ ] I have documented the migration path for users
-- [ ] I have updated MIGRATION.md (if applicable)
+- [ ] The CHANGELOG entry explains what users must do
 
 ## Screenshots (if applicable)
 
@@ -96,11 +101,11 @@ Fixes #<!-- issue number -->
 
 Before submitting, please verify:
 
-- [ ] I have read [CONTRIBUTING.md](../CONTRIBUTING.md)
-- [ ] I have read [DESIGN_PRINCIPLES.md](../DESIGN_PRINCIPLES.md)
-- [ ] This PR addresses an approved issue that was assigned to me
+- [ ] I have read [CONTRIBUTING.md](https://github.com/lfnovo/open-notebook/blob/main/docs/7-DEVELOPMENT/contributing.md)
+- [ ] I have read [VISION.md](https://github.com/lfnovo/open-notebook/blob/main/VISION.md)
+- [ ] This PR addresses an approved Issue assigned to me, **or** it's a small obvious fix (typo, docs, tiny bug) that doesn't need one — ideas and features begin in Discussions; reproducible bugs begin in Issues
 - [ ] I have not included unrelated changes in this PR
-- [ ] My PR title follows conventional commits format (e.g., "feat: add user authentication")
+- [ ] My PR title follows the Conventional Commits format (e.g., "fix(podcasts): …", "feat: …")
 
 ---
 

.github/workflows/build-and-release.yml

@@ -27,7 +27,7 @@ jobs:
       has_dockerhub_secrets: ${{ steps.check.outputs.has_dockerhub_secrets }}
     steps:
       - name: Checkout
-        uses: actions/checkout@v4
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
 
       - name: Extract version from pyproject.toml
         id: version
@@ -55,7 +55,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v4
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
 
       - name: Free up disk space
         run: |
@@ -67,10 +67,10 @@ jobs:
           df -h
 
       - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v3
+        uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4
 
       - name: Login to GitHub Container Registry
-        uses: docker/login-action@v3
+        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
         with:
           registry: ghcr.io
           username: ${{ github.actor }}
@@ -78,13 +78,13 @@ jobs:
 
       - name: Login to Docker Hub
         if: needs.extract-version.outputs.has_dockerhub_secrets == 'true'
-        uses: docker/login-action@v3
+        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
         with:
           username: ${{ secrets.DOCKER_USERNAME }}
           password: ${{ secrets.DOCKER_PASSWORD }}
 
       - name: Cache Docker layers
-        uses: actions/cache@v3
+        uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: /tmp/.buildx-cache
           key: ${{ runner.os }}-buildx-regular-${{ github.sha }}
@@ -126,10 +126,11 @@ jobs:
           echo "Generated tags: ${TAGS}"
 
       - name: Build and push regular image
-        uses: docker/build-push-action@v5
+        uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7
         with:
           context: .
           file: ./Dockerfile
+          target: runtime
           platforms: linux/amd64,linux/arm64
           push: true
           tags: ${{ steps.tags-regular.outputs.tags }}
@@ -146,7 +147,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v4
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
 
       - name: Free up disk space
         run: |
@@ -158,10 +159,10 @@ jobs:
           df -h
 
       - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v3
+        uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4
 
       - name: Login to GitHub Container Registry
-        uses: docker/login-action@v3
+        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
         with:
           registry: ghcr.io
           username: ${{ github.actor }}
@@ -169,13 +170,13 @@ jobs:
 
       - name: Login to Docker Hub
         if: needs.extract-version.outputs.has_dockerhub_secrets == 'true'
-        uses: docker/login-action@v3
+        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
         with:
           username: ${{ secrets.DOCKER_USERNAME }}
           password: ${{ secrets.DOCKER_PASSWORD }}
 
       - name: Cache Docker layers
-        uses: actions/cache@v3
+        uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: /tmp/.buildx-cache-single
           key: ${{ runner.os }}-buildx-single-${{ github.sha }}
@@ -217,10 +218,11 @@ jobs:
           echo "Generated tags: ${TAGS}"
 
       - name: Build and push single-container image
-        uses: docker/build-push-action@v5
+        uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7
         with:
           context: .
-          file: ./Dockerfile.single
+          file: ./Dockerfile
+          target: single
           platforms: linux/amd64,linux/arm64
           push: true
           tags: ${{ steps.tags-single.outputs.tags }}

.github/workflows/build-dev.yml

@@ -39,7 +39,7 @@ jobs:
       is_push_to_main: ${{ steps.check.outputs.is_push_to_main }}
     steps:
       - name: Checkout
-        uses: actions/checkout@v4
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
 
       - name: Extract version from pyproject.toml
         id: version
@@ -72,12 +72,50 @@ jobs:
             echo "This is a PR or manual run - test build only"
           fi
 
+  # Detect whether this PR touches anything that ends up in the Docker image.
+  # build-regular is a required status check, so it must always REPORT — the
+  # expensive build steps below are gated on this output instead of filtering
+  # the workflow trigger (a path-filtered required check never reports and
+  # deadlocks merge, even for admins). On push to main the build always runs.
+  changes:
+    runs-on: ubuntu-latest
+    permissions:
+      pull-requests: read
+    outputs:
+      image: ${{ steps.filter.outputs.image }}
+    steps:
+      - name: Checkout
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+
+      - name: Detect image-affecting changes
+        uses: dorny/paths-filter@ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d # v4.0.3
+        id: filter
+        with:
+          filters: |
+            image:
+              - 'Dockerfile*'
+              - 'pyproject.toml'
+              - 'uv.lock'
+              - 'open_notebook/**'
+              - 'api/**'
+              - 'commands/**'
+              - 'frontend/**'
+              - '.github/workflows/build-dev.yml'
+
+  # The job always runs (so the required `build-regular` check always reports),
+  # but every build step is gated on: push to main OR the PR touched an
+  # image-affecting path (needs.changes.outputs.image). Uses needs.* directly in
+  # each step `if:` (fully supported) rather than a job-level env indirection.
   build-regular:
-    needs: extract-version
+    needs: [extract-version, changes]
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v4
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+
+      - name: Skip notice
+        if: needs.extract-version.outputs.is_push_to_main != 'true' && needs.changes.outputs.image != 'true'
+        run: echo "No image-affecting paths changed — skipping the Docker build. Reporting success."
 
       - name: Free up disk space
         if: needs.extract-version.outputs.is_push_to_main == 'true'
@@ -90,11 +128,12 @@ jobs:
           df -h
 
       - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v3
+        if: needs.extract-version.outputs.is_push_to_main == 'true' || needs.changes.outputs.image == 'true'
+        uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4
 
       - name: Login to GitHub Container Registry
         if: needs.extract-version.outputs.is_push_to_main == 'true'
-        uses: docker/login-action@v3
+        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
         with:
           registry: ghcr.io
           username: ${{ github.actor }}
@@ -102,13 +141,14 @@ jobs:
 
       - name: Login to Docker Hub
         if: needs.extract-version.outputs.is_push_to_main == 'true' && needs.extract-version.outputs.has_dockerhub_secrets == 'true'
-        uses: docker/login-action@v3
+        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
         with:
           username: ${{ secrets.DOCKER_USERNAME }}
           password: ${{ secrets.DOCKER_PASSWORD }}
 
       - name: Cache Docker layers
-        uses: actions/cache@v3
+        if: needs.extract-version.outputs.is_push_to_main == 'true' || needs.changes.outputs.image == 'true'
+        uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: /tmp/.buildx-cache-dev
           key: ${{ runner.os }}-buildx-dev-${{ github.sha }}
@@ -117,6 +157,7 @@ jobs:
 
       - name: Prepare Docker tags
         id: tags
+        if: needs.extract-version.outputs.is_push_to_main == 'true' || needs.changes.outputs.image == 'true'
         run: |
           if [[ "${{ needs.extract-version.outputs.is_push_to_main }}" == "true" ]]; then
             # Push to main: build and push v1-dev tags
@@ -135,10 +176,12 @@ jobs:
           fi
 
       - name: Build and push regular image
-        uses: docker/build-push-action@v5
+        if: needs.extract-version.outputs.is_push_to_main == 'true' || needs.changes.outputs.image == 'true'
+        uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7
         with:
           context: .
           file: ./Dockerfile
+          target: runtime
           platforms: ${{ steps.tags.outputs.platforms }}
           push: ${{ steps.tags.outputs.push }}
           tags: ${{ steps.tags.outputs.tags }}
@@ -146,6 +189,7 @@ jobs:
           cache-to: type=local,dest=/tmp/.buildx-cache-dev-new,mode=max
 
       - name: Move cache
+        if: needs.extract-version.outputs.is_push_to_main == 'true' || needs.changes.outputs.image == 'true'
         run: |
           rm -rf /tmp/.buildx-cache-dev
           mv /tmp/.buildx-cache-dev-new /tmp/.buildx-cache-dev
@@ -157,7 +201,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v4
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
 
       - name: Free up disk space
         run: |
@@ -169,10 +213,10 @@ jobs:
           df -h
 
       - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v3
+        uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4
 
       - name: Login to GitHub Container Registry
-        uses: docker/login-action@v3
+        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
         with:
           registry: ghcr.io
           username: ${{ github.actor }}
@@ -180,13 +224,13 @@ jobs:
 
       - name: Login to Docker Hub
         if: needs.extract-version.outputs.has_dockerhub_secrets == 'true'
-        uses: docker/login-action@v3
+        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
         with:
           username: ${{ secrets.DOCKER_USERNAME }}
           password: ${{ secrets.DOCKER_PASSWORD }}
 
       - name: Cache Docker layers
-        uses: actions/cache@v3
+        uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: /tmp/.buildx-cache-dev-single
           key: ${{ runner.os }}-buildx-dev-single-${{ github.sha }}
@@ -203,10 +247,11 @@ jobs:
           echo "tags=${TAGS}" >> $GITHUB_OUTPUT
 
       - name: Build and push single-container image
-        uses: docker/build-push-action@v5
+        uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7
         with:
           context: .
-          file: ./Dockerfile.single
+          file: ./Dockerfile
+          target: single
           platforms: linux/amd64,linux/arm64
           push: true
           tags: ${{ steps.tags.outputs.tags }}

.github/workflows/claude-code-review.yml

@@ -1,48 +0,0 @@
-name: Claude Code Review
-
-on:
-  pull_request:
-    types: [opened, synchronize, ready_for_review, reopened]
-  pull_request_target:
-    types: [opened, synchronize, ready_for_review, reopened]
-    # Optional: Only run on specific file changes
-    # paths:
-    #   - "src/**/*.ts"
-    #   - "src/**/*.tsx"
-    #   - "src/**/*.js"
-    #   - "src/**/*.jsx"
-
-jobs:
-  claude-review:
-    # Run for fork PRs (via pull_request_target) OR same-repo PRs (via pull_request), but not both
-    if: |
-      (github.event_name == 'pull_request_target' && github.event.pull_request.head.repo.full_name != github.repository) ||
-      (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository)
-
-    runs-on: ubuntu-latest
-    permissions:
-      contents: read
-      pull-requests: write
-      issues: write
-      id-token: write
-
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@v4
-        with:
-          ref: ${{ github.event.pull_request.head.sha }}
-          fetch-depth: 1
-          persist-credentials: false
-
-      - name: Run Claude Code Review
-        id: claude-review
-        uses: anthropics/claude-code-action@v1
-        with:
-          github_token: ${{ secrets.GITHUB_TOKEN }}
-          claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
-          plugin_marketplaces: 'https://github.com/anthropics/claude-code.git'
-          plugins: 'code-review@claude-code-plugins'
-          prompt: '/code-review:code-review ${{ github.repository }}/pull/${{ github.event.pull_request.number }}'
-          # See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md
-          # or https://code.claude.com/docs/en/cli-reference for available options
-

.github/workflows/claude.yml

@@ -1,51 +0,0 @@
-name: Claude Code
-
-on:
-  issue_comment:
-    types: [created]
-  pull_request_review_comment:
-    types: [created]
-  issues:
-    types: [opened, assigned]
-  pull_request_review:
-    types: [submitted]
-
-jobs:
-  claude:
-    if: |
-      (github.event_name == 'issue_comment' && contains(github.event.comment.body, '@claude')) ||
-      (github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude')) ||
-      (github.event_name == 'pull_request_review' && contains(github.event.review.body, '@claude')) ||
-      (github.event_name == 'issues' && (contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')))
-    runs-on: ubuntu-latest
-    permissions:
-      contents: read
-      pull-requests: write
-      issues: write
-      id-token: write
-      actions: read # Required for Claude to read CI results on PRs
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@v4
-        with:
-          fetch-depth: 1
-
-      - name: Run Claude Code
-        id: claude
-        uses: anthropics/claude-code-action@v1
-        with:
-          github_token: ${{ secrets.GITHUB_TOKEN }}
-          claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
-
-          # This is an optional setting that allows Claude to read CI results on PRs
-          additional_permissions: |
-            actions: read
-
-          # Optional: Give a custom prompt to Claude. If this is not specified, Claude will perform the instructions specified in the comment that tagged it.
-          # prompt: 'Update the pull request description to include a summary of changes.'
-
-          # Optional: Add claude_args to customize behavior and configuration
-          # See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md
-          # or https://code.claude.com/docs/en/cli-reference for available options
-          # claude_args: '--allowed-tools Bash(gh pr:*)'
-

.github/workflows/docs-links.yml

@@ -0,0 +1,19 @@
+name: Docs Link Check
+
+on:
+  pull_request:
+    paths:
+      - "**/*.md"
+      - "scripts/check_md_links.py"
+      - ".github/workflows/docs-links.yml"
+
+permissions:
+  contents: read
+
+jobs:
+  check-links:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+      - name: Check relative markdown links
+        run: python3 scripts/check_md_links.py

.github/workflows/test.yml

@@ -19,10 +19,10 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v4
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
 
       - name: Set up uv
-        uses: astral-sh/setup-uv@v4
+        uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
         with:
           enable-cache: true
 
@@ -32,8 +32,59 @@ jobs:
       - name: Install dependencies
         run: uv sync
 
-      - name: Run tests
-        run: uv run pytest tests/ -v
+      - name: Run tests with coverage
+        run: uv run pytest tests/ -v --cov=open_notebook --cov=api --cov-report=term-missing --cov-report=xml
+
+      - name: Upload coverage artifact
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: backend-coverage
+          path: coverage.xml
+
+  backend-lint:
+    name: Backend Lint
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+
+      - name: Set up uv
+        uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
+        with:
+          enable-cache: true
+
+      - name: Set up Python
+        run: uv python install
+
+      - name: Install dependencies
+        run: uv sync
+
+      - name: Run ruff
+        run: uv run ruff check .
+
+      - name: Check formatting
+        run: uv run ruff format --check .
+
+  backend-typecheck:
+    name: Backend Typecheck
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+
+      - name: Set up uv
+        uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
+        with:
+          enable-cache: true
+
+      - name: Set up Python
+        run: uv python install
+
+      - name: Install dependencies
+        run: uv sync
+
+      - name: Run mypy
+        run: uv run python -m mypy .
 
   frontend:
     name: Frontend Tests
@@ -43,10 +94,10 @@ jobs:
         working-directory: frontend
     steps:
       - name: Checkout
-        uses: actions/checkout@v4
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
 
       - name: Set up Node.js
-        uses: actions/setup-node@v4
+        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
           node-version: 22
           cache: npm
@@ -55,5 +106,57 @@ jobs:
       - name: Install dependencies
         run: npm ci
 
-      - name: Run tests
-        run: npm test
+      - name: Run tests with coverage
+        run: npm run test:coverage
+
+      - name: Upload coverage artifact
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: frontend-coverage
+          path: frontend/coverage/
+
+  frontend-lint:
+    name: Frontend Lint
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: frontend
+    steps:
+      - name: Checkout
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+
+      - name: Set up Node.js
+        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+        with:
+          node-version: 22
+          cache: npm
+          cache-dependency-path: frontend/package-lock.json
+
+      - name: Install dependencies
+        run: npm ci
+
+      - name: Run ESLint
+        run: npm run lint
+
+  frontend-build:
+    name: Frontend Build
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: frontend
+    steps:
+      - name: Checkout
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+
+      - name: Set up Node.js
+        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+        with:
+          node-version: 22
+          cache: npm
+          cache-dependency-path: frontend/package-lock.json
+
+      - name: Install dependencies
+        run: npm ci
+
+      - name: Build
+        run: npm run build

.gitignore

@@ -130,7 +130,10 @@ docs/custom_gpt
 doc_exports/
 
 specs/
-.claude
+# .claude: personal state stays local; skills and agents are shared project tooling
+.claude/*
+!.claude/skills
+!.claude/agents
 .sisyphus
 
 .playwright-mcp/
@@ -141,4 +144,17 @@ specs/
 **/*.local.md
 .harness/
 
-.mcp.json
+.mcp.json
+
+# Local Docker Compose override (auto-merged by `docker compose up`).
+# Use it for host-specific tweaks like re-exposing the SurrealDB port;
+# see docker-compose.override.yml.example.
+docker-compose.override.yml
+
+# Local QA/debug artifacts
+screenshot-*.png
+history.txt
+
+# oss-maintainer local state
+.maintainer/profile.local.toml
+.maintainer/state/

.maintainer/PROFILE.md

@@ -0,0 +1,37 @@
+# Maintainer profile — scope and tone
+
+## What we own
+
+- This repository: the FastAPI backend (`api/`, `open_notebook/`), the surreal-commands
+  worker (`commands/`), the prompt templates (`prompts/`), the Next.js frontend (`frontend/`),
+  the SurrealDB schema and migrations, the Docker images (regular and `-single`) and the
+  `docker-compose` files, the documentation under `docs/`.
+- The product decisions recorded in `VISION.md` and `docs/7-DEVELOPMENT/decisions/`.
+
+## What we do not own
+
+- Upstream libraries with their own repositories: `content-core` (extraction),
+  `esperanto` (provider abstraction), `podcast-creator`, `surreal-commands`,
+  `surreal-basics`. A defect that lives there gets the `upstream` label plus the library's
+  label (`content-core`, `esperanto`, `podcast-creator`); the fix is an upstream issue first,
+  then a bump here.
+- SurrealDB itself, the AI providers' services and their pricing or availability.
+- Community packaging channels (Unraid, Helm charts, Podman recipes, desktop wrappers): valid
+  ideas that wait for a champion.
+- The third-party "Installation Assistant" GPT linked from the issue forms.
+
+## Tone in public text
+
+- Answer first; no filler, no marketing, no "great question". Warmth comes from specificity
+  (acknowledging a working prototype, a good decomposition), never from compliments.
+- Say what was verified in the code, separately from opinion ("I checked the code: …").
+- No promises of features or timelines ("no commitment on timing yet"); constraints come with
+  their reasons and a link to the decision record.
+- A "no" is said in the first paragraph, with the reasons; it never hides behind questions.
+- Everything posted publicly is in English.
+
+## Never cite in public
+
+- `.tmp-context/`, `CLAUDE.local.md`, private maintainer drafts and internal plans.
+- Unpublished vision specifics: "aligns with where the product is heading" is the most that
+  can be said. Public anchors are `VISION.md`, the decision records, open issues and PRs.

.maintainer/README.md

@@ -0,0 +1,26 @@
+# .maintainer
+
+Maintainer profile read by the [oss-maintainer](https://github.com/lfnovo/oss-maintainer)
+plugin for Claude Code and Codex. It holds what the maintenance workflows of this repository
+need and cannot derive. It is not a list of people; see
+[docs/7-DEVELOPMENT/maintainer-guide.md](../docs/7-DEVELOPMENT/maintainer-guide.md).
+
+Each fact has one home: executable commands are documented in the `Makefile` or `AGENTS.md`
+and `profile.toml` invokes them by their canonical form (the same invocations CI runs — a gate
+never mutates the tree, so the validator runs `ruff check .` without `--fix`) rather than
+re-implementing them; `profile.toml` holds the fields the plugin reads; `release/runbook.md`
+holds the release sequence and policy; `.github/RELEASE_PROCESS.md` holds the human release
+process and its known gotchas; `gotchas.md` holds the lessons outside the release. Other files
+link to those instead of repeating them.
+
+| File | Purpose |
+|---|---|
+| `profile.toml` | the fields the plugin reads (schema v1) |
+| `PROFILE.md` | scope, tone, what must never be cited in public |
+| `gotchas.md` | fragile areas and known issues, fed by retros |
+| `triage.md` | triage rules specific to this repository |
+| `release/runbook.md` | the release sequence and policy |
+| `release/test-matrix.md` | the recurring risks, seed of each release's coverage table |
+| `smoke/journey.md` | the product journey the smoke test executes |
+| `profile.local.toml` | gitignored local preferences (upstream checkouts, URLs) |
+| `state/` | gitignored run records and reports |

.maintainer/gotchas.md

@@ -0,0 +1,31 @@
+# Gotchas
+
+Fragile areas and lessons learned. Retros append here; a gotcha that holds up three times
+graduates to the process document or to a test.
+
+Release-specific gotchas (version bump hygiene, re-cut after a post-tag fix, RC stack ports,
+local tags shadowing pushed images, opt-in runtime gating) have their home in
+[`.github/RELEASE_PROCESS.md` → Known Gotchas](../.github/RELEASE_PROCESS.md#known-gotchas);
+do not duplicate them here.
+
+- **The backend suite runs against the live dev database** when a developer `.env` is loaded.
+  Snapshot per-table counts (at least `credential`) before and after; a diff means a test
+  leaks writes (48 leaked credentials caught in v1.12.0).
+- **Search and Ask are global by design**: `POST /api/search` and `/api/search/ask/simple`
+  have no `notebook_id`; Pydantic silently drops unknown fields, so a scoped-looking call
+  is an unscoped one. Scoping to selected notebooks landed in #1331 — check the request
+  models before assuming either behavior.
+- **Provider aliases are mapped in more than one place.** `anthropic_compatible` is mapped to
+  Esperanto's `anthropic` in `open_notebook/ai/models.py`; any resolver that bypasses
+  `ModelManager.get_model()` (podcasts did, #1347) must use the shared helper or it breaks.
+- **Async jobs need the worker.** Podcasts, embeddings and source processing queue forever
+  without `make worker-start`; a "hangs" report with no worker log is usually this.
+- **Containerized app + host services**: credentials pointing at Ollama or LM Studio on the
+  host need `http://host.docker.internal:<port>`, not `localhost`.
+- **Dev-machine ports may belong to other projects**: check who owns 3000/5055/8000 before
+  starting or killing anything; the frontend runs fine on `PORT=3001 npm run dev`.
+- **ADR numbers are assigned by the maintainer at merge, not by contributors.** In
+  September 2026 four open PRs (#1360, #1332, #1367, #1376) each created `ADR-009-*.md`.
+  Ask contributors to name the file `ADR-0XX-<slug>.md`; at merge, take the next free
+  number from `docs/7-DEVELOPMENT/decisions/README.md` and add the index row there (#1360
+  landed without its row).

.maintainer/profile.toml

@@ -0,0 +1,116 @@
+# .maintainer/profile.toml — profile for the oss-maintainer plugin (schema v1)
+# Every path is relative to the repository root. Values marked TODO block the capability
+# that needs them; values starting with "CONFIRM:" are detected proposals awaiting confirmation.
+schema_version = 1
+
+[project]
+name = "Open Notebook"
+repo = "lfnovo/open-notebook"
+artifact = "app-docker"
+commands_doc = "AGENTS.md"                  # canonical commands: AGENTS.md → Commands, Makefile
+
+# Backend validator: the three required CI gates (AGENTS.md → Commands). `ruff check .` without
+# `--fix`, as .github/workflows/test.yml runs it: a gate never mutates the tree.
+[commands.validator]
+run = "uv run pytest tests/ && ruff check . && uv run python -m mypy ."
+cwd = "."
+timeout = "20m"
+
+# Frontend gate (lint, unit tests, production build), as AGENTS.md documents it.
+[commands.frontend]
+run = "npm run lint && npm run test && npm run build"
+cwd = "frontend"
+timeout = "20m"
+
+[contributing]
+conventions = "docs/7-DEVELOPMENT/contributing.md"   # root CONTRIBUTING.md only points here
+
+[release]
+process_doc = ".github/RELEASE_PROCESS.md"  # the human process: model, confidence process, cut, gotchas
+change_delivery = "pr"                      # every repo change goes through a PR; never push to main
+versioning = "semver"
+changelog = "CHANGELOG.md"
+version_files = ["pyproject.toml"]
+lock_command = "uv lock"           # uv.lock records the package's own version (line 2053)
+# First action that can start public distribution: the CI workflow pushes the <version> and
+# <version>-single tags to both registries (public, but v1-latest untouched). Publication of
+# the GitHub release is what promotes v1-latest — see latest_promotion.
+distribution_trigger = "gh workflow run build-and-release.yml --ref main -f push_latest=false"
+publish_workflow = ".github/workflows/build-and-release.yml"
+latest_promotion = "publishing a non-prerelease GitHub release (gh release create v<ver> --latest) re-runs build-and-release.yml, which adds the v1-latest and v1-latest-single tags on both registries; manual equivalent: the same workflow with push_latest=true"
+notes_source = "CHANGELOG.md"
+consumer_surfaces = [
+  "REST API under /api (the UI is one client; integrations are others)",
+  "environment variables and config keys (docs/5-CONFIGURATION/environment-reference.md)",
+  "docker-compose files, image variants (regular, -single) and the v1-latest / v1-dev tags",
+  "SurrealDB schema and migrations (upgrades run them on boot)",
+  "transformation, prompt-template and command extension points",
+]
+
+[release.gates]
+# What each gate means: .maintainer/release/runbook.md
+mandatory = ["validator", "frontend", "image-gate", "owner-checks", "notes-approved", "security-alerts"]
+optional = []
+alerts_policy = "open Dependabot highs are resolved or explicitly accepted by the owner before GO"
+merge_own_prs = "ask-once-per-session"      # ask once per session ("merge when clean?") and honor the answer
+
+[artifacts.docker]
+registries = ["lfnovo/open_notebook", "ghcr.io/lfnovo/open-notebook"]
+variants = ["", "-single"]                  # runtime target and --target single, same Dockerfile
+platforms = ["amd64", "arm64"]
+dev_tag = "v1-dev"                          # rebuilt from main on every push (build-dev.yml)
+gate = "make docker-build-local && make release-test TAG=<ver> OLD_TAG=<prev>"
+identity = "manifest-digest"
+
+[labels]                                    # canonical -> real
+needs_triage = "needs-triage"               # referenced by the maintainer guide and the issue forms; recreate if missing
+needs_design = "needs-design"
+needs_vision = "needs-vision"
+awaiting_demand = "awaiting-demand"         # not part of the curated set; recognised only, never assigned
+ready = "ready"
+released = "released"
+bug = "bug"
+
+[triage]
+preset = "maturity-ladder"
+assignable = ["close", "needs-design", "ready", "needs-info"]
+extra_states = { needs-info = "Waiting on the reporter to confirm or provide more information" }
+rules = ".maintainer/triage.md"
+batch_approval = "one-at-a-time"
+
+[review]
+docs = [
+  "AGENTS.md",
+  "VISION.md",
+  "docs/7-DEVELOPMENT/architecture.md",
+  "docs/7-DEVELOPMENT/design-principles.md",
+  "docs/7-DEVELOPMENT/code-standards.md",
+  "docs/7-DEVELOPMENT/decisions/README.md",
+]
+reviewers = ["cubic"]                       # findings are hypotheses to verify, never verdicts
+
+[discussions]
+batch_approval = "one-at-a-time"
+categories = { ideas = "DIC_kwDONDsQ184CjkD_", feedback-requests = "DIC_kwDONDsQ184DBrfp", q-a = "DIC_kwDONDsQ184CjkD-", general = "DIC_kwDONDsQ184CjkD9", announcements = "DIC_kwDONDsQ184CjkD8" }
+regenerate = "gh api graphql -f query='{ repository(owner: \"lfnovo\", name: \"open-notebook\") { discussionCategories(first: 20) { nodes { id name } } } }'"
+public_anchors = ["VISION.md", "docs/7-DEVELOPMENT/decisions/README.md", "docs/7-DEVELOPMENT/contributing.md"]
+never_cite = [".tmp-context/", "CLAUDE.local.md", "private maintainer drafts", "unpublished vision specifics"]
+graduation = "pull"                         # an Issue is born when someone will build it (ratified 2026-09-02)
+close_on = ["answer", "graduated-work-landed"]
+
+[smoke]
+api_url = "http://localhost:5055"
+frontend_url = "http://localhost:3000"
+journey = ".maintainer/smoke/journey.md"
+mandatory_surfaces = ["api", "ui"]
+report_path = ".maintainer/state/smoke-report.md"
+poll_interval = "5s"
+step_timeout = "5m"
+
+[comms]
+owner_language = "pt-BR"
+public_language = "en"
+agent_attribution = "none"
+
+[channels]
+announce = ["discord"]                      # texts are delivered to the owner, never posted by a skill

.maintainer/release/runbook.md

@@ -0,0 +1,160 @@
+# Release runbook
+
+The sequence and the policy of a release of this repository, in one place. The plugin's
+`release` skill supplies the steps and the gates; `profile.toml` holds the executable
+references (`[commands.validator]`, `[commands.frontend]`, `[artifacts.docker].gate`,
+`[release].distribution_trigger`); this file holds what those fields cannot express: order,
+environment, manual steps, exact command sequences and the reasons behind them. The human
+process and its known gotchas live in [`.github/RELEASE_PROCESS.md`](../../.github/RELEASE_PROCESS.md)
+(designed in [ADR-005](../../docs/7-DEVELOPMENT/decisions/ADR-005-release-confidence-process.md));
+this file does not repeat them.
+
+## Policy
+
+- A release happens in a **single session**, phase by phase, with the owner seeing progress.
+- Every repository change goes through a **PR** (branch → PR → CI + cubic → merge). Never push
+  to `main`. Merging own PRs: ask once per session and honor the answer (`merge_own_prs`).
+- Autonomous: tests, builds, probes, local services, branches, commits, PRs, local image
+  builds, the image gate, the RC stack, and dispatching *Build and Release* with
+  `push_latest=false` (version tags only — the agreed pre-verification push).
+- Explicit in-session GO required: publishing the GitHub release (the point of no return —
+  it promotes `v1-latest`), anything else that moves `v1-latest`, creating issues, mass
+  labeling (`released`), touching the owner's dev data (copies only, never originals).
+- Never: publish a prerelease or release to route around a blocked step; mark a phase
+  complete with failing checks ("GO with known issues is worse than a NO-GO").
+- **Re-test policy after each fix merge**: the cheap suite (validator + frontend) always;
+  smoke-e2e and the image gate only if the fix touches what they cover; the owner's manual
+  checks only if the fix touches what they verified; the final image gate always runs on the
+  exact release artifact.
+- **GO** when: validator and frontend green · image gate green (fresh + upgrade) · owner's
+  checks signed off · release notes approved · no open release-regression findings ·
+  Dependabot highs resolved or explicitly accepted.
+
+## Validate
+
+- Canonical validator and mandatory checks: `[commands.validator]`, `[commands.frontend]`,
+  `[release.gates].mandatory`. Run `uv sync --group dev` first if mypy is missing locally;
+  `npm ci` in `frontend/` if dependencies changed (a stale `node_modules` fakes build failures).
+- Bucket A also runs the **smoke-e2e** skill against the local dev stack (database → api →
+  worker → frontend; see `[smoke]`) and the **dev-DB leak check** (per-table counts before
+  and after the backend suite).
+- Artifact gate: `[artifacts.docker].gate`, built from the candidate commit.
+  `docker-build-local` tags with the current `pyproject.toml` version, so `docker pull` the
+  genuine previous tag before the upgrade scenario.
+- Environment or credentials the checks need (names only, never values):
+  - `.env` with `OPEN_NOTEBOOK_ENCRYPTION_KEY` and a `SURREAL_URL` (check which local
+    SurrealDB it points at), at least one provider credential with chat + embedding defaults
+  - Docker with buildx; `docker login` on both registries only for the local push fallback
+  - `gh` authenticated with `workflow` scope (dispatching *Build and Release*)
+
+## Owner's manual checks
+
+Delivered as a concrete checklist tailored to what the release touched and to the
+credentials the owner has (`GET /api/credentials` tells you). Standing items:
+
+- Provider connection tests with **real credentials** (prioritize providers whose code
+  changed); one discover-models; one chat per main provider.
+- One podcast with real TTS on a dense notebook.
+- Visual/UX tour (~10 min) of every UI change, plus dark-mode sampling.
+- The pushed image on the RC stack (see *Verify from the registry*).
+
+Verify error-path items against the provisioning code before listing them: transformation
+and tools defaults deliberately fall back to the chat default (`open_notebook/ai/models.py`).
+
+## Cut
+
+- The cut is the **last** step, on its own branch, committed immediately: bump
+  `pyproject.toml`, run `[release].lock_command`, date `[Unreleased]` as `[<ver>] - <date>`,
+  open the cut PR. Never carry an uncommitted bump across branches.
+- After merge: `make tag` (creates and pushes `v<ver>` from `pyproject.toml`).
+
+## Publish, only after the GO
+
+- Trigger: `[release].distribution_trigger` — pushes `<ver>` and `<ver>-single` to Docker
+  Hub and GHCR (both arches). Watch it:
+
+```bash
+gh run list --workflow=build-and-release.yml --limit 1     # grab the id
+gh run watch <run-id> --exit-status
+```
+
+- Then verify from the registry (next section) and, on the owner's re-GO, promote:
+
+```bash
+gh release create v<ver> --title "v<ver> — <theme>" --notes-file <notes.md> --latest
+# publication (non-prerelease) re-runs the workflow, which pushes v1-latest and v1-latest-single
+gh run list --workflow=build-and-release.yml --limit 1 && gh run watch <id> --exit-status
+```
+
+- Release notes follow `.github/RELEASE_PROCESS.md` → Communication (verdict, sections,
+  behavior changes for self-hosters, **Thanks** by handle — never skipped). Announcement text
+  for Discord is delivered to the owner after `v1-latest` is live.
+
+## Verify from the registry
+
+```bash
+for ref in lfnovo/open_notebook:<ver> lfnovo/open_notebook:<ver>-single ghcr.io/lfnovo/open-notebook:<ver> ghcr.io/lfnovo/open-notebook:<ver>-single; do
+  docker manifest inspect "$ref" | python3 -c "import json,sys; d=json.load(sys.stdin); print(sorted(set(m['platform']['architecture'] for m in d.get('manifests',[]) if m['platform']['architecture']!='unknown')))"
+done
+# expect ['amd64', 'arm64'] for all four; repeat with v1-latest and v1-latest-single after publication
+```
+
+RC stack on the pushed image, optionally with a copy of the owner's dev data (copies only):
+
+```bash
+# 1. Find which SurrealDB instance dev uses: SURREAL_URL and SURREAL_DATABASE in .env
+# 2. Consistent export from the RUNNING instance (originals untouched)
+docker exec <that-container> /surreal export --conn http://localhost:8000 \
+  --user root --pass root --ns open_notebook --db <that-db> /dev/stdout > /tmp/dev-dump.surql
+# 3. Boot (rc-stack.sh docker-pulls the tag, so a local build cannot shadow the registry artifact)
+make release-stack TAG=<ver> DUMP=/tmp/dev-dump.surql
+#    opt-in heavy runtimes on the pushed image (first boot installs them, minutes):
+#    bash scripts/release-test/rc-stack.sh up <ver> /tmp/dev-dump.surql --with-runtimes
+# 4. Credentials decrypt with the dev key from .env
+curl -s http://localhost:15055/api/credentials | python3 -c "import json,sys; c=json.load(sys.stdin); print(len(c), 'creds,', sum(1 for x in c if x.get('decryption_error')), 'decrypt errors')"
+# 5. GET /api/capabilities reports docling/crawl4ai false until --with-runtimes installs them
+```
+
+## Re-cut after a post-tag fix
+
+A blocker found after the tag exists but before publication: the fix goes through the normal
+PR flow, the version stays, the tag moves, **and the images are rebuilt and re-gated** — a
+stale tag or registry image is what publication would promote.
+
+```bash
+git checkout main && git pull && grep '^version' pyproject.toml   # still <ver>
+uv run pytest tests/ -q && ruff check .                            # + frontend if touched
+git tag -d v<ver> && git push origin :refs/tags/v<ver> && make tag
+git rev-parse v<ver> && git rev-parse HEAD                          # must match
+docker rmi lfnovo/open_notebook:<ver> lfnovo/open_notebook:local 2>/dev/null
+make docker-build-local && make release-test TAG=<ver> OLD_TAG=<prev>
+gh workflow run build-and-release.yml --ref main -f push_latest=false   # overwrites the stale images
+# re-verify the manifests, re-boot the RC stack, wait for the owner's re-GO
+```
+
+## After publication
+
+Label shipped issues `released` (after the owner's OK; only actual closed issues — changelog
+references mix issue and PR numbers):
+
+```bash
+for n in <numbers>; do
+  STATE=$(gh api "repos/lfnovo/open-notebook/issues/$n" --jq 'if .pull_request then "pr" else .state end')
+  [ "$STATE" = "closed" ] && gh issue edit "$n" --add-label released
+done
+```
+
+## Cleanup
+
+```bash
+make release-stack-down
+rm -f /tmp/dev-dump.surql; rm -rf /tmp/onrel-*
+docker ps --format '{{.Names}}' | grep onrel   # must be empty
+git status --short                              # must be clean on main
+```
+
+## Retro
+
+Close every release by asking what should improve. Apply the accepted improvements
+immediately: this file, `.github/RELEASE_PROCESS.md`, `scripts/release-test/`, `gotchas.md`
+and the decision log, while the context is fresh.

.maintainer/release/test-matrix.md

@@ -0,0 +1,58 @@
+# Coverage table
+
+The risks this repository's releases keep checking, as a starting point for each release's
+coverage table. The `release` skill instantiates it against the actual diff, keeps the
+canonical validator and mandatory checks, and adds the risk-selected probes. The unit of
+planning is the **risk**, not the feature: for each change ask *what can this break, and for
+whom?* Use commands and selectors the repository supports; no secret values belong here.
+
+The rows are the release's **Bucket A** (`agent` in *Who*: automated, run now) and
+**Bucket C** (`owner` in *Who*: the release owner runs it by hand; the *Who / paid scope*
+column says which of those need real credentials or paid calls); **Bucket B** (automatable
+with investment) follows the table.
+
+| Check | Change / risk | Real probe | Observable success | Prerequisites | Who / paid scope | Stage | Mandatory |
+|---|---|---|---|---|---|---|---|
+| Backend suite | any backend change | `[commands.validator]` | pytest, ruff and mypy exit 0 | `uv sync --group dev`, `.env` | agent / free | pre-GO | yes |
+| Frontend | any frontend change | `[commands.frontend]` | lint, tests and production build pass | `npm ci` in `frontend/` | agent / free | pre-GO | yes |
+| Dev-DB leak | new or changed tests | per-table counts before and after the suite | no diff (at least `credential`) | live dev DB reachable | agent / free | pre-GO | yes |
+| Smoke journey | any change to sources, chat, ask, transformations, podcasts, search | `smoke-e2e` skill on the dev stack, `[smoke].journey` | GO with every mandatory surface run | stack up, default chat + embedding models | agent / small paid (one chat, one embed, one podcast) | pre-GO | yes |
+| Image gate | Dockerfile, deps, migrations, startup, nginx | `[artifacts.docker].gate` | fresh install and upgrade scenarios pass | Docker, previous tag pulled | agent / free | pre-GO | yes |
+| Dependency audit | any dependency bump | Dependabot alerts + `npm audit` | no open highs, or explicit acceptance | — | agent / free | pre-GO | yes |
+| Provider matrix | provider or Esperanto changes | connection test + discover-models + one chat per main provider | responses from every configured provider | real credentials | owner / paid | pre-GO | yes |
+| Real podcast | podcast, TTS or prompt changes | one episode on a dense notebook | audio plays, transcript in the selected language | TTS credential | owner / paid | pre-GO | yes |
+| UX tour | any UI change | ~10 min walk of every changed screen, light and dark | nothing broken or off-theme | dev stack | owner / free | pre-GO | yes |
+| Pushed image | every release | `make release-stack TAG=<ver> [DUMP=…]` on the registry artifact | core flows work; credentials decrypt; opt-in runtimes gated | images pushed, dev-data copy | owner / free | post-trigger, pre-promotion | yes |
+| Manifests | every release | `docker manifest inspect` per registry × variant | `['amd64','arm64']` for all four, then for `v1-latest*` | images pushed | agent / free | post-trigger | yes |
+
+## Probe library (extend per release)
+
+Regression-of-legitimate-use probes proven in v1.11.0 — adapt endpoints and values:
+
+- Upload just under / just over the body cap → accepted / 413.
+- Source ingestion of a `localhost` URL → accepted (self-hosted is legitimate);
+  link-local or metadata URL → rejected with a clear 4xx.
+- Frontend `/config` with clean vs. malformed `Host` → sane URL or fallback, never 5xx.
+- SSE endpoints stream progressively (first byte ≪ total time via `curl -N -w`).
+- CORS preflight with and without `CORS_ORIGINS` set.
+- Every enum or allowlisted query param exercised with **each** valid value plus one
+  invalid (`sort_by=title` 500'd while its siblings passed — test the whole surface).
+- Oversized array inputs and unknown-provider payloads → clean 422, not 500.
+- Anything an LLM or the UI writes through: verify the full path in a real browser, not just
+  the API (the frontend dropped a field AND the API ignored null — only end-to-end caught it).
+
+## Bucket B — automatable with investment
+
+Decide per item with the owner: build it if it compounds for future releases and costs less
+than the manual verification it replaces; otherwise verify manually this once and note it
+here. The image gate graduated from here to `make release-test`. Standing candidates: new
+end-to-end scenarios for the release's features; CI-ification of any probe that proved
+valuable twice; anything the owner keeps verifying by hand.
+
+## The project's inverse questions, checked on every release
+
+- Does a protection break legitimate use? (SSRF guard vs. self-hosted Ollama on localhost;
+  body-size cap vs. big uploads; Host validation vs. reverse proxies.)
+- Does the abstraction hold parity across the provider matrix? (A feature that works only on
+  one provider is a PDR-002 question, not a release note.)
+- Does an upgrade on an existing volume survive? (Migrations, encrypted credentials, data.)

.maintainer/smoke/journey.md

@@ -0,0 +1,148 @@
+# Smoke journey
+
+The product journey the `smoke-e2e` skill executes against a running instance: notebook →
+sources → chat → ask → transformation → note → podcast → search → cleanup, then the UI.
+
+## Conventions
+- Base URL for API calls: `<api_url>/api`; JSON unless a step says multipart.
+- Ids look like `notebook:xxxx`, `source:xxxx`, `chat_session:xxxx`.
+- Source creation is **multipart/form-data**; booleans are the strings `"true"`/`"false"`;
+  `notebooks` is a JSON-array string. `embed=true` is required for embeddings to be created.
+- Async work (processing, embeddings, insights, podcasts) needs the worker running
+  (`make worker-start`); without it every poll times out.
+
+## Health
+- GET `<api_url>/docs` → 200
+- GET `<frontend_url>` → 200
+- GET `<api_url>/api/credentials` → at least one credential
+- GET `<api_url>/api/models/defaults` → `default_chat_model` and a default embedding model set
+save: default_chat_model
+
+## Phase 1: API
+
+### Step: create notebook
+request: POST /api/notebooks {"name": "Smoke Test Notebook", "description": "Automated smoke test — safe to delete"}
+expect: 200; body.id present
+save: notebook_id
+
+### Step: add URL source
+request: POST /api/sources (multipart) type=link, notebooks=["<notebook_id>"], url=https://en.wikipedia.org/wiki/Turing_test, async_processing=true, embed=true
+expect: 200; body.id present
+save: url_source_id
+
+### Step: add text source
+request: POST /api/sources (multipart) type=text, notebooks=["<notebook_id>"], title="AI Overview - Smoke Test", content="Artificial intelligence (AI) is the simulation of human intelligence processes by computer systems. These processes include learning, reasoning, and self-correction. AI applications include expert systems, natural language processing, speech recognition, and machine vision.", async_processing=true, embed=true
+expect: 200; body.id present
+save: text_source_id
+
+### Step: add PDF source
+request: POST /api/sources (multipart) type=upload, notebooks=["<notebook_id>"], file=@tests/fixtures/smoke-test.pdf, async_processing=true, embed=true
+expect: 200; body.id present — not-run when the fixture is absent (two sources are sufficient)
+save: pdf_source_id
+
+### Step: wait for processing
+request: GET /api/sources/<url_source_id>/status and GET /api/sources/<text_source_id>/status
+expect: status == completed for each (failed → FAIL with the error)
+timeout: 5m
+
+### Step: verify content and embeddings
+request: GET /api/sources/<text_source_id>
+expect: full_text non-empty; embedded == true; embedded_chunks > 0 (if embedded is false: POST /api/embed {"item_id": "<text_source_id>", "item_type": "source", "async_processing": false}, wait 15s, re-check)
+
+### Step: chat session
+request: POST /api/chat/sessions {"notebook_id": "<notebook_id>"}
+expect: 200; body.id present
+save: session_id
+
+### Step: chat context
+request: POST /api/chat/context {"notebook_id": "<notebook_id>", "context_config": {}}
+expect: 200; body.context has sources and notes arrays
+save: context
+
+### Step: chat message
+request: POST /api/chat/execute {"session_id": "<session_id>", "message": "What are the main topics covered in the sources?", "context": <context>}
+expect: 200; body.messages contains a message of type ai referencing the sources' content
+timeout: 3m
+
+### Step: ask
+request: POST /api/search/ask/simple {"question": "What is the Turing test and how does it relate to AI?", "strategy_model": "<default_chat_model>", "answer_model": "<default_chat_model>", "final_answer_model": "<default_chat_model>"}
+expect: 200; body.answer non-empty; body.question echoes the question (scoping to notebooks landed in #1331 — check the request model before adding a notebook field)
+timeout: 3m
+
+### Step: list transformations
+request: GET /api/transformations
+expect: 200; a transformation such as "Dense Summary" present
+save: transformation_id
+
+### Step: run transformation
+request: POST /api/sources/<text_source_id>/insights {"transformation_id": "<transformation_id>"}
+expect: 202 (the insight is generated by the worker); then GET /api/sources/<text_source_id>/insights returns at least one insight
+save: insight_id
+timeout: 3m
+
+### Step: save insight as note
+request: POST /api/insights/<insight_id>/save-as-note {"notebook_id": "<notebook_id>"}
+expect: 200; note with non-empty content
+save: note_id
+
+### Step: podcast profiles
+request: GET /api/episode-profiles and GET /api/speaker-profiles
+expect: 200; at least one of each — not-run for the podcast steps when none is configured
+save: episode_profile_name, speaker_profile_name
+
+### Step: generate podcast
+request: POST /api/podcasts/generate {"notebook_id": "<notebook_id>", "episode_profile": "<episode_profile_name>", "speaker_profile": "<speaker_profile_name>", "episode_name": "Smoke Test Episode"}
+expect: 200; body.job_id present (profiles are referenced by NAME, not id)
+save: job_id
+
+### Step: wait for podcast
+request: GET /api/podcasts/jobs/<job_id>
+expect: status == completed; result.episode_id present (the top-level episode_id may be empty)
+save: episode_id
+timeout: 10m
+
+### Step: podcast audio
+request: GET /api/podcasts/episodes/<episode_id>/audio
+expect: 200
+
+### Step: semantic search
+request: POST /api/search {"query": "artificial intelligence", "type": "vector"}
+expect: 200; body.results non-empty; at least one result from this notebook's sources (the field is `type`, not `search_type`)
+
+### Step: delete preview
+request: GET /api/notebooks/<notebook_id>/delete-preview
+expect: 200
+
+### Step: delete notebook
+request: DELETE /api/notebooks/<notebook_id>?delete_exclusive_sources=true
+expect: 200; GET /api/notebooks/<notebook_id> → 404; GET /api/sources/<text_source_id> → 404
+
+### Step: delete episode
+request: DELETE /api/podcasts/episodes/<episode_id>
+expect: 200 (skip when no episode was created)
+
+## Phase 2: UI
+Create a fresh notebook "UI Smoke Test" with one text source via the API first (same
+payloads as above), wait for processing, then:
+- `<frontend_url>/notebooks` lists "UI Smoke Test"
+- `<frontend_url>/notebooks/<ui_notebook_id>` shows the source in the sources panel; console has no errors
+- the source detail (open it from the list) shows the extracted content
+- the chat panel sends a message and renders an AI reply
+- `<frontend_url>/search` returns results for "Turing test"
+- `<frontend_url>/podcasts` lists the smoke episode when one was generated
+- after deleting the UI notebook via the API, `<frontend_url>/notebooks` no longer lists it
+- SSR pages may take up to 30s on a cold dev server
+
+## Cleanup
+- DELETE `/api/notebooks/<ui_notebook_id>?delete_exclusive_sources=true` → 200
+- every notebook, source, note and episode created by the journey is gone (`GET` → 404)
+- nothing named "Smoke Test" remains in `/api/notebooks` or `/api/podcasts/episodes`
+
+## Quirks
+- Processing can take minutes after heavy operations; the dev server slows down.
+- The frontend may run on an alternate port (`PORT=3001 npm run dev`) when 3000 belongs to
+  another project — pass the real `frontend_url`.
+- A dev venv may have `docling`/`crawl4ai` installed out-of-band, so `GET /api/capabilities`
+  does not reflect the lean image; judge opt-in gating on the RC stack, not here.
+- Unknown JSON fields are silently ignored by Pydantic: a wrong field name turns a vector
+  search into a text search without an error.

.maintainer/triage.md

@@ -0,0 +1,78 @@
+# Triage rules
+
+Repository-specific rules layered over the plugin's maturity-ladder preset. They refine the
+preset; they never contradict it silently. The human version, with reply templates, is
+[docs/7-DEVELOPMENT/maintainer-guide.md → Issue Management](../docs/7-DEVELOPMENT/maintainer-guide.md#issue-management).
+
+## What reaches Issues
+
+Ideas, feature requests and design proposals start in **Discussions** (category Ideas) and
+graduate into Issues only when someone will build them (`pull`, see `[discussions]` in the
+profile). Issues are for **reproducible bugs**, **installation reports** and **approved work**
+a maintainer created from a Discussion. An idea opened as an Issue is not triaged into a
+state: it is converted to a Discussion with the guide's template reply (GitHub UI; `gh` cannot
+convert) — or closed with the same text when conversion is not possible.
+
+## The queue
+
+The bug and installation forms are meant to apply `needs-triage`; the label must exist
+(recreate it if missing). An issue needs triage when it carries `needs-triage` **or no state
+label at all**. Skip issues labeled `tracked-in-umbrella` (they are triaged inside their
+umbrella) and `umbrella` issues themselves (graduated as a whole when the vision call is made;
+#712 stays open because PDR-001 references it).
+
+## States (one per issue)
+
+| Label | Meaning | Assigned by triage |
+|---|---|---|
+| `needs-triage` | new report not yet classified | no (intake) |
+| `needs-info` | waiting on the reporter to confirm or provide more information | yes |
+| `needs-design` | wanted, but the *how* is not resolved | yes |
+| `needs-vision` | unsure if/how it fits — strategic call for the maintainers, judged against VISION.md | no (recognised; the maintainer sets it) |
+| `ready` | approved and sufficiently specified — the dev loop can pick it up | yes |
+| closed | not a bug, duplicate, superseded, already fixed, upstream-only, or feature request rerouted | yes, with the reason and an open door |
+
+`ready` is a promise of execution: an issue lands there when its problem, pointers and
+acceptance criteria are complete **and** someone will build it next. A well-specified bug
+with an open PR that fixes it is `ready` (the PR closes it).
+
+## Labels that are not states (never touched by triage)
+
+- **Type** (one): `bug`, `enhancement`, `documentation`, `installation` (intake, applied by
+  the form). Feature requests carry `enhancement` only when they legitimately live in Issues.
+- **Area** (one, always): `area: chat`, `area: search`, `area: sources`, `area: notebooks`,
+  `area: podcast`, `area: providers`, `area: embeddings`, `area: ui`, `area: database`,
+  `area: offline`, `area: deploy` (installation reports route here), `area: i18n`.
+  The guide asks triage to add the area; the plugin's triage only proposes it — the
+  maintainer applies it.
+- **Ecosystem / bundling** (any number): `upstream`, `content-core`, `esperanto`,
+  `podcast-creator`, `bundled`, `umbrella`, `tracked-in-umbrella`, `good first issue`,
+  `help wanted`, `released`.
+
+The label set is curated — **never invent a label**; raise it instead.
+
+## Close criteria particular to this project
+
+- **Feature request in an Issue** → Discussion (see above). Not a rejection; say so.
+- **Root cause upstream** (`content-core`, `esperanto`, `podcast-creator`): keep the issue
+  open with `upstream` + the library label only when a downstream bump or docs change is
+  needed here; otherwise close pointing at the upstream issue (open it first).
+- **Not reproducible with the supplied steps** and no answer after `needs-info`: close with
+  an open door ("reopen with the logs and we will look again").
+- **Already fixed on `main`**: close naming the PR; point testers at the `v1-dev` image
+  (rebuilt on every push to `main`) and say it ships with the next release.
+- **Duplicate**: close with the canonical link; a duplicate report that adds a new
+  environment or a new symptom is a comment on the canonical issue, not a separate one.
+
+## Vision-fit heuristics
+
+- Fit is judged against `VISION.md` (the *IS / IS NOT* list and the principles) and the
+  decision records in `docs/7-DEVELOPMENT/decisions/` — never against taste. The
+  load-bearing ones: PDR-001 (single-user first, multi-user compatible), PDR-002
+  (provider-agnostic core; provider-exclusive capabilities need a PDR), ADR-007 (opt-in
+  heavy runtimes), ADR-008 (notebook-scoped search).
+- Current posture (VISION.md → Current Posture): get the basics solid across the provider
+  matrix before new product surfaces. A request for a new surface that fits the identity is
+  `needs-vision`, not `needs-design`.
+- A request that keeps coming back against a principle is a signal to revisit the principle
+  through a decision record — not to make a quiet exception.

.pre-commit-config.yaml

@@ -0,0 +1,51 @@
+# Pre-commit hooks mirroring the CI gates so local commits fail for the same
+# reasons PRs would. See .github/workflows/test.yml for the CI equivalents.
+#
+# Install: uv run pre-commit install
+# Run on all files: uv run pre-commit run --all-files
+# Update hook versions: uv run pre-commit autoupdate
+
+repos:
+  # ── Basic hygiene ──────────────────────────────────────────────────
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v5.0.0
+    hooks:
+      - id: check-added-large-files   # blocks >500 KB binaries
+      - id: check-merge-conflict      # catches unresolved conflict markers
+      - id: check-yaml                # validates YAML syntax
+      - id: check-toml                # validates TOML syntax
+      - id: end-of-file-fixer         # ensures files end with a newline
+      - id: trailing-whitespace       # removes stray trailing whitespace
+        args: [--markdown-linebreak-ext=md]
+
+  # ── Ruff: Python lint & format (mirrors CI backend-lint job) ──────
+  # Version kept in sync with pyproject.toml [dependency-groups] dev.
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.14.13
+    hooks:
+      # `ruff check --fix` — applies safe auto-fixes (e.g. import sorting)
+      # and reports remaining violations. CI runs `ruff check .` without
+      # --fix; the local auto-fix saves iteration with no drift risk.
+      - id: ruff
+        args: [--fix]
+      # `ruff format` — reformats files to match ruff's formatter. CI runs
+      # `ruff format --check .` with the same tool config, so there is no drift.
+      - id: ruff-format
+
+  # ── Mypy: Python type check (mirrors CI backend-typecheck job) ────
+  # Uses language: system so mypy runs through `uv` in the project's own
+  # virtualenv (the same way CI and `make lint` run it). Pre-commit's
+  # isolated mirror-mypy would miss errors from missing project deps like
+  # langgraph and langchain — that breaks the "no drift" rule.
+  #
+  # Note: type-checking the full project on every commit is slower than
+  # the per-file hooks above. Skip it with `SKIP=mypy git commit`.
+  - repo: local
+    hooks:
+      - id: mypy
+        name: mypy
+        entry: uv run python -m mypy .
+        language: system
+        types: [python]
+        pass_filenames: false
+        verbose: true

AGENTS.md

@@ -0,0 +1,49 @@
+# Open Notebook — Agent Rules
+
+Open Notebook is an open-source, privacy-focused alternative to Google's Notebook LM: an AI-powered research assistant with multi-provider AI support, fully self-hostable.
+
+This file holds the project-wide rules every coding session needs. Component rules: [open_notebook/AGENTS.md](open_notebook/AGENTS.md) (backend — also covers `api/`, `commands/`, `prompts/`) and [frontend/AGENTS.md](frontend/AGENTS.md). Knowledge lives in the docs (see [Where to look](#where-to-look)) — read it on demand instead of guessing.
+
+## Stack, ports, startup order
+
+Three tiers: Next.js frontend (3000) → FastAPI (5055) → SurrealDB (8000).
+
+Start in this order — each tier depends on the one below:
+
+1. `make database` — SurrealDB (API fails without it)
+2. `make api` — FastAPI; **schema migrations run automatically on startup** (check logs)
+3. `make worker-start` — surreal-commands worker. **Required**: podcasts, embeddings and source processing are async jobs that silently queue forever without it
+4. `make frontend` — UI (depends on the API for all data)
+
+Or all at once: `make start-all` (status: `make status`, stop: `make stop-all`). First-time setup (`uv sync`, `npm install`, a `.env` with `SURREAL_URL=ws://localhost:8000/rpc`): [development-setup.md](docs/7-DEVELOPMENT/development-setup.md).
+
+## Commands
+
+- Tests: `uv run pytest tests/`
+- Python lint/format/typecheck: `uv run ruff check . --fix` · `uv run ruff format .` · `uv run python -m mypy .`
+- Frontend (inside `frontend/`): `npm run lint` · `npm run test` · `npm run build`
+- CI runs `uv run ruff check .`, `uv run ruff format --check .`, mypy, pytest and, in `frontend/`, `npm run lint`, `npm run test:coverage` and `npm run build`; all must pass
+- Docker release: `make docker-release` (see `.github/RELEASE_PROCESS.md`)
+
+## Hard rules
+
+- **Async-first**: every DB query, graph invocation and AI call is `await`-ed. No sync DB access.
+- **Never commit secrets.** Credentials are encrypted at rest and require `OPEN_NOTEBOOK_ENCRYPTION_KEY` to be set.
+- CORS is wide-open and auth is a simple password middleware — **dev defaults, not production hardening**. Don't build features that assume otherwise.
+- Product direction questions (does this feature fit?) → [VISION.md](VISION.md). Past decisions ("why is it like this?") → [docs/7-DEVELOPMENT/decisions/](docs/7-DEVELOPMENT/decisions/). Structural decisions made while coding should produce a new decision record there.
+- **Conventional Commits** for commits and PR titles (`fix(podcasts): …`, `feat(providers): …`, `docs: …`).
+- **Every user-visible change adds a CHANGELOG entry** under `## [Unreleased]`, in the one section for its type (`### Added` / `### Changed` / `### Deprecated` / `### Removed` / `### Fixed` / `### Security`); never create a duplicate section. Details: [contributing.md](docs/7-DEVELOPMENT/contributing.md#changelog).
+- Maintainer profile lives in `.maintainer/`; do not run release, triage or discussions workflows without it.
+
+## Where to look
+
+| Need | Location |
+|---|---|
+| Architecture (processes, layout, data model, workflows) | [docs/7-DEVELOPMENT/architecture.md](docs/7-DEVELOPMENT/architecture.md) |
+| Step-by-step recipes (add endpoint, provider, migration, command, i18n…) | [docs/7-DEVELOPMENT/change-playbooks.md](docs/7-DEVELOPMENT/change-playbooks.md) |
+| Dev environment setup | [docs/7-DEVELOPMENT/development-setup.md](docs/7-DEVELOPMENT/development-setup.md) |
+| Code standards & testing | [docs/7-DEVELOPMENT/code-standards.md](docs/7-DEVELOPMENT/code-standards.md) · [testing.md](docs/7-DEVELOPMENT/testing.md) |
+| Product identity & current posture | [VISION.md](VISION.md) |
+| Decision log (ADRs/PDRs) | [docs/7-DEVELOPMENT/decisions/](docs/7-DEVELOPMENT/decisions/) |
+| Contribution process (Discussions → Issues → PRs) | [docs/7-DEVELOPMENT/contributing.md](docs/7-DEVELOPMENT/contributing.md) |
+| User/operator docs (install, configure, troubleshoot) | [docs/](docs/index.md) |

CHANGELOG.md

@@ -7,6 +7,339 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Changed
+- **Documentation rewritten against the code.** Every page under `docs/` was checked against v1.15.0 and rewritten where it described an older or imagined product (#1458, #1459, #1460, #1461):
+  - **Install:** one canonical path built on the shipped `docker-compose.yml`, with small override files for the Ollama and LM Studio variants. The network-exposure warning (bind ports to `127.0.0.1` or set a password) comes before the first `docker compose up -d`. Example encryption keys are generated, never copied. First-run setup lives in one place, [Connect a provider](docs/4-AI-PROVIDERS/index.md#connect-a-provider), and ends in a working chat via **Auto-assign Defaults**. The provider table is built from the provider registry: 24 providers, with the model types each one offers
+  - **User guide and concepts:** insights are a first-class concept and the context mode is *Insights only*. Ask is described as three stages. UI labels match the current app. Invented features are gone (tags, export/share, batch transformations, page-number citations and others). About 6,200 lines became about 1,450
+  - **Configuration and troubleshooting:** `environment-reference.md` is the one complete list of variables, each marked with which process reads it. Troubleshooting entries start from the exact error message the app shows; a new page covers source processing and podcast failures. Security docs cover PBKDF2 credential encryption and the back-up-before-migrating warning
+  - **Development:** `architecture.md`, `testing.md`, `api-reference.md` and `code-standards.md` are rewritten from the code. A new playbook covers adding an AI provider. Dev setup lives on one page. The AGENTS files agree with the docs. Contributor conventions (Conventional Commits, CHANGELOG sections, pre-PR checks that match CI) are documented
+
+### Fixed
+- **Docs no longer recommend settings that do nothing.** `API_CLIENT_TIMEOUT` was never read by Open Notebook; the timeout guidance now covers what actually applies: `ESPERANTO_LLM_TIMEOUT` per model call (keep it below 600) and the web UI's 10-minute request limit (`NEXT_PUBLIC_API_TIMEOUT_MS`, build time only). Remaining `SURREAL_COMMANDS_MAX_TASKS` advice now uses `OPEN_NOTEBOOK_WORKER_MAX_TASKS`. Docker instructions no longer say to edit `.env` and `docker compose restart`: the shipped compose file doesn't load `.env` into the container and `restart` doesn't reload environment changes, so they now point to the `open_notebook` service's `environment:` block and `docker compose up -d`
+- **Timeouts no longer read as connection failures.** A model that doesn't answer within `ESPERANTO_LLM_TIMEOUT` used to report "Could not connect to the AI provider"; it now says the provider took too long and how to raise the limit. Connection failures (including connect timeouts) keep the old message
+- Error hints point to screens that exist: an authentication failure sends you to Manage → Models (not "Settings → Credentials"), and an unsupported podcast voice to Podcasts → Profiles. The empty Podcasts page points to the Generate Podcast button instead of chat screens that can't start a podcast
+- **Italian is selectable in the language menu**, and dates are formatted in Italian. The menu now lists every registered language, so a new translation can't be left out again
+- `make start-all` starts SurrealDB again (it pointed to a compose file removed long ago)
+- The legacy `SURREAL_ADDRESS` / `SURREAL_PORT` fallback builds a valid database URL (`ws://host:port/rpc`, was `ws://host/rpc:port`); an address that already includes the port is kept as is
+- `.env.example` names the chunking variables Open Notebook actually reads (`OPEN_NOTEBOOK_CHUNK_SIZE` / `OPEN_NOTEBOOK_CHUNK_OVERLAP`)
+
+## [1.15.0] - 2026-10-04
+
+### Added
+- **OpenDocument uploads and YouTube unblocking options.** `.odt`, `.ods` and `.odp` files (and `.htm`) can be uploaded as sources (content-core 2.2). New content-core settings are documented: `CCORE_YOUTUBE_PROXY` and `CCORE_YOUTUBE_COOKIES_FILE` get YouTube transcripts through when YouTube blocks the server's IP, and `CCORE_AUDIO_SEGMENT_MINUTES` controls how long audio is split for transcription (#1439)
+- **SiliconFlow and Z.ai providers.** Both are available in Manage → Models with credential setup, connection test and model discovery (language models). SiliconFlow serves hosted DeepSeek, Qwen, GLM and Kimi models; mainland China accounts set the credential's Base URL to `https://api.siliconflow.cn/v1`. Z.ai serves the GLM family. Model discovery honors their Base URL override (from the credential or `SILICONFLOW_BASE_URL` / `ZAI_BASE_URL`), as it already did for OpenAI (#1409, #1437)
+- **MiniMax text-to-speech.** MiniMax credentials now offer text-to-speech, and Discover Models lists `speech-2.8-hd` and `speech-2.8-turbo`, so MiniMax voices can be used in podcast speaker profiles. The connection test and model suggestions move to `MiniMax-M3` (1M-token context) (#1438)
+- **New "Quiet Green" design foundation** — the first of three PRs applying the visual identity co-designed with the community in Discussion #1202. The design token system lands in `globals.css` (core palette, neutral reading surfaces, ink ramp, hairline borders, content-type hues, evidence/citation classes, context-state colors, a squared 4–6px radius scale and a near-flat shadow scale where popovers own the only real shadow); fonts move from Inter to Bricolage Grotesque (display) + Instrument Sans (UI) + Spline Sans Mono (data) via `next/font`; the shadcn primitives are restyled to the foundation's rules (fern green primary actions, red reserved exclusively for destructive/error, clay for warnings, teal as the AI/system voice, underline tabs, quiet neutral badges, teal focus ring); and hard-coded palette colors across 38 components were swept into the semantic tokens, which re-resolve automatically per theme. A dev-only living styleguide at `/dev/design` (404 in production) renders every token and primitive in both themes and serves as the review reference for the follow-up screen-by-screen PRs. Purely visual — no behavior, navigation or i18n changes
+- **Screen-by-screen reskin in the "Quiet Green" language** (the second redesign PR): the app shell gets the recomposed tri-hue pebble wordmark (fern/gold/teal — no red left), a fern spine on the active nav item and destination-hued nav icons; the notebook workspace gets display-type titles, panel headers with identity ticks (sources sage / notes gold / chat teal), one-line source-card metadata with the overflow menu at top-right, and de-washed chat bubbles (the AI speaks in teal accents, never washed backgrounds); the notebooks home separates compact recently-viewed rows from active-notebook cards; the sources table gets the library treatment (content-type pebbles, quiet embedded pills, hover that raises the surface); dialogs and the source viewer are flattened (no card-in-card boxes, uniform breathing room, no ⋮/close collision); and Models/Settings/Podcasts/Transformations/Ask-Search get the badge diet, quiet audio-player containers, mono for data and teal AI accents. Still purely visual — no behavior, columns, navigation or i18n changes
+
+### Changed
+- **Podcasts on podcast-creator 0.13.** Outlines and transcripts are requested as schema-validated JSON, with an automatic fallback to plain JSON for endpoints that don't support JSON schema (0.13.1). The default output budget is now 8192 tokens for the outline and the transcript (was 3000 / 5000); set an episode profile's Max output tokens lower for models with a smaller limit. Episodes now have a ~400 ms pause between speaker turns, and audio combining no longer cuts the end of a turn (#1435)
+- Podcast generation failures now carry a hint that matches the failure: a placeholder/renamed speaker name, a `voice_id` its TTS model doesn't support, and a response that was truncated rather than swallowed by `<think>` tags each get their own explanation. The GPT-5 extended-thinking note used to be the only hint, so the two most common real failures got either nothing or advice about the wrong provider (#1238)
+- Community contribution intake now separates exploration from execution: feature requests, product/design/architecture ideas and contribution proposals start in GitHub Discussions, while Issues are reserved for reproducible bugs and maintainer-approved work items. The Issue chooser routes contributors accordingly, a structured Ideas Discussion form starts from user goals and outcomes, and the contributor/maintainer docs plus PR template now describe the Discussion → Issue → PR graduation path (#1204).
+- Release image gate gained a `probe` scenario (`make release-test` runs it as part of `all`): container-level checks that a Python test suite can't cover because they depend on the shipped image's process supervision — `OPEN_NOTEBOOK_WORKER_MAX_TASKS` reaching the in-image worker (the supervisord `sh -c` expansion), and the worker surviving startup with `HTTP_PROXY` set while a user's `NO_PROXY` value is preserved (the internal SurrealDB websocket not being tunneled). Both were manual probes during the v1.14.0 release; they now run automatically. Release-process docs gained the post-tag re-cut sequence and a note on never leaving the version bump uncommitted (v1.14.0 retro)
+- Python code is now formatted with `ruff format` across the repository, and CI fails when it isn't (`ruff format --check .` in the Backend Lint job). Formatting-only change (59 files); run `uv run ruff format .` before pushing
+- Search page explains what each mode covers: text search matches source titles and full text, insights, and note titles and content, while vector search matches embedded source chunks, insights and note content only. Documented in `docs/3-FEATURES/search.md` and shown as a mode-aware hint under the search box, translated across all 14 locales (#1295, #1267)
+- README star-history chart points at a working renderer; the previous endpoint had been returning a "GitHub restricted access to star data" placeholder (#1262)
+- **Credential encryption now uses PBKDF2-HMAC-SHA256 (600k iterations) with versioned `pbkdf2v1:` ciphertext.** New keys are written in the new format; legacy values keep decrypting, and a one-shot `POST /api/credentials/migrate-encryption` pass upgrades stored keys (idempotent, fail-closed per record). Downgrade warning: versions predating this change cannot read the new format — back up the database before migrating (#1317)
+
+### Fixed
+- **Podcasts generate on a fresh install again.** Generation hands every speaker profile to podcast-creator, which requires a voice model on each; the seeded profiles ship without one, so any unconfigured profile failed every podcast with `validation errors for SpeakerConfig`, even when generating with a complete profile. Speaker profiles without a voice model are now left out of the config (with a warning), like unresolvable episode profiles already were; the selected profile still fails early with a clear error if it has no voice model (#1450)
+- **The worker no longer stalls when a source is deleted before it is processed.** `Source.get()` raises `NotFoundError` for a missing record, so the job was retried as a transient failure (up to 15 attempts with backoff) and every job behind it waited; a missing source or transformation now fails the job immediately. Relatedly, loading a record no longer reports a database failure (e.g. a SurrealDB transaction conflict) as "not found": it raises a database error, so it stays retryable in jobs and returns 500 instead of 404 in the API (#1363)
+- **Model calls default to a 180-second timeout.** esperanto 2.28 now enforces `ESPERANTO_LLM_TIMEOUT` on every provider (60 s by default); before, most providers used their SDK default and Ollama waited indefinitely. Open Notebook now sets 180 s when the variable is unset, so long answers and slower models aren't cut off at 60 s. **Ollama and other slow local setups** that relied on unlimited waits should set `ESPERANTO_LLM_TIMEOUT` explicitly, below 600 s (the web UI's request timeout) (#1434)
+- Batch large embedding inserts into groups of 50 records and reuse one database connection across batches to reduce oversized WebSocket payloads and repeated sign-ins.
+- `GET /api/notebooks` now normalizes `order_by` through the shared `_validate_order_by()` guard, keeping its stricter `name`/`created`/`updated` allowlist on top, so the two ORDER BY validators can no longer drift (#1416)
+- **Source chat no longer repeats earlier answers in the streamed reply.** The stream sent every AI message in the checkpointed history, so the second and later answers arrived concatenated with all previous ones; it now sends only the new answer. The context badge ("1 Sources") is also restored from the saved session after a page refresh (#1393)
+- API password authentication now accepts UTF-8 non-ASCII passwords; clients sending non-ASCII passwords as Latin-1 bytes must switch to UTF-8 (#1290)
+- **Failed sources fail once, with a clear reason.** content-core 2.2 raises typed errors instead of returning empty content (unreachable host, page not found, malformed URL, corrupted file, unsupported type, YouTube without a transcript or blocked, misconfigured engine or speech-to-text model). Source processing now turns each into a short, actionable message, kept with the original cause in the worker log, and fails the job on the first attempt instead of retrying it up to 15 times. A blocked YouTube request points to the new `CCORE_YOUTUBE_PROXY` / `CCORE_YOUTUBE_COOKIES_FILE` settings (#1433)
+- **Podcast episode profiles honor their `language` again.** The app's `prompts/podcast/{outline,transcript}.jinja` shadow podcast-creator's bundled templates (the library resolves `Path.cwd()/prompts/podcast/` before its own resources) and had lost the `{{ language }}` block, so a profile set to `he-IL` produced an English outline and English segment titles — the field looked supported and did nothing. A regression test now fails when a variable the bundled template uses is missing from the app's copy (#1238)
+- **A copied prompt placeholder no longer aborts a podcast mid-generation.** Both podcast templates showed the model a fill-in JSON skeleton (`"speaker": "[Actual Speaker Name]"`, a bare `...`) inside a ```json fence while also instructing it not to use fences; models returned the skeleton verbatim, which failed podcast-creator's speaker-name validation and discarded the outline and every segment already generated. The examples now carry the episode's real speaker names and written-out sample values, so a verbatim copy is valid output; placeholders and truncation are banned explicitly; and for a run with a `language` set no sample is shown at all, since the only sample that can be hard-coded is English (#1238)
+- **Ask answers are no longer silently truncated.** The three Ask stages (search strategy, per-search answers, final synthesis) were capped at 2000 output tokens, well below the 8192 that chat and transformations use. Token-dense languages such as Japanese hit the cap mid-sentence, and reasoning models spent the whole budget thinking and returned blank search terms, so Ask answered "no documents found" from a corpus that had the answer. All three stages now share an 8192 budget; a strategy with no usable search terms fails with an explicit error instead of running empty searches, and thinking-only partial answers are dropped before synthesis (#1221)
+- A source embedding job that fails part-way through the batched insert now removes the embeddings it already wrote, so the source is no longer shown as embedded while search only sees part of it (#1390)
+- **Remote Crawl4AI servers that require a bearer token work again.** Crawl4AI Docker ≥ 0.9.0 rejects unauthenticated external connections by default, so pointing `CRAWL4AI_API_URL` at a current instance failed URL processing. content-core is bumped to 2.0.7, which sends `CRAWL4AI_API_TOKEN` as `Authorization: Bearer`; the variable is documented in the environment reference, `.env.example` and `docker-compose.yml`. No behavior change when the token is unset (#1269, lfnovo/content-core#80)
+- **Source Chat streaming no longer drops or corrupts tokens.** The SSE reader decoded each network chunk in isolation, so a `data:` line split across two chunks was silently discarded and a multibyte character straddling a chunk boundary rendered as `<60>`. The reader now keeps a carry-over buffer and streaming decoder, matching the pattern Ask already used (#1289)
+- **Transformations no longer run on sources with no text.** A source whose extraction produced nothing (e.g. a link that failed to fetch) used to send the model an empty input and save whatever it invented as an insight. Generating an insight on such a source now returns "Source has no text content" and nothing is queued; the transformation graph rejects empty content too (#1394)
+- **Markdown editor follows the app theme.** The note editor's preview/edit surface was hardcoded to light mode, making it unreadable in dark mode; it now tracks the effective theme, gated on store hydration so there is no light flash before the persisted theme loads (#1294, #1268)
+- **Bullet and numbered list markers are visible again in the note editor preview.** Tailwind v4's preflight reset removed list markers that the preview library relies on the browser default for; two low-specificity rules restore them without affecting nested-list numbering or task lists (#1300, #1299)
+- Long voice-model names no longer overflow the speaker-profile card in Podcasts → Episode Profiles; the badge truncates with a tooltip (#1259, #1196)
+- Clearing a notebook or transformation description now persists after a refresh: the UI dropped the empty value from the update, and the API now treats an explicit empty or null `description` as a clear while an absent field still leaves it untouched. (#1396)
+- Theme, Language and Sign Out actions in the expanded sidebar footer are left-aligned again (the Theme button was missing the icon padding the others get) (#1288)
+- **YouTube transcript language preference is honored again.** `ContentSettings.youtube_preferred_languages` (exposed via `GET`/`PUT /api/settings`) was silently ignored since the content-core 2.x migration — source processing always used a hardcoded list, so videos with transcripts only in an unlisted language failed to import. The setting now reaches content-core's `youtube_languages`, and both the built-in default and the setting default gained Simplified and Traditional Chinese (`zh-CN`, `zh-TW`). Supersedes #1287
+- **Vector search results are always returned in descending similarity order.** `fn::vector_search` orders inside the function, but the outer `SELECT *` wrapper did not reliably preserve that order for multi-result queries, so `POST /api/search` could return a weaker match first. Results are now re-sorted by similarity in the domain layer (#1306, #1301)
+- **An empty model reply is now an error, not an answer.** Notebook chat and source chat stored a blank AI message that was replayed as context on later turns, and streaming Ask finished with an empty final answer and no error. All three now surface "The model returned an empty response" (transformations already did). A failed chat turn also no longer leaves the question in the conversation history, so retrying does not duplicate it, and streamed errors keep their specific message instead of a generic "AI service error" (#1392)
+- **Anthropic connection test no longer reports retired Haiku ids as HTTP 403.** The status-code check matched "403" inside the date suffix of `claude-3-haiku-20240307`, turning a plain "model not found" into a permissions error. Status codes are now matched as standalone numbers for 401/403/429, and the registry's test model moved to a current Haiku id (#1303, #1282)
+- **Context-length errors are no longer retried in background jobs.** A source whose content exceeds the model's context window used to be retried up to 15 times (about 25 minutes) before failing, because the error was classified like a transient provider outage. A dedicated `ContextLengthExceededError` now stops the retry loop immediately (#1275)
+
+### Security
+- Bumped esperanto to 2.28.0, content-core to 2.2.0 and podcast-creator to 0.13.1. podcast-creator no longer depends on moviepy, so the `pillow>=12.2.0` override is gone (content-core now floors `pillow>=12.3.0` itself); content-core's extraction logs, which it disables for library consumers since 2.1, are re-enabled in the API and worker (#1408)
+- Dependency security update: `next` 16.3.8 (critical RCE advisory in `next/og` `ImageResponse`; the app does not use `next/og`), `axios` 1.20.0, and the npm overrides for `postcss` (8.5.23) and `brace-expansion` (1.1.21); Python `anyio` 4.14.2, `pyjwt` 2.15.1, `tornado` 6.5.10, `urllib3` 2.8.0 and `virtualenv` 21.14.5. Remaining `npm audit` findings are moderate and limited to the test runner (`vitest`/`@vitest/mocker`, `fflate`). Dependabot now skips odd-numbered (non-LTS) Node majors for the image base
+
+## [1.14.0] - 2026-07-20
+
+### Added
+- **Docling formula & vision enrichment toggles** in Settings → Content Processing. Two new opt-in checkboxes surface content-core 2.x's `docling_formulas` (extract mathematical formulas as structured markup) and `docling_vision` (describe images and extract chart data with a vision model) enrichment flags, mirroring the existing OCR toggle. Both default off; the vision help text warns it is significantly slower and may call a vision model. Like OCR, the toggles only take effect through the Docling engine, so they are gated on Docling availability. The settings persist via `GET`/`PUT /api/settings` and are threaded into content-core extraction alongside `docling_ocr` (migration 23 backfills the new fields); labels and help are translated across all 14 locales (#1131)
+- **OpenRouter now supports text-to-speech and speech-to-text**, not just LLMs and embeddings (esperanto 2.25.0). The provider's modalities were extended so the Settings UI offers TTS/STT when registering OpenRouter models. Because OpenRouter's `/models` listing does not reliably tag audio models, discovery seeds the working defaults esperanto ships — `microsoft/mai-voice-2` (TTS) and `openai/whisper-1` + `openai/whisper-large-v3` (STT) — alongside the API-discovered language models; any other `vendor/model` id can be added manually. Note that `microsoft/mai-voice-2` uses Microsoft neural voice names (e.g. `en-US-AvaNeural`), not OpenAI's `alloy`/`nova` set (#987)
+- Newer ElevenLabs models are now offered in model discovery: `eleven_v3`, `eleven_flash_v2_5` and `eleven_flash_v2` for text-to-speech, and `scribe_v2` for speech-to-text — alongside the existing models. Esperanto passes the `model_id` straight through to ElevenLabs, so no provider changes were needed (#1167)
+- **`anthropic_compatible` credential provider**, mirroring the existing `openai_compatible` one, so endpoints that only speak the Anthropic Messages format (e.g. Kimi) can be connected with a base URL plus API key — from Settings → API Keys or via `ANTHROPIC_COMPATIBLE_BASE_URL` / `ANTHROPIC_COMPATIBLE_API_KEY`. The connection test carries one Anthropic-specific wrinkle: a 404/405 from `GET /models` is reported as "listing unsupported, register models manually" rather than a failed connection, since Messages-only endpoints are legitimately unable to list. Model discovery uses `GET /v1/models` with the Anthropic headers and shares the base-URL normalization and request-time re-validation used by provisioning. Provider card and credential form translated across all 14 locales (#675)
+- **First-class oMLX provider** in Settings → API Keys: native Esperanto `omlx` profile identity (language + embedding), default base URL `http://localhost:11435/v1` (avoids SurrealDB on 8000), optional API key, connection test and `/v1/models` discovery. No remapping to `openai-compatible` and no `OPENAI_COMPATIBLE_*` env mirroring. Docs under `docs/5-CONFIGURATION/omlx.md` (#1048; upstream lfnovo/esperanto#228)
+- **Four new AI providers** (surfaced by esperanto 2.25.1) are now in the provider matrix. **Cohere** (`COHERE_API_KEY`) adds language and embedding models via Cohere's native v2 API (`command-*` chat, `embed-*`); model discovery is bespoke (esperanto's `AIFactory.get_provider_models`) since Cohere is not OpenAI-compatible, and Cohere reranking is intentionally left out of scope (tracked at #1087). **PayPerQ / PPQ** (`PPQ_API_KEY`, `https://api.ppq.ai/v1`) is a multi-modality OpenAI-compatible gateway offering language, embedding, speech-to-text and text-to-speech. **Novita** (`NOVITA_API_KEY`, `https://api.novita.ai/openai`) is an OpenAI-compatible LLM gateway. PPQ and Novita auto-discover their models through the standard `/models` endpoint. In addition, the existing **Deepgram** provider now also supports **speech-to-text** (Nova/Whisper transcription models, e.g. `nova-3`) alongside its Aura text-to-speech voices. All four are configured from Settings → Models exactly like the existing providers (#1170)
+- **Worker concurrency is now configurable** via `OPEN_NOTEBOOK_WORKER_MAX_TASKS`. The surreal-commands worker always ran with a hardcoded concurrency of 5, which overloads single-GPU and local-LLM setups and trips provider rate limits when several sources are processed at once. The variable is wired into every launch point — `make worker-start`/`make start-all`, `dev-init.sh`, and the Docker `supervisord.conf` (wrapped in `sh -c` so the shell actually expands it, since `command=` does not run through a shell) — and defaults to 5; set it to 1 for strictly sequential processing. It is read at worker launch time, so changing it requires restarting the worker. This also removes a phantom entry from the docs: `SURREAL_COMMANDS_MAX_TASKS` was documented but read nowhere (the worker's `--max-tasks` Typer option has no envvar binding) and has been replaced with the real variable in `.env.example` and the environment reference (#893)
+- Episode profiles now expose an optional **"Max output tokens"** field in the UI. The `max_tokens` value already existed on the podcast episode profile model but could only be set through the API; the profile form now offers it as a number input (positive integer, clearable to unset), mirroring how `num_segments` is handled — zod validation, form defaults/reset and submit payload — and the episode card shows the configured value in its details. The field carries help text explaining that it caps output tokens per generation step (the outline and the full transcript — not a per-turn or conversation limit) and that leaving it blank uses the built-in defaults (3000 for the outline, 5000 for the transcript), plus a placeholder to the same effect; clearing it no longer trips a spurious "must be a positive integer" error (an empty or transient number-input state now reads as unset). Label, help and placeholder translated across all 14 locales (#991)
+- Local commits are now gated by the same checks as CI: a new `.pre-commit-config.yaml` runs ruff lint (with `--fix`), ruff format, mypy (through `uv run` so it matches CI exactly) and basic hygiene hooks (large files, merge conflict markers, YAML/TOML syntax, trailing whitespace, EOF newlines), with install/run/skip instructions in the development-setup docs (#940)
+- The i18n locale parity test now also compares **interpolation placeholders**, not just keys. It previously only checked that every locale defined the same key set, so a value carrying a stray single-brace `{count}` instead of i18next's `{{count}}` — or a placeholder that does not exist in the en-US reference — passed silently and rendered as literal text at runtime. Every leaf string is now checked against en-US for both missing/extra `{{...}}` placeholders and stray single-brace tokens; this caught and fixed `podcasts.generationStartedDesc` in zh-CN and zh-TW, which referenced a `{{name}}` placeholder the reference string does not have (#1159)
+- The dev Docker build (`build-regular`) is gated on image-affecting paths, so docs- and config-only PRs no longer wait several minutes for a byte-identical image. A `paths-filter` job decides whether the build steps run; the job itself always executes so the required status check always reports — deliberately avoiding the workflow-level paths filter that would leave the required check permanently pending and deadlock merges. PRs touching the Dockerfile, dependencies, app or frontend build as before, and the release build is untouched (#1134)
+- Release tooling: `rc-stack.sh up` now `docker pull`s `lfnovo/open_notebook:<tag>` before booting so a same-named local build cannot shadow the registry artifact during release verification (non-fatal for local-only tags), and a new `--with-runtimes` flag enables the opt-in Docling/Crawl4AI heavy engines on the RC stack so the published image's opt-in install path can be exercised with real data. Both gotchas are documented in `RELEASE_PROCESS.md`; the default release-test compose leaves the runtimes off to keep fresh/upgrade runs fast (#1133)
+- README now links to [@lfnovo on X](https://x.com/lfnovo) for project updates (#1158)
+
+### Changed
+- Bumped the esperanto floor to `>=2.25.1`. This is the dependency change that unlocks the oMLX provider, the Cohere/Deepgram STT/PPQ/Novita providers, OpenRouter TTS/STT, Vertex service-account auth and the native Anthropic `to_langchain()` base URL forwarding listed elsewhere in this release. A clean drop-in — no application code changes were required (#1176)
+- Removed the `ChatAnthropic` re-injection shim used by `anthropic_compatible` models. esperanto 2.25.1 forwards a custom `base_url` natively through `to_langchain()`, so the `_to_langchain()` helper in `provision.py` and the `_open_notebook_provider` marker it depended on (its only consumer) are gone; provisioning now calls `model.to_langchain()` directly. No behavior change — custom Anthropic-compatible endpoints keep working, now through the upstream path (#1055)
+- Japanese (ja-JP) translations reviewed and improved throughout: "トランスフォーメーション" replaced with the natural 変換, duration formatting given proper spacing, and accessibility labels, error messages and assorted phrasings rewritten to read naturally rather than as literal translations (#998)
+
+### Fixed
+- **A content-processing engine whose runtime is not installed no longer breaks ingestion silently.** The engine choice is persisted in the database, but the opt-in runtimes that serve it (Docling, local Crawl4AI) are installed from environment flags evaluated at boot — so a redeploy that drops `OPEN_NOTEBOOK_ENABLE_CRAWL4AI`/`_DOCLING`, a data volume moved to a new deployment, or a failed on-demand install leaves a stored selection pointing at a runtime that is absent. That selection was passed straight to content-core, and every affected extraction failed with "Could not extract any text content from this source" — naming neither the engine, the runtime, nor the flag that would fix it; with the URL engine set to Crawl4AI it broke URL ingestion entirely. The source graph now checks runtime availability first and degrades to content-core's `auto` chain, logging a warning that names the engine and the variable that would enable it; engines needing no opt-in runtime (`auto`/`simple`/`firecrawl`/`jina`) are untouched. The availability probes moved to `open_notebook/utils/runtime_capabilities.py` so the graph can share them with `GET /api/capabilities`, whose behavior is unchanged (#1194)
+- **HTTP proxy no longer breaks worker/API startup.** `websockets` 15.0 began auto-detecting `HTTP_PROXY`/`HTTPS_PROXY` and tunnels even `ws://` connections through the proxy, so with a proxy set the internal SurrealDB websocket (`ws://host.docker.internal:8018/rpc` or `ws://surrealdb:8000/rpc`) was routed through the external proxy, which rejected the internal host with HTTP 403 and killed the worker on startup. Open Notebook now injects `host.docker.internal,surrealdb,localhost,127.0.0.1` into `no_proxy`/`NO_PROXY` at startup (merged with any user value, never clobbering it), and the `.env.example` / docs `NO_PROXY` examples now include the internal DB hosts (#1160)
+- Auto-assign no longer silently re-populates optional model defaults that were deliberately cleared. Previously `POST /models/auto-assign` treated every empty slot as "missing" and filled it, so an optional slot a user intentionally cleared (to fall back to the chat model) got re-assigned on the next run. Auto-assign now fills only the two required slots (chat, embedding); the optional slots (transformation, tools, large context, TTS, STT) are left untouched. `large_context` now also falls back to the chat model when unset (matching transformation/tools) instead of returning nothing, and the Settings UI shows an inline hint on each empty optional slot — "using chat model (…)" for the text slots, "not configured" for TTS/STT (#1098)
+- PPQ model discovery now lists all modalities, not just chat. PPQ's `/v1/models` returns only chat/language models by default; the discovery URL now requests `?type=all` so the embedding, speech-to-text and text-to-speech models this multi-modality gateway advertises actually surface in the provider matrix (`PPQ_MODEL_TYPES` classifies them into the right slots) (#1180)
+- Vertex credentials now configure text-to-speech (and other Vertex providers) correctly when used through a stored credential rather than environment variables. `Credential.to_esperanto_config()` emitted the generic `project`/`location` keys, but esperanto's Vertex providers expect `vertex_project`/`vertex_location` — so credential-linked Vertex TTS crashed with `__init__() got an unexpected keyword argument 'project'`. The config now maps to the Vertex-specific key names for the `vertex` provider only; the `Credential` schema/API fields stay `project`/`location`, and non-Vertex providers are unaffected (#1151)
+- Deleting a notebook now also deletes its chat sessions instead of leaving orphaned `chat_session` records behind. The notebook `delete()` cascade enumerates the notebook's chat sessions (via the existing `refers_to` relation) and deletes each one alongside the notes and exclusive sources it already handled (#1124)
+- **Outbound requests to user-supplied provider endpoints are now pinned to a DNS-vetted IP address.** Validating a URL and then handing it to httpx left a DNS-rebinding TOCTOU window: httpx re-resolves the hostname at connect time, so a hostname that passed validation could resolve to a link-local or cloud metadata address (e.g. `169.254.169.254`, `fd00:ec2::254`) on the actual connection. A new `prepare_pinned_http_target()` helper resolves once, rejects dangerous addresses, and rewrites the request URL to the vetted IP while preserving the original hostname in the `Host` header and TLS SNI (IDNA-encoded), so routing and certificate verification are unaffected. Applied consistently to the credential-save, connection-test and model-discovery paths for `openai_compatible`, `ollama`, `azure` and the OpenAI provider's custom `base_url` (#1063)
+- Per-transformation model selection now actually takes effect. A transformation's assigned `model_id` was persisted but ignored at execution time: both call sites that run the transformation graph (the source-processing graph and the `run_transformation` background command) invoked it without a config, so the graph node always read a `None` model and fell back to the global default transformation model. Both now forward `transformation.model_id` through the LangGraph `configurable` config, so a transformation configured to use a specific model runs on that model (an unset `model_id` still falls back to the default) (#1137)
+- Error text surfaced by the sources API is now capped at 200 characters. `GET` source status and synchronous processing failures returned the raw command/result `error_message` unbounded, which could push arbitrary internal exception text (including stack detail from third-party extraction libraries) to clients and did not match the truncation already applied elsewhere in the API. Both paths now go through a `None`-safe truncation helper that appends an ellipsis when it cuts (#1136)
+- **Typing in the chat composer no longer lags in long conversations**, which was especially painful for IME users (Japanese, Chinese, Korean) whose per-keystroke composition events made the whole message history re-render. The composer input state moved into a dedicated `ChatComposer` child so keystrokes stay local to it, each message row is memoized with `React.memo`, and the reference-click handler is stabilized with `useCallback` — which in turn required memoizing `openModal`/`closeModal` in `useModalManager`, since their changing identities were re-rendering every memoized row on each composer keystroke. Send behavior and the Ctrl/Cmd+Enter shortcut are unchanged (#1147)
+- The quick-start guide and the two Ollama compose examples set `OLLAMA_BASE_URL`, which no code path reads — the variable the provider registry actually looks for is `OLLAMA_API_BASE`. Following the examples verbatim left Ollama permanently unavailable with no error to explain why. All three shipped files now use the correct name (#1148)
+
+## [1.13.0] - 2026-07-13
+
+### Added
+- **Opt-in heavy extraction runtimes.** Docling (layout parsing + OCR + image sources) and local Crawl4AI (JavaScript rendering) are now **opt-in**, installed automatically on first container startup when enabled — instead of being bundled into (or missing from) every image. Set `OPEN_NOTEBOOK_ENABLE_DOCLING=true` and/or `OPEN_NOTEBOOK_ENABLE_CRAWL4AI=true`; the downloads are cached on the `/app/data` volume so only the first boot is slow, and a failed install degrades gracefully (the app still starts, the engine is reported unavailable) with loud logs. A new `GET /api/capabilities` probe reports what's actually installed, and Settings → Content Processing disables the Docling engine, the Crawl4AI engine and the OCR toggle (with an env-var hint) until their runtime is available — so the UI never advertises an engine that isn't there. This keeps the default image lean (no Chromium, no multi-GB ML stack) while making both runtimes available on demand. A remote Crawl4AI server via `CRAWL4AI_API_URL` needs no local install. New hint strings added across all 14 locales; recorded as ADR-007 (#1122)
+- **OCR toggle** in Settings → Content Processing: a new "Enable OCR" checkbox controls whether the Docling engine runs OCR on scanned PDFs and images. It's on by default (matching content-core's behavior); turn it off to speed up processing of text-native documents. The setting is passed to content-core's `docling_ocr` config, and the label/help are translated across all 14 locales. OCR only runs through Docling, so the toggle takes effect once Docling is enabled (see #1122) (#1104)
+- **Crawl4AI** is now selectable as a URL processing engine in Settings → Content Processing, alongside Firecrawl, Jina and Simple. It renders JavaScript-heavy pages locally with no API key, or offloads to a Crawl4AI server when `CRAWL4AI_API_URL` is set. Local Crawl4AI (and its Chromium browser) is an opt-in runtime installed on first startup when `OPEN_NOTEBOOK_ENABLE_CRAWL4AI=true` (see #1122) — the default image stays lean. As part of this, the persisted document/URL engine choices now actually take effect: the source-processing graph reads the saved Content Settings and passes them to content-core (previously it always ran with hard-coded `auto` engines and silently ignored the selection). New engine label added across all 14 locales (#432)
+- Source extraction now logs the **effective content-core engine** at INFO (`url_engine` / `document_engine` / `docling_ocr`) right before it runs. content-core only logs its own engine dispatch at DEBUG, so operators previously had no way to confirm which engine actually processed a source — e.g. whether a persisted Crawl4AI/Docling selection took effect or the request silently fell back to the `auto` chain (#1125)
+
+### Changed
+- Upgraded the content extraction dependency from content-core 1.14.x to 2.x (2.0.4). The source-processing graph was adapted to content-core's new keyword-only `extract_content()` API: engine/model overrides now travel through a `ContentCoreConfig` object instead of the input dict, and the extraction result (`ExtractionOutput`) no longer echoes the source `url`/`file_path` back, so those are carried from the request state into the saved source asset. Because content-core 2.x no longer deletes the uploaded file after extraction, the graph now honors the `delete_source` flag itself. Transitively this replaces the AGPL-licensed PyMuPDF with MIT-licensed pdfplumber for PDF extraction and drops moviepy in favor of direct ffmpeg calls (which fixes audio extraction from MP3 files carrying chapter metadata). No user-facing configuration changes in this step — document and URL engines stay on their `auto` defaults; new engine/OCR options are tracked separately under #939 (#1103)
+- Podcast episode audio paths are now stored relative to the podcasts folder (`episodes/<uuid>/audio/<name>.mp3`) instead of as absolute filesystem paths, validated at write time so the database can never hold an absolute or root-escaping value, and resolved + containment-checked through a single shared helper instead of per-endpoint guards. Migration 21 converts existing rows written under the known roots (plain `file:///` URIs, `/app/data/podcasts/`, `/data/podcasts/`, `./data/podcasts/`, `data/podcasts/`); previously generated episodes now survive a `DATA_FOLDER` relocation. Rows under any other root (e.g. a source checkout at a custom absolute path) or in exotic legacy forms (percent-encoded or host-qualified `file://` URIs) are left untouched and treated as legacy-invalid — the same 403/audio-unavailable handling out-of-root rows already had; regenerate the episode to restore playback. Podcast jobs whose audio combination fails (podcast-creator reports an in-band `ERROR:` value) now fail with the real ffmpeg/clip error instead of reporting success for an episode with no playable audio (#1030)
+- The source detail view (dialog and full page) now fetches through the shared `useSource` React Query hook instead of a hand-rolled fetch, matching the insight/note dialogs: caching and the never-retry-404 policy come from React Query, title edits and deletes go through the shared mutation hooks (so source lists refresh and a deleted source can't be served from the cache), and the `key={sourceId}` remount workaround on the parents was removed — the component resets its own per-source UI state (#1106)
+- The settings frontend now fetches the provider list from `GET /api/providers` (cached for the session) instead of keeping its own hardcoded provider tables, so adding a provider to the backend registry needs zero frontend edits: unknown modalities render with a fallback icon, the backend registry owns the display order, and the regex-based frontend/backend sync test was removed along with the duplicated tables in `frontend/src/lib/providers.tsx` (#1082)
+
+### Removed
+- The legacy provider/model string fields on podcast profiles (`outline_provider`, `outline_model`, `transcript_provider`, `transcript_model` on episode profiles; `tts_provider`, `tts_model` on speaker profiles) are gone from the database, the API and the UI — the app has ignored them since the Model registry references landed in v1.11. Migration 22 first best-effort maps any profile whose `outline_llm`/`transcript_llm`/`voice_model` reference is still empty to an existing model record (matching provider + name + type; no auto-creation, since a migration must not touch credentials), then drops the six columns; the startup data migration that used to retry this mapping on every boot (`open_notebook/podcasts/migration.py`) was deleted. Accepted trade-off: profiles whose mapping never converged (e.g. the provider credential was never configured) lose the legacy strings and stay unresolved — they were already non-functional and the UI already flags them as needing model selection, so you just re-pick the models in the profile form once (#1107)
+
+### Fixed
+- Uploading a file content-core can't extract now fails immediately at ingestion with a clear `415 Unsupported Media Type` error that names the detected MIME type, instead of enqueueing a background job that retried up to 15 times over ~1 hour before surfacing a generic "Failed" with no actionable detail. The pre-flight uses content-core 2.x's header-only `check_file_support()` — the same routing real extraction uses, so the verdict can't disagree with what would happen downstream — and the source-retry endpoint is guarded the same way; unexpected check errors (e.g. a file removed before a retry) fall through to normal extraction rather than becoming a hard rejection (#975)
+- Podcast episode cards no longer show "— / —" for the outline, transcript and speaker model rows on new episodes: the API now resolves the snapshot's model references (`outline_llm`/`transcript_llm`/`voice_model`) to provider/name display fields at serialization time — batched into a single query per request, so listing episodes never does a per-row model lookup — and the card falls back to the legacy snapshot strings for old episodes and degrades to "—" when a referenced model was deleted (#1114)
+- Renaming a speaker profile no longer breaks the episode profiles that use it: `episode_profile.speaker_config` now stores a `record<speaker_profile>` reference instead of the profile name (migration 20 converts existing rows; references whose speaker profile no longer exists at migration time become null, and any reference that later stops resolving is treated as "needs setup" — the UI asks you to pick a speaker again). The `POST /api/podcasts/generate` contract is unchanged — it still accepts the speaker profile by name and resolves it at the API boundary (#630)
+- Clicking a chat/Ask citation that points at a deleted source, insight, or note now shows a shared, friendly "this content no longer exists" state in all three dialogs (instead of a raw error, a blank dialog, or an empty editable note editor), 404 lookups are no longer retried, and non-404 failures show a distinct "unable to load" message (#455)
+- Source insights now get `created`/`updated` timestamps stamped at creation (migration 19 mirrors the defaults used by the other tables), and the insights API returns `null` — instead of the literal string `"None"` — for legacy insights that predate the migration (#1045)
+- `uv sync` alone now provides the full dev toolchain: the legacy `[project.optional-dependencies].dev` list was merged into `[dependency-groups].dev` (mypy included — the documented `uv run python -m mypy .` previously failed on a fresh clone), Jupyter-only packages moved to a separate `notebooks` group, and the CI typecheck job no longer needs `--extra dev` (#1101)
+- Optional model defaults (transformation, tools, large context, TTS, STT) can now be cleared: `PUT /api/models/defaults` honors explicit `null` (field absent still means "keep"; chat and embedding defaults reject `null`), and the default-model selects offer a "None" / "Use fallback (chat default)" option for the optional defaults (#1091)
+- `docker-compose.yml` now uses the YAML list (exec) form for the SurrealDB `command`, so `SURREAL_USER` / `SURREAL_PASSWORD` values containing spaces are passed as single arguments instead of being split; the mirrored snippets in the installation docs and README (which had drifted — no credential interpolation, SurrealDB port published on all interfaces) are back in sync with the shipped file (#1093)
+- zh-CN and zh-TW podcast toast descriptions (speaker/episode profile created/updated/deleted/duplicated) now include the profile name via the `{{name}}` placeholder, matching the other 12 locales (#1084)
+- Docker images now force the Next.js frontend to bind to `0.0.0.0` in the supervisord command itself, so container runtimes that inject `HOSTNAME` (e.g. Podman pods, where it resolves to `127.0.1.1`) can no longer make the UI unreachable. The `HOSTNAME` variable is no longer honored as a frontend bind override — set the new `FRONTEND_BIND_HOST` variable instead (#994)
+- Podcast generation now honors an explicitly supplied `speaker_profile` all the way into `generate_podcast_command`. The v1.12 fix stopped at the API boundary — the command still re-derived the speaker from the episode profile's `speaker_config` and silently dropped the override, so a direct caller's valid profile was ignored and generation failed when the episode profile pointed at a renamed/deleted speaker. The command now resolves `speaker_profile or episode_profile.speaker_config`, and the persisted episode plus the retry path reflect the actually-resolved profile (#1058)
+- Creating a transformation from the empty state works again: after deleting every transformation, the empty-state "New Transformation" button did nothing because the editor dialog was only mounted in the non-empty branch. The dialog is now rendered in both branches so the create flow works with zero existing transformations (#999)
+
+## [1.12.0] - 2026-07-12
+
+### Fixed
+- Setup snippets no longer teach publishing SurrealDB on `0.0.0.0` — the compose and `docker run` examples across the README, quick starts, installation, configuration and development docs, and the `examples/docker-compose-*.yml` files now bind port 8000 to `127.0.0.1` (matching the shipped `docker-compose.yml`), with docs pointing to `docker-compose.override.yml.example` for opt-in remote access behind a firewall or SSH tunnel; the override example itself gained the `!override` tag it needs to actually replace the base port binding instead of colliding with it (#1034)
+
+### Added
+- Docs: cubic platform mechanics recorded as comments in `cubic.yaml` (agent limits, config precedence, memory/learning) and a "Merging PR Batches" playbook added to the maintainer guide (squash policy, CHANGELOG conflict resolution, fork rebases, competing-PR checks) (#1086)
+- New `GET /api/providers` endpoint returning provider metadata from the registry (name, display name, modalities, docs URL, whether it is configured via environment variables), so clients can enumerate supported providers instead of hardcoding them (#1075)
+- Release confidence process, documented and executable: `.github/RELEASE_PROCESS.md` now covers the risk-based test matrix (buckets A/B/C), the Docker image gate, the fix-loop re-test policy and the communication/credits/retro structure, backed by a new decision record (ADR-005) and versioned tooling under `scripts/release-test/` — `make release-test TAG= OLD_TAG=` runs fresh-install + upgrade scenarios against real images, and `make release-stack TAG= [DUMP=]` boots a browsable, isolated release-candidate stack (optionally with a copy of dev data) for manual verification (#1052)
+- CI now gates every PR on `ruff check` (backend lint), `npm run lint` (frontend ESLint) and `npm run build` (frontend production build), in addition to the existing test suites (#1068)
+- CI now also gates every PR on `mypy` (backend typecheck): the repo-wide baseline went from 197 errors to 0 (enabling the pydantic mypy plugin resolved most of them; the rest got real annotations), so new type errors are blocked from here on. The `ignore_errors` burn-down also started: `open_notebook.graphs.transformation`, `open_notebook.graphs.ask` and `api.routers.models` are now type-checked (plus two stale entries for deleted modules removed); only `open_notebook.domain.notebook` remains exempt pending the surreal-basics migration (#1076)
+
+### Changed
+- cubic AI review now skips `CHANGELOG.md`, `uv.lock` and `frontend/package-lock.json` (no reviewable logic; preserves the monthly reviewed-line quota) (#1080)
+- Context building consolidated into a single implementation (`open_notebook/utils/context_builder.py`): the copy-pasted source/note assembly loops behind `POST /api/chat/context` and the removed notebook-context endpoint, plus the 495-line generalized `ContextBuilder` class (whose only caller was the source-chat graph), are now two focused functions — `build_notebook_context()` (backs `POST /api/chat/context`, unchanged request/response shapes and config semantics) and `build_source_context()` (backs the source-chat graph, same context shape and 50k-token budget). Pinned by new characterization tests — no behavior change for the surviving paths (#1079)
+- **Removed** `POST /api/notebooks/{notebook_id}/context`: it duplicated `POST /api/chat/context` (same assembly logic, slightly different response envelope) and had zero callers — frontend, docs and tests only use `/api/chat/context`. If you called it programmatically, switch to `POST /api/chat/context` (body: `{notebook_id, context_config}`; response fields: `context.sources`/`context.notes`, `token_count`, `char_count`) (#1079)
+- Backend provider metadata now lives in a single registry (`open_notebook/ai/provider_registry.py`): env var config, modalities, connection-test models, OpenAI-compatible discovery URLs and docs links are defined once per provider, and `PROVIDER_ENV_CONFIG`, `PROVIDER_MODALITIES`, `TEST_MODELS` and `OPENAI_COMPAT_PROVIDERS` are derived from it. Adding a provider drops from ~6 hand-synced dicts to the registry plus two manual copies (the `SupportedProvider` Literal and the frontend provider table), both enforced by tests (#1075)
+- Frontend convention cleanup (no user-facing change): hook files unified to kebab-case (`useNotebookChat.ts`/`useSourceChat.ts` → `use-notebook-chat.ts`/`use-source-chat.ts`), `src/components/source/` merged into `src/components/sources/`, the localStorage auth-token parsing ritual extracted into a single `getAuthToken()` helper (`src/lib/auth-token.ts`), and non-streaming raw `fetch` calls routed through `apiClient` (podcast audio download, auth-status check). SSE/streaming paths and the login/checkAuth credential probes deliberately keep raw `fetch` (#1077)
+- Pruned unused langchain packages: removed `langchain-community` and `langchain-deepseek` from the dependencies (nothing imports them — DeepSeek and xAI route through esperanto's OpenAI-compatible path, which uses `langchain-openai`). The remaining `langchain-*` provider packages are documented as runtime requirements of esperanto's dynamic `to_langchain()` and the whole langchain/langgraph family now carries explicit upper bounds; `langchain-core` and `langchain-text-splitters` (both directly imported but previously only transitive) are now declared explicitly (#1073)
+- The two Docker images (regular and single-container) are now built from a single multi-stage `Dockerfile` with shared stages — regular is the default (`runtime`) target, single-container is `--target single` — so deploy fixes (tiktoken pre-cache, env defaults, npm retry logic) no longer have to be applied twice. `Dockerfile.single` and `supervisord.single.conf` were removed; the single image appends a small `supervisord.surrealdb.conf` to the shared `supervisord.conf` at build time. Published image names and tags are unchanged (#1066)
+- Model discovery is now table-driven: the eight providers with OpenAI-compatible `/models` endpoints (OpenAI, Groq, Mistral, DeepSeek, xAI, OpenRouter, DashScope, MiniMax) share one generic discovery function configured by `OPENAI_COMPAT_PROVIDERS`, replacing eight near-identical copies (provider-specific quirks like Mistral's capability flags and OpenRouter's descriptions are preserved as hooks) (#1070)
+- Internal refactor: extracted the session/source verification, record-ID normalization, LangGraph message extraction and shared response models duplicated across the chat and source-chat routers into `api/routers/_chat_shared.py`, pinned by new characterization tests — no behavior change (#1072)
+- Internal refactor of the sources API router: extracted a shared `SourceResponse` builder (was hand-rolled 5×), a single upload-cleanup helper (was pasted 6×), split the 293-line create-source endpoint into validation + sync/async path functions, and unified the duplicated paginated list query. No behavior change; all security checks (atomic filename claim, path-traversal containment, SSRF/LFI guards) preserved verbatim (#1069)
+- Re-enabled the ruff rules for unused imports (`F401`), unused local variables (`F841`) and bare `except:` (`E722`) that were ignored to silence legacy Streamlit-era noise, and cleaned up the remaining fallout (10 unused imports, 2 unused test variables; no bare excepts remained) (#1062)
+- Internal refactor with no user-facing change: split the 1,441-line API Keys settings page into focused components under `frontend/src/components/settings/` and moved the provider config tables to `frontend/src/lib/providers.tsx`, deduplicating the default-model select in the process (#1065)
+- Chat, source chat, Ask and transformation prompts now steer models to write math as `$$...$$` (display) / `$...$` (inline) so formulas render via KaTeX, reserving fenced `latex` code blocks for when the user explicitly asks for the LaTeX source (#1051)
+- Frontend locale files are now type-checked at compile time: every non-en-US locale declares `satisfies TranslationShape` (derived from the en-US object), so a missing or extra i18n key fails `tsc` in the editor instead of only the runtime parity test. Also removed two unused frontend dependencies (`next-themes`, `@monaco-editor/react`) and fixed `frontend/AGENTS.md` drift (14 locales, not 7; dark mode is the hand-rolled zustand theme-store, not next-themes) (#1061)
+
+### Fixed
+- Eight podcast toast descriptions (speaker/episode profile created/updated/deleted/duplicated) showed the literal `{name}` placeholder instead of the profile name: the locale strings used single braces (which i18next ignores) and the `t()` call sites passed no values. Placeholders normalized to `{{name}}` across all locales and the actual name is now passed in from the mutation response/variables (#1077)
+- Typed domain errors now return their documented HTTP status codes instead of a generic 500: the API routers used to wrap endpoint bodies in a broad `except Exception` that swallowed the `open_notebook.exceptions` hierarchy before the global handlers could map it (`NotFoundError`→404, `InvalidInputError`→400, `ConfigurationError`→422, `RateLimitError`→429, `NetworkError`/`ExternalServiceError`→502). All 18 affected routers now re-raise `HTTPException` and `OpenNotebookError` and only convert genuinely unexpected exceptions into sanitized 500s. Most visible changes: a missing/unconfigured model (`ConfigurationError`) now returns 422 with an actionable message instead of 500; getting or deleting a source-chat session whose session isn't related to the source returns 404 (was a 500 wrapping the inner 404); fetching a missing credential returns 404 (was 500) (#1078)
+- Frontend translations now use i18next interpolation (`t('key', { count })`) instead of manual `.replace('{count}', ...)` string surgery across ~75 call sites — locale placeholders changed from `{name}` to `{{name}}` in all 14 locales. This restores proper pluralization (e.g. "used by N episodes" now goes through i18next plural forms) and lets translators reorder placeholders freely (#1074)
+- Podcast generation dialog: the token/char counter no longer fires a request storm on rapid checkbox toggling (debounced, with a stale-response guard so a slow response can't overwrite a fresher count) and the dialog now closes as soon as the episode-list refetch completes instead of after a fixed 500ms timer; the 983-line component was also split (content selection panel and selection helpers extracted, duplicated context-config logic deduplicated) with no behavior changes (#1067)
+- Anthropic models are now discovered live from `GET https://api.anthropic.com/v1/models` (paginated) instead of a hardcoded claude-3-era list — the code comment claiming "Anthropic doesn't have a model listing API" was wrong. A refreshed static list (current Claude 4.x/5 aliases) remains as a fallback when the API call fails, and the credential-based discovery path (`discover_with_config`) uses the same live-with-fallback logic (#1070)
+- Deduplicated the embedding commands (`commands/embedding_commands.py`, ~100 lines less): `embed_note`/`embed_insight`/`embed_source` now share one load→embed→write core with a single error-handling epilogue, the rebuild command uses one submission-loop helper for all three kinds, and the thrice-copied `full_model_dump()` moved to `open_notebook/utils/model_utils.py`. Pure refactor — same outputs, logs and retry behavior (#1071)
+
+### Removed
+- Dead Streamlit-era service layer (~2,000 lines): `api/client.py` (a synchronous HTTP client that called the app's own API) and 13 `api/*_service.py` wrappers that consumed the app's own HTTP API — none were imported by any router, command or test. Also removed the toy `process_text`/`analyze_data` demo commands (`commands/example_commands.py`) from the background worker (#1054)
+- Pre-1.6 embedding job compatibility shims (the `embed_single_item`, `embed_chunk` and `vectorize_source` command handlers) — they existed only so jobs queued by a pre-1.6 version could drain after an upgrade, and any worker restarted on 1.6+ has no such jobs. **Upgrade note:** if you are upgrading from a version older than 1.6 with embedding jobs still queued, drain the queue on a 1.x release before upgrading past this change. Also removed dead tooling config from `pyproject.toml`: the `[tool.mypy]` block (the real config is `mypy.ini`) and Streamlit-era ruff per-file-ignores for files that no longer exist (#1056)
+- Committed QA screenshots (12 files) and a stray debug `history.txt` were removed from the repo root, with `.gitignore` rules added so they can't come back (#1053)
+
+### Fixed
+- Podcast generation now honors the `speaker_profile` parameter of `POST /api/podcasts/generate` — previously it was silently ignored and the speaker was always re-derived from the episode profile's `speaker_config`, which failed when that pointed at a renamed/deleted speaker profile even if the caller supplied a valid one (#1044)
+
+## [1.11.0] - 2026-07-11
+
+### Added
+- `VISION.md` — the product's source of truth in two layers: durable identity (what Open Notebook is and is not, core principles) and current posture (the phase we're in, directional constraints, and the horizon clusters under consideration)
+- Decision records at `docs/7-DEVELOPMENT/decisions/` — short, immutable ADRs/PDRs answering "why is it like this?", seeded with 4 retroactive ADRs (SurrealDB, delegation to external libraries, Streamlit→Next.js, background workers) and 2 PDRs (single-user first, provider-agnostic core)
+- `AGENTS.md` files (root, `open_notebook/`, `frontend/`) with the normative rules for coding agents and humans — commands, hard rules, and gotchas not derivable from the code; `CLAUDE.md` files are now one-line pointers to them
+- Five new engineering docs pages under `docs/7-DEVELOPMENT/`: credentials, content processing, podcasts, prompts, and frontend architecture
+- Contribution guidelines for AI-assisted and agent-generated PRs in the contributing guide — the operator owns the PR, issue-first still applies, tests must have actually run
+- CI check for broken relative links in markdown (`scripts/check_md_links.py` + `docs-links` workflow on PRs touching `*.md`)
+- `cubic.yaml` — AI review settings as code: PR-contract instructions, three custom review agents (vision & principles alignment backed by `VISION.md`, known mechanical caveats, security & testability) and automatic ultrareviews for auth/credential/encryption/migration changes
+- Documented the flow-driven release process in `.github/RELEASE_PROCESS.md`, including the `ready` to `main` to stable release path, dev/stable image labels, and maintainer verification checklist (#938)
+- List view for the Notebooks page — a tile/list toggle in the header lets you switch between the visual card grid and a compact row layout (name, description, source/note counts, last updated) for easier scanning of large collections. The choice is remembered across reloads and translated across all 14 locales (#885)
+- Documented the `ESPERANTO_TTS_TIMEOUT` environment variable (default `300`s) in the environment reference; raise it for slow or self-hosted TTS providers so long podcast segments don't fail with a timeout (#937)
+- `SECURITY.md` with a coordinated-disclosure policy: how to privately report a vulnerability via GitHub's private vulnerability reporting, supported versions, and response expectations (#943)
+- LaTeX math rendering (KaTeX) now also applies to source content, source insights, Ask answers, transformation output, and the note editor preview — previously only chat had it (#269)
+- Syntax highlighting for fenced code blocks in chat responses, source content, insights and Ask/Search answers — light/dark aware, with 25 common languages bundled (others render as plain text) (#783)
+- "Recently Viewed" section on the Notebooks page — a collapsible grid of the last 12 notebooks and sources you opened, newest first, hidden when there's no view history. Backed by a new `last_viewed_at` timestamp stamped on read and a `GET /api/recently-viewed` endpoint (translated across all 14 locales) (#850)
+- Per-transformation model selection — each transformation can now be assigned its own language model from the transformation editor, overriding the global transformation default for that transformation only. Runs without an explicit model keep using the system default as before (#776)
+- "Refresh content" action on web-link sources — re-fetches the URL and re-embeds the source so its content stays current, available from the source card menu once processing has completed (translated across all 14 locales) (#259)
+- Sources table can now be sorted by every column — type, title, insights count, embedded status, created and updated (a new "Updated" column was added) — via clickable column headers backed by new `GET /api/sources` sort fields (translated across all locales) (#895)
+- EasyPanel deployment template under `examples/easypanel/` — provisions the app plus a dedicated SurrealDB service with auto-generated database/encryption secrets — plus an EasyPanel section in the single-container install guide (#189)
+- Test coverage measurement in CI: backend via `pytest-cov` (terminal + XML reports), frontend via `@vitest/coverage-v8` and a new `test:coverage` script (#942)
+
+### Changed
+- Developer documentation restructured: 17 knowledge-heavy `CLAUDE.md` files consolidated into the 3 `AGENTS.md` + docs pages above; `README.dev.md` became a pointer after its unique content moved into `development-setup.md` (make-workflow matrix), `.github/RELEASE_PROCESS.md` (Docker publishing) and the change playbooks (add-a-language); the maintainer guide now carries the curated label taxonomy (state funnel, `area:` labels, consolidation rules)
+- Fixed stale developer docs while migrating: real migration path/format (`open_notebook/database/migrations/N.surrealql` + `AsyncMigrationManager` registration), provider count (17), locale list (7), and 9 README links that pointed at documentation pages that never existed
+- The API's listen interface in the Docker images is now configurable via a new `API_HOST` environment variable instead of a hardcoded `--host 0.0.0.0`. The default is unchanged (`0.0.0.0`); set `API_HOST=::` to serve IPv6/dual-stack environments (#985)
+- `docker-compose.yml` now sources the SurrealDB credentials from `SURREAL_USER` / `SURREAL_PASSWORD` (applied to both the database server and the app), defaulting to `root:root` so the zero-config quick start is unchanged. Set them in a `.env` file to use your own credentials before exposing the instance; `.env.example` and the compose file note this (#946)
+- Docs no longer claim a hardcoded default API password (`open-notebook-change-me`) exists; the actual behavior is that auth is disabled entirely when `OPEN_NOTEBOOK_PASSWORD` is unset. Also removed the dead `check_api_password` helper that had been superseded by the auth middleware (#1026)
+
+### Fixed
+- Testing a valid Google/Vertex credential no longer fails after Google retires a Gemini model. The connection test used a hard-coded model id that Google shuts down on a schedule (`gemini-2.0-flash`), so a valid key surfaced as a broken connection (#970). The Google/Vertex test now uses Google's floating `gemini-flash-latest` alias, and the provider connection test was reframed so only a rejected key, missing permissions, or an unreachable endpoint count as failures — a missing/retired/rate-limited model still reports the credentials as valid. Deprecated `gemini-1.5`/`gemini-2.0` model references were also removed from the connection-test model lists and documentation
+- API startup no longer crashes when SurrealDB isn't ready yet (e.g. docker-compose race on host reboot: `Temporary failure in name resolution`). The lifespan now polls a lightweight readiness probe with bounded exponential backoff (~50s budget, 5s per-probe timeout) before running migrations; migration errors themselves still fail fast (#708)
+- Markdown typography styles (`prose` classes) are active again: the Tailwind v4 migration left the old `tailwind.config.ts` (which loaded `@tailwindcss/typography`) silently ignored, so rendered markdown lost its typographic styling. The plugin and class-based dark mode are now configured in `globals.css`, and markdown rendering is centralized in a shared `MarkdownRenderer` component (#783)
+- Podcast generation no longer truncates on dense, long-form content (`LengthFinishReasonError` / `OUTPUT_PARSING_FAILURE`): episode profiles now support an optional `max_tokens` that is passed through to podcast_creator's outline/transcript generation, overriding its defaults — settable via the episode profile API (UI follow-up in #991) (#639)
+- API no longer freezes for all requests while a chat waits on the LLM. Both the notebook chat (`execute_chat`) and source chat handlers ran LangGraph's synchronous `invoke()` directly on the event loop; they now run it via `asyncio.to_thread()` (matching the existing `get_state` calls), so other requests stay responsive — and the source-chat SSE can flush its early events instead of stalling until the model finishes (#704)
+- Windows native install guide no longer points users at a `start-open-notebook.bat` that doesn't exist in the repo; the Quick Start now documents starting the four services manually with `uv run`, plus an optional sample launcher you can save yourself (#846)
+- OpenRouter (and other providers') "Discover models" dialog no longer cuts off the submit button: the dialog now uses a fixed header/footer with a scrollable body (`grid-rows-[auto_1fr_auto]`) instead of scrolling the whole content, so the "Add" button stays visible regardless of how many models are listed (#816)
+- Chat references using the short `[insight:<id>]` form (emitted by some models) are now rendered as clickable citations like `[source_insight:<id>]` and `[note:<id>]` already were; `insight` is treated as an alias for `source_insight`, so clicking it opens the insight (#490)
+- CRUD endpoints now return `404` (not `500`) for a non-existent resource. `ObjectModel.get()` raises `NotFoundError` rather than returning a falsy value, so the broad `except Exception` in each handler was masking it as a server error. Added an explicit `NotFoundError → 404` arm to the notebook (update / delete / delete-preview / add-source / remove-source), note (get / update / delete / list / create), model (delete), credential (update / delete) and embed handlers (#862)
+- Token counting no longer raises `ValueError: disallowed special token '<|endoftext|>'` when source/context content contains special-token sequences; `token_count()` now encodes with `disallowed_special=()` so such substrings are treated as ordinary text (#667)
+- Single-container image no longer hangs at "API not ready yet" on a brand-new instance. `supervisord.single.conf` ran the API and worker with `uv run` (without `--no-sync`), so at startup `uv` tried to sync dev dependencies it couldn't resolve against the `--no-dev` build. Both processes now use `uv run --no-sync`, matching the multi-container `supervisord.conf` (#609)
+- Note editor now expands to fill the dialog instead of being capped at `500px`; removed the `max-h-[500px]` constraint that overrode the `flex-1` parent and cramped editing on tall windows (#932)
+- Ask and source-chat responses now stream progressively instead of hanging at "Processing..." until the full answer is ready. The API's streaming endpoints now declare `text/event-stream` (with no-buffering headers), and dedicated Next.js route handlers pass the SSE body through as a stream — Next.js `rewrites()` buffers SSE responses to completion (#770)
+- Chat, notebook-context and podcast generation now build their context with a single batched insight query instead of one query per source (14 → 3 queries on a 12-source notebook), via the new `SourceInsight.get_for_sources()` (#1008)
+- File uploads no longer block the event loop: `save_uploaded_file()` now writes via `asyncio.to_thread()`, keeping the API responsive during large uploads (#1009)
+- URL validation no longer blocks the event loop on DNS resolution: `validate_url()` is now async and resolves hostnames via `asyncio.to_thread()`, so a slow DNS lookup on the model-provisioning path can't stall concurrent requests (#1011)
+- Creating a credential with an unknown provider name now fails with a clear `422` at the API boundary instead of an opaque error deep in the domain layer; `provider` is validated against the 17 supported providers, and a test keeps the frontend/backend provider lists in sync (#1016)
+- Podcast episode listing now batch-fetches job statuses in one query instead of one per episode, speeding up notebooks with many episodes; podcast audio-file paths are additionally verified to stay within the podcasts folder before streaming/deleting (#1018)
+- Transformations no longer report success while silently losing their insight when the embedding job fails to queue: `Source.add_insight()` now raises on submission failure (handled by job-level retry), note auto-embedding degrades gracefully instead of turning a note save into a 500, and the explicit note-embed endpoint surfaces queue failures as errors (#1019)
+- Clearing a credential field in the edit dialog (Ollama/OpenAI-compatible `base_url`, Vertex `project`/`location`/`credentials_path`) now actually clears it. Two mirror-image bugs made it impossible: the frontend dropped emptied fields from the PUT body (`undefined` keys are stripped by `JSON.stringify`), and the API ignored explicit `null`s (`is not None` guards) — so the old value survived while the UI reported success. The frontend now sends explicit `null` and the API keys partial updates on field presence (`model_fields_set`) (#1046)
+
+### Security
+- Resolved dependency audit findings: added npm `overrides` for vulnerable transitive frontend packages (`ws`, `brace-expansion`, `ajv`, `@eslint/plugin-kit`, `postcss`) — `npm audit` now reports 0 vulnerabilities — refreshed `uv.lock` (`langsmith`, `pydantic-settings`, `pip`), and hardened external `window.open(..., '_blank')` calls with `noopener,noreferrer` (#962)
+- SurrealQL injection via record ids in `repo_relate()`/`repo_upsert()`/`repo_update()`: a crafted `notebook_id` on the save-insight-as-note flow could execute arbitrary SurrealQL. Record identifiers are now bound as query parameters, and the target notebook's existence is validated before relating (#1002)
+- The API password is now compared with `secrets.compare_digest()` instead of `!=`, closing a timing side-channel on authentication (#1003)
+- User-authored transformation prompts are no longer compiled as Jinja2 template source (a DoS vector via template loops); they are passed as plain variables into fixed developer-authored templates, so Jinja syntax inside a prompt renders as inert text. Output is unchanged for legitimate prompts (#1004)
+- SSRF protection on source-URL ingestion: adding a web-link source now runs the same `validate_url()` guard already used for credential URLs, rejecting internal/private/cloud-metadata addresses (#1005)
+- Provider-credential URLs are re-validated immediately before every outbound request (connection tests, model discovery and inference) instead of only at save time, closing a DNS-rebinding window; AWS's IPv6 metadata address was added to the blocklist (#1006)
+- The note/transformation markdown preview now sanitizes raw HTML via `rehype-sanitize`: `<iframe>`/`<script>`/`<style>` tags and `javascript:` URLs are stripped while math, syntax highlighting and GFM still render — closing an HTML-injection path via AI-generated note content (#1007)
+- Vertex credential-test errors no longer reveal whether a `credentials_path` file is missing, invalid JSON or wrong-shape JSON (a filesystem oracle); all three cases now return one generic message (#1012)
+- CORS no longer combines the wildcard origin with `allow_credentials=True` (which made Starlette reflect any Origin verbatim for credentialed requests); credentials are now only allowed when `CORS_ORIGINS` is explicitly configured (#1013)
+- Request bodies are now capped before auth and routing by a new `MaxBodySizeMiddleware` — default 100 MB, configurable via the new `OPEN_NOTEBOOK_MAX_UPLOAD_SIZE_MB` environment variable; chunked uploads are caught by a streaming byte-count (#1014)
+- Source upload hardening: unique filenames are claimed atomically (closing a TOCTOU race between the exists-check and the write), path-containment checks require a trailing separator so sibling directories can't pass, and the `notebooks`/`transformations` arrays on source creation are capped at 50 items (#1015)
+- API 500 responses from the sources and podcast endpoints no longer echo internal exception text (which could leak DB hostnames or connection details); responses use fixed generic messages and details remain in server logs (#1017)
+- `Credential.get_all()` no longer builds its ORDER BY clause from a raw f-string; ordering fields now go through the base-class allowlist validation (not reachable from the current API surface — defense in depth) (#1021)
+- The frontend runtime-config endpoint validates `Host` and `X-Forwarded-Proto` before using them to build the browser-facing API URL, preventing a spoofed Host header from redirecting browser API traffic (including the bearer token) to an attacker-controlled origin; malformed values fall back to localhost (#1024)
+- `docker-compose.yml` now binds SurrealDB's published port to `127.0.0.1` instead of all interfaces, so the database (root:root by default) is no longer reachable from other machines out of the box; a new `docker-compose.override.yml.example` shows how to re-expose it deliberately (#1025)
+- Forced **Pillow to 12.3.0**, resolving 6 open Dependabot advisories (3 high: PSD out-of-bounds writes, FITS GZIP decompression bomb; 3 moderate: PDF trailer DoS, font integer overflow, heap buffer overflow). The only blocker was moviepy's `pillow<12` cap (pulled in via podcast-creator) — moviepy only touches PIL in its video modules, which the audio-only podcast pipeline never imports, so a documented `[tool.uv] override-dependencies` entry forces the safe version until podcast-creator ships without moviepy (#1041)
+
+## [1.10.0] - 2026-06-17
+
+### Security
+- Bumped **Starlette to 1.2.1** and **FastAPI to 0.136.3** to address **CVE-2026-48710** ("BadHost"), a denial-of-service in Starlette's host header handling (#859)
+
+### Added
+- LaTeX math rendering in chat — inline (`$...$`) and display (`$$...$$`) expressions are now rendered with KaTeX (#606)
+- `NEXT_PUBLIC_API_TIMEOUT_MS` environment variable to configure the frontend API request timeout (default `600000` = 10 minutes; set `0` to disable). Lets slow/long-running chat models finish without editing source (#880)
+- Bulk chat-context actions in a notebook, via a "Context" menu in the Sources and Notes column headers — translated across all 14 locales (#223):
+  - Sources: "Include all (insights only)" (sources without insights are left out rather than forced to full), "Include all (full content)", and "Exclude all from context"
+  - Notes: "Include all in context" / "Exclude all from context"
+- **Turkish (tr-TR) localization** — the UI is now fully translated into Turkish (#871)
+
+### Changed
+- Failed source cards now show a prominent "Retry processing" button directly on the card instead of only inside the 3-dot dropdown; clicking it no longer also opens the source (the click was missing `stopPropagation`) (#726)
+- Docker base image updated to **Debian trixie** and **Node.js 22.x** (#914)
+
+### Fixed
+- Podcast generation now uses the notebook's real content. `Notebook.get_context()` was missing, so generation ran against empty context; it now assembles source and note content as expected (#864)
+- `PUT` profile handlers now use `model_dump(exclude_unset=True)`, so partial updates no longer overwrite unspecified fields with defaults (#860)
+- OpenRouter embedding models are now correctly recognized via their embedding modality (#842)
+- Search and Ask results now use page-level scrolling instead of being confined to a cramped, height-capped (`60vh`) bottom container, so the full result set is readable (#882)
+- `POST /sources/{id}/retry` no longer returns `400 "Source is not associated with any notebooks"` for every source; it now queries the `reference` graph edge by its `in`/`out` columns instead of a non-existent `source` column (#861)
+- `POST /sources/{id}/retry` no longer returns a `500` ("too many values to unpack") after successfully queuing the retry job; the command ID was being double-prefixed (`command:command:…`) before being saved to the source. Retrying a failed source now succeeds and updates the source's command reference
+- `GET /sources/{id}` for a missing or deleted source now returns `404` instead of `500`; the handler caught `NotFoundError` in its generic `except` and mapped it to a server error
+- Sources that fail to ingest (e.g. an unreachable or invalid URL) are now marked `failed` instead of silently saved as `completed` with the extraction error as their body. This means the "Retry processing" button (#726) actually appears for the most common failure mode; previously the job returned a failure payload but the command still completed, so the source never reached a retryable state (#726)
+- Text search no longer returns a 500 when SurrealDB's `search::highlight` hits a "position overflow" on large or multi-byte document chunks; it now falls back to vector search and returns results (#648)
+- `POST /api/search` now rejects a non-positive `limit` with a `422` instead of passing `LIMIT -1`/`LIMIT 0` to SurrealDB (which caused a 500 or a silently empty result set) (#863)
+- Ollama `num_ctx` credential override is now persisted. The `credential` table gained a flexible `config` object (migration 15) and provider-specific tuning options are stored there instead of being dropped by the SCHEMAFULL table; future per-credential options can be added without a schema migration (#875)
+- Worker no longer crashes on queued jobs from older versions; legacy embedding command aliases (`embed_single_item`, `embed_chunk`, `vectorize_source`) are registered and delegate to the current commands so stale queues drain cleanly (#695, #876)
+
+### Performance
+- Notebook source list no longer re-renders every `SourceCard` on unrelated state changes (layout toggles, context selection), and completed sources no longer each open a status-polling query. Both scaled with the number of sources and caused UI lag on large notebooks (#503)
+
+## [1.9.0] - 2026-06-02
+
+### Added
+- **New audio providers**, surfacing the capabilities added in Esperanto 2.21–2.22:
+  - **Mistral Voxtral** speech-to-text (`voxtral-*-latest`) and text-to-speech (`voxtral-mini-tts`), reusing the existing Mistral credential (#827)
+  - **Deepgram** text-to-speech (Aura voice catalog) as a new provider (`DEEPGRAM_API_KEY`) (#827)
+  - **xAI** text-to-speech (#827)
+  - **Google** speech-to-text & text-to-speech, **Vertex** text-to-speech, and **ElevenLabs** speech-to-text (Scribe), completing the audio provider matrix (#828)
+- Optional per-credential **`num_ctx`** (context window) override for Ollama models, configurable in Settings → API Keys and translated across all 13 locales (#825)
+- `OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE` environment variable to override the embedding batch size; default remains `50`. Helps with CPU-only local embedding and stricter OpenAI-compatible endpoints (#735)
+- `CORS_ORIGINS` environment variable to configure the API's allowed origins (comma-separated). Default remains `*` for backward compatibility; the API now logs a startup warning prompting users to set it for production deployments. Exception responses honor the configured origins when explicitly set (#585, #597, #730)
+- `OPEN_NOTEBOOK_MIN_CHUNK_SIZE` environment variable (default: 5 tokens) to filter out degenerate tiny chunks before embedding. Set to `0` to disable.
+
+### Changed
+- Bumped **Esperanto 2.20.0 → 2.22.0**. Beyond the new audio providers above, this inherits several upstream fixes and behavior changes (see below).
+
+### Inherited from Esperanto 2.21–2.22
+- **Fixed:** OpenRouter LLM and embedding requests now send a proper JSON body (previously sent a malformed form-encoded payload).
+- **Fixed:** OpenAI-compatible endpoints (e.g. llama.cpp) that return null embeddings now raise a clear, descriptive error instead of an opaque `TypeError`.
+- **Fixed:** Streaming tool calls now return proper `ToolCall` objects across Anthropic, Google, Vertex, and Ollama.
+- **Fixed:** `base_url` trailing slashes are normalized across providers, preventing double-slash URLs (and 301 redirects) for Ollama and other self-hosted endpoints.
+- **Fixed:** Ollama "thinking" models (e.g. Qwen) now merge their reasoning content correctly.
+- **Fixed:** Model discovery honors a custom `base_url` (LiteLLM/vLLM/OpenAI-compatible proxies).
+- **Behavior change:** the Ollama default context window (`num_ctx`) is now **8192** (was 128000) to avoid out-of-memory errors on consumer GPUs. Raise it per-credential via the new `num_ctx` field if your hardware allows.
+- **Behavior change:** the Google embedding default model is now `gemini-embedding-001` (the previous default, `text-embedding-004`, was removed from Google's API). If you used Google embeddings with the old default, re-create the model and re-embed your content (embedding dimensions changed).
+- **Fixed:** Google TTS default model updated to a currently-working preview model.
+
+### Fixed
+- URL source embedding no longer crashes with `TypeError: float() argument must be a string or a real number, not 'NoneType'` when header-based splitters emit single-character fragments from complex HTML pages (e.g. Wikipedia, Project Gutenberg). Such chunks are now filtered before being sent to the embedding provider (#764)
+- Language toggle now uses `t('common.german')` instead of a hardcoded "Deutsch" label, matching the pattern used by every other language entry (follow-up to #794)
+- Speech-to-text model connection tests now transcribe a short bundled speech clip instead of silence, so a passing test returns real text instead of a blank transcription (#838)
+
+## [1.8.5] - 2026-04-14
+
+### Changed
+- Embedding chunking is now token-based instead of character-based, improving chunk sizing consistency for CJK and mixed-language content (#542, #749)
+- `OPEN_NOTEBOOK_CHUNK_SIZE` and `OPEN_NOTEBOOK_CHUNK_OVERLAP` semantics changed from characters to tokens; default reduced from 1200 characters to 400 tokens to stay safely below the 512-token ceiling of BERT-family embedders (e.g. mxbai-embed-large) after accounting for tokenizer mismatch and splitter overshoot. Existing stored embeddings are unaffected; only new ingestions use the new chunking.
+
+### Fixed
+- Credentials endpoint no longer crashes (500) when encryption key doesn't match stored credentials (#740)
+- Broken credentials are now shown with a decryption warning and can still be deleted
+- DELETE endpoint for broken credentials supports model migration (`migrate_to` parameter)
+
+## [1.8.4] - 2026-04-09
+
+### Security
+- Fix Remote Code Execution (RCE) via Jinja2 Server-Side Template Injection in transformations (CVSS 9.2 Critical)
+- Fix arbitrary file write via path traversal in file upload (CVSS 7.0 High)
+- Fix arbitrary file read via Local File Inclusion in source creation (CVSS 8.2 High)
+
+### Dependencies
+- Bump ai-prompter to >=0.4.0 (uses Jinja2 SandboxedEnvironment to prevent SSTI)
+
 ## [1.8.3] - 2026-04-07
 
 ### Security

CLAUDE.md

@@ -1,218 +1 @@
-# Open Notebook - Root CLAUDE.md
-
-This file provides architectural guidance for contributors working on Open Notebook at the project level.
-
-## Project Overview
-
-**Open Notebook** is an open-source, privacy-focused alternative to Google's Notebook LM. It's an AI-powered research assistant enabling users to upload multi-modal content (PDFs, audio, video, web pages), generate intelligent notes, search semantically, chat with AI models, and produce professional podcasts—all with complete control over data and choice of AI providers.
-
-**Key Values**: Privacy-first, multi-provider AI support, fully self-hosted option, open-source transparency.
-
----
-
-## Three-Tier Architecture
-
-```
-┌─────────────────────────────────────────────────────────┐
-│              Frontend (React/Next.js)                    │
-│              frontend/ @ port 3000                       │
-├─────────────────────────────────────────────────────────┤
-│ - Notebooks, sources, notes, chat, podcasts, search UI  │
-│ - Zustand state management, TanStack Query (React Query)│
-│ - Shadcn/ui component library with Tailwind CSS         │
-└────────────────────────┬────────────────────────────────┘
-                         │ HTTP REST
-┌────────────────────────▼────────────────────────────────┐
-│              API (FastAPI)                              │
-│              api/ @ port 5055                           │
-├─────────────────────────────────────────────────────────┤
-│ - REST endpoints for notebooks, sources, notes, chat    │
-│ - LangGraph workflow orchestration                      │
-│ - Job queue for async operations (podcasts)             │
-│ - Multi-provider AI provisioning via Esperanto          │
-└────────────────────────┬────────────────────────────────┘
-                         │ SurrealQL
-┌────────────────────────▼────────────────────────────────┐
-│         Database (SurrealDB)                            │
-│         Graph database @ port 8000                      │
-├─────────────────────────────────────────────────────────┤
-│ - Records: Notebook, Source, Note, ChatSession, Credential│
-│ - Relationships: source-to-notebook, note-to-source     │
-│ - Vector embeddings for semantic search                 │
-└─────────────────────────────────────────────────────────┘
-```
-
----
-
-## Useful sources
-
-User documentation is at @docs/
-
-## Tech Stack
-
-### Frontend (`frontend/`)
-- **Framework**: Next.js 16 (React 19)
-- **Language**: TypeScript
-- **State Management**: Zustand
-- **Data Fetching**: TanStack Query (React Query)
-- **Styling**: Tailwind CSS + Shadcn/ui
-- **Build Tool**: Webpack (via Next.js)
-- **i18n compatible**: All front-end changes must also consider the translation keys
-
-### API Backend (`api/` + `open_notebook/`)
-- **Framework**: FastAPI 0.104+
-- **Language**: Python 3.11+
-- **Workflows**: LangGraph state machines
-- **Database**: SurrealDB async driver
-- **AI Providers**: Esperanto library (8+ providers: OpenAI, Anthropic, Google, Groq, Ollama, Mistral, DeepSeek, xAI)
-- **Job Queue**: Surreal-Commands for async jobs (podcasts)
-- **Logging**: Loguru
-- **Validation**: Pydantic v2
-- **Testing**: Pytest
-
-### Database
-- **SurrealDB**: Graph database with built-in embedding storage and vector search
-- **Schema Migrations**: Automatic on API startup via AsyncMigrationManager
-
-### Additional Services
-- **Content Processing**: content-core library (file/URL extraction)
-- **Prompts**: AI-Prompter with Jinja2 templating
-- **Podcast Generation**: podcast-creator library
-- **Embeddings**: Multi-provider via Esperanto
-
----
-
-## Architecture Highlights
-
-### 1. Async-First Design
-- All database queries, graph invocations, and API calls are async (await)
-- SurrealDB async driver with connection pooling
-- FastAPI handles concurrent requests efficiently
-
-### 2. LangGraph Workflows
-- **source.py**: Content ingestion (extract → embed → save)
-- **chat.py**: Conversational agent with message history
-- **ask.py**: Search + synthesis (retrieve relevant sources → LLM)
-- **transformation.py**: Custom transformations on sources
-- All use `provision_langchain_model()` for smart model selection
-
-### 3. Multi-Provider AI
-- **Esperanto library**: Unified interface to 8+ AI providers
-- **Credential system**: Individual encrypted credential records per provider; models link to credentials for direct config
-- **ModelManager**: Factory pattern with fallback logic; uses credential config when available, env vars as fallback
-- **Smart selection**: Detects large contexts, prefers long-context models
-- **Override support**: Per-request model configuration
-
-### 4. Database Schema
-- **Automatic migrations**: AsyncMigrationManager runs on API startup
-- **SurrealDB graph model**: Records with relationships and embeddings
-- **Vector search**: Built-in semantic search across all content
-- **Transactions**: Repo functions handle ACID operations
-
-### 5. Authentication
-- **Current**: Simple password middleware (insecure, dev-only)
-- **Production**: Replace with OAuth/JWT (see CONFIGURATION.md)
-
----
-
-## Important Quirks & Gotchas
-
-### API Startup
-- **Migrations run automatically** on startup; check logs for errors
-- **Must start API before UI**: UI depends on API for all data
-- **SurrealDB must be running**: API fails without database connection
-
-### Frontend-Backend Communication
-- **Base API URL**: Configured in `.env.local` (default: http://localhost:5055)
-- **CORS enabled**: Configured in `api/main.py` (allow all origins in dev)
-- **Rate limiting**: Not built-in; add at proxy layer for production
-
-### LangGraph Workflows
-- **Blocking operations**: Chat/podcast workflows may take minutes; no timeout
-- **State persistence**: Uses SQLite checkpoint storage in `/data/sqlite-db/`
-- **Model fallback**: If primary model fails, falls back to cheaper/smaller model
-
-### Podcast Generation
-- **Async job queue**: `podcast_service.py` submits jobs but doesn't wait
-- **Track status**: Use `/commands/{command_id}` endpoint to poll status
-- **TTS failures**: Fall back to silent audio if speech synthesis fails
-
-### Content Processing
-- **File extraction**: Uses content-core library; supports 50+ file types
-- **URL handling**: Extracts text + metadata from web pages
-- **Large files**: Content processing is sync; may block API briefly
-
----
-
-## Component References
-
-See dedicated CLAUDE.md files for detailed guidance:
-
-- **[frontend/CLAUDE.md](frontend/CLAUDE.md)**: React/Next.js architecture, state management, API integration
-- **[api/CLAUDE.md](api/CLAUDE.md)**: FastAPI structure, service pattern, endpoint development
-- **[open_notebook/CLAUDE.md](open_notebook/CLAUDE.md)**: Backend core, domain models, LangGraph workflows, AI provisioning
-- **[open_notebook/domain/CLAUDE.md](open_notebook/domain/CLAUDE.md)**: Data models, repository pattern, search functions
-- **[open_notebook/ai/CLAUDE.md](open_notebook/ai/CLAUDE.md)**: ModelManager, AI provider integration, Esperanto usage
-- **[open_notebook/graphs/CLAUDE.md](open_notebook/graphs/CLAUDE.md)**: LangGraph workflow design, state machines
-- **[open_notebook/database/CLAUDE.md](open_notebook/database/CLAUDE.md)**: SurrealDB operations, migrations, async patterns
-
----
-
-## Documentation Map
-
-- **[README.md](README.md)**: Project overview, features, quick start
-- **[docs/index.md](docs/index.md)**: Complete user & deployment documentation
-- **[CONFIGURATION.md](CONFIGURATION.md)**: Environment variables, model configuration
-- **[CONTRIBUTING.md](CONTRIBUTING.md)**: Contribution guidelines
-- **[MAINTAINER_GUIDE.md](MAINTAINER_GUIDE.md)**: Release & maintenance procedures
-
----
-
-## Testing Strategy
-
-- **Unit tests**: `tests/test_domain.py`, `test_models_api.py`
-- **Graph tests**: `tests/test_graphs.py` (workflow integration)
-- **Utils tests**: `tests/test_utils.py`, `tests/test_chunking.py`, `tests/test_embedding.py`
-- **Run all**: `uv run pytest tests/`
-- **Coverage**: Check with `pytest --cov`
-
----
-
-## Common Tasks
-
-### Add a New API Endpoint
-1. Create router in `api/routers/feature.py`
-2. Create service in `api/feature_service.py`
-3. Define schemas in `api/models.py`
-4. Register router in `api/main.py`
-5. Test via http://localhost:5055/docs
-
-### Add a New LangGraph Workflow
-1. Create `open_notebook/graphs/workflow_name.py`
-2. Define StateDict and node functions
-3. Build graph with `.add_node()` / `.add_edge()`
-4. Invoke in service: `graph.ainvoke({"input": ...}, config={"..."})`
-5. Test with sample data in `tests/`
-
-### Add Database Migration
-1. Create `migrations/XXX_description.surql`
-2. Write SurrealQL schema changes
-3. Create `migrations/XXX_description_down.surql` (optional rollback)
-4. API auto-detects on startup; migration runs if newer than recorded version
-
-### Deploy to Production
-1. Review [CONFIGURATION.md](CONFIGURATION.md) for security settings
-2. Use `make docker-release` for multi-platform image
-3. Push to Docker Hub / GitHub Container Registry
-4. Deploy `docker compose --profile multi up`
-5. Verify migrations via API logs
-
----
-
-## Support & Community
-
-- **Documentation**: https://open-notebook.ai
-- **Discord**: https://discord.gg/37XJPXfz2w
-- **Issues**: https://github.com/lfnovo/open-notebook/issues
-- **License**: MIT (see LICENSE)
-
+@AGENTS.md

CODE_OF_CONDUCT.md

@@ -0,0 +1,134 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, caste, color, religion, or sexual
+identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+- Demonstrating empathy and kindness toward other people
+- Being respectful of differing opinions, viewpoints, and experiences
+- Giving and gracefully accepting constructive feedback
+- Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+- Focusing on what is best not just for us as individuals, but for the overall
+  community
+
+Examples of unacceptable behavior include:
+
+- The use of sexualized language or imagery, and sexual attention or advances of
+  any kind
+- Trolling, insulting or derogatory comments, and personal or political attacks
+- Public or private harassment
+- Publishing others' private information, such as a physical or email address,
+  without their explicit permission
+- Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official email address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement through the
+contact form at **[open-notebook.ai](https://www.open-notebook.ai)** or via our
+**[Discord community](https://discord.gg/37XJPXfz2w)**.
+
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series of
+actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or permanent
+ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior, harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within the
+community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.1, available at
+[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
+
+Community Impact Guidelines were inspired by
+[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
+
+For answers to common questions about this code of conduct, see the FAQ at
+[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
+[https://www.contributor-covenant.org/translations][translations].
+
+[homepage]: https://www.contributor-covenant.org
+[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
+[Mozilla CoC]: https://github.com/mozilla/diversity
+[FAQ]: https://www.contributor-covenant.org/faq
+[translations]: https://www.contributor-covenant.org/translations

CONFIGURATION.md

@@ -13,7 +13,6 @@ All configuration documentation has been consolidated into the new documentation
 - **AI Provider Setup** → [AI Providers](docs/5-CONFIGURATION/ai-providers.md)
 - **Environment Variables Reference** → [Environment Reference](docs/5-CONFIGURATION/environment-reference.md)
 - **Database Configuration** → [Database Setup](docs/5-CONFIGURATION/database.md)
-- **Server Configuration** → [Server Settings](docs/5-CONFIGURATION/server.md)
 - **Security Setup** → [Security Configuration](docs/5-CONFIGURATION/security.md)
 - **Reverse Proxy** → [Reverse Proxy Setup](docs/5-CONFIGURATION/reverse-proxy.md)
 - **Advanced Tuning** → [Advanced Configuration](docs/5-CONFIGURATION/advanced.md)

CONTRIBUTING.md

@@ -12,15 +12,19 @@ All contribution guidelines have been consolidated into the new development docu
 
 - **Want to contribute code?** → [Contributing Guide](docs/7-DEVELOPMENT/contributing.md)
 - **Want to understand the architecture?** → [Architecture Overview](docs/7-DEVELOPMENT/architecture.md)
-- **Want to understand our design philosophy?** → [Design Principles](docs/7-DEVELOPMENT/design-principles.md)
+- **Want to understand what we're building?** → [Vision & Principles](VISION.md)
+- **Want to understand our engineering practices?** → [Design Principles](docs/7-DEVELOPMENT/design-principles.md)
 - **Are you a maintainer?** → [Maintainer Guide](docs/7-DEVELOPMENT/maintainer-guide.md)
-- **New developer?** → [Quick Start](docs/7-DEVELOPMENT/quick-start.md)
+- **Setting up locally?** → [Development Setup](docs/7-DEVELOPMENT/development-setup.md)
 
 ---
 
-## The Issue-First Workflow
+## The Contribution Workflow
 
-**TL;DR**: Create an issue first, get it assigned, THEN code.
+**TL;DR**: Feature requests, product/design/architecture proposals, and other non-trivial ideas
+start in [GitHub Discussions](https://github.com/lfnovo/open-notebook/discussions/new?category=ideas).
+Reproducible bugs start in Issues. Coding starts from an approved and assigned Issue. Small obvious
+fixes (typos, docs, tiny bugs) can still go straight to a PR.
 
 This prevents wasted effort and ensures your work aligns with the project. [See details →](docs/7-DEVELOPMENT/contributing.md)
 

Dockerfile

@@ -1,35 +1,57 @@
-# Build stage
-FROM python:3.12-slim-bookworm AS builder
+# Single source of truth for both published image variants:
+#   - regular (multi-container, SurrealDB external): default build / --target runtime
+#   - single-container (app + SurrealDB):            --target single
+# Shared stages below guarantee that fixes (tiktoken pre-cache, env defaults,
+# npm retry logic, ...) apply to both variants at once.
+
+# Stage 1: Frontend builder
+FROM node:22-slim AS frontend-builder
+WORKDIR /app/frontend
+
+# Copy dependency files first to leverage cache
+COPY frontend/package.json frontend/package-lock.json ./
+ARG NPM_REGISTRY=https://registry.npmjs.org/
+RUN npm config set registry ${NPM_REGISTRY} \
+ && npm config set fetch-retries 5 \
+ && npm config set fetch-retry-mintimeout 20000 \
+ && npm config set fetch-retry-maxtimeout 120000
+# Retry npm ci to survive transient registry ECONNRESETs, which are common on
+# the QEMU-emulated arm64 leg of the multi-arch build.
+RUN i=0; until npm ci; do \
+      i=$((i+1)); \
+      if [ "$i" -ge 5 ]; then echo "npm ci failed after $i attempts"; exit 1; fi; \
+      echo "npm ci failed (attempt $i); retrying in 15s"; sleep 15; \
+    done
+
+# Copy the rest of the frontend source and build
+COPY frontend/ ./
+RUN npm run build
+
+# Stage 2: Backend builder
+FROM python:3.12-slim-trixie AS backend-builder
+
+# Install build dependencies (uv downloads pre-built wheels for most packages)
+RUN apt-get update && apt-get upgrade -y && apt-get install -y --no-install-recommends \
+    build-essential \
+    && rm -rf /var/lib/apt/lists/*
 
 # Install uv using the official method
 COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
 
-# Install system dependencies required for building certain Python packages
-# Add Node.js 20.x LTS for building frontend
-# NOTE: gcc/g++/make removed - uv should download pre-built wheels. Add back if build fails.
-# NOTE: gcc/g++/make required for some python dependencies
-RUN apt-get update && apt-get install -y --no-install-recommends \
-    curl \
-    build-essential \
-    && curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
-    && apt-get install -y nodejs \
-    && rm -rf /var/lib/apt/lists/*
+WORKDIR /app
 
 # Set build optimization environment variables
-ENV MAKEFLAGS="-j$(nproc)"
 ENV PYTHONDONTWRITEBYTECODE=1
 ENV PYTHONUNBUFFERED=1
 ENV UV_COMPILE_BYTECODE=1
 ENV UV_LINK_MODE=copy
-
-# Set the working directory in the container to /app
-WORKDIR /app
+ENV UV_HTTP_TIMEOUT=120
 
 # Copy dependency files and minimal package structure first for better layer caching
 COPY pyproject.toml uv.lock ./
 COPY open_notebook/__init__.py ./open_notebook/__init__.py
 
-# Install dependencies with optimizations (this layer will be cached unless dependencies change)
+# Install dependencies (this layer is cached unless dependencies change)
 RUN uv sync --frozen --no-dev
 
 # Pre-download tiktoken encoding so the app works offline (issue #264).
@@ -40,79 +62,69 @@ ENV TIKTOKEN_CACHE_DIR=/app/tiktoken-cache
 RUN mkdir -p /app/tiktoken-cache && \
     .venv/bin/python -c "import tiktoken; tiktoken.get_encoding('o200k_base')"
 
-# Copy the rest of the application code
-COPY . /app
+# Stage 3: SurrealDB binary (pinned to v2 to match docker-compose.yml; used by the single target only)
+FROM surrealdb/surrealdb:v2 AS surreal-binary
 
-# Install frontend dependencies and build
-WORKDIR /app/frontend
-ARG NPM_REGISTRY=https://registry.npmjs.org/
-COPY frontend/package.json frontend/package-lock.json ./
-RUN npm config set registry ${NPM_REGISTRY}
-RUN npm ci
-COPY frontend/ ./
-RUN npm run build
-
-# Return to app root
-WORKDIR /app
-
-# Runtime stage
-FROM python:3.12-slim-bookworm AS runtime
+# Stage 4: Shared runtime base (everything common to both variants)
+FROM python:3.12-slim-trixie AS runtime-base
 
 # Install only runtime system dependencies (no build tools)
-# Add Node.js 20.x LTS for running frontend
+# Add Node.js 22.x LTS for running the frontend
 RUN apt-get update && apt-get upgrade -y && apt-get install -y --no-install-recommends \
     ffmpeg \
     supervisor \
     curl \
-    && curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
-    && apt-get install -y nodejs \
+    && curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
+    && apt-get install -y --no-install-recommends nodejs \
     && rm -rf /var/lib/apt/lists/*
 
 # Install uv using the official method
 COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
 
-# Set the working directory in the container to /app
 WORKDIR /app
 
-# Copy the virtual environment from builder stage
-COPY --from=builder /app/.venv /app/.venv
+# Copy the virtual environment from the backend builder
+COPY --from=backend-builder /app/.venv /app/.venv
 
-# Copy the source code (the rest)
+# Copy the source code
 COPY . /app
 
 # Copy pre-downloaded tiktoken encoding from builder (outside /data/ — volume-mount safe)
-COPY --from=builder /app/tiktoken-cache /app/tiktoken-cache
+COPY --from=backend-builder /app/tiktoken-cache /app/tiktoken-cache
+
+# Copy built frontend from standalone output
+COPY --from=frontend-builder /app/frontend/.next/standalone /app/frontend/
+COPY --from=frontend-builder /app/frontend/.next/static /app/frontend/.next/static
+COPY --from=frontend-builder /app/frontend/public /app/frontend/public
+COPY --from=frontend-builder /app/frontend/start-server.js /app/frontend/start-server.js
 
 # Ensure uv uses the existing venv without attempting network operations
 ENV UV_NO_SYNC=1
 ENV VIRTUAL_ENV=/app/.venv
 # Point the app at the pre-baked tiktoken encoding (see open_notebook/config.py)
 ENV TIKTOKEN_CACHE_DIR=/app/tiktoken-cache
+# Bind the API to all interfaces (IPv4). Set API_HOST=:: for IPv6 dual-stack environments
+ENV API_HOST=0.0.0.0
 
-# Bind Next.js to all interfaces (required for Docker networking and reverse proxies)
-ENV HOSTNAME=0.0.0.0
+# Caches for the opt-in heavy extraction runtimes (Docling, Crawl4AI local).
+# These live UNDER /app/data so the user's volume mount persists them across
+# container restarts and upgrades: wheels (uv), the Chromium browser (playwright)
+# and Docling's ML models (huggingface) are downloaded once, then reused.
+# See scripts/docker-entrypoint.sh and docs/7-DEVELOPMENT/decisions/ADR-007-optin-runtimes.md.
+ENV UV_CACHE_DIR=/app/data/.cache/uv
+ENV PLAYWRIGHT_BROWSERS_PATH=/app/data/.cache/playwright
+ENV HF_HOME=/app/data/.cache/huggingface
 
-# Copy built frontend from builder stage
-COPY --from=builder /app/frontend/.next/standalone /app/frontend/
-COPY --from=builder /app/frontend/.next/static /app/frontend/.next/static
-COPY --from=builder /app/frontend/public /app/frontend/public
-COPY --from=builder /app/frontend/start-server.js /app/frontend/start-server.js
+# Data directory (volume-mounted by users) and supervisor log directory
+RUN mkdir -p /app/data /var/log/supervisor \
+    && chmod +x /app/scripts/wait-for-api.sh /app/scripts/docker-entrypoint.sh
+
+# Copy supervisord configuration (shared programs: api, worker, frontend)
+COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf
 
 # Expose ports for Frontend and API
 EXPOSE 8502 5055
 
-RUN mkdir -p /app/data
-
-# Copy and make executable the wait-for-api script
-COPY scripts/wait-for-api.sh /app/scripts/wait-for-api.sh
-RUN chmod +x /app/scripts/wait-for-api.sh
-
-# Copy supervisord configuration
-COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf
-
-# Create log directories
-RUN mkdir -p /var/log/supervisor
-
 # Runtime API URL Configuration
 # The API_URL environment variable can be set at container runtime to configure
 # where the frontend should connect to the API. This allows the same Docker image
@@ -123,4 +135,24 @@ RUN mkdir -p /var/log/supervisor
 #
 # Example: docker run -e API_URL=https://your-domain.com/api ...
 
+# The entrypoint installs any opt-in heavy runtimes (Docling, Crawl4AI local)
+# enabled via OPEN_NOTEBOOK_ENABLE_* before handing off to CMD (supervisord).
+ENTRYPOINT ["/app/scripts/docker-entrypoint.sh"]
 CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/conf.d/supervisord.conf"]
+
+# Stage 5: Single-container variant (adds SurrealDB on top of the shared runtime)
+# Build with: docker build --target single .
+FROM runtime-base AS single
+
+# Install SurrealDB (copied from pinned v2 image to match docker-compose.yml)
+COPY --from=surreal-binary /surreal /usr/local/bin/surreal
+
+# SurrealDB data directory (volume-mounted by users)
+RUN mkdir -p /mydata
+
+# Enable the surrealdb program in supervisord (appended to the shared config)
+RUN cat /app/supervisord.surrealdb.conf >> /etc/supervisor/conf.d/supervisord.conf
+
+# Stage 6 (default): Regular multi-container image (SurrealDB runs externally).
+# Kept last so a plain `docker build .` produces this variant.
+FROM runtime-base AS runtime

Dockerfile.single

@@ -1,97 +0,0 @@
-# Stage 1: Frontend Builder
-FROM node:20-slim AS frontend-builder
-WORKDIR /app/frontend
-
-# Copy dependency files first to leverage cache
-COPY frontend/package.json frontend/package-lock.json ./
-ARG NPM_REGISTRY=https://registry.npmjs.org/
-RUN npm config set registry ${NPM_REGISTRY}
-RUN npm ci
-
-# Copy the rest of the frontend source
-COPY frontend/ ./
-# Build the frontend
-RUN npm run build
-
-# Stage 2: SurrealDB binary (pinned to v2 to match docker-compose.yml)
-FROM surrealdb/surrealdb:v2 AS surreal-binary
-
-# Stage 4: Backend Builder
-FROM python:3.12-slim-bookworm AS backend-builder
-# Install build dependencies
-RUN apt-get update && apt-get upgrade -y && apt-get install -y --no-install-recommends build-essential && rm -rf /var/lib/apt/lists/*
-# Install uv
-COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
-WORKDIR /app
-
-# Set build optimization environment variables
-ENV UV_HTTP_TIMEOUT=120
-
-# Copy dependency files first
-COPY pyproject.toml uv.lock ./
-COPY open_notebook/__init__.py ./open_notebook/__init__.py
-# Install dependencies
-RUN uv sync --frozen --no-dev
-
-# Pre-download tiktoken encoding so the app works offline (issue #264).
-# /app/tiktoken-cache is intentionally outside /app/data/ so that volume mounts
-# of /app/data (for user data persistence) do not hide the pre-baked encoding.
-# config.py reads TIKTOKEN_CACHE_DIR from the environment to pick up this path.
-ENV TIKTOKEN_CACHE_DIR=/app/tiktoken-cache
-RUN mkdir -p /app/tiktoken-cache && \
-    .venv/bin/python -c "import tiktoken; tiktoken.get_encoding('o200k_base')"
-
-# Stage 5: Runtime
-FROM python:3.12-slim-bookworm AS runtime
-
-# Install runtime dependencies
-RUN apt-get update && apt-get upgrade -y && apt-get install -y \
-    ffmpeg \
-    supervisor \
-    curl \
-    && curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
-    && apt-get install -y nodejs \
-    && rm -rf /var/lib/apt/lists/*
-
-# Install SurrealDB (copied from pinned v2 image to match docker-compose.yml)
-COPY --from=surreal-binary /surreal /usr/local/bin/surreal
-
-# Install uv (optional but helpful for some scripts)
-COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
-
-WORKDIR /app
-
-# Copy backend virtualenv and source code
-COPY --from=backend-builder /app/.venv /app/.venv
-COPY . /app/
-
-# Copy pre-downloaded tiktoken encoding from builder (outside /data/ — volume-mount safe)
-COPY --from=backend-builder /app/tiktoken-cache /app/tiktoken-cache
-
-# Copy built frontend from standalone output
-COPY --from=frontend-builder /app/frontend/.next/standalone /app/frontend/
-COPY --from=frontend-builder /app/frontend/.next/static /app/frontend/.next/static
-COPY --from=frontend-builder /app/frontend/public /app/frontend/public
-
-# Bind Next.js to all interfaces (required for Docker networking and reverse proxies)
-ENV HOSTNAME=0.0.0.0
-# Point the app at the pre-baked tiktoken encoding (see open_notebook/config.py)
-ENV TIKTOKEN_CACHE_DIR=/app/tiktoken-cache
-
-# Setup directories and permissions
-RUN mkdir -p /app/data /mydata
-
-# Ensure wait-for-api script is executable
-RUN chmod +x /app/scripts/wait-for-api.sh
-
-# Copy supervisord configuration
-COPY supervisord.single.conf /etc/supervisor/conf.d/supervisord.conf
-
-# Create log directories
-RUN mkdir -p /var/log/supervisor
-
-# Expose ports
-EXPOSE 8502 5055
-
-# Set startup command
-CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/conf.d/supervisord.conf"]

Makefile

@@ -1,6 +1,7 @@
 .PHONY: run frontend check ruff database lint api start-all stop-all status clean-cache worker worker-start worker-stop worker-restart
 .PHONY: docker-buildx-prepare docker-buildx-clean docker-buildx-reset
 .PHONY: docker-push docker-push-latest docker-release docker-build-local tag export-docs
+.PHONY: release-test release-stack release-stack-down
 
 # Get version from pyproject.toml
 VERSION := $(shell grep -m1 version pyproject.toml | cut -d'"' -f2)
@@ -43,6 +44,25 @@ docker-buildx-clean:
 docker-buildx-reset: docker-buildx-clean docker-buildx-prepare
 	@echo "✅ Buildx reset complete!"
 
+# === Release Testing (see .github/RELEASE_PROCESS.md) ===
+
+# Automated image gate: fresh install + upgrade against real images.
+# Usage: make release-test TAG=1.12.0 OLD_TAG=1.11.0
+release-test:
+	@test -n "$(TAG)" || (echo "usage: make release-test TAG=<new> [OLD_TAG=<previous>]"; exit 1)
+	bash scripts/release-test/release-image-test.sh all \
+		"$(DOCKERHUB_IMAGE):$(TAG)" \
+		$(if $(OLD_TAG),"$(DOCKERHUB_IMAGE):$(OLD_TAG)")
+
+# Browsable RC stack for manual verification (optionally with a data dump).
+# Usage: make release-stack TAG=1.12.0 [DUMP=/tmp/dev-dump.surql]
+release-stack:
+	@test -n "$(TAG)" || (echo "usage: make release-stack TAG=<tag> [DUMP=<dump.surql>]"; exit 1)
+	bash scripts/release-test/rc-stack.sh up "$(TAG)" $(DUMP)
+
+release-stack-down:
+	bash scripts/release-test/rc-stack.sh down "$(or $(TAG),unused)"
+
 # === Docker Build Targets ===
 
 # Build production image for local platform only (no push)
@@ -70,7 +90,7 @@ docker-push: docker-buildx-prepare
 	docker buildx build --pull \
 		--platform $(PLATFORMS) \
 		--progress=plain \
-		-f Dockerfile.single \
+		--target single \
 		-t $(DOCKERHUB_IMAGE):$(VERSION)-single \
 		-t $(GHCR_IMAGE):$(VERSION)-single \
 		--push \
@@ -100,7 +120,7 @@ docker-push-latest: docker-buildx-prepare
 	docker buildx build --pull \
 		--platform $(PLATFORMS) \
 		--progress=plain \
-		-f Dockerfile.single \
+		--target single \
 		-t $(DOCKERHUB_IMAGE):$(VERSION)-single \
 		-t $(DOCKERHUB_IMAGE):v1-latest-single \
 		-t $(GHCR_IMAGE):$(VERSION)-single \
@@ -127,10 +147,10 @@ tag:
 
 
 dev:
-	docker compose -f docker-compose.dev.yml up --build 
+	docker compose -f examples/docker-compose-dev.yml --project-directory . up --build
 
 full:
-	docker compose -f docker-compose.full.yml up --build 
+	docker compose -f examples/docker-compose-full-local.yml --project-directory . up --build
 
 
 api:
@@ -142,7 +162,7 @@ worker: worker-start
 
 worker-start:
 	@echo "Starting surreal-commands worker..."
-	uv run --env-file .env surreal-commands-worker --import-modules commands
+	uv run --env-file .env surreal-commands-worker --import-modules commands --max-tasks "$${OPEN_NOTEBOOK_WORKER_MAX_TASKS:-5}"
 
 worker-stop:
 	@echo "Stopping surreal-commands worker..."
@@ -156,13 +176,13 @@ worker-restart: worker-stop
 start-all:
 	@echo "🚀 Starting Open Notebook (Database + API + Worker + Frontend)..."
 	@echo "📊 Starting SurrealDB..."
-	@docker compose -f docker-compose.dev.yml up -d surrealdb
+	@docker compose up -d surrealdb
 	@sleep 3
 	@echo "🔧 Starting API backend..."
 	@uv run run_api.py &
 	@sleep 3
 	@echo "⚙️ Starting background worker..."
-	@uv run --env-file .env surreal-commands-worker --import-modules commands &
+	@uv run --env-file .env surreal-commands-worker --import-modules commands --max-tasks "$${OPEN_NOTEBOOK_WORKER_MAX_TASKS:-5}" &
 	@sleep 2
 	@echo "🌐 Starting Next.js frontend..."
 	@echo "✅ All services started!"

README.dev.md

@@ -1,449 +1,10 @@
 # Developer Guide
 
-This guide is for developers working on Open Notebook. For end-user documentation, see [README.md](README.md) and [docs/](docs/).
-
-## Quick Start for Development
-
-```bash
-# 1. Clone and setup
-git clone https://github.com/lfnovo/open-notebook.git
-cd open-notebook
-
-# 2. Copy environment files
-cp .env.example .env
-cp .env.example docker.env
-
-# 3. Install dependencies
-uv sync
-
-# 4. Start all services (recommended for development)
-make start-all
-```
-
-## Development Workflows
-
-### When to Use What?
-
-| Workflow | Use Case | Speed | Production Parity |
-|----------|----------|-------|-------------------|
-| **Local Services** (`make start-all`) | Day-to-day development, fastest iteration | ⚡⚡⚡ Fast | Medium |
-| **Docker Compose** (`make dev`) | Testing containerized setup | ⚡⚡ Medium | High |
-| **Local Docker Build** (`make docker-build-local`) | Testing Dockerfile changes | ⚡ Slow | Very High |
-| **Multi-platform Build** (`make docker-push`) | Publishing releases | 🐌 Very Slow | Exact |
-
----
-
-## 1. Local Development (Recommended)
-
-**Best for:** Daily development, hot reload, debugging
-
-### Setup
-
-```bash
-# Start database
-make database
-
-# Start all services (DB + API + Worker + Frontend)
-make start-all
-```
-
-### What This Does
-
-1. Starts SurrealDB in Docker (port 8000)
-2. Starts FastAPI backend (port 5055)
-3. Starts background worker (surreal-commands)
-4. Starts Next.js frontend (port 3000)
-
-### Individual Services
-
-```bash
-# Just the database
-make database
-
-# Just the API
-make api
-
-# Just the frontend
-make frontend
-
-# Just the worker
-make worker
-```
-
-### Checking Status
-
-```bash
-# See what's running
-make status
-
-# Stop everything
-make stop-all
-```
-
-### Advantages
-- ✅ Fastest iteration (hot reload)
-- ✅ Easy debugging (direct process access)
-- ✅ Low resource usage
-- ✅ Direct log access
-
-### Disadvantages
-- ❌ Doesn't test Docker build
-- ❌ Environment may differ from production
-- ❌ Requires local Python/Node setup
-
----
-
-## 2. Docker Compose Development
-
-**Best for:** Testing containerized setup, CI/CD verification
-
-```bash
-# Start with dev profile
-make dev
-
-# Or full stack
-make full
-```
-
-### Configuration Files
-
-- `docker-compose.dev.yml` - Development setup
-- `docker-compose.full.yml` - Full stack setup
-- `docker-compose.yml` - Base configuration
-
-### Advantages
-- ✅ Closer to production environment
-- ✅ Isolated dependencies
-- ✅ Easy to share exact environment
-
-### Disadvantages
-- ❌ Slower rebuilds
-- ❌ More complex debugging
-- ❌ Higher resource usage
-
----
-
-## 3. Testing Production Docker Images
-
-**Best for:** Verifying Dockerfile changes before publishing
-
-### Build Locally
-
-```bash
-# Build production image for your platform only
-make docker-build-local
-```
-
-This creates two tags:
-- `lfnovo/open_notebook:<version>` (from pyproject.toml)
-- `lfnovo/open_notebook:local`
-
-### Run Locally
-
-```bash
-docker run -p 5055:5055 -p 3000:3000 lfnovo/open_notebook:local
-```
-
-### When to Use
-- ✅ Before pushing to registry
-- ✅ Testing Dockerfile changes
-- ✅ Debugging production-specific issues
-- ✅ Verifying build process
-
----
-
-## 4. Publishing Docker Images
-
-### Workflow
-
-```bash
-# 1. Test locally first
-make docker-build-local
-
-# 2. If successful, push version tag (no latest update)
-make docker-push
-
-# 3. Test the pushed version in staging/production
-
-# 4. When ready, promote to latest
-make docker-push-latest
-```
-
-### Available Commands
-
-| Command | What It Does | Updates Latest? |
-|---------|--------------|-----------------|
-| `make docker-build-local` | Build for current platform only | No registry push |
-| `make docker-push` | Push version tags to registries | ❌ No |
-| `make docker-push-latest` | Push version + update v1-latest | ✅ Yes |
-| `make docker-release` | Full release (same as docker-push-latest) | ✅ Yes |
-
-### Publishing Details
-
-- **Platforms:** `linux/amd64`, `linux/arm64`
-- **Registries:** Docker Hub + GitHub Container Registry
-- **Image Variants:** Regular + Single-container (`-single`)
-- **Version Source:** `pyproject.toml`
-
-### Creating Git Tags
-
-```bash
-# Create and push git tag matching pyproject.toml version
-make tag
-```
-
----
-
-## Code Quality
-
-```bash
-# Run linter with auto-fix
-make ruff
-
-# Run type checking
-make lint
-
-# Run tests
-uv run pytest tests/
-
-# Clean cache directories
-make clean-cache
-```
-
----
-
-## Common Development Tasks
-
-### Adding a New Feature
-
-1. Create feature branch
-2. Develop using `make start-all`
-3. Write tests
-4. Run `make ruff` and `make lint`
-5. Test with `make docker-build-local`
-6. Create PR
-
-### Fixing a Bug
-
-1. Reproduce locally with `make start-all`
-2. Add test case demonstrating bug
-3. Fix the bug
-4. Verify test passes
-5. Check with `make docker-build-local`
-
-### Updating Dependencies
-
-```bash
-# Add Python dependency
-uv add package-name
-
-# Update dependencies
-uv sync
-
-# Frontend dependencies
-cd frontend && npm install package-name
-```
-
-### Adding a New Language (i18n)
-
-Open Notebook supports internationalization. To add a new language:
-
-1. **Create locale file**: Copy an existing locale as template
-   ```bash
-   cp frontend/src/lib/locales/en-US/index.ts frontend/src/lib/locales/pt-BR/index.ts
-   ```
-
-2. **Translate all strings** in the new file. The structure includes:
-   - `common`: Shared UI elements (buttons, labels)
-   - `notebooks`, `sources`, `notes`: Feature-specific strings
-   - `chat`, `search`, `podcasts`: Module-specific strings
-   - `apiErrors`: Error message translations
-
-3. **Register the locale** in `frontend/src/lib/locales/index.ts`:
-   ```typescript
-   import { ptBR } from './pt-BR'
-
-   export const locales = {
-     'en-US': enUS,
-     'zh-CN': zhCN,
-     'zh-TW': zhTW,
-     'pt-BR': ptBR,  // Add your locale
-   }
-   ```
-
-4. **Add date-fns locale** in `frontend/src/lib/utils/date-locale.ts`:
-   ```typescript
-   import { zhCN, enUS, zhTW, ptBR } from 'date-fns/locale'
-
-   const LOCALE_MAP: Record<string, Locale> = {
-     'zh-CN': zhCN,
-     'zh-TW': zhTW,
-     'en-US': enUS,
-     'pt-BR': ptBR,  // Add your locale
-   }
-   ```
-
-5. **Test**: Switch languages using the language toggle in the UI header.
-
-### Database Migrations
-
-Database migrations run **automatically** when the API starts.
-
-1. Create migration file: `migrations/XXX_description.surql`
-2. Write SurrealQL schema changes
-3. (Optional) Create rollback: `migrations/XXX_description_down.surql`
-4. Restart API - migration runs on startup
-
----
-
-## Troubleshooting
-
-### Services Won't Start
-
-```bash
-# Check status
-make status
-
-# Check database
-docker compose ps surrealdb
-
-# View logs
-docker compose logs surrealdb
-
-# Restart everything
-make stop-all
-make start-all
-```
-
-### Port Already in Use
-
-```bash
-# Find process using port
-lsof -i :5055
-lsof -i :3000
-lsof -i :8000
-
-# Kill stuck processes
-make stop-all
-```
-
-### Database Connection Issues
-
-```bash
-# Verify SurrealDB is running
-docker compose ps surrealdb
-
-# Check connection settings in .env
-cat .env | grep SURREAL
-```
-
-### Docker Build Fails
-
-```bash
-# Clean Docker cache
-docker builder prune
-
-# Reset buildx
-make docker-buildx-reset
-
-# Try local build first
-make docker-build-local
-```
-
----
-
-## Project Structure
-
-```
-open-notebook/
-├── api/                    # FastAPI backend
-├── frontend/               # Next.js React frontend
-├── open_notebook/          # Python core library
-│   ├── domain/            # Domain models
-│   ├── graphs/            # LangGraph workflows
-│   ├── ai/                # AI provider integration
-│   └── database/          # SurrealDB operations
-├── migrations/             # Database migrations
-├── tests/                  # Test suite
-├── docs/                   # User documentation
-└── Makefile               # Development commands
-```
-
-See component-specific CLAUDE.md files for detailed architecture:
-- [frontend/CLAUDE.md](frontend/CLAUDE.md)
-- [api/CLAUDE.md](api/CLAUDE.md)
-- [open_notebook/CLAUDE.md](open_notebook/CLAUDE.md)
-
----
-
-## Environment Variables
-
-### Required for Local Development
-
-```bash
-# .env file
-SURREAL_URL=ws://localhost:8000
-SURREAL_USER=root
-SURREAL_PASS=root
-SURREAL_DB=open_notebook
-SURREAL_NS=production
-
-# AI Provider (at least one required)
-OPENAI_API_KEY=sk-...
-# OR
-ANTHROPIC_API_KEY=sk-ant-...
-# OR configure other providers (see docs/5-CONFIGURATION/)
-```
-
-See [docs/5-CONFIGURATION/](docs/5-CONFIGURATION/) for complete configuration guide.
-
----
-
-## Performance Tips
-
-### Speed Up Local Development
-
-1. **Use `make start-all`** instead of Docker for daily work
-2. **Keep SurrealDB running** between sessions (`make database`)
-3. **Use `make docker-build-local`** only when testing Dockerfile changes
-4. **Skip multi-platform builds** until ready to publish
-
-### Reduce Resource Usage
-
-```bash
-# Stop unused services
-make stop-all
-
-# Clean up Docker
-docker system prune -a
-
-# Clean Python cache
-make clean-cache
-```
-
----
-
-## TODO: Sections to Add
-
-- [ ] Frontend development guide (hot reload, component structure)
-- [ ] API development guide (adding endpoints, services)
-- [ ] LangGraph workflow development
-- [ ] Testing strategy and coverage
-- [ ] Debugging tips (VSCode/PyCharm setup)
-- [ ] CI/CD pipeline overview
-- [ ] Release process checklist
-- [ ] Common error messages and solutions
-
----
-
-## Resources
-
-- **Documentation:** https://open-notebook.ai
-- **Discord:** https://discord.gg/37XJPXfz2w
-- **Issues:** https://github.com/lfnovo/open-notebook/issues
-- **Contributing:** [CONTRIBUTING.md](CONTRIBUTING.md)
-- **Maintainer Guide:** [MAINTAINER_GUIDE.md](MAINTAINER_GUIDE.md)
-
----
-
-**Last Updated:** January 2025
+Developer documentation lives in [docs/7-DEVELOPMENT/](docs/7-DEVELOPMENT/index.md).
+
+- **Run it from source** → [Development Setup](docs/7-DEVELOPMENT/development-setup.md)
+- **Contribute** → [Contributing Guide](docs/7-DEVELOPMENT/contributing.md) (workflow, commit and CHANGELOG conventions, checks CI runs)
+- **Make a common change** → [Change Playbooks](docs/7-DEVELOPMENT/change-playbooks.md)
+- **Understand the code** → [Architecture](docs/7-DEVELOPMENT/architecture.md)
+- **Coding-agent rules** → [AGENTS.md](AGENTS.md)
+- **Publish Docker images** → [Release Process](.github/RELEASE_PROCESS.md)

README.md

@@ -23,6 +23,8 @@
     <br />
     <a href="https://www.open-notebook.ai"><strong>Checkout our website »</strong></a>
     <br />
+    Follow <a href="https://x.com/lfnovo">@lfnovo on X</a> for updates
+    <br />
     <br />
     <a href="docs/0-START-HERE/index.md">📚 Get Started</a>
     ·
@@ -58,12 +60,12 @@ In a world dominated by Artificial Intelligence, having the ability to think
 
 **Open Notebook empowers you to:**
 - 🔒 **Control your data** - Keep your research private and secure
-- 🤖 **Choose your AI models** - Support for 18+ providers including OpenAI, Anthropic, Ollama, LM Studio, and more
+- 🤖 **Choose your AI models** - 20+ providers including OpenAI, Anthropic, Google, Mistral, Ollama and any OpenAI-compatible server (LM Studio, vLLM)
 - 📚 **Organize multi-modal content** - PDFs, videos, audio, web pages, and more
 - 🎙️ **Generate professional podcasts** - Advanced multi-speaker podcast generation
 - 🔍 **Search intelligently** - Full-text and vector search across all your content
 - 💬 **Chat with context** - AI conversations powered by your research
-- 🌐 **Multi-language UI** - English, Portuguese, Chinese (Simplified & Traditional), Japanese, Russian, and Bengali support
+- 🌐 **Multi-language UI** - English, Bengali, Catalan, Chinese (Simplified & Traditional), French, German, Italian, Japanese, Polish, Portuguese, Russian, Spanish and Turkish
 
 Learn more about our project at [https://www.open-notebook.ai](https://www.open-notebook.ai)
 
@@ -74,12 +76,12 @@ Learn more about our project at [https://www.open-notebook.ai](https://www.open-
 | Feature | Open Notebook | Google Notebook LM | Advantage |
 |---------|---------------|--------------------|-----------|
 | **Privacy & Control** | Self-hosted, your data | Google cloud only | Complete data sovereignty |
-| **AI Provider Choice** | 18+ providers (OpenAI, Anthropic, Ollama, LM Studio, etc.) | Google models only | Flexibility and cost optimization |
+| **AI Provider Choice** | 20+ providers, cloud or local (OpenAI, Anthropic, Google, Ollama, LM Studio, etc.) | Google models only | Flexibility and cost optimization |
 | **Podcast Speakers** | 1-4 speakers with custom profiles | 2 speakers only | Extreme flexibility |
 | **Content Transformations** | Custom and built-in | Limited options | Unlimited processing power |
 | **API Access** | Full REST API | No API | Complete automation |
 | **Deployment** | Docker, cloud, or local | Google hosted only | Deploy anywhere |
-| **Citations** | Basic references (will improve) | Comprehensive with sources | Research integrity |
+| **Citations** | Answers cite the sources, notes and insights they used | Comprehensive with sources | Research integrity |
 | **Customization** | Open source, fully customizable | Closed system | Unlimited extensibility |
 | **Cost** | Pay only for AI usage | Free tier + Monthly subscription | Transparent and controllable |
 
@@ -94,86 +96,59 @@ Learn more about our project at [https://www.open-notebook.ai](https://www.open-
 
 [![Python][Python]][Python-url] [![Next.js][Next.js]][Next-url] [![React][React]][React-url] [![SurrealDB][SurrealDB]][SurrealDB-url] [![LangChain][LangChain]][LangChain-url]
 
-## 🚀 Quick Start (2 Minutes)
+## 🚀 Quick Start
 
 ### Prerequisites
-- [Docker Desktop](https://www.docker.com/products/docker-desktop/) installed
-- That's it! (API keys configured later in the UI)
+- [Docker Desktop](https://www.docker.com/products/docker-desktop/) (macOS, Windows) or Docker Engine with the Compose plugin (Linux)
+- An API key from an AI provider, or a local model server such as Ollama. You add it in the UI, not in config files.
 
-### Step 1: Get docker-compose.yml
-
-**Option A:** Download directly
+### Step 1: Download docker-compose.yml
 ```bash
+mkdir open-notebook && cd open-notebook
 curl -o docker-compose.yml https://raw.githubusercontent.com/lfnovo/open-notebook/main/docker-compose.yml
 ```
 
-**Option B:** Create the file manually
-Copy this into a new file called `docker-compose.yml`:
+This is the official [docker-compose.yml](docker-compose.yml) from this repository. It runs two services: `surrealdb` (the database) and `open_notebook` (UI, API and background worker).
 
-```yaml
-services:
-  surrealdb:
-    image: surrealdb/surrealdb:v2
-    command: start --log info --user root --pass root rocksdb:/mydata/mydatabase.db
-    user: root
-    ports:
-      - "8000:8000"
-    volumes:
-      - ./surreal_data:/mydata
-    restart: always
-
-  open_notebook:
-    image: lfnovo/open_notebook:v1-latest
-    ports:
-      - "8502:8502"
-      - "5055:5055"
-    environment:
-      - OPEN_NOTEBOOK_ENCRYPTION_KEY=change-me-to-a-secret-string
-      - SURREAL_URL=ws://surrealdb:8000/rpc
-      - SURREAL_USER=root
-      - SURREAL_PASSWORD=root
-      - SURREAL_NAMESPACE=open_notebook
-      - SURREAL_DATABASE=open_notebook
-    volumes:
-      - ./notebook_data:/app/data
-    depends_on:
-      - surrealdb
-    restart: always
-```
-
-### Step 2: Set Your Encryption Key
+### Step 2: Set your encryption key
 Edit `docker-compose.yml` and change this line:
 ```yaml
 - OPEN_NOTEBOOK_ENCRYPTION_KEY=change-me-to-a-secret-string
 ```
-to any secret value (e.g., `my-super-secret-key-123`)
+to a long random secret you generate yourself (for example with `openssl rand -hex 32`; [other options](docs/1-INSTALLATION/docker-compose.md#step-2-set-your-encryption-key)). It encrypts the API keys you store; keep it.
 
-### Step 3: Start Services
+> **Shared network or server?** The UI (`8502`) and API (`5055`) are published on all network interfaces and there is no password by default. If other devices can reach this machine, change those two port lines to `"127.0.0.1:8502:8502"` and `"127.0.0.1:5055:5055"`, or add `- OPEN_NOTEBOOK_PASSWORD=your-password` to the `environment:` block, before starting.
+
+### Step 3: Start services
 ```bash
 docker compose up -d
 ```
 
-Wait 15-20 seconds, then open: **http://localhost:8502**
+Wait about 30 seconds, then open: **http://localhost:8502**
 
-### Step 4: Configure AI Provider
-1. Go to **Settings** → **API Keys**
-2. Click **Add Credential**
-3. Choose your provider (OpenAI, Anthropic, Google, etc.)
-4. Paste your API key and click **Save**
-5. Click **Test Connection** → **Discover Models** → **Register Models**
+### Step 4: Connect an AI provider
+1. In the sidebar, open **Models** (under Manage).
+2. Find your provider (OpenAI, Anthropic, Google, etc.) and click **Add Configuration**.
+3. Enter a name and your API key (local providers such as Ollama need a **Base URL** instead), then click **Add Configuration**.
+4. Click **Test** on the new configuration to check the connection.
+5. Click **Models**, choose a **Model Type**, tick the models to add and click **Add**. Add at least one **Language** model and one **Embedding** model.
+6. Under **Default Model Assignments**, click **Auto-assign Defaults** (or pick the models yourself).
 
-Done! You're ready to create your first notebook.
+Done! Create a notebook, add a source and start chatting. Full walkthrough: [Connect a provider](docs/4-AI-PROVIDERS/index.md#connect-a-provider).
 
 > **Need an API key?** Get one from:
-> [OpenAI](https://platform.openai.com/api-keys) · [Anthropic](https://console.anthropic.com/) · [Google](https://aistudio.google.com/) · [Groq](https://console.groq.com/) (free tier)
+> [OpenAI](https://platform.openai.com/api-keys) · [Anthropic](https://console.anthropic.com/settings/keys) · [Google](https://aistudio.google.com/app/apikey) · [Groq](https://console.groq.com/keys)
+>
+> Anthropic and Groq have no embedding models; pair them with a provider that does (see the [matrix](#provider-support-matrix)).
 
-> **Want free local AI?** See [examples/docker-compose-ollama.yml](examples/) for Ollama setup
+> **Want free local AI?** See the [Local Quick Start](docs/0-START-HERE/quick-start-local.md) (Ollama in Docker).
 
 ---
 
 ### 📚 More Installation Options
 
-- **[With Ollama (Free Local AI)](examples/docker-compose-ollama.yml)** - Run models locally without API costs
+- **[Local AI with Ollama](docs/0-START-HERE/quick-start-local.md)** - Run models locally without API costs
+- **[Docker Compose guide](docs/1-INSTALLATION/docker-compose.md)** - Settings, backups, updates, remote access
 - **[From Source (Developers)](docs/1-INSTALLATION/from-source.md)** - For development and contributions
 - **[Complete Installation Guide](docs/1-INSTALLATION/index.md)** - All deployment scenarios
 
@@ -190,42 +165,49 @@ Done! You're ready to create your first notebook.
 
 ## Star History
 
-[![Star History Chart](https://api.star-history.com/svg?repos=lfnovo/open-notebook&type=date&legend=top-left)](https://www.star-history.com/#lfnovo/open-notebook&type=date&legend=top-left)
+[![Star History Chart](https://star-history.dera.page/svg?repos=lfnovo/open-notebook&type=date&legend=top-left)](https://star-history.dera.page/#lfnovo/open-notebook&type=date&legend=top-left)
 
 
 ## Provider Support Matrix
 
-Thanks to the [Esperanto](https://github.com/lfnovo/esperanto) library, we support this providers out of the box!
+Thanks to the [Esperanto](https://github.com/lfnovo/esperanto) library, we support these providers out of the box. The columns show the model types each provider offers; what you can use depends on your account.
 
 | Provider     | LLM Support | Embedding Support | Speech-to-Text | Text-to-Speech |
 |--------------|-------------|------------------|----------------|----------------|
 | OpenAI       | ✅          | ✅               | ✅             | ✅             |
 | Anthropic    | ✅          | ❌               | ❌             | ❌             |
+| Google (GenAI) | ✅          | ✅               | ✅             | ✅             |
 | Groq         | ✅          | ❌               | ✅             | ❌             |
-| Google (GenAI) | ✅          | ✅               | ❌             | ✅             |
-| Vertex AI    | ✅          | ✅               | ❌             | ✅             |
-| Ollama       | ✅          | ✅               | ❌             | ❌             |
-| Perplexity   | ✅          | ❌               | ❌             | ❌             |
-| ElevenLabs   | ❌          | ❌               | ✅             | ✅             |
-| Azure OpenAI | ✅          | ✅               | ❌             | ❌             |
-| Mistral      | ✅          | ✅               | ❌             | ❌             |
+| Mistral      | ✅          | ✅               | ✅             | ✅             |
 | DeepSeek     | ✅          | ❌               | ❌             | ❌             |
-| Voyage       | ❌          | ✅               | ❌             | ❌             |
-| xAI          | ✅          | ❌               | ❌             | ❌             |
-| OpenRouter   | ✅          | ❌               | ❌             | ❌             |
+| xAI          | ✅          | ❌               | ❌             | ✅             |
+| OpenRouter   | ✅          | ✅               | ✅             | ✅             |
 | DashScope (Qwen) | ✅          | ❌               | ❌             | ❌             |
-| MiniMax      | ✅          | ❌               | ❌             | ❌             |
-| OpenAI Compatible* | ✅          | ❌               | ❌             | ❌             |
+| MiniMax      | ✅          | ❌               | ❌             | ✅             |
+| Novita       | ✅          | ❌               | ❌             | ❌             |
+| SiliconFlow  | ✅          | ❌               | ❌             | ❌             |
+| Z.ai         | ✅          | ❌               | ❌             | ❌             |
+| PayPerQ (PPQ) | ✅          | ✅               | ✅             | ✅             |
+| Cohere       | ✅          | ✅               | ❌             | ❌             |
+| Voyage       | ❌          | ✅               | ❌             | ❌             |
+| ElevenLabs   | ❌          | ❌               | ✅             | ✅             |
+| Deepgram     | ❌          | ❌               | ✅             | ✅             |
+| Ollama       | ✅          | ✅               | ❌             | ❌             |
+| oMLX         | ✅          | ✅               | ❌             | ❌             |
+| Azure OpenAI | ✅          | ✅               | ✅             | ✅             |
+| Vertex AI    | ✅          | ✅               | ❌             | ✅             |
+| OpenAI Compatible* | ✅          | ✅               | ✅             | ✅             |
+| Anthropic Compatible | ✅          | ❌               | ❌             | ❌             |
 
-*Supports LM Studio and any OpenAI-compatible endpoint
+*Supports LM Studio and any OpenAI-compatible endpoint. Prefer the native **oMLX** provider for [oMLX](https://omlx.ai/) (Apple Silicon); see [docs/5-CONFIGURATION/omlx.md](docs/5-CONFIGURATION/omlx.md).
 
 ## ✨ Key Features
 
 ### Core Capabilities
-- **🔒 Privacy-First**: Your data stays under your control - no cloud dependencies
+- **🔒 Privacy-First**: Self-hosted; with local models nothing leaves your machine
 - **🎯 Multi-Notebook Organization**: Manage multiple research projects seamlessly
 - **📚 Universal Content Support**: PDFs, videos, audio, web pages, Office docs, and more
-- **🤖 Multi-Model AI Support**: 18+ providers including OpenAI, Anthropic, Ollama, Google, LM Studio, and more
+- **🤖 Multi-Model AI Support**: 20+ providers including OpenAI, Anthropic, Ollama, Google, LM Studio, and more
 - **🎙️ Professional Podcast Generation**: Advanced multi-speaker podcasts with Episode Profiles
 - **🔍 Intelligent Search**: Full-text and vector search across all your content
 - **💬 Context-Aware Chat**: AI conversations powered by your research materials
@@ -248,47 +230,36 @@ Thanks to the [Esperanto](https://github.com/lfnovo/esperanto) library, we suppo
 
 ### Getting Started
 - **[📖 Introduction](docs/0-START-HERE/index.md)** - Learn what Open Notebook offers
-- **[⚡ Quick Start](docs/0-START-HERE/quick-start.md)** - Get up and running in 5 minutes
+- **[⚡ Quick Start with a Cloud Provider](docs/0-START-HERE/quick-start-cloud.md)** - OpenAI, Anthropic, Google and more in 5 minutes
 - **[🔧 Installation](docs/1-INSTALLATION/index.md)** - Comprehensive setup guide
-- **[🎯 Your First Notebook](docs/0-START-HERE/first-notebook.md)** - Step-by-step tutorial
+- **[🎯 Run It Fully Local](docs/0-START-HERE/quick-start-local.md)** - Ollama/LM Studio, completely private
 
 ### User Guide
 - **[📱 Interface Overview](docs/3-USER-GUIDE/interface-overview.md)** - Understanding the layout
-- **[📚 Notebooks](docs/3-USER-GUIDE/notebooks.md)** - Organizing your research
-- **[📄 Sources](docs/3-USER-GUIDE/sources.md)** - Managing content types
-- **[📝 Notes](docs/3-USER-GUIDE/notes.md)** - Creating and managing notes
-- **[💬 Chat](docs/3-USER-GUIDE/chat.md)** - AI conversations
+- **[📚 Notebooks, Sources & Notes](docs/2-CORE-CONCEPTS/notebooks-sources-notes.md)** - Organizing your research
+- **[📄 Adding Sources](docs/3-USER-GUIDE/adding-sources.md)** - Managing content types
+- **[📝 Working with Notes](docs/3-USER-GUIDE/working-with-notes.md)** - Creating and managing notes
+- **[💬 Chatting Effectively](docs/3-USER-GUIDE/chat-effectively.md)** - AI conversations
 - **[🔍 Search](docs/3-USER-GUIDE/search.md)** - Finding information
 
 ### Advanced Topics
-- **[🎙️ Podcast Generation](docs/2-CORE-CONCEPTS/podcasts.md)** - Create professional podcasts
-- **[🔧 Content Transformations](docs/2-CORE-CONCEPTS/transformations.md)** - Customize content processing
-- **[🤖 AI Models](docs/4-AI-PROVIDERS/index.md)** - AI model configuration
+- **[🎙️ Podcast Generation](docs/2-CORE-CONCEPTS/podcasts-explained.md)** - Create professional podcasts
+- **[🔧 Content Transformations](docs/3-USER-GUIDE/transformations.md)** - Customize content processing
+- **[🤖 AI Providers](docs/4-AI-PROVIDERS/index.md)** - Connect providers and set default models
 - **[🔌 MCP Integration](docs/5-CONFIGURATION/mcp-integration.md)** - Connect with Claude Desktop, VS Code and other MCP clients
 - **[🔧 REST API Reference](docs/7-DEVELOPMENT/api-reference.md)** - Complete API documentation
 - **[🔐 Security](docs/5-CONFIGURATION/security.md)** - Password protection and privacy
 - **[🚀 Deployment](docs/1-INSTALLATION/index.md)** - Complete deployment guides for all scenarios
+- **[🧭 Vision & Principles](VISION.md)** - What Open Notebook is, and where it's going
+- **[🛠️ Developer Docs](docs/7-DEVELOPMENT/index.md)** - Architecture, setup, contributing, decision records
 
 <p align="right">(<a href="#readme-top">back to top</a>)</p>
 
-## 🗺️ Roadmap
+## 🗺️ Where We're Going
 
-### Upcoming Features
-- **Live Front-End Updates**: Real-time UI updates for smoother experience
-- **Async Processing**: Faster UI through asynchronous content processing
-- **Cross-Notebook Sources**: Reuse research materials across projects
-- **Bookmark Integration**: Connect with your favorite bookmarking apps
+Right now the focus is on making the core experience (sources, chat, search, notes, podcasts) solid across every provider and deployment, before adding new product surfaces. The bigger directions under consideration, such as multi-user support, new output formats and agents operating Open Notebook through MCP, are described in the **Horizon** section of [VISION.md](VISION.md#horizon). They are directions, not dated promises.
 
-### Recently Completed ✅
-- **Next.js Frontend**: Modern React-based frontend with improved performance
-- **Comprehensive REST API**: Full programmatic access to all functionality
-- **Multi-Model Support**: 18+ AI providers including OpenAI, Anthropic, Ollama, LM Studio
-- **Advanced Podcast Generator**: Professional multi-speaker podcasts with Episode Profiles
-- **Content Transformations**: Powerful customizable actions for content processing
-- **Enhanced Citations**: Improved layout and finer control for source citations
-- **Multiple Chat Sessions**: Manage different conversations within notebooks
-
-See the [open issues](https://github.com/lfnovo/open-notebook/issues) for a full list of proposed features and known issues.
+Explore [GitHub Discussions](https://github.com/lfnovo/open-notebook/discussions/categories/ideas) for proposed features and product ideas, and [open Issues](https://github.com/lfnovo/open-notebook/issues) for known bugs and approved work. Recent changes are in the [CHANGELOG](CHANGELOG.md).
 
 <p align="right">(<a href="#readme-top">back to top</a>)</p>
 
@@ -297,13 +268,15 @@ See the [open issues](https://github.com/lfnovo/open-notebook/issues) for a full
 - **🤖 AI Installation Assistant**: We have a [CustomGPT built to help you install Open Notebook](https://chatgpt.com/g/g-68776e2765b48191bd1bae3f30212631-open-notebook-installation-assistant) - it will guide you through each step!
 - **New to Open Notebook?** Start with our [Getting Started Guide](docs/0-START-HERE/index.md)
 - **Need installation help?** Check our [Installation Guide](docs/1-INSTALLATION/index.md)
-- **Want to see it in action?** Try our [Quick Start Tutorial](docs/0-START-HERE/quick-start.md)
+- **Want to see it in action?** Try our [Quick Start Tutorial](docs/0-START-HERE/index.md)
 
 ## 🤝 Community & Contributing
 
 ### Join the Community
 - 💬 **[Discord Server](https://discord.gg/37XJPXfz2w)** - Get help, share ideas, and connect with other users
-- 🐛 **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Report bugs and request features
+- 𝕏 **[Follow @lfnovo on X](https://x.com/lfnovo)** - Project updates and news from the maintainer
+- 💡 **[GitHub Discussions](https://github.com/lfnovo/open-notebook/discussions)** - Ask questions and shape features, product direction, design, and architecture
+- 🐛 **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Report reproducible bugs and find approved work
 - ⭐ **Star this repo** - Show your support and help others discover Open Notebook
 
 ### Contributing
@@ -314,9 +287,8 @@ We welcome contributions! We're especially looking for help with:
 - **Documentation**: Improve guides and tutorials
 
 **Current Tech Stack**: Python, FastAPI, Next.js, React, SurrealDB
-**Future Roadmap**: Real-time updates, enhanced async processing
 
-See our [Contributing Guide](CONTRIBUTING.md) for detailed information on how to get started.
+See our [Contributing Guide](CONTRIBUTING.md) for detailed information on how to get started, including our guidelines for [AI-assisted contributions](docs/7-DEVELOPMENT/contributing.md#ai-assisted-and-agent-generated-prs). To understand what we're building (and what we'll say no to), read [VISION.md](VISION.md).
 
 <p align="right">(<a href="#readme-top">back to top</a>)</p>
 
@@ -328,7 +300,9 @@ Open Notebook is MIT licensed. See the [LICENSE](LICENSE) file for details.
 
 **Community Support**:
 - 💬 [Discord Server](https://discord.gg/37XJPXfz2w) - Get help, share ideas, and connect with users
-- 🐛 [GitHub Issues](https://github.com/lfnovo/open-notebook/issues) - Report bugs and request features
+- 𝕏 [Follow @lfnovo on X](https://x.com/lfnovo) - Project updates and news from the maintainer
+- 💡 [GitHub Discussions](https://github.com/lfnovo/open-notebook/discussions) - Ask questions and shape ideas
+- 🐛 [GitHub Issues](https://github.com/lfnovo/open-notebook/issues) - Report reproducible bugs and find approved work
 - 🌐 [Website](https://www.open-notebook.ai) - Learn more about the project
 
 <p align="right">(<a href="#readme-top">back to top</a>)</p>
@@ -345,10 +319,9 @@ Open Notebook is MIT licensed. See the [LICENSE](LICENSE) file for details.
 [issues-shield]: https://img.shields.io/github/issues/lfnovo/open-notebook.svg?style=for-the-badge
 [issues-url]: https://github.com/lfnovo/open-notebook/issues
 [license-shield]: https://img.shields.io/github/license/lfnovo/open-notebook.svg?style=for-the-badge
-[license-url]: https://github.com/lfnovo/open-notebook/blob/master/LICENSE.txt
+[license-url]: https://github.com/lfnovo/open-notebook/blob/main/LICENSE
 [linkedin-shield]: https://img.shields.io/badge/-LinkedIn-black.svg?style=for-the-badge&logo=linkedin&colorB=555
 [linkedin-url]: https://linkedin.com/in/lfnovo
-[product-screenshot]: images/screenshot.png
 [Next.js]: https://img.shields.io/badge/Next.js-000000?style=for-the-badge&logo=next.js&logoColor=white
 [Next-url]: https://nextjs.org/
 [React]: https://img.shields.io/badge/React-61DAFB?style=for-the-badge&logo=react&logoColor=black

SECURITY.md

@@ -0,0 +1,65 @@
+# Security Policy
+
+## Supported Versions
+
+Open Notebook is an actively developed project. Security fixes are applied to the
+**latest released version** only; there are no long-term support branches.
+
+| Version | Supported          |
+| ------- | ------------------ |
+| Latest release (`1.x`, current minor) | :white_check_mark: |
+| Older releases | :x: |
+
+If you are running an older version, please upgrade to the latest release before
+reporting an issue — the problem may already be fixed.
+
+## Reporting a Vulnerability
+
+**Please do not report security vulnerabilities through public GitHub issues,
+discussions, or pull requests.**
+
+Instead, report them privately through GitHub's built-in **private vulnerability
+reporting**:
+
+1. Go to the [Security tab](https://github.com/lfnovo/open-notebook/security) of
+   the repository.
+2. Click **"Report a vulnerability"**.
+3. Fill out the form with as much detail as you can.
+
+This keeps the report private between you and the maintainers until a fix is
+available.
+
+When reporting, please include where relevant:
+
+- A description of the vulnerability and its impact.
+- Steps to reproduce (a proof of concept, affected endpoint/component, or sample
+  configuration).
+- The Open Notebook version and how you are running it (Docker Compose,
+  single-container, from source).
+- Any suggested remediation, if you have one.
+
+## What to Expect
+
+- **Acknowledgement:** we aim to acknowledge a report within **5 business days**.
+- **Assessment:** we will investigate, confirm the issue, and determine the
+  affected versions.
+- **Fix & disclosure:** once a fix is ready we will release it and, with your
+  consent, credit you in the release notes. We follow a coordinated-disclosure
+  approach and ask that you keep the report private until a fix is published.
+
+## Scope
+
+Open Notebook is **self-hosted**: you run the API, frontend, and SurrealDB
+yourself, and you control the AI provider credentials. Please keep in mind:
+
+- The built-in password middleware (`OPEN_NOTEBOOK_PASSWORD`) is a basic access
+  control, not a full authentication system. See
+  [docs/5-CONFIGURATION/security.md](docs/5-CONFIGURATION/security.md) for
+  hardening guidance (encryption key, reverse proxy, CORS, default credentials).
+- Misconfiguration of your own deployment (e.g. exposing SurrealDB with default
+  credentials, or running without `OPEN_NOTEBOOK_ENCRYPTION_KEY`) is a
+  deployment concern covered by that hardening guide rather than a vulnerability
+  in the project — though we welcome reports where the defaults or docs actively
+  steer users toward an insecure setup.
+
+Thank you for helping keep Open Notebook and its users safe.

VISION.md

@@ -0,0 +1,79 @@
+# Open Notebook — Vision & Principles
+
+This document is the product's source of truth in two layers with different lifespans: **Identity** (durable — what Open Notebook is and refuses to be) and **Current Posture** (temporal — where we are in the journey and what's on the horizon). Triage and design decisions are evaluated against this document; the reasoning behind each rule lives in the [decision records](docs/7-DEVELOPMENT/decisions/README.md).
+
+---
+
+## Identity
+
+Open Notebook is a **privacy-focused, self-hosted alternative to Google's Notebook LM** that empowers users to:
+
+1. **Own their research data** — full control over where data lives and who can access it
+2. **Choose their AI providers** — any provider, or fully local models
+3. **Customize their workflows** — adapt the tool to different research needs
+4. **Access their work anywhere** — web UI, API, or integrations
+
+### What Open Notebook IS
+
+- A **research assistant** for managing and understanding content
+- A **platform** that connects various AI providers
+- A **privacy-first** tool that keeps your data under your control
+- An **extensible system** with APIs and customization options
+
+### What Open Notebook IS NOT
+
+- A document editor (use Google Docs, Notion, etc.)
+- A file storage system (use Dropbox, S3, etc.)
+- A general-purpose chatbot (use ChatGPT, Claude, etc.)
+- A replacement for your entire workflow (it's one tool in your toolkit)
+
+### Principles
+
+Durable, normative rules. Each links to the decision record that established it.
+
+| Principle | Rule |
+|---|---|
+| **Privacy first** | User data stays under user control by default. Self-hosted is the primary use case; no telemetry without opt-in; no hard dependency on specific cloud services. |
+| **Provider-agnostic core** | The default is portable: features must work across the provider matrix. Adopting a provider-exclusive capability is allowed but is a deliberate decision that requires a [PDR](docs/7-DEVELOPMENT/decisions/README.md) ([PDR-002](docs/7-DEVELOPMENT/decisions/PDR-002-provider-agnostic-core.md)). |
+| **Simplicity over features** | Easy to understand and use, even if it means fewer features. Sensible defaults; advanced options behind progressive disclosure. |
+| **API-first** | Every capability is accessible via the REST API — the UI is a client, never the only door ([ADR-003](docs/7-DEVELOPMENT/decisions/ADR-003-streamlit-to-nextjs.md)). |
+| **Extensibility through standards** | Extension happens through well-defined interfaces (transformations, commands, prompt templates), not forks. |
+| **Async-first** | Long-running operations never block the UI or the API ([ADR-004](docs/7-DEVELOPMENT/decisions/ADR-004-background-workers.md)). |
+
+### How we evaluate requests
+
+A feature request that conflicts with the IS NOT list or a principle gets closed with a pointer here — kindly, and with the reasoning. A "no" protects the core value proposition; it's not a judgment of the idea. If a request keeps coming back and the principle starts to feel wrong, that's a signal to revisit the principle through a decision record — not to make a quiet exception.
+
+---
+
+## Current Posture
+
+> **Reviewed: 2026-07.** This section is expected to change. Updating it is not a reversal — it's a phase change, recorded with a short PDR and an edit here.
+
+**The phase we're in: get the basics working well for everyone before expanding.** Priority goes to making the core experience (sources, chat, search, notes, podcasts) solid across the full provider matrix and deployment surface, ahead of new product surfaces.
+
+### Directional constraints
+
+Decisions about the future we haven't made yet — recorded as "which door to keep open":
+
+- **Single-user first, multi-user compatible.** Open Notebook is a single-user tool today, but multi-user is under active consideration ([#712](https://github.com/lfnovo/open-notebook/issues/712)). New features must not gratuitously preclude multi-user (schema, auth, data scoping) ([PDR-001](docs/7-DEVELOPMENT/decisions/PDR-001-single-user-first.md)).
+- **Portable by default.** Provider-exclusive capabilities (including paid-only ones) are on the table for the future — deliberately, via PDR, never by accident ([PDR-002](docs/7-DEVELOPMENT/decisions/PDR-002-provider-agnostic-core.md)).
+
+### Horizon
+
+The big clusters under consideration — direction, not roadmap; no dates. Each has an umbrella issue where the thinking happens:
+
+| Cluster | What it is | Where |
+|---|---|---|
+| **Platform v-next** | SurrealDB v3 migration, possible frontend/backend Docker image split, possible Surreal Commands → Celery move — evaluated together as one coordinated breaking change | [#372](https://github.com/lfnovo/open-notebook/issues/372) · [#378](https://github.com/lfnovo/open-notebook/issues/378) · [#381](https://github.com/lfnovo/open-notebook/issues/381) |
+| **Multi-user** | Deep platform redesign: auth, data scoping, what "multi-user" means for a self-hosted tool | [#712](https://github.com/lfnovo/open-notebook/issues/712) |
+| **Content modes & artifacts** | The output side: generated artifacts, videos, explainers, presentations, mind maps — as one coherent product surface, not a pile of features | [#203](https://github.com/lfnovo/open-notebook/issues/203) |
+| **Agents operating Open Notebook** | Role inversion via MCP: AI agents use Open Notebook on the user's behalf — the platform becomes the research memory of agents, not just a UI | [#878](https://github.com/lfnovo/open-notebook/issues/878) · [#693](https://github.com/lfnovo/open-notebook/issues/693) · [#973](https://github.com/lfnovo/open-notebook/issues/973) |
+
+---
+
+## How this document changes
+
+- **Identity** changes rarely and deliberately: a decision record marks the old rule as superseded, then this document is updated.
+- **Posture** changes when the phase changes: a short PDR captures the why, the section above is edited, and the "Reviewed" stamp is bumped.
+- Engineering practices (code standards, anti-patterns, decision framework) live in [docs/7-DEVELOPMENT/design-principles.md](docs/7-DEVELOPMENT/design-principles.md).

api/AGENTS.md

@@ -0,0 +1,3 @@
+# API Rules (api/)
+
+The rules for `api/` are part of the backend rules in [../open_notebook/AGENTS.md](../open_notebook/AGENTS.md) (which also cover `commands/` and `prompts/`). Read that file before changing code here. Project-wide rules are in the root [AGENTS.md](../AGENTS.md).

api/CLAUDE.md

@@ -1,260 +1 @@
-# API Module
-
-FastAPI-based REST backend exposing services for notebooks, sources, notes, chat, podcasts, and AI model management.
-
-## Purpose
-
-FastAPI application serving three architectural layers: routes (HTTP endpoints), services (business logic), and models (request/response schemas). Integrates LangGraph workflows (chat, ask, source_chat), SurrealDB persistence, and AI providers via Esperanto.
-
-## Architecture Overview
-
-**Three layers**:
-1. **Routes** (`routers/*`): HTTP endpoints mapping to services
-2. **Services** (`*_service.py`): Business logic orchestrating domain models, database, graphs, AI providers
-3. **Models** (`models.py`): Pydantic request/response schemas with validation
-
-**Startup flow**:
-- Load .env environment variables
-- Initialize CORS middleware + password auth middleware
-- Run database migrations via AsyncMigrationManager on lifespan startup
-- Run podcast profile data migration (legacy string to model registry conversion)
-- Register all routers
-
-**Key services**:
-- `chat_service.py`: Invokes chat graph with messages, context
-- `podcast_service.py`: Orchestrates outline + transcript generation
-- `sources_service.py`: Content ingestion, vectorization, metadata
-- `notes_service.py`: Note creation, linking to sources/insights
-- `transformations_service.py`: Applies transformations to content
-- `models_service.py`: Manages AI provider/model configuration
-- `episode_profiles_service.py`: Manages podcast speaker/episode profiles
-
-## Component Catalog
-
-### Main Application
-- **main.py**: FastAPI app initialization, CORS setup, auth middleware, lifespan event, router registration
-- **Lifespan handler**: Runs AsyncMigrationManager on startup (database schema migration)
-- **Auth middleware**: PasswordAuthMiddleware protects endpoints (password-based access control)
-
-### Services (Business Logic)
-- **chat_service.py**: Invokes chat.py graph; handles message history via SqliteSaver
-- **podcast_service.py**: Generates outline (outline.jinja), then transcript (transcript.jinja) for episodes
-- **sources_service.py**: Ingests files/URLs (content_core), extracts text, vectorizes, saves to SurrealDB
-- **transformations_service.py**: Applies transformations via transformation.py graph
-- **models_service.py**: Manages ModelManager config (AI provider overrides)
-- **episode_profiles_service.py**: CRUD for EpisodeProfile and SpeakerProfile models
-- **insights_service.py**: Generates and retrieves source insights
-- **notes_service.py**: Creates notes linked to sources/insights
-
-### Models (Schemas)
-- **models.py**: Pydantic schemas for request/response validation
-- Request bodies: ChatRequest, CreateNoteRequest, PodcastGenerationRequest, etc.
-- Response bodies: ChatResponse, NoteResponse, PodcastResponse, etc.
-- Custom validators for enum fields, file paths, model references
-
-### Routers
-- **routers/chat.py**: POST /chat
-- **routers/source_chat.py**: POST /source/{source_id}/chat
-- **routers/podcasts.py**: POST /podcasts, GET /podcasts/{id}, POST /podcasts/episodes/{id}/retry, etc.
-- **routers/notes.py**: POST /notes, GET /notes/{id}
-- **routers/sources.py**: POST /sources, GET /sources/{id}, DELETE /sources/{id}
-- **routers/models.py**: GET /models, POST /models/config
-- **routers/credentials.py**: CRUD + test + discover + migrate for credential management
-- **routers/transformations.py**: POST /transformations
-- **routers/insights.py**: GET /sources/{source_id}/insights
-- **routers/auth.py**: POST /auth/password (password-based auth)
-- **routers/languages.py**: GET /languages (available podcast languages via pycountry+babel)
-- **routers/commands.py**: GET /commands/{command_id} (job status tracking)
-
-## Common Patterns
-
-- **Service injection via FastAPI**: Routers import services directly; no DI framework
-- **Async/await throughout**: All DB queries, graph invocations, AI calls are async
-- **SurrealDB transactions**: Services use repo_query, repo_create, repo_upsert from database layer
-- **Config override pattern**: Models/config override via models_service passed to graph.ainvoke(config=...)
-- **Error handling**: Custom exception hierarchy (`open_notebook.exceptions`) with global FastAPI exception handlers mapping to HTTP status codes (see Error Handling section below). LangGraph nodes use `classify_error()` to convert raw LLM provider errors into typed exceptions with user-friendly messages.
-- **Logging**: loguru logger in main.py; services expected to log key operations
-- **Response normalization**: All responses follow standard schema (data + metadata structure)
-
-## Key Dependencies
-
-- `fastapi`: FastAPI app, routers, HTTPException
-- `pydantic`: Validation models with Field, field_validator
-- `open_notebook.graphs`: chat, ask, source_chat, source, transformation graphs
-- `open_notebook.database`: SurrealDB repository functions (repo_query, repo_create, repo_upsert)
-- `open_notebook.domain`: Notebook, Source, Note, SourceInsight models
-- `open_notebook.ai.provision`: provision_langchain_model() factory
-- `ai_prompter`: Prompter for template rendering
-- `content_core`: extract_content() for file/URL processing
-- `esperanto`: AI provider client library (LLM, embeddings, TTS)
-- `surreal_commands`: Job queue for async operations (podcast generation)
-- `loguru`: Structured logging
-
-## Important Quirks & Gotchas
-
-- **Migration auto-run**: Database schema migrations run on every API startup (via lifespan); no manual migration steps
-- **PasswordAuthMiddleware is basic**: Uses simple password check; production deployments should replace with OAuth/JWT
-- **No request rate limiting**: No built-in rate limiting; deployment must add via proxy/middleware
-- **Service state is stateless**: Services don't cache results; each request re-queries database/AI models
-- **Graph invocation is blocking**: chat/podcast workflows may take minutes; no timeout handling in services
-- **Command job fire-and-forget**: podcast_service.py submits jobs but doesn't wait (async job queue pattern)
-- **Model override scoping**: Model config override via RunnableConfig is per-request only (not persistent)
-- **CORS open by default**: main.py CORS settings allow all origins (restrict before production)
-- **No OpenAPI security scheme**: API docs available without auth (disable before production)
-- **Services don't validate user permission**: All endpoints trust authentication layer; no per-notebook permission checks
-
-## Error Handling
-
-### Global Exception Handlers (`main.py`)
-
-FastAPI exception handlers map custom exception types from `open_notebook.exceptions` to HTTP status codes. All error responses include CORS headers.
-
-| Exception Class | HTTP Status | Use Case |
-|----------------|-------------|----------|
-| `NotFoundError` | 404 | Resource not found |
-| `InvalidInputError` | 400 | Bad request data |
-| `AuthenticationError` | 401 | Invalid/missing API key |
-| `RateLimitError` | 429 | Provider rate limit exceeded |
-| `ConfigurationError` | 422 | Wrong model name, missing config |
-| `NetworkError` | 502 | Cannot reach AI provider |
-| `ExternalServiceError` | 502 | Provider returned error (500/503, context length) |
-| `OpenNotebookError` (base) | 500 | Any other application error |
-
-### Error Classification (`open_notebook.utils.error_classifier`)
-
-The `classify_error()` function maps raw exceptions from LLM providers/Esperanto/LangChain into the typed exceptions above with user-friendly messages. Used in all LangGraph graph nodes and SSE streaming handlers.
-
-**Flow**: Raw exception → keyword matching → `(ExceptionClass, user_message)` → raised → caught by global handler → HTTP response with descriptive message.
-
-### Frontend Integration
-
-The frontend `getApiErrorMessage()` helper (`lib/utils/error-handler.ts`) tries i18n mapping first, then falls back to displaying the backend's descriptive error message directly.
-
----
-
-## How to Add New Endpoint
-
-1. Create router file in `routers/` (e.g., `routers/new_feature.py`)
-2. Import router into `main.py` and register: `app.include_router(new_feature.router, tags=["new_feature"])`
-3. Create service in `new_feature_service.py` with business logic
-4. Define request/response schemas in `models.py` (or create `new_feature_models.py`)
-5. Implement router functions calling service methods
-6. Test with `uv run uvicorn api.main:app --host 0.0.0.0 --port 5055`
-
-## Testing Patterns
-
-- **Interactive docs**: http://localhost:5055/docs (Swagger UI)
-- **Direct service tests**: Import service, call methods directly with test data
-- **Mock graphs**: Replace graph.ainvoke() with mock for testing service logic
-- **Database: Use test database** (separate SurrealDB instance or mock repo_query)
-
----
-
-## Credential Management (API Configuration UI)
-
-The Credential Management system enables users to configure AI provider credentials through the UI instead of environment variables. Keys are stored securely in SurrealDB (encrypted via Fernet) with database-first fallback to environment variables.
-
-### Router: `routers/credentials.py`
-
-**Endpoints**:
-
-| Method | Endpoint | Description |
-|--------|----------|-------------|
-| GET | `/credentials` | List all credentials (optional `?provider=` filter) |
-| GET | `/credentials/by-provider/{provider}` | List credentials for a provider |
-| POST | `/credentials` | Create a new credential |
-| GET | `/credentials/{credential_id}` | Get a specific credential |
-| PUT | `/credentials/{credential_id}` | Update a credential |
-| DELETE | `/credentials/{credential_id}` | Delete a credential |
-| POST | `/credentials/{credential_id}/test` | Test connection using credential |
-| POST | `/credentials/{credential_id}/discover` | Discover available models |
-| POST | `/credentials/{credential_id}/register-models` | Register discovered models |
-| POST | `/credentials/migrate-from-provider-config` | Migrate from legacy ProviderConfig |
-
-**Supported Providers** (13 total):
-- Simple API key: `openai`, `anthropic`, `google`, `groq`, `mistral`, `deepseek`, `xai`, `openrouter`, `voyage`, `elevenlabs`
-- URL-based: `ollama`
-- Multi-field: `azure`, `vertex`, `openai_compatible`
-
-**Security Features**:
-- NEVER returns actual API key values (only metadata)
-- URL validation (SSRF protection) on all URL fields via `_validate_url()`
-- Allows private IPs and localhost for self-hosted services (Ollama, LM Studio)
-- Requires `OPEN_NOTEBOOK_ENCRYPTION_KEY` to be set for storing credentials
-
-### Domain Model: `Credential` (`open_notebook/domain/credential.py`)
-
-Individual credential records replacing the old `ProviderConfig` singleton. Each credential stores:
-- Provider name, display name, modalities
-- Encrypted API key (via Fernet)
-- Provider-specific config (base_url, endpoint, api_version, etc.)
-
-### Integration with Key Provider (`open_notebook/ai/key_provider.py`)
-
-The `key_provider` module provisions DB-stored credentials into environment variables for Esperanto compatibility:
-
-**Database-first Pattern**:
-1. API endpoint saves keys to `Credential` records (encrypted in SurrealDB)
-2. Before model provisioning, `provision_provider_keys(provider)` checks DB, then env vars
-3. Keys from DB are set as environment variables for Esperanto compatibility
-4. Existing env vars remain unchanged if no DB config exists
-
-**Key Functions**:
-- `get_api_key(provider)`: Get API key (DB first, env fallback)
-- `provision_provider_keys(provider)`: Set env vars from DB for a provider
-- `provision_all_keys()`: Load all provider keys from DB into env vars
-
-### Authentication
-
-No changes to authentication. The `credentials` router uses the same `PasswordAuthMiddleware` as all other endpoints. Keys are protected by the same password-based auth.
-
-**Auth Flow** (unchanged from `api/auth.py`):
-- `PasswordAuthMiddleware`: Global middleware checking `Authorization: Bearer {password}` header
-- Default password: `open-notebook-change-me` (set `OPEN_NOTEBOOK_PASSWORD` in production)
-- Docker secrets support via `OPEN_NOTEBOOK_PASSWORD_FILE`
-
-### Connection Testing (`open_notebook/ai/connection_tester.py`)
-
-The `/credentials/{credential_id}/test` endpoint uses minimal API calls to verify credentials:
-- Loads Credential via `Credential.get(config_id)`, uses `credential.to_esperanto_config()`
-- Uses cheapest/smallest models per provider (TEST_MODELS map)
-- Returns success status and descriptive message
-- Special handlers for ollama, openai_compatible, and azure providers
-
-### Migration Workflows
-
-Two migration endpoints help users transition to the credential system:
-
-**From environment variables** (`POST /credentials/migrate-from-env`):
-1. Checks each provider for env var presence
-2. Creates Credential records from env var values
-3. Returns summary: migrated, skipped, errors
-
-**From legacy ProviderConfig** (`POST /credentials/migrate-from-provider-config`):
-1. Reads old ProviderConfig records from database
-2. Converts each to individual Credential records
-3. Returns summary: migrated, skipped, errors
-
-### Example Usage
-
-```python
-# Check status
-GET /credentials/status
-# Response: {"configured": {"openai": true, "anthropic": false}, "source": {"openai": "database", "anthropic": "none"}, "encryption_configured": true}
-
-# Create credential
-POST /credentials
-{"name": "My OpenAI Key", "provider": "openai", "modalities": ["language", "embedding"], "api_key": "sk-proj-..."}
-
-# Test connection
-POST /credentials/{credential_id}/test
-# Response: {"provider": "openai", "success": true, "message": "Connection successful"}
-
-# Discover models
-POST /credentials/{credential_id}/discover
-# Response: {"provider": "openai", "models": [{"model_id": "gpt-4", "name": "gpt-4", ...}], "credential_id": "..."}
-
-# Migrate from env
-POST /credentials/migrate-from-env
-# Response: {"message": "Migration complete. Migrated 3 providers.", "migrated": ["openai", "anthropic", "groq"], "skipped": [], "errors": []}
-```
+@../open_notebook/AGENTS.md

api/auth.py

@@ -1,10 +1,10 @@
+import secrets
 from typing import Optional
 
-from fastapi import Depends, HTTPException, Request
-from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
-from loguru import logger
-from starlette.middleware.base import BaseHTTPMiddleware
-from starlette.responses import JSONResponse
+from fastapi import Request
+from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
+from starlette.responses import JSONResponse, Response
+from starlette.types import ASGIApp
 
 from open_notebook.utils.encryption import get_secret_from_env
 
@@ -12,14 +12,17 @@ from open_notebook.utils.encryption import get_secret_from_env
 class PasswordAuthMiddleware(BaseHTTPMiddleware):
     """
     Middleware to check password authentication for all API requests.
-    Always active with default password if OPEN_NOTEBOOK_PASSWORD is not set.
+    Auth is fully disabled (no hardcoded default password) if
+    OPEN_NOTEBOOK_PASSWORD is not set.
     Supports Docker secrets via OPEN_NOTEBOOK_PASSWORD_FILE.
     """
 
-    def __init__(self, app, excluded_paths: Optional[list] = None):
+    def __init__(
+        self, app: ASGIApp, excluded_paths: Optional[list[str]] = None
+    ) -> None:
         super().__init__(app)
         self.password = get_secret_from_env("OPEN_NOTEBOOK_PASSWORD")
-        self.excluded_paths = excluded_paths or [
+        self.excluded_paths: list[str] = excluded_paths or [
             "/",
             "/health",
             "/docs",
@@ -27,7 +30,9 @@ class PasswordAuthMiddleware(BaseHTTPMiddleware):
             "/redoc",
         ]
 
-    async def dispatch(self, request: Request, call_next):
+    async def dispatch(
+        self, request: Request, call_next: RequestResponseEndpoint
+    ) -> Response:
         # Skip authentication if no password is set
         if not self.password:
             return await call_next(request)
@@ -62,8 +67,10 @@ class PasswordAuthMiddleware(BaseHTTPMiddleware):
                 headers={"WWW-Authenticate": "Bearer"},
             )
 
-        # Check password
-        if credentials != self.password:
+        # Check password (constant-time to avoid a timing side-channel)
+        if not secrets.compare_digest(
+            credentials.encode("latin-1"), self.password.encode("utf-8")
+        ):
             return JSONResponse(
                 status_code=401,
                 content={"detail": "Invalid password"},
@@ -73,42 +80,3 @@ class PasswordAuthMiddleware(BaseHTTPMiddleware):
         # Password is correct, proceed with the request
         response = await call_next(request)
         return response
-
-
-# Optional: HTTPBearer security scheme for OpenAPI documentation
-security = HTTPBearer(auto_error=False)
-
-
-def check_api_password(
-    credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
-) -> bool:
-    """
-    Utility function to check API password.
-    Can be used as a dependency in individual routes if needed.
-    Supports Docker secrets via OPEN_NOTEBOOK_PASSWORD_FILE.
-    Returns True without checking credentials if OPEN_NOTEBOOK_PASSWORD is not configured.
-    Raises 401 if credentials are missing or don't match the configured password.
-    """
-    password = get_secret_from_env("OPEN_NOTEBOOK_PASSWORD")
-
-    # No password configured - skip authentication
-    if not password:
-        return True
-
-    # No credentials provided
-    if not credentials:
-        raise HTTPException(
-            status_code=401,
-            detail="Missing authorization",
-            headers={"WWW-Authenticate": "Bearer"},
-        )
-
-    # Check password
-    if credentials.credentials != password:
-        raise HTTPException(
-            status_code=401,
-            detail="Invalid password",
-            headers={"WWW-Authenticate": "Bearer"},
-        )
-
-    return True

api/chat_service.py

@@ -1,168 +0,0 @@
-"""
-Chat service for API operations.
-Provides async interface for chat functionality.
-"""
-
-import os
-from typing import Any, Dict, List, Optional
-
-import httpx
-from loguru import logger
-
-
-class ChatService:
-    """Service for chat-related API operations"""
-
-    def __init__(self):
-        self.base_url = os.getenv("API_BASE_URL", "http://127.0.0.1:5055")
-        # Add authentication header if password is set
-        self.headers = {}
-        password = os.getenv("OPEN_NOTEBOOK_PASSWORD")
-        if password:
-            self.headers["Authorization"] = f"Bearer {password}"
-
-    async def get_sessions(self, notebook_id: str) -> List[Dict[str, Any]]:
-        """Get all chat sessions for a notebook"""
-        try:
-            async with httpx.AsyncClient() as client:
-                response = await client.get(
-                    f"{self.base_url}/api/chat/sessions",
-                    params={"notebook_id": notebook_id},
-                    headers=self.headers,
-                )
-                response.raise_for_status()
-                return response.json()
-        except Exception as e:
-            logger.error(f"Error fetching chat sessions: {str(e)}")
-            raise
-
-    async def create_session(
-        self,
-        notebook_id: str,
-        title: Optional[str] = None,
-        model_override: Optional[str] = None,
-    ) -> Dict[str, Any]:
-        """Create a new chat session"""
-        try:
-            data: Dict[str, Any] = {"notebook_id": notebook_id}
-            if title is not None:
-                data["title"] = title
-            if model_override is not None:
-                data["model_override"] = model_override
-
-            async with httpx.AsyncClient() as client:
-                response = await client.post(
-                    f"{self.base_url}/api/chat/sessions",
-                    json=data,
-                    headers=self.headers,
-                )
-                response.raise_for_status()
-                return response.json()
-        except Exception as e:
-            logger.error(f"Error creating chat session: {str(e)}")
-            raise
-
-    async def get_session(self, session_id: str) -> Dict[str, Any]:
-        """Get a specific session with messages"""
-        try:
-            async with httpx.AsyncClient() as client:
-                response = await client.get(
-                    f"{self.base_url}/api/chat/sessions/{session_id}",
-                    headers=self.headers,
-                )
-                response.raise_for_status()
-                return response.json()
-        except Exception as e:
-            logger.error(f"Error fetching session: {str(e)}")
-            raise
-
-    async def update_session(
-        self,
-        session_id: str,
-        title: Optional[str] = None,
-        model_override: Optional[str] = None,
-    ) -> Dict[str, Any]:
-        """Update session properties"""
-        try:
-            data: Dict[str, Any] = {}
-            if title is not None:
-                data["title"] = title
-            if model_override is not None:
-                data["model_override"] = model_override
-
-            if not data:
-                raise ValueError(
-                    "At least one field must be provided to update a session"
-                )
-
-            async with httpx.AsyncClient() as client:
-                response = await client.put(
-                    f"{self.base_url}/api/chat/sessions/{session_id}",
-                    json=data,
-                    headers=self.headers,
-                )
-                response.raise_for_status()
-                return response.json()
-        except Exception as e:
-            logger.error(f"Error updating session: {str(e)}")
-            raise
-
-    async def delete_session(self, session_id: str) -> Dict[str, Any]:
-        """Delete a chat session"""
-        try:
-            async with httpx.AsyncClient() as client:
-                response = await client.delete(
-                    f"{self.base_url}/api/chat/sessions/{session_id}",
-                    headers=self.headers,
-                )
-                response.raise_for_status()
-                return response.json()
-        except Exception as e:
-            logger.error(f"Error deleting session: {str(e)}")
-            raise
-
-    async def execute_chat(
-        self,
-        session_id: str,
-        message: str,
-        context: Dict[str, Any],
-        model_override: Optional[str] = None,
-    ) -> Dict[str, Any]:
-        """Execute a chat request"""
-        try:
-            data = {"session_id": session_id, "message": message, "context": context}
-            if model_override is not None:
-                data["model_override"] = model_override
-
-            # Short connect timeout (10s), long read timeout (10 min) for Ollama/local LLMs
-            timeout = httpx.Timeout(connect=10.0, read=600.0, write=30.0, pool=10.0)
-            async with httpx.AsyncClient(timeout=timeout) as client:
-                response = await client.post(
-                    f"{self.base_url}/api/chat/execute", json=data, headers=self.headers
-                )
-                response.raise_for_status()
-                return response.json()
-        except Exception as e:
-            logger.error(f"Error executing chat: {str(e)}")
-            raise
-
-    async def build_context(
-        self, notebook_id: str, context_config: Dict[str, Any]
-    ) -> Dict[str, Any]:
-        """Build context for a notebook"""
-        try:
-            data = {"notebook_id": notebook_id, "context_config": context_config}
-
-            async with httpx.AsyncClient() as client:
-                response = await client.post(
-                    f"{self.base_url}/api/chat/context", json=data, headers=self.headers
-                )
-                response.raise_for_status()
-                return response.json()
-        except Exception as e:
-            logger.error(f"Error building context: {str(e)}")
-            raise
-
-
-# Global instance
-chat_service = ChatService()

api/client.py

@@ -1,529 +0,0 @@
-"""
-API client for Open Notebook API.
-This module provides a client interface to interact with the Open Notebook API.
-"""
-
-import os
-from typing import Any, Dict, List, Optional, Union
-
-import httpx
-from loguru import logger
-
-
-class APIClient:
-    """Client for Open Notebook API."""
-
-    def __init__(self, base_url: Optional[str] = None):
-        self.base_url = base_url or os.getenv("API_BASE_URL", "http://127.0.0.1:5055")
-        # Timeout increased to 5 minutes (300s) to accommodate slow LLM operations
-        # (transformations, insights) on slower hardware (Ollama, LM Studio, remote APIs)
-        # Configurable via API_CLIENT_TIMEOUT environment variable (in seconds)
-        timeout_str = os.getenv("API_CLIENT_TIMEOUT", "300.0")
-        try:
-            timeout_value = float(timeout_str)
-            # Validate timeout is within reasonable bounds (30s - 3600s / 1 hour)
-            if timeout_value < 30:
-                logger.warning(
-                    f"API_CLIENT_TIMEOUT={timeout_value}s is too low, using minimum of 30s"
-                )
-                timeout_value = 30.0
-            elif timeout_value > 3600:
-                logger.warning(
-                    f"API_CLIENT_TIMEOUT={timeout_value}s is too high, using maximum of 3600s"
-                )
-                timeout_value = 3600.0
-            self.timeout = timeout_value
-        except ValueError:
-            logger.error(
-                f"Invalid API_CLIENT_TIMEOUT value '{timeout_str}', using default 300s"
-            )
-            self.timeout = 300.0
-
-        # Add authentication header if password is set
-        self.headers = {}
-        password = os.getenv("OPEN_NOTEBOOK_PASSWORD")
-        if password:
-            self.headers["Authorization"] = f"Bearer {password}"
-
-    def _make_request(
-        self, method: str, endpoint: str, timeout: Optional[float] = None, **kwargs
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Make HTTP request to the API."""
-        url = f"{self.base_url}{endpoint}"
-        request_timeout = timeout if timeout is not None else self.timeout
-
-        # Merge headers
-        headers = kwargs.get("headers", {})
-        headers.update(self.headers)
-        kwargs["headers"] = headers
-
-        try:
-            with httpx.Client(timeout=request_timeout) as client:
-                response = client.request(method, url, **kwargs)
-                response.raise_for_status()
-                return response.json()
-        except httpx.RequestError as e:
-            logger.error(f"Request error for {method} {url}: {str(e)}")
-            raise ConnectionError(f"Failed to connect to API: {str(e)}")
-        except httpx.HTTPStatusError as e:
-            logger.error(
-                f"HTTP error {e.response.status_code} for {method} {url}: {e.response.text}"
-            )
-            raise RuntimeError(
-                f"API request failed: {e.response.status_code} - {e.response.text}"
-            )
-        except Exception as e:
-            logger.error(f"Unexpected error for {method} {url}: {str(e)}")
-            raise
-
-    # Notebooks API methods
-    def get_notebooks(
-        self, archived: Optional[bool] = None, order_by: str = "updated desc"
-    ) -> List[Dict[Any, Any]]:
-        """Get all notebooks."""
-        params: Dict[str, Any] = {"order_by": order_by}
-        if archived is not None:
-            params["archived"] = str(archived).lower()
-
-        result = self._make_request("GET", "/api/notebooks", params=params)
-        return result if isinstance(result, list) else [result]
-
-    def create_notebook(
-        self, name: str, description: str = ""
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Create a new notebook."""
-        data = {"name": name, "description": description}
-        return self._make_request("POST", "/api/notebooks", json=data)
-
-    def get_notebook(
-        self, notebook_id: str
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Get a specific notebook."""
-        return self._make_request("GET", f"/api/notebooks/{notebook_id}")
-
-    def update_notebook(
-        self, notebook_id: str, **updates
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Update a notebook."""
-        return self._make_request("PUT", f"/api/notebooks/{notebook_id}", json=updates)
-
-    def delete_notebook(
-        self, notebook_id: str
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Delete a notebook."""
-        return self._make_request("DELETE", f"/api/notebooks/{notebook_id}")
-
-    # Search API methods
-    def search(
-        self,
-        query: str,
-        search_type: str = "text",
-        limit: int = 100,
-        search_sources: bool = True,
-        search_notes: bool = True,
-        minimum_score: float = 0.2,
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Search the knowledge base."""
-        data = {
-            "query": query,
-            "type": search_type,
-            "limit": limit,
-            "search_sources": search_sources,
-            "search_notes": search_notes,
-            "minimum_score": minimum_score,
-        }
-        return self._make_request("POST", "/api/search", json=data)
-
-    def ask_simple(
-        self,
-        question: str,
-        strategy_model: str,
-        answer_model: str,
-        final_answer_model: str,
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Ask the knowledge base a question (simple, non-streaming)."""
-        data = {
-            "question": question,
-            "strategy_model": strategy_model,
-            "answer_model": answer_model,
-            "final_answer_model": final_answer_model,
-        }
-        # Use configured timeout for long-running ask operations
-        return self._make_request(
-            "POST", "/api/search/ask/simple", json=data, timeout=self.timeout
-        )
-
-    # Models API methods
-    def get_models(self, model_type: Optional[str] = None) -> List[Dict[Any, Any]]:
-        """Get all models with optional type filtering."""
-        params = {}
-        if model_type:
-            params["type"] = model_type
-        result = self._make_request("GET", "/api/models", params=params)
-        return result if isinstance(result, list) else [result]
-
-    def create_model(
-        self, name: str, provider: str, model_type: str
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Create a new model."""
-        data = {
-            "name": name,
-            "provider": provider,
-            "type": model_type,
-        }
-        return self._make_request("POST", "/api/models", json=data)
-
-    def delete_model(
-        self, model_id: str
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Delete a model."""
-        return self._make_request("DELETE", f"/api/models/{model_id}")
-
-    def get_default_models(self) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Get default model assignments."""
-        return self._make_request("GET", "/api/models/defaults")
-
-    def update_default_models(
-        self, **defaults
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Update default model assignments."""
-        return self._make_request("PUT", "/api/models/defaults", json=defaults)
-
-    # Transformations API methods
-    def get_transformations(self) -> List[Dict[Any, Any]]:
-        """Get all transformations."""
-        result = self._make_request("GET", "/api/transformations")
-        return result if isinstance(result, list) else [result]
-
-    def create_transformation(
-        self,
-        name: str,
-        title: str,
-        description: str,
-        prompt: str,
-        apply_default: bool = False,
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Create a new transformation."""
-        data = {
-            "name": name,
-            "title": title,
-            "description": description,
-            "prompt": prompt,
-            "apply_default": apply_default,
-        }
-        return self._make_request("POST", "/api/transformations", json=data)
-
-    def get_transformation(
-        self, transformation_id: str
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Get a specific transformation."""
-        return self._make_request("GET", f"/api/transformations/{transformation_id}")
-
-    def update_transformation(
-        self, transformation_id: str, **updates
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Update a transformation."""
-        return self._make_request(
-            "PUT", f"/api/transformations/{transformation_id}", json=updates
-        )
-
-    def delete_transformation(
-        self, transformation_id: str
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Delete a transformation."""
-        return self._make_request("DELETE", f"/api/transformations/{transformation_id}")
-
-    def execute_transformation(
-        self, transformation_id: str, input_text: str, model_id: str
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Execute a transformation on input text."""
-        data = {
-            "transformation_id": transformation_id,
-            "input_text": input_text,
-            "model_id": model_id,
-        }
-        # Use configured timeout for transformation operations
-        return self._make_request(
-            "POST", "/api/transformations/execute", json=data, timeout=self.timeout
-        )
-
-    # Notes API methods
-    def get_notes(self, notebook_id: Optional[str] = None) -> List[Dict[Any, Any]]:
-        """Get all notes with optional notebook filtering."""
-        params = {}
-        if notebook_id:
-            params["notebook_id"] = notebook_id
-        result = self._make_request("GET", "/api/notes", params=params)
-        return result if isinstance(result, list) else [result]
-
-    def create_note(
-        self,
-        content: str,
-        title: Optional[str] = None,
-        note_type: str = "human",
-        notebook_id: Optional[str] = None,
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Create a new note."""
-        data = {
-            "content": content,
-            "note_type": note_type,
-        }
-        if title:
-            data["title"] = title
-        if notebook_id:
-            data["notebook_id"] = notebook_id
-        return self._make_request("POST", "/api/notes", json=data)
-
-    def get_note(self, note_id: str) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Get a specific note."""
-        return self._make_request("GET", f"/api/notes/{note_id}")
-
-    def update_note(
-        self, note_id: str, **updates
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Update a note."""
-        return self._make_request("PUT", f"/api/notes/{note_id}", json=updates)
-
-    def delete_note(self, note_id: str) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Delete a note."""
-        return self._make_request("DELETE", f"/api/notes/{note_id}")
-
-    # Embedding API methods
-    def embed_content(
-        self, item_id: str, item_type: str, async_processing: bool = False
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Embed content for vector search."""
-        data = {
-            "item_id": item_id,
-            "item_type": item_type,
-            "async_processing": async_processing,
-        }
-        # Use configured timeout for embedding operations
-        return self._make_request("POST", "/api/embed", json=data, timeout=self.timeout)
-
-    def rebuild_embeddings(
-        self,
-        mode: str = "existing",
-        include_sources: bool = True,
-        include_notes: bool = True,
-        include_insights: bool = True,
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Rebuild embeddings in bulk.
-
-        Note: This operation can take a long time for large databases.
-        Consider increasing API_CLIENT_TIMEOUT to 600-900s for bulk rebuilds.
-        """
-        data = {
-            "mode": mode,
-            "include_sources": include_sources,
-            "include_notes": include_notes,
-            "include_insights": include_insights,
-        }
-        # Use double the configured timeout for bulk rebuild operations (or configured value if already high)
-        rebuild_timeout = max(self.timeout, min(self.timeout * 2, 3600.0))
-        return self._make_request(
-            "POST", "/api/embeddings/rebuild", json=data, timeout=rebuild_timeout
-        )
-
-    def get_rebuild_status(
-        self, command_id: str
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Get status of a rebuild operation."""
-        return self._make_request("GET", f"/api/embeddings/rebuild/{command_id}/status")
-
-    # Settings API methods
-    def get_settings(self) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Get all application settings."""
-        return self._make_request("GET", "/api/settings")
-
-    def update_settings(
-        self, **settings
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Update application settings."""
-        return self._make_request("PUT", "/api/settings", json=settings)
-
-    # Context API methods
-    def get_notebook_context(
-        self, notebook_id: str, context_config: Optional[Dict] = None
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Get context for a notebook."""
-        data: Dict[str, Any] = {"notebook_id": notebook_id}
-        if context_config:
-            data["context_config"] = context_config
-        result = self._make_request(
-            "POST", f"/api/notebooks/{notebook_id}/context", json=data
-        )
-        return result if isinstance(result, dict) else {}
-
-    # Sources API methods
-    def get_sources(self, notebook_id: Optional[str] = None) -> List[Dict[Any, Any]]:
-        """Get all sources with optional notebook filtering."""
-        params = {}
-        if notebook_id:
-            params["notebook_id"] = notebook_id
-        result = self._make_request("GET", "/api/sources", params=params)
-        return result if isinstance(result, list) else [result]
-
-    def create_source(
-        self,
-        notebook_id: Optional[str] = None,
-        notebooks: Optional[List[str]] = None,
-        source_type: str = "text",
-        url: Optional[str] = None,
-        file_path: Optional[str] = None,
-        content: Optional[str] = None,
-        title: Optional[str] = None,
-        transformations: Optional[List[str]] = None,
-        embed: bool = False,
-        delete_source: bool = False,
-        async_processing: bool = False,
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Create a new source."""
-        data = {
-            "type": source_type,
-            "embed": embed,
-            "delete_source": delete_source,
-            "async_processing": async_processing,
-        }
-
-        # Handle backward compatibility for notebook_id vs notebooks
-        if notebooks:
-            data["notebooks"] = notebooks
-        elif notebook_id:
-            data["notebook_id"] = notebook_id
-        else:
-            raise ValueError("Either notebook_id or notebooks must be provided")
-
-        if url:
-            data["url"] = url
-        if file_path:
-            data["file_path"] = file_path
-        if content:
-            data["content"] = content
-        if title:
-            data["title"] = title
-        if transformations:
-            data["transformations"] = transformations
-
-        # Use configured timeout for source creation (especially PDF processing with OCR)
-        return self._make_request(
-            "POST", "/api/sources/json", json=data, timeout=self.timeout
-        )
-
-    def get_source(self, source_id: str) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Get a specific source."""
-        return self._make_request("GET", f"/api/sources/{source_id}")
-
-    def get_source_status(
-        self, source_id: str
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Get processing status for a source."""
-        return self._make_request("GET", f"/api/sources/{source_id}/status")
-
-    def update_source(
-        self, source_id: str, **updates
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Update a source."""
-        return self._make_request("PUT", f"/api/sources/{source_id}", json=updates)
-
-    def delete_source(
-        self, source_id: str
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Delete a source."""
-        return self._make_request("DELETE", f"/api/sources/{source_id}")
-
-    # Insights API methods
-    def get_source_insights(self, source_id: str) -> List[Dict[Any, Any]]:
-        """Get all insights for a specific source."""
-        result = self._make_request("GET", f"/api/sources/{source_id}/insights")
-        return result if isinstance(result, list) else [result]
-
-    def get_insight(
-        self, insight_id: str
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Get a specific insight."""
-        return self._make_request("GET", f"/api/insights/{insight_id}")
-
-    def delete_insight(
-        self, insight_id: str
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Delete a specific insight."""
-        return self._make_request("DELETE", f"/api/insights/{insight_id}")
-
-    def save_insight_as_note(
-        self, insight_id: str, notebook_id: Optional[str] = None
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Convert an insight to a note."""
-        data = {}
-        if notebook_id:
-            data["notebook_id"] = notebook_id
-        return self._make_request(
-            "POST", f"/api/insights/{insight_id}/save-as-note", json=data
-        )
-
-    def create_source_insight(
-        self, source_id: str, transformation_id: str, model_id: Optional[str] = None
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Create a new insight for a source by running a transformation."""
-        data = {"transformation_id": transformation_id}
-        if model_id:
-            data["model_id"] = model_id
-        return self._make_request(
-            "POST", f"/api/sources/{source_id}/insights", json=data
-        )
-
-    # Episode Profiles API methods
-    def get_episode_profiles(self) -> List[Dict[Any, Any]]:
-        """Get all episode profiles."""
-        result = self._make_request("GET", "/api/episode-profiles")
-        return result if isinstance(result, list) else [result]
-
-    def get_episode_profile(
-        self, profile_name: str
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Get a specific episode profile by name."""
-        return self._make_request("GET", f"/api/episode-profiles/{profile_name}")
-
-    def create_episode_profile(
-        self,
-        name: str,
-        description: str = "",
-        speaker_config: str = "",
-        outline_provider: str = "",
-        outline_model: str = "",
-        transcript_provider: str = "",
-        transcript_model: str = "",
-        default_briefing: str = "",
-        num_segments: int = 5,
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Create a new episode profile."""
-        data = {
-            "name": name,
-            "description": description,
-            "speaker_config": speaker_config,
-            "outline_provider": outline_provider,
-            "outline_model": outline_model,
-            "transcript_provider": transcript_provider,
-            "transcript_model": transcript_model,
-            "default_briefing": default_briefing,
-            "num_segments": num_segments,
-        }
-        return self._make_request("POST", "/api/episode-profiles", json=data)
-
-    def update_episode_profile(
-        self, profile_id: str, **updates
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Update an episode profile."""
-        return self._make_request(
-            "PUT", f"/api/episode-profiles/{profile_id}", json=updates
-        )
-
-    def delete_episode_profile(
-        self, profile_id: str
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Delete an episode profile."""
-        return self._make_request("DELETE", f"/api/episode-profiles/{profile_id}")
-
-
-# Global client instance
-api_client = APIClient()

api/command_service.py

@@ -27,7 +27,7 @@ class CommandService:
             # surreal-commands expects: submit_command(app_name, command_name, args)
             cmd_id = submit_command(
                 module_name,  # This is actually the app name (e.g., "open_notebook")
-                command_name,  # Command name (e.g., "process_text")
+                command_name,  # Command name (e.g., "generate_podcast")
                 command_args,  # Input data
             )
             # Convert RecordID to string if needed

api/context_service.py

@@ -1,29 +0,0 @@
-"""
-Context service layer using API.
-"""
-
-from typing import Any, Dict, List, Optional, Union
-
-from loguru import logger
-
-from api.client import api_client
-
-
-class ContextService:
-    """Service layer for context operations using API."""
-
-    def __init__(self):
-        logger.info("Using API for context operations")
-
-    def get_notebook_context(
-        self, notebook_id: str, context_config: Optional[Dict] = None
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Get context for a notebook."""
-        result = api_client.get_notebook_context(
-            notebook_id=notebook_id, context_config=context_config
-        )
-        return result
-
-
-# Global service instance
-context_service = ContextService()

api/credentials_service.py

@@ -7,187 +7,55 @@ Extracted from the credentials router to follow the service layer pattern.
 All functions raise ValueError for business errors (router converts to HTTPException).
 """
 
-import ipaddress
 import os
-import socket
-from typing import Dict, List, Optional
-from urllib.parse import urlparse
+from typing import Any, Dict, List, Literal
 
 import httpx
 from loguru import logger
 from pydantic import SecretStr
 
-from api.models import CredentialResponse
+from api.models import CredentialResponse, validate_url_key_provider_required_fields
+from open_notebook.ai.connection_tester import normalize_anthropic_compatible_base_url
+from open_notebook.ai.model_discovery import (
+    ANTHROPIC_FALLBACK_MODELS,
+    audio_seed,
+    classify_model_type,
+    fetch_anthropic_model_ids,
+)
+from open_notebook.ai.provider_registry import PROVIDERS
 from open_notebook.domain.credential import Credential
-from open_notebook.utils.encryption import get_secret_from_env
+from open_notebook.utils.encryption import (
+    PBKDF2_MARKER,
+    decrypt_value,
+    encrypt_value,
+    get_secret_from_env,
+)
+from open_notebook.utils.ssl_config import httpx_verify_setting
+from open_notebook.utils.url_validation import (
+    prepare_pinned_http_target,
+)
+from open_notebook.utils.url_validation import (
+    validate_url as validate_url,  # re-export for routers
+)
 
 # =============================================================================
 # Constants
 # =============================================================================
 
-# Provider environment variable configuration.
+# Provider environment variable configuration, derived from the provider
+# registry (open_notebook/ai/provider_registry.py — the source of truth).
 # - "required": ALL listed env vars must be set for the provider to be considered configured.
 # - "required_any": at least ONE of the listed env vars must be set.
 # - "optional": additional env vars used during migration but not required.
 PROVIDER_ENV_CONFIG: Dict[str, dict] = {
-    "openai": {"required": ["OPENAI_API_KEY"]},
-    "anthropic": {"required": ["ANTHROPIC_API_KEY"]},
-    "google": {"required_any": ["GOOGLE_API_KEY", "GEMINI_API_KEY"]},
-    "groq": {"required": ["GROQ_API_KEY"]},
-    "mistral": {"required": ["MISTRAL_API_KEY"]},
-    "deepseek": {"required": ["DEEPSEEK_API_KEY"]},
-    "xai": {"required": ["XAI_API_KEY"]},
-    "openrouter": {"required": ["OPENROUTER_API_KEY"]},
-    "voyage": {"required": ["VOYAGE_API_KEY"]},
-    "elevenlabs": {"required": ["ELEVENLABS_API_KEY"]},
-    "ollama": {"required": ["OLLAMA_API_BASE"]},
-    "vertex": {
-        "required": ["VERTEX_PROJECT", "VERTEX_LOCATION"],
-        "optional": ["GOOGLE_APPLICATION_CREDENTIALS"],
-    },
-    "azure": {
-        "required": ["AZURE_OPENAI_API_KEY", "AZURE_OPENAI_ENDPOINT", "AZURE_OPENAI_API_VERSION"],
-        "optional": [
-            "AZURE_OPENAI_ENDPOINT_LLM",
-            "AZURE_OPENAI_ENDPOINT_EMBEDDING",
-            "AZURE_OPENAI_ENDPOINT_STT",
-            "AZURE_OPENAI_ENDPOINT_TTS",
-        ],
-    },
-    "openai_compatible": {
-        "required_any": ["OPENAI_COMPATIBLE_BASE_URL", "OPENAI_COMPATIBLE_API_KEY"],
-    },
-    "dashscope": {"required": ["DASHSCOPE_API_KEY"]},
-    "minimax": {"required": ["MINIMAX_API_KEY"]},
+    name: spec.env_config() for name, spec in PROVIDERS.items()
 }
 
 PROVIDER_MODALITIES: Dict[str, List[str]] = {
-    "openai": ["language", "embedding", "speech_to_text", "text_to_speech"],
-    "anthropic": ["language"],
-    "google": ["language", "embedding"],
-    "groq": ["language", "speech_to_text"],
-    "mistral": ["language", "embedding"],
-    "deepseek": ["language"],
-    "xai": ["language"],
-    "openrouter": ["language"],
-    "voyage": ["embedding"],
-    "elevenlabs": ["text_to_speech"],
-    "ollama": ["language", "embedding"],
-    "vertex": ["language", "embedding"],
-    "azure": ["language", "embedding", "speech_to_text", "text_to_speech"],
-    "openai_compatible": ["language", "embedding", "speech_to_text", "text_to_speech"],
-    "dashscope": ["language"],
-    "minimax": ["language"],
+    name: list(spec.modalities) for name, spec in PROVIDERS.items()
 }
 
 
-# =============================================================================
-# URL Validation (SSRF protection)
-# =============================================================================
-
-
-def validate_url(url: str, provider: str) -> None:
-    """
-    Validate URL format for API endpoints.
-
-    This is a self-hosted application, so we allow:
-    - Private IPs (10.x, 172.16-31.x, 192.168.x) for self-hosted services
-    - Localhost for local services (Ollama, LM Studio, etc.)
-
-    We only block:
-    - Invalid schemes (must be http or https)
-    - Malformed URLs
-    - Link-local addresses (169.254.x.x) - used for cloud metadata endpoints
-    - Hostnames that resolve to link-local addresses
-
-    Args:
-        url: The URL to validate
-        provider: The provider name (for logging/context)
-
-    Raises:
-        ValueError: If the URL is invalid
-    """
-    if not url or not url.strip():
-        return  # Empty URLs handled elsewhere
-
-    try:
-        parsed = urlparse(url.strip())
-
-        # Validate scheme - only http/https allowed
-        if parsed.scheme not in ("http", "https"):
-            raise ValueError(
-                f"Invalid URL scheme: '{parsed.scheme}'. Only http and https are allowed."
-            )
-
-        # Extract hostname
-        hostname = parsed.hostname
-        if not hostname:
-            raise ValueError("Invalid URL: hostname could not be determined.")
-
-        # Try to parse as IP address to check for dangerous addresses
-        try:
-            ip = ipaddress.ip_address(hostname)
-
-            # Block link-local addresses (169.254.x.x) - used for cloud metadata
-            # These are dangerous as they can expose cloud instance credentials
-            if ip.is_link_local:
-                raise ValueError(
-                    "Link-local addresses (169.254.x.x) are not allowed for security reasons. "
-                    "These addresses are used for cloud metadata endpoints."
-                )
-
-            # Block IPv4-mapped IPv6 addresses pointing to link-local
-            # e.g. ::ffff:169.254.169.254 bypasses IPv6 is_link_local check
-            if hasattr(ip, "ipv4_mapped") and ip.ipv4_mapped and ip.ipv4_mapped.is_link_local:
-                raise ValueError(
-                    "Link-local addresses (169.254.x.x) are not allowed for security reasons. "
-                    "These addresses are used for cloud metadata endpoints."
-                )
-
-        except ValueError as ve:
-            # Re-raise our own ValueErrors
-            if "Link-local" in str(ve) or "Invalid URL" in str(ve):
-                raise
-            # Not an IP address, it's a hostname - need to resolve and check
-            try:
-                # Resolve hostname to IP address
-                resolved_ips = socket.getaddrinfo(hostname, None)
-                for family, _, _, _, sockaddr in resolved_ips:
-                    ip_addr = sockaddr[0]
-                    try:
-                        parsed_ip = ipaddress.ip_address(ip_addr)
-                        if parsed_ip.is_link_local:
-                            raise ValueError(
-                                f"Hostname '{hostname}' resolves to a link-local address (169.254.x.x) which is not allowed for security reasons. "
-                                "These addresses are used for cloud metadata endpoints."
-                            )
-                        # Block IPv4-mapped IPv6 addresses pointing to link-local
-                        if (
-                            hasattr(parsed_ip, "ipv4_mapped")
-                            and parsed_ip.ipv4_mapped
-                            and parsed_ip.ipv4_mapped.is_link_local
-                        ):
-                            raise ValueError(
-                                f"Hostname '{hostname}' resolves to a link-local address (169.254.x.x) which is not allowed for security reasons. "
-                                "These addresses are used for cloud metadata endpoints."
-                            )
-                    except ValueError as inner_ve:
-                        if "link-local" in str(inner_ve).lower() or "Link-local" in str(inner_ve):
-                            raise
-                        # Skip non-IP addresses (e.g., IPv6 zones)
-                        continue
-            except socket.gaierror:
-                # Could not resolve hostname - allow it since the URL may be
-                # valid in the deployment environment (e.g., Azure endpoints,
-                # internal DNS names). We only block link-local addresses.
-                pass
-
-    except ValueError:
-        raise
-    except Exception:
-        raise ValueError("Invalid URL format. Check server logs for details.")
-
-
 # =============================================================================
 # Helpers
 # =============================================================================
@@ -202,7 +70,9 @@ def require_encryption_key() -> None:
         )
 
 
-def credential_to_response(cred: Credential, model_count: int = 0) -> CredentialResponse:
+def credential_to_response(
+    cred: Credential, model_count: int = 0
+) -> CredentialResponse:
     """Convert a Credential domain object to API response."""
     return CredentialResponse(
         id=cred.id or "",
@@ -219,10 +89,12 @@ def credential_to_response(cred: Credential, model_count: int = 0) -> Credential
         project=cred.project,
         location=cred.location,
         credentials_path=cred.credentials_path,
+        num_ctx=cred.num_ctx,
         has_api_key=cred.api_key is not None,
         created=str(cred.created) if cred.created else "",
         updated=str(cred.updated) if cred.updated else "",
         model_count=model_count,
+        decryption_error=cred.decryption_error,
     )
 
 
@@ -244,6 +116,14 @@ def get_default_modalities(provider: str) -> List[str]:
     return PROVIDER_MODALITIES.get(provider.lower(), ["language"])
 
 
+def ensure_provider_required_fields(cred: Credential) -> None:
+    """Validate provider-specific required fields on a merged credential before
+    save (update path). Delegates to the shared rule in api.models so the create
+    and update paths cannot drift."""
+    api_key = cred.api_key.get_secret_value() if cred.api_key else None
+    validate_url_key_provider_required_fields(cred.provider, cred.base_url, api_key)
+
+
 def create_credential_from_env(provider: str) -> Credential:
     """Create a Credential from environment variables for a given provider."""
     modalities = get_default_modalities(provider)
@@ -256,6 +136,15 @@ def create_credential_from_env(provider: str) -> Credential:
             modalities=modalities,
             base_url=os.environ.get("OLLAMA_API_BASE"),
         )
+    elif provider == "omlx":
+        api_key = os.environ.get("OMLX_API_KEY")
+        return Credential(
+            name=name,
+            provider=provider,
+            modalities=modalities,
+            api_key=SecretStr(api_key) if api_key else None,
+            base_url=os.environ.get("OMLX_API_BASE"),
+        )
     elif provider == "vertex":
         return Credential(
             name=name,
@@ -278,14 +167,15 @@ def create_credential_from_env(provider: str) -> Credential:
             endpoint_stt=os.environ.get("AZURE_OPENAI_ENDPOINT_STT"),
             endpoint_tts=os.environ.get("AZURE_OPENAI_ENDPOINT_TTS"),
         )
-    elif provider == "openai_compatible":
-        api_key = os.environ.get("OPENAI_COMPATIBLE_API_KEY")
+    elif provider in ("openai_compatible", "anthropic_compatible"):
+        env_prefix = provider.upper()
+        api_key = os.environ.get(f"{env_prefix}_API_KEY")
         return Credential(
             name=name,
             provider=provider,
             modalities=modalities,
             api_key=SecretStr(api_key) if api_key else None,
-            base_url=os.environ.get("OPENAI_COMPATIBLE_BASE_URL"),
+            base_url=os.environ.get(f"{env_prefix}_BASE_URL"),
         )
     elif provider == "google":
         # Support both GOOGLE_API_KEY and GEMINI_API_KEY (fallback)
@@ -297,16 +187,22 @@ def create_credential_from_env(provider: str) -> Credential:
             api_key=SecretStr(api_key) if api_key else None,
         )
     else:
-        # Simple API key providers
+        # Simple API key providers (plus an optional *_BASE_URL endpoint
+        # override for providers that declare one, e.g. regional endpoints)
         config = PROVIDER_ENV_CONFIG.get(provider, {})
         required = config.get("required", [])
         env_var = required[0] if required else None
         api_key = os.environ.get(env_var) if env_var else None
+        spec = PROVIDERS.get(provider)
+        base_url_env = spec.base_url_env if spec else None
         return Credential(
             name=name,
             provider=provider,
             modalities=modalities,
             api_key=SecretStr(api_key) if api_key else None,
+            base_url=(os.environ.get(base_url_env, "").strip() or None)
+            if base_url_env
+            else None,
         )
 
 
@@ -369,9 +265,12 @@ async def test_credential(credential_id: str) -> dict:
         config = cred.to_esperanto_config()
 
         from open_notebook.ai.connection_tester import (
+            _is_vertex_credentials_file_error,
+            _test_anthropic_compatible_connection,
             _test_azure_connection,
             _test_ollama_connection,
             _test_openai_compatible_connection,
+            classify_provider_test_error,
         )
 
         provider = cred.provider.lower()
@@ -396,6 +295,29 @@ async def test_credential(credential_id: str) -> dict:
             )
             return {"provider": provider, "success": success, "message": message}
 
+        if provider == "anthropic_compatible":
+            base_url = config.get("base_url")
+            api_key = config.get("api_key")
+            if not base_url:
+                return {
+                    "provider": provider,
+                    "success": False,
+                    "message": "No base URL configured",
+                }
+            success, message = await _test_anthropic_compatible_connection(
+                base_url, api_key
+            )
+            return {"provider": provider, "success": success, "message": message}
+
+        if provider == "omlx":
+            # Esperanto oMLX profile default; port 11435 avoids SurrealDB on 8000.
+            base_url = config.get("base_url") or "http://localhost:11435/v1"
+            api_key = config.get("api_key")
+            success, message = await _test_openai_compatible_connection(
+                base_url, api_key
+            )
+            return {"provider": provider, "success": success, "message": message}
+
         if provider == "azure":
             success, message = await _test_azure_connection(
                 endpoint=config.get("endpoint"),
@@ -430,17 +352,27 @@ async def test_credential(credential_id: str) -> dict:
             )
             lc_model = model.to_langchain()
             await lc_model.ainvoke("Hi")
-            return {"provider": provider, "success": True, "message": "Connection successful"}
+            return {
+                "provider": provider,
+                "success": True,
+                "message": "Connection successful",
+            }
 
         elif test_type == "embedding":
-            model = AIFactory.create_embedding(
+            embedding_model = AIFactory.create_embedding(
                 model_name=test_model, provider=provider, config=config
             )
-            await model.aembed(["test"])
-            return {"provider": provider, "success": True, "message": "Connection successful"}
+            await embedding_model.aembed(["test"])
+            return {
+                "provider": provider,
+                "success": True,
+                "message": "Connection successful",
+            }
 
         elif test_type == "text_to_speech":
-            AIFactory.create_text_to_speech(model_name=test_model, provider=provider, config=config)
+            AIFactory.create_text_to_speech(
+                model_name=test_model, provider=provider, config=config
+            )
             return {
                 "provider": provider,
                 "success": True,
@@ -454,19 +386,21 @@ async def test_credential(credential_id: str) -> dict:
         }
 
     except Exception as e:
+        if provider == "vertex" and _is_vertex_credentials_file_error(e):
+            logger.debug(
+                f"Vertex credentials file error for credential {credential_id}: {e}"
+            )
+            return {
+                "provider": provider,
+                "success": False,
+                "message": "Invalid or inaccessible credentials file",
+            }
+
         error_msg = str(e)
-        if "401" in error_msg or "unauthorized" in error_msg.lower():
-            return {"provider": provider, "success": False, "message": "Invalid API key"}
-        elif "403" in error_msg or "forbidden" in error_msg.lower():
-            return {"provider": provider, "success": False, "message": "API key lacks required permissions"}
-        elif "rate" in error_msg.lower() and "limit" in error_msg.lower():
-            return {"provider": provider, "success": True, "message": "Rate limited - but connection works"}
-        elif "not found" in error_msg.lower() and "model" in error_msg.lower():
-            return {"provider": provider, "success": True, "message": "API key valid (test model not available)"}
-        else:
+        success, message = classify_provider_test_error(error_msg)
+        if not success:
             logger.debug(f"Test connection error for credential {credential_id}: {e}")
-            truncated = error_msg[:100] + "..." if len(error_msg) > 100 else error_msg
-            return {"provider": provider, "success": False, "message": f"Error: {truncated}"}
+        return {"provider": provider, "success": success, "message": message}
 
 
 async def discover_with_config(provider: str, config: dict) -> List[dict]:
@@ -479,56 +413,100 @@ async def discover_with_config(provider: str, config: dict) -> List[dict]:
     api_key = config.get("api_key")
     base_url = config.get("base_url")
 
+    def models_endpoint(url: str) -> str:
+        trimmed = url.rstrip("/")
+        if trimmed.endswith("/models"):
+            return trimmed
+        return f"{trimmed}/models"
+
     # Static model lists for providers without a listing API
     STATIC_MODELS: Dict[str, List[str]] = {
-        "anthropic": [
-            "claude-opus-4-20250514",
-            "claude-sonnet-4-20250514",
-            "claude-3-5-sonnet-20241022",
-            "claude-3-5-haiku-20241022",
-            "claude-3-opus-20240229",
-            "claude-3-sonnet-20240229",
-            "claude-3-haiku-20240307",
-        ],
         "voyage": [
-            "voyage-3", "voyage-3-lite", "voyage-code-3",
-            "voyage-finance-2", "voyage-law-2", "voyage-multilingual-2",
+            "voyage-3",
+            "voyage-3-lite",
+            "voyage-code-3",
+            "voyage-finance-2",
+            "voyage-law-2",
+            "voyage-multilingual-2",
         ],
         "elevenlabs": [
-            "eleven_multilingual_v2", "eleven_turbo_v2_5",
-            "eleven_turbo_v2", "eleven_monolingual_v1",
+            "eleven_multilingual_v2",
+            "eleven_turbo_v2_5",
+            "eleven_turbo_v2",
+            "eleven_monolingual_v1",
+            "scribe_v1",  # speech-to-text
+        ],
+        "deepgram": [
+            # TTS (Aura) voices
+            "aura-2-thalia-en",
+            "aura-2-andromeda-en",
+            "aura-2-helena-en",
+            "aura-2-apollo-en",
+            "aura-2-arcas-en",
+            "aura-2-asteria-en",
+            "aura-2-athena-en",
+            "aura-2-hera-en",
+            "aura-2-hermes-en",
+            "aura-2-atlas-en",
+            # STT (Nova / Whisper) transcription models
+            "nova-3",
+            "nova-2",
+            "whisper-large",
+            "whisper-medium",
+            "whisper-small",
+            "whisper-base",
+            "whisper-tiny",
         ],
     }
 
     if provider in STATIC_MODELS:
         if not api_key and provider != "ollama":
             return []
-        return [
-            {"name": m, "provider": provider}
-            for m in STATIC_MODELS[provider]
-        ]
+        return [{"name": m, "provider": provider} for m in STATIC_MODELS[provider]]
 
-    # API-based discovery URLs (OpenAI-style /models endpoints)
+    if provider == "anthropic":
+        if not api_key:
+            return []
+        try:
+            model_names = await fetch_anthropic_model_ids(api_key)
+        except Exception as e:
+            logger.warning(
+                f"Failed to discover Anthropic models, using static fallback: {e}"
+            )
+            model_names = list(ANTHROPIC_FALLBACK_MODELS)
+        return [{"name": m, "provider": "anthropic"} for m in model_names]
+
+    # API-based discovery URLs (OpenAI-style /models endpoints), from the registry
     url_map = {
-        "openai": "https://api.openai.com/v1/models",
-        "groq": "https://api.groq.com/openai/v1/models",
-        "mistral": "https://api.mistral.ai/v1/models",
-        "deepseek": "https://api.deepseek.com/models",
-        "xai": "https://api.x.ai/v1/models",
-        "openrouter": "https://openrouter.ai/api/v1/models",
-        "dashscope": "https://dashscope.aliyuncs.com/compatible-mode/v1/models",
-        "minimax": "https://api.minimax.io/v1/models",
+        name: spec.openai_compat_discovery_url
+        for name, spec in PROVIDERS.items()
+        if spec.openai_compat_discovery_url
     }
 
     if provider == "ollama":
         ollama_url = base_url or "http://localhost:11434"
         try:
-            async with httpx.AsyncClient() as client:
-                response = await client.get(f"{ollama_url}/api/tags", timeout=10.0)
+            # Pin DNS at request time so httpx cannot re-resolve to a
+            # metadata address after validation (DNS rebinding TOCTOU).
+            target = await prepare_pinned_http_target(
+                f"{ollama_url.rstrip('/')}/api/tags", "ollama"
+            )
+            headers = dict(target.headers)
+            async with httpx.AsyncClient(verify=httpx_verify_setting()) as client:
+                response = await client.get(
+                    target.url,
+                    headers=headers,
+                    timeout=10.0,
+                    extensions=target.extensions,
+                )
                 response.raise_for_status()
                 data = response.json()
                 return [
-                    {"name": m.get("name", ""), "provider": "ollama"}
+                    {
+                        "name": m.get("name", ""),
+                        "provider": "ollama",
+                        "model_type": classify_model_type(m.get("name", ""), "ollama"),
+                    }
                     for m in data.get("models", [])
                     if m.get("name")
                 ]
@@ -540,12 +518,20 @@ async def discover_with_config(provider: str, config: dict) -> List[dict]:
         if not base_url:
             return []
         try:
-            headers = {}
+            # Pin DNS at request time so httpx cannot re-resolve to a
+            # metadata address after validation (DNS rebinding TOCTOU).
+            target = await prepare_pinned_http_target(
+                models_endpoint(base_url), "openai_compatible"
+            )
+            headers = dict(target.headers)
             if api_key:
                 headers["Authorization"] = f"Bearer {api_key}"
-            async with httpx.AsyncClient() as client:
+            async with httpx.AsyncClient(verify=httpx_verify_setting()) as client:
                 response = await client.get(
-                    f"{base_url.rstrip('/')}/models", headers=headers, timeout=30.0,
+                    target.url,
+                    headers=headers,
+                    timeout=30.0,
+                    extensions=target.extensions,
                 )
                 response.raise_for_status()
                 data = response.json()
@@ -558,16 +544,82 @@ async def discover_with_config(provider: str, config: dict) -> List[dict]:
             logger.warning(f"Failed to discover openai_compatible models: {e}")
             return []
 
+    if provider == "anthropic_compatible":
+        if not base_url or not api_key:
+            return []
+        try:
+            normalized_base_url = normalize_anthropic_compatible_base_url(base_url)
+            models_url = f"{normalized_base_url}/models"
+            # Pin DNS at request time (DNS-rebinding TOCTOU), mirroring the
+            # openai_compatible discovery path above.
+            target = await prepare_pinned_http_target(
+                models_url, "anthropic_compatible"
+            )
+            headers = dict(target.headers)
+            headers["x-api-key"] = api_key
+            headers["anthropic-version"] = "2023-06-01"
+            async with httpx.AsyncClient(verify=httpx_verify_setting()) as client:
+                response = await client.get(
+                    target.url,
+                    headers=headers,
+                    timeout=30.0,
+                    extensions=target.extensions,
+                )
+                response.raise_for_status()
+                data = response.json()
+                return [
+                    {"name": m.get("id", ""), "provider": "anthropic_compatible"}
+                    for m in data.get("data", [])
+                    if m.get("id")
+                ]
+        except Exception as e:
+            logger.warning(f"Failed to discover anthropic_compatible models: {e}")
+            return []
+
+    if provider == "omlx":
+        omlx_url = base_url or "http://localhost:11435/v1"
+        try:
+            target = await prepare_pinned_http_target(models_endpoint(omlx_url), "omlx")
+            headers = dict(target.headers)
+            if api_key:
+                headers["Authorization"] = f"Bearer {api_key}"
+            async with httpx.AsyncClient(verify=httpx_verify_setting()) as client:
+                response = await client.get(
+                    target.url,
+                    headers=headers,
+                    timeout=30.0,
+                    extensions=target.extensions,
+                )
+                response.raise_for_status()
+                data = response.json()
+                return [
+                    {"name": m.get("id", ""), "provider": "omlx"}
+                    for m in data.get("data", [])
+                    if m.get("id")
+                ]
+        except Exception as e:
+            logger.warning(f"Failed to discover omlx models: {e}")
+            return []
+
     if provider == "azure":
         endpoint = config.get("endpoint")
         api_version = config.get("api_version", "2024-10-21")
         if not endpoint or not api_key:
             return []
         try:
+            # Pin DNS at request time so httpx cannot re-resolve to a
+            # metadata address after validation (DNS rebinding TOCTOU).
             url = f"{endpoint.rstrip('/')}/openai/models?api-version={api_version}"
-            headers = {"api-key": api_key}
-            async with httpx.AsyncClient() as client:
-                response = await client.get(url, headers=headers, timeout=30.0)
+            target = await prepare_pinned_http_target(url, "azure")
+            headers = dict(target.headers)
+            headers["api-key"] = api_key
+            async with httpx.AsyncClient(verify=httpx_verify_setting()) as client:
+                response = await client.get(
+                    target.url,
+                    headers=headers,
+                    timeout=30.0,
+                    extensions=target.extensions,
+                )
                 response.raise_for_status()
                 data = response.json()
                 return [
@@ -583,10 +635,9 @@ async def discover_with_config(provider: str, config: dict) -> List[dict]:
         # Vertex AI requires service-account OAuth2 for model listing.
         # Return a curated static list of well-known Vertex models instead.
         VERTEX_MODELS = [
-            "gemini-2.0-flash",
-            "gemini-2.0-flash-lite",
-            "gemini-1.5-pro",
-            "gemini-1.5-flash",
+            "gemini-3.5-flash",
+            "gemini-2.5-flash",
+            "gemini-2.5-flash-lite",
             "text-embedding-005",
         ]
         return [{"name": m, "provider": "vertex"} for m in VERTEX_MODELS]
@@ -594,7 +645,7 @@ async def discover_with_config(provider: str, config: dict) -> List[dict]:
     if provider == "google":
         try:
             headers = {"X-Goog-Api-Key": api_key} if api_key else {}
-            async with httpx.AsyncClient() as client:
+            async with httpx.AsyncClient(verify=httpx_verify_setting()) as client:
                 response = await client.get(
                     "https://generativelanguage.googleapis.com/v1/models",
                     headers=headers,
@@ -615,22 +666,66 @@ async def discover_with_config(provider: str, config: dict) -> List[dict]:
             logger.warning(f"Failed to discover Google models: {e}")
             return []
 
+    if provider == "cohere":
+        # Cohere is not OpenAI-compatible; use esperanto's bespoke discovery.
+        if not api_key:
+            return []
+        try:
+            import asyncio
+
+            from esperanto import AIFactory
+
+            models = await asyncio.to_thread(
+                AIFactory.get_provider_models, "cohere", api_key=api_key
+            )
+            return [
+                {"name": m.id, "provider": "cohere"}
+                for m in models
+                if m.type in ("language", "embedding")
+            ]
+        except Exception as e:
+            logger.warning(f"Failed to discover Cohere models: {e}")
+            return []
+
     # Standard OpenAI-style API discovery
     discovery_url = url_map.get(provider)
+    user_supplied_url = False
+    # A credential's base URL override is also where its models are listed:
+    # OpenAI (gateways) and providers that declare a *_BASE_URL override, e.g.
+    # SiliconFlow's mainland-China api.siliconflow.cn. Other providers keep
+    # their registry URL, which may carry provider-specific query params.
+    spec = PROVIDERS.get(provider)
+    honors_base_url = provider == "openai" or bool(spec and spec.base_url_env)
+    if base_url and base_url.strip() and discovery_url and honors_base_url:
+        discovery_url = models_endpoint(base_url.strip())
+        user_supplied_url = True
     if not discovery_url or not api_key:
         return []
 
     try:
-        async with httpx.AsyncClient() as client:
+        headers = {"Authorization": f"Bearer {api_key}"}
+        if user_supplied_url:
+            # Pin DNS at request time so httpx cannot re-resolve a
+            # user-supplied host to a metadata address after validation
+            # (DNS rebinding TOCTOU) — mirrors the openai_compatible path.
+            target = await prepare_pinned_http_target(discovery_url, provider)
+            request_url = target.url
+            headers.update(target.headers)
+            extensions = target.extensions
+        else:
+            request_url = discovery_url
+            extensions = {}
+        async with httpx.AsyncClient(verify=httpx_verify_setting()) as client:
             response = await client.get(
-                discovery_url,
-                headers={"Authorization": f"Bearer {api_key}"},
+                request_url,
+                headers=headers,
                 timeout=30.0,
+                extensions=extensions,
             )
             response.raise_for_status()
             data = response.json()
 
-            return [
+            discovered = [
                 {
                     "name": m.get("id", ""),
                     "provider": provider,
@@ -639,6 +734,16 @@ async def discover_with_config(provider: str, config: dict) -> List[dict]:
                 for m in data.get("data", [])
                 if m.get("id")
             ]
+            # Some /models listings (OpenRouter, MiniMax) don't surface their
+            # TTS/STT catalog, so seed the audio model ids esperanto supports
+            # (only on top of a successful listing).
+            if discovered:
+                seen = {m["name"] for m in discovered}
+                for name, _model_type in audio_seed(provider):
+                    if name not in seen:
+                        discovered.append({"name": name, "provider": provider})
+                        seen.add(name)
+            return discovered
     except Exception as e:
         logger.warning(f"Failed to discover {provider} models: {e}")
         return []
@@ -827,7 +932,9 @@ async def migrate_from_env() -> dict:
                 not_configured.append(provider)
                 continue
 
-            logger.info(f"[{provider}] Env vars detected, checking for existing credentials")
+            logger.info(
+                f"[{provider}] Env vars detected, checking for existing credentials"
+            )
 
             existing = await Credential.get_by_provider(provider)
             if existing:
@@ -839,6 +946,9 @@ async def migrate_from_env() -> dict:
 
             logger.info(f"[{provider}] Creating credential from env vars")
             cred = create_credential_from_env(provider)
+            # Same URL checks as credentials created through the API.
+            if cred.base_url:
+                await validate_url(cred.base_url, provider)
             await cred.save()
             logger.info(f"[{provider}] Credential saved successfully (id={cred.id})")
 
@@ -887,3 +997,215 @@ async def migrate_from_env() -> dict:
         "not_configured": not_configured,
         "errors": errors,
     }
+
+
+# Placeholder the domain layer substitutes when a stored key cannot be
+# decrypted (open_notebook/domain/credential.py). Must never be re-encrypted:
+# it is a marker of a previous failure, not a secret.
+_UNDECRYPTABLE_PLACEHOLDER = "UNDECRYPTABLE"
+
+# Closed set of per-record outcomes for the encryption-scheme pass.
+ReencryptStatus = Literal["migrated", "skipped", "error"]
+
+
+def _reencrypt_stored_value(stored: object) -> tuple[ReencryptStatus, str]:
+    """
+    Decide the fate of one stored api_key value.
+
+    Returns (status, payload) where status is one of migrated, skipped, or
+    error. Payload is the new marked value for migrated rows, otherwise a
+    short reason code. Reasons carry record context only, never key material.
+
+    Args:
+        stored: The raw api_key value read from the database.
+
+    Returns:
+        Tuple of status and payload.
+    """
+    if stored is None or stored == "":
+        return ("skipped", "empty")
+    if not isinstance(stored, str):
+        return ("error", "unexpected-type")
+    if stored == _UNDECRYPTABLE_PLACEHOLDER:
+        return ("error", "decrypt-placeholder")
+    if stored.startswith(PBKDF2_MARKER):
+        try:
+            decrypt_value(stored)
+        except ValueError:
+            return ("error", "undecryptable-marked")
+        return ("skipped", "already-migrated")
+    try:
+        secret = decrypt_value(stored)
+    except ValueError:
+        return ("error", "undecryptable-legacy")
+    return ("migrated", encrypt_value(secret))
+
+
+def _record_settled_outcome(
+    status: ReencryptStatus,
+    payload: str,
+    label: str,
+    skipped: List[str],
+    errors: List[str],
+) -> None:
+    """
+    Record a skipped/error outcome plus its log line.
+
+    Only called for settled (non-migrated) outcomes; migrated rows take
+    their own write path. Reasons carry record context only, never key
+    material.
+
+    Args:
+        status: Settled outcome (skipped or error).
+        payload: Skip reason or error reason.
+        label: Record label for logs and summaries.
+        skipped: Summary list to append skip entries to.
+        errors: Summary list to append error entries to.
+    """
+    if status == "skipped":
+        skipped.append(f"{label}: {payload}")
+    else:
+        logger.warning(f"[{label}] Left untouched: {payload}")
+        errors.append(f"{label}: {payload}")
+
+
+def _migrate_provider_config_entries(
+    creds: Dict[str, Any],
+    skipped: List[str],
+    errors: List[str],
+) -> List[str]:
+    """
+    Re-encrypt nested provider_configs entries in memory.
+
+    Mutates entries with successfully decrypted values; undecryptable
+    entries are left untouched and reported. Never writes to the database:
+    the caller persists the map and owns the migrated labels, so a failed
+    rewrite is never reported as migrated.
+
+    Args:
+        creds: The raw credentials map from the singleton record.
+        skipped: Summary list to append skip entries to.
+        errors: Summary list to append error entries to.
+
+    Returns:
+        Labels re-encrypted in memory, pending persistence.
+    """
+    pending: List[str] = []
+    for provider, entries in creds.items():
+        if not isinstance(entries, list):
+            errors.append(f"provider_configs/{provider}: unexpected-group-type")
+            continue
+        for index, entry in enumerate(entries):
+            if not isinstance(entry, dict):
+                errors.append(
+                    f"provider_configs/{provider}/{index}: unexpected-entry-type"
+                )
+                continue
+            label = f"provider_configs/{provider}/{entry.get('name', index)}"
+            try:
+                status, payload = _reencrypt_stored_value(entry.get("api_key"))
+            except Exception as e:
+                logger.error(f"[{label}] Re-encryption FAILED: {type(e).__name__}")
+                errors.append(f"{label}: unexpected-failure")
+                continue
+            if status == "migrated":
+                entry["api_key"] = payload
+                pending.append(label)
+            else:
+                _record_settled_outcome(status, payload, label, skipped, errors)
+    return pending
+
+
+async def migrate_encryption_scheme() -> dict:
+    """
+    Re-encrypt stored API keys into the versioned PBKDF2 format.
+
+    One-shot, idempotent, and fail-closed per record. Reads raw database
+    rows (never domain-decrypted objects, which can carry UNDECRYPTABLE
+    placeholders) and writes a row back only when its value decrypted
+    successfully. Undecryptable rows are left untouched and reported.
+
+    Run as a single-admin operation: there is no locking, so concurrent
+    credential edits during the pass can be overwritten. Re-running over a
+    migrated corpus changes nothing.
+
+    Returns dict with message, migrated, skipped, errors.
+    """
+    from open_notebook.database.repository import (
+        ensure_record_id,
+        repo_query,
+        repo_update,
+        repo_upsert,
+    )
+    from open_notebook.domain.provider_config import ProviderConfig
+
+    logger.info("=== Starting encryption-scheme migration ===")
+
+    require_encryption_key()
+    logger.info("Encryption key verified")
+
+    migrated: List[str] = []
+    skipped: List[str] = []
+    errors: List[str] = []
+
+    rows = await repo_query("SELECT id, api_key FROM credential")
+    for row in rows:
+        record_id = str(row.get("id", ""))
+        try:
+            status, payload = _reencrypt_stored_value(row.get("api_key"))
+        except Exception as e:
+            logger.error(f"[{record_id}] Re-encryption FAILED: {type(e).__name__}")
+            errors.append(f"{record_id}: unexpected-failure")
+            continue
+        if status == "migrated":
+            try:
+                await repo_update("credential", record_id, {"api_key": payload})
+            except Exception as e:
+                logger.error(f"[{record_id}] Write FAILED: {type(e).__name__}")
+                errors.append(f"{record_id}: write-failed")
+                continue
+            logger.info(f"[{record_id}] Re-encrypted to versioned format")
+            migrated.append(record_id)
+        else:
+            _record_settled_outcome(status, payload, record_id, skipped, errors)
+
+    singleton_rows = await repo_query(
+        "SELECT * FROM ONLY $record_id",
+        {"record_id": ensure_record_id(ProviderConfig.record_id)},
+    )
+    if singleton_rows:
+        data = singleton_rows[0] if isinstance(singleton_rows, list) else singleton_rows
+        creds = data.get("credentials") if isinstance(data, dict) else None
+        if isinstance(creds, dict):
+            pending = _migrate_provider_config_entries(creds, skipped, errors)
+            if pending:
+                try:
+                    await repo_upsert(
+                        "open_notebook",
+                        ProviderConfig.record_id,
+                        {"credentials": creds},
+                    )
+                except Exception as e:
+                    logger.error(f"Singleton rewrite FAILED: {type(e).__name__}")
+                    errors.extend(f"{label}: write-failed" for label in pending)
+                    pending = []
+                migrated.extend(pending)
+
+    logger.info(
+        "=== Encryption-scheme migration complete === "
+        f"migrated={len(migrated)} skipped={len(skipped)} errors={len(errors)}"
+    )
+    if migrated:
+        logger.info(f"  Migrated: {', '.join(migrated)}")
+    if errors:
+        logger.error(f"  Errors: {'; '.join(errors)}")
+
+    return {
+        "message": (
+            f"Encryption migration complete. Migrated {len(migrated)} "
+            f"record(s) with {len(errors)} error(s)."
+        ),
+        "migrated": migrated,
+        "skipped": skipped,
+        "errors": errors,
+    }

api/embedding_service.py

@@ -1,27 +0,0 @@
-"""
-Embedding service layer using API.
-"""
-
-from typing import Any, Dict, List, Union
-
-from loguru import logger
-
-from api.client import api_client
-
-
-class EmbeddingService:
-    """Service layer for embedding operations using API."""
-
-    def __init__(self):
-        logger.info("Using API for embedding operations")
-
-    def embed_content(
-        self, item_id: str, item_type: str
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Embed content for vector search."""
-        result = api_client.embed_content(item_id=item_id, item_type=item_type)
-        return result
-
-
-# Global service instance
-embedding_service = EmbeddingService()

api/episode_profiles_service.py

@@ -1,112 +0,0 @@
-"""
-Episode profiles service layer using API.
-"""
-
-from typing import List
-
-from loguru import logger
-
-from api.client import api_client
-from open_notebook.podcasts.models import EpisodeProfile
-
-
-class EpisodeProfilesService:
-    """Service layer for episode profiles operations using API."""
-
-    def __init__(self):
-        logger.info("Using API for episode profiles operations")
-
-    def get_all_episode_profiles(self) -> List[EpisodeProfile]:
-        """Get all episode profiles."""
-        profiles_data = api_client.get_episode_profiles()
-        # Convert API response to EpisodeProfile objects
-        profiles = []
-        for profile_data in profiles_data:
-            profile = EpisodeProfile(
-                name=profile_data["name"],
-                description=profile_data.get("description", ""),
-                speaker_config=profile_data["speaker_config"],
-                outline_provider=profile_data["outline_provider"],
-                outline_model=profile_data["outline_model"],
-                transcript_provider=profile_data["transcript_provider"],
-                transcript_model=profile_data["transcript_model"],
-                default_briefing=profile_data["default_briefing"],
-                num_segments=profile_data["num_segments"],
-            )
-            profile.id = profile_data["id"]
-            profiles.append(profile)
-        return profiles
-
-    def get_episode_profile(self, profile_name: str) -> EpisodeProfile:
-        """Get a specific episode profile by name."""
-        profile_response = api_client.get_episode_profile(profile_name)
-        profile_data = (
-            profile_response
-            if isinstance(profile_response, dict)
-            else profile_response[0]
-        )
-        profile = EpisodeProfile(
-            name=profile_data["name"],
-            description=profile_data.get("description", ""),
-            speaker_config=profile_data["speaker_config"],
-            outline_provider=profile_data["outline_provider"],
-            outline_model=profile_data["outline_model"],
-            transcript_provider=profile_data["transcript_provider"],
-            transcript_model=profile_data["transcript_model"],
-            default_briefing=profile_data["default_briefing"],
-            num_segments=profile_data["num_segments"],
-        )
-        profile.id = profile_data["id"]
-        return profile
-
-    def create_episode_profile(
-        self,
-        name: str,
-        description: str = "",
-        speaker_config: str = "",
-        outline_provider: str = "",
-        outline_model: str = "",
-        transcript_provider: str = "",
-        transcript_model: str = "",
-        default_briefing: str = "",
-        num_segments: int = 5,
-    ) -> EpisodeProfile:
-        """Create a new episode profile."""
-        profile_response = api_client.create_episode_profile(
-            name=name,
-            description=description,
-            speaker_config=speaker_config,
-            outline_provider=outline_provider,
-            outline_model=outline_model,
-            transcript_provider=transcript_provider,
-            transcript_model=transcript_model,
-            default_briefing=default_briefing,
-            num_segments=num_segments,
-        )
-        profile_data = (
-            profile_response
-            if isinstance(profile_response, dict)
-            else profile_response[0]
-        )
-        profile = EpisodeProfile(
-            name=profile_data["name"],
-            description=profile_data.get("description", ""),
-            speaker_config=profile_data["speaker_config"],
-            outline_provider=profile_data["outline_provider"],
-            outline_model=profile_data["outline_model"],
-            transcript_provider=profile_data["transcript_provider"],
-            transcript_model=profile_data["transcript_model"],
-            default_briefing=profile_data["default_briefing"],
-            num_segments=profile_data["num_segments"],
-        )
-        profile.id = profile_data["id"]
-        return profile
-
-    def delete_episode_profile(self, profile_id: str) -> bool:
-        """Delete an episode profile."""
-        api_client.delete_episode_profile(profile_id)
-        return True
-
-
-# Global service instance
-episode_profiles_service = EpisodeProfilesService()

api/insights_service.py

@@ -1,100 +0,0 @@
-"""
-Insights service layer using API.
-"""
-
-from typing import List, Optional
-
-from loguru import logger
-
-from api.client import api_client
-from open_notebook.domain.notebook import Note, SourceInsight
-
-
-class InsightsService:
-    """Service layer for insights operations using API."""
-
-    def __init__(self):
-        logger.info("Using API for insights operations")
-
-    def get_source_insights(self, source_id: str) -> List[SourceInsight]:
-        """Get all insights for a specific source."""
-        insights_data = api_client.get_source_insights(source_id)
-        # Convert API response to SourceInsight objects
-        insights = []
-        for insight_data in insights_data:
-            insight = SourceInsight(
-                insight_type=insight_data["insight_type"],
-                content=insight_data["content"],
-            )
-            insight.id = insight_data["id"]
-            insight.created = insight_data["created"]
-            insight.updated = insight_data["updated"]
-            insights.append(insight)
-        return insights
-
-    def get_insight(self, insight_id: str) -> SourceInsight:
-        """Get a specific insight."""
-        insight_response = api_client.get_insight(insight_id)
-        insight_data = (
-            insight_response
-            if isinstance(insight_response, dict)
-            else insight_response[0]
-        )
-        insight = SourceInsight(
-            insight_type=insight_data["insight_type"],
-            content=insight_data["content"],
-        )
-        insight.id = insight_data["id"]
-        insight.created = insight_data["created"]
-        insight.updated = insight_data["updated"]
-        # Note: source_id from API response is not stored; use await insight.get_source() if needed
-        return insight
-
-    def delete_insight(self, insight_id: str) -> bool:
-        """Delete a specific insight."""
-        api_client.delete_insight(insight_id)
-        return True
-
-    def save_insight_as_note(
-        self, insight_id: str, notebook_id: Optional[str] = None
-    ) -> Note:
-        """Convert an insight to a note."""
-        note_response = api_client.save_insight_as_note(insight_id, notebook_id)
-        note_data = (
-            note_response if isinstance(note_response, dict) else note_response[0]
-        )
-        note = Note(
-            title=note_data["title"],
-            content=note_data["content"],
-            note_type=note_data["note_type"],
-        )
-        note.id = note_data["id"]
-        note.created = note_data["created"]
-        note.updated = note_data["updated"]
-        return note
-
-    def create_source_insight(
-        self, source_id: str, transformation_id: str, model_id: Optional[str] = None
-    ) -> SourceInsight:
-        """Create a new insight for a source by running a transformation."""
-        insight_response = api_client.create_source_insight(
-            source_id, transformation_id, model_id
-        )
-        insight_data = (
-            insight_response
-            if isinstance(insight_response, dict)
-            else insight_response[0]
-        )
-        insight = SourceInsight(
-            insight_type=insight_data["insight_type"],
-            content=insight_data["content"],
-        )
-        insight.id = insight_data["id"]
-        insight.created = insight_data["created"]
-        insight.updated = insight_data["updated"]
-        # Note: source_id from API response is not stored; use await insight.get_source() if needed
-        return insight
-
-
-# Global service instance
-insights_service = InsightsService()

api/main.py

@@ -3,6 +3,15 @@ from dotenv import load_dotenv
 
 load_dotenv()
 
+# Keep the internal SurrealDB websocket out of any configured HTTP proxy
+# (issue #1160). Must run after load_dotenv() (so a proxy set in .env is
+# already visible) and before the DB is touched.
+from open_notebook.utils.proxy import ensure_internal_no_proxy
+
+ensure_internal_no_proxy()
+
+import asyncio
+import os
 from contextlib import asynccontextmanager
 
 from fastapi import FastAPI, Request
@@ -12,21 +21,12 @@ from loguru import logger
 from starlette.exceptions import HTTPException as StarletteHTTPException
 
 from api.auth import PasswordAuthMiddleware
-from open_notebook.exceptions import (
-    AuthenticationError,
-    ConfigurationError,
-    ExternalServiceError,
-    InvalidInputError,
-    NetworkError,
-    NotFoundError,
-    OpenNotebookError,
-    RateLimitError,
-)
+from api.middleware import MaxBodySizeMiddleware, get_max_upload_size_bytes
 from api.routers import (
     auth,
+    capabilities,
     chat,
     config,
-    context,
     credentials,
     embedding,
     embedding_rebuild,
@@ -37,6 +37,7 @@ from api.routers import (
     notebooks,
     notes,
     podcasts,
+    providers,
     search,
     settings,
     source_chat,
@@ -46,8 +47,78 @@ from api.routers import (
 )
 from api.routers import commands as commands_router
 from open_notebook.database.async_migrate import AsyncMigrationManager
+from open_notebook.exceptions import (
+    AuthenticationError,
+    ConfigurationError,
+    ExternalServiceError,
+    InvalidInputError,
+    NetworkError,
+    NotFoundError,
+    OpenNotebookError,
+    RateLimitError,
+    UnsupportedTypeException,
+)
 from open_notebook.utils.encryption import get_secret_from_env
 
+
+def _parse_cors_origins(raw: str) -> list[str]:
+    """Parse CORS_ORIGINS env value into a list of origins."""
+    value = raw.strip()
+    if value == "*":
+        return ["*"]
+    return [origin.strip() for origin in value.split(",") if origin.strip()]
+
+
+# Parsed once at module load; CORS_ORIGINS changes require a restart.
+_cors_origins_raw = os.getenv("CORS_ORIGINS")
+CORS_ALLOWED_ORIGINS = _parse_cors_origins(_cors_origins_raw or "*")
+CORS_IS_DEFAULT_WILDCARD = _cors_origins_raw is None
+# Keyed on the parsed list, not on whether the env var was set: an operator
+# who explicitly sets CORS_ORIGINS=* must get the same wildcard treatment as
+# the default, or credentials would combine with a wildcard origin - the
+# exact reflect-any-Origin behavior this flag exists to prevent.
+CORS_ALLOW_CREDENTIALS = "*" not in CORS_ALLOWED_ORIGINS
+
+# Parsed once at module load; OPEN_NOTEBOOK_MAX_UPLOAD_SIZE_MB changes require a restart.
+MAX_UPLOAD_SIZE_BYTES = get_max_upload_size_bytes()
+
+DATABASE_STARTUP_RETRY_ATTEMPTS = 12
+DATABASE_STARTUP_RETRY_INITIAL_DELAY_SECONDS = 1
+DATABASE_STARTUP_RETRY_MAX_DELAY_SECONDS = 5
+# Per-probe ceiling so a hung connection cannot exceed the retry budget or
+# block startup indefinitely. A probe that exceeds this is treated as a
+# transient failure and retried like any other unreachable-database attempt.
+DATABASE_STARTUP_RETRY_PROBE_TIMEOUT_SECONDS = 5
+
+
+def _cors_headers(request: Request) -> dict[str, str]:
+    """
+    Build CORS headers for error responses.
+
+    Mirrors Starlette CORSMiddleware behavior: reflects the request Origin
+    when the origin is allowed (or when wildcard is configured, since
+    browsers reject `Access-Control-Allow-Origin: *` combined with
+    credentials). Omits `Access-Control-Allow-Origin` for disallowed
+    origins so the browser blocks the error body from leaking cross-origin.
+    Only claims Access-Control-Allow-Credentials when the real CORSMiddleware
+    would (see its allow_credentials comment above) - otherwise error
+    responses would grant credentialed access the success path doesn't.
+    """
+    origin = request.headers.get("origin")
+    headers: dict[str, str] = {
+        "Access-Control-Allow-Methods": "*",
+        "Access-Control-Allow-Headers": "*",
+    }
+    if CORS_ALLOW_CREDENTIALS:
+        headers["Access-Control-Allow-Credentials"] = "true"
+
+    if origin and ("*" in CORS_ALLOWED_ORIGINS or origin in CORS_ALLOWED_ORIGINS):
+        headers["Access-Control-Allow-Origin"] = origin
+        headers["Vary"] = "Origin"
+
+    return headers
+
+
 # Import commands to register them in the API process
 try:
     logger.info("Commands imported in API process")
@@ -55,14 +126,66 @@ except Exception as e:
     logger.error(f"Failed to import commands in API process: {e}")
 
 
+async def _wait_for_database(migration_manager: AsyncMigrationManager) -> None:
+    """
+    Wait for SurrealDB to accept connections before running migrations.
+
+    Docker Compose can start the API before the database name is resolvable. Keep
+    migration errors fail-fast by only retrying this lightweight readiness probe.
+    """
+    attempts = max(1, DATABASE_STARTUP_RETRY_ATTEMPTS)
+    delay = DATABASE_STARTUP_RETRY_INITIAL_DELAY_SECONDS
+
+    for attempt in range(1, attempts + 1):
+        try:
+            await asyncio.wait_for(
+                migration_manager.ping(),
+                timeout=DATABASE_STARTUP_RETRY_PROBE_TIMEOUT_SECONDS,
+            )
+            if attempt > 1:
+                logger.info(f"Database became reachable on attempt {attempt}")
+            return
+        except Exception as e:
+            if attempt == attempts:
+                logger.error(
+                    f"Database did not become reachable after {attempts} attempts"
+                )
+                raise
+
+            logger.warning(
+                "Database is not reachable yet "
+                f"(attempt {attempt}/{attempts}): {str(e)}. "
+                f"Retrying in {delay:g} seconds..."
+            )
+            await asyncio.sleep(delay)
+            delay = min(delay * 2, DATABASE_STARTUP_RETRY_MAX_DELAY_SECONDS)
+
+
+async def _run_database_migrations() -> None:
+    """Run startup database migrations after SurrealDB is reachable."""
+    migration_manager = AsyncMigrationManager()
+    await _wait_for_database(migration_manager)
+
+    current_version = await migration_manager.get_current_version()
+    logger.info(f"Current database version: {current_version}")
+
+    if await migration_manager.needs_migration():
+        logger.warning("Database migrations are pending. Running migrations...")
+        await migration_manager.run_migration_up()
+        new_version = await migration_manager.get_current_version()
+        logger.success(
+            f"Migrations completed successfully. Database is now at version {new_version}"
+        )
+    else:
+        logger.info("Database is already at the latest version. No migrations needed.")
+
+
 @asynccontextmanager
 async def lifespan(app: FastAPI):
     """
     Lifespan event handler for the FastAPI application.
     Runs database migrations automatically on startup.
     """
-    import os
-
     # Startup: Security checks
     logger.info("Starting API initialization...")
 
@@ -77,36 +200,13 @@ async def lifespan(app: FastAPI):
     # Run database migrations
 
     try:
-        migration_manager = AsyncMigrationManager()
-        current_version = await migration_manager.get_current_version()
-        logger.info(f"Current database version: {current_version}")
-
-        if await migration_manager.needs_migration():
-            logger.warning("Database migrations are pending. Running migrations...")
-            await migration_manager.run_migration_up()
-            new_version = await migration_manager.get_current_version()
-            logger.success(
-                f"Migrations completed successfully. Database is now at version {new_version}"
-            )
-        else:
-            logger.info(
-                "Database is already at the latest version. No migrations needed."
-            )
+        await _run_database_migrations()
     except Exception as e:
         logger.error(f"CRITICAL: Database migration failed: {str(e)}")
         logger.exception(e)
         # Fail fast - don't start the API with an outdated database schema
         raise RuntimeError(f"Failed to run database migrations: {str(e)}") from e
 
-    # Run podcast profile data migration (legacy strings -> Model registry)
-    try:
-        from open_notebook.podcasts.migration import migrate_podcast_profiles
-
-        await migrate_podcast_profiles()
-    except Exception as e:
-        logger.warning(f"Podcast profile migration encountered errors: {e}")
-        # Non-fatal: profiles can be migrated manually via UI
-
     logger.success("API initialization completed successfully")
 
     # Yield control to the application
@@ -122,6 +222,16 @@ app = FastAPI(
     lifespan=lifespan,
 )
 
+if CORS_IS_DEFAULT_WILDCARD:
+    logger.warning(
+        "CORS_ORIGINS is not set — API accepts cross-origin requests from any "
+        "origin (default: '*'). For production deployments, set CORS_ORIGINS to "
+        "your frontend origin(s), e.g. "
+        "CORS_ORIGINS=https://notebook.example.com"
+    )
+else:
+    logger.info(f"CORS allowed origins: {CORS_ALLOWED_ORIGINS}")
+
 # Add password authentication middleware first
 # Exclude /api/auth/status and /api/config from authentication
 app.add_middleware(
@@ -137,11 +247,31 @@ app.add_middleware(
     ],
 )
 
-# Add CORS middleware last (so it processes first)
+# Reject oversized request bodies before they reach auth or routing - added
+# after PasswordAuthMiddleware (so it wraps around it) so a too-large request
+# is rejected before spending any work checking credentials.
+logger.info(
+    f"Max request body size: {MAX_UPLOAD_SIZE_BYTES / (1024 * 1024):g}MB "
+    "(set OPEN_NOTEBOOK_MAX_UPLOAD_SIZE_MB to change)"
+)
+app.add_middleware(MaxBodySizeMiddleware, max_body_size=MAX_UPLOAD_SIZE_BYTES)
+
+# Add CORS middleware last (so it processes first, and so it can attach
+# CORS headers to a 413 raised by MaxBodySizeMiddleware)
+#
+# allow_credentials is tied to whether CORS_ORIGINS resolves to specific
+# origins: combining allow_origins=["*"] with allow_credentials=True makes
+# Starlette reflect the request's Origin header verbatim (browsers reject a
+# literal "*" alongside credentials), which defeats the origin allowlist.
+# The frontend never sends credentialed requests (withCredentials: false)
+# and auth is a Bearer header, not a cookie, so this isn't independently
+# exploitable today - but there's no reason to allow it for any wildcard
+# case. Once an operator explicitly scopes CORS_ORIGINS to real origins,
+# credentialed cross-origin requests to those origins are safe to allow.
 app.add_middleware(
     CORSMiddleware,
-    allow_origins=["*"],  # In production, replace with specific origins
-    allow_credentials=True,
+    allow_origins=CORS_ALLOWED_ORIGINS,
+    allow_credentials=CORS_ALLOW_CREDENTIALS,
     allow_methods=["*"],
     allow_headers=["*"],
 )
@@ -159,31 +289,13 @@ async def custom_http_exception_handler(request: Request, exc: StarletteHTTPExce
     FastAPI, this handler won't be called. In that case, configure your reverse proxy
     to add CORS headers to error responses.
     """
-    # Get the origin from the request
-    origin = request.headers.get("origin", "*")
-
     return JSONResponse(
         status_code=exc.status_code,
         content={"detail": exc.detail},
-        headers={
-            **(exc.headers or {}), "Access-Control-Allow-Origin": origin,
-            "Access-Control-Allow-Credentials": "true",
-            "Access-Control-Allow-Methods": "*",
-            "Access-Control-Allow-Headers": "*",
-        },
+        headers={**(exc.headers or {}), **_cors_headers(request)},
     )
 
 
-def _cors_headers(request: Request) -> dict[str, str]:
-    origin = request.headers.get("origin", "*")
-    return {
-        "Access-Control-Allow-Origin": origin,
-        "Access-Control-Allow-Credentials": "true",
-        "Access-Control-Allow-Methods": "*",
-        "Access-Control-Allow-Headers": "*",
-    }
-
-
 @app.exception_handler(NotFoundError)
 async def not_found_error_handler(request: Request, exc: NotFoundError):
     return JSONResponse(
@@ -247,6 +359,17 @@ async def external_service_error_handler(request: Request, exc: ExternalServiceE
     )
 
 
+@app.exception_handler(UnsupportedTypeException)
+async def unsupported_type_error_handler(
+    request: Request, exc: UnsupportedTypeException
+):
+    return JSONResponse(
+        status_code=415,
+        content={"detail": str(exc)},
+        headers=_cors_headers(request),
+    )
+
+
 @app.exception_handler(OpenNotebookError)
 async def open_notebook_error_handler(request: Request, exc: OpenNotebookError):
     return JSONResponse(
@@ -269,7 +392,6 @@ app.include_router(
     embedding_rebuild.router, prefix="/api/embeddings", tags=["embeddings"]
 )
 app.include_router(settings.router, prefix="/api", tags=["settings"])
-app.include_router(context.router, prefix="/api", tags=["context"])
 app.include_router(sources.router, prefix="/api", tags=["sources"])
 app.include_router(insights.router, prefix="/api", tags=["insights"])
 app.include_router(commands_router.router, prefix="/api", tags=["commands"])
@@ -279,6 +401,8 @@ app.include_router(speaker_profiles.router, prefix="/api", tags=["speaker-profil
 app.include_router(chat.router, prefix="/api", tags=["chat"])
 app.include_router(source_chat.router, prefix="/api", tags=["source-chat"])
 app.include_router(credentials.router, prefix="/api", tags=["credentials"])
+app.include_router(providers.router, prefix="/api", tags=["providers"])
+app.include_router(capabilities.router, prefix="/api", tags=["capabilities"])
 app.include_router(languages.router, prefix="/api", tags=["languages"])
 
 

api/middleware.py

@@ -0,0 +1,122 @@
+import os
+
+from loguru import logger
+from starlette.datastructures import Headers
+from starlette.types import ASGIApp, Message, Receive, Scope, Send
+
+# Matches the file-size guidance already documented in
+# docs/3-USER-GUIDE/adding-sources.md ("Very large files (>100MB) - Timeout").
+DEFAULT_MAX_UPLOAD_SIZE_MB = 100
+
+
+def get_max_upload_size_bytes() -> int:
+    """Read the configured max request body size, in bytes.
+
+    Configurable via OPEN_NOTEBOOK_MAX_UPLOAD_SIZE_MB for deployments that
+    need larger audio/video uploads; falls back to the default on unset,
+    malformed, or non-positive values (a zero/negative limit would reject
+    every request that has a body).
+    """
+    raw = os.environ.get("OPEN_NOTEBOOK_MAX_UPLOAD_SIZE_MB", "").strip()
+    try:
+        mb = float(raw) if raw else DEFAULT_MAX_UPLOAD_SIZE_MB
+    except ValueError:
+        mb = DEFAULT_MAX_UPLOAD_SIZE_MB
+    if mb <= 0:
+        logger.warning(
+            f"OPEN_NOTEBOOK_MAX_UPLOAD_SIZE_MB={raw!r} is not a positive size; "
+            f"using the default of {DEFAULT_MAX_UPLOAD_SIZE_MB}MB"
+        )
+        mb = DEFAULT_MAX_UPLOAD_SIZE_MB
+    return int(mb * 1024 * 1024)
+
+
+class _RequestBodyTooLarge(Exception):
+    pass
+
+
+class MaxBodySizeMiddleware:
+    """
+    Raw ASGI middleware rejecting requests whose body exceeds a configured
+    maximum, so a large upload can't exhaust memory/disk on a deployment
+    with no fronting proxy enforcing its own limit (e.g. the shipped
+    docker-compose.yml, which exposes the API directly).
+
+    Implemented at the raw ASGI level (not BaseHTTPMiddleware) so the check
+    can run ahead of FastAPI's own body/form parsing instead of after it.
+    Rejects on the `Content-Length` header up front when present (the common
+    case, and cheap), and also counts bytes as the body streams in - a
+    client can lie about Content-Length or omit it entirely with chunked
+    transfer-encoding.
+    """
+
+    def __init__(self, app: ASGIApp, max_body_size: int) -> None:
+        self.app = app
+        self.max_body_size = max_body_size
+
+    async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
+        if scope["type"] != "http":
+            await self.app(scope, receive, send)
+            return
+
+        headers = Headers(scope=scope)
+        content_length = headers.get("content-length")
+        if content_length is not None:
+            try:
+                if int(content_length) > self.max_body_size:
+                    logger.warning(
+                        f"Rejected {scope.get('method', '?')} {scope.get('path', '?')}: "
+                        f"declared body of {content_length} bytes exceeds the "
+                        f"{self.max_body_size}-byte limit"
+                    )
+                    await _send_413(send)
+                    return
+            except ValueError:
+                pass  # malformed header - fall through to streaming enforcement
+
+        total_size = 0
+        response_started = False
+
+        async def send_wrapper(message: Message) -> None:
+            nonlocal response_started
+            if message["type"] == "http.response.start":
+                response_started = True
+            await send(message)
+
+        async def receive_wrapper() -> Message:
+            nonlocal total_size
+            message = await receive()
+            if message["type"] == "http.request":
+                total_size += len(message.get("body") or b"")
+                if total_size > self.max_body_size:
+                    raise _RequestBodyTooLarge()
+            return message
+
+        try:
+            await self.app(scope, receive_wrapper, send_wrapper)
+        except _RequestBodyTooLarge:
+            logger.warning(
+                f"Rejected {scope.get('method', '?')} {scope.get('path', '?')}: "
+                f"streamed body exceeded the {self.max_body_size}-byte limit"
+            )
+            if not response_started:
+                await _send_413(send)
+            # Else the app already started responding - nothing safe to send;
+            # let the connection drop rather than violate the ASGI protocol
+            # with a second response.start.
+
+
+async def _send_413(send: Send) -> None:
+    await send(
+        {
+            "type": "http.response.start",
+            "status": 413,
+            "headers": [(b"content-type", b"application/json")],
+        }
+    )
+    await send(
+        {
+            "type": "http.response.body",
+            "body": b'{"detail":"Request body exceeds the maximum allowed upload size"}',
+        }
+    )

api/models.py

@@ -28,11 +28,46 @@ class NotebookResponse(BaseModel):
     note_count: int
 
 
+class RecentlyViewedResponse(BaseModel):
+    type: Literal["notebook", "source"]
+    id: str
+    title: str
+    last_viewed_at: str
+
+
 # Search models
-class SearchRequest(BaseModel):
+class NotebookScopeMixin(BaseModel):
+    """Optional notebook scope shared by Search and Ask requests (#574, #87).
+
+    Both `notebook_id` (single, the shape #574 proposed and existing clients
+    already send) and `notebook_ids` (several) are accepted; `scope_notebook_ids`
+    merges them. An empty scope means the whole knowledge base.
+    """
+
+    notebook_id: Optional[str] = Field(
+        None, description="Restrict results to a single notebook"
+    )
+    notebook_ids: Optional[List[str]] = Field(
+        None,
+        max_length=50,
+        description="Restrict results to these notebooks (omit or empty for all)",
+    )
+
+    @property
+    def scope_notebook_ids(self) -> List[str]:
+        # Keep empty strings so validation rejects them instead of silently
+        # widening the scope to the whole knowledge base.
+        merged: List[str] = []
+        for nb_id in [self.notebook_id, *(self.notebook_ids or [])]:
+            if nb_id is not None and nb_id not in merged:
+                merged.append(nb_id)
+        return merged
+
+
+class SearchRequest(NotebookScopeMixin):
     query: str = Field(..., description="Search query")
     type: Literal["text", "vector"] = Field("text", description="Search type")
-    limit: int = Field(100, description="Maximum number of results", le=1000)
+    limit: int = Field(100, description="Maximum number of results", ge=1, le=1000)
     search_sources: bool = Field(True, description="Include sources in search")
     search_notes: bool = Field(True, description="Include notes in search")
     minimum_score: float = Field(
@@ -46,7 +81,7 @@ class SearchResponse(BaseModel):
     search_type: str = Field(..., description="Type of search performed")
 
 
-class AskRequest(BaseModel):
+class AskRequest(NotebookScopeMixin):
     question: str = Field(..., description="Question to ask the knowledge base")
     strategy_model: str = Field(..., description="Model ID for query strategy")
     answer_model: str = Field(..., description="Model ID for individual answers")
@@ -112,6 +147,9 @@ class TransformationCreate(BaseModel):
     apply_default: bool = Field(
         False, description="Whether to apply this transformation by default"
     )
+    model_id: Optional[str] = Field(
+        None, description="Model ID to use by default for this transformation"
+    )
 
 
 class TransformationUpdate(BaseModel):
@@ -126,6 +164,9 @@ class TransformationUpdate(BaseModel):
     apply_default: Optional[bool] = Field(
         None, description="Whether to apply this transformation by default"
     )
+    model_id: Optional[str] = Field(
+        None, description="Model ID to use by default for this transformation"
+    )
 
 
 class TransformationResponse(BaseModel):
@@ -135,6 +176,7 @@ class TransformationResponse(BaseModel):
     description: str
     prompt: str
     apply_default: bool
+    model_id: Optional[str] = None
     created: str
     updated: str
 
@@ -146,7 +188,9 @@ class TransformationExecuteRequest(BaseModel):
         ..., description="ID of the transformation to execute"
     )
     input_text: str = Field(..., description="Text to transform")
-    model_id: str = Field(..., description="Model ID to use for the transformation")
+    model_id: Optional[str] = Field(
+        None, description="Model ID to use for this transformation run"
+    )
 
 
 class TransformationExecuteResponse(BaseModel):
@@ -154,7 +198,7 @@ class TransformationExecuteResponse(BaseModel):
 
     output: str = Field(..., description="Transformed text")
     transformation_id: str = Field(..., description="ID of the transformation used")
-    model_id: str = Field(..., description="Model ID used")
+    model_id: Optional[str] = Field(None, description="Model ID used")
 
 
 # Default Prompt API models
@@ -261,6 +305,9 @@ class SettingsResponse(BaseModel):
     default_content_processing_engine_url: Optional[str] = None
     default_embedding_option: Optional[str] = None
     auto_delete_files: Optional[str] = None
+    docling_ocr: Optional[bool] = None
+    docling_formulas: Optional[bool] = None
+    docling_vision: Optional[bool] = None
     youtube_preferred_languages: Optional[List[str]] = None
 
 
@@ -269,6 +316,9 @@ class SettingsUpdate(BaseModel):
     default_content_processing_engine_url: Optional[str] = None
     default_embedding_option: Optional[str] = None
     auto_delete_files: Optional[str] = None
+    docling_ocr: Optional[bool] = None
+    docling_formulas: Optional[bool] = None
+    docling_vision: Optional[bool] = None
     youtube_preferred_languages: Optional[List[str]] = None
 
 
@@ -285,7 +335,9 @@ class SourceCreate(BaseModel):
     )
     # New multi-notebook support
     notebooks: Optional[List[str]] = Field(
-        None, description="List of notebook IDs to add the source to"
+        None,
+        max_length=50,
+        description="List of notebook IDs to add the source to (max 50)",
     )
     # Required fields
     type: str = Field(..., description="Source type: link, upload, or text")
@@ -294,7 +346,9 @@ class SourceCreate(BaseModel):
     content: Optional[str] = Field(None, description="Text content for text type")
     title: Optional[str] = Field(None, description="Source title")
     transformations: Optional[List[str]] = Field(
-        default_factory=list, description="Transformation IDs to apply"
+        default_factory=list,
+        max_length=50,
+        description="Transformation IDs to apply (max 50)",
     )
     embed: bool = Field(False, description="Whether to embed content for vector search")
     delete_source: bool = Field(
@@ -366,38 +420,16 @@ class SourceListResponse(BaseModel):
     processing_info: Optional[Dict[str, Any]] = None
 
 
-# Context API models
-class ContextConfig(BaseModel):
-    sources: Dict[str, str] = Field(
-        default_factory=dict, description="Source inclusion config {source_id: level}"
-    )
-    notes: Dict[str, str] = Field(
-        default_factory=dict, description="Note inclusion config {note_id: level}"
-    )
-
-
-class ContextRequest(BaseModel):
-    notebook_id: str = Field(..., description="Notebook ID to get context for")
-    context_config: Optional[ContextConfig] = Field(
-        None, description="Context configuration"
-    )
-
-
-class ContextResponse(BaseModel):
-    notebook_id: str
-    sources: List[Dict[str, Any]] = Field(..., description="Source context data")
-    notes: List[Dict[str, Any]] = Field(..., description="Note context data")
-    total_tokens: Optional[int] = Field(None, description="Estimated token count")
-
-
 # Insights API models
 class SourceInsightResponse(BaseModel):
     id: str
     source_id: str
     insight_type: str
     content: str
-    created: str
-    updated: str
+    # Optional: insights created before migration 19 have no timestamps,
+    # and the API must return null for them (never the string "None").
+    created: Optional[str] = None
+    updated: Optional[str] = None
 
 
 class InsightCreationResponse(BaseModel):
@@ -447,12 +479,8 @@ class SetApiKeyRequest(BaseModel):
     base_url: Optional[str] = Field(
         None, description="Base URL for URL-based providers (Ollama, OpenAI-compatible)"
     )
-    endpoint: Optional[str] = Field(
-        None, description="Endpoint URL for Azure OpenAI"
-    )
-    api_version: Optional[str] = Field(
-        None, description="API version for Azure OpenAI"
-    )
+    endpoint: Optional[str] = Field(None, description="Endpoint URL for Azure OpenAI")
+    api_version: Optional[str] = Field(None, description="API version for Azure OpenAI")
     endpoint_llm: Optional[str] = Field(
         None, description="Service-specific endpoint for LLM (Azure)"
     )
@@ -554,11 +582,106 @@ class MigrationResult(BaseModel):
 
 # Notebook delete cascade models
 # Credential models
+
+# Kept in sync with the provider registry
+# (open_notebook/ai/provider_registry.py PROVIDERS — the backend source of
+# truth). A Literal can't be built at runtime, so this is the one remaining
+# manual copy; tests/test_credential_provider_validation.py enforces the sync.
+# The frontend consumes GET /api/providers at runtime and needs no edit.
+SupportedProvider = Literal[
+    "openai",
+    "anthropic",
+    "google",
+    "groq",
+    "mistral",
+    "deepseek",
+    "xai",
+    "openrouter",
+    "dashscope",
+    "minimax",
+    "novita",
+    "siliconflow",
+    "zai",
+    "ppq",
+    "cohere",
+    "voyage",
+    "elevenlabs",
+    "deepgram",
+    "ollama",
+    "omlx",
+    "azure",
+    "vertex",
+    "openai_compatible",
+    "anthropic_compatible",
+]
+
+
+class ProviderInfoResponse(BaseModel):
+    """Provider metadata from the provider registry."""
+
+    name: str = Field(..., description="Provider identifier (e.g. openai)")
+    display_name: str = Field(..., description="Human-friendly provider name")
+    modalities: List[str] = Field(
+        ..., description="Default modalities supported by the provider"
+    )
+    docs_url: Optional[str] = Field(
+        None, description="Where to get an API key / set the provider up"
+    )
+    env_configured: bool = Field(
+        ..., description="Whether the provider is configured via environment variables"
+    )
+
+
+class CapabilitiesResponse(BaseModel):
+    """Runtime availability of the opt-in heavy extraction engines.
+
+    Reflects what is actually importable/reachable in this container — not merely
+    what the OPEN_NOTEBOOK_ENABLE_* flags request — so the UI can gate engine
+    options honestly (e.g. still show "unavailable" while a first-boot install
+    is in progress). See docs/7-DEVELOPMENT/decisions/ADR-007-optin-runtimes.md.
+    """
+
+    docling_available: bool = Field(
+        ...,
+        description="Docling is installed: the docling document engine, OCR toggle and image sources work.",
+    )
+    crawl4ai_available: bool = Field(
+        ...,
+        description="Crawl4AI is usable: the local package is installed OR a remote server is configured.",
+    )
+    crawl4ai_remote_configured: bool = Field(
+        ...,
+        description="A remote Crawl4AI endpoint is configured via CRAWL4AI_API_URL (no local install needed).",
+    )
+
+
+def validate_url_key_provider_required_fields(
+    provider: Optional[str],
+    base_url: Optional[str],
+    api_key: Optional[str],
+) -> None:
+    """Shared required-field rule for providers that need BOTH a base URL and an
+    API key (currently anthropic_compatible).
+
+    Called from both the create path (CreateCredentialRequest validator, which sees
+    the full request payload) and the update path
+    (credentials_service.ensure_provider_required_fields, which runs against the
+    merged credential). Raises ValueError when a required field is missing.
+    """
+    if (provider or "").lower() == "anthropic_compatible":
+        if not base_url or not str(base_url).strip():
+            raise ValueError("Anthropic-compatible credentials require a base URL")
+        if not api_key or not str(api_key).strip():
+            raise ValueError("Anthropic-compatible credentials require an API key")
+
+
 class CreateCredentialRequest(BaseModel):
     """Request to create a new credential."""
 
     name: str = Field(..., description="Credential name")
-    provider: str = Field(..., description="Provider name (openai, anthropic, etc.)")
+    provider: SupportedProvider = Field(
+        ..., description="Provider name (openai, anthropic, etc.)"
+    )
     modalities: List[str] = Field(
         default_factory=list,
         description="Supported modalities (language, embedding, text_to_speech, speech_to_text)",
@@ -576,6 +699,16 @@ class CreateCredentialRequest(BaseModel):
     credentials_path: Optional[str] = Field(
         None, description="Credentials file path (Vertex)"
     )
+    num_ctx: Optional[int] = Field(
+        None, description="Context window size (Ollama only; defaults to 8192)"
+    )
+
+    @model_validator(mode="after")
+    def _validate_provider_required_fields(self):
+        validate_url_key_provider_required_fields(
+            self.provider, self.base_url, self.api_key
+        )
+        return self
 
 
 class UpdateCredentialRequest(BaseModel):
@@ -594,6 +727,9 @@ class UpdateCredentialRequest(BaseModel):
     project: Optional[str] = Field(None, description="Project ID")
     location: Optional[str] = Field(None, description="Location")
     credentials_path: Optional[str] = Field(None, description="Credentials path")
+    num_ctx: Optional[int] = Field(
+        None, description="Context window size (Ollama only; defaults to 8192)"
+    )
 
 
 class CredentialResponse(BaseModel):
@@ -613,10 +749,12 @@ class CredentialResponse(BaseModel):
     project: Optional[str] = None
     location: Optional[str] = None
     credentials_path: Optional[str] = None
+    num_ctx: Optional[int] = None
     has_api_key: bool = False
     created: str
     updated: str
     model_count: int = 0
+    decryption_error: Optional[str] = None
 
 
 class CredentialDeleteResponse(BaseModel):
@@ -683,3 +821,6 @@ class NotebookDeleteResponse(BaseModel):
     unlinked_sources: int = Field(
         ..., description="Number of sources unlinked from notebook"
     )
+    deleted_chat_sessions: int = Field(
+        ..., description="Number of chat sessions deleted"
+    )

api/models_service.py

@@ -1,112 +0,0 @@
-"""
-Models service layer using API.
-"""
-
-from typing import List, Optional
-
-from loguru import logger
-
-from api.client import api_client
-from open_notebook.ai.models import DefaultModels, Model
-
-
-class ModelsService:
-    """Service layer for models operations using API."""
-
-    def __init__(self):
-        logger.info("Using API for models operations")
-
-    def get_all_models(self, model_type: Optional[str] = None) -> List[Model]:
-        """Get all models with optional type filtering."""
-        models_data = api_client.get_models(model_type=model_type)
-        # Convert API response to Model objects
-        models = []
-        for model_data in models_data:
-            model = Model(
-                name=model_data["name"],
-                provider=model_data["provider"],
-                type=model_data["type"],
-            )
-            model.id = model_data["id"]
-            model.created = model_data["created"]
-            model.updated = model_data["updated"]
-            models.append(model)
-        return models
-
-    def create_model(self, name: str, provider: str, model_type: str) -> Model:
-        """Create a new model."""
-        response = api_client.create_model(name, provider, model_type)
-        model_data = response if isinstance(response, dict) else response[0]
-        model = Model(
-            name=model_data["name"],
-            provider=model_data["provider"],
-            type=model_data["type"],
-        )
-        model.id = model_data["id"]
-        model.created = model_data["created"]
-        model.updated = model_data["updated"]
-        return model
-
-    def delete_model(self, model_id: str) -> bool:
-        """Delete a model."""
-        api_client.delete_model(model_id)
-        return True
-
-    def get_default_models(self) -> DefaultModels:
-        """Get default model assignments."""
-        response = api_client.get_default_models()
-        defaults_data = response if isinstance(response, dict) else response[0]
-        defaults = DefaultModels()
-
-        # Set the values from API response
-        defaults.default_chat_model = defaults_data.get("default_chat_model")
-        defaults.default_transformation_model = defaults_data.get(
-            "default_transformation_model"
-        )
-        defaults.large_context_model = defaults_data.get("large_context_model")
-        defaults.default_text_to_speech_model = defaults_data.get(
-            "default_text_to_speech_model"
-        )
-        defaults.default_speech_to_text_model = defaults_data.get(
-            "default_speech_to_text_model"
-        )
-        defaults.default_embedding_model = defaults_data.get("default_embedding_model")
-        defaults.default_tools_model = defaults_data.get("default_tools_model")
-
-        return defaults
-
-    def update_default_models(self, defaults: DefaultModels) -> DefaultModels:
-        """Update default model assignments."""
-        updates = {
-            "default_chat_model": defaults.default_chat_model,
-            "default_transformation_model": defaults.default_transformation_model,
-            "large_context_model": defaults.large_context_model,
-            "default_text_to_speech_model": defaults.default_text_to_speech_model,
-            "default_speech_to_text_model": defaults.default_speech_to_text_model,
-            "default_embedding_model": defaults.default_embedding_model,
-            "default_tools_model": defaults.default_tools_model,
-        }
-
-        response = api_client.update_default_models(**updates)
-        defaults_data = response if isinstance(response, dict) else response[0]
-
-        # Update the defaults object with the response
-        defaults.default_chat_model = defaults_data.get("default_chat_model")
-        defaults.default_transformation_model = defaults_data.get(
-            "default_transformation_model"
-        )
-        defaults.large_context_model = defaults_data.get("large_context_model")
-        defaults.default_text_to_speech_model = defaults_data.get(
-            "default_text_to_speech_model"
-        )
-        defaults.default_speech_to_text_model = defaults_data.get(
-            "default_speech_to_text_model"
-        )
-        defaults.default_embedding_model = defaults_data.get("default_embedding_model")
-        defaults.default_tools_model = defaults_data.get("default_tools_model")
-
-        return defaults
-
-
-# Global service instance
-models_service = ModelsService()

api/notebook_service.py

@@ -1,87 +0,0 @@
-"""
-Notebook service layer using API.
-"""
-
-from typing import List, Optional
-
-from loguru import logger
-
-from api.client import api_client
-from open_notebook.domain.notebook import Notebook
-
-
-class NotebookService:
-    """Service layer for notebook operations using API."""
-
-    def __init__(self):
-        logger.info("Using API for notebook operations")
-
-    def get_all_notebooks(self, order_by: str = "updated desc") -> List[Notebook]:
-        """Get all notebooks."""
-        notebooks_data = api_client.get_notebooks(order_by=order_by)
-        # Convert API response to Notebook objects
-        notebooks = []
-        for nb_data in notebooks_data:
-            nb = Notebook(
-                name=nb_data["name"],
-                description=nb_data["description"],
-                archived=nb_data["archived"],
-            )
-            nb.id = nb_data["id"]
-            nb.created = nb_data["created"]
-            nb.updated = nb_data["updated"]
-            notebooks.append(nb)
-        return notebooks
-
-    def get_notebook(self, notebook_id: str) -> Optional[Notebook]:
-        """Get a specific notebook."""
-        response = api_client.get_notebook(notebook_id)
-        nb_data = response if isinstance(response, dict) else response[0]
-        nb = Notebook(
-            name=nb_data["name"],
-            description=nb_data["description"],
-            archived=nb_data["archived"],
-        )
-        nb.id = nb_data["id"]
-        nb.created = nb_data["created"]
-        nb.updated = nb_data["updated"]
-        return nb
-
-    def create_notebook(self, name: str, description: str = "") -> Notebook:
-        """Create a new notebook."""
-        response = api_client.create_notebook(name, description)
-        nb_data = response if isinstance(response, dict) else response[0]
-        nb = Notebook(
-            name=nb_data["name"],
-            description=nb_data["description"],
-            archived=nb_data["archived"],
-        )
-        nb.id = nb_data["id"]
-        nb.created = nb_data["created"]
-        nb.updated = nb_data["updated"]
-        return nb
-
-    def update_notebook(self, notebook: Notebook) -> Notebook:
-        """Update a notebook."""
-        updates = {
-            "name": notebook.name,
-            "description": notebook.description,
-            "archived": notebook.archived,
-        }
-        response = api_client.update_notebook(notebook.id or "", **updates)
-        nb_data = response if isinstance(response, dict) else response[0]
-        # Update the notebook object with the response
-        notebook.name = nb_data["name"]
-        notebook.description = nb_data["description"]
-        notebook.archived = nb_data["archived"]
-        notebook.updated = nb_data["updated"]
-        return notebook
-
-    def delete_notebook(self, notebook: Notebook) -> bool:
-        """Delete a notebook."""
-        api_client.delete_notebook(notebook.id or "")
-        return True
-
-
-# Global service instance
-notebook_service = NotebookService()

api/notes_service.py

@@ -1,103 +0,0 @@
-"""
-Notes service layer using API.
-"""
-
-from typing import List, Optional
-
-from loguru import logger
-
-from api.client import api_client
-from open_notebook.domain.notebook import Note
-
-
-class NotesService:
-    """Service layer for notes operations using API."""
-
-    def __init__(self):
-        logger.info("Using API for notes operations")
-
-    def get_all_notes(self, notebook_id: Optional[str] = None) -> List[Note]:
-        """Get all notes with optional notebook filtering."""
-        notes_data = api_client.get_notes(notebook_id=notebook_id)
-        # Convert API response to Note objects
-        notes = []
-        for note_data in notes_data:
-            note = Note(
-                title=note_data["title"],
-                content=note_data["content"],
-                note_type=note_data["note_type"],
-            )
-            note.id = note_data["id"]
-            note.created = note_data["created"]
-            note.updated = note_data["updated"]
-            notes.append(note)
-        return notes
-
-    def get_note(self, note_id: str) -> Note:
-        """Get a specific note."""
-        note_response = api_client.get_note(note_id)
-        note_data = (
-            note_response if isinstance(note_response, dict) else note_response[0]
-        )
-        note = Note(
-            title=note_data["title"],
-            content=note_data["content"],
-            note_type=note_data["note_type"],
-        )
-        note.id = note_data["id"]
-        note.created = note_data["created"]
-        note.updated = note_data["updated"]
-        return note
-
-    def create_note(
-        self,
-        content: str,
-        title: Optional[str] = None,
-        note_type: str = "human",
-        notebook_id: Optional[str] = None,
-    ) -> Note:
-        """Create a new note."""
-        note_response = api_client.create_note(
-            content=content, title=title, note_type=note_type, notebook_id=notebook_id
-        )
-        note_data = (
-            note_response if isinstance(note_response, dict) else note_response[0]
-        )
-        note = Note(
-            title=note_data["title"],
-            content=note_data["content"],
-            note_type=note_data["note_type"],
-        )
-        note.id = note_data["id"]
-        note.created = note_data["created"]
-        note.updated = note_data["updated"]
-        return note
-
-    def update_note(self, note: Note) -> Note:
-        """Update a note."""
-        updates = {
-            "title": note.title,
-            "content": note.content,
-            "note_type": note.note_type,
-        }
-        note_response = api_client.update_note(note.id or "", **updates)
-        note_data = (
-            note_response if isinstance(note_response, dict) else note_response[0]
-        )
-
-        # Update the note object with the response
-        note.title = note_data["title"]
-        note.content = note_data["content"]
-        note.note_type = note_data["note_type"]
-        note.updated = note_data["updated"]
-
-        return note
-
-    def delete_note(self, note_id: str) -> bool:
-        """Delete a note."""
-        api_client.delete_note(note_id)
-        return True
-
-
-# Global service instance
-notes_service = NotesService()

api/podcast_api_service.py

@@ -1,125 +0,0 @@
-"""
-Podcast service layer using API client.
-This replaces direct httpx calls in the Streamlit pages.
-"""
-
-from typing import Any, Dict, List
-
-from loguru import logger
-
-from api.client import api_client
-
-
-class PodcastAPIService:
-    """Service layer for podcast operations using API client."""
-
-    def __init__(self):
-        logger.info("Using API client for podcast operations")
-
-    # Episode methods
-    def get_episodes(self) -> List[Dict[Any, Any]]:
-        """Get all podcast episodes."""
-        result = api_client._make_request("GET", "/api/podcasts/episodes")
-        return result if isinstance(result, list) else [result]
-
-    def delete_episode(self, episode_id: str) -> bool:
-        """Delete a podcast episode."""
-        try:
-            api_client._make_request("DELETE", f"/api/podcasts/episodes/{episode_id}")
-            return True
-        except Exception as e:
-            logger.error(f"Failed to delete episode: {e}")
-            return False
-
-    # Episode Profile methods
-    def get_episode_profiles(self) -> List[Dict]:
-        """Get all episode profiles."""
-        return api_client.get_episode_profiles()
-
-    def create_episode_profile(self, profile_data: Dict) -> bool:
-        """Create a new episode profile."""
-        try:
-            api_client.create_episode_profile(**profile_data)
-            return True
-        except Exception as e:
-            logger.error(f"Failed to create episode profile: {e}")
-            return False
-
-    def update_episode_profile(self, profile_id: str, profile_data: Dict) -> bool:
-        """Update an episode profile."""
-        try:
-            api_client.update_episode_profile(profile_id, **profile_data)
-            return True
-        except Exception as e:
-            logger.error(f"Failed to update episode profile: {e}")
-            return False
-
-    def delete_episode_profile(self, profile_id: str) -> bool:
-        """Delete an episode profile."""
-        try:
-            api_client.delete_episode_profile(profile_id)
-            return True
-        except Exception as e:
-            logger.error(f"Failed to delete episode profile: {e}")
-            return False
-
-    def duplicate_episode_profile(self, profile_id: str) -> bool:
-        """Duplicate an episode profile."""
-        try:
-            api_client._make_request(
-                "POST", f"/api/episode-profiles/{profile_id}/duplicate"
-            )
-            return True
-        except Exception as e:
-            logger.error(f"Failed to duplicate episode profile: {e}")
-            return False
-
-    # Speaker Profile methods
-    def get_speaker_profiles(self) -> List[Dict[Any, Any]]:
-        """Get all speaker profiles."""
-        result = api_client._make_request("GET", "/api/speaker-profiles")
-        return result if isinstance(result, list) else [result]
-
-    def create_speaker_profile(self, profile_data: Dict) -> bool:
-        """Create a new speaker profile."""
-        try:
-            api_client._make_request("POST", "/api/speaker-profiles", json=profile_data)
-            return True
-        except Exception as e:
-            logger.error(f"Failed to create speaker profile: {e}")
-            return False
-
-    def update_speaker_profile(self, profile_id: str, profile_data: Dict) -> bool:
-        """Update a speaker profile."""
-        try:
-            api_client._make_request(
-                "PUT", f"/api/speaker-profiles/{profile_id}", json=profile_data
-            )
-            return True
-        except Exception as e:
-            logger.error(f"Failed to update speaker profile: {e}")
-            return False
-
-    def delete_speaker_profile(self, profile_id: str) -> bool:
-        """Delete a speaker profile."""
-        try:
-            api_client._make_request("DELETE", f"/api/speaker-profiles/{profile_id}")
-            return True
-        except Exception as e:
-            logger.error(f"Failed to delete speaker profile: {e}")
-            return False
-
-    def duplicate_speaker_profile(self, profile_id: str) -> bool:
-        """Duplicate a speaker profile."""
-        try:
-            api_client._make_request(
-                "POST", f"/api/speaker-profiles/{profile_id}/duplicate"
-            )
-            return True
-        except Exception as e:
-            logger.error(f"Failed to duplicate speaker profile: {e}")
-            return False
-
-
-# Global service instance
-podcast_api_service = PodcastAPIService()

api/podcast_service.py

@@ -49,8 +49,9 @@ class PodcastService:
             if not episode_profile:
                 raise ValueError(f"Episode profile '{episode_profile_name}' not found")
 
-            # Validate speaker profile exists
-            speaker_profile = await SpeakerProfile.get_by_name(speaker_profile_name)
+            # Resolve the user-facing speaker profile name to a record ID at
+            # the API boundary (#630) - everything downstream works with IDs.
+            speaker_profile = await SpeakerProfile.resolve(speaker_profile_name)
             if not speaker_profile:
                 raise ValueError(f"Speaker profile '{speaker_profile_name}' not found")
 
@@ -75,10 +76,10 @@ class PodcastService:
                     "Content is required - provide either content or notebook_id"
                 )
 
-            # Prepare command arguments
+            # Prepare command arguments (speaker profile as record ID)
             command_args = {
                 "episode_profile": episode_profile_name,
-                "speaker_profile": speaker_profile_name,
+                "speaker_profile": str(speaker_profile.id),
                 "episode_name": episode_name,
                 "content": str(content),
                 "briefing_suffix": briefing_suffix,
@@ -108,7 +109,7 @@ class PodcastService:
             logger.error(f"Failed to submit podcast generation job: {e}")
             raise HTTPException(
                 status_code=500,
-                detail=f"Failed to submit podcast generation job: {str(e)}",
+                detail="Failed to submit podcast generation job",
             )
 
     @staticmethod
@@ -133,9 +134,7 @@ class PodcastService:
             }
         except Exception as e:
             logger.error(f"Failed to get podcast job status: {e}")
-            raise HTTPException(
-                status_code=500, detail=f"Failed to get job status: {str(e)}"
-            )
+            raise HTTPException(status_code=500, detail="Failed to get job status")
 
     @staticmethod
     async def list_episodes() -> list:
@@ -145,9 +144,7 @@ class PodcastService:
             return episodes
         except Exception as e:
             logger.error(f"Failed to list podcast episodes: {e}")
-            raise HTTPException(
-                status_code=500, detail=f"Failed to list episodes: {str(e)}"
-            )
+            raise HTTPException(status_code=500, detail="Failed to list episodes")
 
     @staticmethod
     async def get_episode(episode_id: str) -> PodcastEpisode:
@@ -157,7 +154,7 @@ class PodcastService:
             return episode
         except Exception as e:
             logger.error(f"Failed to get podcast episode {episode_id}: {e}")
-            raise HTTPException(status_code=404, detail=f"Episode not found: {str(e)}")
+            raise HTTPException(status_code=404, detail="Episode not found")
 
 
 class DefaultProfiles:

api/routers/_chat_shared.py

@@ -0,0 +1,94 @@
+"""Shared helpers for the chat and source-chat routers.
+
+Both `api/routers/chat.py` and `api/routers/source_chat.py` operate on
+`chat_session` records linked to their parent (notebook or source) via the
+`refers_to` relation, and both convert LangGraph state messages into API
+response models. This module holds the single definition of those pieces.
+
+Behavior notes:
+- The helpers raise exactly what the previously inlined blocks raised
+  (`NotFoundError` propagates from `ObjectModel.get`, `HTTPException(404)` for
+  a missing relation), so each router's existing try/except arms keep mapping
+  them to the same status codes and messages as before.
+"""
+
+from typing import Any, Iterable, List, Optional, Tuple
+
+from fastapi import HTTPException
+from pydantic import BaseModel, Field
+
+from open_notebook.database.repository import ensure_record_id, repo_query
+from open_notebook.domain.notebook import ChatSession, Source
+
+
+# Shared response models
+class ChatMessage(BaseModel):
+    id: str = Field(..., description="Message ID")
+    type: str = Field(..., description="Message type (human|ai)")
+    content: str = Field(..., description="Message content")
+    timestamp: Optional[str] = Field(None, description="Message timestamp")
+
+
+class SuccessResponse(BaseModel):
+    success: bool = Field(True, description="Operation success status")
+    message: str = Field(..., description="Success message")
+
+
+def normalize_record_id(table: str, record_id: str) -> str:
+    """Ensure a record ID carries its table prefix (`table:id`)."""
+    prefix = f"{table}:"
+    return record_id if record_id.startswith(prefix) else f"{prefix}{record_id}"
+
+
+async def get_source_or_404(source_id: str) -> Tuple[str, Source]:
+    """Normalize a source ID and fetch the source, 404 if missing."""
+    full_source_id = normalize_record_id("source", source_id)
+    source = await Source.get(full_source_id)
+    if not source:
+        raise HTTPException(status_code=404, detail="Source not found")
+    return full_source_id, source
+
+
+async def get_session_or_404(session_id: str) -> Tuple[str, ChatSession]:
+    """Normalize a session ID and fetch the chat session, 404 if missing."""
+    full_session_id = normalize_record_id("chat_session", session_id)
+    session = await ChatSession.get(full_session_id)
+    if not session:
+        raise HTTPException(status_code=404, detail="Session not found")
+    return full_session_id, session
+
+
+async def get_verified_source_session(
+    source_id: str, session_id: str
+) -> Tuple[str, Source, str, ChatSession]:
+    """Verify the source exists, the session exists, and the session refers to
+    the source. Returns the normalized IDs plus both records."""
+    full_source_id, source = await get_source_or_404(source_id)
+    full_session_id, session = await get_session_or_404(session_id)
+
+    relation_query = await repo_query(
+        "SELECT * FROM refers_to WHERE in = $session_id AND out = $source_id",
+        {
+            "session_id": ensure_record_id(full_session_id),
+            "source_id": ensure_record_id(full_source_id),
+        },
+    )
+    if not relation_query:
+        raise HTTPException(status_code=404, detail="Session not found for this source")
+
+    return full_source_id, source, full_session_id, session
+
+
+def extract_chat_messages(raw_messages: Iterable[Any]) -> List[ChatMessage]:
+    """Convert LangGraph/LangChain state messages into `ChatMessage` models."""
+    messages: List[ChatMessage] = []
+    for msg in raw_messages:
+        messages.append(
+            ChatMessage(
+                id=getattr(msg, "id", f"msg_{len(messages)}"),
+                type=msg.type if hasattr(msg, "type") else "unknown",
+                content=msg.content if hasattr(msg, "content") else str(msg),
+                timestamp=None,  # LangChain messages don't have timestamps by default
+            )
+        )
+    return messages

api/routers/auth.py

@@ -24,4 +24,4 @@ async def get_auth_status():
         "message": "Authentication is required"
         if auth_enabled
         else "Authentication is disabled",
-    }
+    }

api/routers/capabilities.py

@@ -0,0 +1,40 @@
+"""
+Capabilities Router
+
+Reports the runtime availability of the opt-in heavy extraction engines
+(Docling, Crawl4AI local) so the frontend can gate the corresponding engine
+options and the OCR toggle. These runtimes are installed on demand at container
+startup (see scripts/docker-entrypoint.sh and
+docs/7-DEVELOPMENT/decisions/ADR-007-optin-runtimes.md), so this endpoint probes
+what is *actually* importable/reachable rather than trusting the enable flags.
+
+The probes themselves live in open_notebook.utils.runtime_capabilities because
+the source-processing graph needs the same signal (it must not pass content-core
+an engine whose runtime is absent).
+
+Endpoints:
+- GET /capabilities - Availability of Docling and Crawl4AI runtimes
+"""
+
+from fastapi import APIRouter
+
+from api.models import CapabilitiesResponse
+from open_notebook.utils.runtime_capabilities import (
+    crawl4ai_local_ready,
+    crawl4ai_remote_configured,
+    docling_available,
+)
+
+router = APIRouter(prefix="/capabilities", tags=["capabilities"])
+
+
+@router.get("", response_model=CapabilitiesResponse)
+async def get_capabilities():
+    """Report which opt-in extraction runtimes are available in this container."""
+    crawl4ai_remote = crawl4ai_remote_configured()
+    crawl4ai_local = crawl4ai_local_ready()
+    return CapabilitiesResponse(
+        docling_available=docling_available(),
+        crawl4ai_available=crawl4ai_local or crawl4ai_remote,
+        crawl4ai_remote_configured=crawl4ai_remote,
+    )

api/routers/chat.py

@@ -1,19 +1,32 @@
 import asyncio
 import traceback
 from typing import Any, Dict, List, Optional
+from uuid import uuid4
 
 from fastapi import APIRouter, HTTPException, Query
 from langchain_core.runnables import RunnableConfig
 from loguru import logger
 from pydantic import BaseModel, Field
 
+from api.routers._chat_shared import (
+    ChatMessage,
+    SuccessResponse,
+    extract_chat_messages,
+    get_session_or_404,
+)
 from open_notebook.database.repository import ensure_record_id, repo_query
-from open_notebook.domain.notebook import ChatSession, Note, Notebook, Source
+from open_notebook.domain.notebook import ChatSession, Notebook
 from open_notebook.exceptions import (
     NotFoundError,
+    OpenNotebookError,
 )
 from open_notebook.graphs.chat import graph as chat_graph
-from open_notebook.utils.graph_utils import get_session_message_count
+from open_notebook.utils import token_count
+from open_notebook.utils.context_builder import build_notebook_context
+from open_notebook.utils.graph_utils import (
+    get_session_message_count,
+    invoke_chat_turn,
+)
 
 router = APIRouter()
 
@@ -34,13 +47,6 @@ class UpdateSessionRequest(BaseModel):
     )
 
 
-class ChatMessage(BaseModel):
-    id: str = Field(..., description="Message ID")
-    type: str = Field(..., description="Message type (human|ai)")
-    content: str = Field(..., description="Message content")
-    timestamp: Optional[str] = Field(None, description="Message timestamp")
-
-
 class ChatSessionResponse(BaseModel):
     id: str = Field(..., description="Session ID")
     title: str = Field(..., description="Session title")
@@ -88,11 +94,6 @@ class BuildContextResponse(BaseModel):
     char_count: int = Field(..., description="Character count")
 
 
-class SuccessResponse(BaseModel):
-    success: bool = Field(True, description="Operation success status")
-    message: str = Field(..., description="Success message")
-
-
 @router.get("/chat/sessions", response_model=List[ChatSessionResponse])
 async def get_sessions(notebook_id: str = Query(..., description="Notebook ID")):
     """Get all chat sessions for a notebook."""
@@ -127,6 +128,10 @@ async def get_sessions(notebook_id: str = Query(..., description="Notebook ID"))
         return results
     except NotFoundError:
         raise HTTPException(status_code=404, detail="Notebook not found")
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error fetching chat sessions: {str(e)}")
         raise HTTPException(
@@ -165,6 +170,10 @@ async def create_session(request: CreateSessionRequest):
         )
     except NotFoundError:
         raise HTTPException(status_code=404, detail="Notebook not found")
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error creating chat session: {str(e)}")
         raise HTTPException(
@@ -178,16 +187,8 @@ async def create_session(request: CreateSessionRequest):
 async def get_session(session_id: str):
     """Get a specific session with its messages."""
     try:
-        # Get session
-        # Ensure session_id has proper table prefix
-        full_session_id = (
-            session_id
-            if session_id.startswith("chat_session:")
-            else f"chat_session:{session_id}"
-        )
-        session = await ChatSession.get(full_session_id)
-        if not session:
-            raise HTTPException(status_code=404, detail="Session not found")
+        # Get session (normalizes the ID and 404s if missing)
+        full_session_id, session = await get_session_or_404(session_id)
 
         # Get session state from LangGraph to retrieve messages
         # Use sync get_state() in a thread since SqliteSaver doesn't support async
@@ -199,24 +200,9 @@ async def get_session(session_id: str):
         # Extract messages from state
         messages: list[ChatMessage] = []
         if thread_state and thread_state.values and "messages" in thread_state.values:
-            for msg in thread_state.values["messages"]:
-                messages.append(
-                    ChatMessage(
-                        id=getattr(msg, "id", f"msg_{len(messages)}"),
-                        type=msg.type if hasattr(msg, "type") else "unknown",
-                        content=msg.content if hasattr(msg, "content") else str(msg),
-                        timestamp=None,  # LangChain messages don't have timestamps by default
-                    )
-                )
+            messages = extract_chat_messages(thread_state.values["messages"])
 
         # Find notebook_id (we need to query the relationship)
-        # Ensure session_id has proper table prefix
-        full_session_id = (
-            session_id
-            if session_id.startswith("chat_session:")
-            else f"chat_session:{session_id}"
-        )
-
         notebook_query = await repo_query(
             "SELECT out FROM refers_to WHERE in = $session_id",
             {"session_id": ensure_record_id(full_session_id)},
@@ -242,6 +228,10 @@ async def get_session(session_id: str):
         )
     except NotFoundError:
         raise HTTPException(status_code=404, detail="Session not found")
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error fetching session: {str(e)}")
         raise HTTPException(status_code=500, detail=f"Error fetching session: {str(e)}")
@@ -251,15 +241,8 @@ async def get_session(session_id: str):
 async def update_session(session_id: str, request: UpdateSessionRequest):
     """Update session title."""
     try:
-        # Ensure session_id has proper table prefix
-        full_session_id = (
-            session_id
-            if session_id.startswith("chat_session:")
-            else f"chat_session:{session_id}"
-        )
-        session = await ChatSession.get(full_session_id)
-        if not session:
-            raise HTTPException(status_code=404, detail="Session not found")
+        # Get session (normalizes the ID and 404s if missing)
+        full_session_id, session = await get_session_or_404(session_id)
 
         update_data = request.model_dump(exclude_unset=True)
 
@@ -272,12 +255,6 @@ async def update_session(session_id: str, request: UpdateSessionRequest):
         await session.save()
 
         # Find notebook_id
-        # Ensure session_id has proper table prefix
-        full_session_id = (
-            session_id
-            if session_id.startswith("chat_session:")
-            else f"chat_session:{session_id}"
-        )
         notebook_query = await repo_query(
             "SELECT out FROM refers_to WHERE in = $session_id",
             {"session_id": ensure_record_id(full_session_id)},
@@ -298,6 +275,10 @@ async def update_session(session_id: str, request: UpdateSessionRequest):
         )
     except NotFoundError:
         raise HTTPException(status_code=404, detail="Session not found")
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error updating session: {str(e)}")
         raise HTTPException(status_code=500, detail=f"Error updating session: {str(e)}")
@@ -307,21 +288,18 @@ async def update_session(session_id: str, request: UpdateSessionRequest):
 async def delete_session(session_id: str):
     """Delete a chat session."""
     try:
-        # Ensure session_id has proper table prefix
-        full_session_id = (
-            session_id
-            if session_id.startswith("chat_session:")
-            else f"chat_session:{session_id}"
-        )
-        session = await ChatSession.get(full_session_id)
-        if not session:
-            raise HTTPException(status_code=404, detail="Session not found")
+        # Get session (normalizes the ID and 404s if missing)
+        _full_session_id, session = await get_session_or_404(session_id)
 
         await session.delete()
 
         return SuccessResponse(success=True, message="Session deleted successfully")
     except NotFoundError:
         raise HTTPException(status_code=404, detail="Session not found")
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error deleting session: {str(e)}")
         raise HTTPException(status_code=500, detail=f"Error deleting session: {str(e)}")
@@ -331,16 +309,17 @@ async def delete_session(session_id: str):
 async def execute_chat(request: ExecuteChatRequest):
     """Execute a chat request and get AI response."""
     try:
-        # Verify session exists
-        # Ensure session_id has proper table prefix
-        full_session_id = (
-            request.session_id
-            if request.session_id.startswith("chat_session:")
-            else f"chat_session:{request.session_id}"
+        # Verify session exists (normalizes the ID and 404s if missing)
+        full_session_id, session = await get_session_or_404(request.session_id)
+
+        # Fetch notebook linked to this session
+        notebook_query = await repo_query(
+            "SELECT out FROM refers_to WHERE in = $session_id",
+            {"session_id": ensure_record_id(full_session_id)},
         )
-        session = await ChatSession.get(full_session_id)
-        if not session:
-            raise HTTPException(status_code=404, detail="Session not found")
+        notebook = None
+        if notebook_query:
+            notebook = await Notebook.get(notebook_query[0]["out"])
 
         # Determine model override (per-request override takes precedence over session-level)
         model_override = (
@@ -360,43 +339,48 @@ async def execute_chat(request: ExecuteChatRequest):
         state_values = current_state.values if current_state else {}
         state_values["messages"] = state_values.get("messages", [])
         state_values["context"] = request.context
+        state_values["notebook"] = notebook
         state_values["model_override"] = model_override
 
         # Add user message to state
         from langchain_core.messages import HumanMessage
 
-        user_message = HumanMessage(content=request.message)
+        # Explicit id so a failed turn can remove it from the checkpoint.
+        user_message = HumanMessage(content=request.message, id=str(uuid4()))
         state_values["messages"].append(user_message)
 
-        # Execute chat graph
-        result = chat_graph.invoke(
-            input=state_values,  # type: ignore[arg-type]
-            config=RunnableConfig(
+        # Execute chat graph in a thread so the synchronous LangGraph invoke
+        # (SqliteSaver checkpoints are sync) doesn't block the event loop and
+        # freeze the rest of the API while the LLM responds. Mirrors the
+        # get_state() calls above.
+        # invoke_chat_turn also drops the question from the checkpoint when the
+        # turn fails, so a retry doesn't add it twice.
+        result = await asyncio.to_thread(
+            invoke_chat_turn,
+            chat_graph,
+            state_values,
+            RunnableConfig(
                 configurable={
                     "thread_id": full_session_id,
                     "model_id": model_override,
                 }
             ),
+            user_message,
         )
 
         # Update session timestamp
         await session.save()
 
         # Convert messages to response format
-        messages: list[ChatMessage] = []
-        for msg in result.get("messages", []):
-            messages.append(
-                ChatMessage(
-                    id=getattr(msg, "id", f"msg_{len(messages)}"),
-                    type=msg.type if hasattr(msg, "type") else "unknown",
-                    content=msg.content if hasattr(msg, "content") else str(msg),
-                    timestamp=None,
-                )
-            )
+        messages = extract_chat_messages(result.get("messages", []))
 
         return ExecuteChatResponse(session_id=request.session_id, messages=messages)
     except NotFoundError:
         raise HTTPException(status_code=404, detail="Session not found")
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         # Log detailed error with context for debugging
         logger.error(
@@ -417,100 +401,20 @@ async def build_context(request: BuildContextRequest):
         if not notebook:
             raise HTTPException(status_code=404, detail="Notebook not found")
 
-        context_data: dict[str, list[dict[str, str]]] = {"sources": [], "notes": []}
-        total_content = ""
+        context_data, total_content = await build_notebook_context(
+            notebook, request.context_config
+        )
 
-        # Process context configuration if provided
-        if request.context_config:
-            # Process sources
-            for source_id, status in request.context_config.get("sources", {}).items():
-                if "not in" in status:
-                    continue
-
-                try:
-                    # Add table prefix if not present
-                    full_source_id = (
-                        source_id
-                        if source_id.startswith("source:")
-                        else f"source:{source_id}"
-                    )
-
-                    try:
-                        source = await Source.get(full_source_id)
-                    except Exception:
-                        continue
-
-                    if "insights" in status:
-                        source_context = await source.get_context(context_size="short")
-                        context_data["sources"].append(source_context)
-                        total_content += str(source_context)
-                    elif "full content" in status:
-                        source_context = await source.get_context(context_size="long")
-                        context_data["sources"].append(source_context)
-                        total_content += str(source_context)
-                except Exception as e:
-                    logger.warning(f"Error processing source {source_id}: {str(e)}")
-                    continue
-
-            # Process notes
-            for note_id, status in request.context_config.get("notes", {}).items():
-                if "not in" in status:
-                    continue
-
-                try:
-                    # Add table prefix if not present
-                    full_note_id = (
-                        note_id if note_id.startswith("note:") else f"note:{note_id}"
-                    )
-                    note = await Note.get(full_note_id)
-                    if not note:
-                        continue
-
-                    if "full content" in status:
-                        note_context = note.get_context(context_size="long")
-                        context_data["notes"].append(note_context)
-                        total_content += str(note_context)
-                except Exception as e:
-                    logger.warning(f"Error processing note {note_id}: {str(e)}")
-                    continue
-        else:
-            # Default behavior - include all sources and notes with short context
-            sources = await notebook.get_sources()
-            for source in sources:
-                try:
-                    source_context = await source.get_context(context_size="short")
-                    context_data["sources"].append(source_context)
-                    total_content += str(source_context)
-                except Exception as e:
-                    logger.warning(f"Error processing source {source.id}: {str(e)}")
-                    continue
-
-            notes = await notebook.get_notes()
-            for note in notes:
-                try:
-                    note_context = note.get_context(context_size="short")
-                    context_data["notes"].append(note_context)
-                    total_content += str(note_context)
-                except Exception as e:
-                    logger.warning(f"Error processing note {note.id}: {str(e)}")
-                    continue
-
-        # Calculate character and token counts
         char_count = len(total_content)
-        # Use token count utility if available
-        try:
-            from open_notebook.utils import token_count
-
-            estimated_tokens = token_count(total_content) if total_content else 0
-        except ImportError:
-            # Fallback to simple estimation
-            estimated_tokens = char_count // 4
+        estimated_tokens = token_count(total_content) if total_content else 0
 
         return BuildContextResponse(
             context=context_data, token_count=estimated_tokens, char_count=char_count
         )
     except HTTPException:
         raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error building context: {str(e)}")
         raise HTTPException(status_code=500, detail=f"Error building context: {str(e)}")

api/routers/commands.py

@@ -6,13 +6,14 @@ from pydantic import BaseModel, Field
 from surreal_commands import registry
 
 from api.command_service import CommandService
+from open_notebook.exceptions import OpenNotebookError
 
 router = APIRouter()
 
 
 class CommandExecutionRequest(BaseModel):
     command: str = Field(
-        ..., description="Command function name (e.g., 'process_text')"
+        ..., description="Command function name (e.g., 'generate_podcast')"
     )
     app: str = Field(..., description="Application name (e.g., 'open_notebook')")
     input: Dict[str, Any] = Field(..., description="Arguments to pass to the command")
@@ -42,11 +43,13 @@ async def execute_command(request: CommandExecutionRequest):
 
     Example request:
     {
-        "command": "process_text",
+        "command": "generate_podcast",
         "app": "open_notebook",
         "input": {
-            "text": "Hello world",
-            "operation": "uppercase"
+            "episode_profile": "tech_experts",
+            "speaker_profile": "tech_experts",
+            "episode_name": "My Episode",
+            "content": "Content to discuss"
         }
     }
     """
@@ -64,11 +67,13 @@ async def execute_command(request: CommandExecutionRequest):
             message=f"Command '{request.command}' submitted successfully",
         )
 
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error submitting command: {str(e)}")
-        raise HTTPException(
-            status_code=500, detail="Failed to submit command"
-        )
+        raise HTTPException(status_code=500, detail="Failed to submit command")
 
 
 @router.get("/commands/jobs/{job_id}", response_model=CommandJobStatusResponse)
@@ -78,11 +83,13 @@ async def get_command_job_status(job_id: str):
         status_data = await CommandService.get_command_status(job_id)
         return CommandJobStatusResponse(**status_data)
 
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error fetching job status: {str(e)}")
-        raise HTTPException(
-            status_code=500, detail="Failed to fetch job status"
-        )
+        raise HTTPException(status_code=500, detail="Failed to fetch job status")
 
 
 @router.get("/commands/jobs", response_model=List[Dict[str, Any]])
@@ -98,11 +105,13 @@ async def list_command_jobs(
         )
         return jobs
 
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error listing command jobs: {str(e)}")
-        raise HTTPException(
-            status_code=500, detail="Failed to list command jobs"
-        )
+        raise HTTPException(status_code=500, detail="Failed to list command jobs")
 
 
 @router.delete("/commands/jobs/{job_id}")
@@ -112,11 +121,13 @@ async def cancel_command_job(job_id: str):
         success = await CommandService.cancel_command_job(job_id)
         return {"job_id": job_id, "cancelled": success}
 
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error cancelling command job: {str(e)}")
-        raise HTTPException(
-            status_code=500, detail="Failed to cancel command job"
-        )
+        raise HTTPException(status_code=500, detail="Failed to cancel command job")
 
 
 @router.get("/commands/registry/debug")

api/routers/config.py

@@ -1,5 +1,4 @@
 import asyncio
-import os
 import time
 import tomllib
 from pathlib import Path

api/routers/context.py

@@ -1,115 +0,0 @@
-from fastapi import APIRouter, HTTPException
-from loguru import logger
-
-from api.models import ContextRequest, ContextResponse
-from open_notebook.domain.notebook import Note, Notebook, Source
-from open_notebook.exceptions import InvalidInputError
-from open_notebook.utils import token_count
-
-router = APIRouter()
-
-
-@router.post("/notebooks/{notebook_id}/context", response_model=ContextResponse)
-async def get_notebook_context(notebook_id: str, context_request: ContextRequest):
-    """Get context for a notebook based on configuration."""
-    try:
-        # Verify notebook exists
-        notebook = await Notebook.get(notebook_id)
-        if not notebook:
-            raise HTTPException(status_code=404, detail="Notebook not found")
-
-        context_data: dict[str, list[dict[str, str]]] = {"note": [], "source": []}
-        total_content = ""
-
-        # Process context configuration if provided
-        if context_request.context_config:
-            # Process sources
-            for source_id, status in context_request.context_config.sources.items():
-                if "not in" in status:
-                    continue
-
-                try:
-                    # Add table prefix if not present
-                    full_source_id = (
-                        source_id
-                        if source_id.startswith("source:")
-                        else f"source:{source_id}"
-                    )
-
-                    try:
-                        source = await Source.get(full_source_id)
-                    except Exception:
-                        continue
-
-                    if "insights" in status:
-                        source_context = await source.get_context(context_size="short")
-                        context_data["source"].append(source_context)
-                        total_content += str(source_context)
-                    elif "full content" in status:
-                        source_context = await source.get_context(context_size="long")
-                        context_data["source"].append(source_context)
-                        total_content += str(source_context)
-                except Exception as e:
-                    logger.warning(f"Error processing source {source_id}: {str(e)}")
-                    continue
-
-            # Process notes
-            for note_id, status in context_request.context_config.notes.items():
-                if "not in" in status:
-                    continue
-
-                try:
-                    # Add table prefix if not present
-                    full_note_id = (
-                        note_id if note_id.startswith("note:") else f"note:{note_id}"
-                    )
-                    note = await Note.get(full_note_id)
-                    if not note:
-                        continue
-
-                    if "full content" in status:
-                        note_context = note.get_context(context_size="long")
-                        context_data["note"].append(note_context)
-                        total_content += str(note_context)
-                except Exception as e:
-                    logger.warning(f"Error processing note {note_id}: {str(e)}")
-                    continue
-        else:
-            # Default behavior - include all sources and notes with short context
-            sources = await notebook.get_sources()
-            for source in sources:
-                try:
-                    source_context = await source.get_context(context_size="short")
-                    context_data["source"].append(source_context)
-                    total_content += str(source_context)
-                except Exception as e:
-                    logger.warning(f"Error processing source {source.id}: {str(e)}")
-                    continue
-
-            notes = await notebook.get_notes()
-            for note in notes:
-                try:
-                    note_context = note.get_context(context_size="short")
-                    context_data["note"].append(note_context)
-                    total_content += str(note_context)
-                except Exception as e:
-                    logger.warning(f"Error processing note {note.id}: {str(e)}")
-                    continue
-
-        # Calculate estimated token count
-        estimated_tokens = token_count(total_content) if total_content else 0
-
-        return ContextResponse(
-            notebook_id=notebook_id,
-            sources=context_data["source"],
-            notes=context_data["note"],
-            total_tokens=estimated_tokens,
-        )
-
-    except HTTPException:
-        raise
-    except InvalidInputError as e:
-        raise HTTPException(status_code=400, detail=str(e))
-    except Exception as e:
-        logger.error(f"Error getting context for notebook {notebook_id}: {str(e)}")
-        raise HTTPException(status_code=500, detail=f"Error getting context: {str(e)}")

api/routers/credentials.py

@@ -27,16 +27,26 @@ from pydantic import SecretStr
 from api.credentials_service import (
     credential_to_response,
     discover_with_config,
-    migrate_from_env as svc_migrate_from_env,
-    migrate_from_provider_config as svc_migrate_from_provider_config,
+    ensure_provider_required_fields,
+    get_provider_status,
     register_models,
     require_encryption_key,
-    test_credential as svc_test_credential,
     validate_url,
 )
 from api.credentials_service import (
     get_env_status as svc_get_env_status,
-    get_provider_status,
+)
+from api.credentials_service import (
+    migrate_encryption_scheme as svc_migrate_encryption_scheme,
+)
+from api.credentials_service import (
+    migrate_from_env as svc_migrate_from_env,
+)
+from api.credentials_service import (
+    migrate_from_provider_config as svc_migrate_from_provider_config,
+)
+from api.credentials_service import (
+    test_credential as svc_test_credential,
 )
 from api.models import (
     CreateCredentialRequest,
@@ -48,14 +58,20 @@ from api.models import (
     RegisterModelsResponse,
     UpdateCredentialRequest,
 )
+from open_notebook.database.repository import ensure_record_id, repo_delete, repo_query
 from open_notebook.domain.credential import Credential
+from open_notebook.exceptions import (
+    NotFoundError,
+    OpenNotebookError,
+)
 
 router = APIRouter(prefix="/credentials", tags=["credentials"])
 
 
 def _handle_value_error(e: ValueError, status_code: int = 400) -> HTTPException:
     """Convert a ValueError from the service layer to an HTTPException."""
-    return HTTPException(status_code=status_code, detail=str(e))
+    # Truncate so upstream validation messages can't leak unbounded internals.
+    return HTTPException(status_code=status_code, detail=str(e)[:200])
 
 
 # =============================================================================
@@ -71,6 +87,10 @@ async def get_status():
     """
     try:
         return await get_provider_status()
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error fetching status: {e}")
         raise HTTPException(status_code=500, detail="Failed to fetch credential status")
@@ -81,9 +101,15 @@ async def get_env_status():
     """Check what's configured via environment variables."""
     try:
         return await svc_get_env_status()
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error checking env status: {e}")
-        raise HTTPException(status_code=500, detail="Failed to check environment status")
+        raise HTTPException(
+            status_code=500, detail="Failed to check environment status"
+        )
 
 
 # =============================================================================
@@ -109,6 +135,10 @@ async def list_credentials(
 
         return result
 
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error listing credentials: {e}")
         raise HTTPException(status_code=500, detail="Failed to list credentials")
@@ -124,9 +154,15 @@ async def list_credentials_by_provider(provider: str):
             models = await cred.get_linked_models()
             result.append(credential_to_response(cred, len(models)))
         return result
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error listing credentials for {provider}: {e}")
-        raise HTTPException(status_code=500, detail="Failed to list credentials for provider")
+        raise HTTPException(
+            status_code=500, detail="Failed to list credentials for provider"
+        )
 
 
 @router.post("", response_model=CredentialResponse, status_code=201)
@@ -139,12 +175,16 @@ async def create_credential(request: CreateCredentialRequest):
 
     # Validate all URL fields
     for url_field in [
-        request.base_url, request.endpoint, request.endpoint_llm,
-        request.endpoint_embedding, request.endpoint_stt, request.endpoint_tts,
+        request.base_url,
+        request.endpoint,
+        request.endpoint_llm,
+        request.endpoint_embedding,
+        request.endpoint_stt,
+        request.endpoint_tts,
     ]:
         if url_field:
             try:
-                validate_url(url_field, request.provider)
+                await validate_url(url_field, request.provider)
             except ValueError as e:
                 raise _handle_value_error(e)
 
@@ -164,10 +204,15 @@ async def create_credential(request: CreateCredentialRequest):
             project=request.project,
             location=request.location,
             credentials_path=request.credentials_path,
+            num_ctx=request.num_ctx,
         )
         await cred.save()
         return credential_to_response(cred, 0)
 
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error creating credential: {e}")
         raise HTTPException(status_code=500, detail="Failed to create credential")
@@ -180,6 +225,10 @@ async def get_credential(credential_id: str):
         cred = await Credential.get(credential_id)
         models = await cred.get_linked_models()
         return credential_to_response(cred, len(models))
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error fetching credential {credential_id}: {e}")
         raise HTTPException(status_code=404, detail="Credential not found")
@@ -195,44 +244,63 @@ async def update_credential(credential_id: str, request: UpdateCredentialRequest
 
     # Validate all URL fields being updated
     for url_field in [
-        request.base_url, request.endpoint, request.endpoint_llm,
-        request.endpoint_embedding, request.endpoint_stt, request.endpoint_tts,
+        request.base_url,
+        request.endpoint,
+        request.endpoint_llm,
+        request.endpoint_embedding,
+        request.endpoint_stt,
+        request.endpoint_tts,
     ]:
         if url_field:
             try:
-                validate_url(url_field, "update")
+                await validate_url(url_field, "update")
             except ValueError as e:
                 raise _handle_value_error(e)
 
     try:
         cred = await Credential.get(credential_id)
 
+        # Partial-update semantics keyed on field PRESENCE, not value:
+        # a field absent from the payload is left untouched, while an explicit
+        # null (or "") clears it. `is not None` checks would silently ignore
+        # a null sent to clear a field — the old value survived while the
+        # client saw success.
+        sent = request.model_fields_set
+
         if request.name is not None:
             cred.name = request.name
         if request.modalities is not None:
             cred.modalities = request.modalities
         if request.api_key is not None:
             cred.api_key = SecretStr(request.api_key)
-        if request.base_url is not None:
+        if "base_url" in sent:
             cred.base_url = request.base_url or None
-        if request.endpoint is not None:
+        if "endpoint" in sent:
             cred.endpoint = request.endpoint or None
-        if request.api_version is not None:
+        if "api_version" in sent:
             cred.api_version = request.api_version or None
-        if request.endpoint_llm is not None:
+        if "endpoint_llm" in sent:
             cred.endpoint_llm = request.endpoint_llm or None
-        if request.endpoint_embedding is not None:
+        if "endpoint_embedding" in sent:
             cred.endpoint_embedding = request.endpoint_embedding or None
-        if request.endpoint_stt is not None:
+        if "endpoint_stt" in sent:
             cred.endpoint_stt = request.endpoint_stt or None
-        if request.endpoint_tts is not None:
+        if "endpoint_tts" in sent:
             cred.endpoint_tts = request.endpoint_tts or None
-        if request.project is not None:
+        if "project" in sent:
             cred.project = request.project or None
-        if request.location is not None:
+        if "location" in sent:
             cred.location = request.location or None
-        if request.credentials_path is not None:
+        if "credentials_path" in sent:
             cred.credentials_path = request.credentials_path or None
+        if "num_ctx" in sent:
+            # 0/null/falsy clears the override and falls back to esperanto's default
+            cred.num_ctx = request.num_ctx or None
+
+        try:
+            ensure_provider_required_fields(cred)
+        except ValueError as e:
+            raise _handle_value_error(e)
 
         await cred.save()
         models = await cred.get_linked_models()
@@ -240,6 +308,10 @@ async def update_credential(credential_id: str, request: UpdateCredentialRequest
 
     except HTTPException:
         raise
+    except NotFoundError:
+        raise HTTPException(status_code=404, detail="Credential not found")
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error updating credential {credential_id}: {e}")
         raise HTTPException(status_code=500, detail="Failed to update credential")
@@ -260,7 +332,56 @@ async def delete_credential(
     - Otherwise, linked models are cascade-deleted automatically
     """
     try:
-        cred = await Credential.get(credential_id)
+        try:
+            cred = await Credential.get(credential_id)
+        except ValueError as decrypt_err:
+            # Credential exists but can't be decrypted (wrong encryption key).
+            # Fall back to direct DB operations for deletion.
+            logger.warning(
+                f"Cannot decrypt credential {credential_id}, "
+                f"falling back to direct delete: {decrypt_err}"
+            )
+
+            # Query linked models
+            linked = await repo_query(
+                "SELECT * FROM model WHERE credential = $cred_id",
+                {"cred_id": ensure_record_id(credential_id)},
+            )
+            deleted_models = 0
+
+            if linked and migrate_to:
+                # Migrate models to another credential
+                target_cred = await Credential.get(migrate_to)
+                for model_row in linked:
+                    model_id = str(model_row.get("id", ""))
+                    if model_id:
+                        await repo_query(
+                            "UPDATE $model_id SET credential = $target_id",
+                            {
+                                "model_id": ensure_record_id(model_id),
+                                # A fetched credential always has an id; fall
+                                # back to the requested id for the type checker.
+                                "target_id": ensure_record_id(
+                                    target_cred.id or migrate_to
+                                ),
+                            },
+                        )
+            elif linked:
+                # Cascade-delete linked models
+                for model_row in linked:
+                    model_id = str(model_row.get("id", ""))
+                    if model_id:
+                        await repo_delete(model_id)
+                        deleted_models += 1
+
+            # Delete the credential itself
+            await repo_delete(credential_id)
+
+            return CredentialDeleteResponse(
+                message="Credential deleted successfully",
+                deleted_models=deleted_models,
+            )
+
         linked_models = await cred.get_linked_models()
 
         deleted_models = 0
@@ -288,6 +409,10 @@ async def delete_credential(
 
     except HTTPException:
         raise
+    except NotFoundError:
+        raise HTTPException(status_code=404, detail="Credential not found")
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error deleting credential {credential_id}: {e}")
         raise HTTPException(status_code=500, detail="Failed to delete credential")
@@ -327,6 +452,10 @@ async def discover_models_for_credential(credential_id: str):
             ],
         )
 
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error discovering models for credential {credential_id}: {e}")
         raise HTTPException(status_code=500, detail="Failed to discover models")
@@ -340,6 +469,10 @@ async def register_models_for_credential(
     try:
         result = await register_models(credential_id, request.models)
         return RegisterModelsResponse(**result)
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error registering models for credential {credential_id}: {e}")
         raise HTTPException(status_code=500, detail="Failed to register models")
@@ -357,9 +490,42 @@ async def migrate_from_provider_config():
         return await svc_migrate_from_provider_config()
     except ValueError as e:
         raise _handle_value_error(e)
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
-        logger.error(f"ProviderConfig migration FAILED: {type(e).__name__}: {e}", exc_info=True)
-        raise HTTPException(status_code=500, detail="Migration from provider config failed")
+        logger.error(
+            f"ProviderConfig migration FAILED: {type(e).__name__}: {e}", exc_info=True
+        )
+        raise HTTPException(
+            status_code=500, detail="Migration from provider config failed"
+        )
+
+
+@router.post("/migrate-encryption")
+async def migrate_encryption():
+    """Re-encrypt stored API keys into the versioned PBKDF2 format.
+
+    Inherits the global password auth with no exemption: when
+    OPEN_NOTEBOOK_PASSWORD is unset, auth is disabled instance-wide and
+    this endpoint is reachable by anyone with network access, exactly like
+    the other migration endpoints. Set a password before running the pass
+    on a shared network.
+    """
+    try:
+        return await svc_migrate_encryption_scheme()
+    except ValueError as e:
+        raise _handle_value_error(e)
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
+    except Exception as e:
+        logger.error(
+            f"Encryption migration FAILED: {type(e).__name__}: {e}", exc_info=True
+        )
+        raise HTTPException(status_code=500, detail="Encryption migration failed")
 
 
 @router.post("/migrate-from-env")
@@ -369,6 +535,12 @@ async def migrate_from_env():
         return await svc_migrate_from_env()
     except ValueError as e:
         raise _handle_value_error(e)
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Env migration FAILED: {type(e).__name__}: {e}", exc_info=True)
-        raise HTTPException(status_code=500, detail="Migration from environment variables failed")
+        raise HTTPException(
+            status_code=500, detail="Migration from environment variables failed"
+        )

api/routers/embedding.py

@@ -5,6 +5,10 @@ from api.command_service import CommandService
 from api.models import EmbedRequest, EmbedResponse
 from open_notebook.ai.models import model_manager
 from open_notebook.domain.notebook import Note, Source
+from open_notebook.exceptions import (
+    NotFoundError,
+    OpenNotebookError,
+)
 
 router = APIRouter()
 
@@ -78,8 +82,6 @@ async def embed_content(embed_request: EmbedRequest):
             # Get the item and submit embedding job
             if item_type == "source":
                 source_item = await Source.get(item_id)
-                if not source_item:
-                    raise HTTPException(status_code=404, detail="Source not found")
 
                 # Submit embed_source job (returns command_id for tracking)
                 command_id = await source_item.vectorize()
@@ -87,11 +89,19 @@ async def embed_content(embed_request: EmbedRequest):
 
             elif item_type == "note":
                 note_item = await Note.get(item_id)
-                if not note_item:
-                    raise HTTPException(status_code=404, detail="Note not found")
 
-                # Note.save() internally submits embed_note command and returns command_id
+                # Note.save() internally submits embed_note command and
+                # returns command_id. Unlike Source.vectorize(), save()'s
+                # embed submission is best-effort (a hiccup there shouldn't
+                # fail an otherwise-successful note save) - but this
+                # endpoint's whole point is submitting the embedding job,
+                # so a submission failure here (content present, no
+                # command_id) must still surface as a failure.
                 command_id = await note_item.save()
+                if not command_id and note_item.content and note_item.content.strip():
+                    raise HTTPException(
+                        status_code=500, detail="Failed to submit note embedding job"
+                    )
                 message = "Note embedding job submitted"
 
             return EmbedResponse(
@@ -104,6 +114,12 @@ async def embed_content(embed_request: EmbedRequest):
 
     except HTTPException:
         raise
+    except NotFoundError:
+        raise HTTPException(
+            status_code=404, detail=f"{embed_request.item_type} not found"
+        )
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(
             f"Error embedding {embed_request.item_type} {embed_request.item_id}: {str(e)}"

api/routers/embedding_rebuild.py

@@ -11,6 +11,7 @@ from api.models import (
     RebuildStatusResponse,
 )
 from open_notebook.database.repository import repo_query
+from open_notebook.exceptions import OpenNotebookError
 
 router = APIRouter()
 
@@ -112,6 +113,10 @@ async def start_rebuild(request: RebuildRequest):
             message=f"Rebuild operation started. Estimated {total_estimate} items to process.",
         )
 
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Failed to start rebuild: {e}")
         logger.exception(e)
@@ -184,6 +189,8 @@ async def get_rebuild_status(command_id: str):
 
     except HTTPException:
         raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Failed to get rebuild status: {e}")
         logger.exception(e)

api/routers/episode_profiles.py

@@ -1,10 +1,11 @@
-from typing import List, Optional
+from typing import Dict, List, Optional
 
 from fastapi import APIRouter, HTTPException
 from loguru import logger
 from pydantic import BaseModel, Field
 
-from open_notebook.podcasts.models import EpisodeProfile
+from open_notebook.exceptions import InvalidInputError, OpenNotebookError
+from open_notebook.podcasts.models import EpisodeProfile, SpeakerProfile
 
 router = APIRouter()
 
@@ -13,48 +14,83 @@ class EpisodeProfileResponse(BaseModel):
     id: str
     name: str
     description: str
-    speaker_config: str
+    speaker_config: Optional[str] = Field(
+        None, description="speaker_profile record ID (null when orphaned)"
+    )
+    speaker_config_name: Optional[str] = Field(
+        None, description="Resolved speaker profile name (for display)"
+    )
     outline_llm: Optional[str] = None
     transcript_llm: Optional[str] = None
     language: Optional[str] = None
     default_briefing: str
     num_segments: int
-    # Legacy fields (for display/migration awareness)
-    outline_provider: Optional[str] = None
-    outline_model: Optional[str] = None
-    transcript_provider: Optional[str] = None
-    transcript_model: Optional[str] = None
+    max_tokens: Optional[int] = None
 
 
-def _profile_to_response(profile: EpisodeProfile) -> EpisodeProfileResponse:
+async def _speaker_names_by_id() -> Dict[str, str]:
+    """Map speaker_profile record ID -> name for list serialization."""
+    speakers = await SpeakerProfile.get_all()
+    return {str(speaker.id): speaker.name for speaker in speakers}
+
+
+async def _speaker_name_for(speaker_config: Optional[str]) -> Optional[str]:
+    """Resolve one profile's speaker_config record ID to the speaker name.
+
+    Returns None for a missing or dangling reference - the frontend renders
+    that as "needs setup"."""
+    if not speaker_config:
+        return None
+    speaker = await SpeakerProfile.resolve(speaker_config)
+    return speaker.name if speaker else None
+
+
+def _profile_to_response(
+    profile: EpisodeProfile, speaker_name: Optional[str]
+) -> EpisodeProfileResponse:
     return EpisodeProfileResponse(
         id=str(profile.id),
         name=profile.name,
         description=profile.description or "",
         speaker_config=profile.speaker_config,
+        speaker_config_name=speaker_name,
         outline_llm=profile.outline_llm,
         transcript_llm=profile.transcript_llm,
         language=profile.language,
         default_briefing=profile.default_briefing,
         num_segments=profile.num_segments,
-        outline_provider=profile.outline_provider,
-        outline_model=profile.outline_model,
-        transcript_provider=profile.transcript_provider,
-        transcript_model=profile.transcript_model,
+        max_tokens=profile.max_tokens,
     )
 
 
+async def _resolve_speaker_config(value: str) -> SpeakerProfile:
+    """Resolve an incoming speaker_config (record ID, or name for backward
+    compatibility) to the referenced SpeakerProfile."""
+    speaker = await SpeakerProfile.resolve(value)
+    if not speaker:
+        raise InvalidInputError(f"Speaker profile '{value}' not found")
+    return speaker
+
+
 @router.get("/episode-profiles", response_model=List[EpisodeProfileResponse])
 async def list_episode_profiles():
     """List all available episode profiles"""
     try:
         profiles = await EpisodeProfile.get_all(order_by="name asc")
-        return [_profile_to_response(p) for p in profiles]
+        speaker_names = await _speaker_names_by_id()
+        return [
+            _profile_to_response(
+                p, speaker_names.get(p.speaker_config) if p.speaker_config else None
+            )
+            for p in profiles
+        ]
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Failed to fetch episode profiles: {e}")
-        raise HTTPException(
-            status_code=500, detail="Failed to fetch episode profiles"
-        )
+        raise HTTPException(status_code=500, detail="Failed to fetch episode profiles")
 
 
 @router.get("/episode-profiles/{profile_name}", response_model=EpisodeProfileResponse)
@@ -68,21 +104,29 @@ async def get_episode_profile(profile_name: str):
                 status_code=404, detail=f"Episode profile '{profile_name}' not found"
             )
 
-        return _profile_to_response(profile)
+        return _profile_to_response(
+            profile, await _speaker_name_for(profile.speaker_config)
+        )
 
     except HTTPException:
         raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Failed to fetch episode profile '{profile_name}': {e}")
-        raise HTTPException(
-            status_code=500, detail="Failed to fetch episode profile"
-        )
+        raise HTTPException(status_code=500, detail="Failed to fetch episode profile")
 
 
 class EpisodeProfileCreate(BaseModel):
     name: str = Field(..., description="Unique profile name")
     description: str = Field("", description="Profile description")
-    speaker_config: str = Field(..., description="Reference to speaker profile name")
+    speaker_config: str = Field(
+        ...,
+        description=(
+            "speaker_profile record ID (a profile name is also accepted "
+            "for backward compatibility)"
+        ),
+    )
     outline_llm: Optional[str] = Field(None, description="Model record ID for outline")
     transcript_llm: Optional[str] = Field(
         None, description="Model record ID for transcript"
@@ -90,40 +134,39 @@ class EpisodeProfileCreate(BaseModel):
     language: Optional[str] = Field(None, description="Podcast language code")
     default_briefing: str = Field(..., description="Default briefing template")
     num_segments: int = Field(default=5, description="Number of podcast segments")
-    # Legacy fields (accepted but not required)
-    outline_provider: Optional[str] = None
-    outline_model: Optional[str] = None
-    transcript_provider: Optional[str] = None
-    transcript_model: Optional[str] = None
+    max_tokens: Optional[int] = Field(
+        None,
+        description="Max output tokens for outline/transcript generation",
+    )
 
 
 @router.post("/episode-profiles", response_model=EpisodeProfileResponse)
 async def create_episode_profile(profile_data: EpisodeProfileCreate):
     """Create a new episode profile"""
     try:
+        speaker = await _resolve_speaker_config(profile_data.speaker_config)
         profile = EpisodeProfile(
             name=profile_data.name,
             description=profile_data.description,
-            speaker_config=profile_data.speaker_config,
+            speaker_config=str(speaker.id),
             outline_llm=profile_data.outline_llm,
             transcript_llm=profile_data.transcript_llm,
             language=profile_data.language,
             default_briefing=profile_data.default_briefing,
             num_segments=profile_data.num_segments,
-            outline_provider=profile_data.outline_provider,
-            outline_model=profile_data.outline_model,
-            transcript_provider=profile_data.transcript_provider,
-            transcript_model=profile_data.transcript_model,
+            max_tokens=profile_data.max_tokens,
         )
 
         await profile.save()
-        return _profile_to_response(profile)
+        return _profile_to_response(profile, speaker.name)
 
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Failed to create episode profile: {e}")
-        raise HTTPException(
-            status_code=500, detail="Failed to create episode profile"
-        )
+        raise HTTPException(status_code=500, detail="Failed to create episode profile")
 
 
 @router.put("/episode-profiles/{profile_id}", response_model=EpisodeProfileResponse)
@@ -137,29 +180,27 @@ async def update_episode_profile(profile_id: str, profile_data: EpisodeProfileCr
                 status_code=404, detail=f"Episode profile '{profile_id}' not found"
             )
 
-        profile.name = profile_data.name
-        profile.description = profile_data.description
-        profile.speaker_config = profile_data.speaker_config
-        profile.outline_llm = profile_data.outline_llm
-        profile.transcript_llm = profile_data.transcript_llm
-        profile.language = profile_data.language
-        profile.default_briefing = profile_data.default_briefing
-        profile.num_segments = profile_data.num_segments
-        profile.outline_provider = profile_data.outline_provider
-        profile.outline_model = profile_data.outline_model
-        profile.transcript_provider = profile_data.transcript_provider
-        profile.transcript_model = profile_data.transcript_model
+        update_data = profile_data.model_dump(exclude_unset=True)
+        speaker_name: Optional[str] = None
+        if "speaker_config" in update_data:
+            speaker = await _resolve_speaker_config(update_data["speaker_config"])
+            update_data["speaker_config"] = str(speaker.id)
+            speaker_name = speaker.name
+        for field, value in update_data.items():
+            setattr(profile, field, value)
 
         await profile.save()
-        return _profile_to_response(profile)
+        if speaker_name is None:
+            speaker_name = await _speaker_name_for(profile.speaker_config)
+        return _profile_to_response(profile, speaker_name)
 
     except HTTPException:
         raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Failed to update episode profile: {e}")
-        raise HTTPException(
-            status_code=500, detail="Failed to update episode profile"
-        )
+        raise HTTPException(status_code=500, detail="Failed to update episode profile")
 
 
 @router.delete("/episode-profiles/{profile_id}")
@@ -179,11 +220,11 @@ async def delete_episode_profile(profile_id: str):
 
     except HTTPException:
         raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Failed to delete episode profile: {e}")
-        raise HTTPException(
-            status_code=500, detail="Failed to delete episode profile"
-        )
+        raise HTTPException(status_code=500, detail="Failed to delete episode profile")
 
 
 @router.post(
@@ -208,17 +249,18 @@ async def duplicate_episode_profile(profile_id: str):
             language=original.language,
             default_briefing=original.default_briefing,
             num_segments=original.num_segments,
-            outline_provider=original.outline_provider,
-            outline_model=original.outline_model,
-            transcript_provider=original.transcript_provider,
-            transcript_model=original.transcript_model,
+            max_tokens=original.max_tokens,
         )
 
         await duplicate.save()
-        return _profile_to_response(duplicate)
+        return _profile_to_response(
+            duplicate, await _speaker_name_for(duplicate.speaker_config)
+        )
 
     except HTTPException:
         raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Failed to duplicate episode profile: {e}")
         raise HTTPException(

api/routers/insights.py

@@ -3,7 +3,11 @@ from loguru import logger
 
 from api.models import NoteResponse, SaveAsNoteRequest, SourceInsightResponse
 from open_notebook.domain.notebook import SourceInsight
-from open_notebook.exceptions import InvalidInputError
+from open_notebook.exceptions import (
+    InvalidInputError,
+    NotFoundError,
+    OpenNotebookError,
+)
 
 router = APIRouter()
 
@@ -24,11 +28,13 @@ async def get_insight(insight_id: str):
             source_id=source.id or "",
             insight_type=insight.insight_type,
             content=insight.content,
-            created=str(insight.created),
-            updated=str(insight.updated),
+            created=insight.created.isoformat() if insight.created else None,
+            updated=insight.updated.isoformat() if insight.updated else None,
         )
     except HTTPException:
         raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error fetching insight {insight_id}: {str(e)}")
         raise HTTPException(status_code=500, detail="Error fetching insight")
@@ -47,6 +53,8 @@ async def delete_insight(insight_id: str):
         return {"message": "Insight deleted successfully"}
     except HTTPException:
         raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error deleting insight {insight_id}: {str(e)}")
         raise HTTPException(status_code=500, detail="Error deleting insight")
@@ -73,10 +81,12 @@ async def save_insight_as_note(insight_id: str, request: SaveAsNoteRequest):
         )
     except HTTPException:
         raise
+    except NotFoundError:
+        raise HTTPException(status_code=404, detail="Notebook not found")
     except InvalidInputError as e:
         raise HTTPException(status_code=400, detail=str(e))
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error saving insight {insight_id} as note: {str(e)}")
-        raise HTTPException(
-            status_code=500, detail="Error saving insight as note"
-        )
+        raise HTTPException(status_code=500, detail="Error saving insight as note")

api/routers/models.py

@@ -13,7 +13,6 @@ from api.models import (
     ModelResponse,
     ProviderAvailabilityResponse,
 )
-from open_notebook.domain.credential import Credential
 from open_notebook.ai.connection_tester import test_individual_model
 from open_notebook.ai.key_provider import provision_provider_keys
 from open_notebook.ai.model_discovery import (
@@ -23,7 +22,12 @@ from open_notebook.ai.model_discovery import (
     sync_provider_models,
 )
 from open_notebook.ai.models import DefaultModels, Model
-from open_notebook.exceptions import InvalidInputError
+from open_notebook.domain.credential import Credential
+from open_notebook.exceptions import (
+    InvalidInputError,
+    NotFoundError,
+    OpenNotebookError,
+)
 
 router = APIRouter()
 
@@ -104,11 +108,11 @@ PROVIDER_PRIORITY = [
 MODEL_PREFERENCES = {
     "openai": ["gpt-4o", "gpt-4", "gpt-3.5-turbo"],
     "anthropic": ["claude-3-5-sonnet", "claude-3-opus", "claude-3-sonnet"],
-    "google": ["gemini-2.0", "gemini-1.5-pro", "gemini-pro"],
+    "google": ["gemini-3.5-flash", "gemini-2.5-flash", "gemini-2.5-pro"],
     "mistral": ["mistral-large", "mixtral"],
     "groq": ["llama-3.3", "llama-3.1", "mixtral"],
     "dashscope": ["qwen-max", "qwen-plus", "qwen-turbo"],
-    "minimax": ["MiniMax-M2.5", "MiniMax-M2.5-highspeed"],
+    "minimax": ["MiniMax-M3", "MiniMax-M2.5", "MiniMax-M2.5-highspeed"],
 }
 
 
@@ -189,6 +193,10 @@ async def get_models(
             )
             for model in models
         ]
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error fetching models: {str(e)}")
         raise HTTPException(status_code=500, detail=f"Error fetching models: {str(e)}")
@@ -244,6 +252,8 @@ async def create_model(model_data: ModelCreate):
         raise
     except InvalidInputError as e:
         raise HTTPException(status_code=400, detail=str(e))
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error creating model: {str(e)}")
         raise HTTPException(status_code=500, detail=f"Error creating model: {str(e)}")
@@ -254,14 +264,16 @@ async def delete_model(model_id: str):
     """Delete a model configuration."""
     try:
         model = await Model.get(model_id)
-        if not model:
-            raise HTTPException(status_code=404, detail="Model not found")
 
         await model.delete()
 
         return {"message": "Model deleted successfully"}
     except HTTPException:
         raise
+    except NotFoundError:
+        raise HTTPException(status_code=404, detail="Model not found")
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error deleting model {model_id}: {str(e)}")
         raise HTTPException(status_code=500, detail=f"Error deleting model: {str(e)}")
@@ -276,6 +288,8 @@ async def test_model(model_id: str):
             raise HTTPException(status_code=404, detail="Model not found")
     except HTTPException:
         raise
+    except OpenNotebookError:
+        raise
     except Exception:
         raise HTTPException(status_code=404, detail="Model not found")
 
@@ -305,6 +319,10 @@ async def get_default_models():
             default_embedding_model=defaults.default_embedding_model,  # type: ignore[attr-defined]
             default_tools_model=defaults.default_tools_model,  # type: ignore[attr-defined]
         )
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error fetching default models: {str(e)}")
         raise HTTPException(
@@ -312,33 +330,35 @@ async def get_default_models():
         )
 
 
+# Defaults the app cannot function without — they can be reassigned but
+# never cleared (the optional ones fall back to the chat default or are
+# simply skipped when unset).
+REQUIRED_DEFAULTS = {"default_chat_model", "default_embedding_model"}
+
+
 @router.put("/models/defaults", response_model=DefaultModelsResponse)
 async def update_default_models(defaults_data: DefaultModelsResponse):
-    """Update default model assignments."""
+    """Update default model assignments.
+
+    Partial-update semantics keyed on field PRESENCE, not value: a field
+    absent from the payload is left untouched, while an explicit null clears
+    the default (except required ones). `is not None` checks would silently
+    ignore a null sent to clear a default — the old value survived while the
+    client saw success (same anti-pattern fixed for credentials in #1046).
+    """
     try:
         defaults = await DefaultModels.get_instance()
 
-        # Update only provided fields
-        if defaults_data.default_chat_model is not None:
-            defaults.default_chat_model = defaults_data.default_chat_model  # type: ignore[attr-defined]
-        if defaults_data.default_transformation_model is not None:
-            defaults.default_transformation_model = (
-                defaults_data.default_transformation_model
-            )  # type: ignore[attr-defined]
-        if defaults_data.large_context_model is not None:
-            defaults.large_context_model = defaults_data.large_context_model  # type: ignore[attr-defined]
-        if defaults_data.default_text_to_speech_model is not None:
-            defaults.default_text_to_speech_model = (
-                defaults_data.default_text_to_speech_model
-            )  # type: ignore[attr-defined]
-        if defaults_data.default_speech_to_text_model is not None:
-            defaults.default_speech_to_text_model = (
-                defaults_data.default_speech_to_text_model
-            )  # type: ignore[attr-defined]
-        if defaults_data.default_embedding_model is not None:
-            defaults.default_embedding_model = defaults_data.default_embedding_model  # type: ignore[attr-defined]
-        if defaults_data.default_tools_model is not None:
-            defaults.default_tools_model = defaults_data.default_tools_model  # type: ignore[attr-defined]
+        sent = defaults_data.model_fields_set
+        for field in DefaultModelsResponse.model_fields:
+            if field not in sent:
+                continue
+            value = getattr(defaults_data, field)
+            if value is None and field in REQUIRED_DEFAULTS:
+                raise InvalidInputError(
+                    f"{field} is required and cannot be cleared, only reassigned"
+                )
+            setattr(defaults, field, value)
 
         await defaults.update()
 
@@ -355,6 +375,8 @@ async def update_default_models(defaults_data: DefaultModelsResponse):
         )
     except HTTPException:
         raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error updating default models: {str(e)}")
         raise HTTPException(
@@ -381,9 +403,16 @@ async def get_provider_availability():
             "openrouter": "OPENROUTER_API_KEY",
             "voyage": "VOYAGE_API_KEY",
             "elevenlabs": "ELEVENLABS_API_KEY",
+            "deepgram": "DEEPGRAM_API_KEY",
             "ollama": "OLLAMA_API_BASE",
+            "omlx": "OMLX_API_BASE",
             "dashscope": "DASHSCOPE_API_KEY",
             "minimax": "MINIMAX_API_KEY",
+            "novita": "NOVITA_API_KEY",
+            "siliconflow": "SILICONFLOW_API_KEY",
+            "zai": "ZAI_API_KEY",
+            "ppq": "PPQ_API_KEY",
+            "cohere": "COHERE_API_KEY",
         }
 
         provider_status = {}
@@ -414,13 +443,19 @@ async def get_provider_availability():
         )
 
         # OpenAI-compatible: DB credential or env vars
-        provider_status["openai-compatible"] = (
+        provider_status["openai_compatible"] = (
             await _check_provider_has_credential("openai_compatible")
             or _check_openai_compatible_support("LLM")
             or _check_openai_compatible_support("EMBEDDING")
             or _check_openai_compatible_support("STT")
             or _check_openai_compatible_support("TTS")
         )
+        provider_status["anthropic_compatible"] = await _check_provider_has_credential(
+            "anthropic_compatible"
+        ) or (
+            bool((os.environ.get("ANTHROPIC_COMPATIBLE_BASE_URL") or "").strip())
+            and bool((os.environ.get("ANTHROPIC_COMPATIBLE_API_KEY") or "").strip())
+        )
 
         available_providers = [k for k, v in provider_status.items() if v]
         unavailable_providers = [k for k, v in provider_status.items() if not v]
@@ -442,12 +477,15 @@ async def get_provider_availability():
             }
 
             # Special handling for openai-compatible to check mode-specific availability
-            if provider == "openai-compatible":
+            if provider == "openai_compatible":
+                # Esperanto exposes this provider with a hyphen ("openai-compatible"),
+                # while the rest of the codebase uses the underscore form.
+                esperanto_name = "openai-compatible"
                 has_db_cred = await _check_provider_has_credential("openai_compatible")
                 for model_type, mode in mode_mapping.items():
                     if (
                         model_type in esperanto_available
-                        and provider in esperanto_available[model_type]
+                        and esperanto_name in esperanto_available[model_type]
                     ):
                         if has_db_cred or _check_openai_compatible_support(mode):
                             supported_types[provider].append(model_type)
@@ -461,6 +499,12 @@ async def get_provider_availability():
                     ):
                         if has_db_cred or _check_azure_support(mode):
                             supported_types[provider].append(model_type)
+            elif provider == "anthropic_compatible":
+                if (
+                    "language" in esperanto_available
+                    and "anthropic" in esperanto_available["language"]
+                ):
+                    supported_types[provider].append("language")
             else:
                 # Standard provider detection
                 for model_type, providers in esperanto_available.items():
@@ -472,6 +516,10 @@ async def get_provider_availability():
             unavailable=unavailable_providers,
             supported_types=supported_types,
         )
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error checking provider availability: {str(e)}")
         raise HTTPException(
@@ -484,9 +532,7 @@ async def get_provider_availability():
 # =============================================================================
 
 
-@router.get(
-    "/models/discover/{provider}", response_model=List[DiscoveredModelResponse]
-)
+@router.get("/models/discover/{provider}", response_model=List[DiscoveredModelResponse])
 async def discover_models(provider: str):
     """
     Discover available models from a provider without registering them.
@@ -508,10 +554,15 @@ async def discover_models(provider: str):
             )
             for m in discovered
         ]
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error discovering models for {provider}: {str(e)}")
         raise HTTPException(
-            status_code=500, detail="Error discovering models. Check server logs for details."
+            status_code=500,
+            detail="Error discovering models. Check server logs for details.",
         )
 
 
@@ -537,9 +588,16 @@ async def sync_models(provider: str):
             new=new,
             existing=existing,
         )
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error syncing models for {provider}: {str(e)}")
-        raise HTTPException(status_code=500, detail="Error syncing models. Check server logs for details.")
+        raise HTTPException(
+            status_code=500,
+            detail="Error syncing models. Check server logs for details.",
+        )
 
 
 @router.post("/models/sync", response_model=AllProvidersSyncResponse)
@@ -573,6 +631,10 @@ async def sync_all_models():
             total_discovered=total_discovered,
             total_new=total_new,
         )
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error syncing all models: {str(e)}")
         raise HTTPException(
@@ -596,6 +658,10 @@ async def get_model_count(provider: str):
             counts=counts,
             total=total,
         )
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error getting model count for {provider}: {str(e)}")
         raise HTTPException(
@@ -630,11 +696,13 @@ async def get_models_by_provider(provider: str):
             )
             for model in models
         ]
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error fetching models for {provider}: {str(e)}")
-        raise HTTPException(
-            status_code=500, detail=f"Error fetching models: {str(e)}"
-        )
+        raise HTTPException(status_code=500, detail=f"Error fetching models: {str(e)}")
 
 
 def _get_preferred_model(
@@ -722,21 +790,22 @@ async def auto_assign_defaults():
                 models_by_type[model_type].append(model)
 
         # Define slot configuration: (slot_name, model_type, current_value)
-        slot_configs = [
+        #
+        # Only REQUIRED slots are auto-assigned. Optional slots
+        # (transformation, tools, large_context, TTS, STT) are intentionally
+        # left untouched: they fall back to the chat model when empty, so an
+        # empty optional slot may be a deliberate user choice (see #1097/#1098).
+        # Re-populating them here would silently undo that intent.
+        required_slot_configs = [
             ("default_chat_model", "language", defaults.default_chat_model),  # type: ignore[attr-defined]
-            ("default_transformation_model", "language", defaults.default_transformation_model),  # type: ignore[attr-defined]
-            ("default_tools_model", "language", defaults.default_tools_model),  # type: ignore[attr-defined]
-            ("large_context_model", "language", defaults.large_context_model),  # type: ignore[attr-defined]
             ("default_embedding_model", "embedding", defaults.default_embedding_model),  # type: ignore[attr-defined]
-            ("default_text_to_speech_model", "text_to_speech", defaults.default_text_to_speech_model),  # type: ignore[attr-defined]
-            ("default_speech_to_text_model", "speech_to_text", defaults.default_speech_to_text_model),  # type: ignore[attr-defined]
         ]
 
         assigned: Dict[str, str] = {}
         skipped: List[str] = []
         missing: List[str] = []
 
-        for slot_name, model_type, current_value in slot_configs:
+        for slot_name, model_type, current_value in required_slot_configs:
             if current_value:
                 # Slot already has a value
                 skipped.append(slot_name)
@@ -769,6 +838,10 @@ async def auto_assign_defaults():
             missing=missing,
         )
 
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error auto-assigning defaults: {str(e)}")
         raise HTTPException(

api/routers/notebooks.py

@@ -9,14 +9,55 @@ from api.models import (
     NotebookDeleteResponse,
     NotebookResponse,
     NotebookUpdate,
+    RecentlyViewedResponse,
 )
 from open_notebook.database.repository import ensure_record_id, repo_query
 from open_notebook.domain.notebook import Notebook, Source
-from open_notebook.exceptions import InvalidInputError
+from open_notebook.exceptions import (
+    InvalidInputError,
+    NotFoundError,
+    OpenNotebookError,
+)
 
 router = APIRouter()
 
 
+def _last_viewed_sort_key(item: RecentlyViewedResponse) -> str:
+    return item.last_viewed_at
+
+
+async def _stamp_notebook_view(notebook_id: str) -> None:
+    # Best-effort write-on-read: recording the view timestamp must never turn a
+    # successful read into a 500. Log and move on if the stamp update fails.
+    try:
+        await repo_query(
+            "UPDATE $notebook_id SET last_viewed_at = time::now();",
+            {"notebook_id": ensure_record_id(notebook_id)},
+        )
+    except Exception as e:
+        logger.warning(
+            f"Failed to stamp last_viewed_at for notebook {notebook_id}: {e}"
+        )
+
+
+def _recently_viewed_notebook(row: dict) -> RecentlyViewedResponse:
+    return RecentlyViewedResponse(
+        type="notebook",
+        id=str(row.get("id", "")),
+        title=row.get("title") or row.get("name") or "Untitled notebook",
+        last_viewed_at=str(row.get("last_viewed_at", "")),
+    )
+
+
+def _recently_viewed_source(row: dict) -> RecentlyViewedResponse:
+    return RecentlyViewedResponse(
+        type="source",
+        id=str(row.get("id", "")),
+        title=row.get("title") or "Untitled source",
+        last_viewed_at=str(row.get("last_viewed_at", "")),
+    )
+
+
 @router.get("/notebooks", response_model=List[NotebookResponse])
 async def get_notebooks(
     archived: Optional[bool] = Query(None, description="Filter by archived status"),
@@ -24,30 +65,26 @@ async def get_notebooks(
 ):
     """Get all notebooks with optional filtering and ordering."""
     try:
-        # Validate order_by against allowlist to prevent SurrealQL injection
+        # Normalize through the central validator (SurrealQL injection guard),
+        # then apply this endpoint's stricter allowlist: a single clause on one
+        # of the sortable notebook fields.
         allowed_fields = {"name", "created", "updated"}
-        allowed_directions = {"asc", "desc"}
-
-        parts = order_by.strip().lower().split()
-        if len(parts) == 1:
-            if parts[0] not in allowed_fields:
-                raise HTTPException(
-                    status_code=400,
-                    detail=f"Invalid order_by field: '{order_by}'. Allowed fields: {', '.join(sorted(allowed_fields))}",
-                )
-            validated_order_by = parts[0]
-        elif len(parts) == 2:
-            if parts[0] not in allowed_fields or parts[1] not in allowed_directions:
-                raise HTTPException(
-                    status_code=400,
-                    detail=f"Invalid order_by: '{order_by}'. Allowed fields: {', '.join(sorted(allowed_fields))}. Allowed directions: asc, desc",
-                )
-            validated_order_by = f"{parts[0]} {parts[1]}"
-        else:
-            raise HTTPException(
-                status_code=400,
-                detail=f"Invalid order_by format: '{order_by}'. Expected 'field' or 'field direction'",
-            )
+        allowed_message = (
+            f"Allowed fields: {', '.join(sorted(allowed_fields))}. "
+            "Allowed directions: asc, desc"
+        )
+        invalid = InvalidInputError(
+            f"Invalid order_by: '{order_by}'. {allowed_message}"
+        )
+        try:
+            validated_order_by = Notebook._validate_order_by(order_by)
+        except InvalidInputError:
+            raise invalid from None
+        if (
+            "," in validated_order_by
+            or validated_order_by.split()[0] not in allowed_fields
+        ):
+            raise invalid
 
         # Build the query with counts
         query = f"""
@@ -79,6 +116,8 @@ async def get_notebooks(
         ]
     except HTTPException:
         raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error fetching notebooks: {str(e)}")
         raise HTTPException(
@@ -108,6 +147,10 @@ async def create_notebook(notebook: NotebookCreate):
         )
     except InvalidInputError as e:
         raise HTTPException(status_code=400, detail=str(e))
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error creating notebook: {str(e)}")
         raise HTTPException(
@@ -115,6 +158,52 @@ async def create_notebook(notebook: NotebookCreate):
         )
 
 
+@router.get("/recently-viewed", response_model=List[RecentlyViewedResponse])
+async def get_recently_viewed(
+    limit: int = Query(12, ge=1, le=50, description="Number of items to return"),
+):
+    """Get recently viewed notebooks and sources, newest first."""
+    try:
+        notebooks = await repo_query(
+            """
+            SELECT id, name AS title, last_viewed_at
+            FROM notebook
+            WHERE last_viewed_at != NONE AND last_viewed_at != NULL
+            ORDER BY last_viewed_at DESC
+            LIMIT $limit
+            """,
+            {"limit": limit},
+        )
+        sources = await repo_query(
+            """
+            SELECT id, title, last_viewed_at
+            FROM source
+            WHERE last_viewed_at != NONE AND last_viewed_at != NULL
+            ORDER BY last_viewed_at DESC
+            LIMIT $limit
+            """,
+            {"limit": limit},
+        )
+
+        items = [
+            *[_recently_viewed_notebook(nb) for nb in notebooks],
+            *[_recently_viewed_source(src) for src in sources],
+        ]
+        items.sort(key=_last_viewed_sort_key, reverse=True)
+        return items[:limit]
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
+    except Exception as e:
+        # Log full context server-side; return a generic message so internal
+        # details are not leaked to clients.
+        logger.exception(f"Error fetching recently viewed items: {e}")
+        raise HTTPException(
+            status_code=500, detail="Error fetching recently viewed items"
+        )
+
+
 @router.get(
     "/notebooks/{notebook_id}/delete-preview", response_model=NotebookDeletePreview
 )
@@ -122,8 +211,6 @@ async def get_notebook_delete_preview(notebook_id: str):
     """Get a preview of what will be deleted when this notebook is deleted."""
     try:
         notebook = await Notebook.get(notebook_id)
-        if not notebook:
-            raise HTTPException(status_code=404, detail="Notebook not found")
 
         preview = await notebook.get_delete_preview()
 
@@ -136,6 +223,10 @@ async def get_notebook_delete_preview(notebook_id: str):
         )
     except HTTPException:
         raise
+    except NotFoundError:
+        raise HTTPException(status_code=404, detail="Notebook not found")
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error getting delete preview for notebook {notebook_id}: {e}")
         raise HTTPException(
@@ -160,6 +251,8 @@ async def get_notebook(notebook_id: str):
         if not result:
             raise HTTPException(status_code=404, detail="Notebook not found")
 
+        await _stamp_notebook_view(notebook_id)
+
         nb = result[0]
         return NotebookResponse(
             id=str(nb.get("id", "")),
@@ -173,6 +266,8 @@ async def get_notebook(notebook_id: str):
         )
     except HTTPException:
         raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error fetching notebook {notebook_id}: {str(e)}")
         raise HTTPException(
@@ -185,14 +280,14 @@ async def update_notebook(notebook_id: str, notebook_update: NotebookUpdate):
     """Update a notebook."""
     try:
         notebook = await Notebook.get(notebook_id)
-        if not notebook:
-            raise HTTPException(status_code=404, detail="Notebook not found")
 
         # Update only provided fields
         if notebook_update.name is not None:
             notebook.name = notebook_update.name
-        if notebook_update.description is not None:
-            notebook.description = notebook_update.description
+        # An explicit "" (or null) clears the description; only an absent
+        # field leaves it untouched.
+        if "description" in notebook_update.model_fields_set:
+            notebook.description = notebook_update.description or ""
         if notebook_update.archived is not None:
             notebook.archived = notebook_update.archived
 
@@ -233,8 +328,12 @@ async def update_notebook(notebook_id: str, notebook_update: NotebookUpdate):
         )
     except HTTPException:
         raise
+    except NotFoundError:
+        raise HTTPException(status_code=404, detail="Notebook not found")
     except InvalidInputError as e:
         raise HTTPException(status_code=400, detail=str(e))
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error updating notebook {notebook_id}: {str(e)}")
         raise HTTPException(
@@ -246,15 +345,9 @@ async def update_notebook(notebook_id: str, notebook_update: NotebookUpdate):
 async def add_source_to_notebook(notebook_id: str, source_id: str):
     """Add an existing source to a notebook (create the reference)."""
     try:
-        # Check if notebook exists
-        notebook = await Notebook.get(notebook_id)
-        if not notebook:
-            raise HTTPException(status_code=404, detail="Notebook not found")
-
-        # Check if source exists
-        source = await Source.get(source_id)
-        if not source:
-            raise HTTPException(status_code=404, detail="Source not found")
+        # Verify the notebook and source exist (raises NotFoundError -> 404)
+        await Notebook.get(notebook_id)
+        await Source.get(source_id)
 
         # Check if reference already exists (idempotency)
         existing_ref = await repo_query(
@@ -278,6 +371,10 @@ async def add_source_to_notebook(notebook_id: str, source_id: str):
         return {"message": "Source linked to notebook successfully"}
     except HTTPException:
         raise
+    except NotFoundError:
+        raise HTTPException(status_code=404, detail="Notebook or source not found")
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(
             f"Error linking source {source_id} to notebook {notebook_id}: {str(e)}"
@@ -291,10 +388,8 @@ async def add_source_to_notebook(notebook_id: str, source_id: str):
 async def remove_source_from_notebook(notebook_id: str, source_id: str):
     """Remove a source from a notebook (delete the reference)."""
     try:
-        # Check if notebook exists
-        notebook = await Notebook.get(notebook_id)
-        if not notebook:
-            raise HTTPException(status_code=404, detail="Notebook not found")
+        # Verify the notebook exists (raises NotFoundError -> 404)
+        await Notebook.get(notebook_id)
 
         # Delete the reference record linking source to notebook
         await repo_query(
@@ -308,6 +403,10 @@ async def remove_source_from_notebook(notebook_id: str, source_id: str):
         return {"message": "Source removed from notebook successfully"}
     except HTTPException:
         raise
+    except NotFoundError:
+        raise HTTPException(status_code=404, detail="Notebook not found")
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(
             f"Error removing source {source_id} from notebook {notebook_id}: {str(e)}"
@@ -334,19 +433,24 @@ async def delete_notebook(
     """
     try:
         notebook = await Notebook.get(notebook_id)
-        if not notebook:
-            raise HTTPException(status_code=404, detail="Notebook not found")
 
-        result = await notebook.delete(delete_exclusive_sources=delete_exclusive_sources)
+        result = await notebook.delete(
+            delete_exclusive_sources=delete_exclusive_sources
+        )
 
         return NotebookDeleteResponse(
             message="Notebook deleted successfully",
             deleted_notes=result["deleted_notes"],
             deleted_sources=result["deleted_sources"],
             unlinked_sources=result["unlinked_sources"],
+            deleted_chat_sessions=result["deleted_chat_sessions"],
         )
     except HTTPException:
         raise
+    except NotFoundError:
+        raise HTTPException(status_code=404, detail="Notebook not found")
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error deleting notebook {notebook_id}: {str(e)}")
         raise HTTPException(

api/routers/notes.py

@@ -5,7 +5,11 @@ from loguru import logger
 
 from api.models import NoteCreate, NoteResponse, NoteUpdate
 from open_notebook.domain.notebook import Note
-from open_notebook.exceptions import InvalidInputError
+from open_notebook.exceptions import (
+    InvalidInputError,
+    NotFoundError,
+    OpenNotebookError,
+)
 
 router = APIRouter()
 
@@ -21,8 +25,6 @@ async def get_notes(
             from open_notebook.domain.notebook import Notebook
 
             notebook = await Notebook.get(notebook_id)
-            if not notebook:
-                raise HTTPException(status_code=404, detail="Notebook not found")
             notes = await notebook.get_notes()
         else:
             # Get all notes
@@ -41,6 +43,10 @@ async def get_notes(
         ]
     except HTTPException:
         raise
+    except NotFoundError:
+        raise HTTPException(status_code=404, detail="Notebook not found")
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error fetching notes: {str(e)}")
         raise HTTPException(status_code=500, detail=f"Error fetching notes: {str(e)}")
@@ -56,8 +62,10 @@ async def create_note(note_data: NoteCreate):
             from open_notebook.graphs.prompt import graph as prompt_graph
 
             prompt = "Based on the Note below, please provide a Title for this content, with max 15 words"
-            result = await prompt_graph.ainvoke(
-                {  # type: ignore[arg-type]
+            # LangGraph accepts a partial state dict at runtime, but its typed
+            # overloads require the full state type (langgraph typing limitation).
+            result = await prompt_graph.ainvoke(  # type: ignore[call-overload]
+                {
                     "input_text": note_data.content,
                     "prompt": prompt,
                 }
@@ -84,9 +92,8 @@ async def create_note(note_data: NoteCreate):
         if note_data.notebook_id:
             from open_notebook.domain.notebook import Notebook
 
-            notebook = await Notebook.get(note_data.notebook_id)
-            if not notebook:
-                raise HTTPException(status_code=404, detail="Notebook not found")
+            # Verify the notebook exists (raises NotFoundError -> 404)
+            await Notebook.get(note_data.notebook_id)
             await new_note.add_to_notebook(note_data.notebook_id)
 
         return NoteResponse(
@@ -100,8 +107,12 @@ async def create_note(note_data: NoteCreate):
         )
     except HTTPException:
         raise
+    except NotFoundError:
+        raise HTTPException(status_code=404, detail="Notebook not found")
     except InvalidInputError as e:
         raise HTTPException(status_code=400, detail=str(e))
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error creating note: {str(e)}")
         raise HTTPException(status_code=500, detail=f"Error creating note: {str(e)}")
@@ -112,8 +123,6 @@ async def get_note(note_id: str):
     """Get a specific note by ID."""
     try:
         note = await Note.get(note_id)
-        if not note:
-            raise HTTPException(status_code=404, detail="Note not found")
 
         return NoteResponse(
             id=note.id or "",
@@ -125,6 +134,10 @@ async def get_note(note_id: str):
         )
     except HTTPException:
         raise
+    except NotFoundError:
+        raise HTTPException(status_code=404, detail="Note not found")
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error fetching note {note_id}: {str(e)}")
         raise HTTPException(status_code=500, detail=f"Error fetching note: {str(e)}")
@@ -135,8 +148,6 @@ async def update_note(note_id: str, note_update: NoteUpdate):
     """Update a note."""
     try:
         note = await Note.get(note_id)
-        if not note:
-            raise HTTPException(status_code=404, detail="Note not found")
 
         # Update only provided fields
         if note_update.title is not None:
@@ -164,8 +175,12 @@ async def update_note(note_id: str, note_update: NoteUpdate):
         )
     except HTTPException:
         raise
+    except NotFoundError:
+        raise HTTPException(status_code=404, detail="Note not found")
     except InvalidInputError as e:
         raise HTTPException(status_code=400, detail=str(e))
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error updating note {note_id}: {str(e)}")
         raise HTTPException(status_code=500, detail=f"Error updating note: {str(e)}")
@@ -176,14 +191,16 @@ async def delete_note(note_id: str):
     """Delete a note."""
     try:
         note = await Note.get(note_id)
-        if not note:
-            raise HTTPException(status_code=404, detail="Note not found")
 
         await note.delete()
 
         return {"message": "Note deleted successfully"}
     except HTTPException:
         raise
+    except NotFoundError:
+        raise HTTPException(status_code=404, detail="Note not found")
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error deleting note {note_id}: {str(e)}")
         raise HTTPException(status_code=500, detail=f"Error deleting note: {str(e)}")

api/routers/podcasts.py

@@ -1,6 +1,4 @@
-from pathlib import Path
 from typing import List, Optional
-from urllib.parse import unquote, urlparse
 
 from fastapi import APIRouter, HTTPException
 from fastapi.responses import FileResponse
@@ -12,9 +10,101 @@ from api.podcast_service import (
     PodcastGenerationResponse,
     PodcastService,
 )
+from open_notebook.ai.models import Model
+from open_notebook.exceptions import OpenNotebookError
+from open_notebook.podcasts.audio_paths import resolve_contained_audio_path
+from open_notebook.podcasts.models import PodcastEpisode
 
 router = APIRouter()
 
+# Model reference fields stored in the denormalized profile snapshots on an
+# episode, mapped to the resolved display fields the frontend renders
+# ("provider / name" rows in EpisodeCard). Mirrors the speaker_config ->
+# speaker_config_name precedent in api/routers/episode_profiles.py.
+_EPISODE_PROFILE_MODEL_FIELDS = {
+    "outline_llm": ("outline_model_provider", "outline_model_name"),
+    "transcript_llm": ("transcript_model_provider", "transcript_model_name"),
+}
+_SPEAKER_PROFILE_MODEL_FIELDS = {
+    "voice_model": ("voice_model_provider", "voice_model_name"),
+}
+
+
+def _collect_snapshot_model_ids(episodes: List[PodcastEpisode]) -> List[str]:
+    """Collect the distinct model record IDs referenced by episode snapshots."""
+    ids = set()
+    for episode in episodes:
+        for field in _EPISODE_PROFILE_MODEL_FIELDS:
+            ref = (episode.episode_profile or {}).get(field)
+            if ref:
+                ids.add(str(ref))
+        for field in _SPEAKER_PROFILE_MODEL_FIELDS:
+            ref = (episode.speaker_profile or {}).get(field)
+            if ref:
+                ids.add(str(ref))
+    return sorted(ids)
+
+
+def _with_resolved_model_fields(
+    snapshot: dict,
+    field_map: dict,
+    models_by_id: dict,
+) -> dict:
+    """Return a copy of a profile snapshot with resolved model display fields.
+
+    Only sets the display fields when the reference resolves; unresolvable
+    references (deleted model) and legacy snapshots without references are
+    left untouched so the frontend can fall back to the historical
+    provider/model strings, then to a placeholder.
+    """
+    enriched = dict(snapshot or {})
+    for ref_field, (provider_field, name_field) in field_map.items():
+        ref = enriched.get(ref_field)
+        info = models_by_id.get(str(ref)) if ref else None
+        if info:
+            enriched[provider_field] = info["provider"]
+            enriched[name_field] = info["name"]
+    return enriched
+
+
+async def _resolve_snapshot_models(
+    episodes: List[PodcastEpisode],
+) -> dict:
+    """Batch-resolve every model reference in the episodes' snapshots.
+
+    One query for the whole list (see Model.get_display_info_for_ids) - a
+    failure degrades to no resolved fields rather than failing the request.
+    """
+    try:
+        return await Model.get_display_info_for_ids(
+            _collect_snapshot_model_ids(episodes)
+        )
+    except Exception as e:
+        logger.warning(f"Error batch-resolving snapshot model references: {str(e)}")
+        return {}
+
+
+def _delete_episode_audio(episode: PodcastEpisode, episode_id: str) -> None:
+    """Best-effort unlink of an episode's audio file, refusing invalid paths.
+
+    Shared by the delete and retry endpoints. Legacy/escaping audio_file
+    values (resolve_contained_audio_path -> None) are logged and skipped.
+    """
+    if not episode.audio_file:
+        return
+    audio_path = resolve_contained_audio_path(episode.audio_file)
+    if audio_path is None:
+        logger.warning(
+            f"Refusing to delete audio file outside podcasts directory "
+            f"for episode {episode_id}: {episode.audio_file}"
+        )
+    elif audio_path.exists():
+        try:
+            audio_path.unlink()
+            logger.info(f"Deleted audio file: {audio_path}")
+        except Exception as e:
+            logger.warning(f"Failed to delete audio file {audio_path}: {e}")
+
 
 class PodcastEpisodeResponse(BaseModel):
     id: str
@@ -31,13 +121,6 @@ class PodcastEpisodeResponse(BaseModel):
     error_message: Optional[str] = None
 
 
-def _resolve_audio_path(audio_file: str) -> Path:
-    if audio_file.startswith("file://"):
-        parsed = urlparse(audio_file)
-        return Path(unquote(parsed.path))
-    return Path(audio_file)
-
-
 @router.post("/podcasts/generate", response_model=PodcastGenerationResponse)
 async def generate_podcast(request: PodcastGenerationRequest):
     """
@@ -62,11 +145,13 @@ async def generate_podcast(request: PodcastGenerationRequest):
             episode_name=request.episode_name,
         )
 
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error generating podcast: {str(e)}")
-        raise HTTPException(
-            status_code=500, detail="Failed to generate podcast"
-        )
+        raise HTTPException(status_code=500, detail="Failed to generate podcast")
 
 
 @router.get("/podcasts/jobs/{job_id}")
@@ -76,11 +161,13 @@ async def get_podcast_job_status(job_id: str):
         status_data = await PodcastService.get_job_status(job_id)
         return status_data
 
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error fetching podcast job status: {str(e)}")
-        raise HTTPException(
-            status_code=500, detail="Failed to fetch job status"
-        )
+        raise HTTPException(status_code=500, detail="Failed to fetch job status")
 
 
 @router.get("/podcasts/episodes", response_model=List[PodcastEpisodeResponse])
@@ -89,6 +176,22 @@ async def list_podcast_episodes():
     try:
         episodes = await PodcastService.list_episodes()
 
+        # Batch-fetch job status for every episode with a command in one
+        # query instead of one round trip per episode (see
+        # PodcastEpisode.get_job_details_for_commands docstring).
+        try:
+            details_by_command = await PodcastEpisode.get_job_details_for_commands(
+                [episode.command for episode in episodes if episode.command]
+            )
+        except Exception as e:
+            logger.warning(f"Error batch-fetching podcast job statuses: {str(e)}")
+            details_by_command = {}
+
+        # Batch-resolve the snapshots' model references (outline_llm,
+        # transcript_llm, voice_model) to display fields in one query
+        # instead of one lookup per episode.
+        models_by_id = await _resolve_snapshot_models(episodes)
+
         response_episodes = []
         for episode in episodes:
             # Skip incomplete episodes without command or audio
@@ -99,28 +202,35 @@ async def list_podcast_episodes():
             job_status = None
             error_message = None
             if episode.command:
-                try:
-                    detail = await episode.get_job_detail()
+                detail = details_by_command.get(str(episode.command))
+                if detail is not None:
                     job_status = detail["status"]
                     error_message = detail["error_message"]
-                except Exception:
+                else:
                     job_status = "unknown"
             else:
                 # No command but has audio file = completed import
                 job_status = "completed"
 
             audio_url = None
-            if episode.audio_file:
-                audio_path = _resolve_audio_path(episode.audio_file)
-                if audio_path.exists():
-                    audio_url = f"/api/podcasts/episodes/{episode.id}/audio"
+            audio_path = resolve_contained_audio_path(episode.audio_file)
+            if audio_path is not None and audio_path.exists():
+                audio_url = f"/api/podcasts/episodes/{episode.id}/audio"
 
             response_episodes.append(
                 PodcastEpisodeResponse(
                     id=str(episode.id),
                     name=episode.name,
-                    episode_profile=episode.episode_profile,
-                    speaker_profile=episode.speaker_profile,
+                    episode_profile=_with_resolved_model_fields(
+                        episode.episode_profile,
+                        _EPISODE_PROFILE_MODEL_FIELDS,
+                        models_by_id,
+                    ),
+                    speaker_profile=_with_resolved_model_fields(
+                        episode.speaker_profile,
+                        _SPEAKER_PROFILE_MODEL_FIELDS,
+                        models_by_id,
+                    ),
                     briefing=episode.briefing,
                     audio_file=episode.audio_file,
                     audio_url=audio_url,
@@ -134,11 +244,13 @@ async def list_podcast_episodes():
 
         return response_episodes
 
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error listing podcast episodes: {str(e)}")
-        raise HTTPException(
-            status_code=500, detail="Failed to list podcast episodes"
-        )
+        raise HTTPException(status_code=500, detail="Failed to list podcast episodes")
 
 
 @router.get("/podcasts/episodes/{episode_id}", response_model=PodcastEpisodeResponse)
@@ -162,16 +274,25 @@ async def get_podcast_episode(episode_id: str):
             job_status = "completed" if episode.audio_file else "unknown"
 
         audio_url = None
-        if episode.audio_file:
-            audio_path = _resolve_audio_path(episode.audio_file)
-            if audio_path.exists():
-                audio_url = f"/api/podcasts/episodes/{episode.id}/audio"
+        audio_path = resolve_contained_audio_path(episode.audio_file)
+        if audio_path is not None and audio_path.exists():
+            audio_url = f"/api/podcasts/episodes/{episode.id}/audio"
+
+        models_by_id = await _resolve_snapshot_models([episode])
 
         return PodcastEpisodeResponse(
             id=str(episode.id),
             name=episode.name,
-            episode_profile=episode.episode_profile,
-            speaker_profile=episode.speaker_profile,
+            episode_profile=_with_resolved_model_fields(
+                episode.episode_profile,
+                _EPISODE_PROFILE_MODEL_FIELDS,
+                models_by_id,
+            ),
+            speaker_profile=_with_resolved_model_fields(
+                episode.speaker_profile,
+                _SPEAKER_PROFILE_MODEL_FIELDS,
+                models_by_id,
+            ),
             briefing=episode.briefing,
             audio_file=episode.audio_file,
             audio_url=audio_url,
@@ -182,6 +303,10 @@ async def get_podcast_episode(episode_id: str):
             error_message=error_message,
         )
 
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error fetching podcast episode: {str(e)}")
         raise HTTPException(status_code=404, detail="Episode not found")
@@ -194,6 +319,8 @@ async def stream_podcast_episode_audio(episode_id: str):
         episode = await PodcastService.get_episode(episode_id)
     except HTTPException:
         raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error fetching podcast episode for audio: {str(e)}")
         raise HTTPException(status_code=404, detail="Episode not found")
@@ -201,7 +328,14 @@ async def stream_podcast_episode_audio(episode_id: str):
     if not episode.audio_file:
         raise HTTPException(status_code=404, detail="Episode has no audio file")
 
-    audio_path = _resolve_audio_path(episode.audio_file)
+    audio_path = resolve_contained_audio_path(episode.audio_file)
+    if audio_path is None:
+        logger.warning(
+            f"Blocked audio access outside podcasts directory for episode "
+            f"{episode_id}: {episode.audio_file}"
+        )
+        raise HTTPException(status_code=403, detail="Access to file denied")
+
     if not audio_path.exists():
         raise HTTPException(status_code=404, detail="Audio file not found on disk")
 
@@ -239,13 +373,7 @@ async def retry_podcast_episode(episode_id: str):
             )
 
         # Delete audio file if any
-        if episode.audio_file:
-            audio_path = _resolve_audio_path(episode.audio_file)
-            if audio_path.exists():
-                try:
-                    audio_path.unlink()
-                except Exception as e:
-                    logger.warning(f"Failed to delete audio file {audio_path}: {e}")
+        _delete_episode_audio(episode, episode_id)
 
         # Delete the failed episode
         await episode.delete()
@@ -262,11 +390,11 @@ async def retry_podcast_episode(episode_id: str):
 
     except HTTPException:
         raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error retrying podcast episode: {str(e)}")
-        raise HTTPException(
-            status_code=500, detail="Failed to retry episode"
-        )
+        raise HTTPException(status_code=500, detail="Failed to retry episode")
 
 
 @router.delete("/podcasts/episodes/{episode_id}")
@@ -277,14 +405,7 @@ async def delete_podcast_episode(episode_id: str):
         episode = await PodcastService.get_episode(episode_id)
 
         # Delete the physical audio file if it exists
-        if episode.audio_file:
-            audio_path = _resolve_audio_path(episode.audio_file)
-            if audio_path.exists():
-                try:
-                    audio_path.unlink()
-                    logger.info(f"Deleted audio file: {audio_path}")
-                except Exception as e:
-                    logger.warning(f"Failed to delete audio file {audio_path}: {e}")
+        _delete_episode_audio(episode, episode_id)
 
         # Delete the episode from the database
         await episode.delete()
@@ -292,8 +413,10 @@ async def delete_podcast_episode(episode_id: str):
         logger.info(f"Deleted podcast episode: {episode_id}")
         return {"message": "Episode deleted successfully", "episode_id": episode_id}
 
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error deleting podcast episode: {str(e)}")
-        raise HTTPException(
-            status_code=500, detail="Failed to delete episode"
-        )
+        raise HTTPException(status_code=500, detail="Failed to delete episode")

api/routers/providers.py

@@ -0,0 +1,35 @@
+"""
+Providers Router
+
+Exposes the provider registry (open_notebook/ai/provider_registry.py) so
+clients can enumerate supported providers and their metadata instead of
+keeping their own copies.
+
+Endpoints:
+- GET /providers - List all supported providers with metadata
+"""
+
+from typing import List
+
+from fastapi import APIRouter
+
+from api.credentials_service import check_env_configured
+from api.models import ProviderInfoResponse
+from open_notebook.ai.provider_registry import PROVIDERS
+
+router = APIRouter(prefix="/providers", tags=["providers"])
+
+
+@router.get("", response_model=List[ProviderInfoResponse])
+async def list_providers():
+    """List all supported AI providers with their registry metadata."""
+    return [
+        ProviderInfoResponse(
+            name=spec.name,
+            display_name=spec.display_name,
+            modalities=list(spec.modalities),
+            docs_url=spec.docs_url,
+            env_configured=check_env_configured(spec.name),
+        )
+        for spec in PROVIDERS.values()
+    ]

api/routers/search.py

@@ -1,5 +1,5 @@
 import json
-from typing import AsyncGenerator
+from typing import AsyncGenerator, List
 
 from fastapi import APIRouter, HTTPException
 from fastapi.responses import StreamingResponse
@@ -7,8 +7,16 @@ from loguru import logger
 
 from api.models import AskRequest, AskResponse, SearchRequest, SearchResponse
 from open_notebook.ai.models import Model, model_manager
-from open_notebook.domain.notebook import text_search, vector_search
-from open_notebook.exceptions import DatabaseOperationError, InvalidInputError
+from open_notebook.domain.notebook import (
+    resolve_notebook_scope,
+    text_search,
+    vector_search,
+)
+from open_notebook.exceptions import (
+    DatabaseOperationError,
+    InvalidInputError,
+    OpenNotebookError,
+)
 from open_notebook.graphs.ask import graph as ask_graph
 
 router = APIRouter()
@@ -18,6 +26,8 @@ router = APIRouter()
 async def search_knowledge_base(search_request: SearchRequest):
     """Search the knowledge base using text or vector search."""
     try:
+        notebook_ids = await resolve_notebook_scope(search_request.scope_notebook_ids)
+
         if search_request.type == "vector":
             # Check if embedding model is available for vector search
             if not await model_manager.get_embedding_model():
@@ -32,6 +42,7 @@ async def search_knowledge_base(search_request: SearchRequest):
                 source=search_request.search_sources,
                 note=search_request.search_notes,
                 minimum_score=search_request.minimum_score,
+                notebook_ids=notebook_ids,
             )
         else:
             # Text search
@@ -40,6 +51,7 @@ async def search_knowledge_base(search_request: SearchRequest):
                 results=search_request.limit,
                 source=search_request.search_sources,
                 note=search_request.search_notes,
+                notebook_ids=notebook_ids,
             )
 
         return SearchResponse(
@@ -53,20 +65,30 @@ async def search_knowledge_base(search_request: SearchRequest):
     except DatabaseOperationError as e:
         logger.error(f"Database error during search: {str(e)}")
         raise HTTPException(status_code=500, detail=f"Search failed: {str(e)}")
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Unexpected error during search: {str(e)}")
         raise HTTPException(status_code=500, detail=f"Search failed: {str(e)}")
 
 
 async def stream_ask_response(
-    question: str, strategy_model: Model, answer_model: Model, final_answer_model: Model
+    question: str,
+    strategy_model: Model,
+    answer_model: Model,
+    final_answer_model: Model,
+    notebook_ids: List[str],
 ) -> AsyncGenerator[str, None]:
     """Stream the ask response as Server-Sent Events."""
     try:
         final_answer = None
 
-        async for chunk in ask_graph.astream(
-            input=dict(question=question),  # type: ignore[arg-type]
+        # LangGraph accepts a partial state dict at runtime, but its typed
+        # overloads require the full state type (langgraph typing limitation).
+        async for chunk in ask_graph.astream(  # type: ignore[call-overload]
+            input=dict(question=question, notebook_ids=notebook_ids),
             config=dict(
                 configurable=dict(
                     strategy_model=strategy_model.id,
@@ -104,7 +126,12 @@ async def stream_ask_response(
     except Exception as e:
         from open_notebook.utils.error_classifier import classify_error
 
-        _, user_message = classify_error(e)
+        # Typed errors already carry a user-facing message; only raw provider
+        # exceptions need classifying.
+        if isinstance(e, OpenNotebookError):
+            user_message = str(e)
+        else:
+            _, user_message = classify_error(e)
         logger.error(f"Error in ask streaming: {str(e)}")
         error_data = {"type": "error", "message": user_message}
         yield f"data: {json.dumps(error_data)}\n\n"
@@ -114,6 +141,10 @@ async def stream_ask_response(
 async def ask_knowledge_base(ask_request: AskRequest):
     """Ask the knowledge base a question using AI models."""
     try:
+        # Cheapest check first: a malformed or unknown scope fails before any
+        # model lookup or embedding check can mask it.
+        notebook_ids = await resolve_notebook_scope(ask_request.scope_notebook_ids)
+
         # Validate models exist
         strategy_model = await Model.get(ask_request.strategy_model)
         answer_model = await Model.get(ask_request.answer_model)
@@ -145,13 +176,24 @@ async def ask_knowledge_base(ask_request: AskRequest):
         # For streaming response
         return StreamingResponse(
             stream_ask_response(
-                ask_request.question, strategy_model, answer_model, final_answer_model
+                ask_request.question,
+                strategy_model,
+                answer_model,
+                final_answer_model,
+                notebook_ids,
             ),
-            media_type="text/plain",
+            media_type="text/event-stream",
+            headers={
+                "Cache-Control": "no-cache",
+                "Connection": "keep-alive",
+                "X-Accel-Buffering": "no",
+            },
         )
 
     except HTTPException:
         raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error in ask endpoint: {str(e)}")
         raise HTTPException(status_code=500, detail=f"Ask operation failed: {str(e)}")
@@ -161,6 +203,10 @@ async def ask_knowledge_base(ask_request: AskRequest):
 async def ask_knowledge_base_simple(ask_request: AskRequest):
     """Ask the knowledge base a question and return a simple response (non-streaming)."""
     try:
+        # Cheapest check first: a malformed or unknown scope fails before any
+        # model lookup or embedding check can mask it.
+        notebook_ids = await resolve_notebook_scope(ask_request.scope_notebook_ids)
+
         # Validate models exist
         strategy_model = await Model.get(ask_request.strategy_model)
         answer_model = await Model.get(ask_request.answer_model)
@@ -191,8 +237,10 @@ async def ask_knowledge_base_simple(ask_request: AskRequest):
 
         # Run the ask graph and get final result
         final_answer = None
-        async for chunk in ask_graph.astream(
-            input=dict(question=ask_request.question),  # type: ignore[arg-type]
+        # LangGraph accepts a partial state dict at runtime, but its typed
+        # overloads require the full state type (langgraph typing limitation).
+        async for chunk in ask_graph.astream(  # type: ignore[call-overload]
+            input=dict(question=ask_request.question, notebook_ids=notebook_ids),
             config=dict(
                 configurable=dict(
                     strategy_model=strategy_model.id,
@@ -212,6 +260,8 @@ async def ask_knowledge_base_simple(ask_request: AskRequest):
 
     except HTTPException:
         raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error in ask simple endpoint: {str(e)}")
         raise HTTPException(status_code=500, detail=f"Ask operation failed: {str(e)}")

api/routers/settings.py

@@ -3,7 +3,10 @@ from loguru import logger
 
 from api.models import SettingsResponse, SettingsUpdate
 from open_notebook.domain.content_settings import ContentSettings
-from open_notebook.exceptions import InvalidInputError
+from open_notebook.exceptions import (
+    InvalidInputError,
+    OpenNotebookError,
+)
 
 router = APIRouter()
 
@@ -19,13 +22,18 @@ async def get_settings():
             default_content_processing_engine_url=settings.default_content_processing_engine_url,
             default_embedding_option=settings.default_embedding_option,
             auto_delete_files=settings.auto_delete_files,
+            docling_ocr=settings.docling_ocr,
+            docling_formulas=settings.docling_formulas,
+            docling_vision=settings.docling_vision,
             youtube_preferred_languages=settings.youtube_preferred_languages,
         )
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error fetching settings: {str(e)}")
-        raise HTTPException(
-            status_code=500, detail="Error fetching settings"
-        )
+        raise HTTPException(status_code=500, detail="Error fetching settings")
 
 
 @router.put("/settings", response_model=SettingsResponse)
@@ -47,7 +55,7 @@ async def update_settings(settings_update: SettingsUpdate):
             from typing import Literal, cast
 
             settings.default_content_processing_engine_url = cast(
-                Literal["auto", "firecrawl", "jina", "simple"],
+                Literal["auto", "firecrawl", "jina", "crawl4ai", "simple"],
                 settings_update.default_content_processing_engine_url,
             )
         if settings_update.default_embedding_option is not None:
@@ -63,6 +71,12 @@ async def update_settings(settings_update: SettingsUpdate):
             settings.auto_delete_files = cast(
                 Literal["yes", "no"], settings_update.auto_delete_files
             )
+        if settings_update.docling_ocr is not None:
+            settings.docling_ocr = settings_update.docling_ocr
+        if settings_update.docling_formulas is not None:
+            settings.docling_formulas = settings_update.docling_formulas
+        if settings_update.docling_vision is not None:
+            settings.docling_vision = settings_update.docling_vision
         if settings_update.youtube_preferred_languages is not None:
             settings.youtube_preferred_languages = (
                 settings_update.youtube_preferred_languages
@@ -75,14 +89,17 @@ async def update_settings(settings_update: SettingsUpdate):
             default_content_processing_engine_url=settings.default_content_processing_engine_url,
             default_embedding_option=settings.default_embedding_option,
             auto_delete_files=settings.auto_delete_files,
+            docling_ocr=settings.docling_ocr,
+            docling_formulas=settings.docling_formulas,
+            docling_vision=settings.docling_vision,
             youtube_preferred_languages=settings.youtube_preferred_languages,
         )
     except HTTPException:
         raise
     except InvalidInputError as e:
         raise HTTPException(status_code=400, detail=str(e))
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error updating settings: {str(e)}")
-        raise HTTPException(
-            status_code=500, detail="Error updating settings"
-        )
+        raise HTTPException(status_code=500, detail="Error updating settings")

api/routers/source_chat.py

@@ -1,6 +1,7 @@
 import asyncio
 import json
 from typing import AsyncGenerator, List, Optional
+from uuid import uuid4
 
 from fastapi import APIRouter, HTTPException, Path
 from fastapi.responses import StreamingResponse
@@ -9,13 +10,24 @@ from langchain_core.runnables import RunnableConfig
 from loguru import logger
 from pydantic import BaseModel, Field
 
+from api.routers._chat_shared import (
+    ChatMessage,
+    SuccessResponse,
+    extract_chat_messages,
+    get_source_or_404,
+    get_verified_source_session,
+)
 from open_notebook.database.repository import ensure_record_id, repo_query
-from open_notebook.domain.notebook import ChatSession, Source
+from open_notebook.domain.notebook import ChatSession
 from open_notebook.exceptions import (
     NotFoundError,
+    OpenNotebookError,
 )
 from open_notebook.graphs.source_chat import source_chat_graph as source_chat_graph
-from open_notebook.utils.graph_utils import get_session_message_count
+from open_notebook.utils.graph_utils import (
+    get_session_message_count,
+    invoke_chat_turn,
+)
 
 router = APIRouter()
 
@@ -28,18 +40,13 @@ class CreateSourceChatSessionRequest(BaseModel):
         None, description="Optional model override for this session"
     )
 
+
 class UpdateSourceChatSessionRequest(BaseModel):
     title: Optional[str] = Field(None, description="New session title")
     model_override: Optional[str] = Field(
         None, description="Model override for this session"
     )
 
-class ChatMessage(BaseModel):
-    id: str = Field(..., description="Message ID")
-    type: str = Field(..., description="Message type (human|ai)")
-    content: str = Field(..., description="Message content")
-    timestamp: Optional[str] = Field(None, description="Message timestamp")
-
 
 class ContextIndicator(BaseModel):
     sources: List[str] = Field(
@@ -52,6 +59,7 @@ class ContextIndicator(BaseModel):
         default_factory=list, description="Note IDs used in context"
     )
 
+
 class SourceChatSessionResponse(BaseModel):
     id: str = Field(..., description="Session ID")
     title: str = Field(..., description="Session title")
@@ -65,6 +73,7 @@ class SourceChatSessionResponse(BaseModel):
         None, description="Number of messages in session"
     )
 
+
 class SourceChatSessionWithMessagesResponse(SourceChatSessionResponse):
     messages: List[ChatMessage] = Field(
         default_factory=list, description="Session messages"
@@ -73,16 +82,13 @@ class SourceChatSessionWithMessagesResponse(SourceChatSessionResponse):
         None, description="Context indicators from last response"
     )
 
+
 class SendMessageRequest(BaseModel):
     message: str = Field(..., description="User message content")
     model_override: Optional[str] = Field(
         None, description="Optional model override for this message"
     )
 
-class SuccessResponse(BaseModel):
-    success: bool = Field(True, description="Operation success status")
-    message: str = Field(..., description="Success message")
-
 
 @router.post(
     "/sources/{source_id}/chat/sessions", response_model=SourceChatSessionResponse
@@ -93,13 +99,8 @@ async def create_source_chat_session(
 ):
     """Create a new chat session for a source."""
     try:
-        # Verify source exists
-        full_source_id = (
-            source_id if source_id.startswith("source:") else f"source:{source_id}"
-        )
-        source = await Source.get(full_source_id)
-        if not source:
-            raise HTTPException(status_code=404, detail="Source not found")
+        # Verify source exists (normalizes the ID and 404s if missing)
+        full_source_id, _source = await get_source_or_404(source_id)
 
         # Create new session with model_override support
         session = ChatSession(
@@ -122,6 +123,10 @@ async def create_source_chat_session(
         )
     except NotFoundError:
         raise HTTPException(status_code=404, detail="Source not found")
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error creating source chat session: {str(e)}")
         raise HTTPException(
@@ -135,13 +140,8 @@ async def create_source_chat_session(
 async def get_source_chat_sessions(source_id: str = Path(..., description="Source ID")):
     """Get all chat sessions for a source."""
     try:
-        # Verify source exists
-        full_source_id = (
-            source_id if source_id.startswith("source:") else f"source:{source_id}"
-        )
-        source = await Source.get(full_source_id)
-        if not source:
-            raise HTTPException(status_code=404, detail="Source not found")
+        # Verify source exists (normalizes the ID and 404s if missing)
+        full_source_id, _source = await get_source_or_404(source_id)
 
         # Get sessions that refer to this source - first get relations, then sessions
         relations = await repo_query(
@@ -183,6 +183,10 @@ async def get_source_chat_sessions(source_id: str = Path(..., description="Sourc
         return sessions
     except NotFoundError:
         raise HTTPException(status_code=404, detail="Source not found")
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error fetching source chat sessions: {str(e)}")
         raise HTTPException(
@@ -200,37 +204,13 @@ async def get_source_chat_session(
 ):
     """Get a specific source chat session with its messages."""
     try:
-        # Verify source exists
-        full_source_id = (
-            source_id if source_id.startswith("source:") else f"source:{source_id}"
-        )
-        source = await Source.get(full_source_id)
-        if not source:
-            raise HTTPException(status_code=404, detail="Source not found")
-
-        # Get session
-        full_session_id = (
-            session_id
-            if session_id.startswith("chat_session:")
-            else f"chat_session:{session_id}"
-        )
-        session = await ChatSession.get(full_session_id)
-        if not session:
-            raise HTTPException(status_code=404, detail="Session not found")
-
-        # Verify session is related to this source
-        relation_query = await repo_query(
-            "SELECT * FROM refers_to WHERE in = $session_id AND out = $source_id",
-            {
-                "session_id": ensure_record_id(full_session_id),
-                "source_id": ensure_record_id(full_source_id),
-            },
-        )
-
-        if not relation_query:
-            raise HTTPException(
-                status_code=404, detail="Session not found for this source"
-            )
+        # Verify source + session exist and are related (404s otherwise)
+        (
+            _full_source_id,
+            _source,
+            full_session_id,
+            session,
+        ) = await get_verified_source_session(source_id, session_id)
 
         # Get session state from LangGraph to retrieve messages
         # Use sync get_state() in a thread since SqliteSaver doesn't support async
@@ -246,17 +226,7 @@ async def get_source_chat_session(
         if thread_state and thread_state.values:
             # Extract messages
             if "messages" in thread_state.values:
-                for msg in thread_state.values["messages"]:
-                    messages.append(
-                        ChatMessage(
-                            id=getattr(msg, "id", f"msg_{len(messages)}"),
-                            type=msg.type if hasattr(msg, "type") else "unknown",
-                            content=msg.content
-                            if hasattr(msg, "content")
-                            else str(msg),
-                            timestamp=None,  # LangChain messages don't have timestamps by default
-                        )
-                    )
+                messages = extract_chat_messages(thread_state.values["messages"])
 
             # Extract context indicators from the last state
             if "context_indicators" in thread_state.values:
@@ -280,6 +250,10 @@ async def get_source_chat_session(
         )
     except NotFoundError:
         raise HTTPException(status_code=404, detail="Source or session not found")
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error fetching source chat session: {str(e)}")
         raise HTTPException(
@@ -298,37 +272,13 @@ async def update_source_chat_session(
 ):
     """Update source chat session title and/or model override."""
     try:
-        # Verify source exists
-        full_source_id = (
-            source_id if source_id.startswith("source:") else f"source:{source_id}"
-        )
-        source = await Source.get(full_source_id)
-        if not source:
-            raise HTTPException(status_code=404, detail="Source not found")
-
-        # Get session
-        full_session_id = (
-            session_id
-            if session_id.startswith("chat_session:")
-            else f"chat_session:{session_id}"
-        )
-        session = await ChatSession.get(full_session_id)
-        if not session:
-            raise HTTPException(status_code=404, detail="Session not found")
-
-        # Verify session is related to this source
-        relation_query = await repo_query(
-            "SELECT * FROM refers_to WHERE in = $session_id AND out = $source_id",
-            {
-                "session_id": ensure_record_id(full_session_id),
-                "source_id": ensure_record_id(full_source_id),
-            },
-        )
-
-        if not relation_query:
-            raise HTTPException(
-                status_code=404, detail="Session not found for this source"
-            )
+        # Verify source + session exist and are related (404s otherwise)
+        (
+            _full_source_id,
+            _source,
+            full_session_id,
+            session,
+        ) = await get_verified_source_session(source_id, session_id)
 
         # Update session fields
         if request.title is not None:
@@ -352,6 +302,10 @@ async def update_source_chat_session(
         )
     except NotFoundError:
         raise HTTPException(status_code=404, detail="Source or session not found")
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error updating source chat session: {str(e)}")
         raise HTTPException(
@@ -368,37 +322,13 @@ async def delete_source_chat_session(
 ):
     """Delete a source chat session."""
     try:
-        # Verify source exists
-        full_source_id = (
-            source_id if source_id.startswith("source:") else f"source:{source_id}"
-        )
-        source = await Source.get(full_source_id)
-        if not source:
-            raise HTTPException(status_code=404, detail="Source not found")
-
-        # Get session
-        full_session_id = (
-            session_id
-            if session_id.startswith("chat_session:")
-            else f"chat_session:{session_id}"
-        )
-        session = await ChatSession.get(full_session_id)
-        if not session:
-            raise HTTPException(status_code=404, detail="Session not found")
-
-        # Verify session is related to this source
-        relation_query = await repo_query(
-            "SELECT * FROM refers_to WHERE in = $session_id AND out = $source_id",
-            {
-                "session_id": ensure_record_id(full_session_id),
-                "source_id": ensure_record_id(full_source_id),
-            },
-        )
-
-        if not relation_query:
-            raise HTTPException(
-                status_code=404, detail="Session not found for this source"
-            )
+        # Verify source + session exist and are related (404s otherwise)
+        (
+            _full_source_id,
+            _source,
+            full_session_id,
+            session,
+        ) = await get_verified_source_session(source_id, session_id)
 
         await session.delete()
 
@@ -407,6 +337,10 @@ async def delete_source_chat_session(
         )
     except NotFoundError:
         raise HTTPException(status_code=404, detail="Source or session not found")
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error deleting source chat session: {str(e)}")
         raise HTTPException(
@@ -433,31 +367,41 @@ async def stream_source_chat_response(
         state_values["model_override"] = model_override
 
         # Add user message to state
-        user_message = HumanMessage(content=message)
+        # Explicit id so a failed turn can remove it from the checkpoint.
+        user_message = HumanMessage(content=message, id=str(uuid4()))
         state_values["messages"].append(user_message)
 
         # Send user message event
         user_event = {"type": "user_message", "content": message, "timestamp": None}
         yield f"data: {json.dumps(user_event)}\n\n"
 
-        # Execute source chat graph synchronously (like notebook chat does)
-        result = source_chat_graph.invoke(
-            input=state_values,  # type: ignore[arg-type]
-            config=RunnableConfig(
+        # Run the synchronous LangGraph invoke in a thread so it doesn't block the
+        # event loop. While blocked, even the already-yielded SSE events can't
+        # flush and every other request stalls until the LLM finishes. Mirrors the
+        # get_state() calls above.
+        # invoke_chat_turn also drops the question from the checkpoint when the
+        # turn fails, so a retry doesn't add it twice.
+        result = await asyncio.to_thread(
+            invoke_chat_turn,
+            source_chat_graph,
+            state_values,
+            RunnableConfig(
                 configurable={"thread_id": session_id, "model_id": model_override}
             ),
+            user_message,
         )
 
-        # Stream the complete AI response
-        if "messages" in result:
-            for msg in result["messages"]:
-                if hasattr(msg, "type") and msg.type == "ai":
-                    ai_event = {
-                        "type": "ai_message",
-                        "content": msg.content if hasattr(msg, "content") else str(msg),
-                        "timestamp": None,
-                    }
-                    yield f"data: {json.dumps(ai_event)}\n\n"
+        # Stream this turn's AI response. result["messages"] is the full
+        # checkpointed history, so only the last message is new.
+        if result.get("messages"):
+            msg = result["messages"][-1]
+            if getattr(msg, "type", None) == "ai":
+                ai_event = {
+                    "type": "ai_message",
+                    "content": msg.content if hasattr(msg, "content") else str(msg),
+                    "timestamp": None,
+                }
+                yield f"data: {json.dumps(ai_event)}\n\n"
 
         # Stream context indicators
         if "context_indicators" in result:
@@ -474,9 +418,14 @@ async def stream_source_chat_response(
     except Exception as e:
         from open_notebook.utils.error_classifier import classify_error
 
-        _, user_message = classify_error(e)
+        # Typed errors already carry a user-facing message; only raw provider
+        # exceptions need classifying.
+        if isinstance(e, OpenNotebookError):
+            error_message = str(e)
+        else:
+            _, error_message = classify_error(e)
         logger.error(f"Error in source chat streaming: {str(e)}")
-        error_event = {"type": "error", "message": user_message}
+        error_event = {"type": "error", "message": error_message}
         yield f"data: {json.dumps(error_event)}\n\n"
 
 
@@ -488,37 +437,13 @@ async def send_message_to_source_chat(
 ):
     """Send a message to source chat session with SSE streaming response."""
     try:
-        # Verify source exists
-        full_source_id = (
-            source_id if source_id.startswith("source:") else f"source:{source_id}"
-        )
-        source = await Source.get(full_source_id)
-        if not source:
-            raise HTTPException(status_code=404, detail="Source not found")
-
-        # Verify session exists and is related to source
-        full_session_id = (
-            session_id
-            if session_id.startswith("chat_session:")
-            else f"chat_session:{session_id}"
-        )
-        session = await ChatSession.get(full_session_id)
-        if not session:
-            raise HTTPException(status_code=404, detail="Session not found")
-
-        # Verify session is related to this source
-        relation_query = await repo_query(
-            "SELECT * FROM refers_to WHERE in = $session_id AND out = $source_id",
-            {
-                "session_id": ensure_record_id(full_session_id),
-                "source_id": ensure_record_id(full_source_id),
-            },
-        )
-
-        if not relation_query:
-            raise HTTPException(
-                status_code=404, detail="Session not found for this source"
-            )
+        # Verify source + session exist and are related (404s otherwise)
+        (
+            full_source_id,
+            _source,
+            full_session_id,
+            session,
+        ) = await get_verified_source_session(source_id, session_id)
 
         if not request.message:
             raise HTTPException(status_code=400, detail="Message content is required")
@@ -539,16 +464,18 @@ async def send_message_to_source_chat(
                 message=request.message,
                 model_override=model_override,
             ),
-            media_type="text/plain",
+            media_type="text/event-stream",
             headers={
                 "Cache-Control": "no-cache",
                 "Connection": "keep-alive",
-                "Content-Type": "text/plain; charset=utf-8",
+                "X-Accel-Buffering": "no",
             },
         )
 
     except HTTPException:
         raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error sending message to source chat: {str(e)}")
         raise HTTPException(status_code=500, detail=f"Error sending message: {str(e)}")

api/routers/speaker_profiles.py

@@ -4,6 +4,7 @@ from fastapi import APIRouter, HTTPException
 from loguru import logger
 from pydantic import BaseModel, Field
 
+from open_notebook.exceptions import OpenNotebookError
 from open_notebook.podcasts.models import SpeakerProfile
 
 router = APIRouter()
@@ -15,9 +16,6 @@ class SpeakerProfileResponse(BaseModel):
     description: str
     voice_model: Optional[str] = None
     speakers: List[Dict[str, Any]]
-    # Legacy fields (for display/migration awareness)
-    tts_provider: Optional[str] = None
-    tts_model: Optional[str] = None
 
 
 def _profile_to_response(profile: SpeakerProfile) -> SpeakerProfileResponse:
@@ -27,8 +25,6 @@ def _profile_to_response(profile: SpeakerProfile) -> SpeakerProfileResponse:
         description=profile.description or "",
         voice_model=profile.voice_model,
         speakers=profile.speakers,
-        tts_provider=profile.tts_provider,
-        tts_model=profile.tts_model,
     )
 
 
@@ -38,11 +34,13 @@ async def list_speaker_profiles():
     try:
         profiles = await SpeakerProfile.get_all(order_by="name asc")
         return [_profile_to_response(p) for p in profiles]
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Failed to fetch speaker profiles: {e}")
-        raise HTTPException(
-            status_code=500, detail="Failed to fetch speaker profiles"
-        )
+        raise HTTPException(status_code=500, detail="Failed to fetch speaker profiles")
 
 
 @router.get("/speaker-profiles/{profile_name}", response_model=SpeakerProfileResponse)
@@ -60,11 +58,11 @@ async def get_speaker_profile(profile_name: str):
 
     except HTTPException:
         raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Failed to fetch speaker profile '{profile_name}': {e}")
-        raise HTTPException(
-            status_code=500, detail="Failed to fetch speaker profile"
-        )
+        raise HTTPException(status_code=500, detail="Failed to fetch speaker profile")
 
 
 class SpeakerProfileCreate(BaseModel):
@@ -74,9 +72,6 @@ class SpeakerProfileCreate(BaseModel):
     speakers: List[Dict[str, Any]] = Field(
         ..., description="Array of speaker configurations"
     )
-    # Legacy fields (accepted but not required)
-    tts_provider: Optional[str] = None
-    tts_model: Optional[str] = None
 
 
 @router.post("/speaker-profiles", response_model=SpeakerProfileResponse)
@@ -88,18 +83,18 @@ async def create_speaker_profile(profile_data: SpeakerProfileCreate):
             description=profile_data.description,
             voice_model=profile_data.voice_model,
             speakers=profile_data.speakers,
-            tts_provider=profile_data.tts_provider,
-            tts_model=profile_data.tts_model,
         )
 
         await profile.save()
         return _profile_to_response(profile)
 
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Failed to create speaker profile: {e}")
-        raise HTTPException(
-            status_code=500, detail="Failed to create speaker profile"
-        )
+        raise HTTPException(status_code=500, detail="Failed to create speaker profile")
 
 
 @router.put("/speaker-profiles/{profile_id}", response_model=SpeakerProfileResponse)
@@ -113,23 +108,19 @@ async def update_speaker_profile(profile_id: str, profile_data: SpeakerProfileCr
                 status_code=404, detail=f"Speaker profile '{profile_id}' not found"
             )
 
-        profile.name = profile_data.name
-        profile.description = profile_data.description
-        profile.voice_model = profile_data.voice_model
-        profile.speakers = profile_data.speakers
-        profile.tts_provider = profile_data.tts_provider
-        profile.tts_model = profile_data.tts_model
+        for field, value in profile_data.model_dump(exclude_unset=True).items():
+            setattr(profile, field, value)
 
         await profile.save()
         return _profile_to_response(profile)
 
     except HTTPException:
         raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Failed to update speaker profile: {e}")
-        raise HTTPException(
-            status_code=500, detail="Failed to update speaker profile"
-        )
+        raise HTTPException(status_code=500, detail="Failed to update speaker profile")
 
 
 @router.delete("/speaker-profiles/{profile_id}")
@@ -149,11 +140,11 @@ async def delete_speaker_profile(profile_id: str):
 
     except HTTPException:
         raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Failed to delete speaker profile: {e}")
-        raise HTTPException(
-            status_code=500, detail="Failed to delete speaker profile"
-        )
+        raise HTTPException(status_code=500, detail="Failed to delete speaker profile")
 
 
 @router.post(
@@ -174,8 +165,6 @@ async def duplicate_speaker_profile(profile_id: str):
             description=original.description,
             voice_model=original.voice_model,
             speakers=original.speakers,
-            tts_provider=original.tts_provider,
-            tts_model=original.tts_model,
         )
 
         await duplicate.save()
@@ -183,6 +172,8 @@ async def duplicate_speaker_profile(profile_id: str):
 
     except HTTPException:
         raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Failed to duplicate speaker profile: {e}")
         raise HTTPException(

api/routers/transformations.py

@@ -20,6 +20,20 @@ from open_notebook.graphs.transformation import graph as transformation_graph
 router = APIRouter()
 
 
+def _transformation_response(transformation: Transformation) -> TransformationResponse:
+    return TransformationResponse(
+        id=transformation.id or "",
+        name=transformation.name,
+        title=transformation.title,
+        description=transformation.description,
+        prompt=transformation.prompt,
+        apply_default=transformation.apply_default,
+        model_id=transformation.model_id,
+        created=str(transformation.created),
+        updated=str(transformation.updated),
+    )
+
+
 @router.get("/transformations", response_model=List[TransformationResponse])
 async def get_transformations():
     """Get all transformations."""
@@ -27,18 +41,13 @@ async def get_transformations():
         transformations = await Transformation.get_all(order_by="name asc")
 
         return [
-            TransformationResponse(
-                id=transformation.id or "",
-                name=transformation.name,
-                title=transformation.title,
-                description=transformation.description,
-                prompt=transformation.prompt,
-                apply_default=transformation.apply_default,
-                created=str(transformation.created),
-                updated=str(transformation.updated),
-            )
+            _transformation_response(transformation)
             for transformation in transformations
         ]
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error fetching transformations: {str(e)}")
         raise HTTPException(
@@ -50,27 +59,30 @@ async def get_transformations():
 async def create_transformation(transformation_data: TransformationCreate):
     """Create a new transformation."""
     try:
+        # Reject unknown model references up front (same check as execute);
+        # otherwise an invalid model_id is stored and only fails at run time.
+        if transformation_data.model_id:
+            model = await Model.get(transformation_data.model_id)
+            if not model:
+                raise HTTPException(status_code=404, detail="Model not found")
+
         new_transformation = Transformation(
             name=transformation_data.name,
             title=transformation_data.title,
             description=transformation_data.description,
             prompt=transformation_data.prompt,
             apply_default=transformation_data.apply_default,
+            model_id=transformation_data.model_id,
         )
         await new_transformation.save()
 
-        return TransformationResponse(
-            id=new_transformation.id or "",
-            name=new_transformation.name,
-            title=new_transformation.title,
-            description=new_transformation.description,
-            prompt=new_transformation.prompt,
-            apply_default=new_transformation.apply_default,
-            created=str(new_transformation.created),
-            updated=str(new_transformation.updated),
-        )
+        return _transformation_response(new_transformation)
+    except HTTPException:
+        raise
     except InvalidInputError as e:
         raise HTTPException(status_code=400, detail=str(e))
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error creating transformation: {str(e)}")
         raise HTTPException(
@@ -87,24 +99,30 @@ async def execute_transformation(execute_request: TransformationExecuteRequest):
         if not transformation:
             raise HTTPException(status_code=404, detail="Transformation not found")
 
-        # Validate model exists
-        model = await Model.get(execute_request.model_id)
-        if not model:
-            raise HTTPException(status_code=404, detail="Model not found")
+        model_id = execute_request.model_id or transformation.model_id
 
-        # Execute the transformation
-        result = await transformation_graph.ainvoke(
-            dict(  # type: ignore[arg-type]
+        # Validate explicit or transformation-specific model exists.
+        # None is allowed so the graph can use the configured transformation default.
+        if model_id:
+            model = await Model.get(model_id)
+            if not model:
+                raise HTTPException(status_code=404, detail="Model not found")
+
+        # Execute the transformation.
+        # LangGraph accepts a partial state dict at runtime, but its typed
+        # overloads require the full state type (langgraph typing limitation).
+        result = await transformation_graph.ainvoke(  # type: ignore[call-overload]
+            dict(
                 input_text=execute_request.input_text,
                 transformation=transformation,
             ),
-            config=dict(configurable={"model_id": execute_request.model_id}),
+            config=dict(configurable={"model_id": model_id}),
         )
 
         return TransformationExecuteResponse(
             output=result["output"],
             transformation_id=execute_request.transformation_id,
-            model_id=execute_request.model_id,
+            model_id=model_id,
         )
 
     except HTTPException:
@@ -128,6 +146,10 @@ async def get_default_prompt():
             transformation_instructions=default_prompts.transformation_instructions
             or ""
         )
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error fetching default prompt: {str(e)}")
         raise HTTPException(
@@ -149,6 +171,10 @@ async def update_default_prompt(prompt_update: DefaultPromptUpdate):
         return DefaultPromptResponse(
             transformation_instructions=default_prompts.transformation_instructions
         )
+    except HTTPException:
+        raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error updating default prompt: {str(e)}")
         raise HTTPException(
@@ -166,18 +192,11 @@ async def get_transformation(transformation_id: str):
         if not transformation:
             raise HTTPException(status_code=404, detail="Transformation not found")
 
-        return TransformationResponse(
-            id=transformation.id or "",
-            name=transformation.name,
-            title=transformation.title,
-            description=transformation.description,
-            prompt=transformation.prompt,
-            apply_default=transformation.apply_default,
-            created=str(transformation.created),
-            updated=str(transformation.updated),
-        )
+        return _transformation_response(transformation)
     except HTTPException:
         raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error fetching transformation {transformation_id}: {str(e)}")
         raise HTTPException(
@@ -202,29 +221,31 @@ async def update_transformation(
             transformation.name = transformation_update.name
         if transformation_update.title is not None:
             transformation.title = transformation_update.title
-        if transformation_update.description is not None:
-            transformation.description = transformation_update.description
+        # An explicit "" (or null) clears the description; only an absent
+        # field leaves it untouched.
+        if "description" in transformation_update.model_fields_set:
+            transformation.description = transformation_update.description or ""
         if transformation_update.prompt is not None:
             transformation.prompt = transformation_update.prompt
         if transformation_update.apply_default is not None:
             transformation.apply_default = transformation_update.apply_default
+        if "model_id" in transformation_update.model_fields_set:
+            # Validate a newly supplied model reference (allow clearing to None).
+            if transformation_update.model_id:
+                model = await Model.get(transformation_update.model_id)
+                if not model:
+                    raise HTTPException(status_code=404, detail="Model not found")
+            transformation.model_id = transformation_update.model_id
 
         await transformation.save()
 
-        return TransformationResponse(
-            id=transformation.id or "",
-            name=transformation.name,
-            title=transformation.title,
-            description=transformation.description,
-            prompt=transformation.prompt,
-            apply_default=transformation.apply_default,
-            created=str(transformation.created),
-            updated=str(transformation.updated),
-        )
+        return _transformation_response(transformation)
     except HTTPException:
         raise
     except InvalidInputError as e:
         raise HTTPException(status_code=400, detail=str(e))
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error updating transformation {transformation_id}: {str(e)}")
         raise HTTPException(
@@ -245,6 +266,8 @@ async def delete_transformation(transformation_id: str):
         return {"message": "Transformation deleted successfully"}
     except HTTPException:
         raise
+    except OpenNotebookError:
+        raise
     except Exception as e:
         logger.error(f"Error deleting transformation {transformation_id}: {str(e)}")
         raise HTTPException(

api/search_service.py

@@ -1,58 +0,0 @@
-"""
-Search service layer using API.
-"""
-
-from typing import Any, Dict, List, Union
-
-from loguru import logger
-
-from api.client import api_client
-
-
-class SearchService:
-    """Service layer for search operations using API."""
-
-    def __init__(self):
-        logger.info("Using API for search operations")
-
-    def search(
-        self,
-        query: str,
-        search_type: str = "text",
-        limit: int = 100,
-        search_sources: bool = True,
-        search_notes: bool = True,
-        minimum_score: float = 0.2,
-    ) -> List[Dict[str, Any]]:
-        """Search the knowledge base."""
-        response = api_client.search(
-            query=query,
-            search_type=search_type,
-            limit=limit,
-            search_sources=search_sources,
-            search_notes=search_notes,
-            minimum_score=minimum_score,
-        )
-        if isinstance(response, dict):
-            return response.get("results", [])
-        return []
-
-    def ask_knowledge_base(
-        self,
-        question: str,
-        strategy_model: str,
-        answer_model: str,
-        final_answer_model: str,
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Ask the knowledge base a question."""
-        response = api_client.ask_simple(
-            question=question,
-            strategy_model=strategy_model,
-            answer_model=answer_model,
-            final_answer_model=final_answer_model,
-        )
-        return response
-
-
-# Global service instance
-search_service = SearchService()

api/settings_service.py

@@ -1,79 +0,0 @@
-"""
-Settings service layer using API.
-"""
-
-from loguru import logger
-
-from api.client import api_client
-from open_notebook.domain.content_settings import ContentSettings
-
-
-class SettingsService:
-    """Service layer for settings operations using API."""
-
-    def __init__(self):
-        logger.info("Using API for settings operations")
-
-    def get_settings(self) -> ContentSettings:
-        """Get application settings."""
-        settings_response = api_client.get_settings()
-        settings_data = (
-            settings_response
-            if isinstance(settings_response, dict)
-            else settings_response[0]
-        )
-
-        # Create ContentSettings object from API response
-        settings = ContentSettings(
-            default_content_processing_engine_doc=settings_data.get(
-                "default_content_processing_engine_doc"
-            ),
-            default_content_processing_engine_url=settings_data.get(
-                "default_content_processing_engine_url"
-            ),
-            default_embedding_option=settings_data.get("default_embedding_option"),
-            auto_delete_files=settings_data.get("auto_delete_files"),
-            youtube_preferred_languages=settings_data.get(
-                "youtube_preferred_languages"
-            ),
-        )
-
-        return settings
-
-    def update_settings(self, settings: ContentSettings) -> ContentSettings:
-        """Update application settings."""
-        updates = {
-            "default_content_processing_engine_doc": settings.default_content_processing_engine_doc,
-            "default_content_processing_engine_url": settings.default_content_processing_engine_url,
-            "default_embedding_option": settings.default_embedding_option,
-            "auto_delete_files": settings.auto_delete_files,
-            "youtube_preferred_languages": settings.youtube_preferred_languages,
-        }
-
-        settings_response = api_client.update_settings(**updates)
-        settings_data = (
-            settings_response
-            if isinstance(settings_response, dict)
-            else settings_response[0]
-        )
-
-        # Update the settings object with the response
-        settings.default_content_processing_engine_doc = settings_data.get(
-            "default_content_processing_engine_doc"
-        )
-        settings.default_content_processing_engine_url = settings_data.get(
-            "default_content_processing_engine_url"
-        )
-        settings.default_embedding_option = settings_data.get(
-            "default_embedding_option"
-        )
-        settings.auto_delete_files = settings_data.get("auto_delete_files")
-        settings.youtube_preferred_languages = settings_data.get(
-            "youtube_preferred_languages"
-        )
-
-        return settings
-
-
-# Global service instance
-settings_service = SettingsService()

api/sources_service.py

@@ -1,324 +0,0 @@
-"""
-Sources service layer using API.
-"""
-
-from dataclasses import dataclass
-from typing import Dict, List, Optional, Union
-
-from loguru import logger
-
-from api.client import api_client
-from open_notebook.domain.notebook import Asset, Source
-
-
-@dataclass
-class SourceProcessingResult:
-    """Result of source creation with optional async processing info."""
-
-    source: Source
-    is_async: bool = False
-    command_id: Optional[str] = None
-    status: Optional[str] = None
-    processing_info: Optional[Dict] = None
-
-
-@dataclass
-class SourceWithMetadata:
-    """Source object with additional metadata from API."""
-
-    source: Source
-    embedded_chunks: int
-
-    # Expose common source properties for easy access
-    @property
-    def id(self):
-        return self.source.id
-
-    @property
-    def title(self):
-        return self.source.title
-
-    @title.setter
-    def title(self, value):
-        self.source.title = value
-
-    @property
-    def topics(self):
-        return self.source.topics
-
-    @property
-    def asset(self):
-        return self.source.asset
-
-    @property
-    def full_text(self):
-        return self.source.full_text
-
-    @property
-    def created(self):
-        return self.source.created
-
-    @property
-    def updated(self):
-        return self.source.updated
-
-
-class SourcesService:
-    """Service layer for sources operations using API."""
-
-    def __init__(self):
-        logger.info("Using API for sources operations")
-
-    def get_all_sources(
-        self, notebook_id: Optional[str] = None
-    ) -> List[SourceWithMetadata]:
-        """Get all sources with optional notebook filtering."""
-        sources_data = api_client.get_sources(notebook_id=notebook_id)
-        # Convert API response to SourceWithMetadata objects
-        sources = []
-        for source_data in sources_data:
-            source = Source(
-                title=source_data["title"],
-                topics=source_data["topics"],
-                asset=Asset(
-                    file_path=source_data["asset"]["file_path"]
-                    if source_data["asset"]
-                    else None,
-                    url=source_data["asset"]["url"] if source_data["asset"] else None,
-                )
-                if source_data["asset"]
-                else None,
-            )
-            source.id = source_data["id"]
-            source.created = source_data["created"]
-            source.updated = source_data["updated"]
-
-            # Wrap in SourceWithMetadata
-            source_with_metadata = SourceWithMetadata(
-                source=source, embedded_chunks=source_data.get("embedded_chunks", 0)
-            )
-            sources.append(source_with_metadata)
-        return sources
-
-    def get_source(self, source_id: str) -> SourceWithMetadata:
-        """Get a specific source."""
-        response = api_client.get_source(source_id)
-        source_data = response if isinstance(response, dict) else response[0]
-        source = Source(
-            title=source_data["title"],
-            topics=source_data["topics"],
-            full_text=source_data["full_text"],
-            asset=Asset(
-                file_path=source_data["asset"]["file_path"]
-                if source_data["asset"]
-                else None,
-                url=source_data["asset"]["url"] if source_data["asset"] else None,
-            )
-            if source_data["asset"]
-            else None,
-        )
-        source.id = source_data["id"]
-        source.created = source_data["created"]
-        source.updated = source_data["updated"]
-
-        return SourceWithMetadata(
-            source=source, embedded_chunks=source_data.get("embedded_chunks", 0)
-        )
-
-    def create_source(
-        self,
-        notebook_id: Optional[str] = None,
-        source_type: str = "text",
-        url: Optional[str] = None,
-        file_path: Optional[str] = None,
-        content: Optional[str] = None,
-        title: Optional[str] = None,
-        transformations: Optional[List[str]] = None,
-        embed: bool = False,
-        delete_source: bool = False,
-        notebooks: Optional[List[str]] = None,
-        async_processing: bool = False,
-    ) -> Union[Source, SourceProcessingResult]:
-        """
-        Create a new source with support for async processing.
-
-        Args:
-            notebook_id: Single notebook ID (deprecated, use notebooks parameter)
-            source_type: Type of source (link, upload, text)
-            url: URL for link sources
-            file_path: File path for upload sources
-            content: Text content for text sources
-            title: Optional source title
-            transformations: List of transformation IDs to apply
-            embed: Whether to embed content for vector search
-            delete_source: Whether to delete uploaded file after processing
-            notebooks: List of notebook IDs to add source to (preferred over notebook_id)
-            async_processing: Whether to process source asynchronously
-
-        Returns:
-            Source object for sync processing (backward compatibility)
-            SourceProcessingResult for async processing (contains additional metadata)
-        """
-        source_data = api_client.create_source(
-            notebook_id=notebook_id,
-            notebooks=notebooks,
-            source_type=source_type,
-            url=url,
-            file_path=file_path,
-            content=content,
-            title=title,
-            transformations=transformations,
-            embed=embed,
-            delete_source=delete_source,
-            async_processing=async_processing,
-        )
-
-        # Create Source object from response
-        response_data = source_data if isinstance(source_data, dict) else source_data[0]
-        source = Source(
-            title=response_data["title"],
-            topics=response_data.get("topics") or [],
-            full_text=response_data.get("full_text"),
-            asset=Asset(
-                file_path=response_data["asset"]["file_path"]
-                if response_data.get("asset")
-                else None,
-                url=response_data["asset"]["url"]
-                if response_data.get("asset")
-                else None,
-            )
-            if response_data.get("asset")
-            else None,
-        )
-        source.id = response_data["id"]
-        source.created = response_data["created"]
-        source.updated = response_data["updated"]
-
-        # Check if this is an async processing response
-        if (
-            response_data.get("command_id")
-            or response_data.get("status")
-            or response_data.get("processing_info")
-        ):
-            # Ensure source_data is a dict for accessing attributes
-            source_data_dict = (
-                source_data if isinstance(source_data, dict) else source_data[0]
-            )
-            # Return enhanced result for async processing
-            return SourceProcessingResult(
-                source=source,
-                is_async=True,
-                command_id=source_data_dict.get("command_id"),
-                status=source_data_dict.get("status"),
-                processing_info=source_data_dict.get("processing_info"),
-            )
-        else:
-            # Return simple Source for backward compatibility
-            return source
-
-    def get_source_status(self, source_id: str) -> Dict:
-        """Get processing status for a source."""
-        response = api_client.get_source_status(source_id)
-        return response if isinstance(response, dict) else response[0]
-
-    def create_source_async(
-        self,
-        notebook_id: Optional[str] = None,
-        source_type: str = "text",
-        url: Optional[str] = None,
-        file_path: Optional[str] = None,
-        content: Optional[str] = None,
-        title: Optional[str] = None,
-        transformations: Optional[List[str]] = None,
-        embed: bool = False,
-        delete_source: bool = False,
-        notebooks: Optional[List[str]] = None,
-    ) -> SourceProcessingResult:
-        """
-        Create a new source with async processing enabled.
-
-        This is a convenience method that always uses async processing.
-        Returns a SourceProcessingResult with processing status information.
-        """
-        result = self.create_source(
-            notebook_id=notebook_id,
-            notebooks=notebooks,
-            source_type=source_type,
-            url=url,
-            file_path=file_path,
-            content=content,
-            title=title,
-            transformations=transformations,
-            embed=embed,
-            delete_source=delete_source,
-            async_processing=True,
-        )
-
-        # Since we forced async_processing=True, this should always be a SourceProcessingResult
-        if isinstance(result, SourceProcessingResult):
-            return result
-        else:
-            # Fallback: wrap Source in SourceProcessingResult
-            return SourceProcessingResult(
-                source=result,
-                is_async=False,  # This shouldn't happen, but handle it gracefully
-            )
-
-    def is_source_processing_complete(self, source_id: str) -> bool:
-        """
-        Check if a source's async processing is complete.
-
-        Returns True if processing is complete (success or failure),
-        False if still processing or queued.
-        """
-        try:
-            status_data = self.get_source_status(source_id)
-            status = status_data.get("status")
-            return status in [
-                "completed",
-                "failed",
-                None,
-            ]  # None indicates legacy/sync source
-        except Exception as e:
-            logger.error(f"Error checking source processing status: {e}")
-            return True  # Assume complete on error
-
-    def update_source(self, source: Source) -> Source:
-        """Update a source."""
-        if not source.id:
-            raise ValueError("Source ID is required for update")
-
-        updates = {
-            "title": source.title,
-            "topics": source.topics,
-        }
-        source_data = api_client.update_source(source.id, **updates)
-
-        # Ensure source_data is a dict
-        source_data_dict = (
-            source_data if isinstance(source_data, dict) else source_data[0]
-        )
-
-        # Update the source object with the response
-        source.title = source_data_dict["title"]
-        source.topics = source_data_dict["topics"]
-        source.updated = source_data_dict["updated"]
-
-        return source
-
-    def delete_source(self, source_id: str) -> bool:
-        """Delete a source."""
-        api_client.delete_source(source_id)
-        return True
-
-
-# Global service instance
-sources_service = SourcesService()
-
-# Export important classes for easy importing
-__all__ = [
-    "SourcesService",
-    "SourceWithMetadata",
-    "SourceProcessingResult",
-    "sources_service",
-]

api/transformations_service.py

@@ -1,141 +0,0 @@
-"""
-Transformations service layer using API.
-"""
-
-from datetime import datetime
-from typing import Any, Dict, List, Union
-
-from loguru import logger
-
-from api.client import api_client
-from open_notebook.domain.transformation import Transformation
-
-
-class TransformationsService:
-    """Service layer for transformations operations using API."""
-
-    def __init__(self):
-        logger.info("Using API for transformations operations")
-
-    def get_all_transformations(self) -> List[Transformation]:
-        """Get all transformations."""
-        transformations_data = api_client.get_transformations()
-        # Convert API response to Transformation objects
-        transformations = []
-        for trans_data in transformations_data:
-            transformation = Transformation(
-                name=trans_data["name"],
-                title=trans_data["title"],
-                description=trans_data["description"],
-                prompt=trans_data["prompt"],
-                apply_default=trans_data["apply_default"],
-            )
-            transformation.id = trans_data["id"]
-            transformation.created = datetime.fromisoformat(
-                trans_data["created"].replace("Z", "+00:00")
-            )
-            transformation.updated = datetime.fromisoformat(
-                trans_data["updated"].replace("Z", "+00:00")
-            )
-            transformations.append(transformation)
-        return transformations
-
-    def get_transformation(self, transformation_id: str) -> Transformation:
-        """Get a specific transformation."""
-        response = api_client.get_transformation(transformation_id)
-        trans_data = response if isinstance(response, dict) else response[0]
-        transformation = Transformation(
-            name=trans_data["name"],
-            title=trans_data["title"],
-            description=trans_data["description"],
-            prompt=trans_data["prompt"],
-            apply_default=trans_data["apply_default"],
-        )
-        transformation.id = trans_data["id"]
-        transformation.created = datetime.fromisoformat(
-            trans_data["created"].replace("Z", "+00:00")
-        )
-        transformation.updated = datetime.fromisoformat(
-            trans_data["updated"].replace("Z", "+00:00")
-        )
-        return transformation
-
-    def create_transformation(
-        self,
-        name: str,
-        title: str,
-        description: str,
-        prompt: str,
-        apply_default: bool = False,
-    ) -> Transformation:
-        """Create a new transformation."""
-        response = api_client.create_transformation(
-            name=name,
-            title=title,
-            description=description,
-            prompt=prompt,
-            apply_default=apply_default,
-        )
-        trans_data = response if isinstance(response, dict) else response[0]
-        transformation = Transformation(
-            name=trans_data["name"],
-            title=trans_data["title"],
-            description=trans_data["description"],
-            prompt=trans_data["prompt"],
-            apply_default=trans_data["apply_default"],
-        )
-        transformation.id = trans_data["id"]
-        transformation.created = datetime.fromisoformat(
-            trans_data["created"].replace("Z", "+00:00")
-        )
-        transformation.updated = datetime.fromisoformat(
-            trans_data["updated"].replace("Z", "+00:00")
-        )
-        return transformation
-
-    def update_transformation(self, transformation: Transformation) -> Transformation:
-        """Update a transformation."""
-        if not transformation.id:
-            raise ValueError("Transformation ID is required for update")
-
-        updates = {
-            "name": transformation.name,
-            "title": transformation.title,
-            "description": transformation.description,
-            "prompt": transformation.prompt,
-            "apply_default": transformation.apply_default,
-        }
-        response = api_client.update_transformation(transformation.id, **updates)
-        trans_data = response if isinstance(response, dict) else response[0]
-
-        # Update the transformation object with the response
-        transformation.name = trans_data["name"]
-        transformation.title = trans_data["title"]
-        transformation.description = trans_data["description"]
-        transformation.prompt = trans_data["prompt"]
-        transformation.apply_default = trans_data["apply_default"]
-        transformation.updated = datetime.fromisoformat(
-            trans_data["updated"].replace("Z", "+00:00")
-        )
-
-        return transformation
-
-    def delete_transformation(self, transformation_id: str) -> bool:
-        """Delete a transformation."""
-        api_client.delete_transformation(transformation_id)
-        return True
-
-    def execute_transformation(
-        self, transformation_id: str, input_text: str, model_id: str
-    ) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
-        """Execute a transformation on input text."""
-        result = api_client.execute_transformation(
-            transformation_id=transformation_id,
-            input_text=input_text,
-            model_id=model_id,
-        )
-        return result
-
-
-# Global service instance
-transformations_service = TransformationsService()

commands/CLAUDE.md

@@ -1,68 +0,0 @@
-# Commands Module
-
-**Purpose**: Defines async command handlers for long-running operations via `surreal-commands` job queue system.
-
-## Key Components
-
-### Embedding Commands
-
-- **`embed_note_command`**: Embeds a single note using unified embedding pipeline with content-type aware processing. Uses MARKDOWN content type detection. Retry: 5 attempts, exponential jitter 1-60s.
-- **`embed_insight_command`**: Embeds a single source insight. Uses MARKDOWN content type. Retry: 5 attempts, exponential jitter 1-60s.
-- **`embed_source_command`**: Embeds a source by chunking full_text with content-type aware splitters (HTML, Markdown, plain), then batch embedding all chunks (batches of 50 with per-batch retry). Retry: 5 attempts, exponential jitter 1-60s.
-- **`create_insight_command`**: Creates a source insight with automatic retry on transaction conflicts. Creates the DB record, then submits `embed_insight` command (fire-and-forget). Retry: 5 attempts, exponential jitter 1-60s. Used by `Source.add_insight()`.
-- **`rebuild_embeddings_command`**: Submits individual embed_* commands for all sources/notes/insights. Returns immediately; actual embedding happens async. No retry (coordinator only).
-
-### Other Commands
-
-- **`process_source_command`**: Ingests content through `source_graph`, creates embeddings (optional), and generates insights. Retries on transaction conflicts (exp. jitter, max 15×, 1-120s).
-- **`run_transformation_command`**: Runs a transformation on an existing source to generate an insight. Executes the transformation graph (LLM call) then creates insight via `create_insight_command`. Used by `POST /sources/{id}/insights` API endpoint. Retry: 5 attempts, exponential jitter 1-60s.
-- **`generate_podcast_command`**: Creates podcasts via podcast-creator library. Resolves model registry references and credentials for all profiles before invoking podcast-creator. Validates that outline_llm, transcript_llm, and voice_model are configured.
-- **`process_text_command`** (example): Test fixture for text operations (uppercase, lowercase, reverse, word_count).
-- **`analyze_data_command`** (example): Test fixture for numeric aggregations.
-
-## Important Patterns
-
-- **Pydantic I/O**: All commands use `CommandInput`/`CommandOutput` subclasses for type safety and serialization.
-- **Error handling**: Permanent errors (ValueError) return failure output; all other exceptions auto-retry via surreal-commands.
-- **Retry configuration**: Uses `stop_on: [ValueError]` (blocklist approach) - retries all exceptions EXCEPT ValueError. This is more resilient than allowlist as new exception types auto-retry.
-- **Fire-and-forget embedding**: Domain models submit embed_* commands via `submit_command()` without waiting. Commands process asynchronously.
-- **Content-type aware chunking**: `embed_source_command` uses `chunk_text()` with automatic content type detection (HTML, Markdown, plain text) for optimal text splitting. Default: 1500 char chunks with 225 char overlap.
-- **Batch embedding**: `embed_source_command` uses `generate_embeddings()` which automatically batches texts (default 50) with per-batch retry to avoid exceeding provider payload limits.
-- **Mean pooling for large content**: `embed_note_command` and `embed_insight_command` use `generate_embedding()` which handles content larger than chunk size via mean pooling.
-- **Model dumping**: Recursive `full_model_dump()` utility converts Pydantic models → dicts for DB/API responses.
-- **Logging**: Uses `loguru.logger` throughout; logs execution start/end and key metrics (processing time, counts).
-- **Time tracking**: All commands measure `start_time` → `processing_time` for monitoring.
-
-## Dependencies
-
-**External**: `surreal_commands` (command decorator, job queue, submit_command), `loguru`, `pydantic`, `podcast_creator`
-**Internal**: `open_notebook.domain.notebook` (Source, Note, SourceInsight), `open_notebook.utils.chunking` (chunk_text, detect_content_type), `open_notebook.utils.embedding` (generate_embedding, generate_embeddings), `open_notebook.database.repository` (repo_query, repo_insert)
-
-## Quirks & Edge Cases
-
-- **source_commands**: `ensure_record_id()` wraps command IDs for DB storage; transaction conflicts trigger exponential backoff retry. ValueError exceptions are permanent (not retried).
-- **embedding_commands**: Content type detection uses file extension as primary source, heuristics as fallback. Chunks >1800 chars trigger secondary splitting. Empty/whitespace-only content returns ValueError (not retried).
-- **rebuild_embeddings_command**: Returns "jobs_submitted" not "processed_items" - embedding is async. Individual commands handle failures with their own retries.
-- **podcast_commands**: Profiles loaded from SurrealDB by name; model configs (credentials) resolved for ALL profiles before podcast-creator validation. Validates outline_llm/transcript_llm/voice_model are set. Episode records created mid-execution.
-- **Example commands**: Accept optional `delay_seconds` for testing async behavior; not for production.
-
-## Code Example
-
-```python
-@command("process_source", app="open_notebook", retry={
-    "max_attempts": 5,
-    "wait_strategy": "exponential_jitter",
-    "stop_on": [ValueError],  # Don't retry validation errors
-})
-async def process_source_command(input_data: SourceProcessingInput) -> SourceProcessingOutput:
-    start_time = time.time()
-    try:
-        transformations = [await Transformation.get(id) for id in input_data.transformations]
-        source = await Source.get(input_data.source_id)
-        result = await source_graph.ainvoke({...})
-        return SourceProcessingOutput(success=True, ...)
-    except ValueError as e:
-        return SourceProcessingOutput(success=False, error_message=str(e))  # No retry
-    except Exception as e:
-        raise  # Retry all other exceptions
-```

commands/__init__.py

@@ -1,12 +1,19 @@
 """Surreal-commands integration for Open Notebook"""
 
+# The worker starts via `surreal-commands-worker --import-modules commands`,
+# so this package is imported before the worker connects to SurrealDB. Inject
+# the internal DB hosts into no_proxy first so the DB websocket is never
+# tunnelled through a configured HTTP proxy (issue #1160).
+from open_notebook.utils.proxy import ensure_internal_no_proxy
+
+ensure_internal_no_proxy()
+
 from .embedding_commands import (
     embed_insight_command,
     embed_note_command,
     embed_source_command,
     rebuild_embeddings_command,
 )
-from .example_commands import analyze_data_command, process_text_command
 from .podcast_commands import generate_podcast_command
 from .source_commands import process_source_command
 
@@ -19,6 +26,4 @@ __all__ = [
     # Other commands
     "generate_podcast_command",
     "process_source_command",
-    "process_text_command",
-    "analyze_data_command",
 ]

commands/embedding_commands.py

@@ -1,27 +1,45 @@
 import time
-from typing import Dict, List, Literal, Optional
+from typing import (
+    Any,
+    Awaitable,
+    Callable,
+    Dict,
+    List,
+    Literal,
+    Optional,
+    Tuple,
+)
 
 from loguru import logger
-from pydantic import BaseModel
 from surreal_commands import CommandInput, CommandOutput, command, submit_command
 
 from open_notebook.ai.models import model_manager
 from open_notebook.database.repository import ensure_record_id, repo_insert, repo_query
-from open_notebook.exceptions import ConfigurationError
 from open_notebook.domain.notebook import Note, Source, SourceInsight
+from open_notebook.exceptions import (
+    ConfigurationError,
+    ContextLengthExceededError,
+    NotFoundError,
+)
 from open_notebook.utils.chunking import ContentType, chunk_text, detect_content_type
 from open_notebook.utils.embedding import generate_embedding, generate_embeddings
 
-
-def full_model_dump(model):
-    if isinstance(model, BaseModel):
-        return model.model_dump()
-    elif isinstance(model, dict):
-        return {k: full_model_dump(v) for k, v in model.items()}
-    elif isinstance(model, list):
-        return [full_model_dump(item) for item in model]
-    else:
-        return model
+# NOTE: `stop_on` below can never trigger in practice — each command catches
+# ValueError (and NotFoundError) internally and returns success=False instead
+# of raising, so the retry layer never sees it. Kept as-is on purpose; to be revisited in a
+# dedicated error-handling PR.
+EMBED_RETRY_CONFIG = {
+    "max_attempts": 5,
+    "wait_strategy": "exponential_jitter",
+    "wait_min": 1,
+    "wait_max": 60,
+    "stop_on": [
+        ValueError,
+        ConfigurationError,
+        ContextLengthExceededError,
+    ],  # Don't retry validation/config errors
+    "retry_log_level": "warning",
+}
 
 
 def get_command_id(input_data: CommandInput) -> str:
@@ -31,6 +49,99 @@ def get_command_id(input_data: CommandInput) -> str:
     return "unknown"
 
 
+async def _embed_record(
+    input_data: CommandInput,
+    *,
+    kind: str,
+    record_id: str,
+    embed: Callable[[], Awaitable[Tuple[Dict[str, Any], str]]],
+) -> Tuple[Optional[Dict[str, Any]], float, Optional[str]]:
+    """
+    Shared core for the embed_* commands: run the embedding work with the
+    common logging and error-handling epilogue.
+
+    Args:
+        input_data: The command input (used for command_id logging).
+        kind: Record kind for log messages ("note", "insight", "source").
+        record_id: The record being embedded.
+        embed: Async callable doing the actual load/validate/embed/write work.
+            Returns (extra_output_fields, success_log_detail).
+
+    Returns:
+        (extra_output_fields, processing_time, error_message)
+        extra_output_fields is None and error_message is set on permanent
+        (ValueError, or NotFoundError for a record deleted before the job ran)
+        failure. Transient failures re-raise so the retry layer
+        can handle them.
+    """
+    start_time = time.time()
+
+    try:
+        logger.info(f"Starting embedding for {kind}: {record_id}")
+
+        extra_fields, log_detail = await embed()
+
+        processing_time = time.time() - start_time
+        logger.info(
+            f"Successfully embedded {kind} {record_id}{log_detail} in {processing_time:.2f}s"
+        )
+        return extra_fields, processing_time, None
+
+    except (ValueError, NotFoundError) as e:
+        # Permanent failure - don't retry. NotFoundError means the record was
+        # deleted before the job ran (ObjectModel.get raises it only for a
+        # missing record; DB failures are DatabaseOperationError and retry).
+        processing_time = time.time() - start_time
+        cmd_id = get_command_id(input_data)
+        logger.error(f"Failed to embed {kind} {record_id} (command: {cmd_id}): {e}")
+        return None, processing_time, str(e)
+    except Exception as e:
+        # Transient failure - will be retried (surreal-commands logs final failure)
+        cmd_id = get_command_id(input_data)
+        logger.debug(
+            f"Transient error embedding {kind} {record_id} (command: {cmd_id}): {e}"
+        )
+        raise
+
+
+async def _embed_markdown_record(
+    input_data: CommandInput,
+    *,
+    label: str,
+    record_id: str,
+    loader: Callable[[str], Awaitable[Any]],
+) -> Tuple[Dict[str, Any], str]:
+    """
+    Load a record, validate its content, embed it as markdown and UPSERT the
+    embedding back onto the record. Shared by embed_note and embed_insight.
+    """
+    # 1. Load record
+    record = await loader(record_id)
+    if not record:
+        raise ValueError(f"{label} '{record_id}' not found")
+
+    if not record.content or not record.content.strip():
+        raise ValueError(f"{label} '{record_id}' has no content to embed")
+
+    # 2. Generate embedding (auto-chunks + mean pools if needed)
+    # Notes and insights are typically markdown content
+    cmd_id = get_command_id(input_data)
+    embedding = await generate_embedding(
+        record.content, content_type=ContentType.MARKDOWN, command_id=cmd_id
+    )
+
+    # 3. UPSERT embedding into the record
+    await repo_query(
+        "UPDATE $record_id SET embedding = $embedding",
+        {
+            "record_id": ensure_record_id(record_id),
+            "embedding": embedding,
+        },
+    )
+
+    return {}, ""
+
+
 class RebuildEmbeddingsInput(CommandInput):
     mode: Literal["existing", "all"]
     include_sources: bool = True
@@ -50,11 +161,6 @@ class RebuildEmbeddingsOutput(CommandOutput):
     error_message: Optional[str] = None
 
 
-# =============================================================================
-# NEW EMBEDDING COMMANDS (Phase 3)
-# =============================================================================
-
-
 class CreateInsightInput(CommandInput):
     """Input for creating a source insight with automatic retry on conflicts."""
 
@@ -118,18 +224,7 @@ class EmbedSourceOutput(CommandOutput):
     error_message: Optional[str] = None
 
 
-@command(
-    "embed_note",
-    app="open_notebook",
-    retry={
-        "max_attempts": 5,
-        "wait_strategy": "exponential_jitter",
-        "wait_min": 1,
-        "wait_max": 60,
-        "stop_on": [ValueError, ConfigurationError],  # Don't retry validation/config errors
-        "retry_log_level": "debug",
-    },
-)
+@command("embed_note", app="open_notebook", retry=EMBED_RETRY_CONFIG)
 async def embed_note_command(input_data: EmbedNoteInput) -> EmbedNoteOutput:
     """
     Generate and store embedding for a single note.
@@ -145,83 +240,34 @@ async def embed_note_command(input_data: EmbedNoteInput) -> EmbedNoteOutput:
     Retry Strategy:
     - Retries up to 5 times for transient failures (network, timeout, etc.)
     - Uses exponential-jitter backoff (1-60s)
-    - Does NOT retry permanent failures (ValueError for validation errors)
+    - Does NOT retry permanent failures (ValueError for validation errors,
+      NotFoundError for a record deleted before the job ran)
     """
-    start_time = time.time()
 
-    try:
-        logger.info(f"Starting embedding for note: {input_data.note_id}")
-
-        # 1. Load note
-        note = await Note.get(input_data.note_id)
-        if not note:
-            raise ValueError(f"Note '{input_data.note_id}' not found")
-
-        if not note.content or not note.content.strip():
-            raise ValueError(f"Note '{input_data.note_id}' has no content to embed")
-
-        # 2. Generate embedding (auto-chunks + mean pools if needed)
-        # Notes are typically markdown content
-        cmd_id = get_command_id(input_data)
-        embedding = await generate_embedding(
-            note.content, content_type=ContentType.MARKDOWN, command_id=cmd_id
+    async def embed() -> Tuple[Dict[str, Any], str]:
+        return await _embed_markdown_record(
+            input_data,
+            label="Note",
+            record_id=input_data.note_id,
+            loader=Note.get,
         )
 
-        # 3. UPSERT embedding into note record
-        await repo_query(
-            "UPDATE $note_id SET embedding = $embedding",
-            {
-                "note_id": ensure_record_id(input_data.note_id),
-                "embedding": embedding,
-            },
-        )
+    _, processing_time, error_message = await _embed_record(
+        input_data,
+        kind="note",
+        record_id=input_data.note_id,
+        embed=embed,
+    )
 
-        processing_time = time.time() - start_time
-        logger.info(
-            f"Successfully embedded note {input_data.note_id} in {processing_time:.2f}s"
-        )
-
-        return EmbedNoteOutput(
-            success=True,
-            note_id=input_data.note_id,
-            processing_time=processing_time,
-        )
-
-    except ValueError as e:
-        # Permanent failure - don't retry
-        processing_time = time.time() - start_time
-        cmd_id = get_command_id(input_data)
-        logger.error(
-            f"Failed to embed note {input_data.note_id} (command: {cmd_id}): {e}"
-        )
-        return EmbedNoteOutput(
-            success=False,
-            note_id=input_data.note_id,
-            processing_time=processing_time,
-            error_message=str(e),
-        )
-    except Exception as e:
-        # Transient failure - will be retried (surreal-commands logs final failure)
-        cmd_id = get_command_id(input_data)
-        logger.debug(
-            f"Transient error embedding note {input_data.note_id} "
-            f"(command: {cmd_id}): {e}"
-        )
-        raise
+    return EmbedNoteOutput(
+        success=error_message is None,
+        note_id=input_data.note_id,
+        processing_time=processing_time,
+        error_message=error_message,
+    )
 
 
-@command(
-    "embed_insight",
-    app="open_notebook",
-    retry={
-        "max_attempts": 5,
-        "wait_strategy": "exponential_jitter",
-        "wait_min": 1,
-        "wait_max": 60,
-        "stop_on": [ValueError, ConfigurationError],  # Don't retry validation/config errors
-        "retry_log_level": "debug",
-    },
-)
+@command("embed_insight", app="open_notebook", retry=EMBED_RETRY_CONFIG)
 async def embed_insight_command(input_data: EmbedInsightInput) -> EmbedInsightOutput:
     """
     Generate and store embedding for a single source insight.
@@ -237,85 +283,34 @@ async def embed_insight_command(input_data: EmbedInsightInput) -> EmbedInsightOu
     Retry Strategy:
     - Retries up to 5 times for transient failures (network, timeout, etc.)
     - Uses exponential-jitter backoff (1-60s)
-    - Does NOT retry permanent failures (ValueError for validation errors)
+    - Does NOT retry permanent failures (ValueError for validation errors,
+      NotFoundError for a record deleted before the job ran)
     """
-    start_time = time.time()
 
-    try:
-        logger.info(f"Starting embedding for insight: {input_data.insight_id}")
-
-        # 1. Load insight
-        insight = await SourceInsight.get(input_data.insight_id)
-        if not insight:
-            raise ValueError(f"Insight '{input_data.insight_id}' not found")
-
-        if not insight.content or not insight.content.strip():
-            raise ValueError(
-                f"Insight '{input_data.insight_id}' has no content to embed"
-            )
-
-        # 2. Generate embedding (auto-chunks + mean pools if needed)
-        # Insights are typically markdown content (generated by LLM)
-        cmd_id = get_command_id(input_data)
-        embedding = await generate_embedding(
-            insight.content, content_type=ContentType.MARKDOWN, command_id=cmd_id
+    async def embed() -> Tuple[Dict[str, Any], str]:
+        return await _embed_markdown_record(
+            input_data,
+            label="Insight",
+            record_id=input_data.insight_id,
+            loader=SourceInsight.get,
         )
 
-        # 3. UPSERT embedding into insight record
-        await repo_query(
-            "UPDATE $insight_id SET embedding = $embedding",
-            {
-                "insight_id": ensure_record_id(input_data.insight_id),
-                "embedding": embedding,
-            },
-        )
+    _, processing_time, error_message = await _embed_record(
+        input_data,
+        kind="insight",
+        record_id=input_data.insight_id,
+        embed=embed,
+    )
 
-        processing_time = time.time() - start_time
-        logger.info(
-            f"Successfully embedded insight {input_data.insight_id} in {processing_time:.2f}s"
-        )
-
-        return EmbedInsightOutput(
-            success=True,
-            insight_id=input_data.insight_id,
-            processing_time=processing_time,
-        )
-
-    except ValueError as e:
-        # Permanent failure - don't retry
-        processing_time = time.time() - start_time
-        cmd_id = get_command_id(input_data)
-        logger.error(
-            f"Failed to embed insight {input_data.insight_id} (command: {cmd_id}): {e}"
-        )
-        return EmbedInsightOutput(
-            success=False,
-            insight_id=input_data.insight_id,
-            processing_time=processing_time,
-            error_message=str(e),
-        )
-    except Exception as e:
-        # Transient failure - will be retried (surreal-commands logs final failure)
-        cmd_id = get_command_id(input_data)
-        logger.debug(
-            f"Transient error embedding insight {input_data.insight_id} "
-            f"(command: {cmd_id}): {e}"
-        )
-        raise
+    return EmbedInsightOutput(
+        success=error_message is None,
+        insight_id=input_data.insight_id,
+        processing_time=processing_time,
+        error_message=error_message,
+    )
 
 
-@command(
-    "embed_source",
-    app="open_notebook",
-    retry={
-        "max_attempts": 5,
-        "wait_strategy": "exponential_jitter",
-        "wait_min": 1,
-        "wait_max": 60,
-        "stop_on": [ValueError, ConfigurationError],  # Don't retry validation/config errors
-        "retry_log_level": "debug",
-    },
-)
+@command("embed_source", app="open_notebook", retry=EMBED_RETRY_CONFIG)
 async def embed_source_command(input_data: EmbedSourceInput) -> EmbedSourceOutput:
     """
     Generate and store embeddings for a source document.
@@ -334,13 +329,11 @@ async def embed_source_command(input_data: EmbedSourceInput) -> EmbedSourceOutpu
     Retry Strategy:
     - Retries up to 5 times for transient failures (network, timeout, etc.)
     - Uses exponential-jitter backoff (1-60s)
-    - Does NOT retry permanent failures (ValueError for validation errors)
+    - Does NOT retry permanent failures (ValueError for validation errors,
+      NotFoundError for a record deleted before the job ran)
     """
-    start_time = time.time()
-
-    try:
-        logger.info(f"Starting embedding for source: {input_data.source_id}")
 
+    async def embed() -> Tuple[Dict[str, Any], str]:
         # 1. Load source
         source = await Source.get(input_data.source_id)
         if not source:
@@ -371,7 +364,7 @@ async def embed_source_command(input_data: EmbedSourceInput) -> EmbedSourceOutpu
             f"Created {total_chunks} chunks for source {input_data.source_id} "
             f"(sizes: min={min(chunk_sizes) if chunk_sizes else 0}, "
             f"max={max(chunk_sizes) if chunk_sizes else 0}, "
-            f"avg={sum(chunk_sizes)//len(chunk_sizes) if chunk_sizes else 0} chars)"
+            f"avg={sum(chunk_sizes) // len(chunk_sizes) if chunk_sizes else 0} chars)"
         )
 
         if total_chunks == 0:
@@ -401,57 +394,44 @@ async def embed_source_command(input_data: EmbedSourceInput) -> EmbedSourceOutpu
         ]
 
         logger.debug(f"Inserting {len(records)} source_embedding records")
-        await repo_insert("source_embedding", records)
+        try:
+            await repo_insert("source_embedding", records)
+        except Exception:
+            # repo_insert writes in batches, so a failure on a later batch
+            # leaves the earlier ones behind. A partial set would mark the
+            # source as embedded while search sees only part of it: remove
+            # them before re-raising (the original error drives retry/stop).
+            try:
+                await repo_query(
+                    "DELETE source_embedding WHERE source = $source_id",
+                    {"source_id": ensure_record_id(input_data.source_id)},
+                )
+            except Exception as cleanup_error:
+                logger.error(
+                    f"Failed to clean up partial embeddings for source "
+                    f"{input_data.source_id}: {cleanup_error}"
+                )
+            raise
 
-        processing_time = time.time() - start_time
-        logger.info(
-            f"Successfully embedded source {input_data.source_id}: "
-            f"{total_chunks} chunks in {processing_time:.2f}s"
-        )
+        return {"chunks_created": total_chunks}, f": {total_chunks} chunks"
 
-        return EmbedSourceOutput(
-            success=True,
-            source_id=input_data.source_id,
-            chunks_created=total_chunks,
-            processing_time=processing_time,
-        )
+    extra_fields, processing_time, error_message = await _embed_record(
+        input_data,
+        kind="source",
+        record_id=input_data.source_id,
+        embed=embed,
+    )
 
-    except ValueError as e:
-        # Permanent failure - don't retry
-        processing_time = time.time() - start_time
-        cmd_id = get_command_id(input_data)
-        logger.error(
-            f"Failed to embed source {input_data.source_id} (command: {cmd_id}): {e}"
-        )
-        return EmbedSourceOutput(
-            success=False,
-            source_id=input_data.source_id,
-            chunks_created=0,
-            processing_time=processing_time,
-            error_message=str(e),
-        )
-    except Exception as e:
-        # Transient failure - will be retried (surreal-commands logs final failure)
-        cmd_id = get_command_id(input_data)
-        logger.debug(
-            f"Transient error embedding source {input_data.source_id} "
-            f"(command: {cmd_id}): {e}"
-        )
-        raise
+    return EmbedSourceOutput(
+        success=error_message is None,
+        source_id=input_data.source_id,
+        chunks_created=(extra_fields or {}).get("chunks_created", 0),
+        processing_time=processing_time,
+        error_message=error_message,
+    )
 
 
-@command(
-    "create_insight",
-    app="open_notebook",
-    retry={
-        "max_attempts": 5,
-        "wait_strategy": "exponential_jitter",
-        "wait_min": 1,
-        "wait_max": 60,
-        "stop_on": [ValueError, ConfigurationError],  # Don't retry validation/config errors
-        "retry_log_level": "debug",
-    },
-)
+@command("create_insight", app="open_notebook", retry=EMBED_RETRY_CONFIG)
 async def create_insight_command(
     input_data: CreateInsightInput,
 ) -> CreateInsightOutput:
@@ -470,7 +450,8 @@ async def create_insight_command(
     Retry Strategy:
     - Retries up to 5 times for transient failures (network, timeout, etc.)
     - Uses exponential-jitter backoff (1-60s)
-    - Does NOT retry permanent failures (ValueError for validation errors)
+    - Does NOT retry permanent failures (ValueError for validation errors,
+      NotFoundError for a record deleted before the job ran)
     """
     start_time = time.time()
 
@@ -619,6 +600,37 @@ async def collect_items_for_rebuild(
     return items
 
 
+def _submit_embedding_jobs(
+    kind: str, command_name: str, id_field: str, item_ids: List[str]
+) -> Tuple[int, int]:
+    """
+    Submit one embedding command per item, logging progress every 50 items.
+
+    Returns:
+        (submitted_count, failed_count)
+    """
+    logger.info(f"\nSubmitting {len(item_ids)} {kind} embedding jobs...")
+    submitted = 0
+    failed = 0
+    for idx, item_id in enumerate(item_ids, 1):
+        try:
+            submit_command(
+                "open_notebook",
+                command_name,
+                {id_field: item_id},
+            )
+            submitted += 1
+
+            if idx % 50 == 0 or idx == len(item_ids):
+                logger.info(f"  Progress: {idx}/{len(item_ids)} {kind} jobs submitted")
+
+        except Exception as e:
+            logger.error(f"Failed to submit {command_name} for {item_id}: {e}")
+            failed += 1
+
+    return submitted, failed
+
+
 @command("rebuild_embeddings", app="open_notebook", retry=None)
 async def rebuild_embeddings_command(
     input_data: RebuildEmbeddingsInput,
@@ -681,71 +693,17 @@ async def rebuild_embeddings_command(
                 processing_time=time.time() - start_time,
             )
 
-        # Initialize counters
-        sources_submitted = 0
-        notes_submitted = 0
-        insights_submitted = 0
-        failed_submissions = 0
-
-        # Submit embed_source commands for sources
-        logger.info(f"\nSubmitting {len(items['sources'])} source embedding jobs...")
-        for idx, source_id in enumerate(items["sources"], 1):
-            try:
-                submit_command(
-                    "open_notebook",
-                    "embed_source",
-                    {"source_id": source_id},
-                )
-                sources_submitted += 1
-
-                if idx % 50 == 0 or idx == len(items["sources"]):
-                    logger.info(
-                        f"  Progress: {idx}/{len(items['sources'])} source jobs submitted"
-                    )
-
-            except Exception as e:
-                logger.error(f"Failed to submit embed_source for {source_id}: {e}")
-                failed_submissions += 1
-
-        # Submit embed_note commands for notes
-        logger.info(f"\nSubmitting {len(items['notes'])} note embedding jobs...")
-        for idx, note_id in enumerate(items["notes"], 1):
-            try:
-                submit_command(
-                    "open_notebook",
-                    "embed_note",
-                    {"note_id": note_id},
-                )
-                notes_submitted += 1
-
-                if idx % 50 == 0 or idx == len(items["notes"]):
-                    logger.info(
-                        f"  Progress: {idx}/{len(items['notes'])} note jobs submitted"
-                    )
-
-            except Exception as e:
-                logger.error(f"Failed to submit embed_note for {note_id}: {e}")
-                failed_submissions += 1
-
-        # Submit embed_insight commands for insights
-        logger.info(f"\nSubmitting {len(items['insights'])} insight embedding jobs...")
-        for idx, insight_id in enumerate(items["insights"], 1):
-            try:
-                submit_command(
-                    "open_notebook",
-                    "embed_insight",
-                    {"insight_id": insight_id},
-                )
-                insights_submitted += 1
-
-                if idx % 50 == 0 or idx == len(items["insights"]):
-                    logger.info(
-                        f"  Progress: {idx}/{len(items['insights'])} insight jobs submitted"
-                    )
-
-            except Exception as e:
-                logger.error(f"Failed to submit embed_insight for {insight_id}: {e}")
-                failed_submissions += 1
+        # Submit one embedding command per item, per kind
+        sources_submitted, sources_failed = _submit_embedding_jobs(
+            "source", "embed_source", "source_id", items["sources"]
+        )
+        notes_submitted, notes_failed = _submit_embedding_jobs(
+            "note", "embed_note", "note_id", items["notes"]
+        )
+        insights_submitted, insights_failed = _submit_embedding_jobs(
+            "insight", "embed_insight", "insight_id", items["insights"]
+        )
+        failed_submissions = sources_failed + notes_failed + insights_failed
 
         processing_time = time.time() - start_time
         jobs_submitted = sources_submitted + notes_submitted + insights_submitted

commands/example_commands.py

@@ -1,142 +0,0 @@
-import asyncio
-import time
-from typing import List, Optional
-
-from loguru import logger
-from pydantic import BaseModel
-from surreal_commands import command
-
-
-class TextProcessingInput(BaseModel):
-    text: str
-    operation: str = "uppercase"  # uppercase, lowercase, word_count, reverse
-    delay_seconds: Optional[int] = None  # For testing async behavior
-
-
-class TextProcessingOutput(BaseModel):
-    success: bool
-    original_text: str
-    processed_text: Optional[str] = None
-    word_count: Optional[int] = None
-    processing_time: float
-    error_message: Optional[str] = None
-
-
-class DataAnalysisInput(BaseModel):
-    numbers: List[float]
-    analysis_type: str = "basic"  # basic, detailed
-    delay_seconds: Optional[int] = None
-
-
-class DataAnalysisOutput(BaseModel):
-    success: bool
-    analysis_type: str
-    count: int
-    sum: Optional[float] = None
-    average: Optional[float] = None
-    min_value: Optional[float] = None
-    max_value: Optional[float] = None
-    processing_time: float
-    error_message: Optional[str] = None
-
-
-@command("process_text", app="open_notebook")
-async def process_text_command(input_data: TextProcessingInput) -> TextProcessingOutput:
-    """
-    Example command for text processing. Tests basic command functionality
-    and demonstrates different processing types.
-    """
-    start_time = time.time()
-
-    try:
-        logger.info(f"Processing text with operation: {input_data.operation}")
-
-        # Simulate processing delay if specified
-        if input_data.delay_seconds:
-            await asyncio.sleep(input_data.delay_seconds)
-
-        processed_text = None
-        word_count = None
-
-        if input_data.operation == "uppercase":
-            processed_text = input_data.text.upper()
-        elif input_data.operation == "lowercase":
-            processed_text = input_data.text.lower()
-        elif input_data.operation == "reverse":
-            processed_text = input_data.text[::-1]
-        elif input_data.operation == "word_count":
-            word_count = len(input_data.text.split())
-            processed_text = f"Word count: {word_count}"
-        else:
-            raise ValueError(f"Unknown operation: {input_data.operation}")
-
-        processing_time = time.time() - start_time
-
-        return TextProcessingOutput(
-            success=True,
-            original_text=input_data.text,
-            processed_text=processed_text,
-            word_count=word_count,
-            processing_time=processing_time,
-        )
-
-    except Exception as e:
-        processing_time = time.time() - start_time
-        logger.error(f"Text processing failed: {e}")
-        return TextProcessingOutput(
-            success=False,
-            original_text=input_data.text,
-            processing_time=processing_time,
-            error_message=str(e),
-        )
-
-
-@command("analyze_data", app="open_notebook")
-async def analyze_data_command(input_data: DataAnalysisInput) -> DataAnalysisOutput:
-    """
-    Example command for data analysis. Tests command with complex input/output
-    and demonstrates error handling.
-    """
-    start_time = time.time()
-
-    try:
-        logger.info(
-            f"Analyzing {len(input_data.numbers)} numbers with {input_data.analysis_type} analysis"
-        )
-
-        # Simulate processing delay if specified
-        if input_data.delay_seconds:
-            await asyncio.sleep(input_data.delay_seconds)
-
-        if not input_data.numbers:
-            raise ValueError("No numbers provided for analysis")
-
-        count = len(input_data.numbers)
-        sum_value = sum(input_data.numbers)
-        average = sum_value / count
-        min_value = min(input_data.numbers)
-        max_value = max(input_data.numbers)
-
-        processing_time = time.time() - start_time
-
-        return DataAnalysisOutput(
-            success=True,
-            analysis_type=input_data.analysis_type,
-            count=count,
-            sum=sum_value,
-            average=average,
-            min_value=min_value,
-            max_value=max_value,
-            processing_time=processing_time,
-        )
-
-    except Exception as e:
-        processing_time = time.time() - start_time
-        logger.error(f"Data analysis failed: {e}")
-        return DataAnalysisOutput(
-            success=False,
-            analysis_type=input_data.analysis_type,
-            count=0,
-            processing_time=processing_time,
-            error_message=str(e),
-        )

commands/podcast_commands.py

@@ -4,17 +4,18 @@ from pathlib import Path
 from typing import Optional
 
 from loguru import logger
-from pydantic import BaseModel
 from surreal_commands import CommandInput, CommandOutput, command
 
-from open_notebook.config import DATA_FOLDER
+from open_notebook.config import PODCASTS_FOLDER
 from open_notebook.database.repository import ensure_record_id, repo_query
+from open_notebook.podcasts.audio_paths import to_relative_audio_path
 from open_notebook.podcasts.models import (
     EpisodeProfile,
     PodcastEpisode,
     SpeakerProfile,
     _resolve_model_config,
 )
+from open_notebook.utils.model_utils import full_model_dump
 
 try:
     from podcast_creator import configure, create_podcast
@@ -23,34 +24,84 @@ except ImportError as e:
     raise ValueError("podcast_creator library not available")
 
 
-def build_episode_output_dir(data_folder: str) -> tuple[str, Path]:
+def build_episode_output_dir(
+    podcasts_folder: str = PODCASTS_FOLDER,
+) -> tuple[str, Path]:
     """Build a filesystem-safe output directory path for a podcast episode.
 
     Uses a UUID as the directory name so the path is safe regardless of
     what the user typed as episode name (spaces, special chars, etc.).
 
+    Builds under PODCASTS_FOLDER — the same root to_relative_audio_path()
+    validates against at write time (#1030) — so the two can't drift apart.
+
     Returns:
         A tuple of (episode_dir_name, output_dir_path).
     """
     episode_dir_name = str(uuid.uuid4())
-    output_dir = Path(f"{data_folder}/podcasts/episodes/{episode_dir_name}")
+    output_dir = Path(podcasts_folder) / "episodes" / episode_dir_name
     return episode_dir_name, output_dir
 
 
-def full_model_dump(model):
-    if isinstance(model, BaseModel):
-        return model.model_dump()
-    elif isinstance(model, dict):
-        return {k: full_model_dump(v) for k, v in model.items()}
-    elif isinstance(model, list):
-        return [full_model_dump(item) for item in model]
-    else:
-        return model
+def explain_generation_failure(error_msg: str) -> Optional[str]:
+    """Map a podcast-generation failure to an actionable hint, or None.
+
+    Ordered most specific first. The GPT-5 extended-thinking hint used to be
+    the only one, so the two most common real failures got either nothing
+    (`Invalid speaker name`) or advice about the wrong provider - a truncated
+    Gemini response was told to switch to gpt-4o (#1238).
+    """
+    if "Invalid speaker name" in error_msg:
+        return (
+            "The transcript model returned a speaker name that is not in the "
+            "speaker profile - usually a placeholder copied from the prompt "
+            'such as "..." rather than an invented person. Speaker names must '
+            "match the profile exactly. Retrying the episode often succeeds, "
+            "since each attempt is a fresh sample."
+        )
+
+    if "Voice name" in error_msg and "not supported" in error_msg:
+        return (
+            "The speaker profile's voice_id is not valid for its TTS model. "
+            "Check the speaker profile voices in Podcasts -> Profiles against the ones "
+            "your voice model provides (the profiles seeded on install use "
+            "OpenAI voice names)."
+        )
+
+    if "Requested entity was not found" in error_msg:
+        return (
+            "Google returns this for any resource it cannot find, without "
+            "naming which one. Two candidates, likeliest first: a speaker "
+            "profile voice_id that its TTS model doesn't provide (the profiles "
+            "seeded on install use OpenAI voice names, which Gemini voice "
+            "models reject with exactly this message), or a model id in the "
+            "episode profile that doesn't exist for its provider. If the "
+            "transcript finished and the failure came during audio, it is the "
+            "voice."
+        )
+
+    if "Invalid json output" in error_msg or "Expecting value" in error_msg:
+        return (
+            "The model's response could not be parsed as JSON. Two common "
+            "causes: (1) the response was truncated - podcast-creator requests "
+            "up to 8192 output tokens per outline or transcript step unless the "
+            "episode profile sets max_tokens; if the model stops earlier (a "
+            "lower output limit, or a token-expensive language), set max_tokens "
+            "to its limit or use fewer and shorter segments; (2) a model using "
+            "extended thinking (e.g. "
+            "GPT-5) put all of its output inside <think> tags, leaving nothing "
+            "to parse - try gpt-4o, gpt-4o-mini or gpt-4-turbo instead."
+        )
+
+    return None
 
 
 class PodcastGenerationInput(CommandInput):
     episode_profile: str
-    speaker_profile: str
+    # Speaker profile record ID or name (the API boundary resolves the
+    # user-facing name to a record ID before submitting; both are accepted
+    # here for robustness).
+    speaker_profile: Optional[str] = None
     episode_name: str
     content: str
     briefing_suffix: Optional[str] = None
@@ -88,12 +139,25 @@ async def generate_podcast_command(
                 f"Episode profile '{input_data.episode_profile}' not found"
             )
 
-        speaker_profile = await SpeakerProfile.get_by_name(
-            episode_profile.speaker_config
-        )
-        if not speaker_profile:
+        # Honor the explicitly requested speaker profile when provided,
+        # falling back to the episode profile's configured speaker
+        # (a speaker_profile record ID since migration 20, None when the
+        # referenced profile no longer exists).
+        speaker_ref = input_data.speaker_profile or episode_profile.speaker_config
+        if not speaker_ref:
             raise ValueError(
-                f"Speaker profile '{episode_profile.speaker_config}' not found"
+                f"Episode profile '{episode_profile.name}' has no speaker "
+                "profile configured. Please update the profile to select a "
+                "speaker profile."
+            )
+        speaker_profile = await SpeakerProfile.resolve(speaker_ref)
+        if not speaker_profile:
+            if input_data.speaker_profile:
+                raise ValueError(f"Speaker profile '{speaker_ref}' not found")
+            raise ValueError(
+                f"Episode profile '{episode_profile.name}' references a "
+                "speaker profile that no longer exists. Please update the "
+                "profile to select a speaker profile."
             )
 
         logger.info(f"Loaded episode profile: {episode_profile.name}")
@@ -117,15 +181,21 @@ async def generate_podcast_command(
             )
 
         # 3. Resolve model configs with credentials
-        outline_provider, outline_model_name, outline_config = (
-            await episode_profile.resolve_outline_config()
-        )
-        transcript_provider, transcript_model_name, transcript_config = (
-            await episode_profile.resolve_transcript_config()
-        )
-        tts_provider, tts_model_name, tts_config = (
-            await speaker_profile.resolve_tts_config()
-        )
+        (
+            outline_provider,
+            outline_model_name,
+            outline_config,
+        ) = await episode_profile.resolve_outline_config()
+        (
+            transcript_provider,
+            transcript_model_name,
+            transcript_config,
+        ) = await episode_profile.resolve_transcript_config()
+        (
+            tts_provider,
+            tts_model_name,
+            tts_config,
+        ) = await speaker_profile.resolve_tts_config()
 
         logger.info(
             f"Resolved models - outline: {outline_provider}/{outline_model_name}, "
@@ -145,22 +215,55 @@ async def generate_podcast_command(
             profile["name"]: profile for profile in speaker_profiles
         }
 
+        # Map speaker_profile record ID -> name so podcast-creator keeps
+        # receiving speaker names (its EpisodeProfile.speaker_config is a
+        # required non-empty name string, cross-referenced against the
+        # speakers config keyed by name).
+        speaker_name_by_id = {
+            str(profile["id"]): profile["name"] for profile in speaker_profiles
+        }
+
         # 5. Inject resolved model configs into profile dicts
         # Resolve ALL episode profiles (podcast-creator validates all).
         # Remove profiles that fail resolution to prevent validation errors.
         for ep_name in list(episode_profiles_dict.keys()):
             ep_dict = episode_profiles_dict[ep_name]
+
+            # Since migration 20, speaker_config stores a record ID (and is
+            # None when the referenced speaker profile no longer exists).
+            # Rewrite it back to the speaker name for podcast-creator; drop
+            # profiles whose reference doesn't resolve so a single orphaned
+            # profile can't fail validation for the whole config. The profile
+            # being generated always resolves: its speaker was validated above.
+            speaker_ref = ep_dict.get("speaker_config")
+            speaker_name = (
+                speaker_name_by_id.get(str(speaker_ref)) if speaker_ref else None
+            )
+            if not speaker_name and ep_name == episode_profile.name:
+                speaker_name = speaker_profile.name
+            if not speaker_name:
+                logger.warning(
+                    f"Episode profile '{ep_name}' references a speaker profile "
+                    f"that no longer exists ({speaker_ref!r}), removing from "
+                    "config to prevent validation errors"
+                )
+                del episode_profiles_dict[ep_name]
+                continue
+            ep_dict["speaker_config"] = speaker_name
+
             try:
                 if ep_dict.get("outline_llm"):
                     prov, model, conf = await _resolve_model_config(
-                        str(ep_dict["outline_llm"])
+                        str(ep_dict["outline_llm"]),
+                        max_tokens=ep_dict.get("max_tokens"),
                     )
                     ep_dict["outline_provider"] = prov
                     ep_dict["outline_model"] = model
                     ep_dict["outline_config"] = conf
                 if ep_dict.get("transcript_llm"):
                     prov, model, conf = await _resolve_model_config(
-                        str(ep_dict["transcript_llm"])
+                        str(ep_dict["transcript_llm"]),
+                        max_tokens=ep_dict.get("max_tokens"),
                     )
                     ep_dict["transcript_provider"] = prov
                     ep_dict["transcript_model"] = model
@@ -176,6 +279,18 @@ async def generate_podcast_command(
         # Remove profiles that fail resolution to prevent validation errors.
         for sp_name in list(speaker_profiles_dict.keys()):
             sp_dict = speaker_profiles_dict[sp_name]
+            if not sp_dict.get("voice_model") and sp_name != speaker_profile.name:
+                # podcast-creator requires tts_provider/tts_model on every
+                # profile in the config, so one unconfigured profile (the
+                # seeded ones ship without a voice model) would fail every
+                # generation (#1450). The selected profile never gets here
+                # without a voice model: it was validated above.
+                logger.warning(
+                    f"Speaker profile '{sp_name}' has no voice model, removing "
+                    "from config to prevent validation errors"
+                )
+                del speaker_profiles_dict[sp_name]
+                continue
             if sp_dict.get("voice_model"):
                 try:
                     prov, model, conf = await _resolve_model_config(
@@ -228,6 +343,18 @@ async def generate_podcast_command(
         )
         await episode.save()
 
+        # SECURITY NOTE for future work: podcast_creator also supports
+        # configure("templates", {...}), which compiles the given string
+        # directly as Jinja2 template *source* (Prompter(template_text=...)
+        # in podcast_creator/config.py) - the exact SSTI shape already fixed
+        # in open_notebook/graphs/transformation.py (GHSA-f35w-wx37-26q7).
+        # We don't call it today (confirmed: no code path here sets the
+        # "templates" key, so podcast generation always uses the file-based
+        # prompts/podcast/*.jinja templates in this repo). If a "custom
+        # podcast template" feature is ever added, do NOT wire user/profile
+        # text into configure("templates", ...) - render it through a
+        # fixed, developer-authored template with the user text passed in
+        # as a plain variable instead, matching transformation.py's fix.
         configure("speakers_config", {"profiles": speaker_profiles_dict})
         configure("episode_config", {"profiles": episode_profiles_dict})
 
@@ -236,7 +363,7 @@ async def generate_podcast_command(
         logger.info(f"Generated briefing (length: {len(briefing)} chars)")
 
         # 7. Create output directory using UUID for filesystem-safe paths
-        episode_dir_name, output_dir = build_episode_output_dir(DATA_FOLDER)
+        episode_dir_name, output_dir = build_episode_output_dir()
         output_dir.mkdir(parents=True, exist_ok=True)
 
         logger.info(f"Created output directory: {output_dir}")
@@ -253,15 +380,38 @@ async def generate_podcast_command(
             episode_profile=episode_profile.name,
         )
 
-        episode.audio_file = (
-            str(result.get("final_output_file_path")) if result else None
+        # podcast-creator reports audio-combination failures IN-BAND: on
+        # ffmpeg/clip errors combine_audio_files() returns an "ERROR: ..."
+        # string in final_output_file_path instead of a path. Detect it
+        # before path conversion so the real error surfaces (below, after
+        # the transcript/outline are persisted) instead of a misleading
+        # "outside the podcasts folder" ValueError.
+        raw_audio_path = result.get("final_output_file_path") if result else None
+        audio_error: Optional[str] = None
+        if raw_audio_path is not None and str(raw_audio_path).startswith("ERROR:"):
+            audio_error = str(raw_audio_path)
+            raw_audio_path = None
+
+        # Store the audio path RELATIVE to PODCASTS_FOLDER (#1030). The
+        # validation inside to_relative_audio_path guarantees the DB never
+        # holds an absolute or root-escaping value; a violation raises
+        # ValueError, which marks the job permanently failed (no retry).
+        audio_file_rel = (
+            to_relative_audio_path(raw_audio_path) if raw_audio_path else None
         )
+        episode.audio_file = audio_file_rel
         episode.transcript = {
             "transcript": full_model_dump(result["transcript"]) if result else None
         }
         episode.outline = full_model_dump(result["outline"]) if result else None
         await episode.save()
 
+        if audio_error:
+            # Transcript/outline are saved above; fail the job with the real
+            # audio-combination error instead of reporting a silent success
+            # for an episode with no playable audio.
+            raise RuntimeError(f"Podcast audio generation failed: {audio_error}")
+
         processing_time = time.time() - start_time
         logger.info(
             f"Successfully generated podcast episode: {episode.id} in {processing_time:.2f}s"
@@ -270,9 +420,7 @@ async def generate_podcast_command(
         return PodcastGenerationOutput(
             success=True,
             episode_id=str(episode.id),
-            audio_file_path=str(result.get("final_output_file_path"))
-            if result
-            else None,
+            audio_file_path=audio_file_rel,
             transcript={"transcript": full_model_dump(result["transcript"])}
             if result.get("transcript")
             else None,
@@ -282,19 +430,27 @@ async def generate_podcast_command(
             processing_time=processing_time,
         )
 
-    except ValueError:
-        raise
+    except ValueError as e:
+        # ValueError is the command layer's "permanent failure, do not retry"
+        # signal (retry config uses stop_on=[ValueError]), so the type has to
+        # survive - but LangChain's OutputParserException and
+        # json.JSONDecodeError are ValueError subclasses too. Every parser
+        # failure therefore left through here, past the hint mapper below:
+        # the placeholder speaker name and the truncated-JSON cases reached
+        # the user with no guidance at all (#1238).
+        hint = explain_generation_failure(str(e))
+        if not hint:
+            raise
+        logger.error(f"Podcast generation failed: {e}")
+        raise ValueError(f"{e}\n\nNOTE: {hint}") from e
 
     except Exception as e:
         logger.error(f"Podcast generation failed: {e}")
         logger.exception(e)
 
         error_msg = str(e)
-        if "Invalid json output" in error_msg or "Expecting value" in error_msg:
-            error_msg += (
-                "\n\nNOTE: This error commonly occurs with GPT-5 models that use extended thinking. "
-                "The model may be putting all output inside <think> tags, leaving nothing to parse. "
-                "Try using gpt-4o, gpt-4o-mini, or gpt-4-turbo instead in your episode profile."
-            )
+        hint = explain_generation_failure(error_msg)
+        if hint:
+            error_msg += f"\n\nNOTE: {hint}"
 
         raise RuntimeError(error_msg) from e

commands/source_commands.py

@@ -1,14 +1,20 @@
 import time
 from typing import Any, Dict, List, Optional
 
+from langchain_core.runnables import RunnableConfig
 from loguru import logger
-from pydantic import BaseModel
 from surreal_commands import CommandInput, CommandOutput, command
 
 from open_notebook.database.repository import ensure_record_id
 from open_notebook.domain.notebook import Source
 from open_notebook.domain.transformation import Transformation
-from open_notebook.exceptions import ConfigurationError
+from open_notebook.exceptions import (
+    ConfigurationError,
+    ContextLengthExceededError,
+    IncompleteGenerationError,
+    InvalidInputError,
+    NotFoundError,
+)
 
 try:
     from open_notebook.graphs.source import source_graph
@@ -18,17 +24,6 @@ except ImportError as e:
     raise ValueError("graphs not available")
 
 
-def full_model_dump(model):
-    if isinstance(model, BaseModel):
-        return model.model_dump()
-    elif isinstance(model, dict):
-        return {k: full_model_dump(v) for k, v in model.items()}
-    elif isinstance(model, list):
-        return [full_model_dump(item) for item in model]
-    else:
-        return model
-
-
 class SourceProcessingInput(CommandInput):
     source_id: str
     content_state: Dict[str, Any]
@@ -54,7 +49,15 @@ class SourceProcessingOutput(CommandOutput):
         "wait_strategy": "exponential_jitter",
         "wait_min": 1,
         "wait_max": 120,  # Allow queue to drain
-        "stop_on": [ValueError, ConfigurationError],  # Don't retry validation/config errors
+        "stop_on": [
+            ValueError,
+            ConfigurationError,
+            ContextLengthExceededError,
+            IncompleteGenerationError,
+            NotFoundError,
+        ],  # Don't retry validation/config errors, incomplete generations, or a
+        # record deleted mid-processing (NotFoundError means "missing", never a
+        # DB failure: ObjectModel.get raises DatabaseOperationError for those)
         "retry_log_level": "debug",  # Avoid log noise during transaction conflicts
     },
 )
@@ -76,7 +79,11 @@ async def process_source_command(
         transformations = []
         for trans_id in input_data.transformations:
             logger.info(f"Loading transformation: {trans_id}")
-            transformation = await Transformation.get(trans_id)
+            try:
+                transformation = await Transformation.get(trans_id)
+            except NotFoundError as e:
+                # Same as a deleted source below: permanent, not transient.
+                raise ValueError(f"Transformation '{trans_id}' no longer exists") from e
             if not transformation:
                 raise ValueError(f"Transformation '{trans_id}' not found")
             transformations.append(transformation)
@@ -84,7 +91,18 @@ async def process_source_command(
         logger.info(f"Loaded {len(transformations)} transformations")
 
         # 2. Get existing source record to update its command field
-        source = await Source.get(input_data.source_id)
+        try:
+            source = await Source.get(input_data.source_id)
+        except NotFoundError as e:
+            # The source was removed after this job was queued (e.g. a sync tool
+            # deleted it, or the record was cleaned up). The job can never
+            # succeed: raise a permanent error (ValueError is in `stop_on`) so
+            # surreal-commands marks it failed instead of spending 15 retries
+            # with exponential backoff, which starves every job behind it.
+            raise ValueError(
+                f"Source '{input_data.source_id}' no longer exists "
+                "(deleted before processing?)"
+            ) from e
         if not source:
             raise ValueError(f"Source '{input_data.source_id}' not found")
 
@@ -101,9 +119,11 @@ async def process_source_command(
         # 3. Process source with all notebooks
         logger.info(f"Processing source with {len(input_data.notebook_ids)} notebooks")
 
-        # Execute source_graph with all notebooks
-        result = await source_graph.ainvoke(
-            {  # type: ignore[arg-type]
+        # Execute source_graph with all notebooks.
+        # LangGraph accepts a partial state dict at runtime, but its typed
+        # overloads require the full state type (langgraph typing limitation).
+        result = await source_graph.ainvoke(  # type: ignore[call-overload]
+            {
                 "content_state": input_data.content_state,
                 "notebook_ids": input_data.notebook_ids,  # Use notebook_ids (plural) as expected by SourceState
                 "apply_transformations": transformations,
@@ -126,9 +146,7 @@ async def process_source_command(
         logger.info(
             f"Successfully processed source: {processed_source.id} in {processing_time:.2f}s"
         )
-        logger.info(
-            f"Created {insights_created} insights, embedding {embed_status}"
-        )
+        logger.info(f"Created {insights_created} insights, embedding {embed_status}")
 
         return SourceProcessingOutput(
             success=True,
@@ -138,21 +156,30 @@ async def process_source_command(
             processing_time=processing_time,
         )
 
-    except ValueError as e:
-        # Validation errors are permanent failures - don't retry
-        processing_time = time.time() - start_time
-        logger.error(f"Source processing failed: {e}")
-        return SourceProcessingOutput(
-            success=False,
-            source_id=input_data.source_id,
-            processing_time=processing_time,
-            error_message=str(e),
+    except IncompleteGenerationError as e:
+        logger.error(
+            f"Generation failed (permanent) for source {input_data.source_id}: {e}"
         )
+        raise  # Preserve failed job status; stop_on prevents automatic retries.
+    except NotFoundError as e:
+        # E.g. the source was deleted while extraction ran (save_source re-reads
+        # it). Permanent: stop_on prevents retries that would starve the queue.
+        logger.error(
+            f"Source processing failed (permanent), record no longer exists: {e}"
+        )
+        raise
+    except ValueError as e:
+        # Validation errors are permanent failures. Re-raise so surreal-commands
+        # marks the job as `failed` (stop_on=[ValueError] already prevents
+        # pointless retries). Returning a success=False result instead marks the
+        # job `completed` (is_success() checks job status, not the payload),
+        # which hid extraction failures and left the source without a retryable
+        # `failed` status in the UI.
+        logger.error(f"Source processing failed (permanent): {e}")
+        raise
     except Exception as e:
         # Transient failure - will be retried (surreal-commands logs final failure)
-        logger.debug(
-            f"Transient error processing source {input_data.source_id}: {e}"
-        )
+        logger.debug(f"Transient error processing source {input_data.source_id}: {e}")
         raise
 
 
@@ -186,8 +213,16 @@ class RunTransformationOutput(CommandOutput):
         "wait_strategy": "exponential_jitter",
         "wait_min": 1,
         "wait_max": 60,
-        "stop_on": [ValueError, ConfigurationError],  # Don't retry validation/config errors
-        "retry_log_level": "debug",
+        "stop_on": [
+            ValueError,
+            ConfigurationError,
+            ContextLengthExceededError,
+            IncompleteGenerationError,
+            InvalidInputError,
+            NotFoundError,
+        ],  # Don't retry validation/config errors, incomplete generations, or
+        # a source/transformation deleted before the job ran
+        "retry_log_level": "warning",
     },
 )
 async def run_transformation_command(
@@ -229,9 +264,12 @@ async def run_transformation_command(
                 f"Transformation '{input_data.transformation_id}' not found"
             )
 
-        # Run transformation graph (includes LLM call + insight creation)
-        await transform_graph.ainvoke(
-            input=dict(source=source, transformation=transformation)
+        # Run transformation graph (includes LLM call + insight creation).
+        # LangGraph accepts a partial state dict at runtime, but its typed
+        # overloads require the full state type (langgraph typing limitation).
+        await transform_graph.ainvoke(  # type: ignore[call-overload]
+            input=dict(source=source, transformation=transformation),
+            config=RunnableConfig(configurable={"model_id": transformation.model_id}),
         )
 
         processing_time = time.time() - start_time
@@ -247,6 +285,13 @@ async def run_transformation_command(
             processing_time=processing_time,
         )
 
+    except (IncompleteGenerationError, InvalidInputError, NotFoundError) as e:
+        # e.g. the source has no text to transform
+        logger.error(
+            f"Generation failed (permanent) for transformation "
+            f"{input_data.transformation_id} on source {input_data.source_id}: {e}"
+        )
+        raise  # Preserve failed job status; stop_on prevents automatic retries.
     except ValueError as e:
         # Validation errors are permanent failures - don't retry
         processing_time = time.time() - start_time

cubic.yaml

@@ -0,0 +1,102 @@
+# yaml-language-server: $schema=https://cubic.dev/schema/cubic-repository-config.schema.json
+#
+# cubic AI review settings (settings as code).
+# Docs: https://docs.cubic.dev/configure/cubic-yaml
+#
+# Platform mechanics (verified against docs.cubic.dev, Jul 2026):
+# - Max 5 active custom agents (custom_rules) per repo. Each agent's text plus any
+#   file-backed content (file_paths) shares a 10,000-character limit that is truncated
+#   SILENTLY — only link short, stable files.
+# - Config precedence: repo cubic.yaml > org config (`cubic-config` repo) > dashboard UI
+#   > defaults. Schema: https://cubic.dev/schema/cubic-repository-config.schema.json
+# - Replying to a cubic review comment teaches it permanently (team-scoped); manage
+#   learnings under AI review settings → Memory & Learning.
+# - cubic auto-detects context files (AGENTS.md, README, .github/) — those inform the
+#   review; custom agents are what actually enforce rules.
+
+version: 1
+
+reviews:
+  # Skip files with no reviewable logic to preserve the monthly reviewed-line quota:
+  # CHANGELOG entries are prose (and every PR + rebase re-reviews them); lockfiles
+  # are generated.
+  ignore:
+    files:
+      - 'CHANGELOG.md'
+      - 'uv.lock'
+      - 'frontend/package-lock.json'
+
+  custom_instructions: |
+    This project follows an issue-first workflow for anything non-trivial: features and
+    architecture changes must reference an approved, assigned issue and stay scoped to it.
+    Small obvious fixes (typos, docs, tiny bugs, i18n completions) are welcome without an
+    issue. A sizeable PR without one should be converted to draft with an issue opened —
+    don't flag small fixes for lacking an issue. Unrelated refactors belong in separate PRs.
+    Test evidence must be real (actual output), especially for bug fixes, which should
+    include a regression test. Many PRs are agent-generated; hold them to the same bar.
+    Normative rules live in AGENTS.md (root, open_notebook/, frontend/); product direction
+    lives in VISION.md; past decisions in docs/7-DEVELOPMENT/decisions/.
+
+  custom_rules:
+    - name: Vision & principles alignment
+      description: |
+        Check the PR against the product identity and current posture defined in the
+        linked VISION.md. Flag:
+        - Features that gratuitously preclude multi-user (hard-coded single-tenancy in
+          schema, auth, or data scoping) — see PDR-001.
+        - Capabilities that only work on one AI provider without a PDR justifying the
+          exclusivity — the core is provider-agnostic by default (PDR-002).
+        - Features that conflict with the "What Open Notebook IS NOT" list.
+        - Structural/architectural decisions introduced without a decision record in
+          docs/7-DEVELOPMENT/decisions/ (half a page, same PR).
+      file_paths:
+        - VISION.md
+
+    - name: Known caveats
+      description: |
+        Enforce this project's recurring mechanical pitfalls (full rules in
+        open_notebook/AGENTS.md and frontend/AGENTS.md):
+        - i18n: every user-visible frontend string goes through t('section.key') and the
+          key must exist in ALL 7 locales (en-US, pt-BR, zh-CN, zh-TW, ja-JP, ru-RU, bn-IN).
+        - Providers: adding/changing a provider = edit the registry
+          (open_notebook/ai/provider_registry.py PROVIDERS) plus the SupportedProvider
+          Literal in api/models.py; everything else (TEST_MODELS, PROVIDER_ENV_CONFIG,
+          the frontend via GET /api/providers) is derived from the registry.
+        - Migrations: a new open_notebook/database/migrations/N.surrealql file must also be
+          registered in AsyncMigrationManager (async_migrate.py) — no auto-discovery.
+        - Graphs: LLM calls in LangGraph nodes must use provision_langchain_model() and wrap
+          errors with classify_error(); model output goes through clean_thinking_content().
+        - Domain: Source.save() does not auto-embed — vectorize() must be called explicitly;
+          RecordModel singletons need clear_instance() in tests.
+        - Async jobs (podcasts, embeddings, source processing) run on the surreal-commands
+          worker; features depending on them must not assume inline execution. Permanent
+          failures raise ValueError (no retry); podcasts use max_attempts=1 by design.
+        - Frontend: FormData nested fields must be JSON.stringify-ed; don't re-add
+          Content-Type; check hasHydrated before rendering persisted Zustand state.
+
+    - name: Security & testability
+      description: |
+        Security:
+        - API endpoints must never return API key or secret values — metadata only.
+        - Every user-supplied URL must pass validate_url() (SSRF protection; private IPs are
+          intentionally allowed for self-hosted services).
+        - No secrets committed, logged, or echoed in error messages; credential fields use
+          SecretStr and Fernet encryption (OPEN_NOTEBOOK_ENCRYPTION_KEY).
+        - Raise typed exceptions from open_notebook.exceptions instead of bare HTTPException
+          or generic 500s.
+        Testability:
+        - Business logic belongs in services/domain, not in routers or React components,
+          so it can be tested directly.
+        - Critical paths (auth, credentials, encryption, migrations, data deletion) need
+          test coverage in the same PR.
+        - Bug fixes should include a test that reproduces the bug.
+
+  auto_ultrareview: custom
+  auto_ultrareview_custom_prompt: |
+    Run an ultrareview when the PR touches authentication, credential storage, encryption,
+    database migrations, or data deletion/cascade logic.
+  auto_ultrareview_file_patterns:
+    - 'open_notebook/database/migrations/**'
+    - '**/credential*'
+    - '**/auth*'
+    - '**/encryption*'

dev-init.sh

@@ -29,7 +29,7 @@ sleep 3
 
 # Start background worker in background
 echo "Starting background worker..."
-uv run --env-file .env surreal-commands-worker --import-modules commands &
+uv run --env-file .env surreal-commands-worker --import-modules commands --max-tasks "${OPEN_NOTEBOOK_WORKER_MAX_TASKS:-5}" &
 sleep 2
 
 # Start frontend (foreground)

docker-compose.override.yml.example

@@ -0,0 +1,23 @@
+# Local Docker Compose override — copy to `docker-compose.override.yml`
+# (which `docker compose up` merges automatically) to apply host-specific
+# tweaks without touching the tracked docker-compose.yml.
+#
+#   cp docker-compose.override.yml.example docker-compose.override.yml
+#
+# The example below re-exposes SurrealDB's port on all interfaces. The
+# shipped default binds it to 127.0.0.1 only, because the database starts
+# with root:root credentials and exposing it on 0.0.0.0 lets anyone who can
+# reach the host connect as root. Only re-expose it if you actually need to
+# reach the database from another machine (e.g. Surrealist on your laptop),
+# and put it behind a firewall or an SSH tunnel and set SURREAL_USER /
+# SURREAL_PASSWORD to real credentials first.
+
+services:
+  surrealdb:
+    # `!override` replaces the base file's port list instead of merging with
+    # it — without it, compose keeps both entries and the container fails to
+    # start with "port is already allocated". Requires Docker Compose
+    # v2.24.4+; on older clients, edit the ports entry in docker-compose.yml
+    # directly instead.
+    ports: !override
+      - "8000:8000"

docker-compose.yml

@@ -1,10 +1,21 @@
 services:
   surrealdb:
     image: surrealdb/surrealdb:v2
-    command: start --log info --user root --pass root rocksdb:/mydata/mydatabase.db
+    # Credentials default to root:root for a zero-config local setup. Before
+    # exposing this instance to a network, set SURREAL_USER / SURREAL_PASSWORD
+    # in a .env file (see .env.example) — they are applied here and to the
+    # open_notebook service below, so the two always stay in sync.
+    # List (exec) form so each interpolated value stays a single argument —
+    # a password containing spaces would otherwise be split into several.
+    command: ["start", "--log", "info", "--user", "${SURREAL_USER:-root}", "--pass", "${SURREAL_PASSWORD:-root}", "rocksdb:/mydata/mydatabase.db"]
     user: root  # Required for bind mounts on Linux
     ports:
-      - "8000:8000"
+      # Bound to localhost only: the open_notebook service reaches this over
+      # the internal compose network regardless, so the host port is purely
+      # for local debugging (e.g. Surrealist, `surreal sql`). Exposing this
+      # on 0.0.0.0 would let anyone who can reach the host connect with the
+      # default root:root credentials.
+      - "127.0.0.1:8000:8000"
     volumes:
       - ./surreal_data:/mydata
     environment:
@@ -22,12 +33,27 @@ services:
       # This encrypts your API keys in the database
       - OPEN_NOTEBOOK_ENCRYPTION_KEY=change-me-to-a-secret-string
 
-      # Database connection (default values - no need to change)
+      # Database connection. SURREAL_USER / SURREAL_PASSWORD default to root:root
+      # for local use; override them in a .env file before exposing the instance
+      # (the same values configure the surrealdb service above).
       - SURREAL_URL=ws://surrealdb:8000/rpc
-      - SURREAL_USER=root
-      - SURREAL_PASSWORD=root
+      - SURREAL_USER=${SURREAL_USER:-root}
+      - SURREAL_PASSWORD=${SURREAL_PASSWORD:-root}
       - SURREAL_NAMESPACE=open_notebook
       - SURREAL_DATABASE=open_notebook
+
+      # OPTIONAL: heavy extraction runtimes, installed on first startup when
+      # enabled (downloads are cached on the ./notebook_data volume, so only the
+      # first boot is slow). See docs/5-CONFIGURATION/environment-reference.md.
+      # - OPEN_NOTEBOOK_ENABLE_DOCLING=true    # Docling engine + OCR + image sources (large ML stack)
+      # - OPEN_NOTEBOOK_ENABLE_CRAWL4AI=true   # local Crawl4AI (bundles a Chromium browser)
+      # - CRAWL4AI_API_URL=http://crawl4ai:11235  # use a remote Crawl4AI server instead (no local install)
+      # - CRAWL4AI_API_TOKEN=${CRAWL4AI_API_TOKEN}  # bearer token for that server (required by Crawl4AI >= 0.9.0)
+
+      # OPTIONAL: max concurrent background tasks the worker processes at once
+      # (default 5). Set to 1 for sequential processing on single-GPU / local
+      # LLM setups to avoid overloading the model with parallel requests.
+      # - OPEN_NOTEBOOK_WORKER_MAX_TASKS=1
     volumes:
       - ./notebook_data:/app/data
     depends_on: