From ae53779ccd8a9504090f7144f238cea0d3b1159e Mon Sep 17 00:00:00 2001 From: Ouroboros Date: Wed, 10 Jun 2026 15:28:34 +0300 Subject: [PATCH] =?UTF-8?q?release:=20Ouroboros=20v6.26.0-rc.1=20=E2=80=94?= =?UTF-8?q?=20systemic=20hardening:=20immune=20system,=20custody,=20memory?= =?UTF-8?q?=20integrity,=20native=20multimodal=20chat?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Owner-approved big release (plan: большой_релиз_v6.26.0). Workstreams: - WS-A lint gate: ruff F-rules step in CI quick-test; fixed the real F823 NameError class (supervisor/events.py utc_now_iso shadowing); make lint/health. - WS-B memory integrity: atomic dialogue_blocks with corrupt-quarantine (.corrupt-.bak + memory_store_corrupt event); honest scratchpad journal (block_append_failed, corrupt storage renders as corruption not "(empty)"); full Pattern Register window (16K cap vs 3K cut); merge-aware scratchpad consolidation under the sidecar lock; backlog writes through locked helpers; chat omission notes ("[N older unconsolidated messages omitted]"). - WS-C provider SSOT: provider registry (prefixes/credentials/resolution) in provider_models.py replacing 5 duplicated knowledge sites; credential-aware consolidation model; pricing empty-fetch retry; direct-route correctness (o-series max_completion_tokens, reasoning_effort, Anthropic error bodies, per-request timeouts on cached clients); deep/plan review budgets reserve output headroom inside the 1M window (min(SSOT, window − output − margin)). - WS-D races/state: utils.update_json_locked (locked RMW, loud TimeoutError); task_results merge under per-file lock (cancel-latch safe); update_state migration for owner binding, evolution counters, budget_line, post-task activation; queue-lock coverage (enqueue, timeouts, worker health, task_done, snapshot); visible supervisor death (supervisor_error + owner notice + ghost consciousness/chat-agent cleanup on re-init); chat.jsonl rotation via os.replace under the append lock; unbounded _outbox removed; host-service port probe + actual-port panic sweep; skill lifecycle lane deadline (OUROBOROS_SKILL_LIFECYCLE_TIMEOUT_SEC, dedupe-leak-safe); consilium force-plan structured flag (plan_review_aggregate from the FULL result); atomic writers (update-intent, repo-manifest, metrics cache, post-task pair); review_state.save_state raises on lock timeout (advisory ledger honesty); apply_pending_request no-ops while a campaign is active; api_update_apply kills workers only AFTER update validation, with respawn on aborted checkout and "interrupted" terminal status. - WS-E immune hardening: triad anti-refusal coverage contract (empty array needs NO_FINDINGS sentinel or bare-[] body; refusal prose with [] is parse_failure and never enters quorum); BIBLE P3 "Owner-chosen enforcement, loud advisory" bound + CHECKLISTS sync; every advisory pass-through of a blocking signal writes review_advisory_override + persistent advisory_overrides (surfaced by review_status); ToolEntry.mutates_worktree with dispatcher-level before/after worktree diff invalidation (covers error paths; read-only runs no longer invalidate; redundant manual calls removed); scope review fails closed without its checklist; advisory final write is a locked re-read merge; synthesis severity defaults to critical with WARNING fallbacks; find removed from SAFE_SHELL_COMMANDS; anti-thrashing state survives advisory criticals; preflight env scrubs secret-class variables; .git/index.lock age gate in worker startup checks. - WS-F security: file-browser symlink containment on the RESOLVED path across all endpoints (out-of-root symlink targets are listed but inert; tests rewritten to the new contract deliberately); HMAC-signed session cookies (server-side persisted key, 30-day TTL, Secure on TLS) replacing the permanent password-derived cookie; password-class settings mask to a constant placeholder; conservative SSRF guard for the MAIN agent (link-local /cloud-metadata only, LAN stays reachable, per-request route re-validation); ClawHub zip-slip hardening (":"/backslash segments rejected + post-join containment) and lazy no-proxy OuroborosHub opener; single-execution signature dispatch for extension handlers (no TypeError re-run after side effects); onboarding postMessage origin checks; SHA256-pinned python-standalone download with pipefail; payload-resident dependency fingerprints only corroborate durable deps.json; skill payload re-hash immediately before spawn (TOCTOU narrowing). - WS-G process custody: ouroboros/process_custody.py — spawn_supervised chokepoint + durable data/state/process_ledger.jsonl (pid, pgid, fingerprint{start_time, cmd_sha256}, purpose, scope task|session|daemon, owner_task, session_id); platform_layer.process_start_time primitive; startup + periodic reaper killing ONLY strict-fingerprint matches from dead generations/tasks (never by command-line class); migrations: services (the orphan hole), workspace executor + local model + extension companions (ledger write-through), worker_pids (write-through; legacy path retained); parent lifelines (ppid watchdog, group-suicide only as group leader) in worker_main, extension runner, Claude readonly child; conformance test pinning the Popen allowlist; ARCHITECTURE/DEVELOPMENT/CHECKLISTS entries. - WS-H native multimodal chat: supports_vision capability map (static prefixes + OpenRouter /models input_modalities overlay); web chat uploads ride the WS frame as structured attachments (additive ChatInbound field) and image uploads become NATIVE image blocks via the existing Path B; browser screenshots inject natively for vision models via the multipart user-merge (tool result stays a string; file persisted under data/uploads/screenshots for re-view); K=3 newest-image eviction with caption placeholders carrying the vlm_query re-view path; image-aware token estimates (fixed ~1.1K-token equivalent instead of base64 length, fixing permanent emergency-compaction wedges); compaction renders images as captions (no base64 into the summarizer); GigaChat/local lanes emit explicit "[image omitted: model has no vision]"; internal _caption/_source_path metadata stripped from provider payloads. - WS-I housekeeping (partial): SETTLED_STATUSES SSOT (+ cycle-safe mirror pin); owner_inject.py renamed to owner_mailbox.py; version-neutral envelope wording; files.py import-block cleanup. Remaining WS-I/WS-J/WS-K items are deferred with the owner's context-budget priority on review+release. Review notes: triad+scope ran via scripts/run_external_review.py on the core pack across 3 rounds to convergence (scope responded=PASS each round; round-2 criticals fixed: update_apply kill-order + respawn, lifecycle dedupe leak on lane timeout, OUROBOROS_MAX_ROUNDS hot-reload + docs, toggle_evolution NameError, async Anthropic timeout forwarding, supports_vision local check, budget-update lock visibility, ChatInbound additive attachment contract, file-browser doc sync). The FULL combined diff exceeds every triad model's context window (~1.59M tokens > 1.05M) — reviewed in packs; remaining cross-pack findings were verified as slicing artifacts. Adversarial critics (GPT; Gemini/Opus rounds) ran on the working tree. Deliberate tradeoffs: metadata-based eviction captions (no light-LLM call in the hot path); worker ledger records use live-cmdline fingerprints with a synthetic-arg fallback only where the OS offers no cmdline. --- .github/workflows/ci.yml | 4 +- BIBLE.md | 19 +- .../4419367024 | 10 + .../4712967760 | 10 + Makefile | 11 +- README.md | 4 +- VERSION | 2 +- .../benchmarks/swe_bench_pro/grade_pro.py | 2 +- docs/ARCHITECTURE.md | 27 +- docs/CHECKLISTS.md | 7 + docs/DEVELOPMENT.md | 16 + ouroboros/agent.py | 2 +- ouroboros/agent_startup_checks.py | 11 +- ouroboros/agent_task_pipeline.py | 55 +-- ouroboros/cli.py | 3 +- ouroboros/config.py | 7 +- ouroboros/consciousness.py | 8 +- ouroboros/consolidator.py | 104 ++++-- ouroboros/context.py | 23 +- ouroboros/context_budget.py | 17 + ouroboros/context_compaction.py | 19 +- ouroboros/contracts/plugin_api.py | 2 +- ouroboros/deep_self_review.py | 64 ++-- ouroboros/extension_companion.py | 17 + ouroboros/extension_process_runner.py | 40 ++- ouroboros/gateway/contracts.py | 12 + ouroboros/gateway/control.py | 32 +- ouroboros/gateway/files.py | 23 +- ouroboros/gateway/settings.py | 20 +- ouroboros/gateway/tasks.py | 4 +- ouroboros/gateway/ws.py | 48 +++ ouroboros/gateways/claude_code.py | 6 + ouroboros/headless.py | 6 +- ouroboros/launcher_bootstrap.py | 8 +- ouroboros/llm.py | 85 ++++- ouroboros/local_model.py | 16 +- ouroboros/loop.py | 92 ++++- ouroboros/loop_llm_call.py | 3 - ouroboros/loop_tool_execution.py | 5 + ouroboros/marketplace/fetcher.py | 12 + ouroboros/marketplace/isolated_deps.py | 7 +- ouroboros/marketplace/ouroboroshub.py | 25 +- ouroboros/mcp_client.py | 4 +- ouroboros/memory.py | 86 +++-- ouroboros/observability.py | 1 - ouroboros/outcomes.py | 2 +- .../{owner_inject.py => owner_mailbox.py} | 0 ouroboros/platform_layer.py | 34 ++ ouroboros/post_task_evolution.py | 33 +- ouroboros/preflight_runner.py | 11 +- ouroboros/pricing.py | 46 +-- ouroboros/process_custody.py | 338 ++++++++++++++++++ ouroboros/provider_models.py | 118 +++++- ouroboros/reflection.py | 10 +- ouroboros/review_evidence.py | 13 + ouroboros/review_state.py | 11 +- ouroboros/review_substrate.py | 2 +- ouroboros/safety.py | 4 +- ouroboros/server_auth.py | 79 +++- ouroboros/server_control.py | 10 +- ouroboros/skill_lifecycle_queue.py | 57 ++- ouroboros/skill_loader.py | 3 - ouroboros/skill_review.py | 1 - ouroboros/skill_review_runner.py | 5 - ouroboros/task_results.py | 67 ++-- ouroboros/task_status.py | 8 + ouroboros/tools/browser.py | 109 +++++- ouroboros/tools/ci.py | 2 +- ouroboros/tools/claude_advisory_review.py | 34 +- ouroboros/tools/compact_context.py | 2 +- ouroboros/tools/core.py | 3 +- ouroboros/tools/extension_dispatch.py | 11 +- ouroboros/tools/git.py | 7 +- ouroboros/tools/git_pr.py | 38 +- ouroboros/tools/github.py | 2 +- ouroboros/tools/health.py | 1 - ouroboros/tools/knowledge.py | 22 +- ouroboros/tools/memory_tools.py | 1 - ouroboros/tools/parallel_review.py | 2 - ouroboros/tools/plan_review.py | 12 +- ouroboros/tools/query_code.py | 1 - ouroboros/tools/registry.py | 57 ++- ouroboros/tools/review.py | 54 ++- ouroboros/tools/review_helpers.py | 4 +- ouroboros/tools/review_synthesis.py | 13 +- ouroboros/tools/scope_review.py | 16 +- ouroboros/tools/services.py | 17 +- ouroboros/tools/shell.py | 16 +- ouroboros/tools/skill_exec.py | 20 ++ ouroboros/tools/tool_discovery.py | 8 +- ouroboros/tools/vision.py | 5 +- ouroboros/triad_review.py | 30 ++ ouroboros/utils.py | 92 ++++- ouroboros/workspace_executor.py | 20 ++ pyproject.toml | 2 +- scripts/build_repo_bundle.py | 1 - scripts/download_python_standalone.sh | 22 +- server.py | 121 ++++++- supervisor/events.py | 108 +++--- supervisor/git_ops.py | 9 +- supervisor/message_bus.py | 61 ++-- supervisor/queue.py | 86 +++-- supervisor/state.py | 41 ++- supervisor/workers.py | 37 +- tests/test_acting_subagents.py | 3 - tests/test_advisory_observability.py | 2 +- tests/test_advisory_preflight.py | 2 - tests/test_browser_isolation.py | 6 +- tests/test_budget_limits.py | 1 - tests/test_budget_tracking.py | 4 +- tests/test_bughunt_fixes.py | 2 - tests/test_build_scripts.py | 2 +- tests/test_claude_code_gateway.py | 9 +- tests/test_code_inventory_v1.py | 2 +- tests/test_commit_gate.py | 1 - tests/test_compaction.py | 2 +- tests/test_consolidator.py | 1 - tests/test_context.py | 2 - tests/test_crash_report.py | 1 - tests/test_deep_self_review.py | 9 +- tests/test_event_bus.py | 2 +- tests/test_evolution_redesign.py | 1 - tests/test_evolution_stop_and_cost.py | 1 - tests/test_extension_isolated_deps.py | 33 -- tests/test_extension_loader.py | 5 +- tests/test_extension_process_runner.py | 17 - tests/test_extension_surfaces.py | 20 -- tests/test_extensions_api.py | 8 +- tests/test_file_browser_api.py | 35 +- tests/test_git_review_pipeline.py | 8 +- tests/test_immune_hardening.py | 191 ++++++++++ tests/test_isolated_deps.py | 1 - tests/test_llm_no_proxy.py | 22 +- tests/test_llm_provider_routing.py | 1 + tests/test_llm_supported_parameters.py | 1 - tests/test_loop_compaction_policy.py | 26 +- tests/test_loop_misc.py | 3 + tests/test_marketplace_fetcher.py | 1 - tests/test_marketplace_provenance_contract.py | 1 - tests/test_max_tokens_constants.py | 52 ++- tests/test_mcp_api.py | 2 +- tests/test_memory_blocks.py | 1 - tests/test_memory_tool_hints.py | 1 - tests/test_message_routing.py | 16 +- tests/test_multimodal_chat.py | 222 ++++++++++++ tests/test_observability_outcomes_v2.py | 1 - tests/test_observability_retention.py | 2 +- tests/test_oop_extension_parity.py | 1 - tests/test_per_skill_version_resync.py | 1 - tests/test_platform_guard.py | 15 - tests/test_post_task_evolution.py | 9 +- tests/test_pricing.py | 4 - tests/test_process_custody.py | 226 ++++++++++++ tests/test_process_memory.py | 1 - tests/test_release_sync.py | 1 - tests/test_repo_commit_dirty_tree.py | 1 - tests/test_repo_read_limits.py | 2 - tests/test_restart_reconnect.py | 5 + tests/test_review_fidelity.py | 13 +- tests/test_review_observability.py | 6 +- tests/test_review_readiness_gate.py | 2 - tests/test_runtime_mode_elevation.py | 2 - tests/test_scope_review.py | 1 - tests/test_scratchpad_consolidation.py | 1 - tests/test_send_photo.py | 2 - tests/test_settings_budget_hotreload.py | 8 +- tests/test_settings_effort.py | 1 - tests/test_shell_regex_hint.py | 2 - tests/test_skill_exec.py | 5 - tests/test_skill_loader.py | 6 +- tests/test_skill_review.py | 1 - tests/test_skill_review_persist_guard.py | 2 +- tests/test_smoke.py | 6 +- tests/test_task_constraint_server_routing.py | 1 + tests/test_task_constraint_tools.py | 1 - tests/test_task_done_ordering.py | 2 - tests/test_tool_capabilities.py | 1 - tests/test_transcript_seal.py | 1 - tests/test_transport_commands.py | 5 + tests/test_vision.py | 6 +- tests/test_worker_crash_retry.py | 6 +- tests/test_zombie_prevention.py | 1 - web/modules/api_types.js | 2 +- web/modules/chat.js | 11 + web/modules/onboarding_overlay.js | 8 +- web/modules/onboarding_wizard.js | 4 +- web/package.json | 2 +- 187 files changed, 3159 insertions(+), 857 deletions(-) create mode 100644 MagicMock/mock.drive_root.__truediv__().__truediv__()/4419367024 create mode 100644 MagicMock/mock.drive_root.__truediv__().__truediv__()/4712967760 rename ouroboros/{owner_inject.py => owner_mailbox.py} (100%) create mode 100644 ouroboros/process_custody.py create mode 100644 tests/test_immune_hardening.py create mode 100644 tests/test_multimodal_chat.py create mode 100644 tests/test_process_custody.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 143b1593a..29ad345ed 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -61,7 +61,9 @@ jobs: - name: Install dependencies run: | pip install -r requirements.txt - pip install pytest + pip install pytest ruff + - name: Lint (deterministic F-rule gate — catches the NameError-under-except class) + run: python -m ruff check . --select F - name: Run tests run: python -m pytest tests/ -q --tb=short - name: Guard extracted transport imports stay out of core diff --git a/BIBLE.md b/BIBLE.md index ce1cffafa..16c29bcce 100644 --- a/BIBLE.md +++ b/BIBLE.md @@ -203,10 +203,13 @@ may not weaken it. ### Components -1. **Multi-model blocking review** at every commit. Diff reviewers - score the staged diff against the checklists in - [docs/CHECKLISTS.md](docs/CHECKLISTS.md); a blocking scope reviewer - examines goal / intent / coupling using broader repository context. +1. **Multi-model review** at every commit. Diff reviewers score the + staged diff against the checklists in + [docs/CHECKLISTS.md](docs/CHECKLISTS.md); a scope reviewer examines + goal / intent / coupling using broader repository context. The gate + is blocking under `blocking` enforcement; under owner-chosen + `advisory` enforcement it still runs in full and every decision that + blocking would have stopped is loudly and durably recorded. 2. **Advisory pre-review**: a cheaper preflight on the same snapshot, mandatory before the blocking review; staleness-aware (any worktree-mutating tool invalidates the snapshot). @@ -257,6 +260,14 @@ following bounds are constitutional: [docs/CHECKLISTS.md](docs/CHECKLISTS.md), is a named exception scoped to cosmetic changes and release-artifact metadata. Every bypass is durably audited. Silent bypass is forbidden. +- **Owner-chosen enforcement, loud advisory.** The owner selects review + enforcement (`blocking` or `advisory`). Advisory enforcement is + legitimate ONLY while every decision that blocking enforcement would + have stopped — critical findings, quorum failure, infrastructure + failure, missing advisory provider — leaves a loud, durable, + owner-visible trace at the moment it happens. Silent advisory is + forbidden; an advisory mode that hides what it waved through is a + weakened immune system, not a configured one. - **Self-modification gate.** Any change to these bounds (scope floor, context floor, bypass rules, durable-memory permanence) is itself a constitutional change and requires plan review. Changes that diff --git a/MagicMock/mock.drive_root.__truediv__().__truediv__()/4419367024 b/MagicMock/mock.drive_root.__truediv__().__truediv__()/4419367024 new file mode 100644 index 000000000..7290ac138 --- /dev/null +++ b/MagicMock/mock.drive_root.__truediv__().__truediv__()/4419367024 @@ -0,0 +1,10 @@ +{ + "count": 1, + "recent": [ + { + "ts": "2026-06-10T12:38:44.359372+00:00", + "block_reason": "review_quorum", + "message_head": "⚠️ REVIEW_BLOCKED: Only 1 of 1 review models responded successfully (minimum 2 required). Unavailable/failed: .\nRetry the commit — transient model failures usually resolve quickly." + } + ] +} \ No newline at end of file diff --git a/MagicMock/mock.drive_root.__truediv__().__truediv__()/4712967760 b/MagicMock/mock.drive_root.__truediv__().__truediv__()/4712967760 new file mode 100644 index 000000000..bb49955d3 --- /dev/null +++ b/MagicMock/mock.drive_root.__truediv__().__truediv__()/4712967760 @@ -0,0 +1,10 @@ +{ + "count": 1, + "recent": [ + { + "ts": "2026-06-10T12:30:14.271758+00:00", + "block_reason": "review_quorum", + "message_head": "⚠️ REVIEW_BLOCKED: Only 1 of 1 review models responded successfully (minimum 2 required). Unavailable/failed: .\nRetry the commit — transient model failures usually resolve quickly." + } + ] +} \ No newline at end of file diff --git a/Makefile b/Makefile index a4d685e9b..497f66c9f 100644 --- a/Makefile +++ b/Makefile @@ -11,12 +11,17 @@ test: test-v: python3 -m pytest tests/ -v --tb=long +# Lint: deterministic F-rule gate (NameError class); matches the CI quick-test step +lint: + python3 -m ruff check . --select F + # Run codebase health check (requires ouroboros importable) health: - python3 -c "from ouroboros.review import compute_complexity_metrics; \ + python3 -c "from ouroboros.review import collect_sections, compute_complexity_metrics; \ import pathlib, json; \ - m = compute_complexity_metrics(pathlib.Path('.')); \ - print(json.dumps(m, indent=2, default=str))" + sections, stats = collect_sections(pathlib.Path('.'), pathlib.Path('../data')); \ + m = compute_complexity_metrics(sections); \ + print(json.dumps({'repo': stats, **m}, indent=2, default=str))" # Clean Python cache files clean: diff --git a/README.md b/README.md index 6968fc01f..0645b77f2 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ [![Linux](https://img.shields.io/badge/Linux-x86__64-orange.svg)](https://github.com/razzant/ouroboros/releases) [![Windows](https://img.shields.io/badge/Windows-x64-blue.svg)](https://github.com/razzant/ouroboros/releases) [![OuroborosHub](https://img.shields.io/badge/OuroborosHub-skills%20marketplace-8A2BE2.svg)](https://github.com/razzant/OuroborosHub) -[![Version 6.25.0-rc.3](https://img.shields.io/badge/version-6.25.0--rc.3-green.svg)](VERSION) +[![Version 6.26.0-rc.1](https://img.shields.io/badge/version-6.26.0--rc.1-green.svg)](VERSION) A self-modifying AI agent that writes its own code, rewrites its own mind, and evolves autonomously. Born February 16, 2026. @@ -520,6 +520,7 @@ the contribution guide only routes to those sources. | Version | Date | Description | |---------|------|-------------| +| 6.26.0-rc.1 | 2026-06-10 | **release: systemic hardening — immune system, custody, memory integrity, native multimodal chat.** Provider registry SSOT + credential-aware lanes; locked RMW state (`update_json_locked`, queue locks, visible supervisor death); memory integrity (atomic+quarantined dialogue blocks, honest scratchpad journal, full Pattern Register window); immune hardening (triad anti-refusal NO_FINDINGS contract, BIBLE P3 loud-advisory bound with durable `advisory_overrides`, dispatcher-level `mutates_worktree` invalidation, fail-closed scope checklist, `find` de-whitelisted); security (file-browser symlink containment, HMAC session cookies, SSRF metadata guard, zip-slip hardening, single-exec dispatch, SHA256-pinned runtime download); process custody (`process_custody.py` ledger + strict-fingerprint reaper + parent lifelines); native multimodal chat (vision capability map, web uploads and browser screenshots as native image blocks, K=3 eviction with captions, image-aware token estimates); deterministic ruff F-gate in CI. | | 6.25.0-rc.3 | 2026-06-10 | **refactor(consolidation): first function-count paydown and gate headroom.** Owner-approved consolidation pass: remove 14 dead functions, fold ~10 duplicate-body clusters into SSOT survivors, inline ~30 trivial wrappers (3009 → 2953 gated functions, behavior preserved); raise `MAX_TOTAL_FUNCTIONS` to 3500 to stop gate churn after external PR merges landed the branch at zero slack; compress the gate-comment archaeology; drop the dormant file-size-budget parser (its DEVELOPMENT.md section no longer exists); sync DEVELOPMENT.md and add structural-gate rationale to ARCHITECTURE.md. Carries the rc.2 fix-forward surface unchanged. | | 6.25.0-rc.1 | 2026-06-09 | **feat(code-intelligence): add query_code, ripgrep search, and clearer self-improvement settings.** Adds a deterministic code-intelligence v2 layer with an incremental JSON index, derived-only symbol/reference/call facts for Python and JS/TS, the new read-only `query_code` tool (`symbols`, `definition`, `references`, `callers`, `callees`, `impact`, `structural`, `relevant_files`), and a ripgrep-backed `search_code` path with the existing safety filters and Python fallback. The review atlas and codebase digest consume the upgraded facts without changing review budgets or selection policy. Behavior settings now expose post-task self-improvement through one clear trigger selector, move background-cognition timing into its own section, remove the unused evolution cost-threshold setting, and clarify that Every-N counts every eligible task. Subagent tool calls are mirrored into canonical tool logs for swarm observability. | | 6.24.0-rc.4 | 2026-06-09 | **fix(self-evolution): fix-forward rc.4 after rc.3 CI exposed a stale browser-isolation assertion.** Keeps the rc.3 server-driven self-evolution hardening intact, updates the browser module-state test to expect both owner-control browser routes (`/api/owner/context-mode` and `/api/settings`), and ships the reviewed fix-forward release metadata after the `v6.24.0-rc.3` tag failed CI. No runtime behavior change beyond the rc.3 hardening. | @@ -529,7 +530,6 @@ the contribution guide only routes to those sources. | 6.23.3 | 2026-06-08 | **feat(self-evolution): thin per-project facts store + configurable memory carry (C2 / Phase 3).** A project-scoped task (an external/workspace task, or one given an explicit `--project-id`) now keeps its learned facts in a per-project knowledge store under the canonical data dir (`projects//knowledge`, `ouroboros/project_facts.py`), resolved via S7 (explicit id wins, else a stable hash of the workspace path). The knowledge tool (`tools/knowledge.py`) and context loader (`context.py`) redirect to that store when project-scoped, so prior project facts are available and isolated from the global `memory/knowledge` tree. The store lives under the canonical data dir (not a task's child drive) so it persists across forked/empty runs, and the leak guard (red-team R3.1) suppresses the post-task canonical dual-run for project-scoped tasks so project facts can never contaminate global memory or another project. No per-project identity; non-workspace tasks are unchanged (canonical memory). Configurable memory carry is unchanged in semantics (forked/empty/shared via `prepare_task_drive`); the evolutionary driver uses a persistent isolated data root. | | 6.23.2 | 2026-06-08 | **feat(self-evolution): improvement-backlog overhaul (C2 / Phase 2).** The durable improvement backlog stops losing signal: recurring items bump a `count` + `last_seen` instead of being dropped (and a recurrence re-opens a closed item); the digest is ranked by `priority` (high/med/low) then recurrence count then recency, so an important older item outranks a junk burst; `close_backlog_items` marks addressed items `done` with `closed_at`, and a promoted item is closed only when its reviewed self-mod commit is restart-verified and absorbed (`apply_pending_request` records `post_task_backlog_id`; `verify_restart` closes it on absorb — not merely on commit); and `groom_backlog` runs an LLM grooming pass (merge near-duplicates, mark resolved, cap the list) on a size-triggered, non-error-gated schedule, re-serialized through the locked parser-safe writer. Reflection nominations now carry optional `priority` and a `kind` (`bug`/`improvement`/`capability_idea`) so forward-looking capability ideas are captured, not only bugs. `CONSCIOUSNESS.md` backlog upkeep now points at the automated grooming instead of hand-editing. Additive schema; existing items remain valid. | | 6.23.1 | 2026-06-08 | **feat(self-evolution): post-task evolution envelope + budget refusal guard (C1).** Adds an owner-gated, default-OFF post-task self-evolution capability: `OUROBOROS_POST_TASK_EVOLUTION` (+ `_CADENCE` default `llm`, `_BUDGET_USD`) in `config.py`. After a qualifying task the worker may make an LLM-first decision (`ouroboros/post_task_evolution.py`) to promote ONE high-value code-class improvement by writing a durable signal; the supervisor's idle tick applies it through the existing gated `enqueue_evolution_task_if_needed` (idle/restart-verify/3-fail-breaker/reserve/advanced-pro/owner) — never enqueuing from the worker, never from evolution/subagent tasks, preserving the backlog `requires_plan_review` gate, with one-shot autostop. New `supervisor.state.reset_per_task_budget` lets evolutionary drivers reset the per-task budget ONLY in an isolated `OUROBOROS_DATA_DIR` (refuses the live data root, BIBLE P8). A zero-core `devtools/benchmarks/evolve_smoke.py` harness verifies carry + budget-reset + live-untouched on a throwaway clone. Identity stays candidate-only; default behavior unchanged. | -| 6.23.0-rc.1 | 2026-06-08 | **feat(subagents): adaptive progress-aware planning-swarm wait (completes the swarm fan-out/waits set).** `plan_task` planning scouts no longer fail to a fixed cutoff that discards in-flight work: `_collect_planning_handoffs` polls in slices and keeps extending while at least one non-terminal scout is RUNNING with a fresh heartbeat (from `queue_snapshot.json`), up to a generous `OUROBOROS_PLAN_TASK_SWARM_MAX_WAIT_SEC` ceiling (default 900s), then fails closed with a precise `wait_stop_reason` (`stalled`/`saturated`/`ceiling`). The wait/stop decision is a structured fact (terminal status + heartbeat freshness), not keyword matching (P5). Adds a "Timeout & Wait Control" guideline to DEVELOPMENT.md and a pointed progress-aware-waiting line to CHECKLISTS.md item #18. This rc completes the v6.23.0 swarm work begun in 6.22.2 (parallel `schedule_subagent` emission + parent-side lock) and 6.22.3 (fan-out telemetry + accept/reject markers). Dedup is unchanged. | Older releases are preserved in Git tags and GitHub releases. Older 6.x rows, the 5.2.0 through 5.33.0-rc.6 rows, and former `4.0.0` rows are rolled off to respect the P9 changelog cap; their full bodies remain at their git tags. --- diff --git a/VERSION b/VERSION index e2180cb10..b8e81e9a2 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -6.25.0-rc.3 +6.26.0-rc.1 diff --git a/devtools/benchmarks/swe_bench_pro/grade_pro.py b/devtools/benchmarks/swe_bench_pro/grade_pro.py index 4188a1f1f..c6d598fe0 100644 --- a/devtools/benchmarks/swe_bench_pro/grade_pro.py +++ b/devtools/benchmarks/swe_bench_pro/grade_pro.py @@ -8,7 +8,7 @@ per-instance output files. python pro/grade_pro.py --predictions runs/pro_smoke/predictions.jsonl --workers 4 """ from __future__ import annotations -import argparse, ast, csv, json, os, pathlib, subprocess, sys +import argparse, ast, csv, json, pathlib, subprocess, sys if __package__ in {None, ""}: sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[3])) diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index d641b2537..6c677baaf 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -1,4 +1,4 @@ -# Ouroboros v6.25.0-rc.3 — Architecture & Reference +# Ouroboros v6.26.0-rc.1 — Architecture & Reference This file is NOT a changelog. Version history lives in README.md, git tags, and commit log. @@ -77,7 +77,7 @@ server.py (Starlette+uvicorn) ← HTTP + WebSocket on configurable host:port (de ├── triad_review.py ← Shared multi-model review primitives: JSON-array extraction is reused by repo + skill review; per-actor records, quorum/degraded accounting, and model-error events power the skill-review path ├── onboarding_wizard.py ← Shared desktop/web onboarding bootstrap + validation ├── settings_setup_contract.py ← SSOT for Settings/Onboarding setup contract, derived bootstrap state, and setup payload validation - ├── owner_inject.py ← Per-task user message mailbox (compat module name) + ├── owner_mailbox.py ← Per-task user message mailbox (compat module name) ├── launcher_bootstrap.py ← Bundle-to-repo bootstrap and managed sync helpers (used by launcher.py) ├── provider_models.py ← Provider-specific model ID helpers, direct-provider defaults (OpenAI, Anthropic, Cloud.ru, GigaChat) ├── runtime_mode_policy.py ← Runtime-mode protected-path policy (safety-critical files, frozen contracts, release/managed invariants) shared by registry, git tools, and Claude gateway guards @@ -175,6 +175,18 @@ server.py (Starlette+uvicorn) ← HTTP + WebSocket on configurable host:port (de │ └── subagent_integration.py ← integrate_subagent_patch: parent's manifest-first integration of an acting subagent's workspace.patch. For self_worktree children it applies into ctx.active_repo_dir() (sha256-verified, 3-way --index, protected-path gated, top-only lineage check, genesis refused), stages but never commits. For external_workspace children it verifies the child wrote in the same active external workspace and records an audited verdict without re-applying the patch. Also compare_subagent_patches: read-only best-of-N helper that shows several children's candidate patches side by side for LLM-first synthesis └── platform_layer.py ← Cross-platform process/path/locking helpers + ouroboros/process_custody.py ← Supervised spawning + durable orphan ledger + (v6.26.0): `spawn_supervised()` records every long-lived child in + `data/state/process_ledger.jsonl` ({pid, pgid, fingerprint{start_time, + cmd_sha256}, purpose, scope task|session|daemon, owner_task, session_id}); + the reaper (server startup + 10-min supervisor tick) kills entries whose + generation/task owner is gone, matching by STRICT fingerprint only — + never by command-line class, so dev and packaged instances can coexist. + `start_parent_lifeline()` gives our python entrypoints (workers, + extension runner, claude readonly child) a ppid watchdog that + group-suicides when the parent dies. Panic layers (`_active_subprocesses`, + port sweeps, Windows Job Objects) are unchanged complements. + # Build & CI (not part of runtime) .github/workflows/ci.yml ← Four-tier CI (quick / full / integration / build+release) build.sh ← macOS build (PyInstaller → .dmg) @@ -790,7 +802,7 @@ History sync is intentionally two-pass: progress/system entries are replayed fir ### Files -`web/modules/files.js` is the browser file manager: directory tree, breadcrumbs, preview/editor, upload/download, copy/move, and write guards. Backend policy lives in `gateway/files.py`: root confinement is lexical; symlink targets may intentionally resolve outside the root; owner-only state and skill control-plane sidecars are protected. +`web/modules/files.js` is the browser file manager: directory tree, breadcrumbs, preview/editor, upload/download, copy/move, and write guards. Backend policy lives in `gateway/files.py`: root confinement is enforced on the RESOLVED path (v6.26.0) — symlinks whose target leaves the configured root are listed but not readable/writable/traversable; owner-only state and skill control-plane sidecars are protected. ### Skills, Marketplace, Widgets @@ -825,6 +837,13 @@ If `OUROBOROS_NETWORK_PASSWORD` is configured, non-loopback HTTP/WebSocket acces The executable route SSOT is `ouroboros/gateway/router.py`; file-browser routes come from `gateway/files.py::file_browser_routes()`, the contract index is `gateway/contracts.py::HTTP_ENDPOINTS`, and Host Service routes come from `gateway/host_service.py::create_host_service_app`. +File-browser symlink containment (v6.26.0): every `/api/files/*` endpoint +resolves the requested path and rejects it when the RESOLUTION leaves the +configured root (`Path escapes file browser root`). In-root symlinks keep +working; symlinks pointing outside the root are listed (with +`is_symlink: true`) but cannot be read, written, deleted, or traversed — +the old pass-through behavior was a root escape. + | Method | Path | Handler | |---|---|---| | GET | `/` | `server.index_page` | @@ -1333,6 +1352,8 @@ Runtime floors: | USE_LOCAL_LIGHT | false | Route light model to local server | | USE_LOCAL_CONSCIOUSNESS | false | Route background consciousness model slot to local server | | USE_LOCAL_FALLBACK | false | Route fallback model to local server | +| OUROBOROS_MAX_ROUNDS | 200 | Main-loop LLM round ceiling per task (hot-reloadable) | +| OUROBOROS_SKILL_LIFECYCLE_TIMEOUT_SEC | 1800 | Skill lifecycle lane deadline before a wedged job fails loudly | | OUROBOROS_BG_MAX_ROUNDS | 10 | Max LLM rounds per consciousness cycle | | OUROBOROS_BG_WAKEUP_MIN | 30 | Min wakeup interval (seconds) | | OUROBOROS_BG_WAKEUP_MAX | 7200 | Max wakeup interval (seconds) | diff --git a/docs/CHECKLISTS.md b/docs/CHECKLISTS.md index 3473a7941..ba3077b75 100644 --- a/docs/CHECKLISTS.md +++ b/docs/CHECKLISTS.md @@ -35,6 +35,12 @@ When a new reviewable concern appears, add it here — not in prompts or docs. `advisory_obligations_acknowledged` to `events.jsonl`; stale advisory still blocks. Under `blocking`, `commit_reviewed` can proceed only when no open obligations or commit-readiness debt remain. +- **Loud advisory enforcement (BIBLE P3 bound):** the owner chooses enforcement; + `advisory` is legitimate ONLY while every decision blocking enforcement would + have stopped (critical findings, quorum failure, infrastructure failure, + missing advisory provider) leaves a durable trace: a `review_advisory_override` + event in `events.jsonl` plus the persistent `advisory_overrides_count` / + recent-overrides fields in `review_status`. Silent advisory is forbidden. - Once advisory is fresh → call commit_reviewed immediately without further edits. - Bypass (`skip_advisory_review=True`) is an **absolute** escape hatch: it short-circuits the entire commit gate (freshness + open obligations + open commit-readiness debt). Every bypass is durably audited in events.jsonl. Open obligations/debt stay visible in `review_status` (`repo_commit_ready=false`) but do NOT block the bypassed commit. Reach for it when advisory cannot run (provider outage, rate limit) or when the stale signals are known to be obsolete. @@ -112,6 +118,7 @@ or Intent/Scope checklists are. | 14 | Changing extension loader/dispatch or isolated deps? | Native-risk extension imports and tool/route/WS handlers must stay out-of-process. Add or run regression tests where a native-risk plugin aborts during import and the host survives, plus tool/route child-dispatch tests. Do not "fix" failures by importing native-risk plugin code in `server.py`. | | 15 | Changing `supervisor/git_ops.py`, `launcher.py`, `server.py`, `ouroboros/tools/review_helpers.py`, `ouroboros/tools/git.py`, tests, or evolution scheduling/checkpoint code? | Prove two invariants before review spend: (1) pytest/preflight cannot mutate the live repo or live `data/` (`OUROBOROS_DATA_DIR` / `OUROBOROS_SETTINGS_PATH` must be isolated, and `OUROBOROS_MANAGED_BY_LAUNCHER` must not leak into test subprocesses); (2) autonomous restart/reset cannot erase active evolution work — it must either land a reviewed local commit or preserve a rescue/transaction recovery pointer and pause/stop the campaign. | | 16 | Changing `devtools/benchmarks/`? | Confirm it preserves official benchmark boundaries: no replacement scoring, no benchmark-specific prompt/routing hacks, no generated benchmark outputs under `repo/`, no secrets printed or committed, and no runtime-core imports from `devtools/`. Touched `devtools` files are reviewable executable operator code, even though unrelated `devtools` files use Atlas `excluded_dir` coverage-manifest entries and stay compact in broad packs. | +| 17 | Diff spawns OS processes (`subprocess.Popen` / `mp.Process` without a bounded wait)? | Route it through `ouroboros.process_custody.spawn_supervised` (or `record_process` write-through) with an explicit `scope` (`task`/`session`/`daemon`) so the orphan reaper can find it; `tests/test_process_custody.py` enforces the allowlist. | Rule: read before write. Never reconstruct `VERSION`, `pyproject.toml` `version`, or the README badge from memory — one stale reconstruction creates diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md index 71888aa1b..7fa3fc4c0 100644 --- a/docs/DEVELOPMENT.md +++ b/docs/DEVELOPMENT.md @@ -589,6 +589,22 @@ Before every commit, verify the following: --- +## Process Custody Rule + +Long-lived OS processes (anything `subprocess.Popen`-ed or `mp.Process`-ed +without a bounded wait in the same call) **MUST** be spawned through +`ouroboros.process_custody.spawn_supervised(cmd, drive_root=..., purpose=..., +scope=...)` — or, when an existing manager owns the Popen call, registered via +`record_process(...)` write-through immediately after spawn. The custody +ledger (`data/state/process_ledger.jsonl`) is what lets the orphan reaper find +children after an abrupt worker/server death; an unledgered process orphans +invisibly and forever. Scopes: `task` (dies with its task), `session` (dies +with the server generation), `daemon` (launcher-managed; reaper only prunes). +The reaper kills strictly by (pid, start_time, cmd_sha256) fingerprint — never +add command-line-class matching, which would let a dev instance reap a +packaged instance's processes. `tests/test_process_custody.py` enforces the +chokepoint with an explicit allowlist for bounded synchronous helpers. + ## Platform Abstraction Rule All platform-specific code **MUST** go through `ouroboros/platform_layer.py`. diff --git a/ouroboros/agent.py b/ouroboros/agent.py index 6da47ef59..665b1de70 100644 --- a/ouroboros/agent.py +++ b/ouroboros/agent.py @@ -38,7 +38,7 @@ from ouroboros.agent_startup_checks import ( verify_system_state, ) from ouroboros.agent_task_pipeline import ( - build_trace_summary, emit_task_results, build_review_context, + emit_task_results, build_review_context, ) from ouroboros.task_results import STATUS_RUNNING, write_task_result from ouroboros.contracts.task_constraint import normalize_task_constraint diff --git a/ouroboros/agent_startup_checks.py b/ouroboros/agent_startup_checks.py index a0502eb26..6b6578e35 100644 --- a/ouroboros/agent_startup_checks.py +++ b/ouroboros/agent_startup_checks.py @@ -6,6 +6,7 @@ import logging import os import re import subprocess +import time from typing import Any, Dict, Tuple from ouroboros.utils import atomic_write_json, utc_now_iso, read_text, append_jsonl, read_json_dict @@ -25,8 +26,14 @@ def check_uncommitted_changes(env: Any) -> Tuple[dict, int]: lock_path = env.repo_path(".git/index.lock") if lock_path.exists(): try: - lock_path.unlink() - log.warning("Removed stale git index.lock") + # Age gate (mirrors supervisor.git_ops._stale_git_lock_paths): + # an unconditional unlink could yank index.lock from under a + # LIVE supervisor git operation during worker boot, corrupting + # the index. Only locks orphaned by dead processes are stale. + age_sec = time.time() - lock_path.stat().st_mtime + if age_sec >= 15.0: + lock_path.unlink() + log.warning("Removed stale git index.lock (age %.0fs)", age_sec) except OSError: pass diff --git a/ouroboros/agent_task_pipeline.py b/ouroboros/agent_task_pipeline.py index cfd7f1c80..6a7ec25f6 100644 --- a/ouroboros/agent_task_pipeline.py +++ b/ouroboros/agent_task_pipeline.py @@ -4,7 +4,6 @@ from __future__ import annotations import json import logging -import os import pathlib import threading import time @@ -33,58 +32,8 @@ from ouroboros.utils import utc_now_iso, append_jsonl, truncate_review_artifact log = logging.getLogger(__name__) -def _resolve_task_summary_model(default_model: str) -> str: - prefix_to_provider = { - "openai::": "openai", - "anthropic::": "anthropic", - "cloudru::": "cloudru", - "gigachat::": "gigachat", - "openai-compatible::": "openai-compatible", - "openrouter::": "openrouter", - } - provider_env_keys: dict[str, list[str]] = { - "openai": ["OPENAI_API_KEY"], - "anthropic": ["ANTHROPIC_API_KEY"], - "cloudru": ["CLOUDRU_FOUNDATION_MODELS_API_KEY"], - "gigachat": ["GIGACHAT_CREDENTIALS"], - "openrouter": ["OPENROUTER_API_KEY"], - } - - def model_has_credentials(model: str) -> bool: - name = str(model or "").strip() - provider = "openrouter" - for prefix, candidate_provider in prefix_to_provider.items(): - if name.startswith(prefix): - provider = candidate_provider - break - if provider == "openai-compatible": - compat = str(os.environ.get("OPENAI_COMPATIBLE_API_KEY", "") or "").strip() - legacy_key = str(os.environ.get("OPENAI_API_KEY", "") or "").strip() - legacy_base = str(os.environ.get("OPENAI_BASE_URL", "") or "").strip() - return bool(compat or (legacy_key and legacy_base)) - if provider == "gigachat": - creds = str(os.environ.get("GIGACHAT_CREDENTIALS", "") or "").strip() - user = str(os.environ.get("GIGACHAT_USER", "") or "").strip() - password = str(os.environ.get("GIGACHAT_PASSWORD", "") or "").strip() - return bool(creds or (user and password)) - for env_key in provider_env_keys.get(provider, ["OPENROUTER_API_KEY"]): - if str(os.environ.get(env_key, "") or "").strip(): - return True - return False - - if model_has_credentials(default_model): - return default_model - - for env_name in ( - "OUROBOROS_MODEL_LIGHT", - "OUROBOROS_MODEL_FALLBACK", - "OUROBOROS_MODEL", - "OUROBOROS_MODEL_CODE", - ): - candidate = str(os.environ.get(env_name, "") or "").strip() - if candidate and model_has_credentials(candidate): - return candidate - return default_model +# Credential-aware model selection lives in the provider registry SSOT. +from ouroboros.provider_models import resolve_credentialed_model as _resolve_task_summary_model def build_trace_summary(llm_trace: dict) -> str: diff --git a/ouroboros/cli.py b/ouroboros/cli.py index 39d780bd2..6eb8955f1 100644 --- a/ouroboros/cli.py +++ b/ouroboros/cli.py @@ -802,7 +802,8 @@ def _patch_from_result( def _is_terminal_result(result: Dict[str, Any]) -> bool: status = str(result.get("status") or "").lower() - if status not in {"completed", "failed", "cancelled", "rejected_duplicate"}: + from ouroboros.task_status import SETTLED_STATUSES + if status not in SETTLED_STATUSES: return False bundle = result.get("artifact_bundle") if isinstance(result.get("artifact_bundle"), dict) else {} artifact_status = str(bundle.get("status") or result.get("artifact_status") or "").lower() diff --git a/ouroboros/config.py b/ouroboros/config.py index 44067dd61..a584a48b3 100644 --- a/ouroboros/config.py +++ b/ouroboros/config.py @@ -13,7 +13,7 @@ import pathlib import re import sys import time -from typing import Optional +from typing import Any, Optional from ouroboros.platform_layer import pid_lock_acquire as _compat_pid_lock_acquire from ouroboros.platform_layer import pid_lock_release as _compat_pid_lock_release @@ -103,6 +103,10 @@ SETTINGS_DEFAULTS = { "OUROBOROS_PLAN_TASK_SWARM_HEARTBEAT_STALE_SEC": 120, "TOTAL_BUDGET": 10.0, "OUROBOROS_PER_TASK_COST_USD": 20.0, + # Main-loop round ceiling (was an inline literal in loop.py — hot-reloadable now). + "OUROBOROS_MAX_ROUNDS": 200, + # Skill lifecycle lane deadline (wedged-job loud-failure bound). + "OUROBOROS_SKILL_LIFECYCLE_TIMEOUT_SEC": 1800, "OUROBOROS_SOFT_TIMEOUT_SEC": 600, "OUROBOROS_HARD_TIMEOUT_SEC": 1800, "OUROBOROS_FINALIZATION_GRACE_SEC": FINALIZATION_GRACE_DEFAULT_SEC, @@ -991,6 +995,7 @@ def apply_settings_to_env(settings: dict) -> None: "OUROBOROS_PLAN_TASK_SWARM_HEARTBEAT_STALE_SEC", "TOTAL_BUDGET", "OUROBOROS_PER_TASK_COST_USD", "GITHUB_TOKEN", "GITHUB_REPO", "OUROBOROS_TOOL_TIMEOUT_SEC", "OUROBOROS_FINALIZATION_GRACE_SEC", + "OUROBOROS_MAX_ROUNDS", "OUROBOROS_BG_MAX_ROUNDS", "OUROBOROS_BG_WAKEUP_MIN", "OUROBOROS_BG_WAKEUP_MAX", "OUROBOROS_WEBSEARCH_MODEL", "OUROBOROS_POST_TASK_EVOLUTION", "OUROBOROS_POST_TASK_EVOLUTION_CADENCE", diff --git a/ouroboros/consciousness.py b/ouroboros/consciousness.py index 6011f5ab0..a4b415f24 100644 --- a/ouroboros/consciousness.py +++ b/ouroboros/consciousness.py @@ -9,9 +9,11 @@ import os import pathlib import queue import threading -import time import traceback -from typing import Any, Callable, Dict, List, Optional +from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional + +if TYPE_CHECKING: + from ouroboros.tools.registry import ToolRegistry from ouroboros.loop_tool_execution import StatefulToolExecutor, _truncate_tool_result from ouroboros.utils import ( @@ -530,7 +532,7 @@ class BackgroundConsciousness: def _build_registry(self) -> "ToolRegistry": """Create a ToolRegistry scoped to background-allowed tools.""" - from ouroboros.tools.registry import ToolRegistry, ToolContext, ToolEntry + from ouroboros.tools.registry import ToolRegistry, ToolEntry registry = ToolRegistry(repo_dir=self._repo_dir, drive_root=self._drive_root) diff --git a/ouroboros/consolidator.py b/ouroboros/consolidator.py index f159679c1..80ce68c9b 100644 --- a/ouroboros/consolidator.py +++ b/ouroboros/consolidator.py @@ -2,7 +2,6 @@ import json import logging import os import pathlib -import hashlib from typing import Any, Dict, List, Optional, Tuple from ouroboros.contracts.chat_id_policy import is_a2a_chat_id @@ -20,6 +19,19 @@ BLOCK_SIZE = 100 # Messages per consolidation block MAX_SUMMARY_BLOCKS = 10 # Compress into era when exceeded ERA_COMPRESS_COUNT = 4 # Oldest blocks to compress per era CONSOLIDATION_MODEL = "google/gemini-3.5-flash" + + +def _consolidation_model() -> str: + """Credential-aware consolidation model (Provider Independence invariant). + + The default routes through OpenRouter; on single-direct-provider installs + that would silently fail every consolidation pass and freeze long-term + memory (offset never advances). Resolve through the provider registry so + any credentialed configured lane keeps memory consolidation alive. + """ + from ouroboros.provider_models import resolve_credentialed_model + + return resolve_credentialed_model(CONSOLIDATION_MODEL) CONSOLIDATION_REASONING_EFFORT = "medium" def should_consolidate( @@ -169,7 +181,7 @@ def _call_consolidation_llm(llm_client: Any, prompt: str, label: str) -> Tuple[s try: msg, usage = llm_client.chat( messages=[{"role": "user", "content": prompt}], - model=CONSOLIDATION_MODEL, + model=_consolidation_model(), tools=None, reasoning_effort="low", max_tokens=16384, @@ -283,19 +295,58 @@ def _load_blocks(path: pathlib.Path) -> List[Dict[str, Any]]: try: return json.loads(read_text(path)) except (json.JSONDecodeError, ValueError): - log.warning("Corrupt blocks file %s, starting fresh", path) + # Memory loss must never be silent (P1): quarantine the corrupt store + # for forensic recovery instead of overwriting it on the next write. + quarantine = path.with_name(f"{path.name}.corrupt-{utc_now_iso().replace(':', '')}.bak") + try: + os.replace(path, quarantine) + log.error("Corrupt blocks file %s — quarantined to %s, starting fresh", path, quarantine) + except OSError: + log.error("Corrupt blocks file %s — quarantine failed, starting fresh", path, exc_info=True) + try: + from ouroboros.utils import append_jsonl + + append_jsonl(path.parent.parent / "logs" / "events.jsonl", { + "ts": utc_now_iso(), + "type": "memory_store_corrupt", + "path": str(path), + "quarantine": str(quarantine), + }) + except Exception: + log.debug("Failed to emit memory_store_corrupt event", exc_info=True) return [] def _write_locked_json(path: pathlib.Path, payload: Any) -> None: + """Write JSON under the cross-process write lock, atomically. + + The lock serializes concurrent consolidators; the write itself goes + through a temp file + rename so a crash mid-write can never leave a + truncated dialogue_blocks.json (the long-term memory store). + """ + _mutate_locked_json_list(path, lambda _current: payload) + + +def _mutate_locked_json_list(path: pathlib.Path, mutator: Any) -> Any: + """Locked read-modify-write for a JSON list store (atomic replace). + + ``mutator(current_list) -> new_list`` runs while the sidecar lock is held, + so concurrent appenders cannot be lost between the re-read and the write. + """ path.parent.mkdir(parents=True, exist_ok=True) fd = None try: - fd = os.open(str(path), os.O_RDWR | os.O_CREAT, 0o644) + fd = os.open(str(path) + ".lock", os.O_RDWR | os.O_CREAT, 0o644) _lock_ex(fd) - os.lseek(fd, 0, os.SEEK_SET) - os.ftruncate(fd, 0) - os.write(fd, json.dumps(payload, ensure_ascii=False, indent=2).encode("utf-8")) + current: Any = [] + if path.exists(): + try: + current = json.loads(read_text(path)) + except (json.JSONDecodeError, ValueError): + current = [] + updated = mutator(current if isinstance(current, list) else []) + atomic_write_json(path, updated) + return updated finally: if fd is not None: try: @@ -308,19 +359,7 @@ def _load_meta(path: pathlib.Path) -> Dict[str, Any]: return read_json_dict(path) or {} -def _chat_log_signature(path: pathlib.Path) -> Dict[str, Any]: - if not path.exists(): - return {} - try: - stat = path.stat() - with path.open("r", encoding="utf-8") as handle: - first = next((line.strip() for line in handle if line.strip()), "") - return { - "first_line_sha256": hashlib.sha256(first.encode("utf-8", errors="replace")).hexdigest(), - "size": int(stat.st_size), - } - except OSError: - return {} +from ouroboros.utils import jsonl_generation_signature as _chat_log_signature def _count_lines(path: pathlib.Path) -> int: @@ -368,7 +407,9 @@ def _rebuild_knowledge_index(knowledge_dir: pathlib.Path) -> None: except Exception: log.warning("Failed to rebuild knowledge index", exc_info=True) -SCRATCHPAD_CONSOLIDATION_THRESHOLD = 30000 +from ouroboros.context_budget import ( + SCRATCHPAD_CONSOLIDATION_THRESHOLD_CHARS as SCRATCHPAD_CONSOLIDATION_THRESHOLD, +) def should_consolidate_scratchpad(memory: Any) -> bool: @@ -405,7 +446,6 @@ def _consolidate_scratchpad_blocks( compress_count = max(2, len(blocks) // 2) old_blocks = blocks[:compress_count] - recent_blocks = blocks[compress_count:] old_content = "\n\n---\n\n".join( f"[{b.get('ts', '?')[:16]} \u2014 {b.get('source', '?')}]\n{b.get('content', '')}" @@ -439,7 +479,7 @@ Respond with JSON only (no fences): try: msg, usage = llm_client.chat( messages=[{"role": "user", "content": prompt}], - model=CONSOLIDATION_MODEL, + model=_consolidation_model(), reasoning_effort="low", max_tokens=16384, use_local=os.environ.get("USE_LOCAL_LIGHT", "").lower() in ("true", "1"), @@ -463,9 +503,23 @@ Respond with JSON only (no fences): "source": "consolidation", "content": compressed_text.strip(), } - new_blocks = [compressed_block] + recent_blocks - _write_locked_json(memory.scratchpad_blocks_path(), new_blocks) + # Merge-aware replace UNDER the write lock: blocks appended DURING the + # slow LLM call live only on disk — building the new list from the + # pre-call snapshot would silently drop them. Re-read inside the lock + # and keep every block outside the compressed window (ts+source key). + compressed_keys = { + (str(b.get("ts") or ""), str(b.get("source") or "")) for b in old_blocks + } + + def _merge_survivors(live_blocks: List[Dict[str, Any]]) -> List[Dict[str, Any]]: + survivors = [ + b for b in live_blocks + if (str(b.get("ts") or ""), str(b.get("source") or "")) not in compressed_keys + ] + return [compressed_block] + survivors + + new_blocks = _mutate_locked_json_list(memory.scratchpad_blocks_path(), _merge_survivors) memory.regenerate_scratchpad_md() log.info("Scratchpad blocks consolidated: %d blocks (%d chars) -> %d blocks (%d chars)", diff --git a/ouroboros/context.py b/ouroboros/context.py index e613542f5..86dde1095 100644 --- a/ouroboros/context.py +++ b/ouroboros/context.py @@ -18,6 +18,8 @@ from ouroboros.context_budget import ( CONTEXT_SOFT_CAP_TOKENS, LARGE_CONTEXT_SECTION_CHARS, MAX_RECENT_CHAT_TAIL, + SCRATCHPAD_BLOAT_WARN_CHARS, + SCRATCHPAD_SECTION_BUDGET_CHARS, ) from ouroboros.context_layout import ( architecture_context_section, @@ -65,7 +67,13 @@ def build_user_content(task: Dict[str, Any]) -> Any: combined_text = "\n".join(part for part in (image_caption, text if text != image_caption else "") if part) or "Analyze the screenshot" return [ {"type": "text", "text": combined_text}, - {"type": "image_url", "image_url": {"url": f"data:{task.get('image_mime', 'image/jpeg')};base64,{image_b64}"}}, + { + "type": "image_url", + "image_url": {"url": f"data:{task.get('image_mime', 'image/jpeg')};base64,{image_b64}"}, + # Eviction metadata (stripped before provider calls): the K-newest + # image policy replaces older blocks with this caption. + "_caption": str(image_caption or "")[:200], + }, ] @@ -271,7 +279,7 @@ def build_governance_sections(env: Any, *, warn_large: bool = False, warn_label: return sections -_SECTION_BUDGETS = {"scratchpad": 90_000, "identity": 80_000, "registry": 30_000} +_SECTION_BUDGETS = {"scratchpad": SCRATCHPAD_SECTION_BUDGET_CHARS, "identity": 80_000, "registry": 30_000} def _warn_if_over_budget(name: str, content: str) -> None: @@ -395,7 +403,9 @@ def build_recent_sections(memory: Memory, env: Any, task_id: str = "") -> List[s consolidated_offset, _chat_tail, ) - chat_summary = memory.summarize_chat(chat_entries) + # Pass the same tail intent down: summarize_chat's internal default cap + # would silently re-cut the low-mode full-window read to 1000 lines. + chat_summary = memory.summarize_chat(chat_entries, limit=_chat_tail) if chat_summary: sections.append("## Recent chat\n\n" + chat_summary) @@ -646,7 +656,7 @@ def build_health_invariants(env: Any) -> str: sp_len = len(read_text(env.drive_path("memory/scratchpad.md")).strip()) if sp_len < 50: checks.append("WARNING: EMPTY SCRATCHPAD — scratchpad is nearly empty. Memory loss signal.") - elif sp_len > 50000: + elif sp_len > SCRATCHPAD_BLOAT_WARN_CHARS: checks.append(f"WARNING: BLOATED SCRATCHPAD — {sp_len} chars. Extract durable insights to knowledge base.") else: checks.append(f"OK: scratchpad size ({sp_len} chars)") @@ -969,11 +979,6 @@ def apply_message_token_soft_cap( return messages, info -from ouroboros.context_compaction import ( - _COMPACTION_PROTECTED_TOOLS, - compact_tool_history, - compact_tool_history_llm, -) def safe_read(path: pathlib.Path, fallback: str = "") -> str: diff --git a/ouroboros/context_budget.py b/ouroboros/context_budget.py index 77cb3ef1f..1e56c9efe 100644 --- a/ouroboros/context_budget.py +++ b/ouroboros/context_budget.py @@ -53,3 +53,20 @@ MAX_RECENT_CHAT_TAIL = 1000 # Low fires emergency tool-history compaction earlier (~100K tokens at chars/4) # to fit a ~200K window, and (unlike max) also enables remote routine compaction. LOW_EMERGENCY_COMPACTION_CHARS = 400_000 + +# --- Native image blocks (v6.26.0 multimodal chat) --------------------------- +# Char-equivalent for ONE image block in chars/4 token estimates (~1.1K tokens): +# vision models bill per tile, not per base64 char. +IMAGE_BLOCK_CHAR_EQUIVALENT = 4_400 +# Live image blocks kept in the transcript (single counter across owner +# uploads, browser screenshots, and transport injections). Older images are +# replaced by a caption placeholder pointing to the re-view path. +MAX_LIVE_IMAGE_BLOCKS = 3 + +# --- Scratchpad size thresholds (SSOT; previously scattered literals) ------- +# Context-section soft budget for the rendered scratchpad (warn-only). +SCRATCHPAD_SECTION_BUDGET_CHARS = 90_000 +# Health-invariant bloat warning ("extract durable insights to knowledge"). +SCRATCHPAD_BLOAT_WARN_CHARS = 50_000 +# Block-storage consolidation trigger (consolidator compresses oldest blocks). +SCRATCHPAD_CONSOLIDATION_THRESHOLD_CHARS = 30_000 diff --git a/ouroboros/context_compaction.py b/ouroboros/context_compaction.py index 505bd5431..395fb5914 100644 --- a/ouroboros/context_compaction.py +++ b/ouroboros/context_compaction.py @@ -147,7 +147,22 @@ def _render_round_block(messages: list, start: int, end: int) -> str: lines.append(f"TOOL_RESULT {tool_name}:") lines.append(_excerpt_for_summary(content)) elif role == "user": - content = str(msg.get("content") or "") + content = msg.get("content") + if isinstance(content, list): + # Multipart user content: never str() an image block (the + # base64 payload would flood the light summarizer's prompt). + parts = [] + for block in content: + if not isinstance(block, dict): + parts.append(str(block)) + elif str(block.get("type") or "") in ("image_url", "image"): + caption = str(block.get("_caption") or "").strip() + parts.append(f"[image: {caption or 'omitted'}]") + else: + parts.append(str(block.get("text", ""))) + content = "\n".join(part for part in parts if part) + else: + content = str(content or "") lines.append("USER_INPUT:") lines.append(_excerpt_for_summary(content)) return "\n".join(lines).strip() @@ -254,7 +269,7 @@ def compact_tool_history_llm( return messages, None spans_to_keep = spans[-keep_recent:] - keep_starts = {start for start, _ in spans_to_keep} + {start for start, _ in spans_to_keep} compactable_spans = [] protected_starts = set() for start, end in spans[:-keep_recent]: diff --git a/ouroboros/contracts/plugin_api.py b/ouroboros/contracts/plugin_api.py index 65a49b5a4..ac307bbb4 100644 --- a/ouroboros/contracts/plugin_api.py +++ b/ouroboros/contracts/plugin_api.py @@ -10,7 +10,7 @@ from __future__ import annotations import enum import pathlib -from typing import Any, Awaitable, Callable, Dict, List, Protocol, Sequence, runtime_checkable +from typing import Any, Awaitable, Callable, Dict, Protocol, Sequence, runtime_checkable from ouroboros.skill_token import SkillToken diff --git a/ouroboros/deep_self_review.py b/ouroboros/deep_self_review.py index 8ea0e4fa9..fa21d9c04 100644 --- a/ouroboros/deep_self_review.py +++ b/ouroboros/deep_self_review.py @@ -15,12 +15,12 @@ from ouroboros.tools.review_context_atlas import ( # noqa: E402 compile_review_context_atlas, ) from ouroboros.tools.review_helpers import ( # noqa: E402 - _BINARY_SNIFF_BYTES, _MAX_FULL_REPO_FILE_BYTES, - _is_probably_binary, + REVIEW_PROMPT_TOKEN_BUDGET, ) from ouroboros.utils import atomic_write_json, estimate_tokens, utc_now_iso # noqa: E402 from ouroboros.config import get_context_mode, get_deep_self_review_model, resolve_effort # noqa: E402 +from ouroboros.provider_models import provider_for_model, provider_has_credentials # noqa: E402 from ouroboros.context_layout import generate_doc_nav_map # noqa: E402 # Non-agent visual assets. @@ -28,6 +28,18 @@ _SKIP_DIR_PREFIXES = ( "assets/", ) +# Output reservation inside the reviewer's 1M window (same class of fix as +# scope_review._SCOPE_INPUT_TOKEN_LIMIT): 920K input + 100K output exceeds 1M +# and yields a deterministic provider 400, so the assembled INPUT prompt is +# gated on min(SSOT budget, window − output − tokenizer margin). +_DEEP_MAX_OUTPUT_TOKENS = 100_000 +_DEEP_MODEL_CONTEXT_WINDOW = 1_000_000 +_DEEP_OUTPUT_MARGIN_TOKENS = 155_000 +_DEEP_INPUT_TOKEN_LIMIT = min( + REVIEW_PROMPT_TOKEN_BUDGET, + _DEEP_MODEL_CONTEXT_WINDOW - _DEEP_MAX_OUTPUT_TOKENS - _DEEP_OUTPUT_MARGIN_TOKENS, +) + _MEMORY_WHITELIST = [ "memory/identity.md", "memory/scratchpad.md", @@ -162,8 +174,8 @@ def build_review_pack( tracked_paths=tuple(tracked), already_included=already_included, fixed_prompt_tokens=atlas_fixed_tokens, - target_total_tokens=850_000, - hard_total_tokens=920_000, + target_total_tokens=min(850_000, _DEEP_INPUT_TOKEN_LIMIT), + hard_total_tokens=_DEEP_INPUT_TOKEN_LIMIT, include_tests=False, title="Generated Deep Self-Review Atlas", ) @@ -197,29 +209,27 @@ def build_review_pack( def is_review_available() -> Tuple[bool, Optional[str]]: - """Return whether a suitable large-context review model is configured.""" + """Return whether a suitable large-context review model is configured. + + Provider/credential knowledge comes from the provider registry SSOT; the + one deliberate deep-review-specific rule kept here: ``openai::`` is only + trusted when ``OPENAI_BASE_URL`` is unset (a redirected endpoint cannot be + assumed to honor the 1M-context contract this review depends on). + """ configured = get_deep_self_review_model() - if configured.startswith("openai::"): - if os.environ.get("OPENAI_API_KEY") and not os.environ.get("OPENAI_BASE_URL"): + provider = provider_for_model(configured) + if provider == "openai": + if provider_has_credentials("openai") and not os.environ.get("OPENAI_BASE_URL"): return True, configured return False, None if configured.startswith("openai/"): - if os.environ.get("OPENROUTER_API_KEY"): + # OpenRouter route with a direct-OpenAI rewrite fallback. + if provider_has_credentials("openrouter"): return True, configured - if os.environ.get("OPENAI_API_KEY") and not os.environ.get("OPENAI_BASE_URL"): + if provider_has_credentials("openai") and not os.environ.get("OPENAI_BASE_URL"): return True, "openai::" + configured.split("/", 1)[1] return False, None - if configured.startswith("anthropic::"): - return (True, configured) if os.environ.get("ANTHROPIC_API_KEY") else (False, None) - if configured.startswith("cloudru::"): - return (True, configured) if os.environ.get("CLOUDRU_FOUNDATION_MODELS_API_KEY") else (False, None) - if configured.startswith("gigachat::"): - has_giga = bool(os.environ.get("GIGACHAT_CREDENTIALS") or (os.environ.get("GIGACHAT_USER") and os.environ.get("GIGACHAT_PASSWORD"))) - return (True, configured) if has_giga else (False, None) - if configured.startswith("openai-compatible::"): - has_compat = bool(os.environ.get("OPENAI_COMPATIBLE_API_KEY") or (os.environ.get("OPENAI_API_KEY") and os.environ.get("OPENAI_BASE_URL"))) - return (True, configured) if has_compat else (False, None) - if os.environ.get("OPENROUTER_API_KEY"): + if provider_has_credentials(provider): return True, configured return False, None @@ -253,16 +263,18 @@ def run_deep_self_review( + (f", {len(stats['skipped'])} skipped" if stats["skipped"] else "") ) - # Gate full system+pack like scope/plan review; chars/4 undercounts near - # the 1M window, so the prompt budget remains a best-effort guard. - from ouroboros.tools.review_helpers import REVIEW_PROMPT_TOKEN_BUDGET + # Gate full system+pack like scope review: reserve output headroom + # inside the 1M window (min(SSOT, window − output − margin)) so a large + # pack cannot trigger the deterministic input+output>window provider 400. full_prompt_chars = len(_SYSTEM_PROMPT) + len(pack_text) estimated_tokens = estimate_tokens(_SYSTEM_PROMPT + pack_text) - if estimated_tokens > REVIEW_PROMPT_TOKEN_BUDGET: + if estimated_tokens > _DEEP_INPUT_TOKEN_LIMIT: return ( f"❌ Review pack too large: ~{estimated_tokens:,} tokens " f"({full_prompt_chars:,} chars of system+pack, {stats['file_count']} files). " - f"Maximum is ~{REVIEW_PROMPT_TOKEN_BUDGET:,} tokens. Reduce codebase size or split review." + f"Maximum is ~{_DEEP_INPUT_TOKEN_LIMIT:,} tokens " + f"(window minus {_DEEP_MAX_OUTPUT_TOKENS:,} output reserve). " + "Reduce codebase size or split review." ), {} if not model: @@ -306,7 +318,7 @@ def run_deep_self_review( model=model, tools=None, reasoning_effort=resolve_effort("deep_self_review"), - max_tokens=100_000, + max_tokens=_DEEP_MAX_OUTPUT_TOKENS, temperature=None, no_proxy=True, ) diff --git a/ouroboros/extension_companion.py b/ouroboros/extension_companion.py index dcb539c0b..efa9eb7a5 100644 --- a/ouroboros/extension_companion.py +++ b/ouroboros/extension_companion.py @@ -131,6 +131,23 @@ class CompanionSupervisor: popen_kwargs.update(subprocess_new_group_kwargs()) popen_kwargs = merge_hidden_kwargs(popen_kwargs) proc = subprocess.Popen(descriptor.command, **popen_kwargs) # noqa: S603 + # Write-through into the custody ledger (daemon scope: companions + # are launcher-managed and must survive server generations; the + # reaper only prunes their dead entries). The launcher-facing + # extension_companions.json contract stays untouched. + try: + from ouroboros.config import DATA_DIR as _data_dir + from ouroboros.process_custody import record_process + + record_process( + pathlib.Path(_data_dir), + pid=proc.pid, + cmd=list(descriptor.command), + purpose=f"companion:{descriptor.skill_name}:{descriptor.name}", + scope="daemon", + ) + except Exception: + log.debug("companion custody record failed", exc_info=True) job_handle = None if IS_WINDOWS and os.environ.get("OUROBOROS_MANAGED_BY_LAUNCHER") != "1": job_handle = create_kill_on_close_job() diff --git a/ouroboros/extension_process_runner.py b/ouroboros/extension_process_runner.py index d4ef209df..b8901d1ce 100644 --- a/ouroboros/extension_process_runner.py +++ b/ouroboros/extension_process_runner.py @@ -594,13 +594,39 @@ async def _call_tool(surface: str, args: Dict[str, Any], drive_root: pathlib.Pat ToolContext(repo_dir=repo_dir, drive_root=drive_root), dict(ctx_payload or {}), ) - try: - result = handler(ctx, **dict(args or {})) - except TypeError: - result = handler(**dict(args or {})) + # Signature-based dispatch: the old try/except TypeError retry re-EXECUTED + # a handler whose body raised TypeError after side effects had happened. + result = ( + handler(ctx, **dict(args or {})) + if _handler_wants_ctx(handler) + else handler(**dict(args or {})) + ) return await _run_maybe_async(result) +def _handler_wants_ctx(handler: Any) -> bool: + """True when the handler's first parameter is a ctx slot (canonical form). + + Extension tool handlers are either ``fn(ctx, **args)`` (canonical) or + ``fn(**args)`` / ``fn(named=...)`` (ctx-less). Dispatch on the first + parameter's name so the args dict can still bind named parameters. + """ + import inspect + + try: + params = list(inspect.signature(handler).parameters.values()) + except (TypeError, ValueError): + return True # builtins/C callables: keep the historical ctx-first call + if not params: + return False + first = params[0] + if first.kind == first.VAR_POSITIONAL: + return True + if first.kind in (first.POSITIONAL_ONLY, first.POSITIONAL_OR_KEYWORD): + return first.name in {"ctx", "context", "_ctx", "tool_context"} + return False + + async def _request_from_payload(payload: Dict[str, Any], drive_root: pathlib.Path, repo_dir: pathlib.Path) -> Request: body = base64.b64decode(str(payload.get("body_b64") or "")) sent = False @@ -775,4 +801,10 @@ def _child_main(input_path: str) -> None: if __name__ == "__main__": if len(sys.argv) != 2: raise SystemExit("usage: python -m ouroboros.extension_process_runner ") + try: + from ouroboros.process_custody import start_parent_lifeline + + start_parent_lifeline(label="extension-runner") + except Exception: + pass _child_main(sys.argv[1]) diff --git a/ouroboros/gateway/contracts.py b/ouroboros/gateway/contracts.py index 63ba0c518..0ddfee7fe 100644 --- a/ouroboros/gateway/contracts.py +++ b/ouroboros/gateway/contracts.py @@ -14,6 +14,17 @@ except ImportError: # pragma: no cover - CI supports Python 3.10. from typing_extensions import Literal, NotRequired, Required, TypedDict # type: ignore[assignment] +class ChatAttachmentInbound(TypedDict, total=False): + """One uploaded chat attachment reference (file already stored by + /api/chat/upload under data/uploads/; ``filename`` is the stored + basename). Image attachments are delivered to vision models as native + image blocks (v6.26.0).""" + + filename: str + display_name: str + mime: str + + class ChatInbound(TypedDict): """Inbound WS chat message. ``type`` and ``content`` are required.""" @@ -22,6 +33,7 @@ class ChatInbound(TypedDict): sender_session_id: NotRequired[str] client_message_id: NotRequired[str] force_plan: NotRequired[bool] + attachments: NotRequired[list] # list[ChatAttachmentInbound] (additive, v6.26.0) class TaskConstraintInbound(TypedDict, total=False): diff --git a/ouroboros/gateway/control.py b/ouroboros/gateway/control.py index 5b8519e36..58feb2e7c 100644 --- a/ouroboros/gateway/control.py +++ b/ouroboros/gateway/control.py @@ -3,7 +3,7 @@ from __future__ import annotations import asyncio -import pathlib +import logging import time from typing import Any, Dict @@ -16,6 +16,8 @@ from ouroboros.gateway.ws import broadcast_ws_sync from ouroboros.outcomes import public_task_result from ouroboros.utils import utc_now_iso +log = logging.getLogger(__name__) + _RECENT_VISIBLE_COMMANDS: Dict[str, float] = {} _VISIBLE_COMMAND_DEDUPE_SEC = 5.0 _evo_cache: Dict[str, Any] = {} @@ -204,6 +206,16 @@ async def api_update_check(_request: Request) -> JSONResponse: return json_exception(exc) +def _respawn_workers_after_failed_update() -> None: + """Revive workers when an update aborts after they were stopped (no restart follows).""" + try: + from supervisor.workers import spawn_workers + + spawn_workers() + except Exception: + log.warning("update_apply: failed to respawn workers after aborted update", exc_info=True) + + async def api_update_apply(request: Request) -> JSONResponse: """Prepare a managed update and restart so safe_restart applies it.""" body = await request_json_or(request, {}, exceptions=(Exception,)) @@ -214,20 +226,36 @@ async def api_update_apply(request: Request) -> JSONResponse: ok, payload = prepare_managed_update(strategy) if not ok: return JSONResponse(payload, status_code=409) + # Stop workers AFTER the update is validated/prepared but BEFORE the + # hard reset of the live repo: a self-modifying task writing between + # the rescue snapshot and the reset would have its edits destroyed + # unrescued. Doing this pre-validation killed all workers on a plain + # 409 (no update available) with no restart to revive them. + try: + from supervisor.workers import kill_workers + + kill_workers( + result_reason="Task interrupted by an owner-requested managed update (restart follows).", + terminal_status="interrupted", + ) + except Exception: + log.warning("update_apply: failed to stop workers before reset", exc_info=True) try: checkout_ok, checkout_msg = checkout_and_reset( BRANCH_DEV, reason="ui_update_apply", - unsynced_policy="ignore", + unsynced_policy="rescue_and_reset", ) except Exception as checkout_exc: _clear_update_intent() + _respawn_workers_after_failed_update() return JSONResponse( {"error": f"Prepared update but checkout failed: {checkout_exc}", **payload}, status_code=409, ) if not checkout_ok: _clear_update_intent() + _respawn_workers_after_failed_update() return JSONResponse( {"error": f"Prepared update but checkout failed: {checkout_msg}", **payload}, status_code=409, diff --git a/ouroboros/gateway/files.py b/ouroboros/gateway/files.py index bf3cc13d4..15c144320 100644 --- a/ouroboros/gateway/files.py +++ b/ouroboros/gateway/files.py @@ -98,6 +98,15 @@ def _resolve_target(request: Request, rel_path: str) -> tuple[pathlib.Path, path except ValueError as exc: raise ValueError("Path escapes file browser root.") from exc resolved = requested.resolve(strict=False) + # Symlink containment: the lexical check above cannot see where a link + # POINTS. Every endpoint operates within the configured root, so a path + # whose resolution leaves the root (e.g. root/link -> /etc) is rejected + # outright — including symlinks themselves (deleting such a link via the + # API is intentionally blocked rather than special-cased). + try: + resolved.relative_to(root_dir) + except ValueError as exc: + raise ValueError("Path escapes file browser root (symlink target outside root).") from exc return root_dir, requested, resolved @@ -348,7 +357,7 @@ async def api_files_read(request: Request) -> JSONResponse: try: if not rel_path: return json_error("Missing path.", status=400) - root_dir, target, _ = _resolve_target(request, rel_path) + root_dir, _requested, target = _resolve_target(request, rel_path) if not target.exists(): return JSONResponse({"error": f"Path not found: {rel_path}"}, status_code=404) if not target.is_file(): @@ -403,7 +412,7 @@ async def api_files_download(request: Request) -> FileResponse | JSONResponse: try: if not rel_path: return json_error("Missing path.", status=400) - _, target, _ = _resolve_target(request, rel_path) + _, _requested, target = _resolve_target(request, rel_path) if not target.exists(): return JSONResponse({"error": f"Path not found: {rel_path}"}, status_code=404) if not target.is_file(): @@ -422,7 +431,7 @@ async def api_files_content(request: Request) -> FileResponse | JSONResponse: try: if not rel_path: return json_error("Missing path.", status=400) - _, target, _ = _resolve_target(request, rel_path) + _, _requested, target = _resolve_target(request, rel_path) if not target.exists(): return JSONResponse({"error": f"Path not found: {rel_path}"}, status_code=404) if not target.is_file(): @@ -762,16 +771,8 @@ def file_browser_routes() -> list[Route]: ] -import mimetypes -import os -import pathlib import uuid -from starlette.datastructures import UploadFile -from starlette.requests import Request -from starlette.responses import JSONResponse - - _CHAT_UPLOAD_MAX_BYTES = 50 * 1024 * 1024 # 50 MB _CHUNK = 64 * 1024 # 64 KB diff --git a/ouroboros/gateway/settings.py b/ouroboros/gateway/settings.py index 0e74285fd..d8c5361b0 100644 --- a/ouroboros/gateway/settings.py +++ b/ouroboros/gateway/settings.py @@ -129,6 +129,20 @@ def _build_network_meta(bind_host: str, bind_port: int) -> dict: } +# Password-class secrets are usually short human-chosen strings: an 8-char +# prefix can BE most of the password. They mask to a constant placeholder; +# long machine-generated API keys keep the recognizable 8-char prefix. +_PASSWORD_CLASS_KEYS = { + "OUROBOROS_NETWORK_PASSWORD", + "GIGACHAT_PASSWORD", + "GIGACHAT_CREDENTIALS", +} + + +def _mask_password_class(value: Any) -> str: + return "***set***" if str(value or "").strip() else "" + + def _mask_secret_value(value: Any) -> str: text = str(value or "") return text[:8] + "..." if len(text) > 8 else "***" @@ -501,7 +515,11 @@ async def api_settings_get(request: Request) -> JSONResponse: safe = {k: v for k, v in settings.items()} for key in _SECRET_SETTING_KEYS: if safe.get(key): - safe[key] = _mask_secret_value(safe[key]) + safe[key] = ( + _mask_password_class(safe[key]) + if key in _PASSWORD_CLASS_KEYS + else _mask_secret_value(safe[key]) + ) safe["MCP_SERVERS"] = _mask_mcp_servers_payload(safe.get("MCP_SERVERS") or []) for key, value in list(safe.items()): if key in _SECRET_SETTING_KEYS or key in _SETTINGS_DEFAULTS: diff --git a/ouroboros/gateway/tasks.py b/ouroboros/gateway/tasks.py index 63e409eb4..a5c56a203 100644 --- a/ouroboros/gateway/tasks.py +++ b/ouroboros/gateway/tasks.py @@ -9,7 +9,7 @@ import subprocess import time import uuid from datetime import datetime, timezone -from typing import Any, Dict, Iterable, List, Optional +from typing import Any, Dict, List, Optional from starlette.requests import Request from starlette.responses import FileResponse, JSONResponse, StreamingResponse @@ -41,7 +41,7 @@ from ouroboros.task_status import ( load_effective_task_result, ) from ouroboros.tool_access import path_is_relative_to, paths_overlap_casefold -from ouroboros.utils import iter_jsonl_objects, utc_now_iso +from ouroboros.utils import iter_jsonl_objects from ouroboros.workspace_preflight import ( collect_workspace_preflight, render_workspace_preflight_summary, diff --git a/ouroboros/gateway/ws.py b/ouroboros/gateway/ws.py index 567d96364..7eb40b4af 100644 --- a/ouroboros/gateway/ws.py +++ b/ouroboros/gateway/ws.py @@ -28,6 +28,48 @@ def set_event_loop(loop: asyncio.AbstractEventLoop | None) -> None: _event_loop = loop +_MAX_NATIVE_IMAGE_BYTES = 8 * 1024 * 1024 + + +def _first_image_attachment(attachments: Any) -> tuple[str, str, str]: + """Resolve the first image upload into (base64, mime, caption). + + Attachments reference files already stored by /api/chat/upload under + data/uploads/ — only a validated basename is accepted (no traversal), so + the WS frame cannot read arbitrary paths. Non-image/missing/oversized + attachments keep the text-label-only behavior. + """ + if not isinstance(attachments, list): + return "", "", "" + import base64 + import os as _os + + uploads_dir = (pathlib.Path(DATA_DIR) / "uploads").resolve(strict=False) + for item in attachments: + if not isinstance(item, dict): + continue + mime = str(item.get("mime") or "").lower() + if not mime.startswith("image/"): + continue + name = _os.path.basename(str(item.get("filename") or "").strip()) + if not name or name in {".", ".."}: + continue + path = (uploads_dir / name).resolve(strict=False) + try: + path.relative_to(uploads_dir) + except ValueError: + continue + try: + if not path.is_file() or path.stat().st_size > _MAX_NATIVE_IMAGE_BYTES: + continue + encoded = base64.b64encode(path.read_bytes()).decode("ascii") + except OSError: + continue + caption = str(item.get("display_name") or name) + return encoded, mime, f"[user attachment: {caption}]" + return "", "", "" + + def has_ws_clients() -> bool: with _ws_lock: return bool(_ws_clients) @@ -219,10 +261,16 @@ async def ws_endpoint(websocket: WebSocket) -> None: bridge = get_bridge() if msg_type == "chat": force_plan = bool(msg.get("force_plan")) + image_b64, image_mime, image_caption = _first_image_attachment( + msg.get("attachments") + ) bridge.ui_send( payload, sender_session_id=str(msg.get("sender_session_id", "") or ""), client_message_id=str(msg.get("client_message_id", "") or ""), + image_base64=image_b64, + image_mime=image_mime, + image_caption=image_caption, task_metadata={ "force_plan": force_plan, "force_plan_source": "consilium" if force_plan else "", diff --git a/ouroboros/gateways/claude_code.py b/ouroboros/gateways/claude_code.py index b8b74dda5..0a921daed 100644 --- a/ouroboros/gateways/claude_code.py +++ b/ouroboros/gateways/claude_code.py @@ -662,6 +662,12 @@ def run_readonly( def _main() -> int: if len(sys.argv) >= 2 and sys.argv[1] == "--readonly-child": + try: + from ouroboros.process_custody import start_parent_lifeline + + start_parent_lifeline(label="claude-readonly-child") + except Exception: + pass try: payload = json.loads(sys.stdin.read() or "{}") except Exception as exc: diff --git a/ouroboros/headless.py b/ouroboros/headless.py index 73ab829d3..2e533a304 100644 --- a/ouroboros/headless.py +++ b/ouroboros/headless.py @@ -14,7 +14,6 @@ import shutil import subprocess import tempfile import threading -import time from datetime import datetime, timezone from hashlib import sha256 from typing import Any, BinaryIO, Dict, Iterable, List, Optional, Sequence, Tuple @@ -45,7 +44,10 @@ ARTIFACT_TERMINAL_STATUSES = { ARTIFACT_STATUS_FAILED, } -_FINAL_STATUSES = {"completed", "failed", "cancelled", "rejected_duplicate"} +# Mirrors task_status.SETTLED_STATUSES; a module-level import would close the +# headless → task_status → outcomes → headless cycle, and the smoke test below +# pins equality so the literal cannot drift from the SSOT. +_FINAL_STATUSES = frozenset({"completed", "failed", "cancelled", "rejected_duplicate"}) _ARTIFACT_LIFECYCLE_FIELDS = { "artifact_status", "artifact_error", diff --git a/ouroboros/launcher_bootstrap.py b/ouroboros/launcher_bootstrap.py index cf85d4617..47ae15ad7 100644 --- a/ouroboros/launcher_bootstrap.py +++ b/ouroboros/launcher_bootstrap.py @@ -150,9 +150,11 @@ def load_repo_manifest(repo_dir: pathlib.Path) -> dict[str, Any]: def _write_repo_manifest(repo_dir: pathlib.Path, manifest: dict[str, Any]) -> None: - meta_path = _managed_meta_path(repo_dir) - meta_path.parent.mkdir(parents=True, exist_ok=True) - meta_path.write_text(json.dumps(manifest, ensure_ascii=True, indent=2) + "\n", encoding="utf-8") + # Atomic: a torn manifest makes the next launch misdetect the installed + # repo generation and re-bootstrap over a healthy checkout. + from ouroboros.utils import atomic_write_json + + atomic_write_json(_managed_meta_path(repo_dir), manifest, trailing_newline=True) def _mark_bootstrap_pin_pending(repo_dir: pathlib.Path) -> None: diff --git a/ouroboros/llm.py b/ouroboros/llm.py index 6da9ca97d..6d3a282f2 100644 --- a/ouroboros/llm.py +++ b/ouroboros/llm.py @@ -11,7 +11,7 @@ import time import copy from typing import Any, Dict, List, Optional, Set, Tuple -from ouroboros.provider_models import normalize_anthropic_model_id, normalize_model_identity +from ouroboros.provider_models import PROVIDER_PREFIXES, normalize_anthropic_model_id, normalize_model_identity from ouroboros.utils import in_worker_process log = logging.getLogger(__name__) @@ -26,11 +26,19 @@ class LocalContextTooLargeError(RuntimeError): def _estimate_message_chars(messages: List[Dict[str, Any]]) -> int: + from ouroboros.context_budget import IMAGE_BLOCK_CHAR_EQUIVALENT + total = 0 for msg in messages: content = msg.get("content") if isinstance(content, list): - total += sum(len(str(block.get("text", ""))) for block in content if isinstance(block, dict)) + for block in content: + if not isinstance(block, dict): + continue + if str(block.get("type") or "") in ("image_url", "image"): + total += IMAGE_BLOCK_CHAR_EQUIVALENT + continue + total += len(str(block.get("text", ""))) else: total += len(str(content or "")) return total @@ -263,11 +271,20 @@ class LLMClient: resp.status_code, ) return + from ouroboros.provider_models import update_vision_overlay + for m in resp.json().get("data", []) or []: mid = m.get("id") or "" sp = m.get("supported_parameters") if mid and isinstance(sp, list) and sp: cls._SUPPORTED_PARAMS_CACHE[mid] = set(sp) + # Vision overlay for supports_vision(): authoritative + # input_modalities from the same /models payload. + arch = m.get("architecture") + if mid and isinstance(arch, dict): + modalities = arch.get("input_modalities") + if isinstance(modalities, list) and modalities: + update_vision_overlay(mid, "image" in modalities) except Exception: log.debug("Failed to fetch OpenRouter model capabilities", exc_info=True) @@ -358,14 +375,7 @@ class LLMClient: @staticmethod def _parse_provider_model(model: str) -> Tuple[str, str]: model_name = str(model or "").strip() - for prefix, provider in ( - ("openai::", "openai"), - ("anthropic::", "anthropic"), - ("cloudru::", "cloudru"), - ("gigachat::", "gigachat"), - ("openai-compatible::", "openai-compatible"), - ("openrouter::", "openrouter"), - ): + for prefix, provider in PROVIDER_PREFIXES: if model_name.startswith(prefix): return provider, model_name[len(prefix):].strip() return "openrouter", model_name @@ -616,6 +626,11 @@ class LLMClient: block["cache_control"] = {"type": "ephemeral"} else: block.pop("cache_control", None) + # Internal metadata (image eviction captions/paths) + # never leaves the process — strict providers 400 on + # unknown content-block fields. + for key in [k for k in block if str(k).startswith("_")]: + block.pop(key, None) return cleaned @staticmethod @@ -835,6 +850,7 @@ class LLMClient: tool_choice, temperature, no_proxy, + timeout, ) if target.get("provider") == "gigachat": # The gigachat library client is synchronous; offload to a thread @@ -881,6 +897,10 @@ class LLMClient: kwargs = self._build_remote_kwargs( target, messages, reasoning_effort, max_tokens, tool_choice, temperature, tools ) + if timeout and timeout > 0: + # Cached clients are built without a timeout; honor the caller's + # per-request timeout instead of silently using the SDK default. + kwargs["timeout"] = float(timeout) prompt_cache_ttl = self._prompt_cache_ttl_from_payload( kwargs.get("messages"), kwargs.get("tools"), @@ -955,6 +975,16 @@ class LLMClient: flatten_tool_content_blocks=True, ) ) + # Local llama.cpp lane has no vision: replace image blocks with an + # explicit placeholder (a raw image_url block would be str()-flattened + # into the prompt as base64 noise). + for msg in clean_messages: + content = msg.get("content") + if not isinstance(content, list): + continue + for idx, block in enumerate(content): + if isinstance(block, dict) and str(block.get("type") or "") in ("image_url", "image"): + content[idx] = {"type": "text", "text": "[image omitted: model has no vision]"} local_max = min(max_tokens, 2048) ctx_len = 0 try: @@ -1503,7 +1533,15 @@ class LLMClient: sent = session.post(url, headers=headers, json=candidate, timeout=request_timeout) else: sent = requests.post(url, headers=headers, json=candidate, timeout=request_timeout) - sent.raise_for_status() + if sent.status_code >= 400: + # Include the Anthropic error body: requests' bare HTTPError text + # ("400 Client Error … for url …") carries no parameter names, so + # the sampling-reject retry matcher could never fire on this path. + body_preview = (sent.text or "")[:2000] + raise requests.HTTPError( + f"{sent.status_code} {sent.reason} for url {sent.url}: {body_preview}", + response=sent, + ) return sent try: @@ -1573,6 +1611,13 @@ class LLMClient: parts: List[str] = [] for block in content: if isinstance(block, dict): + if str(block.get("type") or "") in ("image_url", "image"): + # Explicit placeholder instead of a silent drop: the + # model (and the transcript reader) must know an image + # was present but not deliverable on this lane. + caption = str(block.get("_caption") or "").strip() + parts.append(f"[image omitted: model has no vision{f' — {caption}' if caption else ''}]") + continue parts.append(str(block.get("text", ""))) else: parts.append(str(block)) @@ -1843,9 +1888,13 @@ class LLMClient: ) -> Dict[str, Any]: messages = self._normalize_system_message_placement(messages) resolved_model = str(target.get("resolved_model") or "") - token_limit_key = "max_tokens" - if str(target.get("provider") or "") == "openai" and resolved_model.startswith("gpt-5"): - token_limit_key = "max_completion_tokens" + provider = str(target.get("provider") or "") + # OpenAI reasoning models (gpt-5*, o-series) reject legacy max_tokens + # with a deterministic 400 — they require max_completion_tokens. + openai_reasoning_model = provider == "openai" and resolved_model.startswith( + ("gpt-5", "o1", "o3", "o4") + ) + token_limit_key = "max_completion_tokens" if openai_reasoning_model else "max_tokens" if not target.get("supports_openrouter_extensions"): # Non-OpenRouter providers do not accept cache_control. clean_messages = self._strip_openrouter_roundtrip_metadata( @@ -1860,6 +1909,10 @@ class LLMClient: "messages": clean_messages, token_limit_key: max_tokens, } + if openai_reasoning_model: + # Direct-OpenAI route honors the configured OUROBOROS_EFFORT_* + # lanes instead of silently dropping them (OpenRouter parity). + kwargs["reasoning_effort"] = normalize_reasoning_effort(reasoning_effort) if temperature is not None: kwargs["temperature"] = temperature if tools: @@ -2111,6 +2164,10 @@ class LLMClient: kwargs = self._build_remote_kwargs( target, messages, reasoning_effort, max_tokens, tool_choice, temperature, tools ) + if timeout and timeout > 0: + # Cached clients are built without a timeout; honor the caller's + # per-request timeout instead of silently using the SDK default. + kwargs["timeout"] = float(timeout) prompt_cache_ttl = self._prompt_cache_ttl_from_payload( kwargs.get("messages"), kwargs.get("tools"), diff --git a/ouroboros/local_model.py b/ouroboros/local_model.py index 576f42fff..363bbdb3e 100644 --- a/ouroboros/local_model.py +++ b/ouroboros/local_model.py @@ -4,6 +4,7 @@ from __future__ import annotations import logging import os +import pathlib import subprocess import sys import threading @@ -11,7 +12,7 @@ import time from typing import Any, Callable, Dict, Optional from ouroboros.platform_layer import ( - IS_MACOS, IS_WINDOWS, terminate_process_tree, kill_process_tree, + IS_MACOS, terminate_process_tree, kill_process_tree, subprocess_new_group_kwargs, subprocess_hidden_kwargs, ) @@ -449,6 +450,19 @@ class LocalModelManager: ) _popen_kwargs.update(subprocess_new_group_kwargs()) self._proc = subprocess.Popen(cmd, **_with_hidden_subprocess(_popen_kwargs)) + try: + from ouroboros.config import DATA_DIR + from ouroboros.process_custody import record_process + + record_process( + pathlib.Path(DATA_DIR), + pid=self._proc.pid, + cmd=cmd, + purpose="local_model_server", + scope="session", + ) + except Exception: + log.debug("local model custody record failed", exc_info=True) except FileNotFoundError: self._status = "error" self._error = "Python executable not found. Cannot start local model server." diff --git a/ouroboros/loop.py b/ouroboros/loop.py index f6ce921ed..792badd73 100644 --- a/ouroboros/loop.py +++ b/ouroboros/loop.py @@ -25,9 +25,6 @@ from ouroboros.utils import estimate_tokens from ouroboros.loop_tool_execution import ( StatefulToolExecutor, handle_tool_calls, - _truncate_tool_result, - _TOOL_RESULT_LIMITS, - _DEFAULT_TOOL_RESULT_LIMIT, ) from ouroboros.loop_llm_call import call_llm_with_retry, emit_llm_usage_event, estimate_cost @@ -53,6 +50,8 @@ class _CompactionRoundContext: def _estimate_messages_chars(messages: List[Dict[str, Any]]) -> int: """Estimate mutable transcript size; excludes the static cached system block.""" + from ouroboros.context_budget import IMAGE_BLOCK_CHAR_EQUIVALENT + total = 0 for msg in messages: content = msg.get("content") @@ -61,7 +60,14 @@ def _estimate_messages_chars(messages: List[Dict[str, Any]]) -> int: elif isinstance(content, list): for block in content: if isinstance(block, dict): - # Count whole multipart blocks, including images/cache markers. + if str(block.get("type") or "") in ("image_url", "image"): + # Vision tokens are billed per tile, not per base64 + # char: counting the raw payload made ONE image look + # like ~300K tokens and permanently wedged emergency + # compaction. + total += IMAGE_BLOCK_CHAR_EQUIVALENT + continue + # Count whole multipart blocks, including cache markers. try: import json as _json2 total += len(_json2.dumps(block, ensure_ascii=False)) @@ -210,15 +216,21 @@ def _skill_finalization_message(drive_root: pathlib.Path, llm_trace: Dict[str, A def _force_plan_completed(llm_trace: Dict[str, Any]) -> bool: + """True when a reviewed plan_task completed in this trace. + + Reads the structured ``plan_review_aggregate`` flag captured from the FULL + tool result at execution time (loop_tool_execution); the old substring + check against the 700-char trace preview could never see the aggregate + marker at the end of a long plan output, wedging Consilium tasks in the + force-plan reminder loop. + """ for call in llm_trace.get("tool_calls") or []: if not isinstance(call, dict): continue - result_text = str(call.get("result") or "") if ( str(call.get("tool") or "") == "plan_task" and not bool(call.get("is_error")) - and "## Plan Review Results" in result_text - and "AGGREGATE:" in result_text + and bool(call.get("plan_review_aggregate")) ): return True return False @@ -252,14 +264,16 @@ def _check_budget_limits( task_cost = accumulated_usage.get("cost", 0) if budget_remaining_usd <= 0: - finish_reason = f"🚫 Task rejected. Total budget exhausted. Please increase TOTAL_BUDGET in settings." + finish_reason = "🚫 Task rejected. Total budget exhausted. Please increase TOTAL_BUDGET in settings." accumulated_usage["execution_status"] = "failed" accumulated_usage["reason_code"] = "budget_exhausted" return finish_reason, accumulated_usage, llm_trace budget_pct = task_cost / budget_remaining_usd if budget_remaining_usd > 0 else 1.0 - per_task_limit = float(os.environ.get("OUROBOROS_PER_TASK_COST_USD", "20.0") or 20.0) + from ouroboros.config import SETTINGS_DEFAULTS as _DEFAULTS + _per_task_default = str(_DEFAULTS["OUROBOROS_PER_TASK_COST_USD"]) + per_task_limit = float(os.environ.get("OUROBOROS_PER_TASK_COST_USD", _per_task_default) or _per_task_default) if task_cost >= per_task_limit and round_idx % 10 == 0: _append_or_merge_user_message( messages, @@ -400,8 +414,52 @@ def _append_or_merge_user_message(messages: List[Dict[str, Any]], text: str) -> _append_or_merge_user_content(messages, text) +def _evict_stale_image_blocks(messages: List[Dict[str, Any]], *, incoming: int = 0) -> None: + """Keep only the newest MAX_LIVE_IMAGE_BLOCKS image blocks in the transcript. + + Single counter across ALL image sources (owner uploads, browser + screenshots, transport injections). Evicted blocks become a text + placeholder carrying the caption and the re-view path, so the dialogue + HORIZON is preserved while the heavy payload is dropped (P1: granularity + varies, history does not silently vanish). ``incoming`` reserves room for + blocks about to be appended. + """ + from ouroboros.context_budget import MAX_LIVE_IMAGE_BLOCKS + + image_refs: List[tuple] = [] # (message_idx, block_idx) + for m_idx, msg in enumerate(messages): + content = msg.get("content") + if not isinstance(content, list): + continue + for b_idx, block in enumerate(content): + if isinstance(block, dict) and str(block.get("type") or "") in ("image_url", "image"): + image_refs.append((m_idx, b_idx)) + excess = len(image_refs) + max(0, int(incoming)) - MAX_LIVE_IMAGE_BLOCKS + if excess <= 0: + return + for m_idx, b_idx in image_refs[:excess]: + content = messages[m_idx]["content"] + block = content[b_idx] + caption = str(block.get("_caption") or "").strip() + source_path = str(block.get("_source_path") or "").strip() + placeholder = "[image evicted" + if caption: + placeholder += f": {caption}" + if source_path: + placeholder += f"; re-view: vlm_query file_path={source_path}" + placeholder += "]" + content[b_idx] = {"type": "text", "text": placeholder} + + def _append_or_merge_user_content(messages: List[Dict[str, Any]], content: Any) -> None: """Append user content without flattening multipart blocks.""" + if isinstance(content, list): + incoming_images = sum( + 1 for b in content + if isinstance(b, dict) and str(b.get("type") or "") in ("image_url", "image") + ) + if incoming_images: + _evict_stale_image_blocks(messages, incoming=incoming_images) if messages and messages[-1].get("role") == "user": prior = messages[-1].get("content") if isinstance(content, list): @@ -747,7 +805,7 @@ def _setup_dynamic_tools(tools_registry, tool_schemas, messages): parts = [] if enabled: parts.append( - "✅ Tools are registered in the active v6.17 envelope: " + "✅ Tools are registered in the active capability envelope: " + ", ".join(enabled) ) if not_found: @@ -806,7 +864,7 @@ def _drain_incoming_messages( break if drive_root is not None and task_id: - from ouroboros.owner_inject import drain_owner_messages + from ouroboros.owner_mailbox import drain_owner_messages drive_msgs = drain_owner_messages(drive_root, task_id=task_id, seen_ids=_owner_msg_seen) for dmsg in drive_msgs: _append_or_merge_user_message(messages, _owner_marked_content(dmsg)) @@ -926,11 +984,13 @@ def run_llm_loop( tools._ctx.messages = messages stateful_executor = StatefulToolExecutor() _owner_msg_seen: set = set() + from ouroboros.config import SETTINGS_DEFAULTS as _DEFAULTS + _max_rounds_default = int(_DEFAULTS["OUROBOROS_MAX_ROUNDS"]) try: - MAX_ROUNDS = max(1, int(os.environ.get("OUROBOROS_MAX_ROUNDS", "200"))) + MAX_ROUNDS = max(1, int(os.environ.get("OUROBOROS_MAX_ROUNDS", str(_max_rounds_default)))) except (ValueError, TypeError): - MAX_ROUNDS = 200 - log.warning("Invalid OUROBOROS_MAX_ROUNDS, defaulting to 200") + MAX_ROUNDS = _max_rounds_default + log.warning("Invalid OUROBOROS_MAX_ROUNDS, defaulting to %s", _max_rounds_default) round_idx = 0 try: while True: @@ -1132,7 +1192,7 @@ def run_llm_loop( emit_progress(progress_text.strip()) llm_trace["reasoning_notes"].append(progress_text.strip()) - error_count = handle_tool_calls( + handle_tool_calls( tool_calls, tools, drive_logs, task_id, stateful_executor, messages, llm_trace, emit_progress ) @@ -1173,7 +1233,7 @@ def run_llm_loop( except Exception: log.debug("Failed to stop task services", exc_info=True) try: - from ouroboros.owner_inject import cleanup_task_mailbox + from ouroboros.owner_mailbox import cleanup_task_mailbox cleanup_task_mailbox(drive_root, task_id) except Exception: log.debug("Failed to cleanup task mailbox", exc_info=True) diff --git a/ouroboros/loop_llm_call.py b/ouroboros/loop_llm_call.py index c87deecab..5732baecd 100644 --- a/ouroboros/loop_llm_call.py +++ b/ouroboros/loop_llm_call.py @@ -8,7 +8,6 @@ Extracted from loop.py to keep the main loop orchestrator focused. from __future__ import annotations -import json import pathlib import queue import time @@ -421,7 +420,6 @@ def call_llm_with_retry( (None, 0.0) on failure after max_retries """ msg = None - last_error: Optional[Exception] = None drive_root = pathlib.Path(drive_logs).parent execution_id = str(accumulated_usage.setdefault("execution_id", new_execution_id())) round_id = f"{execution_id}:round:{round_idx}" @@ -649,7 +647,6 @@ def call_llm_with_retry( return msg, cost except Exception as e: - last_error = e if _record_llm_call_error( e, _LlmErrorContext( diff --git a/ouroboros/loop_tool_execution.py b/ouroboros/loop_tool_execution.py index 0278e1962..b2a228557 100644 --- a/ouroboros/loop_tool_execution.py +++ b/ouroboros/loop_tool_execution.py @@ -299,6 +299,11 @@ def _extract_result_metadata(fn_name: str, result: Any, is_error: bool) -> Dict[ # late ARTIFACT_OUTPUTS marker (e.g. a stopped service after a long log tail). if not is_error and "ARTIFACT_OUTPUTS" in text: meta["artifact_registered"] = True + # Same full-result capture for the Consilium force-plan gate: the review + # aggregate marker sits at the END of a long plan_task result, far past the + # 700-char trace preview the gate used to substring-match against. + if fn_name == "plan_task" and not is_error and "## Plan Review Results" in text and "AGGREGATE:" in text: + meta["plan_review_aggregate"] = True exit_match = _EXIT_CODE_RE.search(text) if exit_match: try: diff --git a/ouroboros/marketplace/fetcher.py b/ouroboros/marketplace/fetcher.py index 2966c4bb6..a5b8261de 100644 --- a/ouroboros/marketplace/fetcher.py +++ b/ouroboros/marketplace/fetcher.py @@ -105,6 +105,10 @@ def _validate_member_path(name: str) -> pathlib.PurePosixPath: parts = posix.parts _reject_if(any(part == ".." for part in parts), f"Archive member uses '..' traversal: {name!r}") _reject_if(any(part.startswith("/") for part in parts), f"Archive member has malformed segment: {name!r}") + # Windows drive/ADS escapes: a "C:" segment re-roots pathlib joins on + # Windows and ":" opens NTFS alternate data streams. + _reject_if(any(":" in part or "\\" in part for part in parts), + f"Archive member contains ':' or backslash segment: {name!r}") return posix @@ -210,6 +214,14 @@ def stage( file_count += 1 _reject_if(file_count > _MAX_FILE_COUNT, f"Archive exceeds file count cap {_MAX_FILE_COUNT}") target_path = staging_dir / pathlib.Path(*rel_path.parts) + # Post-join containment: belt-and-braces against any + # platform-specific join surprise the lexical checks missed. + _reject_if( + not target_path.resolve(strict=False).is_relative_to( + staging_dir.resolve(strict=False) + ), + f"Archive member escapes staging dir after join: {rel_path}", + ) target_path.parent.mkdir(parents=True, exist_ok=True) with zf.open(member, "r") as src: # Zip-bomb defense: trust actual cap+1 read, not file_size metadata. diff --git a/ouroboros/marketplace/isolated_deps.py b/ouroboros/marketplace/isolated_deps.py index 26e72d457..274c89722 100644 --- a/ouroboros/marketplace/isolated_deps.py +++ b/ouroboros/marketplace/isolated_deps.py @@ -218,10 +218,15 @@ def read_deps_state( return state fingerprint = read_json_dict(isolated_env_dir(skill_dir) / FINGERPRINT_FILENAME) or {} if str(state.get("status") or "") != "installed": + # The payload-resident fingerprint.json is AGENT-WRITABLE (it lives in + # the skill dir): it may only ever corroborate a durable deps.json + # record, never substitute for one. A skill shipping a forged + # "installed" fingerprint without the runtime-side install record + # stays non-executable. if str(fingerprint.get("status") or "") == "installed": state_hash = str(state.get("specs_hash") or "") fingerprint_hash = str(fingerprint.get("specs_hash") or "") - if not state_hash or state_hash == fingerprint_hash: + if state_hash and state_hash == fingerprint_hash: return fingerprint return state state_hash = str(state.get("specs_hash") or "") diff --git a/ouroboros/marketplace/ouroboroshub.py b/ouroboros/marketplace/ouroboroshub.py index a01dae7ef..a39023686 100644 --- a/ouroboros/marketplace/ouroboroshub.py +++ b/ouroboros/marketplace/ouroboroshub.py @@ -38,10 +38,25 @@ def _raise_if(condition: bool, message: str) -> None: raise OuroborosHubError(message) -_OPENER = urllib.request.build_opener(AllowlistRedirectHandler( - _ALLOWED_HOSTS, - lambda target: urllib.error.URLError(f"OuroborosHub redirect host {target!r} is not allowed"), -)) +# Lazy, proxy-free opener: an import-time build_opener snapshots the process +# proxy environment (and triggers macOS proxy lookup in forked workers); the +# clawhub module's lazy no-proxy pattern is the SSOT behavior to match. +_OPENER: urllib.request.OpenerDirector | None = None + + +def _hub_opener() -> urllib.request.OpenerDirector: + global _OPENER + if _OPENER is None: + _OPENER = urllib.request.build_opener( + urllib.request.ProxyHandler({}), + AllowlistRedirectHandler( + _ALLOWED_HOSTS, + lambda target: urllib.error.URLError( + f"OuroborosHub redirect host {target!r} is not allowed" + ), + ), + ) + return _OPENER @dataclass @@ -87,7 +102,7 @@ def _fetch_bytes(url: str, *, max_bytes: int, timeout_sec: int = 15) -> bytes: _raise_if(parsed.scheme not in {"https", "http"}, f"URL must use https:// (or localhost http): {url}") _raise_if(parsed.scheme == "http" and parsed.hostname not in {"localhost", "127.0.0.1"}, f"URL must use https:// for non-localhost hosts: {url}") _raise_if(parsed.hostname not in _ALLOWED_HOSTS, f"Host {parsed.hostname!r} is not allowed for OuroborosHub") - with _OPENER.open(url, timeout=timeout_sec) as resp: # noqa: S310 - host allowlist above + with _hub_opener().open(url, timeout=timeout_sec) as resp: # noqa: S310 - host allowlist above data = resp.read(max_bytes + 1) _raise_if(len(data) > max_bytes, f"Response exceeded {max_bytes} bytes: {url}") return data diff --git a/ouroboros/mcp_client.py b/ouroboros/mcp_client.py index 61d36b258..25f266f72 100644 --- a/ouroboros/mcp_client.py +++ b/ouroboros/mcp_client.py @@ -307,7 +307,7 @@ def _mask_token(value: str) -> str: def looks_masked_secret(value: Any) -> bool: text = str(value or "").strip() - return text == "***" or text.endswith("...") + return text in ("***", "***set***") or text.endswith("...") def _redact_error_text(text: Any, cfg: Optional[MCPServerConfig] = None) -> str: @@ -483,7 +483,7 @@ def _serialize_content_part(item: Any) -> Dict[str, Any]: def _run_async(coro_factory: Callable[[], Awaitable[Any]], *, join_timeout: Optional[int] = None) -> Any: """Run async work from sync code; the factory avoids reusing closed coroutines.""" try: - loop = asyncio.get_running_loop() + asyncio.get_running_loop() except RuntimeError: return asyncio.run(coro_factory()) diff --git a/ouroboros/memory.py b/ouroboros/memory.py index e32628f6f..b62aa1b2e 100644 --- a/ouroboros/memory.py +++ b/ouroboros/memory.py @@ -4,7 +4,6 @@ import json import logging import os import pathlib -import hashlib from collections import Counter from typing import Any, Dict, List, Optional @@ -46,12 +45,15 @@ class Memory: return default def load_scratchpad_blocks(self) -> List[Dict[str, Any]]: + # Lock the STABLE sidecar (not the data fd): writers atomically replace + # the data file via rename, so an fd-lock on the data inode would + # synchronize against an orphaned inode after a swap. bp = self.scratchpad_blocks_path() if not bp.exists(): return [] fd = None try: - fd = os.open(str(bp), os.O_RDONLY) + fd = os.open(str(bp) + ".lock", os.O_RDONLY | os.O_CREAT, 0o644) _lock_sh(fd) data = bp.read_text(encoding="utf-8") blocks = json.loads(data) if data.strip() else [] @@ -104,18 +106,17 @@ class Memory: if metadata: new_block["metadata"] = dict(metadata) + # Same stable sidecar lock as load/consolidation; the data file itself + # is replaced atomically so a crash mid-append cannot truncate memory. fd = None try: - fd = os.open(str(bp), os.O_RDWR | os.O_CREAT, 0o644) + fd = os.open(str(bp) + ".lock", os.O_RDWR | os.O_CREAT, 0o644) _lock_ex(fd) - raw = b"" - while True: - chunk = os.read(fd, 65536) - if not chunk: - break - raw += chunk - text = raw.decode("utf-8", errors="replace").strip() + try: + text = bp.read_text(encoding="utf-8").strip() if bp.exists() else "" + except OSError: + text = "" blocks = json.loads(text) if text else [] if not isinstance(blocks, list): blocks = [] @@ -133,11 +134,24 @@ class Memory: }) blocks = blocks[-_SCRATCHPAD_MAX_BLOCKS:] - os.lseek(fd, 0, os.SEEK_SET) - os.ftruncate(fd, 0) - os.write(fd, json.dumps(blocks, ensure_ascii=False, indent=2).encode("utf-8")) + from ouroboros.utils import atomic_write_json + + atomic_write_json(bp, blocks) except Exception: + # An honest journal (P1): a failed write must be journaled as a + # failure and surfaced to the caller — the old path logged + # block_appended success for a block that was never persisted. log.error("Failed to append scratchpad block", exc_info=True) + try: + append_jsonl(self.journal_path(), { + "ts": utc_now_iso(), + "type": "block_append_failed", + "source": source, + "block": dict(new_block), + }) + except Exception: + log.debug("Failed to journal block_append_failed", exc_info=True) + raise finally: if fd is not None: try: @@ -166,6 +180,17 @@ class Memory: def regenerate_scratchpad_md(self) -> None: blocks = self.load_scratchpad_blocks() if not blocks: + bp = self.scratchpad_blocks_path() + if bp.exists() and bp.stat().st_size > 2: + # Storage exists but did not parse — rendering the default + # "(empty)" scratchpad would mask memory corruption as amnesia. + write_text( + self.scratchpad_path(), + "# Scratchpad\n\n⚠️ scratchpad_blocks.json exists but could not be " + "parsed — working memory storage is corrupt, NOT empty. " + "Inspect/restore the file before appending new blocks.\n", + ) + return write_text(self.scratchpad_path(), self._default_scratchpad()) return @@ -296,28 +321,25 @@ class Memory: return suffix[-max_entries:] if max_entries < len(suffix) else suffix def jsonl_generation_signature(self, log_name: str) -> Dict[str, Any]: - path = self.logs_path(log_name) - if not path.exists(): - return {} - try: - stat = path.stat() - first = "" - with path.open("r", encoding="utf-8") as handle: - for line in handle: - if line.strip(): - first = line.strip() - break - return { - "first_line_sha256": hashlib.sha256(first.encode("utf-8", errors="replace")).hexdigest(), - "size": int(stat.st_size), - } - except OSError: - return {} + from ouroboros.utils import jsonl_generation_signature - def summarize_chat(self, entries: List[Dict[str, Any]]) -> str: + return jsonl_generation_signature(self.logs_path(log_name)) + + def summarize_chat(self, entries: List[Dict[str, Any]], limit: int = 1000) -> str: + """Render recent chat entries; never hide a horizon cut silently (P1). + + Callers that want the FULL window (e.g. low-context mode passes a huge + tail intent) pass a large ``limit``; when truncation does happen the + output says exactly how many older unconsolidated messages were omitted. + """ if not entries: return "" - return "\n".join(self._format_chat_line(e, compact=True) for e in entries[-1000:]) + limit = max(1, int(limit)) + shown = entries[-limit:] + prefix = "" + if len(entries) > len(shown): + prefix = f"[{len(entries) - len(shown)} older unconsolidated messages omitted]\n" + return prefix + "\n".join(self._format_chat_line(e, compact=True) for e in shown) @staticmethod def _format_chat_line(e: Dict[str, Any], *, compact: bool) -> str: diff --git a/ouroboros/observability.py b/ouroboros/observability.py index 8da03fe82..5886e7d8d 100644 --- a/ouroboros/observability.py +++ b/ouroboros/observability.py @@ -13,7 +13,6 @@ import json import os import pathlib import re -import time import uuid from dataclasses import dataclass, field from typing import Any, Dict, List, Tuple diff --git a/ouroboros/outcomes.py b/ouroboros/outcomes.py index ebfcf427d..737bba069 100644 --- a/ouroboros/outcomes.py +++ b/ouroboros/outcomes.py @@ -435,7 +435,7 @@ def normalize_outcome_axes(result: Dict[str, Any]) -> Dict[str, Any]: def public_task_result(result: Dict[str, Any], *, include_outcome_axes: bool = True) -> Dict[str, Any]: - """Project persisted/effective task results onto the v6.17 public contract.""" + """Project persisted/effective task results onto the public task-result contract.""" if not isinstance(result, dict): return {} diff --git a/ouroboros/owner_inject.py b/ouroboros/owner_mailbox.py similarity index 100% rename from ouroboros/owner_inject.py rename to ouroboros/owner_mailbox.py diff --git a/ouroboros/platform_layer.py b/ouroboros/platform_layer.py index c9b1ff53f..f48673684 100644 --- a/ouroboros/platform_layer.py +++ b/ouroboros/platform_layer.py @@ -552,6 +552,40 @@ def current_process_group_id() -> int: return 0 +def process_start_time(pid: int) -> str: + """Best-effort stable start-time token for (pid, start_time) fingerprints. + + POSIX: ``ps -o lstart=`` (portable across macOS/Linux); Linux fallback + reads /proc//stat field 22 (clock ticks since boot). Windows: + empty string — callers degrade to pid-liveness semantics there. + Returns "" when the pid is gone or the platform offers no stable token. + """ + if pid <= 0: + return "" + if os.name == "nt": + return "" + try: + out = subprocess.run( + ["ps", "-o", "lstart=", "-p", str(pid)], + capture_output=True, text=True, timeout=5, + ) + text = (out.stdout or "").strip() + if out.returncode == 0 and text: + return text + except Exception: + pass + try: + stat_path = pathlib.Path(f"/proc/{pid}/stat") + if stat_path.exists(): + fields = stat_path.read_text(encoding="utf-8", errors="replace").rsplit(")", 1)[-1].split() + # rsplit removed fields 1-2 (pid, comm); starttime is field 22 → index 19 here. + if len(fields) >= 20: + return fields[19] + except Exception: + pass + return "" + + def process_command(pid: int) -> str: """Return a best-effort command line for a Unix process.""" if IS_WINDOWS: diff --git a/ouroboros/post_task_evolution.py b/ouroboros/post_task_evolution.py index b893e710a..b7a957b71 100644 --- a/ouroboros/post_task_evolution.py +++ b/ouroboros/post_task_evolution.py @@ -23,7 +23,6 @@ from __future__ import annotations import json import logging -import os import pathlib from typing import Any, Dict, Optional @@ -79,8 +78,9 @@ def _counter_due(drive_root: pathlib.Path, k: int) -> bool: n = 0 n += 1 try: - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(json.dumps({"n": n}), encoding="utf-8") + from ouroboros.utils import atomic_write_json + + atomic_write_json(path, {"n": n}) except Exception: pass return (n % max(1, k)) == 0 @@ -188,12 +188,11 @@ def _write_request(drive_root: pathlib.Path, decision: Dict[str, Any], task: Dic "origin_task_id": str(task.get("id") or ""), } path = drive_root / _REQUEST_REL - path.parent.mkdir(parents=True, exist_ok=True) # Atomic publish: the supervisor polls every tick, so a partial write must # never be observable (else it could parse-fail and drop the signal). - tmp = path.with_name(f".{path.name}.tmp") - tmp.write_text(json.dumps(req, ensure_ascii=False, indent=2), encoding="utf-8") - os.replace(str(tmp), str(path)) + from ouroboros.utils import atomic_write_json + + atomic_write_json(path, req) def maybe_promote(env: Any, task: Dict[str, Any], reflection_entry: Optional[Dict[str, Any]], @@ -272,7 +271,7 @@ def apply_pending_request(drive_root: Any) -> bool: return False from supervisor.queue import evolution_block_reason, start_evolution_campaign - from supervisor.state import load_state, save_state + from supervisor.state import load_state if evolution_block_reason(): # light runtime mode, etc. _safe_unlink(path) @@ -283,6 +282,11 @@ def apply_pending_request(drive_root: Any) -> bool: # never run. Drop the stale request rather than leaking it. _safe_unlink(path) return False + if bool(st.get("evolution_mode_enabled")): + # A campaign is already enabled (owner-driven or a previous + # promotion). Activating another would hijack its objective and + # clear its failure counter; leave the request for a later tick. + return False # Per-window budget floor (V4 envelope): if configured, do not start a # post-task cycle unless at least that much budget remains. from ouroboros.config import get_post_task_evolution_budget_usd @@ -326,11 +330,14 @@ def apply_pending_request(drive_root: Any) -> bool: _write_evolution_campaign(camp) except Exception: pass - st = load_state() - st["evolution_mode_enabled"] = True - st["evolution_consecutive_failures"] = 0 - st["post_task_autostop"] = True - save_state(st) + from supervisor.state import update_state + + def _activate_one_shot(live: dict) -> None: + live["evolution_mode_enabled"] = True + live["evolution_consecutive_failures"] = 0 + live["post_task_autostop"] = True + + update_state(_activate_one_shot) _safe_unlink(path) log.info("post_task_evolution: promotion applied -> gated evolution campaign activated") return True diff --git a/ouroboros/preflight_runner.py b/ouroboros/preflight_runner.py index c76fc6901..581271b46 100644 --- a/ouroboros/preflight_runner.py +++ b/ouroboros/preflight_runner.py @@ -8,7 +8,7 @@ import shutil import subprocess import sys import tempfile -from typing import List, Optional, Sequence +from typing import Optional, Sequence DEFAULT_PYTEST_ARGS = ["tests/", "-q", "--tb=line", "--no-header"] @@ -60,6 +60,15 @@ def _copy_untracked(repo_dir: pathlib.Path, worktree: pathlib.Path) -> None: def _preflight_env(temp_root: pathlib.Path, repo_worktree: pathlib.Path) -> dict: env = dict(os.environ) env.pop("OUROBOROS_MANAGED_BY_LAUNCHER", None) + # Scrub secret-class variables: the hermetic preflight pytest must not + # inherit live credentials — a (possibly self-written) test could exfiltrate + # or spend with them. OUROBOROS_* wiring stays (suite needs it). + secret_suffixes = ("_API_KEY", "_TOKEN", "_PASSWORD", "_CREDENTIALS", "_SECRET") + for key in list(env): + if key.startswith("OUROBOROS_"): + continue + if key.endswith(secret_suffixes) or key.startswith("GH_"): + env.pop(key, None) temp_root = pathlib.Path(temp_root).resolve(strict=False) repo_worktree = pathlib.Path(repo_worktree).resolve(strict=False) data_dir = (temp_root / "data").resolve(strict=False) diff --git a/ouroboros/pricing.py b/ouroboros/pricing.py index 578f2b14a..d6e1fe0e2 100644 --- a/ouroboros/pricing.py +++ b/ouroboros/pricing.py @@ -14,7 +14,7 @@ from typing import Any, Dict, Optional, Tuple import logging -from ouroboros.provider_models import normalize_model_identity +from ouroboros.provider_models import normalize_model_identity, provider_for_model from ouroboros.utils import utc_now_iso log = logging.getLogger(__name__) @@ -118,7 +118,15 @@ def get_pricing(*, allow_live_fetch: bool = True) -> Dict[str, Tuple[float, ...] _live = fetch_openrouter_pricing() if _live and len(_live) > 5: _cached_pricing.update(_live) - _pricing_fetched = True + _pricing_fetched = True + else: + # fetch_openrouter_pricing swallows network errors and returns + # {} — keep the flag false so the next call retries instead of + # pricing unknown models at $0 for the process lifetime. + import logging as _log + _log.getLogger(__name__).warning( + "OpenRouter pricing fetch returned no data; will retry on next call" + ) except Exception as e: import logging as _log _log.getLogger(__name__).warning("Failed to sync pricing from OpenRouter: %s", e) @@ -178,18 +186,11 @@ def infer_api_key_type(model: str, provider: Optional[str] = None) -> str: if provider_name in {"local", "openrouter", "openai", "anthropic", "openai-compatible", "cloudru", "gigachat"}: return provider_name raw_model = str(model or "").strip() - if raw_model.endswith(" (local)"): - return "local" - if raw_model.startswith("openai::"): - return "openai" - if raw_model.startswith("anthropic::"): - return "anthropic" - if raw_model.startswith("openai-compatible::"): - return "openai-compatible" - if raw_model.startswith("cloudru::"): - return "cloudru" - if raw_model.startswith("gigachat::"): - return "gigachat" + direct_provider = provider_for_model(raw_model) + # ``openrouter::``-prefixed and un-prefixed ids both bill OpenRouter and + # fall through to the normalized-identity heuristics below. + if direct_provider not in ("openrouter",): + return direct_provider normalized = normalize_model_identity(raw_model) if normalized.startswith("openai/"): return "openrouter" @@ -220,20 +221,9 @@ def infer_provider_from_model(model: str) -> str: Used by review-pipeline emitters to ensure /api/cost-breakdown attribution is correct regardless of which provider the model actually routes through. """ - raw = str(model or "").strip() - if raw.endswith(" (local)"): - raw = raw[:-8] - if raw.startswith("anthropic::"): - return "anthropic" - if raw.startswith("openai::"): - return "openai" - if raw.startswith("openai-compatible::"): - return "openai-compatible" - if raw.startswith("cloudru::"): - return "cloudru" - if raw.startswith("gigachat::"): - return "gigachat" - return "openrouter" + provider = provider_for_model(model) + # Historical billing attribution: local-suffixed ids billed as openrouter. + return "openrouter" if provider == "local" else provider def infer_model_category(model: str) -> str: diff --git a/ouroboros/process_custody.py b/ouroboros/process_custody.py new file mode 100644 index 000000000..d9e46fdd0 --- /dev/null +++ b/ouroboros/process_custody.py @@ -0,0 +1,338 @@ +"""Process custody: supervised spawning plus a durable orphan ledger. + +Every long-lived OS process Ouroboros spawns should go through +``spawn_supervised`` so its identity lands in ``data/state/process_ledger.jsonl`` +BEFORE it can be orphaned. The reaper (server startup + periodic tick) then +kills ledger entries whose owning generation is gone, matching processes by a +STRICT (pid, start_time, cmd_sha256) fingerprint — never by command-line +class, so a dev instance can never reap a packaged instance's processes (or +vice versa). + +Scopes: + - ``task``: dies with its owning task; reapable as soon as the task is no + longer running (and always across server generations). + - ``session``: dies with the server generation (session_id mismatch → reap). + - ``daemon``: intentionally outlives generations (e.g. launcher-managed + companions); the reaper only prunes dead entries, never kills. + +This module deliberately lives OUTSIDE platform_layer (primitives-only) and +adds no policy to the panic layers it complements (``_active_subprocesses``, +port sweeps, Windows Job Objects all stay). +""" + +from __future__ import annotations + +import hashlib +import logging +import os +import pathlib +import subprocess +import uuid +from typing import Any, Dict, List, Optional + +from ouroboros.platform_layer import ( + kill_process_group_id, + pid_is_alive, + process_command, + process_group_id, + process_start_time, + subprocess_new_group_kwargs, +) +from ouroboros.utils import append_jsonl, utc_now_iso + +log = logging.getLogger(__name__) + +LEDGER_FILENAME = "process_ledger.jsonl" +# One ledger generation per server process; recorded into every entry so the +# reaper can tell "mine" from "previous generation" without guessing. +_SESSION_ID = uuid.uuid4().hex +_VALID_SCOPES = ("task", "session", "daemon") + + +def current_custody_session_id() -> str: + return _SESSION_ID + + +def ledger_path(drive_root: pathlib.Path) -> pathlib.Path: + return pathlib.Path(drive_root) / "state" / LEDGER_FILENAME + + +def _cmd_sha256(cmd: Any) -> str: + try: + if isinstance(cmd, (list, tuple)): + text = "\0".join(str(part) for part in cmd) + else: + text = str(cmd or "") + return hashlib.sha256(text.encode("utf-8", errors="replace")).hexdigest() + except Exception: + return "" + + +def _live_cmd_sha256(pid: int) -> str: + command = process_command(pid) + if not command: + return "" + return hashlib.sha256(command.encode("utf-8", errors="replace")).hexdigest() + + +def record_process( + drive_root: pathlib.Path, + *, + pid: int, + cmd: Any, + purpose: str, + scope: str, + owner_task_id: str = "", +) -> Dict[str, Any]: + """Append a custody record for an already-spawned process.""" + if scope not in _VALID_SCOPES: + raise ValueError(f"process custody scope must be one of {_VALID_SCOPES}, got {scope!r}") + try: + pgid = process_group_id(pid) + except Exception: + pgid = 0 + entry = { + "ts": utc_now_iso(), + "pid": int(pid), + "pgid": int(pgid or 0), + "fingerprint": { + "start_time": process_start_time(pid), + # The LIVE command line (what the OS reports) is the reap-time + # comparison anchor; the argv we passed may differ cosmetically. + "cmd_sha256": _live_cmd_sha256(pid) or _cmd_sha256(cmd), + }, + "purpose": str(purpose or "")[:200], + "scope": scope, + "owner_task": str(owner_task_id or ""), + "session_id": _SESSION_ID, + } + append_jsonl(ledger_path(drive_root), entry) + return entry + + +def spawn_supervised( + cmd: Any, + *, + drive_root: pathlib.Path, + purpose: str, + scope: str, + owner_task_id: str = "", + new_process_group: bool = True, + **popen_kwargs: Any, +) -> subprocess.Popen: + """Popen + durable custody record (the single supervised chokepoint). + + The record is written immediately after spawn, so even a SIGKILL of the + spawning worker cannot orphan the child invisibly — the reaper finds it + in the ledger on the next generation. + """ + if new_process_group: + merged = dict(subprocess_new_group_kwargs()) + merged.update(popen_kwargs) + popen_kwargs = merged + proc = subprocess.Popen(cmd, **popen_kwargs) # noqa: S603 — callers pass vetted argv lists + try: + record_process( + drive_root, + pid=proc.pid, + cmd=cmd, + purpose=purpose, + scope=scope, + owner_task_id=owner_task_id, + ) + except Exception: + log.warning("process custody record failed for pid %s (%s)", proc.pid, purpose, exc_info=True) + return proc + + +def _fingerprint_matches(entry: Dict[str, Any]) -> bool: + """STRICT identity: the live process must still BE the recorded one. + + pid alive + same start_time (when we have one) + same command hash (when + we have one). A recycled pid fails this and is left alone. We never match + by command-line class. + """ + pid = int(entry.get("pid") or 0) + if pid <= 0 or not pid_is_alive(pid): + return False + fp = entry.get("fingerprint") if isinstance(entry.get("fingerprint"), dict) else {} + recorded_start = str(fp.get("start_time") or "") + if recorded_start: + live_start = process_start_time(pid) + if live_start and live_start != recorded_start: + return False + recorded_cmd = str(fp.get("cmd_sha256") or "") + if recorded_cmd: + live_cmd = _live_cmd_sha256(pid) + if live_cmd and live_cmd != recorded_cmd: + return False + if not live_cmd and not recorded_start: + # Windows: no command line and no start time — liveness is all we + # have (same degradation as workspace_executor records). + return True + return True + + +def _read_ledger(drive_root: pathlib.Path) -> List[Dict[str, Any]]: + path = ledger_path(drive_root) + if not path.exists(): + return [] + entries: List[Dict[str, Any]] = [] + try: + import json + + for line in path.read_text(encoding="utf-8", errors="replace").splitlines(): + line = line.strip() + if not line: + continue + try: + obj = json.loads(line) + except ValueError: + continue + if isinstance(obj, dict) and obj.get("pid"): + entries.append(obj) + except OSError: + return [] + # Last record per pid wins (a pid may be re-registered by a newer spawn). + by_pid: Dict[int, Dict[str, Any]] = {} + for entry in entries: + try: + by_pid[int(entry.get("pid") or 0)] = entry + except (TypeError, ValueError): + continue + by_pid.pop(0, None) + return list(by_pid.values()) + + +def _rewrite_ledger(drive_root: pathlib.Path, entries: List[Dict[str, Any]]) -> None: + import json + + path = ledger_path(drive_root) + try: + from ouroboros.utils import jsonl_append_lock_path + from ouroboros.platform_layer import acquire_exclusive_file_lock, release_exclusive_file_lock + + lock_path = jsonl_append_lock_path(path) + lock_fd = acquire_exclusive_file_lock(lock_path, timeout_sec=2.0, stale_sec=10.0) + try: + tmp = path.with_name(path.name + f".tmp.{os.getpid()}") + tmp.write_text( + "".join(json.dumps(entry, ensure_ascii=False) + "\n" for entry in entries), + encoding="utf-8", + ) + os.replace(tmp, path) + finally: + release_exclusive_file_lock(lock_path, lock_fd) + except Exception: + log.debug("process ledger rewrite failed", exc_info=True) + + +def start_parent_lifeline(*, poll_sec: float = 5.0, label: str = "") -> None: + """Daemon watchdog: group-suicide when the parent process dies (POSIX). + + For OUR python entrypoints only (workers, extension runner, claude child): + when the parent dies, the child is reparented to init (ppid==1) and would + otherwise keep burning CPU/budget invisibly. Arbitrary-argv services and + skills cannot get a watchdog injected — they are covered by the ledger + + reaper instead. + """ + if os.name == "nt": + return # Windows children are covered by Job Objects + + import threading + import time as _time + + def _suicide() -> None: + log.warning("parent process died — lifeline group-suicide (%s)", label or "child") + try: + from ouroboros.platform_layer import current_process_group_id + + pgid = current_process_group_id() + # Kill the whole group ONLY when we lead it (worker/runner + # entrypoints setsid/new-group first). A non-leader sharing the + # spawner's group must not take unrelated siblings down with it. + if pgid == os.getpid(): + kill_process_group_id(pgid) + except Exception: + pass + os._exit(1) + + initial_ppid = os.getppid() + if initial_ppid <= 1: + # The parent died before we even got here (import-delay race after an + # abrupt supervisor kill). These entrypoints are always spawned by a + # live Ouroboros parent, so an orphan at startup is already a leak. + _suicide() + return + + def _watch() -> None: + while True: + _time.sleep(poll_sec) + # Any reparenting means the original parent died (orphans go to + # init/launchd or the nearest subreaper). + if os.getppid() != initial_ppid: + _suicide() + + threading.Thread(target=_watch, daemon=True, name=f"parent-lifeline-{label or 'child'}").start() + + +def reap_orphaned_processes( + drive_root: pathlib.Path, + *, + running_task_ids: Optional[set] = None, +) -> List[int]: + """Kill ledgered processes whose owning generation/task is gone. + + Rules: + - dead pid / fingerprint mismatch → prune the entry, never kill; + - current session's entries → keep (their owners are alive); + EXCEPT task-scoped entries whose owner task is no longer running; + - previous generations: task/session scopes → kill group + reap event; + daemon scope → keep (launcher-managed lifecycles). + """ + drive_root = pathlib.Path(drive_root) + entries = _read_ledger(drive_root) + if not entries: + return [] + reaped: List[int] = [] + survivors: List[Dict[str, Any]] = [] + for entry in entries: + pid = int(entry.get("pid") or 0) + scope = str(entry.get("scope") or "task") + same_session = str(entry.get("session_id") or "") == _SESSION_ID + if not _fingerprint_matches(entry): + continue # dead or recycled pid: prune silently + owner_task = str(entry.get("owner_task") or "") + task_owner_gone = ( + scope == "task" + and running_task_ids is not None + and owner_task + and owner_task not in running_task_ids + ) + if scope == "daemon" or (same_session and not task_owner_gone): + survivors.append(entry) + continue + try: + pgid = int(entry.get("pgid") or 0) + if pgid > 0: + kill_process_group_id(pgid) + else: + from ouroboros.platform_layer import kill_pid_tree + + kill_pid_tree(pid) + reaped.append(pid) + append_jsonl(drive_root / "logs" / "supervisor.jsonl", { + "ts": utc_now_iso(), + "type": "process_reaped", + "pid": pid, + "pgid": int(entry.get("pgid") or 0), + "purpose": entry.get("purpose"), + "scope": scope, + "owner_task": owner_task, + "stale_session": str(entry.get("session_id") or ""), + }) + except Exception: + log.warning("Failed to reap ledgered process %s", pid, exc_info=True) + survivors.append(entry) + _rewrite_ledger(drive_root, survivors) + return reaped diff --git a/ouroboros/provider_models.py b/ouroboros/provider_models.py index e67bca0f2..2da95c775 100644 --- a/ouroboros/provider_models.py +++ b/ouroboros/provider_models.py @@ -1,7 +1,85 @@ -"""Provider-specific model ID helpers and direct-provider defaults.""" +"""Provider-specific model ID helpers, direct-provider defaults, and the +provider registry (SSOT for prefix→provider→credentials knowledge that was +previously duplicated across llm.py, pricing.py, agent_task_pipeline.py and +deep_self_review.py).""" from __future__ import annotations +import os + +# Direct-provider prefix → canonical provider name. Un-prefixed models route +# through OpenRouter. Order matters only for readability; prefixes are disjoint. +PROVIDER_PREFIXES: tuple[tuple[str, str], ...] = ( + ("openai::", "openai"), + ("anthropic::", "anthropic"), + ("cloudru::", "cloudru"), + ("gigachat::", "gigachat"), + ("openai-compatible::", "openai-compatible"), + ("openrouter::", "openrouter"), +) + +# Primary credential env var per provider (single-key providers). +PROVIDER_ENV_KEYS: dict[str, str] = { + "openai": "OPENAI_API_KEY", + "anthropic": "ANTHROPIC_API_KEY", + "cloudru": "CLOUDRU_FOUNDATION_MODELS_API_KEY", + "openrouter": "OPENROUTER_API_KEY", +} + + +def provider_for_model(model: str) -> str: + """Return the execution provider for a model id (``local`` for local lanes).""" + name = str(model or "").strip() + if name.endswith(" (local)"): + return "local" + for prefix, provider in PROVIDER_PREFIXES: + if name.startswith(prefix): + return provider + return "openrouter" + + +def provider_has_credentials(provider: str) -> bool: + """Return True when the environment carries usable credentials for a provider.""" + if provider == "local": + return True + if provider == "openai-compatible": + compat = str(os.environ.get("OPENAI_COMPATIBLE_API_KEY", "") or "").strip() + legacy_key = str(os.environ.get("OPENAI_API_KEY", "") or "").strip() + legacy_base = str(os.environ.get("OPENAI_BASE_URL", "") or "").strip() + return bool(compat or (legacy_key and legacy_base)) + if provider == "gigachat": + creds = str(os.environ.get("GIGACHAT_CREDENTIALS", "") or "").strip() + user = str(os.environ.get("GIGACHAT_USER", "") or "").strip() + password = str(os.environ.get("GIGACHAT_PASSWORD", "") or "").strip() + return bool(creds or (user and password)) + env_key = PROVIDER_ENV_KEYS.get(provider, "OPENROUTER_API_KEY") + return bool(str(os.environ.get(env_key, "") or "").strip()) + + +def model_has_credentials(model: str) -> bool: + """Return True when the model's provider has usable credentials configured.""" + return provider_has_credentials(provider_for_model(model)) + + +def resolve_credentialed_model(default_model: str) -> str: + """Return ``default_model`` if its provider is credentialed, else the first + configured model slot whose provider has credentials (light → fallback → + main → code). Falls back to ``default_model`` when nothing is credentialed + so callers surface the original provider error rather than a silent swap.""" + if model_has_credentials(default_model): + return default_model + for env_name in ( + "OUROBOROS_MODEL_LIGHT", + "OUROBOROS_MODEL_FALLBACK", + "OUROBOROS_MODEL", + "OUROBOROS_MODEL_CODE", + ): + candidate = str(os.environ.get(env_name, "") or "").strip() + if candidate and model_has_credentials(candidate): + return candidate + return default_model + + OPENAI_DIRECT_DEFAULTS = { "main": "openai::gpt-5.5", "code": "openai::gpt-5.5", @@ -97,6 +175,44 @@ def compute_direct_review_models_fallback( return [main] * int(review_runs or 3) +# Conservative static vision map by normalized id/prefix. The OpenRouter +# /models overlay (llm.py) refines this at runtime; static knowledge only +# covers families whose vision support is long-established. +_VISION_MODEL_PREFIXES: tuple[str, ...] = ( + "openai/gpt-5", "openai/gpt-4o", "openai/gpt-4.1", "openai/o3", "openai/o4", + "google/gemini-", "anthropic/claude-", + "x-ai/grok-4", "x-ai/grok-3", + "qwen/qwen-vl", "qwen/qwen2.5-vl", "qwen/qwen3-vl", + "mistralai/pixtral", "meta-llama/llama-4", "meta-llama/llama-3.2-90b-vision", + "openai/gpt-5.5", +) + +# Runtime overlay: model_id → bool, fed from OpenRouter /models +# architecture.input_modalities by llm.py (same lifecycle as its +# supported-parameters cache). +_VISION_OVERLAY: dict = {} + + +def update_vision_overlay(model_id: str, supports: bool) -> None: + normalized = normalize_model_identity(model_id) + if normalized: + _VISION_OVERLAY[normalized] = bool(supports) + + +def supports_vision(model_id: str) -> bool: + """True when the model accepts native image input blocks.""" + # Local lanes have no vision regardless of family name; check the RAW id — + # normalize_model_identity strips the " (local)" suffix. + if str(model_id or "").strip().endswith(" (local)"): + return False + normalized = normalize_model_identity(model_id) + if not normalized: + return False + if normalized in _VISION_OVERLAY: + return _VISION_OVERLAY[normalized] + return normalized.startswith(_VISION_MODEL_PREFIXES) + + def normalize_model_identity(model: str) -> str: text = str(model or "").strip() if text.endswith(" (local)"): diff --git a/ouroboros/reflection.py b/ouroboros/reflection.py index dd86eb2ad..fb6c5fbdf 100644 --- a/ouroboros/reflection.py +++ b/ouroboros/reflection.py @@ -4,7 +4,6 @@ from __future__ import annotations import json import logging -import os import pathlib from typing import Any, Dict, List, Optional @@ -533,14 +532,19 @@ def _update_patterns(drive_root: pathlib.Path, entry: Dict[str, Any]) -> None: else: current = _PATTERNS_HEADER - current_truncated = _truncate_with_notice(current, 3000) + # The register is bounded by the prompt contract (max 20 rows), which fits + # well under this cap — the old 3000-char cut fed the LLM a PARTIAL table + # and the full-replace write then dropped every unseen row (memory loss). + # The cap remains only as a backstop against a pathologically bloated file. + _register_cap = 16_000 + current_truncated = _truncate_with_notice(current, _register_cap) prompt = _PATTERNS_PROMPT.format( current_patterns=( current_truncated + ( "\n\n[IMPORTANT: The current register was compacted for prompt size. " "Preserve existing rows unless you are intentionally merging or updating them.]" - if len(current) > 3000 else "" + if len(current) > _register_cap else "" ) ), goal=_truncate_with_notice(entry.get("goal", "?"), 200), diff --git a/ouroboros/review_evidence.py b/ouroboros/review_evidence.py index ab0920e87..653f9645d 100644 --- a/ouroboros/review_evidence.py +++ b/ouroboros/review_evidence.py @@ -193,6 +193,12 @@ def build_review_projection( effective_status = matching_run.status if matching_run else ("stale" if latest else "none") open_obligations = state.get_open_obligations(repo_key=repo_filter) open_debts = state.get_open_commit_readiness_debts(repo_key=repo_filter) + try: + from ouroboros.utils import read_json_dict + + advisory_overrides = read_json_dict(drive_root_path / "state" / "advisory_overrides.json") or {} + except Exception: + advisory_overrides = {} return { "state": state, "filters": { @@ -223,6 +229,7 @@ def build_review_projection( "open_debts": open_debts, "repo_commit_ready": bool(effective_is_fresh and not open_obligations and not open_debts), "retry_anchor": "commit_readiness_debt" if open_debts else None, + "advisory_overrides": advisory_overrides, } @@ -250,6 +257,12 @@ def build_review_status_payload(projection: Dict[str, Any], *, next_step: str, i "next_step": next_step, } payload["message"] = payload["status_summary"] + # Persistent advisory-enforcement visibility (BIBLE P3 loud-advisory bound): + # how many blocking-grade signals advisory enforcement waved through. + overrides = projection.get("advisory_overrides") + if isinstance(overrides, dict) and overrides.get("count"): + payload["advisory_overrides_count"] = int(overrides.get("count") or 0) + payload["advisory_overrides_recent"] = list(overrides.get("recent") or []) if include_raw and selected_attempt is not None: payload["raw_evidence"] = { "attempt_ts": selected_attempt.ts, diff --git a/ouroboros/review_state.py b/ouroboros/review_state.py index 88eedfc3c..886b2baf2 100644 --- a/ouroboros/review_state.py +++ b/ouroboros/review_state.py @@ -1129,12 +1129,17 @@ def _save_state_unlocked(drive_root: pathlib.Path, state: AdvisoryReviewState) - def save_state(drive_root: pathlib.Path, state: AdvisoryReviewState) -> None: - """Persist review state atomically under a best-effort lock.""" + """Persist review state atomically under the review-state lock. + + Raises ``TimeoutError`` on lock failure (matching ``update_state``): a + silently skipped save left the advisory ledger reporting a stale "fresh" + pre-review, which the commit gate then trusted — an immune-system hole, + not a tolerable degradation. + """ lock_path = drive_root / _LOCK_RELPATH lock_fd = acquire_review_state_lock(drive_root) if lock_fd is None: - log.warning("Failed to acquire review state lock at %s; skipping save", lock_path) - return + raise TimeoutError(f"Could not acquire review state lock for {lock_path}") try: _save_state_unlocked(drive_root, state) finally: diff --git a/ouroboros/review_substrate.py b/ouroboros/review_substrate.py index df4d1b3c1..69973c3bb 100644 --- a/ouroboros/review_substrate.py +++ b/ouroboros/review_substrate.py @@ -21,7 +21,7 @@ from ouroboros.config import get_review_models from ouroboros.llm import LLMClient from ouroboros.observability import new_call_id, persist_call from ouroboros.triad_review import extract_json_array -from ouroboros.utils import sanitize_tool_result_for_log, truncate_review_artifact, utc_now_iso +from ouroboros.utils import sanitize_tool_result_for_log, truncate_review_artifact @dataclass(frozen=True) diff --git a/ouroboros/safety.py b/ouroboros/safety.py index 48546e823..d8bd55b12 100644 --- a/ouroboros/safety.py +++ b/ouroboros/safety.py @@ -143,8 +143,10 @@ TOOL_POLICY: Dict[str, str] = { # run_command safe-subject whitelist. # ``pip`` mutates the Python env and must route through the LLM check. +# ``find`` is NOT safe: -delete / -exec rm make it a mutator, so it routes +# through the LLM safety check like other mutating commands. SAFE_SHELL_COMMANDS = frozenset([ - "ls", "cat", "head", "tail", "grep", "rg", "find", "wc", + "ls", "cat", "head", "tail", "grep", "rg", "wc", "git", "pytest", "pwd", "whoami", "date", "which", "file", "stat", "diff", "tree", "du", "df", diff --git a/ouroboros/server_auth.py b/ouroboros/server_auth.py index 6b38342a1..8581e7ee9 100644 --- a/ouroboros/server_auth.py +++ b/ouroboros/server_auth.py @@ -8,6 +8,9 @@ import hmac import html import ipaddress import os +import pathlib +import secrets +import time from http.cookies import SimpleCookie from urllib.parse import parse_qs @@ -69,8 +72,67 @@ def _headers_map(scope: Scope) -> dict[str, str]: } -def _session_value(password: str) -> str: - return hashlib.sha256(f"ouroboros-auth\0{password}".encode("utf-8")).hexdigest() +# Sessions expire; a leaked cookie is no longer a permanent credential. +_AUTH_SESSION_TTL_SEC = 30 * 24 * 3600 +_auth_secret_cache: bytes | None = None + + +def _auth_secret() -> bytes: + """Server-side random HMAC key, persisted across restarts (0600). + + The old cookie was a bare ``sha256(password)`` — deterministic, never + expiring, and forgeable by anyone who ever saw it. Sessions are now + HMAC-signed with this key, so they cannot be minted without server state + and they die with the key file (one re-login after key loss is accepted). + """ + global _auth_secret_cache + if _auth_secret_cache: + return _auth_secret_cache + try: + from ouroboros import config as _cfg + + path = pathlib.Path(_cfg.DATA_DIR) / "state" / "auth_secret.key" + path.parent.mkdir(parents=True, exist_ok=True) + if path.exists(): + text = path.read_text(encoding="utf-8").strip() + if text: + _auth_secret_cache = bytes.fromhex(text) + return _auth_secret_cache + secret = secrets.token_hex(32) + path.write_text(secret + "\n", encoding="utf-8") + try: + os.chmod(path, 0o600) + except OSError: + pass + _auth_secret_cache = bytes.fromhex(secret) + except Exception: + # No durable key (read-only data dir): fall back to a process-lifetime + # key — sessions survive within the run and re-login after restart. + _auth_secret_cache = secrets.token_bytes(32) + return _auth_secret_cache + + +def _session_value(password: str, issued_at: int | None = None) -> str: + issued = int(issued_at if issued_at is not None else time.time()) + pw_digest = hashlib.sha256(f"ouroboros-auth\0{password}".encode("utf-8")).hexdigest() + signature = hmac.new( + _auth_secret(), f"{issued}\0{pw_digest}".encode("utf-8"), hashlib.sha256 + ).hexdigest() + return f"{issued}.{signature}" + + +def _cookie_session_valid(cookie_val: str, password: str) -> bool: + issued_text, sep, _signature = (cookie_val or "").partition(".") + if not sep: + return False + try: + issued = int(issued_text) + except (TypeError, ValueError): + return False + now = time.time() + if issued > now + 300 or now - issued > _AUTH_SESSION_TTL_SEC: + return False + return hmac.compare_digest(cookie_val, _session_value(password, issued)) def _cookie_value(scope: Scope, cookie_name: str) -> str: @@ -107,7 +169,7 @@ def _candidate_password(scope: Scope) -> str: def _is_authenticated(scope: Scope, password: str) -> bool: cookie_val = _cookie_value(scope, AUTH_COOKIE_NAME) - if cookie_val and hmac.compare_digest(cookie_val, _session_value(password)): + if cookie_val and _cookie_session_valid(cookie_val, password): return True candidate = _candidate_password(scope) return bool(candidate) and hmac.compare_digest(candidate, password) @@ -212,7 +274,16 @@ async def _handle_login(scope: Scope, receive: Receive, send: Send, password: st response = JSONResponse({"ok": True, "next": next_url}, status_code=200) else: response = RedirectResponse(next_url, status_code=303) - response.set_cookie(AUTH_COOKIE_NAME, _session_value(password), httponly=True, samesite="lax") + # Secure only when actually served over TLS: setting it on plain-http LAN + # would make the browser drop the cookie and lock the user in a login loop. + response.set_cookie( + AUTH_COOKIE_NAME, + _session_value(password), + max_age=_AUTH_SESSION_TTL_SEC, + httponly=True, + samesite="lax", + secure=str(scope.get("scheme") or "").lower() in {"https", "wss"}, + ) await response(scope, receive, send) diff --git a/ouroboros/server_control.py b/ouroboros/server_control.py index 12167f1de..083bfd99b 100644 --- a/ouroboros/server_control.py +++ b/ouroboros/server_control.py @@ -119,8 +119,14 @@ def execute_panic_stop( force_kill_pid(child.pid) except (ProcessLookupError, PermissionError): pass - kill_process_on_port(8765) - kill_process_on_port(8766) + # Sweep the actually bound main port (not hardcoded 8765/8766 — a + # custom-port install would panic-kill an unrelated listener). + try: + import server as _server_mod + + kill_process_on_port(_server_mod._actual_bound_port()) + except Exception: + kill_process_on_port(8765) kill_process_on_port(host_service_port()) except Exception: pass diff --git a/ouroboros/skill_lifecycle_queue.py b/ouroboros/skill_lifecycle_queue.py index 24e1346f1..296b773f7 100644 --- a/ouroboros/skill_lifecycle_queue.py +++ b/ouroboros/skill_lifecycle_queue.py @@ -6,6 +6,7 @@ import asyncio import contextlib import logging import os +import time import pathlib import re import threading @@ -136,12 +137,36 @@ def _notify_chat_progress(job: LifecycleJob, phase: str) -> None: return +def _lifecycle_deadline_sec() -> float: + """Upper bound for waiting on the lifecycle lane / running one job. + + A single wedged job used to freeze the whole skill-lifecycle queue forever + (unbounded join + infinite lock-acquire loops); the deadline converts that + silent wedge into a loud, attributable failure. Generous default — skill + reviews legitimately run for many minutes. + """ + raw = os.environ.get("OUROBOROS_SKILL_LIFECYCLE_TIMEOUT_SEC", "") + try: + parsed = float(raw) + if parsed > 0: + return parsed + except (TypeError, ValueError): + pass + return 1800.0 + + @contextlib.asynccontextmanager async def _async_thread_lock(lock: threading.Lock): + deadline = time.monotonic() + _lifecycle_deadline_sec() acquired = False while not acquired: acquired = lock.acquire(blocking=False) if not acquired: + if time.monotonic() >= deadline: + raise TimeoutError( + "Skill lifecycle lane is busy beyond the deadline — a previous " + "job appears wedged. Restarting Ouroboros recovers the lane." + ) await asyncio.sleep(0.01) try: yield @@ -192,6 +217,7 @@ async def async_skill_lifecycle_file_lock(drive_root: pathlib.Path): lock_dir = pathlib.Path(drive_root) / "state" lock_dir.mkdir(parents=True, exist_ok=True) lock_path = lock_dir / "skill_lifecycle.lock" + deadline = time.monotonic() + _lifecycle_deadline_sec() with lock_path.open("a+") as fh: acquired = False while not acquired: @@ -199,6 +225,11 @@ async def async_skill_lifecycle_file_lock(drive_root: pathlib.Path): file_lock_exclusive_nb(fh.fileno()) acquired = True except OSError: + if time.monotonic() >= deadline: + raise TimeoutError( + "skill_lifecycle.lock is held beyond the deadline — a " + "previous job appears wedged. Restarting Ouroboros recovers it." + ) await asyncio.sleep(0.05) try: yield @@ -312,6 +343,22 @@ async def run_lifecycle_job( if not terminal_notified: _notify_chat(job) raise + except BaseException as exc: + # Lane-acquisition failures (e.g. the new _async_thread_lock deadline + # TimeoutError) raise BEFORE the inner finally exists — without this + # branch the job stayed "queued" in _dedupe_jobs forever, blocking + # every future job with the same dedupe key. + if not terminal_notified: + job.status = "failed" + job.error = job.error or (str(exc) or type(exc).__name__) + job.finished_at = job.finished_at or _now_iso() + _release_dedupe(job) + if _active is job: + _active = None + if opts.progress_target is not None: + opts.progress_target.release() + _notify_chat(job) + raise async def run_blocking_preserving_cancellation( @@ -393,7 +440,15 @@ def run_lifecycle_job_blocking( thread = threading.Thread(target=_thread_main, name=f"skill-lifecycle-{kind}", daemon=False) thread.start() - thread.join() + thread.join(timeout=_lifecycle_deadline_sec()) + if thread.is_alive(): + # The job thread is wedged. We cannot kill it safely (it may hold the + # cross-process flock, which the OS releases only on process exit), but + # the caller must not hang forever pretending the job is progressing. + raise TimeoutError( + f"Skill lifecycle job '{kind}:{target}' exceeded its deadline and is " + "wedged; the lifecycle lane stays blocked until restart." + ) if "error" in box: raise box["error"] return box.get("result") diff --git a/ouroboros/skill_loader.py b/ouroboros/skill_loader.py index f5e027a90..42cf7ceef 100644 --- a/ouroboros/skill_loader.py +++ b/ouroboros/skill_loader.py @@ -9,11 +9,8 @@ pending review. Scripts execute via ``skill_exec``; extensions load through from __future__ import annotations import hashlib -import json import logging import pathlib -import re -import tempfile from dataclasses import dataclass, field from typing import Any, Dict, List, Optional diff --git a/ouroboros/skill_review.py b/ouroboros/skill_review.py index 8130ac434..622582a3a 100644 --- a/ouroboros/skill_review.py +++ b/ouroboros/skill_review.py @@ -215,7 +215,6 @@ def _build_skill_file_pack( if len(files) > _MAX_SKILL_FILES: # Never truncate the executable review surface. raise _SkillPackTooLarge(len(files), _MAX_SKILL_FILES) - extras = 0 blocks: List[str] = [] for file_path in files: diff --git a/ouroboros/skill_review_runner.py b/ouroboros/skill_review_runner.py index e4c4b00e9..4d2e04e88 100644 --- a/ouroboros/skill_review_runner.py +++ b/ouroboros/skill_review_runner.py @@ -17,19 +17,14 @@ from ouroboros.skill_lifecycle_queue import ( LifecycleJob, LifecycleJobOptions, run_blocking_preserving_cancellation, - run_lifecycle_job, run_lifecycle_job_blocking, ) from ouroboros.skill_loader import ( SkillPayloadUnreadable, - SkillReviewState, compute_content_hash, find_skill, - requested_core_setting_keys, review_status_allows_execution, save_enabled, - save_review_state, - save_skill_grants, skill_review_gate, skill_state_dir, ) diff --git a/ouroboros/task_results.py b/ouroboros/task_results.py index 7b88cf673..5bf74e8af 100644 --- a/ouroboros/task_results.py +++ b/ouroboros/task_results.py @@ -7,7 +7,7 @@ import pathlib import re from typing import Any, Dict, List, Optional -from ouroboros.utils import atomic_write_json, read_json_dict, utc_now_iso +from ouroboros.utils import atomic_write_json, read_json_dict, update_json_locked, utc_now_iso log = logging.getLogger(__name__) @@ -137,30 +137,47 @@ def write_task_result( status: str, **fields: Any, ) -> Dict[str, Any]: + """Merge-write a task result under a per-file lock. + + Worker processes, the supervisor thread, and gateway handlers all + read-modify-write the same ``task_results/.json``; the lock makes the + monotonic-status guard evaluate the CURRENT on-disk status (closing the + "cancel_requested latch erased by a concurrent completed write" window). + """ path = task_result_path(results_drive_root, task_id) - existing = load_task_result(results_drive_root, task_id) or {} + explicit_ts = str(fields.pop("ts", "") or "") - # Monotonic lifecycle: never let a stale scheduled/running mirror overwrite a - # cancel-intent latch or a terminal outcome. This is the structural guard - # against "ghost" tasks that keep reporting scheduled/running after they were - # cancelled or finished. - if existing and _is_status_regression(existing.get("status"), status): - # Surface the blocked transition: when debugging a "stuck" task this is - # the only signal that a stale/late write was intentionally dropped. - log.debug("Blocked status regression %s -> %s for task %s", - existing.get("status"), status, task_id) - return existing + def _merge(existing: Dict[str, Any]) -> Optional[Dict[str, Any]]: + # Monotonic lifecycle: never let a stale scheduled/running mirror + # overwrite a cancel-intent latch or a terminal outcome. This is the + # structural guard against "ghost" tasks that keep reporting + # scheduled/running after they were cancelled or finished. + if existing and _is_status_regression(existing.get("status"), status): + # Surface the blocked transition: when debugging a "stuck" task this + # is the only signal that a stale/late write was intentionally dropped. + log.debug("Blocked status regression %s -> %s for task %s", + existing.get("status"), status, task_id) + return None + now = utc_now_iso() + return { + **existing, + **fields, + "task_id": task_id, + "status": status, + "ts": explicit_ts or str(existing.get("ts") or now), + "updated_at": now, + } - now = utc_now_iso() - ts = str(fields.pop("ts", "") or existing.get("ts") or now) - payload = { - **existing, - **fields, - "task_id": task_id, - "status": status, - "ts": ts, - "updated_at": now, - } - - atomic_write_json(path, payload) - return payload + try: + return update_json_locked(path, _merge) + except TimeoutError: + # Last-resort visibility: a wedged sibling holding the lock must not + # silently drop a (possibly terminal) result. Log loudly and fall back + # to the previous unlocked merge so the durable record still lands. + log.error("task_results lock timeout for %s; falling back to unlocked merge", task_id) + existing = load_task_result(results_drive_root, task_id) or {} + merged = _merge(dict(existing)) + if merged is None: + return existing + atomic_write_json(path, merged) + return merged diff --git a/ouroboros/task_status.py b/ouroboros/task_status.py index f929d37af..b14a9cede 100644 --- a/ouroboros/task_status.py +++ b/ouroboros/task_status.py @@ -54,6 +54,14 @@ NONTERMINAL_STATUSES: frozenset[str] = frozenset({ STATUS_SCHEDULED, STATUS_RUNNING, }) +# Truly settled outcomes (no cancel-intent latch): callers that must NOT treat +# cancel_requested as terminal (e.g. wait loops surfacing the final record). +SETTLED_STATUSES: frozenset[str] = frozenset({ + STATUS_COMPLETED, + STATUS_FAILED, + STATUS_CANCELLED, + STATUS_REJECTED_DUPLICATE, +}) ARTIFACT_TERMINAL_STATUSES: frozenset[str] = frozenset({ ARTIFACT_STATUS_READY, ARTIFACT_STATUS_FAILED, diff --git a/ouroboros/tools/browser.py b/ouroboros/tools/browser.py index 544408d70..fc7b4503e 100644 --- a/ouroboros/tools/browser.py +++ b/ouroboros/tools/browser.py @@ -22,9 +22,7 @@ except ImportError: _HAS_STEALTH = False from ouroboros.tools.registry import ToolContext, ToolEntry -from ouroboros.contracts.task_constraint import normalize_task_constraint from ouroboros.server_auth import is_loopback_host -from ouroboros.tool_capabilities import LOCAL_READONLY_SUBAGENT_MODE from ouroboros.config import AGENT_SERVER_PORT log = logging.getLogger(__name__) @@ -141,6 +139,54 @@ def _is_blocked_subagent_ip(ip: ipaddress._BaseAddress) -> bool: ) +# AWS IMDSv6 endpoint; the IPv4 metadata services all live in 169.254.0.0/16 +# (link-local), which ``is_link_local`` covers including decimal/hex URL +# spellings once ipaddress normalizes the resolved address. +_METADATA_IPV6_ADDRESSES = frozenset({ipaddress.ip_address("fd00:ec2::254")}) + + +def _is_metadata_ip(ip: ipaddress._BaseAddress) -> bool: + # Unwrap IPv4-mapped IPv6 (http://[::ffff:169.254.169.254]/) so the + # link-local check sees the real IPv4 — mirrors mcp_client's guard. + mapped = getattr(ip, "ipv4_mapped", None) + if mapped is not None: + ip = mapped + return bool(ip.is_link_local) or ip in _METADATA_IPV6_ADDRESSES + + +def _is_metadata_blocked_browser_url(url: str) -> bool: + """Main-agent guard: True only for link-local/cloud-metadata destinations.""" + parsed = urlparse(str(url or "")) + if parsed.scheme not in {"http", "https"}: + return False + host = (parsed.hostname or "").strip().rstrip(".").lower() + if not host: + return False + try: + return _is_metadata_ip(ipaddress.ip_address(host)) + except ValueError: + pass + if _NONSTANDARD_NUMERIC_IPV4_RE.match(host): + # Decimal/hex IPv4 spellings (e.g. http://2852039166/) bypass naive + # string checks; resolve via inet_aton normalization below. + try: + packed = socket.inet_aton(host) + return _is_metadata_ip(ipaddress.ip_address(socket.inet_ntoa(packed))) + except OSError: + return True + try: + infos = socket.getaddrinfo(host, None, type=socket.SOCK_STREAM) + except OSError: + return False # unresolvable hosts fail naturally at fetch time + for info in infos: + try: + if _is_metadata_ip(ipaddress.ip_address(str(info[4][0]))): + return True + except ValueError: + continue + return False + + def _hostname_resolves_to_blocked_ip(host: str) -> bool: try: infos = socket.getaddrinfo(host, None, type=socket.SOCK_STREAM) @@ -505,6 +551,18 @@ def _ensure_browser(ctx: ToolContext, *, engine: str = "chromium", device: str = if _is_subagent_blocked_browser_url(route.request.url, ctx) else route.continue_(), ) + else: + # Main-agent SSRF guard (conservative): block ONLY link-local / + # cloud-metadata endpoints (169.254.0.0/16 incl. decimal/hex spellings, + # fd00:ec2::254). Private/LAN stays reachable — owners legitimately + # browse their own LAN services. Route interception re-validates every + # hop, so redirects cannot smuggle a metadata fetch. + bs_context.route( + "**/*", + lambda route: route.abort() + if _is_metadata_blocked_browser_url(route.request.url) + else route.continue_(), + ) return bs.page @@ -665,6 +723,49 @@ _MARKDOWN_JS = """() => { }""" +def _inject_native_screenshot(ctx: ToolContext, b64: str) -> str: + """Hand a fresh screenshot to a vision-capable active model natively. + + The screenshot is saved to ``data/uploads/screenshots/.png`` (the + re-view path used by eviction placeholders) and injected as a user-role + image block via the existing multipart-preserving merge. The TOOL result + stays a plain string — the tool-message contract is unchanged. Non-vision + models keep the analyze_screenshot/vlm_query flow. + """ + try: + from ouroboros.provider_models import supports_vision + + active_model = str(os.environ.get("OUROBOROS_MODEL", "") or "") + if not supports_vision(active_model): + return "" + messages = getattr(ctx, "messages", None) + if not isinstance(messages, list): + return "" + from ouroboros.utils import utc_now_iso + + ts = utc_now_iso().replace(":", "").replace("-", "")[:15] + shot_dir = pathlib.Path(ctx.drive_root) / "uploads" / "screenshots" + shot_dir.mkdir(parents=True, exist_ok=True) + shot_path = shot_dir / f"{ts}.png" + shot_path.write_bytes(base64.b64decode(b64)) + caption = f"[browser screenshot {ts}]" + from ouroboros.loop import _append_or_merge_user_content + + _append_or_merge_user_content(messages, [ + {"type": "text", "text": caption}, + { + "type": "image_url", + "image_url": {"url": f"data:image/png;base64,{b64}"}, + "_caption": caption, + "_source_path": str(shot_path), + }, + ]) + return "The screenshot is attached to your context natively (vision model). " + except Exception: + log.debug("native screenshot injection failed", exc_info=True) + return "" + + def _extract_page_output(page: Any, output: str, ctx: ToolContext) -> str: """Extract page content in the requested format.""" if output == "screenshot": @@ -676,9 +777,11 @@ def _extract_page_output(page: Any, output: str, ctx: ToolContext) -> str: f"Screenshot captured ({len(b64)} bytes base64). " "Use analyze_screenshot to inspect it." ) + native_note = _inject_native_screenshot(ctx, b64) return ( f"Screenshot captured ({len(b64)} bytes base64). " - f"Call send_photo(image_base64='__last_screenshot__') to deliver it to the user." + + (native_note or "Use analyze_screenshot to inspect it. ") + + "Call send_photo(image_base64='__last_screenshot__') to deliver it to the user." ) elif output == "html": html = page.content() diff --git a/ouroboros/tools/ci.py b/ouroboros/tools/ci.py index 302e04ffb..bde6b65df 100644 --- a/ouroboros/tools/ci.py +++ b/ouroboros/tools/ci.py @@ -10,7 +10,7 @@ import time import urllib.error import urllib.parse import urllib.request -from typing import Any, Dict, List, Optional, Tuple +from typing import List, Optional, Tuple from ouroboros.config import load_settings from ouroboros.tools.registry import ToolContext, ToolEntry diff --git a/ouroboros/tools/claude_advisory_review.py b/ouroboros/tools/claude_advisory_review.py index b1318c831..00201ba7c 100644 --- a/ouroboros/tools/claude_advisory_review.py +++ b/ouroboros/tools/claude_advisory_review.py @@ -21,10 +21,8 @@ from ouroboros.review_state import ( AdvisoryRunRecord, AdvisoryReviewState, compute_snapshot_hash, - format_status_section, load_state, make_repo_key, - save_state, update_state, _utc_now, ) @@ -926,6 +924,19 @@ def _record_bypass(ctx: ToolContext, state: "AdvisoryReviewState", snapshot_hash )) update_state(drive_root, _mutate) + # Persistent visibility (same mechanism as advisory-enforcement overrides): + # review_status surfaces how often the advisory layer was bypassed/absent. + try: + from ouroboros.utils import update_json_locked, utc_now_iso as _now_iso + + def _bump(current: dict) -> dict: + recent = list(current.get("recent") or []) + recent.append({"ts": _now_iso(), "block_reason": f"advisory_bypass: {reason}"[:200], "message_head": str(commit_message or "")[:200]}) + return {"count": int(current.get("count") or 0) + 1, "recent": recent[-10:]} + + update_json_locked(pathlib.Path(drive_root) / "state" / "advisory_overrides.json", _bump) + except Exception: + log.debug("Failed to persist advisory bypass visibility", exc_info=True) if "ANTHROPIC_API_KEY" in reason: msg = ( "⚠️ ANTHROPIC_API_KEY is not set — advisory review skipped automatically. " @@ -1406,11 +1417,20 @@ def _handle_advisory_pre_review( prompt_chars=prompt_chars, model_used=model_used, session_id=advisory_session_id, duration_sec=_advisory_duration, ) - state.add_run(run) + + # Locked read-modify-write against the LIVE ledger: the SDK call above runs + # for minutes, and a state object loaded before it would clobber stale-marks + # and concurrent runs recorded meanwhile (the pre-SDK `state` snapshot is + # only used for gating decisions, never persisted from here on). + def _record_run(live_state: "AdvisoryReviewState") -> None: + live_state.add_run(run) + if run_status != "parse_failure" and items: + _resolve_matching_obligations(live_state, items, snapshot_hash, repo_key=repo_key) + + update_state(drive_root, _record_run) # Surface parse failures explicitly. if run_status == "parse_failure": - save_state(drive_root, state) return _json_response({ "status": "parse_failure", "snapshot_hash": snapshot_hash, @@ -1424,12 +1444,6 @@ def _handle_advisory_pre_review( ), }) - # Resolve only unambiguous PASSed obligations, even if unrelated findings fail. - if items: - _resolve_matching_obligations(state, items, snapshot_hash, repo_key=repo_key) - - save_state(drive_root, state) - # Build human-readable summary. findings_summary: List[str] = [] for item in critical_fails: diff --git a/ouroboros/tools/compact_context.py b/ouroboros/tools/compact_context.py index 89cb998d8..33dcb5525 100644 --- a/ouroboros/tools/compact_context.py +++ b/ouroboros/tools/compact_context.py @@ -3,7 +3,7 @@ from __future__ import annotations import logging -from typing import Any, Dict, List +from typing import List from ouroboros.tools.registry import ToolEntry diff --git a/ouroboros/tools/core.py b/ouroboros/tools/core.py index 025e857b3..897f4756a 100644 --- a/ouroboros/tools/core.py +++ b/ouroboros/tools/core.py @@ -26,7 +26,6 @@ from ouroboros.tool_access import ( resource_root_path, user_files_path_block_reason, ) -from ouroboros.tool_capabilities import LOCAL_READONLY_SUBAGENT_MODE from ouroboros.utils import atomic_write_json, read_text, safe_relpath, utc_now_iso from ouroboros.contracts.task_constraint import normalize_task_constraint, resolve_payload_path from ouroboros.contracts.skill_payload_policy import ( @@ -1428,7 +1427,7 @@ def _codebase_digest(ctx: ToolContext) -> str: def _forward_to_worker(ctx: ToolContext, task_id: str, message: str) -> str: """Forward a message to a running worker task's mailbox.""" - from ouroboros.owner_inject import write_owner_message + from ouroboros.owner_mailbox import write_owner_message from ouroboros.task_results import STATUS_RUNNING, validate_task_id from ouroboros.task_status import FINAL_STATUSES, load_effective_task_result diff --git a/ouroboros/tools/extension_dispatch.py b/ouroboros/tools/extension_dispatch.py index 3d8e92415..2be511b9c 100644 --- a/ouroboros/tools/extension_dispatch.py +++ b/ouroboros/tools/extension_dispatch.py @@ -57,9 +57,14 @@ def dispatch_extension_tool(ctx: Any, name: str, ext_tool: Dict[str, Any], args: handler = ext_tool["handler"] try: - result = handler(ctx, **call_args) - except TypeError: - result = handler(**call_args) + # Signature-based dispatch (single execution): the old TypeError retry + # re-ran handlers whose BODY raised TypeError after side effects. + from ouroboros.extension_process_runner import _handler_wants_ctx + + if _handler_wants_ctx(handler): + result = handler(ctx, **call_args) + else: + result = handler(**call_args) except Exception as exc: return f"⚠️ TOOL_ERROR ({name}): extension tool failed: {type(exc).__name__}: {exc}" diff --git a/ouroboros/tools/git.py b/ouroboros/tools/git.py index 2af1d3e20..87a84edaf 100644 --- a/ouroboros/tools/git.py +++ b/ouroboros/tools/git.py @@ -9,7 +9,6 @@ import os import pathlib import re import subprocess -import sys import time from typing import Any, Dict, List, Optional @@ -1563,7 +1562,7 @@ def get_tools() -> List[ToolEntry]: "name": "vcs_pull_ff", "description": "Fetch from origin and fast-forward merge. Safe: never rewrites history.", "parameters": {"type": "object", "properties": {}, "required": []}, - }, _pull_from_remote, is_code_tool=True), + }, _pull_from_remote, is_code_tool=True, mutates_worktree=True), ToolEntry("vcs_restore", { "name": "vcs_restore", "description": "Discard uncommitted changes, restoring to last committed state (HEAD).", @@ -1571,7 +1570,7 @@ def get_tools() -> List[ToolEntry]: "confirm": {"type": "boolean", "description": "Must be true to execute."}, "paths": {"type": "array", "items": {"type": "string"}, "description": "Specific files to restore"}, }, "required": ["confirm"]}, - }, _restore_to_head, is_code_tool=True), + }, _restore_to_head, is_code_tool=True, mutates_worktree=True), ToolEntry("vcs_revert", { "name": "vcs_revert", "description": "Revert a specific commit by creating a new undo commit. Safe: no history rewrite.", @@ -1579,5 +1578,5 @@ def get_tools() -> List[ToolEntry]: "sha": {"type": "string", "description": "Commit SHA to revert"}, "confirm": {"type": "boolean", "description": "Must be true to execute."}, }, "required": ["sha", "confirm"]}, - }, _revert_commit, is_code_tool=True), + }, _revert_commit, is_code_tool=True, mutates_worktree=True), ] diff --git a/ouroboros/tools/git_pr.py b/ouroboros/tools/git_pr.py index 6f0c849dc..ba0dd9a1b 100644 --- a/ouroboros/tools/git_pr.py +++ b/ouroboros/tools/git_pr.py @@ -10,7 +10,6 @@ import subprocess from typing import List, Optional from ouroboros.tools.registry import ToolContext, ToolEntry -from ouroboros.tools.commit_gate import _invalidate_advisory from ouroboros.tools.git import _acquire_git_lock, _release_git_lock, _sanitize_git_error log = logging.getLogger(__name__) @@ -216,11 +215,6 @@ def _rollback_failed_amend( amend_err = (amend_r.stderr or amend_r.stdout or "").strip() _g(["reset", "--hard", "HEAD~1"], repo_dir) applied.pop() - if applied: - _invalidate_advisory( - ctx, changed_paths=[], mutation_root=repo_dir, - source_tool="cherry_pick_pr_commits", - ) return ( f"⚠️ CHERRY_PICK_ERROR: author amend failed on {sha[:12]} " f"(rolled back to pre-commit state):\n{amend_err[:500]}\n\n" @@ -327,11 +321,6 @@ def _cherry_pick_pr_commits( if override_author is not None: date_r = _g(["log", "-1", "--format=%aI", sha], repo_dir) if date_r.returncode != 0 or not date_r.stdout.strip(): - if applied: - _invalidate_advisory( - ctx, changed_paths=[], mutation_root=repo_dir, - source_tool="cherry_pick_pr_commits", - ) return ( f"⚠️ CHERRY_PICK_ERROR: Cannot read author date for {sha[:12]} " f"(git log returned {date_r.returncode}). Aborting before " @@ -346,11 +335,6 @@ def _cherry_pick_pr_commits( err = (result.stderr or result.stdout or "").strip() _g(["cherry-pick", "--abort"], repo_dir) if stop_on_conflict: - if applied: - _invalidate_advisory( - ctx, changed_paths=[], mutation_root=repo_dir, - source_tool="cherry_pick_pr_commits", - ) return ( f"⚠️ CHERRY_PICK_CONFLICT on {sha[:12]}:\n{err[:500]}\n\n" f"Applied before conflict: {[s[:12] for s in applied] or 'none'}\n" @@ -384,9 +368,6 @@ def _cherry_pick_pr_commits( f"Skipped (conflict): {skipped}" ) - _invalidate_advisory(ctx, changed_paths=[], mutation_root=repo_dir, - source_tool="cherry_pick_pr_commits") - override_note = "" if override_author is not None: override_note = ( @@ -460,9 +441,6 @@ def _stage_adaptations(ctx: ToolContext) -> str: finally: _release_git_lock(lock) - _invalidate_advisory(ctx, changed_paths=[], mutation_root=repo_dir, - source_tool="stage_adaptations") - files = staged.splitlines() return ( f"✅ Staged {len(files)} file(s) on '{current_branch}' (NOT committed):\n" @@ -519,7 +497,6 @@ def _stage_pr_merge( ) adaptation_patch: bytes = b"" - adapt_paths: list = [] staged_before = _g(["diff", "--cached", "--name-only"], repo_dir).stdout.strip() if staged_before: diff_r = subprocess.run( @@ -533,7 +510,7 @@ def _stage_pr_merge( f"Staged changes are preserved. Fix the repo state and retry." ) adaptation_patch = diff_r.stdout - adapt_paths = staged_before.splitlines() + staged_before.splitlines() _g(["reset", "--hard", "HEAD"], repo_dir) def _restore_on_error() -> None: @@ -609,9 +586,6 @@ def _stage_pr_merge( if log_r.returncode == 0: authors = sorted(set(log_r.stdout.strip().splitlines())) - _invalidate_advisory(ctx, changed_paths=[], mutation_root=repo_dir, - source_tool="stage_pr_merge") - co_authored = "\n".join(f"Co-authored-by: {a}" for a in authors) if authors else "" author_hint = ( f"\n\nAttribution — include in commit_reviewed message:\n{co_authored}" @@ -645,7 +619,7 @@ def get_tools() -> List[ToolEntry]: "remote": {"type": "string", "default": "origin", "description": "Git remote name (default: origin)"}, }, "required": ["pr_number"]}, - }, _fetch_pr_ref, is_code_tool=True), + }, _fetch_pr_ref, is_code_tool=True, mutates_worktree=True), ToolEntry("create_integration_branch", { "name": "create_integration_branch", @@ -659,7 +633,7 @@ def get_tools() -> List[ToolEntry]: "base_branch": {"type": "string", "default": "ouroboros", "description": "Branch to create from"}, }, "required": ["pr_number"]}, - }, _create_integration_branch, is_code_tool=True), + }, _create_integration_branch, is_code_tool=True, mutates_worktree=True), ToolEntry("cherry_pick_pr_commits", { "name": "cherry_pick_pr_commits", @@ -703,7 +677,7 @@ def get_tools() -> List[ToolEntry]: "additionalProperties": False, }, }, "required": ["shas"]}, - }, _cherry_pick_pr_commits, is_code_tool=True), + }, _cherry_pick_pr_commits, is_code_tool=True, mutates_worktree=True), ToolEntry("stage_adaptations", { "name": "stage_adaptations", @@ -715,7 +689,7 @@ def get_tools() -> List[ToolEntry]: "Must be on an integrate/pr-N branch." ), "parameters": {"type": "object", "properties": {}}, - }, _stage_adaptations, is_code_tool=True), + }, _stage_adaptations, is_code_tool=True, mutates_worktree=True), ToolEntry("stage_pr_merge", { "name": "stage_pr_merge", @@ -732,5 +706,5 @@ def get_tools() -> List[ToolEntry]: "branch": {"type": "string", "description": "Integration branch to merge (e.g. integrate/pr-17)"}, }, "required": ["branch"]}, - }, _stage_pr_merge, is_code_tool=True), + }, _stage_pr_merge, is_code_tool=True, mutates_worktree=True), ] diff --git a/ouroboros/tools/github.py b/ouroboros/tools/github.py index 5ee032ed6..e14c22d1e 100644 --- a/ouroboros/tools/github.py +++ b/ouroboros/tools/github.py @@ -278,7 +278,7 @@ def _get_pr(ctx: ToolContext, number: int) -> str: diff_raw = _gh_cmd(["pr", "diff", str(number)], ctx, timeout=60) if not diff_raw.startswith("⚠️") and diff_raw.strip(): - lines.append(f"\n**Diff (truncated to 8000 chars):**\n```diff") + lines.append("\n**Diff (truncated to 8000 chars):**\n```diff") lines.append(_truncate_with_notice(diff_raw, 8000)) lines.append("```") diff --git a/ouroboros/tools/health.py b/ouroboros/tools/health.py index f4b2ea2dd..3d25536bc 100644 --- a/ouroboros/tools/health.py +++ b/ouroboros/tools/health.py @@ -3,7 +3,6 @@ import logging import os import pathlib -from typing import Any, Dict from ouroboros.tools.registry import ToolContext, ToolEntry diff --git a/ouroboros/tools/knowledge.py b/ouroboros/tools/knowledge.py index 30692ce33..acd479fc1 100644 --- a/ouroboros/tools/knowledge.py +++ b/ouroboros/tools/knowledge.py @@ -200,7 +200,27 @@ def _knowledge_write(ctx: ToolContext, topic: str, content: str, mode: str = "ov _ensure_dir(ctx) old_content = path.read_text(encoding="utf-8") if path.exists() else "" - if mode == "append": + if sanitized_topic == "improvement-backlog": + # The backlog is concurrently rewritten by its locked helpers (groomer, + # post-task promotion close). A generic unlocked write can interleave + # with a groom and lose items — take the SAME file lock for the write. + from ouroboros.improvement_backlog import _locked_text_file + + # "a+" for both modes: "r+" raises FileNotFoundError on the very first + # overwrite of a fresh drive (the topic file is created on first write). + path.parent.mkdir(parents=True, exist_ok=True) + with _locked_text_file(path, mode="a+") as fh: + if mode == "append": + fh.seek(0) + tail = fh.read() + if tail and not tail.endswith("\n"): + fh.write("\n") + fh.write(content) + else: + fh.seek(0) + fh.truncate(0) + fh.write(content) + elif mode == "append": needs_newline = False if path.exists() and path.stat().st_size > 0: with open(path, "rb") as rf: diff --git a/ouroboros/tools/memory_tools.py b/ouroboros/tools/memory_tools.py index 7b267ec54..911fdfc36 100644 --- a/ouroboros/tools/memory_tools.py +++ b/ouroboros/tools/memory_tools.py @@ -6,7 +6,6 @@ from pathlib import Path from typing import List from ouroboros.tools.registry import ToolEntry, ToolContext -from ouroboros.utils import utc_now_iso log = logging.getLogger(__name__) diff --git a/ouroboros/tools/parallel_review.py b/ouroboros/tools/parallel_review.py index e61deb0da..122431648 100644 --- a/ouroboros/tools/parallel_review.py +++ b/ouroboros/tools/parallel_review.py @@ -4,12 +4,10 @@ from __future__ import annotations import concurrent.futures as _cf import hashlib import logging -from typing import Optional from ouroboros.utils import run_cmd from ouroboros.tools.review_helpers import build_scope_actor_record, format_review_history_entry from ouroboros.tools.scope_review import ( - ScopeReviewResult, run_scope_review, _degraded_scope_requested, _get_scope_model, diff --git a/ouroboros/tools/plan_review.py b/ouroboros/tools/plan_review.py index 691dc3890..bc4e51d72 100644 --- a/ouroboros/tools/plan_review.py +++ b/ouroboros/tools/plan_review.py @@ -63,7 +63,15 @@ def _effective_swarm_max_wait() -> float: from ouroboros.tools.review_helpers import REVIEW_PROMPT_TOKEN_BUDGET as _REVIEW_BUDGET -_PLAN_BUDGET_TOKEN_LIMIT = _REVIEW_BUDGET +# Reserve output headroom inside the reviewer's 1M window (same class of fix as +# scope_review/deep_self_review): SSOT input budget + max output must not +# exceed the window, or atlas-heavy plan packs hit a deterministic provider 400. +_PLAN_MODEL_CONTEXT_WINDOW = 1_000_000 +_PLAN_OUTPUT_MARGIN_TOKENS = 155_000 +_PLAN_BUDGET_TOKEN_LIMIT = min( + _REVIEW_BUDGET, + _PLAN_MODEL_CONTEXT_WINDOW - _PLAN_REVIEW_MAX_TOKENS - _PLAN_OUTPUT_MARGIN_TOKENS, +) def get_tools(): @@ -711,7 +719,7 @@ async def _run_plan_review_async( if atlas.status == "budget_exceeded": estimated = int((atlas.manifest or {}).get("estimated_total_tokens") or 0) return ( - f"⚠️ PLAN_REVIEW_SKIPPED: generated repository atlas exceeded hard budget" + "⚠️ PLAN_REVIEW_SKIPPED: generated repository atlas exceeded hard budget" + (f" ({estimated:,} estimated tokens)" if estimated else "") + ". Split the plan into a smaller scope or choose a smaller context_level." ) diff --git a/ouroboros/tools/query_code.py b/ouroboros/tools/query_code.py index 4ef9b65fb..00f9cd613 100644 --- a/ouroboros/tools/query_code.py +++ b/ouroboros/tools/query_code.py @@ -2,7 +2,6 @@ from __future__ import annotations -import json import pathlib import re from typing import Any, List diff --git a/ouroboros/tools/registry.py b/ouroboros/tools/registry.py index 992c86a82..56307340d 100644 --- a/ouroboros/tools/registry.py +++ b/ouroboros/tools/registry.py @@ -4,7 +4,6 @@ from __future__ import annotations import copy import hashlib -import json import logging import os import pathlib @@ -15,7 +14,6 @@ from typing import Any, Callable, Dict, List, Optional from ouroboros.runtime_mode_policy import ( PROTECTED_RUNTIME_PATHS, - core_patch_notice, mode_allows_protected_write, protected_paths_in, protected_write_block_message, @@ -55,7 +53,7 @@ from ouroboros.protected_artifacts import shell_block_reason as protected_artifa from ouroboros.git_shell_policy import run_shell_git_block_reason, workspace_git_safety_violation from ouroboros.tool_access import light_cognitive_or_root_redirect, normalize_root, resolve_shell_cwd, workspace_mode_block_reason from ouroboros.utils import safe_relpath -from ouroboros.contracts.task_constraint import TaskConstraint, VALID_WRITE_SURFACES, normalize_task_constraint, resolve_payload_path +from ouroboros.contracts.task_constraint import TaskConstraint, VALID_WRITE_SURFACES, normalize_task_constraint from ouroboros.contracts.skill_payload_policy import ( SKILL_OWNER_STATE_FILENAMES, SKILL_OWNER_STATE_STEMS, @@ -599,6 +597,11 @@ class ToolEntry: handler: Callable # fn(ctx: ToolContext, **args) -> str is_code_tool: bool = False timeout_sec: int = 360 + # Capability flag: tool can mutate the live repo worktree. The dispatcher + # snapshots `git status --porcelain` around flagged tools and invalidates + # advisory freshness when the worktree ACTUALLY changed — covering error + # and timeout paths uniformly, and never invalidating for read-only runs. + mutates_worktree: bool = False class ToolRegistry: @@ -1731,12 +1734,23 @@ class ToolRegistry: if name in _PROCESS_COMMAND_TOOLS and workspace_mode else None ) + worktree_before = ( + self._worktree_status_snapshot() if entry.mutates_worktree else None + ) try: - result = entry.handler(self._ctx, **args) - except TypeError as e: - return f"⚠️ TOOL_ARG_ERROR ({name}): {e}" - except Exception as e: - return f"⚠️ TOOL_ERROR ({name}): {e}" + try: + result = entry.handler(self._ctx, **args) + except TypeError as e: + return f"⚠️ TOOL_ARG_ERROR ({name}): {e}" + except Exception as e: + return f"⚠️ TOOL_ERROR ({name}): {e}" + finally: + # Central advisory invalidation by OBSERVED worktree diff: runs on + # success, tool error, and exception paths alike (the per-tool + # manual calls missed early-return/error paths), and skips + # invalidation when a flagged tool ran read-only. + if worktree_before is not None: + self._invalidate_advisory_if_worktree_changed(name, worktree_before) if name in _PROCESS_COMMAND_TOOLS: result = self._run_shell_post_checks( result, @@ -1750,6 +1764,31 @@ class ToolRegistry: return f"{safety_msg}\n\n---\n{result}" return result + def _worktree_status_snapshot(self) -> str: + try: + from ouroboros.utils import run_cmd + + return run_cmd(["git", "status", "--porcelain"], cwd=self._ctx.repo_dir, timeout=20) + except Exception: + return "" + + def _invalidate_advisory_if_worktree_changed(self, tool_name: str, before: str) -> None: + after = self._worktree_status_snapshot() + if after == before: + return + try: + from ouroboros.review_state import invalidate_advisory_after_mutation + + invalidate_advisory_after_mutation( + pathlib.Path(self._ctx.drive_root), + mutation_root=pathlib.Path(self._ctx.repo_dir), + source_tool=tool_name, + ) + except Exception: + logging.getLogger(__name__).debug( + "Central advisory invalidation failed for %s", tool_name, exc_info=True + ) + def override_handler(self, name: str, handler) -> None: """Override the handler for a registered tool (used for closure injection).""" entry = self._entries.get(name) @@ -1758,7 +1797,9 @@ class ToolRegistry: name=entry.name, schema=entry.schema, handler=handler, + is_code_tool=entry.is_code_tool, timeout_sec=entry.timeout_sec, + mutates_worktree=entry.mutates_worktree, ) @property diff --git a/ouroboros/tools/review.py b/ouroboros/tools/review.py index 0e13f8892..5fd55e7d0 100644 --- a/ouroboros/tools/review.py +++ b/ouroboros/tools/review.py @@ -12,6 +12,7 @@ from ouroboros.utils import ( run_cmd, append_jsonl, truncate_review_artifact, + utc_now_iso, ) from ouroboros import config as _cfg from ouroboros.tools.registry import ToolEntry, ToolContext @@ -90,7 +91,6 @@ from ouroboros.tools.review_helpers import ( REPO_ANTI_PATTERN_LOCK_GUARD, REVIEW_JSON_ARRAY_CONTRACT, REVIEW_PREAMBLE, - normalize_reviewer_items, build_self_verification_template, build_review_history_section as _build_review_history_section, emit_review_usage, @@ -758,12 +758,54 @@ def _handle_review_block_or_warning( """Either block immediately or downgrade to advisory warning.""" if blocking_review: return blocked_msg + _record_advisory_override(ctx, blocked_msg) _append_review_warning(ctx, advisory_prefix + blocked_msg) ctx._review_iteration_count = 0 ctx._review_history = [] return None +def _record_advisory_override(ctx: ToolContext, blocked_msg: str) -> None: + """Durable trace of a blocking signal waved through by advisory enforcement. + + Constitutional requirement (BIBLE P3 "Owner-chosen enforcement, loud + advisory"): every decision blocking enforcement would have stopped must + leave a durable, owner-visible trace. Persisted to events.jsonl AND to a + persistent counter file surfaced by the review_status tool. + """ + reason = str(getattr(ctx, "_last_review_block_reason", "") or "unknown") + try: + append_jsonl(ctx.drive_logs() / "events.jsonl", { + "ts": utc_now_iso(), + "type": "review_advisory_override", + "block_reason": reason, + "message_head": str(blocked_msg or "")[:600], + "task_id": str(getattr(ctx, "task_id", "") or ""), + }) + except Exception: + log.debug("Failed to emit review_advisory_override event", exc_info=True) + try: + from ouroboros.utils import update_json_locked + + path = ctx.drive_root / "state" / "advisory_overrides.json" + + def _bump(current: dict) -> dict: + recent = list(current.get("recent") or []) + recent.append({ + "ts": utc_now_iso(), + "block_reason": reason, + "message_head": str(blocked_msg or "")[:300], + }) + return { + "count": int(current.get("count") or 0) + 1, + "recent": recent[-10:], + } + + update_json_locked(path, _bump) + except Exception: + log.warning("Failed to persist advisory override visibility", exc_info=True) + + def _collect_review_findings(ctx: ToolContext, model_results: list) -> tuple[list[str], list[str], list[str], list[dict]]: parsed = parse_model_review_results({"results": model_results}) critical_fails: List[str] = [] @@ -1094,6 +1136,7 @@ def _run_unified_review(ctx: ToolContext, commit_message: str, ctx, commit_message, critical_fails, advisory_warns, errored_note, ) + _record_advisory_override(ctx, "; ".join(critical_fails[:5])) _append_review_warning( ctx, "Review enforcement=Advisory: critical review findings did not block commit.", @@ -1105,9 +1148,12 @@ def _run_unified_review(ctx: ToolContext, commit_message: str, if errored_note: _append_review_warning(ctx, errored_note) - # All clear: reset iteration state. - ctx._review_iteration_count = 0 - ctx._review_history = [] + if not critical_fails: + # All clear: reset iteration state. With critical findings present + # (advisory enforcement), the anti-thrashing history must SURVIVE so + # repeat findings on the next attempt are still recognized as repeats. + ctx._review_iteration_count = 0 + ctx._review_history = [] if errored_note: advisory_warns.append(errored_note.strip()) diff --git a/ouroboros/tools/review_helpers.py b/ouroboros/tools/review_helpers.py index a87cdb1af..4338b966e 100644 --- a/ouroboros/tools/review_helpers.py +++ b/ouroboros/tools/review_helpers.py @@ -282,6 +282,9 @@ Return ONLY a JSON array. Each element: "severity": "critical" | "advisory", "reason": "" } +If you reviewed everything and found NOTHING to report, return the empty array +followed by the sentinel word NO_FINDINGS on its own line. An empty array +without NO_FINDINGS is treated as a non-response and excluded from quorum. """ REVIEW_SEVERITY_THRESHOLDS = """\ @@ -1231,7 +1234,6 @@ def build_scope_section(scope: str = "") -> str: def get_advisory_runtime_diagnostics(model: str, prompt_chars: int, touched_paths: list) -> dict: """Collect best-effort advisory SDK diagnostics; never raises.""" - import sys diag: dict = { "model": model, diff --git a/ouroboros/tools/review_synthesis.py b/ouroboros/tools/review_synthesis.py index 17365adb6..ac57957df 100644 --- a/ouroboros/tools/review_synthesis.py +++ b/ouroboros/tools/review_synthesis.py @@ -131,7 +131,10 @@ def _parse_synthesis_output(raw: str) -> Optional[List[Dict[str, Any]]]: continue canonical = { "item": str(entry.get("item", "") or ""), - "severity": str(entry.get("severity", "advisory") or "advisory"), + # The synthesizer's INPUT is exclusively critical findings, so a + # missing severity must stay critical — an "advisory" default + # silently downgraded blocking findings out of the gate. + "severity": str(entry.get("severity", "critical") or "critical"), "reason": str(entry.get("reason", "") or ""), "obligation_id": str(entry.get("obligation_id", "") or ""), "evidence_from_reviewers": _normalize_evidence(entry.get("evidence_from_reviewers")), @@ -183,16 +186,16 @@ def synthesize_to_canonical_issues( try: raw_response = _call_synthesis_llm(prompt, ctx=ctx) except Exception as exc: - log.debug("review_synthesis: LLM call raised exception: %s — using original findings", exc) + log.warning("review_synthesis: LLM call raised exception: %s — using original findings", exc) return critical_findings if raw_response is None: - log.debug("review_synthesis: LLM call returned None — using original findings") + log.warning("review_synthesis: LLM call returned None — using original findings") return critical_findings canonical = _parse_synthesis_output(raw_response) if canonical is None: - log.debug("review_synthesis: failed to parse LLM output — using original findings") + log.warning("review_synthesis: failed to parse LLM output — using original findings") return critical_findings log.debug( @@ -249,7 +252,7 @@ def _call_synthesis_llm(prompt: str, *, ctx: Any = None) -> Optional[str]: return str(content) if content else None except Exception as exc: - log.debug("review_synthesis: LLM call failed: %s", exc) + log.warning("review_synthesis: LLM call failed: %s", exc) return None diff --git a/ouroboros/tools/scope_review.py b/ouroboros/tools/scope_review.py index 6127e3c33..b0329eb0f 100644 --- a/ouroboros/tools/scope_review.py +++ b/ouroboros/tools/scope_review.py @@ -9,7 +9,6 @@ fail closed; oversized prompts are the explicit non-blocking skip path. from __future__ import annotations -import asyncio import contextvars import inspect import logging @@ -119,7 +118,7 @@ class _ScopeAtlasBudgetExceeded(RuntimeError): self.manifest = dict(manifest or {}) token_count = int(self.manifest.get("estimated_total_tokens") or 0) super().__init__( - f"Generated Scope Atlas exceeded hard budget" + "Generated Scope Atlas exceeded hard budget" + (f" (~{token_count:,} estimated tokens)" if token_count else "") ) @@ -410,10 +409,15 @@ def _build_scope_prompt( ) -> tuple: """Build the scope prompt or a touched-context/budget status sentinel.""" _SCOPE_CONTEXT_MANIFEST.set({}) - try: - scope_checklist = load_checklist_section("Intent / Scope Review Checklist") - except Exception: - scope_checklist = "(Intent / Scope Review Checklist not found in docs/CHECKLISTS.md)" + # Fail-closed (immune-system parity with the triad): a scope review running + # WITHOUT its checklist silently reviews against nothing. Raising turns it + # into an explicit SCOPE_REVIEW_BLOCKED error instead of a placeholder pass. + scope_checklist = load_checklist_section("Intent / Scope Review Checklist") + if not str(scope_checklist or "").strip(): + raise RuntimeError( + "Intent / Scope Review Checklist could not be loaded from docs/CHECKLISTS.md — " + "scope review cannot run without its checklist (fail-closed)." + ) goal_section = build_goal_section(goal, scope, commit_message) scope_section = build_scope_section(scope) diff --git a/ouroboros/tools/services.py b/ouroboros/tools/services.py index 1fed01027..a5f9cb808 100644 --- a/ouroboros/tools/services.py +++ b/ouroboros/tools/services.py @@ -19,7 +19,6 @@ from ouroboros.platform_layer import ( kill_process_group_id, kill_process_tree, process_group_id, - subprocess_new_group_kwargs, ) from ouroboros.tools.registry import ToolContext, ToolEntry from ouroboros.tool_access import resolve_shell_cwd @@ -378,14 +377,22 @@ def _start_service( log_fh = log_path.open("ab") try: bootstrap_process_path() - proc = subprocess.Popen( + # Supervised spawn: the durable custody record means a SIGKILLed worker + # can no longer orphan this service invisibly — the reaper finds it in + # the ledger on the next server generation. + from ouroboros.process_custody import spawn_supervised + + proc = spawn_supervised( [str(part) for part in cmd], + drive_root=pathlib.Path(ctx.drive_root), + purpose=f"service:{service_name}", + scope="task", + owner_task_id=task_id, cwd=str(workdir), stdout=log_fh, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL, env=_service_env(), - **subprocess_new_group_kwargs(), ) pgid = process_group_id(proc.pid) log_fh.close() @@ -721,7 +728,7 @@ def get_tools() -> List[ToolEntry]: "readiness": {"type": "object", "default": {}, "description": "Optional {log_contains|stdout_contains, timeout_sec} readiness probe."}, "outputs": {"type": "array", "items": {"type": "string"}, "default": [], "description": "Files generated by the service to copy into the task artifact store when the service stops."}, }, "required": ["cmd"]}, - }, _start_service, is_code_tool=True, timeout_sec=30), + }, _start_service, is_code_tool=True, timeout_sec=30, mutates_worktree=True), ToolEntry("service_status", { "name": "service_status", "description": "Return pid/state/readiness/uptime for a task-scoped service.", @@ -743,5 +750,5 @@ def get_tools() -> List[ToolEntry]: "parameters": {"type": "object", "properties": { "name": {"type": "string", "default": "service"}, }, "required": []}, - }, _stop_service, is_code_tool=True, timeout_sec=30), + }, _stop_service, is_code_tool=True, timeout_sec=30, mutates_worktree=True), ] diff --git a/ouroboros/tools/shell.py b/ouroboros/tools/shell.py index 42897cae6..89a70b3d3 100644 --- a/ouroboros/tools/shell.py +++ b/ouroboros/tools/shell.py @@ -22,7 +22,7 @@ from subprocess import Popen, CompletedProcess from typing import Any, Dict, List from ouroboros.artifacts import artifact_store_path_block_reason, copy_directory_to_task_artifacts, copy_file_to_task_artifacts -from ouroboros.platform_layer import IS_WINDOWS, bootstrap_process_path, kill_process_tree, subprocess_new_group_kwargs +from ouroboros.platform_layer import bootstrap_process_path, kill_process_tree, subprocess_new_group_kwargs from ouroboros.config import get_runtime_mode, load_settings from ouroboros.runtime_mode_policy import ( core_patch_notice, @@ -973,6 +973,9 @@ def _run_shell(ctx: ToolContext, cmd, cwd: str = "", outputs: List[str] | None = return autocorrect_note + f"⚠️ SHELL_EXIT_ERROR: command exited with {_describe_returncode(res.returncode, cwd=work_dir)}.\n\n{_format_process_output(res.stdout or '', res.stderr or '')}{executor_note}" after_changed = _status_snapshot(repo_root) if after_changed != before_changed: + # Kept (nonstandard case): repo_root here is the RESOLVED cwd root, + # which may be a workspace/skill repo the central live-repo + # dispatcher check does not watch; this call passes precise paths. _invalidate_advisory( ctx, changed_paths=after_changed or before_changed, @@ -1008,7 +1011,7 @@ def _run_shell(ctx: ToolContext, cmd, cwd: str = "", outputs: List[str] | None = if artifact_failed: return ( autocorrect_note - + f"⚠️ ARTIFACT_OUTPUT_ERROR: command succeeded but declared output registration failed. " + + "⚠️ ARTIFACT_OUTPUT_ERROR: command succeeded but declared output registration failed. " + f"{_describe_returncode(0, cwd=work_dir)}\n" + f"{_format_process_output(res.stdout or '', res.stderr or '')}" + artifact_note @@ -1344,6 +1347,9 @@ def _claude_code_edit(ctx: ToolContext, prompt: str, cwd: str = "", budget: floa after_changed = _status_snapshot(target_repo_root) if repo_mode and after_changed != before_changed: + # Kept (nonstandard case): target_repo_root may be a skill + # payload/workspace repo outside the central live-repo check, + # and result.changed_files gives precise invalidation paths. _invalidate_advisory( ctx, changed_paths=result.changed_files or after_changed or before_changed, @@ -1534,7 +1540,7 @@ def get_tools() -> List[ToolEntry]: "description": "Generated file paths to copy/register into the task artifact store after success.", }, }, "required": ["cmd"]}, - }, _run_shell, is_code_tool=True, timeout_sec=_RUN_SHELL_DEFAULT_TIMEOUT_SEC), + }, _run_shell, is_code_tool=True, timeout_sec=_RUN_SHELL_DEFAULT_TIMEOUT_SEC, mutates_worktree=True), ToolEntry("claude_code_edit", { "name": "claude_code_edit", "description": ( @@ -1568,7 +1574,7 @@ def get_tools() -> List[ToolEntry]: "description": "Generated file paths to copy/register into the task artifact store after a successful edit.", }, }, "required": ["prompt"]}, - }, _claude_code_edit, is_code_tool=True, timeout_sec=1200), + }, _claude_code_edit, is_code_tool=True, timeout_sec=1200, mutates_worktree=True), ToolEntry("run_script", { "name": "run_script", "description": ( @@ -1588,5 +1594,5 @@ def get_tools() -> List[ToolEntry]: "description": "Generated file paths to copy/register into the task artifact store after success.", }, }, "required": ["script"]}, - }, _run_script, is_code_tool=True, timeout_sec=_RUN_SHELL_DEFAULT_TIMEOUT_SEC), + }, _run_script, is_code_tool=True, timeout_sec=_RUN_SHELL_DEFAULT_TIMEOUT_SEC, mutates_worktree=True), ] diff --git a/ouroboros/tools/skill_exec.py b/ouroboros/tools/skill_exec.py index 61e5f80fe..6cca5f5ef 100644 --- a/ouroboros/tools/skill_exec.py +++ b/ouroboros/tools/skill_exec.py @@ -679,6 +679,26 @@ def _handle_skill_exec( except Exception: log.debug("Could not augment skill env with isolated dependencies", exc_info=True) + # TOCTOU narrowing: re-hash the payload immediately before spawn. The + # gate-time hash above ran before grants/env/deps resolution — a write + # landing in that window would execute unreviewed code under a PASS verdict. + try: + spawn_hash = compute_content_hash( + loaded.skill_dir, + manifest_entry=loaded.manifest.entry, + manifest_scripts=loaded.manifest.scripts, + ) + except SkillPayloadUnreadable as exc: + return ( + f"⚠️ SKILL_EXEC_ERROR: skill {skill_name!r} payload became unreadable " + f"right before execution ({exc})." + ) + if spawn_hash != current_hash: + return ( + f"⚠️ SKILL_EXEC_BLOCKED: skill {skill_name!r} payload changed between " + "the review-freshness check and execution. Re-run skill_review." + ) + try: returncode, stdout_bytes, stderr_bytes, overflowed = _run_skill_subprocess( cmd, diff --git a/ouroboros/tools/tool_discovery.py b/ouroboros/tools/tool_discovery.py index aa782392e..9c15cf0f9 100644 --- a/ouroboros/tools/tool_discovery.py +++ b/ouroboros/tools/tool_discovery.py @@ -1,13 +1,13 @@ """Tool discovery meta-tools. -v6.17 starts tasks with the full selected capability envelope. These handlers +Tasks start with the full selected capability envelope. These handlers remain as a compatibility/discovery surface: they can confirm whether a named tool is registered, but they no longer grant delayed core capabilities. """ from __future__ import annotations import logging -from typing import Any, Dict, List, Optional, TYPE_CHECKING +from typing import List, Optional, TYPE_CHECKING from ouroboros.tools.registry import ToolContext, ToolEntry from ouroboros.tool_policy import list_non_core_tools as _policy_list_non_core @@ -91,7 +91,7 @@ def get_tools() -> List[ToolEntry]: "name": "list_available_tools", "description": ( "List tools omitted from the active envelope, if any. In the normal " - "v6.17 envelope this usually reports that all tools are already active." + "full-envelope model this usually reports that all tools are already active." ), "parameters": { "type": "object", @@ -106,7 +106,7 @@ def get_tools() -> List[ToolEntry]: schema={ "name": "enable_tools", "description": ( - "Compatibility check for named tools (comma-separated). v6.17 starts " + "Compatibility check for named tools (comma-separated). Tasks start " "with the selected envelope active, so this confirms registration instead " "of granting delayed core tools." ), diff --git a/ouroboros/tools/vision.py b/ouroboros/tools/vision.py index 4e297b397..18e420bdb 100644 --- a/ouroboros/tools/vision.py +++ b/ouroboros/tools/vision.py @@ -3,6 +3,7 @@ from __future__ import annotations import logging +import pathlib import os from typing import Any, Dict, List, Tuple @@ -189,8 +190,8 @@ def _vlm_query(ctx: ToolContext, prompt: str, image_url: str = "", image_base64: mime = _detect_image_mime_for_vlm(raw) if not mime: return ( - f"⚠️ File does not appear to be a supported image (PNG/JPEG/GIF/WEBP). " - f"Only image files may be sent to the VLM via file_path." + "⚠️ File does not appear to be a supported image (PNG/JPEG/GIF/WEBP). " + "Only image files may be sent to the VLM via file_path." ) images.append(_image_payload_from_bytes(raw, mime)) elif image_url: diff --git a/ouroboros/triad_review.py b/ouroboros/triad_review.py index 907b493d6..f517416ec 100644 --- a/ouroboros/triad_review.py +++ b/ouroboros/triad_review.py @@ -165,6 +165,28 @@ def _normalize_items(items: List[Any]) -> List[Any]: return items +def _empty_array_is_verified_clean(raw_text: str) -> bool: + """True when an EMPTY findings array is a verifiable clean verdict. + + Accepted markers: the explicit ``NO_FINDINGS`` sentinel anywhere in the + response, or a response whose entire body (modulo code fences) is the bare + array itself. + """ + text = str(raw_text or "") + if "NO_FINDINGS" in text: + return True + stripped = text.strip() + if "```" in stripped: + # Unwrap a single fenced block: ```json\n[]\n``` or ```\n[]\n```. + parts = [chunk.strip() for chunk in stripped.split("```") if chunk.strip()] + if len(parts) == 1: + body = parts[0] + if body.startswith("json"): + body = body[4:].strip() + stripped = body + return stripped == "[]" + + def parse_model_review_results( result_json: Dict[str, Any], *, @@ -192,6 +214,14 @@ def parse_model_review_results( if parsed is None: records.append(_actor_record(actor, idx=idx, model_label=model_label, status="parse_failure", raw_text=raw_text)) continue + if not required and not parsed and not _empty_array_is_verified_clean(raw_text): + # Anti-refusal coverage contract: an empty array counts as a real + # "no findings" verdict only with the explicit NO_FINDINGS sentinel + # (or a bare `[]`-only response). A `[]` buried in refusal prose + # ("I cannot review this diff... []") must not enter the quorum as + # a clean PASS. + records.append(_actor_record(actor, idx=idx, model_label=model_label, status="parse_failure", raw_text=raw_text)) + continue actor_findings: List[Dict[str, Any]] = [] covered_items: set[str] = set() for entry in parsed: diff --git a/ouroboros/utils.py b/ouroboros/utils.py index c4fb71018..63d775924 100644 --- a/ouroboros/utils.py +++ b/ouroboros/utils.py @@ -69,6 +69,27 @@ def emit_log_event( def sha256_text(s: str) -> str: return hashlib.sha256(s.encode("utf-8")).hexdigest() + +def jsonl_generation_signature(path: pathlib.Path) -> dict: + """Identity signature of a JSONL log generation: first-line hash + size. + + SSOT shared by the chat-log consolidation writer (consolidator) and the + memory reader so a rotation/rewrite is detected identically on both sides. + """ + path = pathlib.Path(path) + if not path.exists(): + return {} + try: + stat = path.stat() + with path.open("r", encoding="utf-8") as handle: + first = next((line.strip() for line in handle if line.strip()), "") + return { + "first_line_sha256": hashlib.sha256(first.encode("utf-8", errors="replace")).hexdigest(), + "size": int(stat.st_size), + } + except OSError: + return {} + def read_text(path: pathlib.Path) -> str: return path.read_text(encoding="utf-8") @@ -80,12 +101,12 @@ def write_text(path: pathlib.Path, content: str) -> None: def atomic_write_json( path: pathlib.Path, - payload: Dict[str, Any], + payload: Any, *, trailing_newline: bool = False, fsync: bool = False, ) -> None: - """Atomically persist a JSON object using a unique sibling temp file.""" + """Atomically persist a JSON value (object or list) via a sibling temp file.""" path = pathlib.Path(path) path.parent.mkdir(parents=True, exist_ok=True) tmp_name = ( @@ -169,6 +190,56 @@ def read_json_dict(path: pathlib.Path) -> Optional[Dict[str, Any]]: return data if isinstance(data, dict) else None +def update_json_locked( + path: pathlib.Path, + mutator: Any, + *, + timeout_sec: float = 4.0, + stale_sec: float = 90.0, +) -> Dict[str, Any]: + """Locked read-modify-write of a durable JSON dict file. + + Acquires a sidecar ``.lock``, re-reads the CURRENT on-disk JSON + inside the lock (so the mutator always sees the latest state, closing the + lost-update window of unlocked load→merge→write sequences), applies + ``mutator(current) -> dict | None`` (``None`` aborts: the file is left + unchanged and the pre-mutation snapshot is returned), atomically writes + the result, and releases the lock. + + Raises ``TimeoutError`` on lock timeout — proceeding unlocked would + silently reintroduce the exact lost-update class this helper removes. + """ + from ouroboros.platform_layer import ( + acquire_exclusive_file_lock, + release_exclusive_file_lock, + ) + + path = pathlib.Path(path) + lock_path = path.with_name(path.name + ".lock") + lock_fd = acquire_exclusive_file_lock( + lock_path, timeout_sec=timeout_sec, stale_sec=stale_sec + ) + if lock_fd is None: + raise TimeoutError( + f"update_json_locked: could not acquire {lock_path} within {timeout_sec}s" + ) + try: + current = read_json_dict(path) or {} + updated = mutator(current) + if updated is None: + return current + atomic_write_json(path, updated) + return updated + finally: + release_exclusive_file_lock(lock_path, lock_fd) + + +def jsonl_append_lock_path(path: pathlib.Path) -> pathlib.Path: + """Sidecar lock path shared by ``append_jsonl`` writers and log rotation.""" + path_hash = hashlib.sha256(str(path.resolve()).encode("utf-8")).hexdigest()[:12] + return path.parent / f".append_jsonl_{path_hash}.lock" + + def append_jsonl(path: pathlib.Path, obj: Dict[str, Any]) -> bool: """Append a JSON object as a line to a JSONL file (concurrent-safe). @@ -190,8 +261,7 @@ def append_jsonl(path: pathlib.Path, obj: Dict[str, Any]) -> bool: write_retries = 3 retry_sleep_base_sec = 0.01 - path_hash = hashlib.sha256(str(path.resolve()).encode("utf-8")).hexdigest()[:12] - lock_path = path.parent / f".append_jsonl_{path_hash}.lock" + lock_path = jsonl_append_lock_path(path) lock_fd = None lock_acquired = False _written = False @@ -807,15 +877,11 @@ async def collect_evolution_metrics(repo_dir: str, data_dir: str | None = None) if cache_path and new_points: try: - cache_path.parent.mkdir(parents=True, exist_ok=True) - cache_path.write_text( - json.dumps({ - "schema": 1, - "points": points_by_tag, - "updated_at": utc_now_iso(), - }, ensure_ascii=False, indent=2, sort_keys=True), - encoding="utf-8", - ) + atomic_write_json(cache_path, { + "schema": 1, + "points": points_by_tag, + "updated_at": utc_now_iso(), + }) except OSError: log.warning("Failed to write evolution metrics cache: %s", cache_path, exc_info=True) diff --git a/ouroboros/workspace_executor.py b/ouroboros/workspace_executor.py index 718c2aa3e..1fb3fd889 100644 --- a/ouroboros/workspace_executor.py +++ b/ouroboros/workspace_executor.py @@ -737,6 +737,26 @@ def start_service( record.local_proc = proc record.backend_pid = str(proc.pid) record.backend_log_path = str(log_path) + # Write-through into the custody ledger: workspace records keep their + # own validation semantics, but the generation reaper now sees these + # too (previously swept only on clean shutdown). + try: + from ouroboros.process_custody import record_process + + record_process( + pathlib.Path(getattr(ctx, "drive_root")), + pid=proc.pid, + cmd=record.cmd, + purpose=f"workspace_service:{name}", + scope="task", + owner_task_id=record.task_id, + ) + except Exception: + import logging as _logging + + _logging.getLogger(__name__).debug( + "workspace service custody record failed", exc_info=True + ) else: if executor.network == "none": _assert_docker_network_none(executor.container_name) diff --git a/pyproject.toml b/pyproject.toml index 4fb00509a..abf4459e7 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "ouroboros" -version = "6.25.0rc3" +version = "6.26.0rc1" description = "Self-creating AI agent with constitution, background consciousness, and persistent identity" readme = "README.md" license = {text = "MIT"} diff --git a/scripts/build_repo_bundle.py b/scripts/build_repo_bundle.py index 5f5102a78..3629a3529 100644 --- a/scripts/build_repo_bundle.py +++ b/scripts/build_repo_bundle.py @@ -10,7 +10,6 @@ import os import pathlib import re import subprocess -import sys from urllib.parse import urlsplit, urlunsplit diff --git a/scripts/download_python_standalone.sh b/scripts/download_python_standalone.sh index 4222ccca2..1987c1286 100644 --- a/scripts/download_python_standalone.sh +++ b/scripts/download_python_standalone.sh @@ -1,5 +1,5 @@ #!/bin/bash -set -e +set -euo pipefail # Downloads python-build-standalone for macOS (arm64 + x86_64) and Linux (x86_64) # Run from repo root: bash scripts/download_python_standalone.sh @@ -14,8 +14,10 @@ ARCH=$(uname -m) if [ "$OS" = "Darwin" ]; then if [ "$ARCH" = "arm64" ]; then PLATFORM="aarch64-apple-darwin" + SHA256="e6634b06afa2ae79e664cf34174dad5d31b30117f90a01d533de3f4e9db6974e" elif [ "$ARCH" = "x86_64" ]; then PLATFORM="x86_64-apple-darwin" + SHA256="8d08ff2d9bb20566223f20307c8d9c31bdf020b1f1a237c2aa6efd1de651bf7b" else echo "Unsupported macOS architecture: $ARCH" exit 1 @@ -23,8 +25,10 @@ if [ "$OS" = "Darwin" ]; then elif [ "$OS" = "Linux" ]; then if [ "$ARCH" = "x86_64" ]; then PLATFORM="x86_64-unknown-linux-gnu" + SHA256="d71df61d1cdb59af4443912da8eeca744a52e782ff5deefa966ead893235a39e" elif [ "$ARCH" = "aarch64" ]; then PLATFORM="aarch64-unknown-linux-gnu" + SHA256="f2916a20f3de5500df5129e37dd0a213281dea29e226ec1ffe7fecdb10955533" else echo "Unsupported Linux architecture: $ARCH" exit 1 @@ -43,7 +47,21 @@ echo "URL: ${URL}" rm -rf "$DEST" _python_tmp mkdir -p _python_tmp -curl -L --progress-bar "$URL" | tar xz -C _python_tmp +# Pinned SHA256 (from the release's SHA256SUMS): a swapped/truncated archive +# fails here instead of becoming the packaged runtime. Update the pins when +# bumping RELEASE/PY_VERSION. +curl -L --fail --progress-bar "$URL" -o _python_tmp/"$FILENAME" +if command -v shasum >/dev/null 2>&1; then + ACTUAL="$(shasum -a 256 "_python_tmp/${FILENAME}" | awk '{print $1}')" +else + ACTUAL="$(sha256sum "_python_tmp/${FILENAME}" | awk '{print $1}')" +fi +if [ "$ACTUAL" != "$SHA256" ]; then + echo "SHA256 mismatch for ${FILENAME}: expected ${SHA256}, got ${ACTUAL} — refusing to install." + exit 1 +fi +tar xz -C _python_tmp -f _python_tmp/"$FILENAME" +rm -f _python_tmp/"$FILENAME" # Archive extracts to python/ — rename to python-standalone/ mv _python_tmp/python "$DEST" diff --git a/server.py b/server.py index ce2b63fcc..44d936efd 100644 --- a/server.py +++ b/server.py @@ -3,6 +3,7 @@ import asyncio import json import logging +import socket import os import pathlib @@ -100,18 +101,15 @@ def _restart_current_process(host: str, port: int) -> None: _restart_current_process_impl(host, port, repo_dir=REPO_DIR, log=log) from ouroboros.config import ( - SETTINGS_DEFAULTS as _SETTINGS_DEFAULTS, load_settings, save_settings, apply_settings_to_env as _apply_settings_to_env, ) from ouroboros.server_runtime import ( apply_runtime_provider_defaults, - has_local_routing, has_startup_ready_provider, needs_local_model_autostart, setup_remote_if_configured, ws_heartbeat_loop, ) -from ouroboros.onboarding_wizard import build_onboarding_html _supervisor_ready = threading.Event() _supervisor_error: Optional[str] = None @@ -141,7 +139,7 @@ def _describe_bg_consciousness_state(requested_enabled: bool) -> dict: elif requested_enabled and running: status = "running" detail = ( - f"Background consciousness is idle between wakeups." + "Background consciousness is idle between wakeups." + (f" Next wakeup in {next_wakeup_sec}s." if next_wakeup_sec > 0 else "") ) elif requested_enabled: @@ -212,7 +210,6 @@ def _process_bridge_updates(bridge, offset: int, ctx: Any) -> int: st = ctx.load_state() owner_id = st.get("owner_id") - owner_chat_id = st.get("owner_chat_id") lowered = text.strip().lower() is_slash_command = lowered.startswith("/") is_external_transport = source != "web" @@ -220,10 +217,7 @@ def _process_bridge_updates(bridge, offset: int, ctx: Any) -> int: # Global owner = primary chat for outbound notices (web on desktop, the # first transport on headless Colab). Bound once, on the first message. if owner_id is None and external_identity_present: - st["owner_id"] = user_id - st["owner_chat_id"] = chat_id owner_id = user_id - owner_chat_id = chat_id from supervisor.message_bus import log_chat @@ -255,8 +249,15 @@ def _process_bridge_updates(bridge, offset: int, ctx: Any) -> int: "transport": transport, "chat_id": chat_id, }) - st["last_owner_message_at"] = now_iso - ctx.save_state(st) + # Atomic owner-binding + activity stamp: the old load→(log/broadcast)→save + # span could overwrite concurrent budget/state writers with stale data. + def _stamp_owner_activity(live: dict) -> None: + if live.get("owner_id") is None and external_identity_present: + live["owner_id"] = user_id + live["owner_chat_id"] = chat_id + live["last_owner_message_at"] = now_iso + + ctx.update_state(_stamp_owner_activity) if not text and not image_base64: continue @@ -272,10 +273,13 @@ def _process_bridge_updates(bridge, offset: int, ctx: Any) -> int: owner_ext_id = st.get("owner_external_id") owner_ext_chat_id = st.get("owner_external_chat_id") if owner_ext_id is None: - st["owner_external_id"] = user_id - st["owner_external_chat_id"] = chat_id - st["owner_external_bound_at"] = now_iso - ctx.save_state(st) + def _bind_external_owner(live: dict) -> None: + if live.get("owner_external_id") is None: + live["owner_external_id"] = user_id + live["owner_external_chat_id"] = chat_id + live["owner_external_bound_at"] = now_iso + + ctx.update_state(_bind_external_owner) ctx.send_with_budget(chat_id, "✅ Owner chat registered. Send the command again to execute it.") continue try: @@ -504,6 +508,23 @@ def _run_supervisor(settings: dict) -> None: _apply_settings_to_env(settings) + # Supervisor revival (e.g. settings POST after a loop death) must not leak + # the previous generation: the old BackgroundConsciousness daemon thread + # would keep burning budget unreachable by /bg stop, and the cached direct + # chat agent stays bound to the OLD event queue (messages to a dead queue). + if _consciousness is not None: + try: + _consciousness.stop() + except Exception: + log.debug("Failed to stop previous consciousness instance", exc_info=True) + _consciousness = None + try: + from supervisor import workers as _workers_mod + + _workers_mod._chat_agent = None + except Exception: + log.debug("Failed to reset cached chat agent", exc_info=True) + try: from supervisor.message_bus import init as bus_init from supervisor.message_bus import LocalChatBridge @@ -521,7 +542,7 @@ def _run_supervisor(settings: dict) -> None: chat_bridge=bridge, ) - from supervisor.state import init as state_init, init_state, load_state, save_state + from supervisor.state import init as state_init, init_state, load_state, save_state, update_state from supervisor.state import append_jsonl, update_budget_from_usage, rotate_chat_log_if_needed state_init(DATA_DIR, float(settings.get("TOTAL_BUDGET", 10.0))) init_state() @@ -587,6 +608,14 @@ def _run_supervisor(settings: dict) -> None: }) except Exception: log.debug("Headless task drive prune failed", exc_info=True) + try: + from ouroboros.process_custody import reap_orphaned_processes + + reaped = reap_orphaned_processes(DATA_DIR) + if reaped: + log.info("Process custody reaper killed %d orphaned process(es): %s", len(reaped), reaped) + except Exception: + log.debug("Process custody startup reap failed", exc_info=True) try: from ouroboros import subagent_worktrees @@ -660,6 +689,7 @@ def _run_supervisor(settings: dict) -> None: bridge=bridge, WORKERS=WORKERS, PENDING=PENDING, RUNNING=RUNNING, MAX_WORKERS=max_workers, send_with_budget=send_with_budget, load_state=load_state, save_state=save_state, + update_state=update_state, update_budget_from_usage=update_budget_from_usage, append_jsonl=append_jsonl, enqueue_task=enqueue_task, cancel_task_by_id=cancel_task_by_id, queue_deep_self_review_task=queue_deep_self_review_task, persist_queue_snapshot=persist_queue_snapshot, @@ -682,6 +712,7 @@ def _run_supervisor(settings: dict) -> None: offset = 0 crash_count = 0 + _last_custody_reap = [time.time()] while not _restart_requested.is_set(): try: rotate_chat_log_if_needed(DATA_DIR) @@ -707,6 +738,19 @@ def _run_supervisor(settings: dict) -> None: check_scheduled_tasks() except Exception: log.warning("Scheduled task check failed", exc_info=True) + # Periodic custody reap: catches task-scoped processes whose owning + # task finished (or was SIGKILLed) within this server generation. + if time.time() - _last_custody_reap[0] > 600: + _last_custody_reap[0] = time.time() + try: + from ouroboros.process_custody import reap_orphaned_processes + from supervisor.queue import RUNNING as _running_tasks + + reap_orphaned_processes( + DATA_DIR, running_task_ids=set(_running_tasks.keys()), + ) + except Exception: + log.debug("Periodic custody reap failed", exc_info=True) try: from ouroboros.post_task_evolution import apply_pending_request from supervisor import state as _pte_state @@ -727,7 +771,24 @@ def _run_supervisor(settings: dict) -> None: crash_count += 1 log.error("Supervisor loop crash #%d: %s", crash_count, exc, exc_info=True) if crash_count >= 3: - log.critical("Supervisor exceeded max retries.") + # Visible death: previously the loop returned with + # _supervisor_ready still set and no _supervisor_error, so + # tasks silently stopped being assigned with a healthy-looking + # /api/state. Record the failure and tell the owner. + _supervisor_error = f"Supervisor loop died after 3 consecutive crashes: {exc}" + _supervisor_ready.clear() + log.critical("Supervisor exceeded max retries: %s", _supervisor_error) + try: + st = load_state() + if st.get("owner_chat_id"): + send_with_budget( + int(st["owner_chat_id"]), + "🛑 Supervisor loop died after repeated crashes; tasks are no " + "longer being assigned. Saving settings or restarting the app " + f"will revive it. Last error: {exc}", + ) + except Exception: + log.debug("Failed to notify owner about supervisor death", exc_info=True) return time.sleep(min(30, 2 ** crash_count)) _supervisor_thread = None @@ -913,6 +974,18 @@ async def lifespan(app): init_global_supervisor(lifespan_drive_root) host_service_app = create_host_service_app(lifespan_drive_root) host_port = host_service_port() + # Probe the port first: uvicorn's Server.startup() calls sys.exit(1) on a + # bind error, and SystemExit raised inside an asyncio task escapes + # run_forever and takes down the WHOLE main server (a stale prior + # instance still holding the port is exactly the realistic trigger). + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as _probe: + _probe.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + try: + _probe.bind((DEFAULT_HOST_SERVICE_HOST, host_port)) + except OSError as bind_exc: + raise RuntimeError( + f"Host Service port {host_port} is busy: {bind_exc}" + ) from bind_exc host_service_config = uvicorn.Config( host_service_app, host=DEFAULT_HOST_SERVICE_HOST, @@ -1080,6 +1153,14 @@ app.app.state.default_port = DEFAULT_PORT # type: ignore[attr-defined] app.app.state.start_supervisor_if_needed = _start_supervisor_if_needed # type: ignore[attr-defined] +_ACTUAL_BOUND_PORT: Optional[int] = None + + +def _actual_bound_port() -> int: + """Port the server actually bound (set in main(); DEFAULT_PORT before that).""" + return _ACTUAL_BOUND_PORT if _ACTUAL_BOUND_PORT else DEFAULT_PORT + + def _emergency_process_cleanup(*, port_sweep: bool = True) -> None: """Kill child processes, workers, companions, and runtime port holders.""" try: @@ -1128,8 +1209,10 @@ def _emergency_process_cleanup(*, port_sweep: bool = True) -> None: except Exception: pass if port_sweep: - kill_process_on_port(DEFAULT_PORT) - kill_process_on_port(8766) + # Sweep the ACTUALLY bound port (find_free_port may have moved off + # DEFAULT_PORT); the old hardcoded 8765/8766 pair could kill an + # unrelated process on a custom-port install. + kill_process_on_port(_actual_bound_port()) try: from ouroboros.extension_companion import panic_kill_all from ouroboros.gateway.host_service import host_service_port @@ -1159,6 +1242,8 @@ def main() -> int: actual_port = find_free_port(args.host, args.port) if actual_port != args.port: log.info("Port %d busy on %s, using %d instead", args.port, args.host, actual_port) + global _ACTUAL_BOUND_PORT + _ACTUAL_BOUND_PORT = actual_port write_port_file(PORT_FILE, actual_port) log.info("Starting Ouroboros server on %s:%d", args.host, actual_port) config = uvicorn.Config( diff --git a/supervisor/events.py b/supervisor/events.py index c8ebfa159..1f5dd7505 100644 --- a/supervisor/events.py +++ b/supervisor/events.py @@ -2,16 +2,14 @@ from __future__ import annotations -import json import logging import pathlib import subprocess -import sys import time import uuid from typing import Any, Dict, Optional -from ouroboros.utils import truncate_for_log, utc_now_iso +from ouroboros.utils import append_jsonl, truncate_for_log, utc_now_iso from ouroboros.config import get_max_active_subagents_per_root, get_max_subagent_depth from ouroboros.tool_capabilities import ACTING_SUBAGENT_MODE, LOCAL_READONLY_SUBAGENT_MODE from ouroboros.contracts.task_constraint import VALID_WRITE_SURFACES @@ -348,27 +346,28 @@ def _handle_llm_usage(evt: Dict[str, Any], ctx: Any) -> None: usage: Dict[str, Any] = usage_raw if isinstance(usage_raw, dict) else {} # Normalize usage across loop.py, web_search, and claude_code_edit producers. - prompt_tokens = int( - usage.get("prompt_tokens") - or usage.get("input_tokens") - or evt.get("prompt_tokens") - or 0 + # Tolerant coercion: one malformed token field must not raise and drop the + # whole round from the budget ledger and events.jsonl (the exception would + # be swallowed by dispatch_event and the cost silently lost). + def _tolerant_int(*candidates: Any) -> int: + for value in candidates: + if value in (None, ""): + continue + try: + return int(float(value)) + except (TypeError, ValueError): + log.warning("llm_usage: non-numeric token field %r ignored", value) + return 0 + + prompt_tokens = _tolerant_int( + usage.get("prompt_tokens"), usage.get("input_tokens"), evt.get("prompt_tokens") ) - completion_tokens = int( - usage.get("completion_tokens") - or usage.get("output_tokens") - or evt.get("completion_tokens") - or 0 + completion_tokens = _tolerant_int( + usage.get("completion_tokens"), usage.get("output_tokens"), evt.get("completion_tokens") ) - cached_tokens = int( - usage.get("cached_tokens") - or evt.get("cached_tokens") - or 0 - ) - cache_write_tokens = int( - usage.get("cache_write_tokens") - or evt.get("cache_write_tokens") - or 0 + cached_tokens = _tolerant_int(usage.get("cached_tokens"), evt.get("cached_tokens")) + cache_write_tokens = _tolerant_int( + usage.get("cache_write_tokens"), evt.get("cache_write_tokens") ) prompt_cache_ttl = str( usage.get("prompt_cache_ttl") @@ -395,7 +394,6 @@ def _handle_llm_usage(evt: Dict[str, Any], ctx: Any) -> None: } ctx.update_budget_from_usage(usage_for_budget) - from ouroboros.utils import utc_now_iso, append_jsonl try: append_jsonl(ctx.DRIVE_ROOT / "logs" / "events.jsonl", { "ts": evt.get("ts", utc_now_iso()), @@ -545,7 +543,6 @@ def _handle_task_done(evt: Dict[str, Any], ctx: Any) -> None: final_task_result = {} # Persist here so send_message reaches the UI before task_done collapses the card. - from ouroboros.utils import utc_now_iso, append_jsonl outcome_axes = normalize_outcome_axes({**evt, **(final_task_result if isinstance(final_task_result, dict) else {})}) reason_code = final_task_result.get("reason_code") or evt.get("reason_code") artifact_status = final_task_result.get("artifact_status") or evt.get("artifact_status") @@ -588,7 +585,6 @@ def _handle_task_done(evt: Dict[str, Any], ctx: Any) -> None: log.warning("Failed to log task_done to events.jsonl", exc_info=True) if task_type == "evolution": - st = ctx.load_state() # Meaningful evolution work has non-trivial cost plus at least one round. # eff_* falls back to the persisted (reconstructed) result on abnormal # termination so a zeroed terminal event cannot understate the tally or @@ -644,29 +640,39 @@ def _handle_task_done(evt: Dict[str, Any], ctx: Any) -> None: if replayed_evolution_terminal: pass elif not failed_by_axes and rounds >= 1: - st["evolution_consecutive_failures"] = 0 - ctx.save_state(st) + from supervisor.state import update_state + + update_state(lambda live: live.update(evolution_consecutive_failures=0)) else: - failures = int(st.get("evolution_consecutive_failures") or 0) + 1 - st["evolution_consecutive_failures"] = failures - ctx.save_state(st) + from supervisor.state import update_state + + failures_box: Dict[str, int] = {} + + def _bump_failures(live: Dict[str, Any]) -> None: + failures_box["n"] = int(live.get("evolution_consecutive_failures") or 0) + 1 + live["evolution_consecutive_failures"] = failures_box["n"] + + update_state(_bump_failures) ctx.append_jsonl( ctx.DRIVE_ROOT / "logs" / "supervisor.jsonl", { "ts": utc_now_iso(), "type": "evolution_task_failure_tracked", "task_id": task_id, - "consecutive_failures": failures, + "consecutive_failures": failures_box.get("n", 0), "cost_usd": cost, "rounds": rounds, }, ) try: - cur = ctx.load_state() - if cur.get("post_task_autostop"): - cur["evolution_mode_enabled"] = False - cur["post_task_autostop"] = False - ctx.save_state(cur) + from supervisor.state import update_state + + def _consume_autostop(live: Dict[str, Any]) -> None: + if live.get("post_task_autostop"): + live["evolution_mode_enabled"] = False + live["post_task_autostop"] = False + + update_state(_consume_autostop) except Exception: log.debug("Post-task evolution autostop failed", exc_info=True) @@ -710,9 +716,13 @@ def _handle_task_done(evt: Dict[str, Any], ctx: Any) -> None: "artifact_status": str(effective_result.get("artifact_status") or ""), }, ) - ctx.RUNNING.pop(str(task_id), None) - if wid in ctx.WORKERS and ctx.WORKERS[wid].busy_task_id == task_id: - ctx.WORKERS[wid].busy_task_id = None + from supervisor.queue import _queue_lock + + with _queue_lock: + if task_id: + ctx.RUNNING.pop(str(task_id), None) + if wid in ctx.WORKERS and ctx.WORKERS[wid].busy_task_id == task_id: + ctx.WORKERS[wid].busy_task_id = None ctx.persist_queue_snapshot(reason="task_done") try: ctx.bridge.push_log(task_done_event) @@ -1532,14 +1542,17 @@ def _handle_toggle_evolution(evt: Dict[str, Any], ctx: Any) -> None: if st.get("owner_chat_id"): ctx.send_with_budget(int(st["owner_chat_id"]), block) return - st = ctx.load_state() - st["evolution_mode_enabled"] = enabled - if enabled: - st["evolution_consecutive_failures"] = 0 - # Symmetry with the owner /evolve path: an explicit toggle must not inherit a - # stale post-task one-shot autostop that would disable the campaign after one cycle. - st["post_task_autostop"] = False - ctx.save_state(st) + from supervisor.state import update_state + + def _toggle_evolution(live: Dict[str, Any]) -> None: + live["evolution_mode_enabled"] = enabled + if enabled: + live["evolution_consecutive_failures"] = 0 + # Symmetry with the owner /evolve path: an explicit toggle must not inherit a + # stale post-task one-shot autostop that would disable the campaign after one cycle. + live["post_task_autostop"] = False + + st = update_state(_toggle_evolution) try: from supervisor.queue import pause_evolution_campaign, start_evolution_campaign @@ -1648,7 +1661,6 @@ def _handle_send_video(evt: Dict[str, Any], ctx: Any) -> None: def _handle_owner_message_injected(evt: Dict[str, Any], ctx: Any) -> None: """Log owner injections so health checks can detect duplicate processing.""" - from ouroboros.utils import utc_now_iso try: ctx.append_jsonl(ctx.DRIVE_ROOT / "logs" / "events.jsonl", { "ts": evt.get("ts", utc_now_iso()), diff --git a/supervisor/git_ops.py b/supervisor/git_ops.py index 9634fcd36..f4c6c25f8 100644 --- a/supervisor/git_ops.py +++ b/supervisor/git_ops.py @@ -177,9 +177,12 @@ def _read_update_intent() -> Dict[str, Any]: def _write_update_intent(payload: Dict[str, Any]) -> None: + # Atomic: a torn marker would make the next restart silently skip the + # prepared managed update (reader fails closed on parse errors). + from ouroboros.utils import atomic_write_json + path = _update_intent_marker_path() - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(json.dumps(payload, ensure_ascii=True, indent=2) + "\n", encoding="utf-8") + atomic_write_json(path, payload, trailing_newline=True) def _clear_update_intent() -> None: @@ -1030,7 +1033,7 @@ def safe_restart( if t2["ok"]: return True, f"OK: fell back to {BRANCH_STABLE}" - return False, f"Both branches failed import (dev and stable)" + return False, "Both branches failed import (dev and stable)" def list_versions(max_count: int = 50) -> List[Dict[str, Any]]: diff --git a/supervisor/message_bus.py b/supervisor/message_bus.py index ad1748541..4a5dce772 100644 --- a/supervisor/message_bus.py +++ b/supervisor/message_bus.py @@ -7,11 +7,11 @@ import logging import queue import re import threading -from typing import Any, Dict, List, Optional +from typing import Any, Dict, List, Optional, Tuple from ouroboros.contracts.chat_id_policy import is_a2a_chat_id from ouroboros.event_bus import CHAT_OUTBOUND, CHAT_PHOTO, CHAT_TYPING, CHAT_VIDEO, publish_event -from supervisor.state import append_jsonl, load_state, save_state +from supervisor.state import append_jsonl, load_state from ouroboros.utils import utc_now_iso log = logging.getLogger(__name__) @@ -70,7 +70,6 @@ class LocalChatBridge: def __init__(self, settings: Optional[Dict[str, Any]] = None): self._inbox = queue.Queue() # user -> agent - self._outbox = queue.Queue() # agent -> UI self._log_queue: queue.Queue = queue.Queue(maxsize=1000) self._update_counter = 0 self._broadcast_fn = None # set by server.py for WebSocket streaming @@ -167,10 +166,13 @@ class LocalChatBridge: *, sender_session_id: str = "", client_message_id: str = "", + image_base64: str = "", + image_mime: str = "", + image_caption: str = "", task_metadata: Optional[Dict[str, Any]] = None, ) -> None: clean_text = str(text or "").strip() - if not clean_text: + if not clean_text and not image_base64: return ts = utc_now_iso() if self._broadcast_fn: @@ -191,6 +193,9 @@ class LocalChatBridge: sender_label="", sender_session_id=sender_session_id, client_message_id=client_message_id, + image_base64=image_base64, + image_mime=image_mime, + image_caption=image_caption, task_metadata=task_metadata, ) @@ -265,7 +270,6 @@ class LocalChatBridge: } if meta: msg.update(meta) - self._outbox.put(msg) with self._response_subs_lock: subs = [(sid, cb) for sid, (cid, cb) in self._response_subs.items() if cid == chat_id and not is_progress] @@ -307,10 +311,6 @@ class LocalChatBridge: """Send typing indicator to UI/event subscribers.""" if is_a2a_chat_id(chat_id): return True - self._outbox.put({ - "type": "action", - "content": action, - }) if self._broadcast_fn: self._broadcast_fn({"type": "typing", "action": action}) typing_transport = dict(self._chat_transports.get(int(chat_id or 0), {}) or {}) @@ -336,7 +336,6 @@ class LocalChatBridge: "caption": caption, "ts": utc_now_iso(), } - self._outbox.put(msg) if self._broadcast_fn: self._broadcast_fn(msg) photo_transport = dict(self._chat_transports.get(int(chat_id or 0), {}) or {}) @@ -369,7 +368,6 @@ class LocalChatBridge: "caption": caption, "ts": utc_now_iso(), } - self._outbox.put(msg) if self._broadcast_fn: self._broadcast_fn(msg) video_transport = dict(self._chat_transports.get(int(chat_id or 0), {}) or {}) @@ -417,6 +415,9 @@ class LocalChatBridge: sender_session_id: str = "", client_message_id: str = "", suppress_chat_log: bool = False, + image_base64: str = "", + image_mime: str = "", + image_caption: str = "", task_constraint: Optional[Dict[str, Any]] = None, task_metadata: Optional[Dict[str, Any]] = None, ): @@ -426,6 +427,9 @@ class LocalChatBridge: text, sender_session_id=sender_session_id, client_message_id=client_message_id, + image_base64=image_base64, + image_mime=image_mime, + image_caption=image_caption, task_metadata=task_metadata, ) return @@ -436,12 +440,6 @@ class LocalChatBridge: task_metadata=task_metadata, ) - def ui_receive(self, timeout: float = 0.1) -> Optional[Dict[str, Any]]: - """Poll agent messages for the web UI.""" - try: - return self._outbox.get(timeout=timeout) - except queue.Empty: - return None def _strip_markdown(text: str) -> str: @@ -495,22 +493,25 @@ def _format_budget_line(st: Dict[str, Any]) -> str: def budget_line(force: bool = False) -> str: try: - st = load_state() + from supervisor.state import update_state + every = max(1, int(BUDGET_REPORT_EVERY_MESSAGES)) - if force: - st["budget_messages_since_report"] = 0 - save_state(st) - return _format_budget_line(st) + report_box: Dict[str, Any] = {"emit": False} - counter = int(st.get("budget_messages_since_report") or 0) + 1 - if counter < every: - st["budget_messages_since_report"] = counter - save_state(st) - return "" + def _tick_counter(live: Dict[str, Any]) -> None: + if force: + live["budget_messages_since_report"] = 0 + report_box["emit"] = True + return + counter = int(live.get("budget_messages_since_report") or 0) + 1 + if counter < every: + live["budget_messages_since_report"] = counter + return + live["budget_messages_since_report"] = 0 + report_box["emit"] = True - st["budget_messages_since_report"] = 0 - save_state(st) - return _format_budget_line(st) + st = update_state(_tick_counter) + return _format_budget_line(st) if report_box["emit"] else "" except Exception: log.debug("Suppressed exception in budget_line", exc_info=True) return "" diff --git a/supervisor/queue.py b/supervisor/queue.py index eb1c144ec..470dd6183 100644 --- a/supervisor/queue.py +++ b/supervisor/queue.py @@ -13,9 +13,8 @@ from typing import Any, Dict, List, Optional, Tuple from zoneinfo import ZoneInfo from supervisor.state import ( - load_state, save_state, append_jsonl, atomic_write_text, - QUEUE_SNAPSHOT_PATH, budget_pct, TOTAL_BUDGET_LIMIT, - budget_remaining, EVOLUTION_BUDGET_RESERVE, reconstruct_task_cost, + load_state, append_jsonl, atomic_write_text, + QUEUE_SNAPSHOT_PATH, budget_remaining, EVOLUTION_BUDGET_RESERVE, reconstruct_task_cost, ) from supervisor.message_bus import send_with_budget from ouroboros.config import FINALIZATION_GRACE_DEFAULT_SEC, get_finalization_grace_sec @@ -133,18 +132,20 @@ def drain_all_pending() -> list: def enqueue_task(task: Dict[str, Any], front: bool = False) -> Dict[str, Any]: - """Add task to PENDING.""" + """Add task to PENDING (thread-safe: HTTP handlers enqueue concurrently + with the supervisor main loop, so the mutation must hold the queue lock).""" t = dict(task) attach_task_contract(t) - QUEUE_SEQ_COUNTER_REF["value"] += 1 - seq = QUEUE_SEQ_COUNTER_REF["value"] - t.setdefault("priority", _task_priority(str(t.get("type") or ""))) - _att = t.get("_attempt") - t.setdefault("_attempt", int(_att) if _att is not None else 1) - t["_queue_seq"] = -seq if front else seq - t["queued_at"] = utc_now_iso() - PENDING.append(t) - sort_pending() + with _queue_lock: + QUEUE_SEQ_COUNTER_REF["value"] += 1 + seq = QUEUE_SEQ_COUNTER_REF["value"] + t.setdefault("priority", _task_priority(str(t.get("type") or ""))) + _att = t.get("_attempt") + t.setdefault("_attempt", int(_att) if _att is not None else 1) + t["_queue_seq"] = -seq if front else seq + t["queued_at"] = utc_now_iso() + PENDING.append(t) + sort_pending() return t @@ -734,9 +735,20 @@ def update_evolution_campaign_after_task( def persist_queue_snapshot(reason: str = "") -> None: - """Persist queue snapshot for restart/recovery diagnostics.""" + """Persist queue snapshot for restart/recovery diagnostics. + + Snapshots PENDING/RUNNING under the queue lock: iterating the live dicts + while HTTP handlers mutate them raised "dictionary changed size during + iteration" in the supervisor loop (counted toward its crash limit). + """ + with _queue_lock: + pending_items = [dict(t) for t in PENDING] + running_items = [ + (task_id, dict(meta) if isinstance(meta, dict) else {}) + for task_id, meta in RUNNING.items() + ] pending_rows = [] - for t in PENDING: + for t in pending_items: pending_rows.append({ "id": t.get("id"), "type": t.get("type"), "priority": t.get("priority"), "attempt": t.get("_attempt"), "queued_at": t.get("queued_at"), @@ -773,7 +785,7 @@ def persist_queue_snapshot(reason: str = "") -> None: }) running_rows = [] now = time.time() - for task_id, meta in RUNNING.items(): + for task_id, meta in running_items: task = meta.get("task") if isinstance(meta, dict) else {} started = float(meta.get("started_at") or 0.0) if isinstance(meta, dict) else 0.0 hb = float(meta.get("last_heartbeat_at") or 0.0) if isinstance(meta, dict) else 0.0 @@ -787,7 +799,7 @@ def persist_queue_snapshot(reason: str = "") -> None: payload = { "ts": utc_now_iso(), "reason": reason, - "pending_count": len(PENDING), "running_count": len(RUNNING), + "pending_count": len(pending_items), "running_count": len(running_items), "pending": pending_rows, "running": running_rows, } try: @@ -1083,16 +1095,29 @@ def cancel_running_evolution_tasks(reason: str = "evolution stopped") -> List[st def enforce_task_timeouts() -> None: - """Enforce soft/hard timeouts for running tasks.""" + """Enforce soft/hard timeouts for running tasks. + + Holds the queue lock for the whole pass: RUNNING pops and worker respawn + decisions raced with HTTP cancel handlers (double respawn → orphaned + worker; wrong-task dequeue). The RLock keeps nested respawn/assign calls + re-entrant. + """ # Avoid circular dependency during module load. from supervisor import workers - + if not RUNNING: return now = time.time() st = load_state() owner_chat_id = int(st.get("owner_chat_id") or 0) + with _queue_lock: + _enforce_task_timeouts_locked(workers, now, owner_chat_id, st) + + +def _enforce_task_timeouts_locked( + workers: Any, now: float, owner_chat_id: int, st: Dict[str, Any] +) -> None: for task_id, meta in list(RUNNING.items()): if not isinstance(meta, dict): continue @@ -1503,19 +1528,22 @@ def enqueue_evolution_task_if_needed() -> None: # Defensive net: light mode must never run evolution even if the flag was # left enabled (e.g. carried across a restart into light mode). Disable and # pause once; entry points already refuse new starts up front. + from supervisor.state import update_state + + def _disable_evolution(live: Dict[str, Any]) -> None: + live["evolution_mode_enabled"] = False + block = evolution_block_reason() if block: pause_evolution_campaign("blocked in light runtime mode") - st["evolution_mode_enabled"] = False - save_state(st) + update_state(_disable_evolution) send_with_budget(int(owner_chat_id), block) return consecutive_failures = int(st.get("evolution_consecutive_failures") or 0) if consecutive_failures >= 3: pause_evolution_campaign("paused after consecutive failures") - st["evolution_mode_enabled"] = False - save_state(st) + update_state(_disable_evolution) send_with_budget( int(owner_chat_id), f"🧬⚠️ Evolution paused: {consecutive_failures} consecutive failures. " @@ -1526,8 +1554,7 @@ def enqueue_evolution_task_if_needed() -> None: remaining = budget_remaining(st) if remaining < EVOLUTION_BUDGET_RESERVE: pause_evolution_campaign("budget reserve reached") - st["evolution_mode_enabled"] = False - save_state(st) + update_state(_disable_evolution) send_with_budget(int(owner_chat_id), f"💸 Evolution stopped: ${remaining:.2f} remaining (reserve ${EVOLUTION_BUDGET_RESERVE:.0f} for conversations).") return cycle = int(st.get("evolution_cycle") or 0) + 1 @@ -1544,8 +1571,11 @@ def enqueue_evolution_task_if_needed() -> None: } attach_task_contract(task) enqueue_task(task) - st["evolution_cycle"] = cycle - st["last_evolution_task_at"] = utc_now_iso() - save_state(st) + + def _record_cycle(live: Dict[str, Any]) -> None: + live["evolution_cycle"] = cycle + live["last_evolution_task_at"] = utc_now_iso() + + update_state(_record_cycle) # The generic "Evolution task started." lifecycle message (workers.py) # already announces the cycle start, so no extra enqueue bubble here. diff --git a/supervisor/state.py b/supervisor/state.py index 9c83d65a6..565b5c5b7 100644 --- a/supervisor/state.py +++ b/supervisor/state.py @@ -8,7 +8,7 @@ import os import pathlib import time import uuid -from typing import Any, Dict, Optional, Tuple +from typing import Any, Dict, List, Optional, Tuple from ouroboros.platform_layer import acquire_exclusive_file_lock, release_exclusive_file_lock from ouroboros.utils import append_jsonl, iter_llm_usage_events, llm_usage_cost, utc_now_iso @@ -141,8 +141,20 @@ def _save_state_unlocked(st: Dict[str, Any]) -> None: atomic_write_text(STATE_LAST_GOOD_PATH, payload) +def _warn_state_unlocked(op: str, lock_fd: Optional[int]) -> None: + """Loud trail when the state lock could not be acquired. + + Proceeding unlocked is a deliberate availability tradeoff (a wedged lock + must not freeze the supervisor), but it must never be silent: an unlocked + write is exactly the lost-update class this lock exists to prevent. + """ + if lock_fd is None: + log.error("state.json %s proceeding WITHOUT lock (timeout on %s)", op, STATE_LOCK_PATH) + + def load_state() -> Dict[str, Any]: lock_fd = acquire_file_lock(STATE_LOCK_PATH) + _warn_state_unlocked("load", lock_fd) try: return _load_state_unlocked() finally: @@ -151,6 +163,7 @@ def load_state() -> Dict[str, Any]: def save_state(st: Dict[str, Any]) -> None: lock_fd = acquire_file_lock(STATE_LOCK_PATH) + _warn_state_unlocked("save", lock_fd) try: _save_state_unlocked(st) finally: @@ -175,6 +188,7 @@ def update_state(mutator) -> Dict[str, Any]: STATE_LOCK is not re-entrant within a process, so re-entering would block. """ lock_fd = acquire_file_lock(STATE_LOCK_PATH) + _warn_state_unlocked("update", lock_fd) try: st = _load_state_unlocked() mutator(st) @@ -357,6 +371,7 @@ def update_budget_from_usage(usage: Dict[str, Any]) -> None: # Keep the lock around local counters only; network check runs outside it. lock_fd = acquire_file_lock(STATE_LOCK_PATH) + _warn_state_unlocked("budget-update", lock_fd) try: st = _load_state_unlocked() cost = usage.get("cost") if isinstance(usage, dict) else None @@ -624,7 +639,12 @@ def status_text(workers_dict: Dict[int, Any], pending_list: list, running_dict: def rotate_chat_log_if_needed(drive_root: pathlib.Path, max_bytes: int = 800_000) -> None: - """Rotate chat log if it exceeds max_bytes.""" + """Rotate chat log if it exceeds max_bytes. + + Rotation is an atomic ``os.replace`` rename performed under the SAME + sidecar lock that ``append_jsonl`` writers take — the old copy+truncate + destroyed any line appended between the read and the truncate. + """ chat = drive_root / "logs" / "chat.jsonl" if not chat.exists(): return @@ -633,5 +653,18 @@ def rotate_chat_log_if_needed(drive_root: pathlib.Path, max_bytes: int = 800_000 ts = utc_now_iso().replace("-", "").replace(":", "").split(".")[0] archive_path = drive_root / "archive" / f"chat_{ts}.jsonl" archive_path.parent.mkdir(parents=True, exist_ok=True) - archive_path.write_bytes(chat.read_bytes()) - chat.write_text("", encoding="utf-8") + + from ouroboros.utils import jsonl_append_lock_path + + lock_path = jsonl_append_lock_path(chat) + lock_fd = acquire_exclusive_file_lock(lock_path, timeout_sec=2.0, stale_sec=10.0) + if lock_fd is None: + log.warning("chat.jsonl rotation skipped: append lock busy") + return + try: + if not chat.exists() or chat.stat().st_size < max_bytes: + return + os.replace(chat, archive_path) + chat.touch() + finally: + release_exclusive_file_lock(lock_path, lock_fd) diff --git a/supervisor/workers.py b/supervisor/workers.py index 959efa2eb..c3b5993cb 100644 --- a/supervisor/workers.py +++ b/supervisor/workers.py @@ -16,7 +16,6 @@ from dataclasses import dataclass from typing import Any, Dict, List, Optional, Tuple, Union from supervisor.state import load_state, append_jsonl, reconstruct_task_cost -from supervisor import git_ops from supervisor.message_bus import send_with_budget from ouroboros.outcomes import EXECUTION_INFRA_FAILED, terminal_outcome_axes from ouroboros.utils import utc_now_iso @@ -319,6 +318,14 @@ def worker_main(wid: int, in_q: Any, out_q: Any, repo_dir: str, drive_root: str) _os.environ["OUROBOROS_IN_WORKER"] = "1" from ouroboros.platform_layer import create_new_session create_new_session() + # Lifeline: if the supervisor dies abruptly, this worker is reparented to + # init and would keep running LLM rounds invisibly — group-suicide instead. + try: + from ouroboros.process_custody import start_parent_lifeline + + start_parent_lifeline(label=f"worker-{wid}") + except Exception: + pass # Stream this worker's append_jsonl log lines to the dashboard Logs panel. # The WS log sink lives only in the main process, so without this every # worker-task log line (queued/evolution/review/subagent) is written to file @@ -634,6 +641,22 @@ def _record_worker_pids() -> None: ) except Exception: log.debug("Failed to record worker pids", exc_info=True) + # Write-through into the custody ledger (SSOT for the generation reaper); + # worker_pids.json stays as the legacy session-leader reap path. + try: + from ouroboros.process_custody import record_process + + for w in WORKERS.values(): + if w.proc.pid: + record_process( + DRIVE_ROOT, + pid=int(w.proc.pid), + cmd=f"ouroboros-worker-{w.wid}", + purpose=f"worker:{w.wid}", + scope="session", + ) + except Exception: + log.debug("Failed to ledger worker pids", exc_info=True) def reap_orphaned_workers() -> int: @@ -987,10 +1010,22 @@ def assign_tasks() -> None: queue.persist_queue_snapshot(reason="assign_task") def ensure_workers_healthy() -> None: + """Detect dead workers, finalize/requeue their tasks, respawn. + + Runs under the queue lock: the RUNNING pops and respawn decisions here + raced with HTTP cancel handlers (double respawn → orphaned worker, and + "dict changed size" crashes in concurrent iteration). RLock keeps the + nested enqueue/respawn/persist calls re-entrant. + """ from supervisor import queue # Workers need init time after spawn. if (time.time() - _LAST_SPAWN_TIME) < _SPAWN_GRACE_SEC: return + with _queue_lock: + _ensure_workers_healthy_locked(queue) + + +def _ensure_workers_healthy_locked(queue: Any) -> None: busy_crashes = 0 dead_detections = 0 crashed_tasks = [] diff --git a/tests/test_acting_subagents.py b/tests/test_acting_subagents.py index 117699f32..00334ef52 100644 --- a/tests/test_acting_subagents.py +++ b/tests/test_acting_subagents.py @@ -425,7 +425,6 @@ def test_integrate_protected_blocked_in_advanced(tmp_path): def test_integrate_corrupt_sha_refused(tmp_path): from ouroboros.tools.subagent_integration import _integrate_subagent_patch - from ouroboros.artifacts import task_artifact_dir_path repo = tmp_path / "repo" _init_repo(repo, {"a.txt": "hi\n"}) drive = tmp_path / "data"; drive.mkdir() @@ -513,7 +512,6 @@ def test_integrate_acting_into_own_worktree_ok(tmp_path): def test_integrate_external_workspace_verifies_shared_files_without_reapplying(tmp_path): - from ouroboros.task_results import task_result_path from ouroboros.tools.subagent_integration import _integrate_subagent_patch workspace = tmp_path / "workspace" @@ -542,7 +540,6 @@ def test_integrate_external_workspace_verifies_shared_files_without_reapplying(t def test_integrate_external_workspace_rejects_existing_but_mismatched_files(tmp_path): - from ouroboros.task_results import task_result_path from ouroboros.tools.subagent_integration import _integrate_subagent_patch workspace = tmp_path / "workspace" diff --git a/tests/test_advisory_observability.py b/tests/test_advisory_observability.py index d419c3e66..9aa323125 100644 --- a/tests/test_advisory_observability.py +++ b/tests/test_advisory_observability.py @@ -483,7 +483,7 @@ def test_skipped_run_hash_mismatch_reported_as_stale(monkeypatch, tmp_path): # Write a skipped run with a fake (stale) hash directly into state from ouroboros.review_state import ( - AdvisoryRunRecord, AdvisoryReviewState, load_state, save_state, + AdvisoryRunRecord, AdvisoryReviewState, save_state, ) old_hash = "000000000000000000000000000000000000000000000000" run = AdvisoryRunRecord( diff --git a/tests/test_advisory_preflight.py b/tests/test_advisory_preflight.py index 18e392b5d..8a1ed7f34 100644 --- a/tests/test_advisory_preflight.py +++ b/tests/test_advisory_preflight.py @@ -10,7 +10,6 @@ from __future__ import annotations import pathlib import subprocess -import tempfile import unittest.mock as mock import pytest @@ -448,7 +447,6 @@ class TestHandleAdvisoryPreReviewSurfacesPreflightBlocked: def test_advisory_auto_syncs_release_metadata_when_version_staged(self, tmp_path, monkeypatch): from ouroboros.tools import claude_advisory_review as adv - import json as _json repo_root = tmp_path / "repo" repo_root.mkdir(parents=True, exist_ok=True) diff --git a/tests/test_browser_isolation.py b/tests/test_browser_isolation.py index 654a2ed2f..a6c28034d 100644 --- a/tests/test_browser_isolation.py +++ b/tests/test_browser_isolation.py @@ -106,10 +106,12 @@ class TestBrowserModuleState: assert "Chrome/131.0.0.0" in contexts[-1].kwargs["user_agent"] assert getattr(ctx.browser_state, "_thread_id", None) is not None assert getattr(ctx.browser_state, "_browser_engine", None) == "chromium" - assert routes == [ + assert routes[:2] == [ ("**/api/owner/context-mode", browser_mod._block_context_mode_owner_post), ("**/api/settings", browser_mod._block_owner_settings_post), ] + # v6.26.0: the main agent gets a metadata-only SSRF route guard too. + assert len(routes) == 3 and routes[2][0] == "**/*" browser_mod._ensure_browser(ctx, engine="webkit", device="iphone 13") assert contexts[-1].kwargs["viewport"] == {"width": 390, "height": 844} @@ -323,7 +325,6 @@ class TestSetPlaywrightBrowsersPathIfBundled: def test_no_op_when_env_already_set(self, monkeypatch, tmp_path): monkeypatch.setenv("PLAYWRIGHT_BROWSERS_PATH", "/some/custom/path") - import importlib import ouroboros.tools.browser as bmod monkeypatch.setattr(bmod.sys, "platform", "darwin", raising=False) # Should not overwrite existing env var @@ -394,7 +395,6 @@ class TestSetPlaywrightBrowsersPathIfBundled: def test_import_time_side_effect_sets_env_when_bundled(self, monkeypatch, tmp_path): """Module-import calls _set_playwright_browsers_path_if_bundled(); reloading the module with a fake bundled Chromium present must set PLAYWRIGHT_BROWSERS_PATH=0.""" - import importlib import os monkeypatch.delenv("PLAYWRIGHT_BROWSERS_PATH", raising=False) # Build fake playwright package with a non-empty platform dir diff --git a/tests/test_budget_limits.py b/tests/test_budget_limits.py index 53a8764da..b6dbab7cd 100644 --- a/tests/test_budget_limits.py +++ b/tests/test_budget_limits.py @@ -2,7 +2,6 @@ import os import queue from unittest.mock import MagicMock, patch -import pytest from ouroboros.loop import _check_budget_limits diff --git a/tests/test_budget_tracking.py b/tests/test_budget_tracking.py index 8b67e82ec..f08ccef96 100644 --- a/tests/test_budget_tracking.py +++ b/tests/test_budget_tracking.py @@ -670,8 +670,8 @@ class TestConsolidationCostTracking: ] chat_path.write_text("\n".join(json.dumps(e) for e in entries) + "\n") - blocks_path = tmp_path / "dialogue_blocks.json" - meta_path = tmp_path / "dialogue_meta.json" + tmp_path / "dialogue_blocks.json" + tmp_path / "dialogue_meta.json" class FakeEnv: drive_root = tmp_path diff --git a/tests/test_bughunt_fixes.py b/tests/test_bughunt_fixes.py index e98927dec..535cd3a58 100644 --- a/tests/test_bughunt_fixes.py +++ b/tests/test_bughunt_fixes.py @@ -12,8 +12,6 @@ import json import os import pathlib import sys -import time -from unittest import mock import pytest diff --git a/tests/test_build_scripts.py b/tests/test_build_scripts.py index 8a648ba3a..789a75239 100644 --- a/tests/test_build_scripts.py +++ b/tests/test_build_scripts.py @@ -412,7 +412,7 @@ class TestDockerfile: is downloaded, so the binary can find its runtime libraries on first launch.""" src = _read("Dockerfile") deps_pos = src.find("playwright install-deps chromium webkit") - binary_pos = src.find("playwright install chromium webkit") + src.find("playwright install chromium webkit") # binary_pos must not match the install-deps line itself # find the standalone 'playwright install chromium webkit' (not install-deps) import re as _re diff --git a/tests/test_claude_code_gateway.py b/tests/test_claude_code_gateway.py index df7db79cf..b92a7863a 100644 --- a/tests/test_claude_code_gateway.py +++ b/tests/test_claude_code_gateway.py @@ -14,9 +14,7 @@ We test: import asyncio import json import pathlib -import subprocess import sys -import types import pytest @@ -46,9 +44,6 @@ from ouroboros.gateways.claude_code import ( # noqa: E402 # Orchestration helpers now live in shell.py from ouroboros.tools.shell import ( # noqa: E402 _load_project_context, - _get_changed_files, - _get_diff_stat, - _run_validation, ) @@ -457,7 +452,6 @@ class TestSDKOnlyPath: def test_claude_code_edit_returns_error_when_sdk_missing(self, monkeypatch, tmp_path): """When SDK ImportError → tool returns install hint, not a crash.""" from types import SimpleNamespace - import ouroboros.tools.shell as shell_mod ctx = SimpleNamespace( repo_dir=tmp_path, @@ -604,7 +598,7 @@ class TestRunReadonlyEffortParam: captured.update(kwargs) import asyncio - from unittest.mock import AsyncMock, MagicMock, patch + from unittest.mock import patch class FakeSDKClient: def __init__(self, options=None): @@ -694,7 +688,6 @@ class TestSDKStatusPayload: def test_status_payload_reflects_sdk_installed_with_key(self, monkeypatch): """When SDK is importable and API key set, status is ready.""" - import importlib.metadata from ouroboros.platform_layer import ClaudeRuntimeState def mock_resolve(): diff --git a/tests/test_code_inventory_v1.py b/tests/test_code_inventory_v1.py index f51512e5b..9d71c0d92 100644 --- a/tests/test_code_inventory_v1.py +++ b/tests/test_code_inventory_v1.py @@ -75,7 +75,7 @@ def test_code_inventory_rebuilds_v1_cache(tmp_path): repo.mkdir() (repo / "app.py").write_text("def run():\n return 1\n", encoding="utf-8") - inventory = build_code_inventory(repo, drive_root=data, persist=True) + build_code_inventory(repo, drive_root=data, persist=True) cache_file = next((data / "state" / "code_intel").glob("*/inventory.json")) cached = json.loads(cache_file.read_text(encoding="utf-8")) cached["schema_version"] = 1 diff --git a/tests/test_commit_gate.py b/tests/test_commit_gate.py index e7c45f49e..874eb8956 100644 --- a/tests/test_commit_gate.py +++ b/tests/test_commit_gate.py @@ -397,7 +397,6 @@ def test_advisory_gate_lives_in_shared_reviewed_stage_cycle(): def test_advisory_freshness_blocks_without_fresh_run(tmp_path): """_check_advisory_freshness must return ADVISORY_PRE_REVIEW_REQUIRED if no fresh run.""" - import pathlib git_mod = _get_git_module() class FakeCtx: diff --git a/tests/test_compaction.py b/tests/test_compaction.py index 52a1afd7e..107e88d70 100644 --- a/tests/test_compaction.py +++ b/tests/test_compaction.py @@ -1,5 +1,5 @@ """Tests for tool-history compaction protection (context_compaction.py).""" -from ouroboros.context_compaction import compact_tool_history, _COMPACTION_PROTECTED_TOOLS +from ouroboros.context_compaction import compact_tool_history def _make_messages(tool_name: str, result_content: str, num_rounds: int = 8): diff --git a/tests/test_consolidator.py b/tests/test_consolidator.py index b194d874a..91647e59f 100644 --- a/tests/test_consolidator.py +++ b/tests/test_consolidator.py @@ -10,7 +10,6 @@ from ouroboros.consolidator import ( _load_meta, atomic_write_json as _save_meta, _count_lines, - _read_chat_entries, _format_entries_for_block, _load_blocks, _write_locked_json as _save_blocks, diff --git a/tests/test_context.py b/tests/test_context.py index e48e2e0ca..37357e382 100644 --- a/tests/test_context.py +++ b/tests/test_context.py @@ -3,8 +3,6 @@ from __future__ import annotations import json -import pathlib -import tempfile from ouroboros.context import build_health_invariants, build_runtime_section, build_user_content diff --git a/tests/test_crash_report.py b/tests/test_crash_report.py index a943783ae..b82e780c6 100644 --- a/tests/test_crash_report.py +++ b/tests/test_crash_report.py @@ -4,7 +4,6 @@ Verifies: - crash_report.json is NOT deleted during startup verification - build_health_invariants detects crash_report.json """ -import importlib import inspect import json import os diff --git a/tests/test_deep_self_review.py b/tests/test_deep_self_review.py index 3e4e3b643..3240fd141 100644 --- a/tests/test_deep_self_review.py +++ b/tests/test_deep_self_review.py @@ -9,11 +9,11 @@ from unittest import mock import pytest from ouroboros.deep_self_review import ( - _is_probably_binary, build_review_pack, is_review_available, run_deep_self_review, ) +from ouroboros.tools.review_helpers import _is_probably_binary def _make_dulwich_mock(file_list: list[str]): @@ -304,7 +304,7 @@ class TestIsProbablyBinary: def test_only_first_sniff_bytes_read(self, tmp_path): """_is_probably_binary only reads _BINARY_SNIFF_BYTES bytes, not the whole file.""" - from ouroboros.deep_self_review import _BINARY_SNIFF_BYTES + from ouroboros.tools.review_helpers import _BINARY_SNIFF_BYTES # File is mostly text but has NUL in the first 8KB window payload = b"text data\x00more" + b"a" * (_BINARY_SNIFF_BYTES * 2) f = tmp_path / "big.bin" @@ -641,6 +641,9 @@ class TestReviewPackOverflow: ) assert "too large" in result - assert "920,000" in result + # The gate reserves output headroom inside the window: + # min(SSOT 920K, 1M − 100K output − 155K margin) = 745K. + from ouroboros.deep_self_review import _DEEP_INPUT_TOKEN_LIMIT + assert f"{_DEEP_INPUT_TOKEN_LIMIT:,}" in result assert usage == {} mock_llm.chat.assert_not_called() diff --git a/tests/test_event_bus.py b/tests/test_event_bus.py index 33efd2b38..ed8341b77 100644 --- a/tests/test_event_bus.py +++ b/tests/test_event_bus.py @@ -1,4 +1,4 @@ -from ouroboros.event_bus import CHAT_OUTBOUND, CHAT_VIDEO, SKILL_LIFECYCLE, EventBus +from ouroboros.event_bus import CHAT_OUTBOUND, SKILL_LIFECYCLE, EventBus import base64 import asyncio import json diff --git a/tests/test_evolution_redesign.py b/tests/test_evolution_redesign.py index 19d1b6ec8..0526ccec1 100644 --- a/tests/test_evolution_redesign.py +++ b/tests/test_evolution_redesign.py @@ -1,6 +1,5 @@ from __future__ import annotations -import pathlib import json from types import SimpleNamespace diff --git a/tests/test_evolution_stop_and_cost.py b/tests/test_evolution_stop_and_cost.py index 88984e47e..8d77a1ac6 100644 --- a/tests/test_evolution_stop_and_cost.py +++ b/tests/test_evolution_stop_and_cost.py @@ -12,7 +12,6 @@ Covers: import json from types import SimpleNamespace -import pytest def _write_events(tmp_path, rows): diff --git a/tests/test_extension_isolated_deps.py b/tests/test_extension_isolated_deps.py index 7988fad60..ba29233f1 100644 --- a/tests/test_extension_isolated_deps.py +++ b/tests/test_extension_isolated_deps.py @@ -2,40 +2,23 @@ from __future__ import annotations import asyncio -import base64 -import json import pathlib -import queue -import re import sys import threading -import time -from types import SimpleNamespace -from typing import Any, Dict import pytest from ouroboros import extension_loader -from ouroboros.contracts.plugin_api import ( - ExtensionRegistrationError, - FORBIDDEN_EXTENSION_SETTINGS, - PluginAPI, - VALID_EXTENSION_PERMISSIONS, -) from ouroboros.skill_loader import ( SkillReviewState, - compute_content_hash, find_skill, save_enabled, save_review_state, ) from tests._shared import clean_extension_runtime_state from tests.test_extension_loader import ( - _add_fake_native_dep, - _isolated_site_packages_dir, _mark_isolated_deps_installed, _prepare_extension, - _write_ext_skill, ) @@ -131,7 +114,6 @@ def test_load_extension_ignores_isolated_env_symlinks_when_staging(tmp_path): def test_load_extension_does_not_import_untracked_isolated_python_deps(tmp_path): - import sys loaded, repo_root, drive_root = _prepare_extension( tmp_path, @@ -164,7 +146,6 @@ def test_load_extension_does_not_import_untracked_isolated_python_deps(tmp_path) def test_load_extension_imports_and_unloads_isolated_python_deps(tmp_path): import importlib - import sys loaded, repo_root, drive_root = _prepare_extension( tmp_path, @@ -208,7 +189,6 @@ def test_load_extension_imports_and_unloads_isolated_python_deps(tmp_path): def test_load_extension_does_not_execute_isolated_deps_pth_files(tmp_path): - import sys marker = tmp_path / "pth_executed.txt" loaded, repo_root, drive_root = _prepare_extension( @@ -248,7 +228,6 @@ def test_load_extension_does_not_execute_isolated_deps_pth_files(tmp_path): def test_isolated_python_deps_do_not_leak_to_other_extensions(tmp_path): - import sys loaded_a, repo_root, drive_root = _prepare_extension( tmp_path, @@ -299,7 +278,6 @@ def test_isolated_python_deps_do_not_leak_to_other_extensions(tmp_path): def test_isolated_namespace_packages_are_purged_after_import_scope(tmp_path): - import sys loaded, repo_root, drive_root = _prepare_extension( tmp_path, @@ -335,7 +313,6 @@ def test_isolated_namespace_packages_are_purged_after_import_scope(tmp_path): def test_isolated_regular_parent_namespace_child_is_purged_after_import_scope(tmp_path): - import sys loaded, repo_root, drive_root = _prepare_extension( tmp_path, @@ -374,7 +351,6 @@ def test_isolated_regular_parent_namespace_child_is_purged_after_import_scope(tm def test_isolated_regular_parent_namespace_child_is_purged_after_async_handler(tmp_path): - import asyncio import sys loaded, repo_root, drive_root = _prepare_extension( @@ -436,7 +412,6 @@ def test_isolated_site_scope_releases_execution_lock_when_cleanup_fails(tmp_path def test_async_isolated_site_scope_releases_execution_lock_when_cleanup_fails(tmp_path, monkeypatch): - import asyncio from ouroboros import extension_isolated_deps @@ -472,7 +447,6 @@ def test_isolated_site_scope_releases_execution_lock_when_inject_fails(tmp_path, def test_async_isolated_site_scope_releases_execution_lock_when_inject_fails(tmp_path, monkeypatch): - import asyncio from ouroboros import extension_isolated_deps @@ -493,7 +467,6 @@ def test_async_isolated_site_scope_releases_execution_lock_when_inject_fails(tmp def test_async_isolated_site_scope_cancel_while_waiting_does_not_wedge_lock(tmp_path): - import asyncio from ouroboros import extension_isolated_deps @@ -517,7 +490,6 @@ def test_async_isolated_site_scope_cancel_while_waiting_does_not_wedge_lock(tmp_ def test_release_isolated_site_dirs_removes_path_when_module_scan_fails(tmp_path, monkeypatch): - import sys import types from ouroboros import extension_isolated_deps @@ -556,7 +528,6 @@ def test_release_isolated_site_dirs_removes_path_when_module_scan_fails(tmp_path def test_release_isolated_site_dirs_removes_preexisting_env_parent_path(tmp_path): - import sys from ouroboros import extension_isolated_deps @@ -587,7 +558,6 @@ def test_release_isolated_site_dirs_removes_preexisting_env_parent_path(tmp_path def test_inject_isolated_site_dirs_tracks_preexisting_env_path(tmp_path): - import sys from ouroboros import extension_isolated_deps @@ -617,8 +587,6 @@ def test_inject_isolated_site_dirs_tracks_preexisting_env_path(tmp_path): def test_isolated_python_deps_do_not_leak_during_overlapping_handlers(tmp_path): - import sys - import threading started = threading.Event() release = threading.Event() @@ -692,7 +660,6 @@ def test_isolated_python_deps_do_not_leak_during_overlapping_handlers(tmp_path): def test_isolated_python_deps_do_not_leak_during_overlapping_async_handlers(tmp_path): - import asyncio import sys loaded_a, repo_root, drive_root = _prepare_extension( diff --git a/tests/test_extension_loader.py b/tests/test_extension_loader.py index 49cd79f1f..aabba0a21 100644 --- a/tests/test_extension_loader.py +++ b/tests/test_extension_loader.py @@ -7,7 +7,6 @@ namespace enforcement + unload cleanup. from __future__ import annotations import json -import base64 import pathlib import re import sys @@ -24,14 +23,12 @@ from ouroboros.extension_reconcile_queue import ( request_extension_reconcile, ) from ouroboros.contracts.plugin_api import ( - ExtensionRegistrationError, FORBIDDEN_EXTENSION_SETTINGS, PluginAPI, VALID_EXTENSION_PERMISSIONS, ) from ouroboros.skill_loader import ( SkillReviewState, - compute_content_hash, find_skill, save_enabled, save_review_state, @@ -99,7 +96,7 @@ def _prepare_extension( repo_root = tmp_path / "skills" drive_root = tmp_path / "drive" drive_root.mkdir(exist_ok=True) - skill_dir = _write_ext_skill( + _write_ext_skill( repo_root, name, plugin_body=plugin_body, diff --git a/tests/test_extension_process_runner.py b/tests/test_extension_process_runner.py index 78c04378f..fede0722e 100644 --- a/tests/test_extension_process_runner.py +++ b/tests/test_extension_process_runner.py @@ -3,32 +3,20 @@ from __future__ import annotations import asyncio import base64 -import json import pathlib -import queue -import re import sys -import threading import time from types import SimpleNamespace -from typing import Any, Dict import pytest from ouroboros import extension_loader from ouroboros.contracts.plugin_api import ( ExtensionRegistrationError, - FORBIDDEN_EXTENSION_SETTINGS, - PluginAPI, - VALID_EXTENSION_PERMISSIONS, ) from ouroboros.skill_loader import ( - SkillReviewState, - compute_content_hash, find_skill, - save_enabled, save_skill_grants, - save_review_state, ) from tests._shared import clean_extension_runtime_state from tests.test_extension_loader import ( @@ -36,7 +24,6 @@ from tests.test_extension_loader import ( _isolated_site_packages_dir, _mark_isolated_deps_installed, _prepare_extension, - _write_ext_skill, ) @@ -341,7 +328,6 @@ def test_native_risk_extension_streaming_route_is_materialized_out_of_process(tm def test_native_risk_extension_gateway_route_child_failure_returns_502(tmp_path, monkeypatch): - import asyncio from types import SimpleNamespace from starlette.requests import Request from ouroboros.gateway.extensions import api_extension_dispatch @@ -397,7 +383,6 @@ def test_native_risk_extension_gateway_route_child_failure_returns_502(tmp_path, def test_native_risk_extension_gateway_route_rejects_oversized_body_before_child(tmp_path, monkeypatch): - import asyncio from types import SimpleNamespace from starlette.requests import Request from ouroboros.gateway.extensions import _CHILD_DISPATCH_BODY_CAP, api_extension_dispatch @@ -552,7 +537,6 @@ def test_isolated_dependency_extension_oversized_child_result_is_capped(tmp_path def test_parallel_isolated_extension_children_do_not_sweep_each_other_import_tree(tmp_path): import concurrent.futures - import time from ouroboros.tools.registry import ToolRegistry plugin = ( @@ -779,7 +763,6 @@ def test_native_risk_extension_ws_dispatches_out_of_process(tmp_path): def test_native_risk_extension_gateway_ws_child_failure_is_log_message(tmp_path, monkeypatch): import json as _json - from types import SimpleNamespace from ouroboros.gateway.ws import _dispatch_extension_message plugin = ( diff --git a/tests/test_extension_surfaces.py b/tests/test_extension_surfaces.py index 51fbaa308..3892ad8ee 100644 --- a/tests/test_extension_surfaces.py +++ b/tests/test_extension_surfaces.py @@ -1,27 +1,10 @@ """Split extension-loader regression coverage kept below module size gates.""" from __future__ import annotations -import asyncio -import base64 -import json -import pathlib -import queue -import re -import sys -import threading -import time -from types import SimpleNamespace -from typing import Any, Dict import pytest from ouroboros import extension_loader -from ouroboros.contracts.plugin_api import ( - ExtensionRegistrationError, - FORBIDDEN_EXTENSION_SETTINGS, - PluginAPI, - VALID_EXTENSION_PERMISSIONS, -) from ouroboros.skill_loader import ( SkillReviewState, compute_content_hash, @@ -31,9 +14,6 @@ from ouroboros.skill_loader import ( ) from tests._shared import clean_extension_runtime_state from tests.test_extension_loader import ( - _add_fake_native_dep, - _isolated_site_packages_dir, - _mark_isolated_deps_installed, _prepare_extension, _write_ext_skill, ) diff --git a/tests/test_extensions_api.py b/tests/test_extensions_api.py index cf4d7d786..0e6ad8749 100644 --- a/tests/test_extensions_api.py +++ b/tests/test_extensions_api.py @@ -111,7 +111,11 @@ def _patch_lifespan_for_drive_root_test(monkeypatch, srv, settings: dict): monkeypatch.setattr(srv, "apply_runtime_provider_defaults", lambda s: (s, False, [])) monkeypatch.setattr(srv, "_apply_settings_to_env", lambda *_a, **_k: None) monkeypatch.setattr(srv, "has_startup_ready_provider", lambda *_a, **_k: False) - monkeypatch.setattr(srv, "has_local_routing", lambda *_a, **_k: False) + # has_local_routing now lives only in server_runtime (server.py stopped + # importing it after the provider-check consolidation). + monkeypatch.setattr( + "ouroboros.server_runtime.has_local_routing", lambda *_a, **_k: False + ) monkeypatch.setattr(srv, "_start_supervisor_if_needed", lambda *_a, **_k: None) monkeypatch.setattr(srv.uvicorn, "Server", _FakeUvicornServer) monkeypatch.setattr("ouroboros.launcher_bootstrap.ensure_data_skills_seeded", lambda: None) @@ -340,7 +344,7 @@ def test_api_extensions_index_marks_widget_only_extensions_as_ui_pending( def test_api_skill_toggle_enables_and_loads_extension(tmp_path, monkeypatch): from ouroboros import extension_loader - from ouroboros.skill_loader import SkillReviewState, save_review_state, find_skill + from ouroboros.skill_loader import SkillReviewState, save_review_state from ouroboros.skill_loader import compute_content_hash skills_root = tmp_path / "skills" diff --git a/tests/test_file_browser_api.py b/tests/test_file_browser_api.py index 5425562d9..757c01112 100644 --- a/tests/test_file_browser_api.py +++ b/tests/test_file_browser_api.py @@ -30,7 +30,7 @@ def test_network_requests_require_explicit_file_root(monkeypatch: pytest.MonkeyP assert "OUROBOROS_FILE_BROWSER_DEFAULT" in response.json()["error"] -def test_external_symlink_file_is_visible_readable_writable_and_deletable( +def test_external_symlink_file_is_listed_but_not_accessible( monkeypatch: pytest.MonkeyPatch, tmp_path: pathlib.Path, ): @@ -55,24 +55,27 @@ def test_external_symlink_file_is_visible_readable_writable_and_deletable( assert "escape.txt" in names assert entries["escape.txt"]["is_symlink"] is True + # v6.26.0 containment contract: a symlink RESOLVING outside the root is + # listed (the link itself lives in-root) but cannot be read, written, + # or deleted through the API — that was a root escape. read_response = client.get("/api/files/read?path=escape.txt") - assert read_response.status_code == 200 - assert read_response.json()["content"] == "secret" + assert read_response.status_code == 400 + assert "escapes file browser root" in read_response.json()["error"] write_response = client.post( "/api/files/write", json={"path": "escape.txt", "content": "changed"}, ) - assert write_response.status_code == 200 - assert outside.read_text(encoding="utf-8") == "changed" + assert write_response.status_code == 400 + assert outside.read_text(encoding="utf-8") == "secret" delete_response = client.post("/api/files/delete", json={"path": "escape.txt"}) - assert delete_response.status_code == 200 - assert not escape.exists() + assert delete_response.status_code == 400 + assert escape.is_symlink() assert outside.exists() -def test_external_symlink_dir_allows_mkdir_upload_and_copy( +def test_external_symlink_dir_mutations_are_blocked( monkeypatch: pytest.MonkeyPatch, tmp_path: pathlib.Path, ): @@ -98,12 +101,14 @@ def test_external_symlink_dir_allows_mkdir_upload_and_copy( pytest.skip("Symlink creation is not available in this test environment.") with _make_client(root, monkeypatch) as client: + # v6.26.0 containment contract: mutations THROUGH an out-of-root + # symlink are blocked (they were a root escape). mkdir_response = client.post( "/api/files/mkdir", json={"path": "external-dir", "name": "child"}, ) - assert mkdir_response.status_code == 200 - assert (outside_dir / "child").is_dir() + assert mkdir_response.status_code == 400 + assert not (outside_dir / "child").exists() copy_response = client.post( "/api/files/transfer", @@ -113,10 +118,8 @@ def test_external_symlink_dir_allows_mkdir_upload_and_copy( "mode": "copy", }, ) - assert copy_response.status_code == 200 - copied = copy_dir / "external-file.txt" - assert copied.is_symlink() - assert copied.resolve() == outside_file.resolve() + assert copy_response.status_code == 400 + assert not (copy_dir / "external-file.txt").exists() class _Client: host = "testclient" @@ -131,8 +134,8 @@ def test_external_symlink_dir_allows_mkdir_upload_and_copy( } upload_response = asyncio.run(file_browser_api.api_files_upload(_UploadRequest())) - assert upload_response.status_code == 200 - assert (outside_dir / "uploaded.txt").read_bytes() == b"payload" + assert upload_response.status_code == 400 + assert not (outside_dir / "uploaded.txt").exists() def test_root_delete_is_rejected_and_image_url_is_encoded( diff --git a/tests/test_git_review_pipeline.py b/tests/test_git_review_pipeline.py index 942bc2a44..27444b516 100644 --- a/tests/test_git_review_pipeline.py +++ b/tests/test_git_review_pipeline.py @@ -24,7 +24,6 @@ import os import pathlib import subprocess import sys -import tempfile import pytest @@ -416,7 +415,9 @@ class TestReviewEnforcementModes: or (isinstance(w, str) and "broken" in w) for w in ctx._review_advisory ) - assert ctx._review_iteration_count == 0 + # Anti-thrashing state survives an advisory pass-through of critical + # findings: repeats on the next attempt must still be recognized. + assert ctx._review_iteration_count == 1 def test_advisory_mode_downgrades_quorum_failure(self, review_ctx, monkeypatch): review, ctx = review_ctx @@ -994,7 +995,6 @@ class TestAdvisorySkipTests: def test_skip_tests_true_bypasses_test_gate(self, tmp_path, monkeypatch): """skip_tests=True skips the test gate and reaches the SDK call.""" - import json as _json from ouroboros.tools import claude_advisory_review as adv monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-ant-fake") @@ -1026,7 +1026,6 @@ class TestAdvisorySkipTests: def test_passing_tests_proceed_to_sdk(self, tmp_path, monkeypatch): """When tests pass, advisory continues to the SDK call.""" - import json as _json from ouroboros.tools import claude_advisory_review as adv monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-ant-fake") @@ -1420,7 +1419,6 @@ class TestBypassPathTestsRun: must still run in _run_reviewed_stage_cycle even though skip_advisory_pre_review is False. This covers the missing-key auto-bypass path documented in the bypass gate condition: `skip_advisory_pre_review or not os.environ.get("ANTHROPIC_API_KEY", "")`""" - import os from ouroboros.tools import git as git_mod ctx = self._make_staged_repo(tmp_path) diff --git a/tests/test_immune_hardening.py b/tests/test_immune_hardening.py new file mode 100644 index 000000000..5e4cd287d --- /dev/null +++ b/tests/test_immune_hardening.py @@ -0,0 +1,191 @@ +"""Immune-system hardening invariants (WS-E, v6.26.0). + +Covers: triad anti-refusal coverage contract, durable advisory-override +visibility, dispatcher-level worktree-mutation invalidation, and the +`find` removal from the SAFE shell whitelist. +""" + +import json +import pathlib + +from ouroboros.triad_review import parse_model_review_results + + +def _actor(text: str, model: str = "m1") -> dict: + return {"model": model, "text": text, "tokens_in": 1, "tokens_out": 1} + + +class TestTriadRefusalContract: + def test_bare_empty_array_only_response_is_responded(self): + parsed = parse_model_review_results({"results": [_actor("[]")]}) + assert parsed.actor_records[0].status == "responded" + + def test_fenced_empty_array_only_response_is_responded(self): + parsed = parse_model_review_results({"results": [_actor("```json\n[]\n```")]}) + assert parsed.actor_records[0].status == "responded" + + def test_empty_array_with_sentinel_is_responded(self): + parsed = parse_model_review_results( + {"results": [_actor("Everything checks out.\n[]\nNO_FINDINGS")]} + ) + assert parsed.actor_records[0].status == "responded" + + def test_empty_array_inside_refusal_prose_is_parse_failure(self): + refusal = ( + "I'm sorry, but I cannot review this diff because it is too large " + "and may contain content I cannot process. [] Please try again." + ) + parsed = parse_model_review_results({"results": [_actor(refusal)]}) + assert parsed.actor_records[0].status == "parse_failure" + assert not parsed.quorum_met + + def test_refusal_actor_does_not_count_toward_quorum(self): + good = '[{"item":"code_quality","verdict":"PASS","severity":"advisory","reason":"ok"}]' + refusal = "I cannot comply with this request. []" + parsed = parse_model_review_results( + {"results": [_actor(good, "m1"), _actor(refusal, "m2"), _actor(refusal, "m3")]} + ) + assert len(parsed.responsive_models) == 1 + assert not parsed.quorum_met + + def test_non_empty_findings_array_still_responded(self): + text = '[{"item":"security","verdict":"FAIL","severity":"critical","reason":"bad"}]' + parsed = parse_model_review_results({"results": [_actor(text)]}) + assert parsed.actor_records[0].status == "responded" + assert parsed.findings and parsed.findings[0]["verdict"] == "FAIL" + + +class TestAdvisoryOverrideVisibility: + def test_override_leaves_durable_trace(self, tmp_path, monkeypatch): + from ouroboros.tools import review as review_mod + + class Ctx: + drive_root = tmp_path + task_id = "t1" + _last_review_block_reason = "critical_findings" + + def drive_logs(self): + logs = tmp_path / "logs" + logs.mkdir(parents=True, exist_ok=True) + return logs + + review_mod._record_advisory_override(Ctx(), "⚠️ REVIEW_BLOCKED: example") + + events = (tmp_path / "logs" / "events.jsonl").read_text(encoding="utf-8") + assert "review_advisory_override" in events + + overrides = json.loads((tmp_path / "state" / "advisory_overrides.json").read_text()) + assert overrides["count"] == 1 + assert overrides["recent"][0]["block_reason"] == "critical_findings" + + def test_review_status_payload_surfaces_overrides(self): + from ouroboros.review_evidence import build_review_status_payload + + projection = { + "effective_status": "none", + "effective_hash": None, + "stale_from_edit": False, + "stale_from_edit_ts": None, + "stale_reason": None, + "filters": {}, + "runs": [], + "attempts": [], + "selected_attempt": None, + "open_obligations": [], + "open_debts": [], + "repo_commit_ready": False, + "retry_anchor": None, + "advisory_overrides": {"count": 3, "recent": [{"ts": "t", "block_reason": "x"}]}, + "guidance_run": None, + "state": None, + "current_hash": "", + "effective_is_fresh": False, + "latest_run": None, + "matching_run": None, + } + payload = build_review_status_payload(projection, next_step="n/a") + assert payload["advisory_overrides_count"] == 3 + assert payload["advisory_overrides_recent"] + + +class TestCentralWorktreeInvalidation: + def test_mutating_tool_invalidates_only_on_real_change(self, tmp_path, monkeypatch): + from ouroboros.tools.registry import ToolRegistry + + registry = ToolRegistry.__new__(ToolRegistry) + registry._entries = {} + registry._capability_omissions = [] + + class _Ctx: + repo_dir = tmp_path + drive_root = tmp_path / "data" + + registry._ctx = _Ctx() + + calls = {"invalidate": 0} + # Call 1 sees " M a.py" (differs from before="") → invalidate; + # call 2 sees " M a.py" (equals before) → no invalidate. + snapshots = iter([" M a.py", " M a.py"]) + monkeypatch.setattr( + ToolRegistry, "_worktree_status_snapshot", lambda self: next(snapshots) + ) + + def _fake_invalidate(drive_root, **kwargs): + calls["invalidate"] += 1 + + import ouroboros.review_state as review_state + + monkeypatch.setattr( + review_state, "invalidate_advisory_after_mutation", _fake_invalidate + ) + + # Change observed (before="", after=" M a.py") → invalidate. + registry._invalidate_advisory_if_worktree_changed("run_command", "") + # No change (before==after) → no invalidate. + registry._invalidate_advisory_if_worktree_changed("run_command", " M a.py") + assert calls["invalidate"] == 1 + + def test_mutating_tools_are_flagged(self): + from ouroboros.tools import git_pr, services, shell + + flagged = { + entry.name + for mod in (shell, services, git_pr) + for entry in mod.get_tools() + if entry.mutates_worktree + } + for name in ("run_command", "run_script", "claude_code_edit", + "start_service", "stop_service", "cherry_pick_pr_commits", + "stage_pr_merge", "stage_adaptations"): + assert name in flagged, f"{name} must be mutates_worktree-flagged" + + +class TestFindNotSafe: + def test_find_routes_through_llm_safety(self): + from ouroboros.safety import SAFE_SHELL_COMMANDS + + assert "find" not in SAFE_SHELL_COMMANDS + # Read-only staples stay whitelisted. + assert "grep" in SAFE_SHELL_COMMANDS + assert "ls" in SAFE_SHELL_COMMANDS + + +class TestScopeChecklistFailClosed: + def test_missing_checklist_raises(self, monkeypatch): + import pytest + + import ouroboros.tools.scope_review as scope_review + + monkeypatch.setattr(scope_review, "load_checklist_section", lambda *_a, **_k: "") + with pytest.raises(RuntimeError, match="fail-closed"): + scope_review._build_scope_prompt(pathlib.Path("."), "msg") + + +class TestStatusSetSSOT: + def test_headless_final_statuses_mirror_settled_ssot(self): + # headless cannot import task_status at module level (import cycle); + # this pin keeps its mirrored literal equal to the SSOT. + from ouroboros.headless import _FINAL_STATUSES + from ouroboros.task_status import SETTLED_STATUSES + + assert _FINAL_STATUSES == SETTLED_STATUSES diff --git a/tests/test_isolated_deps.py b/tests/test_isolated_deps.py index fdc1987f2..66a8bf3b0 100644 --- a/tests/test_isolated_deps.py +++ b/tests/test_isolated_deps.py @@ -2,7 +2,6 @@ from __future__ import annotations import json import os -import pathlib import subprocess from ouroboros.marketplace.install_specs import normalize_install_specs diff --git a/tests/test_llm_no_proxy.py b/tests/test_llm_no_proxy.py index c1e087289..cb1e4cade 100644 --- a/tests/test_llm_no_proxy.py +++ b/tests/test_llm_no_proxy.py @@ -11,10 +11,8 @@ Covers: from __future__ import annotations import asyncio -import json -from unittest.mock import AsyncMock, MagicMock, patch, call +from unittest.mock import AsyncMock, MagicMock, patch -import pytest # --------------------------------------------------------------------------- @@ -41,6 +39,7 @@ def test_chat_anthropic_no_proxy_uses_session_trust_env_false(): fake_response = MagicMock() fake_response.raise_for_status = MagicMock() + fake_response.status_code = 200 fake_response.json.return_value = { "content": [{"type": "text", "text": "Hi"}], "usage": {"input_tokens": 10, "output_tokens": 5}, @@ -51,9 +50,7 @@ def test_chat_anthropic_no_proxy_uses_session_trust_env_false(): captured_session_trust_env = [] - import requests as _requests - original_session = _requests.Session class FakeSession: def __init__(self): @@ -100,6 +97,7 @@ def test_chat_anthropic_no_proxy_false_uses_requests_post(): ok_response = MagicMock() ok_response.raise_for_status = MagicMock() + ok_response.status_code = 200 ok_response.json.return_value = { "content": [{"type": "text", "text": "Hi"}], "usage": {"input_tokens": 10, "output_tokens": 5}, @@ -107,9 +105,10 @@ def test_chat_anthropic_no_proxy_false_uses_requests_post(): "role": "assistant", } rejected_response = MagicMock() - rejected_response.raise_for_status.side_effect = RuntimeError( - "temperature is not supported for this model" - ) + rejected_response.status_code = 400 + rejected_response.reason = "Bad Request" + rejected_response.url = "https://api.anthropic.com/v1/messages" + rejected_response.text = "temperature is not supported for this model" session_called = [] @@ -163,6 +162,7 @@ def test_chat_anthropic_honors_explicit_timeout(): } fake_response = MagicMock() fake_response.raise_for_status = MagicMock() + fake_response.status_code = 200 fake_response.json.return_value = { "content": [{"type": "text", "text": "Hi"}], "usage": {"input_tokens": 10, "output_tokens": 5}, @@ -195,12 +195,12 @@ def test_chat_async_no_proxy_anthropic_path(): captured_no_proxy = [] - def fake_chat_anthropic(target, messages, tools, effort, max_tokens, tool_choice, temp, np=False): + def fake_chat_anthropic(target, messages, tools, effort, max_tokens, tool_choice, temp, np=False, timeout=None): captured_no_proxy.append(np) return {"role": "assistant", "content": "Hi"}, {"prompt_tokens": 10, "completion_tokens": 5} with patch.object(client, "_chat_anthropic", side_effect=fake_chat_anthropic): - result = asyncio.run( + asyncio.run( client.chat_async(messages=messages, model=model, no_proxy=True) ) @@ -225,7 +225,6 @@ def test_chat_async_no_proxy_non_anthropic_uses_httpx_async_client(): captured_httpx_kwargs = [] - import httpx as _httpx class FakeAsyncClient: def __init__(self, **kwargs): @@ -447,7 +446,6 @@ def test_scope_review_call_scope_llm_uses_no_proxy(): no_proxy=True. We test the asyncio.run() fallback path (RuntimeError branch) by ensuring no running loop is active during the call. """ - import asyncio from ouroboros.tools import scope_review captured_kwargs = [] diff --git a/tests/test_llm_provider_routing.py b/tests/test_llm_provider_routing.py index 96572c87b..c29dfe49e 100644 --- a/tests/test_llm_provider_routing.py +++ b/tests/test_llm_provider_routing.py @@ -475,6 +475,7 @@ def test_chat_anthropic_sends_tool_cache_control_without_ttl(monkeypatch): captured = {} fake_response = SimpleNamespace( raise_for_status=lambda: None, + status_code=200, json=lambda: {"content": [{"type": "text", "text": "ok"}], "usage": {}}, ) monkeypatch.setattr( diff --git a/tests/test_llm_supported_parameters.py b/tests/test_llm_supported_parameters.py index 4c8a0962a..78cdc4aa8 100644 --- a/tests/test_llm_supported_parameters.py +++ b/tests/test_llm_supported_parameters.py @@ -48,7 +48,6 @@ def _install_fake_response(monkeypatch, data: dict[str, Any]) -> dict[str, int]: call_count["n"] += 1 return _Resp() - import ouroboros.llm as llm_mod # _fetch_openrouter_capabilities does `import requests` lazily inside the function, # so we patch the module attribute that it will see after the lazy import. import requests as _real_requests diff --git a/tests/test_loop_compaction_policy.py b/tests/test_loop_compaction_policy.py index ad06aa178..edd33b70d 100644 --- a/tests/test_loop_compaction_policy.py +++ b/tests/test_loop_compaction_policy.py @@ -30,7 +30,7 @@ import json import queue import types import unittest -from unittest.mock import MagicMock, patch, call as mock_call +from unittest.mock import MagicMock, patch # --------------------------------------------------------------------------- @@ -194,14 +194,14 @@ class TestEstimateMessagesChars(unittest.TestCase): self.assertEqual(_emc(msgs), expected) def test_multipart_image_url_block_counted(self): - """Non-text multipart blocks (image_url) must be fully counted.""" + """image_url blocks count as the fixed vision-token equivalent (v6.26.0).""" + from ouroboros.context_budget import IMAGE_BLOCK_CHAR_EQUIVALENT + block = {"type": "image_url", "image_url": {"url": "data:image/png;base64," + "A" * 100}} msgs = [_msg("user", [block])] result = _emc(msgs) - # Must be substantially more than just len("") = 0 - self.assertGreater(result, 100, - "image_url block must be counted in size estimate") - self.assertEqual(result, len(json.dumps(block, ensure_ascii=False))) + self.assertGreater(result, 100, "image_url block must be counted in size estimate") + self.assertEqual(result, IMAGE_BLOCK_CHAR_EQUIVALENT) def test_tool_calls_counted(self): tc = [{"id": "t1", "function": {"name": "foo", "arguments": "{}"}}] @@ -265,19 +265,23 @@ class TestCompactionPolicyRemote(unittest.TestCase): ) def test_emergency_counts_image_url_blocks(self): - """image_url multipart blocks must be counted by the emergency guard.""" + """image_url blocks count as a FIXED token equivalent (v6.26.0): vision + models bill per tile, so one image must never look like ~300K tokens + and permanently wedge emergency compaction.""" + from ouroboros.context_budget import IMAGE_BLOCK_CHAR_EQUIVALENT + big_image_block = { "type": "image_url", "image_url": {"url": "data:image/png;base64," + "A" * 1_300_000}, } messages = [{"role": "user", "content": [big_image_block]}] size = _emc(messages) - self.assertGreater(size, 1_200_000, - "_estimate_messages_chars must count image_url blocks in size") + self.assertEqual(size, IMAGE_BLOCK_CHAR_EQUIVALENT, + "an image block must count as the fixed equivalent, not base64 length") calls = _run_loop(messages, use_local=False, rounds_before_stop=1) - self.assertTrue( + self.assertFalse( any(c["keep_recent"] == 50 for c in calls), - "Emergency compaction must fire when image_url content pushes size over threshold", + "ONE image must NOT trigger emergency compaction anymore", ) def test_emergency_compaction_skips_when_checkpoint_persist_fails(self): diff --git a/tests/test_loop_misc.py b/tests/test_loop_misc.py index e44be2efe..1c92038c5 100644 --- a/tests/test_loop_misc.py +++ b/tests/test_loop_misc.py @@ -338,6 +338,9 @@ def test_run_llm_loop_enforces_consilium_force_plan_before_final(tmp_path, monke "args": {}, "result": "## Plan Review Results\n\nAGGREGATE: GREEN", "is_error": False, + # v6.26.0: the force-plan gate reads this structured flag (captured + # from the FULL tool result), not a substring of the 700-char preview. + "plan_review_aggregate": True, }) request_messages.append({"role": "tool", "tool_call_id": tool_calls[0]["id"], "content": "## Plan Review Results\n\nAGGREGATE: GREEN"}) return 0 diff --git a/tests/test_marketplace_fetcher.py b/tests/test_marketplace_fetcher.py index c9f3d9b32..4e3118f30 100644 --- a/tests/test_marketplace_fetcher.py +++ b/tests/test_marketplace_fetcher.py @@ -305,7 +305,6 @@ def test_stage_rejects_zip_bomb_with_falsified_file_size(): per-file cap). The fix uses ``src.read(cap+1)`` which bounds peak memory at cap+1 regardless of the forged header. """ - import zlib payload = b"x" * (9 * 1024 * 1024) # 9 MB > 8 MB per-file cap buf = io.BytesIO() diff --git a/tests/test_marketplace_provenance_contract.py b/tests/test_marketplace_provenance_contract.py index 54082d640..49bbaec4f 100644 --- a/tests/test_marketplace_provenance_contract.py +++ b/tests/test_marketplace_provenance_contract.py @@ -16,7 +16,6 @@ import asyncio import pathlib from unittest import mock -import pytest # The minimal canonical provenance record any v5 marketplace install diff --git a/tests/test_max_tokens_constants.py b/tests/test_max_tokens_constants.py index ed5e4779e..19dfd1d55 100644 --- a/tests/test_max_tokens_constants.py +++ b/tests/test_max_tokens_constants.py @@ -121,9 +121,22 @@ def test_review_prompt_token_budget_is_ssot(): f"_SCOPE_BUDGET_TOKEN_LIMIT ({_SCOPE_BUDGET_TOKEN_LIMIT}) must equal " f"the SSOT REVIEW_PROMPT_TOKEN_BUDGET ({REVIEW_PROMPT_TOKEN_BUDGET})." ) - assert _PLAN_BUDGET_TOKEN_LIMIT == REVIEW_PROMPT_TOKEN_BUDGET, ( - f"_PLAN_BUDGET_TOKEN_LIMIT ({_PLAN_BUDGET_TOKEN_LIMIT}) must equal " - f"the SSOT REVIEW_PROMPT_TOKEN_BUDGET ({REVIEW_PROMPT_TOKEN_BUDGET})." + # Plan review reserves output headroom inside the reviewer window (same + # class of fix as scope review): the limit is min(SSOT, window − output − + # margin) and must never exceed the SSOT. + from ouroboros.tools.plan_review import ( + _PLAN_MODEL_CONTEXT_WINDOW, + _PLAN_OUTPUT_MARGIN_TOKENS, + _PLAN_REVIEW_MAX_TOKENS, + ) + assert _PLAN_BUDGET_TOKEN_LIMIT == min( + REVIEW_PROMPT_TOKEN_BUDGET, + _PLAN_MODEL_CONTEXT_WINDOW - _PLAN_REVIEW_MAX_TOKENS - _PLAN_OUTPUT_MARGIN_TOKENS, + ) + assert _PLAN_BUDGET_TOKEN_LIMIT <= REVIEW_PROMPT_TOKEN_BUDGET + assert _PLAN_BUDGET_TOKEN_LIMIT + _PLAN_REVIEW_MAX_TOKENS <= _PLAN_MODEL_CONTEXT_WINDOW, ( + "plan input cap + reserved output exceeds the reviewer window; " + "the provider would hard-400." ) @@ -162,18 +175,18 @@ def test_scope_input_budget_reserves_output_within_window(): def test_deep_self_review_budget_uses_ssot(): - """``deep_self_review`` must gate on the SSOT constant (not a hardcoded - literal) and must gate on the FULL assembled prompt (system + user) - using the shared ``estimate_tokens(chars/4)`` helper, matching - scope_review and plan_review. + """``deep_self_review`` must gate the FULL assembled prompt (system + user) + on an input limit derived from the SSOT constant WITH output reservation + (min(SSOT, window − output − margin)) — matching scope_review/plan_review — + using the shared ``estimate_tokens(chars/4)`` helper. """ import pathlib src = pathlib.Path("ouroboros/deep_self_review.py").read_text(encoding="utf-8") assert "REVIEW_PROMPT_TOKEN_BUDGET" in src, ( - "deep_self_review must import the SSOT constant from review_helpers" + "deep_self_review must derive its gate from the SSOT constant" ) - assert "estimated_tokens > REVIEW_PROMPT_TOKEN_BUDGET" in src, ( - "deep_self_review must compare against the SSOT constant, not a literal" + assert "estimated_tokens > _DEEP_INPUT_TOKEN_LIMIT" in src, ( + "deep_self_review must gate on the output-reserving input limit" ) assert "estimate_tokens(_SYSTEM_PROMPT + pack_text)" in src, ( "deep_self_review must gate on the FULL assembled prompt " @@ -190,10 +203,26 @@ def test_deep_self_review_budget_uses_ssot(): "deep_self_review must not use its old chars/3.5 estimator" ) + from ouroboros.deep_self_review import ( + _DEEP_INPUT_TOKEN_LIMIT, + _DEEP_MAX_OUTPUT_TOKENS, + _DEEP_MODEL_CONTEXT_WINDOW, + _DEEP_OUTPUT_MARGIN_TOKENS, + ) + from ouroboros.tools.review_helpers import REVIEW_PROMPT_TOKEN_BUDGET + + assert _DEEP_INPUT_TOKEN_LIMIT == min( + REVIEW_PROMPT_TOKEN_BUDGET, + _DEEP_MODEL_CONTEXT_WINDOW - _DEEP_MAX_OUTPUT_TOKENS - _DEEP_OUTPUT_MARGIN_TOKENS, + ) + assert _DEEP_INPUT_TOKEN_LIMIT + _DEEP_MAX_OUTPUT_TOKENS <= _DEEP_MODEL_CONTEXT_WINDOW, ( + "deep review input cap + reserved output exceeds the reviewer window; " + "the provider would hard-400." + ) + def test_tool_timeout_uses_max_of_settings_and_per_tool(): """_get_tool_timeout must return max(settings, per_tool) not just settings.""" - import importlib from unittest.mock import patch import ouroboros.loop_tool_execution as mod @@ -275,7 +304,6 @@ def test_full_repo_pack_excludes_junk_dirs(): def test_summary_and_reflection_callers_use_bounded_evidence(): """Summary and reflection prompt builders must call format_review_evidence_for_prompt with max_chars.""" - import ast from pathlib import Path for filename in ("ouroboros/agent_task_pipeline.py", "ouroboros/reflection.py"): diff --git a/tests/test_mcp_api.py b/tests/test_mcp_api.py index b8b19e4cd..579337a4d 100644 --- a/tests/test_mcp_api.py +++ b/tests/test_mcp_api.py @@ -311,7 +311,7 @@ def test_settings_post_rehydrates_masked_auth_token(tmp_path, monkeypatch): drive_root = tmp_path / "drive" drive_root.mkdir() - settings_path = tmp_path / "settings.json" + tmp_path / "settings.json" # Seed: a real auth token already saved on disk. real_token = "Bearer plaintext-live-token-DO-NOT-LEAK" diff --git a/tests/test_memory_blocks.py b/tests/test_memory_blocks.py index 6249ad8a0..eb5bf9281 100644 --- a/tests/test_memory_blocks.py +++ b/tests/test_memory_blocks.py @@ -1,6 +1,5 @@ """Tests for block-based memory pipeline (scratchpad blocks, dialogue blocks).""" import json -import pathlib import pytest diff --git a/tests/test_memory_tool_hints.py b/tests/test_memory_tool_hints.py index 393d49f58..f914abaa9 100644 --- a/tests/test_memory_tool_hints.py +++ b/tests/test_memory_tool_hints.py @@ -18,7 +18,6 @@ from __future__ import annotations import pathlib from dataclasses import dataclass -from typing import Any import pytest diff --git a/tests/test_message_routing.py b/tests/test_message_routing.py index 18967de57..7a5561633 100644 --- a/tests/test_message_routing.py +++ b/tests/test_message_routing.py @@ -16,7 +16,7 @@ sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..')) class TestOwnerInjectPerTask(unittest.TestCase): - """Test per-task mailbox in owner_inject.py.""" + """Test per-task mailbox in owner_mailbox.py.""" def setUp(self): self._tmpdir = tempfile.TemporaryDirectory() @@ -26,7 +26,7 @@ class TestOwnerInjectPerTask(unittest.TestCase): self._tmpdir.cleanup() def test_write_creates_per_task_file(self): - from ouroboros.owner_inject import write_owner_message, _mailbox_path + from ouroboros.owner_mailbox import write_owner_message, _mailbox_path write_owner_message(self.drive_root, "hello", task_id="abc123", msg_id="m1") path = _mailbox_path(self.drive_root, "abc123") self.assertTrue(path.exists()) @@ -36,7 +36,7 @@ class TestOwnerInjectPerTask(unittest.TestCase): self.assertEqual(entry["msg_id"], "m1") def test_drain_reads_only_own_task(self): - from ouroboros.owner_inject import write_owner_message, drain_owner_messages + from ouroboros.owner_mailbox import write_owner_message, drain_owner_messages write_owner_message(self.drive_root, "for task A", task_id="taskA", msg_id="m1") write_owner_message(self.drive_root, "for task B", task_id="taskB", msg_id="m2") @@ -47,7 +47,7 @@ class TestOwnerInjectPerTask(unittest.TestCase): self.assertEqual(msgs_b, ["for task B"]) def test_drain_dedup_with_seen_ids(self): - from ouroboros.owner_inject import write_owner_message, drain_owner_messages + from ouroboros.owner_mailbox import write_owner_message, drain_owner_messages write_owner_message(self.drive_root, "msg1", task_id="t1", msg_id="id1") write_owner_message(self.drive_root, "msg2", task_id="t1", msg_id="id2") @@ -62,7 +62,7 @@ class TestOwnerInjectPerTask(unittest.TestCase): self.assertIn("id3", seen) def test_cleanup_removes_file(self): - from ouroboros.owner_inject import write_owner_message, cleanup_task_mailbox, _mailbox_path + from ouroboros.owner_mailbox import write_owner_message, cleanup_task_mailbox, _mailbox_path write_owner_message(self.drive_root, "hello", task_id="t1", msg_id="m1") path = _mailbox_path(self.drive_root, "t1") self.assertTrue(path.exists()) @@ -71,13 +71,13 @@ class TestOwnerInjectPerTask(unittest.TestCase): self.assertFalse(path.exists()) def test_drain_nonexistent_task_returns_empty(self): - from ouroboros.owner_inject import drain_owner_messages + from ouroboros.owner_mailbox import drain_owner_messages msgs = drain_owner_messages(self.drive_root, task_id="nonexistent") self.assertEqual(msgs, []) def test_messages_not_cleared_on_read(self): """Messages persist after read (append-only). Only cleanup removes them.""" - from ouroboros.owner_inject import write_owner_message, drain_owner_messages, _mailbox_path + from ouroboros.owner_mailbox import write_owner_message, drain_owner_messages, _mailbox_path write_owner_message(self.drive_root, "persistent", task_id="t1", msg_id="m1") drain_owner_messages(self.drive_root, task_id="t1") @@ -87,7 +87,7 @@ class TestOwnerInjectPerTask(unittest.TestCase): self.assertIn("persistent", path.read_text()) def test_mailbox_rejects_unsafe_task_id(self): - from ouroboros.owner_inject import _mailbox_path + from ouroboros.owner_mailbox import _mailbox_path with self.assertRaises(ValueError): _mailbox_path(self.drive_root, "../settings") diff --git a/tests/test_multimodal_chat.py b/tests/test_multimodal_chat.py new file mode 100644 index 000000000..9ad5eac05 --- /dev/null +++ b/tests/test_multimodal_chat.py @@ -0,0 +1,222 @@ +"""Native multimodal chat (WS-H, v6.26.0): vision capability, attachment +blocks, eviction, image-aware token estimates, compaction safety, and +non-vision lane placeholders.""" + +import base64 +import json + +from ouroboros.context_budget import IMAGE_BLOCK_CHAR_EQUIVALENT, MAX_LIVE_IMAGE_BLOCKS + + +def _image_block(tag: str = "x", caption: str = "") -> dict: + return { + "type": "image_url", + "image_url": {"url": f"data:image/png;base64,{tag * 8}"}, + "_caption": caption, + } + + +class TestSupportsVision: + def test_static_map(self): + from ouroboros.provider_models import supports_vision + + assert supports_vision("openai/gpt-5.5") is True + assert supports_vision("google/gemini-3.5-flash") is True + assert supports_vision("anthropic::claude-opus-4-8") is True + assert supports_vision("deepseek/deepseek-chat") is False + assert supports_vision("") is False + assert supports_vision("some-model (local)") is False + + def test_overlay_wins(self): + from ouroboros import provider_models + + provider_models.update_vision_overlay("deepseek/deepseek-vl", True) + assert provider_models.supports_vision("deepseek/deepseek-vl") is True + provider_models.update_vision_overlay("openai/gpt-5.5", False) + try: + assert provider_models.supports_vision("openai/gpt-5.5") is False + finally: + provider_models._VISION_OVERLAY.pop("openai/gpt-5.5", None) + provider_models._VISION_OVERLAY.pop("deepseek/deepseek-vl", None) + + +class TestWebAttachmentBlocks: + def test_first_image_attachment_resolves_upload(self, tmp_path, monkeypatch): + import ouroboros.gateway.ws as ws_mod + + uploads = tmp_path / "uploads" + uploads.mkdir(parents=True) + (uploads / "abc_cat.png").write_bytes(b"\x89PNG fake") + monkeypatch.setattr(ws_mod, "DATA_DIR", tmp_path) + + b64, mime, caption = ws_mod._first_image_attachment([ + {"filename": "abc_cat.png", "mime": "image/png", "display_name": "cat.png"}, + ]) + assert base64.b64decode(b64) == b"\x89PNG fake" + assert mime == "image/png" + assert "cat.png" in caption + + def test_traversal_and_non_image_rejected(self, tmp_path, monkeypatch): + import ouroboros.gateway.ws as ws_mod + + (tmp_path / "uploads").mkdir(parents=True) + secret = tmp_path / "settings.json" + secret.write_text("{}", encoding="utf-8") + monkeypatch.setattr(ws_mod, "DATA_DIR", tmp_path) + + assert ws_mod._first_image_attachment( + [{"filename": "../settings.json", "mime": "image/png"}] + ) == ("", "", "") + assert ws_mod._first_image_attachment( + [{"filename": "doc.pdf", "mime": "application/pdf"}] + ) == ("", "", "") + + def test_build_user_content_attaches_caption_metadata(self): + from ouroboros.context import build_user_content + + content = build_user_content({ + "text": "look", + "image_base64": "QUJD", + "image_mime": "image/png", + "image_caption": "[user attachment: cat.png]", + }) + assert isinstance(content, list) + image_blocks = [b for b in content if b.get("type") == "image_url"] + assert image_blocks and image_blocks[0]["_caption"] == "[user attachment: cat.png]" + + +class TestImageEviction: + def test_keeps_only_newest_k(self): + from ouroboros.loop import _append_or_merge_user_content + + messages = [] + for idx in range(MAX_LIVE_IMAGE_BLOCKS + 2): + _append_or_merge_user_content( + messages, + [ + {"type": "text", "text": f"img {idx}"}, + _image_block(str(idx), caption=f"shot-{idx}"), + ], + ) + messages.append({"role": "assistant", "content": "ok"}) + + live = [ + block + for msg in messages + if isinstance(msg.get("content"), list) + for block in msg["content"] + if isinstance(block, dict) and block.get("type") == "image_url" + ] + assert len(live) == MAX_LIVE_IMAGE_BLOCKS + rendered = json.dumps(messages, ensure_ascii=False) + assert "[image evicted: shot-0]" in rendered + assert "[image evicted: shot-1]" in rendered + + def test_placeholder_includes_reviewable_path(self): + from ouroboros.loop import _evict_stale_image_blocks + + block = _image_block("a", caption="screen") + block["_source_path"] = "/data/uploads/screenshots/x.png" + messages = [{"role": "user", "content": [block]}] + _evict_stale_image_blocks(messages, incoming=MAX_LIVE_IMAGE_BLOCKS) + text = messages[0]["content"][0]["text"] + assert "vlm_query file_path=/data/uploads/screenshots/x.png" in text + + +class TestImageTokenEstimates: + def test_loop_estimate_uses_fixed_equivalent(self): + from ouroboros.loop import _estimate_messages_chars + + huge_b64 = "A" * 1_000_000 + messages = [{ + "role": "user", + "content": [ + {"type": "text", "text": "hi"}, + {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{huge_b64}"}}, + ], + }] + total = _estimate_messages_chars(messages) + assert total < IMAGE_BLOCK_CHAR_EQUIVALENT + 1000, ( + "image block must count as the fixed equivalent, not base64 length" + ) + + def test_llm_estimate_symmetric(self): + from ouroboros.llm import _estimate_message_chars + + huge_b64 = "A" * 1_000_000 + messages = [{ + "role": "user", + "content": [{"type": "image_url", "image_url": {"url": huge_b64}}], + }] + assert _estimate_message_chars(messages) == IMAGE_BLOCK_CHAR_EQUIVALENT + + +class TestCompactionAndLanes: + def test_render_round_block_replaces_image(self): + from ouroboros.context_compaction import _render_round_block + + messages = [{ + "role": "user", + "content": [ + {"type": "text", "text": "see this"}, + _image_block("a", caption="login page"), + ], + }] + rendered = _render_round_block(messages, 0, 0) + assert "[image: login page]" in rendered + assert "base64" not in rendered + + def test_gigachat_text_placeholder(self): + from ouroboros.llm import LLMClient + + text = LLMClient._gigachat_text([ + {"type": "text", "text": "hello "}, + _image_block("a"), + ]) + assert "hello" in text + assert "[image omitted: model has no vision" in text + + def test_provider_payload_strips_internal_metadata(self): + from ouroboros.llm import LLMClient + + cleaned = LLMClient._copy_messages_with_cache_policy( + [{"role": "user", "content": [ + {"type": "image_url", "image_url": {"url": "u"}, "_caption": "c", "_source_path": "p"}, + ]}], + allow_message_cache_control=False, + flatten_tool_content_blocks=False, + ) + block = cleaned[0]["content"][0] + assert "_caption" not in block and "_source_path" not in block + + +class TestNativeScreenshotInjection: + def test_injects_for_vision_model(self, tmp_path, monkeypatch): + import ouroboros.tools.browser as browser_mod + + monkeypatch.setenv("OUROBOROS_MODEL", "openai/gpt-5.5") + + class Ctx: + drive_root = tmp_path + messages = [{"role": "user", "content": "start"}] + + note = browser_mod._inject_native_screenshot(Ctx(), base64.b64encode(b"png").decode()) + assert "natively" in note + content = Ctx.messages[-1]["content"] + assert isinstance(content, list) + assert any(b.get("type") == "image_url" for b in content if isinstance(b, dict)) + shots = list((tmp_path / "uploads" / "screenshots").glob("*.png")) + assert shots, "screenshot must be persisted for re-view" + + def test_skipped_for_non_vision_model(self, tmp_path, monkeypatch): + import ouroboros.tools.browser as browser_mod + + monkeypatch.setenv("OUROBOROS_MODEL", "deepseek/deepseek-chat") + + class Ctx: + drive_root = tmp_path + messages = [{"role": "user", "content": "start"}] + + note = browser_mod._inject_native_screenshot(Ctx(), base64.b64encode(b"png").decode()) + assert note == "" + assert Ctx.messages[-1]["content"] == "start" diff --git a/tests/test_observability_outcomes_v2.py b/tests/test_observability_outcomes_v2.py index 7fab3e1c7..32a43b99e 100644 --- a/tests/test_observability_outcomes_v2.py +++ b/tests/test_observability_outcomes_v2.py @@ -16,7 +16,6 @@ from ouroboros.outcomes import ( EXECUTION_INFRA_FAILED, EXECUTION_OK, OBJECTIVE_NOT_EVALUATED, - RESULT_FAILED, RESULT_INFRA_FAILED, artifact_bundle_from_result, build_verification_ledger, diff --git a/tests/test_observability_retention.py b/tests/test_observability_retention.py index af448b3ec..cadf57279 100644 --- a/tests/test_observability_retention.py +++ b/tests/test_observability_retention.py @@ -5,7 +5,7 @@ from ouroboros.observability import persist_call, prune_observability_blobs, wri def test_observability_retention_preserves_old_manifests_and_orphan_blobs(tmp_path): - old = persist_call( + persist_call( tmp_path, task_id="old-task", call_id="old-call", diff --git a/tests/test_oop_extension_parity.py b/tests/test_oop_extension_parity.py index 34952df34..d2a42a467 100644 --- a/tests/test_oop_extension_parity.py +++ b/tests/test_oop_extension_parity.py @@ -7,7 +7,6 @@ on_unload failure) or drop the live->broken health memory. """ from __future__ import annotations -import pathlib import pytest diff --git a/tests/test_per_skill_version_resync.py b/tests/test_per_skill_version_resync.py index d78d74964..fc97e120c 100644 --- a/tests/test_per_skill_version_resync.py +++ b/tests/test_per_skill_version_resync.py @@ -22,7 +22,6 @@ This module pins all six. from __future__ import annotations -import json import logging import pathlib import textwrap diff --git a/tests/test_platform_guard.py b/tests/test_platform_guard.py index 776cf6ca0..293f3ba9b 100644 --- a/tests/test_platform_guard.py +++ b/tests/test_platform_guard.py @@ -184,21 +184,6 @@ def test_platform_layer_exports_core_symbols(): IS_WINDOWS, IS_MACOS, IS_LINUX, - kill_process_tree, - terminate_process_tree, - force_kill_pid, - kill_pid_tree, - kill_process_on_port, - pid_lock_acquire, - pid_lock_release, - file_lock_exclusive, - file_lock_shared, - file_lock_exclusive_nb, - file_unlock, - create_new_session, - get_system_memory, - get_cpu_info, - git_install_hint, ) # Smoke check: flags are booleans assert isinstance(IS_WINDOWS, bool) diff --git a/tests/test_post_task_evolution.py b/tests/test_post_task_evolution.py index 91b0a3741..56a720d9d 100644 --- a/tests/test_post_task_evolution.py +++ b/tests/test_post_task_evolution.py @@ -6,7 +6,6 @@ import json import pathlib import types -import pytest import ouroboros.config as config import ouroboros.post_task_evolution as pte @@ -138,6 +137,14 @@ def test_v5_apply_pending_request_activates_gated_campaign(tmp_path, monkeypatch monkeypatch.setattr(st, "load_state", lambda: {"owner_chat_id": 7}) monkeypatch.setattr(st, "save_state", lambda s: saved.update(s)) + def _fake_update_state(mutator): + live = {"owner_chat_id": 7} + mutator(live) + saved.update(live) + return live + + monkeypatch.setattr(st, "update_state", _fake_update_state) + assert pte.apply_pending_request(tmp_path) is True assert "plan_task" in started["objective"] # requires_plan_review carried in assert started["source"] == "post_task" diff --git a/tests/test_pricing.py b/tests/test_pricing.py index da72b9cba..e0f702308 100644 --- a/tests/test_pricing.py +++ b/tests/test_pricing.py @@ -368,7 +368,6 @@ class TestGetPricing: assert pricing["anthropic/claude-opus-4-7"] == (5.0, 0.5, 25.0) def test_caches_after_successful_fetch(self): - import ouroboros.pricing as mod live = {f"p/m-{i}": (1.0, 0.1, 2.0) for i in range(6)} mock_fetch = MagicMock(return_value=live) with patch(FETCH_PRICING_PATH, mock_fetch): @@ -377,7 +376,6 @@ class TestGetPricing: mock_fetch.assert_called_once() def test_retries_after_failed_fetch(self): - import ouroboros.pricing as mod mock_fetch = MagicMock(side_effect=Exception("down")) with patch(FETCH_PRICING_PATH, mock_fetch): get_pricing() @@ -386,7 +384,6 @@ class TestGetPricing: def test_ignores_small_live_pricing(self): """If fetch returns < 5 entries, don't merge (probably broken).""" - import ouroboros.pricing as mod live = {"a/b": (1, 0.1, 2)} # Only 1 entry with patch(FETCH_PRICING_PATH, return_value=live): pricing = get_pricing() @@ -394,7 +391,6 @@ class TestGetPricing: def test_thread_safety(self): """Multiple threads calling get_pricing() simultaneously shouldn't crash.""" - import ouroboros.pricing as mod results = [] errors = [] diff --git a/tests/test_process_custody.py b/tests/test_process_custody.py new file mode 100644 index 000000000..416bcbd0b --- /dev/null +++ b/tests/test_process_custody.py @@ -0,0 +1,226 @@ +"""Process custody: supervised spawn chokepoint, ledger, reaper, lifeline.""" + +import json +import os +import pathlib +import re +import subprocess +import sys +import time + +import pytest + +from ouroboros import process_custody +from ouroboros.process_custody import ( + ledger_path, + reap_orphaned_processes, + record_process, + spawn_supervised, +) + +REPO_ROOT = pathlib.Path(__file__).resolve().parents[1] + +# Popen call sites that legitimately bypass spawn_supervised: short-lived +# synchronous helpers (waited within the call), panic/cleanup layers, the +# launcher (custody host), and custody itself. Adding a NEW long-lived spawn +# site requires routing it through spawn_supervised/record_process or +# explicitly justifying it here. +_POPEN_ALLOWLIST = { + "launcher.py", # custody host process (pre-runtime) + "ouroboros/process_custody.py", # the chokepoint itself + "ouroboros/platform_layer.py", # primitives (hidden_run helpers) + "ouroboros/packaged_cli.py", # user-facing CLI wrapper (foreground) + "ouroboros/cli.py", # dev CLI (foreground) + "ouroboros/server_control.py", # restart exec path + "ouroboros/headless.py", # waited synchronous child + "ouroboros/preflight_runner.py", # waited hermetic pytest child + "ouroboros/tools/shell.py", # bounded foreground commands (waited + tracked) + "ouroboros/tools/skill_exec.py", # bounded skill run (waited + tracked) + "ouroboros/tools/skill_preflight.py", # waited preflight child + "ouroboros/marketplace/isolated_deps.py", # waited installer child + "ouroboros/gateways/claude_code.py", # waited readonly child (timeout-bound) + "ouroboros/extension_process_runner.py", # waited extension child + "ouroboros/workspace_executor.py", # custody write-through added at spawn + "ouroboros/local_model.py", # custody record added at spawn + "ouroboros/extension_companion.py", # custody write-through added at spawn + "ouroboros/tools/services.py", # routed through spawn_supervised +} + + +def test_popen_sites_are_custodied_or_allowlisted(): + pattern = re.compile(r"subprocess\.Popen\(|[^.\w]Popen\(") + offenders = [] + for base in ("ouroboros", "supervisor"): + for path in (REPO_ROOT / base).rglob("*.py"): + rel = path.relative_to(REPO_ROOT).as_posix() + text = path.read_text(encoding="utf-8", errors="replace") + if pattern.search(text) and rel not in _POPEN_ALLOWLIST: + offenders.append(rel) + for name in ("server.py", "launcher.py"): + path = REPO_ROOT / name + if path.exists() and pattern.search(path.read_text(encoding="utf-8", errors="replace")): + if name not in _POPEN_ALLOWLIST: + offenders.append(name) + assert not offenders, ( + "New raw Popen call sites outside the custody allowlist: " + f"{offenders}. Route long-lived spawns through " + "process_custody.spawn_supervised (or record_process write-through) " + "or extend the allowlist with a justification comment." + ) + + +def test_spawn_supervised_records_ledger_entry(tmp_path): + proc = spawn_supervised( + [sys.executable, "-c", "import time; time.sleep(30)"], + drive_root=tmp_path, + purpose="test-sleeper", + scope="task", + owner_task_id="t123", + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + try: + lines = ledger_path(tmp_path).read_text(encoding="utf-8").strip().splitlines() + entry = json.loads(lines[-1]) + assert entry["pid"] == proc.pid + assert entry["purpose"] == "test-sleeper" + assert entry["scope"] == "task" + assert entry["owner_task"] == "t123" + assert entry["session_id"] == process_custody.current_custody_session_id() + if os.name != "nt": + assert entry["fingerprint"]["start_time"] + assert entry["fingerprint"]["cmd_sha256"] + finally: + proc.kill() + proc.wait(timeout=5) + + +def test_reaper_kills_stale_session_entry(tmp_path, monkeypatch): + proc = spawn_supervised( + [sys.executable, "-c", "import time; time.sleep(60)"], + drive_root=tmp_path, + purpose="stale-service", + scope="task", + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + try: + # Simulate a NEW server generation. + monkeypatch.setattr(process_custody, "_SESSION_ID", "next-generation") + reaped = reap_orphaned_processes(tmp_path) + assert proc.pid in reaped + deadline = time.time() + 5 + while time.time() < deadline and proc.poll() is None: + time.sleep(0.05) + assert proc.poll() is not None, "stale-session process must be dead" + events = (tmp_path / "logs" / "supervisor.jsonl").read_text(encoding="utf-8") + assert "process_reaped" in events + finally: + if proc.poll() is None: + proc.kill() + proc.wait(timeout=5) + + +def test_reaper_never_kills_fingerprint_mismatch(tmp_path, monkeypatch): + proc = subprocess.Popen( + [sys.executable, "-c", "import time; time.sleep(60)"], + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, + ) + try: + # Ledger entry claims this pid but with a FOREIGN fingerprint — + # models a recycled pid from another install. Strict rule: never kill. + record = record_process( + tmp_path, pid=proc.pid, cmd=["sleep", "60"], + purpose="foreign", scope="task", + ) + entries = [dict(record, fingerprint={"start_time": "FOREIGN", "cmd_sha256": "deadbeef"})] + process_custody._rewrite_ledger(tmp_path, entries) + monkeypatch.setattr(process_custody, "_SESSION_ID", "next-generation") + reaped = reap_orphaned_processes(tmp_path) + assert proc.pid not in reaped + assert proc.poll() is None, "fingerprint-mismatched process must survive" + finally: + proc.kill() + proc.wait(timeout=5) + + +def test_reaper_keeps_daemon_and_same_session(tmp_path): + proc = spawn_supervised( + [sys.executable, "-c", "import time; time.sleep(60)"], + drive_root=tmp_path, + purpose="companion", + scope="daemon", + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + proc2 = spawn_supervised( + [sys.executable, "-c", "import time; time.sleep(60)"], + drive_root=tmp_path, + purpose="live-session-service", + scope="session", + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + try: + reaped = reap_orphaned_processes(tmp_path) + assert proc.pid not in reaped + assert proc2.pid not in reaped + assert proc.poll() is None + assert proc2.poll() is None + finally: + for p in (proc, proc2): + p.kill() + p.wait(timeout=5) + + +def test_task_scope_reaped_when_owner_task_gone(tmp_path): + proc = spawn_supervised( + [sys.executable, "-c", "import time; time.sleep(60)"], + drive_root=tmp_path, + purpose="task-service", + scope="task", + owner_task_id="finished-task", + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + try: + reaped = reap_orphaned_processes(tmp_path, running_task_ids={"some-other-task"}) + assert proc.pid in reaped + finally: + if proc.poll() is None: + proc.kill() + proc.wait(timeout=5) + + +@pytest.mark.skipif(os.name == "nt", reason="lifeline is POSIX-only") +def test_lifeline_kills_child_when_parent_dies(tmp_path): + # Parent spawns a child that starts the lifeline, then the parent exits. + child_src = ( + "import sys; sys.path.insert(0, %r);" + "from ouroboros.process_custody import start_parent_lifeline;" + "start_parent_lifeline(poll_sec=0.2, label='test');" + "import time; time.sleep(60)" + ) % str(REPO_ROOT) + parent_src = ( + "import subprocess, sys, pathlib;" + f"child = subprocess.Popen([sys.executable, '-c', {child_src!r}]);" + "pathlib.Path(sys.argv[1]).write_text(str(child.pid));" + ) + pid_file = tmp_path / "child_pid" + subprocess.run( + [sys.executable, "-c", parent_src, str(pid_file)], + check=True, timeout=15, + ) + child_pid = int(pid_file.read_text()) + deadline = time.time() + 10 + while time.time() < deadline: + try: + os.kill(child_pid, 0) + except ProcessLookupError: + return # lifeline fired + time.sleep(0.2) + try: + os.kill(child_pid, 9) + except ProcessLookupError: + return + raise AssertionError("child outlived dead parent despite lifeline") diff --git a/tests/test_process_memory.py b/tests/test_process_memory.py index eba02822f..79d6647e4 100644 --- a/tests/test_process_memory.py +++ b/tests/test_process_memory.py @@ -10,7 +10,6 @@ import inspect import os import sys -import pytest REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) sys.path.insert(0, REPO) diff --git a/tests/test_release_sync.py b/tests/test_release_sync.py index 19352f49b..d3f231b31 100644 --- a/tests/test_release_sync.py +++ b/tests/test_release_sync.py @@ -1,6 +1,5 @@ """Tests for ouroboros/tools/release_sync.py (standalone library, no wire-up).""" -import textwrap from pathlib import Path import pytest diff --git a/tests/test_repo_commit_dirty_tree.py b/tests/test_repo_commit_dirty_tree.py index 65b3ffb6f..d235874ca 100644 --- a/tests/test_repo_commit_dirty_tree.py +++ b/tests/test_repo_commit_dirty_tree.py @@ -19,7 +19,6 @@ import pathlib from types import SimpleNamespace from unittest.mock import patch -import pytest def _fake_run_cmd_sequence(*returns): diff --git a/tests/test_repo_read_limits.py b/tests/test_repo_read_limits.py index dad5e363e..522a1c717 100644 --- a/tests/test_repo_read_limits.py +++ b/tests/test_repo_read_limits.py @@ -229,7 +229,6 @@ def test_repo_read_can_read_architecture_md_in_one_call(tmp_path): def test_consciousness_context_includes_architecture_md(tmp_path): """BackgroundConsciousness._build_context must include ARCHITECTURE.md section.""" - import pathlib import queue from unittest.mock import patch, MagicMock @@ -319,7 +318,6 @@ def test_consciousness_context_architecture_before_knowledge_base(tmp_path): def test_triad_review_prompt_includes_architecture_md(tmp_path): """Triad review prompt must include ARCHITECTURE.md even when it is not in touched files.""" - import pathlib from ouroboros.tools.review import _REVIEW_PROMPT_TEMPLATE from ouroboros.tools.review_helpers import CRITICAL_FINDING_CALIBRATION, load_governance_doc diff --git a/tests/test_restart_reconnect.py b/tests/test_restart_reconnect.py index 7f4e50a66..fb0252944 100644 --- a/tests/test_restart_reconnect.py +++ b/tests/test_restart_reconnect.py @@ -323,6 +323,11 @@ def test_owner_restart_cleans_flags_when_worker_shutdown_fails(tmp_path, monkeyp def save_state(self, _state): return None + def update_state(self, mutator): + live = {} + mutator(live) + return live + def send_with_budget(self, _chat_id, text): messages.append(text) diff --git a/tests/test_review_fidelity.py b/tests/test_review_fidelity.py index daf5cc745..64b48a223 100644 --- a/tests/test_review_fidelity.py +++ b/tests/test_review_fidelity.py @@ -297,7 +297,7 @@ class TestCommitGateFreshnessMessages: def _setup(self): """Return (tmp_dir, ctx, repo_dir, drive_root, save_state, load_state, rs_mod).""" - from ouroboros.review_state import save_state, load_state, AdvisoryReviewState + from ouroboros.review_state import save_state, load_state import importlib rs_mod = importlib.import_module("ouroboros.review_state") @@ -324,7 +324,7 @@ class TestCommitGateFreshnessMessages: def test_fresh_with_open_obligations_shows_all(self, monkeypatch): """fresh advisory + >5 obligations: all obligations appear in warning text.""" from ouroboros.review_state import ( - AdvisoryReviewState, AdvisoryRunRecord, save_state, compute_snapshot_hash, + AdvisoryRunRecord, compute_snapshot_hash, ) from ouroboros.tools.commit_gate import _check_advisory_freshness @@ -358,7 +358,7 @@ class TestCommitGateFreshnessMessages: def test_parse_failure_with_open_obligations_shows_all(self): """parse_failure advisory + >5 obligations: all obligations appear in warning text.""" from ouroboros.review_state import ( - AdvisoryReviewState, AdvisoryRunRecord, save_state, compute_snapshot_hash, + AdvisoryRunRecord, compute_snapshot_hash, ) from ouroboros.tools.commit_gate import _check_advisory_freshness @@ -390,7 +390,7 @@ class TestCommitGateFreshnessMessages: def test_stale_with_open_obligations_shows_all(self): """No fresh advisory (stale/no-run) + >5 obligations: all obligations listed.""" from ouroboros.review_state import ( - AdvisoryReviewState, AdvisoryRunRecord, save_state, compute_snapshot_hash, + AdvisoryRunRecord, ) from ouroboros.tools.commit_gate import _check_advisory_freshness @@ -788,7 +788,7 @@ class TestPersistencePathNotTruncated: def test_long_commit_message_in_obligation_source(self): """_update_obligations_from_attempt must store full commit_message in source_attempt_msg.""" from ouroboros.review_state import ( - AdvisoryReviewState, CommitAttemptRecord, ObligationItem, + AdvisoryReviewState, ) long_msg = "fix: " + "B" * 300 # 305 chars — well above old [:200] cap @@ -946,7 +946,6 @@ class TestGitCommitFailureForensicMetadata: def _make_ctx(self, tmp_dir): """Build a minimal ToolContext-like object with forensic model metadata.""" import pathlib - from ouroboros.tools.registry import ToolContext class _MinimalCtx: pass @@ -964,7 +963,7 @@ class TestGitCommitFailureForensicMetadata: def test_record_commit_attempt_infra_failure_stores_triad_models(self): """_record_commit_attempt with status=failed must persist triad_models.""" import pathlib, tempfile - from ouroboros.review_state import load_state, save_state, AdvisoryReviewState + from ouroboros.review_state import load_state from ouroboros.tools.git import _record_commit_attempt with tempfile.TemporaryDirectory() as tmp: diff --git a/tests/test_review_observability.py b/tests/test_review_observability.py index beb092e47..5bfaeaa50 100644 --- a/tests/test_review_observability.py +++ b/tests/test_review_observability.py @@ -16,12 +16,9 @@ Verifies that: from __future__ import annotations import json -import types import pathlib -import tempfile from unittest.mock import MagicMock, patch -import pytest # ── helpers ─────────────────────────────────────────────────────────────────── @@ -573,7 +570,6 @@ def test_parse_failure_does_not_count_toward_quorum(tmp_path): was counted as a successful quorum participant, silently undermining epistemic integrity. """ - from ouroboros.tools import review as review_mod from ouroboros.tools.review import _collect_review_findings ctx = _make_ctx(tmp_path) @@ -689,7 +685,7 @@ def test_scope_budget_exceeded_has_prompt_chars(tmp_path): Before this fix, _handle_prompt_signals set prompt_chars=0 on the budget_exceeded path, losing the only forensic fact about why scope review was skipped. """ - from ouroboros.tools.scope_review import ScopeReviewResult, _handle_prompt_signals, _TouchedContextStatus + from ouroboros.tools.scope_review import _handle_prompt_signals, _TouchedContextStatus token_count = 800_000 # exceeds the 750K gate context_status = _TouchedContextStatus(status="budget_exceeded", token_count=token_count) diff --git a/tests/test_review_readiness_gate.py b/tests/test_review_readiness_gate.py index 6346ff11a..1bd5a0c95 100644 --- a/tests/test_review_readiness_gate.py +++ b/tests/test_review_readiness_gate.py @@ -1,10 +1,8 @@ """Tests for the review readiness gate (cheap deterministic pre-advisory checks).""" -import pathlib import subprocess from unittest.mock import patch, MagicMock -import pytest from ouroboros.tools.review_helpers import check_worktree_readiness diff --git a/tests/test_runtime_mode_elevation.py b/tests/test_runtime_mode_elevation.py index 2797a3aca..44d45d0d6 100644 --- a/tests/test_runtime_mode_elevation.py +++ b/tests/test_runtime_mode_elevation.py @@ -795,7 +795,6 @@ def test_set_tool_timeout_cannot_smuggle_elevation(isolated_settings, monkeypatc carries an elevated mode that the on-disk baseline does not. """ from ouroboros.config import load_settings - from ouroboros.tools import control as control_mod # Step 1: legitimate baseline = light. _seed_disk(isolated_settings, {"OUROBOROS_RUNTIME_MODE": "light"}) @@ -1017,7 +1016,6 @@ def test_launcher_skill_key_grant_supports_extensions(monkeypatch, tmp_path): runs in the launcher process and would not affect the server). """ import launcher - from io import BytesIO class _Manifest: env_from_settings = ["OPENROUTER_API_KEY"] diff --git a/tests/test_scope_review.py b/tests/test_scope_review.py index 0516bbeff..02e86a8db 100644 --- a/tests/test_scope_review.py +++ b/tests/test_scope_review.py @@ -16,7 +16,6 @@ import importlib import inspect import json import os -import pathlib import subprocess import sys diff --git a/tests/test_scratchpad_consolidation.py b/tests/test_scratchpad_consolidation.py index 070b03bcf..d7b6fb521 100644 --- a/tests/test_scratchpad_consolidation.py +++ b/tests/test_scratchpad_consolidation.py @@ -12,7 +12,6 @@ import os import pathlib import tempfile -import pytest from ouroboros.memory import Memory diff --git a/tests/test_send_photo.py b/tests/test_send_photo.py index d4fb72ca4..01aa5d942 100644 --- a/tests/test_send_photo.py +++ b/tests/test_send_photo.py @@ -1,9 +1,7 @@ """Tests for send_photo file_path support and MIME detection.""" import base64 -import pathlib import types -import pytest from ouroboros.tools.core import _send_photo, _detect_image_mime, _MAX_PHOTO_FILE_BYTES diff --git a/tests/test_settings_budget_hotreload.py b/tests/test_settings_budget_hotreload.py index 5fd9e3b0f..8d7e4319c 100644 --- a/tests/test_settings_budget_hotreload.py +++ b/tests/test_settings_budget_hotreload.py @@ -31,11 +31,11 @@ def _settings_client(monkeypatch, tmp_path, current: dict): def test_settings_post_updates_budget_limits_and_per_task_threshold(monkeypatch, tmp_path): - import server as srv import supervisor.message_bus as bus_mod import supervisor.state as state_mod - current = dict(srv._SETTINGS_DEFAULTS) + from ouroboros.config import SETTINGS_DEFAULTS as _defaults + current = dict(_defaults) current["TOTAL_BUDGET"] = 10.0 monkeypatch.setattr(state_mod, "TOTAL_BUDGET_LIMIT", 10.0) monkeypatch.setattr(bus_mod, "TOTAL_BUDGET_LIMIT", 10.0) @@ -81,10 +81,10 @@ def test_settings_post_updates_budget_limits_and_per_task_threshold(monkeypatch, def test_settings_post_rejects_malformed_evolution_cadence(monkeypatch, tmp_path): """A direct API client must not be able to persist a malformed post-task evolution cadence (e.g. every_n:0) — backend half of the strict every_n validation contract.""" - import server as srv key = "OUROBOROS_POST_TASK_EVOLUTION_CADENCE" - current = dict(srv._SETTINGS_DEFAULTS) + from ouroboros.config import SETTINGS_DEFAULTS as _defaults + current = dict(_defaults) current[key] = "llm" client = _settings_client(monkeypatch, tmp_path, current) diff --git a/tests/test_settings_effort.py b/tests/test_settings_effort.py index 0437860ba..31532e59c 100644 --- a/tests/test_settings_effort.py +++ b/tests/test_settings_effort.py @@ -9,7 +9,6 @@ from ouroboros.config import ( get_review_enforcement, get_scope_review_models, get_task_review_mode, - get_auto_grant_enabled, get_context_mode, ) diff --git a/tests/test_shell_regex_hint.py b/tests/test_shell_regex_hint.py index 4b91daa65..ef5b2792f 100644 --- a/tests/test_shell_regex_hint.py +++ b/tests/test_shell_regex_hint.py @@ -12,9 +12,7 @@ Pinned tests so the regression class doesn't reopen. from __future__ import annotations from types import SimpleNamespace -from unittest.mock import patch -import pytest from ouroboros.tools.shell import _run_shell diff --git a/tests/test_skill_exec.py b/tests/test_skill_exec.py index 7203bc900..8a1da42ce 100644 --- a/tests/test_skill_exec.py +++ b/tests/test_skill_exec.py @@ -9,11 +9,9 @@ from __future__ import annotations import asyncio import json -import os import pathlib import shutil import sys -import tempfile import threading from unittest.mock import patch @@ -314,7 +312,6 @@ def test_run_shell_restores_obfuscated_self_authored_state_marker(tmp_path): def test_claude_code_edit_resolves_skill_cwd_under_drive_root(tmp_path, monkeypatch): from ouroboros.tools import shell as shell_mod - import sys import types ctx = _make_ctx(tmp_path) @@ -1421,7 +1418,6 @@ def test_skill_exec_kills_runaway_stdout_output(tmp_path, monkeypatch): assert "SKILL_EXEC_OVERFLOW" in result, result[:500] # Output in the returned payload must be bounded by the cap. import json as _json - sentinel = "SKILL_EXEC_OVERFLOW" json_start = result.find("{") payload = _json.loads(result[json_start:]) # Streamed stdout buffer must be close to the cap, not megabytes. @@ -1656,7 +1652,6 @@ def test_toggle_skill_refuses_when_load_error_set(tmp_path, monkeypatch): both skills with load_error. ``toggle_skill`` must not mutate state for such skills — otherwise the two directories would still end up sharing ``enabled.json``.""" - import os skills_root = tmp_path / "skills" _build_skill(skills_root, "hello world") _build_skill(skills_root, "hello_world") diff --git a/tests/test_skill_loader.py b/tests/test_skill_loader.py index ecf041780..740c16c60 100644 --- a/tests/test_skill_loader.py +++ b/tests/test_skill_loader.py @@ -145,7 +145,7 @@ def test_load_skill_surfaces_unreadable_manifest(tmp_path): """Phase 3 round 16 regression: an existing-but-unreadable manifest must surface as ``load_error`` instead of silently looking like "not a skill dir at all".""" - import os, platform, stat + import os, platform if platform.system() == "Windows": pytest.skip("chmod-based permission test not portable to Windows") drive_root = tmp_path / "drive" @@ -406,7 +406,7 @@ def test_available_for_execution_rejects_unsupported_runtime(tmp_path): drive_root = tmp_path / "drive" drive_root.mkdir() repo_root = tmp_path / "skills" - skill_dir = _write_skill( + _write_skill( repo_root, "alpha", manifest=_valid_script_manifest("alpha").replace("runtime: python3", "runtime: perl"), @@ -564,7 +564,7 @@ def test_symlink_escape_excluded_from_pack(tmp_path): outside.write_text("SECRET_PAYLOAD\n", encoding="utf-8") escape_link = skill_dir / "escape.txt" os.symlink(outside, escape_link) - files = _iter_payload_files_list = None + _iter_payload_files_list = None # Use the private walker directly — this is the "would the hash / # review pack see this file" question. from ouroboros.skill_loader import _iter_payload_files diff --git a/tests/test_skill_review.py b/tests/test_skill_review.py index 6d7ac2e30..b12f242ae 100644 --- a/tests/test_skill_review.py +++ b/tests/test_skill_review.py @@ -15,7 +15,6 @@ from __future__ import annotations import json import pathlib -from typing import List from unittest.mock import patch import pytest diff --git a/tests/test_skill_review_persist_guard.py b/tests/test_skill_review_persist_guard.py index 79e3789a5..4bd71ba3e 100644 --- a/tests/test_skill_review_persist_guard.py +++ b/tests/test_skill_review_persist_guard.py @@ -128,7 +128,7 @@ def test_mismatched_review_job_hash_cannot_late_write_pass(tmp_path, monkeypatch monkeypatch.setenv("OUROBOROS_SKILLS_REPO_PATH", str(skills_root)) ctx = ToolContext(repo_dir=repo_dir, drive_root=drive_root) ctx._skill_review_lifecycle_guard = True - content_hash = compute_content_hash(skill_dir) + compute_content_hash(skill_dir) save_review_state( drive_root, diff --git a/tests/test_smoke.py b/tests/test_smoke.py index c2cd5ba30..cd85cb18d 100644 --- a/tests/test_smoke.py +++ b/tests/test_smoke.py @@ -360,7 +360,7 @@ def test_no_hardcoded_replies(): if "{" in line or "f'" in line or 'f"' in line: continue violations.append(f"{path.name}:{i}: {line.strip()}") - assert len(violations) < 5, f"Possible hardcoded replies:\n" + "\n".join(violations) + assert len(violations) < 5, "Possible hardcoded replies:\n" + "\n".join(violations) def test_version_file_exists(): @@ -416,7 +416,7 @@ def test_no_env_dumping(): continue if dangerous.search(line): violations.append(f"{path.name}:{i}: {line.strip()[:80]}") - assert len(violations) == 0, f"Dangerous env dumping:\n" + "\n".join(violations) + assert len(violations) == 0, "Dangerous env dumping:\n" + "\n".join(violations) def test_no_oversized_modules(): @@ -462,7 +462,7 @@ def test_no_bare_except_pass(): if next_line and next_line == "pass": violations.append(f"{path.name}:{i}: bare except: pass") break - assert len(violations) == 0, f"Bare except:pass found:\n" + "\n".join(violations) + assert len(violations) == 0, "Bare except:pass found:\n" + "\n".join(violations) # ── AST-based function size check ─────────────────────────────── diff --git a/tests/test_task_constraint_server_routing.py b/tests/test_task_constraint_server_routing.py index 6a3d9d3b2..73d441ee6 100644 --- a/tests/test_task_constraint_server_routing.py +++ b/tests/test_task_constraint_server_routing.py @@ -28,6 +28,7 @@ def test_constrained_repair_is_not_injected_into_busy_agent(monkeypatch): ctx = SimpleNamespace( load_state=lambda: {"owner_id": 1}, save_state=lambda st: None, + update_state=lambda mutator: (lambda st: (mutator(st), st)[1])({"owner_id": 1}), consciousness=SimpleNamespace(inject_observation=lambda *_: None, pause=lambda: None, resume=lambda: None), get_chat_agent=lambda: agent, handle_chat_direct=lambda cid, txt, img, task_constraint=None, task_metadata=None: calls["direct"].append(task_constraint), diff --git a/tests/test_task_constraint_tools.py b/tests/test_task_constraint_tools.py index be10f5a11..55dcd80b9 100644 --- a/tests/test_task_constraint_tools.py +++ b/tests/test_task_constraint_tools.py @@ -1,4 +1,3 @@ -import pathlib from ouroboros.contracts.task_constraint import TaskConstraint, resolve_payload_path from ouroboros.tools.core import _data_write diff --git a/tests/test_task_done_ordering.py b/tests/test_task_done_ordering.py index 690ca6b75..945c29e2a 100644 --- a/tests/test_task_done_ordering.py +++ b/tests/test_task_done_ordering.py @@ -5,10 +5,8 @@ ensuring that send_message reaches the UI before task_done (causal ordering). """ import pathlib -import tempfile import json -import pytest def _make_fake_env(drive_root: pathlib.Path): diff --git a/tests/test_tool_capabilities.py b/tests/test_tool_capabilities.py index bc599ec15..5ed96fbed 100644 --- a/tests/test_tool_capabilities.py +++ b/tests/test_tool_capabilities.py @@ -17,7 +17,6 @@ import re import sys import tempfile -import pytest # --------------------------------------------------------------------------- # SSOT drift tests diff --git a/tests/test_transcript_seal.py b/tests/test_transcript_seal.py index b8ac973d5..a26f875fd 100644 --- a/tests/test_transcript_seal.py +++ b/tests/test_transcript_seal.py @@ -1,6 +1,5 @@ """Tests for seal_task_transcript — transcript cache-boundary sealing.""" -import pytest from ouroboros.loop import seal_task_transcript, _extract_plain_text_from_content diff --git a/tests/test_transport_commands.py b/tests/test_transport_commands.py index 51d1c5fc7..cd0d55387 100644 --- a/tests/test_transport_commands.py +++ b/tests/test_transport_commands.py @@ -18,6 +18,11 @@ class Ctx: return dict(self.state) def save_state(self, state): self.state = dict(state) + def update_state(self, mutator): + st = dict(self.state) + mutator(st) + self.state = st + return st def send_with_budget(self, chat_id, text, **_kwargs): self.sent.append((chat_id, text)) diff --git a/tests/test_vision.py b/tests/test_vision.py index 9f38b4d56..231a6646d 100644 --- a/tests/test_vision.py +++ b/tests/test_vision.py @@ -3,7 +3,7 @@ import sys import os import unittest -from unittest.mock import MagicMock, patch, PropertyMock +from unittest.mock import MagicMock, patch import pathlib # Add project root to path @@ -266,7 +266,6 @@ class TestVlmQueryTool(unittest.TestCase): def test_vlm_query_with_file_path(self): """vlm_query reads a local PNG from uploads dir and passes base64 to VLM.""" - import tempfile import base64 as b64mod from ouroboros.tools.vision import _vlm_query @@ -341,7 +340,6 @@ class TestVlmQueryTool(unittest.TestCase): def test_vlm_query_path_outside_uploads_rejected(self): """vlm_query rejects paths outside the allowed uploads directory.""" from ouroboros.tools.vision import _vlm_query - import tempfile ctx = self._make_ctx() tmpdir, uploads = self._make_uploads_dir() @@ -368,7 +366,7 @@ class TestVlmQueryTool(unittest.TestCase): # Configure a custom data dir tmpdir, custom_uploads = self._make_uploads_dir() # The "default" home uploads path is different - home_uploads = pathlib.Path("~/Ouroboros/data/uploads").expanduser().resolve() + pathlib.Path("~/Ouroboros/data/uploads").expanduser().resolve() # Create a valid PNG in home_uploads area (mocked via a separate temp dir) home_tmp, _ = self._make_uploads_dir() diff --git a/tests/test_worker_crash_retry.py b/tests/test_worker_crash_retry.py index 2ff518312..c84fc02f1 100644 --- a/tests/test_worker_crash_retry.py +++ b/tests/test_worker_crash_retry.py @@ -10,13 +10,9 @@ Covers: from __future__ import annotations -import json -import pathlib import time -import types -from unittest.mock import MagicMock, patch, call +from unittest.mock import MagicMock, patch -import pytest # --------------------------------------------------------------------------- diff --git a/tests/test_zombie_prevention.py b/tests/test_zombie_prevention.py index 163c23c3a..c58357985 100644 --- a/tests/test_zombie_prevention.py +++ b/tests/test_zombie_prevention.py @@ -8,7 +8,6 @@ Covers: import json from unittest import mock -import pytest # --------------------------------------------------------------------------- diff --git a/web/modules/api_types.js b/web/modules/api_types.js index cfb3965fc..1e50c052a 100644 --- a/web/modules/api_types.js +++ b/web/modules/api_types.js @@ -312,4 +312,4 @@ * @property {boolean=} ok */ -export const GATEWAY_CONTRACT_VERSION = '6.25.0-rc.3'; +export const GATEWAY_CONTRACT_VERSION = '6.26.0-rc.1'; diff --git a/web/modules/chat.js b/web/modules/chat.js index f91fe29d6..3ba5734a4 100644 --- a/web/modules/chat.js +++ b/web/modules/chat.js @@ -1865,6 +1865,7 @@ export function initChat({ ws, state, updateUnreadBadge, openSettingsTab, openDa let text = input.value.trim(); const hasAttachments = pendingAttachments.length > 0; let uploadedAttachments = []; + let attachmentMeta = []; if (!text && !pendingAttachments.length) return; if (pendingAttachments.length) { // Upload immediately before send; offline queueing would orphan files. @@ -1890,6 +1891,7 @@ export function initChat({ ws, state, updateUnreadBadge, openSettingsTab, openDa filename: data.filename || '', path: data.path || '', display_name: data.display_name || stagedItem.display_name, + mime: data.mime || stagedItem.file?.type || '', }); } if (ws.ws?.readyState !== WebSocket.OPEN) throw new Error('Connection closed after upload. Reconnect and try again.'); @@ -1898,6 +1900,14 @@ export function initChat({ ws, state, updateUnreadBadge, openSettingsTab, openDa .map((item) => `[Attached file: ${item.display_name} saved to ${item.path}]`) .join('\n'); text += (text ? '\n\n' : '') + attachmentLines; + // Structured attachment metadata rides the WS frame so the + // gateway can hand image uploads to the model as NATIVE image + // blocks (vision models) instead of only a path label. + attachmentMeta = uploaded.map((item) => ({ + filename: item.filename, + display_name: item.display_name, + mime: item.mime || '', + })); } catch (e) { await cleanupUploadedAttachments(uploaded); showToast('Upload error: ' + e.message, 'error'); @@ -1914,6 +1924,7 @@ export function initChat({ ws, state, updateUnreadBadge, openSettingsTab, openDa content: text, sender_session_id: chatSessionId, force_plan: forcePlan, + ...(attachmentMeta.length ? { attachments: attachmentMeta } : {}), }, hasAttachments ? { queue: false } : undefined); if (hasAttachments && result?.status !== 'sent') { await cleanupUploadedAttachments(uploadedAttachments); diff --git a/web/modules/onboarding_overlay.js b/web/modules/onboarding_overlay.js index 1da95c7f0..190eebe91 100644 --- a/web/modules/onboarding_overlay.js +++ b/web/modules/onboarding_overlay.js @@ -21,12 +21,14 @@ function mountOverlay(html) { } function escapeHtml(value) { - return String(value ?? '').replace(/[&<>"']/g, (ch) => ({ + // Backtick escaped too (defense-in-depth parity with utils.escapeHtmlAttr). + return String(value ?? '').replace(/[&<>"'`]/g, (ch) => ({ '&': '&', '<': '<', '>': '>', '"': '"', "'": ''', + '`': '`', }[ch])); } @@ -55,6 +57,10 @@ function showRestartRequiredOverlay(runtimeMode) { export async function initOnboardingOverlay() { function handleMessage(event) { + // Same-origin only: any web page can postMessage into this window; + // without the origin check a foreign page could dismiss onboarding or + // spoof restart prompts. + if (event.origin !== window.location.origin) return; if (event?.data?.type !== 'ouroboros:onboarding-complete') return; if (event.data.restart_required) { showRestartRequiredOverlay(event.data.runtime_mode); diff --git a/web/modules/onboarding_wizard.js b/web/modules/onboarding_wizard.js index 0ec300eb1..ff8ad434b 100644 --- a/web/modules/onboarding_wizard.js +++ b/web/modules/onboarding_wizard.js @@ -1145,11 +1145,13 @@ headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ mode: runtimeMode }), }); + // Target our own origin explicitly (paired with the receiver's + // origin check) instead of broadcasting to any embedding page. window.parent?.postMessage({ type: 'ouroboros:onboarding-complete', restart_required: Boolean(runtimeResult?.restart_required), runtime_mode: runtimeResult?.runtime_mode || runtimeMode, - }, '*'); + }, window.location.origin); if (!window.parent || window.parent === window) { window.location.replace('/'); } diff --git a/web/package.json b/web/package.json index 6e663f845..d127e1239 100644 --- a/web/package.json +++ b/web/package.json @@ -1,6 +1,6 @@ { "name": "ouroboros-web", - "version": "6.25.0-rc.3", + "version": "6.26.0-rc.1", "private": true, "type": "module", "description": "Ouroboros browser UI package boundary",