ouroboros/docs/architecture/05-supervisor-loop.md

5. Supervisor Loop

server.py::_run_supervisor() owns the pooled scheduling tick and lifecycle lock. Cancellation intent precedes teardown. Its admission, waits, controls and maintenance keep intent, outcome and money separate; unknown is not zero or absence.

Tick and locks

Only state.init_state creates state, on first-boot proof before bridge publication (PERSISTENCE.md). Each tick publishes liveness, rotates logs, checks workers, drains one bounded batch (runtime_limits.py) of the two-lane bus (control before telemetry, so telemetry never delays an answer; event_taxonomy.EventDisposition.lane; mechanics: supervisor/event_observation.py docstring), takes bridge input, projects dirty money, enforces deadlines, dispatches maintenance, admits and assigns work, snapshots, publishes direct roots and ticks consciousness. Backlog skips idle sleep, never owner intake. Three tick failures clear readiness and notify the owner; shutdown errors are not crashes (_SignalStopServer, §9).

supervisor/event_observation.py indexes pending idle inputs across both lanes and holds the outstanding-delivery membership (put_unless_outstanding); event_taxonomy.py names the reducers that change idle inputs. Pending inputs defer only idle grace/reaping, granting no progress or lease (its docstring); heartbeats, telemetry and host narration cannot indefinitely spare a hung task, and dequeue does not certify a failed handler's effects.

state.json money is display, not admission (§10.28): totals persist, usage_projection owns per-root values, and usage_writer_snapshot renders once per dirty tick after intake. Ledger (compaction_epoch, seq) ordering rejects stale renders. Quarantine (integrity_degraded) freezes display until explicit repair; failed writes stay dirty until BUDGET_PROJECTION_RETRY_SEC. The asynchronous OpenRouter comparison never changes admission.

supervisor.queue._queue_lock (Q) owns queue transitions; task_admission reserves identity before effects (§1). Assignment skips registered inline ids, reaping, holds, closed projects, competing writers and exhausted root capacity. Cold external holders retain their project lane (§6); capacity counts idle assignable workers. Outer-lock authority probes use store.acquisition_budget(0); confirmed contention defers unsent work off locks, unknown grants nothing. Only Evolution has mode fences. execution_owner.kind governs carrier policy and pooled completion notifications; _is_direct_chat stays historical. First detached_at (epoch)/detached_by survive queue/result metadata.

Admission and addressing

Headless addressing is fixed at admission (log_addressing.ingress_chat_id). A registered Project's explicit chat_id must match its thread; any other address, hidden included, gets a typed 400, so a Main card never detaches from its Project work. Without a Project, ordinary API tasks default to HIDDEN_CHAT_ID (0). Confirmed browser Publish requests Main with source="web"/WEB_UI_CHAT_ID (caller-declared addressing, not authentication); other non-Project conversation addresses are refused. Active registered Projects admit work into their thread. Main gets one host-stamped completion only for work addressed or bound there, never by registration alone; other work stays in the hidden partition, silent in chats, readable via terminal, --result-json-out, Logs and GET /api/tasks/<id>. Reserved inactive Projects keep their chat valid so the queue gives its typed lifecycle refusal.

Admission names runs without model calls; task_named never uses status as a title. Authored title (ouroboros run --title/top-level contract) fills title/suggested_name; metadata.title/metadata.project_id get 400. First-line fallback fills only suggested_name, keeping later names. Model start_task requires a root title for separate new work. Owner/API/queued events keep fallback; historical routing cards remain resolvable.

supervisor/queue_snapshot.py publishes queue_snapshot.json, queue_dispatch.json and direct_roots.json under Q per changed generation; queue.alive (generation, ts, heartbeats) fences restore. Formats, dispatch fences, direct-root collection, age, Restart-hold and legacy-ts rules: PERSISTENCE.md. queue_observation.py reports missing/mismatched evidence as UNKNOWN: reserve and final-dispatch probes unwind and retry proven-unsent work outside money locks, never certify a stale permissive file. Restore fences RUNNING survivors and taken direct roots (direct_roots.owner_process_ended), snapshot or not, with server_shutdown, except saved work without Stop (restart_retention.recover_saved_running, §6), and cancels their pending children. Scratchpad return needs pending_restart_verify.json.

_claim_worker_launch puts only after its dispatch marks commit in one write, before anything says possible (_commit_launch docstring); snapshots never replay a put. Contention is demand, not refusal: a contended claim stays residual demand, so a contended tick retires no ready worker. Dispatch, hold release and deadline expiry read the loop's governing_continuation (its docstring); saved successors need the frozen source and prior admitted_dispatch_attempt (§6). Proven no-put callers bind source/attempt/claim to terminalization, rechecked by the reaper; terminal rows stay terminal, and accepted/sealed roots' descendants remain fenced except exact pauses. task_status.execution_observation keeps fresh activity apart from lifecycle; only pooled work admits worker-boot orphan inference.

supervisor/workspace_preparation.py moves Git/copy/index off the event thread: token/generation precede FIFO entry, contention waits outside Q, admission rechecks identity/controls/scope and copies use preparation-time bytes; a rolled-back scheduled receipt retries only after strict absence and exact compensation (its module docstring). Neither direct-child-count pre-check (the worker's schedule_subagent, the supervisor's early check) refuses an unknown count; the interlocked re-check decides (control_delegation.schedule_delegation_refusal docstring). Startup settles proved preparation-only loss as never admitted and keeps custody/resources for uncertain admission (§10 invariant 19). Entry: events_schedule_task, task_admission.

Schedules and follow-ups

supervisor/queue_schedules.py alone writes state/scheduled_tasks.json, every mutation inside schedule_transaction (Q, then the sidecar lock). load_schedule_store reads only absence as empty; an unreadable table raises ScheduleStoreUnreadable, refuses mutation and reads as unknown.

schedule_occurrence.py admits at most one root per due occurrence, enqueued with a frozen receipt; a typed hold is a wait, not a failed root; the same token and task id re-prepare after restart, and restore follows the receipt (positive possible held, a proven-unsent first attempt unsent, unreadable or foreign results and receipt-less admissions unknown: its module docstring). Edits/disable govern only unstarted work; deletion returns delete_deferred for owed or unknown work. A waiting cron row keeps one overdue occurrence, then advances. A kind:"notify" row is a note, not an occurrence: the pass consumes it and schedule_notes.py shows the mind's words as one signed reminder System row; an unconfirmed show becomes last_error, never retried.

followup_policy.py binds explicit related/independent relationships; host provenance (continuation_of, origin) grants neither relationship nor billing authority; related work keeps its origin's group cap, source and deadline. Lifecycle controls govern future work (running_or_queued); workers hold in flight on missing, unreadable or stale (_QUEUE_SNAPSHOT_MAX_AGE_SEC) snapshot evidence; Restart and Stop persist restrictions before marker consumption, and a Stop's followup_stop lands on the original result even if completion wins (module docstring). stop_action_id names the ingress action apart from custody request_id (exact retries rejoin, altered arguments refuse, omitted ids claim no replay: new_stop_fields docstring). Restore with expected_hold_id releases only that hold; generic Restore can re-enable a schedule and cancel pending deletion. schedule_mutation intent/outcome rows share one operation_id (queue_schedules._audit_schedule_mutation docstring); outcome failure reports changed_audit_incomplete without unaudited rollback. manage_schedules and POST /api/schedules/{id}/action share this seam with explicit action/reason, no text inference (owner edits: supervisor/schedule_lifecycle.py). Final enqueue and worker put share followup_policy.scheduled_start (lock order: its module docstring); preparation and provider waits stay outside.

Completion, owner waits and restarts

Pooled completion separates a file-save attempt from publishable terminal truth: workers prepare terminal files (headless.prepare_terminal_task_files) before task_done; the host publishes only once headless.terminal_task_files_ready confirms them current, and unpublishable completions stay RUNNING under enqueue_terminal_file_recovery on the health cadence (never a false Done or replayed model work). Only confirmed absence of a terminal source reaches the lifecycle-fault owner, which marks execution infra_failed while the authored answer, review and cost survive.

Answer first: agent_task_pipeline.emit_task_results owes a root's final frame in the durable outbox and puts it on the live bus BEFORE cost reconstruction, review-evidence/input capture and _store_task_result, so the answer never waits for bookkeeping; task_done stays buffered until after the store (pre-/post-store carriers and the task_finalization_timing row, basis: post_answer_store: its and finalization_timing.emit_post_answer_timing docstrings). Timeout reaping and dead-worker recovery adopt an owed, exhausted, delivered or unreadable registration; direct boot-fence custody adopts a readable one (working_checkpoint.adopt_registered_answer), as failed/infra_failed/finalization_incomplete with a host notice, not a replayed prompt.

worker_owner_wait keeps task, worker and browser RUNNING; only a durable park lends capacity. Contended parks retry on that row; pre-ACK Stop/deadline/ceiling bind. Recorded clocks measure pre-ACK sleep; native restart carries one open interval, folded once by worker and pool. Capped Resume returns the same attempt, never replay. Main/Project waits survive planned restart and protect unread input; mail wakes, only owner answers answer. Settled Project results reject steering during post-work (owner_mailbox.mailbox_drain_ended). Direct waits (owner_wait.direct_owner_wait) lend no pooled slot or cold restart.

Crashes, timeouts, restart returns and same-server update aborts follow §6 Saved working state, minus OS browser sessions. Managed updates prepare handoffs before pool stop (preserve_running_task_ids for parked ids): waits requeue; failed preparation blocks update. Manual rollback prepares no wait handoff. Cold continuation restores economics but rereads cost-ceiling authority (task_pacing.restore_cost_ceiling); calendar deadlines and owner-wait time stay unchanged.

Cancellation

Cancellation is intent-then-custody (§10 invariants 14-15): intent and outcome are separate fields, since one field carrying both wedges a task forever:

1. Intent. Every cancel ingress (cancel_task tool, HTTP single/cascade, evolution stop, project deletion, cascade sweep, boot recovery of cancel_requested files) writes one durable row via ouroboros/cancel_intents.request_cancel into locked state/cancel_intents.json; a failed write tears nothing down (tool CANCEL_INTENT_WRITE_FAILED, HTTP 503). Owner evolution stop is sticky: the campaign stays open under evolution_owner_stopped until an owner start clears it.
2. Scope. Recorded scope only widens (single to cascade; Stop-now cannot shrink a cascade). Reaping is no cancel ingress: its custody survives health hand-back until replay settles. Stop needs explicit intent; a replay in flight keeps it pending.
3. Claim. supervisor.task_lifecycle.cancel_task_custody, the one settle owner, takes the exclusive claim (§10 invariant 14); racing custodies never double-settle; the cancel_task event only wakes the intent. A live direct-chat turn stops through the same custody (supervisor/worker_chat_lane.py writes one typed finalize_now; the loop ends at its next round boundary without another model call); Stop during post-task synthesis discloses skipped paid stages as post_task_stop_reason. A still-live turn releases the claim for the sweep, never publishing a fabricated cancelled row.
4. Kill and re-check. Custody confirms process death, then rereads the child's settled result; natural completion wins (§10).
5. Reconcile, capture, settle. Open delegated runs are re-audited from durable custody rows; workspace artifacts come from the real tree (a failed or unrunnable capture is failed, never missing); the result carries reconstructed-or-unknown cost, never a fabricated $0, and only here is parent_decision stamped; cancel_publication keeps cancel_origin provenance through every read (its docstring). HTTP proves transport, never a personal owner.
6. Owe, then publish. The owner's terminal answer is owed in the durable outbox before the terminal row (after the claim re-check; pending-deadline expiry owes inside its locked section the same way), before the intent settles and before task_done publishes (a completion that wins withdraws that salvage); a crash between settle and send replays it. A cascade delivers one root message with a children digest (delivery id cascade:<root_tid>:<request_id>). The Done card appears strictly after the answer: ouroboros/terminal_projection.py publishes a root's card only once the outbox owes no terminal message of any family (final:, cancel:<tid>:<request_id>, cascade:), and an unknown or exhausted delivery is shown on it, never hidden (holds and settle-on-ack: _delivery_disposition, events_chat_delivery._handoff_delivered_review docstrings).
7. Watchdog. The tick's off-drain sweep (server_maintenance.py) has sweep_cancel_intents re-feed unclaimed/abandoned intents into custody (cascades as cascades); a lost control event or custody dead mid-teardown cannot wedge cancellation.

Effective results show typed cancel_state: "pending" with cancel_reason until settlement (UI: §3 Stop and cancel controls). Steering writes (tell_task, mailbox follow-ups, forward_to_worker) are refused typed during a pending cancel, keeping the single-turn owner-stop rail safe; restore and pre-assignment read the projection under Q, so a cancelled pending task never starts. task_done asserts a settled outcome validated against the durable result (events_task_done docstrings); a mismatch is a lifecycle fault, left to custody while cancellation is pending.

supervisor/terminal_delivery.py owns the bounded state/terminal_deliveries.json outbox: debt before enqueue, boot/tick replay, exact delivery_id dedupe. Worker/post-task producers use the injected bus (§Event bus). Refusal keeps debt; queued, already-delivered, queued-unregistered and unavailable remain distinct. Replay neither copies nor spends attempts on queued/in-flight answers, including the worker's original (put_unless_outstanding; accounting: _put_unless_outstanding). Exhaustion or capacity eviction retains payload and preserved copy in exhausted until that id's receipt (_registry_document, _disclose_exhausted_delivery). Card dedupe scans only the live chat and archives rotated since readiness (terminal_projection._already_in_chat). Canonical Main readiness retains its frozen event/retry clock until strict publication receipt (§3); eviction cannot discharge or exhaust it. Cancel/non-retry reap sends lineage-routed unreviewed salvage; retryable reap sends nothing. Handler receipt proves neither display nor reading (§10).

Stop policy and hurry

Stop policy is an axis on the same durable intent, independent of cascade scope. An omitted or empty-body cancellation is immediate: 200, 404 when nothing is left, or 202 pending (CLIENT_CONTRACT §8). stop_policy=finalize_then_cancel answers 202 with the intent open and runs one bounded finalization episode: descendants settle first, then the root's finalize_now allows at most one tool-less model turn, terminalizing under the honest owner reason, not a false deadline (supervisor/owner_stop.py docstring). OWNER_STOP_OUTER_CAP_SEC bounds grace; explicit deadline and absolute ceiling stay independent. Transitions are monotonic (immediate hardens pending graceful; graceful never softens accepted immediate); Panic bypasses both.

Owner "hurry" is a typed task-local kind=hurry owner-mailbox control (ouroboros/owner_hurry.py docstring; gateway/task_hurry.py): it skips the next otherwise-eligible acceptance panel with a typed reason, zeroes remaining improvement passes and makes that task's force-plan projection advisory; never a chat message, settings mutation or weakening of the P3 commit/review gates; attempt-scoped (retry_reset strips it on every same-id requeue).

Event bus, timeouts and worker readiness

The server owns one process-lifetime manager-backed bus (lanes: §Tick and locks); worker generations and direct actors receive its producer endpoint. Respawn never rotates it, producer lookup never creates one (absent transport creates no child Manager), producer exit never shuts down its owner, and a force-killed producer cannot corrupt another's feeder frame. A pooled task's endpoint (log_addressing.task_event_queue) stamps chat and lineage by value, so telemetry handled after its task left RUNNING stays addressed (TurnEventQueue docstring). Workers write their own checkpoint rows before the one live frame (loop_messages._emit_checkpoint_event); the nested branch persists only task_start_settings_reload_failed. utils.append_jsonl streams only runtime logs/*.jsonl, minus types with a dedicated live sender (WORKER_LOG_SINK_SUPPRESSED_TYPES, server superset SERVER_LOG_SINK_SUPPRESSED_TYPES): one live frame per recorded event, one durable llm_api_error per call failure.

Heartbeat proves liveness; progress and model-usage events prove task advancement. Progress spares orchestrators' idle rail, never deadline, Stop, budget or execution ceiling (solving/finalization, not settled post-work; §6). queue_timeouts also expires queued deadlines undispatched. Timeout freezes under Q, marks reaping, and delegates kill/join/salvage/retry/respawn to task_reaper; live descendants forbid blind orchestrator replay (a duplicate tree), and unconfirmed death keeps the reservation, emits task_reaper_wedged with a Restart hint and permits no terminal/retry/replacement rather than race a live process. Typed incidents retain TIMEOUT_TERMINAL_REASONS identity.

One Python worker starts; more come on demand and quiet surplus is retired, with no idle timer (owner choice: idle memory beats burst startup); explicit positive counts and MAX stay. Boot is cheap: startup verification runs once per server generation and code version (state/startup_verification.json; _log_worker_boot_once docstring; worker_boot stays per process) and a worker caches bytecode in a prefix inside its data dir (worker_process.write_bytecode_to_prefix_only). worker_assignment forecasts residual demand with dispatch's side-effecting chooser, claiming and consuming nothing; worker_pool_lifecycle grows only that deficit and keeps one ready spare at quiescence, booting slots forecast capacity and unresolved reservations stay charged (reconcile_worker_capacity docstring).

Assignment/free counts require live current PID/birth-bound worker_ready: births are compared only when both sides measured one, else the pid (_births_agree, _record_worker_processes docstrings). SHA equality proves neither loaded code nor review. worker_starting grants one bounded extension (runtime_limits.py); watcher errors use bounded replace/park. Per-slot exhaustion leaves unused MAX headroom. All-spent capacity closes pooled ingress (/review too); disable_exhausted_worker_pool fails unstarted rows with a Restart hint, sparing chat/control/boot/update recovery.

Surplus retirement and cap-full warm Resume share a quiescence token probe; a worker answers once its local reviews, tool futures with late callbacks and abandoned senders settle (worker_process.local_work_settled; review_operation._LIVE counts wait.close and final pointer publication, unknown defers, no history scans). Retirement reserves the slot, kills with kept services through the off-loop task_reaper and frees capacity only on confirmed death; the lifecycle serializer guards membership, never a kill or join, so assignment and health keep their tick (request_worker_quiescence, retire_quiescent_worker, reap_idle_worker docstrings). No idle timer, rotation or setting; intentional retirement is not a crash.

worker_health.recover_confirmed_dead_worker reserves exact custody under Q for the reaper's confirmed_dead_worker: saved terminal bytes (task_exception after handle_task failure included) win over signal death, live exact pauses re-park (§6), unknown/incomplete publication stays TerminalFileRecoveryPending; only confirmed absence reaches crash policy, where eligible crashes retry within QUEUE_MAX_RETRIES and budget continuations forbid ordinary retry (its docstring; storms: the module docstring). After process custody, startup recovers addressed pending_drives, nonterminal/synthesis obligations and saved answers before orphan healing.

Maintenance and custody

server_maintenance.py gives schedules, cancel, delivery, pause-notice, usage, post-task recovery, custody and reconcile separate single-flight latches; domain claims/receipts, not pass order, arbitrate. bind_body_lifespan starts after store publication; replacement joins closing jobs off-loop; cold holders enter orphan-exclusion before recovery (§6). Generation closure/returning Stop admits nothing and proves no exit (§10.24). An invalid_source refusal is retried only when its watched file, rows or generation change (source_retention._unavailable_basis); cost reads treat busy as no observation and prove absence in one statement. Delegated sweeps leave review-owned invocations to panels (delegate_terminal); no-file captures mean no changes only after success, otherwise nothing captured (delegate_directory.capture_directory_result). Reaping checks PID/birth/command/task/generation, never command classes or another install. Watchdogs journal stalls, report wedged actors and suggest Restart; they kill no thread.

Packer tick reserves its existing thread: stop-aware, archive-first discovery exits at the first changed stamp. Contention defers; quiet nonempty roots scan without spawning. Discovery stays in the exit census, releasing reconcile without hiding readers. The child retains its own fresh PackSweep baseline/refresh/journal and unlink authority.

Cooperative project checkpointing (supervisor/events_coop_checkpoint.py) runs on two equivalent quiescence triggers, a root settling without live descendants and the last child settling under an already-terminal root, because a root-scope budget stop terminalizes the root before its children (its handler docstrings). The bounded git chain revalidates quiescence under Q just before mutation. Only host-minted project roots qualify: owner-attached folders are never auto-committed, credential-shaped files stay excluded and disclosed, and each material success, skip or error gets a durable receipt.

Bridge intake and commands

Bridge intake keeps message identity, transport and durable acceptance; failed handlers requeue only their unprocessed in-memory tail. Accepted rows never auto-replay. Ordinary room input reaches its sole live reader; a warm question wait counts, queued or held work does not. Explicit tell writes owed mail without resuming. Question answers try the reading asker, then proven admitted successors, rechecking each under its admission lock, then named fresh-head ingress at the saved publication address. Mail stays owed until saved-read acknowledgement. Panic uses its emergency ingress.

The bridge recognizes /panic, /restart, /review, /evolve [on|off], /bg [start|stop|status] and /status; other text enters ordinary agent routing. External transports invoke them only with positive owner identity and a transport-specific owner-chat binding known current; unknown refuses and registers nobody; Panic keeps its positively observed immutable binding until owner Reset closes it (PanicIngress docstring). Commands reuse runtime-mode, queue, cancellation and typed-result authority, not parallel control paths. Archive readers keep rotated runtime logs' retained timelines (§1).

Confirmed budget/owner pauses register pause_notices with the confirming result update (pause_notices.py): one obligation set, a saved System chat row as the receipt, a failed append still owed, older episodes retained by Resume and newer pauses (its docstring). Consciousness reports paused/pausing or unknown, never guesses settlement.