mirror of
https://github.com/razzant/ouroboros.git
synced 2026-10-21 14:12:28 +00:00
Merge pull request #1643 from razzant/sprint3-known-spend
fix: use known spend limits and keep accounting diagnostics quiet
This commit is contained in:
commit
b32aae0947
127 changed files with 3537 additions and 1056 deletions
|
|
@ -47,7 +47,7 @@ Rows may import columns (`[graph].allowed`). `·` = forbidden direction.
|
|||
| **D09** | ✓ | · | · | · | · | ✓ | · | ✓ | · | · | · | · | · | · | · | ✓ | · | ✓ | · | · |
|
||||
| **D10** | · | · | · | ✓ | ✓ | ✓ | · | ✓ | · | · | · | ✓ | ✓ | · | · | · | · | ✓ | ✓ | · |
|
||||
| **D11** | ✓ | ✓ | ✓ | ✓ | ✓ | · | ✓ | ✓ | ✓ | ✓ | · | ✓ | ✓ | ✓ | · | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| **D12** | ✓ | ✓ | ✓ | · | · | · | ✓ | · | · | · | ✓ | · | ✓ | · | · | · | · | ✓ | · | · |
|
||||
| **D12** | · | ✓ | ✓ | · | · | · | ✓ | · | · | · | ✓ | · | ✓ | · | · | · | · | ✓ | · | · |
|
||||
| **D13** | ✓ | ✓ | · | ✓ | · | · | · | · | · | · | · | ✓ | · | · | · | · | ✓ | ✓ | · | · |
|
||||
| **D14** | · | ✓ | · | ✓ | ✓ | ✓ | · | · | ✓ | ✓ | ✓ | ✓ | ✓ | · | · | ✓ | · | ✓ | ✓ | · |
|
||||
| **D15** | ✓ | · | ✓ | ✓ | · | · | · | · | · | · | · | ✓ | · | · | · | ✓ | · | ✓ | ✓ | · |
|
||||
|
|
@ -65,7 +65,7 @@ Rows may import columns (`[graph].allowed`). `·` = forbidden direction.
|
|||
|
||||
## Hidden coupling (classified out of the strict graph)
|
||||
|
||||
- lazy-only cross-domain pairs: **117**
|
||||
- lazy-only cross-domain pairs: **118**
|
||||
- D01->D08
|
||||
- D01->D10
|
||||
- D01->D11
|
||||
|
|
@ -124,6 +124,7 @@ Rows may import columns (`[graph].allowed`). `·` = forbidden direction.
|
|||
- D10->D17
|
||||
- D11->D06
|
||||
- D11->D15
|
||||
- D12->D01
|
||||
- D12->D05
|
||||
- D12->D06
|
||||
- D12->D10
|
||||
|
|
|
|||
|
|
@ -144,7 +144,7 @@ scanned data-relative path to be covered by a row here (count-anchored both ways
|
|||
| `logs/containment_faults.jsonl` | `ouroboros/delegate_custody.py` (mirrored to events.jsonl) | `type` ∈ CONTAINMENT_FAULT/RESOLVED joined on run_id | unbounded BY DESIGN — read whole so an open fault never ages out — accepted | health invariant degrades to the 4 MB events tail scan (the regression this file fixed) |
|
||||
| `logs/chat_annotations.jsonl` | `ouroboros/project_dialogue.py` (`append_jsonl` under the shared sidecar append lock) | `type: chat_annotation` keyed by `(client_message_id, routing_token)`; latest row per key wins (an owner message keeps one row per routing act; task-authored acts ride synthetic `agent-steer:<token>` ids) | self-compacting at 800 KB under the append lock: the latest row of every message still present in the chat chain (live `chat.jsonl` + the 3 newest `archive/chat_*.jsonl`) is rewritten, the rest is DROPPED — presentation state, so it is not rotated into `archive/` | annotation cards fall back to their plain chat rows; the one named exception (#198) also loses the durable picker decision-card token, so a pending manual routing choice must be made again |
|
||||
| `logs/tasks/task_<id>.txt` | `ouroboros/utils.py` log sanitization (`write_text`, best-effort) | raw oversized task text, no envelope; the log row keeps `text_full_path` | unbounded: no retention sweep names `logs/tasks/` | the truncated text in the log row stays; only the spilled full text of oversized task prompts is lost |
|
||||
| `logs/agent_stdout.log` | `launcher.py` pipe-copy thread | unstructured text | bounded ~8 MB (2 MB × `.1..3` backups, rotated by the copy thread) | pre-logging crash output lost; nothing parses it |
|
||||
| `logs/agent_stdout.log` | `launcher.py` pipe-copy thread (`ouroboros/process_logging.py` `copy_capped_output`) | the server's raw stdout/stderr bytes, plus host `[output gap: …]` lines | what the copier writes stays ≤ 2 MiB per file × `.1..3` (~8 MiB): it rotates before a write would pass the cap; while open, rotation or write fails it keeps draining the pipe, drops and counts chunks (known not written apart from uncertain), never truncates the live file or appends past the cap, marks the gap in the file on recovery and reports to `launcher.log`; an inherited oversized file rotates whole into `.1` and ages out, uncleaned | pre-logging crash output lost; nothing parses it |
|
||||
| `logs/server.log` (+`.1..3`), `logs/launcher.log` | stdlib `RotatingFileHandler` with secret-redacting filter: `server.log` by `ouroboros/process_logging.py` from `server.main()` only, `launcher.log` by `launcher.py`; each pool worker configures a stream handler only at the top of `worker_main` (one rotator per file; `OUROBOROS_WORKER_START_METHOD=fork` inherits the parent's handlers unchanged) | text | bounded ~8 MB (2 MB × 4) — the model citizen | stdlib log history lost; nothing parses it |
|
||||
|
||||
## 6. `memory/` (Ouroboros cognition — operator read-only)
|
||||
|
|
|
|||
|
|
@ -84,7 +84,10 @@ window rows through the `(category, ts_last)` and root indexes. Terminal
|
|||
maintenance takes recovery candidates from the open-set index and projection
|
||||
candidates from `Txn.dirty_owners()`. After an addressed projection succeeds or
|
||||
is equal, `ack_dirty_owner(owner_id, revision)` removes only that revision; a
|
||||
new receipt, failure, missing result or live ownership keeps the debt.
|
||||
new receipt, failure, missing result or live ownership keeps the debt. A
|
||||
missing result is one `stat` before the ownership read: its exact revision
|
||||
stays, so a result that lands after the last receipt (child copyback, a body
|
||||
restored from quarantine) is projected on a later pass without a new receipt.
|
||||
|
||||
## 5. Lock tiers
|
||||
|
||||
|
|
@ -186,3 +189,6 @@ orphan accusation.
|
|||
name lock is per host).
|
||||
- Result-only edits do not dirty money: ordinary receipts maintain projection debt;
|
||||
a result-only accounting repair belongs to explicit repair/import.
|
||||
- An owner whose result never appears stays in `dirty_owners` and costs one
|
||||
`stat` per maintenance pass; absence is never acknowledged, so only a
|
||||
projection (or that explicit repair) retires it.
|
||||
|
|
|
|||
|
|
@ -144,7 +144,7 @@ server.py (Starlette+uvicorn) ← HTTP + WebSocket on configurable host:port (de
|
|||
├── deadline_utils.py ← Shared deadline parsing/remaining-time helpers + the transport-vs-logical wait seam for loop milestones and process-tool/review timeouts
|
||||
├── observability.py, source_retention.py ← Private call/source history: redaction, gzip CAS, manifests, exact iterative retention (§10)
|
||||
├── finalization_timing.py ← Live final-event phase fields and task_finalization_timing (§6)
|
||||
├── process_logging.py ← Per-process logging bootstrap (server `main()`, each worker after its fork preamble): redacted local handlers, the server as sole `server.log` writer, uncaught exceptions into `logging`
|
||||
├── process_logging.py ← Per-process logging bootstrap (server `main()`, each worker after its fork preamble): redacted local handlers, the server as sole `server.log` writer, uncaught exceptions into `logging`; the launcher's byte-capped, storage-error-resilient server-pipe copy into `agent_stdout.log` (stdlib-only)
|
||||
├── model_send_seal.py ← The runtime invariant `model-visible ⟺ logged` for `model_send`: a reconstruction mismatch is a typed durable fact, and the call is NOT blocked — dispatch authority stays with the pre-existing in-memory identity re-check
|
||||
├── cancel_intents.py ← Durable cancel-intent projection: locked `state/cancel_intents.json` of ACTIVE intents (claim owner/pid + claim GENERATION fencing every mutation, `scope` single-vs-cascade) + forensic `cancel_intent` ledger rows; the ONE ingress `request_cancel`; strict fail-closed reads (`CancelIntentProjectionCorrupt`; a malformed row is disclosed once per row content, so the ~20 s watchdog cannot repeat it forever); owns `claim_is_abandoned` and `allow_settled_target` (§5; §10 invariants 14–15)
|
||||
├── owner_hurry.py ← Owner "hurry": a typed TASK-LOCAL acceleration latch, never a chat message; its durable `owner_hurry` projection is written by `update_json_locked` on its own keys only — never `write_task_result`, whose status-regression guard could drop concurrent terminal fields; effects `acceptance_skip_applied`, zero improvement passes via `effective_budget_profile`, advisory force-plan; dies with the attempt (`retry_reset`; `not_applied_before_terminal`) (§5)
|
||||
|
|
@ -169,7 +169,7 @@ server.py (Starlette+uvicorn) ← HTTP + WebSocket on configurable host:port (de
|
|||
├── pricing.py ← Exact-route best-effort provider-catalog lookup with nullable estimates; no static model tariffs (they go stale) and not the monetary ledger (§6 Budget tracking)
|
||||
├── usage_accounting.py ← Physical attempts: reserved→dispatched→settled|unresolved, reserved→released; global/root/group admission, candidate/manifest, raw `local_answer_owner_pid` (§6)
|
||||
├── _usage_response.py ← Accounting usage normalizer; adapters also read raw usage (§6)
|
||||
├── usage_admission.py ← Whole-work group binding/admission and review-wave fit (§6)
|
||||
├── usage_admission.py ← Whole-work group binding/admission and review-wave admission (§6)
|
||||
├── _usage_rows.py ← The one reducer (per-row summary deltas), limits/integrity, call counts, breakdowns and Skill Review projections
|
||||
├── _usage_money.py ← Precision-60 Decimal cash; six-place half-even admission; raw literals retained
|
||||
├── _usage_wait.py ← Owned pre-send lock slices and joined async bridge; existing controls/custody
|
||||
|
|
|
|||
|
|
@ -323,7 +323,7 @@ Logs merges live WebSocket log frames with bounded REST backfill from the events
|
|||
|
||||
Activity joins queue entries with the same refresh's `/api/state` census. Queue identity keeps title/runtime/budget controls; warm sleep says sleeping, unknown stays unconfirmed; census-only turns show kind/phase/elapsed time and shared controls. Partial census cannot prove idleness. Independent sections show the consciousness alarm (enabled/status/autonomy, next/last wake/outcome, rolling-24h allowance and running tasks) and schedules; a failed read keeps the other sections. Authoritative controls: cascade cancellation, consciousness start/stop and schedule enable/disable/delete/restore. Standing schedules stay listed (`waiting: <reason>` for held occurrences, `deletion pending` too, any stored `last error`; a `kind:"notify"` note is named by its own words); fired one-shots and suppressed skill rows share a `History & suppressed` disclosure whose open state survives refresh. Consumed rows cannot Enable; pending deletion may offer explicit Cancel deletion, which exact hold release keeps; Restore re-evaluates suppressed skills, never enabling them. Every schedule button, Delete included, sends action/reason to the agent tool's audited `POST /api/schedules/{id}/action`, not a stale record rewrite. One reply helper distinguishes unchanged, changed with lost audit outcome, and changed with admitted work still running or unobservable. An unavailable queue never renders empty.
|
||||
|
||||
Costs is a projection of the physical-attempt ledger distinguishing confirmed, reserved, unresolved upper-bound, unknown/unmetered usage, open rows and finality; unavailable data renders as unavailable, not `$0`. Breakdowns by model, key, model category and task category are views over the same ledger. Total budget can hot-apply; the per-task value is a hard cost cap over the whole root tree of the next task, not an own-task soft warning (§7 Default settings), and raising a cap does not resume work that already finalized or paused.
|
||||
Costs is a projection of the physical-attempt ledger: known spend (confirmed + estimated, both shown) against the limit, then reserved and unresolved upper-bound holds that are not counted as spending, unknown/unmetered usage, open rows and finality; unavailable data renders as unavailable, not `$0`. Breakdowns by model, key, model category and task category are views over the same ledger. Total budget can hot-apply; the per-task value is a hard cost cap over the whole root tree of the next task, not an own-task soft warning (§7 Default settings), and raising a cap does not resume work that already finalized or paused.
|
||||
|
||||
Evolution shows the current evolution state and the consciousness alarm clock, campaign objective/progress, queue/failure/budget information and durable history. Starting a campaign uses the shared input dialog: cancellation starts nothing, confirmed empty input selects the backend's default autonomous objective, entered text becomes the objective. Light runtime mode disables self-modifying campaigns rather than presenting a control the backend will refuse.
|
||||
|
||||
|
|
|
|||
|
|
@ -42,7 +42,7 @@ Pooled completion separates a finished file-save attempt from publishable termin
|
|||
|
||||
A required owner wait keeps the task RUNNING, its worker and browser after a completed-tool checkpoint. Capacity is lent; a capped grant resumes the same attempt/start without replay, consuming continuation authority before dispatch. A new owner-bound wait stamps its own `parked_at` beside, never instead of, the task start; the whole wait row rides every later write, so a planned restart keeps it. Any mail wakes, only an owner answer answers; Stop, deadline and ceiling still bind. A settled Project result is no longer steerable even if post-work holds RUNNING: fast mail and typed steer refuse it and a quiz answer takes the late-answer path (one actor-drive settlement predicate, `owner_mailbox.mailbox_drain_ended`), so cleanup cannot eat unread input. Ordinary Main/Project waits stay in-process (`owner_wait.direct_owner_wait`), lend no pooled slot and resume the live stack/browser on addressed text or quiz; a saved source grants no cold restart.
|
||||
|
||||
A crash or automatic timeout retry keeps the current-attempt checkpoint as evidence, never blindly replaying; cold continuation exists only for a confirmed planned restart, which cannot preserve an OS browser session. An owner-requested MANAGED UPDATE is such a restart: its writer fence prepares the handoff before stopping the pool and passes the parked ids as `preserve_running_task_ids`, so a wait is requeued instead of interrupted, and preparation failure blocks the update rather than terminalizing the wait; the re-exec seam arms the transaction only in the phases `_safe_restart_serialized` allows (launcher mode observes exit code 42), no-resume flags suppress it, and an aborted update's leftover record never authorizes a later manual Restart or rollback. A manual ROLLBACK (to an older runtime) deliberately parks nothing. Cold continuation resumes saved economics (original CostCeiling, hard clocks, saved-model ContextFit, TaskModelWait choices, the saved round's pending budget decision); calendar deadlines and owner-wait time stay unchanged.
|
||||
A crash or automatic timeout retry keeps the current-attempt checkpoint as evidence, never blindly replaying; cold continuation exists only for a confirmed planned restart, which cannot preserve an OS browser session. An owner-requested MANAGED UPDATE is such a restart: its writer fence prepares the handoff before stopping the pool and passes the parked ids as `preserve_running_task_ids`, so a wait is requeued instead of interrupted, and preparation failure blocks the update rather than terminalizing the wait; the re-exec seam arms the transaction only in the phases `_safe_restart_serialized` allows (launcher mode observes exit code 42), no-resume flags suppress it, and an aborted update's leftover record never authorizes a later manual Restart or rollback. A manual ROLLBACK (to an older runtime) deliberately parks nothing. Cold continuation resumes saved economics (the CostCeiling under its re-read authority, `task_pacing.restore_cost_ceiling`: an explicit profile's saved point, a producer allowance at its actual value, a removed default dropped, an unreadable root's number kept as `legacy_policy_unverified`; hard clocks, saved-model ContextFit, TaskModelWait choices, the saved round's pending budget decision); calendar deadlines and owner-wait time stay unchanged.
|
||||
|
||||
Cancellation is intent-then-custody; intent and outcome are separate fields, because one field carrying both wedges a task forever. The skeleton:
|
||||
|
||||
|
|
|
|||
|
|
@ -39,7 +39,7 @@ Task acceptance grades evidence and advises useful authorized work separately fr
|
|||
|
||||
`acceptance_history.py` freezes zero-physical root/retry skips: answer, criteria, owner inputs and retained HOST refs. Copyback follows owned carriers, never agent data/arguments/quotes. No urgent bulk capture/closure; uncaptured bytes stay gaps, never current-file reconstructions.
|
||||
|
||||
`task_acceptance_review(late_review=...)` needs a NEW owner chat/quiz/mailbox `owner_source`; Main may target any Project. `amend_cap` (bare-cap default) only sets an absolute original-root cap: no review; Pause/fences/spend stay. `review` runs the full `acceptance_late` triad on frozen terminal roots/retries, even mode-off, after any cap. Original-root/group spend and holds (successors included), live global caps apply; finite/unlimited/unknown keep provenance. Caller/target/root Stop, Panic, prohibitions, calendar bounds bind. Hurry/Wrapup block unstarted automatic work, not later explicit authority. Debt-paid identity makes paid/unknown work collect-only.
|
||||
`task_acceptance_review(late_review=...)` needs a NEW owner chat/quiz/mailbox `owner_source`; Main may target any Project. `amend_cap` (bare-cap default) only sets an absolute original-root cap: no review; Pause/fences/spend stay. `review` runs the full `acceptance_late` triad on frozen terminal roots/retries, even mode-off, after any cap. Original-root/group known spend (successors included; holds disclosed), live global caps apply; finite/unlimited/unknown keep provenance. Caller/target/root Stop, Panic, prohibitions, calendar bounds bind. Hurry/Wrapup block unstarted automatic work, not later explicit authority. Debt-paid identity makes paid/unknown work collect-only.
|
||||
|
||||
`events_chat_delivery`/file-ready `task_done` hand off to `review_operation.prepare_historical_operation`. NEW automatic panels need calendar headroom above the acceptance floor, no author reserve; controls recheck after writer drain and before purchase. Supervisor maps prove local ownership; workers read pooled/direct projections and direct postwork checkpoints, conservatively on unknown. The preparation excludes itself. Sources close before freeze; paid/unknown replays stay collection-only. Cold settlement keeps the proven recipient.
|
||||
|
||||
|
|
@ -79,7 +79,7 @@ Whole first-send sizing includes scope/checklist/wrappers/schemas. Oversize beco
|
|||
|
||||
Child calls select ≤1 configured `reviewer_slot_id` (implicit if sole), retaining delivery/money/deadline, not limiting reasoning rounds. No child cycle ledger or restart-dedupe proof. Root `off` disables automatic review; explicit calls use the full configured advisory panel.
|
||||
|
||||
Wave admission prices one API first send per row (sessions unpriced), no rounds multiplier/repricing. Each send checks the wallet; an unaffordable floor is `review_wave_budget_insufficient`. Packet repair grants no extra retrieving episode/session.
|
||||
Wave admission prices one API first send per row (sessions unpriced), no rounds multiplier/repricing, as disclosure: only known spend already at a limit is `review_wave_budget_insufficient`. Each send checks the wallet. Packet repair grants no extra retrieving episode/session.
|
||||
|
||||
Paid identity (`acceptance_paid_identity`) binds that semantic subject plus substantive nonempty obligation dispositions; forensic source hashes and ingress counters alone buy no panel. A resubmit with the same paid identity replays its recorded verdict free — a clean replay can authorize acceptance, a non-clean one keeps its verdict and `identical_acceptance_refused` — and real criteria or evidence changes need no cosmetic edit.
|
||||
|
||||
|
|
@ -95,7 +95,7 @@ Every outcome permits author inspection, correction or rebuttal. `loop_acceptanc
|
|||
|
||||
`task_pacing.review_launch_allowed` admits a new critic only with cycle capacity, the wallet admission above, and MORE than `OUROBOROS_ACCEPTANCE_REVIEW_EST_SEC` (minimum 200 s) above finalization reserve. This floor predicts no duration and applies once, at loop admission before evidence build. `improvement_pass_allowed` instead uses ordinary time above reserve, budget, cancellation and round limits, with no reviewer floor/multiplier. Their refusals are `review_skipped_deadline_reserve` and `improvement_window_inside_reserve` respectively.
|
||||
|
||||
At first physical dispatch, `review_dispatch.task_acceptance_paid_dispatch_stamp` claims the locked `task_results.task_acceptance_review_accounting` wallet, checking only cancellation/cycles. A claim without a recoverable terminal host run is UNKNOWN, not resend authority. Launched review retains deadline/task-ceiling and per-send money bounds; deadline cuts are DEGRADED, not free skips. Evidence build can consume the admitted margin; packet retries/repairs and native rounds can exceed one floor wave. `task_acceptance_review_timing` (`delivery`, `deliveries`, `native_rounds`, `native_rows`) is telemetry, never gate input; `review_status` mirrors the structured axis. Revision rows name the pass and recorded causes: DEGRADED with an improvement capsule differs from a no-quorum terminal.
|
||||
At first physical dispatch, `review_dispatch.task_acceptance_paid_dispatch_stamp` claims the locked `task_results.task_acceptance_review_accounting` wallet, checking only cancellation/cycles. A claim without a recoverable terminal host run is UNKNOWN, not resend authority. Launched review retains deadline/task-ceiling and per-send money bounds; deadline cuts are DEGRADED, not free skips. Evidence build can consume the admitted margin; packet retries/repairs and native rounds can exceed the one-send wave estimate. `task_acceptance_review_timing` (`delivery`, `deliveries`, `native_rounds`, `native_rows`) is telemetry, never gate input; `review_status` mirrors the structured axis. Revision rows name the pass and recorded causes: DEGRADED with an improvement capsule differs from a no-quorum terminal.
|
||||
|
||||
A forced turn sends the round's exact tool envelope (same schemas and server-web flag) so the provider prefix stays a cache hit; "tool-less" describes the instruction and the host, which executes no call, so a reply still requesting a tool, with or without prose, is incomplete on every rail and degrades through the host fallback path instead of publishing as the model's final. Every forced rail uses the common terminal recorder; an eligible task without a panel run records an eligible bypass with zero runs and the rail's trigger. No forced rail takes another model round, so one shared helper terminalizes a dangling `revision_requested` as `finalized_unaccepted` (`revision_unavailable_on_forced_rail`) on EVERY forced rail, naming the prior reason; `accepted` and `finalized_unaccepted` are never overwritten, no bypass reason is stamped over a panel that ran, and the pair stays outside the blocked-terminal set, keeping the objective best_effort. Historical `children_unabsorbed` outcomes remain readable; reminders no longer force completion. A superseded panel stays an audit row, `run_count` intact; `pending_delivery_acceptance` is transient eligibility, never terminal. A deadline or late unread message retains unaccepted work rather than claiming a new subject was reviewed.
|
||||
|
||||
|
|
@ -222,7 +222,7 @@ Complete selected input is inline when actual capacity, tools and answer reserve
|
|||
|
||||
Observe retains a narrow positive path. `consciousness_authority.OBSERVE_ARGUMENT_NARROWED` lists the mutating-table names Observe still SEES, each with a genuine read-only or own-work use that withholding the name would destroy with the mutation: `schedule_subagent` and `delegate_start` for read-only research, `write_file`/`edit_text`/`journal_write`/`workpad_write` for its own notes, `cancel_task` for children it started. One predicate, `observe_argument_refusal`, consulted directly by the dispatch guard, decides what such a name may be ASKED to do — no fail-open wrapper, since a policy that cannot be consulted must not read as permission. It refuses a write root outside `task_drive`/`artifact_store`, a mutative `write_surface`/`may_mutate`, and a delegated session asked for `workspace_write` or a payload selector; an OMITTED access is the read-only shape (`_derive_authority` pins it) and stays allowed. `cancel_task` alone narrows by durable lineage, not arguments: it rides the tool's EXISTING own-child custody check (`join_ledger._cancel_task`), the seam a delegated task meets, so Observe may stop children it started, never the owner's running work. A read-only child may delegate read-only descendants under the inherited depth cap but never passes a mutating delegation budget onward (`delegation_may_mutate`). `manage_schedules` (`tools/followup.py`, beside `schedule_followup`: one module, two tools, one table) is argument-aware, not name-wide: the root turn may list and change, a delegated task only list, a Presence conversation neither — Presence authority is not expanded here.
|
||||
|
||||
Three metadata facts distinguish a wake-up from an owner's turn. ORIGIN is `metadata.initiator == "consciousness"`, inherited by everything the wake starts and labelling every surface; no owner door stamped it, so it speaks to tasks as a task, never the owner (§6 Owner routing verbs). AUTHORITY is the owner's autonomy level (`metadata.consciousness_autonomy`, owned by `consciousness_authority.py`): Observe researches, keeps internal memory/project notes, stops children it started, inspects and governs existing schedules through the audited narrow surface and may delegate read-only research, while the dispatch guard refuses owner-file, source, skill/settings and publication writes; Act (the default) is everything the runtime mode allows except editing Ouroboros's own code and prompts, evolution, restart and settings; Full adds evolution. A level has two structural consequences — an exception list in `task_contract.disabled_tools` and a per-task `runtime_mode_cap` of `light` below Full — plus Observe's argument-level narrowing (`observe_argument_refusal`). For a consciousness-origin task the list binds at DISPATCH ONLY: tool schemas and cached prompt prefix stay byte-identical to an owner turn's, a withheld call gets the typed dispatch refusal when made, and the wake message names the level. The stricter of install mode and cap binds at the repo-mutation, protected-path, `start_service` and shell-write gates, the acting-child write surface, subagent patch integration and a started task's Runtime block, so the owner's level holds even on a Cyber Pro install; a capped tree is refused a self_worktree child and system-repo patch integration in every install mode, the mutative-subagent toggle included. MONEY is one rolling 24-hour allowance over wake-ups and every root tree they start, read off the usage ledger (`consciousness_allowance.py`) by the alarm before waking and by the ONE admission door (`supervisor/queue.py::enqueue_task`) before requested work starts — no "consciousness task" type, only existing tools through the existing door; a Presence cycle a wake starts (Act and above) is an independent root outside it. The remainder (capped by the owner's per-task cap) is the wake tree's graceful ceiling (`root_cost_ceiling_usd`, inherited by its members): the in-task stop lands a planning margin before it while the ledger fence stays at the owner's per-task cap, so a wake never dies before its first call; a remainder at/below that margin counts as exhausted.
|
||||
Three metadata facts distinguish a wake-up from an owner's turn. ORIGIN is `metadata.initiator == "consciousness"`, inherited by everything the wake starts and labelling every surface; no owner door stamped it, so it speaks to tasks as a task, never the owner (§6 Owner routing verbs). AUTHORITY is the owner's autonomy level (`metadata.consciousness_autonomy`, owned by `consciousness_authority.py`): Observe researches, keeps internal memory/project notes, stops children it started, inspects and governs existing schedules through the audited narrow surface and may delegate read-only research, while the dispatch guard refuses owner-file, source, skill/settings and publication writes; Act (the default) is everything the runtime mode allows except editing Ouroboros's own code and prompts, evolution, restart and settings; Full adds evolution. A level has two structural consequences — an exception list in `task_contract.disabled_tools` and a per-task `runtime_mode_cap` of `light` below Full — plus Observe's argument-level narrowing (`observe_argument_refusal`). For a consciousness-origin task the list binds at DISPATCH ONLY: tool schemas and cached prompt prefix stay byte-identical to an owner turn's, a withheld call gets the typed dispatch refusal when made, and the wake message names the level. The stricter of install mode and cap binds at the repo-mutation, protected-path, `start_service` and shell-write gates, the acting-child write surface, subagent patch integration and a started task's Runtime block, so the owner's level holds even on a Cyber Pro install; a capped tree is refused a self_worktree child and system-repo patch integration in every install mode, the mutative-subagent toggle included. MONEY is one rolling 24-hour allowance over wake-ups and every root tree they start, read off the usage ledger (`consciousness_allowance.py`) by the alarm before waking and by the ONE admission door (`supervisor/queue.py::enqueue_task`) before requested work starts — no "consciousness task" type, only existing tools through the existing door; a Presence cycle a wake starts (Act and above) is an independent root outside it. It decides on known spend, holds disclosed beside it. The remainder (capped by the owner's per-task cap) is the wake tree's producer ceiling (`root_cost_ceiling_usd`, honoured by its members): the in-task stop fires when known spend reaches it, with no planning margin; exact-continuation eligibility decides saved pause versus a terminal result. The ledger fence keeps the owner's per-task cap; only a zero daily remainder blocks the wake before its first call.
|
||||
|
||||
An active campaign owns its objective, id and transaction/task claim (`supervisor/evolution_lifecycle.py`); `evolution_mode_enabled` only projects scheduling. Dispatch, review, commit, publication and restart revalidate the claim. Restored work without a live uncommitted claim is cancelled; publication binds the exact reviewed SHA. Authority lost after commit preserves it on a private ref and removes attempt-created tags from the normal namespace without resetting concurrent edits. Terminal replay completes missing effects without double-counting. `state/pending_restart_verify.json` binds restart verification to the claim, including when auto-restart is off; the shared writer serves both supervisor and restart tool.
|
||||
|
||||
|
|
@ -361,7 +361,7 @@ Both task waits use `task_status.SETTLED_STATUSES`; pending cancellation stays `
|
|||
| estimated amount | the amount | `cost_final=false` — an estimated zero is not a proven free session |
|
||||
| undisclosed | `cost_usd: null`, increments `unknown_unmetered` | drops `cost_final` for the projection |
|
||||
|
||||
An undisclosed spend contributes `0.0` to `accounted_usd` — inventing a conservative bound would fabricate a number the harness never gave (BIBLE P1) — so a `TOTAL_BUDGET` fence cannot stop spend it was never told about; the honest consequence is loss of finality, not a guessed charge.
|
||||
An undisclosed spend contributes `0.0` to known `settled_usd` and `accounted_usd` — inventing a conservative bound would fabricate a number the harness never gave (BIBLE P1) — so a `TOTAL_BUDGET` fence cannot stop spend it was never told about; the honest consequence is loss of finality, not a guessed charge.
|
||||
|
||||
**What a delegated run READ.** Harnesses count input tokens with incompatible conventions, so `settle_run` carries the engine's own normalized split, `summary.inputTokenUsage`, onto the `subscription_session` row as the optional `input_token_usage` object: `total_tokens`, `cache_read_tokens`, `cache_write_tokens`, each a nonnegative integer or `null` for unknown. `record_subscription_session` is the one validator and writer: all three keys are required together, and a partial, extra-keyed, negative or fractional object is unknown as a WHOLE, not repaired field by field, because a repaired counter reads exactly like a measured one. An engine that reports nothing leaves the row unchanged, and the object stays OUT of the idempotent row identity, so an engine that starts reporting never rewrites or duplicates a session already settled without it; the legacy `prompt_tokens`/`cached_tokens` axes keep their own meanings, compaction never folds these idempotency-bearing rows, and direct physical attempts take no copy.
|
||||
|
||||
|
|
@ -533,7 +533,7 @@ The review surfaces:
|
|||
- Advisory pre-review (`claude_advisory_review.py`) — a cheap, staleness-aware error-finding pass; the audited skip covers only advisory admission, never authoritative review, test policy or snapshot binding.
|
||||
- Triad diff review (`tools/review.py`) — configured reviewer slots cover the layered Change Review Checklist (the universal core, plus the Ouroboros Body Layer when the subject is the body — `review_helpers.load_checklist_layers`) with JSON findings under `config.adaptive_quorum` (the same SSOT as scope/plan/skill/acceptance review).
|
||||
- Scope review (`tools/scope_review.py`) — intent/scope/coupling across the whole repository by retrieval on every row, in every context mode; the exact candidate, required-source manifest and diagnostic reading coverage stay distinct from substantive findings and configured enforcement (§6 Scope review by retrieval).
|
||||
- Parallel orchestration (`tools/parallel_review.py`) — two-phase admission: the triad packet is assembled/fit-checked and every scope brief prepared before any reviewer dispatches, so a deterministic assembly block anywhere dispatches nothing and spends $0 (a lane paid while its sibling was doomed is real money for a half-verdict). Money admission (`tools/review_admission.py`: `commit_gate_paid_seats` / `admit_commit_gate_wave`) is all-or-nothing and scope-first: every PAID seat (blocking scope seats FIRST, then the triad) is priced with the reservation math `reserve_attempt` will apply (exact first send, native work-order and tool schemas included, plus its own output reservation; later native rounds reserve themselves), under its substrate's usage scope (`review_usage_category` + slot, so a warm split of the caller's transcript never stands in for a seat's cold prefix), against the global `TOTAL_BUDGET`, current root and original billing-group fences. Admission uses the reservation's `_merge_scope` resolution, including missing durable group bindings and explicit owner amendments; a raised root cap alone never raises its group — and admitted as ONE wave through `review_wave_budget_gate(surface="commit_gate")`. A wave that does not fit records typed `$0 not_dispatched` on every seat plus one `review_wave_budget_insufficient` event with `binding_axis` (`global`/`root`/`group`) and the remainders; the block names the fence, shortfall and actual remedy — never a half-dispatched panel whose non-blocking seats hold the blocking reviewer's money. Admission is a read-only pre-check without a wave-level hold; per-seat reservation stays the enforcement, so a concurrent task's global spend between admission and a seat's reservation can still refuse that seat (disclosed residual). A paid seat is any api row (packet or native episode); an agent-session row rides the owner's subscription (ledger row written at settlement, never reserved) and is neither priced nor waited for. Every executor transition toward a seat runs under `contextvars.copy_context()`, so reviewer rows reserve under the admitted usage scope, bound root fence included; a mid-turn settings reload of the per-task limit changes neither side. An admission that raises is fail-open like the gate's own unknowns, but loud and typed (`review_wave_admission_unavailable`; the wave dispatches unadmitted). An admitted wave submits the scope seat first and starts the triad only once that seat's OWN reservation is on the ledger (nothing else releases the hold), bounded by `NESTED_SETTLEMENT_MARGIN_SEC` and by the scope seat itself; a hold ending unobserved emits a typed `review_scope_lead_unobserved` event. Admission touches no transport or logical timeout contract.
|
||||
- Parallel orchestration (`tools/parallel_review.py`) — two-phase admission: the triad packet is assembled/fit-checked and every scope brief prepared before any reviewer dispatches, so a deterministic assembly block anywhere dispatches nothing and spends $0 (a lane paid while its sibling was doomed is real money for a half-verdict). Money admission (`tools/review_admission.py`: `commit_gate_paid_seats` / `admit_commit_gate_wave`) admits the wave while KNOWN spend is below the global `TOTAL_BUDGET`, current root and original billing-group fences, scope-first: every PAID seat (blocking scope seats FIRST, then the triad) is priced, as disclosure, with the reservation math `reserve_attempt` will apply (exact first send, native work-order and tool schemas included, plus its own output reservation; later native rounds reserve themselves), under its substrate's usage scope (`review_usage_category` + slot, so a warm split of the caller's transcript never stands in for a seat's cold prefix). Admission uses the reservation's `_merge_scope` resolution, including missing durable group bindings and explicit owner amendments; a raised root cap alone never raises its group — and admitted as ONE wave through `review_wave_budget_gate(surface="commit_gate")`. A refused wave records typed `$0 not_dispatched` on every seat plus one `review_wave_budget_insufficient` event with `binding_axis` (`global`/`root`/`group`) and the remainders; the block names the fence and actual remedy. Admission is a read-only pre-check without a wave-level hold; per-seat reservation and dispatch stay the enforcement, so a fence that known spend reaches mid-wave refuses the remaining seats with truthful custody. A paid seat is any api row (packet or native episode); an agent-session row rides the owner's subscription (ledger row written at settlement, never reserved) and is neither priced nor waited for. Every executor transition toward a seat runs under `contextvars.copy_context()`, so reviewer rows reserve under the admitted usage scope, bound root fence included; a mid-turn settings reload of the per-task limit changes neither side. An admission that raises is fail-open like the gate's own unknowns, but loud and typed (`review_wave_admission_unavailable`; the wave dispatches unadmitted). An admitted wave submits the scope seat first and starts the triad only once that seat's OWN reservation is on the ledger (nothing else releases the hold), bounded by `NESTED_SETTLEMENT_MARGIN_SEC` and by the scope seat itself; a hold ending unobserved emits a typed `review_scope_lead_unobserved` event. Admission touches no transport or logical timeout contract.
|
||||
- Shared helpers (`review_helpers.py`, `triad_review.py`) — packet/brief building, checklist loading, JSON extraction, usage events, obligations scaffolding and reviewer actor records.
|
||||
|
||||
A free refusal never wears the form of a verdict: one that spent nothing is recorded as a typed `not_dispatched` fact plus its reason, never as a DEGRADED panel or a synthetic actor. That is the one shape across every $0 exit — a plan-review locator the evidence policy cannot attach is a named omission row the panel is dispatched with, an acceptance packet that overflows takes the ladder rather than reporting a verdict, a truncated or self-pageable row names its cut, and a request never sent is one seat with `operation_state='not_dispatched'` that stays in the denominator. `task_acceptance_review` refused before dispatch (a child with no selectable row, an invalid roster: `status=not_dispatched`, no run/actor evidence) stays a tool result outside `review_runs`; alone it cannot degrade the review axis. Dispatched malformed, overflowed or degraded results still do (`loop_tool_execution.process_tool_results`).
|
||||
|
|
@ -642,7 +642,7 @@ Paid cycles are bounded by the shared `OUROBOROS_REVIEW_MAX_CYCLES`; `cycles_pai
|
|||
|
||||
Before fan-out the engine captures one panel health snapshot (`subagents.route_health`, route-level evidence): a slot with positive structural evidence of a spent lane becomes a $0 typed skip row that stays in the denominator; unknown health dispatches (fail-open). The wave records the health epoch and reviewer-roster fingerprint, and a recorded DEGRADED wave replays free only under an identical envelope, matching epoch and unchanged roster — otherwise another paid panel needs remaining cycle capacity (replay/epoch casuistry: `plan_review.py` docstrings). When the wave's own typed rows prove the quorum structurally unreachable, it carries `quorum_unreachable` plus the earliest recorded reset, and under blocking enforcement the finalization gate RELEASES while the review stays open: the agent may finalize `blocked_with_evidence` (reason `plan_review_quorum_unreachable`), wait through a one-shot `schedule_followup`, or ask the owner — the host adds facts only, never an answer template.
|
||||
|
||||
Under blocking an open wave otherwise holds implementation and an exhausted cap escalates as typed `review_cycles_exhausted` (free disposition and exact pending custody keep their rules); under advisory the agent may proceed with the wave open, after typed owner-visible `plan_review_advisory_open` events (one per announced outcome: the dispatch snapshot, then each settled failure with its code and reported cause) and with the open wave recorded in the task's state and result; the model-facing guidance, the `plan_task` description and the cap head say that the agent's own answer is where the open wave is stated, never that a host narrator will state it for it. A forced rail hands its ONE model call the same limitations as typed `key=value` facts inside `_prepare_forced_prompt`, so they are priced by the existing wrap-up reservation probe and no no-call rail gains a call. A hold the agent was told about that the owner's hurry then released reaches it once before its last word; every other gate transition already arrives through the agent's own `plan_task` result, the settled-wave task message, or that forced prompt. Unavailability, invalid state, budget refusal and deadline rails stay typed non-authoritative attempts, never substitutes for GREEN, and a reviewer request the host cannot attach is never refused before dispatch: the only $0 exits are those attempts, `not_dispatched` slot rows and the `pending_dispatch` rows of a barrier-released wave. Every disclosure states the wave's CURRENT state, names the late-result fact while a slot can still settle, never calls an open wave ended, and renders a rail with no recorded reason as absence, not an internal token.
|
||||
Under blocking an open wave otherwise holds implementation and an exhausted cap escalates as typed `review_cycles_exhausted` (free disposition and exact pending custody keep their rules); under advisory the agent may proceed with the wave open, after typed owner-visible `plan_review_advisory_open` events (one per announced outcome: the dispatch snapshot, then each settled failure with its code and reported cause) and with the open wave recorded in the task's state and result; the model-facing guidance, the `plan_task` description and the cap head say that the agent's own answer is where the open wave is stated, never that a host narrator will state it for it. A forced rail hands its ONE model call the same limitations as typed `key=value` facts inside `_prepare_forced_prompt`, so they ride the forced call's own reservation (and an explicit profile's wrap-up probe) and no no-call rail gains a call. A hold the agent was told about that the owner's hurry then released reaches it once before its last word; every other gate transition already arrives through the agent's own `plan_task` result, the settled-wave task message, or that forced prompt. Unavailability, invalid state, budget refusal and deadline rails stay typed non-authoritative attempts, never substitutes for GREEN, and a reviewer request the host cannot attach is never refused before dispatch: the only $0 exits are those attempts, `not_dispatched` slot rows and the `pending_dispatch` rows of a barrier-released wave. Every disclosure states the wave's CURRENT state, names the late-result fact while a slot can still settle, never calls an open wave ended, and renders a rail with no recorded reason as absence, not an internal token.
|
||||
|
||||
#### Deep self-review
|
||||
|
||||
|
|
@ -755,19 +755,19 @@ The time/cost/intrinsic pacing checkpoints carry `resource_facts`: incremental p
|
|||
|
||||
`cost_projection.py` owns producer amounts/openness: `accounted_upper_bound_usd` = settled+reserved+unresolved, null is unknown, finality is never invented, and `COST_OPENNESS_FIELDS` accompanies amounts. Retired `cost_usd[_with_children]` stays read-only. `cost_presentation` binds amount/facts to one bucket (§11.1): `reconstruct_task_cost` owns own scope; root terminal/heartbeat/checkpoint/synthesis paths use root-tree, never falling from null subtree to own zero. `_usage_rows._summary`'s weighted `priced_rows`, `tracked_nonfinal_rows`, `accounting_open_rows` hold for imported aggregates. A price or retained bound evidences zero; empty/unpriced rows do not. Settled unpriced rows leave the known subtotal exact; estimates (zero included), retained bounds, open attempts and integrity gaps stay nonfinal. Terminal refresh/copyback, synthesis, heartbeat and history preserve scope; unknown never becomes sticky-final. Sums and `cost_final` are unchanged.
|
||||
|
||||
A proven abandoned send closes administratively as `settled` with `settle_reason="abandoned"`, `cost_usd=None` and `cost_final=false`: its reservation bound stays accounted, not confirmed spend, and the price remains non-final. A real late receipt may correct the same attempt once (`late_receipt`); a positive never-started receipt may instead release it through the existing `before_dispatch_failed:` seam. Ordinary terminal settlements and releases remain immutable. The store's writes and its one-time journal import share one transition table, so eligibility is a fact of the attempt's one current row; imported baseline groups stay aggregates ([Usage store](../USAGE_STORE.md)).
|
||||
A proven abandoned send closes administratively as `settled` with `settle_reason="abandoned"`, `cost_usd=None` and `cost_final=false`: its reservation bound stays unresolved exposure, not known spend, and the price remains non-final. A real late receipt may correct the same attempt once (`late_receipt`); a positive never-started receipt may instead release it through the existing `before_dispatch_failed:` seam. Ordinary terminal settlements and releases remain immutable. The store's writes and its one-time journal import share one transition table, so eligibility is a fact of the attempt's one current row; imported baseline groups stay aggregates ([Usage store](../USAGE_STORE.md)).
|
||||
|
||||
`terminal_cost_reconciliation` reads open attempts in Phase A; one result parse feeds ownership checks outside the queue lock, with memory rechecked before mutation. Receipts precede HTTP; terminal/gone facts stop probes but keep late-receipt rights, provider Retry-After sets due times, and `duty_unresolved` summarizes changed uncertainty. Phase B projects addressed task/root summaries from `dirty_owners` and acks only that revision; failed, missing or live results retain debt. Missing/unreadable owner-pause authority emits a changed-signature `terminal-cost-projection` duty with `task_ids`, separately from attempt uncertainty; one warning summarizes it, debug retains the stack. Repair retries the same debt; unexpected failures keep warning tracebacks. Only money changes; delivery replay is unchanged.
|
||||
`terminal_cost_reconciliation` reads open attempts in Phase A; one result parse feeds ownership checks outside the queue lock, with memory rechecked before mutation. Receipts precede HTTP; terminal/gone facts stop probes but keep late-receipt rights, and provider Retry-After sets due times. Phase B projects addressed task/root summaries from `dirty_owners` and acks only that revision; failed, missing or live results retain debt. It stats the canonical result before the ownership read: an absent one keeps its exact revision without the fence's quarantine traversal, so a body copied back or restored after the last receipt is projected next pass (absence never retires debt; one stat per such owner per pass). `duty_unresolved` publishes each subject's (`attempt`, `cost_projection`) id→reason map (class plus errno/code) only on change; a reason new to the subject logs one representative traceback, an id whose outcome the pass did not prove (skipped, live, lost race) keeps its last reason, and a failed publish or gateway close is logged once per change. Only money changes; delivery replay is unchanged.
|
||||
|
||||
Summary/reflection/consolidation share a frozen ledger snapshot: settled subtree cost, reservations, unresolved bounds/counts, unknown exposure, integrity and non-final state. Final terminal checkpoint alone owns final cost; read failure is unavailable/null, never zero. No reconciliation LLM or parallel ledger exists. Cache-hit health uses only rounds carrying `cached_tokens`, and stays absent with insufficient samples: unmeasured is unknown, not zero, so providers lacking telemetry cannot dilute measured ratios. Unknown spend remains the existing `unknown_unmetered` field, not a third accounting status.
|
||||
|
||||
`root_phase_checkpoint.accounting` retains a cumulative ledger observation: root ID, accounted/reserved/unresolved amounts, non-final/unresolved counts, unknown exposure and integrity. Separate from own-cost and read-time `TaskCostBreakdown`, it survives public detail/recovery and canonical replica protection. The checkpoint owner refreshes it, explicitly null/unavailable on failure. These facts prove neither an invoice nor local paid-work closure; phase/execution ownership is independent.
|
||||
|
||||
**Planning threshold** (`task_pacing.py`). One unreserved `CostCeiling` (`disabled`, `active`, `exhausted_soft_land`, `unknown`) owns runtime `in_task_cost_ceiling` disclosure and loop checks. Roots use min(global-remainder percentage, root cap) minus margin; uncapped roots use starting wallet. Enabled descendants inherit that number, not later percentages. Disabled profiles disable only pacing; monetary admission stays independent and surfaces name the binding bound. Rooted tasks compare subtree spend/holds even without root caps; own-spend fallback is a disclosed lower bound.
|
||||
**Planning threshold** (`task_pacing.py`). One unreserved `CostCeiling` (`disabled`, `active`, `exhausted_soft_land`, `unknown`) owns runtime `in_task_cost_ceiling` disclosure and loop checks. An ordinary task has none (`no_default_cost_stop`): only an explicit `cost_hard_stop_pct` keeps min(global-remainder share, root cap minus margin), and a producer allowance (a wake's remainder) stops at its actual value. Descendants inherit the root's number under its authority, re-read by `restore_cost_ceiling` at every cold restore; an unreadable root keeps the number as `legacy_policy_unverified`, never widened. Monetary admission stays independent and surfaces name the binding bound. Rooted tasks compare subtree known spend even without root caps, holds beside it; own-spend fallback is a disclosed lower bound.
|
||||
|
||||
`loop_budget._finish_tool_round_budget` and `_finish_no_tool_round_budget` share `_check_budget_limits` after unfinished spending rounds. No-tool tails, including cold Resume (`resume_point.budget_tail`), neither advance nanny baselines nor arm tool controls. Candidates survive; READY returns before this check without an extra paid final. Eligible actors pause, even before work on a soft threshold. Ineligible actors retain the priced terminal rail (`budget_wrapup_unaffordable` if it cannot fit); global rejection before work buys no call. Tool requests remain incomplete even beside `replace`.
|
||||
`loop_budget._finish_tool_round_budget` and `_finish_no_tool_round_budget` share `_check_budget_limits` after unfinished spending rounds. No-tool tails, including cold Resume (`resume_point.budget_tail`), neither advance nanny baselines nor arm tool controls. Candidates survive; READY returns before this check without an extra paid final. Eligible actors pause, even before work on a soft threshold. Ineligible actors at an explicit profile's stop retain the priced terminal rail (`budget_wrapup_unaffordable` if it cannot fit); its send still meets the known-spend fence, so a reached cap buys no recap. At a producer allowance or unverified number, which the fence never checks, the host notice ends them unpaid; global rejection before work buys no call. Tool requests remain incomplete even beside `replace`.
|
||||
|
||||
Affordability probes reserve nothing; full-cap ledger admission decides. Unknown spend is not zero; unknown price fails open. `_usage_cache_splits.py` reuses settled provider/normalized-route/review-surface splits: cache loss reprices full write, expiry may under-reserve one write, competitors may consume room. Proxy stops/native images require exact transcript-copy pricing before service finalization. `task_pacing.main_loop_wire_options` owns wire options with no later payload additions. Identity drift refuses pre-send (`forced_candidate_drift`); terminal handling retries once unpredicated, still ledger-priced, so drift cannot lose the answer.
|
||||
Affordability probes (explicit profiles only) reserve nothing; full-cap ledger admission on known spend decides. Unknown spend is not zero; unknown price fails open. `_usage_cache_splits.py` reuses settled provider/normalized-route/review-surface splits: cache loss reprices full write, expiry may under-reserve one write, competitors may consume room. Proxy stops/native images require exact transcript-copy pricing before service finalization. `task_pacing.main_loop_wire_options` owns wire options with no later payload additions. Identity drift refuses pre-send (`forced_candidate_drift`); terminal handling retries once unpredicated, still ledger-priced, so drift cannot lose the answer.
|
||||
|
||||
#### Exact budget pause and Resume
|
||||
|
||||
|
|
@ -787,7 +787,7 @@ Affordability probes reserve nothing; full-cap ledger admission decides. Unknown
|
|||
|
||||
**Descendant selection.** `events_budget.hold_root_resume_descendants` makes children eligible, not runnable: exact rows stay paused; zero-dispatch/fence-only siblings get `root_fence_lifted_pending_selection`, rebound to the current root grant on each Resume. `resume_child_task` (`budget_resume_child`, recorded `selected_by`) selects within lineage: root selects stored descendants, an intermediate parent only direct children; owner selection shares the path. With a live latch, `budget_pause.budget_fence_selected` (assignment, its launch handoff, and `reserve_attempt` via the snapshot's running/pending row) binds one member to its fence generation: a selected hold or, deciding alone, an exact `explicit_resume` handoff by the owner or a positive root-grant generation. Re-paused, stale-consumed, held and sleep-readiness rows (never launched under a monetary latch), siblings and the unselected root stay fenced; later sends need no new grant; caps bind. Legacy root Resume selects only root. A root parking unselected under a child's latch keeps it; one released past it pauses anew. New root pauses invalidate pending child grants; assignment rechecks root grant and fence for exact grants and zero-dispatch selections (stale → unselected hold). The grant binds the queue row's `root_task_id`, the one assignment rechecks, never a saved pause's own root field. `request_pause` resolves one tree for its seed launch lock and durable row (an explicit root verbatim, else like `member_fence`: `task_metadata` first, a `ToolContext` has no root field); the owner-fence reopen reads the same tree. Other roots/cancelled/completed/stopped members never revive.
|
||||
|
||||
**Consumption.** `resume_paused_loop` rejects spent/revoked/foreign grants and discloses drift/custody before effects; a consumption publication that fails HOLDS (`resume_grant_consumption_unwritable`; identities kept), a grant revoked underneath re-parks; never FAILED. Graceful Resume refreshes ledger-authorized headroom via the same strict tree read and admits ONE fitting reservation (`_second_reservation_fits`, `budget_resume_last_fit_admitted`), avoiding the early-margin re-pause. Hard exhaustion needs owner increase; full-cap admission binds. Browser/services/hidden state and crash recovery are not restored.
|
||||
**Consumption.** `resume_paused_loop` rejects spent/revoked/foreign grants and discloses drift/custody before effects; a consumption publication that fails HOLDS (`resume_grant_consumption_unwritable`; identities kept), a grant revoked underneath re-parks; never FAILED. Graceful Resume of an explicit profile refreshes ledger-authorized headroom via the same strict tree read and admits ONE fitting reservation (`_second_reservation_fits`, `budget_resume_last_fit_admitted`), avoiding the early-margin re-pause; a producer allowance (`producer_allowance_not_expanded`), an unverified number (`policy_unverified_not_expanded`) or a disabled ceiling is never widened or re-armed; the notice says the launch allowance still binds even if its window freed more, re-pausing before another model call. Hard exhaustion needs owner increase; full-cap admission binds. Browser/services/hidden state and crash recovery are not restored.
|
||||
|
||||
#### The model's own sleep
|
||||
|
||||
|
|
@ -844,7 +844,7 @@ Occurrence/token, frozen receipt/live queue/result `dispatch_possible` and owner
|
|||
|
||||
**Live global limit.** One agent resolver: saved settings, reparsed on change; failure (including refused benchmark pins) falls back to environment without caching failure. Workers refresh environment at task start; live file reads keep reservation/wallet `TOTAL_BUDGET` current. Planning thresholds stay separate. Absent means shipped default; non-positive means unbounded, silencing the loop global axis. A run cap set by a launcher before settings load (`OUROBOROS_REVIEW_RUN_CAP_USD`, only the isolated contributor review) replaces the saved budget for that process tree, whose ledger starts empty; an unreadable cap is a zero allowance. Pre-existing gap: supervisor startup/reload parse raw settings and treat absence as unlimited.
|
||||
|
||||
`pricing.py` uses bounded best-effort exact normalized-route catalog lookup, never manual tables, prefix/fallback prices or admission allowlists. Unknown price admits while known spend fits: reserve None, settle reported cost or later exact price, otherwise None/non-final. Only structural pre-generation evidence with zero usage permits confirmed-zero rejection/release (`_usage_response.py` normalizes; adapters retain raw usage). `usage_admission.review_wave_admission`, compatibly exported by `usage_accounting` without eager cycles, uses the shared estimator and tighter global/root/group remainder before skill/plan/task/commit review and managed-update destructive apply. Unpayable slots bypass without substitution; unknown prices stay disclosed without disabling priced siblings.
|
||||
`pricing.py` uses bounded best-effort exact normalized-route catalog lookup, never manual tables, prefix/fallback prices or admission allowlists. Every limit refuses once known (settled) spend reaches it, at reserve and again at dispatch; holds and unresolved bounds are exposure, never spending, and concurrent or late charges can overshoot. Unknown price: reserve None, settle reported cost or later exact price, otherwise None/non-final. Only structural pre-generation evidence with zero usage permits confirmed-zero rejection/release (`_usage_response.py` normalizes; adapters retain raw usage). `usage_admission.review_wave_admission`, compatibly exported by `usage_accounting` without eager cycles, admits a skill/plan/task/commit wave while known spend is below every global/root/group limit; the shared estimator's wave total is disclosure, and managed update only records it (`managed_update_wave_estimate`). Unpayable slots bypass without substitution; unknown prices stay disclosed without disabling priced siblings.
|
||||
|
||||
**Whole-work billing group.** `usage_admission.py` keeps task/root/parent IDs distinct from `billing_group_id`/`billing_group_limit_usd`. One store transaction checks global/root/group, successors and original-root late work included. Continue carries the initial sourced/revisioned `billing_group`, else the earliest live ledger authority, else the configured cap (`legacy_default`). `BindingIndex` keeps unlimited `None`; imported uncarried block rows bind from their own cap literal (`legacy_live`) when they agree, else the member stays open; the store keeps each binding ([store](../USAGE_STORE.md)). Review-wave `binding_axis=group`, pacing and strict Resume share this authority; late subscriptions retain own root/start custody. Only the group owner amends its cap; successor caps and spend remain independent. `task_money_snapshot` keeps distinct root/group axes over shared spend. Cap-only does not Resume, review or replace pacing. A rootless descendant refuses; unknown prices stay disclosed; late overruns bar admission without a bill guarantee. Standalone UI/marketplace contexts declare a non-task role; inherited managed scope wins. Global and cumulative service-label root caps remain: role repair does not cure lifetime-cap exhaustion.
|
||||
|
||||
|
|
|
|||
|
|
@ -116,8 +116,8 @@ A registry of `config.SETTINGS_DEFAULTS` (exact defaults canonical in `settings_
|
|||
| OUROBOROS_DELIVERABLES_ROOT | (empty) | Deliverables root (empty derives the `~/Ouroboros/Deliverables` sibling; `tool_access.py`) |
|
||||
| OUROBOROS_GC_RETENTION_DAYS | 7 | Unified GC retention (`retention.py`) |
|
||||
| OUROBOROS_RESTART_DRAIN_MAX_SEC | 120 | Restart drain bound |
|
||||
| TOTAL_BUDGET | 200.0 | Global budget (USD); an absent key resolves to this product default, a non-positive value means no finite limit; resolved live from the saved document, so a change binds running tasks at their next model call (`resolve_total_budget_usd`) |
|
||||
| OUROBOROS_PER_TASK_COST_USD | 50.0 | Per-task cost cap and tree ceiling basis: the root resolves min(global share, cap minus margin), descendants retain it, admission stays independent, and the wrap-up affordability rail soft-lands under it (`task_pacing.py`, §6 Budget tracking) |
|
||||
| TOTAL_BUDGET | 200.0 | Global budget (USD), reached by known spend; an absent key resolves to this product default, a non-positive value means no finite limit; resolved live from the saved document, so a change binds running tasks at their next model call (`resolve_total_budget_usd`) |
|
||||
| OUROBOROS_PER_TASK_COST_USD | 50.0 | Hard cap over one task's whole tree: new paid calls are refused once its known spend (settled: confirmed + estimated) reaches it; holds are not spending. No default early stop precedes it; only an explicit `cost_hard_stop_pct` profile resolves min(global share, cap minus margin) (`task_pacing.py`, §6 Budget tracking) |
|
||||
| OUROBOROS_RUB_USD_RATE | (empty) | Manual RUB→USD rate for RUB-priced providers |
|
||||
| OUROBOROS_PRICING_TTL_SEC | 21600 | Provider-catalog pricing cache TTL |
|
||||
| OUROBOROS_TOOL_TIMEOUT_SEC | 600 | Default tool timeout |
|
||||
|
|
@ -205,7 +205,7 @@ A registry of `config.SETTINGS_DEFAULTS` (exact defaults canonical in `settings_
|
|||
| OUROBOROS_BG_WAKEUP_MIN | 900 | Lower bound (s) of the model-chosen wake interval (`set_next_wakeup`), clamped into [min, max] and re-read at each alarm, so a change needs no restart |
|
||||
| OUROBOROS_BG_WAKEUP_MAX | 14400 | Upper bound (s), never below the minimum; with no chosen interval the alarm uses `runtime_limits.WAKE_DEFAULT_SEC` (3300 s, just under the 1 h cache TTL, keeping the shared prefix warm) — a constant rather than a third knob |
|
||||
| OUROBOROS_CONSCIOUSNESS_AUTONOMY | act | Closed enum for a wake (else the default): `observe` researches and performs internal work, can start and stop its own read-only children and use owner-governed schedule controls, and cannot write shell/user files/source/skills/settings or publish; `act` adds all the runtime mode allows except own code/prompts, evolution, restart, settings; `full` includes evolution |
|
||||
| OUROBOROS_CONSCIOUSNESS_DAILY_USD | 20.0 | Rolling-24h ceiling over consciousness wakes and the tasks they start; exhausted blocks NEW wakes and tasks until spend leaves the window; `0` is a real choice |
|
||||
| OUROBOROS_CONSCIOUSNESS_DAILY_USD | 20.0 | Rolling-24h ceiling over consciousness wakes and the tasks they start, on known spend; exhausted blocks NEW wakes and tasks until spend leaves the window; `0` is a real choice |
|
||||
| OUROBOROS_CONSCIOUSNESS_MAX_TASKS | 2 | How many consciousness-started tasks may run at once; `0` = it never starts tasks |
|
||||
| OUROBOROS_POST_TASK_EVOLUTION | false | Post-task evolution promotion toggle; self-enablement is blocked at the shell/browser/settings/data-write guards, choosing an objective runs on Main because it is a high-leverage decision, execution stays behind review and owner gates |
|
||||
| OUROBOROS_POST_TASK_EVOLUTION_CADENCE | llm | Promotion cadence `llm` or `every_n:k` (malformed normalizes to `llm`) |
|
||||
|
|
|
|||
|
|
@ -741,9 +741,9 @@ and what enforces each.
|
|||
facts reuse the note cadence and cached money projections; typed tool results count
|
||||
incrementally on the loop usage carrier; durations are overlapping observations, not
|
||||
inferred sleep/poll time or a behavior gate (`tests/test_budget_resource_facts.py`).
|
||||
- Tree-spend pacing decides on root-subtree spend including in-flight holds,
|
||||
publishes the same `CostCeiling` object the loop decides on, and prices the wrap-up
|
||||
with the fence's own cache-aware reservation (`tests/test_network_budget_wallet.py`).
|
||||
- Tree-spend pacing decides on root-subtree known (settled) spend, holds beside it,
|
||||
publishes the same `CostCeiling` object the loop decides on, and prices an explicit
|
||||
profile's wrap-up with the fence's own cache-aware reservation (`tests/test_network_budget_wallet.py`).
|
||||
Explicitly disabled profiles and real monetary fences stay independent; the
|
||||
configured global budget is read LIVE through the one resolver — never an inline
|
||||
default, never a per-task capture (`tests/test_budget_resource_facts.py`).
|
||||
|
|
@ -1110,8 +1110,8 @@ and what enforces each.
|
|||
- Task-acceptance actors are the configured triad rows
|
||||
(`reviewer_slot_config.triad_delivery_slots`; malformed config refuses typed), one
|
||||
substantive interaction each on its own delivery; the retrieving work order,
|
||||
`evidence_refs` against the FULL packet, the money rule (one work-order send per paid
|
||||
row, no rounds multiplier, no second pricing pass), the once-per-panel launch floor
|
||||
`evidence_refs` against the FULL packet, the money rule (known spend below every limit;
|
||||
one work-order send per paid row disclosed, no rounds multiplier), the once-per-panel launch floor
|
||||
(`task_pacing.review_launch_allowed`, `task_acceptance_paid_dispatch_stamp._claim`),
|
||||
the clamps on a running panel and the deadline-cut residual are stated once in
|
||||
ARCHITECTURE §6 "Task acceptance". Format-repair resends are packet-row only; child
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
Machine extraction of the `docs/ARCHITECTURE.md` "Data layout (`~/Ouroboros/`)" tree — the durable-file orientation carrier (this tree's counterpart of the reference PERSISTENCE_OWNERS derivation checklist) — regenerated by `python scripts/regenerate_inventories.py`. Do not edit. Every entry is probed against reality: repo entries must exist as tracked paths; data-plane entries must appear as a literal in the runtime sources that construct them. A durable file renamed or removed in code while its tree row survives = red (`tests/test_generated_inventories.py`).
|
||||
|
||||
Source: `docs/architecture/01-high-level-architecture.md`, physical LF lines 693-793; UTF-8 SHA-256 `f595e658ea8df62cc0f12c1618705f8437c75ba7c2ecc03c3ca5f4816bac2c9d`.
|
||||
Source: `docs/architecture/01-high-level-architecture.md`, physical LF lines 693-793; UTF-8 SHA-256 `82ec0f6fea094375583ed57bb7db429529cef7cacff4173b5ebc2f3dcb5ecbc4`.
|
||||
|
||||
- entries: **90** (code-ref: 83, repo-dir: 6, repo-path: 1)
|
||||
|
||||
|
|
|
|||
52
launcher.py
52
launcher.py
|
|
@ -370,6 +370,9 @@ def _cleanup_recorded_server_group_for_pid(pid: int, reason: str = "agent_exit")
|
|||
def start_agent(port: int = AGENT_SERVER_PORT) -> subprocess.Popen:
|
||||
"""Start server.py as the managed agent subprocess."""
|
||||
global _agent_proc, _agent_job
|
||||
# Stdlib-only, so it loads even when the server's own imports are broken; imported
|
||||
# before the spawn, because an undrained pipe would block the server.
|
||||
from ouroboros.process_logging import copy_capped_output
|
||||
|
||||
settings = _load_settings()
|
||||
_apply_settings_to_env(settings)
|
||||
|
|
@ -459,52 +462,9 @@ def start_agent(port: int = AGENT_SERVER_PORT) -> subprocess.Popen:
|
|||
log.info("Agent pid %d assigned to Windows Job Object", proc.pid)
|
||||
|
||||
_write_server_process_record(proc, port=port, server_py=server_py, server_host_source=host_source)
|
||||
|
||||
def _stream_output() -> None:
|
||||
# Match server.log rotation; rotation failure leaves streaming alive.
|
||||
max_bytes = 2 * 1024 * 1024
|
||||
backups = 3
|
||||
|
||||
def _rotate(log_path: pathlib.Path) -> None:
|
||||
try:
|
||||
for index in range(backups - 1, 0, -1):
|
||||
older = log_path.with_name(f"{log_path.name}.{index}")
|
||||
if older.exists():
|
||||
os.replace(older, log_path.with_name(f"{log_path.name}.{index + 1}"))
|
||||
if log_path.exists():
|
||||
os.replace(log_path, log_path.with_name(f"{log_path.name}.1"))
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
log_path = DATA_DIR / "logs" / "agent_stdout.log"
|
||||
try:
|
||||
written = log_path.stat().st_size if log_path.exists() else 0
|
||||
except OSError:
|
||||
written = 0
|
||||
handle = None
|
||||
try:
|
||||
handle = open(log_path, "a", encoding="utf-8")
|
||||
for line in iter(proc.stdout.readline, b""):
|
||||
decoded = line.decode("utf-8", errors="replace")
|
||||
if written + len(decoded) > max_bytes:
|
||||
handle.close()
|
||||
handle = None
|
||||
_rotate(log_path)
|
||||
handle = open(log_path, "a", encoding="utf-8")
|
||||
written = 0
|
||||
handle.write(decoded)
|
||||
handle.flush()
|
||||
written += len(decoded)
|
||||
except Exception:
|
||||
pass
|
||||
finally:
|
||||
if handle is not None:
|
||||
try:
|
||||
handle.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
threading.Thread(target=_stream_output, daemon=True).start()
|
||||
# The server blocks once its output pipe fills, so the copy drains until the pipe ends.
|
||||
threading.Thread(target=copy_capped_output, args=(proc.stdout, DATA_DIR / "logs" / "agent_stdout.log"),
|
||||
kwargs={"report": log}, name="agent-output-copy", daemon=True).start()
|
||||
return proc
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -127,10 +127,15 @@ def render_cash(total: tuple) -> dict:
|
|||
return dict(zip((*CASH_KEYS, "accounted_usd"), map(float, rounded_cash(total))))
|
||||
|
||||
|
||||
def exceeds_limit(total: tuple, limit, bound=None, *, dispatch=False) -> bool:
|
||||
"""Admission uses rounded buckets and the existing nanodollar allowance."""
|
||||
def exceeds_limit(total: tuple, limit) -> bool:
|
||||
"""Whether KNOWN spend has reached ``limit`` (owner Q4-A, #1487).
|
||||
|
||||
Known is the settled bucket: confirmed prices plus disclosed estimates.
|
||||
Reservations and unresolved bounds stay in the accounted exposure for
|
||||
display; they are not counted as spending and refuse nothing here. The
|
||||
same predicate guards reservation and dispatch, so a crossing in between
|
||||
refuses without sending. Concurrent and late charges can still overshoot.
|
||||
"""
|
||||
with exact_money():
|
||||
cap, spent, allowance = decimal_of(limit), rounded_cash(total)[-1], Decimal("1e-9")
|
||||
return cap <= 0 or (spent > cap + allowance if dispatch else
|
||||
spent >= cap - allowance or
|
||||
bound is not None and spent + decimal_of(bound) > cap + allowance)
|
||||
cap, known, allowance = decimal_of(limit), rounded_cash(total)[0], Decimal("1e-9")
|
||||
return cap <= 0 or known >= cap - allowance
|
||||
|
|
|
|||
|
|
@ -455,11 +455,12 @@ def _summary(rows: Sequence[Dict[str, Any]]) -> Dict[str, Any]:
|
|||
|
||||
|
||||
def _with_limit(summary: Dict[str, Any], limit: Optional[float]) -> Dict[str, Any]:
|
||||
"""Decorate a summary with its configured limit and remaining headroom."""
|
||||
"""Decorate a summary with its configured limit and the room left above KNOWN
|
||||
(settled) spend — the admission rule's own number; open holds stay beside it."""
|
||||
if limit is None:
|
||||
return summary
|
||||
summary["limit_usd"] = round(max(0.0, float(limit)), 6)
|
||||
summary["remaining_known_usd"] = round(max(0.0, summary["limit_usd"] - float(summary["accounted_usd"])), 6)
|
||||
summary["remaining_known_usd"] = round(max(0.0, summary["limit_usd"] - float(summary["settled_usd"])), 6)
|
||||
return summary
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -783,8 +783,8 @@ class OuroborosAgent:
|
|||
budget_root = pathlib.Path(budget_root_text) if budget_root_text else self.env.drive_root
|
||||
total_budget = resolve_total_budget_usd()
|
||||
projection = usage_projection(budget_root, global_limit_usd=total_budget)
|
||||
if total_budget is not None:
|
||||
budget_remaining = max(0.0, total_budget - float(projection.get("accounted_usd") or 0.0))
|
||||
if total_budget is not None: # room above KNOWN spend, the admission rule's own number (#1487)
|
||||
budget_remaining = max(0.0, total_budget - float(projection.get("settled_usd") or 0.0))
|
||||
except Exception:
|
||||
budget_accounting_status = "unavailable"
|
||||
log.error("Budget authority unavailable while building task context", exc_info=True)
|
||||
|
|
|
|||
|
|
@ -495,7 +495,7 @@ def check_budget(env: Any) -> Tuple[dict, int]:
|
|||
from ouroboros.usage_accounting import usage_projection
|
||||
|
||||
accounting = usage_projection(accounting_root, global_limit_usd=total_budget)
|
||||
spent = float(accounting.get("accounted_usd") or 0.0)
|
||||
spent = float(accounting.get("settled_usd") or 0.0)
|
||||
remaining = float(accounting.get("remaining_known_usd") or 0.0)
|
||||
integrity_degraded = bool(accounting.get("integrity_degraded"))
|
||||
|
||||
|
|
|
|||
|
|
@ -1225,11 +1225,14 @@ def load_budget_pause(ctx: Any, handoff: Optional[Dict[str, Any]] = None) -> Dic
|
|||
|
||||
def _refresh_planning_threshold(ctx: Any, budget_remaining_usd: Optional[float],
|
||||
usage: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
|
||||
"""Owner Q10: after an explicit Resume of a GRACEFUL stop, the planning
|
||||
threshold moves forward within the money still authorized, so the task is
|
||||
not paused again on the very number that paused it. The hard tree cap and
|
||||
the global ledger fence are untouched; the planning margin is what the
|
||||
owner's explicit act spends. Returns the disclosure row.
|
||||
"""Owner Q10: after an explicit Resume of a GRACEFUL stop, an explicit
|
||||
``cost_hard_stop_pct`` profile's planning threshold moves forward within
|
||||
the money still authorized, so the task is not paused again on the very
|
||||
number that paused it. The hard tree cap and the global ledger fence are
|
||||
untouched; the planning margin is what the owner's explicit act spends.
|
||||
Only that explicit authority moves: a producer allowance (a wake's daily
|
||||
remainder) is never widened because a cap or the wallet is larger, and a
|
||||
disabled or unknown ceiling is never re-armed. Returns the disclosure row.
|
||||
|
||||
Every number is read from the AUTHORITATIVE ledger NOW: the global wallet
|
||||
from the usage projection, the tree's cumulative spend and its ACTUAL
|
||||
|
|
@ -1246,8 +1249,18 @@ def _refresh_planning_threshold(ctx: Any, budget_remaining_usd: Optional[float],
|
|||
from ouroboros.loop_budget import _loop_tree_accounting, _wrapup_global_remaining
|
||||
|
||||
old = getattr(ctx, "_cost_ceiling", None)
|
||||
if not isinstance(old, task_pacing.CostCeiling):
|
||||
return {"refreshed": False, "reason": "no_ceiling"}
|
||||
if not isinstance(old, task_pacing.CostCeiling) or old.state not in {
|
||||
task_pacing.COST_CEILING_ACTIVE, task_pacing.COST_CEILING_EXHAUSTED_SOFT_LAND}:
|
||||
return {"refreshed": False, "reason": "no_ceiling",
|
||||
**({"ceiling_state": old.state, "ceiling_basis": old.basis}
|
||||
if isinstance(old, task_pacing.CostCeiling) else {})}
|
||||
authority = task_pacing.cost_stop_authority(ctx)
|
||||
if authority != task_pacing.COST_STOP_EXPLICIT:
|
||||
# A producer allowance, or an inherited number whose author is unverified,
|
||||
# keeps its existing bound: Resume never widens it.
|
||||
reason = ("producer_allowance_not_expanded" if authority == task_pacing.COST_STOP_PRODUCER
|
||||
else "policy_unverified_not_expanded")
|
||||
return {"refreshed": False, "reason": reason, "authority": authority, "ceiling_usd": old.ceiling_usd}
|
||||
try:
|
||||
fresh = _wrapup_global_remaining()
|
||||
except Exception:
|
||||
|
|
@ -1267,7 +1280,7 @@ def _refresh_planning_threshold(ctx: Any, budget_remaining_usd: Optional[float],
|
|||
return {"refreshed": False, "reason": "tree_spend_unavailable", "wallet_basis": "ledger_projection"}
|
||||
if tree.get("integrity_degraded"):
|
||||
return {"refreshed": False, "reason": "tree_accounting_degraded", "wallet_basis": "ledger_projection"}
|
||||
if tree.get("accounted_usd") is None:
|
||||
if tree.get("settled_usd") is None:
|
||||
return {"refreshed": False, "reason": "tree_spend_unknown", "wallet_basis": "ledger_projection"}
|
||||
if tree_cap is not None:
|
||||
root_cap, root_cap_basis = float(tree_cap), "root_accounting"
|
||||
|
|
@ -1276,7 +1289,7 @@ def _refresh_planning_threshold(ctx: Any, budget_remaining_usd: Optional[float],
|
|||
usage = usage if isinstance(usage, dict) else (getattr(ctx, "_accumulated_usage", None) or {})
|
||||
task_cost = usage.get("cost")
|
||||
deciding, basis = task_pacing.resolve_deciding_spend(
|
||||
tree_cost_usd=tree.get("accounted_usd") if tree else None,
|
||||
tree_cost_usd=tree.get("settled_usd") if tree else None,
|
||||
task_cost_usd=float(task_cost) if task_cost is not None else None,
|
||||
root_cap_usd=root_cap,
|
||||
)
|
||||
|
|
@ -1287,9 +1300,9 @@ def _refresh_planning_threshold(ctx: Any, budget_remaining_usd: Optional[float],
|
|||
if root_cap is not None:
|
||||
components.append(root_cap - spent)
|
||||
if float(fresh) > 0:
|
||||
profile = task_pacing.resolve_budget_profile(ctx)
|
||||
pct = profile.get("cost_hard_stop_pct")
|
||||
pct = task_pacing._DEFAULT_COST_HARD_STOP_PCT if pct is None else max(0, min(100, int(pct)))
|
||||
# The authored percentage: the task's own, or the root's it was resolved under.
|
||||
profile = task_pacing.cost_stop_policy(ctx)["profile"]
|
||||
pct = max(0, min(100, int(profile.get("cost_hard_stop_pct") or 0)))
|
||||
if pct > 0:
|
||||
components.append(float(fresh) * pct / 100.0)
|
||||
room = min(components) if components else None
|
||||
|
|
@ -1438,10 +1451,10 @@ def resume_paused_loop(tools: Any, state: Dict[str, Any], messages: list, trace:
|
|||
# that no longer exists, and the resumed attempt registers its own.
|
||||
forget_tool_scope(ctx)
|
||||
setattr(ctx, "_budget_pausing", False)
|
||||
if state.get("cost_ceiling") is not None:
|
||||
from ouroboros.task_pacing import CostCeiling
|
||||
from ouroboros.task_pacing import restore_cost_ceiling
|
||||
|
||||
ctx._cost_ceiling = CostCeiling(**state["cost_ceiling"])
|
||||
# The start's own authority, never the saved number alone (#1128).
|
||||
ctx._cost_ceiling = restore_cost_ceiling(ctx, state.get("cost_ceiling"))
|
||||
refresh: Dict[str, Any] = {"refreshed": False, "reason": "hard_rail"}
|
||||
graceful = str(row.get("rail") or "") in GRACEFUL_RAILS
|
||||
if graceful:
|
||||
|
|
@ -1484,6 +1497,12 @@ def resume_paused_loop(tools: Any, state: Dict[str, Any], messages: list, trace:
|
|||
return (ctx.active_model, ctx.active_effort, ctx.active_use_local,
|
||||
mode, int(state["round_idx"]), plan)
|
||||
kind = "owner Pause" if str(row.get("reason") or "") == REASON_OWNER else "budget pause"
|
||||
held = ""
|
||||
if refresh.get("reason") == "producer_allowance_not_expanded" and refresh.get("ceiling_usd") is not None:
|
||||
# Said, not implied: Resume keeps the allowance the task was launched with.
|
||||
held = (f"; the ${float(refresh['ceiling_usd']):.2f} producer allowance this task was launched with "
|
||||
"still binds, never extended by Resume even if its producer's window has since freed more, "
|
||||
"so known spend still at it pauses the task again before another model call")
|
||||
for call_id in pending:
|
||||
# Transcript validity for the provider AND the honest fact: unknown, not
|
||||
# "did not run" and not "ran". The host never re-executes it.
|
||||
|
|
@ -1495,7 +1514,8 @@ def resume_paused_loop(tools: Any, state: Dict[str, Any], messages: list, trace:
|
|||
messages.append({"role": "user", "content": (
|
||||
f"[SYSTEM NOTICE]\nThis task continued from its {kind} after an explicit owner Resume "
|
||||
f"(paused {ctx._budget_paused_sec:.0f}s; rail: {row.get('rail')}; planning threshold "
|
||||
f"{'refreshed to $%.2f' % refresh['ceiling_usd'] if refresh.get('refreshed') else 'not refreshed: ' + str(refresh.get('reason'))}). "
|
||||
f"{'refreshed to $%.2f' % refresh['ceiling_usd'] if refresh.get('refreshed') else 'not refreshed: ' + str(refresh.get('reason'))}"
|
||||
f"{' (' + str(refresh['ceiling_basis']) + ')' if refresh.get('ceiling_basis') else ''}{held}). "
|
||||
"Cumulative spend, rounds and elapsed execution time were NOT reset. Prior tool results remain "
|
||||
"recorded; do not repeat completed effects. The pause ended the previous browser process and "
|
||||
"task-local services; their recorded results remain evidence, not proof they are still running. "
|
||||
|
|
|
|||
|
|
@ -8,12 +8,13 @@ registry, no observation inbox, no pause/resume: the supervisor loop calls ``tic
|
|||
per pass and the ``DirectActivityRegistry`` (the census of live direct turns) is the only
|
||||
liveness truth. Tick order, each step a typed outcome: disabled → a live wake → a live owner
|
||||
turn → not yet due → the rolling-24h allowance (an unreadable ledger is the disclosed skip
|
||||
``allowance_unknown``, never a silent block; a remainder at or below the graceful stop's planning
|
||||
margin counts as exhausted — less than one planned turn) → an owner chat must be bound → launch.
|
||||
What is left of that allowance is the launched tree's graceful ceiling
|
||||
(``metadata.root_cost_ceiling_usd``, В26=A): the in-task stop lands the wake a planning margin
|
||||
before it, while the ledger fence stays at the owner's per-task cap, so a wake never dies before
|
||||
its first call on a nearly spent day. The next
|
||||
``allowance_unknown``, never a silent block; known spend at the daily limit is
|
||||
``allowance_exhausted``) → an owner chat must be bound → launch.
|
||||
What is left of that allowance is the launched tree's producer ceiling
|
||||
(``metadata.root_cost_ceiling_usd``, В26=A): the in-task stop pauses the wake when its known
|
||||
spend reaches it — no extra planning margin is subtracted or skipped (owner 2026-10-07) — while the
|
||||
ledger fence stays at the owner's per-task cap, so a wake never dies before its first call on a
|
||||
nearly spent day. The next
|
||||
wake is ``last finish + interval``: the MODEL's choice (``set_next_wakeup`` persists
|
||||
``consciousness_next_interval_sec``) or ``WAKE_DEFAULT_SEC``, clamped into the owner's
|
||||
[min, max]; a runner failure — or a wake the lane could not admit — doubles it up to max until a
|
||||
|
|
@ -55,7 +56,6 @@ from ouroboros.consciousness_allowance import STATUS_EXHAUSTED, STATUS_UNKNOWN,
|
|||
from ouroboros.consciousness_authority import is_consciousness_origin
|
||||
from ouroboros.consciousness_wake import bind_wake_observation, observe_wake, render_wake_message, wake_task_metadata
|
||||
from ouroboros.deadline_utils import parse_deadline_ts
|
||||
from ouroboros.task_pacing import COST_PLANNING_MARGIN_USD
|
||||
from ouroboros.utils import append_jsonl, utc_now_iso
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
|
@ -248,9 +248,9 @@ class BackgroundConsciousness:
|
|||
window = self._allowance_view(now, fresh=True)
|
||||
if window.get("status") == STATUS_UNKNOWN:
|
||||
return self._skip("allowance_unknown", now + self.floor, now=now, error=str(window.get("error") or ""))
|
||||
# Nothing left — or less than one planned turn (the graceful stop's planning
|
||||
# margin): a wake started there would only be told to land at once.
|
||||
if window.get("status") == STATUS_EXHAUSTED or float(window.get("remaining_usd") or 0.0) <= COST_PLANNING_MARGIN_USD:
|
||||
# Known spend reached the daily allowance (#1487: unresolved bounds are
|
||||
# disclosed exposure, not spending). No extra early margin skips a wake.
|
||||
if window.get("status") == STATUS_EXHAUSTED or float(window.get("remaining_usd") or 0.0) <= 0.0:
|
||||
resets = parse_deadline_ts(str(window.get("resets_at") or ""))
|
||||
return self._skip("allowance_exhausted", max(resets.timestamp() if resets else 0.0, now + self.floor), now=now)
|
||||
if not self._owner_chat_id():
|
||||
|
|
@ -274,13 +274,13 @@ class BackgroundConsciousness:
|
|||
per_task_cap = 0.0
|
||||
# В26=A: a wake's whole tree may spend at most what is left of the rolling-24h
|
||||
# allowance, and never more than the owner's per-task cap (a cap of 0 disables
|
||||
# that half). It travels as the tree's GRACEFUL ceiling (`root_cost_ceiling_usd`,
|
||||
# honored for the root itself by `task_pacing.resolve_cost_ceiling` and inherited
|
||||
# by the members): the in-task stop lands a planning margin early, while the
|
||||
# ledger fence keeps the owner's per-task cap — one Main attempt reserves several
|
||||
# dollars up front, and a fence narrowed below that refused every wake of a
|
||||
# nearly spent day before its first call. The tick never launches with less
|
||||
# than the planning margin left (that window is `allowance_exhausted`).
|
||||
# that half). It travels as the tree's producer ceiling (`root_cost_ceiling_usd`,
|
||||
# honored for the root itself by `task_pacing.resolve_cost_ceiling` and by the
|
||||
# members): the in-task stop pauses the tree when its known spend reaches it, with
|
||||
# no planning margin subtracted (owner 2026-10-07), while the ledger fence keeps the
|
||||
# owner's per-task cap — one Main attempt reserves several dollars up front, and a
|
||||
# fence narrowed below that refused every wake of a nearly spent day before its
|
||||
# first call.
|
||||
ceiling = min(per_task_cap, remaining) if per_task_cap > 0 else remaining
|
||||
metadata = {**self._routing_facts(chat_id),
|
||||
**wake_task_metadata(level, reason, root_cost_ceiling_usd=ceiling)}
|
||||
|
|
@ -294,7 +294,7 @@ class BackgroundConsciousness:
|
|||
def render(events: str) -> str:
|
||||
return render_wake_message(
|
||||
self._repo_dir, reason=reason, last_wake_at=self._last_wake_at, now=now, level=level,
|
||||
disabled_tools=list(metadata.get("disabled_tools") or []), spent_usd=window.get("accounted_usd"),
|
||||
disabled_tools=list(metadata.get("disabled_tools") or []), spent_usd=window.get("settled_usd"),
|
||||
spent_is_floor=int(window.get("unknown_unmetered") or 0) > 0,
|
||||
daily_usd=window.get("limit_usd") or 0.0, running=self._running_roots(),
|
||||
max_tasks=get_consciousness_max_tasks(), interval=self._interval(), events=events)
|
||||
|
|
@ -413,7 +413,7 @@ class BackgroundConsciousness:
|
|||
window = self._allowance_view(time.time())
|
||||
values = (self._enabled, get_consciousness_autonomy(), _iso(self._next_wake_at), self._pending_reason or "",
|
||||
_iso(self._last_wake_at), self._last_wake_task_id, self._last_wake_outcome, self._last_error,
|
||||
window.get("accounted_usd"), window.get("limit_usd"), str(window.get("resets_at") or ""),
|
||||
window.get("settled_usd"), window.get("limit_usd"), str(window.get("resets_at") or ""),
|
||||
self._running_roots(), int(get_consciousness_max_tasks()), wake,
|
||||
int(window.get("unknown_unmetered") or 0), bool(window.get("integrity_degraded")))
|
||||
return dict(zip(SNAPSHOT_KEYS, values))
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
Owner decision В11/В18 (PLAN 5.5, 5.13 п.11, 5.14 п.1): the daily allowance
|
||||
gates NEW starts (a wake, a promoted/scheduled root) on the money the whole
|
||||
consciousness tree accounted in the last 24 hours. There is no root index —
|
||||
consciousness tree spent in the last 24 hours. There is no root index —
|
||||
the roots are read off the usage store: every attempt row whose transition
|
||||
timestamp (``ts_last``) lies in the last 48 h and whose category is
|
||||
``consciousness`` (a wake's own rows) or ``consciousness_task`` (a started
|
||||
|
|
@ -12,13 +12,16 @@ root's rows) names a consciousness root; the window is then every money row
|
|||
roots whose transition timestamp lies in the last 24 h, open rows included,
|
||||
reduced by the ONE money reducer ``_usage_rows._summary``. Both selections are
|
||||
addressed (the ``(category, ts_last)`` and ``root_task_id`` indexes), so the
|
||||
answer never depends on unrelated history. ``accounted_usd`` (settled +
|
||||
reserved + unresolved) is the number: a hanging reservation can read as
|
||||
exhausted for its lifetime, disclosed rather than hidden;
|
||||
``unknown_unmetered`` makes it "at least". The TIME filter runs on every call,
|
||||
so an exhausted window frees itself as rows age out without a new write. A
|
||||
store that cannot be read yields the typed ``allowance_unknown`` outcome —
|
||||
the caller refuses the start honestly.
|
||||
answer never depends on unrelated history. ``settled_usd`` — the KNOWN spend,
|
||||
confirmed prices and disclosed estimates — is the number that decides (owner
|
||||
Q4-A, #1487), the same rule as every other money limit: a reservation or the
|
||||
upper bound of an unresolved call is shown beside it as ``accounted_usd``
|
||||
exposure, never counted as spent, so one refused oversized request cannot
|
||||
close the allowance for a day. ``unknown_unmetered`` makes the known spend "at
|
||||
least". The TIME filter runs on every call, so an exhausted window frees
|
||||
itself as rows age out without a new write. A store that cannot be read
|
||||
yields the typed ``allowance_unknown`` outcome — the caller refuses the start
|
||||
honestly.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -69,20 +72,22 @@ def allowance_window(
|
|||
degraded = usage_store.integrity_degraded(root)
|
||||
except Exception as exc: # noqa: BLE001 — every read failure is the one typed outcome
|
||||
return {"status": STATUS_UNKNOWN, "error": f"{type(exc).__name__}: {exc}",
|
||||
"limit_usd": limit, "accounted_usd": None, "remaining_usd": None, "resets_at": ""}
|
||||
"limit_usd": limit, "settled_usd": None, "accounted_usd": None, "remaining_usd": None,
|
||||
"resets_at": ""}
|
||||
window = [(ts, row) for ts, row in ((row_ts_epoch(row), row) for row in rows) if ts is not None]
|
||||
summary = _summary([row for _ts, row in window])
|
||||
accounted = float(summary["accounted_usd"])
|
||||
known = float(summary["settled_usd"])
|
||||
oldest = min((ts for ts, _row in window), default=None)
|
||||
resets_at = (
|
||||
_dt.datetime.fromtimestamp(oldest + WINDOW_SEC, tz=_dt.timezone.utc).isoformat()
|
||||
if oldest is not None else ""
|
||||
)
|
||||
return {
|
||||
"status": STATUS_EXHAUSTED if accounted >= limit else STATUS_AVAILABLE,
|
||||
"status": STATUS_EXHAUSTED if known >= limit else STATUS_AVAILABLE,
|
||||
"limit_usd": limit,
|
||||
"accounted_usd": accounted,
|
||||
"remaining_usd": round(max(0.0, limit - accounted), 6),
|
||||
"settled_usd": known,
|
||||
"accounted_usd": float(summary["accounted_usd"]),
|
||||
"remaining_usd": round(max(0.0, limit - known), 6),
|
||||
"unknown_unmetered": int(summary["unknown_unmetered"]),
|
||||
"non_final_rows": int(summary["non_final_rows"]),
|
||||
"roots": sorted(roots),
|
||||
|
|
|
|||
|
|
@ -196,10 +196,13 @@ def _runtime_budget_info(env: Any, task: Dict[str, Any], ctx: Any = None) -> Dic
|
|||
total_usd = resolve_total_budget_usd()
|
||||
budget_root = pathlib.Path(task.get("budget_drive_root") or env.drive_root)
|
||||
projection = usage_projection(budget_root, global_limit_usd=total_usd)
|
||||
spent_usd = float(projection.get("accounted_usd") or 0.0)
|
||||
# Known spend (confirmed + estimated) is what the limit decides on; open
|
||||
# holds ride beside it, never added in (#1487).
|
||||
spent_usd = float(projection.get("settled_usd") or 0.0)
|
||||
budget_info = {
|
||||
"status": "available" if total_usd is not None else "no_global_limit", "total_usd": total_usd,
|
||||
"spent_usd": spent_usd, "remaining_usd": None if total_usd is None else total_usd - spent_usd,
|
||||
"spent_usd": spent_usd, "spent_basis": "known_settled_incl_estimates",
|
||||
"remaining_usd": None if total_usd is None else max(0.0, total_usd - spent_usd),
|
||||
"reserved_usd": float(projection.get("reserved_usd") or 0.0),
|
||||
"unresolved_upper_bound_usd": float(projection.get("unresolved_upper_bound_usd") or 0.0),
|
||||
"unknown_unmetered": int(projection.get("unknown_unmetered") or 0),
|
||||
|
|
@ -215,8 +218,13 @@ def _runtime_budget_info(env: Any, task: Dict[str, Any], ctx: Any = None) -> Dic
|
|||
budget_info["per_task_tree_cap_usd"] = root_cap
|
||||
budget_info["per_task_tree_cap_rule"] = (
|
||||
"Hard cap for THIS task's WHOLE tree (own model calls + all subagents), enforced "
|
||||
"by the physical-attempt ledger: dispatches are refused once the tree's accounted "
|
||||
"spend reaches it and the task is force-stopped. Budget checkpoints during the task report the live tree number."
|
||||
"by the physical-attempt ledger: new dispatches are refused once the tree's KNOWN "
|
||||
"spend (confirmed and estimated) reaches it. A task with an exact continuation then "
|
||||
"pauses with its work saved; an actor without one (no continuation owner, task id or "
|
||||
"durable root) ends with a typed budget_exhausted result and no paid recap. "
|
||||
"Open reservations and unresolved charges are not counted as spending, and calls "
|
||||
"in flight or charged late can take the tree past the cap. Budget checkpoints "
|
||||
"during the task report the live tree number."
|
||||
)
|
||||
if ctx is not None:
|
||||
budget_info["in_task_cost_ceiling"] = _in_task_cost_ceiling(ctx, budget_info.get("remaining_usd"))
|
||||
|
|
|
|||
|
|
@ -204,12 +204,13 @@ def _opt_pct(value: Any) -> Any:
|
|||
|
||||
|
||||
def _opt_cost_hard_stop_pct(value: Any) -> Any:
|
||||
"""Like ``_opt_pct`` but FAIL-SAFE for the one percentage whose 0 is the
|
||||
maximally-permissive setting (0 = NO in-task cost stop). A malformed value
|
||||
must NOT silently collapse to 0 and disable the safety stop: a negative
|
||||
number, a non-numeric, or a ``0 < v < 1`` fraction (a likely fraction-vs-
|
||||
percent mix-up, e.g. 0.5 meaning "half") maps to None — the historical 50%
|
||||
default — not to 0. An explicit 0 / 0.0 / "0" is honored verbatim."""
|
||||
"""Like ``_opt_pct`` but never collapsing to 0 for the one percentage whose 0
|
||||
is an explicit bench contract (0 = NO in-task cost stop). A negative number,
|
||||
a non-numeric, or a ``0 < v < 1`` fraction (a likely fraction-vs-percent
|
||||
mix-up, e.g. 0.5 meaning "half") maps to None — no explicit percentage, the
|
||||
same as an absent one — not to 0. An explicit 0 / 0.0 / "0" is honored
|
||||
verbatim. The accepted-input contract is unchanged since v6.56.0; only the
|
||||
meaning of None changed (owner 2026-10-07): it is no longer a 50% stop."""
|
||||
if value is None or (isinstance(value, str) and not value.strip()):
|
||||
return None
|
||||
try:
|
||||
|
|
@ -235,17 +236,19 @@ def normalize_budget_profile(value: Any) -> Dict[str, Any]:
|
|||
adaptive (passes stop early when the remaining window can no longer fit a
|
||||
review comfortably).
|
||||
|
||||
``cost_hard_stop_pct`` (v6.56.0, additive): the in-task cost hard-stop as a
|
||||
percentage of the budget remaining at task start. None -> the historical
|
||||
default (50: the global component of the stop is half the remaining
|
||||
budget). 0 -> NO in-task cost stop at all — the deadline/rounds axes and the
|
||||
global between-task budget gate remain the only bounds, and cost milestones
|
||||
become informational against the start snapshot. The ceiling is resolved in
|
||||
``task_pacing.resolve_cost_ceiling`` (typed; 0 maps to the ``disabled``
|
||||
state, never a $0 ceiling; a per-task root cap contributes a second
|
||||
min-component). A MALFORMED value (negative / non-numeric / a ``0<v<1``
|
||||
fraction) maps to None (the 50% default), NOT to 0 — it must not silently
|
||||
disable the stop (see ``_opt_cost_hard_stop_pct``).
|
||||
``cost_hard_stop_pct`` (v6.56.0, additive): an EXPLICIT experiment's
|
||||
in-task cost stop as a percentage of the budget remaining at task start.
|
||||
None -> no explicit percentage: an ordinary task has no early cost stop of
|
||||
the host's own (owner 2026-10-07, #1128 — formerly the historical 50%
|
||||
default); new paid calls are refused only when known spend reaches the real
|
||||
global/root/group limits, and a producer's allowance (a consciousness wake)
|
||||
still stops at its actual value. A positive value keeps the authored math
|
||||
(that share of the starting wallet, min the per-task cap minus a planning
|
||||
margin). 0 -> NO in-task cost stop at all (bench contract). The ceiling is
|
||||
resolved in ``task_pacing.resolve_cost_ceiling`` (typed; 0 and None map to
|
||||
``disabled``, never a $0 ceiling). A MALFORMED value (negative /
|
||||
non-numeric / a ``0<v<1`` fraction) maps to None like an absent one, never
|
||||
to 0 (see ``_opt_cost_hard_stop_pct``).
|
||||
"""
|
||||
v = value if isinstance(value, Mapping) else {}
|
||||
policy = str(v.get("improvement_policy") or "").strip().lower()
|
||||
|
|
|
|||
|
|
@ -870,7 +870,6 @@ allowed = [
|
|||
"D11->D18",
|
||||
"D11->D19",
|
||||
"D11->D20",
|
||||
"D12->D01",
|
||||
"D12->D02",
|
||||
"D12->D03",
|
||||
"D12->D07",
|
||||
|
|
@ -993,6 +992,7 @@ lazy_only = [
|
|||
"D10->D17",
|
||||
"D11->D06",
|
||||
"D11->D15",
|
||||
"D12->D01",
|
||||
"D12->D05",
|
||||
"D12->D06",
|
||||
"D12->D10",
|
||||
|
|
|
|||
|
|
@ -551,7 +551,8 @@ class UpdateApplyErrorResponse(TypedDict):
|
|||
merge_plan: NotRequired[UpdateMergePlan]
|
||||
smoke: NotRequired[Dict[str, Any]]
|
||||
# Stash-first prologue disclosures (additive): how the owner's stashed work
|
||||
# was unwound on an aborted update, and the wave-floor admission numbers.
|
||||
# was unwound on an aborted update. The wave numbers rode the retired
|
||||
# wave-floor refusal (#1487 removed it); the optional keys stay in the shape.
|
||||
stash_note: NotRequired[str]
|
||||
estimated_wave_usd: NotRequired[Optional[float]]
|
||||
remaining_usd: NotRequired[Optional[float]]
|
||||
|
|
|
|||
|
|
@ -723,15 +723,16 @@ def _start_assisted_merge_fenced(plan: dict, tx: dict) -> JSONResponse:
|
|||
**({"stash_note": note} if note else {})},
|
||||
status_code=409,
|
||||
)
|
||||
# Affordability floor: a resolution that cannot buy even ONE full triad+scope
|
||||
# review wave would mutate the live tree into a conflicted merge and then
|
||||
# stall mid-review. "One full wave" is priced HONESTLY at the review packs'
|
||||
# own worst-case caps — the shared 920K-token input SSOT per API row, the
|
||||
# triad's default output reserve, and the scope reviewer's 100K output
|
||||
# reserve — with the shared reservation math (agent-session rows ride
|
||||
# subscriptions, not USD budget); fail-open on estimator errors, mirroring
|
||||
# review_wave_admission's own contract.
|
||||
admission = {"fits": True}
|
||||
# Money admission is the known-spend rule every reviewer seat's reservation
|
||||
# applies (#1487), checked just above: known spend below TOTAL_BUDGET admits
|
||||
# the update, and a review that later reaches the limit pauses on its own
|
||||
# fences. One full triad+scope wave is still priced at the review packs' own
|
||||
# worst-case caps — the shared 920K-token input SSOT per API row, the triad's
|
||||
# default output reserve, and the scope reviewer's 100K output reserve — but
|
||||
# as DISCLOSURE, never a second, earlier refusal: a prospective wave larger
|
||||
# than the remainder used to refuse an update the owner's limit still
|
||||
# allowed. Agent-session rows ride subscriptions, not USD budget; an
|
||||
# estimator error is recorded, never a zero.
|
||||
try:
|
||||
from ouroboros.reviewer_slot_config import commit_scope_rows, commit_triad_rows
|
||||
from ouroboros.tools.review_helpers import REVIEW_PROMPT_TOKEN_BUDGET
|
||||
|
|
@ -740,10 +741,7 @@ def _start_assisted_merge_fenced(plan: dict, tx: dict) -> JSONResponse:
|
|||
# Native-retrieving actor rows (subagent_id + api route) are priced at
|
||||
# the SAME one-pack-call convention as packet rows: their true worst
|
||||
# case is bounded by the episode's own rails (round cap x transcript
|
||||
# cap) and can exceed this estimate, but the typical episode is
|
||||
# pack-sized or smaller, and this floor is an explicitly fail-open
|
||||
# affordability heuristic — over-refusing assisted updates on a
|
||||
# theoretical ceiling would cost more than it protects.
|
||||
# cap) and can exceed this estimate.
|
||||
triad_models = [
|
||||
row.target_id for row in commit_triad_rows()
|
||||
if not row.is_session and row.target_id
|
||||
|
|
@ -778,70 +776,33 @@ def _start_assisted_merge_fenced(plan: dict, tx: dict) -> JSONResponse:
|
|||
session_slots = sum(
|
||||
1 for row in (*commit_triad_rows(), *commit_scope_rows()) if row.is_session
|
||||
)
|
||||
if any_estimate:
|
||||
admission = {
|
||||
"fits": estimated_total <= float(remaining) + 1e-9,
|
||||
"estimated_wave_usd": round(estimated_total, 6),
|
||||
"remaining_usd": float(remaining),
|
||||
"unpriced_slots": unpriced_total,
|
||||
"session_slots": session_slots,
|
||||
}
|
||||
if admission.get("fits", True) and (unpriced_total or (session_slots and not any_estimate)):
|
||||
# An ADMITTED wave with unknowable parts must not read as a fully
|
||||
# priced estimate later (P1: represent the gap) — one durable line.
|
||||
try:
|
||||
from supervisor.git_ops import DRIVE_ROOT as _dr
|
||||
from ouroboros.utils import append_jsonl as _aj, utc_now_iso as _n
|
||||
try:
|
||||
from supervisor.git_ops import DRIVE_ROOT as _dr
|
||||
from ouroboros.utils import append_jsonl as _aj, utc_now_iso as _n
|
||||
|
||||
_aj(_dr / "logs" / "supervisor.jsonl", {
|
||||
"ts": _n(), "type": "managed_update_wave_floor_partial_unknown",
|
||||
"estimated_wave_usd": admission.get("estimated_wave_usd"),
|
||||
"unpriced_slots": unpriced_total, "session_slots": session_slots,
|
||||
"remaining_usd": float(remaining),
|
||||
})
|
||||
except Exception:
|
||||
log.debug("wave-floor partial-unknown event write failed", exc_info=True)
|
||||
# One durable disclosure of the wave estimate beside the known room;
|
||||
# unknowable parts are counted, never filled in (P1).
|
||||
_aj(_dr / "logs" / "supervisor.jsonl", {
|
||||
"ts": _n(), "type": "managed_update_wave_estimate",
|
||||
"estimated_wave_usd": round(estimated_total, 6) if any_estimate else None,
|
||||
"exceeds_known_remaining": bool(any_estimate and estimated_total > float(remaining) + 1e-9),
|
||||
"unpriced_slots": unpriced_total, "session_slots": session_slots,
|
||||
"remaining_usd": float(remaining),
|
||||
})
|
||||
except Exception:
|
||||
log.debug("wave estimate event write failed", exc_info=True)
|
||||
except Exception:
|
||||
log.debug("assisted admission wave estimate failed open", exc_info=True)
|
||||
admission = {"fits": True}
|
||||
log.debug("assisted admission wave estimate failed", exc_info=True)
|
||||
try:
|
||||
from supervisor.git_ops import DRIVE_ROOT as _dr2
|
||||
from ouroboros.utils import append_jsonl as _aj2, utc_now_iso as _n2
|
||||
|
||||
_aj2(_dr2 / "logs" / "supervisor.jsonl", {
|
||||
"ts": _n2(), "type": "managed_update_wave_floor_estimator_failed",
|
||||
"ts": _n2(), "type": "managed_update_wave_estimate_failed",
|
||||
"remaining_usd": float(remaining),
|
||||
})
|
||||
except Exception:
|
||||
log.debug("estimator-failure event write failed", exc_info=True)
|
||||
if not admission.get("fits", True):
|
||||
note = _unwind_stashed_update(tx, "assisted_admission_failed")
|
||||
_respawn_workers_after_failed_update()
|
||||
estimated = admission.get("estimated_wave_usd")
|
||||
try:
|
||||
from supervisor.git_ops import DRIVE_ROOT
|
||||
from ouroboros.utils import append_jsonl, utc_now_iso as _now
|
||||
|
||||
append_jsonl(DRIVE_ROOT / "logs" / "supervisor.jsonl", {
|
||||
"ts": _now(), "type": "managed_update_wave_floor_refused",
|
||||
"estimated_wave_usd": estimated, "remaining_usd": admission.get("remaining_usd"),
|
||||
})
|
||||
except Exception:
|
||||
log.debug("wave-floor refusal event write failed", exc_info=True)
|
||||
return JSONResponse(
|
||||
{"error": (
|
||||
"Assisted update needs enough model budget for at least one full "
|
||||
f"review wave ({'at least ' if admission.get('unpriced_slots') else ''}≈${estimated} "
|
||||
"estimated at the review packs' worst-case caps for the configured reviewer panel"
|
||||
+ (f"; {admission['unpriced_slots']} slot(s) unpriced" if admission.get("unpriced_slots") else "")
|
||||
+ f", ${round(float(remaining), 2)} remaining); nothing was changed."
|
||||
),
|
||||
"estimated_wave_usd": estimated,
|
||||
"remaining_usd": admission.get("remaining_usd"),
|
||||
"unpriced_slots": admission.get("unpriced_slots", 0),
|
||||
**({"stash_note": note} if note else {})},
|
||||
status_code=409,
|
||||
)
|
||||
|
||||
_create_rescue_snapshot(
|
||||
branch, "ui_update_assisted_merge", _collect_repo_sync_state(),
|
||||
|
|
|
|||
|
|
@ -115,8 +115,9 @@ def make_cost_breakdown_endpoint(data_dir: pathlib.Path):
|
|||
"available": True,
|
||||
"authority": "physical_attempt_ledger",
|
||||
"limit_usd": round(limit, 6),
|
||||
# Room above KNOWN (settled) spend, the admission rule's own number.
|
||||
"remaining_known_usd": (
|
||||
round(max(0.0, limit - float(breakdown.get("accounted_usd") or 0.0)), 6)
|
||||
round(max(0.0, limit - float(breakdown.get("settled_usd") or 0.0)), 6)
|
||||
if limit > 0
|
||||
else None
|
||||
),
|
||||
|
|
|
|||
|
|
@ -178,10 +178,10 @@ def _state_snapshot(request: Request) -> Dict[str, Any]:
|
|||
log.exception("Physical-attempt accounting unavailable for /api/state")
|
||||
accounting_available = False
|
||||
breakdown, accounting = {}, {}
|
||||
# Compatibility header/bar uses the conservative dispatch authority:
|
||||
# settled + live reservations + unresolved upper bounds. Actual paid
|
||||
# cost remains separately visible as accounting.settled_usd/confirmed.
|
||||
spent = float(accounting.get("accounted_usd") or 0.0) if accounting_available else None
|
||||
# Compatibility header/bar shows the dispatch authority's own number: KNOWN
|
||||
# spend (settled: confirmed + estimated, #1487). Reservations and unresolved
|
||||
# upper bounds stay separately visible in the accounting block.
|
||||
spent = float(accounting.get("settled_usd") or 0.0) if accounting_available else None
|
||||
# De-triplication: hand the evolution snapshot the projection this request
|
||||
# already computed, so budget_remaining does not replay the ledger again —
|
||||
# but ONLY when this request's drive root IS the supervisor's root and the
|
||||
|
|
|
|||
|
|
@ -447,9 +447,10 @@ def _execute_task_acceptance_panel(ctx: _TaskAcceptanceContext) -> Any:
|
|||
)
|
||||
if _admission is not None:
|
||||
return _refused(
|
||||
"review_wave_budget_insufficient: estimated "
|
||||
f"~${_admission.get('estimated_wave_usd')} > remaining "
|
||||
f"${_admission.get('remaining_usd')} (no reviewer was called)"
|
||||
"review_wave_budget_insufficient: known spend has reached the "
|
||||
f"{_admission.get('binding_axis') or 'budget'} limit (remaining "
|
||||
f"${_admission.get('remaining_usd')}; wave estimated ~${_admission.get('estimated_wave_usd')}; "
|
||||
"no reviewer was called)"
|
||||
)
|
||||
free_result = _free_dispatch(request, slots, drive_root=drive_root, usage_ctx=ctx.tools._ctx)
|
||||
if free_result is not None:
|
||||
|
|
|
|||
|
|
@ -93,7 +93,7 @@ def _check_budget_limits(
|
|||
# global-share ceiling is global money.
|
||||
pause_scope = "root" if cost_ceiling.root_cap_usd is not None else "global"
|
||||
tree_info = _loop()._loop_tree_accounting(refresh=True, max_age_sec=_loop()._TREE_ACCOUNTING_MAX_STALE_SEC)
|
||||
tree_cost = tree_info.get("accounted_usd") if isinstance(tree_info, dict) else None
|
||||
tree_cost = tree_info.get("settled_usd") if isinstance(tree_info, dict) else None
|
||||
deciding, spend_basis = task_pacing.resolve_deciding_spend(
|
||||
tree_cost_usd=tree_cost,
|
||||
task_cost_usd=task_cost,
|
||||
|
|
@ -110,9 +110,14 @@ def _check_budget_limits(
|
|||
# very number that paused the task; the ledger fence at the full cap still
|
||||
# arbitrates every send. A stop that needs no wrap-up room at all
|
||||
# (``wrapup_fits is False``) is unchanged.
|
||||
last_fit_relaxed = bool(getattr(getattr(getattr(ctx, "tools", None), "_ctx", None),
|
||||
"_budget_resume_last_fit_relaxed", False))
|
||||
if prompt_estimate > 0 and (global_remaining is not None or (cost_ceiling.root_cap_usd is not None and deciding is not None)):
|
||||
tool_ctx = getattr(getattr(ctx, "tools", None), "_ctx", None)
|
||||
last_fit_relaxed = bool(getattr(tool_ctx, "_budget_resume_last_fit_relaxed", False))
|
||||
# The prospective wrap-up probes are an explicit profile's authored margin. A
|
||||
# producer allowance stops when known spend reaches it, never one call early.
|
||||
authority = task_pacing.COST_STOP_EXPLICIT if tool_ctx is None else task_pacing.cost_stop_authority(tool_ctx)
|
||||
explicit = authority == task_pacing.COST_STOP_EXPLICIT
|
||||
if explicit and prompt_estimate > 0 and (
|
||||
global_remaining is not None or (cost_ceiling.root_cap_usd is not None and deciding is not None)):
|
||||
finish_reason = task_pacing.wrapup_last_fit_text(deciding, cost_ceiling, global_remaining)
|
||||
forced_prompt = f"[BUDGET LIMIT] {finish_reason} {_loop()._FORCED_BEST_EFFORT_TAIL}"
|
||||
request_args = dict(model=ctx.active_model, prompt_tokens=prompt_estimate,
|
||||
|
|
@ -187,17 +192,21 @@ def _check_budget_limits(
|
|||
# Presence arm the prepared prompt set is withdrawn: it would silence the ordinary reply.
|
||||
if tools_ctx is not None:
|
||||
tools_ctx._presence_forced_declaration, tools_ctx._presence_forced_pending = presence_arm
|
||||
if deciding is not None and ceiling_usd is not None and deciding > ceiling_usd:
|
||||
# A producer allowance is spent once known spend REACHES it, like every ledger
|
||||
# limit; an explicit profile keeps its authored edge (strictly over its point).
|
||||
reached = deciding is not None and ceiling_usd is not None and (
|
||||
deciding >= ceiling_usd if authority == task_pacing.COST_STOP_PRODUCER else deciding > ceiling_usd)
|
||||
if reached:
|
||||
if spend_basis == task_pacing.SPEND_BASIS_TREE:
|
||||
spent_text = (
|
||||
f"Task tree spent ${deciding:.3f} (ledger-accounted incl. in-flight holds, "
|
||||
f"subagents included; own calls ${task_cost:.3f})"
|
||||
f"Task tree spent ${deciding:.3f} (known spend, subagents included; "
|
||||
f"own calls ${task_cost:.3f})"
|
||||
if task_cost is not None
|
||||
else f"Task tree spent ${deciding:.3f} (ledger-accounted incl. in-flight holds)"
|
||||
else f"Task tree spent ${deciding:.3f} (known spend, subagents included)"
|
||||
)
|
||||
elif spend_basis == task_pacing.SPEND_BASIS_OWN_TREE_UNKNOWN:
|
||||
spent_text = (
|
||||
f"Task spent ${deciding:.3f} on its OWN calls (the tree-accounted total "
|
||||
f"Task spent ${deciding:.3f} on its OWN calls (the tree's known total "
|
||||
"is unavailable right now, so subagent spend is not included — this is a "
|
||||
"lower bound)"
|
||||
)
|
||||
|
|
@ -208,12 +217,22 @@ def _check_budget_limits(
|
|||
if cost_ceiling.root_cap_usd is not None else ""
|
||||
)
|
||||
finish_reason = (
|
||||
f"{spent_text}, over the in-task cost ceiling ${ceiling_usd:.2f}{cap_text}. "
|
||||
"Budget exhausted."
|
||||
f"{spent_text}, {'over' if deciding > ceiling_usd else 'at'} the in-task cost ceiling ${ceiling_usd:.2f} "
|
||||
f"({cost_ceiling.basis or 'resolved at task start'}){cap_text}. Budget exhausted."
|
||||
)
|
||||
accumulated_usage["cost_stop_spend_basis"] = spend_basis
|
||||
budget_pause.request_pause(ctx, rail=budget_pause.RAIL_GRACEFUL_CEILING,
|
||||
scope=pause_scope, reason_text=finish_reason)
|
||||
if not explicit:
|
||||
# Only an explicit profile authored room for a final answer. The fence checks the
|
||||
# wider tree cap and wallet, never a producer allowance (or an inherited number
|
||||
# whose author is unreadable), so a recap here would be paid past it: an actor
|
||||
# without a continuation ends on the host's notice, services finalized first.
|
||||
accumulated_usage.update(execution_status="failed", reason_code="budget_exhausted")
|
||||
trace = ctx.llm_trace if isinstance(ctx.llm_trace, dict) else {}
|
||||
_loop()._finalize_forced_services(ctx, trace)
|
||||
return _loop()._forced_fallback_result(ctx, trace, finish_reason, "budget_exhausted",
|
||||
source="budget_host_fallback")
|
||||
return _loop()._forced_final_answer(
|
||||
ctx,
|
||||
prompt=f"[BUDGET LIMIT] {finish_reason} {_loop()._FORCED_BEST_EFFORT_TAIL}",
|
||||
|
|
@ -345,7 +364,7 @@ def _soft_land_exhausted_ceiling(
|
|||
)
|
||||
tree_info = _loop()._loop_tree_accounting(refresh=True, max_age_sec=0.0)
|
||||
deciding, spend_basis = task_pacing.resolve_deciding_spend(
|
||||
tree_cost_usd=tree_info.get("accounted_usd") if isinstance(tree_info, dict) else None,
|
||||
tree_cost_usd=tree_info.get("settled_usd") if isinstance(tree_info, dict) else None,
|
||||
task_cost_usd=float(limit_ctx.accumulated_usage.get("cost") or 0.0),
|
||||
root_cap_usd=cost_ceiling.root_cap_usd,
|
||||
)
|
||||
|
|
@ -866,7 +885,7 @@ def authored_completion_budget_exhausted(ctx: Any, budget_remaining: float | Non
|
|||
if cap is None:
|
||||
return False
|
||||
tree = _loop()._loop_tree_accounting(refresh=True, max_age_sec=0.0)
|
||||
deciding, basis = task_pacing.resolve_deciding_spend(tree_cost_usd=(tree or {}).get("accounted_usd"),
|
||||
deciding, basis = task_pacing.resolve_deciding_spend(tree_cost_usd=(tree or {}).get("settled_usd"),
|
||||
task_cost_usd=ctx.accumulated_usage.get("cost"), root_cap_usd=cap)
|
||||
if deciding is not None and deciding >= cap:
|
||||
ctx.accumulated_usage["cost_stop_spend_basis"] = basis
|
||||
|
|
|
|||
|
|
@ -206,12 +206,14 @@ def _maybe_inject_self_check(
|
|||
# ledger number (DEVELOPMENT cache_friendliness item 28). The fence
|
||||
# counts the whole tree; own cost alone hid two tree deaths.
|
||||
tree_line = ""
|
||||
tree_known: Optional[float] = None
|
||||
tree_accounted: Optional[float] = None
|
||||
tree_cap: Optional[float] = None
|
||||
tree_info = _loop()._loop_tree_accounting(refresh=True, max_age_sec=30.0)
|
||||
rendered = task_pacing.tree_spend_line(tree_info, cost_ceiling)
|
||||
if rendered:
|
||||
tree_accounted = float(tree_info["accounted_usd"])
|
||||
tree_known = float(tree_info["settled_usd"])
|
||||
tree_accounted = float(tree_info.get("accounted_usd") or tree_known)
|
||||
raw_cap = tree_info.get("root_limit_usd")
|
||||
tree_cap = float(raw_cap) if raw_cap is not None else None
|
||||
tree_line = f"{rendered}\n"
|
||||
|
|
@ -256,7 +258,8 @@ def _maybe_inject_self_check(
|
|||
"context_tokens": ctx_tokens,
|
||||
"task_cost": task_cost,
|
||||
}
|
||||
if tree_accounted is not None:
|
||||
if tree_known is not None:
|
||||
checkpoint_payload["tree_known_usd"] = round(tree_known, 4)
|
||||
checkpoint_payload["tree_accounted_usd"] = round(tree_accounted, 4)
|
||||
checkpoint_payload["tree_cap_usd"] = round(tree_cap, 4) if tree_cap is not None else None
|
||||
_loop()._emit_checkpoint_event(event_queue, task_id, drive_logs, checkpoint_payload)
|
||||
|
|
@ -304,9 +307,9 @@ def _maybe_inject_cost_budget_milestone(
|
|||
) -> bool:
|
||||
"""Thin transport over the task_pacing cost axis (v6.56.0): content,
|
||||
thresholds, and latch state live in ouroboros/task_pacing.py. The deciding
|
||||
spend under a root cap is the tree-accounted stash (free read; refreshed by
|
||||
every dispatch) with a bounded staleness cap — never a per-round ledger
|
||||
read, see ``_TREE_ACCOUNTING_MAX_STALE_SEC``."""
|
||||
spend under a root cap is the tree's KNOWN spend from the root stash (free
|
||||
read; refreshed by every dispatch) with a bounded staleness cap — never a
|
||||
per-round ledger read, see ``_TREE_ACCOUNTING_MAX_STALE_SEC``."""
|
||||
ceiling_usd = (
|
||||
cost_ceiling.ceiling_usd
|
||||
if cost_ceiling is not None and cost_ceiling.state == task_pacing.COST_CEILING_ACTIVE
|
||||
|
|
@ -315,7 +318,7 @@ def _maybe_inject_cost_budget_milestone(
|
|||
tree_info = _loop()._loop_tree_accounting(
|
||||
refresh=True, max_age_sec=_loop()._TREE_ACCOUNTING_MAX_STALE_SEC,
|
||||
)
|
||||
tree_cost = tree_info.get("accounted_usd") if isinstance(tree_info, dict) else None
|
||||
tree_cost = tree_info.get("settled_usd") if isinstance(tree_info, dict) else None
|
||||
note = task_pacing.build_cost_budget_note(
|
||||
tools._ctx,
|
||||
start_remaining_usd=budget_remaining_usd,
|
||||
|
|
|
|||
|
|
@ -275,10 +275,10 @@ def load_owner_wait(ctx: Any, handoff: dict | None = None) -> dict:
|
|||
or state.get("wait_id") != current.get("wait_id")
|
||||
or state.get("task_attempt") != int(ctx.task_attempt or 1)):
|
||||
raise ValueError("owner wait continuation identity mismatch")
|
||||
if state.get("cost_ceiling") is not None:
|
||||
from ouroboros.task_pacing import CostCeiling
|
||||
from ouroboros.task_pacing import restore_cost_ceiling
|
||||
|
||||
ctx._cost_ceiling = CostCeiling(**state["cost_ceiling"])
|
||||
# The start's own authority, never the saved number alone (#1128).
|
||||
ctx._cost_ceiling = restore_cost_ceiling(ctx, state.get("cost_ceiling"))
|
||||
return state
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -6,12 +6,15 @@ whatever the entry point (``python server.py``, ``ouroboros server``, the
|
|||
``ouroboros-web`` script, Colab), each pool worker at the top of ``worker_main``
|
||||
after its fork preamble (so a forkserver parent stays single-threaded), and the
|
||||
launcher keeps its own ``launcher.log`` setup because it ships frozen inside the
|
||||
app bundle, sharing only ``install_exception_hooks``. The server is the single
|
||||
writer of ``logs/server.log``; a worker logs to its stderr, which the launcher
|
||||
copies into ``logs/agent_stdout.log`` (desktop) and Docker keeps as the container
|
||||
log, so two processes never rotate one file. Every local handler carries the
|
||||
shared ``SecretRedactingLogFilter``; the module adds handlers beside any a
|
||||
process already has and never replaces them.
|
||||
app bundle, sharing ``install_exception_hooks`` and ``copy_capped_output``. The
|
||||
server is the single writer of ``logs/server.log``; a worker logs to its stderr,
|
||||
which the launcher copies into ``logs/agent_stdout.log`` (desktop) and Docker
|
||||
keeps as the container log, so two processes never rotate one file. Every local
|
||||
handler carries the shared ``SecretRedactingLogFilter``; the module adds handlers
|
||||
beside any a process already has and never replaces them. Importing the module
|
||||
loads only the standard library: the launcher's copier must keep draining and
|
||||
recording a server whose own imports are broken, which is the crash output
|
||||
``agent_stdout.log`` exists for.
|
||||
|
||||
The hooks route an unhandled exception of a thread, or of the main thread, into
|
||||
``logging`` instead of the interpreter's raw stderr print, so it is formatted like
|
||||
|
|
@ -25,18 +28,19 @@ catches it first), which is why ``worker_main`` records its own crashes.
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
import errno
|
||||
import logging
|
||||
import os
|
||||
import pathlib
|
||||
import sys
|
||||
import threading
|
||||
from logging.handlers import RotatingFileHandler
|
||||
from typing import Any
|
||||
|
||||
from ouroboros.observability import SecretRedactingLogFilter
|
||||
|
||||
LOG_FORMAT = "%(asctime)s [%(levelname)s] %(name)s: %(message)s"
|
||||
SERVER_LOG_MAX_BYTES = 2 * 1024 * 1024
|
||||
SERVER_LOG_BACKUP_COUNT = 3
|
||||
OUTPUT_CHUNK_BYTES = 64 * 1024
|
||||
UNCAUGHT_LOGGER_NAME = "ouroboros.uncaught"
|
||||
|
||||
_configured = False
|
||||
|
|
@ -53,6 +57,8 @@ def configure_process_logging(*, drive_logs: pathlib.Path | None) -> None:
|
|||
global _configured
|
||||
if _configured:
|
||||
return
|
||||
from ouroboros.observability import SecretRedactingLogFilter
|
||||
|
||||
_configured = True
|
||||
handlers: list[logging.Handler] = []
|
||||
if drive_logs is not None:
|
||||
|
|
@ -122,3 +128,161 @@ def _log_uncaught(message: str, exc_info: Any) -> bool:
|
|||
except Exception:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def copy_capped_output(stream: Any, path: pathlib.Path, *, report: logging.Logger,
|
||||
max_bytes: int = SERVER_LOG_MAX_BYTES,
|
||||
backups: int = SERVER_LOG_BACKUP_COUNT) -> None:
|
||||
"""Drain a child's output pipe into ``path``, continuing after storage errors.
|
||||
|
||||
A child whose pipe fills blocks in its next write (for the server: its
|
||||
stderr log handler, so every thread that logs), so a storage error must
|
||||
not end the read loop. Filesystem calls remain synchronous: a hung device
|
||||
can still delay draining. Each read is at most 64 KiB of raw bytes, one line when it fits:
|
||||
newline-free output holds bounded memory, and the cap counts bytes on disk
|
||||
with no decoding or newline translation. The file rotates like
|
||||
``server.log`` BEFORE a write would pass ``max_bytes``. A chunk that cannot
|
||||
be stored within the cap (open, rotation or write failure) is dropped and
|
||||
counted while reading continues, and storage is retried with the next
|
||||
chunk; a failed rotation never truncates the live file or appends past the
|
||||
cap. A file this copier did not write (an inherited oversized one) rotates
|
||||
whole into ``.1`` and ages out with the backups.
|
||||
|
||||
Gaps are disclosed, never invented: a chunk refused before any byte reached
|
||||
the file is ``not written``; the bytes of a failed write call are of
|
||||
``uncertain`` durability. When storage resumes, a bounded marker line,
|
||||
counted toward the cap, states both before the next output. ``report`` (the
|
||||
launcher's own log, never this file) receives one warning per failure kind
|
||||
per file generation and its recovery, a gap still open at end of input, and
|
||||
a failed read; a failing report never stops the copy. End of input, or a
|
||||
failed read of a closed or broken pipe, ends it: nothing retries the input.
|
||||
"""
|
||||
sink = _CappedFile(pathlib.Path(path), max_bytes, backups, report)
|
||||
limit = min(OUTPUT_CHUNK_BYTES, max_bytes)
|
||||
try:
|
||||
while True:
|
||||
try:
|
||||
chunk = stream.readline(limit)
|
||||
except Exception as exc:
|
||||
_tell(report.warning, "Output pipe read failed (%s); the copy into %s stopped",
|
||||
_failure_kind(exc), sink.path.name)
|
||||
return
|
||||
if not chunk:
|
||||
return
|
||||
sink.store(chunk)
|
||||
finally:
|
||||
sink.close()
|
||||
|
||||
|
||||
def _tell(emit: Any, message: str, *args: Any) -> None:
|
||||
try:
|
||||
emit(message, *args)
|
||||
except Exception:
|
||||
pass # A failing report never stops the pipe from draining.
|
||||
|
||||
|
||||
def _failure_kind(exc: BaseException) -> str:
|
||||
"""Class plus errno name: stable across chunks, so one kind is one report."""
|
||||
code = getattr(exc, "errno", None)
|
||||
name = errno.errorcode.get(code, code) if isinstance(code, int) else None
|
||||
return f"{type(exc).__name__}:{name}" if name else type(exc).__name__
|
||||
|
||||
|
||||
class _CappedFile:
|
||||
"""One live file and its numbered backups, written through an unbuffered handle.
|
||||
|
||||
Unbuffered writes leave nothing in a buffer that a later close could flush
|
||||
past the cap, and each returned count is what reached the file; the size is
|
||||
re-read from the file whenever it is (re)opened.
|
||||
"""
|
||||
|
||||
def __init__(self, path: pathlib.Path, max_bytes: int, backups: int, report: logging.Logger) -> None:
|
||||
self.path, self.max_bytes, self.backups, self.report = path, max_bytes, backups, report
|
||||
self.handle: Any = None
|
||||
self.size = 0
|
||||
self.failure = "" # The kind that opened the current gap; empty while storage works.
|
||||
self.not_written = self.uncertain = 0
|
||||
self.gap_reported = False
|
||||
self.warned: set[str] = set() # Kinds already reported in this file generation.
|
||||
|
||||
def store(self, chunk: bytes) -> None:
|
||||
if self.failure:
|
||||
marker = (f"[output gap: {self.not_written} bytes not written, {self.uncertain} bytes "
|
||||
f"of uncertain durability ({self.failure})]\n").encode("ascii")
|
||||
if not self._put(marker[:self.max_bytes], marker=True):
|
||||
self.not_written += len(chunk)
|
||||
return
|
||||
if self.gap_reported:
|
||||
_tell(self.report.info, "%s storage recovered: %d bytes not written, %d bytes of uncertain durability",
|
||||
self.path.name, self.not_written, self.uncertain)
|
||||
self.failure, self.not_written, self.uncertain = "", 0, 0
|
||||
self._put(chunk)
|
||||
|
||||
def close(self) -> None:
|
||||
if self.failure:
|
||||
_tell(self.report.warning, "%s output ended with storage failing (%s): %d bytes not written, "
|
||||
"%d bytes of uncertain durability", self.path.name, self.failure, self.not_written, self.uncertain)
|
||||
self._close()
|
||||
|
||||
def _put(self, data: bytes, *, marker: bool = False) -> bool:
|
||||
stage, pending = "open", memoryview(data)
|
||||
try:
|
||||
if self.handle is None:
|
||||
self._open()
|
||||
if self.size and self.size + len(data) > self.max_bytes:
|
||||
stage = "rotate"
|
||||
self._close()
|
||||
self._rotate()
|
||||
self._open()
|
||||
if self.size and self.size + len(data) > self.max_bytes:
|
||||
raise FileExistsError(errno.EEXIST, "the live file is still full after rotation")
|
||||
stage = "write"
|
||||
while pending:
|
||||
written = self.handle.write(pending)
|
||||
if not written:
|
||||
raise OSError(errno.EIO, "the write made no progress")
|
||||
self.size += written
|
||||
pending = pending[written:]
|
||||
except Exception as exc:
|
||||
self._close()
|
||||
if marker:
|
||||
pass # Host-authored bytes are no output; the gap keeps its counts.
|
||||
elif stage == "write":
|
||||
self.uncertain += len(pending)
|
||||
else:
|
||||
self.not_written += len(pending)
|
||||
if not self.failure:
|
||||
self.failure = f"{stage}:{_failure_kind(exc)}"
|
||||
self.gap_reported = self.failure not in self.warned
|
||||
if self.gap_reported:
|
||||
self.warned.add(self.failure)
|
||||
_tell(self.report.warning, "%s storage failed (%s); output keeps draining and is dropped "
|
||||
"until a write succeeds", self.path.name, self.failure)
|
||||
return False
|
||||
return True
|
||||
|
||||
def _open(self) -> None:
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||
self.handle = open(self.path, "ab", buffering=0)
|
||||
self.size = os.fstat(self.handle.fileno()).st_size
|
||||
|
||||
def _close(self) -> None:
|
||||
handle, self.handle = self.handle, None
|
||||
if handle is not None:
|
||||
try:
|
||||
handle.close()
|
||||
except Exception:
|
||||
pass # Unbuffered: nothing is left to flush, so a failed close loses no output.
|
||||
|
||||
def _rotate(self) -> None:
|
||||
names = [self.path, *(self.path.with_name(f"{self.path.name}.{index}")
|
||||
for index in range(1, self.backups + 1))]
|
||||
# Shift only the occupied run before the first free slot (the oldest is
|
||||
# overwritten when none is free): a retry after a failed live rename finds
|
||||
# ``.1`` free and moves nothing else, so repeated failures discard no backup.
|
||||
free = next((index for index in range(1, len(names)) if not names[index].exists()), len(names) - 1)
|
||||
for index in range(free, 1, -1):
|
||||
os.replace(names[index - 1], names[index])
|
||||
if self.path.exists():
|
||||
os.replace(self.path, names[1])
|
||||
self.warned.clear()
|
||||
|
|
|
|||
|
|
@ -21,7 +21,6 @@ from ouroboros.secret_masking import (
|
|||
CONFIGURED_SECRET_PLACEHOLDER,
|
||||
MASKED_SECRET_SETTING_KEYS as SECRET_SETTING_KEYS,
|
||||
)
|
||||
from ouroboros.task_pacing import COST_PLANNING_MARGIN_USD
|
||||
from ouroboros.model_slots import (
|
||||
MODEL_ACCOUNTS_KEY, MODEL_CONTEXT_WINDOWS_KEY, MODEL_PROCESSING_PREFERENCES_KEY,
|
||||
PROCESSING_PREFERENCE_KEY, normalize_model_role_options, normalize_processing_preference,
|
||||
|
|
@ -235,17 +234,15 @@ _BUDGET_FIELDS = [
|
|||
"settingsInputId": "s-settings-per-task-cost",
|
||||
"title": "Per-task cost cap",
|
||||
"label": "Per-task Cost Cap (USD)",
|
||||
# The wrap-up sentence is only true above the planning margin: a cap at
|
||||
# or below it resolves to `exhausted_soft_land`, which force-finalizes at
|
||||
# the TOP of round 0 — no work rounds at all. The field still accepts
|
||||
# such a cap (owner power stays), so the note states the consequence
|
||||
# instead of the setting silently meaning something else.
|
||||
# Owner 2026-10-03 Q4-A / 2026-10-07: the cap is the limit itself, decided
|
||||
# on known spend; no default share of the wallet or margin before it.
|
||||
"note": (
|
||||
"Hard cap over one task's WHOLE tree, subagents included: further model calls are "
|
||||
"refused and the task is force-stopped once the tree's accounted spend reaches this "
|
||||
"(a graceful wrap-up fires just before). The wrap-up itself needs about "
|
||||
f"${COST_PLANNING_MARGIN_USD:.2f} of room, so a cap at or below that finalizes the "
|
||||
"task immediately instead of running any work rounds."
|
||||
"Hard cap over one task's WHOLE tree, subagents included: new model calls are "
|
||||
"refused once the tree's known spend (confirmed and estimated) reaches it. A resumable "
|
||||
"task then pauses with its work saved; a run that cannot be resumed ends as "
|
||||
"budget-exhausted instead. Calls already in flight settle normally and can take the "
|
||||
"total past the cap. Reservations and unresolved charges are shown separately, not "
|
||||
"counted as spending. Raising the cap does not resume a paused task."
|
||||
),
|
||||
"default": float(SETTINGS_DEFAULTS.get("OUROBOROS_PER_TASK_COST_USD", 50.0)),
|
||||
"min": "0.01",
|
||||
|
|
|
|||
|
|
@ -712,10 +712,10 @@ def review_wave_budget_block(
|
|||
return None
|
||||
fence, remedy = review_wave_binding_fence(admission)
|
||||
return (
|
||||
"review wave declined before dispatch: estimated reviewer-wave cost "
|
||||
f"~${admission.get('estimated_wave_usd')} exceeds the remaining budget "
|
||||
f"${admission.get('remaining_usd')} ({fence}). No reviewer was called; the skill "
|
||||
f"stays pending. Wait for in-flight attempts to settle, {remedy}, or re-run the review in a fresh task."
|
||||
f"review wave declined before dispatch: known spend has reached the {fence} "
|
||||
f"(remaining ${admission.get('remaining_usd')}; the wave was estimated at "
|
||||
f"~${admission.get('estimated_wave_usd')}). No reviewer was called; the skill "
|
||||
f"stays pending. {remedy[:1].upper() + remedy[1:]}, or re-run the review in a fresh task."
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -312,8 +312,11 @@ _TIME_BUDGET_THRESHOLDS = ((0.50, "50%"), (0.25, "25%"), (0.10, "10%"))
|
|||
# contract's budget_profile.cost_hard_stop_pct, not a global).
|
||||
_COST_BUDGET_THRESHOLDS = ((0.50, "50%"), (0.25, "25%"), (0.10, "10%"))
|
||||
_COST_WRAPUP_SPENT_FRACTION = 0.80
|
||||
# The historical in-task hard stop: half the budget remaining at task start.
|
||||
_DEFAULT_COST_HARD_STOP_PCT = 50
|
||||
# Disclosed bases: no default in-task stop (owner 2026-10-07, #1128: no share of the
|
||||
# wallet, no margin before a cap); and an inherited pre-upgrade number whose author is
|
||||
# unreadable now, kept provisionally and never widened (a compatibility gap, named).
|
||||
COST_BASIS_NO_DEFAULT_STOP = "no_default_cost_stop"
|
||||
COST_BASIS_POLICY_UNVERIFIED = "legacy_policy_unverified"
|
||||
|
||||
# Typed cost-ceiling states (v6.91): ``None`` is deliberately NOT overloaded to
|
||||
# mean both "unlimited" and "exhausted" — a $0.50 bench root cap under a $3
|
||||
|
|
@ -323,23 +326,24 @@ COST_CEILING_ACTIVE = "active"
|
|||
COST_CEILING_EXHAUSTED_SOFT_LAND = "exhausted_soft_land"
|
||||
COST_CEILING_UNKNOWN = "unknown"
|
||||
|
||||
# Planning margin subtracted from the root cap before the graceful in-task stop:
|
||||
# an ABSOLUTE emit-window's worth of money (~2 forced-wrap-up call reservation
|
||||
# bounds), NEVER a percentage of the cap (a pct reserve amputated ~54 min from a
|
||||
# 6h task — v6.54.4 adversarial r1, see effective_finalization_reserve_sec) and
|
||||
# NOT ledger-held — the ledger fence still binds at the full cap; the margin
|
||||
# only pulls the graceful stop earlier so the wrap-up call fits under the fence.
|
||||
# Planning margin an EXPLICIT ``cost_hard_stop_pct`` profile subtracts from the
|
||||
# root cap before its graceful in-task stop (authored experiment math, verbatim;
|
||||
# ordinary tasks have none, owner 2026-10-07): an ABSOLUTE emit-window's worth of
|
||||
# money (~2 forced-wrap-up call reservation bounds), NEVER a percentage of the cap
|
||||
# (a pct reserve amputated ~54 min from a 6h task — v6.54.4 adversarial r1, see
|
||||
# effective_finalization_reserve_sec) and NOT ledger-held: the fence binds at the full cap.
|
||||
_WRAPUP_CALL_RESERVATION_BOUND_USD = 1.50
|
||||
COST_PLANNING_MARGIN_USD = max(1.0, 2.0 * _WRAPUP_CALL_RESERVATION_BOUND_USD)
|
||||
|
||||
|
||||
# Deciding-spend basis vocabulary (v6.91). The tree-accounted number is the
|
||||
# authority for every rooted task (with a root cap the ledger fence counts the
|
||||
# TREE; without one the in-task ceiling still decides on the subtree); when it
|
||||
# Deciding-spend basis vocabulary (v6.91). The tree's KNOWN (settled) spend is
|
||||
# the authority for every rooted task (#1487: with a root cap the ledger fence
|
||||
# counts the TREE's known spend; without one the in-task ceiling still decides
|
||||
# on the subtree); open holds are disclosed beside it, never counted. When it
|
||||
# is momentarily unavailable the own-cost number still decides — but the
|
||||
# substitution is DISCLOSED as a lower bound, never silent (BIBLE P1). Only a
|
||||
# task with no root at all has no tree to read, so its own cost is complete.
|
||||
SPEND_BASIS_TREE = "tree_accounted"
|
||||
SPEND_BASIS_TREE = "tree_known"
|
||||
SPEND_BASIS_OWN_TREE_UNKNOWN = "own_fallback_tree_unknown"
|
||||
SPEND_BASIS_OWN_NO_TREE_CAP = "own_only_no_tree_cap"
|
||||
|
||||
|
|
@ -371,17 +375,19 @@ class CostCeiling:
|
|||
"""Typed in-task cost-stop state, resolved ONCE at loop start.
|
||||
|
||||
``state``:
|
||||
- ``disabled``: no in-task stop — explicit ``cost_hard_stop_pct=0`` (bench
|
||||
contract, e.g. SWE-Pro) or no finite budget on either axis (e.g. GAIA);
|
||||
the whole cost axis stays silent.
|
||||
- ``active``: ``ceiling_usd`` is the root's strictly-positive original
|
||||
graceful-stop point, or a disclosed legacy local resolution when the
|
||||
original carrier is unavailable.
|
||||
- ``exhausted_soft_land``: the root cap leaves no room above the planning
|
||||
margin — the loop must enter its graceful best-effort wrap-up
|
||||
immediately; it must NEVER run uncapped.
|
||||
- ``unknown``: resolution inputs errored; the axis stays silent but the
|
||||
gap is represented, never filled in (BIBLE P1)."""
|
||||
- ``disabled``: no in-task stop — an ordinary task (no explicit
|
||||
``cost_hard_stop_pct``, no producer allowance: owner 2026-10-07), an
|
||||
explicit ``cost_hard_stop_pct=0`` (bench contract, e.g. SWE-Pro), or no
|
||||
finite budget on either axis (e.g. GAIA); the ledger fences still bind.
|
||||
- ``active``: ``ceiling_usd`` is a real early stop: an explicit profile's
|
||||
authored point or a producer's allowance (a wake's daily remainder), also
|
||||
as a member's inherited number, or one whose root authority is unreadable
|
||||
now (``legacy_policy_unverified``).
|
||||
- ``exhausted_soft_land``: an explicit profile whose root cap leaves no room
|
||||
above its planning margin — the loop must enter its graceful best-effort
|
||||
wrap-up immediately; it must NEVER run uncapped.
|
||||
- ``unknown``: resolution inputs errored; the axis stays silent and only
|
||||
the hard money limits bind, the gap represented, never filled in (P1)."""
|
||||
|
||||
state: str
|
||||
ceiling_usd: Optional[float] = None
|
||||
|
|
@ -390,6 +396,14 @@ class CostCeiling:
|
|||
basis: str = ""
|
||||
|
||||
|
||||
def _positive(value: Any) -> Optional[float]:
|
||||
try:
|
||||
number = None if value is None or isinstance(value, bool) else float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
return number if number is not None and math.isfinite(number) and number > 0 else None
|
||||
|
||||
|
||||
def resolve_cost_ceiling(
|
||||
budget_remaining_start_usd: Optional[float],
|
||||
profile: Dict[str, Any],
|
||||
|
|
@ -400,39 +414,44 @@ def resolve_cost_ceiling(
|
|||
) -> CostCeiling:
|
||||
"""The in-task cost stop, computed ONCE at loop start (typed; v6.91).
|
||||
|
||||
The GLOBAL component keeps the historical semantics: ``cost_hard_stop_pct``
|
||||
None -> 50% of the global remaining at task start; 0 -> the whole in-task
|
||||
stop is disabled (bench contract). The ROOT component is the per-task tree
|
||||
cap (``OUROBOROS_PER_TASK_COST_USD`` -> ``UsageScope.root_limit_usd``, the
|
||||
SAME value the ledger fence enforces) minus the ABSOLUTE planning margin —
|
||||
deliberately NOT pct-scaled (pct applies to the global axis only; scaling
|
||||
the owner's chosen cap would silently halve it). The ceiling is
|
||||
min(available components); NEVER a computed $0 — a root cap at or below the
|
||||
margin resolves to ``exhausted_soft_land`` instead.
|
||||
Without an explicit ``cost_hard_stop_pct`` the host has no stop of its own
|
||||
(owner 2026-10-07, #1128): only a producer allowance stops early, at its
|
||||
actual value — a ROOT's ``root_ceiling_usd`` (a wake's daily remainder,
|
||||
``producer_allowance``) or a member's inherited number (``root_resolved_ceiling``;
|
||||
the caller decides which carry that authority); otherwise ``disabled``. The
|
||||
ledger fences still bind every send at the full limits, on known spend.
|
||||
|
||||
An enabled non-root member keeps the propagated original root ceiling;
|
||||
a later global balance never re-mints that early threshold. A ROOT may
|
||||
carry a producer's own ``root_cost_ceiling_usd`` below its hard cap (a
|
||||
consciousness wake-up: what is left of its allowance); it lands the same
|
||||
planning margin early, and at or below the margin it is an immediate soft
|
||||
landing, exactly like a cap. Actual global
|
||||
and root dispatch fences still bind independently. Legacy missing carriers
|
||||
retain a disclosed local resolution, never a guessed original root fact.
|
||||
An EXPLICIT percentage keeps the authored math verbatim. The GLOBAL
|
||||
component is that share of the global remaining at task start; 0 disables
|
||||
the whole in-task stop (bench contract). The ROOT component is the per-task
|
||||
tree cap (``OUROBOROS_PER_TASK_COST_USD`` -> ``UsageScope.root_limit_usd``,
|
||||
the SAME value the ledger fence enforces) minus the ABSOLUTE planning margin
|
||||
— deliberately NOT pct-scaled. The ceiling is min(available components);
|
||||
NEVER a computed $0 — a root cap at or below the margin resolves to
|
||||
``exhausted_soft_land`` instead. An enabled non-root member keeps the
|
||||
propagated original root ceiling; a later global balance never re-mints
|
||||
that early threshold. A ROOT's producer ceiling lands the same planning
|
||||
margin early under an explicit profile. Legacy missing carriers retain a
|
||||
disclosed local resolution, never a guessed original root fact.
|
||||
|
||||
Stated plainly rather than implied: the ``room <= 0`` bail is the owner's
|
||||
"$0 ceiling" rule EXACTLY, no wider. A cap just ABOVE the margin therefore
|
||||
yields a real but tiny ceiling — ``root_cap_usd = COST_PLANNING_MARGIN_USD
|
||||
+ 0.01`` gives ``ceiling_usd == 0.01``, which the first round's spend
|
||||
crosses — so a positive ``ceiling_usd`` is not by itself a promise of
|
||||
working room. Both numbers are disclosed on the carrier (``ceiling_usd``,
|
||||
``root_cap_usd``, ``planning_margin_usd``) and printed in the stop text, so
|
||||
a reader sees the tiny ceiling instead of inferring a healthy one. Widening
|
||||
the bail into a minimum-room FLOOR would move caps the owner deliberately
|
||||
allows into immediate soft-land; that is an owner call, not a code one."""
|
||||
Stated plainly rather than implied: under an explicit profile the
|
||||
``room <= 0`` bail is the owner's "$0 ceiling" rule EXACTLY, no wider. A cap
|
||||
just ABOVE the margin therefore yields a real but tiny ceiling —
|
||||
``root_cap_usd = COST_PLANNING_MARGIN_USD + 0.01`` gives
|
||||
``ceiling_usd == 0.01``, which the first round's spend crosses — so a
|
||||
positive ``ceiling_usd`` is not by itself a promise of working room. Both
|
||||
numbers are disclosed on the carrier (``ceiling_usd``, ``root_cap_usd``,
|
||||
``planning_margin_usd``) and printed in the stop text."""
|
||||
try:
|
||||
pct = profile.get("cost_hard_stop_pct")
|
||||
if pct is None:
|
||||
pct = _DEFAULT_COST_HARD_STOP_PCT
|
||||
cap = _positive(root_cap_usd)
|
||||
allowance = _positive(root_ceiling_usd)
|
||||
if allowance is None:
|
||||
return CostCeiling(state=COST_CEILING_DISABLED, root_cap_usd=cap,
|
||||
basis=COST_BASIS_NO_DEFAULT_STOP)
|
||||
return CostCeiling(state=COST_CEILING_ACTIVE, ceiling_usd=allowance, root_cap_usd=cap,
|
||||
basis="root_resolved_ceiling" if non_root_member else "producer_allowance")
|
||||
pct = max(0, min(100, int(pct)))
|
||||
if pct == 0:
|
||||
return CostCeiling(
|
||||
|
|
@ -504,22 +523,61 @@ def resolve_cost_ceiling(
|
|||
return CostCeiling(state=COST_CEILING_UNKNOWN, basis="resolve_error")
|
||||
|
||||
|
||||
def resolve_task_cost_ceiling(ctx: Any, budget_remaining_usd: Optional[float]) -> CostCeiling:
|
||||
"""The typed in-task cost stop of ONE task, resolved once per task.
|
||||
# Who authored a task's early cost stop: an explicit percentage profile, a
|
||||
# producer allowance, nobody (the removed default), or unreadable.
|
||||
COST_STOP_EXPLICIT, COST_STOP_PRODUCER, COST_STOP_NONE, COST_STOP_UNKNOWN = (
|
||||
"explicit", "producer", "none", "unknown")
|
||||
|
||||
The root cap comes from the bound usage scope -- the SAME
|
||||
``OUROBOROS_PER_TASK_COST_USD``-derived value the ledger fence enforces
|
||||
(``agent.py`` wires it as ``UsageScope.root_limit_usd``), so the graceful
|
||||
stop and the fence can never disagree about the cap. The same scope says
|
||||
whether this task is the root of its tree or one of its members."""
|
||||
|
||||
def _root_cost_stop_authority(scope: Any) -> Tuple[str, Optional[float]]:
|
||||
"""Who authored the number a member inherited, from the root's canonical row.
|
||||
|
||||
ONE strict read, only for the ambiguous member (an inherited number, no
|
||||
explicit percentage of its own): the number cannot say whether the root took
|
||||
it from an explicit profile (returned: that percentage), a producer allowance
|
||||
(returned: its actual value) or the removed default (a child scheduled before
|
||||
the upgrade; a readable root with neither). A root row that cannot be read,
|
||||
or lacks its contract, is ``unknown``: the caller keeps the number, never
|
||||
guessing either way."""
|
||||
from ouroboros.contracts.task_contract import normalize_budget_profile
|
||||
from ouroboros.task_results import load_task_result
|
||||
|
||||
try:
|
||||
row = load_task_result(pathlib.Path(scope.drive_root), scope.root_task_id, strict=True)
|
||||
except Exception:
|
||||
log.warning("Root %s cost-stop authority unreadable", getattr(scope, "root_task_id", ""), exc_info=True)
|
||||
return COST_STOP_UNKNOWN, None
|
||||
contract = row.get("task_contract") if isinstance(row, dict) else None
|
||||
if not isinstance(contract, dict):
|
||||
return COST_STOP_UNKNOWN, None
|
||||
pct = normalize_budget_profile(contract.get("budget_profile")).get("cost_hard_stop_pct")
|
||||
if pct is not None:
|
||||
return COST_STOP_EXPLICIT, float(pct)
|
||||
metadata = row.get("metadata") if isinstance(row.get("metadata"), dict) else {}
|
||||
producer = _positive(row.get("root_cost_ceiling_usd")) or _positive(metadata.get("root_cost_ceiling_usd"))
|
||||
return (COST_STOP_PRODUCER, producer) if producer is not None else (COST_STOP_NONE, None)
|
||||
|
||||
|
||||
def _cost_stop_inputs(ctx: Any) -> Dict[str, Any]:
|
||||
"""The policy authority and resolver inputs of ONE task: the one reading a start
|
||||
(``resolve_task_cost_ceiling``), a cold restore (``restore_cost_ceiling``) and
|
||||
the Resume refresh share. Authority is never inferred from a number, a saved
|
||||
basis label, or a child's local ``None``."""
|
||||
root_cap = None
|
||||
root_ceiling = None
|
||||
non_root_member = False
|
||||
scope = None
|
||||
try:
|
||||
from ouroboros.usage_accounting import current_usage_scope
|
||||
|
||||
scope = current_usage_scope()
|
||||
if scope is not None:
|
||||
# Who authored the stop is copied first: the group projection below can fail (a
|
||||
# store timeout), and that must leave the scope's carried cap, never erase the stop.
|
||||
root_ceiling = getattr(scope, "root_cost_ceiling_usd", None)
|
||||
non_root_member = bool(
|
||||
scope.root_task_id and scope.task_id and scope.root_task_id != scope.task_id
|
||||
)
|
||||
root_cap = getattr(scope, "root_limit_usd", None)
|
||||
if root_cap is None: # a Continue's successor: the original root's carried group cap
|
||||
root_cap = getattr(scope, "billing_group_limit_usd", None)
|
||||
|
|
@ -529,21 +587,107 @@ def resolve_task_cost_ceiling(ctx: Any, budget_remaining_usd: Optional[float]) -
|
|||
root_limit=scope.root_limit_usd)
|
||||
if snapshot is not None:
|
||||
root_cap = snapshot["root_limit_usd"]
|
||||
root_ceiling = getattr(scope, "root_cost_ceiling_usd", None)
|
||||
non_root_member = bool(
|
||||
scope.root_task_id and scope.task_id and scope.root_task_id != scope.task_id
|
||||
)
|
||||
except Exception:
|
||||
log.debug("Usage scope unavailable for cost ceiling resolution", exc_info=True)
|
||||
log.debug("Usage scope or group cap projection unavailable for cost ceiling resolution", exc_info=True)
|
||||
profile = resolve_budget_profile(ctx)
|
||||
if profile.get("cost_hard_stop_pct") is not None:
|
||||
authority = COST_STOP_EXPLICIT
|
||||
elif _positive(root_ceiling) is None:
|
||||
authority, root_ceiling = COST_STOP_NONE, None
|
||||
elif not non_root_member:
|
||||
authority = COST_STOP_PRODUCER
|
||||
else:
|
||||
authority, value = _root_cost_stop_authority(scope)
|
||||
if authority == COST_STOP_EXPLICIT:
|
||||
# The root's authored percentage governs the member it handed its number to.
|
||||
profile = {**profile, "cost_hard_stop_pct": int(value)}
|
||||
elif authority != COST_STOP_UNKNOWN:
|
||||
root_ceiling = value
|
||||
return {"authority": authority, "profile": profile, "root_cap_usd": root_cap,
|
||||
"non_root_member": non_root_member, "root_ceiling_usd": root_ceiling}
|
||||
|
||||
|
||||
def _resolve_from_inputs(inputs: Dict[str, Any], budget_remaining_usd: Optional[float]) -> CostCeiling:
|
||||
if inputs["authority"] == COST_STOP_UNKNOWN:
|
||||
# Neither removed as a default nor confirmed as authored: the number the
|
||||
# member already carries keeps binding, labelled, until its root is readable.
|
||||
return CostCeiling(state=COST_CEILING_ACTIVE, ceiling_usd=_positive(inputs["root_ceiling_usd"]),
|
||||
root_cap_usd=_positive(inputs["root_cap_usd"]), basis=COST_BASIS_POLICY_UNVERIFIED)
|
||||
return resolve_cost_ceiling(
|
||||
budget_remaining_usd,
|
||||
resolve_budget_profile(ctx),
|
||||
root_cap_usd=root_cap,
|
||||
non_root_member=non_root_member,
|
||||
root_ceiling_usd=root_ceiling,
|
||||
inputs["profile"],
|
||||
root_cap_usd=inputs["root_cap_usd"],
|
||||
non_root_member=inputs["non_root_member"],
|
||||
root_ceiling_usd=inputs["root_ceiling_usd"],
|
||||
)
|
||||
|
||||
|
||||
def resolve_task_cost_ceiling(ctx: Any, budget_remaining_usd: Optional[float]) -> CostCeiling:
|
||||
"""The typed in-task cost stop of ONE task, resolved once per task.
|
||||
|
||||
The root cap comes from the bound usage scope -- the SAME
|
||||
``OUROBOROS_PER_TASK_COST_USD``-derived value the ledger fence enforces
|
||||
(``agent.py`` wires it as ``UsageScope.root_limit_usd``), so the graceful
|
||||
stop and the fence can never disagree about the cap. The same scope says
|
||||
whether this task is the root of its tree or one of its members, and which
|
||||
authority (``_cost_stop_inputs``) stands behind any early stop."""
|
||||
inputs = _cost_stop_inputs(ctx)
|
||||
_remember_cost_stop_policy(ctx, inputs)
|
||||
return _resolve_from_inputs(inputs, budget_remaining_usd)
|
||||
|
||||
|
||||
def _remember_cost_stop_policy(ctx: Any, inputs: Dict[str, Any]) -> None:
|
||||
try:
|
||||
setattr(ctx, "_cost_stop_policy", dict(inputs))
|
||||
except Exception:
|
||||
log.debug("Cost-stop policy could not be stashed on the tool context", exc_info=True)
|
||||
|
||||
|
||||
def cost_stop_policy(ctx: Any) -> Dict[str, Any]:
|
||||
"""The authority and resolver inputs behind this task's ceiling: the read its
|
||||
latest start or restore made (read now only when neither did in this process)."""
|
||||
policy = getattr(ctx, "_cost_stop_policy", None)
|
||||
if not isinstance(policy, dict):
|
||||
policy = _cost_stop_inputs(ctx)
|
||||
_remember_cost_stop_policy(ctx, policy)
|
||||
return policy
|
||||
|
||||
|
||||
def cost_stop_authority(ctx: Any) -> str:
|
||||
"""The authority behind this task's ceiling (``cost_stop_policy``)."""
|
||||
return str(cost_stop_policy(ctx)["authority"])
|
||||
|
||||
|
||||
def restore_cost_ceiling(ctx: Any, saved: Any) -> Optional[CostCeiling]:
|
||||
"""The ceiling a cold continuation (budget pause, owner-wait restart) resumes under.
|
||||
|
||||
Every restore re-reads the start's authority, never the saved number alone:
|
||||
an explicit percentage keeps its saved threshold (Q10 Resume moves it later);
|
||||
a producer allowance is re-read at its actual value (no old margin kept); a
|
||||
stop saved under the removed default is dropped, disclosed in the basis; an
|
||||
unreadable authority keeps the bound it had (``legacy_policy_unverified``),
|
||||
never re-armed or widened, until a later restore can read it. ``None`` only
|
||||
for an explicit policy with nothing saved: the loop resolves it as at any
|
||||
start. Callers assign the result."""
|
||||
previous = CostCeiling(**saved) if isinstance(saved, dict) else None
|
||||
inputs = _cost_stop_inputs(ctx)
|
||||
_remember_cost_stop_policy(ctx, inputs)
|
||||
if inputs["authority"] == COST_STOP_EXPLICIT:
|
||||
return previous
|
||||
if inputs["authority"] == COST_STOP_UNKNOWN and previous is not None:
|
||||
if previous.state not in {COST_CEILING_ACTIVE, COST_CEILING_EXHAUSTED_SOFT_LAND}:
|
||||
return previous
|
||||
basis = previous.basis if previous.basis.startswith(COST_BASIS_POLICY_UNVERIFIED) else (
|
||||
f"{COST_BASIS_POLICY_UNVERIFIED}({previous.basis})")
|
||||
return replace(previous, basis=basis)
|
||||
restored = _resolve_from_inputs(inputs, None)
|
||||
if (restored.state == COST_CEILING_DISABLED and previous is not None
|
||||
and previous.state in {COST_CEILING_ACTIVE, COST_CEILING_EXHAUSTED_SOFT_LAND}):
|
||||
removed = "soft_land" if previous.ceiling_usd is None else f"${float(previous.ceiling_usd):.2f}"
|
||||
restored = replace(restored, basis=f"{restored.basis}(saved default stop {removed} removed)")
|
||||
return restored
|
||||
|
||||
|
||||
def cost_ceiling_disclosure(ceiling: CostCeiling) -> Dict[str, Any]:
|
||||
"""The start-of-task shape of the ceiling the loop will actually decide on."""
|
||||
return {
|
||||
|
|
@ -554,14 +698,18 @@ def cost_ceiling_disclosure(ceiling: CostCeiling) -> Dict[str, Any]:
|
|||
"basis": ceiling.basis,
|
||||
"allocation": "unreserved_shared_pool",
|
||||
"rule": (
|
||||
"The graceful in-task cost stop of THIS task's whole tree, resolved once at task "
|
||||
"start: the root resolves min(configured share of global remaining, hard tree cap "
|
||||
"minus a planning margin); enabled descendants retain that original number. "
|
||||
"Legacy members without it disclose their local resolution. Crossing it asks for a "
|
||||
"best-effort final answer; the ledger fence at the full cap still binds "
|
||||
"independently. This threshold reserves no money: concurrent tasks share the "
|
||||
"global pool and may consume it before this task reaches its ceiling. "
|
||||
"Budget checkpoints report observed spend and unreserved shared headroom."
|
||||
"The early in-task cost stop of THIS task's whole tree, if any. Ordinary tasks have none: new paid calls "
|
||||
"are refused only when known spend (confirmed and estimated, subagents included) reaches the hard tree "
|
||||
"cap, the whole-work group cap or the shared Total budget. An active stop is an explicit "
|
||||
"cost_hard_stop_pct profile's point (resolved once at task start, retained by descendants; spend over it "
|
||||
"stops) or a producer's allowance such as a wake's daily remainder (known spend reaching it stops). At "
|
||||
"any of these a task with an exact continuation pauses with its work saved; an actor without one ends "
|
||||
"budget_exhausted instead (only an explicit profile's stop may still buy it one final answer, fenced by "
|
||||
"the hard limits; at any other stop the host's notice ends it with no paid recap). "
|
||||
"Basis legacy_policy_unverified: a number inherited before this version whose author is unreadable now, "
|
||||
"kept provisionally, never widened, re-read at the next start or restore. Nothing here reserves money: "
|
||||
"concurrent tasks share the global pool, and calls in flight or charged late can take spend past any "
|
||||
"limit. Checkpoints report known spend, open holds beside it, and unreserved shared headroom."
|
||||
),
|
||||
}
|
||||
|
||||
|
|
@ -584,11 +732,12 @@ def in_task_cost_ceiling_disclosure(ctx: Any, budget_remaining_usd: Optional[flo
|
|||
def tree_spend_line(tree_info: Any, ceiling: Optional[CostCeiling] = None) -> str:
|
||||
"""The one live tree-spend line the checkpoint and the pacing note share.
|
||||
|
||||
Known (settled) spend decides; open holds ride beside it, never added in.
|
||||
Names the BINDING bound: the in-task ceiling when one is active (that is
|
||||
what stops the task first), with the hard tree cap the ledger fence
|
||||
enforces beside it. Empty string when tree accounting is unavailable --
|
||||
unknown is never rendered as $0."""
|
||||
if not isinstance(tree_info, dict) or tree_info.get("accounted_usd") is None:
|
||||
if not isinstance(tree_info, dict) or tree_info.get("settled_usd") is None:
|
||||
return ""
|
||||
raw_cap = tree_info.get("root_limit_usd")
|
||||
cap = float(raw_cap) if raw_cap is not None else None
|
||||
|
|
@ -603,9 +752,11 @@ def tree_spend_line(tree_info: Any, ceiling: Optional[CostCeiling] = None) -> st
|
|||
bound += f" (${cap:.2f} hard tree cap)"
|
||||
else:
|
||||
bound = f" of ${cap:.2f} hard tree cap" if cap is not None else ""
|
||||
known = float(tree_info["settled_usd"])
|
||||
holds = max(0.0, float(tree_info.get("accounted_usd") or known) - known)
|
||||
return (
|
||||
f"Task tree spend: ~${float(tree_info['accounted_usd']):.2f}{bound} "
|
||||
"(ledger-accounted incl. in-flight holds, subagents included; ceiling is unreserved)"
|
||||
f"Task tree spend: ~${known:.2f}{bound} "
|
||||
f"(known, subagents included; +${holds:.2f} open holds not counted; ceiling is unreserved)"
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -931,10 +1082,10 @@ def build_cost_budget_note(
|
|||
budget) keeps the axis silent. ADVISORY only — the hard stop itself lives
|
||||
in the loop's budget gate, not here (P5).
|
||||
|
||||
``tree_cost_usd`` (v6.91) is the root subtree's ledger-accounted spend
|
||||
(settled + reserved + unresolved holds, subagents included) — when known it
|
||||
is the DECIDING spend, because the ledger fence counts the tree, not this
|
||||
task's own calls (waves died at tree $84-94 while own showed $41-49).
|
||||
``tree_cost_usd`` (v6.91) is the root subtree's KNOWN (settled) spend,
|
||||
subagents included — when known it is the DECIDING spend, because the
|
||||
ledger fence counts the tree, not this task's own calls (waves died at tree
|
||||
$84-94 while own showed $41-49); open holds are not spending (#1487).
|
||||
``task_cost`` (own accumulated cost) stays the diagnostic line. Unknown tree
|
||||
spend falls back to own cost — never coerced to $0, and never SILENTLY
|
||||
substituted: under a ``root_cap_usd`` the fallback is a lower bound and the
|
||||
|
|
@ -960,11 +1111,11 @@ def build_cost_budget_note(
|
|||
own_text = f"; own calls ~${task_cost:.2f}" if task_cost is not None else ""
|
||||
spent_line = (
|
||||
f"Spent this task tree: ~${deciding:.2f} "
|
||||
f"(ledger-accounted incl. in-flight holds, subagents included{own_text})"
|
||||
f"(known spend, subagents included; open holds not counted{own_text})"
|
||||
)
|
||||
elif spend_basis == SPEND_BASIS_OWN_TREE_UNKNOWN:
|
||||
spent_line = (
|
||||
f"Spent this task: ~${deciding:.2f} (OWN calls only — the tree-accounted "
|
||||
f"Spent this task: ~${deciding:.2f} (OWN calls only — the tree's known "
|
||||
"total is unavailable right now, so subagent spend is NOT included; treat "
|
||||
"this as a lower bound against the tree cap)"
|
||||
)
|
||||
|
|
@ -1010,11 +1161,11 @@ def build_cost_budget_note(
|
|||
)
|
||||
_tree_tail = _TREE_FLUSH_SENTENCE if _workspace_delivery(ctx) else ""
|
||||
if tree_basis:
|
||||
_tree_amount = f"tree-accounted ~${deciding:.2f} of ~${base:.2f}"
|
||||
_tree_amount = f"tree known spend ~${deciding:.2f} of ~${base:.2f}"
|
||||
elif spend_basis == SPEND_BASIS_OWN_TREE_UNKNOWN:
|
||||
_tree_amount = (
|
||||
f"~${deciding:.2f} of ~${base:.2f}, counting OWN calls only — the "
|
||||
"tree-accounted total is unavailable right now, so this is a lower bound"
|
||||
"tree's known total is unavailable right now, so this is a lower bound"
|
||||
)
|
||||
else:
|
||||
_tree_amount = f"~${deciding:.2f} of ~${base:.2f}"
|
||||
|
|
@ -1134,7 +1285,7 @@ def _acceptance_rails_line_inner(
|
|||
)
|
||||
root = usage_projection(scope.drive_root, root_task_id=scope.root_task_id)
|
||||
remaining = projection.get("remaining_known_usd")
|
||||
money_bits.append(_headroom_phrase(remaining, rails.get("cost_ceiling_usd"), root.get("accounted_usd")))
|
||||
money_bits.append(_headroom_phrase(remaining, rails.get("cost_ceiling_usd"), root.get("settled_usd")))
|
||||
except Exception:
|
||||
log.debug("rails: budget projection unavailable", exc_info=True)
|
||||
if money_bits:
|
||||
|
|
@ -1296,8 +1447,8 @@ def build_intrinsic_pacing_note(
|
|||
no deterministic time/round/cost stop (finalization stays P5 judgment).
|
||||
|
||||
``tree_cost_provider`` (v6.91): a zero-arg callable returning the root
|
||||
subtree's accounting snapshot (``{"accounted_usd", "root_limit_usd"}`` or
|
||||
None). Called ONLY when the note actually fires (a rare, already
|
||||
subtree's accounting snapshot (``{"settled_usd", "accounted_usd",
|
||||
"root_limit_usd"}`` or None). Called ONLY when the note actually fires (a rare, already
|
||||
cache-breaking surface — never per round), so a fresh ledger read at most
|
||||
once per pacing interval keeps the number honest after long child waits.
|
||||
Unknown stays "unknown", never $0."""
|
||||
|
|
@ -1316,6 +1467,7 @@ def build_intrinsic_pacing_note(
|
|||
cost = float(raw_cost) if raw_cost is not None else None
|
||||
cost_text = f"~${cost:.2f}" if cost is not None else "unknown"
|
||||
tree_line = ""
|
||||
tree_known: Optional[float] = None
|
||||
tree_accounted: Optional[float] = None
|
||||
tree_cap: Optional[float] = None
|
||||
if callable(tree_cost_provider):
|
||||
|
|
@ -1325,7 +1477,8 @@ def build_intrinsic_pacing_note(
|
|||
tree_info = None
|
||||
rendered = tree_spend_line(tree_info, getattr(ctx, "_cost_ceiling", None))
|
||||
if rendered:
|
||||
tree_accounted = float(tree_info["accounted_usd"])
|
||||
tree_known = float(tree_info["settled_usd"])
|
||||
tree_accounted = float(tree_info.get("accounted_usd") or tree_known)
|
||||
raw_cap = tree_info.get("root_limit_usd")
|
||||
tree_cap = float(raw_cap) if raw_cap is not None else None
|
||||
tree_line = f" | {rendered}"
|
||||
|
|
@ -1348,7 +1501,8 @@ def build_intrinsic_pacing_note(
|
|||
"rounds": int(round_idx),
|
||||
"cost": round(cost, 4) if cost is not None else None,
|
||||
}
|
||||
if tree_accounted is not None:
|
||||
if tree_known is not None:
|
||||
checkpoint["tree_known_usd"] = round(tree_known, 4)
|
||||
checkpoint["tree_accounted_usd"] = round(tree_accounted, 4)
|
||||
checkpoint["tree_cap_usd"] = round(tree_cap, 4) if tree_cap is not None else None
|
||||
ceiling = getattr(ctx, "_cost_ceiling", None)
|
||||
|
|
@ -1439,7 +1593,7 @@ def with_resource_facts(note: PacingNote, ctx: Any, usage: Optional[Dict[str, An
|
|||
log.debug("Pacing spend facts unavailable", exc_info=True)
|
||||
facts = {"tools": tools, "spend": spend}
|
||||
return PacingNote(
|
||||
text=note.text + "\nObserved resource facts (accounted money includes open holds):\n"
|
||||
text=note.text + "\nObserved resource facts (known spend is settled_usd; accounted money adds open holds):\n"
|
||||
+ json.dumps(facts, ensure_ascii=False, sort_keys=True, separators=(",", ":")),
|
||||
checkpoint={**note.checkpoint, "resource_facts": facts},
|
||||
)
|
||||
|
|
|
|||
|
|
@ -2,13 +2,18 @@
|
|||
|
||||
Recovery custody is separate from money. Dirty owners are acknowledged only
|
||||
through the revision whose cost was projected; a later receipt retains debt.
|
||||
Each pass observes what stays unresolved, per subject (open attempts, cost
|
||||
projections), as one id-to-reason map; only a change of that map is published,
|
||||
and only a reason new to the subject logs a representative traceback.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import errno
|
||||
import logging
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import re
|
||||
import threading
|
||||
from collections import Counter
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
|
@ -18,7 +23,13 @@ log = logging.getLogger(__name__)
|
|||
_PROBING: set[tuple[str, str]] = set()
|
||||
_PROBE_LOCK = threading.Lock()
|
||||
_UNRESOLVED_LOCK = threading.Lock()
|
||||
_LAST_UNRESOLVED: dict[tuple[str, str], tuple] = {}
|
||||
# (root, subject) -> the last observed id->reason map, the last published
|
||||
# signature, the last publication failure and (attempts) the last gateway-close
|
||||
# failure; current entries only.
|
||||
_LAST_UNRESOLVED: dict[tuple[str, str], dict] = {}
|
||||
ATTEMPTS, COST_PROJECTIONS = "attempt", "cost_projection"
|
||||
_ID_FIELDS = {ATTEMPTS: "attempt_ids", COST_PROJECTIONS: "task_ids"}
|
||||
_CODE = re.compile(r"[a-z][a-z0-9_.:-]{0,63}")
|
||||
|
||||
|
||||
class _RecoveryGateway:
|
||||
|
|
@ -69,30 +80,84 @@ def _unresolved(events, row, basis):
|
|||
events[str(row['attempt_id'])] = str(basis)
|
||||
|
||||
|
||||
def _publish_unresolved(root, observations, *, duty="terminal-maintenance", id_field="attempt_ids"):
|
||||
def _reason(exc):
|
||||
"""A stable failure reason: class plus errno or code, never a per-id message or path."""
|
||||
code = getattr(exc, "errno", None)
|
||||
code = errno.errorcode.get(code, code) if isinstance(code, int) else getattr(exc, "code", None)
|
||||
if code is None and _CODE.fullmatch(str(exc)):
|
||||
code = str(exc) # A typed refusal such as owner_pause_authority_missing.
|
||||
return f"{type(exc).__name__}:{code}" if code not in (None, "") else type(exc).__name__
|
||||
|
||||
|
||||
def _subject_state(root, subject):
|
||||
"""The subject's diagnostic memory; the caller holds ``_UNRESOLVED_LOCK``."""
|
||||
return _LAST_UNRESOLVED.setdefault((str(root), subject),
|
||||
{"observed": {}, "published": None, "fault": None, "close": None})
|
||||
|
||||
|
||||
def _close_gateway(root, gateway):
|
||||
"""A failed close of the pass's borrowed gateway retains nothing; it is logged once per change."""
|
||||
try:
|
||||
gateway.close()
|
||||
exc = reason = None
|
||||
except Exception as error:
|
||||
exc, reason = error, _reason(error)
|
||||
with _UNRESOLVED_LOCK:
|
||||
state = _subject_state(root, ATTEMPTS)
|
||||
repeated, state["close"] = state["close"] == reason, reason
|
||||
if reason and not repeated:
|
||||
log.debug("Usage recovery gateway close failed (%s)", reason, exc_info=exc)
|
||||
|
||||
|
||||
def _publish_unresolved(root, observations, *, subject=ATTEMPTS, faults=None, keep=(), complete=True):
|
||||
"""Publish one changed summary, without an interprocess lock or polling wait.
|
||||
|
||||
Outstanding custody stays in the store. These log rows are observations,
|
||||
like the generic log appender's unlocked fallback, never authority.
|
||||
``observations`` maps each id this pass found unresolved to its reason;
|
||||
an id in ``keep`` (still a candidate whose outcome this pass did not prove)
|
||||
keeps its previous reason, so a skipped or partial pass proves no recovery. The
|
||||
observed map is replaced every pass and holds only current ids. A reason
|
||||
absent from the previous map logs one representative traceback from
|
||||
``faults``; an unchanged pass logs nothing at any level. Publication is
|
||||
separate: an incomplete pass never publishes, and a failed write retries
|
||||
on the next pass with one warning per failure kind, so neither an
|
||||
unwritable log nor a stable error repeats a stack. Outstanding custody
|
||||
stays in the store; these rows are observations, never authority.
|
||||
"""
|
||||
from ouroboros.utils import _write_fd_fully, assert_test_data_path, utc_now_iso
|
||||
from supervisor.message_bus import try_get_bridge
|
||||
|
||||
ids = tuple(sorted(observations))
|
||||
by_basis = dict(sorted(Counter(observations.values()).items()))
|
||||
signature = len(ids), tuple(by_basis.items()), ids
|
||||
key = str(root), duty
|
||||
with _UNRESOLVED_LOCK:
|
||||
state = _subject_state(root, subject)
|
||||
previous = state["observed"]
|
||||
current = {key: previous[key] for key in keep if key in previous}
|
||||
current.update(observations)
|
||||
state["observed"] = current
|
||||
known = set(previous.values())
|
||||
fresh = {}
|
||||
for key, reason in sorted(current.items()):
|
||||
if reason not in known and (faults or {}).get(key) is not None:
|
||||
fresh.setdefault(reason, (key, faults[key]))
|
||||
for reason, (example, exc) in fresh.items():
|
||||
count = sum(1 for value in current.values() if value == reason)
|
||||
log.warning("Terminal maintenance %s: %d unresolved with %s (e.g. %s)", subject, count, reason, example,
|
||||
exc_info=exc)
|
||||
if not complete:
|
||||
return
|
||||
ids = tuple(sorted(current))
|
||||
by_basis = dict(sorted(Counter(current.values()).items()))
|
||||
signature = tuple(sorted(current.items()))
|
||||
path = root / "logs" / "supervisor.jsonl"
|
||||
assert_test_data_path(path)
|
||||
try:
|
||||
with _UNRESOLVED_LOCK:
|
||||
previous = _LAST_UNRESOLVED.get(key)
|
||||
if previous == signature or previous is None and not ids:
|
||||
if state["published"] == signature or state["published"] is None and not ids:
|
||||
return
|
||||
# 50 is the hard identifier-list size limit of one log row. Compare
|
||||
# the FULL id set above so changes beyond this display cap still emit.
|
||||
event = {"type": "duty_unresolved", "duty": duty, "ts": utc_now_iso(),
|
||||
"count": len(ids), "by_basis": by_basis, id_field: list(ids[:50])}
|
||||
# the FULL id->reason map above so changes beyond this display cap still emit.
|
||||
shown = ids[:50]
|
||||
event = {"type": "duty_unresolved", "duty": "terminal-maintenance", "subject": subject,
|
||||
"ts": utc_now_iso(), "count": len(ids), "by_basis": by_basis,
|
||||
_ID_FIELDS[subject]: list(shown), "reasons": {key: current[key] for key in shown}}
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
data = (json.dumps(event, ensure_ascii=False) + '\n').encode('utf-8')
|
||||
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o644)
|
||||
|
|
@ -100,14 +165,15 @@ def _publish_unresolved(root, observations, *, duty="terminal-maintenance", id_f
|
|||
_write_fd_fully(fd, data, path)
|
||||
finally:
|
||||
os.close(fd)
|
||||
_LAST_UNRESOLVED[key] = signature
|
||||
if duty == "terminal-cost-projection" and ids:
|
||||
log.warning("Terminal cost projection unresolved for %d owners: %s", len(ids), by_basis)
|
||||
state["published"], state["fault"] = signature, None
|
||||
bridge = try_get_bridge()
|
||||
if bridge is not None:
|
||||
bridge.push_log(event)
|
||||
except Exception:
|
||||
log.warning("Unresolved duty observations could not be published", exc_info=True)
|
||||
except Exception as exc:
|
||||
with _UNRESOLVED_LOCK:
|
||||
repeated, state["fault"] = state["fault"] == _reason(exc), _reason(exc)
|
||||
if not repeated:
|
||||
log.warning("Unresolved %s observations could not be published", subject, exc_info=True)
|
||||
|
||||
|
||||
def imported_projection_debt(root, buckets, *, degraded=False):
|
||||
|
|
@ -146,7 +212,7 @@ def imported_projection_debt(root, buckets, *, degraded=False):
|
|||
|
||||
def _refresh_costs(root: pathlib.Path, reads) -> None:
|
||||
from ouroboros import usage_store
|
||||
from ouroboros.task_results import validate_task_id
|
||||
from ouroboros.task_results import task_result_path, validate_task_id
|
||||
from ouroboros.task_status import SETTLED_STATUSES
|
||||
from ouroboros.post_task_checkpoint import post_task_synthesis_is_open
|
||||
from supervisor.events_task_done import _refresh_terminal_task_cost
|
||||
|
|
@ -167,12 +233,23 @@ def _refresh_costs(root: pathlib.Path, reads) -> None:
|
|||
|
||||
with usage_store.read(root) as txn:
|
||||
owners = txn.dirty_owners()
|
||||
acknowledged, observations = [], {}
|
||||
acknowledged, pending, failures, faults = [], set(), {}, {}
|
||||
for owner, revision in owners:
|
||||
try:
|
||||
task_id = validate_task_id(owner)
|
||||
except ValueError:
|
||||
continue # System accounting scopes own no task result.
|
||||
pending.add(task_id)
|
||||
try:
|
||||
# Presence precedes the ownership read: an absent body keeps this
|
||||
# exact revision without the fence's quarantine traversal, and the
|
||||
# next pass projects a body copied back after the last ledger write.
|
||||
task_result_path(root, task_id, create=False).stat()
|
||||
except FileNotFoundError:
|
||||
failures[task_id] = "result_absent"
|
||||
continue
|
||||
except OSError:
|
||||
pass # Only absence skips the read; the ownership read reports the rest.
|
||||
try:
|
||||
current = reads.load(task_id)
|
||||
if (not current or current.get("status") not in SETTLED_STATUSES
|
||||
|
|
@ -184,21 +261,16 @@ def _refresh_costs(root: pathlib.Path, reads) -> None:
|
|||
and _refresh_terminal_task_cost(root, tid, current=row)
|
||||
for tid, row in projection_targets):
|
||||
acknowledged.append((owner, revision))
|
||||
pending.discard(task_id)
|
||||
except Exception as exc:
|
||||
if isinstance(exc, ValueError) and str(exc) in {
|
||||
"owner_pause_authority_missing", "owner_pause_authority_unreadable",
|
||||
}:
|
||||
observations[task_id] = str(exc)
|
||||
log.debug("Task cost authority unavailable for %s", task_id, exc_info=True)
|
||||
else:
|
||||
log.warning("Reconciled task cost refresh failed for %s", task_id, exc_info=True)
|
||||
_publish_unresolved(root, observations, duty="terminal-cost-projection", id_field="task_ids")
|
||||
failures[task_id], faults[task_id] = _reason(exc), exc
|
||||
if acknowledged:
|
||||
# One short acknowledgement transaction after all result I/O. A crash
|
||||
# before it merely repeats successful projections; newer receipts win.
|
||||
with usage_store.hold(root) as txn:
|
||||
for owner, revision in acknowledged:
|
||||
txn.ack_dirty_owner(owner, revision)
|
||||
_publish_unresolved(root, failures, subject=COST_PROJECTIONS, faults=faults, keep=pending)
|
||||
|
||||
|
||||
def reconcile_abandoned_usage(drive_root: pathlib.Path) -> None:
|
||||
|
|
@ -220,7 +292,11 @@ def reconcile_abandoned_usage(drive_root: pathlib.Path) -> None:
|
|||
root = pathlib.Path(drive_root).resolve()
|
||||
with usage_store.read(root) as txn:
|
||||
rows = txn.open_attempts()
|
||||
reads, eligible, observations = TaskOwnershipRead(root), {}, {}
|
||||
reads, eligible, observations, faults = TaskOwnershipRead(root), {}, {}, {}
|
||||
# An open attempt keeps its last reason unless this pass observes a new one or
|
||||
# proves an outcome: a recovered receipt, a release, a terminal settlement or
|
||||
# terminal remote custody. Skipped, live or lost-race outcomes prove nothing.
|
||||
unproven = {str(row["attempt_id"]) for row in rows}
|
||||
probes_complete = True
|
||||
gateway, gateway_unavailable = None, False
|
||||
|
||||
|
|
@ -283,6 +359,7 @@ def reconcile_abandoned_usage(drive_root: pathlib.Path) -> None:
|
|||
probes_complete = False # A partial observation cannot declare the set empty.
|
||||
continue
|
||||
_PROBING.add(probe_key)
|
||||
attempt_id = str(row["attempt_id"])
|
||||
reservation = usage.AttemptReservation(
|
||||
str(row["attempt_id"]), root, str(row.get("model") or ""),
|
||||
str(row.get("provider") or ""), row.get("reservation_upper_bound_usd"),
|
||||
|
|
@ -304,14 +381,17 @@ def reconcile_abandoned_usage(drive_root: pathlib.Path) -> None:
|
|||
if disposition == "settled":
|
||||
usage.settle_attempt(reservation, reported, cost_usd=cost, cost_final=final,
|
||||
expected_revision=row["revision"])
|
||||
unproven.discard(attempt_id) # The receipt is the proof, whichever settlement won.
|
||||
elif disposition == "released":
|
||||
if not release_pre_dispatch_attempt(reservation, ProviderNotDispatched("recovered model operation never started"),
|
||||
expected_revision=row["revision"]):
|
||||
continue
|
||||
unproven.discard(attempt_id)
|
||||
elif disposition == "abandoned":
|
||||
if abandoned:
|
||||
if remote and (row.get("recovery") or {}).get("outcome") not in {"terminal", "operation_gone"}:
|
||||
_recovery_fact(root, row, "terminal", "recover_model_attempt:abandoned")
|
||||
unproven.discard(attempt_id)
|
||||
continue
|
||||
state = usage.terminalize_abandoned_attempt(
|
||||
reservation, reason="owner_task_terminal", expected_revision=row.get("revision"))
|
||||
|
|
@ -320,10 +400,12 @@ def reconcile_abandoned_usage(drive_root: pathlib.Path) -> None:
|
|||
if remote and recovered is not None:
|
||||
_recovery_fact(root, row, "terminal", "recover_model_attempt:abandoned",
|
||||
revision=row["revision"] + 1)
|
||||
unproven.discard(attempt_id)
|
||||
else:
|
||||
continue
|
||||
except ClaudexorUnavailable as exc:
|
||||
if exc.code == "recovery_terminal":
|
||||
unproven.discard(attempt_id) # Terminal remote custody is already a durable fact.
|
||||
continue
|
||||
if exc.code == "recovery_not_due":
|
||||
basis = (row.get("recovery") or {}).get("basis") or {}
|
||||
|
|
@ -339,25 +421,22 @@ def reconcile_abandoned_usage(drive_root: pathlib.Path) -> None:
|
|||
_recovery_fact(root, row, "operation_gone", {"operation_get_status": exc.status_code,
|
||||
"code": str(exc)},
|
||||
revision=row["revision"] + (0 if abandoned else 1))
|
||||
if state in {"settled", "released"}:
|
||||
unproven.discard(attempt_id)
|
||||
continue
|
||||
if due := _retry_after(exc.retry_after):
|
||||
_recovery_fact(root, row, "unresolved", {"header": "Retry-After", "value": exc.retry_after,
|
||||
"code": exc.code}, due_at=due)
|
||||
gateway_unavailable = gateway_unavailable or exc.code == "daemon_unreachable"
|
||||
_unresolved(observations, row, exc.code)
|
||||
log.debug("Model usage custody deferred for %s: %s", row["attempt_id"], exc.code)
|
||||
except Exception:
|
||||
_unresolved(observations, row, "recovery_failed")
|
||||
log.warning("Usage reconciliation deferred for %s", row["attempt_id"], exc_info=True)
|
||||
except Exception as exc:
|
||||
_unresolved(observations, row, f"recovery_failed:{_reason(exc)}")
|
||||
faults[attempt_id] = exc
|
||||
finally:
|
||||
with _PROBE_LOCK:
|
||||
_PROBING.discard(probe_key)
|
||||
finally:
|
||||
if gateway is not None:
|
||||
try:
|
||||
gateway.close()
|
||||
except Exception:
|
||||
log.debug("Usage recovery gateway close failed", exc_info=True)
|
||||
if probes_complete:
|
||||
_publish_unresolved(root, observations)
|
||||
_close_gateway(root, gateway)
|
||||
_publish_unresolved(root, observations, faults=faults, keep=unproven, complete=probes_complete)
|
||||
_refresh_costs(root, reads)
|
||||
|
|
|
|||
|
|
@ -647,9 +647,9 @@ def _budget_refusal(ctx: ToolContext, callable_slots: list, delivery: dict) -> s
|
|||
return ""
|
||||
fence, remedy = review_wave_binding_fence(admission)
|
||||
return ("⚠️ PLAN_REVIEW_SKIPPED_BUDGET: the reviewer wave was declined before dispatch — "
|
||||
f"estimated cost ~${admission.get('estimated_wave_usd')} exceeds the remaining budget "
|
||||
f"${admission.get('remaining_usd')} ({fence}). No reviewer was called. Shrink the evidence, "
|
||||
f"split the plan, or {remedy}.")
|
||||
f"known spend has reached the {fence} (remaining ${admission.get('remaining_usd')}; "
|
||||
f"the wave was estimated at ~${admission.get('estimated_wave_usd')}). No reviewer was called. "
|
||||
f"To continue, {remedy}.")
|
||||
|
||||
|
||||
def _carried_dispositions(existing: dict, named: list) -> list:
|
||||
|
|
|
|||
|
|
@ -669,13 +669,15 @@ def commit_gate_paid_seats(triad_prepared, triad_exited, scope_rows) -> list:
|
|||
|
||||
|
||||
def admit_commit_gate_wave(ctx, seats) -> str | None:
|
||||
"""All-or-nothing money admission of one commit-gate wave (owner decision
|
||||
2026-09-05): every paid seat's reservation upper bound must fit TOGETHER,
|
||||
against every fence ``reserve_attempt`` enforces (the global TOTAL_BUDGET
|
||||
remainder, root and original group fences), before ANY seat is dispatched. Returns the
|
||||
typed refusal text ($0, nothing dispatched) naming the binding axis, or
|
||||
None; fail-open on unknowns like the task-level surfaces that already ride
|
||||
``review_wave_budget_gate``."""
|
||||
"""Money admission of one commit-gate wave (owner decision 2026-09-05, on
|
||||
the known-spend rule of #1487): before ANY seat is dispatched, KNOWN spend
|
||||
must be below every fence ``reserve_attempt`` enforces (the global
|
||||
TOTAL_BUDGET, root and original group fences). The seats' summed reservation
|
||||
bounds are disclosed, not an earlier refusal; a fence reached mid-wave
|
||||
refuses the remaining seats at their own reservation with truthful custody.
|
||||
Returns the typed refusal text ($0, nothing dispatched) naming the binding
|
||||
axis, or None; fail-open on unknowns like the task-level surfaces that
|
||||
already ride ``review_wave_budget_gate``."""
|
||||
if not seats:
|
||||
return None
|
||||
from ouroboros.review_substrate import review_usage_category
|
||||
|
|
@ -698,32 +700,30 @@ def admit_commit_gate_wave(ctx, seats) -> str | None:
|
|||
usd = lambda value: "unknown" if value is None else f"${float(value):.6f}" # noqa: E731
|
||||
bounds = list(admission.get("slot_bounds") or []) + [None] * len(seats)
|
||||
wave, remaining = admission.get("estimated_wave_usd"), admission.get("remaining_usd")
|
||||
shortfall = None if wave is None or remaining is None else max(0.0, float(wave) - float(remaining))
|
||||
limit, accounted = admission.get("limit_usd"), admission.get("accounted_usd")
|
||||
root_remaining = None if limit is None or accounted is None else max(0.0, float(limit) - float(accounted))
|
||||
limit, known = admission.get("limit_usd"), admission.get("known_usd")
|
||||
root_remaining = None if limit is None or known is None else max(0.0, float(limit) - float(known))
|
||||
if admission.get("binding_axis") == "global":
|
||||
# The refusal names the fence that binds and the knob that moves it — never
|
||||
# a per-task fence the wave would have fit.
|
||||
fence = (
|
||||
f"the global budget TOTAL_BUDGET {usd(admission.get('global_limit_usd'))}: "
|
||||
f"accounted={usd(admission.get('global_accounted_usd'))} across every task (of which "
|
||||
f"{usd(admission.get('global_reserved_usd'))} is reserved by other in-flight attempts), "
|
||||
f"remaining={usd(remaining)}, shortfall={usd(shortfall)}; the per-task budget fence "
|
||||
f"known spend={usd(admission.get('global_known_usd'))} across every task (plus "
|
||||
f"{usd(admission.get('global_reserved_usd'))} of open holds, not counted), "
|
||||
f"remaining={usd(remaining)}; the per-task budget fence "
|
||||
f"{usd(limit)} alone would leave {usd(root_remaining)}"
|
||||
)
|
||||
else:
|
||||
label = "whole-work billing-group budget fence" if admission.get("binding_axis") == "group" else "per-task budget fence"
|
||||
fence = (
|
||||
f"the {label} {usd(limit)}: accounted={usd(accounted)} (of which "
|
||||
f"{usd(admission.get('reserved_usd'))} is reserved by other in-flight attempts), "
|
||||
f"remaining={usd(remaining)}, shortfall={usd(shortfall)}; the global budget "
|
||||
f"the {label} {usd(limit)}: known spend={usd(known)} (plus "
|
||||
f"{usd(admission.get('reserved_usd'))} of open holds, not counted), "
|
||||
f"remaining={usd(remaining)}; the global budget "
|
||||
f"{usd(admission.get('global_limit_usd'))} alone would leave {usd(admission.get('global_remaining_usd'))}"
|
||||
)
|
||||
remedy = review_wave_binding_fence(admission)[1]
|
||||
return (
|
||||
"⚠️ REVIEW_BLOCKED: commit-gate review wave declined before dispatch ($0 spent). "
|
||||
f"The wave's reservation upper bound {usd(wave)} ("
|
||||
f"Known spend has reached {fence}. The wave's reservation upper bound would have been {usd(wave)} ("
|
||||
+ "; ".join(f"{s['surface']}:{s['slot_id']} {s['model']} {usd(bounds[i])}" for i, s in enumerate(seats))
|
||||
+ f") does not fit {fence}. No reviewer seat was dispatched (scope and triad alike): wait for "
|
||||
f"in-flight attempts to settle or {remedy}, then retry the same commit."
|
||||
+ f"). No reviewer seat was dispatched (scope and triad alike): {remedy}, then retry the same commit."
|
||||
)
|
||||
|
|
|
|||
|
|
@ -246,9 +246,11 @@ def review_wave_budget_gate(
|
|||
reads the seat's own observed cache split rather than the caller's), against
|
||||
every fence ``reserve_attempt`` enforces — global TOTAL_BUDGET, the task's
|
||||
current root and its original billing group — with the binding axis and
|
||||
remainders named in the event. A wave that fits at admission time is
|
||||
dispatched whole; one that does not is refused before any seat spends.
|
||||
Fail-open on any error/unknown."""
|
||||
remainders named in the event. The wave is admitted while KNOWN spend is
|
||||
below every limit (#1487): its summed seat bounds are disclosed, never an
|
||||
earlier refusal. A limit reached mid-wave refuses the remaining seats at
|
||||
their own reservation and dispatch fences, with custody of what was sent.
|
||||
A wave declined here spent nothing. Fail-open on any error/unknown."""
|
||||
try:
|
||||
from ouroboros.usage_accounting import current_usage_scope
|
||||
from ouroboros.usage_admission import review_wave_admission
|
||||
|
|
@ -278,10 +280,12 @@ def review_wave_budget_gate(
|
|||
"estimated_wave_usd": admission.get("estimated_wave_usd"),
|
||||
"remaining_usd": admission.get("remaining_usd"),
|
||||
"limit_usd": admission.get("limit_usd"),
|
||||
"known_usd": admission.get("known_usd"),
|
||||
"accounted_usd": admission.get("accounted_usd"),
|
||||
"reserved_usd": admission.get("reserved_usd"),
|
||||
**{key: admission.get(key) for key in (
|
||||
"binding_axis", "global_limit_usd", "global_accounted_usd", "global_reserved_usd", "global_remaining_usd")},
|
||||
"binding_axis", "global_limit_usd", "global_known_usd", "global_accounted_usd",
|
||||
"global_reserved_usd", "global_remaining_usd")},
|
||||
"slots": admission.get("slots"),
|
||||
"slot_bounds": admission.get("slot_bounds"),
|
||||
"unpriced_slots": unpriced,
|
||||
|
|
@ -311,12 +315,12 @@ def review_wave_binding_fence(admission: dict) -> tuple[str, str]:
|
|||
"""Name the binding global/root/group fence and its actual remedy."""
|
||||
usd = lambda key: "unknown" if admission.get(key) is None else f"${float(admission[key]):.6f}" # noqa: E731
|
||||
if admission.get("binding_axis") == "global":
|
||||
return (f"global budget TOTAL_BUDGET {usd('global_limit_usd')}, accounted {usd('global_accounted_usd')} "
|
||||
return (f"global budget TOTAL_BUDGET {usd('global_limit_usd')}, known spend {usd('global_known_usd')} "
|
||||
"across every task", "raise TOTAL_BUDGET")
|
||||
if admission.get("binding_axis") == "group":
|
||||
return (f"whole-work billing-group budget fence {usd('limit_usd')}, accounted {usd('accounted_usd')}",
|
||||
return (f"whole-work billing-group budget fence {usd('limit_usd')}, known spend {usd('known_usd')}",
|
||||
"amend the original billing-group owner's cap explicitly; changing the per-task setting does not amend it")
|
||||
return f"per-task budget fence {usd('limit_usd')}, accounted {usd('accounted_usd')}", (
|
||||
return f"per-task budget fence {usd('limit_usd')}, known spend {usd('known_usd')}", (
|
||||
"raise the per-task budget (OUROBOROS_PER_TASK_COST_USD)")
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -90,13 +90,16 @@ _ROOT_ACCOUNTING_TELEMETRY_CAP = 64
|
|||
_ROOT_RESERVATIONS_KEPT = 8 # identities of the newest appended reservations per root
|
||||
def _stash_root_accounting(
|
||||
root_task_id: str,
|
||||
accounted_usd: Optional[float],
|
||||
spend: Any,
|
||||
root_limit_usd: Optional[float],
|
||||
reservation: Optional[Dict[str, Any]] = None,
|
||||
*,
|
||||
integrity_degraded: bool = False,
|
||||
) -> None:
|
||||
"""Refresh the process-local root snapshot. ``reservation`` is the identity
|
||||
"""Refresh the process-local root snapshot. ``spend`` is the rendered
|
||||
bucket: its ``settled_usd`` is the known spend money readers decide on, its
|
||||
``accounted_usd`` the exposure including holds; a missing field stays
|
||||
unknown (``None``), never borrowed from the other. ``reservation`` is the identity
|
||||
of a row this call has just APPENDED (attempt id, task, category, review
|
||||
slot): only a successful ``reserve_attempt`` passes one, so a reader that
|
||||
finds its own identity here has observed its own reservation — a refresh,
|
||||
|
|
@ -104,6 +107,9 @@ def _stash_root_accounting(
|
|||
root_task_id = str(root_task_id or "").strip()
|
||||
if not root_task_id:
|
||||
return
|
||||
if not isinstance(spend, dict):
|
||||
raise TypeError("root accounting snapshot needs a rendered money bucket")
|
||||
accounted_usd, settled_usd = _number(spend.get("accounted_usd")), _number(spend.get("settled_usd"))
|
||||
with _ROOT_ACCOUNTING_TELEMETRY_LOCK:
|
||||
if (
|
||||
root_task_id not in _ROOT_ACCOUNTING_TELEMETRY
|
||||
|
|
@ -119,6 +125,7 @@ def _stash_root_accounting(
|
|||
if reservation:
|
||||
kept = (kept + [{**reservation, "reserved_monotonic": now}])[-_ROOT_RESERVATIONS_KEPT:]
|
||||
_ROOT_ACCOUNTING_TELEMETRY[root_task_id] = {
|
||||
"settled_usd": None if settled_usd is None else float(settled_usd),
|
||||
"accounted_usd": None if accounted_usd is None else float(accounted_usd),
|
||||
"root_limit_usd": None if root_limit_usd is None else float(root_limit_usd),
|
||||
# The projection's own integrity verdict rides the snapshot (#1196): a
|
||||
|
|
@ -130,9 +137,10 @@ def _stash_root_accounting(
|
|||
}
|
||||
|
||||
def last_root_accounting(root_task_id: str) -> Optional[Dict[str, Any]]:
|
||||
"""Newest process-local root snapshot, including in-flight holds and the
|
||||
identities of the newest appended reservations (each with its own
|
||||
``age_sec``)."""
|
||||
"""Newest process-local root snapshot: the KNOWN spend (``settled_usd``)
|
||||
limits decide on, the exposure including in-flight holds
|
||||
(``accounted_usd``), and the identities of the newest appended
|
||||
reservations (each with its own ``age_sec``)."""
|
||||
with _ROOT_ACCOUNTING_TELEMETRY_LOCK:
|
||||
entry = _ROOT_ACCOUNTING_TELEMETRY.get(str(root_task_id or "").strip())
|
||||
if entry is None:
|
||||
|
|
@ -175,7 +183,7 @@ def refresh_root_accounting(
|
|||
projection = usage_projection(drive_root, root_task_id="" if group else root_task_id, billing_group_id=group)
|
||||
_stash_root_accounting(
|
||||
root_task_id,
|
||||
_number(projection.get("accounted_usd")),
|
||||
projection,
|
||||
_number(projection.get("limit_usd")),
|
||||
integrity_degraded=bool(projection.get("integrity_degraded")),
|
||||
)
|
||||
|
|
@ -184,8 +192,15 @@ def refresh_root_accounting(
|
|||
log.debug("root accounting refresh failed for %s", root_task_id, exc_info=True)
|
||||
return cached
|
||||
|
||||
def _known_spend_text(summary: Dict[str, Any]) -> str:
|
||||
"""A refusal's money: the known spend that decided it, the open holds beside it."""
|
||||
holds = (_number(summary.get("reserved_usd")) or 0.0) + (_number(summary.get("unresolved_upper_bound_usd")) or 0.0)
|
||||
return (f"known=${float(_number(summary.get('settled_usd')) or 0.0):.6f} "
|
||||
f"(open holds ${holds:.6f} not counted as spending)")
|
||||
|
||||
|
||||
class BudgetExceeded(UsageAccountingError):
|
||||
"""Raised before dispatch when a known budget would be exceeded."""
|
||||
"""Raised before dispatch when known spend has reached an applicable limit."""
|
||||
|
||||
def __init__(self, message: str, *, limit_scope: str = "global", root_task_id: str = "") -> None:
|
||||
super().__init__(message)
|
||||
|
|
@ -703,30 +718,29 @@ def reserve_attempt(request: AttemptRequest) -> AttemptReservation:
|
|||
with acquire(root) as view:
|
||||
_check_dispatch_fences(scope, root)
|
||||
global_limit = _global_limit(request)
|
||||
accounted = view.summary()["accounted_usd"]
|
||||
if view.exceeds_limit(global_limit, bound):
|
||||
if view.exceeds_limit(global_limit):
|
||||
raise BudgetExceeded(
|
||||
f"global model budget exhausted: accounted=${accounted:.6f}, "
|
||||
f"reservation={'unknown' if bound is None else f'${bound:.6f}'}, limit=${global_limit:.6f}",
|
||||
f"global model budget exhausted: {_known_spend_text(view.summary())}, "
|
||||
f"limit=${global_limit:.6f}",
|
||||
limit_scope="global",
|
||||
root_task_id=scope.root_task_id,
|
||||
)
|
||||
root_limit: Optional[float] = None
|
||||
if scope.root_task_id: # every rooted attempt refreshes the subtree telemetry, cap or not
|
||||
root_accounted = view.summary(scope.root_task_id)["accounted_usd"]
|
||||
root_summary = view.summary(scope.root_task_id)
|
||||
root_limit = None if scope.root_limit_usd is None else max(0.0, float(scope.root_limit_usd))
|
||||
_stash_root_accounting(scope.root_task_id, root_accounted, root_limit) # pre-append subtree sum
|
||||
_stash_root_accounting(scope.root_task_id, root_summary, root_limit) # pre-append subtree sum
|
||||
if root_limit is not None:
|
||||
if view.exceeds_limit(root_limit, bound, root_task_id=scope.root_task_id):
|
||||
if view.exceeds_limit(root_limit, root_task_id=scope.root_task_id):
|
||||
raise BudgetExceeded(
|
||||
f"root model budget exhausted for {scope.root_task_id}: "
|
||||
f"accounted=${root_accounted:.6f}, limit=${root_limit:.6f}",
|
||||
f"{_known_spend_text(root_summary)}, limit=${root_limit:.6f}",
|
||||
limit_scope="root",
|
||||
root_task_id=scope.root_task_id,
|
||||
)
|
||||
from ouroboros.usage_admission import raise_group_refusal, scope_group
|
||||
|
||||
raise_group_refusal(view, scope, bound)
|
||||
raise_group_refusal(view, scope)
|
||||
group_id, group_limit = scope_group(scope)
|
||||
view.write(
|
||||
{
|
||||
|
|
@ -778,11 +792,11 @@ def reserve_attempt(request: AttemptRequest) -> AttemptReservation:
|
|||
)
|
||||
if group_id:
|
||||
_stash_root_accounting(f"group:{group_id}",
|
||||
view.summary(billing_group_id=group_id)["accounted_usd"], group_limit)
|
||||
view.summary(billing_group_id=group_id), group_limit)
|
||||
if scope.root_task_id:
|
||||
_stash_root_accounting(
|
||||
scope.root_task_id,
|
||||
view.summary(scope.root_task_id)["accounted_usd"],
|
||||
view.summary(scope.root_task_id),
|
||||
root_limit,
|
||||
reservation={
|
||||
"attempt_id": attempt_id, "task_id": scope.task_id,
|
||||
|
|
@ -1017,12 +1031,14 @@ def _transition(reservation: AttemptReservation, state: str, **fields: Any) -> D
|
|||
("global", _global_limit(limit_request), None),
|
||||
("root", scope.root_limit_usd, scope.root_task_id),
|
||||
):
|
||||
if limit is not None and view.exceeds_limit(limit, root_task_id=identity, dispatch=True):
|
||||
# The reservation's own predicate: known spend reached the limit
|
||||
# since this attempt was reserved, so nothing is sent.
|
||||
if limit is not None and view.exceeds_limit(limit, root_task_id=identity):
|
||||
raise BudgetExceeded(f"{axis} model budget changed before dispatch", limit_scope=axis,
|
||||
root_task_id=scope.root_task_id)
|
||||
from ouroboros.usage_admission import raise_group_refusal
|
||||
|
||||
raise_group_refusal(view, scope, dispatch=True)
|
||||
raise_group_refusal(view, scope)
|
||||
replaced = {"seq", "ts", "revision", "pre_compaction_seq", "settle_reason", "reason"}
|
||||
if state == "settled":
|
||||
replaced.update(("effort", "effort_resolution", "processing", "speed", "service_tier", "cost_basis", "cost_evidence"))
|
||||
|
|
@ -1050,11 +1066,11 @@ def _transition(reservation: AttemptReservation, state: str, **fields: Any) -> D
|
|||
group_id = billing_group_key(current)
|
||||
if group_id:
|
||||
_stash_root_accounting(f"group:{group_id}",
|
||||
view.summary(billing_group_id=group_id)["accounted_usd"],
|
||||
view.summary(billing_group_id=group_id),
|
||||
_number(current.get("billing_group_limit_usd", current.get("root_limit_usd"))))
|
||||
root_task_id = str(current.get("root_task_id") or "")
|
||||
if root_task_id:
|
||||
_stash_root_accounting(root_task_id, view.summary(root_task_id)["accounted_usd"],
|
||||
_stash_root_accounting(root_task_id, view.summary(root_task_id),
|
||||
_number(current.get("root_limit_usd")))
|
||||
return copy.deepcopy(appended)
|
||||
|
||||
|
|
|
|||
|
|
@ -16,11 +16,13 @@ by the original root without being rewritten. The check is symmetric: the
|
|||
original root's own later work (its reviews, its post-work) is a member of the
|
||||
same group and sees every successor's spend. Nothing here is a second ledger:
|
||||
the group axis is the store's ``group`` summary (``billing_group_key``), kept
|
||||
in the same transaction as every attempt row. Unknown-priced admission keeps its
|
||||
existing policy (an unknown bound is admitted while the known accounting still
|
||||
fits); the projection discloses unknown liabilities and never promises a bound
|
||||
on an eventual bill. A late charge that exceeds the cap is recorded and bars
|
||||
the next admission; it was not prevented.
|
||||
in the same transaction as every attempt row. Every axis admits while its
|
||||
KNOWN spend (the settled bucket: confirmed prices and disclosed estimates) is
|
||||
below its limit (owner Q4-A, #1487); reservations and unresolved bounds are
|
||||
disclosed exposure, never counted as spending, and an unknown price is never a
|
||||
zero. The projection never promises a bound on an eventual bill: concurrent,
|
||||
in-flight and late charges can exceed the cap. Such a charge is recorded and
|
||||
bars the next admission; it was not prevented.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -234,8 +236,8 @@ def settlement_billing_fields(root: Any, task_id: str, root_task_id: str) -> Dic
|
|||
return binding
|
||||
|
||||
|
||||
def raise_group_refusal(view: Any, scope: Any, bound: Optional[float] = None, *, dispatch: bool = False) -> None:
|
||||
"""The group axis of ONE reservation, inside the caller's store transaction."""
|
||||
def raise_group_refusal(view: Any, scope: Any) -> None:
|
||||
"""The group axis of ONE reservation or dispatch, inside the caller's store transaction."""
|
||||
from ouroboros import usage_accounting as ua
|
||||
|
||||
group, limit = scope_group(scope)
|
||||
|
|
@ -244,10 +246,10 @@ def raise_group_refusal(view: Any, scope: Any, bound: Optional[float] = None, *,
|
|||
limit_scope="root", root_task_id=str(getattr(scope, "root_task_id", "") or ""))
|
||||
if not group or limit is None:
|
||||
return
|
||||
accounted = view.summary(billing_group_id=group)["accounted_usd"]
|
||||
if view.exceeds_limit(limit, bound, billing_group_id=group, dispatch=dispatch):
|
||||
if view.exceeds_limit(limit, billing_group_id=group):
|
||||
raise ua.BudgetExceeded(
|
||||
f"whole-work budget exhausted for group {group}: accounted=${accounted:.6f}, limit=${limit:.6f}",
|
||||
f"whole-work budget exhausted for group {group}: "
|
||||
f"{ua._known_spend_text(view.summary(billing_group_id=group))}, limit=${limit:.6f}",
|
||||
limit_scope="root", root_task_id=str(getattr(scope, "root_task_id", "") or group))
|
||||
|
||||
|
||||
|
|
@ -304,18 +306,23 @@ def review_wave_admission(
|
|||
slot_ids: str | Sequence[str] = "",
|
||||
processing_preferences: str | Sequence[str] = "",
|
||||
) -> Dict[str, Any]:
|
||||
"""Read-only whole-wave admission through each slot's reservation math.
|
||||
"""Read-only wave admission on the reservation's own known-spend rule.
|
||||
|
||||
Standalone callers may supply remaining_usd_override. Otherwise the tighter
|
||||
global/root remainder binds, including every in-flight hold; unknown prices
|
||||
stay unknown. An explicit root_limit_usd is the caller's current fence,
|
||||
otherwise the ledger's historical minimum governs. Global None resolves
|
||||
settings; a non-positive configured limit is unbounded.
|
||||
The wave is admitted while KNOWN spend is below every applicable limit
|
||||
(global, root, original group), exactly what each seat's reservation will
|
||||
check (#1487). The summed seat bounds are information, never an earlier
|
||||
refusal: a wave that crosses the limit mid-dispatch is refused per seat by
|
||||
the reservation and dispatch fences, with truthful custody of what was sent.
|
||||
Standalone callers may supply remaining_usd_override (known room). An
|
||||
explicit root_limit_usd is the caller's current fence, otherwise the
|
||||
ledger's historical minimum governs. Global None resolves settings; a
|
||||
non-positive configured limit is unbounded. Open holds are disclosed beside
|
||||
the known spend; unknown prices stay unknown.
|
||||
|
||||
Input sizes, outputs, categories, slots and processing can be scalar or
|
||||
aligned per-slot values. Price each seat under its own sending scope, so the
|
||||
caller's warm cache split cannot stand in for a reviewer's cold prefix.
|
||||
Returned per-slot bounds and both remainders disclose the binding cause.
|
||||
Returned per-slot bounds and both remainders disclose the binding axis.
|
||||
"""
|
||||
from ouroboros import usage_accounting as ua
|
||||
|
||||
|
|
@ -326,11 +333,12 @@ def review_wave_admission(
|
|||
"limit_usd": None,
|
||||
"slots": len(list(models or [])),
|
||||
"unpriced_slots": 0,
|
||||
"known_usd": None,
|
||||
"accounted_usd": None,
|
||||
"reserved_usd": None,
|
||||
"slot_bounds": [],
|
||||
**{key: None for key in ("global_limit_usd", "global_accounted_usd", "global_remaining_usd",
|
||||
"global_reserved_usd", "binding_axis")},
|
||||
**{key: None for key in ("global_limit_usd", "global_known_usd", "global_accounted_usd",
|
||||
"global_remaining_usd", "global_reserved_usd", "binding_axis")},
|
||||
}
|
||||
root_task_id = str(root_task_id or "").strip()
|
||||
if not root_task_id or not models:
|
||||
|
|
@ -341,8 +349,8 @@ def review_wave_admission(
|
|||
if remaining_usd_override is not None:
|
||||
remaining = float(remaining_usd_override)
|
||||
else:
|
||||
# Every OPEN hold counts as reserved-by-others: a reserved row and a
|
||||
# dispatched (in-flight) row both bind their upper bound on the fence.
|
||||
# Open holds (reserved and in-flight/unresolved bounds) are disclosed
|
||||
# exposure beside the known spend; they do not shrink the room.
|
||||
holds = lambda p: round(float(ua._number(p.get("reserved_usd")) or 0.0) # noqa: E731
|
||||
+ float(ua._number(p.get("unresolved_upper_bound_usd")) or 0.0), 6)
|
||||
projection = ua.usage_projection(drive_root, root_task_id=root_task_id)
|
||||
|
|
@ -350,12 +358,12 @@ def review_wave_admission(
|
|||
max(0.0, float(root_limit_usd)) if root_limit_usd is not None
|
||||
else None if root_limit_source else ua._number(projection.get("limit_usd"))
|
||||
)
|
||||
accounted = ua._number(projection.get("accounted_usd"))
|
||||
known = ua._number(projection.get("settled_usd"))
|
||||
remaining = None
|
||||
if limit is not None and accounted is not None:
|
||||
remaining = round(max(0.0, limit - accounted), 6)
|
||||
result.update(limit_usd=limit, accounted_usd=accounted, reserved_usd=holds(projection),
|
||||
binding_axis="root")
|
||||
if limit is not None and known is not None:
|
||||
remaining = round(max(0.0, limit - known), 6)
|
||||
result.update(limit_usd=limit, known_usd=known, accounted_usd=ua._number(projection.get("accounted_usd")),
|
||||
reserved_usd=holds(projection), binding_axis="root")
|
||||
# Resolve the same task-bound durable group and owner amendments
|
||||
# as a seat reservation, without reserving money or pinning a binding.
|
||||
# A raised in-memory root fence alone cannot raise the original group.
|
||||
|
|
@ -369,16 +377,19 @@ def review_wave_admission(
|
|||
return {**result, "fits": False, "binding_axis": "group", "reason": "billing_authority_unavailable"}
|
||||
if group and group_limit is not None:
|
||||
group_projection = ua.usage_projection(drive_root, billing_group_id=group)
|
||||
group_accounted = ua._number(group_projection.get("accounted_usd"))
|
||||
if group_accounted is not None:
|
||||
group_remaining = round(max(0.0, group_limit - group_accounted), 6)
|
||||
group_known = ua._number(group_projection.get("settled_usd"))
|
||||
if group_known is not None:
|
||||
group_remaining = round(max(0.0, group_limit - group_known), 6)
|
||||
if remaining is None or group_remaining < remaining:
|
||||
remaining = group_remaining
|
||||
result.update(limit_usd=group_limit, accounted_usd=group_accounted,
|
||||
result.update(limit_usd=group_limit, known_usd=group_known,
|
||||
accounted_usd=ua._number(group_projection.get("accounted_usd")),
|
||||
reserved_usd=holds(group_projection),
|
||||
binding_axis="group", billing_group_id=group)
|
||||
# The global axis reserve_attempt checks FIRST (all roots' rows, open holds included).
|
||||
# The global axis reserve_attempt checks FIRST (all roots' rows).
|
||||
gp = ua.usage_projection(drive_root, global_limit_usd=global_limit_usd, include_roots=False)
|
||||
result.update(global_limit_usd=ua._number(gp.get("limit_usd")),
|
||||
global_known_usd=ua._number(gp.get("settled_usd")),
|
||||
global_accounted_usd=ua._number(gp.get("accounted_usd")),
|
||||
global_remaining_usd=ua._number(gp.get("remaining_known_usd")), global_reserved_usd=holds(gp))
|
||||
global_remaining = result["global_remaining_usd"]
|
||||
|
|
@ -422,7 +433,8 @@ def review_wave_admission(
|
|||
continue
|
||||
total += float(bound)
|
||||
result["estimated_wave_usd"] = round(total, 6)
|
||||
result["fits"] = total <= remaining + 1e-9
|
||||
# The reservation's equality: no room once known spend reached the limit.
|
||||
result["fits"] = remaining > 1e-9
|
||||
return result
|
||||
except Exception:
|
||||
log.debug("review_wave_admission failed open", exc_info=True)
|
||||
|
|
@ -484,14 +496,16 @@ def amended_projection(root, identity, projection, *, group=False):
|
|||
limit = 0.0
|
||||
result.update({key: value for key, value in fields.items() if key.endswith(("source", "revision"))})
|
||||
if limit is not None:
|
||||
result.update(limit_usd=limit, remaining_known_usd=round(max(0., limit - result["accounted_usd"]), 6))
|
||||
result.update(limit_usd=limit, remaining_known_usd=round(max(0., limit - result["settled_usd"]), 6))
|
||||
return result
|
||||
|
||||
|
||||
def task_money_snapshot(root, task, root_id, *, root_limit=None):
|
||||
"""One ledger observation of both independent ceilings, on group spend basis.
|
||||
|
||||
root_limit_usd is the effective allowance on that basis; actual authored
|
||||
Rooms are limit minus KNOWN (settled) spend, the reservation's own rule;
|
||||
``accounted_usd`` is the group's exposure including open holds, disclosed.
|
||||
root_limit_usd is the effective allowance on the known basis; actual authored
|
||||
caps and attribution are exposed in root_axis/group_axis. Never cached
|
||||
under a shared group key, and never an amendment to a saved cost ceiling.
|
||||
"""
|
||||
|
|
@ -508,17 +522,18 @@ def task_money_snapshot(root, task, root_id, *, root_limit=None):
|
|||
cap = fields.get("root_limit_usd")
|
||||
if cap is None and not fields.get("root_limit_source") and initial.get("root_limit_usd") is not None:
|
||||
cap = ua._number(initial["root_limit_usd"])
|
||||
axes = {"root": {"accounted_usd": own["accounted_usd"], "limit_usd": cap,
|
||||
axes = {"root": {"settled_usd": own["settled_usd"], "accounted_usd": own["accounted_usd"], "limit_usd": cap,
|
||||
"source": fields.get("root_limit_source") or initial.get("root_limit_source"),
|
||||
"revision": fields.get("root_limit_revision")},
|
||||
"group": {"accounted_usd": shared["accounted_usd"], "limit_usd": fields.get("billing_group_limit_usd"),
|
||||
"group": {"settled_usd": shared["settled_usd"], "accounted_usd": shared["accounted_usd"],
|
||||
"limit_usd": fields.get("billing_group_limit_usd"),
|
||||
"source": fields.get("billing_group_limit_source"),
|
||||
"revision": fields.get("billing_group_limit_revision")}}
|
||||
rooms = {key: axis["limit_usd"] - axis["accounted_usd"] for key, axis in axes.items() if axis["limit_usd"] is not None}
|
||||
rooms = {key: axis["limit_usd"] - axis["settled_usd"] for key, axis in axes.items() if axis["limit_usd"] is not None}
|
||||
binding = min(rooms, key=rooms.get) if rooms else None
|
||||
room = rooms[binding] if binding else None
|
||||
return {"accounted_usd": shared["accounted_usd"],
|
||||
"root_limit_usd": None if room is None else shared["accounted_usd"] + room,
|
||||
return {"settled_usd": shared["settled_usd"], "accounted_usd": shared["accounted_usd"],
|
||||
"root_limit_usd": None if room is None else shared["settled_usd"] + room,
|
||||
"remaining_known_usd": room, "root_axis": axes["root"], "group_axis": axes["group"],
|
||||
"accounting_basis": "billing_group", "binding_axis": binding,
|
||||
"integrity_degraded": usage_store.integrity_degraded(ua._drive_root(root)), "age_sec": 0.0}
|
||||
|
|
|
|||
|
|
@ -427,8 +427,8 @@ class Txn:
|
|||
def summary(self, root_task_id: Optional[str] = None, *, billing_group_id: Optional[str] = None) -> dict:
|
||||
return render_cash(self.totals(root_task_id, billing_group_id))
|
||||
|
||||
def exceeds_limit(self, limit, bound=None, *, root_task_id=None, billing_group_id=None, dispatch=False) -> bool:
|
||||
return exceeds_limit(self.totals(root_task_id, billing_group_id), limit, bound, dispatch=dispatch)
|
||||
def exceeds_limit(self, limit, *, root_task_id=None, billing_group_id=None) -> bool:
|
||||
return exceeds_limit(self.totals(root_task_id, billing_group_id), limit)
|
||||
|
||||
# -- writes
|
||||
def ack_dirty_owner(self, owner_id: str, revision: int) -> bool:
|
||||
|
|
|
|||
|
|
@ -14,4 +14,4 @@ Wake facts (context only):
|
|||
- reason: {reason}; last wake: {last_wake_ago}
|
||||
- recent facts: {events}
|
||||
- autonomy: {level} — {level_line}; unavailable tools: {withheld_tools}
|
||||
- allowance accounting (last 24 h): {spent_usd} / {daily_usd} USD; tasks running: {running}/{max_tasks}; next interval: {interval} s
|
||||
- allowance known spend (last 24 h): {spent_usd} / {daily_usd} USD; tasks running: {running}/{max_tasks}; next interval: {interval} s
|
||||
|
|
|
|||
|
|
@ -138,12 +138,13 @@ def _root_money_refusal(result_root: Any, task: Dict[str, Any], root_task_id: st
|
|||
if tree.get("integrity_degraded"):
|
||||
return {"ok": False, "error": "root_accounting_degraded",
|
||||
"action": "retry_or_cancel"}
|
||||
limit, accounted = tree.get("root_limit_usd"), tree.get("accounted_usd")
|
||||
# Known spend decides, the reservation's own rule (#1487); holds do not.
|
||||
limit, known = tree.get("root_limit_usd"), tree.get("settled_usd")
|
||||
if limit is not None:
|
||||
if accounted is None:
|
||||
if known is None:
|
||||
return {"ok": False, "error": "root_accounting_degraded",
|
||||
"action": "retry_or_cancel"}
|
||||
if float(accounted) >= float(limit) - 1e-9:
|
||||
if float(known) >= float(limit) - 1e-9:
|
||||
return {"ok": False, "error": "root_hard_cap_exhausted",
|
||||
"action": "increase_budget_then_resume"}
|
||||
return None
|
||||
|
|
|
|||
|
|
@ -1263,7 +1263,7 @@ def budget_line(force: bool = False) -> str:
|
|||
if total > 0
|
||||
else usage_writer_snapshot(DATA_DIR, allow_stale=True)
|
||||
)
|
||||
display_state["spent_usd"] = float(accounting.get("accounted_usd") or 0.0)
|
||||
display_state["spent_usd"] = float(accounting.get("settled_usd") or 0.0) # known; holds in the detail
|
||||
display_state["usage_accounting"] = accounting
|
||||
display_state["_budget_accounting_available"] = True
|
||||
except Exception:
|
||||
|
|
|
|||
|
|
@ -541,10 +541,14 @@ def _consciousness_admission_block(task: Dict[str, Any], window: Optional[Dict[s
|
|||
if not window["limit_usd"]:
|
||||
return ("consciousness_allowance_exhausted",
|
||||
"OUROBOROS_CONSCIOUSNESS_DAILY_USD=0: consciousness may not spend")
|
||||
at_least = " (at least)" if window["unknown_unmetered"] else ""
|
||||
at_least = " (at least)" if window.get("unknown_unmetered") else ""
|
||||
known = window.get("settled_usd")
|
||||
open_holds = 0.0 if known is None else max(0.0, float(window.get("accounted_usd") or 0.0) - float(known))
|
||||
holds = f" (plus ${open_holds:.2f} of open holds, not counted)" if open_holds > 0 else ""
|
||||
spent = "unknown" if known is None else f"${float(known):.2f}{at_least}"
|
||||
return ("consciousness_allowance_exhausted", (
|
||||
f"${window['accounted_usd']:.2f}{at_least} of ${window['limit_usd']:.2f} "
|
||||
f"spent in the last 24 h; resets at {window['resets_at'] or 'unknown'}"
|
||||
f"{spent} of ${window['limit_usd']:.2f} "
|
||||
f"known spend in the last 24 h{holds}; resets at {window['resets_at'] or 'unknown'}"
|
||||
))
|
||||
return None
|
||||
|
||||
|
|
|
|||
|
|
@ -1305,7 +1305,9 @@ def status_text(workers_dict: Dict[int, Any], pending_list: list,
|
|||
if TOTAL_BUDGET_LIMIT > 0
|
||||
else ledger_breakdown
|
||||
)
|
||||
spent = float(ledger_projection.get("accounted_usd") or 0.0)
|
||||
spent = float(ledger_projection.get("settled_usd") or 0.0) # known: what the limit decides on
|
||||
open_holds = float(ledger_projection.get("reserved_usd") or 0.0) + float(
|
||||
ledger_projection.get("unresolved_upper_bound_usd") or 0.0)
|
||||
pct = (spent / TOTAL_BUDGET_LIMIT * 100.0) if TOTAL_BUDGET_LIMIT > 0 else 0.0
|
||||
budget_remaining_usd = (
|
||||
float(ledger_projection.get("remaining_known_usd") or 0.0)
|
||||
|
|
@ -1319,7 +1321,7 @@ def status_text(workers_dict: Dict[int, Any], pending_list: list,
|
|||
except Exception:
|
||||
log.exception("Budget ledger unavailable while building status")
|
||||
accounting_available = False
|
||||
spent = pct = budget_remaining_usd = None
|
||||
spent = pct = budget_remaining_usd = open_holds = None
|
||||
spent_calls = prompt_tokens = completion_tokens = cached_tokens = None
|
||||
lines.append(f"budget_total: ${TOTAL_BUDGET_LIMIT:.0f}")
|
||||
if not accounting_available:
|
||||
|
|
@ -1336,6 +1338,9 @@ def status_text(workers_dict: Dict[int, Any], pending_list: list,
|
|||
lines.append(f"spent_usd: ${spent:.2f} ({pct:.1f}% of budget)")
|
||||
else:
|
||||
lines.append(f"spent_usd: ${spent:.2f}")
|
||||
if open_holds > 0:
|
||||
lines.append(f"open_holds_usd: ${open_holds:.2f} (reservations and unresolved upper bounds; "
|
||||
"not counted as spending)")
|
||||
lines.append(f"spent_calls: {spent_calls}")
|
||||
lines.append(
|
||||
f"prompt_tokens: {prompt_tokens}, completion_tokens: {completion_tokens}, "
|
||||
|
|
|
|||
|
|
@ -7,14 +7,18 @@ from ouroboros import task_pacing
|
|||
from ouroboros.contracts.task_contract import normalize_budget_profile
|
||||
from ouroboros.loop_round_limits import _RoundLimitContext
|
||||
|
||||
# An explicit experiment profile: its authored stop math is kept verbatim.
|
||||
EXPLICIT_50 = normalize_budget_profile({"cost_hard_stop_pct": 50})
|
||||
|
||||
|
||||
def _make_args(**overrides):
|
||||
"""Build default kwargs for _check_budget_limits.
|
||||
|
||||
``cost_ceiling`` defaults to the typed resolution with no root cap, so the
|
||||
legacy guard tests keep exercising the historical 50%-of-global semantics
|
||||
the runtime gets from ``task_pacing.resolve_cost_ceiling`` with an absent
|
||||
profile.
|
||||
``cost_ceiling`` defaults to the typed resolution of an EXPLICIT
|
||||
``cost_hard_stop_pct=50`` experiment profile with no root cap, so the guard
|
||||
tests keep exercising that authored 50%-of-global stop. An ordinary task has
|
||||
no such stop (owner 2026-10-07): pass the absent profile's resolution to
|
||||
test that.
|
||||
"""
|
||||
llm = MagicMock()
|
||||
llm.chat.return_value = (
|
||||
|
|
@ -40,7 +44,7 @@ def _make_args(**overrides):
|
|||
defaults.update(overrides)
|
||||
if "cost_ceiling" not in defaults:
|
||||
defaults["cost_ceiling"] = task_pacing.resolve_cost_ceiling(
|
||||
defaults["budget_remaining_usd"], normalize_budget_profile(None),
|
||||
defaults["budget_remaining_usd"], EXPLICIT_50,
|
||||
)
|
||||
budget_remaining_usd = defaults.pop("budget_remaining_usd")
|
||||
cost_ceiling = defaults.pop("cost_ceiling")
|
||||
|
|
|
|||
|
|
@ -0,0 +1,4 @@
|
|||
113
|
||||
#1569: the process_logging.py module row names its new second job, the launcher's byte-capped,
|
||||
storage-error-resilient copy of the server pipe into agent_stdout.log, and that the module imports only
|
||||
the standard library so the launcher can rely on it. The row's older description is kept whole.
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
6
|
||||
The usage_admission.py module-map row names review-wave admission instead of the retired wave fit (#1487).
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
89
|
||||
The Costs page sets known spend (confirmed + estimated, both shown) against the limit, with reserved and unresolved holds beside it and not counted as spending (#1487).
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
227
|
||||
Cold continuation restores the cost ceiling under its re-read authority (task_pacing.restore_cost_ceiling): explicit point kept, producer allowance at its value, removed default dropped, unreadable root kept as legacy_policy_unverified (#1128).
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
857
|
||||
Money paragraphs restate the known-spend rule in place of the old accounted/wave-fit wording (#1487) and the removal of the default in-task stop (#1128): planning threshold, wake allowance and producer ceiling, commit-gate and acceptance wave admission, forced rails, Resume refresh and pricing.
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
583
|
||||
#1569: the terminal-maintenance paragraph replaces its one-clause `duty_unresolved` mention with
|
||||
what the duty now does: Phase B stats the canonical result before the ownership read and keeps an
|
||||
absent owner's exact revision (so a later copyback or quarantine restore is projected without a
|
||||
receipt, at one stat per such owner per pass), and diagnostics publish a per-subject id->reason map
|
||||
only on change, with one representative traceback per new reason, an id whose outcome the pass did
|
||||
not prove keeping its last reason, and a failed publish or gateway close logged once per change.
|
||||
No older text describes either rule.
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
254
|
||||
The forced-rail and Resume-consumption paragraphs say an ineligible actor at a producer allowance or unverified inherited number ends on the host notice with no paid recap, and that Resume keeps the launch-time producer allowance even after its window frees more.
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
116
|
||||
The TOTAL_BUDGET, per-task cap and consciousness allowance rows say they are reached by known spend and that no default early stop precedes the per-task cap (#1487, #1128).
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
42
|
||||
Tree-spend pacing and the acceptance money rule name known spend, holds beside it; the wrap-up probe is an explicit profile's (#1487, #1128).
|
||||
|
|
@ -51,10 +51,34 @@ from tests.test_acceptance_delivery import (
|
|||
)
|
||||
|
||||
|
||||
def test_the_floor_priced_wave_that_does_not_fit_is_still_refused(monkeypatch, tmp_path):
|
||||
"""The floor is the admission line, not a bypass: when even one send per
|
||||
paid row does not fit the remaining root budget, the panel is refused as
|
||||
before — through the real gate — and no reviewer is called."""
|
||||
class _KnownPricedLLM(_EpisodeLLM):
|
||||
"""The delivery suite's scripted reviewer whose provider reports a FINAL price
|
||||
for each send: KNOWN spend, the only money a limit decides on (#1487). The
|
||||
suite's own replies carry no usable price, so they stay unresolved exposure."""
|
||||
|
||||
def chat(self, **kwargs):
|
||||
from ouroboros import usage_accounting as ua
|
||||
|
||||
request = ua.AttemptRequest(reservation_usd=self.reservation_usd, drive_root=self.drive_root,
|
||||
model="openai/fake-reviewer", provider="openrouter")
|
||||
return ua.execute_physical_attempt(request, lambda: self._reply(kwargs),
|
||||
extractor=lambda reply: (reply[1], self.reservation_usd, True))
|
||||
|
||||
|
||||
def _known_charge(scope, cost):
|
||||
from ouroboros import usage_accounting as ua
|
||||
|
||||
with ua.usage_scope(scope):
|
||||
held = ua.reserve_attempt(ua.AttemptRequest(model="openai/fake-reviewer", provider="openrouter",
|
||||
reservation_usd=cost))
|
||||
ua.mark_dispatched(held)
|
||||
ua.settle_attempt(held, {"prompt_tokens": 1, "completion_tokens": 1}, cost_usd=cost, cost_final=True)
|
||||
|
||||
|
||||
def test_a_wave_at_the_known_root_limit_is_still_refused(monkeypatch, tmp_path):
|
||||
"""The floor is the admission line, not a bypass: once the root's KNOWN spend
|
||||
has reached its limit (#1487: the reservation's own rule), the panel is refused
|
||||
as before — through the real gate — and no reviewer is called."""
|
||||
from ouroboros import loop as loop_mod
|
||||
from ouroboros import usage_accounting as ua
|
||||
|
||||
|
|
@ -64,7 +88,8 @@ def test_the_floor_priced_wave_that_does_not_fit_is_still_refused(monkeypatch, t
|
|||
llm = _EpisodeLLM(tmp_path, [{"content": json.dumps(_CLEAN_VERDICT)}] * 2, scoped=True)
|
||||
_real_panel(monkeypatch, llm, stub_gate=False)
|
||||
scope = _root_scope(tmp_path, root_limit_usd=0.5)
|
||||
_seed_root_ledger(scope, cost=0.45) # $0.05 left: less than one send (~$0.07)
|
||||
_seed_root_ledger(scope)
|
||||
_known_charge(scope, 0.5) # known spend is at the $0.50 limit
|
||||
ctx = _acceptance_ctx(tmp_path, evidence=dict(_ACCEPTANCE_PACKET), repo_dir=str(governance),
|
||||
workspace_root=str(workspace), workspace_mode="project")
|
||||
with ua.usage_scope(scope):
|
||||
|
|
@ -193,11 +218,11 @@ def test_projecting_an_exhausted_cap_emits_no_review_cycles_exhausted_event(monk
|
|||
# Required+Blocking count cap is exercised by the `fixed`/`adaptive` policies alone.
|
||||
def test_the_per_send_wallet_fence_still_binds_after_an_admitted_dispatch(monkeypatch, tmp_path):
|
||||
"""The per-send wallet binding at dispatch is what actually protects money,
|
||||
proven with PRICED sends: the wave gate admits on ONE send per paid row, so
|
||||
a nearly spent wallet admits the panel; the native episode's first send is
|
||||
reserved and settled, its second send is refused by the ledger
|
||||
(`budget_exhausted`), and the accounted total never exceeds the root
|
||||
limit. The coarse admission is a filter, never the fence."""
|
||||
proven with PRICED sends: the wave gate admits while known spend is below
|
||||
the limit; the native episode's first send is reserved and settled at a
|
||||
final price that brings known spend to the root limit, so its second send
|
||||
is refused by the ledger (`budget_exhausted`). The coarse admission is a
|
||||
filter, never the fence."""
|
||||
from ouroboros import loop as loop_mod
|
||||
from ouroboros import usage_accounting as ua
|
||||
|
||||
|
|
@ -205,13 +230,13 @@ def test_the_per_send_wallet_fence_still_binds_after_an_admitted_dispatch(monkey
|
|||
_offline_env(monkeypatch, _ROW_NATIVE)
|
||||
_priced_offline_model(monkeypatch)
|
||||
governance, workspace = _roots(tmp_path)
|
||||
llm = _EpisodeLLM(
|
||||
llm = _KnownPricedLLM(
|
||||
tmp_path, [], scoped=True, reservation_usd=0.06,
|
||||
native_script=[{"tool_calls": [_tool_call("read_file", {"path": "greeting.txt"})]},
|
||||
{"content": json.dumps(_CLEAN_VERDICT)}],
|
||||
)
|
||||
_real_panel(monkeypatch, llm, stub_gate=False)
|
||||
limit = 0.1 # one priced send (0.06) fits; the second (0.12 cumulative) does not
|
||||
limit = 0.06 # the first send's known $0.06 reaches the limit; the second is refused
|
||||
scope = _root_scope(tmp_path, root_limit_usd=limit)
|
||||
_seed_root_ledger(scope)
|
||||
ctx = _acceptance_ctx(tmp_path, evidence=dict(_ACCEPTANCE_PACKET), repo_dir=str(governance),
|
||||
|
|
@ -224,7 +249,8 @@ def test_the_per_send_wallet_fence_still_binds_after_an_admitted_dispatch(monkey
|
|||
assert result.aggregate_signal == "DEGRADED"
|
||||
projection = ua.usage_projection(tmp_path, root_task_id="root-delivery")
|
||||
spent = float(projection["limit_usd"]) - float(projection["remaining_known_usd"])
|
||||
assert projection["limit_usd"] == limit and 0.06 <= spent <= limit
|
||||
assert projection["limit_usd"] == limit and spent == pytest.approx(0.06)
|
||||
assert projection["settled_usd"] == pytest.approx(0.06)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
|
|||
|
|
@ -603,6 +603,8 @@ def test_real_budget_pause_cap_amendment_preserves_all_controls_and_ledger(tmp_p
|
|||
settle_attempt(spent, cost_usd=2, cost_final=True)
|
||||
hold = reserve_attempt(AttemptRequest(model="synthetic", provider="test", drive_root=f.root,
|
||||
task_id=f.accounting, root_task_id=f.accounting, reservation_usd=2, global_limit_usd=100, root_limit_usd=4))
|
||||
mark_dispatched(hold)
|
||||
settle_attempt(hold, cost_usd=2, cost_final=True) # known spend reaches the $4 cap (#1487)
|
||||
with pytest.raises(BudgetExceeded) as exhausted:
|
||||
reserve_attempt(AttemptRequest(model="synthetic", provider="test", drive_root=f.root,
|
||||
task_id=f.accounting, root_task_id=f.accounting, reservation_usd=0.1, global_limit_usd=100, root_limit_usd=4))
|
||||
|
|
|
|||
|
|
@ -398,33 +398,53 @@ def test_explicit_receipt_requires_id_routed_chat_and_exact_bytes(late, tmp_path
|
|||
assert not late.calls and not load_task_result(f.root, f.accounting).get('task_acceptance_review_accounting')
|
||||
|
||||
|
||||
@pytest.mark.parametrize('cap', ['unlimited', 'unknown', 'prior_hold', 'prior_spend', 'live_global'])
|
||||
@pytest.mark.parametrize('cap', ['unlimited', 'unknown', 'prior_hold', 'prior_spend', 'spent_at_cap',
|
||||
'spent_over_cap', 'live_global', 'live_global_spent'])
|
||||
def test_original_cap_and_live_money_fences_use_original_wallet(late, tmp_path, monkeypatch, cap):
|
||||
import time
|
||||
from ouroboros import pricing
|
||||
from ouroboros.usage_accounting import AttemptRequest, reserve_attempt
|
||||
from ouroboros import pricing, review_operation
|
||||
from ouroboros.usage_accounting import AttemptRequest, execute_physical_attempt, reserve_attempt, usage_projection
|
||||
# The wave fence needs a priced reviewer seat: an unpriced one adds nothing and fits. Pin the
|
||||
# catalog row; a live fetch that times out, or a test's leftover 30 s retry_after, leaves none.
|
||||
monkeypatch.setitem(pricing._cached_pricing, 'openrouter', {'openai/gpt-4.1-nano': (0.1, 0.025, None, 0.4)})
|
||||
monkeypatch.setitem(pricing._pricing_fetched_at, 'openrouter', time.time())
|
||||
f = delivered(tmp_path, monkeypatch, retry=True, cap='unlimited' if cap in {'unlimited', 'live_global'} else 'unknown' if cap == 'unknown' else 'finite')
|
||||
live = cap.startswith('live_global')
|
||||
f = delivered(tmp_path, monkeypatch, retry=True,
|
||||
cap='unlimited' if cap == 'unlimited' or live else 'unknown' if cap == 'unknown' else 'finite')
|
||||
monkeypatch.setenv('OUROBOROS_PER_TASK_COST_USD', '0.00001')
|
||||
|
||||
def paid(usd, root):
|
||||
# A settled actual price is KNOWN spend: admitted while known is below the
|
||||
# cap, and recorded whole even where it lands beyond it (#1487).
|
||||
execute_physical_attempt(AttemptRequest(model='openai/gpt-4.1-nano', provider='openai', drive_root=f.root,
|
||||
task_id=root, root_task_id=root, root_limit_usd=4, reservation_usd=usd),
|
||||
lambda: 'original paid work', extractor=lambda _response: ({}, usd, True))
|
||||
if cap == 'prior_hold':
|
||||
# An open hold is disclosed exposure, never spending: it refuses nothing.
|
||||
reserve_attempt(AttemptRequest(model='openai/gpt-4.1-nano', provider='openai', drive_root=f.root,
|
||||
task_id=f.accounting, root_task_id=f.accounting, root_limit_usd=4, reservation_usd=3.9999))
|
||||
if cap == 'prior_spend':
|
||||
from ouroboros.usage_accounting import execute_physical_attempt
|
||||
execute_physical_attempt(AttemptRequest(model='openai/gpt-4.1-nano', provider='openai', drive_root=f.root,
|
||||
task_id=f.accounting, root_task_id=f.accounting, root_limit_usd=4, reservation_usd=3.9999),
|
||||
lambda: 'original paid work', extractor=lambda _response: ({}, 3.9999, True))
|
||||
if cap == 'live_global':
|
||||
if cap in {'prior_spend', 'spent_at_cap', 'spent_over_cap'}:
|
||||
paid({'prior_spend': 3.9999, 'spent_at_cap': 4, 'spent_over_cap': 4.5}[cap], f.accounting)
|
||||
if cap == 'live_global_spent':
|
||||
paid(0.00001, 'unrelated-root') # another root's known spend reaches the live global cap exactly
|
||||
if live:
|
||||
monkeypatch.setattr('ouroboros.settings_setup_contract.resolve_total_budget_usd', lambda: 0.00001)
|
||||
ctx = _caller(f)
|
||||
gate = threading.Event()
|
||||
late.gates.append(gate)
|
||||
result = _request(f, ctx, _source(ctx))
|
||||
if cap == 'unlimited':
|
||||
if cap in {'unlimited', 'prior_hold', 'prior_spend', 'live_global'}:
|
||||
assert result['status'] in {'pending', 'announced', 'published', 'settled'}, result
|
||||
# The gate holds every settlement: all three seats pass reserve and dispatch
|
||||
# on the same known spend, whatever the prior and sibling holds add up to.
|
||||
until(lambda: len(late.calls) == 3)
|
||||
assert all(scope.root_limit_usd is None and scope.root_limit_source for scope, _ in late.calls)
|
||||
gate.set()
|
||||
until(lambda: not review_operation._LIVE)
|
||||
assert all(scope.root_limit_usd == (4 if cap.startswith('prior') else None) and scope.root_limit_source
|
||||
for scope, _ in late.calls)
|
||||
if cap == 'prior_spend':
|
||||
# Seats sent in parallel on known $3.9999 overshoot the $4 cap: recorded, not prevented.
|
||||
assert usage_projection(f.root, root_task_id=f.accounting)['settled_usd'] == pytest.approx(4.0299)
|
||||
else:
|
||||
assert result['status'] == 'owed', result
|
||||
assert result['reason'] == ('original_root_cap_unknown' if cap == 'unknown' else 'review_wave_budget_insufficient')
|
||||
|
|
|
|||
|
|
@ -62,14 +62,11 @@ def test_retired_local_answer_owner_releases_same_continue_without_settling_mone
|
|||
assert len(workers.PENDING) == 1
|
||||
assert ledger_rows(tmp_path) == before, "writer classification must not mutate money"
|
||||
projection = ua.usage_projection(tmp_path, billing_group_id="pred-1")
|
||||
# Releasing a local-writer hold neither settles nor forgets its liability: the
|
||||
# bound stays disclosed exposure, and (#1487) it is never counted as spending.
|
||||
assert projection["unresolved_upper_bound_usd"] == 0.5
|
||||
# Releasing a local-writer hold does not restore its monetary room.
|
||||
sent = []
|
||||
with ua.usage_scope(ua.UsageScope(drive_root=tmp_path, task_id=successor, root_task_id=successor)):
|
||||
with pytest.raises(ua.BudgetExceeded):
|
||||
ua.execute_physical_attempt(ua.AttemptRequest(
|
||||
model="m", provider="test", reservation_usd=20.0), lambda: sent.append(True))
|
||||
assert sent == [] and ledger_rows(tmp_path) == before
|
||||
assert projection["settled_usd"] == 0.0 and projection["accounted_usd"] == 0.5
|
||||
assert ledger_rows(tmp_path) == before
|
||||
# An actual late receipt still settles the same liability.
|
||||
reservation = ua.AttemptReservation(final["attempt_id"], tmp_path, "m", "test", 0.5)
|
||||
ua.settle_attempt(reservation, {}, cost_usd=0.7, cost_final=True)
|
||||
|
|
@ -401,7 +398,7 @@ root = Path({str(tmp_path)!r})
|
|||
ua._reservation_cost = lambda request: 0.25
|
||||
scope = ua.UsageScope(drive_root=root, task_id="successor", root_task_id="successor",
|
||||
root_limit_usd=10, global_limit_usd=100, billing_group_id="original",
|
||||
billing_group_limit_usd=0.1, billing_group_limit_source="initial_task_admission")
|
||||
billing_group_limit_usd=0.0, billing_group_limit_source="initial_task_admission")
|
||||
with ua.usage_scope(scope):
|
||||
verdict = review_wave_budget_gate(SimpleNamespace(task_id="successor", pending_events=[]),
|
||||
surface="test", models=["test/model"], prompt_chars=100)
|
||||
|
|
|
|||
|
|
@ -112,7 +112,8 @@ def test_pre_send_wait_revalidates_late_charge_on_each_axis(root, monkeypatch, a
|
|||
prepared.append(reservation)
|
||||
# The same writer receives the original/sibling's delayed receipt after reservation.
|
||||
with ua.usage_scope(None):
|
||||
ua.settle_attempt(late, cost_usd=1.5, cost_final=True)
|
||||
# Its known price reaches the $2 cap exactly (#1487: equality refuses).
|
||||
ua.settle_attempt(late, cost_usd=2.0, cost_final=True)
|
||||
lock_owner.enter_context(held_lock(root))
|
||||
req = request(root, root_task_id="successor", global_limit_usd=2 if axis == "global" else 100)
|
||||
async def send():
|
||||
|
|
|
|||
|
|
@ -35,22 +35,22 @@ def test_owner_cap_is_total_group_allowance_with_successor_and_late_charge(late,
|
|||
assert ledger_rows(f.root) == before
|
||||
assert load_task_result(f.root, f.accounting)["billing_group"] == binding
|
||||
with ua.usage_scope(_scope(f.root, "successor", "successor", group=f.accounting, group_limit=4)):
|
||||
paid = ua.reserve_attempt(ua.AttemptRequest(model="m", provider="test", reservation_usd=5.8))
|
||||
paid = ua.reserve_attempt(ua.AttemptRequest(model="m", provider="test", reservation_usd=6.0))
|
||||
assert paid.scope.billing_group_limit_usd == 9
|
||||
assert paid.scope.billing_group_limit_source == "owner_amendment"
|
||||
ua.mark_dispatched(paid)
|
||||
ua.settle_attempt(paid, {}, cost_usd=5.8, cost_final=True)
|
||||
ua.settle_attempt(paid, {}, cost_usd=6.0, cost_final=True) # known spend reaches the amended $9
|
||||
with pytest.raises(ua.BudgetExceeded):
|
||||
_spend(f.root, _scope(f.root, f.accounting, f.accounting, group=f.accounting, group_limit=4), .3)
|
||||
projection = ua.usage_projection(f.root, billing_group_id=f.accounting)
|
||||
assert projection["accounted_usd"] == pytest.approx(8.8) and projection["limit_usd"] == 9
|
||||
assert projection["accounted_usd"] == pytest.approx(9.0) and projection["limit_usd"] == 9
|
||||
from ouroboros.usage_admission import task_money_snapshot
|
||||
snapshot = task_money_snapshot(f.root, {"id": f.accounting}, f.accounting)
|
||||
assert snapshot["remaining_known_usd"] == pytest.approx(.2)
|
||||
assert snapshot["remaining_known_usd"] == pytest.approx(0.0)
|
||||
assert snapshot["group_axis"]["source"] == "owner_amendment"
|
||||
with ua.usage_scope(_scope(f.root, f.accounting, f.accounting, group=f.accounting, group_limit=4)):
|
||||
from ouroboros.loop_budget import _loop_tree_accounting
|
||||
assert _loop_tree_accounting(refresh=True, strict=True)["accounted_usd"] == pytest.approx(8.8)
|
||||
assert _loop_tree_accounting(refresh=True, strict=True)["settled_usd"] == pytest.approx(9.0)
|
||||
monkeypatch.setattr(ua, "_reservation_cost", lambda _request: .1)
|
||||
requested = _request(f, ctx, _source(ctx, text="Now review the historical answer"), action="review")
|
||||
assert requested["reason"] == "review_wave_budget_insufficient", requested
|
||||
|
|
@ -60,7 +60,7 @@ def test_owner_cap_is_total_group_allowance_with_successor_and_late_charge(late,
|
|||
def test_successor_cap_amendment_cannot_widen_original_group(late, tmp_path, monkeypatch): # noqa: F811
|
||||
f = delivered(tmp_path, monkeypatch)
|
||||
bind_group(f, "original")
|
||||
_spend(f.root, _scope(f.root, "original", "original", group="original", group_limit=4), 3.8)
|
||||
_spend(f.root, _scope(f.root, "original", "original", group="original", group_limit=4), 4.0)
|
||||
ctx = _caller(f)
|
||||
assert _request(f, ctx, _source(ctx), action="amend_cap", new_original_root_cap_usd=9)["status"] == "amended"
|
||||
with pytest.raises(ua.BudgetExceeded):
|
||||
|
|
|
|||
|
|
@ -54,8 +54,9 @@ def test_actual_host_role_preserves_configured_lifetime_cap(data_root, monkeypat
|
|||
reservation_upper_bound_usd="1.0", pricing_known=True)
|
||||
write_compacted_journal(data_root, [{**aggregate, "attempt_id": "old-first", "model": "z", "root_limit_usd": "2.0"},
|
||||
{**aggregate, "attempt_id": "old-later", "model": "a", "root_limit_usd": "100.0"}])
|
||||
for skill in ("one", "two"):
|
||||
result = _coordinator_reservation(ctx, monkeypatch, skill=skill, amount=.4)
|
||||
# Known spend: the open block's $1.00, then $0.40 and $0.60 reach the configured $2 cap.
|
||||
for skill, amount in (("one", .4), ("two", .6)):
|
||||
result = _coordinator_reservation(ctx, monkeypatch, skill=skill, amount=amount)
|
||||
assert "attempt" in result
|
||||
scope = result["scope"]
|
||||
assert scope.non_task_operation and scope.root_limit_usd == 2.0
|
||||
|
|
@ -63,13 +64,16 @@ def test_actual_host_role_preserves_configured_lifetime_cap(data_root, monkeypat
|
|||
assert scope.root_task_id == ctx.task_id
|
||||
exhausted = _coordinator_reservation(ctx, monkeypatch, skill="three", amount=.3)
|
||||
assert "root model budget exhausted" in exhausted["refusal"]
|
||||
assert ua.usage_projection(data_root, root_task_id=ctx.task_id)["accounted_usd"] == pytest.approx(1.8)
|
||||
assert ua.usage_projection(data_root, root_task_id=ctx.task_id)["accounted_usd"] == pytest.approx(2.0)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("ctx_type", [_ApiReviewCtx, _MarketplaceReviewCtx])
|
||||
def test_host_role_keeps_global_exhaustion(data_root, monkeypatch, ctx_type):
|
||||
ctx = ctx_type(data_root, data_root.parent / "repo")
|
||||
with ua.usage_scope(ua.UsageScope(drive_root=data_root, non_task_operation=True, global_limit_usd=.01)):
|
||||
spent = ua.reserve_attempt(ua.AttemptRequest(model="fixture", provider="fixture", reservation_usd=.01))
|
||||
ua.mark_dispatched(spent)
|
||||
ua.settle_attempt(spent, {}, cost_usd=.01, cost_final=True) # known spend reaches the $0.01 wallet
|
||||
result = _coordinator_reservation(ctx, monkeypatch)
|
||||
assert result["scope"].global_limit_usd == .01
|
||||
assert "global model budget exhausted" in result["refusal"]
|
||||
|
|
@ -82,7 +86,7 @@ def test_host_hint_never_overrides_inherited_continue_group(data_root, monkeypat
|
|||
parent_task_id="successor", billing_group_id="original", billing_group_limit_usd=1.0,
|
||||
billing_group_limit_source="ledger_first_row", root_limit_usd=50.0)
|
||||
with ua.usage_scope(inherited):
|
||||
first = _coordinator_reservation(ctx, monkeypatch, amount=.75)
|
||||
first = _coordinator_reservation(ctx, monkeypatch, amount=1.0) # known spend reaches the group's $1
|
||||
second = _coordinator_reservation(ctx, monkeypatch, amount=.3)
|
||||
assert "attempt" in first
|
||||
scope = first["scope"]
|
||||
|
|
|
|||
|
|
@ -53,20 +53,24 @@ T = dict(group="P", group_limit=20.0) # the successor's helper
|
|||
def test_every_member_sees_every_other_including_the_original_roots_late_review(data_root):
|
||||
_spend(data_root, _scope(data_root, "P", "P", **P), 8.0)
|
||||
_spend(data_root, _scope(data_root, "S", "S", **S), 10.0)
|
||||
# P's late review ($4) would reach $22: refused under P's OWN scope.
|
||||
# Known $18 of $20: P's late review is admitted although its own $4 bound would
|
||||
# pass the cap (#1487: a bound is exposure, not spending). It settles at $2.
|
||||
_spend(data_root, _scope(data_root, "P-review", "P", category="review", **P), 2.0, bound=4.0)
|
||||
# Known spend reached the cap: P's own later work, S and S's helper T are all refused.
|
||||
with pytest.raises(ua.BudgetExceeded, match="whole-work budget exhausted for group P"):
|
||||
_spend(data_root, _scope(data_root, "P-review", "P", category="review", **P), 4.0)
|
||||
# S's helper T cannot spend the same remainder either.
|
||||
_spend(data_root, _scope(data_root, "T", "S", parent_task_id="S", **T), 1.5)
|
||||
_spend(data_root, _scope(data_root, "P-review", "P", category="review", **P), 0.5)
|
||||
with pytest.raises(ua.BudgetExceeded, match=r"known=\$20\.000000"):
|
||||
_spend(data_root, _scope(data_root, "T", "S", parent_task_id="S", **T), 0.5)
|
||||
with pytest.raises(ua.BudgetExceeded):
|
||||
_spend(data_root, _scope(data_root, "S", "S", **S), 1.0)
|
||||
projection = ua.usage_projection(data_root, billing_group_id="P")
|
||||
assert projection["accounted_usd"] == pytest.approx(19.5) and projection["limit_usd"] == 20.0
|
||||
assert projection["settled_usd"] == pytest.approx(20.0) and projection["limit_usd"] == 20.0
|
||||
assert projection["remaining_known_usd"] == 0.0
|
||||
# Genuine ids stay genuine: each row names its own task and root.
|
||||
rows = ledger_rows(data_root) # one current row per admitted attempt; refusals left none
|
||||
attributed = {(r["task_id"], r["root_task_id"], r["billing_group_id"]) for r in rows}
|
||||
assert attributed == {("P", "P", "P"), ("S", "S", "P"), ("T", "S", "P")}
|
||||
assert ua.usage_projection(data_root, root_task_id="S")["accounted_usd"] == pytest.approx(11.5)
|
||||
assert attributed == {("P", "P", "P"), ("S", "S", "P"), ("P-review", "P", "P")}
|
||||
assert ua.usage_projection(data_root, root_task_id="S")["accounted_usd"] == pytest.approx(10.0)
|
||||
|
||||
|
||||
def test_legacy_rows_of_the_original_root_join_the_group_without_a_rewrite(data_root):
|
||||
|
|
@ -76,9 +80,9 @@ def test_legacy_rows_of_the_original_root_join_the_group_without_a_rewrite(data_
|
|||
# Journaled before the group field existed: no ``billing_group_id`` anywhere;
|
||||
# the store's one-time import carries the row over as it is.
|
||||
write_journal(data_root, [{**legacy, "state": "reserved"}, {**legacy, "state": "dispatched"},
|
||||
{**legacy, "state": "settled", "cost_usd": 15.0, "cost_final": True}])
|
||||
with pytest.raises(ua.BudgetExceeded):
|
||||
_spend(data_root, _scope(data_root, "S", "S", **S), 6.0)
|
||||
{**legacy, "state": "settled", "cost_usd": 20.0, "cost_final": True}])
|
||||
with pytest.raises(ua.BudgetExceeded): # P's known $20 legacy spend binds its successor
|
||||
_spend(data_root, _scope(data_root, "S", "S", **S), 0.5)
|
||||
[imported] = [row for row in ledger_rows(data_root) if row["attempt_id"] == "legacy-1"]
|
||||
assert "billing_group_id" not in imported, "history is never rewritten"
|
||||
from ouroboros.usage_admission import original_group_limit
|
||||
|
|
@ -111,21 +115,21 @@ def test_an_imported_aggregate_keeps_the_group_and_the_pacing_cache_reads_it(dat
|
|||
# What the retired compaction folded 16 of P's and 10 of S's settled attempts into.
|
||||
write_compacted_journal(data_root, [
|
||||
{**common, "attempt_id": "fold-P", "task_id": "P", "root_task_id": "P", "root_limit_usd": 20.0,
|
||||
"folded_attempt_count": 16, "cost_usd": 8.0, "reservation_upper_bound_usd": 8.0},
|
||||
"folded_attempt_count": 16, "cost_usd": 12.0, "reservation_upper_bound_usd": 12.0},
|
||||
{**common, "attempt_id": "fold-S", "task_id": "S", "root_task_id": "S",
|
||||
"folded_attempt_count": 10, "cost_usd": 5.0, "reservation_upper_bound_usd": 5.0},
|
||||
"folded_attempt_count": 10, "cost_usd": 8.0, "reservation_upper_bound_usd": 8.0},
|
||||
])
|
||||
assert original_group_limit(data_root, "P")["limit_usd"] == 20.0
|
||||
aggregates = [row for row in ledger_rows(data_root) if row.get("kind") == "usage_baseline_group"]
|
||||
assert [(row["billing_group_id"], row["billing_group_limit_usd"]) for row in aggregates] == [("P", 20.0)] * 2
|
||||
projection = ua.usage_projection(data_root, billing_group_id="P")
|
||||
assert projection["accounted_usd"] == pytest.approx(13.0)
|
||||
assert projection["accounted_usd"] == pytest.approx(20.0)
|
||||
assert projection["attempt_counts"] == {"settled": 26} # counts weighted by the folded attempts
|
||||
with ua.usage_scope(_scope(data_root, "S", "S", **S)):
|
||||
tree = _loop_tree_accounting(refresh=True, max_age_sec=0.0, strict=True)
|
||||
assert tree["accounted_usd"] == pytest.approx(13.0) and tree["root_limit_usd"] == 20.0
|
||||
assert tree["settled_usd"] == pytest.approx(20.0) and tree["root_limit_usd"] == 20.0
|
||||
with pytest.raises(ua.BudgetExceeded):
|
||||
_spend(data_root, _scope(data_root, "S", "S", **S), 8.0)
|
||||
_spend(data_root, _scope(data_root, "S", "S", **S), 0.5)
|
||||
|
||||
|
||||
def test_the_review_wave_binds_on_the_group_when_it_is_tighter(data_root):
|
||||
|
|
@ -178,9 +182,11 @@ def test_the_continue_admission_carries_the_original_cap_and_the_resume_grant_re
|
|||
assert task_accounting_key(tmp_path, row, row["id"]) == "group:pred-1"
|
||||
|
||||
|
||||
def test_concurrent_siblings_cannot_each_spend_the_same_remainder(data_root):
|
||||
"""S and its helper T race for the group's last $8 with $6 bounds each: the
|
||||
one locked read/check/append admits exactly one, whichever wins."""
|
||||
def test_concurrent_siblings_are_admitted_on_known_spend_and_the_exposure_is_disclosed(data_root):
|
||||
"""Owner Q4-A (#1487): S and its helper T race for the group's last $8 with $6
|
||||
bounds each. Known spend ($12) is below the cap, so BOTH are admitted: concurrent
|
||||
overshoot is accepted, never prevented by worst-case holds, and the $24 exposure
|
||||
is disclosed beside the known $12."""
|
||||
import threading
|
||||
|
||||
_spend(data_root, _scope(data_root, "P", "P", **P), 12.0)
|
||||
|
|
@ -204,8 +210,11 @@ def test_concurrent_siblings_cannot_each_spend_the_same_remainder(data_root):
|
|||
thread.start()
|
||||
for thread in threads:
|
||||
thread.join(timeout=60)
|
||||
assert sorted(outcomes.values()) == ["admitted", "refused"], outcomes
|
||||
assert ua.usage_projection(data_root, billing_group_id="P")["accounted_usd"] <= 20.0 + 1e-9
|
||||
assert sorted(outcomes.values()) == ["admitted", "admitted"], outcomes
|
||||
projection = ua.usage_projection(data_root, billing_group_id="P")
|
||||
assert projection["settled_usd"] == pytest.approx(12.0)
|
||||
assert projection["accounted_usd"] == pytest.approx(24.0)
|
||||
assert projection["accounted_usd"] > projection["limit_usd"] == 20.0
|
||||
|
||||
|
||||
def test_late_session_uses_start_custody_after_root_loss_not_callers_group(data_root):
|
||||
|
|
@ -220,19 +229,20 @@ def test_late_session_uses_start_custody_after_root_loss_not_callers_group(data_
|
|||
task_result_path(data_root, "S").unlink()
|
||||
with ua.usage_scope(_scope(data_root, "unrelated", "unrelated", group="unrelated", group_limit=999.0)):
|
||||
ua.record_subscription_session("late-session", drive_root=data_root, route="test", model="m",
|
||||
task_id="S", root_task_id="S", spend_usd=19.5)
|
||||
assert ua.usage_projection(data_root, billing_group_id="P")["accounted_usd"] == pytest.approx(19.5)
|
||||
assert ua.usage_projection(data_root, root_task_id="S")["accounted_usd"] == pytest.approx(19.5)
|
||||
task_id="S", root_task_id="S", spend_usd=20.0)
|
||||
assert ua.usage_projection(data_root, billing_group_id="P")["accounted_usd"] == pytest.approx(20.0)
|
||||
assert ua.usage_projection(data_root, root_task_id="S")["accounted_usd"] == pytest.approx(20.0)
|
||||
assert ua.usage_projection(data_root, billing_group_id="unrelated")["accounted_usd"] == 0.0
|
||||
with pytest.raises(ua.BudgetExceeded):
|
||||
_spend(data_root, _scope(data_root, "P", "P", **P), 1.0)
|
||||
|
||||
|
||||
def test_sibling_reservations_share_one_atomic_remainder(data_root):
|
||||
def test_sibling_reservations_share_one_atomic_known_spend(data_root):
|
||||
"""Known spend exactly at the cap: the one locked check refuses BOTH racers."""
|
||||
import threading
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
_spend(data_root, _scope(data_root, "P", "P", **P), 19.0)
|
||||
_spend(data_root, _scope(data_root, "P", "P", **P), 20.0)
|
||||
ready = threading.Barrier(2)
|
||||
def claim(tid):
|
||||
with ua.usage_scope(_scope(data_root, tid, tid, **S)):
|
||||
|
|
@ -243,16 +253,18 @@ def test_sibling_reservations_share_one_atomic_remainder(data_root):
|
|||
except ua.BudgetExceeded:
|
||||
return "refused"
|
||||
with ThreadPoolExecutor(max_workers=2) as pool:
|
||||
assert sorted(pool.map(claim, ["S", "T"])) == ["refused", "reserved"]
|
||||
assert ua.usage_projection(data_root, billing_group_id="P")["accounted_usd"] == pytest.approx(19.75)
|
||||
assert sorted(pool.map(claim, ["S", "T"])) == ["refused", "refused"]
|
||||
assert ua.usage_projection(data_root, billing_group_id="P")["accounted_usd"] == pytest.approx(20.0)
|
||||
|
||||
|
||||
def test_root_cap_and_group_cap_are_both_required(data_root):
|
||||
with pytest.raises(ua.BudgetExceeded):
|
||||
_spend(data_root, _scope(data_root, "S", "S", group="P", group_limit=20.0, root_limit=2.0), 3.0)
|
||||
_spend(data_root, _scope(data_root, "P", "P", **P), 19.0)
|
||||
with pytest.raises(ua.BudgetExceeded):
|
||||
_spend(data_root, _scope(data_root, "S", "S", group="P", group_limit=20.0, root_limit=5.0), 2.0)
|
||||
tight = _scope(data_root, "S", "S", group="P", group_limit=20.0, root_limit=2.0)
|
||||
_spend(data_root, tight, 2.0) # S's own known spend reaches its root cap
|
||||
with pytest.raises(ua.BudgetExceeded, match="root model budget exhausted for S"):
|
||||
_spend(data_root, tight, 0.01) # although the group still has $18
|
||||
_spend(data_root, _scope(data_root, "P", "P", **P), 18.0) # the group's known spend reaches $20
|
||||
with pytest.raises(ua.BudgetExceeded, match="whole-work budget exhausted for group P"):
|
||||
_spend(data_root, _scope(data_root, "S", "S", group="P", group_limit=20.0, root_limit=5.0), 0.01)
|
||||
|
||||
|
||||
def test_explicit_other_root_cannot_inherit_the_callers_billing_group(data_root):
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ from unittest.mock import MagicMock, patch
|
|||
from ouroboros import task_pacing
|
||||
from ouroboros.contracts.task_contract import normalize_budget_profile
|
||||
from ouroboros.loop_budget import _check_budget_limits
|
||||
from tests._budget_limits_helpers import _make_args
|
||||
from tests._budget_limits_helpers import EXPLICIT_50, _make_args
|
||||
|
||||
|
||||
# --- The retired per-task soft reminder (v6.91) ---
|
||||
|
|
@ -46,7 +46,23 @@ class TestPerTaskSoftNoteRetired:
|
|||
# --- Global budget guard ---
|
||||
|
||||
class TestGlobalBudgetGuard:
|
||||
"""Existing global budget percentage checks."""
|
||||
"""Global exhaustion, and an EXPLICIT ``cost_hard_stop_pct=50`` profile's stop
|
||||
(``_make_args``' default ceiling); ordinary tasks have no percentage stop."""
|
||||
|
||||
def test_an_ordinary_task_is_not_stopped_at_half_the_wallet(self, tmp_path):
|
||||
"""Owner 2026-10-07 (#1128): no default early stop. 4.5 of 8 remaining is no
|
||||
reason to stop an ordinary task; only real limits on known spend are."""
|
||||
args = _make_args(
|
||||
budget_remaining_usd=8.0, accumulated_usage={"cost": 7.9}, drive_logs=tmp_path,
|
||||
cost_ceiling=task_pacing.resolve_cost_ceiling(8.0, normalize_budget_profile(None)),
|
||||
)
|
||||
assert args["cost_ceiling"].state == task_pacing.COST_CEILING_DISABLED
|
||||
with (
|
||||
patch.dict(os.environ, {"OUROBOROS_PER_TASK_COST_USD": "10.0"}),
|
||||
patch("ouroboros.loop.call_llm_with_retry") as model_call,
|
||||
):
|
||||
assert _check_budget_limits(**args) is None
|
||||
model_call.assert_not_called()
|
||||
|
||||
def test_no_global_budget_and_no_root_cap_is_silent(self, tmp_path):
|
||||
"""Neither axis finite (GAIA shape, no per-task cap) → the whole cost
|
||||
|
|
@ -139,9 +155,19 @@ class TestCostCeilingResolution:
|
|||
"""task_pacing.resolve_cost_ceiling: typed states, global pct component,
|
||||
root-cap-minus-margin component."""
|
||||
|
||||
def test_absent_profile_means_historical_50pct(self):
|
||||
def test_absent_profile_has_no_default_stop(self):
|
||||
"""Owner 2026-10-07 (#1128): no half-wallet share and no cap-minus-margin for
|
||||
an ordinary task — not at a large wallet, a small one, or under any cap."""
|
||||
profile = normalize_budget_profile(None)
|
||||
ceiling = task_pacing.resolve_cost_ceiling(10.0, profile)
|
||||
for wallet, cap in ((10.0, None), (480.0, 400.0), (1900.0, 100.0), (None, 50.0), (100.0, 0.5)):
|
||||
ceiling = task_pacing.resolve_cost_ceiling(wallet, profile, root_cap_usd=cap)
|
||||
assert ceiling.state == task_pacing.COST_CEILING_DISABLED, (wallet, cap)
|
||||
assert ceiling.ceiling_usd is None and ceiling.planning_margin_usd is None
|
||||
assert ceiling.root_cap_usd == cap
|
||||
assert ceiling.basis == task_pacing.COST_BASIS_NO_DEFAULT_STOP
|
||||
|
||||
def test_explicit_50_keeps_the_authored_half_of_the_wallet(self):
|
||||
ceiling = task_pacing.resolve_cost_ceiling(10.0, EXPLICIT_50)
|
||||
assert ceiling.state == task_pacing.COST_CEILING_ACTIVE
|
||||
assert ceiling.ceiling_usd == 5.0
|
||||
assert ceiling.root_cap_usd is None
|
||||
|
|
@ -172,9 +198,9 @@ class TestCostCeilingResolution:
|
|||
assert task_pacing.resolve_cost_ceiling(0.0, profile).state == task_pacing.COST_CEILING_DISABLED
|
||||
|
||||
def test_root_cap_component_binds_when_smaller(self):
|
||||
"""min(pct-of-global, cap − margin): the live wave1/2 shape — a huge
|
||||
global remaining must not hide a $100 tree cap."""
|
||||
profile = normalize_budget_profile(None)
|
||||
"""Explicit profile: min(pct-of-global, cap − margin): the live wave1/2 shape —
|
||||
a huge global remaining must not hide a $100 tree cap."""
|
||||
profile = EXPLICIT_50
|
||||
ceiling = task_pacing.resolve_cost_ceiling(1900.0, profile, root_cap_usd=100.0)
|
||||
assert ceiling.state == task_pacing.COST_CEILING_ACTIVE
|
||||
assert ceiling.root_cap_usd == 100.0
|
||||
|
|
@ -182,14 +208,14 @@ class TestCostCeilingResolution:
|
|||
assert ceiling.planning_margin_usd == task_pacing.COST_PLANNING_MARGIN_USD
|
||||
|
||||
def test_global_pct_component_binds_when_smaller(self):
|
||||
profile = normalize_budget_profile(None)
|
||||
profile = EXPLICIT_50
|
||||
ceiling = task_pacing.resolve_cost_ceiling(10.0, profile, root_cap_usd=100.0)
|
||||
assert ceiling.state == task_pacing.COST_CEILING_ACTIVE
|
||||
assert ceiling.ceiling_usd == 5.0
|
||||
|
||||
def test_root_cap_only_no_finite_global(self):
|
||||
"""A per-task cap with an unbounded global still yields an active stop."""
|
||||
profile = normalize_budget_profile(None)
|
||||
"""Explicit profile: a per-task cap with an unbounded global still yields an active stop."""
|
||||
profile = EXPLICIT_50
|
||||
ceiling = task_pacing.resolve_cost_ceiling(None, profile, root_cap_usd=50.0)
|
||||
assert ceiling.state == task_pacing.COST_CEILING_ACTIVE
|
||||
assert ceiling.ceiling_usd == 50.0 - task_pacing.COST_PLANNING_MARGIN_USD
|
||||
|
|
@ -197,8 +223,9 @@ class TestCostCeilingResolution:
|
|||
def test_cap_at_or_below_margin_soft_lands_never_uncapped(self):
|
||||
"""A root cap at/below the planning margin must resolve to the typed
|
||||
soft-land state — the pre-typed shape returned the same None as
|
||||
'unlimited' (a $0.50 bench cap would have run uncapped)."""
|
||||
profile = normalize_budget_profile(None)
|
||||
'unlimited' (a $0.50 bench cap would have run uncapped). An explicit
|
||||
profile's math; the absent profile has no margin to land under."""
|
||||
profile = EXPLICIT_50
|
||||
for cap in (0.5, task_pacing.COST_PLANNING_MARGIN_USD):
|
||||
ceiling = task_pacing.resolve_cost_ceiling(100.0, profile, root_cap_usd=cap)
|
||||
assert ceiling.state == task_pacing.COST_CEILING_EXHAUSTED_SOFT_LAND, cap
|
||||
|
|
@ -206,7 +233,7 @@ class TestCostCeilingResolution:
|
|||
assert ceiling.root_cap_usd == cap
|
||||
|
||||
def test_ceiling_is_never_computed_zero(self):
|
||||
profile = normalize_budget_profile(None)
|
||||
profile = EXPLICIT_50
|
||||
just_above = task_pacing.COST_PLANNING_MARGIN_USD + 0.01
|
||||
ceiling = task_pacing.resolve_cost_ceiling(1000.0, profile, root_cap_usd=just_above)
|
||||
assert ceiling.state == task_pacing.COST_CEILING_ACTIVE
|
||||
|
|
@ -225,13 +252,12 @@ class TestCostCeilingResolution:
|
|||
)
|
||||
assert at_margin.state == task_pacing.COST_CEILING_EXHAUSTED_SOFT_LAND
|
||||
|
||||
def test_per_task_cap_setting_note_states_the_immediate_finalization(self):
|
||||
"""The owner-facing note must not promise a wrap-up a small cap cannot get.
|
||||
|
||||
The field still accepts a cap below the wrap-up margin (owner power is
|
||||
preserved), but such a cap resolves to `exhausted_soft_land`, which the
|
||||
loop turns into a forced final answer at the TOP of round 0 — zero work
|
||||
rounds. The note said only "a graceful wrap-up fires just before"."""
|
||||
def test_per_task_cap_setting_note_states_the_known_spend_limit(self):
|
||||
"""The owner reads the per-task cap as the task's limit (#1128): the note says
|
||||
the cap itself is where known spend stops new calls, that a resumable task pauses
|
||||
with its work saved (any other run ends budget-exhausted), and that in-flight calls
|
||||
can overshoot — and promises no wrap-up margin, because an ordinary task has none
|
||||
(even at a tiny cap)."""
|
||||
from ouroboros.settings_setup_contract import build_setup_contract
|
||||
|
||||
fields = {
|
||||
|
|
@ -239,33 +265,34 @@ class TestCostCeilingResolution:
|
|||
for field in build_setup_contract().get("budgetFields", [])
|
||||
}
|
||||
note = str(fields["OUROBOROS_PER_TASK_COST_USD"]["note"])
|
||||
assert f"${task_pacing.COST_PLANNING_MARGIN_USD:.2f}" in note
|
||||
assert "finalizes the task immediately" in note
|
||||
# A cap at the documented boundary really does behave that way.
|
||||
assert "known spend" in note and "pauses" in note and "budget-exhausted" in note and "in flight" in note
|
||||
assert "Raising the cap does not resume a paused task" in note
|
||||
assert "wrap-up" not in note and "margin" not in note
|
||||
assert task_pacing.resolve_cost_ceiling(
|
||||
1000.0, normalize_budget_profile(None),
|
||||
root_cap_usd=task_pacing.COST_PLANNING_MARGIN_USD,
|
||||
).state == task_pacing.COST_CEILING_EXHAUSTED_SOFT_LAND
|
||||
).state == task_pacing.COST_CEILING_DISABLED
|
||||
|
||||
def test_planning_margin_is_absolute_not_pct(self):
|
||||
"""The margin must not scale with the cap (a pct reserve amputated the
|
||||
tail of long tasks — v6.54.4 r1; the money-axis analogue is pinned)."""
|
||||
profile = normalize_budget_profile(None)
|
||||
"""An explicit profile's margin must not scale with the cap (a pct reserve
|
||||
amputated the tail of long tasks — v6.54.4 r1; the money-axis analogue)."""
|
||||
profile = EXPLICIT_50
|
||||
small = task_pacing.resolve_cost_ceiling(None, profile, root_cap_usd=10.0)
|
||||
large = task_pacing.resolve_cost_ceiling(None, profile, root_cap_usd=1000.0)
|
||||
assert small.ceiling_usd == 10.0 - task_pacing.COST_PLANNING_MARGIN_USD
|
||||
assert large.ceiling_usd == 1000.0 - task_pacing.COST_PLANNING_MARGIN_USD
|
||||
|
||||
def test_malformed_pct_fails_safe_to_default_not_zero(self):
|
||||
"""A garbage cost_hard_stop_pct must NOT silently become 0 (= no in-task
|
||||
stop, the most permissive setting): negative / non-numeric / a 0<v<1
|
||||
fraction map to None (the 50% default), while an explicit 0 is honored."""
|
||||
def test_malformed_pct_keeps_its_input_contract_and_reads_as_absent(self):
|
||||
"""The accepted-input contract is unchanged: negative / non-numeric / a 0<v<1
|
||||
fraction map to None, never to an explicit 0. None now means what an absent
|
||||
profile means — no default stop — and is disclosed as that, never as the
|
||||
explicit-zero bench contract."""
|
||||
for bad in (-5, -0.1, 0.5, "0.5", "abc", [1]):
|
||||
profile = normalize_budget_profile({"cost_hard_stop_pct": bad})
|
||||
assert profile["cost_hard_stop_pct"] is None, bad
|
||||
ceiling = task_pacing.resolve_cost_ceiling(10.0, profile)
|
||||
assert ceiling.state == task_pacing.COST_CEILING_ACTIVE, bad
|
||||
assert ceiling.ceiling_usd == 5.0, bad
|
||||
assert ceiling.state == task_pacing.COST_CEILING_DISABLED, bad
|
||||
assert ceiling.basis == task_pacing.COST_BASIS_NO_DEFAULT_STOP, bad
|
||||
# explicit 0 (and "0") stays a deliberate no-stop; whole percents clamp.
|
||||
assert normalize_budget_profile({"cost_hard_stop_pct": 0})["cost_hard_stop_pct"] == 0
|
||||
assert normalize_budget_profile({"cost_hard_stop_pct": "0"})["cost_hard_stop_pct"] == 0
|
||||
|
|
@ -316,15 +343,27 @@ class TestWakeRootCeiling:
|
|||
assert self._scope(tmp_path, monkeypatch, metadata={"root_limit_usd": 0.66}).root_limit_usd == 50.0
|
||||
assert self._scope(tmp_path, monkeypatch, metadata={"root_limit_usd": 0.66}, per_task_cap="0").root_limit_usd is None
|
||||
|
||||
def test_a_thin_root_ceiling_soft_lands_the_root_immediately(self, tmp_path, monkeypatch):
|
||||
"""$0.66 left of the allowance is below the planning margin, so the wake gets its
|
||||
one best-effort final answer — which the fence at the full cap still admits."""
|
||||
def test_a_thin_producer_allowance_is_the_working_ceiling_without_a_margin(self, tmp_path, monkeypatch):
|
||||
"""Owner 2026-10-07: $0.66 left of the allowance is $0.66 of work, not an
|
||||
immediate landing under a $3 margin. The allowance stays a real restriction."""
|
||||
from ouroboros.usage_accounting import usage_scope
|
||||
|
||||
ctx = SimpleNamespace()
|
||||
with usage_scope(self._scope(tmp_path, monkeypatch, metadata={"root_cost_ceiling_usd": 0.66})):
|
||||
ceiling = task_pacing.resolve_task_cost_ceiling(SimpleNamespace(), 1000.0)
|
||||
assert ceiling.state == task_pacing.COST_CEILING_EXHAUSTED_SOFT_LAND
|
||||
assert ceiling.root_cap_usd == 50.0 and ceiling.basis == "root_ceiling_at_or_below_planning_margin"
|
||||
ceiling = task_pacing.resolve_task_cost_ceiling(ctx, 1000.0)
|
||||
assert ceiling.state == task_pacing.COST_CEILING_ACTIVE and ceiling.ceiling_usd == 0.66
|
||||
assert ceiling.root_cap_usd == 50.0 and ceiling.basis == "producer_allowance"
|
||||
assert ceiling.planning_margin_usd is None
|
||||
assert task_pacing.cost_stop_authority(ctx) == task_pacing.COST_STOP_PRODUCER
|
||||
|
||||
def test_an_explicit_profile_keeps_its_margin_under_a_producer(self):
|
||||
"""Explicit experiment math is kept verbatim, including its producer margin."""
|
||||
thin = task_pacing.resolve_cost_ceiling(1000.0, EXPLICIT_50, root_cap_usd=50.0, root_ceiling_usd=0.66)
|
||||
assert thin.state == task_pacing.COST_CEILING_EXHAUSTED_SOFT_LAND
|
||||
assert thin.basis == "root_ceiling_at_or_below_planning_margin"
|
||||
wide = task_pacing.resolve_cost_ceiling(1000.0, EXPLICIT_50, root_cap_usd=50.0, root_ceiling_usd=9.0)
|
||||
assert wide.ceiling_usd == 9.0 - task_pacing.COST_PLANNING_MARGIN_USD
|
||||
assert "root_ceiling_minus_margin" in wide.basis
|
||||
|
||||
def test_a_root_ceiling_above_the_margin_is_the_working_ceiling(self, tmp_path, monkeypatch):
|
||||
from ouroboros.usage_accounting import usage_scope
|
||||
|
|
@ -333,8 +372,7 @@ class TestWakeRootCeiling:
|
|||
ceiling = task_pacing.resolve_task_cost_ceiling(SimpleNamespace(), 1000.0)
|
||||
assert ceiling.state == task_pacing.COST_CEILING_ACTIVE
|
||||
assert ceiling.root_cap_usd == 50.0
|
||||
assert ceiling.ceiling_usd == 9.0 - task_pacing.COST_PLANNING_MARGIN_USD
|
||||
assert "root_ceiling_minus_margin" in ceiling.basis
|
||||
assert ceiling.ceiling_usd == 9.0 and ceiling.basis == "producer_allowance"
|
||||
|
||||
def test_a_member_still_inherits_the_resolved_ceiling(self):
|
||||
"""The root's number reaches its members as before; a member's own carrier is the
|
||||
|
|
@ -381,9 +419,7 @@ class TestCostCeilingStop:
|
|||
args = _make_args(
|
||||
budget_remaining_usd=100.0,
|
||||
accumulated_usage={"cost": 26.0},
|
||||
cost_ceiling=task_pacing.resolve_cost_ceiling(
|
||||
50.0, normalize_budget_profile(None),
|
||||
),
|
||||
cost_ceiling=task_pacing.resolve_cost_ceiling(50.0, EXPLICIT_50),
|
||||
llm=llm,
|
||||
drive_logs=tmp_path,
|
||||
)
|
||||
|
|
@ -392,13 +428,11 @@ class TestCostCeilingStop:
|
|||
assert result is not None
|
||||
|
||||
def test_cost_equal_to_ceiling_does_not_stop(self, tmp_path):
|
||||
"""Strict > preserves the historical edge (budget_pct > 0.5)."""
|
||||
"""Strict > preserves the explicit profile's historical edge (budget_pct > 0.5)."""
|
||||
args = _make_args(
|
||||
budget_remaining_usd=100.0,
|
||||
accumulated_usage={"cost": 25.0},
|
||||
cost_ceiling=task_pacing.resolve_cost_ceiling(
|
||||
50.0, normalize_budget_profile(None),
|
||||
),
|
||||
cost_ceiling=task_pacing.resolve_cost_ceiling(50.0, EXPLICIT_50),
|
||||
drive_logs=tmp_path,
|
||||
)
|
||||
with patch.dict(os.environ, {"OUROBOROS_PER_TASK_COST_USD": "999"}):
|
||||
|
|
|
|||
|
|
@ -2,19 +2,22 @@
|
|||
v6.91 tree-fed deciding value, root-accounting telemetry and the latched
|
||||
v6.56.0 cost milestones."""
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from ouroboros import task_pacing
|
||||
from ouroboros.contracts.task_contract import normalize_budget_profile
|
||||
from ouroboros.loop_budget import _check_budget_limits
|
||||
from tests._budget_limits_helpers import _make_args
|
||||
from tests._budget_limits_helpers import EXPLICIT_50, _make_args
|
||||
|
||||
|
||||
# --- v6.91 tree-fed deciding value ---
|
||||
|
||||
class TestTreeFedDecidingValue:
|
||||
"""Under a root cap the deciding spend is the root subtree's ledger-accounted
|
||||
"""Under a root cap the deciding spend is the root subtree's KNOWN (settled)
|
||||
number from the reserve-time scope telemetry — own cost stays a diagnostic.
|
||||
These run an explicit ``cost_hard_stop_pct=50`` profile, the only ordinary-free
|
||||
source of an in-task ceiling besides a producer allowance.
|
||||
The waves died at tree $84-94 while own cost showed $41-49 and no warning
|
||||
ever fired; these pin the closed class."""
|
||||
|
||||
|
|
@ -32,7 +35,7 @@ class TestTreeFedDecidingValue:
|
|||
llm = MagicMock()
|
||||
llm.chat.return_value = ({"content": "done"}, {"prompt_tokens": 1, "completion_tokens": 1})
|
||||
ceiling = task_pacing.resolve_cost_ceiling(
|
||||
1900.0, normalize_budget_profile(None), root_cap_usd=100.0,
|
||||
1900.0, EXPLICIT_50, root_cap_usd=100.0,
|
||||
)
|
||||
assert ceiling.state == task_pacing.COST_CEILING_ACTIVE
|
||||
args = _make_args(
|
||||
|
|
@ -43,7 +46,7 @@ class TestTreeFedDecidingValue:
|
|||
drive_logs=tmp_path,
|
||||
)
|
||||
with self._scoped("root-tree-1", 100.0):
|
||||
usage_accounting._stash_root_accounting("root-tree-1", 98.5, 100.0)
|
||||
usage_accounting._stash_root_accounting("root-tree-1", {"settled_usd": 98.5, "accounted_usd": 98.5}, 100.0)
|
||||
result = _check_budget_limits(**args)
|
||||
assert result is not None
|
||||
text, usage, _ = result
|
||||
|
|
@ -53,7 +56,7 @@ class TestTreeFedDecidingValue:
|
|||
from ouroboros import usage_accounting
|
||||
|
||||
ceiling = task_pacing.resolve_cost_ceiling(
|
||||
1900.0, normalize_budget_profile(None), root_cap_usd=100.0,
|
||||
1900.0, EXPLICIT_50, root_cap_usd=100.0,
|
||||
)
|
||||
args = _make_args(
|
||||
budget_remaining_usd=1900.0,
|
||||
|
|
@ -62,7 +65,7 @@ class TestTreeFedDecidingValue:
|
|||
drive_logs=tmp_path,
|
||||
)
|
||||
with self._scoped("root-tree-2", 100.0):
|
||||
usage_accounting._stash_root_accounting("root-tree-2", 60.0, 100.0)
|
||||
usage_accounting._stash_root_accounting("root-tree-2", {"settled_usd": 60.0, "accounted_usd": 60.0}, 100.0)
|
||||
result = _check_budget_limits(**args)
|
||||
assert result is None
|
||||
|
||||
|
|
@ -75,7 +78,7 @@ class TestTreeFedDecidingValue:
|
|||
llm = MagicMock()
|
||||
llm.chat.return_value = ({"content": "done"}, {"prompt_tokens": 1, "completion_tokens": 1})
|
||||
ceiling = task_pacing.resolve_cost_ceiling(
|
||||
10.0, normalize_budget_profile(None),
|
||||
10.0, EXPLICIT_50,
|
||||
)
|
||||
usage = {"cost": 6.0} # own over the $5 ceiling
|
||||
args = _make_args(
|
||||
|
|
@ -104,7 +107,7 @@ class TestTreeFedDecidingValue:
|
|||
llm = MagicMock()
|
||||
llm.chat.return_value = ({"content": "done"}, {"prompt_tokens": 1, "completion_tokens": 1})
|
||||
ceiling = task_pacing.resolve_cost_ceiling(
|
||||
1900.0, normalize_budget_profile(None), root_cap_usd=10.0,
|
||||
1900.0, EXPLICIT_50, root_cap_usd=10.0,
|
||||
)
|
||||
assert ceiling.state == task_pacing.COST_CEILING_ACTIVE
|
||||
usage = {"cost": 9.0} # over the $10 − margin ceiling
|
||||
|
|
@ -137,7 +140,7 @@ class TestTreeFedDecidingValue:
|
|||
llm = MagicMock()
|
||||
llm.chat.return_value = ({"content": "done"}, {"prompt_tokens": 1, "completion_tokens": 1})
|
||||
ceiling = task_pacing.resolve_cost_ceiling(
|
||||
None, normalize_budget_profile(None), root_cap_usd=100.0,
|
||||
None, EXPLICIT_50, root_cap_usd=100.0,
|
||||
)
|
||||
assert ceiling.state == task_pacing.COST_CEILING_ACTIVE
|
||||
args = _make_args(
|
||||
|
|
@ -148,7 +151,7 @@ class TestTreeFedDecidingValue:
|
|||
drive_logs=tmp_path,
|
||||
)
|
||||
with self._scoped("root-tree-noglobal", 100.0):
|
||||
usage_accounting._stash_root_accounting("root-tree-noglobal", 98.5, 100.0)
|
||||
usage_accounting._stash_root_accounting("root-tree-noglobal", {"settled_usd": 98.5, "accounted_usd": 98.5}, 100.0)
|
||||
result = _check_budget_limits(**args)
|
||||
assert result is not None
|
||||
_text, usage, _ = result
|
||||
|
|
@ -174,12 +177,12 @@ class TestTreeFedDecidingValue:
|
|||
budget_remaining_usd=1900.0,
|
||||
accumulated_usage={"cost": 1.0},
|
||||
cost_ceiling=task_pacing.resolve_cost_ceiling(
|
||||
1900.0, normalize_budget_profile(None), root_cap_usd=100.0,
|
||||
1900.0, EXPLICIT_50, root_cap_usd=100.0,
|
||||
),
|
||||
drive_logs=tmp_path,
|
||||
)
|
||||
with self._scoped("root-tree-fresh", 100.0):
|
||||
usage_accounting._stash_root_accounting("root-tree-fresh", 5.0, 100.0)
|
||||
usage_accounting._stash_root_accounting("root-tree-fresh", {"settled_usd": 5.0, "accounted_usd": 5.0}, 100.0)
|
||||
assert _check_budget_limits(**args) is None
|
||||
assert calls["n"] == 0
|
||||
|
||||
|
|
@ -195,21 +198,21 @@ class TestTreeFedDecidingValue:
|
|||
|
||||
def _fresh_projection(drive_root, **kwargs):
|
||||
reads["n"] += 1
|
||||
return {"accounted_usd": 98.5, "limit_usd": 100.0}
|
||||
return {"settled_usd": 98.5, "accounted_usd": 98.5, "limit_usd": 100.0}
|
||||
|
||||
monkeypatch.setattr(usage_accounting, "usage_projection", _fresh_projection)
|
||||
args = _make_args(
|
||||
budget_remaining_usd=1900.0,
|
||||
accumulated_usage={"cost": 41.0},
|
||||
cost_ceiling=task_pacing.resolve_cost_ceiling(
|
||||
1900.0, normalize_budget_profile(None), root_cap_usd=100.0,
|
||||
1900.0, EXPLICIT_50, root_cap_usd=100.0,
|
||||
),
|
||||
llm=llm,
|
||||
drive_logs=tmp_path,
|
||||
)
|
||||
with self._scoped("root-tree-stale", 100.0):
|
||||
# Stash a pre-block number, then age it past the bound.
|
||||
usage_accounting._stash_root_accounting("root-tree-stale", 40.0, 100.0)
|
||||
usage_accounting._stash_root_accounting("root-tree-stale", {"settled_usd": 40.0, "accounted_usd": 40.0}, 100.0)
|
||||
with usage_accounting._ROOT_ACCOUNTING_TELEMETRY_LOCK:
|
||||
usage_accounting._ROOT_ACCOUNTING_TELEMETRY["root-tree-stale"][
|
||||
"updated_monotonic"
|
||||
|
|
@ -221,12 +224,11 @@ class TestTreeFedDecidingValue:
|
|||
assert usage["cost_stop_spend_basis"] == task_pacing.SPEND_BASIS_TREE
|
||||
|
||||
def test_current_inflight_reservation_participates_in_the_stop(self, tmp_path, monkeypatch):
|
||||
"""G3-4 regression: the stash the loop trusts for 120s must include the
|
||||
reservation of the call currently in flight, not the pre-append sum.
|
||||
The pre-fix shape: a tree near its cap reserved+settled one more call,
|
||||
the loop still saw the pre-call number, and the hard ledger fence fired
|
||||
on the next send before the graceful wrap-up ever ran. Here the ceiling
|
||||
check stops on the un-settled hold alone, with zero fresh ledger reads."""
|
||||
"""G3-4 + #1487: the stash the loop trusts for 120s follows every transition,
|
||||
and it carries the in-flight hold as disclosed EXPOSURE, not known spend. The
|
||||
hold alone stops nothing (owner Q4-A); the moment the call settles, its price
|
||||
is known and the ceiling check stops on it — still with zero fresh ledger
|
||||
reads, because the settlement refreshed the stash itself."""
|
||||
from ouroboros import usage_accounting
|
||||
from ouroboros.usage_accounting import AttemptRequest, UsageScope, usage_scope
|
||||
|
||||
|
|
@ -240,7 +242,7 @@ class TestTreeFedDecidingValue:
|
|||
budget_remaining_usd=1900.0,
|
||||
accumulated_usage={"cost": 0.5},
|
||||
cost_ceiling=task_pacing.resolve_cost_ceiling(
|
||||
1900.0, normalize_budget_profile(None), root_cap_usd=100.0,
|
||||
1900.0, EXPLICIT_50, root_cap_usd=100.0,
|
||||
),
|
||||
llm=llm,
|
||||
drive_logs=tmp_path,
|
||||
|
|
@ -250,14 +252,19 @@ class TestTreeFedDecidingValue:
|
|||
global_limit_usd=1900.0, root_limit_usd=100.0,
|
||||
)
|
||||
with usage_scope(scope):
|
||||
usage_accounting.reserve_attempt(AttemptRequest(
|
||||
reservation = usage_accounting.reserve_attempt(AttemptRequest(
|
||||
model="test/model", provider="openrouter",
|
||||
reservation_usd=99.5, drive_root=tmp_path,
|
||||
))
|
||||
# The attempt is still in flight (never settled) — its hold alone
|
||||
# must already be visible to the deciding surface.
|
||||
# The attempt is still in flight (never settled): its hold is visible
|
||||
# exposure, and known spend is still $0 — no stop on a worst case.
|
||||
entry = usage_accounting.last_root_accounting("root-inflight")
|
||||
assert entry is not None and entry["accounted_usd"] == 99.5
|
||||
assert entry is not None and entry["accounted_usd"] == 99.5 and entry["settled_usd"] == 0.0
|
||||
assert _check_budget_limits(**args) is None
|
||||
usage_accounting.mark_dispatched(reservation)
|
||||
usage_accounting.settle_attempt(reservation, cost_usd=98.0, cost_final=False)
|
||||
entry = usage_accounting.last_root_accounting("root-inflight")
|
||||
assert entry["settled_usd"] == 98.0 and entry["accounted_usd"] == 98.0 # over the $97 ceiling
|
||||
result = _check_budget_limits(**args)
|
||||
assert result is not None
|
||||
_text, usage, _ = result
|
||||
|
|
@ -269,7 +276,7 @@ class TestRootAccountingTelemetry:
|
|||
def test_stash_roundtrip_and_age(self):
|
||||
from ouroboros import usage_accounting
|
||||
|
||||
usage_accounting._stash_root_accounting("root-t-1", 12.5, 100.0)
|
||||
usage_accounting._stash_root_accounting("root-t-1", {"settled_usd": 12.5, "accounted_usd": 12.5}, 100.0)
|
||||
entry = usage_accounting.last_root_accounting("root-t-1")
|
||||
assert entry is not None
|
||||
assert entry["accounted_usd"] == 12.5
|
||||
|
|
@ -291,12 +298,17 @@ class TestRootAccountingTelemetry:
|
|||
)
|
||||
entry = usage_accounting.refresh_root_accounting(tmp_path, "root-t-2")
|
||||
assert entry is not None and entry["accounted_usd"] == 7.25
|
||||
# A bucket that carries no known figure stays unknown: exposure is never
|
||||
# relabelled as known spend (#1487).
|
||||
assert entry["settled_usd"] is None
|
||||
assert usage_accounting.last_root_accounting("root-t-2")["root_limit_usd"] == 25.0
|
||||
with pytest.raises(TypeError):
|
||||
usage_accounting._stash_root_accounting("root-t-2", 7.25, 25.0)
|
||||
|
||||
def test_refresh_failure_returns_stale_stash_not_zero(self, tmp_path, monkeypatch):
|
||||
from ouroboros import usage_accounting
|
||||
|
||||
usage_accounting._stash_root_accounting("root-t-3", 3.0, 10.0)
|
||||
usage_accounting._stash_root_accounting("root-t-3", {"settled_usd": 3.0, "accounted_usd": 3.0}, 10.0)
|
||||
|
||||
def _boom(*a, **k):
|
||||
raise RuntimeError("ledger unavailable")
|
||||
|
|
@ -312,7 +324,7 @@ class TestRootAccountingTelemetry:
|
|||
refreshes the display cache; the display reader keeps its fallback."""
|
||||
from ouroboros import usage_accounting
|
||||
|
||||
usage_accounting._stash_root_accounting("root-strict", 3.0, 10.0)
|
||||
usage_accounting._stash_root_accounting("root-strict", {"settled_usd": 3.0, "accounted_usd": 3.0}, 10.0)
|
||||
assert usage_accounting.last_root_accounting("root-strict")["age_sec"] < 1.0
|
||||
|
||||
def _boom(*a, **k):
|
||||
|
|
@ -364,8 +376,11 @@ class TestRootAccountingTelemetry:
|
|||
def _stashed():
|
||||
entry = usage_accounting.last_root_accounting("root-transitions")
|
||||
assert entry is not None
|
||||
known.append(entry["settled_usd"])
|
||||
return entry["accounted_usd"]
|
||||
|
||||
known = []
|
||||
|
||||
scope = UsageScope(
|
||||
drive_root=tmp_path, task_id="root-transitions", root_task_id="root-transitions",
|
||||
global_limit_usd=100.0, root_limit_usd=50.0,
|
||||
|
|
@ -387,6 +402,8 @@ class TestRootAccountingTelemetry:
|
|||
assert _stashed() == 4.0 # settled + the new hold
|
||||
usage_accounting.release_attempt(second, "not_dispatched")
|
||||
assert _stashed() == 1.0 # released hold no longer counts
|
||||
# Known spend moves only on the settled price, never on a hold.
|
||||
assert known == [0.0, 0.0, 1.0, 1.0, 1.0]
|
||||
|
||||
|
||||
# --- v6.56.0 cost axis: latched milestones + wrap-up (task_pacing content) ---
|
||||
|
|
@ -443,17 +460,17 @@ class TestCostMilestones:
|
|||
assert "Remaining: ~$0.00" in note.text
|
||||
|
||||
def test_tree_cost_is_the_deciding_value_and_is_labeled(self):
|
||||
"""v6.91: the tree-accounted number decides the crossing and is labeled
|
||||
honestly (incl. in-flight holds); own cost rides as the diagnostic."""
|
||||
"""v6.91 + #1487: the tree's known number decides the crossing and is labeled
|
||||
honestly (open holds not counted); own cost rides as the diagnostic."""
|
||||
ctx = SimpleNamespace()
|
||||
note = task_pacing.build_cost_budget_note(
|
||||
ctx, start_remaining_usd=200.0, cost_ceiling_usd=97.0,
|
||||
task_cost=41.0, tree_cost_usd=50.0,
|
||||
)
|
||||
assert note is not None and "50% remaining" in note.text
|
||||
assert "in-flight holds" in note.text
|
||||
assert "known spend" in note.text and "open holds not counted" in note.text
|
||||
assert "own calls ~$41.00" in note.text
|
||||
assert note.checkpoint["spend_basis"] == "tree_accounted"
|
||||
assert note.checkpoint["spend_basis"] == "tree_known"
|
||||
# The deciding (tree) number and this task's own cost are BOTH recorded,
|
||||
# each under the name that means it — see the meaning-stability pin below.
|
||||
assert note.checkpoint["deciding_spend_usd"] == 50.0
|
||||
|
|
|
|||
|
|
@ -282,17 +282,19 @@ def test_root_grant_refuses_a_cached_tree_snapshot_and_admits_a_fresh_read(tmp_p
|
|||
queue, state, workers = _install_queue(tmp_path, monkeypatch)
|
||||
monkeypatch.setattr(state, "budget_remaining", lambda _st, **_k: 5.0)
|
||||
task, _row = _parked(tmp_path, monkeypatch, task_id="strict-root", scope="root")
|
||||
usage_accounting._stash_root_accounting("strict-root", 1.0, 10.0)
|
||||
usage_accounting._stash_root_accounting("strict-root", {"settled_usd": 1.0, "accounted_usd": 1.0}, 10.0)
|
||||
monkeypatch.setattr(usage_accounting, "usage_projection",
|
||||
lambda *_a, **_k: (_ for _ in ()).throw(OSError("ledger unavailable")))
|
||||
refused = queue.resume_budget_paused_task("strict-root")
|
||||
assert refused["error"] == "root_accounting_unavailable" and refused["action"] == "retry_or_cancel"
|
||||
assert "_budget_pause" in task and "_budget_pause_resume" not in task
|
||||
monkeypatch.setattr(usage_accounting, "usage_projection",
|
||||
lambda *_a, **_k: {"accounted_usd": 10.0, "limit_usd": 10.0})
|
||||
lambda *_a, **_k: {"settled_usd": 10.0, "accounted_usd": 10.0, "limit_usd": 10.0})
|
||||
assert queue.resume_budget_paused_task("strict-root")["error"] == "root_hard_cap_exhausted"
|
||||
# #1487: a $25 unresolved bound beside $1 of known spend is exposure, not spending:
|
||||
# explicit Resume within the authorized $10 is granted.
|
||||
monkeypatch.setattr(usage_accounting, "usage_projection",
|
||||
lambda *_a, **_k: {"accounted_usd": 1.0, "limit_usd": 10.0})
|
||||
lambda *_a, **_k: {"settled_usd": 1.0, "accounted_usd": 26.0, "limit_usd": 10.0})
|
||||
assert queue.resume_budget_paused_task("strict-root")["ok"] is True
|
||||
|
||||
|
||||
|
|
@ -470,7 +472,7 @@ def test_graceful_rail_refreshes_planning_threshold_within_authorized_money(tmp_
|
|||
# wallet observation and a fresh, undegraded root-accounting read.
|
||||
monkeypatch.setattr(loop_budget, "_wrapup_global_remaining", lambda: 100.0)
|
||||
monkeypatch.setattr(loop_budget, "_loop_tree_accounting",
|
||||
lambda **_k: {"accounted_usd": 8.0, "age_sec": 0.0})
|
||||
lambda **_k: {"settled_usd": 8.0, "accounted_usd": 8.0, "age_sec": 0.0})
|
||||
monkeypatch.setattr(task_pacing, "resolve_budget_profile", lambda _c: {"cost_hard_stop_pct": 50})
|
||||
disclosure = budget_pause._refresh_planning_threshold(ctx, budget_remaining_usd=100.0)
|
||||
assert disclosure["refreshed"] is True
|
||||
|
|
@ -487,7 +489,7 @@ def test_graceful_rail_refreshes_planning_threshold_within_authorized_money(tmp_
|
|||
# authorized room, and the owner's explicit act cannot invent any.
|
||||
ctx._accumulated_usage = {"cost": 10.0}
|
||||
monkeypatch.setattr(loop_budget, "_loop_tree_accounting",
|
||||
lambda **_k: {"accounted_usd": 10.0, "age_sec": 0.0})
|
||||
lambda **_k: {"settled_usd": 10.0, "accounted_usd": 10.0, "age_sec": 0.0})
|
||||
spent = budget_pause._refresh_planning_threshold(ctx, budget_remaining_usd=100.0)
|
||||
assert spent["refreshed"] is False and spent["reason"] == "no_authorized_room"
|
||||
|
||||
|
|
@ -514,7 +516,7 @@ def test_threshold_refresh_refuses_every_unknown_or_stale_money_fact(tmp_path, m
|
|||
ctx._accumulated_usage = {"cost": 8.0}
|
||||
monkeypatch.setattr(loop_budget, "_wrapup_global_remaining", lambda: None)
|
||||
monkeypatch.setattr(loop_budget, "_loop_tree_accounting",
|
||||
lambda **_k: {"accounted_usd": 8.0, "age_sec": 0.0})
|
||||
lambda **_k: {"settled_usd": 8.0, "accounted_usd": 8.0, "age_sec": 0.0})
|
||||
assert budget_pause._refresh_planning_threshold(ctx, budget_remaining_usd=100.0) == {
|
||||
"refreshed": False, "reason": "wallet_unavailable", "wallet_basis": "ledger_unavailable",
|
||||
"dispatch_time_remaining_usd": 100.0}
|
||||
|
|
@ -523,8 +525,8 @@ def test_threshold_refresh_refuses_every_unknown_or_stale_money_fact(tmp_path, m
|
|||
monkeypatch.setattr(loop_budget, "_wrapup_global_remaining", lambda: 100.0)
|
||||
for tree, reason in (
|
||||
(None, "tree_spend_unavailable"),
|
||||
({"accounted_usd": 8.0, "age_sec": 0.0, "integrity_degraded": True}, "tree_accounting_degraded"),
|
||||
({"accounted_usd": None, "age_sec": 0.0}, "tree_spend_unknown"),
|
||||
({"settled_usd": 8.0, "accounted_usd": 8.0, "age_sec": 0.0, "integrity_degraded": True}, "tree_accounting_degraded"),
|
||||
({"settled_usd": None, "accounted_usd": None, "age_sec": 0.0}, "tree_spend_unknown"),
|
||||
):
|
||||
ctx._cost_ceiling = _ceiling()
|
||||
monkeypatch.setattr(loop_budget, "_loop_tree_accounting", lambda _t=tree, **_k: _t)
|
||||
|
|
@ -547,7 +549,7 @@ def test_threshold_refresh_reads_the_ledger_now_never_a_cached_snapshot(tmp_path
|
|||
ctx._accumulated_usage = {"cost": 8.0}
|
||||
monkeypatch.setattr(task_pacing, "resolve_budget_profile", lambda _c: {"cost_hard_stop_pct": 50})
|
||||
monkeypatch.setattr(loop_budget, "_wrapup_global_remaining", lambda: 100.0)
|
||||
usage_accounting._stash_root_accounting("q10-root", 8.0, 10.0) # fresh, 0-age display cache
|
||||
usage_accounting._stash_root_accounting("q10-root", {"settled_usd": 8.0, "accounted_usd": 8.0}, 10.0) # fresh, 0-age display cache
|
||||
monkeypatch.setattr(usage_accounting, "usage_projection",
|
||||
lambda *_a, **_k: (_ for _ in ()).throw(OSError("ledger unavailable")))
|
||||
with usage_scope(UsageScope(drive_root=tmp_path, task_id="q10-task", root_task_id="q10-root")):
|
||||
|
|
@ -555,7 +557,7 @@ def test_threshold_refresh_reads_the_ledger_now_never_a_cached_snapshot(tmp_path
|
|||
assert refused == {"refreshed": False, "reason": "tree_spend_unavailable", "wallet_basis": "ledger_projection"}
|
||||
assert ctx._cost_ceiling.ceiling_usd == 7.0
|
||||
monkeypatch.setattr(usage_accounting, "usage_projection",
|
||||
lambda *_a, **_k: {"accounted_usd": 8.0, "limit_usd": 10.0})
|
||||
lambda *_a, **_k: {"settled_usd": 8.0, "accounted_usd": 8.0, "limit_usd": 10.0})
|
||||
granted = budget_pause._refresh_planning_threshold(ctx, budget_remaining_usd=100.0)
|
||||
assert granted["refreshed"] is True and granted["root_cap_basis"] == "root_accounting"
|
||||
assert ctx._cost_ceiling.ceiling_usd == pytest.approx(10.0)
|
||||
|
|
|
|||
|
|
@ -63,6 +63,16 @@ def _reserve(tmp_path, task_id, *, request=None, **limits):
|
|||
return (exc.limit_scope, str(exc))
|
||||
|
||||
|
||||
def _settle_known(tmp_path, task_id, cost):
|
||||
"""A send of the task's tree settled at a final price: KNOWN spend (#1487)."""
|
||||
from ouroboros import usage_accounting as ua
|
||||
|
||||
with ua.usage_scope(ua.UsageScope(drive_root=tmp_path, task_id=task_id, root_task_id=ROOT)):
|
||||
held = ua.reserve_attempt(ua.AttemptRequest(model="m", provider="p", task_id=task_id, reservation_usd=cost))
|
||||
ua.mark_dispatched(held)
|
||||
ua.settle_attempt(held, {}, cost_usd=cost, cost_final=True)
|
||||
|
||||
|
||||
def _fixed_price(monkeypatch):
|
||||
monkeypatch.setattr("ouroboros.usage_accounting._reservation_cost", lambda _r: 0.01)
|
||||
|
||||
|
|
@ -167,6 +177,7 @@ def test_the_real_caps_still_bind_a_selected_child_and_its_clocks_carry_the_paus
|
|||
meta = workers.RUNNING[CHILD]
|
||||
_consume(monkeypatch, child_ctx, child_limit, sent[0]["_budget_pause_resume"])
|
||||
grant = budget_pause.budget_pause_row(tmp_path, CHILD)["grant"]
|
||||
_settle_known(tmp_path, CHILD, 0.01) # the caps below decide on this known spend, not on holds
|
||||
root_cap = _reserve(tmp_path, CHILD, root_limit_usd=0.005)
|
||||
global_cap = _reserve(tmp_path, CHILD, request={"global_limit_usd": 0.005})
|
||||
_record("selected_child_caps", {"root_cap": root_cap, "global_cap": global_cap,
|
||||
|
|
|
|||
|
|
@ -34,6 +34,14 @@ def rows(root):
|
|||
return ledger_rows(root)
|
||||
|
||||
|
||||
def spend(root, amount, **kwargs):
|
||||
"""KNOWN spend: what every money limit decides on (#1487)."""
|
||||
held = accounting.reserve_attempt(request(root, reservation_usd=amount, **kwargs))
|
||||
accounting.mark_dispatched(held)
|
||||
accounting.settle_attempt(held, {}, cost_usd=amount, cost_final=True)
|
||||
return held
|
||||
|
||||
|
||||
def test_fallback_limit_is_the_value_applied_and_survives_settlement(root, monkeypatch):
|
||||
reservation = accounting.reserve_attempt(request(root))
|
||||
monkeypatch.setenv("TOTAL_BUDGET", "500")
|
||||
|
|
@ -90,7 +98,7 @@ def test_a_running_task_follows_the_budget_the_owner_saves_mid_run(root, monkeyp
|
|||
monkeypatch.setenv("TOTAL_BUDGET", "1.5")
|
||||
|
||||
def scoped(_task):
|
||||
accounting.reserve_attempt(request(root))
|
||||
spend(root, 1.5) # known spend reaches the $1.50 the task started under
|
||||
with pytest.raises(accounting.BudgetExceeded):
|
||||
accounting.reserve_attempt(request(root))
|
||||
settings.write_text(json.dumps({"TOTAL_BUDGET": 10})) # the owner tops the budget up; this process's env is untouched
|
||||
|
|
@ -116,7 +124,7 @@ def test_the_wrapup_wallet_observation_follows_the_saved_budget_too(root, monkey
|
|||
settings.write_text(json.dumps({"TOTAL_BUDGET": 100}))
|
||||
|
||||
def scoped(_task):
|
||||
accounting.reserve_attempt(request(root, reservation_usd=90))
|
||||
spend(root, 90)
|
||||
before = _wrapup_global_remaining()
|
||||
settings.write_text(json.dumps({"TOTAL_BUDGET": 500}))
|
||||
return before, _wrapup_global_remaining()
|
||||
|
|
@ -201,7 +209,8 @@ def test_unbounded_fallback_is_explicit_without_nonfinite_json(root, monkeypatch
|
|||
|
||||
def test_soft_ceilings_reserve_nothing_and_concurrent_sends_still_share_one_pool(root, monkeypatch):
|
||||
monkeypatch.setenv("TOTAL_BUDGET", "10")
|
||||
ceilings = [pacing.resolve_cost_ceiling(10, {}) for _ in range(2)]
|
||||
assert pacing.resolve_cost_ceiling(10, {}).state == pacing.COST_CEILING_DISABLED # ordinary: no early stop
|
||||
ceilings = [pacing.resolve_cost_ceiling(10, {"cost_hard_stop_pct": 50}) for _ in range(2)]
|
||||
assert [ceiling.ceiling_usd for ceiling in ceilings] == [5, 5]
|
||||
assert not ledger_rows(root)
|
||||
assert all(pacing.cost_ceiling_disclosure(c)["allocation"] == "unreserved_shared_pool" for c in ceilings)
|
||||
|
|
@ -218,9 +227,18 @@ def test_soft_ceilings_reserve_nothing_and_concurrent_sends_still_share_one_pool
|
|||
|
||||
with ThreadPoolExecutor(max_workers=2) as pool:
|
||||
results = list(pool.map(send, ("a", "b")))
|
||||
assert sum(isinstance(result, accounting.AttemptReservation) for result in results) == 1
|
||||
assert sum(isinstance(result, accounting.BudgetExceeded) for result in results) == 1
|
||||
assert accounting.usage_projection(root)["accounted_usd"] == 6
|
||||
# Owner Q4-A: both concurrent sends are admitted on $0 known spend — the pool's
|
||||
# overshoot is accepted and disclosed as exposure, never prevented by holds.
|
||||
assert all(isinstance(result, accounting.AttemptReservation) for result in results)
|
||||
assert accounting.usage_projection(root)["accounted_usd"] == 12
|
||||
for result in results:
|
||||
accounting.mark_dispatched(result)
|
||||
accounting.settle_attempt(result, {}, cost_usd=6, cost_final=True)
|
||||
assert accounting.usage_projection(root)["settled_usd"] == 12 # a recorded overrun of $2
|
||||
with pytest.raises(accounting.BudgetExceeded):
|
||||
accounting.reserve_attempt(accounting.AttemptRequest(
|
||||
model="fixture", provider="openai", drive_root=root, reservation_usd=0.01,
|
||||
task_id="c", root_task_id="c"))
|
||||
|
||||
|
||||
def delivered_tool(name, ident, *, duration=None, error=False):
|
||||
|
|
@ -284,9 +302,12 @@ def test_real_note_transport_keeps_counts_measured_time_and_open_spend(root, mon
|
|||
assert spend["delegated_tree"]["subscription_sessions"] == 2
|
||||
assert spend["delegated_tree"]["unknown_unmetered"] == 1
|
||||
assert spend["delegated_tree"]["cost_final"] is False
|
||||
assert spend["global"]["remaining_known_usd"] == 96.75
|
||||
# Room above KNOWN spend ($1.25); the $2 unresolved bound rides as exposure (#1487).
|
||||
assert spend["global"]["remaining_known_usd"] == 98.75
|
||||
assert spend["global"]["settled_usd"] == 1.25 and spend["global"]["accounted_usd"] == 3.25
|
||||
assert spend["global"]["allocation"] == "unreserved_shared_pool"
|
||||
rendered = messages[-1]["content"].split("Observed resource facts (accounted money includes open holds):\n")[1]
|
||||
rendered = messages[-1]["content"].split(
|
||||
"Observed resource facts (known spend is settled_usd; accounted money adds open holds):\n")[1]
|
||||
assert json.loads(rendered) == facts
|
||||
monkeypatch.setattr(accounting, "usage_breakdown", lambda *_a, **_kw: pytest.fail("read before next milestone"))
|
||||
assert not inject()
|
||||
|
|
|
|||
279
tests/test_capped_output_copy.py
Normal file
279
tests/test_capped_output_copy.py
Normal file
|
|
@ -0,0 +1,279 @@
|
|||
"""#1569: the launcher's pipe copy is byte-capped and keeps draining after storage errors.
|
||||
|
||||
Owner decision of 8 October 2026 (quiz 9928636c): keep the size limit; while storage
|
||||
refuses writes, keep draining and drop new chunks with an explicit gap, never truncate
|
||||
the old live file after a failed rename and never append past the cap.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import errno
|
||||
import io
|
||||
import os
|
||||
import pathlib
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
|
||||
import pytest
|
||||
|
||||
from ouroboros import process_logging
|
||||
from ouroboros.process_logging import OUTPUT_CHUNK_BYTES, SERVER_LOG_MAX_BYTES, copy_capped_output
|
||||
|
||||
|
||||
class Report:
|
||||
"""The launcher's own log, recorded."""
|
||||
|
||||
def __init__(self):
|
||||
self.lines = []
|
||||
|
||||
def warning(self, message, *args):
|
||||
self.lines.append(("warning", message % args))
|
||||
|
||||
def info(self, message, *args):
|
||||
self.lines.append(("info", message % args))
|
||||
|
||||
|
||||
class Reads(io.BytesIO):
|
||||
def __init__(self, data):
|
||||
super().__init__(data)
|
||||
self.sizes = []
|
||||
|
||||
def readline(self, limit=-1):
|
||||
chunk = super().readline(limit)
|
||||
self.sizes.append(len(chunk))
|
||||
return chunk
|
||||
|
||||
|
||||
def generations(path):
|
||||
"""Oldest first: .3, .2, .1, live."""
|
||||
names = [path.with_name(f"{path.name}.{index}") for index in (3, 2, 1)] + [path]
|
||||
return [name for name in names if name.exists()]
|
||||
|
||||
|
||||
def stored(path):
|
||||
return b"".join(name.read_bytes() for name in generations(path))
|
||||
|
||||
|
||||
def mixed_lines(total):
|
||||
"""Multibyte, CRLF and malformed UTF-8 lines of varied length."""
|
||||
pieces, size, index = [], 0, 0
|
||||
while size < total:
|
||||
body = ("é漢" * (index % 400)).encode("utf-8") + b"\xff\xfe" * (index % 3)
|
||||
line = b"%06d " % index + body + (b"\r\n" if index % 2 else b"\n")
|
||||
pieces.append(line)
|
||||
size += len(line)
|
||||
index += 1
|
||||
return b"".join(pieces)
|
||||
|
||||
|
||||
def test_raw_bytes_are_copied_exactly_and_every_file_stays_within_the_cap(tmp_path):
|
||||
path, cap, data = tmp_path / "agent_stdout.log", 100_000, mixed_lines(1_000_000)
|
||||
stream = Reads(data)
|
||||
copy_capped_output(stream, path, report=Report(), max_bytes=cap)
|
||||
|
||||
files = generations(path)
|
||||
assert len(files) == 4 and all(name.stat().st_size <= cap for name in files)
|
||||
kept = stored(path)
|
||||
assert data.endswith(kept) and len(kept) > 3 * cap - OUTPUT_CHUNK_BYTES # No decoding, no newline translation.
|
||||
assert all(name.read_bytes().startswith(b"0") for name in files) # Rotation falls between lines.
|
||||
assert stream.tell() == len(data)
|
||||
|
||||
|
||||
def test_a_newline_free_flood_is_read_in_bounded_chunks(tmp_path):
|
||||
path, data = tmp_path / "agent_stdout.log", b"x" * (5 * 1024 * 1024)
|
||||
stream = Reads(data)
|
||||
copy_capped_output(stream, path, report=Report())
|
||||
|
||||
assert max(stream.sizes) == OUTPUT_CHUNK_BYTES
|
||||
assert all(name.stat().st_size <= SERVER_LOG_MAX_BYTES for name in generations(path))
|
||||
assert stored(path) == data # Three generations hold 5 MiB; nothing older existed.
|
||||
|
||||
|
||||
def test_an_inherited_oversized_file_rotates_whole_and_is_not_cleaned(tmp_path):
|
||||
path, cap = tmp_path / "agent_stdout.log", 4096
|
||||
inherited = b"old writer line\n" * 1000 # 16 000 bytes, written by someone else
|
||||
path.write_bytes(inherited)
|
||||
copy_capped_output(Reads(b"new line\n" * 10), path, report=Report(), max_bytes=cap)
|
||||
|
||||
assert path.with_name("agent_stdout.log.1").read_bytes() == inherited # Disclosed, aged out later.
|
||||
assert path.read_bytes() == b"new line\n" * 10
|
||||
|
||||
|
||||
@pytest.mark.parametrize("refused", ["live_rename", "every_rename"])
|
||||
def test_a_failed_rotation_never_truncates_never_passes_the_cap_and_keeps_draining(tmp_path, monkeypatch, refused):
|
||||
path, cap = tmp_path / "agent_stdout.log", 4096
|
||||
live = b"L" * (cap - 5)
|
||||
path.write_bytes(live)
|
||||
olds = {index: bytes([48 + index]) * 100 for index in (1, 2, 3)}
|
||||
for index, body in olds.items():
|
||||
path.with_name(f"agent_stdout.log.{index}").write_bytes(body)
|
||||
real_replace, attempts = os.replace, []
|
||||
|
||||
def replace(src, dst):
|
||||
attempts.append(pathlib.Path(src).name)
|
||||
if refused == "every_rename" or pathlib.Path(src) == path:
|
||||
raise PermissionError(errno.EACCES, "held by another process")
|
||||
real_replace(src, dst)
|
||||
|
||||
monkeypatch.setattr(process_logging.os, "replace", replace)
|
||||
data = b"0123456789\n" * 200
|
||||
stream, report = Reads(data), Report()
|
||||
copy_capped_output(stream, path, report=report, max_bytes=cap)
|
||||
|
||||
assert stream.tell() == len(data) # Every chunk was drained.
|
||||
assert path.read_bytes() == live # Never truncated, never appended past the cap.
|
||||
backups = {index: path.with_name(f"agent_stdout.log.{index}").read_bytes()
|
||||
for index in (1, 2, 3) if path.with_name(f"agent_stdout.log.{index}").exists()}
|
||||
if refused == "every_rename":
|
||||
assert backups == olds
|
||||
else:
|
||||
# The first attempt shifted the chain once (the oldest generation is what any
|
||||
# rotation discards); every retry found .1 free and moved nothing else.
|
||||
assert backups == {2: olds[1], 3: olds[2]}
|
||||
assert attempts[:3] == ["agent_stdout.log.2", "agent_stdout.log.1", "agent_stdout.log"]
|
||||
assert set(attempts[3:]) == {"agent_stdout.log"}
|
||||
assert report.lines == [
|
||||
("warning", "agent_stdout.log storage failed (rotate:PermissionError:EACCES); output keeps draining "
|
||||
"and is dropped until a write succeeds"),
|
||||
("warning", f"agent_stdout.log output ended with storage failing (rotate:PermissionError:EACCES): "
|
||||
f"{len(data)} bytes not written, 0 bytes of uncertain durability")]
|
||||
|
||||
|
||||
def test_storage_that_recovers_writes_one_bounded_marker_counted_toward_the_cap(tmp_path, monkeypatch):
|
||||
path, cap = tmp_path / "agent_stdout.log", 200
|
||||
path.write_bytes(b"p" * (cap - 15))
|
||||
real_open, refusals = open, [3]
|
||||
|
||||
def flaky_open(*args, **kwargs):
|
||||
if refusals[0]:
|
||||
refusals[0] -= 1
|
||||
raise OSError(errno.ENOSPC, "No space left on device")
|
||||
return real_open(*args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(process_logging, "open", flaky_open, raising=False)
|
||||
lines = [b"line-%d....\n" % index for index in range(5)] # 11 bytes each
|
||||
report = Report()
|
||||
copy_capped_output(Reads(b"".join(lines)), path, report=report, max_bytes=cap)
|
||||
|
||||
marker = (b"[output gap: 33 bytes not written, 0 bytes of uncertain durability "
|
||||
b"(open:OSError:ENOSPC)]\n")
|
||||
assert path.with_name("agent_stdout.log.1").read_bytes() == b"p" * (cap - 15)
|
||||
assert path.read_bytes() == marker + lines[3] + lines[4] # The marker rotated the file first.
|
||||
assert all(name.stat().st_size <= cap for name in generations(path))
|
||||
assert report.lines == [
|
||||
("warning", "agent_stdout.log storage failed (open:OSError:ENOSPC); output keeps draining and is "
|
||||
"dropped until a write succeeds"),
|
||||
("info", "agent_stdout.log storage recovered: 33 bytes not written, 0 bytes of uncertain durability")]
|
||||
|
||||
|
||||
def test_a_failed_write_is_uncertain_and_kept_apart_from_unwritten_bytes(tmp_path, monkeypatch):
|
||||
path = tmp_path / "agent_stdout.log"
|
||||
real_open, plan = open, ["partial", "refuse", "ok"]
|
||||
|
||||
class PartialWrite:
|
||||
"""Four bytes reach the file, then the device fails."""
|
||||
|
||||
def __init__(self, handle):
|
||||
self.handle, self.calls = handle, 0
|
||||
|
||||
def write(self, data):
|
||||
self.calls += 1
|
||||
if self.calls == 1:
|
||||
return self.handle.write(data) # The first chunk, whole
|
||||
if self.calls == 2:
|
||||
return self.handle.write(bytes(data[:4]))
|
||||
raise OSError(errno.EIO, "Input/output error")
|
||||
|
||||
def fileno(self):
|
||||
return self.handle.fileno()
|
||||
|
||||
def close(self):
|
||||
self.handle.close()
|
||||
|
||||
def opener(*args, **kwargs):
|
||||
step = plan.pop(0)
|
||||
if step == "refuse":
|
||||
raise PermissionError(errno.EACCES, "denied")
|
||||
handle = real_open(*args, **kwargs)
|
||||
return PartialWrite(handle) if step == "partial" else handle
|
||||
|
||||
monkeypatch.setattr(process_logging, "open", opener, raising=False)
|
||||
chunks = [b"first-ok.\n", b"second-10\n", b"third-10.\n", b"fourth-ok\n"]
|
||||
report = Report()
|
||||
copy_capped_output(Reads(b"".join(chunks)), path, report=report)
|
||||
|
||||
# The healthy handle writes chunk 1; 4 bytes of chunk 2 reach the file and its other
|
||||
# 6 are uncertain; chunk 3 meets a refused open and is known not written.
|
||||
marker = b"[output gap: 10 bytes not written, 6 bytes of uncertain durability (write:OSError:EIO)]\n"
|
||||
assert path.read_bytes() == chunks[0] + chunks[1][:4] + marker + chunks[3]
|
||||
assert report.lines[-1] == (
|
||||
"info", "agent_stdout.log storage recovered: 10 bytes not written, 6 bytes of uncertain durability")
|
||||
|
||||
|
||||
def test_a_failing_report_never_stops_the_copy(tmp_path):
|
||||
class Broken:
|
||||
def warning(self, *_args):
|
||||
raise RuntimeError("launcher log is gone")
|
||||
|
||||
info = warning
|
||||
|
||||
blocker = tmp_path / "not-a-directory"
|
||||
blocker.write_text("file", encoding="utf-8")
|
||||
data = b"line\n" * 1000
|
||||
stream = Reads(data)
|
||||
copy_capped_output(stream, blocker / "agent_stdout.log", report=Broken())
|
||||
assert stream.tell() == len(data)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("failure", [OSError(errno.EIO, "Input/output error"), ValueError("I/O on closed file")])
|
||||
def test_a_broken_input_ends_the_copy_instead_of_spinning(tmp_path, failure):
|
||||
class Failing:
|
||||
calls = 0
|
||||
|
||||
def readline(self, _limit):
|
||||
self.calls += 1
|
||||
if self.calls == 1:
|
||||
return b"last words\n"
|
||||
raise failure
|
||||
|
||||
stream, report = Failing(), Report()
|
||||
copy_capped_output(stream, tmp_path / "agent_stdout.log", report=report)
|
||||
assert stream.calls == 2 and (tmp_path / "agent_stdout.log").read_bytes() == b"last words\n"
|
||||
assert report.lines == [("warning", f"Output pipe read failed ({type(failure).__name__}"
|
||||
f"{':EIO' if isinstance(failure, OSError) else ''}); "
|
||||
"the copy into agent_stdout.log stopped")]
|
||||
|
||||
|
||||
PRODUCER = ("import sys\nout = sys.stdout.buffer\n"
|
||||
"for index in range(10240):\n out.write(b'%07d ' % index + b'x' * 1015 + b'\\n')\nout.flush()\n")
|
||||
|
||||
|
||||
@pytest.mark.serial
|
||||
@pytest.mark.parametrize("sink", ["unwritable", "writable"])
|
||||
def test_a_real_child_writing_past_pipe_capacity_never_blocks(tmp_path, sink):
|
||||
"""10 MiB through a real pipe: the child exits with a working sink or immediate storage errors."""
|
||||
blocker = tmp_path / "blocker"
|
||||
if sink == "unwritable":
|
||||
blocker.write_text("a file where the logs directory should be", encoding="utf-8")
|
||||
path = blocker / "agent_stdout.log"
|
||||
child = subprocess.Popen([sys.executable, "-c", PRODUCER], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
|
||||
report = Report()
|
||||
copier = threading.Thread(target=copy_capped_output, args=(child.stdout, path), kwargs={"report": report})
|
||||
copier.start()
|
||||
try:
|
||||
assert child.wait(timeout=120) == 0
|
||||
finally:
|
||||
copier.join(timeout=120)
|
||||
child.stdout.close()
|
||||
assert not copier.is_alive()
|
||||
total = 10240 * 1024
|
||||
if sink == "unwritable":
|
||||
assert [level for level, _ in report.lines] == ["warning", "warning"]
|
||||
assert report.lines[1][1].endswith(f"{total} bytes not written, 0 bytes of uncertain durability")
|
||||
else:
|
||||
assert report.lines == []
|
||||
kept = stored(path)
|
||||
assert len(generations(path)) == 4 and all(name.stat().st_size <= SERVER_LOG_MAX_BYTES
|
||||
for name in generations(path))
|
||||
assert kept.endswith(b"%07d " % 10239 + b"x" * 1015 + b"\n") and len(kept) % 1024 == 0
|
||||
assert len(kept) > 3 * SERVER_LOG_MAX_BYTES
|
||||
|
|
@ -10,6 +10,7 @@ from tests.test_acceptance_delivery import (
|
|||
_CLEAN_VERDICT, _EpisodeLLM, _ROW_API, _ROW_NATIVE, _ROW_SESSION,
|
||||
_fake_session, _offline_env, _priced_offline_model, _real_panel, _roots, _tool_call,
|
||||
)
|
||||
from tests.test_acceptance_floor_admission import _KnownPricedLLM
|
||||
from tests._usage_store_testing import ledger_rows
|
||||
|
||||
|
||||
|
|
@ -37,7 +38,10 @@ def _ctx(tmp_path, workspace, governance, *, child=True, expired=False):
|
|||
pending_events=[])
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("limit", "expired", "sends"), [(1.0, False, 2), (0.15, False, 1), (1.0, True, 0)])
|
||||
# Each send settles a KNOWN $0.10. Known $0.10 below $0.15 admits the second send
|
||||
# although its own bound would cross the cap; known $0.10 AT a $0.10 cap refuses it.
|
||||
@pytest.mark.parametrize(("limit", "expired", "sends"),
|
||||
[(1.0, False, 2), (0.15, False, 2), (0.1, False, 1), (1.0, True, 0)])
|
||||
def test_child_one_native_reviewer_can_reason_again_but_inherits_money_and_deadline(monkeypatch, tmp_path, limit, expired, sends):
|
||||
from ouroboros import usage_accounting as ua
|
||||
|
||||
|
|
@ -45,8 +49,8 @@ def test_child_one_native_reviewer_can_reason_again_but_inherits_money_and_deadl
|
|||
_priced_offline_model(monkeypatch)
|
||||
governance, workspace = _roots(tmp_path)
|
||||
ctx = _ctx(tmp_path, workspace, governance, expired=expired)
|
||||
llm = _EpisodeLLM(tmp_path, [{"tool_calls": [_tool_call("read_file", {"path": "greeting.txt"})]},
|
||||
{"content": json.dumps(_CLEAN_VERDICT)}], scoped=True, reservation_usd=0.1)
|
||||
llm = _KnownPricedLLM(tmp_path, [{"tool_calls": [_tool_call("read_file", {"path": "greeting.txt"})]},
|
||||
{"content": json.dumps(_CLEAN_VERDICT)}], scoped=True, reservation_usd=0.1)
|
||||
seen = _real_panel(monkeypatch, llm, stub_gate=False)
|
||||
with ua.usage_scope(ua.UsageScope(drive_root=tmp_path, task_id="child", root_task_id="root", root_limit_usd=limit)):
|
||||
result = _call(ctx, reviewer_slot_id="t_actor")
|
||||
|
|
@ -55,9 +59,12 @@ def test_child_one_native_reviewer_can_reason_again_but_inherits_money_and_deadl
|
|||
assert seen[0].deadline_at == ctx.task_metadata["deadline_at"]
|
||||
if sends == 2:
|
||||
assert any(m.get("role") == "tool" and "hello" in str(m) for m in llm.calls[-1]["messages"])
|
||||
if sends == 1:
|
||||
assert result["actors"][0]["usage"]["native_end_reason"] == "budget_exhausted"
|
||||
if sends:
|
||||
rows = ledger_rows(tmp_path)
|
||||
assert all(r["root_task_id"] == "root" and r["task_id"] == "child" for r in rows)
|
||||
assert ua.usage_projection(tmp_path, root_task_id="root")["settled_usd"] == pytest.approx(0.1 * sends)
|
||||
assert not (tmp_path / "state/child_review_cycles.json").exists() # no extra cycle owner
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -1,14 +1,16 @@
|
|||
"""Commit-gate wave admission (owner decision 2026-09-05, answer 2 = A).
|
||||
"""Commit-gate wave admission (owner decision 2026-09-05, rebased on #1487).
|
||||
|
||||
The scope reviewer — the only constitutionally blocking seat — reserves its
|
||||
budget FIRST, and the commit-gate wave (scope seats + triad seats) is admitted
|
||||
all-or-nothing against every fence the ledger enforces at reservation — the
|
||||
global TOTAL_BUDGET remainder, the task's current root and its original billing
|
||||
group — BEFORE any paid seat is dispatched. A wave that does not fit is a
|
||||
typed $0 pre-dispatch refusal naming the binding axis and the shortfall, never
|
||||
a half-dispatched panel (the 4 September paid run: two triad seats held the
|
||||
money, the third seat and the scope seat were refused mid-wave, and the commit
|
||||
blocked with ~$5 of the $8 fence never spent).
|
||||
budget FIRST. The commit-gate wave (scope seats + triad seats) is admitted while
|
||||
KNOWN spend is below every fence the ledger enforces at reservation — the global
|
||||
TOTAL_BUDGET, the task's current root and its original billing group — the
|
||||
reservation's own rule (owner Q4-A, 2026-10-03). The seats' summed reservation
|
||||
bounds are disclosed, never an earlier refusal: open holds of other in-flight
|
||||
attempts are exposure, not spending. A wave declined at admission (known spend
|
||||
already at a fence) is a typed $0 pre-dispatch refusal naming the binding axis.
|
||||
A fence reached mid-wave refuses the remaining seats at their own reservation,
|
||||
with truthful custody of what was sent (partial dispatch is accepted overshoot
|
||||
behaviour, not hidden).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -66,6 +68,28 @@ class LedgerLLM:
|
|||
return reply, {"prompt_tokens": 4, "completion_tokens": 2, "cost": 0.01}
|
||||
|
||||
|
||||
class PricedLedgerLLM(LedgerLLM):
|
||||
"""``LedgerLLM`` whose seats settle at chosen real prices."""
|
||||
|
||||
def __init__(self, costs, delays=None):
|
||||
super().__init__(delays)
|
||||
self.costs = dict(costs)
|
||||
|
||||
def chat(self, **kwargs):
|
||||
model = kwargs["model"]
|
||||
time.sleep(self.delays.get(model, 0.0))
|
||||
reply = {"content": _scope_matrix() if model == SCOPE_MODEL else "[]"}
|
||||
price = self.costs.get(model, 0.01)
|
||||
ua.execute_physical_attempt(
|
||||
ua.AttemptRequest(model=model, provider="test", reservation_usd=BOUNDS[model]),
|
||||
lambda: reply,
|
||||
extractor=lambda _r: ({"prompt_tokens": 4, "completion_tokens": 2}, price, True),
|
||||
)
|
||||
with self.lock:
|
||||
self.calls.append(model)
|
||||
return reply, {"prompt_tokens": 4, "completion_tokens": 2, "cost": price}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def gate(tmp_path, monkeypatch):
|
||||
root = tmp_path / "data"
|
||||
|
|
@ -147,21 +171,36 @@ def test_fence_that_fits_the_whole_wave_dispatches_every_seat(gate, tmp_path, mo
|
|||
assert not [e for e in ctx.pending_events if e.get("type") == "review_wave_budget_insufficient"]
|
||||
|
||||
|
||||
def test_fence_that_fits_only_the_triad_refuses_the_wave_at_zero_dollars(gate, tmp_path, monkeypatch):
|
||||
"""Scope $3 + triad $1 + $1 = $5 against a $4 fence: the triad alone would
|
||||
have fit (the 4 September failure shape), so the WHOLE wave is refused
|
||||
before dispatch — no seat reserved, no ledger row, every seat typed $0."""
|
||||
def test_a_wave_larger_than_the_known_remainder_is_admitted_whole(gate, tmp_path, monkeypatch):
|
||||
"""Scope $3 + triad $1 + $1 = $5 of worst case against a $4 fence with $0 known:
|
||||
the prospective sum is information, never an earlier refusal (#1487) — every seat
|
||||
is dispatched and each settles at its real price."""
|
||||
llm = LedgerLLM()
|
||||
ctx, (review_err, scope_result, _reason, _adv) = _run(gate, tmp_path, monkeypatch, llm, fence=4.0)
|
||||
|
||||
assert review_err is None, review_err
|
||||
assert scope_result.blocked is False and scope_result.status == "responded"
|
||||
assert sorted(llm.calls) == sorted([SCOPE_MODEL, *TRIAD_MODELS])
|
||||
assert sorted(row["state"] for row in _ledger(gate)) == ["settled"] * 3
|
||||
assert not [e for e in ctx.pending_events if e.get("type") == "review_wave_budget_insufficient"]
|
||||
|
||||
|
||||
def test_known_spend_at_the_fence_declines_the_wave_at_zero_dollars(gate, tmp_path, monkeypatch):
|
||||
"""Known spend already at the $4 fence: the WHOLE wave is declined before dispatch —
|
||||
no seat reserved, no new ledger row, every seat typed $0."""
|
||||
_seed_settled(gate, fence=4.0, cost=4.0)
|
||||
before = _ledger(gate)
|
||||
llm = LedgerLLM()
|
||||
ctx, (review_err, scope_result, block_reason, _adv) = _run(gate, tmp_path, monkeypatch, llm, fence=4.0)
|
||||
|
||||
assert llm.calls == [] and _ledger(gate) == []
|
||||
assert llm.calls == [] and _ledger(gate) == before
|
||||
assert review_err and "commit-gate review wave declined before dispatch ($0 spent)" in review_err
|
||||
assert "reservation upper bound $5.000000" in review_err
|
||||
assert "per-task budget fence $4.000000" in review_err
|
||||
assert "accounted=$0.000000 (of which $0.000000 is reserved by other in-flight attempts)" in review_err
|
||||
assert "remaining=$4.000000, shortfall=$1.000000" in review_err
|
||||
assert "Known spend has reached the per-task budget fence $4.000000" in review_err
|
||||
assert "known spend=$4.000000 (plus $0.000000 of open holds, not counted)" in review_err
|
||||
assert "remaining=$0.000000" in review_err
|
||||
assert "reservation upper bound would have been $5.000000" in review_err
|
||||
# The root axis binds; the global axis is disclosed beside it with its own remainder.
|
||||
assert "the global budget $100.000000 alone would leave $100.000000" in review_err
|
||||
assert "the global budget $100.000000 alone would leave $96.000000" in review_err
|
||||
assert "raise the per-task budget (OUROBOROS_PER_TASK_COST_USD)" in review_err
|
||||
# Every seat is named with its own bound, scope first.
|
||||
assert review_err.index("scope_review:scope_slot_1 scope/model $3.000000") < review_err.index(
|
||||
|
|
@ -178,31 +217,27 @@ def test_fence_that_fits_only_the_triad_refuses_the_wave_at_zero_dollars(gate, t
|
|||
assert events[0]["seats"] == ["scope_review:scope_slot_1", "multi_model_review:slot_1",
|
||||
"multi_model_review:slot_2"]
|
||||
assert events[0]["slot_bounds"] == [3.0, 1.0, 1.0]
|
||||
assert events[0]["binding_axis"] == "root" and events[0]["remaining_usd"] == 4.0
|
||||
assert (events[0]["global_limit_usd"], events[0]["global_remaining_usd"]) == (100.0, 100.0)
|
||||
assert events[0]["binding_axis"] == "root" and events[0]["remaining_usd"] == 0.0
|
||||
assert (events[0]["global_limit_usd"], events[0]["global_remaining_usd"]) == (100.0, 96.0)
|
||||
|
||||
|
||||
def test_global_budget_that_does_not_fit_refuses_the_wave_before_any_seat_reserves(gate, tmp_path, monkeypatch):
|
||||
"""rc.14 audit point 2 (verified repro): root fence $10 fits the $5 wave, the
|
||||
global TOTAL_BUDGET $100 has $4 left after $96 settled under ANOTHER root.
|
||||
Before the fix admission read only the root axis, two seats reserved and
|
||||
paid, and the ledger's global check — the FIRST one ``reserve_attempt``
|
||||
runs — refused the third mid-wave. Now the wave is refused before any seat
|
||||
reserves, and the refusal names the global axis and its own knob, never the
|
||||
per-task fence the wave would have fit."""
|
||||
def test_global_known_spend_at_the_limit_declines_the_wave_before_any_seat_reserves(gate, tmp_path, monkeypatch):
|
||||
"""rc.14 audit point 2, on the known-spend rule: the root fence $10 has room, the
|
||||
global TOTAL_BUDGET $100 is fully spent under ANOTHER root. The wave is declined
|
||||
before any seat reserves, and the refusal names the global axis and its own knob,
|
||||
never the per-task fence."""
|
||||
with ua.usage_scope(ua.UsageScope(drive_root=gate, task_id="other-root", root_task_id="other-root")):
|
||||
other = ua.reserve_attempt(ua.AttemptRequest(model="triad/a", provider="test", source="main"))
|
||||
ua.mark_dispatched(other)
|
||||
ua.settle_attempt(other, {"prompt_tokens": 1, "completion_tokens": 1}, cost_usd=96.0, cost_final=True)
|
||||
ua.settle_attempt(other, {"prompt_tokens": 1, "completion_tokens": 1}, cost_usd=100.0, cost_final=True)
|
||||
llm = LedgerLLM()
|
||||
ctx, (review_err, scope_result, block_reason, _adv) = _run(gate, tmp_path, monkeypatch, llm, fence=10.0)
|
||||
|
||||
assert llm.calls == [] and len(_ledger(gate)) == 1 # the other root's attempt only (one current row)
|
||||
assert review_err and "commit-gate review wave declined before dispatch ($0 spent)" in review_err
|
||||
assert "reservation upper bound $5.000000" in review_err
|
||||
assert "does not fit the global budget TOTAL_BUDGET $100.000000: accounted=$96.000000 across every task" in review_err
|
||||
assert "(of which $0.000000 is reserved by other in-flight attempts)" in review_err
|
||||
assert "remaining=$4.000000, shortfall=$1.000000" in review_err
|
||||
assert "Known spend has reached the global budget TOTAL_BUDGET $100.000000: known spend=$100.000000 across every task" in review_err
|
||||
assert "(plus $0.000000 of open holds, not counted)" in review_err
|
||||
assert "remaining=$0.000000" in review_err
|
||||
assert "the per-task budget fence $10.000000 alone would leave $10.000000" in review_err
|
||||
assert "raise TOTAL_BUDGET" in review_err and "OUROBOROS_PER_TASK_COST_USD" not in review_err
|
||||
assert block_reason == "review_wave_budget_insufficient"
|
||||
|
|
@ -210,9 +245,26 @@ def test_global_budget_that_does_not_fit_refuses_the_wave_before_any_seat_reserv
|
|||
assert scope_result.blocked is True and scope_result.status == "not_dispatched"
|
||||
events = [e for e in ctx.pending_events if e.get("type") == "review_wave_budget_insufficient"]
|
||||
assert len(events) == 1 and events[0]["binding_axis"] == "global"
|
||||
assert (events[0]["remaining_usd"], events[0]["global_remaining_usd"]) == (4.0, 4.0)
|
||||
assert (events[0]["global_limit_usd"], events[0]["global_accounted_usd"]) == (100.0, 96.0)
|
||||
assert (events[0]["limit_usd"], events[0]["accounted_usd"]) == (10.0, 0.0)
|
||||
assert (events[0]["remaining_usd"], events[0]["global_remaining_usd"]) == (0.0, 0.0)
|
||||
assert (events[0]["global_limit_usd"], events[0]["global_known_usd"]) == (100.0, 100.0)
|
||||
assert (events[0]["limit_usd"], events[0]["known_usd"]) == (10.0, 0.0)
|
||||
|
||||
|
||||
def test_a_fence_reached_mid_wave_refuses_the_remaining_seats_with_custody(gate, tmp_path, monkeypatch):
|
||||
"""The wave is admitted at $0 known; the scope seat settles at the full $4 fence
|
||||
before the (slower) triad seats reserve, so each triad seat is refused at its OWN
|
||||
reservation: nothing further is sent, and the outcome says what happened."""
|
||||
llm = PricedLedgerLLM(costs={SCOPE_MODEL: 4.0}, delays={"triad/a": 0.3, "triad/b": 0.3})
|
||||
ctx, (review_err, scope_result, _reason, _adv) = _run(gate, tmp_path, monkeypatch, llm, fence=4.0)
|
||||
|
||||
assert llm.calls == [SCOPE_MODEL]
|
||||
assert scope_result.status == "responded"
|
||||
rows = _ledger(gate)
|
||||
assert [(row["model"], row["state"]) for row in rows] == [(SCOPE_MODEL, "settled")]
|
||||
assert ua.usage_projection(gate, root_task_id=ROOT)["settled_usd"] == pytest.approx(4.0)
|
||||
triad = ctx._last_triad_raw_results
|
||||
assert len(triad) == 2 and all(r.get("status") != "ok" for r in triad), triad
|
||||
assert all("budget" in json.dumps(r, default=str).lower() for r in triad), triad
|
||||
|
||||
|
||||
def test_review_wave_admission_binds_on_the_tighter_of_root_and_global_axes(gate, monkeypatch):
|
||||
|
|
@ -223,26 +275,27 @@ def test_review_wave_admission_binds_on_the_tighter_of_root_and_global_axes(gate
|
|||
axis, and an internal exception keeps the fail-open skeleton."""
|
||||
wave = dict(root_task_id=ROOT, models=[SCOPE_MODEL, *TRIAD_MODELS], prompt_chars=10, task_id=ROOT)
|
||||
|
||||
# The $5 estimate exceeds both remainders, and the wave still fits: known room decides.
|
||||
root_bound = ua.review_wave_admission(gate, root_limit_usd=4.0, global_limit_usd=10.0, **wave)
|
||||
assert (root_bound["fits"], root_bound["binding_axis"], root_bound["estimated_wave_usd"]) == (False, "root", 5.0)
|
||||
assert (root_bound["fits"], root_bound["binding_axis"], root_bound["estimated_wave_usd"]) == (True, "root", 5.0)
|
||||
assert (root_bound["remaining_usd"], root_bound["global_remaining_usd"]) == (4.0, 10.0)
|
||||
global_bound = ua.review_wave_admission(gate, root_limit_usd=10.0, global_limit_usd=4.0, **wave)
|
||||
assert (global_bound["fits"], global_bound["binding_axis"]) == (False, "global")
|
||||
assert (global_bound["fits"], global_bound["binding_axis"]) == (True, "global")
|
||||
assert (global_bound["remaining_usd"], global_bound["limit_usd"], global_bound["global_limit_usd"]) == (
|
||||
4.0, 10.0, 4.0)
|
||||
assert (global_bound["global_accounted_usd"], global_bound["global_reserved_usd"]) == (0.0, 0.0)
|
||||
admitted = ua.review_wave_admission(gate, root_limit_usd=10.0, global_limit_usd=10.0, **wave)
|
||||
assert admitted["fits"] is True and admitted["remaining_usd"] == 10.0
|
||||
assert (global_bound["global_known_usd"], global_bound["global_reserved_usd"]) == (0.0, 0.0)
|
||||
no_room = ua.review_wave_admission(gate, root_limit_usd=0.0, global_limit_usd=10.0, **wave)
|
||||
assert (no_room["fits"], no_room["binding_axis"], no_room["remaining_usd"]) == (False, "root", 0.0)
|
||||
|
||||
monkeypatch.setenv("TOTAL_BUDGET", "0") # no finite global budget: the root axis alone
|
||||
root_only = ua.review_wave_admission(gate, root_limit_usd=4.0, **wave)
|
||||
assert (root_only["fits"], root_only["binding_axis"], root_only["remaining_usd"]) == (False, "root", 4.0)
|
||||
assert (root_only["fits"], root_only["binding_axis"], root_only["remaining_usd"]) == (True, "root", 4.0)
|
||||
assert (root_only["global_limit_usd"], root_only["global_remaining_usd"]) == (None, None)
|
||||
unfenced_unbounded = ua.review_wave_admission(gate, **wave) # no fence, no rows, no global budget
|
||||
assert unfenced_unbounded["fits"] is True and unfenced_unbounded["binding_axis"] is None
|
||||
monkeypatch.setenv("TOTAL_BUDGET", "4")
|
||||
unfenced = ua.review_wave_admission(gate, **wave) # no root fence: the global axis still binds
|
||||
assert (unfenced["fits"], unfenced["binding_axis"], unfenced["limit_usd"]) == (False, "global", None)
|
||||
assert (unfenced["fits"], unfenced["binding_axis"], unfenced["limit_usd"]) == (True, "global", None)
|
||||
assert (unfenced["remaining_usd"], unfenced["global_limit_usd"]) == (4.0, 4.0)
|
||||
|
||||
monkeypatch.setattr(ua, "usage_projection", lambda *_a, **_k: (_ for _ in ()).throw(RuntimeError("ledger")))
|
||||
|
|
@ -251,9 +304,9 @@ def test_review_wave_admission_binds_on_the_tighter_of_root_and_global_axes(gate
|
|||
assert skeleton["binding_axis"] is None and skeleton["global_remaining_usd"] is None
|
||||
|
||||
|
||||
def test_refusal_names_money_held_by_other_in_flight_attempts(gate, tmp_path, monkeypatch):
|
||||
"""The fence compares against settled + reserved holds: a refusal must say
|
||||
how much of `accounted` is a hold, not a bill."""
|
||||
def test_open_holds_alone_never_decline_the_wave(gate, tmp_path, monkeypatch):
|
||||
"""A $1 in-flight main-loop send and $2.50 known under an $8 fence: the hold is
|
||||
exposure, not a bill, so the wave is admitted and dispatched (#1487)."""
|
||||
with ua.usage_scope(ua.UsageScope(drive_root=gate, task_id=ROOT, root_task_id=ROOT, root_limit_usd=8.0)):
|
||||
held = ua.reserve_attempt(ua.AttemptRequest(model="triad/a", provider="test", source="main"))
|
||||
ua.mark_dispatched(held) # an in-flight main-loop send: $1 upper bound held
|
||||
|
|
@ -261,11 +314,27 @@ def test_refusal_names_money_held_by_other_in_flight_attempts(gate, tmp_path, mo
|
|||
ua.mark_dispatched(settled)
|
||||
ua.settle_attempt(settled, {"prompt_tokens": 1, "completion_tokens": 1}, cost_usd=2.5, cost_final=True)
|
||||
llm = LedgerLLM()
|
||||
_ctx, (review_err, scope_result, _reason, _adv) = _run(gate, tmp_path, monkeypatch, llm, fence=8.0)
|
||||
|
||||
assert review_err is None and scope_result.status == "responded"
|
||||
assert sorted(llm.calls) == sorted([SCOPE_MODEL, *TRIAD_MODELS])
|
||||
|
||||
|
||||
def test_refusal_names_open_holds_beside_known_spend(gate, tmp_path, monkeypatch):
|
||||
"""Known spend at the $8 fence with a $1 hold still in flight: the refusal says how
|
||||
much is known spend and how much is an uncounted hold, never one blended number."""
|
||||
with ua.usage_scope(ua.UsageScope(drive_root=gate, task_id=ROOT, root_task_id=ROOT, root_limit_usd=8.0)):
|
||||
held = ua.reserve_attempt(ua.AttemptRequest(model="triad/a", provider="test", source="main"))
|
||||
ua.mark_dispatched(held)
|
||||
settled = ua.reserve_attempt(ua.AttemptRequest(model="triad/b", provider="test", source="main"))
|
||||
ua.mark_dispatched(settled)
|
||||
ua.settle_attempt(settled, {"prompt_tokens": 1, "completion_tokens": 1}, cost_usd=8.0, cost_final=True)
|
||||
llm = LedgerLLM()
|
||||
ctx, (review_err, _scope, _reason, _adv) = _run(gate, tmp_path, monkeypatch, llm, fence=8.0)
|
||||
|
||||
assert llm.calls == []
|
||||
assert "accounted=$3.500000 (of which $1.000000 is reserved by other in-flight attempts)" in review_err
|
||||
assert "remaining=$4.500000, shortfall=$0.500000" in review_err
|
||||
assert "known spend=$8.000000 (plus $1.000000 of open holds, not counted)" in review_err
|
||||
assert "remaining=$0.000000" in review_err
|
||||
assert len(_ledger(gate)) == 2 # the two seeded attempts only (one current row each)
|
||||
|
||||
|
||||
|
|
@ -331,9 +400,11 @@ def test_review_wave_admission_prices_per_slot_and_discloses_holds(gate, monkeyp
|
|||
assert [(r.prompt_tokens_estimate, r.max_completion_tokens, r.task_id) for r in requests] == [
|
||||
(100, 1000, "fresh-root"), (400, 2000, "fresh-root"),
|
||||
]
|
||||
# No ledger row yet: the caller's bound fence is the limit, not a fail-open.
|
||||
assert admission["limit_usd"] == 0.75 and admission["accounted_usd"] == 0.0
|
||||
assert admission["estimated_wave_usd"] == 1.0 and admission["fits"] is False
|
||||
# No ledger row yet: the caller's bound fence is the limit, not a fail-open. The
|
||||
# $1.00 estimate above the $0.75 room is disclosed; known room decides (#1487).
|
||||
assert admission["limit_usd"] == 0.75 and admission["known_usd"] == 0.0
|
||||
assert admission["accounted_usd"] == 0.0
|
||||
assert admission["estimated_wave_usd"] == 1.0 and admission["fits"] is True
|
||||
assert admission["slot_bounds"] == [0.5, 0.5] and admission["reserved_usd"] == 0.0
|
||||
# A scalar still broadcasts (the task-level callers are unchanged), and a
|
||||
# root with neither a ledger row nor a bound fence keeps failing open.
|
||||
|
|
@ -511,10 +582,12 @@ def test_current_root_fence_governs_admission_over_the_ledgers_historical_minimu
|
|||
kwargs = dict(root_task_id=ROOT, models=["triad/a"], prompt_chars=10, task_id=ROOT)
|
||||
|
||||
raised = ua.review_wave_admission(gate, root_limit_usd=50.0, **kwargs)
|
||||
assert (raised["limit_usd"], raised["accounted_usd"], raised["remaining_usd"]) == (50.0, 3.0, 47.0)
|
||||
assert (raised["limit_usd"], raised["known_usd"], raised["remaining_usd"]) == (50.0, 3.0, 47.0)
|
||||
assert raised["fits"] is True
|
||||
lowered = ua.review_wave_admission(gate, root_limit_usd=6.0, **kwargs)
|
||||
assert (lowered["limit_usd"], lowered["remaining_usd"], lowered["fits"]) == (6.0, 3.0, False)
|
||||
assert (lowered["limit_usd"], lowered["remaining_usd"], lowered["fits"]) == (6.0, 3.0, True)
|
||||
at_known = ua.review_wave_admission(gate, root_limit_usd=3.0, **kwargs) # lowered to the known spend
|
||||
assert (at_known["limit_usd"], at_known["remaining_usd"], at_known["fits"]) == (3.0, 0.0, False)
|
||||
unfenced = ua.review_wave_admission(gate, **kwargs) # no fence of its own: the ledger's $8 row
|
||||
assert (unfenced["limit_usd"], unfenced["remaining_usd"], unfenced["fits"]) == (8.0, 5.0, True)
|
||||
|
||||
|
|
@ -549,8 +622,8 @@ def test_scope_first_hold_observes_only_the_scope_seats_own_reservation(gate, tm
|
|||
ua.refresh_root_accounting(gate, ROOT)
|
||||
with ua.usage_scope(base):
|
||||
ua.settle_attempt(earlier, {"prompt_tokens": 1, "completion_tokens": 1}, cost_usd=0.5, cost_final=True)
|
||||
with ua.usage_scope(ua.UsageScope(drive_root=gate, task_id=ROOT, root_task_id=ROOT, root_limit_usd=0.75)):
|
||||
with pytest.raises(ua.BudgetExceeded): # refused: pre-fence refresh only, no identity
|
||||
with ua.usage_scope(ua.UsageScope(drive_root=gate, task_id=ROOT, root_task_id=ROOT, root_limit_usd=0.5)):
|
||||
with pytest.raises(ua.BudgetExceeded): # refused (known $0.50 = fence): pre-fence refresh only
|
||||
ua.reserve_attempt(ua.AttemptRequest(model="triad/a", provider="test", source="main"))
|
||||
with ua.usage_scope(triad_seat):
|
||||
ua.reserve_attempt(ua.AttemptRequest(model="triad/a", provider="test"))
|
||||
|
|
@ -635,10 +708,10 @@ def test_seats_reserve_against_exactly_the_fence_the_wave_was_admitted_with(gate
|
|||
def test_raised_root_without_group_amendment_refuses_the_whole_wave(gate, tmp_path, monkeypatch):
|
||||
"""#1599: durable group8 beats an in-memory root50, before any seat sends.
|
||||
|
||||
This fails on the old admission reader without relying on settlement order:
|
||||
reservations recover group8, but the old wave precheck admitted against50.
|
||||
Known spend has reached the original group's $8: the wave is declined although
|
||||
the in-memory root fence $50 has room.
|
||||
"""
|
||||
_seed_settled(gate, fence=8.0, cost=4.0)
|
||||
_seed_settled(gate, fence=8.0, cost=8.0)
|
||||
before = _ledger(gate)
|
||||
llm = LedgerLLM()
|
||||
ctx, (review_err, scope_result, reason, _adv) = _run(
|
||||
|
|
@ -646,7 +719,7 @@ def test_raised_root_without_group_amendment_refuses_the_whole_wave(gate, tmp_pa
|
|||
|
||||
assert llm.calls == [] and _ledger(gate) == before
|
||||
assert reason == "review_wave_budget_insufficient" and scope_result.status == "not_dispatched"
|
||||
assert review_err and "remaining=$4.000000, shortfall=$1.000000" in review_err
|
||||
assert review_err and "remaining=$0.000000" in review_err
|
||||
event = next(e for e in ctx.pending_events if e.get("type") == "review_wave_budget_insufficient")
|
||||
assert event["binding_axis"] == "group" and event["limit_usd"] == 8.0
|
||||
assert "whole-work billing-group budget fence" in review_err
|
||||
|
|
@ -673,22 +746,22 @@ def test_wave_group_resolution_matches_reservation_without_pinning_authority(
|
|||
models=[SCOPE_MODEL, *TRIAD_MODELS], prompt_chars=10)
|
||||
assert resolved.billing_group_id == "original-root"
|
||||
assert resolved.billing_group_limit_usd == group_limit
|
||||
assert admission["fits"] is (group_limit is None or group_limit >= 5.0)
|
||||
assert admission["fits"] is (group_limit is None or group_limit > 0.0) # known $0 decides
|
||||
assert load_task_result(gate, ROOT) == before and _ledger(gate) == []
|
||||
|
||||
|
||||
def test_bound_fence_that_does_not_fit_refuses_even_when_the_environment_is_roomier(gate, tmp_path, monkeypatch):
|
||||
"""The converse: bound $8 (admission's fence), environment $50, a $4 history
|
||||
row: the $5 wave does not fit the bound fence and nothing is dispatched —
|
||||
the roomier setting never buys a seat the admitting fence refused."""
|
||||
_seed_settled(gate, fence=8.0, cost=4.0)
|
||||
"""The converse: bound $8 (admission's fence), environment $50, $8 of known
|
||||
history: the bound fence is reached and nothing is dispatched — the roomier
|
||||
setting never buys a seat the admitting fence refused."""
|
||||
_seed_settled(gate, fence=8.0, cost=8.0)
|
||||
llm = LedgerLLM()
|
||||
ctx, (review_err, scope_result, block_reason, _adv) = _run(
|
||||
gate, tmp_path, monkeypatch, llm, fence=8.0, env_fence=50.0)
|
||||
|
||||
assert llm.calls == [] and len(_ledger(gate)) == 1 # the seeded attempt only (one current row)
|
||||
assert review_err and "per-task budget fence $8.000000" in review_err
|
||||
assert "remaining=$4.000000, shortfall=$1.000000" in review_err
|
||||
assert "remaining=$0.000000" in review_err
|
||||
assert block_reason == "review_wave_budget_insufficient"
|
||||
assert scope_result.status == "not_dispatched"
|
||||
|
||||
|
|
|
|||
|
|
@ -93,9 +93,12 @@ def test_authored_stop_money_check_uses_real_cap_not_planning_margin(monkeypatch
|
|||
monkeypatch.setattr('ouroboros.loop_budget._wrapup_global_remaining', lambda: 10)
|
||||
ctx = SimpleNamespace(active_use_local=False, accumulated_usage={'cost': 1.0})
|
||||
cap = SimpleNamespace(root_cap_usd=10.0, ceiling_usd=7.0)
|
||||
monkeypatch.setattr(loop, '_loop_tree_accounting', lambda **kw: {'accounted_usd': 8.0})
|
||||
monkeypatch.setattr(loop, '_loop_tree_accounting', lambda **kw: {'settled_usd': 8.0, 'accounted_usd': 8.0})
|
||||
assert authored_completion_budget_exhausted(ctx, 10, cap) is False
|
||||
monkeypatch.setattr(loop, '_loop_tree_accounting', lambda **kw: {'accounted_usd': 10.0})
|
||||
# Open holds are not spending (#1487): $8 known with $5 in flight is not exhaustion.
|
||||
monkeypatch.setattr(loop, '_loop_tree_accounting', lambda **kw: {'settled_usd': 8.0, 'accounted_usd': 13.0})
|
||||
assert authored_completion_budget_exhausted(ctx, 10, cap) is False
|
||||
monkeypatch.setattr(loop, '_loop_tree_accounting', lambda **kw: {'settled_usd': 10.0, 'accounted_usd': 10.0})
|
||||
assert authored_completion_budget_exhausted(ctx, 10, cap) is True
|
||||
monkeypatch.setattr('ouroboros.loop_budget._wrapup_global_remaining', lambda: 0)
|
||||
assert authored_completion_budget_exhausted(ctx, 10, None) is True
|
||||
|
|
|
|||
|
|
@ -28,8 +28,8 @@ T0 = 1_800_000_000.0
|
|||
def _iso(ts):
|
||||
return clock_module._iso(ts)
|
||||
FLOOR, CEILING, DEFAULT = 900, 14400, 3300
|
||||
AVAILABLE = {"status": "available", "limit_usd": 20.0, "accounted_usd": 2.5, "remaining_usd": 17.5,
|
||||
"resets_at": "", "unknown_unmetered": 0}
|
||||
AVAILABLE = {"status": "available", "limit_usd": 20.0, "settled_usd": 2.5, "accounted_usd": 2.5,
|
||||
"remaining_usd": 17.5, "resets_at": "", "unknown_unmetered": 0}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
|
|
@ -494,13 +494,18 @@ def test_launch_cap_is_the_remaining_allowance_when_no_per_task_cap(clock, monke
|
|||
assert clock.launches[0]["metadata"]["root_cost_ceiling_usd"] == 17.5
|
||||
|
||||
|
||||
def test_less_than_one_planned_turn_left_is_exhausted(clock, monkeypatch):
|
||||
"""A remainder at or below the graceful stop's planning margin would only wake the
|
||||
mind to be told to land at once: the tick skips it as exhausted instead."""
|
||||
from ouroboros.task_pacing import COST_PLANNING_MARGIN_USD
|
||||
|
||||
thin = dict(AVAILABLE, remaining_usd=COST_PLANNING_MARGIN_USD, accounted_usd=20.0 - COST_PLANNING_MARGIN_USD)
|
||||
def test_a_thin_allowance_still_wakes_and_only_known_exhaustion_skips(clock, monkeypatch):
|
||||
"""Owner 2026-10-07: no extra $3 margin skip. $0.50 of known allowance left is a
|
||||
wake under a $0.50 producer ceiling; only known spend at the limit skips."""
|
||||
thin = dict(AVAILABLE, remaining_usd=0.5, settled_usd=19.5, accounted_usd=60.0)
|
||||
monkeypatch.setattr(clock_module, "allowance_window", lambda root, now=None, **_display_read: dict(thin))
|
||||
assert clock.clock.tick(T0 + FLOOR + 1) == "launched"
|
||||
assert clock.launches[-1]["metadata"]["root_cost_ceiling_usd"] == 0.5
|
||||
clock.launches.clear()
|
||||
clock.clock._last_wake_task_id, clock.clock._next_wake_at = "", 0.0
|
||||
clock.live = None
|
||||
spent = dict(AVAILABLE, status="exhausted", remaining_usd=0.0, settled_usd=20.0, accounted_usd=20.0)
|
||||
monkeypatch.setattr(clock_module, "allowance_window", lambda root, now=None, **_display_read: dict(spent))
|
||||
assert clock.clock.tick(T0 + FLOOR + 1) == "skipped:allowance_exhausted"
|
||||
assert clock.launches == []
|
||||
# On an exhausted day every root completion would otherwise pull the clock to "now" and cost a
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@ import pytest
|
|||
|
||||
from ouroboros import consciousness_authority as ca
|
||||
|
||||
AVAILABLE = {"status": "available", "limit_usd": 20.0, "accounted_usd": 4.0, "remaining_usd": 16.0,
|
||||
AVAILABLE = {"status": "available", "limit_usd": 20.0, "settled_usd": 4.0, "accounted_usd": 4.0, "remaining_usd": 16.0,
|
||||
"unknown_unmetered": 0, "resets_at": ""}
|
||||
|
||||
|
||||
|
|
@ -78,15 +78,18 @@ def test_max_tasks_zero_means_consciousness_never_starts_tasks(door, monkeypatch
|
|||
def test_an_exhausted_allowance_refuses_with_the_window_facts(door, monkeypatch):
|
||||
queue, pending, _running = door
|
||||
monkeypatch.setattr("ouroboros.consciousness_allowance.allowance_window", lambda root, now=None: {
|
||||
**AVAILABLE, "status": "exhausted", "accounted_usd": 21.5, "remaining_usd": 0.0,
|
||||
**AVAILABLE, "status": "exhausted", "settled_usd": 21.5, "accounted_usd": 25.5, "remaining_usd": 0.0,
|
||||
"unknown_unmetered": 1, "resets_at": "2026-09-17T08:00:00+00:00"})
|
||||
refused = queue.enqueue_task(_root("c1"))
|
||||
assert refused["_admission_blocked"] == "consciousness_allowance_exhausted"
|
||||
# Known spend decides and is called spent; open holds are shown beside it, uncounted (#1487).
|
||||
assert refused["_admission_detail"] == (
|
||||
"$21.50 (at least) of $20.00 spent in the last 24 h; resets at 2026-09-17T08:00:00+00:00")
|
||||
"$21.50 (at least) of $20.00 known spend in the last 24 h (plus $4.00 of open holds, "
|
||||
"not counted); resets at 2026-09-17T08:00:00+00:00")
|
||||
assert pending == []
|
||||
monkeypatch.setattr("ouroboros.consciousness_allowance.allowance_window", lambda root, now=None: {
|
||||
**AVAILABLE, "status": "exhausted", "limit_usd": 0.0, "accounted_usd": 0.0, "remaining_usd": 0.0})
|
||||
**AVAILABLE, "status": "exhausted", "limit_usd": 0.0, "settled_usd": 0.0, "accounted_usd": 0.0,
|
||||
"remaining_usd": 0.0})
|
||||
zero = queue.enqueue_task(_root("c2"))
|
||||
assert zero["_admission_blocked"] == "consciousness_allowance_exhausted"
|
||||
assert "OUROBOROS_CONSCIOUSNESS_DAILY_USD=0" in zero["_admission_detail"]
|
||||
|
|
|
|||
|
|
@ -235,11 +235,31 @@ def test_a_known_reservation_bound_stays_accounted_while_the_price_is_unknown(da
|
|||
ua.mark_dispatched(reservation)
|
||||
ua.mark_unresolved(reservation, "the provider went dark")
|
||||
window = allowance.allowance_window(data_root, now=T0)
|
||||
assert window["accounted_usd"] == pytest.approx(5.0) # 1.0 settled + the 4.0 bound
|
||||
assert window["remaining_usd"] == pytest.approx(15.0)
|
||||
assert window["accounted_usd"] == pytest.approx(5.0) # 1.0 settled + the 4.0 bound, disclosed
|
||||
assert window["settled_usd"] == pytest.approx(1.0)
|
||||
assert window["remaining_usd"] == pytest.approx(19.0) # known spend decides (#1487)
|
||||
assert window["non_final_rows"] == 1
|
||||
|
||||
|
||||
def test_a_refused_oversized_request_never_closes_the_allowance_for_a_day(data_root, monkeypatch):
|
||||
"""The 2026-10-05 incident (#1487 comment): $31.68 known of a $50 allowance, then a
|
||||
provider-refused fallback recorded `unresolved` at its $41.27 bound. Its worst case is
|
||||
exposure beside the known spend, never spending: the next wake is admitted with the
|
||||
known $18.32 left, instead of 21 hours of `allowance_exhausted`."""
|
||||
monkeypatch.setenv("OUROBOROS_CONSCIOUSNESS_DAILY_USD", "50")
|
||||
_wake(data_root, monkeypatch, T0 - 2 * HOUR, 31.68)
|
||||
_at(monkeypatch, T0 - HOUR)
|
||||
refused = ua.reserve_attempt(_request(data_root, reservation_usd=41.27, task_id="wake-1",
|
||||
root_task_id="wake-1", category="consciousness"))
|
||||
ua.mark_dispatched(refused)
|
||||
ua.mark_unresolved(refused, "HTTP 400 total text input size exceeds 8 MB")
|
||||
window = allowance.allowance_window(data_root, now=T0)
|
||||
assert window["status"] == allowance.STATUS_AVAILABLE
|
||||
assert window["settled_usd"] == pytest.approx(31.68)
|
||||
assert window["remaining_usd"] == pytest.approx(18.32)
|
||||
assert window["accounted_usd"] == pytest.approx(72.95) # the open bound stays disclosed
|
||||
|
||||
|
||||
def test_a_positive_charge_and_a_descendants_charge_both_land_on_the_one_allowance(data_root, monkeypatch):
|
||||
"""The wake, the root it started and that root's descendant share one window."""
|
||||
monkeypatch.setenv("OUROBOROS_CONSCIOUSNESS_DAILY_USD", "20")
|
||||
|
|
@ -323,3 +343,27 @@ def test_the_wake_prompt_money_framing_matches_what_the_ledger_actually_records(
|
|||
assert session["subscription_sessions"] == 1
|
||||
assert session["subscription_windows"] == {"r": "2026-09-17T00:00:00Z"}
|
||||
assert "limit_usd" not in session
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("settled", "unresolved", "status", "remaining"), [
|
||||
((2.0, True), 20.0, allowance.STATUS_AVAILABLE, 8.0), # $2 known beside a $20 unknown
|
||||
((9.0, False), None, allowance.STATUS_AVAILABLE, 1.0), # a $9 estimate is known spend
|
||||
((10.0, False), None, allowance.STATUS_EXHAUSTED, 0.0), # equality: the allowance is reached
|
||||
])
|
||||
def test_the_daily_allowance_decides_on_known_spend(data_root, monkeypatch, settled, unresolved, status, remaining):
|
||||
"""The same rule as every money limit (#1487): known spend (estimates included)
|
||||
decides; an unresolved bound is disclosed exposure."""
|
||||
monkeypatch.setenv("OUROBOROS_CONSCIOUSNESS_DAILY_USD", "10")
|
||||
cost, final = settled
|
||||
_at(monkeypatch, T0 - 2 * HOUR)
|
||||
_settle(data_root, cost=cost, cost_final=final, task_id="wake-1", root_task_id="wake-1", category="consciousness")
|
||||
if unresolved is not None:
|
||||
_at(monkeypatch, T0 - HOUR)
|
||||
held = ua.reserve_attempt(_request(data_root, reservation_usd=unresolved, task_id="wake-1",
|
||||
root_task_id="wake-1", category="consciousness"))
|
||||
ua.mark_dispatched(held)
|
||||
ua.mark_unresolved(held, "provider outcome unknown")
|
||||
window = allowance.allowance_window(data_root, now=T0)
|
||||
assert window["status"] == status and window["remaining_usd"] == pytest.approx(remaining)
|
||||
assert window["settled_usd"] == pytest.approx(cost)
|
||||
assert window["accounted_usd"] == pytest.approx(cost + (unresolved or 0.0))
|
||||
|
|
|
|||
|
|
@ -718,7 +718,7 @@ def test_deep_review_request_carries_the_origin_to_the_one_door(tmp_path, monkey
|
|||
monkeypatch.setattr("supervisor.workers._worker_pool_execution_state",
|
||||
lambda: {"available": True, "disabled_reason": ""})
|
||||
monkeypatch.setattr("ouroboros.consciousness_allowance.allowance_window",
|
||||
lambda root, now=None: {"status": "available", "limit_usd": 20.0, "accounted_usd": 0.0,
|
||||
lambda root, now=None: {"status": "available", "limit_usd": 20.0, "settled_usd": 0.0, "accounted_usd": 0.0,
|
||||
"remaining_usd": 20.0, "unknown_unmetered": 0, "resets_at": ""})
|
||||
handed: list = []
|
||||
sup = types.SimpleNamespace(queue_deep_self_review_task=lambda **kw: handed.append(kw))
|
||||
|
|
@ -749,7 +749,7 @@ def test_the_allowance_is_read_before_the_queue_lock(tmp_path, monkeypatch):
|
|||
|
||||
def _window(root, now=None):
|
||||
seen.append(queue._queue_lock._is_owned())
|
||||
return {"status": "available", "limit_usd": 20.0, "accounted_usd": 0.0, "remaining_usd": 20.0,
|
||||
return {"status": "available", "limit_usd": 20.0, "settled_usd": 0.0, "accounted_usd": 0.0, "remaining_usd": 20.0,
|
||||
"unknown_unmetered": 0, "resets_at": ""}
|
||||
|
||||
monkeypatch.setattr("ouroboros.consciousness_allowance.allowance_window", _window)
|
||||
|
|
@ -796,7 +796,7 @@ def test_campaign_keeps_the_origin_and_its_cycle_tasks_inherit_it(tmp_path, monk
|
|||
monkeypatch.setattr(queue, "send_with_budget", lambda *a, **k: None)
|
||||
monkeypatch.setattr(queue, "persist_queue_snapshot", lambda reason="": None)
|
||||
monkeypatch.setattr("ouroboros.consciousness_allowance.allowance_window",
|
||||
lambda root, now=None: {"status": "available", "limit_usd": 20.0, "accounted_usd": 0.0,
|
||||
lambda root, now=None: {"status": "available", "limit_usd": 20.0, "settled_usd": 0.0, "accounted_usd": 0.0,
|
||||
"remaining_usd": 20.0, "unknown_unmetered": 0, "resets_at": ""})
|
||||
queue.enqueue_evolution_task_if_needed()
|
||||
assert len(pending) == 1
|
||||
|
|
@ -811,7 +811,7 @@ def test_campaign_keeps_the_origin_and_its_cycle_tasks_inherit_it(tmp_path, monk
|
|||
sent: list = []
|
||||
monkeypatch.setattr(queue, "send_with_budget", lambda cid, text, **kw: sent.append(text))
|
||||
monkeypatch.setattr("ouroboros.consciousness_allowance.allowance_window",
|
||||
lambda root, now=None: {"status": "exhausted", "limit_usd": 20.0, "accounted_usd": 21.0,
|
||||
lambda root, now=None: {"status": "exhausted", "limit_usd": 20.0, "settled_usd": 21.0, "accounted_usd": 21.0,
|
||||
"remaining_usd": 0.0, "unknown_unmetered": 0, "resets_at": "2027-01-01T00:00:00+00:00"})
|
||||
queue.enqueue_evolution_task_if_needed()
|
||||
queue.enqueue_evolution_task_if_needed()
|
||||
|
|
|
|||
|
|
@ -532,7 +532,7 @@ def test_render_substitutes_every_placeholder_and_keeps_every_event(tmp_path, mo
|
|||
assert text.startswith("You are Ouroboros. No one has asked for a task")
|
||||
assert "1 h 30 min ago" in text and "autonomy: act — everything your runtime mode allows except" in text
|
||||
assert "toggle_evolution, request_restart" in text
|
||||
assert "allowance accounting (last 24 h): 4.00 / 20.00 USD" in text and "tasks running: 1/2" in text
|
||||
assert "allowance known spend (last 24 h): 4.00 / 20.00 USD" in text and "tasks running: 1/2" in text
|
||||
assert "next interval: 3300 s" in text
|
||||
assert "- wake cause: task t14 finished (completed)" in text
|
||||
assert text.count("- task t") == 15 and "more; see" not in text # the trigger's own terminal stays an event
|
||||
|
|
@ -541,7 +541,7 @@ def test_render_substitutes_every_placeholder_and_keeps_every_event(tmp_path, mo
|
|||
disabled_tools=[], spent_usd=None, daily_usd=0, running=0, max_tasks=0, interval=900,
|
||||
events=_observe(tmp_path / "empty", since=T0 + 1).full_text())
|
||||
assert "no wake since this process started" in quiet and "wake cause: scheduled heartbeat" in quiet
|
||||
assert "unavailable tools: none" in quiet and "allowance accounting (last 24 h): unknown / 0.00 USD" in quiet
|
||||
assert "unavailable tools: none" in quiet and "allowance known spend (last 24 h): unknown / 0.00 USD" in quiet
|
||||
assert "including evolution" in quiet
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -124,10 +124,14 @@ def test_public_api_is_stable():
|
|||
def test_budget_profile_frozen_key_set():
|
||||
"""§11.1 additive ABI pin (v6.56.0): the normalized budget_profile key set.
|
||||
|
||||
``cost_hard_stop_pct`` is the additive in-task cost hard-stop knob
|
||||
(None -> historical 50%-of-remaining stop; 0 -> no in-task stop, never a
|
||||
$0 ceiling). Removing or renaming any key here is a deliberate ABI break.
|
||||
``cost_hard_stop_pct`` is the additive in-task cost hard-stop knob of an
|
||||
EXPLICIT experiment profile. The key and its accepted inputs are frozen; the
|
||||
meaning of None changed by owner decision 2026-10-07 (#1128): None is no
|
||||
explicit stop — not the old 50%-of-remaining default — and 0 stays the
|
||||
explicit no-stop bench contract, never a $0 ceiling. Removing or renaming
|
||||
any key here is a deliberate ABI break.
|
||||
"""
|
||||
from ouroboros import task_pacing
|
||||
from ouroboros.contracts.task_contract import normalize_budget_profile
|
||||
|
||||
profile = normalize_budget_profile(None)
|
||||
|
|
@ -141,6 +145,11 @@ def test_budget_profile_frozen_key_set():
|
|||
assert profile["cost_hard_stop_pct"] is None
|
||||
assert normalize_budget_profile({"cost_hard_stop_pct": 0})["cost_hard_stop_pct"] == 0
|
||||
assert normalize_budget_profile({"cost_hard_stop_pct": "37"})["cost_hard_stop_pct"] == 37
|
||||
# The semantic half of the pin: None resolves to no stop, explicit 37 to its authored share.
|
||||
ordinary = task_pacing.resolve_cost_ceiling(100.0, profile, root_cap_usd=400.0)
|
||||
assert (ordinary.state, ordinary.basis) == ("disabled", task_pacing.COST_BASIS_NO_DEFAULT_STOP)
|
||||
authored = task_pacing.resolve_cost_ceiling(100.0, normalize_budget_profile({"cost_hard_stop_pct": "37"}))
|
||||
assert (authored.state, authored.ceiling_usd) == ("active", 37.0)
|
||||
|
||||
|
||||
def test_task_contract_preserves_protected_artifact_policy():
|
||||
|
|
|
|||
|
|
@ -223,15 +223,24 @@ def run_review_change(ctx, **arguments):
|
|||
return reserve_attempt(AttemptRequest(model="synthetic", provider="test", task_id="probe",
|
||||
root_task_id="probe", reservation_usd=usd))
|
||||
|
||||
spent = admit(1.0)
|
||||
mark_dispatched(spent)
|
||||
settle_attempt(spent, cost_usd=1.0, cost_final=True)
|
||||
for label, usd in (("hold_2_5", 2.5), ("over_cap", 10.0)):
|
||||
def spend(usd):
|
||||
held = admit(usd)
|
||||
mark_dispatched(held)
|
||||
settle_attempt(held, cost_usd=usd, cost_final=True)
|
||||
|
||||
def probe(label, step):
|
||||
try:
|
||||
release_attempt(admit(usd))
|
||||
step()
|
||||
report[label] = "admitted"
|
||||
except BudgetExceeded as exc:
|
||||
report[label] = "refused: " + str(exc)
|
||||
|
||||
probe("spend_1", lambda: spend(1.0))
|
||||
# Known spend decides (#1487): a hold above the remainder is exposure, not spending.
|
||||
probe("hold_10", lambda: release_attempt(admit(10.0)))
|
||||
if report["spend_1"] == "admitted":
|
||||
spend(3.0) # known spend reaches the explicit cap
|
||||
probe("at_cap", lambda: release_attempt(admit(0.01)))
|
||||
slots = {row.slot_id: row for row in load_reviewer_slot_config().triad}
|
||||
report["configured_triad"] = sorted(slots)
|
||||
record_reviewer_slot_executions(
|
||||
|
|
@ -346,8 +355,9 @@ def test_full_entrypoint_keeps_a_legacy_host_untouched(tmp_path, engine):
|
|||
assert report["saved_total_budget"] == 500.0
|
||||
assert report["global_limit_usd"] == 4.0
|
||||
assert (report["ledger_at_start"]["accounted_usd"], report["ledger_at_start"]["limit_usd"]) == (0.0, 4.0)
|
||||
assert report["hold_2_5"] == "admitted" # $1 spent + $2.50 held fits $4
|
||||
assert report["over_cap"].startswith("refused")
|
||||
assert report["spend_1"] == "admitted"
|
||||
assert report["hold_10"] == "admitted" # $1 known: a $10 hold is exposure, not spending
|
||||
assert report["at_cap"].startswith("refused") # known $4 reached the $4 cap, not the saved $500
|
||||
assert (drive / "state" / "usage.sqlite").is_file()
|
||||
sys.path.insert(0, str(REPO))
|
||||
from ouroboros import usage_store
|
||||
|
|
@ -356,8 +366,8 @@ def test_full_entrypoint_keeps_a_legacy_host_untouched(tmp_path, engine):
|
|||
return [row for row in usage_store.read_usage_records(drive)
|
||||
if row.get("task_id") == "probe" and row.get("state") == state]
|
||||
|
||||
assert [float(row["cost_usd"]) for row in probe_rows("settled")] == [1.0]
|
||||
assert len(probe_rows("released")) == 1 # the $2.50 hold, admitted then let go
|
||||
assert [float(row["cost_usd"]) for row in probe_rows("settled")] == [1.0, 3.0]
|
||||
assert len(probe_rows("released")) == 1 # the $10 hold, admitted then let go
|
||||
# The configured rows came from the host settings; the marker landed in the drive.
|
||||
assert report["configured_triad"] == ["t1"]
|
||||
marker = json.loads((drive / "state" / "reviewer_slot_last_execution.json").read_text(encoding="utf-8"))
|
||||
|
|
@ -400,14 +410,14 @@ def test_full_entrypoint_keeps_a_legacy_host_untouched(tmp_path, engine):
|
|||
host_values = (_HOST_PROVIDER_VALUE.encode(), _TOKEN.encode())
|
||||
assert len(outputs) > 3 and not [blob for blob in outputs if any(value in blob for value in host_values)]
|
||||
|
||||
# Continuation with the SAME cap on the same drive: the spend already
|
||||
# recorded counts, so the hold that fit before no longer does.
|
||||
# Continuation with the SAME cap on the same drive: the known spend already
|
||||
# recorded counts, so the spend that fit before no longer does.
|
||||
again = _review(repo, host, out, "--run-cap-usd=4", "--attach-host-engine")
|
||||
assert again.returncode == 3, again.stderr[-4000:]
|
||||
second = json.loads((out / "probe.json").read_text(encoding="utf-8"))
|
||||
assert (second["ledger_at_start"]["accounted_usd"], second["global_limit_usd"]) == (1.0, 4.0)
|
||||
assert second["hold_2_5"].startswith("refused") # $2 spent + $2.50 > $4
|
||||
assert [float(row["cost_usd"]) for row in probe_rows("settled")] == [1.0, 1.0]
|
||||
assert (second["ledger_at_start"]["settled_usd"], second["global_limit_usd"]) == (4.0, 4.0)
|
||||
assert second["spend_1"].startswith("refused") and second["hold_10"].startswith("refused")
|
||||
assert [float(row["cost_usd"]) for row in probe_rows("settled")] == [1.0, 3.0]
|
||||
# A continuation never changes the cap it was opened with.
|
||||
changed = _review(repo, host, out, "--run-cap-usd=9", "--attach-host-engine")
|
||||
assert changed.returncode == 3 and "keeps that cap" in changed.stderr
|
||||
|
|
|
|||
183
tests/test_cost_stop_cold_restore.py
Normal file
183
tests/test_cost_stop_cold_restore.py
Normal file
|
|
@ -0,0 +1,183 @@
|
|||
"""A graceful budget pause saved before the upgrade resumes under its real authority (#1128).
|
||||
|
||||
Owner 2026-10-07: ordinary tasks on new AND existing installations lose the default
|
||||
early stop (half the wallet, the margin before the cap); explicit experiment profiles
|
||||
keep theirs; a producer allowance stays a real restriction at its actual value; an
|
||||
explicit Resume never widens what was not authored. Each case drives the real pause,
|
||||
the real exact-Resume grant and ``budget_pause.resume_paused_loop``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import asdict
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from ouroboros import task_pacing
|
||||
from ouroboros import usage_accounting as accounting
|
||||
from tests import _budget_pause_exact_helpers as helpers
|
||||
from tests._budget_pause_exact_helpers import _install_queue
|
||||
|
||||
|
||||
def _resume(tmp_path, monkeypatch, task_id, ceiling, *, contract=None, scope=None, ledger=None):
|
||||
"""Pause on the graceful-ceiling rail with ``ceiling`` saved, grant, and resume."""
|
||||
from ouroboros import budget_pause, loop_budget, owner_wait
|
||||
from supervisor import workers
|
||||
|
||||
queue, state, _workers = _install_queue(tmp_path, monkeypatch)
|
||||
monkeypatch.setattr(state, "budget_remaining", lambda _st, **_k: 500.0)
|
||||
original = helpers._loop_ctx
|
||||
|
||||
def with_saved_policy(*args, **kwargs):
|
||||
context, limits = original(*args, **kwargs)
|
||||
context._cost_ceiling = ceiling
|
||||
if contract is not None:
|
||||
context.task_contract = contract
|
||||
return context, limits
|
||||
|
||||
monkeypatch.setattr(helpers, "_loop_ctx", with_saved_policy)
|
||||
_ctx, _limit, _pause_row = helpers._pause(tmp_path, monkeypatch, task_id=task_id,
|
||||
rail=budget_pause.RAIL_GRACEFUL_CEILING)
|
||||
budget_pause.end_dispatch_fence(task_id)
|
||||
row = budget_pause.budget_pause_row(tmp_path, task_id)
|
||||
marker = budget_pause.exact_pause_marker(row, default_root=task_id)
|
||||
task = {"id": task_id, "type": "task", "chat_id": 0, "root_task_id": task_id, "_attempt": 1,
|
||||
"_budget_pause": marker}
|
||||
workers.PENDING.append(task)
|
||||
budget_pause.set_budget_pause(tmp_path, task_id, {**row, "state": budget_pause.STATE_PAUSED})
|
||||
assert queue.resume_budget_paused_task(task_id)["ok"] is True
|
||||
ctx, _limit = with_saved_policy(tmp_path, task_id)
|
||||
ctx._cost_ceiling = None # a fresh process: nothing restored yet
|
||||
ctx.budget_pause_resume = task["_budget_pause_resume"]
|
||||
state_blob = budget_pause.load_budget_pause(ctx)
|
||||
assert state_blob["cost_ceiling"] == (None if ceiling is None else asdict(ceiling))
|
||||
monkeypatch.setattr(owner_wait, "rebind_restored_route", lambda *_a, **_k: (None, "max"))
|
||||
# The Resume refresh reads the ledger NOW: wallet $500, the tree's known $8 (cap $10).
|
||||
monkeypatch.setattr(loop_budget, "_wrapup_global_remaining", lambda: 500.0)
|
||||
monkeypatch.setattr(loop_budget, "_loop_tree_accounting", lambda **_k: ledger or {
|
||||
"settled_usd": 8.0, "accounted_usd": 30.0, "root_limit_usd": 10.0, "age_sec": 0.0})
|
||||
messages, usage = [], {}
|
||||
with accounting.usage_scope(scope or accounting.UsageScope(
|
||||
drive_root=tmp_path, task_id=task_id, root_task_id=task_id, root_limit_usd=10.0)):
|
||||
budget_pause.resume_paused_loop(SimpleNamespace(_ctx=ctx), state_blob, messages, {}, usage, set(),
|
||||
budget_remaining_usd=500.0)
|
||||
return ctx, usage["budget_pause_resume"]["threshold_refresh"], messages[-1]["content"]
|
||||
|
||||
|
||||
def test_an_ordinary_pre_upgrade_pause_resumes_without_the_removed_default(tmp_path, monkeypatch):
|
||||
"""The #1128 shape: paused at $7 of a $10 cap by cap-minus-margin. Resume removes the
|
||||
hidden ceiling instead of re-arming it, and the model is told why."""
|
||||
legacy = task_pacing.CostCeiling(state="active", ceiling_usd=7.0, root_cap_usd=10.0,
|
||||
planning_margin_usd=3.0, basis="min(global_pct, root_cap_minus_margin)")
|
||||
ctx, refresh, notice = _resume(tmp_path, monkeypatch, "ordinary-1", legacy)
|
||||
assert ctx._cost_ceiling.state == task_pacing.COST_CEILING_DISABLED
|
||||
assert refresh == {"refreshed": False, "reason": "no_ceiling", "ceiling_state": "disabled",
|
||||
"ceiling_basis": "no_default_cost_stop(saved default stop $7.00 removed)"}
|
||||
assert "not refreshed: no_ceiling (no_default_cost_stop(saved default stop $7.00 removed))" in notice
|
||||
assert "Cumulative spend, rounds and elapsed execution time were NOT reset" in notice
|
||||
|
||||
|
||||
def test_an_explicit_profile_keeps_its_saved_point_and_the_q10_refresh(tmp_path, monkeypatch):
|
||||
explicit = task_pacing.CostCeiling(state="active", ceiling_usd=7.0, root_cap_usd=10.0,
|
||||
planning_margin_usd=3.0, basis="min(global_pct, root_cap_minus_margin)")
|
||||
ctx, refresh, notice = _resume(tmp_path, monkeypatch, "explicit-1", explicit,
|
||||
contract={"budget_profile": {"cost_hard_stop_pct": 50}})
|
||||
# Spend is the tree's KNOWN $8; the $22 of open holds does not shrink the room.
|
||||
assert refresh["refreshed"] is True and refresh["spent_usd"] == 8.0
|
||||
assert ctx._cost_ceiling.ceiling_usd == pytest.approx(10.0) # 8 + min(10 - 8, 50% of 500)
|
||||
assert ctx._cost_ceiling.basis.startswith("owner_resume_refresh")
|
||||
assert "refreshed to $10.00" in notice
|
||||
|
||||
|
||||
def test_a_producer_allowance_is_restored_at_its_value_and_never_widened(tmp_path, monkeypatch):
|
||||
"""A wake paused under the old $6 (its $9 allowance minus the margin). The restore
|
||||
re-reads the producer's $9; the Resume does not stretch it to the $50 cap."""
|
||||
old = task_pacing.CostCeiling(state="active", ceiling_usd=6.0, root_cap_usd=50.0,
|
||||
planning_margin_usd=3.0, basis="min(root_cap_minus_margin, root_ceiling_minus_margin)")
|
||||
scope = accounting.UsageScope(drive_root=tmp_path, task_id="wake-1", root_task_id="wake-1",
|
||||
root_limit_usd=50.0, root_cost_ceiling_usd=9.0)
|
||||
ctx, refresh, notice = _resume(tmp_path, monkeypatch, "wake-1", old, scope=scope)
|
||||
assert ctx._cost_ceiling == task_pacing.CostCeiling(
|
||||
state="active", ceiling_usd=9.0, root_cap_usd=50.0, basis="producer_allowance")
|
||||
assert refresh["refreshed"] is False and refresh["reason"] == "producer_allowance_not_expanded"
|
||||
assert ctx._cost_ceiling.ceiling_usd == 9.0
|
||||
# Said, not implied: the launch-time allowance still binds even if its window has freed more.
|
||||
assert "not refreshed: producer_allowance_not_expanded" in notice
|
||||
assert "the $9.00 producer allowance this task was launched with still binds" in notice
|
||||
|
||||
|
||||
def test_a_grouped_producer_allowance_survives_an_unreadable_group_projection(tmp_path, monkeypatch):
|
||||
"""The same wake carrying a whole-work billing group, whose group projection times
|
||||
out at restore. Who authored the stop is copied from the scope BEFORE that fallible
|
||||
read, so the producer's $9 is restored (never classified as the removed default and
|
||||
dropped), and the cap the scope carries stands in for the unreadable projection."""
|
||||
from ouroboros import usage_admission
|
||||
|
||||
def store_timeout(*_args, **_kwargs):
|
||||
raise TimeoutError("usage store lock timed out")
|
||||
|
||||
monkeypatch.setattr(usage_admission, "task_money_snapshot", store_timeout)
|
||||
old = task_pacing.CostCeiling(state="active", ceiling_usd=6.0, root_cap_usd=50.0,
|
||||
planning_margin_usd=3.0, basis="min(root_cap_minus_margin, root_ceiling_minus_margin)")
|
||||
scope = accounting.UsageScope(drive_root=tmp_path, task_id="wake-2", root_task_id="wake-2",
|
||||
root_limit_usd=50.0, root_cost_ceiling_usd=9.0, billing_group_id="group-wake",
|
||||
billing_group_limit_usd=50.0, billing_group_limit_source="fixture")
|
||||
ctx, refresh, notice = _resume(tmp_path, monkeypatch, "wake-2", old, scope=scope)
|
||||
assert ctx._cost_ceiling == task_pacing.CostCeiling(
|
||||
state="active", ceiling_usd=9.0, root_cap_usd=50.0, basis="producer_allowance")
|
||||
assert refresh["refreshed"] is False and refresh["reason"] == "producer_allowance_not_expanded"
|
||||
assert "removed" not in notice
|
||||
|
||||
|
||||
def test_an_unverified_inherited_number_keeps_its_bound_and_is_not_widened(tmp_path, monkeypatch):
|
||||
"""A pre-upgrade member whose root row cannot be read: its saved $7 stays, labelled,
|
||||
and the Resume refresh reports that it needs authority rather than moving it."""
|
||||
saved = task_pacing.CostCeiling(state="active", ceiling_usd=7.0, root_cap_usd=10.0,
|
||||
basis="min(root_resolved_ceiling)")
|
||||
scope = accounting.UsageScope(drive_root=tmp_path, task_id="member-1", root_task_id="lost-root",
|
||||
parent_task_id="lost-root", root_limit_usd=10.0, root_cost_ceiling_usd=7.0)
|
||||
ctx, refresh, _notice = _resume(tmp_path, monkeypatch, "member-1", saved, scope=scope)
|
||||
assert ctx._cost_ceiling.state == task_pacing.COST_CEILING_ACTIVE and ctx._cost_ceiling.ceiling_usd == 7.0
|
||||
assert ctx._cost_ceiling.basis == "legacy_policy_unverified(min(root_resolved_ceiling))"
|
||||
assert refresh == {"refreshed": False, "reason": "policy_unverified_not_expanded",
|
||||
"authority": task_pacing.COST_STOP_UNKNOWN, "ceiling_usd": 7.0}
|
||||
|
||||
|
||||
def test_every_restore_re_reads_the_root_authority_never_a_stashed_policy(tmp_path):
|
||||
"""The policy a start or an earlier restore stashed on the context is never reused:
|
||||
each restore reads the root again. Unreadable -> the saved bound stays, labelled;
|
||||
readable ordinary root -> the default it was is removed; readable explicit root ->
|
||||
the saved point is kept verbatim."""
|
||||
from ouroboros.contracts.task_contract import build_task_contract
|
||||
from ouroboros.task_results import task_result_path, write_task_result
|
||||
from ouroboros.tools.registry import ToolContext
|
||||
|
||||
def root_row(profile):
|
||||
task_result_path(tmp_path, "root-2").unlink(missing_ok=True)
|
||||
write_task_result(tmp_path, "root-2", "running", metadata={}, task_contract=build_task_contract(
|
||||
{"id": "root-2", "type": "task", "task_contract": {"budget_profile": profile}}))
|
||||
|
||||
ctx = ToolContext(repo_dir=tmp_path, drive_root=tmp_path)
|
||||
ctx.task_id, ctx.task_contract = "member-2", build_task_contract({"id": "member-2", "type": "task"})
|
||||
saved = asdict(task_pacing.CostCeiling(state="active", ceiling_usd=7.0, root_cap_usd=10.0,
|
||||
basis="min(root_resolved_ceiling)"))
|
||||
scope = accounting.UsageScope(drive_root=tmp_path, task_id="member-2", root_task_id="root-2",
|
||||
parent_task_id="root-2", root_limit_usd=10.0, root_cost_ceiling_usd=7.0)
|
||||
with accounting.usage_scope(scope):
|
||||
first = task_pacing.restore_cost_ceiling(ctx, saved)
|
||||
assert (first.state, first.ceiling_usd) == ("active", 7.0)
|
||||
assert first.basis == "legacy_policy_unverified(min(root_resolved_ceiling))"
|
||||
assert task_pacing.cost_stop_authority(ctx) == task_pacing.COST_STOP_UNKNOWN
|
||||
|
||||
root_row({})
|
||||
second = task_pacing.restore_cost_ceiling(ctx, saved)
|
||||
assert second.state == task_pacing.COST_CEILING_DISABLED
|
||||
assert second.basis == "no_default_cost_stop(saved default stop $7.00 removed)"
|
||||
assert task_pacing.cost_stop_authority(ctx) == task_pacing.COST_STOP_NONE
|
||||
|
||||
root_row({"cost_hard_stop_pct": 20})
|
||||
ctx._cost_stop_policy = {"authority": task_pacing.COST_STOP_NONE} # a stale stash is ignored
|
||||
third = task_pacing.restore_cost_ceiling(ctx, saved)
|
||||
assert third == task_pacing.CostCeiling(**saved)
|
||||
assert task_pacing.cost_stop_policy(ctx)["profile"]["cost_hard_stop_pct"] == 20
|
||||
|
|
@ -977,9 +977,11 @@ def test_production_budget_wrapup_propagates_budget_exceeded(tmp_path, monkeypat
|
|||
import ouroboros.usage_accounting as accounting
|
||||
from ouroboros import task_pacing
|
||||
|
||||
loop, _registry, ctx, trace = _forced_test_context(
|
||||
loop, registry, ctx, trace = _forced_test_context(
|
||||
tmp_path, usage={"cost": 5.0},
|
||||
)
|
||||
# The paid wrap-up at a reached ceiling is an explicit experiment profile's authored rail.
|
||||
registry._ctx.task_contract = {**registry._ctx.task_contract, "budget_profile": {"cost_hard_stop_pct": 50}}
|
||||
|
||||
def reject_dispatch(*_args, **_kwargs):
|
||||
raise accounting.BudgetExceeded(
|
||||
|
|
@ -1003,9 +1005,11 @@ def test_wrapup_is_forced_while_real_ledger_admission_still_fits(tmp_path, monke
|
|||
from ouroboros import task_pacing
|
||||
import ouroboros.usage_accounting as accounting
|
||||
|
||||
loop, _registry, ctx, _trace = _forced_test_context(
|
||||
loop, registry, ctx, _trace = _forced_test_context(
|
||||
tmp_path, usage={"cost": 1.0, "_context_prompt_estimate": 1000},
|
||||
)
|
||||
# The last-fit rail is an explicit experiment profile's authored rail (#1128).
|
||||
registry._ctx.task_contract = {**registry._ctx.task_contract, "budget_profile": {"cost_hard_stop_pct": 50}}
|
||||
scope = accounting.UsageScope(
|
||||
drive_root=tmp_path, task_id="parent1", root_task_id="parent1",
|
||||
global_limit_usd=1000.0, root_limit_usd=100.0,
|
||||
|
|
|
|||
|
|
@ -87,7 +87,7 @@ def test_consciousness_status_snapshot_exposes_the_alarm_facts(monkeypatch, tmp_
|
|||
|
||||
monkeypatch.setattr(state, "load_state", lambda: {"bg_consciousness_enabled": True, "owner_chat_id": 1})
|
||||
monkeypatch.setattr(clock_module, "allowance_window", lambda root, now=None, **_display_read: {
|
||||
"status": "available", "limit_usd": 20.0, "accounted_usd": 3.0, "remaining_usd": 17.0, "resets_at": ""})
|
||||
"status": "available", "limit_usd": 20.0, "settled_usd": 3.0, "accounted_usd": 3.0, "remaining_usd": 17.0, "resets_at": ""})
|
||||
monkeypatch.setattr(BackgroundConsciousness, "_running_roots", staticmethod(lambda: 0))
|
||||
clock = BackgroundConsciousness(tmp_path, tmp_path / "repo", lambda: 1, now=1_800_000_000.0)
|
||||
clock.notify("task_finished:t1:completed")
|
||||
|
|
|
|||
|
|
@ -344,7 +344,7 @@ def test_the_budget_soft_landing_wraps_up_with_the_model_not_the_host_text(actin
|
|||
acting.ctx.messages = [{"role": "user", "content": "Please finish"}]
|
||||
acting.ctx.llm_trace = {}
|
||||
acting.tools._ctx.model_turn_state = ModelTurnState(deepcopy(TURN))
|
||||
ceiling = task_pacing.resolve_cost_ceiling(100.0, normalize_budget_profile(None), root_cap_usd=0.5)
|
||||
ceiling = task_pacing.resolve_cost_ceiling(100.0, normalize_budget_profile({"cost_hard_stop_pct": 50}), root_cap_usd=0.5)
|
||||
text, _usage, _trace = loop_module._soft_land_exhausted_ceiling(acting.ctx, ceiling)
|
||||
assert "Ответ 🐍" in text and "no working room" not in text
|
||||
assert acting.gateway.uploads[0][0]["nativeContinuation"] == TURN
|
||||
|
|
|
|||
|
|
@ -249,7 +249,7 @@ def test_group_scope_includes_sibling_and_original_late_charge(world):
|
|||
queue.check_scheduled_tasks()
|
||||
[task] = pending
|
||||
fields = task_billing_fields(task, task['id'], 999.0, root)
|
||||
for who, amount in ((ORIGIN, 4.0), ('sibling', 4.0)):
|
||||
for who, amount in ((ORIGIN, 5.0), ('sibling', 5.0)): # known spend reaches the group's $10
|
||||
with ua.usage_scope(ua.UsageScope(drive_root=root, task_id=who, root_task_id=who,
|
||||
source='test', root_limit_usd=10.0, **BINDING)):
|
||||
reservation = ua.reserve_attempt(ua.AttemptRequest(model='openai/gpt-5.2', provider='openai', reservation_usd=amount))
|
||||
|
|
|
|||
|
|
@ -124,7 +124,8 @@ def test_cost_breakdown_uses_ledger_not_later_compatibility_events(tmp_path, mon
|
|||
"attempt_counts": {"reserved": 1, "settled": 2, "unresolved": 1},
|
||||
"authority": "physical_attempt_ledger",
|
||||
"limit_usd": 7.5,
|
||||
"remaining_known_usd": 5.75,
|
||||
# Room above KNOWN spend ($0.25): the open $1.50 of holds is exposure (#1487).
|
||||
"remaining_known_usd": 7.25,
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -192,13 +193,13 @@ def test_api_state_money_and_call_count_are_ledger_projections(tmp_path, monkeyp
|
|||
payload = json.loads(response.body)
|
||||
|
||||
assert response.status_code == 200
|
||||
assert payload["spent_usd"] == 1.75
|
||||
assert payload["spent_usd"] == 0.25 # the header shows known spend; holds ride in accounting
|
||||
assert payload["spent_calls"] == 3
|
||||
assert payload["budget_limit"] == 7.5
|
||||
assert payload["accounting"]["authority"] == "physical_attempt_ledger"
|
||||
assert payload["accounting"]["accounted_usd"] == 1.75
|
||||
assert payload["accounting"]["unknown_unmetered"] == 1
|
||||
assert payload["accounting"]["remaining_known_usd"] == 5.75
|
||||
assert payload["accounting"]["remaining_known_usd"] == 7.25
|
||||
|
||||
|
||||
@pytest.mark.serial
|
||||
|
|
@ -544,4 +545,4 @@ def test_api_state_reads_the_slim_writer_snapshot_never_the_full_breakdown(tmp_p
|
|||
assert call() == with_full_breakdown_available
|
||||
assert with_full_breakdown_available["spent_calls"] == 3
|
||||
assert with_full_breakdown_available["accounting"]["accounted_usd"] == 1.75
|
||||
assert with_full_breakdown_available["accounting"]["remaining_known_usd"] == (5.75 if limit else None)
|
||||
assert with_full_breakdown_available["accounting"]["remaining_known_usd"] == (7.25 if limit else None)
|
||||
|
|
|
|||
381
tests/test_known_spend_admission.py
Normal file
381
tests/test_known_spend_admission.py
Normal file
|
|
@ -0,0 +1,381 @@
|
|||
"""Known spend decides every money limit (owner Q4-A 2026-10-03, #1487).
|
||||
|
||||
Known spend is the settled bucket: confirmed prices AND disclosed estimates. A
|
||||
reservation or the upper bound of an unresolved call is exposure, shown beside
|
||||
it and never counted as spending. The global, root and original-group axes use
|
||||
ONE predicate at reservation and again at dispatch: refuse once known spend has
|
||||
reached the limit (equality refuses). Concurrent, in-flight and late charges can
|
||||
still overshoot; nothing here promises a maximum. Every case runs the real
|
||||
``reserve_attempt`` / ``mark_dispatched`` over the real usage store.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import replace
|
||||
|
||||
import pytest
|
||||
|
||||
from ouroboros import usage_accounting as ua
|
||||
from tests import fixtures_usage_store as _fixtures
|
||||
from tests._usage_store_testing import ledger_rows
|
||||
|
||||
data_root = _fixtures.data_root
|
||||
|
||||
AXES = ("global", "root", "group")
|
||||
|
||||
|
||||
def _scope(root, axis, task_id="t", root_task_id="t"):
|
||||
"""A scope whose ONLY finite $10 limit is ``axis``."""
|
||||
common = dict(drive_root=root, task_id=task_id, root_task_id=root_task_id, source="test")
|
||||
if axis == "global":
|
||||
return ua.UsageScope(global_limit_usd=10.0, **common)
|
||||
if axis == "root":
|
||||
return ua.UsageScope(global_limit_usd=1000.0, root_limit_usd=10.0, **common)
|
||||
return ua.UsageScope(global_limit_usd=1000.0, billing_group_id="G", billing_group_limit_usd=10.0,
|
||||
billing_group_limit_source="fixture", **common)
|
||||
|
||||
|
||||
def _neighbour(root, axis):
|
||||
"""Another worker on the same axis: another root (global), another member (root), a Continue (group)."""
|
||||
if axis == "global":
|
||||
return _scope(root, axis, "o", "o")
|
||||
if axis == "root":
|
||||
return _scope(root, axis, "t2", "t")
|
||||
return _scope(root, axis, "s", "s")
|
||||
|
||||
|
||||
def _request(amount=1.0):
|
||||
return ua.AttemptRequest(model="openai/gpt-5.2", provider="openai", reservation_usd=amount)
|
||||
|
||||
|
||||
def _settled(scope, cost, *, final=True):
|
||||
with ua.usage_scope(scope):
|
||||
held = ua.reserve_attempt(_request(cost))
|
||||
ua.mark_dispatched(held)
|
||||
ua.settle_attempt(held, {"prompt_tokens": 1, "completion_tokens": 1}, cost_usd=cost, cost_final=final)
|
||||
|
||||
|
||||
def _unresolved(scope, bound):
|
||||
with ua.usage_scope(scope):
|
||||
held = ua.reserve_attempt(_request(bound))
|
||||
ua.mark_dispatched(held)
|
||||
ua.mark_unresolved(held, "provider outcome unknown")
|
||||
|
||||
|
||||
def _state(root, attempt_id):
|
||||
return {row["attempt_id"]: row for row in ledger_rows(root)}[attempt_id]["state"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("axis", AXES)
|
||||
def test_two_dollars_known_beside_a_twenty_dollar_unknown_admits_under_ten(data_root, axis):
|
||||
scope = _scope(data_root, axis)
|
||||
_settled(scope, 2.0)
|
||||
_unresolved(scope, 20.0)
|
||||
with ua.usage_scope(scope):
|
||||
held = ua.reserve_attempt(_request(5.0))
|
||||
ua.mark_dispatched(held) # the dispatch seam admits on the same known $2
|
||||
assert _state(data_root, held.attempt_id) == "dispatched"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("axis", AXES)
|
||||
def test_nine_estimated_and_two_confirmed_refuse_ten(data_root, axis):
|
||||
"""Estimates are known spend: known-only is not confirmed-only."""
|
||||
scope = _scope(data_root, axis)
|
||||
_settled(scope, 9.0, final=False)
|
||||
_settled(scope, 2.0, final=True)
|
||||
with ua.usage_scope(scope), pytest.raises(ua.BudgetExceeded, match=r"known=\$11\.000000") as refused:
|
||||
ua.reserve_attempt(_request(0.01))
|
||||
assert refused.value.limit_scope == ("global" if axis == "global" else "root")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("axis", AXES)
|
||||
def test_known_spend_exactly_at_the_limit_refuses_both_seams(data_root, axis):
|
||||
scope = _scope(data_root, axis)
|
||||
with ua.usage_scope(scope):
|
||||
held = ua.reserve_attempt(_request(0.5)) # reserved while known is $0
|
||||
_settled(_neighbour(data_root, axis), 10.0) # known spend reaches exactly $10
|
||||
with ua.usage_scope(scope):
|
||||
with pytest.raises(ua.BudgetExceeded):
|
||||
ua.reserve_attempt(_request(0.01))
|
||||
with pytest.raises(ua.BudgetExceeded, match="changed before dispatch|exhausted"):
|
||||
ua.mark_dispatched(held)
|
||||
# The crossing between reserve and dispatch never sent: the row is still only reserved.
|
||||
assert _state(data_root, held.attempt_id) == "reserved"
|
||||
ua.release_attempt(held, "before_dispatch_failed:limit")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("axis", AXES)
|
||||
def test_a_crossing_between_reserve_and_dispatch_never_sends(data_root, axis, monkeypatch):
|
||||
sends, crossed = [], []
|
||||
scope = _scope(data_root, axis)
|
||||
_settled(scope, 9.99)
|
||||
original_dispatch = ua.mark_dispatched
|
||||
|
||||
def cross_then_dispatch(reservation, **kwargs):
|
||||
if not crossed: # once: the neighbour's own settlement dispatches normally
|
||||
crossed.append(True)
|
||||
_settled(_neighbour(data_root, axis), 0.01) # lands after the reservation, before the send
|
||||
return original_dispatch(reservation, **kwargs)
|
||||
|
||||
monkeypatch.setattr(ua, "mark_dispatched", cross_then_dispatch)
|
||||
with ua.usage_scope(scope), pytest.raises(ua.BudgetExceeded):
|
||||
ua.execute_physical_attempt(_request(0.5), lambda: sends.append("sent"))
|
||||
assert crossed and sends == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("axis", AXES)
|
||||
def test_a_late_charge_past_the_limit_is_recorded_not_prevented(data_root, axis):
|
||||
"""Owner Q4-A accepts overshoot: two in-flight calls admitted at $9 settle to $13.
|
||||
The overrun is recorded, and the next admission is refused on it."""
|
||||
scope = _scope(data_root, axis)
|
||||
_settled(scope, 9.0)
|
||||
with ua.usage_scope(scope):
|
||||
first, second = ua.reserve_attempt(_request(2.0)), ua.reserve_attempt(_request(2.0))
|
||||
ua.mark_dispatched(first)
|
||||
ua.mark_dispatched(second) # both in flight at known $9
|
||||
for held in (first, second):
|
||||
ua.settle_attempt(held, {"prompt_tokens": 1, "completion_tokens": 1}, cost_usd=2.0, cost_final=True)
|
||||
with pytest.raises(ua.BudgetExceeded):
|
||||
ua.reserve_attempt(_request(0.01))
|
||||
if axis == "group":
|
||||
projection = ua.usage_projection(data_root, billing_group_id="G")
|
||||
elif axis == "root":
|
||||
projection = ua.usage_projection(data_root, root_task_id="t")
|
||||
else:
|
||||
projection = ua.usage_projection(data_root, global_limit_usd=10.0)
|
||||
assert projection["settled_usd"] == pytest.approx(13.0)
|
||||
|
||||
|
||||
def test_a_zero_limit_admits_nothing_and_no_limit_admits_on_any_exposure(data_root, monkeypatch):
|
||||
with ua.usage_scope(replace(_scope(data_root, "global"), global_limit_usd=0.0)):
|
||||
with pytest.raises(ua.BudgetExceeded):
|
||||
ua.reserve_attempt(_request(0.01))
|
||||
with ua.usage_scope(replace(_scope(data_root, "root"), root_limit_usd=0.0)):
|
||||
with pytest.raises(ua.BudgetExceeded):
|
||||
ua.reserve_attempt(_request(0.01))
|
||||
monkeypatch.setenv("TOTAL_BUDGET", "0") # no finite global budget
|
||||
unlimited = ua.UsageScope(drive_root=data_root, task_id="u", root_task_id="u", source="test")
|
||||
_unresolved(unlimited, 500.0)
|
||||
_settled(unlimited, 400.0)
|
||||
with ua.usage_scope(unlimited):
|
||||
ua.release_attempt(ua.reserve_attempt(_request(1.0)))
|
||||
|
||||
|
||||
def test_unavailable_group_authority_is_refused_never_a_healthy_empty_store(data_root):
|
||||
"""Unknown price is not a zero, and missing group authority is not an empty group."""
|
||||
scope = ua.UsageScope(drive_root=data_root, task_id="t", root_task_id="t", global_limit_usd=1000.0,
|
||||
billing_group_id="unavailable:t", billing_group_limit_usd=0.0)
|
||||
with ua.usage_scope(scope), pytest.raises(ua.BudgetExceeded, match="billing group authority unavailable"):
|
||||
ua.reserve_attempt(_request(0.01))
|
||||
|
||||
|
||||
def test_remaining_known_usd_is_one_number_at_every_producer(data_root, monkeypatch):
|
||||
"""The field keeps its name and gains its meaning: limit minus KNOWN spend, at
|
||||
the ledger projection, the root projection and the Costs page alike."""
|
||||
import asyncio
|
||||
import json
|
||||
|
||||
from ouroboros.gateway.history import make_cost_breakdown_endpoint
|
||||
from supervisor import state as supervisor_state
|
||||
|
||||
scope = ua.UsageScope(drive_root=data_root, task_id="t", root_task_id="t", global_limit_usd=10.0,
|
||||
root_limit_usd=10.0)
|
||||
_settled(scope, 2.0)
|
||||
_unresolved(scope, 20.0)
|
||||
monkeypatch.setattr(supervisor_state, "TOTAL_BUDGET_LIMIT", 10.0)
|
||||
global_view = ua.usage_projection(data_root, global_limit_usd=10.0)
|
||||
root_view = ua.usage_projection(data_root, root_task_id="t")
|
||||
costs = json.loads(asyncio.run(make_cost_breakdown_endpoint(data_root)(None)).body)["accounting"]
|
||||
for view in (global_view, root_view, costs):
|
||||
assert view["remaining_known_usd"] == 8.0
|
||||
assert view["settled_usd"] == 2.0 and view["accounted_usd"] == 22.0 # exposure kept beside it
|
||||
|
||||
|
||||
def test_an_ineligible_actor_is_stopped_by_the_fence_with_no_funded_recap(data_root, monkeypatch, tmp_path):
|
||||
"""An actor without an exact continuation keeps the existing priced wrap-up rail
|
||||
(not redesigned here), but that rail cannot cross the physical money fence: with
|
||||
known spend at the cap, the forced send's own reservation is refused, nothing is
|
||||
sent, and the refusal ends the task in the typed terminal projection without any
|
||||
model-written recap being bought."""
|
||||
from types import SimpleNamespace
|
||||
|
||||
from ouroboros import loop, task_pacing
|
||||
from ouroboros.contracts.task_contract import normalize_budget_profile
|
||||
from tests._budget_limits_helpers import _make_args
|
||||
|
||||
sends = []
|
||||
scope = ua.UsageScope(drive_root=data_root, task_id="ineligible", root_task_id="ineligible",
|
||||
global_limit_usd=1000.0, root_limit_usd=10.0)
|
||||
_settled(scope, 10.0) # the tree's known spend is at its $10 cap
|
||||
before = ledger_rows(data_root)
|
||||
|
||||
def funded_send(*_args, **_kwargs):
|
||||
ua.execute_physical_attempt(_request(0.5), lambda: sends.append("recap"))
|
||||
return {"role": "assistant", "content": "recap"}, 0.5
|
||||
|
||||
monkeypatch.setattr(loop, "call_llm_with_retry", funded_send)
|
||||
monkeypatch.setattr(loop, "_loop_tree_accounting", lambda **_k: {"settled_usd": 10.0, "accounted_usd": 10.0})
|
||||
ceiling = task_pacing.resolve_cost_ceiling(1000.0, normalize_budget_profile({"cost_hard_stop_pct": 50}),
|
||||
root_cap_usd=10.0)
|
||||
args = _make_args(task_id="ineligible", accumulated_usage={"cost": 10.0}, cost_ceiling=ceiling,
|
||||
budget_remaining_usd=1000.0, drive_logs=tmp_path)
|
||||
tool_ctx = SimpleNamespace(task_id="ineligible", drive_root=data_root, budget_drive_root=data_root,
|
||||
owner_wait_callback=None, task_metadata={}, is_direct_chat=False,
|
||||
task_contract={"budget_profile": {"cost_hard_stop_pct": 50}}) # the profile it stops on
|
||||
args["ctx"].tools = SimpleNamespace(_ctx=tool_ctx)
|
||||
with ua.usage_scope(scope):
|
||||
with pytest.raises(ua.BudgetExceeded):
|
||||
loop._check_budget_limits(**args) # graceful stop -> ineligible -> priced rail -> fence
|
||||
usage = args["ctx"].accumulated_usage
|
||||
assert usage["exact_pause_unavailable"] == "no_continuation_owner"
|
||||
assert sends == [] and ledger_rows(data_root) == before # no recap reserved, none sent
|
||||
|
||||
exit_ctx = loop._LoopExitContext(
|
||||
tools=SimpleNamespace(_ctx=tool_ctx), drive_root=data_root, task_id="ineligible", event_queue=None,
|
||||
drive_logs=tmp_path, accumulated_usage=usage, llm_trace={})
|
||||
monkeypatch.setattr(loop, "_finalize_task_services", lambda _ctx: None)
|
||||
text, terminal, trace = loop._handle_budget_exceeded(
|
||||
ua.BudgetExceeded("root model budget exhausted", limit_scope="root", root_task_id="ineligible"),
|
||||
exit_ctx)
|
||||
assert terminal["execution_status"] == "failed" and terminal["reason_code"] == "budget_exhausted"
|
||||
assert trace["resource_limit"]["status"] == "resource_limited"
|
||||
assert trace["resource_limit"]["auto_resume"] is False
|
||||
assert "Resource limit reached before another model dispatch" in text
|
||||
assert sends == [] and ledger_rows(data_root) == before
|
||||
|
||||
|
||||
def test_an_ordinary_ineligible_actor_meets_only_the_real_cap_with_no_funded_recap(data_root, monkeypatch, tmp_path):
|
||||
"""The same actor without an explicit profile (owner 2026-10-07): no default early
|
||||
stop fires before the cap, so the budget gate buys nothing; the REAL root cap
|
||||
refuses its next send at reservation once known spend reached it, and the typed
|
||||
terminal follows with no model-written recap bought on the way out."""
|
||||
from types import SimpleNamespace
|
||||
|
||||
from ouroboros import loop, task_pacing
|
||||
from ouroboros.contracts.task_contract import normalize_budget_profile
|
||||
from tests._budget_limits_helpers import _make_args
|
||||
|
||||
sends = []
|
||||
scope = ua.UsageScope(drive_root=data_root, task_id="ordinary", root_task_id="ordinary",
|
||||
global_limit_usd=1000.0, root_limit_usd=10.0)
|
||||
_settled(scope, 10.0) # known spend is at the $10 cap
|
||||
before = ledger_rows(data_root)
|
||||
monkeypatch.setattr(loop, "call_llm_with_retry", lambda *_a, **_k: pytest.fail("no recap may be bought"))
|
||||
ceiling = task_pacing.resolve_cost_ceiling(1000.0, normalize_budget_profile(None), root_cap_usd=10.0)
|
||||
assert ceiling.state == task_pacing.COST_CEILING_DISABLED and ceiling.basis == task_pacing.COST_BASIS_NO_DEFAULT_STOP
|
||||
args = _make_args(task_id="ordinary", accumulated_usage={"cost": 10.0}, cost_ceiling=ceiling,
|
||||
budget_remaining_usd=1000.0, drive_logs=tmp_path)
|
||||
tool_ctx = SimpleNamespace(task_id="ordinary", drive_root=data_root, budget_drive_root=data_root,
|
||||
owner_wait_callback=None, task_metadata={}, is_direct_chat=False)
|
||||
args["ctx"].tools = SimpleNamespace(_ctx=tool_ctx)
|
||||
with ua.usage_scope(scope):
|
||||
assert loop._check_budget_limits(**args) is None # no host stop of its own
|
||||
with pytest.raises(ua.BudgetExceeded) as refused: # the next ordinary send meets the real cap
|
||||
ua.execute_physical_attempt(_request(0.5), lambda: sends.append("work"))
|
||||
assert refused.value.limit_scope == "root" and "known=$10.000000" in str(refused.value)
|
||||
assert sends == [] and ledger_rows(data_root) == before
|
||||
|
||||
exit_ctx = loop._LoopExitContext(
|
||||
tools=SimpleNamespace(_ctx=tool_ctx), drive_root=data_root, task_id="ordinary", event_queue=None,
|
||||
drive_logs=tmp_path, accumulated_usage=args["ctx"].accumulated_usage, llm_trace={})
|
||||
monkeypatch.setattr(loop, "_finalize_task_services", lambda _ctx: None)
|
||||
text, terminal, trace = loop._handle_budget_exceeded(refused.value, exit_ctx)
|
||||
assert terminal["execution_status"] == "failed" and terminal["reason_code"] == "budget_exhausted"
|
||||
assert trace["resource_limit"]["scope"] == "root" and trace["resource_limit"]["auto_resume"] is False
|
||||
assert "Resource limit reached before another model dispatch" in text
|
||||
assert sends == [] and ledger_rows(data_root) == before
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("author", "known", "stops"), [
|
||||
("producer", 8.99, False), ("producer", 9.0, True), ("producer", 9.5, True),
|
||||
("explicit", 9.0, False), ("explicit", 9.01, True),
|
||||
("unverified", 9.0, False), ("unverified", 9.5, True),
|
||||
], ids=["producer-below", "producer-equal", "producer-above", "explicit-equal", "explicit-above",
|
||||
"unverified-equal", "unverified-above"])
|
||||
def test_the_budget_gate_stops_a_producer_allowance_once_known_spend_reaches_it(
|
||||
data_root, monkeypatch, tmp_path, author, known, stops):
|
||||
"""A wake's $9 daily remainder is a real limit on known spend, so, like the ledger
|
||||
caps, reaching it stops the task (owner 2026-10-07). An explicit profile's $9 point
|
||||
keeps its authored edge: only spend over it stops. Real ledger rows, the real
|
||||
task-start resolver, the real budget gate and the real forced rail; the model send
|
||||
is funded through the real ledger fence.
|
||||
|
||||
Only an explicit profile buys the final answer it authored room for. The fence
|
||||
checks the $50 tree cap and the $1000 wallet, never the $9 allowance, so a recap
|
||||
after the allowance (or an inherited number whose author is unreadable, kept and
|
||||
never widened) would be paid past it: that actor ends on the host's own notice."""
|
||||
from types import SimpleNamespace
|
||||
|
||||
from ouroboros import loop, task_pacing
|
||||
from tests._budget_limits_helpers import _make_args
|
||||
|
||||
sends = []
|
||||
|
||||
def funded_send(*_args, **_kwargs):
|
||||
ua.execute_physical_attempt(_request(0.5), lambda: sends.append("recap"))
|
||||
return {"role": "assistant", "content": "recap"}, 0.5
|
||||
|
||||
monkeypatch.setattr(loop, "call_llm_with_retry", funded_send)
|
||||
task_id = f"gate-{author}-{known}"
|
||||
profile = {"cost_hard_stop_pct": 50} if author == "explicit" else None
|
||||
root_id = task_id if author != "unverified" else f"lost-root-{known}" # a root with no readable row
|
||||
scope = ua.UsageScope(drive_root=data_root, task_id=task_id, root_task_id=root_id, global_limit_usd=1000.0, **(
|
||||
{} if author == "explicit" else {"root_limit_usd": 50.0, "root_cost_ceiling_usd": 9.0}),
|
||||
**({"parent_task_id": root_id} if author == "unverified" else {}))
|
||||
# The tree's known spend: the member's root spent it before the member's check.
|
||||
_settled(scope if author != "unverified" else ua.UsageScope(
|
||||
drive_root=data_root, task_id=root_id, root_task_id=root_id, global_limit_usd=1000.0,
|
||||
root_limit_usd=50.0), known)
|
||||
before = ledger_rows(data_root)
|
||||
tool_ctx = SimpleNamespace(task_id=task_id, drive_root=data_root, budget_drive_root=data_root,
|
||||
owner_wait_callback=None, task_metadata={}, is_direct_chat=False,
|
||||
task_contract={"budget_profile": profile} if profile else None)
|
||||
with ua.usage_scope(scope):
|
||||
ceiling = task_pacing.resolve_task_cost_ceiling(tool_ctx, 18.0)
|
||||
assert ceiling.state == task_pacing.COST_CEILING_ACTIVE and ceiling.ceiling_usd == 9.0
|
||||
assert task_pacing.cost_stop_authority(tool_ctx) == {
|
||||
"explicit": task_pacing.COST_STOP_EXPLICIT, "producer": task_pacing.COST_STOP_PRODUCER,
|
||||
"unverified": task_pacing.COST_STOP_UNKNOWN}[author]
|
||||
args = _make_args(task_id=task_id, accumulated_usage={"cost": known}, cost_ceiling=ceiling,
|
||||
budget_remaining_usd=1000.0, drive_logs=tmp_path)
|
||||
args["ctx"].tools = SimpleNamespace(_ctx=tool_ctx)
|
||||
result = loop._check_budget_limits(**args)
|
||||
usage = args["ctx"].accumulated_usage
|
||||
if not stops:
|
||||
assert result is None and sends == [] and "exact_pause_unavailable" not in usage
|
||||
return
|
||||
assert result is not None
|
||||
assert usage["cost_stop_spend_basis"] == task_pacing.SPEND_BASIS_TREE
|
||||
assert usage["exact_pause_unavailable"] == "no_continuation_owner" # the graceful pause rail ran
|
||||
assert usage["execution_status"] == "failed" and usage["reason_code"] == "budget_exhausted"
|
||||
if author == "explicit":
|
||||
assert sends == ["recap"] and result[0] == "recap" # its authored final, fenced by the hard limits
|
||||
return
|
||||
assert sends == [] and ledger_rows(data_root) == before # nothing reserved, nothing sent
|
||||
assert f"spent ${known:.3f}" in result[0] and "$9.00" in result[0]
|
||||
|
||||
|
||||
def test_limit_texts_promise_a_saved_pause_only_where_an_exact_continuation_exists(data_root, monkeypatch):
|
||||
"""The owner setting, the runtime fact and the ceiling disclosure describe the SAME
|
||||
two endings the ineligible-actor tests above establish: an exact continuation pauses
|
||||
with its work saved; an actor without one ends budget-exhausted. None promises the
|
||||
saved pause to every actor, and none grants an actor a new capability."""
|
||||
from types import SimpleNamespace
|
||||
|
||||
from ouroboros import task_pacing
|
||||
from ouroboros.context_runtime_facts import _runtime_budget_info
|
||||
from ouroboros.settings_setup_contract import build_setup_contract
|
||||
|
||||
monkeypatch.setenv("OUROBOROS_PER_TASK_COST_USD", "10")
|
||||
fields = {str(field.get("settingKey")): field for field in build_setup_contract()["budgetFields"]}
|
||||
texts = {
|
||||
"setting": str(fields["OUROBOROS_PER_TASK_COST_USD"]["note"]),
|
||||
"runtime": _runtime_budget_info(SimpleNamespace(drive_root=data_root), {})["per_task_tree_cap_rule"],
|
||||
"disclosure": task_pacing.cost_ceiling_disclosure(task_pacing.resolve_cost_ceiling(
|
||||
1000.0, {"cost_hard_stop_pct": None}, root_cap_usd=10.0))["rule"],
|
||||
}
|
||||
for surface, text in texts.items():
|
||||
assert "pauses with its work saved" in text, surface
|
||||
assert "the task pauses" not in text and "the task then pauses" not in text, surface
|
||||
assert "budget-exhausted" in text or "budget_exhausted" in text, surface
|
||||
assert "no paid recap" in texts["runtime"] # at the hard cap the fence refuses the recap too
|
||||
|
|
@ -332,7 +332,7 @@ def test_start_agent_exports_presentation_posture(monkeypatch, tmp_path):
|
|||
captured = {}
|
||||
|
||||
class FakeStdout:
|
||||
def readline(self):
|
||||
def readline(self, _limit=-1):
|
||||
return b""
|
||||
|
||||
class FakeProcess:
|
||||
|
|
@ -382,7 +382,7 @@ def test_start_agent_unix_uses_process_group_and_writes_server_record(monkeypatc
|
|||
captured = {}
|
||||
|
||||
class FakeStdout:
|
||||
def readline(self):
|
||||
def readline(self, _limit=-1):
|
||||
return b""
|
||||
|
||||
class FakeProcess:
|
||||
|
|
@ -582,7 +582,7 @@ def test_start_agent_windows_assigns_job_before_resume_and_records(monkeypatch,
|
|||
calls: list[tuple[str, object]] = []
|
||||
|
||||
class FakeStdout:
|
||||
def readline(self):
|
||||
def readline(self, _limit=-1):
|
||||
return b""
|
||||
|
||||
class FakeProcess:
|
||||
|
|
|
|||
|
|
@ -13,6 +13,21 @@ from __future__ import annotations
|
|||
import asyncio
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _isolated_request_capabilities(monkeypatch, tmp_path):
|
||||
"""Each fake provider starts without another test's learned wire policy."""
|
||||
from ouroboros import request_wire_contract
|
||||
from ouroboros.llm import LLMClient
|
||||
|
||||
# A successful retry persists a drop-field action. Clearing the legacy
|
||||
# rejected-params cache cannot reset that evidence, and a warm /models
|
||||
# cache can also strip temperature before the fake's first rejection.
|
||||
monkeypatch.setattr(request_wire_contract, "canonical_wire_evidence_root", lambda: tmp_path)
|
||||
monkeypatch.setattr(LLMClient, "_SUPPORTED_PARAMS_CACHE", {})
|
||||
monkeypatch.setattr(LLMClient, "_SUPPORTED_PARAMS_FETCHED", False)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -308,12 +323,6 @@ def test_chat_remote_no_proxy_retries_openrouter_parameter_rejection(monkeypatch
|
|||
"""OpenRouter no_proxy path retries once without optional sampling params."""
|
||||
from ouroboros.llm import LLMClient
|
||||
|
||||
LLMClient._REJECTED_PARAMS_CACHE.clear()
|
||||
# Premise: a cold no_proxy worker with no capability metadata yet strips nothing
|
||||
# up front, so temperature reaches the first send and only the provider's
|
||||
# rejection removes it. Metadata warmed by an earlier test would strip it first.
|
||||
monkeypatch.setattr(LLMClient, "_SUPPORTED_PARAMS_FETCHED", False)
|
||||
monkeypatch.setattr(LLMClient, "_SUPPORTED_PARAMS_CACHE", {})
|
||||
client = LLMClient(api_key="test-or-key")
|
||||
target = client._resolve_remote_target("anthropic/claude-opus-4.8")
|
||||
messages = [{"role": "user", "content": "hello"}]
|
||||
|
|
|
|||
|
|
@ -655,7 +655,7 @@ def test_budget_line_replays_unresolved_attempt_not_stale_state(monkeypatch, tmp
|
|||
ua.mark_unresolved(reservation, "timeout")
|
||||
|
||||
stale = {
|
||||
"spent_usd": 0,
|
||||
"spent_usd": 5,
|
||||
"spent_calls": 0,
|
||||
"current_branch": "ouroboros",
|
||||
"current_sha": "abcdef123456",
|
||||
|
|
@ -671,10 +671,12 @@ def test_budget_line_replays_unresolved_attempt_not_stale_state(monkeypatch, tmp
|
|||
|
||||
line = message_bus.budget_line(force=True)
|
||||
|
||||
assert "$1.0000 / $10.00" in line
|
||||
# The ledger, not the stale $5 counter: known spend is $0 and the unresolved
|
||||
# attempt's $1 bound is disclosed beside it, not counted as spending (#1487).
|
||||
assert "$0.0000 / $10.00" in line
|
||||
assert "unresolved <=$1.0000" in line
|
||||
assert "ouroboros@abcdef12" in line
|
||||
assert "$0.0000 / $10.00" not in line
|
||||
assert "$5.0000" not in line and "$1.0000 / $10.00" not in line
|
||||
|
||||
|
||||
@pytest.mark.parametrize("damage", ["journal", "store"])
|
||||
|
|
|
|||
|
|
@ -13,29 +13,40 @@ def _request(amount, task="other"):
|
|||
reservation_usd=amount, task_id=task, root_task_id=task)
|
||||
|
||||
|
||||
def _spend(amount, task="other"):
|
||||
"""KNOWN spend: the only money the wallet and its fence decide on (#1487)."""
|
||||
held = accounting.reserve_attempt(_request(amount, task))
|
||||
accounting.mark_dispatched(held)
|
||||
accounting.settle_attempt(held, {}, cost_usd=amount, cost_final=True)
|
||||
return held
|
||||
|
||||
|
||||
@pytest.mark.parametrize("root_cap", [None, 50.0])
|
||||
def test_actual_global_holds_leave_one_same_route_final_then_the_atomic_fence_binds(tmp_path, root_cap):
|
||||
def test_known_global_spend_leaves_one_same_route_final_then_the_atomic_fence_binds(tmp_path, root_cap):
|
||||
scope = accounting.UsageScope(drive_root=tmp_path, task_id="main", root_task_id="main",
|
||||
global_limit_usd=10.0, root_limit_usd=root_cap)
|
||||
with accounting.usage_scope(scope):
|
||||
other = accounting.reserve_attempt(_request(8.0))
|
||||
_spend(8.0)
|
||||
request = _request(1.5, "main")
|
||||
remaining = _wrapup_global_remaining()
|
||||
assert remaining == 2.0
|
||||
args = dict(request=request, root_cap_usd=root_cap, deciding_usd=0.0,
|
||||
global_remaining_usd=remaining)
|
||||
# An explicit profile's probe math is unchanged: one final fits, two do not.
|
||||
assert task_pacing.wrapup_reservation_fits(**args) is True
|
||||
assert task_pacing.wrapup_reservation_fits(**args, reservation_count=2) is False
|
||||
final = accounting.reserve_attempt(request)
|
||||
assert _wrapup_global_remaining() == 2.0 # the final's own hold is exposure, not spending
|
||||
unknown = accounting.reserve_attempt(_request(1.0))
|
||||
accounting.mark_dispatched(unknown)
|
||||
accounting.mark_unresolved(unknown, "provider outcome unknown")
|
||||
assert _wrapup_global_remaining() == 2.0 # an unknown outcome is neither spending nor a refund
|
||||
accounting.mark_dispatched(final)
|
||||
accounting.settle_attempt(final, {}, cost_usd=2.0, cost_final=False) # an estimate is known
|
||||
assert _wrapup_global_remaining() == 0.0
|
||||
with pytest.raises(accounting.BudgetExceeded) as refused:
|
||||
accounting.reserve_attempt(request)
|
||||
assert refused.value.limit_scope == "global"
|
||||
assert _wrapup_global_remaining() == 0.5
|
||||
accounting.mark_dispatched(other)
|
||||
accounting.mark_unresolved(other, "provider outcome unknown")
|
||||
assert _wrapup_global_remaining() == 0.5 # a dead/unknown call is not a refund
|
||||
accounting.release_attempt(final, "controlled test did not send")
|
||||
assert _wrapup_global_remaining() == 2.0
|
||||
|
||||
|
||||
@pytest.mark.parametrize("global_remaining,root_cap,deciding,expected", [
|
||||
|
|
@ -56,7 +67,7 @@ def test_a_concurrent_reservation_after_the_check_can_still_refuse_the_final(tmp
|
|||
request = _request(1.5, "main")
|
||||
assert task_pacing.wrapup_reservation_fits(request=request, root_cap_usd=None, deciding_usd=0,
|
||||
global_remaining_usd=_wrapup_global_remaining()) is True
|
||||
accounting.reserve_attempt(_request(1.0))
|
||||
_spend(2.0) # another task's known spend lands after the observation
|
||||
with pytest.raises(accounting.BudgetExceeded):
|
||||
accounting.reserve_attempt(request) # observation did not reserve a share
|
||||
|
||||
|
|
@ -64,7 +75,7 @@ def test_a_concurrent_reservation_after_the_check_can_still_refuse_the_final(tmp
|
|||
def test_explicit_scope_limit_and_canonical_root_own_the_wallet_read(tmp_path):
|
||||
scope = accounting.UsageScope(drive_root=tmp_path, task_id="main", root_task_id="main", global_limit_usd=10.0)
|
||||
with accounting.usage_scope(scope):
|
||||
accounting.reserve_attempt(_request(3.0))
|
||||
_spend(3.0)
|
||||
assert _wrapup_global_remaining() == 7.0
|
||||
with accounting.usage_scope(replace(scope, global_limit_usd=20.0)):
|
||||
assert _wrapup_global_remaining() == 17.0
|
||||
|
|
@ -110,7 +121,7 @@ def test_local_final_call_path_does_not_request_an_unneeded_wallet_projection(tm
|
|||
monkeypatch.setattr(accounting, "_ROOT_ACCOUNTING_TELEMETRY", {})
|
||||
scope = accounting.UsageScope(drive_root=tmp_path, task_id="local", root_task_id="local", global_limit_usd=10.0)
|
||||
with accounting.usage_scope(scope):
|
||||
accounting._stash_root_accounting("local", 0.0, None)
|
||||
accounting._stash_root_accounting("local", {"settled_usd": 0.0, "accounted_usd": 0.0}, None)
|
||||
assert loop._check_budget_limits(_ctx(active_use_local=True), 10.0,
|
||||
task_pacing.CostCeiling(state="active", ceiling_usd=5.0)) is None
|
||||
assert reads == []
|
||||
|
|
@ -186,13 +197,13 @@ def test_live_wallet_triggers_the_existing_final_call_path_without_a_root_cap(tm
|
|||
return "verified final", actual_ctx.accumulated_usage, actual_ctx.llm_trace
|
||||
monkeypatch.setattr(loop, "_forced_final_answer", finish)
|
||||
with accounting.usage_scope(scope):
|
||||
accounting.reserve_attempt(_request(6.0))
|
||||
_spend(6.0)
|
||||
assert loop._check_budget_limits(ctx, 10.0, ceiling) is None
|
||||
assert prepared == [] and admitted == []
|
||||
accounting.reserve_attempt(_request(2.0, "another-root"))
|
||||
_spend(2.0, "another-root")
|
||||
result = loop._check_budget_limits(ctx, 10.0, ceiling)
|
||||
assert result[0] == "verified final" and len(admitted) == 1 and len(prepared) == 1
|
||||
assert "$2.00 left across all tasks" in prepared[0]
|
||||
assert ctx.accumulated_usage["cost_stop_rail"] == "wrapup_reservation_last_fit"
|
||||
assert _wrapup_global_remaining() == 0.5
|
||||
assert _wrapup_global_remaining() == 2.0 # the admitted final's hold is not known spend
|
||||
accounting.release_attempt(admitted[0], "controlled final callback did not send")
|
||||
|
|
|
|||
|
|
@ -40,7 +40,7 @@ def _ctx(**overrides):
|
|||
|
||||
def _ceiling(root_cap):
|
||||
return task_pacing.resolve_cost_ceiling(
|
||||
None, normalize_budget_profile(None), root_cap_usd=root_cap,
|
||||
None, normalize_budget_profile({"cost_hard_stop_pct": 50}), root_cap_usd=root_cap,
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -49,7 +49,7 @@ def test_an_unfinished_no_tool_round_takes_the_same_over_ceiling_exit(monkeypatc
|
|||
from ouroboros import loop as loop_mod
|
||||
|
||||
ctx = _ctx()
|
||||
monkeypatch.setattr(loop_mod, "_loop_tree_accounting", lambda **_k: {"accounted_usd": 99.0})
|
||||
monkeypatch.setattr(loop_mod, "_loop_tree_accounting", lambda **_k: {"settled_usd": 99.0, "accounted_usd": 99.0})
|
||||
monkeypatch.setattr(
|
||||
loop_mod, "_forced_final_answer",
|
||||
lambda ctx_, **kwargs: ("wrapped up", ctx_.accumulated_usage, {"kwargs": kwargs}),
|
||||
|
|
@ -68,7 +68,7 @@ def test_it_never_reaches_for_the_tool_only_bookkeeping_or_delivery_arming(monke
|
|||
from ouroboros import loop as loop_mod
|
||||
|
||||
touched = []
|
||||
monkeypatch.setattr(loop_mod, "_loop_tree_accounting", lambda **_k: {"accounted_usd": 99.0})
|
||||
monkeypatch.setattr(loop_mod, "_loop_tree_accounting", lambda **_k: {"settled_usd": 99.0, "accounted_usd": 99.0})
|
||||
monkeypatch.setattr(loop_mod, "_prepare_post_tool_budget_context",
|
||||
lambda *a, **k: touched.append("prepare"))
|
||||
monkeypatch.setattr(loop_mod, "_note_nanny_delegate_activity",
|
||||
|
|
@ -85,7 +85,7 @@ def test_it_never_reaches_for_the_tool_only_bookkeeping_or_delivery_arming(monke
|
|||
def test_below_the_ceiling_and_a_disabled_ceiling_both_continue_unchanged(monkeypatch):
|
||||
from ouroboros import loop as loop_mod
|
||||
|
||||
monkeypatch.setattr(loop_mod, "_loop_tree_accounting", lambda **_k: {"accounted_usd": 1.0})
|
||||
monkeypatch.setattr(loop_mod, "_loop_tree_accounting", lambda **_k: {"settled_usd": 1.0, "accounted_usd": 1.0})
|
||||
assert _finish_no_tool_round_budget(_ctx(), None, _ceiling(50.0)) is None
|
||||
# An explicitly disabled ceiling is untouched by this unification.
|
||||
disabled = task_pacing.resolve_cost_ceiling(
|
||||
|
|
@ -105,7 +105,7 @@ def test_the_current_candidate_survives_the_budget_exit(monkeypatch):
|
|||
|
||||
candidate = SimpleNamespace(full_text="the answer so far")
|
||||
ctx = _ctx(delivery_candidate=candidate)
|
||||
monkeypatch.setattr(loop_mod, "_loop_tree_accounting", lambda **_k: {"accounted_usd": 99.0})
|
||||
monkeypatch.setattr(loop_mod, "_loop_tree_accounting", lambda **_k: {"settled_usd": 99.0, "accounted_usd": 99.0})
|
||||
monkeypatch.setattr(
|
||||
loop_mod, "_forced_final_answer",
|
||||
lambda ctx_, **kwargs: ("wrapped up", ctx_.accumulated_usage, {}),
|
||||
|
|
@ -166,7 +166,7 @@ def test_preparation_failure_pauses_and_resumes_the_no_tool_tail(full_loop, monk
|
|||
# route rebuilding is exercised by the owner-wait/ContextFit suites.
|
||||
monkeypatch.setattr(owner_wait, "rebind_restored_route", lambda *_a, **_k: (None, "max"))
|
||||
f.ctx._cost_ceiling = _ceiling(50.0)
|
||||
tree = {"accounted_usd": 49.0, "root_limit_usd": 50.0, "age_sec": 0.0}
|
||||
tree = {"settled_usd": 49.0, "accounted_usd": 49.0, "root_limit_usd": 50.0, "age_sec": 0.0}
|
||||
monkeypatch.setattr(loop, "_loop_tree_accounting", lambda **_k: dict(tree))
|
||||
monkeypatch.setattr(loop_budget, "_loop_tree_accounting", lambda **_k: dict(tree))
|
||||
monkeypatch.setattr(loop_budget, "_wrapup_global_remaining", lambda: 100.0)
|
||||
|
|
|
|||
|
|
@ -78,7 +78,112 @@ def test_legacy_child_never_labels_its_own_fallback_as_the_original_root(tmp_pat
|
|||
ctx = ToolContext(repo_dir=tmp_path, drive_root=tmp_path)
|
||||
ctx.task_id, ctx.task_depth, ctx.event_queue = "legacy-child", 1, queue.Queue()
|
||||
ctx.task_metadata = {"root_task_id": "root"}
|
||||
ctx._cost_ceiling = task_pacing.resolve_cost_ceiling(160.0, normalize_budget_profile(None), non_root_member=True)
|
||||
explicit = normalize_budget_profile({"cost_hard_stop_pct": 50})
|
||||
ctx._cost_ceiling = task_pacing.resolve_cost_ceiling(160.0, explicit, non_root_member=True)
|
||||
assert ctx._cost_ceiling.ceiling_usd == 80.0
|
||||
assert "original_root_ceiling_unavailable" in ctx._cost_ceiling.basis
|
||||
assert _schedule(ctx)["root_cost_ceiling_usd"] is None
|
||||
|
||||
|
||||
def test_an_ordinary_root_hands_its_children_no_number(tmp_path, monkeypatch):
|
||||
"""Owner 2026-10-07: an ordinary root has no early stop, so the real scheduler
|
||||
stamps nothing a child could later mistake for an authored ceiling."""
|
||||
monkeypatch.setattr(control, "load_settings", lambda: _settings(_api_row()))
|
||||
contract = build_task_contract({"id": "root", "type": "task"})
|
||||
ctx = ToolContext(repo_dir=tmp_path, drive_root=tmp_path)
|
||||
ctx.task_id, ctx.task_contract, ctx.event_queue = "root", contract, queue.Queue()
|
||||
ctx.task_metadata = {"root_task_id": "root", "task_contract": contract}
|
||||
scope = accounting.UsageScope(drive_root=tmp_path, task_id="root", root_task_id="root",
|
||||
global_limit_usd=480.0, root_limit_usd=400.0)
|
||||
with accounting.usage_scope(scope):
|
||||
ctx._cost_ceiling = task_pacing.resolve_task_cost_ceiling(ctx, 480.0)
|
||||
payload = _schedule(ctx)
|
||||
assert ctx._cost_ceiling.state == task_pacing.COST_CEILING_DISABLED
|
||||
assert payload["root_cost_ceiling_usd"] is None
|
||||
|
||||
|
||||
def _root_row(tmp_path, *, profile=None, metadata=None, contract=True):
|
||||
"""The canonical RUNNING row a root's start writes (``agent._persist_running_record``)."""
|
||||
from ouroboros.task_results import write_task_result
|
||||
|
||||
fields = {"metadata": dict(metadata or {})}
|
||||
if contract:
|
||||
fields["task_contract"] = build_task_contract(
|
||||
{"id": "root", "type": "task", "task_contract": {"budget_profile": profile or {}}})
|
||||
write_task_result(tmp_path, "root", "running", **fields)
|
||||
|
||||
|
||||
def _member(tmp_path, inherited):
|
||||
"""A child scheduled before the upgrade: an inherited number, no percentage of its own."""
|
||||
ctx = ToolContext(repo_dir=tmp_path, drive_root=tmp_path)
|
||||
ctx.task_id, ctx.task_contract = "child", build_task_contract({"id": "child", "type": "task"})
|
||||
scope = accounting.UsageScope(drive_root=tmp_path, task_id="child", root_task_id="root",
|
||||
parent_task_id="root", root_limit_usd=400.0, root_cost_ceiling_usd=inherited)
|
||||
with accounting.usage_scope(scope):
|
||||
return ctx, task_pacing.resolve_task_cost_ceiling(ctx, 480.0)
|
||||
|
||||
|
||||
def test_a_pre_upgrade_child_of_an_ordinary_root_loses_the_removed_default(tmp_path):
|
||||
"""The #1128 shape: $239.66 was half the wallet the ROOT saw. The root's own
|
||||
canonical contract proves it had no explicit percentage and no producer, so the
|
||||
child keeps no hidden ceiling; only the hard limits bind."""
|
||||
_root_row(tmp_path)
|
||||
ctx, ceiling = _member(tmp_path, 239.66)
|
||||
assert ceiling.state == task_pacing.COST_CEILING_DISABLED
|
||||
assert ceiling.basis == task_pacing.COST_BASIS_NO_DEFAULT_STOP
|
||||
assert task_pacing.cost_stop_authority(ctx) == task_pacing.COST_STOP_NONE
|
||||
|
||||
|
||||
def test_a_child_of_an_explicit_root_keeps_the_inherited_number(tmp_path):
|
||||
_root_row(tmp_path, profile={"cost_hard_stop_pct": 20})
|
||||
ctx, ceiling = _member(tmp_path, 39.0)
|
||||
assert ceiling.state == task_pacing.COST_CEILING_ACTIVE and ceiling.ceiling_usd == 39.0
|
||||
assert "root_resolved_ceiling" in ceiling.basis
|
||||
policy = task_pacing.cost_stop_policy(ctx)
|
||||
assert policy["authority"] == task_pacing.COST_STOP_EXPLICIT
|
||||
assert policy["profile"]["cost_hard_stop_pct"] == 20 # the Resume refresh's percentage
|
||||
|
||||
|
||||
def test_a_child_of_a_wake_uses_the_producer_allowance_not_an_old_margin(tmp_path):
|
||||
"""A wake started with $9 of allowance; the pre-upgrade child inherited $6 ($9 minus
|
||||
the old margin). The real restriction is the producer's $9, never widened further."""
|
||||
_root_row(tmp_path, metadata={"root_cost_ceiling_usd": 9.0, "initiator": "consciousness"})
|
||||
ctx, ceiling = _member(tmp_path, 6.0)
|
||||
assert ceiling.state == task_pacing.COST_CEILING_ACTIVE and ceiling.ceiling_usd == 9.0
|
||||
assert ceiling.planning_margin_usd is None
|
||||
assert task_pacing.cost_stop_authority(ctx) == task_pacing.COST_STOP_PRODUCER
|
||||
|
||||
|
||||
@pytest.mark.parametrize("damage", ["missing", "no_contract", "corrupt"])
|
||||
def test_unreadable_root_authority_keeps_the_inherited_number_provisionally(tmp_path, damage):
|
||||
"""Neither the default removed on a guess nor an authored stop widened: the
|
||||
member keeps the number it already carries, labelled ``legacy_policy_unverified``,
|
||||
and the next start re-reads the root once it is readable again."""
|
||||
from ouroboros.task_results import task_result_path
|
||||
|
||||
if damage == "no_contract":
|
||||
_root_row(tmp_path, contract=False)
|
||||
elif damage == "corrupt":
|
||||
path = task_result_path(tmp_path, "root")
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text("{not json", encoding="utf-8")
|
||||
ctx, ceiling = _member(tmp_path, 239.66)
|
||||
assert ceiling.state == task_pacing.COST_CEILING_ACTIVE and ceiling.ceiling_usd == 239.66
|
||||
assert ceiling.basis == task_pacing.COST_BASIS_POLICY_UNVERIFIED
|
||||
assert ceiling.root_cap_usd == 400.0 and ceiling.planning_margin_usd is None
|
||||
assert task_pacing.cost_stop_authority(ctx) == task_pacing.COST_STOP_UNKNOWN
|
||||
assert "legacy_policy_unverified" in task_pacing.cost_ceiling_disclosure(ceiling)["rule"]
|
||||
# Once the root's authority is readable (an ordinary root), the next start drops it.
|
||||
task_result_path(tmp_path, "root").unlink(missing_ok=True)
|
||||
_root_row(tmp_path)
|
||||
_ctx, later = _member(tmp_path, 239.66)
|
||||
assert later.state == task_pacing.COST_CEILING_DISABLED
|
||||
|
||||
|
||||
def test_an_unreadable_root_with_no_inherited_number_invents_no_ceiling(tmp_path):
|
||||
"""Nothing inherited, nothing to keep: the member resolves like any ordinary task
|
||||
(no stop), without reading the missing root at all."""
|
||||
ctx, ceiling = _member(tmp_path, None)
|
||||
assert ceiling.state == task_pacing.COST_CEILING_DISABLED and ceiling.ceiling_usd is None
|
||||
assert ceiling.basis == task_pacing.COST_BASIS_NO_DEFAULT_STOP
|
||||
assert task_pacing.cost_stop_authority(ctx) == task_pacing.COST_STOP_NONE
|
||||
|
|
|
|||
|
|
@ -557,10 +557,12 @@ def test_degraded_owner_line_bounds_each_reason():
|
|||
|
||||
|
||||
def test_redaction_filter_ssot_is_observability():
|
||||
"""process_logging defines no filter of its own; the bootstrap attaches observability's
|
||||
(tests/test_process_logging.py checks the attached class in a fresh interpreter)."""
|
||||
from ouroboros import process_logging
|
||||
from ouroboros.observability import SecretRedactingLogFilter
|
||||
|
||||
assert process_logging.SecretRedactingLogFilter is SecretRedactingLogFilter
|
||||
assert not [value for value in vars(process_logging).values()
|
||||
if isinstance(value, type) and issubclass(value, logging.Filter)]
|
||||
|
||||
|
||||
@pytest.mark.skipif(sys.platform == "win32", reason="POSIX pgrep/getuid semantics")
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ import pytest
|
|||
from ouroboros import budget_pause, loop, owner_wait, pricing, task_pacing, usage_accounting as accounting
|
||||
from ouroboros.contracts.task_contract import normalize_budget_profile
|
||||
from ouroboros.owner_mailbox import write_owner_message
|
||||
from tests.test_owner_wait_cold_loop import cold_registry
|
||||
from tests.test_owner_wait_cold_loop import EXPLICIT, cold_registry
|
||||
from tests.test_loop_transport_wait import _loop_kwargs
|
||||
|
||||
|
||||
|
|
@ -65,12 +65,15 @@ def test_cold_grant_checks_saved_budget_before_ordinary_dispatch(tmp_path, monke
|
|||
monkeypatch.setattr(loop, "_dispatch_round_model", ordinary)
|
||||
monkeypatch.setattr(loop, "_call_forced_model_once", forced)
|
||||
with accounting.usage_scope(scope):
|
||||
ceiling = task_pacing.resolve_cost_ceiling(200, normalize_budget_profile(None), root_cap_usd=50)
|
||||
# An explicit experiment profile's authored stop ($50 cap minus its margin): the
|
||||
# saved threshold a cold grant must re-check before ordinary dispatch.
|
||||
ceiling = task_pacing.resolve_cost_ceiling(200, normalize_budget_profile(EXPLICIT["budget_profile"]),
|
||||
root_cap_usd=50)
|
||||
with accounting.usage_scope(replace(scope, task_id="prior-child", parent_task_id="t-wait")):
|
||||
settle(46.9)
|
||||
|
||||
def registry():
|
||||
tools = cold_registry(tmp_path, monkeypatch, ceiling)
|
||||
tools = cold_registry(tmp_path, monkeypatch, ceiling, EXPLICIT)
|
||||
tools._ctx._owner_wait_requested = ""
|
||||
tools._ctx.current_chat_id, tools._ctx.current_task_type = 1, "task"
|
||||
tools._ctx.task_model_override = "same-model"
|
||||
|
|
|
|||
|
|
@ -46,10 +46,23 @@ def test_provider_isolation_preserves_event_loop_self_pipe(monkeypatch):
|
|||
event_loop.close()
|
||||
|
||||
|
||||
def cold_registry(tmp_path, monkeypatch, ceiling=None):
|
||||
EXPLICIT = {"budget_profile": {"cost_hard_stop_pct": 50}} # an authored experiment profile
|
||||
|
||||
|
||||
def _settle(amount, *, provider="openai"):
|
||||
"""KNOWN spend under the bound scope: the only money a limit decides on (#1487)."""
|
||||
held = accounting.reserve_attempt(accounting.AttemptRequest(
|
||||
model="fixture", provider=provider, reservation_usd=amount))
|
||||
accounting.mark_dispatched(held)
|
||||
accounting.settle_attempt(held, {}, cost_usd=amount, cost_final=True)
|
||||
|
||||
|
||||
def cold_registry(tmp_path, monkeypatch, ceiling=None, contract=None):
|
||||
registry = ToolRegistry(repo_dir=tmp_path, drive_root=tmp_path)
|
||||
ctx = registry._ctx
|
||||
ctx.task_id, ctx.task_attempt = "t-wait", 1
|
||||
if contract is not None:
|
||||
ctx.task_contract = dict(contract)
|
||||
ctx.active_model, ctx.active_effort = "same-model", "high"
|
||||
ctx.active_use_local, ctx.active_context_mode = False, "max"
|
||||
ctx._owner_wait_requested = "quiz"
|
||||
|
|
@ -113,20 +126,19 @@ def test_cold_switched_model_keeps_actual_overflow_reclaim_and_retry(tmp_path, m
|
|||
def test_cold_loop_keeps_original_ceiling_while_live_wallet_still_binds(tmp_path, monkeypatch):
|
||||
scope = accounting.UsageScope(drive_root=tmp_path, task_id="t-wait", root_task_id="t-wait",
|
||||
global_limit_usd=200.0, root_limit_usd=50.0)
|
||||
explicit = normalize_budget_profile(EXPLICIT["budget_profile"])
|
||||
with accounting.usage_scope(scope):
|
||||
original = task_pacing.resolve_cost_ceiling(200.0, normalize_budget_profile(None), root_cap_usd=50.0)
|
||||
original = task_pacing.resolve_cost_ceiling(200.0, explicit, root_cap_usd=50.0)
|
||||
assert original.ceiling_usd == 47.0
|
||||
accounting.reserve_attempt(accounting.AttemptRequest(model="fixture", provider="openai", reservation_usd=10.0))
|
||||
registry = cold_registry(tmp_path, monkeypatch, original)
|
||||
others = []
|
||||
_settle(10.0)
|
||||
registry = cold_registry(tmp_path, monkeypatch, original, EXPLICIT)
|
||||
for index in range(4):
|
||||
with accounting.usage_scope(replace(scope, task_id=f"other-{index}", root_task_id=f"other-{index}")):
|
||||
others.append(accounting.reserve_attempt(accounting.AttemptRequest(
|
||||
model="fixture", provider="openai", reservation_usd=40.0)))
|
||||
fresh = task_pacing.resolve_cost_ceiling(30.0, normalize_budget_profile(None), root_cap_usd=50.0)
|
||||
_settle(40.0)
|
||||
fresh = task_pacing.resolve_cost_ceiling(30.0, explicit, root_cap_usd=50.0)
|
||||
assert fresh.ceiling_usd == 15.0
|
||||
registry._ctx._cost_ceiling = fresh # new context-builder disclosure before loading saved messages
|
||||
accounting.reserve_attempt(accounting.AttemptRequest(model="fixture", provider="openai", reservation_usd=6.0))
|
||||
_settle(6.0)
|
||||
captured = []
|
||||
soft_land = loop._soft_land_exhausted_ceiling
|
||||
forced = []
|
||||
|
|
@ -155,18 +167,35 @@ def test_cold_loop_keeps_original_ceiling_while_live_wallet_still_binds(tmp_path
|
|||
monkeypatch.setattr(loop, "_soft_land_exhausted_ceiling", inspect_ceiling)
|
||||
result, _, _ = loop.run_llm_loop(**{**_loop_kwargs(tmp_path, registry, []), "budget_remaining_usd": 30.0})
|
||||
assert result == "ceiling retained" and captured == [original]
|
||||
permitted = accounting.reserve_attempt(accounting.AttemptRequest(model="fixture", provider="openai", reservation_usd=1.0))
|
||||
_settle(24.0) # the live wallet's known spend reaches its $200
|
||||
with pytest.raises(accounting.BudgetExceeded) as global_refusal:
|
||||
accounting.reserve_attempt(accounting.AttemptRequest(model="fixture", provider="openai", reservation_usd=25.0))
|
||||
accounting.reserve_attempt(accounting.AttemptRequest(model="fixture", provider="openai", reservation_usd=0.01))
|
||||
assert global_refusal.value.limit_scope == "global"
|
||||
accounting.release_attempt(permitted, "test did not send")
|
||||
for held in others:
|
||||
accounting.release_attempt(held, "test did not send")
|
||||
with accounting.usage_scope(replace(scope, global_limit_usd=1000.0)):
|
||||
_settle(10.0) # the tree's known spend reaches its $50 cap ($10 + $6 + $24 + $10)
|
||||
with pytest.raises(accounting.BudgetExceeded) as root_refusal:
|
||||
accounting.reserve_attempt(accounting.AttemptRequest(model="fixture", provider="openai", reservation_usd=35.0))
|
||||
accounting.reserve_attempt(accounting.AttemptRequest(model="fixture", provider="openai", reservation_usd=0.01))
|
||||
assert root_refusal.value.limit_scope == "root"
|
||||
|
||||
|
||||
def test_an_ordinary_cold_restore_drops_a_saved_default_ceiling(tmp_path, monkeypatch):
|
||||
"""Owner 2026-10-07 for existing installs: an owner wait saved before the upgrade
|
||||
carries the removed default ($47 = cap minus margin). With no explicit profile and no
|
||||
producer, the planned-restart restore resumes without it and says so."""
|
||||
from ouroboros.owner_wait import load_owner_wait
|
||||
|
||||
scope = accounting.UsageScope(drive_root=tmp_path, task_id="t-wait", root_task_id="t-wait",
|
||||
global_limit_usd=200.0, root_limit_usd=50.0)
|
||||
legacy = task_pacing.CostCeiling(state="active", ceiling_usd=47.0, root_cap_usd=50.0,
|
||||
planning_margin_usd=3.0, basis="min(global_pct, root_cap_minus_margin)")
|
||||
with accounting.usage_scope(scope):
|
||||
registry = cold_registry(tmp_path, monkeypatch, legacy)
|
||||
load_owner_wait(registry._ctx)
|
||||
restored = registry._ctx._cost_ceiling
|
||||
assert restored.state == task_pacing.COST_CEILING_DISABLED and restored.ceiling_usd is None
|
||||
assert restored.basis == "no_default_cost_stop(saved default stop $47.00 removed)"
|
||||
|
||||
|
||||
def test_agent_cold_context_shows_saved_ceiling_and_current_wallet(tmp_path, monkeypatch):
|
||||
from ouroboros import agent as agent_module
|
||||
from tests._context_shared import _make_health_env
|
||||
|
|
@ -176,16 +205,14 @@ def test_agent_cold_context_shows_saved_ceiling_and_current_wallet(tmp_path, mon
|
|||
scope = accounting.UsageScope(drive_root=tmp_path, task_id="t-wait", root_task_id="t-wait",
|
||||
global_limit_usd=200.0, root_limit_usd=50.0)
|
||||
with accounting.usage_scope(scope):
|
||||
original = task_pacing.resolve_task_cost_ceiling(None, 200.0)
|
||||
original = task_pacing.resolve_task_cost_ceiling(SimpleNamespace(task_contract=EXPLICIT), 200.0)
|
||||
assert original.ceiling_usd == 47.0
|
||||
registry = cold_registry(tmp_path, monkeypatch, original)
|
||||
registry = cold_registry(tmp_path, monkeypatch, original, EXPLICIT)
|
||||
handoff = registry._ctx.owner_wait_resume
|
||||
accounting.reserve_attempt(accounting.AttemptRequest(
|
||||
model="fixture", provider="openai", reservation_usd=10.0))
|
||||
_settle(10.0)
|
||||
for index in range(4):
|
||||
with accounting.usage_scope(replace(scope, task_id=f"other-{index}", root_task_id=f"other-{index}")):
|
||||
accounting.reserve_attempt(accounting.AttemptRequest(
|
||||
model="fixture", provider="openai", reservation_usd=40.0))
|
||||
_settle(40.0)
|
||||
|
||||
env = _make_health_env(tmp_path)
|
||||
env.branch_dev, env.budget_drive_root = "ouroboros", tmp_path
|
||||
|
|
@ -212,6 +239,7 @@ def test_agent_cold_context_shows_saved_ceiling_and_current_wallet(tmp_path, mon
|
|||
monkeypatch.setattr(agent_module, "build_llm_messages", runtime_messages)
|
||||
task = {"id": "t-wait", "root_task_id": "t-wait", "delegation_role": "root",
|
||||
"type": "task", "text": "Continue saved work", "_attempt": 1,
|
||||
"budget_profile": dict(EXPLICIT["budget_profile"]),
|
||||
"_owner_wait_resume": handoff, "budget_drive_root": str(tmp_path)}
|
||||
ctx, messages, caps = agent._prepare_task_context(task)
|
||||
assert caps["budget_remaining"] == 30.0
|
||||
|
|
@ -232,7 +260,8 @@ def test_agent_cold_context_shows_saved_ceiling_and_current_wallet(tmp_path, mon
|
|||
"budget_remaining_usd": caps["budget_remaining"],
|
||||
})
|
||||
assert result == "Continued saved work" and len(captured) == 1
|
||||
_settle(30.0) # known spend reaches the $200 wallet
|
||||
with pytest.raises(accounting.BudgetExceeded) as refused:
|
||||
accounting.reserve_attempt(accounting.AttemptRequest(
|
||||
model="fixture", provider="openai", reservation_usd=31.0))
|
||||
model="fixture", provider="openai", reservation_usd=0.01))
|
||||
assert refused.value.limit_scope == "global"
|
||||
|
|
|
|||
|
|
@ -208,7 +208,8 @@ def test_api_state_warm_path_replays_ledger_zero_times_and_projects_once(
|
|||
# be the de-triplication regression this test exists to catch).
|
||||
assert len(counters["projections"]) == 1
|
||||
assert "budget_reserve_usd" in payload["evolution_state"] # real snapshot ran
|
||||
assert payload["spent_usd"] == 1.25 # settled 0.25 + reserved bound 1.0
|
||||
assert payload["spent_usd"] == 0.25 # known (settled) spend; the $1.00 hold rides in accounting
|
||||
assert payload["accounting"]["accounted_usd"] == 1.25
|
||||
assert payload["accounting"]["authority"] == "physical_attempt_ledger"
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -316,15 +316,42 @@ def test_iter_jsonl_chain_objects_chronological(tmp_path):
|
|||
assert [row["i"] for row in iter_jsonl_chain_objects(path)] == [0, 1, 2]
|
||||
|
||||
|
||||
def test_launcher_stdout_copy_is_size_capped():
|
||||
"""Source pin (CPL4-C5): the pipe-copy thread rotates agent_stdout.log."""
|
||||
import inspect
|
||||
@pytest.mark.serial
|
||||
def test_launcher_stdout_copy_is_size_capped(monkeypatch, tmp_path):
|
||||
"""CPL4-C5: the launcher drains a real server pipe into a rotated agent_stdout.log.
|
||||
|
||||
The copier's own byte, failure and gap rules: tests/test_capped_output_copy.py.
|
||||
"""
|
||||
import subprocess
|
||||
import threading
|
||||
|
||||
import launcher
|
||||
from ouroboros import process_logging
|
||||
from ouroboros.process_logging import SERVER_LOG_MAX_BYTES
|
||||
|
||||
src = inspect.getsource(launcher.start_agent)
|
||||
assert "max_bytes = 2 * 1024 * 1024" in src
|
||||
assert 'os.replace(log_path, log_path.with_name(f"{log_path.name}.1"))' in src
|
||||
data_dir, repo_dir = tmp_path / "data", tmp_path / "repo"
|
||||
repo_dir.mkdir()
|
||||
(repo_dir / "server.py").write_text("", encoding="utf-8")
|
||||
line = b"x" * 1023 + b"\n"
|
||||
producer = f"import sys\nfor _ in range(5120): sys.stdout.buffer.write({line!r})\n"
|
||||
monkeypatch.setattr(launcher, "IS_WINDOWS", False)
|
||||
monkeypatch.setattr(launcher, "DATA_DIR", data_dir)
|
||||
monkeypatch.setattr(launcher, "REPO_DIR", repo_dir)
|
||||
monkeypatch.setattr(launcher, "_load_settings", lambda: {})
|
||||
monkeypatch.setattr(launcher, "_apply_settings_to_env", lambda _settings: None)
|
||||
monkeypatch.setattr(launcher, "process_group_id", lambda _pid: None)
|
||||
monkeypatch.setattr(launcher, "_hidden_popen",
|
||||
lambda _cmd, **kwargs: subprocess.Popen([sys.executable, "-c", producer], **kwargs))
|
||||
drained, copy = threading.Event(), process_logging.copy_capped_output
|
||||
monkeypatch.setattr(process_logging, "copy_capped_output", lambda *a, **k: (copy(*a, **k), drained.set()))
|
||||
|
||||
proc = launcher.start_agent(port=9877)
|
||||
assert proc.wait(timeout=120) == 0 and drained.wait(timeout=120)
|
||||
proc.stdout.close()
|
||||
log_path = data_dir / "logs" / "agent_stdout.log"
|
||||
rotated = [log_path.with_name(f"agent_stdout.log.{index}") for index in (2, 1)]
|
||||
assert [path.stat().st_size for path in (*rotated, log_path)] == [SERVER_LOG_MAX_BYTES] * 2 + [1024 * 1024]
|
||||
assert b"".join(path.read_bytes() for path in (*rotated, log_path)) == line * 5120
|
||||
|
||||
|
||||
def test_supervisor_tick_rotates_the_train():
|
||||
|
|
|
|||
|
|
@ -339,7 +339,10 @@ def test_native_preflight_uses_existing_episode_and_monetary_custody(preflight,
|
|||
assert skipped["status"] == "bypassed"
|
||||
if end != "success":
|
||||
projection = usage.usage_projection(preflight.drive_root, global_limit_usd=10.0)
|
||||
assert projection["accounted_usd"] == 1.0 and projection["remaining_known_usd"] == 9.0
|
||||
# The unknown outcome keeps its $1 bound as disclosed exposure; it is not
|
||||
# known spend, so the room the limit decides on is untouched (#1487).
|
||||
assert projection["accounted_usd"] == 1.0 and projection["unresolved_upper_bound_usd"] == 1.0
|
||||
assert projection["settled_usd"] == 0.0 and projection["remaining_known_usd"] == 10.0
|
||||
if rows:
|
||||
retained = load_state(preflight.drive_root).advisory_runs[0]
|
||||
assert retained.execution["operation_state"] == "custody_lost"
|
||||
|
|
|
|||
|
|
@ -309,9 +309,11 @@ def test_a_budget_repricing_arms_presence_only_for_the_forced_call_it_commits(
|
|||
from ouroboros import task_pacing
|
||||
from ouroboros.contracts.task_contract import normalize_budget_profile
|
||||
|
||||
ceiling = task_pacing.resolve_cost_ceiling(None, normalize_budget_profile(None), root_cap_usd=50.0)
|
||||
ceiling = task_pacing.resolve_cost_ceiling(None, normalize_budget_profile({"cost_hard_stop_pct": 50}), root_cap_usd=50.0)
|
||||
monkeypatch.setattr(loop, "_resolve_task_cost_ceiling", lambda *_a: ceiling)
|
||||
monkeypatch.setattr(loop, "_loop_tree_accounting", lambda **_k: {"accounted_usd": 20.0})
|
||||
# The last-fit rail is an explicit profile's authored rail (#1128): that task's authority.
|
||||
monkeypatch.setattr(task_pacing, "cost_stop_authority", lambda _ctx: task_pacing.COST_STOP_EXPLICIT)
|
||||
monkeypatch.setattr(loop, "_loop_tree_accounting", lambda **_k: {"settled_usd": 20.0, "accounted_usd": 20.0})
|
||||
# proxy last-fit -> exact probe last-fit -> the prepared candidate decides
|
||||
answers = iter((True, False, True, False, *prepared_fits))
|
||||
monkeypatch.setattr(task_pacing, "wrapup_reservation_fits", lambda **_k: next(answers, True))
|
||||
|
|
|
|||
|
|
@ -48,6 +48,7 @@ def test_server_bootstrap_writes_server_log_through_redacting_handlers(tmp_path)
|
|||
logs = tmp_path / "logs"
|
||||
completed = _run(f"""
|
||||
import json, logging, pathlib, threading
|
||||
from ouroboros.observability import SecretRedactingLogFilter
|
||||
from ouroboros.process_logging import configure_process_logging
|
||||
configure_process_logging(drive_logs=pathlib.Path({str(logs)!r}))
|
||||
configure_process_logging(drive_logs=pathlib.Path({str(logs)!r})) # idempotent
|
||||
|
|
@ -57,7 +58,7 @@ def test_server_bootstrap_writes_server_log_through_redacting_handlers(tmp_path)
|
|||
handler.flush()
|
||||
print("STATE", json.dumps({{
|
||||
"handlers": sorted(type(h).__name__ for h in root.handlers),
|
||||
"filtered": all(any(type(f).__name__ == "SecretRedactingLogFilter" for f in h.filters)
|
||||
"filtered": all(any(type(f) is SecretRedactingLogFilter for f in h.filters)
|
||||
for h in root.handlers),
|
||||
"httpx": logging.getLogger("httpx").level,
|
||||
"thread_hook": getattr(threading.excepthook, "_ouroboros_hook", False),
|
||||
|
|
@ -297,15 +298,32 @@ def test_every_uvicorn_config_in_the_server_leaves_logging_to_the_root_handlers(
|
|||
|
||||
|
||||
def test_the_launcher_routes_its_uncaught_exceptions_through_the_shared_hooks():
|
||||
"""The frozen launcher keeps its own launcher.log handlers and shares only the
|
||||
uncaught-exception hooks; the call sits after its redaction filter loop."""
|
||||
"""The frozen launcher keeps its own launcher.log handlers and shares the
|
||||
uncaught-exception hooks, called after its guarded redaction filter loop, and
|
||||
the output copier, imported unguarded because the module loads only the stdlib."""
|
||||
tree = ast.parse((REPO / "launcher.py").read_text(encoding="utf-8"))
|
||||
calls = [node for node in ast.walk(tree) if isinstance(node, ast.Call)
|
||||
and isinstance(node.func, ast.Name) and node.func.id == "install_exception_hooks"]
|
||||
assert len(calls) == 1
|
||||
imports = [node for node in ast.walk(tree) if isinstance(node, ast.ImportFrom)
|
||||
and node.module == "ouroboros.process_logging"]
|
||||
assert imports and all(alias.name == "install_exception_hooks" for node in imports for alias in node.names)
|
||||
imports = {alias.name: node for node in ast.walk(tree) if isinstance(node, ast.ImportFrom)
|
||||
and node.module == "ouroboros.process_logging" for alias in node.names}
|
||||
assert set(imports) == {"install_exception_hooks", "copy_capped_output"}
|
||||
start_agent = next(node for node in tree.body if isinstance(node, ast.FunctionDef) and node.name == "start_agent")
|
||||
assert imports["copy_capped_output"] in start_agent.body # Unguarded, before the server is spawned.
|
||||
|
||||
|
||||
@pytest.mark.serial
|
||||
def test_the_module_loads_only_the_standard_library():
|
||||
"""The launcher drains and records a server whose own imports are broken (the crash
|
||||
output agent_stdout.log exists for), so nothing project-owned loads with the copier."""
|
||||
tree = ast.parse((REPO / "ouroboros" / "process_logging.py").read_text(encoding="utf-8"))
|
||||
roots = {alias.name.split(".")[0] for node in tree.body if isinstance(node, ast.Import) for alias in node.names}
|
||||
roots |= {node.module.split(".")[0] for node in tree.body if isinstance(node, ast.ImportFrom)}
|
||||
assert roots - {"__future__"} <= set(sys.stdlib_module_names)
|
||||
completed = subprocess.run(
|
||||
[sys.executable, "-c", "import sys, ouroboros.process_logging; print('ouroboros.observability' in sys.modules)"],
|
||||
cwd=REPO, env={**os.environ, "PYTHONPATH": str(REPO)}, capture_output=True, text=True, timeout=120)
|
||||
assert completed.stdout.strip() == "False", completed.stderr[-2000:]
|
||||
|
||||
|
||||
@pytest.mark.serial
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue