mirror of
https://github.com/razzant/ouroboros.git
synced 2026-10-23 07:09:35 +00:00
Merge pull request #1872 from ouroboros-agent/contrib/phantom-cancel-card-20261015
Retire the false cancel notice in Main and route every task row to its own room
This commit is contained in:
commit
e57ed17951
84 changed files with 2422 additions and 866 deletions
|
|
@ -654,9 +654,11 @@ class IsolatedServer:
|
|||
callers (a driver cleaning up after its own ``wait_task`` deadline) keep the
|
||||
byte-identical legacy single-task request they have always sent.
|
||||
|
||||
The cascade lane answers only once the subtree is actually torn down, hence the
|
||||
wide default timeout. Returns the ``_api_status`` envelope; the refusal statuses are
|
||||
part of the contract under test, so nothing is raised or swallowed.
|
||||
A hard Stop records the intent first and answers 200 once the teardown was driven,
|
||||
404 when nothing was live, or 202 ``cancel_state=pending`` while the intent is still
|
||||
open for the watchdog; the cascade lane still waits for the kills it asked for, hence
|
||||
the wide default timeout. Returns the ``_api_status`` envelope; the refusal statuses
|
||||
are part of the contract under test, so nothing is raised or swallowed.
|
||||
"""
|
||||
body: dict = {}
|
||||
if cascade:
|
||||
|
|
|
|||
|
|
@ -242,7 +242,7 @@ In a Project response, `window.latest_message` names the latest eligible non-own
|
|||
- **Deliveries:** `msg_type` is `document`, `photo`, `video`, `links` or `quiz`, with `filename`, `mime`, `download_url`, `caption` and `size_bytes`; `actions` and `title` for links; `quiz` for a question (`quiz.quiz_id`, `quiz.state`, the recorded answer).
|
||||
- **Progress rows** (`is_progress: true`) carry the same task-progress facts live `chat` progress frames carry (`ChatOutbound`).
|
||||
- **Rows that are not messages.** Do not show these as bubbles:
|
||||
- replay evidence: `system_type` `quiz_answer` or `task_evidence`, or a row with `summary_kind` and `historical_terminal`. They carry facts for cards whose rows lie on other pages.
|
||||
- replay evidence: `system_type` `quiz_answer` or `task_evidence`, or a row with `summary_kind` and `historical_terminal`. They carry facts for cards whose rows lie on other pages. A `task_evidence` carrier may be an in-room parent's only evidence (zero tool calls included; `task_terminal_status` only when the parent is terminal): a client mints a parent card only from the parent's own rows, carrier or live state, never from a child's `parent_task_id`; a child whose parent has no such evidence here renders as its own card and re-nests when that evidence arrives.
|
||||
- `system_type: "task_model_wait"`: a model wait's latest state, in `model_waits`, for the task's card.
|
||||
- **A rebuilt row:** `lifecycle_virtual: true` describes the skill operation running now. It is rebuilt with a fresh `ts` on every read and has no `history_id`, so replace it rather than merge it.
|
||||
- **Derived and synthesized rows:** in Main a Project question appears as `system_type: "project_question_pointer"` (with `quiz_id`), carrying the question row's `history_id`. A Project's start message rebuilt from its binding (`origin_projected: true`) has no `history_id`. A synthesized `origin_omission` System notice says further archived origins were not rendered; it also has no stored `history_id`.
|
||||
|
|
@ -340,11 +340,11 @@ data: {"seq":12,"type":"task_result","task_id":"…","data":{…}}
|
|||
## 8. Stop, pause, continue, resume
|
||||
|
||||
**Stop.** `POST /api/tasks/{id}/cancel` takes an optional `TaskCancelRequest`: `cascade`, `stop_policy` and `stop_action_id`.
|
||||
- **Safe retries:** `stop_action_id` names one Stop action. While that action is pending, an identical request joins it and reusing the id with different arguments answers 409. An immediate Stop normally answers after teardown; once the task is no longer active, even an identical retry can answer 404. Read task detail to distinguish the recorded outcome.
|
||||
- **Graceful stop:** `202` with `cancel_state: "pending"` means a graceful stop is under way.
|
||||
- **Already stopped:** `404` for a task that is no longer active also covers repeating a Stop that already finished.
|
||||
- **Nothing cancelled:** `503 cancel_intent_write_failed` means nothing was cancelled, so retry.
|
||||
- **Still running:** a `503` saying the cancellation "did not settle" means the stop was recorded but the task is still running. Read `GET /api/tasks/{id}` (`cancel_state`) before anything else.
|
||||
- **Safe retries:** `stop_action_id` names one Stop action. While that action is pending, an identical request joins it and reusing the id with different arguments answers 409. An immediate Stop records the stop first and answers after it drove the teardown; once the task is no longer active, even an identical retry can answer 404. Read task detail to distinguish the recorded outcome.
|
||||
- **Stopped:** `200` means the teardown settled (with `cascade: true` when the whole tree was stopped).
|
||||
- **Recorded, still stopping:** `202` with `cancel_state: "pending"` means the stop is recorded and the supervisor finishes it: a graceful stop under way, or an immediate one whose teardown is not confirmed yet (`reason_code: "teardown_unconfirmed"`, with the stop's `request_id`). No retry is needed: read `GET /api/tasks/{id}` (`cancel_state`) until it settles; an immediate Stop still hardens a graceful one.
|
||||
- **Nothing to stop:** `404` means the task already finished (completion wins), the id names no task, or a repeated Stop already finished. A task that was created but never started is stopped, not 404.
|
||||
- **Nothing recorded:** `503 cancel_intent_write_failed` means nothing was recorded or cancelled, so retry; `503 cancel_intent_projection_corrupt` cannot succeed until the projection file is repaired.
|
||||
- **When to offer Stop:** chat frames carry `cancelable` to say when Stop applies. Do not offer it otherwise.
|
||||
|
||||
`stop_policy` is `immediate` (also the omitted/empty default) or `finalize_then_cancel`. A graceful request cannot soften a previously recorded immediate Stop. Task detail has `cancel_state:"pending"` while cancellation is owed, otherwise the field is absent; there is no public `cancel_state:"settled"`.
|
||||
|
|
|
|||
|
|
@ -57,7 +57,7 @@ Messages, media bubbles and task-card roots sort by numeric timestamps, ties by
|
|||
|
||||
Progress callbacks capture address, lineage and attempt at binding, so a late review cannot inherit another turn; unkeyed progress and summary rows never mutate a live card. `terminal_projection.py` owes the Project row and the deferred Main mirror together: `write_task_result` stamps `canonical_terminal_projection_origin=terminal_transition` only on an accepted root terminal transition, so markerless historical rows are never replayed; publication and retirement require `headless.terminal_task_files_ready`: synthesis alone does not prove artifact completion; publication errors are isolated from task-done and retried at checkpoint, startup and the `server_maintenance.py` sweep; the bounded outbox and at-least-once transport preclude exactly-once.
|
||||
|
||||
Main receives its own dialogue, Project quiz mirrors and `project_dialogue.MAIN_PINNED_ROW_TYPES` (`project_handoff`, `project_started`, `project_completion_summary`, and a root's `main_notice` from `send_user_message(destination="main")`), pinned at supervisor send and `room_membership` replay; other Project traffic stays in its room. `projects_registry.mirrored_answer` carries the whole `completion_answer` only for a nonempty `model_final` in a settled phase within `MIRRORED_ANSWER_MAX_CHARS`; otherwise it is absent, never cut (`project_answer.js::decorateProjectRow`; DESIGN "Project completion mirror"). `ORIGIN_ADDRESSED_NOTICE_TYPES` (`task_not_started`, `task_start_unconfirmed`, `steer_not_delivered`) stay in their issuing chat on replay despite target binding.
|
||||
Main receives its own dialogue, Project quiz mirrors and `project_dialogue.MAIN_PINNED_ROW_TYPES` (`project_handoff`, `project_started`, `project_completion_summary`, and a root's `main_notice` from `send_user_message(destination="main")`), pinned at supervisor send and `room_membership` replay; other task rows follow their task's room (`project_chat_for_task_tree`). `projects_registry.mirrored_answer` carries the whole `completion_answer` only for a nonempty `model_final` in a settled phase within `MIRRORED_ANSWER_MAX_CHARS`; otherwise it is absent, never cut (`project_answer.js::decorateProjectRow`; DESIGN "Project completion mirror"). `ORIGIN_ADDRESSED_NOTICE_TYPES` (`task_not_started`, `task_start_unconfirmed`, `steer_not_delivered`) stay in their issuing chat on replay despite target binding.
|
||||
|
||||
Every unanswered Project quiz is mirrored into Main from one keyed pointer, without a second durable message: `chat_decision.buildQuizCard` renders the Project's own form plus the Project reference (DESIGN "Project question mirror") over the existing decision ingress; only Main's copy retires after the first confirmed `answered` observation, and a settled question never reopens. `project_dialogue.project_question_pointer`, live `message_bus.send_quiz` and the census `required_question` carry the same question fields, so a narrower redelivery never blanks a known field; `question_presentation.js` owns the lifecycle wording, with the Python twin `project_dialogue.QUESTION_STATUS` pinned by `question_presentation_parity.json`. Waiting requires positive evidence, carried by history too: ordinary input can resume a task without a `quiz_state` frame.
|
||||
|
||||
|
|
@ -115,7 +115,7 @@ A card's cost is one scope's accounting facts (amount, finality, subtree, reserv
|
|||
|
||||
#### Stop and cancel controls
|
||||
|
||||
Cancel appears only on unfinished, unconverted root cards whose host attests `cancelable=true`; card shape grants no control, and native direct tasks follow §5 custody. The shared dropdown sends `cascade:true`: hard stop waits for teardown or refusal, Wrap up records durable finalize intent and returns 202 pending. Completion wins, a missing live task reconciles from durable state, and unproven death stays a visible refusal. Reason lines: `log_events.taskReasonDetail` over `cancel_origin` (§3 Task cards, errors and reason lines).
|
||||
Cancel appears only on unfinished, unconverted root cards whose host attests `cancelable=true`; card shape grants no control, and native direct tasks follow §5 custody. The shared dropdown sends `cascade:true`; Wrap up, or a Stop now not yet confirmed, returns 202 pending. Completion wins and a missing live task reconciles from durable state. Reason lines: `log_events.taskReasonDetail` over `cancel_origin` (§3 Task cards, errors and reason lines).
|
||||
|
||||
#### Turning a card into a Project
|
||||
|
||||
|
|
|
|||
|
|
@ -46,7 +46,7 @@ Cancellation is intent-then-custody (§10 invariants 14-15): intent and outcome
|
|||
|
||||
1. **Intent.** Every cancel ingress (the `cancel_task` tool, HTTP single and cascade, evolution stop, project deletion, the cascade sweep, boot recovery of `cancel_requested` files) writes one durable row through `ouroboros/cancel_intents.request_cancel` into locked `state/cancel_intents.json`. A failed write tears nothing down (tool `CANCEL_INTENT_WRITE_FAILED`, HTTP 503): no teardown without a watchdog-replayable fence. The owner's evolution stop is sticky: the campaign stays open under `evolution_owner_stopped`, which only an owner start clears.
|
||||
2. **Scope.** Recorded scope only widens (single to cascade; Stop-now cannot shrink a cascade). Timeout reaping is not a cancel ingress: the reaper keeps its own `reaping` custody; cancellation is reserved for explicit intent.
|
||||
3. **Claim.** `supervisor.task_lifecycle.cancel_task_custody` is the one settle owner: it claims the intent before any mutation and a refused claim exits untouched; racing custodies never double-settle. A live direct-chat turn stops through the same custody (`supervisor/worker_chat_lane.py` writes one typed `finalize_now` control; the loop ends at its next round boundary with no further model call); a Stop during post-task synthesis discloses skipped paid stages as `post_task_stop_reason`. A still-live turn releases the claim for the sweep, never publishing a fabricated `cancelled` row.
|
||||
3. **Claim.** `supervisor.task_lifecycle.cancel_task_custody` is the one settle owner: it claims the intent before any mutation and a refused claim exits untouched; racing custodies never double-settle. The `cancel_task` event only wakes the intent. A live direct-chat turn stops through the same custody (`supervisor/worker_chat_lane.py` writes one typed `finalize_now` control; the loop ends at its next round boundary with no further model call); a Stop during post-task synthesis discloses skipped paid stages as `post_task_stop_reason`. A still-live turn releases the claim for the sweep, never publishing a fabricated `cancelled` row.
|
||||
4. **Kill and re-check.** Custody confirms process death, then re-reads the child's settled result. Natural completion wins: a child finished before the kill keeps its result, artifacts and cost, and the cancel settles as already-settled.
|
||||
5. **Reconcile and capture.** Open delegated runs are re-audited from durable custody rows; workspace artifacts are captured from the real tree, a failed or unrunnable capture being `failed`, never `missing`.
|
||||
6. **Settle.** The result carries reconstructed-or-unknown cost, never a fabricated `$0`; `parent_decision` is stamped only here. `cancel_publication` keeps the provenance as `cancel_origin` through the terminal event, task detail, history and result reads. HTTP proves transport, never a personal owner.
|
||||
|
|
@ -59,7 +59,7 @@ Effective results show typed `cancel_state: "pending"` with `cancel_reason` unti
|
|||
|
||||
### Stop policy and hurry
|
||||
|
||||
Stop policy is an axis on the same durable intent, independent of cascade scope. An omitted or empty-body cancellation is synchronous immediate teardown, keeping programmatic callers' budgets bounded. `stop_policy=finalize_then_cancel` answers 202 with the intent open and runs one bounded finalization episode (`supervisor/owner_stop.py`): live descendants settle first, then the root's `finalize_now` control allows at most one tool-less model turn over a bounded child-result projection, terminalizing under the honest owner reason, not a false deadline. Grace is bounded by `OWNER_STOP_OUTER_CAP_SEC`; explicit deadline and absolute ceiling stay independent. Transitions are monotonic: immediate hardens a pending graceful intent and graceful never softens an accepted immediate one; Panic bypasses both.
|
||||
Stop policy is an axis on the same durable intent, independent of cascade scope. An omitted or empty-body cancellation is immediate: 200, 404 when nothing is left, or 202 pending (CLIENT_CONTRACT §8). `stop_policy=finalize_then_cancel` answers 202 with the intent open and runs one bounded finalization episode (`supervisor/owner_stop.py`): live descendants settle first, then the root's `finalize_now` control allows at most one tool-less model turn over a bounded child-result projection, terminalizing under the honest owner reason, not a false deadline. Grace is bounded by `OWNER_STOP_OUTER_CAP_SEC`; explicit deadline and absolute ceiling stay independent. Transitions are monotonic: immediate hardens a pending graceful intent and graceful never softens an accepted immediate one; Panic bypasses both.
|
||||
|
||||
The owner "hurry" is a typed task-local `kind=hurry` owner-mailbox control (`ouroboros/owner_hurry.py`, `gateway/task_hurry.py`) that skips the next otherwise-eligible acceptance panel with a typed reason, zeroes the remaining improvement passes and makes the force-plan projection advisory for that task. It is never a chat message, a settings mutation or a weakening of the P3 commit and review gates, and it is attempt-scoped: a shared `retry_reset` strips it on every same-id requeue.
|
||||
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ Numbered rules identify cross-cutting owners; the continuity map separates exact
|
|||
11. **UI resources carry a disposer.** Every subscription, listener, observer, timer, stream and live page instance has explicit teardown; navigation leaves no hidden instance mutating visible or durable state.
|
||||
12. **Frozen contracts extend explicitly.** `ouroboros/contracts/` is a versioned, backward-compatible ABI of typed shapes with their parsing, normalization and policy helpers (§11); new capability extends the frozen shape or ships a versioned successor, and existing consumers keep working.
|
||||
13. **Provider wire adaptation stays exact-route and success-confirmed.** Canonical history stays provider-neutral; a typed physical projection may change values, fields or a registered dialect on one provider/endpoint/API/model only. Failed candidates teach nothing durable, task-local degradation never becomes dispatch authority, and the physical-attempt ledger stays distinct from terminal request-wire history.
|
||||
14. **Cancellation is intent-then-custody.** Intent never rides the canonical task status, and no teardown begins before the durable, watchdog-replayable intent row exists; the settle owner takes an exclusive claim before any custody mutation; a settled result does not prove a dead worker, so live ownership gates a settled-target cancel; natural completion wins a late cancel and the stored terminal row survives byte-identical, because the kill is about the process, never the result; the deliverable is registered as owed before the intent settles; a cascade acts on lineage roots. Flow: §5. Owners: `supervisor/task_lifecycle.py`, `ouroboros/cancel_intents.py`, `supervisor/cancel_publication.py`, `supervisor/terminal_delivery.py`.
|
||||
14. **Cancellation is intent-then-custody.** Intent never rides the canonical task status, and no custody runs before the durable, watchdog-replayable intent row exists; the settle owner takes an exclusive claim before any custody mutation; a settled result does not prove a dead worker, so live ownership gates a settled-target cancel; natural completion wins a late cancel and the stored terminal row survives byte-identical, because the kill is about the process, never the result; the deliverable is registered as owed before the intent settles; a cascade acts on lineage roots. Flow: §5. Owners: `supervisor/task_lifecycle.py`, `ouroboros/cancel_intents.py`, `supervisor/cancel_publication.py`, `supervisor/terminal_delivery.py`.
|
||||
15. **Cancellation registry corruption stays visible.** In `state/cancel_intents.json` and `state/terminal_deliveries.json` an absent file reads as empty; a malformed file or row is disclosed and its bytes kept, and no mutation rebuilds an existing malformed store from a `{}` collapse.
|
||||
16. **Verification receipts reconcile by one typed identity key.** Sameness is equality of the single (kind, value) key, never a match across kinds, so reconciliation fails safe toward fewer reconciliations; disclosed parts are never the comparison (`_outcome_receipts.py`).
|
||||
17. **Review spend has one ceiling.** Every paid review gate shares `OUROBOROS_REVIEW_MAX_CYCLES` (SSOT `review_cycles.py`; per-gate meanings: §6 Review stack). The ceiling never removes the last author reaction; a blocking correction or stop grants no approval.
|
||||
|
|
|
|||
|
|
@ -42,4 +42,4 @@ Companion processes are host-supervised: reviewed manifest-declared descriptors
|
|||
|
||||
### Chat-id delivery
|
||||
|
||||
Each message has exactly ONE destination. Chat IDs are values; absence is `None` (§11.1 `chat_id_policy`). `HIDDEN_CHAT_ID` (0) is the real hidden destination for Skill Review and unbound headless work, with no browser reader. `message_bus.notification_chat_route` decides where a notice goes; a `chat_id=0` history query coerces to Main, which excludes chat-0 rows. Only durable lineage binding brings them to a Project; negative A2A ids never enter human streams. An exhausted terminal outbox notice skips chat 0 because it has no reader, while the durable incident and saved message remain; only positive same-id, same-destination evidence suppresses that notice (`terminal_delivery`), never a bare delivered id or a missing ACK. Memory and consolidation provenance follows the same value/`None` policy: a per-window registry snapshot names Main and the current Projects with stable chat ids, missing or ambiguous values stay explicit and never default to Main, and a presence room is named only by transport facts that re-derive its chat id. These labels are read-only source attribution and alter no routing, visibility or delivery state.
|
||||
Each message has exactly ONE destination. Chat IDs are values; absence is `None` (§11.1 `chat_id_policy`). `HIDDEN_CHAT_ID` (0) is the real hidden destination for Skill Review and unbound headless work, with no browser reader. `message_bus.notification_chat_route` decides where a notice goes; a `chat_id=0` history query coerces to Main, excluding chat-0 rows. A lineage binding or the task's admitted Project chat brings them to a Project; negative A2A ids never enter human streams. An exhausted terminal outbox notice skips chat 0 (no reader) while the durable incident and saved message remain; only positive same-id, same-destination evidence suppresses that notice (`terminal_delivery`), never a bare delivered id or a missing ACK. Memory and consolidation provenance follows the same value/`None` policy: a per-window registry snapshot names Main and the current Projects with stable chat ids, missing or ambiguous values stay explicit and never default to Main, and a presence room is named only by transport facts that re-derive its chat id. These labels are read-only source attribution and alter no routing, visibility or delivery state.
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
AST-derived inventory of compatibility facades, regenerated by `python scripts/regenerate_inventories.py`. Do not edit. A facade row is any runtime module whose top-level `from <population module> import ...` statements carry the `noqa: F401` re-export marker — the codebase's declared "this binding exists for its binding, not for this module's own use" convention (reference FACADE_CONSUMERS method). Leaf domains come from `ouroboros/domains.toml`; a leaf outside the facade's domain is marked ✗ (that edge also appears in the manifest's pinned direction matrix). `tests/test_generated_inventories.py` pins byte-identity, so any re-export surface change must regenerate this file.
|
||||
|
||||
- facade modules: **67**; marked re-export bindings: **2368**; cross-domain facade→leaf pairs: **131**
|
||||
- facade modules: **67**; marked re-export bindings: **2370**; cross-domain facade→leaf pairs: **131**
|
||||
|
||||
| facade | domain | bindings | leaves |
|
||||
|---|---|---:|---|
|
||||
|
|
@ -69,7 +69,7 @@ AST-derived inventory of compatibility facades, regenerated by `python scripts/r
|
|||
| `supervisor/queue.py` | D08 | 115 | `ouroboros/config.py` (6 ✗D12)<br>`ouroboros/contracts/task_contract.py` (3 ✗D19)<br>`ouroboros/schedule_contract.py` (2)<br>`ouroboros/skill_loader.py` (1 ✗D14)<br>`ouroboros/utils.py` (3 ✗D18)<br>`supervisor/evolution_lifecycle.py` (12 ✗D15)<br>`supervisor/message_bus.py` (3)<br>`supervisor/queue_schedules.py` (19)<br>`supervisor/queue_snapshot.py` (5)<br>`supervisor/queue_timeouts.py` (9)<br>`supervisor/queue_transitions.py` (5)<br>`supervisor/schedule_lifecycle.py` (3)<br>`supervisor/schedule_time.py` (7)<br>`supervisor/state.py` (6)<br>`supervisor/task_admission.py` (9)<br>`supervisor/task_lifecycle.py` (17 ✗D09)<br>`supervisor/task_reaper.py` (5 ✗D09) |
|
||||
| `supervisor/queue_transitions.py` | D08 | 2 | `supervisor/budget_resume.py` (2) |
|
||||
| `supervisor/state.py` | D08 | 4 | `ouroboros/utils.py` (4 ✗D18) |
|
||||
| `supervisor/task_lifecycle.py` | D09 | 31 | `supervisor/cancel_publication.py` (20)<br>`supervisor/queue_transitions.py` (11 ✗D08) |
|
||||
| `supervisor/task_lifecycle.py` | D09 | 33 | `supervisor/cancel_publication.py` (21)<br>`supervisor/queue_transitions.py` (12 ✗D08) |
|
||||
| `supervisor/task_model_wait.py` | D08 | 1 | `ouroboros/model_wait.py` (1 ✗D09) |
|
||||
| `supervisor/update_merge.py` | D10 | 27 | `supervisor/update_candidate.py` (24)<br>`supervisor/update_merge_plan.py` (3) |
|
||||
| `supervisor/workers.py` | D08 | 43 | `supervisor/log_addressing.py` (1)<br>`supervisor/message_bus.py` (1)<br>`supervisor/worker_assignment.py` (3)<br>`supervisor/worker_chat_lane.py` (6)<br>`supervisor/worker_health.py` (4)<br>`supervisor/worker_pool_lifecycle.py` (13)<br>`supervisor/worker_process.py` (6)<br>`supervisor/worker_promotion.py` (9) |
|
||||
|
|
|
|||
|
|
@ -406,12 +406,12 @@ def iter_room_rows(root: Any, room_id: Any, *, from_addr: Any = None, to_addr: A
|
|||
own address: a redelivered message is not folded into its first copy.
|
||||
"""
|
||||
from ouroboros.project_dialogue import room_membership, source_refs_for_project
|
||||
from ouroboros.projects_registry import all_task_bindings, list_reserved_projects
|
||||
from ouroboros.projects_registry import list_reserved_projects, RoomResolver
|
||||
|
||||
root, chat = pathlib.Path(root), int(room_id)
|
||||
projects = {int(project["chat_id"]) for project in list_reserved_projects(root)}
|
||||
refs = source_refs_for_project(root, chat) if chat in projects else []
|
||||
member = room_membership(chat, projects, refs, all_task_bindings(root))
|
||||
member = room_membership(chat, projects, refs, RoomResolver(root))
|
||||
tasks = None if task_ids is None else {str(task) for task in task_ids}
|
||||
lower, upper = _period_bounds(period)
|
||||
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ from ouroboros.project_dialogue import (
|
|||
bound_room_chat, entry_matches_source_ref, project_origin_rows,
|
||||
room_membership, source_refs_for_project, latest_chat_annotations,
|
||||
)
|
||||
from ouroboros.projects_registry import all_task_bindings, list_reserved_projects
|
||||
from ouroboros.projects_registry import list_reserved_projects, RoomResolver
|
||||
from ouroboros.utils import jsonl_archive_segments, jsonl_generation_signature, utc_now_iso
|
||||
|
||||
# Structural producer fields, not a semantic selection of what counts as an
|
||||
|
|
@ -107,9 +107,8 @@ def read_room_source(drive_root: Any, chat_id: int, *, task_id: str = "",
|
|||
projects = {int(p["chat_id"]): p for p in list_reserved_projects(root)}
|
||||
if chat_id < 0:
|
||||
return None
|
||||
bindings = all_task_bindings(root)
|
||||
refs = source_refs_for_project(root, chat_id) if chat_id in projects else []
|
||||
matches = room_membership(chat_id, set(projects), refs, bindings)
|
||||
matches = room_membership(chat_id, set(projects), refs, RoomResolver(root))
|
||||
rows, coverage = Memory(root).read_chat_generations(predicate=lambda row: matches(_chat_id(row), row))
|
||||
if chat_id not in projects and chat_id not in {0, 1} and not rows:
|
||||
return None
|
||||
|
|
@ -263,7 +262,7 @@ def own_room_chat(ctx: Any, drive_root: Any) -> int | None:
|
|||
stored = task_room_record(drive_root, task_id)
|
||||
meta = getattr(ctx, "task_metadata", {}) or {}
|
||||
task = {**stored, **meta, "task_id": task_id}
|
||||
bound = bound_room_chat(all_task_bindings(drive_root), task)
|
||||
bound = bound_room_chat(RoomResolver(drive_root), task)
|
||||
if bound:
|
||||
return bound
|
||||
for value in (getattr(ctx, "current_chat_id", None), task.get("chat_id")):
|
||||
|
|
|
|||
|
|
@ -136,7 +136,8 @@ class ChatOutbound(TypedDict):
|
|||
comment: NotRequired[str]
|
||||
host_facts: NotRequired[str] # the mirrored card's host sentence (QuizOutbound.host_facts)
|
||||
task_incident: NotRequired[str]
|
||||
# A cancellation fault names the PHYSICAL task it could not settle when it differs from the logical task id.
|
||||
# Historical rows only: the retired cancellation-fault notice named the PHYSICAL task it could not
|
||||
# settle when that differed from the logical id. Declared so stored rows keep their shape.
|
||||
cancel_physical_task_id: NotRequired[str]
|
||||
toast_once: NotRequired[str]
|
||||
# The one-shot toast's valence (warn/ok/error; the reaper's rail ``warning`` is normalizeTone's ``warn``
|
||||
|
|
@ -1312,10 +1313,13 @@ class TaskCancelResponse(TypedDict, total=False):
|
|||
task_id: str
|
||||
# Cascade echoes subtree scope; immediate success follows teardown.
|
||||
cascade: bool
|
||||
# Soft 202 keeps custody pending and echoes EFFECTIVE policy, never softening
|
||||
# an immediate intent. Legacy immediate replies omit these additive fields.
|
||||
# Every 202 keeps the recorded intent pending and echoes the EFFECTIVE policy,
|
||||
# never softening an immediate intent: a soft stop, or an immediate teardown not
|
||||
# yet confirmed (request_id, reason_code "teardown_unconfirmed"). 200 omits them.
|
||||
cancel_state: str
|
||||
stop_policy: str
|
||||
request_id: str
|
||||
reason_code: str
|
||||
error: str
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -268,7 +268,7 @@ HTTP_SHAPES: dict[str, Shape] = {
|
|||
note="v2 cursor: send back the cursor of the last event processed"),
|
||||
"POST /api/tasks/{task_id}/cancel": Shape(
|
||||
f"200,202:TaskCancelResponse {_TASK_ERRORS}", "TaskCancelRequest",
|
||||
note="stop_action_id identifies one Stop; an exact retry joins a pending Stop; after teardown it may return 404; 202 = a graceful stop is pending"),
|
||||
note="stop_action_id identifies one Stop; an exact retry joins a pending Stop; after teardown it may return 404; 202 = the Stop is recorded and still pending (graceful, or an immediate teardown not yet confirmed)"),
|
||||
"POST /api/tasks/{task_id}/hurry": Shape(f"200:TaskHurryResponse {_TASK_ERRORS}", "TaskHurryRequest"),
|
||||
"POST /api/tasks/{task_id}/pause": Shape(f"200,202:TaskPauseResponse {_TASK_ERRORS}", "TaskPauseRequest"),
|
||||
"POST /api/tasks/{task_id}/continue": Shape(f"200:TaskContinueResponse {_TASK_ERRORS}", "TaskContinueRequest",
|
||||
|
|
|
|||
|
|
@ -138,11 +138,11 @@ def _stored_chat_id(value: Any, default: int = 1) -> int:
|
|||
def _project_history_context(
|
||||
data_dir: pathlib.Path,
|
||||
thread_id: int,
|
||||
) -> tuple[set[int], list[dict], Dict[str, Any], Dict[str, int], bool]:
|
||||
) -> tuple[set[int], list[dict], Dict[str, Any], Dict[str, int], Any, bool]:
|
||||
"""Read-only Project lenses and registry completeness, in the assembly worker.
|
||||
|
||||
Preload bindings once per request to avoid per-row task/parent/root reads
|
||||
of state/project_task_bindings.json; no extra event-loop thread hops.
|
||||
Preload bindings (the cursor fingerprint's explicit roots) and the room rule
|
||||
once per request; no per-row registry reads, no extra event-loop thread hops.
|
||||
"""
|
||||
from ouroboros.projects_registry import reserved_project_chat_ids
|
||||
|
||||
|
|
@ -164,13 +164,10 @@ def _project_history_context(
|
|||
annotations = latest_chat_annotations(data_dir)
|
||||
except Exception:
|
||||
annotations = {}
|
||||
try:
|
||||
from ouroboros.projects_registry import all_task_bindings
|
||||
from ouroboros.projects_registry import RoomResolver, all_task_bindings
|
||||
|
||||
bindings_by_task = all_task_bindings(data_dir)
|
||||
except Exception:
|
||||
bindings_by_task = {}
|
||||
return project_chat_ids, source_refs, annotations, bindings_by_task, classified
|
||||
# Neither raises: an unreadable bindings store reads as no bindings in both.
|
||||
return project_chat_ids, source_refs, annotations, all_task_bindings(data_dir), RoomResolver(data_dir), classified
|
||||
|
||||
|
||||
def _user_annotation(
|
||||
|
|
@ -441,6 +438,7 @@ def _annotate_terminal_task_truth(
|
|||
result_task_ids.add(task_id)
|
||||
if progress or summary:
|
||||
carriers[task_id] = message
|
||||
# A parent anchored by its child's row carries that row's ts; a census anchor has none.
|
||||
for task_id, ts in (evidence_anchors or {}).items():
|
||||
represented.setdefault(task_id, ts)
|
||||
tool_evidence_by_task = replay_evidence_for_tasks(data_dir, represented)
|
||||
|
|
@ -454,10 +452,11 @@ def _annotate_terminal_task_truth(
|
|||
if isinstance(row, dict) and type(row.get("calls")) is int and row["calls"] > 0))
|
||||
known_calls = max((n for n in counts if type(n) is int and n > 0), default=0)
|
||||
coverage = evidence.get("coverage") or {}
|
||||
# Global byte/archive bounds do not establish missing calls for
|
||||
# this task. Positive task counts and actual read gaps do.
|
||||
# Global byte/archive bounds do not establish missing calls for this task; positive counts and read gaps
|
||||
# do. A parent anchored by its child's row is always carried (zero tools included): never a bare shell (A5).
|
||||
incomplete = (coverage.get("gaps") or max(known_calls, coverage.get("matched", 0)) > coverage.get("shown", 0))
|
||||
if task_id not in carriers and (evidence.get("observations") or evidence.get("legacy", {}).get("calls") or incomplete):
|
||||
if task_id not in carriers and (evidence.get("observations") or evidence.get("legacy", {}).get("calls")
|
||||
or incomplete or (evidence_anchors or {}).get(task_id)):
|
||||
carrier = {"task_id": task_id, "ts": represented[task_id], "role": "system",
|
||||
"system_type": "task_evidence", "text": "", "is_progress": False, "narration": False}
|
||||
combined.append(carrier)
|
||||
|
|
@ -668,13 +667,13 @@ def _make_thread_filter(
|
|||
thread_id: int,
|
||||
project_chat_ids: set,
|
||||
project_source_refs: list,
|
||||
bindings_by_task: Dict[str, int],
|
||||
rooms: Any,
|
||||
):
|
||||
"""One per-request room predicate shared by durable stream and evidence readers."""
|
||||
from ouroboros.project_dialogue import bound_room_chat, matching_project_origin, room_membership
|
||||
from ouroboros.project_dialogue import matching_project_origin, room_membership
|
||||
|
||||
belongs = room_membership(thread_id if thread_id in project_chat_ids else 1,
|
||||
project_chat_ids, project_source_refs, bindings_by_task)
|
||||
project_chat_ids, project_source_refs, rooms)
|
||||
|
||||
def _row_matches_thread(entry_chat: int, entry: Optional[dict] = None) -> bool:
|
||||
if _question_project_chat(entry):
|
||||
|
|
@ -691,7 +690,7 @@ def _make_thread_filter(
|
|||
quiz = entry.get("quiz")
|
||||
if not isinstance(quiz, dict):
|
||||
return 0
|
||||
chat = bound_room_chat(bindings_by_task, {"task_id": entry.get("task_id")}) or _stored_chat_id(entry.get("chat_id"), 1)
|
||||
chat = rooms(entry.get("task_id")) or _stored_chat_id(entry.get("chat_id"), 1)
|
||||
return chat if chat in project_chat_ids else 0
|
||||
|
||||
_row_matches_thread.question_project_chat = _question_project_chat
|
||||
|
|
@ -1433,11 +1432,11 @@ def _assemble_history_response(
|
|||
Room lenses, source reads, projection, terminal truth and JSON encoding all
|
||||
stay off the event loop. Both selectors use the same row transformers.
|
||||
"""
|
||||
project_chat_ids, project_source_refs, chat_annotations, bindings_by_task, classified = (
|
||||
project_chat_ids, project_source_refs, chat_annotations, bindings_by_task, rooms, classified = (
|
||||
_project_history_context(data_dir, thread_id)
|
||||
)
|
||||
row_matches_thread = _make_thread_filter(
|
||||
thread_id, project_chat_ids, project_source_refs, bindings_by_task
|
||||
thread_id, project_chat_ids, project_source_refs, rooms
|
||||
)
|
||||
chat_path = data_dir / "logs" / "chat.jsonl"
|
||||
progress_path = data_dir / "logs" / "progress.jsonl"
|
||||
|
|
@ -1448,7 +1447,7 @@ def _assemble_history_response(
|
|||
{"chat": _chat_quota_predicate(row_matches_thread),
|
||||
"progress": progress_quota_predicate(row_matches_thread, _stored_chat_id)},
|
||||
{"human": _MAX_N_HUMAN, "progress": _MAX_N_PROGRESS},
|
||||
room_segment_lens(thread_id, project_chat_ids, project_source_refs, bindings_by_task))
|
||||
room_segment_lens(thread_id, project_chat_ids, project_source_refs, rooms))
|
||||
selections, before, recent = page["selections"], page["before"], page["recent"]
|
||||
n_human, n_progress = page["quotas"]["human"], page["quotas"]["progress"]
|
||||
historical_terminals: Dict[str, dict] = {}
|
||||
|
|
@ -1497,15 +1496,15 @@ def _assemble_history_response(
|
|||
data_dir, bindings_by_task, row_matches_thread.evidence_matches), "")
|
||||
represented = {str(row.get("task_id") or "") for row in messages} | evidence_anchors.keys()
|
||||
for row in messages:
|
||||
parent = str(row.get("parent_task_id") or "")
|
||||
# A notice names its target, never a speaker (no parent); a legacy child row's parent is in the facts index.
|
||||
parent = "" if row.get("system_type") in ("project_question_pointer", *ORIGIN_ADDRESSED_NOTICE_TYPES) else str(
|
||||
row.get("parent_task_id") or rooms.lineage(row.get("task_id")).get("parent_task_id") or "")
|
||||
if not parent or parent in represented or (floor and str(row.get("ts") or "") < floor
|
||||
and row.get("task_id") not in anchored_children):
|
||||
continue
|
||||
# A child's room does not address its parent. Unknown parent routes
|
||||
# retain their topology without disclosing unscoped tool evidence.
|
||||
parent_chat = bindings_by_task.get(parent)
|
||||
if parent_chat is None:
|
||||
parent_chat = _load_terminal_result(data_dir, parent, result_cache).get("chat_id")
|
||||
# A child's room does not address its parent: the parent's own room decides (the room rule, else its
|
||||
# admitted chat); an in-room parent is anchored at its child's row, so its carrier carries that row's ts.
|
||||
parent_chat = rooms(parent) or rooms.lineage(parent).get("chat_id")
|
||||
if parent_chat is not None and row_matches_thread.evidence_matches(parent_chat, {"task_id": parent}):
|
||||
evidence_anchors.setdefault(parent, str(row.get("ts") or ""))
|
||||
_annotate_terminal_task_truth(
|
||||
|
|
@ -1514,8 +1513,8 @@ def _assemble_history_response(
|
|||
historical_terminals=historical_terminals,
|
||||
deferred_lineage=deferred_lineage,
|
||||
evidence_anchors=evidence_anchors,
|
||||
evidence_scope=lambda task_id: task_id not in bindings_by_task or row_matches_thread.evidence_matches(
|
||||
bindings_by_task[task_id], {"task_id": task_id}),
|
||||
evidence_scope=lambda task_id: row_matches_thread.evidence_matches(
|
||||
rooms(task_id) or thread_id, {"task_id": task_id}),
|
||||
)
|
||||
for source in ("chat", "progress"):
|
||||
before[source] = max([before[source], *(entry["_history_end"] for entry in selections[source][0] or ()
|
||||
|
|
|
|||
|
|
@ -65,11 +65,12 @@ class HistoryCursorError(ValueError):
|
|||
|
||||
|
||||
def room_view_fingerprint(thread_id, project_ids, source_refs, bindings):
|
||||
"""Fingerprint the existing room lens, without introducing membership state."""
|
||||
"""Fingerprint the existing room lens, without introducing membership state.
|
||||
|
||||
Every room hashes every binding: a task's resolved room outranks the room its
|
||||
rows were written to, so a conversion anywhere can move rows out of this room."""
|
||||
project = thread_id in project_ids
|
||||
relevant = {key: value for key, value in bindings.items()
|
||||
if not project or value == thread_id}
|
||||
value = [thread_id, project, [] if project else sorted(project_ids), relevant,
|
||||
value = [thread_id, project, [] if project else sorted(project_ids), dict(bindings),
|
||||
sorted(source_refs, key=lambda row: json.dumps(row, sort_keys=True))]
|
||||
return hashlib.sha256(json.dumps(value, sort_keys=True, separators=(",", ":")).encode()).hexdigest()
|
||||
|
||||
|
|
|
|||
|
|
@ -80,23 +80,25 @@ def segment_summary(path: Any, stat: os.stat_result) -> Optional[SegmentSummary]
|
|||
|
||||
|
||||
def room_segment_lens(thread_id: int, project_chat_ids: set, source_refs: list,
|
||||
bindings_by_task: Dict[str, int]) -> Optional[Callable[[SegmentSummary], bool]]:
|
||||
rooms: Any) -> Optional[Callable[[SegmentSummary], bool]]:
|
||||
"""``may_hold(summary)`` for a Project room; None (read everything) for any other room.
|
||||
|
||||
Mirrors ``room_membership``'s three Project admissions: the room's own chat,
|
||||
a task bound to it (by task, parent or root id), or the owner row a binding
|
||||
names. Main and the rooms that share its lens hold rows in nearly every archive.
|
||||
a task the room rule (``rooms``, the request's ``projects_registry.RoomResolver``)
|
||||
puts in it by any id the archive names, or the owner row a binding names. Main
|
||||
and the rooms that share its lens hold rows in nearly every archive.
|
||||
"""
|
||||
if thread_id not in project_chat_ids:
|
||||
return None
|
||||
from ouroboros.project_dialogue import _source_ref_identity # lazy: gateway -> dialogue
|
||||
|
||||
bound = frozenset(task for task, chat in bindings_by_task.items() if chat == thread_id)
|
||||
origins = frozenset((key[0], key[3]) for ref in source_refs if isinstance(ref, dict)
|
||||
if (key := _source_ref_identity(ref)) is not None)
|
||||
|
||||
def may_hold(summary: SegmentSummary) -> bool:
|
||||
return (thread_id in summary.chat_ids or not bound.isdisjoint(summary.task_ids)
|
||||
or not origins.isdisjoint(summary.owner_rows))
|
||||
if thread_id in summary.chat_ids or not origins.isdisjoint(summary.owner_rows):
|
||||
return True
|
||||
rooms.warm(summary.task_ids) # one read for the archive's unseen ids, then dict answers
|
||||
return any(rooms(task) == thread_id for task in summary.task_ids)
|
||||
|
||||
return may_hold
|
||||
|
|
|
|||
|
|
@ -994,11 +994,11 @@ def _record_cascade_incident(task_id: str, kind: str, detail: str = "") -> None:
|
|||
|
||||
|
||||
def _run_cascade_cancel(task_id: str) -> bool:
|
||||
"""Subtree cancel for the HTTP cascade path, AWAITED by its caller.
|
||||
"""Subtree cancel for the Host Service ``/chat/cancel`` path, AWAITED by its caller.
|
||||
|
||||
Returns True when the subtree is settled — cancelled, or already terminal
|
||||
(the benign completion-wins race) — and False when the teardown failed or
|
||||
refused while the tree is STILL live, which the endpoint reports rather than
|
||||
refused while the tree is STILL live, which the caller reports rather than
|
||||
answering ok:true for a cancellation that did not happen. Failures stay
|
||||
durable and owner-visible as incidents either way.
|
||||
"""
|
||||
|
|
@ -1023,6 +1023,104 @@ def _run_cascade_cancel(task_id: str) -> bool:
|
|||
return False
|
||||
|
||||
|
||||
def _record_stop(task_id: str, *, cascade: bool, stop_action_id: str) -> Optional[Dict[str, Any]]:
|
||||
"""Record a hard Stop's durable intent before anything is torn down.
|
||||
|
||||
No liveness probe precedes it: a live, queued, or persisted but never
|
||||
admitted target gets its intent first, so an admission racing the click is
|
||||
never torn down without its watchdog-replayable fence. A settled target
|
||||
mints nothing (completion wins) unless live ownership remains — stored
|
||||
completion can precede the worker's exit, and a settled root can still
|
||||
coordinate live descendants — so ownership is asked only once the request
|
||||
found the target settled, and the intent is then requested with
|
||||
``allow_settled_target``. A cascade needs a recorded tree: over an id with
|
||||
no durable row it is minted only while something of it is live, because
|
||||
the cascade postcondition would settle that phantom tree as cancelled.
|
||||
Returns the intent, or None when there is nothing to stop.
|
||||
"""
|
||||
from supervisor.queue import DRIVE_ROOT, task_has_live_ownership, task_subtree_is_live
|
||||
|
||||
from ouroboros.cancel_intents import SCOPE_CASCADE, STOP_POLICY_IMMEDIATE, request_cancel
|
||||
|
||||
def still_owned() -> bool:
|
||||
return task_has_live_ownership(task_id) or (cascade and task_subtree_is_live(task_id))
|
||||
|
||||
if cascade:
|
||||
from supervisor.cancel_publication import live_presence_row
|
||||
|
||||
row = load_task_result(DRIVE_ROOT, task_id, strict=True)
|
||||
# A live Presence turn is its runner's (as on the single Stop's miss lane).
|
||||
if live_presence_row(row) or (not row and not still_owned()):
|
||||
return None
|
||||
source = "http_cascade" if cascade else "http_single"
|
||||
record = functools.partial(
|
||||
request_cancel, DRIVE_ROOT, task_id, source=source, stop_action_id=stop_action_id,
|
||||
observation=observe_cancellation_target(
|
||||
DRIVE_ROOT, task_id, request_origin={"kind": "http_client", "source": source}),
|
||||
# §13.1: an omitted/empty-body or explicit-immediate request IS the
|
||||
# immediate policy — it monotonically HARDENS a pending graceful
|
||||
# intent (Stop-now during the wait) and mints a legacy row otherwise.
|
||||
requested_stop_policy=STOP_POLICY_IMMEDIATE,
|
||||
**({"scope": SCOPE_CASCADE} if cascade else {}),
|
||||
)
|
||||
intent = record(allow_settled_target=False)
|
||||
if intent.get("already_settled") and still_owned():
|
||||
intent = record(allow_settled_target=True)
|
||||
return None if intent.get("already_settled") else intent
|
||||
|
||||
|
||||
def _drive_stop(task_id: str, intent: Dict[str, Any], *, cascade: bool) -> JSONResponse:
|
||||
"""Drive a recorded hard Stop through its canonical owner, then answer.
|
||||
|
||||
``drive_cancel_intent_scope`` re-reads the durable scope right before the
|
||||
teardown, so an intent already widened to a cascade runs as one whatever
|
||||
this request's body said; kills and joins run off the event loop and
|
||||
outside the queue lock, and a large tree makes the caller wait for the
|
||||
kills it asked for. Cancelled answers 200 with the legacy envelope;
|
||||
nothing to cancel — completion won, or an unknown id settled ``not_found``
|
||||
without a task row — is the legacy 404. Any other outcome is judged by the
|
||||
durable intent alone: still open (busy store, unknown ownership, a refused
|
||||
or failed teardown, an unmet cascade postcondition) answers 202
|
||||
``cancel_state: "pending"``, which the cancel-intent watchdog settles and
|
||||
task detail shows meanwhile; closed by a concurrent settle, 404. A cascade
|
||||
that raised or did not settle is also an owner-visible incident.
|
||||
"""
|
||||
from supervisor.queue import (
|
||||
CANCEL_ALREADY_SETTLED, CANCEL_CANCELLED, CANCEL_NOT_FOUND, DRIVE_ROOT,
|
||||
drive_cancel_intent_scope,
|
||||
)
|
||||
|
||||
from ouroboros.cancel_intents import SCOPE_CASCADE, active_intent, stop_policy
|
||||
|
||||
target = str(intent.get("task_id") or task_id)
|
||||
cascade = cascade or str(intent.get("scope") or "") == SCOPE_CASCADE
|
||||
echo = {"cascade": True} if cascade else {}
|
||||
try:
|
||||
outcome = drive_cancel_intent_scope(target)
|
||||
except Exception as exc:
|
||||
log.warning("Stop teardown for %s raised; its intent stays open", target, exc_info=True)
|
||||
if cascade:
|
||||
_record_cascade_incident(target, "task_cancel_cascade_error", repr(exc))
|
||||
outcome = ""
|
||||
if outcome == CANCEL_CANCELLED:
|
||||
return JSONResponse({"ok": True, "task_id": task_id, **echo})
|
||||
if outcome not in (CANCEL_ALREADY_SETTLED, CANCEL_NOT_FOUND):
|
||||
try:
|
||||
pending = active_intent(DRIVE_ROOT, target, strict=True)
|
||||
except Exception:
|
||||
pending = intent # recorded; an unreadable projection proves no settle
|
||||
if pending:
|
||||
if cascade and outcome:
|
||||
_record_cascade_incident(target, "task_cancel_cascade_noop")
|
||||
return JSONResponse({
|
||||
"ok": True, "task_id": task_id, "cancel_state": "pending",
|
||||
"stop_policy": stop_policy(pending),
|
||||
"request_id": str(pending.get("request_id") or ""),
|
||||
"reason_code": "teardown_unconfirmed", **echo,
|
||||
}, status_code=202)
|
||||
return json_error("task not found or not active", 404, task_id=task_id)
|
||||
|
||||
|
||||
# Sentinel telling "the body did not parse" apart from a legitimate JSON null.
|
||||
_NO_BODY = object()
|
||||
|
||||
|
|
@ -1146,187 +1244,45 @@ async def api_task_cancel(request: Request) -> JSONResponse:
|
|||
if migrating := await run_sync_to_completion(migrating_error, request_drive_root(request)):
|
||||
return migrating # P1: Cancel writes the result's custody; 503 while the boot import owns the store
|
||||
if stop_policy_value == "finalize_then_cancel":
|
||||
# Graceful ingress: immediate typed pending acknowledgement; the
|
||||
# synchronous teardown contract below stays hard/legacy-only.
|
||||
# Graceful ingress: immediate typed pending acknowledgement; the hard
|
||||
# Stop below records its intent first and answers once it was driven.
|
||||
return await _graceful_stop_acknowledgement(task_id, cascade=cascade, stop_action_id=action_id)
|
||||
from supervisor.followup_policy import StopActionConflict
|
||||
|
||||
intent_target = {"task_id": task_id, "scope": ""}
|
||||
from ouroboros.cancel_intents import CancelIntentProjectionCorrupt
|
||||
|
||||
def _record_http_intent(
|
||||
source: str, *, cascade_scope: bool = False, allow_settled: bool = False,
|
||||
) -> bool:
|
||||
"""Persist the watchdog fence before teardown; failure refuses ingress.
|
||||
|
||||
Cascade ingress records widen-only scope and allows settled roots with
|
||||
live descendants. Single ingress supplies allow_settled from live physical
|
||||
ownership, since stored completion can precede worker exit. Return an
|
||||
empty string on success or a typed write/corruption/action-conflict refusal.
|
||||
"""
|
||||
try:
|
||||
from supervisor.followup_policy import StopActionConflict
|
||||
from supervisor.queue import DRIVE_ROOT as _drive_root
|
||||
|
||||
from ouroboros.cancel_intents import (
|
||||
CancelIntentProjectionCorrupt,
|
||||
SCOPE_CASCADE,
|
||||
STOP_POLICY_IMMEDIATE,
|
||||
request_cancel,
|
||||
)
|
||||
except Exception:
|
||||
log.warning("HTTP cancel-intent machinery unavailable for %s", task_id,
|
||||
exc_info=True)
|
||||
return "write_failed"
|
||||
try:
|
||||
intent = request_cancel(
|
||||
_drive_root, task_id, source=source,
|
||||
stop_action_id=action_id,
|
||||
observation=observe_cancellation_target(_drive_root, task_id, request_origin={"kind": "http_client", "source": source}),
|
||||
**({"scope": SCOPE_CASCADE} if cascade_scope else {}),
|
||||
allow_settled_target=bool(cascade_scope or allow_settled),
|
||||
# §13.1: an omitted/empty-body or explicit-immediate request IS
|
||||
# the immediate policy — it monotonically HARDENS a pending
|
||||
# graceful intent (Stop-now during the wait) and mints a
|
||||
# byte-identical legacy row when no intent exists.
|
||||
requested_stop_policy=STOP_POLICY_IMMEDIATE,
|
||||
)
|
||||
intent_target["task_id"] = str(intent.get("task_id") or task_id)
|
||||
intent_target["scope"] = str(intent.get("scope") or "")
|
||||
return ""
|
||||
except StopActionConflict:
|
||||
return "stop_action_conflict"
|
||||
except CancelIntentProjectionCorrupt:
|
||||
log.error("HTTP cancel refused for %s: intent projection corrupt", task_id)
|
||||
return "projection_corrupt"
|
||||
except Exception:
|
||||
log.warning("HTTP cancel-intent write failed for %s; cancel refused",
|
||||
task_id, exc_info=True)
|
||||
return "write_failed"
|
||||
|
||||
def _intent_write_refused(kind: str) -> JSONResponse:
|
||||
if kind == "stop_action_conflict":
|
||||
return json_error("stop_action_id reused with a different action", 409,
|
||||
task_id=task_id, reason_code=kind)
|
||||
if kind == "projection_corrupt":
|
||||
# GR4-8: honest wording — "retry" cannot succeed while the file is
|
||||
# malformed. The corrupt state/cancel_intents.json was PRESERVED
|
||||
# (never overwritten) and a projection_corrupt_refused forensic row
|
||||
# was recorded in logs/supervisor.jsonl.
|
||||
return json_error(
|
||||
"the cancel-intent projection (state/cancel_intents.json) is corrupt; "
|
||||
"nothing was cancelled and retrying cannot succeed until the file is "
|
||||
"repaired — the malformed file was preserved (no overwrite) and a "
|
||||
"projection_corrupt_refused forensic row was recorded in "
|
||||
"logs/supervisor.jsonl",
|
||||
503, task_id=task_id, reason_code="cancel_intent_projection_corrupt",
|
||||
)
|
||||
try:
|
||||
intent = await asyncio.to_thread(functools.partial(
|
||||
_record_stop, task_id, cascade=cascade, stop_action_id=action_id,
|
||||
))
|
||||
except StopActionConflict:
|
||||
return json_error("stop_action_id reused with a different action", 409,
|
||||
task_id=task_id, reason_code="stop_action_conflict")
|
||||
except CancelIntentProjectionCorrupt:
|
||||
# GR4-8: honest wording — "retry" cannot succeed while the file is
|
||||
# malformed. The corrupt state/cancel_intents.json was PRESERVED
|
||||
# (never overwritten) and a projection_corrupt_refused forensic row
|
||||
# was recorded in logs/supervisor.jsonl.
|
||||
log.error("HTTP cancel refused for %s: intent projection corrupt", task_id)
|
||||
return json_error(
|
||||
"the cancel-intent projection (state/cancel_intents.json) is corrupt; "
|
||||
"nothing was cancelled and retrying cannot succeed until the file is "
|
||||
"repaired — the malformed file was preserved (no overwrite) and a "
|
||||
"projection_corrupt_refused forensic row was recorded in "
|
||||
"logs/supervisor.jsonl",
|
||||
503, task_id=task_id, reason_code="cancel_intent_projection_corrupt",
|
||||
)
|
||||
except Exception:
|
||||
# Nothing was recorded, so nothing was torn down: no teardown without
|
||||
# the watchdog-replayable fence.
|
||||
log.warning("HTTP cancel-intent write failed for %s; cancel refused", task_id, exc_info=True)
|
||||
return json_error(
|
||||
"durable cancel intent could not be recorded; nothing was cancelled — retry",
|
||||
503, task_id=task_id, reason_code="cancel_intent_write_failed",
|
||||
)
|
||||
|
||||
if not cascade:
|
||||
try:
|
||||
from supervisor.queue import (
|
||||
CANCEL_CANCELLED, CANCEL_FAILED, cancel_task_custody,
|
||||
task_has_live_ownership as _live_ownership,
|
||||
task_subtree_is_live as _live_check,
|
||||
)
|
||||
|
||||
# Intent only for a LIVE task: the plain path's legacy contract
|
||||
# answers 404 for an inactive id, and an intent minted for a dead id
|
||||
# would sit open until the watchdog settles it as not_found.
|
||||
# LIVE OWNERSHIP (GR6-1) widens the gate: a settled result whose
|
||||
# worker is still alive is not "inactive" — the intent is minted
|
||||
# with ``allow_settled`` so custody kills the spending worker.
|
||||
live_own = await asyncio.to_thread(_live_ownership, task_id)
|
||||
if live_own or await asyncio.to_thread(_live_check, task_id):
|
||||
refused = await asyncio.to_thread(functools.partial(
|
||||
_record_http_intent, "http_single", allow_settled=live_own,
|
||||
))
|
||||
if refused:
|
||||
return _intent_write_refused(refused)
|
||||
if intent_target["scope"] == "cascade":
|
||||
# Scope is widen-only durable authority. Stop-now over an
|
||||
# existing graceful cascade must execute that cascade now even
|
||||
# when the new HTTP body omitted ``cascade``; replaying the raw
|
||||
# single shape can kill only the root or fail on a retry leaf.
|
||||
settled = await asyncio.to_thread(
|
||||
_run_cascade_cancel, intent_target["task_id"],
|
||||
)
|
||||
if not settled:
|
||||
return json_error(
|
||||
"subtree cancellation did not settle; the tree is still live",
|
||||
503,
|
||||
task_id=task_id,
|
||||
)
|
||||
return JSONResponse({
|
||||
"ok": True,
|
||||
"task_id": task_id,
|
||||
"cascade": True,
|
||||
})
|
||||
# The TYPED outcome, not a boolean: a task whose worker refused to die
|
||||
# is neither cancelled nor absent, and answering 404 for it would tell
|
||||
# the caller the task is gone while it keeps running.
|
||||
outcome = await asyncio.to_thread(
|
||||
cancel_task_custody, intent_target["task_id"],
|
||||
)
|
||||
except Exception as exc:
|
||||
return json_exception(exc, 503)
|
||||
if outcome == CANCEL_FAILED:
|
||||
return json_error(
|
||||
"cancellation did not settle; the task is still live",
|
||||
503, task_id=task_id,
|
||||
)
|
||||
if outcome != CANCEL_CANCELLED:
|
||||
# LEGACY CONTRACT preserved: the plain path has always answered 404 for
|
||||
# an INACTIVE task, and one that already settled on its own is exactly
|
||||
# that — the typed outcome must not silently widen the envelope.
|
||||
return json_error("task not found or not active", 404, task_id=task_id)
|
||||
return JSONResponse({"ok": True, "task_id": task_id})
|
||||
# Cascade path: ONE synchronous transaction. The caller is answered only once
|
||||
# the subtree is actually torn down, which is what makes the whole
|
||||
# split-transaction family (durable pre-acknowledgement latch, partial-latch
|
||||
# taxonomy, ownership handed to a background teardown, rollbacks that could
|
||||
# withdraw a concurrent cascade's fences) unnecessary rather than merely
|
||||
# guarded. The cost is honest and bounded: a large tree makes the caller wait
|
||||
# for the worker kills and joins it asked for. Off the event loop; process
|
||||
# kills and joins deliberately happen outside the supervisor queue lock. Repeats are
|
||||
# idempotent (the per-task cancel finalizes-on-miss) and a fully-cancelled tree
|
||||
# is no longer live, so it answers 404 like any other inactive task.
|
||||
try:
|
||||
from supervisor.queue import (
|
||||
task_has_live_ownership as _cascade_live_ownership,
|
||||
task_subtree_is_live,
|
||||
)
|
||||
|
||||
# GR7-1b: the 404 pre-check consults the SAME live-ownership predicate
|
||||
# the single lane uses. `task_subtree_is_live` deliberately excludes a
|
||||
# settled-but-RUNNING root (so the cascade postcondition can converge
|
||||
# over a winding-down finalizer), which made this pre-check answer 404
|
||||
# for a settled root whose worker was still burning post-task
|
||||
# cognition — before any intent was minted. A settled-but-LIVE root
|
||||
# proceeds to mint + custody (the kill path preserves the stored
|
||||
# result); a genuinely settled-AND-dead tree keeps the 404 envelope.
|
||||
if not await asyncio.to_thread(
|
||||
_cascade_live_ownership, task_id,
|
||||
) and not await asyncio.to_thread(task_subtree_is_live, task_id):
|
||||
return json_error("task not found or not active", 404, task_id=task_id)
|
||||
refused = await asyncio.to_thread(
|
||||
functools.partial(_record_http_intent, "http_cascade", cascade_scope=True),
|
||||
)
|
||||
if refused:
|
||||
return _intent_write_refused(refused)
|
||||
settled = await asyncio.to_thread(_run_cascade_cancel, task_id)
|
||||
if not settled:
|
||||
# The teardown refused or failed while the subtree is STILL live: an
|
||||
# ok:true here would report a cancellation that did not happen.
|
||||
return json_error(
|
||||
"subtree cancellation did not settle; the tree is still live",
|
||||
503, task_id=task_id,
|
||||
)
|
||||
except Exception as exc:
|
||||
return json_exception(exc, 503)
|
||||
return JSONResponse({"ok": True, "task_id": task_id, "cascade": True})
|
||||
if intent is None:
|
||||
return json_error("task not found or not active", 404, task_id=task_id)
|
||||
return await asyncio.to_thread(functools.partial(_drive_stop, task_id, intent, cascade=cascade))
|
||||
|
||||
|
||||
async def api_task_resume(request: Request) -> JSONResponse:
|
||||
|
|
|
|||
|
|
@ -99,10 +99,12 @@ def row_meta(row: Mapping[str, Any]) -> Dict[str, Any]:
|
|||
|
||||
|
||||
class MembershipFacts(NamedTuple):
|
||||
"""The registry facts room membership reads, taken once per call."""
|
||||
"""The registry facts room membership reads, taken once per call; ``rooms`` is
|
||||
the room rule over the ``bindings`` snapshot the fingerprint names."""
|
||||
project_chat_ids: FrozenSet[int]
|
||||
bindings: Dict[str, int]
|
||||
projects_by_source_key: Dict[Tuple[Any, ...], FrozenSet[int]]
|
||||
rooms: Any
|
||||
|
||||
def fingerprint(self) -> str:
|
||||
payload = json.dumps([sorted(self.project_chat_ids), sorted(self.bindings.items()),
|
||||
|
|
@ -114,7 +116,7 @@ class MembershipFacts(NamedTuple):
|
|||
def membership_facts(root: Any) -> MembershipFacts:
|
||||
"""Reserved Project rooms, task bindings and the bound owner messages, as ``iter_room_rows`` reads them."""
|
||||
from ouroboros.project_dialogue import _source_ref_identity # D15->D17 is lazy-only
|
||||
from ouroboros.projects_registry import all_task_bindings, list_reserved_projects, project_task_bindings
|
||||
from ouroboros.projects_registry import all_task_bindings, list_reserved_projects, project_task_bindings, RoomResolver
|
||||
|
||||
projects = frozenset(int(project["chat_id"]) for project in list_reserved_projects(root))
|
||||
by_key: Dict[Tuple[Any, ...], set] = {}
|
||||
|
|
@ -128,7 +130,7 @@ def membership_facts(root: Any) -> MembershipFacts:
|
|||
if key is not None:
|
||||
by_key.setdefault(key, set()).add(chat)
|
||||
return MembershipFacts(projects, dict(all_task_bindings(root)),
|
||||
{key: frozenset(chats) for key, chats in by_key.items()})
|
||||
{key: frozenset(chats) for key, chats in by_key.items()}, RoomResolver(root))
|
||||
|
||||
|
||||
def rooms_of_row(meta: Mapping[str, Any], facts: MembershipFacts) -> FrozenSet[str]:
|
||||
|
|
@ -146,12 +148,12 @@ def rooms_of_row(meta: Mapping[str, Any], facts: MembershipFacts) -> FrozenSet[s
|
|||
if is_a2a_chat_id(chat):
|
||||
return frozenset()
|
||||
kind = meta.get("type")
|
||||
bound = 0 if kind in ORIGIN_ADDRESSED_NOTICE_TYPES else bound_room_chat(facts.bindings, meta)
|
||||
bound = 0 if kind in ORIGIN_ADDRESSED_NOTICE_TYPES else bound_room_chat(facts.rooms, meta)
|
||||
pinned = kind in MAIN_PINNED_ROW_TYPES
|
||||
projects = facts.project_chat_ids
|
||||
rooms = set()
|
||||
if not pinned:
|
||||
rooms.update(room for room in (bound, chat) if room in projects)
|
||||
if not pinned: # the resolved room is authoritative over the stored address (``room_membership``)
|
||||
rooms.update(room for room in ((bound,) if bound else (chat,)) if room in projects)
|
||||
for key in meta.get("source_keys") or ():
|
||||
rooms.update(facts.projects_by_source_key.get(tuple(key), ()))
|
||||
if MAIN_ROOM not in projects and chat != HIDDEN_CHAT_ID and chat not in projects and (pinned or not bound):
|
||||
|
|
|
|||
|
|
@ -273,21 +273,21 @@ def project_origin_rows(drive_root: Any, project_chat_id: int) -> List[Dict[str,
|
|||
return rows
|
||||
|
||||
|
||||
def bound_room_chat(bindings: Dict[str, int], row: Dict[str, Any]) -> int:
|
||||
"""Resolve a row's immutable task binding in delivery lineage order."""
|
||||
for field in ("task_id", "parent_task_id", "root_task_id"):
|
||||
chat = bindings.get(str(row.get(field) or "").strip())
|
||||
if chat:
|
||||
return int(chat)
|
||||
return 0
|
||||
def bound_room_chat(rooms: Callable[..., int], row: Dict[str, Any]) -> int:
|
||||
"""The Project room a row's task lives in by the one room rule (``rooms`` is
|
||||
``projects_registry.RoomResolver``), 0 when it lives in none: a row that
|
||||
carries only its task id resolves through the task's own recorded lineage."""
|
||||
return int(rooms(row.get("task_id"), row.get("parent_task_id"), row.get("root_task_id")) or 0)
|
||||
|
||||
|
||||
def room_membership(chat_id: int, project_chat_ids: set, source_refs: list,
|
||||
bindings: Dict[str, int]):
|
||||
rooms: Callable[..., int]):
|
||||
"""Canonical room membership shared by history and evidence readers.
|
||||
|
||||
``gateway.history_segments.room_segment_lens`` mirrors the Project admissions
|
||||
below to skip archives; change both together.
|
||||
``rooms`` is the room rule (``projects_registry.RoomResolver``), the same one
|
||||
the live bus addresses by, so a task's rows replay in the room its frames went
|
||||
to. ``gateway.history_segments.room_segment_lens`` mirrors the Project
|
||||
admissions below to skip archives; change both together.
|
||||
|
||||
Presentation-only hiding and cross-room question pointers belong to the UI
|
||||
caller. A room source retains the actual cognitive result as well.
|
||||
|
|
@ -300,10 +300,13 @@ def room_membership(chat_id: int, project_chat_ids: set, source_refs: list,
|
|||
return False
|
||||
# A routing refusal belongs to the issuing chat, even when the target
|
||||
# is bound to another Project. Its lineage must not move the notice.
|
||||
bound = 0 if row.get("type") in ORIGIN_ADDRESSED_NOTICE_TYPES else bound_room_chat(bindings, row)
|
||||
bound = 0 if row.get("type") in ORIGIN_ADDRESSED_NOTICE_TYPES else bound_room_chat(rooms, row)
|
||||
lifecycle = row.get("type") in MAIN_PINNED_ROW_TYPES
|
||||
if chat_id in project_chat_ids:
|
||||
return not lifecycle and (bound == chat_id or entry_chat == chat_id
|
||||
# A task's resolved room is authoritative over the address its row was
|
||||
# written with: a later conversion moves the subtree's earlier rows too,
|
||||
# out of the old room (live delivery already went to the new one).
|
||||
return not lifecycle and ((bound == chat_id if bound else entry_chat == chat_id)
|
||||
or entry_matches_source_ref(row, source_refs))
|
||||
if chat_id != 1:
|
||||
return entry_chat == chat_id and not bound
|
||||
|
|
@ -657,6 +660,13 @@ MAIN_NOTICE_TYPE = "main_notice"
|
|||
# Rows pinned to Main whatever the sender's Project binding: the supervisor
|
||||
# send handler and room_membership replay read this one set.
|
||||
MAIN_PINNED_ROW_TYPES = frozenset({"project_started", "project_handoff", "project_completion_summary", MAIN_NOTICE_TYPE})
|
||||
# Rows whose address is the producer's, never their task's room: Main pins, the
|
||||
# origin-addressed refusals, the late-acceptance supplement (its retained chat is
|
||||
# the send handler's decision) and the Main question pointer. The bus write seam
|
||||
# (``message_bus.send_with_budget``) reads this one set; the readers above keep
|
||||
# each component's own reading.
|
||||
ROOM_FIXED_ROW_TYPES = MAIN_PINNED_ROW_TYPES | ORIGIN_ADDRESSED_NOTICE_TYPES | frozenset({
|
||||
"acceptance_late_settlement", "project_question_pointer"})
|
||||
|
||||
# Statuses of an act that LANDED (or is still in flight): no cause sentence.
|
||||
_LANDED_ROUTING_STATUSES = frozenset({"scheduled", "delivered", "pending", "dispatch_pending", "accepted"})
|
||||
|
|
|
|||
|
|
@ -10,6 +10,7 @@ creates fresh execution incarnations; lost folders/tombstones are not recovered
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import uuid
|
||||
import pathlib
|
||||
|
|
@ -20,6 +21,7 @@ from typing import Any, Dict, Iterator, List, Optional
|
|||
from ouroboros.contracts.chat_id_policy import project_chat_id
|
||||
from ouroboros.contracts.schema_versions import with_schema_version
|
||||
from ouroboros.project_facts import sanitize_project_id
|
||||
from ouroboros.store import open_store
|
||||
from ouroboros.utils import atomic_write_json, iter_jsonl_objects, read_json_dict, utc_now_iso, strip_markdown
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
|
@ -42,7 +44,6 @@ PROJECT_ACTIVE = "active"
|
|||
PROJECT_DELETING = "deleting"
|
||||
PROJECT_TOMBSTONED = "tombstoned"
|
||||
PROJECT_LIFECYCLES = frozenset({PROJECT_ACTIVE, PROJECT_DELETING, PROJECT_TOMBSTONED})
|
||||
_DEPRECATED_CHAT_IDS_EVENTS: set[str] = set()
|
||||
|
||||
|
||||
@contextmanager
|
||||
|
|
@ -341,25 +342,10 @@ def project_task_bindings(drive_root: Any) -> Dict[str, Dict[str, Any]]:
|
|||
|
||||
|
||||
def all_task_bindings(drive_root: Any) -> Dict[str, int]:
|
||||
"""Map task_id -> project chat_id for ALL post-hoc 'Turn into project' bindings.
|
||||
|
||||
Cognition/history isolation consults this so a bound task's rows (which keep
|
||||
their ORIGINAL main chat_id) are still treated as project-owned. One bounded
|
||||
read; no per-row lock (atomic writes guarantee complete reads)."""
|
||||
out: Dict[str, int] = {}
|
||||
try:
|
||||
for tid, row in _load_bindings(drive_root).get("bindings", {}).items():
|
||||
if not isinstance(row, dict):
|
||||
continue
|
||||
try:
|
||||
cid = int(row.get("project_chat_id") or 0)
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
if cid:
|
||||
out[str(tid)] = cid
|
||||
except Exception:
|
||||
log.debug("all_task_bindings failed", exc_info=True)
|
||||
return out
|
||||
"""Map task_id -> project chat_id for ALL post-hoc 'Turn into project' bindings:
|
||||
the explicit roots (cursor fingerprints, the census). One bounded read; no
|
||||
per-row lock (atomic writes guarantee complete reads)."""
|
||||
return {tid: int(row["chat_id"]) for tid, row in all_task_project_bindings(drive_root).items()}
|
||||
|
||||
|
||||
def all_task_project_bindings(drive_root: Any, *, strict: bool = False) -> Dict[str, Dict[str, Any]]:
|
||||
|
|
@ -664,35 +650,137 @@ def _bindings_lens(drive_root: Any) -> Dict[str, Any]:
|
|||
return bindings
|
||||
|
||||
|
||||
_TASK_LINEAGE: Dict[tuple, Dict[str, Any]] = {}
|
||||
_TASK_LINEAGE_LIMIT = 50_000 # a memory bound, not a policy: the facts are re-read on the next sight
|
||||
_ROOM_ANSWERS: Dict[str, tuple] = {}
|
||||
_LINEAGE_SELECT = ("SELECT t.task_id, t.parent_task_id, t.root_task_id, t.delegation_role, "
|
||||
"json_extract(b.body, '$.chat_id') FROM tasks t LEFT JOIN task_bodies b ON b.task_id = t.task_id "
|
||||
"WHERE t.task_id IN (SELECT value FROM json_each(?))")
|
||||
|
||||
|
||||
def _read_lineage(drive_root: Any, task_ids: Any) -> Optional[Dict[str, Dict[str, Any]]]:
|
||||
"""ONE indexed read of the lineage facts of ``task_ids`` (parent, root, role and the
|
||||
admitted ``chat_id``), remembered per task id for the process (an existing task's
|
||||
lineage never changes); an id the store does not hold is absent from the answer
|
||||
(admission creates it later); ``None`` when the store cannot answer right now."""
|
||||
try:
|
||||
with open_store(drive_root, write=False) as txn:
|
||||
records = txn.conn.execute(_LINEAGE_SELECT, (json.dumps(sorted(task_ids)),)).fetchall()
|
||||
except Exception:
|
||||
log.debug("task lineage read failed", exc_info=True)
|
||||
return None
|
||||
if len(_TASK_LINEAGE) >= _TASK_LINEAGE_LIMIT:
|
||||
_TASK_LINEAGE.clear()
|
||||
facts = {}
|
||||
for record in records:
|
||||
try:
|
||||
chat = int(record[4]) if record[4] not in (None, "") else None
|
||||
except (TypeError, ValueError):
|
||||
chat = None
|
||||
facts[str(record[0])] = {"parent_task_id": str(record[1] or ""), "root_task_id": str(record[2] or ""),
|
||||
"delegation_role": str(record[3] or ""), "chat_id": chat}
|
||||
for tid, row in facts.items():
|
||||
_TASK_LINEAGE[(str(drive_root), tid)] = row
|
||||
return facts
|
||||
|
||||
|
||||
class RoomResolver:
|
||||
"""``room(task)``, the one rule live routing, history and cognition share: the
|
||||
nearest explicit «Turn into project» binding on task → parent → … (the root
|
||||
only after the parent chain: converting a sub-task moves its own subtree);
|
||||
else the task's own admitted chat when it is a registered Project chat (never
|
||||
``project_id``, which a Main direct turn can carry too); else 0 and the caller
|
||||
keeps its address. Lineage is the caller's or the store's. Answers hold per
|
||||
task id for one bindings snapshot (a conversion invalidates them, a commit does
|
||||
not), never one reached while an ancestor's read was unavailable (retried next
|
||||
call); an id the store has no row for is read once per resolver (request, frame)."""
|
||||
|
||||
__slots__ = ("root", "bindings", "answers", "misses", "_projects")
|
||||
|
||||
def __init__(self, drive_root: Any) -> None:
|
||||
self.root = drive_root
|
||||
self.bindings = _bindings_lens(drive_root) # one ``stat`` per resolver
|
||||
key = str(_bindings_path(drive_root))
|
||||
cached = _ROOM_ANSWERS.get(key)
|
||||
if cached is None or cached[0] is not self.bindings:
|
||||
cached = _ROOM_ANSWERS[key] = (self.bindings, {})
|
||||
self.answers: Dict[str, int] = cached[1]
|
||||
self.misses: set = set()
|
||||
self._projects: Optional[frozenset] = None
|
||||
|
||||
def _facts(self, task_id: str) -> Optional[Dict[str, Any]]:
|
||||
"""Cached or read facts; ``{}`` for a known miss; ``None`` when unavailable now."""
|
||||
facts = _TASK_LINEAGE.get((str(self.root), task_id))
|
||||
if facts is None and task_id not in self.misses:
|
||||
read = _read_lineage(self.root, [task_id])
|
||||
if read is None:
|
||||
return None
|
||||
facts = read.get(task_id)
|
||||
if facts is None:
|
||||
self.misses.add(task_id)
|
||||
return facts or {}
|
||||
|
||||
def lineage(self, task_id: Any) -> Dict[str, Any]:
|
||||
"""The task's facts (``{}`` when unknown or unavailable now): what a reader anchors by."""
|
||||
tid = str(task_id or "").strip()
|
||||
return (self._facts(tid) if tid else None) or {}
|
||||
|
||||
def _bound(self, task_id: str) -> int:
|
||||
row = self.bindings.get(task_id)
|
||||
try:
|
||||
return int(row.get("project_chat_id") or 0) if isinstance(row, dict) else 0
|
||||
except (TypeError, ValueError):
|
||||
return 0
|
||||
|
||||
def __call__(self, task_id: Any, parent_task_id: Any = "", root_task_id: Any = "") -> int:
|
||||
tid = str(task_id or "").strip()
|
||||
if not tid:
|
||||
return 0
|
||||
answer = self.answers.get(tid)
|
||||
if answer is not None:
|
||||
return answer
|
||||
reads: List[Optional[Dict[str, Any]]] = [] # ``None``: an ancestor the store could not answer
|
||||
chain: List[str] = []
|
||||
cur, parent, chat = tid, str(parent_task_id or "").strip(), 0
|
||||
while cur and cur not in chain:
|
||||
chain.append(cur)
|
||||
if chat := self._bound(cur):
|
||||
break
|
||||
if not parent:
|
||||
reads.append(self._facts(cur))
|
||||
parent = (reads[-1] or {}).get("parent_task_id", "")
|
||||
cur, parent = parent, ""
|
||||
own = {} if chat else self._facts(tid)
|
||||
reads.append(own)
|
||||
own = own or {}
|
||||
if not chat:
|
||||
root = str(root_task_id or "").strip() or own.get("root_task_id", "")
|
||||
chat = self._bound(root) if root and root not in chain else 0
|
||||
if not chat and own.get("chat_id"):
|
||||
if self._projects is None:
|
||||
self._projects = project_thread_chat_ids(self.root)
|
||||
chat = own["chat_id"] if own["chat_id"] in self._projects else 0
|
||||
if (chat or own) and None not in reads: # an incomplete ancestry read is never memoized
|
||||
self.answers[tid] = chat
|
||||
return chat
|
||||
|
||||
def warm(self, task_ids: Any) -> None:
|
||||
"""Read every unseen id's lineage in ONE transaction (an archive summary names hundreds)."""
|
||||
unseen = {tid for tid in map(str, task_ids) if tid and (str(self.root), tid) not in _TASK_LINEAGE}
|
||||
read = _read_lineage(self.root, unseen - self.misses) if unseen - self.misses else {}
|
||||
if read is not None:
|
||||
self.misses.update(unseen - read.keys())
|
||||
|
||||
|
||||
def project_chat_for_task_tree(
|
||||
drive_root: Any, task_id: Any, parent_task_id: Any = "", root_task_id: Any = ""
|
||||
) -> int:
|
||||
"""Resolve the project chat for a task by its TASK TREE: the task's OWN binding
|
||||
wins; else inherit from its parent; else its root. A subagent is never bound
|
||||
itself, so this is how its live frames + history are recognized as belonging to
|
||||
its root's project and route to the project thread instead of staying in the main
|
||||
chat (the cyber-racing "subagents vanished from the project" gap). Membership is
|
||||
DERIVED from lineage — no per-child binding store, one SSOT."""
|
||||
# One cached bindings view for all three probes (runs per live log event).
|
||||
"""The room rule (``RoomResolver``) under its live-addressing name: 0 when none; never raises."""
|
||||
try:
|
||||
bindings = _bindings_lens(drive_root)
|
||||
return RoomResolver(drive_root)(task_id, parent_task_id, root_task_id)
|
||||
except Exception:
|
||||
log.debug("project_chat_for_task_tree bindings read failed", exc_info=True)
|
||||
log.debug("project_chat_for_task_tree failed", exc_info=True)
|
||||
return 0
|
||||
for tid in (task_id, parent_task_id, root_task_id):
|
||||
tid = str(tid or "").strip()
|
||||
if not tid:
|
||||
continue
|
||||
row = bindings.get(tid)
|
||||
if not isinstance(row, dict):
|
||||
continue
|
||||
try:
|
||||
chat = int(row.get("project_chat_id") or 0)
|
||||
except (TypeError, ValueError):
|
||||
chat = 0
|
||||
if chat:
|
||||
return chat
|
||||
return 0
|
||||
|
||||
|
||||
def list_reserved_projects(drive_root: Any, *, strict: bool = False) -> List[Dict[str, Any]]:
|
||||
|
|
@ -790,28 +878,6 @@ def stamp_project_thread(drive_root: Any, payload: Dict[str, Any]) -> None:
|
|||
log.debug("Project thread lens failed", exc_info=True)
|
||||
|
||||
|
||||
def registered_project_chat_ids(drive_root: Any) -> set:
|
||||
"""One-minor compatibility alias for :func:`reserved_project_chat_ids`."""
|
||||
key = str(pathlib.Path(drive_root).resolve(strict=False))
|
||||
if key not in _DEPRECATED_CHAT_IDS_EVENTS:
|
||||
_DEPRECATED_CHAT_IDS_EVENTS.add(key)
|
||||
try:
|
||||
from ouroboros.utils import append_jsonl
|
||||
|
||||
append_jsonl(
|
||||
pathlib.Path(drive_root) / "logs" / "events.jsonl",
|
||||
{
|
||||
"ts": utc_now_iso(),
|
||||
"type": "deprecated_project_chat_ids_alias_used",
|
||||
"alias": "registered_project_chat_ids",
|
||||
"replacement": "reserved_project_chat_ids",
|
||||
},
|
||||
)
|
||||
except Exception:
|
||||
log.debug("Failed to record Project chat-id alias use", exc_info=True)
|
||||
return reserved_project_chat_ids(drive_root)
|
||||
|
||||
|
||||
def get_project(drive_root: Any, project_id: str, *, strict: bool = False) -> Optional[Dict[str, Any]]:
|
||||
project = get_reserved_project(drive_root, project_id, strict=strict)
|
||||
return project if project and project.get("lifecycle") == PROJECT_ACTIVE else None
|
||||
|
|
@ -1481,11 +1547,11 @@ __all__ = [
|
|||
"project_chat_for_task",
|
||||
"project_thread_note_for_task",
|
||||
"project_chat_for_task_tree",
|
||||
"RoomResolver",
|
||||
"project_id_for_origin",
|
||||
"project_task_bindings",
|
||||
"task_presentation_snapshot",
|
||||
"task_presentation_name",
|
||||
"registered_project_chat_ids",
|
||||
"reserved_project_chat_ids",
|
||||
"projects_summary",
|
||||
"reconcile_projects",
|
||||
|
|
|
|||
|
|
@ -730,26 +730,32 @@ def _normalize_swarm_efficiency(data: Dict[str, Any]) -> Dict[str, Any]:
|
|||
|
||||
|
||||
def load_task_result(
|
||||
drive_root: Any, task_id: str, *, strict: bool = False,
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
drive_root: Any, task_id: str, *, strict: bool = False, signed: bool = False,
|
||||
) -> Any:
|
||||
"""Read one exact result through schema admission, tolerating old fan-out names.
|
||||
|
||||
Fail-soft refusal quarantines with a batched event and returns None; strict
|
||||
refusal raises without moving, so unreadable identities cannot be readmitted.
|
||||
Neither mode materializes deliverables or independent state axes."""
|
||||
Neither mode materializes deliverables or independent state axes. ``signed``
|
||||
returns ``(result, signature)`` from ONE read transaction: an ownership check
|
||||
(``supervisor.task_ownership``) re-checks the stamp of the very body it holds,
|
||||
so a writer landing between two reads is a change and a busy store is unknown,
|
||||
never a false change."""
|
||||
try:
|
||||
tid = validate_task_id(task_id)
|
||||
except ValueError:
|
||||
if strict:
|
||||
raise
|
||||
return None
|
||||
data = read_document(drive_root, tid)
|
||||
return (None, None) if signed else None
|
||||
data, stamp = read_document(drive_root, tid, signed=True) if signed else (read_document(drive_root, tid), None)
|
||||
if data is None:
|
||||
if strict:
|
||||
_refuse_occupied(drive_root, tid, write=False)
|
||||
return None # This read saw absence even if a writer publishes immediately after it.
|
||||
_admit_stored(data, tid, strict=strict, noun="task result authority")
|
||||
return _normalize_swarm_efficiency(data)
|
||||
result = None # This read saw absence even if a writer publishes immediately after it.
|
||||
else:
|
||||
_admit_stored(data, tid, strict=strict, noun="task result authority")
|
||||
result = _normalize_swarm_efficiency(data)
|
||||
return (result, stamp) if signed else result
|
||||
|
||||
|
||||
def _refuse_occupied(drive_root: Any, task_id: str, *, write: bool) -> None:
|
||||
|
|
|
|||
|
|
@ -21,20 +21,25 @@ FACT_KEYS = ("task_id", "status", "ts", "updated_at", "delegation_role", "parent
|
|||
_NEWEST_ORDER = "ORDER BY ts DESC, task_id DESC" # NULL ts sorts last: a row without one is oldest
|
||||
|
||||
|
||||
def read_document(root: Any, task_id: str, *, txn: Optional[Txn] = None) -> Optional[Dict[str, Any]]:
|
||||
def read_document(root: Any, task_id: str, *, txn: Optional[Txn] = None, signed: bool = False) -> Any:
|
||||
"""One coherent document; implicit reads decode only after releasing the store.
|
||||
|
||||
A supplied transaction keeps its own read/modify/write snapshot and lifetime.
|
||||
``signed=True`` returns ``(document, signature)`` captured in that same read
|
||||
transaction, so an ownership check compares the stamp of the body it holds and
|
||||
never a second read's.
|
||||
"""
|
||||
if txn is not None:
|
||||
return txn.load(str(task_id))
|
||||
document = txn.load(str(task_id))
|
||||
return (document, signature(root, task_id, txn=txn)) if signed else document
|
||||
task_id = str(task_id)
|
||||
with open_store(root, write=False) as txn:
|
||||
stamp = signature(root, task_id, txn=txn) if signed else None
|
||||
body = txn.body_text(task_id)
|
||||
rows = txn.handoff_records([task_id]) if body is not None else []
|
||||
if body is None:
|
||||
return None
|
||||
return store.project_handoffs(json.loads(body), store.decode_handoffs([task_id], rows)[task_id])
|
||||
document = None if body is None else store.project_handoffs(
|
||||
json.loads(body), store.decode_handoffs([task_id], rows)[task_id])
|
||||
return (document, stamp) if signed else document
|
||||
|
||||
|
||||
def exists(root: Any, task_id: str) -> bool:
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ from typing import Any
|
|||
|
||||
from ouroboros.dialogue_evidence import own_room_chat, read_room_source, task_room_record
|
||||
from ouroboros.owner_words import owner_words_text
|
||||
from ouroboros.projects_registry import all_task_bindings, list_reserved_projects
|
||||
from ouroboros.projects_registry import all_task_bindings, list_reserved_projects, RoomResolver
|
||||
from ouroboros.task_results import load_plan_review_state
|
||||
|
||||
|
||||
|
|
@ -15,13 +15,13 @@ def related_rooms(ctx: Any, root: pathlib.Path, own_chat: int | None) -> list[di
|
|||
"""Only Main and task-lineage rooms, using existing registry activity facts."""
|
||||
task_id = str(getattr(ctx, "task_id", "") or "")
|
||||
task = {**(task_room_record(root, task_id)), **(getattr(ctx, "task_metadata", {}) or {}), "task_id": task_id}
|
||||
bindings = all_task_bindings(root)
|
||||
bindings, rooms = all_task_bindings(root), RoomResolver(root)
|
||||
chats = {1} if own_chat != 1 else set()
|
||||
for field in ("task_id", "parent_task_id", "root_task_id"):
|
||||
tid = str(task.get(field) or "")
|
||||
if tid:
|
||||
source = task_room_record(root, tid)
|
||||
chat = bindings.get(tid) or source.get("chat_id")
|
||||
chat = rooms(tid) or source.get("chat_id")
|
||||
if chat is not None:
|
||||
chats.add(int(chat))
|
||||
from ouroboros.task_results import resolve_task_lineage
|
||||
|
|
|
|||
|
|
@ -681,7 +681,7 @@ def _run_supervisor(settings: dict) -> None:
|
|||
from supervisor.queue import (
|
||||
enqueue_task, enforce_task_timeouts, enqueue_evolution_task_if_needed,
|
||||
persist_queue_snapshot, restore_pending_from_snapshot,
|
||||
cancel_task_by_id, queue_deep_self_review_task, sort_pending,
|
||||
queue_deep_self_review_task, sort_pending,
|
||||
)
|
||||
from supervisor.workers import (
|
||||
init as workers_init, get_event_q, WORKERS, PENDING, RUNNING,
|
||||
|
|
@ -777,7 +777,7 @@ def _run_supervisor(settings: dict) -> None:
|
|||
send_with_budget=send_with_budget, load_state=load_state, save_state=save_state,
|
||||
update_state=update_state,
|
||||
update_budget_from_usage=update_budget_from_usage, append_jsonl=append_jsonl,
|
||||
enqueue_task=enqueue_task, cancel_task_by_id=cancel_task_by_id,
|
||||
enqueue_task=enqueue_task,
|
||||
queue_deep_self_review_task=queue_deep_self_review_task, persist_queue_snapshot=persist_queue_snapshot,
|
||||
safe_restart=safe_restart, kill_workers=kill_workers, spawn_workers=spawn_workers,
|
||||
sort_pending=sort_pending, consciousness=_consciousness,
|
||||
|
|
@ -1324,7 +1324,7 @@ async def lifespan(app):
|
|||
log.warning("Native skills bootstrap failed", exc_info=True)
|
||||
|
||||
# Boot-reconcile the project registry BEFORE /api/state and context-building
|
||||
# can rely on registered_project_chat_ids (the multi-project isolation SSOT):
|
||||
# can rely on reserved_project_chat_ids (the multi-project isolation SSOT):
|
||||
# register any pre-existing data/projects/<id>/ store whose row is missing, so
|
||||
# an inherited project's raw chat is partitioned from turn one (not only after
|
||||
# the 300s periodic tick). Idempotent and never prunes.
|
||||
|
|
|
|||
|
|
@ -737,16 +737,26 @@ def stop_paused_late_phase_custody(q: Any, task_id: str, *, intent: Optional[Dic
|
|||
return True
|
||||
|
||||
|
||||
def live_presence_row(row: Any) -> bool:
|
||||
"""An unsettled row a live Presence turn owns: its runner settles it, not cancel custody."""
|
||||
from supervisor.queue_transitions import settled_status_of
|
||||
|
||||
owner = row.get("execution_owner") if isinstance(row, dict) else None
|
||||
return isinstance(owner, dict) and owner.get("kind") == "presence" and not settled_status_of(row)
|
||||
|
||||
|
||||
def _finalize_cancel_intent_on_miss(
|
||||
q: Any, task_id: str, *, intent: Optional[Dict[str, Any]] = None,
|
||||
) -> str:
|
||||
"""Neither queued nor running: settle an open cancel intent (or a legacy
|
||||
``cancel_requested`` latch file) as cancelled with reconstructed cost.
|
||||
|
||||
Two things this lane must NOT do. It must not invent a task: an intent for an
|
||||
id that has no durable result at all names a task that never existed, and
|
||||
Three things this lane must NOT do. It must not invent a task: an intent for
|
||||
an id that has no durable result at all names a task that never existed, and
|
||||
fabricating a ``cancelled`` row with $0 for it would put a phantom task in the
|
||||
ledger — it settles as ``not_found`` instead. And it must not bury a child
|
||||
ledger — it settles as ``not_found`` instead. It must not take a row whose
|
||||
execution owner is a live Presence turn (no supervisor map knows it): that
|
||||
settles ``not_found`` too, the row untouched. And it must not bury a child
|
||||
that finished: when the row names a child drive, the child's own result is
|
||||
copied back BEFORE the cancelled write, so a crash of the split-drive
|
||||
copy-back window cannot cost a completed answer.
|
||||
|
|
@ -780,6 +790,14 @@ def _finalize_cancel_intent_on_miss(
|
|||
_settle_intent(q, task_id, outcome="not_found",
|
||||
detail="no durable task result for this id", intent=intent)
|
||||
return CANCEL_NOT_FOUND
|
||||
if live_presence_row(existing):
|
||||
# A live Presence turn runs on its own thread outside every supervisor
|
||||
# map: nothing here can capture it, and a cancelled write would only
|
||||
# strand a turn that keeps spending. The row stays its runner's and
|
||||
# the Stop answers "not active", as it did before intent-first minting.
|
||||
_settle_intent(q, task_id, outcome="not_found",
|
||||
detail="execution owned by a live Presence turn", intent=intent)
|
||||
return CANCEL_NOT_FOUND
|
||||
# A concurrent custody attempt may have captured this task between our
|
||||
# own capture miss and here (the pending double-settle probe). If the
|
||||
# live claim is no longer ours, it owns the settle — refuse and let it,
|
||||
|
|
|
|||
|
|
@ -150,41 +150,24 @@ def _handle_cancel_task(evt: Dict[str, Any], ctx: Any) -> None:
|
|||
|
||||
|
||||
def _drive_cancel_task_event(evt: Dict[str, Any], ctx: Any) -> None:
|
||||
"""Drive one agent-requested cancel through custody — TYPED outcome end to end.
|
||||
"""Wake the durable intent's own driver; the event itself says nothing.
|
||||
|
||||
Custody publishes the settled truth itself: a cancelled child's
|
||||
``task_summary``/salvage row lands in the task's own thread through the
|
||||
terminal-delivery seam, an already-settled child keeps the result it
|
||||
already published, and the calling agent received the typed tool result.
|
||||
A second host acknowledgement here (#624) reached the global owner chat as
|
||||
an untyped assistant row with no task identity — duplicate presentation in
|
||||
Main Chat, and for a pending-drop race it misstated the causal history.
|
||||
Only the FAILED outcome still speaks: the task is live, the intent stays
|
||||
open, and the owner needs the typed incident."""
|
||||
A ``cancel_task`` event is only the fast trigger for an intent the tool
|
||||
already minted. Every outcome is published where it belongs: a cancelled
|
||||
child's ``task_summary``/salvage row lands in the task's own thread through
|
||||
the terminal-delivery seam, an already-settled child keeps the result it
|
||||
published, a closed or claimed intent is a silent no-op, and a FAILED
|
||||
attempt leaves the intent open for ``sweep_cancel_intents`` with its typed
|
||||
trail in logs/supervisor.jsonl. The host sentence this used to send on
|
||||
FAILED (#624) named a child with no lineage in the owner's chat and, for an
|
||||
event drained long after the watchdog had settled the intent, stated a
|
||||
failure that never happened: the card and the journal are the record."""
|
||||
task_id = str(evt.get("task_id") or "").strip()
|
||||
requested_task_id = str(evt.get("requested_task_id") or "").strip()
|
||||
display_task_id = requested_task_id or task_id
|
||||
st = ctx.load_state()
|
||||
owner_chat_id = st.get("owner_chat_id")
|
||||
from supervisor.queue import CANCEL_FAILED, CANCEL_NOT_FOUND, drive_cancel_intent_scope
|
||||
|
||||
outcome = drive_cancel_intent_scope(task_id) if task_id else CANCEL_NOT_FOUND
|
||||
if not owner_chat_id or outcome != CANCEL_FAILED:
|
||||
if not task_id:
|
||||
return
|
||||
incident_meta = {
|
||||
"task_incident": "cancellation_fault",
|
||||
"toast_once": f"{display_task_id or 'unknown'}:cancellation_fault",
|
||||
}
|
||||
if task_id and display_task_id != task_id:
|
||||
incident_meta["cancel_physical_task_id"] = task_id
|
||||
ctx.send_with_budget(
|
||||
int(owner_chat_id),
|
||||
f"❌ cancel {display_task_id or '?'} did not settle — the task is still live; "
|
||||
"the durable cancel intent stays open and the supervisor watchdog retries (event)",
|
||||
is_progress=True,
|
||||
task_id=display_task_id,
|
||||
progress_meta=incident_meta,
|
||||
role="system", system_type="cancellation_notice")
|
||||
from supervisor.queue import drive_cancel_intent_scope
|
||||
|
||||
drive_cancel_intent_scope(task_id)
|
||||
|
||||
|
||||
EVOLUTION_CONTROL_KEYS = ("evolution_mode_enabled", "evolution_owner_stopped",
|
||||
|
|
|
|||
|
|
@ -20,7 +20,8 @@ from supervisor.state import append_jsonl, load_state
|
|||
from supervisor.message_ingress import _INGRESS_LOCK, _accepted_web_message, _take_undispatched, _write_marked, _enter_dispatch, delivery_facts
|
||||
from supervisor.message_ingress import ( # noqa: F401 - the named ingress's public names stay importable here
|
||||
_UNDISPATCHED, accept_local_message, accepted_chat_message, acceptance_undispatched, record_inbound_message)
|
||||
from ouroboros.projects_registry import stamp_project_thread
|
||||
from ouroboros.project_dialogue import ROOM_FIXED_ROW_TYPES
|
||||
from ouroboros.projects_registry import RoomResolver, stamp_project_thread
|
||||
from ouroboros.tools.core import (
|
||||
LinkActionsValidationError,
|
||||
QuizValidationError,
|
||||
|
|
@ -1452,6 +1453,21 @@ def send_with_budget(chat_id: int, text: str, log_text: Optional[str] = None,
|
|||
meta["narration"] = bool(narration)
|
||||
meta.setdefault("narration", False)
|
||||
progress_meta = meta
|
||||
if task_id and DATA_DIR and system_type not in ROOM_FIXED_ROW_TYPES and meta.get("_is_direct_chat") is not True:
|
||||
# The one room rule (A4) at the one send seam: a task-addressed host row
|
||||
# lands in its task's Project room on the durable row, the WebSocket frame
|
||||
# and every transport alike, whichever producer addressed it. A child row
|
||||
# that reached here without its lineage borrows it from the same cached
|
||||
# facts (never a result read), so its card nests it live and on replay.
|
||||
try:
|
||||
rooms = RoomResolver(DATA_DIR)
|
||||
chat_id = rooms(task_id, meta.get("parent_task_id"), meta.get("root_task_id")) or chat_id
|
||||
lineage = {} if meta.get("parent_task_id") else rooms.lineage(task_id)
|
||||
for key in ("parent_task_id", "root_task_id", "delegation_role"):
|
||||
if lineage.get(key) and not meta.get(key):
|
||||
meta[key] = lineage[key]
|
||||
except Exception:
|
||||
log.debug("room rule failed for %s; the producer's address stands", task_id, exc_info=True)
|
||||
|
||||
if is_progress and DATA_DIR:
|
||||
progress_record = {
|
||||
|
|
|
|||
|
|
@ -647,11 +647,6 @@ from supervisor.queue_transitions import ( # noqa: E402, F401 -- intentional pu
|
|||
)
|
||||
|
||||
|
||||
def _cancel_task_by_id_single(task_id: str) -> bool:
|
||||
"""Boolean facade for the pre-v6.82 single-task callers."""
|
||||
return cancel_task_custody(task_id) in {CANCEL_CANCELLED, CANCEL_ALREADY_SETTLED}
|
||||
|
||||
|
||||
# Evolution-stop transitions (GR2-13) live in supervisor.queue_transitions
|
||||
# (module-size boundary); re-exported below with the other transition helpers
|
||||
# so `supervisor.queue` stays the single import surface for callers.
|
||||
|
|
|
|||
|
|
@ -945,17 +945,23 @@ def _settled_status(drive_root: object, task_id: str) -> str:
|
|||
"""The task's own already-settled durable status, or "" — fail-soft."""
|
||||
try:
|
||||
from ouroboros.task_results import load_task_result
|
||||
from ouroboros.task_status import SETTLED_STATUSES
|
||||
|
||||
status = str((load_task_result(drive_root, task_id) or {}).get("status") or "")
|
||||
return status if status in SETTLED_STATUSES else ""
|
||||
return settled_status_of(load_task_result(drive_root, task_id) or {})
|
||||
except Exception:
|
||||
log.debug("settled-status read failed for %s", task_id, exc_info=True)
|
||||
return ""
|
||||
|
||||
|
||||
def settled_status_of(row: Any) -> str:
|
||||
"""The row's terminal status, or "" (custody judges the row it prepared outside the lock)."""
|
||||
from ouroboros.task_status import SETTLED_STATUSES
|
||||
|
||||
status = str((row or {}).get("status") or "")
|
||||
return status if status in SETTLED_STATUSES else ""
|
||||
|
||||
|
||||
def task_subtree_is_live(task_id: str, *, ignore_intents: bool = False) -> bool:
|
||||
"""Cheap liveness pre-check for the HTTP cascade-cancel path (v6.82).
|
||||
"""Cheap liveness check the HTTP Stop asks only for a settled or row-less cascade target.
|
||||
|
||||
True when the task itself is queued/running, when it still has live
|
||||
descendants in the queue, when THIS process still runs its paid post-task
|
||||
|
|
@ -1029,7 +1035,12 @@ def _live_retry_target_locked(q: Any, task_id: str, *, results) -> Tuple[str, st
|
|||
host-written result chain (old points to new; new points back to old), then
|
||||
require the live task row to satisfy the existing root-retry contract.
|
||||
Corrupt, overlong, cyclic, or ambiguous chains raise so custody leaves the
|
||||
intent open and fails closed.
|
||||
intent open and fails closed. The chain is walked — and its rows re-checked
|
||||
under the lock — only when the requested row has ``superseded_by``: any
|
||||
other task is its own physical id, and its live worker rewriting the row
|
||||
between preparation and this lock changes nothing about who owns it (a
|
||||
retry admitted meanwhile is still caught below as a live root-shaped row
|
||||
outside the prepared chain).
|
||||
"""
|
||||
requested = str(task_id or "").strip()
|
||||
if not requested:
|
||||
|
|
@ -1050,7 +1061,7 @@ def _live_retry_target_locked(q: Any, task_id: str, *, results) -> Tuple[str, st
|
|||
|
||||
from ouroboros.task_results import resolve_task_lineage
|
||||
|
||||
requested_result = results.load(requested)
|
||||
requested_result = results.prepared(requested)
|
||||
requested_shape = (
|
||||
requested_result
|
||||
if requested_result
|
||||
|
|
@ -1080,8 +1091,9 @@ def _live_retry_target_locked(q: Any, task_id: str, *, results) -> Tuple[str, st
|
|||
except (TypeError, ValueError, OverflowError):
|
||||
max_edges = 0
|
||||
logical_root = requested
|
||||
walks_chain = bool(str(requested_result.get("superseded_by") or "").strip())
|
||||
while True:
|
||||
current = results.load(current_id) or {}
|
||||
current = (results.load(current_id) if walks_chain else requested_result) or {}
|
||||
if current_id == requested:
|
||||
logical_root = str(current.get("root_task_id") or requested)
|
||||
superseded_by = str(current.get("superseded_by") or "").strip()
|
||||
|
|
@ -1400,7 +1412,7 @@ def run_project_deletion(
|
|||
errors.append(f"{task_id}: cancel_intent_write_failed")
|
||||
continue
|
||||
try:
|
||||
q.cancel_task_by_id(task_id, cascade=True)
|
||||
q.cancel_task_by_id(task_id)
|
||||
except Exception as exc:
|
||||
errors.append(f"{task_id}: {type(exc).__name__}: {exc}")
|
||||
first_pass = False
|
||||
|
|
|
|||
|
|
@ -35,6 +35,7 @@ from supervisor.cancel_publication import ( # noqa: F401 -- intentional public
|
|||
_cascade_delivery_row_locked,
|
||||
_custody_disclosure_fields,
|
||||
_deliver_on_miss,
|
||||
_finalize_cancel_intent_on_miss,
|
||||
_intent_outcome_fields,
|
||||
_is_workspace_task_record,
|
||||
_load_result_row,
|
||||
|
|
@ -241,17 +242,8 @@ def _queue_module():
|
|||
return queue
|
||||
|
||||
|
||||
def _finalize_cancel_intent_on_miss(
|
||||
task_id: str, *, intent: Optional[Dict[str, Any]] = None,
|
||||
) -> str:
|
||||
"""Preserve the lifecycle import surface while publication owns the work."""
|
||||
from supervisor.cancel_publication import _finalize_cancel_intent_on_miss as finalize
|
||||
|
||||
return finalize(_queue_module(), task_id, intent=intent)
|
||||
|
||||
|
||||
def cancel_task_by_id(task_id: str, *, cascade: bool = False) -> bool:
|
||||
"""Cancel a task and, when requested, its atomically captured live subtree.
|
||||
def cancel_task_by_id(task_id: str) -> bool:
|
||||
"""Cancel a task and its atomically captured live subtree.
|
||||
|
||||
The cascade is SYNCHRONOUS end to end (v6.82.0): the caller is answered only
|
||||
once the tree is actually torn down. That single decision is what removes the
|
||||
|
|
@ -278,13 +270,14 @@ def cancel_task_by_id(task_id: str, *, cascade: bool = False) -> bool:
|
|||
Completion always wins: custody is never taken from a task that already
|
||||
reached its own terminal result, and the durable write is refused by the
|
||||
result writer's monotonic guard if it settles mid-teardown.
|
||||
|
||||
Cascade-only: a single cancel is ``cancel_task_custody`` with its typed
|
||||
outcome; the no-intent boolean facade is gone (invariant 14).
|
||||
"""
|
||||
q = _queue_module()
|
||||
task_id = str(task_id or "").strip()
|
||||
if not task_id:
|
||||
return False
|
||||
if not cascade:
|
||||
return q._cancel_task_by_id_single(task_id)
|
||||
# Close admission for the TARGET before the first snapshot (a schedule event
|
||||
# already draining would otherwise slip in), then each sweep fences every id it
|
||||
# captures — so a child naming a since-removed descendant as its parent is still
|
||||
|
|
@ -294,6 +287,10 @@ def cancel_task_by_id(task_id: str, *, cascade: bool = False) -> bool:
|
|||
# Scope on the ROOT intent: a watchdog replay must re-run the cascade, not a
|
||||
# single cancel that would settle the root and leave descendants live.
|
||||
_record_cascade_scope(q, task_id)
|
||||
# That root intent is the tree's explicit authority (A2): every custody of
|
||||
# this cascade runs under it, so a child whose own mint failed is still
|
||||
# stopped; a tree nobody asked to cancel has none and nothing is touched.
|
||||
cascade_claim = _active_intent(q, task_id)
|
||||
with q._queue_lock:
|
||||
CANCELLED_ROOT_FENCES[task_id] = utc_now_iso()
|
||||
_ACTIVE_CASCADE_FENCES[cascade_token] = {task_id}
|
||||
|
|
@ -324,7 +321,7 @@ def cancel_task_by_id(task_id: str, *, cascade: bool = False) -> bool:
|
|||
all_outcomes: Dict[str, str] = {}
|
||||
try:
|
||||
for _sweep in range(4):
|
||||
swept, outcomes = _cancel_subtree_sweep(q, task_id, already, cascade_token)
|
||||
swept, outcomes = _cancel_subtree_sweep(q, task_id, already, cascade_token, cascade_claim)
|
||||
cancelled = swept or cancelled
|
||||
if not outcomes:
|
||||
break
|
||||
|
|
@ -428,8 +425,9 @@ def drive_cancel_intent_scope(task_id: str, *, deliver: bool = True) -> str:
|
|||
Ingress may widen a previously-single request to ``cascade`` and Stop-now
|
||||
may harden an already-graceful cascade. Every in-band consumer therefore
|
||||
reads the durable row immediately before teardown instead of replaying the
|
||||
raw request shape. Absence keeps the legacy direct-custody behavior;
|
||||
unreadable authority fails closed.
|
||||
raw request shape. No open intent is a typed no-op that touches nothing
|
||||
(``already_settled`` for a settled result, else ``not_found``: invariant
|
||||
14 gives a stale wake no authority); unreadable authority fails closed.
|
||||
"""
|
||||
q = _queue_module()
|
||||
task_id = str(task_id or "").strip()
|
||||
|
|
@ -449,23 +447,24 @@ def drive_cancel_intent_scope(task_id: str, *, deliver: bool = True) -> str:
|
|||
exc_info=True,
|
||||
)
|
||||
return CANCEL_FAILED
|
||||
if isinstance(intent, dict):
|
||||
try:
|
||||
from supervisor.owner_stop import revoke_hardened_owner_stop_control
|
||||
if not isinstance(intent, dict):
|
||||
return CANCEL_ALREADY_SETTLED if _settled_status(q.DRIVE_ROOT, task_id) else CANCEL_NOT_FOUND
|
||||
try:
|
||||
from supervisor.owner_stop import revoke_hardened_owner_stop_control
|
||||
|
||||
revoke_hardened_owner_stop_control(q, task_id, intent)
|
||||
except Exception:
|
||||
log.debug(
|
||||
"Hardened owner-stop cleanup failed for %s",
|
||||
task_id,
|
||||
exc_info=True,
|
||||
)
|
||||
if str(intent.get("scope") or "") == SCOPE_CASCADE:
|
||||
return (
|
||||
CANCEL_CANCELLED
|
||||
if q.cancel_task_by_id(task_id, cascade=True)
|
||||
else CANCEL_FAILED
|
||||
)
|
||||
revoke_hardened_owner_stop_control(q, task_id, intent)
|
||||
except Exception:
|
||||
log.debug(
|
||||
"Hardened owner-stop cleanup failed for %s",
|
||||
task_id,
|
||||
exc_info=True,
|
||||
)
|
||||
if str(intent.get("scope") or "") == SCOPE_CASCADE:
|
||||
return (
|
||||
CANCEL_CANCELLED
|
||||
if q.cancel_task_by_id(task_id)
|
||||
else CANCEL_FAILED
|
||||
)
|
||||
return q.cancel_task_custody(task_id, deliver=deliver)
|
||||
|
||||
|
||||
|
|
@ -502,7 +501,9 @@ def _record_cascade_scope(q: Any, task_id: str) -> None:
|
|||
log.debug("cascade-scope forensic append failed for %s", task_id, exc_info=True)
|
||||
|
||||
|
||||
def cancel_task_custody(task_id: str, *, deliver: bool = True) -> str:
|
||||
def cancel_task_custody(
|
||||
task_id: str, *, deliver: bool = True, cascade_claim: Optional[Dict[str, Any]] = None,
|
||||
) -> str:
|
||||
"""Cancel one task and return a TYPED outcome, never a bare boolean.
|
||||
|
||||
The ONE settle owner for cancellation (phase A): every ingress records a
|
||||
|
|
@ -517,9 +518,10 @@ def cancel_task_custody(task_id: str, *, deliver: bool = True) -> str:
|
|||
mutation. Two custody attempts racing the same task used to interleave —
|
||||
the loser entered the capture-miss lane before the winner claimed, saw
|
||||
no live claim, and double-settled (two ``cancelled`` writes, two
|
||||
``task_done`` events). A refused claim is now ``failed`` with ZERO
|
||||
mutation; ``{}`` (no intent at all) keeps the legacy path, where the
|
||||
capture under the queue lock is the mutual exclusion.
|
||||
``task_done`` events). A refused claim is ``failed`` with ZERO
|
||||
mutation; no intent at all is a typed no-op with ZERO mutation (A2,
|
||||
invariant 14) unless ``cascade_claim`` — the root's open cascade intent
|
||||
a subtree sweep runs under — authorizes this descendant.
|
||||
1. UNDER THE QUEUE LOCK — capture. A pending task leaves q.PENDING; a running
|
||||
task keeps its authoritative q.RUNNING row and its worker slot is marked
|
||||
``reaping`` so no other actor can dispatch, reap, or respawn it.
|
||||
|
|
@ -540,7 +542,8 @@ def cancel_task_custody(task_id: str, *, deliver: bool = True) -> str:
|
|||
patch-capture seam, so its diff does not prove which actor authored it.
|
||||
|
||||
``deliver=False`` suppresses the per-task salvage chat delivery (cascade
|
||||
sweeps deliver ONE root message with a children digest instead).
|
||||
sweeps deliver ONE root message with a children digest instead); under a
|
||||
``cascade_claim`` a child whose own intent write failed is still stopped.
|
||||
"""
|
||||
q = _queue_module()
|
||||
from supervisor import workers
|
||||
|
|
@ -562,10 +565,13 @@ def cancel_task_custody(task_id: str, *, deliver: bool = True) -> str:
|
|||
prepare_retry_chain(q, task_id, reads.load)
|
||||
with q._queue_lock:
|
||||
retry_target, settled_retry_status = _live_retry_target_locked(q, task_id, results=reads)
|
||||
except Exception:
|
||||
except Exception as exc:
|
||||
# A busy store or a changed chain row under the lock is UNKNOWN: nothing
|
||||
# was claimed or touched, the intent stays open, the watchdog retries.
|
||||
log.error(
|
||||
"Cancellation retry-lineage authority is indeterminate for %s",
|
||||
"Cancellation retry-lineage authority is indeterminate for %s (%s)",
|
||||
task_id,
|
||||
exc,
|
||||
exc_info=True,
|
||||
)
|
||||
return CANCEL_FAILED
|
||||
|
|
@ -577,6 +583,11 @@ def cancel_task_custody(task_id: str, *, deliver: bool = True) -> str:
|
|||
return _cancel_live_retry_successor(
|
||||
q, task_id, retry_target, deliver=deliver,
|
||||
)
|
||||
# The capture judges the row prepared outside the lock (A3): a full parse
|
||||
# under the queue lock makes every admission wait on SQLite, and a result
|
||||
# landing meanwhile is re-read after the kill (completion wins) or on miss.
|
||||
prepared = reads.prepared(task_id)
|
||||
settled = settled_status_of(prepared)
|
||||
|
||||
# Read the durable intent BEFORE claiming it. The pre-claim row is what the
|
||||
# reaping-takeover gate below judges: a slot already marked ``reaping`` is
|
||||
|
|
@ -596,6 +607,9 @@ def cancel_task_custody(task_id: str, *, deliver: bool = True) -> str:
|
|||
intent = _claim_intent(q, task_id)
|
||||
if intent.get("claim_refused"):
|
||||
return CANCEL_FAILED
|
||||
if not intent and not cascade_claim:
|
||||
# No durable request, no authority: typed from the stored result, nothing touched.
|
||||
return CANCEL_ALREADY_SETTLED if settled else CANCEL_NOT_FOUND
|
||||
generation = intent.get("generation")
|
||||
request_id = str(intent.get("request_id") or "")
|
||||
# Takeover authority (AR2-11, re-based on claim-first): our claim proves a
|
||||
|
|
@ -619,7 +633,6 @@ def cancel_task_custody(task_id: str, *, deliver: bool = True) -> str:
|
|||
captured_meta = None
|
||||
captured_running = None
|
||||
with q._queue_lock:
|
||||
settled = _settled_status(q.DRIVE_ROOT, task_id)
|
||||
if settled:
|
||||
# Natural completion (or an earlier cancel) already decided this task.
|
||||
# A QUEUED row for a task with a terminal result is a ghost and is
|
||||
|
|
@ -645,7 +658,7 @@ def cancel_task_custody(task_id: str, *, deliver: bool = True) -> str:
|
|||
from ouroboros.headless import terminal_task_files_ready
|
||||
bound_task = (q.RUNNING.get(task_id) or {}).get("task") or {"id": task_id}
|
||||
if (settled and not _worker_possibly_alive(worker)
|
||||
and terminal_task_files_ready(q.DRIVE_ROOT, bound_task, _load_result_row(q, task_id))):
|
||||
and terminal_task_files_ready(q.DRIVE_ROOT, bound_task, prepared)):
|
||||
# Settled result AND provably dead process: no live
|
||||
# ownership remains — the fast path below settles and
|
||||
# recovers a stranded ``reaping`` marker. Only a
|
||||
|
|
@ -763,7 +776,7 @@ def cancel_task_custody(task_id: str, *, deliver: bool = True) -> str:
|
|||
intent=intent, deliver=deliver, settled_status=settled,
|
||||
captured_running=captured_running,
|
||||
)
|
||||
return _finalize_cancel_intent_on_miss(task_id, intent=intent)
|
||||
return _finalize_cancel_intent_on_miss(q, task_id, intent=intent)
|
||||
except Exception:
|
||||
# A crash BETWEEN the capture and the respawn is what strands a slot at
|
||||
# ``reaping`` forever (the reaper's step-5 self-heal has the same
|
||||
|
|
@ -883,10 +896,9 @@ def _claim_intent(q: Any, task_id: str) -> Dict[str, Any]:
|
|||
|
||||
The two remaining shapes are deliberately DISTINCT (AR2-2):
|
||||
|
||||
- ``{}`` means NO ACTIVE INTENT exists — the legacy/no-intent path. Custody
|
||||
may proceed: capture under the queue lock is the mutual exclusion for a
|
||||
task nobody minted an intent for (pre-migration legacy latches, direct
|
||||
custody callers), and the later ``_settle_intent`` no-ops harmlessly.
|
||||
- ``{}`` means NO ACTIVE INTENT exists, and custody stops there (A2,
|
||||
invariant 14): nothing to claim, settle or replay, so the caller answers
|
||||
typed from the stored result and mutates nothing (a cascade claim aside).
|
||||
- A claim attempt that RAISED cannot tell whether a live owner exists, so
|
||||
it is treated as refused: proceeding would settle without the exclusivity
|
||||
the fence exists to prove.
|
||||
|
|
@ -958,7 +970,9 @@ def _cancel_live_retry_successor(
|
|||
intent = _claim_intent(q, intent_task_id)
|
||||
if intent.get("claim_refused"):
|
||||
return CANCEL_FAILED
|
||||
source_intent = intent or intent_before
|
||||
if not intent:
|
||||
# A2: no durable request on the logical root mints none on its retry.
|
||||
return CANCEL_NOT_FOUND
|
||||
try:
|
||||
from ouroboros.cancel_intents import (
|
||||
request_cancel,
|
||||
|
|
@ -967,9 +981,9 @@ def _cancel_live_retry_successor(
|
|||
request_cancel(
|
||||
q.DRIVE_ROOT,
|
||||
retry_task_id,
|
||||
reason=str(source_intent.get("reason") or "logical retry cancellation"),
|
||||
reason=str(intent.get("reason") or "logical retry cancellation"),
|
||||
source="timeout_retry_lineage",
|
||||
requested_by=str(source_intent.get("requested_by") or ""),
|
||||
requested_by=str(intent.get("requested_by") or ""),
|
||||
allow_settled_target=True,
|
||||
)
|
||||
except Exception:
|
||||
|
|
@ -979,31 +993,28 @@ def _cancel_live_retry_successor(
|
|||
retry_task_id,
|
||||
exc_info=True,
|
||||
)
|
||||
if intent:
|
||||
_release_intent_claim(
|
||||
q, intent_task_id,
|
||||
error="retry-lineage cancel intent write failed",
|
||||
intent=intent,
|
||||
)
|
||||
_release_intent_claim(
|
||||
q, intent_task_id,
|
||||
error="retry-lineage cancel intent write failed",
|
||||
intent=intent,
|
||||
)
|
||||
return CANCEL_FAILED
|
||||
|
||||
outcome = cancel_task_custody(retry_task_id, deliver=deliver)
|
||||
if outcome not in _CANCEL_TERMINALIZED:
|
||||
if intent:
|
||||
_release_intent_claim(
|
||||
q, intent_task_id,
|
||||
error=f"retry-lineage custody did not settle ({outcome})",
|
||||
intent=intent,
|
||||
)
|
||||
return CANCEL_FAILED
|
||||
if intent:
|
||||
_settle_intent(
|
||||
q,
|
||||
intent_task_id,
|
||||
outcome=outcome,
|
||||
detail=f"logical cancellation followed live retry {retry_task_id}",
|
||||
_release_intent_claim(
|
||||
q, intent_task_id,
|
||||
error=f"retry-lineage custody did not settle ({outcome})",
|
||||
intent=intent,
|
||||
)
|
||||
return CANCEL_FAILED
|
||||
_settle_intent(
|
||||
q,
|
||||
intent_task_id,
|
||||
outcome=outcome,
|
||||
detail=f"logical cancellation followed live retry {retry_task_id}",
|
||||
intent=intent,
|
||||
)
|
||||
return outcome
|
||||
|
||||
|
||||
|
|
@ -1419,9 +1430,11 @@ def sweep_cancel_intents(*, now: Optional[float] = None) -> Dict[str, str]:
|
|||
|
||||
def _cancel_subtree_sweep(
|
||||
q: Any, task_id: str, already: set[str], cascade_token: str = "",
|
||||
cascade_claim: Optional[Dict[str, Any]] = None,
|
||||
) -> Tuple[bool, Dict[str, str]]:
|
||||
"""One snapshot pass over the live subtree of ``task_id``: fence what it sees,
|
||||
then cancel it. Returns ``(cancelled_anything, {task_id: typed_outcome})``."""
|
||||
then cancel it under the root's ``cascade_claim``. Returns
|
||||
``(cancelled_anything, {task_id: typed_outcome})``."""
|
||||
with q._queue_lock:
|
||||
live: Dict[str, Dict[str, Any]] = {
|
||||
str(task["id"]): task
|
||||
|
|
@ -1507,10 +1520,11 @@ def _cancel_subtree_sweep(
|
|||
except Exception:
|
||||
# AR2-1 (owner batch-4 1=A): a child whose durable intent could not
|
||||
# be written is NOT silently left unfenced. Custody still runs on it
|
||||
# this sweep (killing it now beats leaving it live), but the failure
|
||||
# is surfaced — warning + typed forensic row — and the ROOT's open
|
||||
# ``scope=cascade`` intent makes the watchdog replay the WHOLE
|
||||
# cascade after a crash, re-attempting this mint.
|
||||
# this sweep under the root's ``cascade_claim`` (killing it now
|
||||
# beats leaving it live), but the failure is surfaced — warning +
|
||||
# typed forensic row — and the ROOT's open ``scope=cascade`` intent
|
||||
# makes the watchdog replay the WHOLE cascade after a crash,
|
||||
# re-attempting this mint.
|
||||
log.warning(
|
||||
"cascade descendant intent write failed for %s (child of %s); "
|
||||
"custody still runs this sweep, watchdog re-feeds via the root's "
|
||||
|
|
@ -1530,7 +1544,7 @@ def _cancel_subtree_sweep(
|
|||
# Children first (cancel_order is depth-sorted), each with its own typed
|
||||
# result — the parent's success can never mask a child's refusal.
|
||||
# deliver=False: the cascade delivers ONE root summary, not N messages.
|
||||
outcomes[live_id] = q.cancel_task_custody(live_id, deliver=False)
|
||||
outcomes[live_id] = q.cancel_task_custody(live_id, deliver=False, cascade_claim=cascade_claim)
|
||||
cancelled = outcomes[live_id] == q.CANCEL_CANCELLED or cancelled
|
||||
return cancelled, outcomes
|
||||
|
||||
|
|
@ -1571,6 +1585,7 @@ from supervisor.queue_transitions import ( # noqa: E402, F401 -- intentional pu
|
|||
_live_retry_target_locked,
|
||||
_settled_status,
|
||||
clear_acceptance_fence_for_root,
|
||||
settled_status_of,
|
||||
resume_budget_paused_task,
|
||||
resume_project_deletions,
|
||||
run_project_deletion,
|
||||
|
|
|
|||
|
|
@ -1,14 +1,17 @@
|
|||
"""Addressed result reads shared by ownership checks before queue mutation.
|
||||
|
||||
A pass reads each body once. Reuse checks its atomic-replacement stamp; a changed
|
||||
or unreadable row means unknown ownership, never permission to mutate. Locked
|
||||
callers supply the reads they prepared before acquiring the queue RLock.
|
||||
A pass reads each body once, signed by the store in the same transaction. A reuse
|
||||
re-checks that signature — UNCHANGED, CHANGED or BUSY — without waiting under the
|
||||
queue RLock: a changed or unreadable row means unknown ownership, never permission
|
||||
to mutate, and a busy store is neither a change nor proof of sameness. Locked
|
||||
callers supply the reads they prepared before acquiring the lock.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import pathlib
|
||||
import threading
|
||||
|
||||
UNCHANGED, CHANGED, BUSY = "unchanged", "changed", "busy"
|
||||
_SETTLEMENT_READS = threading.local()
|
||||
# A hard memory bound on unconsumed preparations per thread (each holds parsed result bodies);
|
||||
# a multi-occupant settlement prepares one per occupant, far below it. An evicted one is
|
||||
|
|
@ -35,14 +38,15 @@ class TaskOwnershipRead:
|
|||
if task_id in self.errors:
|
||||
raise self.errors[task_id]
|
||||
if task_id in self.rows:
|
||||
if not self.unchanged((task_id,)):
|
||||
raise ValueError('ownership result changed during this pass')
|
||||
state = self.recheck((task_id,))
|
||||
if state != UNCHANGED:
|
||||
raise ValueError(f'ownership result {state} during this pass')
|
||||
return self.rows[task_id]
|
||||
if queue._queue_lock._is_owned():
|
||||
raise RuntimeError('prepare durable ownership before taking the queue lock')
|
||||
stamp = self._stamp(task_id)
|
||||
try:
|
||||
self.rows[task_id] = load_task_result(self.root, task_id, strict=True) or {}
|
||||
row, stamp = load_task_result(self.root, task_id, strict=True, signed=True)
|
||||
self.rows[task_id] = row or {}
|
||||
if stamp is None:
|
||||
from ouroboros.owner_pause import read_fence
|
||||
read_fence(self.root, task_id) # Preserve missing/quarantined fence authority.
|
||||
|
|
@ -50,17 +54,33 @@ class TaskOwnershipRead:
|
|||
self.errors[task_id] = exc
|
||||
raise
|
||||
self.stamps[task_id] = stamp
|
||||
if not self.unchanged((task_id,)):
|
||||
raise ValueError('ownership result changed while reading')
|
||||
return self.rows[task_id]
|
||||
|
||||
def unchanged(self, task_ids):
|
||||
def prepared(self, task_id):
|
||||
"""The row as read before the queue lock, with no re-check: a task without a retry
|
||||
pointer is addressed by its physical id, and its live worker rewriting the row
|
||||
between preparation and the lock changes nothing about who owns it."""
|
||||
if task_id in self.rows:
|
||||
return self.rows[task_id]
|
||||
return self.load(task_id)
|
||||
|
||||
def recheck(self, task_ids):
|
||||
"""UNCHANGED, CHANGED or BUSY for prepared rows. Under the queue lock the signature
|
||||
probe never waits (``acquisition_budget(0)``): a busy store is unknown."""
|
||||
from contextlib import nullcontext
|
||||
from ouroboros.store import StoreBusy, acquisition_budget
|
||||
from supervisor import queue
|
||||
|
||||
budget = acquisition_budget(0) if queue._queue_lock._is_owned() else nullcontext()
|
||||
try:
|
||||
with acquisition_budget(0):
|
||||
return all(self.stamps[tid] == self._stamp(tid) for tid in task_ids)
|
||||
with budget:
|
||||
return UNCHANGED if all(self.stamps[tid] == self._stamp(tid) for tid in task_ids) else CHANGED
|
||||
except StoreBusy:
|
||||
return False # A busy signature cannot prove unchanged ownership, and never waits under Q.
|
||||
return BUSY
|
||||
|
||||
def unchanged(self, task_ids):
|
||||
"""True only for a proven-unchanged set; busy or changed cannot authorize a mutation."""
|
||||
return self.recheck(task_ids) == UNCHANGED
|
||||
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -919,7 +919,7 @@ def _settle_retry_cancel_handoff(
|
|||
if str(intent.get("scope") or "") == SCOPE_CASCADE:
|
||||
outcomes[cancel_target] = (
|
||||
q.CANCEL_CANCELLED
|
||||
if q.cancel_task_by_id(cancel_target, cascade=True)
|
||||
if q.cancel_task_by_id(cancel_target)
|
||||
else q.CANCEL_FAILED
|
||||
)
|
||||
elif sweep_owner_stop_hold(
|
||||
|
|
|
|||
27
tests/_room_shared.py
Normal file
27
tests/_room_shared.py
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
"""The room rule shaped from an explicit-roots map, for readers tested without a store.
|
||||
|
||||
``projects_registry.RoomResolver`` answers from the bindings file and the task store;
|
||||
a test that only has ``{task_id: project_chat_id}`` in hand gets the same callable
|
||||
shape here: own id, then parent, then root, as the explicit map names them.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict
|
||||
|
||||
|
||||
class RoomsFromBindings:
|
||||
def __init__(self, bindings: Dict[str, int]) -> None:
|
||||
self.bindings = dict(bindings)
|
||||
|
||||
def __call__(self, task_id: Any, parent_task_id: Any = "", root_task_id: Any = "") -> int:
|
||||
for candidate in (task_id, parent_task_id, root_task_id):
|
||||
chat = self.bindings.get(str(candidate or "").strip())
|
||||
if chat:
|
||||
return int(chat)
|
||||
return 0
|
||||
|
||||
def lineage(self, task_id: Any) -> Dict[str, Any]:
|
||||
return {}
|
||||
|
||||
def warm(self, task_ids: Any) -> None:
|
||||
return None
|
||||
10
tests/fixtures/chat_logs_ui_static_checks.json
vendored
10
tests/fixtures/chat_logs_ui_static_checks.json
vendored
|
|
@ -933,7 +933,13 @@
|
|||
"id": "test_sync_history_sweep_rederives_block_presence",
|
||||
"path": "web/modules/chat.js",
|
||||
"kind": "contains",
|
||||
"value": " for (const rec of liveCardRecords.values()) {\n reorderDirtyCardIfNeeded(rec);\n ensureLiveCardVisible(rec);\n }"
|
||||
"value": " for (const [id, record] of liveCardRecords) {\n mountReplayedCard(record);"
|
||||
},
|
||||
{
|
||||
"id": "test_sync_history_sweep_rederives_block_presence",
|
||||
"path": "web/modules/chat.js",
|
||||
"kind": "contains",
|
||||
"value": " function mountReplayedCard(rec) {\n if (rec?._anchorOrderDirty && !rec.isSubagent && rec.root?.isConnected) {\n insertMessageNode(rec.root, { reorderExisting: true });\n rec._anchorOrderDirty = false;\n }\n ensureLiveCardVisible(rec);\n }"
|
||||
},
|
||||
{
|
||||
"id": "test_sync_history_sweep_rederives_block_presence",
|
||||
|
|
@ -975,7 +981,7 @@
|
|||
"id": "test_history_replay_preserves_failed_task_summary_phase",
|
||||
"path": "web/modules/chat.js",
|
||||
"kind": "contains",
|
||||
"value": "finishLiveCard(taskId, msg.task_terminal_status ? taskTerminalPhase(msg) : replayTerminalPhase(record));"
|
||||
"value": "finishLiveCard(taskId, msg.task_terminal_status ? taskTerminalPhase(msg) : replayTerminalPhase(liveCardRecords.get(taskId)));"
|
||||
},
|
||||
{
|
||||
"id": "test_history_replay_preserves_failed_task_summary_phase",
|
||||
|
|
|
|||
|
|
@ -21,8 +21,9 @@ the same WS chat and HTTP cancel surfaces the SPA drives.
|
|||
DETERMINISM. "Mid-turn" is not a sleep. The loopback model HOLDS the turn's first
|
||||
tool-bearing round on a ``ModelGate`` (harness): the request thread blocks until the
|
||||
scenario releases it, so the owner's stop provably lands while the turn is inside
|
||||
a model round, the endpoint's typed "still live" is the ONLY answer it can give
|
||||
(a turn cannot reach its round boundary while its round is held), and the armed
|
||||
a model round, the endpoint's typed 202 ``cancel_state: pending`` is the ONLY
|
||||
answer it can give (a turn cannot reach its round boundary while its round is
|
||||
held), and the armed
|
||||
control, the toast and the marker are all read back DURABLY before the release.
|
||||
Everything else is durable-event polling over ``ArtifactOracle`` readers.
|
||||
|
||||
|
|
@ -35,16 +36,19 @@ WHAT S26 ASSERTS, in order:
|
|||
``_is_direct_chat=true`` and the client_message_id link (``task_started`` is a
|
||||
session-only live log, never persisted);
|
||||
2. STOP-NOW over the UI's ``POST /api/tasks/{id}/cancel`` (empty body = immediate)
|
||||
mid-round is NOT 404: the typed 503 "still live" naming the task; the
|
||||
cooperative control is ARMED — exactly one ``finalize_now`` row with the
|
||||
mid-round is NOT 404: the intent is recorded first and the teardown is not yet
|
||||
confirmed, so the answer is the typed 202 ``cancel_state: pending``
|
||||
(``reason_code=teardown_unconfirmed``) naming the task and its durable
|
||||
request; the cooperative control is ARMED — exactly one ``finalize_now`` row with the
|
||||
direct-turn reason in the canonical mailbox; the durable intent is open with
|
||||
the forensic trail requested(http_single, single) → claimed → claim_released
|
||||
("has not reached its next step"); the owner toast went out as a progress frame
|
||||
carrying the host-attested ``cancelable`` marker (the direct-turn branch of the
|
||||
delivery seam) — durable in progress.jsonl and live on /ws; the durable row is
|
||||
still honestly ``running`` (no fabricated cancelled over a live turn);
|
||||
3. a SECOND stop-now while armed is typed and idempotent — 503 again, still ONE
|
||||
control, still ONE toast, no model round in between (stamp latch, no re-arm);
|
||||
3. a SECOND stop-now while armed is typed and idempotent — 202 pending again on
|
||||
the same request, still ONE control, still ONE toast, no model round in
|
||||
between (stamp latch, no re-arm);
|
||||
4. RELEASE — the turn ends at its next step under the owner-stop reason with ZERO
|
||||
further model rounds: the durable terminal carries
|
||||
``reason_code=owner_requested_finalization`` and the typed no-further-work
|
||||
|
|
@ -182,7 +186,6 @@ S26_PROBE = (f"[{S26_MARKER}] List the repository root and keep listing it "
|
|||
# loop_round_limits._handle_direct_turn_hard_stop's typed no-candidate answer).
|
||||
S26_TOAST_TEXT = "The owner stopped chat turn"
|
||||
S26_STOP_TEXT = "The owner stopped this chat turn"
|
||||
S26_STILL_LIVE = "still live"
|
||||
S26_NOT_ACTIVE = "task not found or not active"
|
||||
|
||||
|
||||
|
|
@ -354,11 +357,14 @@ def test_s26_direct_chat_turn_owner_stop_mid_round_is_typed_and_ends_the_turn(
|
|||
"client_message_id") == client_message_id, received_task
|
||||
|
||||
# 2. STOP-NOW over the UI's endpoint, mid-round: typed, never 404,
|
||||
# the cooperative control ARMED.
|
||||
# the cooperative control ARMED. The intent is recorded first and
|
||||
# the teardown is not yet confirmed: 202 ``cancel_state: pending``
|
||||
# (``teardown_unconfirmed``), never a fabricated 200.
|
||||
first = server.cancel_task(task_id)
|
||||
assert first.get("status") == 503, first
|
||||
assert first.get("status") == 202, first
|
||||
first_body = first.get("body") or {}
|
||||
assert S26_STILL_LIVE in str(first_body.get("error") or ""), first
|
||||
assert first_body.get("cancel_state") == "pending", first
|
||||
assert first_body.get("reason_code") == "teardown_unconfirmed", first
|
||||
assert first_body.get("task_id") == task_id, first
|
||||
controls = _mailbox_controls(server.data_root, task_id)
|
||||
assert len(controls) == 1, controls
|
||||
|
|
@ -369,6 +375,7 @@ def test_s26_direct_chat_turn_owner_stop_mid_round_is_typed_and_ends_the_turn(
|
|||
# ``requested`` for the sweep with the typed "still live" error.
|
||||
intent = oracle.cancel_intents().get(task_id) or {}
|
||||
assert intent, oracle.cancel_intents()
|
||||
assert first_body.get("request_id") == intent.get("request_id"), (first_body, intent)
|
||||
assert stop_policy(intent) == STOP_POLICY_IMMEDIATE, intent
|
||||
assert intent.get("state") == INTENT_REQUESTED, intent
|
||||
assert "not reached its next step" in str(intent.get("last_error") or ""), intent
|
||||
|
|
@ -395,10 +402,13 @@ def test_s26_direct_chat_turn_owner_stop_mid_round_is_typed_and_ends_the_turn(
|
|||
# No fabricated terminal over a live turn.
|
||||
assert oracle.task_result(task_id).get("status") == "running", oracle.task_result(task_id)
|
||||
|
||||
# 3. A SECOND stop while armed: typed and idempotent.
|
||||
# 3. A SECOND stop while armed: typed and idempotent — the same
|
||||
# pending envelope on the same durable request.
|
||||
second = server.cancel_task(task_id)
|
||||
assert second.get("status") == 503, second
|
||||
assert S26_STILL_LIVE in str((second.get("body") or {}).get("error") or ""), second
|
||||
assert second.get("status") == 202, second
|
||||
second_body = second.get("body") or {}
|
||||
assert second_body.get("cancel_state") == "pending", second
|
||||
assert second_body.get("request_id") == intent.get("request_id"), (second_body, intent)
|
||||
assert len(_mailbox_controls(server.data_root, task_id)) == 1, "the stop was re-armed"
|
||||
assert len(_toast_rows(oracle, task_id)) == 1, "the owner was toasted twice"
|
||||
assert gate.matched == 1 and gate.held == 1, (gate.matched, gate.held)
|
||||
|
|
|
|||
|
|
@ -21,6 +21,7 @@ from ouroboros.task_status import load_effective_task_result
|
|||
from supervisor.events_worker_reports import _handle_log_event
|
||||
from supervisor.worker_process import WORKER_LOG_SINK_SUPPRESSED_TYPES
|
||||
from tests.test_acceptance_publication import _run
|
||||
from tests._room_shared import RoomsFromBindings
|
||||
|
||||
pytestmark = pytest.mark.serial
|
||||
|
||||
|
|
@ -106,14 +107,14 @@ def test_terminal_pipeline_routes_review_and_downloads_without_bookkeeping_deliv
|
|||
chats = {project_chat} if project_bound else set()
|
||||
main_rows, _ = _collect_progress_rows(
|
||||
tmp_path / "logs" / "progress.jsonl", tmp_path / "logs" / "archive", 20,
|
||||
_make_thread_filter(1, chats, [], bindings),
|
||||
_make_thread_filter(1, chats, [], RoomsFromBindings(bindings)),
|
||||
)
|
||||
main_references = [row for row in main_rows if row.get("system_type") == "review_reference"]
|
||||
assert bool(main_references) is (chat_id == 1 and not project_bound)
|
||||
if project_bound:
|
||||
project_rows, _ = _collect_progress_rows(
|
||||
tmp_path / "logs" / "progress.jsonl", tmp_path / "logs" / "archive", 20,
|
||||
_make_thread_filter(project_chat, chats, [], bindings),
|
||||
_make_thread_filter(project_chat, chats, [], RoomsFromBindings(bindings)),
|
||||
)
|
||||
assert any(row.get("system_type") == "review_reference" for row in project_rows)
|
||||
|
||||
|
|
|
|||
|
|
@ -516,7 +516,8 @@ def test_terminal_http_stop_reaches_the_live_historical_operation(late, tmp_path
|
|||
app.state.drive_root = f.root
|
||||
with TestClient(app) as client:
|
||||
response = client.post(f'/api/tasks/{target}/cancel', json={})
|
||||
assert response.status_code == 503, response.text # still owned, never a false 404 or completed cancellation
|
||||
assert response.status_code == 202, response.text # still owned, never a false 404 or completed cancellation
|
||||
assert response.json()['cancel_state'] == 'pending'
|
||||
assert cancel_pending(f.root, ctx.task_id if stopped == 'caller' else f.tid, strict=True)
|
||||
operation = next(op for op in review_operation._LIVE.values() if op.task_id == f.tid)
|
||||
until(lambda: operation.control() == 'cancelled')
|
||||
|
|
|
|||
|
|
@ -579,6 +579,7 @@ _TRUNCATION_DECISIONS: dict[str, tuple[bool, str]] = {
|
|||
"artifact_relpath_invalid": (False, "gateway/tasks.py: invalid exact artifact selector; HTTP refusal, not a task terminal"),
|
||||
"invalid_action_nonce": (False, "task_cancel rejects malformed action identity; no task transition"),
|
||||
"stop_action_conflict": (False, "task_cancel refuses conflicting action replay; no task transition"),
|
||||
"teardown_unconfirmed": (False, "gateway/tasks.py: an HTTP 202 Stop acknowledgement; the recorded intent is still being finished, never a task outcome"),
|
||||
"artifact_unavailable": (False, "gateway/task_archive.py: confined single-file read unavailable; HTTP refusal, not a task terminal"),
|
||||
"artifact_unverified": (False, "gateway/task_archive.py: single-file drift or capture verification failure; HTTP refusal, not a task terminal"),
|
||||
"upload_unavailable": (False, "gateway/files.py: a chat attachment is missing or not a regular file; HTTP 404, not a task terminal"),
|
||||
|
|
|
|||
|
|
@ -138,7 +138,7 @@ def test_cascade_over_a_settled_root_with_live_children_still_delivers(qenv, mon
|
|||
]
|
||||
write_task_result(qenv.drive, "kid1", "scheduled")
|
||||
|
||||
assert qenv.tl.cancel_task_by_id("root-dead", cascade=True) is True
|
||||
assert qenv.tl.cancel_task_by_id("root-dead") is True
|
||||
|
||||
assert delivered, "a settled root with live children must still report to chat"
|
||||
(row,) = delivered
|
||||
|
|
@ -158,7 +158,7 @@ def test_cascade_mints_child_intents_and_records_scope(qenv, monkeypatch):
|
|||
]
|
||||
ci.request_cancel(qenv.drive, "c-root", reason="stop the tree")
|
||||
|
||||
assert qenv.tl.cancel_task_by_id("c-root", cascade=True) is False # custody refused
|
||||
assert qenv.tl.cancel_task_by_id("c-root") is False # custody refused
|
||||
|
||||
intents = ci.active_intents(qenv.drive)
|
||||
assert "c-kid" in intents, "every captured descendant carries its own intent"
|
||||
|
|
@ -183,7 +183,7 @@ def test_watchdog_replays_a_cascade_intent_as_a_cascade(qenv, monkeypatch):
|
|||
|
||||
qenv.tl.sweep_cancel_intents(now=1_000_000.0)
|
||||
|
||||
assert calls == [("casc-root", {"cascade": True})]
|
||||
assert calls == [("casc-root", {})]
|
||||
|
||||
@pytest.mark.serial
|
||||
def test_unreconciled_delegated_runs_are_disclosed_on_the_cancelled_result(
|
||||
|
|
@ -336,7 +336,7 @@ def test_cascade_descendant_intent_failure_is_surfaced_not_silent(qenv, monkeypa
|
|||
]
|
||||
real_request(qenv.drive, "d-root", reason="stop the tree")
|
||||
|
||||
assert qenv.tl.cancel_task_by_id("d-root", cascade=True) is True
|
||||
assert qenv.tl.cancel_task_by_id("d-root") is True
|
||||
assert "d-kid" in calls, "custody still runs on the child this sweep"
|
||||
trail = (qenv.drive / "logs" / "supervisor.jsonl").read_text(encoding="utf-8")
|
||||
assert "cascade_descendant_intent_write_failed" in trail
|
||||
|
|
@ -366,7 +366,7 @@ def test_cascade_over_settled_root_keeps_the_intent_until_the_postcondition(qenv
|
|||
qenv.q, "cancel_task_custody",
|
||||
lambda tid, **kw: qenv.q.CANCEL_FAILED if tid == "sr1-kid" else real_custody(tid, **kw),
|
||||
)
|
||||
assert qenv.tl.cancel_task_by_id("sr1", cascade=True) is False
|
||||
assert qenv.tl.cancel_task_by_id("sr1") is False
|
||||
row = ci.active_intent(qenv.drive, "sr1")
|
||||
assert row is not None and row["scope"] == ci.SCOPE_CASCADE, (
|
||||
"the durable cascade intent must survive a failed sweep — it is the "
|
||||
|
|
@ -375,7 +375,7 @@ def test_cascade_over_settled_root_keeps_the_intent_until_the_postcondition(qenv
|
|||
|
||||
# The watchdog replay converges: custody works now, postcondition settles it.
|
||||
monkeypatch.setattr(qenv.q, "cancel_task_custody", real_custody)
|
||||
assert qenv.tl.cancel_task_by_id("sr1", cascade=True) is True
|
||||
assert qenv.tl.cancel_task_by_id("sr1") is True
|
||||
assert ci.active_intent(qenv.drive, "sr1") is None
|
||||
assert load_task_result(qenv.drive, "sr1-kid")["status"] == STATUS_CANCELLED
|
||||
assert delivered, "the tree's summary still reaches chat"
|
||||
|
|
|
|||
|
|
@ -19,6 +19,14 @@ def _isolate_queue(monkeypatch, tmp_path, tasks):
|
|||
return queue, pending
|
||||
|
||||
|
||||
def _request(drive, task_id, *, cascade=False):
|
||||
"""Every cancel ingress records the durable intent first (invariant 14); a cascade
|
||||
records it on the root with the cascade scope."""
|
||||
from ouroboros.cancel_intents import SCOPE_CASCADE, request_cancel
|
||||
|
||||
request_cancel(drive, task_id, scope=SCOPE_CASCADE if cascade else "", allow_settled_target=True)
|
||||
|
||||
|
||||
def test_root_cancel_snapshots_and_cancels_whole_live_subtree(tmp_path, monkeypatch):
|
||||
from ouroboros.task_results import STATUS_CANCELLED, load_task_result
|
||||
|
||||
|
|
@ -47,7 +55,8 @@ def test_root_cancel_snapshots_and_cancels_whole_live_subtree(tmp_path, monkeypa
|
|||
order: list[str] = []
|
||||
monkeypatch.setattr(queue, "_emit_cancel_task_done", lambda _task, task_id, **_kw: order.append(task_id))
|
||||
|
||||
assert queue.cancel_task_by_id("root", cascade=True) is True
|
||||
_request(tmp_path, "root", cascade=True)
|
||||
assert queue.cancel_task_by_id("root") is True
|
||||
|
||||
assert order == ["grandchild", "child", "root"]
|
||||
assert [task["id"] for task in pending] == ["other"]
|
||||
|
|
@ -71,11 +80,13 @@ def test_midtree_cancel_uses_combined_pending_lineage(tmp_path, monkeypatch):
|
|||
],
|
||||
)
|
||||
|
||||
assert queue.cancel_task_by_id("child", cascade=True) is True
|
||||
_request(tmp_path, "child", cascade=True)
|
||||
assert queue.cancel_task_by_id("child") is True
|
||||
assert [task["id"] for task in pending] == ["root"]
|
||||
|
||||
|
||||
def test_default_cancel_preserves_live_children(tmp_path, monkeypatch):
|
||||
def test_single_cancel_preserves_live_children(tmp_path, monkeypatch):
|
||||
"""A single-scope intent drives one custody; the subtree is a cascade's business."""
|
||||
queue, pending = _isolate_queue(
|
||||
monkeypatch,
|
||||
tmp_path,
|
||||
|
|
@ -85,7 +96,8 @@ def test_default_cancel_preserves_live_children(tmp_path, monkeypatch):
|
|||
],
|
||||
)
|
||||
|
||||
assert queue.cancel_task_by_id("root") is True
|
||||
_request(tmp_path, "root")
|
||||
assert queue.drive_cancel_intent_scope("root") == queue.CANCEL_CANCELLED
|
||||
assert [task["id"] for task in pending] == ["child"]
|
||||
|
||||
|
||||
|
|
@ -116,7 +128,7 @@ def test_cascade_resweeps_descendants_admitted_after_the_first_snapshot(monkeypa
|
|||
monkeypatch.setattr(q, "cancel_task_custody", lambda tid, **_kw: q.CANCEL_CANCELLED if _fake_single(tid) else q.CANCEL_FAILED)
|
||||
monkeypatch.setattr(q, "append_jsonl", lambda *a, **k: None)
|
||||
|
||||
assert cancel_task_by_id("root", cascade=True) is True
|
||||
assert cancel_task_by_id("root") is True
|
||||
assert "late-child" in cancelled, cancelled
|
||||
assert cancelled.count("root") == 1, cancelled
|
||||
|
||||
|
|
@ -144,7 +156,7 @@ def test_cancelled_root_fences_late_descendant_admission(monkeypatch):
|
|||
monkeypatch.setattr(q, "cancel_task_custody", _custody)
|
||||
monkeypatch.setattr(q, "append_jsonl", lambda *a, **k: None)
|
||||
|
||||
assert cancel_task_by_id("root", cascade=True) is True
|
||||
assert cancel_task_by_id("root") is True
|
||||
# A schedule event that drained AFTER the whole sweep loop is refused.
|
||||
admitted = q.enqueue_task({"id": "late", "parent_task_id": "root", "root_task_id": "root"})
|
||||
assert admitted.get("_admission_blocked") == "root_cancelled"
|
||||
|
|
@ -181,7 +193,7 @@ def test_mid_tree_cancel_fences_a_grandchild_scheduled_by_a_live_descendant(monk
|
|||
monkeypatch.setattr(q, "cancel_task_custody", lambda tid, **_kw: q.CANCEL_CANCELLED if _remove_from_live(tid) else q.CANCEL_FAILED)
|
||||
monkeypatch.setattr(q, "append_jsonl", lambda *a, **k: None)
|
||||
|
||||
assert cancel_task_by_id("mid", cascade=True) is True
|
||||
assert cancel_task_by_id("mid") is True
|
||||
assert "deep" not in q.RUNNING # the cancelled subtree really is gone
|
||||
|
||||
admitted = q.enqueue_task(
|
||||
|
|
@ -223,7 +235,7 @@ def test_a_500_child_cascade_never_evicts_its_own_fence(monkeypatch):
|
|||
monkeypatch.setattr(q, "cancel_task_custody", lambda tid, **_kw: q.CANCEL_CANCELLED if _remove(tid) else q.CANCEL_FAILED)
|
||||
monkeypatch.setattr(q, "append_jsonl", lambda *a, **k: None)
|
||||
|
||||
assert cancel_task_by_id("root", cascade=True) is True
|
||||
assert cancel_task_by_id("root") is True
|
||||
assert "root" in tl.CANCELLED_ROOT_FENCES
|
||||
assert "c499" in tl.CANCELLED_ROOT_FENCES
|
||||
# A schedule event draining after the final sweep is still refused.
|
||||
|
|
@ -274,7 +286,8 @@ def test_completion_wins_over_a_cascade_that_is_already_tearing_down(monkeypatch
|
|||
|
||||
write_task_result(tmp_path, "root", "completed", result="Finished on its own.")
|
||||
|
||||
queue.cancel_task_by_id("root", cascade=True)
|
||||
_request(tmp_path, "root", cascade=True)
|
||||
queue.cancel_task_by_id("root")
|
||||
|
||||
assert load_task_result(tmp_path, "root")["status"] == "completed"
|
||||
assert load_task_result(tmp_path, "child")["status"] == "cancelled"
|
||||
|
|
@ -329,7 +342,8 @@ def test_a_child_that_refuses_to_die_fails_the_whole_cascade(monkeypatch, tmp_pa
|
|||
|
||||
monkeypatch.setattr(q, "cancel_task_custody", _custody)
|
||||
|
||||
assert tl.cancel_task_by_id("root", cascade=True) is False
|
||||
_request(tmp_path, "root", cascade=True)
|
||||
assert tl.cancel_task_by_id("root") is False
|
||||
assert any(str(t.get("id")) == "child" for t in q.PENDING), "the child really is still live"
|
||||
|
||||
|
||||
|
|
@ -346,6 +360,7 @@ def test_a_stubborn_process_is_refused_not_reported_cancelled(monkeypatch, tmp_p
|
|||
monkeypatch.setattr("ouroboros.platform_layer.kill_pid_tree", lambda *a, **k: None)
|
||||
emitted: list[str] = []
|
||||
monkeypatch.setattr(q, "_emit_cancel_task_done", lambda *a, **k: emitted.append("emit"))
|
||||
_request(tmp_path, "stuck")
|
||||
|
||||
assert q.cancel_task_custody("stuck") == q.CANCEL_FAILED
|
||||
assert "stuck" in q.RUNNING, "custody is restored so the tree stays honest"
|
||||
|
|
@ -362,6 +377,7 @@ def test_persistence_failure_restores_custody_and_publishes_nothing(monkeypatch,
|
|||
_isolate_queue(monkeypatch, tmp_path, [{"id": "root", "chat_id": 0, "root_task_id": "root"}])
|
||||
emitted: list[str] = []
|
||||
monkeypatch.setattr(q, "_emit_cancel_task_done", lambda *a, **k: emitted.append("emit"))
|
||||
_request(tmp_path, "root")
|
||||
|
||||
def _boom(*a, **k):
|
||||
raise OSError("disk full")
|
||||
|
|
@ -403,6 +419,8 @@ def test_concurrent_cascades_on_overlapping_trees_both_settle(monkeypatch, tmp_p
|
|||
{"id": "leaf", "chat_id": 0, "root_task_id": "root", "parent_task_id": "mid"},
|
||||
])
|
||||
monkeypatch.setattr(tl, "CANCELLED_ROOT_FENCES", {}, raising=False)
|
||||
_request(tmp_path, "root", cascade=True)
|
||||
_request(tmp_path, "mid", cascade=True)
|
||||
|
||||
def _custody(tid, **_kw):
|
||||
# Mirrors the real contract: custody is the ONE settle owner, so the
|
||||
|
|
@ -425,8 +443,8 @@ def test_concurrent_cascades_on_overlapping_trees_both_settle(monkeypatch, tmp_p
|
|||
|
||||
results: dict[str, bool] = {}
|
||||
threads = [
|
||||
threading.Thread(target=lambda: results.__setitem__("root", tl.cancel_task_by_id("root", cascade=True))),
|
||||
threading.Thread(target=lambda: results.__setitem__("mid", tl.cancel_task_by_id("mid", cascade=True))),
|
||||
threading.Thread(target=lambda: results.__setitem__("root", tl.cancel_task_by_id("root"))),
|
||||
threading.Thread(target=lambda: results.__setitem__("mid", tl.cancel_task_by_id("mid"))),
|
||||
]
|
||||
for t in threads:
|
||||
t.start()
|
||||
|
|
@ -457,6 +475,7 @@ def test_a_settled_task_with_a_live_worker_is_killed_and_keeps_its_result(monkey
|
|||
emitted: list[str] = []
|
||||
monkeypatch.setattr(q, "_emit_cancel_task_done", lambda *a, **k: emitted.append("emit"))
|
||||
write_task_result(tmp_path, "done-ish", "completed", result="Finished on its own.")
|
||||
_request(tmp_path, "done-ish")
|
||||
|
||||
assert q.cancel_task_custody("done-ish") == q.CANCEL_ALREADY_SETTLED
|
||||
assert not state["alive"], "the settled-but-live worker IS killed (GR6-1b)"
|
||||
|
|
@ -490,6 +509,7 @@ def test_cancelling_a_running_task_completes_the_whole_publish_phase(monkeypatch
|
|||
)
|
||||
emitted: list[str] = []
|
||||
monkeypatch.setattr(q, "_emit_cancel_task_done", lambda *a, **k: emitted.append("emit"))
|
||||
_request(tmp_path, "live-one")
|
||||
|
||||
assert q.cancel_task_custody("live-one") == q.CANCEL_CANCELLED
|
||||
assert not state["alive"], "the process is confirmed dead before publishing"
|
||||
|
|
@ -523,6 +543,7 @@ def test_custody_marks_the_slot_so_the_health_check_keeps_its_hands_off(monkeypa
|
|||
monkeypatch.setattr("ouroboros.platform_layer.kill_pid_tree", _kill)
|
||||
respawned: list[int] = []
|
||||
monkeypatch.setattr(workers, "respawn_worker", lambda wid: respawned.append(wid), raising=False)
|
||||
_request(tmp_path, "live-two")
|
||||
|
||||
assert q.cancel_task_custody("live-two") == q.CANCEL_CANCELLED
|
||||
assert seen_reaping == [True], "the slot is marked BEFORE the lock is released"
|
||||
|
|
@ -543,6 +564,7 @@ def test_a_raising_kill_restores_custody_and_clears_the_slot_marker(monkeypatch,
|
|||
raise OSError("kill failed")
|
||||
|
||||
monkeypatch.setattr("ouroboros.platform_layer.kill_pid_tree", _boom)
|
||||
_request(tmp_path, "boom-one")
|
||||
|
||||
assert q.cancel_task_custody("boom-one") == q.CANCEL_FAILED
|
||||
assert "boom-one" in q.RUNNING, "the task is visible to the liveness check again"
|
||||
|
|
@ -619,6 +641,7 @@ def test_a_natural_result_written_before_the_kill_still_resolves_the_card(monkey
|
|||
state["alive"] = False
|
||||
|
||||
monkeypatch.setattr("ouroboros.platform_layer.kill_pid_tree", _kill)
|
||||
_request(tmp_path, "racer")
|
||||
|
||||
assert q.cancel_task_custody("racer") == q.CANCEL_ALREADY_SETTLED
|
||||
assert load_task_result(tmp_path, "racer")["status"] == "completed"
|
||||
|
|
@ -654,6 +677,7 @@ def test_split_drive_completion_is_promoted_before_cancelled_write(monkeypatch,
|
|||
lambda _task, task_id, **kw: emitted.append({"task_id": task_id, **kw}),
|
||||
)
|
||||
|
||||
_request(tmp_path, "split-racer")
|
||||
assert q.cancel_task_custody("split-racer") == q.CANCEL_ALREADY_SETTLED
|
||||
stored = load_task_result(tmp_path, "split-racer")
|
||||
assert stored["status"] == "completed"
|
||||
|
|
@ -686,6 +710,7 @@ def test_publication_failures_after_the_durable_write_never_undo_the_cancellatio
|
|||
monkeypatch.setattr(q, "_emit_cancel_task_done", _boom)
|
||||
monkeypatch.setattr(workers, "respawn_worker", _boom, raising=False)
|
||||
monkeypatch.setattr(q, "persist_queue_snapshot", _boom, raising=False)
|
||||
_request(tmp_path, "post-boundary")
|
||||
|
||||
assert q.cancel_task_custody("post-boundary") == q.CANCEL_CANCELLED
|
||||
assert load_task_result(tmp_path, "post-boundary")["status"] == "cancelled"
|
||||
|
|
@ -716,7 +741,7 @@ def test_a_cascade_settles_when_a_child_is_terminal_but_still_winding_down(monke
|
|||
|
||||
monkeypatch.setattr(q, "cancel_task_custody", _custody)
|
||||
|
||||
assert tl.cancel_task_by_id("root", cascade=True) is True, (
|
||||
assert tl.cancel_task_by_id("root") is True, (
|
||||
"a terminal-but-unreclaimed child is settled, not live"
|
||||
)
|
||||
|
||||
|
|
@ -750,6 +775,8 @@ def test_overlapping_cascade_cannot_confirm_success_while_the_child_kill_is_in_f
|
|||
"child": {"task": {"id": "child", "root_task_id": "root", "parent_task_id": "root"}},
|
||||
}, raising=False)
|
||||
|
||||
_request(tmp_path, "root", cascade=True)
|
||||
_request(tmp_path, "child")
|
||||
outcome = {}
|
||||
child_thread = threading.Thread(
|
||||
target=lambda: outcome.__setitem__("child", q.cancel_task_custody("child")),
|
||||
|
|
@ -757,7 +784,7 @@ def test_overlapping_cascade_cannot_confirm_success_while_the_child_kill_is_in_f
|
|||
child_thread.start()
|
||||
assert kill_started.wait(timeout=5)
|
||||
try:
|
||||
assert tl.cancel_task_by_id("root", cascade=True) is False, (
|
||||
assert tl.cancel_task_by_id("root") is False, (
|
||||
"no success while the child's kill is in flight"
|
||||
)
|
||||
assert tl.task_subtree_is_live("root") is True, "the in-flight child stays visible"
|
||||
|
|
@ -765,7 +792,7 @@ def test_overlapping_cascade_cannot_confirm_success_while_the_child_kill_is_in_f
|
|||
release_kill.set()
|
||||
child_thread.join(timeout=10)
|
||||
assert outcome.get("child") == q.CANCEL_CANCELLED
|
||||
assert tl.cancel_task_by_id("root", cascade=True) is True, "settled after confirmed death"
|
||||
assert tl.cancel_task_by_id("root") is True, "settled after confirmed death"
|
||||
|
||||
|
||||
def test_a_natural_task_done_in_the_kill_window_keeps_one_owner(monkeypatch, tmp_path):
|
||||
|
|
|
|||
|
|
@ -806,15 +806,22 @@ def test_custody_refuses_when_the_claim_cannot_be_read(qenv, monkeypatch):
|
|||
assert [t["id"] for t in qenv.q.PENDING] == ["claim-io"], "capture returned"
|
||||
assert load_task_result(qenv.drive, "claim-io")["status"] == "scheduled"
|
||||
|
||||
def test_custody_without_any_intent_is_the_documented_legacy_path(qenv, monkeypatch):
|
||||
"""AR2-2: claim → None (no active intent) is the legacy/no-intent path —
|
||||
capture under the queue lock is the mutual exclusion and custody proceeds."""
|
||||
def test_custody_without_any_intent_is_a_typed_noop(qenv, monkeypatch):
|
||||
"""A2 (invariant 14): claim → None (no active intent) is NO authority. Custody
|
||||
captures nothing, kills nothing and writes nothing; the answer is typed from the
|
||||
stored result (``not_found`` for a live task, ``already_settled`` for a settled one)."""
|
||||
qenv.q.PENDING[:] = [{"id": "no-intent", "chat_id": 1}]
|
||||
write_task_result(qenv.drive, "no-intent", "scheduled")
|
||||
monkeypatch.setattr(qenv.q, "_emit_cancel_task_done", lambda *_a, **_kw: None)
|
||||
emitted: list = []
|
||||
monkeypatch.setattr(qenv.q, "_emit_cancel_task_done", lambda *_a, **_kw: emitted.append(_a))
|
||||
|
||||
assert qenv.tl.cancel_task_custody("no-intent") == qenv.tl.CANCEL_CANCELLED
|
||||
assert load_task_result(qenv.drive, "no-intent")["status"] == STATUS_CANCELLED
|
||||
assert qenv.tl.cancel_task_custody("no-intent") == qenv.tl.CANCEL_NOT_FOUND
|
||||
assert [t["id"] for t in qenv.q.PENDING] == ["no-intent"]
|
||||
assert load_task_result(qenv.drive, "no-intent")["status"] == "scheduled"
|
||||
assert emitted == []
|
||||
write_task_result(qenv.drive, "no-intent", "completed")
|
||||
assert qenv.tl.cancel_task_custody("no-intent") == qenv.tl.CANCEL_ALREADY_SETTLED
|
||||
assert [t["id"] for t in qenv.q.PENDING] == ["no-intent"]
|
||||
|
||||
def test_two_concurrent_custodies_on_a_pending_task_settle_exactly_once(qenv, monkeypatch):
|
||||
"""GR2-2 (sol's repro shape): two threads racing custody over one pending
|
||||
|
|
@ -884,31 +891,6 @@ def test_double_takeover_loser_restores_the_reaping_marker_as_found(qenv, monkey
|
|||
"the loser must restore the marker as found — the winner is mid-kill behind it"
|
||||
)
|
||||
|
||||
def test_task_lifecycle_keeps_capture_miss_calling_convention(monkeypatch):
|
||||
from supervisor import cancel_publication, task_lifecycle
|
||||
|
||||
queue_sentinel = object()
|
||||
intent = {"request_id": "compat-request"}
|
||||
seen = {}
|
||||
|
||||
def fake_finalize(q, task_id, *, intent=None):
|
||||
seen.update(q=q, task_id=task_id, intent=intent)
|
||||
return "compat-result"
|
||||
|
||||
monkeypatch.setattr(task_lifecycle, "_queue_module", lambda: queue_sentinel)
|
||||
monkeypatch.setattr(cancel_publication, "_finalize_cancel_intent_on_miss", fake_finalize)
|
||||
|
||||
assert (
|
||||
task_lifecycle._finalize_cancel_intent_on_miss("compat-task", intent=intent)
|
||||
== "compat-result"
|
||||
)
|
||||
assert seen == {
|
||||
"q": queue_sentinel,
|
||||
"task_id": "compat-task",
|
||||
"intent": intent,
|
||||
}
|
||||
|
||||
|
||||
def _reaper_kill_audit(runs: list) -> dict:
|
||||
return {
|
||||
"task_id": "audited", "trigger": "reaper_idle_timeout", "outcomes": [],
|
||||
|
|
|
|||
425
tests/test_cancel_intent_authority.py
Normal file
425
tests/test_cancel_intent_authority.py
Normal file
|
|
@ -0,0 +1,425 @@
|
|||
"""Cancellation authority is the durable intent, and only the intent (invariant 14).
|
||||
|
||||
Phantom cancelled-subagent card in Main (14.10.2026): a ``cancel_task`` event drained
|
||||
51 minutes late ran custody with no open intent, a busy ownership read was reported as
|
||||
"changed", and the resulting "❌ cancel … did not settle" row named a child without
|
||||
lineage in the owner's chat. Three owner decisions close the class:
|
||||
|
||||
- A1: the event is a silent wake of ``drive_cancel_intent_scope``; no intent is a typed
|
||||
no-op that touches nothing.
|
||||
- A2: ``cancel_task_custody`` refuses without a claimed intent or an explicit root-cascade
|
||||
claim; the dead no-intent facade is gone.
|
||||
- A3: the ownership read is one signed transaction outside the queue lock; under the lock
|
||||
the re-check is non-blocking and tri-state, and the retry chain is walked only when the
|
||||
task has ``superseded_by``.
|
||||
|
||||
Every test here fails with the corresponding change reverted and keeps the still-working
|
||||
case beside it.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from ouroboros import cancel_intents as ci
|
||||
from ouroboros.task_results import STATUS_CANCELLED, load_task_result, write_task_result
|
||||
from tests._cancel_intents_shared import qenv as _qenv
|
||||
|
||||
qenv = _qenv
|
||||
|
||||
|
||||
def _event_ctx(sent):
|
||||
return SimpleNamespace(load_state=lambda: {"owner_chat_id": 1},
|
||||
send_with_budget=lambda *a, **k: sent.append((a, k)))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- A1 -------------------------
|
||||
|
||||
def test_a1_cancel_event_without_open_intent_never_tears_down(qenv, monkeypatch):
|
||||
"""A stale wake for a live task that nobody asked to cancel: nothing happens, nothing is said."""
|
||||
from supervisor.events_runtime_controls import _drive_cancel_task_event
|
||||
|
||||
qenv.q.PENDING[:] = [{"id": "live-no-intent", "chat_id": 1}]
|
||||
monkeypatch.setattr(qenv.q, "_emit_cancel_task_done", lambda *_a, **_k: None)
|
||||
write_task_result(qenv.drive, "live-no-intent", "scheduled")
|
||||
sent = []
|
||||
_drive_cancel_task_event({"task_id": "live-no-intent"}, _event_ctx(sent))
|
||||
assert [t["id"] for t in qenv.q.PENDING] == ["live-no-intent"]
|
||||
assert load_task_result(qenv.drive, "live-no-intent")["status"] == "scheduled"
|
||||
assert sent == []
|
||||
|
||||
|
||||
def test_a1_wake_with_a_closed_intent_is_typed_and_touches_the_queue_not_at_all(qenv, monkeypatch):
|
||||
"""The incident shape: the watchdog settled the intent an hour ago; the event arrives now.
|
||||
``already_settled`` is the answer; a queued row for that id is NOT scrubbed here (custody
|
||||
owns ghost rows, under its own claim)."""
|
||||
custody_calls = []
|
||||
monkeypatch.setattr(qenv.q, "cancel_task_custody",
|
||||
lambda tid, **kw: custody_calls.append(tid) or qenv.tl.CANCEL_CANCELLED)
|
||||
write_task_result(qenv.drive, "settled-child", STATUS_CANCELLED)
|
||||
qenv.q.PENDING[:] = [{"id": "settled-child", "chat_id": 1}]
|
||||
|
||||
assert qenv.tl.drive_cancel_intent_scope("settled-child") == qenv.tl.CANCEL_ALREADY_SETTLED
|
||||
assert [t["id"] for t in qenv.q.PENDING] == ["settled-child"]
|
||||
assert custody_calls == []
|
||||
|
||||
write_task_result(qenv.drive, "live-child", "scheduled")
|
||||
qenv.q.PENDING.append({"id": "live-child", "chat_id": 1})
|
||||
assert qenv.tl.drive_cancel_intent_scope("live-child") == qenv.tl.CANCEL_NOT_FOUND
|
||||
assert [t["id"] for t in qenv.q.PENDING] == ["settled-child", "live-child"]
|
||||
assert custody_calls == []
|
||||
|
||||
|
||||
def test_a1_wake_with_an_open_intent_drives_custody_once_and_still_says_nothing(qenv, monkeypatch):
|
||||
from supervisor.events_runtime_controls import _drive_cancel_task_event
|
||||
|
||||
custody_calls = []
|
||||
monkeypatch.setattr(qenv.q, "cancel_task_custody",
|
||||
lambda tid, **kw: custody_calls.append(tid) or qenv.tl.CANCEL_CANCELLED)
|
||||
write_task_result(qenv.drive, "wanted", "scheduled")
|
||||
qenv.q.PENDING[:] = [{"id": "wanted", "chat_id": 1}]
|
||||
ci.request_cancel(qenv.drive, "wanted", reason="stop")
|
||||
sent = []
|
||||
_drive_cancel_task_event({"task_id": "wanted"}, _event_ctx(sent))
|
||||
assert custody_calls == ["wanted"]
|
||||
assert sent == []
|
||||
|
||||
|
||||
def test_a1_wake_claimed_by_another_worker_is_silent_and_leaves_the_claim(qenv, monkeypatch):
|
||||
"""The eighth phantom row: the event lost the claim race to the watchdog 10 s before
|
||||
settlement and announced a failure. A refused claim is a typed outcome the wake ignores."""
|
||||
from supervisor.events_runtime_controls import _drive_cancel_task_event
|
||||
|
||||
write_task_result(qenv.drive, "raced", "scheduled")
|
||||
qenv.q.PENDING[:] = [{"id": "raced", "chat_id": 1}]
|
||||
ci.request_cancel(qenv.drive, "raced", reason="stop")
|
||||
owner = ci.claim_intent(qenv.drive, "raced", owner="watchdog")
|
||||
assert owner and not owner.get("claim_refused")
|
||||
sent = []
|
||||
_drive_cancel_task_event({"task_id": "raced"}, _event_ctx(sent))
|
||||
assert sent == []
|
||||
current = ci.active_intent(qenv.drive, "raced")
|
||||
assert (current["state"], current["claim_owner"], current["generation"]) == (
|
||||
ci.INTENT_CLAIMED, "watchdog", owner["generation"])
|
||||
assert [t["id"] for t in qenv.q.PENDING] == ["raced"]
|
||||
assert load_task_result(qenv.drive, "raced")["status"] == "scheduled"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- A2 -------------------------
|
||||
|
||||
def test_a2_the_no_intent_boolean_facade_is_gone(qenv, monkeypatch):
|
||||
"""cancel_task_by_id is cascade-only by signature; the single shape ran custody without
|
||||
any intent. A cascade over a tree nobody asked to cancel tears nothing down."""
|
||||
import inspect
|
||||
|
||||
assert list(inspect.signature(qenv.tl.cancel_task_by_id).parameters) == ["task_id"]
|
||||
assert not hasattr(qenv.q, "_cancel_task_by_id_single")
|
||||
monkeypatch.setattr(qenv.q, "_emit_cancel_task_done", lambda *_a, **_k: None)
|
||||
qenv.q.PENDING[:] = [{"id": "facade", "chat_id": 1}]
|
||||
write_task_result(qenv.drive, "facade", "scheduled")
|
||||
assert qenv.tl.cancel_task_by_id("facade") is False
|
||||
assert [t["id"] for t in qenv.q.PENDING] == ["facade"]
|
||||
assert load_task_result(qenv.drive, "facade")["status"] == "scheduled"
|
||||
|
||||
|
||||
def test_a2_cascade_stops_a_child_whose_own_mint_failed_only_under_the_root_claim(qenv, monkeypatch):
|
||||
"""The one authority besides a task's own intent is the root's durable cascade
|
||||
intent, passed explicitly by the sweep. Without it a cascade touches nothing; with
|
||||
it a child whose intent write failed is still stopped, its forensic row kept."""
|
||||
import json
|
||||
|
||||
monkeypatch.setattr(qenv.q, "_emit_cancel_task_done", lambda *_a, **_k: None)
|
||||
qenv.q.PENDING[:] = [
|
||||
{"id": "tree", "chat_id": 1, "root_task_id": "tree"},
|
||||
{"id": "kid", "chat_id": 1, "root_task_id": "tree", "parent_task_id": "tree"},
|
||||
]
|
||||
write_task_result(qenv.drive, "tree", "scheduled")
|
||||
write_task_result(qenv.drive, "kid", "scheduled", parent_task_id="tree", root_task_id="tree")
|
||||
real_request = ci.request_cancel
|
||||
|
||||
def _unwritable_for_the_child(root, tid, **kw):
|
||||
if tid == "kid":
|
||||
raise OSError("intent store unwritable")
|
||||
return real_request(root, tid, **kw)
|
||||
|
||||
monkeypatch.setattr(ci, "request_cancel", _unwritable_for_the_child)
|
||||
|
||||
assert qenv.tl.cancel_task_by_id("tree") is False
|
||||
assert sorted(t["id"] for t in qenv.q.PENDING) == ["kid", "tree"]
|
||||
assert load_task_result(qenv.drive, "kid")["status"] == "scheduled"
|
||||
|
||||
real_request(qenv.drive, "tree", scope=ci.SCOPE_CASCADE, reason="stop the tree")
|
||||
assert qenv.tl.cancel_task_by_id("tree") is True
|
||||
assert qenv.q.PENDING == []
|
||||
assert load_task_result(qenv.drive, "kid")["status"] == STATUS_CANCELLED
|
||||
assert load_task_result(qenv.drive, "tree")["status"] == STATUS_CANCELLED
|
||||
rows = [json.loads(line) for line in
|
||||
(qenv.drive / "logs" / "supervisor.jsonl").read_text(encoding="utf-8").splitlines()]
|
||||
assert any(row.get("type") == "cascade_descendant_intent_write_failed" and row.get("task_id") == "kid"
|
||||
for row in rows)
|
||||
|
||||
|
||||
def test_a2_retry_transfer_mints_no_successor_authority_after_an_empty_claim(qenv, monkeypatch):
|
||||
"""The legacy retry-transfer lane runs before the main claim; an empty claim on the
|
||||
logical root must not mint an intent on the physical retry and tear it down."""
|
||||
from tests._cancel_intents_shared import _write_root_retry_pair
|
||||
|
||||
monkeypatch.setattr(qenv.q, "_emit_cancel_task_done", lambda *_a, **_k: None)
|
||||
monkeypatch.setattr(qenv.q, "QUEUE_MAX_RETRIES", 2)
|
||||
_write_root_retry_pair(qenv.drive, "old-root", "new-root")
|
||||
qenv.q.PENDING[:] = [{
|
||||
"id": "new-root", "type": "task", "chat_id": 0, "depth": 0, "root_task_id": "old-root",
|
||||
"parent_task_id": "", "delegation_role": "root", "original_task_id": "old-root",
|
||||
"timeout_retry_from": "old-root",
|
||||
}]
|
||||
|
||||
assert qenv.tl.cancel_task_custody("old-root") == qenv.tl.CANCEL_NOT_FOUND
|
||||
assert ci.active_intents(qenv.drive) == {}
|
||||
assert [t["id"] for t in qenv.q.PENDING] == ["new-root"]
|
||||
|
||||
minted = ci.request_cancel(qenv.drive, "old-root", reason="stop")
|
||||
assert minted["task_id"] == "new-root"
|
||||
assert qenv.tl.cancel_task_custody("new-root") == qenv.tl.CANCEL_CANCELLED
|
||||
assert qenv.q.PENDING == []
|
||||
assert load_task_result(qenv.drive, "new-root")["status"] == STATUS_CANCELLED
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- A3 -------------------------
|
||||
|
||||
def _signature_busy_at_zero_budget(monkeypatch):
|
||||
"""A writer is active: a normal-budget read waits and succeeds; a zero-budget
|
||||
probe (``acquisition_budget(0)``) cannot acquire and raises StoreBusy. Returns
|
||||
a callable that lets the store answer again."""
|
||||
from ouroboros import store, task_store
|
||||
|
||||
real_sig = task_store.signature
|
||||
|
||||
def _sig(root, tid, *, txn=None):
|
||||
if store._WAIT_BUDGET.get() == 0:
|
||||
raise store.StoreBusy("writer active")
|
||||
return real_sig(root, tid, txn=txn)
|
||||
|
||||
monkeypatch.setattr(task_store, "signature", _sig)
|
||||
return lambda: monkeypatch.setattr(task_store, "signature", real_sig)
|
||||
|
||||
|
||||
def test_a3_load_outside_queue_lock_is_one_signed_transaction(tmp_path, monkeypatch):
|
||||
"""The incident's false FAILED: body read, then a zero-budget signature re-check that a
|
||||
concurrent writer turned into "changed while reading". Outside Q the body and its
|
||||
signature come from one transaction with the normal budget and nothing is re-checked."""
|
||||
from ouroboros import task_store
|
||||
from supervisor.task_ownership import TaskOwnershipRead
|
||||
|
||||
write_task_result(tmp_path, "t-a3", "running")
|
||||
_signature_busy_at_zero_budget(monkeypatch)
|
||||
reads = TaskOwnershipRead(tmp_path)
|
||||
assert reads.load("t-a3")["status"] == "running"
|
||||
assert reads.stamps["t-a3"] == task_store.signature(tmp_path, "t-a3")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("shape", ["subagent", "root"])
|
||||
def test_a3_a_task_without_superseded_by_is_not_rechecked_under_the_lock(qenv, monkeypatch, shape):
|
||||
"""'Walk the retry chain only if the task has superseded_by': a task whose live worker
|
||||
rewrites its own row between preparation and the locked resolver is still cancelled."""
|
||||
from supervisor import task_ownership
|
||||
|
||||
lineage = ({"delegation_role": "subagent", "parent_task_id": "root-a3", "root_task_id": "root-a3"}
|
||||
if shape == "subagent" else {"delegation_role": "root", "root_task_id": "kid-a3"})
|
||||
ci.request_cancel(qenv.drive, "kid-a3", reason="stop")
|
||||
qenv.q.PENDING[:] = [{"id": "kid-a3", "chat_id": 1, **lineage}]
|
||||
write_task_result(qenv.drive, "kid-a3", "scheduled", **lineage)
|
||||
real_prepare = task_ownership.prepare_retry_chain
|
||||
|
||||
def _prepare_then_worker_writes(q, task_id, read):
|
||||
real_prepare(q, task_id, read)
|
||||
write_task_result(qenv.drive, task_id, "scheduled", progress_note="worker wrote meanwhile")
|
||||
|
||||
monkeypatch.setattr(task_ownership, "prepare_retry_chain", _prepare_then_worker_writes)
|
||||
monkeypatch.setattr(qenv.q, "_emit_cancel_task_done", lambda *_a, **_k: None)
|
||||
assert qenv.tl.cancel_task_custody("kid-a3") == qenv.tl.CANCEL_CANCELLED
|
||||
assert load_task_result(qenv.drive, "kid-a3")["status"] == STATUS_CANCELLED
|
||||
assert qenv.q.PENDING == []
|
||||
|
||||
|
||||
def _retry_chain(qenv, monkeypatch):
|
||||
from tests._cancel_intents_shared import _write_root_retry_pair
|
||||
|
||||
monkeypatch.setattr(qenv.q, "_emit_cancel_task_done", lambda *_a, **_k: None)
|
||||
monkeypatch.setattr(qenv.q, "QUEUE_MAX_RETRIES", 2)
|
||||
_write_root_retry_pair(qenv.drive, "old-root", "new-root")
|
||||
qenv.q.PENDING[:] = [{
|
||||
"id": "new-root", "type": "task", "chat_id": 0, "depth": 0, "root_task_id": "old-root",
|
||||
"parent_task_id": "", "delegation_role": "root", "original_task_id": "old-root",
|
||||
"timeout_retry_from": "old-root",
|
||||
}]
|
||||
minted = ci.request_cancel(qenv.drive, "old-root", reason="stop")
|
||||
assert minted["task_id"] == "new-root"
|
||||
|
||||
|
||||
def test_a3_busy_store_under_the_lock_is_unknown_and_the_watchdog_finishes(qenv, monkeypatch, caplog):
|
||||
"""With superseded_by the chain IS re-checked under Q with a zero budget: BUSY is
|
||||
neither a change nor sameness — no capture, no claim taken, the intent stays open
|
||||
(requested), the logged reason says busy (not the incident's false "changed"), and
|
||||
the next sweep settles it once the store answers."""
|
||||
import logging
|
||||
from datetime import datetime
|
||||
|
||||
_retry_chain(qenv, monkeypatch)
|
||||
store_answers_again = _signature_busy_at_zero_budget(monkeypatch)
|
||||
with caplog.at_level(logging.ERROR, logger="supervisor.task_lifecycle"):
|
||||
assert qenv.tl.cancel_task_custody("old-root") == qenv.tl.CANCEL_FAILED
|
||||
assert "ownership result busy during this pass" in caplog.text
|
||||
assert "changed" not in caplog.text
|
||||
intent = ci.active_intent(qenv.drive, "new-root")
|
||||
assert intent["state"] == ci.INTENT_REQUESTED
|
||||
assert [t["id"] for t in qenv.q.PENDING] == ["new-root"]
|
||||
assert load_task_result(qenv.drive, "new-root")["status"] == "scheduled"
|
||||
|
||||
store_answers_again()
|
||||
requested_at = datetime.fromisoformat(intent["requested_at"].replace("Z", "+00:00")).timestamp()
|
||||
outcomes = qenv.tl.sweep_cancel_intents(now=requested_at + 60)
|
||||
assert outcomes == {"new-root": qenv.tl.CANCEL_CANCELLED}
|
||||
assert load_task_result(qenv.drive, "new-root")["status"] == STATUS_CANCELLED
|
||||
assert ci.active_intents(qenv.drive) == {}
|
||||
|
||||
|
||||
def test_a3_a_changed_chain_row_under_the_lock_refuses_without_touching_anything(qenv, monkeypatch):
|
||||
from supervisor import task_ownership
|
||||
|
||||
_retry_chain(qenv, monkeypatch)
|
||||
real_prepare = task_ownership.prepare_retry_chain
|
||||
|
||||
def _prepare_then_leaf_rewritten(q, task_id, read):
|
||||
real_prepare(q, task_id, read)
|
||||
write_task_result(qenv.drive, "new-root", "scheduled", progress_note="rewritten meanwhile")
|
||||
|
||||
monkeypatch.setattr(task_ownership, "prepare_retry_chain", _prepare_then_leaf_rewritten)
|
||||
assert qenv.tl.cancel_task_custody("old-root") == qenv.tl.CANCEL_FAILED
|
||||
assert ci.active_intent(qenv.drive, "new-root")["state"] == ci.INTENT_REQUESTED
|
||||
assert [t["id"] for t in qenv.q.PENDING] == ["new-root"]
|
||||
assert load_task_result(qenv.drive, "new-root")["status"] == "scheduled"
|
||||
|
||||
|
||||
def test_a3_custody_parses_no_result_under_the_queue_lock(qenv, monkeypatch):
|
||||
"""The capture judges the row prepared outside the lock; a result that settles
|
||||
between preparation and capture is still honoured (completion wins)."""
|
||||
from ouroboros import task_results
|
||||
from supervisor import task_ownership
|
||||
|
||||
parses_under_lock = []
|
||||
real_load = task_results.load_task_result
|
||||
|
||||
def _load(root, tid, **kw):
|
||||
if qenv.q._queue_lock._is_owned():
|
||||
parses_under_lock.append(tid)
|
||||
return real_load(root, tid, **kw)
|
||||
|
||||
monkeypatch.setattr(task_results, "load_task_result", _load)
|
||||
monkeypatch.setattr(qenv.q, "_emit_cancel_task_done", lambda *_a, **_k: None)
|
||||
ci.request_cancel(qenv.drive, "queued", reason="stop")
|
||||
qenv.q.PENDING[:] = [{"id": "queued", "chat_id": 1}]
|
||||
write_task_result(qenv.drive, "queued", "scheduled")
|
||||
assert qenv.tl.cancel_task_custody("queued") == qenv.tl.CANCEL_CANCELLED
|
||||
assert parses_under_lock == []
|
||||
|
||||
ci.request_cancel(qenv.drive, "finisher", reason="stop")
|
||||
write_task_result(qenv.drive, "finisher", "running")
|
||||
real_prepare = task_ownership.prepare_retry_chain
|
||||
|
||||
def _prepare_then_completes(q, task_id, read):
|
||||
real_prepare(q, task_id, read)
|
||||
write_task_result(qenv.drive, task_id, "completed", result="done just in time")
|
||||
|
||||
monkeypatch.setattr(task_ownership, "prepare_retry_chain", _prepare_then_completes)
|
||||
assert qenv.tl.cancel_task_custody("finisher") == qenv.tl.CANCEL_ALREADY_SETTLED
|
||||
assert load_task_result(qenv.drive, "finisher")["status"] == "completed"
|
||||
assert parses_under_lock == []
|
||||
assert ci.active_intents(qenv.drive) == {}
|
||||
|
||||
|
||||
# ------------------------------------------------- fix round 1: Presence-owned rows ----------
|
||||
|
||||
def _presence_row(drive, task_id, kind="presence"):
|
||||
from ouroboros.task_status import execution_owner_record
|
||||
|
||||
write_task_result(drive, task_id, "running", result="Task is running.", _is_direct_chat=True,
|
||||
execution_owner=execution_owner_record(drive, {"id": task_id}, kind))
|
||||
|
||||
|
||||
def test_miss_lane_leaves_a_live_presence_turns_row_to_its_runner(qenv, monkeypatch):
|
||||
"""Intent-first minting reaches custody for a RUNNING row no supervisor map owns. A
|
||||
Presence turn (its own thread, execution_owner kind 'presence') must not be rewritten
|
||||
as cancelled while it keeps running: the intent settles not_found, the row stays.
|
||||
A pooled row no map owns is still finalized on miss."""
|
||||
from supervisor import terminal_delivery
|
||||
|
||||
monkeypatch.setattr(qenv.q, "_emit_cancel_task_done", lambda *_a, **_k: None)
|
||||
monkeypatch.setattr(terminal_delivery, "deliver_miss_lane_outcome", lambda *a, **kw: True, raising=False)
|
||||
_presence_row(qenv.drive, "presence-turn")
|
||||
ci.request_cancel(qenv.drive, "presence-turn", reason="stop")
|
||||
|
||||
assert qenv.tl.cancel_task_custody("presence-turn") == qenv.tl.CANCEL_NOT_FOUND
|
||||
assert load_task_result(qenv.drive, "presence-turn")["status"] == "running"
|
||||
assert ci.active_intents(qenv.drive) == {}
|
||||
|
||||
_presence_row(qenv.drive, "orphan-pooled", kind="pooled")
|
||||
ci.request_cancel(qenv.drive, "orphan-pooled", reason="stop")
|
||||
assert qenv.tl.cancel_task_custody("orphan-pooled") == qenv.tl.CANCEL_CANCELLED
|
||||
assert load_task_result(qenv.drive, "orphan-pooled")["status"] == STATUS_CANCELLED
|
||||
assert ci.active_intents(qenv.drive) == {}
|
||||
|
||||
|
||||
def test_http_stop_on_a_live_presence_turn_answers_404_and_leaves_no_intent(tmp_path, monkeypatch):
|
||||
from starlette.applications import Starlette
|
||||
from starlette.routing import Route
|
||||
from starlette.testclient import TestClient
|
||||
|
||||
from ouroboros.gateway.tasks import api_task_cancel
|
||||
from supervisor import queue, workers
|
||||
|
||||
monkeypatch.setattr(queue, "DRIVE_ROOT", tmp_path)
|
||||
monkeypatch.setattr(queue, "PENDING", [])
|
||||
monkeypatch.setattr(queue, "RUNNING", {})
|
||||
monkeypatch.setattr(workers, "WORKERS", {}, raising=False)
|
||||
monkeypatch.setattr(queue, "persist_queue_snapshot", lambda reason="": None)
|
||||
monkeypatch.setattr(queue, "_emit_cancel_task_done", lambda *_a, **_k: None)
|
||||
_presence_row(tmp_path, "presence-http")
|
||||
app = Starlette(routes=[Route("/api/tasks/{task_id}/cancel", api_task_cancel, methods=["POST"])])
|
||||
app.state.drive_root = tmp_path
|
||||
with TestClient(app) as client:
|
||||
response = client.post("/api/tasks/presence-http/cancel")
|
||||
assert response.status_code == 404, response.text
|
||||
assert load_task_result(tmp_path, "presence-http")["status"] == "running"
|
||||
assert ci.active_intents(tmp_path) == {}
|
||||
|
||||
|
||||
def test_http_cascade_stop_on_a_live_presence_turn_answers_404_and_records_nothing(tmp_path, monkeypatch):
|
||||
"""The cascade path reads the target row before minting; a live Presence turn's row
|
||||
is its runner's, so the whole-tree Stop answers 404 and records no intent, while an
|
||||
ordinary persisted task in the same shape is still accepted and stopped."""
|
||||
from starlette.applications import Starlette
|
||||
from starlette.routing import Route
|
||||
from starlette.testclient import TestClient
|
||||
|
||||
from ouroboros.gateway.tasks import api_task_cancel
|
||||
from supervisor import queue, workers
|
||||
|
||||
monkeypatch.setattr(queue, "DRIVE_ROOT", tmp_path)
|
||||
monkeypatch.setattr(queue, "PENDING", [])
|
||||
monkeypatch.setattr(queue, "RUNNING", {})
|
||||
monkeypatch.setattr(workers, "WORKERS", {}, raising=False)
|
||||
monkeypatch.setattr(queue, "persist_queue_snapshot", lambda reason="": None)
|
||||
monkeypatch.setattr(queue, "_emit_cancel_task_done", lambda *_a, **_k: None)
|
||||
_presence_row(tmp_path, "presence-tree")
|
||||
_presence_row(tmp_path, "pooled-tree", kind="pooled")
|
||||
app = Starlette(routes=[Route("/api/tasks/{task_id}/cancel", api_task_cancel, methods=["POST"])])
|
||||
app.state.drive_root = tmp_path
|
||||
with TestClient(app) as client:
|
||||
presence = client.post("/api/tasks/presence-tree/cancel", json={"cascade": True})
|
||||
pooled = client.post("/api/tasks/pooled-tree/cancel", json={"cascade": True})
|
||||
assert presence.status_code == 404, presence.text
|
||||
assert load_task_result(tmp_path, "presence-tree")["status"] == "running"
|
||||
assert "presence-tree" not in ci.active_intents(tmp_path)
|
||||
assert pooled.status_code in (200, 202), pooled.text
|
||||
assert pooled.status_code == 202 or load_task_result(tmp_path, "pooled-tree")["status"] == "cancelled"
|
||||
|
|
@ -136,10 +136,10 @@ def test_http_observation_records_transport_without_inventing_owner_identity(tmp
|
|||
write_task_result(tmp_path, "task", "running")
|
||||
_queue(tmp_path, "task")
|
||||
monkeypatch.setattr(task_queue, "DRIVE_ROOT", tmp_path)
|
||||
monkeypatch.setattr(task_queue, "task_has_live_ownership", lambda task_id: True)
|
||||
monkeypatch.setattr(task_queue, "task_subtree_is_live", lambda task_id: True)
|
||||
monkeypatch.setattr(task_queue, "cancel_task_custody", lambda task_id: task_queue.CANCEL_CANCELLED)
|
||||
monkeypatch.setattr(tasks, "_run_cascade_cancel", lambda task_id: True)
|
||||
# A running row is recorded with no liveness probe; the teardown owners are
|
||||
# stubbed so the recorded intent stays readable after the 200.
|
||||
monkeypatch.setattr(task_queue, "cancel_task_custody", lambda task_id, **_kw: task_queue.CANCEL_CANCELLED)
|
||||
monkeypatch.setattr(task_queue, "cancel_task_by_id", lambda task_id, **_kw: True)
|
||||
monkeypatch.setattr("ouroboros.delegate_evidence.task_execution_evidence", lambda *_a: pytest.fail("HTTP must not scan delegated descendants"))
|
||||
app = Starlette(routes=[Route("/api/tasks/{task_id}/cancel", tasks.api_task_cancel, methods=["POST"])])
|
||||
app.state.drive_root = tmp_path
|
||||
|
|
|
|||
|
|
@ -46,7 +46,7 @@ def test_cascade_origin_links_the_child_to_its_cancelled_root(qenv, monkeypatch)
|
|||
write_task_result(qenv.drive, task["id"], "scheduled")
|
||||
cancel_intents.request_cancel(qenv.drive, "root", source="http_cascade", scope="cascade")
|
||||
|
||||
assert qenv.tl.cancel_task_by_id("root", cascade=True)
|
||||
assert qenv.tl.cancel_task_by_id("root")
|
||||
|
||||
root = load_task_result(qenv.drive, "root")
|
||||
child = load_task_result(qenv.drive, "child")
|
||||
|
|
|
|||
|
|
@ -492,7 +492,7 @@ def test_cascade_receipt_dedups_the_actual_destination_and_preserves_main(qenv,
|
|||
qenv.drive, "live-kid", "scheduled",
|
||||
parent_task_id="settled-root", root_task_id="settled-root",
|
||||
)
|
||||
assert qenv.tl.cancel_task_by_id("settled-root", cascade=True)
|
||||
assert qenv.tl.cancel_task_by_id("settled-root")
|
||||
event = next(e for e in queue.events if e.get("system_type") == "cancel_receipt")
|
||||
assert event["chat_id"] == 77 and text in event["text"]
|
||||
before = load_task_result(qenv.drive, "settled-root")
|
||||
|
|
|
|||
|
|
@ -292,8 +292,10 @@ def test_panel_chat_zero_stays_hidden_and_request_zero_still_means_main(tmp_path
|
|||
def test_panel_zero_exclusion_does_not_override_project_task_binding():
|
||||
from ouroboros.gateway.history import _make_thread_filter
|
||||
|
||||
project_filter = _make_thread_filter(42, {42}, [], {"root-1": 42})
|
||||
main_filter = _make_thread_filter(1, {42}, [], {"root-1": 42})
|
||||
from tests._room_shared import RoomsFromBindings
|
||||
|
||||
project_filter = _make_thread_filter(42, {42}, [], RoomsFromBindings({"root-1": 42}))
|
||||
main_filter = _make_thread_filter(1, {42}, [], RoomsFromBindings({"root-1": 42}))
|
||||
|
||||
row = {"task_id": "root-1"}
|
||||
assert project_filter(0, row) is True
|
||||
|
|
|
|||
|
|
@ -196,10 +196,13 @@ def test_main_and_native_project_continue_keep_their_own_room(host, case):
|
|||
assert (bound or {}).get("project_id") == (pid if case == "native-bound" else None)
|
||||
rooms = _room_of(host, successor, row)
|
||||
assert rooms["live"] == chat
|
||||
if case == "native-bound":
|
||||
assert rooms == {"live": chat, "late": chat, "child": chat, "tree": chat, "delivery": chat}
|
||||
if case == "main":
|
||||
assert rooms["tree"] == 0, "Main work gains no Project room"
|
||||
elif case == "native": # a room-born root is addressed to its room unbound (the one room rule);
|
||||
# an id the store never admitted ("helper-1") has no room of its own yet: the caller keeps its address
|
||||
assert {key: rooms[key] for key in ("live", "late", "tree", "child")} == {"live": chat, "late": chat, "tree": chat, "child": None}
|
||||
else:
|
||||
assert rooms["tree"] == 0, "unbound work gains no binding"
|
||||
assert rooms == {"live": chat, "late": chat, "child": chat, "tree": chat, "delivery": chat}
|
||||
|
||||
|
||||
def test_a_claim_frozen_before_a_later_conversion_keeps_its_destination(host, monkeypatch):
|
||||
|
|
|
|||
|
|
@ -425,10 +425,10 @@ def _collect_literal_progress_meta_keys(source_path: pathlib.Path) -> set[str]:
|
|||
keys: set[str] = set()
|
||||
meta_helper_names = {"_subagent_rejection_meta", "_subagent_progress_meta", "_subagent_scheduled_meta"}
|
||||
# A producer may build the dict under a local name and hand it over as
|
||||
# ``progress_meta=<name>`` (events_runtime_controls: ``incident_meta``);
|
||||
# every such name is an alias of ``progress_meta`` for the walk below, or
|
||||
# the scan is blind to exactly those emitters and stays green when their
|
||||
# keys drop out of the carriers.
|
||||
# ``progress_meta=<name>`` (the retired cancellation-incident emitter did,
|
||||
# as ``incident_meta``); every such name is an alias of ``progress_meta``
|
||||
# for the walk below, or the scan is blind to exactly those emitters and
|
||||
# stays green when their keys drop out of the carriers.
|
||||
meta_names = {"progress_meta"}
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.keyword) and node.arg == "progress_meta" and isinstance(node.value, ast.Name):
|
||||
|
|
@ -577,7 +577,7 @@ def test_chat_outbound_matches_message_bus_sends():
|
|||
)
|
||||
|
||||
|
||||
def test_chat_outbound_declares_progress_meta_keys_used_by_runtime():
|
||||
def test_chat_outbound_declares_progress_meta_keys_used_by_runtime(tmp_path):
|
||||
"""Progress metadata is merged into ChatOutbound frames by message_bus."""
|
||||
|
||||
from ouroboros.gateway.contracts import ChatOutbound
|
||||
|
|
@ -599,9 +599,19 @@ def test_chat_outbound_declares_progress_meta_keys_used_by_runtime():
|
|||
progress_keys.update(_collect_literal_progress_meta_keys(source_path))
|
||||
|
||||
assert progress_keys, "no literal progress_meta keys found"
|
||||
# The named-dict idiom must be seen: this key has exactly one emitter and
|
||||
# it hands the dict over as ``progress_meta=incident_meta``.
|
||||
assert "cancel_physical_task_id" in progress_keys, sorted(progress_keys)
|
||||
# The named-dict idiom (``progress_meta=<local name>``) has no live emitter
|
||||
# since the cancellation incident row was retired, so the alias walk is
|
||||
# proven on a synthetic producer: a future named-dict emitter cannot slip
|
||||
# past the scan unnoticed.
|
||||
synthetic = tmp_path / "named_dict_producer.py"
|
||||
synthetic.write_text(
|
||||
"def emit(ctx):\n"
|
||||
" incident_meta = {\"task_incident\": \"synthetic_fault\"}\n"
|
||||
" incident_meta[\"synthetic_physical_task_id\"] = \"t\"\n"
|
||||
" ctx.send_with_budget(1, \"x\", progress_meta=incident_meta)\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
assert {"task_incident", "synthetic_physical_task_id"} <= _collect_literal_progress_meta_keys(synthetic)
|
||||
assert progress_keys <= declared, (
|
||||
"progress_meta emits keys not declared in ChatOutbound: "
|
||||
f"{sorted(progress_keys - declared)}"
|
||||
|
|
|
|||
|
|
@ -412,7 +412,9 @@ def test_kill_fast_lane_clears_a_stale_disclosure(tmp_path, monkeypatch):
|
|||
reason_code="delegated_custody_unreconciled",
|
||||
delegated_runs_unreconciled=["run-stale"], delegated_runs_settled=0,
|
||||
)
|
||||
ci.request_cancel(env.drive, "t-fast")
|
||||
# The ingress saw live ownership (a worker still winding down) and minted on the
|
||||
# settled target; by the time custody runs the worker is gone: the fast lane.
|
||||
ci.request_cancel(env.drive, "t-fast", allow_settled_target=True)
|
||||
|
||||
assert env.tl.cancel_task_custody("t-fast", deliver=False) == env.tl.CANCEL_ALREADY_SETTLED
|
||||
|
||||
|
|
@ -434,7 +436,7 @@ def test_kill_fast_lane_adds_no_write_when_nothing_is_stale(tmp_path, monkeypatc
|
|||
|
||||
env = _kill_env(tmp_path, monkeypatch)
|
||||
write_task_result(env.drive, "t-clean-kill", STATUS_FAILED, result="settled")
|
||||
ci.request_cancel(env.drive, "t-clean-kill")
|
||||
ci.request_cancel(env.drive, "t-clean-kill", allow_settled_target=True)
|
||||
row_before = result_signature(tmp_path, "t-clean-kill")
|
||||
|
||||
writes: list = []
|
||||
|
|
|
|||
|
|
@ -14,9 +14,10 @@ resolve a live direct turn through — so a turn the task list shows as
|
|||
running is addressable, and custody stops it COOPERATIVELY (the finalize
|
||||
control on the canonical owner mailbox; there is no worker process to kill).
|
||||
|
||||
The two ``orphan`` cases at the top keep the honest legacy envelope for a
|
||||
durable running row that NOTHING owns (no queue row, no worker, no live
|
||||
chat turn): that is still 404, because there is nothing to stop.
|
||||
The two ``orphan`` cases at the top pin the Stop of a durable running row
|
||||
that NOTHING owns (no queue row, no worker, no live chat turn): the intent is
|
||||
recorded first and custody's miss lane settles the stuck row ``cancelled``,
|
||||
as the agent's ``cancel_task`` does — never a 404 that leaves it running.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -97,13 +98,17 @@ def _mailbox_kinds(tmp_path, task_id=TURN_ID):
|
|||
return [json.loads(line).get("kind") for line in path.read_text(encoding="utf-8").splitlines() if line.strip()]
|
||||
|
||||
|
||||
# --- the honest residual: a running row that nothing owns ----------------------
|
||||
# --- a running row that nothing owns is stuck, and Stop settles it -------------
|
||||
|
||||
@pytest.mark.parametrize("snapshot", ["absent", "fresh_without_row"])
|
||||
def test_orphaned_running_row_keeps_the_404_envelope(tmp_path, monkeypatch, snapshot):
|
||||
def test_orphaned_running_row_is_settled_cancelled_by_stop(tmp_path, monkeypatch, snapshot):
|
||||
"""No queue row, no worker slot, no live chat turn: listed by the durable
|
||||
mirror (until the orphan projection catches it) but there is nothing to
|
||||
stop, so the cancel ingress keeps its legacy 404."""
|
||||
mirror (until the orphan projection catches it) with nothing behind it.
|
||||
Stop records its intent first and custody's miss lane settles the row
|
||||
``cancelled`` (the agent tool's behaviour), instead of a 404 that left it
|
||||
listed as running."""
|
||||
from ouroboros.cancel_intents import active_intent
|
||||
|
||||
_isolate_queue(monkeypatch, tmp_path)
|
||||
write_task_result(tmp_path, TURN_ID, "running", chat_id=0, description="orphan")
|
||||
if snapshot == "fresh_without_row":
|
||||
|
|
@ -114,7 +119,10 @@ def test_orphaned_running_row_keeps_the_404_envelope(tmp_path, monkeypatch, snap
|
|||
cancel = client.post(f"/api/tasks/{TURN_ID}/cancel")
|
||||
|
||||
assert [row["task_id"] for row in listed["tasks"]] == [TURN_ID]
|
||||
assert cancel.status_code == 404
|
||||
assert cancel.status_code == 200, cancel.text
|
||||
assert cancel.json() == {"ok": True, "task_id": TURN_ID}
|
||||
assert load_task_result(tmp_path, TURN_ID)["status"] == "cancelled"
|
||||
assert not active_intent(tmp_path, TURN_ID)
|
||||
|
||||
|
||||
def test_running_row_present_in_queue_is_cancellable(tmp_path, monkeypatch):
|
||||
|
|
@ -198,11 +206,11 @@ def test_stop_now_finalizes_the_direct_turn_cooperatively_and_answers_ok(tmp_pat
|
|||
assert not active_intent(tmp_path, TURN_ID)
|
||||
|
||||
|
||||
def test_stop_now_on_a_turn_inside_a_long_step_answers_still_live_and_keeps_the_stop_armed(tmp_path, monkeypatch):
|
||||
def test_stop_now_on_a_turn_inside_a_long_step_answers_pending_and_keeps_the_stop_armed(tmp_path, monkeypatch):
|
||||
"""A turn that has not reached a round boundary within the wait bound:
|
||||
503 "still live", the durable row is NOT rewritten as cancelled (the
|
||||
miss finalizer must never run for a live turn), the control stays in
|
||||
the mailbox and the intent stays open for the sweep's next custody pass."""
|
||||
202 pending, the durable row is NOT rewritten as cancelled (the miss
|
||||
finalizer must never run for a live turn), the control stays in the
|
||||
mailbox and the intent stays open for the sweep's next custody pass."""
|
||||
import ouroboros.config as config
|
||||
|
||||
_isolate_queue(monkeypatch, tmp_path)
|
||||
|
|
@ -215,13 +223,13 @@ def test_stop_now_on_a_turn_inside_a_long_step_answers_still_live_and_keeps_the_
|
|||
with _client(tmp_path) as client:
|
||||
cancel = client.post(f"/api/tasks/{TURN_ID}/cancel", json={"stop_policy": "immediate"})
|
||||
|
||||
assert cancel.status_code == 503, cancel.text
|
||||
assert "still live" in cancel.json()["error"]
|
||||
assert cancel.status_code == 202, cancel.text
|
||||
assert cancel.json()["cancel_state"] == "pending"
|
||||
assert load_task_result(tmp_path, TURN_ID)["status"] == "running"
|
||||
assert "finalize_now" in _mailbox_kinds(tmp_path)
|
||||
from ouroboros.cancel_intents import active_intent
|
||||
|
||||
assert active_intent(tmp_path, TURN_ID)
|
||||
assert active_intent(tmp_path, TURN_ID)["request_id"] == cancel.json()["request_id"]
|
||||
|
||||
|
||||
def test_wrap_up_arms_the_owner_stop_episode_for_the_direct_turn(tmp_path, monkeypatch):
|
||||
|
|
@ -346,7 +354,7 @@ def test_stop_now_on_a_turn_that_already_ended_arms_nothing_and_answers_already_
|
|||
|
||||
def _turn_ends_after_custody_captured(task_id=""):
|
||||
reads["n"] += 1
|
||||
if reads["n"] == 3: # ownership predicate, custody capture, then the lane's re-check
|
||||
if reads["n"] == 2: # custody capture, then the lane's re-check (no pre-probe: the row is unsettled)
|
||||
write_task_result(tmp_path, TURN_ID, "completed", chat_id=0, description="modify yourself")
|
||||
agent._busy = False
|
||||
return real_reader(task_id)
|
||||
|
|
@ -406,8 +414,8 @@ def test_settled_row_does_not_hide_a_still_busy_direct_turn_from_custody(tmp_pat
|
|||
"""Scope-review finding: the pipeline persists the terminal BEFORE post-task
|
||||
cognition, so a direct turn can be settled on disk and still busy (and
|
||||
spending). Custody must not take the settled fast path past a live turn:
|
||||
it arms the stop and answers "still live" until the agent is really done,
|
||||
then settles already_settled against the stored terminal."""
|
||||
it arms the stop and answers pending until the agent is really done, then
|
||||
settles already_settled against the stored terminal."""
|
||||
import ouroboros.config as config
|
||||
|
||||
_isolate_queue(monkeypatch, tmp_path)
|
||||
|
|
@ -417,7 +425,8 @@ def test_settled_row_does_not_hide_a_still_busy_direct_turn_from_custody(tmp_pat
|
|||
monkeypatch.setattr(config, "get_direct_turn_stop_wait_sec", lambda: 0.3)
|
||||
with _client(tmp_path) as client:
|
||||
first = client.post(f"/api/tasks/{TURN_ID}/cancel", json={"stop_policy": "immediate"})
|
||||
assert first.status_code == 503, first.text # armed, still busy
|
||||
assert first.status_code == 202, first.text # armed, still busy
|
||||
assert first.json()["cancel_state"] == "pending"
|
||||
assert "finalize_now" in _mailbox_kinds(tmp_path)
|
||||
agent._busy = False # the turn ends; no synthesis in flight
|
||||
second = client.post(f"/api/tasks/{TURN_ID}/cancel", json={"stop_policy": "immediate"})
|
||||
|
|
@ -581,7 +590,7 @@ def test_live_ownership_holds_through_the_inflight_synthesis_and_ends_with_its_c
|
|||
def test_stop_now_during_the_inflight_synthesis_is_accepted_and_stops_the_remaining_paid_stages(tmp_path, monkeypatch):
|
||||
"""The audit-point-4 class, end to end over the UI's endpoint: the turn's
|
||||
answer has landed and only its paid synthesis is running. Stop-now is
|
||||
ACCEPTED (typed 503 "still live", the durable immediate intent minted and
|
||||
ACCEPTED (typed 202 pending, the durable immediate intent minted and
|
||||
left open by custody — never 404), the pipeline's per-stage gate then
|
||||
skips every remaining paid stage and discloses them; the next stop after
|
||||
the checkpoint settled answers the honest 404 with the intent drained."""
|
||||
|
|
@ -597,8 +606,8 @@ def test_stop_now_during_the_inflight_synthesis_is_accepted_and_stops_the_remain
|
|||
try:
|
||||
with _client(tmp_path) as client:
|
||||
first = client.post(f"/api/tasks/{TURN_ID}/cancel", json={"stop_policy": "immediate"})
|
||||
assert first.status_code == 503, first.text
|
||||
assert "still live" in first.json()["error"], first.text
|
||||
assert first.status_code == 202, first.text
|
||||
assert first.json()["cancel_state"] == "pending", first.text
|
||||
intent = active_intent(tmp_path, TURN_ID) or {}
|
||||
assert intent.get("state") == INTENT_REQUESTED, intent
|
||||
assert "post-task synthesis" in str(intent.get("last_error") or ""), intent
|
||||
|
|
@ -625,8 +634,8 @@ def test_cascade_stop_now_during_the_inflight_synthesis_keeps_the_intent_open_an
|
|||
audit-point-4 state must hold over the CASCADE path too: the turn's answer
|
||||
landed (stored ``completed``, no queue row, no busy agent) and only its paid
|
||||
synthesis is running. The cascade's PHYSICAL postcondition must see the
|
||||
in-flight synthesis as live — the answer is the single lane's typed 503
|
||||
"still live", the cascade-scope immediate intent stays OPEN for the
|
||||
in-flight synthesis as live — the answer is the typed 202 pending, the
|
||||
cascade-scope immediate intent stays OPEN for the
|
||||
pipeline's per-stage gate, and the remaining paid stages are skipped.
|
||||
(rc.15 review MAJOR 1: the postcondition re-judged custody's refusal
|
||||
against the stored ``completed`` alone, settled the intent and answered
|
||||
|
|
@ -645,8 +654,8 @@ def test_cascade_stop_now_during_the_inflight_synthesis_keeps_the_intent_open_an
|
|||
try:
|
||||
with _client(tmp_path) as client:
|
||||
first = client.post(f"/api/tasks/{TURN_ID}/cancel", json=body)
|
||||
assert first.status_code == 503, first.text
|
||||
assert "still live" in first.json()["error"], first.text
|
||||
assert first.status_code == 202, first.text
|
||||
assert first.json()["cancel_state"] == "pending", first.text
|
||||
intent = active_intent(tmp_path, TURN_ID) or {}
|
||||
assert intent.get("state") == INTENT_REQUESTED, intent
|
||||
assert intent.get("scope") == SCOPE_CASCADE, intent
|
||||
|
|
|
|||
|
|
@ -719,7 +719,7 @@ def test_e11_stop_now_hardens_the_same_intent_without_minting_a_second(e2e_clone
|
|||
first = dict(intents(data_root).get(task_id) or {})
|
||||
assert first.get("request_id"), intents(data_root)
|
||||
|
||||
server.cancel_task(task_id) # Stop-now: 200 while live, 404 if it just settled
|
||||
server.cancel_task(task_id) # Stop-now: 200 once torn down, 202 while still stopping, 404 if it just settled
|
||||
hardened = wait_until(
|
||||
lambda: forensics(data_root, task_id=task_id, event="stop_policy_hardened"), 60,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -69,7 +69,23 @@ def test_reconstruct_task_cost_never_fabricates_zero_when_ledger_unavailable(
|
|||
state.reconstruct_task_cost("paid-task")
|
||||
|
||||
|
||||
def test_stop_evolution_tasks_cancels_only_evolution_pending_and_running(monkeypatch):
|
||||
def _minting(monkeypatch, q, tmp_path):
|
||||
"""The real durable intent, observed: the stop drives custody only through an
|
||||
open intent (A1/A2), so a fixture that mints nothing would stop nothing."""
|
||||
from ouroboros.cancel_intents import request_cancel
|
||||
|
||||
monkeypatch.setattr(q, "DRIVE_ROOT", tmp_path)
|
||||
intents: list = []
|
||||
|
||||
def _mint(root, tid, **kw):
|
||||
intents.append((tid, kw.get("source")))
|
||||
return request_cancel(root, tid, **kw)
|
||||
|
||||
monkeypatch.setattr("ouroboros.cancel_intents.request_cancel", _mint)
|
||||
return intents
|
||||
|
||||
|
||||
def test_stop_evolution_tasks_cancels_only_evolution_pending_and_running(monkeypatch, tmp_path):
|
||||
"""GR2-13: PENDING evolution tasks go through the same durable-intent +
|
||||
typed-custody ingress as running ones (never an in-place prune), and only
|
||||
evolution tasks are touched."""
|
||||
|
|
@ -84,11 +100,7 @@ def test_stop_evolution_tasks_cancels_only_evolution_pending_and_running(monkeyp
|
|||
{"id": "evo-pending", "type": "evolution"},
|
||||
{"id": "plain-pending", "type": "chat"},
|
||||
])
|
||||
intents: list = []
|
||||
monkeypatch.setattr(
|
||||
"ouroboros.cancel_intents.request_cancel",
|
||||
lambda root, tid, **kw: intents.append((tid, kw.get("source"))) or {},
|
||||
)
|
||||
intents = _minting(monkeypatch, q, tmp_path)
|
||||
custody: list = []
|
||||
monkeypatch.setattr(q, "cancel_task_custody",
|
||||
lambda tid, **_kw: custody.append(tid) or q.CANCEL_CANCELLED)
|
||||
|
|
@ -102,7 +114,7 @@ def test_stop_evolution_tasks_cancels_only_evolution_pending_and_running(monkeyp
|
|||
assert not any(bucket for key, bucket in out.items() if key != "cancelled")
|
||||
|
||||
|
||||
def test_stop_evolution_tasks_reports_settled_and_failed_honestly(monkeypatch):
|
||||
def test_stop_evolution_tasks_reports_settled_and_failed_honestly(monkeypatch, tmp_path):
|
||||
"""GR2-13: 'cancelled' names only CANCEL_CANCELLED outcomes; already-settled
|
||||
and failed teardowns are separate buckets, and the report composer marks the
|
||||
stop incomplete only for still-live tasks."""
|
||||
|
|
@ -113,8 +125,7 @@ def test_stop_evolution_tasks_reports_settled_and_failed_honestly(monkeypatch):
|
|||
"evo-stuck": {"task": {"type": "evolution"}},
|
||||
})
|
||||
monkeypatch.setattr(q, "PENDING", [])
|
||||
monkeypatch.setattr("ouroboros.cancel_intents.request_cancel",
|
||||
lambda root, tid, **kw: {})
|
||||
_minting(monkeypatch, q, tmp_path)
|
||||
outcomes = {"evo-done": q.CANCEL_ALREADY_SETTLED, "evo-stuck": q.CANCEL_FAILED}
|
||||
monkeypatch.setattr(q, "cancel_task_custody", lambda tid, **_kw: outcomes[tid])
|
||||
|
||||
|
|
|
|||
|
|
@ -109,7 +109,7 @@ def test_cascade_replay_after_crash_delivers_exactly_one_summary(qenv, monkeypat
|
|||
qenv.drive, "cr1", scope=ci.SCOPE_CASCADE, allow_settled_target=True,
|
||||
)
|
||||
|
||||
assert qenv.tl.cancel_task_by_id("cr1", cascade=True) is True
|
||||
assert qenv.tl.cancel_task_by_id("cr1") is True
|
||||
|
||||
sends = [e for e in queue.events if e.get("type") == "send_message"]
|
||||
assert len(sends) == 1, "the replay/already-down path still owes the summary"
|
||||
|
|
@ -130,7 +130,7 @@ def test_cascade_replay_after_crash_delivers_exactly_one_summary(qenv, monkeypat
|
|||
qenv.drive, "cr1", scope=ci.SCOPE_CASCADE, allow_settled_target=True,
|
||||
)
|
||||
assert second["request_id"] != minted["request_id"]
|
||||
assert qenv.tl.cancel_task_by_id("cr1", cascade=True) is True
|
||||
assert qenv.tl.cancel_task_by_id("cr1") is True
|
||||
later = [e for e in queue.events if e.get("type") == "send_message"]
|
||||
assert len(later) == 1
|
||||
assert later[0]["delivery_id"] == f"cascade:cr1:{second['request_id']}"
|
||||
|
|
@ -630,7 +630,7 @@ def test_stale_sweep_failures_do_not_veto_a_converged_tree(qenv, monkeypatch):
|
|||
|
||||
monkeypatch.setattr(qenv.q, "cancel_task_custody", _concurrently_settled)
|
||||
|
||||
assert qenv.tl.cancel_task_by_id("st-root", cascade=True) is True
|
||||
assert qenv.tl.cancel_task_by_id("st-root") is True
|
||||
assert delivered == ["st-root"], "one summary; no 503 over a settled tree"
|
||||
assert ci.active_intent(qenv.drive, "st-root") is None
|
||||
|
||||
|
|
|
|||
|
|
@ -101,7 +101,7 @@ def test_gr4_1_cascade_summary_registration_failure_keeps_the_intent(qenv, monke
|
|||
ci.request_cancel(qenv.drive, "cs1", scope=ci.SCOPE_CASCADE, allow_settled_target=True)
|
||||
store = _corrupt_delivery_registry(qenv.drive)
|
||||
|
||||
assert qenv.tl.cancel_task_by_id("cs1", cascade=True) is True, (
|
||||
assert qenv.tl.cancel_task_by_id("cs1") is True, (
|
||||
"the teardown itself is not gated on the registry"
|
||||
)
|
||||
row = ci.active_intent(qenv.drive, "cs1")
|
||||
|
|
@ -114,7 +114,7 @@ def test_gr4_1_cascade_summary_registration_failure_keeps_the_intent(qenv, monke
|
|||
|
||||
# Healed registry: the watchdog replay owes the summary and settles.
|
||||
store.unlink()
|
||||
assert qenv.tl.cancel_task_by_id("cs1", cascade=True) is True
|
||||
assert qenv.tl.cancel_task_by_id("cs1") is True
|
||||
assert ci.active_intent(qenv.drive, "cs1") is None, "healed registry converges"
|
||||
|
||||
|
||||
|
|
@ -223,7 +223,7 @@ def test_gr4_2_same_intent_replay_dedups_by_the_per_intent_delivery_id(qenv, mon
|
|||
|
||||
monkeypatch.setattr("ouroboros.cancel_intents.settle_intent", _crashy)
|
||||
|
||||
assert qenv.tl.cancel_task_by_id("cr2", cascade=True) is True
|
||||
assert qenv.tl.cancel_task_by_id("cr2") is True
|
||||
sends = [e for e in queue.events if e.get("type") == "send_message"]
|
||||
assert [e["delivery_id"] for e in sends] == [did]
|
||||
assert ci.active_intent(qenv.drive, "cr2") is not None, "crash shape: intent open"
|
||||
|
|
@ -231,7 +231,7 @@ def test_gr4_2_same_intent_replay_dedups_by_the_per_intent_delivery_id(qenv, mon
|
|||
# Confirmed send → the same-intent replay delivers NOTHING new and settles.
|
||||
td.register_delivery(qenv.drive, did)
|
||||
queue.events.clear()
|
||||
assert qenv.tl.cancel_task_by_id("cr2", cascade=True) is True
|
||||
assert qenv.tl.cancel_task_by_id("cr2") is True
|
||||
assert [e for e in queue.events if e.get("type") == "send_message"] == []
|
||||
assert ci.active_intent(qenv.drive, "cr2") is None
|
||||
|
||||
|
|
@ -241,7 +241,7 @@ def test_gr4_2_same_intent_replay_dedups_by_the_per_intent_delivery_id(qenv, mon
|
|||
)
|
||||
assert second["request_id"] != minted["request_id"]
|
||||
queue.events.clear()
|
||||
assert qenv.tl.cancel_task_by_id("cr2", cascade=True) is True
|
||||
assert qenv.tl.cancel_task_by_id("cr2") is True
|
||||
later = [e for e in queue.events if e.get("type") == "send_message"]
|
||||
assert [e["delivery_id"] for e in later] == [f"cascade:cr2:{second['request_id']}"]
|
||||
|
||||
|
|
|
|||
|
|
@ -204,8 +204,12 @@ def test_gr6_1_every_ingress_passes_the_live_ownership_fact():
|
|||
|
||||
gw = (REPO_ROOT / "ouroboros" / "gateway" / "tasks.py").read_text(encoding="utf-8")
|
||||
assert "task_has_live_ownership as _live_ownership" in gw
|
||||
assert "allow_settled=live_own" in gw
|
||||
assert "allow_settled_target=bool(cascade_scope or allow_settled)" in gw
|
||||
assert "allow_settled_target=bool(cascade or live_own)" in gw # graceful ingress
|
||||
# HTTP Stop: a target the intent request found settled is re-requested with
|
||||
# the live-ownership fact (cascade: also its live subtree).
|
||||
assert "return task_has_live_ownership(task_id) or (cascade and task_subtree_is_live(task_id))" in gw
|
||||
assert ('if intent.get("already_settled") and still_owned():\n'
|
||||
' intent = record(allow_settled_target=True)') in gw
|
||||
|
||||
qt = (REPO_ROOT / "supervisor" / "queue_transitions.py").read_text(encoding="utf-8")
|
||||
assert qt.count("allow_settled_target=task_has_live_ownership(task_id)") == 2, (
|
||||
|
|
|
|||
|
|
@ -4,9 +4,10 @@ GR7-1 ingress gaps of the GR6-1 live-ownership predicate: (a) the worker-side
|
|||
queue-snapshot twin fails OPEN toward liveness — a missing, unreadable,
|
||||
or STALE snapshot cannot prove a dead worker, so the agent cancel tool
|
||||
mints the intent instead of answering "Nothing to cancel" over a
|
||||
burning worker; (b) the HTTP cascade endpoint's 404 pre-check consults
|
||||
the same live-ownership predicate — a settled-but-LIVE root proceeds to
|
||||
mint + custody, a settled-AND-dead tree keeps the 404 envelope;
|
||||
burning worker; (b) the HTTP cascade Stop asks the same live-ownership
|
||||
predicate once its intent request found the root settled — a
|
||||
settled-but-LIVE root mints + runs custody, a settled-AND-dead tree
|
||||
keeps the 404 envelope;
|
||||
GR7-2 verbatim preservation order in the settled-capture kill lane: the
|
||||
settled short-circuit runs ABOVE every mutating step (child copy-back,
|
||||
artifact finalize, memory export) — the stored terminal row survives
|
||||
|
|
@ -174,7 +175,7 @@ def test_gr7_1a_cancel_tool_mints_the_intent_on_missing_and_stale_snapshots(
|
|||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# GR7-1b — the HTTP cascade 404 pre-check consults live ownership
|
||||
# GR7-1b — a settled HTTP cascade target consults live ownership
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
|
|
@ -190,28 +191,40 @@ def _client(tmp_path):
|
|||
|
||||
def test_gr7_1b_cascade_proceeds_for_a_settled_but_live_root(qenv, monkeypatch):
|
||||
"""A settled root still holding its RUNNING row (worker burning post-task
|
||||
cognition) used to 404 BEFORE the intent mint. It must mint + run custody."""
|
||||
import ouroboros.gateway.tasks as tasks_mod
|
||||
cognition) used to 404 BEFORE the intent mint. It must mint + run custody:
|
||||
the worker dies, the stored answer survives."""
|
||||
from supervisor import workers
|
||||
|
||||
monkeypatch.setattr(workers, "get_event_q", lambda *, create=True: _CaptureQueue(), raising=False)
|
||||
monkeypatch.setattr(qenv.q, "_emit_cancel_task_done", lambda *a, **kw: None)
|
||||
state = _install_live_worker(qenv, monkeypatch, "root7")
|
||||
write_task_result(qenv.drive, "root7", STATUS_COMPLETED, chat_id=3,
|
||||
result="settled while the worker burns")
|
||||
qenv.q.RUNNING["root7"] = {"task": {"id": "root7", "chat_id": 3,
|
||||
"root_task_id": "root7"}}
|
||||
"root_task_id": "root7"}, "worker_id": 0}
|
||||
seen: dict = {}
|
||||
real_custody = qenv.q.cancel_task_custody
|
||||
|
||||
def _capture_teardown(task_id):
|
||||
def _custody_sees_the_intent(task_id, **kwargs):
|
||||
seen.update(ci.active_intent(qenv.drive, task_id) or {})
|
||||
return True
|
||||
return real_custody(task_id, **kwargs)
|
||||
|
||||
monkeypatch.setattr(tasks_mod, "_run_cascade_cancel", _capture_teardown)
|
||||
monkeypatch.setattr(qenv.q, "cancel_task_custody", _custody_sees_the_intent)
|
||||
with _client(qenv.drive) as client:
|
||||
response = client.post("/api/tasks/root7/cancel", json={"cascade": True})
|
||||
|
||||
assert response.status_code == 200, response.text
|
||||
assert response.json() == {"ok": True, "task_id": "root7", "cascade": True}
|
||||
assert seen.get("scope") == "cascade", (
|
||||
"GR7-1b: the settled-but-live root proceeds to mint + custody"
|
||||
)
|
||||
assert seen.get("source") == "http_cascade"
|
||||
assert not state["alive"], "the still-spending worker is killed"
|
||||
stored = load_task_result(qenv.drive, "root7")
|
||||
# The kill is about the process, never the result (the cascade's one
|
||||
# summary receipt rides the root row beside it).
|
||||
assert (stored["status"], stored["result"]) == (STATUS_COMPLETED, "settled while the worker burns")
|
||||
assert ci.active_intent(qenv.drive, "root7") is None
|
||||
|
||||
|
||||
def test_gr7_1b_settled_and_dead_root_keeps_the_404_envelope(qenv):
|
||||
|
|
|
|||
|
|
@ -185,16 +185,23 @@ def test_owner_visible_incidents_use_canonical_message_seam() -> None:
|
|||
assert "bridge.send_message(" not in source
|
||||
|
||||
|
||||
def test_cancel_failure_is_progress_incident_not_chat_bubble(monkeypatch) -> None:
|
||||
def test_cancel_failure_wakes_the_intent_driver_and_says_nothing(monkeypatch) -> None:
|
||||
"""The cancel_task event is a wake for the durable intent's driver. A FAILED
|
||||
outcome used to send a "❌ cancel … did not settle" row to the owner chat;
|
||||
seven of the eight phantom cancelled-subagent cards in Main were that row,
|
||||
drained 23–58 minutes after the watchdog had already settled the intent.
|
||||
The intent stays open for the watchdog; the card and supervisor.jsonl are
|
||||
the record, so the event publishes nothing for any outcome."""
|
||||
import supervisor.queue as q
|
||||
from supervisor.events_runtime_controls import _drive_cancel_task_event as _handle_cancel_task
|
||||
|
||||
sent = []
|
||||
# Phase A: the handler drives the TYPED custody outcome directly (the boolean
|
||||
# facade collapsed already_settled into a false "✅ cancel").
|
||||
seen = []
|
||||
monkeypatch.setattr(
|
||||
q, "cancel_task_custody",
|
||||
lambda task_id, **_kw: q.CANCEL_FAILED if task_id == "cancel-me" else q.CANCEL_NOT_FOUND,
|
||||
q, "drive_cancel_intent_scope",
|
||||
lambda task_id, **_kw: seen.append(task_id) or (
|
||||
q.CANCEL_FAILED if task_id == "cancel-me" else q.CANCEL_NOT_FOUND
|
||||
),
|
||||
)
|
||||
|
||||
class _Ctx:
|
||||
|
|
@ -208,32 +215,22 @@ def test_cancel_failure_is_progress_incident_not_chat_bubble(monkeypatch) -> Non
|
|||
|
||||
_handle_cancel_task({"task_id": "cancel-me"}, _Ctx())
|
||||
|
||||
assert len(sent) == 1
|
||||
args, kwargs = sent[0]
|
||||
assert args[0] == 9
|
||||
assert args[1].startswith("❌ cancel cancel-me")
|
||||
assert "watchdog" in args[1] # the intent stays open and is retried
|
||||
assert kwargs == {
|
||||
"role": "system", "system_type": "cancellation_notice",
|
||||
"is_progress": True,
|
||||
"task_id": "cancel-me",
|
||||
"progress_meta": {
|
||||
"task_incident": "cancellation_fault",
|
||||
"toast_once": "cancel-me:cancellation_fault",
|
||||
},
|
||||
}
|
||||
assert seen == ["cancel-me"]
|
||||
assert sent == []
|
||||
|
||||
|
||||
def test_cancel_event_keeps_requested_retry_id_for_owner_presentation(monkeypatch) -> None:
|
||||
def test_cancel_event_drives_the_physical_retry_id_without_owner_presentation(monkeypatch) -> None:
|
||||
"""The event names the physical id the tool resolved; the logical
|
||||
``requested_task_id`` used to shape an owner notice and now shapes nothing."""
|
||||
import supervisor.queue as q
|
||||
from supervisor.events_runtime_controls import _drive_cancel_task_event as _handle_cancel_task
|
||||
|
||||
captured = []
|
||||
monkeypatch.setattr(
|
||||
q,
|
||||
"cancel_task_custody",
|
||||
"drive_cancel_intent_scope",
|
||||
lambda task_id, **_kwargs: (
|
||||
captured.append(("custody", task_id)) or q.CANCEL_FAILED
|
||||
captured.append(("driver", task_id)) or q.CANCEL_FAILED
|
||||
),
|
||||
)
|
||||
|
||||
|
|
@ -253,16 +250,7 @@ def test_cancel_event_keeps_requested_retry_id_for_owner_presentation(monkeypatc
|
|||
_Ctx(),
|
||||
)
|
||||
|
||||
assert captured[0] == ("custody", "physical-retry-7f3a")
|
||||
args, kwargs = captured[1]
|
||||
assert args[1].startswith("❌ cancel stable-root")
|
||||
assert "physical-retry-7f3a" not in args[1]
|
||||
assert kwargs["task_id"] == "stable-root"
|
||||
assert kwargs["progress_meta"] == {
|
||||
"task_incident": "cancellation_fault",
|
||||
"toast_once": "stable-root:cancellation_fault",
|
||||
"cancel_physical_task_id": "physical-retry-7f3a",
|
||||
}
|
||||
assert captured == [("driver", "physical-retry-7f3a")]
|
||||
|
||||
|
||||
def test_routine_cancel_outcomes_publish_nothing_to_the_owner_chat(monkeypatch) -> None:
|
||||
|
|
@ -271,16 +259,16 @@ def test_routine_cancel_outcomes_publish_nothing_to_the_owner_chat(monkeypatch)
|
|||
acknowledgement for cancelled / already-settled / not-found reached the
|
||||
GLOBAL owner chat as an untyped assistant row with no task identity — and
|
||||
for a pending-drop race the "had already finished" wording misstated the
|
||||
causal history. Those three outcomes now publish nothing; only FAILED keeps
|
||||
its typed progress incident (the test above)."""
|
||||
causal history. The event is a wake for ``drive_cancel_intent_scope`` and
|
||||
publishes nothing for any outcome, FAILED included (the test above)."""
|
||||
import supervisor.queue as q
|
||||
from supervisor.events_runtime_controls import _drive_cancel_task_event as _handle_cancel_task
|
||||
|
||||
for outcome in (q.CANCEL_CANCELLED, q.CANCEL_ALREADY_SETTLED, q.CANCEL_NOT_FOUND):
|
||||
for outcome in (q.CANCEL_CANCELLED, q.CANCEL_ALREADY_SETTLED, q.CANCEL_NOT_FOUND, q.CANCEL_FAILED):
|
||||
sent = []
|
||||
seen = []
|
||||
monkeypatch.setattr(
|
||||
q, "cancel_task_custody",
|
||||
q, "drive_cancel_intent_scope",
|
||||
lambda task_id, **_kw: seen.append(task_id) or outcome,
|
||||
)
|
||||
|
||||
|
|
@ -299,9 +287,9 @@ def test_routine_cancel_outcomes_publish_nothing_to_the_owner_chat(monkeypatch)
|
|||
assert seen == ["child-A"], outcome # custody still runs
|
||||
assert sent == [], (outcome, sent) # no Main-chat speech, no untyped row
|
||||
|
||||
# A missing task id never reaches custody and still says nothing.
|
||||
# A missing task id never reaches the driver and still says nothing.
|
||||
sent = []
|
||||
monkeypatch.setattr(q, "cancel_task_custody", lambda task_id, **_kw: q.CANCEL_CANCELLED)
|
||||
monkeypatch.setattr(q, "drive_cancel_intent_scope", lambda task_id, **_kw: q.CANCEL_CANCELLED)
|
||||
|
||||
class _Ctx2:
|
||||
@staticmethod
|
||||
|
|
@ -314,7 +302,7 @@ def test_routine_cancel_outcomes_publish_nothing_to_the_owner_chat(monkeypatch)
|
|||
|
||||
_handle_cancel_task({"task_id": ""}, _Ctx2())
|
||||
assert sent == []
|
||||
# The handler no longer carries the prose that leaked: source pin.
|
||||
# The handler no longer carries the prose that leaked, nor a send at all: source pin.
|
||||
src = inspect.getsource(_handle_cancel_task)
|
||||
for leaked in ("teardown confirmed", "had already finished", "no such live task"):
|
||||
for leaked in ("teardown confirmed", "had already finished", "no such live task", "send_with_budget("):
|
||||
assert leaked not in src
|
||||
|
|
|
|||
|
|
@ -11,6 +11,7 @@ import os
|
|||
from ouroboros.gateway import history_paging
|
||||
from ouroboros.gateway.history_segments import SegmentSummary, room_segment_lens, segment_summary
|
||||
from tests.test_chat_history_paging import isolated_runtime, pages, request, row, write # noqa: F401
|
||||
from tests._room_shared import RoomsFromBindings
|
||||
|
||||
|
||||
def foreign_archives(root, stamp, count=3):
|
||||
|
|
@ -110,9 +111,9 @@ def test_the_read_ceiling_bounds_each_request_but_loses_nothing(tmp_path, monkey
|
|||
|
||||
|
||||
def test_main_and_rooms_sharing_its_lens_never_skip_an_archive():
|
||||
assert room_segment_lens(1, {7}, [], {"task": 7}) is None
|
||||
assert room_segment_lens(42, {7}, [], {"task": 7}) is None
|
||||
may_hold = room_segment_lens(7, {7}, [], {"task": 7, "other": 8})
|
||||
assert room_segment_lens(1, {7}, [], RoomsFromBindings({"task": 7})) is None
|
||||
assert room_segment_lens(42, {7}, [], RoomsFromBindings({"task": 7})) is None
|
||||
may_hold = room_segment_lens(7, {7}, [], RoomsFromBindings({"task": 7, "other": 8}))
|
||||
assert may_hold(SegmentSummary(frozenset({7}), frozenset(), frozenset()))
|
||||
assert may_hold(SegmentSummary(frozenset({1}), frozenset({"task"}), frozenset()))
|
||||
assert not may_hold(SegmentSummary(frozenset({1, 8}), frozenset({"other"}), frozenset()))
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@ from types import SimpleNamespace
|
|||
import pytest
|
||||
|
||||
from ouroboros.gateway import history
|
||||
from ouroboros.gateway.state import _chat_activities_snapshot_safe as _real_census # before the autouse stub
|
||||
from ouroboros.store_import import ensure_current_store
|
||||
from ouroboros.tool_call_log import replay_evidence_for_tasks
|
||||
|
||||
|
|
@ -373,6 +374,138 @@ def test_a_main_rooted_tasks_own_main_notice_still_carries_its_tools(tmp_path):
|
|||
assert [(row["task_id"], row["system_type"]) for row in rows] == [("root", "main_notice"), ("root", "task_evidence")]
|
||||
|
||||
|
||||
# --- A4 read seam and A5 G2: the room rule in history, and the in-room parent's carrier ------------
|
||||
|
||||
def _converted_root_with_child(root, *, child_status="cancelled", parent_status="completed", row_chat=1, lineage=False):
|
||||
"""A root converted by «Turn into project», its child, and the child's progress row as the
|
||||
supervisor recorded it (Main, task id only) unless ``lineage``/``row_chat`` say otherwise."""
|
||||
from ouroboros.projects_registry import bind_task_to_project, create_project
|
||||
from ouroboros.task_results import write_task_result
|
||||
|
||||
project = create_project(root, "room", name="Project room")
|
||||
write_task_result(root, "par", parent_status, root_task_id="par", chat_id=1)
|
||||
bind_task_to_project(root, "par", project["id"], origin={"absent": "system"})
|
||||
write_task_result(root, "kid", child_status, delegation_role="subagent", parent_task_id="par",
|
||||
root_task_id="par", chat_id=1)
|
||||
row = {"ts": "2026-10-14T12:10:20Z", "type": "send_message", "task_id": "kid", "is_progress": True,
|
||||
"direction": "out", "chat_id": row_chat, "text": "Stopped", "content": "Stopped", "role": "system",
|
||||
"system_type": "subagent_terminal_notice", "subagent_event": child_status, "narration": False,
|
||||
**({"delegation_role": "subagent", "parent_task_id": "par", "root_task_id": "par"} if lineage else {})}
|
||||
_write(root / "logs/progress.jsonl", [row])
|
||||
return project
|
||||
|
||||
|
||||
def test_a_recorded_phantom_row_leaves_main_and_appears_in_its_project(tmp_path):
|
||||
project = _converted_root_with_child(tmp_path)
|
||||
assert [row["task_id"] for row in _room_read(tmp_path, 1)["messages"]] == [], "Main keeps no Project child"
|
||||
rows = _room_read(tmp_path, project["chat_id"])["messages"]
|
||||
child = next(row for row in rows if row["task_id"] == "kid")
|
||||
assert (child["parent_task_id"], child["delegation_role"]) == ("par", "subagent"), "replay recovers the lineage"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("parent_status, terminal", [("completed", "completed"), ("cancelled", "cancelled"), ("scheduled", None)])
|
||||
def test_a_zero_tool_in_room_parent_is_carried_at_its_childs_row_with_only_a_real_terminal(tmp_path, parent_status, terminal):
|
||||
project = _converted_root_with_child(tmp_path, parent_status=parent_status)
|
||||
rows = _room_read(tmp_path, project["chat_id"])["messages"]
|
||||
[carrier] = [row for row in rows if row["task_id"] == "par"]
|
||||
assert carrier["system_type"] == "task_evidence" and carrier["ts"] == "2026-10-14T12:10:20Z"
|
||||
assert carrier["tool_evidence"]["observations"] == [] and "tool_calls" not in carrier
|
||||
assert carrier.get("task_terminal_status") == terminal, "a scheduled parent is never finished as Done"
|
||||
if terminal:
|
||||
assert carrier["outcome_phase"] == ("cancelled" if terminal == "cancelled" else "done")
|
||||
else:
|
||||
assert carrier["task_phase"] == "unfinished"
|
||||
|
||||
|
||||
def test_a_main_parent_with_zero_tools_is_carried_for_its_child_in_main(tmp_path):
|
||||
from ouroboros.task_results import write_task_result
|
||||
|
||||
write_task_result(tmp_path, "mpar", "completed", root_task_id="mpar", chat_id=1)
|
||||
write_task_result(tmp_path, "mkid", "completed", delegation_role="subagent", parent_task_id="mpar",
|
||||
root_task_id="mpar", chat_id=1)
|
||||
_write(tmp_path / "logs/progress.jsonl", [{"ts": TS, "type": "send_message", "task_id": "mkid", "is_progress": True,
|
||||
"chat_id": 1, "content": "Child finished", "role": "system", "system_type": "subagent_terminal_notice",
|
||||
"subagent_event": "completed", "delegation_role": "subagent", "parent_task_id": "mpar", "root_task_id": "mpar"}])
|
||||
[carrier] = [row for row in _room_read(tmp_path, 1)["messages"] if row["task_id"] == "mpar"]
|
||||
assert (carrier["system_type"], carrier["ts"], carrier["task_terminal_status"]) == ("task_evidence", TS, "completed")
|
||||
|
||||
|
||||
def test_a_notice_naming_a_child_anchors_no_parent(tmp_path):
|
||||
from ouroboros.task_results import write_task_result
|
||||
|
||||
write_task_result(tmp_path, "npar", "completed", root_task_id="npar", chat_id=1)
|
||||
write_task_result(tmp_path, "nkid", "running", delegation_role="subagent", parent_task_id="npar",
|
||||
root_task_id="npar", chat_id=1)
|
||||
_write(tmp_path / "logs/chat.jsonl", [{"direction": "system", "type": "steer_not_delivered", "task_id": "nkid",
|
||||
"text": "Not delivered", "ts": TS, "chat_id": 1}])
|
||||
assert [row["task_id"] for row in _room_read(tmp_path, 1)["messages"]] == ["nkid"]
|
||||
|
||||
|
||||
def test_an_empty_ts_census_anchor_with_zero_tools_gets_no_carrier(tmp_path, monkeypatch):
|
||||
from ouroboros.task_results import write_task_result
|
||||
|
||||
write_task_result(tmp_path, "live", "running", root_task_id="live", chat_id=1)
|
||||
monkeypatch.setattr("ouroboros.gateway.state._chat_activities_snapshot_safe", lambda *_: [
|
||||
{"activity_id": "live", "chat_id": 1, "kind": "managed_task", "phase": "working"}])
|
||||
_write(tmp_path / "logs/chat.jsonl", [{"direction": "in", "text": "Latest question", "ts": TS, "chat_id": 1}])
|
||||
assert [row["text"] for row in _read(tmp_path)["messages"]] == ["Latest question"]
|
||||
|
||||
|
||||
def test_the_history_cursor_survives_child_admission_and_changes_on_conversion(tmp_path):
|
||||
from ouroboros.gateway.history_paging import room_view_fingerprint
|
||||
from ouroboros.projects_registry import all_task_bindings, bind_task_to_project, create_project
|
||||
from ouroboros.task_results import write_task_result
|
||||
|
||||
project = create_project(tmp_path, "room", name="Project room")
|
||||
write_task_result(tmp_path, "root", "running", root_task_id="root", chat_id=1)
|
||||
bind_task_to_project(tmp_path, "root", project["id"], origin={"absent": "system"})
|
||||
view = lambda: room_view_fingerprint(1, {project["chat_id"]}, [], all_task_bindings(tmp_path)) # noqa: E731
|
||||
before = view()
|
||||
write_task_result(tmp_path, "kid", "running", delegation_role="subagent", parent_task_id="root",
|
||||
root_task_id="root", chat_id=1)
|
||||
assert view() == before, "a child's admission keeps every open Main cursor"
|
||||
other = create_project(tmp_path, "other", name="Other room")
|
||||
bind_task_to_project(tmp_path, "kid", other["id"], origin={"absent": "system"})
|
||||
assert view() != before, "a conversion changes what Main shows"
|
||||
|
||||
|
||||
def test_a_project_rooms_cursor_changes_when_a_conversion_moves_its_rows_away(tmp_path):
|
||||
"""A task's resolved room outranks the room its rows were written to, so binding a
|
||||
task of room A to room B changes what A shows: A's open cursor must not survive."""
|
||||
from ouroboros.gateway.history_paging import room_view_fingerprint
|
||||
from ouroboros.projects_registry import all_task_bindings, bind_task_to_project, create_project
|
||||
from ouroboros.task_results import write_task_result
|
||||
|
||||
room_a = create_project(tmp_path, "room-a", name="Room A")
|
||||
write_task_result(tmp_path, "root-a", "running", root_task_id="root-a", chat_id=room_a["chat_id"])
|
||||
write_task_result(tmp_path, "kid-a", "running", delegation_role="subagent", parent_task_id="root-a",
|
||||
root_task_id="root-a", chat_id=room_a["chat_id"])
|
||||
view = lambda: room_view_fingerprint( # noqa: E731
|
||||
room_a["chat_id"], {room_a["chat_id"]}, [], all_task_bindings(tmp_path))
|
||||
before = view()
|
||||
write_task_result(tmp_path, "kid-a2", "running", delegation_role="subagent", parent_task_id="root-a",
|
||||
root_task_id="root-a", chat_id=room_a["chat_id"])
|
||||
assert view() == before, "a child's admission keeps room A's open cursor"
|
||||
room_b = create_project(tmp_path, "room-b", name="Room B")
|
||||
bind_task_to_project(tmp_path, "kid-a", room_b["id"], origin={"absent": "system"})
|
||||
assert view() != before, "moving kid-a's rows to room B changes what room A shows"
|
||||
|
||||
|
||||
def test_census_rows_carry_started_at_and_never_ts(tmp_path, monkeypatch):
|
||||
"""The carrier rule tells a parent anchor (its child's ts) from a census anchor by the ts alone:
|
||||
the real census producer stamps ``started_at`` on every row and never a ``ts``."""
|
||||
from supervisor import queue
|
||||
|
||||
monkeypatch.setattr(queue, "PENDING", [{"id": "queued-root", "chat_id": 1, "root_task_id": "queued-root",
|
||||
"ts": TS, "queued_at": TS}])
|
||||
monkeypatch.setattr(queue, "RUNNING", {"running-root": {"task": {"id": "running-root", "chat_id": 1,
|
||||
"root_task_id": "running-root", "ts": TS},
|
||||
"started_at": 1_700_000_000.0}})
|
||||
rows = _real_census(tmp_path, {}, direct_turns=[])
|
||||
assert {row["activity_id"] for row in rows} == {"queued-root", "running-root"}
|
||||
assert all("ts" not in row and "started_at" in row for row in rows), rows
|
||||
|
||||
|
||||
@pytest.mark.parametrize("status", ["completed", "failed"])
|
||||
def test_answered_root_with_a_paused_late_phase_is_not_readdressed_at_the_page_end(tmp_path, monkeypatch, status):
|
||||
# D10: the answer was delivered; only its post-task work waits under the owner's Pause.
|
||||
|
|
|
|||
|
|
@ -161,7 +161,6 @@ HOST_ROW_CLASSES = {
|
|||
"acceptance_late_settlement": ("card_row", 1),
|
||||
"budget_notice": ("runtime_state", 1),
|
||||
"cancel_receipt": ("owner_act_receipt", 2),
|
||||
"cancellation_notice": ("card_row", 1),
|
||||
"command_reply": ("owner_act_receipt", 2),
|
||||
"consciousness_notice": ("runtime_state", 1),
|
||||
"deep_self_review_queued": ("owner_act_receipt", 1),
|
||||
|
|
|
|||
|
|
@ -162,7 +162,7 @@ def test_native_post_task_wait_remains_addressable_after_dialogue_closes(phase,
|
|||
return reason
|
||||
|
||||
# Keep paid post-task custody alive until HTTP reports its
|
||||
# existing still-live response; this is a schedule, not a fake stop.
|
||||
# pending response; this is a schedule, not a fake stop.
|
||||
monkeypatch.setattr(owner, "control_reason", hold_observed_stop)
|
||||
else:
|
||||
real_request_cancel = cancel_intents.request_cancel
|
||||
|
|
@ -170,6 +170,8 @@ def test_native_post_task_wait_remains_addressable_after_dialogue_closes(phase,
|
|||
def settle_after_durable_stop(*args, **kwargs):
|
||||
assert post_task_model_wait(f.root, task_id) is owner
|
||||
intent = real_request_cancel(*args, **kwargs)
|
||||
if intent.get("already_settled") and not kwargs.get("allow_settled_target"):
|
||||
return intent # the settled answer asks live ownership before it mints
|
||||
assert intent["request_id"] and intent["state"] == "requested"
|
||||
assert intent["source"] == "http_single"
|
||||
assert not intent.get("already_settled")
|
||||
|
|
@ -187,7 +189,7 @@ def test_native_post_task_wait_remains_addressable_after_dialogue_closes(phase,
|
|||
monkeypatch.setattr(cancel_intents, "request_cancel", settle_after_durable_stop)
|
||||
with _client(f.root) as client:
|
||||
response = client.post(f"/api/tasks/{task_id}/cancel", json={"stop_policy": "immediate"})
|
||||
assert response.status_code == (503 if action == "stop" else 404), response.json()
|
||||
assert response.status_code == (202 if action == "stop" else 404), response.json()
|
||||
if action == "stop":
|
||||
assert stop_observed.wait(5)
|
||||
assert post_task_model_wait(f.root, task_id) is owner and not owner.closed
|
||||
|
|
|
|||
|
|
@ -960,7 +960,7 @@ def test_corrupt_retry_lineage_holds_until_grace_deadline_then_releases_hard_pat
|
|||
now=ostop._requested_ts(intent) + OWNER_STOP_OUTER_CAP_SEC + 1.0,
|
||||
)
|
||||
assert after == {root_id: lifecycle.CANCEL_CANCELLED}
|
||||
assert hard_calls == [(root_id, {"cascade": True})]
|
||||
assert hard_calls == [(root_id, {})]
|
||||
|
||||
|
||||
def test_sweep_feeds_custody_for_settled_pending_or_expired_roots(tmp_path, monkeypatch):
|
||||
|
|
|
|||
|
|
@ -797,6 +797,9 @@ def test_cancelling_a_subagent_rescues_its_partial_result(monkeypatch, tmp_path)
|
|||
monkeypatch.setattr(
|
||||
observability, "latest_llm_response_text", lambda root, tid: "half-written answer",
|
||||
)
|
||||
from ouroboros.cancel_intents import request_cancel
|
||||
|
||||
request_cancel(tmp_path, "live-salvage", allow_settled_target=True)
|
||||
|
||||
assert q.cancel_task_custody("live-salvage") == q.CANCEL_CANCELLED
|
||||
result = load_task_result(tmp_path, "live-salvage")
|
||||
|
|
@ -902,6 +905,9 @@ def test_cancelling_a_subagent_preserves_the_full_output_on_the_canonical_drive(
|
|||
},
|
||||
}, raising=False)
|
||||
monkeypatch.setattr(q, "_emit_cancel_task_done", lambda *a, **k: None)
|
||||
from ouroboros.cancel_intents import request_cancel
|
||||
|
||||
request_cancel(tmp_path, task_id, allow_settled_target=True)
|
||||
|
||||
assert q.cancel_task_custody(task_id) == q.CANCEL_CANCELLED
|
||||
|
||||
|
|
|
|||
|
|
@ -59,11 +59,10 @@ def isolated_project_queue(tmp_path, monkeypatch):
|
|||
lambda project_id, chat_id: broadcasts.append((project_id, chat_id)),
|
||||
)
|
||||
|
||||
def cancel_task(task_id: str, *, cascade: bool = False) -> bool:
|
||||
def cancel_task(task_id: str) -> bool:
|
||||
# GR5-5: the deletion cascades lineage ROOTS only, so the fake must
|
||||
# tear down the whole subtree the real cascade would (root + every
|
||||
# task whose recorded root is the cascaded id).
|
||||
assert cascade is True
|
||||
cancelled.append(task_id)
|
||||
removed = False
|
||||
for task in list(pending):
|
||||
|
|
|
|||
|
|
@ -154,5 +154,6 @@ def test_agent_producers_publish_one_main_receipt_and_none_for_project_origins(t
|
|||
def test_handoff_is_a_main_only_history_row(tmp_path):
|
||||
from ouroboros.project_dialogue import room_membership
|
||||
row = {"type": "project_handoff", "task_id": "root"}
|
||||
assert room_membership(1, {22}, [], {"root": 22})(1, row)
|
||||
assert not room_membership(22, {22}, [], {"root": 22})(1, row)
|
||||
rooms = lambda task_id, parent="", root="": 22 if task_id == "root" else 0 # noqa: E731
|
||||
assert room_membership(1, {22}, [], rooms)(1, row)
|
||||
assert not room_membership(22, {22}, [], rooms)(1, row)
|
||||
|
|
|
|||
|
|
@ -1445,23 +1445,23 @@ def test_promote_chat_to_task_broadcasts_projects_changed(tmp_path, monkeypatch)
|
|||
assert changed[0]["chat_id"] == project_chat_id("proj-x")
|
||||
|
||||
|
||||
def test_registered_project_chat_ids_recognizes_every_project(tmp_path):
|
||||
def test_reserved_project_chat_ids_recognizes_every_project(tmp_path):
|
||||
"""The isolation SSOT recognizes EVERY registered project's chat_id (regardless
|
||||
of sidebar visibility) so its raw chat never re-leaks into the штаб's main
|
||||
context / dialogue consolidation / background consciousness (BIBLE P1). Sidebar
|
||||
visibility is a separate presentation concern (no project statuses, v6.33.0)."""
|
||||
from ouroboros.projects_registry import (
|
||||
create_project,
|
||||
registered_project_chat_ids,
|
||||
reserved_project_chat_ids,
|
||||
update_project,
|
||||
)
|
||||
|
||||
proj = create_project(tmp_path, "old-racer")
|
||||
chat_id = int(proj["chat_id"])
|
||||
assert chat_id in registered_project_chat_ids(tmp_path)
|
||||
assert chat_id in reserved_project_chat_ids(tmp_path)
|
||||
# A rename (or any mutable-field update) never drops it from the isolation set.
|
||||
update_project(tmp_path, "old-racer", name="Old Racer (renamed)")
|
||||
assert chat_id in registered_project_chat_ids(tmp_path)
|
||||
assert chat_id in reserved_project_chat_ids(tmp_path)
|
||||
|
||||
|
||||
def test_chat_history_tool_spans_all_threads_full_awareness(tmp_path):
|
||||
|
|
|
|||
|
|
@ -288,7 +288,7 @@ def test_recent_room_and_history_share_parent_root_and_origin_membership(runtime
|
|||
from ouroboros.chronicle_store import ChronicleStore
|
||||
from ouroboros.gateway.history import _make_thread_filter
|
||||
from ouroboros.memory_inventory import open_room_rows
|
||||
from ouroboros.projects_registry import bind_task_to_project, create_project
|
||||
from ouroboros.projects_registry import RoomResolver, bind_task_to_project, create_project
|
||||
|
||||
project = create_project(runtime.root, "membership", name="Membership")
|
||||
bind_task_to_project(runtime.root, "parent", project["id"], origin={"absent": "system"})
|
||||
|
|
@ -300,8 +300,8 @@ def test_recent_room_and_history_share_parent_root_and_origin_membership(runtime
|
|||
rows = [meta for _address, meta, _pos in open_room_rows(runtime.root, project["chat_id"])]
|
||||
assert {row["task_id"] for row in rows} == {"child-parent_task_id", "child-root_task_id"}
|
||||
assert open_room_rows(runtime.root, 1) == [] # bound by lineage: not Main's
|
||||
project_filter = _make_thread_filter(project["chat_id"], {project["chat_id"]}, [], {"parent": project["chat_id"]})
|
||||
main_filter = _make_thread_filter(1, {project["chat_id"]}, [], {"parent": project["chat_id"]})
|
||||
project_filter = _make_thread_filter(project["chat_id"], {project["chat_id"]}, [], RoomResolver(runtime.root))
|
||||
main_filter = _make_thread_filter(1, {project["chat_id"]}, [], RoomResolver(runtime.root))
|
||||
assert all(project_filter(row["chat_id"], row) and not main_filter(row["chat_id"], row) for row in rows)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -342,7 +342,8 @@ def test_chat_scrolls_to_bottom_after_first_history_load():
|
|||
assert "ensureLiveCardVisible(record, { reorderExisting: true });" in source, \
|
||||
"A mounted task card must be re-sorted if a later event lowers its anchor"
|
||||
assert "record._anchorOrderDirty = true;" in source
|
||||
assert "reorderDirtyCardIfNeeded(rec);" in source, \
|
||||
assert "if (rec?._anchorOrderDirty && !rec.isSubagent && rec.root?.isConnected) {" in source
|
||||
assert "mountReplayedCard(record);" in source, \
|
||||
"Pass 2 must re-sort connected cards dirtied by routine history sync"
|
||||
assert "const parent = liveCardRecords.get(record.parentGroupId);" in source
|
||||
assert "seen.has(record.groupId)" in source, \
|
||||
|
|
|
|||
|
|
@ -18,14 +18,14 @@ from tests import _memory_inventory_shared as shared
|
|||
|
||||
def _canonical(root):
|
||||
from ouroboros.project_dialogue import room_membership, source_refs_for_project
|
||||
from ouroboros.projects_registry import all_task_bindings, list_reserved_projects
|
||||
from ouroboros.projects_registry import list_reserved_projects, RoomResolver
|
||||
|
||||
projects = {int(project["chat_id"]) for project in list_reserved_projects(root)}
|
||||
bindings = all_task_bindings(root)
|
||||
rooms = RoomResolver(root)
|
||||
|
||||
def member(room: int):
|
||||
refs = source_refs_for_project(root, room) if room in projects else []
|
||||
return room_membership(room, projects, refs, bindings)
|
||||
return room_membership(room, projects, refs, rooms)
|
||||
|
||||
return member
|
||||
|
||||
|
|
|
|||
|
|
@ -174,9 +174,9 @@ def test_origin_addressed_notice_rows_stay_in_the_owners_chat(notice_type):
|
|||
"""R12: the refused task id is bound to the project it never started in, so
|
||||
binding lineage would move the notice out of Main into that project on
|
||||
reload. The notice is addressed to the chat the OWNER wrote in."""
|
||||
bindings = {"refused-task": 77}
|
||||
main = room_membership(1, {77}, [], bindings)
|
||||
project = room_membership(77, {77}, [], bindings)
|
||||
rooms = lambda task_id, parent="", root="": 77 if "refused-task" in (task_id, parent, root) else 0 # noqa: E731
|
||||
main = room_membership(1, {77}, [], rooms)
|
||||
project = room_membership(77, {77}, [], rooms)
|
||||
notice = {"direction": "system", "type": notice_type, "task_id": "refused-task", "chat_id": 1}
|
||||
|
||||
assert main(1, notice) is True
|
||||
|
|
|
|||
|
|
@ -53,9 +53,10 @@ def test_replay_learns_subagent_lineage_before_merging_card_rows():
|
|||
src = _read("web/modules/chat.js")
|
||||
# One helper learns lineage from both replay rows and live final frames.
|
||||
assert "function learnSubagentLineage(msg)" in src
|
||||
assert "for (const msg of messages) learnSubagentLineage(msg);" in src
|
||||
lineage_pass = "for (const msg of messages) {\n learnSubagentLineage(msg);"
|
||||
assert lineage_pass in src
|
||||
history = src[src.index("function applyHistoryMessages"):src.index("async function syncHistory")]
|
||||
assert history.index("for (const msg of messages) learnSubagentLineage(msg);") < history.index("admitCardMetadata(msg)")
|
||||
assert history.index(lineage_pass) < history.index("admitCardMetadata(msg)")
|
||||
# A page may contain a child whose parent is already represented elsewhere.
|
||||
# Keep those bindings during replay; only final instance disposal clears them.
|
||||
for collection in ("subagentChildParents", "subagentTerminalChildren"):
|
||||
|
|
@ -68,9 +69,10 @@ def test_replay_learns_subagent_lineage_before_merging_card_rows():
|
|||
assert fanout.index("learnSubagentLineage(msg);") < fanout.index("routeSubagentFinalMessageToCard(explicitTaskId, msg)")
|
||||
# A lineage-known child is minted as its parent's nested card by whichever
|
||||
# path reaches it first (#636) — no sticky force writer: the parent's anchor
|
||||
# follows the child's frame and the predicate always admits a child block.
|
||||
assert "reanchorTaskCard(getLiveCardRecord(parentId), rawTs);" in src
|
||||
assert "const record = getSubagentCardRecord(childId, parentId, role);" in src
|
||||
# follows the child's frame (when this room holds the parent's own card, A5)
|
||||
# and the predicate always admits a child block.
|
||||
assert "if (rawTs) reanchorTaskCard(liveCardRecords.get(parentId), rawTs);" in src
|
||||
assert "const record = getSubagentCardRecord(childId, parentId, role, rawTs);" in src
|
||||
assert "if (!record || record.isSubagent) return true;" in src
|
||||
# A child is locked terminal from EITHER a terminal subagent event OR a
|
||||
# genuinely-settled server task_terminal_status; interrupted stays retryable.
|
||||
|
|
|
|||
|
|
@ -1,10 +1,12 @@
|
|||
"""HTTP contract for /api/tasks/{id}/cancel with the optional cascade body (v6.82 P5).
|
||||
|
||||
The plain (no-body / empty-body) request keeps the pre-v6.82 single-task
|
||||
envelope and now answers from the same TYPED custody outcome as the cascade;
|
||||
{"cascade": true} pre-checks subtree liveness (404 contract preserved) and runs
|
||||
the WHOLE custody-based teardown before answering with the SAME envelope plus
|
||||
"cascade": true — no background task, failures are honest 503s.
|
||||
Both the plain (no-body / empty-body) request and {"cascade": true} record the
|
||||
durable intent FIRST — no liveness pre-probe gates it — and drive it through
|
||||
the canonical ``drive_cancel_intent_scope`` before answering: 200 with the
|
||||
pre-v6.82 single-task envelope (plus "cascade": true) once the teardown
|
||||
settled, the legacy 404 when there is nothing to cancel, and 202
|
||||
``cancel_state: "pending"`` while the recorded intent stays open for the
|
||||
cancel-intent watchdog. 503 means nothing was recorded.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -92,12 +94,12 @@ def test_cascade_acknowledges_and_cancels_whole_live_subtree(tmp_path, monkeypat
|
|||
{"id": "other", "chat_id": 0, "root_task_id": "other", "depth": 0},
|
||||
],
|
||||
)
|
||||
cascade_calls: list[tuple[str, bool]] = []
|
||||
cascade_calls: list[str] = []
|
||||
real = queue.cancel_task_by_id
|
||||
|
||||
def _spy(task_id, *, cascade=False):
|
||||
cascade_calls.append((task_id, cascade))
|
||||
return real(task_id, cascade=cascade)
|
||||
def _spy(task_id):
|
||||
cascade_calls.append(task_id)
|
||||
return real(task_id)
|
||||
|
||||
monkeypatch.setattr(queue, "cancel_task_by_id", _spy)
|
||||
with _client(tmp_path) as client:
|
||||
|
|
@ -106,7 +108,7 @@ def test_cascade_acknowledges_and_cancels_whole_live_subtree(tmp_path, monkeypat
|
|||
assert response.status_code == 200
|
||||
# Same envelope keys as the plain path, plus the cascade echo.
|
||||
assert response.json() == {"ok": True, "task_id": "root", "cascade": True}
|
||||
assert cascade_calls == [("root", True)]
|
||||
assert cascade_calls == ["root"]
|
||||
assert [task["id"] for task in pending] == ["other"]
|
||||
for task_id in ("root", "child", "grandchild"):
|
||||
assert load_task_result(tmp_path, task_id)["status"] == STATUS_CANCELLED
|
||||
|
|
@ -194,17 +196,25 @@ def test_a_failing_teardown_is_reported_and_recorded_never_swallowed(tmp_path, m
|
|||
[{"id": "root", "chat_id": 0, "root_task_id": "root"}],
|
||||
)
|
||||
|
||||
def _boom(task_id, *, cascade=False):
|
||||
def _boom(task_id):
|
||||
raise RuntimeError("teardown exploded")
|
||||
|
||||
monkeypatch.setattr(queue, "cancel_task_by_id", _boom)
|
||||
with _client(tmp_path) as client:
|
||||
response = client.post("/api/tasks/root/cancel", json={"cascade": True})
|
||||
# The cascade is awaited, so a teardown that explodes is REPORTED (never an
|
||||
# ok:true for a cancellation that did not happen) and is also recorded as a
|
||||
# durable owner-visible incident. The server does not crash either way.
|
||||
assert response.status_code == 503
|
||||
assert "still live" in response.json()["error"]
|
||||
# ok:true cancellation that did not happen): the intent was recorded first,
|
||||
# so the answer is the typed pending Stop the watchdog finishes, and the
|
||||
# failure is also a durable owner-visible incident. The server does not crash.
|
||||
assert response.status_code == 202
|
||||
assert response.json() == {
|
||||
"ok": True, "task_id": "root", "cancel_state": "pending", "stop_policy": "immediate",
|
||||
"request_id": response.json()["request_id"], "reason_code": "teardown_unconfirmed",
|
||||
"cascade": True,
|
||||
}
|
||||
from ouroboros.cancel_intents import active_intent
|
||||
|
||||
assert (active_intent(tmp_path, "root") or {}).get("request_id") == response.json()["request_id"]
|
||||
log_path = tmp_path / "logs" / "supervisor.jsonl"
|
||||
rows = [json.loads(line) for line in log_path.read_text(encoding="utf-8").splitlines() if line.strip()]
|
||||
errors = [row for row in rows if row.get("type") == "task_cancel_cascade_error"]
|
||||
|
|
@ -219,14 +229,12 @@ def test_running_task_counts_as_live_for_cascade_precheck(tmp_path, monkeypatch)
|
|||
"RUNNING",
|
||||
{"root": {"task": {"id": "root", "root_task_id": "root"}}},
|
||||
)
|
||||
seen: list[tuple[str, bool]] = []
|
||||
monkeypatch.setattr(
|
||||
queue, "cancel_task_by_id", lambda task_id, *, cascade=False: seen.append((task_id, cascade)) or True,
|
||||
)
|
||||
seen: list[str] = []
|
||||
monkeypatch.setattr(queue, "cancel_task_by_id", lambda task_id: seen.append(task_id) or True)
|
||||
with _client(tmp_path) as client:
|
||||
response = client.post("/api/tasks/root/cancel", json={"cascade": True})
|
||||
assert response.status_code == 200
|
||||
assert seen == [("root", True)]
|
||||
assert seen == ["root"]
|
||||
|
||||
|
||||
def test_non_boolean_cascade_is_a_client_error_not_a_subtree_cancel(tmp_path, monkeypatch):
|
||||
|
|
@ -326,55 +334,62 @@ def _isolate_cancellation_globals_endpoint(monkeypatch, tmp_path):
|
|||
|
||||
|
||||
def test_cascade_answers_only_after_the_teardown_and_reports_an_unsettled_tree(tmp_path, monkeypatch):
|
||||
"""The cascade is ONE synchronous transaction: the response is sent after the
|
||||
teardown, so a tree that is still live when the teardown gives up is reported
|
||||
as a failure rather than acknowledged as cancelled."""
|
||||
import ouroboros.gateway.tasks as tasks_mod
|
||||
"""The cascade runs before the answer: the response is sent after the
|
||||
teardown, so a tree that is still live when the teardown gives up is never
|
||||
acknowledged as cancelled — it is the typed pending Stop whose recorded
|
||||
intent the watchdog keeps driving."""
|
||||
import supervisor.queue as queue
|
||||
|
||||
order: list[str] = []
|
||||
# GR3-11a: the endpoint consults the liveness pre-check through the
|
||||
# supervisor.queue re-export (the single public import surface).
|
||||
# GR3-11a: the endpoint consults liveness through the supervisor.queue
|
||||
# re-export (the single public import surface); "root" has no durable row,
|
||||
# so a cascade is recorded only while something of it is live.
|
||||
monkeypatch.setattr("supervisor.queue.task_subtree_is_live", lambda tid: True)
|
||||
monkeypatch.setattr(
|
||||
tasks_mod, "_run_cascade_cancel",
|
||||
lambda task_id: (order.append("teardown"), True)[1],
|
||||
queue, "cancel_task_by_id",
|
||||
lambda task_id, **_kw: (order.append("teardown"), True)[1],
|
||||
)
|
||||
with _client(tmp_path) as client:
|
||||
response = client.post("/api/tasks/root/cancel", json={"cascade": True})
|
||||
assert response.status_code == 200
|
||||
assert response.json() == {"ok": True, "task_id": "root", "cascade": True}
|
||||
assert order == ["teardown"], "the teardown ran before the answer"
|
||||
|
||||
monkeypatch.setattr(tasks_mod, "_run_cascade_cancel", lambda task_id: False)
|
||||
monkeypatch.setattr(queue, "cancel_task_by_id", lambda task_id, **_kw: False)
|
||||
with _client(tmp_path) as client:
|
||||
unsettled = client.post("/api/tasks/root/cancel", json={"cascade": True})
|
||||
assert unsettled.status_code == 503
|
||||
assert "still live" in unsettled.json()["error"]
|
||||
assert unsettled.status_code == 202
|
||||
assert unsettled.json()["cancel_state"] == "pending"
|
||||
assert unsettled.json()["reason_code"] == "teardown_unconfirmed"
|
||||
assert unsettled.json()["cascade"] is True
|
||||
|
||||
|
||||
def test_single_task_cancel_reports_a_refusal_instead_of_a_false_404(tmp_path, monkeypatch):
|
||||
def test_single_task_cancel_reports_an_unconfirmed_teardown_as_pending_not_404(tmp_path, monkeypatch):
|
||||
"""A worker that refuses to die is neither cancelled nor absent. The legacy
|
||||
boolean collapsed both into 404 — telling the caller the task is gone while it
|
||||
keeps running. The typed outcome answers 503 for that case and keeps 404 for a
|
||||
task that genuinely is not there."""
|
||||
keeps running. A refused teardown whose recorded intent stays open is the
|
||||
typed pending Stop (202), and 404 stays for a task that genuinely is not there."""
|
||||
import supervisor.queue as q
|
||||
|
||||
monkeypatch.setattr(q, "cancel_task_custody", lambda tid: q.CANCEL_FAILED)
|
||||
monkeypatch.setattr(q, "cancel_task_custody", lambda tid, **_kw: q.CANCEL_FAILED)
|
||||
with _client(tmp_path) as client:
|
||||
refused = client.post("/api/tasks/root/cancel")
|
||||
assert refused.status_code == 503
|
||||
assert "still live" in refused.json()["error"]
|
||||
assert refused.status_code == 202
|
||||
assert refused.json()["cancel_state"] == "pending"
|
||||
assert "cascade" not in refused.json()
|
||||
|
||||
monkeypatch.setattr(q, "cancel_task_custody", lambda tid: q.CANCEL_NOT_FOUND)
|
||||
monkeypatch.setattr(q, "cancel_task_custody", lambda tid, **_kw: q.CANCEL_NOT_FOUND)
|
||||
with _client(tmp_path) as client:
|
||||
missing = client.post("/api/tasks/root/cancel")
|
||||
assert missing.status_code == 404
|
||||
|
||||
# ...and an already-settled task keeps the legacy INACTIVE answer (404), which
|
||||
# its own dedicated test pins explicitly.
|
||||
monkeypatch.setattr(q, "cancel_task_custody", lambda tid: q.CANCEL_CANCELLED)
|
||||
monkeypatch.setattr(q, "cancel_task_custody", lambda tid, **_kw: q.CANCEL_CANCELLED)
|
||||
with _client(tmp_path) as client:
|
||||
cancelled = client.post("/api/tasks/root/cancel")
|
||||
assert cancelled.status_code == 200
|
||||
assert cancelled.json() == {"ok": True, "task_id": "root"}
|
||||
|
||||
|
||||
def test_plain_cancel_of_an_already_finished_task_keeps_the_legacy_404(tmp_path, monkeypatch):
|
||||
|
|
@ -382,7 +397,7 @@ def test_plain_cancel_of_an_already_finished_task_keeps_the_legacy_404(tmp_path,
|
|||
settled on its own is INACTIVE, which the plain path has always answered 404."""
|
||||
import supervisor.queue as q
|
||||
|
||||
monkeypatch.setattr(q, "cancel_task_custody", lambda tid: q.CANCEL_ALREADY_SETTLED)
|
||||
monkeypatch.setattr(q, "cancel_task_custody", lambda tid, **_kw: q.CANCEL_ALREADY_SETTLED)
|
||||
with _client(tmp_path) as client:
|
||||
response = client.post("/api/tasks/root/cancel")
|
||||
assert response.status_code == 404
|
||||
|
|
@ -393,9 +408,7 @@ def test_cascade_ingress_mints_the_cascade_scope_itself(tmp_path, monkeypatch):
|
|||
``scope=cascade`` at the ingress — even when the supervisor's own
|
||||
``mark_intent_scope`` stamp never runs (the crash-before-stamp window), a
|
||||
watchdog replay of the intent re-runs a CASCADE, never a single cancel."""
|
||||
import ouroboros.gateway.tasks as tasks_mod
|
||||
|
||||
_isolate_queue(monkeypatch, tmp_path, [
|
||||
queue, _pending = _isolate_queue(monkeypatch, tmp_path, [
|
||||
{"id": "root", "chat_id": 0, "root_task_id": "root"},
|
||||
{"id": "kid", "chat_id": 0, "root_task_id": "root", "parent_task_id": "root"},
|
||||
])
|
||||
|
|
@ -404,13 +417,13 @@ def test_cascade_ingress_mints_the_cascade_scope_itself(tmp_path, monkeypatch):
|
|||
lambda *_a, **_kw: False)
|
||||
seen: dict = {}
|
||||
|
||||
def _capture_teardown(task_id):
|
||||
def _capture_teardown(task_id, **_kw):
|
||||
from ouroboros.cancel_intents import active_intent
|
||||
|
||||
seen.update(active_intent(tmp_path, task_id) or {})
|
||||
return True
|
||||
|
||||
monkeypatch.setattr(tasks_mod, "_run_cascade_cancel", _capture_teardown)
|
||||
monkeypatch.setattr(queue, "cancel_task_by_id", _capture_teardown)
|
||||
with _client(tmp_path) as client:
|
||||
response = client.post("/api/tasks/root/cancel", json={"cascade": True})
|
||||
|
||||
|
|
@ -428,7 +441,6 @@ def test_plain_stop_now_executes_an_existing_durable_cascade(tmp_path, monkeypat
|
|||
the existing cascade immediately, never replay it as a narrower single
|
||||
cancellation.
|
||||
"""
|
||||
import ouroboros.gateway.tasks as tasks_mod
|
||||
import supervisor.queue as queue
|
||||
|
||||
from ouroboros import cancel_intents as ci
|
||||
|
|
@ -452,8 +464,8 @@ def test_plain_stop_now_executes_an_existing_durable_cascade(tmp_path, monkeypat
|
|||
)
|
||||
cascades: list[str] = []
|
||||
monkeypatch.setattr(
|
||||
tasks_mod,
|
||||
"_run_cascade_cancel",
|
||||
queue,
|
||||
"cancel_task_by_id",
|
||||
lambda task_id: cascades.append(task_id) or True,
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
|
|
@ -482,26 +494,226 @@ def test_cascade_over_a_settled_root_leaves_a_replayable_intent_on_crash(tmp_pat
|
|||
"""GR2-1b: a cascade over an ALREADY-SETTLED root with live descendants
|
||||
mints a durable cascade coordination intent — after a simulated crash
|
||||
mid-sweep (teardown reports unsettled) that intent survives as the
|
||||
watchdog's replay trigger for the descendants."""
|
||||
import ouroboros.gateway.tasks as tasks_mod
|
||||
|
||||
watchdog's replay trigger for the descendants, and the answer says so."""
|
||||
from ouroboros.cancel_intents import active_intent
|
||||
from ouroboros.task_results import write_task_result
|
||||
|
||||
_isolate_queue(monkeypatch, tmp_path, [
|
||||
queue, _pending = _isolate_queue(monkeypatch, tmp_path, [
|
||||
{"id": "kid", "chat_id": 7, "root_task_id": "root", "parent_task_id": "root"},
|
||||
])
|
||||
write_task_result(tmp_path, "root", "failed", result="root died on budget")
|
||||
write_task_result(tmp_path, "kid", "scheduled")
|
||||
# Crash mid-sweep: the teardown never settles the tree.
|
||||
monkeypatch.setattr(tasks_mod, "_run_cascade_cancel", lambda task_id: False)
|
||||
monkeypatch.setattr(queue, "cancel_task_by_id", lambda task_id, **_kw: False)
|
||||
|
||||
with _client(tmp_path) as client:
|
||||
response = client.post("/api/tasks/root/cancel", json={"cascade": True})
|
||||
|
||||
assert response.status_code == 503, "an unsettled tree is never acknowledged"
|
||||
assert response.status_code == 202, "an unsettled tree is never acknowledged as cancelled"
|
||||
assert response.json()["cancel_state"] == "pending"
|
||||
row = active_intent(tmp_path, "root")
|
||||
assert row is not None and row["scope"] == "cascade", (
|
||||
"the settled-root cascade intent must survive the crash as the "
|
||||
"watchdog's replay trigger for the live descendants"
|
||||
)
|
||||
assert response.json()["request_id"] == row["request_id"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Intent first (no liveness pre-probe); a recorded, unconfirmed Stop is pending
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _supervisor_rows(tmp_path, kind):
|
||||
path = tmp_path / "logs" / "supervisor.jsonl"
|
||||
if not path.exists():
|
||||
return []
|
||||
rows = [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines() if line.strip()]
|
||||
return [row for row in rows if row.get("type") == kind]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cascade", [False, True])
|
||||
def test_unknown_id_is_404_with_no_task_row_and_no_open_intent(tmp_path, monkeypatch, cascade):
|
||||
"""An id that names no task. The single Stop records its intent and
|
||||
custody's miss lane settles it ``not_found`` in the same call; a cascade has
|
||||
no recorded tree to stop and mints nothing. Neither invents a task row,
|
||||
leaves an intent open or runs a cascade over a phantom tree."""
|
||||
from ouroboros.cancel_intents import active_intents
|
||||
from ouroboros.task_results import load_task_result
|
||||
|
||||
_isolate_queue(monkeypatch, tmp_path, [])
|
||||
with _client(tmp_path) as client:
|
||||
response = client.post("/api/tasks/ghost-id/cancel", json={"cascade": cascade})
|
||||
assert response.status_code == 404
|
||||
assert response.json()["error"] == "task not found or not active"
|
||||
assert load_task_result(tmp_path, "ghost-id") is None
|
||||
assert active_intents(tmp_path) == {}
|
||||
events = [(row.get("event"), row.get("outcome")) for row in _supervisor_rows(tmp_path, "cancel_intent")]
|
||||
if cascade:
|
||||
assert events == []
|
||||
assert _supervisor_rows(tmp_path, "task_cancel_subtree_snapshot") == []
|
||||
else:
|
||||
assert ("requested", None) in events and ("settled", "not_found") in events, events
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cascade", [False, True])
|
||||
def test_finished_and_dead_task_is_404_and_mints_nothing(tmp_path, monkeypatch, cascade):
|
||||
"""Completion wins: a settled task with no live owner left mints no intent
|
||||
(no false "Cancelling…" on a finished card) and keeps its stored answer."""
|
||||
from ouroboros.cancel_intents import active_intents
|
||||
from ouroboros.task_results import load_task_result, write_task_result
|
||||
|
||||
_isolate_queue(monkeypatch, tmp_path, [])
|
||||
write_task_result(tmp_path, "done-1", "completed", result="the finished answer")
|
||||
before = load_task_result(tmp_path, "done-1")
|
||||
with _client(tmp_path) as client:
|
||||
response = client.post("/api/tasks/done-1/cancel", json={"cascade": cascade})
|
||||
assert response.status_code == 404
|
||||
assert active_intents(tmp_path) == {}
|
||||
assert [row for row in _supervisor_rows(tmp_path, "cancel_intent") if row.get("event") == "requested"] == []
|
||||
assert load_task_result(tmp_path, "done-1") == before
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cascade", [False, True])
|
||||
def test_persisted_but_never_admitted_task_is_cancelled_by_stop(tmp_path, monkeypatch, cascade):
|
||||
"""A child stuck at ``requested`` that never reached the queue (the
|
||||
admission stall the agent's cancel_task already settles): Stop records the
|
||||
intent and custody's miss lane settles it cancelled. A liveness pre-probe
|
||||
used to answer 404 here and leave it requested."""
|
||||
from ouroboros.cancel_intents import active_intents
|
||||
from ouroboros.task_results import STATUS_CANCELLED, load_task_result, write_task_result
|
||||
|
||||
_isolate_queue(monkeypatch, tmp_path, [])
|
||||
write_task_result(tmp_path, "stuck-kid", "requested", parent_task_id="coord",
|
||||
root_task_id="coord", delegation_role="subagent")
|
||||
with _client(tmp_path) as client:
|
||||
response = client.post("/api/tasks/stuck-kid/cancel", json={"cascade": cascade})
|
||||
assert response.status_code == 200, response.text
|
||||
assert response.json() == {"ok": True, "task_id": "stuck-kid", **({"cascade": True} if cascade else {})}
|
||||
assert load_task_result(tmp_path, "stuck-kid")["status"] == STATUS_CANCELLED
|
||||
assert active_intents(tmp_path) == {}
|
||||
|
||||
|
||||
def test_settled_task_with_a_live_worker_is_still_killed_and_keeps_its_answer(tmp_path, monkeypatch):
|
||||
"""Stored completion can precede the worker's exit (post-task cognition still
|
||||
spends). Once the intent request finds the target settled, live ownership
|
||||
qualifies it: the worker is killed, the stored answer survives, and
|
||||
completion wins the envelope (404)."""
|
||||
import types
|
||||
|
||||
from supervisor import workers
|
||||
|
||||
from ouroboros.cancel_intents import active_intents
|
||||
from ouroboros.task_results import load_task_result, write_task_result
|
||||
|
||||
queue, _pending = _isolate_queue(monkeypatch, tmp_path, [])
|
||||
monkeypatch.setattr(workers, "respawn_worker", lambda wid: None, raising=False)
|
||||
monkeypatch.setattr(workers, "get_event_q", lambda *, create=True: types.SimpleNamespace(put=lambda evt: None),
|
||||
raising=False)
|
||||
monkeypatch.setattr(queue, "_emit_cancel_task_done", lambda *_a, **_kw: None)
|
||||
alive = {"value": True}
|
||||
proc = types.SimpleNamespace(pid=4242, is_alive=lambda: alive["value"],
|
||||
terminate=lambda: alive.update(value=False), join=lambda timeout=None: None)
|
||||
workers.WORKERS[0] = types.SimpleNamespace(wid=0, proc=proc, busy_task_id="burning", reaping=False)
|
||||
monkeypatch.setattr("ouroboros.platform_layer.kill_pid_tree", lambda *_a, **_kw: alive.update(value=False))
|
||||
queue.RUNNING["burning"] = {"task": {"id": "burning", "chat_id": 0, "root_task_id": "burning"}, "worker_id": 0}
|
||||
write_task_result(tmp_path, "burning", "completed", chat_id=0, result="the stored answer")
|
||||
with _client(tmp_path) as client:
|
||||
response = client.post("/api/tasks/burning/cancel")
|
||||
assert response.status_code == 404, response.text
|
||||
assert alive["value"] is False, "the still-spending worker is killed"
|
||||
stored = load_task_result(tmp_path, "burning")
|
||||
assert (stored["status"], stored["result"]) == ("completed", "the stored answer")
|
||||
assert active_intents(tmp_path) == {}
|
||||
|
||||
|
||||
def test_stop_records_its_intent_before_custody_even_when_admission_races_the_click(tmp_path, monkeypatch):
|
||||
"""The single Stop used to ask liveness first, mint only for a live answer and
|
||||
call custody anyway: a task admitted between the probe and custody was torn
|
||||
down with no durable intent. No probe gates the intent now — even when every
|
||||
liveness observation says "not live", custody starts from a recorded,
|
||||
watchdog-replayable intent."""
|
||||
from ouroboros.cancel_intents import active_intent
|
||||
from ouroboros.task_results import STATUS_CANCELLED, load_task_result, write_task_result
|
||||
|
||||
queue, pending = _isolate_queue(monkeypatch, tmp_path, [{"id": "late", "chat_id": 1, "root_task_id": "late"}])
|
||||
# Observed before scheduler admission: both liveness readers say "not live".
|
||||
monkeypatch.setattr(queue, "task_has_live_ownership", lambda *_a, **_kw: False)
|
||||
monkeypatch.setattr(queue, "task_subtree_is_live", lambda *_a, **_kw: False)
|
||||
write_task_result(tmp_path, "late", "scheduled")
|
||||
intents_at_custody = []
|
||||
real_custody = queue.cancel_task_custody
|
||||
|
||||
def _custody(task_id, **kwargs):
|
||||
intents_at_custody.append(active_intent(tmp_path, task_id))
|
||||
return real_custody(task_id, **kwargs)
|
||||
|
||||
monkeypatch.setattr(queue, "cancel_task_custody", _custody)
|
||||
with _client(tmp_path) as client:
|
||||
response = client.post("/api/tasks/late/cancel")
|
||||
assert response.status_code == 200, response.text
|
||||
assert intents_at_custody and (intents_at_custody[0] or {}).get("source") == "http_single", intents_at_custody
|
||||
assert load_task_result(tmp_path, "late")["status"] == STATUS_CANCELLED
|
||||
assert pending == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cascade", [False, True])
|
||||
def test_unconfirmed_teardown_answers_pending_and_the_watchdog_settles_it(tmp_path, monkeypatch, cascade):
|
||||
"""The intent is recorded but this request cannot confirm the teardown (a
|
||||
busy store or a refused custody leaves it open with nothing captured): 202
|
||||
pending with the intent's request id — never 503 for a Stop that is recorded
|
||||
— and the cancel-intent watchdog finishes it once custody can."""
|
||||
import time
|
||||
|
||||
from supervisor.task_lifecycle import sweep_cancel_intents
|
||||
|
||||
from ouroboros.cancel_intents import active_intent, active_intents
|
||||
from ouroboros.task_results import STATUS_CANCELLED, load_task_result, write_task_result
|
||||
|
||||
queue, pending = _isolate_queue(monkeypatch, tmp_path, [{"id": "busy-1", "chat_id": 0, "root_task_id": "busy-1"}])
|
||||
write_task_result(tmp_path, "busy-1", "scheduled")
|
||||
store = {"busy": True}
|
||||
real_custody = queue.cancel_task_custody
|
||||
|
||||
def _custody(task_id, **kwargs):
|
||||
if store["busy"]:
|
||||
return queue.CANCEL_FAILED # nothing captured, the intent stays open
|
||||
return real_custody(task_id, **kwargs)
|
||||
|
||||
monkeypatch.setattr(queue, "cancel_task_custody", _custody)
|
||||
with _client(tmp_path) as client:
|
||||
response = client.post("/api/tasks/busy-1/cancel", json={"cascade": cascade})
|
||||
intent = active_intent(tmp_path, "busy-1")
|
||||
assert response.status_code == 202, response.text
|
||||
assert response.json() == {
|
||||
"ok": True, "task_id": "busy-1", "cancel_state": "pending", "stop_policy": "immediate",
|
||||
"request_id": intent["request_id"], "reason_code": "teardown_unconfirmed",
|
||||
**({"cascade": True} if cascade else {}),
|
||||
}
|
||||
assert [task["id"] for task in pending] == ["busy-1"], "nothing was torn down yet"
|
||||
|
||||
store["busy"] = False
|
||||
assert sweep_cancel_intents(now=time.time() + 60) == {"busy-1": queue.CANCEL_CANCELLED}
|
||||
assert load_task_result(tmp_path, "busy-1")["status"] == STATUS_CANCELLED
|
||||
assert active_intents(tmp_path) == {}
|
||||
assert pending == []
|
||||
|
||||
|
||||
def test_stop_action_replay_joins_the_pending_intent_and_other_arguments_are_409(tmp_path, monkeypatch):
|
||||
"""The exact retry of one Stop action joins the recorded intent (same request
|
||||
id, still pending); the same stop_action_id with different arguments is
|
||||
refused 409 and changes nothing."""
|
||||
from ouroboros.task_results import write_task_result
|
||||
|
||||
queue, pending = _isolate_queue(monkeypatch, tmp_path, [{"id": "act-1", "chat_id": 0, "root_task_id": "act-1"}])
|
||||
write_task_result(tmp_path, "act-1", "scheduled")
|
||||
monkeypatch.setattr(queue, "cancel_task_custody", lambda task_id, **_kw: queue.CANCEL_FAILED)
|
||||
with _client(tmp_path) as client:
|
||||
first = client.post("/api/tasks/act-1/cancel", json={"stop_action_id": "stop-a"})
|
||||
replay = client.post("/api/tasks/act-1/cancel", json={"stop_action_id": "stop-a"})
|
||||
conflict = client.post("/api/tasks/act-1/cancel", json={"stop_action_id": "stop-a", "cascade": True})
|
||||
assert (first.status_code, replay.status_code) == (202, 202)
|
||||
assert replay.json()["request_id"] == first.json()["request_id"]
|
||||
assert conflict.status_code == 409
|
||||
assert conflict.json()["reason_code"] == "stop_action_conflict"
|
||||
assert [task["id"] for task in pending] == ["act-1"]
|
||||
|
|
|
|||
|
|
@ -110,3 +110,51 @@ def test_unconsumed_settlement_preparations_are_bounded_per_thread(tmp_path):
|
|||
newest = f"t{task_ownership._PREPARED_READS_MAX + 39}"
|
||||
assert task_ownership.settlement_reads(tmp_path, newest, locked=True) is not None
|
||||
pending.clear()
|
||||
|
||||
|
||||
def test_recheck_is_tri_state_and_never_waits_under_the_queue_lock(env, monkeypatch):
|
||||
"""A3: the prepared signature is re-checked as UNCHANGED / CHANGED / BUSY. Under Q the
|
||||
probe uses a zero budget, so a busy store answers BUSY (unknown) — never CHANGED and
|
||||
never UNCHANGED — while ``unchanged()`` keeps its boolean 'proven unchanged' meaning."""
|
||||
from ouroboros import store, task_store
|
||||
from supervisor.task_ownership import BUSY, CHANGED, UNCHANGED
|
||||
|
||||
budgets = []
|
||||
real_budget = store.acquisition_budget
|
||||
|
||||
def _spy_budget(seconds):
|
||||
budgets.append((seconds, queue._queue_lock._is_owned()))
|
||||
return real_budget(seconds)
|
||||
|
||||
monkeypatch.setattr(store, 'acquisition_budget', _spy_budget)
|
||||
reads = TaskOwnershipRead(env)
|
||||
reads.load('root')
|
||||
assert budgets == [], 'the signed read needs no zero-budget probe outside Q'
|
||||
assert reads.recheck(('root',)) == UNCHANGED and reads.unchanged(('root',)) is True
|
||||
assert budgets == []
|
||||
with queue._queue_lock:
|
||||
assert reads.recheck(('root',)) == UNCHANGED
|
||||
assert budgets == [(0, True)]
|
||||
|
||||
task_results.write_task_result(env, 'root', 'completed', result='Rewritten answer')
|
||||
with queue._queue_lock:
|
||||
assert reads.recheck(('root',)) == CHANGED and reads.unchanged(('root',)) is False
|
||||
with pytest.raises(ValueError, match='changed'):
|
||||
reads.load('root')
|
||||
|
||||
reads = TaskOwnershipRead(env)
|
||||
reads.load('root')
|
||||
real_sig = task_store.signature
|
||||
|
||||
def _busy_at_zero(root, tid, *, txn=None):
|
||||
if store._WAIT_BUDGET.get() == 0:
|
||||
raise store.StoreBusy('writer active')
|
||||
return real_sig(root, tid, txn=txn)
|
||||
|
||||
monkeypatch.setattr(task_store, 'signature', _busy_at_zero)
|
||||
with queue._queue_lock:
|
||||
assert reads.recheck(('root',)) == BUSY and reads.unchanged(('root',)) is False
|
||||
with pytest.raises(ValueError, match='busy'):
|
||||
reads.load('root')
|
||||
assert reads.prepared('root')['status'] == 'completed', 'the prepared row needs no re-check'
|
||||
assert reads.recheck(('root',)) == UNCHANGED, 'outside Q the probe waits with the normal budget'
|
||||
|
|
|
|||
304
tests/test_task_room_rule.py
Normal file
304
tests/test_task_room_rule.py
Normal file
|
|
@ -0,0 +1,304 @@
|
|||
"""One room rule (owner decision A4): the bus write seam, replay membership and cognition
|
||||
agree on the room a task-addressed row belongs to, and a warm frame costs no store read.
|
||||
|
||||
Two-sided: every case fails on d375d9fd0 (the bus never re-addressed a row that reached
|
||||
it without lineage, and replay membership read only the explicit-roots map) and asserts
|
||||
the still-working case beside it (a Main direct task with a project_id stays Main, a
|
||||
pinned or origin-addressed row keeps its producer's address).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from ouroboros import projects_registry as pr
|
||||
from ouroboros.project_dialogue import ROOM_FIXED_ROW_TYPES, room_membership
|
||||
from ouroboros.task_results import write_task_result
|
||||
|
||||
pytestmark = pytest.mark.serial
|
||||
|
||||
|
||||
def _bus(monkeypatch, root):
|
||||
from supervisor import message_bus
|
||||
|
||||
sent = []
|
||||
monkeypatch.setattr(message_bus, "DATA_DIR", root)
|
||||
monkeypatch.setattr(message_bus, "load_state", lambda: {"owner_id": 1, "owner_chat_id": 1})
|
||||
monkeypatch.setattr(message_bus, "get_bridge", lambda: SimpleNamespace(send_message=lambda *a, **k: sent.append((a, k))))
|
||||
return message_bus, sent
|
||||
|
||||
|
||||
def _progress_rows(root):
|
||||
return [json.loads(line) for line in (root / "logs/progress.jsonl").read_text().splitlines()]
|
||||
|
||||
|
||||
def _world(root):
|
||||
"""A converted root with a cancelled child, a root bound to A with a child bound to B, a
|
||||
room-born root, and a Main direct task that carries a project_id."""
|
||||
a, b, native = (pr.create_project(root, pid, name=pid) for pid in ("room-a", "room-b", "room-native"))
|
||||
write_task_result(root, "root-1", "running", root_task_id="root-1", chat_id=1)
|
||||
pr.bind_task_to_project(root, "root-1", a["id"], origin={"absent": "system"})
|
||||
write_task_result(root, "kid-1", "cancelled", delegation_role="subagent", parent_task_id="root-1",
|
||||
root_task_id="root-1", chat_id=1)
|
||||
write_task_result(root, "mid-1", "running", delegation_role="subagent", parent_task_id="root-1",
|
||||
root_task_id="root-1", chat_id=1)
|
||||
pr.bind_task_to_project(root, "mid-1", b["id"], origin={"absent": "system"})
|
||||
write_task_result(root, "grand-1", "running", delegation_role="subagent", parent_task_id="mid-1",
|
||||
root_task_id="root-1", chat_id=1)
|
||||
write_task_result(root, "born-1", "running", root_task_id="born-1", chat_id=native["chat_id"],
|
||||
project_id=native["id"])
|
||||
write_task_result(root, "direct-1", "running", root_task_id="direct-1", chat_id=1, _is_direct_chat=True,
|
||||
project_id=native["id"])
|
||||
return {"a": a["chat_id"], "b": b["chat_id"], "native": native["chat_id"]}
|
||||
|
||||
|
||||
def _membership(root, chat):
|
||||
projects = pr.reserved_project_chat_ids(root)
|
||||
return room_membership(chat if chat in projects else 1, projects, [], pr.RoomResolver(root))
|
||||
|
||||
|
||||
@pytest.mark.parametrize("task_id, room", [
|
||||
("kid-1", "a"), # a child of a converted root that carries only its id
|
||||
("grand-1", "b"), # root bound to A, its parent bound to B: the nearest conversion wins
|
||||
("born-1", "native"), # born in a Project, bound to nothing
|
||||
])
|
||||
def test_the_bus_and_replay_put_a_project_tasks_bare_row_in_its_room(tmp_path, monkeypatch, task_id, room):
|
||||
rooms = _world(tmp_path)
|
||||
bus, sent = _bus(monkeypatch, tmp_path)
|
||||
bus.send_with_budget(1, "Stopped by Ouroboros", is_progress=True, task_id=task_id, role="system",
|
||||
system_type="subagent_terminal_notice")
|
||||
row = _progress_rows(tmp_path)[-1]
|
||||
assert row["chat_id"] == rooms[room] == sent[-1][0][0], "the durable row and the live frame share the room"
|
||||
if task_id != "born-1":
|
||||
assert (row["delegation_role"], row["parent_task_id"], row["root_task_id"]) == (
|
||||
"subagent", {"kid-1": "root-1", "grand-1": "mid-1"}[task_id], "root-1"), "lineage rides the row"
|
||||
# Replay: the row as it was recorded before the seam (chat 1, task id only) lives in the same room.
|
||||
legacy = {"type": "send_message", "task_id": task_id, "chat_id": 1}
|
||||
assert _membership(tmp_path, rooms[room])(1, legacy) is True
|
||||
assert _membership(tmp_path, 1)(1, legacy) is False, "Main does not keep another room's task"
|
||||
assert all(_membership(tmp_path, rooms[other])(1, legacy) is False for other in rooms if other != room)
|
||||
|
||||
|
||||
def test_live_routing_and_replay_agree_for_a_deep_chain_with_two_conversions(tmp_path):
|
||||
rooms = _world(tmp_path)
|
||||
assert pr.project_chat_for_task_tree(tmp_path, "grand-1") == rooms["b"]
|
||||
assert pr.project_chat_for_task_tree(tmp_path, "grand-1", "mid-1", "root-1") == rooms["b"], "a supplied root never shortcuts the parent chain"
|
||||
assert pr.project_chat_for_task_tree(tmp_path, "kid-1", "root-1", "root-1") == rooms["a"]
|
||||
row = {"task_id": "grand-1", "parent_task_id": "mid-1", "root_task_id": "root-1"}
|
||||
assert _membership(tmp_path, rooms["b"])(1, row) and not _membership(tmp_path, rooms["a"])(1, row)
|
||||
|
||||
|
||||
def test_a_main_direct_task_with_a_project_id_stays_main(tmp_path, monkeypatch):
|
||||
rooms = _world(tmp_path)
|
||||
bus, sent = _bus(monkeypatch, tmp_path)
|
||||
for meta in ({"_is_direct_chat": True}, {}): # the flagged turn and its unflagged finalization frame
|
||||
bus.send_with_budget(1, "Thinking", is_progress=True, task_id="direct-1", role="system",
|
||||
system_type="host_progress", progress_meta=meta)
|
||||
assert _progress_rows(tmp_path)[-1]["chat_id"] == 1 == sent[-1][0][0]
|
||||
assert pr.project_chat_for_task_tree(tmp_path, "direct-1") == 0, "project_id is a copy, never the room"
|
||||
assert _membership(tmp_path, 1)(1, {"task_id": "direct-1"}) is True
|
||||
assert _membership(tmp_path, rooms["native"])(1, {"task_id": "direct-1"}) is False
|
||||
|
||||
|
||||
@pytest.mark.parametrize("system_type", sorted(ROOM_FIXED_ROW_TYPES))
|
||||
def test_a_fixed_row_type_keeps_its_producers_address_and_borrows_no_lineage(tmp_path, monkeypatch, system_type):
|
||||
rooms = _world(tmp_path)
|
||||
bus, sent = _bus(monkeypatch, tmp_path)
|
||||
bus.send_with_budget(rooms["native"], "Not delivered", task_id="kid-1", role="system", system_type=system_type)
|
||||
stored = [json.loads(line) for line in (tmp_path / "logs/chat.jsonl").read_text().splitlines()][-1]
|
||||
assert stored["chat_id"] == rooms["native"] == sent[-1][0][0], "the producer's address stands"
|
||||
assert not {"delegation_role", "parent_task_id", "root_task_id"} & set(stored), "the target is not the speaker"
|
||||
|
||||
|
||||
def _count_store_work(monkeypatch):
|
||||
from ouroboros import store, task_status, task_store
|
||||
|
||||
counts = {"open_store": 0, "load_effective_task_result": 0}
|
||||
real_open, real_load = store.open_store, task_status.load_effective_task_result
|
||||
|
||||
def _open(*a, **k):
|
||||
counts["open_store"] += 1
|
||||
return real_open(*a, **k)
|
||||
|
||||
def _load(*a, **k):
|
||||
counts["load_effective_task_result"] += 1
|
||||
return real_load(*a, **k)
|
||||
|
||||
monkeypatch.setattr(store, "open_store", _open)
|
||||
monkeypatch.setattr(task_store, "open_store", _open)
|
||||
monkeypatch.setattr(pr, "open_store", _open)
|
||||
monkeypatch.setattr(task_status, "load_effective_task_result", _load)
|
||||
return counts
|
||||
|
||||
|
||||
@pytest.mark.parametrize("through_handler", [False, True], ids=["send_with_budget", "event_queue_handler"])
|
||||
def test_a_warm_frame_opens_no_store_and_a_cold_task_id_opens_at_most_once(tmp_path, monkeypatch, through_handler):
|
||||
from supervisor.events_chat_delivery import _handle_send_message
|
||||
|
||||
rooms = _world(tmp_path)
|
||||
bus, sent = _bus(monkeypatch, tmp_path)
|
||||
counts = _count_store_work(monkeypatch)
|
||||
ctx = SimpleNamespace(DRIVE_ROOT=tmp_path, RUNNING={}, send_with_budget=bus.send_with_budget)
|
||||
|
||||
def frame(task_id, meta, n=10):
|
||||
for i in range(n):
|
||||
if through_handler:
|
||||
_handle_send_message({"type": "send_message", "chat_id": 1, "text": f"frame {i}", "is_progress": True,
|
||||
"role": "system", "system_type": "host_progress", "task_id": task_id,
|
||||
"progress_meta": dict(meta)}, ctx)
|
||||
else:
|
||||
bus.send_with_budget(1, f"frame {i}", is_progress=True, task_id=task_id, progress_meta=dict(meta),
|
||||
role="system", system_type="host_progress")
|
||||
|
||||
lineage = {"delegation_role": "subagent", "parent_task_id": "root-1", "root_task_id": "root-1"}
|
||||
frame("kid-1", lineage)
|
||||
assert counts == {"open_store": 0, "load_effective_task_result": 0}, "a frame carrying its lineage opens nothing"
|
||||
frame("grand-1", {})
|
||||
opened = counts["open_store"]
|
||||
assert 0 < opened <= 2, "one indexed read per unseen id on the chain (grand-1, mid-1 is bound)"
|
||||
frame("grand-1", {})
|
||||
assert counts["open_store"] == opened, "warm frames open nothing"
|
||||
assert counts["load_effective_task_result"] == 0, "the seam never parses a result body"
|
||||
assert {a[0][0] for a in sent} == {rooms["a"], rooms["b"]}
|
||||
|
||||
|
||||
def test_a_conversion_invalidates_room_answers_and_a_store_commit_does_not(tmp_path):
|
||||
rooms = _world(tmp_path)
|
||||
assert pr.project_chat_for_task_tree(tmp_path, "kid-1") == rooms["a"]
|
||||
write_task_result(tmp_path, "kid-1", "cancelled", progress_note="a later commit")
|
||||
assert pr.RoomResolver(tmp_path).answers.get("kid-1") == rooms["a"], "a commit keeps the warm answer"
|
||||
c = pr.create_project(tmp_path, "room-c", name="room-c")
|
||||
pr.bind_task_to_project(tmp_path, "kid-1", c["id"], origin={"absent": "system"})
|
||||
assert pr.project_chat_for_task_tree(tmp_path, "kid-1") == c["chat_id"], "the nearest conversion wins at once"
|
||||
|
||||
|
||||
def test_an_unadmitted_id_is_answered_but_a_miss_is_never_remembered(tmp_path):
|
||||
rooms = _world(tmp_path)
|
||||
assert pr.project_chat_for_task_tree(tmp_path, "later-1") == 0 and pr.RoomResolver(tmp_path).lineage("later-1") == {}
|
||||
assert pr.project_chat_for_task_tree(tmp_path, "later-2", "root-1", "root-1") == rooms["a"], "supplied lineage routes an unseen id"
|
||||
for task_id in ("later-1", "later-2"):
|
||||
write_task_result(tmp_path, task_id, "scheduled", delegation_role="subagent", parent_task_id="root-1",
|
||||
root_task_id="root-1", chat_id=1)
|
||||
assert pr.project_chat_for_task_tree(tmp_path, task_id) == rooms["a"], "admission creates the facts"
|
||||
|
||||
|
||||
def _busy_store(monkeypatch, busy):
|
||||
"""``open_store`` raises StoreBusy while ``busy["on"]``: the store under contention."""
|
||||
from ouroboros import store
|
||||
|
||||
real_open = store.open_store
|
||||
|
||||
def _open(*a, **k):
|
||||
if busy["on"]:
|
||||
raise store.StoreBusy("database is locked")
|
||||
return real_open(*a, **k)
|
||||
|
||||
monkeypatch.setattr(pr, "open_store", _open)
|
||||
return _open
|
||||
|
||||
|
||||
def test_an_unavailable_ancestor_read_is_never_memoized(tmp_path, monkeypatch):
|
||||
"""Astra's counterexample: leaf -> middle -> mid (bound to B) -> root (bound to A); the leaf's
|
||||
facts are warm, the middle's read hits a busy store. The fallback answer of that call may
|
||||
be wrong, but it is not remembered: once the store answers again the leaf resolves to B."""
|
||||
a, b = (pr.create_project(tmp_path, pid, name=pid) for pid in ("room-a", "room-b"))
|
||||
write_task_result(tmp_path, "root", "running", root_task_id="root", chat_id=1)
|
||||
pr.bind_task_to_project(tmp_path, "root", a["id"], origin={"absent": "system"})
|
||||
write_task_result(tmp_path, "mid", "running", delegation_role="subagent", parent_task_id="root", root_task_id="root", chat_id=1)
|
||||
pr.bind_task_to_project(tmp_path, "mid", b["id"], origin={"absent": "system"})
|
||||
write_task_result(tmp_path, "middle", "running", delegation_role="subagent", parent_task_id="mid", root_task_id="root", chat_id=1)
|
||||
write_task_result(tmp_path, "leaf", "running", delegation_role="subagent", parent_task_id="middle", root_task_id="root", chat_id=1)
|
||||
assert pr.RoomResolver(tmp_path).lineage("leaf")["parent_task_id"] == "middle" # warm leaf facts, cold middle
|
||||
busy = {"on": True}
|
||||
_busy_store(monkeypatch, busy)
|
||||
contended = pr.project_chat_for_task_tree(tmp_path, "leaf")
|
||||
assert contended in (0, a["chat_id"]), "a best-effort answer while the middle is unreadable"
|
||||
assert "leaf" not in pr.RoomResolver(tmp_path).answers, "an incomplete ancestry read is not memoized"
|
||||
busy["on"] = False
|
||||
assert pr.project_chat_for_task_tree(tmp_path, "leaf") == b["chat_id"], "the retried read finds the nearer conversion"
|
||||
assert pr.RoomResolver(tmp_path).answers["leaf"] == b["chat_id"], "a complete read is memoized"
|
||||
|
||||
|
||||
def test_a_healthy_warm_frame_still_opens_no_store(tmp_path, monkeypatch):
|
||||
rooms = _world(tmp_path)
|
||||
assert pr.project_chat_for_task_tree(tmp_path, "grand-1") == rooms["b"]
|
||||
counts = _count_store_work(monkeypatch)
|
||||
for _ in range(10):
|
||||
assert pr.project_chat_for_task_tree(tmp_path, "grand-1") == rooms["b"]
|
||||
assert counts["open_store"] == 0
|
||||
|
||||
|
||||
def test_an_absent_id_is_read_once_per_resolver(tmp_path, monkeypatch):
|
||||
_world(tmp_path)
|
||||
counts = _count_store_work(monkeypatch)
|
||||
rooms = pr.RoomResolver(tmp_path)
|
||||
for _ in range(5):
|
||||
assert rooms("ghost-1") == 0 and rooms.lineage("ghost-1") == {}
|
||||
assert counts["open_store"] == 1, "one read per resolver for an id the store holds no row for"
|
||||
rooms.warm(["ghost-1", "ghost-2"])
|
||||
assert counts["open_store"] == 2 and rooms("ghost-2") == 0 and counts["open_store"] == 2
|
||||
assert pr.RoomResolver(tmp_path)("ghost-1") == 0 and counts["open_store"] == 3, "a fresh resolver (a new frame) asks again"
|
||||
|
||||
|
||||
def _request(root, body):
|
||||
async def _json():
|
||||
return body
|
||||
|
||||
return SimpleNamespace(app=SimpleNamespace(state=SimpleNamespace(drive_root=root)), json=_json)
|
||||
|
||||
|
||||
def _history(root, chat):
|
||||
import asyncio
|
||||
|
||||
from ouroboros.gateway.history import make_chat_history_endpoint
|
||||
|
||||
response = asyncio.run(make_chat_history_endpoint(root)(SimpleNamespace(query_params={"chat_id": str(chat)})))
|
||||
assert response.status_code == 200, response.body
|
||||
return [row["task_id"] for row in json.loads(response.body)["messages"] if row.get("task_id")]
|
||||
|
||||
|
||||
def test_converting_a_middle_task_moves_its_subtrees_earlier_rows_out_of_the_old_room(tmp_path, monkeypatch):
|
||||
"""Astra's counterexample through the real conversion endpoint: root R bound to A, its child C
|
||||
emits a row into A, then the owner turns C into project B. Live delivery already goes to B;
|
||||
history, chat_chain and the memory inventory must show C's earlier row in B only."""
|
||||
import asyncio
|
||||
|
||||
from ouroboros import chat_chain, memory_inventory
|
||||
from ouroboros.gateway.projects import api_project_from_task
|
||||
|
||||
monkeypatch.setattr("ouroboros.gateway.state._chat_activities_snapshot_safe", lambda *_a, **_k: [])
|
||||
a = pr.create_project(tmp_path, "room-a", name="Room A")
|
||||
write_task_result(tmp_path, "root-1", "running", root_task_id="root-1", chat_id=1)
|
||||
pr.bind_task_to_project(tmp_path, "root-1", a["id"], origin={"absent": "system"})
|
||||
write_task_result(tmp_path, "kid-1", "running", delegation_role="subagent", parent_task_id="root-1",
|
||||
root_task_id="root-1", chat_id=1)
|
||||
bus, sent = _bus(monkeypatch, tmp_path)
|
||||
bus.send_with_budget(1, "Root planning", is_progress=True, task_id="root-1", role="system", system_type="host_progress")
|
||||
bus.send_with_budget(1, "Child working", is_progress=True, task_id="kid-1", role="system", system_type="host_progress")
|
||||
assert [row["chat_id"] for row in _progress_rows(tmp_path)] == [a["chat_id"]] * 2, "before: both rows live in A"
|
||||
assert _history(tmp_path, a["chat_id"]) == ["root-1", "kid-1"]
|
||||
|
||||
response = asyncio.run(api_project_from_task(_request(tmp_path, {"task_id": "kid-1", "id": "task-kid-1"})))
|
||||
payload = json.loads(response.body)
|
||||
b = int(payload["project"]["chat_id"])
|
||||
assert response.status_code == 200 and payload["binding"]["project_id"] == payload["project"]["id"] and b != a["chat_id"]
|
||||
|
||||
bus.send_with_budget(1, "Child still working", is_progress=True, task_id="kid-1", role="system", system_type="host_progress")
|
||||
assert sent[-1][0][0] == b, "live delivery follows the conversion"
|
||||
assert _history(tmp_path, a["chat_id"]) == ["root-1"], "A no longer shows the converted subtree's earlier row"
|
||||
assert _history(tmp_path, b) == ["kid-1", "kid-1"], "B shows the earlier row beside the new one"
|
||||
assert _history(tmp_path, 1) == [], "Main shows neither"
|
||||
assert [row.get("task_id") for _addr, row, _pos in chat_chain.iter_room_rows(tmp_path, a["chat_id"])] == []
|
||||
assert [row.get("task_id") for _addr, row, _pos in chat_chain.iter_room_rows(tmp_path, b)] == []
|
||||
facts = memory_inventory.membership_facts(tmp_path)
|
||||
early = _progress_rows(tmp_path)[1]
|
||||
assert memory_inventory.rooms_of_row(memory_inventory.row_meta(early), facts) == {str(b)}
|
||||
# Controls: the root's own row stays in its room, a Project-born row in its own room, Main work in Main.
|
||||
assert memory_inventory.rooms_of_row(memory_inventory.row_meta(_progress_rows(tmp_path)[0]), facts) == {str(a["chat_id"])}
|
||||
native = pr.create_project(tmp_path, "room-native", name="Native")
|
||||
write_task_result(tmp_path, "born-1", "running", root_task_id="born-1", chat_id=native["chat_id"])
|
||||
facts = memory_inventory.membership_facts(tmp_path) # the registry facts of the moment, as a capture takes them
|
||||
assert memory_inventory.rooms_of_row({"chat_id": native["chat_id"], "task_id": "born-1"}, facts) == {str(native["chat_id"])}
|
||||
assert memory_inventory.rooms_of_row({"chat_id": 1, "task_id": "direct-9"}, facts) == {"1"}
|
||||
|
|
@ -224,7 +224,9 @@ def test_natural_completion_wins_a_late_cancel(tmp_path, monkeypatch):
|
|||
monkeypatch.setattr(queue_module, "persist_queue_snapshot", lambda reason="": None)
|
||||
monkeypatch.setattr(queue_module, "_emit_cancel_task_done", lambda *_args, **_kwargs: None)
|
||||
|
||||
assert queue_module.cancel_task_by_id("fast-child") is True
|
||||
# No intent was ever minted (the tool saw a settled child), so custody is the
|
||||
# typed no-op already_settled (A2): nothing captured, nothing written.
|
||||
assert task_lifecycle.cancel_task_custody("fast-child") == task_lifecycle.CANCEL_ALREADY_SETTLED
|
||||
stored = load_task_result(tmp_path, "fast-child")
|
||||
assert stored["status"] == STATUS_COMPLETED
|
||||
assert stored["accounted_upper_bound_usd"] == 0.75
|
||||
|
|
@ -234,12 +236,9 @@ def test_natural_completion_wins_a_late_cancel(tmp_path, monkeypatch):
|
|||
# Completion wins WITHOUT a parent_decision stamp: discarding a kept result
|
||||
# stays a separate explicit action (discard_child_result).
|
||||
assert "parent_decision" not in stored
|
||||
# The durable intent settled (already_settled) — nothing left pending.
|
||||
assert active_intent(tmp_path, "fast-child") is None
|
||||
public = public_task_result(stored)
|
||||
assert public["outcome_axes"]["execution"]["status"] == "ok"
|
||||
# The typed custody outcome (not the boolean facade) reports already_settled.
|
||||
assert task_lifecycle.cancel_task_custody("fast-child") == task_lifecycle.CANCEL_ALREADY_SETTLED
|
||||
|
||||
|
||||
def test_cancel_workspace_task_records_terminal_artifact_state(tmp_path, monkeypatch):
|
||||
|
|
@ -277,7 +276,10 @@ def test_cancel_workspace_task_records_terminal_artifact_state(tmp_path, monkeyp
|
|||
result="queued",
|
||||
)
|
||||
|
||||
assert queue_module.cancel_task_by_id("workspacecancel") is True
|
||||
from ouroboros.cancel_intents import request_cancel
|
||||
|
||||
request_cancel(tmp_path, "workspacecancel")
|
||||
assert queue_module.drive_cancel_intent_scope("workspacecancel") == queue_module.CANCEL_CANCELLED
|
||||
|
||||
stored = load_task_result(tmp_path, "workspacecancel")
|
||||
assert stored["status"] == STATUS_CANCELLED
|
||||
|
|
|
|||
|
|
@ -129,7 +129,10 @@ def test_headless_pending_cancel_still_emits_task_done(tmp_path, monkeypatch):
|
|||
monkeypatch.setattr(workers, "get_event_q", lambda: SimpleNamespace(put=events.append))
|
||||
monkeypatch.setattr(queue, "persist_queue_snapshot", lambda reason="": None)
|
||||
|
||||
assert queue.cancel_task_by_id(task["id"]) is True
|
||||
from ouroboros.cancel_intents import request_cancel
|
||||
|
||||
request_cancel(tmp_path, task["id"])
|
||||
assert queue.drive_cancel_intent_scope(task["id"]) == queue.CANCEL_CANCELLED
|
||||
|
||||
terminal = [event for event in events if event.get("type") == "task_done"]
|
||||
assert len(terminal) == 1
|
||||
|
|
|
|||
|
|
@ -516,7 +516,7 @@ function renderProjectsNav(projects, projectChatIds, activityIndex = projectActi
|
|||
const all = projects || [];
|
||||
// Isolation fan-out SSOT: recognize EVERY registered project chat_id (incl.
|
||||
// file-less / no-activity / beyond the sidebar summary cap), matching the
|
||||
// backend registered_project_chat_ids, so chat.js::isMyThread never treats a
|
||||
// backend reserved_project_chat_ids, so chat.js::isMyThread never treats a
|
||||
// project frame as a main-thread frame on the live WS path. Prefer the
|
||||
// COMPLETE /api/state `project_chat_ids` (uncapped); fall back to the (capped)
|
||||
// projects array only if that field is absent. Sidebar visibility is a
|
||||
|
|
|
|||
|
|
@ -495,9 +495,10 @@ export function initActivity({ mount, ws } = {}) {
|
|||
}
|
||||
// Same declared semantics as the chat card (v6.82): the
|
||||
// task AND its live subtree, so stopping an orchestrator
|
||||
// never orphans its running subagents. Soft stop answers
|
||||
// 202 with the intent open; immediate answers after the
|
||||
// teardown — either way the refresh shows honest state.
|
||||
// never orphans its running subagents. A soft stop answers
|
||||
// 202 with the intent open; an immediate stop answers 200
|
||||
// once torn down or 202 pending while the intent is still
|
||||
// being driven — either way the refresh shows honest state.
|
||||
await requestStop(id, action);
|
||||
} catch (exc) {
|
||||
// A 404 is the documented completion race (the run
|
||||
|
|
|
|||
|
|
@ -147,12 +147,15 @@ export function createTask(payload) {
|
|||
}
|
||||
|
||||
/**
|
||||
* Cancel a task. With {cascade:true} the server also cancels the task's live
|
||||
* subtree and answers only once that teardown has finished; without it the
|
||||
* request stays the synchronous single-task cancel (no body — headless compat).
|
||||
* S3 (Q1/Q2): stopPolicy "finalize_then_cancel" requests the soft
|
||||
* finalize-then-stop episode (202 acknowledgement, cancel_state "pending");
|
||||
* "immediate" (or absent) keeps today's hard cancel byte-identical.
|
||||
* Cancel a task. The server records the durable Stop first, then drives it;
|
||||
* with {cascade:true} the task's live subtree is torn down too. Without
|
||||
* options no body is sent (headless compat). Every answer with ok:true is an
|
||||
* accepted Stop: 200 once the teardown settled, 202 cancel_state "pending"
|
||||
* while it is still being finished — the soft finalize-then-stop episode of
|
||||
* stopPolicy "finalize_then_cancel" (S3, Q1/Q2), or an "immediate" (or
|
||||
* absent) Stop whose teardown is not confirmed yet (reason_code
|
||||
* "teardown_unconfirmed"); the supervisor finishes it and task detail shows
|
||||
* the pending state. 404 means there is nothing to stop.
|
||||
* Shared by the Chat live-card stop control and the Activity tab.
|
||||
* @param {string} taskId
|
||||
* @param {{cascade?: boolean, stopPolicy?: string, stopActionId?: string}} [options]
|
||||
|
|
|
|||
|
|
@ -278,8 +278,8 @@
|
|||
* @property {string=} task_terminal_status Typed terminal fact on a frame that IS the turn's conclusion (stamped on direct/ephemeral finals and the direct error branch).
|
||||
* @property {string=} task_incident
|
||||
* @property {string=} cancel_physical_task_id
|
||||
* A cancellation fault names the physical task it could not settle when that
|
||||
* differs from the displayed task id.
|
||||
* Historical rows only: the retired cancellation-fault notice named the
|
||||
* physical task it could not settle when that differed from the displayed id.
|
||||
* @property {string=} toast_once
|
||||
* @property {string=} toast_tone
|
||||
* The incident's valence for the one-shot toast (warn/ok/error); absent =
|
||||
|
|
@ -1393,11 +1393,16 @@
|
|||
* @property {boolean} ok
|
||||
* @property {string} task_id
|
||||
* @property {boolean=} cascade
|
||||
* Echoed only for {"cascade": true}, after subtree cancel completes; single-task envelope unchanged.
|
||||
* Echoed when the Stop ran as a cascade (requested, or an already recorded cascade scope); single-task envelope unchanged.
|
||||
* @property {string=} cancel_state
|
||||
* "pending" on graceful 202 ACK: durable intent stays open during bounded finalization; absent for immediate.
|
||||
* "pending" on every 202: the durable intent stays open — bounded graceful finalization, or an immediate
|
||||
* teardown not confirmed yet that the supervisor finishes; absent on 200.
|
||||
* @property {string=} stop_policy
|
||||
* Effective durable "immediate" | "finalize_then_cancel"; a graceful request never softens a hard intent.
|
||||
* On 202, the effective durable "immediate" | "finalize_then_cancel"; a graceful request never softens a hard intent.
|
||||
* @property {string=} request_id
|
||||
* On an immediate 202, the open intent's custody id (distinct from the client's stop_action_id).
|
||||
* @property {string=} reason_code
|
||||
* On an immediate 202, "teardown_unconfirmed".
|
||||
* @property {string=} error
|
||||
*/
|
||||
/**
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ import { PAGE_ICONS } from './page_icons.js';
|
|||
import { showToast } from './toast.js';
|
||||
import { tx } from './i18n.js';
|
||||
import { hostRowDisplayText } from './host_notice_text.js';
|
||||
import { childRowPlacement, replayConcludesTask } from './chat_row_placement.js';
|
||||
import { ORIGIN_ADDRESSED_NOTICE_TYPES, childRowPlacement, replayConcludesTask } from './chat_row_placement.js';
|
||||
import { decorateProjectRow, syncSavedProjectContext } from './project_answer.js';
|
||||
import { createProjectHandoffs, receiptNotice } from './project_handoff.js';
|
||||
import { bindComposerFileTargets, cleanupUploadedAttachments, createChatMedia, showTaskIncidentToast } from './chat_media.js';
|
||||
|
|
@ -1207,9 +1207,10 @@ export function createChatInstance({
|
|||
// Carrier facts precede presentation returns.
|
||||
if (row.tool_evidence && row.task_id) noteToolMetrics(row.task_id, row, row.ts || row.timestamp || '');
|
||||
if (row.system_type === 'task_evidence') {
|
||||
const record = liveCardRecords.get(row.task_id);
|
||||
if (!record) return false;
|
||||
// The task's own evidence: mints its card at the carrier's row (zero tools included).
|
||||
const record = getLiveCardRecord(row.task_id);
|
||||
const activity = activeDirectActivities.get(row.task_id);
|
||||
if (row.ts) reanchorTaskCard(record, row.ts);
|
||||
if (row.suggested_name) applySuggestedName(row.task_id, row.suggested_name);
|
||||
noteDirectTurn(record, activity ? activity.kind !== 'managed_task' : row._is_direct_chat);
|
||||
if (row.task_phase === 'finalizing') markLiveCardFinalizing(row.task_id, row);
|
||||
|
|
@ -1372,6 +1373,10 @@ export function createChatInstance({
|
|||
// P5: the cancelable marker may have arrived (scheduled progress frame /
|
||||
// history replay) before this card was minted.
|
||||
syncCancelRunButton(record);
|
||||
// Children that arrived before their parent's evidence nest under it now.
|
||||
for (const [childId, info] of subagentChildParents) {
|
||||
if (info.parentId === normalizedGroupId) ensureLiveCardVisible(liveCardRecords.get(childId));
|
||||
}
|
||||
return record;
|
||||
}
|
||||
|
||||
|
|
@ -1417,8 +1422,11 @@ export function createChatInstance({
|
|||
return Boolean(changed && record.root?.isConnected);
|
||||
}
|
||||
|
||||
// A parent card comes only from the parent's OWN evidence here (its record, a census
|
||||
// activity, a replayed row of its own), never a child's bare parent_task_id (A5).
|
||||
function ensureSubagentContainer(parentId = '', depth = 0) {
|
||||
if (!parentId) return null;
|
||||
if (!parentId || !(liveCardRecords.has(parentId) || activeDirectActivities.has(parentId)
|
||||
|| evidencedTaskIds.has(parentId))) return null;
|
||||
// A lineage-known parent is minted as the nested card ITS parent owns,
|
||||
// never as a root-shaped shell (#636); depth bounds a lineage cycle.
|
||||
const lineage = depth < 32 ? subagentChildParents.get(parentId) : null;
|
||||
|
|
@ -1439,13 +1447,15 @@ export function createChatInstance({
|
|||
return container;
|
||||
}
|
||||
|
||||
function getSubagentCardRecord(childId = '', parentId = '', role = '') {
|
||||
// A child's frame reanchors its parent's card (one this room has or just minted) at the frame's time.
|
||||
function getSubagentCardRecord(childId = '', parentId = '', role = '', rawTs = '') {
|
||||
const existing = liveCardRecords.get(childId);
|
||||
const childBefore = captureLiveCardProjection(existing);
|
||||
const parentBefore = captureLiveCardProjection(liveCardRecords.get(parentId));
|
||||
let record = null;
|
||||
withStableViewport(() => {
|
||||
record = getSubagentCardRecordMutation(childId, parentId, role);
|
||||
if (rawTs) reanchorTaskCard(liveCardRecords.get(parentId), rawTs);
|
||||
return liveCardProjectionChanged(childBefore, record)
|
||||
|| liveCardProjectionChanged(parentBefore, liveCardRecords.get(parentId));
|
||||
});
|
||||
|
|
@ -1521,13 +1531,11 @@ export function createChatInstance({
|
|||
{ suppressDomInsert = false, reorderExisting = false } = {},
|
||||
) {
|
||||
if (!record || suppressDomInsert || _syncPass1Active) return;
|
||||
if (record.isSubagent && record.parentGroupId) {
|
||||
// The container mints a lineage-known parent nested (#636); the
|
||||
// recursion carries the chain up to the inserted root.
|
||||
const container = ensureSubagentContainer(record.parentGroupId);
|
||||
if (container && record.root.parentNode !== container) {
|
||||
container.appendChild(record.root);
|
||||
}
|
||||
const container = record.isSubagent ? ensureSubagentContainer(record.parentGroupId) : null;
|
||||
if (container) {
|
||||
// The container nests a lineage-known parent (#636); the recursion
|
||||
// carries the chain up to the inserted root.
|
||||
if (record.root.parentNode !== container) container.appendChild(record.root);
|
||||
const parentRecord = liveCardRecords.get(record.parentGroupId);
|
||||
ensureLiveCardVisible(parentRecord);
|
||||
updateLiveCardCount(parentRecord);
|
||||
|
|
@ -1861,6 +1869,8 @@ export function createChatInstance({
|
|||
|
||||
// Child lineage retains model-less updates; child finality is independent.
|
||||
const subagentChildParents = new Map();
|
||||
// Tasks whose own rows this room replayed: a parent card's evidence (A5).
|
||||
const evidencedTaskIds = new Set();
|
||||
// Late progress must not revive a terminal child.
|
||||
const subagentTerminalChildren = new Set();
|
||||
|
||||
|
|
@ -2010,8 +2020,7 @@ export function createChatInstance({
|
|||
// Interrupted is retryable and therefore non-terminal; the canonical
|
||||
// projector owns that distinction for both live and replay paths.
|
||||
if (summary.terminal) subagentTerminalChildren.add(childId);
|
||||
reanchorTaskCard(getLiveCardRecord(parentId), rawTs);
|
||||
stampNodeTimestamp(getSubagentCardRecord(childId, parentId, role)?.root, rawTs, { anchor: true });
|
||||
stampNodeTimestamp(getSubagentCardRecord(childId, parentId, role, rawTs)?.root, rawTs, { anchor: true });
|
||||
const reviewsChanged = attachTaskDetailReviews(childId, evt);
|
||||
const updated = queueTaskLiveUpdate(
|
||||
summary,
|
||||
|
|
@ -2031,8 +2040,7 @@ export function createChatInstance({
|
|||
const content = String(msg?.content || msg?.text || '').trim();
|
||||
if (!content) return false;
|
||||
const rawTs = msg?.ts || new Date().toISOString();
|
||||
reanchorTaskCard(getLiveCardRecord(parentId), rawTs);
|
||||
const record = getSubagentCardRecord(childId, parentId, role);
|
||||
const record = getSubagentCardRecord(childId, parentId, role, rawTs);
|
||||
const preserveTerminal = Boolean(record?.finished && (ownLine || subagentTerminalChildren.has(childId)));
|
||||
const summary = summarizeSubagentCardFrame(msg, childId, {
|
||||
content,
|
||||
|
|
@ -2060,8 +2068,7 @@ export function createChatInstance({
|
|||
const { parentId, role } = info;
|
||||
const text = String(msg?.content || msg?.text || '').trim();
|
||||
const rawTs = msg?.ts || new Date().toISOString();
|
||||
reanchorTaskCard(getLiveCardRecord(parentId), rawTs);
|
||||
const record = getSubagentCardRecord(childId, parentId, role);
|
||||
const record = getSubagentCardRecord(childId, parentId, role, rawTs);
|
||||
const priorTerminalPhase = record?.finished ? String(record.phaseEl?.dataset?.phase || '') : '';
|
||||
const summary = summarizeSubagentCardFrame(msg, childId, {
|
||||
content: '',
|
||||
|
|
@ -2431,7 +2438,12 @@ export function createChatInstance({
|
|||
for (const entry of localEcho.confirmed) {
|
||||
localEchoJournal.delete(entry.clientMessageId);
|
||||
}
|
||||
for (const msg of messages) learnSubagentLineage(msg);
|
||||
for (const msg of messages) {
|
||||
learnSubagentLineage(msg);
|
||||
// A notice names its target, not a speaker: no evidence of that task here.
|
||||
if (msg.task_id && !ORIGIN_ADDRESSED_NOTICE_TYPES.has(msg.system_type)
|
||||
&& msg.system_type !== 'project_question_pointer') evidencedTaskIds.add(msg.task_id);
|
||||
}
|
||||
|
||||
const historicalTerminalProjections = new Set();
|
||||
for (const msg of messages) if (msg.historical_terminal && msg.task_id) {
|
||||
|
|
@ -2463,11 +2475,10 @@ export function createChatInstance({
|
|||
updateLiveCardFromProgressMessage(msg, { grantCancelAuthority: msg.project_mirror !== true });
|
||||
const historyCard = liveCardRecords.get(taskId);
|
||||
if (historyCard && !historyCard.isSubagent && !activeDirectActivities.has(taskId)
|
||||
&& !msg.task_terminal_status && msg.task_phase !== 'finalizing') {
|
||||
if (setHistoricalUnconfirmed(historyCard)) {
|
||||
syncCancelRunButton(historyCard);
|
||||
renderLiveCardMeta(historyCard);
|
||||
}
|
||||
&& !msg.task_terminal_status && msg.task_phase !== 'finalizing'
|
||||
&& setHistoricalUnconfirmed(historyCard)) {
|
||||
syncCancelRunButton(historyCard);
|
||||
renderLiveCardMeta(historyCard);
|
||||
}
|
||||
if (historyCard && historicalTerminals.has(taskId)) historyCard.historicalTerminal = historicalTerminals.get(taskId);
|
||||
continue;
|
||||
|
|
@ -2478,17 +2489,17 @@ export function createChatInstance({
|
|||
|
||||
// Pass 2 inserts cards at the first visible task message, then finishes them.
|
||||
const insertedCardTaskIds = new Set();
|
||||
function reorderDirtyCardIfNeeded(rec) {
|
||||
if (!rec?._anchorOrderDirty || rec.isSubagent || !rec.root?.isConnected) return;
|
||||
insertMessageNode(rec.root, { reorderExisting: true });
|
||||
rec._anchorOrderDirty = false;
|
||||
function mountReplayedCard(rec) {
|
||||
if (rec?._anchorOrderDirty && !rec.isSubagent && rec.root?.isConnected) {
|
||||
insertMessageNode(rec.root, { reorderExisting: true });
|
||||
rec._anchorOrderDirty = false;
|
||||
}
|
||||
ensureLiveCardVisible(rec);
|
||||
}
|
||||
function insertCardIfNeeded(taskId) {
|
||||
if (!taskId || insertedCardTaskIds.has(taskId)) return;
|
||||
insertedCardTaskIds.add(taskId);
|
||||
const rec = liveCardRecords.get(taskId);
|
||||
reorderDirtyCardIfNeeded(rec);
|
||||
ensureLiveCardVisible(rec);
|
||||
mountReplayedCard(liveCardRecords.get(taskId));
|
||||
}
|
||||
// A stored row as its bubble: its own identity and labels, plus what its kind adds.
|
||||
const addStoredRow = (msg, taskId, extra) => addMessage(msg.text, msg.role, !!msg.markdown, msg.ts || null, false, {
|
||||
|
|
@ -2543,21 +2554,15 @@ export function createChatInstance({
|
|||
insertCardIfNeeded(taskId); routeSubagentProgressToCard(taskId, msg, true); continue;
|
||||
}
|
||||
if (taskId && replayConcludesTask(msg)) {
|
||||
if (subagentChildParents.has(taskId)) {
|
||||
insertCardIfNeeded(taskId);
|
||||
routeSubagentFinalMessageToCard(taskId, msg);
|
||||
const record = liveCardRecords.get(taskId);
|
||||
finishLiveCard(taskId, msg.task_terminal_status ? taskTerminalPhase(msg) : replayTerminalPhase(record));
|
||||
continue;
|
||||
}
|
||||
insertCardIfNeeded(taskId);
|
||||
// A replayed early final must not finalize the card.
|
||||
if (msg.task_phase === 'finalizing') {
|
||||
markLiveCardFinalizing(taskId, msg);
|
||||
} else if (!msg.historical_terminal || positiveTaskTerminalFact(msg)) {
|
||||
const record = liveCardRecords.get(taskId);
|
||||
finishLiveCard(taskId, msg.task_terminal_status ? taskTerminalPhase(msg) : replayTerminalPhase(record));
|
||||
const child = subagentChildParents.has(taskId);
|
||||
if (child) routeSubagentFinalMessageToCard(taskId, msg);
|
||||
// A replayed early final must not finalize a root's card.
|
||||
if (!child && msg.task_phase === 'finalizing') markLiveCardFinalizing(taskId, msg);
|
||||
else if (child || !msg.historical_terminal || positiveTaskTerminalFact(msg)) {
|
||||
finishLiveCard(taskId, msg.task_terminal_status ? taskTerminalPhase(msg) : replayTerminalPhase(liveCardRecords.get(taskId)));
|
||||
}
|
||||
if (child) continue;
|
||||
}
|
||||
// A durable routing receipt retires Sending like its live frame.
|
||||
if (msg.chat_annotation && msg.client_message_id) {
|
||||
|
|
@ -2577,37 +2582,22 @@ export function createChatInstance({
|
|||
}
|
||||
_historyRow = null;
|
||||
// Persisted terminals without task_summary (crash/timeout/cancel) also settle
|
||||
// progress-only cards; otherwise reload revives them as Working.
|
||||
const terminalTaskRecords = new Map();
|
||||
for (const msg of messages) {
|
||||
const tid = msg.task_id || '';
|
||||
if (tid && msg.task_terminal_status) {
|
||||
terminalTaskRecords.set(tid, {
|
||||
...msg,
|
||||
status: String(msg.task_terminal_status),
|
||||
});
|
||||
}
|
||||
}
|
||||
// progress-only cards, or reload revives them as Working. A child's terminal
|
||||
// resolves the child card, never its parent; a root's uses the shared terminal
|
||||
// mapping (P5: a cancelled root replays as "Cancelled", never a generic "Done").
|
||||
const settleReplayedTerminal = (tid, terminalRecord, phase, summary = false) => {
|
||||
if (!taskDoneIsTerminal(terminalRecord)) return;
|
||||
if (subagentChildParents.has(tid)) return routeSubagentTerminalToCard(tid, terminalRecord);
|
||||
if (!liveCardRecords.has(tid)) return;
|
||||
insertCardIfNeeded(tid);
|
||||
if (summary) appendTaskSummaryToLiveCard(terminalRecord); else finishLiveCard(tid, phase);
|
||||
};
|
||||
const terminalTaskRecords = new Map(messages.filter((msg) => msg.task_id && msg.task_terminal_status)
|
||||
.map((msg) => [msg.task_id, { ...msg, status: String(msg.task_terminal_status) }]));
|
||||
for (const [tid, terminalRecord] of terminalTaskRecords) {
|
||||
_historyRow = terminalRecord;
|
||||
if (!taskDoneIsTerminal(terminalRecord)) continue;
|
||||
// Subagent terminal status resolves the child card, not the
|
||||
// parent. Otherwise reload can revive a crashed/cancelled child.
|
||||
if (subagentChildParents.has(tid)) {
|
||||
routeSubagentTerminalToCard(tid, terminalRecord);
|
||||
continue;
|
||||
}
|
||||
const rec = liveCardRecords.get(tid);
|
||||
if (rec) {
|
||||
insertCardIfNeeded(tid);
|
||||
if (terminalRecord.outcome_axes || terminalRecord.review_projection || terminalRecord.reason_code) {
|
||||
appendTaskSummaryToLiveCard(terminalRecord);
|
||||
} else {
|
||||
// P5: shared terminal mapping — a cancelled root replays
|
||||
// as "Cancelled", never as a generic "Done".
|
||||
finishLiveCard(tid, taskTerminalPhase(terminalRecord));
|
||||
}
|
||||
}
|
||||
settleReplayedTerminal(tid, terminalRecord, taskTerminalPhase(terminalRecord),
|
||||
Boolean(terminalRecord.outcome_axes || terminalRecord.review_projection || terminalRecord.reason_code));
|
||||
}
|
||||
_historyRow = null;
|
||||
|
||||
|
|
@ -2616,25 +2606,13 @@ export function createChatInstance({
|
|||
// which may still say finalizing; status alone loses failures.
|
||||
for (const [tid, historicalTerminal] of historicalTerminals) {
|
||||
if (!historicalTerminalProjections.has(tid)) continue;
|
||||
const terminalRecord = { ...historicalTerminal, task_id: tid };
|
||||
if (!taskDoneIsTerminal(terminalRecord)) continue;
|
||||
if (subagentChildParents.has(tid)) {
|
||||
routeSubagentTerminalToCard(tid, terminalRecord);
|
||||
continue;
|
||||
}
|
||||
const rec = liveCardRecords.get(tid);
|
||||
if (!rec) continue;
|
||||
insertCardIfNeeded(tid);
|
||||
finishLiveCard(tid, historicalTerminal.phase);
|
||||
}
|
||||
|
||||
// Every block the predicate admits is in the transcript after a rebuild.
|
||||
for (const rec of liveCardRecords.values()) {
|
||||
reorderDirtyCardIfNeeded(rec);
|
||||
ensureLiveCardVisible(rec);
|
||||
settleReplayedTerminal(tid, { ...historicalTerminal, task_id: tid }, historicalTerminal.phase);
|
||||
}
|
||||
|
||||
// Every block the predicate admits is in the transcript after a rebuild; a card
|
||||
// this replay minted with neither a terminal nor a census entry is unconfirmed.
|
||||
for (const [id, record] of liveCardRecords) {
|
||||
mountReplayedCard(record);
|
||||
if (!knownCards.has(id) && !record.finished && !activeDirectActivities.has(id)) {
|
||||
setHistoricalUnconfirmed(record);
|
||||
syncCancelRunButton(record);
|
||||
|
|
|
|||
|
|
@ -43,7 +43,6 @@ export const HOST_NOTICE_PREFIXES = new Map(Object.entries({
|
|||
// Progress rows: a task card's timeline line (`log_events.summarizeChatLiveEvent`).
|
||||
task_scheduled: ['🗓️ Scheduled subagent ', '🗓️ Scheduled task '],
|
||||
task_error: ['⚠️ Error: '],
|
||||
cancellation_notice: ['❌ cancel '],
|
||||
finalization_notice: ['⏳ Task ', '▶️ Task '],
|
||||
skill_review: ['✅ Skill review round ', '⚠️ Skill review round ', '❌ Skill review round ', '⏳ Skill review round '],
|
||||
host_progress: ['🌐 Connection restored ', '🌐 The new attempt could not reach the provider ',
|
||||
|
|
@ -57,6 +56,7 @@ export const HOST_NOTICE_PREFIXES = new Map(Object.entries({
|
|||
export const RETIRED_HOST_NOTICE_PREFIXES = new Map(Object.entries({
|
||||
attachment_notice: ['📎 Attachment staging report for ', '⚠️ Task not started: every attachment was rejected.',
|
||||
'⚠️ Some declared attachments could not be staged; '],
|
||||
cancellation_notice: ['❌ cancel '],
|
||||
task_admission_notice: ['✅ Repair task '],
|
||||
task_started: ['🧬 Evolution task ', '🔎 Review task '],
|
||||
host_progress: ['🌐 Could not establish a provider connection '],
|
||||
|
|
|
|||
115
web/tests/cancel_pending_stop.test.js
Normal file
115
web/tests/cancel_pending_stop.test.js
Normal file
|
|
@ -0,0 +1,115 @@
|
|||
// The owner's hard Stop answered 202 pending — the durable stop is recorded and
|
||||
// the supervisor finishes it (docs/CLIENT_CONTRACT.md §8). Driven through the
|
||||
// real card control and the shared requestStop: an ok:true answer is accepted,
|
||||
// task detail keeps the card «Cancelling…» and no «Cancel failed» toast is
|
||||
// raised. The control case pins that this harness does see that toast for a
|
||||
// Stop the server refuses, so the pending case is a real negative.
|
||||
import assert from 'node:assert/strict';
|
||||
import test from 'node:test';
|
||||
import { createChatInstance } from '../modules/chat.js';
|
||||
import { ElementStub, installDom, restoreDom, walkCard } from './chat_dom_fixture.js';
|
||||
|
||||
// The flat fixture's querySelector does not descend; the card's Stop trigger
|
||||
// lives in its actions row.
|
||||
const originalQuery = ElementStub.prototype.querySelector;
|
||||
ElementStub.prototype.querySelector = function (selector) {
|
||||
const direct = originalQuery.call(this, selector);
|
||||
if (direct) return direct;
|
||||
for (const child of this.children) { const found = child.querySelector(selector); if (found) return found; }
|
||||
return null;
|
||||
};
|
||||
|
||||
const collect = (node, predicate, out = []) => {
|
||||
if (predicate(node)) out.push(node);
|
||||
for (const child of node?.children || []) collect(child, predicate, out);
|
||||
return out;
|
||||
};
|
||||
|
||||
// A browser click: both controls stop propagation on their own event.
|
||||
const press = (node) => {
|
||||
for (const fn of node.listeners.get('click') || []) fn({ target: node, stopPropagation() {}, preventDefault() {} });
|
||||
};
|
||||
|
||||
async function stopNowAnsweredWith(taskId, cancelStatus, cancelBody) {
|
||||
const calls = [];
|
||||
const env = installDom(async (url, init = {}) => {
|
||||
calls.push(`${init.method || 'GET'} ${url}`);
|
||||
if (String(url) === `/api/tasks/${taskId}/cancel`) {
|
||||
return { ok: cancelStatus < 400, status: cancelStatus, json: async () => cancelBody };
|
||||
}
|
||||
if (String(url) === `/api/tasks/${taskId}`) {
|
||||
return { ok: true, status: 200,
|
||||
json: async () => ({ task_id: taskId, status: 'running', cancel_state: 'pending' }) };
|
||||
}
|
||||
return { ok: true, json: async () => ({ active_direct_turns: [] }) };
|
||||
});
|
||||
const priorObserver = globalThis.IntersectionObserver;
|
||||
globalThis.IntersectionObserver = class { observe() {} disconnect() {} takeRecords() { return []; } };
|
||||
const body = new ElementStub('body', globalThis.document);
|
||||
body.isConnected = true;
|
||||
globalThis.document.body = body;
|
||||
const handlers = new Map();
|
||||
let generation = 0;
|
||||
const instance = createChatInstance({
|
||||
ws: { on(type, fn) { handlers.set(type, fn); return () => handlers.delete(type); },
|
||||
isConnected: () => true, send() {} },
|
||||
state: { activePage: 'chat', projectChatIds: new Set(), unreadCount: 0 },
|
||||
updateUnreadBadge() {},
|
||||
stateSnapshots: {
|
||||
begin: () => ({ generation: ++generation, requestedAt: Date.now() }),
|
||||
gate() { return Promise.resolve(this.begin()); },
|
||||
isCurrent: () => true, apply() {},
|
||||
},
|
||||
chatId: 1, idPrefix: 'chat', mountEl: env.mount,
|
||||
});
|
||||
try {
|
||||
handlers.get('chat')({
|
||||
chat_id: 1, task_id: taskId, role: 'assistant', is_progress: true, cancelable: true,
|
||||
content: 'Working on it.', ts: '2026-10-14T20:00:00Z',
|
||||
});
|
||||
const card = walkCard(globalThis.document.byId.get('chat-messages'), taskId);
|
||||
const trigger = card.querySelector('[data-cancel-run]');
|
||||
assert.ok(trigger, 'the host-attested marker offers Stop');
|
||||
press(trigger);
|
||||
const [stopNow] = collect(body, (node) => node?.dataset?.taskControl === 'stop_now');
|
||||
assert.ok(stopNow, 'the dropdown offers Stop now');
|
||||
press(stopNow);
|
||||
const detailRead = `GET /api/tasks/${taskId}`;
|
||||
for (let i = 0; i < 50 && !calls.includes(detailRead); i += 1) {
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
}
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
const toasts = collect(body, (node) => node?.classList?.contains('toast')).map((node) => node.textContent);
|
||||
return {
|
||||
calls, toasts, triggerDisabled: trigger.disabled, finished: card.dataset.finished,
|
||||
phase: card.querySelector('[data-live-phase]')?.textContent,
|
||||
};
|
||||
} finally {
|
||||
instance.destroy();
|
||||
globalThis.IntersectionObserver = priorObserver;
|
||||
restoreDom(env.prior);
|
||||
}
|
||||
}
|
||||
|
||||
test('a 202 pending Stop keeps the card Cancelling… with no Cancel failed toast', async () => {
|
||||
const seen = await stopNowAnsweredWith('pending-root', 202, {
|
||||
ok: true, task_id: 'pending-root', cancel_state: 'pending', stop_policy: 'immediate',
|
||||
request_id: 'ci_pending', reason_code: 'teardown_unconfirmed', cascade: true,
|
||||
});
|
||||
assert.ok(seen.calls.includes('POST /api/tasks/pending-root/cancel'), seen.calls);
|
||||
assert.ok(seen.calls.includes('GET /api/tasks/pending-root'), 'the card reconciles from task detail');
|
||||
assert.deepEqual(seen.toasts.filter((text) => /Cancel failed/.test(text)), []);
|
||||
assert.equal(seen.phase, 'Cancelling…');
|
||||
assert.equal(seen.finished, '0', 'only the settled task_done closes the card');
|
||||
assert.equal(seen.triggerDisabled, true, 'a hard Stop stays pending; the trigger is not re-armed');
|
||||
});
|
||||
|
||||
test('control: a Stop the server refuses raises the Cancel failed toast in the same harness', async () => {
|
||||
const seen = await stopNowAnsweredWith('refused-root', 503, {
|
||||
error: 'durable cancel intent could not be recorded; nothing was cancelled — retry',
|
||||
task_id: 'refused-root', reason_code: 'cancel_intent_write_failed',
|
||||
});
|
||||
assert.ok(seen.calls.includes('POST /api/tasks/refused-root/cancel'), seen.calls);
|
||||
assert.equal(seen.toasts.filter((text) => /^Cancel failed: durable cancel intent could not be recorded/.test(text)).length, 1,
|
||||
seen.toasts);
|
||||
});
|
||||
|
|
@ -111,6 +111,8 @@ test('inert tool carrier restores a speechless card without invented narration o
|
|||
assert.ok(f.card());
|
||||
assert.match(f.rows().map(row => row.innerHTML).join(' '), /1 tool call/);
|
||||
assert.equal(f.card().querySelector('[data-live-phase]').hidden, true, 'history alone confers no liveness');
|
||||
assert.match(f.meta(), /Activity unconfirmed/, 'the hidden chip is replaced by the meta label');
|
||||
assert.match(f.meta(), /1 tool call/);
|
||||
assert.equal(f.card().querySelector('[data-resume-run]'), null);
|
||||
assert.equal(f.card().querySelector('[data-cancel-run]'), null);
|
||||
assert.equal(f.card().querySelector('[data-live-title]').textContent, '');
|
||||
|
|
|
|||
|
|
@ -1458,3 +1458,108 @@ test('a late acceptance settlement row without a placement fact never rewrites t
|
|||
assert.match(added[added.length - 1].innerHTML, /Reviewers later passed this answer/);
|
||||
} finally { instance?.destroy(); restoreDom(prior); }
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// A5: a parent card comes only from the parent's own evidence in this room; a
|
||||
// child whose parent has none stays visible on its own and nests later.
|
||||
// ---------------------------------------------------------------------------
|
||||
const CHILD_ROW = {
|
||||
chat_id: 2, role: 'system', is_progress: true, system_type: 'subagent_terminal_notice', narration: false,
|
||||
task_id: 'kid-x', subagent_task_id: 'kid-x', parent_task_id: 'par-x', root_task_id: 'par-x',
|
||||
delegation_role: 'subagent', subagent_event: 'cancelled', task_terminal_status: 'cancelled',
|
||||
content: 'Stopped by Ouroboros', text: 'Stopped by Ouroboros', ts: '2026-10-14T12:10:20Z',
|
||||
history_id: 'progress:120', history_position: { source: 'progress', offset: 120 },
|
||||
};
|
||||
const PARENT_CARRIER = {
|
||||
chat_id: 2, role: 'system', system_type: 'task_evidence', is_progress: false, narration: false, text: '',
|
||||
task_id: 'par-x', ts: '2026-10-14T12:10:20Z', task_terminal_status: 'cancelled', outcome_phase: 'cancelled',
|
||||
outcome_final: true, outcome_axes: { lifecycle: { status: 'cancelled' } },
|
||||
tool_evidence: { observations: [], coverage: { shown: 0, matched: 0 } },
|
||||
};
|
||||
|
||||
function a5Instance(pages) {
|
||||
let page = 0;
|
||||
const { prior, mount } = installDom(async (url) => {
|
||||
if (!String(url).startsWith('/api/chat/history')) return { ok: true, json: async () => ({ active_direct_turns: [] }) };
|
||||
const rows = pages[Math.min(page, pages.length - 1)];
|
||||
page += 1;
|
||||
return { ok: true, json: async () => ({ messages: rows, window: { complete: true } }) };
|
||||
});
|
||||
const handlers = new Map();
|
||||
const ws = { on(type, fn) { handlers.set(type, fn); return () => handlers.delete(type); }, isConnected: () => true, send() {} };
|
||||
let generation = 0;
|
||||
const stateSnapshots = {
|
||||
begin: () => ({ generation: ++generation, requestedAt: Date.now() }), gate() { return Promise.resolve(this.begin()); },
|
||||
isCurrent: () => true, apply() {},
|
||||
};
|
||||
const instance = createChatInstance({
|
||||
ws, state: { activePage: 'chat', projectChatIds: new Set([2]), unreadCount: 0 }, updateUnreadBadge() {},
|
||||
stateSnapshots, chatId: 2, idPrefix: 'chat', mountEl: mount, asPanel: true,
|
||||
});
|
||||
const messages = () => globalThis.document.byId.get('chat-messages');
|
||||
const topCards = () => messages().children.filter((n) => n.classList.contains('chat-live-card')).map((n) => n.dataset.taskId);
|
||||
return { instance, handlers, prior, messages, topCards };
|
||||
}
|
||||
|
||||
test('a child of a foreign or unknown parent is its own visible card, never a parent shell (A5)', async () => {
|
||||
const { instance, handlers, prior, messages, topCards } = a5Instance([[CHILD_ROW]]);
|
||||
try {
|
||||
await instance.refreshHistory({ revision: 1 });
|
||||
assert.deepEqual(topCards(), ['kid-x'], 'the child renders at the top level');
|
||||
assert.equal(walkCard(messages(), 'par-x'), null, 'no card is minted from a bare parent_task_id');
|
||||
const child = walkCard(messages(), 'kid-x');
|
||||
assert.equal(child.dataset.finished, '1');
|
||||
assert.equal(child.querySelector('[data-live-phase]')?.dataset.phase, 'cancelled');
|
||||
// Live-frame parity: the same lineage on a live frame mints no parent either.
|
||||
handlers.get('chat')({ ...CHILD_ROW, task_id: 'kid-y', subagent_task_id: 'kid-y', parent_task_id: 'par-y',
|
||||
root_task_id: 'par-y', history_id: undefined, history_position: undefined, ts: '2026-10-14T12:11:00Z' });
|
||||
assert.equal(walkCard(messages(), 'par-y'), null);
|
||||
assert.ok(walkCard(messages(), 'kid-y'), 'the live child is visible');
|
||||
assert.deepEqual(topCards(), ['kid-x', 'kid-y']);
|
||||
} finally { instance.destroy(); restoreDom(prior); }
|
||||
});
|
||||
|
||||
test('an in-room parent with a terminal carrier is a finished card with the child nested (A5)', async () => {
|
||||
const { instance, prior, messages, topCards } = a5Instance([[CHILD_ROW, PARENT_CARRIER]]);
|
||||
try {
|
||||
await instance.refreshHistory({ revision: 1 });
|
||||
assert.deepEqual(topCards(), ['par-x'], 'the parent is the one top-level card');
|
||||
const parent = walkCard(messages(), 'par-x');
|
||||
assert.equal(parent.dataset.finished, '1', 'the carrier finished the parent');
|
||||
assert.equal(parent.querySelector('[data-live-phase]')?.dataset.phase, 'cancelled', 'its real phase, not Done');
|
||||
assert.notEqual(parent.dataset.inert, '1', 'not Activity unconfirmed');
|
||||
assert.equal(walkCard(messages(), 'kid-x').parentNode?.dataset?.subagentsFor, 'par-x');
|
||||
} finally { instance.destroy(); restoreDom(prior); }
|
||||
});
|
||||
|
||||
test('an in-room parent whose carrier has no terminal stays unconfirmed, never Done (A5)', async () => {
|
||||
const carrier = { ...PARENT_CARRIER, task_terminal_status: undefined, outcome_phase: undefined, outcome_final: undefined,
|
||||
outcome_axes: undefined, task_phase: 'unfinished' };
|
||||
const { instance, prior, messages } = a5Instance([[CHILD_ROW, carrier]]);
|
||||
try {
|
||||
await instance.refreshHistory({ revision: 1 });
|
||||
const parent = walkCard(messages(), 'par-x');
|
||||
assert.ok(parent, 'the carrier is the parent\'s own evidence');
|
||||
assert.notEqual(parent.dataset.finished, '1');
|
||||
assert.equal(parent.dataset.inert, '1', 'liveness unconfirmed in this room');
|
||||
assert.match(parent.querySelector('[data-live-meta]')?.innerHTML || '', /Activity unconfirmed/,
|
||||
'the summary row says so, never a blank card');
|
||||
assert.equal(walkCard(messages(), 'kid-x').parentNode?.dataset?.subagentsFor, 'par-x');
|
||||
} finally { instance.destroy(); restoreDom(prior); }
|
||||
});
|
||||
|
||||
test('a parent whose evidence arrives later re-nests its child, keeping the node (A5)', async () => {
|
||||
const parentRow = { chat_id: 2, role: 'system', is_progress: true, system_type: 'host_progress', task_id: 'par-x',
|
||||
content: 'Parent planning', text: 'Parent planning', narration: false, ts: '2026-10-14T12:00:00Z',
|
||||
history_id: 'progress:100', history_position: { source: 'progress', offset: 100 } };
|
||||
const { instance, prior, messages, topCards } = a5Instance([[CHILD_ROW], [CHILD_ROW, parentRow]]);
|
||||
try {
|
||||
await instance.refreshHistory({ revision: 1 });
|
||||
const child = walkCard(messages(), 'kid-x');
|
||||
assert.deepEqual(topCards(), ['kid-x']);
|
||||
await instance.refreshHistory({ revision: 2 });
|
||||
assert.deepEqual(topCards(), ['par-x'], 'the parent\'s own row mints its card');
|
||||
assert.equal(walkCard(messages(), 'kid-x'), child, 'the child keeps its node identity');
|
||||
assert.equal(child.parentNode?.dataset?.subagentsFor, 'par-x', 'and nests under the parent');
|
||||
} finally { instance.destroy(); restoreDom(prior); }
|
||||
});
|
||||
|
|
|
|||
|
|
@ -86,7 +86,7 @@ test('typed terminal status drives an error phase on live and replay cards', ()
|
|||
assert.equal(taskDoneIsTerminal(failed), true);
|
||||
assert.match(
|
||||
chatSource,
|
||||
/finishLiveCard\(taskId, msg\.task_terminal_status \? taskTerminalPhase\(msg\) : replayTerminalPhase\(record\)\);/,
|
||||
/finishLiveCard\(taskId, msg\.task_terminal_status \? taskTerminalPhase\(msg\) : replayTerminalPhase\(liveCardRecords\.get\(taskId\)\)\);/,
|
||||
);
|
||||
assert.match(chatSource, /appendTaskSummaryToLiveCard\(msg\) \|\| changed;/);
|
||||
});
|
||||
|
|
@ -117,7 +117,7 @@ test('interrupted task_done remains retryable and cannot finish a root card', ()
|
|||
phase: 'working', headline: 'Working',
|
||||
});
|
||||
assert.match(chatSource, /if \(eventType === 'task_done' && summary\.terminal\)/);
|
||||
assert.match(chatSource, /if \(!taskDoneIsTerminal\(terminalRecord\)\) continue;/);
|
||||
assert.match(chatSource, /if \(!taskDoneIsTerminal\(terminalRecord\)\) return;/);
|
||||
assert.equal(taskDoneIsTerminal({
|
||||
status: 'completed',
|
||||
root_phase_checkpoint: { post_task_synthesis: 'running' },
|
||||
|
|
@ -377,9 +377,10 @@ test('history replay keeps open summaries live and terminal fallbacks factual',
|
|||
dataset: { phase: 'error' },
|
||||
} }), 'error');
|
||||
assert.equal(replayTerminalPhase({}), 'done');
|
||||
// One replay-conclusion site finishes a child's and a root's card alike (A5 fold).
|
||||
assert.equal(
|
||||
[...chatSource.matchAll(/finishLiveCard\(taskId, msg\.task_terminal_status \? taskTerminalPhase\(msg\) : replayTerminalPhase\(record\)\);/g)].length,
|
||||
2,
|
||||
[...chatSource.matchAll(/finishLiveCard\(taskId, msg\.task_terminal_status \? taskTerminalPhase\(msg\) : replayTerminalPhase\(liveCardRecords\.get\(taskId\)\)\);/g)].length,
|
||||
1,
|
||||
);
|
||||
assert.doesNotMatch(
|
||||
chatSource,
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue