diff --git a/docs/design/2026-10-02-hosted-replaceable-harness.md b/docs/design/2026-10-02-hosted-replaceable-harness.md index 75c8a82c07..9b94af484d 100644 --- a/docs/design/2026-10-02-hosted-replaceable-harness.md +++ b/docs/design/2026-10-02-hosted-replaceable-harness.md @@ -2,8 +2,9 @@ [English](2026-10-02-hosted-replaceable-harness.md) | [简体中文](2026-10-02-hosted-replaceable-harness.zh-CN.md) -Status: Steps 1 and 2 implemented in this PR (D1-D9); D10's Step 3 rows are -named follow-ups. Issue tracker: #12952 (Stage G). Code references are to +Status: Steps 1 and 2 are implemented (D1-D9). Step 3's bounded native-file +approval and main-model slices are implemented on this branch; the broader +acceptance matrix remains open, as recorded in the [Step 3 design](2026-10-09-hosted-g3-step3-recovery.md). Issue tracker: #12952 (Stage G). Code references are to `main` @ `728c13de21` unless a decision notes a later refinement. This design answers Q2 and Q3 of #12952 following the proposal in its comments, and corrects one mechanism description from that proposal (see "Current @@ -515,7 +516,7 @@ happen. `hosted_harness_protocol_error` — instead of failing every Session open with `managed_session_open_failed`. -### D10 — Step 3 stays a named follow-up, minus its cheap row +### D10 — Step 3 bounded slices are implemented; the wider matrix remains open D4 already delivers the proposal table's first row (submission attempted, never admitted → re-submitted after withdrawal). The remaining rows — model @@ -531,6 +532,16 @@ NOT close on the model-round slice alone: the Boundaries bullet keeps #12952's Q2 half open until a separate multi-instance control-plane tracker proves it, so closing waits on both. +The [G3 Step 3 detailed design](2026-10-09-hosted-g3-step3-recovery.md) +refines these follow-ups: retain requested Actions and drive the saved batch +after Allow/Deny; Cancel/expiry settle only with safe ownership evidence. +It also defines saved effective model requests, bounded prefix retraction, +durable cleanup and cursor gates. The detailed design records the current +working-tree implementation and its verified scope; its full acceptance matrix +remains broader than the five packaged gates completed so far. D4's null epoch +proves a missing admission reply, with safety from original-command journal +idempotency and the parked-input refusal as specified in D4 above. + ## Changes and ownership | Layer | Files | Change | diff --git a/docs/design/2026-10-02-hosted-replaceable-harness.zh-CN.md b/docs/design/2026-10-02-hosted-replaceable-harness.zh-CN.md index 9adba6a05d..b3503a5dfb 100644 --- a/docs/design/2026-10-02-hosted-replaceable-harness.zh-CN.md +++ b/docs/design/2026-10-02-hosted-replaceable-harness.zh-CN.md @@ -2,8 +2,9 @@ [English](2026-10-02-hosted-replaceable-harness.md) | [简体中文](2026-10-02-hosted-replaceable-harness.zh-CN.md) -状态：Step 1 与 Step 2 已在本 PR 实现（D1-D9）；D10 的 Step 3 各行是 -点名的后续。跟踪 issue：#12952（Stage G）。代码引用以 `main` @ +状态：Step 1 与 Step 2 已实现（D1-D9）。Step 3 的限定原生文件审批与 +主模型切片已在当前分支实现；更广的验收矩阵仍未完成，详见 +[Step 3 设计](2026-10-09-hosted-g3-step3-recovery.zh-CN.md)。跟踪 issue：#12952（Stage G）。代码引用以 `main` @ `728c13de21` 为准，除非某个决策注明了更晚的修正。本设计按 issue 评论 中的提案回答了 #12952 的 Q2 与 Q3，并修正了该提案中一处机制描述 （见「现状」第 1 条）。 @@ -433,7 +434,7 @@ darwin 是一个未被验证的预期，而不是等待首个不可能发生的 `hosted_harness_protocol_error`——而不是让每个 Session 的 open 都以 `managed_session_open_failed` 失败。 -### D10 —— Step 3 保留为点名后续，去掉其中便宜的一行 +### D10 —— Step 3 限定切片已实现，更广的验收矩阵仍开放 D4 已经交付提案表格的第一行（尝试过提交但从未准入 → 撤回后重新 提交）。其余各行——带半截文本回撤的模型轮重发（`before_model`；机制 @@ -447,6 +448,13 @@ profile）、以及 `turn_settled` 的换绑续读——是架在 D5 类型化 d 保持开放，留待一个独立的多实例控制面 tracker 证明——关闭须两边同时 成立。 +[G3 第三步详细设计](2026-10-09-hosted-g3-step3-recovery.zh-CN.md)细化这些 +后续切片：保留 requested Action，Allow/Deny 后驱动保存的批次，Cancel/expiry +仅在所有权证据证明安全时结算；同时定义保存有效模型请求、有界 prefix 撤回、 +持久清理和游标门禁。详细设计记录当前工作区实现和已验证范围；完整验收矩阵 +仍大于目前已完成的五种打包门禁。D4 的 null epoch 仅证明 admission 回答 +缺失，安全性来自上文 D4 指定的原命令 journal 幂等与停靠输入拒绝。 + ## 改动与属主 | 层 | 文件 | 改动 | diff --git a/docs/design/2026-10-09-hosted-g3-step3-recovery.md b/docs/design/2026-10-09-hosted-g3-step3-recovery.md new file mode 100644 index 0000000000..c0fa711781 --- /dev/null +++ b/docs/design/2026-10-09-hosted-g3-step3-recovery.md @@ -0,0 +1,573 @@ +# Hosted G3 Step 3 — recover parked Turns after Harness replacement + +[English](2026-10-09-hosted-g3-step3-recovery.md) | [简体中文](2026-10-09-hosted-g3-step3-recovery.zh-CN.md) + +Status: implemented on the current branch, 2026-10-09; five scoped Linux packaged acceptance modes passed against immutable snapshot `linux-rebased-gates-3`. Rebased source baseline: upstream `main` at `4a3b8f3c084a1aa82af6cd75684f26123cdc64be`; the snapshot includes production commit `dca809caf4526ade87b12dc46a0797cf744dc0d6`. Part of [#12952][g]; follows the existing G3 native-file recovery work. This does not close the broader G series or Q2 fencing gates. + +The implementation produces native-file approval plans for native-file profiles without Hooks or MCP (packaged gates cover `hosted-workspace-files/1`), exact main-model request snapshots without Hooks, bounded cold-stream retractions, and native Turn cleanup obligations. Cold model reissue is admitted only without Hooks or MCP. Shell profiles retain their existing execution/cleanup owner. Live prefix continuations remain supported, but an unfinished prefix continuation has no cold-reissue snapshot and declines recovery. A committed final answer with a completed main attempt can still receive terminal-only compensation without reissuing inference. Oversize snapshots deliberately remain unsupported. The production protocol and readers are updated together in this tree; the gates below state the required acceptance evidence. + +Exact request snapshots are produced by the native Turn driver for initial admission, approval resume and model redrive. The legacy Runtime-only continuation route retains its existing G1 recovery behavior and does not produce these snapshots. Interrupted channel settlement reconciles original G3 cleanup debt and preserves a cleanup-only block when release fails; request-preparation failures keep both ordinary prompt and wake inputs unsettled for recovery. + +## 1. Problem and existing behavior + +G3 Steps 1–2 let a live Java control plane adopt a replacement Hosted Harness. +Step 3 must finish the original Turn when the crash occurred in model inference +or an approval wait, and preserve already committed terminal events. Recovery +must keep the original Session, Prompt, authority and side-effect identities. + +| Baseline behavior | What Step 3 still owes | +| ------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------- | +| The connector renegotiates a new Harness generation; journal writer fencing and Harness-only restart gates are merged. | Keep that fencing before every recovery write and every new tool admission. | +| `await_runtime` / `results_ready` file-tool recovery queries original executions and consumes their results. | Reuse it; do not rebuild or dispatch those calls from model output. | +| A requested approval can attach, and resolve commits its decision before replying. | A replacement has no original in-memory waiter. It must drive or settle the parked Turn after the durable wait ends. | +| `resolveDurableWait()` advances `await_action` to `model_output_committed`. | This checkpoint transition does not execute the approved batch or finish the Turn. | +| No-tool and first-model-round drive loads decline with `model_start`. | Recover a proven model request boundary instead of starting an ordinary fresh Turn. | +| The load route can compensate a projectable `turn_settled` checkpoint; Java changes the event epoch without advancing the consumed cursor. | Complete fault and lost-response coverage; do not replace these mechanisms with a new projection protocol. | +| Cancellation-only loads can settle some parks with no unpaid Runtime work. | Cover requested approvals, expiry and post-decision crashes without certifying an unobserved stop. | + +Source anchors: `hosted-runtime-recovery.ts` (`recoverHostedRuntimeTurn`), +`hosted-harness-session.ts` (`executeHostedTurn`, `settleCancelledHarnessTurn`, +the load and Action resolve routes), `hosted-tool-approval.ts` +(`resolveHostedAction`), `managed-harness-factory.ts` (`resolveDurableWait`), +and Java `HarnessCoordinator` (`runClaimed`). These are baseline facts, +not new acceptance results. + +`model.attempt: output_committed` currently records a finished provider stream +before the final assistant message is durably written. It is not proof of a +committed answer. A fresh model invocation also starts its loop at round zero; +Step 3 must preserve the original remaining rounds and absolute +`input.accepted.deadline`. + +## 2. Scope and invariants + +This design covers successive Hosted Harness generations reachable through one +Java control plane. Ordinary local Managed activation, two live control-plane +owners, cross-host takeover and Shell execution takeover remain separate. + +- A successful load means the parked Turn has a registered driver, a durable + approval wait with an expiry path, or a payable terminal projection. Merely + making the Session readable is insufficient. +- A model request may be reissued; a possibly dispatched tool, Hook, MCP call + or child task may not be recreated to make recovery progress. +- The journal is authoritative for admission, Action state, execution receipts + and settlement. Java owns public command delivery and its consumed event + cursor; the current fenced Harness owns model and tool orchestration. +- Keep the original `promptId`, user record, Action IDs, model message IDs for + committed outputs, and execution IDs for prepared work. A new inference + attempt gets a new attempt/message identity, not a new Prompt. +- A fresh Prompt, undo and cwd change stay refused while the original Turn or + a cleanup obligation still owns the corresponding resource. Status and + cancellation must report the parked Turn truthfully. +- Store faults are retryable faults, not evidence that a resource or execution + is absent. Missing, corrupt or conflicting durable evidence fails closed. +- A Managed failure never falls back to Legacy. Stale Workspace authorization, + changed capability/configuration pins or writer loss never grant execution. + +## 3. Recovery decisions come from evidence, not phase names + +`HARNESS_MODEL_START_PHASES` is a run-admission vocabulary, not a recovery +allowlist. In particular, `model_output_committed` may mean an approved tool +batch is still owed; `results_ready` owns receipts; `turn_settled` owes only +projection. A bootstrap `before_model` checkpoint with a null Turn is not a +snapshot of the currently accepted model request. + +| Durable evidence for the original Turn | Proposed behavior | +| ------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Submission was marked but Java has no admission epoch/reply. | Keep G3 D4's original-command reconciliation. A null epoch proves only a missing reply: journal command idempotency and the parked-input refusal make this safe. No Step 3 resubmission shortcut. | +| Complete main-model request snapshot; unfinished attempt; no committed output or unsettled side effect for that request. | Reissue that model request under a new attempt after fencing and retraction. | +| Complete assistant tool-call message, but batch has no complete approval continuation evidence. | Do not reissue the model or run the ordinary tool entry. Keep the existing recovery refusal. | +| `await_action`, Action still `requested`, complete continuation evidence. | Retain the same Action, restore its waiter/expiry and cancellation driver. | +| Action `decided` with `allow` or `deny`, complete continuation evidence. | Drive the exact saved batch; apply the saved decision once. Deny becomes a paired tool refusal, then normal model continuation. | +| Action `cancelled` / `expired`, or an admitted Turn cancellation wins before fresh dispatch. | Close the durable wait and settle the original Turn as cancelled, after proving no unpaid effects. | +| Partial prepare or intent evidence exists after an Action decision. | Reconcile original reservations and complete the original batch checkpoint before entering G1/G2; never prepare under replacement IDs. | +| Complete `await_runtime` / `results_ready` execution evidence exists. | Recover the original executions through G1/G2. No fresh prepare under replacement IDs. | +| `results_ready` with complete original tool results. | Existing result continuation; never rerun the tool-producing model round. | +| `turn.settled` already committed but not consumed by Java. | Rebind the event epoch and replay from Java's consumed cursor; no inference or tool execution. | +| `turn_settled` checkpoint is payable but terminal record is missing. | Existing terminal projection compensation, with idempotent write and replay. | +| In-flight Shell, unknown external work, incomplete Hook/MCP/child evidence or ambiguous multiple parked Turns. | Existing typed decline/block; keep the affected ownership fence. | + +The model and approval cases need small, producer-specific durable resources. +There is no new universal recovery domain, public resume API or configuration +matrix. + +## 4. Slice A — finish approval waits (B2) + +### 4.1 Save the continuation before publishing the wait + +The existing `approval.invocationRef` saves one call's input, and its +`optionsRef` identifies the Action. Neither alone preserves a mixed batch, +earlier decisions, rewritten inputs or the point at which a Hook asked again. +Save an immutable `hosted-approval-continuation/1` resource before +`commitDurableWait()`. Proposed Action options version 3 references it through +`continuationRef`; retain `invocationRef`'s current meaning. + +| Continuation content | Required purpose | +| ---------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | +| Session key, original Prompt, assistant message, definition/config/profile/policy pins and source activation. | Reject foreign, stale or differently configured continuation. | +| Ordered call IDs, names, effective input refs/digests, tool definitions, stable `runtimeCallId` and prepare idempotency keys/references. | Reconstruct the original batch without inventing arguments or IDs. | +| Current Action ID, approval ordinal and whether the wait is before PreToolUse or a reapproval after input rewriting. | Resume at the right point without granting a later input an earlier approval. | +| Earlier decisions/refusals and completed permission/Hook evidence references. | Keep decisions and side effects already paid; do not repeat Hooks. | +| Original Runtime owner identity and any existing prepare/receipt evidence. | Distinguish a pre-dispatch plan from work that must go through G1/G2. | +| History boundary and model/Hook continuation inputs. | Continue from the committed assistant call message, without another user record or inference of the same call batch. | + +The Action options, descriptor and nested input references must be present in +the same durable wait transaction. A failure publishes neither a recoverable +wait nor a claim that its inputs are complete. All refs stay inside the same +Session; existing resource and transaction limits apply. + +The first approval snapshot is not necessarily the final execution plan: +PreToolUse may rewrite input and allocate a new `runtimeCallId`, even without +another approval. After Hooks and decisions complete, before file-history or +prepare effects, publish and journal-link the latest immutable effective batch +revision. Use a narrow `hosted.batch.planned` event with `batchId`, +`planRevision` and `planRef`, referencing the same continuation format. Save the +final model-call/runtime-call IDs, normalized payload refs/digests, definitions, +refusals/decisions, original owner and complete Broker prepare reference/key. +Recovery selects that revision, not the initial approval descriptor. No +file-history or prepare effect may precede this durable link. If a crash occurs +before a final plan exists, resume only from proven completed Hook/decision +stages; incomplete producer evidence keeps the refusal. + +Version 3 keeps the existing preview policy: `inputRef` is present only for +tools eligible for the native bounded preview; `continuationRef` is private +recovery data and is never returned as a public Action field. Both the Java +Action projector and TS options reader must support v1/v2/v3 before a writer +emits v3. + +Measure the serialized approval and final batch plans before publication against the Store's 64 KiB inline resource limit. If a plan is oversized, omit that recovery resource and keep ordinary live approval and execution working with the existing v1/v2 options. Such an approval has no new continuation recovery guarantee. A Store publication failure for a fitting plan still blocks; it is not a size fallback. + +Once an approval lacks a captured plan, omit all later plans in that assistant batch, even if a final plan would fit. Recovery also declines any batch containing a durable v1/v2 approval before choosing a final or earlier v3 plan; otherwise a mixed batch could repeat an existing approval or select an unreadable plan. This check starts at the current assistant message's committed sequence, so an earlier fallback does not disable a later batch. + +### 4.2 Restore a driver, then honor the durable winner + +1. Acquire the new writer activation and restore the exact Session/profile, + original Runtime owner and continuation evidence. Check cancellation and + current authorization before permitting fresh effects. +2. Register one parked-Turn owner before load answers. `/status` remains in + flight; `/prompt` cannot overwrite the park. Resolve, cancel, expiry and + repeated loads join the same per-Session driver. Its in-memory latch is only + single-process scheduling; journal CAS and original dispatch IDs provide + crash idempotency. +3. For a requested Action, keep its original ID and deadline. Register its + waiter and one expiry timer, then re-read the Action to cover a decision + that won before registration. No new approval is minted for the same input. +4. For Allow/Deny, verify the original decision digest, input revision and + policy revision. Advance the wait once, restore the saved execution stage, + and continue the saved batch. Reevaluate changed effective inputs against + the applicable policy and obtain a new approval when required; a saved Allow + never authorizes rewritten arguments by itself. +5. Resume completed Hook occurrences from their original records. Admit a new + occurrence only when the next stage was proven not started. An uncertain + Hook/MCP result or existing tool intent diverts to its recovery path, never + to a fresh invocation. Producer adapters without complete evidence decline. + For native files, a crash among prepare calls or between prepare and + `tool.intent` needs batch reconciliation before G1: retry/query prepare with + the saved `runtimeCallId`, original idempotency key/reference and digests; + recover matching `PREPARED` reservations; append missing original intents; + commit the complete original `await_runtime` batch; then enter G1. Broker + prepare uses `dispatch = false` and exact-key replay returns the original + execution receipt. Do not call `prepareRequests()` to reconstruct the plan: + it generates new random IDs. A conflict or unknown reservation blocks. Never + dispatch before the complete batch and required approval/Hook evidence are + durable. +6. For expiry or cancellation, record the Action's final state by CAS, close + `await_action`, pair any committed unanswered call records truthfully, then + write one cancelled Turn terminal. Expiry takes this cancelled outcome; + it does not start another model request after the restart. An already + committed decision wins the Action race; a separate admitted Turn cancel + still prevents fresh dispatch and settles/stops whatever it actually owes. +7. Release the exact Runtime lease only after safe settlement. A failed release + remains a durable cleanup obligation as specified in Section 6, recoverable + after another crash. Never release an uncertain Hook or descendant writer + based on an empty tool list. +8. Repeated resolve/load/cancel after a crash reads the committed decision, + checkpoint, original intents and terminal before acting. It neither asks + again nor executes under a replacement ID. + +The resolve route continues to commit a decision before its response. An Action +response operation can complete when that decision is durably projected; this +does not claim the Turn or tool finished. A failure after decision commit must +not rewrite a successful Action response as a failed decision. Java keeps its +original operation/key and the Turn event consumer running; browser terminal +tracking such as #13609 is a separate consumer of that operation. + +Expiry must work without a browser response: load installs a deadline timer +and immediately processes an already elapsed deadline. Every reattach checks +the same saved deadline, so another process crash cannot reset it. + +### 4.3 Profile and older-record boundaries + +Native files are the first acceptance target. Private Shell before dispatch, +MCP, Hooks and child-agent batches require their own positive continuation and +ownership evidence; Shell after dispatch retains its refusal. Do not enable +H3, public Shell or other domains as part of B2. + +Older v1/v2 waits remain readable. Safe cancel/expire is allowed only when the +old journal and producer records prove all unpaid effects absent or settled. +An old Allow/Deny without a complete recoverable batch keeps the existing +refusal; it is not silently upgraded by reconstructing live memory. Legacy +reconstruction, if later desired, is a separately validated compatibility +slice. Completion claims must name the supported producer/profile versions. + +## 5. Slice B — recover an unfinished main model round + +### 5.1 Persist the request before sending it + +Add an optional `recoveryRef` to a main `model.attempt` event, pointing to an +immutable `hosted-model-request/1` resource. The event's existing `routeRef` +and budget information keep their meanings. Hook-model attempts do not become +reissuable. Do not repurpose `checkpoint.attempt.routeRef`, which currently +also describes tool/approval input, or change the checkpoint schema. + +| Request snapshot | Required purpose | +| --------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | +| Session, original Prompt, attempt, source activation, request digest and journal/history boundary. | Prove which admitted request is being replaced. | +| Original effective Prompt/request Parts and normalized model-visible history/system instruction refs. | Restore Hook-modified input, compaction and already consumed results without reapplying startup Hooks. | +| Pinned model route, sampling, definition/config/capability/profile revisions and tool declaration digest. | Reissue under the same contract; changed or unavailable pins fail closed. | +| Workspace context revision/content ref and completed Hook-context references. | Avoid reading a different Workspace or new instructions from the Harness host. | +| Original budget state, round ordinal and request kind. | Preserve turn limits across replacement instead of resetting the 16-round loop or cost budget. | +| Ownership/continuation evidence for prior results and producer obligations. | Prove that no unconsumed side effect is being skipped or recreated. | + +Capture the prepared, authentication-free provider request through a narrow, +awaitable callback at the final logical request boundary in `LlmChat`, after +automatic compaction, tool-history repair and output-token adjustment, before +the generator sends it. A Hosted-layer copy made before `sendMessageStream()` +is insufficient. The Hosted caller supplies this callback for every main +request; ordinary callers keep their current behavior. Snapshot publication and +`model.attempt: started` commit must finish before the network request. Move the +main attempt's existing pre-send `started` write into this callback so it names +the request actually sent. After tools, take the next snapshot from durably +committed/settled results and completed Hook outcomes. Preserve current result +consumption timing: it advances only after successful model `Finished`, not +before this request. Never store model credentials, authorization headers or +OAuth tokens; current credentials resolve the pinned route. Resource references +carry digests and length checks, with the existing inline/transaction limits. +If the descriptor or new request data cannot fit without a new chunking/storage +mechanism, mark this request unrecoverable, write an old-compatible attempt +without `recoveryRef` and keep ordinary live inference working. After a crash +it retains `model_start` refusal; never claim recovery for that request. A Store +fault is not such a size classification and must not silently downgrade the +durability barrier. No OSS spill or new chunking mechanism is added here. + +Save the normalized generator parameters, history state and original stream +owner namespace. The saved-request entry calls the pinned generator with those +parameters; it skips `sendMessageStream()`'s normal history repair, compaction, +context injection and Prompt preparation. Retain normal response processing, +usage recording, cancellation and tool-output validation. Once that response commits, the next +logical round returns to the normal loop with restored history and budgets. +A byte-equivalent transport retry can reuse the snapshot. Fallback to another +route, reactive compaction or token adjustment that changes the effective +request requires a new immutable snapshot and attempt boundary before send; +never overwrite a ref or attribute a fallback request to the previous attempt. +This does not increase the original logical-round allowance. Interrupted +compaction or Hook-model work without complete evidence retains refusal. + +Before send, bind each attempt to its own pre-assigned stream message ID; keep +that mapping with the saved attempt identity, separate from the normalized +request digest. Restore the latest committed usage/model-metrics ledger as well +as the snapshot's budget baseline, so a finished stream's recorded usage is not +forgotten when its assistant commit was lost. Unknown usage from a crashed +provider call cannot be reconstructed; repeat inference can incur additional +provider cost, and exact external billing limits are not a recovery guarantee. + +Recovery retains the absolute deadline already committed in +`input.accepted.deadline`; preparing a replacement request does not start a new +`deadlineMs` interval. If admission committed but no prepared-request snapshot +did, the first model slice retains refusal. Covering that earlier window would +require a separately durable initial-turn preparation plan and is not implied +by request snapshots. + +### 5.2 Reissue only the saved unfinished request + +1. Find the one unsettled original Prompt and its latest main-model request. + Verify request identity, referenced history and pins, current writer and + absence of committed output or unpaid producer effects for that attempt. + `initial` or a bootstrap checkpoint alone is insufficient proof. +2. If an assistant output/call message was durably committed, recover its + outstanding work or terminal projection instead. Never discard an accepted + call batch by reissuing its producing model request. +3. Mark a former `started` attempt abandoned under its original identity; + preserve a recorded finished-stream state when it already exists. The new + request snapshot names the replaced attempt in either case. Restore its + streamed message IDs, original boot/epoch namespace and bounded delta + sequences from the journal and saved owner evidence, and commit each + required `message.retracted` before new output. A new empty + `HostedTextDeltaStream` cannot retract its predecessor's remembered prefix. +4. Start a new attempt for the same saved request, with a fresh message ID and + reference to the same request digest. Lost retraction responses or another + crash between retraction and send replay the same journal commands. A crash + after send can cause another model inference call; exactly-once inference + and identical regenerated answers are not promised. +5. Invoke the saved normalized-request entry with restored history, request kind, + budget and round ordinal. Do not call ordinary `executeHostedTurn` as a + fresh Prompt: it can write another user record and rerun Prompt Hooks. + No-tool Sessions use this same saved-request path with empty tool declarations. +6. Admit newly produced tools through the normal approved, durable tool path. + Previously committed calls/results stay in history with their original + identities. Cancellation observed before send does not send; cancellation + after send aborts inference and prevents fresh tool admission. + +Extend journal retraction with optional `sourceBootId`, +`sourceEventEpoch` and `throughSequence` on `message.retracted`, alongside the +existing message ID and `fromSequence`. Validate the original namespace and +bounded message range against journal/owner evidence. Retract both the prefix +already projected under its original namespace and copies of the same journal +message/range replayed under later attachment namespaces. The journal reader +also applies it to that message, so cold replay cannot resurrect it. Current +in-band retraction selects only the current boot/epoch; it cannot remove an +already consumed old-generation prefix. Do not call broad +`retractContinuationOutput()`, which removes all old Turn/epoch deltas, including +earlier committed rounds. Retraction and consumed-cursor advance are atomic and +idempotent. Test consumed prefixes, unprojected prefixes and prefixes split +across namespaces; retain all prior committed rounds. + +Keep the existing same-generation retry semantics. This slice permits +replacement-generation retraction with fenced evidence; it does not make a +published same-generation continuation automatically replayable. + +Unfinished old attempts without `recoveryRef` keep the existing `model_start` refusal. A completed main attempt followed by a durable final assistant answer can use terminal-only compensation even without a request snapshot; this never reissues inference. +The new snapshot producer must run for every supported main-model request; +adding a reader or optional parameter without populating it is not delivery. + +## 6. Slice C — close terminal projection and event-cursor gaps + +Reuse `settleProjectablePromptId` / `runSettleProjection`, the sink's terminal +identity and Java's existing epoch CAS. This slice starts as fault coverage, +with only failures it demonstrates receiving production changes. + +- A committed `turn.settled` is replayed, never recomputed. A missing terminal + record is compensated only from complete projectable checkpoint/output and + producer evidence. An Action decision alone is not a completed Turn. +- Rebinding changes boot ID and event epoch, while keeping Java's last + **consumed** journal sequence. The attachment tail is an observation, not a + consumption receipt; never jump the cursor to that tail. +- Retraction is journaled before the new attempt's deltas. Java/public replay + and browser refresh remove the former prefix and keep the replacement + answer, with one terminal event. Retain the existing `fromSequence` semantics; + do not delete older committed rounds or fabricate message IDs on recovery. +- Replaying an already admitted prompt/continue/cancel returns its original + admission watermark, not the current tail. Duplicate terminal delivery + advances the consumer idempotently without another tool or model call. +- Terminal commit and physical cleanup are separate facts. Failed lease + release, file-history marker retirement or producer cleanup stays owed; + Java must not reinterpret a completed Turn as an error, and a new Turn must + not bypass the affected resource fence. Cold restore must find the obligation. + +Use one narrow `hosted.cleanup` journal event with `cleanupId`, +`descriptorRef` and state `owed` / `confirmed`, referencing an immutable +`hosted-turn-cleanup/1` descriptor. It names the original Turn, Runtime +Session/binding owner and, when applicable, native file-history marker IDs. +This is turn cleanup bookkeeping, not a general task domain. Commit `owed` +only when original ownership actually owes cleanup; a no-tool Turn does not +acquire a Runtime Session for this protocol. Save the obligation +before adopting/acquiring the recovered lease and before a terminal/release +path can lose that identity. Cleanup waits for journal and producer evidence of +safe settlement. Release uses the original `runtimeSessionId`; only Broker's +`released = true` and confirmation of every named marker permit committing +`confirmed` under the same cleanup ID. Releasing an already `RELEASED` Runtime +Session is idempotent and does not require acquiring it again. Reconcile partial +success and lost replies by querying/retrying those identities. Owners with +still-active work, unknown evidence, missing identity or binding drift remain +fenced. A failure to save `owed` prevents the +new acquisition/release; an already committed terminal remains authoritative. + +Cold load and lifecycle-close reconciliation enumerate unresolved +`hosted.cleanup` events even without an `unsettledPromptId`, restore the +original owner and retry only eligible cleanup. Keep the affected +Workspace/resource fence until confirmation. A requested approval still owns +its lease and must not be released merely because this marker exists. +In-memory `runtimeLeaseHeld` / `refusedAdoptions` may schedule retries but are +not durable evidence. Missing old cleanup identity or uncertain Hook/descendant +stop evidence blocks cleanup without relabelling a committed terminal. + +## 7. Protocol ownership and compatibility + +No new public REST/WebShell route is needed. Existing private load, resolve, +cancel and events routes stay scoped to the live Session owner; Workspace and +Runtime calls stay inside the persisted binding and original execution owner. +None may fall back to the primary runtime on an unresolved scope. + +The simplest control-plane path is the existing plain attachment plus event +replay: the Harness owns restored model/approval drivers and Java consumes +their journal. Do not disguise them as a Runtime recovery report containing +invented executions, or add a generic public recovery-kind switch. The existing +Runtime report/continue/cancel protocol remains for G1 work. + +The Harness advertises `hosted_approval_resume_v1` and +`hosted_model_round_recovery_v1`; the Java SDK exposes these negotiated features. +They describe support rather than select a new public recovery route. Existing +DRIVE loads inspect durable evidence in the Harness; Store version acknowledgement +and the reader floor enforce format compatibility. An older compatible stack +retains the existing refusal. Once a +Session contains new Step 3 record formats, enforce its reader floor before an +older Harness can attach, not after parsing fails. +Use the existing `qwen_managed_session_journal_head.storage_version` as the +monotonic floor: readers support 1 and 2; the transaction that first commits +v3 options, final batch plans, model recovery data, bounded cross-generation +retraction or cleanup promotes the head to 2 atomically with that record. Keep +the checkpoint schema at 1; the storage envelope's reading contract changes. + +Carry the `maxReadableStorageVersion` contract through one HTTP header, +`X-Qwen-Managed-Max-Readable-Storage-Version` (omitted means 1; new readers send +2). The shared TS Store client sends it on every head-scoped request: acquire, +renew, restore, transaction enumeration, resource read, commit, publication, +recovery and lifecycle operations, including requests outside the JSON wrapper. +The Store first checks its supported head formats, then the caller's ceiling +before lease access, reads or writes. A promoting commit checks ceiling 2 under +the head lock. Check before idempotent replay as well; an old still-valid token +cannot bypass the floor. No new grant table or token capability state is needed. +Replay returns the original commit; the floor never moves backward on rollback or deletion of +a transient model state. Validate the writer/lease and head CAS in the same +promotion transaction. The current old Java `requireHeadScope` and old TS +restore both reject a storage version other than 1; the new protocol must also +reject an old Harness against a new Java Store before it obtains a writer lease. +Reader/Store support for all Step 3 formats lands before any producer, even +when only one feature is advertised for execution. This uses an existing SQL +column, not a new table or a guessed Flyway reservation. Synchronize the actual +frozen contract revision on the landing base. + +Writer acquire/renew replies advertise `supportedStorageVersion: 2`; missing support means 1 to the TS client. A new-format commit requires that acknowledgement and uses `/transactions:commit-v2`, which shares the new Store's atomic implementation. An old Store has no such endpoint and refuses the commit even if a prior acquire reached a new server. This closes the reverse mixed-deployment direction: older Stores otherwise accept arbitrary model-attempt payloads without promoting the head. Renewal refreshes the acknowledgement. Publication-token Worker submissions keep their own original admission protocol. + +All new refs must be followed by TS/Java validation, snapshot/restore, +Hosted-layout verification, W1b bundle enumeration, W1c profile/layout checks +and retention readers before producer enablement. Keep the 64 KiB ordinary +inline resource and 8 MiB transaction limits; shared/missing/corrupt/foreign nested +refs must not pass a reference census. No physical GC or new storage backend +is enabled here. + +## 8. Delivery order and dependencies + +| Independent slice | Deliverable and exit condition | +| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| G3.3-A, B2 | Options/continuation readers first, producer and parked approval driver second. Requested → Allow/Deny/Cancel/expiry, lost replies and post-decision crashes finish the original native-file Turn or truthfully block; next Turn and close work once safe cleanup settles. | +| G3.3-B, model rounds | Main request snapshots and reader floor, then model reissue/retraction for tool-enabled and no-tool Sessions. Original input is admitted once; prior tools execute once; model budget/history survive two replacements. | +| G3.3-C, terminal closure | Real process/cursor and cleanup fault gates around existing projection. Land bounded production fixes only if those gates expose gaps. | + +The C gates apply to A and B before either is accepted; C is not permission to +ship an earlier slice with known cursor or cleanup errors. A final C-only PR is +appropriate only for independent remaining coverage or a bounded proven fix. + +All slices are separately reviewable and revertible before new-record producer +enablement. After new records exist, rollback must honor their reader floor; +source revert alone cannot make their old reader compatible. + +Rebase each implementation onto actual landing main. [#13188][g1fix] owns +post-merge G1 state/cancellation/cleanup fixes; depend on its landed contracts +or explicitly reconcile the necessary overlap instead of copying the branch. +Do not require public Shell, H3 enablement or ordinary local M6 to start native +file/no-tool Step 3 work. H1/H2/H4 producer evidence is required only for the +corresponding recovery case. Broader Q2 binding fencing remains a separate gate +and keeps the overall Stage G tracker open. + +## 9. Verification and acceptance + +The implementation adds deterministic fixture barriers to the packaged-stack +runner. Use real Java/MySQL, Harness, Broker and Runtime operations; the model +fixture counts requests and tools and controls exact crash boundaries. A model +fixture is sufficient for orchestration proofs, not real-provider acceptance. +Run Harness-only replacement first, preserving Spring and its production +lease/timeouts; separately exercise replacement of both owners on Linux. + +| Gate | Required observations | +| ---------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| A1 requested approval → Allow / Deny | Same Action/input/decision identity; Allow dispatches once, Deny dispatches zero times for that call; original Turn settles and a later Turn completes. | +| A2 Cancel / expiry / no browser | Deadline is not reset; wait closes, one cancelled terminal and truthful paired records; `close 202` and next Turn succeed after ownership settles. | +| A3 mixed calls and rewritten inputs | Earlier decisions persist; pending sibling IDs/order survive; reapproved inputs never inherit an earlier Allow; completed Hooks are not rerun. | +| A4 decision/load responses lost | Original response operation/key settles once; one driver, no new approval; replay does not duplicate effects. | +| A5 crash after decision, wait advance, final plan, prepare and result commit | Each barrier resumes the correct stage. Final rewritten plan, original IDs and Runtime status evidence win over the initial approval descriptor. | +| A5a partial prepare / missing intent | Reconcile native reservations under original IDs/digests; complete batch checkpoint precedes dispatch/G1. Conflicts block; no replacement IDs or duplicate file effects. | +| A6 MCP/Hook/child uncertainty | Missing or unknown producer evidence blocks; no external call is recreated and no Workspace lease is silently released. | +| B1 first round, with/without tools | Exactly one original admission and user record; new inference attempt can complete; no-tool needs no Broker dispatch. | +| B2 streamed prefix and two replacements | New attempt/stream identities; durable retraction before new deltas; live view, reload and replay expose only the surviving answer and one terminal. | +| B3 after earlier tools | Request uses saved results/history; each previous execution occurs once; round and cost budget do not reset. | +| B3a Finished versus durable output | Crash after stream Finished but before assistant commit can reissue from complete evidence; crash after assistant commit adds no inference. Never use the attempt state name as output durability proof. | +| B4 committed assistant call output | No model reissue over unpaid calls; drive saved approval/batch or existing Runtime recovery. | +| B5 snapshot faults, pin drift and cancellation | Missing/corrupt/oversized/foreign refs or config/authorization drift cause no new effects; store faults retry; cancellation prevents fresh dispatch. | +| B6 fallback / reactive compaction | Changed normalized parameters get a new snapshot/attempt before send; equivalent retries reuse the snapshot. Saved-request reissue does not prepare or inject context again; logical-round budget is preserved. | +| C1 terminal committed, not delivered | Boot/epoch changes with the consumed cursor unchanged; terminal replays once; zero inference/tool calls. | +| C2 checkpoint payable, terminal missing | Compensation succeeds exactly once through crash/lost reply; no invented result on corrupt evidence. | +| C3 cleanup fails, then another crash | Original cleanup identity remains discoverable and fenced; retry discharges it; completed Turn stays completed. | +| C4 original admission response lost | Original command/prompt is reconciled, not re-admitted; original admission watermark replays. | +| R1 frozen former writer / unsupported engine | Former writer cannot append or dispatch; typed unsupported-engine refusal and no Legacy replay remain. This does not certify a surviving former Spring's binding fencing. | +| R2 old/new readers and storage bundles | v1/v2 and old model attempts stay readable with honest refusal; floor refuses incompatible acquire, renew, reads, writes and replay, including old tokens; bundle/restore reference closure passes. | + +For every physical/tool assertion, inspect actual files, SQL execution rows, +journal IDs and model fixture traffic. Removing the dispatch-count, cursor or +next-Turn assertion must fail its gate; an HTTP 200 or green unit test alone is +not acceptance. Keep initial failed fixture runs and report actual evidence +boundaries. Tests of local JSONL/H2 are additional coverage, not MySQL/Linux +physical-stop evidence. + +### 9.1 Verified implementation evidence + +The immutable `linux-rebased-gates-3` package passed Allow, Deny, Cancel, expiry and model continuation on the first attempt of each mode after rebase and integration fixes. The environment was Linux aarch64 with Node 22.22.1, Java 21.0.12.1 and MySQL 8.4.11. The portable Node bundle and Java JAR were built on macOS and executed on Linux. The actual Java control plane, Broker and Runtime Worker survived while only the Harness was replaced; inference used a controlled local OpenAI fixture. All 1387 artifact hashes matched on both hosts after the gates; production source stayed unchanged. Earlier `linux-delivery-gates-2` results remain historical evidence from the preceding main baseline. + +Allow retained the original Action, prepare key, request digest and Runtime binding, produced one settled execution with dispatch generation 1, and wrote the expected file bytes. Deny, Cancel and expiry produced no execution and no file; expiry retained the original deadline. Continuation retained the original execution, reissued the same normalized provider request body and removed the previously visible partial output. Every mode confirmed original-owner cleanup, completed a subsequent Turn, observed exactly two total terminals for those two Turns, then completed close admission 202 and reached closed Session status. + +Focused verification passed 841 core tests and 111 shared-contract tests. The expanded CLI run passed 779 tests and exposed three invalid legacy approval-fixture records; correcting their policy, choice labels and resource kind made all 60 tests in the affected suite pass. The repeated 13-file run passed 781 of 782 tests, with one `ENOTEMPTY` temporary-directory teardown failure in the unchanged Hosted model integration suite. All eight tests in that suite passed on a standalone rerun, so all 782 tests have passing evidence across scoped runs; the expanded batch is not claimed fully green. Fresh Java verification passed 1522 Server tests with one skipped, plus 55 SDK and 159 Broker tests. Build, typecheck, bundle, source ESLint and all three Java Checkstyle checks passed. Source lint excludes generated Maven `target` and test `coverage` artifacts. Failed diagnostics from the earlier baseline and expanded runs are retained. + +Integration regressions proved that model request preparation failures preserve admitted, unsettled inputs through both the ordinary prompt route and the wake driver, and that interrupted channel settlement confirms the original cleanup owner only after its terminal and file-history retirement. A first release failure retained the cleanup fence; passive resident attachment then released the same binding/generation and added one confirmation without a second terminal. A separate Spring/H2 probe passed the bounded-retraction/replay-floor combination: readers remained usable while the snapshot was rebuilding, and old cursors expired again after snapshot coverage caught up. This probe is not MySQL or process-takeover acceptance. + +The machine-readable reports, manifests and retained failed diagnostics are under `.qwen/investigations/g3-step3-implementation/`; the current consolidated report is `packaged-verification.md`. These five packaged gates do not prove the whole matrix above. First no-tool request crash, double replacement, partial-prepare crash, response loss, malformed/foreign refs, snapshot overflow and cleanup crash have focused or fault-probe coverage only, or remain broader packaged acceptance work. Full-owner restart, real-provider acceptance, a real mixed-version cluster and Q2 former-control-plane fencing remain separate. + +## 10. Implementation surface + +| Area | Expected paths / responsibilities | +| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| CLI orchestration | `hosted-harness-session.ts`, `hosted-runtime-recovery.ts`: registered parked driver, model recovery entry, cancel/expiry dispatch, projection reuse and original ownership. | +| Approval / native tools | `hosted-tool-approval.ts`, `hosted-workspace-tool-turn.ts`: options v3, continuation producer/reader, exact batch resume and no Hook/argument replay. | +| Model and streaming | `hosted-harness-model.ts`, `hosted-text-deltas.ts`, core `core/llm-chat.ts` and its call-options types: prepared-request callback, saved-request continuation and journal-seeded retraction. | +| Core journal | `managed-hook-activation.ts`, `managed-session-records.ts`, shared schema/fixtures and `http-managed-session-store.ts`: optional main-attempt recovery ref, final batch plans, bounded retraction, turn-cleanup records, recursive reference closure and per-request version ceiling; keep checkpoint v1 unchanged. | +| Java control plane/store | `ManagedActionStore`, Session-store validation/reference readers, `QwenHostedHarnessConnector`, `HarnessCoordinator`: v3 projection, reader floor and plain-attach/cursor contract; `ActionResponseCoordinator` retains decision-operation semantics. | +| Other readers / producers | Hosted layout/W1b/W1c readers; Hook, MCP, child and file-history adapters only where their continuation evidence is supported. Audit every declared field's producer and every read site. | +| Verification / documentation | Collocated tests, packaged E2E runner/CI, paired designs and developer event schema. No public contract promotion or deployment enablement without its own acceptance. | + +## 11. Decisions and remaining implementation questions + +Decisions: finish B2 before widening model recovery; preserve requested Actions +across restart; Allow/Deny drive the saved batch; Cancel/expiry settle safely; +reissue only a durable main-model request; use plain attachments and the +existing journal feed; retain checkpoint v1 and existing Runtime recovery; +do not close the Stage G tracker on this functional work alone. + +Before a slice emits its new records, the implementation must test the +persisted storage-version floor and complete nested-ref readers. Before +claiming a producer supported, prove its saved inputs, completed occurrences, +original owner and stop/settlement evidence. These are implementation gates, +not permission to fill missing evidence from the current environment. Real +provider acceptance, multi-instance binding fencing, public Shell enablement, +cross-host stop/ownership proof and production event pruning remain separate. + +## 12. References + +- [Stage G tracker #12952][g] and [architecture proposal #12380][proposal]. +- [Merged G3 Steps 1–2 #13174][g3] and [explicit B2 follow-up decision][b2]. +- [G3 Steps 1–2 design](2026-10-02-hosted-replaceable-harness.md). +- [Actions design](2026-09-30-managed-agent-actions.md). +- [Hosted Hooks design](2026-09-30-managed-hooks-runtime.md). +- [Delivery ledger](managed-agent-delivery-ledger.md). +- [G1 post-merge fixes #13188][g1fix]. + +[g]: https://github.com/QwenLM/qwen-code/issues/12952 +[proposal]: https://github.com/QwenLM/qwen-code/issues/12380 +[g3]: https://github.com/QwenLM/qwen-code/pull/13174 +[b2]: https://github.com/QwenLM/qwen-code/pull/13174#issuecomment-6030365600 +[g1fix]: https://github.com/QwenLM/qwen-code/pull/13188 + +### Review corrections + +A local cancel aborts immediately. If its durable append fails, the route returns 503: the current process stops, but the cancellation is not durable and must be retried before relying on replacement recovery. Cleanup tracks the requested Turn separately from other Turns' debt and clears a held lease only after its original owner is confirmed released. Wake input IDs retain the same path-safe Runtime mapping as ordinary acquisition. Transient cleanup failures continue to fence new work until a successful attachment retry clears the cleanup block. + +Prepared-request capture bounds defensive cloning at 64 KiB; oversized borrowed views are synchronously serialized and rejected by the Hosted publisher rather than retained. Recovered generation configuration lasts only for its request, while history restoration runs the usual registry reconciliation. Hosted closure validation shares one reference memo and one verified-resource cache per transaction; reference metadata conflicts and the complete transaction census remain enforced. + +## Review correction: request intent and cleanup liveness (2026-10-12) + +Prepared requests now carry an optional boolean `allowPersistentRetry` that records primary versus fallback retry eligibility. Replay also requires the current unattended-retry opt-in; it does not persist credentials or enable persistent retries for a fallback. The TS and Java readers accept the legacy six-field prepared shape and the new seven-field shape, with a boolean type check and rejection of unknown fields. Legacy snapshots retain bounded retries because they contain no evidence of primary eligibility. This changes the unmerged G3 nested codec; it does not establish compatibility with earlier revisions of this PR or relax the released G2 reader-floor gate. + +A history containing externalized image references depends on an in-memory payload table that is absent from the durable snapshot. Its ordinary online request keeps the existing image reattachment behavior, but the capture callback marks the snapshot unsupported. The cold reader and direct prepared replay also decline earlier unsafe image-reference snapshots before inference. A completed, committed final answer may still receive terminal-only compensation; this is not image-history restoration. + +Approval continuation validates prepare keys and Runtime Session references using the same path-safe mapping as acquisition, while keeping the original raw prompt identity. Cleanup identity remains unchanged for short IDs; longer compound IDs use a digest so both owed and confirmed events fit the stable-ID byte limit. Historical owed records retain their payload identity and descriptor; only an over-limit confirmation event/command ID is shortened. The descriptor continues to name the original prompt, binding and generation. + +A confirmed cleanup clears only its cleanup block, and successful unblock transitions retire that flag so a later unrelated recovery block is not cleared accidentally. A descriptorless current Runtime returning the exact `404 runtime_session_not_found` refusal is an already-absent handback; original-owner descriptor releases still require confirmation. Closing an adopted Session completes local teardown after a release transport fault, retaining unconfirmed durable debt for later reconciliation. A parked lifecycle request is refused before stamping a closing claim, leaving its ordinary recovery controls operable. Resident model redrive holds the existing recovery busy fence across asynchronous reads and verifies registration before attachment. + +Recovery honors an already-durable cancellation before installing the restored deadline timer. An earlier cancellation therefore stays cancellation after the wall-clock deadline passes; a deadline with no durable cancellation still expires normally. The R3 correction is verified separately from the historical packaged acceptance snapshot above; that snapshot does not certify this new revision. diff --git a/docs/design/2026-10-09-hosted-g3-step3-recovery.zh-CN.md b/docs/design/2026-10-09-hosted-g3-step3-recovery.zh-CN.md new file mode 100644 index 0000000000..70c47488c7 --- /dev/null +++ b/docs/design/2026-10-09-hosted-g3-step3-recovery.zh-CN.md @@ -0,0 +1,476 @@ +# Hosted G3 第三步：Harness 替换后恢复停靠的 Turn + +[English](2026-10-09-hosted-g3-step3-recovery.md) | [简体中文](2026-10-09-hosted-g3-step3-recovery.zh-CN.md) + +状态：已在当前分支实现，2026-10-09；不可变快照 `linux-rebased-gates-3` 的五种限定范围 Linux 打包验收模式全部通过。rebase 后的源码基线为 upstream `main` 的 `4a3b8f3c084a1aa82af6cd75684f26123cdc64be`；快照包含生产提交 `dca809caf4526ade87b12dc46a0797cf744dc0d6`。属于 [#12952][g]，延续已有 G3 原生文件恢复工作。本次不关闭更广的 G 系列或 Q2 fencing 门禁。 + +当前实现为不含 Hooks/MCP 的原生文件 profile 生产审批计划（打包门禁覆盖 `hosted-workspace-files/1`），为不含 Hooks 的主模型请求保存精确快照，并实现冷恢复流的有界撤回及原生 Turn 清理债务。主模型冷重发仅在不含 Hooks/MCP 时准入。Shell profile 保留现有执行和清理 owner。运行中的前缀 continuation 继续可用，但尚未完成的前缀 continuation 不保存冷重发快照，恢复时明确拒绝。主模型 attempt 已完成且最终回答已提交时，仍可只补终态而不重发推理。超限快照保留明确不支持的边界。协议生产者和读取者在本次工作区一同实现；下文门禁定义所需的验收证据。 + +精确请求快照由原生 Turn driver 在首次准入、审批恢复和模型 redrive 路径生产。旧 Runtime-only continuation 路由保留现有 G1 恢复行为，不生产这些快照。渠道中断结算对账原 G3 清理债务，并在释放失败时保留仅由清理债务引起的阻塞；请求准备失败时，普通 prompt 和 wake 输入均保持未结算，留待恢复。 + +## 1. 问题与已有行为 + +G3 第 1–2 步让仍存活的 Java 控制面能够接纳替代 Hosted Harness。第三步需要在 +崩溃发生于模型推理或审批等待时完成原 Turn，并保留已经提交的终态事件。恢复必须 +沿用原 Session、Prompt、权限主体与副作用身份。 + +| 基线行为 | 第三步仍需补齐 | +| --------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | +| connector 会重新协商新的 Harness 代数；journal writer fencing 与仅重启 Harness 的门禁已合入。 | 每次恢复写入与新工具准入之前仍需通过该 fencing。 | +| `await_runtime` / `results_ready` 文件工具恢复查询原执行并消费原结果。 | 复用已有路径；不得从模型输出重新构造并派发这些调用。 | +| requested 审批可以附着，resolve 在回答前提交决定。 | 替代进程没有原来的内存 waiter，必须在持久等待结束后驱动或结算停靠的 Turn。 | +| `resolveDurableWait()` 把 `await_action` 推进至 `model_output_committed`。 | 这一 checkpoint 转换不会执行批准后的批次，也不会完成 Turn。 | +| 无工具会话和首轮模型的 drive load 以 `model_start` 拒绝。 | 恢复有证据的模型请求边界，不得作为普通新 Turn 启动。 | +| load 已能补偿可投影的 `turn_settled` checkpoint；Java 更换事件 epoch 时不推进已消费游标。 | 补齐故障与丢响应覆盖，保留现有机制，不另造投影协议。 | +| cancellation-only load 能结算部分没有未付 Runtime 工作的停靠状态。 | 覆盖 requested 审批、过期与决定后的崩溃，不认证未经观测的停止。 | + +源码定位：`hosted-runtime-recovery.ts` 的 `recoverHostedRuntimeTurn`， +`hosted-harness-session.ts` 的 `executeHostedTurn`、 +`settleCancelledHarnessTurn`、load 与 Action resolve 路由， +`hosted-tool-approval.ts` 的 `resolveHostedAction`， +`managed-harness-factory.ts` 的 `resolveDurableWait`，以及 Java +`HarnessCoordinator` 的 `runClaimed`。以上是基线事实，不是新的验收结果。 + +当前 `model.attempt: output_committed` 在 provider 流结束后、最终 assistant 消息 +持久写入前记录，因此不证明答案已经提交。新的模型调用也会把循环从第零轮开始； +第三步必须保留原剩余轮数与绝对的 `input.accepted.deadline`。 + +## 2. 范围与不变量 + +本设计覆盖同一个 Java 控制面可连接的先后 Hosted Harness 代数。普通本地 +Managed 激活、两个存活的控制面 owner、跨主机接管与 Shell 执行接管仍另行处理。 + +- 成功 load 意味着停靠 Turn 有已注册的 driver、带过期处理的持久审批等待，或 + 可偿付的终态投影。仅让 Session 可读并不足够。 +- 模型请求可以重发；可能已经派发的工具、Hook、MCP 调用或子任务不得为了恢复 + 进度而重新创建。 +- journal 对准入、Action 状态、执行 receipt 与结算具有权威性。Java 拥有公开 + 命令投递和已消费事件游标；当前被 fencing 的 Harness 拥有模型与工具编排。 +- 保留原 `promptId`、用户记录、Action ID、已提交输出的模型消息 ID，以及已 + prepare 工作的 execution ID。新推理 attempt 使用新 attempt/message 身份， + 不创建新 Prompt。 +- 原 Turn 或清理义务仍占有相应资源时，拒绝新 Prompt、undo 和 cwd 变更。状态 + 与取消必须如实报告停靠 Turn。 +- Store 故障是可重试故障，不是资源或执行不存在的证据。持久证据缺失、损坏或 + 冲突时必须 fail closed。 +- Managed 失败不回退到 Legacy。Workspace 授权过期、capability/configuration + pin 改变或 writer 丢失都不能授予执行权限。 + +## 3. 按证据选择恢复路径，不按 phase 名称猜测 + +`HARNESS_MODEL_START_PHASES` 是运行准入词汇，不是恢复白名单。 +`model_output_committed` 可能表示批准后的工具批次仍待执行；`results_ready` +拥有原 receipt；`turn_settled` 只欠投影。Turn 为 null 的 bootstrap +`before_model` checkpoint 不是当前已接纳模型请求的快照。 + +| 原 Turn 的持久证据 | 提议行为 | +| ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | +| 已标记提交，但 Java 没有 admission epoch/回答。 | 保留 G3 D4 的原命令对账。null epoch 仅证明回答缺失；安全性来自 journal 命令幂等和停靠输入拒绝，不增加第三步重提快捷方式。 | +| 完整主模型请求快照，attempt 尚未完成，没有该请求的已提交输出或未结算副作用。 | fencing 和撤回后，在新 attempt 下重发该模型请求。 | +| assistant 工具调用消息完整，但批次缺少完整的审批续跑证据。 | 不重发模型、不进入普通工具入口，保留现有恢复拒绝。 | +| `await_action`，Action 仍为 `requested`，续跑证据完整。 | 保留原 Action，恢复 waiter、过期处理和取消 driver。 | +| Action 为 `decided` 且选择 `allow` 或 `deny`，续跑证据完整。 | 驱动保存的原批次，应用原决定一次；Deny 写配对的工具拒绝，再按正常路径继续模型。 | +| Action 为 `cancelled` / `expired`，或新派发前已有 Turn 取消获准。 | 证明没有未付副作用后，关闭持久等待并把原 Turn 按取消结算。 | +| Action 决定后有部分 prepare 或 intent 证据。 | 先对账原预留并补齐原批次 checkpoint，再进入 G1/G2，不用替代 ID prepare。 | +| 存在完整的 `await_runtime` / `results_ready` 执行证据。 | 通过 G1/G2 恢复原执行，不用替代 ID 重新 prepare。 | +| `results_ready`，原工具结果完整。 | 沿用已有结果续跑，不重跑产生工具的模型轮。 | +| `turn.settled` 已提交但尚未被 Java 消费。 | 重绑事件 epoch，从 Java 已消费游标重放，不推理、不执行工具。 | +| `turn_settled` checkpoint 可偿付，但缺少终态记录。 | 沿用终态投影补偿，保证幂等写入与重放。 | +| Shell 在飞、外部工作结果未知、Hook/MCP/child 证据不完整，或存在多个歧义停靠 Turn。 | 保留已有类型化拒绝/阻塞及相应所有权 fence。 | + +模型与审批需要少量按 producer 定义的持久资源，不新增通用恢复 domain、公开 +resume API 或配置矩阵。 + +## 4. 切片 A：完成审批等待（B2） + +### 4.1 发布等待前保存续跑信息 + +现有 `approval.invocationRef` 保存单个调用的输入，`optionsRef` 标识 Action。 +两者都不能单独保留混合批次、先前决定、改写后的输入，以及 Hook 再次询问的阶段。 +在 `commitDurableWait()` 之前保存不可变的 +`hosted-approval-continuation/1` 资源。提议 Action options 第 3 版通过 +`continuationRef` 引用它，`invocationRef` 保持当前含义。 + +| 续跑内容 | 必要用途 | +| ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------- | +| Session key、原 Prompt、assistant 消息、definition/config/profile/policy pin 与来源 activation。 | 拒绝外部、过期或配置不同的续跑。 | +| 有序 call ID、名称、有效输入 ref/digest、tool definition、稳定 `runtimeCallId` 及 prepare 幂等键/reference。 | 重建原批次，不发明参数或 ID。 | +| 当前 Action ID、审批 ordinal、位于 PreToolUse 前还是输入改写后的再次审批。 | 从正确阶段继续，不把早期审批授予后来的输入。 | +| 先前决定/拒绝与已完成 permission/Hook 证据引用。 | 保留已经偿付的决定和副作用，不重复 Hook。 | +| 原 Runtime owner 身份与已有 prepare/receipt 证据。 | 区分派发前计划与必须转入 G1/G2 的工作。 | +| 历史边界和 model/Hook 续跑输入。 | 从已提交 assistant 调用消息继续，不再写用户记录、不重新推理同一调用批次。 | + +Action options、descriptor 及嵌套输入引用必须处在同一持久等待事务中。失败既 +不能发布可恢复等待，也不能声称其输入完整。全部 ref 位于同一 Session；沿用已有 +资源与事务大小限制。 + +首份审批快照不一定是最终执行计划：PreToolUse 可能改写输入并分配新 +`runtimeCallId`，即使不再次询问审批。在 Hook 和决定全部完成后、file-history +或 prepare 副作用前，发布并通过 journal 引用最新的不可变有效批次 revision。 +提议使用窄的 `hosted.batch.planned` 事件，含 `batchId`、`planRevision`、 +`planRef`，引用同一 continuation 格式。保存最终 model-call/runtime-call ID、 +规范化 payload ref/digest、definition、拒绝/决定、原 owner 和完整 Broker +prepare reference/key。恢复选择该 revision，而非最初审批 descriptor。该持久 +链接提交前不允许 file-history 或 prepare 副作用。如果最终计划生成前崩溃，仅从 +已证明完成的 Hook/decision 阶段恢复；producer 证据不完整时仍拒绝。 + +第 3 版保留现有预览策略：只有可提供原生有界预览的工具才含 `inputRef`； +`continuationRef` 是私有恢复数据，不作为公开 Action 字段返回。Java Action +projector 和 TS options reader 都必须先支持 v1/v2/v3，writer 才能输出 v3。 + +发布前将序列化后的审批计划和最终批次计划与 Store 的 64 KiB 内联资源上限比较。计划超限时省略该恢复资源，沿用 v1/v2 options，维持普通存活进程的审批和执行。这种审批不具备新增的 continuation 恢复保证。能容纳的计划如果发布到 Store 失败，仍须阻塞，不能按大小超限降级。 + +某个审批没有捕获计划后，同一个 assistant 批次的所有后续计划都不再发布，即使最终计划可以容纳。恢复时，在选择最终计划或更早的 v3 计划前，也须拒绝包含持久化 v1/v2 审批的批次；否则混合批次可能重复已有审批或选中无法读取的计划。检查从当前 assistant 消息的提交 sequence 开始，避免前一个批次的回退影响后续批次。 + +### 4.2 恢复 driver，再服从持久化竞争结果 + +1. 获取新 writer activation，恢复准确的 Session/profile、原 Runtime owner 与 + 续跑证据。在允许新副作用之前检查取消与当前授权。 +2. 在 load 回答前注册一个停靠 Turn owner。`/status` 仍报告进行中，`/prompt` + 不能覆盖停靠状态。resolve、cancel、expiry 与重复 load 汇入同一个 + per-Session driver。内存 latch 仅负责单进程调度；journal CAS 与原 dispatch + ID 才提供跨崩溃幂等性。 +3. requested Action 保留原 ID 和 deadline。注册 waiter 和一个过期定时器后再读 + Action，覆盖在注册前已胜出的决定。不为同一输入创建新审批。 +4. Allow/Deny 验证原 decision digest、input revision 和 policy revision。 + 推进等待一次，恢复保存的执行阶段并继续原批次。有效输入改变时，按适用 policy + 重新判断，必要时取得新审批；保存的 Allow 本身不能授权改写参数。 +5. 从原记录恢复已完成 Hook occurrence。只有下一阶段被证明从未开始时才准入新 + occurrence。不确定的 Hook/MCP 结果或已有 tool intent 转入对应恢复路径，不 + 创建新调用。缺少完整证据的 producer adapter 拒绝恢复。 + 对原生文件，在 prepare 调用之间或 prepare 与 `tool.intent` 之间崩溃， + 需要先对账批次再进入 G1：按保存的 `runtimeCallId`、原幂等键/reference 和 + digest 重试/查询 prepare，恢复匹配的 `PREPARED` 预留，补写缺失的原 intent， + 提交完整的原 `await_runtime` 批次，再进入 G1。Broker prepare 使用 + `dispatch = false`，相同 key 的准确重放返回原 execution receipt。不能调用 + `prepareRequests()` 重建计划，它会生成新随机 ID。冲突或未知预留阻塞；完整 + 批次及必要审批/Hook 证据持久化前不能派发。 +6. 过期或取消时，通过 CAS 记录 Action 终态、关闭 `await_action`，如实配对已 + 提交但尚无回复的调用记录，再写一个取消 Turn 终态。过期采用该取消结果，不 + 在重启后再发模型请求。已提交决定赢得 Action 竞争；另外已经获准的 Turn + cancel 仍阻止新派发，并结算/停止它实际欠下的工作。 +7. 只有安全结算后才释放准确的 Runtime lease。释放失败按第 6 节作为可跨再次 + 崩溃恢复的持久清理义务保留，不能因工具列表为空就释放不确定的 Hook 或后代 writer。 +8. 崩溃后重复 resolve/load/cancel，先读已经提交的决定、checkpoint、原 intent + 与终态；既不再次询问，也不以替代 ID 执行。 + +resolve 路由继续在回答之前提交决定。Action response operation 可在决定持久 +投影后完成，这不声称 Turn 或工具已经完成。决定提交后的失败不能把成功的 Action +回答改写为失败决定。Java 保留原 operation/key 并继续消费 Turn 事件；#13609 +这类浏览器终态跟踪是该 operation 的独立消费者。 + +没有浏览器回答时，过期也必须有效：load 安装 deadline timer，对已经到期的等待 +立即处理。每次重新附着都检查同一保存的 deadline，再次进程崩溃不能将其重置。 + +### 4.3 profile 与旧记录的边界 + +原生文件是首个验收目标。派发前的私有 Shell、MCP、Hooks 和 child-agent 批次 +需要各自明确的续跑与所有权证据；已派发 Shell 保持拒绝。B2 不开启 H3、公开 +Shell 或其他 domain。 + +旧 v1/v2 等待保持可读。只有旧 journal 和 producer 记录证明全部未付副作用 +不存在或已结算时，才允许安全取消/过期。缺少完整可恢复批次的旧 Allow/Deny +保留现有拒绝，不通过重建当前内存静默升级。若后续需要兼容性重建，另设验收切片。 +完成声明必须注明支持的 producer/profile 版本。 + +## 5. 切片 B：恢复尚未完成的主模型轮 + +### 5.1 发送前持久化请求 + +给主模型的 `model.attempt` 事件增加可选 `recoveryRef`，指向不可变的 +`hosted-model-request/1` 资源。事件已有的 `routeRef` 与预算信息保留含义。 +Hook-model attempt 不因此变为可重发。不要挪用也描述 tool/approval 输入的 +`checkpoint.attempt.routeRef`，不修改 checkpoint schema。 + +| 请求快照 | 必要用途 | +| -------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | +| Session、原 Prompt、attempt、来源 activation、request digest 与 journal/history 边界。 | 证明替换的是哪个已接纳请求。 | +| 原有效 Prompt/request Parts，以及规范化的模型可见 history/system instruction ref。 | 恢复 Hook 改写输入、压缩上下文和已消费结果，不重跑 startup Hook。 | +| 固定的 model route、sampling、definition/config/capability/profile revision 与 tool declaration digest。 | 按原契约重发；pin 改变或不可用时 fail closed。 | +| Workspace context revision/content ref 与已完成 Hook-context 引用。 | 避免读取不同 Workspace 或 Harness 主机的新指令。 | +| 原 budget 状态、round ordinal 与 request kind。 | 替换后保留 Turn 限制，不重置 16 轮循环或成本预算。 | +| 先前结果与 producer 义务的所有权/续跑证据。 | 证明没有跳过或重新创建未消费副作用。 | + +在 `LlmChat` 的最终逻辑请求边界，用窄的、可等待的 callback 捕获已经准备好的 +无鉴权 provider 请求：自动压缩、tool-history 修复和输出 token 调整之后， +generator 发送之前。在 `sendMessageStream()` 之前仅复制 Hosted 层输入不足以 +满足要求。Hosted 调用者为每个主请求提供该 callback，普通调用者维持已有行为。 +网络请求之前必须完成快照发布与 `model.attempt: started` 提交。把主 attempt +现有的发送前 `started` 写入移到该 callback，以标识实际发送的请求。工具之后， +用持久提交/结算的结果及已完成 Hook 输出生成下一快照。保留现有结果消费时序： +仅在模型成功 `Finished` 后推进，不在该请求之前推进。不保存模型凭据、鉴权 header +或 OAuth token；使用当前凭据解析固定 route。资源引用验证 digest 与长度，并 +沿用 inline/transaction 限制。如果 descriptor 或新请求数据不引入分块/存储 +机制就无法容纳，将该请求标为不可恢复，写入不含 `recoveryRef` 的旧格式兼容 +attempt，维持普通存活进程的推理。崩溃后保留 `model_start` 拒绝，不声称该请求 +可恢复。Store 故障不是大小分类，不能静默降级持久化屏障。不增加 OSS spill 或 +新的分块机制。 + +保存规范化 generator 参数、history 状态和原 stream owner namespace。 +saved-request 入口按这些参数直接调用固定 generator，跳过 +`sendMessageStream()` 的普通 history repair、压缩、context injection 和 +Prompt 准备。保留正常响应处理、usage 记录、取消与工具输出验证。该响应提交后， +下一逻辑轮用恢复的 history/budget 回到正常循环。 +参数字节等价的 transport retry 可复用快照。fallback 切换 route、reactive +compaction 或 token 调整改变有效请求时，必须在发送前建立新的不可变快照和 +attempt 边界，不能覆盖 ref 或把 fallback 请求归到前一个 attempt。这不增加 +原逻辑轮 allowance。中断的压缩或 Hook-model 工作缺少完整证据时保持拒绝。 + +发送前，把每个 attempt 绑定到独立预分配的 stream message ID；将该映射与 +保存的 attempt 身份一起记录，与规范化 request digest 分开。恢复时除快照内 +budget baseline，还要恢复最新已提交 usage/model-metrics ledger，避免流完成 +但 assistant commit 丢失时忘记已记录用量。崩溃 provider 调用的未知用量无法 +重建；重复推理可能增加 provider 成本，恢复不保证外部计费上限精确执行。 + +恢复沿用已经提交的绝对 `input.accepted.deadline`，准备替代请求不能开启新的 +`deadlineMs` 时段。如果 admission 已提交但 prepared-request 快照尚未提交， +首个模型切片保持拒绝。覆盖这一更早窗口需要另一个持久初始 Turn preparation +plan，请求快照本身不提供该能力。 + +### 5.2 只重发保存的未完成请求 + +1. 找到唯一未结算的原 Prompt 及其最近主模型请求。验证请求身份、引用的历史和 + pin、当前 writer，以及该 attempt 没有已提交输出或未付 producer 副作用。 + `initial` 或 bootstrap checkpoint 本身不能作为证明。 +2. assistant 输出/调用消息若已持久提交，则恢复它的待付工作或终态投影。不能 + 重发产生该调用批次的模型请求来丢弃已经接纳的批次。 +3. 将原 `started` attempt 按原身份标为 abandoned；已经记录流完成状态时保留 + 该事实。两种情况下，新请求快照都标识被替换的 attempt。从 journal 恢复它的 + streamed message ID、原 boot/epoch namespace 及有界 delta sequence，结合 + 保存的 owner 证据，在新输出前提交必要的 + `message.retracted`。一个新建的空 `HostedTextDeltaStream` 无法撤回前任 + 内存中记住的 prefix。 +4. 为保存的同一请求开启新 attempt，分配新 message ID，并引用同一 request + digest。撤回响应丢失或撤回后发送前再次崩溃，通过同一 journal command 重放。 + 发送后崩溃可能产生另一轮模型推理，不承诺推理 exactly-once 或再生成答案相同。 +5. 使用恢复的 history、request kind、budget 和 round ordinal 调用保存的规范化请求 + 入口。不得把普通 `executeHostedTurn` 作为新 Prompt 调用：它可能再写用户 + 记录、重跑 Prompt Hook。无工具 Session 使用同一 saved-request 路径，工具 + 声明为空。 +6. 新模型产生的工具经过正常审批与持久工具准入。先前已提交调用/结果保留在历史 + 中并沿用身份。发送前观察到取消则不发送；发送后取消则中断推理，并阻止新工具 + 准入。 + +扩展 journal 撤回：提议在 `message.retracted` 中增加可选 `sourceBootId`、 +`sourceEventEpoch`、`throughSequence`，连同已有 message ID 和 +`fromSequence` 标识原 namespace 与有界消息范围，并用 journal/owner 证据验证。 +撤回已经在原 namespace 投影的 prefix，以及同一 journal message/range 在 +后续 attachment namespace 下重放的副本。journal reader 也对该消息应用撤回， +cold replay 不能让它重新出现。现有 in-band 撤回只选择当前 boot/epoch，无法 +删除已消费的旧代数 prefix。不能调用会清除旧 Turn/epoch 全部 delta 的 +`retractContinuationOutput()`，否则先前已提交轮也被删除。撤回与已消费游标推进 +原子且幂等。覆盖已消费、未投影及跨两个 namespace 的 prefix，保留此前全部已 +提交轮。 + +维持现有同代数重试语义。本切片允许在 fencing 证据下进行替代代数的撤回，不让 +已经发布的同代数 continuation 自动变为可重放。 + +尚未完成且缺少 `recoveryRef` 的旧 attempt 保留 `model_start` 拒绝。已完成主 attempt 后若存在持久 final assistant 答案，即使没有请求快照也可仅补偿终态，且不重发推理。新快照 producer +必须覆盖每个被支持的主模型请求；仅增加 reader 或没人填充的 optional parameter +不代表交付。 + +## 6. 切片 C：闭合终态投影与事件游标缺口 + +复用 `settleProjectablePromptId` / `runSettleProjection`、sink 的终态身份和 +Java 的现有 epoch CAS。首先增加故障覆盖，只对门禁证实的失败修改生产代码。 + +- 已提交的 `turn.settled` 只重放，不重新计算。仅依据完整、可投影的 checkpoint/ + output 及 producer 证据补偿缺失终态。Action 决定不是 Turn 完成证据。 +- 重绑改变 boot ID 和 event epoch，保留 Java 最后**已消费**的 journal sequence。 + attachment tail 只是观测，不是消费 receipt，不能把游标跳到该 tail。 +- 撤回先于新 attempt 的 delta 落 journal。Java/公开重放和浏览器刷新删除前任 + prefix、保留替代答案，且只有一个终态事件。沿用现有 `fromSequence` 语义，不 + 删除以前已提交轮，不在恢复时伪造 message ID。 +- 重放已准入的 prompt/continue/cancel，返回原 admission watermark，不返回 + 当前 tail。重复终态投递幂等推进消费者，不再调用模型或工具。 +- 终态提交与物理清理是独立事实。lease release、file-history marker retirement + 或 producer cleanup 失败时继续欠付。Java 不能把已完成 Turn 改解释为错误， + 新 Turn 也不能绕过受影响资源的 fence。cold restore 必须找到该义务。 + +提议一个窄的 `hosted.cleanup` journal 事件，含 `cleanupId`、`descriptorRef` +及 `owed` / `confirmed` 状态，引用不可变 `hosted-turn-cleanup/1` descriptor。 +它标识原 Turn、Runtime Session/binding owner，以及适用的原生 file-history +marker ID。这是 Turn 清理记账，不是通用任务 domain。仅在原所有权确实欠付清理 +时提交 `owed`；无工具 Turn 不为该协议创建 Runtime Session。在 adopt/acquire 恢复的 +lease 之前、terminal/release 路径可能丢失该身份之前提交 `owed`。只有 journal +与 producer 证据证明安全结算后才清理。release 使用原 `runtimeSessionId`；只有 +Broker 返回 `released = true` 且全部指定 marker 确认后，才按同一 cleanup ID +提交 `confirmed`。已为 `RELEASED` 的 Runtime Session 可幂等 release，无需 +再次 acquire。通过查询/重试原身份对账部分成功和丢失回答。仍有在飞工作、证据 +未知、身份缺失或 binding 漂移的 owner 继续 fenced。保存 `owed` 失败则不进行新的 acquire/release； +已提交终态仍具有权威性。 + +cold load 与 lifecycle-close 对账即使没有 `unsettledPromptId`，也枚举未确认 +的 `hosted.cleanup` 事件、恢复原 owner，只重试有资格的清理。确认前保留受影响 +Workspace/resource fence。requested 审批仍占有 lease，不能仅因存在 marker +就释放。内存 `runtimeLeaseHeld` / `refusedAdoptions` 可调度重试，但不是持久 +证据。旧清理身份缺失或 Hook/后代停止证据不确定时阻塞清理，不重标已提交终态。 + +## 7. 协议归属与兼容性 + +不需要新的公开 REST/WebShell 路由。现有私有 load、resolve、cancel、events +路由仍属于 live Session owner；Workspace 和 Runtime 调用仍位于持久 binding +和原执行 owner 内。scope 未解析时，任何路径都不能回退 primary runtime。 + +最简单的控制面路径是现有 plain attachment 加事件重放：Harness 拥有恢复后的 +model/approval driver，Java 消费它们的 journal。不把这些状态伪装成含虚构 +execution 的 Runtime recovery report，也不增加通用公开 recovery-kind 开关。 +G1 工作继续使用已有 Runtime report/continue/cancel 协议。 + +Harness 宣告 `hosted_approval_resume_v1` 与 +`hosted_model_round_recovery_v1`，Java SDK 暴露协商后的 feature。它们描述支持能力， +不选择新的公开恢复路由。已有 DRIVE load 在 Harness 中检查持久证据；Store 的版本 +支持回执与阅读门槛负责格式兼容性。兼容的旧栈保留已有拒绝。一旦 Session 写入第三步 +的新记录格式，必须在旧 Harness 附着之前执行阅读门槛，不能 +等解析失败再处理。使用已有 +`qwen_managed_session_journal_head.storage_version` 作为单调门槛：reader +支持 1 和 2；首次提交 v3 options、最终批次计划、model recovery data、跨代数 +有界撤回或 cleanup 的事务与记录一起原子地把 head 提升到 2。 +checkpoint schema 仍为 1，改变的是存储 envelope 的阅读契约。 + +通过一个 HTTP header 传递提议的 `maxReadableStorageVersion` 契约： +`X-Qwen-Managed-Max-Readable-Storage-Version`，省略为 1，新 reader 发送 2。 +共享 TS Store client 在全部 head-scoped 请求中携带它：acquire、renew、restore、 +transaction 枚举、resource read、commit、publication、recovery 和 lifecycle， +包括绕过 JSON wrapper 的请求。Store 先检查服务器支持的 head 格式，再在授予 +lease、读取或写入前检查调用者上限。提升格式的 commit 在 head lock 内检查 +ceiling 2，幂等重放前也检查；旧的有效 token 不能绕过门槛。不增加 grant table +或 token capability 状态。重放返回原 commit； +回滚或删除临时模型状态不能降低门槛。在同一提升事务内验证 writer/lease 与 head +CAS。旧 Java 的 `requireHeadScope` 和旧 TS restore 都拒绝非 1 的 storage +version；新协议还必须在旧 Harness 取得 writer lease 前，拒绝它访问新 Java +Store 中的不兼容 Session。即使只宣告一个执行 feature，也先落地全部第三步新格式的 +reader/Store 支持，再开启任一 producer。使用已有 SQL 列，不新建表，不猜测 +Flyway 预留号。实际冻结契约 revision 随落地基线同步。 + +Writer acquire/renew 回执声明 `supportedStorageVersion: 2`；TS 客户端把缺失支持视为 1。新格式提交必须取得该回执，并使用 `/transactions:commit-v2`，复用新版 Store 的同一原子提交实现。旧 Store 没有这个入口，即使此前 acquire 请求命中了新服务，也会拒绝此提交。这补齐反向混部署边界：旧 Store 原本会接受任意 model-attempt payload，却不提升 head 的读取版本。续租会刷新该回执。使用 publication token 的 Worker 提交保留其原始 admission 协议。 + +开启 producer 前，TS/Java validation、snapshot/restore、Hosted-layout +verification、W1b bundle 枚举、W1c profile/layout 检查与 retention reader +必须追踪全部新 ref。维持普通 inline resource 的 64 KiB 和 8 MiB transaction 限制； +共享、缺失、损坏、外部嵌套 ref 不能误通过引用清点。不开启物理 GC 或新存储后端。 + +## 8. 交付顺序与依赖 + +| 独立切片 | 交付物与退出条件 | +| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| G3.3-A，B2 | 先 options/continuation reader，再 producer 与停靠审批 driver。Requested → Allow/Deny/Cancel/expiry、丢响应与决定后崩溃能完成原原生文件 Turn，或如实阻塞；安全清理结算后下一 Turn 与 close 可用。 | +| G3.3-B，模型轮 | 先主请求快照和阅读门槛，再工具/无工具会话的模型重发与撤回。原输入准入一次，先前工具执行一次，两次替换仍保留 budget/history。 | +| G3.3-C，终态闭环 | 围绕现有投影增加真实进程/游标与清理故障门禁；只有门禁暴露缺口时才落地有界生产修复。 | + +C 门禁是 A、B 被接受前的条件，不允许先发布带已知 cursor 或 cleanup 错误的 +切片。仅剩独立覆盖或有界已证修复时，才适合单独提交最终 C PR。 + +全部切片在开启新记录 producer 前可分别评审和回滚。新记录出现后，回滚必须遵守 +其阅读门槛；单纯源码 revert 不会让旧 reader 变得兼容。 + +每个实现都基于实际落地 main。G1 合入后状态/取消/清理修复归 [#13188][g1fix]； +依赖其已落地契约，或明确协调必要重叠，不复制其分支。原生文件/无工具第三步不 +等待公开 Shell、H3 开启或普通本地 M6。对应恢复情形才要求 H1/H2/H4 producer +证据。更广的 Q2 binding fencing 是独立门禁，整体 Stage G tracker 继续开放。 + +## 9. 验证与验收 + +实现为 packaged-stack runner 增加确定性的 fixture barrier。使用真实 +Java/MySQL、Harness、Broker 和 Runtime 操作；model fixture 计数请求与工具， +控制准确的崩溃边界。该 fixture 足以证明编排，不等于真实 provider 验收。先仅 +替换 Harness，保留 Spring 与生产 lease/timeout；另行在 Linux 替换两侧 owner。 + +| 门禁 | 必须观测到的结果 | +| -------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | +| A1 requested approval → Allow / Deny | Action/input/decision 身份相同；Allow 派发一次，Deny 对该调用派发零次；原 Turn 结算，后续 Turn 完成。 | +| A2 Cancel / expiry / 无浏览器 | deadline 不重置；等待关闭，一个取消终态及真实配对记录；所有权结算后 `close 202` 与下一 Turn 成功。 | +| A3 混合调用与输入改写 | 先前决定保留；待付 sibling ID/顺序恢复；再次批准的输入不继承早期 Allow；已完成 Hook 不重跑。 | +| A4 决定/load 响应丢失 | 原 response operation/key 结算一次；只有一个 driver，不新建审批；重放不重复副作用。 | +| A5 决定、wait 推进、最终计划、prepare 与结果提交后的崩溃 | 各 barrier 恢复正确阶段；最终改写计划、原 ID 与 Runtime status 证据优先于最初审批 descriptor。 | +| A5a 部分 prepare / 缺失 intent | 按原 ID/digest 对账原生预留；完整批次 checkpoint 先于 dispatch/G1。冲突阻塞，不分配替代 ID、不重复文件副作用。 | +| A6 MCP/Hook/child 不确定性 | 缺失或未知 producer 证据阻塞；不重新创建外部调用，不静默释放 Workspace lease。 | +| B1 首轮，有/无工具 | 只有一个原 admission 和 user record；新推理 attempt 可以完成；无工具不需要 Broker dispatch。 | +| B2 streamed prefix 与两次替换 | 新 attempt/stream 身份；新 delta 前持久撤回；live view、reload、replay 只保留存活答案和一个终态。 | +| B3 先前工具之后 | 使用保存的 results/history；各先前 execution 一次；轮数和成本 budget 不重置。 | +| B3a Finished 与持久输出 | 流 Finished 后、assistant commit 前崩溃，可在证据完整时重发；assistant commit 后崩溃不再推理。不能用 attempt 状态名称证明输出持久性。 | +| B4 已提交 assistant call 输出 | 不跨过未付调用重发模型；驱动保存的 approval/batch 或既有 Runtime recovery。 | +| B5 snapshot 故障、pin 漂移与取消 | ref 缺失/损坏/过大/外部或 config/authorization 漂移不产生新副作用；Store 故障重试；取消阻止新派发。 | +| B6 fallback / reactive compaction | 规范化参数改变时发送前建立新 snapshot/attempt；等价重试复用快照。saved-request 重发不重新准备或注入 context，逻辑轮 budget 保留。 | +| C1 终态已提交但未投递 | boot/epoch 改变，已消费 cursor 不动；终态重放一次；模型/工具调用零次。 | +| C2 checkpoint 可偿付但缺少终态 | 崩溃/丢响应下补偿一次；证据损坏时不发明结果。 | +| C3 清理失败后再次崩溃 | 原清理身份仍可发现且 fenced；重试清偿；已完成 Turn 仍为已完成。 | +| C4 原 admission 响应丢失 | 对账原 command/prompt，不再次准入；重放原 admission watermark。 | +| R1 冻结旧 writer / 不支持的 engine | 旧 writer 不能 append/dispatch；保留类型化 engine 拒绝与不回放 Legacy。这不认证存活旧 Spring 的 binding fencing。 | +| R2 新旧 reader 与 storage bundle | v1/v2 与旧 model attempt 保持可读并如实拒绝；门槛拒绝不兼容 acquire、renew、读取、写入与重放，包括旧 token；bundle/restore 引用闭包通过。 | + +每个物理/工具断言都检查实际文件、SQL execution 行、journal ID 和 model fixture +流量。删除 dispatch-count、cursor 或 next-Turn 断言必须让门禁失败；HTTP 200 +或单测绿色本身不是验收。保留最初失败的 fixture 运行，报告实际证据边界。 +local JSONL/H2 测试是额外覆盖，不是 MySQL/Linux 物理停止证据。 + +### 9.1 已验证的实现证据 + +不可变的 `linux-rebased-gates-3` 打包产物在 rebase 和集成修正后，各模式的首次运行中通过 Allow、Deny、Cancel、expiry 和模型续写。环境为 Linux aarch64，Node 22.22.1、Java 21.0.12.1、MySQL 8.4.11。可移植 Node bundle 和 Java JAR 在 macOS 构建后于 Linux 执行。真实 Java 控制面、Broker 和 Runtime Worker 保持存活，仅替换 Harness；推理使用受控的本地 OpenAI fixture。门禁结束后，两端的 1387 个产物 hash 均保持一致，生产源码未变化。此前的 `linux-delivery-gates-2` 结果保留为前一 main 基线的历史证据。 + +Allow 保留原 Action、prepare key、request digest 和 Runtime binding，产生一个 dispatch generation 为 1 的 settled execution，并写入预期文件内容。Deny、Cancel、expiry 均无 execution、无文件；expiry 沿用原 deadline。模型续写保留原 execution，重发相同的规范化 provider 请求内容，并撤回先前已公开可见的部分输出。每种模式都确认原 owner 清理，完成后续 Turn，观察到两轮合计恰好两个终态，再完成 close 202 准入并达到 Session closed 状态。 + +定向验证通过 841 条 core 测试和 111 条共享契约测试。扩展后的 CLI 运行通过 779 条，并暴露三条不符合契约的旧审批 fixture 记录；修正 policy、选项标签和资源 kind 后，受影响套件的全部 60 条测试通过。重复的 13 文件运行通过 782 条中的 781 条；未修改的 Hosted 模型集成套件有一次 `ENOTEMPTY` 临时目录清理失败。该套件全部 8 条测试在单独复跑中通过，因此 782 条测试均有分次通过证据；不声称扩展批次全绿。新一轮 Java 验证通过 1522 条 Server 测试、跳过一条，另通过 55 条 SDK 和 159 条 Broker 测试。build、typecheck、bundle、源码 ESLint 和三个 Java 模块的 Checkstyle 均通过。源码 lint 排除生成的 Maven `target` 与测试 `coverage` 产物。此前基线和扩展运行中的失败诊断均保留。 + +集成回归证明：模型请求准备失败时，普通 prompt 路由和 wake driver 均保留已准入、未结算的输入；渠道中断结算仅在终态与 file-history 退役之后确认原清理 owner。首次释放失败时保留清理 fence，随后 passive resident attachment 释放相同 binding/generation，只新增一次确认，不重复终态。独立 Spring/H2 probe 通过有界撤回与 replay floor 的组合：snapshot 重建期间 reader 保持可用，覆盖范围追平后旧 cursor 再次过期。这项 probe 不属于 MySQL 或进程接管验收。 + +机器可读报告、manifest 和保留的失败诊断位于 `.qwen/investigations/g3-step3-implementation/`，当前汇总报告为 `packaged-verification.md`。这五种打包门禁不证明上述整个矩阵。首个无工具请求崩溃、两次替换、部分 prepare 崩溃、响应丢失、损坏/外部 ref、快照超限与清理崩溃，仅有定向测试或故障探针覆盖，或仍属于更广的打包验收工作。两侧 owner 重启、真实 provider、真实混版本集群和 Q2 旧控制面 fencing 仍独立处理。 + +## 10. 实现范围 + +| 区域 | 预期路径与职责 | +| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| CLI 编排 | `hosted-harness-session.ts`、`hosted-runtime-recovery.ts`：注册停靠 driver、模型恢复入口、cancel/expiry 调度、复用投影与原所有权。 | +| 审批/原生工具 | `hosted-tool-approval.ts`、`hosted-workspace-tool-turn.ts`：options v3、continuation producer/reader、准确批次续跑，不重放 Hook/参数。 | +| 模型与流式输出 | `hosted-harness-model.ts`、`hosted-text-deltas.ts`、core `core/llm-chat.ts` 及其 call-options 类型：prepared-request callback、saved-request 续跑与从 journal 恢复撤回身份。 | +| Core journal | `managed-hook-activation.ts`、`managed-session-records.ts`、共享 schema/fixture、`http-managed-session-store.ts`：主 attempt 可选 recovery ref、最终批次计划、有界撤回、Turn 清理记录、递归引用闭包及逐请求 version ceiling，checkpoint v1 保持不变。 | +| Java 控制面/Store | `ManagedActionStore`、Session-store validation/reference reader、`QwenHostedHarnessConnector`、`HarnessCoordinator`：v3 投影、阅读门槛与 plain-attach/cursor 契约；`ActionResponseCoordinator` 保留 decision-operation 语义。 | +| 其他 reader/producer | Hosted layout/W1b/W1c reader；仅在支持续跑证据时修改 Hook、MCP、child 与 file-history adapter。审计每个声明字段的 producer 和全部读取处。 | +| 验证/文档 | 邻接测试、packaged E2E runner/CI、双语设计和 developer event schema；未通过各自验收前，不升级公开契约实现状态、不启用部署。 | + +## 11. 已定决策与实现待验证项 + +决策：先完成 B2，再扩大模型恢复；重启后保留 requested Action；Allow/Deny +驱动保存的批次，Cancel/expiry 安全结算；仅重发持久主模型请求；使用 plain +attachment 和现有 journal feed；保留 checkpoint v1 与原 Runtime recovery； +不能只因完成这些功能就关闭 Stage G tracker。 + +每个切片输出新记录前，必须验证持久 storage-version 门槛并补齐 nested-ref +reader。宣称某个 producer 被支持前，证明其保存输入、已完成 occurrence、原 owner +和停止/结算证据。这些是实现门禁，不允许从当前环境补猜缺失证据。真实 provider +验收、多实例 binding fencing、公开 Shell 开启、跨主机停止/所有权证明及生产事件 +pruning 都另行处理。 + +## 12. 引用 + +- [Stage G tracker #12952][g] 与[架构 proposal #12380][proposal]。 +- [已合入的 G3 第 1–2 步 #13174][g3] 与[明确的 B2 后续决定][b2]。 +- [G3 第 1–2 步设计](2026-10-02-hosted-replaceable-harness.zh-CN.md)。 +- [Actions 设计](2026-09-30-managed-agent-actions.zh-CN.md)。 +- [Hosted Hooks 设计](2026-09-30-managed-hooks-runtime.zh-CN.md)。 +- [交付账本](managed-agent-delivery-ledger.zh-CN.md)。 +- [G1 合入后修复 #13188][g1fix]。 + +[g]: https://github.com/QwenLM/qwen-code/issues/12952 +[proposal]: https://github.com/QwenLM/qwen-code/issues/12380 +[g3]: https://github.com/QwenLM/qwen-code/pull/13174 +[b2]: https://github.com/QwenLM/qwen-code/pull/13174#issuecomment-6030365600 +[g1fix]: https://github.com/QwenLM/qwen-code/pull/13188 + +### 审阅修正 + +本机取消立即触发 abort。若持久化追加失败，接口返回 503：当前进程停止，但取消尚未持久化，需要重试后才能依赖替代进程恢复。清理按请求的 Turn 区分其它 Turn 的债务，只有确认原 owner 释放后才清除持有租约标记。wake 输入 ID 使用与普通 acquire 相同的路径安全 Runtime 映射。暂时清理故障继续阻止新工作，直到 attachment 重试成功后解除清理阻塞。 + +准备请求时，防御性克隆限定为 64 KiB；超限的借用视图由 Hosted publisher 同步序列化并拒绝保存，不予保留。恢复的 generation 配置只在其请求期间生效，历史恢复执行常规 registry 对账。Hosted 引用闭包校验在每个事务中共享一次引用 memo 与已验证资源缓存；引用元数据冲突检查和完整事务 census 仍保持生效。 + +## 复审修正：请求意图与清理可用性（2026-10-12） + +prepared request 新增可选 boolean `allowPersistentRetry`，记录主模型或 fallback 的重试资格。重发还需当前 unattended-retry 显式开启；不持久化凭据，也不为 fallback 开启无限重试。TS 和 Java reader 接受旧六字段 prepared 及新七字段形状，校验 boolean 类型并拒绝未知字段。旧快照没有主模型资格证据，继续使用有限重试。这修改的是尚未合入的 G3 嵌套 codec，不证明与本 PR 早期修订兼容，也不放宽已发布 G2 的 reader-floor 门禁。 + +包含外部化图片引用的 history 依赖内存 payload 表，该表不在持久快照中。普通在线请求保留原有图片重新附加行为，但 capture callback 将该快照标记为不支持。冷 reader 和直接 prepared 重发也会在推理前拒绝旧的不安全图片引用快照。主模型已完成且最终回答已提交时仍可只补终态；这不代表恢复图片历史。 + +审批 continuation 按 acquire 相同的路径安全映射校验 prepare key 和 Runtime Session 引用，原始 prompt 身份仍保留。短 cleanup ID 不变；较长组合 ID 使用 digest，保证 owed 和 confirmed 事件均满足 stable-ID 字节上限。历史 owed 保留原 payload 身份与 descriptor，仅在确认事件或命令 ID 超限时缩短该 ID。descriptor 继续指向原 prompt、binding 和 generation。 + +清理确认只解除对应清理阻塞；成功解除阻塞的状态转换同时退役该标记，避免以后误清其它恢复阻塞。没有 descriptor 的当前 Runtime 返回精确 `404 runtime_session_not_found` 时，按已不存在完成 handback；有原 owner descriptor 的释放仍要求确认。已接管 Session 的释放遇到传输故障时，close 继续完成本地 teardown，未确认持久债务留待后续对账。停靠状态的 lifecycle 请求在写入 closing claim 之前拒绝，使普通恢复控制仍可操作。resident 模型 redrive 在异步读取期间持有现有 recovery busy fence，并在 attachment 之前校验注册身份。 + +恢复先处理已经持久化的取消，再安装恢复后的 deadline timer。因此较早取消在墙钟超过 deadline 后仍保持取消；没有持久取消的 deadline 仍正常过期。R3 修正与上述历史打包验收快照分开验证，该快照不能认证本次新修订。 diff --git a/integration-tests/cli/hosted-harness-process.test.ts b/integration-tests/cli/hosted-harness-process.test.ts index 5d38a9966b..bae9d1730d 100644 --- a/integration-tests/cli/hosted-harness-process.test.ts +++ b/integration-tests/cli/hosted-harness-process.test.ts @@ -178,12 +178,15 @@ function expectNoTools(request: Record) { function assertCompleted(events: Event[], promptId: string, text: string) { const visible = events.filter((event) => event.promptId === promptId); + // Assistant text streams as durable message.delta chunks; the committed + // message then projects as managed_journal_event, not a second chunk. expect(visible.map((event) => event.type)).toEqual([ 'session_update', + 'managed_journal_event', 'turn_complete', ]); expect(visible[0].data.update?.content.text).toBe(text); - expect(visible[1].data.stopReason).toBe('end_turn'); + expect(visible[2].data.stopReason).toBe('end_turn'); } async function replay(after: number, epoch: string, last: number) { @@ -262,18 +265,28 @@ describe( const visible = events.filter( (event) => event.promptId === body.promptId, ); + // The answer streams as a durable message.delta chunk; the committed + // message carries the same messageId and projects as the journal + // record instead of a second chunk. expect(visible.map((event) => event.type)).toEqual([ 'session_update', + 'managed_journal_event', 'turn_complete', ]); expect(visible[0].data.update?.content.text).toBe('HOSTED_REPLY'); - expect(visible[1].data.stopReason).toBe('end_turn'); + expect(visible[2].data.stopReason).toBe('end_turn'); const committed = (await store!.scan()).events; expect(events.map((event) => event.id)).toEqual( committed.map((event) => event.sequence), ); + expect( + committed.find((event) => event.sequence === visible[0].id), + ).toMatchObject({ + kind: 'message.delta', + payload: { role: 'assistant', text: 'HOSTED_REPLY' }, + }); const message = committed.find( - (event) => event.sequence === visible[0].id, + (event) => event.sequence === visible[1].id, )!; expect(message).toMatchObject({ kind: 'message.committed', @@ -293,7 +306,7 @@ describe( message: { role: 'model', parts: [{ text: 'HOSTED_REPLY' }] }, }); expect( - committed.find((event) => event.sequence === visible[1].id), + committed.find((event) => event.sequence === visible[2].id), ).toMatchObject({ kind: 'turn.settled', payload: { diff --git a/integration-tests/helpers/hosted-broker-reply-loss-driver.ts b/integration-tests/helpers/hosted-broker-reply-loss-driver.ts index a087dc41a0..dbfbd85c16 100644 --- a/integration-tests/helpers/hosted-broker-reply-loss-driver.ts +++ b/integration-tests/helpers/hosted-broker-reply-loss-driver.ts @@ -210,6 +210,7 @@ try { `${fault}: fault did not fire`, ); const blocked = !['prepare', 'start'].includes(fault); + const retryRelease = ['release', 'release-before-forward'].includes(fault); assert.equal( (await json(`/session/${sessionId}/status`)).recoveryBlocked, blocked, @@ -238,8 +239,13 @@ try { const terminals = events.filter( (event) => event.promptId === promptId && event.type.startsWith('turn_'), ); - assert.equal(terminals.length, blocked ? 0 : 1, JSON.stringify(events)); - if (!blocked) assert.equal(terminals[0].type, 'turn_complete'); + assert.equal( + terminals.length, + blocked && !retryRelease ? 0 : 1, + JSON.stringify(events), + ); + if (!blocked || retryRelease) + assert.equal(terminals[0].type, 'turn_complete'); const results = events.filter( (event) => event.data.record?.type === 'tool_result', ); @@ -306,7 +312,6 @@ try { } const traffic = exchanges.length; const callsBeforeReload = modelCalls; - const retryRelease = ['release', 'release-before-forward'].includes(fault); const rejectLoad = blocked && !retryRelease; await json(`/session/${sessionId}/detach`, {}, 204); const loaded = await json( @@ -342,7 +347,7 @@ try { event.promptId === promptId && event.type.startsWith('turn_'), ) .map((event) => event.type), - fault === 'release' ? ['turn_complete'] : [], + ['turn_complete'], ); assert.equal( recoveredEvents.filter( @@ -353,11 +358,23 @@ try { } await json(`/session/${sessionId}/detach`, {}, 204); } - assert.deepEqual( - exchanges.slice(traffic).map((entry) => entry.operation), - retryRelease ? ['release'] : [], - 'Reload may only retry the original runtime release', - ); + const cleanupRetries = exchanges.slice(traffic); + if (retryRelease) { + assert(cleanupRetries.length > 0, 'Original cleanup must be retried'); + const original = exchanges.find((entry) => entry.operation === 'release'); + assert(original); + for (const entry of cleanupRetries) { + assert.equal( + entry.operation, + 'release', + 'Reload may only retry the original runtime release', + ); + assert.deepEqual(entry.fields, { + ...original.fields, + requestId: entry.fields['requestId'], + }); + } + } else assert.deepEqual(cleanupRetries, []); assert.equal( modelCalls, callsBeforeReload, diff --git a/integration-tests/helpers/hosted-latency-driver.ts b/integration-tests/helpers/hosted-latency-driver.ts index 5ac181d4b0..473f234c66 100644 --- a/integration-tests/helpers/hosted-latency-driver.ts +++ b/integration-tests/helpers/hosted-latency-driver.ts @@ -338,6 +338,17 @@ try { } sample.storeRequests = turnStoreRequests; samples.push(structuredClone(sample)); + await waitUntil(async () => { + const response = await cli.request( + `/session/${session.sessionId}/status`, + { + headers: cli.headers(clientId), + }, + ); + const text = await response.text(); + assert.equal(response.status, 200, text); + return JSON.parse(text).hasActivePrompt === false; + }); await json(`/session/${session.sessionId}/detach`, {}, 204); } assert.deepEqual(proxyFailures, []); diff --git a/integration-tests/helpers/hosted-session-store.ts b/integration-tests/helpers/hosted-session-store.ts index 27496df16b..ce46a404ab 100644 --- a/integration-tests/helpers/hosted-session-store.ts +++ b/integration-tests/helpers/hosted-session-store.ts @@ -99,7 +99,13 @@ export async function startHostedSessionStore(sessionId: string) { activationEpoch: current.activation?.epoch ?? 0, }; if (route === '/writers:acquire' || route === '/writers:renew') { - json({ ...head, leaseUntil: Date.now() + Number(body.leaseMillis) }); + json({ + ...head, + leaseUntil: Date.now() + Number(body.leaseMillis), + // Matches ManagedSessionReaderVersion.SUPPORTED on the Java store: + // the Harness writes Hosted recovery records at storage version 2. + supportedStorageVersion: 2, + }); } else if (route === '/execution:authorize') { assert.equal(req.method, 'POST'); assert(handle); @@ -116,7 +122,10 @@ export async function startHostedSessionStore(sessionId: string) { recoveryStatus: 'READY', recoveryDetailCode: null, }); - } else if (route === '/transactions:commit') { + } else if ( + route === '/transactions:commit' || + route === '/transactions:commit-v2' + ) { assert(handle); assert.equal(body.writerGeneration, writerGeneration); assert.equal(body.expectedJournalRevision, transactions.length); diff --git a/integration-tests/helpers/hosted-store-failure-driver.ts b/integration-tests/helpers/hosted-store-failure-driver.ts index 93f4516f2e..cebcdf930c 100644 --- a/integration-tests/helpers/hosted-store-failure-driver.ts +++ b/integration-tests/helpers/hosted-store-failure-driver.ts @@ -69,16 +69,24 @@ function events(transaction: Transaction): Event[] { function selected(transaction: Transaction, fault: string) { const committed = events(transaction); - if (fault === 'arguments' || fault === 'intent') { + if (fault === 'arguments') { + return transaction.resources.some( + (resource) => + resource.kind === 'managed-tool-input' && resource.bytesBase64, + ); + } + if (fault === 'intent') { const intent = committed.find((event) => event.kind === 'tool.intent'); if (!intent) return false; - const argument = transaction.resources.find( - (resource) => resource.kind === 'managed-tool-input', - ); - assert(argument?.bytesBase64); - assert.equal( - (intent.payload['argsRef'] as { resourceId: string }).resourceId, - argument.resourceId, + const argument = intent.payload['argsRef'] as { + kind: string; + resourceId: string; + }; + assert.equal(argument.kind, 'managed-tool-input'); + assert( + transaction.resources.some( + (resource) => resource.resourceId === argument.resourceId, + ), ); return true; } @@ -137,7 +145,10 @@ const proxy = createServer(async (req, res) => { ...(body.length ? { body } : {}), signal: AbortSignal.timeout(30_000), }; - const commit = store && url.pathname.endsWith('/transactions:commit'); + const commit = + store && + (url.pathname.endsWith('/transactions:commit') || + url.pathname.endsWith('/transactions:commit-v2')); const target = commit && !restoring && selected(fields, report.fault); if (target) { assert(report.faults < 3, 'Harness exceeded the commit retry budget'); @@ -329,9 +340,9 @@ try { current.fault.startsWith('result-') || current.fault === 'turn-reply'; for (const [operation, count] of [ ['acquire', 1], - ['prepare', 1], + ['prepare', current.fault === 'arguments' ? 0 : 1], ['start', started ? 1 : 0], - ['release', current.fault === 'turn-reply' ? 1 : 0], + ['release', 0], ] as const) assert.equal( current.operations.filter((item) => item === operation).length, @@ -392,10 +403,10 @@ try { restored[0].recordBytesBase64, current.target!.recordBytesBase64, ); - assert.equal( - current.operations.length, - operations, - 'Cold load must not contact Broker', + assert.deepEqual( + current.operations.slice(operations), + completed ? ['release'] : [], + 'Cold load may only release a durably settled Turn', ); assert.equal( current.modelCalls, diff --git a/integration-tests/helpers/hosted-workspace-tool-turn-driver.ts b/integration-tests/helpers/hosted-workspace-tool-turn-driver.ts index 30d3e4bef3..ece9ae8c4e 100644 --- a/integration-tests/helpers/hosted-workspace-tool-turn-driver.ts +++ b/integration-tests/helpers/hosted-workspace-tool-turn-driver.ts @@ -124,6 +124,13 @@ const storeProxy = createServer(async (req, res) => { 'X-Qwen-Managed-Writer-Token': String( req.headers['x-qwen-managed-writer-token'], ), + ...(req.headers['x-qwen-managed-max-readable-storage-version'] + ? { + 'X-Qwen-Managed-Max-Readable-Storage-Version': String( + req.headers['x-qwen-managed-max-readable-storage-version'], + ), + } + : {}), }, ...(payload.length ? { body: payload } : {}), signal: AbortSignal.timeout(30000), @@ -155,7 +162,11 @@ const storeProxy = createServer(async (req, res) => { ), }); } - if (response.ok && req.url?.endsWith('/transactions:commit')) { + if ( + response.ok && + (req.url?.endsWith('/transactions:commit') || + req.url?.endsWith('/transactions:commit-v2')) + ) { for (const line of Buffer.from(body.recordBytesBase64, 'base64') .toString() .trim() diff --git a/packages/cli/src/serve/hosted-approval-continuation.test.ts b/packages/cli/src/serve/hosted-approval-continuation.test.ts new file mode 100644 index 0000000000..97bac4efb8 --- /dev/null +++ b/packages/cli/src/serve/hosted-approval-continuation.test.ts @@ -0,0 +1,284 @@ +/** + * @license + * Copyright 2026 Qwen Team + * SPDX-License-Identifier: Apache-2.0 + */ + +import { createHash, randomUUID } from 'node:crypto'; +import { mkdtemp, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +import { afterEach, expect, it, vi } from 'vitest'; +import { openManagedSession } from '@qwen-code/qwen-code-core/managed-runtime/managed-session-assembly.js'; +import { LocalManagedSessionResourceStore } from '@qwen-code/qwen-code-core/managed-runtime/managed-session-resources.js'; +import { createManagedHarnessHandle } from '@qwen-code/qwen-code-core/managed-runtime/managed-harness-factory.js'; +import { HostedWorkspaceToolTurn } from './hosted-workspace-tool-turn.js'; +import { + HostedWorkspaceBroker, + hostedRuntimeSessionId, +} from './hosted-workspace-broker.js'; +import { + endHostedAction, + HOSTED_TOOL_APPROVAL_POLICY, + HostedApprovalWaiters, + resolveHostedAction, +} from './hosted-tool-approval.js'; +import { findHostedApprovalContinuation } from './hosted-approval-continuation.js'; + +afterEach(() => vi.restoreAllMocks()); + +it.each( + ['allow', 'deny', 'cancelled', 'expired'].flatMap((decision) => + [randomUUID(), 'mon_1:notify:3', `arun_${'a'.repeat(32)}:input`].map( + (promptId) => ({ decision, promptId }), + ), + ), +)( + 'restores the original native batch and final Action under a replacement owner ($decision, $promptId)', + async ({ decision, promptId }) => { + const root = await mkdtemp(path.join(tmpdir(), 'g3-step3-baseline-')); + const sessionKey = { + tenantId: 'tenant', + workspaceId: 'workspace', + sessionId: randomUUID(), + }; + const options = { + runtimeBaseDir: root, + cwd: root, + transcriptPath: path.join(root, 'transcript.jsonl'), + sessionId: sessionKey.sessionId, + sessionKey, + version: 'g3-step3-baseline', + activationLeaseDurationMs: 60_000, + }; + const resources = LocalManagedSessionResourceStore.create({ + runtimeBaseDir: root, + sessionKey, + }); + const first = await openManagedSession({ + ...options, + workerId: 'baseline-original', + create: { + definitionRef: await resources.publish( + 'managed-definition', + Buffer.from( + JSON.stringify({ + engine: 'managed', + toolProfile: 'hosted-workspace-files/1', + }), + ), + ), + rootSnapshotRef: await resources.publish( + 'managed-root', + Buffer.from('{}'), + ), + createdBy: 'baseline-probe', + }, + }); + let closed = false; + const controller = new AbortController(); + let running: Promise | undefined; + try { + const harness = createManagedHarnessHandle(first); + const content = Buffer.from( + JSON.stringify([{ type: 'text', text: 'write original.txt' }]), + ); + await first.authority.submitInput( + { + operation: 'submitInput', + commandId: promptId, + sessionKey, + contentDigest: createHash('sha256').update(content).digest('hex'), + }, + { + inputId: promptId, + turnId: promptId, + source: 'baseline-probe', + contentRef: await resources.publish('managed-input', content), + admissionRef: await resources.publish( + 'managed-admission', + Buffer.from('{}'), + ), + deadline: null, + wakeReason: 'input', + }, + ); + await harness.ensureRunnable(); + vi.spyOn(HostedWorkspaceBroker.prototype, 'warm').mockImplementation( + async function (this: HostedWorkspaceBroker) { + this.runtime = { + bindingId: 'original-binding', + generation: '1', + workspaceGeneration: '1', + }; + }, + ); + vi.spyOn(HostedWorkspaceBroker.prototype, 'acquire').mockResolvedValue(); + vi.spyOn( + HostedWorkspaceBroker.prototype, + 'fileHistory', + ).mockResolvedValue({ + ownerSessionId: sessionKey.sessionId, + snapshots: [], + files: {}, + }); + const prepare = vi.spyOn(HostedWorkspaceBroker.prototype, 'prepare'); + const execute = vi.spyOn(HostedWorkspaceBroker.prototype, 'execute'); + const turn = new HostedWorkspaceToolTurn( + { baseUrl: 'http://127.0.0.1:1', token: 'fixture-no-traffic' }, + first, + harness, + promptId, + async (type, parts, model, identity) => { + const uuid = identity?.uuid ?? randomUUID(); + await first.sink.write({ + uuid, + parentUuid: null, + sessionId: sessionKey.sessionId, + timestamp: identity?.timestamp ?? new Date().toISOString(), + type, + model, + cwd: root, + version: 'g3-step3-baseline', + daemonPromptId: promptId, + message: { role: type === 'assistant' ? 'model' : 'user', parts }, + }); + return uuid; + }, + () => true, + undefined, + undefined, + { + settings: { mode: 'default', timeoutMs: 60_000 }, + waiters: new HostedApprovalWaiters(), + }, + ); + const call = { + name: 'write_file', + callId: 'original-call', + args: { file_path: 'original.txt', content: 'one original batch' }, + isClientInitiated: false, + prompt_id: promptId, + }; + running = turn + .execute( + [call], + [ + { + functionCall: { + id: call.callId, + name: call.name, + args: call.args, + }, + }, + ], + 'fixture-model', + controller.signal, + ) + .catch((cause) => cause); + let requestId = ''; + await vi.waitFor( + async () => { + const authorization = await first.authority.harnessRunAuthorization(); + expect(authorization.status).toBe('runnable'); + if (authorization.status !== 'runnable') + throw new Error('not runnable'); + expect(authorization.checkpoint.continuation.phase).toBe( + 'await_action', + ); + requestId = authorization.checkpoint.approval!.requestId; + expect(first.authority.action(requestId)?.state).toBe('requested'); + }, + { timeout: 10_000 }, + ); + const action = first.authority.action(requestId)!; + const descriptor = JSON.parse( + (await first.resources.read(action.optionsRef!)).toString(), + ); + await first.close(); + closed = true; + controller.abort(); + await running; + const replacement = await openManagedSession({ + ...options, + workerId: 'baseline-replacement', + }); + try { + expect(replacement.authority.action(requestId)?.state).toBe( + 'requested', + ); + if (decision === 'allow' || decision === 'deny') { + const answer = await resolveHostedAction( + replacement, + new HostedApprovalWaiters(), + requestId, + { + optionId: decision, + inputRevision: 1, + policyRevision: HOSTED_TOOL_APPROVAL_POLICY, + }, + ); + expect(answer.status).toBe(200); + } else { + await endHostedAction( + replacement, + requestId, + decision as 'cancelled' | 'expired', + ); + } + const saved = await findHostedApprovalContinuation( + replacement, + promptId, + ); + expect(saved?.plan).toMatchObject({ + promptId, + batchId: first.authority + .eventsInSequenceRange(1, first.authority.committedSequence) + .find( + (event) => + event.kind === 'message.committed' && + event.payload['role'] === 'assistant', + )?.payload['messageId'], + stage: 'approval', + actionId: requestId, + runtime: { + bindingId: 'original-binding', + generation: '1', + runtimeSessionId: hostedRuntimeSessionId(promptId), + }, + }); + expect(saved!.plan.calls[0].call).toEqual(call); + expect(saved!.plan.calls[0].prepareKey).toBe( + `${hostedRuntimeSessionId(promptId)}:${saved!.plan.calls[0].runtimeCallId}`, + ); + expect(saved!.plan.calls[0].prepareReference).toMatchObject({ + sessionId: hostedRuntimeSessionId(promptId), + promptId, + }); + expect(saved!.inputs[0].input).toEqual(call.args); + expect(descriptor.v).toBe(3); + expect(prepare).not.toHaveBeenCalled(); + expect(execute).not.toHaveBeenCalled(); + const harness = createManagedHarnessHandle(replacement); + await harness.resolveDurableWait(); + await harness.adoptPreparedContinuation(promptId); + const adopted = await replacement.authority.harnessRunAuthorization(); + expect(adopted).toMatchObject({ + status: 'runnable', + checkpoint: { + identity: { activationId: replacement.activation.activationId }, + continuation: { phase: 'model_output_committed' }, + }, + }); + } finally { + await replacement.close(); + } + } finally { + controller.abort(); + await running; + if (!closed) await first.close(); + await rm(root, { recursive: true, force: true }); + } + }, + 20_000, +); diff --git a/packages/cli/src/serve/hosted-approval-continuation.ts b/packages/cli/src/serve/hosted-approval-continuation.ts new file mode 100644 index 0000000000..f85667be7c --- /dev/null +++ b/packages/cli/src/serve/hosted-approval-continuation.ts @@ -0,0 +1,362 @@ +/** + * @license + * Copyright 2026 Qwen Team + * SPDX-License-Identifier: Apache-2.0 + */ + +import { createHash } from 'node:crypto'; +import { isDeepStrictEqual } from 'node:util'; +import type { Part } from '@google/genai'; +import { managedToolDigest } from '@qwen-code/qwen-code-core/tools/managed-tool-protocol.js'; +import type { ToolCallRequestInfo } from '@qwen-code/qwen-code-core/core/turn.js'; +import type { ManagedSession } from '@qwen-code/qwen-code-core/managed-runtime/managed-session-assembly.js'; +import { parseHostedRecoveryResource } from '@qwen-code/qwen-code-core/managed-runtime/hosted-recovery-records.js'; +import { + assertManagedSessionDurableRef, + assertManagedSessionStableId, + type ManagedSessionDurableRef, + type ManagedSessionKey, +} from '@qwen-code/qwen-code-core/managed-runtime/managed-session-records.js'; +import { + hostedActionAllowed, + hostedActionDenied, + readHostedActionOptions, +} from './hosted-tool-approval.js'; + +import { hostedRuntimeSessionId } from './hosted-workspace-broker.js'; + +const NATIVE_TOOLS = new Set(['read_file', 'write_file', 'edit', 'glob']); + +export interface HostedNativePlannedCall { + readonly call: ToolCallRequestInfo; + readonly runtimeCallId: string; + readonly inputRef: ManagedSessionDurableRef; + readonly definitionRef: ManagedSessionDurableRef; + readonly requestDigest: string; + readonly argsDigest: string; + readonly prepareKey: string; + readonly prepareReference: { + readonly sessionId: string; + readonly promptId: string; + readonly callId: string; + readonly argsDigest: string; + }; + readonly partIndex: number; + readonly refusal: string | null; + readonly actionId: string | null; +} + +export interface HostedApprovalContinuation { + readonly v: 1; + readonly sessionKey: ManagedSessionKey; + readonly promptId: string; + readonly definitionRef: ManagedSessionDurableRef; + readonly rootSnapshotRef: ManagedSessionDurableRef; + readonly sourceActivation: { + readonly activationId: string; + readonly epoch: number; + }; + readonly runtime: { + readonly runtimeSessionId: string; + readonly bindingId: string; + readonly generation: string; + readonly workspaceGeneration: string; + }; + readonly batchId: string; + readonly assistantRef: ManagedSessionDurableRef; + readonly model: string; + readonly round: number; + readonly stage: 'approval' | 'final'; + readonly approvalOrdinal: number; + readonly actionId: string | null; + readonly calls: readonly HostedNativePlannedCall[]; +} + +export interface RestoredHostedApprovalContinuation { + readonly plan: HostedApprovalContinuation; + readonly parts: Part[]; + readonly inputs: ReadonlyArray<{ + readonly bytes: Buffer; + readonly payloadJson: string; + readonly input: Record; + }>; +} + +export async function readHostedApprovalContinuation( + session: ManagedSession, + ref: ManagedSessionDurableRef, + promptId: string, +): Promise { + if (ref.kind !== 'hosted-approval-continuation' || ref.schemaVersion !== 1) + throw new Error('Unsupported Hosted approval continuation.'); + const raw = parseHostedRecoveryResource( + await session.resources.read(ref), + session.authority.sessionHeader.sessionKey, + ); + const plan = raw as unknown as HostedApprovalContinuation; + const header = session.authority.sessionHeader; + if ( + Object.keys(raw).length !== 15 || + plan.promptId !== promptId || + !['approval', 'final'].includes(plan.stage) || + !isDeepStrictEqual(plan.definitionRef, header.definitionRef) || + !isDeepStrictEqual(plan.rootSnapshotRef, header.rootSnapshotRef) || + !Number.isSafeInteger(plan.round) || + plan.round < 0 || + plan.round >= 16 || + !Array.isArray(plan.calls) || + plan.calls.length < 1 || + !Number.isSafeInteger(plan.approvalOrdinal) || + plan.approvalOrdinal < 0 || + plan.approvalOrdinal >= plan.calls.length || + !plan.runtime || + Object.keys(plan.runtime).length !== 4 || + plan.runtime.runtimeSessionId !== hostedRuntimeSessionId(promptId) || + !/^[1-9][0-9]{0,18}$/u.test(plan.runtime.generation) || + !/^[1-9][0-9]{0,18}$/u.test(plan.runtime.workspaceGeneration) || + BigInt(plan.runtime.generation) > 2n ** 63n - 1n || + BigInt(plan.runtime.workspaceGeneration) > 2n ** 63n - 1n || + !plan.sourceActivation || + Object.keys(plan.sourceActivation).length !== 2 || + !Number.isSafeInteger(plan.sourceActivation.epoch) || + plan.sourceActivation.epoch < 1 + ) + throw new Error( + 'Hosted approval continuation conflicts with the original Turn.', + ); + for (const id of [ + plan.batchId, + plan.runtime.bindingId, + plan.sourceActivation.activationId, + plan.model, + ]) + assertManagedSessionStableId(id, 'Hosted continuation identity'); + const assistant = session.authority + .eventsInSequenceRange(1, session.authority.committedSequence) + .find( + (event) => + event.kind === 'message.committed' && + event.payload['messageId'] === plan.batchId && + event.payload['role'] === 'assistant', + ); + if ( + !assistant || + !isDeepStrictEqual(assistant.payload['contentRef'], plan.assistantRef) + ) + throw new Error('Original assistant batch is missing.'); + const record = JSON.parse( + (await session.resources.read(plan.assistantRef)).toString('utf8'), + ) as { daemonPromptId?: string; message?: { parts?: Part[] } }; + const parts = record.message?.parts; + if ( + record.daemonPromptId !== promptId || + !Array.isArray(parts) || + parts.filter((part) => part.functionCall).length !== plan.calls.length + ) + throw new Error( + 'Original assistant batch does not match the continuation.', + ); + const ids = new Set(); + const runtimeIds = new Set(); + const partIndexes = new Set(); + const inputs = []; + for (const [ordinal, item] of plan.calls.entries()) { + if ( + Object.keys(item).length !== 11 || + !item.call || + !NATIVE_TOOLS.has(item.call.name) || + item.call.prompt_id !== promptId || + item.call.wasOutputTruncated || + item.call.hadIncompleteArguments || + ids.has(item.call.callId) || + runtimeIds.has(item.runtimeCallId) || + partIndexes.has(item.partIndex) || + !/^sha256:[a-f0-9]{64}$/u.test(item.argsDigest) || + !/^sha256:[a-f0-9]{64}$/u.test(item.requestDigest) || + !Number.isSafeInteger(item.partIndex) || + item.partIndex < 0 || + parts[item.partIndex]?.functionCall?.id !== item.call.callId || + parts[item.partIndex]?.functionCall?.name !== item.call.name || + item.prepareKey !== + `${hostedRuntimeSessionId(promptId)}:${item.runtimeCallId}` || + item.prepareReference?.sessionId !== hostedRuntimeSessionId(promptId) || + item.prepareReference.promptId !== promptId || + item.prepareReference.callId !== item.runtimeCallId || + item.prepareReference.argsDigest !== item.requestDigest || + (item.refusal !== null && typeof item.refusal !== 'string') || + (item.actionId !== null && typeof item.actionId !== 'string') + ) + throw new Error('Invalid Hosted native batch member.'); + ids.add(item.call.callId); + runtimeIds.add(item.runtimeCallId); + partIndexes.add(item.partIndex); + assertManagedSessionStableId(item.call.callId, 'model call ID'); + assertManagedSessionStableId(item.runtimeCallId, 'Runtime call ID'); + for (const [resource, kind] of [ + [item.inputRef, 'managed-tool-input'], + [item.definitionRef, 'managed-tool-definition'], + ] as const) { + const checked = assertManagedSessionDurableRef( + resource, + 'Hosted batch resource', + ); + if (checked.kind !== kind || checked.schemaVersion !== 1) + throw new Error('Invalid Hosted batch resource kind.'); + } + const bytes = await session.resources.read(item.inputRef); + const saved = JSON.parse(bytes.toString('utf8')) as { + harnessSessionId?: string; + runtimeSessionId?: string; + payloadJson?: string; + }; + if ( + saved.harnessSessionId !== header.sessionKey.sessionId || + saved.runtimeSessionId !== hostedRuntimeSessionId(promptId) || + typeof saved.payloadJson !== 'string' || + item.requestDigest !== + `sha256:${createHash('sha256').update(saved.payloadJson).digest('hex')}` + ) + throw new Error('Saved native request digest conflicts.'); + const payload = JSON.parse(saved.payloadJson) as { + toolName?: string; + input?: Record; + }; + if ( + payload.toolName !== item.call.name || + !payload.input || + typeof payload.input !== 'object' || + Array.isArray(payload.input) || + item.argsDigest !== `sha256:${managedToolDigest(payload.input)}` + ) + throw new Error('Saved native request is invalid.'); + const definition = JSON.parse( + (await session.resources.read(item.definitionRef)).toString('utf8'), + ) as { name?: string }; + if (definition.name !== item.call.name) + throw new Error('Saved tool definition conflicts.'); + if (item.actionId) { + const action = session.authority.action(item.actionId); + if (!action) throw new Error('Original approval is missing.'); + const options = await readHostedActionOptions(session, action); + if ( + options.v !== 3 || + options.functionCallId !== item.call.callId || + options.turnId !== promptId || + options.toolName !== item.call.name || + (options.inputRef && + !isDeepStrictEqual(options.inputRef, item.inputRef)) || + (action.state === 'decided' && + !hostedActionAllowed(action, options.policyRevision) && + !hostedActionDenied(action, options.policyRevision)) || + (plan.stage === 'approval' && + ordinal < plan.approvalOrdinal && + action.state !== 'decided') || + (plan.stage === 'final' && action.state === 'requested') || + (plan.stage === 'final' && + ['expired', 'cancelled'].includes(action.state) && + item.refusal === null) || + (action.state === 'decided' && + hostedActionDenied(action, options.policyRevision) && + item.refusal === null && + (plan.stage === 'final' || ordinal !== plan.approvalOrdinal)) + ) + throw new Error('Original approval verdict conflicts.'); + } + inputs.push({ + bytes, + payloadJson: saved.payloadJson, + input: payload.input, + }); + } + if ( + plan.stage === 'approval' && + (!plan.actionId || + plan.calls[plan.approvalOrdinal].actionId !== plan.actionId) + ) + throw new Error('Original approval ordinal conflicts.'); + return { plan, parts, inputs }; +} + +export async function findHostedApprovalContinuation( + session: ManagedSession, + promptId: string, +): Promise { + const authorization = await session.authority.harnessRunAuthorization(); + if ( + authorization.status !== 'runnable' || + !['before_model', 'await_action', 'model_output_committed'].includes( + authorization.checkpoint.continuation.phase, + ) + ) + return undefined; + const events = session.authority.eventsInSequenceRange( + 1, + session.authority.committedSequence, + ); + const projected = await session.sink.project(); + const latestAssistant = projected.findLast( + (item) => item.type === 'assistant' && item.daemonPromptId === promptId, + ); + if (!latestAssistant?.message?.parts?.some((part) => part.functionCall)) + return undefined; + const callIds = latestAssistant.message.parts.flatMap((part) => + part.functionCall?.id ? [part.functionCall.id] : [], + ); + const answered = new Set( + projected + .filter( + (item) => + item.type === 'tool_result' && item.daemonPromptId === promptId, + ) + .flatMap( + (item) => + item.message?.parts?.flatMap((part) => + part.functionResponse?.id ? [part.functionResponse.id] : [], + ) ?? [], + ), + ); + if (callIds.every((id) => answered.has(id))) return undefined; + const assistantSequence = events.findLast( + (event) => + event.kind === 'message.committed' && + event.payload['messageId'] === latestAssistant.uuid && + event.payload['role'] === 'assistant', + )?.sequence; + if (assistantSequence === undefined) return undefined; + const batch = [...events] + .reverse() + .find( + (event) => + event.sequence > assistantSequence && + event.kind === 'hosted.batch.planned' && + event.payload['batchId'] === latestAssistant.uuid, + ); + let approvalRef: ManagedSessionDurableRef | undefined; + for (const event of [...events].reverse()) { + if (event.sequence <= assistantSequence) break; + if ( + event.kind !== 'action.changed' || + event.payload['source'] !== 'tool_call' || + event.payload['state'] !== 'requested' + ) + continue; + const action = session.authority.action( + event.payload['requestId'] as string, + ); + if (!action) continue; + const options = await readHostedActionOptions(session, action); + if ( + options.turnId !== promptId || + !callIds.includes(options.functionCallId) + ) + continue; + if (options.v !== 3) return undefined; + approvalRef ??= options.continuationRef; + } + const planRef = batch + ? (batch.payload['planRef'] as unknown as ManagedSessionDurableRef) + : approvalRef; + return planRef + ? readHostedApprovalContinuation(session, planRef, promptId) + : undefined; +} diff --git a/packages/cli/src/serve/hosted-approval-fallback.test.ts b/packages/cli/src/serve/hosted-approval-fallback.test.ts new file mode 100644 index 0000000000..fcc8a2fce4 --- /dev/null +++ b/packages/cli/src/serve/hosted-approval-fallback.test.ts @@ -0,0 +1,438 @@ +/** + * @license + * Copyright 2026 Qwen Team + * SPDX-License-Identifier: Apache-2.0 + */ + +import { createHash, randomUUID } from 'node:crypto'; +import { mkdtemp, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +import { expect, it, vi } from 'vitest'; +import { openManagedSession } from '@qwen-code/qwen-code-core/managed-runtime/managed-session-assembly.js'; +import type { ManagedSession } from '@qwen-code/qwen-code-core/managed-runtime/managed-session-assembly.js'; +import type { ManagedSessionDurableRef } from '@qwen-code/qwen-code-core/managed-runtime/managed-session-records.js'; +import { LocalManagedSessionResourceStore } from '@qwen-code/qwen-code-core/managed-runtime/managed-session-resources.js'; +import { createManagedHarnessHandle } from '@qwen-code/qwen-code-core/managed-runtime/managed-harness-factory.js'; +import { + createHttpManagedSessionStores, + HTTP_MANAGED_SESSION_STORE_CONTRACT, +} from '@qwen-code/qwen-code-core/managed-runtime/http-managed-session-store.js'; +import { HostedWorkspaceBroker } from './hosted-workspace-broker.js'; +import { HostedWorkspaceToolTurn } from './hosted-workspace-tool-turn.js'; +import { findHostedApprovalContinuation } from './hosted-approval-continuation.js'; +import { + HOSTED_TOOL_APPROVAL_POLICY, + HOSTED_APPROVAL_OPTIONS, + HostedApprovalWaiters, + resolveHostedAction, +} from './hosted-tool-approval.js'; + +it.each([ + { padding: 62_000, decision: 'allow' }, + { padding: 63_080, decision: 'allow' }, +])( + 'keeps final-plan recovery consistent with approval at $padding content bytes', + async ({ padding, decision }) => { + const limit = HTTP_MANAGED_SESSION_STORE_CONTRACT.maxInlineResourceBytes; + const root = await mkdtemp(path.join(tmpdir(), 'g3-approval-inline-')); + const sessionKey = { + tenantId: 'tenant', + workspaceId: 'workspace', + sessionId: randomUUID(), + }; + const promptId = randomUUID(); + const resources = LocalManagedSessionResourceStore.create({ + runtimeBaseDir: root, + sessionKey, + }); + const session = await openManagedSession({ + runtimeBaseDir: root, + cwd: root, + transcriptPath: path.join(root, 'transcript.jsonl'), + sessionId: sessionKey.sessionId, + sessionKey, + version: 'approval-inline-test', + workerId: 'approval-inline-test', + activationLeaseDurationMs: 60_000, + create: { + definitionRef: await resources.publish( + 'managed-definition', + Buffer.from( + JSON.stringify({ + engine: 'managed', + toolProfile: 'hosted-workspace-files/1', + }), + ), + ), + rootSnapshotRef: await resources.publish( + 'managed-root', + Buffer.from('{}'), + ), + createdBy: 'approval-inline-test', + }, + }); + const controller = new AbortController(); + let running: Promise | undefined; + let releasePrepare: (() => void) | undefined; + try { + const content = Buffer.from( + JSON.stringify([{ type: 'text', text: 'write original.txt' }]), + ); + await session.authority.submitInput( + { + operation: 'submitInput', + commandId: promptId, + sessionKey, + contentDigest: createHash('sha256').update(content).digest('hex'), + }, + { + inputId: promptId, + turnId: promptId, + source: 'approval-inline-test', + contentRef: await resources.publish('managed-input', content), + admissionRef: await resources.publish( + 'managed-admission', + Buffer.from('{}'), + ), + deadline: null, + wakeReason: 'input', + }, + ); + const harness = createManagedHarnessHandle(session); + await harness.ensureRunnable(); + const remote = createHttpManagedSessionStores({ + baseUrl: 'http://127.0.0.1:1', + sessionKey, + writerId: 'approval-inline-test', + }); + const published: Array<{ + kind: string; + bytes: number; + plan?: Record; + }> = []; + const localPublish = session.resources.publish.bind(session.resources); + vi.spyOn(session.resources, 'publish').mockImplementation( + async (kind, bytes) => { + published.push({ + kind, + bytes: bytes.byteLength, + ...(kind === 'hosted-approval-continuation' + ? { plan: JSON.parse(bytes.toString()) } + : {}), + }); + await remote.resourceStore.publish(kind, bytes); + return localPublish(kind, bytes); + }, + ); + vi.spyOn(HostedWorkspaceBroker.prototype, 'warm').mockImplementation( + async function (this: HostedWorkspaceBroker) { + this.runtime = { + bindingId: 'original-binding', + generation: '1', + workspaceGeneration: '1', + }; + }, + ); + vi.spyOn(HostedWorkspaceBroker.prototype, 'acquire').mockResolvedValue(); + vi.spyOn( + HostedWorkspaceBroker.prototype, + 'fileHistory', + ).mockResolvedValue({ + ownerSessionId: sessionKey.sessionId, + snapshots: [], + files: {}, + }); + const prepare = vi + .spyOn(HostedWorkspaceBroker.prototype, 'prepare') + .mockImplementation( + () => + new Promise((resolve) => { + releasePrepare = () => resolve('original-execution'); + }), + ); + const execute = vi + .spyOn(HostedWorkspaceBroker.prototype, 'execute') + .mockResolvedValue({ + executionStatus: 'success', + responseParts: [{ text: 'file written' }], + }); + const waiters = new HostedApprovalWaiters(); + const turn = new HostedWorkspaceToolTurn( + { baseUrl: 'http://127.0.0.1:1', token: 'fixture-no-traffic' }, + session, + harness, + promptId, + async (type, parts, model, identity) => { + const uuid = identity?.uuid ?? randomUUID(); + await session.sink.write({ + uuid, + parentUuid: null, + sessionId: sessionKey.sessionId, + timestamp: identity?.timestamp ?? new Date().toISOString(), + type, + model, + cwd: root, + version: 'approval-inline-test', + daemonPromptId: promptId, + message: { role: type === 'assistant' ? 'model' : 'user', parts }, + }); + return uuid; + }, + (_type, parts) => Buffer.byteLength(JSON.stringify(parts)) <= limit, + undefined, + undefined, + { settings: { mode: 'default', timeoutMs: 60_000 }, waiters }, + ); + const call = { + name: 'write_file', + callId: 'original-call', + args: { file_path: 'original.txt', content: 'x'.repeat(padding) }, + isClientInitiated: false, + prompt_id: promptId, + }; + running = turn + .execute( + [call], + [ + { + functionCall: { + id: call.callId, + name: call.name, + args: call.args, + }, + }, + ], + 'fixture-model', + controller.signal, + ) + .catch((cause) => cause); + let requestId = ''; + await vi.waitFor( + async () => { + const authorization = + await session.authority.harnessRunAuthorization(); + expect(authorization.status).toBe('runnable'); + if (authorization.status !== 'runnable') + throw new Error('not runnable'); + expect(authorization.checkpoint.continuation.phase).toBe( + 'await_action', + ); + requestId = authorization.checkpoint.approval!.requestId; + expect(session.authority.action(requestId)?.state).toBe('requested'); + }, + { timeout: 5_000 }, + ); + const action = session.authority.action(requestId)!; + const options = JSON.parse( + (await session.resources.read(action.optionsRef!)).toString(), + ); + expect(options.v).toBe(padding === 62_000 ? 3 : 2); + if (padding === 63_080) + expect(options).not.toHaveProperty('continuationRef'); + expect( + published.find((item) => item.kind === 'managed-message')!.bytes, + ).toBeLessThanOrEqual(limit); + expect( + published.find((item) => item.kind === 'managed-tool-input')!.bytes, + ).toBeLessThanOrEqual(limit); + expect(prepare).not.toHaveBeenCalled(); + expect(execute).not.toHaveBeenCalled(); + await expect( + resolveHostedAction(session, waiters, requestId, { + optionId: decision, + inputRevision: 1, + policyRevision: HOSTED_TOOL_APPROVAL_POLICY, + }), + ).resolves.toMatchObject({ status: 200 }); + await vi.waitFor(() => expect(prepare).toHaveBeenCalledTimes(1), { + timeout: 5_000, + }); + expect(execute).not.toHaveBeenCalled(); + const plans = published.filter( + (item) => item.kind === 'hosted-approval-continuation', + ); + const finalEvents = session.authority + .eventsInSequenceRange(1, session.authority.committedSequence) + .filter((event) => event.kind === 'hosted.batch.planned'); + expect(plans).toHaveLength(padding === 62_000 ? 2 : 0); + expect(finalEvents).toHaveLength(padding === 62_000 ? 1 : 0); + expect(published.every((item) => item.bytes <= limit)).toBe(true); + const restored = await findHostedApprovalContinuation(session, promptId); + if (padding === 62_000) expect(restored?.plan.stage).toBe('final'); + else expect(restored).toBeUndefined(); + releasePrepare!(); + expect(await running).not.toBeInstanceOf(Error); + expect(execute).toHaveBeenCalledTimes(1); + expect(JSON.parse(execute.mock.calls[0][1]).input).toEqual(call.args); + await turn.consumeResults(); + await turn.finish(); + } finally { + controller.abort(); + releasePrepare?.(); + await running; + vi.restoreAllMocks(); + await session.close(); + await rm(root, { recursive: true, force: true }); + } + }, + 15_000, +); + +function approvalSelectionFixture( + legacyVersion: 1 | 2, + final: boolean, + previousBatch = false, +) { + const bytes = new Map(); + const resource = (kind: string, body: object): ManagedSessionDurableRef => { + const data = Buffer.from(JSON.stringify(body)); + const ref = { + resourceId: `resource-${bytes.size}`, + kind, + schemaVersion: 1, + byteLength: data.byteLength, + digest: createHash('sha256').update(data).digest('hex'), + }; + bytes.set(ref.resourceId, data); + return ref; + }; + const planRef = resource('hosted-approval-continuation', {}); + const inputRef = resource('managed-tool-input', {}); + const optionsRef = ( + requestId: string, + functionCallId: string, + v: 1 | 2 | 3, + ) => + resource('managed-action-options', { + v, + requestId, + turnId: 'prompt', + functionCallId, + toolName: 'write_file', + policyRevision: HOSTED_TOOL_APPROVAL_POLICY, + inputRevision: 1, + createdAt: 1, + expiresAt: 60_001, + options: HOSTED_APPROVAL_OPTIONS, + ...(v >= 2 ? { inputRef } : {}), + ...(v === 3 ? { continuationRef: planRef } : {}), + }); + const actions = new Map([ + [ + 'first', + { + requestId: 'first', + inputRevision: 1, + optionsRef: optionsRef('first', 'call-first', 3), + }, + ], + [ + 'legacy', + { + requestId: 'legacy', + inputRevision: 1, + optionsRef: optionsRef('legacy', 'call-legacy', legacyVersion), + }, + ], + ]); + const request = (requestId: string, sequence: number) => ({ + kind: 'action.changed', + sequence, + payload: { requestId, source: 'tool_call', state: 'requested' }, + }); + const events = [ + ...(previousBatch ? [request('legacy', 9)] : []), + { + kind: 'message.committed', + sequence: 10, + payload: { messageId: 'current-batch', role: 'assistant' }, + }, + request('first', 11), + ...(previousBatch + ? [ + { + kind: 'action.changed', + sequence: 12, + payload: { + requestId: 'legacy', + source: 'tool_call', + state: 'cancelled', + }, + }, + ] + : [request('legacy', 12)]), + ...(final + ? [ + { + kind: 'hosted.batch.planned', + sequence: 13, + payload: { batchId: 'current-batch', planRef }, + }, + ] + : []), + ]; + const read = vi.fn(async (ref: ManagedSessionDurableRef) => { + if (ref.resourceId === planRef.resourceId) + throw new Error('current plan selected'); + return bytes.get(ref.resourceId)!; + }); + const session = { + authority: { + committedSequence: 13, + harnessRunAuthorization: async () => ({ + status: 'runnable', + checkpoint: { + continuation: { + phase: final ? 'model_output_committed' : 'await_action', + }, + }, + }), + eventsInSequenceRange: () => events, + action: (id: string) => actions.get(id), + }, + sink: { + project: async () => [ + { + uuid: 'current-batch', + type: 'assistant', + daemonPromptId: 'prompt', + message: { + parts: ['call-first', 'call-legacy'].map((id) => ({ + functionCall: { id }, + })), + }, + }, + ], + }, + resources: { read }, + } as unknown as ManagedSession; + return { session, read, planRef }; +} + +it.each([ + { legacyVersion: 1 as const, final: false }, + { legacyVersion: 2 as const, final: false }, + { legacyVersion: 1 as const, final: true }, + { legacyVersion: 2 as const, final: true }, +])( + 'declines a current v$legacyVersion approval before selecting a plan (final=$final)', + async ({ legacyVersion, final }) => { + const { session, read, planRef } = approvalSelectionFixture( + legacyVersion, + final, + ); + await expect( + findHostedApprovalContinuation(session, 'prompt'), + ).resolves.toBeUndefined(); + expect(read).not.toHaveBeenCalledWith(planRef); + }, +); + +it('keeps a later batch recoverable when an earlier legacy Action updates', async () => { + const { session, read, planRef } = approvalSelectionFixture(2, true, true); + await expect( + findHostedApprovalContinuation(session, 'prompt'), + ).rejects.toThrow('current plan selected'); + expect(read).toHaveBeenCalledWith(planRef); +}); diff --git a/packages/cli/src/serve/hosted-approval-inline-limit.test.ts b/packages/cli/src/serve/hosted-approval-inline-limit.test.ts new file mode 100644 index 0000000000..97519308a8 --- /dev/null +++ b/packages/cli/src/serve/hosted-approval-inline-limit.test.ts @@ -0,0 +1,272 @@ +/** + * @license + * Copyright 2026 Qwen Team + * SPDX-License-Identifier: Apache-2.0 + */ + +import { createHash, randomUUID } from 'node:crypto'; +import { mkdtemp, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +import { expect, it, vi } from 'vitest'; +import { openManagedSession } from '@qwen-code/qwen-code-core/managed-runtime/managed-session-assembly.js'; +import { LocalManagedSessionResourceStore } from '@qwen-code/qwen-code-core/managed-runtime/managed-session-resources.js'; +import { createManagedHarnessHandle } from '@qwen-code/qwen-code-core/managed-runtime/managed-harness-factory.js'; +import { + createHttpManagedSessionStores, + HTTP_MANAGED_SESSION_STORE_CONTRACT, +} from '@qwen-code/qwen-code-core/managed-runtime/http-managed-session-store.js'; +import { HostedWorkspaceBroker } from './hosted-workspace-broker.js'; +import { HostedWorkspaceToolTurn } from './hosted-workspace-tool-turn.js'; +import { + HOSTED_TOOL_APPROVAL_POLICY, + HostedApprovalWaiters, + resolveHostedAction, +} from './hosted-tool-approval.js'; + +it.each([ + { padding: 62_000, decision: 'allow' }, + { padding: 62_000, decision: 'deny' }, + { padding: 63_500, decision: 'allow' }, + { padding: 63_500, decision: 'deny' }, +])( + 'completes native approval at $padding content bytes ($decision)', + async ({ padding, decision }) => { + const limit = HTTP_MANAGED_SESSION_STORE_CONTRACT.maxInlineResourceBytes; + const root = await mkdtemp(path.join(tmpdir(), 'g3-approval-inline-')); + const sessionKey = { + tenantId: 'tenant', + workspaceId: 'workspace', + sessionId: randomUUID(), + }; + const promptId = randomUUID(); + const resources = LocalManagedSessionResourceStore.create({ + runtimeBaseDir: root, + sessionKey, + }); + const session = await openManagedSession({ + runtimeBaseDir: root, + cwd: root, + transcriptPath: path.join(root, 'transcript.jsonl'), + sessionId: sessionKey.sessionId, + sessionKey, + version: 'approval-inline-test', + workerId: 'approval-inline-test', + activationLeaseDurationMs: 60_000, + create: { + definitionRef: await resources.publish( + 'managed-definition', + Buffer.from( + JSON.stringify({ + engine: 'managed', + toolProfile: 'hosted-workspace-files/1', + }), + ), + ), + rootSnapshotRef: await resources.publish( + 'managed-root', + Buffer.from('{}'), + ), + createdBy: 'approval-inline-test', + }, + }); + const controller = new AbortController(); + let running: Promise | undefined; + try { + const content = Buffer.from( + JSON.stringify([{ type: 'text', text: 'write original.txt' }]), + ); + await session.authority.submitInput( + { + operation: 'submitInput', + commandId: promptId, + sessionKey, + contentDigest: createHash('sha256').update(content).digest('hex'), + }, + { + inputId: promptId, + turnId: promptId, + source: 'approval-inline-test', + contentRef: await resources.publish('managed-input', content), + admissionRef: await resources.publish( + 'managed-admission', + Buffer.from('{}'), + ), + deadline: null, + wakeReason: 'input', + }, + ); + const harness = createManagedHarnessHandle(session); + await harness.ensureRunnable(); + const remote = createHttpManagedSessionStores({ + baseUrl: 'http://127.0.0.1:1', + sessionKey, + writerId: 'approval-inline-test', + }); + const published: Array<{ kind: string; bytes: number }> = []; + const localPublish = session.resources.publish.bind(session.resources); + vi.spyOn(session.resources, 'publish').mockImplementation( + async (kind, bytes) => { + published.push({ kind, bytes: bytes.byteLength }); + await remote.resourceStore.publish(kind, bytes); + return localPublish(kind, bytes); + }, + ); + vi.spyOn(HostedWorkspaceBroker.prototype, 'warm').mockImplementation( + async function (this: HostedWorkspaceBroker) { + this.runtime = { + bindingId: 'original-binding', + generation: '1', + workspaceGeneration: '1', + }; + }, + ); + vi.spyOn(HostedWorkspaceBroker.prototype, 'acquire').mockResolvedValue(); + vi.spyOn( + HostedWorkspaceBroker.prototype, + 'fileHistory', + ).mockResolvedValue({ + ownerSessionId: sessionKey.sessionId, + snapshots: [], + files: {}, + }); + const prepare = vi + .spyOn(HostedWorkspaceBroker.prototype, 'prepare') + .mockResolvedValue('original-execution'); + const execute = vi + .spyOn(HostedWorkspaceBroker.prototype, 'execute') + .mockResolvedValue({ + executionStatus: 'success', + responseParts: [{ text: 'file written' }], + }); + const waiters = new HostedApprovalWaiters(); + const turn = new HostedWorkspaceToolTurn( + { baseUrl: 'http://127.0.0.1:1', token: 'fixture-no-traffic' }, + session, + harness, + promptId, + async (type, parts, model, identity) => { + const uuid = identity?.uuid ?? randomUUID(); + await session.sink.write({ + uuid, + parentUuid: null, + sessionId: sessionKey.sessionId, + timestamp: identity?.timestamp ?? new Date().toISOString(), + type, + model, + cwd: root, + version: 'approval-inline-test', + daemonPromptId: promptId, + message: { role: type === 'assistant' ? 'model' : 'user', parts }, + }); + return uuid; + }, + (_type, parts) => Buffer.byteLength(JSON.stringify(parts)) <= limit, + undefined, + undefined, + { settings: { mode: 'default', timeoutMs: 60_000 }, waiters }, + ); + const call = { + name: 'write_file', + callId: 'original-call', + args: { file_path: 'original.txt', content: 'x'.repeat(padding) }, + isClientInitiated: false, + prompt_id: promptId, + }; + running = turn + .execute( + [call], + [ + { + functionCall: { + id: call.callId, + name: call.name, + args: call.args, + }, + }, + ], + 'fixture-model', + controller.signal, + ) + .catch((cause) => cause); + let requestId = ''; + await vi.waitFor( + async () => { + const authorization = + await session.authority.harnessRunAuthorization(); + expect(authorization.status).toBe('runnable'); + if (authorization.status !== 'runnable') + throw new Error('not runnable'); + expect(authorization.checkpoint.continuation.phase).toBe( + 'await_action', + ); + requestId = authorization.checkpoint.approval!.requestId; + expect(session.authority.action(requestId)?.state).toBe('requested'); + }, + { timeout: 5_000 }, + ); + const action = session.authority.action(requestId)!; + const options = JSON.parse( + (await session.resources.read(action.optionsRef!)).toString(), + ); + expect(options.v).toBe(padding === 62_000 ? 3 : 2); + if (padding === 63_500) + expect(options).not.toHaveProperty('continuationRef'); + expect( + published.find((item) => item.kind === 'managed-message')!.bytes, + ).toBeLessThanOrEqual(limit); + expect( + published.find((item) => item.kind === 'managed-tool-input')!.bytes, + ).toBeLessThanOrEqual(limit); + expect(prepare).not.toHaveBeenCalled(); + expect(execute).not.toHaveBeenCalled(); + await expect( + resolveHostedAction(session, waiters, requestId, { + optionId: decision, + inputRevision: 1, + policyRevision: HOSTED_TOOL_APPROVAL_POLICY, + }), + ).resolves.toMatchObject({ status: 200 }); + const result = await running; + expect(result).not.toBeInstanceOf(Error); + expect(prepare).toHaveBeenCalledTimes(decision === 'allow' ? 1 : 0); + expect(execute).toHaveBeenCalledTimes(decision === 'allow' ? 1 : 0); + if (decision === 'allow') { + expect(JSON.parse(execute.mock.calls[0][1]).input).toEqual(call.args); + expect(result).toMatchObject([ + { + functionResponse: { + id: call.callId, + response: { executionStatus: 'success' }, + }, + }, + ]); + } else { + expect(JSON.stringify(result)).toContain('owner denied'); + } + const plans = published.filter( + (item) => item.kind === 'hosted-approval-continuation', + ); + expect(plans).toHaveLength(padding === 62_000 ? 2 : 0); + expect(published.every((item) => item.bytes <= limit)).toBe(true); + await turn.consumeResults(); + await turn.finish(); + expect(await session.authority.harnessRunAuthorization()).toMatchObject({ + status: 'runnable', + checkpoint: { + continuation: { + phase: + decision === 'allow' ? 'turn_settled' : 'model_output_committed', + }, + }, + }); + } finally { + controller.abort(); + await running; + vi.restoreAllMocks(); + await session.close(); + await rm(root, { recursive: true, force: true }); + } + }, + 15_000, +); diff --git a/packages/cli/src/serve/hosted-harness-contract.test.ts b/packages/cli/src/serve/hosted-harness-contract.test.ts index addea25ac4..a7b79b3679 100644 --- a/packages/cli/src/serve/hosted-harness-contract.test.ts +++ b/packages/cli/src/serve/hosted-harness-contract.test.ts @@ -45,6 +45,7 @@ describe('Hosted Harness private contract', () => { capabilityDigest: DIGEST, lifecycleProtocolVersion: 1, titleProtocolVersion: 1, + features: ['hosted_approval_resume_v1', 'hosted_model_round_recovery_v1'], }); expect(Object.isFrozen(contract)).toBe(true); expect(Object.isFrozen(contract.protocolVersions)).toBe(true); diff --git a/packages/cli/src/serve/hosted-harness-contract.ts b/packages/cli/src/serve/hosted-harness-contract.ts index 1a2b5c0c63..3d96491a58 100644 --- a/packages/cli/src/serve/hosted-harness-contract.ts +++ b/packages/cli/src/serve/hosted-harness-contract.ts @@ -47,6 +47,10 @@ export function createHostedHarnessContract( capabilityDigest, lifecycleProtocolVersion: 1, titleProtocolVersion: 1, + features: Object.freeze([ + 'hosted_approval_resume_v1', + 'hosted_model_round_recovery_v1', + ]), }); } diff --git a/packages/cli/src/serve/hosted-harness-model.ts b/packages/cli/src/serve/hosted-harness-model.ts index ff2a342ffd..b227e2a8f8 100644 --- a/packages/cli/src/serve/hosted-harness-model.ts +++ b/packages/cli/src/serve/hosted-harness-model.ts @@ -4,11 +4,18 @@ * SPDX-License-Identifier: Apache-2.0 */ +import { Turn, LlmEventType } from '@qwen-code/qwen-code-core/core/turn.js'; +import { + LlmRequestPreparationError, + type LlmPreparedRequest, + type LlmPreparedRequestCallback, +} from '@qwen-code/qwen-code-core/core/llm-chat.js'; +import type { HostedModelRequest } from './hosted-model-recovery.js'; + import type { Content, Part } from '@google/genai'; import { createHash } from 'node:crypto'; import { SendMessageType } from '@qwen-code/qwen-code-core/core/client.js'; import type { ToolCallRequestInfo } from '@qwen-code/qwen-code-core/core/turn.js'; -import { LlmEventType } from '@qwen-code/qwen-code-core/core/turn.js'; import type { ChatRecord } from '@qwen-code/qwen-code-core/services/chatRecordingService.js'; import { loadCliConfig, type CliArgs } from '../config/config.js'; import { loadSettings } from '../config/settings.js'; @@ -73,6 +80,18 @@ export async function runHostedHarnessTextTurn(input: { promptId: string; signal: AbortSignal; resumeFromToolResults?: readonly Part[]; + initialRound?: number; + resumeModel?: HostedModelRequest; + prepareModel?: ( + prepared: LlmPreparedRequest, + round: number, + pendingToolResults: boolean, + context: string | undefined, + captureUnsupported?: boolean, + ) => Promise< + | import('@qwen-code/qwen-code-core/managed-runtime/managed-session-records.js').ManagedSessionDurableRef + | undefined + >; hooks?: HostedHookSession; modelScope?: ManagedHookModelScope; toolTurn?: Pick< @@ -80,7 +99,10 @@ export async function runHostedHarnessTextTurn(input: { 'execute' | 'consumeResults' | 'declarations' | 'setPromptHookRunner' > & Partial< - Pick + Pick< + HostedWorkspaceToolTurn, + 'resumeHookResults' | 'hookStopReason' | 'setModelRound' + > >; workspaceContext?: { read(): string | undefined }; agentDefinition?: HostedAgentDefinition; @@ -163,7 +185,7 @@ export async function runHostedHarnessTextTurn(input: { ] : [], ); - if (!input.resumeFromToolResults) + if (!input.resumeFromToolResults && !input.resumeModel) initialMessages.push({ role: 'user', parts: [{ text: input.prompt }] }); const initializationHooks: Array<{ fields: HookInput; @@ -288,7 +310,7 @@ export async function runHostedHarnessTextTurn(input: { if (runPromptHook) input.toolTurn?.setPromptHookRunner?.(runPromptHook); modelReady = true; let effectivePrompt = input.prompt; - if (input.hooks && !input.resumeFromToolResults) { + if (input.hooks && !input.resumeFromToolResults && !input.resumeModel) { const startupId = `session-start:${input.sessionId}`; const start = input.hooks.hasCompletedOccurrence( HookEventName.SessionStart, @@ -403,13 +425,25 @@ export async function runHostedHarnessTextTurn(input: { input.signal, )) ?? [...input.resumeFromToolResults]) : [{ text: effectivePrompt }]; - let pendingToolResults = input.resumeFromToolResults !== undefined; + let pendingToolResults = + input.resumeFromToolResults !== undefined || + (input.resumeModel?.pendingToolResults ?? false); + let preparedRecovery = input.resumeModel?.prepared; let stopHookActive = (await input.hooks?.wasStopBlocked(input.promptId)) ?? false; - for (let round = 0; round < 16; round++) { + for ( + let round = input.resumeModel?.round ?? input.initialRound ?? 0; + round < 16; + round++ + ) { + input.toolTurn?.setModelRound?.(round); input.signal.throwIfAborted(); const contextAvailable = input.workspaceContext?.read(); - if (contextAvailable && contextAvailable !== injectedContext) { + if ( + !preparedRecovery && + contextAvailable && + contextAvailable !== injectedContext + ) { // setUserMemory alone never reaches the wire: the system instruction // was assembled during initialize() and is cached on the chat. config.setUserMemory(contextWithInstructions(contextAvailable)); @@ -434,28 +468,67 @@ export async function runHostedHarnessTextTurn(input: { let text = ''; let finished = false; let modelFailure = true; - const completeAttempt = await input.modelScope?.beginMainAttempt( - config.getModel(), - ); - const modelOccurrence = + let completeAttempt: + | Awaited> + | undefined = input.prepareModel + ? undefined + : await input.modelScope?.beginMainAttempt(config.getModel()); + let modelOccurrence = completeAttempt?.attemptId ?? `${input.promptId}:${round}`; + const prepare: LlmPreparedRequestCallback = async ( + prepared, + captureUnsupported, + ) => { + if (completeAttempt) { + const previous = completeAttempt; + completeAttempt = undefined; + await previous(finished, usage); + } + const recoveryRef = await input.prepareModel?.( + prepared, + round, + pendingToolResults, + input.workspaceContext?.read(), + captureUnsupported, + ); + completeAttempt = await input.modelScope?.beginMainAttempt( + prepared.request.model, + recoveryRef, + ); + modelOccurrence = completeAttempt?.attemptId ?? modelOccurrence; + finished = false; + usage.length = 0; + }; + prepare.maxSnapshotBytes = 64 * 1024; const usage: unknown[] = []; try { - for await (const event of client.sendMessageStream( - request, - input.signal, - input.promptId, - { - type: - round === 0 && !input.resumeFromToolResults - ? SendMessageType.UserQuery - : SendMessageType.ToolResult, - // Published deltas cannot be un-glued after the fact: a cut that - // already delivered content replays the request, and the RETRY - // handling below retracts the orphaned prefix (#13319). - retractDeliveredOutputOnRetry: true, - }, - )) { + const stream = preparedRecovery + ? new Turn( + client.getChat(), + input.promptId, + undefined, + undefined, + true, + prepare, + ).run( + preparedRecovery.request.model, + [], + input.signal, + preparedRecovery, + ) + : client.sendMessageStream(request, input.signal, input.promptId, { + type: + round === 0 && !input.resumeFromToolResults + ? SendMessageType.UserQuery + : SendMessageType.ToolResult, + // Published deltas cannot be un-glued after the fact: a cut that + // already delivered content replays the request, and the RETRY + // handling below retracts the orphaned prefix (#13319). + retractDeliveredOutputOnRetry: true, + ...(input.prepareModel ? { onPreparedRequest: prepare } : {}), + }); + preparedRecovery = undefined; + for await (const event of stream) { if (event.type === LlmEventType.Content) { text += event.value; await textDeltas?.delta(event.value); @@ -509,7 +582,8 @@ export async function runHostedHarnessTextTurn(input: { if ( modelFailure && !input.signal.aborted && - !(cause instanceof HostedHookRecoveryRequiredError) + !(cause instanceof HostedHookRecoveryRequiredError) && + !(cause instanceof LlmRequestPreparationError) ) { await fire( HookEventName.StopFailure, diff --git a/packages/cli/src/serve/hosted-harness-session.test.ts b/packages/cli/src/serve/hosted-harness-session.test.ts index c4feeb2df2..7bf7a39b18 100644 --- a/packages/cli/src/serve/hosted-harness-session.test.ts +++ b/packages/cli/src/serve/hosted-harness-session.test.ts @@ -29,6 +29,7 @@ import { type MockInstance, } from 'vitest'; import * as hostedHistory from './hosted-file-history.js'; +import * as modelRecovery from './hosted-model-recovery.js'; import type { HostedFileHistoryState } from './hosted-file-history-protocol.js'; import { LocalJsonlManagedSessionJournalHandle, @@ -46,6 +47,11 @@ import { SessionTranscriptChangedError } from '@qwen-code/qwen-code-core/service import { channelInputId } from '@qwen-code/qwen-code-core/managed-runtime/managed-channel-operations.js'; import { HostedChannelSession } from './hosted-channel-session.js'; import { commitHostedFileHistory } from './hosted-file-history.js'; +import { oweHostedTurnCleanup } from './hosted-turn-cleanup.js'; +import { + LlmRequestPreparationError, + type LlmPreparedRequest, +} from '@qwen-code/qwen-code-core/core/llm-chat.js'; import { LocalShellResultCapture } from '@qwen-code/qwen-code-core/managed-runtime/local-shell-result-capture.js'; import { parseToolResultManifestBytes } from '@qwen-code/qwen-code-core/managed-runtime/managed-tool-result.js'; import type { @@ -96,6 +102,7 @@ import { HOSTED_APPROVAL_TIMEOUT_MS, HOSTED_TOOL_APPROVAL_POLICY, HostedApprovalWaiters, + endHostedAction, } from './hosted-tool-approval.js'; import { stripAnsiAndControl } from '@qwen-code/qwen-code-core/utils/textUtils.js'; import * as stdio from '../utils/stdioHelpers.js'; @@ -115,7 +122,10 @@ import { HostedMonitorLoop } from './hosted-monitor-loop.js'; import { HostedMonitorWakeScheduler } from './hosted-monitor-wake.js'; import { HostedChildRunSession } from './hosted-child-run-session.js'; import { LocalShellStreamCapture } from '@qwen-code/qwen-code-core/managed-runtime/local-shell-stream-capture.js'; -import { monitorWakeNeedsRecovery } from './hosted-monitor-wake-turn.js'; +import { + createMonitorWakeRunTurn, + monitorWakeNeedsRecovery, +} from './hosted-monitor-wake-turn.js'; import { HostedChildAgentSession } from './hosted-child-agent-session.js'; import { HostedSessionMessageSession } from './hosted-session-message-session.js'; import { HostedTeamSession } from './hosted-team-session.js'; @@ -1732,6 +1742,59 @@ describe('Hosted Harness no-tool session', () => { }, ); + it('aborts an active Turn and returns 503 when durable cancellation fails', async () => { + const { server, authorize } = await hookApp(); + let signal!: AbortSignal; + let finish!: () => void; + state.model.mockImplementationOnce( + (input) => + new Promise((resolve) => { + signal = input.signal; + finish = () => resolve({ text: '', model: 'test-model' }); + }), + ); + const prompt = [ + { type: 'text', text: 'cancel while Store is unavailable' }, + ]; + const payloadDigest = `sha256:${createHash('sha256').update(JSON.stringify(prompt)).digest('hex')}`; + await authorize(supertest(server).post(`/session/${SESSION_ID}/prompt`)) + .send({ prompt, promptId: PROMPT_ID, payloadDigest }) + .expect(202); + await vi.waitFor(() => expect(state.model).toHaveBeenCalledOnce()); + const original = + LocalManagedSessionAuthority.prototype.appendExecutionEvent; + const append = vi + .spyOn(LocalManagedSessionAuthority.prototype, 'appendExecutionEvent') + .mockImplementation(function ( + this: LocalManagedSessionAuthority, + ...args + ) { + if (args[0].operation === 'requestCancel') + return Promise.reject(new Error('Store unavailable')); + return Reflect.apply(original, this, args); + }); + try { + const response = await authorize( + supertest(server).post(`/session/${SESSION_ID}/cancel`), + ); + expect(response.status).toBe(503); + expect(response.body.code).toBe('hosted_cancel_commit_failed'); + expect(signal.aborted).toBe(true); + } finally { + append.mockRestore(); + finish(); + } + await vi.waitFor(async () => { + const status = await authorize( + supertest(server).get(`/session/${SESSION_ID}/status`), + ); + expect(status.body.hasActivePrompt).toBe(false); + }); + await authorize( + supertest(server).post(`/session/${SESSION_ID}/detach`), + ).expect(204); + }); + it.each([ new TypeError('network timeout'), new ManagedSessionStoreHttpError( @@ -3296,6 +3359,9 @@ describe('Hosted Harness no-tool session', () => { it('refuses a prompt retriably when the wake park probe read nothing', async () => { const inputId = await prewriteAutomationParkedWakeSession(); const repair = mockRepairBroker(); + const kick = vi + .spyOn(HostedMonitorWakeScheduler.prototype, 'kick') + .mockImplementation(() => {}); const server = await app(true); const loaded = await headers( supertest(server).post(`/session/${SESSION_ID}/load`), @@ -3340,6 +3406,8 @@ describe('Hosted Harness no-tool session', () => { // in later tests' stderr spies. The pump's own view is the journal — // watch the consume, not a route. checkpointRead.mockRestore(); + kick.mockRestore(); + capturedWakeSession().monitorWake!.kick(); await vi.waitFor( async () => { const journal = await LocalJsonlManagedSessionJournalStore.read( @@ -4816,7 +4884,21 @@ describe('Hosted Harness no-tool session', () => { ) .map((event) => event.payload['operationId']), ).toEqual([`${runId}:1`, `${runId}:2`]); - await headers(supertest(server).delete(`/session/${SESSION_ID}`)); + await vi.waitFor( + async () => { + expect((await operations(fireBody(1, slot))).body.run.state).toBe( + 'settled', + ); + const status = await authorize( + supertest(server).get(`/session/${SESSION_ID}/status`), + ); + expect(status.body.hasActivePrompt).toBe(false); + }, + { timeout: 10_000 }, + ); + await headers(supertest(server).delete(`/session/${SESSION_ID}`)).expect( + 204, + ); }); it('answers an operation against a dead journal writer with a retryable 503', async () => { @@ -5275,7 +5357,10 @@ describe('Hosted Harness no-tool session', () => { * checkpoint and file_history.pendingTurn. Shared by the attachment * and the transient-retry witnesses. */ - async function prewriteChannelWriteInterruption(): Promise { + async function prewriteChannelWriteInterruption( + cleanupDebt = false, + previousPromptId?: string, + ): Promise { const key = { tenantId: 'tenant', workspaceId: 'workspace', @@ -5309,7 +5394,9 @@ describe('Hosted Harness no-tool session', () => { engine: 'managed', sessionId: SESSION_ID, toolProfile: 'hosted-workspace-files/1', - hookCatalog: hookPin, + ...(cleanupDebt || previousPromptId + ? {} + : { hookCatalog: hookPin }), }), ), ), @@ -5337,6 +5424,50 @@ describe('Hosted Harness no-tool session', () => { }, key, ); + + if (previousPromptId) { + const content = Buffer.from( + JSON.stringify([{ type: 'text', text: 'prior completed turn' }]), + ); + await managed.authority.submitInput( + { + operation: 'submitInput', + commandId: previousPromptId, + sessionKey: key, + contentDigest: createHash('sha256').update(content).digest('hex'), + }, + { + inputId: previousPromptId, + turnId: previousPromptId, + source: 'hosted-harness', + contentRef: await resources.publish('managed-input', content), + admissionRef: await resources.publish( + 'managed-admission', + Buffer.from('{}'), + ), + deadline: null, + wakeReason: 'input', + }, + ); + await createManagedHarnessHandle(managed).ensureRunnable(); + await managed.sink.write({ + uuid: `${previousPromptId}:terminal`, + parentUuid: null, + sessionId: SESSION_ID, + timestamp: new Date().toISOString(), + type: 'system', + subtype: 'turn_result', + cwd: state.root, + version: 'hosted-harness/1', + daemonPromptId: previousPromptId, + systemPayload: { + promptId: previousPromptId, + state: 'completed', + stopReason: 'end_turn', + endedAt: Date.now(), + }, + } as ChatRecord); + } await channels.submitInput({ inputId: channelTurn, channelInstanceId: channelIdentity.channelInstanceId, @@ -5379,6 +5510,19 @@ describe('Hosted Harness no-tool session', () => { } as ChatRecord); const harness = createManagedHarnessHandle(managed); await harness.ensureRunnable(); + if (cleanupDebt) { + const broker = new HostedWorkspaceBroker( + { baseUrl: 'http://127.0.0.1:1', token: 'test' }, + key, + channelTurn, + ); + broker.runtime = { + bindingId: 'binding', + generation: '1', + workspaceGeneration: '1', + }; + await oweHostedTurnCleanup(managed, channelTurn, broker); + } const activation = managed.activation; const executionId = 'exec-channel-write'; const toolDefinitionRef = await resources.publish( @@ -5602,6 +5746,137 @@ describe('Hosted Harness no-tool session', () => { ).toBe(204); }); + it('parks an admitted prompt when model request preparation fails', async () => { + const server = await app(); + const created = await headers(supertest(server).post('/session')) + .send({ + sessionId: SESSION_ID, + sessionScope: 'thread', + managedSessionStore: store(), + }) + .expect(200); + const authorize = (request: supertest.Test) => + headers(request).set('X-Qwen-Client-Id', created.body.clientId as string); + state.model.mockRejectedValueOnce( + new LlmRequestPreparationError('snapshot durability failed'), + ); + const prompt = [{ type: 'text', text: 'hello' }]; + const payloadDigest = `sha256:${createHash('sha256').update(JSON.stringify(prompt)).digest('hex')}`; + await authorize(supertest(server).post(`/session/${SESSION_ID}/prompt`)) + .send({ prompt, promptId: PROMPT_ID, payloadDigest }) + .expect(202); + await vi.waitFor(async () => { + const status = await authorize( + supertest(server).get(`/session/${SESSION_ID}/status`), + ).expect(200); + expect(status.body).toMatchObject({ + hasActivePrompt: false, + recoveryBlocked: true, + }); + }); + const journal = await LocalJsonlManagedSessionJournalStore.read( + path.join(state.root, `${SESSION_ID}.jsonl`), + { tenantId: 'tenant', workspaceId: 'workspace', sessionId: SESSION_ID }, + ); + expect( + journal.events.filter((event) => event.kind === 'input.accepted'), + ).toHaveLength(1); + expect(journal.events.some((event) => event.kind === 'turn.settled')).toBe( + false, + ); + expect(state.model).toHaveBeenCalledTimes(1); + await authorize(supertest(server).delete(`/session/${SESSION_ID}`)).expect( + 204, + ); + }); + + it('retains cleanup debt after interrupted channel settlement until original-owner release succeeds', async () => { + const channelTurn = await prewriteChannelWriteInterruption(true); + mockBrokerBroker(); + vi.spyOn(HostedWorkspaceBroker.prototype, 'status').mockResolvedValue({ + state: 'settled', + }); + const release = vi.mocked(HostedWorkspaceBroker.prototype.release); + release.mockRejectedValueOnce(new Error('release unavailable')); + const server = await app(true); + const loaded = await headers( + supertest(server).post(`/session/${SESSION_ID}/load`), + ).send({ + managedSessionStore: store(), + toolProfile: 'hosted-workspace-files/1', + }); + expect(loaded.status).toBe(200); + const authorize = (request: supertest.Test) => + headers(request).set('X-Qwen-Client-Id', loaded.body.clientId as string); + const readJournal = () => + LocalJsonlManagedSessionJournalStore.read( + path.join(state.root, `${SESSION_ID}.jsonl`), + { tenantId: 'tenant', workspaceId: 'workspace', sessionId: SESSION_ID }, + ); + await vi.waitFor(async () => { + expect(release).toHaveBeenCalledTimes(1); + const status = await authorize( + supertest(server).get(`/session/${SESSION_ID}/status`), + ).expect(200); + expect(status.body.recoveryBlocked).toBe(true); + }); + const parked = await readJournal(); + const terminals = parked.events.filter( + (event) => event.kind === 'turn.settled', + ); + expect(terminals).toHaveLength(1); + expect(terminals[0]!.payload['turnId']).toBe(channelTurn); + expect( + parked.events.filter((event) => event.kind === 'hosted.cleanup'), + ).toHaveLength(1); + expect(release).toHaveBeenLastCalledWith({ + bindingId: 'binding', + generation: '1', + }); + expect(state.model).not.toHaveBeenCalled(); + const nextPrompt = [{ type: 'text', text: 'must remain blocked' }]; + await authorize(supertest(server).post(`/session/${SESSION_ID}/prompt`)) + .send({ + prompt: nextPrompt, + promptId: randomUUID(), + payloadDigest: `sha256:${createHash('sha256').update(JSON.stringify(nextPrompt)).digest('hex')}`, + }) + .expect(409); + expect((await readJournal()).events).toHaveLength(parked.events.length); + expect(state.model).not.toHaveBeenCalled(); + const redriven = await headers( + supertest(server).post(`/session/${SESSION_ID}/load`), + ).send({ + managedSessionStore: store(), + passiveManagedRuntimeRecovery: true, + }); + expect(redriven.status).toBe(200); + expect(redriven.body.clientId).toBe(loaded.body.clientId); + expect(release).toHaveBeenCalledTimes(2); + expect(release).toHaveBeenLastCalledWith({ + bindingId: 'binding', + generation: '1', + }); + const status = await authorize( + supertest(server).get(`/session/${SESSION_ID}/status`), + ).expect(200); + expect(status.body.recoveryBlocked).toBe(false); + const journal = await readJournal(); + const confirmed = journal.events.filter( + (event) => + event.kind === 'hosted.cleanup' && + event.payload['state'] === 'confirmed', + ); + expect(confirmed).toHaveLength(1); + expect(confirmed[0]!.sequence).toBeGreaterThan(terminals[0]!.sequence); + expect( + journal.events.filter((event) => event.kind === 'turn.settled'), + ).toHaveLength(1); + await authorize(supertest(server).delete(`/session/${SESSION_ID}`)).expect( + 204, + ); + }); + it('crosses a durable store fault to 503 instead of mislabelling it as 400 (F10)', async () => { const server = await app(true); const created = await headers(supertest(server).post('/session')).send({ @@ -5838,6 +6113,10 @@ describe('Hosted Harness no-tool session', () => { event.payload['turnId'] === notificationTurnId, ), ).toBe(true); + const status = await headers( + supertest(server).get(`/session/${SESSION_ID}/status`), + ).set('X-Qwen-Client-Id', loaded.body.clientId as string); + expect(status.body.hasActivePrompt).toBe(false); }, { timeout: 15_000, interval: 100 }, ); @@ -12511,13 +12790,574 @@ describe('Hosted Harness no-tool session', () => { ); }); - // Parks a plain Session with its Turn unsettled: the settlement's durable - // write is refused, so the input stays accepted-but-unsettled. - async function parkUnsettledPlainTurn(server: Server): Promise { + it('clears predispatch partial intent file history before the cancelled terminal', async () => { + let originalManaged: Awaited>; + vi.spyOn(HostedWorkspaceBroker.prototype, 'warm').mockImplementation( + async function (this: HostedWorkspaceBroker) { + this.runtime = { + bindingId: 'original-binding', + generation: '1', + workspaceGeneration: '1', + }; + }, + ); + vi.spyOn(HostedWorkspaceBroker.prototype, 'acquire').mockResolvedValue(); + const release = vi + .spyOn(HostedWorkspaceBroker.prototype, 'release') + .mockResolvedValue(); + const execute = vi.spyOn(HostedWorkspaceBroker.prototype, 'execute'); + let prepareOrdinal = 0; + vi.spyOn(HostedWorkspaceBroker.prototype, 'prepare').mockImplementation( + async () => { + if (prepareOrdinal++ === 0) return 'original-execution-1'; + const authority = originalManaged.authority; + const commandId = `hosted-cancel:${PROMPT_ID}`; + await authority.appendExecutionEvent( + { + operation: 'requestCancel', + commandId, + sessionKey: authority.sessionHeader.sessionKey, + contentDigest: createHash('sha256').update(PROMPT_ID).digest('hex'), + }, + (sequence) => ({ + v: 1, + sequence, + eventId: commandId, + sessionKey: authority.sessionHeader.sessionKey, + kind: 'cancel.requested', + occurredAt: Date.now(), + payload: { + requestId: commandId, + target: { turnId: PROMPT_ID }, + reason: 'user', + requestedBy: 'hosted', + }, + }), + { class: 'trusted_entry' }, + ); + throw new Error('crash after first prepared intent'); + }, + ); + state.model.mockImplementationOnce(async ({ toolTurn, signal }) => { + originalManaged = ( + toolTurn as unknown as { session: typeof originalManaged } + ).session; + const calls = [0, 1].map((ordinal) => ({ + name: 'write_file', + callId: `original-call-${ordinal}`, + args: { file_path: `original-${ordinal}.txt`, content: 'original' }, + isClientInitiated: false, + prompt_id: PROMPT_ID, + })); + await toolTurn!.execute( + calls, + calls.map((call) => ({ + functionCall: { id: call.callId, name: call.name, args: call.args }, + })), + 'test-model', + signal, + ); + return { text: 'never reached', model: 'test-model' }; + }); + const server = await app(true); const created = await headers(supertest(server).post('/session')).send({ sessionId: SESSION_ID, sessionScope: 'thread', managedSessionStore: store(), + toolProfile: 'hosted-workspace-files/1', + approvalMode: 'yolo', + }); + expect(created.status).toBe(200); + const prompt = [{ type: 'text', text: 'write original' }]; + await headers(supertest(server).post(`/session/${SESSION_ID}/prompt`)) + .set('X-Qwen-Client-Id', created.body.clientId) + .send({ + prompt, + promptId: PROMPT_ID, + payloadDigest: `sha256:${createHash('sha256').update(JSON.stringify(prompt)).digest('hex')}`, + }) + .expect(202); + await vi.waitFor(async () => { + const status = await headers( + supertest(server).get(`/session/${SESSION_ID}/status`), + ).set('X-Qwen-Client-Id', created.body.clientId); + expect(status.body).toMatchObject({ + hasActivePrompt: false, + recoveryBlocked: true, + }); + }); + const before = await LocalJsonlManagedSessionJournalStore.read( + path.join(state.root, `${SESSION_ID}.jsonl`), + { tenantId: 'tenant', workspaceId: 'workspace', sessionId: SESSION_ID }, + ); + expect( + before.events.filter((event) => event.kind === 'tool.intent'), + ).toHaveLength(1); + expect(execute).not.toHaveBeenCalled(); + await headers(supertest(server).post(`/session/${SESSION_ID}/detach`)) + .set('X-Qwen-Client-Id', created.body.clientId) + .expect(204); + const replacementServer = await app(true); + const loaded = await headers( + supertest(replacementServer).post(`/session/${SESSION_ID}/load`), + ).send({ managedSessionStore: store(), driveRuntimeRecovery: true }); + expect(loaded.status).toBe(200); + await vi.waitFor(async () => { + const status = await headers( + supertest(replacementServer).get(`/session/${SESSION_ID}/status`), + ).set('X-Qwen-Client-Id', loaded.body.clientId); + expect(status.body).toMatchObject({ + hasActivePrompt: false, + recoveryBlocked: false, + }); + }); + const after = await LocalJsonlManagedSessionJournalStore.read( + path.join(state.root, `${SESSION_ID}.jsonl`), + { tenantId: 'tenant', workspaceId: 'workspace', sessionId: SESSION_ID }, + ); + const terminals = after.events.filter( + (event) => event.kind === 'turn.settled', + ); + expect(terminals).toHaveLength(1); + expect(terminals[0].payload['outcome']).toBe('cancelled'); + expect(state.model).toHaveBeenCalledTimes(1); + expect(execute).not.toHaveBeenCalled(); + expect(release).toHaveBeenCalledTimes(1); + await headers( + supertest(replacementServer).post(`/session/${SESSION_ID}/detach`), + ) + .set('X-Qwen-Client-Id', loaded.body.clientId) + .expect(204); + const nextServer = await app(true); + const reloaded = await headers( + supertest(nextServer).post(`/session/${SESSION_ID}/load`), + ).send({ managedSessionStore: store() }); + expect(reloaded.status).toBe(200); + const nextPromptId = '55555555-5555-4555-8555-555555555555'; + const nextPrompt = [{ type: 'text', text: 'say hello after cancellation' }]; + await headers(supertest(nextServer).post(`/session/${SESSION_ID}/prompt`)) + .set('X-Qwen-Client-Id', reloaded.body.clientId) + .send({ + prompt: nextPrompt, + promptId: nextPromptId, + payloadDigest: `sha256:${createHash('sha256').update(JSON.stringify(nextPrompt)).digest('hex')}`, + }) + .expect(202); + await vi.waitFor(async () => { + const journal = await LocalJsonlManagedSessionJournalStore.read( + path.join(state.root, `${SESSION_ID}.jsonl`), + { tenantId: 'tenant', workspaceId: 'workspace', sessionId: SESSION_ID }, + ); + expect( + journal.events.filter( + (event) => + event.kind === 'turn.settled' && + event.payload['turnId'] === nextPromptId, + ), + ).toHaveLength(1); + const status = await headers( + supertest(nextServer).get(`/session/${SESSION_ID}/status`), + ).set('X-Qwen-Client-Id', reloaded.body.clientId); + expect(status.body).toMatchObject({ + hasActivePrompt: false, + recoveryBlocked: false, + }); + }); + expect(state.model).toHaveBeenCalledTimes(2); + expect(execute).not.toHaveBeenCalled(); + await headers(supertest(nextServer).post(`/session/${SESSION_ID}/detach`)) + .set('X-Qwen-Client-Id', reloaded.body.clientId) + .expect(204); + }); + it('targets the original active turn when the durable cancel reply arrives late', async () => { + let originalManaged: Awaited>; + let finishModel!: () => void; + let releaseCancelAck!: () => void; + let cancelCommitted = false; + const modelGate = new Promise((resolve) => { + finishModel = resolve; + }); + const cancelAckGate = new Promise((resolve) => { + releaseCancelAck = resolve; + }); + vi.spyOn(HostedWorkspaceBroker.prototype, 'warm').mockResolvedValue(); + state.model.mockImplementationOnce(async ({ toolTurn }) => { + originalManaged = ( + toolTurn as unknown as { session: typeof originalManaged } + ).session; + await modelGate; + return { text: 'original answer completed', model: 'test-model' }; + }); + const server = await app(true); + const created = await headers(supertest(server).post('/session')).send({ + sessionId: SESSION_ID, + sessionScope: 'thread', + managedSessionStore: store(), + toolProfile: 'hosted-workspace-files/1', + approvalMode: 'yolo', + }); + expect(created.status).toBe(200); + const prompt = [{ type: 'text', text: 'complete original answer' }]; + await headers(supertest(server).post(`/session/${SESSION_ID}/prompt`)) + .set('X-Qwen-Client-Id', created.body.clientId) + .send({ + prompt, + promptId: PROMPT_ID, + payloadDigest: `sha256:${createHash('sha256').update(JSON.stringify(prompt)).digest('hex')}`, + }) + .expect(202); + await vi.waitFor(() => expect(originalManaged).toBeDefined()); + const append = originalManaged!.authority.appendExecutionEvent.bind( + originalManaged!.authority, + ); + vi.spyOn( + originalManaged!.authority, + 'appendExecutionEvent', + ).mockImplementation(async (...args) => { + const result = await append(...args); + if (args[0].operation === 'requestCancel') { + cancelCommitted = true; + await cancelAckGate; + } + return result; + }); + const cancelResponse = headers( + supertest(server).post(`/session/${SESSION_ID}/cancel`), + ) + .set('X-Qwen-Client-Id', created.body.clientId) + .then((response) => response); + await vi.waitFor(() => expect(cancelCommitted).toBe(true)); + finishModel(); + await vi.waitFor(async () => { + const status = await headers( + supertest(server).get(`/session/${SESSION_ID}/status`), + ).set('X-Qwen-Client-Id', created.body.clientId); + expect(status.body).toMatchObject({ + hasActivePrompt: false, + recoveryBlocked: false, + }); + }); + let nextSignal: AbortSignal | undefined; + let finishNextModel!: () => void; + const nextModelGate = new Promise((resolve) => { + finishNextModel = resolve; + }); + state.model.mockImplementationOnce(async ({ signal }) => { + nextSignal = signal; + await nextModelGate; + return { text: 'next answer completed', model: 'test-model' }; + }); + const nextPrompt = [{ type: 'text', text: 'complete the next answer' }]; + await headers(supertest(server).post(`/session/${SESSION_ID}/prompt`)) + .set('X-Qwen-Client-Id', created.body.clientId) + .send({ + prompt: nextPrompt, + promptId: randomUUID(), + payloadDigest: `sha256:${createHash('sha256').update(JSON.stringify(nextPrompt)).digest('hex')}`, + }) + .expect(202); + await vi.waitFor(() => expect(nextSignal).toBeDefined()); + releaseCancelAck(); + const cancelled = await cancelResponse; + expect(cancelled.status).toBe(204); + expect(nextSignal!.aborted).toBe(false); + finishNextModel(); + await vi.waitFor(async () => { + const status = await headers( + supertest(server).get(`/session/${SESSION_ID}/status`), + ).set('X-Qwen-Client-Id', created.body.clientId); + expect(status.body).toMatchObject({ + hasActivePrompt: false, + recoveryBlocked: false, + }); + }); + const journal = await LocalJsonlManagedSessionJournalStore.read( + path.join(state.root, `${SESSION_ID}.jsonl`), + { tenantId: 'tenant', workspaceId: 'workspace', sessionId: SESSION_ID }, + ); + expect( + journal.events.filter((event) => event.kind === 'cancel.requested'), + ).toHaveLength(1); + expect( + journal.events.filter((event) => event.kind === 'turn.settled'), + ).toHaveLength(2); + await headers(supertest(server).post(`/session/${SESSION_ID}/detach`)) + .set('X-Qwen-Client-Id', created.body.clientId) + .expect(204); + }); + + it.each(['expired', 'cancelled'])( + 'settles a saved final native plan without new inference (%s)', + async (actionEnd) => { + let originalManaged: import('@qwen-code/qwen-code-core/managed-runtime/managed-session-assembly.js').ManagedSession; + vi.spyOn(HostedWorkspaceBroker.prototype, 'warm').mockImplementation( + async function (this: HostedWorkspaceBroker) { + this.runtime = { + bindingId: 'original-binding', + generation: '1', + workspaceGeneration: '1', + }; + }, + ); + vi.spyOn(HostedWorkspaceBroker.prototype, 'acquire').mockResolvedValue(); + vi.spyOn(HostedWorkspaceBroker.prototype, 'release').mockResolvedValue(); + vi.spyOn(HostedApprovalWaiters.prototype, 'wait').mockImplementation( + async (requestId) => { + if (actionEnd !== 'cancelled') return; + const authority = originalManaged.authority; + const commandId = `hosted-cancel:${PROMPT_ID}`; + await authority.appendExecutionEvent( + { + operation: 'requestCancel', + commandId, + sessionKey: authority.sessionHeader.sessionKey, + contentDigest: createHash('sha256') + .update(PROMPT_ID) + .digest('hex'), + }, + (sequence) => ({ + v: 1, + sequence, + eventId: commandId, + sessionKey: authority.sessionHeader.sessionKey, + kind: 'cancel.requested', + occurredAt: Date.now(), + payload: { + requestId: commandId, + target: { turnId: PROMPT_ID }, + reason: 'user', + requestedBy: 'hosted', + }, + }), + { class: 'trusted_entry' }, + ); + await endHostedAction(originalManaged, requestId, 'cancelled'); + }, + ); + const originalWrite = ManagedSessionRecordSink.prototype.write; + const crash = vi + .spyOn(ManagedSessionRecordSink.prototype, 'write') + .mockImplementation(function (this: ManagedSessionRecordSink, record) { + if (record.type === 'tool_result') + return Promise.reject(new Error('crash after final plan')); + return originalWrite.call(this, record); + }); + state.model.mockImplementationOnce(async ({ toolTurn, signal }) => { + originalManaged = ( + toolTurn as unknown as { session: typeof originalManaged } + ).session; + const call = { + name: 'write_file', + callId: 'original-call', + args: { file_path: 'original.txt', content: 'original' }, + isClientInitiated: false, + prompt_id: PROMPT_ID, + }; + await toolTurn!.execute( + [call], + [ + { + functionCall: { + id: call.callId, + name: call.name, + args: call.args, + }, + }, + ], + 'test-model', + signal, + ); + return { text: 'never reached', model: 'test-model' }; + }); + const server = await app(true); + const created = await headers(supertest(server).post('/session')).send({ + sessionId: SESSION_ID, + sessionScope: 'thread', + managedSessionStore: store(), + toolProfile: 'hosted-workspace-files/1', + approvalMode: 'default', + approvalTimeoutMs: 1000, + }); + expect(created.status).toBe(200); + const prompt = [{ type: 'text', text: 'write original' }]; + await headers(supertest(server).post(`/session/${SESSION_ID}/prompt`)) + .set('X-Qwen-Client-Id', created.body.clientId) + .send({ + prompt, + promptId: PROMPT_ID, + payloadDigest: `sha256:${createHash('sha256').update(JSON.stringify(prompt)).digest('hex')}`, + }) + .expect(202); + await vi.waitFor(async () => { + const status = await headers( + supertest(server).get(`/session/${SESSION_ID}/status`), + ).set('X-Qwen-Client-Id', created.body.clientId); + expect(status.body.hasActivePrompt).toBe(false); + expect(status.body.recoveryBlocked).toBe(true); + }); + await headers(supertest(server).post(`/session/${SESSION_ID}/detach`)) + .set('X-Qwen-Client-Id', created.body.clientId) + .expect(204); + crash.mockRestore(); + const replacementServer = await app(true); + const loaded = await headers( + supertest(replacementServer).post(`/session/${SESSION_ID}/load`), + ).send({ managedSessionStore: store(), driveRuntimeRecovery: true }); + expect(loaded.status).toBe(200); + await vi.waitFor(async () => { + const journal = await LocalJsonlManagedSessionJournalStore.read( + path.join(state.root, `${SESSION_ID}.jsonl`), + { + tenantId: 'tenant', + workspaceId: 'workspace', + sessionId: SESSION_ID, + }, + ); + const terminals = journal.events.filter( + (event) => event.kind === 'turn.settled', + ); + expect(terminals).toHaveLength(1); + expect(terminals[0].payload['outcome']).toBe('cancelled'); + }); + expect(state.model).toHaveBeenCalledTimes(1); + await vi.waitFor(async () => { + const status = await headers( + supertest(replacementServer).get(`/session/${SESSION_ID}/status`), + ).set('X-Qwen-Client-Id', loaded.body.clientId); + expect(status.body).toMatchObject({ + hasActivePrompt: false, + recoveryBlocked: false, + }); + }); + await headers( + supertest(replacementServer).post(`/session/${SESSION_ID}/detach`), + ) + .set('X-Qwen-Client-Id', loaded.body.clientId) + .expect(204); + }, + ); + + it.each([ + { broker: false, toolProfile: undefined, redrive: false }, + { broker: true, toolProfile: undefined, redrive: false }, + { broker: true, toolProfile: 'hosted-workspace-files/1', redrive: false }, + { broker: false, toolProfile: undefined, redrive: true }, + ])( + 'compensates a committed final answer without inference or unowned release (%o)', + async (scenario) => { + const withBroker = scenario.broker; + const prepared = { + request: { + model: 'test-model', + contents: [{ role: 'user', parts: [{ text: 'park me' }] }], + config: { temperature: 0.3 }, + }, + history: [{ role: 'user', parts: [{ text: 'park me' }] }], + completedToolCallIds: [], + routeSelector: 'openai:test-model', + providerPin: 'a'.repeat(64), + promptTokensForClamp: 12, + }; + state.model.mockImplementation(async (input) => { + const prepare = ( + input as unknown as { + prepareModel: ( + request: unknown, + round: number, + pending: boolean, + context?: string, + ) => Promise< + | import('@qwen-code/qwen-code-core/managed-runtime/managed-session-records.js').ManagedSessionDurableRef + | undefined + >; + } + ).prepareModel; + const ref = await prepare(prepared, 0, false); + const finish = await input.modelScope!.beginMainAttempt( + 'test-model', + ref, + ); + await finish(true, {}); + return { text: 'committed original answer', model: 'test-model' }; + }); + const server = await app(withBroker); + await parkUnsettledPlainTurn(server, scenario.toolProfile); + if (!scenario.redrive) + vi.mocked( + LocalManagedSessionResourceStore.prototype.publish, + ).mockRestore(); + const release = vi + .spyOn(HostedWorkspaceBroker.prototype, 'release') + .mockImplementation(async () => { + throw new Error('no tool profile must not release runtime'); + }); + const replacementServer = await app(withBroker); + const loaded = await headers( + supertest(replacementServer).post(`/session/${SESSION_ID}/load`), + ).send({ managedSessionStore: store(), driveRuntimeRecovery: true }); + expect(loaded.status).toBe(200); + if (scenario.redrive) { + await vi.waitFor(async () => { + const status = await headers( + supertest(replacementServer).get(`/session/${SESSION_ID}/status`), + ).set('X-Qwen-Client-Id', loaded.body.clientId); + expect(status.body).toMatchObject({ + hasActivePrompt: false, + recoveryBlocked: true, + }); + }); + vi.mocked( + LocalManagedSessionResourceStore.prototype.publish, + ).mockRestore(); + await headers( + supertest(replacementServer).post(`/session/${SESSION_ID}/load`), + ) + .send({ managedSessionStore: store(), driveRuntimeRecovery: true }) + .expect(200); + } + await vi.waitFor(async () => { + const journal = await LocalJsonlManagedSessionJournalStore.read( + path.join(state.root, `${SESSION_ID}.jsonl`), + { + tenantId: 'tenant', + workspaceId: 'workspace', + sessionId: SESSION_ID, + }, + ); + expect( + journal.events.filter((event) => event.kind === 'turn.settled'), + ).toHaveLength(1); + }); + expect(state.model).toHaveBeenCalledTimes(1); + expect(release).not.toHaveBeenCalled(); + await vi.waitFor(async () => { + const status = await headers( + supertest(replacementServer).get(`/session/${SESSION_ID}/status`), + ).set('X-Qwen-Client-Id', loaded.body.clientId); + expect(status.body).toMatchObject({ + hasActivePrompt: false, + recoveryBlocked: false, + }); + }); + await headers( + supertest(replacementServer).post(`/session/${SESSION_ID}/detach`), + ) + .set('X-Qwen-Client-Id', loaded.body.clientId) + .expect(204); + }, + ); + + // Parks a plain Session with its Turn unsettled: the settlement's durable + // write is refused, so the input stays accepted-but-unsettled. + async function parkUnsettledPlainTurn( + server: Server, + toolProfile?: string, + ): Promise { + const created = await headers(supertest(server).post('/session')).send({ + sessionId: SESSION_ID, + sessionScope: 'thread', + managedSessionStore: store(), + ...(toolProfile ? { toolProfile } : {}), }); const clientId = created.body.clientId as string; const log = vi @@ -14809,6 +15649,688 @@ describe('Hosted Harness no-tool session', () => { expect(state.model).not.toHaveBeenCalled(); await headers(supertest(server).delete(`/session/${SESSION_ID}`)); }); + function capturedWakeSession() { + return ( + wakeDeps.last as { + session: { + managed: Awaited>; + runtimeLeaseHeld?: { promptId: string; runtimeSessionId: string }; + blocked: boolean; + cleanupBlocked?: boolean; + mcpClosing?: boolean; + monitorWake?: HostedMonitorWakeScheduler; + }; + } + ).session; + } + + async function recoveryFileSession() { + mockBrokerBroker(); + const server = await app(true); + const created = await headers(supertest(server).post('/session')) + .send({ + sessionId: SESSION_ID, + sessionScope: 'thread', + managedSessionStore: store(), + toolProfile: 'hosted-workspace-files/1', + approvalMode: 'yolo', + }) + .expect(200); + const authorize = (request: supertest.Test) => + headers(request).set('X-Qwen-Client-Id', created.body.clientId); + const prompt = [{ type: 'text', text: 'original' }]; + const submit = (deadlineMs?: number) => + authorize(supertest(server).post(`/session/${SESSION_ID}/prompt`)) + .send({ + prompt, + promptId: PROMPT_ID, + payloadDigest: `sha256:${createHash('sha256').update(JSON.stringify(prompt)).digest('hex')}`, + ...(deadlineMs ? { deadlineMs } : {}), + }) + .expect(202); + const status = () => + authorize(supertest(server).get(`/session/${SESSION_ID}/status`)).expect( + 200, + ); + const idle = () => + vi.waitFor( + async () => expect((await status()).body.hasActivePrompt).toBe(false), + { timeout: 10_000 }, + ); + const resident = capturedWakeSession(); + const owe = async (promptId = PROMPT_ID) => { + const broker = new HostedWorkspaceBroker( + { baseUrl: 'http://127.0.0.1:1', token: 'test' }, + resident.managed.authority.sessionHeader.sessionKey, + hostedRuntimeSessionId(promptId), + ); + broker.runtime = { + bindingId: 'binding', + generation: '1', + workspaceGeneration: '1', + }; + await oweHostedTurnCleanup(resident.managed, promptId, broker); + }; + return { server, authorize, submit, status, idle, resident, owe }; + } + + it('clears a cleanup-only block when continue confirms the original owner (R3-14)', async () => { + const f = await recoveryFileSession(); + f.resident.runtimeLeaseHeld = { + promptId: PROMPT_ID, + runtimeSessionId: hostedRuntimeSessionId(PROMPT_ID), + }; + state.model.mockImplementationOnce(async () => { + await f.owe(); + return { text: 'completed', model: 'test-model' }; + }); + vi.spyOn( + HostedWorkspaceToolTurn.prototype, + 'cleanup', + ).mockRejectedValueOnce(new Error('release unavailable')); + await f.submit(); + await f.idle(); + expect(f.resident.cleanupBlocked).toBe(true); + const authorization = + await f.resident.managed.authority.harnessRunAuthorization(); + if (authorization.status !== 'runnable') + throw new Error('fixture checkpoint is not runnable'); + await f + .authorize( + supertest(f.server).post( + `/session/${SESSION_ID}/managed-runtime/continue`, + ), + ) + .send({ + promptId: PROMPT_ID, + checkpointId: authorization.checkpoint.identity.checkpointId, + activationId: f.resident.managed.activation.activationId, + }) + .expect(409); + await vi.waitFor(() => { + expect(f.resident.runtimeLeaseHeld).toBeUndefined(); + expect( + f.resident.managed.authority + .eventsInSequenceRange( + 1, + f.resident.managed.authority.committedSequence, + ) + .some( + (event) => + event.kind === 'hosted.cleanup' && + event.payload['state'] === 'confirmed', + ), + ).toBe(true); + }); + const loaded = await headers( + supertest(f.server).post(`/session/${SESSION_ID}/load`), + ) + .send({ + managedSessionStore: store(), + passiveManagedRuntimeRecovery: true, + }) + .expect(200); + expect(loaded.body.recoveryRequired).not.toBe(true); + expect((await f.status()).body.recoveryBlocked).toBe(false); + expect(f.resident.cleanupBlocked).toBe(false); + await f + .authorize(supertest(f.server).delete(`/session/${SESSION_ID}`)) + .expect(204); + }); + + it('leaves a parked owner operable after lifecycle close requires recovery (R3-5)', async () => { + const f = await recoveryFileSession(); + state.model.mockImplementationOnce(async () => { + await f.owe(); + throw new LlmRequestPreparationError('parked request'); + }); + await f.submit(); + await f.idle(); + state.setLifecycleAuthority.mockClear(); + await f + .authorize(supertest(f.server).post(`/session/${SESSION_ID}/lifecycle`)) + .send({ + kind: 'close', + sessionKey: f.resident.managed.authority.sessionHeader.sessionKey, + authority: { operationId: 'close-parked', claimGeneration: 1 }, + }) + .expect(503); + expect(state.setLifecycleAuthority).not.toHaveBeenCalled(); + const cancel = await f + .authorize( + supertest(f.server).post( + `/session/${SESSION_ID}/managed-runtime/cancel`, + ), + ) + .send({ + promptId: PROMPT_ID, + checkpointId: 'checkpoint', + activationId: f.resident.managed.activation.activationId, + }); + expect(cancel.body.code).not.toBe('hosted_session_closing'); + expect(cancel.body.code).not.toBe('hosted_lifecycle_operation_active'); + await f + .authorize(supertest(f.server).delete(`/session/${SESSION_ID}`)) + .expect(204); + }); + + it('keeps a journaled cancellation after the recovered request deadline passes (R3-2)', async () => { + const f = await recoveryFileSession(); + let modelWaiting = false; + state.model.mockImplementationOnce(async ({ modelScope, signal }) => { + const prepared: LlmPreparedRequest = { + request: { + model: 'fixture', + contents: [{ role: 'user', parts: [{ text: 'original' }] }], + config: { temperature: 0.3 }, + }, + history: [{ role: 'user', parts: [{ text: 'original' }] }], + completedToolCallIds: [], + routeSelector: 'openai:fixture', + providerPin: 'a'.repeat(64), + promptTokensForClamp: 12, + }; + const request = modelRecovery.createHostedModelRequest( + f.resident.managed, + PROMPT_ID, + prepared, + 0, + false, + undefined, + { bootId: BOOT_ID, eventEpoch: 'source-epoch' }, + ); + await modelScope!.beginMainAttempt( + 'fixture', + await modelRecovery.publishHostedModelRequest( + f.resident.managed, + request, + ), + ); + return new Promise((_resolve, reject) => { + signal.addEventListener('abort', () => + reject( + new LlmRequestPreparationError('park cancellation before terminal'), + ), + ); + modelWaiting = true; + }); + }); + await f.submit(10_000); + await vi.waitFor(() => expect(modelWaiting).toBe(true), { + timeout: 10_000, + }); + await f + .authorize(supertest(f.server).post(`/session/${SESSION_ID}/cancel`)) + .expect(204); + await f.idle(); + const events = () => + f.resident.managed.authority.eventsInSequenceRange( + 1, + f.resident.managed.authority.committedSequence, + ); + const deadline = events().find((event) => event.kind === 'input.accepted')! + .payload['deadline'] as number; + const cancellation = events().find( + (event) => event.kind === 'cancel.requested', + )!; + expect(cancellation.occurredAt).toBeLessThan(deadline); + const now = Date.now.bind(Date); + const elapsedStart = now(); + const clock = vi + .spyOn(Date, 'now') + .mockImplementation(() => deadline + 1 + now() - elapsedStart); + try { + state.model.mockImplementationOnce(async ({ signal }) => { + signal.throwIfAborted(); + return { text: 'unexpected', model: 'fixture' }; + }); + await headers(supertest(f.server).post(`/session/${SESSION_ID}/load`)) + .send({ + managedSessionStore: store(), + toolProfile: 'hosted-workspace-files/1', + driveRuntimeRecovery: true, + }) + .expect(200); + await vi.waitFor( + () => + expect( + events().filter((event) => event.kind === 'turn.settled'), + ).toHaveLength(1), + { timeout: 10_000 }, + ); + expect( + events().find((event) => event.kind === 'turn.settled')!.payload, + ).toMatchObject({ outcome: 'cancelled', stopReason: 'cancelled' }); + } finally { + clock.mockRestore(); + } + await f.idle(); + await f + .authorize(supertest(f.server).delete(`/session/${SESSION_ID}`)) + .expect(204); + }); + + it('keeps resident recovery and concurrent close consistent (R3-8)', async () => { + const f = await recoveryFileSession(); + const publish = LocalManagedSessionResourceStore.prototype.publish; + const refuse = vi + .spyOn(LocalManagedSessionResourceStore.prototype, 'publish') + .mockImplementation(function ( + this: LocalManagedSessionResourceStore, + kind, + bytes, + ) { + return kind === 'managed-turn-result' + ? Promise.reject(new Error('terminal unavailable')) + : publish.call(this, kind, bytes); + }); + state.model.mockImplementationOnce(async ({ modelScope }) => { + const finish = await modelScope!.beginMainAttempt('fixture'); + await finish(true, {}); + return { text: 'committed answer', model: 'fixture' }; + }); + await f.submit(); + await f.idle(); + refuse.mockRestore(); + const find = modelRecovery.findHostedCommittedModelOutput; + let release!: () => void; + let reading = false; + const gate = new Promise((resolve) => (release = resolve)); + vi.spyOn( + modelRecovery, + 'findHostedCommittedModelOutput', + ).mockImplementationOnce(async (...args) => { + const committed = await find(...args); + expect(committed).toBeDefined(); + reading = true; + await gate; + return committed; + }); + const loading = headers( + supertest(f.server).post(`/session/${SESSION_ID}/load`), + ) + .send({ + managedSessionStore: store(), + toolProfile: 'hosted-workspace-files/1', + driveRuntimeRecovery: true, + }) + .then((response) => response); + try { + await vi.waitFor(() => expect(reading).toBe(true), { timeout: 10_000 }); + f.resident.mcpClosing = true; + const closingRefusal = await headers( + supertest(f.server).post(`/session/${SESSION_ID}/load`), + ).send({ + managedSessionStore: store(), + toolProfile: 'hosted-workspace-files/1', + driveRuntimeRecovery: true, + }); + expect(closingRefusal.status).toBe(409); + expect(closingRefusal.body.code).toBe('hosted_session_closing'); + f.resident.mcpClosing = false; + await headers(supertest(f.server).post(`/session/${SESSION_ID}/load`)) + .send({ + managedSessionStore: store(), + toolProfile: 'hosted-workspace-files/1', + driveRuntimeRecovery: true, + }) + .expect(409); + const closed = await f.authorize( + supertest(f.server).delete(`/session/${SESSION_ID}`), + ); + release(); + const loaded = await loading; + if (closed.status === 204) { + expect([404, 409]).toContain(loaded.status); + await f + .authorize(supertest(f.server).get(`/session/${SESSION_ID}/status`)) + .expect(404); + } else { + expect(closed.status).toBe(409); + expect(closed.body.code).toBe('hosted_turn_active'); + expect(loaded.status).toBe(200); + await f.status(); + await f.idle(); + await f + .authorize(supertest(f.server).delete(`/session/${SESSION_ID}`)) + .expect(204); + } + } finally { + f.resident.mcpClosing = false; + release(); + await loading; + } + }); + + it.each(['ordinary', 'wake'])( + 'retries completed native-file cleanup on the real scheduler cadence (%s)', + async (source) => { + const f = await recoveryFileSession(); + vi.mocked(HostedWorkspaceBroker.prototype.warm).mockImplementation( + async function (this: HostedWorkspaceBroker) { + this.runtime = { + bindingId: 'binding', + generation: '1', + workspaceGeneration: '1', + }; + }, + ); + const prepare = vi + .spyOn(HostedWorkspaceBroker.prototype, 'prepare') + .mockImplementation(async () => randomUUID()); + const execute = vi + .spyOn(HostedWorkspaceBroker.prototype, 'execute') + .mockResolvedValue({ + executionStatus: 'success', + responseParts: [{ text: 'written' }], + }); + let unavailable = true; + const releases: Array<{ + owner: string; + options: unknown; + failed: boolean; + at: number; + }> = []; + vi.mocked(HostedWorkspaceBroker.prototype.release).mockImplementation( + async function (this: HostedWorkspaceBroker, options) { + releases.push({ + owner: this.runtimeSessionId, + options, + failed: unavailable, + at: Date.now(), + }); + if (unavailable) + throw new HostedWorkspaceBrokerRejection( + 503, + 'runtime_unavailable', + ); + }, + ); + state.model.mockImplementationOnce(async ({ toolTurn, signal }) => { + const call = { + name: 'write_file', + callId: 'call-1', + args: { file_path: 'a.txt', content: 'x' }, + isClientInitiated: false, + prompt_id: PROMPT_ID, + }; + await toolTurn!.execute( + [call], + [ + { + functionCall: { + id: call.callId, + name: call.name, + args: call.args, + }, + }, + ], + 'test-model', + signal, + ); + return { text: 'done', model: 'test-model' }; + }); + try { + if (source === 'ordinary') await f.submit(); + else { + const managed = f.resident.managed; + const content = Buffer.from(JSON.stringify({ text: 'write a.txt' })); + await managed.authority.submitInput( + { + operation: 'submitInput', + commandId: PROMPT_ID, + sessionKey: managed.authority.sessionHeader.sessionKey, + contentDigest: createHash('sha256').update(content).digest('hex'), + }, + { + inputId: PROMPT_ID, + turnId: PROMPT_ID, + source: 'monitor', + contentRef: await managed.resources.publish( + 'managed-input', + content, + ), + admissionRef: await managed.resources.publish( + 'managed-admission', + Buffer.from('{}'), + ), + deadline: null, + wakeReason: 'input', + }, + ); + ( + f.resident as unknown as { monitorWake: HostedMonitorWakeScheduler } + ).monitorWake.kick(); + } + await vi.waitFor( + () => { + expect(f.resident.cleanupBlocked).toBe(true); + expect(releases.length).toBeGreaterThan(0); + }, + { timeout: 10_000 }, + ); + await vi.waitFor( + () => expect(releases.filter((item) => item.failed)).toHaveLength(4), + { timeout: 10_000 }, + ); + expect(releases[3].at - releases[2].at).toBeGreaterThanOrEqual(400); + expect(releases).toHaveLength(4); + expect(f.resident.blocked).toBe(true); + unavailable = false; + await vi.waitFor( + () => { + expect(f.resident.blocked).toBe(false); + expect(f.resident.cleanupBlocked).toBe(false); + }, + { timeout: 10_000 }, + ); + expect(releases.filter((item) => !item.failed)).toHaveLength(1); + expect( + releases.every( + (item) => item.owner === hostedRuntimeSessionId(PROMPT_ID), + ), + ).toBe(true); + expect(releases.at(-1)!.options).toEqual({ + bindingId: 'binding', + generation: '1', + }); + expect(state.model).toHaveBeenCalledOnce(); + expect(prepare).toHaveBeenCalledOnce(); + expect(execute).toHaveBeenCalledOnce(); + const events = f.resident.managed.authority.eventsInSequenceRange( + 1, + f.resident.managed.authority.committedSequence, + ); + expect( + events.filter( + (event) => + event.kind === 'turn.settled' && + event.payload['turnId'] === PROMPT_ID, + ), + ).toHaveLength(1); + expect( + events.filter((event) => event.kind === 'hosted.cleanup').at(-1)! + .payload['state'], + ).toBe('confirmed'); + } finally { + unavailable = false; + await f + .authorize(supertest(f.server).delete(`/session/${SESSION_ID}`)) + .expect(204); + } + }, + ); + + it('retries consumed channel cleanup while keeping new turns blocked', async () => { + const previous = 'previous-prompt'; + const channelTurn = await prewriteChannelWriteInterruption(false, previous); + mockBrokerBroker(); + vi.spyOn(HostedMonitorWakeScheduler.prototype, 'kick').mockImplementation( + () => {}, + ); + vi.spyOn(HostedWorkspaceBroker.prototype, 'status').mockResolvedValue({ + state: 'settled', + }); + const owners: string[] = []; + let faults = 0; + vi.mocked(HostedWorkspaceBroker.prototype.release).mockImplementation( + async function (this: HostedWorkspaceBroker) { + owners.push(this.runtimeSessionId); + if (this.runtimeSessionId === channelTurn) + throw new HostedWorkspaceBrokerRejection( + 404, + 'runtime_session_not_found', + ); + if (this.runtimeSessionId === previous && faults-- > 0) + throw new HostedWorkspaceBrokerRejection(503, 'runtime_unavailable'); + }, + ); + const server = await app(true); + const loaded = await headers( + supertest(server).post(`/session/${SESSION_ID}/load`), + ) + .send({ + managedSessionStore: store(), + toolProfile: 'hosted-workspace-files/1', + }) + .expect(200); + const resident = capturedWakeSession(); + const broker = new HostedWorkspaceBroker( + { baseUrl: 'http://127.0.0.1:1', token: 'test' }, + resident.managed.authority.sessionHeader.sessionKey, + previous, + ); + broker.runtime = { + bindingId: 'binding', + generation: '1', + workspaceGeneration: '1', + }; + await oweHostedTurnCleanup(resident.managed, previous, broker); + faults = 2; + const params = wakeDeps.last as Parameters< + typeof createMonitorWakeRunTurn + >[0]; + expect( + await createMonitorWakeRunTurn(params)({ + turnId: channelTurn, + text: 'write deploy.txt', + source: 'channel', + }), + ).toBe('busy'); + const scheduler = ( + resident as unknown as { + monitorWake: { + deps: { + next: () => Promise; + recoverBlocked: () => Promise; + }; + }; + } + ).monitorWake; + expect(await scheduler.deps.next()).toBeUndefined(); + expect(resident.blocked).toBe(true); + expect(resident.cleanupBlocked).toBe(true); + await scheduler.deps.recoverBlocked(); + expect(owners.filter((id) => id === previous)).toHaveLength(2); + expect(resident.blocked).toBe(true); + await scheduler.deps.recoverBlocked(); + expect(owners.filter((id) => id === previous)).toHaveLength(3); + expect(resident.blocked).toBe(false); + expect(resident.cleanupBlocked).toBe(false); + expect(state.model).not.toHaveBeenCalled(); + const events = resident.managed.authority.eventsInSequenceRange( + 1, + resident.managed.authority.committedSequence, + ); + expect( + events.filter( + (event) => + event.kind === 'turn.settled' && + event.payload['turnId'] === channelTurn, + ), + ).toHaveLength(1); + expect( + events.filter((event) => event.kind === 'hosted.cleanup').at(-1)!.payload[ + 'state' + ], + ).toBe('confirmed'); + await headers(supertest(server).delete(`/session/${SESSION_ID}`)) + .set('X-Qwen-Client-Id', loaded.body.clientId) + .expect(204); + }); + + it('settles a channel wake whose missing Runtime has another prompt cleanup owner (R2-63)', async () => { + const previous = 'previous-prompt'; + const channelTurn = await prewriteChannelWriteInterruption(false, previous); + mockBrokerBroker(); + vi.spyOn(HostedMonitorWakeScheduler.prototype, 'kick').mockImplementation( + () => {}, + ); + vi.spyOn(HostedWorkspaceBroker.prototype, 'status').mockResolvedValue({ + state: 'settled', + }); + const release = vi.mocked(HostedWorkspaceBroker.prototype.release); + release.mockImplementation(async function (this: HostedWorkspaceBroker) { + if (this.runtimeSessionId === channelTurn) + throw new HostedWorkspaceBrokerRejection( + 404, + 'runtime_session_not_found', + ); + }); + const server = await app(true); + const loaded = await headers( + supertest(server).post(`/session/${SESSION_ID}/load`), + ) + .send({ + managedSessionStore: store(), + toolProfile: 'hosted-workspace-files/1', + }) + .expect(200); + const resident = capturedWakeSession(); + const broker = new HostedWorkspaceBroker( + { baseUrl: 'http://127.0.0.1:1', token: 'test' }, + resident.managed.authority.sessionHeader.sessionKey, + previous, + ); + broker.runtime = { + bindingId: 'binding', + generation: '1', + workspaceGeneration: '1', + }; + await oweHostedTurnCleanup(resident.managed, previous, broker); + const params = wakeDeps.last as Parameters< + typeof createMonitorWakeRunTurn + >[0]; + const runTurn = createMonitorWakeRunTurn(params); + await expect( + runTurn({ + turnId: channelTurn, + text: 'write deploy.txt', + source: 'channel', + }), + ).resolves.toBe('settled'); + expect(resident.blocked).toBe(false); + expect(state.model).not.toHaveBeenCalled(); + const events = resident.managed.authority.eventsInSequenceRange( + 1, + resident.managed.authority.committedSequence, + ); + expect( + events.filter( + (event) => + event.kind === 'turn.settled' && + event.payload['turnId'] === channelTurn, + ), + ).toHaveLength(1); + expect( + events.filter((event) => event.kind === 'hosted.cleanup').at(-1)!.payload[ + 'state' + ], + ).toBe('confirmed'); + await headers(supertest(server).delete(`/session/${SESSION_ID}`)) + .set('X-Qwen-Client-Id', loaded.body.clientId) + .expect(204); + }); }); describe('Hosted Harness tool approvals', () => { @@ -16912,7 +18434,7 @@ describe('Hosted Harness Runtime turn takeover', () => { }); }); - function replacementApp() { + async function replacementApp() { const result = express(); result.use(express.json()); const contract = createHostedHarnessContract( @@ -16924,7 +18446,12 @@ describe('Hosted Harness Runtime turn takeover', () => { baseUrl: 'http://127.0.0.1:1', token: 'test', }); - return result; + const listener = createServer(result); + listeners.add(listener); + await new Promise((resolve) => + listener.listen(0, '127.0.0.1', resolve), + ); + return listener; } function replacementHeaders(request: T): T { @@ -16955,7 +18482,14 @@ describe('Hosted Harness Runtime turn takeover', () => { vi.spyOn(HostedWorkspaceBroker.prototype, 'warm').mockResolvedValue(); acquireSpy = vi .spyOn(HostedWorkspaceBroker.prototype, 'acquire') - .mockResolvedValue(); + .mockImplementation(async function (this: HostedWorkspaceBroker) { + if (toolProfile === 'hosted-workspace-mcp/1') + this.runtime = { + bindingId: 'binding', + generation: '1', + workspaceGeneration: '1', + }; + }); vi.spyOn(HostedWorkspaceBroker.prototype, 'prepare').mockResolvedValue( '66666666-6666-4666-8666-666666666666', ); @@ -16980,6 +18514,17 @@ describe('Hosted Harness Runtime turn takeover', () => { sessionScope: 'thread', managedSessionStore: store(), toolProfile, + ...(toolProfile === 'hosted-workspace-mcp/1' + ? { + mcpServers: [ + { + serverId: 'demo', + serverRevision: 1, + definitionDigest: 'a'.repeat(64), + }, + ], + } + : {}), }); expect(created.status).toBe(200); state.model.mockImplementationOnce( @@ -17037,7 +18582,7 @@ describe('Hosted Harness Runtime turn takeover', () => { } async function loadReplacement(passive = false, toolProfile = FILE_PROFILE) { - const server = replacementApp(); + const server = await replacementApp(); const loaded = await replacementHeaders( supertest(server).post(`/session/${SESSION_ID}/load`), ).send({ @@ -17050,6 +18595,220 @@ describe('Hosted Harness Runtime turn takeover', () => { return { server, loaded }; } + it.each(['cancel', 'close', 'continue-refusal'])( + 'preserves the actual passive MCP owner through %s', + async (path) => { + const mcpServers = [ + { + serverId: 'demo', + serverRevision: 1, + definitionDigest: 'a'.repeat(64), + }, + ]; + vi.spyOn(HostedMonitorWakeScheduler.prototype, 'kick').mockImplementation( + () => {}, + ); + const replies = new Map(); + vi.spyOn(HostedWorkspaceBroker.prototype, 'control').mockImplementation( + async (operation) => { + if (operation.kind === 'mcp-configure') { + const result: ManagedMcpOperationView = { + operationId: operation.operationId, + state: 'settled', + catalog: { + serverId: operation.serverId, + serverRevision: operation.serverRevision, + definitionDigest: operation.definitionDigest, + configRevision: operation.configRevision, + connectionGeneration: operation.configRevision, + catalogRevision: operation.configRevision, + tools: [], + resources: [], + prompts: [], + discovery: { + tools: 'complete', + resources: 'complete', + prompts: 'complete', + }, + }, + }; + replies.set(operation.operationId, result); + return result; + } + if (operation.kind === 'mcp-discover') + return { + ...replies.get(operation.grant.operationId)!, + operationId: operation.operationId, + }; + if ( + operation.kind === 'mcp-status' || + operation.kind === 'mcp-cancel' + ) + return ( + replies.get(operation.targetOperationId) ?? { + operationId: operation.targetOperationId, + state: 'outcome_unknown', + } + ); + return { operationId: operation.operationId, state: 'settled' }; + }, + ); + await parkToolTurn(true, 'hosted-workspace-mcp/1'); + const originalResident = ( + wakeDeps.last as Parameters[0] + ).session; + await ( + originalResident.managed as Awaited< + ReturnType + > + ).close(); + let stopped = false; + vi.spyOn(HostedWorkspaceBroker.prototype, 'status').mockImplementation( + async () => ({ state: stopped ? 'settled' : 'prepared' }), + ); + vi.mocked(HostedWorkspaceBroker.prototype.cancel).mockImplementation( + async () => { + stopped = true; + }, + ); + const releases: string[] = []; + const releaseArgs: Array<{ owner: string; options: unknown }> = []; + const acquisitions: string[] = []; + const retired = new Set(); + acquireSpy.mockImplementation(async function ( + this: HostedWorkspaceBroker, + ) { + if (retired.has(this.runtimeSessionId)) + throw new HostedWorkspaceBrokerRejection( + 409, + 'runtime_session_not_acquirable', + ); + acquisitions.push(this.runtimeSessionId); + this.runtime = { + bindingId: 'binding', + generation: '1', + workspaceGeneration: '1', + }; + }); + vi.mocked(HostedWorkspaceBroker.prototype.release).mockImplementation( + async function (this: HostedWorkspaceBroker, options) { + releases.push(this.runtimeSessionId); + retired.add(this.runtimeSessionId); + releaseArgs.push({ + owner: this.runtimeSessionId, + options: options ?? null, + }); + if (this.runtimeSessionId === PROMPT_ID) + throw new HostedWorkspaceBrokerRejection( + 404, + 'runtime_session_not_found', + ); + }, + ); + const server = await replacementApp(); + const loaded = await replacementHeaders( + supertest(server).post(`/session/${SESSION_ID}/load`), + ).send({ + managedSessionStore: storeFor(BOOT_ID_2), + toolProfile: 'hosted-workspace-mcp/1', + mcpServers, + passiveManagedRuntimeRecovery: true, + }); + expect(loaded.status).toBe(200); + const recovery = loaded.body._meta['qwen.daemon.managedRuntimeRecovery']; + expect(recovery.phase).toBe('await_runtime'); + const owner = recovery.executions[0].runtimeSessionId; + expect(owner.startsWith('mcp-')).toBe(true); + if (path === 'close') { + const closed = await replacementHeaders( + supertest(server).delete(`/session/${SESSION_ID}`), + ).set('X-Qwen-Client-Id', loaded.body.clientId); + expect(closed.status).toBe(204); + expect(releases).toContain(owner); + expect(releases).not.toContain(PROMPT_ID); + expect(releaseArgs[0].options).toBeNull(); + return; + } + if (path === 'continue-refusal') { + const refused = await replacementHeaders( + supertest(server).post( + `/session/${SESSION_ID}/managed-runtime/continue`, + ), + ) + .set('X-Qwen-Client-Id', loaded.body.clientId) + .send({ + promptId: PROMPT_ID, + checkpointId: recovery.checkpointId, + activationId: recovery.activationId, + }); + expect(refused.status).toBe(409); + expect(refused.body.code).toBe('hosted_turn_recovery_required'); + expect(refused.body.reason).toBe('continuation_not_ready'); + expect(releases).toEqual([]); + } + const cancelled = await replacementHeaders( + supertest(server).post(`/session/${SESSION_ID}/managed-runtime/cancel`), + ) + .set('X-Qwen-Client-Id', loaded.body.clientId) + .send({ + promptId: PROMPT_ID, + checkpointId: recovery.checkpointId, + activationId: recovery.activationId, + }); + await vi.waitFor(async () => { + const status = await replacementHeaders( + supertest(server).get(`/session/${SESSION_ID}/status`), + ).set('X-Qwen-Client-Id', loaded.body.clientId); + expect(status.body.hasActivePrompt).toBe(false); + }); + const terminalReleases = [...releases]; + const terminalArgs = [...releaseArgs]; + const closed = await replacementHeaders( + supertest(server).delete(`/session/${SESSION_ID}`), + ).set('X-Qwen-Client-Id', loaded.body.clientId); + expect(cancelled.status).toBe(200); + expect(closed.status).toBe(204); + expect(terminalReleases).toEqual([]); + expect(releases).toContain(owner); + expect(terminalReleases).not.toContain(PROMPT_ID); + expect( + terminalArgs.every( + (entry) => + entry.options === null || + !(entry.options as { drain?: boolean }).drain, + ), + ).toBe(true); + expect(releaseArgs.at(-1)?.options).toBeNull(); + }, + ); + + it('closes an adopted Session after a transient handback transport fault (R2-1)', async () => { + await parkToolTurn(); + vi.spyOn(HostedWorkspaceBroker.prototype, 'status').mockResolvedValue({ + state: 'prepared', + }); + const { server, loaded } = await loadReplacement(true); + expect(loaded.status).toBe(200); + const release = vi.mocked(HostedWorkspaceBroker.prototype.release); + const releaseActivation = vi.spyOn( + LocalManagedSessionAuthority.prototype, + 'releaseActivation', + ); + release.mockRejectedValueOnce( + new TypeError('Broker handback transport fault'), + ); + await replacementHeaders( + supertest(server).delete(`/session/${SESSION_ID}`), + ).expect(204); + expect(release).toHaveBeenCalledOnce(); + expect(releaseActivation).toHaveBeenCalledOnce(); + await replacementHeaders( + supertest(server).get(`/session/${SESSION_ID}/status`), + ) + .set('X-Qwen-Client-Id', loaded.body.clientId) + .expect(404); + }); + /** * Drives a Workspace turn to the parked agent wait through the * PRODUCTION admission itself (R1-8): the model launches a foreground @@ -17834,7 +19593,7 @@ describe('Hosted Harness Runtime turn takeover', () => { const log = vi .spyOn(stdio, 'writeStderrLineSafe') .mockImplementation(() => {}); - const server = replacementApp(); + const server = await replacementApp(); const loaded = await replacementHeaders( supertest(server).post(`/session/${SESSION_ID}/load`), ).send({ @@ -17864,7 +19623,7 @@ describe('Hosted Harness Runtime turn takeover', () => { const log = vi .spyOn(stdio, 'writeStderrLineSafe') .mockImplementation(() => {}); - const server = replacementApp(); + const server = await replacementApp(); const loaded = await replacementHeaders( supertest(server).post(`/session/${SESSION_ID}/load`), ).send({ @@ -17898,7 +19657,7 @@ describe('Hosted Harness Runtime turn takeover', () => { const log = vi .spyOn(stdio, 'writeStderrLineSafe') .mockImplementation(() => {}); - const server = replacementApp(); + const server = await replacementApp(); const loaded = await replacementHeaders( supertest(server).post(`/session/${SESSION_ID}/load`), ).send({ @@ -17932,7 +19691,7 @@ describe('Hosted Harness Runtime turn takeover', () => { const log = vi .spyOn(stdio, 'writeStderrLineSafe') .mockImplementation(() => {}); - const server = replacementApp(); + const server = await replacementApp(); const loaded = await replacementHeaders( supertest(server).post(`/session/${SESSION_ID}/load`), ).send({ @@ -17973,7 +19732,7 @@ describe('Hosted Harness Runtime turn takeover', () => { const log = vi .spyOn(stdio, 'writeStderrLineSafe') .mockImplementation(() => {}); - const server = replacementApp(); + const server = await replacementApp(); const loaded = await replacementHeaders( supertest(server).post(`/session/${SESSION_ID}/load`), ).send({ @@ -18024,7 +19783,7 @@ describe('Hosted Harness Runtime turn takeover', () => { const log = vi .spyOn(stdio, 'writeStderrLineSafe') .mockImplementation(() => {}); - const server = replacementApp(); + const server = await replacementApp(); const loaded = await replacementHeaders( supertest(server).post(`/session/${SESSION_ID}/load`), ).send({ @@ -18723,11 +20482,16 @@ describe('Hosted Harness Runtime turn takeover', () => { const { server, loaded } = await loadReplacement(); expect(loaded.status).toBe(200); release.mockClear(); + acquireSpy.mockClear(); + let acquireEntered = false; let finishAcquire!: () => void; const acquireGate = new Promise((resolve) => { finishAcquire = resolve; }); - acquireSpy.mockImplementationOnce(() => acquireGate as never); + acquireSpy.mockImplementationOnce(() => { + acquireEntered = true; + return acquireGate as never; + }); let redriving: Promise | undefined; try { redriving = replacementHeaders( @@ -18739,11 +20503,14 @@ describe('Hosted Harness Runtime turn takeover', () => { driveRuntimeRecovery: true, }) .then((response) => response); - await vi.waitFor(() => expect(acquireSpy).toHaveBeenCalledOnce(), { + await vi.waitFor(() => expect(acquireEntered).toBe(true), { timeout: 10_000, }); - // The close route fences an active Turn, MCP work and Hooks — not a - // parked redrive whose await has not recorded its lease yet. + expect(acquireSpy).toHaveBeenCalledOnce(); + expect( + (acquireSpy.mock.contexts[0] as HostedWorkspaceBroker).runtimeSessionId, + ).toBe(PROMPT_ID); + // This fixture has no G3 request/approval snapshot and reaches the G1 acquire window. await replacementHeaders( supertest(server).delete(`/session/${SESSION_ID}`), ).expect(204); @@ -18755,6 +20522,9 @@ describe('Hosted Harness Runtime turn takeover', () => { // adoption is named instead of released, because a release would // persist RELEASED and wedge every retried acquire of the identity. expect(release).toHaveBeenCalledOnce(); + expect( + (release.mock.contexts[0] as HostedWorkspaceBroker).runtimeSessionId, + ).toBe(PROMPT_ID); expect(owedLines()).toHaveLength(1); } finally { finishAcquire(); @@ -19004,7 +20774,7 @@ describe('Hosted Harness Runtime turn takeover', () => { return Buffer.from(JSON.stringify(definition)); }); const drive = await replacementHeaders( - supertest(replacementApp()).post(`/session/${SESSION_ID}/load`), + supertest(await replacementApp()).post(`/session/${SESSION_ID}/load`), ).send({ managedSessionStore: storeFor(BOOT_ID_2), driveRuntimeRecovery: true, @@ -19016,7 +20786,7 @@ describe('Hosted Harness Runtime turn takeover', () => { // The decline closed its authority, releasing the writer latch for // the cancellation arm's load. const passive = await replacementHeaders( - supertest(replacementApp()).post(`/session/${SESSION_ID}/load`), + supertest(await replacementApp()).post(`/session/${SESSION_ID}/load`), ).send({ managedSessionStore: storeFor(BOOT_ID_2), passiveManagedRuntimeRecovery: true, @@ -19618,7 +21388,7 @@ describe('Hosted Harness Runtime turn takeover', () => { .mockImplementation(() => {}); for (const passive of [false, true]) { const refused = await replacementHeaders( - supertest(replacementApp()).post(`/session/${SESSION_ID}/load`), + supertest(await replacementApp()).post(`/session/${SESSION_ID}/load`), ).send({ managedSessionStore: storeFor(BOOT_ID_2), toolProfile: FILE_PROFILE, @@ -19740,7 +21510,7 @@ describe('Hosted Harness Runtime turn takeover', () => { .spyOn(stdio, 'writeStderrLineSafe') .mockImplementation(() => {}); const refused = await replacementHeaders( - supertest(replacementApp()).post(`/session/${SESSION_ID}/load`), + supertest(await replacementApp()).post(`/session/${SESSION_ID}/load`), ).send({ managedSessionStore: storeFor(BOOT_ID_2), toolProfile: FILE_PROFILE, @@ -19849,14 +21619,14 @@ describe('Hosted Harness Runtime turn takeover', () => { // retriable refusal first: nothing may mint a canned CANCELLED record // for a wait whose owner could still exist (the collision guard). const parkedRefused = await replacementHeaders( - supertest(replacementApp()).post(`/session/${SESSION_ID}/load`), + supertest(await replacementApp()).post(`/session/${SESSION_ID}/load`), ).send({ managedSessionStore: storeFor(BOOT_ID_2), passiveManagedRuntimeRecovery: true, }); expect(parkedRefused.status).toBe(409); expect(parkedRefused.body.code).toBe('hosted_turn_recovery_required'); - const replacement = replacementApp(); + const replacement = await replacementApp(); const taken = await replacementHeaders( supertest(replacement).post(`/session/${SESSION_ID}/load`), ).send({ @@ -19975,7 +21745,7 @@ describe('Hosted Harness Runtime turn takeover', () => { .spyOn(stdio, 'writeStderrLineSafe') .mockImplementation(() => {}); const taken = await replacementHeaders( - supertest(replacementApp()).post(`/session/${SESSION_ID}/load`), + supertest(await replacementApp()).post(`/session/${SESSION_ID}/load`), ).send({ managedSessionStore: storeFor(BOOT_ID_2), passiveManagedRuntimeRecovery: true, @@ -20025,7 +21795,7 @@ describe('Hosted Harness Runtime turn takeover', () => { 'action', ).mockReturnValue({ state: durableState } as never); const parkedRefused = await replacementHeaders( - supertest(replacementApp()).post(`/session/${SESSION_ID}/load`), + supertest(await replacementApp()).post(`/session/${SESSION_ID}/load`), ).send({ managedSessionStore: storeFor(BOOT_ID_2), toolProfile: FILE_PROFILE, @@ -20034,7 +21804,7 @@ describe('Hosted Harness Runtime turn takeover', () => { expect(parkedRefused.status).toBe(409); expect(parkedRefused.body.code).toBe('hosted_turn_recovery_required'); const taken = await replacementHeaders( - supertest(replacementApp()).post(`/session/${SESSION_ID}/load`), + supertest(await replacementApp()).post(`/session/${SESSION_ID}/load`), ).send({ managedSessionStore: storeFor(BOOT_ID_2), toolProfile: FILE_PROFILE, @@ -21602,7 +23372,7 @@ describe('Hosted Harness Runtime turn takeover', () => { it('refuses a takeover on a Session without any tool profile', async () => { await parkBlockedTextTurn(false); - const server = replacementApp(); + const server = await replacementApp(); // The drive shape is G3's typed terminal decline; the retriable refusal // with its ground survives on the cancellation-only shape. const loaded = await replacementHeaders( @@ -21621,7 +23391,7 @@ describe('Hosted Harness Runtime turn takeover', () => { it('refuses a takeover whose parked turn is not a Runtime wait', async () => { await parkBlockedTextTurn(true); - const server = replacementApp(); + const server = await replacementApp(); const loaded = await replacementHeaders( supertest(server).post(`/session/${SESSION_ID}/load`), ).send({ @@ -22057,7 +23827,7 @@ describe('Hosted Harness Runtime turn takeover', () => { .status, ).toBe(204); // The coordinator's cancellation takeover on a replacement Harness. - const replacement = replacementApp(); + const replacement = await replacementApp(); const loaded = await replacementHeaders( supertest(replacement).post(`/session/${SESSION_ID}/load`), ).send({ diff --git a/packages/cli/src/serve/hosted-harness-session.ts b/packages/cli/src/serve/hosted-harness-session.ts index 076230f4ef..7f6204fb00 100644 --- a/packages/cli/src/serve/hosted-harness-session.ts +++ b/packages/cli/src/serve/hosted-harness-session.ts @@ -4,6 +4,35 @@ * SPDX-License-Identifier: Apache-2.0 */ +import { + findHostedModelRequest, + findHostedCommittedModelOutput, + createHostedModelRequest, + publishHostedModelRequest, + type HostedModelRequest, +} from './hosted-model-recovery.js'; +import { LlmRequestPreparationError } from '@qwen-code/qwen-code-core/core/llm-chat.js'; +import { + findHostedApprovalContinuation, + type RestoredHostedApprovalContinuation, +} from './hosted-approval-continuation.js'; +import { + reconcileHostedTurnCleanup, + hasHostedCleanupDebt, +} from './hosted-turn-cleanup.js'; +import { + readHostedActionOptions, + hostedActionAllowed, + hostedActionDenied, + endHostedAction, + parseHostedActionOptions, +} from './hosted-tool-approval.js'; + +import { + HOSTED_RECOVERY_RESOURCE_KINDS, + parseHostedRecoveryResource, + hostedRecoveryReferences, +} from '@qwen-code/qwen-code-core/managed-runtime/hosted-recovery-records.js'; import { readHostedFileHistory, commitHostedFileHistory, @@ -233,6 +262,10 @@ const RESTORE_READ_BATCH = 32; * what they reference is verified too; the rest are not read again. */ const RESTORE_CONTAINER_KINDS = new Set([ + 'managed-action-options', + 'hosted-approval-continuation', + 'hosted-model-request', + 'hosted-turn-cleanup', 'managed-definition', 'managed-session_metadata', 'managed-file_history', @@ -276,6 +309,7 @@ function settledTurnOutcome(abort: AbortController): { } interface HostedSession { + streamOwner: { bootId: string; eventEpoch: string }; managed: ManagedSession; storeBaseUrl: string; definition: Record | null; @@ -336,6 +370,7 @@ interface HostedSession { storeDescriptor: ReturnType; lifecycle?: ManagedSessionLifecycleAuthority; lifecycleKind?: 'close' | 'delete'; + cleanupBlocked?: boolean; /** Fetched Workspace instructions; undefined until the first fetch. */ workspaceContext?: string; monitorWake?: HostedMonitorWakeScheduler; @@ -352,7 +387,7 @@ interface HostedSession { * READY lease is re-admitted against the current checkpoint or * re-acquired idempotently. The continuation route keeps its #13083 * handback discipline (a recorded follow-up). */ - runtimeLeaseHeld?: string; + runtimeLeaseHeld?: { promptId: string; runtimeSessionId: string }; /** #13856: promptIds of turns whose Runtime Session ended held by a * live background run (the designed hold — the broker refused the * routine release with `runtime_session_held` and stopped nothing). @@ -1639,6 +1674,7 @@ export async function settleCancelledHookTurn( ), ); session.blocked = false; + session.cleanupBlocked = false; } finally { session.hooksBusy = false; } @@ -1710,14 +1746,24 @@ function runSettleProjection( sessionId: string, promptId: string, brokerOptions: HostedWorkspaceBrokerOptions | undefined, + committedModelOutput = false, ): void { const abort = new AbortController(); session.active = { promptId, digest: '', abort }; void (async () => { const harness = createManagedHarnessHandle(session.managed); + if (committedModelOutput) + await harness.adoptPreparedContinuation(promptId, true); await harness.run(async () => { await harness.settleConsumedRuntimeContinuation(); - if (!session.hooks && !session.mcp) + if ( + brokerOptions && + session.toolProfile && + !committedModelOutput && + !session.hooks && + !session.mcp && + !hasHostedCleanupDebt(session.managed) + ) await new HostedWorkspaceBroker( brokerOptions!, session.managed.authority.sessionHeader.sessionKey, @@ -1734,7 +1780,10 @@ function runSettleProjection( }, }), ); + if (brokerOptions) await retryHostedCleanup(session, brokerOptions); }); + session.blocked = false; + session.cleanupBlocked = false; })() .catch((cause: unknown) => { session.blocked = true; @@ -1748,6 +1797,26 @@ function runSettleProjection( }); } +async function retryHostedCleanup( + session: HostedSession, + options: HostedWorkspaceBrokerOptions, +): Promise { + if (unsettledPromptId(session)) return; + try { + if (hasHostedCleanupDebt(session.managed)) + await reconcileHostedTurnCleanup(session.managed, options); + if (session.cleanupBlocked) session.blocked = false; + session.cleanupBlocked = false; + } catch (cause) { + if (!session.blocked) { + session.cleanupBlocked = true; + session.blocked = true; + session.monitorWake?.kick(); + } + throw cause; + } +} + async function readShellReceipt( session: HostedSession, receipt: ManagedSessionEvent, @@ -1870,6 +1939,20 @@ async function verifyWorkspaceRestore( metadata['previousRecordRef'] as unknown as ManagedSessionDurableRef, ); } + if ( + HOSTED_RECOVERY_RESOURCE_KINDS.has(ref.kind) || + ref.kind === 'managed-action-options' + ) { + const body = + ref.kind === 'managed-action-options' + ? parseHostedActionOptions(bytes) + : parseHostedRecoveryResource( + bytes, + authority.sessionHeader.sessionKey, + ); + for (const nested of hostedRecoveryReferences(body)) + await readRef(nested); + } if (ref.kind === 'managed-definition') { const definition = object(JSON.parse(bytes.toString('utf8'))); const pin = parseHostedAgentDefinition(definition?.['agentDefinition']); @@ -2352,6 +2435,13 @@ async function eventEnvelope( turnId: event.payload['turnId'], messageId: event.payload['messageId'], fromSequence: event.payload['fromSequence'], + ...(event.payload['throughSequence'] !== undefined + ? { + throughSequence: event.payload['throughSequence'], + sourceBootId: event.payload['sourceBootId'], + sourceEventEpoch: event.payload['sourceEventEpoch'], + } + : {}), }, }; } @@ -2457,6 +2547,90 @@ async function eventEnvelope( }; } +async function startHostedRecoveredTurn( + session: HostedSession, + sessionId: string, + cwd: string, + resume: { promptId: string; text: string; parts: Part[] }, + brokerOptions: HostedWorkspaceBrokerOptions | undefined, + resumeApproval?: RestoredHostedApprovalContinuation, + resumeModel?: HostedModelRequest, + cancellationTakeover = false, +): Promise { + const abort = new AbortController(); + const accepted = session.managed.authority + .eventsInSequenceRange(1, session.managed.authority.committedSequence) + .find( + (event) => + event.kind === 'input.accepted' && + event.payload['turnId'] === resume.promptId, + ); + const deadline = accepted?.payload['deadline']; + const cancelled = + cancellationTakeover || + session.managed.authority + .eventsInSequenceRange(1, session.managed.authority.committedSequence) + .some( + (event) => + event.kind === 'cancel.requested' && + object(event.payload['target'])?.['turnId'] === resume.promptId, + ); + const timer = + !cancelled && typeof deadline === 'number' + ? setTimeout( + () => abort.abort(HOSTED_TURN_DEADLINE), + Math.max(0, deadline - Date.now()), + ) + : undefined; + timer?.unref(); + if (cancelled) abort.abort(); + else if (typeof deadline === 'number' && deadline <= Date.now()) + abort.abort(HOSTED_TURN_DEADLINE); + session.active = { + promptId: resume.promptId, + digest: '', + abort, + }; + let resolveReady!: () => void; + let rejectReady!: (cause: unknown) => void; + const ready = new Promise((resolve, reject) => { + resolveReady = resolve; + rejectReady = reject; + }); + const resumed = executeHostedTurn( + session, + sessionId, + cwd, + resume.promptId, + resume.text, + abort, + brokerOptions, + resumeApproval || resumeModel ? undefined : resume.parts, + undefined, + resolveReady, + resumeApproval, + resumeModel, + ); + // Do not attach a Session whose original continuation cannot acquire + // Workspace ownership. The caller can retry load without losing it. + void resumed.catch(rejectReady); + void resumed + .catch((cause: unknown) => { + session.blocked = true; + writeStderrLineSafe( + 'qwen serve: Hosted Harness recovery remained blocked: ' + + String(cause), + ); + }) + .finally(() => { + if (timer) clearTimeout(timer); + session.active = undefined; + }); + await ready; + session.blocked = false; + session.cleanupBlocked = false; +} + async function executeHostedTurn( session: HostedSession, sessionId: string, @@ -2468,9 +2642,173 @@ async function executeHostedTurn( resumeFromToolResults?: Part[], onTurnResult?: (result: ChatRecord) => void, onResumeReady?: () => void, + resumeApproval?: RestoredHostedApprovalContinuation, + resumeModel?: HostedModelRequest, ): Promise { const authority = session.managed.authority; const harness = createManagedHarnessHandle(session.managed); + if (resumeModel) { + await harness.adoptPreparedContinuation(promptId, true); + session.workspaceContext = resumeModel.workspaceContext ?? undefined; + await new HostedTextDeltaStream(session.managed, promptId).retractOriginal( + resumeModel.messageId, + resumeModel, + ); + const attempt = authority + .eventsInSequenceRange(1, authority.committedSequence) + .findLast( + (event) => + event.kind === 'model.attempt' && + typeof event.payload['attemptId'] === 'string' && + event.payload['attemptId'].startsWith(`${promptId}:main:`), + ); + if (attempt?.payload['state'] === 'started') { + const activation = session.managed.activation; + const commandId = `${attempt.payload['attemptId']}:abandoned`; + await authority.appendExecutionEvent( + { + operation: 'hostedModelAttempt', + commandId, + sessionKey: authority.sessionHeader.sessionKey, + contentDigest: ( + attempt.payload['routeRef'] as unknown as ManagedSessionDurableRef + ).digest, + }, + (sequence) => ({ + v: 1, + sequence, + eventId: commandId, + sessionKey: authority.sessionHeader.sessionKey, + kind: 'model.attempt', + occurredAt: Date.now(), + subject: { + type: 'activation', + scopeId: activation.activationId, + ...activation, + }, + payload: { ...attempt.payload, state: 'abandoned' }, + }), + { class: 'harness', activation }, + ); + } + } + if (resumeApproval) { + const plan = resumeApproval.plan; + if (plan.stage === 'approval') { + const actionId = plan.actionId!; + const original = authority.action(actionId)!; + const options = await readHostedActionOptions(session.managed, original); + onResumeReady?.(); + await session.waiters.wait( + actionId, + options.expiresAt, + abort.signal, + () => + authority.action(actionId)?.state !== 'requested' || + authority.writesStopped, + ); + if (authority.action(actionId)?.state === 'requested') + await endHostedAction( + session.managed, + actionId, + abort.signal.aborted ? 'cancelled' : 'expired', + ); + const action = authority.action(actionId)!; + await harness.resolveDurableWait(); + if ( + action.state === 'decided' && + !hostedActionAllowed(action, options.policyRevision) && + !hostedActionDenied(action, options.policyRevision) + ) + throw new HostedToolRecoveryRequiredError( + 'Unknown original approval verdict.', + ); + const calls = plan.calls.map((item, index) => + index === plan.approvalOrdinal + ? { + ...item, + refusal: hostedActionAllowed(action, options.policyRevision) + ? null + : 'Original approval denied the call.', + } + : item, + ); + resumeApproval = { ...resumeApproval, plan: { ...plan, calls } }; + } + const expired = plan.calls.some( + (item) => + item.actionId && authority.action(item.actionId)?.state === 'expired', + ); + const cancelled = plan.calls.some( + (item) => + item.actionId && authority.action(item.actionId)?.state === 'cancelled', + ); + if (abort.signal.aborted || expired || cancelled) { + if (!(await parkNeedsNoRuntimeSettlement(session.managed))) + throw new HostedToolRecoveryRequiredError( + 'Original cancellation still owes Runtime outcomes.', + ); + const fileHistory = await readHostedFileHistory(session.managed); + if (fileHistory?.pendingUndo) + throw new HostedToolRecoveryRequiredError( + 'Original file history undo is still owed.', + ); + if (fileHistory?.pendingTurn === promptId) + await commitHostedFileHistory(session.managed, { + ...fileHistory, + pendingTurn: null, + pendingMessageId: undefined, + }); + const answered = new Set( + (await session.managed.sink.project()) + .filter( + (item) => + item.type === 'tool_result' && item.daemonPromptId === promptId, + ) + .flatMap( + (item) => + item.message?.parts?.flatMap((part) => + part.functionResponse?.id ? [part.functionResponse.id] : [], + ) ?? [], + ), + ); + const results = plan.calls.flatMap((item) => + answered.has(item.call.callId) + ? [] + : convertToFunctionErrorResponse( + item.call.name, + item.call.callId, + [], + expired + ? 'Original approval expired.' + : 'Original Turn was cancelled.', + ), + ); + if (results.length) + await session.managed.sink.write( + record(session, sessionId, 'tool_result', plan.batchId, { + daemonPromptId: promptId, + model: plan.model, + message: { role: 'user', parts: results }, + }), + ); + await settleCancelledHarnessTurn( + session.managed, + session, + sessionId, + promptId, + ); + const terminal = (await session.managed.sink.project()).findLast( + (item) => + item.subtype === 'turn_result' && + object(item.systemPayload)?.['promptId'] === promptId, + )!; + if (brokerOptions) await retryHostedCleanup(session, brokerOptions); + onResumeReady?.(); + return terminal; + } + await harness.adoptPreparedContinuation(promptId); + } let turnResult: ChatRecord | undefined; let toolTurn: HostedWorkspaceToolTurn | undefined; const running = new ManagedHookActivationController(session.managed).runTurn( @@ -2490,7 +2828,8 @@ async function executeHostedTurn( ? projected.filter( (entry) => settledPrompts.has(entry.daemonPromptId) || - (resumeFromToolResults && entry.daemonPromptId === promptId), + ((resumeFromToolResults || resumeApproval || resumeModel) && + entry.daemonPromptId === promptId), ) : projected ).filter( @@ -2502,7 +2841,7 @@ async function executeHostedTurn( ), ); let parentUuid = projected.at(-1)?.uuid ?? null; - if (!resumeFromToolResults) { + if (!resumeFromToolResults && !resumeApproval && !resumeModel) { const user = record(session, sessionId, 'user', parentUuid, { daemonPromptId: promptId, message: { role: 'user', parts: [{ text }] }, @@ -2522,9 +2861,7 @@ async function executeHostedTurn( message: { role: type === 'assistant' ? 'model' : 'user', parts }, ...identity, }); - const deltas = session.toolProfile - ? new HostedTextDeltaStream(session.managed, promptId) - : undefined; + const deltas = new HostedTextDeltaStream(session.managed, promptId); const commit = async ( type: 'assistant' | 'tool_result', parts: Part[], @@ -2583,7 +2920,24 @@ async function executeHostedTurn( }, ) : undefined; - if (resumeFromToolResults) { + if (resumeApproval) { + if (!toolTurn) + throw new HostedToolRecoveryRequiredError( + 'Saved tool owner is unavailable.', + ); + const authorization = await authority.harnessRunAuthorization(); + if (authorization.status !== 'runnable') + throw new HostedToolRecoveryRequiredError( + 'Saved tool continuation is blocked.', + ); + const parts = await toolTurn.resumeSavedNativeBatch( + resumeApproval, + resumeApproval.plan.approvalOrdinal + 1, + abort.signal, + ); + resumeFromToolResults = parts; + onResumeReady?.(); + } else if (resumeFromToolResults || resumeModel?.pendingToolResults) { if (!toolTurn) throw new HostedToolRecoveryRequiredError( 'Tool turn is unavailable.', @@ -2596,6 +2950,7 @@ async function executeHostedTurn( } onResumeReady?.(); } + if (resumeModel && !resumeModel.pendingToolResults) onResumeReady?.(); let state: 'completed' | 'cancelled' | 'error' = 'completed'; let stopReason = 'end_turn'; try { @@ -2616,6 +2971,38 @@ async function executeHostedTurn( ...(session.hooks ? { hooks: session.hooks } : {}), ...(toolTurn ? { toolTurn } : {}), ...(resumeFromToolResults ? { resumeFromToolResults } : {}), + ...(resumeApproval + ? { initialRound: resumeApproval.plan.round + 1 } + : {}), + ...(resumeModel ? { resumeModel } : {}), + ...(!session.hooks + ? { + prepareModel: async ( + prepared, + round, + pendingToolResults, + context, + captureUnsupported, + ) => { + if (captureUnsupported) return undefined; + const snapshot = createHostedModelRequest( + session.managed, + promptId, + prepared, + round, + pendingToolResults, + context, + session.streamOwner, + ); + const ref = await publishHostedModelRequest( + session.managed, + snapshot, + ); + deltas.bindMessageId(snapshot.messageId); + return ref; + }, + } + : {}), ...(deltas ? { textDeltas: deltas } : {}), }); await commit( @@ -2627,7 +3014,8 @@ async function executeHostedTurn( if ( cause instanceof HostedToolRecoveryRequiredError || cause instanceof HostedMcpRecoveryRequiredError || - cause instanceof HostedHookRecoveryRequiredError + cause instanceof HostedHookRecoveryRequiredError || + cause instanceof LlmRequestPreparationError ) throw cause; const outcome = settledTurnOutcome(abort); @@ -2663,6 +3051,19 @@ async function executeHostedTurn( }); onTurnResult?.(turnResult); await session.managed.sink.write(turnResult); + try { + await toolTurn?.cleanup(); + } catch (cause) { + if (!session.blocked) { + session.cleanupBlocked = true; + session.blocked = true; + session.monitorWake?.kick(); + } + writeStderrLineSafe( + 'qwen serve: Turn completed with cleanup still owed: ' + + String(cause), + ); + } // H4b: the tool-arm results this turn answered are consumed // facts only once the turn's own settlement commits durably — a // crash before this point leaves accepted-not-consumed evidence @@ -2787,6 +3188,7 @@ export function registerHostedHarnessSessionRoutes( } refusedAdoptions.delete(sessionId); resident.blocked = false; + resident.cleanupBlocked = false; sendAttachment(res, sessionId, resident); if (settle !== null) runSettleProjection(resident, sessionId, settle, brokerOptions); @@ -3028,6 +3430,20 @@ export function registerHostedHarnessSessionRoutes( // The owner proven above restates the host's capability; a later // Turn reads it, never a stale one from the first attach. resident.childWorkspaces = childWorkspaces; + if ( + !resident.active && + brokerOptions && + hasHostedCleanupDebt(resident.managed) && + !unsettledPromptId(resident) + ) { + try { + await retryHostedCleanup(resident, brokerOptions); + } catch (cause) { + writeStderrLineSafe( + 'qwen serve: Cleanup remains owed: ' + String(cause), + ); + } + } // The cancellation signal pays identically on an attached re-answer // (R9-2): the settle separation lived only on the first-load branch, // so a cancellation takeover forced onto an attached Session fell @@ -3117,6 +3533,10 @@ export function registerHostedHarnessSessionRoutes( sendAttachment(res, sessionId, resident); return; } + if (resident.active?.promptId === parked) { + sendAttachment(res, sessionId, resident); + return; + } if (resident.active !== undefined) { error(res, 409, 'hosted_session_already_attached'); return; @@ -3124,6 +3544,95 @@ export function registerHostedHarnessSessionRoutes( // Only a load this resident Session admits may stop its messages, // and before the recovery it drives can start one. if (stopMessages) resident.messagesStopped = true; + if (!resident.mcp) { + if (resident.mcpClosing) { + noteOwedAdoption(resident, sessionId); + return error(res, 409, 'hosted_session_closing'); + } + if (resident.mcpRecovering) + return error(res, 409, 'hosted_turn_active'); + resident.mcpRecovering += 1; + try { + const committed = await findHostedCommittedModelOutput( + resident.managed, + parked, + ); + if (committed) { + if (sessions.get(sessionId) !== resident) { + noteOwedAdoption(resident, sessionId); + return error(res, 404, 'hosted_session_not_found'); + } + if (resident.mcpClosing) { + noteOwedAdoption(resident, sessionId); + return error(res, 409, 'hosted_session_closing'); + } + runSettleProjection( + resident, + sessionId, + parked, + brokerOptions, + true, + ); + sendAttachment(res, sessionId, resident); + return; + } + const model = await findHostedModelRequest(resident.managed, parked); + const approval = + resident.toolProfile && brokerOptions + ? await findHostedApprovalContinuation(resident.managed, parked) + : undefined; + if (model || approval) { + const user = (await resident.managed.sink.project()).find( + (item) => item.type === 'user' && item.daemonPromptId === parked, + ); + const text = user?.message?.parts + ?.flatMap((part) => + typeof part.text === 'string' ? [part.text] : [], + ) + .join('\n'); + if (!text) + throw new HostedToolRecoveryRequiredError( + 'Original Prompt is missing.', + ); + if (sessions.get(sessionId) !== resident) { + noteOwedAdoption(resident, sessionId); + return error(res, 404, 'hosted_session_not_found'); + } + if (resident.mcpClosing) { + noteOwedAdoption(resident, sessionId); + return error(res, 409, 'hosted_session_closing'); + } + await startHostedRecoveredTurn( + resident, + sessionId, + cwd, + { promptId: parked, text, parts: [] }, + brokerOptions, + approval, + model, + ); + if (sessions.get(sessionId) !== resident) { + noteOwedAdoption(resident, sessionId); + return error(res, 404, 'hosted_session_not_found'); + } + if (resident.mcpClosing) { + noteOwedAdoption(resident, sessionId); + return error(res, 409, 'hosted_session_closing'); + } + sendAttachment(res, sessionId, resident); + return; + } + } catch (cause) { + resident.blocked = true; + writeStderrLineSafe( + 'qwen serve: Hosted recovery redrive failed: ' + String(cause), + ); + error(res, 409, 'hosted_turn_recovery_required'); + return; + } finally { + resident.mcpRecovering -= 1; + } + } if (resident.toolProfile === undefined || !brokerOptions) { recoveryDeclined(res, 'model_start'); return; @@ -3137,7 +3646,7 @@ export function registerHostedHarnessSessionRoutes( promptId: parked, brokerOptions, passive: false, - leaseAlreadyHeld: resident.runtimeLeaseHeld !== undefined, + leaseAlreadyHeld: resident.runtimeLeaseHeld?.promptId === parked, }); if (outcome.kind === 'declined') { recoveryDeclined(res, outcome.reason); @@ -3149,9 +3658,10 @@ export function registerHostedHarnessSessionRoutes( } recovery = outcome.turn.report; if (outcome.turn.acquiredRuntime) - resident.runtimeLeaseHeld = - outcome.turn.report.executions[0]?.runtimeSessionId ?? - outcome.turn.promptId; + resident.runtimeLeaseHeld = { + promptId: outcome.turn.promptId, + runtimeSessionId: outcome.turn.runtimeSessionId, + }; } catch (cause) { writeStderrLineSafe( `qwen serve: Hosted Harness recovery re-answer of session ${sessionId} failed: ${sanitizeDaemonLogLine(String(cause))}`, @@ -3255,7 +3765,10 @@ export function registerHostedHarnessSessionRoutes( brokerOptions, passive: true, onPassiveRuntimeAcquired: (runtimeSessionId) => { - resident.runtimeLeaseHeld = runtimeSessionId; + resident.runtimeLeaseHeld = { + promptId: parked, + runtimeSessionId, + }; }, }); if (outcome.kind === 'recovered') { @@ -3478,6 +3991,7 @@ export function registerHostedHarnessSessionRoutes( return; } const session: HostedSession = { + streamOwner: { bootId: contract.bootId, eventEpoch: epoch }, managed, storeBaseUrl: store.baseUrl, definition, @@ -3857,19 +4371,37 @@ export function registerHostedHarnessSessionRoutes( // The recovered lease hands back only once the terminal // record is durable: a release persisted earlier would // wedge the retry on runtime_session_not_acquirable. - await runtime.broker?.release().catch((cause: unknown) => { - if ( - cause instanceof HostedWorkspaceBrokerRejection && - cause.status === 404 - ) - return; - writeStderrLineSafe( - 'qwen serve: Interrupted channel wake turn ' + - turn.turnId + - ' could not hand back its recovered Runtime: ' + - String(cause), - ); - }); + if (brokerOptions && hasHostedCleanupDebt(session.managed)) { + try { + if ( + runtime.broker && + !(await releaseHeldRuntime(session, brokerOptions, { + promptId: turn.turnId, + runtimeSessionId: runtime.broker.runtimeSessionId, + })) + ) + return 'busy'; + await retryHostedCleanup(session, brokerOptions); + } catch (cause) { + if (!session.blocked) session.cleanupBlocked = true; + session.blocked = true; + throw cause; + } + } else { + await runtime.broker?.release().catch((cause: unknown) => { + if ( + cause instanceof HostedWorkspaceBrokerRejection && + cause.status === 404 + ) + return; + writeStderrLineSafe( + 'qwen serve: Interrupted channel wake turn ' + + turn.turnId + + ' could not hand back its recovered Runtime: ' + + String(cause), + ); + }); + } return settled ? 'settled' : false; }, writeStderr: writeStderrLineSafe, @@ -3964,6 +4496,7 @@ export function registerHostedHarnessSessionRoutes( const aftermath = await wakeAftermath(turn.turnId); if (aftermath === 'settled') { session.blocked = false; + session.cleanupBlocked = false; } } if ( @@ -4002,10 +4535,21 @@ export function registerHostedHarnessSessionRoutes( // renewal) no-ops at the aftermath gates; anything else // already carries its own retry. if (!session.blocked || wakeBusy()) return; + if (session.cleanupBlocked && brokerOptions) { + try { + await retryHostedCleanup(session, brokerOptions); + } catch { + return; + } + if (!session.blocked) return; + } const first = firstPendingWakeInput(session); if (first === undefined) return; const aftermath = await wakeAftermath(first.turnId); - if (aftermath === 'settled') session.blocked = false; + if (aftermath === 'settled') { + session.blocked = false; + session.cleanupBlocked = false; + } }, failed: (cause) => { session.blocked = true; @@ -4186,7 +4730,76 @@ export function registerHostedHarnessSessionRoutes( } } let resume: { promptId: string; text: string; parts: Part[] } | undefined; - let settlePromptId: string | undefined; + let resumeModel: HostedModelRequest | undefined; + let committedModelOutput: { promptId: string } | undefined; + if ( + takeover && + unsettled && + body?.['cancellationTakeover'] !== true && + !session.hooks && + !session.mcp + ) { + committedModelOutput = await findHostedCommittedModelOutput( + managed, + unsettled, + ); + if (!committedModelOutput) + resumeModel = await findHostedModelRequest(managed, unsettled); + if (resumeModel) { + const user = (await managed.sink.project()).find( + (item) => item.type === 'user' && item.daemonPromptId === unsettled, + ); + const text = user?.message?.parts + ?.flatMap((part) => + typeof part.text === 'string' ? [part.text] : [], + ) + .join('\n'); + if (!text) + throw new HostedToolRecoveryRequiredError( + 'Original model Prompt is missing.', + ); + resume = { promptId: unsettled, text, parts: [] }; + } + } + let resumeApproval: RestoredHostedApprovalContinuation | undefined; + if ( + takeover && + unsettled && + session.toolProfile && + !session.hooks && + !session.mcp && + brokerOptions + ) { + const verdict = await managed.authority.harnessRunAuthorization(); + if ( + verdict.status === 'runnable' && + ['before_model', 'await_action', 'model_output_committed'].includes( + verdict.checkpoint.continuation.phase, + ) + ) { + resumeApproval = await findHostedApprovalContinuation( + managed, + unsettled, + ); + if (resumeApproval) { + const user = (await managed.sink.project()).find( + (item) => + item.type === 'user' && item.daemonPromptId === unsettled, + ); + const text = user?.message?.parts + ?.flatMap((part) => + typeof part.text === 'string' ? [part.text] : [], + ) + .join('\n'); + if (!text) + throw new HostedToolRecoveryRequiredError( + 'Original Prompt is missing.', + ); + resume = { promptId: unsettled, text, parts: [] }; + } + } + } + let settlePromptId: string | undefined = committedModelOutput?.promptId; let recovery: HostedRuntimeRecoveryReport | undefined; // A takeover that answers inapplicable has spoken: the plain attach // must not fail the bare-load refusal below on its empty recovery. @@ -4194,7 +4807,10 @@ export function registerHostedHarnessSessionRoutes( if ( restore.recoveryStatus === 'ok' && unsettled !== undefined && - takeover + takeover && + !resumeApproval && + !resumeModel && + !settlePromptId ) { // A parked Runtime turn is taken over, not refused: settle its // executions under their original ids (or report them for a @@ -4277,7 +4893,10 @@ export function registerHostedHarnessSessionRoutes( brokerOptions, passive: body?.['passiveManagedRuntimeRecovery'] === true, onPassiveRuntimeAcquired: (runtimeSessionId) => { - session.runtimeLeaseHeld = runtimeSessionId; + session.runtimeLeaseHeld = { + promptId: unsettled, + runtimeSessionId, + }; }, }); if (outcome.kind === 'inapplicable') { @@ -4333,9 +4952,10 @@ export function registerHostedHarnessSessionRoutes( } else if (outcome.kind === 'recovered') { recovery = outcome.turn.report; if (outcome.turn.acquiredRuntime) - session.runtimeLeaseHeld = - outcome.turn.report.executions[0]?.runtimeSessionId ?? - outcome.turn.promptId; + session.runtimeLeaseHeld = { + promptId: outcome.turn.promptId, + runtimeSessionId: outcome.turn.runtimeSessionId, + }; } // inapplicable: nothing a takeover owes this payload — the load // continues as the plain attach it was before G3, so a requested @@ -4526,46 +5146,30 @@ export function registerHostedHarnessSessionRoutes( ) session.blocked = true; if (!session.lifecycle) await settleCancelledHookTurn(session); - if (resume) { - const abort = new AbortController(); - session.active = { - promptId: resume.promptId, - digest: '', - abort, - }; - let resolveReady!: () => void; - let rejectReady!: (cause: unknown) => void; - const ready = new Promise((resolve, reject) => { - resolveReady = resolve; - rejectReady = reject; - }); - const resumed = executeHostedTurn( + if (resume) + await startHostedRecoveredTurn( session, sessionId, cwd, - resume.promptId, - resume.text, - abort, + resume, brokerOptions, - resume.parts, - undefined, - resolveReady, + resumeApproval, + resumeModel, + body?.['cancellationTakeover'] === true, ); - // Do not attach a Session whose original continuation cannot acquire - // Workspace ownership. The caller can retry load without losing it. - void resumed.catch(rejectReady); - await ready; - void resumed - .catch((cause: unknown) => { - session.blocked = true; - writeStderrLineSafe( - 'qwen serve: Hosted Harness recovery remained blocked: ' + - String(cause), - ); - }) - .finally(() => { - session.active = undefined; - }); + if ( + brokerOptions && + hasHostedCleanupDebt(managed) && + !unsettledPromptId(session) + ) { + try { + await retryHostedCleanup(session, brokerOptions); + } catch (cause) { + session.blocked = true; + writeStderrLineSafe( + 'qwen serve: Cleanup remains owed: ' + String(cause), + ); + } } sessions.set(sessionId, session); // H6c: a settle → settle-revision crash window closes here. It runs @@ -4593,7 +5197,13 @@ export function registerHostedHarnessSessionRoutes( refusedAdoptions.delete(sessionId); sendAttachment(res, sessionId, session, recovery); if (settlePromptId) - runSettleProjection(session, sessionId, settlePromptId, brokerOptions); + runSettleProjection( + session, + sessionId, + settlePromptId, + brokerOptions, + !!committedModelOutput, + ); } catch (cause) { await managed?.close().catch(() => undefined); await stores.close().catch(() => undefined); @@ -4746,6 +5356,8 @@ export function registerHostedHarnessSessionRoutes( session.hooksBusy ) return error(res, 409, 'hosted_turn_active'); + if (hasHostedCleanupDebt(session.managed) && unsettledPromptId(session)) + return error(res, 503, 'hosted_lifecycle_recovery_required'); const previousAuthority = session.lifecycle; session.stores!.setLifecycleAuthority(authority); session.hooksBusy = true; @@ -4756,6 +5368,13 @@ export function registerHostedHarnessSessionRoutes( session.lifecycle = authority; session.lifecycleKind = kind; session.mcpClosing = true; + if (brokerOptions) + await retryHostedCleanup(session, { + ...brokerOptions, + lifecycleAuthority: () => session.lifecycle, + }); + if (hasHostedCleanupDebt(session.managed)) + throw new Error('Runtime cleanup remains owed.'); const events = kind === 'close' ? [HookEventName.SessionEnd] @@ -5163,7 +5782,8 @@ export function registerHostedHarnessSessionRoutes( if ( cause instanceof HostedToolRecoveryRequiredError || cause instanceof HostedMcpRecoveryRequiredError || - cause instanceof HostedHookRecoveryRequiredError + cause instanceof HostedHookRecoveryRequiredError || + cause instanceof LlmRequestPreparationError ) { if (admitted) session.blocked = true; else if (!res.headersSent) @@ -5604,15 +6224,53 @@ export function registerHostedHarnessSessionRoutes( session: HostedSession, sessionId: string, ): void => { - const runtimeSessionId = session.runtimeLeaseHeld; - if (runtimeSessionId === undefined || refusedAdoptions.has(sessionId)) - return; + const lease = session.runtimeLeaseHeld; + if (lease === undefined || refusedAdoptions.has(sessionId)) return; + const { runtimeSessionId } = lease; refusedAdoptions.set(sessionId, runtimeSessionId); writeStderrLineSafe( `qwen serve: Hosted Harness takeover of session ${sessionId} adopted Runtime Session ${runtimeSessionId} but refuses the load: the lease stays owed until this session loads successfully or retires.`, ); }; + const releaseHeldRuntime = async ( + session: HostedSession, + options: HostedWorkspaceBrokerOptions, + lease: NonNullable, + drain = false, + ): Promise => { + const { promptId, runtimeSessionId } = lease; + const states = await reconcileHostedTurnCleanup( + session.managed, + options, + promptId, + drain, + ); + if (states.get(promptId) === 'owed') return false; + if ( + runtimeSessionId !== hostedRuntimeSessionId(promptId) || + !states.has(promptId) + ) { + try { + await new HostedWorkspaceBroker( + options, + session.managed.authority.sessionHeader.sessionKey, + runtimeSessionId, + ).release(drain ? { drain: true } : undefined); + } catch (cause) { + if ( + !(cause instanceof HostedWorkspaceBrokerRejection) || + cause.status !== 404 || + cause.code !== 'runtime_session_not_found' + ) + throw cause; + } + } + if (!hasHostedCleanupDebt(session.managed)) + await retryHostedCleanup(session, options); + return true; + }; + // A recovery load may hold the Runtime Session. On the cancellation // path, terminal routes hand it back — or the workspace lease stays // pinned forever — but retry-inviting refusals must not (see the field @@ -5621,25 +6279,25 @@ export function registerHostedHarnessSessionRoutes( // once the release is confirmed, so a failed handback stays owed and the // next terminal route retries it. const releaseRecoveredRuntime = (session: HostedSession): void => { - const promptId = session.runtimeLeaseHeld; - if (promptId === undefined || !brokerOptions) return; - new HostedWorkspaceBroker( - brokerOptions, - session.managed.authority.sessionHeader.sessionKey, - hostedRuntimeSessionId(promptId), + const lease = session.runtimeLeaseHeld; + if (lease === undefined || !brokerOptions) return; + // MCP configurations own their Runtime until their own drain completes. + if ( + lease.runtimeSessionId !== hostedRuntimeSessionId(lease.promptId) && + (session.mcp || unsettledPromptId(session) === lease.promptId) ) - .release() - .then( - () => { - if (session.runtimeLeaseHeld === promptId) - session.runtimeLeaseHeld = undefined; - }, - (cause: unknown) => { - writeStderrLineSafe( - `qwen serve: Hosted Harness release of recovered Runtime ${promptId} failed: ${String(cause)}`, - ); - }, - ); + return; + void releaseHeldRuntime(session, brokerOptions, lease).then( + (confirmed) => { + if (confirmed && session.runtimeLeaseHeld === lease) + session.runtimeLeaseHeld = undefined; + }, + (cause: unknown) => { + writeStderrLineSafe( + `qwen serve: Hosted Harness release of recovered Runtime ${lease.runtimeSessionId} failed: ${String(cause)}`, + ); + }, + ); }; // Awaited variant for exits after which no route can retry the handback @@ -5648,23 +6306,31 @@ export function registerHostedHarnessSessionRoutes( // a recovered Runtime can hold live background runs (#13856), and only // the drain arm may stop them. const releaseLeaseNow = async (session: HostedSession): Promise => { - const promptId = session.runtimeLeaseHeld; - if (promptId === undefined || !brokerOptions) return; - await new HostedWorkspaceBroker( - brokerOptions, - session.managed.authority.sessionHeader.sessionKey, - hostedRuntimeSessionId(promptId), + const lease = session.runtimeLeaseHeld; + if (lease === undefined || !brokerOptions) return; + if ( + session.mcp && + lease.runtimeSessionId !== hostedRuntimeSessionId(lease.promptId) ) - .release({ drain: true }) - .then(() => { - if (session.runtimeLeaseHeld === promptId) + return; + await releaseHeldRuntime( + session, + { ...brokerOptions, lifecycleAuthority: () => session.lifecycle }, + lease, + true, + ).then( + (confirmed) => { + if (!confirmed) + throw new Error('Original Runtime cleanup is still owed.'); + if (session.runtimeLeaseHeld === lease) session.runtimeLeaseHeld = undefined; - }) - .catch((cause: unknown) => { + }, + (cause: unknown) => { writeStderrLineSafe( - `qwen serve: Hosted Harness release of recovered Runtime ${promptId} failed: ${String(cause)}`, + `qwen serve: Hosted Harness release of recovered Runtime ${lease.runtimeSessionId} failed: ${String(cause)}`, ); - }); + }, + ); }; /** @@ -7023,7 +7689,9 @@ export function registerHostedHarnessSessionRoutes( // never held one — its recovery acquires nothing (acquiredRuntime // false), so there is no lease to hand back here. const handedBack = - broker === undefined + broker === undefined || + (session.mcp && + broker.runtimeSessionId !== hostedRuntimeSessionId(promptId)) ? false : await broker.release().then( () => true, @@ -7213,7 +7881,7 @@ export function registerHostedHarnessSessionRoutes( lastSeenAt: Date.now(), }); }); - app.post('/session/:id/cancel', (req, res) => { + app.post('/session/:id/cancel', async (req, res) => { const session = identity(req, sessions); if (!session) return error(res, 404, 'hosted_session_not_found'); // Honest refusal instead of a silent no-op 204 (R10-3): with no live @@ -7230,7 +7898,40 @@ export function registerHostedHarnessSessionRoutes( 'hosted_turn_recovery_required', 'inputs_unsettled', ); - session.active?.abort.abort(); + if (session.active) { + const active = session.active; + active.abort.abort(); + const promptId = active.promptId; + const authority = session.managed.authority; + const commandId = `hosted-cancel:${promptId}`; + try { + await authority.appendExecutionEvent( + { + operation: 'requestCancel', + commandId, + sessionKey: authority.sessionHeader.sessionKey, + contentDigest: createHash('sha256').update(promptId).digest('hex'), + }, + (sequence) => ({ + v: 1, + sequence, + eventId: commandId, + sessionKey: authority.sessionHeader.sessionKey, + kind: 'cancel.requested', + occurredAt: Date.now(), + payload: { + requestId: commandId, + target: { turnId: promptId }, + reason: 'user', + requestedBy: 'hosted', + }, + }), + { class: 'trusted_entry' }, + ); + } catch { + return error(res, 503, 'hosted_cancel_commit_failed'); + } + } res.sendStatus(204); }); app.post('/session/:id/actions/:requestId/resolve', (req, res) => { @@ -7637,7 +8338,16 @@ export function registerHostedHarnessSessionRoutes( // closes last. await session.shell?.publisher?.close(); await session.backgroundLane?.publisher?.close(); + const mcpLease = session.runtimeLeaseHeld; await session.mcp?.close(); + if ( + session.mcp && + mcpLease !== undefined && + mcpLease.runtimeSessionId !== + hostedRuntimeSessionId(mcpLease.promptId) && + session.runtimeLeaseHeld === mcpLease + ) + session.runtimeLeaseHeld = undefined; // No monitor, channel or automation notification may park the // Session: every pending one settles cancelled here, model-free, // before the log closes; a run whose input settled that way ends @@ -7856,6 +8566,7 @@ export function registerHostedHarnessSessionRoutes( const aftermath = await session.wakeAftermath(first.turnId); if (aftermath === 'settled') { session.blocked = false; + session.cleanupBlocked = false; repaired = true; } } diff --git a/packages/cli/src/serve/hosted-hook-session.test.ts b/packages/cli/src/serve/hosted-hook-session.test.ts index b1780f31b5..89c8453d70 100644 --- a/packages/cli/src/serve/hosted-hook-session.test.ts +++ b/packages/cli/src/serve/hosted-hook-session.test.ts @@ -5,6 +5,8 @@ */ import { randomUUID } from 'node:crypto'; +import { createServer } from 'node:http'; +import { WORKSPACE_CAPABILITY_DIGEST } from './managed-workspace-activation.js'; import { setTimeout as delay } from 'node:timers/promises'; import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; import { tmpdir } from 'node:os'; @@ -147,9 +149,20 @@ beforeEach(async () => { generation: '1', workspaceGeneration: '1', }; + Object.defineProperty(this, 'acquired', { + value: true, + configurable: true, + }); + }, + ); + vi.spyOn(HostedWorkspaceBroker.prototype, 'release').mockImplementation( + async function (this: HostedWorkspaceBroker) { + Object.defineProperty(this, 'acquired', { + value: false, + configurable: true, + }); }, ); - vi.spyOn(HostedWorkspaceBroker.prototype, 'release').mockResolvedValue(); vi.spyOn(HostedWorkspaceBroker.prototype, 'hookControl').mockImplementation( async (operation) => { requests.push(operation); @@ -3219,3 +3232,94 @@ it.each([false, true])( expect(runner).toHaveBeenCalledOnce(); }, ); + +it.each(['none', 'hook', 'borrowed'] as const)( + 'releases only acquired Hook owners after actual HTTP warm (%s)', + async (mode) => { + vi.mocked(HostedWorkspaceBroker.prototype.warm).mockRestore(); + vi.mocked(HostedWorkspaceBroker.prototype.acquire).mockRestore(); + vi.mocked(HostedWorkspaceBroker.prototype.release).mockRestore(); + const brokerRequests: Array<{ + path: string; + owner: string; + drain: unknown; + }> = []; + let held = false; + const server = createServer(async (req, res) => { + const chunks: Buffer[] = []; + for await (const chunk of req) chunks.push(Buffer.from(chunk)); + const body = JSON.parse(Buffer.concat(chunks).toString()); + const pathname = req.url!; + brokerRequests.push({ + path: pathname, + owner: body.runtimeSessionId, + drain: body.drain ?? false, + }); + const envelope = { + protocolVersion: 1, + requestId: body.requestId, + harnessSessionId: body.harnessSessionId, + runtimeSessionId: body.runtimeSessionId, + scope: { + tenantId: 'tenant', + workspaceId: 'workspace', + workspaceGeneration: '1', + capabilityDigest: WORKSPACE_CAPABILITY_DIGEST, + }, + runtime: { bindingId: 'binding', generation: '1' }, + }; + res.setHeader('Content-Type', 'application/json'); + if (pathname.endsWith('/runtimes:warm')) + res.end(JSON.stringify(envelope)); + else if (pathname.endsWith('/tool-sessions:acquire')) { + held = true; + res.end(JSON.stringify({ ...envelope, acquired: true })); + } else if (pathname.endsWith(':release') && held) { + held = false; + res.end(JSON.stringify({ ...envelope, released: true })); + } else { + res.statusCode = 404; + res.end(JSON.stringify({ code: 'runtime_session_not_found' })); + } + }); + await new Promise((resolve) => + server.listen(0, '127.0.0.1', resolve), + ); + try { + const port = (server.address() as { port: number }).port; + hooks = new HostedHookSession( + { baseUrl: `http://127.0.0.1:${port}`, token: 'test' }, + session, + pin, + ); + await hooks.broker.warm(); + expect(hooks.broker.runtime).toEqual({ + bindingId: 'binding', + generation: '1', + workspaceGeneration: '1', + }); + if (mode === 'hook') await hooks.acquire(); + if (mode === 'borrowed') await hooks.broker.acquire(); + expect(hooks.mountHeld).toBe(mode !== 'none'); + let failure: unknown; + try { + await hooks.close(); + } catch (cause) { + failure = cause; + } + expect(failure).toBeUndefined(); + expect( + brokerRequests.filter((r) => r.path.endsWith(':release')), + ).toHaveLength(mode === 'none' ? 0 : 1); + if (mode !== 'none') + expect( + brokerRequests.filter((r) => r.path.endsWith(':release'))[0].drain, + ).toBe(true); + } finally { + await new Promise((resolve) => { + server.close(() => resolve()); + server.closeAllConnections(); + }); + } + }, +); diff --git a/packages/cli/src/serve/hosted-hook-session.ts b/packages/cli/src/serve/hosted-hook-session.ts index 96185c6058..acb4ed5304 100644 --- a/packages/cli/src/serve/hosted-hook-session.ts +++ b/packages/cli/src/serve/hosted-hook-session.ts @@ -489,7 +489,7 @@ export class HostedHookSession { * acquisition-free state when a foreground child would need the mount. */ get mountHeld(): boolean { - return this.acquired; + return this.acquired || this.broker.acquired; } /** @@ -1787,7 +1787,7 @@ export class HostedHookSession { // before releasing the Session's own owner: the Session stays attached // with it, so a later turn can still run and retry the fenced release. if (this.fencedOwners.size > 0) throw new HostedHookRecoveryRequiredError(); - if (this.ownsBroker && this.broker.runtime) { + if (this.ownsBroker && this.mountHeld) { try { await this.broker.release({ drain: true }); } catch (cause) { diff --git a/packages/cli/src/serve/hosted-model-recovery.ts b/packages/cli/src/serve/hosted-model-recovery.ts new file mode 100644 index 0000000000..c689a6cff7 --- /dev/null +++ b/packages/cli/src/serve/hosted-model-recovery.ts @@ -0,0 +1,287 @@ +/** + * @license + * Copyright 2026 Qwen Team + * SPDX-License-Identifier: Apache-2.0 + */ +import { createHash, randomUUID } from 'node:crypto'; +import { isDeepStrictEqual } from 'node:util'; +import type { LlmPreparedRequest } from '@qwen-code/qwen-code-core/core/llm-chat.js'; +import { hasImagePayloadReferences } from '@qwen-code/qwen-code-core/services/image-payload-references.js'; +import type { ManagedSession } from '@qwen-code/qwen-code-core/managed-runtime/managed-session-assembly.js'; +import { parseHostedRecoveryResource } from '@qwen-code/qwen-code-core/managed-runtime/hosted-recovery-records.js'; +import { + assertManagedSessionDurableRef, + assertManagedSessionStableId, + type ManagedSessionDurableRef, +} from '@qwen-code/qwen-code-core/managed-runtime/managed-session-records.js'; + +export interface HostedModelRequest { + v: 1; + sessionKey: ManagedSession['authority']['sessionHeader']['sessionKey']; + promptId: string; + definitionRef: ManagedSessionDurableRef; + rootSnapshotRef: ManagedSessionDurableRef; + sourceActivation: ManagedSession['activation']; + sourceBootId: string; + sourceEventEpoch: string; + messageId: string; + round: number; + throughSequence: number; + pendingToolResults: boolean; + workspaceContext: string | null; + prepared: LlmPreparedRequest; + requestDigest: string; +} + +export function createHostedModelRequest( + session: ManagedSession, + promptId: string, + prepared: LlmPreparedRequest, + round: number, + pendingToolResults: boolean, + workspaceContext: string | undefined, + owner: { bootId: string; eventEpoch: string }, +): HostedModelRequest { + const header = session.authority.sessionHeader; + return { + v: 1, + sessionKey: header.sessionKey, + promptId, + definitionRef: header.definitionRef, + rootSnapshotRef: header.rootSnapshotRef, + sourceActivation: session.activation, + sourceBootId: owner.bootId, + sourceEventEpoch: owner.eventEpoch, + messageId: randomUUID(), + round, + throughSequence: session.authority.committedSequence, + pendingToolResults, + workspaceContext: workspaceContext ?? null, + prepared, + requestDigest: createHash('sha256') + .update(JSON.stringify(prepared)) + .digest('hex'), + }; +} + +export async function publishHostedModelRequest( + session: ManagedSession, + request: HostedModelRequest, +): Promise { + const bytes = Buffer.from(JSON.stringify(request)); + if (bytes.length > 64 * 1024) return undefined; + return session.resources.publish('hosted-model-request', bytes); +} + +async function readHostedModelRequest( + session: ManagedSession, + promptId: string, + allowCommittedOutput: boolean, +): Promise { + const events = session.authority.eventsInSequenceRange( + 1, + session.authority.committedSequence, + ); + const attempt = [...events] + .reverse() + .find( + (event) => + event.kind === 'model.attempt' && + typeof event.payload['attemptId'] === 'string' && + event.payload['attemptId'].startsWith(`${promptId}:main:`), + ); + if (!attempt || !attempt.payload['recoveryRef']) return undefined; + const ref = assertManagedSessionDurableRef( + attempt.payload['recoveryRef'], + 'model recovery request', + ); + if (ref.kind !== 'hosted-model-request' || ref.schemaVersion !== 1) + throw new Error('Invalid model request reference.'); + const raw = parseHostedRecoveryResource( + await session.resources.read(ref), + session.authority.sessionHeader.sessionKey, + ); + if (raw['promptId'] !== promptId) return undefined; + const saved = raw as unknown as HostedModelRequest; + if ( + Object.keys(raw).length !== 15 || + !isDeepStrictEqual( + saved.definitionRef, + session.authority.sessionHeader.definitionRef, + ) || + !isDeepStrictEqual( + saved.rootSnapshotRef, + session.authority.sessionHeader.rootSnapshotRef, + ) || + !Number.isSafeInteger(saved.round) || + saved.round < 0 || + saved.round >= 16 || + !Number.isSafeInteger(saved.throughSequence) || + saved.throughSequence < 1 || + saved.throughSequence >= attempt.sequence || + typeof saved.pendingToolResults !== 'boolean' || + !( + saved.workspaceContext === null || + typeof saved.workspaceContext === 'string' + ) || + !saved.sourceActivation || + Object.keys(saved.sourceActivation).length !== 2 || + !Number.isSafeInteger(saved.sourceActivation.epoch) || + saved.sourceActivation.epoch < 1 || + !saved.prepared || + Object.keys(saved.prepared).length !== + 6 + (Object.hasOwn(saved.prepared, 'allowPersistentRetry') ? 1 : 0) || + (Object.hasOwn(saved.prepared, 'allowPersistentRetry') && + typeof saved.prepared.allowPersistentRetry !== 'boolean') || + !Array.isArray(saved.prepared.history) || + !Array.isArray(saved.prepared.completedToolCallIds) || + !Array.isArray(saved.prepared.request?.contents) || + Object.keys(saved.prepared.request).length !== 3 || + !saved.prepared.request.config || + typeof saved.prepared.request.config !== 'object' || + Array.isArray(saved.prepared.request.config) || + ['abortSignal', 'httpOptions', 'apiKey', 'headers', 'authorization'].some( + (field) => Object.hasOwn(saved.prepared.request.config, field), + ) || + typeof saved.prepared.request.model !== 'string' || + !/^[a-f0-9]{64}$/u.test(saved.prepared.providerPin) || + !Number.isSafeInteger(saved.prepared.promptTokensForClamp) || + saved.prepared.promptTokensForClamp < 0 || + saved.requestDigest !== + createHash('sha256').update(JSON.stringify(saved.prepared)).digest('hex') + ) + throw new Error('Invalid saved model request.'); + for (const id of [ + saved.messageId, + saved.sourceBootId, + saved.sourceEventEpoch, + saved.sourceActivation.activationId, + saved.prepared.routeSelector, + ]) + assertManagedSessionStableId(id, 'saved model identity'); + if ( + !/^[^:]+:.+$/u.test(saved.prepared.routeSelector) || + new Set(saved.prepared.completedToolCallIds).size !== + saved.prepared.completedToolCallIds.length + ) + throw new Error('Invalid saved model route or completed calls.'); + for (const id of saved.prepared.completedToolCallIds) + assertManagedSessionStableId(id, 'completed call'); + for (const content of [ + ...saved.prepared.history, + ...saved.prepared.request.contents, + ]) + if ( + !content || + !['user', 'model'].includes(content.role ?? '') || + !Array.isArray(content.parts) || + content.parts.some( + (part) => !part || typeof part !== 'object' || Array.isArray(part), + ) + ) + throw new Error('Invalid saved model content.'); + const started = events.find( + (event) => + event.kind === 'model.attempt' && + event.payload['attemptId'] === attempt.payload['attemptId'] && + event.payload['state'] === 'started', + ); + if ( + started?.subject?.type !== 'activation' || + started.subject.activationId !== saved.sourceActivation.activationId || + started.subject.epoch !== saved.sourceActivation.epoch + ) + throw new Error('Saved model source differs from original attempt.'); + // Any committed output after this request belongs to a different recovery path. + if ( + !allowCommittedOutput && + events.some( + (event) => + event.sequence > saved.throughSequence && + ((event.kind === 'message.committed' && + event.payload['role'] === 'assistant') || + event.kind === 'tool.intent'), + ) + ) + return undefined; + const verdict = await session.authority.harnessRunAuthorization(); + if ( + verdict.status !== 'runnable' || + verdict.checkpoint.approval || + !(verdict.checkpoint.tools?.items ?? []).every( + (item) => item.state === 'settled', + ) || + ![ + 'before_model', + 'model_output_committed', + 'results_ready', + 'turn_settled', + ].includes(verdict.checkpoint.continuation.phase) + ) + return undefined; + if ( + !allowCommittedOutput && + hasImagePayloadReferences(saved.prepared.history) + ) + return undefined; + return saved; +} + +export async function findHostedModelRequest( + session: ManagedSession, + promptId: string, +): Promise { + return readHostedModelRequest(session, promptId, false); +} + +export async function findHostedCommittedModelOutput( + session: ManagedSession, + promptId: string, +): Promise<{ promptId: string } | undefined> { + const events = session.authority.eventsInSequenceRange( + 1, + session.authority.committedSequence, + ); + const latestAttempt = events.findLast( + (event) => + event.kind === 'model.attempt' && + typeof event.payload['attemptId'] === 'string' && + event.payload['attemptId'].startsWith(`${promptId}:main:`), + ); + if (latestAttempt?.payload['state'] !== 'output_committed') return; + const saved = latestAttempt.payload['recoveryRef'] + ? await readHostedModelRequest(session, promptId, true) + : undefined; + if (latestAttempt.payload['recoveryRef'] && !saved) return; + const current = (await session.sink.project()).filter( + (item) => item.daemonPromptId === promptId, + ); + const last = current.at(-1); + if ( + last?.type !== 'assistant' || + (saved && last.uuid !== saved.messageId) || + !last.message?.parts?.length || + last.message.parts.some((part) => part.functionCall) + ) + return; + const committed = events.findLast( + (event) => + event.kind === 'message.committed' && + event.payload['messageId'] === last.uuid, + ); + if (!committed || committed.sequence <= latestAttempt.sequence) return; + const verdict = await session.authority.harnessRunAuthorization(); + if ( + verdict.status !== 'runnable' || + verdict.checkpoint.approval || + ![ + 'before_model', + 'model_output_committed', + 'results_ready', + 'turn_settled', + ].includes(verdict.checkpoint.continuation.phase) || + (verdict.checkpoint.tools?.items ?? []).some((item) => !item.consumed) + ) + return; + return saved ?? { promptId }; +} diff --git a/packages/cli/src/serve/hosted-monitor-wake-turn.ts b/packages/cli/src/serve/hosted-monitor-wake-turn.ts index 2379ba570b..dd9339beda 100644 --- a/packages/cli/src/serve/hosted-monitor-wake-turn.ts +++ b/packages/cli/src/serve/hosted-monitor-wake-turn.ts @@ -13,6 +13,7 @@ import type { HostedChildAgentSession } from './hosted-child-agent-session.js'; import { HostedToolRecoveryRequiredError } from './hosted-workspace-tool-turn.js'; import { HostedMcpRecoveryRequiredError } from './hosted-mcp-session.js'; import { HostedHookRecoveryRequiredError } from './hosted-hook-session.js'; +import { LlmRequestPreparationError } from '@qwen-code/qwen-code-core/core/llm-chat.js'; /** * The Session's one wake-recovery classification, shared by its wiring @@ -24,7 +25,8 @@ export function monitorWakeNeedsRecovery(cause: unknown): boolean { return ( cause instanceof HostedToolRecoveryRequiredError || cause instanceof HostedMcpRecoveryRequiredError || - cause instanceof HostedHookRecoveryRequiredError + cause instanceof HostedHookRecoveryRequiredError || + cause instanceof LlmRequestPreparationError ); } @@ -63,8 +65,8 @@ export function createMonitorWakeRunTurn(params: { * re-read with the busy verdict right before the turn starts. */ readonly held?: (turn: HostedMonitorWakeTurn) => boolean; /** - * The recovery-required classification. It is injected because the three - * RecoveryRequiredError classes must be matched by type — their + * The recovery-required classification. It is injected because these + * error classes must be matched by type — their * constructors never assign `name`, so every instance reads `name === * 'Error'` and a name comparison silently reclassifies a recovery * exception as an ordinary failure, settling its unconsumed input. diff --git a/packages/cli/src/serve/hosted-monitor-wake.test.ts b/packages/cli/src/serve/hosted-monitor-wake.test.ts index 5fbf7c3be0..feef38a203 100644 --- a/packages/cli/src/serve/hosted-monitor-wake.test.ts +++ b/packages/cli/src/serve/hosted-monitor-wake.test.ts @@ -31,6 +31,7 @@ import { import { HostedToolRecoveryRequiredError } from './hosted-workspace-tool-turn.js'; import { HostedMcpRecoveryRequiredError } from './hosted-mcp-session.js'; import { HostedHookRecoveryRequiredError } from './hosted-hook-session.js'; +import { LlmRequestPreparationError } from '@qwen-code/qwen-code-core/core/llm-chat.js'; // monitor_run is enabled by the H3 enablement slice; the close-side settle // rig commits a notification input ahead of it, like the funnel suite does. @@ -533,6 +534,7 @@ describe('createMonitorWakeRunTurn', () => { ['tool', () => new HostedToolRecoveryRequiredError(new Error('parked'))], ['mcp', () => new HostedMcpRecoveryRequiredError()], ['hook', () => new HostedHookRecoveryRequiredError()], + ['model preparation', () => new LlmRequestPreparationError('store lost')], ])( 'leaves a %s recovery-required input unsettled rather than consuming it', async (_kind, makeCause) => { diff --git a/packages/cli/src/serve/hosted-recovery.test.ts b/packages/cli/src/serve/hosted-recovery.test.ts new file mode 100644 index 0000000000..5b8cf9fda3 --- /dev/null +++ b/packages/cli/src/serve/hosted-recovery.test.ts @@ -0,0 +1,827 @@ +/** + * @license + * Copyright 2026 Qwen Team + * SPDX-License-Identifier: Apache-2.0 + */ +import { createHash, randomUUID } from 'node:crypto'; +import { mkdtemp, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +import { afterEach, expect, it, vi } from 'vitest'; +import { openManagedSession } from '@qwen-code/qwen-code-core/managed-runtime/managed-session-assembly.js'; +import { LocalManagedSessionResourceStore } from '@qwen-code/qwen-code-core/managed-runtime/managed-session-resources.js'; +import { createManagedHarnessHandle } from '@qwen-code/qwen-code-core/managed-runtime/managed-harness-factory.js'; +import { ManagedHookActivationController } from '@qwen-code/qwen-code-core/managed-runtime/managed-hook-activation.js'; +import type { LlmPreparedRequest } from '@qwen-code/qwen-code-core/core/llm-chat.js'; +import type { Part } from '@google/genai'; +import { + hostedRuntimeSessionId, + HostedWorkspaceBroker, +} from './hosted-workspace-broker.js'; +import { HostedWorkspaceToolTurn } from './hosted-workspace-tool-turn.js'; +import { HostedApprovalWaiters } from './hosted-tool-approval.js'; +import { + hasHostedCleanupDebt, + oweHostedTurnCleanup, + reconcileHostedTurnCleanup, +} from './hosted-turn-cleanup.js'; +import { + createHostedModelRequest, + findHostedModelRequest, + findHostedCommittedModelOutput, + publishHostedModelRequest, +} from './hosted-model-recovery.js'; +import { HostedTextDeltaStream } from './hosted-text-deltas.js'; + +const directories: string[] = []; +afterEach(async () => { + vi.restoreAllMocks(); + await Promise.all( + directories + .splice(0) + .map((root) => rm(root, { recursive: true, force: true })), + ); +}); + +async function fixture(promptId: string = randomUUID()) { + const root = await mkdtemp(path.join(tmpdir(), 'hosted-recovery-')); + directories.push(root); + const sessionKey = { + tenantId: 'tenant', + workspaceId: 'workspace', + sessionId: randomUUID(), + }; + const resources = LocalManagedSessionResourceStore.create({ + runtimeBaseDir: root, + sessionKey, + }); + const options = { + runtimeBaseDir: root, + cwd: root, + transcriptPath: path.join(root, 'session.jsonl'), + sessionId: sessionKey.sessionId, + sessionKey, + version: 'test', + activationLeaseDurationMs: 60_000, + }; + const session = await openManagedSession({ + ...options, + workerId: 'first', + create: { + definitionRef: await resources.publish( + 'managed-definition', + Buffer.from('{}'), + ), + rootSnapshotRef: await resources.publish( + 'managed-root', + Buffer.from('{}'), + ), + createdBy: 'test', + }, + }); + const content = Buffer.from('[]'); + await session.authority.submitInput( + { + operation: 'submitInput', + commandId: promptId, + sessionKey, + contentDigest: createHash('sha256').update(content).digest('hex'), + }, + { + inputId: promptId, + turnId: promptId, + source: 'test', + contentRef: await resources.publish('managed-input', content), + admissionRef: await resources.publish( + 'managed-admission', + Buffer.from('{}'), + ), + deadline: null, + wakeReason: 'input', + }, + ); + await createManagedHarnessHandle(session).ensureRunnable(); + const terminal = () => + session.sink.write({ + uuid: randomUUID(), + parentUuid: null, + sessionId: sessionKey.sessionId, + timestamp: new Date().toISOString(), + type: 'system', + subtype: 'turn_result', + cwd: root, + version: 'test', + systemPayload: { + promptId, + state: 'completed', + stopReason: 'end_turn', + endedAt: Date.now(), + }, + }); + const brokerOptions = { baseUrl: 'http://127.0.0.1:1', token: 'fixture' }; + const broker = new HostedWorkspaceBroker( + brokerOptions, + sessionKey, + hostedRuntimeSessionId(promptId), + ); + broker.runtime = { + bindingId: 'original-binding', + generation: '7', + workspaceGeneration: '3', + }; + return { + session, + promptId, + broker, + brokerOptions, + terminal, + reopen: () => openManagedSession({ ...options, workerId: 'replacement' }), + }; +} + +it.each(['ordinary-prompt', 'monitor:wake:run'])( + 'reports scoped cleanup confirmation for %s', + async (promptId) => { + const f = await fixture(promptId); + try { + await oweHostedTurnCleanup(f.session, f.promptId, f.broker); + const release = vi + .spyOn(HostedWorkspaceBroker.prototype, 'release') + .mockResolvedValue(); + expect( + await reconcileHostedTurnCleanup( + f.session, + f.brokerOptions, + 'other-prompt', + ), + ).toEqual(new Map()); + expect( + await reconcileHostedTurnCleanup( + f.session, + f.brokerOptions, + f.promptId, + ), + ).toEqual(new Map([[f.promptId, 'owed']])); + expect(release).not.toHaveBeenCalled(); + await f.terminal(); + expect( + await reconcileHostedTurnCleanup( + f.session, + f.brokerOptions, + f.promptId, + ), + ).toEqual(new Map([[f.promptId, 'confirmed']])); + expect(release).toHaveBeenCalledOnce(); + await reconcileHostedTurnCleanup(f.session, f.brokerOptions, f.promptId); + expect(release).toHaveBeenCalledOnce(); + } finally { + await f.session.close(); + } + }, +); + +it.each([ + ['a'.repeat(243), false], + ['a'.repeat(244), true], + ['a'.repeat(496), true], + ['猫'.repeat(140), true], +] as const)( + 'preserves cleanup identity through confirmation (case %#)', + async (promptId, hashed) => { + const f = await fixture(promptId); + const original = `hosted-cleanup:${promptId}:${f.broker.runtimeSessionId}`; + const expected = hashed + ? `hosted-cleanup:${createHash('sha256').update(original).digest('hex')}` + : original; + const cleanupEvents = () => + f.session.authority + .eventsInSequenceRange(1, f.session.authority.committedSequence) + .filter((event) => event.kind === 'hosted.cleanup'); + try { + await oweHostedTurnCleanup(f.session, promptId, f.broker); + await oweHostedTurnCleanup(f.session, promptId, f.broker); + expect(cleanupEvents()).toHaveLength(1); + expect(cleanupEvents()[0]!.payload['cleanupId']).toBe(expected); + expect(cleanupEvents()[0]!.eventId).toBe(`${expected}:owed`); + const release = vi + .spyOn(HostedWorkspaceBroker.prototype, 'release') + .mockResolvedValue(); + await f.terminal(); + await reconcileHostedTurnCleanup(f.session, f.brokerOptions, promptId); + await reconcileHostedTurnCleanup(f.session, f.brokerOptions, promptId); + expect(release).toHaveBeenCalledOnce(); + expect(release).toHaveBeenCalledWith({ + bindingId: 'original-binding', + generation: '7', + }); + expect(cleanupEvents().map((event) => event.payload['state'])).toEqual([ + 'owed', + 'confirmed', + ]); + expect(cleanupEvents()[1]!.payload['cleanupId']).toBe(expected); + expect(cleanupEvents()[1]!.eventId).toBe(`${expected}:confirmed`); + expect( + Buffer.byteLength(cleanupEvents()[1]!.eventId), + ).toBeLessThanOrEqual(512); + expect(hasHostedCleanupDebt(f.session)).toBe(false); + } finally { + await f.session.close(); + } + }, +); + +it('journals cleanup for a maximum-length stable prompt without changing its owner', async () => { + const f = await fixture(); + const promptId = 'a'.repeat(512); + const broker = new HostedWorkspaceBroker( + f.brokerOptions, + f.session.authority.sessionHeader.sessionKey, + hostedRuntimeSessionId(promptId), + ); + broker.runtime = f.broker.runtime; + try { + await oweHostedTurnCleanup(f.session, promptId, broker); + const event = f.session.authority + .eventsInSequenceRange(1, f.session.authority.committedSequence) + .findLast((event) => event.kind === 'hosted.cleanup')!; + const original = `hosted-cleanup:${promptId}:${promptId}`; + expect(event.payload['cleanupId']).toBe( + `hosted-cleanup:${createHash('sha256').update(original).digest('hex')}`, + ); + const descriptor = JSON.parse( + ( + await f.session.resources.read( + event.payload['descriptorRef'] as unknown as Parameters< + typeof f.session.resources.read + >[0], + ) + ).toString('utf8'), + ); + expect(descriptor).toMatchObject({ + promptId, + runtimeSessionId: promptId, + fileHistoryTurnId: promptId, + ...broker.runtime, + }); + } finally { + await f.session.close(); + } +}); + +it.each([244, 245])( + 'confirms an existing raw cleanup identity after cold recovery (%s bytes)', + async (length) => { + const promptId = 'a'.repeat(length); + const f = await fixture(promptId); + const cleanupId = `hosted-cleanup:${promptId}:${f.broker.runtimeSessionId}`; + const descriptorRef = await f.session.resources.publish( + 'hosted-turn-cleanup', + Buffer.from( + JSON.stringify({ + v: 1, + sessionKey: f.session.authority.sessionHeader.sessionKey, + promptId, + runtimeSessionId: f.broker.runtimeSessionId, + ...f.broker.runtime, + fileHistoryTurnId: promptId, + }), + ), + ); + try { + // This exact legacy owed event fits; only its confirmation ID does not. + await f.session.authority.appendExecutionEvent( + { + operation: 'hostedCleanup', + commandId: `${cleanupId}:owed`, + sessionKey: f.session.authority.sessionHeader.sessionKey, + contentDigest: createHash('sha256') + .update(`${descriptorRef.digest}:owed`) + .digest('hex'), + }, + (sequence) => ({ + v: 1, + sequence, + eventId: `${cleanupId}:owed`, + sessionKey: f.session.authority.sessionHeader.sessionKey, + kind: 'hosted.cleanup', + occurredAt: Date.now(), + subject: { + type: 'activation', + scopeId: f.session.activation.activationId, + ...f.session.activation, + }, + payload: { cleanupId, descriptorRef, state: 'owed' }, + }), + { class: 'harness', activation: f.session.activation }, + ); + await f.terminal(); + } finally { + await f.session.close(); + } + const replacement = await f.reopen(); + const events = () => + replacement.authority + .eventsInSequenceRange(1, replacement.authority.committedSequence) + .filter((event) => event.kind === 'hosted.cleanup'); + try { + await oweHostedTurnCleanup(replacement, promptId, f.broker); + expect(events()).toHaveLength(1); + const release = vi + .spyOn(HostedWorkspaceBroker.prototype, 'release') + .mockResolvedValue(); + await reconcileHostedTurnCleanup(replacement, f.brokerOptions, promptId); + await reconcileHostedTurnCleanup(replacement, f.brokerOptions, promptId); + expect(release).toHaveBeenCalledOnce(); + expect(release).toHaveBeenCalledWith({ + bindingId: 'original-binding', + generation: '7', + }); + expect(events().map((event) => event.payload)).toEqual([ + { cleanupId, descriptorRef, state: 'owed' }, + { cleanupId, descriptorRef, state: 'confirmed' }, + ]); + expect(events()[1]!.eventId).not.toBe(`${cleanupId}:confirmed`); + expect(Buffer.byteLength(events()[1]!.eventId)).toBeLessThanOrEqual(512); + expect(hasHostedCleanupDebt(replacement)).toBe(false); + await expect( + oweHostedTurnCleanup(replacement, promptId, f.broker), + ).rejects.toThrow('already released'); + } finally { + await replacement.close(); + } + }, +); + +it.each([undefined, 'already loaded'])( + 'refreshes saved-batch context only when missing (%s)', + async (context) => { + const f = await fixture(); + try { + vi.spyOn(f.broker, 'acquire').mockResolvedValue(); + const fetchWorkspaceContext = vi.fn().mockResolvedValue(undefined); + const receiver = { + session: f.session, + broker: f.broker, + warmed: Promise.resolve(), + promptId: f.promptId, + declarations: vi.fn().mockResolvedValue([]), + executeNative: vi.fn().mockResolvedValue([]), + fetchWorkspaceContext, + context: { read: () => context, write: vi.fn() }, + }; + await Reflect.apply( + HostedWorkspaceToolTurn.prototype.resumeSavedNativeBatch, + receiver, + [ + { + plan: { + promptId: f.promptId, + calls: [], + runtime: { runtimeSessionId: f.promptId, ...f.broker.runtime }, + round: 1, + model: 'fixture', + }, + inputs: [], + parts: [], + }, + 0, + new AbortController().signal, + ], + ); + expect(fetchWorkspaceContext).toHaveBeenCalledTimes( + context === undefined ? 1 : 0, + ); + expect(receiver.declarations).toHaveBeenCalledOnce(); + } finally { + await f.session.close(); + } + }, +); + +it('retains terminal cleanup debt across a failed release and a cold replacement', async () => { + const f = await fixture(); + await oweHostedTurnCleanup(f.session, f.promptId, f.broker); + const release = vi + .spyOn(HostedWorkspaceBroker.prototype, 'release') + .mockRejectedValueOnce(new Error('lost release reply')) + .mockResolvedValue(); + await reconcileHostedTurnCleanup(f.session, f.brokerOptions); + expect(release).not.toHaveBeenCalled(); + await f.terminal(); + await expect( + reconcileHostedTurnCleanup(f.session, f.brokerOptions), + ).rejects.toThrow('lost release reply'); + expect(hasHostedCleanupDebt(f.session)).toBe(true); + await f.session.close(); + const replacement = await f.reopen(); + try { + await reconcileHostedTurnCleanup(replacement, f.brokerOptions); + expect(release).toHaveBeenLastCalledWith({ + bindingId: 'original-binding', + generation: '7', + }); + expect(hasHostedCleanupDebt(replacement)).toBe(false); + await reconcileHostedTurnCleanup(replacement, f.brokerOptions); + expect(release).toHaveBeenCalledTimes(2); + expect( + replacement.authority + .eventsInSequenceRange(1, replacement.authority.committedSequence) + .filter((event) => event.kind === 'turn.settled'), + ).toHaveLength(1); + await expect( + oweHostedTurnCleanup(replacement, f.promptId, f.broker), + ).rejects.toThrow('already released'); + } finally { + await replacement.close(); + } +}); + +it('releases recovered file-history work only after its durable Turn terminal', async () => { + const f = await fixture(); + try { + vi.spyOn(HostedWorkspaceBroker.prototype, 'warm').mockImplementation( + async function (this: HostedWorkspaceBroker) { + this.runtime = f.broker.runtime; + }, + ); + vi.spyOn(HostedWorkspaceBroker.prototype, 'acquire').mockResolvedValue(); + let crashSnapshot = true; + vi.spyOn(HostedWorkspaceBroker.prototype, 'fileHistory').mockImplementation( + async (operation) => { + if (operation.action === 'snapshot' && crashSnapshot) { + crashSnapshot = false; + throw new Error('crash before history marker clears'); + } + return { + ownerSessionId: + f.session.authority.sessionHeader.sessionKey.sessionId, + snapshots: [], + files: {}, + }; + }, + ); + vi.spyOn(HostedWorkspaceBroker.prototype, 'prepare').mockResolvedValue( + 'original-execution', + ); + const execute = vi + .spyOn(HostedWorkspaceBroker.prototype, 'execute') + .mockResolvedValue({ + executionStatus: 'success', + responseParts: [{ text: 'written once' }], + }); + vi.spyOn(HostedWorkspaceBroker.prototype, 'cancel').mockResolvedValue(); + const release = vi + .spyOn(HostedWorkspaceBroker.prototype, 'release') + .mockImplementation(async () => { + expect( + f.session.authority + .eventsInSequenceRange(1, f.session.authority.committedSequence) + .filter((event) => event.kind === 'turn.settled'), + ).toHaveLength(1); + }); + const harness = createManagedHarnessHandle(f.session); + const commit = async ( + type: 'assistant' | 'tool_result', + parts: Part[], + model: string, + identity?: { uuid: string; timestamp: string }, + ) => { + const uuid = identity?.uuid ?? randomUUID(); + await f.session.sink.write({ + uuid, + parentUuid: null, + sessionId: f.session.authority.sessionHeader.sessionKey.sessionId, + timestamp: identity?.timestamp ?? new Date().toISOString(), + type, + model, + cwd: '/fixture', + version: 'test', + daemonPromptId: f.promptId, + message: { role: type === 'assistant' ? 'model' : 'user', parts }, + }); + return uuid; + }; + const original = new HostedWorkspaceToolTurn( + f.brokerOptions, + f.session, + harness, + f.promptId, + commit, + () => true, + undefined, + undefined, + { + settings: { mode: 'yolo', timeoutMs: 15 }, + waiters: new HostedApprovalWaiters(), + }, + ); + const call = { + name: 'write_file', + callId: 'original-call', + args: { file_path: 'original.txt', content: 'written once' }, + isClientInitiated: false, + prompt_id: f.promptId, + }; + await expect( + original.execute( + [call], + [ + { + functionCall: { id: call.callId, name: call.name, args: call.args }, + }, + ], + 'fixture', + new AbortController().signal, + ), + ).rejects.toThrow('Hosted tool turn requires recovery'); + const authorization = await f.session.authority.harnessRunAuthorization(); + expect(authorization.status).toBe('runnable'); + if (authorization.status !== 'runnable') throw new Error('Not runnable'); + expect(authorization.checkpoint.continuation.phase).toBe('results_ready'); + const replacement = new HostedWorkspaceToolTurn( + f.brokerOptions, + f.session, + harness, + f.promptId, + async () => { + throw new Error('Recovery must not reexecute tools'); + }, + () => true, + ); + await replacement.resumeCommittedResults(new AbortController().signal); + await replacement.consumeResults(); + await replacement.finish(); + expect(release).not.toHaveBeenCalled(); + expect(hasHostedCleanupDebt(f.session)).toBe(true); + await f.terminal(); + await replacement.cleanup(); + expect(release).toHaveBeenCalledOnce(); + expect(release).toHaveBeenCalledWith({ + bindingId: 'original-binding', + generation: '7', + }); + expect(execute).toHaveBeenCalledOnce(); + expect(hasHostedCleanupDebt(f.session)).toBe(false); + } finally { + await f.session.close(); + } +}); + +const prepared: LlmPreparedRequest = { + request: { + model: 'fixture', + contents: [{ role: 'user', parts: [{ text: 'original' }] }], + config: { temperature: 0.3 }, + }, + history: [{ role: 'user', parts: [{ text: 'original' }] }], + completedToolCallIds: [], + routeSelector: 'openai:fixture', + providerPin: 'a'.repeat(64), + promptTokensForClamp: 12, +}; + +it.each( + [false, true].flatMap((finished) => + [undefined, false, true].map((allowPersistentRetry) => ({ + finished, + allowPersistentRetry, + })), + ), +)( + 'restores an unfinished request even when stream Finished was recorded (%s)', + async ({ finished, allowPersistentRetry }) => { + const f = await fixture(); + const request = createHostedModelRequest( + f.session, + f.promptId, + { + ...prepared, + ...(allowPersistentRetry === undefined ? {} : { allowPersistentRetry }), + }, + 0, + false, + undefined, + { bootId: 'source-boot', eventEpoch: 'source-epoch' }, + ); + await new ManagedHookActivationController(f.session).runTurn( + f.promptId, + async (scope) => { + const finish = await scope.beginMainAttempt( + 'fixture', + await publishHostedModelRequest(f.session, request), + ); + if (finished) await finish(true, {}); + }, + ); + await f.session.close(); + const replacement = await f.reopen(); + try { + expect(await findHostedModelRequest(replacement, f.promptId)).toEqual( + request, + ); + await createManagedHarnessHandle(replacement).adoptPreparedContinuation( + f.promptId, + true, + ); + expect( + (await replacement.authority.harnessRunAuthorization()).status, + ).toBe('runnable'); + } finally { + await replacement.close(); + } + }, +); + +it('does not reuse an older snapshot after a newer unsupported attempt', async () => { + const f = await fixture(); + try { + const request = createHostedModelRequest( + f.session, + f.promptId, + prepared, + 0, + false, + undefined, + { bootId: 'source-boot', eventEpoch: 'source-epoch' }, + ); + await new ManagedHookActivationController(f.session).runTurn( + f.promptId, + async (scope) => { + await scope.beginMainAttempt( + 'fixture', + await publishHostedModelRequest(f.session, request), + ); + await scope.beginMainAttempt('fixture'); + }, + ); + expect(await findHostedModelRequest(f.session, f.promptId)).toBeUndefined(); + expect( + await publishHostedModelRequest(f.session, { + ...request, + workspaceContext: 'x'.repeat(64 * 1024), + }), + ).toBeUndefined(); + } finally { + await f.session.close(); + } +}); + +it('rejects saved authentication fields before attempting inference', async () => { + const f = await fixture(); + try { + const request = createHostedModelRequest( + f.session, + f.promptId, + { + ...prepared, + request: { + ...prepared.request, + config: { + ...prepared.request.config, + httpOptions: { headers: { Authorization: 'fixture-secret' } }, + } as LlmPreparedRequest['request']['config'], + }, + }, + 0, + false, + undefined, + { bootId: 'source-boot', eventEpoch: 'source-epoch' }, + ); + await new ManagedHookActivationController(f.session).runTurn( + f.promptId, + async (scope) => { + await scope.beginMainAttempt( + 'fixture', + await publishHostedModelRequest(f.session, request), + ); + }, + ); + await expect(findHostedModelRequest(f.session, f.promptId)).rejects.toThrow( + 'Invalid saved model request', + ); + } finally { + await f.session.close(); + } +}); + +it('retracts only the original streamed message after replacement, once', async () => { + const f = await fixture(); + const prior = new HostedTextDeltaStream(f.session, f.promptId); + await prior.delta('committed earlier round'); + const priorId = prior.takeMessageId(); + const orphan = new HostedTextDeltaStream(f.session, f.promptId); + await orphan.delta('orphan one'); + await orphan.delta('orphan two'); + const id = orphan.takeMessageId()!; + const source = { + sourceActivation: f.session.activation, + sourceBootId: 'old-boot', + sourceEventEpoch: 'old-epoch', + }; + const oldDeltas = f.session.authority + .eventsInSequenceRange(1, f.session.authority.committedSequence) + .filter( + (event) => + event.kind === 'message.delta' && event.payload['messageId'] === id, + ); + await f.session.close(); + const replacement = await f.reopen(); + try { + const stream = new HostedTextDeltaStream(replacement, f.promptId); + await stream.retractOriginal(id, source); + await stream.retractOriginal(id, source); + const retracts = replacement.authority + .eventsInSequenceRange(1, replacement.authority.committedSequence) + .filter((event) => event.kind === 'message.retracted'); + expect(retracts).toHaveLength(1); + expect(retracts[0].payload).toEqual({ + messageId: id, + turnId: f.promptId, + fromSequence: oldDeltas[0].sequence, + throughSequence: oldDeltas.at(-1)!.sequence, + sourceBootId: 'old-boot', + sourceEventEpoch: 'old-epoch', + }); + expect(retracts[0].payload['messageId']).not.toBe(priorId); + stream.bindMessageId(randomUUID()); + await stream.delta('surviving answer'); + } finally { + await replacement.close(); + } +}); + +it.each([true, false, 'image-history'])( + 'recognizes committed final output as terminal-only compensation (saved request=%s)', + async (saved) => { + const f = await fixture(); + try { + const request = createHostedModelRequest( + f.session, + f.promptId, + saved === 'image-history' + ? { + ...prepared, + history: [ + { + role: 'user', + parts: [ + { text: '[Image #0123456789ab: image/png, 3 bytes]' }, + ], + }, + ], + } + : prepared, + 0, + false, + undefined, + { bootId: 'source-boot', eventEpoch: 'source-epoch' }, + ); + await new ManagedHookActivationController(f.session).runTurn( + f.promptId, + async (scope) => { + const finish = await scope.beginMainAttempt( + 'fixture', + saved + ? await publishHostedModelRequest(f.session, request) + : undefined, + ); + await finish(true, {}); + }, + ); + if (saved === 'image-history') + expect( + await findHostedModelRequest(f.session, f.promptId), + ).toBeUndefined(); + await f.session.sink.write({ + uuid: request.messageId, + parentUuid: null, + sessionId: f.session.authority.sessionHeader.sessionKey.sessionId, + timestamp: new Date().toISOString(), + type: 'assistant', + daemonPromptId: f.promptId, + model: 'fixture', + cwd: '/fixture', + version: 'test', + message: { role: 'model', parts: [{ text: 'already committed' }] }, + }); + expect( + await findHostedModelRequest(f.session, f.promptId), + ).toBeUndefined(); + expect( + await findHostedCommittedModelOutput(f.session, f.promptId), + ).toEqual(saved ? request : { promptId: f.promptId }); + await f.terminal(); + expect( + f.session.authority + .eventsInSequenceRange(1, f.session.authority.committedSequence) + .filter( + (event) => + event.kind === 'model.attempt' && + event.payload['state'] === 'started', + ), + ).toHaveLength(1); + } finally { + await f.session.close(); + } + }, +); diff --git a/packages/cli/src/serve/hosted-runtime-recovery.test.ts b/packages/cli/src/serve/hosted-runtime-recovery.test.ts index c9d18c29d8..12e4b43442 100644 --- a/packages/cli/src/serve/hosted-runtime-recovery.test.ts +++ b/packages/cli/src/serve/hosted-runtime-recovery.test.ts @@ -33,6 +33,7 @@ import { import { HARNESS_MODEL_START_PHASES } from '@qwen-code/qwen-code-core/managed-runtime/managed-harness-checkpoint.js'; import { HostedWorkspaceBroker } from './hosted-workspace-broker.js'; import { hostedRuntimeSessionId } from './hosted-workspace-tool-turn.js'; +import { HOSTED_TOOL_APPROVAL_POLICY } from './hosted-tool-approval.js'; import { HTTP_MANAGED_SESSION_STORE_CONTRACT } from '@qwen-code/qwen-code-core/managed-runtime/http-managed-session-store.js'; const SESSION_ID = '22222222-2222-4222-8222-222222222222'; @@ -496,7 +497,7 @@ describe('recoverHostedRuntimeTurn', () => { kind: 'execute', source: 'tool_call', optionsRef: await session.resources.publish( - 'managed-approval', + 'managed-action-options', Buffer.from( JSON.stringify({ v: 1, @@ -504,11 +505,14 @@ describe('recoverHostedRuntimeTurn', () => { turnId: PROMPT_ID, functionCallId: 'call-1', toolName: 'write_file', - policyRevision: 'pol-1', + policyRevision: HOSTED_TOOL_APPROVAL_POLICY, inputRevision: 1, createdAt: expiresAt - 20_000, expiresAt, - options: [{ id: 'allow' }, { id: 'deny' }], + options: [ + { id: 'allow', label: 'Allow once' }, + { id: 'deny', label: 'Deny' }, + ], }), ), ), diff --git a/packages/cli/src/serve/hosted-runtime-recovery.ts b/packages/cli/src/serve/hosted-runtime-recovery.ts index 8968fd81ef..076e6e8379 100644 --- a/packages/cli/src/serve/hosted-runtime-recovery.ts +++ b/packages/cli/src/serve/hosted-runtime-recovery.ts @@ -95,13 +95,13 @@ export interface HostedRuntimeRecoveryReport { executions: HostedRuntimeRecoveryExecution[]; } -export interface HostedRecoveryTurn { +export type HostedRecoveryTurn = { promptId: string; report: HostedRuntimeRecoveryReport; - /** Whether the recovery acquired the Runtime Session, which a later - * continue/cancel must release. */ - acquiredRuntime: boolean; -} +} & ( + | { acquiredRuntime: false } + | { acquiredRuntime: true; runtimeSessionId: string } +); /** * Why a takeover load cannot continue the parked Turn. Every reason is a @@ -1660,7 +1660,12 @@ export async function recoverHostedRuntimeTurn(input: { }); return recovered({ promptId, - acquiredRuntime, + ...(acquiredRuntime + ? { + acquiredRuntime: true as const, + runtimeSessionId: broker.runtimeSessionId, + } + : { acquiredRuntime: false as const }), report: { phase: finalCheckpoint.continuation.phase === 'results_ready' diff --git a/packages/cli/src/serve/hosted-text-deltas.ts b/packages/cli/src/serve/hosted-text-deltas.ts index 026853956e..e9da7ef7f9 100644 --- a/packages/cli/src/serve/hosted-text-deltas.ts +++ b/packages/cli/src/serve/hosted-text-deltas.ts @@ -27,6 +27,88 @@ export class HostedTextDeltaStream { private readonly turnId: string, ) {} + bindMessageId(messageId: string): void { + if (this.firstSequence !== undefined) + throw new Error('Stream message still owns published deltas.'); + this.messageId = messageId; + this.ordinal = 0; + } + + async retractOriginal( + messageId: string, + source: { + sourceBootId: string; + sourceEventEpoch: string; + sourceActivation: ManagedSession['activation']; + }, + ): Promise { + const authority = this.session.authority; + const events = authority.eventsInSequenceRange( + 1, + authority.committedSequence, + ); + if ( + events.some( + (event) => + event.kind === 'message.retracted' && + event.payload['messageId'] === messageId, + ) + ) + return; + const deltas = events.filter( + (event) => + event.kind === 'message.delta' && + event.payload['messageId'] === messageId && + event.payload['turnId'] === this.turnId, + ); + if (!deltas.length) return; + if ( + deltas.some( + (event) => + event.subject?.type !== 'activation' || + event.subject.activationId !== source.sourceActivation.activationId || + event.subject.epoch !== source.sourceActivation.epoch, + ) + ) + throw new Error('Original stream owner conflicts.'); + const fromSequence = deltas[0].sequence; + const throughSequence = deltas.at(-1)!.sequence; + const activation = this.session.activation; + const commandId = `assistant-retract:${this.turnId}:${messageId}`; + await authority.appendExecutionEvent( + { + operation: 'assistantRetract', + commandId, + sessionKey: authority.sessionHeader.sessionKey, + contentDigest: createHash('sha256') + .update(`${messageId}:${fromSequence}:${throughSequence}`) + .digest('hex'), + }, + (sequence) => ({ + v: 1, + sequence, + eventId: commandId, + sessionKey: authority.sessionHeader.sessionKey, + kind: 'message.retracted', + occurredAt: Date.now(), + subject: { + type: 'activation', + scopeId: activation.activationId, + ...activation, + }, + payload: { + messageId, + turnId: this.turnId, + fromSequence, + throughSequence, + sourceBootId: source.sourceBootId, + sourceEventEpoch: source.sourceEventEpoch, + }, + }), + { class: 'harness', activation }, + ); + } + /** The messageId the in-flight assistant message must commit under. */ takeMessageId(): string | undefined { const id = this.messageId; diff --git a/packages/cli/src/serve/hosted-tool-approval.ts b/packages/cli/src/serve/hosted-tool-approval.ts index b09eeb4e2d..dd475af808 100644 --- a/packages/cli/src/serve/hosted-tool-approval.ts +++ b/packages/cli/src/serve/hosted-tool-approval.ts @@ -5,8 +5,15 @@ */ import { createHash } from 'node:crypto'; +import { isDeepStrictEqual } from 'node:util'; import type { ManagedSession } from '@qwen-code/qwen-code-core/managed-runtime/managed-session-assembly.js'; -import type { ManagedSessionDurableRef } from '@qwen-code/qwen-code-core/managed-runtime/managed-session-records.js'; +import { + type ManagedSessionDurableRef, + assertManagedSessionDurableRef, + assertManagedSessionStableId, + parseManagedSessionRecordJson, + type ManagedSessionJsonValue, +} from '@qwen-code/qwen-code-core/managed-runtime/managed-session-records.js'; import { ManagedSessionConflictError, type ManagedSessionAction, @@ -134,8 +141,82 @@ export type HostedActionOptions = HostedActionOptionsBase & ( | { readonly v: 1 } | { readonly v: 2; readonly inputRef: ManagedSessionDurableRef } + | { + readonly v: 3; + readonly inputRef?: ManagedSessionDurableRef; + readonly continuationRef: ManagedSessionDurableRef; + } ); +export function parseHostedActionOptions(bytes: Buffer): HostedActionOptions { + const value = parseManagedSessionRecordJson( + bytes.toString('utf8'), + 64 * 1024, + ); + if (!value || typeof value !== 'object' || Array.isArray(value)) + throw new Error('Invalid Hosted Action options.'); + const options = value as Record; + const fields = [ + 'v', + 'requestId', + 'turnId', + 'functionCallId', + 'toolName', + 'policyRevision', + 'inputRevision', + 'createdAt', + 'expiresAt', + 'options', + ]; + if (options['v'] === 2 || (options['v'] === 3 && options['inputRef'])) + fields.push('inputRef'); + if (options['v'] === 3) fields.push('continuationRef'); + if ( + ![1, 2, 3].includes(options['v'] as number) || + Object.keys(options).length !== fields.length || + fields.some((field) => !Object.hasOwn(options, field)) || + !Number.isSafeInteger(options['inputRevision']) || + (options['inputRevision'] as number) < 1 || + options['policyRevision'] !== HOSTED_TOOL_APPROVAL_POLICY || + !Number.isSafeInteger(options['createdAt']) || + !Number.isSafeInteger(options['expiresAt']) || + (options['createdAt'] as number) < 0 || + (options['expiresAt'] as number) <= (options['createdAt'] as number) || + !Array.isArray(options['options']) || + options['options'].length !== 2 || + !isDeepStrictEqual( + options['options'].map((option) => + option && typeof option === 'object' && !Array.isArray(option) + ? option['id'] + : null, + ), + ['allow', 'deny'], + ) || + options['options'].some( + (option) => + !option || + typeof option !== 'object' || + Array.isArray(option) || + Object.keys(option).length !== 2 || + typeof option['label'] !== 'string' || + !option['label'], + ) + ) + throw new Error('Invalid Hosted Action options.'); + for (const field of ['requestId', 'turnId', 'functionCallId', 'toolName']) + assertManagedSessionStableId(options[field], field); + for (const [field, kind] of [ + ['inputRef', 'managed-tool-input'], + ['continuationRef', 'hosted-approval-continuation'], + ]) { + if (!Object.hasOwn(options, field)) continue; + const ref = assertManagedSessionDurableRef(options[field], field); + if (ref.kind !== kind || ref.schemaVersion !== 1) + throw new Error('Invalid Hosted Action reference.'); + } + return options as unknown as HostedActionOptions; +} + function decisionBytes( optionId: string, inputRevision: number, @@ -146,6 +227,19 @@ function decisionBytes( ); } +export function hostedActionDenied( + action: ManagedSessionAction, + policyRevision: string, +): boolean { + return ( + action.state === 'decided' && + action.decisionRef?.digest === + createHash('sha256') + .update(decisionBytes('deny', action.inputRevision, policyRevision)) + .digest('hex') + ); +} + /** * Whether a decided Action chose `allow` under the policy revision its options * recorded. Decision bytes are deterministic, so their recorded digest says @@ -245,9 +339,15 @@ export async function readHostedActionOptions( ): Promise { if (action.optionsRef === null) throw new Error(`Action ${action.requestId} has no options.`); - return JSON.parse( - (await session.resources.read(action.optionsRef)).toString('utf8'), - ) as HostedActionOptions; + const options = parseHostedActionOptions( + await session.resources.read(action.optionsRef), + ); + if ( + options.requestId !== action.requestId || + options.inputRevision !== action.inputRevision + ) + throw new Error('Hosted Action options do not match the Action.'); + return options; } /** The closed vocabulary the Hosted refusals carry in 'reason'. */ diff --git a/packages/cli/src/serve/hosted-turn-cleanup.ts b/packages/cli/src/serve/hosted-turn-cleanup.ts new file mode 100644 index 0000000000..4045c04ae6 --- /dev/null +++ b/packages/cli/src/serve/hosted-turn-cleanup.ts @@ -0,0 +1,234 @@ +/** + * @license + * Copyright 2026 Qwen Team + * SPDX-License-Identifier: Apache-2.0 + */ +import { createHash } from 'node:crypto'; +import { isDeepStrictEqual } from 'node:util'; +import type { ManagedSession } from '@qwen-code/qwen-code-core/managed-runtime/managed-session-assembly.js'; +import { parseHostedRecoveryResource } from '@qwen-code/qwen-code-core/managed-runtime/hosted-recovery-records.js'; +import { + assertManagedSessionDurableRef, + assertManagedSessionStableId, + MANAGED_SESSION_LIMITS, + type ManagedSessionDurableRef, + type ManagedSessionJsonValue, +} from '@qwen-code/qwen-code-core/managed-runtime/managed-session-records.js'; +import { + HostedWorkspaceBroker, + hostedRuntimeSessionId, + type HostedWorkspaceBrokerOptions, +} from './hosted-workspace-broker.js'; +import { readHostedFileHistory } from './hosted-file-history.js'; + +interface CleanupDescriptor { + v: 1; + sessionKey: ManagedSession['authority']['sessionHeader']['sessionKey']; + promptId: string; + runtimeSessionId: string; + bindingId: string; + generation: string; + workspaceGeneration: string; + fileHistoryTurnId: string; +} + +async function commitCleanup( + session: ManagedSession, + cleanupId: string, + descriptorRef: ManagedSessionDurableRef, + state: 'owed' | 'confirmed', +): Promise { + const authority = session.authority; + const activation = session.activation; + const rawEventId = `${cleanupId}:${state}`; + const eventId = + Buffer.byteLength(rawEventId) <= MANAGED_SESSION_LIMITS.maxIdBytes + ? rawEventId + : `hosted-cleanup:${createHash('sha256').update(rawEventId).digest('hex')}`; + await authority.appendExecutionEvent( + { + operation: 'hostedCleanup', + commandId: eventId, + sessionKey: authority.sessionHeader.sessionKey, + contentDigest: createHash('sha256') + .update(`${descriptorRef.digest}:${state}`) + .digest('hex'), + }, + (sequence) => ({ + v: 1, + sequence, + eventId, + sessionKey: authority.sessionHeader.sessionKey, + kind: 'hosted.cleanup', + occurredAt: Date.now(), + subject: { + type: 'activation', + scopeId: activation.activationId, + ...activation, + }, + payload: { cleanupId, descriptorRef, state }, + }), + { class: 'harness', activation }, + ); +} + +export async function oweHostedTurnCleanup( + session: ManagedSession, + promptId: string, + broker: HostedWorkspaceBroker, +): Promise { + if (!broker.runtime) return; + const originalId = `hosted-cleanup:${promptId}:${broker.runtimeSessionId}`; + let cleanupId = originalId; + if ( + Buffer.byteLength(`${cleanupId}:confirmed`) > + MANAGED_SESSION_LIMITS.maxIdBytes + ) + cleanupId = `hosted-cleanup:${createHash('sha256').update(cleanupId).digest('hex')}`; + const existing = session.authority + .eventsInSequenceRange(1, session.authority.committedSequence) + .findLast( + (event) => + event.kind === 'hosted.cleanup' && + (event.payload['cleanupId'] === cleanupId || + event.payload['cleanupId'] === originalId), + ); + if (existing) { + if (existing.payload['state'] !== 'owed') + throw new Error('Original Runtime owner was already released.'); + const descriptor = await readDescriptor( + session, + existing.payload['descriptorRef'], + ); + if ( + !isDeepStrictEqual( + { + bindingId: descriptor.bindingId, + generation: descriptor.generation, + workspaceGeneration: descriptor.workspaceGeneration, + }, + broker.runtime, + ) + ) + throw new Error('Cleanup Runtime owner changed.'); + return; + } + const descriptor: CleanupDescriptor = { + v: 1, + sessionKey: session.authority.sessionHeader.sessionKey, + promptId, + runtimeSessionId: broker.runtimeSessionId, + ...broker.runtime, + fileHistoryTurnId: promptId, + }; + const ref = await session.resources.publish( + 'hosted-turn-cleanup', + Buffer.from(JSON.stringify(descriptor)), + ); + await commitCleanup(session, cleanupId, ref, 'owed'); +} + +async function readDescriptor( + session: ManagedSession, + value: unknown, +): Promise { + const ref = assertManagedSessionDurableRef( + value as ManagedSessionJsonValue, + 'cleanup descriptor', + ); + if (ref.kind !== 'hosted-turn-cleanup' || ref.schemaVersion !== 1) + throw new Error('Invalid cleanup resource.'); + const body = parseHostedRecoveryResource( + await session.resources.read(ref), + session.authority.sessionHeader.sessionKey, + ); + if ( + Object.keys(body).length !== 8 || + body['runtimeSessionId'] !== + hostedRuntimeSessionId(body['promptId'] as string) || + body['fileHistoryTurnId'] !== body['promptId'] + ) + throw new Error('Invalid cleanup owner.'); + assertManagedSessionStableId(body['bindingId'], 'cleanup binding'); + for (const field of ['generation', 'workspaceGeneration']) + if ( + typeof body[field] !== 'string' || + !/^[1-9][0-9]{0,18}$/u.test(body[field] as string) || + BigInt(body[field] as string) > 2n ** 63n - 1n + ) + throw new Error('Invalid cleanup generation.'); + return body as unknown as CleanupDescriptor; +} + +export function hasHostedCleanupDebt(session: ManagedSession): boolean { + const states = new Map(); + for (const event of session.authority.eventsInSequenceRange( + 1, + session.authority.committedSequence, + )) + if (event.kind === 'hosted.cleanup') + states.set(event.payload['cleanupId'] as string, event.payload['state']); + return [...states.values()].some((state) => state === 'owed'); +} + +export async function reconcileHostedTurnCleanup( + session: ManagedSession, + options: HostedWorkspaceBrokerOptions, + promptId?: string, + drain = false, +): Promise> { + const events = session.authority.eventsInSequenceRange( + 1, + session.authority.committedSequence, + ); + const latest = new Map(); + const states = new Map(); + for (const event of events) + if (event.kind === 'hosted.cleanup') + latest.set(event.payload['cleanupId'] as string, event); + for (const [cleanupId, event] of latest) { + const descriptor = await readDescriptor( + session, + event.payload['descriptorRef'], + ); + if (promptId && descriptor.promptId !== promptId) continue; + const state = event.payload['state'] as 'owed' | 'confirmed'; + states.set(descriptor.promptId, state); + if (state !== 'owed') continue; + if ( + !events.some( + (item) => + item.kind === 'turn.settled' && + item.payload['turnId'] === descriptor.promptId, + ) + ) + continue; + const history = await readHostedFileHistory(session); + if ( + history?.pendingUndo || + history?.pendingTurn === descriptor.fileHistoryTurnId + ) + throw new Error('Original file history marker is still owed.'); + const broker = new HostedWorkspaceBroker( + options, + descriptor.sessionKey, + descriptor.runtimeSessionId, + ); + await broker.release({ + bindingId: descriptor.bindingId, + generation: descriptor.generation, + ...(drain ? { drain: true } : {}), + }); + await commitCleanup( + session, + cleanupId, + assertManagedSessionDurableRef( + event.payload['descriptorRef'], + 'cleanup descriptor', + ), + 'confirmed', + ); + states.set(descriptor.promptId, 'confirmed'); + } + return states; +} diff --git a/packages/cli/src/serve/hosted-workspace-broker.test.ts b/packages/cli/src/serve/hosted-workspace-broker.test.ts index 7afa5f43a5..a2cf6c23f9 100644 --- a/packages/cli/src/serve/hosted-workspace-broker.test.ts +++ b/packages/cli/src/serve/hosted-workspace-broker.test.ts @@ -57,6 +57,28 @@ async function fixture( ); } +it.each([false, true])( + 'carries the original cleanup fence with drain=%s', + async (drain) => { + const requests: Array> = []; + const broker = await fixture((_path, body) => { + requests.push(body); + return { body: { ...identity, released: true } }; + }); + await broker.release({ + bindingId: 'original-binding', + generation: '7', + ...(drain ? { drain: true } : {}), + }); + expect(requests[0]).toMatchObject({ + runtimeSessionId: 'turn', + cleanupBindingId: 'original-binding', + cleanupGeneration: '7', + }); + expect(requests[0]['drain']).toBe(drain ? true : undefined); + }, +); + it.each(['tenantId', 'workspaceId', 'capabilityDigest'])( 'rejects a mismatched %s at acquisition', async (field) => { diff --git a/packages/cli/src/serve/hosted-workspace-broker.ts b/packages/cli/src/serve/hosted-workspace-broker.ts index 5ae844593b..39dab1f4dd 100644 --- a/packages/cli/src/serve/hosted-workspace-broker.ts +++ b/packages/cli/src/serve/hosted-workspace-broker.ts @@ -12,7 +12,7 @@ import type { RawFileHistoryOperation, HostedFileHistoryState, } from './hosted-file-history-protocol.js'; -import { randomUUID } from 'node:crypto'; +import { createHash, randomUUID } from 'node:crypto'; import { setTimeout as delay } from 'node:timers/promises'; import type { ManagedSessionKey } from '@qwen-code/qwen-code-core/managed-runtime/managed-session-records.js'; import type { @@ -71,6 +71,7 @@ export function isHostedFileHistoryRefusal( export class HostedWorkspaceBroker { private readonly baseUrl: URL; + private leaseAcquired = false; private readonly identity: { harnessSessionId: string; runtimeSessionId: string; @@ -85,6 +86,10 @@ export class HostedWorkspaceBroker { return this.identity.runtimeSessionId; } + get acquired(): boolean { + return this.leaseAcquired; + } + constructor( private readonly options: HostedWorkspaceBrokerOptions, private readonly key: ManagedSessionKey, @@ -139,7 +144,8 @@ export class HostedWorkspaceBroker { } async warm(): Promise { - await this.request('/runtimes:warm', {}); + const response = await this.request('/runtimes:warm', {}); + if (response['runtime'] !== undefined) this.readRuntime(response); } async authorizeLifecycle(): Promise { @@ -160,9 +166,15 @@ export class HostedWorkspaceBroker { } : {}), }); + if (response['acquired'] !== true) + throw new Error('Runtime did not acquire the Session.'); + this.readRuntime(response); + this.leaseAcquired = true; + } + + private readRuntime(response: Record): void { const scope = object(response['scope']); if ( - response['acquired'] !== true || scope['tenantId'] !== this.key.tenantId || scope['workspaceId'] !== this.key.workspaceId || scope['capabilityDigest'] !== WORKSPACE_CAPABILITY_DIGEST @@ -624,16 +636,27 @@ export class HostedWorkspaceBroker { } } - async release(options?: { readonly drain?: boolean }): Promise { + async release( + options?: + | { bindingId: string; generation: string; readonly drain?: boolean } + | { readonly drain?: boolean }, + ): Promise { + const expected = options && 'bindingId' in options ? options : undefined; const response = await this.request( `/tool-sessions/${encodeURIComponent(this.identity.runtimeSessionId)}:release`, - // #13856: the drain arm stops the Session's live background holds - // before releasing; a routine release is refused - // `runtime_session_held` while a run is provably live. - options?.drain === true ? { drain: true } : {}, + { + ...(expected + ? { + cleanupBindingId: expected.bindingId, + cleanupGeneration: expected.generation, + } + : {}), + ...(options?.drain === true ? { drain: true } : {}), + }, ); if (response['released'] !== true) throw new Error('Runtime Session release is unconfirmed.'); + this.leaseAcquired = false; } private async request( @@ -709,3 +732,18 @@ export class HostedWorkspaceBroker { return parsed; } } + +/** + * The Broker admits only path-safe Runtime Session ids, while a wake + * turn's id is an input id (`arun_…:input`, `:notify:`): + * such an id is mapped to a stable path-safe digest instead of being + * refused at acquire. The mapped form is path-safe itself, so layering + * this over an id that was already mapped stays idempotent. + */ +export function hostedRuntimeSessionId(promptId: string): string { + return /^[A-Za-z0-9._-]{1,512}$/.test(promptId) && + promptId !== '.' && + !promptId.includes('..') + ? promptId + : `wake-${createHash('sha256').update(promptId).digest('hex')}`; +} diff --git a/packages/cli/src/serve/hosted-workspace-tool-turn.child-agent.test.ts b/packages/cli/src/serve/hosted-workspace-tool-turn.child-agent.test.ts index d691121b23..ea516c41e4 100644 --- a/packages/cli/src/serve/hosted-workspace-tool-turn.child-agent.test.ts +++ b/packages/cli/src/serve/hosted-workspace-tool-turn.child-agent.test.ts @@ -64,6 +64,7 @@ vi.mock('./hosted-workspace-broker.js', async (importOriginal) => ({ ...(await importOriginal()), HostedWorkspaceBroker: class { readonly runtimeSessionId = 'prompt'; + acquired = false; runtime = { bindingId: 'binding', generation: '1', @@ -71,14 +72,20 @@ vi.mock('./hosted-workspace-broker.js', async (importOriginal) => ({ }; fileHistory = broker.fileHistory; warm = broker.warm; - acquire = broker.acquire; + acquire = async (...args: Parameters) => { + await broker.acquire(...args); + this.acquired = true; + }; prepare = broker.prepare; prepareV3 = broker.prepareV3; execute = broker.execute; executeV3 = broker.executeV3; acknowledgeV3 = broker.acknowledgeV3; cancel = broker.cancel; - release = broker.release; + release = async (...args: Parameters) => { + await broker.release(...args); + this.acquired = false; + }; registerPublisher = broker.registerPublisher; acknowledge = broker.acknowledge; hookControl = broker.hookControl; diff --git a/packages/cli/src/serve/hosted-workspace-tool-turn.ts b/packages/cli/src/serve/hosted-workspace-tool-turn.ts index ca5093fbb3..1a5ad4f8ca 100644 --- a/packages/cli/src/serve/hosted-workspace-tool-turn.ts +++ b/packages/cli/src/serve/hosted-workspace-tool-turn.ts @@ -4,6 +4,11 @@ * SPDX-License-Identifier: Apache-2.0 */ +import { + oweHostedTurnCleanup, + reconcileHostedTurnCleanup, +} from './hosted-turn-cleanup.js'; + import { readHostedFileHistory, commitHostedFileHistory, @@ -64,6 +69,10 @@ import { ManagedSessionStoreTransportError, } from '@qwen-code/qwen-code-core/managed-runtime/http-managed-session-store.js'; import { writeStderrLineSafe } from '../utils/stdioHelpers.js'; +import type { + HostedApprovalContinuation, + RestoredHostedApprovalContinuation, +} from './hosted-approval-continuation.js'; import { InvalidWorkspaceRelativePathError, normalizeWorkspaceRelativePath, @@ -71,6 +80,7 @@ import { import { WORKSPACE_CAPABILITY_DIGEST } from './managed-workspace-activation.js'; import { HostedWorkspaceBroker, + hostedRuntimeSessionId, HostedWorkspaceBrokerRejection, isHostedFileHistoryRefusal, type HostedWorkspaceBrokerOptions, @@ -931,20 +941,7 @@ function physicalToolStatus( return response?.['error'] ? 'error' : 'success'; } -/** - * The Broker admits only path-safe Runtime Session ids, while a wake - * turn's id is an input id (`arun_…:input`, `:notify:`): - * such an id is mapped to a stable path-safe digest instead of being - * refused at acquire. The mapped form is path-safe itself, so layering - * this over an id that was already mapped stays idempotent. - */ -export function hostedRuntimeSessionId(promptId: string): string { - return /^[A-Za-z0-9._-]{1,512}$/.test(promptId) && - promptId !== '.' && - !promptId.includes('..') - ? promptId - : `wake-${createHash('sha256').update(promptId).digest('hex')}`; -} +export { hostedRuntimeSessionId } from './hosted-workspace-broker.js'; export class HostedToolRecoveryRequiredError extends Error { constructor(cause: unknown) { @@ -1073,11 +1070,45 @@ export function fitChildResultInline( throw new Error('Child agent result cannot be recorded inline.'); } +interface HostedPreparedRequest { + call: ToolCallRequestInfo; + validationError?: string; + input: Record; + isShell: boolean; + inputDigest?: string; + mcp: boolean; + payloadJson: string; + inputBytes: Buffer; + digest: string; + argsDigest: string; + publicationId: string | null; + runtimeCallId: string; + background: boolean; + monitoring: boolean; + agent: boolean; + agentBackground: boolean; + team: boolean; + message: boolean; +} + export class HostedWorkspaceToolTurn { + private logicalRound = 0; + private nativeBatch?: { + messageId: string; + model: string; + parts: Part[]; + requests: HostedPreparedRequest[]; + refusals: Array; + inputRefs: Map; + definitionRefs: Map; + actionIds: Map; + uncaptured?: boolean; + }; hookStopReason?: string; private readonly broker: HostedWorkspaceBroker; private readonly warmed: Promise; private acquired = false; + private cleanupOwed = false; private uncertain = false; // Agent calls never write a `tool.intent` (the Broker pipeline they // bypass owns that marker), so completeHookResults needs the admitted @@ -1433,6 +1464,15 @@ export class HostedWorkspaceToolTurn { originalId, ); await original.warm(); + if ( + original === this.broker && + !this.hooks && + !this.mcp && + !isHostedWorkspaceShellProfile(this.profile) + ) { + await oweHostedTurnCleanup(this.session, this.promptId, original); + this.cleanupOwed = original.runtime !== undefined; + } await original.acquire(); const state = await original.fileHistory({ kind: 'raw-file-history', @@ -1475,7 +1515,26 @@ export class HostedWorkspaceToolTurn { for (;;) { try { if (this.hooks && !this.mcp) await this.hooks.acquire(); - else await this.broker.acquire(); + else { + if ( + !this.cleanupOwed && + !this.mcp && + !isHostedWorkspaceShellProfile(this.profile) + ) + await oweHostedTurnCleanup( + this.session, + this.promptId, + this.broker, + ); + if ( + !this.mcp && + !this.hooks && + !isHostedWorkspaceShellProfile(this.profile) && + this.broker.runtime + ) + this.cleanupOwed = true; + await this.broker.acquire(); + } break; } catch (cause) { // A definite busy refusal before claiming storage means another @@ -1837,11 +1896,254 @@ export class HostedWorkspaceToolTurn { this.promptHookRunner = runner; } + setModelRound(round: number): void { + this.logicalRound = round; + } + + async resumeSavedNativeBatch( + saved: RestoredHostedApprovalContinuation, + nextApprovalOrdinal: number, + signal: AbortSignal, + ): Promise { + const plan = saved.plan; + this.unanswered = plan.calls.some( + (item) => + item.actionId && + this.session.authority.action(item.actionId)?.state === 'expired', + ); + if ( + this.hooks || + this.mcp || + this.promptId !== plan.promptId || + this.broker.runtimeSessionId !== plan.runtime.runtimeSessionId + ) + throw new HostedToolRecoveryRequiredError( + 'Unsupported saved batch owner.', + ); + this.uncertain = true; + try { + await waitForTurn(this.warmed, signal); + this.broker.runtime = { + bindingId: plan.runtime.bindingId, + generation: plan.runtime.generation, + workspaceGeneration: plan.runtime.workspaceGeneration, + }; + await oweHostedTurnCleanup(this.session, this.promptId, this.broker); + this.cleanupOwed = true; + await this.broker.acquire({ + runtimeBindingId: plan.runtime.bindingId, + generation: plan.runtime.generation, + }); + this.acquired = true; + if ( + !this.broker.runtime || + this.broker.runtime.workspaceGeneration !== + plan.runtime.workspaceGeneration + ) + throw new Error('Original Workspace generation changed.'); + if (this.context?.read() === undefined) + await this.fetchWorkspaceContext(signal); + const declarations = await this.declarations(signal); + const requests: HostedPreparedRequest[] = []; + for (const [ordinal, item] of plan.calls.entries()) { + const definition = JSON.parse( + (await this.session.resources.read(item.definitionRef)).toString( + 'utf8', + ), + ) as unknown; + if ( + !isDeepStrictEqual( + definition, + declarations.find((tool) => tool.name === item.call.name), + ) + ) + throw new Error('Original native tool definition changed.'); + const input = saved.inputs[ordinal]; + if (item.argsDigest !== `sha256:${managedToolDigest(input.input)}`) + throw new Error('Original native arguments changed.'); + requests.push({ + call: item.call, + input: input.input, + runtimeCallId: item.runtimeCallId, + payloadJson: input.payloadJson, + inputBytes: input.bytes, + digest: item.requestDigest, + argsDigest: item.argsDigest, + isShell: false, + mcp: false, + publicationId: null, + background: false, + monitoring: false, + agent: false, + agentBackground: true, + team: false, + message: false, + }); + } + this.logicalRound = plan.round; + return await this.executeNative( + plan.calls.map((item) => item.call), + saved.parts, + plan.model, + signal, + { saved, requests, nextApprovalOrdinal }, + ); + } catch (cause) { + throw new HostedToolRecoveryRequiredError(cause); + } + } + + private async captureNativeBatch( + stage: 'approval' | 'final', + actionId: string | null, + approvalOrdinal: number, + ): Promise { + const batch = this.nativeBatch; + const runtime = this.broker.runtime; + if ( + !batch || + batch.uncaptured || + !runtime || + isHostedWorkspaceShellProfile(this.profile) || + this.hooks || + this.mcp || + batch.requests.some( + (request) => + !['read_file', 'write_file', 'edit', 'glob'].includes( + request.call.name, + ), + ) + ) + return undefined; + const authority = this.session.authority; + const header = authority.sessionHeader; + const assistant = authority + .eventsInSequenceRange(1, authority.committedSequence) + .find( + (event) => + event.kind === 'message.committed' && + event.payload['messageId'] === batch.messageId, + ); + if (!assistant) throw new Error('Original assistant batch is not durable.'); + const calls = []; + for (const [ordinal, request] of batch.requests.entries()) { + let inputRef = batch.inputRefs.get(ordinal); + if (!inputRef) { + inputRef = await this.session.resources.publish( + 'managed-tool-input', + request.inputBytes, + ); + batch.inputRefs.set(ordinal, inputRef); + } + let definitionRef = batch.definitionRefs.get(ordinal); + if (!definitionRef) { + definitionRef = await this.session.resources.publish( + 'managed-tool-definition', + Buffer.from( + JSON.stringify( + this.advertised!.find((tool) => tool.name === request.call.name), + ), + ), + ); + batch.definitionRefs.set(ordinal, definitionRef); + } + calls.push({ + call: request.call, + runtimeCallId: request.runtimeCallId, + inputRef, + definitionRef, + requestDigest: request.digest, + argsDigest: request.argsDigest, + prepareKey: `${this.broker.runtimeSessionId}:${request.runtimeCallId}`, + prepareReference: { + sessionId: this.broker.runtimeSessionId, + promptId: this.promptId, + callId: request.runtimeCallId, + argsDigest: request.digest, + }, + partIndex: batch.parts.findIndex( + (part) => part.functionCall?.id === request.call.callId, + ), + refusal: batch.refusals[ordinal] ?? null, + actionId: batch.actionIds.get(ordinal) ?? null, + }); + } + const plan: HostedApprovalContinuation = { + v: 1, + sessionKey: header.sessionKey, + promptId: this.promptId, + definitionRef: header.definitionRef, + rootSnapshotRef: header.rootSnapshotRef, + sourceActivation: this.session.activation, + runtime: { runtimeSessionId: this.broker.runtimeSessionId, ...runtime }, + batchId: batch.messageId, + assistantRef: assistant.payload[ + 'contentRef' + ] as unknown as ManagedSessionDurableRef, + model: batch.model, + round: this.logicalRound, + stage, + approvalOrdinal, + actionId, + calls, + }; + const planBytes = Buffer.from(JSON.stringify(plan)); + if ( + planBytes.byteLength > + HTTP_MANAGED_SESSION_STORE_CONTRACT.maxInlineResourceBytes + ) + return undefined; + const planRef = await this.session.resources.publish( + 'hosted-approval-continuation', + planBytes, + ); + if (stage === 'final') { + const activation = this.session.activation; + const planRevision = + authority + .eventsInSequenceRange(1, authority.committedSequence) + .filter( + (event) => + event.kind === 'hosted.batch.planned' && + event.payload['batchId'] === batch.messageId, + ).length + 1; + await authority.appendExecutionEvent( + { + operation: 'hostedBatchPlan', + commandId: `hosted-batch:${batch.messageId}:${planRevision}`, + sessionKey: header.sessionKey, + contentDigest: planRef.digest, + }, + (sequence) => ({ + v: 1, + sequence, + eventId: `hosted-batch:${batch.messageId}:${planRevision}`, + sessionKey: header.sessionKey, + kind: 'hosted.batch.planned', + occurredAt: Date.now(), + subject: { + type: 'activation', + scopeId: activation.activationId, + ...activation, + }, + payload: { batchId: batch.messageId, planRevision, planRef }, + }), + { class: 'harness', activation }, + ); + } + return planRef; + } + private async executeNative( calls: ToolCallRequestInfo[], parts: Part[], model: string, signal: AbortSignal, + restored?: { + saved: RestoredHostedApprovalContinuation; + requests: HostedPreparedRequest[]; + nextApprovalOrdinal: number; + }, ): Promise { signal.throwIfAborted(); if (this.mcp) await waitForTurn(this.warmed, signal); @@ -1889,7 +2191,9 @@ export class HostedWorkspaceToolTurn { }), ); } - const prepareRequests = (source: ToolCallRequestInfo[]) => { + const prepareRequests = ( + source: ToolCallRequestInfo[], + ): HostedPreparedRequest[] => { const ids = new Set(); return source.map((call) => { const runtimeCallId = randomUUID(); @@ -2298,7 +2602,7 @@ export class HostedWorkspaceToolTurn { }; }); }; - let requests = prepareRequests(calls); + let requests = restored?.requests ?? prepareRequests(calls); if (!this.messageFitsInline('assistant', parts, model)) throw new Error( 'Hosted assistant record exceeds the inline Session Store limit.', @@ -2391,7 +2695,18 @@ export class HostedWorkspaceToolTurn { let renewGrants: (() => Promise) | undefined; let messageId: string; let refusals: Array; - const inputRefs = new Map(); + const inputRefs = new Map( + restored?.saved.plan.calls.map((item, ordinal) => [ + ordinal, + item.inputRef, + ]), + ); + const definitionRefs = new Map( + restored?.saved.plan.calls.map((item, ordinal) => [ + ordinal, + item.definitionRef, + ]), + ); try { this.uncertain = true; if ( @@ -2432,8 +2747,39 @@ export class HostedWorkspaceToolTurn { await this.publisher.start(), ); } - messageId = await this.commit('assistant', parts, model); - refusals = await this.approve(requests, messageId, inputRefs, signal); + messageId = + restored?.saved.plan.batchId ?? + (await this.commit('assistant', parts, model)); + this.nativeBatch = { + messageId, + model, + parts, + requests, + refusals: + restored?.saved.plan.calls.map((item) => item.refusal ?? undefined) ?? + requests.map((request) => + this.hookPermission.get(request.call.callId) === 'deny' + ? 'PermissionRequest Hook denied the call.' + : undefined, + ), + inputRefs, + definitionRefs, + actionIds: new Map( + restored?.saved.plan.calls.flatMap((item, index) => + item.actionId ? [[index, item.actionId] as const] : [], + ), + ), + }; + refusals = + restored?.saved.plan.stage === 'final' + ? this.nativeBatch.refusals + : await this.approve( + requests, + messageId, + inputRefs, + signal, + restored?.nextApprovalOrdinal ?? 0, + ); if (this.hooks) { const updated = requests.map((request) => request.call); const askAgain = new Set(); @@ -2532,6 +2878,14 @@ export class HostedWorkspaceToolTurn { } } } + this.nativeBatch.requests = requests; + this.nativeBatch.refusals = refusals; + if (restored?.saved.plan.stage !== 'final') + await this.captureNativeBatch( + 'final', + null, + Math.max(0, requests.length - 1), + ); } catch (cause) { throw new HostedToolRecoveryRequiredError(cause); } @@ -2678,14 +3032,16 @@ export class HostedWorkspaceToolTurn { continue; } reserved.set(ordinal, executionCallId); - const toolDefinitionRef = await this.session.resources.publish( - 'managed-tool-definition', - Buffer.from( - JSON.stringify( - declarations.find((tool) => tool.name === request.call.name), + const toolDefinitionRef = + definitionRefs.get(ordinal) ?? + (await this.session.resources.publish( + 'managed-tool-definition', + Buffer.from( + JSON.stringify( + declarations.find((tool) => tool.name === request.call.name), + ), ), - ), - ); + )); const authority = this.session.authority; const activation = this.session.activation; const argsRef = @@ -2695,36 +3051,58 @@ export class HostedWorkspaceToolTurn { Buffer.from(JSON.stringify(request.input)), ) : routeRef; - const intent = await authority.appendExecutionEvent( - { - operation: 'toolIntent', - commandId: `tool-intent:${executionCallId}`, - sessionKey: authority.sessionHeader.sessionKey, - contentDigest: routeRef.digest, - }, - (sequence) => ({ - v: 1, - sequence, - eventId: `tool-intent:${executionCallId}`, - sessionKey: authority.sessionHeader.sessionKey, - kind: 'tool.intent', - occurredAt: Date.now(), - subject: { - type: 'activation', - scopeId: activation.activationId, - ...activation, - }, - payload: { - executionCallId, - batchId: messageId, - ordinal, + const existingIntent = + restored && + authority + .eventsInSequenceRange(1, authority.committedSequence) + .find( + (event) => + event.kind === 'tool.intent' && + event.payload['executionCallId'] === executionCallId, + ); + if ( + existingIntent && + (existingIntent.payload['batchId'] !== messageId || + existingIntent.payload['ordinal'] !== ordinal || + !isDeepStrictEqual(existingIntent.payload['argsRef'], argsRef) || + !isDeepStrictEqual( + existingIntent.payload['toolDefinitionRef'], toolDefinitionRef, - argsRef, - outcomeSource: 'runtime', - }, - }), - { class: 'harness', activation }, - ); + )) + ) + throw new Error('Original prepared intent conflicts.'); + const intent = existingIntent + ? { lastSequence: existingIntent.sequence } + : await authority.appendExecutionEvent( + { + operation: 'toolIntent', + commandId: `tool-intent:${executionCallId}`, + sessionKey: authority.sessionHeader.sessionKey, + contentDigest: routeRef.digest, + }, + (sequence) => ({ + v: 1, + sequence, + eventId: `tool-intent:${executionCallId}`, + sessionKey: authority.sessionHeader.sessionKey, + kind: 'tool.intent', + occurredAt: Date.now(), + subject: { + type: 'activation', + scopeId: activation.activationId, + ...activation, + }, + payload: { + executionCallId, + batchId: messageId, + ordinal, + toolDefinitionRef, + argsRef, + outcomeSource: 'runtime', + }, + }), + { class: 'harness', activation }, + ); if (prepared) { shellBindings.set(executionCallId, { publicationId: request.publicationId!, @@ -4752,7 +5130,7 @@ export class HostedWorkspaceToolTurn { ref = (await owner.request( `/publications/${publicationId}/admissions/prepare`, outcome, - )) as ManagedSessionDurableRef; + )) as unknown as ManagedSessionDurableRef; break; } catch (error) { const uncertain = @@ -4833,17 +5211,22 @@ export class HostedWorkspaceToolTurn { messageId: string, inputRefs: Map, signal: AbortSignal, + nextOrdinal = 0, ): Promise> { - const refusals: Array = requests.map((request) => - this.hookPermission.get(request.call.callId) === 'deny' - ? 'PermissionRequest Hook denied the call.' - : undefined, - ); + const refusals: Array = + this.nativeBatch?.refusals ?? + requests.map((request) => + this.hookPermission.get(request.call.callId) === 'deny' + ? 'PermissionRequest Hook denied the call.' + : undefined, + ); + if (this.nativeBatch) this.nativeBatch.refusals = refusals; const approval = this.approval; if (!approval) return refusals; const searchProfile = this.searchProfile; let asked = false; for (const [index, request] of requests.entries()) { + if (index < nextOrdinal) continue; if ( refusals[index] || this.hookPermission.get(request.call.callId) === 'allow' || @@ -4884,10 +5267,28 @@ export class HostedWorkspaceToolTurn { const authority = this.session.authority; const requestId = `tool_approval_${randomBytes(16).toString('hex')}`; const createdAt = Date.now(); + const ordinal = + this.nativeBatch?.requests.findIndex( + (request) => request.call.callId === call.callId, + ) ?? -1; + if (ordinal >= 0) this.nativeBatch!.actionIds.set(ordinal, requestId); + const continuationRef = + ordinal < 0 + ? undefined + : await this.captureNativeBatch('approval', requestId, ordinal); + if (ordinal >= 0 && !continuationRef) this.nativeBatch!.uncaptured = true; const options: HostedActionOptions = { - ...(HOSTED_INPUT_PREVIEW_TOOLS.includes(call.name) - ? { v: 2 as const, inputRef } - : { v: 1 as const }), + ...(continuationRef + ? { + v: 3 as const, + continuationRef, + ...(HOSTED_INPUT_PREVIEW_TOOLS.includes(call.name) + ? { inputRef } + : {}), + } + : HOSTED_INPUT_PREVIEW_TOOLS.includes(call.name) + ? { v: 2 as const, inputRef } + : { v: 1 as const }), requestId, turnId: this.promptId, functionCallId: call.callId, @@ -4969,8 +5370,10 @@ export class HostedWorkspaceToolTurn { if (this.hookStopReason) await this.harness.settleHookStoppedRuntimeContinuation(); else await this.harness.settleConsumedRuntimeContinuation(); - if (!this.mcp && !this.hooks) await this.broker.release(); - this.acquired = false; + // Physical release follows the durable Turn terminal on the G3 stack. + if (!this.mcp && !this.hooks && !this.cleanupOwed) + await this.broker.release(); + if (this.mcp || this.hooks || !this.cleanupOwed) this.acquired = false; } catch (cause) { if ( cause instanceof HostedWorkspaceBrokerRejection && @@ -4990,6 +5393,18 @@ export class HostedWorkspaceToolTurn { // deliberately does not touch it: the Session's ordered close (H3's // fifth slice) drains the stores, and background traffic keeps its // endpoint until then. + async cleanup(): Promise { + if (this.hooks || this.mcp) return; + if (this.cleanupOwed) + await reconcileHostedTurnCleanup( + this.session, + this.options, + this.promptId, + ); + else if (this.acquired) await this.broker.release(); + this.acquired = false; + } + async close(): Promise {} } diff --git a/packages/cli/src/serve/types.ts b/packages/cli/src/serve/types.ts index 26e9f7df55..87923a2903 100644 --- a/packages/cli/src/serve/types.ts +++ b/packages/cli/src/serve/types.ts @@ -450,6 +450,7 @@ export interface ServeOptions { * `v` is the wire schema version; bumped only on breaking frame changes. */ export interface HostedHarnessCapabilities { + readonly features?: readonly string[]; readonly lifecycleProtocolVersion?: 1; readonly titleProtocolVersion?: 1; readonly protocolVersions: { diff --git a/packages/cli/src/serve/workspace-recovery-session.ts b/packages/cli/src/serve/workspace-recovery-session.ts index 75698e718e..109e2f4232 100644 --- a/packages/cli/src/serve/workspace-recovery-session.ts +++ b/packages/cli/src/serve/workspace-recovery-session.ts @@ -5,6 +5,11 @@ */ import path from 'node:path'; +import { + HOSTED_RECOVERY_RESOURCE_KINDS, + parseHostedRecoveryResource, + hostedRecoveryReferences, +} from '@qwen-code/qwen-code-core/managed-runtime/hosted-recovery-records.js'; import { createHash } from 'node:crypto'; import { isDeepStrictEqual } from 'node:util'; import { @@ -180,6 +185,7 @@ const RESOURCE_KINDS = new Set([ 'managed-tool-args', 'managed-tool-outcome', 'managed-action-options', + ...HOSTED_RECOVERY_RESOURCE_KINDS, 'managed-action-decision', 'managed-message', 'managed-message-part', @@ -332,6 +338,7 @@ export async function verifyRecoverySession( boundary: unknown; state: HarnessCheckpointV1; } | null = null; + const cleanupDebt = new Map(); let activeTurn: string | null = null; let hasContinuation = false; let lastWorkSequence = 0; @@ -351,7 +358,7 @@ export async function verifyRecoverySession( !head || (head.sessionId === source.sessionId && head.tenantId === source.binding.tenantId && - head.storageVersion === 1 && + [1, 2].includes(head.storageVersion) && head.journalRevision >= 1 && head.compactedThroughRevision === 0 && head.recoveryStatus === 'READY'), @@ -831,6 +838,11 @@ export async function verifyRecoverySession( event.sequence === sequence + 1, 'noncontiguous event sequence', ); + if (event.kind === 'hosted.cleanup') + cleanupDebt.set( + event.payload['cleanupId'] as string, + event.payload['state'], + ); if (event.kind === 'input.accepted') { requireValue(activeTurn === null, 'overlapping unresolved inputs'); activeTurn = event.payload['turnId'] as string; @@ -966,6 +978,7 @@ export async function verifyRecoverySession( ); requireValue( activeTurn === null && + ![...cleanupDebt.values()].some((state) => state !== 'confirmed') && !latestHistoryPending && (latestCheckpoint ? (latestCheckpoint.boundary === HARNESS_TURN_COMPLETE_BOUNDARY || @@ -996,6 +1009,11 @@ export async function verifyRecoverySession( 'checkpoint closure owner conflicts', ); await enqueueRefs(state, io); + } else if (HOSTED_RECOVERY_RESOURCE_KINDS.has(ref.kind)) { + for (const nested of hostedRecoveryReferences( + parseHostedRecoveryResource(bytes, key ?? undefined), + )) + await io.enqueue(nested); } else if (ref.kind === 'managed-agent-instructions') { requireValue( bytes.byteLength <= 64 * 1024, diff --git a/packages/core/src/core/client.ts b/packages/core/src/core/client.ts index 6625d1d455..4e92321935 100644 --- a/packages/core/src/core/client.ts +++ b/packages/core/src/core/client.ts @@ -4,6 +4,8 @@ * SPDX-License-Identifier: Apache-2.0 */ +import type { LlmPreparedRequestCallback } from './llm-chat.js'; + import { captureHookExecutionOwner, runWithHookExecutionOwner, @@ -246,6 +248,7 @@ interface StopHookChain { } export interface SendMessageOptions { + onPreparedRequest?: LlmPreparedRequestCallback; type: SendMessageType; /** User-submitted text captured before prompt expansion. */ submittedPrompt?: string; @@ -4502,6 +4505,7 @@ export class LlmClient { // stays unmarked, so a replaced identified turn fails closed. messageType === SendMessageType.UserQuery ? prompt_id : undefined, options?.retractDeliveredOutputOnRetry, + options?.onPreparedRequest, ); // Assemble the outgoing request. IDE context is merged into the diff --git a/packages/core/src/core/llm-chat.test.ts b/packages/core/src/core/llm-chat.test.ts index 6eb55cc42c..97bd57bd2b 100644 --- a/packages/core/src/core/llm-chat.test.ts +++ b/packages/core/src/core/llm-chat.test.ts @@ -17,6 +17,10 @@ import { ApiError } from '@google/genai'; import { AuthType, type ContentGenerator } from './contentGenerator.js'; import { LlmChat, + LlmRequestPreparationError, + llmPreparedProviderPin, + type LlmPreparedRequest, + type LlmPreparedRequestCallback, InvalidStreamError, approvedPlanRedactionText, redactApprovedPlansInHistory, @@ -840,6 +844,322 @@ describe('LlmChat', async () => { }); }); + describe('prepared model request recovery', () => { + it('awaits durability after normalization and before the first provider call', async () => { + let release!: () => void; + const gate = new Promise((resolve) => { + release = resolve; + }); + let entered!: () => void; + const preparedReady = new Promise((resolve) => { + entered = resolve; + }); + let saved: LlmPreparedRequest | undefined; + streamMock().mockResolvedValue( + streamOf(stopResponse([{ text: 'done' }])), + ); + const response = await chat.sendMessageStream( + 'test-model', + { + message: 'original', + config: { + abortSignal: new AbortController().signal, + httpOptions: { headers: { Authorization: 'secret-test' } }, + }, + }, + 'prompt', + undefined, + { + onPreparedRequest: async (prepared) => { + saved = prepared; + entered(); + await gate; + }, + }, + ); + const done = drain(response); + await preparedReady; + expect(streamMock()).not.toHaveBeenCalled(); + expect(saved!.history.at(-1)?.parts).toEqual([{ text: 'original' }]); + expect(saved!.request.config.maxOutputTokens).toBeGreaterThan(0); + expect(saved!.request.config).not.toHaveProperty('abortSignal'); + expect(saved!.request.config).not.toHaveProperty('httpOptions'); + expect(JSON.stringify(saved)).not.toContain('secret-test'); + release(); + await done; + expect(streamMock()).toHaveBeenCalledOnce(); + }); + + it('avoids snapshot copies above the capture budget while sending live', async () => { + streamMock().mockResolvedValue( + streamOf(stopResponse([{ text: 'done' }])), + ); + let copiedDuringCapture = -1; + const clone = vi.spyOn(globalThis, 'structuredClone'); + const callback: LlmPreparedRequestCallback = async (prepared) => { + copiedDuringCapture = clone.mock.calls.length; + expect(JSON.stringify(prepared).length).toBeGreaterThan(64 * 1024); + }; + callback.maxSnapshotBytes = 64 * 1024; + const response = await chat.sendMessageStream( + 'test-model', + { message: 'x'.repeat(100_000) }, + 'large-prompt', + undefined, + { onPreparedRequest: callback }, + ); + clone.mockClear(); + await drain(response); + expect(copiedDuringCapture).toBe(0); + expect(streamMock()).toHaveBeenCalledOnce(); + }); + + it('does not send, retry or fall back after a failed durability barrier', async () => { + const callback = vi + .fn() + .mockRejectedValue(new Error('Store unavailable')); + const response = await chat.sendMessageStream( + 'test-model', + { message: 'original' }, + 'prompt', + undefined, + { onPreparedRequest: callback }, + ); + await expect(drain(response)).rejects.toBeInstanceOf( + LlmRequestPreparationError, + ); + expect(callback).toHaveBeenCalledOnce(); + expect(streamMock()).not.toHaveBeenCalled(); + expect(mockRetryWithBackoff).not.toHaveBeenCalled(); + }); + + it('reissues saved normalized history without another push, compaction or clamp', async () => { + const routeConfig = { + model: 'test-model', + authType: AuthType.USE_GEMINI, + samplingParams: { temperature: 0.3 }, + }; + vi.mocked(mockConfig.getContentGeneratorConfig).mockReturnValue( + routeConfig, + ); + vi.mocked(mockConfig.getBaseLlmClient).mockReturnValue({ + resolveForModel: vi.fn().mockResolvedValue({ + model: 'test-model', + contentGenerator: mockContentGenerator, + contentGeneratorConfig: routeConfig, + }), + } as unknown as ReturnType); + const saved: LlmPreparedRequest = { + request: { + model: 'test-model', + contents: [userText('normalized request')], + config: { + maxOutputTokens: 77, + systemInstruction: 'original context', + }, + }, + history: [userText('normalized request')], + completedToolCallIds: [], + routeSelector: `${AuthType.USE_GEMINI}:test-model`, + providerPin: llmPreparedProviderPin(routeConfig), + promptTokensForClamp: 15, + }; + const originalConfig = chat['generationConfig']; + const compression = vi.spyOn(chat, 'tryCompress'); + streamMock().mockResolvedValue( + streamOf(stopResponse([{ text: 'recovered' }])), + ); + const prepared = vi.fn(); + await drain( + await chat.sendPreparedMessageStream( + saved, + 'original-prompt', + new AbortController().signal, + prepared, + ), + ); + expect(chat['generationConfig']).toBe(originalConfig); + expect(compression).not.toHaveBeenCalled(); + expect(streamMock().mock.calls[0][0]).toMatchObject(saved.request); + expect(prepared.mock.calls[0][0]).toMatchObject(saved); + expect( + chat.getHistory().filter((item) => item.role === 'user'), + ).toHaveLength(1); + expect(chat.getHistory().at(-1)?.parts).toContainEqual({ + text: 'recovered', + }); + }); + + it('preserves primary persistent retry intent through prepared replay', async () => { + vi.stubEnv('QWEN_CODE_UNATTENDED_RETRY', 'true'); + try { + const routeConfig = { + model: 'test-model', + authType: AuthType.USE_GEMINI, + }; + mockGeneratorConfig(); + vi.mocked(mockConfig.getBaseLlmClient).mockReturnValue({ + resolveForModel: vi.fn().mockResolvedValue({ + model: 'test-model', + contentGenerator: mockContentGenerator, + contentGeneratorConfig: routeConfig, + }), + } as unknown as ReturnType); + let saved: LlmPreparedRequest | undefined; + streamMock().mockImplementation(async () => textStream('answer')); + await drain( + await chat.sendMessageStream( + 'test-model', + { message: 'original' }, + 'primary-replay', + undefined, + { + onPreparedRequest: async (prepared) => { + saved = prepared; + }, + }, + ), + ); + expect(saved!.allowPersistentRetry).toBe(true); + expect(mockRetryWithBackoff.mock.calls.at(-1)?.[1]).toMatchObject({ + persistentMode: true, + heartbeatFn: expect.any(Function), + }); + mockRetryWithBackoff.mockClear(); + await drain( + await newChat().sendPreparedMessageStream( + saved!, + 'primary-replay', + new AbortController().signal, + ), + ); + expect(mockRetryWithBackoff).toHaveBeenCalledOnce(); + expect(mockRetryWithBackoff.mock.calls[0]?.[1]).toMatchObject({ + persistentMode: true, + heartbeatFn: expect.any(Function), + }); + vi.stubEnv('QWEN_CODE_UNATTENDED_RETRY', 'false'); + await drain( + await newChat().sendPreparedMessageStream( + saved!, + 'primary-replay-without-opt-in', + new AbortController().signal, + ), + ); + expect(mockRetryWithBackoff.mock.calls.at(-1)?.[1].persistentMode).toBe( + false, + ); + } finally { + vi.unstubAllEnvs(); + } + }); + + it('keeps externalized images live while refusing lossy prepared replay', async () => { + mockGeneratorConfig(); + vi.mocked(mockConfig.getBaseLlmClient).mockReturnValue({ + resolveForModel: vi.fn().mockResolvedValue({ + model: 'test-model', + contentGenerator: mockContentGenerator, + contentGeneratorConfig: mockConfig.getContentGeneratorConfig(), + }), + } as unknown as ReturnType); + vi.mocked(mockConfig.getChatCompression).mockReturnValue({ + imagePayloadThreshold: 1, + maxRecentImagesToRetain: 1, + }); + chat.setHistory([ + content('user', { + inlineData: { mimeType: 'image/png', data: 'earlier-image' }, + }), + modelText('I saw the image'), + ]); + streamMock().mockImplementation(async () => textStream('answer')); + const prepared = vi.fn(); + for (const message of ['first question', 'next question']) { + await drain( + await chat.sendMessageStream( + 'test-model', + { message }, + message, + undefined, + { onPreparedRequest: prepared }, + ), + ); + } + expect(prepared).toHaveBeenCalledTimes(2); + for (const [saved, captureUnsupported] of prepared.mock.calls) { + expect(captureUnsupported).toBe(true); + expect(JSON.stringify(saved.history)).toContain('[Image #'); + expect(JSON.stringify(saved.history)).not.toContain('"inlineData"'); + } + for (const [request] of streamMock().mock.calls) { + expect( + (request.contents as Content[]) + .flatMap((entry) => entry.parts ?? []) + .filter((part) => part.inlineData), + ).toEqual([ + { + inlineData: { + mimeType: 'image/png', + data: 'earlier-image', + displayName: undefined, + }, + }, + ]); + } + streamMock().mockClear(); + mockRetryWithBackoff.mockClear(); + const fresh = newChat(); + await expect( + fresh.sendPreparedMessageStream( + prepared.mock.calls[0]![0], + 'old-image-snapshot', + new AbortController().signal, + ), + ).rejects.toBeInstanceOf(LlmRequestPreparationError); + expect(streamMock()).not.toHaveBeenCalled(); + expect(mockRetryWithBackoff).not.toHaveBeenCalled(); + expect(fresh.getHistory()).toEqual([]); + await drain( + await fresh.sendPreparedMessageStream( + { + ...prepared.mock.calls[0]![0], + history: [content('user', { text: 'original' })], + }, + 'safe-image-control', + new AbortController().signal, + ), + ); + expect(streamMock()).toHaveBeenCalledOnce(); + }); + + it('pins provider semantics while excluding credential rotation', () => { + const base = { + model: 'test', + authType: AuthType.USE_OPENAI, + baseUrl: 'https://example.test', + samplingParams: { temperature: 0.3 }, + }; + expect(llmPreparedProviderPin({ ...base, apiKey: 'first' })).toBe( + llmPreparedProviderPin({ ...base, apiKey: 'second' }), + ); + for (const changed of [ + { customHeaders: { 'X-Route': 'other' } }, + { proxy: 'http://other-proxy.test' }, + ]) { + expect(llmPreparedProviderPin({ ...base, ...changed })).not.toBe( + llmPreparedProviderPin(base), + ); + } + expect(llmPreparedProviderPin(base)).not.toBe( + llmPreparedProviderPin({ + ...base, + samplingParams: { temperature: 0.4 }, + }), + ); + }); + }); + describe('sendMessageStream', () => { type Message = Parameters[1]['message']; @@ -5459,6 +5779,66 @@ describe('LlmChat', async () => { ); }); + it('keeps a captured fallback request outside persistent retry on replay', async () => { + const fallbackGenerate = vi + .fn() + .mockImplementation(async () => textStream('fallback answer')); + const fallbackConfig = { + model: 'fallback-model', + authType: AuthType.USE_GEMINI, + }; + const resolver = wireFallbacks( + ['fallback-model'], + vi + .fn() + .mockResolvedValue( + fallbackRoute('fallback-model', fallbackGenerate, fallbackConfig), + ), + ); + streamMock().mockRejectedValue(tpmCapacityError()); + const saved: LlmPreparedRequest[] = []; + vi.stubEnv('QWEN_CODE_UNATTENDED_RETRY', 'false'); + try { + await drain( + await chat.sendMessageStream( + 'test-model', + { message: 'original' }, + 'fallback-replay', + undefined, + { + onPreparedRequest: async (prepared) => { + saved.push(prepared); + }, + }, + ), + ); + const fallback = saved.find( + (prepared) => prepared.request.model === 'fallback-model', + ); + expect(fallback).toBeDefined(); + expect(fallback!.allowPersistentRetry).toBe(false); + vi.stubEnv('QWEN_CODE_UNATTENDED_RETRY', 'true'); + mockRetryWithBackoff.mockClear(); + resolver.mockClear(); + await drain( + await newChat().sendPreparedMessageStream( + fallback!, + 'fallback-replay', + new AbortController().signal, + ), + ); + expect(mockRetryWithBackoff).toHaveBeenCalledOnce(); + expect(mockRetryWithBackoff.mock.calls[0]?.[1].persistentMode).toBe( + false, + ); + expect(mockRetryWithBackoff.mock.calls[0]?.[1]).not.toHaveProperty( + 'heartbeatFn', + ); + } finally { + vi.unstubAllEnvs(); + } + }); + it('does not enter the fallback chain in unattended retry mode', async () => { vi.stubEnv('QWEN_CODE_UNATTENDED_RETRY', '1'); try { @@ -10379,6 +10759,38 @@ describe('LlmChat', async () => { return served; } + it.each([false, true])( + 'marks prefix continuations and propagates a durability refusal (%s)', + async (refuse) => { + const served = serveStreams( + truncated('discarded'), + truncated('kept prefix'), + streamOf(stopResponse([{ text: 'suffix' }])), + ); + const flags: boolean[] = []; + const response = await chat.sendMessageStream( + 'gemini-pro', + { message: 'original' }, + 'prepared-continuation', + undefined, + { + onPreparedRequest: async (_request, continuing) => { + flags.push(continuing === true); + if (refuse && continuing) + throw new Error('Store unavailable during continuation'); + }, + }, + ); + if (refuse) + await expect(drain(response)).rejects.toBeInstanceOf( + LlmRequestPreparationError, + ); + else await drain(response); + expect(flags).toEqual([false, false, true]); + expect(served.calls).toBe(refuse ? 2 : 3); + }, + ); + const sendOn = ( model: string, message: string | Part[], @@ -13045,3 +13457,21 @@ describe('LlmChat', async () => { }); }); }); + +it('pins the reasoning registry route independently of credentials', () => { + const config = { + model: 'fixture', + authType: AuthType.USE_OPENAI, + baseUrl: 'https://fixture.invalid', + reasoningRouteBaseUrl: 'https://route-a.invalid', + }; + expect(llmPreparedProviderPin(config)).not.toBe( + llmPreparedProviderPin({ + ...config, + reasoningRouteBaseUrl: 'https://route-b.invalid', + }), + ); + expect(llmPreparedProviderPin(config)).toBe( + llmPreparedProviderPin({ ...config, apiKey: 'rotated' }), + ); +}); diff --git a/packages/core/src/core/llm-chat.ts b/packages/core/src/core/llm-chat.ts index 47999dedb2..b642574857 100644 --- a/packages/core/src/core/llm-chat.ts +++ b/packages/core/src/core/llm-chat.ts @@ -4,6 +4,8 @@ * SPDX-License-Identifier: Apache-2.0 */ +import { createHash } from 'node:crypto'; + // DISCLAIMER: This is a copied version of https://github.com/googleapis/js-genai/blob/main/src/chats.ts with the intention of working around a key bug // where function responses are not treated as "valid" responses: https://b.corp.google.com/issues/420354090 @@ -56,6 +58,7 @@ import { import type { Config } from '../config/config.js'; import type { ContentGenerator, + ContentGeneratorConfig, InputModalities, PromptCacheSharingParameters, } from './contentGenerator.js'; @@ -101,6 +104,7 @@ import { InMemoryImagePayloadStore, buildReattachParts, countAllInlineImages, + hasImagePayloadReferences, replaceImagePayloadsInPlace, } from '../services/image-payload-references.js'; import { @@ -589,7 +593,79 @@ export type StreamEvent = | { type: StreamEventType.COMPRESSED; info: ChatCompressionInfo } | { type: StreamEventType.MODEL_FALLBACK; info: ModelFallbackInfo }; +export interface LlmPreparedRequest { + request: { + model: string; + contents: Content[]; + config: Omit; + }; + history: Content[]; + completedToolCallIds: string[]; + routeSelector: string; + providerPin: string; + promptTokensForClamp: number; + allowPersistentRetry?: boolean; +} +export type LlmPreparedRequestCallback = (( + prepared: LlmPreparedRequest, + captureUnsupported?: boolean, +) => Promise) & { + /** Over-budget requests are borrowed views and must not be retained. */ + maxSnapshotBytes?: number; +}; +export class LlmRequestPreparationError extends Error { + constructor(cause: unknown) { + super('Model request durability preparation failed.', { cause }); + } +} +export function llmPreparedProviderPin( + config: ContentGeneratorConfig | undefined, +): string { + const fields = [ + 'model', + 'baseUrl', + 'customHeaders', + 'proxy', + 'vertexai', + 'authType', + 'samplingParams', + 'reasoning', + 'reasoningSnapshot', + 'reasoningRouteBaseUrl', + 'thinkingMandatory', + 'extra_body', + 'modalities', + 'schemaCompliance', + 'contextWindowSize', + 'splitToolMedia', + 'toolResultContentFormat', + 'enableCacheControl', + 'enableRequestMetadata', + 'forceGlobalCacheScope', + 'cacheRetention', + 'cacheRetentionByBlock', + ] satisfies Array; + const pin = Object.fromEntries(fields.map((key) => [key, config?.[key]])); + const stable = (value: unknown): unknown => + Array.isArray(value) + ? value.map(stable) + : value && typeof value === 'object' + ? Object.fromEntries( + Object.entries(value) + .sort(([a], [b]) => a.localeCompare(b)) + .map(([key, child]) => [key, stable(child)]), + ) + : value; + return createHash('sha256') + .update( + JSON.stringify(stable({ adapter: 'qwen-code-generator/1', ...pin })), + ) + .digest('hex'); +} + export interface LlmChatSendOptions { + onPreparedRequest?: LlmPreparedRequestCallback; + preparedRequest?: LlmPreparedRequest; /** Skip only the configured model fallback chain for this request. */ disableModelFallbacks?: boolean; /** Internal identity for the user prompt added to model history. */ @@ -2397,7 +2473,7 @@ export class LlmChat { */ constructor( private readonly config: Config, - private readonly generationConfig: GenerateContentConfig = {}, + private generationConfig: GenerateContentConfig = {}, private history: Content[] = [], private readonly chatRecordingService?: ChatRecordingService, private readonly telemetryService?: UiTelemetryService, @@ -3014,6 +3090,34 @@ export class LlmChat { * } * ``` */ + async sendPreparedMessageStream( + prepared: LlmPreparedRequest, + promptId: string, + signal: AbortSignal, + onPreparedRequest?: LlmPreparedRequestCallback, + ): Promise> { + signal.throwIfAborted(); + if (hasImagePayloadReferences(prepared.history)) + throw new LlmRequestPreparationError( + 'Saved image payloads are unavailable.', + ); + return this.sendMessageStream( + `${prepared.routeSelector}\0`, + { + message: [], + config: { ...prepared.request.config, abortSignal: signal }, + }, + promptId, + undefined, + { + preparedRequest: prepared, + onPreparedRequest, + retractDeliveredOutputOnRetry: true, + disableModelFallbacks: true, + }, + ); + } + async sendMessageStream( model: string, params: SendMessageParameters, @@ -3111,382 +3215,412 @@ export class LlmChat { cgConfigForThresholds?.contextWindowSize ?? DEFAULT_TOKEN_LIMIT; let promptTokensForClamp = 0; + const previousGenerationConfig = options?.preparedRequest + ? this.generationConfig + : undefined; let currentUserContent: Content | undefined; try { - // The send-lock above is held but the generator's `finally` (which - // resolves it) has not run yet. Any setup error before returning the - // generator must release the lock or subsequent sends will block forever - // at `await this.sendPromise`. - // Build the user content BEFORE compression so the cheap-gate can size - // the upcoming prompt — closes the "first send after inherited history" - // gap where `lastPromptTokenCount === 0` and the gate would otherwise - // see only the stale prior-turn count (0). - let userContent = createUserContent(params.message); - const toolOutputBudget = this.config.getToolOutputBatchBudget?.(); - if ( - toolOutputBudget !== undefined && - Number.isFinite(toolOutputBudget) && - userContent.parts - ) { - const [guarded] = enforceFunctionResponseBudget( - [ - { - callId: 'send-boundary', - toolName: 'tool-response-batch', - responseParts: userContent.parts, - }, - ], - toolOutputBudget, + if (options?.preparedRequest) { + const saved = options.preparedRequest; + if ( + saved.request.model !== model || + saved.providerPin !== + llmPreparedProviderPin( + exactRoute?.contentGeneratorConfig ?? + this.config.getContentGeneratorConfig(), + ) + ) + throw new LlmRequestPreparationError('Saved model route changed.'); + this.setHistory( + structuredClone(saved.history), + saved.completedToolCallIds, ); - if (guarded.responseParts !== userContent.parts) { - debugLogger.warn( - `Tool response send guard reduced an unfinalized batch to ${toolOutputBudget} characters.`, - ); - userContent = { ...userContent, parts: guarded.responseParts }; - } - } - - // Hard-tier rescue: when the estimated prompt size is at or above the - // hard threshold (effectiveWindow - HARD_BUFFER), force compaction in - // this send instead of waiting for the API to reject the request as too - // large. - // - // We compute `effectiveTokens` ONCE here and pass it through to - // tryCompress → service.compress so the cheap-gate doesn't redo the - // estimation (which involves another `getHistory(true)` clone). This - // reuse also fixes a per-config-knob inconsistency: previously the - // hard-tier rescue used the default imageTokenEstimate while the - // cheap-gate inside tryCompress used the user's resolved value. - // (review #4168 R1.3 + R1.4) - // - // The cheap-gate consecutive-failure counter is NOT pre-reset here. - // force=true already bypasses that breaker, while hard-rescue itself is - // bounded by hardRescueFailureCount so persistent pre-send rescue - // failures fall through to reactive overflow after a few strikes. - // Thresholds gate on the full window: the output clamp guarantees the - // response fits, so nothing needs to be pre-reserved for it. - const { hard } = computeThresholds( - contextWindowForClamp, - this.config.getAutoCompactThreshold(), - ); - const imageTokenEstimate = resolveSlimmingConfig( - this.config.getChatCompression(), - ).imageTokenEstimate; - // When lastPromptTokenCount > 0, estimatePromptTokens uses the - // API-authoritative previous prompt count + the previous response's - // output token count + a tiny estimate of just the new user message. - // It does NOT touch the history at all in that branch, so skip the - // costly `getHistory(true)` clone on the steady-state path. - // The lastPromptTokenCount=0 branch (first send after --continue - // restore / subagent inheritance) walks history with a char/4 - // heuristic that can under-count by ~15-20K tokens; the reactive - // overflow recovery path inside the async iterator below (the - // `getContextLengthExceededInfo` → `tryCompress` → RETRY branch) - // is the documented safety net when this under-count causes - // hard-rescue to miss. - const effectiveTokens = estimatePromptTokens( - this.lastPromptTokenCount > 0 ? [] : this.getHistoryShallow(true), - userContent, - this.lastPromptTokenCount, - this.lastOutputTokenCount, - imageTokenEstimate, - ); - const isHardTier = effectiveTokens >= hard; - const shouldForceFromHard = - !exactRoute && - isHardTier && - this.hardRescueFailureCount < MAX_CONSECUTIVE_FAILURES; - const historyBeforeHardRescue = shouldForceFromHard - ? this.getHistoryShallow() - : undefined; - const completedToolCallIdsBeforeHardRescue = this.completedToolCallIds; - const lastPromptTokenCountBeforeHardRescue = this.lastPromptTokenCount; - const lastPromptTokenCountWasEstimatedBeforeHardRescue = - this.lastPromptTokenCountIsEstimated; - // The rescue's COMPRESSED stamp clears response metadata (via - // setLastPromptTokenCount), so rollback must restore both counts. - const lastOutputTokenCountBeforeHardRescue = this.lastOutputTokenCount; - const lastCachedContentTokenCountBeforeHardRescue = - this.lastCachedContentTokenCount; - // tryCompress re-stamps tokenCountsRouteKey to the ACTIVE route (via - // setLastPromptTokenCount on the success path) even though this send - // targets the REQUEST route — and hard-rescue only fires for - // non-exact sends, whose request key can differ from the active one. - // Capture the key so the rollback below restores the resurrected - // count's original route attribution along with the count itself. - const tokenCountsRouteKeyBeforeHardRescue = this.tokenCountsRouteKey; - // Snapshot the retention map too: the rescue's compression consumes - // retained entries mid-flight (ChatCompressionService's keyless getter - // reads adopt the active route, deleting-and-consuming its entry) and - // a successful compression clears the map outright. Without the - // snapshot the rollback would restore the slots but not the map, - // leaving the resurrected route's count nowhere (#9506). - const retainedTokenCountsBeforeHardRescue = new Map( - this.tokenCountsByRouteKey, - ); - const hardRescueFailureCountBeforeHardRescue = - this.hardRescueFailureCount; - if (shouldForceFromHard) { - debugLogger.warn( - `[compaction] hard-tier rescue triggered: prompt_id=${prompt_id}, effectiveTokens=${effectiveTokens}, hard=${hard}, hardRescueAttempt=${this.hardRescueFailureCount + 1}, consecutiveFailures=${this.consecutiveFailures}.`, - ); - } else if (isHardTier && !exactRoute) { - debugLogger.warn( - `[compaction] hard-tier rescue skipped after ${this.hardRescueFailureCount} failed attempts; relying on reactive overflow recovery. prompt_id=${prompt_id}, effectiveTokens=${effectiveTokens}, hard=${hard}.`, - ); - } - - // Compression derives prompt ids before the user content is pushed. - markApiHistoryPrompt(userContent, options?.promptId); - if (exactRoute || (isHardTier && !shouldForceFromHard)) { + this.generationConfig = structuredClone(saved.request.config); + requestContents = structuredClone(saved.request.contents); + currentUserContent = this.history.at(-1); + promptTokensForClamp = saved.promptTokensForClamp; compressionInfo = { - originalTokenCount: effectiveTokens, - newTokenCount: effectiveTokens, + originalTokenCount: promptTokensForClamp, + newTokenCount: promptTokensForClamp, compressionStatus: CompressionStatus.NOOP, }; } else { - compressionInfo = await this.tryCompress( - prompt_id, - shouldForceFromHard, - params.config?.abortSignal, - { - pendingUserMessage: userContent, - precomputedEffectiveTokens: effectiveTokens, - requestGenerationConfig: params.config, - requestRouteKey, - deferChatCompressionRecord: shouldForceFromHard, - // Hard-rescue is force=true to bypass the cheap-gate breaker - // but it remains a semantically AUTOMATIC trigger. Tag the - // compactTrigger explicitly as 'auto' so PostCompact hooks are - // classified correctly while the pending user message preserves - // any active tool-call / response pairing. - trigger: shouldForceFromHard ? 'auto' : undefined, - }, - ); - } - const localPromptTokensAfterCompression = shouldForceFromHard - ? estimatePromptTokens( - this.lastPromptTokenCount > 0 ? [] : this.getHistoryShallow(true), - userContent, - this.lastPromptTokenCount, - this.lastOutputTokenCount, - imageTokenEstimate, - ) - : 0; - if ( - shouldStopAfterHardRescue( - shouldForceFromHard, - hard, - localPromptTokensAfterCompression, - ) - ) { - const message = getHardRescueFailureMessage( - effectiveTokens, - hard, - compressionInfo, - localPromptTokensAfterCompression, - ); - if (shouldForceFromHard) { - this.hardRescueFailureCount = - hardRescueFailureCountBeforeHardRescue + 1; - } + // The send-lock above is held but the generator's `finally` (which + // resolves it) has not run yet. Any setup error before returning the + // generator must release the lock or subsequent sends will block forever + // at `await this.sendPromise`. + // Build the user content BEFORE compression so the cheap-gate can size + // the upcoming prompt — closes the "first send after inherited history" + // gap where `lastPromptTokenCount === 0` and the gate would otherwise + // see only the stale prior-turn count (0). + let userContent = createUserContent(params.message); + const toolOutputBudget = this.config.getToolOutputBatchBudget?.(); if ( - compressionInfo.compressionStatus === CompressionStatus.COMPRESSED && - historyBeforeHardRescue + toolOutputBudget !== undefined && + Number.isFinite(toolOutputBudget) && + userContent.parts ) { - // Hard-rescue compression mutates in-memory history before this - // guard can compare the compressed prompt size. If the compressed - // prompt is still too large to send, restore the pre-compression - // state. The JSONL compression checkpoint is intentionally not - // written because the send is about to be rejected. - this.setHistory( - historyBeforeHardRescue, - completedToolCallIdsBeforeHardRescue, - ); - // setHistory conservatively cleared loaded-skill tracking; the - // restored bodies re-arm it on their next invoke. - this.lastPromptTokenCount = lastPromptTokenCountBeforeHardRescue; - this.lastPromptTokenCountIsEstimated = - lastPromptTokenCountWasEstimatedBeforeHardRescue; - this.lastOutputTokenCount = lastOutputTokenCountBeforeHardRescue; - this.lastCachedContentTokenCount = - lastCachedContentTokenCountBeforeHardRescue; - this.tokenCountsRouteKey = tokenCountsRouteKeyBeforeHardRescue; - // Restore the retention map alongside the slots: the rescue's - // compression consumed/cleared entries mid-flight, and without - // the restore the resurrected route's count would survive - // nowhere — its next gate read would pass with 0 (#9506). The - // snapshot predates the rescue, so it already satisfies the - // invariant (no entry for the resurrected slot key). - this.tokenCountsByRouteKey.clear(); - for (const [ - retainedRouteKey, - retainedCounts, - ] of retainedTokenCountsBeforeHardRescue) { - this.tokenCountsByRouteKey.set(retainedRouteKey, retainedCounts); - } - this.telemetryService?.setLastPromptTokenCount( - lastPromptTokenCountBeforeHardRescue, - ); - this.telemetryService?.setLastCachedContentTokenCount( - lastCachedContentTokenCountBeforeHardRescue, - ); - } - const compressionStatus = - CompressionStatus[compressionInfo.compressionStatus] ?? - String(compressionInfo.compressionStatus); - debugLogger.warn( - `[compaction] hard-tier rescue stopped oversized prompt: ` + - `prompt_id=${prompt_id}, effectiveTokens=${effectiveTokens}, ` + - `hard=${hard}, localPromptTokensAfterCompression=` + - `${localPromptTokensAfterCompression}, compressionStatus=` + - `${compressionStatus}, newTokenCount=` + - `${compressionInfo.newTokenCount}, hardRescueFailureCount=` + - `${this.hardRescueFailureCount}, consecutiveFailures=` + - `${this.consecutiveFailures}. ${message}`, - ); - throw new Error(message); - } - if ( - shouldForceFromHard && - compressionInfo.compressionStatus === CompressionStatus.COMPRESSED - ) { - // Keep the pending question with the compressed answer on resume. - this.chatRecordingService?.recordChatCompression({ - info: compressionInfo, - compressedHistory: [...this.getHistoryShallow(), userContent], - completedToolCallIds: this.completedToolCallIds, - }); - } - - if (this.manualPlanExitNoticesEnabled) { - const notice = this.config.takePendingManualPlanExitNotice(); - if (notice) { - manualPlanExitNoticeVersion = notice.version; - manualPlanExitNoticeText = getManualPlanExitSystemReminder( - notice.currentMode, - ); - userContent = { - ...userContent, - parts: [ - ...(userContent.parts ?? []), + const [guarded] = enforceFunctionResponseBudget( + [ { - text: manualPlanExitNoticeText, + callId: 'send-boundary', + toolName: 'tool-response-batch', + responseParts: userContent.parts, }, ], - }; + toolOutputBudget, + ); + if (guarded.responseParts !== userContent.parts) { + debugLogger.warn( + `Tool response send guard reduced an unfinalized batch to ${toolOutputBudget} characters.`, + ); + userContent = { ...userContent, parts: guarded.responseParts }; + } } - } - // Publish the acceptance snapshot for a caller-side settlement - // carrier (see `userContentPushSnapshotKey`) immediately before the - // push — no await between the snapshot and this push, so no - // concurrent send can supply the counter growth it observes. - // `params.message` is the caller's own request array (Turn passes - // it through unchanged), so the publication reaches the caller even - // though this method never returns on the pre-push error paths. - if (Array.isArray(params.message)) { - (params.message as unknown as Record)[ - userContentPushSnapshotKey - ] = this.userContentPushCount; - } - // Add user content to history ONCE before any attempts. Later object - // spreads preserve the identity marked before compression. - this.history.push(userContent); - this.syncReviewedSchemasForContent(userContent); - currentUserContent = userContent; - userContentAdded = true; - // Record that the user content landed (see `userContentPushCount`). The - // setup-error path below decrements this if it rolls the push back. - this.userContentPushCount++; - // Per-send orphan repair (belt-and-suspenders alongside the - // startChat load-time pass). Runs AFTER user content lands so a - // user-supplied tool_result closes the pair before we synthesize - // anything. An ordinary prompt that races a restore re-hang must - // still close the pair — `model[functionCall] → user[text]` is - // rejected by Anthropic-compatible providers. Restore itself sends - // the real functionResponse, so this pass is a no-op on that path. - const inlineRepair = repairOrphanedToolUseTurns( - this.history, - ORPHAN_TOOL_USE_REPAIR_REASON, - ); - if (inlineRepair.injected.length > 0) { - debugLogger.warn( - `[REPAIR] sendMessageStream inline pass synthesized ` + - `${inlineRepair.injected.length} functionResponse(s): ` + - inlineRepair.injected - .map((entry) => `${entry.name}(${entry.callId})`) - .join(', '), + // Hard-tier rescue: when the estimated prompt size is at or above the + // hard threshold (effectiveWindow - HARD_BUFFER), force compaction in + // this send instead of waiting for the API to reject the request as too + // large. + // + // We compute `effectiveTokens` ONCE here and pass it through to + // tryCompress → service.compress so the cheap-gate doesn't redo the + // estimation (which involves another `getHistory(true)` clone). This + // reuse also fixes a per-config-knob inconsistency: previously the + // hard-tier rescue used the default imageTokenEstimate while the + // cheap-gate inside tryCompress used the user's resolved value. + // (review #4168 R1.3 + R1.4) + // + // The cheap-gate consecutive-failure counter is NOT pre-reset here. + // force=true already bypasses that breaker, while hard-rescue itself is + // bounded by hardRescueFailureCount so persistent pre-send rescue + // failures fall through to reactive overflow after a few strikes. + // Thresholds gate on the full window: the output clamp guarantees the + // response fits, so nothing needs to be pre-reserved for it. + const { hard } = computeThresholds( + contextWindowForClamp, + this.config.getAutoCompactThreshold(), ); - } - if (inlineRepair.droppedDuplicates.length > 0) { - debugLogger.warn( - `[REPAIR] sendMessageStream inline pass dropped ` + - `${inlineRepair.droppedDuplicates.length} duplicate ` + - `functionResponse(s): ` + - inlineRepair.droppedDuplicates - .map((entry) => `${entry.name}(${entry.callId})`) - .join(', '), + const imageTokenEstimate = resolveSlimmingConfig( + this.config.getChatCompression(), + ).imageTokenEstimate; + // When lastPromptTokenCount > 0, estimatePromptTokens uses the + // API-authoritative previous prompt count + the previous response's + // output token count + a tiny estimate of just the new user message. + // It does NOT touch the history at all in that branch, so skip the + // costly `getHistory(true)` clone on the steady-state path. + // The lastPromptTokenCount=0 branch (first send after --continue + // restore / subagent inheritance) walks history with a char/4 + // heuristic that can under-count by ~15-20K tokens; the reactive + // overflow recovery path inside the async iterator below (the + // `getContextLengthExceededInfo` → `tryCompress` → RETRY branch) + // is the documented safety net when this under-count causes + // hard-rescue to miss. + const effectiveTokens = estimatePromptTokens( + this.lastPromptTokenCount > 0 ? [] : this.getHistoryShallow(true), + userContent, + this.lastPromptTokenCount, + this.lastOutputTokenCount, + imageTokenEstimate, ); - } - requestContents = this.getRequestHistoryForRoute( - currentUserContent, - requestModalities, - ); + const isHardTier = effectiveTokens >= hard; + const shouldForceFromHard = + !exactRoute && + isHardTier && + this.hardRescueFailureCount < MAX_CONSECUTIVE_FAILURES; + const historyBeforeHardRescue = shouldForceFromHard + ? this.getHistoryShallow() + : undefined; + const completedToolCallIdsBeforeHardRescue = this.completedToolCallIds; + const lastPromptTokenCountBeforeHardRescue = this.lastPromptTokenCount; + const lastPromptTokenCountWasEstimatedBeforeHardRescue = + this.lastPromptTokenCountIsEstimated; + // The rescue's COMPRESSED stamp clears response metadata (via + // setLastPromptTokenCount), so rollback must restore both counts. + const lastOutputTokenCountBeforeHardRescue = this.lastOutputTokenCount; + const lastCachedContentTokenCountBeforeHardRescue = + this.lastCachedContentTokenCount; + // tryCompress re-stamps tokenCountsRouteKey to the ACTIVE route (via + // setLastPromptTokenCount on the success path) even though this send + // targets the REQUEST route — and hard-rescue only fires for + // non-exact sends, whose request key can differ from the active one. + // Capture the key so the rollback below restores the resurrected + // count's original route attribution along with the count itself. + const tokenCountsRouteKeyBeforeHardRescue = this.tokenCountsRouteKey; + // Snapshot the retention map too: the rescue's compression consumes + // retained entries mid-flight (ChatCompressionService's keyless getter + // reads adopt the active route, deleting-and-consuming its entry) and + // a successful compression clears the map outright. Without the + // snapshot the rollback would restore the slots but not the map, + // leaving the resurrected route's count nowhere (#9506). + const retainedTokenCountsBeforeHardRescue = new Map( + this.tokenCountsByRouteKey, + ); + const hardRescueFailureCountBeforeHardRescue = + this.hardRescueFailureCount; + if (shouldForceFromHard) { + debugLogger.warn( + `[compaction] hard-tier rescue triggered: prompt_id=${prompt_id}, effectiveTokens=${effectiveTokens}, hard=${hard}, hardRescueAttempt=${this.hardRescueFailureCount + 1}, consecutiveFailures=${this.consecutiveFailures}.`, + ); + } else if (isHardTier && !exactRoute) { + debugLogger.warn( + `[compaction] hard-tier rescue skipped after ${this.hardRescueFailureCount} failed attempts; relying on reactive overflow recovery. prompt_id=${prompt_id}, effectiveTokens=${effectiveTokens}, hard=${hard}.`, + ); + } - // Window-clamp the output request AFTER compression has settled the - // history: max_tokens = min(ceiling, window − prompt − margin), floored - // at MIN_CLAMPED_OUTPUT_TOKENS. Computed here in the send path — not in - // the shared provider code — so the API-authoritative - // lastPromptTokenCount is in scope and side queries (which set their - // own maxOutputTokens via getBaseLlmClient()) stay exempt by - // construction. This makes `prompt + max_tokens ≤ window` an invariant - // on every main-turn request (issue #5950). - // - // When lastPromptTokenCount > 0 (steady state, or refreshed to - // newTokenCount by compression/resume), re-estimate from the counts — - // cheap, no history walk. When it is still 0, reuse the pre-push gate - // estimate: userContent is already in history here, so a fresh history - // walk would double-count it. Estimate-derived counts can omit the - // system prompt, tool definitions, and skill content (see - // estimatePromptTokens — "typically ~15-20K of under-estimate"). Some - // counts based on prior API usage already preserve part of that - // overhead, but conservatively double-counting it is safe and - // self-corrects when provider usage arrives. An under-count is the ONE - // way `prompt + max_tokens` can still overflow the window, so keep the - // pad until provider usage replaces the estimate. - promptTokensForClamp = - this.lastPromptTokenCount > 0 + // Compression derives prompt ids before the user content is pushed. + markApiHistoryPrompt(userContent, options?.promptId); + if (exactRoute || (isHardTier && !shouldForceFromHard)) { + compressionInfo = { + originalTokenCount: effectiveTokens, + newTokenCount: effectiveTokens, + compressionStatus: CompressionStatus.NOOP, + }; + } else { + compressionInfo = await this.tryCompress( + prompt_id, + shouldForceFromHard, + params.config?.abortSignal, + { + pendingUserMessage: userContent, + precomputedEffectiveTokens: effectiveTokens, + requestGenerationConfig: params.config, + requestRouteKey, + deferChatCompressionRecord: shouldForceFromHard, + // Hard-rescue is force=true to bypass the cheap-gate breaker + // but it remains a semantically AUTOMATIC trigger. Tag the + // compactTrigger explicitly as 'auto' so PostCompact hooks are + // classified correctly while the pending user message preserves + // any active tool-call / response pairing. + trigger: shouldForceFromHard ? 'auto' : undefined, + }, + ); + } + const localPromptTokensAfterCompression = shouldForceFromHard ? estimatePromptTokens( - [], + this.lastPromptTokenCount > 0 ? [] : this.getHistoryShallow(true), userContent, this.lastPromptTokenCount, this.lastOutputTokenCount, imageTokenEstimate, - /* conservative= */ true, ) - : effectiveTokens; - if (this.promptCountIsEstimateDerived()) { - promptTokensForClamp += ESTIMATE_CLAMP_OVERHEAD_PAD; - debugLogger.debug( - `[clamp] estimate-derived prompt count; padded by ` + - `${ESTIMATE_CLAMP_OVERHEAD_PAD}: ` + - `promptTokensForClamp=${promptTokensForClamp}, ` + - `count=${this.lastPromptTokenCount}`, + : 0; + if ( + shouldStopAfterHardRescue( + shouldForceFromHard, + hard, + localPromptTokensAfterCompression, + ) + ) { + const message = getHardRescueFailureMessage( + effectiveTokens, + hard, + compressionInfo, + localPromptTokensAfterCompression, + ); + if (shouldForceFromHard) { + this.hardRescueFailureCount = + hardRescueFailureCountBeforeHardRescue + 1; + } + if ( + compressionInfo.compressionStatus === + CompressionStatus.COMPRESSED && + historyBeforeHardRescue + ) { + // Hard-rescue compression mutates in-memory history before this + // guard can compare the compressed prompt size. If the compressed + // prompt is still too large to send, restore the pre-compression + // state. The JSONL compression checkpoint is intentionally not + // written because the send is about to be rejected. + this.setHistory( + historyBeforeHardRescue, + completedToolCallIdsBeforeHardRescue, + ); + // setHistory conservatively cleared loaded-skill tracking; the + // restored bodies re-arm it on their next invoke. + this.lastPromptTokenCount = lastPromptTokenCountBeforeHardRescue; + this.lastPromptTokenCountIsEstimated = + lastPromptTokenCountWasEstimatedBeforeHardRescue; + this.lastOutputTokenCount = lastOutputTokenCountBeforeHardRescue; + this.lastCachedContentTokenCount = + lastCachedContentTokenCountBeforeHardRescue; + this.tokenCountsRouteKey = tokenCountsRouteKeyBeforeHardRescue; + // Restore the retention map alongside the slots: the rescue's + // compression consumed/cleared entries mid-flight, and without + // the restore the resurrected route's count would survive + // nowhere — its next gate read would pass with 0 (#9506). The + // snapshot predates the rescue, so it already satisfies the + // invariant (no entry for the resurrected slot key). + this.tokenCountsByRouteKey.clear(); + for (const [ + retainedRouteKey, + retainedCounts, + ] of retainedTokenCountsBeforeHardRescue) { + this.tokenCountsByRouteKey.set(retainedRouteKey, retainedCounts); + } + this.telemetryService?.setLastPromptTokenCount( + lastPromptTokenCountBeforeHardRescue, + ); + this.telemetryService?.setLastCachedContentTokenCount( + lastCachedContentTokenCountBeforeHardRescue, + ); + } + const compressionStatus = + CompressionStatus[compressionInfo.compressionStatus] ?? + String(compressionInfo.compressionStatus); + debugLogger.warn( + `[compaction] hard-tier rescue stopped oversized prompt: ` + + `prompt_id=${prompt_id}, effectiveTokens=${effectiveTokens}, ` + + `hard=${hard}, localPromptTokensAfterCompression=` + + `${localPromptTokensAfterCompression}, compressionStatus=` + + `${compressionStatus}, newTokenCount=` + + `${compressionInfo.newTokenCount}, hardRescueFailureCount=` + + `${this.hardRescueFailureCount}, consecutiveFailures=` + + `${this.consecutiveFailures}. ${message}`, + ); + throw new Error(message); + } + if ( + shouldForceFromHard && + compressionInfo.compressionStatus === CompressionStatus.COMPRESSED + ) { + // Keep the pending question with the compressed answer on resume. + this.chatRecordingService?.recordChatCompression({ + info: compressionInfo, + compressedHistory: [...this.getHistoryShallow(), userContent], + completedToolCallIds: this.completedToolCallIds, + }); + } + + if (this.manualPlanExitNoticesEnabled) { + const notice = this.config.takePendingManualPlanExitNotice(); + if (notice) { + manualPlanExitNoticeVersion = notice.version; + manualPlanExitNoticeText = getManualPlanExitSystemReminder( + notice.currentMode, + ); + userContent = { + ...userContent, + parts: [ + ...(userContent.parts ?? []), + { + text: manualPlanExitNoticeText, + }, + ], + }; + } + } + + // Publish the acceptance snapshot for a caller-side settlement + // carrier (see `userContentPushSnapshotKey`) immediately before the + // push — no await between the snapshot and this push, so no + // concurrent send can supply the counter growth it observes. + // `params.message` is the caller's own request array (Turn passes + // it through unchanged), so the publication reaches the caller even + // though this method never returns on the pre-push error paths. + if (Array.isArray(params.message)) { + (params.message as unknown as Record)[ + userContentPushSnapshotKey + ] = this.userContentPushCount; + } + // Add user content to history ONCE before any attempts. Later object + // spreads preserve the identity marked before compression. + this.history.push(userContent); + this.syncReviewedSchemasForContent(userContent); + currentUserContent = userContent; + userContentAdded = true; + // Record that the user content landed (see `userContentPushCount`). The + // setup-error path below decrements this if it rolls the push back. + this.userContentPushCount++; + // Per-send orphan repair (belt-and-suspenders alongside the + // startChat load-time pass). Runs AFTER user content lands so a + // user-supplied tool_result closes the pair before we synthesize + // anything. An ordinary prompt that races a restore re-hang must + // still close the pair — `model[functionCall] → user[text]` is + // rejected by Anthropic-compatible providers. Restore itself sends + // the real functionResponse, so this pass is a no-op on that path. + const inlineRepair = repairOrphanedToolUseTurns( + this.history, + ORPHAN_TOOL_USE_REPAIR_REASON, ); + if (inlineRepair.injected.length > 0) { + debugLogger.warn( + `[REPAIR] sendMessageStream inline pass synthesized ` + + `${inlineRepair.injected.length} functionResponse(s): ` + + inlineRepair.injected + .map((entry) => `${entry.name}(${entry.callId})`) + .join(', '), + ); + } + if (inlineRepair.droppedDuplicates.length > 0) { + debugLogger.warn( + `[REPAIR] sendMessageStream inline pass dropped ` + + `${inlineRepair.droppedDuplicates.length} duplicate ` + + `functionResponse(s): ` + + inlineRepair.droppedDuplicates + .map((entry) => `${entry.name}(${entry.callId})`) + .join(', '), + ); + } + requestContents = this.getRequestHistoryForRoute( + currentUserContent, + requestModalities, + ); + + // Window-clamp the output request AFTER compression has settled the + // history: max_tokens = min(ceiling, window − prompt − margin), floored + // at MIN_CLAMPED_OUTPUT_TOKENS. Computed here in the send path — not in + // the shared provider code — so the API-authoritative + // lastPromptTokenCount is in scope and side queries (which set their + // own maxOutputTokens via getBaseLlmClient()) stay exempt by + // construction. This makes `prompt + max_tokens ≤ window` an invariant + // on every main-turn request (issue #5950). + // + // When lastPromptTokenCount > 0 (steady state, or refreshed to + // newTokenCount by compression/resume), re-estimate from the counts — + // cheap, no history walk. When it is still 0, reuse the pre-push gate + // estimate: userContent is already in history here, so a fresh history + // walk would double-count it. Estimate-derived counts can omit the + // system prompt, tool definitions, and skill content (see + // estimatePromptTokens — "typically ~15-20K of under-estimate"). Some + // counts based on prior API usage already preserve part of that + // overhead, but conservatively double-counting it is safe and + // self-corrects when provider usage arrives. An under-count is the ONE + // way `prompt + max_tokens` can still overflow the window, so keep the + // pad until provider usage replaces the estimate. + promptTokensForClamp = + this.lastPromptTokenCount > 0 + ? estimatePromptTokens( + [], + userContent, + this.lastPromptTokenCount, + this.lastOutputTokenCount, + imageTokenEstimate, + /* conservative= */ true, + ) + : effectiveTokens; + if (this.promptCountIsEstimateDerived()) { + promptTokensForClamp += ESTIMATE_CLAMP_OVERHEAD_PAD; + debugLogger.debug( + `[clamp] estimate-derived prompt count; padded by ` + + `${ESTIMATE_CLAMP_OVERHEAD_PAD}: ` + + `promptTokensForClamp=${promptTokensForClamp}, ` + + `count=${this.lastPromptTokenCount}`, + ); + } + const clampedMaxOutputTokens = clampOutputTokensToWindow( + outputCeiling, + contextWindowForClamp, + promptTokensForClamp, + ); + params = { + ...params, + config: { + ...params.config, + maxOutputTokens: clampedMaxOutputTokens, + }, + }; } - const clampedMaxOutputTokens = clampOutputTokensToWindow( - outputCeiling, - contextWindowForClamp, - promptTokensForClamp, - ); - params = { - ...params, - config: { - ...params.config, - maxOutputTokens: clampedMaxOutputTokens, - }, - }; } catch (error) { if (userContentAdded) { this.history.pop(); @@ -3501,6 +3635,8 @@ export class LlmChat { manualPlanExitNoticeVersion, ); } + if (previousGenerationConfig) + this.generationConfig = previousGenerationConfig; streamDoneResolver!(); throw error; } @@ -3589,8 +3725,11 @@ export class LlmChat { const requestOverrides = exactRoute ? { contentGenerator: exactRoute.contentGenerator, + contentGeneratorConfig: exactRoute.contentGeneratorConfig, retryAuthType: exactRoute.retryAuthType, retryErrorCodes: exactRoute.retryErrorCodes, + allowPersistentRetry: + options?.preparedRequest?.allowPersistentRetry === true, } : undefined; const maxRateLimitRetries = @@ -3742,6 +3881,8 @@ export class LlmChat { ? transportContinuationPrefix : undefined, acceptQuietToolResultCompletion, + options?.onPreparedRequest, + promptTokensForClamp, ); streamEstablished = true; @@ -3814,6 +3955,7 @@ export class LlmChat { transportContinuationPrefix = []; break; } catch (error) { + if (error instanceof LlmRequestPreparationError) throw error; lastError = error; if (params.config?.abortSignal?.aborted) throw error; // This attempt is over; fold what it delivered into the running @@ -4625,6 +4767,9 @@ export class LlmChat { turnGoalContext, undefined, acceptQuietToolResultCompletion, + options?.onPreparedRequest, + promptTokensForClamp, + retryEvent.isContinuation === true, ); for await (const chunk of stream) { yield { type: StreamEventType.CHUNK, value: chunk }; @@ -4881,6 +5026,8 @@ export class LlmChat { successfulRecoveries++; activeRecoveryUser = undefined; } catch (recoveryError) { + if (recoveryError instanceof LlmRequestPreparationError) + throw recoveryError; if (params.config?.abortSignal?.aborted) throw recoveryError; rollbackRecoveryAttempt(); debugLogger.warn( @@ -4969,6 +5116,7 @@ export class LlmChat { // Resolve the fallback model's content generator let fallbackGenerator: ContentGenerator; + let fallbackConfig: ContentGeneratorConfig | undefined; let fallbackRetryAuthType: string | undefined; let fallbackRetryErrorCodes: readonly number[] | undefined; let resolvedFallbackModel: string; @@ -4978,6 +5126,7 @@ export class LlmChat { .getBaseLlmClient() .resolveForModel(fallbackModelId, { failClosed: true }); fallbackGenerator = resolved.contentGenerator; + fallbackConfig = resolved.contentGeneratorConfig; fallbackRetryAuthType = resolved.retryAuthType; fallbackRetryErrorCodes = resolved.retryErrorCodes; resolvedFallbackModel = resolved.model; @@ -5057,6 +5206,9 @@ export class LlmChat { fallbackRetryErrorCodes, requestRouteKey, turnGoalContext, + options?.onPreparedRequest, + promptTokensForClamp, + fallbackConfig, )) { const emittedUserVisibleOutput = event.type !== StreamEventType.CHUNK || @@ -5077,6 +5229,8 @@ export class LlmChat { ); return; } catch (fallbackError) { + if (fallbackError instanceof LlmRequestPreparationError) + throw fallbackError; if ( params.config?.abortSignal?.aborted || isAbortError(fallbackError) @@ -5201,6 +5355,8 @@ export class LlmChat { if (successfulRecoveries > 0) { self.coalesceRecoveryPairs(successfulRecoveries); } + if (previousGenerationConfig) + self.generationConfig = previousGenerationConfig; sleepInhibitorHandle.release(); streamDoneResolver!(); // Flush any deferred partial-tool_use record. Covers both the @@ -5245,37 +5401,72 @@ export class LlmChat { prompt_id: string, overrides?: { contentGenerator: ContentGenerator; + contentGeneratorConfig?: ContentGeneratorConfig; retryAuthType?: string; retryErrorCodes?: readonly number[]; + allowPersistentRetry?: boolean; }, routeKey = this.currentRouteKey(), goalContext?: GoalTurnPermit, transportContinuationPrefix?: Part[], acceptQuietToolResultCompletion = false, + onPreparedRequest?: LlmPreparedRequestCallback, + promptTokensForClamp = 0, + continuationInFlight = false, ): Promise> { const generator = overrides?.contentGenerator ?? this.config.getContentGenerator(); - const apiCall = () => { - // A continuation attempt's replay gate is already shut by the - // accumulated prefix, so the pipeline must release a parked tool-call - // finish rather than withhold it for a replay that cannot happen. - const request: PromptCacheSharingParameters = { - model, - contents: requestContents, - config: { ...this.generationConfig, ...params.config }, - ...(transportContinuationPrefix !== undefined && { - continuationInFlight: true, - }), - }; - return generator.generateContentStream(request, prompt_id); + const request: PromptCacheSharingParameters = { + model, + contents: requestContents, + config: { ...this.generationConfig, ...params.config }, + ...(transportContinuationPrefix !== undefined && { + continuationInFlight: true, + }), }; + const actualConfig = + overrides?.contentGeneratorConfig ?? + this.config.getContentGeneratorConfig(); + if (onPreparedRequest) { + const { + abortSignal: _signal, + httpOptions: _http, + ...config + } = request.config ?? {}; + try { + const prepared: LlmPreparedRequest = { + request: { model, contents: requestContents, config }, + history: this.history, + completedToolCallIds: [...this.completedToolCallIds], + routeSelector: `${actualConfig?.authType ?? ''}:${model}`, + providerPin: llmPreparedProviderPin(actualConfig), + promptTokensForClamp, + allowPersistentRetry: overrides?.allowPersistentRetry ?? !overrides, + }; + const overBudget = + onPreparedRequest.maxSnapshotBytes !== undefined && + Buffer.byteLength(JSON.stringify(prepared)) > + onPreparedRequest.maxSnapshotBytes; + await onPreparedRequest( + overBudget ? prepared : structuredClone(prepared), + continuationInFlight || + transportContinuationPrefix !== undefined || + hasImagePayloadReferences(this.history), + ); + } catch (cause) { + throw new LlmRequestPreparationError(cause); + } + } + params.config?.abortSignal?.throwIfAborted(); + const apiCall = () => generator.generateContentStream(request, prompt_id); const cgConfig = this.config.getContentGeneratorConfig(); const authType = overrides?.retryAuthType ?? cgConfig?.authType; const extraRetryErrorCodes = overrides?.retryErrorCodes ?? cgConfig?.retryErrorCodes; // Fallback models never enter persistent retry mode — persistent mode // is the caller's explicit opt-in for the primary model only. - const persistentMode = overrides ? false : isUnattendedMode(); + const persistentMode = + (overrides?.allowPersistentRetry ?? !overrides) && isUnattendedMode(); const streamResponse = await retryWithBackoff(apiCall, { shouldRetryOnError: (error: unknown) => { if (error instanceof ResponsesHttpError) { @@ -5358,15 +5549,27 @@ export class LlmChat { retryErrorCodes?: readonly number[], routeKey?: string, goalContext?: GoalTurnPermit, + onPreparedRequest?: LlmPreparedRequestCallback, + promptTokensForClamp = 0, + contentGeneratorConfig?: ContentGeneratorConfig, ): AsyncGenerator { const stream = await this.makeApiCallAndProcessStream( model, requestContents, params, prompt_id, - { contentGenerator, retryAuthType, retryErrorCodes }, + { + contentGenerator, + contentGeneratorConfig, + retryAuthType, + retryErrorCodes, + }, routeKey, goalContext, + undefined, + false, + onPreparedRequest, + promptTokensForClamp, ); for await (const chunk of stream) { diff --git a/packages/core/src/core/turn.ts b/packages/core/src/core/turn.ts index 8b6fe0b2e0..19ccaeb96a 100644 --- a/packages/core/src/core/turn.ts +++ b/packages/core/src/core/turn.ts @@ -30,7 +30,12 @@ import { UnauthorizedError, toFriendlyError, } from '../utils/errors.js'; -import type { LlmChat } from './llm-chat.js'; +import { + LlmRequestPreparationError, + type LlmChat, + type LlmPreparedRequest, + type LlmPreparedRequestCallback, +} from './llm-chat.js'; import type { RetryInfo } from '../utils/rateLimit.js'; import { getThoughtSummary, @@ -707,6 +712,7 @@ export class Turn { goalContext?: GoalTurnPermit, private readonly promptIdentity?: string, private readonly retractDeliveredOutputOnRetry?: boolean, + private readonly onPreparedRequest?: LlmPreparedRequestCallback, ) { this.goalContext = goalContext ? { ...goalContext } : undefined; } @@ -715,6 +721,7 @@ export class Turn { model: string, req: PartListUnion, signal: AbortSignal, + prepared?: LlmPreparedRequest, ): AsyncGenerator { try { // Note: This assumes `sendMessageStream` yields events like @@ -723,26 +730,37 @@ export class Turn { // `undefined`, as before either option existed. const sendOptions = this.promptIdentity !== undefined || - this.retractDeliveredOutputOnRetry === true + this.retractDeliveredOutputOnRetry === true || + this.onPreparedRequest !== undefined ? { + ...(this.onPreparedRequest + ? { onPreparedRequest: this.onPreparedRequest } + : {}), ...(this.promptIdentity ? { promptId: this.promptIdentity } : {}), ...(this.retractDeliveredOutputOnRetry ? { retractDeliveredOutputOnRetry: true } : {}), } : undefined; - const responseStream = await this.chat.sendMessageStream( - model, - { - message: req, - config: { - abortSignal: signal, - }, - }, - this.prompt_id, - this.goalContext, - sendOptions, - ); + const responseStream = prepared + ? await this.chat.sendPreparedMessageStream( + prepared, + this.prompt_id, + signal, + this.onPreparedRequest, + ) + : await this.chat.sendMessageStream( + model, + { + message: req, + config: { + abortSignal: signal, + }, + }, + this.prompt_id, + this.goalContext, + sendOptions, + ); for await (const streamEvent of responseStream) { if (signal?.aborted) { @@ -869,6 +887,7 @@ export class Turn { } } } catch (e) { + if (e instanceof LlmRequestPreparationError) throw e; if (signal.aborted) { yield { type: LlmEventType.UserCancelled }; // Regular cancellation error, fail gracefully. diff --git a/packages/core/src/managed-runtime/contracts/managed-event-envelope-v1.fixtures.json b/packages/core/src/managed-runtime/contracts/managed-event-envelope-v1.fixtures.json index a60c7cb219..26ba4002dd 100644 --- a/packages/core/src/managed-runtime/contracts/managed-event-envelope-v1.fixtures.json +++ b/packages/core/src/managed-runtime/contracts/managed-event-envelope-v1.fixtures.json @@ -20,7 +20,9 @@ "domain.committed", "message.delta", "message.retracted", - "operation.replayed" + "operation.replayed", + "hosted.batch.planned", + "hosted.cleanup" ], "streams": ["authoritative_journal"], "forbiddenFields": [ @@ -1564,6 +1566,42 @@ "digest": "631f42ee57f8e39d9491a76ba9477d32a11f8c2b8a3a4ff3c21b30760c600000" } } + }, + { + "id": "kind-hosted-batch-planned", + "valid": true, + "envelope": { + "v": 1, + "sessionId": "session-1", + "tenantId": "tenant-1", + "workspaceId": "workspace-1", + "stream": "authoritative_journal", + "sequence": 118, + "eventId": "evt-000118", + "kind": "hosted.batch.planned", + "occurredAt": 1760000000016, + "payloadRef": { + "digest": "631f42ee57f8e39d9491a76ba9477d32a11f8c2b8a3a4ff3c21b30760c600000" + } + } + }, + { + "id": "kind-hosted-cleanup", + "valid": true, + "envelope": { + "v": 1, + "sessionId": "session-1", + "tenantId": "tenant-1", + "workspaceId": "workspace-1", + "stream": "authoritative_journal", + "sequence": 119, + "eventId": "evt-000119", + "kind": "hosted.cleanup", + "occurredAt": 1760000000016, + "payloadRef": { + "digest": "631f42ee57f8e39d9491a76ba9477d32a11f8c2b8a3a4ff3c21b30760c600000" + } + } } ], "dedupeCases": [ diff --git a/packages/core/src/managed-runtime/contracts/managed-event-envelope-v1.schema.json b/packages/core/src/managed-runtime/contracts/managed-event-envelope-v1.schema.json index a099b8021b..3040901057 100644 --- a/packages/core/src/managed-runtime/contracts/managed-event-envelope-v1.schema.json +++ b/packages/core/src/managed-runtime/contracts/managed-event-envelope-v1.schema.json @@ -46,7 +46,9 @@ "domain.committed", "message.delta", "message.retracted", - "operation.replayed" + "operation.replayed", + "hosted.batch.planned", + "hosted.cleanup" ] }, "streams": { @@ -226,7 +228,9 @@ "domain.committed", "message.delta", "message.retracted", - "operation.replayed" + "operation.replayed", + "hosted.batch.planned", + "hosted.cleanup" ] }, "occurredAt": { diff --git a/packages/core/src/managed-runtime/contracts/managed-session-store-v1.fixtures.json b/packages/core/src/managed-runtime/contracts/managed-session-store-v1.fixtures.json index 7b20eafc07..13f612b8d6 100644 --- a/packages/core/src/managed-runtime/contracts/managed-session-store-v1.fixtures.json +++ b/packages/core/src/managed-runtime/contracts/managed-session-store-v1.fixtures.json @@ -35,7 +35,9 @@ "domain.committed", "message.delta", "message.retracted", - "operation.replayed" + "operation.replayed", + "hosted.batch.planned", + "hosted.cleanup" ], "sessionKey": { "tenantId": "tenant-fixture", diff --git a/packages/core/src/managed-runtime/hosted-recovery-records.test.ts b/packages/core/src/managed-runtime/hosted-recovery-records.test.ts new file mode 100644 index 0000000000..7e47c2d8f9 --- /dev/null +++ b/packages/core/src/managed-runtime/hosted-recovery-records.test.ts @@ -0,0 +1,69 @@ +/** + * @license + * Copyright 2026 Qwen Team + * SPDX-License-Identifier: Apache-2.0 + */ +import { expect, it } from 'vitest'; +import { + hostedRecoveryReferences, + parseHostedRecoveryResource, +} from './hosted-recovery-records.js'; + +const key = { + tenantId: 'tenant', + workspaceId: 'workspace', + sessionId: 'session', +}; +const ref = { + resourceId: 'definition', + kind: 'managed-definition', + schemaVersion: 1, + byteLength: 2, + digest: 'a'.repeat(64), +}; + +it('follows only protocol reference slots, preserving arbitrary model and tool data', () => { + const businessData = { + resourceId: 'business', + kind: 'business-record', + schemaVersion: 1, + byteLength: 12, + digest: 'ordinary-user-digest', + }; + expect( + hostedRecoveryReferences({ + definitionRef: ref, + prepared: { + history: [{ parts: [{ functionCall: { args: businessData } }] }], + request: { config: { tools: [businessData] } }, + }, + calls: [ + { call: { args: businessData }, inputRef: ref, definitionRef: ref }, + ], + }), + ).toEqual([ref]); +}); + +it('rejects malformed protocol references and conflicting reference metadata', () => { + expect(() => + hostedRecoveryReferences({ definitionRef: { resourceId: 'missing' } }), + ).toThrow(); + expect(() => + hostedRecoveryReferences({ + definitionRef: ref, + rootSnapshotRef: { ...ref, byteLength: 3 }, + }), + ).toThrow('conflict'); +}); + +it.each([ + { ...key, sessionId: 'foreign' }, + { ...key, tenantId: 'foreign' }, +])('rejects recovery resources naming a foreign scope', (sessionKey) => { + expect(() => + parseHostedRecoveryResource( + Buffer.from(JSON.stringify({ v: 1, sessionKey, promptId: 'prompt' })), + key, + ), + ).toThrow('another Session'); +}); diff --git a/packages/core/src/managed-runtime/hosted-recovery-records.ts b/packages/core/src/managed-runtime/hosted-recovery-records.ts new file mode 100644 index 0000000000..030a2fd0ca --- /dev/null +++ b/packages/core/src/managed-runtime/hosted-recovery-records.ts @@ -0,0 +1,94 @@ +/** + * @license + * Copyright 2026 Qwen Team + * SPDX-License-Identifier: Apache-2.0 + */ + +import { + assertManagedSessionDurableRef, + assertManagedSessionKey, + assertManagedSessionStableId, + parseManagedSessionRecordJson, + ManagedSessionRecordError, + type ManagedSessionDurableRef, + type ManagedSessionJsonValue, + type ManagedSessionKey, +} from './managed-session-records.js'; + +export const HOSTED_RECOVERY_RESOURCE_KINDS = new Set([ + 'hosted-approval-continuation', + 'hosted-model-request', + 'hosted-turn-cleanup', +]); + +export function parseHostedRecoveryResource( + bytes: Buffer, + expectedKey?: ManagedSessionKey, +): Record { + const value = parseManagedSessionRecordJson( + bytes.toString('utf8'), + 64 * 1024, + ); + if ( + !value || + typeof value !== 'object' || + Array.isArray(value) || + value['v'] !== 1 + ) + throw new ManagedSessionRecordError('Invalid Hosted recovery resource.'); + const key = assertManagedSessionKey( + value['sessionKey'], + 'Hosted recovery sessionKey', + ); + assertManagedSessionStableId(value['promptId'], 'Hosted recovery promptId'); + if ( + expectedKey && + (key.tenantId !== expectedKey.tenantId || + key.workspaceId !== expectedKey.workspaceId || + key.sessionId !== expectedKey.sessionId) + ) + throw new ManagedSessionRecordError( + 'Hosted recovery resource names another Session.', + ); + return value; +} + +export function hostedRecoveryReferences( + value: unknown, +): ManagedSessionDurableRef[] { + const refs = new Map(); + if (!value || typeof value !== 'object' || Array.isArray(value)) return []; + const body = value as Record; + const pending: unknown[] = [ + 'definitionRef', + 'rootSnapshotRef', + 'assistantRef', + 'inputRef', + 'continuationRef', + ].flatMap((field) => (body[field] ? [body[field]] : [])); + if (Array.isArray(body['calls'])) + for (const call of body['calls']) { + if (!call || typeof call !== 'object') continue; + const member = call as Record; + pending.push(member['inputRef'], member['definitionRef']); + } + for (const item of pending) { + const ref = assertManagedSessionDurableRef( + item as ManagedSessionJsonValue, + 'Hosted recovery reference', + ); + const previous = refs.get(ref.resourceId); + if ( + previous && + (previous.kind !== ref.kind || + previous.schemaVersion !== ref.schemaVersion || + previous.byteLength !== ref.byteLength || + previous.digest !== ref.digest) + ) + throw new ManagedSessionRecordError( + 'Hosted recovery references conflict.', + ); + refs.set(ref.resourceId, ref); + } + return [...refs.values()]; +} diff --git a/packages/core/src/managed-runtime/http-managed-session-store.test.ts b/packages/core/src/managed-runtime/http-managed-session-store.test.ts index e06eb85e81..153cd018c9 100644 --- a/packages/core/src/managed-runtime/http-managed-session-store.test.ts +++ b/packages/core/src/managed-runtime/http-managed-session-store.test.ts @@ -249,6 +249,59 @@ describe('HTTP Managed Session store', () => { return { stores, session }; } + it.each([undefined, 1, 2])( + 'requires Store acknowledgement before a bounded retraction (version=%s)', + async (version) => { + const server = new FakeManagedSessionStore(); + server.supportedStorageVersion = version; + const { session } = await bootStoresAndSession(server); + try { + const before = server.commits.length; + const commit = session.authority.appendExecutionEvent( + { + operation: 'assistantRetract', + commandId: 'bounded', + sessionKey: SESSION_KEY, + contentDigest: 'a'.repeat(64), + }, + (sequence) => ({ + v: 1, + sequence, + eventId: 'bounded', + sessionKey: SESSION_KEY, + kind: 'message.retracted', + occurredAt: Date.now(), + subject: { + type: 'activation', + scopeId: session.activation.activationId, + ...session.activation, + }, + payload: { + messageId: 'message', + turnId: 'turn', + fromSequence: 1, + sourceBootId: 'boot', + sourceEventEpoch: 'epoch', + throughSequence: 1, + }, + }), + { class: 'harness', activation: session.activation }, + ); + if (version === 2) { + await commit; + expect(server.commits.length).toBe(before + 1); + } else { + await expect(commit).rejects.toThrow( + 'does not support Hosted recovery storage version 2', + ); + expect(server.commits.length).toBe(before); + } + } finally { + await session.close().catch(() => undefined); + } + }, + ); + // A committed message body must be the reader-facing record the cold // projection will replay: the commit-time fence refuses anything else. const MESSAGE_BODY = Buffer.from( @@ -327,6 +380,9 @@ describe('HTTP Managed Session store', () => { const headers = new Headers(init?.headers); expect(headers.get('X-Qwen-Tenant-Id')).toBe(SESSION_KEY.tenantId); expect(headers.get('X-Qwen-Managed-Writer-Token')).toBe(TOKEN_A); + expect(headers.get('X-Qwen-Managed-Max-Readable-Storage-Version')).toBe( + '2', + ); expect(init?.method).toBe('POST'); expect(JSON.parse(String(init?.body))).toEqual(request); return jsonResponse(request); @@ -1485,9 +1541,18 @@ describe('HTTP Managed Session store', () => { const referencedResources = server.commits.at(-1)!['resources'] as Array<{ resourceId: string; }>; + // The settled Turn carries the checkpoint, the checkpoint's own api + // history dependency, and the payload-referenced result record a cold + // projection replays. expect( referencedResources.map(({ resourceId }) => resourceId).sort(), - ).toEqual([checkpointRef.resourceId, turnResultRef.resourceId].sort()); + ).toEqual( + [ + checkpointRef.resourceId, + historyRef.resourceId, + turnResultRef.resourceId, + ].sort(), + ); const snapshotResources = new Map< string, { @@ -3018,7 +3083,7 @@ describe('HTTP Managed Session store', () => { it.each([ ['recoveryStatus', 'BLOCKED_RESOURCE'], - ['storageVersion', 2], + ['storageVersion', 3], ['state', 'SEALED'], ['writerGeneration', 2], ['compactedThroughRevision', 1], @@ -3033,7 +3098,7 @@ describe('HTTP Managed Session store', () => { await expect(journal.read()).resolves.toBeDefined(); server.headOverrides[key] = value; await expect(journal.read()).rejects.toThrow( - /not readable by this v1 writer/, + /not readable by this writer/, ); await session.close(); }, @@ -3446,6 +3511,7 @@ describe('HTTP Managed Session store', () => { }); class FakeManagedSessionStore { + supportedStorageVersion: number | undefined = 2; readonly commits: Array> = []; readonly recoveryBlocks: Array> = []; readonly headOverrides: Record = {}; @@ -3551,7 +3617,10 @@ class FakeManagedSessionStore { ...this.pageOverrides, }); } - if (suffix === '/transactions:commit') { + if ( + suffix === '/transactions:commit' || + suffix === '/transactions:commit-v2' + ) { const body = JSON.parse(String(init?.body)) as Record; this.commits.push(body); const resources = body['resources'] as Array>; @@ -3670,6 +3739,9 @@ class FakeManagedSessionStore { private grant(): Record { return { + ...(this.supportedStorageVersion === undefined + ? {} + : { supportedStorageVersion: this.supportedStorageVersion }), writerGeneration: this.writerGeneration, leaseUntil: this.leaseUntil, journalRevision: this.transactions.length, diff --git a/packages/core/src/managed-runtime/http-managed-session-store.ts b/packages/core/src/managed-runtime/http-managed-session-store.ts index 5cdba930ee..2f25b16989 100644 --- a/packages/core/src/managed-runtime/http-managed-session-store.ts +++ b/packages/core/src/managed-runtime/http-managed-session-store.ts @@ -28,6 +28,11 @@ import { } from './managed-session-records.js'; import type { ManagedSessionJsonValue } from './managed-session-inbox.js'; import { tryParseHarnessCheckpointV1 } from './managed-harness-checkpoint.js'; +import { + HOSTED_RECOVERY_RESOURCE_KINDS, + hostedRecoveryReferences, + parseHostedRecoveryResource, +} from './hosted-recovery-records.js'; import { MANAGED_EXTENSION_RECORD_BODIES } from './managed-extension-projection.js'; import { MANAGED_TOOL_RESULT_KINDS, @@ -179,7 +184,7 @@ export function readOnlyManagedSessionSnapshot(value: unknown) { ); const head = parseRestoreHead(snapshot['head']); if ( - head.storageVersion !== 1 || + ![1, 2].includes(head.storageVersion) || !['ACTIVE', 'SEALED'].includes(head.state) || head.recoveryStatus !== 'READY' || head.compactedThroughRevision !== 0 @@ -482,7 +487,9 @@ class HttpManagedSessionResourceStore implements ManagedSessionResourceStore { return this.client.readResource(validated); } - commitResources(refs: readonly ManagedSessionDurableRef[]): CommitResource[] { + async commitResources( + refs: readonly ManagedSessionDurableRef[], + ): Promise { const closure = new Map(); const pending = [...refs]; while (pending.length > 0) { @@ -505,6 +512,8 @@ class HttpManagedSessionResourceStore implements ManagedSessionResourceStore { if (staged !== undefined) { requireSameRef(staged.ref, ref); pending.push(...collectNestedResourceRefs(ref, staged.bytes)); + } else if (isReferenceContainer(ref.kind)) { + pending.push(...collectNestedResourceRefs(ref, await this.read(ref))); } } return [...closure.values()].map((ref) => { @@ -533,6 +542,7 @@ class HttpManagedSessionResourceStore implements ManagedSessionResourceStore { } interface WriterGrant { + readonly supportedStorageVersion: number; readonly writerGeneration: number; readonly leaseUntil: number; readonly journalRevision: number; @@ -770,6 +780,7 @@ class ManagedSessionStoreHttpClient { throw corrupt('journal bytes do not match the durable head.'); } this.grant = { + ...this.requireGrant(), writerGeneration: head.writerGeneration, leaseUntil: this.requireGrant().leaseUntil, journalRevision: head.journalRevision, @@ -795,7 +806,37 @@ class ManagedSessionStoreHttpClient { 'the transaction does not extend the HTTP journal head.', ); } - const commitResources = this.resources.commitResources(descriptor.refs); + const commitResources = await this.resources.commitResources( + descriptor.refs, + ); + const needsVersionTwo = + commitResources.some( + (resource) => + HOSTED_RECOVERY_RESOURCE_KINDS.has(resource.kind) || + (resource.kind === 'managed-action-options' && + resource.bytesBase64 && + JSON.parse( + Buffer.from(resource.bytesBase64, 'base64').toString('utf8'), + ).v === 3), + ) || + records.some((record) => { + const event = ( + record as { + managedSession?: { + kind?: string; + payload?: Record; + }; + } + ).managedSession; + return ( + event?.kind === 'message.retracted' && + event.payload?.['sourceBootId'] !== undefined + ); + }); + if (needsVersionTwo && grant.supportedStorageVersion < 2) + throw new ManagedSessionRecordError( + 'Managed Session Store does not support Hosted recovery storage version 2.', + ); const publicationIds = new Set( descriptor.refs.flatMap((ref) => { const id = this.publicationAdmissions.get(ref.resourceId); @@ -836,7 +877,13 @@ class ManagedSessionStoreHttpClient { try { committed = publicationId === undefined - ? await this.json('/transactions:commit', 'POST', commitBody) + ? await this.json( + needsVersionTwo + ? '/transactions:commit-v2' + : '/transactions:commit', + 'POST', + commitBody, + ) : await this.publicationRequest( `/publications/${encodeURIComponent(publicationId)}/receipts/commit`, commitBody, @@ -965,6 +1012,7 @@ class ManagedSessionStoreHttpClient { method: readOnly ? 'GET' : 'POST', headers: { Accept: 'application/json', + 'X-Qwen-Managed-Max-Readable-Storage-Version': '2', ...(readOnly ? {} : { 'Content-Type': 'application/json' }), [HTTP_MANAGED_SESSION_STORE_CONTRACT.tenantHeader]: this.sessionKey.tenantId, @@ -1188,7 +1236,11 @@ class ManagedSessionStoreHttpClient { if (renewed.writerGeneration !== grant.writerGeneration) { throw corrupt('writer generation changed during renewal.'); } - this.grant = { ...this.requireGrant(), leaseUntil: renewed.leaseUntil }; + this.grant = { + ...this.requireGrant(), + leaseUntil: renewed.leaseUntil, + supportedStorageVersion: renewed.supportedStorageVersion, + }; this.scheduleRenewal(); })(); this.renewPromise = renewal.finally(() => { @@ -1222,12 +1274,12 @@ class ManagedSessionStoreHttpClient { const grant = this.requireGrant(); if ( head.state !== 'ACTIVE' || - head.storageVersion !== 1 || + ![1, 2].includes(head.storageVersion) || head.writerGeneration !== grant.writerGeneration || head.recoveryStatus !== 'READY' || head.compactedThroughRevision !== 0 ) { - throw corrupt('restore head is not readable by this v1 writer.'); + throw corrupt('restore head is not readable by this writer.'); } // A head below this writer's committed position is a server-side // regression; re-baselining onto it would silently forget our own @@ -1356,6 +1408,7 @@ class ManagedSessionStoreHttpClient { redirect: 'error', headers: { Accept: accept, + 'X-Qwen-Managed-Max-Readable-Storage-Version': '2', [HTTP_MANAGED_SESSION_STORE_CONTRACT.tenantHeader]: this.sessionKey.tenantId, [HTTP_MANAGED_SESSION_STORE_CONTRACT.writerTokenHeader]: @@ -1681,6 +1734,13 @@ export function requireStoredTransactionMatches( function parseWriterGrant(value: unknown): WriterGrant { const record = asRecord(value, 'writer grant'); return { + supportedStorageVersion: + record['supportedStorageVersion'] === undefined + ? 1 + : safeCounter( + record['supportedStorageVersion'], + 'supportedStorageVersion', + ), writerGeneration: safeCounter( record['writerGeneration'], 'writerGeneration', @@ -1750,6 +1810,9 @@ export function collectNestedResourceRefs( const parsed = tryParseHarnessCheckpointV1(bytes); return parsed.ok ? collectRefs([parsed.checkpoint]) : []; } + if (HOSTED_RECOVERY_RESOURCE_KINDS.has(ref.kind)) { + return hostedRecoveryReferences(parseHostedRecoveryResource(bytes)); + } if ( EXTENSION_RECORD_KINDS.has(ref.kind) || ref.kind === 'managed-definition' || @@ -1799,6 +1862,20 @@ export function collectNestedResourceRefs( return []; } +function isReferenceContainer(kind: string): boolean { + return ( + EXTENSION_RECORD_KINDS.has(kind) || + HOSTED_RECOVERY_RESOURCE_KINDS.has(kind) || + [ + 'managed-checkpoint', + 'managed-action-options', + 'managed-hook-plan', + 'managed-hook-message-chunks', + MANAGED_MESSAGE_CHUNKS_KIND, + ].includes(kind) + ); +} + function requireSameRef( left: ManagedSessionDurableRef, right: ManagedSessionDurableRef, diff --git a/packages/core/src/managed-runtime/managed-harness-factory.test.ts b/packages/core/src/managed-runtime/managed-harness-factory.test.ts index db620c9ac4..789704b648 100644 --- a/packages/core/src/managed-runtime/managed-harness-factory.test.ts +++ b/packages/core/src/managed-runtime/managed-harness-factory.test.ts @@ -562,7 +562,25 @@ describe('managed harness factory', () => { const handle = createManagedHarnessHandle(session); await handle.ensureRunnable(); const refs = await waitRefs(session); + const beforeWait = session.authority.committedSequence; const boundary = await handle.commitDurableWait(waitCommit(refs)); + const waitEvents = session.authority.eventsInSequenceRange( + beforeWait + 1, + session.authority.committedSequence, + ); + expect(waitEvents.map((event) => event.kind)).toEqual([ + 'action.changed', + 'checkpoint.committed', + ]); + const journal = await fs.readFile(workspace.transcriptPath, 'utf8'); + const records = journal + .trim() + .split('\n') + .map((line) => JSON.parse(line)); + const markers = records.filter( + (record) => record.subtype === 'managed_session_commit_v1', + ); + expect(markers.at(-1).managedSession.eventCount).toBe(2); expect(boundary.kind).toBe('durable_wait'); expect(session.authority.latestCheckpoint?.boundary).toBe( HARNESS_DURABLE_WAIT_BOUNDARY, diff --git a/packages/core/src/managed-runtime/managed-harness-factory.ts b/packages/core/src/managed-runtime/managed-harness-factory.ts index 92cdfa064e..96df4de452 100644 --- a/packages/core/src/managed-runtime/managed-harness-factory.ts +++ b/packages/core/src/managed-runtime/managed-harness-factory.ts @@ -199,6 +199,11 @@ export interface ManagedHarnessHandle { * `model_output_committed`. No-op when the handle is not waiting. */ resolveDurableWait(): Promise; + /** Rebinds a proven saved native pre-dispatch plan to the current owner. */ + adoptPreparedContinuation( + promptId: string, + settledResults?: boolean, + ): Promise; /** * Commits the agent-wait safety point: admitted foreground child runs as * `await_agent` with `durable_wait`. Agent calls bypass the Broker @@ -492,25 +497,6 @@ class LocalManagedHarnessHandle implements ManagedHarnessHandle { }, } : runnable; - if (request.source === 'tool_call') { - await this.authority.requestToolAction( - { - operation: 'requestToolAction', - commandId: `requestToolAction:${request.requestId}`, - sessionKey: this.authority.sessionHeader.sessionKey, - contentDigest: createHash('sha256') - .update(request.optionsRef.digest) - .digest('hex'), - }, - { - requestId: request.requestId, - kind: request.kind, - inputRevision: 1, - optionsRef: request.optionsRef, - }, - { class: 'harness', activation: this.activation }, - ); - } const identity = this.nextCheckpointIdentity(); const checkpoint = createAwaitActionHarnessCheckpoint({ previous, @@ -540,6 +526,7 @@ class LocalManagedHarnessHandle implements ManagedHarnessHandle { `harness:await_action:${this.activation.activationId}:${request.requestId}:${identity.coveredSequence}`, checkpoint, HARNESS_DURABLE_WAIT_BOUNDARY, + request.source === 'tool_call' ? request : undefined, ); const committed = this.authority.latestCheckpoint; if (committed === undefined) { @@ -581,7 +568,13 @@ class LocalManagedHarnessHandle implements ManagedHarnessHandle { } const identity = this.nextCheckpointIdentity(); const checkpoint = createModelOutputCommittedHarnessCheckpoint({ - previous, + previous: { + ...previous, + identity: { + ...previous.identity, + activationId: this.activation.activationId, + }, + }, ...identity, }); await this.commitHarnessCheckpoint( @@ -840,6 +833,67 @@ class LocalManagedHarnessHandle implements ManagedHarnessHandle { }); } + async adoptPreparedContinuation( + promptId: string, + settledResults?: boolean, + ): Promise { + await this.mutateCheckpoint(async () => { + this.assertNotDetached(); + this.assertCurrentActivation(); + const previous = (await this.requireRunnableAuthorization()).checkpoint; + const startsNewTurn = + previous.continuation.phase === 'turn_settled' && + (previous.tools?.items ?? []).every( + (item) => item.state === 'settled' && item.consumed, + ) && + this.authority + .eventsInSequenceRange(1, this.authority.committedSequence) + .some( + (event) => + event.kind === 'turn.settled' && + event.payload['turnId'] === previous.identity.turnId, + ); + if ( + ![ + 'before_model', + 'model_output_committed', + ...(settledResults ? ['results_ready', 'turn_settled'] : []), + ...(startsNewTurn ? ['turn_settled'] : []), + ].includes(previous.continuation.phase) || + (previous.continuation.phase !== 'before_model' && + !startsNewTurn && + previous.identity.promptId !== promptId) || + !(previous.tools?.items ?? []).every( + (item) => + item.state === 'settled' && (settledResults || item.consumed), + ) + ) + throw new ManagedSessionConflictError( + 'A saved native plan cannot adopt unpaid Runtime work.', + ); + const identity = this.nextCheckpointIdentity(); + const checkpoint = { + ...previous, + resume: { + ...previous.resume, + throughSequence: identity.coveredSequence, + }, + identity: { + ...previous.identity, + ...identity, + activationId: this.activation.activationId, + turnId: promptId, + promptId, + }, + }; + await this.commitHarnessCheckpoint( + `harness:prepared_adopt:${this.activation.activationId}:${promptId}:${checkpoint.identity.coveredSequence}`, + checkpoint, + null, + ); + }); + } + async commitAwaitRuntime( request: ManagedAwaitRuntimeCommit, ): Promise { @@ -1280,6 +1334,7 @@ class LocalManagedHarnessHandle implements ManagedHarnessHandle { commandId: string, checkpoint: HarnessCheckpointV1, boundary: string | null, + action?: ManagedDurableWaitCommit, ): Promise { const header = this.authority.sessionHeader; const state = encodeHarnessCheckpointV1(checkpoint); @@ -1290,7 +1345,20 @@ class LocalManagedHarnessHandle implements ManagedHarnessHandle { sessionKey: header.sessionKey, contentDigest: createHash('sha256').update(state).digest('hex'), }, - { state, boundary }, + { + state, + boundary, + ...(action + ? { + toolAction: { + requestId: action.requestId, + kind: action.kind, + inputRevision: 1, + optionsRef: action.optionsRef, + }, + } + : {}), + }, { class: 'harness', activation: this.activation }, ); } diff --git a/packages/core/src/managed-runtime/managed-hook-activation.ts b/packages/core/src/managed-runtime/managed-hook-activation.ts index a3c2437d26..85d7781cba 100644 --- a/packages/core/src/managed-runtime/managed-hook-activation.ts +++ b/packages/core/src/managed-runtime/managed-hook-activation.ts @@ -132,7 +132,10 @@ class LocalManagedHookModelScope { if (shared.snapshot) budget.beginTurn(shared.snapshot); } - async beginMainAttempt(model: string): Promise { + async beginMainAttempt( + model: string, + recoveryRef?: ManagedSessionDurableRef, + ): Promise { this.assertOwner(); if (this.operation) throw new ManagedSessionConflictError( @@ -140,10 +143,15 @@ class LocalManagedHookModelScope { ); const attemptId = `${this.turnId}:main:${randomUUID()}`; return Object.assign( - await this.beginAttempt(attemptId, 'hosted', { - turnId: this.turnId, - model, - }), + await this.beginAttempt( + attemptId, + 'hosted', + { + turnId: this.turnId, + model, + }, + recoveryRef, + ), { attemptId }, ); } @@ -152,6 +160,7 @@ class LocalManagedHookModelScope { attemptId: string, kind: 'hook' | 'hosted', route: object, + recoveryRef?: ManagedSessionDurableRef, ): Promise<(success: boolean, usage: unknown) => Promise> { const authority = this.session.authority; const activation = this.session.activation; @@ -185,7 +194,14 @@ class LocalManagedHookModelScope { scopeId: activation.activationId, ...activation, }, - payload: { attemptId, routeRef, inputCheckpointRef, state, usageRef }, + payload: { + attemptId, + routeRef, + inputCheckpointRef, + state, + usageRef, + ...(recoveryRef ? { recoveryRef } : {}), + }, }), { class: 'harness', activation }, ); diff --git a/packages/core/src/managed-runtime/managed-session-authority.ts b/packages/core/src/managed-runtime/managed-session-authority.ts index e6c4eedf3b..4e52d346de 100644 --- a/packages/core/src/managed-runtime/managed-session-authority.ts +++ b/packages/core/src/managed-runtime/managed-session-authority.ts @@ -1098,7 +1098,16 @@ export class LocalManagedSessionAuthority { */ async commitCheckpoint( command: ManagedSessionCommand, - request: { state: Buffer; boundary: string | null }, + request: { + state: Buffer; + boundary: string | null; + toolAction?: { + readonly requestId: string; + readonly kind: string; + readonly inputRevision: number; + readonly optionsRef: ManagedSessionDurableRef; + }; + }, actor: ManagedSessionActor, ): Promise { const store = this.resources; @@ -1114,8 +1123,12 @@ export class LocalManagedSessionAuthority { ); } return this.runSerial(async () => { + const action = request.toolAction; + if (action && this.actions.has(action.requestId)) { + throw new ManagedSessionConflictError('The approval already exists.'); + } const previous = this.checkpoint; - const checkpointId = `ckpt-${this.committed + 1}`; + const checkpointId = `ckpt-${this.committed + (action ? 2 : 1)}`; const covered = this.committed; const previousCheckpointId = previous?.checkpointId ?? null; const parsed = tryParseHarnessCheckpointV1(request.state); @@ -1143,9 +1156,33 @@ export class LocalManagedSessionAuthority { const receipt = await this.commit( command, [ + ...(action + ? [ + { + v: MANAGED_SESSION_FORMAT_VERSION, + sequence: this.committed + 1, + eventId: `action:${action.requestId}:requested`, + sessionKey: command.sessionKey, + kind: 'action.changed', + occurredAt: this.now(), + subject: { + type: 'activation', + scopeId: held.activationId, + activationId: held.activationId, + epoch: held.epoch, + }, + payload: { + ...action, + source: 'tool_call', + state: 'requested', + decisionRef: null, + }, + }, + ] + : []), { v: MANAGED_SESSION_FORMAT_VERSION, - sequence: this.committed + 1, + sequence: this.committed + (action ? 2 : 1), eventId: checkpointId, sessionKey: command.sessionKey, kind: 'checkpoint.committed', @@ -1165,7 +1202,7 @@ export class LocalManagedSessionAuthority { }, }, ], - [actor], + action ? [actor, actor] : [actor], ); const checkpoint = this.checkpoint; if (checkpoint === undefined) { diff --git a/packages/core/src/managed-runtime/managed-session-records.test.ts b/packages/core/src/managed-runtime/managed-session-records.test.ts index fd4edc62e8..be9facbaa8 100644 --- a/packages/core/src/managed-runtime/managed-session-records.test.ts +++ b/packages/core/src/managed-runtime/managed-session-records.test.ts @@ -1291,3 +1291,88 @@ describe('managed session raw record parsing', () => { expect(parseJson(over, cap)).toThrow(/exceeds 65536 UTF-8 bytes/); }); }); + +describe('Hosted recovery envelopes', () => { + it('keeps old attempts readable and validates new references', () => { + expect(parseManagedSessionEvent(harnessEvent()).kind).toBe('model.attempt'); + const payload = { + attemptId: 'a', + routeRef: ref(), + inputCheckpointRef: null, + state: 'started', + usageRef: null, + recoveryRef: ref('hosted-model-request'), + }; + expect( + parseManagedSessionEvent(harnessEvent({ payload })).payload[ + 'recoveryRef' + ], + ).toEqual(payload.recoveryRef); + expect(() => + parseManagedSessionEvent( + harnessEvent({ payload: { ...payload, recoveryRef: ref() } }), + ), + ).toThrow(/recoveryRef/); + }); + it('requires a complete bounded retraction and ordered range', () => { + const event = (payload: unknown) => + harnessEvent({ kind: 'message.retracted', payload }); + const payload = { messageId: 'msg', turnId: 'turn', fromSequence: 2 }; + expect(parseManagedSessionEvent(event(payload)).kind).toBe( + 'message.retracted', + ); + expect(() => + parseManagedSessionEvent(event({ ...payload, throughSequence: 3 })), + ).toThrow(/together/); + const bounded = { + ...payload, + throughSequence: 3, + sourceBootId: 'boot', + sourceEventEpoch: 'epoch', + }; + expect(parseManagedSessionEvent(event(bounded)).payload).toEqual(bounded); + expect(() => + parseManagedSessionEvent(event({ ...bounded, throughSequence: 1 })), + ).toThrow(/range/); + }); + it('validates batch plans and cleanup states under Harness ownership', () => { + for (const [kind, payload] of [ + [ + 'hosted.batch.planned', + { + batchId: 'batch', + planRevision: 1, + planRef: ref('hosted-approval-continuation'), + }, + ], + [ + 'hosted.cleanup', + { + cleanupId: 'cleanup', + descriptorRef: ref('hosted-turn-cleanup'), + state: 'owed', + }, + ], + ] as const) { + const event = parseManagedSessionEvent(harnessEvent({ kind, payload })); + expect(() => + assertManagedSessionEventActor(event, 'harness'), + ).not.toThrow(); + expect(() => + assertManagedSessionEventActor(event, 'trusted_entry'), + ).toThrow(); + } + expect(() => + parseManagedSessionEvent( + harnessEvent({ + kind: 'hosted.cleanup', + payload: { + cleanupId: 'c', + descriptorRef: ref('hosted-turn-cleanup'), + state: 'released', + }, + }), + ), + ).toThrow(); + }); +}); diff --git a/packages/core/src/managed-runtime/managed-session-records.ts b/packages/core/src/managed-runtime/managed-session-records.ts index 70d6ac29b9..556ca48a37 100644 --- a/packages/core/src/managed-runtime/managed-session-records.ts +++ b/packages/core/src/managed-runtime/managed-session-records.ts @@ -70,6 +70,8 @@ export const MANAGED_SESSION_EVENT_KINDS = [ 'message.delta', 'message.retracted', 'operation.replayed', + 'hosted.batch.planned', + 'hosted.cleanup', ] as const; export type ManagedSessionEventKind = @@ -888,7 +890,9 @@ const EVENT_SCHEMAS: Readonly> = inputCheckpointRef: 'refOrNull', state: 'text', usageRef: 'refOrNull', + recoveryRef: 'ref', }, + optional: ['recoveryRef'], }, 'message.committed': { fields: { @@ -926,7 +930,17 @@ const EVENT_SCHEMAS: Readonly> = messageId: 'id', turnId: 'id', fromSequence: 'sequence', + sourceBootId: 'id', + sourceEventEpoch: 'id', + throughSequence: 'sequence', }, + optional: ['sourceBootId', 'sourceEventEpoch', 'throughSequence'], + }, + 'hosted.batch.planned': { + fields: { batchId: 'id', planRevision: 'sequence', planRef: 'ref' }, + }, + 'hosted.cleanup': { + fields: { cleanupId: 'id', descriptorRef: 'ref', state: 'text' }, }, 'action.changed': { fields: { @@ -1035,6 +1049,8 @@ const EVENT_ACTORS: Readonly< 'tool.intent': ['harness'], 'message.delta': ['harness'], 'message.retracted': ['harness'], + 'hosted.batch.planned': ['harness'], + 'hosted.cleanup': ['harness'], 'action.changed': ['harness', 'trusted_entry'], 'tool.receipt': ['trusted_entry'], 'checkpoint.committed': ['harness'], @@ -1058,6 +1074,8 @@ const ACTIVATION_SUBJECT_KINDS: Readonly< 'tool.intent': true, 'message.delta': true, 'message.retracted': true, + 'hosted.batch.planned': true, + 'hosted.cleanup': true, 'action.changed': false, 'tool.receipt': false, 'checkpoint.committed': true, @@ -1221,6 +1239,59 @@ function assertPayloadRules( if (state === 'started' && payload['usageRef'] !== null) { fail(`${at}.usageRef must be null while the attempt is started.`); } + if (Object.hasOwn(payload, 'recoveryRef')) { + const ref = payload[ + 'recoveryRef' + ] as unknown as ManagedSessionDurableRef; + if (ref.kind !== 'hosted-model-request' || ref.schemaVersion !== 1) { + fail(`${at}.recoveryRef must be a hosted-model-request/1.`); + } + } + return; + } + case 'hosted.batch.planned': { + if ((payload['planRevision'] as number) < 1) { + fail(`${at}.planRevision must start at 1.`); + } + const ref = payload['planRef'] as unknown as ManagedSessionDurableRef; + if ( + ref.kind !== 'hosted-approval-continuation' || + ref.schemaVersion !== 1 + ) { + fail(`${at}.planRef must be a hosted-approval-continuation/1.`); + } + return; + } + case 'hosted.cleanup': { + assertEnum( + payload['state'], + ['owed', 'confirmed'] as const, + `${at}.state`, + ); + const ref = payload[ + 'descriptorRef' + ] as unknown as ManagedSessionDurableRef; + if (ref.kind !== 'hosted-turn-cleanup' || ref.schemaVersion !== 1) { + fail(`${at}.descriptorRef must be a hosted-turn-cleanup/1.`); + } + return; + } + case 'message.retracted': { + const fields = ['sourceBootId', 'sourceEventEpoch', 'throughSequence']; + const count = fields.filter((name) => + Object.hasOwn(payload, name), + ).length; + if (count !== 0 && count !== fields.length) { + fail(`${at} bounded source fields must be present together.`); + } + if ( + (payload['fromSequence'] as number) < 1 || + (count > 0 && + (payload['throughSequence'] as number) < + (payload['fromSequence'] as number)) + ) { + fail(`${at} retraction range is invalid.`); + } return; } case 'action.changed': { @@ -1348,7 +1419,6 @@ function assertPayloadRules( case 'input.accepted': case 'tool.intent': case 'message.delta': - case 'message.retracted': case 'tool.receipt': case 'cancel.requested': case 'turn.settled': diff --git a/packages/core/src/services/image-payload-references.ts b/packages/core/src/services/image-payload-references.ts index 4b025fd462..45cd46f1e2 100644 --- a/packages/core/src/services/image-payload-references.ts +++ b/packages/core/src/services/image-payload-references.ts @@ -57,6 +57,10 @@ export function countAllInlineImages(contents: Content[]): number { return count; } +export function hasImagePayloadReferences(contents: Content[]): boolean { + return collectReferencedImageIds(contents).size > 0; +} + /** * Replace image payloads in-place with text references, storing the * originals in the provided store. This mutates the history so that diff --git a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/api/ManagedSessionStoreController.java b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/api/ManagedSessionStoreController.java index dc0023570e..5332ce5087 100644 --- a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/api/ManagedSessionStoreController.java +++ b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/api/ManagedSessionStoreController.java @@ -114,7 +114,7 @@ public class ManagedSessionStoreController { writerToken, request); } - @PostMapping("/transactions:commit") + @PostMapping({"/transactions:commit", "/transactions:commit-v2"}) public CommitReceipt commit(TenantContext tenant, @PathVariable String sessionId, @RequestHeader(ManagedSessionStoreModels.WRITER_TOKEN_HEADER) diff --git a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/service/HarnessCoordinator.java b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/service/HarnessCoordinator.java index ff3b6119ae..c109ebcfb9 100644 --- a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/service/HarnessCoordinator.java +++ b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/service/HarnessCoordinator.java @@ -713,10 +713,22 @@ public class HarnessCoordinator { flush(turn, stream.eventEpoch(), batch, leaseLost); batchBytes = 0; flushAt = 0; - store.retractHarnessTurnOutput(turn.tenantId(), - turn.sessionId(), turn.turnId(), owner, - stream.eventEpoch(), retractionFromSequence(source), - source.id()); + Map retraction = source.data() instanceof Map data ? data : Map.of(); + if (retraction.containsKey("throughSequence")) { + Object through = retraction.get("throughSequence"); + Object sourceBoot = retraction.get("sourceBootId"); + Object sourceEpoch = retraction.get("sourceEventEpoch"); + if (!(through instanceof Number) || !(sourceBoot instanceof String) || !(sourceEpoch instanceof String)) { + throw new IllegalStateException("Invalid bounded Hosted retraction source"); + } + store.retractHarnessTurnOutput(turn.tenantId(), turn.sessionId(), turn.turnId(), owner, + stream.eventEpoch(), retractionFromSequence(source), source.id(), + (String) sourceBoot, (String) sourceEpoch, ((Number) through).longValue()); + } else { + store.retractHarnessTurnOutput(turn.tenantId(), turn.sessionId(), turn.turnId(), owner, + stream.eventEpoch(), retractionFromSequence(source), source.id()); + } + // The replay's first chunk is the new first visible text: // the transcript it replaces was just blanked. flushFirstVisibleText = true; diff --git a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/service/ManagedActionService.java b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/service/ManagedActionService.java index 4d15525e7e..a3739f5dc2 100644 --- a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/service/ManagedActionService.java +++ b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/service/ManagedActionService.java @@ -267,7 +267,7 @@ public class ManagedActionService { JsonNode options = action.options(); String tool = options.path("toolName").asText(); if (!"requested".equals(action.state()) - || options.path("v").asLong() != 2 + || !List.of(2L, 3L).contains(options.path("v").asLong()) || !PREVIEW_TOOLS.contains(tool)) { return null; } diff --git a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/AgentStateStore.java b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/AgentStateStore.java index 58e82f1b25..4e4cc5b529 100644 --- a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/AgentStateStore.java +++ b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/AgentStateStore.java @@ -475,6 +475,11 @@ public interface AgentStateStore { String turnId, String owner, String eventEpoch, long fromSourceId, long retractionSourceId); + void retractHarnessTurnOutput(String tenantId, String sessionId, + String turnId, String owner, String eventEpoch, + long fromSourceId, long retractionSourceId, String sourceBootId, + String sourceEventEpoch, Long throughSourceId); + void recordHarnessEvents(String tenantId, String sessionId, String turnId, String owner, String eventEpoch, List events); diff --git a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedActionStore.java b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedActionStore.java index b081216404..f72da05493 100644 --- a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedActionStore.java +++ b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedActionStore.java @@ -152,6 +152,11 @@ public class ManagedActionStore { int eventCount, byte[] bytes, Function resources) { + apply(tenantId, workspaceId, sessionId, firstSequence, eventCount, bytes, resources, new java.util.HashMap<>()); + } + + void apply(String tenantId, String workspaceId, String sessionId, long firstSequence, + int eventCount, byte[] bytes, Function resources, Map hostedReferences) { int index = -1; for (String line : new String(bytes, StandardCharsets.UTF_8).split("\n")) { index++; @@ -233,14 +238,17 @@ public class ManagedActionStore { "createdAt", "expiresAt", "options")); - if (options.path("v").asLong() == 2) { + if (options.path("v").asLong() == 2 || options.path("v").asLong() == 3 && options.has("inputRef")) { optionFields.add("inputRef"); } + if (options.path("v").asLong() == 3) { + optionFields.add("continuationRef"); + } closed(options, optionFields.toArray(String[]::new)); require( safeNumber(options.path("v")) && (options.path("v").asLong() == 1 - || options.path("v").asLong() == 2) + || options.path("v").asLong() == 2 || options.path("v").asLong() == 3) && id.equals(options.path("requestId").asText()) && safeNumber(options.path("inputRevision")) && options.path("inputRevision").asLong() @@ -264,9 +272,12 @@ public class ManagedActionStore { closed(option, "id", "label"); require(text(option.path("label"))); } - if (options.path("v").asLong() == 2) { + if (options.has("inputRef")) { resource(options.path("inputRef"), "managed-tool-input", resources); } + if (options.path("v").asLong() == 3) { + ManagedHostedRecoveryRecords.reference(options.path("continuationRef"), "hosted-approval-continuation", tenantId, workspaceId, sessionId, resources, hostedReferences); + } Action previous = find(tenantId, sessionId, id).orElse(null); require( previous == null diff --git a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedAgentStore.java b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedAgentStore.java index 218d9ec790..58fb652e93 100644 --- a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedAgentStore.java +++ b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedAgentStore.java @@ -1394,7 +1394,7 @@ public class ManagedAgentStore implements AgentStateStore { requireNoOpenOperation(tenantId, sessionId); validateOperationStart(session, kind); if (protocolVersion == 1) { - WorkspaceLifecycleStore.requireIdleJournal(jdbc, objectMapper, tenantId, sessionId); + WorkspaceLifecycleStore.requireIdleJournal(jdbc, objectMapper, tenantId, sessionId, true); } long now = lifecycleDatabaseTime(); String operationId = publicId("op"); @@ -3067,6 +3067,20 @@ public class ManagedAgentStore implements AgentStateStore { public void retractHarnessTurnOutput(String tenantId, String sessionId, String turnId, String owner, String eventEpoch, long fromSourceId, long retractionSourceId) { + retractHarnessTurnOutput(tenantId, sessionId, turnId, owner, eventEpoch, + fromSourceId, retractionSourceId, null, null, null); + } + + @Transactional + public void retractHarnessTurnOutput(String tenantId, String sessionId, + String turnId, String owner, String eventEpoch, + long fromSourceId, long retractionSourceId, String sourceBootId, + String sourceEventEpoch, Long throughSourceId) { + boolean bounded = throughSourceId != null; + if (bounded && (sourceBootId == null || sourceEventEpoch == null + || fromSourceId < 1 || throughSourceId < fromSourceId || throughSourceId >= retractionSourceId)) { + throw new IllegalArgumentException("Invalid bounded Hosted retraction"); + } SessionRecord session = requireSessionForUpdate(tenantId, sessionId); TurnRecord turn = requireTurnForUpdate(tenantId, sessionId, turnId); long now = clock.millis(); @@ -3079,9 +3093,10 @@ public class ManagedAgentStore implements AgentStateStore { throw new IllegalStateException( "Hosted Harness event epoch changed"); } - String sourcePrefix = session.harnessBootId() + ":" + eventEpoch + ":"; + String sourcePrefix = (bounded ? sourceBootId : session.harnessBootId()) + ":" + + (bounded ? sourceEventEpoch : eventEpoch) + ":"; String reconciliationKey = "reconcile:inband:" + sourcePrefix + turnId - + ":" + fromSourceId; + + ":" + fromSourceId + (bounded ? ":" + throughSourceId : ""); if (!hasSourceEvent(tenantId, sessionId, reconciliationKey)) { jdbc.queryForObject("SELECT covered_sequence FROM" + " managed_agent_consumer_progress WHERE tenant_id" @@ -3097,8 +3112,9 @@ public class ManagedAgentStore implements AgentStateStore { long firstRetracted = Long.MAX_VALUE; for (EventRecord event : deltas) { if (event.sourceKey() == null - || !event.sourceKey().startsWith(sourcePrefix) - || sourceIdOf(event.sourceKey()) < fromSourceId) { + || (!bounded && !event.sourceKey().startsWith(sourcePrefix)) + || sourceIdOf(event.sourceKey()) < fromSourceId + || (bounded && sourceIdOf(event.sourceKey()) > throughSourceId)) { continue; } Map data = new LinkedHashMap<>(event.data()); diff --git a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedEventEnvelopes.java b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedEventEnvelopes.java index 196e312ba5..ded7f52433 100644 --- a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedEventEnvelopes.java +++ b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedEventEnvelopes.java @@ -31,7 +31,7 @@ public final class ManagedEventEnvelopes { // assert rejects exactly MAX_SAFE_INTEGER as "cannot advance"). static final long MAX_SEQUENCE = 9007199254740990L; - // Ordered (the failure message joins it) — the frozen contract's 18-kind + // Ordered (the failure message joins it) — the frozen contract's 20-kind // vocabulary in fixture order. static final List KINDS = List.of( "input.accepted", "wake.requested", "activation.changed", @@ -39,7 +39,8 @@ public final class ManagedEventEnvelopes { "action.changed", "tool.receipt", "checkpoint.committed", "context.compacted", "cancel.requested", "turn.settled", "config.bound", "lifecycle.changed", "domain.committed", - "message.delta", "message.retracted", "operation.replayed"); + "message.delta", "message.retracted", "operation.replayed", + "hosted.batch.planned", "hosted.cleanup"); static final Set FORBIDDEN_FIELDS = Set.of( "absolutePath", "localPath", "pid", "pod", "runtimeBindingId", diff --git a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedExtensionRecords.java b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedExtensionRecords.java index b74db84a44..786c27f062 100644 --- a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedExtensionRecords.java +++ b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedExtensionRecords.java @@ -42,7 +42,7 @@ public final class ManagedExtensionRecords { "tool.receipt", "checkpoint.committed", "context.compacted", "cancel.requested", "turn.settled", "config.bound", "lifecycle.changed", "domain.committed", "message.delta", - "message.retracted", "operation.replayed"); + "message.retracted", "operation.replayed", "hosted.batch.planned", "hosted.cleanup"); public static final int MAX_ID_BYTES = 512; public static final int MAX_TEXT_BYTES = 4096; public static final int MAX_GRANT_PHASES = 16; diff --git a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedHostedRecoveryRecords.java b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedHostedRecoveryRecords.java new file mode 100644 index 0000000000..2c1ef7abfa --- /dev/null +++ b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedHostedRecoveryRecords.java @@ -0,0 +1,296 @@ +package com.alibaba.qwen.code.managedagent.store; + +import com.alibaba.qwen.code.managedagent.api.ApiException; +import com.alibaba.qwen.code.managedagent.store.ManagedSessionStoreModels.StoredResource; +import com.fasterxml.jackson.databind.JsonNode; +import java.nio.charset.StandardCharsets; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.function.Function; +import org.springframework.http.HttpStatus; + +/** The private Hosted continuation formats and their reference closure. */ +public final class ManagedHostedRecoveryRecords { + public static final Set KINDS = Set.of("hosted-approval-continuation", + "hosted-model-request", "hosted-turn-cleanup"); + + private ManagedHostedRecoveryRecords() {} + + public static boolean requiresVersionTwo(byte[] records, + List resources) { + for (var resource : resources == null ? List.of() : resources) { + if (KINDS.contains(resource.kind())) { + return true; + } + } + for (String line : new String(records, StandardCharsets.UTF_8).split("\n")) { + JsonNode record = ManagedExtensionRecordStore.parse(line); + if (record == null) { + continue; + } + JsonNode event = record.path("managedSession"); + String kind = event.path("kind").asText(); + if (List.of("hosted.batch.planned", "hosted.cleanup").contains(kind) + || "model.attempt".equals(kind) && event.path("payload").has("recoveryRef") + || "message.retracted".equals(kind) && event.path("payload").has("throughSequence")) { + return true; + } + } + return false; + } + + public static void validateResource(StoredResource stored, String tenant, + String workspace, String session, Function resources) { + validateResource(stored, tenant, workspace, session, resources, new HashMap<>()); + } + + public static void validateResource(StoredResource stored, String tenant, + String workspace, String session, Function resources, + Map seen) { + if (!KINDS.contains(stored.kind()) && !"managed-action-options".equals(stored.kind())) { + return; + } + JsonNode body = ManagedExtensionRecordStore.parse(new String(stored.bytes(), StandardCharsets.UTF_8)); + require(body != null && stored.schemaVersion() == 1, "Invalid Hosted recovery resource."); + if ("managed-action-options".equals(stored.kind()) && body.path("v").asInt() != 3) { + return; + } + if (KINDS.contains(stored.kind())) { + validateBody(stored.kind(), body, tenant, workspace, session); + } + for (JsonNode ref : references(body)) { + ManagedExtensionRecords.durableRef(ref, "Hosted nested reference"); + closure(ref, tenant, workspace, session, resources, seen); + } + } + + public static void validateEvent(String kind, JsonNode payload, + String tenant, String workspace, String session, + Function resources) { + validateEvent(kind, payload, tenant, workspace, session, resources, new HashMap<>()); + } + + public static void validateEvent(String kind, JsonNode payload, + String tenant, String workspace, String session, + Function resources, Map seen) { + try { + if ("hosted.batch.planned".equals(kind)) { + ManagedExtensionRecords.closed(payload, Set.of("batchId", "planRevision", "planRef"), "batch plan"); + ManagedExtensionRecords.id(payload.get("batchId"), "batchId"); + ManagedExtensionRecords.count(payload.get("planRevision"), 1, + ManagedSessionStoreModels.MAX_SAFE_COUNTER, "planRevision"); + reference(payload.get("planRef"), "hosted-approval-continuation", tenant, workspace, session, resources, seen); + } else if ("hosted.cleanup".equals(kind)) { + ManagedExtensionRecords.closed(payload, Set.of("cleanupId", "descriptorRef", "state"), "cleanup"); + ManagedExtensionRecords.id(payload.get("cleanupId"), "cleanupId"); + ManagedExtensionRecords.oneOf(payload.get("state"), List.of("owed", "confirmed"), "cleanup state"); + reference(payload.get("descriptorRef"), "hosted-turn-cleanup", tenant, workspace, session, resources, seen); + } else if ("model.attempt".equals(kind) && payload.has("recoveryRef")) { + reference(payload.get("recoveryRef"), "hosted-model-request", tenant, workspace, session, resources, seen); + } else if ("message.retracted".equals(kind)) { + int fields = 0; + for (String field : List.of("sourceBootId", "sourceEventEpoch", "throughSequence")) { + fields += payload.has(field) ? 1 : 0; + } + require(fields == 0 || fields == 3, "Retraction source fields must be present together."); + if (fields == 3) { + ManagedExtensionRecords.closed(payload, Set.of("messageId", "turnId", "fromSequence", + "sourceBootId", "sourceEventEpoch", "throughSequence"), "retraction"); + for (String field : List.of("messageId", "turnId", "sourceBootId", "sourceEventEpoch")) { + ManagedExtensionRecords.id(payload.get(field), field); + } + long from = ManagedExtensionRecords.count(payload.get("fromSequence"), 1, + ManagedSessionStoreModels.MAX_SAFE_COUNTER, "fromSequence"); + ManagedExtensionRecords.count(payload.get("throughSequence"), from, + ManagedSessionStoreModels.MAX_SAFE_COUNTER, "throughSequence"); + } + } + } catch (IllegalArgumentException error) { + throw rejected(error.getMessage()); + } + } + + public static void reference(JsonNode ref, String expectedKind, + String tenant, String workspace, String session, + Function resources) { + reference(ref, expectedKind, tenant, workspace, session, resources, new HashMap<>()); + } + + public static void reference(JsonNode ref, String expectedKind, + String tenant, String workspace, String session, + Function resources, Map seen) { + ManagedExtensionRecords.durableRef(ref, "Hosted recovery reference"); + require(ref.path("schemaVersion").asInt() == 1, "Unsupported Hosted recovery reference version."); + require(expectedKind.equals(ref.path("kind").asText()), "Hosted recovery reference kind conflicts."); + closure(ref, tenant, workspace, session, resources, seen); + } + + private static void closure(JsonNode node, String tenant, String workspace, + String session, Function resources, Map seen) { + if (node == null || node.isNull()) { + return; + } + if (node.isObject() && node.has("resourceId") && node.has("kind") + && node.has("schemaVersion") && node.has("byteLength") && node.has("digest")) { + ManagedExtensionRecords.durableRef(node, "Hosted nested reference"); + String id = node.get("resourceId").asText(); + JsonNode previous = seen.putIfAbsent(id, node); + require(previous == null || previous.equals(node), "Conflicting Hosted nested reference."); + if (previous != null) { + return; + } + require(seen.size() <= ManagedSessionStoreModels.MAX_RESOURCES_PER_TRANSACTION, "Hosted reference closure is too large."); + StoredResource stored = resources.apply(id); + require(stored.kind().equals(node.get("kind").asText()) + && stored.schemaVersion() == node.get("schemaVersion").asInt() + && stored.byteLength() == node.get("byteLength").asLong() + && stored.digest().equals(node.get("digest").asText()), "Hosted nested resource conflicts."); + if (KINDS.contains(stored.kind()) || List.of("managed-action-options", "managed-checkpoint", + "managed-hook-plan", "managed-hook-message-chunks", "managed-message-chunks").contains(stored.kind()) + || ManagedExtensionRecords.DOMAINS.stream().anyMatch(domain -> ("managed-" + domain).equals(stored.kind()))) { + JsonNode body = ManagedExtensionRecordStore.parse(new String(stored.bytes(), StandardCharsets.UTF_8)); + require(body != null && stored.schemaVersion() == 1, "Invalid Hosted recovery resource."); + if (KINDS.contains(stored.kind())) { + validateBody(stored.kind(), body, tenant, workspace, session); + } + if (KINDS.contains(stored.kind())) { + for (JsonNode ref : references(body)) { + ManagedExtensionRecords.durableRef(ref, "Hosted nested reference"); + closure(ref, tenant, workspace, session, resources, seen); + } + } else { + closure(body, tenant, workspace, session, resources, seen); + } + } + return; + } + for (JsonNode child : node) { + closure(child, tenant, workspace, session, resources, seen); + } + } + + private static String runtimeSessionId(String promptId) { + return promptId.matches("[A-Za-z0-9._-]{1,512}") && !".".equals(promptId) && !promptId.contains("..") + ? promptId : "wake-" + ToolPublicationContract.sha256(promptId.getBytes(StandardCharsets.UTF_8)); + } + + private static List references(JsonNode body) { + var refs = new java.util.ArrayList(); + for (String field : List.of("definitionRef", "rootSnapshotRef", "assistantRef", "inputRef", "continuationRef")) { + if (body.hasNonNull(field)) { + refs.add(body.get(field)); + } + } + if (body.path("calls").isArray()) { + for (JsonNode call : body.get("calls")) { + refs.add(call.path("inputRef")); + refs.add(call.path("definitionRef")); + } + } + return refs; + } + + private static void validateBody(String kind, JsonNode body, String tenant, String workspace, String session) { + ManagedExtensionRecords.count(body.get("v"), 1, 1, "Hosted recovery version"); + JsonNode key = body.get("sessionKey"); + ManagedExtensionRecords.closed(key, Set.of("tenantId", "workspaceId", "sessionId"), "Hosted recovery Session key"); + for (String field : List.of("tenantId", "workspaceId", "sessionId")) { + ManagedExtensionRecords.id(key.get(field), field); + } + require(tenant.equals(key.get("tenantId").asText()) && workspace.equals(key.get("workspaceId").asText()) + && session.equals(key.get("sessionId").asText()), "Hosted recovery resource names another Session."); + ManagedExtensionRecords.id(body.get("promptId"), "promptId"); + if ("hosted-turn-cleanup".equals(kind)) { + ManagedExtensionRecords.closed(body, Set.of("v", "sessionKey", "promptId", "runtimeSessionId", + "bindingId", "generation", "workspaceGeneration", "fileHistoryTurnId"), "cleanup descriptor"); + ManagedExtensionRecords.id(body.get("bindingId"), "bindingId"); + require(runtimeSessionId(body.get("promptId").asText()).equals(body.path("runtimeSessionId").asText()) + && body.get("promptId").equals(body.get("fileHistoryTurnId")), "Cleanup owner conflicts."); + generation(body.get("generation")); + generation(body.get("workspaceGeneration")); + return; + } + boolean model = "hosted-model-request".equals(kind); + ManagedExtensionRecords.closed(body, model + ? Set.of("v", "sessionKey", "promptId", "definitionRef", "rootSnapshotRef", "sourceActivation", + "sourceBootId", "sourceEventEpoch", "messageId", "round", "throughSequence", + "pendingToolResults", "workspaceContext", "prepared", "requestDigest") + : Set.of("v", "sessionKey", "promptId", "definitionRef", "rootSnapshotRef", "sourceActivation", + "runtime", "batchId", "assistantRef", "model", "round", "stage", "approvalOrdinal", "actionId", "calls"), + "Hosted recovery descriptor"); + JsonNode source = body.get("sourceActivation"); + ManagedExtensionRecords.closed(source, Set.of("activationId", "epoch"), "source activation"); + ManagedExtensionRecords.id(source.get("activationId"), "activationId"); + ManagedExtensionRecords.count(source.get("epoch"), 1, ManagedSessionStoreModels.MAX_SAFE_COUNTER, "source epoch"); + ManagedExtensionRecords.count(body.get("round"), 0, 15, "round"); + for (String field : List.of("definitionRef", "rootSnapshotRef")) { + ManagedExtensionRecords.durableRef(body.get(field), field); + } + if (model) { + for (String field : List.of("sourceBootId", "sourceEventEpoch", "messageId")) { + ManagedExtensionRecords.id(body.get(field), field); + } + ManagedExtensionRecords.count(body.get("throughSequence"), 1, ManagedSessionStoreModels.MAX_SAFE_COUNTER, "throughSequence"); + require(body.get("pendingToolResults").isBoolean() + && (body.get("workspaceContext").isNull() || body.get("workspaceContext").isTextual()), "Invalid model context."); + require(body.get("requestDigest").isTextual() && body.get("requestDigest").asText().matches("[a-f0-9]{64}"), "Invalid model digest."); + JsonNode prepared = body.get("prepared"); + Set preparedKeys = new HashSet<>(Set.of("request", "history", "completedToolCallIds", + "routeSelector", "providerPin", "promptTokensForClamp")); + if (prepared.has("allowPersistentRetry")) { + preparedKeys.add("allowPersistentRetry"); + require(prepared.get("allowPersistentRetry").isBoolean(), "Invalid model retry policy."); + } + ManagedExtensionRecords.closed(prepared, preparedKeys, "prepared model request"); + ManagedExtensionRecords.closed(prepared.get("request"), Set.of("model", "contents", "config"), "model request"); + ManagedExtensionRecords.id(prepared.get("routeSelector"), "routeSelector"); + ManagedExtensionRecords.id(prepared.path("request").get("model"), "model"); + require(prepared.path("history").isArray() && prepared.path("completedToolCallIds").isArray() + && prepared.path("request").path("contents").isArray() && prepared.path("request").path("config").isObject() + && prepared.path("providerPin").isTextual() && prepared.path("providerPin").asText().matches("[a-f0-9]{64}"), "Invalid prepared request."); + ManagedExtensionRecords.count(prepared.get("promptTokensForClamp"), 0, ManagedSessionStoreModels.MAX_SAFE_COUNTER, "promptTokensForClamp"); + for (String field : List.of("abortSignal", "httpOptions", "apiKey", "headers", "authorization")) { + require(!prepared.path("request").path("config").has(field), "Prepared request contains authentication or transport state."); + } + } else { + ManagedExtensionRecords.durableRef(body.get("assistantRef"), "assistantRef"); + for (String field : List.of("batchId", "model")) { + ManagedExtensionRecords.id(body.get(field), field); + } + ManagedExtensionRecords.oneOf(body.get("stage"), List.of("approval", "final"), "stage"); + JsonNode runtime = body.get("runtime"); + ManagedExtensionRecords.closed(runtime, Set.of("runtimeSessionId", "bindingId", "generation", "workspaceGeneration"), "native Runtime"); + require(runtimeSessionId(body.get("promptId").asText()).equals(runtime.path("runtimeSessionId").asText()), "Native Runtime owner conflicts."); + ManagedExtensionRecords.id(runtime.get("bindingId"), "bindingId"); + generation(runtime.get("generation")); + generation(runtime.get("workspaceGeneration")); + require(body.path("calls").isArray() && !body.path("calls").isEmpty(), "Native batch is empty."); + ManagedExtensionRecords.count(body.get("approvalOrdinal"), 0, body.path("calls").size() - 1, "approvalOrdinal"); + for (JsonNode call : body.get("calls")) { + ManagedExtensionRecords.closed(call, Set.of("call", "runtimeCallId", "inputRef", "definitionRef", + "requestDigest", "argsDigest", "prepareKey", "prepareReference", "partIndex", "refusal", "actionId"), "native batch member"); + ManagedExtensionRecords.durableRef(call.get("inputRef"), "inputRef"); + ManagedExtensionRecords.durableRef(call.get("definitionRef"), "definitionRef"); + } + } + } + + private static void generation(JsonNode node) { + require(node != null && node.isTextual() && node.asText().matches("[1-9][0-9]{0,18}") + && new java.math.BigInteger(node.asText()).compareTo(java.math.BigInteger.valueOf(Long.MAX_VALUE)) <= 0, + "Invalid Runtime generation."); + } + + private static void require(boolean value, String message) { + if (!value) { + throw rejected(message); + } + } + + private static ApiException rejected(String message) { + return new ApiException(HttpStatus.CONFLICT, "managed_hosted_recovery_record_rejected", message); + } +} diff --git a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedSessionReaderVersion.java b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedSessionReaderVersion.java new file mode 100644 index 0000000000..7514544783 --- /dev/null +++ b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedSessionReaderVersion.java @@ -0,0 +1,36 @@ +package com.alibaba.qwen.code.managedagent.store; + +import com.alibaba.qwen.code.managedagent.api.ApiException; +import com.alibaba.qwen.code.managedagent.api.PublicSurface; +import org.springframework.http.HttpStatus; +import org.springframework.web.context.request.RequestContextHolder; +import org.springframework.web.context.request.ServletRequestAttributes; + +/** The ceiling of a remote Store reader; in-process readers support both formats. */ +final class ManagedSessionReaderVersion { + static final String HEADER = "X-Qwen-Managed-Max-Readable-Storage-Version"; + static final int SUPPORTED = 2; + + private ManagedSessionReaderVersion() {} + + static void requireReadable(int floor) { + int ceiling = SUPPORTED; + if (RequestContextHolder.getRequestAttributes() instanceof ServletRequestAttributes attributes) { + var request = attributes.getRequest(); + String path = PublicSurface.pathWithinApplication(request); + if (path.startsWith("/internal/managed-session-store/") + || path.startsWith("/internal/managed-tool-publications/")) { + String header = request.getHeader(HEADER); + if (header != null && !header.matches("[12]")) { + throw new ApiException(HttpStatus.BAD_REQUEST, "invalid_managed_session_reader_version", + "The Managed Session reader version is invalid."); + } + ceiling = header == null ? 1 : Integer.parseInt(header); + } + } + if (floor > ceiling) { + throw new ApiException(HttpStatus.CONFLICT, "managed_session_reader_version_unsupported", + "This reader cannot read the Managed Session storage version."); + } + } +} diff --git a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedSessionStore.java b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedSessionStore.java index b0dc30ccc7..7ae21655a9 100644 --- a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedSessionStore.java +++ b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedSessionStore.java @@ -33,6 +33,7 @@ import java.sql.Timestamp; import java.time.temporal.ChronoUnit; import java.util.ArrayList; import java.util.Base64; +import java.util.HashMap; import java.util.HashSet; import java.util.HexFormat; import java.util.List; @@ -299,6 +300,7 @@ public class ManagedSessionStore { throw conflict("managed_session_not_writable", "The Session is closing or closed."); } String tokenHash = tokenHash(writerToken); + ManagedSessionReaderVersion.requireReadable(1); Timestamp createdAt = databaseNow(); Timestamp initialLeaseUntil = plusMillis(createdAt, request.leaseMillis()); @@ -503,6 +505,17 @@ public class ManagedSessionStore { ValidatedCommit validated = validateCommit(request); HeadRow head = requireHeadForUpdate(tenantId, sessionId); requireHeadScope(head, tenantId, request.workspaceId(), sessionId); + boolean promotes = ManagedHostedRecoveryRecords.requiresVersionTwo(validated.recordBytes(), request.resources()) + || validated.resources().stream().anyMatch(resource -> { + if (!"managed-action-options".equals(resource.resource().kind()) || resource.bytes() == null) { + return false; + } + JsonNode body = ManagedExtensionRecordStore.parse(new String(resource.bytes(), StandardCharsets.UTF_8)); + return body != null && body.path("v").asInt() == 3; + }); + if (promotes) { + ManagedSessionReaderVersion.requireReadable(2); + } TransactionRow existing = findTransactionByCommand(tenantId, sessionId, commandKeyHash(request.operation(), request.commandId())); @@ -520,8 +533,36 @@ public class ManagedSessionStore { String scopeKey = sessionScopeKey(tenantId, sessionId); commitResources(scopeKey, tenantId, sessionId, request, revision, now, validated.resources()); + Set declaredResources = new HashSet<>(); + for (var resource : request.resources() == null ? List.of() : request.resources()) { + declaredResources.add(resource.resourceId()); + } + var resourceCache = new HashMap(); + var hostedReferences = new HashMap(); + java.util.function.Function reader = resourceId -> resourceCache.computeIfAbsent(resourceId, + id -> storedResource(scopeKey, tenantId, request.workspaceId(), sessionId, id)); + java.util.function.Function closureReader = resourceId -> { + if (!declaredResources.contains(resourceId)) { + throw invalid("Hosted nested reference is absent from the transaction census."); + } + return reader.apply(resourceId); + }; + for (var resource : request.resources() == null ? List.of() : request.resources()) { + if (ManagedHostedRecoveryRecords.KINDS.contains(resource.kind()) || "managed-action-options".equals(resource.kind())) { + ManagedHostedRecoveryRecords.validateResource(closureReader.apply(resource.resourceId()), tenantId, + request.workspaceId(), sessionId, closureReader, hostedReferences); + } + } + for (String line : new String(validated.recordBytes(), StandardCharsets.UTF_8).split("\n")) { + JsonNode record = ManagedExtensionRecordStore.parse(line); + if (record != null && record.has("managedSession")) { + JsonNode event = record.get("managedSession"); + ManagedHostedRecoveryRecords.validateEvent(event.path("kind").asText(), event.path("payload"), tenantId, + request.workspaceId(), sessionId, closureReader, hostedReferences); + } + } if (authority != null && extensionRecords.hasNewLifecycleDispatch(tenantId, sessionId, validated.recordBytes(), - resourceId -> storedResource(scopeKey, tenantId, request.workspaceId(), sessionId, resourceId))) { + reader)) { if (lifecycleExecution == null || lifecycleSessions == null) { throw WorkspaceLifecycleStore.blocked("workspace_lifecycle_protocol_unavailable"); } @@ -539,13 +580,11 @@ public class ManagedSessionStore { } if (authority == null && lifecycleProtocolFenced(tenantId, sessionId)) { extensionRecords.requireLifecycleSettlement(tenantId, sessionId, validated.recordBytes(), - resourceId -> storedResource(scopeKey, tenantId, request.workspaceId(), sessionId, resourceId)); + reader); } var applied = extensionRecords.apply(tenantId, request.workspaceId(), sessionId, request.firstSequence(), request.eventCount(), - validated.recordBytes(), resourceId -> storedResource( - scopeKey, tenantId, request.workspaceId(), sessionId, - resourceId)); + validated.recordBytes(), reader); var receiptEvents = applied.receipts(); if (actions != null) { actions.apply( @@ -553,13 +592,7 @@ public class ManagedSessionStore { request.workspaceId(), sessionId, request.firstSequence(), request.eventCount(), validated.recordBytes(), - resourceId -> - storedResource( - scopeKey, - tenantId, - request.workspaceId(), - sessionId, - resourceId)); + reader, hostedReferences); } jdbc.update("INSERT INTO qwen_managed_session_journal_tx" + " (tenant_id, workspace_id, session_id," @@ -619,7 +652,7 @@ public class ManagedSessionStore { + " activation_id = ?, activation_phase = ?," + " activation_event_epoch = ?," + " activation_expires_at = ?," - + " activation_head_revision = ?, updated_at = ?" + + " activation_head_revision = ?, storage_version = ?, updated_at = ?" + " WHERE tenant_id = ?" + " AND session_id = ?", revision, request.lastSequence(), request.commitDigest(), @@ -633,7 +666,7 @@ public class ManagedSessionStore { : fits ? activation.epoch() : null, activation == null ? head.activationExpiresAt() : fits ? activation.expiresAt() : null, - activationHeadRevision, now, tenantId, sessionId); + activationHeadRevision, Math.max(head.storageVersion(), promotes ? 2 : 1), now, tenantId, sessionId); if (toolResults != null) { toolResults.captureEvents(tenantId, request.workspaceId(), sessionId, revision, receiptEvents); } @@ -1557,10 +1590,11 @@ public class ManagedSessionStore { || !sessionId.equals(head.sessionId())) { throw sessionNotFound(); } - if (head.storageVersion() != STORAGE_VERSION) { + if (head.storageVersion() < 1 || head.storageVersion() > ManagedSessionReaderVersion.SUPPORTED) { throw conflict("managed_session_storage_version_unsupported", "The Managed Session storage version is unsupported."); } + ManagedSessionReaderVersion.requireReadable(head.storageVersion()); if (!HEAD_STATES.contains(head.state()) || !RECOVERY_STATES.contains(head.recoveryStatus()) || head.writerGeneration() < 1 diff --git a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedSessionStoreModels.java b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedSessionStoreModels.java index b081479559..613b6bc925 100644 --- a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedSessionStoreModels.java +++ b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ManagedSessionStoreModels.java @@ -98,7 +98,12 @@ public final class ManagedSessionStoreModels { public record WriterGrant(long writerGeneration, long leaseUntil, long journalRevision, long committedSequence, String lastCommitDigest, long activationEpoch, - boolean replayed) { + boolean replayed, int supportedStorageVersion) { + public WriterGrant(long writerGeneration, long leaseUntil, long journalRevision, + long committedSequence, String lastCommitDigest, long activationEpoch, boolean replayed) { + this(writerGeneration, leaseUntil, journalRevision, committedSequence, lastCommitDigest, + activationEpoch, replayed, ManagedSessionReaderVersion.SUPPORTED); + } } public record SealReceipt(long writerGeneration, String state, diff --git a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ToolPublicationAdmissionStore.java b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ToolPublicationAdmissionStore.java index fdaa540bb5..22cccdee6b 100644 --- a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ToolPublicationAdmissionStore.java +++ b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ToolPublicationAdmissionStore.java @@ -135,8 +135,11 @@ public final class ToolPublicationAdmissionStore { "Original referenced publication conflicts"); long revision = positiveCounter(publication.get("receipt_revision")); long sequence = positiveCounter(publication.get("receipt_sequence")); - var head = jdbc.queryForMap("SELECT journal_revision, committed_sequence FROM qwen_managed_session_journal_head" + var head = jdbc.queryForMap("SELECT journal_revision, committed_sequence, storage_version FROM qwen_managed_session_journal_head" + " WHERE tenant_id = ? AND session_id = ?" + suffix, text(key, "tenantId"), text(key, "sessionId")); + require(((Number) head.get("storage_version")).intValue() >= 1 + && ((Number) head.get("storage_version")).intValue() <= ManagedSessionReaderVersion.SUPPORTED, + "Managed Session storage version is unsupported"); require(positiveCounter(head.get("journal_revision")) >= revision && positiveCounter(head.get("committed_sequence")) >= sequence, "Original receipt exceeds the committed Session head"); diff --git a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ToolPublicationStore.java b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ToolPublicationStore.java index d100ea7b06..8f4dffa0dd 100644 --- a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ToolPublicationStore.java +++ b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/ToolPublicationStore.java @@ -236,13 +236,16 @@ public final class ToolPublicationStore { require(row.tenant().equals(text(key, "tenantId")) && row.workspace().equals(text(key, "workspaceId")) && row.session().equals(text(key, "sessionId")), "Publication scope conflicts"); - var head = jdbc.queryForMap("SELECT workspace_id, state, writer_id, writer_generation," + var head = jdbc.queryForMap("SELECT workspace_id, state, storage_version, writer_id, writer_generation," + " CASE WHEN writer_lease_until > CURRENT_TIMESTAMP(6) THEN 1 ELSE 0 END AS writer_live," + " recovery_status, activation_epoch, journal_revision," + " activation_id, activation_phase, activation_event_epoch, activation_expires_at," + " activation_head_revision" + " FROM qwen_managed_session_journal_head WHERE tenant_id = ? AND session_id = ? FOR UPDATE", row.tenant(), row.session()); + require(((Number) head.get("storage_version")).intValue() >= 1 + && ((Number) head.get("storage_version")).intValue() <= ManagedSessionReaderVersion.SUPPORTED, + "Managed Session storage version is unsupported."); var current = jdbc.queryForMap("SELECT token_hash, state, expires_at, binding_digest, binding_json," + " tenant_id, workspace_id, session_id, CASE WHEN quarantined THEN 1 ELSE 0 END AS quarantined" + " FROM qwen_tool_publication WHERE scope_key = ? AND publication_id = ? FOR UPDATE", diff --git a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/WorkspaceCsiCheckpointSnapshotStore.java b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/WorkspaceCsiCheckpointSnapshotStore.java index c316a0f149..bf4a7ab685 100644 --- a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/WorkspaceCsiCheckpointSnapshotStore.java +++ b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/WorkspaceCsiCheckpointSnapshotStore.java @@ -128,7 +128,7 @@ public final class WorkspaceCsiCheckpointSnapshotStore { "SELECT * FROM qwen_managed_session_journal_head WHERE tenant_id = ? AND session_id = ?", key.path("tenantId").asText(), key.path("sessionId").asText()); scope(head, key); - require(number(head, "storage_version") == 1 && "READY".equals(head.get("recovery_status")) + require((number(head, "storage_version") == 1 || number(head, "storage_version") == 2) && "READY".equals(head.get("recovery_status")) && Set.of("ACTIVE", "SEALED").contains(text(head, "state")) && number(head, "compacted_through_revision") == 0 && number(head, "activation_epoch") == binding.path("activationEpoch").longValue()); diff --git a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/WorkspaceLifecycleStore.java b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/WorkspaceLifecycleStore.java index 6473979f3b..9fb8633888 100644 --- a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/WorkspaceLifecycleStore.java +++ b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/WorkspaceLifecycleStore.java @@ -85,6 +85,10 @@ public class WorkspaceLifecycleStore { } public static void requireIdleJournal(JdbcTemplate jdbc, ObjectMapper json, String tenant, String session) { + requireIdleJournal(jdbc, json, tenant, session, false); + } + + public static void requireIdleJournal(JdbcTemplate jdbc, ObjectMapper json, String tenant, String session, boolean allowCleanup) { var heads = jdbc.queryForList("SELECT compacted_through_revision FROM qwen_managed_session_journal_head" + " WHERE tenant_id = ? AND session_id = ? FOR UPDATE", tenant, session); if (heads.isEmpty()) { @@ -94,12 +98,16 @@ public class WorkspaceLifecycleStore { throw blocked("workspace_lifecycle_journal_unverified"); } var pending = new java.util.HashSet(); + var cleanup = new java.util.HashMap(); jdbc.query("SELECT record_bytes FROM qwen_managed_session_journal_tx WHERE tenant_id = ? AND session_id = ?" + " ORDER BY journal_revision FOR UPDATE", row -> { for (String line : new String(row.getBytes(1), StandardCharsets.UTF_8).split("\n")) { try { JsonNode event = json.readTree(line).path("managedSession"); String turn = event.path("payload").path("turnId").asText(); + if ("hosted.cleanup".equals(event.path("kind").asText())) { + cleanup.put(event.path("payload").path("cleanupId").asText(), event.path("payload").path("state").asText()); + } if ("input.accepted".equals(event.path("kind").asText())) { pending.add(turn); } else if ("turn.settled".equals(event.path("kind").asText())) { @@ -110,7 +118,7 @@ public class WorkspaceLifecycleStore { } } }, tenant, session); - if (!pending.isEmpty()) { + if (!pending.isEmpty() || !allowCleanup && cleanup.values().stream().anyMatch(state -> !"confirmed".equals(state))) { throw new ApiException(HttpStatus.CONFLICT, "turn_active", "The Session has an active Turn."); } } @@ -125,6 +133,11 @@ public class WorkspaceLifecycleStore { if (saved != null) { return parse(saved); } + try { + requireIdleJournal(jdbc, json, operation.tenantId(), operation.sessionId()); + } catch (ApiException blocked) { + return null; + } JsonNode header = header(operation.tenantId(), operation.sessionId()); var receipt = envelope(operation); var effects = receipt.putArray("effects"); @@ -207,6 +220,7 @@ public class WorkspaceLifecycleStore { } private void verifyEffects(OperationRecord operation, JsonNode receipt) { + requireIdleJournal(jdbc, json, operation.tenantId(), operation.sessionId()); if (receipt.path("protocolVersion").asInt() != 1 || !receipt.path("sessionKey").equals(sessionKey(operation)) || !operation.operationId().equals(receipt.path("operationId").asText()) || !operation.kind().name().toLowerCase(java.util.Locale.ROOT).equals(receipt.path("kind").asText()) diff --git a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/WorkspaceRecoveryStore.java b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/WorkspaceRecoveryStore.java index b0ea30cd19..792847eff9 100644 --- a/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/WorkspaceRecoveryStore.java +++ b/packages/sdk-java/managed-agent-server/src/main/java/com/alibaba/qwen/code/managedagent/store/WorkspaceRecoveryStore.java @@ -567,7 +567,7 @@ public final class WorkspaceRecoveryStore { Timestamp lease = (Timestamp) head.get("writer_lease_until"); check(!"ACTIVE".equals(head.get("state")) || lease != null && !lease.after(now(jdbc)), "source_drift"); check(Set.of("ACTIVE", "SEALED", "DELETING", "DELETED").contains(head.get("state")) - && ((Number) head.get("storage_version")).intValue() == 1 + && List.of(1, 2).contains(((Number) head.get("storage_version")).intValue()) && ((Number) head.get("compacted_through_revision")).longValue() == 0 && "READY".equals(head.get("recovery_status")), "source_drift"); ObjectNode value = source.putObject("head"); diff --git a/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/HostedCancellationProbe.java b/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/HostedCancellationProbe.java index a35390a5fc..e0d97c7e45 100644 --- a/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/HostedCancellationProbe.java +++ b/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/HostedCancellationProbe.java @@ -123,7 +123,7 @@ final class HostedCancellationProbe implements AutoCloseable { assertThat(count(executions, id)).isEqualTo(1); assertThat(count(completions, id)).as("Cancellation ACK is not physical settlement").isZero(); assertThat(count(cancellations, id)).isEqualTo(blocked ? 2 : 1); - assertJournal(session, false, false, "await_runtime"); + assertJournal(session, false, false, "await_runtime", false); } else { assertThat(phase).isIn("release", "finished"); assertThat(execution.get("execution_state")).isEqualTo("SETTLED"); @@ -133,8 +133,8 @@ final class HostedCancellationProbe implements AutoCloseable { assertThat(count(cancellations, id)).isEqualTo(prepared ? 0 : blocked ? 2 : 1); assertThat(Files.readString(Path.of(session.get("directory").toString()).resolve("proof.txt"))) .isEqualTo(prepared ? "x" : "xx"); - assertJournal(session, !blocked, phase.equals("finished") && !blocked, - blocked ? "await_runtime" : phase.equals("release") ? "results_ready" : "before_model"); + assertJournal(session, !blocked, !blocked, + blocked ? "await_runtime" : "before_model", phase.equals("finished") && !blocked); } assertThat(((Number) execution.get("dispatch_generation")).longValue()) .isEqualTo(phase.equals("prepared") || prepared ? 0 : 1); @@ -168,7 +168,8 @@ final class HostedCancellationProbe implements AutoCloseable { System.out.println("FG6D_LEDGER " + session.get("fault") + " " + JSON.writeValueAsString(check(session, "finished"))); } - private void assertJournal(Map session, boolean result, boolean terminal, String phase) throws Exception { + private void assertJournal(Map session, boolean result, boolean terminal, String phase, + boolean cleanupConfirmed) throws Exception { String id = session.get("sessionId").toString(); var rows = jdbc.queryForList("SELECT * FROM qwen_managed_session_journal_tx" + " WHERE tenant_id = ? AND session_id = ? ORDER BY journal_revision", tenant, id); @@ -203,6 +204,14 @@ final class HostedCancellationProbe implements AutoCloseable { assertThat(terminals.getFirst().path("payload").path("outcome").asText()).isEqualTo("cancelled"); assertThat(terminals.getFirst().path("payload").path("turnId").asText()).isEqualTo(execution.get("turn_id")); } + var cleanup = events.stream().filter(event -> event.path("kind").asText().equals("hosted.cleanup")).toList(); + assertThat(cleanup).hasSize(cleanupConfirmed ? 2 : 1); + assertThat(cleanup.getFirst().path("payload").path("state").asText()).isEqualTo("owed"); + if (cleanupConfirmed) { + assertThat(cleanup.getLast().path("payload").path("state").asText()).isEqualTo("confirmed"); + assertThat(cleanup.getLast().path("sequence").asLong()) + .isGreaterThan(terminals.getFirst().path("sequence").asLong()); + } assertThat(events.stream().filter(event -> event.path("kind").asText().equals("message.committed") && event.path("payload").path("role").asText().equals("tool_result"))).hasSize(result ? 1 : 0); var outcomes = jdbc.queryForList("SELECT inline_bytes FROM qwen_managed_session_resource WHERE tenant_id = ?" diff --git a/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/HostedWorkspaceToolTurnIT.java b/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/HostedWorkspaceToolTurnIT.java index 112d5e3ed2..5999268612 100644 --- a/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/HostedWorkspaceToolTurnIT.java +++ b/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/HostedWorkspaceToolTurnIT.java @@ -450,7 +450,7 @@ class HostedWorkspaceToolTurnIT { List> executions = jdbc.queryForList("SELECT execution_call_id, idempotency_key," + " runtime_session_id, dispatch_generation, execution_state, execution_status" + " FROM qwen_tool_execution WHERE harness_session_id = ?", session.get("sessionId")); - assertThat(executions).as(fault + " reservations").hasSize(fault.equals("acquire") ? 0 : 1); + assertThat(executions).as(fault + " reservations").hasSize(List.of("acquire", "arguments").contains(fault) ? 0 : 1); for (Map execution : executions) { assertThat(execution.get("execution_call_id")).isEqualTo(report.path("executionCallId").asText()); assertThat(execution.get("idempotency_key")).isEqualTo(report.path("idempotencyKey").asText()); diff --git a/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/ManagedAgentServerIntegrationTest.java b/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/ManagedAgentServerIntegrationTest.java index c969622823..fd4286f46c 100644 --- a/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/ManagedAgentServerIntegrationTest.java +++ b/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/ManagedAgentServerIntegrationTest.java @@ -1683,6 +1683,102 @@ class ManagedAgentServerIntegrationTest { "event epoch changed")); } + + + @Test + void boundedRetractionRemovesFullyConsumedOriginalNamespace() { + assertBoundedRetractionAcrossAttachment(4, "consumed"); + } + + @Test + void boundedRetractionRemovesUnconsumedReplayNamespace() { + assertBoundedRetractionAcrossAttachment(2, "unconsumed"); + } + + @Test + void boundedRetractionRemovesSplitOriginalAndReplayNamespaces() { + assertBoundedRetractionAcrossAttachment(3, "split"); + } + + private void assertBoundedRetractionAcrossAttachment(int consumedThrough, String caseName) { + pauseRecoveryScanning(); + String tenant = "tenant-bounded-" + caseName + "-" + UUID.randomUUID(); + Admission session = store.insertSessionCommand(tenant, "CREATE_SESSION", "bounded-create", + "sha256:" + "4".repeat(64), "qwen-code", null, null, List.of(), null); + Admission turn = store.insertTurnCommand(tenant, "SUBMIT_TURN", "bounded-turn", + "sha256:" + "5".repeat(64), session.sessionId(), List.of(), "sha256:" + "6".repeat(64)); + String owner = "bounded-owner"; + assertThat(store.claimTurn(tenant, session.sessionId(), turn.turnId(), owner, Duration.ofMinutes(1))).isPresent(); + assertThat(store.bindHarness(tenant, session.sessionId(), turn.turnId(), owner, "boot_old")).isTrue(); + store.markSubmissionAttempted(tenant, session.sessionId(), turn.turnId(), owner); + store.recordAdmission(tenant, session.sessionId(), turn.turnId(), owner, "epoch_old", 1); + java.util.ArrayList old = new java.util.ArrayList<>(); + old.add(new HarnessEvent(2, "boot_old:epoch_old:2", new ProjectedEvent("item.output_text.delta", + Map.of("text", "earlier-round"), false, null, null, null))); + for (int sequence = 3; sequence <= consumedThrough; sequence++) { + old.add(new HarnessEvent(sequence, "boot_old:epoch_old:" + sequence, + new ProjectedEvent(sequence == 3 ? "item.output_text.delta" : "item.reasoning.delta", + Map.of("text", "orphaned-" + sequence), false, null, null, null))); + } + store.recordHarnessEvents(tenant, session.sessionId(), turn.turnId(), owner, "epoch_old", old); + assertThat(store.bindRecoveredHarness(tenant, session.sessionId(), turn.turnId(), owner, "boot_old", "boot_new")).isTrue(); + store.recordRecoveryAdmission(tenant, session.sessionId(), turn.turnId(), owner, + "epoch_old", "epoch_old", "epoch_new", consumedThrough); + java.util.ArrayList replay = new java.util.ArrayList<>(); + for (int sequence = consumedThrough + 1; sequence <= 4; sequence++) { + replay.add(new HarnessEvent(sequence, "boot_new:epoch_new:" + sequence, + new ProjectedEvent(sequence == 3 ? "item.output_text.delta" : "item.reasoning.delta", + Map.of("text", "orphaned-" + sequence), false, null, null, null))); + } + store.recordHarnessEvents(tenant, session.sessionId(), turn.turnId(), owner, "epoch_new", replay); + store.materializeNextBatch(tenant, session.sessionId(), 100); + long publicSequence = store.requireSession(tenant, session.sessionId()).lastSequence(); + for (long[] invalid : List.of(new long[] {3, 5, 5}, new long[] {5, 4, 5}, new long[] {0, 4, 5})) { + assertThatThrownBy(() -> store.retractHarnessTurnOutput(tenant, session.sessionId(), turn.turnId(), owner, + "epoch_new", invalid[0], invalid[2], "boot_old", "epoch_old", invalid[1])) + .isInstanceOf(IllegalArgumentException.class); + assertThat(store.requireSession(tenant, session.sessionId()).lastSequence()).isEqualTo(publicSequence); + assertThat(store.findTurn(tenant, session.sessionId(), turn.turnId())).get() + .satisfies(record -> assertThat(record.harnessLastEventId()).isEqualTo(4)); + } + store.retractHarnessTurnOutput(tenant, session.sessionId(), turn.turnId(), owner, + "epoch_new", 3, 5, "boot_old", "epoch_old", 4L); + assertThat(store.findEvents(tenant, session.sessionId(), 0, 100)) + .filteredOn(event -> "boot_old:epoch_old:2".equals(event.sourceKey())).singleElement() + .satisfies(event -> assertThat(event.data()).containsEntry("text", "earlier-round")); + assertThat(store.findEvents(tenant, session.sessionId(), 0, 100)) + .filteredOn(event -> ("item.output_text.delta".equals(event.type()) || "item.reasoning.delta".equals(event.type())) && event.sourceKey() != null && + (event.sourceKey().endsWith(":3") || event.sourceKey().endsWith(":4"))) + .hasSize(2).allSatisfy(event -> { + assertThat(event.data()).containsEntry("text", ""); + assertThat(event.itemId()).isNull(); + assertThat(event.contentPartId()).isNull(); + }); + store.recordHarnessEvents(tenant, session.sessionId(), turn.turnId(), owner, "epoch_new", + List.of(new HarnessEvent(6, "boot_new:epoch_new:6", new ProjectedEvent("item.output_text.delta", + Map.of("text", "replacement"), false, null, null, null)))); + store.retractHarnessTurnOutput(tenant, session.sessionId(), turn.turnId(), owner, + "epoch_new", 3, 7, "boot_old", "epoch_old", 4L); + assertThat(store.findEvents(tenant, session.sessionId(), 0, 100)) + .filteredOn(event -> "stream.reconciled".equals(event.type())).hasSize(1); + assertThat(store.findEvents(tenant, session.sessionId(), 0, 100)) + .filteredOn(event -> "boot_new:epoch_new:6".equals(event.sourceKey())).singleElement() + .satisfies(event -> assertThat(event.data()).containsEntry("text", "replacement")); + assertThat(store.findTurn(tenant, session.sessionId(), turn.turnId())).get() + .satisfies(record -> assertThat(record.harnessLastEventId()).isEqualTo(7)); + store.materializeNextBatch(tenant, session.sessionId(), 100); + assertThat(store.findSnapshot(tenant, session.sessionId())).isPresent(); + store.recordHarnessEvents(tenant, session.sessionId(), turn.turnId(), owner, "epoch_new", + List.of(new HarnessEvent(8, "boot_new:epoch_new:8", new ProjectedEvent("turn.completed", + Map.of(), true, "COMPLETED", null, null)))); + assertThat(store.findEvents(tenant, session.sessionId(), 0, 100)) + .filteredOn(event -> "turn.completed".equals(event.type()) && turn.turnId().equals(event.turnId())).hasSize(1); + store.materializeNextBatch(tenant, session.sessionId(), 100); + assertThat(store.findSnapshot(tenant, session.sessionId())).get().satisfies(snapshot -> + assertThat(snapshot.coveredSequence()).isEqualTo(store.requireSession(tenant, session.sessionId()).lastSequence())); + assertEventsNameTheSnapshot(tenant, session.sessionId()); + } + @Test void doesNotPublishRolledBackEvents() throws Exception { String tenant = "tenant-rollback-" + UUID.randomUUID(); @@ -2280,4 +2376,61 @@ class ManagedAgentServerIntegrationTest { } } + @Test + void preAcquireCleanupUsesCurrentLifecycleAuthorityAndDoesNotAcquire() { + String tenant = "tenant-cleanup-authority-" + UUID.randomUUID(); + String harnessSession = "harness-" + UUID.randomUUID(); + String operation = "operation-" + UUID.randomUUID(); + var scope = new com.alibaba.qwen.code.runtimebroker.RuntimeScope(tenant, "workspace", "1", "/workspace", "capability", "session"); + var bindings = new com.alibaba.qwen.code.runtimebroker.JdbcRuntimeBindingRepository(jdbc.getDataSource(), + new com.alibaba.qwen.code.runtimebroker.AesGcmSecretProtector("test", new byte[32])); + var sessions = new com.alibaba.qwen.code.runtimebroker.JdbcRuntimeSessionRepository(jdbc.getDataSource()); + var executions = new com.alibaba.qwen.code.runtimebroker.JdbcToolExecutionRepository(jdbc.getDataSource()); + var provisioner = org.mockito.Mockito.mock(com.alibaba.qwen.code.runtimebroker.RuntimeProvisioner.class, + org.mockito.Mockito.CALLS_REAL_METHODS); + var transport = org.mockito.Mockito.mock(com.alibaba.qwen.code.runtimebroker.RuntimeTransport.class); + var lease = new com.alibaba.qwen.code.runtimebroker.RuntimeLease("runtime", java.net.URI.create("http://127.0.0.1:4000"), "token", "lease", 1); + when(provisioner.provision(org.mockito.ArgumentMatchers.any(com.alibaba.qwen.code.runtimebroker.RuntimeProvisionRequest.class))) + .thenReturn(java.util.concurrent.CompletableFuture.completedFuture(lease)); + java.util.function.Consumer admissionClosed = error -> { + Throwable cause = error; + while (cause.getCause() != null) cause = cause.getCause(); + assertThat(cause).isInstanceOfSatisfying(RuntimeBrokerException.class, + conflict -> assertThat(conflict.getCode()).isEqualTo("runtime_admission_closed")); + }; + try (var broker = new com.alibaba.qwen.code.runtimebroker.RuntimeBrokerService( + ignored -> java.util.concurrent.CompletableFuture.completedFuture(scope), provisioner, transport, + bindings, sessions, executions, "cleanup-probe", Duration.ofMinutes(1), Duration.ofMinutes(1))) { + var binding = broker.warm(harnessSession).toCompletableFuture().join(); + jdbc.execute((org.springframework.jdbc.core.ConnectionCallback) connection -> { + com.alibaba.qwen.code.runtimebroker.JdbcRuntimeBindingRepository.beginHarnessLifecycle(connection, tenant, harnessSession, operation); + return null; + }); + jdbc.update("UPDATE qwen_runtime_harness_drain SET claim_generation = 7, claim_lease_until = ? WHERE tenant_id = ? AND harness_session_id = ?", + System.currentTimeMillis() + 60_000, tenant, harnessSession); + var authority = new com.alibaba.qwen.code.runtimebroker.RuntimeLifecycleAuthority(operation, 7); + for (var invalid : java.util.Arrays.asList(null, + new com.alibaba.qwen.code.runtimebroker.RuntimeLifecycleAuthority(operation, 8), + new com.alibaba.qwen.code.runtimebroker.RuntimeLifecycleAuthority("wrong-operation", 7))) { + assertThatThrownBy(() -> broker.releaseOriginal(harnessSession, "runtime-a", binding.getBindingId(), + binding.getGeneration(), invalid).toCompletableFuture().join()).satisfies(admissionClosed); + assertThat(sessions.findById(scope, "runtime-a")).isNull(); + } + jdbc.update("UPDATE qwen_runtime_harness_drain SET claim_lease_until = 0 WHERE tenant_id = ? AND harness_session_id = ?", tenant, harnessSession); + assertThatThrownBy(() -> broker.releaseOriginal(harnessSession, "runtime-a", binding.getBindingId(), + binding.getGeneration(), authority).toCompletableFuture().join()).satisfies(admissionClosed); + assertThat(sessions.findById(scope, "runtime-a")).isNull(); + jdbc.update("UPDATE qwen_runtime_harness_drain SET claim_lease_until = ? WHERE tenant_id = ? AND harness_session_id = ?", + System.currentTimeMillis() + 60_000, tenant, harnessSession); + assertThat(broker.releaseOriginal(harnessSession, "runtime-a", binding.getBindingId(), binding.getGeneration(), authority) + .toCompletableFuture().join()).isTrue(); + assertThat(sessions.findById(scope, "runtime-a").getState()).isEqualTo(com.alibaba.qwen.code.runtimebroker.RuntimeSessionRecord.State.RELEASED); + assertThat(broker.releaseOriginal(harnessSession, "runtime-a", binding.getBindingId(), binding.getGeneration(), authority) + .toCompletableFuture().join()).isTrue(); + org.mockito.Mockito.verify(transport, org.mockito.Mockito.never()).acquire(org.mockito.ArgumentMatchers.any(), org.mockito.ArgumentMatchers.any()); + org.mockito.Mockito.verify(transport, org.mockito.Mockito.never()).release(org.mockito.ArgumentMatchers.any(), org.mockito.ArgumentMatchers.any()); + org.mockito.Mockito.verify(provisioner, org.mockito.Mockito.times(1)).provision(org.mockito.ArgumentMatchers.any(com.alibaba.qwen.code.runtimebroker.RuntimeProvisionRequest.class)); + } + } + } diff --git a/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/ManagedSessionStoreIntegrationTest.java b/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/ManagedSessionStoreIntegrationTest.java index b53cbfa055..2fa3416c79 100644 --- a/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/ManagedSessionStoreIntegrationTest.java +++ b/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/ManagedSessionStoreIntegrationTest.java @@ -77,6 +77,67 @@ class ManagedSessionStoreIntegrationTest { @Autowired private ManagedExtensionRecordStore records; + @Test + void promotesAtomicallyAndRefusesOlderHttpReadersIncludingReplay() throws Exception { + String session = "g3-reader-" + UUID.randomUUID(); + String base = "/internal/managed-session-store/v1/sessions/" + session; + String ceiling = "X-Qwen-Managed-Max-Readable-Storage-Version"; + var writer = writerRequest(WRITER_A); + mvc.perform(post(base + "/writers:acquire").header(TenantContextFilter.HEADER, TENANT) + .header(ManagedSessionStoreModels.WRITER_TOKEN_HEADER, TOKEN_A) + .contentType(MediaType.APPLICATION_JSON).content(writer.toString())).andExpect(status().isOk()); + ObjectNode body = objectMapper.createObjectNode().put("v", 1).put("promptId", "original-prompt") + .put("runtimeSessionId", "original-prompt").put("bindingId", "original-binding") + .put("generation", "1").put("workspaceGeneration", "1").put("fileHistoryTurnId", "original-prompt"); + body.putObject("sessionKey").put("tenantId", TENANT).put("workspaceId", WORKSPACE).put("sessionId", session); + byte[] bytes = body.toString().getBytes(StandardCharsets.UTF_8); + ObjectNode commit = genesisRequest("{\"subtype\":\"session_execution_engine\"}\n{\"subtype\":\"managed_session_header_v1\"}\n", bytes); + ((ObjectNode) commit.withArray("resources").get(0)).put("kind", "hosted-turn-cleanup"); + mvc.perform(post(base + "/transactions:commit").header(TenantContextFilter.HEADER, TENANT) + .header(ManagedSessionStoreModels.WRITER_TOKEN_HEADER, TOKEN_A) + .contentType(MediaType.APPLICATION_JSON).content(commit.toString())) + .andExpect(status().isConflict()).andExpect(jsonPath("$.error.code").value("managed_session_reader_version_unsupported")); + assertThat(jdbc.queryForObject("SELECT storage_version FROM qwen_managed_session_journal_head WHERE session_id = ?", Integer.class, session)).isEqualTo(1); + assertThat(jdbc.queryForObject("SELECT COUNT(*) FROM qwen_managed_session_resource WHERE session_id = ?", Integer.class, session)).isZero(); + mvc.perform(post(base + "/transactions:commit").header(TenantContextFilter.HEADER, TENANT) + .header(ManagedSessionStoreModels.WRITER_TOKEN_HEADER, TOKEN_A).header(ceiling, "2") + .contentType(MediaType.APPLICATION_JSON).content(commit.toString())).andExpect(status().isOk()); + for (String header : List.of("", "1")) { + for (String path : List.of("/restore", "/transactions", "/resources/resource-context")) { + var read = get(base + path).header(TenantContextFilter.HEADER, TENANT) + .header(ManagedSessionStoreModels.WRITER_TOKEN_HEADER, TOKEN_A).param("workspaceId", WORKSPACE); + if (!header.isEmpty()) read.header(ceiling, header); + mvc.perform(read).andExpect(status().isConflict()) + .andExpect(jsonPath("$.error.code").value("managed_session_reader_version_unsupported")); + } + var replay = post(base + "/transactions:commit").header(TenantContextFilter.HEADER, TENANT) + .header(ManagedSessionStoreModels.WRITER_TOKEN_HEADER, TOKEN_A) + .contentType(MediaType.APPLICATION_JSON).content(commit.toString()); + if (!header.isEmpty()) replay.header(ceiling, header); + mvc.perform(replay).andExpect(status().isConflict()) + .andExpect(jsonPath("$.error.code").value("managed_session_reader_version_unsupported")); + var before = jdbc.queryForMap("SELECT * FROM qwen_managed_session_journal_head WHERE session_id = ?", session); + var acquire = post(base + "/writers:acquire").header(TenantContextFilter.HEADER, TENANT) + .header(ManagedSessionStoreModels.WRITER_TOKEN_HEADER, TOKEN_A) + .contentType(MediaType.APPLICATION_JSON).content(writer.toString()); + if (!header.isEmpty()) acquire.header(ceiling, header); + mvc.perform(acquire).andExpect(status().isConflict()).andExpect(jsonPath("$.error.code").value("managed_session_reader_version_unsupported")); + var renewBody = objectMapper.createObjectNode().put("workspaceId", WORKSPACE).put("writerId", WRITER_A).put("writerGeneration", 1).put("leaseMillis", 60_000); + var renew = post(base + "/writers:renew").header(TenantContextFilter.HEADER, TENANT) + .header(ManagedSessionStoreModels.WRITER_TOKEN_HEADER, TOKEN_A).contentType(MediaType.APPLICATION_JSON).content(renewBody.toString()); + if (!header.isEmpty()) renew.header(ceiling, header); + mvc.perform(renew).andExpect(status().isConflict()).andExpect(jsonPath("$.error.code").value("managed_session_reader_version_unsupported")); + assertThat(jdbc.queryForMap("SELECT * FROM qwen_managed_session_journal_head WHERE session_id = ?", session)).isEqualTo(before); + } + mvc.perform(post(base + "/transactions:commit").header(TenantContextFilter.HEADER, TENANT) + .header(ManagedSessionStoreModels.WRITER_TOKEN_HEADER, TOKEN_A).header(ceiling, "2") + .contentType(MediaType.APPLICATION_JSON).content(commit.toString())) + .andExpect(status().isOk()).andExpect(jsonPath("$.replayed").value(true)); + mvc.perform(get(base + "/restore").header(TenantContextFilter.HEADER, TENANT) + .header(ManagedSessionStoreModels.WRITER_TOKEN_HEADER, TOKEN_A).header(ceiling, "2") + .param("workspaceId", WORKSPACE)).andExpect(status().isOk()).andExpect(jsonPath("$.storageVersion").value(2)); + } + @Test void reportsAMigrationFenceAsADefiniteConflictWithoutChangingTheWriter() throws Exception { String session = "migration-" + UUID.randomUUID(); diff --git a/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/api/SurfaceRegistry.java b/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/api/SurfaceRegistry.java index 46050e2773..0530b1c3df 100644 --- a/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/api/SurfaceRegistry.java +++ b/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/api/SurfaceRegistry.java @@ -313,6 +313,11 @@ public enum SurfaceRegistry { + "/transactions:commit", Surface.INTERNAL, RuleClass.INTERNAL_WRITER, EnumSet.of(Capability.STORE_TRANSACTION_COMMIT)), + INTERNAL_TRANSACTION_COMMIT_V2(Method.POST, + "/internal/managed-session-store/v1/sessions/{sessionId}" + + "/transactions:commit-v2", + Surface.INTERNAL, RuleClass.INTERNAL_WRITER, + EnumSet.of(Capability.STORE_TRANSACTION_COMMIT)), INTERNAL_RESTORE(Method.GET, "/internal/managed-session-store/v1/sessions/{sessionId}/restore", Surface.INTERNAL, RuleClass.INTERNAL_WRITER, diff --git a/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/store/ManagedHostedRecoveryRecordsTest.java b/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/store/ManagedHostedRecoveryRecordsTest.java new file mode 100644 index 0000000000..1ae44f7364 --- /dev/null +++ b/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/store/ManagedHostedRecoveryRecordsTest.java @@ -0,0 +1,180 @@ +package com.alibaba.qwen.code.managedagent.store; + +import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.assertj.core.api.Assertions.assertThatCode; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.databind.node.ObjectNode; +import java.nio.charset.StandardCharsets; +import org.junit.jupiter.api.Test; + +class ManagedHostedRecoveryRecordsTest { + private final ObjectMapper json = new ObjectMapper(); + + private ObjectNode cleanup() { + ObjectNode body = json.createObjectNode().put("v", 1).put("promptId", "prompt") + .put("runtimeSessionId", "prompt").put("bindingId", "binding") + .put("generation", "1").put("workspaceGeneration", "1").put("fileHistoryTurnId", "prompt"); + body.putObject("sessionKey").put("tenantId", "tenant").put("workspaceId", "workspace").put("sessionId", "session"); + return body; + } + + private void validate(String kind, ObjectNode body) { + byte[] bytes = body.toString().getBytes(StandardCharsets.UTF_8); + ManagedHostedRecoveryRecords.validateResource(new ManagedSessionStoreModels.StoredResource( + "resource", kind, 1, bytes.length, "a".repeat(64), bytes), "tenant", "workspace", "session", + id -> { throw new AssertionError("No nested resource belongs to this cleanup descriptor"); }); + } + + @Test + void requiresCompleteClosedRecoveryEnvelopes() { + for (String kind : ManagedHostedRecoveryRecords.KINDS) { + ObjectNode incomplete = cleanup(); + incomplete.remove("bindingId"); + assertThatThrownBy(() -> validate(kind, incomplete)).isInstanceOf(RuntimeException.class); + } + ObjectNode extra = cleanup().put("unexpected", true); + assertThatThrownBy(() -> validate("hosted-turn-cleanup", extra)).isInstanceOf(RuntimeException.class); + assertThatCode(() -> validate("hosted-turn-cleanup", cleanup())).doesNotThrowAnyException(); + } + + @Test + void rejectsCoercedVersionsForeignScopesAndWrongOwners() { + assertThatThrownBy(() -> validate("hosted-turn-cleanup", cleanup().put("v", "1"))).isInstanceOf(RuntimeException.class); + assertThatThrownBy(() -> validate("hosted-turn-cleanup", cleanup().put("v", 1.5))).isInstanceOf(RuntimeException.class); + ObjectNode foreign = cleanup(); + ((ObjectNode) foreign.get("sessionKey")).put("sessionId", "foreign"); + assertThatThrownBy(() -> validate("hosted-turn-cleanup", foreign)).isInstanceOf(RuntimeException.class); + assertThatThrownBy(() -> validate("hosted-turn-cleanup", cleanup().put("runtimeSessionId", "other"))).isInstanceOf(RuntimeException.class); + assertThatThrownBy(() -> validate("hosted-turn-cleanup", cleanup().put("generation", "9223372036854775808"))).isInstanceOf(RuntimeException.class); + } + + @Test + void validatesTypedReferencesBeforeResolvingThem() { + var partial = json.createObjectNode().put("resourceId", "missing").put("kind", "hosted-model-request"); + assertThatThrownBy(() -> ManagedHostedRecoveryRecords.reference(partial, "hosted-model-request", "tenant", "workspace", "session", + id -> { throw new AssertionError("Malformed ref reached storage"); })).isInstanceOf(RuntimeException.class); + var ref = partial.put("schemaVersion", 2).put("byteLength", 2).put("digest", "a".repeat(64)); + assertThatThrownBy(() -> ManagedHostedRecoveryRecords.reference(ref, "hosted-model-request", "tenant", "workspace", "session", + id -> { throw new AssertionError("Unsupported ref reached storage"); })).isInstanceOf(RuntimeException.class); + } + + @Test + void acceptsCanonicalWakeRuntimeOwner() { + String prompt = "monitor:wake:run"; + ObjectNode body = cleanup().put("promptId", prompt).put("fileHistoryTurnId", prompt) + .put("runtimeSessionId", "wake-" + ToolPublicationContract.sha256(prompt.getBytes(StandardCharsets.UTF_8))); + assertThatCode(() -> validate("hosted-turn-cleanup", body)).doesNotThrowAnyException(); + assertThatThrownBy(() -> validate("hosted-turn-cleanup", body.put("runtimeSessionId", prompt))) + .isInstanceOf(RuntimeException.class); + } + + @Test + void sharesClosureMemoAcrossRootsAndStillRejectsConflictingMetadata() { + byte[] bytes = "{}".getBytes(StandardCharsets.UTF_8); + var nested = new ManagedSessionStoreModels.StoredResource("shared", "managed-definition", 1, + bytes.length, "a".repeat(64), bytes); + var ref = json.createObjectNode().put("resourceId", "shared").put("kind", nested.kind()) + .put("schemaVersion", 1).put("byteLength", bytes.length).put("digest", nested.digest()); + var seen = new java.util.HashMap(); + var reads = new java.util.concurrent.atomic.AtomicInteger(); + java.util.function.Function reader = id -> { + reads.incrementAndGet(); + return nested; + }; + for (int i = 0; i < 12; i++) { + byte[] root = json.createObjectNode().put("v", 3).set("definitionRef", ref).toString().getBytes(StandardCharsets.UTF_8); + ManagedHostedRecoveryRecords.validateResource(new ManagedSessionStoreModels.StoredResource( + "root-" + i, "managed-action-options", 1, root.length, "b".repeat(64), root), + "tenant", "workspace", "session", reader, seen); + } + org.assertj.core.api.Assertions.assertThat(reads.get()).isEqualTo(1); + assertThatThrownBy(() -> ManagedHostedRecoveryRecords.reference(ref.deepCopy().put("digest", "c".repeat(64)), + "managed-definition", "tenant", "workspace", "session", reader, seen)) + .isInstanceOf(RuntimeException.class); + org.assertj.core.api.Assertions.assertThat(reads.get()).isEqualTo(1); + } + + @Test + void acceptsDigestCleanupIdentityForTheMaximumStablePrompt() { + String prompt = "a".repeat(512); + String original = "hosted-cleanup:" + prompt + ":" + prompt; + String cleanupId = "hosted-cleanup:" + ToolPublicationContract.sha256(original.getBytes(StandardCharsets.UTF_8)); + ObjectNode body = cleanup().put("promptId", prompt).put("runtimeSessionId", prompt).put("fileHistoryTurnId", prompt); + var resource = stored("cleanup", "hosted-turn-cleanup", body); + ObjectNode payload = json.createObjectNode().put("cleanupId", cleanupId).put("state", "owed"); + payload.set("descriptorRef", reference(resource)); + for (String state : java.util.List.of("owed", "confirmed")) { + payload.put("state", state); + assertThatCode(() -> ManagedHostedRecoveryRecords.validateEvent("hosted.cleanup", payload, + "tenant", "workspace", "session", id -> resource)).doesNotThrowAnyException(); + } + payload.put("cleanupId", original); + assertThatThrownBy(() -> ManagedHostedRecoveryRecords.validateEvent("hosted.cleanup", payload, + "tenant", "workspace", "session", id -> resource)).hasMessageContaining("512 UTF-8 bytes"); + } + + @Test + void acceptsLegacyAndBooleanPreparedRetryPolicies() { + ObjectNode legacy = modelRequest(); + assertThatCode(() -> validateModel(legacy)).doesNotThrowAnyException(); + for (boolean allow : new boolean[] {true, false}) { + ObjectNode body = modelRequest(); + ((ObjectNode) body.get("prepared")).put("allowPersistentRetry", allow); + assertThatCode(() -> validateModel(body)).doesNotThrowAnyException(); + } + } + + @Test + void rejectsInvalidPreparedRetryTypesAndUnknownFields() { + for (var value : java.util.List.of(json.getNodeFactory().nullNode(), json.getNodeFactory().textNode("true"), + json.getNodeFactory().numberNode(1), json.createObjectNode(), json.createArrayNode())) { + ObjectNode body = modelRequest(); + ((ObjectNode) body.get("prepared")).set("allowPersistentRetry", value); + assertThatThrownBy(() -> validateModel(body)).hasMessageContaining("retry policy"); + } + ObjectNode legacyExtra = modelRequest(); + ((ObjectNode) legacyExtra.get("prepared")).put("unexpected", true); + assertThatThrownBy(() -> validateModel(legacyExtra)).hasMessageContaining("exactly"); + ObjectNode currentExtra = modelRequest(); + ((ObjectNode) currentExtra.get("prepared")).put("allowPersistentRetry", true).put("unexpected", true); + assertThatThrownBy(() -> validateModel(currentExtra)).hasMessageContaining("exactly"); + } + + private ObjectNode modelRequest() { + ObjectNode body = json.createObjectNode().put("v", 1).put("promptId", "prompt") + .put("sourceBootId", "boot").put("sourceEventEpoch", "epoch").put("messageId", "message") + .put("round", 0).put("throughSequence", 1).put("pendingToolResults", false).putNull("workspaceContext"); + body.putObject("sessionKey").put("tenantId", "tenant").put("workspaceId", "workspace").put("sessionId", "session"); + body.set("definitionRef", reference(stored("definition", "managed-definition", json.createObjectNode()))); + body.set("rootSnapshotRef", reference(stored("root", "managed-root", json.createObjectNode()))); + body.putObject("sourceActivation").put("activationId", "activation").put("epoch", 1); + ObjectNode prepared = body.putObject("prepared").put("routeSelector", "openai:fixture") + .put("providerPin", "a".repeat(64)).put("promptTokensForClamp", 0); + ObjectNode request = prepared.putObject("request").put("model", "fixture"); + request.putArray("contents"); + request.putObject("config"); + prepared.putArray("history"); + prepared.putArray("completedToolCallIds"); + return body; + } + + private void validateModel(ObjectNode body) { + body.put("requestDigest", ToolPublicationContract.sha256(body.get("prepared").toString().getBytes(StandardCharsets.UTF_8))); + var resources = java.util.Map.of( + "definition", stored("definition", "managed-definition", json.createObjectNode()), + "root", stored("root", "managed-root", json.createObjectNode())); + ManagedHostedRecoveryRecords.validateResource(stored("request", "hosted-model-request", body), + "tenant", "workspace", "session", resources::get); + } + + private ManagedSessionStoreModels.StoredResource stored(String id, String kind, ObjectNode body) { + byte[] bytes = body.toString().getBytes(StandardCharsets.UTF_8); + return new ManagedSessionStoreModels.StoredResource(id, kind, 1, bytes.length, ToolPublicationContract.sha256(bytes), bytes); + } + + private ObjectNode reference(ManagedSessionStoreModels.StoredResource resource) { + return json.createObjectNode().put("resourceId", resource.resourceId()).put("kind", resource.kind()) + .put("schemaVersion", resource.schemaVersion()).put("byteLength", resource.byteLength()).put("digest", resource.digest()); + } +} diff --git a/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/store/ManagedSessionReaderVersionTest.java b/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/store/ManagedSessionReaderVersionTest.java new file mode 100644 index 0000000000..c3ba22f20f --- /dev/null +++ b/packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/store/ManagedSessionReaderVersionTest.java @@ -0,0 +1,62 @@ +package com.alibaba.qwen.code.managedagent.store; + +import static org.junit.jupiter.api.Assertions.assertDoesNotThrow; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; + +import com.alibaba.qwen.code.managedagent.api.ApiException; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; +import org.springframework.http.HttpStatus; +import org.springframework.mock.web.MockHttpServletRequest; +import org.springframework.web.context.request.RequestContextHolder; +import org.springframework.web.context.request.ServletRequestAttributes; + +class ManagedSessionReaderVersionTest { + @AfterEach + void resetRequest() { + RequestContextHolder.resetRequestAttributes(); + } + + @ParameterizedTest + @ValueSource(strings = { + "/internal/managed-session-store/session/restore", + "/%69%6Eternal/managed-session-store/session/restore", + "/internal/managed-session-store;x=y/session/restore", + "/internal/managed-tool-publications/session/replay", + "/%69%6Eternal/managed-tool-publications/session/replay", + "/internal/managed-tool-publications;x=y/session/replay" + }) + void normalizedInternalRoutesRequireTheRemoteReaderCeiling(String path) { + var request = new MockHttpServletRequest(); + request.setContextPath("/app"); + request.setRequestURI("/app" + path); + RequestContextHolder.setRequestAttributes(new ServletRequestAttributes(request)); + + assertDoesNotThrow(() -> ManagedSessionReaderVersion.requireReadable(1)); + assertUnsupported(); + request.addHeader(ManagedSessionReaderVersion.HEADER, "1"); + assertUnsupported(); + request.removeHeader(ManagedSessionReaderVersion.HEADER); + request.addHeader(ManagedSessionReaderVersion.HEADER, "2"); + assertDoesNotThrow(() -> ManagedSessionReaderVersion.requireReadable(2)); + } + + @Test + void publicAndInProcessReadersRetainBothFormats() { + assertDoesNotThrow(() -> ManagedSessionReaderVersion.requireReadable(2)); + var request = new MockHttpServletRequest(); + request.setRequestURI("/v1/agents/agent/sessions/session"); + RequestContextHolder.setRequestAttributes(new ServletRequestAttributes(request)); + assertDoesNotThrow(() -> ManagedSessionReaderVersion.requireReadable(2)); + } + + private static void assertUnsupported() { + var failure = assertThrows(ApiException.class, + () -> ManagedSessionReaderVersion.requireReadable(2)); + assertEquals(HttpStatus.CONFLICT, failure.getStatus()); + assertEquals("managed_session_reader_version_unsupported", failure.getCode()); + } +} diff --git a/packages/sdk-java/qwencode/src/main/java/com/alibaba/qwen/code/daemon/HostedHarnessCapabilities.java b/packages/sdk-java/qwencode/src/main/java/com/alibaba/qwen/code/daemon/HostedHarnessCapabilities.java index 9d49bad854..9266c907af 100644 --- a/packages/sdk-java/qwencode/src/main/java/com/alibaba/qwen/code/daemon/HostedHarnessCapabilities.java +++ b/packages/sdk-java/qwencode/src/main/java/com/alibaba/qwen/code/daemon/HostedHarnessCapabilities.java @@ -11,8 +11,11 @@ public final class HostedHarnessCapabilities { private final String bootId; private final String capabilityDigest; private final int lifecycleProtocolVersion; + private final List features; private final int titleProtocolVersion; + public List getFeatures() { return features; } + public int getTitleProtocolVersion() { return titleProtocolVersion; } public int getLifecycleProtocolVersion() { return lifecycleProtocolVersion; } @@ -25,11 +28,12 @@ public final class HostedHarnessCapabilities { HostedHarnessCapabilities(int currentProtocolVersion, List supportedProtocolVersions, String bootId, String capabilityDigest, int lifecycleProtocolVersion) { - this(currentProtocolVersion, supportedProtocolVersions, bootId, capabilityDigest, lifecycleProtocolVersion, 0); + this(currentProtocolVersion, supportedProtocolVersions, bootId, capabilityDigest, lifecycleProtocolVersion, 0, List.of()); } HostedHarnessCapabilities(int currentProtocolVersion, List supportedProtocolVersions, - String bootId, String capabilityDigest, int lifecycleProtocolVersion, int titleProtocolVersion) { + String bootId, String capabilityDigest, int lifecycleProtocolVersion, int titleProtocolVersion, List features) { + this.features = List.copyOf(features); this.titleProtocolVersion = titleProtocolVersion; this.lifecycleProtocolVersion = lifecycleProtocolVersion; this.currentProtocolVersion = currentProtocolVersion; diff --git a/packages/sdk-java/qwencode/src/main/java/com/alibaba/qwen/code/daemon/HostedHarnessClient.java b/packages/sdk-java/qwencode/src/main/java/com/alibaba/qwen/code/daemon/HostedHarnessClient.java index f816308d84..c22e0dbe40 100644 --- a/packages/sdk-java/qwencode/src/main/java/com/alibaba/qwen/code/daemon/HostedHarnessClient.java +++ b/packages/sdk-java/qwencode/src/main/java/com/alibaba/qwen/code/daemon/HostedHarnessClient.java @@ -982,11 +982,24 @@ public final class HostedHarnessClient implements AutoCloseable { throw new HostedHarnessCapabilityMismatchException( expectedDigest, digest); } + List hostedFeatures = new ArrayList<>(); + if (hosted.containsKey("features")) { + if (!(hosted.get("features") instanceof List)) { + throw new DaemonProtocolException("Hosted Harness features must be an array"); + } + for (Object value : (List) hosted.get("features")) { + if (!(value instanceof String) || ((String) value).isBlank() || hostedFeatures.contains(value)) { + throw new DaemonProtocolException("Invalid Hosted Harness feature"); + } + hostedFeatures.add((String) value); + } + } return new HostedHarnessCapabilities(current, supported, bootId, digest, hosted.containsKey("lifecycleProtocolVersion") ? JsonSupport.requiredInt(hosted, "lifecycleProtocolVersion", "capabilities.hostedHarness") : 0, hosted.containsKey("titleProtocolVersion") - ? JsonSupport.requiredInt(hosted, "titleProtocolVersion", "capabilities.hostedHarness") : 0); + ? JsonSupport.requiredInt(hosted, "titleProtocolVersion", "capabilities.hostedHarness") : 0, + hostedFeatures); } private HarnessSessionRef parseSession(String body, diff --git a/packages/sdk-java/qwencode/src/test/java/com/alibaba/qwen/code/daemon/HostedHarnessClientTest.java b/packages/sdk-java/qwencode/src/test/java/com/alibaba/qwen/code/daemon/HostedHarnessClientTest.java index fbd81da834..2d106b23ca 100644 --- a/packages/sdk-java/qwencode/src/test/java/com/alibaba/qwen/code/daemon/HostedHarnessClientTest.java +++ b/packages/sdk-java/qwencode/src/test/java/com/alibaba/qwen/code/daemon/HostedHarnessClientTest.java @@ -1063,7 +1063,7 @@ class HostedHarnessClientTest { server.removeContext("/capabilities"); server.createContext("/capabilities", exchange -> sendJson(exchange, 200, capabilitiesJson(DIGEST, BOOT_ID).replace("\"hostedHarness\":{", - "\"hostedHarness\":{\"titleProtocolVersion\":1,"), false)); + "\"hostedHarness\":{\"titleProtocolVersion\":1,\"features\":[\"hosted_approval_resume_v1\",\"hosted_model_round_recovery_v1\"],"), false)); AtomicReference body = new AtomicReference<>(); server.createContext("/session/" + SESSION_ID + "/title", exchange -> { body.set(readBody(exchange)); @@ -1071,6 +1071,9 @@ class HostedHarnessClientTest { + "\",\"persisted\":true,\"managedRenameRevision\":\"7\"}"); }); try (HostedHarnessClient client = newClient()) { + assertEquals(1, client.capabilities().getTitleProtocolVersion()); + assertEquals(List.of("hosted_approval_resume_v1", "hosted_model_round_recovery_v1"), + client.capabilities().getFeatures()); client.updateSessionTitle(createSession(client), "A", 7); } assertTrue(body.get().contains("\"managedRenameRevision\":\"7\"")); diff --git a/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/InMemoryRuntimeBindingRepository.java b/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/InMemoryRuntimeBindingRepository.java index e3b4fc6720..c0feecc0f4 100644 --- a/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/InMemoryRuntimeBindingRepository.java +++ b/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/InMemoryRuntimeBindingRepository.java @@ -181,6 +181,16 @@ public final class InMemoryRuntimeBindingRepository return sessions.findOrCreate(candidate); } + @Override + public synchronized RuntimeSessionRecord createSessionTombstone( + RuntimeSessionRepository sessions, RuntimeSessionRecord candidate) { + if (candidate.getState() != RuntimeSessionRecord.State.RELEASED) { + throw new IllegalArgumentException("Tombstone must be RELEASED"); + } + RuntimeAdmission.requireRelease(findById(candidate.getBindingId()), candidate); + return sessions.findOrCreate(candidate); + } + @Override public synchronized ToolExecutionRecord admitExecution( RuntimeSessionRepository sessions, ToolExecutionRepository executions, diff --git a/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/InMemoryRuntimeSessionRepository.java b/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/InMemoryRuntimeSessionRepository.java index b103f50b0a..06c53d045f 100644 --- a/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/InMemoryRuntimeSessionRepository.java +++ b/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/InMemoryRuntimeSessionRepository.java @@ -25,10 +25,10 @@ public final class InMemoryRuntimeSessionRepository public synchronized RuntimeSessionRecord findOrCreate( RuntimeSessionRecord candidate) { if (candidate == null || candidate.getVersion() != 0 - || candidate.getState() - != RuntimeSessionRecord.State.ACQUIRING) { + || (candidate.getState() != RuntimeSessionRecord.State.ACQUIRING + && candidate.getState() != RuntimeSessionRecord.State.RELEASED)) { throw new IllegalArgumentException( - "candidate must be a new acquiring Session"); + "candidate must be a new acquiring Session or release tombstone"); } RuntimeScope scope = candidate.getSession().getScope(); Map scopedRecords = records.computeIfAbsent( diff --git a/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/JdbcRuntimeBindingRepository.java b/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/JdbcRuntimeBindingRepository.java index 9d33151421..afdd6ba584 100644 --- a/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/JdbcRuntimeBindingRepository.java +++ b/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/JdbcRuntimeBindingRepository.java @@ -314,6 +314,37 @@ public final class JdbcRuntimeBindingRepository }); } + @Override + public RuntimeSessionRecord createSessionTombstone(RuntimeSessionRepository sessions, + RuntimeSessionRecord candidate) { + if (!(sessions instanceof JdbcRuntimeSessionRepository jdbcSessions) + || !jdbcSessions.usesDataSource(dataSource)) { + throw new IllegalArgumentException("Retirement requires the same DataSource"); + } + JdbcRuntimeSessionRepository.requireCandidate(candidate); + if (candidate.getState() != RuntimeSessionRecord.State.RELEASED) { + throw new IllegalArgumentException("Tombstone must be RELEASED"); + } + return JdbcRepositorySupport.transaction(dataSource, connection -> { + RuntimeScope scope = candidate.getSession().getScope(); + lockPlacementDomain(connection, scope.getTenantId()); + RuntimeBindingRecord binding = selectById(connection, candidate.getBindingId(), true); + RuntimeAdmission.requireRelease(binding, candidate); + requireHarnessAdmission(connection, scope, candidate.getSession().getHarnessSessionId(), + scope.getLifecycleAuthority(), false, true); + RuntimeSessionRecord existing = JdbcRuntimeSessionRepository.selectSession( + connection, scope, candidate.getRuntimeSessionId(), true); + if (existing != null) { + if (!existing.sameIdentity(candidate)) { + throw new IllegalArgumentException("Session identity differs"); + } + return existing; + } + JdbcRuntimeSessionRepository.insertSession(connection, candidate); + return candidate; + }); + } + @Override public ToolExecutionRecord admitExecution(RuntimeSessionRepository sessions, ToolExecutionRepository executions, ToolExecutionRecord candidate) { @@ -461,6 +492,11 @@ public final class JdbcRuntimeBindingRepository private static void requireHarnessAdmission(Connection connection, RuntimeScope scope, String harnessSessionId, RuntimeLifecycleAuthority authority, boolean legacyHook) throws SQLException { + requireHarnessAdmission(connection, scope, harnessSessionId, authority, legacyHook, false); + } + + private static void requireHarnessAdmission(Connection connection, RuntimeScope scope, + String harnessSessionId, RuntimeLifecycleAuthority authority, boolean legacyHook, boolean retirement) throws SQLException { if (!"session".equals(scope.getIsolationClass())) { if (authority != null) { throw admissionClosed(); @@ -483,6 +519,9 @@ public final class JdbcRuntimeBindingRepository || !harnessSessionId.equals(row.getString("harness_session_id"))) { throw admissionClosed(); } + if (retirement && authority == null && "DRAINING".equals(row.getString("phase"))) { + return; + } if (legacyHook && authority == null && "DRAINING".equals(row.getString("phase")) && row.getString("operation_id") == null) { return; diff --git a/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/JdbcRuntimeSessionRepository.java b/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/JdbcRuntimeSessionRepository.java index 8c26419555..a1b4911578 100644 --- a/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/JdbcRuntimeSessionRepository.java +++ b/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/JdbcRuntimeSessionRepository.java @@ -372,10 +372,10 @@ public final class JdbcRuntimeSessionRepository static void requireCandidate(RuntimeSessionRecord candidate) { if (candidate == null || candidate.getVersion() != 0 - || candidate.getState() - != RuntimeSessionRecord.State.ACQUIRING) { + || (candidate.getState() != RuntimeSessionRecord.State.ACQUIRING + && candidate.getState() != RuntimeSessionRecord.State.RELEASED)) { throw new IllegalArgumentException( - "candidate must be a new acquiring Session"); + "candidate must be a new acquiring Session or release tombstone"); } } diff --git a/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/RuntimeBindingRepository.java b/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/RuntimeBindingRepository.java index fae2c93484..d0226f71fb 100644 --- a/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/RuntimeBindingRepository.java +++ b/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/RuntimeBindingRepository.java @@ -10,6 +10,13 @@ public interface RuntimeBindingRepository { RuntimeSessionRecord admitSession(RuntimeSessionRepository sessions, RuntimeSessionRecord candidate); + /** Creates a RELEASED receipt under the same generation lock as admission. */ + default RuntimeSessionRecord createSessionTombstone(RuntimeSessionRepository sessions, + RuntimeSessionRecord candidate) { + throw new RuntimeBrokerException(501, "runtime_session_tombstone_unavailable", + "Runtime repository does not support atomic Session retirement", false); + } + /** Existing idempotency receipts remain readable after admission closes. */ ToolExecutionRecord admitExecution(RuntimeSessionRepository sessions, ToolExecutionRepository executions, ToolExecutionRecord candidate); diff --git a/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/RuntimeBrokerHttpServer.java b/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/RuntimeBrokerHttpServer.java index eb840821c9..86cecdb545 100644 --- a/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/RuntimeBrokerHttpServer.java +++ b/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/RuntimeBrokerHttpServer.java @@ -110,8 +110,13 @@ public final class RuntimeBrokerHttpServer implements AutoCloseable { requireProtocol(body); JsonCodec.requiredString(body, "requestId", "warm request"); String sessionId = JsonCodec.requiredString(body, "harnessSessionId", "warm request"); - complete(exchange, service.warm(sessionId, lifecycleAuthority(exchange)), ignored -> Map.of( - "protocolVersion", 1, "harnessSessionId", sessionId, "ready", true)); + complete(exchange, service.warm(sessionId, lifecycleAuthority(exchange)), record -> Map.of( + "protocolVersion", 1, "harnessSessionId", sessionId, "ready", true, + "scope", Map.of("tenantId", record.getRequest().getScope().getTenantId(), + "workspaceId", record.getRequest().getScope().getWorkspaceId(), + "workspaceGeneration", record.getRequest().getScope().getWorkspaceGeneration(), + "capabilityDigest", record.getRequest().getScope().getCapabilityDigest()), + "runtime", Map.of("bindingId", record.getBindingId(), "generation", Long.toString(record.getGeneration())))); return; } if ("POST".equals(exchange.getRequestMethod()) && "/runtimes:authorize-lifecycle".equals(relative)) { @@ -246,12 +251,13 @@ public final class RuntimeBrokerHttpServer implements AutoCloseable { JsonCodec.requiredString(body, "requestId", "release request"); String harnessSessionId = JsonCodec.requiredString(body, "harnessSessionId", "release request"); - // `drain` is the close arm: it stops the Session's live - // background holds before releasing. A routine release omits it - // and is refused `runtime_session_held` while a run is live. boolean drain = Boolean.TRUE.equals(body.get("drain")); - complete(exchange, service.release(harnessSessionId, - runtimeSessionId, drain), released -> envelope(harnessSessionId, + CompletionStage release = body.containsKey("cleanupBindingId") + ? service.releaseOriginal(harnessSessionId, runtimeSessionId, + JsonCodec.requiredString(body, "cleanupBindingId", "release request"), + Long.parseLong(JsonCodec.requiredString(body, "cleanupGeneration", "release request")), lifecycleAuthority(exchange), drain) + : service.release(harnessSessionId, runtimeSessionId, drain); + complete(exchange, release, released -> envelope(harnessSessionId, runtimeSessionId, "released", released)); return; } diff --git a/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/RuntimeBrokerService.java b/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/RuntimeBrokerService.java index 149b7645c8..ecea42e754 100644 --- a/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/RuntimeBrokerService.java +++ b/packages/sdk-java/runtime-broker/src/main/java/com/alibaba/qwen/code/runtimebroker/RuntimeBrokerService.java @@ -1668,6 +1668,41 @@ public final class RuntimeBrokerService implements AutoCloseable { } } + public CompletionStage releaseOriginal(String harnessSessionId, String runtimeSessionId, + String bindingId, long generation) { + return releaseOriginal(harnessSessionId, runtimeSessionId, bindingId, generation, null); + } + + public CompletionStage releaseOriginal(String harnessSessionId, String runtimeSessionId, + String bindingId, long generation, RuntimeLifecycleAuthority authority) { + return releaseOriginal(harnessSessionId, runtimeSessionId, bindingId, generation, authority, false); + } + + public CompletionStage releaseOriginal(String harnessSessionId, String runtimeSessionId, + String bindingId, long generation, RuntimeLifecycleAuthority authority, boolean drain) { + requireOpen(); + RuntimeBindingRecord binding = bindingRepository.findById(bindingId); + if (binding == null || binding.getGeneration() != generation + || !harnessSessionId.equals(binding.getRequest().getIsolationKey()) + || !"session".equals(binding.getRequest().getScope().getIsolationClass())) { + return failed(conflict("runtime_session_conflict", "Cleanup owner differs from original binding")); + } + RuntimeSessionRecord stored = sessionRepository.findById(binding.getRequest().getScope(), runtimeSessionId); + if (stored == null) { + // Admission and this tombstone serialize under the binding lock. A late + // acquire must observe RELEASED instead of creating a new owner. + stored = bindingRepository.createSessionTombstone(sessionRepository, new RuntimeSessionRecord( + new RuntimeSession(harnessSessionId, runtimeSessionId, "bootstrap", binding.getRequest().getScope().withLifecycleAuthority(authority)), + bindingId, generation, RuntimeSessionRecord.State.RELEASED, 0, clock.instant())); + } + if (!bindingId.equals(stored.getBindingId()) || stored.getRuntimeGeneration() != generation + || !harnessSessionId.equals(stored.getSession().getHarnessSessionId())) { + return failed(conflict("runtime_session_conflict", "Cleanup owner differs from original Session")); + } + return stored.getState() == RuntimeSessionRecord.State.RELEASED + ? CompletableFuture.completedFuture(true) : release(harnessSessionId, runtimeSessionId, drain); + } + /** * A routine release holds nothing back and stops nothing: a Session * with a provably live hold row is refused {@code runtime_session_held} diff --git a/packages/sdk-java/runtime-broker/src/test/java/com/alibaba/qwen/code/runtimebroker/RuntimeBrokerServiceTest.java b/packages/sdk-java/runtime-broker/src/test/java/com/alibaba/qwen/code/runtimebroker/RuntimeBrokerServiceTest.java index d3ca13aa74..df246b833b 100644 --- a/packages/sdk-java/runtime-broker/src/test/java/com/alibaba/qwen/code/runtimebroker/RuntimeBrokerServiceTest.java +++ b/packages/sdk-java/runtime-broker/src/test/java/com/alibaba/qwen/code/runtimebroker/RuntimeBrokerServiceTest.java @@ -47,6 +47,46 @@ class RuntimeBrokerServiceTest { "tenant", "workspace", "generation", "/workspace", "capability", "session"); + @Test + void cleanupBeforeAcquireTombstonesTheOriginalOwnerWithoutAcquiringALease() { + try (Fixture fixture = new Fixture(SESSION_SCOPE)) { + RuntimeBindingRecord binding = join(fixture.service.warm("harness-a")); + assertTrue(join(fixture.service.releaseOriginal("harness-a", "runtime-a", binding.getBindingId(), binding.getGeneration()))); + assertTrue(join(fixture.service.releaseOriginal("harness-a", "runtime-a", binding.getBindingId(), binding.getGeneration()))); + assertEquals(RuntimeSessionRecord.State.RELEASED, fixture.sessionRepository.findById(SESSION_SCOPE, "runtime-a").getState()); + assertEquals(0, fixture.transport.acquireCalls.get()); + assertEquals(0, fixture.transport.releaseCalls.get()); + assertEquals("runtime_session_not_acquirable", failure(fixture.service.acquire("harness-a", "runtime-a", "bootstrap")).getCode()); + assertEquals(0, fixture.transport.acquireCalls.get()); + } + } + + @Test + void cleanupBeforeAcquireRemainsAvailableAfterAdmissionCloses() { + for (var state : List.of(RuntimeBindingRecord.State.READY, RuntimeBindingRecord.State.DRAINING)) { + try (Fixture fixture = new Fixture(SESSION_SCOPE)) { + RuntimeBindingRecord binding = join(fixture.service.warm("harness-a")); + binding = fixture.bindingRepository.findById(binding.getBindingId()); + binding = fixture.bindingRepository.compareAndSet(binding, + binding.withState(state, binding.getLease(), START).withDrainRequested(true, START)); + assertTrue(join(fixture.service.releaseOriginal("harness-a", "runtime-a", + binding.getBindingId(), binding.getGeneration()))); + assertTrue(join(fixture.service.releaseOriginal("harness-a", "runtime-a", + binding.getBindingId(), binding.getGeneration()))); + assertEquals(RuntimeSessionRecord.State.RELEASED, + fixture.sessionRepository.findById(SESSION_SCOPE, "runtime-a").getState()); + assertEquals(0, fixture.transport.acquireCalls.get()); + assertEquals(0, fixture.transport.releaseCalls.get()); + fixture.bindingRepository.compareAndSet(binding, + binding.withState(RuntimeBindingRecord.State.READY, binding.getLease(), START) + .withDrainRequested(false, START)); + assertEquals("runtime_session_not_acquirable", + failure(fixture.service.acquire("harness-a", "runtime-a", "bootstrap")).getCode()); + assertEquals(0, fixture.transport.acquireCalls.get()); + } + } + } + @Test void coldReleaseRejectsHistoricalAmbiguityWithAStableConflict() { var target = new RuntimeScope("tenant", "workspace", "generation", "/target", @@ -1725,6 +1765,15 @@ class RuntimeBrokerServiceTest { @Test void monitorWatchHoldsItsRuntimeLikeABackgroundShell() throws Exception { + verifyMonitorHoldRelease(false); + } + + @Test + void originalOwnerFencePreservesTheHoldAndAllowsOnlyExplicitDrain() throws Exception { + verifyMonitorHoldRelease(true); + } + + private void verifyMonitorHoldRelease(boolean originalFence) throws Exception { // The Monitor family rides the same hold-row shape as the Shell: a // routine release answers held and stops nothing, the drain issues // the watch's stop through its own maintenance kinds (#13856). @@ -1751,8 +1800,8 @@ class RuntimeBrokerServiceTest { "bindingGeneration", "1")); } }; - try (Fixture fixture = new Fixture(WORKSPACE_SCOPE, verifier)) { - join(fixture.service.acquire("harness", "runtime", "bootstrap")); + try (Fixture fixture = new Fixture(originalFence ? SESSION_SCOPE : WORKSPACE_SCOPE, verifier)) { + RuntimeSessionRecord owner = join(fixture.service.acquire("harness", "runtime", "bootstrap")); Map reference = Map.of("sessionId", "runtime", "promptId", "prompt", "callId", "call", "argsDigest", "sha256:" + "a".repeat(64)); fixture.transport.executeV3Result = CompletableFuture.completedFuture( @@ -1775,7 +1824,10 @@ class RuntimeBrokerServiceTest { .completedFuture(Map.of("operationId", "call", "state", "running", "unitName", "qwen-mon-call")); assertEquals("runtime_session_held", - failure(fixture.service.release("harness", "runtime")).getCode()); + failure(originalFence + ? fixture.service.releaseOriginal("harness", "runtime", owner.getBindingId(), + owner.getRuntimeGeneration(), null, false) + : fixture.service.release("harness", "runtime")).getCode()); assertEquals(java.util.List.of("monitor-status"), fixture.transport.controls.stream() .map(each -> (String) each.get("kind")).toList()); @@ -1794,7 +1846,15 @@ class RuntimeBrokerServiceTest { "evidence", Map.of("exitSignal", "SIGTERM")) : Map.of("operationId", "call", "state", "running", "unitName", "qwen-mon-call")); - assertTrue(join(fixture.service.release("harness", "runtime", true))); + if (originalFence) { + assertEquals("runtime_session_conflict", failure(fixture.service.releaseOriginal( + "harness", "runtime", owner.getBindingId(), owner.getRuntimeGeneration() + 1, null, true)).getCode()); + assertEquals(0, fixture.transport.releaseCalls.get()); + } + assertTrue(join(originalFence + ? fixture.service.releaseOriginal("harness", "runtime", owner.getBindingId(), + owner.getRuntimeGeneration(), null, true) + : fixture.service.release("harness", "runtime", true))); assertEquals(java.util.List.of("monitor-status", "monitor-stop"), fixture.transport.controls.stream() .map(each -> (String) each.get("kind")).toList()); diff --git a/packages/sdk-java/runtime-broker/src/test/java/com/alibaba/qwen/code/runtimebroker/RuntimeSessionTombstoneTest.java b/packages/sdk-java/runtime-broker/src/test/java/com/alibaba/qwen/code/runtimebroker/RuntimeSessionTombstoneTest.java new file mode 100644 index 0000000000..5a75580987 --- /dev/null +++ b/packages/sdk-java/runtime-broker/src/test/java/com/alibaba/qwen/code/runtimebroker/RuntimeSessionTombstoneTest.java @@ -0,0 +1,98 @@ +package com.alibaba.qwen.code.runtimebroker; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertThrows; + +import java.time.Instant; +import java.time.Duration; +import java.util.UUID; +import org.h2.jdbcx.JdbcDataSource; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; + +class RuntimeSessionTombstoneTest { + private static final RuntimeScope SCOPE = new RuntimeScope( + "tenant", "workspace", "generation", "/workspace", "capability", "session"); + + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void retirementUsesReleaseRulesAndKeepsAdmissionClosed(boolean jdbc) { + var fixture = fixture(jdbc); + var binding = fixture.binding; + binding = fixture.bindings.compareAndSet(binding, + binding.withState(RuntimeBindingRecord.State.DRAINING, null, Instant.now()) + .withDrainRequested(true, Instant.now())); + fixture.bindings.requestHarnessDrain("tenant", "harness"); + var tombstone = record(binding, "runtime", RuntimeSessionRecord.State.RELEASED); + assertEquals(RuntimeSessionRecord.State.RELEASED, + fixture.bindings.createSessionTombstone(fixture.sessions, tombstone).getState()); + assertEquals(RuntimeSessionRecord.State.RELEASED, + fixture.bindings.createSessionTombstone(fixture.sessions, tombstone).getState()); + var closed = binding; + var failure = assertThrows(RuntimeBrokerException.class, + () -> fixture.bindings.admitSession(fixture.sessions, + record(closed, "new-runtime", RuntimeSessionRecord.State.ACQUIRING))); + assertEquals("runtime_admission_closed", failure.getCode()); + assertNull(fixture.sessions.findById(SCOPE, "new-runtime")); + } + + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void tombstoneCannotOverwriteAnAcquiredOwnerOrOpenAdmission(boolean jdbc) { + var fixture = fixture(jdbc); + var acquired = fixture.bindings.admitSession(fixture.sessions, + record(fixture.binding, "runtime", RuntimeSessionRecord.State.ACQUIRING)); + assertEquals(acquired.getState(), fixture.bindings.createSessionTombstone(fixture.sessions, + record(fixture.binding, "runtime", RuntimeSessionRecord.State.RELEASED)).getState()); + assertThrows(IllegalArgumentException.class, + () -> fixture.bindings.createSessionTombstone(fixture.sessions, + record(fixture.binding, "new-runtime", RuntimeSessionRecord.State.READY))); + assertNull(fixture.sessions.findById(SCOPE, "new-runtime")); + } + + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void retirementRejectsAnUnreconciledOriginalGeneration(boolean jdbc) { + var fixture = fixture(jdbc); + var binding = fixture.binding; + fixture.bindings.compareAndSet(binding, + binding.withState(RuntimeBindingRecord.State.LOST, null, Instant.now())); + var failure = assertThrows(RuntimeBrokerException.class, + () -> fixture.bindings.createSessionTombstone(fixture.sessions, + record(binding, "runtime", RuntimeSessionRecord.State.RELEASED))); + assertEquals("runtime_reconciliation_required", failure.getCode()); + assertEquals(503, failure.getStatusCode()); + assertNull(fixture.sessions.findById(SCOPE, "runtime")); + } + + private static RuntimeSessionRecord record(RuntimeBindingRecord binding, String id, + RuntimeSessionRecord.State state) { + return new RuntimeSessionRecord(new RuntimeSession("harness", id, "bootstrap", SCOPE), + binding.getBindingId(), binding.getGeneration(), state, 0, Instant.now()); + } + + private static Fixture fixture(boolean jdbc) { + RuntimeBindingRepository bindings; + RuntimeSessionRepository sessions; + if (jdbc) { + var source = new JdbcDataSource(); + source.setURL("jdbc:h2:mem:tombstone-" + UUID.randomUUID() + + ";MODE=MySQL;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=TRUE"); + JdbcRuntimeBrokerSchema.initialize(source); + bindings = new JdbcRuntimeBindingRepository(source, new AesGcmSecretProtector("test", new byte[32])); + sessions = new JdbcRuntimeSessionRepository(source); + } else { + bindings = new InMemoryRuntimeBindingRepository(); + sessions = new InMemoryRuntimeSessionRepository(); + } + var binding = bindings.findOrCreate(new RuntimeProvisionRequest(SCOPE, "harness")); + binding = bindings.claimOperation(binding.getBindingId(), "test-owner", Duration.ofMinutes(1)); + binding = bindings.compareAndSet(binding, + binding.withState(RuntimeBindingRecord.State.READY, null, Instant.now())); + return new Fixture(bindings, sessions, binding); + } + + private record Fixture(RuntimeBindingRepository bindings, RuntimeSessionRepository sessions, + RuntimeBindingRecord binding) {} +} diff --git a/scripts/run-managed-agent-server-e2e.ts b/scripts/run-managed-agent-server-e2e.ts index 9c1f16d4a3..dc2492a51c 100644 --- a/scripts/run-managed-agent-server-e2e.ts +++ b/scripts/run-managed-agent-server-e2e.ts @@ -17,8 +17,59 @@ import { fakeToolCall, startFakeOpenAIServer, } from '../integration-tests/fake-openai-server.js'; +import { isDeepStrictEqual } from 'node:util'; const root = process.cwd(); +const verificationMode = process.env['G3_VERIFY_MODE']; +if ( + verificationMode && + !['allow', 'deny', 'cancel', 'expiry', 'continuation'].includes( + verificationMode, + ) +) { + throw new Error( + 'G3_VERIFY_MODE must be allow, deny, cancel, expiry or continuation', + ); +} +const approvalRecovery = + !!verificationMode && verificationMode !== 'continuation'; +const verificationDirectory = path.resolve( + root, + process.env['G3_VERIFY_REPORT_DIR'] ?? + `.qwen/investigations/g3-step3-implementation/packaged-${verificationMode}-${Date.now()}`, +); +if (verificationMode) mkdirSync(verificationDirectory, { recursive: true }); +const reliefMarker = 'G3_STEP3_RELIEF_TURN'; +const reliefResponse = 'G3_STEP3_RELIEF_COMPLETED'; +let originalApprovalId: string | undefined; +interface SavedApprovalOptions { + v: number; + policyRevision: string; + continuationRef: { resourceId: string }; +} +interface SavedApprovalPlan { + calls: { prepareKey: string; requestDigest: string }[]; + actionId: string; + stage: string; + batchId: string; + runtime: { + runtimeSessionId: string; + bindingId: string; + generation: string; + workspaceGeneration: string; + }; +} +let originalApprovalOptions: SavedApprovalOptions | undefined; +let originalApprovalPlan: SavedApprovalPlan | undefined; +let verifiedOriginalTerminal: string | undefined; +function saveEvidence(name: string, body: unknown): void { + if (!verificationMode) return; + writeFileSync( + path.join(verificationDirectory, name), + JSON.stringify(body, null, 2) + '\n', + ); +} + const argumentsList = process.argv.slice(2); let model = 'moonshot/kimi-k3'; let runtimeDelayMs = 0; @@ -80,6 +131,12 @@ if (freeze && (!continuationFailover || harnessOnly)) { ); } +if (verificationMode && (!continuationFailover || !harnessOnly || freeze)) { + throw new Error( + 'This isolated Step 3 probe requires --continuation-failover --harness-only', + ); +} + const durableFailover = sessionFailover || inflightFailover || continuationFailover || bigOutput; const runtimeTakeover = inflightFailover || continuationFailover; @@ -928,6 +985,8 @@ let dumpPort: number | undefined; // the post-finally throws, so an interrupt can never put a clean pass // on stdout for a run the operator explicitly stopped. let resultJson: string | undefined; +let approvalResultJson: string | undefined; +let g3SummaryJson: string | undefined; // The freeze arm's wake record reads as a pass line too, so it defers the // same way and prints after the main result, as in the base's ordering. let fencedWriterJson: string | undefined; @@ -956,6 +1015,9 @@ try { fake = await startFakeOpenAIServer(({ body }) => { const messages = Array.isArray(body['messages']) ? body['messages'] : []; const serialized = JSON.stringify(messages); + // Match the new current input before any marker from restored history. + if (serialized.includes(reliefMarker)) return { content: reliefResponse }; + if (continuationFailover && serialized.includes(continuationMarker)) { if (!serialized.includes('"role":"tool"')) { return { @@ -971,6 +1033,7 @@ try { ], }; } + if (approvalRecovery) return { content: continuationResponse }; if (!acceptReplacementContinuation) { return { contentChunks: [continuationPartial], @@ -1103,7 +1166,9 @@ try { SPRING_DATASOURCE_PASSWORD: '', SPRING_DATASOURCE_URL: `jdbc:mysql://127.0.0.1:${mysqlPort}/qwen_managed_agent?useSSL=false&allowPublicKeyRetrieval=true`, SPRING_DATASOURCE_USERNAME: 'root', - QWEN_MANAGED_AGENT_APPROVAL_MODE: 'yolo', + QWEN_MANAGED_AGENT_APPROVAL_MODE: approvalRecovery ? 'default' : 'yolo', + QWEN_MANAGED_AGENT_APPROVAL_TIMEOUT: + verificationMode === 'expiry' ? '30s' : '5m', QWEN_MANAGED_AGENT_CAPABILITY_DIGEST: capabilityDigest, QWEN_MANAGED_AGENT_HARNESS_BASE_URL: `http://127.0.0.1:${harnessPort}`, QWEN_MANAGED_AGENT_HARNESS_ENABLED: 'true', @@ -1118,7 +1183,8 @@ try { QWEN_MANAGED_AGENT_WORKSPACE_FILES_ENABLED: 'true', ...(runtimeTakeover ? { - QWEN_MANAGED_AGENT_RUNTIME_DURABLE_LOCAL_PROCESS: 'true', + QWEN_MANAGED_AGENT_RUNTIME_DURABLE_LOCAL_PROCESS: + process.platform === 'linux' ? 'true' : 'false', } : { QWEN_MANAGED_AGENT_RUNTIME_DURABLE_LOCAL_PROCESS: 'false', @@ -1340,6 +1406,132 @@ try { `Runtime execution start boundary failed; proxy=${heldStartProxy.observations().join(' | ') || 'no requests'}`, ); } + } else if (approvalRecovery) { + await waitUntil( + 'Original requested approval', + async () => { + const page = await fetchJson>>( + `${springUrl}/v1/agents/sessions/${session.id}/actions?limit=100`, + { headers: tenantHeaders(tenant) }, + ); + const requested = page.data.filter( + (action) => action.state === 'requested', + ); + if (requested.length !== 1) return false; + originalApprovalId = String(requested[0].id); + return originalApprovalId.startsWith('tool_approval_'); + }, + 120_000, + harness, + ); + const actionFilter = `tenant_id=${sqlString(tenant)} AND session_id=${sqlString(session.id)} AND action_id=${sqlString(originalApprovalId!)}`; + originalApprovalOptions = JSON.parse( + runMysql( + mysqlPort, + `SELECT options_json FROM qwen_managed_agent.managed_agent_action WHERE ${actionFilter}`, + ), + ); + if ( + originalApprovalOptions?.v !== 3 || + !originalApprovalOptions.continuationRef + ) { + throw new Error( + `New approval omitted the recovery continuation: version=${originalApprovalOptions?.v}`, + ); + } + originalApprovalPlan = JSON.parse( + runMysql( + mysqlPort, + `SELECT CONVERT(inline_bytes USING utf8mb4) FROM qwen_managed_agent.qwen_managed_session_resource WHERE tenant_id=${sqlString(tenant)} AND session_id=${sqlString(session.id)} AND resource_id=${sqlString(originalApprovalOptions.continuationRef.resourceId)}`, + ), + ); + if ( + !originalApprovalPlan || + originalApprovalPlan.calls?.length !== 1 || + originalApprovalPlan.actionId !== originalApprovalId || + originalApprovalPlan.stage !== 'approval' + ) { + throw new Error( + 'Original saved approval plan does not identify exactly one original call', + ); + } + originalRuntimeSessionId = originalApprovalPlan.runtime.runtimeSessionId; + const runtime = runMysql( + mysqlPort, + `SELECT binding_id, runtime_generation, workspace_generation, session_state FROM qwen_managed_agent.qwen_runtime_session WHERE runtime_session_id=${sqlString(originalRuntimeSessionId!)}`, + ).split('\t'); + if ( + runtime[0] !== originalApprovalPlan.runtime.bindingId || + runtime[1] !== originalApprovalPlan.runtime.generation || + runtime[2] !== originalApprovalPlan.runtime.workspaceGeneration || + runtime[3] !== 'READY' + ) { + throw new Error( + `Saved approval Runtime binding differs from the real Broker: ${runtime.join(',')}`, + ); + } + firstRuntimeHeartbeat = runtimeSessionHeartbeat( + originalRuntimeSessionId!, + ); + await waitUntil( + 'Original durable await_action checkpoint', + () => { + const id = runMysql( + mysqlPort, + `SELECT latest_checkpoint_resource_id FROM qwen_managed_agent.qwen_managed_session_journal_head WHERE tenant_id=${sqlString(tenant)} AND session_id=${sqlString(session.id)}`, + ); + if (!id) return false; + const bytes = runMysql( + mysqlPort, + `SELECT CONVERT(inline_bytes USING utf8mb4) FROM qwen_managed_agent.qwen_managed_session_resource WHERE tenant_id=${sqlString(tenant)} AND session_id=${sqlString(session.id)} AND resource_id=${sqlString(id)}`, + ); + if (!bytes) return false; + const checkpoint = JSON.parse(bytes); + return ( + checkpoint.continuation?.phase === 'await_action' && + checkpoint.approval?.requestId === originalApprovalId + ); + }, + 10_000, + harness, + ); + const privateHead = runMysql( + mysqlPort, + `SELECT storage_version, latest_checkpoint_resource_id FROM qwen_managed_agent.qwen_managed_session_journal_head WHERE tenant_id=${sqlString(tenant)} AND session_id=${sqlString(session.id)}`, + ).split('\t'); + const checkpoint = JSON.parse( + runMysql( + mysqlPort, + `SELECT CONVERT(inline_bytes USING utf8mb4) FROM qwen_managed_agent.qwen_managed_session_resource WHERE tenant_id=${sqlString(tenant)} AND session_id=${sqlString(session.id)} AND resource_id=${sqlString(privateHead[1])}`, + ), + ); + if ( + privateHead[0] !== '2' || + checkpoint.continuation?.phase !== 'await_action' || + checkpoint.approval?.requestId !== originalApprovalId + ) { + throw new Error( + 'Approval snapshot was not committed before the crash barrier', + ); + } + const executionCount = Number( + runMysql( + mysqlPort, + 'SELECT COUNT(*) FROM qwen_managed_agent.qwen_tool_execution', + ), + ); + if (executionCount !== 0 || existsSync(inflightSideEffect)) + throw new Error('Tool ran before the original approval'); + saveEvidence('original-approval.json', { + actionId: originalApprovalId, + options: originalApprovalOptions, + plan: originalApprovalPlan, + originalRuntime: runtime, + storageVersion: Number(privateHead[0]), + checkpointPhase: checkpoint.continuation.phase, + executionCount, + physicalFileExists: existsSync(inflightSideEffect), + }); } else if (continuationFailover) { await waitUntil( 'Continuation partial text', @@ -1575,7 +1767,11 @@ try { SPRING_DATASOURCE_PASSWORD: '', SPRING_DATASOURCE_URL: `jdbc:mysql://127.0.0.1:${mysqlPort}/qwen_managed_agent?useSSL=false&allowPublicKeyRetrieval=true`, SPRING_DATASOURCE_USERNAME: 'root', - QWEN_MANAGED_AGENT_APPROVAL_MODE: 'yolo', + QWEN_MANAGED_AGENT_APPROVAL_MODE: approvalRecovery + ? 'default' + : 'yolo', + QWEN_MANAGED_AGENT_APPROVAL_TIMEOUT: + verificationMode === 'expiry' ? '30s' : '5m', QWEN_MANAGED_AGENT_CAPABILITY_DIGEST: capabilityDigest, QWEN_MANAGED_AGENT_HARNESS_BASE_URL: `http://127.0.0.1:${replacementHarnessPort}`, QWEN_MANAGED_AGENT_HARNESS_ENABLED: 'true', @@ -1586,7 +1782,8 @@ try { QWEN_MANAGED_AGENT_WORKSPACE_FILES_ENABLED: 'true', ...(runtimeTakeover ? { - QWEN_MANAGED_AGENT_RUNTIME_DURABLE_LOCAL_PROCESS: 'true', + QWEN_MANAGED_AGENT_RUNTIME_DURABLE_LOCAL_PROCESS: + process.platform === 'linux' ? 'true' : 'false', } : { QWEN_MANAGED_AGENT_RUNTIME_DURABLE_LOCAL_PROCESS: 'false', @@ -1692,7 +1889,256 @@ try { replacementHarness, ); - if (inflightFailover) { + if (approvalRecovery) { + await waitUntil( + 'Original approval attached to replacement Harness', + () => { + const boot = runMysql( + mysqlPort, + `SELECT harness_boot_id FROM qwen_managed_agent.managed_agent_session WHERE ${sessionFilter}`, + ); + return boot.length > 0 && boot !== firstBootId; + }, + 120_000, + replacementHarness, + ); + const actionBeforeResponse = await fetchJson>( + `${replacementSpringUrl}/v1/agents/sessions/${session.id}/actions/${originalApprovalId}`, + { headers: tenantHeaders(tenant) }, + ); + if ( + actionBeforeResponse.id !== originalApprovalId || + (verificationMode !== 'expiry' && + actionBeforeResponse.state !== 'requested') + ) { + throw new Error( + 'Replacement did not preserve the original requested Action', + ); + } + let responseOperation: Record | undefined; + if (verificationMode === 'allow' || verificationMode === 'deny') { + const response = await fetch( + `${replacementSpringUrl}/v1/agents/sessions/${session.id}/actions/${originalApprovalId}/responses`, + { + method: 'POST', + headers: { + 'content-type': 'application/json', + 'idempotency-key': `g3-answer-${originalApprovalId}`, + ...tenantHeaders(tenant), + }, + body: JSON.stringify({ + kind: 'permission', + input_revision: 1, + policy_revision: originalApprovalOptions!.policyRevision, + option_id: verificationMode, + }), + }, + ); + if (response.status !== 202) + throw new Error( + `Original Action response returned ${response.status}: ${await response.text()}`, + ); + responseOperation = await response.json(); + } else if (verificationMode === 'cancel') { + const response = await fetch( + `${replacementSpringUrl}/v1/agents/sessions/${session.id}/events`, + { + method: 'POST', + headers: { + 'content-type': 'application/json', + 'idempotency-key': `g3-cancel-${originalApprovalId}`, + ...tenantHeaders(tenant), + }, + body: JSON.stringify({ + type: 'agent.session.cancel', + turn_id: actionBeforeResponse.turn_id, + }), + }, + ); + if (response.status !== 202) + throw new Error( + `Original Turn cancel returned ${response.status}: ${await response.text()}`, + ); + } + const recovered = await waitForTerminal( + replacementSpringUrl, + tenant, + session.id, + 0, + replacementSpring, + 120_000, + ); + const terminals = recovered.events.filter((event) => event.terminal); + const expectedTerminal = ['allow', 'deny'].includes(verificationMode) + ? 'turn.completed' + : 'turn.cancelled'; + if (terminals.length !== 1 || terminals[0].type !== expectedTerminal) { + throw new Error( + `Original approval terminal differs: expected=${expectedTerminal} actual=${JSON.stringify(terminals)}`, + ); + } + verifiedOriginalTerminal = terminals[0].type; + const rows = runMysql( + mysqlPort, + `SELECT execution_call_id, idempotency_key, binding_id, runtime_generation, runtime_session_id, execution_state, dispatch_generation, request_digest FROM qwen_managed_agent.qwen_tool_execution ORDER BY execution_call_id`, + ); + const executions = rows + ? rows.split('\n').map((row) => row.split('\t')) + : []; + const expectedExecutions = verificationMode === 'allow' ? 1 : 0; + if (executions.length !== expectedExecutions) + throw new Error( + `Approval execution count ${executions.length} differs from ${expectedExecutions}`, + ); + if (executions.length === 1) { + const execution = executions[0]; + const plan = originalApprovalPlan!; + if ( + execution[1] !== plan.calls[0].prepareKey || + execution[2] !== plan.runtime.bindingId || + execution[3] !== plan.runtime.generation || + execution[4] !== plan.runtime.runtimeSessionId || + execution[5] !== 'SETTLED' || + execution[6] !== '1' || + execution[7] !== plan.calls[0].requestDigest + ) { + throw new Error( + `Replacement changed the original prepared execution identity: ${execution.join(',')}`, + ); + } + } + const bytes = existsSync(inflightSideEffect) + ? readFileSync(inflightSideEffect, 'utf8') + : ''; + if ( + verificationMode === 'allow' + ? bytes !== inflightSideEffectContent + : existsSync(inflightSideEffect) + ) { + throw new Error( + `Approval physical file assertion failed for ${verificationMode}`, + ); + } + const actionRows = runMysql( + mysqlPort, + `SELECT action_id, state FROM qwen_managed_agent.managed_agent_action WHERE ${sessionFilter}`, + ); + const expectedActionState = + verificationMode === 'expiry' + ? 'expired' + : verificationMode === 'cancel' + ? 'cancelled' + : 'decided'; + if (actionRows !== `${originalApprovalId}\t${expectedActionState}`) + throw new Error( + `Original approval identity/state differs: ${actionRows}`, + ); + const requestBodies = (fake?.requests ?? []).filter((request) => + JSON.stringify(request.body.messages).includes(continuationMarker), + ); + const initialRequests = requestBodies.filter( + (request) => + !JSON.stringify(request.body.messages).includes('"role":"tool"'), + ); + const modelAfterResults = requestBodies.filter((request) => + JSON.stringify(request.body.messages).includes('"role":"tool"'), + ); + const expectedModelsAfterResults = ['allow', 'deny'].includes( + verificationMode, + ) + ? 1 + : 0; + if ( + initialRequests.length !== 1 || + modelAfterResults.length !== expectedModelsAfterResults + ) + throw new Error( + `Approval replayed inference or skipped result inference: ${initialRequests.length}+${modelAfterResults.length}`, + ); + if (responseOperation) { + await waitUntil( + 'Approval response command completion', + async () => { + const operation = await fetchJson>( + `${replacementSpringUrl}/v1/agents/sessions/${session.id}/operations/${responseOperation!.id}`, + { headers: tenantHeaders(tenant) }, + ); + if (operation.status === 'failed') + throw new Error( + `Approval response operation failed: ${JSON.stringify(operation)}`, + ); + return operation.status === 'completed'; + }, + 30_000, + replacementSpring, + ); + } + await waitUntil( + 'Original Runtime cleanup confirmed', + () => { + const state = runMysql( + mysqlPort, + `SELECT session_state FROM qwen_managed_agent.qwen_runtime_session WHERE runtime_session_id=${sqlString(originalRuntimeSessionId!)}`, + ); + const confirmed = runMysql( + mysqlPort, + `SELECT COUNT(*) FROM qwen_managed_agent.qwen_managed_session_journal_tx WHERE ${sessionFilter} AND command_id=${sqlString(`hosted-cleanup:${originalRuntimeSessionId}:${originalRuntimeSessionId}:confirmed`)}`, + ); + return state === 'RELEASED' && confirmed === '1'; + }, + 30_000, + replacementHarness, + ); + const runtimeAfter = runMysql( + mysqlPort, + `SELECT binding_id, runtime_generation, workspace_generation, session_state FROM qwen_managed_agent.qwen_runtime_session WHERE runtime_session_id=${sqlString(originalRuntimeSessionId!)}`, + ).split('\t'); + const cleanupConfirmed = Number( + runMysql( + mysqlPort, + `SELECT COUNT(*) FROM qwen_managed_agent.qwen_managed_session_journal_tx WHERE ${sessionFilter} AND command_id=${sqlString(`hosted-cleanup:${originalRuntimeSessionId}:${originalRuntimeSessionId}:confirmed`)}`, + ), + ); + if ( + runtimeAfter[0] !== originalApprovalPlan!.runtime.bindingId || + runtimeAfter[1] !== originalApprovalPlan!.runtime.generation || + runtimeAfter[2] !== originalApprovalPlan!.runtime.workspaceGeneration || + runtimeAfter[3] !== 'RELEASED' || + cleanupConfirmed !== 1 + ) + throw new Error( + `Original cleanup did not release the original Runtime: state=${runtimeAfter.join(',')} confirmed=${cleanupConfirmed}`, + ); + saveEvidence('approval-result.json', { + mode: verificationMode, + sessionId: session.id, + actionId: originalApprovalId, + replacementActionBeforeResponse: actionBeforeResponse, + expectedTerminal, + terminalCount: terminals.length, + executions, + physicalFileExists: existsSync(inflightSideEffect), + physicalFileBytes: bytes, + modelRequests: initialRequests.length + modelAfterResults.length, + initialRequests: initialRequests.length, + modelAfterResults: modelAfterResults.length, + runtimeAfter, + cleanupConfirmed, + }); + approvalResultJson = JSON.stringify( + { + mode: verificationMode, + originalAction: originalApprovalId, + originalBatch: originalApprovalPlan!.batchId, + originalBinding: originalApprovalPlan!.runtime.bindingId, + executions: executions.length, + terminal: expectedTerminal, + cleanupConfirmed, + }, + null, + 2, + ); + } else if (inflightFailover) { const recoveredTurn = await waitForTerminal( replacementSpringUrl, tenant, @@ -1847,6 +2293,30 @@ try { const requests = (fake?.requests ?? []).filter(({ body }) => JSON.stringify(body['messages']).includes(continuationMarker), ); + const originalAndReplacementRequests = requests.filter(({ body }) => + JSON.stringify(body['messages']).includes('"role":"tool"'), + ); + if ( + originalAndReplacementRequests.length === 2 && + !isDeepStrictEqual( + originalAndReplacementRequests[0].body, + originalAndReplacementRequests[1].body, + ) + ) { + throw new Error( + 'The replacement model request does not equal the original effective request', + ); + } + saveEvidence('model-request-parity.json', { + requests: originalAndReplacementRequests.map((request) => request.body), + equal: + originalAndReplacementRequests.length === 2 && + isDeepStrictEqual( + originalAndReplacementRequests[0].body, + originalAndReplacementRequests[1].body, + ), + }); + const initialModelRequests = requests.filter( ({ body }) => !JSON.stringify(body['messages']).includes('"role":"tool"'), @@ -2164,6 +2634,161 @@ try { 2, ); } + if (verificationMode) { + // These assertions run after every A/B primary gate, never in place of it. + await waitUntil( + 'Runtime cleanup before next Turn', + () => { + const state = runMysql( + mysqlPort, + `SELECT session_state FROM qwen_managed_agent.qwen_runtime_session WHERE runtime_session_id=${sqlString(originalRuntimeSessionId!)}`, + ); + const confirmed = runMysql( + mysqlPort, + `SELECT COUNT(*) FROM qwen_managed_agent.qwen_managed_session_journal_tx WHERE ${sessionFilter} AND command_id=${sqlString(`hosted-cleanup:${originalRuntimeSessionId}:${originalRuntimeSessionId}:confirmed`)}`, + ); + return state === 'RELEASED' && confirmed === '1'; + }, + 30_000, + replacementHarness, + ); + const originalPage = await fetchJson>( + `${replacementSpringUrl}/v1/agents/sessions/${session.id}/events?after=0&limit=100`, + { headers: tenantHeaders(tenant) }, + ); + if (originalPage.data.filter((event) => event.terminal).length !== 1) + throw new Error( + 'Original Turn must have exactly one public terminal before relief', + ); + const originalCursor = Math.max( + 0, + ...originalPage.data.map((event) => event.sequence), + ); + const relief = await fetch( + `${replacementSpringUrl}/v1/agents/sessions/${session.id}/events`, + { + method: 'POST', + headers: { + 'content-type': 'application/json', + 'idempotency-key': `g3-relief-${session.id}`, + ...tenantHeaders(tenant), + }, + body: JSON.stringify({ + type: 'agent.session.input.message', + input: [ + { + type: 'text', + text: `${reliefMarker}. Reply exactly ${reliefResponse}. Use no tools.`, + }, + ], + }), + }, + ); + if (relief.status !== 202) + throw new Error( + `Relief Turn admission returned ${relief.status}: ${await relief.text()}`, + ); + const reliefTurn = await waitForTerminal( + replacementSpringUrl, + tenant, + session.id, + originalCursor, + replacementSpring, + 60_000, + ); + const reliefTerminals = reliefTurn.events.filter( + (event) => event.terminal, + ); + const reliefText = reliefTurn.events + .filter((event) => event.type === 'item.output_text.delta') + .map(eventText) + .join(''); + if ( + reliefTerminals.length !== 1 || + reliefTerminals[0].type !== 'turn.completed' || + reliefText !== reliefResponse + ) + throw new Error( + `Relief Turn did not finish: terminals=${JSON.stringify(reliefTerminals)} text=${JSON.stringify(reliefText)}`, + ); + const reliefRequests = (fake?.requests ?? []).filter((request) => + JSON.stringify(request.body.messages).includes(reliefMarker), + ); + if (reliefRequests.length !== 1) + throw new Error( + `Relief Turn inference count differs: ${reliefRequests.length}`, + ); + const finalTerminalCount = Number( + runMysql( + mysqlPort, + `SELECT COUNT(*) FROM qwen_managed_agent.managed_agent_event WHERE ${sessionFilter} AND terminal=TRUE`, + ), + ); + if (finalTerminalCount !== 2) + throw new Error( + `Expected one terminal per original/relief Turn, observed ${finalTerminalCount}`, + ); + saveEvidence('post-turn-events.json', { + cursorBeforeRelief: originalCursor, + original: originalPage.data, + relief: reliefTurn.events, + sqlTerminalCount: finalTerminalCount, + }); + const close = await fetch( + `${replacementSpringUrl}/v1/agents/sessions/${session.id}/close`, + { + method: 'POST', + headers: { + 'idempotency-key': `g3-close-${session.id}`, + ...tenantHeaders(tenant), + }, + }, + ); + if (close.status !== 202) + throw new Error( + `Close admission returned ${close.status}: ${await close.text()}`, + ); + const closeOperation = (await close.json()) as Record; + await waitUntil( + 'Reliable close operation completion', + async () => { + const operation = await fetchJson>( + `${replacementSpringUrl}/v1/agents/sessions/${session.id}/operations/${closeOperation.id}`, + { headers: tenantHeaders(tenant) }, + ); + if (operation.status === 'failed') + throw new Error( + `Close operation failed: ${JSON.stringify(operation)}`, + ); + return operation.status === 'completed'; + }, + 60_000, + replacementSpring, + ); + const closedSession = await fetchJson>( + `${replacementSpringUrl}/v1/agents/sessions/${session.id}`, + { headers: tenantHeaders(tenant) }, + ); + if (closedSession.status !== 'closed') + throw new Error( + `Reliable close completed with Session status ${closedSession.status}`, + ); + const summary = { + mode: verificationMode, + sessionId: session.id, + originalRuntimeSessionId, + originalActionId: originalApprovalId, + originalTerminal: verifiedOriginalTerminal ?? 'turn.completed', + nextTurnCompleted: true, + nextTurnRequests: reliefRequests.length, + terminalCount: finalTerminalCount, + closeAdmission: close.status, + closeCompleted: true, + oldHarnessHomeDeleted: !existsSync(harnessHome), + }; + saveEvidence('summary.json', summary); + g3SummaryJson = JSON.stringify(summary, null, 2); + } } else { const idempotencyKey = `create-${Date.now()}`; const prompt = [ @@ -2413,6 +3038,8 @@ if (failure) throw failure; // payload stays a variable until both guards have run, so the decision // precedes the record. if (receivedSignal) throw new Error(`Interrupted by ${receivedSignal}`); +if (approvalResultJson !== undefined) console.log(approvalResultJson); +if (g3SummaryJson !== undefined) console.log(g3SummaryJson); if (resultJson !== undefined) { console.log(resultJson); if (fencedWriterJson !== undefined) console.log(fencedWriterJson); diff --git a/scripts/tests/managed-agent-server-e2e.test.js b/scripts/tests/managed-agent-server-e2e.test.js index 26f3ce8386..58d588cf6b 100644 --- a/scripts/tests/managed-agent-server-e2e.test.js +++ b/scripts/tests/managed-agent-server-e2e.test.js @@ -265,9 +265,9 @@ describe('managed-agent-server e2e runner', () => { ).toHaveLength(2); expect( source.match( - /\.\.\.\(runtimeTakeover\s*\?\s*\{\s*QWEN_MANAGED_AGENT_RUNTIME_DURABLE_LOCAL_PROCESS:\s*'true'/g, + /\.\.\.\(runtimeTakeover\s*\?\s*\{\s*QWEN_MANAGED_AGENT_RUNTIME_DURABLE_LOCAL_PROCESS:\s*process\.platform === 'linux' \? 'true' : 'false'/g, ), - 'both durable local process branches must follow runtimeTakeover', + 'both durable local process branches must follow runtimeTakeover, Linux-gated', ).toHaveLength(2); }); @@ -811,6 +811,10 @@ describe('managed-agent-server e2e runner', () => { // pass line on stdout too, so it defers the same way — no eager // console.log of a JSON payload remains inside the try at all. expect(source.match(/resultJson = JSON\.stringify\(/g)).toHaveLength(4); + expect( + source.match(/approvalResultJson = JSON\.stringify\(/g), + ).toHaveLength(1); + expect(source.match(/g3SummaryJson = JSON\.stringify\(/g)).toHaveLength(1); expect(source.match(/fencedWriterJson = JSON\.stringify\(/g)).toHaveLength( 1, ); @@ -819,6 +823,8 @@ describe('managed-agent-server e2e runner', () => { // module-scope binding the deferred print reads is pinned by text: // dropped, the print below throws ReferenceError on the success path. expect(source).toMatch(/^let resultJson: string \| undefined;$/m); + expect(source).toMatch(/^let approvalResultJson: string \| undefined;$/m); + expect(source).toMatch(/^let g3SummaryJson: string \| undefined;$/m); expect(source).toMatch(/^let fencedWriterJson: string \| undefined;$/m); // The behavioural tests inject the signal through this harness's // synthesized setter, so the production link — the runner's own @@ -846,6 +852,11 @@ describe('managed-agent-server e2e runner', () => { const print = source.indexOf('console.log(resultJson)'); expect(failureThrow).toBeGreaterThan(-1); expect(signalThrow).toBeGreaterThan(failureThrow); + const approvalPrint = source.indexOf('console.log(approvalResultJson)'); + const summaryPrint = source.indexOf('console.log(g3SummaryJson)'); + expect(approvalPrint).toBeGreaterThan(signalThrow); + expect(summaryPrint).toBeGreaterThan(approvalPrint); + expect(summaryPrint).toBeLessThan(print); expect( print, 'the success payload must print only after both exit guards',