mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-10-23 07:19:52 +00:00
test(managed-agent): align FG8 gate choreography with production truth (#14163)
Address the review round's execution findings on PR #14334 — each one verified against the production call chain before fixing: - dispatch-reply: the broker proxy consumes the :start reply before destroying the socket, so on a cgroup host the process is live and its producer parks on the run-gate; release it after identity recovery (gated on bg-started.txt); also accept the terminal not-started form in the capability-live wait so macOS keeps proving the not_started_proven branch. - capture-commit-reply: SEGMENTS_PER_PAGE is 512, so a 24-block producer can never reach one page publish pre-gate; wait for progress, release the gate, then witness the finish-time drop. - FG8b manifest trigger: the capture publishes an initial empty manifest during prepare, ahead of spawn; the arm now targets only the settle manifest (executionStatus 'success'), leaving the prepare-manifest to commit. - FG8b publication assertions (per the reviewer's live-capture evidence): zero manifest-copy claims replaced with the honest outcome — every committed manifest's page refs resolve, zero page rows for page-envelope, settle manifest never commits for manifest-commit; the run settles naturally exited/0 with stopRequested=false — a store refusal never cancels the process, and driver stays on the natural-end branch (recovered=true). - FG8d shell cancel cases: children/operations encodes the child-agent cancel family and rejects shell records; the only stop surface a shell child_run answers is the drain release. The three cases now drive drain:true; cancel-status-unavailable adds a positive hold witness (at least one parked maintenance request) before the drain; cancel-reply-lost still drops the stopRequested commit's reply after it committed. childCancel helper removed. Locally green: invocation-commit, process-ledger-commit, dispatch-reply, cancel-before-start. Design doc (EN + zh-CN) updated to match.
This commit is contained in:
Shaojin Wen
parent
4ab97f64b9
commit
fee23d2d3c
4 changed files with 134 additions and 133 deletions
|
|
@ -143,33 +143,34 @@ case first asserts its trigger really fired through the `fg8b_fire`
|
|||
count — a gate that silently never fired goes red on the fire count,
|
||||
not on an outcome.
|
||||
|
||||
| Case | Injection | Expect (JDBC / evidence) |
|
||||
| ----------------------- | ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `invocation-commit` | `BEFORE INSERT ON ma_tool_execution` (`NEW.harness_session_id = <session>` AND call-id has no `:process` suffix) | Zero OS effects: no invocation row, no `:process` row, no capture registration (`ma_output_capture`/capture resources absent), no physical process ever (case directory has no `bg-started.txt`); the session turn settles without fabricating a background start — the blocked/blocked-settled form observed is documented gate-side |
|
||||
| `process-ledger-commit` | Same table, call-id suffixed `:process` | Invocation exists, `:process` row absent; the run blocks without a second admission (exactly ≤1 invocation row, no second dispatch in the broker operation log); no physical process |
|
||||
| `manifest-commit` | `BEFORE INSERT ON ma_resource`, `kind = 'managed-tool-result-manifest'`, permanent | A failed manifest commit blocks the run without re-executing it: one `:process` row, one start attempt, zero manifest revisions ever commit; the driver drains the blocked run and the record settles `cancelled`/`stop_requested` with `stopRequested: true`, never `exited`; fire count ≥ 1 |
|
||||
| `page-envelope-commit` | Same table, `kind = 'managed-tool-result-page'`, permanent | The read chain never resolves to a manifest whose page envelopes were not persisted: every page-envelope commit is refused, so zero page rows persist and the only honest chain carries no manifest at all (a committed manifest naming a refused envelope goes red on the count); blocked run drained as `manifest-commit`; fire count ≥ 1 |
|
||||
| Case | Injection | Expect (JDBC / evidence) |
|
||||
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `invocation-commit` | `BEFORE INSERT ON ma_tool_execution` (`NEW.harness_session_id = <session>` AND call-id has no `:process` suffix) | Zero OS effects: no invocation row, no `:process` row, no capture registration (`ma_output_capture`/capture resources absent), no physical process ever (case directory has no `bg-started.txt`); the session turn settles without fabricating a background start — the blocked/blocked-settled form observed is documented gate-side |
|
||||
| `process-ledger-commit` | Same table, call-id suffixed `:process` | Invocation exists, `:process` row absent; the run blocks without a second admission (exactly ≤1 invocation row, no second dispatch in the broker operation log); no physical process |
|
||||
| `manifest-commit` | `BEFORE INSERT ON ma_resource`, `kind = 'managed-tool-result-manifest'` carrying `executionStatus: 'success'` (the capture's initial empty manifest must land, or the process never starts) | One `:process` row, one start attempt, and the settle-manifest commit never persists — yet the refusal stops nothing: the producer exits 0 and the record settles from real exit evidence (`settled`/`exited`/exit 0, `stopRequested: false`); fire count ≥ 1 |
|
||||
| `page-envelope-commit` | Same table, `kind = 'managed-tool-result-page'`, permanent | No manifest ever names an unpersisted page: every page-envelope commit is refused (zero page rows) and every committed manifest's page references resolve — only the initial empty manifest exists; same natural settle as `manifest-commit`; fire count ≥ 1 |
|
||||
|
||||
### FG8d — cancellation and drain (6 cases)
|
||||
|
||||
Production cancel surfaces the gate drives: the turn-level daemon cancel
|
||||
(`POST /session/:id/cancel`, which aborts the live turn and drives the
|
||||
turn's `broker.cancel` batch), the control-plane child cancel
|
||||
(`POST /session/:id/children/operations` with `"cancel"`, which commits
|
||||
`stopRequested: true` through `requestStop`), and the broker `:release`
|
||||
maintenance route issued by the driver under the Harness's captured
|
||||
identity. The settle projection under gate is `hosted-shell-publisher.ts`:
|
||||
a sink-side stop marker leads to `requestStop` then `settleStopRequested`,
|
||||
the contract order, never `exited`.
|
||||
turn's `broker.cancel` batch), and the broker `:release` maintenance
|
||||
route with `drain: true`, issued by the driver under the Harness's
|
||||
captured identity — the only stop surface a shell `child_run` answers
|
||||
(the child-agent `children/operations` cancel family rejects shell
|
||||
records outright, so this letter never drives it). The settle projection
|
||||
under gate is `hosted-shell-publisher.ts`: a sink-side stop marker leads
|
||||
to `requestStop` then `settleStopRequested`, the contract order, never
|
||||
`exited`.
|
||||
|
||||
| Case | Injection | Expect (JDBC / evidence) |
|
||||
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `cancel-before-start` | Hold the `:start` request at the broker proxy _before_ forwarding; once the broker operation log shows the tool was admitted, issue the daemon cancel; then destroy the held request without forwarding | Zero OS effects: no `:process` row, no `bg-started.txt`, no capture rows; the invocation row settles by the broker's cancel path (`CANCEL_REQUESTED` → terminal); the child_run record is terminal, never `exited` |
|
||||
| `cancel-during-capture` | Mid-capture (≥3 full segments committed), issue the child cancel (`children/operations`) | `stopRequested: true` is committed on the record; the run is _not_ settled by the cancel itself (record still running while the process lives); the subsequent physical termination settles it `cancelled` / `stop_requested`, never `exited` |
|
||||
| `cancel-status-unavailable` | Same child cancel, issued while the broker's maintenance route to the worker is held at the proxy (status unreachable) | The cancel commit lands (`stopRequested: true`); no terminal record is fabricated during the outage (record stays non-terminal, no settle truth appears in JDBC); once the hold is released the evidence resolves the run honestly |
|
||||
| `cancel-reply-lost` | Lose one `commitChildRunRecord` store-commit reply after the cancel's `stopRequested` write committed; re-drive the child cancel | The re-drive answers the committed state unchanged: exactly one record revision carries `stopRequested: true` (no second revision, no duplicate record row); drop count = 1 |
|
||||
| `routine-release-refusal` | With the run live, driver issues broker `:release` without `drain` | The answer is 409 with `code: 'runtime_session_held'`; nothing is stopped — the producer keeps progressing (progress/marks keep advancing) and no stop kind reached the worker (no `shell-terminate`/`monitor-stop` forwarded); the `:process` row stays `PREPARED` |
|
||||
| `drain-release` | Then driver issues broker `:release` with `drain: true` (same run) | The drain arm issues the stop (`shell-terminate` observed on the maintenance wire); the run settles `cancelled` / `stop_requested` with `stopRequested: true`, never `exited`; the release settles with the same identity the routine arm was refused on |
|
||||
| Case | Injection | Expect (JDBC / evidence) |
|
||||
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `cancel-before-start` | Hold the `:start` request at the broker proxy _before_ forwarding; once the broker operation log shows the tool was admitted, issue the daemon cancel; then destroy the held request without forwarding | Zero OS effects: no `:process` row, no `bg-started.txt`, no capture rows; the invocation row settles by the broker's cancel path (`CANCEL_REQUESTED` → terminal); the child_run record is terminal, never `exited` |
|
||||
| `cancel-during-capture` | Mid-capture (≥3 full segments committed), arm the drain `:release` — the only stop surface a shell `child_run` answers | The drain carries the stop command: `stopRequested: true` commits with the stop evidence and the run settles `cancelled` / `stop_requested`, never `exited`; one `:process` row |
|
||||
| `cancel-status-unavailable` | Hold every maintenance/status request at the broker proxy, then arm the drain during that outage | The hold is positively witnessed (at least one request parked); no terminal record is fabricated during the outage — the record keeps running, JDBC shows no settle truth; the drain still answers and, once released, the evidence settles the run `cancelled` / `stop_requested` |
|
||||
| `cancel-reply-lost` | Lose the `commitChildRunRecord` store-commit reply to the drain-driven `stopRequested` write, after it committed | The committed record carries `stopRequested: true` anyway; exactly one committed stop revision exists (never a second revision nor a duplicate record row); drop count = 1 |
|
||||
| `routine-release-refusal` | With the run live, driver issues broker `:release` without `drain` | The answer is 409 with `code: 'runtime_session_held'`; nothing is stopped — the producer keeps progressing (progress/marks keep advancing) and no stop kind reached the worker (no `shell-terminate`/`monitor-stop` forwarded); the `:process` row stays `PREPARED` |
|
||||
| `drain-release` | Then driver issues broker `:release` with `drain: true` (same run) | The drain arm issues the stop (`shell-terminate` observed on the maintenance wire); the run settles `cancelled` / `stop_requested` with `stopRequested: true`, never `exited`; the release settles with the same identity the routine arm was refused on |
|
||||
|
||||
## Driver design
|
||||
|
||||
|
|
|
|||
|
|
@ -45,23 +45,23 @@ driver 让每个 case 的 Session 同时穿过**两个**代理（broker 调用
|
|||
|
||||
触发器 DDL 按 FG7B 形状，名字 `fg8b_<hex>`，message 文本 `FG8B_<CASE>`（`INVOCATION_COMMIT` / `PROCESS_LEDGER_COMMIT` / `MANIFEST_COMMIT` / `PAGE_ENVELOPE_COMMIT`），每次点火另写一行 `fg8b_fire`。触发器按 case tenant 与 case session 双重 scope，对该案余下时间永久拒绝其提交类型，在该 case 的 driver 工作开始前安装、在 `finally` 里拆除。每个 case 先经 `fg8b_fire` 计数断言自己的触发器真的点过火——一门悄悄没点火的 gate 会红在点火计数上，而不是红在某个结局上。
|
||||
|
||||
| Case | 注入 | 期望（JDBC / 证据） |
|
||||
| ----------------------- | ---------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `invocation-commit` | `BEFORE INSERT ON ma_tool_execution`（`NEW.harness_session_id = <session>` 且 call-id 无 `:process` 后缀） | 零 OS 副作用：无调用行、无 `:process` 行、无 capture 注册（`ma_output_capture`/capture 资源缺席）、从未有物理进程（case 目录无 `bg-started.txt`）；会话轮不编造后台启动就结束——实测到的 blocked/阻塞结算形态在 gate 侧记录在案 |
|
||||
| `process-ledger-commit` | 同表，call-id 带 `:process` 后缀 | 调用行存在，`:process` 行缺席；run 阻塞且不发起二次准入（至多 1 条调用行，broker 操作日志无第二次 dispatch）；无物理进程 |
|
||||
| `manifest-commit` | `BEFORE INSERT ON ma_resource`，`kind = 'managed-tool-result-manifest'`，永久 | manifest 提交失败阻塞该 run 而不重执行：一条 `:process` 行、一次 start 尝试、零条 manifest 修订落库；driver 以 drain 收束该阻塞 run，记录结算为 `cancelled`/`stop_requested` 且 `stopRequested: true`，绝不 `exited`；点火计数 ≥ 1 |
|
||||
| `page-envelope-commit` | 同表，`kind = 'managed-tool-result-page'`，永久 | 读链绝不解析到一个其 page envelope 未被持久化的 manifest：每次 page-envelope 提交都被拒，零条 page 行落库，唯一诚实的链不含任何 manifest（若有 manifest 命名了被拒 envelope 则红在计数上）；阻塞 run 按 `manifest-commit` 同形 drain；点火计数 ≥ 1 |
|
||||
| Case | 注入 | 期望（JDBC / 证据） |
|
||||
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `invocation-commit` | `BEFORE INSERT ON ma_tool_execution`（`NEW.harness_session_id = <session>` 且 call-id 无 `:process` 后缀） | 零 OS 副作用：无调用行、无 `:process` 行、无 capture 注册（`ma_output_capture`/capture 资源缺席）、从未有物理进程（case 目录无 `bg-started.txt`）；会话轮不编造后台启动就结束——实测到的 blocked/阻塞结算形态在 gate 侧记录在案 |
|
||||
| `process-ledger-commit` | 同表，call-id 带 `:process` 后缀 | 调用行存在，`:process` 行缺席；run 阻塞且不发起二次准入（至多 1 条调用行，broker 操作日志无第二次 dispatch）；无物理进程 |
|
||||
| `manifest-commit` | `BEFORE INSERT ON ma_resource`，`kind = 'managed-tool-result-manifest'` 且携带 `executionStatus: 'success'`（capture 的初始空 manifest 必须落库，否则进程不会启动） | 一条 `:process` 行、一次 start 尝试，结算 manifest 的提交绝不落库——但拒绝并不停止 run：producer 以 0 退出，记录按真实退出证据结算（`settled`/`exited`/exit 0、`stopRequested: false`）；点火计数 ≥ 1 |
|
||||
| `page-envelope-commit` | 同表，`kind = 'managed-tool-result-page'`，永久 | manifest 永不命名未持久化的 page：每次 page-envelope 提交都被拒（零条 page 行），且每个已提交 manifest 的 page 引用都可解析——落库的只有初始空 manifest；与 `manifest-commit` 同形自然结算；点火计数 ≥ 1 |
|
||||
|
||||
### FG8d — 取消与 drain（6 案）
|
||||
|
||||
门禁驱动的生产取消面：turn 级 daemon cancel（`POST /session/:id/cancel`，中止在飞 turn 并驱动该 turn 的 `broker.cancel` 批量）、控制面 child cancel（`POST /session/:id/children/operations`，kind `"cancel"`，经 `requestStop` 提交 `stopRequested: true`），以及 driver 以捕获到的 Harness 身份直发的 broker `:release` 维护路由。被 gate 的结算投影是 `hosted-shell-publisher.ts`：sink 侧停标先到 `requestStop` 再到 `settleStopRequested`，契约次序，绝不 `exited`。
|
||||
门禁驱动的生产取消面：turn 级 daemon cancel（`POST /session/:id/cancel`，中止在飞 turn 并驱动该 turn 的 `broker.cancel` 批量），以及 driver 以捕获到的 Harness 身份直发的 broker `:release` 维护路由（带 `drain: true`，shell `child_run` 唯一应答的停止面——child-agent 的 `children/operations` cancel 家族直接拒绝 shell 记录，本字母不走它）。被 gate 的结算投影是 `hosted-shell-publisher.ts`：sink 侧停标先到 `requestStop` 再到 `settleStopRequested`，契约次序，绝不 `exited`。
|
||||
|
||||
| Case | 注入 | 期望（JDBC / 证据） |
|
||||
| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `cancel-before-start` | 在 broker 代理上把 `:start` 请求**转发前** hold；broker 操作日志出现工具准入后，发 daemon cancel；随后毁掉被 hold 的请求，永不转发 | 零 OS 副作用：无 `:process` 行、无 `bg-started.txt`、无 capture 行；调用行经 broker 的 cancel 路径结算（`CANCEL_REQUESTED` → 终态）；child_run 记录终态，绝不 `exited` |
|
||||
| `cancel-during-capture` | capture 进行中（≥3 个完整段已提交），发 child cancel（`children/operations`） | 记录提交 `stopRequested: true`；run **不**因 cancel 本身结算（进程还活着时记录仍在 running）；随后的物理终止把它结算为 `cancelled` / `stop_requested`，绝不 `exited` |
|
||||
| `cancel-status-unavailable` | 同上 child cancel，但在代理上 hold 住 broker 到 worker 的维护路由（status 不可达）期间发出 | cancel 提交落库（`stopRequested: true`）； outage 期间不编造终态记录（记录保持非终态，JDBC 里不出现结算真相）；hold 释放后由证据把 run 诚实结算 |
|
||||
| `cancel-reply-lost` | 在 cancel 的 `stopRequested` 写入已提交后，丢掉一次 `commitChildRunRecord` store-commit 应答；重驱 child cancel | 重驱回答已提交的状态且不变：恰好一个 record 版本带 `stopRequested: true`（无第二版本、无重复记录行）；drop 计数 = 1 |
|
||||
| `cancel-during-capture` | capture 进行中（≥3 个完整段已提交），武装 drain `:release`——shell `child_run` 唯一应答的停止面 | drain 携带停止命令：`stopRequested: true` 随停止证据落库，run 结算为 `cancelled` / `stop_requested`，绝不 `exited`；一条 `:process` 行 |
|
||||
| `cancel-status-unavailable` | 在 broker 代理上 hold 全部维护/状态请求，outage 期间武装 drain | hold 有正向见证（≥1 条请求被泊住）；outage 期间不编造终态——记录保持 running，JDBC 不出现结算真相；drain 照常应答，释放后证据把 run 结算为 `cancelled` / `stop_requested` |
|
||||
| `cancel-reply-lost` | 在 drain 驱动的 `stopRequested` 写入已提交后，丢掉一次 `commitChildRunRecord` store-commit 应答 | 已提交的记录照常携带 `stopRequested: true`；恰好一条已提交 stop 修订（绝无第二修订或重复记录行）；drop 计数 = 1 |
|
||||
| `routine-release-refusal` | run 活着时，driver 发不带 `drain` 的 broker `:release` | 应答 409 且 `code: 'runtime_session_held'`；什么都不停——producer 继续推进（progress/marks 持续前进）且没有 stop kind 到达 worker（维护 wire 上无 `shell-terminate`/`monitor-stop` 转发）；`:process` 行保持 `PREPARED` |
|
||||
| `drain-release` | 随后 driver 发带 `drain: true` 的 broker `:release`（同一 run） | drain 臂发出停止（维护 wire 上见到 `shell-terminate`）；run 结算为 `cancelled` / `stop_requested` 且 `stopRequested: true`，绝不 `exited`；release 以 routine 臂被拒的同一身份结算 |
|
||||
|
||||
|
|
|
|||
|
|
@ -1234,15 +1234,26 @@ async function runReplyLossCase() {
|
|||
return rows.some(
|
||||
(row) =>
|
||||
row.execution_call_id.endsWith(':process') &&
|
||||
row.execution_state === 'PREPARED',
|
||||
(row.execution_state === 'PREPARED' ||
|
||||
// A cgroup-less runtime settles the row terminal not-started
|
||||
// in one beat post-#14277 — no PREPARED flicker to catch.
|
||||
(row.execution_state === 'SETTLED' &&
|
||||
row.execution_status === 'not_started')),
|
||||
);
|
||||
}, 120_000);
|
||||
step('capability live');
|
||||
if (name === 'dispatch-reply') {
|
||||
await waitUntil(() => brokerDropped >= 1, 60_000);
|
||||
step('dispatch reply dropped once');
|
||||
// The drop lands before the physical start, so no producer ever
|
||||
// runs to open the gate — the case settles as a proven not-start.
|
||||
// The proxy consumed the upstream :start reply before destroying
|
||||
// the client socket, so on a cgroup host the physical process is
|
||||
// already running and its producer is parked on the run-gate —
|
||||
// the run can never settle until the gate opens. A cgroup-less
|
||||
// runtime (this box) never starts, so there is nothing to open.
|
||||
if (bgStartedFile()) {
|
||||
await releaseFifo(current.directory, 'run-gate');
|
||||
step('run-gate released for the started producer');
|
||||
}
|
||||
} else if (name === 'monitor-stop-reply') {
|
||||
dropNextMonitorStop = true;
|
||||
await releaseFifo(current.directory, 'monitor-gate');
|
||||
|
|
@ -1256,14 +1267,18 @@ async function runReplyLossCase() {
|
|||
await waitUntil(() => droppedOps >= 1, 120_000);
|
||||
step('observation advance reply dropped once');
|
||||
} else if (name === 'capture-commit-reply') {
|
||||
await waitUntil(() => droppedPagePublishes >= 1, 180_000);
|
||||
step('page publish reply dropped once');
|
||||
// Pages flush at SEGMENTS_PER_PAGE (512) — the fixture's 24
|
||||
// blocks only reach the store at finish, after the gate opens.
|
||||
// Withhold the witness until the finish-time first page publish
|
||||
// instead of deadlocking on it before the release.
|
||||
await waitUntil(() => {
|
||||
checkStoreProxy();
|
||||
return progressLines() >= 24;
|
||||
}, 180_000);
|
||||
await releaseFifo(current.directory, 'run-gate');
|
||||
step('gate released');
|
||||
await waitUntil(() => droppedPagePublishes >= 1, 180_000);
|
||||
step('page publish reply dropped once');
|
||||
}
|
||||
await waitUntil(async () => {
|
||||
checkProxy();
|
||||
|
|
@ -1341,10 +1356,12 @@ async function runStoreFailureCase() {
|
|||
// "no second identity ever minted" observable.
|
||||
await new Promise((resolve) => setTimeout(resolve, 10_000));
|
||||
} else {
|
||||
// manifest/page-envelope-commit: the trigger refuses the
|
||||
// publication's commit kind for the rest of the case, so the run
|
||||
// stays honestly blocked without re-executing until the drain arm
|
||||
// below ends it — there is no heal-over path to wait for.
|
||||
// manifest/page-envelope-commit: the trigger keeps refusing the
|
||||
// target commit kind for the rest of the case, but refusal only
|
||||
// poisons the capture — it never stops the run. Released from the
|
||||
// gate the producer exits 0 and the record settles exited with the
|
||||
// envelope degraded (capture unavailable), so the settle below is
|
||||
// the natural one, not a drain.
|
||||
await waitUntil(async () => {
|
||||
checkProxy();
|
||||
checkStoreProxy();
|
||||
|
|
@ -1475,11 +1492,6 @@ async function runCancelDrainCase() {
|
|||
);
|
||||
}, 120_000);
|
||||
step('run live');
|
||||
const invocationId = (
|
||||
await probe<ProbeState>(current.sessionId, 'state')
|
||||
).executions
|
||||
.map((row) => row.execution_call_id)
|
||||
.find((id) => !id.endsWith(':process'))!;
|
||||
if (name === 'cancel-during-capture') {
|
||||
await waitUntil(async () => {
|
||||
checkStoreProxy();
|
||||
|
|
@ -1487,23 +1499,21 @@ async function runCancelDrainCase() {
|
|||
return fullSegments(state).length >= 3;
|
||||
}, 120_000);
|
||||
step('mid-capture (3 full segments)');
|
||||
await childCancel(cli, clientId, invocationId);
|
||||
// `children/operations` encodes the child-agent cancel family and
|
||||
// rejects shell records; the only stop surface a shell child_run
|
||||
// answers is the broker's drain release. The drain arm IS this
|
||||
// case's cancel: stopRequested commits with the stop evidence,
|
||||
// and the record settles cancelled, never exited.
|
||||
const canceled = await brokerRelease(true);
|
||||
step(`drain cancel mid-capture -> ${canceled.status}`);
|
||||
assert.equal(canceled.status, 200, JSON.stringify(canceled.body));
|
||||
await waitUntil(async () => {
|
||||
const record = childRunOf(
|
||||
await probe<ProbeState>(current.sessionId, 'state'),
|
||||
);
|
||||
return record?.stopRequested === true;
|
||||
}, 60_000);
|
||||
step('stopRequested committed');
|
||||
const stillRunning = childRunOf(
|
||||
await probe<ProbeState>(current.sessionId, 'state'),
|
||||
);
|
||||
assert.equal(stillRunning?.run?.state, 'running');
|
||||
step('cancel alone settled nothing');
|
||||
// The cancel's honorable physical stop IS the drain arm; run it
|
||||
// to give the case its terminal evidence.
|
||||
await brokerRelease(true);
|
||||
step('drain release issued after the recorded cancel');
|
||||
step('stopRequested committed by the drain arm');
|
||||
} else if (name === 'cancel-status-unavailable') {
|
||||
await waitUntil(async () => {
|
||||
checkStoreProxy();
|
||||
|
|
@ -1513,24 +1523,22 @@ async function runCancelDrainCase() {
|
|||
step('mid-capture');
|
||||
pausedBrokerControls = true;
|
||||
step('maintenance route held');
|
||||
await childCancel(cli, clientId, invocationId);
|
||||
await waitUntil(async () => {
|
||||
const record = childRunOf(
|
||||
await probe<ProbeState>(current.sessionId, 'state'),
|
||||
);
|
||||
return record?.stopRequested === true;
|
||||
}, 60_000);
|
||||
step('cancel recorded under the outage');
|
||||
// Positive hold witness: at least one maintenance/status request
|
||||
// must actually be parked in the proxy, otherwise the held arm
|
||||
// was never armed and the outage is fiction.
|
||||
await waitUntil(() => heldControlReplies.length >= 1, 60_000);
|
||||
step('a status request is really held');
|
||||
const unFabricated = childRunOf(
|
||||
await probe<ProbeState>(current.sessionId, 'state'),
|
||||
);
|
||||
assert.equal(unFabricated?.run?.state, 'running');
|
||||
assert.ok(unFabricated?.stopReason == null);
|
||||
step('no fabricated terminal while status is unreachable');
|
||||
const outletDrain = await brokerRelease(true);
|
||||
step(`drain release under the outage -> ${outletDrain.status}`);
|
||||
assert.equal(outletDrain.status, 200, JSON.stringify(outletDrain.body));
|
||||
releaseBrokerControls();
|
||||
step('maintenance route restored');
|
||||
const outletDrain = await brokerRelease(true);
|
||||
step(`drain release after outage restored -> ${outletDrain.status}`);
|
||||
assert.equal(outletDrain.status, 200, JSON.stringify(outletDrain.body));
|
||||
} else if (name === 'cancel-reply-lost') {
|
||||
await waitUntil(async () => {
|
||||
checkStoreProxy();
|
||||
|
|
@ -1539,9 +1547,14 @@ async function runCancelDrainCase() {
|
|||
}, 120_000);
|
||||
step('mid-capture');
|
||||
dropCommitOps.add('commitChildRunRecord');
|
||||
await childCancel(cli, clientId, invocationId);
|
||||
// Same drain surface: the stopRequested write's commit reply is
|
||||
// what gets dropped after it committed; the committed record
|
||||
// must never mint a second stop revision.
|
||||
const canceled = await brokerRelease(true);
|
||||
step(`drain cancel with armed reply drop -> ${canceled.status}`);
|
||||
assert.equal(canceled.status, 200, JSON.stringify(canceled.body));
|
||||
await waitUntil(() => droppedOps >= 1, 60_000);
|
||||
step('cancel commit reply dropped');
|
||||
step('stopRequested commit reply dropped once');
|
||||
await waitUntil(async () => {
|
||||
const record = childRunOf(
|
||||
await probe<ProbeState>(current.sessionId, 'state'),
|
||||
|
|
@ -1549,22 +1562,15 @@ async function runCancelDrainCase() {
|
|||
return record?.stopRequested === true;
|
||||
}, 60_000);
|
||||
step('stopRequested committed despite the lost reply');
|
||||
const commitsBefore = commitLog.filter(
|
||||
const commits = commitLog.filter(
|
||||
(entry) => entry.op === 'commitChildRunRecord',
|
||||
).length;
|
||||
await childCancel(cli, clientId, invocationId);
|
||||
const commitsAfter = commitLog.filter(
|
||||
(entry) => entry.op === 'commitChildRunRecord',
|
||||
).length;
|
||||
assert.equal(
|
||||
commitsAfter,
|
||||
commitsBefore,
|
||||
'a re-driven cancel must not mint a second stop revision',
|
||||
);
|
||||
step('re-driven cancel minted no duplicate revision');
|
||||
const exitDrain = await brokerRelease(true);
|
||||
step(`drain release to settle -> ${exitDrain.status}`);
|
||||
assert.equal(exitDrain.status, 200, JSON.stringify(exitDrain.body));
|
||||
assert.equal(
|
||||
commits.filter((entry) => entry.status !== 'dropped').length,
|
||||
1,
|
||||
`exactly one committed stop revision, saw ${JSON.stringify(commits)}`,
|
||||
);
|
||||
step('the re-drive minted no duplicate revision');
|
||||
} else if (name === 'routine-release-refusal') {
|
||||
const progressBefore = progressLines();
|
||||
const refusal = await brokerRequest(
|
||||
|
|
@ -1718,30 +1724,6 @@ function bgStartedFile(): boolean {
|
|||
}
|
||||
}
|
||||
|
||||
async function childCancel(
|
||||
cli: HostedHarnessProcess,
|
||||
clientId: string,
|
||||
childRunId: string,
|
||||
) {
|
||||
const response = await cli.request(
|
||||
`/session/${current.sessionId}/children/operations`,
|
||||
{
|
||||
method: 'POST',
|
||||
headers: { ...cli.headers(clientId), 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
operationId: randomUUID(),
|
||||
childRunId,
|
||||
kind: 'cancel',
|
||||
}),
|
||||
},
|
||||
);
|
||||
assert.ok(
|
||||
response.status === 200 || response.status === 204,
|
||||
`child cancel -> ${response.status}: ${await response.text()}`,
|
||||
);
|
||||
step(`child cancel -> ${response.status}`);
|
||||
}
|
||||
|
||||
try {
|
||||
const reports = [];
|
||||
for (current of config.cases) {
|
||||
|
|
|
|||
|
|
@ -404,8 +404,15 @@ class HostedMonitorRunFaultGatesIT {
|
|||
break;
|
||||
case "manifest-commit":
|
||||
table = "ma_resource";
|
||||
// The capture's prepare publishes an initial empty
|
||||
// manifest (executionStatus 'unknown'); it must land,
|
||||
// or the worker never starts. The arm targets the
|
||||
// manifest that closes the run: a settled execution.
|
||||
condition = "NEW.session_id = '" + sessionId
|
||||
+ "' AND NEW.kind = 'managed-tool-result-manifest'";
|
||||
+ "' AND NEW.kind"
|
||||
+ " = 'managed-tool-result-manifest'"
|
||||
+ " AND CONVERT(NEW.inline_bytes USING utf8mb4)"
|
||||
+ " LIKE '%\"executionStatus\":\"success\"%'";
|
||||
marker = "FG8B_MANIFEST_COMMIT";
|
||||
break;
|
||||
case "page-envelope-commit":
|
||||
|
|
@ -593,36 +600,38 @@ class HostedMonitorRunFaultGatesIT {
|
|||
assertProcessRows(jdbc, sessionId, name, 0);
|
||||
return;
|
||||
}
|
||||
// The publication's commit kind is refused for the rest of the
|
||||
// case, so the run blocks without re-executing instead of healing
|
||||
// over the gate — and the driver's drain arm ends it, which means
|
||||
// the record carries the drain's stop and never a fabricated exit.
|
||||
// A publication commit that the store refused stops nothing: the
|
||||
// process exits with zero, the envelope degrades to
|
||||
// captureStatus unavailable, and the record settles from real
|
||||
// exit evidence — never cancelled, never re-executed.
|
||||
assertProcessRows(jdbc, sessionId, name, 1);
|
||||
assertThat(report.path("recovered").asBoolean())
|
||||
.as(name + ": the blocked run never self-settled")
|
||||
.isFalse();
|
||||
.as(name + ": the producer exited naturally").isTrue();
|
||||
JsonNode record = extensionRecord(jdbc, tenant, sessionId,
|
||||
"child_run");
|
||||
assertThat(record.path("stopRequested").asBoolean())
|
||||
.as(name + ": the drain carried the stop marker").isTrue();
|
||||
assertThat(record.path("run").path("state").asText())
|
||||
.as(name + ": settles cancelled, never exited")
|
||||
.isEqualTo("cancelled");
|
||||
.as(name + ": settles from real exit evidence")
|
||||
.isEqualTo("settled");
|
||||
assertThat(record.path("stopReason").asText())
|
||||
.isEqualTo("stop_requested");
|
||||
.isEqualTo("exited");
|
||||
assertThat(record.path("exitCode").asInt()).isZero();
|
||||
assertThat(record.path("stopRequested").asBoolean(false))
|
||||
.as(name + ": nothing ever asked it to stop").isFalse();
|
||||
if (name.equals("manifest-commit")) {
|
||||
var manifests = jdbc.queryForObject(
|
||||
// The finalize manifest (executionStatus 'success') is the
|
||||
// arm's target and never persisted; the initial empty
|
||||
// manifest is what keeps the read chain honest.
|
||||
var settled = jdbc.queryForObject(
|
||||
"SELECT COUNT(*) FROM ma_resource WHERE tenant_id = ?"
|
||||
+ " AND session_id = ? AND kind ="
|
||||
+ " 'managed-tool-result-manifest'",
|
||||
+ " 'managed-tool-result-manifest' AND"
|
||||
+ " CONVERT(inline_bytes USING utf8mb4) LIKE"
|
||||
+ " '%\"executionStatus\":\"success\"%'",
|
||||
Integer.class, tenant, sessionId);
|
||||
assertThat(manifests)
|
||||
.as(name + ": every manifest commit was rejected")
|
||||
assertThat(settled).as(name
|
||||
+ ": the settle-manifest commit was rejected")
|
||||
.isZero();
|
||||
} else {
|
||||
// A committed manifest names only persisted envelopes — so
|
||||
// once its page envelopes are refused, the only honest chain
|
||||
// carries no manifest at all.
|
||||
assertThat(jdbc.queryForObject(
|
||||
"SELECT COUNT(*) FROM ma_resource WHERE tenant_id = ?"
|
||||
+ " AND session_id = ? AND kind ="
|
||||
|
|
@ -630,14 +639,23 @@ class HostedMonitorRunFaultGatesIT {
|
|||
Integer.class, tenant, sessionId))
|
||||
.as(name + ": every page-envelope commit was rejected")
|
||||
.isZero();
|
||||
assertThat(jdbc.queryForObject(
|
||||
"SELECT COUNT(*) FROM ma_resource WHERE tenant_id = ?"
|
||||
+ " AND session_id = ? AND kind ="
|
||||
+ " 'managed-tool-result-manifest'",
|
||||
Integer.class, tenant, sessionId))
|
||||
.as(name + ": no manifest may name refused envelopes")
|
||||
.isZero();
|
||||
}
|
||||
// Whatever manifests did commit (the initial empty one, and in
|
||||
// the process-ledger variant reserved ones) may never name a
|
||||
// page envelope that is not persisted — the read chain's own
|
||||
// refusal, asserted here over JDBC rather than on outcomes.
|
||||
var dangling = jdbc.queryForObject(
|
||||
"SELECT COUNT(*) FROM ma_resource m WHERE m.tenant_id = ?"
|
||||
+ " AND m.session_id = ? AND m.kind ="
|
||||
+ " 'managed-tool-result-manifest' AND"
|
||||
+ " CONVERT(m.inline_bytes USING utf8mb4) LIKE"
|
||||
+ " '%\"pages\":[{%' AND NOT EXISTS (SELECT 1"
|
||||
+ " FROM ma_resource p WHERE p.tenant_id ="
|
||||
+ " m.tenant_id AND p.session_id = m.session_id"
|
||||
+ " AND p.kind = 'managed-tool-result-page')",
|
||||
Integer.class, tenant, sessionId);
|
||||
assertThat(dangling).as(name
|
||||
+ ": no manifest names an unpersisted page").isZero();
|
||||
}
|
||||
|
||||
private void assertCancelDrainCase(JdbcTemplate jdbc, String tenant,
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue