From b6debf84a3fdd73697da269d2a825e5833f394de Mon Sep 17 00:00:00 2001 From: Shaojin Wen Date: Mon, 19 Oct 2026 00:12:46 +0800 Subject: [PATCH] test(managed-agent): drop the FG13d hook-cancel reply through a broker proxy (#14555) * test(managed-agent): drop the hook-cancel reply through a broker proxy Front the embedded broker with the existing forwarding proxy and arm a DROP on the hook-cancel control reply. The owner already aborted, so the gate must resolve the same cancellation through a later status poll, drain the unit exactly once, and never issue a second physical Hook call. * test(managed-agent): budget the FG13d dropped-reply cancel cell The class gains a fourth fixture, so the shared Hosted fork grows by 150 s: healthy 4908, fork 5518, Verify 105 minutes and job 200, with the three pinned comments kept in step. * docs(managed-agent): record the FG13d dropped-reply cancel case and witness --- .github/workflows/sdk-java.yml | 14 +++--- ...-10-18-hosted-hook-cancel-drain-gates-d.md | 13 ++++-- ...-hosted-hook-cancel-drain-gates-d.zh-CN.md | 11 +++-- packages/managed-agent-server/pom.xml | 9 +++- .../HostedHookCancelFaultGatesIT.java | 44 +++++++++++++++++-- scripts/tests/hosted-process-ci.test.js | 12 ++--- 6 files changed, 79 insertions(+), 24 deletions(-) diff --git a/.github/workflows/sdk-java.yml b/.github/workflows/sdk-java.yml index 9bd0da8e20..05a6bc7fd5 100644 --- a/.github/workflows/sdk-java.yml +++ b/.github/workflows/sdk-java.yml @@ -685,11 +685,13 @@ jobs: # Verify to 100 minutes, the FG13d worker-stop cell to 5368 s and the # FG11b store faults to 5498 s, which moved Verify to 105 minutes, # FG11b slice 2 to 5643 s, still inside them, and the event-stream - # liveness gates to 6083 s, which moves Verify to 115 minutes so it - # wraps its fork + 600 s plus margin; the Broker fault gates keep 10, - # and admission plus O4 keep per-fork ceilings) plus ~13 minutes for - # the uncapped setup steps (Checkout.. cgroup delegation, the report - # checks): 208 + 2. The failover arms run in hosted-failover-e2e. + # liveness gates to 6083 s, which moves Verify to 115 minutes; the + # FG13d dropped-reply cancel cell adds 150 s and moves the fork to + # 6233 s, still inside those 115 minutes, so Verify wraps its fork + + # 600 s plus margin; the Broker fault gates keep 10, and admission plus + # O4 keep per-fork ceilings) plus ~13 minutes for the uncapped setup + # steps (Checkout.. cgroup delegation, the report checks): 208 + 2. The + # failover arms run in hosted-failover-e2e. timeout-minutes: 210 services: mysql: @@ -779,7 +781,7 @@ jobs: # fork leaves no per-test failure lines, so the main-CI failure # analyzer can only file an undiagnosable per-commit issue (#13503). # hosted-process-ci.test.js pins this ordering; the fork budget is - # 6083 s, so the step wraps to fork + 600 s plus margin. + # 6233 s, so the step wraps to fork + 600 s plus margin. timeout-minutes: 115 env: MAVEN_ARGS: '--settings ${{ runner.temp }}/setup-java-m2/settings.xml --toolchains ${{ runner.temp }}/setup-java-m2/toolchains.xml' diff --git a/docs/design/2026-10-18-hosted-hook-cancel-drain-gates-d.md b/docs/design/2026-10-18-hosted-hook-cancel-drain-gates-d.md index 5f5f6861d2..c310c38afc 100644 --- a/docs/design/2026-10-18-hosted-hook-cancel-drain-gates-d.md +++ b/docs/design/2026-10-18-hosted-hook-cancel-drain-gates-d.md @@ -39,6 +39,7 @@ once the unit is empty (`hook-command-cgroup.ts:227-248`, | `cancel-held-command` | A `PreToolUse` command Hook holds with a descendant that ignores SIGTERM; cancel the operation through the Harness route. | The command gets SIGTERM; the descendant survives it, so the unit is only emptied by `cgroup.kill` after the grace. The record settles `cancelled` and only then reports; the unit reaches `populated 0`; exactly one physical call; `failClosed` turns the cancelled outcome into a block and the tool never runs. | | `timeout-held-command` | The same held command with a 2 s Hook timeout and a descendant that ignores SIGTERM. | The command times out, its output is not applied, the record settles with `outcome=timeout`, the unit is killed and reaches `populated 0`, and the model sees the block instead of the file body. | | `cancel-with-stopped-worker` | The same held command; SIGSTOP the durable worker, then cancel through the Harness route. | The cancel is recorded, but the owner cannot answer: the record stays `recovery_blocked` / `outcome_unknown`, the command and its descendant keep the unit populated, and no second call happens. | +| `cancel-with-dropped-reply` | The same held command; the broker proxy drops the `hook-cancel` control reply after the worker aborted. | The cancel is already recorded, so the later status poll of the same owner settles it: the record is `cancelled`, the unit drains once, exactly one physical call, and the proxy log proves the drop. | ## Mutation witnesses @@ -59,13 +60,17 @@ Each witness must turn its cell red before the slice merges. (the branch that records `recovery_blocked` / `outcome_unknown` when the owner cannot answer). The record stays `running`, so the state assertion fails. +- `cancel-with-dropped-reply`: break the operation-id idempotency of the + cancel dispatch (or remove the status re-poll that resolves the same + occurrence). The retry then issues a second physical call, or the record + stays unknown; the one-call or `cancelled` assertion fails. ## Non-goals -The rest of FG13d — cancel before dispatch, a dropped `hook-cancel` reply, -and the DELETE async-drain with `SessionEnd`/`SessionDelete` — stays on -#14449. Also out: FG13a, FG13b, FG13e, FG13f, local non-managed Hooks, -Windows/macOS and #13133's design items. +The rest of FG13d — cancel before dispatch and the DELETE async-drain with +`SessionEnd`/`SessionDelete` — stays on #14449. Also out: FG13a, FG13b, +FG13e, FG13f, local non-managed Hooks, Windows/macOS and #13133's design +items. ## Validation diff --git a/docs/design/2026-10-18-hosted-hook-cancel-drain-gates-d.zh-CN.md b/docs/design/2026-10-18-hosted-hook-cancel-drain-gates-d.zh-CN.md index 39c64ae308..5edc05fedc 100644 --- a/docs/design/2026-10-18-hosted-hook-cancel-drain-gates-d.zh-CN.md +++ b/docs/design/2026-10-18-hosted-hook-cancel-drain-gates-d.zh-CN.md @@ -33,6 +33,7 @@ owner 重启矩阵；本切片是 **FG13d 的前半部分：取消与超时**， | `cancel-held-command` | `PreToolUse` 命令 Hook 带一个忽略 SIGTERM 的后代保持运行；通过 Harness 路由取消该操作。 | 命令收到 SIGTERM；后代存活，因此只有宽限期后的 `cgroup.kill` 能清空单元。记录结算为 `cancelled` 且只在那之后报告；单元到达 `populated 0`；恰好一次物理调用；`failClosed` 把取消结果变成阻断，工具不运行。 | | `timeout-held-command` | 同样的保持型命令，Hook 超时 2 s，后代忽略 SIGTERM。 | 命令超时，输出不被采用，记录结算为 `outcome=timeout`，单元被杀并到达 `populated 0`，模型看到阻断而不是文件正文。 | | `cancel-with-stopped-worker` | 同样的保持型命令；对 durable worker 发 SIGSTOP，再通过 Harness 路由取消。 | 取消被记录，但 owner 无法应答：记录停在 `recovery_blocked` / `outcome_unknown`，命令与后代继续占据单元，且不会发生第二次调用。 | +| `cancel-with-dropped-reply` | 同样的保持型命令；broker 代理在 worker 中止后丢弃 `hook-cancel` 控制调用的回复。 | 取消已记录，因此同一 owner 的后续状态轮询会结算它：记录为 `cancelled`、单元只 drain 一次、恰好一次物理调用，代理日志证明这次丢弃。 | ## 变异见证 @@ -49,13 +50,15 @@ owner 重启矩阵；本切片是 **FG13d 的前半部分：取消与超时**， - `cancel-with-stopped-worker`：去掉 `status()` 里如实记录 unknown 的提交分支 （owner 无法应答时写 `recovery_blocked` / `outcome_unknown` 的那一支）。 记录会停在 `running`，状态断言失败。 +- `cancel-with-dropped-reply`：破坏取消派发的 operation-id 幂等（或移除解析同一 + occurrence 的状态轮询）。重试会产生第二次物理调用，或记录停在未知态；一次 + 调用或 `cancelled` 断言失败。 ## 非目标 -FG13d 其余部分——派发前取消、`hook-cancel` 回复丢失、以及带 -`SessionEnd`/`SessionDelete` 的 DELETE 异步 drain——仍留在 #14449。同样不 -在范围：FG13a、FG13b、FG13e、FG13f、本地非托管 Hook、Windows/macOS，以及 -#13133 的设计项。 +FG13d 其余部分——派发前取消、以及带 `SessionEnd`/`SessionDelete` 的 DELETE +异步 drain——仍留在 #14449。同样不在范围：FG13a、FG13b、FG13e、FG13f、本地 +非托管 Hook、Windows/macOS，以及 #13133 的设计项。 ## 验证 diff --git a/packages/managed-agent-server/pom.xml b/packages/managed-agent-server/pom.xml index 0d65812684..8f6091c9f8 100644 --- a/packages/managed-agent-server/pom.xml +++ b/packages/managed-agent-server/pom.xml @@ -453,8 +453,13 @@ across four runs, ~440 s budgeted at 2x) raise the healthy sum to 5473 s, so the fork moves to 6083 s and the - Verify step to 115 minutes. --> - 6083 + Verify step to 115 minutes. The FG13d + dropped-reply cancel cell (issue + #14449) adds 150 s: the healthy sum + becomes 5623 s, so the fork moves to + 6233 s, still inside those 115 + minutes. --> + 6233 diff --git a/packages/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/HostedHookCancelFaultGatesIT.java b/packages/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/HostedHookCancelFaultGatesIT.java index dcf8a94405..995690fe2d 100644 --- a/packages/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/HostedHookCancelFaultGatesIT.java +++ b/packages/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/HostedHookCancelFaultGatesIT.java @@ -85,6 +85,7 @@ class HostedHookCancelFaultGatesIT { private ServletWebServerApplicationContext spring; private JdbcTemplate jdbc; private Process harness; + private HostedSessionRelayForwardingProxy proxy; @Test @Timeout(300) @@ -190,6 +191,37 @@ class HostedHookCancelFaultGatesIT { assertThat(modelFailure.get()).isNull(); } + @Test + @Timeout(300) + void fg13dCancelWithDroppedReplySettlesThroughStatusOnMySql() throws Exception { + boot("60000", "-1", "hook-ignore-term"); + proxy.arm(HostedSessionRelayForwardingProxy.Rule.drop( + (method, path, body) -> path.contains("/control") && body.contains("\"kind\":\"hook-cancel\""), + "hook-cancel-reply")); + JsonNode definition = api("POST", "/v1/agents", definition(), "definition", 202); + String session = createSession(definition.path("id").asText(), "initial"); + submit(session, "FG13_CANCEL_DROP"); + waitFor(() -> { + assertThat(events("hook.call")).hasSize(1); + assertThat(events("hook.child")).hasSize(1); + }); + JsonNode hook = events("hook.call").getFirst(); + JsonNode child = events("hook.child").getFirst(); + assertThat(running(hook)).isTrue(); + assertThat(Files.readString(cgroup(hook).resolve("cgroup.events"))).contains("populated 1"); + String executionId = hookExecution(session).path("hookExecutionId").asText(); + + cancelHook(session, executionId); + // The cancel reply is gone, but the poll through the record's owner + // resolves the same cancellation, and the gate drains exactly once. + awaitTerminalDrained(session, "cancelled", child); + assertThat(events("hook.call")).hasSize(1); + assertThat(events("hook.allow")).isEmpty(); + assertThat(hookResult(session).path("outcome").asText()).isEqualTo("cancelled"); + assertThat(Files.readString(temporary.resolve("proxy.log"))).contains("PROXY dropped hook-cancel-reply"); + assertThat(modelFailure.get()).isNull(); + } + // The record may report a terminal state only once the unit is empty: // inspect the first observation, not a later poll, so a regression that // settles early while the drain continues still fails the cell. @@ -298,6 +330,8 @@ class HostedHookCancelFaultGatesIT { tenant, WorkspaceExecutionProfile.CONFIG_REF, WorkspaceExecutionProfile.POLICY_REF); jdbc.update("INSERT INTO ma_workspace_access (tenant_id, workspace_id, actor_id, role)" + " VALUES (?, 'workspace', ?, 'OPERATOR')", tenant, "actor".getBytes(StandardCharsets.UTF_8)); + proxy = HostedSessionRelayForwardingProxy.start("hook-broker", temporary.resolve("proxy.log")); + proxy.forwardTo(URI.create("http://127.0.0.1:" + brokerPort)); startHarness(); } @@ -400,7 +434,7 @@ class HostedHookCancelFaultGatesIT { ProcessBuilder builder = new ProcessBuilder(node, cli.toString(), "serve", "--profile", "hosted-harness", "--http-bridge", "--hostname", "127.0.0.1", "--port", Integer.toString(harnessPort), "--require-auth", "--no-web", "--workspace", temporary.resolve("decoy").toString(), "--managed-runtime-broker-url", - "http://127.0.0.1:" + brokerPort, "--managed-runtime-broker-token", TOKEN) + proxy.baseUrl(), "--managed-runtime-broker-token", TOKEN) .directory(temporary.resolve("decoy").toFile()).redirectErrorStream(true) .redirectOutput(temporary.resolve("harness-" + boots + ".log").toFile()); var env = builder.environment(); @@ -446,7 +480,7 @@ class HostedHookCancelFaultGatesIT { request.path("tools").forEach(tool -> names.add(tool.at("/function/name").asText())); assertThat(names).contains("read_file"); boolean call = results.isEmpty() - && List.of("FG13_CANCEL", "FG13_CANCEL_STOP", "FG13_TIMEOUT").contains(prompt); + && List.of("FG13_CANCEL", "FG13_CANCEL_DROP", "FG13_CANCEL_STOP", "FG13_TIMEOUT").contains(prompt); var chunk = json.createObjectNode().put("id", "hook-fault").put("object", "chat.completion.chunk") .put("created", 0).put("model", "hook-fault-fixture"); var choice = chunk.putArray("choices").addObject().put("index", 0); @@ -585,6 +619,10 @@ class HostedHookCancelFaultGatesIT { Files.delete(entry); } } + }, () -> { + if (proxy != null) { + proxy.close(); + } }, http::close); } @@ -595,7 +633,7 @@ class HostedHookCancelFaultGatesIT { Path destination = Files.createDirectories(Path.of("target/failsafe-reports/hosted-hook-fault-evidence", tenant)); System.out.println("HOSTED_HOOK_FAULT_EVIDENCE " + destination.toAbsolutePath()); json.writeValue(destination.resolve("model-requests.json").toFile(), modelRequests); - for (String name : List.of("processes.jsonl", "harness-1.log", "harness-2.log")) { + for (String name : List.of("processes.jsonl", "harness-1.log", "harness-2.log", "proxy.log")) { Path source = temporary.resolve(name); if (Files.exists(source)) { Files.copy(source, destination.resolve(source.getFileName())); diff --git a/scripts/tests/hosted-process-ci.test.js b/scripts/tests/hosted-process-ci.test.js index 7ff9fd207c..5b82db5e85 100644 --- a/scripts/tests/hosted-process-ci.test.js +++ b/scripts/tests/hosted-process-ci.test.js @@ -604,13 +604,14 @@ describe('Hosted real-process gates', () => { // ~130 s, so the fork moves to 5498 and the Verify step to 105 // minutes; FG11b slice 2's Harness-side store faults add ~145 s, so // the fork moves to 5643, still inside those 105 minutes; the - // event-stream liveness gates add ~440 s, so the fork moves to 6083 - // and the Verify step to 115 minutes): + // event-stream liveness gates add ~440 s, so the fork moves to 6083; + // the FG13d dropped-reply cancel cell adds 150 s, so the fork moves + // to 6233 and the Verify step to 115 minutes): // the relational check above tolerates // anything up to step*60, so the value is pinned literally here — // an unnoticed raise has no other red to land on. if (profile === 'hosted-harness-mysql') { - expect(forkSeconds).toBe(6083); + expect(forkSeconds).toBe(6233); } }, ); @@ -659,10 +660,11 @@ describe('Hosted real-process gates', () => { // event-stream liveness gates (HostedEventStreamLivenessIT, four cases on // the shipped 15s keepalive and 45s bound) measure 214.4–215.0s for the // class locally on JDK 21 and 25 + MySQL 8.4.6 across four acceptance runs - // (~440s budgeted at 2x), so it becomes 5473s. + // (~440s budgeted at 2x), so it becomes 5473s; the FG13d dropped-reply + // cancel cell adds 150s, so it becomes 5623s. // A larger ceiling lets failsafe kill the fork mid-run, and the classes // left unrun never report, failing the gate undiagnosably (#13780). - const hostedForkHealthySeconds = 5473; + const hostedForkHealthySeconds = 5623; const hostedForkSeconds = () => Number( read('packages/managed-agent-server/pom.xml')