mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-10-19 12:43:39 +00:00
test(managed-agent): drop the FG13d hook-cancel reply through a broker proxy (#14555)
* test(managed-agent): drop the hook-cancel reply through a broker proxy Front the embedded broker with the existing forwarding proxy and arm a DROP on the hook-cancel control reply. The owner already aborted, so the gate must resolve the same cancellation through a later status poll, drain the unit exactly once, and never issue a second physical Hook call. * test(managed-agent): budget the FG13d dropped-reply cancel cell The class gains a fourth fixture, so the shared Hosted fork grows by 150 s: healthy 4908, fork 5518, Verify 105 minutes and job 200, with the three pinned comments kept in step. * docs(managed-agent): record the FG13d dropped-reply cancel case and witness
This commit is contained in:
Shaojin Wen
GitHub
parent
46bf492d05
commit
b6debf84a3
6 changed files with 79 additions and 24 deletions
14
.github/workflows/sdk-java.yml
vendored
14
.github/workflows/sdk-java.yml
vendored
|
|
@ -685,11 +685,13 @@ jobs:
|
|||
# Verify to 100 minutes, the FG13d worker-stop cell to 5368 s and the
|
||||
# FG11b store faults to 5498 s, which moved Verify to 105 minutes,
|
||||
# FG11b slice 2 to 5643 s, still inside them, and the event-stream
|
||||
# liveness gates to 6083 s, which moves Verify to 115 minutes so it
|
||||
# wraps its fork + 600 s plus margin; the Broker fault gates keep 10,
|
||||
# and admission plus O4 keep per-fork ceilings) plus ~13 minutes for
|
||||
# the uncapped setup steps (Checkout.. cgroup delegation, the report
|
||||
# checks): 208 + 2. The failover arms run in hosted-failover-e2e.
|
||||
# liveness gates to 6083 s, which moves Verify to 115 minutes; the
|
||||
# FG13d dropped-reply cancel cell adds 150 s and moves the fork to
|
||||
# 6233 s, still inside those 115 minutes, so Verify wraps its fork +
|
||||
# 600 s plus margin; the Broker fault gates keep 10, and admission plus
|
||||
# O4 keep per-fork ceilings) plus ~13 minutes for the uncapped setup
|
||||
# steps (Checkout.. cgroup delegation, the report checks): 208 + 2. The
|
||||
# failover arms run in hosted-failover-e2e.
|
||||
timeout-minutes: 210
|
||||
services:
|
||||
mysql:
|
||||
|
|
@ -779,7 +781,7 @@ jobs:
|
|||
# fork leaves no per-test failure lines, so the main-CI failure
|
||||
# analyzer can only file an undiagnosable per-commit issue (#13503).
|
||||
# hosted-process-ci.test.js pins this ordering; the fork budget is
|
||||
# 6083 s, so the step wraps to fork + 600 s plus margin.
|
||||
# 6233 s, so the step wraps to fork + 600 s plus margin.
|
||||
timeout-minutes: 115
|
||||
env:
|
||||
MAVEN_ARGS: '--settings ${{ runner.temp }}/setup-java-m2/settings.xml --toolchains ${{ runner.temp }}/setup-java-m2/toolchains.xml'
|
||||
|
|
|
|||
|
|
@ -39,6 +39,7 @@ once the unit is empty (`hook-command-cgroup.ts:227-248`,
|
|||
| `cancel-held-command` | A `PreToolUse` command Hook holds with a descendant that ignores SIGTERM; cancel the operation through the Harness route. | The command gets SIGTERM; the descendant survives it, so the unit is only emptied by `cgroup.kill` after the grace. The record settles `cancelled` and only then reports; the unit reaches `populated 0`; exactly one physical call; `failClosed` turns the cancelled outcome into a block and the tool never runs. |
|
||||
| `timeout-held-command` | The same held command with a 2 s Hook timeout and a descendant that ignores SIGTERM. | The command times out, its output is not applied, the record settles with `outcome=timeout`, the unit is killed and reaches `populated 0`, and the model sees the block instead of the file body. |
|
||||
| `cancel-with-stopped-worker` | The same held command; SIGSTOP the durable worker, then cancel through the Harness route. | The cancel is recorded, but the owner cannot answer: the record stays `recovery_blocked` / `outcome_unknown`, the command and its descendant keep the unit populated, and no second call happens. |
|
||||
| `cancel-with-dropped-reply` | The same held command; the broker proxy drops the `hook-cancel` control reply after the worker aborted. | The cancel is already recorded, so the later status poll of the same owner settles it: the record is `cancelled`, the unit drains once, exactly one physical call, and the proxy log proves the drop. |
|
||||
|
||||
## Mutation witnesses
|
||||
|
||||
|
|
@ -59,13 +60,17 @@ Each witness must turn its cell red before the slice merges.
|
|||
(the branch that records `recovery_blocked` / `outcome_unknown` when the
|
||||
owner cannot answer). The record stays `running`, so the state assertion
|
||||
fails.
|
||||
- `cancel-with-dropped-reply`: break the operation-id idempotency of the
|
||||
cancel dispatch (or remove the status re-poll that resolves the same
|
||||
occurrence). The retry then issues a second physical call, or the record
|
||||
stays unknown; the one-call or `cancelled` assertion fails.
|
||||
|
||||
## Non-goals
|
||||
|
||||
The rest of FG13d — cancel before dispatch, a dropped `hook-cancel` reply,
|
||||
and the DELETE async-drain with `SessionEnd`/`SessionDelete` — stays on
|
||||
#14449. Also out: FG13a, FG13b, FG13e, FG13f, local non-managed Hooks,
|
||||
Windows/macOS and #13133's design items.
|
||||
The rest of FG13d — cancel before dispatch and the DELETE async-drain with
|
||||
`SessionEnd`/`SessionDelete` — stays on #14449. Also out: FG13a, FG13b,
|
||||
FG13e, FG13f, local non-managed Hooks, Windows/macOS and #13133's design
|
||||
items.
|
||||
|
||||
## Validation
|
||||
|
||||
|
|
|
|||
|
|
@ -33,6 +33,7 @@ owner 重启矩阵；本切片是 **FG13d 的前半部分：取消与超时**，
|
|||
| `cancel-held-command` | `PreToolUse` 命令 Hook 带一个忽略 SIGTERM 的后代保持运行；通过 Harness 路由取消该操作。 | 命令收到 SIGTERM；后代存活，因此只有宽限期后的 `cgroup.kill` 能清空单元。记录结算为 `cancelled` 且只在那之后报告；单元到达 `populated 0`；恰好一次物理调用；`failClosed` 把取消结果变成阻断，工具不运行。 |
|
||||
| `timeout-held-command` | 同样的保持型命令，Hook 超时 2 s，后代忽略 SIGTERM。 | 命令超时，输出不被采用，记录结算为 `outcome=timeout`，单元被杀并到达 `populated 0`，模型看到阻断而不是文件正文。 |
|
||||
| `cancel-with-stopped-worker` | 同样的保持型命令；对 durable worker 发 SIGSTOP，再通过 Harness 路由取消。 | 取消被记录，但 owner 无法应答：记录停在 `recovery_blocked` / `outcome_unknown`，命令与后代继续占据单元，且不会发生第二次调用。 |
|
||||
| `cancel-with-dropped-reply` | 同样的保持型命令；broker 代理在 worker 中止后丢弃 `hook-cancel` 控制调用的回复。 | 取消已记录，因此同一 owner 的后续状态轮询会结算它：记录为 `cancelled`、单元只 drain 一次、恰好一次物理调用，代理日志证明这次丢弃。 |
|
||||
|
||||
## 变异见证
|
||||
|
||||
|
|
@ -49,13 +50,15 @@ owner 重启矩阵；本切片是 **FG13d 的前半部分：取消与超时**，
|
|||
- `cancel-with-stopped-worker`：去掉 `status()` 里如实记录 unknown 的提交分支
|
||||
（owner 无法应答时写 `recovery_blocked` / `outcome_unknown` 的那一支）。
|
||||
记录会停在 `running`，状态断言失败。
|
||||
- `cancel-with-dropped-reply`：破坏取消派发的 operation-id 幂等（或移除解析同一
|
||||
occurrence 的状态轮询）。重试会产生第二次物理调用，或记录停在未知态；一次
|
||||
调用或 `cancelled` 断言失败。
|
||||
|
||||
## 非目标
|
||||
|
||||
FG13d 其余部分——派发前取消、`hook-cancel` 回复丢失、以及带
|
||||
`SessionEnd`/`SessionDelete` 的 DELETE 异步 drain——仍留在 #14449。同样不
|
||||
在范围：FG13a、FG13b、FG13e、FG13f、本地非托管 Hook、Windows/macOS，以及
|
||||
#13133 的设计项。
|
||||
FG13d 其余部分——派发前取消、以及带 `SessionEnd`/`SessionDelete` 的 DELETE
|
||||
异步 drain——仍留在 #14449。同样不在范围：FG13a、FG13b、FG13e、FG13f、本地
|
||||
非托管 Hook、Windows/macOS，以及 #13133 的设计项。
|
||||
|
||||
## 验证
|
||||
|
||||
|
|
|
|||
|
|
@ -453,8 +453,13 @@
|
|||
across four runs, ~440 s budgeted at
|
||||
2x) raise the healthy sum to 5473 s,
|
||||
so the fork moves to 6083 s and the
|
||||
Verify step to 115 minutes. -->
|
||||
<forkedProcessTimeoutInSeconds>6083</forkedProcessTimeoutInSeconds>
|
||||
Verify step to 115 minutes. The FG13d
|
||||
dropped-reply cancel cell (issue
|
||||
#14449) adds 150 s: the healthy sum
|
||||
becomes 5623 s, so the fork moves to
|
||||
6233 s, still inside those 115
|
||||
minutes. -->
|
||||
<forkedProcessTimeoutInSeconds>6233</forkedProcessTimeoutInSeconds>
|
||||
</configuration>
|
||||
<executions>
|
||||
<execution>
|
||||
|
|
|
|||
|
|
@ -85,6 +85,7 @@ class HostedHookCancelFaultGatesIT {
|
|||
private ServletWebServerApplicationContext spring;
|
||||
private JdbcTemplate jdbc;
|
||||
private Process harness;
|
||||
private HostedSessionRelayForwardingProxy proxy;
|
||||
|
||||
@Test
|
||||
@Timeout(300)
|
||||
|
|
@ -190,6 +191,37 @@ class HostedHookCancelFaultGatesIT {
|
|||
assertThat(modelFailure.get()).isNull();
|
||||
}
|
||||
|
||||
@Test
|
||||
@Timeout(300)
|
||||
void fg13dCancelWithDroppedReplySettlesThroughStatusOnMySql() throws Exception {
|
||||
boot("60000", "-1", "hook-ignore-term");
|
||||
proxy.arm(HostedSessionRelayForwardingProxy.Rule.drop(
|
||||
(method, path, body) -> path.contains("/control") && body.contains("\"kind\":\"hook-cancel\""),
|
||||
"hook-cancel-reply"));
|
||||
JsonNode definition = api("POST", "/v1/agents", definition(), "definition", 202);
|
||||
String session = createSession(definition.path("id").asText(), "initial");
|
||||
submit(session, "FG13_CANCEL_DROP");
|
||||
waitFor(() -> {
|
||||
assertThat(events("hook.call")).hasSize(1);
|
||||
assertThat(events("hook.child")).hasSize(1);
|
||||
});
|
||||
JsonNode hook = events("hook.call").getFirst();
|
||||
JsonNode child = events("hook.child").getFirst();
|
||||
assertThat(running(hook)).isTrue();
|
||||
assertThat(Files.readString(cgroup(hook).resolve("cgroup.events"))).contains("populated 1");
|
||||
String executionId = hookExecution(session).path("hookExecutionId").asText();
|
||||
|
||||
cancelHook(session, executionId);
|
||||
// The cancel reply is gone, but the poll through the record's owner
|
||||
// resolves the same cancellation, and the gate drains exactly once.
|
||||
awaitTerminalDrained(session, "cancelled", child);
|
||||
assertThat(events("hook.call")).hasSize(1);
|
||||
assertThat(events("hook.allow")).isEmpty();
|
||||
assertThat(hookResult(session).path("outcome").asText()).isEqualTo("cancelled");
|
||||
assertThat(Files.readString(temporary.resolve("proxy.log"))).contains("PROXY dropped hook-cancel-reply");
|
||||
assertThat(modelFailure.get()).isNull();
|
||||
}
|
||||
|
||||
// The record may report a terminal state only once the unit is empty:
|
||||
// inspect the first observation, not a later poll, so a regression that
|
||||
// settles early while the drain continues still fails the cell.
|
||||
|
|
@ -298,6 +330,8 @@ class HostedHookCancelFaultGatesIT {
|
|||
tenant, WorkspaceExecutionProfile.CONFIG_REF, WorkspaceExecutionProfile.POLICY_REF);
|
||||
jdbc.update("INSERT INTO ma_workspace_access (tenant_id, workspace_id, actor_id, role)"
|
||||
+ " VALUES (?, 'workspace', ?, 'OPERATOR')", tenant, "actor".getBytes(StandardCharsets.UTF_8));
|
||||
proxy = HostedSessionRelayForwardingProxy.start("hook-broker", temporary.resolve("proxy.log"));
|
||||
proxy.forwardTo(URI.create("http://127.0.0.1:" + brokerPort));
|
||||
startHarness();
|
||||
}
|
||||
|
||||
|
|
@ -400,7 +434,7 @@ class HostedHookCancelFaultGatesIT {
|
|||
ProcessBuilder builder = new ProcessBuilder(node, cli.toString(), "serve", "--profile", "hosted-harness",
|
||||
"--http-bridge", "--hostname", "127.0.0.1", "--port", Integer.toString(harnessPort), "--require-auth",
|
||||
"--no-web", "--workspace", temporary.resolve("decoy").toString(), "--managed-runtime-broker-url",
|
||||
"http://127.0.0.1:" + brokerPort, "--managed-runtime-broker-token", TOKEN)
|
||||
proxy.baseUrl(), "--managed-runtime-broker-token", TOKEN)
|
||||
.directory(temporary.resolve("decoy").toFile()).redirectErrorStream(true)
|
||||
.redirectOutput(temporary.resolve("harness-" + boots + ".log").toFile());
|
||||
var env = builder.environment();
|
||||
|
|
@ -446,7 +480,7 @@ class HostedHookCancelFaultGatesIT {
|
|||
request.path("tools").forEach(tool -> names.add(tool.at("/function/name").asText()));
|
||||
assertThat(names).contains("read_file");
|
||||
boolean call = results.isEmpty()
|
||||
&& List.of("FG13_CANCEL", "FG13_CANCEL_STOP", "FG13_TIMEOUT").contains(prompt);
|
||||
&& List.of("FG13_CANCEL", "FG13_CANCEL_DROP", "FG13_CANCEL_STOP", "FG13_TIMEOUT").contains(prompt);
|
||||
var chunk = json.createObjectNode().put("id", "hook-fault").put("object", "chat.completion.chunk")
|
||||
.put("created", 0).put("model", "hook-fault-fixture");
|
||||
var choice = chunk.putArray("choices").addObject().put("index", 0);
|
||||
|
|
@ -585,6 +619,10 @@ class HostedHookCancelFaultGatesIT {
|
|||
Files.delete(entry);
|
||||
}
|
||||
}
|
||||
}, () -> {
|
||||
if (proxy != null) {
|
||||
proxy.close();
|
||||
}
|
||||
}, http::close);
|
||||
}
|
||||
|
||||
|
|
@ -595,7 +633,7 @@ class HostedHookCancelFaultGatesIT {
|
|||
Path destination = Files.createDirectories(Path.of("target/failsafe-reports/hosted-hook-fault-evidence", tenant));
|
||||
System.out.println("HOSTED_HOOK_FAULT_EVIDENCE " + destination.toAbsolutePath());
|
||||
json.writeValue(destination.resolve("model-requests.json").toFile(), modelRequests);
|
||||
for (String name : List.of("processes.jsonl", "harness-1.log", "harness-2.log")) {
|
||||
for (String name : List.of("processes.jsonl", "harness-1.log", "harness-2.log", "proxy.log")) {
|
||||
Path source = temporary.resolve(name);
|
||||
if (Files.exists(source)) {
|
||||
Files.copy(source, destination.resolve(source.getFileName()));
|
||||
|
|
|
|||
|
|
@ -604,13 +604,14 @@ describe('Hosted real-process gates', () => {
|
|||
// ~130 s, so the fork moves to 5498 and the Verify step to 105
|
||||
// minutes; FG11b slice 2's Harness-side store faults add ~145 s, so
|
||||
// the fork moves to 5643, still inside those 105 minutes; the
|
||||
// event-stream liveness gates add ~440 s, so the fork moves to 6083
|
||||
// and the Verify step to 115 minutes):
|
||||
// event-stream liveness gates add ~440 s, so the fork moves to 6083;
|
||||
// the FG13d dropped-reply cancel cell adds 150 s, so the fork moves
|
||||
// to 6233 and the Verify step to 115 minutes):
|
||||
// the relational check above tolerates
|
||||
// anything up to step*60, so the value is pinned literally here —
|
||||
// an unnoticed raise has no other red to land on.
|
||||
if (profile === 'hosted-harness-mysql') {
|
||||
expect(forkSeconds).toBe(6083);
|
||||
expect(forkSeconds).toBe(6233);
|
||||
}
|
||||
},
|
||||
);
|
||||
|
|
@ -659,10 +660,11 @@ describe('Hosted real-process gates', () => {
|
|||
// event-stream liveness gates (HostedEventStreamLivenessIT, four cases on
|
||||
// the shipped 15s keepalive and 45s bound) measure 214.4–215.0s for the
|
||||
// class locally on JDK 21 and 25 + MySQL 8.4.6 across four acceptance runs
|
||||
// (~440s budgeted at 2x), so it becomes 5473s.
|
||||
// (~440s budgeted at 2x), so it becomes 5473s; the FG13d dropped-reply
|
||||
// cancel cell adds 150s, so it becomes 5623s.
|
||||
// A larger ceiling lets failsafe kill the fork mid-run, and the classes
|
||||
// left unrun never report, failing the gate undiagnosably (#13780).
|
||||
const hostedForkHealthySeconds = 5473;
|
||||
const hostedForkHealthySeconds = 5623;
|
||||
const hostedForkSeconds = () =>
|
||||
Number(
|
||||
read('packages/managed-agent-server/pom.xml')
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue