fix(managed-agent): publish the stale-Harness wait instead of hiding it

The generation-mismatch wait was budget-exempt and invisible. The terminal
arm is vetoed by !staleBoot, and the exempt reschedule writes
budget_exempt_attempt = attempt_count, so the effective budget stayed at zero
and the row read RUNNING/PENDING with error_code NULL — byte-identical to a
healthy retry — for as long as this replica kept its stale Harness boot, which
only a Java restart clears. The writer wait, the other unbounded external
wait, already publishes recovery_blocked with its code past the budget; there
was no reason for this one to be the invisible half, and an operator cannot
restart the replica that a row does not name.

Publishes it the same way, as hosted_harness_generation_mismatch. The wait
stays unbounded and the recovery scan still re-drives it, so nothing
terminates earlier than before — the row is only legible now. Gated on
!cwdChange because a cwd change never reaches settle() and is bounded by its
own CWD_CHANGE_ATTEMPT_BUDGET.

Contract and docs follow: the failure_code description on both OpenAPI
surfaces and the regenerated client, and the durable-lifecycle design doc in
both languages, which also claimed "the one exception" and then described a
second one four lines later.

Pinned by aStaleHarnessViewPublishesItsWaitPastTheBudget (renamed from
...KeepsTheOperationWaitingPastTheBudget, which asserted the invisible retry)
plus aStaleHarnessViewReschedulesExemptlyBelowTheBudget, so publication stays
budget-gated rather than becoming the first response to a generation error.
Mutant-verified: deleting the new branch turns the first test red.
This commit is contained in:
wenshao 2026-10-07 12:06:08 +08:00
parent dfcb2a282a
commit 9989e1cea6
6 changed files with 110 additions and 38 deletions

View file

@ -201,22 +201,27 @@ which for a delete of an already closed bound Session is no call at all —
and terminates `failed` with the settle's failure
code, so a Session whose settle can never succeed still reaches a terminal
state instead of looping forever: the Session keeps its pending status, no
`session.closed` is appended, and no completion is certified. The one
exception is a live journal writer, whatever status the operation was
admitted on — a close or delete of an active Session whose settle waits for
the writer, or a delete of a closed Session whose retention retirement
refuses while a residual writer holds the journal: either can still succeed
once that writer stops, so the budget does not terminate it and it keeps
waiting until the writer stops or its lease lapses. The wait's attempts stay
budget-exempt — only attempts that could have made progress consume the
budget — and once the budget is spent the row reads `recovery_blocked` with
the `session_close_writer_live` code, so a wedged close is visible instead of
reading byte-identical to a healthy retry. After the Hosted Harness
restarts, the Java
connector keeps the previous boot, so its calls fail with a generation error
until Java restarts too, as Turn dispatch does; the operation waits meanwhile
— budget-exempt like the writer wait — and then until the old process's
writer lease expires.
`session.closed` is appended, and no completion is certified. Two conditions
wait instead, because either can still succeed. A live journal writer,
whatever status the operation was admitted on — a close or delete of an
active Session whose settle waits for the writer, or a delete of a closed
Session whose retention retirement refuses while a residual writer holds the
journal — stops being an obstacle once that writer stops or its lease lapses.
Java's own stale view of a restarted Harness is the other: after the Hosted
Harness restarts the connector keeps the previous boot, so its calls fail
with a generation error until Java restarts too, as Turn dispatch does, and
the close can still succeed once this replica refreshes; the operation waits
meanwhile, and then until the old process's writer lease expires. Neither
wait keys on the journal alone — the writer wait also requires the failure to
still be retryable, so a permanent refusal thrown before the Harness was ever
asked to stop reaches the terminal arm instead of waiting on its own writer
lease. Both waits' attempts stay budget-exempt — only attempts that could
have made progress consume the budget — and once the budget is spent each row
reads `recovery_blocked` with its own code, `session_close_writer_live` or
`hosted_harness_generation_mismatch`, so a wedged close is visible instead of
reading byte-identical to a healthy retry. The recovery scan still re-drives
both, so publishing bounds nothing; it only makes an unbounded wait legible
to the operator who has to clear its cause.
A budget-terminated operation leaves its Session in the pending status
(`closing` or `deleting`) with the failure code on the operation row, and no
@ -252,13 +257,13 @@ Session and drains its own Runtime binding.
### 4.6 Operation fields
| Field | Values |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `status` | `pending` when admitted, `running` from the first claim, `completed` at the end, `failed` when the retry budget is spent (4.5). |
| `admission_stage` | `java_durable`; at completion `harness_confirmed` if the Harness that held the Session acknowledged its close. |
| `delivery_state` | `pending` between attempts, `leased` during one, `confirmed` at the terminal state, `blocked` while recovery-blocked. |
| `receipt_id` | An opaque `rcpt_` receipt that Java issues when the operation reaches a terminal state. |
| `failure_code` | The settle failure that spent the budget, or the code a `recovery_blocked` operation waits on: the workspace-close refusal, or `session_close_writer_live` while a Harness still holds the journal writer. |
| Field | Values |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `status` | `pending` when admitted, `running` from the first claim, `completed` at the end, `failed` when the retry budget is spent (4.5). |
| `admission_stage` | `java_durable`; at completion `harness_confirmed` if the Harness that held the Session acknowledged its close. |
| `delivery_state` | `pending` between attempts, `leased` during one, `confirmed` at the terminal state, `blocked` while recovery-blocked. |
| `receipt_id` | An opaque `rcpt_` receipt that Java issues when the operation reaches a terminal state. |
| `failure_code` | The settle failure that spent the budget, or the code a `recovery_blocked` operation waits on: the workspace-close refusal, `session_close_writer_live` while a Harness still holds the journal writer, or `hosted_harness_generation_mismatch` while this replica's view of the Harness boot is stale. |
The Harness that held the Session is the one whose boot ID the Session
recorded: Turn dispatch records the boot it attaches to, and a rename records

View file

@ -92,7 +92,7 @@ archive 只需要 Java 这一个权威,因此在同一个事务里完成:Ses
3. 排空该 Session 的 Runtime 绑定。内嵌的 Runtime Broker 只是不再预热该 Session,目前还没有 Harness 级别的回收。
4. 在一个锁住 Session 行的事务中,确认自己的认领仍然有效,完成 operation,把 Session 设为 `closed` 或墓碑,并追加 `session.closed` 或终态的 `session.deleted`。
失败的尝试让 operation 回到待定状态并计数,退避沿用 dispatch 设置:从 `dispatch.retry-initial-delay` 起翻倍,直到 `dispatch.retry-max-delay`。重试受 `dispatch.max-operation-retries`(默认 10 次)约束:预算用尽之前,operation 只有在各步骤成功之后才会完成,因此 `202` 从不表示工具已停止;Harness 调用持续失败时,operation 保持 `running`。预算是按尝试次数计量的,因此它对应的墙钟时长取决于故障应答的速度:在默认退避下,尝试立即失败(generation 错误立即拒绝、journal 受阻的 Harness 每次都立即应答 `503`)时历次认领跨越约五分钟,而每次尝试都耗满 Harness 请求超时时约为两倍。预算耗尽后,operation 仍会沿 settle() 的路由尽力释放 Runtime 绑定——对已关闭绑定 Session 的 delete 而言即不做任何调用——并以 `failed` 终结且带上 settle 的失败码:Session 保持其待定状态,不追加 `session.closed`,不为任何完成出具凭证。唯一的例外是存活的 journal writer,与 operation 受理时的 Session 状态无关——活动 Session 上 settle 等待 writer 的 close 或 delete,或已关闭 Session 上因残余 writer 持有 journal 而被 retention 清理拒绝的 delete:两者在该 writer 停止后都仍能成功,因此预算不会终结它,它会一直等到 writer 停止或其租约过期。等待期间的尝试不计入预算——只有本可推进的尝试才消耗预算——而预算耗尽后该行读作 `recovery_blocked` 并带 `session_close_writer_live` 失败码,因此卡死的 close 是可见的,而不是与健康重试逐字节相同。Hosted Harness 重启之后,Java 连接器仍沿用之前的 boot,因此它的调用会以 generation 错误失败,直到 Java 也重启为止,与 Turn 分发的情况相同;期间 operation 的等待同样不计入预算,之后还要等旧进程的 writer 租约过期。
失败的尝试让 operation 回到待定状态并计数,退避沿用 dispatch 设置:从 `dispatch.retry-initial-delay` 起翻倍,直到 `dispatch.retry-max-delay`。重试受 `dispatch.max-operation-retries`(默认 10 次)约束:预算用尽之前,operation 只有在各步骤成功之后才会完成,因此 `202` 从不表示工具已停止;Harness 调用持续失败时,operation 保持 `running`。预算是按尝试次数计量的,因此它对应的墙钟时长取决于故障应答的速度:在默认退避下,尝试立即失败(generation 错误立即拒绝、journal 受阻的 Harness 每次都立即应答 `503`)时历次认领跨越约五分钟,而每次尝试都耗满 Harness 请求超时时约为两倍。预算耗尽后,operation 仍会沿 settle() 的路由尽力释放 Runtime 绑定——对已关闭绑定 Session 的 delete 而言即不做任何调用——并以 `failed` 终结且带上 settle 的失败码:Session 保持其待定状态,不追加 `session.closed`,不为任何完成出具凭证。有两种情况会等待而不终结,因为两者都仍可能成功。其一是存活的 journal writer,与 operation 受理时的 Session 状态无关——活动 Session 上 settle 等待 writer 的 close 或 delete,或已关闭 Session 上因残余 writer 持有 journal 而被 retention 清理拒绝的 delete:该 writer 停止或其租约过期之后,障碍即消失。其二是 Java 自身对已重启 Harness 的陈旧视图:Hosted Harness 重启之后,Java 连接器仍沿用之前的 boot,因此它的调用会以 generation 错误失败,直到 Java 也重启为止,与 Turn 分发的情况相同;本副本刷新之后 close 仍能成功,因此期间 operation 持续等待,之后还要等旧进程的 writer 租约过期。两种等待都不只依据 journal——writer 等待还要求失败仍然可重试,因此在 Harness 尚未被要求停止之前就抛出的永久性拒绝会走到终态分支,而不是守着自己的 writer 租约空等。两种等待期间的尝试都不计入预算——只有本可推进的尝试才消耗预算——而预算耗尽后各自读作 `recovery_blocked` 并带自己的失败码,`session_close_writer_live` 或 `hosted_harness_generation_mismatch`,因此卡死的 close 是可见的,而不是与健康重试逐字节相同。恢复扫描仍会重新驱动这两类行,所以发布状态并不为其设定上界;它只是让一个无界等待对必须消除其成因的运维人员可读。
预算终结的 operation 会让其 Session 停留在待定状态(`closing` 或 `deleting`),失败码记录在 operation 行上,而目前没有任何路由允许在该状态上受理新的 operation:运维人员在消除失败码所指明的原因后手工对账该 Session(operation 行即审计记录);允许从 API 对其受理新的 close 或 delete 属于后续工作。
@ -104,13 +104,13 @@ delete 不动共享 Workspace:worker 只关闭该 Session 并排空它自己
### 4.6 operation 字段
| 字段 | 取值 |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `status` | 受理时为 `pending`,首次认领起为 `running`,结束时为 `completed`,重试预算耗尽时为 `failed`(4.5)。 |
| `admission_stage` | `java_durable`;完成时,如果持有该 Session 的 Harness 确认了关闭,则为 `harness_confirmed`。 |
| `delivery_state` | 两次尝试之间为 `pending`,尝试期间为 `leased`,到达终态时为 `confirmed`,恢复受阻时为 `blocked`。 |
| `receipt_id` | operation 到达终态时由 Java 签发的不透明回执 `rcpt_…`。 |
| `failure_code` | 耗尽预算的 settle 失败原因,或 `recovery_blocked` 的 operation 正在等待的失败码:workspace 关闭拒绝,或 Harness 仍持有 journal writer 时的 `session_close_writer_live`。 |
| 字段 | 取值 |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `status` | 受理时为 `pending`,首次认领起为 `running`,结束时为 `completed`,重试预算耗尽时为 `failed`(4.5)。 |
| `admission_stage` | `java_durable`;完成时,如果持有该 Session 的 Harness 确认了关闭,则为 `harness_confirmed`。 |
| `delivery_state` | 两次尝试之间为 `pending`,尝试期间为 `leased`,到达终态时为 `confirmed`,恢复受阻时为 `blocked`。 |
| `receipt_id` | operation 到达终态时由 Java 签发的不透明回执 `rcpt_…`。 |
| `failure_code` | 耗尽预算的 settle 失败原因,或 `recovery_blocked` 的 operation 正在等待的失败码:workspace 关闭拒绝、Harness 仍持有 journal writer 时的 `session_close_writer_live`,或本副本对 Harness boot 的视图已陈旧时的 `hosted_harness_generation_mismatch`。 |
持有该 Session 的 Harness,是该 Session 所记录的 boot ID 对应的那个 Harness:Turn 分发会记录它挂接的 boot,而 rename 只在尚未记录任何 boot 时才记录,因此只经过 rename 之后,记录可能是较旧的 boot,这时 close 虽已封存,仍报告为 `java_durable`。新的 archive、删除已关闭或已归档的 Session,以及关闭或删除从未被任何 Harness 持有的 Session,完成时都保持 `java_durable`:没有任何 Harness 确认过该 Session 的任何事。Harness 因重启被替换的 Session 同样如此:新的 Harness 回答它没有持有该 Session,operation 要等旧进程的 writer 租约过期之后才完成,而那个 writer 没有被封存。V17 之前受理、迁移过来的 archive 是在活动 Session 上受理的,因此它会先关闭该 Session,可能以 `harness_confirmed` 完成(4.10)。workspace 结算被 Runtime Broker 持续拒绝的 close 在预算耗尽前处于 `recovery_blocked`,`delivery_state` 为 `blocked`,`failure_code` 为拒绝的原因。不会产生 `cancelled` 状态。预算终结的 operation 读作 `failed` 并带 `failure_code`,因此放弃与已结算的 close 总是可区分的。

View file

@ -39,7 +39,9 @@ import org.springframework.stereotype.Component;
* wait is published as recovery_blocked with the
* session_close_writer_live code), and a generation error from
* Java's stale view of a restarted Harness (the close can still
* succeed once this replica refreshes). A cwd change is settled
* succeed once this replica refreshes; past the budget that wait
* is published as recovery_blocked with the
* hosted_harness_generation_mismatch code). A cwd change is settled
* without Harness or worker involvement — a read-only mount probe
* and one revision-CAS commit. A structural probe refusal or a
* moved fact is a terminal failure that is never retried; a
@ -196,7 +198,10 @@ public class SessionLifecycleCoordinator {
// generation error from Java's stale view of a restarted
// Harness (the close can still succeed once this replica
// refreshes). Both reschedule through the budget-exempt
// baseline.
// baseline, and past the budget both are published as
// recovery_blocked with their own code: an unbounded wait has
// to stay visible, because neither row is otherwise
// distinguishable from a healthy pending retry.
// A cwd change never reaches settle(): the close-specific
// waits (live writer, stale boot) and the terminal budget arm
// are not its semantics — its rethrown probe refusal takes the
@ -303,6 +308,17 @@ public class SessionLifecycleCoordinator {
operationId, owner, claimed.claimGeneration(),
"session_close_writer_live",
Math.addExact(clock.millis(), delay), true);
} else if (valid.get() && !cwdChange && staleBoot
&& claimed.attemptCount() >= maxOperationRetries) {
// The stale-view wait is unbounded like the writer wait —
// only a Java restart clears it — so past the budget it is
// published the same way rather than reading as a healthy
// pending retry. The recovery scan still re-drives it, so
// the wait itself stays unbounded.
store.blockLifecycleOperation(tenantId, sessionId,
operationId, owner, claimed.claimGeneration(),
"hosted_harness_generation_mismatch",
Math.addExact(clock.millis(), delay), true);
} else if (valid.get() && (writerLive || staleBoot)) {
store.retryOperation(tenantId, sessionId, operationId, owner,
claimed.claimGeneration(),

View file

@ -6469,7 +6469,7 @@
"type": "string",
"minLength": 1,
"maxLength": 128,
"description": "Reason a durable response or task cancellation failed. Action response failures come from the original committed Action, a definitive invalid response, or a delivery that exhausted its retry budget (action_response_delivery_failed). Workspace close recovery_blocked reports original resource identity or unsettled execution, or session_close_writer_live while a Harness still holds the journal writer; the Session remains closing. A lifecycle operation that exhausted its retry budget carries the settle's failure (session_lifecycle_delivery_failed or the workspace close refusal)."
"description": "Reason a durable response or task cancellation failed. Action response failures come from the original committed Action, a definitive invalid response, or a delivery that exhausted its retry budget (action_response_delivery_failed). Workspace close recovery_blocked reports original resource identity or unsettled execution, session_close_writer_live while a Harness still holds the journal writer, or hosted_harness_generation_mismatch while this replica's view of the Harness boot is stale; the Session remains closing. A lifecycle operation that exhausted its retry budget carries the settle's failure (session_lifecycle_delivery_failed or the workspace close refusal)."
},
"replayed": {
"type": "boolean"
@ -6855,7 +6855,7 @@
"type": "string",
"minLength": 1,
"maxLength": 128,
"description": "Reason a durable response or task cancellation failed. Action response failures come from the original committed Action, a definitive invalid response, or a delivery that exhausted its retry budget (action_response_delivery_failed). Workspace close recovery_blocked reports original resource identity or unsettled execution, or session_close_writer_live while a Harness still holds the journal writer; the Session remains closing. A lifecycle operation that exhausted its retry budget carries the settle's failure (session_lifecycle_delivery_failed or the workspace close refusal)."
"description": "Reason a durable response or task cancellation failed. Action response failures come from the original committed Action, a definitive invalid response, or a delivery that exhausted its retry budget (action_response_delivery_failed). Workspace close recovery_blocked reports original resource identity or unsettled execution, session_close_writer_live while a Harness still holds the journal writer, or hosted_harness_generation_mismatch while this replica's view of the Harness boot is stale; the Session remains closing. A lifecycle operation that exhausted its retry budget carries the settle's failure (session_lifecycle_delivery_failed or the workspace close refusal)."
},
"replayed": {
"type": "boolean"

View file

@ -148,9 +148,11 @@ class OperationRetryTerminalStateTest {
// generation error until Java restarts too. That failure is this
// replica's stale view, not the close's own failure: the operation keeps
// waiting past the budget instead of recording an unrecoverable
// terminal (review round 6, R6-3).
// terminal (review round 6, R6-3) — but the wait is published, because
// an unbounded wait that reads as a healthy pending retry is invisible
// to the operator who has to restart this replica.
@Test
void aStaleHarnessViewKeepsTheOperationWaitingPastTheBudget() {
void aStaleHarnessViewPublishesItsWaitPastTheBudget() {
AgentStateStore store = mock(AgentStateStore.class);
ManagedSessionStore sessionStore = mock(ManagedSessionStore.class);
HarnessConnector harness = mock(HarnessConnector.class);
@ -167,6 +169,52 @@ class OperationRetryTerminalStateTest {
when(sessionStore.hasLiveWriter("tenant", "session"))
.thenReturn(false);
SessionLifecycleCoordinator coordinator =
new SessionLifecycleCoordinator(store, sessionStore, harness,
mock(RuntimeWarmer.class),
CoordinatorTestSupport.directExecutor(),
Clock.systemUTC(), new ManagedAgentProperties());
try {
coordinator.dispatch("tenant", "session", "op-close");
// Past the budget the stale-view wait is published like the
// writer wait: recovery_blocked with its own code, still
// budget-exempt so the wait itself stays unbounded.
verify(store).blockLifecycleOperation(eq("tenant"), eq("session"),
eq("op-close"), anyString(), eq(1L),
eq("hosted_harness_generation_mismatch"), anyLong(),
eq(true));
verify(store, never()).failOperation(anyString(), anyString(),
anyString(), anyString(), anyLong(), anyString());
verify(store, never()).retryOperation(anyString(), anyString(),
anyString(), anyString(), anyLong(), anyLong(),
anyBoolean());
} finally {
coordinator.stopRenewals();
}
}
// Below the budget the same stale view is still a plain budget-exempt
// reschedule: publication is what spending the budget buys, not the
// first response to a generation error.
@Test
void aStaleHarnessViewReschedulesExemptlyBelowTheBudget() {
AgentStateStore store = mock(AgentStateStore.class);
ManagedSessionStore sessionStore = mock(ManagedSessionStore.class);
HarnessConnector harness = mock(HarnessConnector.class);
OperationRecord claimed = lifecycleOperation(9);
when(store.claimOperation(eq("tenant"), eq("session"),
eq("op-close"), anyString(), any(Duration.class)))
.thenReturn(Optional.of(claimed));
when(store.requireSession("tenant", "session")).thenReturn(
new SessionRecord("tenant", "session", "qwen-code", null,
"ACTIVE", "boot-1", null, 0, 0, 1, 1, null, 1));
when(harness.isAvailable()).thenReturn(true);
when(harness.closeSession("tenant", "session")).thenThrow(
mock(HostedHarnessGenerationException.class));
when(sessionStore.hasLiveWriter("tenant", "session"))
.thenReturn(false);
SessionLifecycleCoordinator coordinator =
new SessionLifecycleCoordinator(store, sessionStore, harness,
mock(RuntimeWarmer.class),
@ -178,6 +226,9 @@ class OperationRetryTerminalStateTest {
verify(store).retryOperation(eq("tenant"), eq("session"),
eq("op-close"), anyString(), eq(1L), anyLong(),
eq(true));
verify(store, never()).blockLifecycleOperation(anyString(),
anyString(), anyString(), anyString(), anyLong(),
anyString(), anyLong(), anyBoolean());
verify(store, never()).failOperation(anyString(), anyString(),
anyString(), anyString(), anyLong(), anyString());
} finally {

View file

@ -767,7 +767,7 @@ export interface components {
deliveryState: "pending" | "leased" | "confirmed" | "blocked";
receiptId?: string;
actionResolution?: components["schemas"]["WebShellActionResolution"];
/** @description Reason a durable response or task cancellation failed. Action response failures come from the original committed Action, a definitive invalid response, or a delivery that exhausted its retry budget (action_response_delivery_failed). Workspace close recovery_blocked reports original resource identity or unsettled execution, or session_close_writer_live while a Harness still holds the journal writer; the Session remains closing. A lifecycle operation that exhausted its retry budget carries the settle's failure (session_lifecycle_delivery_failed or the workspace close refusal). */
/** @description Reason a durable response or task cancellation failed. Action response failures come from the original committed Action, a definitive invalid response, or a delivery that exhausted its retry budget (action_response_delivery_failed). Workspace close recovery_blocked reports original resource identity or unsettled execution, session_close_writer_live while a Harness still holds the journal writer, or hosted_harness_generation_mismatch while this replica's view of the Harness boot is stale; the Session remains closing. A lifecycle operation that exhausted its retry budget carries the settle's failure (session_lifecycle_delivery_failed or the workspace close refusal). */
failureCode?: string;
replayed: boolean;
} & (unknown & unknown & unknown & unknown);