fix(ci): classify host lock starvation as infrastructure (#14550) (#14552)

The main CI watcher and the PR scan share one infrastructure-signature
list. A per-host SDK Java test lock starving past its flock budget carried
no signature, so main run 39104863059 filed #14550 for a lane in which no
test ever ran.

The annotation names a lock budget, not an assertion: several runner
instances share one HOME on the ECS pool, so a fast-tier job can wait
behind sibling Maven holds that outlast its own acquisition budget. Main
runs now rerun that class once instead of filing, and a red check on a PR
is reran once; the existing attempt-2 guard still files when the
starvation is not transient.

The new pattern also covers the E2E and release docker lock messages,
which share the same starvation semantics.
This commit is contained in:
Shaojin Wen 2026-10-18 16:43:20 +08:00 • committed by GitHub
parent 00f7093a71
commit 9ef35af42e
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 22 additions and 5 deletions

View file

@ -132,7 +132,7 @@ jobs:
# test pins the equality. There the same annotation class auto-reruns
# a PR's red check; here it suppresses the per-commit issue.
# Full rationale → qwen-autofix.md#af-076
INFRA_FAILURE_SIGNATURES: 'lost communication with the server|No space left on device|ENOSPC|received a shutdown signal|The runner has received|Failed to initialize container|runner (was|has been) (lost|terminated)|invalid index-pack output|RPC failed|failed to be acquired'
INFRA_FAILURE_SIGNATURES: 'lost communication with the server|No space left on device|ENOSPC|received a shutdown signal|The runner has received|Failed to initialize container|runner (was|has been) (lost|terminated)|invalid index-pack output|RPC failed|failed to be acquired|lock not acquired within'
run: |-
jobs_json="${RUNNER_TEMP}/run-jobs.json"

View file

@ -2355,7 +2355,9 @@ the PR and clears on a re-run (observed: #7490's E2E "runner lost
communication"; #6506's checkout "RPC failed; curl 92" / "fetch-pack:
invalid index-pack output" — both green on the rerun; #13622's Classify
PR, "repeatedly failed to be acquired" — the hosted pool never assigned a
runner, so zero steps executed). The scan
runner, so zero steps executed; #14550's SDK Java lane, "sdk-java host
lock not acquired within 40 minutes" — the per-host test lock starved
behind cross-run pool traffic and no test ran). The scan
auto-reruns those failed jobs ONCE, guarded by run_attempt so a persistent
infra problem cannot loop. Deliberately conservative — only unambiguous
machine/transport failures, never a bare test-level timeout, which could be

View file

@ -147,10 +147,12 @@ env:
NON_BLOCKING_CHECKS: '["review-pr"]'
# Failed-check annotation patterns that mean the INFRASTRUCTURE died, not the
# code — a self-hosted runner losing the server, the disk filling, a runner
# shutdown, a git fetch/clone dying mid-transfer, or the hosted pool never
# handing the job a runner (#13622: "repeatedly failed to be acquired").
# shutdown, a git fetch/clone dying mid-transfer, the hosted pool never
# handing the job a runner (#13622: "repeatedly failed to be acquired"), or a
# shared-host test lock starving past its per-acquisition budget (#14550:
# "sdk-java host lock not acquired within 40 minutes").
# Full rationale → qwen-autofix.md#af-076
INFRA_FAILURE_SIGNATURES: 'lost communication with the server|No space left on device|ENOSPC|received a shutdown signal|The runner has received|Failed to initialize container|runner (was|has been) (lost|terminated)|invalid index-pack output|RPC failed|failed to be acquired'
INFRA_FAILURE_SIGNATURES: 'lost communication with the server|No space left on device|ENOSPC|received a shutdown signal|The runner has received|Failed to initialize container|runner (was|has been) (lost|terminated)|invalid index-pack output|RPC failed|failed to be acquired|lock not acquired within'
# Upper bound on review targets emitted per scan (fan-out defense-in-depth;
# excess is logged and deferred to the next scan).
# Full rationale → qwen-autofix.md#af-077

View file

@ -305,6 +305,11 @@ describe('main CI failure issue workflow', () => {
expect(infraStep.env.INFRA_FAILURE_SIGNATURES).toContain(
'lost communication with the server',
);
// A per-host lock starvation (#14550) is the same class: the annotation
// names a lock budget, not an assertion.
expect(infraStep.env.INFRA_FAILURE_SIGNATURES).toContain(
'lock not acquired within',
);
});
it('classifies per failed job and only when EVERY one is infrastructure', () => {

View file

@ -1320,6 +1320,14 @@ describe('qwen-autofix workflow', () => {
'The self-hosted runner lost communication with the server',
}),
).toEqual({ reran: true, continued: true });
// A per-host test lock starving past its budget (#14550) is capacity
// contention on the shared pool, not the PR's change: no test ran.
expect(
run({
checks: [FAIL],
annotations: 'sdk-java host lock not acquired within 40 minutes',
}),
).toEqual({ reran: true, continued: true });
// A REAL failure (no infra signature in the annotation) → never rerun; the
// agent/human handles it. This is the gate that stops masking real bugs.
expect(