From 9ef35af42ed336ea8a903728ecc4bb156b02c604 Mon Sep 17 00:00:00 2001 From: Shaojin Wen Date: Sun, 18 Oct 2026 16:43:20 +0800 Subject: [PATCH] fix(ci): classify host lock starvation as infrastructure (#14550) (#14552) The main CI watcher and the PR scan share one infrastructure-signature list. A per-host SDK Java test lock starving past its flock budget carried no signature, so main run 39104863059 filed #14550 for a lane in which no test ever ran. The annotation names a lock budget, not an assertion: several runner instances share one HOME on the ECS pool, so a fast-tier job can wait behind sibling Maven holds that outlast its own acquisition budget. Main runs now rerun that class once instead of filing, and a red check on a PR is reran once; the existing attempt-2 guard still files when the starvation is not transient. The new pattern also covers the E2E and release docker lock messages, which share the same starvation semantics. --- .github/workflows/main-ci-failure-issue.yml | 2 +- .github/workflows/qwen-autofix.md | 4 +++- .github/workflows/qwen-autofix.yml | 8 +++++--- scripts/tests/main-ci-failure-issue-workflow.test.js | 5 +++++ scripts/tests/qwen-autofix-workflow.test.js | 8 ++++++++ 5 files changed, 22 insertions(+), 5 deletions(-) diff --git a/.github/workflows/main-ci-failure-issue.yml b/.github/workflows/main-ci-failure-issue.yml index b02ae8c837..9cd7ad4914 100644 --- a/.github/workflows/main-ci-failure-issue.yml +++ b/.github/workflows/main-ci-failure-issue.yml @@ -132,7 +132,7 @@ jobs: # test pins the equality. There the same annotation class auto-reruns # a PR's red check; here it suppresses the per-commit issue. # Full rationale → qwen-autofix.md#af-076 - INFRA_FAILURE_SIGNATURES: 'lost communication with the server|No space left on device|ENOSPC|received a shutdown signal|The runner has received|Failed to initialize container|runner (was|has been) (lost|terminated)|invalid index-pack output|RPC failed|failed to be acquired' + INFRA_FAILURE_SIGNATURES: 'lost communication with the server|No space left on device|ENOSPC|received a shutdown signal|The runner has received|Failed to initialize container|runner (was|has been) (lost|terminated)|invalid index-pack output|RPC failed|failed to be acquired|lock not acquired within' run: |- jobs_json="${RUNNER_TEMP}/run-jobs.json" diff --git a/.github/workflows/qwen-autofix.md b/.github/workflows/qwen-autofix.md index fe9b1c15b7..12266ce519 100644 --- a/.github/workflows/qwen-autofix.md +++ b/.github/workflows/qwen-autofix.md @@ -2355,7 +2355,9 @@ the PR and clears on a re-run (observed: #7490's E2E "runner lost communication"; #6506's checkout "RPC failed; curl 92" / "fetch-pack: invalid index-pack output" — both green on the rerun; #13622's Classify PR, "repeatedly failed to be acquired" — the hosted pool never assigned a -runner, so zero steps executed). The scan +runner, so zero steps executed; #14550's SDK Java lane, "sdk-java host +lock not acquired within 40 minutes" — the per-host test lock starved +behind cross-run pool traffic and no test ran). The scan auto-reruns those failed jobs ONCE, guarded by run_attempt so a persistent infra problem cannot loop. Deliberately conservative — only unambiguous machine/transport failures, never a bare test-level timeout, which could be diff --git a/.github/workflows/qwen-autofix.yml b/.github/workflows/qwen-autofix.yml index 6055d18157..c9eb24947d 100644 --- a/.github/workflows/qwen-autofix.yml +++ b/.github/workflows/qwen-autofix.yml @@ -147,10 +147,12 @@ env: NON_BLOCKING_CHECKS: '["review-pr"]' # Failed-check annotation patterns that mean the INFRASTRUCTURE died, not the # code — a self-hosted runner losing the server, the disk filling, a runner - # shutdown, a git fetch/clone dying mid-transfer, or the hosted pool never - # handing the job a runner (#13622: "repeatedly failed to be acquired"). + # shutdown, a git fetch/clone dying mid-transfer, the hosted pool never + # handing the job a runner (#13622: "repeatedly failed to be acquired"), or a + # shared-host test lock starving past its per-acquisition budget (#14550: + # "sdk-java host lock not acquired within 40 minutes"). # Full rationale → qwen-autofix.md#af-076 - INFRA_FAILURE_SIGNATURES: 'lost communication with the server|No space left on device|ENOSPC|received a shutdown signal|The runner has received|Failed to initialize container|runner (was|has been) (lost|terminated)|invalid index-pack output|RPC failed|failed to be acquired' + INFRA_FAILURE_SIGNATURES: 'lost communication with the server|No space left on device|ENOSPC|received a shutdown signal|The runner has received|Failed to initialize container|runner (was|has been) (lost|terminated)|invalid index-pack output|RPC failed|failed to be acquired|lock not acquired within' # Upper bound on review targets emitted per scan (fan-out defense-in-depth; # excess is logged and deferred to the next scan). # Full rationale → qwen-autofix.md#af-077 diff --git a/scripts/tests/main-ci-failure-issue-workflow.test.js b/scripts/tests/main-ci-failure-issue-workflow.test.js index d2278bd015..ed12c3e51e 100644 --- a/scripts/tests/main-ci-failure-issue-workflow.test.js +++ b/scripts/tests/main-ci-failure-issue-workflow.test.js @@ -305,6 +305,11 @@ describe('main CI failure issue workflow', () => { expect(infraStep.env.INFRA_FAILURE_SIGNATURES).toContain( 'lost communication with the server', ); + // A per-host lock starvation (#14550) is the same class: the annotation + // names a lock budget, not an assertion. + expect(infraStep.env.INFRA_FAILURE_SIGNATURES).toContain( + 'lock not acquired within', + ); }); it('classifies per failed job and only when EVERY one is infrastructure', () => { diff --git a/scripts/tests/qwen-autofix-workflow.test.js b/scripts/tests/qwen-autofix-workflow.test.js index 8db4e29e67..c80575bf68 100644 --- a/scripts/tests/qwen-autofix-workflow.test.js +++ b/scripts/tests/qwen-autofix-workflow.test.js @@ -1320,6 +1320,14 @@ describe('qwen-autofix workflow', () => { 'The self-hosted runner lost communication with the server', }), ).toEqual({ reran: true, continued: true }); + // A per-host test lock starving past its budget (#14550) is capacity + // contention on the shared pool, not the PR's change: no test ran. + expect( + run({ + checks: [FAIL], + annotations: 'sdk-java host lock not acquired within 40 minutes', + }), + ).toEqual({ reran: true, continued: true }); // A REAL failure (no infra signature in the annotation) → never rerun; the // agent/human handles it. This is the gate that stops masking real bugs. expect(