mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-10-24 15:59:48 +00:00
6c3d84d361
* fix(cua): sign Windows UIAccess release workers * fix(cua): select imported code-signing certificate correctly * fix(ci): keep CUA signing within workflow size limit * fix(cua): isolate dry-run signing credentials Keep production certificate secrets out of branch dry runs and fall back to PowerShell 7 only when the legacy signature module cannot load. * fix(cua): select the release signing certificate by EKU extension The production signing path filtered on `$_.EnhancedKeyUsageList.ObjectId`, but `EnhancedKeyUsageList` elements are Certificate-provider display strings and `Oid` exposes only FriendlyName and Value. Neither type has an `ObjectId` member, so the filter matched nothing and every real release threw on a correctly configured PFX. Match the raw EKU extension instead; that route does not depend on provider script properties being attached to the object `Import-PfxCertificate` returns. Verified by execution on PowerShell 7.4.6 / .NET 8 against openssl-generated PFX fixtures: the old predicate selects 0 for a genuine code-signing cert, while the replacement selects it and rejects a serverAuth-only cert. Also narrow the installer's powershell-to-pwsh retry to the non-localized `CouldNotAutoloadMatchingModule` error id observed on the windows-2025 runner during the release dry run. The previous `Microsoft.PowerShell.Security` substring also appears in ordinary Get-AuthenticodeSignature errors, where the retry hid the real result behind a shell users usually do not have. Replace the test assertion that pinned the dead filter with assertions on the EKU-extension mechanism, and forbid the removed member access. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(cua): fail closed on half-configured cert pairs and clean up the imported key Review follow-ups on the Windows UIAccess signing path: - sign-cua-windows-worker.ps1: throw on a half-configured WINDOWS_CERTIFICATE / WIN_CSC_LINK pair (mirroring desktop-release.yml) instead of silently falling through to the other identity, and remove the imported leaf certificate from Cert:\CurrentUser\My in a finally after the sign + verify lines, so the key is not resident while the later scripts-enabled clean-install step runs under the same user. - install-native.mjs: chain the original Windows PowerShell 5.1 CouldNotAutoloadMatchingModule diagnostic into the pwsh retry failure (the entry renderer prints only error.message), and map EPERM/EACCES on the deploy mkdir to an elevation message naming the elevated install.ps1 alternative. installUiAccessWorker is exported with the signature check and mkdir injectable so both are testable off Windows. - computer-use SKILL.md / user guide: state the Windows elevation precondition where the install actually runs, with the elevated install.ps1 alternative and the already-installed short-circuit. - release-workflow.test.js: pin the new guards, the whole clean-install signature verdict (comparison and throw, not just the cmdlet token), the Remove-Item-before-cargo-build ordering the `Valid` gate now depends on, and the retry guard's polarity — replacing the source-text ban on the Microsoft.PowerShell.Security module name that a deeper 5.1 fix needs. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(cua): integrate SignPath signing and verify opt-in UIAccess install * fix(cua): clarify UIAccess recovery and narrow signing scope * fix(cua): test-sign the Windows worker through the SignPath API The SignPath GitHub connector needs the SignPath GitHub App, which this repository cannot install. Branch dry runs now submit the UIAccess worker directly to the SignPath REST API with the configured token, verify the test certificate, and install the signed worker. Release builds keep shipping the unsigned worker so ordinary-mode Windows releases never depend on a production certificate. * fix(cua): supply the browser-use tarball to the Windows clean-install step The clean-install step ran `npm install` with only the SDK tarball, so npm had to resolve the SDK's exact-pinned `@qwen-code/browser-use@<version>` from the public registry. `validate-version` forces that pin to equal the release version, and the only publisher of it is `publish-browser`, which needs `release`, which needs `build-windows`. The step carries no `if:`, so tag releases run it too: cutting `cua-driver-rs-v0.27.0` would fail both Windows legs with ETARGET before `upload-artifact` and skip every downstream publish job, and a re-run fails identically because the only publisher sits behind the job that just failed. It stays invisible today only because `0.26.0` already happens to be on npm. Mirror what `verify-sdk-package` already does in this same file: depend on `verify-browser-package` (acyclic, it needs only `validate-version`), download its `browser-use-npm-<version>` artifact, and pass both tarballs to one `npm install`. Pin the fix with two assertions that go red if the sibling tarball or the job edge is dropped. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> Patrol-Run: qwen-pr-closeout/jmvl4fms0ro --------- Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
3590 lines
149 KiB
JavaScript
3590 lines
149 KiB
JavaScript
/**
|
|
* @license
|
|
* Copyright 2026 Qwen Team
|
|
* SPDX-License-Identifier: Apache-2.0
|
|
*/
|
|
|
|
import { spawnSync } from 'node:child_process';
|
|
import {
|
|
chmodSync,
|
|
existsSync,
|
|
lstatSync,
|
|
mkdirSync,
|
|
mkdtempSync,
|
|
readdirSync,
|
|
readFileSync,
|
|
realpathSync,
|
|
rmSync,
|
|
symlinkSync,
|
|
writeFileSync,
|
|
} from 'node:fs';
|
|
import { tmpdir } from 'node:os';
|
|
import { dirname, join } from 'node:path';
|
|
import { globSync } from 'glob';
|
|
import { describe, expect, it } from 'vitest';
|
|
import { parse } from 'yaml';
|
|
import { PUBLISHED_PACKAGES } from '../assert-release-version.mjs';
|
|
import { RELEASE_WORKSPACES } from '../release-packages.mjs';
|
|
import { isAlreadyPublishedDryRunFailure } from '../../packages/cua-driver/scripts/publish-dry-run-skip.mjs';
|
|
import { getTestCiWorkspacePackageJsonPaths } from '../workspaces.js';
|
|
|
|
// `realpath -m` (the script's canonicalization line) is a GNU coreutils
|
|
// extension. Probe the host before asserting GNU-specific path behavior.
|
|
const hasGnuRealpath =
|
|
spawnSync('realpath', ['-m', '--', '/'], { stdio: 'ignore' }).status === 0 &&
|
|
spawnSync('realpath', ['-m', '-s', '--', '/'], { stdio: 'ignore' }).status ===
|
|
0;
|
|
|
|
const workflow = readFileSync('.github/workflows/release.yml', 'utf8');
|
|
const audioPrebuildsWorkflow = readFileSync(
|
|
'.github/workflows/audio-capture-prebuilds.yml',
|
|
'utf8',
|
|
);
|
|
const audioPrebuildsYaml = parse(audioPrebuildsWorkflow);
|
|
const releaseYaml = parse(workflow);
|
|
const releaseStepScriptPath = '.github/scripts/run-release-step.sh';
|
|
const releaseStepScriptAbsolutePath = join(
|
|
process.cwd(),
|
|
releaseStepScriptPath,
|
|
);
|
|
const releaseStepScript = readFileSync(releaseStepScriptPath, 'utf8');
|
|
const workspaceTestScriptPath =
|
|
'.github/scripts/run-release-workspace-tests.sh';
|
|
const workspaceTestScript = readFileSync(workspaceTestScriptPath, 'utf8');
|
|
const dockerIntegrationScript = readFileSync(
|
|
'.github/scripts/run-release-docker-integration.sh',
|
|
'utf8',
|
|
);
|
|
const cuaReleaseWorkflow = readFileSync(
|
|
'.github/workflows/cd-cua-driver.yml',
|
|
'utf8',
|
|
);
|
|
const cuaReleaseJobs = parse(cuaReleaseWorkflow).jobs;
|
|
const nodeReplPackage = JSON.parse(
|
|
readFileSync('packages/node-repl/package.json', 'utf8'),
|
|
);
|
|
const cuaSdkPackage = JSON.parse(
|
|
readFileSync('packages/cua-driver/typescript/package.json', 'utf8'),
|
|
);
|
|
const cuaSdkInstallScript = readFileSync(
|
|
'packages/cua-driver/typescript/scripts/install-native.mjs',
|
|
'utf8',
|
|
);
|
|
const testSignWorkerScript = readFileSync(
|
|
'.github/scripts/test-sign-cua-windows-worker.ps1',
|
|
'utf8',
|
|
);
|
|
const browserUsePackage = JSON.parse(
|
|
readFileSync('packages/browser-use/package.json', 'utf8'),
|
|
);
|
|
const workspaceLock = parse(readFileSync('pnpm-lock.yaml', 'utf8'));
|
|
const cuaInstallScript = readFileSync(
|
|
'packages/cua-driver/scripts/install.sh',
|
|
'utf8',
|
|
);
|
|
const computerUseGuide = readFileSync(
|
|
'docs/users/features/computer-use.md',
|
|
'utf8',
|
|
);
|
|
const desktopReleaseWorkflow = readFileSync(
|
|
'.github/workflows/desktop-release.yml',
|
|
'utf8',
|
|
);
|
|
const liveHostInstaller = readFileSync(
|
|
'packages/cli/src/serve/live/live-host-installer.ts',
|
|
'utf8',
|
|
);
|
|
const liveHostCiWorkflow = readFileSync(
|
|
'.github/workflows/live-host.yml',
|
|
'utf8',
|
|
);
|
|
const liveHostReleaseWorkflow = readFileSync(
|
|
'.github/workflows/live-host-release.yml',
|
|
'utf8',
|
|
);
|
|
const liveHostOssWorkflow = readFileSync(
|
|
'.github/workflows/sync-live-host-to-oss.yml',
|
|
'utf8',
|
|
);
|
|
|
|
describe('CUA release workflow', () => {
|
|
it('excludes the Computer Use distribution from the CLI release', () => {
|
|
for (const name of [
|
|
'@qwen-code/browser-use',
|
|
'@qwen-code/cua-sdk',
|
|
'@qwen-code/node-repl-mcp',
|
|
]) {
|
|
expect(RELEASE_WORKSPACES).not.toContain(name);
|
|
}
|
|
});
|
|
|
|
it('keeps the Computer Use distribution on one version', () => {
|
|
expect(nodeReplPackage.name).toBe('@qwen-code/node-repl-mcp');
|
|
expect(nodeReplPackage.version).toBe(cuaSdkPackage.version);
|
|
expect(browserUsePackage.version).toBe(cuaSdkPackage.version);
|
|
expect(nodeReplPackage.dependencies['@qwen-code/cua-sdk']).toBe(
|
|
cuaSdkPackage.version,
|
|
);
|
|
expect(cuaSdkPackage.dependencies['@qwen-code/browser-use']).toBe(
|
|
cuaSdkPackage.version,
|
|
);
|
|
expect(cuaReleaseWorkflow).toContain(
|
|
"node_repl_version: '${{ steps.release.outputs.node_repl_version }}'",
|
|
);
|
|
expect(cuaReleaseWorkflow).not.toContain(
|
|
'NODE_REPL_VERSION does not match release version',
|
|
);
|
|
expect(
|
|
workspaceLock.importers['packages/cua-driver/typescript'],
|
|
).toBeDefined();
|
|
});
|
|
|
|
it('dry-runs and clean-installs the packed Node REPL MCP server', () => {
|
|
expect(cuaReleaseWorkflow).toMatch(
|
|
/verify-node-repl-package:[\s\S]*?pnpm install --frozen-lockfile --ignore-scripts[\s\S]*?npm run typecheck[\s\S]*?npm test[\s\S]*?npm run smoke:mcp[\s\S]*?npm run smoke:lifecycle[\s\S]*?node packages\/node-repl\/scripts\/verify-package\.mjs[\s\S]*?node-repl-mcp-npm-\$\{\{[\s\S]*?node_repl_version/,
|
|
);
|
|
});
|
|
|
|
it('publishes the verified Node REPL tarball immutably with provenance', () => {
|
|
const job = cuaReleaseJobs['publish-node-repl'];
|
|
expect(job.needs).toContain('publish-sdk');
|
|
const commands = job.steps.map((step) => step.run ?? '').join('\n');
|
|
expect(commands).toMatch(
|
|
/npm view "@qwen-code\/node-repl-mcp@\$\{VERSION\}" dist\.integrity[\s\S]*?npm publish "\$TARBALL" --provenance --access public --tag "\$NPM_TAG"/,
|
|
);
|
|
expect(
|
|
job.steps.some((step) => step.name === 'Verify npm registry integrity'),
|
|
).toBe(true);
|
|
});
|
|
|
|
it('keeps installer version changes in the feature PR', () => {
|
|
expect(cuaReleaseWorkflow).not.toContain('sync-installer-version:');
|
|
expect(cuaReleaseWorkflow).not.toContain('gh pr create');
|
|
expect(cuaReleaseWorkflow).not.toContain('gh pr merge');
|
|
expect(cuaInstallScript).toContain(
|
|
`CUA_DRIVER_RS_VERSION=${cuaSdkPackage.version}`,
|
|
);
|
|
expect(cuaInstallScript).toContain(
|
|
`CUA_DRIVER_VERSION=${cuaSdkPackage.version}`,
|
|
);
|
|
expect(cuaReleaseWorkflow).toContain(
|
|
'INSTALL_ENTRYPOINT_RS_VERSION=$(sed -nE',
|
|
);
|
|
});
|
|
|
|
it('ships the Windows UIAccess worker for target-machine signing', () => {
|
|
expect(cuaReleaseWorkflow).toMatch(
|
|
/Build \(release\)[\s\S]*?Remove-Item[^\n]*cua-driver-uia\.exe[^\n]*\n[^\n]*cargo build[\s\S]*?Verify unsigned UIAccess worker/,
|
|
);
|
|
expect(cuaReleaseWorkflow).toMatch(
|
|
/Build \(release\)[\s\S]*?Verify unsigned UIAccess worker[\s\S]*?Get-AuthenticodeSignature[\s\S]*?NotSigned[\s\S]*?qwen-cua-driver-uia\.exe[\s\S]*?release artifact contract/,
|
|
);
|
|
expect(cuaReleaseWorkflow).not.toMatch(
|
|
/WINDOWS_CERTIFICATE|WIN_CSC_LINK|New-SelfSignedCertificate/,
|
|
);
|
|
expect(cuaReleaseWorkflow).toContain(
|
|
'- **Windows**: unsigned UIAccess worker + native SDK payload',
|
|
);
|
|
});
|
|
|
|
it('test-signs the Windows worker only for branch dry runs', () => {
|
|
const job = cuaReleaseJobs['build-windows'];
|
|
const steps = job.steps;
|
|
const index = (name) => steps.findIndex((step) => step.name === name);
|
|
const verifyIndex = index('Verify unsigned UIAccess worker');
|
|
const signIndex = index('Test-sign UIAccess worker (dry run)');
|
|
const packageIndex = index('Package');
|
|
const installIndex = index(
|
|
'Clean-install Windows SDK from packaged payload',
|
|
);
|
|
const uploadIndex = steps.findIndex(
|
|
(step) => step.uses === 'actions/upload-artifact@v4',
|
|
);
|
|
|
|
expect(job.permissions).toEqual({ contents: 'read' });
|
|
expect(job.env.CUA_TEST_SIGNING).toBe(
|
|
"${{ github.event_name == 'workflow_dispatch' && inputs.dry_run == true && !startsWith(github.ref, 'refs/tags/') }}",
|
|
);
|
|
expect(cuaReleaseJobs.release.if).toContain(
|
|
"startsWith(github.ref, 'refs/tags/cua-driver-rs-v')",
|
|
);
|
|
expect(verifyIndex).toBeGreaterThan(-1);
|
|
expect(signIndex).toBeGreaterThan(verifyIndex);
|
|
expect(packageIndex).toBeGreaterThan(signIndex);
|
|
expect(installIndex).toBeGreaterThan(packageIndex);
|
|
expect(uploadIndex).toBeGreaterThan(installIndex);
|
|
|
|
const signStep = steps[signIndex];
|
|
expect(signStep.if).toBe("env.CUA_TEST_SIGNING == 'true'");
|
|
expect(signStep.env.SIGNPATH_API_TOKEN).toBe(
|
|
'${{ secrets.SIGNPATH_API_TOKEN }}',
|
|
);
|
|
expect(signStep.run).toContain(
|
|
'.github/scripts/test-sign-cua-windows-worker.ps1',
|
|
);
|
|
expect(cuaReleaseWorkflow).not.toContain(
|
|
'github-action-submit-signing-request',
|
|
);
|
|
expect(testSignWorkerScript).toContain(
|
|
'/SigningRequests/SubmitWithArtifact',
|
|
);
|
|
expect(testSignWorkerScript).toContain(
|
|
"signingPolicySlug = 'test-signing'",
|
|
);
|
|
expect(testSignWorkerScript).toContain(
|
|
"-ne '0B27B4BA9D646BCDB1FD59B4F229C0DAA7E8EEF0'",
|
|
);
|
|
expect(testSignWorkerScript).toMatch(
|
|
/Thumbprint -ne[\s\S]*?Cert:\\LocalMachine\\Root[\s\S]*?\.Status -ne 'Valid'[\s\S]*?Copy-Item/,
|
|
);
|
|
|
|
const install = steps[installIndex];
|
|
expect(install.env.CUA_DRIVER_USE_UIACCESS).toBe(
|
|
"${{ env.CUA_TEST_SIGNING == 'true' && '1' || '0' }}",
|
|
);
|
|
expect(install.run).toContain('npm install');
|
|
expect(install.run).toContain('--ignore-scripts=false');
|
|
expect(job.needs).toContain('verify-browser-package');
|
|
expect(install.run).toContain('qwen-code-browser-use-');
|
|
expect(install.run).toContain("throw 'Windows SDK postinstall failed'");
|
|
expect(install.run).toContain(
|
|
"if ((Get-AuthenticodeSignature -LiteralPath $worker).Status -ne 'Valid') {",
|
|
);
|
|
expect(install.run).toContain(
|
|
"throw 'Ordinary installation must not deploy the UIAccess worker'",
|
|
);
|
|
expect(cuaSdkInstallScript).toContain('await run("powershell", args');
|
|
expect(cuaSdkInstallScript).toMatch(
|
|
/CouldNotAutoloadMatchingModule[\s\S]*?run\("pwsh"/,
|
|
);
|
|
expect(cuaSdkInstallScript).toContain(
|
|
'if (!String(error?.stderr).includes("CouldNotAutoloadMatchingModule")) {',
|
|
);
|
|
});
|
|
|
|
it('pins exact runtime registration and dependency versions', () => {
|
|
expect(computerUseGuide).toContain(
|
|
`@qwen-code/node-repl-mcp@${nodeReplPackage.version}`,
|
|
);
|
|
expect(computerUseGuide).toContain(
|
|
'installs the matching SDK and browser SDK as dependencies',
|
|
);
|
|
expect(cuaReleaseWorkflow).not.toContain('SKILL_NODE_REPL_VERSION=');
|
|
expect(cuaReleaseWorkflow).toContain(
|
|
'USER_GUIDE_NODE_REPL_VERSION=$(sed -nE',
|
|
);
|
|
expect(cuaReleaseWorkflow).not.toContain('SKILL_SDK_VERSION=');
|
|
expect(cuaReleaseWorkflow).toContain('SDK_BROWSER_VERSION=');
|
|
expect(cuaReleaseWorkflow).not.toContain('USER_GUIDE_SDK_VERSION=');
|
|
expect(cuaReleaseWorkflow).not.toContain(
|
|
'grep -Fq "@qwen-code/node-repl-mcp@',
|
|
);
|
|
expect(cuaReleaseWorkflow).not.toContain('grep -Fq "@qwen-code/cua-sdk@');
|
|
});
|
|
|
|
it('releases dependencies before the same-version runtime', () => {
|
|
expect(cuaReleaseWorkflow).not.toContain('node_repl_only');
|
|
expect(cuaReleaseJobs['publish-sdk'].needs).toContain('publish-browser');
|
|
expect(cuaReleaseJobs['publish-node-repl'].needs).toContain('publish-sdk');
|
|
expect(cuaReleaseWorkflow).toContain(
|
|
'A production dispatch must run from protected main',
|
|
);
|
|
});
|
|
});
|
|
|
|
// The canonical workspace restoration script shared by all nine
|
|
// 'Restore workspace ownership' copies in release.yml. The full wipe (vs
|
|
// serve-ab.yml's keep-and-scrub) is deliberate on this lane: release
|
|
// checkouts carry CI_BOT_PAT and the npm OIDC id-token, so no pre-existing
|
|
// repo state may survive into them. Pin the WHOLE body by equality, the way
|
|
// review-worktree-cleanup-workflow.test.js pins its sweep copies: a
|
|
// commented-out find, an inserted early exit, or a uniformly dropped
|
|
// ownership ladder all ship green under substring pins, and each of those
|
|
// mutants reopens the incident class this step exists for.
|
|
const canonicalWipe = `set -uo pipefail
|
|
# Reject symlinked or non-canonical paths before recursive changes.
|
|
RWS="\${RUNNER_WORKSPACE:?}"
|
|
while [ "\${RWS%/}" != "$RWS" ]; do RWS="\${RWS%/}"; done
|
|
if [ -L "$RWS" ]; then
|
|
echo "::error::refusing to wipe: runner workspace is a symlink: \${RWS}"
|
|
exit 1
|
|
fi
|
|
RWS_LEX="$(realpath -m -s -- "$RWS" 2>/dev/null)" || { echo "::error::refusing to wipe: realpath unavailable, cannot canonicalize \${RUNNER_WORKSPACE}"; exit 1; }
|
|
RWS="$(realpath -m -- "$RWS" 2>/dev/null)" || { echo "::error::refusing to wipe: realpath unavailable, cannot canonicalize \${RUNNER_WORKSPACE}"; exit 1; }
|
|
if [ "$RWS" != "$RWS_LEX" ]; then
|
|
echo "::error::refusing to wipe: runner workspace resolves through a symlinked component: \${RWS_LEX} resolves to \${RWS}"
|
|
exit 1
|
|
fi
|
|
while [ "\${RWS%/}" != "$RWS" ]; do RWS="\${RWS%/}"; done
|
|
if [ -z "$RWS" ]; then echo "::error::refusing to wipe: runner workspace resolved to /"; exit 1; fi
|
|
case "$RWS" in
|
|
..|../*|*/..|*/../*) echo "::error::refusing runner workspace path containing '..': \${RWS}"; exit 1 ;;
|
|
esac
|
|
WS="\${GITHUB_WORKSPACE:?}"
|
|
while [ "\${WS%/}" != "$WS" ]; do WS="\${WS%/}"; done
|
|
# Heal only a corrupt leaf whose canonical parent remains contained.
|
|
if [ -L "$WS" ] || [ ! -d "$WS" ]; then
|
|
HEAL_PARENT="$(realpath -m -- "$(dirname -- "$WS")" 2>/dev/null)" || { echo "::error::refusing to heal: realpath unavailable, cannot canonicalize the parent of \${WS}"; exit 1; }
|
|
case "$HEAL_PARENT" in
|
|
"$RWS"|"$RWS"/*) ;;
|
|
*) echo "::error::refusing to heal workspace outside the runner workspace: \${WS} (parent: \${HEAL_PARENT}, runner workspace: \${RWS})"; exit 1 ;;
|
|
esac
|
|
if [ -L "$WS" ]; then
|
|
# Keep attacker-controlled link text out of workflow commands.
|
|
heal_target="$(readlink -- "$WS" 2>/dev/null || printf '%s' '<unreadable>')"
|
|
heal_target="$(printf '%s' "$heal_target" | tr -d '\\r\\n' | cut -c1-200)"
|
|
echo "::warning::healing workspace \${WS}: it was a symlink"
|
|
printf 'heal: %s pointed at %s\\n' "$WS" "$heal_target"
|
|
else
|
|
echo "::warning::healing workspace \${WS}: it was not a directory"
|
|
fi
|
|
# Remove the raw link path; never resolve its target.
|
|
rm -f -- "$WS" || { echo "::error::refusing to continue: could not remove \${WS}"; exit 1; }
|
|
mkdir -- "$WS" || { echo "::error::refusing to continue: could not recreate \${WS}"; exit 1; }
|
|
fi
|
|
WS_LEX="$(realpath -m -s -- "$WS" 2>/dev/null)" || { echo "::error::refusing to wipe: realpath unavailable, cannot canonicalize \${GITHUB_WORKSPACE}"; exit 1; }
|
|
WS="$(realpath -m -- "$WS" 2>/dev/null)" || { echo "::error::refusing to wipe: realpath unavailable, cannot canonicalize \${GITHUB_WORKSPACE}"; exit 1; }
|
|
if [ "$WS" != "$WS_LEX" ]; then
|
|
echo "::error::refusing to wipe: workspace resolves through a symlinked component: \${WS_LEX} resolves to \${WS}"
|
|
exit 1
|
|
fi
|
|
while [ "\${WS%/}" != "$WS" ]; do WS="\${WS%/}"; done
|
|
case "$WS" in
|
|
..|../*|*/..|*/../*) echo "::error::refusing to wipe path containing '..': \${WS}"; exit 1 ;;
|
|
esac
|
|
# Deny known roots, then allow only runner-workspace descendants.
|
|
case "$WS" in
|
|
/|/home|/root|/usr*|/etc*|/var|"") echo "::error::refusing to wipe suspicious workspace path: \${WS}"; exit 1 ;;
|
|
esac
|
|
case "$WS" in
|
|
"$RWS"/*) ;;
|
|
*) echo "::error::refusing to wipe workspace outside the runner workspace: \${WS} (runner workspace: \${RWS})"; exit 1 ;;
|
|
esac
|
|
# Path geometry is validated before ownership changes.
|
|
RUNNER_UID="$(id -u)"
|
|
RUNNER_GID="$(id -g)"
|
|
if [ "$RUNNER_UID" != "0" ]; then
|
|
chown -R "$RUNNER_UID:$RUNNER_GID" "$GITHUB_WORKSPACE" 2>/dev/null || sudo -n chown -R "$RUNNER_UID:$RUNNER_GID" "$GITHUB_WORKSPACE" || echo "::warning::could not restore workspace ownership; checkout may fail on leftover root-owned files"
|
|
fi
|
|
chmod -R u+rwX "$GITHUB_WORKSPACE" 2>/dev/null || sudo -n chmod -R u+rwX "$GITHUB_WORKSPACE" || echo "::warning::could not restore workspace write permissions; checkout may fail on leftover read-only files"
|
|
find "$WS" -mindepth 1 -maxdepth 1 -exec rm -rf -- {} +
|
|
# Isolate Git, npm, Docker, and gh state inherited across pool jobs.
|
|
release_state="$(mktemp -d "\${RUNNER_TEMP:?}/release-state.XXXXXX")" || exit 1
|
|
: > "\${release_state}/gitconfig" || exit 1
|
|
: > "\${release_state}/npmrc" || exit 1
|
|
mkdir "\${release_state}/docker" || exit 1
|
|
mkdir "\${release_state}/gh" || exit 1
|
|
{
|
|
echo 'GIT_CONFIG_COUNT=0'
|
|
echo 'GIT_CONFIG_NOSYSTEM=1'
|
|
echo 'GIT_CONFIG_PARAMETERS='
|
|
echo "GIT_CONFIG_GLOBAL=\${release_state}/gitconfig"
|
|
echo "NPM_CONFIG_USERCONFIG=\${release_state}/npmrc"
|
|
echo "DOCKER_CONFIG=\${release_state}/docker"
|
|
echo "GH_CONFIG_DIR=\${release_state}/gh"
|
|
} >> "\${GITHUB_ENV:?}"`;
|
|
|
|
describe('release workflow', () => {
|
|
// The shard-completeness pin and the zero-test ratchet must gate on the
|
|
// same workspace set, so the selection lives once in
|
|
// getTestCiWorkspacePackageJsonPaths (scripts/workspaces.js) -- the same
|
|
// resolver unit-vitest-configs.test.ts consumes -- instead of letting
|
|
// each test carry its own selection copy. The pairs here only attach the
|
|
// parsed package.json the pins below assert on.
|
|
const getTestCiWorkspaces = () =>
|
|
getTestCiWorkspacePackageJsonPaths(process.cwd()).map((path) => [
|
|
path,
|
|
JSON.parse(readFileSync(path, 'utf8')),
|
|
]);
|
|
|
|
it('cleans every shared ECS workspace before checkout', () => {
|
|
// The subject set has to be derived from a property independent of the
|
|
// wipe itself: selecting on "has a Restore workspace ownership step"
|
|
// would filter out any newly added pool-routed job before an assertion
|
|
// ever ran. The pool marker is what makes state survive across jobs, and
|
|
// it cannot be swapped for the hosted label because every pool-routed
|
|
// runs-on expression names 'ubuntu-latest' as its fallback branch.
|
|
const isPoolRouted = (job) =>
|
|
/ecs-qwen-(?:hk4-host|release-host)/.test(String(job['runs-on'] ?? ''));
|
|
const wipesWorkspace = (job) =>
|
|
(job.steps ?? []).some(
|
|
(step) => step.name === 'Restore workspace ownership',
|
|
);
|
|
const checkoutJobs = Object.entries(releaseYaml.jobs).filter(
|
|
([, job]) =>
|
|
(job.steps ?? []).some((step) =>
|
|
String(step.uses ?? '').includes('actions/checkout'),
|
|
) &&
|
|
(isPoolRouted(job) || wipesWorkspace(job)),
|
|
);
|
|
|
|
expect(checkoutJobs.map(([id]) => id)).toEqual([
|
|
'prepare',
|
|
'quality_static',
|
|
'quality_build',
|
|
'quality_typecheck',
|
|
'workspace_tests',
|
|
'quality_scripts',
|
|
'integration_none',
|
|
'integration_docker',
|
|
'publish',
|
|
]);
|
|
for (const [id, job] of checkoutJobs) {
|
|
const restoreIndex = job.steps.findIndex(
|
|
(step) => step.name === 'Restore workspace ownership',
|
|
);
|
|
// The step must stay FIRST: it is the only defence between
|
|
// cross-job-persistent state and every later state read in the job,
|
|
// so a demotion must fail even while it remains ahead of checkout.
|
|
expect(restoreIndex, id).toBe(0);
|
|
expect(job.steps[restoreIndex]?.if, id).toBe(
|
|
"${{ runner.environment == 'self-hosted' }}",
|
|
);
|
|
const checkoutIndex = job.steps.findIndex((step) =>
|
|
String(step.uses ?? '').includes('actions/checkout'),
|
|
);
|
|
expect(checkoutIndex, id).toBeGreaterThan(0);
|
|
// Full-string equality against the shared constant: commenting out
|
|
// the find, inserting an early exit, or dropping the chown/chmod
|
|
// ladder uniformly from all nine copies keeps every substring and
|
|
// equality-across-copies pin green while reopening the incident.
|
|
expect(job.steps[restoreIndex]?.run, id).toBe(canonicalWipe);
|
|
}
|
|
// Pool-routed jobs inherit root-owned leftovers whether or not they
|
|
// check out, so the exemption above must never extend to them.
|
|
for (const [id, job] of Object.entries(releaseYaml.jobs)) {
|
|
if (!isPoolRouted(job)) {
|
|
continue;
|
|
}
|
|
expect(wipesWorkspace(job), id).toBe(true);
|
|
}
|
|
});
|
|
|
|
it('gates every pool-routed job on a disk floor before its heavy steps', () => {
|
|
// The release lane's slice of #10035. ci.yml has gated its heavy jobs on
|
|
// check-disk-floor.sh since that incident; release validation went
|
|
// without until a saturated instance died on ENOSPC mid-step — no log,
|
|
// no annotation — and took the release with it (runs 34998771277 and
|
|
// 35024357480). The gate fails the job fast instead, and a re-run lands
|
|
// on an instance with headroom. Jobs without a checkout are exempt: the
|
|
// gate rides a checkout, and the only such pool-routed job (quality) is
|
|
// a workspace-free three-second aggregation with nothing heavy to lose.
|
|
const isPoolRouted = (job) =>
|
|
/ecs-qwen-(?:hk4-host|release-host)/.test(String(job['runs-on'] ?? ''));
|
|
const gated = [];
|
|
for (const [id, job] of Object.entries(releaseYaml.jobs)) {
|
|
if (!isPoolRouted(job)) continue;
|
|
if (
|
|
!(job.steps ?? []).some((step) =>
|
|
String(step.uses ?? '').includes('actions/checkout'),
|
|
)
|
|
) {
|
|
continue;
|
|
}
|
|
gated.push(id);
|
|
const steps = job.steps ?? [];
|
|
const gateIndex = steps.findIndex(
|
|
(step) => step.name === 'Disk floor gate (self-hosted)',
|
|
);
|
|
expect(gateIndex, id).toBeGreaterThanOrEqual(0);
|
|
const gate = steps[gateIndex];
|
|
expect(gate.if, id).toBe("${{ runner.environment == 'self-hosted' }}");
|
|
// The gate script rides the selected ref's checkout, so the gate can
|
|
// only stand between that checkout and the first heavy step, and a ref
|
|
// that predates the script must skip the gate rather than fail on it.
|
|
expect(gate.run, id).toBe(
|
|
'if [ -f .github/scripts/check-disk-floor.sh ]; then\n' +
|
|
' bash .github/scripts/check-disk-floor.sh "${GITHUB_WORKSPACE}" "${RUNNER_TEMP:-/tmp}"\n' +
|
|
'fi',
|
|
);
|
|
const checkoutIndex = steps.findIndex((step) =>
|
|
String(step.uses ?? '').includes('actions/checkout'),
|
|
);
|
|
const installIndex = steps.findIndex(
|
|
(step) => step.name === 'Install Dependencies',
|
|
);
|
|
expect(checkoutIndex, id).toBeGreaterThanOrEqual(0);
|
|
expect(installIndex, id).toBeGreaterThan(checkoutIndex);
|
|
expect(gateIndex, id).toBeGreaterThan(checkoutIndex);
|
|
expect(gateIndex, id).toBeLessThan(installIndex);
|
|
}
|
|
expect(gated.sort()).toEqual([
|
|
'integration_docker',
|
|
'integration_none',
|
|
'prepare',
|
|
'publish',
|
|
'quality_build',
|
|
'quality_scripts',
|
|
'quality_static',
|
|
'quality_typecheck',
|
|
'workspace_tests',
|
|
]);
|
|
});
|
|
|
|
it('uses shallow history only for validation jobs', () => {
|
|
const checkoutDepth = (id) =>
|
|
releaseYaml.jobs[id].steps.find((step) =>
|
|
String(step.uses ?? '').includes('actions/checkout'),
|
|
).with['fetch-depth'];
|
|
|
|
expect(checkoutDepth('prepare')).toBe(0);
|
|
expect(checkoutDepth('publish')).toBe(0);
|
|
for (const id of [
|
|
'quality_static',
|
|
'quality_build',
|
|
'quality_typecheck',
|
|
'workspace_tests',
|
|
'quality_scripts',
|
|
'integration_none',
|
|
'integration_docker',
|
|
]) {
|
|
expect(checkoutDepth(id), id).toBe(1);
|
|
}
|
|
});
|
|
|
|
it('loads extracted runners from the workflow commit for old release refs', () => {
|
|
for (const jobId of [
|
|
'prepare',
|
|
'quality_build',
|
|
'workspace_tests',
|
|
'integration_docker',
|
|
'publish',
|
|
'notify_failure',
|
|
]) {
|
|
const steps = releaseYaml.jobs[jobId].steps;
|
|
const targetCheckoutIndex = steps.findIndex(
|
|
(step) => step.name === 'Checkout',
|
|
);
|
|
const scriptCheckoutIndex = steps.findIndex(
|
|
(step) => step.name === 'Checkout release workflow scripts',
|
|
);
|
|
const scriptCheckout = steps[scriptCheckoutIndex];
|
|
|
|
expect(scriptCheckout.with, jobId).toEqual({
|
|
ref: '${{ github.workflow_sha }}',
|
|
'fetch-depth': 1,
|
|
'persist-credentials': false,
|
|
// The tarball gate rides this checkout too: run from the selected
|
|
// ref's own `scripts/`, it is missing on any ref that predates it.
|
|
'sparse-checkout':
|
|
'/.github/scripts\n/scripts/assert-release-version.mjs\n/scripts/check-release-tarball-exec-bits.mjs\n/scripts/release-packages.mjs\n/scripts/workspaces.js\n/package.json\n/packages/*/package.json\n/packages/channels/*/package.json\n/integrations/*/package.json',
|
|
'sparse-checkout-cone-mode': false,
|
|
path: '.release-workflow',
|
|
});
|
|
if (jobId !== 'notify_failure') {
|
|
expect(scriptCheckoutIndex, jobId).toBeGreaterThan(targetCheckoutIndex);
|
|
}
|
|
for (const [index, step] of steps.entries()) {
|
|
if (String(step.run ?? '').includes('.release-workflow/')) {
|
|
expect(index, `${jobId}:${step.name}`).toBeGreaterThan(
|
|
scriptCheckoutIndex,
|
|
);
|
|
}
|
|
}
|
|
}
|
|
});
|
|
|
|
it('re-pins trusted runners before every step that runs one', () => {
|
|
// Swept, not hand-listed: a list of protected step names cannot fail for
|
|
// the steps it omits, which is how `Build Bundle and Prepare Package`,
|
|
// `Build Standalone Archives` and `Verify Standalone Archives` came to
|
|
// execute the trusted runner from a tree that selected-ref npm code had
|
|
// already run over. Every step invoking `.release-workflow/` is checked.
|
|
let swept = 0;
|
|
for (const [jobId, job] of Object.entries(releaseYaml.jobs)) {
|
|
const steps = job.steps ?? [];
|
|
for (const [index, step] of steps.entries()) {
|
|
if (!String(step.run ?? '').includes('.release-workflow/')) continue;
|
|
swept += 1;
|
|
const label = `${jobId}:${step.name}`;
|
|
const checkoutIndex = steps.findLastIndex(
|
|
(candidate, candidateIndex) =>
|
|
candidateIndex < index &&
|
|
candidate.name === 'Checkout release workflow scripts',
|
|
);
|
|
expect(checkoutIndex, label).toBeGreaterThanOrEqual(0);
|
|
// Nothing between the trusted checkout and the step it protects may
|
|
// run selected-ref npm code, which could overwrite the runner in
|
|
// place — including npm lifecycle scripts fired by `npm publish`.
|
|
expect(
|
|
steps
|
|
.slice(checkoutIndex + 1, index)
|
|
.some((candidate) => /\bnpm\b/.test(String(candidate.run ?? ''))),
|
|
label,
|
|
).toBe(false);
|
|
}
|
|
|
|
// Every re-checkout discards the previous tree first; the job's first
|
|
// checkout has nothing to reset.
|
|
const checkouts = steps
|
|
.map((step, index) =>
|
|
step.name === 'Checkout release workflow scripts' ? index : -1,
|
|
)
|
|
.filter((index) => index >= 0);
|
|
for (const [ordinal, index] of checkouts.entries()) {
|
|
if (ordinal === 0) continue;
|
|
expect(steps[index - 1]?.name, `${jobId}:checkout@${index}`).toBe(
|
|
'Reset release workflow scripts',
|
|
);
|
|
}
|
|
}
|
|
expect(swept).toBeGreaterThan(10);
|
|
});
|
|
|
|
it('uses workflow-pinned manifests for publishing and the version guard', () => {
|
|
const directory = mkdtempSync(join(tmpdir(), 'release-selection-'));
|
|
try {
|
|
// A release-ref cwd must not change the workflow's trusted selection.
|
|
writeFileSync(join(directory, 'package.json'), '{"workspaces":[]}');
|
|
const result = spawnSync(
|
|
process.execPath,
|
|
[join(process.cwd(), 'scripts/release-packages.mjs')],
|
|
{
|
|
cwd: directory,
|
|
encoding: 'utf8',
|
|
env: {
|
|
...process.env,
|
|
PUBLISH_AUDIO_CAPTURE: 'true',
|
|
PUBLISH_EXTERNAL_CONTEXT_MEM0: 'true',
|
|
},
|
|
},
|
|
);
|
|
expect(result.status, result.stderr).toBe(0);
|
|
expect([
|
|
'@qwen-code/qwen-code',
|
|
...result.stdout.trim().split('\n'),
|
|
]).toEqual(PUBLISHED_PACKAGES);
|
|
} finally {
|
|
rmSync(directory, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it('keeps the workflow focused on orchestration', () => {
|
|
// 880, raised from 870 for quality_scripts' standalone runtime cache
|
|
// step: the same four-line export of a host-wide cache path as ci.yml's
|
|
// test job — orchestration; the cache logic lives in
|
|
// scripts/lib/runtime-archive-cache.js.
|
|
// 870, raised from 850 for moving publish/collect onto the opt-in
|
|
// ecs-qwen-release-host lane and quality onto the pool its checks already
|
|
// use: the added lines are two runs-on expressions, two anchored steps,
|
|
// and their comments — orchestration, not logic (2026-10-15 hosted-pool
|
|
// saturation queued publish for 64 minutes behind PR automation).
|
|
// 850 was raised from 830 for the pool-wide disk-floor gate: one anchored
|
|
// step plus one alias per pool-routed job is still orchestration — the
|
|
// gate's logic lives in .github/scripts/check-disk-floor.sh (#10035).
|
|
// 830 was raised from 800 for the notify_failure fallback: that job may
|
|
// not depend on the trusted checkout or the extracted runner, because it
|
|
// is the job that reports their failure, so its last-resort issue filing
|
|
// is deliberately inline. Every other step stays under the per-step cap
|
|
// below, which is the rule that actually keeps logic out of the YAML.
|
|
expect(workflow.split('\n').length).toBeLessThan(880);
|
|
for (const [jobId, job] of Object.entries(releaseYaml.jobs)) {
|
|
for (const step of job.steps ?? []) {
|
|
if (step.name === 'Restore workspace ownership' || !step.run) continue;
|
|
expect(
|
|
String(step.run).split('\n').length,
|
|
`${jobId}:${step.name}`,
|
|
).toBeLessThanOrEqual(12);
|
|
}
|
|
}
|
|
});
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'executes release flag classification from the extracted script',
|
|
() => {
|
|
const directory = mkdtempSync(join(tmpdir(), 'release-flags-'));
|
|
const output = join(directory, 'output');
|
|
writeFileSync(output, '');
|
|
try {
|
|
const result = spawnSync(
|
|
'bash',
|
|
[releaseStepScriptAbsolutePath, 'set-flags'],
|
|
{
|
|
encoding: 'utf8',
|
|
env: {
|
|
...process.env,
|
|
CREATE_NIGHTLY_RELEASE: 'false',
|
|
CREATE_PREVIEW_RELEASE: 'false',
|
|
CRON: '0 21 * * *',
|
|
DRY_RUN_INPUT: 'true',
|
|
GITHUB_OUTPUT: output,
|
|
},
|
|
},
|
|
);
|
|
expect(result.status).toBe(0);
|
|
expect(readFileSync(output, 'utf8')).toBe(
|
|
'is_nightly=true\nis_preview=false\nis_dry_run=true\n',
|
|
);
|
|
} finally {
|
|
rmSync(directory, { recursive: true, force: true });
|
|
}
|
|
},
|
|
);
|
|
|
|
it.skipIf(process.platform === 'win32').each([
|
|
{ dryRun: 'false', optional: 'false', failure: '0', cliPublished: '1' },
|
|
{ dryRun: 'false', optional: 'true', failure: '0', cliPublished: '0' },
|
|
{ dryRun: 'true', optional: 'true', failure: '0', cliPublished: '0' },
|
|
{ dryRun: 'false', optional: 'true', failure: '7', cliPublished: '1' },
|
|
])(
|
|
'publishes the selected workspaces and CLI safely: %j',
|
|
({ dryRun, optional, failure, cliPublished }) => {
|
|
const directory = mkdtempSync(join(tmpdir(), 'release-publish-'));
|
|
const bin = join(directory, 'bin');
|
|
const publishLog = join(directory, 'published');
|
|
mkdirSync(bin);
|
|
mkdirSync(join(directory, 'dist'));
|
|
writeFileSync(
|
|
join(directory, 'dist/package.json'),
|
|
JSON.stringify({ name: '@qwen-code/qwen-code' }),
|
|
);
|
|
// `npm view` drives the already-published skip; `npm publish` is the
|
|
// generated CLI's own publish call, logged like the corepack one so the
|
|
// assertions below cover both. The CLI goes through npm because pnpm's
|
|
// packer drops the exec bit on `vendor/ripgrep/*/rg` (#12668).
|
|
writeFileSync(
|
|
join(bin, 'npm'),
|
|
'#!/bin/sh\nif [ "$1" = "view" ]; then exit "$CLI_PUBLISHED"; fi\nprintf "%s\\t%s\\n" "$PWD" "$*" >> "$PUBLISH_LOG"\nexit "$PUBLISH_FAILURE"\n',
|
|
{ mode: 0o755 },
|
|
);
|
|
writeFileSync(
|
|
join(bin, 'corepack'),
|
|
'#!/bin/sh\nprintf "%s\\t%s\\n" "$PWD" "$*" >> "$PUBLISH_LOG"\nexit "$PUBLISH_FAILURE"\n',
|
|
{ mode: 0o755 },
|
|
);
|
|
try {
|
|
const result = spawnSync(
|
|
'bash',
|
|
[releaseStepScriptAbsolutePath, 'publish-packages'],
|
|
{
|
|
cwd: directory,
|
|
encoding: 'utf8',
|
|
env: {
|
|
...process.env,
|
|
PATH: `${bin}:${process.env.PATH}`,
|
|
IS_DRY_RUN: dryRun,
|
|
NPM_TAG: 'preview',
|
|
PUBLISH_AUDIO_CAPTURE: optional,
|
|
PUBLISH_EXTERNAL_CONTEXT_MEM0: optional,
|
|
PUBLISH_FAILURE: failure,
|
|
CLI_PUBLISHED: cliPublished,
|
|
PUBLISH_LOG: publishLog,
|
|
RELEASE_VERSION: '1.2.3',
|
|
},
|
|
},
|
|
);
|
|
expect(result.status, result.stderr).toBe(Number(failure));
|
|
const canonicalDirectory = realpathSync(directory);
|
|
const publishCalls = readFileSync(publishLog, 'utf8')
|
|
.trim()
|
|
.split('\n')
|
|
.map((line) => line.split('\t'));
|
|
const publishesCli =
|
|
failure === '0' && (dryRun === 'true' || cliPublished !== '0');
|
|
expect(publishCalls.map(([cwd]) => cwd)).toEqual([
|
|
canonicalDirectory,
|
|
...(publishesCli ? [join(canonicalDirectory, 'dist')] : []),
|
|
]);
|
|
const selected = publishCalls[0][1]
|
|
.split(' ')
|
|
.filter((arg) => arg.startsWith('--filter='))
|
|
.map((arg) => arg.slice('--filter='.length));
|
|
expect(selected).toEqual(
|
|
RELEASE_WORKSPACES.filter(
|
|
(name) =>
|
|
optional === 'true' ||
|
|
![
|
|
'@qwen-code/audio-capture',
|
|
'@qwen-code/external-context-mem0',
|
|
].includes(name),
|
|
),
|
|
);
|
|
expect(publishCalls[0][1]).toContain('pnpm -r publish');
|
|
if (publishesCli) {
|
|
// The generated CLI is the only package carrying vendored binaries
|
|
// and the only one published with npm: pnpm's packer writes 0644 for
|
|
// every entry outside `bin`, which shipped an unspawnable
|
|
// `vendor/ripgrep/*/rg` from 0.24.4 on (#12668).
|
|
expect(publishCalls[1][1]).toMatch(/^publish --provenance /);
|
|
expect(publishCalls[1][1]).not.toContain('pnpm');
|
|
}
|
|
for (const [cwd, args] of publishCalls) {
|
|
expect(args, cwd).toContain('--access public');
|
|
expect(args, cwd).toContain('--tag=preview');
|
|
expect(args, cwd).toContain('--provenance');
|
|
expect(args.includes('--dry-run'), cwd).toBe(dryRun === 'true');
|
|
}
|
|
expect(publishCalls[0][1].includes('--force')).toBe(dryRun === 'true');
|
|
} finally {
|
|
rmSync(directory, { recursive: true, force: true });
|
|
}
|
|
},
|
|
);
|
|
|
|
it('runs the step script only from the trusted checkout', () => {
|
|
// The whole point of the extraction is that the decision comes from the
|
|
// workflow-pinned SHA. Every arm pin elsewhere is a substring match that
|
|
// a bare `.github/scripts/...` invocation would satisfy just as well, so
|
|
// pin the prefix itself: no step may reach the release-ref copy.
|
|
let invocations = 0;
|
|
for (const [jobId, job] of Object.entries(releaseYaml.jobs)) {
|
|
for (const step of job.steps ?? []) {
|
|
const run = String(step.run ?? '');
|
|
if (!/\.github\/scripts\/run-release-[a-z-]+\.sh/.test(run)) continue;
|
|
invocations += 1;
|
|
expect(run, `${jobId}:${step.name}`).toContain('.release-workflow/');
|
|
expect(run, `${jobId}:${step.name}`).not.toMatch(
|
|
/(^|[^/])\.github\/scripts\/run-release-/m,
|
|
);
|
|
}
|
|
}
|
|
expect(invocations).toBeGreaterThan(10);
|
|
});
|
|
|
|
it('keeps the extracted release scripts executable', () => {
|
|
// release.yml runs these by bare path, so the mode is load-bearing:
|
|
// a checkout that materializes them 100644 dies with exit 126 before any
|
|
// validation runs. Nothing else pins it — the execution tests all invoke
|
|
// `bash <script>`, which ignores the mode, and release.yml triggers only
|
|
// on schedule/workflow_dispatch, so no pull-request lane ever runs the
|
|
// bare-path form. Assert the *recorded* mode, not the working tree's:
|
|
// that is what actions/checkout materializes.
|
|
const recorded = spawnSync('git', ['ls-tree', 'HEAD', '.github/scripts/'], {
|
|
encoding: 'utf8',
|
|
});
|
|
expect(recorded.status).toBe(0);
|
|
const modes = new Map(
|
|
recorded.stdout
|
|
.trim()
|
|
.split('\n')
|
|
.map((line) => {
|
|
const [meta, path] = line.split('\t');
|
|
return [path, meta.split(' ')[0]];
|
|
}),
|
|
);
|
|
for (const script of [
|
|
'.github/scripts/run-release-step.sh',
|
|
'.github/scripts/run-release-workspace-tests.sh',
|
|
'.github/scripts/run-release-docker-integration.sh',
|
|
]) {
|
|
expect(modes.get(script), script).toBe('100755');
|
|
}
|
|
});
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'maps a manual preview release onto the preview override flags',
|
|
() => {
|
|
// `resolve-version` was the one arm no test executed or text-pinned, so
|
|
// flipping --type=preview to --type=stable left the whole scripts lane
|
|
// green — while at runtime the stable path ignores
|
|
// preview_version_override entirely, publishing the operator's manual
|
|
// preview as a stable version under npm's `latest` dist-tag.
|
|
const directory = mkdtempSync(join(tmpdir(), 'release-resolve-'));
|
|
const bin = join(directory, 'bin');
|
|
const argsLog = join(directory, 'node-args');
|
|
const output = join(directory, 'github-output');
|
|
mkdirSync(bin);
|
|
writeFileSync(
|
|
join(bin, 'node'),
|
|
'#!/bin/sh\n' +
|
|
'printf "%s\\n" "$*" >> "$NODE_ARGS_LOG"\n' +
|
|
'echo \'{"releaseTag":"v1.2.3-preview.0","releaseVersion":"1.2.3-preview.0","npmTag":"preview","previousReleaseTag":"v1.2.2"}\'\n',
|
|
{ mode: 0o755 },
|
|
);
|
|
writeFileSync(output, '');
|
|
try {
|
|
const result = spawnSync(
|
|
'bash',
|
|
[releaseStepScriptAbsolutePath, 'resolve-version'],
|
|
{
|
|
cwd: directory,
|
|
encoding: 'utf8',
|
|
env: {
|
|
...process.env,
|
|
PATH: `${bin}:${process.env.PATH}`,
|
|
IS_NIGHTLY: 'false',
|
|
IS_PREVIEW: 'true',
|
|
MANUAL_VERSION: '1.2.3',
|
|
NODE_ARGS_LOG: argsLog,
|
|
GITHUB_OUTPUT: output,
|
|
},
|
|
},
|
|
);
|
|
expect(result.status).toBe(0);
|
|
const nodeArgs = readFileSync(argsLog, 'utf8');
|
|
expect(nodeArgs).toContain('--type=preview');
|
|
expect(nodeArgs).toContain(
|
|
'--preview_version_override=1.2.3-preview.0',
|
|
);
|
|
expect(nodeArgs).not.toContain('--type=stable');
|
|
expect(readFileSync(output, 'utf8')).toContain('NPM_TAG=preview');
|
|
} finally {
|
|
rmSync(directory, { recursive: true, force: true });
|
|
}
|
|
},
|
|
);
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'rejects a manual preview version that is not X.Y.Z or X.Y.Z-preview.N',
|
|
() => {
|
|
const directory = mkdtempSync(join(tmpdir(), 'release-resolve-bad-'));
|
|
const output = join(directory, 'github-output');
|
|
writeFileSync(output, '');
|
|
try {
|
|
const result = spawnSync(
|
|
'bash',
|
|
[releaseStepScriptAbsolutePath, 'resolve-version'],
|
|
{
|
|
cwd: directory,
|
|
encoding: 'utf8',
|
|
env: {
|
|
...process.env,
|
|
IS_NIGHTLY: 'false',
|
|
IS_PREVIEW: 'true',
|
|
MANUAL_VERSION: 'not-a-version',
|
|
GITHUB_OUTPUT: output,
|
|
},
|
|
},
|
|
);
|
|
expect(result.status).toBe(1);
|
|
expect(result.stdout + result.stderr).toContain(
|
|
'For preview releases, version must be X.Y.Z or X.Y.Z-preview.N',
|
|
);
|
|
} finally {
|
|
rmSync(directory, { recursive: true, force: true });
|
|
}
|
|
},
|
|
);
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'aborts notify-failure instead of filing a duplicate when gh is unreachable',
|
|
() => {
|
|
const directory = mkdtempSync(join(tmpdir(), 'release-notify-'));
|
|
const bin = join(directory, 'bin');
|
|
const ghLog = join(directory, 'gh-calls');
|
|
mkdirSync(bin);
|
|
// Stand in for a connection-level failure: stderr only, exit 1, and
|
|
// nothing at all on stdout, which is what jq then reads as an empty list.
|
|
writeFileSync(
|
|
join(bin, 'gh'),
|
|
'#!/bin/sh\n' +
|
|
'printf "%s\\n" "$*" >> "$GH_CALL_LOG"\n' +
|
|
'if [ "$1" = issue ] && [ "$2" = list ]; then\n' +
|
|
' echo "gh: connection reset by peer" >&2\n' +
|
|
' exit 1\n' +
|
|
'fi\n' +
|
|
'exit 0\n',
|
|
{ mode: 0o755 },
|
|
);
|
|
try {
|
|
const result = spawnSync(
|
|
'bash',
|
|
[releaseStepScriptAbsolutePath, 'notify-failure'],
|
|
{
|
|
cwd: directory,
|
|
encoding: 'utf8',
|
|
env: {
|
|
...process.env,
|
|
PATH: `${bin}:${process.env.PATH}`,
|
|
GH_CALL_LOG: ghLog,
|
|
GH_REPO: 'QwenLM/qwen-code',
|
|
RELEASE_TAG: 'v1.2.3',
|
|
DETAILS_URL: 'https://github.example/run/1',
|
|
PREPARE_RESULT: 'success',
|
|
QUALITY_RESULT: 'failure',
|
|
INTEGRATION_NONE_RESULT: 'skipped',
|
|
INTEGRATION_DOCKER_RESULT: 'skipped',
|
|
PUBLISH_RESULT: 'skipped',
|
|
BUG_LABEL: 'type/bug',
|
|
READY_FOR_AGENT_LABEL: 'status/ready-for-agent',
|
|
AUTOFIX_APPROVED_LABEL: 'autofix/approved',
|
|
},
|
|
},
|
|
);
|
|
// Without pipefail this arm exits 0 and files a fresh autofix/approved
|
|
// twin of the release-failure issue, bypassing all three reuse guards
|
|
// (title-prefix anchor, bot-author refusal, still_eligible) because
|
|
// those only run on the reuse branch.
|
|
expect(result.status, result.stderr).not.toBe(0);
|
|
expect(readFileSync(ghLog, 'utf8')).not.toContain('issue create');
|
|
} finally {
|
|
rmSync(directory, { recursive: true, force: true });
|
|
}
|
|
},
|
|
);
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'records the issue before a failed autofix dispatch',
|
|
() => {
|
|
const directory = mkdtempSync(join(tmpdir(), 'release-notify-dispatch-'));
|
|
const bin = join(directory, 'bin');
|
|
const output = join(directory, 'github-output');
|
|
mkdirSync(bin);
|
|
writeFileSync(output, '');
|
|
writeFileSync(
|
|
join(bin, 'gh'),
|
|
'#!/bin/sh\n' +
|
|
'case "$1 $2" in\n' +
|
|
" 'issue list') echo '[]' ;;\n" +
|
|
" 'issue create') echo 'https://github.example/QwenLM/qwen-code/issues/4242' ;;\n" +
|
|
" 'workflow run') exit 1 ;;\n" +
|
|
'esac\n',
|
|
{ mode: 0o755 },
|
|
);
|
|
try {
|
|
const result = spawnSync(
|
|
'bash',
|
|
[releaseStepScriptAbsolutePath, 'notify-failure'],
|
|
{
|
|
cwd: directory,
|
|
encoding: 'utf8',
|
|
env: {
|
|
...process.env,
|
|
PATH: `${bin}:${process.env.PATH}`,
|
|
GITHUB_OUTPUT: output,
|
|
GH_REPO: 'QwenLM/qwen-code',
|
|
RELEASE_TAG: 'v1.2.3',
|
|
DETAILS_URL: 'https://github.example/run/1',
|
|
PREPARE_RESULT: 'success',
|
|
QUALITY_RESULT: 'failure',
|
|
INTEGRATION_NONE_RESULT: 'skipped',
|
|
INTEGRATION_DOCKER_RESULT: 'skipped',
|
|
PUBLISH_RESULT: 'skipped',
|
|
BUG_LABEL: 'type/bug',
|
|
READY_FOR_AGENT_LABEL: 'status/ready-for-agent',
|
|
AUTOFIX_APPROVED_LABEL: 'autofix/approved',
|
|
},
|
|
},
|
|
);
|
|
expect(result.status).not.toBe(0);
|
|
expect(readFileSync(output, 'utf8')).toBe(
|
|
'issue_url=https://github.example/QwenLM/qwen-code/issues/4242\n',
|
|
);
|
|
const fallback = releaseYaml.jobs.notify_failure.steps.find(
|
|
(step) => step.name === 'File a fallback failure issue',
|
|
);
|
|
expect(fallback.if).toBe(
|
|
"${{ always() && steps.notify.outcome != 'success' && !steps.notify.outputs.issue_url }}",
|
|
);
|
|
} finally {
|
|
rmSync(directory, { recursive: true, force: true });
|
|
}
|
|
},
|
|
);
|
|
|
|
it('pins validation and publishing to the commit resolved by prepare', () => {
|
|
expect(releaseYaml.jobs.prepare.outputs.release_sha).toBe(
|
|
'${{ steps.source.outputs.release_sha }}',
|
|
);
|
|
const sourceStep = releaseYaml.jobs.prepare.steps.find(
|
|
(step) => step.id === 'source',
|
|
);
|
|
expect(sourceStep.run).toContain('run-release-step.sh resolve-commit');
|
|
expect(releaseStepScript).toContain('git rev-parse HEAD');
|
|
|
|
for (const id of [
|
|
'quality_static',
|
|
'quality_build',
|
|
'quality_typecheck',
|
|
'workspace_tests',
|
|
'quality_scripts',
|
|
'integration_none',
|
|
'integration_docker',
|
|
'publish',
|
|
]) {
|
|
const checkout = releaseYaml.jobs[id].steps.find((step) =>
|
|
String(step.uses ?? '').includes('actions/checkout'),
|
|
);
|
|
expect(checkout.with.ref, id).toBe(
|
|
'${{ needs.prepare.outputs.release_sha }}',
|
|
);
|
|
// The ref expression only resolves when the consumer declares prepare
|
|
// in needs; without the edge it evaluates to '' and checkout silently
|
|
// falls back to the event ref (the moving branch tip at fetch time).
|
|
expect([].concat(releaseYaml.jobs[id].needs ?? []), id).toContain(
|
|
'prepare',
|
|
);
|
|
}
|
|
});
|
|
|
|
it('allows build artifact replacement when all jobs are rerun', () => {
|
|
const upload = releaseYaml.jobs.quality_build.steps.find((step) =>
|
|
String(step.uses ?? '').includes('actions/upload-artifact'),
|
|
);
|
|
expect(upload.with.name).toBe('release-quality-build');
|
|
expect(upload.with.overwrite).toBe(true);
|
|
});
|
|
|
|
it('keeps the build artifact alive long enough for re-run failed jobs', () => {
|
|
const upload = releaseYaml.jobs.quality_build.steps.find((step) =>
|
|
String(step.uses ?? '').includes('actions/upload-artifact'),
|
|
);
|
|
// "Re-run failed jobs" more than a day later does not re-run the
|
|
// succeeded producer; with retention-days: 1 its artifact is already
|
|
// expired and every consumer fails the download. Three days keeps the
|
|
// re-run path recoverable without paying for long-term storage.
|
|
expect(upload.with['retention-days']).toBe(3);
|
|
});
|
|
|
|
it('downloads and unpacks the build artifact in every consumer job', () => {
|
|
// The build-once contract has one producer and three consumers; the
|
|
// upload pin alone leaves a consumer free to drop the download/unpack
|
|
// and silently test a checkout without build outputs.
|
|
for (const id of [
|
|
'quality_typecheck',
|
|
'workspace_tests',
|
|
'quality_scripts',
|
|
]) {
|
|
const steps = releaseYaml.jobs[id].steps;
|
|
// The download only waits for the upload when the needs edge
|
|
// exists; without it the consumer races the producer.
|
|
expect([].concat(releaseYaml.jobs[id].needs ?? []), id).toContain(
|
|
'quality_build',
|
|
);
|
|
const downloadIndex = steps.findIndex((step) =>
|
|
String(step.uses ?? '').includes('actions/download-artifact'),
|
|
);
|
|
expect(
|
|
downloadIndex,
|
|
`${id} lost the build download step`,
|
|
).toBeGreaterThanOrEqual(0);
|
|
expect(steps[downloadIndex].with.name, id).toBe('release-quality-build');
|
|
expect(steps[downloadIndex].with.path, id).toBe(
|
|
'${{ runner.temp }}/release-quality-build',
|
|
);
|
|
const unpackIndex = steps.findIndex(
|
|
(step) => step.name === 'Unpack Build Outputs',
|
|
);
|
|
// Unpack must stay after download: reversed, tar would read a path
|
|
// that does not exist yet.
|
|
expect(unpackIndex, id).toBeGreaterThan(downloadIndex);
|
|
expect(steps[unpackIndex].run, id).toBe(
|
|
'tar -xzf "${RUNNER_TEMP}/release-quality-build/release-build.tgz"',
|
|
);
|
|
}
|
|
});
|
|
|
|
it('shares generated web templates with build consumers', () => {
|
|
const pack = releaseYaml.jobs.quality_build.steps.find(
|
|
(step) => step.name === 'Pack Build Outputs',
|
|
);
|
|
expect(pack.run).toContain('run-release-step.sh pack-build');
|
|
expect(releaseStepScript).toContain('packages/web-templates/src/generated');
|
|
// npm ci leaves nested dependency dist dirs under workspace
|
|
// node_modules; the find must prune them so only real build outputs
|
|
// travel in release-quality-build.
|
|
expect(releaseStepScript).toContain('-type d -name node_modules -prune');
|
|
// Log what is packed and fail closed on a silent under-pack: a dropped
|
|
// `-o` in the find turns the two -prune clauses into one conjunction
|
|
// that matches nothing, and tar would then ship only the two hardcoded
|
|
// paths while the symptom lands in downstream consumers.
|
|
expect(releaseStepScript).toContain('printf');
|
|
expect(releaseStepScript).toContain('${#build_paths[@]} -gt 2');
|
|
});
|
|
|
|
it('keeps the dist producer ahead of the pack step', () => {
|
|
// Pack hardcodes the repo-root `dist`, which only exists as a side
|
|
// effect of check:serve-fast-path-bundle (the check runs the esbuild
|
|
// bundle with outdir dist; scripts/build.js never writes it). The
|
|
// dependency must stay documented and ordered: moving the check after
|
|
// Pack, or dropping it, leaves tar without the bundle outputs.
|
|
const names = releaseYaml.jobs.quality_build.steps.map((step) => step.name);
|
|
const producer = names.indexOf('Check Serve Fast Path Bundle');
|
|
const pack = names.indexOf('Pack Build Outputs');
|
|
expect(producer).toBeGreaterThanOrEqual(0);
|
|
expect(pack).toBeGreaterThan(producer);
|
|
expect(workflow).toContain(
|
|
'# This materializes the repo-root dist packed below.',
|
|
);
|
|
expect(releaseStepScript).toContain(
|
|
"build_paths=('dist' 'packages/web-templates/src/generated')",
|
|
);
|
|
});
|
|
|
|
it('fans workspace tests into three complete Vitest shards', () => {
|
|
const job = releaseYaml.jobs.workspace_tests;
|
|
expect(job.strategy).toEqual({
|
|
'fail-fast': false,
|
|
matrix: { shard: [1, 2, 3] },
|
|
});
|
|
const testStep = job.steps.find(
|
|
(step) => step.name === 'Run Workspace Tests',
|
|
);
|
|
expect(testStep.run).toBe(
|
|
'.release-workflow/.github/scripts/run-release-workspace-tests.sh "${{ matrix.shard }}"',
|
|
);
|
|
expect(workspaceTestScript).toContain(
|
|
'npm run test:release:workspaces -- --shard="${shard}/3" --passWithNoTests "${retry_arg[@]}"',
|
|
);
|
|
expect(workspaceTestScript).toContain(
|
|
'::warning title=Workspace tests exited',
|
|
);
|
|
// Every release schedule retries, stable included: running the stable
|
|
// lane with no retry let one flaky test out of ~30k red a release whose
|
|
// other gates were all green. The default is pinned here so a silent
|
|
// drop back to a no-retry stable lane fails this test.
|
|
expect(testStep.env.VITEST_RETRY).toBe(
|
|
"${{ vars.QWEN_RELEASE_VITEST_RETRY || '2' }}",
|
|
);
|
|
// Keep the workflow invocation explicit and the script independently
|
|
// fail-closed, so direct local execution and Actions use the same shell.
|
|
expect(testStep.shell).toBe('bash');
|
|
expect(workspaceTestScript).toContain('set -eo pipefail');
|
|
// The extracted step runner has to fail closed the same way. notify-failure
|
|
// reads `gh issue list ... | jq -c ...` inside a command substitution, and
|
|
// a connection-level gh failure leaves jq exiting 0 on empty input, so
|
|
// without pipefail an unreachable API reads as "no existing issue".
|
|
expect(releaseStepScript).toMatch(/^set -eo pipefail$/m);
|
|
// Vitest colours its summaries from the mere presence of CI, and a
|
|
// coloured summary sits escape bytes between a label and its value, so
|
|
// every anchored pattern in the guard matches nothing: the pass-through
|
|
// is never granted again and each transport timeout reddens the release.
|
|
// Quoted in YAML, so it parses to the string rather than a boolean.
|
|
expect(testStep.env.NO_COLOR).toBe('true');
|
|
|
|
const workspacePackages = getTestCiWorkspaces();
|
|
|
|
expect(workspacePackages.length).toBeGreaterThan(0);
|
|
for (const [path, packageJson] of workspacePackages) {
|
|
// test:release:workspaces appends --shard/--passWithNoTests to the
|
|
// script body, so the flags only reach vitest when the vitest
|
|
// invocation is the LAST command in the chain; commands before it
|
|
// (e.g. sdk-typescript's typecheck:public-surface) stay accepted.
|
|
expect(
|
|
packageJson.scripts['test:ci'].split('&&').pop()?.trim(),
|
|
path,
|
|
).toMatch(/^vitest run(?:\s|$)/);
|
|
}
|
|
});
|
|
|
|
it('passes --retry unless the operator switched it off', () => {
|
|
// The flag reaches every workspace's vitest, where a command line option
|
|
// outranks the config. Every schedule now retries by default; the only
|
|
// way off is the operator sentinel, and it must omit the flag rather
|
|
// than zero it — --retry=0 would switch off a workspace's own retry
|
|
// (packages/sdk-typescript) on this lane alone.
|
|
for (const [retry, expected] of [
|
|
['2', '--retry=2'],
|
|
['', null],
|
|
// 'off' must omit the flag, not pass --retry=0: that would outrank a
|
|
// workspace's own config-level retry.
|
|
['off', null],
|
|
]) {
|
|
const dir = mkdtempSync(join(tmpdir(), 'release-retry-'));
|
|
try {
|
|
const stub = join(dir, 'npm');
|
|
writeFileSync(stub, '#!/bin/sh\nprintf "%s\\0" "$@"\n');
|
|
chmodSync(stub, 0o755);
|
|
|
|
const result = spawnSync(
|
|
'bash',
|
|
['-e', '-o', 'pipefail', workspaceTestScriptPath, '1'],
|
|
{
|
|
env: {
|
|
...process.env,
|
|
PATH: `${dir}:${process.env['PATH']}`,
|
|
VITEST_RETRY: retry,
|
|
},
|
|
encoding: 'utf8',
|
|
},
|
|
);
|
|
|
|
expect(result.status, result.stderr).toBe(0);
|
|
const args = result.stdout.split('\0');
|
|
expect(args.pop()).toBe('');
|
|
expect(args, retry).toContain('--passWithNoTests');
|
|
// An empty positional would reach vitest as a test-name filter.
|
|
expect(args, retry).not.toContain('');
|
|
if (expected) {
|
|
expect(args, retry).toContain(expected);
|
|
} else {
|
|
expect(
|
|
args.some((arg) => arg.startsWith('--retry')),
|
|
retry,
|
|
).toBe(false);
|
|
}
|
|
} finally {
|
|
rmSync(dir, { recursive: true, force: true });
|
|
}
|
|
}
|
|
});
|
|
|
|
it('stands down millisecond latency budgets on the shared ECS pool', () => {
|
|
// ci.yml's unit lane sets this on the same fleet predicate (#10870): a
|
|
// wall-clock budget written on a developer machine measures the
|
|
// neighbours on the shared pool, where one shard of the same suite
|
|
// measures 6.7 or 36 minutes depending on placement. The release lane
|
|
// runs the same suites on the same ecs-qwen hosts, so the budgets stand
|
|
// down here too — a pool-hosted shard otherwise reddens a release on
|
|
// contention while the commit itself is green. The relaxed
|
|
// poolMultiplier bound still applies where the duration is the property
|
|
// under test.
|
|
const testStep = releaseYaml.jobs.workspace_tests.steps.find(
|
|
(step) => step.name === 'Run Workspace Tests',
|
|
);
|
|
expect(testStep.env.QWEN_SKIP_LATENCY_BUDGETS).toBe(
|
|
"${{ startsWith(runner.name, 'ecs-qwen-') && '1' || '' }}",
|
|
);
|
|
});
|
|
|
|
it('lets an operator retune the workspace shard timeout without a PR', () => {
|
|
// A shard's runtime tracks how busy the reserved host is, not the suite:
|
|
// the same third measured 6.7 minutes on a quiet host and 36 on a
|
|
// contended one, and 45 killed shards at the boundary with every executed
|
|
// suite green (run 33713579913, both attempts). release.yml is
|
|
// code-owned, so a literal here costs a review every time the fleet
|
|
// moves; the variable is the same runtime knob QWEN_CI_VITEST_MAX_WORKERS
|
|
// and QWEN_RELEASE_VITEST_RETRY already use.
|
|
expect(workflow).toContain(
|
|
`timeout-minutes: "\${{ fromJSON(vars.QWEN_RELEASE_WORKSPACE_TIMEOUT_MINUTES || '45') }}"`,
|
|
);
|
|
// fromJSON, not the bare variable: timeout-minutes takes a number, and an
|
|
// unset variable has to fall back rather than render an empty string.
|
|
expect(releaseYaml.jobs.workspace_tests['timeout-minutes']).toContain(
|
|
'fromJSON(',
|
|
);
|
|
});
|
|
|
|
it('lets an operator retune the quality lane timeouts without a PR', () => {
|
|
// Run 33963757913 lost both lanes at their timeout boundary on a
|
|
// contended hk4 with no failing step: quality_static at 30m13s in Run
|
|
// Lint, quality_build at 45m13s in Pack Build Outputs. A timeout kill
|
|
// reports as 'cancelled', the quality aggregate fails closed on it, and
|
|
// the release failure was filed against a tree with nothing to fix. Same
|
|
// remedy as workspace_tests: a runtime knob. The static default moved
|
|
// 30 -> 60 on the pricing evidence its lane comment records (#11121);
|
|
// the build default stays a fleet-load call for the operator.
|
|
// Value format, relocated here from the release.yml lane comment this diff
|
|
// condensed to one line: the knob is free text under a name ending in
|
|
// MINUTES and takes a bare positive integer. `1h` is an expression error —
|
|
// GitHub never creates the job and the run blames 'a workflow file issue'
|
|
// instead of naming the variable — while `70 minutes` silently means 70,
|
|
// because fromJSON keeps the leading number and drops the rest. The
|
|
// aggregate still fails closed on a lane that never ran, so a bad value
|
|
// refuses a release rather than shipping one unvalidated. `0` is not 'no
|
|
// limit' either: the string '0' is truthy in a GitHub `||`, so the knob
|
|
// wins over the fallback and GitHub then ignores the zero bound it was
|
|
// handed. quality_static, quality_build and workspace_tests all share this
|
|
// contract, so it lives beside the assertions that pin their expressions.
|
|
expect(releaseYaml.jobs.quality_static['timeout-minutes']).toBe(
|
|
"${{ fromJSON(vars.QWEN_RELEASE_STATIC_TIMEOUT_MINUTES || '60') }}",
|
|
);
|
|
expect(releaseYaml.jobs.quality_build['timeout-minutes']).toBe(
|
|
"${{ fromJSON(vars.QWEN_RELEASE_BUILD_TIMEOUT_MINUTES || '45') }}",
|
|
);
|
|
// A budget the run never states is indistinguishable from a budget nobody
|
|
// set: a misspelled variable name renders '' and the lane dies at its
|
|
// default again, with nothing in the log to reconcile the two. Each
|
|
// tunable lane reports the bound it resolved — through the SAME expression
|
|
// the timeout uses, so the trace cannot drift from what the runner
|
|
// enforces — and whether the variable reached the job at all.
|
|
for (const [id, variable] of [
|
|
['quality_static', 'QWEN_RELEASE_STATIC_TIMEOUT_MINUTES'],
|
|
['quality_build', 'QWEN_RELEASE_BUILD_TIMEOUT_MINUTES'],
|
|
]) {
|
|
const bound = releaseYaml.jobs[id]['timeout-minutes'];
|
|
const report = releaseYaml.jobs[id].steps.find(
|
|
(step) => step.env?.BUDGET === bound,
|
|
);
|
|
expect(report?.name, id).toBe('Report timeout budget');
|
|
expect(report.run, id).toContain(`::notice::${id} timeout budget`);
|
|
expect(report.run, id).toContain(`${variable} set=\${VARIABLE_SET}`);
|
|
expect(report.env.VARIABLE_SET, id).toBe(
|
|
`\${{ vars.${variable} != '' }}`,
|
|
);
|
|
}
|
|
});
|
|
|
|
it('names which failure this is, and never changes the exit code', () => {
|
|
// A shard that died on Vitest's own worker RPC timing out reads
|
|
// identically to a real break, and this release lost two attempts before
|
|
// anyone could tell them apart (run 33713579913). The annotation says
|
|
// which; the child's status is re-raised untouched, so no reading of the
|
|
// log can turn a failure green except the one deliberate pass-through.
|
|
//
|
|
// That pass-through is granted by Vitest's own count of unhandled errors
|
|
// (`Errors N errors`) matching how many carried the transport's message
|
|
// — not by recognising a crash from its header. The header is
|
|
// producer-chosen, so a pattern over headers is incomplete by
|
|
// construction; the rows below carry the shapes that defeat one
|
|
// (suffix-less class, no class at all) and the shapes that defeat the
|
|
// count (an extra error the timeouts do not account for, a summary the
|
|
// run never reached, words a test merely printed).
|
|
const timeout = 'Error: [vitest-worker]: Timeout calling "x"';
|
|
const tally = ' Tests 10614 passed (10614)';
|
|
const passedThrough =
|
|
'::warning title=Workspace tests passed through a Vitest transport timeout::';
|
|
const stands =
|
|
'::warning title=Workspace tests exited 1 on a Vitest transport timeout::';
|
|
|
|
for (const [label, stub, code, annotation, expected] of [
|
|
// A failing test names itself; an annotation would only add noise.
|
|
['failing test', ' FAIL src/a.test.ts > boom', 1, null],
|
|
// Vitest's worker RPC giving up says nothing about the product, and
|
|
// --retry cannot cover it — retries re-run failing TESTS while an
|
|
// unhandled error fails the run outright. Passed only with proof the
|
|
// run reached its end and that the transport accounts for every
|
|
// unhandled error Vitest counted. It cost this release three attempts.
|
|
[
|
|
'transport timeout, run completed',
|
|
`${timeout}\n${tally}\n Errors 1 error`,
|
|
1,
|
|
passedThrough,
|
|
0,
|
|
],
|
|
[
|
|
'transport timeout, killed by a signal',
|
|
`${timeout}\n${tally}\n Errors 1 error`,
|
|
137,
|
|
'::warning title=Workspace tests exited 137 on a Vitest transport timeout::',
|
|
],
|
|
// One more error than the transport accounts for: the run broke on
|
|
// something else as well, whatever its header said.
|
|
[
|
|
'transport timeout beside a real one',
|
|
`${timeout}\nError: write after end\n${tally}\n Errors 2 errors`,
|
|
1,
|
|
stands,
|
|
],
|
|
// The counts can also disagree with no crash in the log at all — the
|
|
// same transport message on two lines inflates `timeouts` past what
|
|
// Vitest counted. That is a fifth way to reach this refusal, so the
|
|
// annotation names it and prints both figures it compared; without
|
|
// them the oncall is told one of four things happened when none did.
|
|
[
|
|
'transport timeout counted twice against one unhandled error',
|
|
`${timeout}\n${timeout}\n${tally}\n Errors 1 error`,
|
|
1,
|
|
'::warning title=Workspace tests exited 1 on a Vitest transport timeout::A transport timeout the run cannot account for — no passing tally, a failing tally, a signal death, an unhandled error that was not the transport, or the two counts disagreeing for a reason this log does not show. The failure stands (status 1, 1 counted error(s) vs 2 transport line(s)); rerun the job.',
|
|
],
|
|
// The two shapes no header pattern reaches. A class whose name carries
|
|
// no Error/Exception suffix is not hypothetical — 26 of this repo's 293
|
|
// Error subclasses are named that way, four of them assigning the bare
|
|
// name to err.name — and a bare string throw prints under Vitest's own
|
|
// `Unknown Error:` heading, which a header matcher misses on the space.
|
|
// The count sees both, because it never looks at the header.
|
|
[
|
|
'transport timeout beside a suffix-less crash header',
|
|
`${timeout}\nPoolTimeout: worker pool exhausted\n${tally}\n Errors 2 errors`,
|
|
1,
|
|
stands,
|
|
],
|
|
[
|
|
'transport timeout beside a bare string throw',
|
|
`${timeout}\nUnknown Error: a bare string, no class header\n${tally}\n Errors 2 errors`,
|
|
1,
|
|
stands,
|
|
],
|
|
// Ordinary `Error:` lines are test output, not evidence of a break: the
|
|
// log of the run this guard was written for carries three of them as
|
|
// fixture data. A matcher over headers refuses the pass-through on
|
|
// those and reddens a release the guard exists to save; the count is
|
|
// unmoved by them.
|
|
[
|
|
'transport timeout beside Error: lines a test printed',
|
|
`${timeout}\nError: boom\nError: Not implemented: navigation\n${tally}\n Errors 1 error`,
|
|
1,
|
|
passedThrough,
|
|
0,
|
|
],
|
|
// `--workspaces` prints one summary per workspace into one log, so both
|
|
// figures are whole-file sums: a passing tally cannot cover a later
|
|
// workspace's crash, and two transport deaths in two workspaces still
|
|
// pass.
|
|
[
|
|
'tally, then a later workspace crashing',
|
|
`${timeout}\n${tally}\n Tests 8 passed (8)\n Errors 2 errors`,
|
|
1,
|
|
stands,
|
|
],
|
|
// The shape of the log this guard was written for: several transport
|
|
// deaths in one run, and the plural summary Vitest prints for more than
|
|
// one of them (run 33713579913).
|
|
[
|
|
'four transport deaths, four unhandled errors',
|
|
`${timeout}\n${timeout}\n${timeout}\n${timeout}\n${tally}\n Errors 4 errors`,
|
|
1,
|
|
passedThrough,
|
|
0,
|
|
],
|
|
[
|
|
'two workspaces, both lost to the transport',
|
|
`${timeout}\n Tests 5 passed (5)\n Errors 1 error\nError: [vitest-worker]: Timeout calling "y"\n Tests 7 passed (7)\n Errors 1 error`,
|
|
1,
|
|
passedThrough,
|
|
0,
|
|
],
|
|
// Absent evidence refuses the pass rather than granting it: no summary
|
|
// line at all, no passing tally, or a failing tally.
|
|
[
|
|
'transport timeout, no error summary to count',
|
|
`${timeout}\n${tally}`,
|
|
1,
|
|
stands,
|
|
],
|
|
[
|
|
'transport timeout, no tally to back it',
|
|
'Error: [vitest-worker]: Timeout calling "onTaskUpdate"',
|
|
1,
|
|
stands,
|
|
],
|
|
// Vitest's own summary is the only thing that grants the pass, so a
|
|
// run that never printed a tally does not get one even when its error
|
|
// count is all transport.
|
|
[
|
|
'error summary with no tally to back it',
|
|
`${timeout}\n Errors 1 error`,
|
|
1,
|
|
stands,
|
|
],
|
|
// `Timeout calling` in a test's own output is not the transport dying:
|
|
// the branch is entered on Vitest's own `[vitest-worker]:` message, so
|
|
// a log carrying only the words is unexplained, not passed through.
|
|
[
|
|
'Timeout calling printed by a test',
|
|
`Timeout calling the vendor API\n${tally}`,
|
|
1,
|
|
'::error title=Workspace tests exited 1 with no failing test::',
|
|
],
|
|
// ...and the count is anchored on the same message, so those words
|
|
// beside a real transport death do not inflate it into a mismatch.
|
|
[
|
|
'transport timeout, and Timeout calling printed by a test',
|
|
`${timeout}\nTimeout calling the vendor API\n${tally}\n Errors 1 error`,
|
|
1,
|
|
passedThrough,
|
|
0,
|
|
],
|
|
// ...and the summary sum is anchored on the section-line shape for the
|
|
// same reason. Vitest echoes a test's console output at column 0, so a
|
|
// workspace printing summary-shaped fixture data lands there; an
|
|
// unanchored `Errors N errors` match would add it to the sum, inflate
|
|
// `errors` past `timeouts`, and refuse a pass-through every test
|
|
// earned — the false-red this PR exists to remove, back again.
|
|
[
|
|
'transport timeout, and a summary-shaped line a test printed',
|
|
`${timeout}\nErrors 2 errors occurred in fixture data\n${tally}\n Errors 1 error`,
|
|
1,
|
|
passedThrough,
|
|
0,
|
|
],
|
|
[
|
|
'transport timeout, failing tally',
|
|
`${timeout}\n Tests 3 failed | 10611 passed (10614)\n Errors 1 error`,
|
|
1,
|
|
stands,
|
|
],
|
|
// One workspace can print a passing tally while a later one fails
|
|
// without ever emitting a FAIL line, so the failing tally is checked
|
|
// across the whole log rather than trusted to the branch above.
|
|
[
|
|
'passing tally in one workspace, failing tally in another',
|
|
`${timeout}\n Tests 10 passed (10)\n Test Files 1 failed (3)\n Errors 1 error`,
|
|
1,
|
|
stands,
|
|
],
|
|
[
|
|
'unexplained',
|
|
'something odd',
|
|
7,
|
|
'::error title=Workspace tests exited 7 with no failing test::',
|
|
],
|
|
]) {
|
|
const dir = mkdtempSync(join(tmpdir(), 'release-failure-'));
|
|
try {
|
|
const stubPath = join(dir, 'npm');
|
|
writeFileSync(stubPath, `#!/bin/sh\necho '${stub}'\nexit ${code}\n`);
|
|
chmodSync(stubPath, 0o755);
|
|
|
|
const result = spawnSync(
|
|
'bash',
|
|
['-e', '-o', 'pipefail', workspaceTestScriptPath, '1'],
|
|
{
|
|
env: {
|
|
...process.env,
|
|
PATH: `${dir}:${process.env['PATH']}`,
|
|
VITEST_RETRY: '2',
|
|
RUNNER_TEMP: dir,
|
|
},
|
|
encoding: 'utf8',
|
|
},
|
|
);
|
|
|
|
expect(result.status, label).toBe(expected ?? code);
|
|
if (annotation) {
|
|
expect(result.stdout, label).toContain(annotation);
|
|
} else {
|
|
expect(result.stdout, label).not.toContain('::warning title=');
|
|
expect(result.stdout, label).not.toContain('::error title=');
|
|
}
|
|
} finally {
|
|
rmSync(dir, { recursive: true, force: true });
|
|
}
|
|
}
|
|
});
|
|
|
|
it('discovers at least one test file in every test:ci workspace', () => {
|
|
// --passWithNoTests lets a shard that received no files exit 0, but it
|
|
// would also turn a workspace that lost every test file green in all
|
|
// three shards; the monolithic test:release exited 1 in that case and
|
|
// blocked the release. This ratchet runs in the same test:scripts lane
|
|
// (quality_scripts) that gates the release, so a workspace whose test
|
|
// discovery comes up empty blocks publish again.
|
|
//
|
|
// Discovery mirrors vitest's default include pattern. Every workspace's
|
|
// test:ci runs vitest with that default or a narrower include (the
|
|
// custom configs list 'src/**/*.test.ts' or 'scripts/**/*.test.js'),
|
|
// so zero matches here means zero discoverable tests under vitest too.
|
|
const testCiWorkspaces = getTestCiWorkspaces();
|
|
expect(testCiWorkspaces.length).toBeGreaterThan(0);
|
|
for (const [path] of testCiWorkspaces) {
|
|
const testFiles = globSync('**/*.{test,spec}.?(c|m)[jt]s?(x)', {
|
|
cwd: dirname(path),
|
|
ignore: ['**/node_modules/**', '**/dist/**', '**/e2e/**'],
|
|
});
|
|
expect(
|
|
testFiles.length,
|
|
`${path} defines test:ci but matches no test file; every shard ` +
|
|
'would pass with zero tests executed for this workspace',
|
|
).toBeGreaterThan(0);
|
|
}
|
|
});
|
|
|
|
it('keeps publishing behind one fail-closed quality aggregate', () => {
|
|
const quality = releaseYaml.jobs.quality;
|
|
expect(quality.needs).toEqual([
|
|
'prepare',
|
|
'quality_static',
|
|
'quality_build',
|
|
'quality_typecheck',
|
|
'workspace_tests',
|
|
'quality_scripts',
|
|
]);
|
|
// !cancelled(), not always(): a cancelled run leaves the aggregate
|
|
// skipped, so notify_failure's needs.quality.result == 'failure' gate
|
|
// stays closed for runs an operator stopped on purpose. Failed
|
|
// components still run the aggregate and fail it. The prepare gate
|
|
// skips the aggregate when prepare never ran (forks): otherwise
|
|
// !cancelled() overrides the needs gate and turns five skipped lanes
|
|
// into a quality failure that opens notify_failure on a fork dispatch.
|
|
expect(quality.if).toBe(
|
|
"${{ !cancelled() && needs.prepare.result == 'success' && github.event.inputs.force_skip_tests != 'true' }}",
|
|
);
|
|
// The wipe anchor precedes the verify step once quality is pool-routed;
|
|
// locate the aggregation by name so the pool hygiene cannot shadow it.
|
|
const verify = quality.steps.find(
|
|
(step) => step.name === 'Verify quality results',
|
|
);
|
|
expect(verify.run).toContain('if [[ "${result}" != \'success\' ]]');
|
|
// Pin the full result mapping, not just the loop's shape: dropping one
|
|
// loop entry or remapping an env var to another job's result must fail
|
|
// here instead of letting a failed component publish.
|
|
expect(verify.env).toEqual({
|
|
STATIC_RESULT: '${{ needs.quality_static.result }}',
|
|
BUILD_RESULT: '${{ needs.quality_build.result }}',
|
|
TYPECHECK_RESULT: '${{ needs.quality_typecheck.result }}',
|
|
WORKSPACE_TEST_RESULT: '${{ needs.workspace_tests.result }}',
|
|
SCRIPT_TEST_RESULT: '${{ needs.quality_scripts.result }}',
|
|
});
|
|
for (const name of [
|
|
'STATIC_RESULT',
|
|
'BUILD_RESULT',
|
|
'TYPECHECK_RESULT',
|
|
'WORKSPACE_TEST_RESULT',
|
|
'SCRIPT_TEST_RESULT',
|
|
]) {
|
|
expect(verify.run, name).toContain('"${' + name + '}"');
|
|
}
|
|
expect(releaseYaml.jobs.notify_failure.if).toContain(
|
|
"needs.quality.result == 'failure'",
|
|
);
|
|
expect(releaseYaml.jobs.publish.needs).toContain('quality');
|
|
expect(releaseYaml.jobs.publish.needs).not.toContain('workspace_tests');
|
|
});
|
|
|
|
it('keeps every component quality job behind the force_skip_tests gate', () => {
|
|
// The aggregate's gate alone is not enough: an emergency
|
|
// `force_skip_tests: 'true'` dispatch must skip each component lane
|
|
// directly too, otherwise a red lane still runs and blocks the very
|
|
// release the override exists to unblock.
|
|
for (const id of [
|
|
'quality_static',
|
|
'quality_build',
|
|
'quality_typecheck',
|
|
'workspace_tests',
|
|
'quality_scripts',
|
|
]) {
|
|
expect(releaseYaml.jobs[id].if, id).toContain(
|
|
"github.event.inputs.force_skip_tests != 'true'",
|
|
);
|
|
}
|
|
});
|
|
|
|
it('keeps workspace cleanup from inspecting or signaling host processes', () => {
|
|
expect(canonicalWipe).not.toMatch(/(?:^|\s)(?:ps|kill|pkill)\s/m);
|
|
});
|
|
|
|
it.skipIf(
|
|
!hasGnuRealpath || process.getuid?.() === 0 || process.platform === 'win32',
|
|
)('executes the workspace wipe against guard branches', () => {
|
|
const wipeScript = canonicalWipe;
|
|
|
|
const runWipe = (envOverrides, { preCreateWorkspace } = {}) => {
|
|
const base = mkdtempSync(join(tmpdir(), 'release-wipe-behavioral-'));
|
|
const workspace = join(base, 'workspace');
|
|
mkdirSync(workspace);
|
|
if (preCreateWorkspace) preCreateWorkspace(base, workspace);
|
|
const env = {
|
|
...process.env,
|
|
GITHUB_WORKSPACE: workspace,
|
|
RUNNER_WORKSPACE: base,
|
|
RUNNER_TEMP: join(base, 'temp'),
|
|
RUNNER_TOOL_CACHE: join(base, 'tool-cache'),
|
|
GITHUB_ENV: join(base, 'github-env'),
|
|
HOME: join(base, 'home'),
|
|
XDG_CONFIG_HOME: join(base, 'home', '.config'),
|
|
...envOverrides,
|
|
};
|
|
mkdirSync(env.HOME);
|
|
mkdirSync(env.RUNNER_TEMP);
|
|
mkdirSync(join(env.RUNNER_TOOL_CACHE, 'node'), { recursive: true });
|
|
mkdirSync(join(env.HOME, '.docker'));
|
|
writeFileSync(
|
|
join(env.HOME, '.gitconfig'),
|
|
'[credential]\n\thelper = !false\n',
|
|
);
|
|
writeFileSync(join(env.HOME, '.gitconfig.lock'), 'stale');
|
|
writeFileSync(
|
|
join(env.HOME, '.docker', 'config.json'),
|
|
'{"proxies":{"default":{"httpProxy":"http://attacker"}}}',
|
|
);
|
|
env.GIT_CONFIG_GLOBAL = join(env.HOME, '.gitconfig');
|
|
env.GIT_CONFIG_COUNT = '1';
|
|
env.GIT_CONFIG_KEY_0 = 'credential.helper';
|
|
env.GIT_CONFIG_VALUE_0 = '!false';
|
|
return {
|
|
result: spawnSync('bash', ['-e', '-o', 'pipefail', '-c', wipeScript], {
|
|
encoding: 'utf8',
|
|
env,
|
|
}),
|
|
base,
|
|
workspace,
|
|
githubEnv: env.GITHUB_ENV,
|
|
env,
|
|
};
|
|
};
|
|
|
|
// Happy path: a normal workspace inside the runner workspace is wiped,
|
|
// including subdirectories (the wipe's core property: recursive removal
|
|
// of all persisted entries, not just files).
|
|
{
|
|
const { result, base, workspace, githubEnv, env } = runWipe(
|
|
{},
|
|
{
|
|
preCreateWorkspace: (_base, ws) => {
|
|
writeFileSync(join(ws, 'leftover.txt'), 'stale');
|
|
mkdirSync(join(ws, 'leftover-dir'));
|
|
writeFileSync(join(ws, 'leftover-dir', 'nested.txt'), 'stale');
|
|
},
|
|
},
|
|
);
|
|
try {
|
|
expect(result.status).toBe(0);
|
|
const entries = readdirSync(workspace);
|
|
expect(entries).toHaveLength(0);
|
|
const stateEnv = readFileSync(githubEnv, 'utf8');
|
|
expect(stateEnv).toContain('GIT_CONFIG_COUNT=0\n');
|
|
expect(stateEnv).toContain('GIT_CONFIG_NOSYSTEM=1\n');
|
|
expect(stateEnv).toContain('GIT_CONFIG_PARAMETERS=\n');
|
|
expect(stateEnv).toMatch(
|
|
/GIT_CONFIG_GLOBAL=.*\/release-state\.[^/]+\/gitconfig\n/,
|
|
);
|
|
expect(stateEnv).toMatch(
|
|
/NPM_CONFIG_USERCONFIG=.*\/release-state\.[^/]+\/npmrc\n/,
|
|
);
|
|
expect(stateEnv).toMatch(
|
|
/DOCKER_CONFIG=.*\/release-state\.[^/]+\/docker\n/,
|
|
);
|
|
expect(stateEnv).toMatch(
|
|
/GH_CONFIG_DIR=.*\/release-state\.[^/]+\/gh\n/,
|
|
);
|
|
const isolatedEnv = { ...env };
|
|
for (const line of stateEnv.trimEnd().split('\n')) {
|
|
const separator = line.indexOf('=');
|
|
isolatedEnv[line.slice(0, separator)] = line.slice(separator + 1);
|
|
}
|
|
expect(
|
|
spawnSync(
|
|
'git',
|
|
['config', '--global', '--get', 'credential.helper'],
|
|
{ env: isolatedEnv },
|
|
).status,
|
|
).not.toBe(0);
|
|
expect(readdirSync(isolatedEnv.DOCKER_CONFIG)).toHaveLength(0);
|
|
// The sibling tool cache SURVIVES the wipe: the sweep is scoped
|
|
// to the workspace, and the pool-wide cache stays untouched on
|
|
// purpose — the pool-routed release lane never reads it, while
|
|
// other pool lanes resolve Node from it through un-gated
|
|
// setup-node.
|
|
expect(lstatSync(join(base, 'tool-cache', 'node')).isDirectory()).toBe(
|
|
true,
|
|
);
|
|
} finally {
|
|
rmSync(base, { recursive: true, force: true });
|
|
}
|
|
}
|
|
|
|
// Symlink heal: a workspace replaced with a symlink inside the runner
|
|
// workspace is removed and recreated, then wiped. The decoy target
|
|
// is a real file so the test can verify `rm -f` removed only the
|
|
// link itself and did not follow/delete the target.
|
|
{
|
|
const { result, base, workspace } = runWipe(
|
|
{},
|
|
{
|
|
preCreateWorkspace: (b, ws) => {
|
|
rmSync(ws, { recursive: true, force: true });
|
|
const decoyTarget = join(b, 'decoy-target');
|
|
writeFileSync(decoyTarget, 'must-survive');
|
|
symlinkSync(decoyTarget, ws);
|
|
},
|
|
},
|
|
);
|
|
try {
|
|
expect(result.status).toBe(0);
|
|
expect(`${result.stdout}${result.stderr}`).toContain(
|
|
'healing workspace',
|
|
);
|
|
const stat = lstatSync(workspace);
|
|
expect(stat.isDirectory()).toBe(true);
|
|
// The decoy target must survive: rm -f on the raw path removes
|
|
// the link itself and never follows it.
|
|
expect(readFileSync(join(base, 'decoy-target'), 'utf8')).toBe(
|
|
'must-survive',
|
|
);
|
|
} finally {
|
|
rmSync(base, { recursive: true, force: true });
|
|
}
|
|
}
|
|
|
|
// Pool geometry: the tool cache is a SIBLING of the runner workspace
|
|
// (<root>/_work/_tool vs <root>/_work/qwen-code) — the standard
|
|
// self-hosted layout. The wipe must leave it untouched: other pool
|
|
// lanes resolve Node from it through un-gated setup-node, while the
|
|
// pool-routed release jobs never read it.
|
|
{
|
|
const runnerRoot = mkdtempSync(join(tmpdir(), 'release-wipe-pool-'));
|
|
const rws = join(runnerRoot, '_work', 'qwen-code');
|
|
const workspace = join(rws, 'qwen-code');
|
|
mkdirSync(workspace, { recursive: true });
|
|
writeFileSync(join(workspace, 'leftover.txt'), 'stale');
|
|
try {
|
|
const env = {
|
|
...process.env,
|
|
GITHUB_WORKSPACE: workspace,
|
|
RUNNER_WORKSPACE: rws,
|
|
RUNNER_TEMP: join(runnerRoot, 'temp'),
|
|
RUNNER_TOOL_CACHE: join(runnerRoot, '_work', '_tool'),
|
|
GITHUB_ENV: join(runnerRoot, 'github-env'),
|
|
HOME: join(runnerRoot, 'home'),
|
|
XDG_CONFIG_HOME: join(runnerRoot, 'home', '.config'),
|
|
};
|
|
mkdirSync(env.HOME);
|
|
mkdirSync(env.RUNNER_TEMP);
|
|
mkdirSync(join(env.RUNNER_TOOL_CACHE, 'node'), { recursive: true });
|
|
writeFileSync(
|
|
join(env.RUNNER_TOOL_CACHE, 'node', 'marker.txt'),
|
|
'pool-node',
|
|
);
|
|
const result = spawnSync(
|
|
'bash',
|
|
['-e', '-o', 'pipefail', '-c', wipeScript],
|
|
{ encoding: 'utf8', env },
|
|
);
|
|
expect(result.status).toBe(0);
|
|
expect(readdirSync(workspace)).toHaveLength(0);
|
|
// The sibling tool cache's node directory SURVIVES the wipe.
|
|
expect(
|
|
lstatSync(join(env.RUNNER_TOOL_CACHE, 'node')).isDirectory(),
|
|
).toBe(true);
|
|
expect(
|
|
readFileSync(
|
|
join(env.RUNNER_TOOL_CACHE, 'node', 'marker.txt'),
|
|
'utf8',
|
|
),
|
|
).toBe('pool-node');
|
|
} finally {
|
|
rmSync(runnerRoot, { recursive: true, force: true });
|
|
}
|
|
}
|
|
|
|
// Workspace outside runner workspace: refused.
|
|
{
|
|
const outside = mkdtempSync(join(tmpdir(), 'release-wipe-outside-'));
|
|
const base = mkdtempSync(join(tmpdir(), 'release-wipe-runner-'));
|
|
try {
|
|
const env = {
|
|
...process.env,
|
|
GITHUB_WORKSPACE: outside,
|
|
RUNNER_WORKSPACE: base,
|
|
RUNNER_TEMP: join(base, 'temp'),
|
|
RUNNER_TOOL_CACHE: join(base, 'tool-cache'),
|
|
GITHUB_ENV: join(base, 'github-env'),
|
|
HOME: join(base, 'home'),
|
|
XDG_CONFIG_HOME: join(base, 'home', '.config'),
|
|
};
|
|
mkdirSync(env.HOME);
|
|
mkdirSync(env.RUNNER_TEMP);
|
|
const result = spawnSync(
|
|
'bash',
|
|
['-e', '-o', 'pipefail', '-c', wipeScript],
|
|
{ encoding: 'utf8', env },
|
|
);
|
|
expect(result.status).not.toBe(0);
|
|
expect(`${result.stdout}${result.stderr}`).toContain(
|
|
'refusing to wipe workspace outside the runner workspace',
|
|
);
|
|
} finally {
|
|
rmSync(outside, { recursive: true, force: true });
|
|
rmSync(base, { recursive: true, force: true });
|
|
}
|
|
}
|
|
|
|
// Path with '..' that realpath resolves inside the runner workspace:
|
|
// canonicalization succeeds, containment passes, wipe proceeds.
|
|
{
|
|
const base = mkdtempSync(join(tmpdir(), 'release-wipe-dots-'));
|
|
const workspace = join(base, 'workspace');
|
|
mkdirSync(workspace);
|
|
mkdirSync(join(base, 'sub'));
|
|
writeFileSync(join(workspace, 'leftover.txt'), 'stale');
|
|
try {
|
|
const env = {
|
|
...process.env,
|
|
// String concatenation preserves the literal '..' segment —
|
|
// path.join would normalize it away before the script sees it.
|
|
GITHUB_WORKSPACE: `${base}/sub/../workspace`,
|
|
RUNNER_WORKSPACE: base,
|
|
RUNNER_TEMP: join(base, 'temp'),
|
|
RUNNER_TOOL_CACHE: join(base, 'tool-cache'),
|
|
GITHUB_ENV: join(base, 'github-env'),
|
|
HOME: join(base, 'home'),
|
|
XDG_CONFIG_HOME: join(base, 'home', '.config'),
|
|
};
|
|
mkdirSync(env.HOME);
|
|
mkdirSync(env.RUNNER_TEMP);
|
|
const result = spawnSync(
|
|
'bash',
|
|
['-e', '-o', 'pipefail', '-c', wipeScript],
|
|
{ encoding: 'utf8', env },
|
|
);
|
|
expect(result.status).toBe(0);
|
|
const entries = readdirSync(workspace);
|
|
expect(entries).toHaveLength(0);
|
|
} finally {
|
|
rmSync(base, { recursive: true, force: true });
|
|
}
|
|
}
|
|
|
|
// Symlinked runner workspace: refused BEFORE any chown/chmod/wipe —
|
|
// a prior pool job may have replaced it with a link to redirect the
|
|
// whole guard chain (heal, containment, wipe) to an attacker-chosen
|
|
// location.
|
|
{
|
|
const outside = mkdtempSync(join(tmpdir(), 'release-rws-target-'));
|
|
mkdirSync(join(outside, 'qwen-code'));
|
|
const decoy = join(outside, 'qwen-code', 'decoy.txt');
|
|
writeFileSync(decoy, 'must-survive');
|
|
chmodSync(decoy, 0o400);
|
|
const base = mkdtempSync(join(tmpdir(), 'release-rws-runner-'));
|
|
const rwsLink = join(base, 'rws-link');
|
|
symlinkSync(outside, rwsLink);
|
|
try {
|
|
const env = {
|
|
...process.env,
|
|
GITHUB_WORKSPACE: join(rwsLink, 'qwen-code'),
|
|
RUNNER_WORKSPACE: rwsLink,
|
|
RUNNER_TEMP: join(base, 'temp'),
|
|
RUNNER_TOOL_CACHE: join(base, 'tool-cache'),
|
|
GITHUB_ENV: join(base, 'github-env'),
|
|
HOME: join(base, 'home'),
|
|
XDG_CONFIG_HOME: join(base, 'home', '.config'),
|
|
};
|
|
mkdirSync(env.HOME);
|
|
mkdirSync(env.RUNNER_TEMP);
|
|
const result = spawnSync(
|
|
'bash',
|
|
['-e', '-o', 'pipefail', '-c', wipeScript],
|
|
{ encoding: 'utf8', env },
|
|
);
|
|
expect(result.status).not.toBe(0);
|
|
expect(`${result.stdout}${result.stderr}`).toContain(
|
|
'refusing to wipe: runner workspace is a symlink',
|
|
);
|
|
// Decoy intact — and the ownership ladder did not reach it.
|
|
expect(readFileSync(decoy, 'utf8')).toBe('must-survive');
|
|
expect(lstatSync(decoy).mode & 0o777).toBe(0o400);
|
|
} finally {
|
|
rmSync(outside, { recursive: true, force: true });
|
|
rmSync(base, { recursive: true, force: true });
|
|
}
|
|
}
|
|
|
|
// Same refusal when the redirected target has no qwen-code subdir:
|
|
// the heal arm must not mkdir at the attacker-chosen location.
|
|
{
|
|
const outside = mkdtempSync(join(tmpdir(), 'release-rws-empty-'));
|
|
const base = mkdtempSync(join(tmpdir(), 'release-rws-runner2-'));
|
|
const rwsLink = join(base, 'rws-link');
|
|
symlinkSync(outside, rwsLink);
|
|
try {
|
|
const env = {
|
|
...process.env,
|
|
GITHUB_WORKSPACE: join(rwsLink, 'qwen-code'),
|
|
RUNNER_WORKSPACE: rwsLink,
|
|
RUNNER_TEMP: join(base, 'temp'),
|
|
RUNNER_TOOL_CACHE: join(base, 'tool-cache'),
|
|
GITHUB_ENV: join(base, 'github-env'),
|
|
HOME: join(base, 'home'),
|
|
XDG_CONFIG_HOME: join(base, 'home', '.config'),
|
|
};
|
|
mkdirSync(env.HOME);
|
|
mkdirSync(env.RUNNER_TEMP);
|
|
const result = spawnSync(
|
|
'bash',
|
|
['-e', '-o', 'pipefail', '-c', wipeScript],
|
|
{ encoding: 'utf8', env },
|
|
);
|
|
expect(result.status).not.toBe(0);
|
|
expect(`${result.stdout}${result.stderr}`).toContain(
|
|
'refusing to wipe: runner workspace is a symlink',
|
|
);
|
|
expect(existsSync(join(outside, 'qwen-code'))).toBe(false);
|
|
} finally {
|
|
rmSync(outside, { recursive: true, force: true });
|
|
rmSync(base, { recursive: true, force: true });
|
|
}
|
|
}
|
|
|
|
// Trailing slash on a symlinked runner workspace: [ -L ] does not see
|
|
// the link through a trailing slash — path resolution dereferences it —
|
|
// while realpath -m canonicalizes THROUGH it, re-rooting the containment
|
|
// allow-list at the link's target. The raw value must be stripped
|
|
// before the -L test (the GITHUB_WORKSPACE side's ordering), or a
|
|
// mangled env carrying one trailing slash defeats the refusal.
|
|
{
|
|
const outside = mkdtempSync(join(tmpdir(), 'release-rws-slash-'));
|
|
mkdirSync(join(outside, 'qwen-code'));
|
|
const decoy = join(outside, 'qwen-code', 'decoy.txt');
|
|
writeFileSync(decoy, 'must-survive');
|
|
const base = mkdtempSync(join(tmpdir(), 'release-rws-slash-runner-'));
|
|
const rwsLink = join(base, 'rws-link');
|
|
symlinkSync(outside, rwsLink);
|
|
try {
|
|
const env = {
|
|
...process.env,
|
|
GITHUB_WORKSPACE: join(rwsLink, 'qwen-code'),
|
|
// String concatenation keeps the literal trailing slash —
|
|
// path.join would normalize it away before the script sees it.
|
|
RUNNER_WORKSPACE: `${rwsLink}/`,
|
|
RUNNER_TEMP: join(base, 'temp'),
|
|
RUNNER_TOOL_CACHE: join(base, 'tool-cache'),
|
|
GITHUB_ENV: join(base, 'github-env'),
|
|
HOME: join(base, 'home'),
|
|
XDG_CONFIG_HOME: join(base, 'home', '.config'),
|
|
};
|
|
mkdirSync(env.HOME);
|
|
mkdirSync(env.RUNNER_TEMP);
|
|
const result = spawnSync(
|
|
'bash',
|
|
['-e', '-o', 'pipefail', '-c', wipeScript],
|
|
{ encoding: 'utf8', env },
|
|
);
|
|
expect(result.status).not.toBe(0);
|
|
expect(`${result.stdout}${result.stderr}`).toContain(
|
|
'refusing to wipe: runner workspace is a symlink',
|
|
);
|
|
expect(readFileSync(decoy, 'utf8')).toBe('must-survive');
|
|
} finally {
|
|
rmSync(outside, { recursive: true, force: true });
|
|
rmSync(base, { recursive: true, force: true });
|
|
}
|
|
}
|
|
|
|
// Symlinked INTERMEDIATE runner-workspace component: [ -L ] only
|
|
// sees the leaf, so a `_work` replaced with a link passes it, and
|
|
// realpath then re-roots the whole chain (heal, containment, wipe)
|
|
// at the link's target — the leaf test alone accepts the geometry
|
|
// and wipes the attacker-chosen tree. The lexical-vs-canonical
|
|
// comparison must refuse BEFORE any chown/chmod/wipe.
|
|
{
|
|
const outside = mkdtempSync(join(tmpdir(), 'release-rws-mid-'));
|
|
mkdirSync(join(outside, 'qwen-code', 'qwen-code'), { recursive: true });
|
|
const decoy = join(outside, 'qwen-code', 'qwen-code', 'decoy.txt');
|
|
writeFileSync(decoy, 'must-survive');
|
|
chmodSync(decoy, 0o400);
|
|
const runnerRoot = mkdtempSync(join(tmpdir(), 'release-rws-mid-runner-'));
|
|
symlinkSync(outside, join(runnerRoot, '_work'));
|
|
const rws = join(runnerRoot, '_work', 'qwen-code');
|
|
try {
|
|
const env = {
|
|
...process.env,
|
|
GITHUB_WORKSPACE: join(rws, 'qwen-code'),
|
|
RUNNER_WORKSPACE: rws,
|
|
RUNNER_TEMP: join(runnerRoot, 'temp'),
|
|
RUNNER_TOOL_CACHE: join(runnerRoot, 'tool-cache'),
|
|
GITHUB_ENV: join(runnerRoot, 'github-env'),
|
|
HOME: join(runnerRoot, 'home'),
|
|
XDG_CONFIG_HOME: join(runnerRoot, 'home', '.config'),
|
|
};
|
|
mkdirSync(env.HOME);
|
|
mkdirSync(env.RUNNER_TEMP);
|
|
const result = spawnSync(
|
|
'bash',
|
|
['-e', '-o', 'pipefail', '-c', wipeScript],
|
|
{ encoding: 'utf8', env },
|
|
);
|
|
expect(result.status).not.toBe(0);
|
|
expect(`${result.stdout}${result.stderr}`).toContain(
|
|
'refusing to wipe: runner workspace resolves through a symlinked component',
|
|
);
|
|
// Decoy intact — and the ownership ladder did not reach it.
|
|
expect(readFileSync(decoy, 'utf8')).toBe('must-survive');
|
|
expect(lstatSync(decoy).mode & 0o777).toBe(0o400);
|
|
expect(existsSync(env.GITHUB_ENV)).toBe(false);
|
|
} finally {
|
|
rmSync(outside, { recursive: true, force: true });
|
|
rmSync(runnerRoot, { recursive: true, force: true });
|
|
}
|
|
}
|
|
|
|
// Same refusal when the redirected target lacks the leaf: without
|
|
// the comparison the heal arm would judge the re-rooted parent
|
|
// INSIDE the re-rooted runner workspace and mkdir the leaf at the
|
|
// attacker-chosen location.
|
|
{
|
|
const outside = mkdtempSync(join(tmpdir(), 'release-rws-mid-empty-'));
|
|
mkdirSync(join(outside, 'qwen-code'));
|
|
const runnerRoot = mkdtempSync(
|
|
join(tmpdir(), 'release-rws-mid-empty-runner-'),
|
|
);
|
|
symlinkSync(outside, join(runnerRoot, '_work'));
|
|
const rws = join(runnerRoot, '_work', 'qwen-code');
|
|
try {
|
|
const env = {
|
|
...process.env,
|
|
GITHUB_WORKSPACE: join(rws, 'qwen-code'),
|
|
RUNNER_WORKSPACE: rws,
|
|
RUNNER_TEMP: join(runnerRoot, 'temp'),
|
|
RUNNER_TOOL_CACHE: join(runnerRoot, 'tool-cache'),
|
|
GITHUB_ENV: join(runnerRoot, 'github-env'),
|
|
HOME: join(runnerRoot, 'home'),
|
|
XDG_CONFIG_HOME: join(runnerRoot, 'home', '.config'),
|
|
};
|
|
mkdirSync(env.HOME);
|
|
mkdirSync(env.RUNNER_TEMP);
|
|
const result = spawnSync(
|
|
'bash',
|
|
['-e', '-o', 'pipefail', '-c', wipeScript],
|
|
{ encoding: 'utf8', env },
|
|
);
|
|
expect(result.status).not.toBe(0);
|
|
expect(`${result.stdout}${result.stderr}`).toContain(
|
|
'refusing to wipe: runner workspace resolves through a symlinked component',
|
|
);
|
|
expect(existsSync(join(outside, 'qwen-code', 'qwen-code'))).toBe(false);
|
|
} finally {
|
|
rmSync(outside, { recursive: true, force: true });
|
|
rmSync(runnerRoot, { recursive: true, force: true });
|
|
}
|
|
}
|
|
|
|
// Symlinked intermediate component BELOW the runner workspace: RWS
|
|
// itself is clean, so only the workspace-side comparison catches
|
|
// the re-rooting. The link points INSIDE the runner workspace,
|
|
// where the containment allow-list alone would pass — without the
|
|
// comparison the wipe would run on the wrong sibling directory.
|
|
{
|
|
const base = mkdtempSync(join(tmpdir(), 'release-ws-mid-'));
|
|
const rws = join(base, 'rws');
|
|
mkdirSync(join(rws, 'elsewhere', 'qwen-code'), { recursive: true });
|
|
const decoy = join(rws, 'elsewhere', 'qwen-code', 'decoy.txt');
|
|
writeFileSync(decoy, 'must-survive');
|
|
symlinkSync(join(rws, 'elsewhere'), join(rws, 'qwen-code'));
|
|
const workspace = join(rws, 'qwen-code', 'qwen-code');
|
|
try {
|
|
const env = {
|
|
...process.env,
|
|
GITHUB_WORKSPACE: workspace,
|
|
RUNNER_WORKSPACE: rws,
|
|
RUNNER_TEMP: join(base, 'temp'),
|
|
RUNNER_TOOL_CACHE: join(base, 'tool-cache'),
|
|
GITHUB_ENV: join(base, 'github-env'),
|
|
HOME: join(base, 'home'),
|
|
XDG_CONFIG_HOME: join(base, 'home', '.config'),
|
|
};
|
|
mkdirSync(env.HOME);
|
|
mkdirSync(env.RUNNER_TEMP);
|
|
const result = spawnSync(
|
|
'bash',
|
|
['-e', '-o', 'pipefail', '-c', wipeScript],
|
|
{ encoding: 'utf8', env },
|
|
);
|
|
expect(result.status).not.toBe(0);
|
|
expect(`${result.stdout}${result.stderr}`).toContain(
|
|
'refusing to wipe: workspace resolves through a symlinked component',
|
|
);
|
|
expect(readFileSync(decoy, 'utf8')).toBe('must-survive');
|
|
} finally {
|
|
rmSync(base, { recursive: true, force: true });
|
|
}
|
|
}
|
|
});
|
|
|
|
it('checks docker availability before the docker checkout', () => {
|
|
const steps = releaseYaml.jobs.integration_docker.steps;
|
|
const preflightIndex = steps.findIndex(
|
|
(step) => step.name === 'Check docker daemon',
|
|
);
|
|
const checkoutIndex = steps.findIndex((step) =>
|
|
String(step.uses ?? '').includes('actions/checkout'),
|
|
);
|
|
expect(preflightIndex).toBeGreaterThanOrEqual(0);
|
|
expect(preflightIndex).toBeLessThan(checkoutIndex);
|
|
// Pin the full fail-closed form, not just a substring: deleting
|
|
// 'exit 1' degrades the preflight to a warning (a dead daemon proceeds
|
|
// into checkout and dies deep in the docker tests), inverting the guard
|
|
// fails every healthy runner, and discarding docker's own output leaves
|
|
// the oncall unable to tell dockerd-down from socket-permission
|
|
// failures without first reaching the runner — all mutants probed
|
|
// green under the old substring pin.
|
|
expect(steps[preflightIndex].run).toMatch(
|
|
/^if ! docker_info_output="\$\(docker info 2>&1\)"; then\n {2}echo "::error::docker daemon is not reachable on this runner; docker integration tests cannot run\."\n {2}printf '%s\\n' "\$docker_info_output"\n {2}exit 1\nfi$/,
|
|
);
|
|
});
|
|
|
|
it('coordinates Docker image use with other shared-pool workflows', () => {
|
|
const steps = releaseYaml.jobs.integration_docker.steps;
|
|
const setupStep = steps.find((step) => step.name === 'Set up Docker');
|
|
const testStep = steps.find(
|
|
(step) => step.name === 'Run Docker Integration Tests',
|
|
);
|
|
|
|
expect(setupStep.if).toBe("${{ runner.environment != 'self-hosted' }}");
|
|
expect(testStep.run).toBe(
|
|
'.release-workflow/.github/scripts/run-release-docker-integration.sh',
|
|
);
|
|
expect(dockerIntegrationScript).toContain(
|
|
'docker-sandbox-build-release-${sandbox_revision}.lock',
|
|
);
|
|
expect(dockerIntegrationScript).toContain('flock --wait 1800 8');
|
|
expect(dockerIntegrationScript).toContain(
|
|
'exec 9>"${HOME}/.cache/qwen-code-ci/docker-sandbox-daemon.lock"',
|
|
);
|
|
expect(dockerIntegrationScript).toContain('-release-${sandbox_revision}');
|
|
expect(dockerIntegrationScript).toContain('--no-prune -i "$sandbox_image"');
|
|
expect(dockerIntegrationScript).toContain(
|
|
'export QWEN_SANDBOX_IMAGE="$sandbox_image_id"',
|
|
);
|
|
expect(dockerIntegrationScript).toContain('flock --shared --wait 1800 9');
|
|
// The daemon lock stays shared for the whole step: upgrading it to
|
|
// exclusive to build an image starves the build behind the test phase of
|
|
// any run already on the host (run 33637097713). Builds serialize on a
|
|
// separate host mutex that no test phase holds.
|
|
expect(dockerIntegrationScript).toContain(
|
|
'exec 7>"${HOME}/.cache/qwen-code-ci/docker-sandbox-build.lock"',
|
|
);
|
|
expect(dockerIntegrationScript).toContain('flock --wait 1800 7');
|
|
expect(dockerIntegrationScript).not.toContain('acquire_daemon_write_lock');
|
|
expect(dockerIntegrationScript).not.toContain('flock --unlock 9');
|
|
expect(dockerIntegrationScript).not.toContain('flock --nonblock 9');
|
|
// A flock lives on the open file description: a descendant inheriting
|
|
// the descriptor past the job would keep the lock on the host.
|
|
expect(dockerIntegrationScript).toContain(
|
|
'-i "$sandbox_image" 7>&- 8>&- 9>&-',
|
|
);
|
|
expect(dockerIntegrationScript).toContain('exec 7>&-');
|
|
expect(dockerIntegrationScript).toContain('exec 8>&-');
|
|
expect(dockerIntegrationScript).toContain('integration-tests cli 9>&-');
|
|
expect(dockerIntegrationScript).toContain(
|
|
'integration-tests interactive 9>&-',
|
|
);
|
|
});
|
|
|
|
it('prunes BuildKit cache and gates the docker data root before the build and the vitest phase', () => {
|
|
// #13479: run 37374675168 passed the job-start disk floor gate and the
|
|
// runner still died on ENOSPC 24 minutes into the docker test step. The
|
|
// lane reclaims dangling images and BuildKit cache on every path — a
|
|
// cached-image re-run reaches the pre-vitest gate with nothing else
|
|
// reclaimed ahead of it — prunes the labelled CI images under the build
|
|
// mutex when it does build, and gates the docker data root at a
|
|
// build-sized floor immediately before the build and at the repo's
|
|
// calibrated job floor before the vitest phase, so a saturated host
|
|
// fails fast with a legible error instead of the runner worker crashing
|
|
// mid-step.
|
|
const danglingPrune = dockerIntegrationScript.indexOf(
|
|
"timeout 20m docker image prune --force --filter 'until=24h'",
|
|
);
|
|
const cachePrune = dockerIntegrationScript.indexOf(
|
|
"timeout 20m docker builder prune --all --force --filter 'until=24h'",
|
|
);
|
|
const imagePrune = dockerIntegrationScript.indexOf(
|
|
"timeout 20m docker image prune --all --force --filter 'label=org.qwen-code.ci.sandbox=true' --filter 'until=24h'",
|
|
);
|
|
expect(danglingPrune).toBeGreaterThanOrEqual(0);
|
|
expect(cachePrune).toBeGreaterThan(danglingPrune);
|
|
expect(imagePrune).toBeGreaterThan(cachePrune);
|
|
// The prunes and the gate's `docker info` can execute while the shared
|
|
// host locks are held, so each carries a time bound: a wedged daemon GC
|
|
// cannot be allowed to starve a peer lane's 30-minute lock wait
|
|
// (run 33637097713). A `docker info` timeout degrades into the gate's
|
|
// warn-and-skip path.
|
|
expect(dockerIntegrationScript).toMatch(
|
|
/timeout [0-9]+m docker builder prune --all --force --filter 'until=24h'/,
|
|
);
|
|
expect(dockerIntegrationScript).toContain(
|
|
"timeout 60 docker info --format '{{.DockerRootDir}}'",
|
|
);
|
|
// The image-presence probe and the image-id read run with the shared
|
|
// host locks held, so both carry the same bound as the prunes — and the
|
|
// probe discriminates a timeout (124) from "image absent" instead of
|
|
// silently taking the rebuild branch against a wedged daemon.
|
|
expect(dockerIntegrationScript).toContain(
|
|
'timeout 60 docker image inspect "$sandbox_image"',
|
|
);
|
|
expect(dockerIntegrationScript).toContain(
|
|
'[ "$image_inspect_status" -eq 124 ]',
|
|
);
|
|
expect(dockerIntegrationScript).toMatch(
|
|
/timeout [0-9]+ docker image inspect --format '\{\{\.Id\}\}'/,
|
|
);
|
|
});
|
|
|
|
const dockerIntegrationScriptAbsolutePath = join(
|
|
process.cwd(),
|
|
'.github/scripts/run-release-docker-integration.sh',
|
|
);
|
|
|
|
// Drives the extracted script against stub git/node/docker/npm/npx
|
|
// binaries so the reclaim-and-gate contract is pinned by execution, not
|
|
// just text. `timeout` (coreutils) and `flock` are stubbed too because
|
|
// stock macOS ships neither, and the lock protocol only runs on the
|
|
// self-hosted path.
|
|
const runDockerIntegrationScript = ({
|
|
gateExit = '0',
|
|
imagePresent = false,
|
|
runnerEnvironment = 'github-hosted',
|
|
pruneFails = false,
|
|
imagePruneFails = false,
|
|
dockerInfoFails = false,
|
|
dockerRootMissing = false,
|
|
helperAbsent = false,
|
|
ambientDiskFloorKb,
|
|
ambientDiskFloorInodes,
|
|
dockerFloorKb,
|
|
dockerFloorInodes,
|
|
dockerPostFloorKb,
|
|
postGateExit,
|
|
inspectTimesOut = false,
|
|
} = {}) => {
|
|
const directory = mkdtempSync(
|
|
join(tmpdir(), 'release-docker-integration-'),
|
|
);
|
|
const bin = join(directory, 'bin');
|
|
const dockerRoot = join(directory, 'docker-root');
|
|
const callsLog = join(directory, 'calls.log');
|
|
mkdirSync(bin);
|
|
if (!dockerRootMissing) {
|
|
mkdirSync(dockerRoot);
|
|
}
|
|
mkdirSync(join(directory, '.github', 'scripts'), { recursive: true });
|
|
writeFileSync(callsLog, '');
|
|
const stub = (name, body) =>
|
|
writeFileSync(join(bin, name), `#!/bin/sh\n${body}\n`, { mode: 0o755 });
|
|
stub('git', 'echo 0123456789abcdef0123456789abcdef01234567');
|
|
stub('node', 'echo ghcr.io/qwenlm/qwen-code:0.0.0-test');
|
|
stub('timeout', 'shift; exec "$@"');
|
|
stub('flock', `printf 'flock %s\\n' "$*" >> "$CALLS_LOG"`);
|
|
stub(
|
|
'docker',
|
|
`printf 'docker %s\\n' "$*" >> "$CALLS_LOG"
|
|
if [ "$1" = 'image' ] && [ "$2" = 'inspect' ]; then
|
|
if [ "$3" = '--format' ]; then
|
|
echo sha256:fakeimageid
|
|
exit 0
|
|
fi
|
|
# The stubbed timeout passes the child's status through, so a 124 here
|
|
# stands in for the probe hitting its bound.
|
|
if [ "$INSPECT_TIMES_OUT" = '1' ]; then
|
|
exit 124
|
|
fi
|
|
if [ "$IMAGE_PRESENT" = '1' ]; then
|
|
exit 0
|
|
fi
|
|
exit 1
|
|
fi
|
|
if [ "$1" = 'image' ] && [ "$2" = 'prune' ] && [ "$IMAGE_PRUNE_FAILS" = '1' ]; then
|
|
echo 'ERROR: failed to prune images' >&2
|
|
exit 1
|
|
fi
|
|
if [ "$1" = 'builder' ] && [ "$2" = 'prune' ] && [ "$PRUNE_FAILS" = '1' ]; then
|
|
echo 'ERROR: failed to prune build cache' >&2
|
|
exit 1
|
|
fi
|
|
if [ "$1" = 'info' ]; then
|
|
# Answer only the exact expression the gate depends on, so a template typo
|
|
# drops the gate from this suite instead of silently disabling it in CI.
|
|
if [ "$DOCKER_INFO_FAILS" != '1' ] && [ "$2" = '--format' ] && [ "$3" = '{{.DockerRootDir}}' ]; then
|
|
echo "$DOCKER_ROOT_DIR"
|
|
exit 0
|
|
fi
|
|
exit 1
|
|
fi
|
|
exit 0`,
|
|
);
|
|
stub('npm', `printf 'npm %s\\n' "$*" >> "$CALLS_LOG"`);
|
|
stub('npx', `printf 'npx %s\\n' "$*" >> "$CALLS_LOG"`);
|
|
if (!helperAbsent) {
|
|
// Both checkpoints share this stub: count invocations so a case can
|
|
// fail the pre-vitest gate while the pre-build gate passes.
|
|
writeFileSync(
|
|
join(directory, '.github', 'scripts', 'check-disk-floor.sh'),
|
|
'#!/bin/sh\n' +
|
|
'count_file="$CALLS_LOG.gate-count"\n' +
|
|
'count=0\n' +
|
|
'if [ -f "$count_file" ]; then count=$(cat "$count_file"); fi\n' +
|
|
'count=$((count + 1))\n' +
|
|
'printf \'%s\' "$count" > "$count_file"\n' +
|
|
'printf \'disk-floor %s floor=%s inodes=%s\\n\' "$*" "${DISK_FLOOR_MIN_FREE_KB:-unset}" "${DISK_FLOOR_MIN_FREE_INODES:-unset}" >> "$CALLS_LOG"\n' +
|
|
'if [ "$count" -gt 1 ] && [ -n "${POST_GATE_EXIT:-}" ]; then\n' +
|
|
' exit "$POST_GATE_EXIT"\n' +
|
|
'fi\n' +
|
|
'exit "${GATE_EXIT:-0}"\n',
|
|
{ mode: 0o755 },
|
|
);
|
|
}
|
|
try {
|
|
const env = {
|
|
...process.env,
|
|
PATH: `${bin}:${process.env.PATH}`,
|
|
HOME: directory,
|
|
RUNNER_ENVIRONMENT: runnerEnvironment,
|
|
RELEASE_CONTAINER_OWNER: 'test-owner',
|
|
CALLS_LOG: callsLog,
|
|
DOCKER_ROOT_DIR: dockerRoot,
|
|
IMAGE_PRESENT: imagePresent ? '1' : '0',
|
|
INSPECT_TIMES_OUT: inspectTimesOut ? '1' : '0',
|
|
GATE_EXIT: gateExit,
|
|
PRUNE_FAILS: pruneFails ? '1' : '0',
|
|
IMAGE_PRUNE_FAILS: imagePruneFails ? '1' : '0',
|
|
DOCKER_INFO_FAILS: dockerInfoFails ? '1' : '0',
|
|
};
|
|
// The floor knobs come from the test, never the ambient environment:
|
|
// the suite's verdict must not depend on the env of whoever runs it.
|
|
delete env.DISK_FLOOR_MIN_FREE_KB;
|
|
delete env.DISK_FLOOR_MIN_FREE_INODES;
|
|
delete env.DISK_FLOOR_DOCKER_MIN_FREE_KB;
|
|
delete env.DISK_FLOOR_DOCKER_MIN_FREE_INODES;
|
|
delete env.DISK_FLOOR_DOCKER_POST_MIN_FREE_KB;
|
|
delete env.POST_GATE_EXIT;
|
|
if (ambientDiskFloorKb !== undefined) {
|
|
env.DISK_FLOOR_MIN_FREE_KB = ambientDiskFloorKb;
|
|
}
|
|
if (ambientDiskFloorInodes !== undefined) {
|
|
env.DISK_FLOOR_MIN_FREE_INODES = ambientDiskFloorInodes;
|
|
}
|
|
if (dockerFloorKb !== undefined) {
|
|
env.DISK_FLOOR_DOCKER_MIN_FREE_KB = dockerFloorKb;
|
|
}
|
|
if (dockerFloorInodes !== undefined) {
|
|
env.DISK_FLOOR_DOCKER_MIN_FREE_INODES = dockerFloorInodes;
|
|
}
|
|
if (dockerPostFloorKb !== undefined) {
|
|
env.DISK_FLOOR_DOCKER_POST_MIN_FREE_KB = dockerPostFloorKb;
|
|
}
|
|
if (postGateExit !== undefined) {
|
|
env.POST_GATE_EXIT = postGateExit;
|
|
}
|
|
const result = spawnSync('bash', [dockerIntegrationScriptAbsolutePath], {
|
|
cwd: directory,
|
|
encoding: 'utf8',
|
|
env,
|
|
});
|
|
return { result, calls: readFileSync(callsLog, 'utf8'), dockerRoot };
|
|
} finally {
|
|
rmSync(directory, { recursive: true, force: true });
|
|
}
|
|
};
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'reclaims without the disk gate or the lock protocol on hosted runners',
|
|
() => {
|
|
const { result, calls } = runDockerIntegrationScript();
|
|
expect(result.status, result.stderr).toBe(0);
|
|
const lines = calls.trim().split('\n');
|
|
const danglingPrune = lines.findIndex((line) =>
|
|
line.startsWith('docker image prune --force'),
|
|
);
|
|
const cachePrune = lines.findIndex((line) =>
|
|
line.startsWith('docker builder prune'),
|
|
);
|
|
const imagePrune = lines.findIndex((line) =>
|
|
line.startsWith('docker image prune --all'),
|
|
);
|
|
const build = lines.findIndex((line) =>
|
|
line.startsWith('npm run build:sandbox'),
|
|
);
|
|
expect(danglingPrune).toBeGreaterThanOrEqual(0);
|
|
expect(cachePrune).toBeGreaterThan(danglingPrune);
|
|
expect(imagePrune).toBeGreaterThan(cachePrune);
|
|
expect(build).toBeGreaterThan(imagePrune);
|
|
// The floor gate is scoped to the persistent self-hosted pool, like
|
|
// every other check-disk-floor.sh call site.
|
|
expect(calls).not.toContain('disk-floor');
|
|
expect(calls).not.toContain('docker info');
|
|
expect(calls).not.toContain('flock');
|
|
expect(
|
|
lines.filter((line) => line.startsWith('npx vitest')),
|
|
).toHaveLength(2);
|
|
},
|
|
);
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'builds the image only after the cache prune and the pre-build disk gate pass',
|
|
() => {
|
|
const { result, calls, dockerRoot } = runDockerIntegrationScript({
|
|
runnerEnvironment: 'self-hosted',
|
|
});
|
|
expect(result.status, result.stderr).toBe(0);
|
|
// The exact `docker info` expression the gate depends on is pinned: the
|
|
// stub answers nothing else, so a template typo drops the gate lines.
|
|
expect(calls).toContain('docker info --format {{.DockerRootDir}}');
|
|
const lines = calls.trim().split('\n');
|
|
const daemonLock = lines.findIndex((line) =>
|
|
line.startsWith('flock --shared --wait 1800 9'),
|
|
);
|
|
const coordinatorLock = lines.findIndex(
|
|
(line) => line === 'flock --wait 1800 8',
|
|
);
|
|
const buildLock = lines.findIndex(
|
|
(line) => line === 'flock --wait 1800 7',
|
|
);
|
|
const danglingPrune = lines.findIndex((line) =>
|
|
line.startsWith('docker image prune --force'),
|
|
);
|
|
const cachePrune = lines.findIndex((line) =>
|
|
line.startsWith('docker builder prune'),
|
|
);
|
|
const imagePrune = lines.findIndex((line) =>
|
|
line.startsWith('docker image prune --all'),
|
|
);
|
|
const gates = lines
|
|
.map((line, index) => (line.startsWith('disk-floor') ? index : -1))
|
|
.filter((index) => index >= 0);
|
|
const build = lines.findIndex((line) =>
|
|
line.startsWith('npm run build:sandbox'),
|
|
);
|
|
const buildUnlock = lines.findIndex(
|
|
(line) => line === 'flock --unlock 7',
|
|
);
|
|
const coordinatorUnlock = lines.findIndex(
|
|
(line) => line === 'flock --unlock 8',
|
|
);
|
|
const firstVitest = lines.findIndex((line) =>
|
|
line.startsWith('npx vitest'),
|
|
);
|
|
expect(daemonLock).toBeGreaterThanOrEqual(0);
|
|
expect(coordinatorLock).toBeGreaterThan(daemonLock);
|
|
// The unlabelled prunes run on every path, outside the build mutex:
|
|
// the daily host sweep runs the same two with no lock at all.
|
|
expect(danglingPrune).toBeGreaterThan(coordinatorLock);
|
|
expect(cachePrune).toBeGreaterThan(danglingPrune);
|
|
expect(buildLock).toBeGreaterThan(cachePrune);
|
|
expect(imagePrune).toBeGreaterThan(buildLock);
|
|
// One gate sample before the build and one after it: the #13479 death
|
|
// landed in the vitest phase, after the build had already finished.
|
|
// The pre-build sample charges the build-sized floor; the pre-vitest
|
|
// sample forwards an empty override so the helper's calibrated job
|
|
// floor applies — echoed as "unset" here, so a re-hardcoded literal
|
|
// turns red instead of silently drifting from check-disk-floor.sh.
|
|
expect(gates).toHaveLength(2);
|
|
expect(lines[gates[0]]).toBe(
|
|
`disk-floor ${dockerRoot} floor=8388608 inodes=unset`,
|
|
);
|
|
expect(lines[gates[1]]).toBe(
|
|
`disk-floor ${dockerRoot} floor=unset inodes=unset`,
|
|
);
|
|
expect(gates[0]).toBeGreaterThan(imagePrune);
|
|
expect(build).toBeGreaterThan(gates[0]);
|
|
expect(buildUnlock).toBeGreaterThan(build);
|
|
expect(gates[1]).toBeGreaterThan(build);
|
|
expect(gates[1]).toBeLessThan(firstVitest);
|
|
expect(coordinatorUnlock).toBeGreaterThan(buildUnlock);
|
|
expect(firstVitest).toBeGreaterThan(coordinatorUnlock);
|
|
expect(
|
|
lines.filter((line) => line.startsWith('npx vitest')),
|
|
).toHaveLength(2);
|
|
},
|
|
);
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'fails before the image build when the pre-build disk gate trips',
|
|
() => {
|
|
const { result, calls } = runDockerIntegrationScript({
|
|
gateExit: '1',
|
|
runnerEnvironment: 'self-hosted',
|
|
});
|
|
expect(result.status).not.toBe(0);
|
|
expect(calls).toContain('disk-floor');
|
|
expect(calls).not.toContain('npm run build:sandbox');
|
|
expect(calls).not.toContain('npx vitest');
|
|
// A gate failure exits holding the host locks (the OS releases them at
|
|
// process exit): no unlock may be recorded past the trip.
|
|
expect(calls).not.toContain('flock --unlock 7');
|
|
expect(calls).not.toContain('flock --unlock 8');
|
|
// The lane re-labels a helper failure with the knobs this gate
|
|
// actually reads, so the operator is not sent to the job-start gate's
|
|
// variables.
|
|
expect(result.stdout).toContain('docker data root floor gate failed');
|
|
expect(result.stdout).toContain('DISK_FLOOR_DOCKER_MIN_FREE_KB');
|
|
},
|
|
);
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'keeps the docker floors when the workspace floor knob is set',
|
|
() => {
|
|
// The docker data-root gates read DISK_FLOOR_DOCKER_MIN_FREE_KB and
|
|
// DISK_FLOOR_DOCKER_POST_MIN_FREE_KB, not the job-start gate's
|
|
// DISK_FLOOR_MIN_FREE_KB: one env setting must not move both floors
|
|
// (#13479).
|
|
const { result, calls, dockerRoot } = runDockerIntegrationScript({
|
|
runnerEnvironment: 'self-hosted',
|
|
ambientDiskFloorKb: '1',
|
|
});
|
|
expect(result.status, result.stderr).toBe(0);
|
|
const floors = calls
|
|
.trim()
|
|
.split('\n')
|
|
.filter((line) => line.startsWith('disk-floor'));
|
|
expect(floors).toHaveLength(2);
|
|
expect(floors[0]).toBe(
|
|
`disk-floor ${dockerRoot} floor=8388608 inodes=unset`,
|
|
);
|
|
expect(floors[1]).toBe(
|
|
`disk-floor ${dockerRoot} floor=unset inodes=unset`,
|
|
);
|
|
},
|
|
);
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'honors the dedicated docker floor knob',
|
|
() => {
|
|
const { result, calls, dockerRoot } = runDockerIntegrationScript({
|
|
runnerEnvironment: 'self-hosted',
|
|
dockerFloorKb: '4194304',
|
|
});
|
|
expect(result.status, result.stderr).toBe(0);
|
|
const floors = calls
|
|
.trim()
|
|
.split('\n')
|
|
.filter((line) => line.startsWith('disk-floor'));
|
|
expect(floors).toHaveLength(2);
|
|
// DISK_FLOOR_DOCKER_MIN_FREE_KB sizes the pre-build sample only: the
|
|
// pre-vitest checkpoint has its own knob so the build budget is
|
|
// charged once.
|
|
expect(floors[0]).toBe(
|
|
`disk-floor ${dockerRoot} floor=4194304 inodes=unset`,
|
|
);
|
|
expect(floors[1]).toBe(
|
|
`disk-floor ${dockerRoot} floor=unset inodes=unset`,
|
|
);
|
|
},
|
|
);
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'honors the dedicated post-build docker floor knob',
|
|
() => {
|
|
const { result, calls, dockerRoot } = runDockerIntegrationScript({
|
|
runnerEnvironment: 'self-hosted',
|
|
dockerPostFloorKb: '3145728',
|
|
});
|
|
expect(result.status, result.stderr).toBe(0);
|
|
const floors = calls
|
|
.trim()
|
|
.split('\n')
|
|
.filter((line) => line.startsWith('disk-floor'));
|
|
expect(floors).toHaveLength(2);
|
|
// DISK_FLOOR_DOCKER_POST_MIN_FREE_KB sizes the pre-vitest sample
|
|
// only: an operator can tune the two checkpoints independently.
|
|
expect(floors[0]).toBe(
|
|
`disk-floor ${dockerRoot} floor=8388608 inodes=unset`,
|
|
);
|
|
expect(floors[1]).toBe(
|
|
`disk-floor ${dockerRoot} floor=3145728 inodes=unset`,
|
|
);
|
|
},
|
|
);
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'stops before the vitest phase when the post-build gate trips',
|
|
() => {
|
|
// A host admitted by the pre-build floor can drop below the
|
|
// pre-vitest floor during the build: the build's own layers are
|
|
// exactly what the 8 GiB budget pays for. The post-build gate must
|
|
// then stop the lane before vitest writes container layers to the
|
|
// same filesystem — #13479 died in that phase.
|
|
const { result, calls } = runDockerIntegrationScript({
|
|
runnerEnvironment: 'self-hosted',
|
|
postGateExit: '1',
|
|
});
|
|
expect(result.status).not.toBe(0);
|
|
expect(calls).toContain('npm run build:sandbox');
|
|
expect(calls).toContain('flock --unlock 7');
|
|
expect(calls).not.toContain('npx vitest');
|
|
},
|
|
);
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'keeps the default docker inode floor when the workspace inode knob is set',
|
|
() => {
|
|
// The job-start gate's DISK_FLOOR_MIN_FREE_INODES must not move the
|
|
// docker data-root floor either: hardening the workspace gate would
|
|
// otherwise silently retune a filesystem the operator never targeted
|
|
// (#13479).
|
|
const { result, calls, dockerRoot } = runDockerIntegrationScript({
|
|
runnerEnvironment: 'self-hosted',
|
|
ambientDiskFloorInodes: '999999999',
|
|
});
|
|
expect(result.status, result.stderr).toBe(0);
|
|
const floors = calls
|
|
.trim()
|
|
.split('\n')
|
|
.filter((line) => line.startsWith('disk-floor'));
|
|
expect(floors).toHaveLength(2);
|
|
expect(floors[0]).toBe(
|
|
`disk-floor ${dockerRoot} floor=8388608 inodes=unset`,
|
|
);
|
|
expect(floors[1]).toBe(
|
|
`disk-floor ${dockerRoot} floor=unset inodes=unset`,
|
|
);
|
|
},
|
|
);
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'honors the dedicated docker inode knob',
|
|
() => {
|
|
const { result, calls, dockerRoot } = runDockerIntegrationScript({
|
|
runnerEnvironment: 'self-hosted',
|
|
dockerFloorInodes: '200000',
|
|
});
|
|
expect(result.status, result.stderr).toBe(0);
|
|
const floors = calls
|
|
.trim()
|
|
.split('\n')
|
|
.filter((line) => line.startsWith('disk-floor'));
|
|
expect(floors).toHaveLength(2);
|
|
expect(floors[0]).toBe(
|
|
`disk-floor ${dockerRoot} floor=8388608 inodes=200000`,
|
|
);
|
|
expect(floors[1]).toBe(
|
|
`disk-floor ${dockerRoot} floor=unset inodes=200000`,
|
|
);
|
|
},
|
|
);
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'reclaims but skips the build and the gate when a hosted runner finds the image cached',
|
|
() => {
|
|
const { result, calls } = runDockerIntegrationScript({
|
|
imagePresent: true,
|
|
});
|
|
expect(result.status, result.stderr).toBe(0);
|
|
// The unlabelled prunes run on every path: a cached image skips the
|
|
// build branch, and without them nothing would reclaim ahead of the
|
|
// pre-vitest gate (#13479). The labelled prune stays inside the
|
|
// image-missing branch under the build mutex.
|
|
expect(calls).toContain('docker image prune --force --filter until=24h');
|
|
expect(calls).toContain(
|
|
'docker builder prune --all --force --filter until=24h',
|
|
);
|
|
expect(calls).not.toContain('docker image prune --all');
|
|
expect(calls).not.toContain('disk-floor');
|
|
expect(calls).not.toContain('npm run build:sandbox');
|
|
expect(calls).toContain('npx vitest run --root ./integration-tests cli');
|
|
},
|
|
);
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'gates the cached-image re-run path before the vitest phase',
|
|
() => {
|
|
const { result, calls, dockerRoot } = runDockerIntegrationScript({
|
|
imagePresent: true,
|
|
runnerEnvironment: 'self-hosted',
|
|
});
|
|
expect(result.status, result.stderr).toBe(0);
|
|
// The build is skipped, but the reclaim is not: a re-run landing on
|
|
// the same still-saturated host with the image now cached must get
|
|
// its reclaim before the data-root gate samples the filesystem, or
|
|
// the gate trips again on the reading that failed the previous
|
|
// attempt and no retry can recover (#13479).
|
|
expect(calls).not.toContain('docker image prune --all');
|
|
expect(calls).not.toContain('npm run build:sandbox');
|
|
expect(calls).toContain('docker info --format {{.DockerRootDir}}');
|
|
const lines = calls.trim().split('\n');
|
|
const danglingPrune = lines.findIndex((line) =>
|
|
line.startsWith('docker image prune --force'),
|
|
);
|
|
const cachePrune = lines.findIndex((line) =>
|
|
line.startsWith('docker builder prune'),
|
|
);
|
|
const gates = lines
|
|
.map((line, index) => (line.startsWith('disk-floor') ? index : -1))
|
|
.filter((index) => index >= 0);
|
|
const firstVitest = lines.findIndex((line) =>
|
|
line.startsWith('npx vitest'),
|
|
);
|
|
expect(danglingPrune).toBeGreaterThanOrEqual(0);
|
|
expect(cachePrune).toBeGreaterThan(danglingPrune);
|
|
expect(gates).toHaveLength(1);
|
|
expect(lines[gates[0]]).toBe(
|
|
`disk-floor ${dockerRoot} floor=unset inodes=unset`,
|
|
);
|
|
expect(gates[0]).toBeGreaterThan(cachePrune);
|
|
expect(firstVitest).toBeGreaterThan(gates[0]);
|
|
},
|
|
);
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'blocks the vitest phase when the cached-image host fails the gate',
|
|
() => {
|
|
const { result, calls } = runDockerIntegrationScript({
|
|
gateExit: '1',
|
|
imagePresent: true,
|
|
runnerEnvironment: 'self-hosted',
|
|
});
|
|
expect(result.status).not.toBe(0);
|
|
expect(calls).toContain('disk-floor');
|
|
expect(calls).not.toContain('npm run build:sandbox');
|
|
expect(calls).not.toContain('npx vitest');
|
|
},
|
|
);
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'fails fast with the docker knob named when a docker floor knob is malformed',
|
|
() => {
|
|
// The helper validates what it is handed under ITS variable names, so
|
|
// without the lane-side check a malformed DISK_FLOOR_DOCKER_* value
|
|
// is reported as a bad DISK_FLOOR_MIN_FREE_KB — and the pre-vitest
|
|
// knob's typo only after the whole image build.
|
|
const { result, calls } = runDockerIntegrationScript({
|
|
runnerEnvironment: 'self-hosted',
|
|
dockerPostFloorKb: '3G',
|
|
});
|
|
expect(result.status).not.toBe(0);
|
|
expect(result.stdout).toContain(
|
|
'DISK_FLOOR_DOCKER_POST_MIN_FREE_KB must be a non-negative integer',
|
|
);
|
|
expect(calls).not.toContain('npm run build:sandbox');
|
|
expect(calls).not.toContain('npx vitest');
|
|
},
|
|
);
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'fails fast when the image-presence probe times out instead of rebuilding',
|
|
() => {
|
|
// The probe runs with the shared host locks held: a wedged daemon
|
|
// must fail the lane on the bound, not read as "image absent" and
|
|
// rebuild against a daemon that cannot answer.
|
|
const { result, calls } = runDockerIntegrationScript({
|
|
runnerEnvironment: 'self-hosted',
|
|
inspectTimesOut: true,
|
|
});
|
|
expect(result.status).not.toBe(0);
|
|
expect(result.stdout).toContain('docker image inspect timed out');
|
|
expect(calls).not.toContain('npm run build:sandbox');
|
|
expect(calls).not.toContain('npx vitest');
|
|
},
|
|
);
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'continues with a warning when the BuildKit cache prune fails',
|
|
() => {
|
|
const { result, calls } = runDockerIntegrationScript({
|
|
pruneFails: true,
|
|
});
|
|
expect(result.status, result.stderr).toBe(0);
|
|
expect(result.stdout).toContain(
|
|
'::warning::docker build cache cleanup failed',
|
|
);
|
|
expect(calls).toContain('npm run build:sandbox');
|
|
expect(
|
|
calls
|
|
.trim()
|
|
.split('\n')
|
|
.filter((line) => line.startsWith('npx vitest')),
|
|
).toHaveLength(2);
|
|
},
|
|
);
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'continues with a warning when the dangling image prune fails',
|
|
() => {
|
|
const { result, calls } = runDockerIntegrationScript({
|
|
imagePruneFails: true,
|
|
});
|
|
expect(result.status, result.stderr).toBe(0);
|
|
expect(result.stdout).toContain(
|
|
'::warning::dangling image cleanup failed',
|
|
);
|
|
expect(calls).toContain('npm run build:sandbox');
|
|
},
|
|
);
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'continues with a warning when the labelled image prune fails',
|
|
() => {
|
|
const { result, calls } = runDockerIntegrationScript({
|
|
imagePruneFails: true,
|
|
});
|
|
expect(result.status, result.stderr).toBe(0);
|
|
expect(result.stdout).toContain(
|
|
'::warning::old CI sandbox image cleanup failed',
|
|
);
|
|
expect(calls).toContain('npm run build:sandbox');
|
|
},
|
|
);
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'warns and still builds when the docker data root cannot be resolved',
|
|
() => {
|
|
const { result, calls } = runDockerIntegrationScript({
|
|
runnerEnvironment: 'self-hosted',
|
|
dockerInfoFails: true,
|
|
});
|
|
expect(result.status, result.stderr).toBe(0);
|
|
expect(calls).not.toContain('disk-floor');
|
|
expect(calls).toContain('npm run build:sandbox');
|
|
expect(result.stdout).toContain(
|
|
'::warning::docker data root floor gate skipped',
|
|
);
|
|
},
|
|
);
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'warns and still builds when the docker data root is not a directory',
|
|
() => {
|
|
const { result, calls } = runDockerIntegrationScript({
|
|
runnerEnvironment: 'self-hosted',
|
|
dockerRootMissing: true,
|
|
});
|
|
expect(result.status, result.stderr).toBe(0);
|
|
expect(calls).not.toContain('disk-floor');
|
|
expect(calls).toContain('npm run build:sandbox');
|
|
expect(result.stdout).toContain(
|
|
'::warning::docker data root floor gate skipped',
|
|
);
|
|
},
|
|
);
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'warns and still builds when the disk floor helper is absent',
|
|
() => {
|
|
const { result, calls } = runDockerIntegrationScript({
|
|
runnerEnvironment: 'self-hosted',
|
|
helperAbsent: true,
|
|
});
|
|
expect(result.status, result.stderr).toBe(0);
|
|
expect(calls).not.toContain('disk-floor');
|
|
expect(calls).toContain('npm run build:sandbox');
|
|
expect(result.stdout).toContain(
|
|
'::warning::docker data root floor gate skipped',
|
|
);
|
|
},
|
|
);
|
|
|
|
it('reaps only the Docker integration containers owned by its job', () => {
|
|
const owner = '${{ github.run_id }}-${{ github.run_attempt }}-release';
|
|
const steps = releaseYaml.jobs.integration_docker.steps;
|
|
const testStep = steps.find(
|
|
(step) => step.name === 'Run Docker Integration Tests',
|
|
);
|
|
const cleanupStep = steps.find(
|
|
(step) => step.name === 'Remove job-owned release containers',
|
|
);
|
|
|
|
expect(testStep.env.RELEASE_CONTAINER_OWNER).toBe(owner);
|
|
expect(testStep.env.SANDBOX_FLAGS).toContain(
|
|
'org.qwen-code.ci.owner=${RELEASE_CONTAINER_OWNER}',
|
|
);
|
|
expect(dockerIntegrationScript).toContain(
|
|
'trap cleanup_release_containers EXIT',
|
|
);
|
|
expect(dockerIntegrationScript).toContain("trap 'exit 1' INT TERM");
|
|
expect(dockerIntegrationScript).toContain(
|
|
'--filter "label=org.qwen-code.ci.owner=${RELEASE_CONTAINER_OWNER}"',
|
|
);
|
|
expect(cleanupStep.if).toContain('always()');
|
|
expect(cleanupStep.env.RELEASE_CONTAINER_OWNER).toBe(owner);
|
|
expect(cleanupStep.run).toBe(
|
|
'.release-workflow/.github/scripts/run-release-docker-integration.sh cleanup',
|
|
);
|
|
expect(dockerIntegrationScript).toContain('docker rm -f > /dev/null');
|
|
expect(dockerIntegrationScript.match(/docker ps -aq/g)).toHaveLength(2);
|
|
expect(dockerIntegrationScript).toContain('release containers remain');
|
|
});
|
|
|
|
it('digest-pins every sandbox base image', () => {
|
|
// integration_docker builds on the shared pool, whose docker daemon
|
|
// store persists across jobs: a co-resident job can retag a mutable
|
|
// base tag with a poisoned image, but a digest cannot be moved by
|
|
// `docker tag`. Every FROM must carry an @sha256: digest.
|
|
const dockerfile = readFileSync('Dockerfile', 'utf8');
|
|
const fromLines = dockerfile
|
|
.split('\n')
|
|
.filter((line) => /^FROM\s/.test(line));
|
|
expect(fromLines.length).toBeGreaterThan(0);
|
|
for (const line of fromLines) {
|
|
expect(line, line).toMatch(/@sha256:[0-9a-f]{64}(\s|$)/);
|
|
}
|
|
});
|
|
|
|
it('bounds shared-pool jobs and skips redundant remote npm caches', () => {
|
|
expect(
|
|
Object.fromEntries(
|
|
[
|
|
'prepare',
|
|
'quality_static',
|
|
'quality_build',
|
|
'quality_typecheck',
|
|
'workspace_tests',
|
|
'quality_scripts',
|
|
'quality',
|
|
'integration_none',
|
|
'integration_docker',
|
|
'notify_failure',
|
|
].map((id) => [id, releaseYaml.jobs[id]['timeout-minutes']]),
|
|
),
|
|
).toEqual({
|
|
prepare: 30,
|
|
// The three bounds an operator can retune without a PR; their defaults
|
|
// are pinned by their own tests above.
|
|
quality_static:
|
|
"${{ fromJSON(vars.QWEN_RELEASE_STATIC_TIMEOUT_MINUTES || '60') }}",
|
|
quality_build:
|
|
"${{ fromJSON(vars.QWEN_RELEASE_BUILD_TIMEOUT_MINUTES || '45') }}",
|
|
quality_typecheck: 30,
|
|
workspace_tests:
|
|
"${{ fromJSON(vars.QWEN_RELEASE_WORKSPACE_TIMEOUT_MINUTES || '45') }}",
|
|
quality_scripts: 30,
|
|
quality: 5,
|
|
integration_none: 120,
|
|
integration_docker: 120,
|
|
notify_failure: 10,
|
|
});
|
|
|
|
for (const id of [
|
|
'prepare',
|
|
'quality_static',
|
|
'quality_build',
|
|
'quality_typecheck',
|
|
'workspace_tests',
|
|
'quality_scripts',
|
|
'integration_none',
|
|
'integration_docker',
|
|
]) {
|
|
const steps = releaseYaml.jobs[id].steps;
|
|
// In-tree precedent says nodejs.org may be unreachable through the
|
|
// ECS egress proxy: pool runs must reuse the machine's Node, with
|
|
// setup-node reserved for the hosted fallback.
|
|
const setupNode = steps.find((step) =>
|
|
String(step.uses ?? '').includes('actions/setup-node'),
|
|
);
|
|
expect(setupNode?.if, id).toBe(
|
|
"${{ runner.environment != 'self-hosted' }}",
|
|
);
|
|
// Dependencies install with pnpm, so an npm download cache would be
|
|
// restored and never read.
|
|
expect(setupNode?.with.cache, id).toBeUndefined();
|
|
expect(setupNode?.with['package-manager-cache'], id).toBe(false);
|
|
const machineNode = steps.find((step) =>
|
|
String(step.uses ?? '').includes('.github/actions/self-hosted-node'),
|
|
);
|
|
expect(machineNode?.if, id).toBe(
|
|
"${{ runner.environment == 'self-hosted' }}",
|
|
);
|
|
}
|
|
// publish stays hosted-only and keeps its unconditional setup-node.
|
|
const publishSetupNode = releaseYaml.jobs.publish.steps.find((step) =>
|
|
String(step.uses ?? '').includes('actions/setup-node'),
|
|
);
|
|
expect(publishSetupNode?.with.cache).toBeUndefined();
|
|
expect(publishSetupNode?.with['package-manager-cache']).toBe(false);
|
|
});
|
|
|
|
it('stages every integration package manifest after versioning', () => {
|
|
expect(releaseStepScript).toContain(
|
|
'git add package.json pnpm-lock.yaml packages/*/package.json packages/channels/*/package.json integrations/*/package.json integrations/*/qwen-extension.json',
|
|
);
|
|
});
|
|
|
|
it('publishes the Mem0 Extension only after trusted publishing bootstrap', () => {
|
|
const publishSteps = releaseYaml.jobs.publish.steps;
|
|
const publishStep = publishSteps.find(
|
|
(step) => step.name === 'Publish npm packages',
|
|
);
|
|
expect(publishStep.env.PUBLISH_EXTERNAL_CONTEXT_MEM0).toContain(
|
|
"vars.NPM_EXTERNAL_CONTEXT_MEM0_TRUSTED_PUBLISHING_ENABLED == 'true'",
|
|
);
|
|
// The audio-capture switch replaced a deleted step-level
|
|
// `if: github.repository == 'QwenLM/qwen-code'` gate. Unpinned, a fork
|
|
// running this workflow would publish @qwen-code/audio-capture.
|
|
expect(publishStep.env.PUBLISH_AUDIO_CAPTURE).toContain(
|
|
"github.repository == 'QwenLM/qwen-code'",
|
|
);
|
|
});
|
|
|
|
it('fires the fleet-moving npm-published dispatch on stable releases only', () => {
|
|
// This gate is the sole protection keeping a nightly/preview/dry-run
|
|
// release from moving the ECS fleet; the triggered update workflow
|
|
// installs whatever version it is handed, so there is no downstream
|
|
// guard. Pin all three clauses together so dropping or inverting one
|
|
// fails review instead of silently shipping a non-stable fleet.
|
|
expect(workflow).toContain(
|
|
'if: |-\n' +
|
|
" ${{ github.repository == 'QwenLM/qwen-code' &&\n" +
|
|
" needs.prepare.outputs.is_dry_run == 'false' &&\n" +
|
|
" needs.prepare.outputs.npm_tag == 'latest' }}",
|
|
);
|
|
expect(releaseStepScript).toContain("-f 'event_type=npm-published'");
|
|
expect(releaseStepScript).toContain(
|
|
'-f "client_payload[version]=${RELEASE_VERSION}"',
|
|
);
|
|
});
|
|
|
|
it('fails the release when the review source stamp did not land', () => {
|
|
// The runtime staleness check degrades to "could not check" without the
|
|
// stamp this step is guarding. The publish job itself does not re-run
|
|
// the scripts suite — the quality_scripts job that gates it does, but
|
|
// `force_skip_tests: 'true'` skips that job entirely — so a future
|
|
// change that removes the stamp step or this guard must fail here
|
|
// instead of shipping a release that silently lost its digest.
|
|
// The ordering — bundle, then the stamp gate, then packaging — not the
|
|
// gate's prose or indentation: rewording the comment above the check must
|
|
// not fail a test whose subject is the guard itself.
|
|
expect(releaseStepScript).toMatch(
|
|
/npm run bundle[\s\S]*?test -f dist\/review-sources\.sha256[\s\S]*?npm run prepare:package/,
|
|
);
|
|
});
|
|
|
|
it('force-pushes the release branch so a retry replaces a failed attempt', () => {
|
|
// A failed attempt leaves release/<tag> on an older head, and the
|
|
// retry's divergent bump commit makes a plain push fail as
|
|
// non-fast-forward, blocking every retry of the release. Force is safe
|
|
// only while nothing for this version has shipped, so the push is
|
|
// pinned INSIDE the dry-run guard: the dry-run contract (release.yml's
|
|
// dispatch input) promises no branch is created, and no other test
|
|
// pins this guard — a force push outside it would turn a dry run into
|
|
// a destructive overwrite of the remote branch.
|
|
expect(releaseStepScript).toMatch(
|
|
/if \[\[ "\$\{IS_DRY_RUN\}" == "false" \]\]; then[\s\S]*?git push --force --set-upstream origin "\$\{release_branch_name\}" --follow-tags\n {4}else\n {6}echo "Dry run enabled\. Skipping push\."/,
|
|
);
|
|
});
|
|
|
|
it('serializes publish jobs per release tag', () => {
|
|
// The pre-push re-validation below is only sound while at most one
|
|
// publish job pushes and publishes a given version at a time; --force
|
|
// removed the non-fast-forward rejection that used to serialize the
|
|
// push itself. The group must sit on the publish job — in-progress
|
|
// runs are never cancelled, and of queued same-tag runs only the
|
|
// latest survives, but whichever run reaches the push re-validates
|
|
// first — and it must be keyed by the computed tag, which only exists
|
|
// as a prepare output, plus is_dry_run, so a dry run (which ships
|
|
// nothing) never queues ahead of the real release for the same tag.
|
|
// timeout-minutes bounds the hold a wedged publish (a stalled npm
|
|
// publish or asset upload) keeps on the group: without it the GitHub
|
|
// default of 360 minutes leaves same-tag retries queued behind it,
|
|
// unable to run, fail, or notify.
|
|
expect(workflow).toMatch(
|
|
/ {2}publish:\n {4}name: 'Publish Release'[\s\S]*?concurrency:\n {6}group: 'release-publish-\$\{\{ needs\.prepare\.outputs\.release_tag \}\}-\$\{\{ needs\.prepare\.outputs\.is_dry_run \}\}'\n {6}cancel-in-progress: false\n {4}timeout-minutes: 90\n {4}environment:\n {6}name: 'production-release'/,
|
|
);
|
|
});
|
|
|
|
it('uses the workflow-pinned guard for old release refs', () => {
|
|
expect(releaseStepScript).toContain(
|
|
'node .release-workflow/scripts/assert-release-version.mjs --assert-unreleased="${RELEASE_VERSION}"',
|
|
);
|
|
expect(releaseStepScript).not.toContain(
|
|
'node scripts/get-release-version.js --assert-unreleased=',
|
|
);
|
|
expect(releaseStepScript).toContain(
|
|
'node .release-workflow/.github/scripts/classify-release-notes.mjs',
|
|
);
|
|
expect(releaseStepScript).toContain(
|
|
'node .release-workflow/.github/scripts/cap-release-notes.mjs',
|
|
);
|
|
expect(releaseStepScript).not.toMatch(
|
|
/node \.github\/scripts\/(?:classify|cap)-release/,
|
|
);
|
|
});
|
|
|
|
it('re-validates that the version is still unshipped right before force-pushing', () => {
|
|
// prepare computed and validated the version minutes to hours before
|
|
// this push (the validation jobs and the production-release approval
|
|
// gate sit in between). A concurrent same-version run can ship in that
|
|
// window; the force push would then replace the branch tip that the
|
|
// shipped npm packages, tag, and merge to main anchor to. Pin the
|
|
// re-validation of prepare's invariant directly before the push, and
|
|
// pinned to the script that owns the published-package list so the
|
|
// guard cannot silently drift from it: the call must sit inside the
|
|
// dry-run guard, before the push, as an anchored full line (moving it
|
|
// out of the guard, inverting it, or narrowing it fails), and the
|
|
// step must receive RELEASE_VERSION — without it the check aborts
|
|
// every release — and GITHUB_TOKEN, which its gh release view probe
|
|
// needs: without a token the probe errors, and the guard fails closed
|
|
// on probe errors, so every push would abort. The script-side checks
|
|
// (every published package, the remote tag, the release, aborting on
|
|
// a hit, failing closed on a probe error) are unit-tested in
|
|
// get-release-version.test.js. The `|| GUARD_STATUS=$?` suffix is
|
|
// pinned too: notify_failure's refusal gate reads the guard's exit
|
|
// code through it, and the retry loop is pinned around the call:
|
|
// GUARD_STATUS is reset each attempt and only exit 2 (a probe
|
|
// failure) retries — exit 0, exit 3 (already shipped) and exit 4
|
|
// (malformed version) all stay decisive on the first attempt.
|
|
const releaseBranchStep = releaseYaml.jobs.publish.steps.find(
|
|
(step) => step.id === 'release_branch',
|
|
);
|
|
const pushReleaseBranchStep = releaseYaml.jobs.publish.steps.find(
|
|
(step) => step.id === 'push_release_branch',
|
|
);
|
|
expect(releaseBranchStep.env.CI_BOT_PAT).toBeUndefined();
|
|
expect(releaseBranchStep.env.GITHUB_TOKEN).toBeUndefined();
|
|
expect(pushReleaseBranchStep.env.GITHUB_TOKEN).toBe('${{ github.token }}');
|
|
expect(pushReleaseBranchStep.env.BRANCH_NAME).toBe(
|
|
'${{ steps.release_branch.outputs.BRANCH_NAME }}',
|
|
);
|
|
expect(pushReleaseBranchStep.env.CI_BOT_PAT).toBe(
|
|
'${{ secrets.CI_BOT_PAT }}',
|
|
);
|
|
expect(pushReleaseBranchStep.env.RELEASE_VERSION).toBe(
|
|
'${{ needs.prepare.outputs.release_version }}',
|
|
);
|
|
expect(pushReleaseBranchStep.run).toContain('push-release-branch');
|
|
const guard = releaseStepScript.indexOf(
|
|
'node .release-workflow/scripts/assert-release-version.mjs --assert-unreleased="${RELEASE_VERSION}" || guard_status=$?',
|
|
);
|
|
expect(guard).toBeGreaterThan(
|
|
releaseStepScript.indexOf('if [[ "${IS_DRY_RUN}" == "false" ]]'),
|
|
);
|
|
expect(releaseStepScript.indexOf('for attempt in 1 2 3; do')).toBeLessThan(
|
|
guard,
|
|
);
|
|
expect(
|
|
releaseStepScript.indexOf(
|
|
'git push --force --set-upstream origin "${release_branch_name}" --follow-tags',
|
|
),
|
|
).toBeGreaterThan(guard);
|
|
});
|
|
|
|
it('keeps a decisive version refusal out of the release-failed notification', () => {
|
|
// The guard's exit 3 means the version already shipped (fully or
|
|
// partially) — a correct refusal, not a release failure. The step
|
|
// must turn exactly that exit into the version_refusal marker, the
|
|
// publish job must export the marker, and notify_failure must skip
|
|
// its "Release Failed" issue + autofix dispatch for it — while any
|
|
// other guard exit (a fail-closed probe error) and every later
|
|
// publish failure still notify — including through the propagation
|
|
// branch pinned verbatim below: without it a probe failure falls
|
|
// through to the force push unverified.
|
|
expect(releaseStepScript).toMatch(
|
|
/node \.release-workflow\/scripts\/assert-release-version\.mjs --assert-unreleased="\$\{RELEASE_VERSION\}" \|\| guard_status=\$\?[\s\S]*?if \[\[ "\$\{guard_status\}" -eq 3 \]\]; then\n {8}echo "version_refusal=true" >> "\$\{GITHUB_OUTPUT\}"\n {8}exit 1\n {6}fi\n {6}if \[\[ "\$\{guard_status\}" -ne 0 \]\]; then\n {8}exit "\$\{guard_status\}"\n {6}fi/,
|
|
);
|
|
expect(workflow).toContain(
|
|
"version_refusal: '${{ steps.push_release_branch.outputs.version_refusal }}'",
|
|
);
|
|
expect(workflow).toMatch(
|
|
/needs\.publish\.result == 'failure' &&\n {12}needs\.publish\.outputs\.version_refusal != 'true'/,
|
|
);
|
|
});
|
|
|
|
it('keeps the guard copy of isExpectedMissingGitHubRelease in sync', () => {
|
|
// The guard ships alone under release.yml's sparse-checkout set, so it
|
|
// cannot import scripts/lib/release-helpers.js — the copy is deliberate.
|
|
// Two copies of a fail-closed hinge can drift, and widening only the
|
|
// guard's copy (a 403 or rate-limit branch) would read a throttled probe
|
|
// as "release absent" and let the force push proceed over a shipped
|
|
// version. Pin the bodies together so a one-sided edit goes red here.
|
|
const bodyOf = (source) =>
|
|
source
|
|
.slice(source.indexOf('function isExpectedMissingGitHubRelease'))
|
|
.split('\n}')[0]
|
|
.replace(/^export /, '');
|
|
const guard = readFileSync('scripts/assert-release-version.mjs', 'utf8');
|
|
const helpers = readFileSync('scripts/lib/release-helpers.js', 'utf8');
|
|
expect(bodyOf(guard)).toBe(bodyOf(helpers));
|
|
});
|
|
|
|
it('wires the guard exit code to the process exit status end to end', () => {
|
|
// The workflow reads the guard's decision from the process exit
|
|
// status. Run the real entry point without mocks — a usage error
|
|
// needs no network — so an inverted or dropped process.exit fails
|
|
// here instead of letting a refusal exit 0 at push time.
|
|
const result = spawnSync(
|
|
process.execPath,
|
|
['scripts/assert-release-version.mjs', '--assert-unreleased='],
|
|
{ encoding: 'utf8' },
|
|
);
|
|
// 4 rather than 2: the retry loop in run-release-step.sh breaks on any
|
|
// status other than 2, so a usage error fails fast instead of being
|
|
// logged three times as a transient probe failure.
|
|
expect(result.status).toBe(4);
|
|
expect(result.stdout).toContain(
|
|
'::error::assert-unreleased requires a version',
|
|
);
|
|
});
|
|
|
|
it.skipIf(process.platform === 'win32')(
|
|
'exits 3 from the real entry point when the version already shipped',
|
|
() => {
|
|
// The workflow reads the refusal through the entry-point exit-status
|
|
// glue, not through runCli(); the usage-error test above never
|
|
// exercises exit 3. A stub npm on PATH that echoes the probed
|
|
// version makes the strict npm scan report "shipped" without
|
|
// network, so an entry point that swallowed exit 3 fails here
|
|
// instead of reading as GUARD_STATUS=0 at push time. The stub is a
|
|
// '#!/bin/sh' script prepended to PATH with ':' — unresolvable on
|
|
// Windows, so win32 skips it and Linux CI remains the authoritative
|
|
// coverage.
|
|
const stubDir = mkdtempSync(join(tmpdir(), 'npm-stub-'));
|
|
writeFileSync(join(stubDir, 'npm'), '#!/bin/sh\necho "${2##*@}"\n', {
|
|
mode: 0o755,
|
|
});
|
|
const result = spawnSync(
|
|
process.execPath,
|
|
['scripts/assert-release-version.mjs', '--assert-unreleased=1.2.3'],
|
|
{
|
|
encoding: 'utf8',
|
|
env: { ...process.env, PATH: `${stubDir}:${process.env.PATH}` },
|
|
},
|
|
);
|
|
expect(result.status).toBe(3);
|
|
expect(result.stdout).toContain('has already shipped');
|
|
},
|
|
);
|
|
|
|
it('keeps a dispatch failure from failing an already-published release', () => {
|
|
// The packages are published before this step runs, so it must not fail
|
|
// the release; but the failure must still surface (as an error, not a
|
|
// warning) so the fleet can be reconciled via a manual re-run.
|
|
expect(workflow).toContain(
|
|
'continue-on-error: true\n' +
|
|
' env:\n' +
|
|
" GITHUB_TOKEN: '${{ secrets.CI_BOT_PAT }}'",
|
|
);
|
|
expect(releaseStepScript).toContain(
|
|
'echo "::error::npm-published dispatch failed;',
|
|
);
|
|
});
|
|
});
|
|
|
|
describe('Live Host feed contract', () => {
|
|
it('keeps Live Host releases independent from desktop releases', () => {
|
|
expect(desktopReleaseWorkflow).not.toContain('live-host:');
|
|
expect(liveHostReleaseWorkflow).toContain(
|
|
"working-directory: 'packages/live-host'",
|
|
);
|
|
expect(liveHostReleaseWorkflow).toContain(
|
|
"run: 'npm run dist:mac:no-publish'",
|
|
);
|
|
});
|
|
|
|
it('resolves the ASAR verifier through the standalone package', () => {
|
|
expect(liveHostCiWorkflow).toContain('npx --no-install asar list');
|
|
expect(liveHostCiWorkflow).toContain('npx --no-install asar extract');
|
|
expect(liveHostCiWorkflow).not.toContain(
|
|
'node_modules/@electron/asar/bin/asar.mjs',
|
|
);
|
|
});
|
|
|
|
it('keeps a producer and recovery path for every installer asset', () => {
|
|
for (const asset of [
|
|
'Qwen-Live-Host-manifest.json',
|
|
'Qwen-Live-Host-arm64.zip',
|
|
'Qwen-Live-Host-x64.zip',
|
|
]) {
|
|
expect(liveHostInstaller).toContain(asset);
|
|
expect(liveHostReleaseWorkflow).toContain(asset);
|
|
expect(liveHostOssWorkflow).toContain(asset);
|
|
}
|
|
expect(liveHostInstaller).toContain(
|
|
'https://github.com/QwenLM/qwen-code/releases/download/live-host-latest',
|
|
);
|
|
expect(liveHostReleaseWorkflow).toContain(
|
|
"FEED_TAG: '${{ env.LIVE_HOST_FEED_TAG }}'",
|
|
);
|
|
expect(liveHostOssWorkflow).toContain(
|
|
"gh release download 'live-host-latest'",
|
|
);
|
|
});
|
|
});
|
|
|
|
describe('release lane runner routing', () => {
|
|
const ecsRunsOn =
|
|
'${{ (github.repository == \'QwenLM/qwen-code\' && vars.MAINTAINER_ECS_RUNNER_DISABLED != \'true\') && fromJSON(\'["self-hosted", "linux", "x64", "ecs-qwen-hk4-host"]\') || fromJSON(\'["ubuntu-latest"]\') }}';
|
|
|
|
it('routes validation jobs to ECS with a hosted emergency fallback', () => {
|
|
const validationJobs = [
|
|
'prepare',
|
|
'quality_static',
|
|
'quality_build',
|
|
'quality_typecheck',
|
|
'workspace_tests',
|
|
'quality_scripts',
|
|
'integration_none',
|
|
'integration_docker',
|
|
];
|
|
for (const name of validationJobs) {
|
|
const job = releaseYaml.jobs[name];
|
|
expect(job, `job missing from release.yml: ${name}`).toBeTruthy();
|
|
expect(job['runs-on'], `runs-on drifted on job: ${name}`).toBe(ecsRunsOn);
|
|
}
|
|
});
|
|
|
|
it('classifies each schedule cron by the exact string it fires with', () => {
|
|
// prepare tells nightly from preview by comparing github.event.schedule
|
|
// against the cron text; a cron edited here without its comparison
|
|
// silently turns that schedule into a no-op run.
|
|
const crons = releaseYaml.on.schedule.map((entry) => entry.cron);
|
|
expect(crons).toHaveLength(2);
|
|
const vars = releaseYaml.jobs.prepare.steps.find(
|
|
(step) => step.id === 'vars',
|
|
);
|
|
for (const cron of crons) {
|
|
expect(vars.run).toContain('run-release-step.sh set-flags');
|
|
expect(releaseStepScript).toContain(`"\${CRON}" == "${cron}"`);
|
|
}
|
|
});
|
|
|
|
it('serializes scheduled release validation without coupling manual runs', () => {
|
|
expect(releaseYaml.concurrency).toEqual({
|
|
group:
|
|
"${{ github.event_name == 'schedule' && 'release-scheduled-validation' || format('release-{0}', github.run_id) }}",
|
|
'cancel-in-progress': false,
|
|
});
|
|
});
|
|
|
|
it('passes the runner environment to integration test configuration', () => {
|
|
for (const [name, expectedSteps] of [
|
|
['integration_none', 2],
|
|
['integration_docker', 1],
|
|
]) {
|
|
const job = releaseYaml.jobs[name];
|
|
expect(job.env.RUNNER_ENVIRONMENT, name).toBeUndefined();
|
|
const testSteps = job.steps.filter((step) => {
|
|
const command = String(step.run ?? '');
|
|
return (
|
|
!command.endsWith(' cleanup') &&
|
|
/(?:vitest|test:integration|run-release-docker-integration)/.test(
|
|
command,
|
|
)
|
|
);
|
|
});
|
|
expect(testSteps, name).toHaveLength(expectedSteps);
|
|
for (const step of testSteps) {
|
|
expect(step.env.RUNNER_ENVIRONMENT, `${name}: ${step.name}`).toBe(
|
|
'${{ runner.environment }}',
|
|
);
|
|
}
|
|
}
|
|
});
|
|
|
|
it('keeps release credentials off PR-shared machines and failure reporting hosted', () => {
|
|
// quality aggregates five pool-routed checks, so it rides the same
|
|
// hk4-host pool they do; on 2026-10-15 it queued 53 minutes on the
|
|
// hosted pool while every machine it depended on sat idle.
|
|
expect(releaseYaml.jobs.quality['runs-on']).toContain('ecs-qwen-hk4-host');
|
|
expect(releaseYaml.jobs.quality['runs-on']).toContain('ubuntu-latest');
|
|
// publish carries CI_BOT_PAT and npm OIDC publishing. ecs-qwen-hk4-host
|
|
// machines are dual-labelled onto the ecs-qwen PR CI pool, so publish may
|
|
// only reach a release-only machine set — and only when explicitly
|
|
// opted in, which keeps the lane hosted until ops provisions it.
|
|
expect(releaseYaml.jobs.publish['runs-on']).toContain(
|
|
'ecs-qwen-release-host',
|
|
);
|
|
expect(releaseYaml.jobs.publish['runs-on']).not.toContain(
|
|
'ecs-qwen-hk4-host',
|
|
);
|
|
expect(releaseYaml.jobs.publish['runs-on']).toContain(
|
|
"vars.ECS_RELEASE_HOST_RUNNER_ENABLED == 'true'",
|
|
);
|
|
expect(releaseYaml.jobs.publish['runs-on']).toContain('ubuntu-latest');
|
|
// Failure reporting must survive whatever broke the pool it reports on.
|
|
expect(releaseYaml.jobs.notify_failure['runs-on']).toBe('ubuntu-latest');
|
|
expect(releaseYaml.jobs.notify_failure['runs-on']).not.toContain(
|
|
'ecs-qwen',
|
|
);
|
|
});
|
|
|
|
it('keeps prebuild collection fail-closed on the release-only pool', () => {
|
|
// collect rides the same opt-in ecs-qwen-release-host lane as publish,
|
|
// so the runs-on expression must keep the repository condition and the
|
|
// hosted fallback that keep forks and the unset default on ubuntu-latest.
|
|
expect(audioPrebuildsYaml.jobs.collect['runs-on']).toContain(
|
|
"github.repository == 'QwenLM/qwen-code'",
|
|
);
|
|
expect(audioPrebuildsYaml.jobs.collect['runs-on']).toContain(
|
|
"vars.ECS_RELEASE_HOST_RUNNER_ENABLED == 'true'",
|
|
);
|
|
expect(audioPrebuildsYaml.jobs.collect['runs-on']).toContain(
|
|
'ecs-qwen-release-host',
|
|
);
|
|
expect(audioPrebuildsYaml.jobs.collect['runs-on']).toContain(
|
|
'ubuntu-latest',
|
|
);
|
|
// On the persistent release-host workspace, download-artifact's
|
|
// merge-multiple merges into existing paths, so a stale prebuilds/ tree
|
|
// from a previous release would leak into the merged artifact. The wipe
|
|
// must stay first and self-hosted-only; dropping it (or reordering it
|
|
// after the merge) re-opens that leak.
|
|
const [wipe, ...rest] = audioPrebuildsYaml.jobs.collect.steps;
|
|
expect(wipe.name).toBe('Clear stale prebuilds (self-hosted)');
|
|
expect(wipe.if).toBe("${{ runner.environment == 'self-hosted' }}");
|
|
expect(wipe.run).toBe('rm -rf prebuilds');
|
|
expect(rest[0].name).toBe('Merge per-platform prebuilds');
|
|
});
|
|
});
|
|
|
|
describe('SDK publish dry-run already-published tolerance', () => {
|
|
// After a release, the packaged version lives in the registry and the
|
|
// dry-run's collision probe is permanently red until the next bump; only
|
|
// that exact registry state may be treated as a skip.
|
|
it.each([
|
|
{
|
|
label: 'npm stderr line',
|
|
text: 'npm error You cannot publish over the previously published versions: 0.26.0.',
|
|
},
|
|
{
|
|
label: 'npm --json error summary',
|
|
text: '{"error":{"code":"E403","summary":"You cannot publish over the previously published versions: 0.26.0."}}',
|
|
},
|
|
])('skips the registry-collision case: $label', ({ text }) => {
|
|
expect(isAlreadyPublishedDryRunFailure(text)).toBe(true);
|
|
});
|
|
|
|
it.each([
|
|
{
|
|
label: 'permission denied',
|
|
text: 'npm error code E403\nnpm error 403 403 Forbidden - PUT https://registry.npmjs.org/@qwen-code%2fcua-sdk - You do not have permission to publish "@qwen-code/cua-sdk".',
|
|
},
|
|
{
|
|
label: 'network failure',
|
|
text: 'npm error code ENOTFOUND\nnpm error network request to https://registry.npmjs.org/@qwen-code%2fcua-sdk failed',
|
|
},
|
|
{
|
|
label: 'bare exit without captured detail',
|
|
text: 'npm exited with status 1',
|
|
},
|
|
{ label: 'empty', text: '' },
|
|
])('keeps failing every other registry failure: $label', ({ text }) => {
|
|
expect(isAlreadyPublishedDryRunFailure(text)).toBe(false);
|
|
});
|
|
});
|