vrr/openclaw
2
0
Fork 0
mirror of https://github.com/openclaw/openclaw.git synced 2026-11-02 05:27:21 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 37

refactor(extensions): remove speculative runtime coordination (#168841)

Browse source 
* refactor(extensions): drop redundant catalog and cache rechecks

Best-effort tradeoffs accepted: an admitted native catalog read can finish after its setting changes, and an edited memory file can leave unused content-addressed vectors until normal pruning. Keep execution authority, cancellation, index publication validation, session tombstones, and Forget behavior unchanged.

* refactor(extensions): simplify speculative runtime coordination

* refactor(browser): use pending shutdown admission without epochs

* fix(browser): finish screencast lifecycle signal cutover

* test(extensions): prune obsolete mock baseline entries

* refactor(extensions): finish consult state and cleanup simplification

* test(extensions): align coverage with simplified runtime paths

* test(qa): retain cleanup errors without replaying failed work

* chore(extensions): remove lint leftovers from runtime simplification

* test(qa): observe committed announcement delivery state

Backport the test-only canonical query from cfcb7bda17 (#168796). The parent observer must read child Gateway commits without reusing an empty-queue cache. Preserve all custody assertions and existing timeouts.
This commit is contained in:
Peter Steinberger 2026-10-10 23:09:20 -07:00 • committed by GitHub
parent 01b25a9da8
commit 7410a1a70d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
134 changed files with 1088 additions and 5139 deletions
Download patch file Download diff file Expand all files Collapse all files

3
config/test-mock-exports-baseline.txt
View file

@ -286,7 +286,6 @@
["extensions/browser/src/plugin-service.test.ts","openclaw/plugin-sdk/plugin-runtime","f388834be47f8a57d33bfa13e4ca34055446b8396de05d2a4d6623f7b2bf9c24",1]
["extensions/browser/src/session-browser-dashboard.test.ts","./browser-control-state.js","fc85331ef3f4c181efd61c3091b852eda7cfdfba1ad7becffb6ed296904ac8fd",1]
["extensions/browser/src/session-browser-dashboard.test.ts","./browser/pw-ai-module.js","3729253210b4d40d753d43f4556cdbffb3f72fcebebff866926a91a582b3d32a",1]
["extensions/browser/src/session-browser-dashboard.test.ts","./browser/server-context.lifecycle.js","a4454fe465eed147574d52e8f68d59fad06a9ae4c91cc7c2f6021c07c4477f35",1]
["extensions/buzz/src/directory.test.ts","./gateway.js","fb61948729773cd9beeb4b1877b31797e40689f1751aaad22023b6dc73b35070",1]
["extensions/buzz/src/gateway.cold-start-recovery.test.ts","./inbound.js","32f0e199331fb4d11efd8c68d851ff2ee4b98e665c8b8181c629c69d9cf6a96e",1]
["extensions/buzz/src/gateway.lifecycle.test.ts","./buzz-bus.js","3ed9e8f54f478b7157eb84d07e12f367776dc8c4b235459a4c4a0bd93d595b4e",1]
@ -1083,8 +1082,6 @@
["extensions/memory-core/src/memory-tool-manager.test-mocks.ts","./tools.runtime.js","2da9c55b891d38b2f9aee8a066fdbfd61b78b543ede9d5a875f2571d45cd479b",1]
["extensions/memory-core/src/memory-workspace-lock.cas.test.ts","./dreaming-state.js","f20ed71534a970e301435288ceeb18d5d0cf6822d21ef85a6807063d9f2af277",1]
["extensions/memory-core/src/memory-workspace-lock.cas.test.ts","openclaw/plugin-sdk/process-runtime","03b18c14758ddfde64a51af9687f05e3cd33a71ba51774bfc74e431f454728c3",1]
["extensions/memory-core/src/memory-workspace-lock.cas.test.ts","openclaw/plugin-sdk/runtime-env","2b6ff10d2e3fef0a26225950a2d784fc3e4d80826dcde9166c07072ad6b7cd12",1]
["extensions/memory-core/src/memory/manager-index-generation-lease.async.test.ts","./manager-sqlite-lease.js","f0b4758d5a57c5730530cde927768bfce7c433778d9868905130f6b9b21e0f43",1]
["extensions/memory-core/src/memory/manager-sync-yield.test.ts","./embeddings.js","ebb8fff40836b2bb2cc9b79ee71c224a01cee2847d37ba26caf51aa1e3371204",1]
["extensions/memory-core/src/memory/manager.fts-only-reindex.test.ts","./embeddings.js","39a5c7e02e6c076f88561b0f11aae022947971660d55c6b29f99960942e66c9e",1]
["extensions/memory-core/src/memory/search-manager.test.ts","../../manager-runtime.js","5f46121d3b36d98dc24f25dfb5b015912e04160c699575eb5e48094948732a51",1]

5
docs/cli/doctor/state-migrations.md
View file

@ -234,8 +234,9 @@ or renaming that file before retrying.
Sibling reindex locks, captures, and SQLite sidecars are not separate agent stores.
Current-layout database sidecars still preserve evidence of a missing main database.
Previously recorded coordination-file holds remain preserved in recovery receipts
but do not produce held-agent warnings. The reindex lock file can remain after
its SQLite lease is released; its presence does not mean an agent was deleted.
but do not produce held-agent warnings. Legacy reindex lock files can remain
after an upgrade; current reindex coordination is in-process, and a lock file's
presence does not mean an agent was deleted.
Session SQLite import and recovery hold existing agent databases and their sidecars
when deletion history is unavailable, preserving legacy sources without importing
or archiving them. Recorded deletion and reconstruction holds and retained plugin

5
docs/plugins/workboard.md
View file

@ -721,8 +721,9 @@ its connections.
Workboard instances invalidate cached card lists and board revisions using the
physical database's in-process writer receipts, including commits through sibling
instances. Card-list reuse and publication are bracketed by that receipt; an
unsettled receipt leaves the read uncached. Change notifications use owner publications;
instances. Card-list reuse checks that receipt; an unsettled receipt leaves the
read uncached. A read overlapping a mutation can return its older snapshot; the
next request observes the writer receipt and reloads it. Change notifications use owner publications;
there is no timer polling SQLite for writes from other processes. If a worker
reply fails after a possible commit, the owner discards cached facts and rereads
them on the next use without replaying the mutation.

16
docs/reference/database-schemas/worker-access.md
View file

@ -3190,8 +3190,8 @@ binary values and is discarded after settlement or owner closure. Source-file in
host. Cache pruning uses that same worker for its live count and oldest-row deletion,
with a transaction recheck before each batch of at most 100 rows. The host releases
admission and yields between batches; only definite pre-entry lock failures retry.
Cache and source-hash reads use the retained publication worker, with caller
authority checked after delivery. Source snapshots use that same publication owner;
Cache and source-hash reads use the retained publication worker, checking that
its handle is open at admission. Source snapshots use that same publication owner;
read-only diagnostics use the retrieval worker. Shadow session publication prepares
its current tombstone predicate in the original published worker while retaining
the workspace lock through commit. Ordinary publication checks that predicate only
@ -3200,12 +3200,12 @@ publication authority. Cold opening remains separate work. Schemas, cache retent
and stored formats are unchanged.
Session-only lexical searches return hits and recall metadata in one retrieval
request. Their existing generation owner retains exclusive admission through
result selection; source mutations and forget retain shared admission through
settlement. Already-admitted ordinary readers can finish on their shared generation.
Rebuild and provider recovery release read custody before waiting, then prepare
fresh results. Forget therefore either precedes the fused read or follows its
acceptance. Cancellation and close join the accepted read and lock cleanup. Mixed and semantic searches
request. An in-process reader/writer queue orders retrieval and publication, and
the workspace queue orders source mutations and Forget through settlement.
Rebuild and provider recovery release the read slot before waiting, then prepare
fresh results. Cancellation and close join accepted work. These queues do not
coordinate foreign processes: CLI and maintenance writers must use the Gateway
or hold exclusive ownership while it is stopped. Mixed and semantic searches
retain their final asynchronous metadata reader. No schema, retention, durability,
SDK, or update migration changes.

47
extensions/acpx/index.test.ts
View file

@ -326,7 +326,7 @@ describe("acpx plugin", () => {
expect(getRuntime).not.toHaveBeenCalled();
});
it.for(["enabled", "disabled", "disposed"] as const)(
it.for(["enabled", "disposed"] as const)(
"inspects a real native catalog without runtime state when the harness becomes %s",
async (lifecycle, { signal }) => {
const directory = await fs.mkdtemp(path.join(os.tmpdir(), "acpx-native-catalog-"));
@ -343,7 +343,7 @@ export { sendReceipt };`,
throw new Error("Catalog inspection must not acquire the runtime");
});
createAcpxRuntimeServiceMock.mockReturnValue({ id: "acpx", getRuntime });
let config: OpenClawPluginApi["config"] = {};
const config: OpenClawPluginApi["config"] = {};
const harnesses = new Map<string, Parameters<OpenClawPluginApi["registerAgentHarness"]>[0]>();
plugin.register(
createTestPluginApi({
@ -410,11 +410,6 @@ export { sendReceipt };`,
]),
signal,
);
if (lifecycle === "disabled") {
config = {
plugins: { entries: { acpx: { config: { nativeAgents: { opencode: false } } } } },
};
}
if (lifecycle === "disposed") {
await harness.dispose?.();
} else {
@ -426,27 +421,23 @@ export { sendReceipt };`,
expect(result.models).toBeUndefined();
} else {
expect(result.error).toBeUndefined();
expect(result.models).toEqual(
lifecycle === "disabled"
? { entries: [] }
: {
entries: [
{
provider: "acp-opencode",
id: "initial",
name: "Initial",
nativeRuntime: "acp-opencode",
},
{
provider: "acp-opencode",
id: "selected",
name: "Selected",
nativeRuntime: "acp-opencode",
},
],
outcomes: [{ provider: "acp-opencode", status: "ready" }],
},
);
expect(result.models).toEqual({
entries: [
{
provider: "acp-opencode",
id: "initial",
name: "Initial",
nativeRuntime: "acp-opencode",
},
{
provider: "acp-opencode",
id: "selected",
name: "Selected",
nativeRuntime: "acp-opencode",
},
],
outcomes: [{ provider: "acp-opencode", status: "ready" }],
});
}
const sessionId = await fs.readFile(
path.join(peerDirectory, "session-new-entered"),

7
extensions/acpx/src/harness.ts
View file

@ -166,9 +166,7 @@ export function createAcpAgentHarness(params: {
: (finiteSecondsToTimerSafeMilliseconds(config.timeoutSeconds) ?? 1),
});
generation.signal.throwIfAborted();
if (!params.isEnabled()) {
return { entries: [] };
}
// A settings change takes effect on the next catalog request.
return {
entries: (models?.availableModels ?? []).map((model) => ({
provider: id,
@ -180,9 +178,6 @@ export function createAcpAgentHarness(params: {
};
} catch (error) {
generation.signal.throwIfAborted();
if (!params.isEnabled()) {
return { entries: [] };
}
// ACP SDK RequestError.authRequired reserves this code/message pair;
// a generic JSON-RPC server error with the same code is not an auth rejection.
const authRequired =

43
extensions/acpx/src/pi-session-catalog.test.ts
View file

@ -472,49 +472,6 @@ describe("Pi session catalog", () => {
);
});
it("rebuilds metadata after a same-file replacement grows", async () => {
const directory = await createPiStore("old session");
const file = path.join(directory, "session.jsonl");
const expireCandidates = usePiCandidateCacheClock();
await listLocalPiSessionPage({ limit: 20 });
const entries = [
{
type: "session",
version: 3,
id: "pi-replaced-session",
timestamp: "2026-07-13T11:00:00Z",
cwd: "/workspace/replaced",
},
{
type: "message",
id: "replacement-user",
parentId: null,
timestamp: "2026-07-13T11:00:01Z",
message: { role: "user", content: `replacement ${"x".repeat(4_096)}` },
},
{
type: "session_info",
id: "replacement-name",
parentId: "replacement-user",
timestamp: "2026-07-13T11:00:02Z",
name: "Replacement session",
},
];
await fs.writeFile(file, `${entries.map((entry) => JSON.stringify(entry)).join("\n")}\n`);
expireCandidates();
await expect(listLocalPiSessionPage({ limit: 20 })).resolves.toMatchObject({
sessions: [
expect.objectContaining({
threadId: "pi-replaced-session",
name: "Replacement session",
cwd: "/workspace/replaced",
}),
],
});
});
it("does not reuse transcript paths after the configured store changes", async () => {
await createPiStore("old store");
await listLocalPiSessionPage({ limit: 20 });

106
extensions/acpx/src/pi-session-store.ts
View file

@ -18,7 +18,6 @@ const SUMMARY_SCAN_BATCH_SIZE = 100;
const MAX_SUMMARY_CACHE_ENTRIES = 256;
const MAX_SESSION_BYTES = 32 * 1024 * 1024;
const MAX_SUMMARY_LINE_BYTES = 1024 * 1024;
const APPEND_PROOF_EDGE_BYTES = 64 * 1024;
const IO_CONCURRENCY = 8;
const PI_FILE_CANDIDATE_CACHE_TTL_MS = 32_000;
const PI_FILE_CANDIDATE_CACHE_MAX_ENTRIES = 8;
@ -46,7 +45,6 @@ type PiSummaryScanState = {
type CachedSummary = PiFileCandidate & {
summary?: PiSessionSummary;
scanState: PiSummaryScanState;
appendProof: { head: Buffer; tail: Buffer };
};
type PiFileCandidateCacheEntry = {
@ -301,44 +299,6 @@ async function scanSummaryAppend(
}
}
async function readAppendProof(
file: string,
size: number,
): Promise<{ head: Buffer; tail: Buffer }> {
const length = Math.min(size, APPEND_PROOF_EDGE_BYTES);
if (length === 0) {
return { head: Buffer.alloc(0), tail: Buffer.alloc(0) };
}
const handle = await fs.open(file, "r");
try {
const head = Buffer.alloc(length);
const tail = Buffer.alloc(length);
const [headRead, tailRead] = await Promise.all([
handle.read(head, 0, length, 0),
handle.read(tail, 0, length, size - length),
]);
return {
head: head.subarray(0, headRead.bytesRead),
tail: tail.subarray(0, tailRead.bytesRead),
};
} finally {
await handle.close();
}
}
async function cachedPrefixIsUnchanged(candidate: PiFileCandidate, cached: CachedSummary) {
if (cached.identity !== candidate.identity || cached.size >= candidate.size) {
return false;
}
// Pi persists established sessions with appendFileSync. Its in-place rewrite
// paths (notably version migration) rewrite the header, so the head proof
// rejects them; the tail proof rejects truncation before later growth.
const current = await readAppendProof(candidate.file, cached.size);
return (
current.head.equals(cached.appendProof.head) && current.tail.equals(cached.appendProof.tail)
);
}
async function readPiSessionSummary(
candidate: PiFileCandidate,
): Promise<PiSessionSummary | undefined> {
@ -352,12 +312,11 @@ async function readPiSessionSummary(
}
let summary: PiSessionSummary | undefined;
let scanState: PiSummaryScanState;
let appendProof: CachedSummary["appendProof"];
try {
// Pi normally appends JSONL. Resume only when bounded edge proofs show the
// previously indexed prefix survived; rewrites rebuild from byte zero.
// Pi appends active sessions. An in-place rewrite that also grows may leave
// this read-only catalog stale until eviction; full reads still verify identity.
const resumable =
cached && (await cachedPrefixIsUnchanged(candidate, cached)) ? cached : undefined;
cached?.identity === candidate.identity && cached.size < candidate.size ? cached : undefined;
scanState = resumable
? {
...resumable.scanState,
@ -369,7 +328,6 @@ async function readPiSessionSummary(
invalid: false,
};
await scanSummaryAppend(candidate, resumable?.size ?? 0, scanState);
appendProof = await readAppendProof(candidate.file, candidate.size);
// A complete final record is valid without a newline. Project it from a
// clone so later appends can still finish the cached pending line once.
const projectedState = { ...scanState, pending: Buffer.from(scanState.pending) };
@ -408,7 +366,7 @@ async function readPiSessionSummary(
if (cached?.summary?.threadId && cached.summary.threadId !== summary?.threadId) {
threadFileCache.delete(threadCacheKey(cached.storeRoot, cached.summary.threadId));
}
cacheSummary(candidate.file, { ...candidate, summary, scanState, appendProof });
cacheSummary(candidate.file, { ...candidate, summary, scanState });
if (summary) {
threadFileCache.set(threadCacheKey(candidate.storeRoot, summary.threadId), candidate.file);
}
@ -512,37 +470,29 @@ export async function readPiSessionById(
env: NodeJS.ProcessEnv,
): Promise<Record<string, unknown>[]> {
const cacheKey = threadCacheKey(piSessionStore(env).root, threadId);
let file = threadFileCache.get(cacheKey);
for (let attempt = 0; attempt < 2; attempt += 1) {
if (!file) {
file = (await findPiSummary(threadId, env))?.file;
}
if (!file) {
throw new Error("Pi session was not found");
}
try {
const stats = await fs.stat(file);
if (!stats.isFile()) {
throw new Error("Pi session is not a file");
}
if (stats.size > MAX_SESSION_BYTES) {
throw new RangeError("Pi session exceeds the 32 MiB read safety limit");
}
const entries = parsePiJsonLines(await fs.readFile(file, "utf8"));
if (entries[0]?.type === "session" && entries[0].id === threadId) {
return entries;
}
} catch (error) {
if (error instanceof RangeError) {
throw error;
}
if (attempt > 0) {
throw new Error("Pi session is unavailable", { cause: error });
}
}
// The cached path can disappear when Pi replaces or prunes a session file.
threadFileCache.delete(cacheKey);
file = undefined;
const file = threadFileCache.get(cacheKey) ?? (await findPiSummary(threadId, env))?.file;
if (!file) {
throw new Error("Pi session was not found");
}
try {
const stats = await fs.stat(file);
if (!stats.isFile()) {
throw new Error("Pi session is not a file");
}
if (stats.size > MAX_SESSION_BYTES) {
throw new RangeError("Pi session exceeds the 32 MiB read safety limit");
}
const entries = parsePiJsonLines(await fs.readFile(file, "utf8"));
if (entries[0]?.type !== "session" || entries[0].id !== threadId) {
throw new Error("Pi session changed during read");
}
return entries;
} catch (error) {
// A replaced or pruned file fails this request; a later catalog scan rediscovers it.
threadFileCache.delete(cacheKey);
if (error instanceof RangeError) {
throw error;
}
throw new Error("Pi session is unavailable", { cause: error });
}
throw new Error("Pi session changed during read");
}

1
extensions/acpx/src/runtime-probe.ts
View file

@ -30,7 +30,6 @@ export class AcpxRuntimeProbe {
const runtime = this.params.createRuntime(agent);
try {
const report = await this.params.runWithLease(agent, () => runtime.doctor());
this.params.assertRunning();
this.health = { agent, ok: report.ok };
return report;
} finally {

13
extensions/acpx/src/runtime-session-store.ts
View file

@ -92,7 +92,6 @@ export type GenerationHandle = OpenClawRuntimeHandle & { [acpxGenerationKey]?: A
export const acpxOperationScope = new AsyncLocalStorage<{
generation: AcpxGeneration;
closeRecord?: AcpLoadedSessionRecord;
recordId?: string;
}>();
export function readSessionRecordName(record: unknown): string {
@ -200,12 +199,6 @@ export function createResetAwareSessionStore(
return undefined;
}
const record = await baseStore.load(sessionId);
if (
scope?.generation.retired ||
freshSessionKeys.has(scope?.generation.resource ?? normalized)
) {
return undefined;
}
if (scope && record) {
captureGenerationRecord(scope.generation, record);
}
@ -221,9 +214,6 @@ export function createResetAwareSessionStore(
if (!lease) {
return record;
}
if (scope?.generation.retired) {
return undefined;
}
const leasedRecord = withOpenClawLeaseSessionMetadata(record, lease);
if (scope) {
captureGenerationRecord(scope.generation, leasedRecord);
@ -281,9 +271,6 @@ export function createResetAwareSessionStore(
})
) {
const existing = await params.leaseStore.load(leaseIdentity.leaseId);
if (scope?.generation.retired) {
return;
}
const ownsExisting =
!existing ||
(existing.gatewayInstanceId === leaseIdentity.gatewayInstanceId &&

43
extensions/acpx/src/runtime.ts
View file

@ -513,12 +513,12 @@ export class AcpxRuntime implements CompleteAcpRuntime {
private async runInGeneration<T>(
target: BridgeSession & { acpxRecordId?: string; bridgeSession?: BridgeSession | null },
scope: { generation: AcpxGeneration; closeRecord?: AcpLoadedSessionRecord; recordId?: string },
scope: { generation: AcpxGeneration; closeRecord?: AcpLoadedSessionRecord },
run: () => Promise<T>,
): Promise<T> {
const release = this.generationRegistry.retainGenerationOperation(
scope.generation,
scope.recordId ?? target.acpxRecordId ?? scope.generation.resource,
target.acpxRecordId ?? scope.generation.resource,
);
try {
return await this.sessionScope.run(resolveBridgeSession(target), () =>
@ -544,40 +544,17 @@ export class AcpxRuntime implements CompleteAcpRuntime {
allowRetired = false,
): Promise<AcpxHandleOperationSnapshot> {
const resource = generation.resource;
if (!allowRetired) {
this.generationRegistry.assertCurrentGeneration(generation);
}
const ownedRecord = generation.records.get(handle.acpxRecordId ?? resource);
if (
ownedRecord &&
((handle.acpxRecordId && ownedRecord.acpxRecordId !== handle.acpxRecordId) ||
(handle.backendSessionId &&
ownedRecord.acpSessionId &&
ownedRecord.acpSessionId !== handle.backendSessionId))
) {
throw new AcpRuntimeError(
"ACP_TURN_FAILED",
"ACP handle no longer owns this runtime generation.",
);
}
let record = allowRetired
const record = allowRetired
? generation.retired
? ownedRecord
: await this.sessionStore.loadForClose(handle.acpxRecordId ?? resource)
: await acpxOperationScope.run({ generation }, () =>
this.sessionStore.load(handle.acpxRecordId ?? resource),
);
// A reset can retire this generation while the snapshot read is pending.
// Prefer its captured record over any replacement now visible in storage.
if (allowRetired && generation.retired && ownedRecord) {
record = ownedRecord;
}
if (allowRetired && record) {
captureGenerationRecord(generation, record);
}
if (!allowRetired) {
this.generationRegistry.assertCurrentGeneration(generation);
}
if (
record &&
((handle.acpxRecordId && handle.acpxRecordId !== record.acpxRecordId) ||
@ -617,9 +594,6 @@ export class AcpxRuntime implements CompleteAcpRuntime {
);
}
}
if (!allowRetired) {
this.generationRegistry.assertCurrentGeneration(generation);
}
return { record, command, generation };
}
@ -628,16 +602,11 @@ export class AcpxRuntime implements CompleteAcpRuntime {
run: (snapshot: AcpxHandleOperationSnapshot) => Promise<T>,
): Promise<T> {
const generation = this.generationForHandle(handle);
// Hold the owner before lookup can yield; the verified record gets its own
// reservation without leaving a gap between snapshot and operation custody.
// Keep the runtime alive through lookup and execution, then check once at the effect.
return await this.runInGeneration(handle, { generation }, async () => {
const snapshot = await this.loadOperationSnapshotForHandle(handle, generation);
this.generationRegistry.assertCurrentGeneration(generation);
return await this.runInGeneration(
handle,
{ generation, recordId: snapshot.record?.acpxRecordId },
() => run(snapshot),
);
return await run(snapshot);
});
}
@ -854,7 +823,6 @@ export class AcpxRuntime implements CompleteAcpRuntime {
sessionKey: resource,
agent: input.agent,
});
this.generationRegistry.assertCurrentGeneration(generation);
// Returned handles keep their reset fence, including closed persisted history.
generation.admitted ||= Boolean(handle);
return handle
@ -1038,7 +1006,6 @@ export class AcpxRuntime implements CompleteAcpRuntime {
});
const turnPromise = this.runWithOperationSnapshot(input.handle, (snapshot) => {
const { command, generation } = snapshot;
this.generationRegistry.assertCurrentGeneration(generation);
const delegate = this.resolveDelegateForOperationSnapshot(input.handle, snapshot);
return this.sessionScope.run(resolveBridgeSession(input.handle), () =>
acpxOperationScope.run({ generation }, () =>

14
extensions/acpx/src/service.ts
View file

@ -195,9 +195,7 @@ async function reapOpenAcpxProcessLeases(params: {
if (pending) {
legacyWrapperRoots.add(lease.wrapperRoot);
}
assertCurrent?.();
await params.leaseStore.markState(lease.leaseId, "closing");
assertCurrent?.();
const result = pending
? await cleanupOpenClawOwnedAcpxPendingLease({
leaseId: lease.leaseId,
@ -225,7 +223,6 @@ async function reapOpenAcpxProcessLeases(params: {
result.skippedReason === "unverified-root" ||
(lease.sessionKey === ACPX_PROBE_LEASE_SESSION_KEY &&
result.skippedReason === "missing-root")));
assertCurrent?.();
await params.leaseStore.markState(
lease.leaseId,
retryableEvidenceFailure ? "open" : result.terminatedPids.length > 0 ? "closed" : "lost",
@ -235,7 +232,6 @@ async function reapOpenAcpxProcessLeases(params: {
// proves this Gateway had an uncertain spawn. Keep aggregate results wholly
// separate from the state transition of any specific lease.
for (const wrapperRoot of legacyWrapperRoots) {
assertCurrent?.();
const legacyResult = await reapStaleOpenClawOwnedAcpxOrphans({
wrapperRoot,
assertCurrent,
@ -275,7 +271,6 @@ export function createAcpxRuntimeService(
assertCurrent();
recoveryPromise ??= measureAcpxStartup(ctx, "process-leases.reap", async () => {
const result = await recover(assertCurrent);
assertCurrent();
if (result.terminatedPids.length > 0) {
ctx.logger.info(
`reaped ${result.terminatedPids.length} stale OpenClaw-owned ACPX processes`,
@ -283,7 +278,6 @@ export function createAcpxRuntimeService(
}
});
await recoveryPromise;
assertCurrent();
};
return {
@ -382,8 +376,6 @@ export function createAcpxRuntimeService(
return;
}
lifecycleRevision += 1;
const currentRevision = lifecycleRevision;
try {
const timeoutSeconds = pluginConfig.timeoutSeconds ?? DEFAULT_ACPX_TIMEOUT_SECONDS;
const doctorReport = await measureAcpxStartup(ctx, "probe.availability", () =>
@ -398,9 +390,6 @@ export function createAcpxRuntimeService(
{ ref: false },
),
);
if (currentRevision !== lifecycleRevision) {
return;
}
if (doctorReport.ok) {
ctx.startupTrace?.detail?.("probe.result", [["healthyCount", 1]]);
ctx.logger.info("embedded acpx runtime backend ready");
@ -411,9 +400,6 @@ export function createAcpxRuntimeService(
`embedded acpx runtime backend check failed: ${formatDoctorFailureMessage(doctorReport)}`,
);
} catch (err) {
if (currentRevision !== lifecycleRevision) {
return;
}
ctx.startupTrace?.detail?.("probe.result", [["healthyCount", 0]]);
ctx.logger.warn(`embedded acpx runtime setup failed: ${formatErrorMessage(err)}`);
}

41
extensions/browser/src/browser-control-state.test.ts
View file

@ -49,33 +49,29 @@ beforeEach(() => {
});
describe("browser control lifecycle", () => {
it("allows a start queued after a no-state stop", async () => {
const stopping = stop("service");
const starting = start();
await expect(stopping).resolves.toBeNull();
await expect(starting).resolves.toBeTruthy();
it("allows a start after a no-state stop settles", async () => {
await expect(stop("service")).resolves.toBeNull();
await expect(start()).resolves.toBeTruthy();
await stop("service");
});
it("rejects a start requested after stop intent but before stop drains", async () => {
it("rejects new starts while ordinary shutdown is pending", async () => {
await start();
let releaseStop!: () => void;
const stopGate = new Promise<void>((resolve) => {
releaseStop = resolve;
});
const gate = Promise.withResolvers<void>();
runtimeMocks.stopBrowserRuntime.mockImplementationOnce(async (params) => {
await stopGate;
await gate.promise;
params.clearState();
});
const stopping = stop("service");
const starting = start();
releaseStop();
await stopping;
await expect(starting).rejects.toThrow("stopping");
try {
await expect(start()).rejects.toThrow("Browser runtime is stopping.");
} finally {
gate.resolve();
await stopping;
}
expect(getBrowserControlState()).toBeNull();
await expect(start()).resolves.toBeTruthy();
await stop("service");
});
it("retains a failed stop owner for an exact retry", async () => {
@ -110,14 +106,11 @@ describe("browser control lifecycle", () => {
expect(runtimeMocks.stopBrowserRuntime).toHaveBeenCalledOnce();
});
it("allows a start queued after a foreground-owned service stop", async () => {
it("allows a start after a foreground-owned service stop settles", async () => {
await start();
await start({} as Server);
const stopping = stop("service");
const starting = start();
await expect(stopping).resolves.toBeNull();
await expect(starting).resolves.toBeTruthy();
await expect(stop("service")).resolves.toBeNull();
await expect(start()).resolves.toBeTruthy();
await stop("server");
});

17
extensions/browser/src/browser-control-state.ts
View file

@ -12,8 +12,8 @@ type BrowserControlOwner = "server" | "service";
let state: BrowserServerState | null = null;
let lifecycleTail = Promise.resolve();
let completedEffectiveStops = 0;
let pendingLifecycles = 0;
let pendingStops = 0;
/** Serialize complete Browser runtime start/stop workflows. */
function enqueueBrowserControlLifecycle<T>(run: () => Promise<T>): Promise<T> {
@ -28,14 +28,13 @@ function enqueueBrowserControlLifecycle<T>(run: () => Promise<T>): Promise<T> {
return result;
}
/** Queue startup, but never turn a request made during shutdown into a post-stop restart. */
/** Serialize startup with the runtime's stop workflow. */
export function withBrowserControlStart<T>(run: () => Promise<T>): Promise<T> {
const effectiveStopsAtRequest = completedEffectiveStops;
if (pendingStops > 0) {
return Promise.reject(new BrowserProfileUnavailableError("Browser runtime is stopping."));
}
return enqueueBrowserControlLifecycle(() => {
if (
completedEffectiveStops !== effectiveStopsAtRequest ||
(state ? !isBrowserRuntimeRunning(state) : false)
) {
if (state && !isBrowserRuntimeRunning(state)) {
throw new BrowserProfileUnavailableError("Browser runtime is stopping.");
}
return run();
@ -94,6 +93,7 @@ export function stopBrowserControlRuntime(params: {
closeServer?: boolean;
onWarn: (message: string) => void;
}): Promise<BrowserServerState | null> {
pendingStops += 1;
return enqueueBrowserControlLifecycle(async () => {
const current = state;
if (!current) {
@ -112,7 +112,8 @@ export function stopBrowserControlRuntime(params: {
closeServer: params.closeServer,
onWarn: params.onWarn,
});
completedEffectiveStops += 1;
return current;
}).finally(() => {
pendingStops -= 1;
});
}

2
extensions/browser/src/browser-dashboard-events.ts
View file

@ -81,8 +81,6 @@ export function bindBrowserDashboardEvents(
(dashboard.definition.agentId === event.agentId &&
parseAgentSessionKey(dashboard.definition.sessionKey)?.rest === event.sessionKey)
) {
// Fence retained frames synchronously; the next operation verifies the current definition.
dashboard.definitionChanged();
void dashboard.assertDefinitionCurrent().catch(() => {});
}
}

1
extensions/browser/src/browser-dashboard.types.ts
View file

@ -40,6 +40,5 @@ export type SessionBrowserDashboard = {
signal: AbortSignal;
assertCurrent: () => void;
assertDefinitionCurrent: () => Promise<void>;
definitionChanged: () => void;
close: () => Promise<void>;
};

160
extensions/browser/src/browser/chrome-mcp-session.ts
View file

@ -26,7 +26,6 @@ type PendingAttach = ReturnType<typeof createChromeMcpSession> & {
waiters: number;
settled: boolean;
cancelled: boolean;
cleanupSettled: boolean;
session?: ChromeMcpSession;
};
@ -122,18 +121,12 @@ class ChromeMcpSessionOwner {
}
private async drainPending(pending: PendingAttach): Promise<void> {
const settled = pending.cleanupSettled;
try {
await pending.cleanup;
} catch (error) {
// Concurrent waiters observe the original failure; later admission retries its retained handle.
if (!settled) {
throw error;
} finally {
if (this.pending === pending) {
this.pending = undefined;
}
await this.drainRetired();
}
if (this.pending === pending) {
this.pending = undefined;
}
}
@ -165,7 +158,6 @@ class ChromeMcpSessionOwner {
waiters: 0,
settled: false,
cancelled: false,
cleanupSettled: false,
};
this.pending = pending;
owners.set(this.key, this);
@ -182,9 +174,6 @@ class ChromeMcpSessionOwner {
.finally(() => {
pending.settled = true;
});
pending.cleanup = creation.cleanup.finally(() => {
pending.cleanupSettled = true;
});
void pending.promise.catch(() => {});
void pending.cleanup.catch(() => {});
return pending;
@ -241,107 +230,84 @@ class ChromeMcpSessionOwner {
async lease(options: ChromeMcpCallOptions): Promise<ChromeMcpSessionLease> {
this.admissions++;
// A caller arriving on a live session must not reconnect after queued cleanup overtakes it.
const admittedSession =
!this.pending && this.session?.transport.pid !== null ? this.session : undefined;
try {
if (!options.ephemeral) {
await stopOwners(this.profileName, this);
}
options.signal?.throwIfAborted();
return await this.acquire(options, admittedSession);
return await this.acquire(options);
} finally {
this.admissions--;
this.forgetIfEmpty();
}
}
private async acquire(
options: ChromeMcpCallOptions,
admittedSession?: ChromeMcpSession,
): Promise<ChromeMcpSessionLease> {
for (let retry = 0; ; retry++) {
if (!admittedSession) {
if (this.pending?.cancelled) {
await this.drainPending(this.pending);
}
await this.drainRetired();
options.signal?.throwIfAborted();
if (this.session?.transport.pid === null) {
await this.close(this.session);
}
if (this.pending?.cancelled) {
continue;
}
}
const temporary = Boolean(
!admittedSession && options.ephemeral && (this.pending || !this.session),
);
let session = admittedSession ?? (this.pending ? undefined : this.session);
if (temporary) {
this.temporary++;
const creation = createChromeMcpSession(
this,
this.profileName,
this.options,
options.signal,
);
private async acquire(options: ChromeMcpCallOptions): Promise<ChromeMcpSessionLease> {
if (this.pending?.cancelled) {
await this.drainPending(this.pending);
}
await this.drainRetired();
options.signal?.throwIfAborted();
if (this.session?.transport.pid === null) {
await this.close(this.session);
}
const temporary = Boolean(options.ephemeral && (this.pending || !this.session));
let session = this.pending ? undefined : this.session;
if (temporary) {
this.temporary++;
const creation = createChromeMcpSession(this, this.profileName, this.options, options.signal);
try {
session = await creation.promise;
await waitForChromeMcpReady(session, this.profileName, options.timeoutMs, options.signal);
} catch (error) {
try {
session = await creation.promise;
await waitForChromeMcpReady(session, this.profileName, options.timeoutMs, options.signal);
} catch (error) {
await creation.cleanup;
if (session) {
await this.close(session);
}
} finally {
this.temporary--;
this.forgetIfEmpty();
}
throw error;
}
} else if (session) {
try {
await waitForChromeMcpReady(session, this.profileName, options.timeoutMs, options.signal);
} catch (error) {
if (!options.ephemeral || !options.signal?.aborted) {
await this.close(session);
}
throw error;
}
} else {
session = await this.join(this.pending ?? this.start(), options);
}
if (!options.ephemeral && session.transport.pid === null) {
if (this.pending?.session === session) {
this.pending = undefined;
}
await this.close(session);
throw new BrowserProfileUnavailableError(
`Chrome MCP existing-session attach failed for profile "${redactChromeMcpProfileLabelForDiagnostic(this.profileName)}". ` +
"The Chrome MCP subprocess exited before it became usable.",
);
}
return {
session,
temporary,
owner: this,
release: async () => {
if (temporary) {
try {
await creation.cleanup;
if (session) {
await this.close(session);
}
await this.close(session);
} finally {
this.temporary--;
this.forgetIfEmpty();
}
throw error;
}
} else if (session) {
try {
await waitForChromeMcpReady(session, this.profileName, options.timeoutMs, options.signal);
} catch (error) {
if (!options.ephemeral || !options.signal?.aborted) {
await this.close(session);
}
throw error;
}
} else {
session = await this.join(this.pending ?? this.start(), options);
}
if (!admittedSession && !options.ephemeral && session.transport.pid === null) {
if (this.pending?.session === session) {
this.pending = undefined;
}
await this.close(session);
if (retry === 0) {
continue;
}
throw new BrowserProfileUnavailableError(
`Chrome MCP existing-session attach failed for profile "${redactChromeMcpProfileLabelForDiagnostic(this.profileName)}". ` +
"The Chrome MCP subprocess exited before it became usable.",
);
}
return {
session,
temporary,
owner: this,
release: async () => {
if (temporary) {
try {
await this.close(session);
} finally {
this.temporary--;
this.forgetIfEmpty();
}
}
},
};
}
},
};
}
}

115
extensions/browser/src/browser/chrome-mcp.test.ts
View file

@ -327,7 +327,6 @@ describe("chrome MCP page parsing", () => {
const { promise: listGate, resolve: releaseList } = createDeferred<void>();
const { promise: closeStarted, resolve: markCloseStarted } = createDeferred<void>();
const { promise: closeGate, resolve: releaseClose } = createDeferred<void>();
let factoryCalls = 0;
const session = createPageSession({
pid: 141,
pages: [{ id: 1, url: "https://a.example" }],
@ -346,7 +345,6 @@ describe("chrome MCP page parsing", () => {
});
session.client.close = close as typeof session.client.close;
setChromeMcpSessionFactoryForTest(async () => {
factoryCalls += 1;
return session;
});
@ -369,9 +367,8 @@ describe("chrome MCP page parsing", () => {
expect(explicitCloseSettled).toBe(false);
releaseClose();
await expect(active).rejects.toThrow(/transport failed before stop/);
await expect(queued).rejects.toThrow(/changed before the operation/);
await expect(queued).rejects.toThrow("Chrome MCP profile session was replaced");
await expect(explicitClose).resolves.toBe(true);
expect(factoryCalls).toBe(1);
});
it.each(["during admission", "while queued"])(
@ -1369,45 +1366,6 @@ describe("chrome MCP page parsing", () => {
expect(closeMock).toHaveBeenCalledOnce();
});
it("blocks replacement after an aborted pending factory fails exact cleanup", async () => {
let factoryCalls = 0;
const { promise: factoryGate, resolve: releaseFactory } = createDeferred<void>();
const closeMock = vi
.fn()
.mockRejectedValueOnce(new Error("pending cleanup failed"))
.mockResolvedValue(undefined);
setChromeMcpSessionFactoryForTest(async () => {
factoryCalls += 1;
if (factoryCalls === 1) {
await factoryGate;
const session = createFakeSession();
session.client.close = closeMock as typeof session.client.close;
return session;
}
return createFakeSession();
});
const ctrl = new AbortController();
const aborted = listChromeMcpTabs("chrome-live", undefined, { signal: ctrl.signal });
const abortedExpectation = expect(aborted).rejects.toThrow("pending cleanup failed");
await waitForChromeMcpState(() => expect(factoryCalls).toBe(1));
ctrl.abort(new Error("caller cancelled"));
const blockedReplacement = listChromeMcpTabs("chrome-live");
const blockedReplacementExpectation =
expect(blockedReplacement).rejects.toThrow("pending cleanup failed");
await new Promise<void>((resolve) => {
setImmediate(resolve);
});
expect(factoryCalls).toBe(1);
releaseFactory();
await Promise.all([abortedExpectation, blockedReplacementExpectation]);
expect(factoryCalls).toBe(1);
await expect(listChromeMcpTabs("chrome-live")).resolves.toHaveLength(2);
expect(factoryCalls).toBe(2);
expect(closeMock).toHaveBeenCalledTimes(2);
});
it("holds ephemeral probes behind cancelled pending-session cleanup", async () => {
let factoryCalls = 0;
const { promise: factoryGate, resolve: releaseFactory } = createDeferred<void>();
@ -1536,77 +1494,6 @@ describe("chrome MCP page parsing", () => {
expect(closeMock).not.toHaveBeenCalled();
});
it("starts a fresh shared session when a ready-pending session loses its transport", async () => {
let factoryCalls = 0;
let firstSession: ChromeMcpSession | undefined;
const { promise: firstReadyGate, resolve: releaseFirstReady } = createDeferred<void>();
const firstReadyThen = vi.spyOn(firstReadyGate, "then");
const closeMocks: Array<ReturnType<typeof vi.fn>> = [];
const factory: ChromeMcpSessionFactory = async () => {
factoryCalls += 1;
const session = createFakeSession();
const closeMock = vi.fn().mockResolvedValue(undefined);
closeMocks.push(closeMock);
session.client.close = closeMock as typeof session.client.close;
if (factoryCalls === 1) {
firstSession = session;
session.ready = firstReadyGate;
}
return session;
};
setChromeMcpSessionFactoryForTest(factory);
const ctrl = new AbortController();
const firstTabsPromise = listChromeMcpTabs("chrome-live", undefined, {
signal: ctrl.signal,
});
const firstTabsExpectation = expect(firstTabsPromise).rejects.toThrow(/first waiter cancelled/);
await waitForChromeMcpState(() => expect(factoryCalls).toBe(1));
await waitForChromeMcpState(() => expect(firstReadyThen).toHaveBeenCalledTimes(1));
if (!firstSession) {
throw new Error("Expected first Chrome MCP session to be created");
}
(firstSession.transport as { pid: number | null }).pid = null;
const tabsPromise = listChromeMcpTabs("chrome-live");
const siblingTabsPromise = listChromeMcpTabs("chrome-live");
ctrl.abort(new Error("first waiter cancelled"));
releaseFirstReady();
await waitForChromeMcpState(() => expect(factoryCalls).toBe(2));
const [tabs, siblingTabs] = await Promise.all([tabsPromise, siblingTabsPromise]);
expect(tabs).toHaveLength(2);
expect(siblingTabs).toHaveLength(2);
await firstTabsExpectation;
await waitForChromeMcpState(() => expect(closeMocks[0]).toHaveBeenCalledTimes(1));
expect(closeMocks[1]).not.toHaveBeenCalled();
});
it("bounds retries when ready sessions keep losing their transport", async () => {
let factoryCalls = 0;
const closeMocks: Array<ReturnType<typeof vi.fn>> = [];
const factory: ChromeMcpSessionFactory = async () => {
factoryCalls += 1;
const session = createFakeSession();
(session.transport as { pid: number | null }).pid = null;
const closeMock = vi.fn().mockResolvedValue(undefined);
closeMocks.push(closeMock);
session.client.close = closeMock as typeof session.client.close;
return session;
};
setChromeMcpSessionFactoryForTest(factory);
await expect(listChromeMcpTabs("chrome-live")).rejects.toThrow(
/subprocess exited before it became usable/,
);
expect(factoryCalls).toBe(2);
await waitForChromeMcpState(() => expect(closeMocks[0]).toHaveBeenCalled());
await waitForChromeMcpState(() => expect(closeMocks[1]).toHaveBeenCalled());
});
it("does not let ephemeral probes persist canceled pending attaches", async () => {
let factoryCalls = 0;
const { promise: firstReadyGate, resolve: releaseFirstReady } = createDeferred<void>();

4
extensions/browser/src/browser/extension-relay/gateway-relay-route.ts
View file

@ -74,13 +74,13 @@ async function resolveGatewayRelay(resource: string) {
throw new Error("Gateway relay profile is unavailable");
}
const lifecycle = getProfileLifecycle(runtime);
const generation = lifecycle.generation;
const profileSignal = lifecycle.controller.signal;
return {
relay,
profileName,
assertCurrent: () => {
if (
lifecycle.generation !== generation ||
profileSignal.aborted ||
lifecycle.transitionReason ||
lifecycle.terminal ||
getBrowserControlState() !== state ||

5
extensions/browser/src/browser/extension-relay/relay-lifecycle.test.ts
View file

@ -198,7 +198,6 @@ describe("extension relay lifecycle", () => {
const first = withProfileOperationLease({
state,
runtime,
configRevision: getProfileLifecycle(runtime).configRevision,
signal: firstController.signal,
run: async (signal) => await ensureExtensionRelayForProfile(state, profile, signal),
});
@ -208,7 +207,6 @@ describe("extension relay lifecycle", () => {
const sibling = withProfileOperationLease({
state,
runtime,
configRevision: getProfileLifecycle(runtime).configRevision,
run: async (signal) => await ensureExtensionRelayForProfile(state, profile, signal),
});
void sibling.catch(() => {});
@ -243,7 +241,6 @@ describe("extension relay lifecycle", () => {
const owner = withProfileOperationLease({
state,
runtime,
configRevision: getProfileLifecycle(runtime).configRevision,
run: async (signal) => await ensureExtensionRelayForProfile(state, profile, signal),
});
void owner.catch(() => {});
@ -253,7 +250,6 @@ describe("extension relay lifecycle", () => {
const sibling = withProfileOperationLease({
state,
runtime,
configRevision: getProfileLifecycle(runtime).configRevision,
signal: siblingController.signal,
run: async (signal) => await ensureExtensionRelayForProfile(state, profile, signal),
});
@ -292,7 +288,6 @@ describe("extension relay lifecycle", () => {
const pending = withProfileOperationLease({
state,
runtime,
configRevision: getProfileLifecycle(runtime).configRevision,
run: async (signal) => await ensureExtensionRelayForProfile(state, profile, signal),
});
void pending.catch(() => {});

1
extensions/browser/src/browser/extension-relay/relay-lifecycle.ts
View file

@ -142,7 +142,6 @@ async function ensureDesiredRelay(params: {
return await withProfileOperationLease({
state,
runtime,
configRevision: getProfileLifecycle(runtime).configRevision,
ownership: "lifecycle",
run: async (signal) => {
const map = (state.extensionRelays ??= new Map());

4
extensions/browser/src/browser/profiles-service.test.ts
View file

@ -585,7 +585,6 @@ describe("BrowserProfilesService", () => {
const starting = enqueueProfileStart({
state,
runtime,
configRevision: 0,
key: "default",
run: async () => {
entered.resolve();
@ -637,7 +636,6 @@ describe("BrowserProfilesService", () => {
enqueueProfileStart({
state,
runtime,
configRevision: getProfileLifecycle(runtime).configRevision,
key: "default",
run: async () => {},
}),
@ -668,7 +666,6 @@ describe("BrowserProfilesService", () => {
const starting = enqueueProfileStart({
state,
runtime,
configRevision: 0,
key: "default",
run: async (signal) => {
entered.resolve();
@ -704,7 +701,6 @@ describe("BrowserProfilesService", () => {
enqueueProfileStart({
state,
runtime,
configRevision: getProfileLifecycle(runtime).configRevision,
key: "after-drift",
run: async () => {},
}),

1
extensions/browser/src/browser/profiles-service.ts
View file

@ -251,7 +251,6 @@ export function createBrowserProfilesService(ctx: BrowserRouteContext) {
runtime,
reason: "profile deletion requested",
terminal: "deleted",
advanceConfigRevision: true,
closeRelay: resolved.driver === "extension",
managedChrome: "release-profile-data",
afterCleanup: persistDelete,

7
extensions/browser/src/browser/pw-session-actions.ts
View file

@ -42,7 +42,6 @@ import {
cachedByCdpUrl,
connectingByCdpUrl,
pageStates,
retainedClosingByCdpUrl,
type BrowserObservedState,
} from "./pw-session-contracts.js";
import {
@ -127,11 +126,7 @@ export async function closePlaywrightBrowserConnection(opts?: { cdpUrl?: string
return;
}
const cdpUrls = new Set([
...cachedByCdpUrl.keys(),
...connectingByCdpUrl.keys(),
...retainedClosingByCdpUrl.keys(),
]);
const cdpUrls = new Set([...cachedByCdpUrl.keys(), ...connectingByCdpUrl.keys()]);
clearBlockedTargetsForCdpUrl();
clearBlockedPageRefsForCdpUrl();
const results = await Promise.allSettled(

124
extensions/browser/src/browser/pw-session-connection.ts
View file

@ -1,4 +1,3 @@
import { toErrorObject } from "openclaw/plugin-sdk/error-runtime";
import { sleepWithAbort } from "openclaw/plugin-sdk/retry-runtime";
import { formatErrorMessage } from "openclaw/plugin-sdk/security-runtime";
import type { SsrFPolicy } from "openclaw/plugin-sdk/security-runtime";
@ -35,10 +34,8 @@ import {
contextStates,
observedContexts,
PLAYWRIGHT_CONNECTION_CLOSE_TIMEOUT_MS,
retainedClosingByCdpUrl,
type ConnectedBrowser,
type ContextState,
type PendingBrowserConnection,
type PlaywrightConnectionRetirement,
} from "./pw-session-contracts.js";
import {
@ -161,12 +158,6 @@ function takeCachedPlaywrightBrowserConnection(cdpUrl: string): ConnectedBrowser
const normalized = normalizeCdpUrl(cdpUrl);
const cur = cachedByCdpUrl.get(normalized);
cachedByCdpUrl.delete(normalized);
const pending = connectingByCdpUrl.get(normalized);
if (pending) {
// Invalidation must also retire an in-flight connect. Otherwise it can
// resolve after cleanup and repopulate the cache with the stale pipe.
pending.attempt.cancelled = true;
}
connectingByCdpUrl.delete(normalized);
if (!cur) {
return null;
@ -185,35 +176,12 @@ class BlockedBrowserTargetError extends Error {
}
}
function retainClosingPlaywrightConnection(connection: ConnectedBrowser): void {
const retained = retainedClosingByCdpUrl.get(connection.cdpUrl) ?? new Set<ConnectedBrowser>();
retained.add(connection);
retainedClosingByCdpUrl.set(connection.cdpUrl, retained);
}
function releaseClosingPlaywrightConnection(connection: ConnectedBrowser): void {
const retained = retainedClosingByCdpUrl.get(connection.cdpUrl);
retained?.delete(connection);
if (retained?.size === 0) {
retainedClosingByCdpUrl.delete(connection.cdpUrl);
}
}
async function closeTrackedPlaywrightConnection(connection: ConnectedBrowser): Promise<void> {
const existing = closeConnectionPromises.get(connection);
if (existing) {
return await existing;
}
retainClosingPlaywrightConnection(connection);
const closing = (async () => {
try {
await connection.browser.close();
releaseClosingPlaywrightConnection(connection);
} catch (error) {
closeConnectionPromises.delete(connection);
throw error;
}
})();
const closing = connection.browser.close();
closeConnectionPromises.set(connection, closing);
return await closing;
}
@ -236,45 +204,31 @@ export function retirePlaywrightBrowserConnectionExact(opts: {
const normalized = normalizeCdpUrl(opts.cdpUrl);
clearBlockedTargetsForCdpUrl(normalized);
clearBlockedPageRefsForCdpUrl(normalized);
const connections = new Map<ConnectedBrowser, Promise<void> | undefined>();
const pendingCollections = new Set<Promise<void>>();
const connections = new Map<ConnectedBrowser, Promise<void>>();
const closing: Promise<void>[] = [];
let retired = false;
const captureConnection = (connection: ConnectedBrowser) => {
const existing = connections.get(connection);
if (existing) {
return existing;
let task = connections.get(connection);
if (!task) {
task = closeTrackedPlaywrightConnection(connection);
connections.set(connection, task);
closing.push(task);
void task.catch(() => {});
}
const closing = closeTrackedPlaywrightConnection(connection);
connections.set(connection, closing);
void closing.catch(() => {});
return closing;
return task;
};
const capture = () => {
const pending = connectingByCdpUrl.get(normalized);
const cached = takeCachedPlaywrightBrowserConnection(normalized);
for (const connection of retainedClosingByCdpUrl.get(normalized) ?? []) {
void captureConnection(connection);
}
if (cached) {
void captureConnection(cached);
}
if (pending) {
const collection = pending.promise.then(
(connection) => {
void captureConnection(connection);
},
() => {
if (pending.attempt.retired) {
void captureConnection(pending.attempt.retired);
}
},
);
pendingCollections.add(collection);
void collection.then(() => {
pendingCollections.delete(collection);
});
const task = pending.then(captureConnection, () => {});
closing.push(task);
void task.catch(() => {});
}
const captured = Boolean(pending || connections.size > 0);
const captured = Boolean(pending || cached);
retired ||= captured;
return captured;
};
@ -285,31 +239,14 @@ export function retirePlaywrightBrowserConnectionExact(opts: {
},
refresh: capture,
close: async () => {
// Failed disconnects are visible to the caller; a process restart is the recovery path.
await withPlaywrightCloseTimeout(
(async () => {
for (const connection of connections.keys()) {
void captureConnection(connection);
Promise.allSettled(closing).then((results) => {
const failed = results.find((result) => result.status === "rejected");
if (failed?.status === "rejected") {
throw failed.reason;
}
await Promise.all(pendingCollections);
const attempts = [...connections.keys()].map(
(connection) => [connection, captureConnection(connection)] as const,
);
const results = await Promise.allSettled(attempts.map(([, closing]) => closing));
let firstError: Error | undefined;
for (const [index, result] of results.entries()) {
if (result.status !== "rejected") {
continue;
}
const [connection, closing] = attempts[index]!;
if (connections.get(connection) === closing) {
connections.set(connection, undefined);
}
firstError ??= toErrorObject(result.reason, "Playwright adapter disconnect failed.");
}
if (firstError) {
throw firstError;
}
})(),
}),
);
},
};
@ -423,16 +360,12 @@ export async function connectBrowser(
}
const connecting = connectingByCdpUrl.get(normalized);
if (connecting) {
return await connecting.promise;
return await connecting;
}
const connectionAttempt: PendingBrowserConnection["attempt"] = { cancelled: false };
const connectWithRetry = async (): Promise<ConnectedBrowser> => {
let lastErr: unknown;
for (let attempt = 0; attempt < 3; attempt += 1) {
if (connectionAttempt.cancelled) {
break;
}
try {
const timeout = 5000 + attempt * 2000;
let endpointDiscoveryError: unknown;
@ -502,11 +435,6 @@ export async function connectBrowser(
}
browser = await connectEndpoint(normalized, configuredPin?.lookup);
}
if (connectionAttempt.cancelled) {
connectionAttempt.retired = { browser, cdpUrl: normalized };
void closeTrackedPlaywrightConnection(connectionAttempt.retired).catch(() => {});
throw new Error("Playwright connection attempt was superseded.");
}
const onDisconnected = () => {
const current = cachedByCdpUrl.get(normalized);
if (current?.browser === browser) {
@ -517,13 +445,15 @@ export async function connectBrowser(
markConnectionScopedBrowser(browser);
}
const connected: ConnectedBrowser = { browser, cdpUrl: normalized, onDisconnected, engine };
cachedByCdpUrl.set(normalized, connected);
if (connectingByCdpUrl.get(normalized) === pending) {
cachedByCdpUrl.set(normalized, connected);
}
browser.on("disconnected", onDisconnected);
observeBrowser(browser);
return connected;
} catch (err) {
lastErr = err;
if (connectionAttempt.cancelled || relay) {
if (relay) {
break;
}
// Don't retry rate-limit errors; retrying worsens the 429.
@ -541,11 +471,11 @@ export async function connectBrowser(
};
const pending = connectWithRetry().finally(() => {
if (connectingByCdpUrl.get(normalized)?.attempt === connectionAttempt) {
if (connectingByCdpUrl.get(normalized) === pending) {
connectingByCdpUrl.delete(normalized);
}
});
connectingByCdpUrl.set(normalized, { attempt: connectionAttempt, promise: pending });
connectingByCdpUrl.set(normalized, pending);
return await pending;
}

17
extensions/browser/src/browser/pw-session-contracts.ts
View file

@ -128,19 +128,14 @@ export type PageState = {
roleRefsFrame?: Frame;
/** Target-cache entry owned by the current role refs. */
roleRefsTargetKey?: string;
/** Cache generation restored or stored by this Page. */
roleRefsTargetGeneration?: number;
/** Main-frame navigation observed before this Page could be bound to its target. */
roleRefsInvalidBeforeGeneration?: number;
/** Any frame changed before target binding; invalidates only page-wide aria refs. */
roleRefsAriaInvalidBeforeGeneration?: number;
/** Navigation observed before target discovery invalidates the eventual cache entry. */
roleRefsInvalidated?: boolean;
};
export type RoleRefs = NonNullable<PageState["roleRefs"]>;
export type RoleRefsCacheEntry = {
refs: RoleRefs;
mode?: NonNullable<PageState["roleRefsMode"]>;
generation: number;
};
export type ContextState = {
@ -157,11 +152,6 @@ export const MAX_NETWORK_REQUESTS = 500;
export const MAX_RECENT_DIALOGS = 20;
export const OBSERVED_DIALOG_TIMEOUT_MS = 120_000;
export type PendingBrowserConnection = {
attempt: { cancelled: boolean; retired?: ConnectedBrowser };
promise: Promise<ConnectedBrowser>;
};
export type PlaywrightConnectionRetirement = {
readonly retired: boolean;
/** Capture handles created by already-admitted work before cleanup begins. */
@ -170,8 +160,7 @@ export type PlaywrightConnectionRetirement = {
};
export const cachedByCdpUrl = new Map<string, ConnectedBrowser>();
export const connectingByCdpUrl = new Map<string, PendingBrowserConnection>();
export const retainedClosingByCdpUrl = new Map<string, Set<ConnectedBrowser>>();
export const connectingByCdpUrl = new Map<string, Promise<ConnectedBrowser>>();
export const closeConnectionPromises = new WeakMap<ConnectedBrowser, Promise<void>>();
export const PLAYWRIGHT_CONNECTION_CLOSE_TIMEOUT_MS = 2_000;
export const blockedTargetsByCdpUrl = new Set<string>();

45
extensions/browser/src/browser/pw-session-state.ts
View file

@ -32,7 +32,6 @@ import {
// for the same CDP target across requests.
const roleRefsByTarget = new Map<string, RoleRefsCacheEntry>();
const MAX_ROLE_REFS_CACHE = 50;
let roleRefsCacheGeneration = 0;
const MAX_OBSERVED_PAGE_TEXT_CHARS = 2_048;
function truncateObservedPageText(value: string): string {
@ -54,24 +53,11 @@ export function bindRoleRefsTarget(page: Page, cdpUrl: string, targetId?: string
}
const state = ensurePageState(page);
const key = targetKey(cdpUrl, normalizedTargetId);
const invalidBeforeGeneration = state.roleRefsInvalidBeforeGeneration;
const ariaInvalidBeforeGeneration = state.roleRefsAriaInvalidBeforeGeneration;
const cached = roleRefsByTarget.get(key);
if (
cached &&
((invalidBeforeGeneration !== undefined && cached.generation <= invalidBeforeGeneration) ||
(ariaInvalidBeforeGeneration !== undefined &&
cached.mode === "aria" &&
cached.generation <= ariaInvalidBeforeGeneration))
) {
if (state.roleRefsInvalidated) {
roleRefsByTarget.delete(key);
}
state.roleRefsInvalidBeforeGeneration = undefined;
state.roleRefsAriaInvalidBeforeGeneration = undefined;
state.roleRefsInvalidated = false;
state.roleRefsTargetKey = key;
if (!state.roleRefs) {
state.roleRefsTargetGeneration = roleRefsByTarget.get(key)?.generation;
}
}
/** Store role refs on the page and target cache. */
@ -94,7 +80,6 @@ export function storeRoleRefsForTarget(opts: {
const targetId = normalizeOptionalString(opts.targetId);
if (!targetId) {
state.roleRefsTargetKey = undefined;
state.roleRefsTargetGeneration = undefined;
return;
}
bindRoleRefsTarget(opts.page, opts.cdpUrl, targetId);
@ -103,29 +88,21 @@ export function storeRoleRefsForTarget(opts: {
// Frame-scoped refs remain local and require a fresh snapshot after reconnect.
if (opts.frameSelector) {
roleRefsByTarget.delete(key);
state.roleRefsTargetGeneration = undefined;
return;
}
const generation = ++roleRefsCacheGeneration;
roleRefsByTarget.set(key, { refs: opts.refs, mode: opts.mode, generation });
roleRefsByTarget.set(key, { refs: opts.refs, mode: opts.mode });
pruneMapToMaxSize(roleRefsByTarget, MAX_ROLE_REFS_CACHE);
state.roleRefsTargetGeneration = generation;
}
function clearRoleRefs(state: PageState): void {
if (state.roleRefsTargetKey) {
const cached = roleRefsByTarget.get(state.roleRefsTargetKey);
// A delayed event from an obsolete Page must not erase refs that a newer
// wrapper stored for the same target after this Page's generation.
if (cached?.generation === state.roleRefsTargetGeneration) {
roleRefsByTarget.delete(state.roleRefsTargetKey);
}
// A delayed event from an older wrapper may evict newer refs; a snapshot repairs it.
roleRefsByTarget.delete(state.roleRefsTargetKey);
}
state.roleRefs = undefined;
state.roleRefsMode = undefined;
state.roleRefsFrame = undefined;
state.roleRefsTargetKey = undefined;
state.roleRefsTargetGeneration = undefined;
}
function currentTargetRoleRefsMode(
@ -134,8 +111,7 @@ function currentTargetRoleRefsMode(
if (!state.roleRefsTargetKey) {
return undefined;
}
const cached = roleRefsByTarget.get(state.roleRefsTargetKey);
return cached && cached.generation === state.roleRefsTargetGeneration ? cached.mode : undefined;
return roleRefsByTarget.get(state.roleRefsTargetKey)?.mode;
}
/** Restore cached role refs onto a newly resolved page. */
@ -159,7 +135,6 @@ export function restoreRoleRefsForTarget(opts: {
return;
}
state.roleRefsTargetKey = cacheKey;
state.roleRefsTargetGeneration = cached.generation;
state.roleRefs = cached.refs;
state.roleRefsMode = cached.mode;
}
@ -276,13 +251,7 @@ export function ensurePageState(page: Page): PageState {
const invalidateFrameRefs = (frame: Frame, navigated: boolean) => {
const isMainFrame = frame === page.mainFrame();
if (!state.roleRefsTargetKey) {
// Discovery can bind a replacement Page after a navigation. Keep its
// generation boundary so an older Page cannot invalidate newer refs.
if (isMainFrame) {
state.roleRefsInvalidBeforeGeneration = roleRefsCacheGeneration;
} else {
state.roleRefsAriaInvalidBeforeGeneration = roleRefsCacheGeneration;
}
state.roleRefsInvalidated = true;
}
const pageWideAriaRefs =
state.roleRefsMode === "aria" || currentTargetRoleRefsMode(state) === "aria";

16
extensions/browser/src/browser/pw-session.connections.test.ts
View file

@ -374,25 +374,11 @@ describe("pw-session connection scoping", () => {
expect(closeSettled).toBe(false);
pendingBrowser.resolve(browser.browser);
await expect(listing).rejects.toThrow("superseded");
await listing.catch(() => {});
await expect(closing).resolves.toBeUndefined();
expect(browser.browserClose).toHaveBeenCalledOnce();
});
it("retains a scoped connection until a failed disconnect succeeds on retry", async () => {
const browser = makeBrowser("A", "https://a.example");
browser.browserClose
.mockRejectedValueOnce(new Error("disconnect failed"))
.mockResolvedValue(undefined);
connectOverCdpSpy.mockResolvedValue(browser.browser);
await listPagesViaPlaywright({ cdpUrl });
await expect(closePlaywrightBrowserConnection({ cdpUrl })).rejects.toThrow("disconnect failed");
await expect(closePlaywrightBrowserConnection({ cdpUrl })).resolves.toBeUndefined();
expect(browser.browserClose).toHaveBeenCalledTimes(2);
});
it("awaits only the retired adapter after a same-URL successor connects", async () => {
const first = makeBrowser("A", "https://a.example");
const closeGate = createDeferred<void>();

30
extensions/browser/src/browser/pw-session.test.ts
View file

@ -179,36 +179,6 @@ describe("pw-session role refs cache", () => {
expect(ensurePageState(pageB).roleRefsMode).toBe("aria");
});
it("does not let an obsolete Page invalidate a newer cache generation", () => {
const cdpUrl = "http://127.0.0.1:9222";
const targetId = "shared-target";
const { page: oldPage, handlers, mainFrame } = fakePage();
storeRoleRefsForTarget({
page: oldPage,
cdpUrl,
targetId,
refs: { e1: { role: "button", name: "Old document" } },
mode: "role",
});
const { page: currentPage } = fakePage();
storeRoleRefsForTarget({
page: currentPage,
cdpUrl,
targetId,
refs: { e1: { role: "heading", name: "Current document" } },
mode: "aria",
});
handlers.get("framenavigated")?.[0]?.(mainFrame);
const { page: replacementPage } = fakePage();
restoreRoleRefsForTarget({ cdpUrl, targetId, page: replacementPage });
expect(ensurePageState(replacementPage).roleRefs).toEqual({
e1: { role: "heading", name: "Current document" },
});
expect(ensurePageState(replacementPage).roleRefsMode).toBe("aria");
});
it("invalidates page-wide aria refs on subframe navigation", () => {
const event = "framenavigated";
const cdpUrl = "http://127.0.0.1:9222";

26
extensions/browser/src/browser/resolved-config-refresh.ts
View file

@ -47,19 +47,13 @@ function queueRemovedProfileCleanup(params: {
current: BrowserServerState;
name: string;
runtime: ProfileRuntimeState;
initial: boolean;
}) {
const actor = getProfileLifecycle(params.runtime);
if (!params.initial && (!actor.blockedReason || actor.transitionReason)) {
return;
}
params.runtime.lastTargetId = null;
void beginProfileTransition({
state: params.current,
runtime: params.runtime,
reason: params.initial ? "profile removed from config" : "profile removal cleanup retry",
reason: "profile removed from config",
terminal: "config-removed",
advanceConfigRevision: params.initial,
closeRelay: params.runtime.profile.driver === "extension",
exposeReason: true,
})
@ -109,25 +103,11 @@ function applyResolvedConfig(
};
for (const [name, runtime] of current.profiles) {
const actor = getProfileLifecycle(runtime);
if (actor.terminal === "config-removed") {
queueRemovedProfileCleanup({ current, name, runtime, initial: false });
continue;
}
if (actor.terminal) {
continue;
}
const nextProfile = resolveProfile(current.resolved, name);
if (nextProfile) {
if (actor.blockedReason && !actor.transitionReason) {
void beginProfileTransition({
state: current,
runtime,
reason: "profile invariant cleanup retry",
captureProfileResources: false,
exposeReason: true,
}).catch(() => {});
continue;
}
const changed = changedProfileInvariants(
runtime.profile,
nextProfile,
@ -141,7 +121,6 @@ function applyResolvedConfig(
state: current,
runtime,
reason,
advanceConfigRevision: true,
closeRelay: previousProfile.driver === "extension",
exposeReason: true,
}).catch(() => {});
@ -150,7 +129,8 @@ function applyResolvedConfig(
runtime.profile = nextProfile;
continue;
}
queueRemovedProfileCleanup({ current, name, runtime, initial: true });
// Failed cleanup stays visible; an explicit stop or restart repairs it.
queueRemovedProfileCleanup({ current, name, runtime });
}
}

2
extensions/browser/src/browser/routes/agent.screencast.test.ts
View file

@ -177,7 +177,7 @@ describe("browser screencast mint route", () => {
await expect(token.checkNavigationAllowed("http://127.0.0.1/")).resolves.toBeUndefined();
state.resolved.ssrfPolicy = { allowPrivateNetwork: false };
await expect(token.checkNavigationAllowed("http://127.0.0.1/")).rejects.toThrow();
getProfileLifecycle(runtime).generation += 1;
getProfileLifecycle(runtime).controller.abort();
expect(() => token.assertCurrent()).toThrow("superseded");
});

9
extensions/browser/src/browser/routes/agent.screencast.ts
View file

@ -5,7 +5,7 @@ import { getBrowserProfileCapabilities } from "../profile-capabilities.js";
import { getPwAiModule } from "../pw-ai-module.js";
import { mintBrowserScreencastToken } from "../screencast/tokens.js";
import type { BrowserRouteContext } from "../server-context.js";
import { getProfileLifecycle, isProfileGenerationCurrent } from "../server-context.lifecycle.js";
import { getProfileLifecycle, isProfileOperationCurrent } from "../server-context.lifecycle.js";
import {
browserNavigationPolicyForProfile,
readBody,
@ -68,14 +68,13 @@ export function registerBrowserAgentScreencastRoutes(
throw new BrowserProfileUnavailableError("Browser profile is no longer available.");
}
const lifecycle = getProfileLifecycle(runtime);
const generation = lifecycle.generation;
const configRevision = lifecycle.configRevision;
const profileSignal = lifecycle.controller.signal;
const assertResourceCurrent = req.screencastAuthority?.assertCurrent;
const assertCurrent = () => {
assertResourceCurrent?.();
if (
state.profiles.get(profileName) !== runtime ||
!isProfileGenerationCurrent({ state, runtime, generation, configRevision })
!isProfileOperationCurrent({ state, runtime, signal: profileSignal })
) {
throw new BrowserProfileUnavailableError("Browser screencast target was superseded.");
}
@ -96,7 +95,7 @@ export function registerBrowserAgentScreencastRoutes(
maxWidth: clampScreencastOption(body.maxWidth, 320, 2000, 1280),
maxHeight: clampScreencastOption(body.maxHeight, 320, 2000, 1280),
quality: clampScreencastOption(body.quality, 30, 90, 70),
lifecycleGeneration: generation,
profileSignal,
lifecycleSignal: req.screencastAuthority
? AbortSignal.any([lifecycle.controller.signal, req.screencastAuthority.signal])
: lifecycle.controller.signal,

4
extensions/browser/src/browser/routes/agent.snapshot-target.ts
View file

@ -26,14 +26,14 @@ export async function captureBrowserOperationTarget(opts: {
throw new Error("Browser operation profile is unavailable");
}
const lifecycle = getProfileLifecycle(runtime);
const generation = lifecycle.generation;
const profileSignal = lifecycle.controller.signal;
let released = false;
const isCurrent = () =>
!released &&
opts.ctx.state() === state &&
state.extensionRelays?.get(opts.profileName) === relay &&
state.profiles.get(opts.profileName) === runtime &&
lifecycle.generation === generation;
!profileSignal.aborted;
const reference =
relay.ownership === "borrowed"
? await relay.client.capture(opts.targetId, () => {

2
extensions/browser/src/browser/runtime-lifecycle.shutdown-race.test.ts
View file

@ -96,7 +96,6 @@ describe("browser runtime shutdown profile races", () => {
enqueueProfileStart({
state,
runtime,
configRevision: 0,
key: "default",
run: async (signal) => {
startSignals[index] = signal;
@ -129,7 +128,6 @@ describe("browser runtime shutdown profile races", () => {
enqueueProfileStart({
state,
runtime: runtimes[0]!,
configRevision: 0,
key: "after-shutdown",
run: async () => {},
}),

4
extensions/browser/src/browser/screencast/session.test.ts
View file

@ -438,7 +438,9 @@ describe("browser screencast sessions", () => {
await flush();
expect(pending.close).toHaveBeenCalledWith(4004, "target_closed");
expect(page.newCDPSession).not.toHaveBeenCalled();
const successor = await attach(screencastParams({ lifecycleGeneration: 1 }));
const successor = await attach(
screencastParams({ profileSignal: new AbortController().signal }),
);
const stale = await attach(params);
expect(stale.close).toHaveBeenCalledWith(4004, "target_closed");
expect(successor.close).not.toHaveBeenCalled();

5
extensions/browser/src/browser/screencast/session.ts
View file

@ -341,10 +341,7 @@ export function attachBrowserScreencastViewer(
};
let session = sessions.get(key);
let previousDrain: Promise<void> | undefined;
if (
session &&
(session.closed || session.params.lifecycleGeneration !== params.lifecycleGeneration)
) {
if (session && (session.closed || session.params.profileSignal !== params.profileSignal)) {
previousDrain = session.close(4004, "target_closed");
session = undefined;
}

4
extensions/browser/src/browser/screencast/test-support.ts
View file

@ -2,6 +2,8 @@ import { EventEmitter } from "node:events";
import { vi } from "vitest";
import type { BrowserScreencastTokenParams } from "./tokens.js";
const profileSignal = new AbortController().signal;
export function screencastParams(
overrides: Partial<BrowserScreencastTokenParams> = {},
): BrowserScreencastTokenParams {
@ -12,7 +14,7 @@ export function screencastParams(
maxWidth: 1280,
maxHeight: 1280,
quality: 70,
lifecycleGeneration: 0,
profileSignal,
lifecycleSignal: new AbortController().signal,
assertCurrent: () => {},
checkNavigationAllowed: async () => {},

2
extensions/browser/src/browser/screencast/tokens.ts
View file

@ -9,7 +9,7 @@ export type BrowserScreencastTokenParams = {
maxWidth: number;
maxHeight: number;
quality: number;
lifecycleGeneration: number;
profileSignal: AbortSignal;
lifecycleSignal: AbortSignal;
requesterSignal?: AbortSignal;
isRequesterCurrent?: () => boolean;

26
extensions/browser/src/browser/server-context.availability.ts
View file

@ -42,7 +42,7 @@ import {
beginProfileTransition,
enqueueProfileStart,
getProfileLifecycle,
isProfileGenerationCurrent,
isProfileOperationCurrent,
isProfileRestartRequiredError,
isWithinProfileOperationLease,
ProfileRestartRequiredError,
@ -63,7 +63,6 @@ type AvailabilityDeps = {
profile: ResolvedBrowserProfile;
state: () => BrowserServerState;
runtime: ProfileRuntimeState;
configRevision: number;
};
type AvailabilityOps = Pick<
@ -136,7 +135,6 @@ export function createProfileAvailability({
profile,
state,
runtime,
configRevision,
}: AvailabilityDeps): AvailabilityOps {
const actor = getProfileLifecycle(runtime);
const redactedProfileCdpUrl = redactCdpUrl(profile.cdpUrl) ?? profile.cdpUrl;
@ -297,13 +295,12 @@ export function createProfileAvailability({
}
};
const adoptRunning = (running: RunningChrome, generation: number, signal: AbortSignal): void => {
const adoptRunning = (running: RunningChrome, signal: AbortSignal): void => {
if (
!isProfileGenerationCurrent({
!isProfileOperationCurrent({
state: state(),
runtime,
configRevision,
generation,
signal,
})
) {
signal.throwIfAborted();
@ -446,7 +443,6 @@ export function createProfileAvailability({
const ensureBrowserAvailableOnce = async (
signal: AbortSignal,
generation: number,
options?: BrowserEnsureOptions,
): Promise<void> => {
signal.throwIfAborted();
@ -538,7 +534,7 @@ export function createProfileAvailability({
}
try {
await waitForCdpReadyAfterLaunch(signal, launched);
adoptRunning(launched, generation, signal);
adoptRunning(launched, signal);
runtime.managedLaunchFailure = undefined;
} catch (err) {
await stopExactRunning(launched);
@ -596,9 +592,8 @@ export function createProfileAvailability({
await withProfileOperationLease({
state: state(),
runtime,
configRevision,
signal: options?.signal,
run: async (signal) => await ensureBrowserAvailableOnce(signal, actor.generation, options),
run: async (signal) => await ensureBrowserAvailableOnce(signal, options),
});
return;
}
@ -609,11 +604,10 @@ export function createProfileAvailability({
await enqueueProfileStart({
state: state(),
runtime,
configRevision,
key,
signal: options?.signal,
run: async (signal, generation) => {
await ensureBrowserAvailableOnce(signal, generation, options);
run: async (signal) => {
await ensureBrowserAvailableOnce(signal, options);
},
});
return;
@ -622,7 +616,7 @@ export function createProfileAvailability({
throw err;
}
// A route may already hold an ordinary lease. Let its wrapper release
// and retry so a restart never waits on its own generation lease.
// and retry so a restart never waits on its own settlement lease.
if (isWithinProfileOperationLease(runtime)) {
throw err;
}
@ -637,7 +631,7 @@ export function createProfileAvailability({
const stopRunningBrowser = async (): Promise<{ stopped: boolean }> => {
const current = state();
assertProfileLifecycleContext({ state: current, runtime, configRevision });
assertProfileLifecycleContext({ state: current, runtime });
runtime.managedLaunchFailure = undefined;
const result = await beginProfileTransition({
state: current,

1
extensions/browser/src/browser/server-context.ensure-browser-available.waits-for-cdp-ready.test.ts
View file

@ -621,7 +621,6 @@ function createAvailability() {
profile,
state: () => state,
runtime,
configRevision: 0,
});
vi.mocked(chromeModule.isChromeCdpReady).mockImplementation(async (...args) => {
await args[4]?.onDiagnostic?.({

66
extensions/browser/src/browser/server-context.hot-reload-profiles.test.ts
View file

@ -135,12 +135,11 @@ function updateProfile(state: BrowserServerState, name: string, config: TestProf
function enqueueCurrentProfileStart(
state: BrowserServerState,
runtime: ProfileRuntimeState,
run: (signal: AbortSignal, generation: number) => Promise<void>,
run: (signal: AbortSignal) => Promise<void>,
) {
return enqueueProfileStart({
state,
runtime,
configRevision: getProfileLifecycle(runtime).configRevision,
key: "default",
run,
});
@ -180,7 +179,6 @@ describe("server-context hot-reload profiles", () => {
chrome: relay.internalToken,
});
expect(state.resolved.extensionRelayToken).toBe(relay.token);
expect(getProfileLifecycle(runtime).configRevision).toBe(0);
expect(runtime.lastTargetId).toBe("shared-tab");
}
expect(relay.close).not.toHaveBeenCalled();
@ -295,7 +293,6 @@ describe("server-context hot-reload profiles", () => {
expect(getProfileLifecycle(runtime).transitionReason).toContain("mcpCommand");
expect(getProfileLifecycle(runtime).transitionReason).toContain("mcpArgs");
expect(getProfileLifecycle(runtime).configRevision).toBe(1);
});
it("rapid A to B to C closes both stale endpoints and adopts only C", async () => {
@ -323,7 +320,7 @@ describe("server-context hot-reload profiles", () => {
const workC = expectDefined(resolveProfile(state.resolved, "work"), "work C missing");
expect(runtime.profile.cdpUrl).toBe(workC.cdpUrl);
await expect(pendingB).rejects.toThrow(/profile config changed|superseded/i);
await expect(pendingB).rejects.toThrow(/profile invariants changed/i);
await getProfileLifecycle(runtime).tail;
await expect(
enqueueCurrentProfileStart(state, runtime, async () => {
@ -385,63 +382,4 @@ describe("server-context hot-reload profiles", () => {
cdpUrl: profile.cdpUrl,
});
});
it("retries a failed removal tombstone before admitting a same-name re-add", async () => {
const { state, runtime: oldRuntime } = createProfileFixture({
name: "work",
config: { cdpPort: 18801, color: "#0066CC" },
});
lifecycleMocks.closePlaywrightBrowserConnection
.mockRejectedValueOnce(new Error("close failed"))
.mockResolvedValue(undefined);
lifecycleMocks.retirePlaywrightBrowserConnection
.mockReturnValueOnce(true)
.mockReturnValue(false);
delete mockState.cfgProfiles.work;
refreshProfiles(state);
await getProfileLifecycle(oldRuntime).tail;
expect(getProfileLifecycle(oldRuntime).blockedReason).toContain("cleanup failed");
expect(state.profiles.get("work")).toBe(oldRuntime);
updateProfile(state, "work", { cdpPort: 18802, color: "#00AA00" });
await getProfileLifecycle(oldRuntime).tail;
await Promise.resolve();
expect(state.profiles.has("work")).toBe(false);
expect(lifecycleMocks.closePlaywrightBrowserConnection).toHaveBeenCalledTimes(2);
});
it("retries failed invariant cleanup before admitting the updated profile", async () => {
const { state, profile, runtime } = createProfileFixture({
name: "work",
config: { cdpPort: 18801, color: "#0066CC" },
});
lifecycleMocks.closePlaywrightBrowserConnection
.mockRejectedValueOnce(new Error("close failed"))
.mockResolvedValue(undefined);
updateProfile(state, "work", { cdpPort: 18802, color: "#00AA00" });
await getProfileLifecycle(runtime).tail;
expect(getProfileLifecycle(runtime).blockedReason).toContain("cleanup failed");
refreshProfiles(state);
await getProfileLifecycle(runtime).tail;
expect(getProfileLifecycle(runtime).blockedReason).toBeNull();
expect(runtime.profile.cdpPort).toBe(18802);
expect(lifecycleMocks.retirePlaywrightBrowserConnection).toHaveBeenCalledWith({
cdpUrl: profile.cdpUrl,
});
expect(lifecycleMocks.closePlaywrightBrowserConnection).toHaveBeenCalledTimes(2);
expect(lifecycleMocks.closePlaywrightBrowserConnection).toHaveBeenNthCalledWith(1, {
cdpUrl: profile.cdpUrl,
});
expect(lifecycleMocks.closePlaywrightBrowserConnection).toHaveBeenNthCalledWith(2, {
cdpUrl: profile.cdpUrl,
});
await expect(
enqueueCurrentProfileStart(state, runtime, async () => {}),
).resolves.toBeUndefined();
});
});

98
extensions/browser/src/browser/server-context.lifecycle.ts
View file

@ -2,8 +2,7 @@
* Per-profile Browser lifecycle actor.
*
* Starts and destructive transitions share one settled serial tail. Ordinary
* tab/action work uses generation leases, so it remains concurrent while a
* transition can still abort and drain all previously admitted work.
* Tab/action work stays concurrent; transitions cancel it and wait for settlement.
*/
import { AsyncLocalStorage } from "node:async_hooks";
import { createDeferred } from "openclaw/plugin-sdk/extension-shared";
@ -21,8 +20,6 @@ import type { BrowserServerState, ProfileRuntimeState } from "./server-context.t
type ProfileLifecycleTerminal = "deleted" | "config-removed";
type ProfileLifecycleActor = {
generation: number;
configRevision: number;
controller: AbortController;
/** Settled-only tail: failed starts/transitions never poison later work. */
tail: Promise<void>;
@ -42,7 +39,6 @@ type ProfileTransitionOptions = {
runtime: ProfileRuntimeState;
reason: string;
terminal?: ProfileLifecycleTerminal;
advanceConfigRevision?: boolean;
closeRelay?: boolean;
captureProfileResources?: boolean;
/** Bridge runtimes must not retire process-global adapters shared by another runtime. */
@ -58,7 +54,6 @@ type ProfileTransitionResult = {
};
type ProfileLeaseContext = {
generation: number;
signal: AbortSignal;
};
@ -68,8 +63,6 @@ const stoppingBrowserRuntimes = new WeakSet<BrowserServerState>();
function createProfileLifecycleActor(): ProfileLifecycleActor {
return {
generation: 0,
configRevision: 0,
controller: new AbortController(),
tail: Promise.resolve(),
starts: new Map(),
@ -137,8 +130,6 @@ function assertRuntimeAdmission(state: BrowserServerState): void {
function assertProfileCurrent(params: {
state: BrowserServerState;
runtime: ProfileRuntimeState;
configRevision: number;
generation?: number;
allowBlocked?: boolean;
}): void {
assertRuntimeAdmission(params.state);
@ -149,19 +140,12 @@ function assertProfileCurrent(params: {
if (actor.blockedReason && !params.allowBlocked) {
throw lifecycleError(params.runtime.profile.name, actor.blockedReason);
}
if (actor.configRevision !== params.configRevision) {
throw lifecycleError(params.runtime.profile.name, "profile config changed");
}
if (params.generation != null && actor.generation !== params.generation) {
throw lifecycleError(params.runtime.profile.name, "operation superseded");
}
}
/** Allow a lifecycle retry to repair a failed cleanup while fencing stale config. */
/** Allow explicit lifecycle operations to repair a failed cleanup. */
export function assertProfileLifecycleContext(params: {
state: BrowserServerState;
runtime: ProfileRuntimeState;
configRevision: number;
}): void {
assertProfileCurrent({ ...params, allowBlocked: true });
}
@ -236,25 +220,23 @@ export function releaseProfileHandle(runtime: ProfileRuntimeState, running: Runn
}
}
/** True only while a captured start still owns the current profile generation. */
export function isProfileGenerationCurrent(params: {
/** Check the captured lifecycle signal at resource effect boundaries. */
export function isProfileOperationCurrent(params: {
state: BrowserServerState;
runtime: ProfileRuntimeState;
configRevision: number;
generation: number;
signal: AbortSignal;
}): boolean {
const actor = getProfileLifecycle(params.runtime);
return (
isBrowserRuntimeRunning(params.state) &&
!actor.terminal &&
!actor.blockedReason &&
actor.configRevision === params.configRevision &&
actor.generation === params.generation
!params.signal.aborted
);
}
/**
* Run ordinary profile work under a concurrent generation lease.
* Run ordinary profile work under a concurrent settlement lease.
*
* Passing the current lifecycle signal denotes nested work already covered by
* an outer lease; this avoids self-deadlock while preserving cancellation.
@ -262,7 +244,6 @@ export function isProfileGenerationCurrent(params: {
export async function withProfileOperationLease<T>(params: {
state: BrowserServerState;
runtime: ProfileRuntimeState;
configRevision: number;
signal?: AbortSignal;
/** Shared producers belong to the lifecycle, never to their first caller. */
ownership?: "caller" | "lifecycle";
@ -276,29 +257,17 @@ export async function withProfileOperationLease<T>(params: {
if (parent) {
const signal = combineSignals(parent.signal, params.signal);
signal.throwIfAborted();
assertProfileCurrent({ ...params, generation: parent.generation });
assertProfileCurrent(params);
const result = await params.run(signal);
signal.throwIfAborted();
assertProfileCurrent({ ...params, generation: parent.generation });
assertProfileCurrent(params);
await params.commit?.(result);
return result;
}
const requestedGeneration = actor.generation;
assertProfileCurrent({ ...params, generation: requestedGeneration });
// The settled actor tail is the readiness barrier for new ordinary work.
// Re-read after every await so a synchronously-started transition cannot be
// skipped between observing an old settled tail and lease admission.
for (;;) {
const ready = actor.tail;
await waitForProfileOperation(ready, params.signal);
if (actor.tail === ready) {
break;
}
}
assertProfileCurrent({ ...params, generation: requestedGeneration });
const generation = requestedGeneration;
const lifecycleSignal = actor.controller.signal;
assertProfileCurrent(params);
await waitForProfileOperation(actor.tail, params.signal);
const signal =
params.ownership === "lifecycle"
? lifecycleSignal
@ -307,10 +276,10 @@ export async function withProfileOperationLease<T>(params: {
const release = createLease(actor);
try {
const leases = new Map(inherited);
leases.set(params.runtime, { generation, signal });
leases.set(params.runtime, { signal });
const result = await profileLeaseStorage.run(leases, async () => await params.run(signal));
signal.throwIfAborted();
assertProfileCurrent({ ...params, generation });
assertProfileCurrent(params);
// This assertion is the operation's linearization point. Once admitted,
// an async persistent commit keeps its lease until complete; a later
// reset/delete/stop drains behind it instead of partially cancelling it.
@ -325,10 +294,9 @@ export async function withProfileOperationLease<T>(params: {
export function enqueueProfileStart(params: {
state: BrowserServerState;
runtime: ProfileRuntimeState;
configRevision: number;
key: string;
signal?: AbortSignal;
run: (signal: AbortSignal, generation: number) => Promise<void>;
run: (signal: AbortSignal) => Promise<void>;
}): Promise<void> {
assertProfileCurrent(params);
params.signal?.throwIfAborted();
@ -338,15 +306,14 @@ export function enqueueProfileStart(params: {
return waitForProfileOperation(existing, params.signal);
}
const generation = actor.generation;
const signal = actor.controller.signal;
const promise = actor.tail.then(async () => {
assertProfileCurrent({ ...params, generation });
assertProfileCurrent(params);
signal.throwIfAborted();
const owned = new Map(profileLeaseStorage.getStore());
owned.set(params.runtime, { generation, signal });
await profileLeaseStorage.run(owned, async () => await params.run(signal, generation));
assertProfileCurrent({ ...params, generation });
owned.set(params.runtime, { signal });
await profileLeaseStorage.run(owned, async () => await params.run(signal));
assertProfileCurrent(params);
signal.throwIfAborted();
});
actor.starts.set(params.key, promise);
@ -462,7 +429,7 @@ async function cleanupProfileResources(params: {
}
/**
* Synchronously invalidate the current generation, eagerly begin owned adapter
* Synchronously cancel current work, eagerly begin owned adapter
* teardown, then serialize exact-handle cleanup behind older starts and leases.
*/
export function beginProfileTransition(
@ -480,11 +447,7 @@ export function beginProfileTransition(
const hadPendingWork = actor.starts.size > 0 || actor.leases.size > 0 || actor.handles.size > 0;
const reason = lifecycleError(params.runtime.profile.name, params.reason);
actor.generation += 1;
params.runtime.externalBrowserMode = undefined;
if (params.advanceConfigRevision) {
actor.configRevision += 1;
}
actor.controller.abort(reason);
actor.controller = new AbortController();
actor.starts.clear();
@ -520,7 +483,6 @@ export function beginProfileTransition(
?.then(({ closeChromeMcpSession }) => closeChromeMcpSession(ownerProfile.name))
.catch(() => false) ?? null)
: null;
const transitionGeneration = actor.generation;
let cleanupCompleted = false;
const transition = actor.tail
.then(async () => {
@ -538,28 +500,22 @@ export function beginProfileTransition(
});
cleanupCompleted = true;
await params.afterCleanup?.();
if (actor.generation === transitionGeneration) {
actor.blockedReason = null;
}
actor.blockedReason = null;
return result;
})
.catch((err: unknown) => {
if (actor.generation === transitionGeneration) {
if (cleanupCompleted) {
if (params.rollbackTerminalOnFailure) {
actor.terminal = null;
}
actor.blockedReason = null;
} else {
actor.blockedReason = `${params.reason} cleanup failed`;
if (cleanupCompleted) {
if (params.rollbackTerminalOnFailure) {
actor.terminal = null;
}
actor.blockedReason = null;
} else {
actor.blockedReason = `${params.reason} cleanup failed`;
}
throw err;
});
const settleTransition = () => {
if (actor.generation === transitionGeneration) {
actor.transitionReason = null;
}
actor.transitionReason = null;
};
actor.tail = transition.then(settleTransition, settleTransition);
return transition;

6
extensions/browser/src/browser/server-context.ts
View file

@ -92,8 +92,6 @@ function createProfileContext(
return runtimeState;
};
const configRevision = getProfileLifecycle(profileState).configRevision;
const rawTabOps = createProfileTabOps({
profile,
state,
@ -105,7 +103,6 @@ function createProfileContext(
profile,
state,
runtime: profileState,
configRevision,
});
const rawSelection = createProfileSelectionOps({
@ -125,7 +122,6 @@ function createProfileContext(
await withProfileOperationLease({
state: state(),
runtime: profileState,
configRevision,
signal: callerSignal,
run: async (lifecycleSignal) => await run(lifecycleSignal, profileState),
commit: options?.commit,
@ -182,7 +178,7 @@ function createProfileContext(
);
}
const userDataDir = resolveOpenClawUserDataDir(profile.name);
assertProfileLifecycleContext({ state: state(), runtime: profileState, configRevision });
assertProfileLifecycleContext({ state: state(), runtime: profileState });
profileState.managedLaunchFailure = undefined;
let result: Awaited<ReturnType<ProfileContext["resetProfile"]>> = {
moved: false,

33
extensions/browser/src/browser/session-tab-cleanup-claim.ts
View file

@ -57,14 +57,6 @@ export type CloseParams = SessionEntryCurrentPreparation & {
onDebug?: (message: string) => void;
};
/** Deduplicate deferrals by row generation across Browser plugin bundle instances. */
type DeferredTabDiagnostic = {
reason: string;
trackedAt: number;
profileFingerprint: string;
browserInstanceFingerprint: string;
};
const deferredTabDiagnosticsSymbol = Symbol.for(
"openclaw.browser.session-tabs.deferred-diagnostics",
);
@ -72,33 +64,19 @@ const deferredTabDiagnosticsSymbol = Symbol.for(
/** Defensive bound: live deferrals are already forgotten as their rows settle. */
const MAX_DEFERRED_TAB_DIAGNOSTICS = 512;
function deferredTabDiagnostics(): Map<string, DeferredTabDiagnostic> {
function deferredTabDiagnostics(): Map<string, string> {
return resolveGlobalMap(deferredTabDiagnosticsSymbol);
}
function sameDeferredTabRow(previous: DeferredTabDiagnostic, tab: DurableTab): boolean {
return (
previous.trackedAt === tab.trackedAt &&
previous.profileFingerprint === tab.profileFingerprint &&
previous.browserInstanceFingerprint === tab.browserInstanceFingerprint
);
}
/** Records a deferral, returning true when the row has not reported it yet. */
function recordDeferredTabDiagnostic(tab: DurableTab, reason: string): boolean {
const diagnostics = deferredTabDiagnostics();
const previous = diagnostics.get(tab.storageKey);
if (previous?.reason === reason && sameDeferredTabRow(previous, tab)) {
if (diagnostics.get(tab.storageKey) === reason) {
return false;
}
// Re-insert so the least recently reported deferral is evicted first.
diagnostics.delete(tab.storageKey);
diagnostics.set(tab.storageKey, {
reason,
trackedAt: tab.trackedAt,
profileFingerprint: tab.profileFingerprint,
browserInstanceFingerprint: tab.browserInstanceFingerprint,
});
diagnostics.set(tab.storageKey, reason);
if (diagnostics.size > MAX_DEFERRED_TAB_DIAGNOSTICS) {
const oldest = diagnostics.keys().next();
if (!oldest.done) {
@ -111,10 +89,7 @@ function recordDeferredTabDiagnostic(tab: DurableTab, reason: string): boolean {
/** Lets a settled row report the next outage instead of staying muted forever. */
function forgetDeferredTabDiagnostic(tab: DurableTab): void {
const diagnostics = deferredTabDiagnostics();
const previous = diagnostics.get(tab.storageKey);
if (previous && sameDeferredTabRow(previous, tab)) {
diagnostics.delete(tab.storageKey);
}
diagnostics.delete(tab.storageKey);
}
/** Warn once per deferral; repeated sweeps remain visible at debug level. */

7
extensions/browser/src/browser/session-tab-process-state.ts
View file

@ -52,13 +52,8 @@ export function volatileTabsBySession(): Map<string, Map<string, VolatileSession
return resolveGlobalMap(volatileStateSymbol);
}
type VolatileTabCleanup = {
registrations: VolatileSessionTab[];
promise: Promise<number>;
};
/** Keeps one in-flight volatile target close shared across Browser plugin bundles. */
export function volatileTabCleanupByTarget(): Map<string, VolatileTabCleanup> {
export function volatileTabCleanupByTarget(): Map<string, Promise<number>> {
return resolveGlobalMap(volatileCleanupStateSymbol);
}

82
extensions/browser/src/browser/session-tab-registry.test.ts
View file

@ -152,49 +152,6 @@ describe("session tab registry", () => {
},
);
it.each([false, true])(
"shares lifecycle cleanup after a preparing sweep is revoked (closeFails=%s)",
async (closeFails) => {
const tab = { sessionKey: "agent:main:main", targetId: "touched-sweep" };
await trackSessionBrowserTab({ ...tab, now: 1_000 });
const release = createDeferred<void>();
const sweep = sweepTrackedBrowserTabs({
now: 10_000,
idleMs: 1,
prepareCurrent: async () => {
await release.promise;
return true;
},
});
const closeTab = vi.fn(() => {
if (closeFails) {
throw new Error("close failed");
}
return Promise.resolve();
});
const lifecycle = () =>
closeTrackedBrowserTabsForSessions({ sessionKeys: [tab.sessionKey], closeTab });
const pending = [sweep, lifecycle(), lifecycle()];
try {
await touchSessionBrowserTab({ ...tab, now: 11_000 });
} finally {
release.resolve();
}
await expect(Promise.all(pending)).resolves.toEqual([0, closeFails ? 0 : 1, 0]);
expect(clientMocks.browserCloseTabByRawTargetId).not.toHaveBeenCalled();
expect(closeTab).toHaveBeenCalledOnce();
const retryClose = vi.fn(async () => {});
await expect(
closeTrackedBrowserTabsForSessions({
sessionKeys: [tab.sessionKey],
closeTab: retryClose,
}),
).resolves.toBe(closeFails ? 1 : 0);
expect(retryClose).toHaveBeenCalledTimes(closeFails ? 1 : 0);
},
);
it("does not adopt a new registration while an earlier selected tab closes", async () => {
const sessionKey = "agent:main:main";
const next = { sessionKey, targetId: "next-tab" };
@ -259,7 +216,7 @@ describe("session tab registry", () => {
} finally {
release.resolve();
}
await expect(cleanup).resolves.toBe(replacementPhase === "during-prepare" ? 0 : 1);
await expect(cleanup).resolves.toBe(replacementPhase === "during-close" ? 1 : 0);
expect(clientMocks.browserCloseTabByRawTargetId).not.toHaveBeenCalled();
const freshClose = vi.fn(async () => {});
await expect(
@ -307,43 +264,6 @@ describe("session tab registry", () => {
},
);
it.each([false, true])(
"binds a queued lifecycle request to its registration (replace=%s)",
async (replace) => {
const tab = { sessionKey: "agent:main:main", targetId: "queued-target" };
await trackSessionBrowserTab({ ...tab, now: 1_000 });
const entered = createDeferred<void>();
const release = createDeferred<void>();
const firstClose = vi.fn(async () => {
entered.resolve();
await release.promise;
});
const first = closeTrackedBrowserTabsForSessions({
sessionKeys: [tab.sessionKey],
closeTab: firstClose,
});
const nextClose = vi.fn(async () => {});
let second: Promise<number>;
try {
await entered.promise;
if (replace) {
await untrackSessionBrowserTab(tab);
await trackSessionBrowserTab({ ...tab, now: 1_000 });
}
second = closeTrackedBrowserTabsForSessions({
sessionKeys: [tab.sessionKey],
closeTab: nextClose,
});
expect(nextClose).not.toHaveBeenCalled();
} finally {
release.resolve();
}
await expect(first).resolves.toBe(1);
await expect(second).resolves.toBe(replace ? 1 : 0);
expect(nextClose).toHaveBeenCalledTimes(replace ? 1 : 0);
},
);
it("isolates volatile aliases by browser surface", async () => {
await trackSessionBrowserTab({
sessionKey: "agent:main:main",

138
extensions/browser/src/browser/session-tab-registry.ts
View file

@ -1,4 +1,3 @@
import { createDeferred } from "openclaw/plugin-sdk/extension-shared";
/**
* Session-owned browser tabs. Host-local durable ownership is canonical in
* plugin SQLite; all other tabs remain process-local.
@ -74,7 +73,13 @@ async function performVolatileCleanup(
? current
: undefined;
};
while (true) {
const existing = inFlight.get(targetKey);
if (existing) {
await existing;
// A replaced registration is picked up by the next sweep, not handed off mid-close.
return 0;
}
const cleanup = Promise.resolve().then(async () => {
if (params.prepareCurrent && !(await params.prepareCurrent())) {
return 0;
}
@ -82,87 +87,70 @@ async function performVolatileCleanup(
return 0;
}
params.authority?.assertCurrent?.();
const current = resolveCurrent();
if (!current) {
let tab = resolveCurrent();
if (!tab) {
return 0;
}
const existing = inFlight.get(targetKey);
if (existing) {
await existing.promise;
if (existing.registrations.some((owned) => owned.registration === candidate.registration)) {
return 0;
}
continue;
}
const { promise: cleanup, resolve: complete } = createDeferred<number>();
// Preparation and dispatch share one reservation, including reentrant closers.
// Completion retires only the acquired registrations.
const owner = { registrations: volatileRegistrationsForTarget(targetKey), promise: cleanup };
const performClose = async () => {
let tab = current;
let closeTab = params.closeTab;
try {
if (!closeTab && tab.route.kind === "browser-control") {
const { browserCloseTabByRawTargetId } = await import("./client-tab-close.runtime.js");
const latest = resolveCurrent();
if (!latest) {
// No dispatch occurred: a lifecycle joiner may retry a touched sweep.
owner.registrations = [];
return 0;
}
tab = latest;
closeTab = ({ baseUrl, targetId, profile }) =>
browserCloseTabByRawTargetId(baseUrl, targetId, { profile });
}
if (closeTab) {
await closeTab({
targetId: tab.targetId,
...(tab.route.kind === "browser-control" && tab.route.baseUrl
? { baseUrl: tab.route.baseUrl }
: {}),
...(tab.route.kind === "node-proxy" ? { route: tab.route } : {}),
...(tab.profile ? { profile: tab.profile } : {}),
});
} else if (tab.route.kind === "node-proxy") {
const outcome = await tab.route.closeTarget({
targetId: tab.targetId,
profile: tab.profile,
ownership: tab.ownership,
});
if (outcome.status === "cancelled" || outcome.status === "unavailable") {
params.onWarn?.(
`deferred tracked browser tab ${tab.targetId}: ${outcome.status === "unavailable" ? outcome.reason : "cleanup cancelled"}`,
);
return 0;
}
if (outcome.status === "ownership-mismatch") {
params.onWarn?.(`retired tracked browser tab ${tab.targetId}: ownership mismatch`);
}
deleteVolatileRegistrations(owner.registrations);
return outcome.status === "closed" ? 1 : 0;
}
} catch (error) {
if (closeTab && tab.route.kind === "browser-control" && isIgnorableTabCloseError(error)) {
deleteVolatileRegistrations(owner.registrations);
const registrations = volatileRegistrationsForTarget(targetKey);
let closeTab = params.closeTab;
try {
if (!closeTab && tab.route.kind === "browser-control") {
const { browserCloseTabByRawTargetId } = await import("./client-tab-close.runtime.js");
const latest = resolveCurrent();
if (!latest) {
return 0;
}
params.onWarn?.(`failed to close tracked browser tab ${tab.targetId}: ${String(error)}`);
tab = latest;
closeTab = ({ baseUrl, targetId, profile }) =>
browserCloseTabByRawTargetId(baseUrl, targetId, { profile });
}
params.authority?.assertCurrent?.();
if (!isCleanupCurrent(params)) {
return 0;
}
deleteVolatileRegistrations(owner.registrations);
return 1;
};
inFlight.set(targetKey, owner);
try {
complete(performClose());
return await cleanup;
} finally {
// Queued handoff callers must see the reservation until its completion settles.
if (inFlight.get(targetKey) === owner) {
inFlight.delete(targetKey);
if (closeTab) {
await closeTab({
targetId: tab.targetId,
...(tab.route.kind === "browser-control" && tab.route.baseUrl
? { baseUrl: tab.route.baseUrl }
: {}),
...(tab.route.kind === "node-proxy" ? { route: tab.route } : {}),
...(tab.profile ? { profile: tab.profile } : {}),
});
} else if (tab.route.kind === "node-proxy") {
const outcome = await tab.route.closeTarget({
targetId: tab.targetId,
profile: tab.profile,
ownership: tab.ownership,
});
if (outcome.status === "cancelled" || outcome.status === "unavailable") {
params.onWarn?.(
`deferred tracked browser tab ${tab.targetId}: ${outcome.status === "unavailable" ? outcome.reason : "cleanup cancelled"}`,
);
return 0;
}
if (outcome.status === "ownership-mismatch") {
params.onWarn?.(`retired tracked browser tab ${tab.targetId}: ownership mismatch`);
}
deleteVolatileRegistrations(registrations);
return outcome.status === "closed" ? 1 : 0;
}
} catch (error) {
if (closeTab && tab.route.kind === "browser-control" && isIgnorableTabCloseError(error)) {
deleteVolatileRegistrations(registrations);
return 0;
}
params.onWarn?.(`failed to close tracked browser tab ${tab.targetId}: ${String(error)}`);
return 0;
}
deleteVolatileRegistrations(registrations);
return 1;
});
inFlight.set(targetKey, cleanup);
try {
return await cleanup;
} finally {
inFlight.delete(targetKey);
}
}

11
extensions/browser/src/session-browser-dashboard.test.ts
View file

@ -36,9 +36,10 @@ vi.mock("./control-service.js", async (importOriginal) => ({
vi.mock("./browser/pw-ai-module.js", () => ({
getPwAiModule: async () => ({ createPageViaPlaywright: mocked.create }),
}));
vi.mock("./browser/server-context.lifecycle.js", () => ({
vi.mock("./browser/server-context.lifecycle.js", async (original) => ({
...(await original<typeof import("./browser/server-context.lifecycle.js")>()),
getProfileLifecycle: () => mocked.lifecycle,
isProfileGenerationCurrent: () => mocked.profileCurrent,
isProfileOperationCurrent: () => mocked.profileCurrent,
}));
const request = { sessionKey: "agent:main:dashboard:one", agentId: "main", name: "review" };
@ -105,7 +106,7 @@ beforeEach(() => {
const state = makeBrowserServerState();
state.profiles.set("openclaw", { profile: makeBrowserProfile(), running: null });
mocked.state = state;
mocked.lifecycle = { generation: 1, configRevision: 1, controller: new AbortController() };
mocked.lifecycle = { controller: new AbortController() };
mocked.profileCurrent = true;
mocked.definition.mockResolvedValue(definition);
mocked.start.mockResolvedValue(true);
@ -197,13 +198,11 @@ describe("isolated session browser owner", () => {
expect(mocked.create).toHaveBeenCalledTimes(2);
});
it("fences a changed board immediately and closes its complete context", async () => {
it("closes the complete context when its board definition changes", async () => {
const owner = authority();
const { resource } = await accessSessionBrowserDashboard(request, owner.value, {
operation: "open",
});
resource.definitionChanged();
expect(() => resource.assertCurrent()).toThrow("definition changed");
mocked.definition.mockResolvedValue({ ...definition, url: "https://replacement.test/" });
await expect(resource.assertDefinitionCurrent()).rejects.toThrow("removed or replaced");
expect(resource.page?.close).toHaveBeenCalledOnce();

24
extensions/browser/src/session-browser-dashboard.ts
View file

@ -17,7 +17,7 @@ import { getPwAiModule } from "./browser/pw-ai-module.js";
import { browserNavigationPolicyForProfile } from "./browser/routes/agent.shared.js";
import {
getProfileLifecycle,
isProfileGenerationCurrent,
isProfileOperationCurrent,
} from "./browser/server-context.lifecycle.js";
import { startBrowserControlServiceFromConfig } from "./control-service.js";
@ -66,8 +66,6 @@ async function createResource(
const session = authority.retainSession();
const controller = new AbortController();
const key = keyFor(definition);
let definitionEpoch = 0;
let verifiedDefinitionEpoch = 0;
let assertProfileCurrent: (() => void) | undefined;
let removeProfileAbort: (() => void) | undefined;
let closing: Promise<void> | undefined;
@ -100,9 +98,6 @@ async function createResource(
throw new Error("The isolated browser context is no longer available. Reopen the dashboard.");
}
assertProfileCurrent?.();
if (verifiedDefinitionEpoch !== definitionEpoch) {
throw new Error("The dashboard definition changed. Refresh the dashboard before continuing.");
}
};
const resource: SessionBrowserDashboard = {
definition,
@ -112,25 +107,17 @@ async function createResource(
assertCurrent,
assertDefinitionCurrent: async () => {
session.assertCurrent();
const epoch = definitionEpoch;
const current = await readBrowserDashboardDefinition(
definition,
getBrowserControlState()?.resolved.defaultProfile,
);
session.assertCurrent();
if (epoch !== definitionEpoch) {
throw new Error("The dashboard changed during verification. Retry the operation.");
}
if (!sameBrowserDashboardDefinition(definition, current)) {
await close();
throw new Error("The dashboard was removed or replaced. Open its current definition.");
}
verifiedDefinitionEpoch = epoch;
assertCurrent();
},
definitionChanged: () => {
definitionEpoch += 1;
},
close,
};
const onAbort = () => {
@ -168,19 +155,18 @@ async function createResource(
throw new Error("Browser profile is unavailable.");
}
const lifecycle = getProfileLifecycle(profile);
const generation = lifecycle.generation;
const configRevision = lifecycle.configRevision;
const profileSignal = lifecycle.controller.signal;
assertProfileCurrent = () => {
if (
getBrowserControlState() !== state ||
state.profiles.get(definition.profile) !== profile ||
!isProfileGenerationCurrent({ state, runtime: profile, generation, configRevision })
!isProfileOperationCurrent({ state, runtime: profile, signal: profileSignal })
) {
throw new Error("The browser profile changed. Reopen the dashboard.");
}
};
lifecycle.controller.signal.addEventListener("abort", onAbort, { once: true });
removeProfileAbort = () => lifecycle.controller.signal.removeEventListener("abort", onAbort);
profileSignal.addEventListener("abort", onAbort, { once: true });
removeProfileAbort = () => profileSignal.removeEventListener("abort", onAbort);
const playwright = await getPwAiModule({ mode: "strict" });
authority.assertCurrent();
assertProfileCurrent();

13
extensions/memory-core/src/dreaming-dreams-file.test.ts
View file

@ -12,12 +12,6 @@ import {
updateDreamsFile,
writeBackfillDiaryEntries,
} from "./dreaming-dreams-file.js";
import {
SHORT_TERM_LOCK_MAX_ENTRIES,
SHORT_TERM_LOCK_NAMESPACE,
memoryCoreWorkspaceStateKey,
openMemoryCoreStateStore,
} from "./dreaming-state.js";
import { forgetMemoryEntries } from "./memory-forget.js";
import { createMemoryCoreTestHarness } from "./test-helpers.js";
@ -54,18 +48,11 @@ describe("dream diary file behavior", () => {
let forgotten: ReturnType<typeof forgetMemoryEntries> | undefined;
try {
await prepared.promise;
const writerOwnsLock = await openMemoryCoreStateStore({
namespace: SHORT_TERM_LOCK_NAMESPACE,
maxEntries: SHORT_TERM_LOCK_MAX_ENTRIES,
}).lookup(memoryCoreWorkspaceStateKey(workspaceDir));
forgotten = forgetMemoryEntries({
cfg: { agents: { entries: { main: { workspace: workspaceDir } } } },
agentId: "main",
sessionIds: ["forgotten"],
});
if (!writerOwnsLock) {
await forgotten;
}
publish.resolve();
await Promise.all([update, forgotten]);
const content = await fs.readFile(dreamsPath, "utf8");

15
extensions/memory-core/src/dreaming-phases.test.ts
View file

@ -19,12 +19,6 @@ import {
runDreamingSweepPhases,
seedHistoricalDailyMemorySignals,
} from "./dreaming-phases.js";
import {
memoryCoreWorkspaceStateKey,
openMemoryCoreStateStore,
SHORT_TERM_LOCK_MAX_ENTRIES,
SHORT_TERM_LOCK_NAMESPACE,
} from "./dreaming-state.js";
import { forgetMemoryEntries } from "./memory-forget.js";
import { previewRemHarness } from "./rem-harness.js";
import { appendSessionCorpusLines } from "./session-ingestion.js";
@ -944,16 +938,7 @@ describe("memory-core dreaming phases", () => {
}),
]);
expect(pendingContent).toContain(claim);
const publisherOwnsLock = await openMemoryCoreStateStore({
namespace: SHORT_TERM_LOCK_NAMESPACE,
maxEntries: SHORT_TERM_LOCK_MAX_ENTRIES,
}).lookup(memoryCoreWorkspaceStateKey(workspaceDir));
forgotten = forgetMemoryEntries({ cfg, agentId: "main", sessionIds: [sessionId] });
// Finish deletion before a writer without a lease resumes. A serialized
// writer must finish first; this exercises both orders without sleeps.
if (!publisherOwnsLock) {
await forgotten;
}
publish.resolve();
await Promise.all([sweep, forgotten]);
for (const file of [

5
extensions/memory-core/src/memory-mutation-authority.test.ts
View file

@ -329,11 +329,6 @@ describe("memory mutation authority at durable effects", () => {
operation,
"Rollback never reached partial failure",
);
const locks = openMemoryCoreStateStore({
namespace: SHORT_TERM_LOCK_NAMESPACE,
maxEntries: SHORT_TERM_LOCK_MAX_ENTRIES,
});
expect(await locks.lookup(memoryCoreWorkspaceStateKey(rollbackWorkspace))).toBeDefined();
expect(settled).toBe(false);
} finally {
finish.resolve();

116
extensions/memory-core/src/memory-search-tool-query.test.ts
View file

@ -13,66 +13,58 @@ vi.mock("./session-search-visibility.js", () => ({
afterEach(() => vi.resetAllMocks());
it.each(["fallback", "abort", "session"] as const)(
"withholds keyword snapshots for %s",
async (event) => {
const hit: MemorySearchResult = {
path: "memory/visible.md",
source: event === "session" ? "sessions" : "memory",
startLine: 1,
endLine: 1,
score: 1,
snippet: "Visible keyword match",
};
const filtering = createDeferred<MemorySearchResult[]>();
const finishSearch = createDeferred<MemorySearchResult[]>();
const searchEntered = createDeferred<void>();
let searchOptions: Parameters<MemorySearchManager["search"]>[1];
const manager: MemorySearchManager = {
search: async (_query, options) => {
searchOptions = options;
options?.onPartialResults?.([hit]);
searchEntered.resolve();
return await finishSearch.promise;
},
status: () => ({ backend: "builtin", provider: "fixture" }),
readFile: async () => ({ status: "not_found", path: "", text: "" }),
probeEmbeddingAvailability: async () => ({ ok: true }),
probeVectorAvailability: async () => false,
};
vi.mocked(filterMemorySearchHitsBySessionVisibility)
.mockImplementationOnce(() => filtering.promise)
.mockResolvedValue([]);
const caller = new AbortController();
const snapshots = vi.fn();
const query = executeMemorySearchToolQuery({
initialManager: { manager },
refreshManager: async () => null,
query: { text: "keyword", resultLimit: 1, defaultSources: ["memory"] },
visibility: { cfg: {}, agentId: "main", sandboxed: false },
signal: caller.signal,
onPartialResults: snapshots,
});
try {
await searchEntered.promise;
expect(filterMemorySearchHitsBySessionVisibility).toHaveBeenCalledTimes(
event === "session" ? 0 : 1,
);
expect(snapshots).toHaveBeenCalledWith(null);
if (event === "fallback") {
searchOptions?.onPartialResults?.(null);
} else if (event === "abort") {
caller.abort(new Error("caller cancelled"));
}
filtering.resolve([hit]);
await filtering.promise;
await Promise.resolve();
await Promise.resolve();
expect(snapshots.mock.calls.every(([result]) => result === null)).toBe(true);
} finally {
filtering.resolve([]);
finishSearch.resolve([]);
await query;
it.each(["abort", "session"] as const)("withholds keyword snapshots for %s", async (event) => {
const hit: MemorySearchResult = {
path: "memory/visible.md",
source: event === "session" ? "sessions" : "memory",
startLine: 1,
endLine: 1,
score: 1,
snippet: "Visible keyword match",
};
const filtering = createDeferred<MemorySearchResult[]>();
const finishSearch = createDeferred<MemorySearchResult[]>();
const searchEntered = createDeferred<void>();
const manager: MemorySearchManager = {
search: async (_query, options) => {
options?.onPartialResults?.([hit]);
searchEntered.resolve();
return await finishSearch.promise;
},
status: () => ({ backend: "builtin", provider: "fixture" }),
readFile: async () => ({ status: "not_found", path: "", text: "" }),
probeEmbeddingAvailability: async () => ({ ok: true }),
probeVectorAvailability: async () => false,
};
vi.mocked(filterMemorySearchHitsBySessionVisibility)
.mockImplementationOnce(() => filtering.promise)
.mockResolvedValue([]);
const caller = new AbortController();
const snapshots = vi.fn();
const query = executeMemorySearchToolQuery({
initialManager: { manager },
query: { text: "keyword", resultLimit: 1, defaultSources: ["memory"] },
visibility: { cfg: {}, agentId: "main", sandboxed: false },
signal: caller.signal,
onPartialResults: snapshots,
});
try {
await searchEntered.promise;
expect(filterMemorySearchHitsBySessionVisibility).toHaveBeenCalledTimes(
event === "session" ? 0 : 1,
);
expect(snapshots).toHaveBeenCalledWith(null);
if (event === "abort") {
caller.abort(new Error("caller cancelled"));
}
},
);
filtering.resolve([hit]);
await filtering.promise;
await Promise.resolve();
await Promise.resolve();
expect(snapshots.mock.calls.every(([result]) => result === null)).toBe(true);
} finally {
filtering.resolve([]);
finishSearch.resolve([]);
await query;
}
});

38
extensions/memory-core/src/memory-search-tool-query.ts
View file

@ -1,4 +1,3 @@
import { formatErrorMessage } from "openclaw/plugin-sdk/error-runtime";
import {
resolveMemoryIndexIdentityDiagnostic,
resolveMemoryIndexSearchDiagnostic,
@ -58,19 +57,8 @@ type MemorySearchToolVisibility = {
sandboxed: boolean;
};
function isClosedMemoryStoreError(error: unknown): boolean {
const message = formatErrorMessage(error).toLowerCase();
return (
message.includes("database is not open") ||
message.includes("database connection is not open") ||
message.includes("database handle is closed") ||
message.includes("memory index manager is closed")
);
}
export async function executeMemorySearchToolQuery(params: {
initialManager: ManagerState;
refreshManager: () => Promise<ManagerState | null>;
query: MemorySearchToolQuery;
visibility: MemorySearchToolVisibility;
signal: AbortSignal;
@ -82,8 +70,7 @@ export async function executeMemorySearchToolQuery(params: {
}) {
const startedAt = Date.now();
const runtimeDebug: MemorySearchRuntimeDebug[] = [];
let active = params.initialManager;
let partialGeneration = 0;
const active = params.initialManager;
const { query, signal, visibility } = params;
// Product recall may index transcripts without adding them to ordinary model search.
// Explicit corpus selection is authorized by the tool owner before this point.
@ -131,7 +118,6 @@ export async function executeMemorySearchToolQuery(params: {
onDebug: (debug) => runtimeDebug.push(debug),
onPartialResults: params.onPartialResults
? (partialCandidates) => {
const generation = ++partialGeneration;
params.onPartialResults?.(null);
// Session visibility can change while semantic retrieval waits. A deadline
// cannot reuse earlier session authority, so retain only durable memory files.
@ -141,7 +127,6 @@ export async function executeMemorySearchToolQuery(params: {
if (!memoryCandidates?.length || signal.aborted) {
return;
}
// Finalization yields; only the latest permitted snapshot survives fallback.
void finalizeMemorySearchToolQuery({
active,
searched: { candidates: memoryCandidates, searchWindow },
@ -150,7 +135,7 @@ export async function executeMemorySearchToolQuery(params: {
effectiveMode: "keyword-only",
}).then(
(result) => {
if (generation === partialGeneration && !signal.aborted) {
if (!signal.aborted) {
params.onPartialResults?.(result.pausedIndexIdentity ? null : result);
}
},
@ -163,24 +148,7 @@ export async function executeMemorySearchToolQuery(params: {
return { searched: { candidates, searchWindow }, status: active.manager.status() };
};
let searched: Awaited<ReturnType<typeof searchOnce>>;
try {
searched = await searchOnce();
} catch (error) {
if (!isClosedMemoryStoreError(error)) {
throw error;
}
partialGeneration += 1;
params.onPartialResults?.(null);
const refreshed = await params.refreshManager();
if (!refreshed) {
throw error;
}
active = refreshed;
searched = await searchOnce();
} finally {
partialGeneration += 1;
}
const searched = await searchOnce();
return await finalizeMemorySearchToolQuery({
active,

132
extensions/memory-core/src/memory-workspace-lock.cas.test.ts
View file

@ -3,10 +3,7 @@ import type {
PluginStateKeyedStore,
} from "openclaw/plugin-sdk/plugin-state-runtime";
import { afterEach, describe, expect, it, vi } from "vitest";
import {
deleteShortTermLockEntryIfCurrent,
withMemoryWorkspaceLock,
} from "./memory-workspace-lock.js";
import { deleteShortTermLockEntryIfCurrent } from "./memory-workspace-lock.js";
import type { ShortTermLockEntry } from "./short-term-promotion-types.js";
const state = vi.hoisted(() => ({ open: vi.fn() }));
@ -21,11 +18,6 @@ vi.mock("openclaw/plugin-sdk/process-runtime", () => ({
getFileLockProcessStartTime: () => 123,
isPidDefinitelyDead: () => false,
}));
vi.mock("openclaw/plugin-sdk/runtime-env", () => ({
sleep: async () => {
throw new Error("Unexpected lock contention");
},
}));
function createLockStore() {
const rows = new Map<string, ShortTermLockEntry>();
@ -89,28 +81,6 @@ describe("memory workspace lock comparisons", () => {
const key = "synthetic-workspace";
const expected: ShortTermLockEntry = { owner: "synthetic-owner", acquiredAt: 1 };
it("reports the observed holder when the acquisition bound expires", async () => {
const store = createLockStore();
const acquiredAt = Date.now();
const holder = { owner: `${process.ppid}:active`, acquiredAt };
await store.register(key, holder);
const clock = vi
.spyOn(Date, "now")
.mockReturnValueOnce(acquiredAt - 10_000)
.mockReturnValue(acquiredAt);
const task = vi.fn(async () => "unreachable");
try {
await expect(withMemoryWorkspaceLock(key, task)).rejects.toMatchObject({
code: "MEMORY_WORKSPACE_LOCK_HELD",
outcome: { kind: "held", holder: { owner: holder.owner, epoch: acquiredAt } },
});
expect(task).not.toHaveBeenCalled();
expect(await store.lookup(key)).toEqual(holder);
} finally {
clock.mockRestore();
}
});
it.each([
{ owner: "replacement-owner", acquiredAt: expected.acquiredAt },
{ owner: expected.owner, acquiredAt: expected.acquiredAt + 1 },
@ -140,106 +110,6 @@ describe("memory workspace lock comparisons", () => {
expect(await store.lookup(key)).toBeUndefined();
});
it("keeps a failed release best effort and recovers before the next owner runs", async () => {
const store = createLockStore();
const compare = store.compareAndApply.getMockImplementation()!;
const failedDeletion = vi.fn(() => {
throw new Error("worker result unavailable");
});
store.compareAndApply.mockImplementation(async (...args) =>
args[2].action === "delete" ? failedDeletion() : compare(...args),
);
const firstTask = vi.fn(async () => "completed");
await expect(withMemoryWorkspaceLock(key, firstTask)).resolves.toBe("completed");
expect(failedDeletion).toHaveBeenCalledOnce();
const completed = await store.lookup(key);
expect(completed).toBeDefined();
store.compareAndApply.mockImplementation(compare);
await expect(
withMemoryWorkspaceLock(key, async () => {
expect(await store.lookup(key)).not.toEqual(completed);
return "recovered";
}),
).resolves.toBe("recovered");
expect(firstTask).toHaveBeenCalledOnce();
expect(await store.lookup(key)).toBeUndefined();
});
it("recovers an applied acquisition whose reply failed without replaying its task", async () => {
const store = createLockStore();
const compare = store.compareAndApply.getMockImplementation()!;
const unavailable = new Error("worker acquisition result unavailable");
store.compareAndApply.mockImplementationOnce(async (...args) => {
expect(await compare(...args)).toEqual({ status: "applied" });
throw unavailable;
});
const originalTask = vi.fn(async () => "must not run");
await expect(withMemoryWorkspaceLock(key, originalTask)).rejects.toMatchObject({
code: "MEMORY_WORKSPACE_LOCK_STORE_UNAVAILABLE",
outcome: { kind: "store-unavailable", reason: "storage-error" },
cause: unavailable,
});
const orphan = await store.lookup(key);
expect(orphan).toBeDefined();
expect(originalTask).not.toHaveBeenCalled();
await expect(
withMemoryWorkspaceLock(key, async () => {
expect(await store.lookup(key)).not.toEqual(orphan);
return "new owner";
}),
).resolves.toBe("new owner");
expect(originalTask).not.toHaveBeenCalled();
expect(await store.lookup(key)).toBeUndefined();
});
it.each(["owner", "acquiredAt"] as const)(
"preserves a live replacement with a different %s after failed cleanup",
async (field) => {
const store = createLockStore();
const compare = store.compareAndApply.getMockImplementation()!;
store.compareAndApply.mockImplementation(async (...args) => {
if (args[2].action === "delete") {
throw new Error("worker result unavailable");
}
return compare(...args);
});
await withMemoryWorkspaceLock(key, async () => "completed");
const completed = await store.lookup(key);
expect(completed).toBeDefined();
if (!completed) {
throw new Error("Expected the completed lease receipt");
}
const replacement = {
...completed,
...(field === "owner"
? { owner: `${process.pid}:independent-live-owner` }
: { acquiredAt: completed.acquiredAt + 1 }),
};
await store.register(key, replacement);
store.compareAndApply.mockImplementation(compare);
const nextTask = vi.fn(async () => "must not run");
await expect(withMemoryWorkspaceLock(key, nextTask)).rejects.toThrow(
"Unexpected lock contention",
);
expect(nextTask).not.toHaveBeenCalled();
expect(await store.lookup(key)).toEqual(replacement);
},
);
it("retains an unknown fresh same-process lease without treating it as completed", async () => {
const store = createLockStore();
const unknown = { owner: `${process.pid}:unknown-owner`, acquiredAt: Date.now() };
await store.register(key, unknown);
const nextTask = vi.fn(async () => "must not run");
await expect(withMemoryWorkspaceLock(key, nextTask)).rejects.toThrow(
"Unexpected lock contention",
);
expect(nextTask).not.toHaveBeenCalled();
expect(await store.lookup(key)).toEqual(unknown);
});
it.each(["observe", "compareAndApply"] as const)(
"requires %s before deletion",
async (method) => {

239
extensions/memory-core/src/memory-workspace-lock.test.ts
View file

@ -1,132 +1,16 @@
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { DatabaseSync } from "node:sqlite";
import { setImmediate as nextTurn } from "node:timers/promises";
import { collectErrorGraphCandidates } from "openclaw/plugin-sdk/error-runtime";
import { createDeferred } from "openclaw/plugin-sdk/extension-shared";
import type { OpenKeyedStoreOptions } from "openclaw/plugin-sdk/plugin-state-runtime";
import {
createPluginStateKeyedStoreForTests,
openOpenClawStateDatabase,
} from "openclaw/plugin-sdk/plugin-state-test-runtime";
import { getFileLockProcessStartTime } from "openclaw/plugin-sdk/process-runtime";
import { afterAll, beforeAll, afterEach, describe, expect, it, vi } from "vitest";
import {
configureMemoryCoreDreamingState,
memoryCoreWorkspaceStateKey,
openMemoryCoreStateStore,
SHORT_TERM_LOCK_MAX_ENTRIES,
SHORT_TERM_LOCK_NAMESPACE,
} from "./dreaming-state.js";
import { describe, expect, it } from "vitest";
import {
withMemoryWorkspaceLock,
withMemoryWorkspacePreparation,
} from "./memory-workspace-lock.js";
import {
auditShortTermPromotionArtifacts,
repairShortTermPromotionArtifacts,
} from "./short-term-promotion-artifacts.js";
import type { ShortTermLockEntry } from "./short-term-promotion-types.js";
import {
configureMemoryCoreDreamingStateForTests,
resetMemoryCoreDreamingStateForTests,
shortTermTestState as testing,
} from "./test-helpers.js";
describe("memory workspace lock orphan recovery", () => {
let fixtureRoot = "";
describe("memory workspace ordering", () => {
let caseId = 0;
beforeAll(async () => {
await configureMemoryCoreDreamingStateForTests();
fixtureRoot = await fs.mkdtemp(path.join(os.tmpdir(), "memory-lock-"));
});
afterAll(async () => {
if (fixtureRoot) {
await fs.rm(fixtureRoot, { recursive: true, force: true });
}
resetMemoryCoreDreamingStateForTests();
});
afterEach(() => {
vi.useRealTimers();
vi.restoreAllMocks();
});
async function makeWorkspace(): Promise<string> {
const workspaceDir = path.join(fixtureRoot, `case-${caseId++}`);
await fs.mkdir(path.join(workspaceDir, "memory", ".dreams"), { recursive: true });
return workspaceDir;
return `/memory-workspace-test/case-${caseId++}`;
}
it("waits for an active short-term lock before repairing", async () => {
const workspaceDir = await makeWorkspace();
await testing.writeRawRecallStore(workspaceDir, {
version: 1,
updatedAt: "2026-04-04T00:00:00.000Z",
entries: {
bad: {
path: "",
},
},
});
const acquiredAt = Date.now();
const activeLock = { owner: `${process.pid}:${acquiredAt}`, acquiredAt };
await testing.writeShortTermLock(workspaceDir, activeLock);
const blocked = createDeferred<void>();
const lockKey = memoryCoreWorkspaceStateKey(workspaceDir);
let activeObservations = 0;
configureMemoryCoreDreamingState(<T>(options: OpenKeyedStoreOptions) => {
const store = createPluginStateKeyedStoreForTests<T>("memory-core", options);
return {
...store,
async observe(...args: Parameters<typeof store.observe>) {
const observation = await store.observe(...args);
if (
options.namespace === SHORT_TERM_LOCK_NAMESPACE &&
args[0] === lockKey &&
activeObservations < 2
) {
expect(observation.value).toEqual(activeLock);
if (++activeObservations === 2) {
blocked.resolve();
}
}
return observation;
},
};
});
let settled = false;
const repairPromise = repairShortTermPromotionArtifacts({ workspaceDir }).then((result) => {
settled = true;
return result;
});
try {
// A second real observation proves the owner waited and kept the active lock intact.
await Promise.race([
blocked.promise,
repairPromise.then(() => {
throw new Error("Repair completed before observing the active lock");
}),
]);
expect(settled).toBe(false);
await testing.deleteShortTermLock(workspaceDir);
const repair = await repairPromise;
expect(repair.changed).toBe(true);
expect(repair.rewroteStore).toBe(true);
expect(repair.removedInvalidEntries).toBe(1);
} finally {
await testing.deleteShortTermLock(workspaceDir);
await Promise.allSettled([repairPromise]);
await configureMemoryCoreDreamingStateForTests();
}
});
it("keeps preparations and writers in the same local FIFO", async () => {
const workspace = await makeWorkspace();
const entered = createDeferred<void>();
@ -215,121 +99,4 @@ describe("memory workspace lock orphan recovery", () => {
}
expect(order).toEqual(["writer", "preparation"]);
});
it("reports unavailable storage with the original SQLite busy cause", async () => {
const workspace = await makeWorkspace();
const blocker = new DatabaseSync(openOpenClawStateDatabase().path);
const task = vi.fn(async () => "unreachable");
try {
blocker.exec("PRAGMA busy_timeout = 0");
blocker.exec("BEGIN EXCLUSIVE");
const failure = await withMemoryWorkspaceLock(workspace, task).catch(
(error: unknown) => error,
);
expect(failure).toMatchObject({
code: "MEMORY_WORKSPACE_LOCK_STORE_UNAVAILABLE",
outcome: { kind: "store-unavailable", reason: "storage-error" },
});
expect(collectErrorGraphCandidates(failure, (record) => [record.cause])).toContainEqual(
expect.objectContaining({ errcode: 5 }),
);
expect(task).not.toHaveBeenCalled();
} finally {
blocker.close();
}
});
it("surfaces SQLite contention after a failed release leaves a fresh local lock", async () => {
const workspace = await makeWorkspace();
const store = openMemoryCoreStateStore<ShortTermLockEntry>({
namespace: SHORT_TERM_LOCK_NAMESPACE,
maxEntries: SHORT_TERM_LOCK_MAX_ENTRIES,
});
const stateBlocker = new DatabaseSync(openOpenClawStateDatabase().path);
try {
await expect(
withMemoryWorkspaceLock(workspace, async () => {
stateBlocker.exec("PRAGMA busy_timeout = 0");
stateBlocker.exec("BEGIN EXCLUSIVE");
return "completed";
}),
).resolves.toBe("completed");
} finally {
stateBlocker.close();
}
const key = memoryCoreWorkspaceStateKey(workspace);
expect(await store.lookup(key)).toMatchObject({
owner: expect.stringMatching(`${process.pid}:`),
});
const dataPath = path.join(workspace, "locked.sqlite");
const data = new DatabaseSync(dataPath);
const dataBlocker = new DatabaseSync(dataPath);
try {
data.exec("CREATE TABLE entries (value TEXT)");
data.exec("PRAGMA busy_timeout = 0");
dataBlocker.exec("BEGIN EXCLUSIVE");
await expect(
withMemoryWorkspaceLock(workspace, async () => {
data.exec("INSERT INTO entries VALUES ('pending')");
}),
).rejects.toMatchObject({ errcode: 5 });
} finally {
dataBlocker.close();
data.close();
}
expect(await store.lookup(key)).toBeUndefined();
});
it.each(["legacy same-PID", "foreign start identity"] as const)(
"reclaims a stale lock after reuse of its %s",
async (identity) => {
vi.useFakeTimers();
const now = Date.now();
vi.setSystemTime(now);
const workspaceDir = await makeWorkspace();
const lock: ShortTermLockEntry = {
owner: `${process.pid}:${now - 120_000}`,
acquiredAt: now - 120_000,
};
if (identity === "foreign start identity") {
const ownerStartTime = getFileLockProcessStartTime(process.ppid);
expect(ownerStartTime).not.toBeNull();
if (ownerStartTime === null) {
throw new Error("Expected the test runner process start identity");
}
lock.owner = `${process.ppid}:${now - 120_000}`;
lock.ownerStartTime = ownerStartTime + 1;
}
await testing.writeShortTermLock(workspaceDir, lock);
const result = withMemoryWorkspaceLock(workspaceDir, async () => "recovered").then(
(value) => ({ status: "resolved" as const, value }),
(error: unknown) => ({ status: "rejected" as const, error: String(error) }),
);
await vi.advanceTimersByTimeAsync(10_040);
expect(await result).toEqual({ status: "resolved", value: "recovered" });
},
);
it("keeps a stale-looking lock while its owner task is active", async () => {
const workspaceDir = await makeWorkspace();
const entered = createDeferred<void>();
const release = createDeferred<void>();
const owner = withMemoryWorkspaceLock(workspaceDir, async () => {
entered.resolve();
await release.promise;
});
await entered.promise;
const now = Date.now();
vi.spyOn(Date, "now").mockReturnValue(now + 120_000);
try {
const audit = await auditShortTermPromotionArtifacts({ workspaceDir });
expect(audit.issues.map((issue) => issue.code)).not.toContain("recall-lock-stale");
} finally {
vi.restoreAllMocks();
release.resolve();
await owner;
}
});
});

168
extensions/memory-core/src/memory-workspace-lock.ts
View file

@ -1,9 +1,7 @@
import { AsyncLocalStorage } from "node:async_hooks";
import { randomUUID } from "node:crypto";
import { KeyedAsyncQueue } from "openclaw/plugin-sdk/keyed-async-queue";
import type {
PluginStateCompareIntent,
PluginStateCompareResult,
PluginStateKeyedStore,
PluginStateObservation,
} from "openclaw/plugin-sdk/plugin-state-runtime";
@ -11,71 +9,15 @@ import {
getFileLockProcessStartTime,
isPidDefinitelyDead,
} from "openclaw/plugin-sdk/process-runtime";
import { sleep } from "openclaw/plugin-sdk/runtime-env";
import {
SHORT_TERM_LOCK_MAX_ENTRIES,
SHORT_TERM_LOCK_NAMESPACE,
memoryCoreStateReference,
memoryCoreWorkspaceStateKey,
openMemoryCoreStateStore,
} from "./dreaming-state.js";
import type { ShortTermLockEntry } from "./short-term-promotion-types.js";
const MEMORY_WORKSPACE_LOCK_WAIT_TIMEOUT_MS = 10_000;
const SHORT_TERM_LOCK_STALE_MS = 60_000;
const MEMORY_WORKSPACE_LOCK_RETRY_DELAY_MS = 40;
const inProcessMemoryWorkspaceLocks = new KeyedAsyncQueue();
const activeMemoryWorkspaceLockOwners = new Map<string, string>();
type MemoryWorkspaceLockReceipt = {
key: string;
entry: ShortTermLockEntry;
observation: PluginStateObservation<ShortTermLockEntry>;
};
const pendingMemoryWorkspaceLockReleases = new Map<string, MemoryWorkspaceLockReceipt>();
function retainMemoryWorkspaceLockRelease(receipt: MemoryWorkspaceLockReceipt): void {
const now = Date.now();
for (const [owner, pending] of pendingMemoryWorkspaceLockReleases) {
if (now - pending.entry.acquiredAt > SHORT_TERM_LOCK_STALE_MS) {
pendingMemoryWorkspaceLockReleases.delete(owner);
}
}
if (now - receipt.entry.acquiredAt > SHORT_TERM_LOCK_STALE_MS) {
return;
}
pendingMemoryWorkspaceLockReleases.set(receipt.entry.owner, receipt);
if (pendingMemoryWorkspaceLockReleases.size > SHORT_TERM_LOCK_MAX_ENTRIES) {
const oldest = pendingMemoryWorkspaceLockReleases.keys().next().value;
if (oldest !== undefined) {
pendingMemoryWorkspaceLockReleases.delete(oldest);
}
}
}
type MemoryWorkspaceLockAcquisitionFailure =
| { kind: "held"; holder: { owner: string; epoch: number } }
| { kind: "store-unavailable"; reason: "storage-error" | "holder-unobserved" };
class MemoryWorkspaceLockAcquisitionError extends Error {
readonly code: "MEMORY_WORKSPACE_LOCK_HELD" | "MEMORY_WORKSPACE_LOCK_STORE_UNAVAILABLE";
constructor(
lockRef: string,
readonly outcome: MemoryWorkspaceLockAcquisitionFailure,
cause?: unknown,
) {
super(
outcome.kind === "held"
? `Timed out waiting for memory workspace lock at ${lockRef} (held by ${outcome.holder.owner})`
: `Memory workspace lock store unavailable at ${lockRef}: ${outcome.reason}`,
{ cause },
);
this.code =
outcome.kind === "held"
? "MEMORY_WORKSPACE_LOCK_HELD"
: "MEMORY_WORKSPACE_LOCK_STORE_UNAVAILABLE";
}
}
type MemoryWorkspaceLease = { key: string; active: boolean };
type MemoryWorkspaceLockScope = {
@ -114,7 +56,7 @@ async function runWorkspaceLockScope<T>(
return await memoryWorkspaceLockScopes.run(scope, task);
} finally {
// Closed async contexts must acquire a new lease. Already accepted children
// finish before the owner releases the cross-process lock.
// finish before the owner admits the next workspace writer.
scope.active = false;
await scope.childTail;
}
@ -130,11 +72,7 @@ function parseLockOwnerPid(raw: string): number | null {
return Number.isInteger(pid) && pid > 0 ? pid : null;
}
export function isShortTermLockStealable(
lockKey: string,
existing: ShortTermLockEntry,
nowMs: number,
): boolean {
export function isShortTermLockStealable(existing: ShortTermLockEntry, nowMs: number): boolean {
if (nowMs - existing.acquiredAt <= SHORT_TERM_LOCK_STALE_MS) {
return false;
}
@ -143,9 +81,8 @@ export function isShortTermLockStealable(
return true;
}
if (ownerPid === process.pid) {
// The current process can own this row only through the tracked local lease.
// A same-PID row without that lease survived a prior process or failed cleanup.
return activeMemoryWorkspaceLockOwners.get(lockKey) !== existing.owner;
// Runtime coordination is process-local; these are legacy diagnostic rows.
return true;
}
if (isPidDefinitelyDead(ownerPid)) {
return true;
@ -220,109 +157,12 @@ export async function withMemoryWorkspaceLock<T>(
);
return await child;
}
const lockRef = resolveLockPath(workspaceDir);
const lockStore = openMemoryCoreStateStore<ShortTermLockEntry>({
namespace: SHORT_TERM_LOCK_NAMESPACE,
maxEntries: SHORT_TERM_LOCK_MAX_ENTRIES,
});
return await inProcessMemoryWorkspaceLocks.enqueue(lockKey, async () => {
const receipt = await acquireMemoryWorkspaceLock(lockStore, lockKey, lockRef);
const lease = { key: lockKey, active: true };
activeMemoryWorkspaceLockOwners.set(lockKey, receipt.entry.owner);
try {
return await runWorkspaceLockScope(lease, task);
} finally {
lease.active = false;
activeMemoryWorkspaceLockOwners.delete(lockKey);
try {
await deleteShortTermLockEntryIfCurrent(
lockStore,
lockKey,
receipt.entry,
receipt.observation,
);
} catch {
retainMemoryWorkspaceLockRelease(receipt);
}
}
});
}
async function acquireMemoryWorkspaceLock(
lockStore: PluginStateKeyedStore<ShortTermLockEntry>,
lockKey: string,
lockRef: string,
): Promise<MemoryWorkspaceLockReceipt> {
const startedAt = Date.now();
while (true) {
let outcome: MemoryWorkspaceLockAcquisitionFailure;
try {
if (!lockStore.observe || !lockStore.compareAndApply) {
throw new Error("memory-core short-term lock store requires atomic comparisons");
}
const observation = await lockStore.observe(lockKey);
let existing = observation.value;
if (existing === undefined) {
const ownerStartTime = getFileLockProcessStartTime(process.pid);
const lockEntry: ShortTermLockEntry = {
owner: `${process.pid}:${randomUUID()}`,
acquiredAt: Date.now(),
...(ownerStartTime === null ? {} : { ownerStartTime }),
};
const receipt = { key: lockKey, entry: lockEntry, observation };
let acquired: PluginStateCompareResult<ShortTermLockEntry>;
try {
acquired = await lockStore.compareAndApply(lockKey, observation.comparison, {
operation: "update",
action: "set",
value: lockEntry,
});
} catch (error) {
// Worker rejection joins native settlement; the task has not started.
retainMemoryWorkspaceLockRelease(receipt);
throw error;
}
if (acquired.status === "applied") {
return receipt;
}
if (acquired.status === "conflict") {
existing = acquired.current.value;
}
} else {
const pending = pendingMemoryWorkspaceLockReleases.get(existing.owner);
if (pending && Date.now() - pending.entry.acquiredAt > SHORT_TERM_LOCK_STALE_MS) {
pendingMemoryWorkspaceLockReleases.delete(existing.owner);
} else if (pending?.key === lockKey && pending.entry.acquiredAt === existing.acquiredAt) {
// The acquisition token fences cleanup to its original physical store.
await deleteShortTermLockEntryIfCurrent(
lockStore,
lockKey,
pending.entry,
pending.observation,
);
pendingMemoryWorkspaceLockReleases.delete(existing.owner);
continue;
}
if (
isShortTermLockStealable(lockKey, existing, Date.now()) &&
(await deleteShortTermLockEntryIfCurrent(lockStore, lockKey, existing, observation))
) {
continue;
}
}
outcome = existing
? { kind: "held", holder: { owner: existing.owner, epoch: existing.acquiredAt } }
: { kind: "store-unavailable", reason: "holder-unobserved" };
} catch (cause) {
throw new MemoryWorkspaceLockAcquisitionError(
lockRef,
{ kind: "store-unavailable", reason: "storage-error" },
cause,
);
}
if (Date.now() - startedAt >= MEMORY_WORKSPACE_LOCK_WAIT_TIMEOUT_MS) {
throw new MemoryWorkspaceLockAcquisitionError(lockRef, outcome);
}
await sleep(MEMORY_WORKSPACE_LOCK_RETRY_DELAY_MS);
}
}

19
extensions/memory-core/src/memory/lifecycle.ts
View file

@ -7,7 +7,6 @@ type ReloadHandle = ReturnType<NonNullable<MemoryPluginRuntime["prepareReload"]>
export type MemoryReloadState = {
retireRuntime: boolean;
adapters: Set<MemoryEmbeddingProviderAdapter>;
generation: number;
};
export type MemoryManagerLifecycle = {
reload?: MemoryReloadState;
@ -34,29 +33,21 @@ export function getMemoryManagerLifecycle(): MemoryManagerLifecycle {
return lifecycle;
}
/** Fence acquisition even before the lazy manager module has loaded. */
/** Pause new acquisition while the plugin owner replaces its adapters. */
export function prepareMemoryManagerReload(
change: ReloadChange,
lifecycle = getMemoryManagerLifecycle(),
): ReloadHandle {
const reload = lifecycle.reload ?? {
retireRuntime: false,
adapters: new Set<MemoryEmbeddingProviderAdapter>(),
generation: 0,
const reload = {
retireRuntime: change.retireRuntime,
adapters: new Set(change.retiringEmbeddingProviders),
};
reload.retireRuntime ||= change.retireRuntime;
for (const adapter of change.retiringEmbeddingProviders) {
reload.adapters.add(adapter);
}
const generation = ++reload.generation;
lifecycle.reload = reload;
const drain = lifecycle.prepare?.(reload) ?? (async () => ({ errors: [] }));
return {
drain,
resume() {
if (lifecycle.reload === reload && generation === reload.generation) {
lifecycle.reload = undefined;
}
lifecycle.reload = undefined;
},
};
}

37
extensions/memory-core/src/memory/manager-captured-session-preparation.test.ts
View file

@ -1,15 +1,7 @@
import type { DatabaseSync } from "node:sqlite";
import { createDeferred } from "openclaw/plugin-sdk/extension-shared";
import { afterEach, describe, expect, it, vi } from "vitest";
import {
memoryCoreWorkspaceStateKey,
openMemoryCoreStateStore,
SHORT_TERM_LOCK_MAX_ENTRIES,
SHORT_TERM_LOCK_NAMESPACE,
} from "../dreaming-state.js";
import { forgetMemoryEntries } from "../memory-forget.js";
import { deleteShortTermLockEntryIfCurrent } from "../memory-workspace-lock.js";
import type { ShortTermLockEntry } from "../short-term-promotion-types.js";
import * as cpu from "./manager-cpu-worker-runtime.js";
import { createManagerIndexFixture } from "./manager-index.test-support.js";
@ -42,35 +34,6 @@ describe("captured session preparation", () => {
return { manager, db, cfg };
}
it("leaves the durable lease available while preparing captured session content", async () => {
const { manager } = await setup();
const store = openMemoryCoreStateStore<ShortTermLockEntry>({
namespace: SHORT_TERM_LOCK_NAMESPACE,
maxEntries: SHORT_TERM_LOCK_MAX_ENTRIES,
});
const key = memoryCoreWorkspaceStateKey(fixture.paths.workspace);
const prepare = cpu.prepareMemoryIndexInWorker;
const sessionClaims: boolean[] = [];
const memoryClaims: boolean[] = [];
vi.spyOn(cpu, "prepareMemoryIndexInWorker").mockImplementation(async (input) => {
const entry = { owner: "independent-writer", acquiredAt: Date.now() };
// The atomic store is the cross-process boundary; bypass only the local queue.
const acquired = await store.registerIfAbsent(key, entry);
(input.source === "sessions" ? sessionClaims : memoryClaims).push(acquired);
if (acquired) {
await deleteShortTermLockEntryIfCurrent(store, key, entry);
}
return prepare(input);
});
await manager.sync({ reason: "cli", force: true });
expect(sessionClaims).toEqual([true]);
expect(memoryClaims.length).toBeGreaterThan(0);
expect(memoryClaims.every((claimed) => !claimed)).toBe(true);
expect(await store.lookup(key)).toBeUndefined();
});
it.each([false, true])(
"does not resurrect a session forgotten between preparation and writing (vectors: %s)",
async (vectorEnabled) => {

58
extensions/memory-core/src/memory/manager-embedding-cache-ops.ts
View file

@ -4,10 +4,7 @@ import {
hasNonTextEmbeddingParts,
type EmbeddingInput,
} from "openclaw/plugin-sdk/memory-core-host-engine-embeddings";
import {
buildFileEntry,
type MemorySource,
} from "openclaw/plugin-sdk/memory-core-host-engine-storage";
import type { MemorySource } from "openclaw/plugin-sdk/memory-core-host-engine-storage";
import { withMemoryWorkspaceLock } from "../memory-workspace-lock.js";
import type { IndexedMemoryChunk } from "./manager-chunk-writer.js";
import {
@ -56,20 +53,6 @@ export abstract class MemoryManagerEmbeddingCacheOps extends MemoryManagerSyncOp
}
}
protected assertEmbeddingCacheGenerationCurrent(
generation: MemorySemanticProviderGeneration,
): void {
if (
this.closed ||
this.syncProviderGeneration !== generation ||
generation.database.closed ||
!generation.database.db.isOpen ||
this.publishedDatabase !== generation.database
) {
throw new Error("Memory embedding generation changed during cache lookup");
}
}
protected async collectCachedEmbeddings(
candidates: MemoryEmbeddingCacheCandidate[],
generation: MemorySemanticProviderGeneration,
@ -84,10 +67,13 @@ export abstract class MemoryManagerEmbeddingCacheOps extends MemoryManagerSyncOp
hashes,
},
},
() => this.assertEmbeddingCacheGenerationCurrent(generation),
() => {
if (!generation.database.db.isOpen) {
throw new Error("Memory database owner is closed");
}
},
)
: new Map<string, number[]>();
this.assertEmbeddingCacheGenerationCurrent(generation);
// Cache hits and new batches must inhabit the same vector space during a sync.
for (const [hash, embedding] of cached) {
if (!isValidMemoryEmbedding(embedding, generation.embeddingDimensions)) {
@ -114,7 +100,6 @@ export abstract class MemoryManagerEmbeddingCacheOps extends MemoryManagerSyncOp
candidates,
generation,
);
this.assertEmbeddingCacheGenerationCurrent(generation);
if (missing.length === 0) {
return embeddings;
@ -253,31 +238,18 @@ export abstract class MemoryManagerEmbeddingCacheOps extends MemoryManagerSyncOp
if (!this.canWriteEmbeddingCache(generation)) {
return;
}
const entryValidity = new Map<MemoryIndexEntry, boolean>();
const accepted: MemoryEmbeddingCacheEntry[] = [];
for (const [index, candidate] of candidates.entries()) {
let valid = entryValidity.get(candidate.entry);
if (valid === undefined) {
if (candidate.source === "memory") {
const current = await (this.memoryFiles?.inspectFile ?? buildFileEntry)(
candidate.entry.absPath,
this.workspaceDir,
this.settings.multimodal,
);
valid = current?.hash === candidate.entry.hash;
} else {
const sessionId = candidate.entry.sessionId;
valid = Boolean(sessionId);
}
entryValidity.set(candidate.entry, valid);
}
if (valid) {
accepted.push({
hash: candidate.chunk.hash,
embedding: embeddings[index] ?? [],
...(candidate.source === "sessions" ? { sessionId: candidate.entry.sessionId } : {}),
});
if (candidate.source === "sessions" && !candidate.entry.sessionId) {
continue;
}
// Changed files may leave unused content-addressed vectors; normal cache eviction
// removes them. The publication worker still checks session tombstones.
accepted.push({
hash: candidate.chunk.hash,
embedding: embeddings[index] ?? [],
...(candidate.source === "sessions" ? { sessionId: candidate.entry.sessionId } : {}),
});
}
if (accepted.length === 0) {
return;

1
extensions/memory-core/src/memory/manager-embedding-ops.ts
View file

@ -203,7 +203,6 @@ export abstract class MemoryManagerEmbeddingOps extends MemoryManagerEmbeddingCa
candidates,
generation,
);
this.assertEmbeddingCacheGenerationCurrent(generation);
if (missing.length === 0) {
return embeddings;
}

138
extensions/memory-core/src/memory/manager-index-generation-lease.async.test.ts
View file

@ -1,138 +0,0 @@
import { createDeferred } from "openclaw/plugin-sdk/extension-shared";
import { beforeEach, describe, expect, it, vi } from "vitest";
import {
acquireMemoryIndexReadGeneration,
withMemoryIndexGeneration,
} from "./manager-index-generation-lease.js";
import type { MemorySqliteLeaseHandle } from "./manager-sqlite-lease.js";
const leases = vi.hoisted(() => ({
acquire: vi.fn<typeof import("./manager-sqlite-lease.js").tryAcquireMemorySqliteLease>(),
acquireWriter: vi.fn<typeof import("./manager-sqlite-lease.js").acquireMemorySqliteWriterLease>(),
}));
vi.mock("./manager-sqlite-lease.js", () => ({
tryAcquireMemorySqliteLease: leases.acquire,
acquireMemorySqliteWriterLease: leases.acquireWriter,
}));
beforeEach(() => {
leases.acquire.mockReset();
leases.acquireWriter.mockReset();
});
describe("memory generation lease cleanup", () => {
it.each([false, true])(
"keeps publication and queued readers waiting for both releases (release failure: %s)",
async (failRelease) => {
const generationStarted = createDeferred<void>();
const generationGate = createDeferred<void>();
const admissionStarted = createDeferred<void>();
const admissionGate = createDeferred<void>();
const releaseError = new Error("generation release failed");
const events: string[] = [];
leases.acquire.mockResolvedValueOnce({
release: async () => {
generationStarted.resolve();
await generationGate.promise;
if (failRelease) {
throw releaseError;
}
},
});
leases.acquire.mockResolvedValue({ release: async () => {} });
leases.acquireWriter.mockResolvedValue({
release: async () => {
admissionStarted.resolve();
await admissionGate.promise;
},
});
const databasePath = `/memory-generation-release-${failRelease}.sqlite`;
const publication = withMemoryIndexGeneration(databasePath, "write", async () => {
events.push("published");
}).then(
() => events.push("released"),
(error: unknown) => {
events.push("release-failed");
return error;
},
);
await generationStarted.promise;
const nextReader = acquireMemoryIndexReadGeneration(databasePath).then(async (release) => {
events.push("next-reader");
await release();
});
try {
await new Promise<void>((resolve) => {
setImmediate(resolve);
});
expect(events).toEqual(["published"]);
expect(leases.acquire).toHaveBeenCalledTimes(1);
generationGate.resolve();
await admissionStarted.promise;
await new Promise<void>((resolve) => {
setImmediate(resolve);
});
expect(events).toEqual(["published"]);
expect(leases.acquire).toHaveBeenCalledTimes(1);
} finally {
generationGate.resolve();
admissionGate.resolve();
await Promise.all([publication, nextReader]);
}
expect(events).toContain("next-reader");
if (failRelease) {
expect(await publication).toBe(releaseError);
} else {
expect(events).toContain("released");
}
},
);
it("drains a lease delivered after cancellation before admitting the next local publisher", async () => {
const acquired = createDeferred<MemorySqliteLeaseHandle>();
const acquisitionStarted = createDeferred<void>();
const releaseStarted = createDeferred<void>();
const releaseGate = createDeferred<void>();
const admissionRelease = vi.fn(async () => {});
const generationRelease = vi.fn(async () => {
releaseStarted.resolve();
await releaseGate.promise;
});
leases.acquire
.mockResolvedValueOnce({ release: admissionRelease })
.mockImplementationOnce(async () => {
acquisitionStarted.resolve();
return acquired.promise;
})
.mockResolvedValue({ release: async () => {} });
const abortReason = new Error("search canceled");
const controller = new AbortController();
const databasePath = "/memory-generation-late-acquisition.sqlite";
const reader = acquireMemoryIndexReadGeneration(databasePath, controller.signal).catch(
(error: unknown) => error,
);
await acquisitionStarted.promise;
controller.abort(abortReason);
acquired.resolve({ release: generationRelease });
await releaseStarted.promise;
let published = false;
leases.acquireWriter.mockResolvedValue({ release: async () => {} });
const nextPublication = withMemoryIndexGeneration(databasePath, "write", async () => {
published = true;
});
try {
await new Promise<void>((resolve) => {
setImmediate(resolve);
});
expect(published).toBe(false);
expect(admissionRelease).not.toHaveBeenCalled();
} finally {
releaseGate.resolve();
await nextPublication;
}
expect(await reader).toMatchObject({ cause: abortReason });
expect(generationRelease).toHaveBeenCalledTimes(1);
expect(admissionRelease).toHaveBeenCalledTimes(1);
expect(published).toBe(true);
});
});

256
extensions/memory-core/src/memory/manager-index-generation-lease.test.ts
View file

@ -1,121 +1,10 @@
// Memory Core tests published-index read and publication ordering.
import { spawn, type ChildProcessWithoutNullStreams } from "node:child_process";
import { once } from "node:events";
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { describe, expect, it, vi } from "vitest";
import {
acquireMemoryIndexReadGeneration,
withMemoryIndexGeneration,
} from "./manager-index-generation-lease.js";
const leaseChildSource = String.raw`
import { once } from "node:events";
import { DatabaseSync } from "node:sqlite";
const [, mode, databasePath] = process.argv;
let reportedContention = false;
const acquire = async (location, leaseMode) => {
while (true) {
const database = new DatabaseSync(location);
database.exec("PRAGMA busy_timeout = 0");
try {
if (leaseMode === "exclusive") {
database.exec("BEGIN EXCLUSIVE");
} else {
database.exec("BEGIN");
database.prepare("SELECT name FROM sqlite_schema LIMIT 1").get();
}
return database;
} catch (error) {
database.close();
if (!/SQLITE_(?:BUSY|LOCKED)|database is locked/iu.test(String(error))) throw error;
if (!reportedContention) {
reportedContention = true;
process.stdout.write("contended\n");
}
await new Promise((resolve) => setTimeout(resolve, 10));
}
}
};
const admission = await acquire(
databasePath + ".generation-writer.sqlite",
mode === "write" ? "exclusive" : "shared",
);
if (mode === "read-admission") {
process.stdout.write("acquired\n");
process.stdin.resume();
await once(process.stdin, "end");
admission.exec("ROLLBACK");
admission.close();
process.exit(0);
}
const generation = await acquire(
databasePath + ".generation-lock.sqlite",
mode === "write" ? "exclusive" : "shared",
);
if (mode === "read") {
admission.exec("ROLLBACK");
admission.close();
}
try {
process.stdout.write("acquired\n");
process.stdin.resume();
await once(process.stdin, "end");
generation.exec("ROLLBACK");
if (mode !== "read") {
admission.exec("ROLLBACK");
}
} finally {
generation.close();
if (mode !== "read") {
admission.close();
}
}
`;
function spawnLeaseFixture(
mode: "read" | "read-admission" | "write" | "mutation",
databasePath: string,
): ChildProcessWithoutNullStreams {
return spawn(
process.execPath,
["--input-type=module", "--eval", leaseChildSource, mode, databasePath],
{
stdio: ["pipe", "pipe", "pipe"],
},
);
}
async function readChildLine(child: ChildProcessWithoutNullStreams): Promise<string> {
const [chunk] = await once(child.stdout, "data");
return String(chunk).trim();
}
async function stopChild(child: ChildProcessWithoutNullStreams): Promise<void> {
if (child.exitCode !== null) {
return;
}
child.stdin.end();
await once(child, "exit");
}
let fixtureRoot = "";
beforeEach(async () => {
fixtureRoot = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-memory-generation-"));
});
afterEach(async () => {
await fs.rm(fixtureRoot, { recursive: true, force: true });
});
function leasePath(name: string): string {
return path.join(fixtureRoot, `${name}.sqlite`);
}
async function withReadGeneration<T>(key: string, run: () => Promise<T>): Promise<T> {
const release = await acquireMemoryIndexReadGeneration(key);
try {
@ -126,47 +15,23 @@ async function withReadGeneration<T>(key: string, run: () => Promise<T>): Promis
}
describe("memory index generation lease", () => {
it("lets existing readers finish while fused retrieval excludes foreign mutations", async () => {
const databasePath = leasePath("fused-retrieval-cross-process");
const releaseOrdinary = await acquireMemoryIndexReadGeneration(databasePath);
const releaseFused = await acquireMemoryIndexReadGeneration(databasePath, undefined, true);
let released = false;
const child = spawnLeaseFixture("mutation", databasePath);
try {
expect(await readChildLine(child)).toBe("contended");
const acquired = readChildLine(child);
await releaseFused();
released = true;
expect(await acquired).toBe("acquired");
} finally {
if (!released) {
await releaseFused();
}
await stopChild(child);
await releaseOrdinary();
}
});
it("admits another reader into the active generation when no writer is queued", async () => {
let releaseFirstReader = () => {};
const firstReaderGate = new Promise<void>((resolve) => {
releaseFirstReader = resolve;
});
const events: string[] = [];
const generationPath = leasePath("shared-reader-generation");
const generationPath = "/memory-test/shared-reader-generation.sqlite";
const firstReader = withReadGeneration(generationPath, async () => {
events.push("first-reader");
await firstReaderGate;
});
await vi.waitFor(() => expect(events).toContain("first-reader"));
const nextReader = withReadGeneration(generationPath, async () => {
events.push("next-reader");
});
await nextReader;
expect(events).toEqual(["first-reader", "next-reader"]);
releaseFirstReader();
await firstReader;
});
@ -177,21 +42,18 @@ describe("memory index generation lease", () => {
releaseFirstReader = resolve;
});
const events: string[] = [];
const generationPath = leasePath("reader-before-publish");
const generationPath = "/memory-test/reader-before-publish.sqlite";
const firstReader = withReadGeneration(generationPath, async () => {
events.push("first-reader-start");
await firstReaderGate;
events.push("first-reader-end");
});
await vi.waitFor(() => expect(events).toContain("first-reader-start"));
const publish = withMemoryIndexGeneration(generationPath, "write", async () => {
events.push("publish");
});
await Promise.resolve();
expect(events).not.toContain("publish");
releaseFirstReader();
await Promise.all([firstReader, publish]);
expect(events).toEqual(["first-reader-start", "first-reader-end", "publish"]);
@ -203,8 +65,7 @@ describe("memory index generation lease", () => {
releaseFirstReader = resolve;
});
const events: string[] = [];
const generationPath = leasePath("publish-before-reader");
const generationPath = "/memory-test/publish-before-reader.sqlite";
const firstReader = withReadGeneration(generationPath, async () => {
events.push("first-reader");
await firstReaderGate;
@ -216,117 +77,8 @@ describe("memory index generation lease", () => {
const nextReader = withReadGeneration(generationPath, async () => {
events.push("next-reader");
});
releaseFirstReader();
await Promise.all([firstReader, publish, nextReader]);
expect(events).toEqual(["first-reader", "publish", "next-reader"]);
});
it("keeps another process from publishing during an active read generation", async () => {
const databasePath = leasePath("reader-cross-process");
const release = await acquireMemoryIndexReadGeneration(databasePath);
let released = false;
const child = spawnLeaseFixture("write", databasePath);
try {
expect(await readChildLine(child)).toBe("contended");
const acquired = readChildLine(child);
await release();
released = true;
expect(await acquired).toBe("acquired");
} finally {
if (!released) {
await release();
}
await stopChild(child);
}
});
it("waits for another process reader before publishing a new generation", async () => {
const databasePath = leasePath("publisher-cross-process");
const child = spawnLeaseFixture("read", databasePath);
try {
expect(await readChildLine(child)).toBe("acquired");
const events: string[] = [];
const publication = withMemoryIndexGeneration(databasePath, "write", async () => {
events.push("published");
});
await new Promise<void>((resolve) => {
setImmediate(resolve);
});
expect(events).toEqual([]);
await stopChild(child);
await publication;
expect(events).toEqual(["published"]);
} finally {
await stopChild(child);
}
});
it("keeps newly arriving process readers behind a waiting publication", async () => {
const databasePath = leasePath("writer-intent-cross-process");
const firstReader = spawnLeaseFixture("read-admission", databasePath);
let nextReader: ChildProcessWithoutNullStreams | undefined;
let publication: Promise<void> | undefined;
try {
expect(await readChildLine(firstReader)).toBe("acquired");
const events: string[] = [];
publication = withMemoryIndexGeneration(databasePath, "write", async () => {
events.push("published");
});
await new Promise<void>((resolve) => {
setTimeout(resolve, 50);
});
nextReader = spawnLeaseFixture("read", databasePath);
expect(await readChildLine(nextReader)).toBe("contended");
const nextReaderAcquired = readChildLine(nextReader).then((line) => {
expect(line).toBe("acquired");
events.push("next-reader");
});
expect(events).toEqual([]);
await stopChild(firstReader);
// Publication releases its lease before resolving, so the reader may already be done.
await publication;
await nextReaderAcquired;
expect(events).toEqual(["published", "next-reader"]);
} finally {
await stopChild(firstReader);
if (nextReader) {
await stopChild(nextReader);
}
await publication;
}
});
it("removes an aborted reader while another process holds publication", async () => {
const databasePath = leasePath("cancelled-reader-cross-process");
const publisher = spawnLeaseFixture("write", databasePath);
const controller = new AbortController();
const abortError = new Error("memory search cancelled while waiting for publication");
let pendingReader: ReturnType<typeof acquireMemoryIndexReadGeneration> | undefined;
try {
expect(await readChildLine(publisher)).toBe("acquired");
pendingReader = acquireMemoryIndexReadGeneration(databasePath, controller.signal);
await new Promise<void>((resolve) => {
setTimeout(resolve, 50);
});
controller.abort(abortError);
const outcome = await Promise.race([
pendingReader.then(
() => "acquired" as const,
(error: unknown) => error,
),
new Promise<"timeout">((resolve) => {
setTimeout(() => resolve("timeout"), 250);
}),
]);
expect(outcome).toMatchObject({ cause: abortError });
} finally {
await stopChild(publisher);
const release = await pendingReader?.catch(() => undefined);
await release?.();
}
});
});

113
extensions/memory-core/src/memory/manager-index-generation-lease.ts
View file

@ -1,16 +1,10 @@
// Memory Core coordinates published-index readers with atomic shadow publication.
import { resolveUserPath } from "openclaw/plugin-sdk/memory-core-host-engine-fs";
import { sleepWithAbort } from "openclaw/plugin-sdk/runtime-env";
import {
acquireMemorySqliteWriterLease,
tryAcquireMemorySqliteLease,
type MemorySqliteLeaseHandle,
} from "./manager-sqlite-lease.js";
type Waiter = {
kind: "read" | "write";
resolve: (release: () => void) => void;
};
type GenerationLeaseKind = Waiter["kind"] | "mutation" | "retrieval";
type GenerationLeaseKind = Waiter["kind"] | "mutation";
type GenerationLeaseState = {
readers: number;
@ -19,7 +13,6 @@ type GenerationLeaseState = {
};
const states = new Map<string, GenerationLeaseState>();
const CROSS_PROCESS_RETRY_DELAY_MS = 25;
function createGenerationLeaseAbortError(signal?: AbortSignal): Error {
return new Error("Memory index generation lease acquisition aborted", { cause: signal?.reason });
@ -31,82 +24,6 @@ function throwIfGenerationLeaseAborted(signal?: AbortSignal): void {
}
}
async function acquireCrossProcessLease(
databasePath: string,
kind: GenerationLeaseKind,
signal?: AbortSignal,
): Promise<MemorySqliteLeaseHandle> {
const acquireSqliteLease = async (
location: string,
mode: "shared" | "exclusive",
): Promise<MemorySqliteLeaseHandle> => {
while (true) {
throwIfGenerationLeaseAborted(signal);
const lease = await tryAcquireMemorySqliteLease(location, mode);
if (lease) {
if (signal?.aborted) {
await lease.release();
throw createGenerationLeaseAbortError(signal);
}
return lease;
}
try {
await sleepWithAbort(CROSS_PROCESS_RETRY_DELAY_MS, signal);
} catch (err) {
throw signal?.aborted ? createGenerationLeaseAbortError(signal) : err;
}
}
};
// Admission prevents new readers from overtaking a waiting publisher across
// processes. Readers release it as soon as their generation lease is secured.
const admissionLocation = `${databasePath}.generation-writer.sqlite`;
let admission: MemorySqliteLeaseHandle;
try {
admission =
kind === "write"
? await acquireMemorySqliteWriterLease(admissionLocation, signal)
: await acquireSqliteLease(
admissionLocation,
kind === "retrieval" ? "exclusive" : "shared",
);
} catch (err) {
throw signal?.aborted ? createGenerationLeaseAbortError(signal) : err;
}
let generation: MemorySqliteLeaseHandle;
try {
generation = await acquireSqliteLease(
`${databasePath}.generation-lock.sqlite`,
kind === "write" ? "exclusive" : "shared",
);
} catch (err) {
await admission.release();
throw err;
}
if (kind === "read") {
try {
await admission.release();
} catch (err) {
await generation.release();
throw err;
}
if (signal?.aborted) {
await generation.release();
throw createGenerationLeaseAbortError(signal);
}
return generation;
}
return {
release: async () => {
try {
await generation.release();
} finally {
await admission.release();
}
},
};
}
function stateFor(key: string): GenerationLeaseState {
const existing = states.get(key);
if (existing) {
@ -201,36 +118,16 @@ async function acquire(
signal?: AbortSignal,
): Promise<() => Promise<void>> {
const key = resolveUserPath(databasePath);
const releaseLocal = await acquireLocal(key, kind === "write" ? "write" : "read", signal);
let crossProcess: MemorySqliteLeaseHandle;
try {
throwIfGenerationLeaseAborted(signal);
crossProcess = await acquireCrossProcessLease(key, kind, signal);
if (signal?.aborted) {
await crossProcess.release();
throw createGenerationLeaseAbortError(signal);
}
} catch (err) {
releaseLocal();
throw err;
}
return async () => {
try {
await crossProcess.release();
} finally {
releaseLocal();
}
};
const release = await acquireLocal(key, kind === "write" ? "write" : "read", signal);
return async () => release();
}
export async function acquireMemoryIndexReadGeneration(
databasePath: string,
signal?: AbortSignal,
excludeMutations = false,
): Promise<() => Promise<void>> {
// Fused reads retain admission exclusively through acceptance, while existing
// ordinary readers keep their shared generation and can finish independently.
return await acquire(databasePath, excludeMutations ? "retrieval" : "read", signal);
// One Gateway owns this database; simultaneous foreign processes are unsupported.
return await acquire(databasePath, "read", signal);
}
export async function withMemoryIndexGeneration<T>(

5
extensions/memory-core/src/memory/manager-memory-source-race.test.ts
View file

@ -101,11 +101,6 @@ describe("memory source changes during indexing", () => {
.join("\n");
expect(indexedText()).toContain("Updated beta sibling");
expect(indexedText()).not.toContain("Obsolete alpha");
expect(
db
.prepare("SELECT hash FROM memory_embedding_cache WHERE hash = ?")
.get(hashText(obsoleteContent)),
).toBeUndefined();
expect(manager.status().dirty).toBe(true);
expect(Reflect.get(manager, "memoryFullRetryDirty")).toBe(false);

1
extensions/memory-core/src/memory/manager-provider-lifecycle.ts
View file

@ -502,7 +502,6 @@ export abstract class MemoryProviderLifecycle extends MemoryManagerEmbeddingOps
try {
await this.awaitProviderIdle(pendingProvider);
await pendingProvider.close?.();
this.releaseProvider(pendingProvider);
this.providersPendingRetirement.delete(pendingProvider);
} catch (err) {
if (!closeFailed) {

63
extensions/memory-core/src/memory/manager-registry.reload.test.ts
View file

@ -92,69 +92,6 @@ it("joins pending manager cleanup across overlapping reload drains", async () =>
}
});
it("owns failed late creation cleanup until explicit close", async () => {
const lifecycle: MemoryManagerLifecycle = {};
const registry = new MemoryManagerRegistry(lifecycle);
const entered = createDeferred<void>();
const released = createDeferred<void>();
const failure = new Error("late manager close failed");
const late = { close: vi.fn().mockRejectedValueOnce(failure).mockResolvedValue(undefined) };
const replacement = { close: vi.fn(async () => {}) };
const transient = { close: vi.fn(async () => {}) };
const pending = registry.acquire(
{ agentId: "main", purpose: "default" },
{
prepare: () => ({
key: "main:late:default",
reuse: () => true,
create: async () => {
entered.resolve();
await released.promise;
return late;
},
}),
},
);
const observed = expect(pending).rejects.toBe(failure);
await entered.promise;
const retirement = prepareMemoryManagerReload(
{ retireRuntime: true, retiringEmbeddingProviders: [] },
lifecycle,
);
const draining = retirement.drain();
released.resolve();
try {
await observed;
await draining;
} finally {
released.resolve();
retirement.resume();
}
expect.soft(registry.canPublishProbe(late)).toBe(false);
expect(late.close).toHaveBeenCalledOnce();
await registry.acquire(
{ agentId: "main", purpose: "default" },
{
prepare: () => ({ key: "main:late:default", reuse: () => true, create: () => replacement }),
},
);
await registry.acquire(
{ agentId: "main", purpose: "status" },
{
prepare: () => ({ key: "main:status", reuse: () => true, create: () => transient }),
},
);
expect(late.close).toHaveBeenCalledOnce();
await registry.closeAll();
expect(late.close).toHaveBeenCalledTimes(2);
expect(replacement.close).toHaveBeenCalledOnce();
expect(transient.close).not.toHaveBeenCalled();
await registry.closeAll();
expect(late.close).toHaveBeenCalledTimes(2);
expect(replacement.close).toHaveBeenCalledOnce();
await transient.close();
});
it("fences acquisition when reload starts before the manager owner initializes", async () => {
const lifecycle: MemoryManagerLifecycle = {};
const reload = prepareMemoryManagerReload(

71
extensions/memory-core/src/memory/manager-registry.ts
View file

@ -2,7 +2,6 @@
import { toErrorObject } from "openclaw/plugin-sdk/error-runtime";
import { enqueueKeyedTask } from "openclaw/plugin-sdk/keyed-async-queue";
import type {
MemoryEmbeddingProvider,
MemoryEmbeddingProviderAdapter,
MemoryEmbeddingProviderCreateResult,
} from "openclaw/plugin-sdk/memory-core-host-engine-embeddings";
@ -81,9 +80,7 @@ export type MemoryManagerProviderFactory = (
type ManagerOwnership = {
key: string;
pending: Map<object, MemoryEmbeddingProviderAdapter>;
providers: Map<MemoryEmbeddingProvider, MemoryEmbeddingProviderAdapter>;
failedAdapters: Set<MemoryEmbeddingProviderAdapter>;
adapters: Set<MemoryEmbeddingProviderAdapter>;
retiring: boolean;
};
@ -106,9 +103,7 @@ export class MemoryManagerRegistry<T extends ClosableMemoryManager> {
if (!owner) {
owner = {
key,
pending: new Map(),
providers: new Map(),
failedAdapters: new Set(),
adapters: new Set(),
retiring: false,
};
this.managers.set(manager, owner);
@ -128,25 +123,9 @@ export class MemoryManagerRegistry<T extends ClosableMemoryManager> {
if (owner.retiring || this.reload?.retireRuntime || this.reload?.adapters.has(adapter)) {
throw new MemoryManagerReloadError();
}
const acquisition = {};
// Record the exact adapter before creation can yield; late results belong to
// this manager's close, including when replacement began during creation.
owner.pending.set(acquisition, adapter);
try {
const result = await create();
if (result.provider) {
owner.providers.set(result.provider, adapter);
owner.failedAdapters.delete(adapter);
} else {
owner.failedAdapters.add(adapter);
}
return result;
} catch (error) {
owner.failedAdapters.add(adapter);
throw error;
} finally {
owner.pending.delete(acquisition);
}
// Remember attempted adapters, including a primary that fell back.
owner.adapters.add(adapter);
return await create();
}
canPublishProbe(manager: T): boolean {
@ -156,19 +135,7 @@ export class MemoryManagerRegistry<T extends ClosableMemoryManager> {
getProbeOwners(manager: T): readonly MemoryEmbeddingProviderAdapter[] {
const owner = this.managers.get(manager);
return owner
? [
...new Set([
...owner.pending.values(),
...owner.providers.values(),
...owner.failedAdapters,
]),
]
: [];
}
releaseProvider(manager: T, provider: MemoryEmbeddingProvider): void {
this.managers.get(manager)?.providers.delete(provider);
return owner ? [...owner.adapters] : [];
}
private prepareManagersForReload(reload: MemoryReloadState) {
@ -221,27 +188,13 @@ export class MemoryManagerRegistry<T extends ClosableMemoryManager> {
throw new MemoryManagerReloadError();
}
return await this.runScopeOperation(params, async () => {
if (this.reload?.retireRuntime) {
throw new MemoryManagerReloadError();
}
const prepared = await callbacks.prepare();
if (!prepared) {
return null;
}
if (this.reload?.retireRuntime) {
throw new MemoryManagerReloadError();
}
const create = async () => {
if (this.reload?.retireRuntime) {
throw new MemoryManagerReloadError();
}
const manager = await prepared.create();
const owner = this.track(manager, prepared.key);
if (this.reload?.retireRuntime) {
owner.retiring = true;
await this.closeEntries([[prepared.key, manager]]);
throw new MemoryManagerReloadError();
}
this.track(manager, prepared.key);
return manager;
};
if (params.purpose !== "default") {
@ -256,16 +209,6 @@ export class MemoryManagerRegistry<T extends ClosableMemoryManager> {
// The scope queue already serializes creation and replacement for this agent.
const existing = this.cache.get(prepared.key);
if (existing) {
// Other sidecars may drain between preparation and memory cleanup.
// Recheck after the scope await without discarding the manager needed for rollback.
const reload = this.reload;
if (
reload &&
(reload.retireRuntime ||
this.getProbeOwners(existing).some((adapter) => reload.adapters.has(adapter)))
) {
throw new MemoryManagerReloadError();
}
return existing;
}
const manager = await create();

38
extensions/memory-core/src/memory/manager-reindex-lock.ts
View file

@ -1,11 +1,12 @@
// Memory Core plugin module serializes full memory reindex builds across processes.
import {
acquireMemorySqliteWriterLease,
type MemorySqliteLeaseHandle,
} from "./manager-sqlite-lease.js";
// Memory Core serializes builds and reset within the owning Gateway.
import { createDeferred } from "openclaw/plugin-sdk/concurrency-runtime";
import { enqueueKeyedTask } from "openclaw/plugin-sdk/keyed-async-queue";
import { resolveUserPath } from "openclaw/plugin-sdk/memory-core-host-engine-fs";
import { racePromiseWithAbortSignal } from "openclaw/plugin-sdk/time-runtime";
export type MemoryReindexLockHandle = MemorySqliteLeaseHandle;
export type MemoryReindexLockHandle = { release: () => Promise<void> };
const builds = new Map<string, Promise<void>>();
const REINDEX_LOCK_WAIT_TIMEOUT_MS = 2_000;
function createMemoryReindexBusyError(lockPath: string): Error & { code: string } {
return Object.assign(
@ -14,19 +15,40 @@ function createMemoryReindexBusyError(lockPath: string): Error & { code: string
);
}
/** Wait asynchronously for the exclusive build lock without blocking the Node event loop. */
/** Old on-disk coordination files are inert; foreign writers must stop the Gateway. */
export async function waitForMemoryReindexLock(
dbPath: string,
options: { waitForActive?: boolean } = {},
): Promise<MemoryReindexLockHandle> {
const lockPath = `${dbPath}.reindex-lock.sqlite`;
const entered = createDeferred();
const released = createDeferred();
const completed = enqueueKeyedTask({
tails: builds,
key: resolveUserPath(dbPath),
task: async () => {
entered.resolve();
await released.promise;
},
});
// Reset refuses a busy index; admitted sync work waits for its writer to settle.
const timeout = options.waitForActive
? undefined
: AbortSignal.timeout(REINDEX_LOCK_WAIT_TIMEOUT_MS);
try {
return await acquireMemorySqliteWriterLease(lockPath, timeout);
if (timeout) {
await racePromiseWithAbortSignal(entered.promise, timeout);
} else {
await entered.promise;
}
return {
release: async () => {
released.resolve();
await completed;
},
};
} catch (error) {
released.resolve();
if (timeout?.aborted) {
throw createMemoryReindexBusyError(lockPath);
}

1
extensions/memory-core/src/memory/manager-search-orchestration.ts
View file

@ -158,7 +158,6 @@ export abstract class MemorySearchOrchestration extends MemoryKeywordRetrieval {
releaseGeneration ??= await acquireMemoryIndexReadGeneration(
this.settings.store.databasePath,
opts?.signal,
fuseRecallMetadata,
);
assertReadOwner();
preparedKeyword = undefined;

124
extensions/memory-core/src/memory/manager-sqlite-lease.ts
View file

@ -1,124 +0,0 @@
// Memory Core owns crash-safe SQLite coordination leases shared across processes.
import type { DatabaseSync } from "node:sqlite";
import { extractErrorCode, toErrorObject } from "openclaw/plugin-sdk/error-runtime";
import { sleepWithAbort } from "openclaw/plugin-sdk/runtime-env";
import { openNodeSqliteDatabase } from "openclaw/plugin-sdk/sqlite-runtime";
export type MemorySqliteLeaseHandle = {
release: () => Promise<void>;
};
const MEMORY_SQLITE_LEASE_RETRY_DELAY_MS = 25;
function isSqliteBusyError(err: unknown): boolean {
const code = extractErrorCode(err);
if (code === "SQLITE_BUSY" || code === "SQLITE_LOCKED") {
return true;
}
const message = err instanceof Error ? err.message : String(err);
return /SQLITE_(?:BUSY|LOCKED)|database is locked/iu.test(message);
}
function openMemoryLeaseDatabase(location: string): DatabaseSync {
const database = openNodeSqliteDatabase(location);
try {
database.exec("PRAGMA busy_timeout = 0");
return database;
} catch (err) {
database.close();
throw err;
}
}
function createMemorySqliteLeaseHandle(
database: DatabaseSync,
transactionActive: boolean,
): MemorySqliteLeaseHandle {
return {
release: async () => {
let releaseError: unknown;
if (transactionActive) {
try {
database.exec("ROLLBACK");
} catch (err) {
releaseError = err;
}
}
try {
database.close();
} catch (err) {
releaseError ??= err;
}
if (releaseError) {
throw toErrorObject(releaseError, "Failed to release memory SQLite lease");
}
},
};
}
export async function tryAcquireMemorySqliteLease(
location: string,
mode: "shared" | "exclusive",
): Promise<MemorySqliteLeaseHandle | undefined> {
const database = openMemoryLeaseDatabase(location);
try {
if (mode === "exclusive") {
database.exec("BEGIN EXCLUSIVE");
} else {
database.exec("BEGIN");
// BEGIN is deferred. Reading sqlite_schema acquires the shared lock without
// requiring a coordination table or touching the live memory database.
database.prepare("SELECT name FROM sqlite_schema LIMIT 1").get();
}
} catch (err) {
database.close();
if (isSqliteBusyError(err)) {
return undefined;
}
throw err;
}
return createMemorySqliteLeaseHandle(database, true);
}
/** Acquire writer admission without dropping PENDING intent after the first reader collision. */
export async function acquireMemorySqliteWriterLease(
location: string,
signal?: AbortSignal,
): Promise<MemorySqliteLeaseHandle> {
while (true) {
signal?.throwIfAborted();
const database = openMemoryLeaseDatabase(location);
try {
database.exec("PRAGMA locking_mode = EXCLUSIVE");
database.exec("BEGIN IMMEDIATE");
// Rewriting the unchanged header field creates a schema-free write transaction.
// A blocked commit retains SQLite's PENDING lock so later readers cannot overtake it.
database.exec("PRAGMA user_version = 0");
} catch (err) {
database.close();
if (!isSqliteBusyError(err)) {
throw err;
}
await sleepWithAbort(MEMORY_SQLITE_LEASE_RETRY_DELAY_MS, signal);
continue;
}
while (true) {
try {
database.exec("COMMIT");
return createMemorySqliteLeaseHandle(database, false);
} catch (err) {
if (!isSqliteBusyError(err)) {
await createMemorySqliteLeaseHandle(database, true).release();
throw err;
}
try {
await sleepWithAbort(MEMORY_SQLITE_LEASE_RETRY_DELAY_MS, signal);
} catch (sleepError) {
await createMemorySqliteLeaseHandle(database, true).release();
throw sleepError;
}
}
}
}
}

3
extensions/memory-core/src/memory/manager-sync-ops.interval.test.ts
View file

@ -31,9 +31,6 @@ class IntervalSyncHarness extends MemorySyncTestHarness {
protected readonly createProvider = (): never => {
throw new Error("Interval harness does not acquire embedding providers");
};
protected releaseProvider(): never {
throw new Error("Interval harness does not own embedding providers");
}
protected readonly cfg = {} as OpenClawConfig;
protected readonly agentId = "main";
protected readonly workspaceDir = "/tmp/openclaw-memory-interval-test";

3
extensions/memory-core/src/memory/manager-sync-ops.startup-catchup.test-support.ts
View file

@ -104,9 +104,6 @@ export class SessionStartupCatchupHarness extends MemorySyncTestHarness {
protected readonly createProvider = (): never => {
throw new Error("Startup catch-up harness does not acquire embedding providers");
};
protected releaseProvider(): never {
throw new Error("Startup catch-up harness does not own embedding providers");
}
protected readonly cfg = {} as OpenClawConfig;
protected readonly agentId = "main";
protected readonly workspaceDir = "/tmp/openclaw-test-workspace";

1
extensions/memory-core/src/memory/manager-sync-ops.ts
View file

@ -97,7 +97,6 @@ export abstract class MemoryManagerSyncOps extends MemoryManagerSourceSyncOps {
}
protected abstract readonly createProvider: MemoryManagerProviderFactory;
protected abstract releaseProvider(provider: EmbeddingProvider): void;
protected fallbackProviderInitPromise: Promise<boolean> | null = null;
protected syncProviderGeneration: MemorySyncProviderGeneration | null = null;

3
extensions/memory-core/src/memory/manager-sync-yield.test.ts
View file

@ -105,9 +105,6 @@ class SessionSyncYieldHarness extends MemorySyncTestHarness {
protected readonly createProvider = (): never => {
throw new Error("Sync yield harness does not acquire embedding providers");
};
protected releaseProvider(): never {
throw new Error("Sync yield harness does not own embedding providers");
}
protected readonly cfg = {} as OpenClawConfig;
protected readonly agentId = "main";
protected readonly workspaceDir = "/tmp/openclaw-test-workspace";

15
extensions/memory-core/src/memory/manager-worker-reads.test.ts
View file

@ -9,7 +9,6 @@ import {
resolveOpenClawAgentSqlitePath,
} from "openclaw/plugin-sdk/sqlite-runtime";
import {
closeOpenClawAgentDatabasesAsync,
closeOpenClawAgentDatabasesForTest,
observeHostDataSql,
} from "openclaw/plugin-sdk/sqlite-runtime-testing";
@ -128,7 +127,7 @@ describe("memory manager retained worker reads", () => {
}
});
it.each(["ready", "rejected", "revoked"] as const)(
it.each(["ready", "rejected"] as const)(
"waits for a %s cache read before requesting embeddings",
async (outcome) => {
const memoryPath = path.join(fixture.paths.memory, "2026-01-12.md");
@ -176,24 +175,14 @@ describe("memory manager retained worker reads", () => {
expect(
publishedDb.prepare("SELECT text FROM memory_index_chunks ORDER BY id").all(),
).toEqual(before);
if (outcome === "revoked") {
closeOpenClawAgentDatabasesForTest();
}
release.resolve();
if (outcome === "ready") {
await sync;
expect(fixture.provider.embeddedBatchTexts).toEqual([replacement]);
} else {
await expect(sync).rejects.toThrow(
outcome === "rejected"
? "controlled cache read rejection"
: "Memory embedding generation changed during cache lookup",
);
await expect(sync).rejects.toThrow("controlled cache read rejection");
expect(fixture.provider.embeddedBatchTexts).toEqual([]);
}
if (outcome === "revoked") {
await closeOpenClawAgentDatabasesAsync();
}
const current = openOpenClawAgentDatabase({ agentId: "main" }).db;
expect(current.prepare("SELECT text FROM memory_index_chunks ORDER BY id").all()).toEqual(
outcome === "ready" ? [{ text: replacement }] : before,

300
extensions/memory-core/src/memory/manager-workspace-release.stress.test.ts
View file

@ -1,274 +1,36 @@
import { spawn, type ChildProcessWithoutNullStreams } from "node:child_process";
import fs from "node:fs/promises";
import path from "node:path";
import { createInterface } from "node:readline";
import type { DatabaseSync } from "node:sqlite";
import { createDeferred } from "openclaw/plugin-sdk/extension-shared";
import { openOpenClawStateDatabase } from "openclaw/plugin-sdk/plugin-state-test-runtime";
import { closeOpenClawStateDatabaseByPathAsync } from "openclaw/plugin-sdk/sqlite-runtime-testing";
import { describe, expect, it, vi } from "vitest";
import {
memoryCoreWorkspaceStateKey,
openMemoryCoreStateStore,
SHORT_TERM_LOCK_MAX_ENTRIES,
SHORT_TERM_LOCK_NAMESPACE,
} from "../dreaming-state.js";
import {
deleteShortTermLockEntryIfCurrent,
withMemoryWorkspaceLock,
} from "../memory-workspace-lock.js";
import type { ShortTermLockEntry } from "../short-term-promotion-types.js";
import { configureMemoryCoreDreamingStateForTests } from "../test-helpers.js";
import * as cpu from "./manager-cpu-worker-runtime.js";
import { createManagerIndexFixture } from "./manager-index.test-support.js";
import { expect, it } from "vitest";
import { withMemoryWorkspaceLock } from "../memory-workspace-lock.js";
const { closeAllMemorySearchManagers, getMemorySearchManager } = await import("./index.js");
const openLocks = () =>
openMemoryCoreStateStore<ShortTermLockEntry>({
namespace: SHORT_TERM_LOCK_NAMESPACE,
maxEntries: SHORT_TERM_LOCK_MAX_ENTRIES,
});
describe("memory workspace release recovery", () => {
const fixture = createManagerIndexFixture({
getMemorySearchManager,
closeAllMemorySearchManagers,
});
async function setupIndex() {
const cfg = fixture.createConfig({
provider: "none",
sources: ["memory"],
vectorEnabled: false,
it("settles accepted child writes before a failed owner releases the next writer", async () => {
const workspace = "/memory-workspace-test/failed-owner";
const entered = createDeferred<void>();
const release = createDeferred<void>();
const events: string[] = [];
const failure = new Error("owner failed after accepting a child write");
let child: Promise<void> | undefined;
const owner = withMemoryWorkspaceLock(workspace, async () => {
child = withMemoryWorkspaceLock(workspace, async () => {
entered.resolve();
await release.promise;
events.push("child committed");
});
const manager = await fixture.getFreshManager(cfg, "cli");
await manager.sync({ reason: "cli", force: true });
const database = Reflect.get(manager, "db") as DatabaseSync;
const readPublished = () =>
database.prepare("SELECT path, text FROM memory_index_chunks ORDER BY path, id").all();
return { cfg, manager, database, readPublished };
await entered.promise;
throw failure;
}).catch((error: unknown) => {
return error;
});
await entered.promise;
const next = withMemoryWorkspaceLock(workspace, async () => {
events.push("next writer");
});
try {
release.resolve();
expect(await owner).toBe(failure);
await next;
expect(events).toEqual(["child committed", "next writer"]);
} finally {
release.resolve();
await Promise.allSettled([owner, next, child]);
}
it("recovers the next sync after a settled SQLite lease-release failure", async () => {
const { cfg, manager, database, readPublished } = await setupIndex();
const publishedBeforeFailure = readPublished();
expect(publishedBeforeFailure.length).toBeGreaterThan(0);
const state = openOpenClawStateDatabase();
const key = memoryCoreWorkspaceStateKey(fixture.paths.workspace);
const locks = openLocks();
await fs.writeFile(
path.join(fixture.paths.memory, "2026-01-12.md"),
"# Log\nCerulean corrected memory survives recovery.\n",
);
state.db.exec(`
CREATE TRIGGER fail_memory_workspace_release BEFORE DELETE ON plugin_state_entries
WHEN OLD.plugin_id = 'memory-core' AND OLD.namespace = 'short-term-locks'
BEGIN SELECT RAISE(ABORT, 'injected workspace cleanup failure'); END;
`);
try {
await expect(manager.sync({ reason: "cli", force: true })).rejects.toBeInstanceOf(Error);
expect(readPublished()).toEqual(publishedBeforeFailure);
state.db.exec("DROP TRIGGER fail_memory_workspace_release");
const orphan = await locks.lookup(key);
expect(orphan).toBeDefined();
await expect(manager.sync({ reason: "cli", force: true })).resolves.toBeUndefined();
expect(readPublished()).toEqual([
{
path: "memory/2026-01-12.md",
text: expect.stringContaining("Cerulean corrected memory survives recovery."),
},
]);
expect(await locks.lookup(key)).toBeUndefined();
expect(database.prepare("PRAGMA integrity_check").get()).toEqual({ integrity_check: "ok" });
await manager.close();
const reopened = await fixture.getFreshManager(cfg, "cli");
expect((await reopened.search("Cerulean")).length).toBeGreaterThan(0);
} finally {
state.db.exec("DROP TRIGGER IF EXISTS fail_memory_workspace_release");
const orphan = await locks.lookup(key);
if (orphan) {
await deleteShortTermLockEntryIfCurrent(locks, key, orphan);
}
}
});
it("recovers after a real external state writer exceeds lease cleanup's SQLite budget", async () => {
const { cfg, manager, database, readPublished } = await setupIndex();
const original = readPublished();
await manager.sync({ reason: "cli", force: true });
expect(readPublished()).toEqual(original);
const state = openOpenClawStateDatabase();
const key = memoryCoreWorkspaceStateKey(fixture.paths.workspace);
const locks = openLocks();
expect(await locks.lookup(key)).toBeUndefined();
await fs.writeFile(
path.join(fixture.paths.memory, "2026-01-12.md"),
"# Log\nCerulean updated memory after external contention.\n",
);
let writer: ChildProcessWithoutNullStreams | undefined;
let writerExited: Promise<number | null> | undefined;
let writerStderr = "";
const writerEvents: string[] = [];
const prepare = cpu.prepareMemoryIndexInWorker;
const gate = vi.spyOn(cpu, "prepareMemoryIndexInWorker").mockImplementation(async (input) => {
const result = await prepare(input);
if (writer || input.source !== "memory") {
return result;
}
const held = createDeferred<void>();
writer = spawn(
process.execPath,
[
"--input-type=module",
"--eval",
`
import { DatabaseSync } from "node:sqlite";
const db = new DatabaseSync(process.argv[1]);
db.exec("PRAGMA busy_timeout = 0; BEGIN IMMEDIATE");
process.stdout.write("held\\n");
const release = () => {
db.exec("ROLLBACK");
db.close();
process.stdout.write("released\\n");
process.exit(0);
};
const timer = setTimeout(release, 6500);
process.stdin.on("end", () => { clearTimeout(timer); release(); });
process.stdin.resume();
`,
path.join(fixture.paths.stateDir, "state", "openclaw.sqlite"),
],
{ stdio: ["pipe", "pipe", "pipe"] },
);
writer.stderr.setEncoding("utf8");
createInterface({ input: writer.stdout }).on("line", (line) => {
writerEvents.push(line);
if (line === "held") {
held.resolve();
}
});
writer.stderr.on("data", (chunk: string) => {
writerStderr += chunk;
});
writer.once("error", held.reject);
writerExited = new Promise((resolve) => {
writer!.once("close", (code) => {
held.reject(new Error(`State contention child exited ${code}: ${writerStderr}`));
resolve(code);
});
});
await held.promise;
return result;
});
try {
const initial = await manager.sync({ reason: "cli", force: true }).then(
() => undefined,
(error: unknown) => error,
);
expect(await writerExited).toBe(0);
expect(writerEvents).toEqual(["held", "released"]);
gate.mockRestore();
const expectedPublication = [
{
path: "memory/2026-01-12.md",
text: expect.stringContaining("Cerulean updated memory after external contention."),
},
];
if (initial === undefined) {
expect(readPublished()).toEqual(expectedPublication);
expect(await locks.lookup(key)).toBeUndefined();
} else {
expect(initial).toBeInstanceOf(Error);
expect(readPublished()).toEqual(original);
}
expect(state.db.prepare("PRAGMA integrity_check").get()).toEqual({ integrity_check: "ok" });
expect(database.prepare("PRAGMA integrity_check").get()).toEqual({ integrity_check: "ok" });
await expect(manager.sync({ reason: "cli", force: true })).resolves.toBeUndefined();
expect(readPublished()).toEqual(expectedPublication);
expect(await locks.lookup(key)).toBeUndefined();
await manager.close();
const reopened = await fixture.getFreshManager(cfg, "cli");
expect((await reopened.search("Cerulean")).length).toBeGreaterThan(0);
} finally {
gate.mockRestore();
if (writer && writer.exitCode === null) {
writer.stdin.end();
}
await writerExited;
const orphan = await locks.lookup(key);
if (orphan) {
await deleteShortTermLockEntryIfCurrent(locks, key, orphan);
}
}
});
it("keeps a completed receipt bound to its original physical state database", async () => {
const workspace = fixture.paths.workspace;
const key = memoryCoreWorkspaceStateKey(workspace);
const originalState = openOpenClawStateDatabase();
const originalLocks = openLocks();
const otherEnv = {
...process.env,
OPENCLAW_STATE_DIR: path.join(fixture.paths.root, "other-state"),
};
let otherLocks: ReturnType<typeof openLocks> | undefined;
try {
await withMemoryWorkspaceLock(workspace, async () => {
originalState.db.exec(`
CREATE TRIGGER fail_memory_workspace_release BEFORE DELETE ON plugin_state_entries
WHEN OLD.plugin_id = 'memory-core' AND OLD.namespace = 'short-term-locks'
BEGIN SELECT RAISE(ABORT, 'injected workspace cleanup failure'); END;
`);
});
originalState.db.exec("DROP TRIGGER fail_memory_workspace_release");
const completed = await originalLocks.lookup(key);
expect(completed).toBeDefined();
if (!completed) {
throw new Error("Expected the retained completed lease");
}
await configureMemoryCoreDreamingStateForTests(otherEnv);
otherLocks = openLocks();
await otherLocks.register(key, completed);
const otherTask = vi.fn(async () => "must not run");
await expect(withMemoryWorkspaceLock(workspace, otherTask)).rejects.toMatchObject({
code: "MEMORY_WORKSPACE_LOCK_STORE_UNAVAILABLE",
outcome: { kind: "store-unavailable", reason: "storage-error" },
cause: { code: "PLUGIN_STATE_INVALID_INPUT" },
});
expect(otherTask).not.toHaveBeenCalled();
expect(await otherLocks.lookup(key)).toEqual(completed);
expect(await originalLocks.lookup(key)).toEqual(completed);
await configureMemoryCoreDreamingStateForTests();
await expect(withMemoryWorkspaceLock(workspace, async () => "recovered")).resolves.toBe(
"recovered",
);
expect(await originalLocks.lookup(key)).toBeUndefined();
expect(await otherLocks.lookup(key)).toEqual(completed);
expect(originalState.db.prepare("PRAGMA integrity_check").get()).toEqual({
integrity_check: "ok",
});
expect(
openOpenClawStateDatabase({ env: otherEnv }).db.prepare("PRAGMA integrity_check").get(),
).toEqual({ integrity_check: "ok" });
} finally {
await configureMemoryCoreDreamingStateForTests();
try {
originalState.db.exec("DROP TRIGGER IF EXISTS fail_memory_workspace_release");
for (const store of [originalLocks, otherLocks]) {
const entry = await store?.lookup(key);
if (store && entry) {
await deleteShortTermLockEntryIfCurrent(store, key, entry);
}
}
} finally {
await closeOpenClawStateDatabaseByPathAsync(
path.join(otherEnv.OPENCLAW_STATE_DIR, "state", "openclaw.sqlite"),
);
}
}
});
});

65
extensions/memory-core/src/memory/manager.borrowed-connection.test.ts
View file

@ -34,10 +34,6 @@ import {
readPublishedSessionIndex,
} from "./manager-index.test-support.js";
import { memoryPublicationFaultEntrypoint } from "./manager-publication-fault-entrypoint.test-support.js";
import {
observePublishedReservations,
reservePublishedWriter,
} from "./manager-publication-observer.test-support.js";
import { MemoryIndexManager } from "./manager.js";
const { closeAllMemorySearchManagers, getMemorySearchManager } = await import("./index.js");
@ -424,67 +420,6 @@ describe("memory manager shared agent connection", () => {
},
);
it("retains the newest cache rows when another holder purges before queued pruning", async () => {
const cfg = fixture.createConfig({
provider: "none",
vectorEnabled: false,
cacheEnabled: true,
sources: ["memory"],
});
const db = sqliteRuntime.openOpenClawAgentDatabase({ agentId: "main" }).db;
const queued = createDeferred<void>();
let armed = false;
let queuedObserved = false;
observePublishedReservations(db, () => {
if (armed) {
queuedObserved = true;
queued.resolve();
}
});
const manager = await fixture.getFreshManager(cfg, "cli");
expect(managerDatabase(manager) === db).toBe(true);
await manager.sync({ reason: "baseline", force: true });
const owner = manager as unknown as { cache: { maxEntries: number } };
owner.cache.maxEntries = 2;
const insert = db.prepare(`INSERT INTO memory_embedding_cache
(provider, model, provider_key, hash, embedding, dims, updated_at)
VALUES ('fixture', 'fixture', 'fixture', ?, ?, 1, ?)`);
for (let index = 0; index < 3; index++) {
insert.run(`cache-${index}`, encodeMemoryEmbedding([index + 1]), index);
}
const readCache = () => db.prepare("SELECT * FROM memory_embedding_cache ORDER BY hash").all();
const readSources = () => db.prepare("SELECT * FROM memory_index_sources ORDER BY id").all();
const before = readCache();
const retained = before.filter((row) => row.hash !== "cache-0");
const sources = readSources();
expect(before).toHaveLength(3);
expect(retained).toHaveLength(2);
const reservation = await reservePublishedWriter(() => {
expect(
db.prepare("DELETE FROM memory_embedding_cache WHERE hash = 'cache-0'").run().changes,
).toBe(1);
expect(readCache()).toEqual(retained);
});
armed = true;
const sync = manager.sync({ reason: "watch" });
void sync.catch(() => undefined);
try {
await Promise.race([queued.promise, sync]);
expect(queuedObserved).toBe(true);
expect(readCache()).toEqual(before);
reservation.release();
await reservation.done;
await sync;
expect(readCache()).toEqual(retained);
expect(readSources()).toEqual(sources);
} finally {
armed = false;
reservation.release();
await Promise.allSettled([sync, reservation.done]);
await manager.close();
}
});
it.each([
"watched-file",
"deleted-memory",

4
extensions/memory-core/src/memory/manager.ts
View file

@ -19,7 +19,6 @@ import { normalizeAgentId } from "openclaw/plugin-sdk/routing";
import { createPluginRuntimeStore } from "openclaw/plugin-sdk/runtime-store";
import type { withOpenClawAgentDatabaseWrite } from "openclaw/plugin-sdk/sqlite-runtime";
import type { MemoryCoreAcquireLocalService } from "./embedding-local-service.js";
import type { EmbeddingProvider } from "./embeddings.js";
import { getMemoryManagerLifecycle } from "./lifecycle.js";
import { MemoryIndexDatabase } from "./manager-database-context.js";
import { isMemoryEmbeddingOperationError } from "./manager-embedding-errors.js";
@ -61,9 +60,6 @@ export class MemoryIndexManager extends MemorySearchOrchestration implements Mem
private readonly managerRegistry: MemoryManagerRegistry<MemoryIndexManager>;
protected readonly createProvider: MemoryManagerProviderFactory = (adapter, create) =>
this.managerRegistry.createProvider(this, adapter, create);
protected releaseProvider(provider: EmbeddingProvider): void {
this.managerRegistry.releaseProvider(this, provider);
}
protected canPublishEmbeddingProbe(): boolean {
return this.managerRegistry.canPublishProbe(this);
}

9
extensions/memory-core/src/short-term-promotion-artifacts.settlement.test.ts
View file

@ -8,7 +8,6 @@ import {
DREAMING_SESSION_INGESTION_FILES_NAMESPACE,
DREAMING_SESSION_INGESTION_SEEN_NAMESPACE,
SESSION_BACKFILL_REWIND_NAMESPACE,
SHORT_TERM_LOCK_NAMESPACE,
SHORT_TERM_PHASE_SIGNAL_NAMESPACE,
SHORT_TERM_RECALL_NAMESPACE,
writeMemoryCoreWorkspaceEntry,
@ -118,7 +117,6 @@ it.each([
const firstFailureIssued = createDeferred<void>();
const failure = new Error("write unavailable");
let failureSelected = false;
let lockReleaseStarted = false;
const beforeMutation = async (namespace: string) => {
if (namespace === scenario.failingNamespace && !failureSelected) {
failureSelected = true;
@ -136,12 +134,6 @@ it.each([
const store = createPluginStateKeyedStoreForTests<T>("memory-core", options);
return {
...store,
async compareAndApply(...args: Parameters<typeof store.compareAndApply>) {
if (options.namespace === SHORT_TERM_LOCK_NAMESPACE && args[2].operation === "delete") {
lockReleaseStarted = true;
}
return await store.compareAndApply(...args);
},
async register(...args: Parameters<typeof store.register>) {
if (scenario.mutation === "register") {
await beforeMutation(options.namespace);
@ -169,7 +161,6 @@ it.each([
vi.useFakeTimers();
try {
await vi.advanceTimersByTimeAsync(0);
expect(lockReleaseStarted).toBe(false);
expect(rollbackSettled).toBe(false);
expect(nextWriterEntered).toBe(false);
} finally {

2
extensions/memory-core/src/short-term-promotion-artifacts.ts
View file

@ -49,7 +49,7 @@ async function inspectStaleShortTermLock(workspaceDir: string, repair: boolean):
maxEntries: SHORT_TERM_LOCK_MAX_ENTRIES,
});
const lockEntry = await lockStore.lookup(lockKey);
if (!lockEntry || !isShortTermLockStealable(lockKey, lockEntry, Date.now())) {
if (!lockEntry || !isShortTermLockStealable(lockEntry, Date.now())) {
return false;
}
return repair ? await deleteShortTermLockEntryIfCurrent(lockStore, lockKey, lockEntry) : true;

32
extensions/memory-core/src/tools.real-manager.test.ts
View file

@ -57,7 +57,6 @@ describe("memory_search real manager", () => {
score: 1,
snippet: "Alpha wiki entry",
};
const alphaQuery = Object.freeze({ query: "alpha", corpus: "memory" });
const zebraQuery = { query: "zebra", corpus: "memory" };
function keywordConfig(sources: Array<"memory" | "sessions"> = ["memory"]) {
@ -88,14 +87,6 @@ describe("memory_search real manager", () => {
});
}
function requireFormatRepair() {
const db = openOpenClawAgentDatabase({ agentId: "main" }).db;
db.prepare(
"UPDATE memory_index_meta SET value = json_set(value, '$.provenanceVersion', 0) WHERE key = 'memory_index_meta_v1'",
).run();
return db;
}
function withWiki() {
registerMemoryCorpusSupplement("wiki-fixture", {
search: async () => [wikiHit],
@ -263,29 +254,6 @@ describe("memory_search real manager", () => {
});
});
it("recovers when the memory manager closes after repair without another rebuild", async () => {
const cfg = createConfig({ vectorEnabled: false, minScore: 0 });
cfg.memory = { ...cfg.memory, search: { ...cfg.memory?.search, cache: { enabled: false } } };
const manager = await indexedManager(cfg, undefined, "cli");
const embeddingCalls = provider.embedBatchCalls;
requireFormatRepair();
const search = manager.search.bind(manager);
vi.spyOn(manager, "search").mockImplementationOnce(async (...args) => {
const results = await search(...args);
await manager.close();
return results;
});
const tool = searchTool(cfg);
const result = await tool.execute("closed-memory-manager", alphaQuery);
expect(result.details).not.toHaveProperty("error");
expect(result.details).toMatchObject({
results: [expect.objectContaining({ path: memoryPath })],
});
expect(result.details).not.toHaveProperty("unavailable");
expect(provider.embedBatchCalls).toBe(embeddingCalls + 1);
expect(result.details).toHaveProperty("warning", expect.stringContaining("provider cost"));
});
it("preserves reindex guidance alongside wiki results after an embedding model change", async () => {
const manager = await indexedManager(
createConfig({ model: "old-embed", vectorEnabled: false }),

6
extensions/memory-core/src/tools.ts
View file

@ -316,12 +316,6 @@ export function createMemorySearchTool(options: MemoryToolOptions) {
return await executeMemorySearchToolQuery({
onRebuildNotice: (read) => rebuildNotices.push(read),
initialManager: { manager: memory.manager, managerMs: memory.debug?.managerMs },
refreshManager: async () => {
const refreshed = await acquireMemoryManager();
return "error" in refreshed
? null
: { manager: refreshed.manager, managerMs: refreshed.debug?.managerMs };
},
query: {
text: query,
resultLimit: maxResults ?? settings.query.maxResults,

7
extensions/qa-lab/src/crabline-discord-replies.ts
View file

@ -34,7 +34,6 @@ export async function startCrablineDiscordReplies(params: {
const lifecycle = new AbortController();
const sockets = new Set<WebSocket>();
const gatewayServer = new WebSocketServer({ noServer: true, maxPayload: 100 * 1024 * 1024 });
let generation = 0;
let lastDelivery: { messageId: string; channelId: string } | undefined;
const server = createServer((req, res) => {
dispatchQaHttpRequest(res, async () => {
@ -42,7 +41,6 @@ export async function startCrablineDiscordReplies(params: {
req.method === "POST" && req.headers.authorization === `Bot ${manifest.botToken}`
? req.url?.match(/^\/api\/v10\/channels\/(\d+)\/messages(?:\?.*)?$/u)?.[1]
: undefined;
const requestGeneration = generation;
if (channelId) {
lastDelivery = undefined;
}
@ -83,9 +81,7 @@ export async function startCrablineDiscordReplies(params: {
if (!messageId || !/^\d+$/u.test(messageId)) {
throw new Error("Discord delivery response omitted its message id");
}
if (requestGeneration === generation) {
lastDelivery = { messageId, channelId };
}
lastDelivery = { messageId, channelId };
}
const pathname = new URL(req.url ?? "/", upstream).pathname;
if (req.method === "GET" && /^\/api\/v10\/gateway(?:\/bot)?$/u.test(pathname)) {
@ -248,7 +244,6 @@ export async function startCrablineDiscordReplies(params: {
apiBaseUrl: `${origin}/api/v10`,
waitForCompletedReply,
reset() {
generation += 1;
lastDelivery = undefined;
},
cleanup() {

10
extensions/qa-lab/src/evidence-invocation.test.ts
View file

@ -112,7 +112,7 @@ describe("evidence invocation owner", () => {
expect(continuation.entries).toEqual(before.entries);
});
it("preserves the parent and pending import when a selected failure would become ineffective", () => {
it("rejects publication when a selected failure would become ineffective", () => {
const parent = createInvocation();
const first = parent.begin(0);
parent.complete(first, { status: "fail", entries: [entry("fail")] });
@ -128,13 +128,7 @@ describe("evidence invocation owner", () => {
const input = child.snapshot(snapshotOptions);
parent.importChild(0, input);
expect(() => parent.select(0, first)).toThrow(/selected observation is ineffective/);
expect(parent.snapshot(snapshotOptions)).toEqual(before);
expect(parent.importChild(0, input)).toBe(retry);
parent.select(0, retry);
expect(projectQaEvidenceScenarioOutcomes(parent.snapshot(snapshotOptions))[0]).toMatchObject({
occurrenceId: retry,
status: "pass",
});
expect(() => parent.snapshot(snapshotOptions)).toThrow(/selected observation is ineffective/);
});
it("accepts an identical pending replay after another instance appends and retains insertion order", () => {

30
extensions/qa-lab/src/evidence-invocation.ts
View file

@ -253,30 +253,27 @@ export function createQaEvidenceInvocation(params: {
function select(index: number, occurrenceId: string | null): string | null;
function select(index: number, occurrenceId: string | null): string | null {
anchorFor(index);
const nextAnchors = structuredClone(anchors);
const nextObservations = structuredClone(observations);
const nextEntries = structuredClone(entries);
const setEffective = (id: string, effective: boolean) => {
for (const entry of nextEntries) {
for (const entry of entries) {
if (entry.binding.occurrenceId === id) {
entry.effective = effective;
}
}
};
const anchor = nextAnchors[index]!;
const anchor = anchors[index]!;
const pending = pendingChildren.get(index);
if (pending) {
for (const completion of pending.completions) {
const offset = nextObservations.findIndex((item) => item.id === completion.id);
nextObservations[offset] = structuredClone(completion);
const offset = observations.findIndex((item) => item.id === completion.id);
observations[offset] = structuredClone(completion);
}
for (const update of pending.updates) {
setEffective(update.occurrenceId, update.effective);
}
nextObservations.splice(pending.observationOffset, 0, ...structuredClone(pending.additions));
nextEntries.splice(pending.entryOffset, 0, ...structuredClone(pending.rows));
observations.splice(pending.observationOffset, 0, ...structuredClone(pending.additions));
entries.splice(pending.entryOffset, 0, ...structuredClone(pending.rows));
}
const byId = new Map(nextObservations.map((occurrence) => [occurrence.id, occurrence]));
const byId = new Map(observations.map((occurrence) => [occurrence.id, occurrence]));
let selected = occurrenceId === null ? null : byId.get(occurrenceId);
if (
occurrenceId !== null &&
@ -309,7 +306,7 @@ export function createQaEvidenceInvocation(params: {
setEffective(priorId, false);
priorId = byId.get(priorId)!.retryOf;
}
for (const occurrence of nextObservations) {
for (const occurrence of observations) {
let ancestor = occurrence.retryOf;
while (ancestor !== null && ancestor !== selectedId) {
ancestor = byId.get(ancestor)!.retryOf;
@ -318,16 +315,13 @@ export function createQaEvidenceInvocation(params: {
setEffective(occurrence.id, false);
}
}
// Validate the full proposed selection before changing authoritative state
// or consuming pending imports; a rejected choice remains retryable by its owner.
// Invalid internal selections abort this invocation; callers discard its
// partial state. Snapshot publication independently validates the same facts.
buildQaOccurrenceEvidenceSummary({
generatedAt: new Date().toISOString(),
occurrences: [...nextAnchors, ...nextObservations],
entries: nextEntries,
occurrences: [...anchors, ...observations],
entries,
});
anchors.splice(0, anchors.length, ...nextAnchors);
observations.splice(0, observations.length, ...nextObservations);
entries.splice(0, entries.length, ...nextEntries);
pendingChildren.delete(index);
return selectedId;
}

6
extensions/qa-lab/src/gateway-child-command.test-support.ts
View file

@ -132,10 +132,8 @@ record({
});
const gatewayAttempts = fs.readFileSync(recordPath, "utf8").trim().split("\\n")
.map((line) => JSON.parse(line)).filter((entry) => entry.kind === "gateway").length;
if (gatewayAttempts === 1 && process.env.QA_STARTUP_RETRY) {
process.stderr.write(process.env.QA_STARTUP_RETRY === "migration"
? "OpenClaw plugin migration inputs changed during startup convergence; refusing readiness."
: "listen EADDRINUSE: address already in use");
if (gatewayAttempts === 1 && process.env.QA_STARTUP_RETRY === "migration") {
process.stderr.write("OpenClaw plugin migration inputs changed during startup convergence; refusing readiness.");
process.exit(18);
}
process.stderr.write("fixture gateway exit");

57
extensions/qa-lab/src/gateway-child-readiness.ts
View file

@ -1,5 +1,4 @@
import { setTimeout as sleep } from "node:timers/promises";
import { formatErrorMessage } from "openclaw/plugin-sdk/error-runtime";
import { resolveTimerTimeoutMs } from "openclaw/plugin-sdk/number-runtime";
import { fetchWithSsrFGuard } from "openclaw/plugin-sdk/ssrf-runtime";
import { QaSuiteInfraError } from "./errors.js";
@ -9,54 +8,17 @@ import {
throwQaGatewayChildFailure,
} from "./gateway-child-process.js";
export const QA_GATEWAY_CHILD_STARTUP_MAX_ATTEMPTS = 5;
const QA_GATEWAY_CHILD_RESTART_BOUNDARY_TIMEOUT_MS = 90_000;
const QA_GATEWAY_MIGRATION_CONVERGENCE_RESTART_PREFIX =
"OpenClaw plugin migration inputs changed during startup convergence;";
type QaGatewayStartupRetryKind = "bind-collision" | "migration-convergence-restart";
type QaGatewayHealthChild = {
exitCode: number | null;
signalCode: NodeJS.Signals | null;
};
function classifyQaGatewayStartupRetry(details: string): QaGatewayStartupRetryKind | null {
if (details.includes(QA_GATEWAY_MIGRATION_CONVERGENCE_RESTART_PREFIX)) {
return "migration-convergence-restart";
}
if (
details.includes("another gateway instance is already listening on ws://") ||
details.includes("failed to bind gateway socket on ws://") ||
details.includes("EADDRINUSE") ||
details.includes("address already in use")
) {
return "bind-collision";
}
return null;
}
export function resolveQaGatewayStartupRetry(params: {
attempt: number;
details: string;
migrationConvergenceRestartUsed: boolean;
}) {
if (params.attempt >= QA_GATEWAY_CHILD_STARTUP_MAX_ATTEMPTS) {
return null;
}
const kind = classifyQaGatewayStartupRetry(params.details);
if (
!kind ||
(kind === "migration-convergence-restart" && params.migrationConvergenceRestartUsed)
) {
return null;
}
return {
kind,
reuseLaunchState: kind === "migration-convergence-restart",
migrationConvergenceRestartUsed:
params.migrationConvergenceRestartUsed || kind === "migration-convergence-restart",
};
export function needsQaGatewayMigrationRestart(details: string) {
return details.includes(QA_GATEWAY_MIGRATION_CONVERGENCE_RESTART_PREFIX);
}
async function fetchLocalGatewayProbe(params: {
@ -152,18 +114,3 @@ export function waitForGatewayReady(params: QaGatewayProbeParams) {
export function waitForGatewayListening(params: QaGatewayProbeParams) {
return waitForGatewayProbe(params, "listening");
}
export function isRetryableRpcStartupError(error: unknown) {
// Startup errors cross the same low-level client/log boundary; timeout and
// token-mismatch retry facts exist only in the formatted diagnostic.
const details = formatErrorMessage(error);
return (
details.includes("gateway timeout after") ||
details.includes("handshake timeout") ||
details.includes("gateway token mismatch") ||
details.includes("token mismatch") ||
details.includes("gateway closed (1000") ||
details.includes("gateway closed (1006") ||
details.includes("gateway closed (1012)")
);
}

197
extensions/qa-lab/src/gateway-child-setup.ts
View file

@ -319,7 +319,6 @@ export async function prepareQaGatewayChild(
let wsUrl = "";
let cfg!: OpenClawConfig;
let env: NodeJS.ProcessEnv | null = null;
let packagedMockAuthStaged = false;
const nodeExecPath = gatewayExecutablePath ?? process.execPath;
const cliArgsPrefix = gatewayCommand?.processBoundary
@ -365,122 +364,109 @@ export async function prepareQaGatewayChild(
configPath,
gatewayToken,
buildGatewayArgs,
async prepareAttempt(reuseStartupLaunchState: boolean) {
async prepareAttempt() {
lifetime.assertOpen();
if (!reuseStartupLaunchState) {
lifetime.portReservation = await reserveQaGatewayPort(net.createServer());
gatewayPort = lifetime.portReservation.port;
baseUrl = `http://127.0.0.1:${gatewayPort}`;
wsUrl = `ws://127.0.0.1:${gatewayPort}`;
cfg = await buildStagedGatewayConfig(gatewayPort);
if (!env) {
const allowedPluginIds = uniqueStrings(
[...(cfg.plugins?.allow ?? []), "openai"].filter(
(pluginId): pluginId is string => typeof pluginId === "string" && pluginId.length > 0,
),
);
if (!usesPackagedCandidate) {
// Register the external root before staging so one lifecycle owner
// also cleans partial copies and host-version resolution failures.
lifetime.stagedBundledPluginsRoot = resolveQaStagedBundledPluginsRoot({
lifetime.portReservation = await reserveQaGatewayPort(net.createServer());
gatewayPort = lifetime.portReservation.port;
baseUrl = `http://127.0.0.1:${gatewayPort}`;
wsUrl = `ws://127.0.0.1:${gatewayPort}`;
cfg = await buildStagedGatewayConfig(gatewayPort);
const allowedPluginIds = uniqueStrings(
[...(cfg.plugins?.allow ?? []), "openai"].filter(
(pluginId): pluginId is string => typeof pluginId === "string" && pluginId.length > 0,
),
);
if (!usesPackagedCandidate) {
// Register the external root before staging so one lifecycle owner
// also cleans partial copies and host-version resolution failures.
lifetime.stagedBundledPluginsRoot = resolveQaStagedBundledPluginsRoot({
repoRoot: params.repoRoot,
tempRoot,
});
}
const stagedPluginRuntime = usesPackagedCandidate
? { bundledPluginsDir: undefined, runtimeHostVersion: undefined }
: {
...(await createQaBundledPluginsDir({
repoRoot: params.repoRoot,
tempRoot,
});
}
const stagedPluginRuntime = usesPackagedCandidate
? { bundledPluginsDir: undefined, runtimeHostVersion: undefined }
: {
...(await createQaBundledPluginsDir({
repoRoot: params.repoRoot,
tempRoot,
allowedPluginIds,
})),
runtimeHostVersion: await resolveQaRuntimeHostVersion({
repoRoot: params.repoRoot,
allowedPluginIds,
}),
};
env = buildQaRuntimeEnv({
configPath,
gatewayToken,
homeDir,
forwardHostHome: params.forwardHostHome,
stateDir,
tempRoot,
xdgConfigHome,
xdgDataHome,
xdgCacheHome,
bundledPluginsDir: stagedPluginRuntime.bundledPluginsDir,
stagedBundledPluginsRoot: lifetime.stagedBundledPluginsRoot,
compatibilityHostVersion: stagedPluginRuntime.runtimeHostVersion,
developmentSourceRoot: usesPackagedCandidate ? null : params.repoRoot,
allowedPluginIds,
})),
runtimeHostVersion: await resolveQaRuntimeHostVersion({
repoRoot: params.repoRoot,
allowedPluginIds,
}),
};
env = buildQaRuntimeEnv({
configPath,
gatewayToken,
homeDir,
forwardHostHome: params.forwardHostHome,
stateDir,
tempRoot,
xdgConfigHome,
xdgDataHome,
xdgCacheHome,
bundledPluginsDir: stagedPluginRuntime.bundledPluginsDir,
stagedBundledPluginsRoot: lifetime.stagedBundledPluginsRoot,
compatibilityHostVersion: stagedPluginRuntime.runtimeHostVersion,
developmentSourceRoot: usesPackagedCandidate ? null : params.repoRoot,
providerMode,
runtimeEnvPatch: {
...params.runtimeEnvPatch,
...buildQaForcedRuntimeEnvPatch({
forcedRuntime: params.forcedRuntime,
runtimeSelection: params.runtimeSelection,
providerMode,
runtimeEnvPatch: {
...params.runtimeEnvPatch,
...buildQaForcedRuntimeEnvPatch({
forcedRuntime: params.forcedRuntime,
runtimeSelection: params.runtimeSelection,
providerMode,
providerBaseUrl: params.providerBaseUrl,
codexModelCatalogPath,
nativeAppServerArgs:
params.runtimeEnvPatch?.OPENCLAW_CODEX_APP_SERVER_ARGS ??
process.env.OPENCLAW_CODEX_APP_SERVER_ARGS,
}),
},
forwardHostHomeForClaudeCli: liveProviderIds.includes("claude-cli"),
claudeCliAuthMode: params.claudeCliAuthMode,
});
}
assertQaLiveCodexAuthAvailable({
cfg,
providerIds: liveProviderIds,
env,
});
await fs.writeFile(configPath, `${JSON.stringify(cfg, null, 2)}\n`, {
encoding: "utf8",
providerBaseUrl: params.providerBaseUrl,
codexModelCatalogPath,
nativeAppServerArgs:
params.runtimeEnvPatch?.OPENCLAW_CODEX_APP_SERVER_ARGS ??
process.env.OPENCLAW_CODEX_APP_SERVER_ARGS,
}),
},
forwardHostHomeForClaudeCli: liveProviderIds.includes("claude-cli"),
claudeCliAuthMode: params.claudeCliAuthMode,
});
assertQaLiveCodexAuthAvailable({
cfg,
providerIds: liveProviderIds,
env,
});
await fs.writeFile(configPath, `${JSON.stringify(cfg, null, 2)}\n`, {
encoding: "utf8",
mode: 0o600,
});
// Bootstrap commands must inspect this child's port without our placeholder listener.
env.OPENCLAW_GATEWAY_PORT = String(gatewayPort);
await lifetime.portReservation?.release();
lifetime.portReservation = null;
const mockAuthProviders = resolvedProvider.mockAuthProviders;
if (usesPackagedCandidate && gatewayCommand && mockAuthProviders?.length) {
const canonicalConfig = await fs.readFile(configPath);
await fs.mkdir(path.dirname(packagedAuthConfigPath), { recursive: true, mode: 0o700 });
await fs.writeFile(packagedAuthConfigPath, canonicalConfig, {
flag: "wx",
mode: 0o600,
});
// Bootstrap commands must inspect this child's port without our placeholder listener.
env.OPENCLAW_GATEWAY_PORT = String(gatewayPort);
await lifetime.portReservation?.release();
lifetime.portReservation = null;
const mockAuthProviders = resolvedProvider.mockAuthProviders;
if (
usesPackagedCandidate &&
gatewayCommand &&
mockAuthProviders?.length &&
!packagedMockAuthStaged
) {
const canonicalConfig = await fs.readFile(configPath);
await fs.mkdir(path.dirname(packagedAuthConfigPath), { recursive: true, mode: 0o700 });
await fs.writeFile(packagedAuthConfigPath, canonicalConfig, {
flag: "wx",
mode: 0o600,
});
await stageQaPackagedMockAuthProfiles({
lifetime,
command: gatewayCommand,
configPath: packagedAuthConfigPath,
cwd: gatewayCwd,
env: createQaPackagedBootstrapEnv(env),
providers: mockAuthProviders,
});
if (!canonicalConfig.equals(await fs.readFile(configPath))) {
throw new Error("installed package mock auth bootstrap mutated canonical config");
}
packagedMockAuthStaged = true;
await stageQaPackagedMockAuthProfiles({
lifetime,
command: gatewayCommand,
configPath: packagedAuthConfigPath,
cwd: gatewayCwd,
env: createQaPackagedBootstrapEnv(env),
providers: mockAuthProviders,
});
if (!canonicalConfig.equals(await fs.readFile(configPath))) {
throw new Error("installed package mock auth bootstrap mutated canonical config");
}
}
if (!env) {
throw new Error("qa gateway runtime env not initialized");
}
// Auth staging opens parent-owned agent stores. Release this fixture's
// leases before packaged repair or Gateway startup takes maintenance ownership.
await closeQaRuntimeStores(tempRoot);
lifetime.assertOpen();
if (!reuseStartupLaunchState && usesPackagedCandidate && gatewayCommand) {
if (usesPackagedCandidate && gatewayCommand) {
const command = {
lifetime,
executablePath: gatewayCommand.executablePath,
@ -489,8 +475,7 @@ export async function prepareQaGatewayChild(
env: createQaPackagedBootstrapEnv(env),
};
// The separate onboarding smoke cannot prepare this child's state.
// Converge every freshly written config; a new-port retry can otherwise
// restore plugin entries the candidate removed before verify-only startup.
// Converge the freshly written config before verify-only startup.
// Published candidates such as 2026.7.1-2 predate capability consent.
const help = await runQaPackagedBootstrap(
"installed package plugin setup failed (update repair --help)",

167
extensions/qa-lab/src/gateway-child.test.ts
View file

@ -30,8 +30,7 @@ import {
throwQaGatewayChildFailure,
} from "./gateway-child-process.js";
import {
isRetryableRpcStartupError,
resolveQaGatewayStartupRetry,
needsQaGatewayMigrationRestart,
waitForGatewayReady,
waitForQaGatewayRestartBoundary,
} from "./gateway-child-readiness.js";
@ -983,58 +982,47 @@ describe("buildQaRuntimeEnv", () => {
expect(new Set(records.map((record) => record.authDbPath)).size).toBe(1);
});
it.each([
{ retry: "bind", configBuilds: 2 },
{ retry: "migration", configBuilds: 1 },
])(
"preserves packaged config repair across $retry startup retries",
async ({ retry, configBuilds }) => {
const mutateConfig = vi.fn((cfg: OpenClawConfig) => cfg);
const { start, recordPath } = await createPackagedFixture(
{
QA_STARTUP_RETRY: retry,
QA_CONFIG_RUNTIME_VERSION: "2026.7.33",
},
mutateConfig,
);
await expect(start()).rejects.toThrow("fixture gateway exit");
const records = await readJsonLines(recordPath);
const gateways = records.filter((record) => record.kind === "gateway");
expect(gateways).toHaveLength(2);
expect(gateways.map((record) => record.sourcePluginConfigured)).toEqual([false, false]);
const repairs = records.filter((record) => record.kind === "plugins");
expect(repairs).toHaveLength(configBuilds);
for (const repair of repairs) {
expect(repair.args).toContain("--accept-capabilities");
}
expect(repairs.map((record) => record.configPort)).toEqual(
retry === "bind" ? gateways.map((record) => record.configPort) : [gateways[0]?.configPort],
);
expect(mutateConfig).toHaveBeenCalledTimes(configBuilds);
expect(records.filter((record) => record.kind === "auth")).toHaveLength(2);
expect(records.map((record) => record.kind)).toEqual([
"auth",
"auth",
"help",
"plugins",
"gateway",
...(retry === "bind" ? ["help", "plugins"] : []),
"gateway",
]);
expect(new Set(records.map((record) => record.stateDir)).size).toBe(1);
for (const gateway of gateways) {
expect(gateway.args).toContainEqual(String(gateway.configPort));
expect(gateway).toMatchObject({
dbExists: true,
authProfileIds: ["qa-mock-openai", "qa-mock-anthropic"],
configVersion: "2026.7.33",
});
}
if (retry === "migration") {
expect(gateways[1]?.configPort).toBe(gateways[0]?.configPort);
}
},
);
it("preserves packaged config repair across migration convergence", async () => {
const mutateConfig = vi.fn((cfg: OpenClawConfig) => cfg);
const { start, recordPath } = await createPackagedFixture(
{
QA_STARTUP_RETRY: "migration",
QA_CONFIG_RUNTIME_VERSION: "2026.7.33",
},
mutateConfig,
);
await expect(start()).rejects.toThrow("fixture gateway exit");
const records = await readJsonLines(recordPath);
const gateways = records.filter((record) => record.kind === "gateway");
expect(gateways).toHaveLength(2);
expect(gateways.map((record) => record.sourcePluginConfigured)).toEqual([false, false]);
const repairs = records.filter((record) => record.kind === "plugins");
expect(repairs).toHaveLength(1);
for (const repair of repairs) {
expect(repair.args).toContain("--accept-capabilities");
}
expect(repairs.map((record) => record.configPort)).toEqual([gateways[0]?.configPort]);
expect(mutateConfig).toHaveBeenCalledTimes(1);
expect(records.filter((record) => record.kind === "auth")).toHaveLength(2);
expect(records.map((record) => record.kind)).toEqual([
"auth",
"auth",
"help",
"plugins",
"gateway",
"gateway",
]);
expect(new Set(records.map((record) => record.stateDir)).size).toBe(1);
for (const gateway of gateways) {
expect(gateway.args).toContainEqual(String(gateway.configPort));
expect(gateway).toMatchObject({
dbExists: true,
authProfileIds: ["qa-mock-openai", "qa-mock-anthropic"],
configVersion: "2026.7.33",
});
}
expect(gateways[1]?.configPort).toBe(gateways[0]?.configPort);
});
it("preserves authored newer-version metadata so the packaged candidate refuses it", async () => {
const { start, recordPath } = await createPackagedFixture(
@ -1294,80 +1282,17 @@ describe("buildQaRuntimeEnv", () => {
}
});
it.each([
["another gateway instance is already listening on ws://127.0.0.1:43124", "bind-collision"],
[
"failed to bind gateway socket on ws://127.0.0.1:43124: Error: listen EADDRINUSE",
"bind-collision",
],
] as const)("classifies %s", (details, expectedKind) => {
expect(
resolveQaGatewayStartupRetry({
attempt: 1,
details,
migrationConvergenceRestartUsed: false,
})?.kind,
).toBe(expectedKind);
});
it("does not retry an incomplete migration-convergence diagnostic", () => {
const details = "OpenClaw plugin migration inputs changed during startup convergence";
expect(
resolveQaGatewayStartupRetry({
attempt: 1,
details,
migrationConvergenceRestartUsed: false,
}),
).toBeNull();
expect(needsQaGatewayMigrationRestart(details)).toBe(false);
});
it("restarts migration convergence once with the same launch state", () => {
const first = resolveQaGatewayStartupRetry({
attempt: 1,
details:
it("recognizes a complete migration convergence diagnostic", () => {
expect(
needsQaGatewayMigrationRestart(
"OpenClaw plugin migration inputs changed during startup convergence; refusing readiness.",
migrationConvergenceRestartUsed: false,
});
expect(first).toEqual({
kind: "migration-convergence-restart",
reuseLaunchState: true,
migrationConvergenceRestartUsed: true,
});
expect(
resolveQaGatewayStartupRetry({
attempt: 2,
details:
"OpenClaw plugin migration inputs changed during startup convergence; refusing readiness.",
migrationConvergenceRestartUsed: first?.migrationConvergenceRestartUsed ?? false,
}),
).toBeNull();
});
it("fails immediately for generic exits and after the startup attempt budget", () => {
expect(
resolveQaGatewayStartupRetry({
attempt: 1,
details: "gateway exited with code 1",
migrationConvergenceRestartUsed: false,
}),
).toBeNull();
expect(
resolveQaGatewayStartupRetry({
attempt: 5,
details: "listen EADDRINUSE",
migrationConvergenceRestartUsed: false,
}),
).toBeNull();
});
it("treats startup token mismatches as retryable rpc startup errors", () => {
expect(
isRetryableRpcStartupError(
"unauthorized: gateway token mismatch (set gateway.remote.token to match gateway.auth.token)",
),
).toBe(true);
expect(isRetryableRpcStartupError("permission denied")).toBe(false);
});
it("preserves only sanitized gateway debug artifacts", async () => {

84
extensions/qa-lab/src/gateway-child.ts
View file

@ -1,5 +1,4 @@
import { spawn } from "node:child_process";
import { setTimeout as sleep } from "node:timers/promises";
import { formatErrorMessage } from "openclaw/plugin-sdk/error-runtime";
import { closeQaRuntimeStores } from "openclaw/plugin-sdk/qa-runtime";
import { runQaGatewayCliCommand } from "./gateway-child-command.js";
@ -12,9 +11,7 @@ import {
type QaChildFailure,
} from "./gateway-child-process.js";
import {
isRetryableRpcStartupError,
QA_GATEWAY_CHILD_STARTUP_MAX_ATTEMPTS,
resolveQaGatewayStartupRetry,
needsQaGatewayMigrationRestart,
waitForGatewayListening,
waitForGatewayReady,
waitForQaGatewayRestartBoundary,
@ -70,7 +67,6 @@ async function startOwnedGatewayChild(
} = setup;
let active!: ReturnType<QaGatewayChildLifecycle["register"]>;
let getChildFailure: (() => QaChildFailure | null) | undefined;
let launch!: Awaited<ReturnType<typeof setup.prepareAttempt>>;
const requireRpcClient = () => {
if (!lifetime.rpcClient) {
throw new Error("qa gateway rpc client is not ready");
@ -160,19 +156,7 @@ async function startOwnedGatewayChild(
token: gatewayToken,
logs,
});
for (let rpcAttempt = 1; ; rpcAttempt += 1) {
lifetime.assertOpen();
try {
await lifetime.rpcClient.request("config.get", {}, { timeoutMs: 30_000 });
break;
} catch (error) {
if (rpcAttempt >= 4 || !isRetryableRpcStartupError(error)) {
throw error;
}
await sleep(500 * rpcAttempt);
await waitForGatewayReady({ ...health, timeoutMs: initial ? 60_000 : 15_000 });
}
}
await lifetime.rpcClient.request("config.get", {}, { timeoutMs: 30_000 });
throwActiveChildFailure();
if (active.identity && lifetime.controller) {
await lifetime.controller.markReady(active.identity);
@ -180,41 +164,24 @@ async function startOwnedGatewayChild(
lifetime.assertOpen();
active.ready = true;
};
let migrationConvergenceRestartUsed = false;
let reuseStartupLaunchState = false;
for (let attempt = 1; attempt <= QA_GATEWAY_CHILD_STARTUP_MAX_ATTEMPTS; attempt += 1) {
launch = await setup.prepareAttempt(reuseStartupLaunchState);
const attemptLogMark = output.mark();
try {
await launchReady(true, attempt);
break;
} catch (error) {
const attemptLogs = output.readRedactedSince(attemptLogMark);
const retry = resolveQaGatewayStartupRetry({
attempt,
details: attemptLogs.trim() ? attemptLogs : formatErrorMessage(error),
migrationConvergenceRestartUsed,
});
const retryableRpcStartup =
attempt < QA_GATEWAY_CHILD_STARTUP_MAX_ATTEMPTS &&
!retry &&
isRetryableRpcStartupError(error);
if (!retry && !retryableRpcStartup) {
throw error;
}
await stopAttempt(error);
lifetime.assertOpen();
migrationConvergenceRestartUsed =
retry?.migrationConvergenceRestartUsed ?? migrationConvergenceRestartUsed;
reuseStartupLaunchState = retry?.reuseLaunchState ?? false;
const retryMessage =
retry?.kind === "migration-convergence-restart"
? `[qa-lab] gateway child startup attempt ${attempt}/${QA_GATEWAY_CHILD_STARTUP_MAX_ATTEMPTS} completed plugin migration convergence; restarting once with the same state, config, and port ${launch.gatewayPort}\n`
: `[qa-lab] gateway child startup attempt ${attempt}/${QA_GATEWAY_CHILD_STARTUP_MAX_ATTEMPTS} hit a transient startup race on port ${launch.gatewayPort}; retrying with a new port\n`;
const retryBuffer = Buffer.from(retryMessage);
output.push("internal", retryBuffer);
stdoutLog.write(retryBuffer);
const launch = await setup.prepareAttempt();
const attemptLogMark = output.mark();
try {
await launchReady(true);
} catch (error) {
const attemptLogs = output.readRedactedSince(attemptLogMark);
if (
!needsQaGatewayMigrationRestart(attemptLogs.trim() ? attemptLogs : formatErrorMessage(error))
) {
throw error;
}
await stopAttempt(error);
const retryBuffer = Buffer.from(
`[qa-lab] gateway child startup attempt 1/2 completed plugin migration convergence; restarting once with the same state, config, and port ${launch.gatewayPort}\n`,
);
output.push("internal", retryBuffer);
stdoutLog.write(retryBuffer);
await launchReady(true, 2);
}
const { cfg, baseUrl, wsUrl, env: runningEnv } = launch;
const signalActiveProcess = async (signal: NodeJS.Signals) => {
@ -302,14 +269,11 @@ async function startOwnedGatewayChild(
try {
await launchReady(false);
} catch (error) {
const retry = resolveQaGatewayStartupRetry({
attempt: 1,
details: [output.readRedactedSince(replacementLogMark), formatErrorMessage(error)].join(
"\n",
),
migrationConvergenceRestartUsed: false,
});
if (retry?.kind !== "migration-convergence-restart") {
if (
!needsQaGatewayMigrationRestart(
[output.readRedactedSince(replacementLogMark), formatErrorMessage(error)].join("\n"),
)
) {
throw error;
}
await stopAttempt(error);

28
extensions/qa-lab/src/live-transports/matrix/adapter.runtime.test.ts
View file

@ -28,7 +28,6 @@ describe("Matrix QA adapter observer recovery", () => {
isExpectedInterruption: () => interrupted,
observer: makeObserver(waitForOptionalRoomEvent),
predicate: () => true,
readInterruptionGeneration: () => 1,
roomId: "!room:matrix.test",
sleepImpl,
timeoutMs: 1_000,
@ -50,7 +49,6 @@ describe("Matrix QA adapter observer recovery", () => {
isExpectedInterruption: () => false,
observer: makeObserver(waitForOptionalRoomEvent),
predicate: () => true,
readInterruptionGeneration: () => 0,
roomId: "!room:matrix.test",
sleepImpl,
timeoutMs: 1_000,
@ -75,7 +73,6 @@ describe("Matrix QA adapter observer recovery", () => {
isExpectedInterruption: () => interrupted,
observer: makeObserver(waitForOptionalRoomEvent),
predicate: () => true,
readInterruptionGeneration: () => 1,
roomId: "!room:matrix.test",
sleepImpl: vi.fn(async () => undefined),
timeoutMs: 1_000,
@ -83,29 +80,4 @@ describe("Matrix QA adapter observer recovery", () => {
).resolves.toEqual({ matched: false, since: "s2" });
expect(waitForOptionalRoomEvent).toHaveBeenCalledTimes(2);
});
it("retries a poll that spans the complete interruption window", async () => {
let interruptionGeneration = 0;
const waitForOptionalRoomEvent = vi
.fn<MatrixQaRoomObserver["waitForOptionalRoomEvent"]>()
.mockImplementationOnce(async () => {
// The poll began before restart and rejected only after begin/end both ran.
interruptionGeneration = 2;
throw new Error("pre-restart poll rejected after recovery");
})
.mockResolvedValueOnce({ matched: false, since: "s3" });
await expect(
waitForMatrixQaObserverEvent({
isExpectedInterruption: () => false,
observer: makeObserver(waitForOptionalRoomEvent),
predicate: () => true,
readInterruptionGeneration: () => interruptionGeneration,
roomId: "!room:matrix.test",
sleepImpl: vi.fn(async () => undefined),
timeoutMs: 1_000,
}),
).resolves.toEqual({ matched: false, since: "s3" });
expect(waitForOptionalRoomEvent).toHaveBeenCalledTimes(2);
});
});

18
extensions/qa-lab/src/live-transports/matrix/adapter.runtime.ts
View file

@ -68,7 +68,6 @@ export async function waitForMatrixQaObserverEvent(params: {
isExpectedInterruption: () => boolean;
observer: MatrixQaRoomObserver;
predicate: (event: MatrixQaObservedEvent) => boolean;
readInterruptionGeneration: () => number;
roomId: string;
sleepImpl?: (ms: number) => Promise<unknown>;
timeoutMs: number;
@ -76,7 +75,6 @@ export async function waitForMatrixQaObserverEvent(params: {
const sleepImpl = params.sleepImpl ?? sleep;
for (;;) {
const expectedInterruptionAtStart = params.isExpectedInterruption();
const interruptionGenerationAtStart = params.readInterruptionGeneration();
try {
return await params.observer.waitForOptionalRoomEvent({
predicate: params.predicate,
@ -84,15 +82,8 @@ export async function waitForMatrixQaObserverEvent(params: {
timeoutMs: params.timeoutMs,
});
} catch (error) {
// The homeserver restart scenario owns this narrow recovery window. The
// generation also catches a poll that spans the complete interruption
// before rejecting. The observer clears its failed pollPromise in finally,
// so the same cursor can safely retry.
if (
!expectedInterruptionAtStart &&
!params.isExpectedInterruption() &&
interruptionGenerationAtStart === params.readInterruptionGeneration()
) {
// A poll spanning the entire restart may fail visibly; rerun that scenario.
if (!expectedInterruptionAtStart && !params.isExpectedInterruption()) {
throw error;
}
await sleepImpl(MATRIX_EXPECTED_INTERRUPTION_RETRY_MS);
@ -213,14 +204,10 @@ export async function createMatrixQaTransportAdapter(
const nativeEventIds = new Map<string, string>();
const busMessageIds = new Map<string, string>();
let expectedTransportInterruption = false;
let transportInterruptionGeneration = 0;
const scenarioEnvironment = createMatrixQaScenarioEnvironment({
accountId,
harness,
onTransportInterruptionStateChange: (active) => {
if (expectedTransportInterruption !== active) {
transportInterruptionGeneration += 1;
}
expectedTransportInterruption = active;
},
observedEvents,
@ -236,7 +223,6 @@ export async function createMatrixQaTransportAdapter(
isExpectedInterruption: () => expectedTransportInterruption,
observer,
predicate: (event) => event.sender === provisioning.sut.userId && Boolean(event.body),
readInterruptionGeneration: () => transportInterruptionGeneration,
roomId,
timeoutMs: 1_000,
});

18
extensions/qa-lab/src/live-transports/shared/credential-lease-health.ts
View file

@ -1,13 +1,5 @@
import { performance } from "node:perf_hooks";
export type QaLeaseClock = { wall: number; monotonic: number };
export function captureQaLeaseClock(): QaLeaseClock {
return { wall: Date.now(), monotonic: performance.now() };
}
/** Timers can pause; every consumer also checks the last confirmed lease directly. */
export function createQaLeaseHealth(leaseTtlMs: number, acquiredAt: QaLeaseClock) {
export function createQaLeaseHealth(leaseTtlMs: number, acquiredAt: number) {
let confirmedAt = acquiredAt;
let closed = false;
let failure: Error | undefined;
@ -16,10 +8,8 @@ export function createQaLeaseHealth(leaseTtlMs: number, acquiredAt: QaLeaseClock
throw new Error("QA credential lease has been released.");
}
if (!failure) {
const now = captureQaLeaseClock();
if (
Math.max(now.wall - confirmedAt.wall, now.monotonic - confirmedAt.monotonic) >= leaseTtlMs
) {
// Broker TTLs elapse during host suspend, unlike a process monotonic clock.
if (Date.now() - confirmedAt >= leaseTtlMs) {
failure = new Error("QA credential lease expired before its owner could renew it.");
}
}
@ -29,7 +19,7 @@ export function createQaLeaseHealth(leaseTtlMs: number, acquiredAt: QaLeaseClock
};
return {
assertHealthy,
confirm(requestStartedAt: QaLeaseClock) {
confirm(requestStartedAt: number) {
assertHealthy();
confirmedAt = requestStartedAt;
assertHealthy();

4
extensions/qa-lab/src/live-transports/shared/credential-lease.runtime.test.ts
View file

@ -122,8 +122,9 @@ describe("credential lease runtime", () => {
vi.useRealTimers();
});
it("releases a credential that expired while its payload was being hydrated", async () => {
it("releases a credential whose TTL expires while the host monotonic clock is paused", async () => {
vi.useFakeTimers();
vi.spyOn(performance, "now").mockReturnValue(42);
const operations: string[] = [];
await expect(
acquireQaCredentialLease({
@ -158,6 +159,7 @@ describe("credential lease runtime", () => {
},
}),
).rejects.toThrow("could not be confirmed before use");
expect(performance.now()).toBe(42);
expect(operations).toEqual(["acquire", "release"]);
});

6
extensions/qa-lab/src/live-transports/shared/credential-lease.runtime.ts
View file

@ -16,7 +16,7 @@ import {
parseQaCredentialResponsePayload,
QA_CREDENTIALS_DEFAULT_ENDPOINT_PREFIX,
} from "../../qa-credentials-common.runtime.js";
import { captureQaLeaseClock, createQaLeaseHealth } from "./credential-lease-health.js";
import { createQaLeaseHealth } from "./credential-lease-health.js";
const DEFAULT_ACQUIRE_TIMEOUT_MS = 90_000;
const DEFAULT_HEARTBEAT_INTERVAL_MS = 30_000;
@ -414,7 +414,7 @@ export async function acquireQaCredentialLease<TPayload>(
opts.signal?.throwIfAborted();
attempt += 1;
try {
const acquiredAt = captureQaLeaseClock();
const acquiredAt = Date.now();
const payload = await postConvexBroker(config, fetchImpl, "acquire", {
kind: opts.kind,
ownerId: config.ownerId,
@ -475,7 +475,7 @@ export async function acquireQaCredentialLease<TPayload>(
payload: parsedPayload,
async heartbeat() {
health.assertHealthy();
const requestStartedAt = captureQaLeaseClock();
const requestStartedAt = Date.now();
const heartbeatPayload = await postConvexBroker(config, fetchImpl, "heartbeat", {
kind: opts.kind,
ownerId: config.ownerId,

51
extensions/qa-lab/src/live-transports/shared/live-gateway-config.runtime.test.ts
View file

@ -17,45 +17,6 @@ describe("live QA gateway config", () => {
);
});
it("re-reads the config once after a stale patch", async () => {
const waitForConfigRestartSettle = vi.fn();
const call = vi
.fn()
.mockResolvedValueOnce({ config: { channels: {} }, hash: "old" })
.mockRejectedValueOnce(new Error("config changed since last load"))
.mockResolvedValueOnce({ config: { channels: {} }, hash: "current" })
.mockResolvedValueOnce({ hash: "applied" })
.mockResolvedValueOnce({
hash: "applied",
configRevisionHash: "applied",
appliedConfigHash: "applied",
});
await patchLiveQaGatewayConfig({
gateway: { call },
patch: { channels: { slack: { enabled: true } } },
replacePaths: ["channels.slack"],
timeoutMs: 45_000,
waitForConfigRestartSettle,
});
expect(call).toHaveBeenNthCalledWith(
4,
"config.patch",
{
raw: JSON.stringify({ channels: { slack: { enabled: true } } }, null, 2),
baseHash: "current",
replacePaths: ["channels.slack"],
restartDelayMs: 0,
},
{ timeoutMs: 60_000 },
);
expect(waitForConfigRestartSettle).toHaveBeenCalledWith({
restartDelayMs: 0,
timeoutMs: 45_000,
});
});
it("waits for the active Gateway to apply the persisted config revision", async () => {
vi.useFakeTimers();
const waitForConfigRestartSettle = vi.fn();
@ -77,6 +38,7 @@ describe("live QA gateway config", () => {
const patching = patchLiveQaGatewayConfig({
gateway: { call },
patch: { channels: { slack: { enabled: true } } },
replacePaths: ["channels.slack"],
timeoutMs: 45_000,
waitForConfigRestartSettle,
});
@ -85,6 +47,17 @@ describe("live QA gateway config", () => {
await patching;
expect(call).toHaveBeenCalledTimes(4);
expect(call).toHaveBeenNthCalledWith(
2,
"config.patch",
{
raw: JSON.stringify({ channels: { slack: { enabled: true } } }, null, 2),
baseHash: "current",
replacePaths: ["channels.slack"],
restartDelayMs: 0,
},
{ timeoutMs: 60_000 },
);
expect(waitForConfigRestartSettle).toHaveBeenCalledTimes(1);
});

68
extensions/qa-lab/src/live-transports/shared/live-gateway-config.runtime.ts
View file

@ -1,14 +1,9 @@
import { setTimeout as sleep } from "node:timers/promises";
import { formatErrorMessage } from "openclaw/plugin-sdk/error-runtime";
type GatewayConfigClient = {
call: (method: string, params?: unknown, options?: { timeoutMs?: number }) => Promise<unknown>;
};
function isStaleConfigPatchError(error: unknown) {
return formatErrorMessage(error).toLowerCase().includes("config changed since last load");
}
async function waitForLiveQaGatewayConfigApplied(params: {
expectedHash: string;
gateway: GatewayConfigClient;
@ -77,44 +72,33 @@ export async function patchLiveQaGatewayConfig(params: {
timeoutMs: number;
}) => Promise<void>;
}) {
for (let attempt = 0; attempt < 2; attempt += 1) {
const snapshot = await readLiveQaGatewayConfig(params.gateway);
let patchResult: { hash?: string; noop?: boolean };
try {
patchResult =
((await params.gateway.call(
"config.patch",
{
raw: JSON.stringify(params.patch, null, 2),
baseHash: snapshot.hash,
...(params.replacePaths?.length ? { replacePaths: params.replacePaths } : {}),
restartDelayMs: 0,
},
{ timeoutMs: 60_000 },
)) as { noop?: boolean } | null | undefined) ?? {};
} catch (error) {
if (attempt === 0 && isStaleConfigPatchError(error)) {
continue;
}
throw error;
}
if (patchResult.noop !== true) {
await params.waitForConfigRestartSettle({
// The suite owns this Gateway's config; concurrent operator writes fail visibly.
const snapshot = await readLiveQaGatewayConfig(params.gateway);
const patchResult =
((await params.gateway.call(
"config.patch",
{
raw: JSON.stringify(params.patch, null, 2),
baseHash: snapshot.hash,
...(params.replacePaths?.length ? { replacePaths: params.replacePaths } : {}),
restartDelayMs: 0,
timeoutMs: params.timeoutMs,
});
if (!patchResult.hash) {
throw new Error("live QA config patch returned no persisted hash");
}
// Restart-required writes acknowledge before SIGUSR2 completes. The old
// Gateway can still look healthy, so require the active runtime revision.
await waitForLiveQaGatewayConfigApplied({
expectedHash: patchResult.hash,
gateway: params.gateway,
timeoutMs: params.timeoutMs,
});
},
{ timeoutMs: 60_000 },
)) as { hash?: string; noop?: boolean } | null | undefined) ?? {};
if (patchResult.noop !== true) {
await params.waitForConfigRestartSettle({
restartDelayMs: 0,
timeoutMs: params.timeoutMs,
});
if (!patchResult.hash) {
throw new Error("live QA config patch returned no persisted hash");
}
return;
// Restart-required writes acknowledge before SIGUSR2 completes. The old
// Gateway can still look healthy, so require the active runtime revision.
await waitForLiveQaGatewayConfigApplied({
expectedHash: patchResult.hash,
gateway: params.gateway,
timeoutMs: params.timeoutMs,
});
}
throw new Error("live QA config patch exhausted retries");
}

11
extensions/qa-lab/src/live-transports/whatsapp/adapter.runtime.test.ts
View file

@ -262,6 +262,17 @@ describe("WhatsApp QA adapter cleanup through the suite", () => {
await vi.advanceTimersByTimeAsync(100);
expect(mocks.heartbeat).not.toHaveBeenCalled();
}
if (
failureAt === "driver close" ||
failureAt === "heartbeat stop" ||
failureAt === "lease release"
) {
await expect(transport.cleanupWithoutGateway()).rejects.toBe(failure);
expect(mocks.closeDriver).toHaveBeenCalledOnce();
expect(mocks.heartbeatStop).toHaveBeenCalledOnce();
expect(mocks.release).toHaveBeenCalledOnce();
created = undefined;
}
},
);

21
extensions/qa-lab/src/qa-transport-registry.test.ts
View file

@ -247,27 +247,6 @@ describe("qa transport registry", () => {
expect(cleanupAfterGatewayStop).toHaveBeenCalledOnce();
});
it("shares a failed cleanup and permits one subsequent retry", async () => {
const failure = new Error("pre-cleanup failed");
const cleanup = vi.fn().mockRejectedValueOnce(failure).mockResolvedValue(undefined);
const created = await createCleanupAdapter(cleanup);
const attempts = await Promise.allSettled([
created.cleanupBeforeGatewayStop(),
created.cleanupBeforeGatewayStop(),
]);
expect(attempts).toEqual([
{ status: "rejected", reason: failure },
{ status: "rejected", reason: failure },
]);
expect(cleanup).toHaveBeenCalledOnce();
await created.cleanupBeforeGatewayStop();
await created.cleanupBeforeGatewayStop();
expect(cleanup).toHaveBeenCalledTimes(2);
});
it("runs post-gateway cleanup when gateway-less pre-cleanup fails", async () => {
const cleanup = vi.fn(async () => {
throw new Error("pre-cleanup failed");

Some files were not shown because too many files have changed in this diff Show more