openclaw/docs/reference/test/performance.md

doc-schema-version	summary	title	read_when
1	Import profiling, CPU and heap profiles, shard timings, and benchmark scripts	Test performance and benchmarks	You are profiling a slow test run You need a startup, gateway, or model latency benchmark

Test performance tooling

• pnpm test:perf:imports: enables Vitest import-duration + import-breakdown reporting, while still using scoped lane routing for explicit file/directory targets. pnpm test:perf:imports:changed scopes the same profiling to files changed since origin/main.
• pnpm test:perf:changed:bench -- --ref <git-ref> benchmarks the routed changed-mode path against the native root-project run for the same committed git diff; pnpm test:perf:changed:bench -- --worktree benchmarks the current worktree change set without committing first.
• pnpm test:perf:profile:main writes a CPU profile for the Vitest main thread; pnpm test:perf:profile:runner writes CPU + heap profiles for each unit worker. Both print their output directory (a temporary directory by default). Use -- --output-dir <dir> or OPENCLAW_VITEST_PROFILE_DIR to retain profiles at a chosen location.
• pnpm test:perf:groups --full-suite --allow-failures --output .artifacts/test-perf/baseline-before.json: runs every full-suite Vitest leaf config serially and writes grouped duration data plus per-config JSON/log artifacts. Full-suite reports isolate files by default so retained module graphs and GC pauses from earlier files are not charged to later assertions; pass -- --no-isolate only when intentionally profiling shared-worker accumulation. pnpm test:perf:groups:compare .artifacts/test-perf/baseline-before.json .artifacts/test-perf/after-agent.json compares grouped reports after a performance-focused change.
• Full, extension, and include-pattern shard runs update local timing data in .artifacts/vitest-shard-timings.json; later whole-config runs use those timings to balance slow and fast shards. Include-pattern CI shards append the shard name to the timing key, which keeps filtered shard timings visible without replacing whole-config timing data. Set OPENCLAW_TEST_PROJECTS_TIMINGS=0 to ignore the local timing artifact.
• pnpm ci:timings:refit: regenerate committed config/ci-test-timings.json from up to five completed scheduled main CI runs using their successful jobs; add --dry-run to preview the changed-entry table. This file owns per-file UI E2E and per-profile compact-group weights, unlike the gitignored .artifacts/vitest-shard-timings.json whole-config timing cache. Independent CI shards use only the committed weights, never that cache. See CI timing refits for the daily refresh and sampling rules.

Runner profiling preserves the selected forks or threads pool, isolation, environment, and custom runners extending Vitest's TestRunner. Capture starts in a Node preload before Vitest worker imports, spans all files assigned to that worker, and finishes both profile files in awaited worker cleanup before teardown is acknowledged. It does not depend on exit-time profile flushing. Root global setup configures every selected project without replacing its reporters or setup. Main capture spans Vitest/Vite startup through run completion and close. Process termination before cleanup, bootstrap failures before runner construction, and teardown timeouts can still prevent output. Browser/VM pools, custom runners without onCleanupWorkerContext, and additional native --cpu-prof/--heap-prof flags are rejected for runner profiling.

Forward Vitest options after the profiler separator. Forwarded options use Vitest's native CLI validation before loading config. Config-only settings, such as runner and globalSetup, belong in the Vitest config file, not CLI flags. For example:

pnpm test:perf:profile:runner -- --output-dir .artifacts/profiles -- --config test/vitest/vitest.unit.config.ts --pool threads

pnpm test:extensions:memory profiles built plugin index entries from dist/extensions (including nested dist output) and package-local extensions/<id>/dist output; TypeScript source entries are excluded. Root artifacts take precedence when both builds exist. Selecting an already-built plugin with --extension <id> reuses its output without requiring unrelated plugin builds; build the plugin package first if its output is not supplied by pnpm build.

Native imports also need the plugin's declared dependencies and a resolvable openclaw host package. The profiler does not install or link dependencies: missing dependencies remain import failures in the JSON report and cause a nonzero exit.

The report labels this cold-import screening, not plugin activation, Gateway startup, or workload coverage. Each fresh child reports process user/system CPU in microseconds through its exit hook and process peak RSS in MiB (the existing maxRssMb field). CPU and peak RSS come from one observation bound to the importing child PID; inherited descendant output cannot replace it. CPU totals reconcile with user plus system counters. Deltas subtract the same empty-child baseline, retain negative values, and must not be summed across plugins. The combined-import case measures a separate process with shared imports. Explicit process exit is not proof of plugin lifecycle cleanup or memory retention.

Source commit/tree and tracked cleanliness, build metadata and selected entry hashes are observed before and after the run. Child Node/V8/ABI/platform/architecture identify the measured runtime. These snapshots do not attest the transitive dependency closure or prove that build metadata describes every generated byte. Package-local builds without their own provenance remain unqualified. Schema 2 preserves the existing repoRoot and absolute results[].file fields for saved-report readers and adds results[].relativeFile for portable screening. Entry provenance paths are repository-relative. Raw reports still contain local paths and captured diagnostics; use the screening consumer’s public projection or redact them before sharing.

qualification.qualified admits only complete cold-import snapshots with matching source/build declarations, stable selected inputs, resource counters, a child-recorded completion after every awaited import, and verified process-group/output and temporary-home cleanup. Missing identity or cleanup proof stays explicit in qualification.gaps; screening consumers must reject unqualified reports even when the observation command exits successfully. Windows currently verifies child/output closure without descendant-group proof, so those observations remain unqualified. Import failures (including exit zero before the sequence completes), missing RSS, timeouts, and cleanup failures produce their nonzero command outcome.

Kitchen Sink Gateway resource comparison

The existing Kitchen Sink RPC walk can compare a fresh Gateway with plugins disabled against a fresh Gateway with only Kitchen Sink conformance active:

OPENCLAW_KITCHEN_SINK_NPM_SPEC=npm-pack:/fixtures/kitchen-sink.tgz \
  pnpm test:plugins:kitchen-sink-rpc -- --resource-profile /out/resources.json

This mode requires Linux Node with process.threadCpuUsage, a built OpenClaw entry in the current package root, dist/build-info.json with a full source commit, and a local npm-pack fixture. It does not download a floating fixture. The normal RPC walk, including its Bun command, is unchanged.

Run only in a prepared secretless container or remote runner. For example, bake the frozen built host, its dependencies and the pinned fixture into a reviewed image, then execute with explicit limits and networking disabled:

docker run --rm --network none --memory 4g --cpus 2 --pids-limit 512 \
  -v "$PWD/proof-output:/out" -w /app \
  -e OPENCLAW_KITCHEN_SINK_NPM_SPEC=npm-pack:/fixtures/kitchen-sink.tgz \
  <prepared-image> \
  node --import ./scripts/tsx.mjs scripts/e2e/kitchen-sink-rpc-walk.mts \
  --resource-profile /out/resources.json

Prepare any managed npm installation prerequisites in the image; an offline install failure is blocked proof, not permission to copy credentials or enable network access. The harness supplies a minimal child environment, but does not enforce container isolation itself. Preserve the image digest and runner limits alongside the report.

Each case records startup from the initialized measurement preload to HTTP readiness, one unmeasured health warmup, a 250 ms idle window, 20 completed health RPCs, and a 250 ms post-work window. The conformance case additionally measures one asserted session-create, then 20 asserted kitchen_sink_text calls with unique idempotency keys, followed by another observation window. The plugin-tool aggregate retains all 20 calls. Its nested breakdown separates plugin-tool-first (the original first call) from plugin-tool-warm (the remaining 19); no call is discarded or added as warmup. Both children share the same raw midpoint snapshot, and the aggregate uses the original outer snapshots. Do not sum the aggregate and its children. The aggregate includes midpoint sampling overhead; these are whole-Gateway costs in an already started host, not isolated schema construction, cold loading or plugin allocation costs. Neither session creation nor plugin tools have an empty-host subtraction.

Setup and package installation are outside measured phases. Failed measured calls are not retried. Missing or invalid boundary samples fail the observation without changing the completed-call count; an unavailable midpoint stops work before warm calls. A later failure preserves the successful first-call receipt.

After those matched phases, the conformance case uses kitchen.resources to verify ten million CPU iterations and a fixed checksum, hold a 16 MiB Buffer, and start a referenced 10 ms timer. Timer progress is observed with bounded status checks. Each calibration operation counts one asserted control step, including its checks; it is not an RPC throughput count. The fixture must include the calibration controls from Kitchen Sink commit 051db418820c2f0a73f8d349c88eb002b3c2d6e2 or later. Older fixtures fail visibly instead of skipping calibration.

The harness resets the controls, asserts empty owner state, then reacquires the Buffer and timer before plugins.setEnabled disables the fixture. It requires an applied runtime receipt, no restart or cleanup warnings, an inactive catalog, and before/after samples from the same Gateway PID. This gives an in-process retirement observation before normal Gateway shutdown. Forced termination or a nonzero Gateway exit fails the report even when the process group is gone.

The report preserves raw phase-boundary snapshots, completed/failed operation counts, provenance hashes and signed conformance-minus-empty deltas. CPU counters cover the Gateway process and main thread, excluding separate child processes. RSS is process-wide; other memory fields describe the main isolate. ArrayBuffers overlap external memory. Boundary samples are not peaks, and no forced GC occurs. Active-resource histograms count the types keeping the event loop alive, not plugin ownership or all live objects. The report labels CPU, held-memory and timer signals as observed or inconclusive; unrelated collection or host timers can obscure them. Reset proves owner-state release, not immediate RSS reclamation. The fixed post-work window is not a plugin drain receipt. Host shutdown is checked separately; post-process-exit sampling and guaranteed reclamation remain unsupported. These observations establish neither a leak nor a budget violation. Repeat comparable pairs through the campaign owner before drawing performance conclusions; do not sum individual plugin costs.

Plugin coverage inventory

Export the source plugin inventory before a profiling campaign:

pnpm --silent plugins:inventory:json > plugin-inventory.json
pnpm --silent plugins:inventory:json --commit <full-commit-sha> > plugin-inventory.json

The command reads committed Git objects, defaults to HEAD, and needs Git 2.45 or newer. It works without installed dependencies or a complete working tree. Local edits do not affect the export. Missing objects fail the command; it does not fetch them or load plugin code.

The versioned JSON includes the commit and tree, plugin IDs, relative paths, package metadata, distribution classes, and declared surfaces. It shares the documentation inventory's collector and distribution rules but excludes docs-only external seeds. sha256 hashes the payload without that field using the shared stable JSON serializer (object keys sorted, array order preserved). Declaration in this inventory is not evidence that a workload was exercised. Keep import, registration, workload, and cleanup coverage separate when joining profiling results to this inventory.

Reusing the resource host

Source-checkout campaign tools can import resolveResourceGatewayRuntime and runResourceGatewayCase from scripts/e2e/kitchen-sink-rpc-walk.mts. Kitchen Sink uses this same host lifecycle. Run from one frozen, built OpenClaw package root per process; this is a testing interface, not a published plugin SDK API.

The preparation callback receives an isolated config path, loopback port, test token and a local-archive installer that verifies the supplied SHA-256. Enable only the selected plugins there. The workload callback receives authenticated CLI-mode RPC calls, resource snapshots and counted measure(name, count, run) phases. Assert the active plugin inventory and operation results in the workload; registration or a successful transport response alone does not establish coverage. Pass { splitFirst: true } as the fourth measure argument to retain a nested first/warm breakdown with shared boundary samples and unchanged operation indices.

The host records startup, preserves failed phases and joins Gateway shutdown before checking service-stop logs. A failed workload, nonzero exit, attempted forced cleanup or shutdown error retains temporary state and fails the case. The campaign owns bounded callback deadlines, mock-service cleanup, runner isolation, repetitions and report publication. Keep mock-service measurements separate from Gateway observations and preserve host, archive and harness hashes. On an outer timeout, the runner must terminate and join the whole container or cgroup: the Gateway has its own process group, so killing the campaign process alone does not clean it up.

Zod schema compilation

Compile individual schemas only after measuring a repeated validation path. Use the pinned Zod package's z.compile(schema) API and retain the compiled schema at its existing owner. The zod/compile side-effect import enables a process-wide hook and is unsuitable for a selective optimization.

Nested tool activity validation compiles its schema on the first matching read or creation and reuses it across Gateway turns. Ordinary transcript rows bypass that compilation. Config, transcript entry, and browser relay schemas retain their existing parsers because their measured caller costs did not justify compilation.

Measure compilation and the first operation separately from warmed operations. Compare valid, invalid, and mixed inputs through the actual caller, including any JSON decoding, copying, or context construction it performs. For predicates that discard parsed output, compare ordinary safeParse(...).success, z.validate(schema, input), and validation with a compiled schema; avoiding error allocation can help independently of compilation.

Keep schemas shared with the strict-CSP Control UI uncompiled: explicit compilation attempts code generation even when jitless is set. Reusing a stable schema across calls is a separate optimization that needs no compiler.

Keep refinement and transform callbacks pure: an invalid compiled parse can fall back to the runtime parser and execute those callbacks twice. Default compilation preserves runtime fallback for unsupported schemas; async parsing and encoding keep their existing runtime behavior.

Benchmarks

This instrumented component benchmark calls the registered TypeSafe Decision provider against a deterministic loopback HTTP server. It checks normalized answers, completed request counts, accepted TCP connections, and explicit transport-pool teardown. No credentials or billed inference are used.

Prepare separate, clean source checkouts with their own frozen dependencies for the baseline and candidate. Overlay the same benchmark builder, worker, and scripts/lib/{benchmark-harness,arg-utils}.mts on both checkouts. Build once in each checkout, outside the measured interval:

node --import ./scripts/tsx.mjs scripts/build-decision-transport-bench.mts <source-sha>

Each build prints its manifest path after a two-request correctness smoke test. Run both manifests on the same otherwise idle host and Node version:

node scripts/bench-decision-transport.mts \
  --baseline <baseline-manifest.json> --candidate <candidate-manifest.json> \
  --runs 20 --requests 50 --json --output .artifacts/decision-transport.json

The driver alternates baseline and candidate in serial ABAB order. It verifies output hashes before and after the run and rejects unequal worker/helper bytes. The report includes per-sample first-evaluation and warm-request latency, process CPU, sampled RSS, and heap/external memory before work, after work, and after teardown. Warm latency summaries compare each sample's median, not pooled requests. Tail percentiles require at least 20 samples per revision.

For a connection-reuse comparison, require identical completed work and zero sockets after teardown. Fewer accepted connections establish reuse, not a latency win. Before measuring, use a 10% median latency reduction as the improvement threshold and a 20% regression in p95 latency, CPU, or sampled peak RSS as a reason to withhold a performance-win claim. Report raw samples and inconclusive results. Record runner CPU quotas separately; affinity and memory capacity are included, but the driver does not resolve CPU quotas.

Builds and eager imports are outside this workload. The first evaluation and total CPU include lazy client loading and validator compilation; warm timings exclude that initialization. CPU also includes the loopback server and answer checks. Model inference, TLS handshakes, and Gateway startup are not measured. RSS is sampled between requests and may miss short peaks. These results are not packaged Gateway or end-to-end agent performance evidence.

Measure SQLite history pages and the Gateway's bounded history reader with synthetic conversations, including sparse markers, dense markers, and resets:

pnpm test:sessions:history:bench --samples 30 --output history.json
pnpm test:sessions:history:bench --profile sparse,trailing,reset --operation recent --analyze --samples 15 --output history-analyzed.json

The second command includes 5,000 trailing compaction markers and refreshes SQLite planner statistics before reading. Compare both statistics states when changing a query. --operation selects recent, page, or gateway-tail; omitting it measures all three.

Each reader runs in a fresh process. Reports separate imports, the first read (including database open), and warm p50/p95 wall and CPU time. OS caches are not flushed. SQL plans, statement counts, rows delivered to JavaScript, and JSON parsing counts come from a separate instrumented read. Heap deltas are uncollected observations, not total allocations. Fixtures are removed afterward.

pnpm tsx scripts/bench-model.ts --runs 10

Optional env: MINIMAX_API_KEY, MINIMAX_BASE_URL, MINIMAX_MODEL, ANTHROPIC_API_KEY. Default prompt: "Reply with a single word: ok. No punctuation or extra text."

pnpm test:startup:bench
pnpm test:startup:bench:smoke
pnpm test:startup:bench:save
pnpm test:startup:bench:update
pnpm test:startup:bench:check
pnpm tsx scripts/bench-cli-startup.ts --runs 12
pnpm tsx scripts/bench-cli-startup.ts --preset real --case status --case gatewayStatus --runs 3
pnpm tsx scripts/bench-cli-startup.ts --entry openclaw.mjs --entry-secondary dist/entry.js --preset all
pnpm tsx scripts/bench-cli-startup.ts --runtime-rss --case status --runs 3

Presets:

• startup: --version, --help, health, health --json, status --json, status
• real: health, status, status --json, sessions, sessions --json, agents list --json, gateway status, gateway status --json, gateway health --json, config get gateway.port
• all: both presets combined

Output includes sampleCount, avg, p50, p95, min/max, exit-code/signal distribution, and RSS per command. The maxRssMb fields use MiB. By default, RSS uses the last preload marker received on stderr, preserving the historical fixture's attribution. A respawning launcher can supply that last marker. Default reports omit memoryMetric and sample memory; no runtime identity or temporary observation files are required. For a silent command, the exit marker can count as first output.

Pass --runtime-rss to opt into runtime-process attribution. primary.memoryMetric identifies cli-runtime-max-rss-v1, and each sample's memory records PID, parent PID, role, and high-water RSS in bytes. The runtime is the terminal process in a unique matching CLI invocation chain; launcher and auxiliary observations are not added to it. This is not simultaneous process-tree memory.

High-water RSS is observed when the preload's exit listener runs. Allocations in later application exit handlers are outside this observation; this is not a full-lifetime OS measurement.

With --runtime-rss, the preload records observations in temporary files, separate from stdout/stderr and first-output timing. Runtime identity does not depend on command output; a silent entry has firstOutputMs: null. Missing or ambiguous runtime identity fails the sample only in this opt-in mode. Both modes are instrumented launches, separate from the no-preload, no-respawn scripts/check-cli-startup-memory.mjs diagnostic. --cpu-prof-dir / --heap-prof-dir write V8 profiles per run.

Saved-report comparison uses report metadata, not --runtime-rss. Comparison, enforced fixture budgets, and source-summary memory trends reject mixed legacy/runtime metrics rather than silently comparing different processes. Historical fixtures cannot be relabeled; any replacement baseline needs separate validation and approval.

Saved output: pnpm test:startup:bench:smoke writes .artifacts/cli-startup-bench-smoke.json; pnpm test:startup:bench:save writes .artifacts/cli-startup-bench-all.json (runs=5 warmup=1). Checked-in fixture: test/fixtures/cli-startup-bench.json, refreshed by pnpm test:startup:bench:update, compared by pnpm test:startup:bench:check.

Gateway startup, restart, and agent concurrency benchmark fixtures use temporary home and state directories, loopback binding, and discovery.mdns.mode: "off" so synthetic Gateways do not advertise on the LAN, including on macOS.

Defaults to the built CLI entry at dist/entry.js; run pnpm build first. Pass --entry scripts/run-node.mjs to measure the source runner instead, and keep those results separate from built-entry baselines.

Startup, restart, and concurrency accept --gateway-runtime <executable> to select only the Gateway executable, leaving the benchmark controller and mock provider on the runtime that launched the script. The default is that same executable. Use an absolute path to select a specific staged runtime. JSON reports record this selection as gatewayRuntime; it is not a measured child version or binary hash. Failed activity-summary diagnostics omit this caller-supplied path, like the entry path; that mode is not performance proof. The existing restart report's node field describes the controller.

On Linux, all three accept --gateway-cpus 0,1,2,3, applied through taskset only to the Gateway. Pin the controller separately when comparing runtimes:

taskset --cpu-list 4-31 node --import ./scripts/tsx.mjs scripts/bench-gateway-startup.ts --case default --gateway-runtime /path/to/bun --gateway-cpus 0,1,2,3 --runs 1 --warmup 0 --output .artifacts/gateway-startup-bun.json

The installed-package --installed-cohort mode rejects both selectors; its input owns the attested runtime. Profiler options remain runtime-specific and require the selected runtime to support their profiling APIs.

pnpm test:startup:gateway -- --runs 5 --warmup 1
pnpm test:startup:gateway -- --case skipChannels --case fiftyPlugins --runs 5
node --import tsx scripts/bench-gateway-startup.ts --case default --runs 5 --output .artifacts/gateway-startup.json
node --import tsx scripts/bench-gateway-startup.ts --case incidentCombined --runs 5 --warmup 1 --timeout-ms 60000 --output .artifacts/gateway-startup-incident.json

Case ids: default, skipChannels (channel startup skipped), oneInternalHook, allInternalHooks, fiftyPlugins (50 manifest plugins), fiftyStartupLazyPlugins (50 startup-lazy manifest plugins), incidentDatabase, incidentNullMetadata, incidentWorkspace, incidentPackagedPlugins, and incidentCombined.

The incident cases are opt-in because each sample builds an isolated, non-sensitive load fixture: current global and agent databases, 100,000 retained audit rows with freelist fragmentation, eight agent workspaces containing 80,000 files (about 800 MB), and the packaged plugin inventory. Run the combined case only on a clean machine with enough free disk space; the fixture directory is removed after each sample. incidentCombined fails when /healthz p95 reaches 30 seconds or /readyz p95 reaches 60 seconds.

Output includes first process output, /healthz, /readyz, HTTP listen log time, Gateway ready log time, CPU time, CPU core ratio, max RSS, heap, startup trace metrics, event-loop delay, and plugin lookup-table detail metrics. The script sets OPENCLAW_GATEWAY_STARTUP_TRACE=1 in the child Gateway environment.

/healthz is liveness (HTTP server can answer). /readyz is usable readiness (startup plugin sidecars, channels, and ready-critical post-attach work have settled). Startup hooks dispatch asynchronously and are not part of the readiness guarantee. Ready log time is the Gateway's internal timestamp, useful for process-side attribution but not a substitute for the external /readyz check.

Use JSON output or --output when comparing changes. Use --cpu-prof-dir only after trace output points at import, compile, or CPU-bound work that phase timings alone cannot explain.

After readiness, the Gateway uses idle turns to prepare common Control UI handler modules and configured local agent skill discovery. Each item yields to admitted foreground work, and shutdown joins preparation that has already started. This work does not execute chat requests, create connection state, or fetch live provider catalogs. Context-window cache preparation shares this sequence and starts no earlier than five seconds after scheduling. Compare immediate first requests with requests after an idle interval; readiness alone does not guarantee every optional cache is warm.

Workspace inventory inspection, raw manifest parsing and comparison, and bulk file hashing use a bounded computation pool on the Gateway and node. Decoded manifests are compared synchronously against typed entries without another inventory copy. Reconciliation prepares the source difference once and sends changed nodes and their ancestors to each preflight. Each preflight still inspects the target filesystem again before mutation or acceptance; it does not reuse local node state. Independent workspaces can progress while the main thread serves requests. The host owns session authority, workspace mutations, Git subprocesses, and durable acceptance; cancellation joins outstanding work before cleanup.

Node capture reuses this pool and its compiled manifest program rather than starting a process per verification. Each capture has a private workspace, hash memo, and limits; the node host owns Git inventory commands. For acceptance and recovery ordering, see session lifecycle.

Compare workspace inventory, manifest capture, and result preparation against a frozen source checkout with its own installed dependencies:

node --import ./scripts/tsx.mjs scripts/bench-workspace-computation.ts \
  --baseline /path/to/baseline-checkout \
  --scenarios inventory,manifest,delta,unchanged \
  --sizes 1000,32000,100000 --concurrency 1,4 \
  --runs 3 --warmup 1 --output .artifacts/workspace-computation.json

Use --changed-files 2000 --scenarios delta --sizes 32000 to include a larger changed payload, and --file-bytes to vary the content hashed during capture. The default workload is smaller: 1,000 entries and one concurrent operation.

The benchmark checks identical inventory bytes, manifest references, and changed payloads. Separate processes measure first invocation and warm throughput, CPU, event-loop delay, memory, and HTTP latency from an external check. Worker task diagnostics distinguish queueing, input preparation, transfer, and execution. The HTTP check measures responsiveness of the computation's owning process; paired-node wire tests provide the full Gateway dispatch and reconciliation proof.

Use --scenarios reconcile --sizes 20000 --changed-files 100 --file-bytes 128 to measure sparse checkpoint updates through real Git patch application and accepted-manifest verification. This scenario creates real files sequentially and uses an in-memory journal adapter. It does not simulate a watcher trigger or measure a full Gateway. --sampling-interval 32768 samples main-isolate allocations, including collected objects; run CPU comparisons separately with sampling disabled.

For externally coordinated comparisons, seed each side with --prepare-root <dir>, then measure using --reuse-root <dir> and matching workload options. --start-at sets the epoch time for the first warm measurement, after cold and warmup calls. --source-commit <sha> permits a single archived source tree without Git metadata; the report labels that revision as operator-supplied and unverified.

For connected clients and fixed-duration closed-loop requests, see Control UI protocol load. That mode has separate client counts, coordinated driver processes, reply correlation, and Linux resource accounting.

Runs synthetic streaming agent turns in parallel sessions on one isolated Gateway. Add tool calls, session history, observers, and control-plane checks to reproduce allocation pressure from a busy Gateway. Build with pnpm build first. The default mock provider needs no key. Dreaming is disabled in this isolated benchmark; ordinary indexing, recaps, and database idle retention keep their normal settings.

If a sample fails, --output and --json retain completed measured runs, completed warmupRuns, and a failedAttempt with its phase, one-based index, available observations, and workload, diagnostics, or cleanup errors. Failed attempts and warmups do not enter aggregate summaries. Null or absent partial measurements mean unavailable, not zero. The command still exits with status 1; live and activity-summary diagnostic failure sidecars remain available.

pnpm test:gateway:concurrency -- --concurrency 16 --tool-events --workspace-fanout --session-count 100 --history-messages 20 --history-clients 4 --subscribers 4 --visible-observer --control-plane --heap-prof-dir .artifacts/gateway-heap --output .artifacts/gateway-concurrency.json
pnpm test:gateway:concurrency -- --concurrency 64 --turns-per-session 8 --tool-events --timeout-ms 600000 --heap-prof-dir .artifacts/gateway-sustained-heap --output .artifacts/gateway-sustained.json

Use --provider openai with OPENAI_API_KEY supplied in the environment for real OpenAI turns:

pnpm test:gateway:concurrency -- --provider openai --runs 1 --warmup 0 \
  --agent-warmup-turns 0 --agent-count 32 --concurrency 32 --turns-per-session 3 \
  --session-count 1000 --history-messages 20 --history-message-chars 1024 \
  --probe-rounds 64 --cadence-ms 100 --session-updates 100 \
  --session-update-clients 2 --history-clients 2 --history-burst 2 \
  --subscribers 4 --control-plane --timeout-ms 120000 \
  --load-cpu-prof-dir .artifacts/gateway-live-cpu \
  --output .artifacts/gateway-live.json

Live mode uses a fixed OpenAI model, denies tools, and limits output to 128 tokens. It permits one run with no warmups and at most 96 turns, and checks streamed replies, terminal receipts, history, and persisted replies after shutdown. It does not report synthetic provider request counts. CPU profiles are instrumented observations; keep them separate from unprofiled latency measurements. The manual workflow runs this workload with repository-managed credentials.

--concurrency controls parallel sessions; --turns-per-session controls serial turns in each session (default 1, maximum 100). The second example completes 512 turns across 64 sessions. Each session starts its next turn as soon as its previous turn completes, retaining its conversation history and workspace; there is no barrier between rounds. The fresh-connection check runs once after every session has started its first turn. --tool-events requires a matching successful exec result and the expected visible final reply on every turn, including follow-ups. Missing or duplicate tool evidence fails the run. The load timeout bounds turns and checks; startup, setup, and check warmup have separate budgets. Health/control sampling is capped at 2,048 samples, while heap sampling continues until the full workload finishes.

The fixture gives the utility model its own structured mock response, preserving the agent model's automatic tool loop. turnEvidence.observerModelDigestTurns counts turns with a published model-derived observer digest. A short run can legitimately report zero; observer correctness proof requires a positive count.

mockRequests retains six mock-server counter checkpoints and their parent monotonic request bounds. Ingress deltas cover startupAndWarmup (readiness, connect, visibility, and check warmup), setup, agentWarmup (optional agent turns on the same Gateway), loadBracket, and postLoad (through Gateway shutdown). The agentWarmup bracket remains present when --agent-warmup-turns is zero (the default); warmup turns are excluded from measured load. These deltas distinguish Responses, Chat Completions, embeddings, and other routes, including rejected request bodies; health and model-catalog reads are excluded. These HTTP brackets are not exact CPU capture windows or causal attribution. selections through the final checkpoint separately count model/global controlled responses and automatic tool/text branches, not completed responses. Auxiliary model requests remain included; neither total is a count of agent turns. Missing, regressing, or replaced-server checkpoints fail instead of becoming zero. Reports retain counters, not raw prompts or request bodies. The mock request journal lives in a separate telemetry directory so its writes do not feed the Gateway's config and skill ancestor watchers during measurement.

--agent-count N distributes the same session inventory round-robin across 1–128 configured agents. It defaults to one agent and cannot exceed the larger of --session-count and --concurrency. Increasing it does not add sessions or turns. Multi-agent runs use main, bench-agent-2, and subsequent IDs with separate workspaces. --workspace-fanout still assigns a distinct workspace per session. Separate browser click targets remain on main.

For example, --agent-count 32 --session-count 1000 --concurrency 32 --turns-per-session 3 seeds 1,000 sessions and completes 96 turns, three per agent. Before the load window and its CPU/allocation profiling, multi-agent runs require each agent's current published configured model through models.list, then verify its complete seeded inventory and reported per-agent SQLite path through all pages of scoped sessions.list reads. This checks the Gateway's reported storage route; it does not independently inspect database files. agentCoverage.beforeLoad retains the model response and every session page. activeTurnAgentIds and completedTurns distinguish the agents handling turns from the configured roster: 128 configured agents with 16 parallel sessions does not mean 128 agents handled turns. Multi-agent history check rows retain sessionKey, which maps successful requests to the independently verified store inventory. These extra setup reads do not run in the default one-agent case.

Use --probe-rounds N for allocation comparisons with equal check work. It attempts exactly N sampler rounds and N history bursts per configured history client, regardless of which finishes first. Each sampler round requests /readyz, the Control UI, and sessions.list; --control-plane adds one each of cron.list and cron.status. Enabling --subscribers adds one subscribe attempt per round and an unsubscribe after each successful subscription. History attempts total N × historyClients × historyBurst, capped at 2048 per run. Slow clients receive the same history budget as fast clients. Failed checks remain recorded failures; counts describe attempts, not successes. Omitting the flag retains adaptive checking until agent turns and mutations end.

Fixed checks can finish before or after agent turns. Every configured workload joins before final memory and allocation capture; an exhausted load deadline fails the run instead of reporting a partial fixed workload as complete. Output records the mode and requested counts in probeWorkload; actual sampler and history counts remain in summary.sampleCount and summary.historySampleCount. Peak RSS is sampled during sampler rounds plus the final memory observation. If those rounds finish early, a later transient RSS peak can be missed; this is not continuous peak-RSS coverage of the entire agent workload. Equal request counts do not equalize their overlap with agent turns or the Gateway's time-dependent background work.

Each run's cpuUsage records user, system, and total CPU milliseconds for the Gateway process and its main thread. Two private IPC snapshots bound the load after setup and profiler activation through completion of all configured work, before the final memory check, profile export, and teardown. Their child-side monotonic timestamps define wallMs; CPU time can exceed wall time when threads run in parallel. Ordinary runs do not connect an inspector or start a profiler.

The process counters include Workers and native threads, including Workers that exit during the load, but exclude separate child processes, the mock provider, and the browser. Main-thread CPU is part of process CPU; do not add them. Their difference estimates work on other threads, with small skew from reading the counters sequentially. The summary reports gatewayProcessCpuMs, gatewayProcessCpuMsPerTurn, gatewayMainThreadCpuMs, and gatewayProcessCpuCoreRatio. CPU per turn includes the configured concurrent checks and mutations. Compare fixed workloads and identical profiling settings; moving work to Workers can improve responsiveness without reducing process CPU. The existing cpuCoreRatio summary remains sampled readiness-window data.

To measure clicking an existing session in the Control UI sidebar during load, build the UI and install Playwright Chromium, then enable the browser check:

pnpm ui:build
pnpm --dir ui exec playwright install chromium
pnpm test:gateway:concurrency -- --session-count 1000 --concurrency 16 --turns-per-session 2 --browser-session-clicks 3 --browser-history-messages 80 --timeout-ms 240000 --no-diagnostics-timeline --output .artifacts/gateway-session-clicks.json

--browser-session-clicks defaults to 0 and accepts up to 20 first visits, followed by one revisit to a recent pane. The check seeds separate idle click targets after the inventory. --browser-history-messages defaults to 80 per target (maximum 500), independent of --history-messages, so a large inventory does not require history in every session. --history-message-chars also sizes the browser targets' synthetic Markdown. The browser uses the built assets in dist/control-ui; PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH can select an installed Chromium executable.

browser.inventory records requested load and click target counts alongside the authoritative unarchived and retained session counts before load, including the click targets. Retained counts include archived sessions; normal inventory maintenance can reduce the unarchived count during seeding.

Each run's browser.clicks records time from the actual browser click event to the target pane's visible history, enabled composer, and successful transcript load followed by a paint opportunity (readyMs). These timings exclude Playwright actionability waits. RPC windowStartMs offsets begin before that wait; latencyMs measures the observed request/response round trip. Request records include socket identity, method, session key, success, error, and response bytes. An inherited request began before the click window and has a negative windowStartMs; its latency includes the earlier wait. Completed click records remain unchanged when a later window observes the response. The check includes message subscribe/unsubscribe requests to expose subscription recovery waits. connections records observed socket, hello, and outer event sequence gaps; negative offsets include recent setup events. Close events report observed inbound silence, without inferring a close reason. paneStates records changes to the selected pane's loading, readiness, connection epoch, and rendered history error. A displayed history failure retains visible-history/composer timings but fails the click instead of counting as ready. Browser longTasks contains the milliseconds each observed long task overlaps the click-to-ready interval, separate from the Gateway's Node CPU, event-loop, and memory samples. Pre-load /new and initial-session timings stay outside the click summary. Compare first visits and cached revisits separately, and check activeLoadAtStart, activeLoadAtFinish, and samplesOutsideActiveLoad before attributing latency to concurrent work. A recorded click failure makes the benchmark exit unsuccessfully after writing its report.

--heap-prof-dir samples allocations in the Gateway's main V8 isolate, starting after startup, session seeding, and check warmup. Sampling ends after the load and its final memory check, before profile serialization and teardown. It uses a 32 KiB sampling interval and includes objects collected by both minor and major GC, so sampledAllocatedBytes estimates gross allocations rather than retained heap. Each run records its .heapprofile path and the twenty largest allocation stacks with scope: "main-isolate"; open the raw file in the Chrome DevTools Memory panel. Worker isolates have separate profiles, described below. Native allocations are outside these V8 profiles.

The summary includes sampled allocation bytes per run and per completed turn. The per-turn figure also includes concurrent checks and session mutations; compare identical workload settings and Node versions across multiple runs. Initial and follow-up turns overlap across sessions, so the allocation profile covers their combined workload rather than attributing separate cold and warm allocations. Compare matched one-turn and sustained runs to study reuse. Sampling is statistical and adds overhead. Use unprofiled runs for latency comparisons. Existing heap/RSS measurements are taken before exporting the profile. --cpu-prof-dir remains available separately and includes startup; the recorded loadWindow identifies the measured interval in that CPU profile.

For CPU attribution during concurrent work, including on Windows, add --load-cpu-prof-dir .artifacts/gateway-load-cpu. This captures the Gateway's main V8 isolate at a 1 ms sampling interval after setup and through the final memory check. The private benchmark IPC channel stops the profiler and writes the .cpuprofile before process teardown, without depending on signal-driven profile flushing. Each run's loadCpuProfile records its path, duration, and sample count with scope: "main-isolate"; open the raw profile in Chrome DevTools. Profiled runs add overhead, so keep them separate from latency comparisons. --cpu-prof-dir retains its startup-inclusive native profiling behavior. --load-cpu-prof-dir and --heap-prof-dir require separate runs so exporting one profile cannot contaminate the other capture.

Both load-profile flags also capture observed Worker isolates over the existing private inspector connection. The main profile summary links workersManifestPath, a .workers.json manifest beside the main profile. Its rows distinguish native threadId from inspectorWorkerId and link each Worker's profile. completed: true means that profile was written; inspect row-level errors for missing identity, retired Workers, or other incomplete captures. A successful profiling command does not mean every Worker produced a usable profile. Workers are not paused at birth, so profiling can miss their earliest work. These files exclude separate child processes.

The Worker manifest's samples use performance.now in the Gateway process, with timestamps taken before asynchronous Worker reads. The 100 ms cadence is nominal; overlapping reads are coalesced. CPU counters are cumulative microseconds: difference observations for the same thread identity instead of summing samples. Those deltas cover each Worker's observed interval and can miss work before its first or after its last successful sample. The sample's memory.rss covers the process; other memory fields describe the main isolate, while each Worker's heap contains its own V8 heap statistics.

Capture windows differ: load CPU counters end before the final memory check; the main V8 profile includes that check; Worker profiles and samples also extend through main-profile serialization before their own stop. Use the raw CPU profiles' timestamps and manifest observations for attribution, and keep these windows separate from cpuUsage. Main-thread CPU plus observed Worker CPU does not account for every native thread or unobserved Worker interval. Profiling and periodic Worker inspection add overhead; compare equally instrumented runs.

Both load-phase profile modes also attach diagnostics and a diagnosticsPath to the run report. The private benchmark preload subscribes only during capture and aggregates the main isolate's redaction, session writer, session list, and worker-task completion records, including fast work below slow-log thresholds. It records counts, totals, and maxima in at most 256 groups; droppedEvents and collectionErrors report incomplete collection. No per-call record history, session or agent IDs, paths, message text, regex patterns, or exception text is retained in these aggregates. Worker artifact basenames identify task groups.

Redaction reports synchronous thread CPU separately from elapsed time and input UTF-16 character counts. These are inclusive measurements: nested redaction operations overlap, so do not add their times. Session writer timing separates queue wait, writer-held elapsed time, and completion delay. List records distinguish sessions.list from the initial sessions.subscribe snapshot and separate projection owners, in-flight followers, and completed cache hits. Worker-task records separate queue, preparation, run, and transfer measurements. Elapsed intervals can overlap across concurrent work and do not measure CPU.

The capture also aggregates main-isolate GC pause entries. Allocation profiles identify allocation sites, including collected objects; they do not establish which objects remain reachable or prove a leak. Raw profiles contain function names and source locations and need inspection before sharing.

These diagnostics channels have no collector in an ordinary Gateway. The benchmark uses isolated synthetic state, private process IPC, and no inspector listener. It does not attach to or modify an existing operator Gateway.

macOS and Linux only (uses SIGUSR2 for in-process restarts; fails immediately on Windows). Same built-entry default and --entry scripts/run-node.mjs override as gateway startup above.

pnpm test:restart:gateway -- --case skipChannels --runs 1 --restarts 5
pnpm test:restart:gateway -- --case default --runs 3 --restarts 3 --warmup 1

Case ids: skipChannels, skipChannelsAcpxProbe (ACPX startup check on), skipChannelsNoAcpxProbe (check off), default, fiftyPlugins.

Output includes next /healthz, next /readyz, downtime, restart ready timing, CPU, RSS, startup trace metrics for the replacement process, and restart trace metrics for signal handling, active-work drain, close phases, next start, ready timing, and memory snapshots. The script sets OPENCLAW_GATEWAY_STARTUP_TRACE=1 and OPENCLAW_GATEWAY_RESTART_TRACE=1.

Use this benchmark when a change touches restart signaling, close handlers, startup-after-restart, sidecar shutdown, service handoff, or readiness after restart. Start with skipChannels to isolate Gateway mechanics from channel startup; use default or plugin-heavy cases only after the narrow case explains the restart path. Trace metrics are attribution hints, not verdicts — judge a restart change from multiple samples, the matching owner span, /healthz//readyz behavior, and the user-visible restart contract.