mirror of
https://github.com/openclaw/openclaw.git
synced 2026-11-18 23:50:21 +00:00
## Problem A warm rebuild after a one-line source edit (`OPENCLAW_RUN_NODE_SKIP_DTS_BUILD=1 pnpm build`) takes about 4 minutes on a loaded 32-core Mac, mostly in work that repeats unchanged results: - Twelve build plugins register unfiltered `transform`/`load` hooks that match one or a few exact files. Rolldown therefore sends every module of ~90 configs through JavaScript, and several hooks `realpath` each id. - `plugins:assets:build` runs its plugin Control UI hooks one at a time (each a short esbuild process). - `external-plugins:local-dist` hits its per-plugin caches but still re-hashes the same SDK and core sources once per plugin: 66,258 file reads per pass for 12,965 unique files, done twice. - A `ui:build` cache hit still runs the report-only Control UI performance check (~20 s of gzip/brotli measurement) on bytes it already measured. - Runtime-only builds always build the sealed cloud-worker deploy bundle, about half of tsdown's CPU, and `check-cli-bootstrap-imports` then walks it. ## Solution - **Hook filters.** Each exact-match hook declares its match as a native Rolldown filter (`scripts/lib/rolldown-id-filter.mts`); handler bodies are unchanged. Filters normalize Windows paths to the forward-slash ids Rolldown matches against. The Bun standalone source adapter evaluates the same two filter shapes (id RegExp, code substring) and still rejects any other shape. - **Plugin asset hooks.** Build-phase hooks run four at a time; each writes only its own package's assets. Every running hook is joined before the step reports. Copy-phase hooks stay serial. - **External plugin signatures.** `hashInputFiles` accepts an optional per-run file-digest map. The step shares it across plugins and drops a plugin's own entries after its build, so regenerated package files are re-read. - **`ui:build` hit.** The report-only performance check runs only after a real build. The precompressed-assets check still runs on every hit, and restores still use APFS clones (`COPYFILE_FICLONE`). - **Worker deploy bundle.** `OPENCLAW_BUILD_WORKER_DEPLOY_BUNDLE=0` skips the sealed deploy entries, only together with `OPENCLAW_RUN_NODE_SKIP_DTS_BUILD=1`. Full, package, CI-artifact and release builds pin the bundle on in `build-all` (they run tsdown without declarations too), so only runtime-only builds honor the opt-out, and the local process helpers in `dist/worker/` are always built. tsdown's output clean removes any earlier bundle. Preparing a worker bundle from such a dist fails with the existing "OpenClaw worker deploy artifact is missing" error. The bootstrap check skips the worker artifact only when the flag is set and the entry is absent. Not changed: `runtime-postbuild` and `check-cli-bootstrap-imports` read tsdown's fresh output, so a content-keyed cache would miss on every source edit; the bootstrap check is ~1 s once the worker walk is skipped. ## Impact Measured on Studio (M3 Ultra, shared, load 40-80), `OPENCLAW_RUN_NODE_SKIP_DTS_BUILD=1 pnpm build`, warm caches, base `77a0a84265` vs this branch with `OPENCLAW_BUILD_WORKER_DEPLOY_BUNDLE=0`: | Warm rebuild | main | this PR | |---|---|---| | one `src/` edit (run 1) | 4m 04s | 1m 36s | | one `src/` edit (run 2) | 4m 15s | 2m 09s | | one `extensions/` edit | 3m 35s | 2m 01s | | no change | 3m 45s | 2m 28s | Per step for the `src/` edit: tsdown 91 s -> 32-42 s, `ui:build` 47-50 s -> 23-27 s, `plugins:assets:build` 16-25 s -> 4-5 s, `external-plugins:local-dist` 22-40 s -> 11-15 s, `check-cli-bootstrap-imports` 10-16 s -> ~1 s. Full cache-off `pnpm build` (declarations, worker bundle built): 6m 16.8s -> 4m 45.2s. ## Evidence - **Re-proven on this head (`73a8e1974e13`, rebased on `3256fe1`):** full `--dts` build of this head vs full build of `3256fe1`: 30,291 `dist`/`dist-runtime` files each; 11 differ, all build identity (stamps, `build-info.json`, `build-cache-inputs.json`, help-text commit, Control UI `index.html`/`sw.js*`) plus two Control UI source maps that differ only in Rolldown's checkout-path-derived `ROLLDOWN_FILE_URL_*` placeholder names (their JS is identical). All 705 declaration files are byte-identical. - **Tests on this head:** 18 files in `test/scripts` (the list above plus `crabbox-wrapper`, `ui-real-gateway-runner`, `ui-tsx-discovery`): 582 passed, 1 skipped; `build-all` passes separately. Timings below were taken on base `77a0a84265`; correctness proof and tests were rerun on the current head. - **Byte-identical full tier:** cache-off full `pnpm build` on base and branch, every dist file hashed: 28,193 files each; the only 10 differing files are build identity (`.buildstamp`, `.runtime-postbuildstamp`, `build-cache-inputs.json`, `build-info.json`, `cli-startup-metadata.json` help text with the commit id, Control UI `index.html`/`sw.js*`, and one Control UI source map that embeds the checkout path). - **Hook filters alone:** no-declaration tsdown step on base vs filters, 3 alternating pairs: 23,424 files, 0 differing in every pair; CPU 189 s -> 179 s. - **Worker skip:** after the skipped build, `dist/worker/` holds only `github-exec-launcher.mjs`, `service-child-group-anchor.mjs`, `service-child-relay.mjs`, `workspace-rsync-receiver.mjs` and the Windows helper; `prepareWorkerBundle` on that checkout rejects with "OpenClaw worker deploy artifact is missing; build the running package at …"; an isolated Gateway (private state, loopback, token auth) on that dist reports healthy in 17 s. - **Tests:** `build-all`, `build-artifact-cache`, `build-external-plugin-local-dist`, `bundled-plugin-assets`, `check-cli-bootstrap-imports`, `ui`, `tsdown-config`, `tsdown-runtime-config`, `worker-deploy-build-plugin`, `bun-standalone-source-{module-urls,transforms}`, `bun-standalone-worker-source-types`, `plugin-inventory-module-refs`, `public-session-locales`, `sqlite-schema-assets`, `state-read-worker-build`, `build-plugin-control-ui`; `tsgo -p tsconfig.scripts.json`. Co-authored-by: Ayaan Zaidi <hi@obviy.us>
72 lines
2.5 KiB
TypeScript
72 lines
2.5 KiB
TypeScript
import fs from "node:fs";
|
|
import path from "node:path";
|
|
import { exactIdFilter } from "./rolldown-id-filter.mts";
|
|
|
|
export const STATE_SCHEMA_INLINE_PLUGIN_NAME = "openclaw:inline-state-schemas";
|
|
|
|
const STATE_SCHEMA_MODULES = [
|
|
{
|
|
modulePath: "extensions/logbook/src/logbook-postgres-schema.ts",
|
|
schemaPath: "extensions/logbook/src/logbook-schema.postgres.sql",
|
|
exportName: "LOGBOOK_POSTGRES_SCHEMA_SQL",
|
|
},
|
|
{
|
|
modulePath: "extensions/workboard/src/workboard-postgres-schema.ts",
|
|
schemaPath: "extensions/workboard/src/workboard-schema.postgres.sql",
|
|
exportName: "WORKBOARD_POSTGRES_SCHEMA_SQL",
|
|
},
|
|
{
|
|
modulePath: "extensions/team-reports/src/store-postgres-schema.ts",
|
|
schemaPath: "extensions/team-reports/src/team-reports-schema.postgres.sql",
|
|
exportName: "TEAM_REPORTS_POSTGRES_SCHEMA_SQL",
|
|
},
|
|
{
|
|
modulePath: "src/state/openclaw-state-schema.ts",
|
|
schemaPath: "src/state/openclaw-state-schema.sql",
|
|
exportName: "OPENCLAW_STATE_SCHEMA_SQL",
|
|
},
|
|
{
|
|
modulePath: "src/state/openclaw-agent-schema.ts",
|
|
schemaPath: "src/state/openclaw-agent-schema.sql",
|
|
exportName: "OPENCLAW_AGENT_SCHEMA_SQL",
|
|
},
|
|
] as const;
|
|
|
|
export const STATE_SCHEMA_GENERATOR_INPUTS = STATE_SCHEMA_MODULES.map(
|
|
({ schemaPath }) => schemaPath,
|
|
);
|
|
|
|
/** Inline canonical schema bytes so bundled consumers need no SQL asset. */
|
|
export function createStateSchemaInlinePlugin(rootDir = process.cwd()) {
|
|
const schemasByModulePath = new Map(
|
|
STATE_SCHEMA_MODULES.map((schema) => [path.resolve(rootDir, schema.modulePath), schema]),
|
|
);
|
|
const cacheKeyForSchema = ({ id }: { id: string }) => {
|
|
const schema = schemasByModulePath.get(path.resolve(id));
|
|
return schema ? fs.readFileSync(path.resolve(rootDir, schema.schemaPath), "utf8") : undefined;
|
|
};
|
|
|
|
return {
|
|
name: STATE_SCHEMA_INLINE_PLUGIN_NAME,
|
|
configureVitest(context: {
|
|
defineCacheKeyGenerator(callback: typeof cacheKeyForSchema): void;
|
|
}) {
|
|
context.defineCacheKeyGenerator(cacheKeyForSchema);
|
|
},
|
|
load: {
|
|
filter: exactIdFilter(...schemasByModulePath.keys()),
|
|
handler(this: { addWatchFile(id: string): void }, id: string) {
|
|
const schema = schemasByModulePath.get(path.resolve(id));
|
|
if (!schema) {
|
|
return null;
|
|
}
|
|
const schemaPath = path.resolve(rootDir, schema.schemaPath);
|
|
this.addWatchFile(schemaPath);
|
|
return {
|
|
code: `export const ${schema.exportName} = ${JSON.stringify(fs.readFileSync(schemaPath, "utf8"))};\n`,
|
|
moduleType: "js" as const,
|
|
};
|
|
},
|
|
},
|
|
};
|
|
}
|