Refactor request routing and provider coordination
Some checks failed
CI / Typecheck (push) Has been cancelled
CI / Test architecture (push) Has been cancelled
CI / Test cli (push) Has been cancelled
CI / Test core (push) Has been cancelled
CI / Test electron (push) Has been cancelled
CI / Test ui (push) Has been cancelled
CI / CLI web E2E (push) Has been cancelled
CI / Docker system smoke (push) Has been cancelled

This commit is contained in:
musi 2026-07-15 13:32:14 +08:00
parent 2aa3529449
commit 30d1f35926
45 changed files with 5366 additions and 232 deletions

32
benchmarks/README.md Normal file
View file

@ -0,0 +1,32 @@
# Request log benchmarks
Run the request-log benchmark with the same Electron/Node native runtime selection used by the test runner:
```sh
npm run benchmark:request-logs -- --label local
```
To persist a machine-readable result:
```sh
npm run benchmark:request-logs -- --label local --output benchmarks/results/request-log-local.json
```
The benchmark has two layers:
- A request-log microbenchmark covering small records without a route trace, small records with a 10-hop route trace, 2 MB request bodies with a 10-hop trace, and 8 MB Base64 image requests.
- A real loopback HTTP service benchmark comparing no logging, synchronous main-thread SQLite logging, and the asynchronous worker logger.
The HTTP benchmark uses a separate load-generator process so client work does not share the server event loop. It reports QPS, durable QPS, p50/p95/p99/max latency, HTTP errors, rejected log events, event-loop p99/max delay, CPU, heap, and RSS changes.
The default web load is 10,000 requests with concurrency 64 and a 1 KB request body. It can be adjusted without editing code:
```sh
npm run benchmark:request-logs -- \
--web-requests 20000 \
--web-concurrency 128 \
--web-body-bytes 4096 \
--label web-load
```
Use `--skip-web` when only the storage microbenchmark is needed, or `--skip-storage` when iterating only on HTTP performance. Both layers report request-path time separately from durable flush time so moving work off the request thread cannot hide storage regressions.

View file

@ -0,0 +1,87 @@
{
"architecture": "worker-runtime",
"cpu": "arm64",
"label": "base64-image-object-scoped",
"node": "v24.16.0",
"platform": "darwin",
"results": [
{
"accepted": 2000,
"batchP99Ms": 1.632,
"degraded": 0,
"dropped": 0,
"durableMs": 64.936,
"durableRecordsPerSecond": 30799.754,
"eventLoopMaxMs": 2.365,
"eventLoopP99Ms": 1.831,
"flushMs": 47.56,
"heapDeltaMb": -0.89,
"name": "small-no-trace",
"records": 2000,
"requestPathMs": 17.375,
"requestPathRecordsPerSecond": 115108.464,
"rssDeltaMb": 21.203
},
{
"accepted": 1000,
"batchP99Ms": 1.15,
"degraded": 0,
"dropped": 0,
"durableMs": 58.399,
"durableRecordsPerSecond": 17123.581,
"eventLoopMaxMs": 2.195,
"eventLoopP99Ms": 2.195,
"flushMs": 45.398,
"heapDeltaMb": 4.493,
"name": "small-10-hops",
"records": 1000,
"requestPathMs": 13.001,
"requestPathRecordsPerSecond": 76915.681,
"rssDeltaMb": 8.891
},
{
"accepted": 50,
"batchP99Ms": 43.127,
"degraded": 50,
"dropped": 0,
"durableMs": 170.224,
"durableRecordsPerSecond": 293.73,
"eventLoopMaxMs": 1.916,
"eventLoopP99Ms": 1.628,
"flushMs": 91.361,
"heapDeltaMb": 36.715,
"name": "large-2mb-10-hops",
"records": 50,
"requestPathMs": 78.863,
"requestPathRecordsPerSecond": 634.013,
"rssDeltaMb": 135.344
},
{
"accepted": 10,
"batchP99Ms": 10.85,
"degraded": 10,
"dropped": 0,
"durableMs": 16.432,
"durableRecordsPerSecond": 608.572,
"eventLoopMaxMs": 1.2,
"eventLoopP99Ms": 1.2,
"flushMs": 5.581,
"heapDeltaMb": 0.062,
"name": "base64-image-8mb",
"records": 10,
"requestPathMs": 10.851,
"requestPathRecordsPerSecond": 921.595,
"rssDeltaMb": 0
}
],
"storageSkipped": false,
"timestamp": "2026-07-15T01:25:30.154Z",
"web": {
"bodyBytes": 1024,
"concurrency": 64,
"requests": 10000,
"results": [],
"skipped": true
},
"runtime": "electron"
}

View file

@ -0,0 +1,62 @@
{
"architecture": "main-thread-store",
"cpu": "arm64",
"label": "baseline",
"node": "v24.16.0",
"platform": "darwin",
"results": [
{
"accepted": 2000,
"batchP99Ms": 6.366,
"degraded": 0,
"dropped": 0,
"durableMs": 201.032,
"durableRecordsPerSecond": 9948.653,
"eventLoopMaxMs": 6.365,
"eventLoopP99Ms": 4.239,
"flushMs": 0.002,
"heapDeltaMb": 1.586,
"name": "small-no-trace",
"records": 2000,
"requestPathMs": 201.03,
"requestPathRecordsPerSecond": 9948.756,
"rssDeltaMb": 18.484
},
{
"accepted": 1000,
"batchP99Ms": 7.765,
"degraded": 0,
"dropped": 0,
"durableMs": 168.879,
"durableRecordsPerSecond": 5921.412,
"eventLoopMaxMs": 7.782,
"eventLoopP99Ms": 7.782,
"flushMs": 0.003,
"heapDeltaMb": -1.761,
"name": "small-10-hops",
"records": 1000,
"requestPathMs": 168.875,
"requestPathRecordsPerSecond": 5921.534,
"rssDeltaMb": 10.172
},
{
"accepted": 50,
"batchP99Ms": 203.52,
"degraded": 0,
"dropped": 0,
"durableMs": 397.142,
"durableRecordsPerSecond": 125.9,
"eventLoopMaxMs": 204.079,
"eventLoopP99Ms": 204.079,
"flushMs": 0.003,
"heapDeltaMb": 2.151,
"name": "large-2mb-10-hops",
"records": 50,
"requestPathMs": 397.139,
"requestPathRecordsPerSecond": 125.901,
"rssDeltaMb": 130.203
}
],
"timestamp": "2026-07-14T23:42:58.245Z",
"runtime": "electron"
}

View file

@ -0,0 +1,43 @@
# Request log performance comparison
Environment: Apple arm64, Electron Node v24.16.0, SQLite WAL, identical benchmark scenarios and batch sizes.
| Scenario | Metric | Baseline | Optimized | Change |
| --- | ---: | ---: | ---: | ---: |
| Small, no trace | Request-path throughput | 9,949 records/s | 173,695 records/s | 17.5x |
| Small, no trace | Durable throughput | 9,949 records/s | 34,267 records/s | 3.4x |
| Small, no trace | Event-loop p99 | 4.24 ms | 1.61 ms | -62.1% |
| Small, 10 hops | Request-path throughput | 5,922 records/s | 66,716 records/s | 11.3x |
| Small, 10 hops | Durable throughput | 5,921 records/s | 17,640 records/s | 3.0x |
| Small, 10 hops | Event-loop p99 | 7.78 ms | 1.30 ms | -83.3% |
| 2 MB, 10 hops | Request-path throughput | 126 records/s | 553 records/s | 4.4x |
| 2 MB, 10 hops | Durable throughput | 126 records/s | 336 records/s | 2.7x |
| 2 MB, 10 hops | Event-loop max | 204.08 ms | 2.04 ms | -99.0% |
| 2 MB, 10 hops | RSS delta | 130.20 MB | 66.58 MB | -48.9% |
The optimized 2 MB result above was measured with the then-current 512 KB default, so it is intentionally marked degraded. Production now retains up to the 50 MB hard safety ceiling for each request/response body by default while preserving original size and truncation metadata.
Raw measurements:
- `request-log-baseline.json`
- `request-log-optimized.json`
## HTTP service benchmark
The HTTP benchmark sends 10,000 POST requests with concurrency 64 and a 1 KB body from a separate load-generator process.
| Mode | QPS | Durable QPS | p50 | p95 | p99 | Event-loop p99 | Errors | Rejected logs |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| No logging control | 32,878 | 32,877 | 1.93 ms | 2.34 ms | 3.02 ms | 1.96 ms | 0 | 0 |
| Synchronous main-thread SQLite | 9,609 | 9,603 | 5.67 ms | 11.44 ms | 13.51 ms | 10.53 ms | 0 | 0 |
| Asynchronous worker logger | 31,879 | 25,757 | 2.02 ms | 3.01 ms | 6.20 ms | 2.06 ms | 0 | 0 |
Compared with synchronous main-thread logging, the asynchronous worker delivers 3.3x request QPS, 2.7x durable QPS, 54.1% lower HTTP p99 latency, and 80.5% lower event-loop p99 delay. Its request-path QPS is 3.0% below the no-logging control, and all 10,000 log events were accepted.
The first unrestricted-network run also exposed a linear scan through the downloaded model-pricing catalog, which limited durable logging to roughly 109 records/s. Building a source-and-model price index once per catalog load removed that bottleneck before the final measurements above.
## Base64 image benchmark
The image scenario reuses an already-received 8 MB JSON request body containing an inline Base64 image, matching the point where request logging starts. Ten records were accepted and compacted. Request-path throughput was 922 records/s, durable throughput was 609 records/s, and no RSS growth was observed in the measured run. The stored JSON retains its structure, MIME information, original wire size, and an image-size descriptor without retaining the encoded pixels. MIME and encoding context are scoped to the same JSON object, so adjacent audio payloads are not compacted.
Raw measurement: `request-log-base64-image.json`.

View file

@ -0,0 +1,149 @@
{
"architecture": "worker-runtime",
"cpu": "arm64",
"label": "optimized-final",
"node": "v24.16.0",
"platform": "darwin",
"results": [
{
"accepted": 2000,
"batchP99Ms": 0.704,
"degraded": 0,
"dropped": 0,
"durableMs": 58.366,
"durableRecordsPerSecond": 34266.696,
"eventLoopMaxMs": 2.052,
"eventLoopP99Ms": 1.605,
"flushMs": 46.851,
"heapDeltaMb": 1.279,
"name": "small-no-trace",
"records": 2000,
"requestPathMs": 11.514,
"requestPathRecordsPerSecond": 173695.29,
"rssDeltaMb": 16
},
{
"accepted": 1000,
"batchP99Ms": 2.092,
"degraded": 0,
"dropped": 0,
"durableMs": 56.691,
"durableRecordsPerSecond": 17639.614,
"eventLoopMaxMs": 2.347,
"eventLoopP99Ms": 1.302,
"flushMs": 41.702,
"heapDeltaMb": 3.953,
"name": "small-10-hops",
"records": 1000,
"requestPathMs": 14.989,
"requestPathRecordsPerSecond": 66716.335,
"rssDeltaMb": 6.313
},
{
"accepted": 50,
"batchP99Ms": 46.632,
"degraded": 50,
"dropped": 0,
"durableMs": 148.846,
"durableRecordsPerSecond": 335.918,
"eventLoopMaxMs": 2.036,
"eventLoopP99Ms": 1.665,
"flushMs": 58.449,
"heapDeltaMb": -2.104,
"name": "large-2mb-10-hops",
"records": 50,
"requestPathMs": 90.396,
"requestPathRecordsPerSecond": 553.12,
"rssDeltaMb": 66.578
}
],
"storageSkipped": false,
"timestamp": "2026-07-15T00:58:51.159Z",
"web": {
"bodyBytes": 1024,
"concurrency": 64,
"requests": 10000,
"results": [
{
"bodyBytes": 1024,
"concurrency": 64,
"cpuSystemMs": 48.203,
"cpuTotalPercent": 50.988,
"cpuUserMs": 106.881,
"durableMs": 304.161,
"durableQps": 32877.361,
"errors": 0,
"eventLoopMaxMs": 2.017,
"eventLoopP99Ms": 1.959,
"flushMs": 0.002,
"heapDeltaMb": 0.561,
"latencyAvgMs": 1.915,
"latencyMaxMs": 9.838,
"latencyP50Ms": 1.927,
"latencyP95Ms": 2.341,
"latencyP99Ms": 3.021,
"logAccepted": 0,
"logRejected": 0,
"mode": "control",
"qps": 32877.509,
"requestPathMs": 304.159,
"requests": 10000,
"rssDeltaMb": 3.531
},
{
"bodyBytes": 1024,
"concurrency": 64,
"cpuSystemMs": 521.521,
"cpuTotalPercent": 90.135,
"cpuUserMs": 416.51,
"durableMs": 1041.303,
"durableQps": 9603.352,
"errors": 0,
"eventLoopMaxMs": 13.369,
"eventLoopP99Ms": 10.527,
"flushMs": 0.609,
"heapDeltaMb": -2.815,
"latencyAvgMs": 6.612,
"latencyMaxMs": 24.691,
"latencyP50Ms": 5.67,
"latencyP95Ms": 11.436,
"latencyP99Ms": 13.508,
"logAccepted": 10000,
"logRejected": 0,
"mode": "sync-main-thread",
"qps": 9608.974,
"requestPathMs": 1040.694,
"requests": 10000,
"rssDeltaMb": -12.125
},
{
"bodyBytes": 1024,
"concurrency": 64,
"cpuSystemMs": 126.52,
"cpuTotalPercent": 146.645,
"cpuUserMs": 333.485,
"durableMs": 388.251,
"durableQps": 25756.503,
"errors": 0,
"eventLoopMaxMs": 2.269,
"eventLoopP99Ms": 2.055,
"flushMs": 74.564,
"heapDeltaMb": -5.156,
"latencyAvgMs": 1.967,
"latencyMaxMs": 11.159,
"latencyP50Ms": 2.018,
"latencyP95Ms": 3.007,
"latencyP99Ms": 6.195,
"logAccepted": 10000,
"logRejected": 0,
"mode": "async-worker",
"qps": 31878.941,
"requestPathMs": 313.687,
"requests": 10000,
"rssDeltaMb": 4.516
}
],
"skipped": false
},
"runtime": "electron"
}

View file

@ -0,0 +1,129 @@
import esbuild from "esbuild";
import { spawnSync } from "node:child_process";
import { existsSync, mkdirSync, rmSync, statSync, writeFileSync } from "node:fs";
import os from "node:os";
import path from "node:path";
import { fileURLToPath } from "node:url";
const buildDir = path.dirname(fileURLToPath(import.meta.url));
const projectRoot = path.resolve(buildDir, "..");
const outputDir = path.join(projectRoot, ".benchmark-dist", "request-logs");
const outputFile = path.join(outputDir, "request-log-runtime.bench.js");
const args = parseArgs(process.argv.slice(2));
rmSync(outputDir, { force: true, recursive: true });
mkdirSync(outputDir, { recursive: true });
try {
await esbuild.build({
absWorkingDir: projectRoot,
bundle: true,
entryPoints: {
"request-log-runtime.bench": path.join(projectRoot, "packages/core/benchmark/request-log-runtime.bench.mjs"),
"request-log-web-load": path.join(projectRoot, "packages/core/benchmark/request-log-web-load.mjs"),
"request-log-worker": path.join(projectRoot, "packages/core/src/observability/request-log-worker.ts")
},
external: ["better-sqlite3", "electron", "undici"],
format: "cjs",
legalComments: "none",
logLevel: "warning",
outdir: outputDir,
platform: "node",
plugins: [packageAliasPlugin()],
target: "node22"
});
const runtime = resolveRuntime();
const executable = runtime === "electron"
? (await import("electron")).default
: process.execPath;
const child = spawnSync(executable, [outputFile], {
cwd: projectRoot,
encoding: "utf8",
env: {
...process.env,
CCR_REQUEST_LOG_BENCHMARK_LABEL: args.label,
CCR_REQUEST_LOG_BENCHMARK_SKIP_STORAGE: args.skipStorage ? "1" : "0",
CCR_REQUEST_LOG_BENCHMARK_SKIP_WEB: args.skipWeb ? "1" : "0",
CCR_REQUEST_LOG_BENCHMARK_WEB_BODY_BYTES: args.webBodyBytes,
CCR_REQUEST_LOG_BENCHMARK_WEB_CONCURRENCY: args.webConcurrency,
CCR_REQUEST_LOG_BENCHMARK_WEB_REQUESTS: args.webRequests,
...(runtime === "electron" ? { ELECTRON_RUN_AS_NODE: "1" } : {})
},
maxBuffer: 16 * 1024 * 1024,
stdio: ["ignore", "pipe", "inherit"]
});
if (child.status !== 0) {
process.stderr.write(child.stdout || `benchmark exited with ${child.status}\n`);
process.exitCode = child.status ?? 1;
} else {
const result = JSON.parse(child.stdout);
result.runtime = runtime;
const json = `${JSON.stringify(result, null, 2)}\n`;
process.stdout.write(json);
if (args.output) {
const destination = path.resolve(projectRoot, args.output);
mkdirSync(path.dirname(destination), { recursive: true });
writeFileSync(destination, json, "utf8");
}
}
} finally {
rmSync(outputDir, { force: true, recursive: true });
}
function parseArgs(values) {
const parsed = {
label: "benchmark",
output: "",
skipStorage: false,
skipWeb: false,
webBodyBytes: "1024",
webConcurrency: "64",
webRequests: "10000"
};
for (let index = 0; index < values.length; index += 1) {
if (values[index] === "--label") parsed.label = values[index + 1] || parsed.label;
if (values[index] === "--output") parsed.output = values[index + 1] || "";
if (values[index] === "--skip-web") parsed.skipWeb = true;
if (values[index] === "--skip-storage") parsed.skipStorage = true;
if (values[index] === "--web-body-bytes") parsed.webBodyBytes = values[index + 1] || parsed.webBodyBytes;
if (values[index] === "--web-concurrency") parsed.webConcurrency = values[index + 1] || parsed.webConcurrency;
if (values[index] === "--web-requests") parsed.webRequests = values[index + 1] || parsed.webRequests;
}
return parsed;
}
function resolveRuntime() {
const probe = spawnSync(process.execPath, [
"-e",
"const Database = require('better-sqlite3'); const db = new Database(':memory:'); db.close();"
], { stdio: "ignore" });
return probe.status === 0 ? "node" : "electron";
}
function packageAliasPlugin() {
const roots = {
cli: path.join(projectRoot, "packages/cli/src"),
core: path.join(projectRoot, "packages/core/src"),
electron: path.join(projectRoot, "packages/electron/src"),
ui: path.join(projectRoot, "packages/ui/src")
};
return {
name: "request-log-benchmark-alias",
setup(build) {
build.onResolve({ filter: /^@ccr\/(cli|core|electron|ui)\// }, (resolveArgs) => {
const match = resolveArgs.path.match(/^@ccr\/(cli|core|electron|ui)\/(.+)$/);
if (!match) return undefined;
return { path: resolvePackageImport(roots[match[1]], match[2]) };
});
}
};
}
function resolvePackageImport(root, importPath) {
const base = path.resolve(root, importPath);
for (const candidate of [base, `${base}.ts`, `${base}.tsx`, `${base}.js`, `${base}.mjs`, path.join(base, "index.ts")]) {
if (existsSync(candidate) && statSync(candidate).isFile()) return candidate;
}
return base;
}

View file

@ -68,6 +68,7 @@ export const cssInput = path.join(rendererRoot, "styles", "globals.css");
export const cssOutput = path.join(rendererAssetsDir, "main.css");
export const webClientBridgeOutput = path.join(rendererAssetsDir, "web-client-bridge.js");
export const electronUndiciProxyAgentInput = path.join(coreSourceRoot, "proxy", "undici-proxy-agent.ts");
export const upstreamHeaderSanitizerInput = path.join(coreSourceRoot, "gateway", "core-runtime", "upstream-header-sanitizer.ts");
const lightweightMcpBundleNames = ["browser-web-search-proxy-mcp.js", "fusion-vision-mcp.js", "fusion-tool-fallback-mcp.js"];
const lightweightMcpBundleMaxBytes = 128 * 1024;
const forbiddenLightweightMcpInputs = [
@ -219,6 +220,8 @@ export function createMainBuildOptions({ mode = "production", plugins = [] } = {
path.join(coreSourceRoot, "mcp", "fusion-vision-mcp.ts"),
path.join(coreSourceRoot, "mcp", "fusion-tool-fallback-mcp.ts"),
path.join(coreSourceRoot, "mcp", "toolhub-mcp.ts"),
path.join(coreSourceRoot, "observability", "request-log-worker.ts"),
upstreamHeaderSanitizerInput,
electronUndiciProxyAgentInput,
path.join(electronSourceRoot, "main", "preload.ts")
],
@ -245,7 +248,9 @@ export function createCliBuildOptions({ mode = "production", plugins = [] } = {}
path.join(cliSourceRoot, "cli.ts"),
path.join(coreSourceRoot, "mcp", "fusion-vision-mcp.ts"),
path.join(coreSourceRoot, "mcp", "fusion-tool-fallback-mcp.ts"),
path.join(coreSourceRoot, "mcp", "toolhub-mcp.ts")
path.join(coreSourceRoot, "mcp", "toolhub-mcp.ts"),
path.join(coreSourceRoot, "observability", "request-log-worker.ts"),
upstreamHeaderSanitizerInput
],
external: nodeExternals.filter((moduleName) => moduleName !== "electron"),
format: "cjs",
@ -269,7 +274,9 @@ export function createCoreServerBuildOptions({ mode = "production", plugins = []
path.join(coreSourceRoot, "entrypoints", "server.ts"),
path.join(coreSourceRoot, "mcp", "fusion-vision-mcp.ts"),
path.join(coreSourceRoot, "mcp", "fusion-tool-fallback-mcp.ts"),
path.join(coreSourceRoot, "mcp", "toolhub-mcp.ts")
path.join(coreSourceRoot, "mcp", "toolhub-mcp.ts"),
path.join(coreSourceRoot, "observability", "request-log-worker.ts"),
upstreamHeaderSanitizerInput
],
external: nodeExternals.filter((moduleName) => moduleName !== "electron"),
format: "cjs",

View file

@ -25,6 +25,8 @@ const testProjects = {
core: {
runtimeEntryPoints: {
"runtime/fusion-vision-mcp": path.join(packageRoots.core, "mcp", "fusion-vision-mcp.ts"),
"runtime/request-log-worker": path.join(packageRoots.core, "observability", "request-log-worker.ts"),
"runtime/upstream-header-sanitizer": path.join(packageRoots.core, "gateway", "core-runtime", "upstream-header-sanitizer.ts"),
"runtime/toolhub-mcp": path.join(packageRoots.core, "mcp", "toolhub-mcp.ts")
},
testDir: path.join(projectRoot, "packages", "core", "test")

View file

@ -59,6 +59,7 @@
"test:e2e:install": "playwright install chromium",
"test:main": "npm run test:core && npm run test:electron",
"test:renderer": "npm run test:ui",
"benchmark:request-logs": "node build/benchmark-request-logs.mjs",
"typecheck": "tsc --noEmit",
"rebuild:sqlite3": "electron-rebuild -f -w better-sqlite3"
},

View file

@ -0,0 +1,482 @@
import { spawn } from "node:child_process";
import { mkdtempSync, rmSync } from "node:fs";
import { createServer } from "node:http";
import { tmpdir } from "node:os";
import path from "node:path";
import { monitorEventLoopDelay, performance } from "node:perf_hooks";
import process from "node:process";
import * as requestLogs from "@ccr/core/observability/request-log-store.ts";
const scenarios = [
{ bodyBytes: 256, hops: 0, name: "small-no-trace", records: 2_000 },
{ bodyBytes: 256, hops: 10, name: "small-10-hops", records: 1_000 },
{ bodyBytes: 2 * 1024 * 1024, hops: 10, name: "large-2mb-10-hops", records: 50 },
{ base64Image: true, bodyBytes: 8 * 1024 * 1024, hops: 0, name: "base64-image-8mb", records: 10 }
];
const webOptions = {
bodyBytes: positiveInteger(process.env.CCR_REQUEST_LOG_BENCHMARK_WEB_BODY_BYTES, 1_024),
concurrency: positiveInteger(process.env.CCR_REQUEST_LOG_BENCHMARK_WEB_CONCURRENCY, 64),
requests: positiveInteger(process.env.CCR_REQUEST_LOG_BENCHMARK_WEB_REQUESTS, 10_000),
skip: process.env.CCR_REQUEST_LOG_BENCHMARK_SKIP_WEB === "1"
};
const skipStorage = process.env.CCR_REQUEST_LOG_BENCHMARK_SKIP_STORAGE === "1";
void main();
async function main() {
const results = [];
if (!skipStorage) {
for (const scenario of scenarios) {
process.stderr.write(`[benchmark] Running storage scenario: ${scenario.name}\n`);
results.push(await runScenario(scenario));
process.stderr.write(`[benchmark] Completed storage scenario: ${scenario.name}\n`);
}
}
const webResults = [];
if (!webOptions.skip) {
for (const mode of ["control", "sync-main-thread", "async-worker"]) {
process.stderr.write(`[benchmark] Running HTTP scenario: ${mode}\n`);
webResults.push(await runWebScenario(mode, webOptions));
process.stderr.write(`[benchmark] Completed HTTP scenario: ${mode}\n`);
}
}
process.stdout.write(JSON.stringify({
architecture: typeof requestLogs.createRequestLogRuntime === "function" ? "worker-runtime" : "main-thread-store",
cpu: process.arch,
label: process.env.CCR_REQUEST_LOG_BENCHMARK_LABEL || "benchmark",
node: process.version,
platform: process.platform,
results,
storageSkipped: skipStorage,
timestamp: new Date().toISOString(),
web: {
bodyBytes: webOptions.bodyBytes,
concurrency: webOptions.concurrency,
requests: webOptions.requests,
results: webResults,
skipped: webOptions.skip
}
}));
}
async function runScenario(scenario) {
const dir = mkdtempSync(path.join(tmpdir(), `ccr-log-bench-${scenario.name}-`));
const target = createTarget(path.join(dir, "request-logs.sqlite"));
const eventLoop = monitorEventLoopDelay({ resolution: 1 });
const batchDurations = [];
const pending = [];
let accepted = 0;
let degraded = 0;
let dropped = 0;
const sharedRequestBody = scenario.base64Image ? createRequestBody(scenario) : undefined;
try {
for (let index = 0; index < Math.min(25, scenario.records); index += 1) {
await Promise.resolve(target.record(createRecord(scenario, `warmup-${index}`, sharedRequestBody)));
}
await target.flush();
if (global.gc) global.gc();
const heapBefore = process.memoryUsage();
eventLoop.enable();
const startedAt = performance.now();
const batchSize = 25;
for (let start = 0; start < scenario.records; start += batchSize) {
const batchStartedAt = performance.now();
pending.length = 0;
for (let index = start; index < Math.min(start + batchSize, scenario.records); index += 1) {
const result = target.record(createRecord(scenario, `${scenario.name}-${index}`, sharedRequestBody));
if (result && typeof result.then === "function") {
pending.push(result);
} else if (result?.accepted === false) {
dropped += 1;
} else {
accepted += 1;
if (result?.degraded) degraded += 1;
}
}
await Promise.all(pending);
await new Promise((resolve) => setImmediate(resolve));
batchDurations.push(performance.now() - batchStartedAt);
}
const requestPathMs = performance.now() - startedAt;
const flushStartedAt = performance.now();
await target.flush();
const flushMs = performance.now() - flushStartedAt;
const durableMs = performance.now() - startedAt;
eventLoop.disable();
const heapAfter = process.memoryUsage();
if (accepted + dropped === 0) accepted = scenario.records;
return {
accepted,
batchP99Ms: round(percentile(batchDurations, 0.99)),
degraded,
dropped,
durableMs: round(durableMs),
durableRecordsPerSecond: round(accepted / (durableMs / 1_000)),
eventLoopMaxMs: round(Number(eventLoop.max) / 1e6),
eventLoopP99Ms: round(Number(eventLoop.percentile(99)) / 1e6),
flushMs: round(flushMs),
heapDeltaMb: round((heapAfter.heapUsed - heapBefore.heapUsed) / 1024 / 1024),
name: scenario.name,
records: scenario.records,
requestPathMs: round(requestPathMs),
requestPathRecordsPerSecond: round(scenario.records / (requestPathMs / 1_000)),
rssDeltaMb: round((heapAfter.rss - heapBefore.rss) / 1024 / 1024)
};
} finally {
await target.close();
rmSync(dir, { force: true, recursive: true });
}
}
async function runWebScenario(mode, options) {
const dir = mkdtempSync(path.join(tmpdir(), `ccr-log-web-bench-${mode}-`));
const target = createWebTarget(mode, path.join(dir, "request-logs.sqlite"), options);
const counters = { accepted: 0, rejected: 0 };
const server = createServer((request, response) => {
void handleWebRequest(request, response, mode, target, counters);
});
server.keepAliveTimeout = 30_000;
server.requestTimeout = 30_000;
try {
await target.ready();
const address = await listen(server);
const url = `http://127.0.0.1:${address.port}/benchmark`;
const warmupRequests = Math.min(1_000, Math.max(100, Math.floor(options.requests / 10)));
await runLoadGenerator(url, {
...options,
concurrency: Math.min(options.concurrency, warmupRequests),
prefix: `${mode}-warmup`,
requests: warmupRequests
});
await target.flush();
counters.accepted = 0;
counters.rejected = 0;
if (global.gc) global.gc();
const eventLoop = monitorEventLoopDelay({ resolution: 1 });
const memoryBefore = process.memoryUsage();
const cpuBefore = process.cpuUsage();
eventLoop.enable();
const load = await runLoadGenerator(url, {
...options,
prefix: `${mode}-measured`
});
eventLoop.disable();
const memoryAfterRequest = process.memoryUsage();
const cpu = process.cpuUsage(cpuBefore);
const flushStartedAt = performance.now();
await target.flush();
const flushMs = performance.now() - flushStartedAt;
const durableMs = load.durationMs + flushMs;
return {
bodyBytes: options.bodyBytes,
concurrency: options.concurrency,
cpuSystemMs: round(cpu.system / 1_000),
cpuTotalPercent: round(((cpu.user + cpu.system) / 1_000) / Math.max(1, load.durationMs) * 100),
cpuUserMs: round(cpu.user / 1_000),
durableMs: round(durableMs),
durableQps: round(load.completed / (durableMs / 1_000)),
errors: load.errors,
eventLoopMaxMs: round(Number(eventLoop.max) / 1e6),
eventLoopP99Ms: round(Number(eventLoop.percentile(99)) / 1e6),
flushMs: round(flushMs),
heapDeltaMb: round((memoryAfterRequest.heapUsed - memoryBefore.heapUsed) / 1024 / 1024),
latencyAvgMs: load.latencyAvgMs,
latencyMaxMs: load.latencyMaxMs,
latencyP50Ms: load.latencyP50Ms,
latencyP95Ms: load.latencyP95Ms,
latencyP99Ms: load.latencyP99Ms,
logAccepted: counters.accepted,
logRejected: counters.rejected,
mode,
qps: load.qps,
requestPathMs: load.durationMs,
requests: load.requests,
rssDeltaMb: round((memoryAfterRequest.rss - memoryBefore.rss) / 1024 / 1024)
};
} finally {
await closeHttpServer(server);
await target.close();
rmSync(dir, { force: true, recursive: true });
}
}
async function handleWebRequest(request, response, mode, target, counters) {
const startedAt = Date.now();
try {
const body = await readBody(request);
if (mode !== "control") {
const input = createWebRecord(
body,
String(request.headers["x-benchmark-request-id"] ?? "benchmark-request"),
startedAt
);
const result = target.record(input);
if (result && typeof result.then === "function") {
await result;
counters.accepted += 1;
} else if (result?.accepted === false) {
counters.rejected += 1;
} else {
counters.accepted += 1;
}
}
response.writeHead(204);
response.end();
} catch (error) {
response.writeHead(500, { "content-type": "text/plain" });
response.end(error instanceof Error ? error.message : String(error));
}
}
function createWebTarget(mode, dbFile, options) {
if (mode === "control") {
return {
close: async () => {},
flush: async () => {},
ready: async () => {},
record: () => undefined
};
}
if (mode === "sync-main-thread") {
const store = new requestLogs.RequestLogStore(dbFile);
return {
close: () => store.close(),
flush: () => store.checkpoint(),
ready: () => store.initialize(),
record: (input) => store.record(input)
};
}
const runtime = requestLogs.createRequestLogRuntime({
batchMaxBytes: 4 * 1024 * 1024,
batchMaxItems: 50,
batchMaxWaitMs: 10,
dbFile,
queueMaxBytes: Math.max(256 * 1024 * 1024, options.bodyBytes * options.requests * 2),
queueMaxItems: Math.max(20_000, options.requests * 2)
});
return {
close: () => runtime.close({ timeoutMs: 30_000 }),
flush: () => runtime.flush({ timeoutMs: 60_000 }),
ready: async () => {},
record: (input) => runtime.enqueueRecord(input)
};
}
function createWebRecord(requestBody, requestId, startedAtMs) {
const startedAt = new Date(startedAtMs).toISOString();
return {
captureBody: true,
client: "web-benchmark",
completedAt: new Date().toISOString(),
durationMs: Date.now() - startedAtMs,
maxBodyBytes: 512 * 1024,
method: "POST",
model: "benchmark-model",
path: "/benchmark",
providerName: "benchmark-provider",
requestBody,
requestHeaders: { "content-type": "application/octet-stream" },
requestId,
responseBodyText: "",
responseHeaders: {},
startedAt,
statusCode: 204,
url: "http://127.0.0.1/benchmark"
};
}
function runLoadGenerator(url, options) {
const loadGenerator = path.join(__dirname, "request-log-web-load.js");
const args = [
loadGenerator,
"--url", url,
"--requests", String(options.requests),
"--concurrency", String(options.concurrency),
"--body-bytes", String(options.bodyBytes),
"--prefix", options.prefix
];
return new Promise((resolve, reject) => {
const child = spawn(process.execPath, args, {
cwd: process.cwd(),
env: process.env,
stdio: ["ignore", "pipe", "pipe"]
});
let stdout = "";
let stderr = "";
child.stdout.setEncoding("utf8");
child.stderr.setEncoding("utf8");
child.stdout.on("data", (chunk) => { stdout += chunk; });
child.stderr.on("data", (chunk) => { stderr += chunk; });
child.on("error", reject);
child.on("exit", (code, signal) => {
if (signal) {
reject(new Error(`Web load generator exited from signal ${signal}: ${stderr}`));
return;
}
if (code !== 0) {
reject(new Error(stderr || stdout || `Web load generator exited with ${code ?? 1}`));
return;
}
try {
resolve(JSON.parse(stdout));
} catch (error) {
reject(new Error(`Invalid web load generator output: ${stdout}\n${stderr}`, { cause: error }));
}
});
});
}
function readBody(request) {
return new Promise((resolve, reject) => {
const chunks = [];
let size = 0;
request.on("data", (chunk) => {
const buffer = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk);
chunks.push(buffer);
size += buffer.byteLength;
});
request.on("end", () => resolve(Buffer.concat(chunks, size)));
request.on("error", reject);
});
}
function listen(server) {
return new Promise((resolve, reject) => {
server.once("error", reject);
server.listen(0, "127.0.0.1", () => {
server.off("error", reject);
resolve(server.address());
});
});
}
function closeHttpServer(server) {
return new Promise((resolve) => {
if (!server.listening) {
resolve();
return;
}
server.close(() => resolve());
server.closeIdleConnections?.();
});
}
function createTarget(dbFile) {
if (typeof requestLogs.createRequestLogRuntime === "function") {
const runtime = requestLogs.createRequestLogRuntime({
batchMaxBytes: 4 * 1024 * 1024,
batchMaxItems: 50,
batchMaxWaitMs: 10,
dbFile,
queueMaxBytes: 256 * 1024 * 1024,
queueMaxItems: 4_000
});
return {
close: () => runtime.close({ timeoutMs: 10_000 }),
flush: () => runtime.flush({ timeoutMs: 30_000 }),
record: (input) => runtime.enqueueRecord(input)
};
}
const store = new requestLogs.RequestLogStore(dbFile);
return {
close: () => store.close(),
flush: async () => {},
record: (input) => store.record(input)
};
}
function createRecord(scenario, requestId, sharedRequestBody) {
const startedAt = new Date().toISOString();
const requestBody = sharedRequestBody ?? createRequestBody(scenario);
return {
client: "benchmark",
completedAt: startedAt,
durationMs: 42,
fallbackModel: "benchmark-provider,benchmark-model",
method: "POST",
maxBodyBytes: scenario.base64Image ? 50 * 1024 * 1024 : undefined,
model: "benchmark-model",
path: "/v1/messages",
providerName: "benchmark-provider",
requestBody,
requestHeaders: { "content-type": "application/json", "x-api-key": "secret" },
requestId,
...(scenario.hops > 0 ? { routeTrace: createTrace(scenario.hops) } : {}),
responseBodyText: JSON.stringify({ usage: { input_tokens: 32, output_tokens: 16 } }),
responseHeaders: { "content-type": "application/json" },
startedAt,
statusCode: 200,
url: "http://127.0.0.1:3456/v1/messages"
};
}
function createRequestBody(scenario) {
const paddingLength = Math.max(0, scenario.bodyBytes - 96);
return Buffer.from(JSON.stringify(scenario.base64Image
? {
messages: [{
content: [{
source: { data: "A".repeat(paddingLength), media_type: "image/png", type: "base64" },
type: "image"
}],
role: "user"
}],
model: "benchmark-model",
stream: true
}
: {
messages: [{ content: "x".repeat(paddingLength), role: "user" }],
model: "benchmark-model",
stream: true
}));
}
function createTrace(hopCount) {
return {
attemptCount: 1,
complete: true,
hopCount,
hops: Array.from({ length: hopCount }, (_, index) => ({
attempt: 1,
changes: [{
after: `model-${index + 1}`,
before: `model-${index}`,
operation: "replace",
path: "/body/model",
scope: "body"
}],
durationMs: 1,
kind: "mutation",
name: `benchmark.hop-${index}`,
phase: index === 0 ? "routing" : "attempt",
seq: index,
startedOffsetMs: index,
status: "ok",
target: { model: `model-${index + 1}`, provider: "benchmark-provider" }
})),
truncated: false,
version: 2
};
}
function percentile(values, fraction) {
if (values.length === 0) return 0;
const sorted = [...values].sort((left, right) => left - right);
return sorted[Math.min(sorted.length - 1, Math.floor(sorted.length * fraction))];
}
function round(value) {
return Math.round(value * 1_000) / 1_000;
}
function positiveInteger(value, fallback) {
const parsed = Number(value);
return Number.isFinite(parsed) && parsed > 0 ? Math.floor(parsed) : fallback;
}

View file

@ -0,0 +1,100 @@
import process from "node:process";
import { performance } from "node:perf_hooks";
import { Pool } from "undici";
void main();
async function main() {
const options = parseArgs(process.argv.slice(2));
const target = new URL(options.url);
const pool = new Pool(target.origin, {
connections: options.concurrency,
pipelining: 1
});
const body = Buffer.alloc(options.bodyBytes, "x");
const latencies = new Float64Array(options.requests);
let nextRequest = 0;
let completed = 0;
let errors = 0;
const startedAt = performance.now();
try {
await Promise.all(Array.from({ length: options.concurrency }, async () => {
while (true) {
const index = nextRequest;
nextRequest += 1;
if (index >= options.requests) return;
const requestStartedAt = performance.now();
try {
const response = await pool.request({
body,
headers: {
"content-type": "application/octet-stream",
"x-benchmark-request-id": `${options.prefix}-${index}`
},
method: "POST",
path: target.pathname
});
await response.body.dump();
if (response.statusCode < 200 || response.statusCode >= 300) errors += 1;
} catch {
errors += 1;
} finally {
latencies[index] = performance.now() - requestStartedAt;
completed += 1;
}
}
}));
} finally {
await pool.close();
}
const durationMs = performance.now() - startedAt;
const sorted = [...latencies].sort((left, right) => left - right);
process.stdout.write(JSON.stringify({
completed,
concurrency: options.concurrency,
durationMs: round(durationMs),
errors,
latencyAvgMs: round(sorted.reduce((total, value) => total + value, 0) / Math.max(1, sorted.length)),
latencyMaxMs: round(sorted.at(-1) ?? 0),
latencyP50Ms: round(percentile(sorted, 0.5)),
latencyP95Ms: round(percentile(sorted, 0.95)),
latencyP99Ms: round(percentile(sorted, 0.99)),
qps: round(completed / (durationMs / 1_000)),
requests: options.requests
}));
}
function parseArgs(values) {
const options = {
bodyBytes: 1_024,
concurrency: 64,
prefix: "benchmark",
requests: 10_000,
url: ""
};
for (let index = 0; index < values.length; index += 1) {
if (values[index] === "--body-bytes") options.bodyBytes = positiveInteger(values[index + 1], options.bodyBytes);
if (values[index] === "--concurrency") options.concurrency = positiveInteger(values[index + 1], options.concurrency);
if (values[index] === "--prefix") options.prefix = values[index + 1] || options.prefix;
if (values[index] === "--requests") options.requests = positiveInteger(values[index + 1], options.requests);
if (values[index] === "--url") options.url = values[index + 1] || "";
}
if (!options.url) throw new Error("--url is required");
return options;
}
function positiveInteger(value, fallback) {
const parsed = Number(value);
return Number.isFinite(parsed) && parsed > 0 ? Math.floor(parsed) : fallback;
}
function percentile(sorted, fraction) {
if (sorted.length === 0) return 0;
return sorted[Math.min(sorted.length - 1, Math.floor((sorted.length - 1) * fraction))];
}
function round(value) {
return Math.round(value * 1_000) / 1_000;
}

View file

@ -8,6 +8,7 @@ import { normalizeGrokProviderAccountConfig } from "@ccr/core/agents/local-provi
import { removeOpenCodeProviderAccountConfig } from "@ccr/core/agents/local-providers/opencode";
import { CLAUDE_CODE_DEFAULT_ENV, CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY_ENV, DEFAULT_OVERVIEW_WIDGETS, DEFAULT_TRAY_COMPONENT_VARIANTS, DEFAULT_TRAY_WIDGETS, DEFAULT_TRAY_WINDOW_MODULES, OVERVIEW_WIDGET_SIZE_VALUES, ROUTER_FALLBACK_MAX_RETRY_COUNT, TRAY_SINGLETON_WIDGET_TYPES, TRAY_TOP_WIDGET_TYPES, TRAY_WINDOW_MODULE_IDS, enforceSingleEnabledGlobalProfilePerAgent } from "@ccr/core/contracts/app";
import { createDefaultAppConfig } from "@ccr/core/config/default-config";
import { maxRequestLogBodyBytes } from "@ccr/core/observability/request-log-limits";
import { findProviderPresetByBaseUrl, providerApiKeySafetyIssue, providerEndpointCanReceiveProviderApiKey } from "@ccr/core/providers/presets/index";
import type {
AppConfig,
@ -695,6 +696,15 @@ function parseObservability(value: unknown): Partial<ObservabilityConfig> | unde
if (typeof value.agentAnalysis === "boolean") {
observability.agentAnalysis = value.agentAnalysis;
}
if (value.requestLogBodyCapture === "all" || value.requestLogBodyCapture === "errors" || value.requestLogBodyCapture === "none") {
observability.requestLogBodyCapture = value.requestLogBodyCapture;
}
if (typeof value.requestLogMaxBodyBytes === "number" && Number.isFinite(value.requestLogMaxBodyBytes)) {
observability.requestLogMaxBodyBytes = Math.max(0, Math.min(maxRequestLogBodyBytes, Math.floor(value.requestLogMaxBodyBytes)));
}
if (typeof value.requestLogSuccessSampleRate === "number" && Number.isFinite(value.requestLogSuccessSampleRate)) {
observability.requestLogSuccessSampleRate = Math.max(0, Math.min(1, value.requestLogSuccessSampleRate));
}
return Object.keys(observability).length ? observability : undefined;
}

View file

@ -7,6 +7,7 @@ import {
type AppConfig,
type ProxyRouteTarget
} from "@ccr/core/contracts/app";
import { defaultRequestLogBodyBytes } from "@ccr/core/observability/request-log-limits";
export const DEFAULT_PROXY_TARGETS: ProxyRouteTarget[] = [
{ host: "api.anthropic.com", paths: ["/v1/messages", "/v1/messages/count_tokens"] },
@ -101,6 +102,9 @@ export function createDefaultAppConfig(options: DefaultAppConfigOptions): AppCon
launchAtLogin: false,
observability: {
agentAnalysis: false,
requestLogBodyCapture: "all",
requestLogMaxBodyBytes: defaultRequestLogBodyBytes,
requestLogSuccessSampleRate: 1,
requestLogs: false
},
preferredProvider: "",

View file

@ -953,6 +953,9 @@ export type ProxyRuntimeConfig = {
export type ObservabilityConfig = {
agentAnalysis: boolean;
requestLogBodyCapture?: "all" | "errors" | "none";
requestLogMaxBodyBytes?: number;
requestLogSuccessSampleRate?: number;
requestLogs: boolean;
};
@ -1708,6 +1711,94 @@ export type RequestLogRetryAttempt = {
status?: string;
};
export type RequestRouteTracePhase =
| "ingress"
| "compatibility"
| "routing"
| "capability"
| "enrichment"
| "planning"
| "attempt"
| "core"
| "outcome";
export type RequestRouteTraceChange = {
after?: unknown;
before?: unknown;
operation: "add" | "remove" | "replace";
path: string;
redacted?: boolean;
scope: "body" | "headers" | "routing" | "url";
truncated?: boolean;
};
export type RequestRouteTraceDecision = {
diagnostics?: Array<{
code: string;
message: string;
model?: string;
ruleId?: string;
source?: string;
}>;
policyId?: string;
reason?: string;
ruleId?: string;
ruleName?: string;
source?: string;
};
export type RequestRouteTraceTarget = {
credentialCandidates?: string[];
credentialId?: string;
model?: string;
protocol?: GatewayProviderProtocol;
provider?: string;
};
export type RequestRouteTraceOutcome = {
error?: string;
fallbackReason?: string;
retryDelayMs?: number;
statusCode?: number;
};
export type RequestRouteTraceSnapshot = {
body?: unknown;
bodySizeBytes: number;
bodyTruncated: boolean;
headers: Record<string, unknown>;
method: string;
routing?: Record<string, unknown>;
url: string;
};
export type RequestRouteTraceHop = {
attempt?: number;
changes: RequestRouteTraceChange[];
decision?: RequestRouteTraceDecision;
durationMs: number;
kind: "attempt" | "decision" | "mutation" | "outcome" | "snapshot";
name: string;
outcome?: RequestRouteTraceOutcome;
phase: RequestRouteTracePhase;
seq: number;
startedOffsetMs: number;
status: "error" | "noop" | "ok";
target?: RequestRouteTraceTarget;
truncated?: boolean;
};
export type RequestRouteTrace = {
attemptCount: number;
complete: boolean;
finalSnapshot?: RequestRouteTraceSnapshot;
hopCount: number;
hops: RequestRouteTraceHop[];
ingressSnapshot?: RequestRouteTraceSnapshot;
truncated: boolean;
version: 1 | 2;
};
export type RequestLogEntry = {
cacheReadTokens: number;
cacheWriteTokens: number;
@ -1733,6 +1824,10 @@ export type RequestLogEntry = {
requestBody: RequestLogBody;
requestHeaders: Record<string, string | string[]>;
requestId: string;
routeAttemptCount: number;
routeHopCount: number;
routeTrace?: RequestRouteTrace;
routeTraceTruncated: boolean;
retryAttempts: RequestLogRetryAttempt[];
responseBody?: RequestLogBody;
responseHeaders: Record<string, string | string[]>;

View file

@ -42,8 +42,7 @@ class GatewayService {
getConfig: () => this.config,
getCoreAuthToken: () => this.coreAuthToken,
getPlugin: () => this.plugin,
getStatus: () => ({ coreEndpoint: this.status.coreEndpoint, endpoint: this.status.endpoint }),
takePendingRawTraceUpdate: (requestId) => this.rawTraceSynchronizer.take(requestId)
getStatus: () => ({ coreEndpoint: this.status.coreEndpoint, endpoint: this.status.endpoint })
});
private browserAutomationMcpIntegration?: BrowserAutomationMcpIntegration;
@ -56,7 +55,9 @@ class GatewayService {
private config?: AppConfig;
private coreAuthToken = "";
private plugin?: ClaudeCodeRouterPlugin;
private readonly rawTraceSynchronizer = new RawTraceSynchronizer();
private readonly rawTraceSynchronizer = new RawTraceSynchronizer({
getConfig: () => this.config
});
private server?: Server;
private status: GatewayStatus = {
coreEndpoint: "",

View file

@ -2,13 +2,14 @@ import { createRequire } from "node:module";
import { EventEmitter } from "node:events";
import os from "node:os";
import path from "node:path";
import { type AppConfig, type ProfileClientKind, type RouterBuiltInAgentRuleId, type RouterFallbackConfig, type RouterRule, type RouterRuleCondition, type RouterRuleRewrite } from "@ccr/core/contracts/app";
import { type AppConfig, type ProfileClientKind, type RequestRouteTraceChange, type RouterBuiltInAgentRuleId, type RouterFallbackConfig, type RouterRule, type RouterRuleCondition, type RouterRuleRewrite } from "@ccr/core/contracts/app";
import { CONFIGDIR } from "@ccr/core/config/constants";
import { applyAgentRequestEnrichers } from "@ccr/core/agents/request-enricher";
import { compileRouterConfig, type CompiledRouterConfig, type CompiledRouterRule } from "@ccr/core/routing/config-compiler";
import type { RouteDecision, RouteDiagnostic, RouteModelRef, RouteRequest, RouteSource } from "@ccr/core/routing/contracts";
import { ModelRegistry, normalizeRouteSelector } from "@ccr/core/routing/model-registry";
import { RoutePolicyEngine, type RoutePolicy } from "@ccr/core/routing/policy-engine";
import type { RouteTraceObserver } from "@ccr/core/observability/route-trace";
export { normalizeRouteSelector } from "@ccr/core/routing/model-registry";
@ -27,6 +28,11 @@ export type ClaudeCodeRouteDecision = {
};
type ConfiguredRouteDecision = Omit<RouteDecision, "diagnostics">;
type ResolvedConfiguredRouteDecision = ConfiguredRouteDecision & {
policyId: string;
ruleId?: string;
ruleName?: string;
};
const requireFromHere = createRequire(__filename);
@ -42,6 +48,7 @@ export class ClaudeCodeRouterPlugin {
body: Record<string, unknown>;
headers: Record<string, HeaderValue>;
method: string;
trace?: RouteTraceObserver;
url: string;
}): Promise<{ body: Record<string, unknown>; decision: ClaudeCodeRouteDecision }> {
const body = cloneRecord(input.body);
@ -52,7 +59,8 @@ export class ClaudeCodeRouterPlugin {
method: input.method,
url: input.url
};
applyAgentRequestEnrichers(request, [{
const enrichmentStartedAt = Date.now();
const appliedEnrichers = applyAgentRequestEnrichers(request, [{
enrich: (matchedRequest) => {
injectClaudeCodeAgentToolDescription(matchedRequest.body, this.config);
injectClaudeCodeToolHubInstructions(matchedRequest.body, this.config);
@ -62,12 +70,33 @@ export class ClaudeCodeRouterPlugin {
id: "claude-code",
matches: (candidate) => builtInAgentRouteMatches(candidate, this.config, "claude-code")
}]);
if (appliedEnrichers.length > 0) {
input.trace?.capture({
changes: [{ operation: "replace", path: "/body", scope: "body" }],
durationMs: Date.now() - enrichmentStartedAt,
kind: "mutation",
name: `agent-enricher.${appliedEnrichers.join("+")}`,
phase: "enrichment",
startedAtMs: enrichmentStartedAt
});
}
const sessionId = resolveSessionId(body, input.headers);
const tokenCount = calculateTokenCount(body.messages, body.system, body.tools);
request.sessionId = sessionId;
request.tokenCount = tokenCount;
const customRouteStartedAt = Date.now();
const requestedCustomModel = await this.resolveCustomRoute(request);
if (this.config.CUSTOM_ROUTER_PATH) {
input.trace?.capture({
durationMs: Date.now() - customRouteStartedAt,
kind: "decision",
name: "custom-router",
phase: "routing",
startedAtMs: customRouteStartedAt,
target: requestedCustomModel ? { model: requestedCustomModel } : undefined
});
}
const customModel = this.compiled.modelRegistry.resolve(requestedCustomModel);
const customDiagnostic: RouteDiagnostic[] = requestedCustomModel && !customModel
? [{
@ -78,13 +107,64 @@ export class ClaudeCodeRouterPlugin {
}]
: [];
const configuredDecision = resolveConfiguredRouteDecision(request, this.config, this.compiled, customModel);
const traceDecision = {
diagnostics: [...this.compiled.diagnostics, ...customDiagnostic],
policyId: configuredDecision.policyId,
reason: configuredDecision.reason,
...(configuredDecision.ruleId ? { ruleId: configuredDecision.ruleId } : {}),
...(configuredDecision.ruleName ? { ruleName: configuredDecision.ruleName } : {}),
source: configuredDecision.source
};
input.trace?.capture({
decision: traceDecision,
kind: "decision",
name: "router.policy",
phase: "routing",
target: configuredDecision.model ? {
model: configuredDecision.model.selector,
...(configuredDecision.model.kind === "provider" ? { provider: configuredDecision.model.provider.name } : {})
} : undefined
});
if (configuredDecision.rewrites.length) {
for (const rewrite of configuredDecision.rewrites) {
applyRouterRewrite(rewrite, request);
const rewriteStartedAt = Date.now();
const change = applyRouterRewrite(rewrite, request);
input.trace?.capture({
changes: change ? [change] : [],
decision: traceDecision,
durationMs: Date.now() - rewriteStartedAt,
kind: "mutation",
name: `router.rewrite:${rewrite.operation ?? "set"}:${rewrite.key}`,
phase: "routing",
startedAtMs: rewriteStartedAt
});
}
}
if (configuredDecision.model) {
const modelStartedAt = Date.now();
const previousModel = body.model;
body.model = configuredDecision.model.selector;
input.trace?.capture({
changes: Object.is(previousModel, configuredDecision.model.selector)
? []
: [{
...(previousModel === undefined ? {} : { before: previousModel }),
after: configuredDecision.model.selector,
operation: previousModel === undefined ? "add" : "replace",
path: "/body/model",
scope: "body"
}],
decision: traceDecision,
durationMs: Date.now() - modelStartedAt,
kind: "mutation",
name: "router.model-selection",
phase: "routing",
startedAtMs: modelStartedAt,
target: {
model: configuredDecision.model.selector,
...(configuredDecision.model.kind === "provider" ? { provider: configuredDecision.model.provider.name } : {})
}
});
}
const routedModel = configuredDecision.model?.selector ?? readString(body.model);
@ -197,7 +277,7 @@ function resolveConfiguredRouteDecision(
config: AppConfig,
compiled: CompiledRouterConfig,
customModel?: RouteModelRef
): ConfiguredRouteDecision {
): ResolvedConfiguredRouteDecision {
const requestedModel = readString(request.body.model);
const explicitModel = normalizeRouteSelector(requestedModel);
const builtInDecision = resolveBuiltInAgentRouteDecision(request, config, compiled.modelRegistry, compiled.fallback);
@ -248,9 +328,20 @@ function resolveConfiguredRouteDecision(
}
];
const match = new RoutePolicyEngine(policies).evaluate(request);
return match?.decision ?? {
if (match) {
const compiledRule = match.policyId.startsWith("rule:")
? compiled.rules.find((rule) => `rule:${rule.rule.id}` === match.policyId)
: undefined;
return {
...match.decision,
policyId: match.policyId,
...(compiledRule ? { ruleId: compiledRule.rule.id, ruleName: compiledRule.rule.name } : {})
};
}
return {
fallback: compiled.fallback,
model: compiled.modelRegistry.resolve(explicitModel),
policyId: "default",
reason: "default",
rewrites: [],
source: "default"
@ -890,32 +981,65 @@ function routerRuleRewriteDecision(
};
}
function applyRouterRewrite(rewrite: RouterRuleRewrite, request: MutableRequestLike): void {
function applyRouterRewrite(rewrite: RouterRuleRewrite, request: MutableRequestLike): RequestRouteTraceChange | undefined {
const parts = rewrite.key
.split(".")
.map((part) => part.trim())
.filter(Boolean);
const [scope, section, ...rest] = parts;
if (scope !== "request") {
return;
return undefined;
}
if (section === "header" || section === "headers") {
const name = rest.join(".").trim().toLowerCase();
if (!name) {
return;
return undefined;
}
const before = request.headers[name];
if ((rewrite.operation ?? "set") === "delete") {
delete request.headers[name];
} else if (rewrite.value !== undefined) {
request.headers[name] = rewrite.value;
}
return;
const after = request.headers[name];
return createReportedRewriteChange("headers", `/headers/${escapeJsonPointer(name)}`, before, after);
}
if (section === "body") {
const before = readPathValue(request.body, rest);
applyBodyRewrite(request.body, rest, rewrite);
const after = readPathValue(request.body, rest);
return createReportedRewriteChange(
"body",
`/body/${rest.map(escapeJsonPointer).join("/")}`,
before,
after
);
}
return undefined;
}
function createReportedRewriteChange(
scope: RequestRouteTraceChange["scope"],
path: string,
before: unknown,
after: unknown
): RequestRouteTraceChange | undefined {
if (Object.is(before, after)) {
return undefined;
}
return {
...(after === undefined ? {} : { after }),
...(before === undefined ? {} : { before }),
operation: before === undefined ? "add" : after === undefined ? "remove" : "replace",
path,
scope
};
}
function escapeJsonPointer(value: string): string {
return value.replaceAll("~", "~0").replaceAll("/", "~1");
}
function applyBodyRewrite(body: Record<string, unknown>, path: string[], rewrite: RouterRuleRewrite): void {

View file

@ -1,6 +1,7 @@
/**
* Extracted from gateway/service.ts. Keep this module focused on its named gateway boundary.
*/
import { join as pathJoin } from "node:path";
import type { AppConfig, GatewayProviderConfig, GatewayProviderProtocol, VirtualModelProfileConfig } from "@ccr/core/contracts/app";
import { codexDefaultBaseUrl, readCodexAuth, readGrokAuth, resolveGrokAuth } from "@ccr/core/agents/local-providers/service";
import { grokAccessTokenExpired, grokClientVersion } from "@ccr/core/agents/local-providers/grok";
@ -18,6 +19,8 @@ import { uniqueStrings } from "@ccr/core/gateway/internal/collections";
import { isLocalClaudeCodeOauthProviderPlugin, mergeAnthropicBetaValues } from "@ccr/core/providers/oauth-plugin";
import { resolveConfiguredProviderModelSelector, resolveUniqueConfiguredProviderModelSelector } from "@ccr/core/routing/model-resolution";
const upstreamHeaderSanitizerPluginKey = "ccr-upstream-header-sanitizer";
export async function compileCoreGatewayConfig(
config: AppConfig,
@ -28,6 +31,11 @@ export async function compileCoreGatewayConfig(
upstreamProxyUrl?: string
): Promise<Record<string, unknown>> {
const pluginCoreGatewayConfig = pluginService.getCoreGatewayConfig();
const configuredGatewayPlugins = Array.isArray(pluginCoreGatewayConfig.plugins)
? pluginCoreGatewayConfig.plugins.filter((plugin) =>
!isRecord(plugin) || stringValue(plugin.key) !== upstreamHeaderSanitizerPluginKey
)
: [];
const pluginBillingConfig = isRecord(pluginCoreGatewayConfig.billing) ? pluginCoreGatewayConfig.billing : {};
const configuredProviderPlugins = normalizeClaudeCodeOauthProviderPlugins([
...(config.providerPlugins ?? []).filter(providerPluginEnabled),
@ -108,6 +116,14 @@ export async function compileCoreGatewayConfig(
enabled: false
},
port: config.gateway.corePort,
plugins: [
...configuredGatewayPlugins,
{
enabled: true,
key: upstreamHeaderSanitizerPluginKey,
modulePath: pathJoin(__dirname, "upstream-header-sanitizer.js")
}
],
upstreamTimeoutMs: Number(config.API_TIMEOUT_MS) || 0,
agent: {
...pluginAgentConfig,

View file

@ -114,6 +114,13 @@ function createGatewayProcessEnv(config: AppConfig, upstreamProxyUrl: string | u
PORT: String(config.gateway.corePort)
};
// The managed gateway must use the generated raw-trace policy. Inheriting
// the upstream gateway's RAW_TRACE_* overrides could bypass CCR privacy,
// size, spool, or sync-endpoint controls.
for (const key of Object.keys(env)) {
if (key.startsWith("RAW_TRACE_")) delete env[key];
}
const noProxy = mergeNoProxy(env.NO_PROXY || env.no_proxy, [
"127.0.0.1",
"localhost",

View file

@ -0,0 +1,48 @@
type UpstreamRequest = {
body: unknown;
bodyEncoding?: "bytes" | "form" | "json" | "none" | "text";
headers: Record<string, string>;
method?: string;
url: string;
};
type ProviderPluginRequestInput = {
upstreamRequest: UpstreamRequest;
};
const ccrAuthHeaderNames = new Set([
"x-auth-api-key-id",
"x-auth-sub"
]);
/**
* Removes CCR-owned routing, authentication and observability metadata at the
* final provider boundary. Provider credentials and non-CCR custom X-Auth
* headers are deliberately preserved.
*/
export function sanitizeUpstreamProviderHeaders(headers: Record<string, string>): Record<string, string> {
const sanitized: Record<string, string> = {};
for (const [name, value] of Object.entries(headers)) {
const normalized = name.trim().toLowerCase();
if (normalized.startsWith("x-ccr-") || ccrAuthHeaderNames.has(normalized)) continue;
sanitized[name] = value;
}
return sanitized;
}
export function createGatewayPlugin() {
return {
providerHooks: [{
key: "ccr-upstream-header-sanitizer",
transformRequest(input: ProviderPluginRequestInput) {
return {
ok: true as const,
value: {
...input.upstreamRequest,
headers: sanitizeUpstreamProviderHeaders(input.upstreamRequest.headers)
}
};
}
}]
};
}

View file

@ -145,7 +145,9 @@ export type ApiKeyWindowCounter = {
export type RawTracePartText = {
contentType?: string;
text: string;
filePath: string;
sizeBytes: number;
truncated: boolean;
};

View file

@ -1,8 +1,8 @@
import { randomUUID } from "node:crypto";
import type { IncomingMessage, ServerResponse } from "node:http";
import { Readable } from "node:stream";
import type { ApiKeyConfig, AppConfig } from "@ccr/core/contracts/app";
import { createSseErrorDetector, recordGatewayRequestLog, updateGatewayRequestLogFromRawTrace, type RequestLogRawTraceUpdateInput } from "@ccr/core/observability/request-log-store";
import type { ApiKeyConfig, AppConfig, RequestRouteTraceChange } from "@ccr/core/contracts/app";
import { createSseErrorDetector, recordGatewayRequestLog } from "@ccr/core/observability/request-log-store";
import { recordGatewayUsageCapture, type UsageCaptureInput } from "@ccr/core/usage/store";
import { ClaudeCodeRouterPlugin } from "@ccr/core/gateway/claude-code-router-plugin";
import { adaptRouteRequestBody, restoreRouteRequestBody } from "@ccr/core/routing/protocol-adapter";
@ -13,7 +13,8 @@ import { prepareCursorOpenAICompatChatBody } from "@ccr/core/gateway/features/cu
import { filteredResponseHeaders, formatError, forwardHeaders, inferGatewayClient, parseJsonObject, readRequestBody, sendJson, shouldCaptureGatewayUsage, shouldSendBody, stripLocalGatewayAuthHeaders } from "@ccr/core/gateway/http/io";
import { createGatewayModelsResponse, prepareClaudeAppDiscoveredModelRequest, prepareClaudeCodeDiscoveredModelRequest, shouldServeGatewayModelsResponse } from "@ccr/core/gateway/features/model-discovery";
import { resolveProviderLogName, resolveResponseProviderProtocol, sanitizeHeaderValue } from "@ccr/core/providers/runtime-topology";
import { createBodySampler, shouldRecordRequestLogs } from "@ccr/core/observability/raw-trace-sync";
import { createBodySampler, requestLogSampled, shouldRecordRequestLogs } from "@ccr/core/observability/raw-trace-sync";
import { RequestRouteTraceRecorder } from "@ccr/core/observability/route-trace";
import { endpoint } from "@ccr/core/gateway/core-runtime/supervisor";
import { coreGatewayUsageAttributionConfig } from "@ccr/core/gateway/core-runtime/config-compiler";
import { clientClosedRequestStatusCode, clientDisconnectMessage, UpstreamRequestError } from "@ccr/core/gateway/internal/shared";
@ -28,9 +29,30 @@ export type GatewayRequestPipelineDependencies = {
getCoreAuthToken: () => string;
getPlugin: () => ClaudeCodeRouterPlugin | undefined;
getStatus: () => { coreEndpoint: string; endpoint: string };
takePendingRawTraceUpdate: (requestId: string) => RequestLogRawTraceUpdateInput | undefined;
};
function reportedRouteChange(
scope: RequestRouteTraceChange["scope"],
path: string,
before: unknown,
after: unknown
): RequestRouteTraceChange | undefined {
if (Object.is(before, after)) {
return undefined;
}
return {
...(after === undefined ? {} : { after }),
...(before === undefined ? {} : { before }),
operation: before === undefined ? "add" : after === undefined ? "remove" : "replace",
path,
scope
};
}
function isReportedRouteChange(change: RequestRouteTraceChange | undefined): change is RequestRouteTraceChange {
return change !== undefined;
}
export class GatewayRequestPipeline {
constructor(private readonly dependencies: GatewayRequestPipelineDependencies) {}
@ -39,7 +61,6 @@ export class GatewayRequestPipeline {
private get coreAuthToken() { return this.dependencies.getCoreAuthToken(); }
private get plugin() { return this.dependencies.getPlugin(); }
private get status() { return this.dependencies.getStatus(); }
private takePendingRawTraceUpdate(requestId: string) { return this.dependencies.takePendingRawTraceUpdate(requestId); }
async proxyRequest(request: IncomingMessage, response: ServerResponse, path: string, apiKey?: ApiKeyConfig): Promise<void> {
if (!this.config || !this.plugin) {
@ -47,46 +68,113 @@ export class GatewayRequestPipeline {
return;
}
const method = request.method ?? "GET";
const requestBody = await readRequestBody(request);
const startedAt = Date.now();
const startedAtIso = new Date(startedAt).toISOString();
const requestId = randomUUID();
const requestUrl = new URL(request.url || path, this.status.endpoint || "http://127.0.0.1").toString();
const routeTrace = shouldRecordRequestLogs(this.config)
? new RequestRouteTraceRecorder(startedAt)
: undefined;
routeTrace?.captureIngress();
const headerNormalizationStartedAt = Date.now();
const headers = forwardHeaders(request.headers);
const previousAuthorization = headers.authorization;
const previousApiKey = headers["x-api-key"];
const previousLegacyApiKey = headers["api-key"];
const previousAuthApiKeyId = headers["x-auth-api-key-id"];
const previousAuthSub = headers["x-auth-sub"];
const previousClientRequestId = headers["x-client-request-id"];
if (apiKey) {
stripLocalGatewayAuthHeaders(headers);
headers["x-auth-api-key-id"] = apiKey.id;
headers["x-auth-sub"] = apiKey.id;
}
const method = request.method ?? "GET";
const requestBody = await readRequestBody(request);
headers["x-client-request-id"] = requestId;
routeTrace?.capture({
changes: [
...(apiKey ? [
reportedRouteChange("headers", "/headers/authorization", previousAuthorization, undefined),
reportedRouteChange("headers", "/headers/x-api-key", previousApiKey, undefined),
reportedRouteChange("headers", "/headers/api-key", previousLegacyApiKey, undefined),
reportedRouteChange("headers", "/headers/x-auth-api-key-id", previousAuthApiKeyId, apiKey.id),
reportedRouteChange("headers", "/headers/x-auth-sub", previousAuthSub, apiKey.id)
] : []),
reportedRouteChange("headers", "/headers/x-client-request-id", previousClientRequestId, requestId)
].filter(isReportedRouteChange),
durationMs: Date.now() - headerNormalizationStartedAt,
kind: "mutation",
name: "gateway.header-normalization",
phase: "ingress",
startedAtMs: headerNormalizationStartedAt
});
const client = inferGatewayClient(apiKey, request.headers);
const cursorCompatStartedAt = Date.now();
const cursorCompatPreparation = prepareCursorOpenAICompatChatBody(this.config, client, method, path, requestBody);
if (cursorCompatPreparation) {
headers["x-ccr-cursor-openai-compat"] = sanitizeHeaderValue(cursorCompatPreparation.diagnostic);
}
let bodyToForward: Buffer | undefined = cursorCompatPreparation?.body ?? requestBody;
if (cursorCompatPreparation) {
routeTrace?.capture({
changes: [
{ operation: "replace", path: "/body", scope: "body" },
{ after: headers["x-ccr-cursor-openai-compat"], operation: "add", path: "/headers/x-ccr-cursor-openai-compat", scope: "headers" }
],
durationMs: Date.now() - cursorCompatStartedAt,
kind: "mutation",
name: "compatibility.cursor-openai",
phase: "compatibility",
startedAtMs: cursorCompatStartedAt
});
}
let routeFallback = this.config.Router.fallback;
let routedModel: string | undefined;
let codexApplyPatchBridgeActive = false;
const claudeModelRewriteStartedAt = Date.now();
const claudeModelRewrite = prepareClaudeCodeDiscoveredModelRequest(this.config, request.headers, method, path, bodyToForward);
if (claudeModelRewrite) {
headers["x-ccr-claude-model-discovery"] = sanitizeHeaderValue(claudeModelRewrite.diagnostic);
bodyToForward = claudeModelRewrite.body;
routeTrace?.capture({
changes: [
{ operation: "replace", path: "/body/model", scope: "body" },
{ after: headers["x-ccr-claude-model-discovery"], operation: "add", path: "/headers/x-ccr-claude-model-discovery", scope: "headers" }
],
durationMs: Date.now() - claudeModelRewriteStartedAt,
kind: "mutation",
name: "model-discovery.claude-code",
phase: "compatibility",
startedAtMs: claudeModelRewriteStartedAt
});
}
const claudeAppModelRewriteStartedAt = Date.now();
const claudeAppModelRewrite = prepareClaudeAppDiscoveredModelRequest(this.config, method, path, bodyToForward);
if (claudeAppModelRewrite) {
headers["x-ccr-claude-app-model-rewrite"] = sanitizeHeaderValue(claudeAppModelRewrite.diagnostic);
bodyToForward = claudeAppModelRewrite.body;
routedModel = claudeAppModelRewrite.routedModel;
routeTrace?.capture({
changes: [
{ after: routedModel, operation: "replace", path: "/body/model", scope: "body" },
{ after: headers["x-ccr-claude-app-model-rewrite"], operation: "add", path: "/headers/x-ccr-claude-app-model-rewrite", scope: "headers" }
],
durationMs: Date.now() - claudeAppModelRewriteStartedAt,
kind: "mutation",
name: "model-discovery.claude-app",
phase: "compatibility",
startedAtMs: claudeAppModelRewriteStartedAt,
target: routedModel ? { model: routedModel } : undefined
});
}
if (!reserveApiKeyLimits(apiKey, request, response, bodyToForward)) {
return;
}
const startedAt = Date.now();
const startedAtIso = new Date(startedAt).toISOString();
const requestId = randomUUID();
headers["x-client-request-id"] = requestId;
const usageAttributionConfig = coreGatewayUsageAttributionConfig(this.config);
const recordUsage = (input: Omit<UsageCaptureInput, "config">) => {
void recordGatewayUsageCapture({ ...input, config: usageAttributionConfig });
};
const requestUrl = new URL(request.url || path, this.status.endpoint || "http://127.0.0.1").toString();
const upstreamAbortController = new AbortController();
let clientDisconnected = false;
let responseCompleted = false;
@ -121,38 +209,44 @@ export class GatewayRequestPipeline {
responseHeaders: Headers,
responseBodyText = "",
responseBodyTruncated = false,
error?: string
error?: string,
responseBodySizeBytes = Buffer.byteLength(responseBodyText)
) => {
const config = this.config;
if (!config || !shouldRecordRequestLogs(config)) {
return;
}
void (async () => {
await recordGatewayRequestLog({
client,
completedAt: new Date().toISOString(),
durationMs: Date.now() - startedAt,
error,
fallbackModel: routedModel,
method,
path,
providerName: resolveProviderLogName(responseHeaders, config, routedModel),
providerProtocol: resolveResponseProviderProtocol(responseHeaders, this.config),
requestBody: shouldSendBody(method) ? bodyToForward ?? Buffer.alloc(0) : Buffer.alloc(0),
requestHeaders: headers,
requestId,
responseBodyText,
responseBodyTruncated,
responseHeaders,
startedAt: startedAtIso,
statusCode,
url: requestUrl
});
const pendingRawTraceUpdate = this.takePendingRawTraceUpdate(requestId);
if (pendingRawTraceUpdate) {
await updateGatewayRequestLogFromRawTrace(pendingRawTraceUpdate);
}
})();
const successful = statusCode >= 200 && statusCode < 400 && !error;
const successSampleRate = config.observability.requestLogSuccessSampleRate ?? 1;
if (successful && !requestLogSampled(requestId, successSampleRate)) {
return;
}
const bodyCapture = config.observability.requestLogBodyCapture ?? "all";
recordGatewayRequestLog({
captureBody: bodyCapture === "all" || (bodyCapture === "errors" && !successful),
client,
completedAt: new Date().toISOString(),
durationMs: Date.now() - startedAt,
error,
fallbackModel: routedModel,
maxBodyBytes: config.observability.requestLogMaxBodyBytes,
method,
model: routedModel,
path,
providerName: resolveProviderLogName(responseHeaders, config, routedModel),
providerProtocol: resolveResponseProviderProtocol(responseHeaders, this.config),
requestBody: shouldSendBody(method) ? bodyToForward ?? Buffer.alloc(0) : Buffer.alloc(0),
requestHeaders: headers,
requestId,
routeTrace: routeTrace?.finish(),
responseBodyText,
responseBodySizeBytes,
responseBodyTruncated,
responseHeaders,
startedAt: startedAtIso,
statusCode,
url: requestUrl
});
};
const shouldCaptureUsage = shouldCaptureGatewayUsage(method, path);
@ -171,11 +265,21 @@ export class GatewayRequestPipeline {
}
if (shouldApplyGatewayRouting(method, path)) {
const routeAdaptationStartedAt = Date.now();
const adaptation = adaptRouteRequestBody(path, parseJsonObject(bodyToForward ?? requestBody));
routeTrace?.capture({
changes: [{ operation: "replace", path: "/body", scope: "body" }],
durationMs: Date.now() - routeAdaptationStartedAt,
kind: "mutation",
name: "protocol-adapter.route-input",
phase: "routing",
startedAtMs: routeAdaptationStartedAt
});
const routed = await this.plugin.routeRequest({
body: adaptation.body,
headers: headers as Record<string, string | string[] | undefined>,
method,
trace: routeTrace,
url: request.url ?? path
});
const serialized = Buffer.from(`${JSON.stringify(restoreRouteRequestBody(routed.body, adaptation))}\n`, "utf8");
@ -191,8 +295,27 @@ export class GatewayRequestPipeline {
routedModel = routed.decision.model;
}
bodyToForward = serialized;
routeTrace?.capture({
changes: [
{ operation: "replace", path: "/body", scope: "body" },
{ after: headers["content-type"], operation: "replace", path: "/headers/content-type", scope: "headers" },
{ after: headers["x-ccr-route-reason"], operation: "add", path: "/headers/x-ccr-route-reason", scope: "headers" },
{ after: headers["x-ccr-route-source"], operation: "add", path: "/headers/x-ccr-route-source", scope: "headers" },
...(routedModel ? [{ after: routedModel, operation: "add" as const, path: "/headers/x-ccr-routed-model", scope: "headers" as const }] : [])
],
decision: {
diagnostics: routed.decision.diagnostics,
reason: routed.decision.reason,
source: routed.decision.source
},
kind: "mutation",
name: "router.route-output",
phase: "routing",
target: routedModel ? { model: routedModel } : undefined
});
}
const codexBridgeStartedAt = Date.now();
const codexApplyPatchBridgeRequest = prepareCodexApplyPatchBridgeRequest({
body: bodyToForward,
config: this.config,
@ -206,8 +329,29 @@ export class GatewayRequestPipeline {
codexApplyPatchBridgeActive = true;
headers["x-ccr-codex-patch-bridge"] = sanitizeHeaderValue(codexApplyPatchBridgeRequest.diagnostic);
headers["content-type"] = "application/json";
routeTrace?.capture({
changes: [
{ operation: "replace", path: "/body", scope: "body" },
{ after: headers["x-ccr-codex-patch-bridge"], operation: "add", path: "/headers/x-ccr-codex-patch-bridge", scope: "headers" },
{ after: headers["content-type"], operation: "replace", path: "/headers/content-type", scope: "headers" }
],
durationMs: Date.now() - codexBridgeStartedAt,
kind: "mutation",
name: "compatibility.codex-apply-patch",
phase: "compatibility",
startedAtMs: codexBridgeStartedAt
});
}
const capabilityRoutingStartedAt = Date.now();
const capabilityBodyBefore = bodyToForward;
const capabilityFallbackBefore = routeFallback;
const capabilityModelBefore = routedModel;
const capabilityProviderHeadersBefore = {
gateway: headers["x-gateway-target-provider"],
list: headers["x-target-providers"],
target: headers["x-target-provider"]
};
const providerCapabilityRouting = applyProviderCapabilityRouting({
body: bodyToForward,
config: this.config,
@ -219,6 +363,22 @@ export class GatewayRequestPipeline {
bodyToForward = providerCapabilityRouting.body;
routeFallback = providerCapabilityRouting.fallback;
routedModel = providerCapabilityRouting.routedModel;
routeTrace?.capture({
changes: [
...(capabilityBodyBefore === bodyToForward ? [] : [{ operation: "replace" as const, path: "/body/model", scope: "body" as const }]),
reportedRouteChange("routing", "/routing/model", capabilityModelBefore, routedModel),
...(capabilityFallbackBefore === routeFallback ? [] : [{ after: routeFallback, before: capabilityFallbackBefore, operation: "replace" as const, path: "/routing/fallback", scope: "routing" as const }]),
reportedRouteChange("headers", "/headers/x-target-provider", capabilityProviderHeadersBefore.target, headers["x-target-provider"]),
reportedRouteChange("headers", "/headers/x-target-providers", capabilityProviderHeadersBefore.list, headers["x-target-providers"]),
reportedRouteChange("headers", "/headers/x-gateway-target-provider", capabilityProviderHeadersBefore.gateway, headers["x-gateway-target-provider"])
].filter(isReportedRouteChange),
durationMs: Date.now() - capabilityRoutingStartedAt,
kind: "mutation",
name: "provider.capability-routing",
phase: "capability",
startedAtMs: capabilityRoutingStartedAt,
target: routedModel ? { model: routedModel } : undefined
});
const hostedWebSearchProtocolContext = createHostedWebSearchProtocolContext({
body: bodyToForward,
@ -241,6 +401,7 @@ export class GatewayRequestPipeline {
});
if (records.length > 0) {
hostedWebSearchProtocolContext.records = records;
const webSearchEnrichmentStartedAt = Date.now();
const webSearchContextBody = prepareHostedWebSearchProtocolRequestBody(
bodyToForward,
records,
@ -250,6 +411,18 @@ export class GatewayRequestPipeline {
bodyToForward = webSearchContextBody;
headers["content-type"] = "application/json";
headers["x-ccr-hosted-web-search-context"] = hostedWebSearchProtocolContext.protocol;
routeTrace?.capture({
changes: [
{ operation: "replace", path: "/body/tools", scope: "body" },
{ after: headers["x-ccr-hosted-web-search-context"], operation: "add", path: "/headers/x-ccr-hosted-web-search-context", scope: "headers" }
],
durationMs: Date.now() - webSearchEnrichmentStartedAt,
kind: "mutation",
name: "enrichment.hosted-web-search",
phase: "enrichment",
startedAtMs: webSearchEnrichmentStartedAt,
target: { protocol: hostedWebSearchProtocolContext.protocol }
});
}
}
if (records.length === 0 && !this.browserWebSearchMcpIntegration) {
@ -277,6 +450,7 @@ export class GatewayRequestPipeline {
claudeCodeWebSearchContinuationContext,
this.browserWebSearchMcpIntegration
);
const webSearchContinuationStartedAt = Date.now();
const webSearchContinuationBody = prepareClaudeCodeWebSearchContinuationRequestBody(
bodyToForward,
records,
@ -286,10 +460,30 @@ export class GatewayRequestPipeline {
bodyToForward = webSearchContinuationBody;
headers["content-type"] = "application/json";
headers["x-ccr-claude-code-web-search-continuation"] = records.length > 0 ? "in-app-browser-evidence" : "tool-result-evidence";
routeTrace?.capture({
changes: [
{ operation: "replace", path: "/body/messages", scope: "body" },
{ after: headers["x-ccr-claude-code-web-search-continuation"], operation: "add", path: "/headers/x-ccr-claude-code-web-search-continuation", scope: "headers" }
],
durationMs: Date.now() - webSearchContinuationStartedAt,
kind: "mutation",
name: "enrichment.web-search-continuation",
phase: "enrichment",
startedAtMs: webSearchContinuationStartedAt
});
}
}
const contentLengthHeader = headers["content-length"];
delete headers["content-length"];
if (contentLengthHeader !== undefined) {
routeTrace?.capture({
changes: [{ before: contentLengthHeader, operation: "remove", path: "/headers/content-length", scope: "headers" }],
kind: "mutation",
name: "gateway.content-length-normalization",
phase: "compatibility"
});
}
const upstreamUrl = new URL(request.url || "/", this.status.coreEndpoint).toString();
let upstreamResult: UpstreamFetchResult;
@ -304,6 +498,7 @@ export class GatewayRequestPipeline {
routedModel,
coreAuthToken: this.coreAuthToken,
signal: upstreamAbortController.signal,
trace: routeTrace,
upstreamUrl
});
} catch (error) {
@ -423,7 +618,8 @@ export class GatewayRequestPipeline {
responseHeaders,
sampler.read(),
sampler.isTruncated(),
error ?? streamDetectedError
error ?? streamDetectedError,
sampler.sizeBytes()
);
};
onClientDisconnect = () => {

View file

@ -7,6 +7,10 @@ export function shouldRestartGatewayForRuntimeConfigChange(previousConfig: AppCo
previousConfig.gateway.port !== nextConfig.gateway.port ||
previousConfig.gateway.coreHost !== nextConfig.gateway.coreHost ||
previousConfig.gateway.corePort !== nextConfig.gateway.corePort ||
previousConfig.observability.requestLogs !== nextConfig.observability.requestLogs ||
previousConfig.observability.agentAnalysis !== nextConfig.observability.agentAnalysis ||
previousConfig.observability.requestLogBodyCapture !== nextConfig.observability.requestLogBodyCapture ||
previousConfig.observability.requestLogMaxBodyBytes !== nextConfig.observability.requestLogMaxBodyBytes ||
previousConfig.proxy.enabled !== nextConfig.proxy.enabled ||
previousConfig.proxy.host !== nextConfig.proxy.host ||
previousConfig.proxy.mode !== nextConfig.proxy.mode ||

View file

@ -21,6 +21,7 @@ import { delay } from "@ccr/core/gateway/internal/clock";
import { retryDelayAfterNetworkError, retryDelayAfterStatus, shouldFallbackAfterStatus } from "@ccr/core/gateway/upstream/retry-policy";
import { claudeCodeOauthBetaHeader, claudeCodeOauthRequiredBeta, UpstreamRequestError } from "@ccr/core/gateway/internal/shared";
import type { ApiKeyLimitUsage, ProviderCredentialRoutingTarget, UpstreamAttempt, UpstreamFailedAttempt, UpstreamFetchResult } from "@ccr/core/gateway/internal/shared";
import type { RouteTraceObserver } from "@ccr/core/observability/route-trace";
const providerCredentialSpilloverThreshold = 0.8;
@ -259,6 +260,7 @@ export async function fetchUpstreamWithFallback(input: {
path: string;
routedModel?: string;
signal?: AbortSignal;
trace?: RouteTraceObserver;
upstreamUrl: string;
}): Promise<UpstreamFetchResult> {
const fallbackMode = input.fallback.mode;
@ -271,6 +273,13 @@ export async function fetchUpstreamWithFallback(input: {
input.routedModel
);
const failedAttempts: UpstreamFailedAttempt[] = [];
input.trace?.capture({
decision: { reason: `fallback:${fallbackMode}`, source: "execution-plan" },
kind: "decision",
name: "fallback.execution-plan",
phase: "planning",
target: attempts[0]?.model ? { model: attempts[0].model } : undefined
});
for (let index = 0; index < attempts.length; index += 1) {
if (input.signal?.aborted) {
@ -287,17 +296,73 @@ export async function fetchUpstreamWithFallback(input: {
path: input.path
});
const hasNextAttempt = index < attempts.length - 1;
const attemptNumber = index + 1;
const attemptStartedAt = Date.now();
const attemptUrl = rewriteRouteModelInUrl(input.upstreamUrl, attempt.model);
const attemptHeaders = withCoreGatewayAuthHeader(
omitLocalObservabilityHeaders(attempt.headers ?? input.headers),
input.coreAuthToken
);
const attemptProvider = attempt.logicalProvider ?? (
attempt.target?.kind === "provider" ? attempt.target.provider.name : undefined
);
input.trace?.capture({
attempt: attemptNumber,
changes: [
...(attempt.model && attempt.model !== input.routedModel
? [{
...(input.routedModel === undefined ? {} : { before: input.routedModel }),
after: attempt.model,
operation: input.routedModel === undefined ? "add" as const : "replace" as const,
path: "/body/model",
scope: "body" as const
}]
: []),
...(attemptUrl !== input.upstreamUrl
? [{ after: attemptUrl, before: input.upstreamUrl, operation: "replace" as const, path: "/url", scope: "url" as const }]
: [])
],
kind: "attempt",
name: "upstream.attempt.prepare",
phase: "attempt",
startedAtMs: attemptStartedAt,
target: {
...(attempt.credentialIds?.[0] ? { credentialId: attempt.credentialIds[0] } : {}),
...(attempt.credentialIds?.length ? { credentialCandidates: attempt.credentialIds } : {}),
...(attempt.model ? { model: attempt.model } : {}),
...(attempt.credentialProtocol ? { protocol: attempt.credentialProtocol } : {}),
...(attemptProvider ? { provider: attemptProvider } : {})
}
});
try {
const response = await fetchWithSystemProxy(rewriteRouteModelInUrl(input.upstreamUrl, attempt.model), {
const response = await fetchWithSystemProxy(attemptUrl, {
body: shouldSendBody(input.method) ? attempt.body?.toString("utf8") : undefined,
headers: withCoreGatewayAuthHeader(omitLocalObservabilityHeaders(attempt.headers ?? input.headers), input.coreAuthToken),
headers: attemptHeaders,
method: input.method,
signal: input.signal
});
if (hasNextAttempt && shouldFallbackAfterStatus(response.status, fallbackMode)) {
const delayMs = retryDelayAfterStatus(response.headers, failedAttempts.length);
input.trace?.capture({
attempt: attemptNumber,
durationMs: Date.now() - attemptStartedAt,
kind: "outcome",
name: "upstream.attempt.outcome",
outcome: {
fallbackReason: `http:${response.status}`,
retryDelayMs: delayMs,
statusCode: response.status
},
phase: "outcome",
startedAtMs: attemptStartedAt,
status: "error",
target: {
...(attempt.model ? { model: attempt.model } : {}),
...(attemptProvider ? { provider: attemptProvider } : {})
}
});
failedAttempts.push({
credentialChain: attempt.credentialChain,
credentialIds: attempt.credentialIds,
@ -313,6 +378,21 @@ export async function fetchUpstreamWithFallback(input: {
continue;
}
input.trace?.capture({
attempt: attemptNumber,
durationMs: Date.now() - attemptStartedAt,
kind: "outcome",
name: "upstream.attempt.outcome",
outcome: { statusCode: response.status },
phase: "outcome",
startedAtMs: attemptStartedAt,
status: response.ok ? "ok" : "error",
target: {
...(attempt.model ? { model: attempt.model } : {}),
...(attemptProvider ? { provider: attemptProvider } : {})
}
});
return {
attempt,
failedAttempts,
@ -323,6 +403,23 @@ export async function fetchUpstreamWithFallback(input: {
const delayMs = hasNextAttempt && !input.signal?.aborted
? retryDelayAfterNetworkError(failedAttempts.length)
: 0;
input.trace?.capture({
attempt: attemptNumber,
durationMs: Date.now() - attemptStartedAt,
kind: "outcome",
name: "upstream.attempt.outcome",
outcome: {
error: message,
...(hasNextAttempt ? { fallbackReason: "network-error", retryDelayMs: delayMs } : {})
},
phase: "outcome",
startedAtMs: attemptStartedAt,
status: "error",
target: {
...(attempt.model ? { model: attempt.model } : {}),
...(attemptProvider ? { provider: attemptProvider } : {})
}
});
failedAttempts.push({
credentialChain: attempt.credentialChain,
credentialIds: attempt.credentialIds,

View file

@ -28,10 +28,12 @@ type ModelPrice = {
};
type PriceCatalog = {
index: PriceIndex;
loadedAt: number;
prices: ModelPrice[];
};
type PriceIndex = Map<ModelPricingSource, Map<string, ModelPrice>>;
const catalogTtlMs = 24 * 60 * 60 * 1000;
const fetchTimeoutMs = 5000;
const liteLlmPricesUrl = "https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json";
@ -48,7 +50,7 @@ export async function estimateUsageCostUsd(input: UsageCostInput): Promise<Usage
}
const prices = await getPriceCatalog();
const price = findModelPrice(prices.prices, model, input.provider);
const price = findModelPrice(prices.index, model, input.provider);
if (!price) {
return undefined;
}
@ -96,8 +98,8 @@ async function loadPriceCatalog(): Promise<PriceCatalog> {
]);
const prices = results.flatMap((result) => result.status === "fulfilled" ? result.value : []);
return {
loadedAt: Date.now(),
prices
index: buildPriceIndex(prices),
loadedAt: Date.now()
};
}
@ -247,7 +249,7 @@ function priceFromTokenCosts(input: {
};
}
function findModelPrice(prices: ModelPrice[], model: string, provider: string | undefined): ModelPrice | undefined {
function findModelPrice(index: PriceIndex, model: string, provider: string | undefined): ModelPrice | undefined {
const candidateKeys = modelCandidateKeys(model, provider);
const providerIsOpenRouter = normalizeKey(provider).includes("openrouter");
const sourcePriority: ModelPricingSource[] = providerIsOpenRouter
@ -255,8 +257,10 @@ function findModelPrice(prices: ModelPrice[], model: string, provider: string |
: ["models.dev", "litellm", "openrouter"];
for (const source of sourcePriority) {
const sourceIndex = index.get(source);
if (!sourceIndex) continue;
for (const key of candidateKeys) {
const price = prices.find((item) => item.source === source && priceMatchesKey(item, key));
const price = sourceIndex.get(key);
if (price) {
return price;
}
@ -265,16 +269,31 @@ function findModelPrice(prices: ModelPrice[], model: string, provider: string |
return undefined;
}
function priceMatchesKey(price: ModelPrice, key: string): boolean {
const keys = new Set<string>([
function buildPriceIndex(prices: ModelPrice[]): PriceIndex {
const index: PriceIndex = new Map();
for (const price of prices) {
let sourceIndex = index.get(price.source);
if (!sourceIndex) {
sourceIndex = new Map();
index.set(price.source, sourceIndex);
}
for (const key of priceIndexKeys(price)) {
if (!sourceIndex.has(key)) sourceIndex.set(key, price);
}
}
return index;
}
function priceIndexKeys(price: ModelPrice): string[] {
const keys = [
normalizeKey(price.model),
normalizeKey(lastPathSegment(price.model))
]);
];
if (price.provider) {
keys.add(normalizeKey(`${price.provider}/${price.model}`));
keys.add(normalizeKey(`${price.provider}/${lastPathSegment(price.model)}`));
keys.push(normalizeKey(`${price.provider}/${price.model}`));
keys.push(normalizeKey(`${price.provider}/${lastPathSegment(price.model)}`));
}
return keys.has(key);
return unique(keys.filter(Boolean));
}
function modelCandidateKeys(model: string, provider: string | undefined): string[] {

View file

@ -1,26 +1,39 @@
/**
* Extracted from gateway/service.ts. Keep this module focused on its named gateway boundary.
*/
import { readFileSync, rmSync } from "node:fs";
import { readFile, rm, stat } from "node:fs/promises";
import { randomUUID } from "node:crypto";
import type { IncomingMessage, ServerResponse } from "node:http";
import { dirname, resolve as pathResolve, sep as pathSep } from "node:path";
import type { AppConfig } from "@ccr/core/contracts/app";
import { RAW_TRACE_SPOOL_DIR } from "@ccr/core/config/constants";
import { updateGatewayRequestLogFromRawTrace, type RequestLogRawTraceUpdateInput } from "@ccr/core/observability/request-log-store";
import {
updateGatewayRequestLogFromRawTrace,
type RequestLogRawTraceFiles,
type RequestLogRawTraceUpdateInput
} from "@ccr/core/observability/request-log-store";
import { resolveRawTraceBodyLimit } from "@ccr/core/observability/request-log-limits";
import { isRecord, numberValue, stringValue } from "@ccr/core/gateway/internal/value";
import { formatError, parseJsonObject, readHeader, readRequestBody, sendJson } from "@ccr/core/gateway/http/io";
import { endpoint } from "@ccr/core/gateway/core-runtime/supervisor";
import { maxUsageCaptureBytes, rawTraceSyncHeader, rawTraceSyncPath } from "@ccr/core/gateway/internal/shared";
import type { RawTracePartText } from "@ccr/core/gateway/internal/shared";
type PendingRawTraceUpdate = RequestLogRawTraceUpdateInput & { receivedAt: number };
const maxPendingRawTraceUpdates = 200;
const pendingRawTraceMaxAgeMs = 5 * 60 * 1000;
type RawTraceSynchronizerDependencies = {
enqueueUpdate?: typeof updateGatewayRequestLogFromRawTrace;
getConfig: () => AppConfig | undefined;
spoolDirectory?: string;
};
type RawTraceRequestLogBundle = {
files: RequestLogRawTraceFiles;
update: RequestLogRawTraceUpdateInput;
};
export class RawTraceSynchronizer {
readonly token = randomUUID();
private readonly pendingUpdates = new Map<string, PendingRawTraceUpdate>();
constructor(private readonly dependencies: RawTraceSynchronizerDependencies) {}
async handle(request: IncomingMessage, response: ServerResponse): Promise<void> {
if (request.method !== "POST") {
@ -33,51 +46,47 @@ export class RawTraceSynchronizer {
}
const manifest = parseJsonObject(await readRequestBody(request));
const update = readRawTraceRequestLogUpdate(manifest);
cleanupRawTraceBundle(manifest);
if (!update) {
const spoolDirectory = this.dependencies.spoolDirectory ?? RAW_TRACE_SPOOL_DIR;
const bundle = await readRawTraceRequestLogBundle(manifest, spoolDirectory);
if (!bundle) {
await cleanupRawTraceBundle(manifest, spoolDirectory);
sendJson(response, 202, { applied: false, ok: true });
return;
}
const applied = await updateGatewayRequestLogFromRawTrace(update);
if (!applied) this.store(update);
sendJson(response, 200, { applied, ok: true });
}
take(requestId: string): RequestLogRawTraceUpdateInput | undefined {
const update = this.pendingUpdates.get(requestId);
if (!update) return undefined;
this.pendingUpdates.delete(requestId);
const { receivedAt: _receivedAt, ...input } = update;
return input;
}
private store(update: RequestLogRawTraceUpdateInput): void {
this.prune();
this.pendingUpdates.set(update.requestId, { ...update, receivedAt: Date.now() });
while (this.pendingUpdates.size > maxPendingRawTraceUpdates) {
const oldestKey = this.pendingUpdates.keys().next().value;
if (!oldestKey) break;
this.pendingUpdates.delete(oldestKey);
const config = this.dependencies.getConfig();
if (!config || !shouldRecordRequestLogs(config)) {
await cleanupRawTraceBundle(manifest, spoolDirectory);
sendJson(response, 202, { accepted: true, ok: true, reason: "disabled" });
return;
}
}
private prune(): void {
const cutoff = Date.now() - pendingRawTraceMaxAgeMs;
for (const [requestId, update] of this.pendingUpdates) {
if (update.receivedAt < cutoff) this.pendingUpdates.delete(requestId);
const policy = applyRawTraceRequestLogPolicy(config, bundle.update);
if (policy.action === "discard") {
await cleanupRawTraceBundle(manifest, spoolDirectory);
sendJson(response, 202, { accepted: true, ok: true, reason: policy.reason });
return;
}
const maxBodyBytes = resolveRawTraceBodyLimit(config.observability.requestLogMaxBodyBytes);
const files = policy.captureBodies
? { ...bundle.files, maxBodyBytes }
: { cleanupDirectory: bundle.files.cleanupDirectory, maxBodyBytes };
const enqueueUpdate = this.dependencies.enqueueUpdate ?? updateGatewayRequestLogFromRawTrace;
const accepted = await enqueueUpdate(policy.update, files);
sendJson(response, accepted ? 200 : 503, { accepted, ok: accepted });
}
}
export function buildRawTraceConfig(config: AppConfig, rawTraceSyncToken: string): Record<string, unknown> {
const enabled = rawTraceEnabledFromEnv() && shouldRecordRequestLogs(config);
const bodyCapture = config.observability.requestLogBodyCapture ?? "all";
const maxBodyBytes = resolveRawTraceBodyLimit(config.observability.requestLogMaxBodyBytes);
const bodyCaptureEnabled = bodyCapture !== "none" && maxBodyBytes >= 1024;
const enabled = rawTraceEnabledFromEnv() && shouldRecordRequestLogs(config) && bodyCaptureEnabled;
return {
deleteLocalAfterUpload: false,
enabled,
maxPartBytes: maxUsageCaptureBytes,
maxPartBytes: maxBodyBytes,
mode: "wire_raw",
spoolDir: RAW_TRACE_SPOOL_DIR,
sync: {
@ -97,13 +106,82 @@ export function shouldRecordRequestLogs(config: AppConfig): boolean {
}
export type RawTraceRequestLogPolicy = {
action: "discard";
reason: "sampled";
} | {
action: "enqueue";
captureBodies: boolean;
update: RequestLogRawTraceUpdateInput;
};
export function applyRawTraceRequestLogPolicy(
config: AppConfig,
input: RequestLogRawTraceUpdateInput
): RawTraceRequestLogPolicy {
const successful = input.statusCode === undefined || (input.statusCode >= 200 && input.statusCode < 400);
if (successful && !requestLogSampled(
input.requestId,
config.observability.requestLogSuccessSampleRate ?? 1
)) {
return { action: "discard", reason: "sampled" };
}
const bodyCapture = config.observability.requestLogBodyCapture ?? "all";
const captureBodies = resolveRawTraceBodyLimit(config.observability.requestLogMaxBodyBytes) > 0 &&
(bodyCapture === "all" || (bodyCapture === "errors" && !successful));
return {
action: "enqueue",
captureBodies,
update: captureBodies ? input : suppressRawTraceBodies(input)
};
}
export function requestLogSampled(requestId: string, rate: number): boolean {
if (rate >= 1) return true;
if (rate <= 0) return false;
let hash = 2166136261;
for (let index = 0; index < requestId.length; index += 1) {
hash ^= requestId.charCodeAt(index);
hash = Math.imul(hash, 16777619);
}
return (hash >>> 0) / 0x1_0000_0000 < rate;
}
function suppressRawTraceBodies(input: RequestLogRawTraceUpdateInput): RequestLogRawTraceUpdateInput {
const requestSize = input.requestBodySizeBytes ??
(input.requestBodyText === undefined ? undefined : Buffer.byteLength(input.requestBodyText));
const responseSize = input.responseBodySizeBytes ??
(input.responseBodyText === undefined ? undefined : Buffer.byteLength(input.responseBodyText));
return {
...input,
...(requestSize === undefined ? {} : {
requestBodySizeBytes: requestSize,
requestBodyText: "",
requestBodyTruncated: Boolean(input.requestBodyTruncated) || requestSize > 0
}),
...(responseSize === undefined ? {} : {
responseBodySizeBytes: responseSize,
responseBodyText: "",
responseBodyTruncated: Boolean(input.responseBodyTruncated) || responseSize > 0
})
};
}
function rawTraceEnabledFromEnv(): boolean {
const value = (process.env.CCR_RAW_TRACE_ENABLED ?? process.env.CCR_RAW_TRACE ?? "").trim().toLowerCase();
return value === "1" || value === "true" || value === "yes" || value === "on";
}
export function readRawTraceRequestLogUpdate(manifest: Record<string, unknown>): RequestLogRawTraceUpdateInput | undefined {
export async function readRawTraceRequestLogBundle(
manifest: Record<string, unknown>,
spoolDirectory = RAW_TRACE_SPOOL_DIR
): Promise<RawTraceRequestLogBundle | undefined> {
const requestId = stringValue(manifest.turnKey);
const parts = Array.isArray(manifest.parts)
? manifest.parts.filter((part): part is Record<string, unknown> => isRecord(part))
@ -112,40 +190,63 @@ export function readRawTraceRequestLogUpdate(manifest: Record<string, unknown>):
return undefined;
}
const upstreamRequestMetadata = readRawTraceJsonPart(parts, "upstream_request_metadata");
const upstreamResponseMetadata = readRawTraceJsonPart(parts, "upstream_response_metadata");
const upstreamRequestBody = readRawTraceTextPart(parts, "upstream_request");
const upstreamResponseStream = readRawTraceTextPart(parts, "response_stream");
const upstreamResponseBody = upstreamResponseStream ?? readRawTraceTextPart(parts, "upstream_response");
const [
upstreamRequestMetadata,
upstreamResponseMetadata,
upstreamRequestBody,
upstreamResponseStream,
fallbackResponseBody
] = await Promise.all([
readRawTraceJsonPart(parts, "upstream_request_metadata", spoolDirectory),
readRawTraceJsonPart(parts, "upstream_response_metadata", spoolDirectory),
readRawTracePart(parts, "upstream_request", spoolDirectory),
readRawTracePart(parts, "response_stream", spoolDirectory),
readRawTracePart(parts, "upstream_response", spoolDirectory)
]);
const upstreamResponseBody = upstreamResponseStream ?? fallbackResponseBody;
const target = isRecord(manifest.target) ? manifest.target : {};
const rawUrl = stringValue(upstreamRequestMetadata?.url);
const url = sanitizeUrlForLog(rawUrl);
return {
method: stringValue(upstreamRequestMetadata?.method) || "POST",
model: stringValue(target.model),
path: pathFromUrl(url),
provider: stringValue(target.providerName) || stringValue(target.provider),
requestBodyContentType: upstreamRequestBody?.contentType,
requestBodyText: upstreamRequestBody?.text,
requestHeaders: headerRecordFromUnknown(upstreamRequestMetadata?.headers),
requestId,
isStream: upstreamResponseStream !== undefined,
responseBodyContentType: upstreamResponseBody?.contentType,
responseBodyText: upstreamResponseBody?.text,
responseHeaders: headerRecordFromUnknown(upstreamResponseMetadata?.headers),
statusCode: numberValue(upstreamResponseMetadata?.statusCode),
url
files: {
cleanupDirectory: rawTraceBundleDirectory(parts, spoolDirectory),
requestBody: upstreamRequestBody,
responseBody: upstreamResponseBody
},
update: {
method: stringValue(upstreamRequestMetadata?.method) || "POST",
model: stringValue(target.model),
path: pathFromUrl(url),
provider: stringValue(target.providerName) || stringValue(target.provider),
requestBodyContentType: upstreamRequestBody?.contentType,
requestBodySizeBytes: upstreamRequestBody?.sizeBytes,
requestBodyTruncated: upstreamRequestBody?.truncated,
requestHeaders: headerRecordFromUnknown(upstreamRequestMetadata?.headers),
requestId,
isStream: upstreamResponseStream !== undefined,
responseBodyContentType: upstreamResponseBody?.contentType,
responseBodySizeBytes: upstreamResponseBody?.sizeBytes,
responseBodyTruncated: upstreamResponseBody?.truncated,
responseHeaders: headerRecordFromUnknown(upstreamResponseMetadata?.headers),
statusCode: numberValue(upstreamResponseMetadata?.statusCode),
url
}
};
}
function readRawTraceJsonPart(parts: Record<string, unknown>[], partType: string): Record<string, unknown> | undefined {
const text = readRawTraceTextPart(parts, partType)?.text;
if (!text) {
async function readRawTraceJsonPart(
parts: Record<string, unknown>[],
partType: string,
spoolDirectory: string
): Promise<Record<string, unknown> | undefined> {
const part = await readRawTracePart(parts, partType, spoolDirectory);
if (!part) {
return undefined;
}
try {
const text = await readFile(part.filePath, "utf8");
const parsed = JSON.parse(text) as unknown;
return isRecord(parsed) ? parsed : undefined;
} catch {
@ -154,16 +255,23 @@ function readRawTraceJsonPart(parts: Record<string, unknown>[], partType: string
}
function readRawTraceTextPart(parts: Record<string, unknown>[], partType: string): RawTracePartText | undefined {
async function readRawTracePart(
parts: Record<string, unknown>[],
partType: string,
spoolDirectory: string
): Promise<RawTracePartText | undefined> {
const part = parts.find((candidate) => stringValue(candidate.partType) === partType);
const filePath = stringValue(part?.filePath);
if (!filePath || !isRawTraceSpoolFile(filePath)) {
if (!filePath || !isRawTraceSpoolFile(filePath, spoolDirectory)) {
return undefined;
}
try {
const storedBytes = (await stat(filePath)).size;
return {
contentType: stringValue(part?.contentType),
text: readFileSync(filePath, "utf8")
filePath,
sizeBytes: Math.max(storedBytes, numberValue(part?.originalBytes) ?? 0),
truncated: storedBytes < (numberValue(part?.originalBytes) ?? storedBytes)
};
} catch (error) {
console.warn(`[gateway] Failed to read raw trace part ${partType}: ${formatError(error)}`);
@ -172,24 +280,33 @@ function readRawTraceTextPart(parts: Record<string, unknown>[], partType: string
}
export function cleanupRawTraceBundle(manifest: Record<string, unknown>): void {
function rawTraceBundleDirectory(parts: Record<string, unknown>[], spoolDirectory: string): string | undefined {
const filePath = parts.map((part) => stringValue(part.filePath)).find((value): value is string => Boolean(value));
return filePath && isRawTraceSpoolFile(filePath, spoolDirectory) ? dirname(filePath) : undefined;
}
export async function cleanupRawTraceBundle(
manifest: Record<string, unknown>,
spoolDirectory = RAW_TRACE_SPOOL_DIR
): Promise<void> {
const parts = Array.isArray(manifest.parts)
? manifest.parts.filter((part): part is Record<string, unknown> => isRecord(part))
: [];
const firstFilePath = parts.map((part) => stringValue(part.filePath)).find((value): value is string => Boolean(value));
if (!firstFilePath || !isRawTraceSpoolFile(firstFilePath)) {
if (!firstFilePath || !isRawTraceSpoolFile(firstFilePath, spoolDirectory)) {
return;
}
try {
rmSync(dirname(firstFilePath), { force: true, recursive: true });
await rm(dirname(firstFilePath), { force: true, recursive: true });
} catch (error) {
console.warn(`[gateway] Failed to clean raw trace bundle: ${formatError(error)}`);
}
}
function isRawTraceSpoolFile(filePath: string): boolean {
const spoolDir = pathResolve(RAW_TRACE_SPOOL_DIR);
function isRawTraceSpoolFile(filePath: string, spoolDirectory: string): boolean {
const spoolDir = pathResolve(spoolDirectory);
const resolvedFile = pathResolve(filePath);
return dirname(resolvedFile) !== spoolDir && resolvedFile.startsWith(`${spoolDir}${pathSep}`);
}
@ -250,32 +367,35 @@ function pathFromUrl(value: string | undefined): string | undefined {
export function createBodySampler() {
const chunks: Buffer[] = [];
let capturedBytes = 0;
let totalBytes = 0;
let truncated = false;
return {
append(chunk: Buffer | string) {
if (truncated) {
return;
}
const buffer = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk);
if (totalBytes + buffer.byteLength > maxUsageCaptureBytes) {
const remaining = Math.max(0, maxUsageCaptureBytes - totalBytes);
totalBytes += buffer.byteLength;
if (truncated) return;
if (capturedBytes + buffer.byteLength > maxUsageCaptureBytes) {
const remaining = Math.max(0, maxUsageCaptureBytes - capturedBytes);
if (remaining > 0) {
chunks.push(buffer.subarray(0, remaining));
totalBytes += remaining;
capturedBytes += remaining;
}
truncated = true;
return;
}
chunks.push(buffer);
totalBytes += buffer.byteLength;
capturedBytes += buffer.byteLength;
},
isTruncated() {
return truncated;
},
read() {
return Buffer.concat(chunks, totalBytes).toString("utf8");
return Buffer.concat(chunks, capturedBytes).toString("utf8");
},
sizeBytes() {
return totalBytes;
}
};
}

View file

@ -0,0 +1,223 @@
export type RequestLogBodyCompaction = {
buffer: Buffer;
compacted: boolean;
imageCount: number;
omittedBytes: number;
};
const base64ImageCompactionThresholdBytes = 16 * 1024;
const base64Marker = Buffer.from(";base64,");
const dataImagePrefix = Buffer.from("data:image/");
const genericDataKey = Buffer.from('"data"');
const base64EncodingMarker = Buffer.from('"base64"');
const inlineDataParentKeys = [Buffer.from("inline_data"), Buffer.from("inlineData")];
const imageDataKeys = [
Buffer.from('"b64_json"'),
Buffer.from('"image_base64"'),
Buffer.from('"imageBase64"')
];
const imageMimeMarker = Buffer.from("image/");
type Base64Range = {
end: number;
start: number;
};
/**
* Replaces large inline Base64 image payloads with small JSON-safe descriptors.
* The scan works on bytes instead of JSON.parse/stringify so a large image does
* not create another full JavaScript string/object graph on the request path.
*/
export function compactBase64ImagePayloads(input: Buffer): RequestLogBodyCompaction {
const ranges = findBase64ImageRanges(input);
if (ranges.length === 0) {
return { buffer: input, compacted: false, imageCount: 0, omittedBytes: 0 };
}
const chunks: Buffer[] = [];
let cursor = 0;
let outputBytes = 0;
let omittedBytes = 0;
for (const range of ranges) {
const prefix = input.subarray(cursor, range.start);
const encodedBytes = range.end - range.start;
const decodedBytes = approximateDecodedBytes(input, range);
const replacement = Buffer.from(
`[base64 image omitted from log; encoded_bytes=${encodedBytes}; decoded_bytes~=${decodedBytes}]`
);
chunks.push(prefix, replacement);
outputBytes += prefix.byteLength + replacement.byteLength;
omittedBytes += encodedBytes;
cursor = range.end;
}
const suffix = input.subarray(cursor);
chunks.push(suffix);
outputBytes += suffix.byteLength;
return {
buffer: Buffer.concat(chunks, outputBytes),
compacted: true,
imageCount: ranges.length,
omittedBytes
};
}
function findBase64ImageRanges(input: Buffer): Base64Range[] {
const candidates = [
...findDataImageUrlRanges(input),
...imageDataKeys.flatMap((key) => findJsonStringValueRanges(input, key, false)),
...findJsonStringValueRanges(input, genericDataKey, true)
].sort((left, right) => left.start - right.start || left.end - right.end);
const ranges: Base64Range[] = [];
for (const candidate of candidates) {
const previous = ranges.at(-1);
if (previous && candidate.start < previous.end) continue;
ranges.push(candidate);
}
return ranges;
}
function findDataImageUrlRanges(input: Buffer): Base64Range[] {
const ranges: Base64Range[] = [];
let cursor = 0;
while (cursor < input.byteLength) {
const prefix = input.indexOf(dataImagePrefix, cursor);
if (prefix < 0) break;
const marker = input.indexOf(base64Marker, prefix + dataImagePrefix.byteLength);
if (marker < 0 || marker - prefix > 192) {
cursor = prefix + dataImagePrefix.byteLength;
continue;
}
const start = marker + base64Marker.byteLength;
const end = scanBase64End(input, start);
if (end - start >= base64ImageCompactionThresholdBytes) ranges.push({ end, start });
cursor = Math.max(end, start + 1);
}
return ranges;
}
function findJsonStringValueRanges(input: Buffer, key: Buffer, requireImageContext: boolean): Base64Range[] {
const ranges: Base64Range[] = [];
let cursor = 0;
while (cursor < input.byteLength) {
const keyIndex = input.indexOf(key, cursor);
if (keyIndex < 0) break;
cursor = keyIndex + key.byteLength;
const start = jsonStringValueStart(input, cursor);
if (start === undefined) continue;
const end = scanBase64End(input, start);
if (requireImageContext && !hasImageContext(input, keyIndex, end)) continue;
if (end - start >= base64ImageCompactionThresholdBytes) ranges.push({ end, start });
cursor = Math.max(cursor, end);
}
return ranges;
}
function hasImageContext(input: Buffer, keyIndex: number, valueEnd: number): boolean {
const objectStart = input.lastIndexOf(0x7b, keyIndex);
if (objectStart < 0) return false;
const objectEnd = findContainingObjectEnd(input, valueEnd);
const hasMime = containsBefore(input, imageMimeMarker, objectStart, keyIndex) ||
containsBefore(input, imageMimeMarker, valueEnd, objectEnd);
const hasImageEncoding = containsBefore(input, base64EncodingMarker, objectStart, keyIndex) ||
containsBefore(input, base64EncodingMarker, valueEnd, objectEnd) ||
objectParentKeyMatches(input, objectStart, inlineDataParentKeys);
return hasMime && hasImageEncoding;
}
function findContainingObjectEnd(input: Buffer, valueEnd: number): number {
let depth = 1;
let inString = false;
let escaped = false;
for (let cursor = Math.min(input.byteLength, valueEnd + 1); cursor < input.byteLength; cursor += 1) {
const byte = input[cursor];
if (inString) {
if (escaped) escaped = false;
else if (byte === 0x5c) escaped = true;
else if (byte === 0x22) inString = false;
continue;
}
if (byte === 0x22) inString = true;
else if (byte === 0x7b) depth += 1;
else if (byte === 0x7d && --depth === 0) return cursor + 1;
}
return input.byteLength;
}
function objectParentKeyMatches(input: Buffer, objectStart: number, keys: Buffer[]): boolean {
let cursor = skipWhitespaceBackward(input, objectStart - 1);
if (input[cursor] !== 0x3a) return false;
cursor = skipWhitespaceBackward(input, cursor - 1);
if (input[cursor] !== 0x22) return false;
const keyEnd = cursor;
cursor -= 1;
while (cursor >= 0 && input[cursor] !== 0x22) cursor -= 1;
if (cursor < 0) return false;
const key = input.subarray(cursor + 1, keyEnd);
return keys.some((candidate) => key.equals(candidate));
}
function containsBefore(input: Buffer, marker: Buffer, start: number, end: number): boolean {
const index = input.indexOf(marker, start);
return index >= 0 && index < end;
}
function jsonStringValueStart(input: Buffer, start: number): number | undefined {
let cursor = skipWhitespace(input, start);
if (input[cursor] !== 0x3a) return undefined;
cursor = skipWhitespace(input, cursor + 1);
return input[cursor] === 0x22 ? cursor + 1 : undefined;
}
function skipWhitespace(input: Buffer, start: number): number {
let cursor = start;
while (cursor < input.byteLength) {
const byte = input[cursor];
if (byte !== 0x20 && byte !== 0x09 && byte !== 0x0a && byte !== 0x0d) break;
cursor += 1;
}
return cursor;
}
function skipWhitespaceBackward(input: Buffer, start: number): number {
let cursor = start;
while (cursor >= 0) {
const byte = input[cursor];
if (byte !== 0x20 && byte !== 0x09 && byte !== 0x0a && byte !== 0x0d) break;
cursor -= 1;
}
return cursor;
}
function scanBase64End(input: Buffer, start: number): number {
const closingQuote = input.indexOf(0x22, start);
const end = closingQuote >= 0 ? closingQuote : input.byteLength;
if (end <= start || !isProbablyBase64Range(input, start, end)) return start;
return end;
}
function isProbablyBase64Range(input: Buffer, start: number, end: number): boolean {
const sampleBytes = 64;
const firstEnd = Math.min(end, start + sampleBytes);
for (let index = start; index < firstEnd; index += 1) {
if (!isBase64Byte(input[index])) return false;
}
const lastStart = Math.max(firstEnd, end - sampleBytes);
for (let index = lastStart; index < end; index += 1) {
if (!isBase64Byte(input[index])) return false;
}
return true;
}
function isBase64Byte(byte: number): boolean {
return (byte >= 0x41 && byte <= 0x5a) ||
(byte >= 0x61 && byte <= 0x7a) ||
(byte >= 0x30 && byte <= 0x39) ||
byte === 0x2b || byte === 0x2f || byte === 0x3d || byte === 0x2d || byte === 0x5f;
}
function approximateDecodedBytes(input: Buffer, range: Base64Range): number {
let padding = 0;
if (input[range.end - 1] === 0x3d) padding += 1;
if (input[range.end - 2] === 0x3d) padding += 1;
return Math.max(0, Math.floor((range.end - range.start) * 3 / 4) - padding);
}

View file

@ -0,0 +1,12 @@
// Raw traces are produced outside the request-log runtime, so keep an
// independent hard ceiling as defense in depth if configuration validation is
// bypassed or its public range grows in the future.
export const rawTraceHardMaxBodyBytes = 50 * 1024 * 1024;
export const maxRequestLogBodyBytes = rawTraceHardMaxBodyBytes;
export const defaultRequestLogBodyBytes = rawTraceHardMaxBodyBytes;
export function resolveRawTraceBodyLimit(value: number | undefined): number {
const configured = value ?? defaultRequestLogBodyBytes;
if (!Number.isFinite(configured)) return defaultRequestLogBodyBytes;
return Math.max(0, Math.min(rawTraceHardMaxBodyBytes, Math.floor(configured)));
}

View file

@ -0,0 +1,680 @@
import { randomUUID } from "node:crypto";
import { realpath, rm } from "node:fs/promises";
import path from "node:path";
import { StringDecoder } from "node:string_decoder";
import { Worker } from "node:worker_threads";
import { RAW_TRACE_SPOOL_DIR } from "@ccr/core/config/constants";
import type {
AgentAnalysisFilter,
AgentAnalysisSnapshot,
AgentAnalysisTracePayloadFullResult,
AgentAnalysisTracePayloadRequest,
RequestLogDetailRequest,
RequestLogEntry,
RequestLogListFilter,
RequestLogPage
} from "@ccr/core/contracts/app";
import type {
RequestLogRawTraceFiles,
RequestLogRawTraceUpdateInput,
RequestLogRecordInput,
RequestLogStoreWriteCommand
} from "@ccr/core/observability/request-log-store";
import {
defaultRequestLogBodyBytes,
maxRequestLogBodyBytes,
resolveRawTraceBodyLimit
} from "@ccr/core/observability/request-log-limits";
import { compactBase64ImagePayloads } from "@ccr/core/observability/request-log-body";
export type RequestLogEnqueueResult = {
accepted: boolean;
degraded: boolean;
reason?: "body_removed" | "closed" | "queue_full" | "writer_unavailable";
};
export type RequestLogRuntimeMetrics = {
accepted: number;
committed: number;
degraded: number;
dropped: number;
inFlightItems: number;
queueBytes: number;
queueItems: number;
writerRestarts: number;
};
export type RequestLogRuntimeOptions = {
batchMaxBytes?: number;
batchMaxItems?: number;
batchMaxWaitMs?: number;
dbFile: string;
queueMaxBytes?: number;
queueMaxItems?: number;
rawTraceSpoolDir?: string;
workerFile?: string;
};
type ResolvedRuntimeOptions = Required<Omit<RequestLogRuntimeOptions, "rawTraceSpoolDir" | "workerFile">> & {
rawTraceSpoolDir: string;
workerFile: string;
};
type QueuedCommand = RequestLogStoreWriteCommand & {
batchBytes: number;
sizeBytes: number;
};
type InFlightBatch = {
bytes: number;
commands: QueuedCommand[];
};
type WorkerResponse = {
batchId?: number;
error?: string;
requestId?: number;
result?: unknown;
type: "ack" | "batch-error" | "ready" | "response";
};
type PendingRpc = {
reject: (error: Error) => void;
resolve: (value: unknown) => void;
};
// One raw-trace event can contain both a maximum-sized request and response.
// Keep room for exactly that case while retaining byte-based backpressure for
// concurrent events.
const defaultQueueMaxBytes = 128 * 1024 * 1024;
const sensitiveQueueHeaderNames = new Set([
"authorization",
"cookie",
"proxy-authorization",
"set-cookie",
"x-api-key",
"x-auth-api-key-id",
"x-auth-sub"
]);
export class RequestLogRuntime {
private accepted = 0;
private batchId = 0;
private closed = false;
private committed = 0;
private readonly cleanupTasks = new Set<Promise<void>>();
private degraded = 0;
private dropped = 0;
private flushTimer?: NodeJS.Timeout;
private readonly inFlight = new Map<number, InFlightBatch>();
private nextRequestId = 0;
private nextSequence = 0;
private readonly options: ResolvedRuntimeOptions;
private outstandingBytes = 0;
private readonly queryRequests = new Map<number, PendingRpc>();
private queryWorker?: Worker;
private queryWorkerReady?: Promise<void>;
private readonly queue: QueuedCommand[] = [];
private readonly writerRequests = new Map<number, PendingRpc>();
private writerRestartCount = 0;
private writerWorker?: Worker;
private writerWorkerReady?: Promise<void>;
constructor(options: RequestLogRuntimeOptions) {
this.options = {
batchMaxBytes: positiveInteger(options.batchMaxBytes, 4 * 1024 * 1024),
batchMaxItems: positiveInteger(options.batchMaxItems, 50),
batchMaxWaitMs: positiveInteger(options.batchMaxWaitMs, 10),
dbFile: options.dbFile,
queueMaxBytes: positiveInteger(options.queueMaxBytes, defaultQueueMaxBytes),
queueMaxItems: positiveInteger(options.queueMaxItems, 2_000),
rawTraceSpoolDir: options.rawTraceSpoolDir ?? RAW_TRACE_SPOOL_DIR,
workerFile: options.workerFile ?? path.join(__dirname, "request-log-worker.js")
};
}
enqueueRecord(input: RequestLogRecordInput): RequestLogEnqueueResult {
const pressure = this.pressureRatio();
const ordinarySuccess = input.statusCode >= 200 && input.statusCode < 400 && !input.error;
if (pressure >= 0.95 && ordinarySuccess) {
this.dropped += 1;
return { accepted: false, degraded: false, reason: "queue_full" };
}
const prepared = prepareRecordForQueue(input, pressure);
const sizeBytes = estimateRecordBytes(prepared.input);
const command: QueuedCommand = {
eventId: randomUUID(),
input: prepared.input,
kind: "record",
sequence: ++this.nextSequence,
batchBytes: sizeBytes,
sizeBytes
};
const result = this.enqueue(command, prepared.degraded ? "body_removed" : undefined);
if (result.accepted && prepared.degraded) this.degraded += 1;
return result;
}
enqueueRawTrace(
input: RequestLogRawTraceUpdateInput,
rawTraceFiles?: RequestLogRawTraceFiles
): RequestLogEnqueueResult {
const maxBodyBytes = resolveRawTraceBodyLimit(rawTraceFiles?.maxBodyBytes);
const prepared = prepareRawTraceForQueue(input, maxBodyBytes);
const sizeBytes = Math.max(
estimateRawTraceBytes(prepared, rawTraceFiles),
rawTraceFileBytes(rawTraceFiles, maxBodyBytes)
);
const command: QueuedCommand = {
input: prepared,
kind: "raw-trace-update",
rawTraceFiles,
sequence: ++this.nextSequence,
batchBytes: sizeBytes,
sizeBytes
};
return this.enqueue(command);
}
async list(filter?: RequestLogListFilter): Promise<RequestLogPage> {
return await this.query<RequestLogPage>("list", [filter]);
}
async getDetail(request: RequestLogDetailRequest): Promise<RequestLogEntry | undefined> {
return await this.query<RequestLogEntry | undefined>("getDetail", [request]);
}
async analyze(filter?: AgentAnalysisFilter): Promise<AgentAnalysisSnapshot> {
return await this.query<AgentAnalysisSnapshot>("analyze", [filter]);
}
async getTracePayload(request: AgentAnalysisTracePayloadRequest): Promise<AgentAnalysisTracePayloadFullResult> {
return await this.query<AgentAnalysisTracePayloadFullResult>("getTracePayload", [request]);
}
metrics(): RequestLogRuntimeMetrics {
return {
accepted: this.accepted,
committed: this.committed,
degraded: this.degraded,
dropped: this.dropped,
inFlightItems: [...this.inFlight.values()].reduce((total, batch) => total + batch.commands.length, 0),
queueBytes: this.outstandingBytes,
queueItems: this.queue.length,
writerRestarts: this.writerRestartCount
};
}
async flush(options: { timeoutMs: number }): Promise<{ pending: number; timedOut: boolean }> {
if (!this.writerWorker && this.queue.length === 0 && this.cleanupTasks.size === 0) {
return { pending: 0, timedOut: false };
}
this.schedulePump(true);
const drained = await waitUntil(
() => this.queue.length === 0 && this.inFlight.size === 0 && this.cleanupTasks.size === 0,
options.timeoutMs
);
if (drained && this.writerWorker) {
await withTimeout(this.writerRpc("flush", []), Math.max(1, options.timeoutMs));
}
return {
pending: this.queue.length + [...this.inFlight.values()].reduce((total, batch) => total + batch.commands.length, 0),
timedOut: !drained
};
}
async close(options: { timeoutMs: number }): Promise<void> {
if (this.closed) return;
this.closed = true;
if (this.flushTimer) clearTimeout(this.flushTimer);
await this.flush(options).catch(() => undefined);
await Promise.all([
this.shutdownWorker("query", this.queryWorker, this.queryRequests, options.timeoutMs),
this.shutdownWorker("writer", this.writerWorker, this.writerRequests, options.timeoutMs)
]);
this.queryWorker = undefined;
this.writerWorker = undefined;
}
private enqueue(command: QueuedCommand, degradedReason?: RequestLogEnqueueResult["reason"]): RequestLogEnqueueResult {
if (this.closed) {
this.dropped += 1;
return { accepted: false, degraded: false, reason: "closed" };
}
if (command.sizeBytes > this.options.queueMaxBytes ||
this.outstandingBytes + command.sizeBytes > this.options.queueMaxBytes ||
this.queue.length + this.inFlightItemCount() >= this.options.queueMaxItems) {
this.dropped += 1;
return { accepted: false, degraded: false, reason: "queue_full" };
}
this.accepted += 1;
this.outstandingBytes += command.sizeBytes;
this.queue.push(command);
this.ensureWriter().catch(() => undefined);
this.schedulePump(this.queue.length >= this.options.batchMaxItems || this.queuedBytes() >= this.options.batchMaxBytes);
return {
accepted: true,
degraded: degradedReason !== undefined,
...(degradedReason ? { reason: degradedReason } : {})
};
}
private pressureRatio(): number {
return Math.max(
this.outstandingBytes / this.options.queueMaxBytes,
(this.queue.length + this.inFlightItemCount()) / this.options.queueMaxItems
);
}
private inFlightItemCount(): number {
return [...this.inFlight.values()].reduce((total, batch) => total + batch.commands.length, 0);
}
private queuedBytes(): number {
return this.queue.reduce((total, command) => total + command.batchBytes, 0);
}
private schedulePump(immediate = false): void {
if (this.inFlight.size > 0 || this.queue.length === 0) return;
if (immediate) {
if (this.flushTimer) clearTimeout(this.flushTimer);
this.flushTimer = undefined;
queueMicrotask(() => void this.pump());
return;
}
this.flushTimer ??= setTimeout(() => {
this.flushTimer = undefined;
void this.pump();
}, this.options.batchMaxWaitMs);
this.flushTimer.unref?.();
}
private async pump(): Promise<void> {
if (this.inFlight.size > 0 || this.queue.length === 0) return;
try {
await this.ensureWriter();
} catch {
return;
}
const commands: QueuedCommand[] = [];
let bytes = 0;
while (this.queue.length > 0 && commands.length < this.options.batchMaxItems) {
const next = this.queue[0];
if (commands.length > 0 && bytes + next.batchBytes > this.options.batchMaxBytes) break;
commands.push(this.queue.shift()!);
bytes += next.batchBytes;
}
if (commands.length === 0 || !this.writerWorker) return;
const batchId = ++this.batchId;
this.inFlight.set(batchId, {
bytes: commands.reduce((total, command) => total + command.sizeBytes, 0),
commands
});
this.writerWorker.postMessage({ batchId, commands: commands.map(withoutSize), type: "batch" });
}
private ensureWriter(): Promise<void> {
if (this.writerWorkerReady) return this.writerWorkerReady;
this.writerWorkerReady = new Promise<void>((resolve, reject) => {
const worker = new Worker(this.options.workerFile, {
workerData: {
dbFile: this.options.dbFile,
mode: "writer",
rawTraceSpoolDir: this.options.rawTraceSpoolDir
}
});
this.writerWorker = worker;
worker.unref();
const onStartupError = (error: Error) => reject(error);
worker.once("error", onStartupError);
worker.on("message", (message: WorkerResponse) => {
if (message.type === "ready") {
worker.off("error", onStartupError);
resolve();
this.schedulePump(true);
return;
}
this.handleWriterMessage(message);
});
worker.on("error", (error) => this.handleWriterFailure(worker, error));
worker.on("exit", (code) => {
if (!this.closed && code !== 0) this.handleWriterFailure(worker, new Error(`request log writer exited with ${code}`));
});
});
return this.writerWorkerReady;
}
private handleWriterMessage(message: WorkerResponse): void {
if (message.type === "ack" && message.batchId !== undefined) {
const batch = this.inFlight.get(message.batchId);
if (!batch) return;
this.inFlight.delete(message.batchId);
this.outstandingBytes = Math.max(0, this.outstandingBytes - batch.bytes);
this.committed += batch.commands.length;
this.scheduleRawTraceCleanup(batch.commands);
this.schedulePump(true);
return;
}
if (message.type === "batch-error") {
this.handleWriterFailure(this.writerWorker, new Error(message.error || "request log batch failed"));
return;
}
settleRpc(this.writerRequests, message);
}
private handleWriterFailure(worker: Worker | undefined, error: Error): void {
if (!worker || worker !== this.writerWorker) return;
this.writerWorker = undefined;
this.writerWorkerReady = undefined;
void worker.terminate().catch(() => undefined);
for (const [, batch] of [...this.inFlight].reverse()) this.queue.unshift(...batch.commands);
this.inFlight.clear();
rejectPending(this.writerRequests, error);
if (this.closed) return;
this.writerRestartCount += 1;
const delayMs = Math.min(5_000, 100 * (2 ** Math.min(6, this.writerRestartCount - 1)));
const timer = setTimeout(() => {
this.ensureWriter().then(() => this.schedulePump(true)).catch(() => undefined);
}, delayMs);
timer.unref?.();
}
private scheduleRawTraceCleanup(commands: QueuedCommand[]): void {
const directories = new Set(commands.flatMap((command) =>
command.kind === "raw-trace-update" && command.rawTraceFiles?.cleanupDirectory
? [command.rawTraceFiles.cleanupDirectory]
: []
));
if (directories.size === 0) return;
const task = this.cleanupRawTraceDirectories(directories).finally(() => {
this.cleanupTasks.delete(task);
});
this.cleanupTasks.add(task);
}
private async cleanupRawTraceDirectories(directories: Set<string>): Promise<void> {
for (const directory of directories) {
try {
const spoolDirectory = await realpath(this.options.rawTraceSpoolDir);
const candidate = await realpath(path.resolve(directory));
if (candidate === spoolDirectory || !candidate.startsWith(`${spoolDirectory}${path.sep}`)) {
throw new Error(`Raw trace path is outside the configured spool directory: ${directory}`);
}
await rm(candidate, { force: true, recursive: true });
} catch (error) {
if (errorCode(error) !== "ENOENT") {
console.warn(`[request-log] Failed to clean committed raw trace bundle: ${formatRuntimeError(error)}`);
}
}
}
}
private async ensureQueryWorker(): Promise<void> {
if (this.queryWorkerReady) return await this.queryWorkerReady;
await this.ensureWriter();
this.queryWorkerReady = new Promise<void>((resolve, reject) => {
const worker = new Worker(this.options.workerFile, {
workerData: {
dbFile: this.options.dbFile,
mode: "query",
rawTraceSpoolDir: this.options.rawTraceSpoolDir
}
});
this.queryWorker = worker;
worker.unref();
const onStartupError = (error: Error) => reject(error);
worker.once("error", onStartupError);
worker.on("message", (message: WorkerResponse) => {
if (message.type === "ready") {
worker.off("error", onStartupError);
resolve();
return;
}
settleRpc(this.queryRequests, message);
});
worker.on("error", (error) => {
if (worker !== this.queryWorker) return;
this.queryWorker = undefined;
this.queryWorkerReady = undefined;
rejectPending(this.queryRequests, error);
});
worker.on("exit", (code) => {
if (worker !== this.queryWorker) return;
this.queryWorker = undefined;
this.queryWorkerReady = undefined;
if (!this.closed && code !== 0) rejectPending(this.queryRequests, new Error(`request log query worker exited with ${code}`));
});
});
return await this.queryWorkerReady;
}
private async query<T>(method: string, args: unknown[]): Promise<T> {
if (this.closed) throw new Error("Request log runtime is closed.");
await this.ensureQueryWorker();
if (!this.queryWorker) throw new Error("Request log query worker is unavailable.");
return await rpc<T>(this.queryWorker, this.queryRequests, ++this.nextRequestId, method, args, {
revision: this.committed
});
}
private async writerRpc<T = unknown>(method: string, args: unknown[]): Promise<T> {
await this.ensureWriter();
if (!this.writerWorker) throw new Error("Request log writer is unavailable.");
return await rpc<T>(this.writerWorker, this.writerRequests, ++this.nextRequestId, method, args);
}
private async shutdownWorker(
mode: "query" | "writer",
worker: Worker | undefined,
requests: Map<number, PendingRpc>,
timeoutMs: number
): Promise<void> {
if (!worker) return;
try {
await withTimeout(rpc(worker, requests, ++this.nextRequestId, "shutdown", []), timeoutMs);
} catch {
await worker.terminate();
} finally {
rejectPending(requests, new Error(`Request log ${mode} worker closed.`));
}
}
}
export function createRequestLogRuntime(options: RequestLogRuntimeOptions): RequestLogRuntime {
return new RequestLogRuntime(options);
}
function prepareRecordForQueue(input: RequestLogRecordInput, pressure: number): { degraded: boolean; input: RequestLogRecordInput } {
const maxBodyBytes = Math.max(0, Math.min(maxRequestLogBodyBytes, input.maxBodyBytes ?? defaultRequestLogBodyBytes));
const ordinarySuccess = input.statusCode >= 200 && input.statusCode < 400 && !input.error;
const removeBodies = input.captureBody === false || (pressure >= 0.7 && ordinarySuccess);
const removeTrace = pressure >= 0.85 && ordinarySuccess;
const compactedRequest = removeBodies
? { buffer: Buffer.alloc(0), compacted: false }
: compactBase64ImagePayloads(input.requestBody);
const requestBody = removeBodies
? Buffer.alloc(0)
: boundedBuffer(compactedRequest.buffer, maxBodyBytes);
const compactedResponse = removeBodies
? { compacted: false, text: "" }
: compactBoundedText(input.responseBodyText ?? "", maxBodyBytes);
const responseBodyText = compactedResponse.text;
const responseBodySizeBytes = input.responseBodySizeBytes ?? Buffer.byteLength(input.responseBodyText ?? "");
const responseBodyCapturedBytes = Buffer.byteLength(responseBodyText);
return {
degraded: removeBodies || removeTrace || compactedRequest.compacted || compactedResponse.compacted ||
requestBody.byteLength < compactedRequest.buffer.byteLength ||
responseBodyCapturedBytes < responseBodySizeBytes,
input: {
...input,
requestBody,
requestBodySizeBytes: input.requestBodySizeBytes ?? input.requestBody.byteLength,
requestBodyTruncated: removeBodies || compactedRequest.compacted || Boolean(input.requestBodyTruncated) ||
requestBody.byteLength < compactedRequest.buffer.byteLength,
requestHeaders: plainHeaderRecord(input.requestHeaders),
routeTrace: removeTrace ? undefined : input.routeTrace,
responseBodyText,
responseBodySizeBytes,
responseBodyTruncated: removeBodies || compactedResponse.compacted || Boolean(input.responseBodyTruncated) ||
responseBodyCapturedBytes < responseBodySizeBytes,
responseHeaders: plainHeaderRecord(input.responseHeaders)
}
};
}
function prepareRawTraceForQueue(
input: RequestLogRawTraceUpdateInput,
maxBodyBytes: number
): RequestLogRawTraceUpdateInput {
const requestBody = input.requestBodyText === undefined
? undefined
: compactBoundedText(input.requestBodyText, maxBodyBytes);
const requestBodyText = requestBody?.text;
const requestBodySizeBytes = input.requestBodySizeBytes ??
(input.requestBodyText === undefined ? undefined : Buffer.byteLength(input.requestBodyText));
const responseBody = input.responseBodyText === undefined
? undefined
: compactBoundedText(input.responseBodyText, maxBodyBytes);
const responseBodyText = responseBody?.text;
const responseBodySizeBytes = input.responseBodySizeBytes ??
(input.responseBodyText === undefined ? undefined : Buffer.byteLength(input.responseBodyText));
return {
...input,
...(requestBodyText === undefined ? {} : {
requestBodySizeBytes,
requestBodyText,
requestBodyTruncated: Boolean(input.requestBodyTruncated) || Boolean(requestBody?.compacted) ||
Buffer.byteLength(requestBodyText) < (requestBodySizeBytes ?? 0)
}),
requestHeaders: plainHeaderRecord(input.requestHeaders),
...(responseBodyText === undefined ? {} : {
responseBodySizeBytes,
responseBodyText,
responseBodyTruncated: Boolean(input.responseBodyTruncated) || Boolean(responseBody?.compacted) ||
Buffer.byteLength(responseBodyText) < (responseBodySizeBytes ?? 0)
}),
responseHeaders: plainHeaderRecord(input.responseHeaders)
};
}
function boundedBuffer(value: Buffer, maxBytes: number): Buffer {
return value.byteLength <= maxBytes ? value : Buffer.from(value.subarray(0, maxBytes));
}
function compactBoundedText(value: string, maxBytes: number): { compacted: boolean; text: string } {
const compacted = compactBase64ImagePayloads(Buffer.from(value));
const buffer = boundedBuffer(compacted.buffer, maxBytes);
return {
compacted: compacted.compacted,
text: new StringDecoder("utf8").write(buffer)
};
}
function plainHeaderRecord(value: Headers | Record<string, string | string[] | undefined> | undefined): Record<string, string | string[]> {
if (!value) return {};
const entries = typeof Headers !== "undefined" && value instanceof Headers
? [...value.entries()]
: Object.entries(value).filter((entry): entry is [string, string | string[]] => entry[1] !== undefined);
return Object.fromEntries(entries.map(([key, headerValue]) => [
key,
sensitiveQueueHeaderNames.has(key.toLowerCase()) ? "[redacted]" : headerValue
]));
}
function estimateRecordBytes(input: RequestLogRecordInput): number {
return input.requestBody.byteLength + Buffer.byteLength(input.responseBodyText ?? "") +
jsonBytes(input.requestHeaders) + jsonBytes(input.responseHeaders) + jsonBytes(input.routeTrace) + 1_024;
}
function estimateRawTraceBytes(
input: RequestLogRawTraceUpdateInput,
rawTraceFiles?: RequestLogRawTraceFiles
): number {
return Buffer.byteLength(input.requestBodyText ?? "") + Buffer.byteLength(input.responseBodyText ?? "") +
jsonBytes(input.requestHeaders) + jsonBytes(input.responseHeaders) + jsonBytes(rawTraceFiles) + 512;
}
function rawTraceFileBytes(rawTraceFiles: RequestLogRawTraceFiles | undefined, maxBodyBytes: number): number {
return Math.min(maxBodyBytes, Math.max(0, rawTraceFiles?.requestBody?.sizeBytes ?? 0)) +
Math.min(maxBodyBytes, Math.max(0, rawTraceFiles?.responseBody?.sizeBytes ?? 0));
}
function jsonBytes(value: unknown): number {
if (value === undefined) return 0;
try {
return Buffer.byteLength(JSON.stringify(value));
} catch {
return 0;
}
}
function withoutSize(command: QueuedCommand): RequestLogStoreWriteCommand {
const { batchBytes: _batchBytes, sizeBytes: _sizeBytes, ...output } = command;
return output;
}
function positiveInteger(value: number | undefined, fallback: number): number {
return Number.isFinite(value) && Number(value) > 0 ? Math.floor(Number(value)) : fallback;
}
function rpc<T>(
worker: Worker,
requests: Map<number, PendingRpc>,
requestId: number,
method: string,
args: unknown[],
metadata: Record<string, unknown> = {}
): Promise<T> {
return new Promise<T>((resolve, reject) => {
requests.set(requestId, { reject, resolve: (value) => resolve(value as T) });
worker.postMessage({ ...metadata, args, method, requestId, type: "request" });
});
}
function settleRpc(requests: Map<number, PendingRpc>, message: WorkerResponse): void {
if (message.type !== "response" || message.requestId === undefined) return;
const pending = requests.get(message.requestId);
if (!pending) return;
requests.delete(message.requestId);
if (message.error) pending.reject(new Error(message.error));
else pending.resolve(message.result);
}
function rejectPending(requests: Map<number, PendingRpc>, error: Error): void {
for (const pending of requests.values()) pending.reject(error);
requests.clear();
}
function errorCode(error: unknown): string | undefined {
return typeof error === "object" && error !== null && "code" in error
? String((error as { code?: unknown }).code ?? "") || undefined
: undefined;
}
function formatRuntimeError(error: unknown): string {
return error instanceof Error ? error.stack || error.message : String(error);
}
async function waitUntil(predicate: () => boolean, timeoutMs: number): Promise<boolean> {
const deadline = Date.now() + Math.max(1, timeoutMs);
while (!predicate()) {
if (Date.now() >= deadline) return false;
await new Promise((resolve) => setTimeout(resolve, 5));
}
return true;
}
async function withTimeout<T>(promise: Promise<T>, timeoutMs: number): Promise<T> {
let timer: NodeJS.Timeout | undefined;
try {
return await Promise.race([
promise,
new Promise<never>((_resolve, reject) => {
timer = setTimeout(() => reject(new Error(`Request log operation timed out after ${timeoutMs}ms.`)), timeoutMs);
timer.unref?.();
})
]);
} finally {
if (timer) clearTimeout(timer);
}
}

View file

@ -3,8 +3,11 @@ import { dirname } from "node:path";
import { StringDecoder } from "node:string_decoder";
import { REQUEST_LOGS_DB_FILE } from "@ccr/core/config/constants";
import { estimateUsageCostUsd } from "@ccr/core/models/pricing-service";
import { createBetterSqliteDatabase, type BetterSqliteDatabase } from "@ccr/core/storage/sqlite-native";
import { createBetterSqliteDatabase, type BetterSqliteDatabase, type BetterSqliteStatement } from "@ccr/core/storage/sqlite-native";
import { normalizeUsageInputTokens } from "@ccr/core/usage/normalization";
import { createRequestLogRuntime, type RequestLogEnqueueResult } from "@ccr/core/observability/request-log-runtime";
import { maxRequestLogBodyBytes, rawTraceHardMaxBodyBytes } from "@ccr/core/observability/request-log-limits";
import { compactBase64ImagePayloads } from "@ccr/core/observability/request-log-body";
import type {
AgentAnalysisAgentRow,
AgentAnalysisFilter,
@ -37,12 +40,14 @@ import type {
RequestLogPage,
RequestLogRetryAttempt,
RequestLogStatusFilter,
RequestRouteTrace,
RequestRouteTraceHop,
RequestRouteTraceSnapshot,
UsageStatsRange
} from "@ccr/core/contracts/app";
type SqlDatabase = BetterSqliteDatabase;
type SqlValue = bigint | Buffer | number | string | null;
type HeaderRecord = Record<string, string | string[] | undefined>;
type UsageNumbers = {
@ -65,20 +70,28 @@ type RequestLogUsageContext = {
provider: string;
};
type RequestLogRecordInput = {
export type RequestLogRecordInput = {
captureBody?: boolean;
client?: string;
completedAt?: string;
durationMs: number;
error?: string;
eventId?: string;
fallbackModel?: string;
maxBodyBytes?: number;
method: string;
model?: string;
path: string;
providerName?: string;
providerProtocol?: GatewayProviderProtocol;
requestBody: Buffer;
requestBodySizeBytes?: number;
requestBodyTruncated?: boolean;
requestHeaders: HeaderRecord;
requestId?: string;
routeTrace?: RequestRouteTrace;
responseBodyText?: string;
responseBodySizeBytes?: number;
responseBodyTruncated?: boolean;
responseHeaders?: Headers | HeaderRecord;
startedAt: string;
@ -92,12 +105,14 @@ export type RequestLogRawTraceUpdateInput = {
path?: string;
provider?: string;
requestBodyContentType?: string;
requestBodySizeBytes?: number;
requestBodyText?: string;
requestBodyTruncated?: boolean;
requestHeaders?: HeaderRecord;
requestId: string;
isStream?: boolean;
responseBodyContentType?: string;
responseBodySizeBytes?: number;
responseBodyText?: string;
responseBodyTruncated?: boolean;
responseHeaders?: HeaderRecord;
@ -105,6 +120,32 @@ export type RequestLogRawTraceUpdateInput = {
url?: string;
};
export type RequestLogRawTraceFile = {
contentType?: string;
filePath: string;
sizeBytes: number;
truncated?: boolean;
};
export type RequestLogRawTraceFiles = {
cleanupDirectory?: string;
maxBodyBytes?: number;
requestBody?: RequestLogRawTraceFile;
responseBody?: RequestLogRawTraceFile;
};
export type RequestLogStoreWriteCommand = {
sequence: number;
} & ({
eventId: string;
input: RequestLogRecordInput;
kind: "record";
} | {
input: RequestLogRawTraceUpdateInput;
kind: "raw-trace-update";
rawTraceFiles?: RequestLogRawTraceFiles;
});
type StoredRequestLogEntry = {
cacheReadTokens: number;
cacheWriteTokens: number;
@ -130,6 +171,10 @@ type StoredRequestLogEntry = {
requestBody: RequestLogBody;
requestHeaders: Record<string, string | string[]>;
requestId: string;
routeAttemptCount: number;
routeHopCount: number;
routeTrace?: RequestRouteTrace;
routeTraceTruncated: boolean;
retryAttempts: RequestLogRetryAttempt[];
responseBody?: RequestLogBody;
responseHeaders: Record<string, string | string[]>;
@ -195,7 +240,7 @@ type ToolCallStreamState = {
indexToId: Map<string, string>;
};
const maxBodyBytes = 2 * 1024 * 1024;
const maxBodyBytes = maxRequestLogBodyBytes;
const maxAgentAnalysisRows = 5000;
const maxAgentSessionDetailRequests = 250;
const maxTracePayloadPreviewChars = 1600;
@ -244,12 +289,63 @@ type AgentAnalysisCacheEntry = {
export class RequestLogStore {
private database?: SqlDatabase;
private initPromise?: Promise<SqlDatabase>;
private insertRequestStatement?: BetterSqliteStatement;
private insertRouteTraceStatement?: BetterSqliteStatement;
private lastRetentionCleanupDay?: string;
private revision = 0;
private analysisCache?: AgentAnalysisCacheEntry;
constructor(private readonly dbFile: string) {}
async initialize(): Promise<void> {
await this.getDatabase();
}
invalidateAnalysisCache(): void {
this.analysisCache = undefined;
}
async checkpoint(): Promise<void> {
const database = await this.getDatabase();
database.pragma("wal_checkpoint(PASSIVE)");
}
async close(): Promise<void> {
const database = this.database ?? (this.initPromise ? await this.initPromise.catch(() => undefined) : undefined);
this.database = undefined;
this.initPromise = undefined;
this.insertRequestStatement = undefined;
this.insertRouteTraceStatement = undefined;
database?.close();
}
async writeBatch(commands: RequestLogStoreWriteCommand[]): Promise<void> {
if (commands.length === 0) return;
const database = await this.getDatabase();
database.exec("BEGIN IMMEDIATE");
try {
for (const command of commands) {
if (command.kind === "record") {
await this.record({ ...command.input, eventId: command.eventId });
const requestId = command.input.requestId?.trim();
if (requestId) {
const pending = this.takePendingRawTraceUpdate(database, requestId);
if (pending) {
await this.updateFromRawTrace(pending);
}
}
continue;
}
const applied = await this.updateFromRawTrace(command.input);
if (!applied) this.storePendingRawTraceUpdate(database, command.input);
}
database.exec("COMMIT");
} catch (error) {
if (database.inTransaction) database.exec("ROLLBACK");
throw error;
}
}
async record(input: RequestLogRecordInput): Promise<void> {
const database = await this.getDatabase();
this.pruneOldRequestLogs(database);
@ -265,7 +361,7 @@ export class RequestLogStore {
usageHint: bodyUsage
}) ?? {};
const route = splitRouteSelector(input.fallbackModel);
const requestModel = extractModelFromBody(input.requestBody.toString("utf8"));
const requestModel = normalizeFilterValue(input.model) ?? extractModelFromBody(input.requestBody.toString("utf8"));
const provider =
normalizeFilterValue(input.providerName) ??
readResponseHeader(input.responseHeaders, "x-gateway-target-provider-name") ??
@ -290,14 +386,20 @@ export class RequestLogStore {
outputTokens,
provider: providerName
});
const requestBody = bodyFromBuffer(
const capturedRequestBody = bodyFromBuffer(
input.requestBody,
headerValue(requestHeaders, "content-type")
);
const requestBody: RequestLogBody = {
...capturedRequestBody,
sizeBytes: Math.max(capturedRequestBody.sizeBytes, normalizeCount(input.requestBodySizeBytes)),
truncated: capturedRequestBody.truncated || Boolean(input.requestBodyTruncated)
};
const responseBody = bodyFromText(
responseBodyText,
headerValue(responseHeaders, "content-type"),
Boolean(input.responseBodyTruncated)
Boolean(input.responseBodyTruncated),
input.responseBodySizeBytes
);
const isStream = inferRequestLogIsStream({
path: input.path,
@ -308,11 +410,12 @@ export class RequestLogStore {
url: input.url
});
const statement = database.prepare(`
INSERT INTO request_logs (
const statement = this.insertRequestStatement ??= database.prepare(`
INSERT OR IGNORE INTO request_logs (
created_at,
completed_at,
request_id,
event_id,
client,
method,
path,
@ -322,6 +425,10 @@ export class RequestLogStore {
credential_chain,
credential_saturated,
model,
route_trace_version,
route_hop_count,
route_attempt_count,
route_trace_truncated,
is_stream,
status_code,
ok,
@ -346,48 +453,68 @@ export class RequestLogStore {
response_body_size_bytes,
response_body_truncated,
error
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`);
statement.run(
input.startedAt,
input.completedAt ?? new Date().toISOString(),
input.requestId ?? "",
normalizeLabel(input.client, "unknown"),
input.method,
input.path,
input.url,
providerName,
credentialInfo.id,
credentialInfo.chain.join(","),
credentialInfo.saturated ? 1 : 0,
model,
isStream ? 1 : 0,
normalizeCount(input.statusCode),
isSuccessStatus(input.statusCode, responseError) ? 1 : 0,
normalizeCount(input.durationMs),
inputTokens,
outputTokens,
reasoningTokens,
cacheReadTokens,
cacheWriteTokens,
totalTokens,
cost?.amountUsd ?? null,
JSON.stringify(requestHeaders),
JSON.stringify(responseHeaders),
requestBody.text,
requestBody.encoding,
requestBody.contentType ?? "",
requestBody.sizeBytes,
requestBody.truncated ? 1 : 0,
responseBody.text,
responseBody.encoding,
responseBody.contentType ?? "",
responseBody.sizeBytes,
responseBody.truncated ? 1 : 0,
responseError ?? ""
);
this.revision += 1;
let inserted = false;
const insert = () => {
const result = statement.run(
input.startedAt,
input.completedAt ?? new Date().toISOString(),
input.requestId ?? "",
input.eventId ?? "",
normalizeLabel(input.client, "unknown"),
input.method,
input.path,
input.url,
providerName,
credentialInfo.id,
credentialInfo.chain.join(","),
credentialInfo.saturated ? 1 : 0,
model,
input.routeTrace?.version ?? 0,
input.routeTrace?.hopCount ?? 0,
input.routeTrace?.attemptCount ?? 0,
input.routeTrace?.truncated ? 1 : 0,
isStream ? 1 : 0,
normalizeCount(input.statusCode),
isSuccessStatus(input.statusCode, responseError) ? 1 : 0,
normalizeCount(input.durationMs),
inputTokens,
outputTokens,
reasoningTokens,
cacheReadTokens,
cacheWriteTokens,
totalTokens,
cost?.amountUsd ?? null,
JSON.stringify(requestHeaders),
JSON.stringify(responseHeaders),
requestBody.text,
requestBody.encoding,
requestBody.contentType ?? "",
requestBody.sizeBytes,
requestBody.truncated ? 1 : 0,
responseBody.text,
responseBody.encoding,
responseBody.contentType ?? "",
responseBody.sizeBytes,
responseBody.truncated ? 1 : 0,
responseError ?? ""
);
if (result.changes === 0) return;
inserted = true;
if (input.routeTrace) {
insertRequestRouteTrace(
this.insertRouteTraceStatement ??= prepareRequestRouteTraceInsert(database),
Number(result.lastInsertRowid),
input.requestId ?? "",
input.routeTrace
);
}
};
if (database.inTransaction) insert();
else database.transaction(insert)();
if (inserted) this.revision += 1;
}
async updateFromRawTrace(input: RequestLogRawTraceUpdateInput): Promise<boolean> {
@ -519,7 +646,9 @@ export class RequestLogStore {
const requestBody = bodyFromText(
input.requestBodyText,
input.requestBodyContentType ?? headerValue(mergedRequestHeaders ?? {}, "content-type"),
Boolean(input.requestBodyTruncated)
Boolean(input.requestBodyTruncated),
input.requestBodySizeBytes,
rawTraceHardMaxBodyBytes
);
pushBodyValues(sets, params, "request", requestBody);
}
@ -527,17 +656,23 @@ export class RequestLogStore {
const responseBody = bodyFromText(
input.responseBodyText,
responseBodyContentType,
Boolean(input.responseBodyTruncated)
Boolean(input.responseBodyTruncated),
input.responseBodySizeBytes,
rawTraceHardMaxBodyBytes
);
pushBodyValues(sets, params, "response", responseBody);
}
if (sets.length === 0) {
return true;
const update = () => {
if (sets.length > 0) {
database.prepare(`UPDATE request_logs SET ${sets.join(", ")} WHERE request_id = ?`).run(...params, requestId);
}
};
if (database.inTransaction) update();
else database.transaction(update)();
if (sets.length > 0) {
this.revision += 1;
}
database.prepare(`UPDATE request_logs SET ${sets.join(", ")} WHERE request_id = ?`).run(...params, requestId);
this.revision += 1;
return true;
}
@ -568,6 +703,10 @@ export class RequestLogStore {
credential_chain,
credential_saturated,
model,
route_trace_version,
route_hop_count,
route_attempt_count,
route_trace_truncated,
is_stream,
status_code,
ok,
@ -616,7 +755,12 @@ export class RequestLogStore {
if (requestLogId <= 0) {
return undefined;
}
return readRequestLogById(database, requestLogId);
const entry = readRequestLogById(database, requestLogId);
if (!entry) {
return undefined;
}
entry.routeTrace = readRequestRouteTrace(database, requestLogId);
return entry;
}
async analyze(filter: AgentAnalysisFilter = {}): Promise<AgentAnalysisSnapshot> {
@ -785,6 +929,7 @@ export class RequestLogStore {
created_at TEXT NOT NULL,
completed_at TEXT NOT NULL DEFAULT '',
request_id TEXT NOT NULL DEFAULT '',
event_id TEXT NOT NULL DEFAULT '',
client TEXT NOT NULL DEFAULT 'unknown',
method TEXT NOT NULL,
path TEXT NOT NULL,
@ -794,6 +939,10 @@ export class RequestLogStore {
credential_chain TEXT NOT NULL DEFAULT '',
credential_saturated INTEGER NOT NULL DEFAULT 0,
model TEXT NOT NULL DEFAULT 'unknown',
route_trace_version INTEGER NOT NULL DEFAULT 0,
route_hop_count INTEGER NOT NULL DEFAULT 0,
route_attempt_count INTEGER NOT NULL DEFAULT 0,
route_trace_truncated INTEGER NOT NULL DEFAULT 0,
is_stream INTEGER NOT NULL DEFAULT 0,
status_code INTEGER NOT NULL DEFAULT 0,
ok INTEGER NOT NULL DEFAULT 0,
@ -819,8 +968,61 @@ export class RequestLogStore {
response_body_truncated INTEGER NOT NULL DEFAULT 0,
error TEXT NOT NULL DEFAULT ''
);
CREATE TABLE IF NOT EXISTS request_route_traces (
request_log_id INTEGER PRIMARY KEY,
request_id TEXT NOT NULL DEFAULT '',
version INTEGER NOT NULL DEFAULT 1,
complete INTEGER NOT NULL DEFAULT 1,
ingress_snapshot_json TEXT NOT NULL DEFAULT '{}',
final_snapshot_json TEXT NOT NULL DEFAULT '{}',
hop_count INTEGER NOT NULL DEFAULT 0,
attempt_count INTEGER NOT NULL DEFAULT 0,
truncated INTEGER NOT NULL DEFAULT 0,
trace_json TEXT NOT NULL DEFAULT '',
FOREIGN KEY(request_log_id) REFERENCES request_logs(id) ON DELETE CASCADE
);
CREATE TABLE IF NOT EXISTS request_route_hops (
id INTEGER PRIMARY KEY AUTOINCREMENT,
request_log_id INTEGER NOT NULL,
request_id TEXT NOT NULL DEFAULT '',
seq INTEGER NOT NULL,
phase TEXT NOT NULL,
name TEXT NOT NULL,
kind TEXT NOT NULL,
attempt_no INTEGER,
started_offset_ms INTEGER NOT NULL DEFAULT 0,
duration_ms INTEGER NOT NULL DEFAULT 0,
status TEXT NOT NULL DEFAULT 'ok',
decision_json TEXT NOT NULL DEFAULT '{}',
target_json TEXT NOT NULL DEFAULT '{}',
changes_json TEXT NOT NULL DEFAULT '[]',
outcome_json TEXT NOT NULL DEFAULT '{}',
truncated INTEGER NOT NULL DEFAULT 0,
FOREIGN KEY(request_log_id) REFERENCES request_logs(id) ON DELETE CASCADE,
UNIQUE(request_log_id, seq)
);
CREATE TABLE IF NOT EXISTS request_log_pending_updates (
request_id TEXT PRIMARY KEY,
received_at INTEGER NOT NULL,
update_json TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS request_route_hops_request_idx
ON request_route_hops(request_log_id, seq);
CREATE INDEX IF NOT EXISTS request_route_traces_request_id_idx
ON request_route_traces(request_id);
`);
ensureRequestLogSchema(database);
ensureRequestRouteTraceSchema(database);
database.exec(`
CREATE UNIQUE INDEX IF NOT EXISTS request_logs_event_id_idx
ON request_logs(event_id)
WHERE event_id <> '';
`);
backfillRequestLogStreamFlags(database);
this.database = database;
@ -855,30 +1057,56 @@ export class RequestLogStore {
).run(cutoff);
this.lastRetentionCleanupDay = dayKey;
}
private storePendingRawTraceUpdate(database: SqlDatabase, input: RequestLogRawTraceUpdateInput): void {
const requestId = input.requestId.trim();
if (!requestId) return;
const now = Date.now();
database.prepare(`
INSERT INTO request_log_pending_updates (request_id, received_at, update_json)
VALUES (?, ?, ?)
ON CONFLICT(request_id) DO UPDATE SET received_at = excluded.received_at, update_json = excluded.update_json
`).run(requestId, now, JSON.stringify(input));
database.prepare("DELETE FROM request_log_pending_updates WHERE received_at < ?").run(now - 5 * 60 * 1_000);
database.exec(`
DELETE FROM request_log_pending_updates
WHERE request_id NOT IN (
SELECT request_id FROM request_log_pending_updates ORDER BY received_at DESC LIMIT 200
)
`);
}
private takePendingRawTraceUpdate(database: SqlDatabase, requestId: string): RequestLogRawTraceUpdateInput | undefined {
const row = queryRows(
database,
"SELECT update_json FROM request_log_pending_updates WHERE request_id = ? LIMIT 1",
[requestId]
)[0];
if (!row) return undefined;
database.prepare("DELETE FROM request_log_pending_updates WHERE request_id = ?").run(requestId);
const parsed = parseJson(String(row.update_json ?? ""));
return isRecord(parsed) ? parsed as RequestLogRawTraceUpdateInput : undefined;
}
}
export const requestLogStore = new RequestLogStore(REQUEST_LOGS_DB_FILE);
export const requestLogRuntime = createRequestLogRuntime({ dbFile: REQUEST_LOGS_DB_FILE });
export { createRequestLogRuntime } from "@ccr/core/observability/request-log-runtime";
export async function recordGatewayRequestLog(input: RequestLogRecordInput): Promise<void> {
try {
await requestLogStore.record(input);
} catch (error) {
console.warn(`[request-log] Failed to record request log: ${formatError(error)}`);
}
export function recordGatewayRequestLog(input: RequestLogRecordInput): RequestLogEnqueueResult {
return requestLogRuntime.enqueueRecord(input);
}
export async function updateGatewayRequestLogFromRawTrace(input: RequestLogRawTraceUpdateInput): Promise<boolean> {
try {
return await requestLogStore.updateFromRawTrace(input);
} catch (error) {
console.warn(`[request-log] Failed to update request log from raw trace: ${formatError(error)}`);
return false;
}
export async function updateGatewayRequestLogFromRawTrace(
input: RequestLogRawTraceUpdateInput,
rawTraceFiles?: RequestLogRawTraceFiles
): Promise<boolean> {
return requestLogRuntime.enqueueRawTrace(input, rawTraceFiles).accepted;
}
export async function getRequestLogs(filter?: RequestLogListFilter): Promise<RequestLogPage> {
try {
return await requestLogStore.list(filter);
return await requestLogRuntime.list(filter);
} catch (error) {
console.warn(`[request-log] Failed to read request logs: ${formatError(error)}`);
throw error;
@ -887,7 +1115,7 @@ export async function getRequestLogs(filter?: RequestLogListFilter): Promise<Req
export async function getRequestLogDetail(request: RequestLogDetailRequest): Promise<RequestLogEntry | undefined> {
try {
return await requestLogStore.getDetail(request);
return await requestLogRuntime.getDetail(request);
} catch (error) {
console.warn(`[request-log] Failed to read request log detail: ${formatError(error)}`);
throw error;
@ -896,7 +1124,7 @@ export async function getRequestLogDetail(request: RequestLogDetailRequest): Pro
export async function getAgentAnalysis(filter?: AgentAnalysisFilter): Promise<AgentAnalysisSnapshot> {
try {
return await requestLogStore.analyze(filter);
return await requestLogRuntime.analyze(filter);
} catch (error) {
console.warn(`[request-log] Failed to analyze agent logs: ${formatError(error)}`);
throw error;
@ -905,13 +1133,21 @@ export async function getAgentAnalysis(filter?: AgentAnalysisFilter): Promise<Ag
export async function getAgentTracePayload(request: AgentAnalysisTracePayloadRequest): Promise<AgentAnalysisTracePayloadFullResult> {
try {
return await requestLogStore.getTracePayload(request);
return await requestLogRuntime.getTracePayload(request);
} catch (error) {
console.warn(`[request-log] Failed to read agent trace payload: ${formatError(error)}`);
throw error;
}
}
export async function flushRequestLogRuntime(timeoutMs = 2_000): Promise<{ pending: number; timedOut: boolean }> {
return await requestLogRuntime.flush({ timeoutMs });
}
export async function closeRequestLogRuntime(timeoutMs = 2_000): Promise<void> {
await requestLogRuntime.close({ timeoutMs });
}
function toAnalyzedAgentRequest(entry: StoredRequestLogEntry): AnalyzedAgentRequest {
const details = extractAgentLogDetails(entry);
const startedAtMs = parseDateMs(entry.createdAt);
@ -2687,6 +2923,7 @@ function ensureRequestLogSchema(database: SqlDatabase): void {
addColumn("created_at", "TEXT NOT NULL DEFAULT ''");
addColumn("completed_at", "TEXT NOT NULL DEFAULT ''");
addColumn("request_id", "TEXT NOT NULL DEFAULT ''");
addColumn("event_id", "TEXT NOT NULL DEFAULT ''");
addColumn("client", "TEXT NOT NULL DEFAULT 'unknown'");
addColumn("method", "TEXT NOT NULL DEFAULT ''");
addColumn("path", "TEXT NOT NULL DEFAULT ''");
@ -2696,6 +2933,10 @@ function ensureRequestLogSchema(database: SqlDatabase): void {
addColumn("credential_chain", "TEXT NOT NULL DEFAULT ''");
addColumn("credential_saturated", "INTEGER NOT NULL DEFAULT 0");
addColumn("model", "TEXT NOT NULL DEFAULT 'unknown'");
addColumn("route_trace_version", "INTEGER NOT NULL DEFAULT 0");
addColumn("route_hop_count", "INTEGER NOT NULL DEFAULT 0");
addColumn("route_attempt_count", "INTEGER NOT NULL DEFAULT 0");
addColumn("route_trace_truncated", "INTEGER NOT NULL DEFAULT 0");
addColumn("is_stream", "INTEGER NOT NULL DEFAULT 0");
addColumn("status_code", "INTEGER NOT NULL DEFAULT 0");
addColumn("ok", "INTEGER NOT NULL DEFAULT 0");
@ -2734,6 +2975,17 @@ function ensureRequestLogSchema(database: SqlDatabase): void {
database.exec("CREATE INDEX IF NOT EXISTS request_logs_status_created_at_idx ON request_logs(ok, created_at DESC)");
}
function ensureRequestRouteTraceSchema(database: SqlDatabase): void {
const columns = new Set(
queryRows(database, "PRAGMA table_info(request_route_traces)")
.map((row) => String(row.name ?? ""))
.filter(Boolean)
);
if (!columns.has("trace_json")) {
database.exec("ALTER TABLE request_route_traces ADD COLUMN trace_json TEXT NOT NULL DEFAULT ''");
}
}
function backfillRequestLogStreamFlags(database: SqlDatabase): void {
const rows = queryRows(
database,
@ -2892,11 +3144,180 @@ function buildLogWhereClause(filter: RequestLogListFilter): { params: SqlValue[]
}
function configureSqliteDatabase(database: SqlDatabase): void {
database.pragma("foreign_keys = ON");
database.pragma("journal_mode = WAL");
database.pragma("synchronous = NORMAL");
database.pragma("busy_timeout = 5000");
}
function insertRequestRouteTrace(
statement: BetterSqliteStatement,
requestLogId: number,
requestId: string,
trace: RequestRouteTrace
): void {
statement.run(
requestLogId,
requestId,
trace.version,
trace.complete ? 1 : 0,
JSON.stringify(trace.ingressSnapshot ?? {}),
JSON.stringify(trace.finalSnapshot ?? {}),
trace.hopCount,
trace.attemptCount,
trace.truncated ? 1 : 0,
JSON.stringify(trace)
);
}
function prepareRequestRouteTraceInsert(database: SqlDatabase): BetterSqliteStatement {
return database.prepare(`
INSERT INTO request_route_traces (
request_log_id,
request_id,
version,
complete,
ingress_snapshot_json,
final_snapshot_json,
hop_count,
attempt_count,
truncated,
trace_json
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`);
}
function readRequestRouteTrace(database: SqlDatabase, requestLogId: number): RequestRouteTrace | undefined {
const traceRow = queryRows(
database,
`
SELECT
version,
complete,
ingress_snapshot_json,
final_snapshot_json,
hop_count,
attempt_count,
truncated,
trace_json
FROM request_route_traces
WHERE request_log_id = ?
LIMIT 1
`,
[requestLogId]
)[0];
if (!traceRow) {
return undefined;
}
const storedTrace = parseRequestRouteTrace(traceRow.trace_json);
if (storedTrace) return storedTrace;
const hops = queryRows(
database,
`
SELECT
seq,
phase,
name,
kind,
attempt_no,
started_offset_ms,
duration_ms,
status,
decision_json,
target_json,
changes_json,
outcome_json,
truncated
FROM request_route_hops
WHERE request_log_id = ?
ORDER BY seq ASC
`,
[requestLogId]
).map(requestRouteHopFromRow);
const ingressSnapshot = parseRouteTraceSnapshot(traceRow.ingress_snapshot_json);
const finalSnapshot = parseRouteTraceSnapshot(traceRow.final_snapshot_json);
return {
attemptCount: normalizeCount(traceRow.attempt_count),
complete: normalizeCount(traceRow.complete) === 1,
...(finalSnapshot ? { finalSnapshot } : {}),
hopCount: Math.max(normalizeCount(traceRow.hop_count), hops.length),
hops,
...(ingressSnapshot ? { ingressSnapshot } : {}),
truncated: normalizeCount(traceRow.truncated) === 1,
version: normalizeCount(traceRow.version) === 2 ? 2 : 1
};
}
function parseRequestRouteTrace(value: SqlValue): RequestRouteTrace | undefined {
if (typeof value !== "string" || !value.trim()) return undefined;
const parsed = parseJson(value);
if (!isRecord(parsed) || !Array.isArray(parsed.hops)) return undefined;
return parsed as RequestRouteTrace;
}
function requestRouteHopFromRow(row: Record<string, SqlValue>): RequestRouteTraceHop {
const decision = parseJson(String(row.decision_json ?? ""));
const target = parseJson(String(row.target_json ?? ""));
const changes = parseJson(String(row.changes_json ?? ""));
const outcome = parseJson(String(row.outcome_json ?? ""));
const attempt = normalizeCount(row.attempt_no);
return {
...(attempt > 0 ? { attempt } : {}),
changes: Array.isArray(changes) ? changes as RequestRouteTraceHop["changes"] : [],
...(isRecord(decision) && Object.keys(decision).length > 0
? { decision: decision as RequestRouteTraceHop["decision"] }
: {}),
durationMs: normalizeCount(row.duration_ms),
kind: requestRouteHopKind(row.kind),
name: String(row.name ?? "route-hop"),
...(isRecord(outcome) && Object.keys(outcome).length > 0
? { outcome: outcome as RequestRouteTraceHop["outcome"] }
: {}),
phase: requestRouteTracePhase(row.phase),
seq: normalizeCount(row.seq),
startedOffsetMs: normalizeCount(row.started_offset_ms),
status: requestRouteHopStatus(row.status),
...(isRecord(target) && Object.keys(target).length > 0
? { target: target as RequestRouteTraceHop["target"] }
: {}),
...(normalizeCount(row.truncated) === 1 ? { truncated: true } : {})
};
}
function parseRouteTraceSnapshot(value: SqlValue): RequestRouteTraceSnapshot | undefined {
if (typeof value !== "string" || !value.trim()) {
return undefined;
}
const parsed = parseJson(value);
if (!isRecord(parsed) || typeof parsed.method !== "string" || typeof parsed.url !== "string") {
return undefined;
}
return parsed as RequestRouteTraceSnapshot;
}
function requestRouteHopKind(value: SqlValue): RequestRouteTraceHop["kind"] {
const kind = String(value ?? "");
return kind === "attempt" || kind === "decision" || kind === "outcome" || kind === "snapshot"
? kind
: "mutation";
}
function requestRouteHopStatus(value: SqlValue): RequestRouteTraceHop["status"] {
const status = String(value ?? "");
return status === "error" || status === "noop" ? status : "ok";
}
function requestRouteTracePhase(value: SqlValue): RequestRouteTraceHop["phase"] {
const phase = String(value ?? "");
return phase === "ingress" || phase === "compatibility" || phase === "routing" ||
phase === "capability" || phase === "enrichment" || phase === "planning" ||
phase === "attempt" || phase === "core" || phase === "outcome"
? phase
: "routing";
}
function queryRows(database: SqlDatabase, sql: string, params: SqlValue[] = []): Record<string, SqlValue>[] {
return database.prepare(sql).all(...params) as Record<string, SqlValue>[];
}
@ -2932,6 +3353,10 @@ function readRequestLogById(database: SqlDatabase, id: number): StoredRequestLog
credential_chain,
credential_saturated,
model,
route_trace_version,
route_hop_count,
route_attempt_count,
route_trace_truncated,
is_stream,
status_code,
ok,
@ -3004,6 +3429,9 @@ function toRequestLogEntry(row: Record<string, SqlValue>): StoredRequestLogEntry
requestBody,
requestHeaders,
requestId: String(row.request_id ?? ""),
routeAttemptCount: normalizeCount(row.route_attempt_count),
routeHopCount: normalizeCount(row.route_hop_count),
routeTraceTruncated: normalizeCount(row.route_trace_truncated) === 1,
retryAttempts: parseRequestLogRetryAttempts(responseHeaders, normalizeCount(row.status_code)),
responseBody,
responseHeaders,
@ -3080,27 +3508,38 @@ function readDistinctValues(database: SqlDatabase, column: "credential_id" | "mo
}
function bodyFromBuffer(buffer: Buffer, contentType?: string): RequestLogBody {
const truncated = buffer.byteLength > maxBodyBytes;
const data = truncated ? buffer.subarray(0, maxBodyBytes) : buffer;
const compacted = compactBase64ImagePayloads(buffer);
const exceedsCaptureLimit = compacted.buffer.byteLength > maxBodyBytes;
const data = exceedsCaptureLimit ? compacted.buffer.subarray(0, maxBodyBytes) : compacted.buffer;
const textLike = isTextLikeContentType(contentType);
return {
contentType,
encoding: textLike ? "utf8" : "base64",
sizeBytes: buffer.byteLength,
text: textLike ? data.toString("utf8") : data.toString("base64"),
truncated
truncated: compacted.compacted || exceedsCaptureLimit
};
}
function bodyFromText(text: string, contentType?: string, alreadyTruncated = false): RequestLogBody {
function bodyFromText(
text: string,
contentType?: string,
alreadyTruncated = false,
originalSizeBytes?: number,
captureLimitBytes = maxBodyBytes
): RequestLogBody {
const buffer = Buffer.from(text);
const truncated = alreadyTruncated || buffer.byteLength > maxBodyBytes;
const data = truncated ? buffer.subarray(0, maxBodyBytes) : buffer;
const sizeBytes = Math.max(buffer.byteLength, normalizeCount(originalSizeBytes));
const compacted = compactBase64ImagePayloads(buffer);
const truncated = alreadyTruncated || compacted.compacted || buffer.byteLength < sizeBytes ||
compacted.buffer.byteLength > captureLimitBytes;
const exceedsCaptureLimit = compacted.buffer.byteLength > captureLimitBytes;
const data = exceedsCaptureLimit ? compacted.buffer.subarray(0, captureLimitBytes) : compacted.buffer;
return {
contentType,
encoding: "utf8",
sizeBytes: buffer.byteLength,
text: data.toString("utf8"),
sizeBytes,
text: exceedsCaptureLimit ? new StringDecoder("utf8").write(data) : data.toString("utf8"),
truncated
};
}

View file

@ -0,0 +1,236 @@
import { closeSync, fstatSync, openSync, readSync, realpathSync } from "node:fs";
import { resolve as pathResolve, sep as pathSep } from "node:path";
import { StringDecoder } from "node:string_decoder";
import { parentPort, workerData } from "node:worker_threads";
import { RAW_TRACE_SPOOL_DIR } from "@ccr/core/config/constants";
import {
RequestLogStore,
type RequestLogRawTraceFile,
type RequestLogRecordInput,
type RequestLogStoreWriteCommand
} from "@ccr/core/observability/request-log-store";
import { resolveRawTraceBodyLimit } from "@ccr/core/observability/request-log-limits";
import { compactBase64ImagePayloads } from "@ccr/core/observability/request-log-body";
type WorkerConfiguration = {
dbFile: string;
mode: "query" | "writer";
rawTraceSpoolDir?: string;
};
type WorkerMessage = {
args?: unknown[];
batchId?: number;
commands?: RequestLogStoreWriteCommand[];
method?: string;
requestId?: number;
revision?: number;
type: "batch" | "request";
};
const configuration = workerData as WorkerConfiguration;
const store = new RequestLogStore(configuration.dbFile);
let chain = Promise.resolve();
let queryRevision = -1;
void store.initialize().then(() => {
parentPort?.postMessage({ type: "ready" });
parentPort?.on("message", (message: WorkerMessage) => {
chain = chain.then(() => handleMessage(message)).catch((error) => {
const detail = formatError(error);
if (message.type === "batch") {
parentPort?.postMessage({ batchId: message.batchId, error: detail, type: "batch-error" });
} else {
parentPort?.postMessage({ error: detail, requestId: message.requestId, type: "response" });
}
});
});
}).catch((error) => {
throw error;
});
async function handleMessage(message: WorkerMessage): Promise<void> {
if (message.type === "batch") {
if (configuration.mode !== "writer") throw new Error("Query worker cannot process writes.");
const commands = message.commands ?? [];
await store.writeBatch(commands.map(reviveCommand));
parentPort?.postMessage({ batchId: message.batchId, type: "ack" });
return;
}
const args = message.args ?? [];
if (configuration.mode === "query" && message.revision !== undefined && message.revision !== queryRevision) {
queryRevision = message.revision;
store.invalidateAnalysisCache();
}
let result: unknown;
switch (message.method) {
case "analyze":
result = await store.analyze(args[0] as Parameters<RequestLogStore["analyze"]>[0]);
break;
case "flush":
await store.checkpoint();
result = true;
break;
case "getDetail":
result = await store.getDetail(args[0] as Parameters<RequestLogStore["getDetail"]>[0]);
break;
case "getTracePayload":
result = await store.getTracePayload(args[0] as Parameters<RequestLogStore["getTracePayload"]>[0]);
break;
case "list":
result = await store.list(args[0] as Parameters<RequestLogStore["list"]>[0]);
break;
case "shutdown":
await store.close();
result = true;
break;
default:
throw new Error(`Unknown request log worker method: ${message.method ?? ""}`);
}
parentPort?.postMessage({ requestId: message.requestId, result, type: "response" });
if (message.method === "shutdown") parentPort?.close();
}
function reviveCommand(command: RequestLogStoreWriteCommand): RequestLogStoreWriteCommand {
if (command.kind === "raw-trace-update") {
const input = { ...command.input };
const maxBodyBytes = resolveRawTraceBodyLimit(command.rawTraceFiles?.maxBodyBytes);
if (command.rawTraceFiles?.requestBody) {
const body = readRawTraceBody(command.rawTraceFiles.requestBody, maxBodyBytes);
if (body) {
input.requestBodyContentType = body.contentType ?? input.requestBodyContentType;
input.requestBodySizeBytes = body.sizeBytes;
input.requestBodyTruncated = body.truncated;
if (shouldApplyRawTraceBodyText(body)) input.requestBodyText = body.text;
}
}
if (command.rawTraceFiles?.responseBody) {
const body = readRawTraceBody(command.rawTraceFiles.responseBody, maxBodyBytes);
if (body) {
input.responseBodyContentType = body.contentType ?? input.responseBodyContentType;
input.responseBodySizeBytes = body.sizeBytes;
input.responseBodyTruncated = body.truncated;
if (shouldApplyRawTraceBodyText(body)) input.responseBodyText = body.text;
}
}
return {
input,
kind: "raw-trace-update",
sequence: command.sequence
};
}
const input = command.input as RequestLogRecordInput & { requestBody: Buffer | Uint8Array };
return {
...command,
input: {
...input,
requestBody: Buffer.isBuffer(input.requestBody) ? input.requestBody : Buffer.from(input.requestBody)
}
};
}
function readRawTraceBody(
file: RequestLogRawTraceFile,
maxBodyBytes: number
): (RequestLogRawTraceFile & { text: string }) | undefined {
let descriptor: number | undefined;
try {
const filePath = verifiedRawTracePath(file.filePath);
descriptor = openSync(filePath, "r");
const storedBytes = fstatSync(descriptor).size;
const captureBytes = Math.min(storedBytes, maxBodyBytes);
const buffer = Buffer.allocUnsafe(captureBytes);
let offset = 0;
while (offset < captureBytes) {
const bytesRead = readSync(descriptor, buffer, offset, captureBytes - offset, offset);
if (bytesRead === 0) break;
offset += bytesRead;
}
const captured = offset === buffer.byteLength ? buffer : buffer.subarray(0, offset);
const compacted = compactBase64ImagePayloads(captured);
return {
...file,
filePath,
sizeBytes: Math.max(file.sizeBytes, storedBytes),
text: new StringDecoder("utf8").write(compacted.buffer),
truncated: Boolean(file.truncated) || compacted.compacted ||
offset < Math.max(file.sizeBytes, storedBytes)
};
} catch (error) {
if (errorCode(error) === "ENOENT") return undefined;
throw error;
} finally {
if (descriptor !== undefined) closeSync(descriptor);
}
}
function shouldApplyRawTraceBodyText(body: RequestLogRawTraceFile & { text: string }): boolean {
if (!body.truncated) return true;
const contentType = body.contentType?.split(";", 1)[0].trim().toLowerCase() ?? "";
const firstContentIndex = skipJsonWhitespace(body.text, 0);
const firstContent = body.text.charCodeAt(firstContentIndex);
const jsonLike = contentType === "application/json" || contentType.endsWith("+json") ||
firstContent === 0x7b || firstContent === 0x5b;
if (!jsonLike) return true;
return isCompleteJsonContainer(body.text, firstContentIndex);
}
function isCompleteJsonContainer(value: string, start: number): boolean {
const first = value.charCodeAt(start);
if (first !== 0x7b && first !== 0x5b) return false;
const stack: number[] = [first];
let escaped = false;
let inString = false;
for (let index = start + 1; index < value.length; index += 1) {
const character = value.charCodeAt(index);
if (inString) {
if (escaped) escaped = false;
else if (character === 0x5c) escaped = true;
else if (character === 0x22) inString = false;
continue;
}
if (character === 0x22) {
inString = true;
continue;
}
if (character === 0x7b || character === 0x5b) {
stack.push(character);
continue;
}
if (character !== 0x7d && character !== 0x5d) continue;
const open = stack.pop();
if ((character === 0x7d && open !== 0x7b) || (character === 0x5d && open !== 0x5b)) return false;
if (stack.length === 0) return skipJsonWhitespace(value, index + 1) === value.length;
}
return false;
}
function skipJsonWhitespace(value: string, start: number): number {
let index = start;
while (index < value.length) {
const character = value.charCodeAt(index);
if (character !== 0x20 && character !== 0x09 && character !== 0x0a && character !== 0x0d) break;
index += 1;
}
return index;
}
function verifiedRawTracePath(value: string): string {
const spoolDirectory = realpathSync(configuration.rawTraceSpoolDir ?? RAW_TRACE_SPOOL_DIR);
const candidate = realpathSync(pathResolve(value));
if (candidate === spoolDirectory || !candidate.startsWith(`${spoolDirectory}${pathSep}`)) {
throw new Error(`Raw trace path is outside the configured spool directory: ${value}`);
}
return candidate;
}
function formatError(error: unknown): string {
return error instanceof Error ? error.stack || error.message : String(error);
}
function errorCode(error: unknown): string | undefined {
return typeof error === "object" && error !== null && "code" in error
? String((error as { code?: unknown }).code ?? "") || undefined
: undefined;
}

View file

@ -0,0 +1,344 @@
import type {
RequestRouteTrace,
RequestRouteTraceChange,
RequestRouteTraceDecision,
RequestRouteTraceHop,
RequestRouteTraceOutcome,
RequestRouteTracePhase,
RequestRouteTraceTarget
} from "@ccr/core/contracts/app";
export type RouteTraceObservation = {
attempt?: number;
changes?: readonly RequestRouteTraceChange[];
decision?: RequestRouteTraceDecision;
durationMs?: number;
kind?: RequestRouteTraceHop["kind"];
name: string;
outcome?: RequestRouteTraceOutcome;
phase: RequestRouteTracePhase;
startedAtMs?: number;
status?: RequestRouteTraceHop["status"];
target?: RequestRouteTraceTarget;
};
export type RouteTraceObserver = {
capture: (observation: RouteTraceObservation) => void;
};
const maxArrayItems = 16;
const maxChangesPerHop = 64;
const maxDepth = 6;
const maxHops = 64;
const maxObjectEntries = 32;
const maxPreviewBytes = 2 * 1024;
const maxStringChars = 1_024;
const maxTraceBytes = 256 * 1024;
const redactedDisplayValue = "[redacted]";
const truncatedDisplayValue = "[truncated]";
const sensitiveNames = /(?:^|[-_.])(authorization|cookie|api[-_]?key|access[-_]?token|refresh[-_]?token|password|secret|proxy[-_]?authorization)(?:$|[-_.])/i;
type PreviewBudget = {
remaining: number;
truncated: boolean;
};
/**
* Records route events supplied by the code that performs each mutation.
*
* This recorder deliberately never snapshots, parses, or compares request
* bodies. Mutation sites must report their own paths and values through
* `changes`, which keeps the tracing cost proportional to the reported
* changes instead of the total request size and hop count.
*/
export class RequestRouteTraceRecorder implements RouteTraceObserver {
private readonly attempts = new Set<number>();
private estimatedBytes = 0;
private finished?: RequestRouteTrace;
private readonly hops: RequestRouteTraceHop[] = [];
private sealed = false;
private truncated = false;
constructor(private readonly startedAtMs: number) {}
captureIngress(): void {
if (this.finished || this.hops.length > 0) {
return;
}
this.pushHop({
changes: [],
durationMs: 0,
kind: "snapshot",
name: "request.ingress",
phase: "ingress",
seq: 0,
startedOffsetMs: 0,
status: "ok"
});
}
capture(observation: RouteTraceObservation): void {
if (this.finished || this.sealed || this.hops.length >= maxHops) {
this.truncated = true;
this.sealed = true;
return;
}
const reportedChanges = observation.changes ?? [];
const changes = reportedChanges
.slice(0, maxChangesPerHop)
.map(sanitizeReportedChange);
const changesTruncated = reportedChanges.length > changes.length || changes.some((change) => change.truncated);
if (observation.attempt !== undefined) {
this.attempts.add(observation.attempt);
}
const observationStartedAtMs = observation.startedAtMs ?? Date.now();
const hop: RequestRouteTraceHop = {
...(observation.attempt === undefined ? {} : { attempt: observation.attempt }),
changes,
...(observation.decision ? { decision: boundedObservationValue(observation.decision) } : {}),
durationMs: Math.max(0, Math.round(observation.durationMs ?? 0)),
kind: observation.kind ?? (changes.length > 0 ? "mutation" : "decision"),
name: observation.name,
...(observation.outcome ? { outcome: boundedObservationValue(observation.outcome) } : {}),
phase: observation.phase,
seq: this.hops.length,
startedOffsetMs: Math.max(0, Math.round(observationStartedAtMs - this.startedAtMs)),
status: observation.status ?? (changes.length > 0 ? "ok" : "noop"),
...(observation.target ? { target: boundedObservationValue(observation.target) } : {}),
...(changesTruncated ? { truncated: true } : {})
};
if (changesTruncated) {
this.truncated = true;
}
this.pushHop(hop);
}
finish(): RequestRouteTrace {
if (this.finished) {
return this.finished;
}
this.finished = {
attemptCount: this.attempts.size,
complete: true,
hopCount: this.hops.length,
hops: this.hops,
truncated: this.truncated,
version: 2
};
return this.finished;
}
private pushHop(hop: RequestRouteTraceHop): void {
const remaining = maxTraceBytes - this.estimatedBytes;
if (remaining <= 0) {
this.truncated = true;
this.sealed = true;
return;
}
let nextHop = hop;
let hopBytes = jsonByteLength(nextHop);
if (hopBytes > remaining && nextHop.changes.length > 0) {
const retained: RequestRouteTraceChange[] = [];
let retainedBytes = jsonByteLength({ ...nextHop, changes: [] });
for (const change of nextHop.changes) {
const changeBytes = jsonByteLength(change) + 1;
if (retainedBytes + changeBytes > remaining) {
break;
}
retained.push(change);
retainedBytes += changeBytes;
}
nextHop = {
...nextHop,
changes: retained,
truncated: retained.length < nextHop.changes.length || nextHop.truncated
};
hopBytes = retainedBytes;
if (nextHop.truncated) {
this.truncated = true;
}
}
if (hopBytes > remaining) {
this.truncated = true;
this.sealed = true;
return;
}
this.hops.push(nextHop);
this.estimatedBytes += hopBytes;
}
}
function boundedObservationValue<T extends object>(value: T): T {
return previewValue(value).value as T;
}
function sanitizeReportedChange(change: RequestRouteTraceChange): RequestRouteTraceChange {
const path = normalizePath(change.path);
const redacted = Boolean(change.redacted) || pathContainsSensitiveName(path);
const before = change.before === undefined || redacted
? undefined
: previewValue(change.scope === "url" ? sanitizeUrlValue(change.before) : change.before);
const after = change.after === undefined || redacted
? undefined
: previewValue(change.scope === "url" ? sanitizeUrlValue(change.after) : change.after);
return {
...(change.after === undefined
? {}
: { after: redacted ? redactedDisplayValue : after?.value }),
...(change.before === undefined
? {}
: { before: redacted ? redactedDisplayValue : before?.value }),
operation: change.operation,
path,
...(redacted ? { redacted: true } : {}),
scope: change.scope,
...((change.truncated || before?.truncated || after?.truncated) ? { truncated: true } : {})
};
}
function sanitizeUrlValue(value: unknown): unknown {
if (typeof value !== "string") {
return value;
}
try {
const url = new URL(value, "http://127.0.0.1");
for (const key of [...url.searchParams.keys()]) {
if (sensitiveNames.test(key) || /^(?:key|token)$/i.test(key)) {
url.searchParams.set(key, redactedDisplayValue);
}
}
return /^[a-z][a-z\d+.-]*:/i.test(value)
? url.toString()
: `${url.pathname}${url.search}${url.hash}`;
} catch {
return value;
}
}
function previewValue(value: unknown): { truncated: boolean; value: unknown } {
const budget: PreviewBudget = { remaining: maxPreviewBytes, truncated: false };
const preview = boundedPreview(value, budget, 0, new WeakSet());
return {
truncated: budget.truncated,
value: preview ?? truncatedDisplayValue
};
}
function boundedPreview(
value: unknown,
budget: PreviewBudget,
depth: number,
seen: WeakSet<object>,
key?: string
): unknown {
if (budget.remaining <= 0) {
budget.truncated = true;
return truncatedDisplayValue;
}
if (key && sensitiveNames.test(key)) {
budget.remaining -= redactedDisplayValue.length;
return redactedDisplayValue;
}
if (value === null) {
budget.remaining -= 4;
return null;
}
if (Buffer.isBuffer(value)) {
budget.remaining -= 32;
return { sizeBytes: value.byteLength, type: "buffer" };
}
if (typeof value === "string") {
const output = value.length <= maxStringChars
? value
: `${value.slice(0, maxStringChars)}…[${value.length - maxStringChars} chars truncated]`;
budget.remaining -= Math.min(maxStringChars, output.length) * 2;
if (output.length !== value.length) budget.truncated = true;
return output;
}
if (typeof value === "number" || typeof value === "boolean") {
budget.remaining -= 16;
return value;
}
if (typeof value === "bigint") {
budget.remaining -= 32;
return value.toString();
}
if (value === undefined || typeof value === "function" || typeof value === "symbol") {
return undefined;
}
if (depth >= maxDepth) {
budget.truncated = true;
return truncatedDisplayValue;
}
if (seen.has(value)) {
budget.truncated = true;
return "[circular]";
}
seen.add(value);
try {
if (Array.isArray(value)) {
const output = value
.slice(0, maxArrayItems)
.map((item) => boundedPreview(item, budget, depth + 1, seen));
if (value.length > maxArrayItems) {
output.push(`[${value.length - maxArrayItems} items omitted]`);
budget.truncated = true;
}
return output;
}
const output: Record<string, unknown> = {};
const record = value as Record<string, unknown>;
let entryCount = 0;
let omitted = false;
for (const entryKey in record) {
if (!Object.prototype.hasOwnProperty.call(record, entryKey)) {
continue;
}
if (entryCount >= maxObjectEntries) {
omitted = true;
break;
}
output[entryKey] = boundedPreview(record[entryKey], budget, depth + 1, seen, entryKey);
entryCount += 1;
if (budget.remaining <= 0) break;
}
if (omitted) {
output[truncatedDisplayValue] = "additional fields omitted";
budget.truncated = true;
}
return output;
} finally {
seen.delete(value);
}
}
function pathContainsSensitiveName(path: string): boolean {
return path
.split("/")
.filter(Boolean)
.map((part) => part.replaceAll("~1", "/").replaceAll("~0", "~"))
.some((part) => sensitiveNames.test(part));
}
function normalizePath(value: string): string {
const path = value.trim();
if (!path) return "/";
const normalized = path.startsWith("/") ? path : `/${path}`;
return normalized.length <= maxStringChars
? normalized
: `${normalized.slice(0, maxStringChars)}…`;
}
function jsonByteLength(value: unknown): number {
try {
return Buffer.byteLength(JSON.stringify(value), "utf8");
} catch {
return maxTraceBytes;
}
}

View file

@ -25,7 +25,7 @@ import { getProfileOpenCommand, getProfileRuntimeStatus, openProfileFromCcr, sto
import { ensureProxyCertificateAuthority } from "@ccr/core/proxy/certificates";
import { proxyService } from "@ccr/core/proxy/service";
import { listMcpServerTools } from "@ccr/core/mcp/tool-discovery";
import { getAgentAnalysis, getAgentTracePayload, getRequestLogDetail, getRequestLogs } from "@ccr/core/observability/request-log-store";
import { closeRequestLogRuntime, getAgentAnalysis, getAgentTracePayload, getRequestLogDetail, getRequestLogs } from "@ccr/core/observability/request-log-store";
import { getUsageStats } from "@ccr/core/usage/store";
import { gatewayService } from "@ccr/core/gateway/service";
import { shouldRestartGatewayForRuntimeConfigChange } from "@ccr/core/gateway/runtime-change";
@ -160,6 +160,7 @@ export async function startWebManagementServer(options: WebManagementServerOptio
close: async () => {
await closeServer(server);
await stopConfiguredServices();
await closeRequestLogRuntime();
},
server,
url

View file

@ -0,0 +1,552 @@
import assert from "node:assert/strict";
import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import test from "node:test";
import { createRequestLogRuntime, RequestLogStore } from "@ccr/core/observability/request-log-store.ts";
const workerFile = [
path.resolve(__dirname, "../../../runtime/request-log-worker.js"),
path.resolve(__dirname, "../../runtime/request-log-worker.js")
].find(existsSync);
assert.ok(workerFile, "compiled request-log-worker.js must exist");
test("RequestLogRuntime writes through a worker and reads through the query worker", async () => {
const dir = mkdtempSync(path.join(tmpdir(), "ccr-request-log-runtime-test-"));
const runtime = createRuntime(dir);
try {
const result = runtime.enqueueRecord(createRecord("worker-request"));
assert.deepEqual(result, { accepted: true, degraded: false });
const flush = await runtime.flush({ timeoutMs: 10_000 });
assert.equal(flush.timedOut, false);
assert.equal(flush.pending, 0);
const page = await runtime.list({ pageSize: 25 });
assert.equal(page.items.length, 1);
assert.equal(page.items[0].requestId, "worker-request");
const detail = await runtime.getDetail({ id: page.items[0].id });
assert.equal(detail?.requestBody.text.includes("worker-model"), true);
assert.equal(runtime.metrics().committed, 1);
} finally {
await runtime.close({ timeoutMs: 5_000 });
rmSync(dir, { force: true, recursive: true });
}
});
test("RequestLogRuntime merges a raw trace update that arrives before its request record", async () => {
const dir = mkdtempSync(path.join(tmpdir(), "ccr-request-log-runtime-order-test-"));
const runtime = createRuntime(dir);
try {
assert.equal(runtime.enqueueRawTrace({
model: "trace-model",
provider: "trace-provider",
requestId: "out-of-order-request",
statusCode: 429
}).accepted, true);
assert.equal(runtime.enqueueRecord(createRecord("out-of-order-request")).accepted, true);
await runtime.flush({ timeoutMs: 10_000 });
const page = await runtime.list({ pageSize: 25 });
assert.equal(page.items.length, 1);
assert.equal(page.items[0].model, "trace-model");
assert.equal(page.items[0].provider, "trace-provider");
assert.equal(page.items[0].statusCode, 429);
} finally {
await runtime.close({ timeoutMs: 5_000 });
rmSync(dir, { force: true, recursive: true });
}
});
test("RequestLogRuntime rejects an event that exceeds its hard queue byte limit", async () => {
const dir = mkdtempSync(path.join(tmpdir(), "ccr-request-log-runtime-bound-test-"));
const runtime = createRequestLogRuntime({
dbFile: path.join(dir, "request-logs.sqlite"),
queueMaxBytes: 1_024,
queueMaxItems: 10,
workerFile
});
try {
const result = runtime.enqueueRecord({
...createRecord("oversized-request"),
requestBody: Buffer.alloc(4 * 1_024, "x")
});
assert.equal(result.accepted, false);
assert.equal(result.reason, "queue_full");
assert.equal(runtime.metrics().dropped, 1);
} finally {
await runtime.close({ timeoutMs: 5_000 });
rmSync(dir, { force: true, recursive: true });
}
});
test("RequestLogRuntime persists unmatched raw trace updates across worker restarts", async () => {
const dir = mkdtempSync(path.join(tmpdir(), "ccr-request-log-runtime-pending-test-"));
const first = createRuntime(dir);
try {
first.enqueueRawTrace({ model: "persisted-trace-model", requestId: "restart-request" });
await first.flush({ timeoutMs: 10_000 });
await first.close({ timeoutMs: 5_000 });
const second = createRuntime(dir);
try {
second.enqueueRecord(createRecord("restart-request"));
await second.flush({ timeoutMs: 10_000 });
const page = await second.list({ pageSize: 25 });
assert.equal(page.items[0].model, "persisted-trace-model");
} finally {
await second.close({ timeoutMs: 5_000 });
}
} finally {
await first.close({ timeoutMs: 5_000 });
rmSync(dir, { force: true, recursive: true });
}
});
test("RequestLogRuntime preserves original response sizes when normal body capture is truncated or disabled", async () => {
const dir = mkdtempSync(path.join(tmpdir(), "ccr-request-log-runtime-response-size-test-"));
const runtime = createRuntime(dir);
try {
const fullResponse = "x".repeat(1024 * 1024);
runtime.enqueueRecord({
...createRecord("truncated-response"),
maxBodyBytes: 512 * 1024,
responseBodyText: fullResponse
});
runtime.enqueueRecord({
...createRecord("metadata-response"),
captureBody: false,
responseBodyText: "exists"
});
await runtime.flush({ timeoutMs: 10_000 });
const page = await runtime.list({ pageSize: 25 });
const truncatedEntry = page.items.find((item) => item.requestId === "truncated-response");
const metadataEntry = page.items.find((item) => item.requestId === "metadata-response");
const truncated = await runtime.getDetail({ id: truncatedEntry.id });
const metadata = await runtime.getDetail({ id: metadataEntry.id });
assert.equal(truncated.responseBody.sizeBytes, Buffer.byteLength(fullResponse));
assert.equal(Buffer.byteLength(truncated.responseBody.text), 512 * 1024);
assert.equal(truncated.responseBody.truncated, true);
assert.equal(metadata.responseBody.sizeBytes, Buffer.byteLength("exists"));
assert.equal(metadata.responseBody.text, "");
assert.equal(metadata.responseBody.truncated, true);
} finally {
await runtime.close({ timeoutMs: 5_000 });
rmSync(dir, { force: true, recursive: true });
}
});
test("RequestLogRuntime retains inline raw trace bodies below the default safety limit", async () => {
const dir = mkdtempSync(path.join(tmpdir(), "ccr-request-log-runtime-raw-body-test-"));
const runtime = createRuntime(dir);
try {
const body = "r".repeat(3 * 1024 * 1024);
runtime.enqueueRecord(createRecord("complete-raw-body"));
runtime.enqueueRawTrace({
requestId: "complete-raw-body",
responseBodyContentType: "text/plain",
responseBodySizeBytes: Buffer.byteLength(body),
responseBodyText: body,
statusCode: 200
});
await runtime.flush({ timeoutMs: 10_000 });
const page = await runtime.list({ pageSize: 25 });
const detail = await runtime.getDetail({ id: page.items[0].id });
assert.equal(detail.responseBody.sizeBytes, Buffer.byteLength(body));
assert.equal(detail.responseBody.text.length, body.length);
assert.equal(detail.responseBody.truncated, false);
} finally {
await runtime.close({ timeoutMs: 5_000 });
rmSync(dir, { force: true, recursive: true });
}
});
test("RequestLogRuntime bounds file-backed raw bodies and cleans bundles only after ACK", async () => {
const dir = mkdtempSync(path.join(tmpdir(), "ccr-request-log-runtime-raw-file-test-"));
const spoolDir = path.join(dir, "raw-trace-spool");
const bundleDir = path.join(spoolDir, "bundle");
const responseFile = path.join(bundleDir, "response_stream.txt");
mkdirSync(bundleDir, { recursive: true });
const body = "f".repeat(700 * 1024);
writeFileSync(responseFile, body);
const runtime = createRuntime(dir, { rawTraceSpoolDir: spoolDir });
try {
runtime.enqueueRecord(createRecord("file-backed-raw-body"));
const result = runtime.enqueueRawTrace({
requestId: "file-backed-raw-body",
responseBodySizeBytes: Buffer.byteLength(body),
statusCode: 200
}, {
cleanupDirectory: bundleDir,
maxBodyBytes: 256 * 1024,
responseBody: {
contentType: "text/plain",
filePath: responseFile,
sizeBytes: Buffer.byteLength(body)
}
});
assert.equal(result.accepted, true);
await runtime.flush({ timeoutMs: 10_000 });
assert.equal(existsSync(bundleDir), false);
const page = await runtime.list({ pageSize: 25 });
const detail = await runtime.getDetail({ id: page.items[0].id });
assert.equal(detail.responseBody.sizeBytes, Buffer.byteLength(body));
assert.equal(detail.responseBody.text.length, 256 * 1024);
assert.equal(detail.responseBody.truncated, true);
} finally {
await runtime.close({ timeoutMs: 5_000 });
rmSync(dir, { force: true, recursive: true });
}
});
test("RequestLogRuntime compacts Base64 images while reading file-backed raw traces", async () => {
const dir = mkdtempSync(path.join(tmpdir(), "ccr-request-log-runtime-raw-base64-test-"));
const spoolDir = path.join(dir, "raw-trace-spool");
const bundleDir = path.join(spoolDir, "bundle");
const requestFile = path.join(bundleDir, "upstream_request.json");
mkdirSync(bundleDir, { recursive: true });
const body = JSON.stringify({
image_url: { url: `data:image/jpeg;base64,${"A".repeat(512 * 1024)}` },
model: "raw-vision-model"
});
writeFileSync(requestFile, body);
const runtime = createRuntime(dir, { rawTraceSpoolDir: spoolDir });
try {
runtime.enqueueRecord(createRecord("file-backed-base64-image"));
const result = runtime.enqueueRawTrace({
requestBodySizeBytes: Buffer.byteLength(body),
requestId: "file-backed-base64-image",
statusCode: 200
}, {
cleanupDirectory: bundleDir,
maxBodyBytes: 50 * 1024 * 1024,
requestBody: {
contentType: "application/json",
filePath: requestFile,
sizeBytes: Buffer.byteLength(body)
}
});
assert.equal(result.accepted, true);
await runtime.flush({ timeoutMs: 10_000 });
const page = await runtime.list({ pageSize: 25 });
const detail = await runtime.getDetail({ id: page.items[0].id });
const captured = JSON.parse(detail.requestBody.text);
assert.equal(detail.requestBody.sizeBytes, Buffer.byteLength(body));
assert.equal(detail.requestBody.truncated, true);
assert.match(captured.image_url.url, /^data:image\/jpeg;base64,\[base64 image omitted from log;/);
assert.equal(existsSync(bundleDir), false);
} finally {
await runtime.close({ timeoutMs: 5_000 });
rmSync(dir, { force: true, recursive: true });
}
});
test("RequestLogRuntime does not let a source-truncated raw image overwrite complete JSON", async () => {
const dir = mkdtempSync(path.join(tmpdir(), "ccr-request-log-runtime-partial-raw-base64-test-"));
const spoolDir = path.join(dir, "raw-trace-spool");
const bundleDir = path.join(spoolDir, "bundle");
const requestFile = path.join(bundleDir, "upstream_request.json");
mkdirSync(bundleDir, { recursive: true });
const fullBody = Buffer.from(JSON.stringify({
image_url: { url: `data:image/png;base64,${"A".repeat(1024 * 1024)}` },
model: "model-after-large-image"
}));
const sourceLimit = 512 * 1024;
writeFileSync(requestFile, fullBody.subarray(0, sourceLimit));
const runtime = createRuntime(dir, { rawTraceSpoolDir: spoolDir });
try {
runtime.enqueueRecord({
...createRecord("partial-file-backed-base64-image"),
maxBodyBytes: sourceLimit,
model: "model-after-large-image",
requestBody: fullBody
});
const result = runtime.enqueueRawTrace({
requestBodySizeBytes: fullBody.byteLength,
requestBodyTruncated: true,
requestId: "partial-file-backed-base64-image",
statusCode: 200
}, {
cleanupDirectory: bundleDir,
maxBodyBytes: sourceLimit,
requestBody: {
contentType: "application/json",
filePath: requestFile,
sizeBytes: fullBody.byteLength,
truncated: true
}
});
assert.equal(result.accepted, true);
await runtime.flush({ timeoutMs: 10_000 });
const page = await runtime.list({ pageSize: 25 });
const detail = await runtime.getDetail({ id: page.items[0].id });
const captured = JSON.parse(detail.requestBody.text);
assert.equal(captured.model, "model-after-large-image");
assert.match(captured.image_url.url, /^data:image\/png;base64,\[base64 image omitted from log;/);
assert.equal(detail.requestBody.sizeBytes, fullBody.byteLength);
} finally {
await runtime.close({ timeoutMs: 5_000 });
rmSync(dir, { force: true, recursive: true });
}
});
test("RequestLogRuntime keeps raw trace files until the writer ACK is received", async () => {
const dir = mkdtempSync(path.join(tmpdir(), "ccr-request-log-runtime-ack-cleanup-test-"));
const spoolDir = path.join(dir, "raw-trace-spool");
const bundleDir = path.join(spoolDir, "bundle");
const responseFile = path.join(bundleDir, "response.txt");
const delayedWorker = path.join(dir, "delayed-ack-worker.cjs");
mkdirSync(bundleDir, { recursive: true });
writeFileSync(responseFile, "body");
writeFileSync(delayedWorker, `
const { parentPort } = require("node:worker_threads");
parentPort.postMessage({ type: "ready" });
parentPort.on("message", (message) => {
if (message.type === "batch") {
setTimeout(() => parentPort.postMessage({ batchId: message.batchId, type: "ack" }), 100);
return;
}
parentPort.postMessage({ requestId: message.requestId, result: true, type: "response" });
if (message.method === "shutdown") parentPort.close();
});
`);
const runtime = createRequestLogRuntime({
batchMaxWaitMs: 1,
dbFile: path.join(dir, "request-logs.sqlite"),
rawTraceSpoolDir: spoolDir,
workerFile: delayedWorker
});
try {
runtime.enqueueRawTrace({ requestId: "delayed-ack" }, {
cleanupDirectory: bundleDir,
maxBodyBytes: 512 * 1024,
responseBody: { filePath: responseFile, sizeBytes: 4 }
});
await new Promise((resolve) => setTimeout(resolve, 25));
assert.equal(existsSync(bundleDir), true);
const flush = await runtime.flush({ timeoutMs: 5_000 });
assert.equal(flush.timedOut, false);
assert.equal(existsSync(bundleDir), false);
} finally {
await runtime.close({ timeoutMs: 5_000 });
rmSync(dir, { force: true, recursive: true });
}
});
test("RequestLogRuntime replays a committed raw trace idempotently when its body file is already missing", async () => {
const dir = mkdtempSync(path.join(tmpdir(), "ccr-request-log-runtime-raw-replay-test-"));
const spoolDir = path.join(dir, "raw-trace-spool");
const bundleDir = path.join(spoolDir, "bundle");
mkdirSync(bundleDir, { recursive: true });
const runtime = createRuntime(dir, { rawTraceSpoolDir: spoolDir });
try {
runtime.enqueueRecord(createRecord("missing-raw-file"));
const result = runtime.enqueueRawTrace({
model: "metadata-survives-replay",
requestId: "missing-raw-file",
statusCode: 200
}, {
cleanupDirectory: bundleDir,
maxBodyBytes: 512 * 1024,
responseBody: {
filePath: path.join(bundleDir, "already-removed.txt"),
sizeBytes: 1024
}
});
assert.equal(result.accepted, true);
const flush = await runtime.flush({ timeoutMs: 10_000 });
assert.equal(flush.timedOut, false);
assert.equal(runtime.metrics().writerRestarts, 0);
assert.equal(existsSync(bundleDir), false);
const page = await runtime.list({ pageSize: 25 });
assert.equal(page.items[0].model, "metadata-survives-replay");
} finally {
await runtime.close({ timeoutMs: 5_000 });
rmSync(dir, { force: true, recursive: true });
}
});
test("RequestLogRuntime retains raw trace bundles when enqueue is rejected", async () => {
const dir = mkdtempSync(path.join(tmpdir(), "ccr-request-log-runtime-raw-reject-test-"));
const spoolDir = path.join(dir, "raw-trace-spool");
const bundleDir = path.join(spoolDir, "bundle");
const responseFile = path.join(bundleDir, "response.txt");
mkdirSync(bundleDir, { recursive: true });
writeFileSync(responseFile, "body");
const runtime = createRequestLogRuntime({
dbFile: path.join(dir, "request-logs.sqlite"),
queueMaxBytes: 1,
queueMaxItems: 10,
rawTraceSpoolDir: spoolDir,
workerFile
});
try {
const result = runtime.enqueueRawTrace({ requestId: "rejected-raw-body" }, {
cleanupDirectory: bundleDir,
responseBody: { filePath: responseFile, sizeBytes: 4 }
});
assert.equal(result.accepted, false);
assert.equal(result.reason, "queue_full");
assert.equal(existsSync(bundleDir), true);
} finally {
await runtime.close({ timeoutMs: 5_000 });
rmSync(dir, { force: true, recursive: true });
}
});
test("RequestLogRuntime accounts bounded file-backed raw bodies against queue memory", async () => {
const dir = mkdtempSync(path.join(tmpdir(), "ccr-request-log-runtime-raw-queue-test-"));
const spoolDir = path.join(dir, "raw-trace-spool");
const bundleDir = path.join(spoolDir, "bundle");
const responseFile = path.join(bundleDir, "response.txt");
mkdirSync(bundleDir, { recursive: true });
writeFileSync(responseFile, "q".repeat(128 * 1024));
const runtime = createRequestLogRuntime({
dbFile: path.join(dir, "request-logs.sqlite"),
queueMaxBytes: 64 * 1024,
queueMaxItems: 10,
rawTraceSpoolDir: spoolDir,
workerFile
});
try {
const result = runtime.enqueueRawTrace({ requestId: "raw-file-over-queue-budget" }, {
cleanupDirectory: bundleDir,
maxBodyBytes: 128 * 1024,
responseBody: { filePath: responseFile, sizeBytes: 128 * 1024 }
});
assert.equal(result.accepted, false);
assert.equal(result.reason, "queue_full");
assert.equal(existsSync(bundleDir), true);
} finally {
await runtime.close({ timeoutMs: 5_000 });
rmSync(dir, { force: true, recursive: true });
}
});
test("RequestLogRuntime default queue accepts one maximum request and response raw trace", async () => {
const dir = mkdtempSync(path.join(tmpdir(), "ccr-request-log-runtime-max-raw-event-test-"));
const spoolDir = path.join(dir, "raw-trace-spool");
const bundleDir = path.join(spoolDir, "bundle");
const requestFile = path.join(bundleDir, "request.txt");
const responseFile = path.join(bundleDir, "response.txt");
mkdirSync(bundleDir, { recursive: true });
writeFileSync(requestFile, "request");
writeFileSync(responseFile, "response");
const runtime = createRequestLogRuntime({
dbFile: path.join(dir, "request-logs.sqlite"),
rawTraceSpoolDir: spoolDir,
workerFile
});
try {
const result = runtime.enqueueRawTrace({ requestId: "maximum-raw-event" }, {
cleanupDirectory: bundleDir,
maxBodyBytes: 50 * 1024 * 1024,
requestBody: { filePath: requestFile, sizeBytes: 50 * 1024 * 1024 },
responseBody: { filePath: responseFile, sizeBytes: 50 * 1024 * 1024 }
});
assert.equal(result.accepted, true);
assert.ok(runtime.metrics().queueBytes >= 100 * 1024 * 1024);
const flush = await runtime.flush({ timeoutMs: 10_000 });
assert.equal(flush.timedOut, false);
} finally {
await runtime.close({ timeoutMs: 5_000 });
rmSync(dir, { force: true, recursive: true });
}
});
test("RequestLogRuntime compacts Base64 images before applying queue and body limits", async () => {
const dir = mkdtempSync(path.join(tmpdir(), "ccr-request-log-runtime-base64-image-test-"));
const runtime = createRuntime(dir, { queueMaxBytes: 64 * 1024 });
try {
const image = "A".repeat(512 * 1024);
const body = Buffer.from(JSON.stringify({
messages: [{
content: [{
source: { data: image, media_type: "image/png", type: "base64" },
type: "image"
}],
role: "user"
}],
model: "vision-model"
}));
const result = runtime.enqueueRecord({
...createRecord("base64-image-request"),
maxBodyBytes: 50 * 1024 * 1024,
model: "vision-model",
requestBody: body
});
assert.equal(result.accepted, true);
assert.equal(result.degraded, true);
await runtime.flush({ timeoutMs: 10_000 });
const page = await runtime.list({ pageSize: 25 });
const detail = await runtime.getDetail({ id: page.items[0].id });
const captured = JSON.parse(detail.requestBody.text);
assert.equal(detail.requestBody.sizeBytes, body.byteLength);
assert.equal(detail.requestBody.truncated, true);
assert.match(captured.messages[0].content[0].source.data, /^\[base64 image omitted from log;/);
assert.ok(Buffer.byteLength(detail.requestBody.text) < 2 * 1024);
} finally {
await runtime.close({ timeoutMs: 5_000 });
rmSync(dir, { force: true, recursive: true });
}
});
test("RequestLogStore deduplicates replayed writer events", async () => {
const dir = mkdtempSync(path.join(tmpdir(), "ccr-request-log-runtime-replay-test-"));
const store = new RequestLogStore(path.join(dir, "request-logs.sqlite"));
try {
const command = {
eventId: "stable-event-id",
input: createRecord("deduplicated-request"),
kind: "record",
sequence: 1
};
await store.writeBatch([command, { ...command, sequence: 2 }]);
const page = await store.list({ pageSize: 25 });
assert.equal(page.items.length, 1);
} finally {
await store.close();
rmSync(dir, { force: true, recursive: true });
}
});
function createRuntime(dir, options = {}) {
return createRequestLogRuntime({
batchMaxItems: 10,
batchMaxWaitMs: 1,
dbFile: path.join(dir, "request-logs.sqlite"),
queueMaxBytes: 8 * 1024 * 1024,
queueMaxItems: 100,
...options,
workerFile
});
}
function createRecord(requestId) {
const now = new Date().toISOString();
return {
completedAt: now,
durationMs: 10,
method: "POST",
model: "worker-model",
path: "/v1/messages",
providerName: "worker-provider",
requestBody: Buffer.from(JSON.stringify({ model: "worker-model", stream: true })),
requestHeaders: { "content-type": "application/json" },
requestId,
responseBodyText: "{}",
responseHeaders: { "content-type": "application/json" },
startedAt: now,
statusCode: 200,
url: "http://127.0.0.1:3456/v1/messages"
};
}

View file

@ -7,6 +7,7 @@ import test from "node:test";
import { promisify } from "node:util";
import { getHeapStatistics } from "node:v8";
import { RequestLogStore } from "@ccr/core/observability/request-log-store.ts";
import { RequestRouteTraceRecorder } from "@ccr/core/observability/route-trace.ts";
import { createBetterSqliteDatabase } from "@ccr/core/storage/sqlite-native.ts";
const execFileAsync = promisify(execFile);
@ -146,6 +147,75 @@ test("RequestLogStore keeps list rows lightweight and detail rows complete", asy
}
});
test("RequestLogStore persists actively reported route hops without synthesizing Core diffs", async () => {
const dir = mkdtempSync(path.join(tmpdir(), "ccr-request-route-trace-test-"));
try {
const store = new RequestLogStore(path.join(dir, "request-logs.sqlite"));
const startedAtMs = Date.now();
const startedAt = new Date(startedAtMs).toISOString();
const recorder = new RequestRouteTraceRecorder(startedAtMs);
recorder.captureIngress();
recorder.capture({
changes: [{
after: "routed-model",
before: "request-model",
operation: "replace",
path: "/body/model",
scope: "body"
}],
decision: { policyId: "rule:test", reason: "test-rule", source: "rule" },
name: "router.policy",
phase: "routing",
target: { model: "routed-model", provider: "test-provider" }
});
await store.record({
completedAt: startedAt,
durationMs: 20,
method: "POST",
path: "/v1/messages",
providerName: "test-provider",
requestBody: Buffer.from(JSON.stringify({ model: "routed-model", stream: true })),
requestHeaders: { "content-type": "application/json" },
requestId: "route-trace-request",
responseBodyText: "{}",
responseHeaders: { "content-type": "application/json" },
routeTrace: recorder.finish(),
startedAt,
statusCode: 200,
url: "http://127.0.0.1:3456/v1/messages"
});
const pageBeforeRawTrace = await store.list({ pageSize: 25 });
assert.equal(pageBeforeRawTrace.items[0].routeHopCount, 2);
assert.equal(pageBeforeRawTrace.items[0].routeAttemptCount, 0);
assert.equal(pageBeforeRawTrace.items[0].routeTrace, undefined);
const applied = await store.updateFromRawTrace({
method: "POST",
model: "wire-model",
path: "/v1/messages",
provider: "wire-provider",
requestBodyContentType: "application/json",
requestBodyText: JSON.stringify({ model: "wire-model", stream: true }),
requestHeaders: { "content-type": "application/json", "x-wire-header": "yes" },
requestId: "route-trace-request",
statusCode: 200,
url: "https://upstream.example/v1/messages"
});
assert.equal(applied, true);
const detail = await store.getDetail({ id: pageBeforeRawTrace.items[0].id });
assert.ok(detail?.routeTrace);
assert.equal(detail.routeTrace.version, 2);
assert.equal(detail.routeTrace.hopCount, 2);
assert.equal(detail.routeTrace.hops.at(-1).name, "router.policy");
assert.ok(detail.routeTrace.hops.at(-1).changes.some((change) => change.path === "/body/model"));
} finally {
rmSync(dir, { force: true, recursive: true });
}
});
test("RequestLogStore redacts secrets and records CCR metadata", async () => {
const dir = mkdtempSync(path.join(tmpdir(), "ccr-request-log-metadata-test-"));
try {

View file

@ -23,6 +23,10 @@ test("core gateway disables the full-trace billing webhook without disabling raw
assert.deepEqual(compiled.billingWebhook, { enabled: false });
assert.deepEqual(compiled.billingQueue, { enabled: false });
assert.deepEqual(compiled.billing, { enabled: true });
const upstreamHeaderSanitizer = compiled.plugins?.at(-1);
assert.equal(upstreamHeaderSanitizer?.enabled, true);
assert.equal(upstreamHeaderSanitizer?.key, "ccr-upstream-header-sanitizer");
assert.match(upstreamHeaderSanitizer?.modulePath, /upstream-header-sanitizer\.js$/);
assert.equal(compiled.rawTrace?.enabled, true);
assert.deepEqual(compiled.rawTrace?.sync, {
enabled: true,

View file

@ -40,3 +40,27 @@ test("upstream proxy config changes restart the gateway runtime", () => {
assert.equal(shouldRestartGatewayForRuntimeConfigChange(previous, next), true);
});
test("raw trace observability config changes restart the gateway runtime", () => {
const mutations = [
(config) => { config.observability.requestLogs = !config.observability.requestLogs; },
(config) => { config.observability.agentAnalysis = !config.observability.agentAnalysis; },
(config) => { config.observability.requestLogBodyCapture = "none"; },
(config) => { config.observability.requestLogMaxBodyBytes = 4 * 1024 * 1024; }
];
for (const mutate of mutations) {
const previous = createDefaultAppConfig({ generatedConfigFile: "/tmp/ccr-gateway.config.json" });
const next = createDefaultAppConfig({ generatedConfigFile: "/tmp/ccr-gateway.config.json" });
mutate(next);
assert.equal(shouldRestartGatewayForRuntimeConfigChange(previous, next), true);
}
});
test("main-process-only observability changes do not restart the gateway runtime", () => {
const previous = createDefaultAppConfig({ generatedConfigFile: "/tmp/ccr-gateway.config.json" });
const next = createDefaultAppConfig({ generatedConfigFile: "/tmp/ccr-gateway.config.json" });
next.observability.requestLogSuccessSampleRate = 0.25;
assert.equal(shouldRestartGatewayForRuntimeConfigChange(previous, next), false);
});

View file

@ -1,6 +1,7 @@
import assert from "node:assert/strict";
import test from "node:test";
import { ClaudeCodeRouterPlugin } from "@ccr/core/gateway/claude-code-router-plugin.ts";
import { RequestRouteTraceRecorder } from "@ccr/core/observability/route-trace.ts";
import {
fallbackRetryDelayAfterNetworkErrorForTest,
fallbackRetryDelayAfterStatusForTest,
@ -872,6 +873,8 @@ test("router rules can add headers after the built-in Claude Code profile route"
const headers = {
"user-agent": "Claude Code"
};
const traceRecorder = new RequestRouteTraceRecorder(Date.now());
traceRecorder.captureIngress();
const result = await plugin.routeRequest({
body: {
messages: [],
@ -879,6 +882,7 @@ test("router rules can add headers after the built-in Claude Code profile route"
},
headers,
method: "POST",
trace: traceRecorder,
url: "/v1/messages"
});
@ -886,6 +890,13 @@ test("router rules can add headers after the built-in Claude Code profile route"
assert.equal(result.body.model, "Provider/claude-sonnet");
assert.equal(result.decision.model, "Provider/claude-sonnet");
assert.equal(result.decision.reason, "rule:target-provider");
const rewriteHop = traceRecorder.finish().hops.find((hop) => hop.name === "router.rewrite:set:request.header.x-target-provider");
assert.deepEqual(rewriteHop?.changes, [{
after: "Provider",
operation: "add",
path: "/headers/x-target-provider",
scope: "headers"
}]);
});
test("router rules override explicit provider model requests", async () => {
@ -935,6 +946,8 @@ test("router rules override explicit provider model requests", async () => {
const headers = {
"anthropic-background": "true"
};
const traceRecorder = new RequestRouteTraceRecorder(Date.now());
traceRecorder.captureIngress();
const result = await plugin.routeRequest({
body: {
messages: [],
@ -942,6 +955,7 @@ test("router rules override explicit provider model requests", async () => {
},
headers,
method: "POST",
trace: traceRecorder,
url: "/v1/messages"
});
@ -950,6 +964,14 @@ test("router rules override explicit provider model requests", async () => {
assert.equal(result.decision.model, "Qwen3-235B-A22B");
assert.equal(result.decision.reason, "rule:background");
assert.deepEqual(result.decision.fallback, ruleFallback);
const modelRewriteHop = traceRecorder.finish().hops.find((hop) => hop.name === "router.rewrite:set:request.body.model");
assert.deepEqual(modelRewriteHop?.changes, [{
after: "Qwen3-235B-A22B",
before: "OpenCode/deepseek-v4-flash",
operation: "replace",
path: "/body/model",
scope: "body"
}]);
});
test("issue 1480 raw user config no longer reproduces the Claude Code profile routing failure", async () => {

View file

@ -0,0 +1,45 @@
import assert from "node:assert/strict";
import test from "node:test";
import {
createGatewayPlugin,
sanitizeUpstreamProviderHeaders
} from "@ccr/core/gateway/core-runtime/upstream-header-sanitizer.ts";
test("provider boundary removes CCR-owned headers and preserves provider headers", () => {
assert.deepEqual(sanitizeUpstreamProviderHeaders({
authorization: "Bearer provider-token",
"X-Auth-API-Key-ID": "profile:claude",
"x-auth-sub": "profile:claude",
"x-auth-token": "provider-specific-token",
"x-ccr-core-auth": "core-secret",
"X-CCR-Route-Reason": "rule:claude",
"x-client-request-id": "request-1"
}), {
authorization: "Bearer provider-token",
"x-auth-token": "provider-specific-token",
"x-client-request-id": "request-1"
});
});
test("gateway sanitizer hook runs on the final upstream request shape", async () => {
const [hook] = createGatewayPlugin().providerHooks;
const upstreamRequest = {
body: { model: "provider-model" },
headers: {
"content-type": "application/json",
"x-ccr-provider-credential-id": "credential-id",
"x-auth-api-key-id": "profile:codex"
},
method: "POST",
url: "https://provider.example/v1/responses"
};
const result = await hook.transformRequest({ upstreamRequest });
assert.equal(result.ok, true);
assert.deepEqual(result.value, {
...upstreamRequest,
headers: { "content-type": "application/json" }
});
assert.equal(upstreamRequest.headers["x-ccr-provider-credential-id"], "credential-id");
});

View file

@ -0,0 +1,167 @@
import assert from "node:assert/strict";
import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import { Readable } from "node:stream";
import test from "node:test";
import { createDefaultAppConfig } from "@ccr/core/config/default-config.ts";
import { rawTraceSyncHeader } from "@ccr/core/gateway/internal/shared.ts";
import { rawTraceHardMaxBodyBytes } from "@ccr/core/observability/request-log-limits.ts";
import {
applyRawTraceRequestLogPolicy,
buildRawTraceConfig,
createBodySampler,
RawTraceSynchronizer
} from "@ccr/core/observability/raw-trace-sync.ts";
test("raw trace applies metadata-only body privacy while retaining original sizes", () => {
const config = createConfig();
config.observability.requestLogBodyCapture = "none";
const policy = applyRawTraceRequestLogPolicy(config, {
requestBodySizeBytes: 1_024,
requestId: "privacy-request",
responseBodySizeBytes: 2_048,
statusCode: 200
});
assert.equal(policy.action, "enqueue");
assert.equal(policy.captureBodies, false);
assert.equal(policy.update.requestBodyText, "");
assert.equal(policy.update.requestBodySizeBytes, 1_024);
assert.equal(policy.update.requestBodyTruncated, true);
assert.equal(policy.update.responseBodyText, "");
assert.equal(policy.update.responseBodySizeBytes, 2_048);
assert.equal(policy.update.responseBodyTruncated, true);
});
test("raw trace uses the same successful-request sampling policy as normal request logs", () => {
const config = createConfig();
config.observability.requestLogSuccessSampleRate = 0;
assert.deepEqual(applyRawTraceRequestLogPolicy(config, {
requestBodySizeBytes: 128,
requestId: "sampled-request",
statusCode: 200
}), { action: "discard", reason: "sampled" });
const errorPolicy = applyRawTraceRequestLogPolicy(config, {
requestBodySizeBytes: 128,
requestId: "error-request",
statusCode: 429
});
assert.equal(errorPolicy.action, "enqueue");
});
test("raw trace captures complete error bodies but suppresses successful bodies in errors-only mode", () => {
const config = createConfig();
config.observability.requestLogBodyCapture = "errors";
const successful = applyRawTraceRequestLogPolicy(config, {
requestBodySizeBytes: 64,
requestId: "successful-request",
statusCode: 200
});
assert.equal(successful.action, "enqueue");
assert.equal(successful.captureBodies, false);
const failed = applyRawTraceRequestLogPolicy(config, {
requestBodySizeBytes: 64,
requestId: "failed-request",
statusCode: 500
});
assert.equal(failed.action, "enqueue");
assert.equal(failed.captureBodies, true);
});
test("raw trace source defaults to the 50 MB hard body ceiling", () => {
const config = createConfig();
const previous = process.env.CCR_RAW_TRACE_ENABLED;
process.env.CCR_RAW_TRACE_ENABLED = "1";
try {
const rawTrace = buildRawTraceConfig(config, "sync-token");
assert.equal(rawTrace.maxPartBytes, rawTraceHardMaxBodyBytes);
config.observability.requestLogMaxBodyBytes = Number.MAX_SAFE_INTEGER;
assert.equal(buildRawTraceConfig(config, "sync-token").maxPartBytes, rawTraceHardMaxBodyBytes);
} finally {
if (previous === undefined) delete process.env.CCR_RAW_TRACE_ENABLED;
else process.env.CCR_RAW_TRACE_ENABLED = previous;
}
});
test("raw trace capture is disabled at the source for metadata-only logging", () => {
const config = createConfig();
config.observability.requestLogBodyCapture = "none";
const previous = process.env.CCR_RAW_TRACE_ENABLED;
process.env.CCR_RAW_TRACE_ENABLED = "1";
try {
assert.equal(buildRawTraceConfig(config, "sync-token").enabled, false);
config.observability.requestLogBodyCapture = "all";
config.observability.requestLogMaxBodyBytes = 0;
assert.equal(buildRawTraceConfig(config, "sync-token").enabled, false);
} finally {
if (previous === undefined) delete process.env.CCR_RAW_TRACE_ENABLED;
else process.env.CCR_RAW_TRACE_ENABLED = previous;
}
});
test("stream sampler retains the original response byte size after capture truncation", () => {
const sampler = createBodySampler();
const body = Buffer.alloc(9 * 1024 * 1024, "x");
sampler.append(body);
assert.equal(sampler.isTruncated(), true);
assert.equal(Buffer.byteLength(sampler.read()), 8 * 1024 * 1024);
assert.equal(sampler.sizeBytes(), body.byteLength);
});
test("raw trace sync retains its bundle when the asynchronous queue rejects it", async () => {
const dir = mkdtempSync(path.join(tmpdir(), "ccr-raw-trace-sync-reject-test-"));
const spoolDirectory = path.join(dir, "spool");
const bundleDirectory = path.join(spoolDirectory, "bundle");
const bodyFile = path.join(bundleDirectory, "upstream_request.json");
mkdirSync(bundleDirectory, { recursive: true });
writeFileSync(bodyFile, '{"message":"keep me"}');
const synchronizer = new RawTraceSynchronizer({
enqueueUpdate: async () => false,
getConfig: createConfig,
spoolDirectory
});
const request = Readable.from([JSON.stringify({
parts: [{
contentType: "application/json",
filePath: bodyFile,
originalBytes: 21,
partType: "upstream_request"
}],
turnKey: "queue-rejected-request"
})]);
request.method = "POST";
request.headers = { [rawTraceSyncHeader]: synchronizer.token };
const result = {};
const response = {
end(body) {
result.body = JSON.parse(body);
},
writeHead(statusCode) {
result.statusCode = statusCode;
}
};
try {
await synchronizer.handle(request, response);
assert.equal(result.statusCode, 503);
assert.equal(result.body.accepted, false);
assert.equal(existsSync(bundleDirectory), true);
} finally {
rmSync(dir, { force: true, recursive: true });
}
});
function createConfig() {
const config = createDefaultAppConfig({ generatedConfigFile: "/tmp/ccr-generated-config.json" });
config.observability.requestLogs = true;
config.observability.requestLogBodyCapture = "all";
config.observability.requestLogSuccessSampleRate = 1;
return config;
}

View file

@ -0,0 +1,65 @@
import assert from "node:assert/strict";
import test from "node:test";
import { compactBase64ImagePayloads } from "@ccr/core/observability/request-log-body.ts";
const largeBase64 = "A".repeat(64 * 1024);
test("request log body compaction replaces data URL images without parsing the JSON tree", () => {
const input = Buffer.from(JSON.stringify({
image_url: { url: `data:image/png;base64,${largeBase64}` },
model: "vision-model"
}));
const compacted = compactBase64ImagePayloads(input);
const parsed = JSON.parse(compacted.buffer.toString("utf8"));
assert.equal(compacted.compacted, true);
assert.equal(compacted.imageCount, 1);
assert.equal(compacted.omittedBytes, largeBase64.length);
assert.equal(parsed.model, "vision-model");
assert.match(parsed.image_url.url, /^data:image\/png;base64,\[base64 image omitted from log;/);
assert.ok(compacted.buffer.byteLength < 512);
});
test("request log body compaction recognizes Anthropic, Gemini, and OpenAI image payloads", () => {
const input = Buffer.from(JSON.stringify({
anthropic: {
source: { data: largeBase64, media_type: "image/jpeg", type: "base64" }
},
gemini: {
inline_data: { data: largeBase64, mime_type: "image/webp" }
},
openai: {
b64_json: largeBase64
}
}));
const compacted = compactBase64ImagePayloads(input);
const parsed = JSON.parse(compacted.buffer.toString("utf8"));
assert.equal(compacted.imageCount, 3);
assert.match(parsed.anthropic.source.data, /^\[base64 image omitted from log;/);
assert.match(parsed.gemini.inline_data.data, /^\[base64 image omitted from log;/);
assert.match(parsed.openai.b64_json, /^\[base64 image omitted from log;/);
});
test("request log body compaction leaves small and non-image Base64 values untouched", () => {
const input = Buffer.from(JSON.stringify({
audio: { data: largeBase64, media_type: "audio/wav", type: "base64" },
image: `data:image/png;base64,${"A".repeat(1024)}`
}));
const compacted = compactBase64ImagePayloads(input);
assert.equal(compacted.compacted, false);
assert.strictEqual(compacted.buffer, input);
});
test("request log body compaction does not borrow image context from an adjacent object", () => {
const audioBase64 = "A".repeat(64 * 1024);
const input = Buffer.from(JSON.stringify([
{ mime_type: "image/png", type: "base64" },
{ data: audioBase64, media_type: "audio/wav", type: "base64" }
]));
const compacted = compactBase64ImagePayloads(input);
assert.equal(compacted.compacted, false);
assert.equal(JSON.parse(compacted.buffer.toString("utf8"))[1].data, audioBase64);
});

View file

@ -0,0 +1,60 @@
import assert from "node:assert/strict";
import test from "node:test";
import { RequestRouteTraceRecorder } from "@ccr/core/observability/route-trace.ts";
test("route trace records actively reported changes and never persists sensitive values", () => {
const startedAt = Date.now();
const recorder = new RequestRouteTraceRecorder(startedAt);
recorder.captureIngress();
recorder.capture({
changes: [
{ after: "model-b", before: "model-a", operation: "replace", path: "/body/model", scope: "body" },
{ after: "body-secret-b", before: "body-secret-a", operation: "replace", path: "/body/api_key", scope: "body" },
{ after: "Bearer header-secret-b", before: "Bearer header-secret-a", operation: "replace", path: "/headers/authorization", scope: "headers" },
{
after: "https://upstream.example/v1/messages?access_token=url-secret-b",
before: "http://127.0.0.1/v1/messages?access_token=url-secret-a",
operation: "replace",
path: "/url",
scope: "url"
}
],
decision: { policyId: "rule:test", reason: "unit-test", source: "rule" },
kind: "mutation",
name: "router.policy",
phase: "routing",
target: { model: "model-b", provider: "provider-b" }
});
const trace = recorder.finish();
const serialized = JSON.stringify(trace);
assert.equal(trace.hopCount, 2);
assert.equal(trace.version, 2);
assert.equal(trace.ingressSnapshot, undefined);
assert.equal(trace.finalSnapshot, undefined);
assert.equal(trace.hops[1].decision.policyId, "rule:test");
assert.ok(trace.hops[1].changes.some((change) => change.path === "/body/model"));
assert.ok(trace.hops[1].changes.some((change) => change.path === "/headers/authorization" && change.redacted));
assert.match(serialized, /\[redacted\]/);
assert.doesNotMatch(serialized, /header-secret|body-secret|url-secret/);
});
test("route trace bounds actively reported values without parsing request bodies", () => {
const recorder = new RequestRouteTraceRecorder(Date.now());
const largeBody = Buffer.alloc(512 * 1024, "x");
recorder.captureIngress();
for (let index = 0; index < 200; index += 1) {
recorder.capture({
changes: [{ after: largeBody, operation: "replace", path: "/body", scope: "body" }],
name: `hop-${index}`,
phase: "routing"
});
}
const trace = recorder.finish();
assert.ok(trace.hopCount <= 64);
assert.equal(trace.truncated, true);
assert.deepEqual(trace.hops[1].changes[0].after, { sizeBytes: largeBody.byteLength, type: "buffer" });
assert.doesNotMatch(JSON.stringify(trace), /xxxxxxxx/);
assert.ok(Buffer.byteLength(JSON.stringify(trace)) <= 256 * 1024 + 4096);
});

View file

@ -15,6 +15,7 @@ import { browserAutomationMcpService } from "./browser-automation-mcp";
import { browserWebSearchMcpService } from "./electron-web-search-mcp";
import { applyNativeThemePreference } from "./native-theme";
import windowsManager from "./windows";
import { closeRequestLogRuntime } from "@ccr/core/observability/request-log-store";
const gotTheLock = app.requestSingleInstanceLock();
const quitProxyRestoreTimeoutMs = 30_000;
@ -179,6 +180,9 @@ function stopServicesForQuit(): Promise<void> {
console.error(`Failed to stop services before quit: ${formatError(error)}`);
})
.finally(async () => {
await closeRequestLogRuntime().catch((error) => {
console.error(`Failed to flush request logs before quit: ${formatError(error)}`);
});
try {
const config = await loadAppConfig();
for (const status of restoreGlobalProfileConfigsOnExit(config.profile.profiles)) {

View file

@ -1,9 +1,10 @@
import { memo } from "react";
import { Maximize2, X } from "lucide-react";
import type { RequestRouteTrace, RequestRouteTraceChange, RequestRouteTraceHop } from "@ccr/core/contracts/app";
import {
AnimatedIconSwap, Check, ChevronDown, ChevronLeft,
ChevronRight, clampNumber, clientInitial, cn, Copy, copyTextToClipboard,
Database, filterLogText, formatBytes, formatCompactNumber, formatDuration,
Database, Dialog, DialogBody, DialogContent, DialogHeader, DialogTitle, filterLogText, formatBytes, formatCompactNumber, formatDuration,
formatLogBodyView, formatLogDateTime, formatLogTokenSummary, formatNetworkRequestRaw, formatNetworkResponseRaw, formatUsdCost,
isJsonContainer, jsonChildPath, logRequestModel,
logResponseModel, logSelectOptions, motion, MoveRight, Network, networkCodeLabel,
@ -706,6 +707,7 @@ function LogExpandedDetails({
<LogMetric label={t("Cost")} value={formatUsdCost(entry.costUsd ?? 0)} />
</div>
{entry.retryAttempts.length > 0 ? <LogRetryAttempts attempts={entry.retryAttempts} /> : null}
{entry.routeTrace ? <LogRouteTrace trace={entry.routeTrace} /> : null}
{detailLoading || detailError ? (
<div className={cn("border-b px-3 py-2 text-[12px] font-semibold", detailError ? "network-error-box" : "network-body-meta")}>
{detailError || t("Loading full payload...")}
@ -726,6 +728,302 @@ function LogExpandedDetails({
);
}
function LogRouteTrace({ trace }: { trace: RequestRouteTrace }) {
const t = useAppText();
const [dialogOpen, setDialogOpen] = useState(false);
return (
<>
<div className="network-body-meta flex min-w-0 flex-wrap items-center justify-between gap-3 border-b px-3 py-2.5 text-[12px]">
<div className="min-w-0">
<div className="font-semibold">{t("Route trace")}</div>
<div className="network-muted mt-0.5 text-[11px]">
{trace.hopCount} {t("hops")} · {trace.attemptCount} {t("attempts")}
{trace.truncated ? ` · ${t("truncated")}` : ""}
</div>
</div>
<div className="flex shrink-0 items-center gap-2">
<span className={cn(
"rounded-full px-2 py-0.5 text-[10px] font-bold uppercase",
trace.complete ? "network-service-running" : "network-service-paused"
)}>
{trace.complete ? t("Complete") : t("Partial")}
</span>
<button
className="network-control-button flex h-8 items-center gap-2 rounded-md border px-3 text-[11px] font-semibold outline-none focus-visible:ring-2 focus-visible:ring-ring/30"
onClick={() => setDialogOpen(true)}
type="button"
>
<Maximize2 className="h-3.5 w-3.5" aria-hidden="true" />
{t("View route graph")}
</button>
</div>
</div>
{dialogOpen ? <LogRouteTraceDialog onClose={() => setDialogOpen(false)} trace={trace} /> : null}
</>
);
}
function LogRouteTraceDialog({
onClose,
trace
}: {
onClose: () => void;
trace: RequestRouteTrace;
}) {
const t = useAppText();
const [activeHopSequence, setActiveHopSequence] = useState<number>();
const activeHopIndex = trace.hops.findIndex((hop) => hop.seq === activeHopSequence);
const activeHop = activeHopIndex >= 0 ? trace.hops[activeHopIndex] : undefined;
useEffect(() => {
const closeOnEscape = (event: KeyboardEvent) => {
if (event.key === "Escape") onClose();
};
window.addEventListener("keydown", closeOnEscape);
return () => window.removeEventListener("keydown", closeOnEscape);
}, [onClose]);
return (
<Dialog className="items-start" onOpenChange={(open) => !open && onClose()} open>
<DialogContent className="h-[calc(100dvh-1.5rem)] max-w-[1240px] origin-top sm:h-[min(820px,calc(100dvh-3rem))]">
<DialogHeader>
<div className="min-w-0">
<DialogTitle>{t("Route graph")}</DialogTitle>
<div className="mt-0.5 flex min-w-0 flex-wrap items-center gap-2 text-[11px] text-muted-foreground">
<span>{trace.hopCount} {t("hops")}</span>
<span aria-hidden="true">·</span>
<span>{trace.attemptCount} {t("attempts")}</span>
{trace.truncated ? <><span aria-hidden="true">·</span><span>{t("truncated")}</span></> : null}
</div>
</div>
<button
aria-label={t("Close")}
className="network-control-button flex h-7 w-7 items-center justify-center rounded-md border outline-none focus-visible:ring-2 focus-visible:ring-ring/30"
onClick={onClose}
title={t("Close")}
type="button"
>
<X className="h-3.5 w-3.5" aria-hidden="true" />
</button>
</DialogHeader>
<DialogBody className="flex min-h-0 flex-col overflow-hidden p-0">
<div className="network-body-meta shrink-0 border-b px-4 py-3">
<div className="network-muted text-[11px] font-semibold">{t("Hover a node to inspect routing operations")}</div>
<div className="mt-3 overflow-x-auto rounded-md border border-[color:var(--network-border)] bg-card/40">
{trace.hops.length === 0 ? (
<div className="network-muted flex min-h-40 items-center justify-center px-4 text-[12px]">{t("No route activity")}</div>
) : (
<div className="flex min-w-max items-center px-5 py-6" role="list" aria-label={t("Route graph")}>
{trace.hops.map((hop, index) => (
<div className="flex items-center" key={`${hop.seq}-${hop.name}`} role="listitem">
{index > 0 ? (
<div className="flex w-12 shrink-0 items-center text-border" aria-hidden="true">
<span className="h-px flex-1 bg-current" />
<ChevronRight className="-ml-1 h-4 w-4" />
</div>
) : null}
<button
aria-label={`${t("Route node")} ${index + 1}: ${routeHopDisplayName(hop.name, t)}`}
className={cn(
"flex h-[116px] w-[184px] shrink-0 flex-col rounded-md border border-border bg-card p-3 text-left outline-none transition-colors focus-visible:ring-2 focus-visible:ring-ring/40",
activeHopSequence === hop.seq && "border-primary/60 bg-primary/5 ring-2 ring-primary/10"
)}
onFocus={() => setActiveHopSequence(hop.seq)}
onMouseEnter={() => setActiveHopSequence(hop.seq)}
type="button"
>
<span className="flex min-w-0 items-center gap-2">
<span className={cn(
"h-2.5 w-2.5 shrink-0 rounded-full",
hop.status === "error"
? "network-dot-error"
: hop.status === "noop"
? "network-service-muted"
: "network-dot-completed"
)} />
<span className="network-muted truncate text-[10px] font-bold uppercase">#{index + 1} · {t(hop.phase)}</span>
{hop.attempt ? <span className="network-muted ml-auto shrink-0 text-[10px]">A{hop.attempt}</span> : null}
</span>
<span className="mt-3 line-clamp-2 break-words text-[12px] font-bold leading-4">{routeHopDisplayName(hop.name, t)}</span>
<span className="network-muted mt-auto flex items-center justify-between gap-2 text-[10px]">
<span>+{formatDuration(hop.startedOffsetMs)}</span>
<span>{formatDuration(hop.durationMs)}</span>
</span>
</button>
</div>
))}
</div>
)}
</div>
</div>
<div className="min-h-0 flex-1 overflow-auto p-4" id="route-hop-details">
{activeHop ? (
<LogRouteHopDetails hop={activeHop} index={activeHopIndex} />
) : (
<div className="network-muted flex min-h-52 items-center justify-center rounded-md border border-dashed border-border px-4 text-center text-[12px]">
{t("Hover over a route node to inspect its operations.")}
</div>
)}
</div>
</DialogBody>
</DialogContent>
</Dialog>
);
}
function LogRouteHopDetails({ hop, index }: { hop: RequestRouteTraceHop; index: number }) {
const t = useAppText();
const target = routeHopTargetSummary(hop);
const outcome = routeHopOutcomeSummary(hop);
const explanation = [hop.decision?.source, hop.decision?.ruleName ?? hop.decision?.ruleId, hop.decision?.reason]
.filter(Boolean)
.join(" · ");
return (
<div className="min-w-0">
<div className="flex min-w-0 flex-wrap items-start justify-between gap-3 border-b border-border pb-3">
<div className="min-w-0">
<div className="flex min-w-0 flex-wrap items-center gap-2">
<span className="network-muted text-[10px] font-bold uppercase">{t("Route node")} #{index + 1}</span>
<span className="rounded bg-muted px-1.5 py-0.5 text-[10px] font-semibold uppercase text-muted-foreground">{t(hop.phase)}</span>
<span className={cn(
"rounded px-1.5 py-0.5 text-[10px] font-bold uppercase",
hop.status === "error" ? "network-state-pill-error" : hop.status === "noop" ? "network-service-muted" : "network-state-pill-completed"
)}>{t(hop.status)}</span>
{hop.attempt ? <span className="network-muted text-[10px]">{t("Attempt")} #{hop.attempt}</span> : null}
</div>
<div className="mt-1 break-words font-mono text-[13px] font-bold">{hop.name}</div>
</div>
<div className="network-muted shrink-0 text-right font-mono text-[10px]">
<div>+{formatDuration(hop.startedOffsetMs)}</div>
<div>{formatDuration(hop.durationMs)}</div>
</div>
</div>
{explanation || target || outcome ? (
<div className="grid gap-2 border-b border-border py-3 text-[11px] md:grid-cols-3">
{explanation ? <RouteHopDetail label={t("Decision")} value={explanation} /> : null}
{target ? <RouteHopDetail label={t("Target")} mono value={target} /> : null}
{outcome ? <RouteHopDetail danger={hop.status === "error"} label={t("Result")} value={outcome} /> : null}
</div>
) : null}
<div className="pt-3">
<div className="mb-2 text-[11px] font-semibold">{t("Routing operations")}</div>
{hop.changes.length > 0 ? (
<div className="overflow-x-auto rounded border border-[color:var(--network-border)]">
{hop.changes.map((change, changeIndex) => (
<LogRouteChange change={change} index={changeIndex} key={`${change.path}-${changeIndex}`} />
))}
</div>
) : <div className="network-muted rounded border border-dashed border-border px-3 py-4 text-[11px]">{t("No request fields changed")}</div>}
{hop.truncated ? <div className="network-service-paused mt-2 text-[10px]">{t("This hop was truncated")}</div> : null}
</div>
</div>
);
}
function RouteHopDetail({
danger,
label,
mono,
value
}: {
danger?: boolean;
label: string;
mono?: boolean;
value: string;
}) {
return (
<div className="min-w-0 rounded-md bg-muted/50 px-3 py-2">
<div className="network-muted text-[10px] font-semibold uppercase">{label}</div>
<div className={cn("mt-1 break-words text-[11px]", mono && "font-mono", danger && "network-error-text")}>{value}</div>
</div>
);
}
function routeHopDisplayName(name: string, t: (value: string) => string): string {
if (name.startsWith("router.rewrite:")) return t("Request rewrite");
const labels: Record<string, string> = {
"compatibility.codex-apply-patch": "Codex patch compatibility",
"compatibility.cursor-openai": "Cursor compatibility",
"custom-router": "Custom router",
"enrichment.hosted-web-search": "Web search enrichment",
"enrichment.web-search-continuation": "Web search continuation",
"fallback.execution-plan": "Execution plan",
"gateway.content-length-normalization": "Content length normalization",
"gateway.header-normalization": "Header normalization",
"model-discovery.claude-app": "Claude App model discovery",
"model-discovery.claude-code": "Claude Code model discovery",
"protocol-adapter.route-input": "Protocol adaptation",
"provider.capability-routing": "Capability routing",
"request.ingress": "Request ingress",
"router.model-selection": "Model selection",
"router.policy": "Route decision",
"router.route-output": "Route output",
"upstream.attempt.outcome": "Upstream result",
"upstream.attempt.prepare": "Prepare upstream"
};
return t(labels[name] ?? name);
}
function LogRouteChange({ change, index }: { change: RequestRouteTraceChange; index: number }) {
const t = useAppText();
return (
<div className={cn(
"grid min-w-[560px] grid-cols-[76px_minmax(160px,0.8fr)_minmax(0,1fr)_28px_minmax(0,1fr)] border-b last:border-b-0 text-[10px]",
index % 2 === 0 ? "network-kv-row-even" : "network-kv-row-odd"
)}>
<div className="border-r border-[color:var(--network-border)] px-2 py-1.5 font-bold uppercase">{t(change.operation)}</div>
<div className="border-r border-[color:var(--network-border)] px-2 py-1.5 font-mono" title={change.path}>{change.path}</div>
<RouteTraceChangeValue value={change.before} />
<div className="network-muted flex items-center justify-center border-r border-[color:var(--network-border)]">→</div>
<RouteTraceChangeValue value={change.after} />
</div>
);
}
function RouteTraceChangeValue({ value }: { value: unknown }) {
const text = formatRouteTraceValue(value);
return (
<div className="max-h-24 overflow-auto whitespace-pre-wrap break-all border-r border-[color:var(--network-border)] px-2 py-1.5 font-mono last:border-r-0" title={text}>
{text || "∅"}
</div>
);
}
function routeHopTargetSummary(hop: RequestRouteTraceHop): string {
const target = hop.target;
if (!target) return "";
return [
target.provider ? `provider=${target.provider}` : "",
target.model ? `model=${target.model}` : "",
target.protocol ? `protocol=${target.protocol}` : "",
target.credentialId ? `credential=${target.credentialId}` : "",
target.credentialCandidates?.length ? `candidates=${target.credentialCandidates.join(" > ")}` : ""
].filter(Boolean).join(" · ");
}
function routeHopOutcomeSummary(hop: RequestRouteTraceHop): string {
const outcome = hop.outcome;
if (!outcome) return "";
return [
outcome.statusCode ? `HTTP ${outcome.statusCode}` : "",
outcome.error ?? "",
outcome.fallbackReason ? `fallback=${outcome.fallbackReason}` : "",
outcome.retryDelayMs !== undefined ? `retry in ${formatDuration(outcome.retryDelayMs)}` : ""
].filter(Boolean).join(" · ");
}
function formatRouteTraceValue(value: unknown): string {
if (value === undefined) return "";
if (typeof value === "string") return value;
try {
return JSON.stringify(value, null, 2);
} catch {
return String(value);
}
}
function LogRetryAttempts({ attempts }: { attempts: RequestLogEntry["retryAttempts"] }) {
const t = useAppText();

View file

@ -446,6 +446,9 @@ export function normalizeObservabilityConfig(config: Partial<AppConfig["observab
...fallbackConfig.observability,
...(config || {}),
agentAnalysis: Boolean(config?.agentAnalysis),
requestLogBodyCapture: config?.requestLogBodyCapture ?? fallbackConfig.observability.requestLogBodyCapture,
requestLogMaxBodyBytes: config?.requestLogMaxBodyBytes ?? fallbackConfig.observability.requestLogMaxBodyBytes,
requestLogSuccessSampleRate: config?.requestLogSuccessSampleRate ?? fallbackConfig.observability.requestLogSuccessSampleRate,
requestLogs: Boolean(config?.requestLogs)
};
}

View file

@ -1068,7 +1068,55 @@ export const appCopy: Record<ResolvedLanguage, AppCopy> = {
"Ready to route": "可以开始路由",
"Restart proxy": "重启代理",
"Route": "路由",
"Route graph": "路由链路图",
"Route hops": "路由节点",
"Route node": "路由节点",
"Route trace": "路由轨迹",
"Route Observability": "路由可观测",
"View route graph": "查看路由链路图",
"Hover a node to inspect routing operations": "将鼠标悬停在节点上查看路由操作",
"Hover over a route node to inspect its operations.": "将鼠标悬停在路由节点上查看操作详情。",
"Routing operations": "路由操作",
"No request fields changed": "没有请求字段变更",
"This hop was truncated": "此路由节点已截断",
"Decision": "决策",
"Complete": "完整",
"Partial": "部分",
"hops": "个节点",
"attempts": "次尝试",
"ingress": "接入",
"compatibility": "兼容处理",
"routing": "路由",
"capability": "能力匹配",
"enrichment": "请求增强",
"planning": "执行规划",
"attempt": "调用尝试",
"core": "核心网关",
"outcome": "结果",
"error": "错误",
"noop": "无变更",
"Request ingress": "请求接入",
"Header normalization": "请求头规范化",
"Protocol adaptation": "协议适配",
"Route decision": "路由决策",
"Request rewrite": "请求改写",
"Model selection": "模型选择",
"Route output": "路由输出",
"Capability routing": "能力路由",
"Content length normalization": "正文长度规范化",
"Execution plan": "执行计划",
"Prepare upstream": "准备上游调用",
"Upstream result": "上游调用结果",
"Cursor compatibility": "Cursor 兼容处理",
"Codex patch compatibility": "Codex Patch 兼容处理",
"Claude Code model discovery": "Claude Code 模型发现",
"Claude App model discovery": "Claude App 模型发现",
"Web search enrichment": "网页搜索增强",
"Web search continuation": "网页搜索续接",
"Custom router": "自定义路由",
"add": "新增",
"replace": "替换",
"remove": "移除",
"Rules": "规则",
"Save": "保存",
"Screen lock": "锁屏",