mirror of
https://github.com/AgentSeal/codeburn.git
synced 2026-10-29 02:23:57 +00:00
22ebfd463e
* Write session shards one member per line A shard is still one JSON object, so older builds read it unchanged, but it is written through a stream and never exists as one string. Past V8's string limit the save used to throw and the cache silently never persisted; reads of such a shard now go member by member. * Hold cached files a ranged query cannot report on as stubs A ranged load streams its in-scope shards and keeps in full only the files with a call in the range, spawn anchors, failure markers, turn-less files and WSL paths. Every other file becomes a stub: its fingerprint, bucket, PR flag, dedup keys per turn and where its member line lives. A stub stands in for its entry everywhere but the summaries it cannot contribute to: reconcile, the seenMsgIds and parser dedup pre-seeds, the query-time key replay in serve order, orphan surfacing, eviction and the emptiness checks. A stub whose file changed loads its member first, so the append or re-parse runs as before, and a wider request in the same process loads the stubs it now needs. A bucket holding stubs is rewritten as a full load would write it, copying the stubs' lines from the shard they came from; unchanged, the published shard is kept. A single-line shard from an older build is rewritten on the next save. Providers whose parse reads cached entries no report shows (hermes, quickdesk, antigravity, devin, gemini) still load in full. * Discover codex projects from a small index Discovery only needs each rollout's project, but it loaded the whole codex results file (hundreds of MB on a large corpus) on every run to get it. flushCodexCache now also writes codex-projects.v15.json, keyed by the same fingerprint, and discovery reads that. A missing index is built once from the results file. * Load the session cache once for status today and month status asks for today, then the month. A command can now declare the union of its ranges, so the first load already holds the month in full and the second request finds it there; each request keeps its own month scope and summaries. * Release the pre-lock cache view before reloading Once the refresh lock is acquired or another process completed the refresh, the snapshot loaded before the wait is never used again, but it stayed referenced while the reload read a second view. * Test the first-paint snapshot serves stubs in load order * Parse a shard whole when the range covers all its months A range that spans a shard's months needs nearly every member, and streaming it one member at a time only added allocation churn (about 200MB more RSS on report -p all over a 1GB cache). The streaming reader also reuses one read buffer instead of allocating a fresh one per chunk. * Parse a shard whole once the range reaches its oldest month The previous rule also required the range to reach past the shard's newest month, which a range ending now never does, so report -p all still streamed every shard. * Carry a shard line that spans reads as decoded text Members of a megabyte and more span the 1MB reads, and joining their pieces as Buffers allocated off-heap memory that only a GC returns: report -p all over a 1GB cache peaked about 170MB above main in RSS. Decoding the pieces with a StringDecoder keeps them on the heap and brings every streamed range below main, so the whole-file parse for wide ranges goes again. * Changelog: bounded session-cache load
67 lines
2.4 KiB
TypeScript
67 lines
2.4 KiB
TypeScript
// Codex discovery needs only each rollout's project, so it reads a small
|
|
// index flushCodexCache writes beside the results file instead of the whole
|
|
// (hundreds-of-MB) results cache.
|
|
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
|
|
import { mkdtempSync, rmSync, writeFileSync, existsSync, readdirSync } from 'fs'
|
|
import { tmpdir } from 'os'
|
|
import { join } from 'path'
|
|
|
|
import {
|
|
clearCodexMemCaches,
|
|
codexCacheFileName,
|
|
codexProjectsFileName,
|
|
fingerprintFile,
|
|
flushCodexCache,
|
|
getCachedCodexProject,
|
|
withCodexCacheDirectory,
|
|
writeCachedCodexResults,
|
|
} from '../src/codex-cache.js'
|
|
|
|
let dir: string
|
|
|
|
beforeEach(() => {
|
|
dir = mkdtempSync(join(tmpdir(), 'codex-projects-'))
|
|
clearCodexMemCaches()
|
|
})
|
|
|
|
afterEach(() => {
|
|
clearCodexMemCaches()
|
|
rmSync(dir, { recursive: true, force: true })
|
|
})
|
|
|
|
describe('codex projects index', () => {
|
|
it('serves discovery without the results file', async () => {
|
|
const rollout = join(dir, 'rollout-2099-05-01T10-00-00-a.jsonl')
|
|
writeFileSync(rollout, '{}\n')
|
|
const fp = (await fingerprintFile(rollout))!
|
|
await withCodexCacheDirectory(dir, async () => {
|
|
await writeCachedCodexResults(rollout, 'indexed-project', [], fp)
|
|
await flushCodexCache()
|
|
})
|
|
expect(existsSync(join(dir, codexProjectsFileName()))).toBe(true)
|
|
|
|
clearCodexMemCaches()
|
|
writeFileSync(join(dir, codexCacheFileName()), 'not the results cache')
|
|
expect(await withCodexCacheDirectory(dir, () => getCachedCodexProject(rollout))).toBe('indexed-project')
|
|
|
|
// An index that lags the file (it grew since) only misses.
|
|
writeFileSync(rollout, '{}\n{}\n')
|
|
clearCodexMemCaches()
|
|
expect(await withCodexCacheDirectory(dir, () => getCachedCodexProject(rollout))).toBeNull()
|
|
expect(readdirSync(dir).filter(f => f.endsWith('.tmp'))).toEqual([])
|
|
})
|
|
|
|
it('builds the index from the results file when it is missing', async () => {
|
|
const rollout = join(dir, 'rollout-2099-05-01T10-00-00-b.jsonl')
|
|
writeFileSync(rollout, '{}\n')
|
|
const fp = (await fingerprintFile(rollout))!
|
|
await withCodexCacheDirectory(dir, async () => {
|
|
await writeCachedCodexResults(rollout, 'from-results', [], fp)
|
|
await flushCodexCache()
|
|
})
|
|
rmSync(join(dir, codexProjectsFileName()))
|
|
clearCodexMemCaches()
|
|
expect(await withCodexCacheDirectory(dir, () => getCachedCodexProject(rollout))).toBe('from-results')
|
|
expect(existsSync(join(dir, codexProjectsFileName()))).toBe(true)
|
|
})
|
|
})
|