Commit graph

33 commits

Author SHA1 Message Date
Resham Joshi
1a7284c39b
fix(copilot): count store-row output for unclosed sessions, price grok-4.6 at xAI rates (#1652)
Session-store rows carried output only for compaction requests; every
other request's output was left to the per-turn assistant.message or the
shutdown rollup residual. CLI 1.0.8x and ACP hosts (JetBrains AI Chat via
copilot --acp) write no per-turn output, and an ACP session usually ends
without session.shutdown, so its whole output was lost. Rows now carry
their own output_tokens; at serve time a row that pairs with a per-turn
call carrying output (older CLIs) serves with output 0, so output counts
once in every shape.

grok-4.6 priced at $1.25/M input: LiteLLM has no bare grok-4.6 row and
the stripped azure_ai/grok-4.6 row claims the bare id ahead of
xai/grok-4.6, in the bundler and the live fetch alike. A builtin price
pins xAI's list rate ($2/$6/$0.50 per M), which is what GitHub bills.

On real data every Copilot model's cost now equals GitHub's
total_nano_aiu to the digit.
2026-10-06 19:15:43 +02:00
Resham Joshi
e5dff62fa1
perf: store the session cache in day pieces (#1550)
* Store the session cache in day pieces with a per-provider index

Layout v10: a provider's files live in pieces by the UTC day of their
newest call, each with a key file, and an index row per file (piece,
fingerprint, call span, v9 month) read on every load. A ranged load reads
the index, the key files and only the members it reports on; a save
rewrites only the pieces it touched. Rows keep v9's full-load order and
month scope, so orphans, the first-paint snapshot and dedup see what they
saw on v9. The v9 month shards re-lay out once, streaming, and are removed
after v10 publishes.

* Skip cached transcripts outside a narrow query's months when unchanged

v9 could not see them, so every narrow query re-parsed each older Claude
transcript it discovered and republished the cache. The index knows their
fingerprint; an unchanged one needs nothing, a changed one still re-parses
as if uncached.

* Read piece members in batches through one buffer, and full loads concurrently

* Write piece key files without a sync; rebuild lost keys from the piece

* Changelog: day pieces

* Seed dedup with the keys of unchanged transcripts outside the query's months before a whole-file parse

* Upgrade-path check: verify the v10 envelope, indexes, pieces and key files
2026-09-24 20:45:35 +02:00
Resham Joshi
2fc62f7876
0.9.25: correctness, energy, menu bar and installer fixes (#1511)
* 0.9.25: correctness + provider fixes, energy guard, logo, default DMG

CLI/engine fixes: local-model savings count reasoning; ZCode reasoning billed;
codex-credits version-boundary tiering; GLM alias repointed to z.ai discounted rate;
openclaw/pi/qwen timestamp mtime fallback; hermes delta timestamp;
sqlite/warp/hermes busy+permission reclassified as failed (no sealed empty);
codex-cache dev+ino; cursor-cache WAL; antigravity server TTL; quota timeout+
grokbot footer+claude near-expiry; serve RSS guard env+hysteresis; CSV export
subfolder; providers pane fixed window; Warp credit-based cost.
Desktop/menubar: unified flame logo; default DMG; docs signed+notarized.

* menubar: one running instance, self-relaunch on language change

The single-instance guard only retired strictly older copies, so two login
items starting in the same instant both survived, and a copy whose bundle
identity did not survive install was never seen at all. The decision is now
total (newest launch wins, ties break on pid) and peers are matched on bundle
id or executable name. --keep-both opts one launch out.

Language changes no longer have the desktop app quit and reopen the menu bar,
which made macOS attribute the new process to the desktop app and re-ask for
the data-access grant. The menu bar relaunches itself via the existing remote
command key; older menu bars that do not watch the key fall back to the old
path after 5s.

* desktop: export toast shows the saved path; resource buckets on app_close

The export toast showed the folder that was picked, not the dated subfolder
the files land in. The handler already parsed the CLI's saved line; it now
returns that path and the toast uses it.

app_close carries cpu/mem (and serveCpu/serveMem when serve reported) as bucket
labels, sampled on the existing flush interval. serve adds cpuSec and peak rssMb
to its answer frames. Unmeasured figures are omitted, never sent as zero.

* menubar guard never leaves zero instances; installer replaces in place; JSON export into folder

Guard: an unreadable launch date on either side could make two copies retire
each other. Unreadable peers now count only when their pid is lower, and a copy
that cannot read its own date retires lower pids only, so a disagreement keeps
at least one instance.

Installer: an existing bundle in /Applications or ~/Applications is replaced
where it lives instead of a second copy being added. The old bundle is moved
aside and restored on any failure; a cross-volume move falls back to a copy that
is re-verified with codesign. An unwritable location is never escalated: the
install goes to ~/Applications and the leftover is named.

Export: -o pointing at an existing folder writes the JSON inside it. Short
sessions omit the cpu buckets. setLanguage reopens the menu bar if its own
relaunch never comes back. AppRelaunch passes the bundle path as an argument.

* menubar guard orders by kernel start time; installer opens the copy that exists and recovers a killed install

Guard: launch dates could be unreadable and pid order can disagree with start
order after pid reuse, so two copies could still retire each other. Ordering now
uses the kernel-reported process start time, which both sides read identically.
A copy whose start time cannot be read is left out, so the worst case is two
menu bars, never none.

Installer: with the only copy in an unwritable /Applications, a plain run tried
to open a path that did not exist. It now opens the installed copy. A placement
killed mid-way is recovered on the next run, a concurrent install is refused,
and a failed restore names where the previous bundle is. Process matching uses
the exact executable name.

Export: a trailing separator names a folder. Telemetry cpu is measured from
telemetry start.

* installer: a recycled pid can no longer block installs; language change waits for the replacement process

An aside left by an earlier install counted as an active install whenever its
pid was live, so once that pid was reused every install was refused for good.
It now also has to be recent (ctime within 10 minutes), a pid we may not signal
is not ours, the refusal names the file, and an orphan that cannot be cleared is
reported instead of aborting the install.

setLanguage confirmed the relaunch with a matcher the dying process still
satisfied, so the reopen fallback never fired. It now waits for a pid that was
not running before.

* desktop: only the headline refreshes live; act, yield and the optimize scan move to slower tiers

An open window ran act report (a fresh CLI process each time), yield and an
optimize scan on every refresh. The headline poll now passes --no-optimize and
takes the snapshot path; act report refreshes on mount, on manual refresh and
every 10 minutes, yield every 5. The optimize scan runs at most once a day per
query scope, is stored in userData with its time, and is forced fresh by the
Optimize page or a manual refresh. Figures that are not live show their age,
and no stored scan means no savings clause rather than a zero.

On battery the live interval doubles; the chosen cadence stays the base.

* desktop: a stored optimize scan is only valid on the day it was computed; cadence changes no longer refetch

The scan cache is keyed on the argv, which says 'today' or '30days', not a date,
so yesterday's scan was served as today's. It is now reused only on the same
local day. Stored findings in Signals carry their age, a failed scan shows the
error instead of scanning forever, and the cold-boot label uses the time the app
received the payload.

usePolled fetched immediately whenever its interval changed, so every power
transition (and launching on battery) refetched every live hook and could spawn
an extra act report. Fetching and timer arming are now separate effects.
Optimize's yield is on the same 5 minute tier as Overview.

* desktop: re-ask for the optimize scan on the first render of a new local day

An app left open across midnight kept yesterday's stored savings under Today,
because the scan was only requested on mount, scope change or manual refresh.
The local day is now one of the hook's triggers; the previous day's figure is
dropped while the new scan runs.

* serve: a menubar poll is never answered from the status snapshot

Inside its settle window the status snapshot returns the pre-change payload on
purpose. The resident serve process then memoized that answer as clean, so after
a single write followed by quiet the headline stayed on the old figure until the
next write or the five minute memo cap. The snapshot is for one-shot runs; the
resident process already holds the parsed state, so it now skips it.

* tests stop writing the real serve pid file; superseded caches are swept; installer notes reach the desktop card

ServeConnection takes its pid file as a parameter, so the Swift tests no longer
share ~/.cache/codeburn/menubar-serve.pid with a running menu bar, and the
reaper only signals a record that is a serve --stdio command.

Superseded session-cache, codex-results and cursor-results files (100MB+ each)
are removed once untouched for 14 days, at most once a day. daily-cache and the
hermes ledger are never swept: they carry sealed days a version bump adopts.

The menu bar installer's leftover-copy advice is emitted as a marked line under
CODEBURN_PROGRESS and shown on the desktop card. The skipped-refresh NSLog is
debug only. The app's vitest workers turn off Node's built-in web storage.

* menubar: anonymous usage statistics, under the same consent as the desktop and the Windows tray

The Swift menu bar sent nothing, so every codeburn-menubar row was a Windows
tray. It now follows the desktop's telemetry.v1.json when that exists (read
only: same install id, same on/off, onboarded required) and otherwise keeps its
own state with the same region default (off in EU/EEA/UK/CH and when the region
is unknown). Settings gains one Privacy toggle, disabled when the desktop owns
the choice. Events use the tray's names; the daily usage snapshot is the CLI's
block passed through a sanitizer and is left to the desktop when it is the
consent source. Debug builds never send.

A resident CLI that gets disabled for the run now says so in the log.

* menubar statistics: private network session, abandoned batch on opt-out, a desktop No survives the desktop going away

The sender used URLSession.shared, so a server or CDN cookie and the default
User-Agent rode along and would have outlived an opt-out. It now owns an
ephemeral session with no cookies, no cache, a fixed agent string and no
redirects. A batch that is out when the toggle goes off is dropped rather than
re-queued under the fresh id. The desktop's answer is recorded locally, so a
missing or damaged state file never falls back to the region default, and the
standalone id is persisted when minted. The toggle stays usable as a local off
switch under desktop consent. Copy states what this app actually sends.

* vercel gateway spend is shown on its own row and kept out of totals by default

The gateway reports daily aggregates per model, with nothing to match against
the local tools that were pointed at it, so every headline counted that traffic
twice. Totals now leave the gateway slice out when all providers are shown; the
row keeps its full amount and is marked not in total, --provider vercel-gateway
is unchanged, and codeburn gateway-totals include folds it back in. The daily
cache still stores the slice, so the choice applies retroactively. Reports with
no gateway credential are byte-identical.

The gateway's request_count now feeds the call count instead of each daily row
counting as one call. Hermes: a mid-session price change lands on the day it is
observed and sealed days are never rewritten; stated in the code and changelog.

* gateway exclusion applies to every report through the parse, and the detail panels follow the headline

The first pass narrowed only the day totals, so By tool, top models, top
projects, activities, status json projects and any project-filtered table still
carried gateway spend next to a headline that did not, and By tool shares could
pass 100%. The rule now lives in parseAllSessions: with all providers in scope
and the opt-in off, aggregate-only sessions are dropped after the memo, and the
callers that derive cached days opt out so stored slices are unchanged.
day.projects is reduced like models. The gateway stays visible once, as a
labelled not-in-total row with no share, and standalone reports note the
excluded amount on stderr so machine output is unchanged.

* dashboard: the background history index follows the gateway exclusion too

The index keeps the whole corpus so it can fill the daily cache, but its
per-period projection handed that unfiltered list to the project, model and
activity panels, so they regained gateway spend once the index landed.

* ci: a hung lock suite fails in 90s and retries once with a warning; slow Windows tests get measured limits

The cache-lock step failed 6 of the last 60 runs with no output for five minutes
after the vitest banner; passing runs take 12-18s on every Node version. Each
attempt is now capped at 90s and retried once, with a warning whenever that
happens. On windows-latest a single CLI spawn takes 1.4-12s, so the four-spawn
status snapshot test and the c6 parser suite get limits taken from those runs,
each with its reason. Both test jobs print their ten slowest cases.

* desktop: Privacy & data uses the same rows as the rest of Settings

The pane mixed two static claims with two controls in its own layout. It now
uses the about-row pattern every other pane uses: title and hint on the left,
one control on the right. The claims became the subtitle. Rows: anonymous usage
statistics (toggle, link to what is sent), device sharing state with Manage,
local report snapshots with Clear, export with Open export.

* desktop: Privacy & data keeps one bounded column and rows that never squeeze

The card stretched with the window, stranding each control hundreds of pixels
from its text. This pane's heading, subtitle and card now share a 760px column
and one text measure. Rows stack on their own width, because the window can
never be narrow enough for the old viewport rule to fire; row buttons do not
wrap. The what-is-sent link sits on its own line, and the snapshot and export
descriptions fit their measure and say this machine.

* menubar tests clean up their scratch defaults and leave the real Keychain alone

Four test files never removed their UserDefaults suites and the rest removed the
domain but left the plist, so thousands of files piled up in
~/Library/Preferences. A shared helper forgets a suite and deletes its file, and
every test that makes a suite now uses it. cfprefsd still rewrites an empty plist
when the test process exits, so a run leaves about a hundred 42-byte files with
no test data in them. The real-Keychain round trip only runs with
CODEBURN_TEST_REAL_KEYCHAIN=1.

* desktop: the resident parser retires after 15 idle minutes and restarts on demand

The renderer stops polling while the window is hidden, but the serve child stayed
resident with its parsed cache, one to two GB on a large corpus. After 15 minutes
without a request it is stopped the way a settings restart stops it, without
spending the unexpected-death budget, and the next request starts a new resident
through the existing lazy path, never a one-shot. CODEBURN_SERVE_IDLE_MS tunes or
disables it.

* desktop: every settings pane keeps a bounded column, and dark buttons have an edge

Cards stretched with the window, so on a wide display controls sat up to 1300px
from their labels and intro lines ran past 140 characters. All panes now share
one left-aligned column (760px, 980px for projects, aliases and pricing, which
hold wide rows) and one text measure. Rows stack on their own width instead of a
viewport rule the 900px minimum window could never reach, and a long path in
Projects no longer overflows the pane.

In dark mode the secondary button matched or sat below its surface, and was
invisible on the plugin install dialog. It gets a face and a border from two new
tokens; light is unchanged. The onboarding consent label matches Settings.

* desktop: this device shows its fingerprint on its own line

The local name and the fingerprint are two adjacent inline spans, so they read as
one run of text.

* menubar: the resident parser retires after 15 idle minutes and restarts on demand

The menu bar kept its serve child, and the parsed cache with it, for the whole
run, even with the popover closed and nothing changing. After 15 minutes without
a request the child is retired without counting as a death, and the next request
starts a new resident through ensureStarted, never the one-shot fallback, whose
snapshot settle window would make the figure lag a whole tick. The window sits
above the longest gap between real requests, so a child stays warm while work is
happening. CodeBurnServeIdleSeconds tunes or disables it.

* serve: a poll re-checks only what the watcher saw change, and shard writes are coalesced

With a session being written, every poll re-listed about 2,400 directories and
re-fingerprinted about 6,000 transcripts, then rewrote the whole current-month
shard (49MB on a large corpus) and read it back on the next poll. The resident
process now reuses the previous listing and fingerprints for paths its root
watcher proves untouched, and falls back to the full sweep whenever the watcher
is missing, unhealthy, overflowed, armed after the sweep, does not cover the
path, the memo is older than 4.5 minutes, or the process was suspended. Shard
publication is held for up to 90s and flushed when stdin closes; a held write is
dropped rather than written over something another process published. One-shot
runs are unchanged. Active poll cost 0.71 to 0.33 CPU-s at a 30s cadence;
time-to-reflect unchanged at about 1.2s across append, new dir, new file, delete,
rename-over, in-flight, watcher overflow and unhealthy watcher.

Changelog: state the telemetry default as it is in this version.

* a retired parser gets to publish its held shards, and durable providers are never held

Both apps retired the serve child with SIGTERM before its stdin-close flush could
run, so every retire discarded the coalesced window. They now close stdin, give
the child a bounded grace to publish and exit, and signal it only if it is still
running; a settings restart does the same. Measured: flushed 0/5 before, 5/5
after, for both clients.

copilot's cache entries are the only record once Copilot prunes its own files, so
a held window that was dropped (retire, crash, kill -9) before the source was
pruned lost that spend for good. A window that a durable provider dirtied is
published on the poll that parsed it, as before coalescing existed. The RSS guard
publishes a held window before it clears the caches instead of discarding it.

* serve: release the lifetime parse after the fill, and make the collection real

The RSS guard dropped its memos and then called globalThis.gc, which exists only
under --expose-gc; no launcher passes it, so the collection had never run and the
freed pages never went back. On a cold start the resident idled at 2.2GB with
364MB of live heap: clearing every memo moved RSS by 0MB, a collection alone by
700MB.

After the background fill has sealed its summaries the process now publishes any
held shard window, drops the memos it can re-read from disk (the cold start had
loaded the cache with scope 'all', which then served every later poll), and
collects; it collects again a minute later for as long as a pass returns 64MB,
at most four times. Cold idle 2198MB to 700MB, warm 900MB to 781MB. Reports are
leaf-identical cold and warm, time-to-reflect unchanged, a released month scope
reloads in 0.3-0.5s.

* warp: the billed cost reaches reports and survives the cache

The provider derived a dollar cost from Warp's own billing record (provider cents,
charged usage, or credits), but warp was missing from the reported-cost
pass-through, so the cached call kept no cost and every report, cold or warm,
priced the conversation from its tokens. A fixture billed at $1.22 reported
$0.15. A call whose cost came from a billing record now keeps it; a call that
only has the token floor stores none and is still re-priced when prices change.

Warp entries re-parse once (provider parse version), and the daily cache moves to
v34 so days whose Warp source still exists re-derive; call counts do not move, so
nothing is double counted or dropped.

* openclaw and pi: a reported cost survives the cache like every other reported cost

Both providers read the cost their tool recorded for a message, and both were
missing from the reported-cost pass-through, so reports priced every call from
tokens instead. Fixtures reported $0.011 against $0.056 (openclaw) and $0.013
against $0.410 (pi). A call with a reported cost now keeps it; a call with none,
or an explicit zero, stores none and is still re-priced when prices change. omp
had been added to the list in #1150 and pi, the same parser, was never revisited.

The provider list is one named set, the unreachable providerCallToTurn is gone,
and both providers re-parse once; the v34 daily cache re-derives their days with
call counts unchanged.

* test: pin parse count per report and per resident poll
2026-09-21 00:16:21 +02:00
iamtoruk
830f3ff6d6 fix(cache): record session-cache completeness per provider
A scan is scoped on two axes, provider and date range, but completeness
was one whole-cache boolean. #1229 stopped a scoped run from stamping it
by adding a second, unfiltered discoverAllSessions to the stamp path,
which left scoped users on a multi-provider machine permanently cold and
paid an extra full discovery on every run.

Completeness now lives on the provider section, with an optional
completeFrom floor for the sources a date range filtered out. The
whole-cache flag keeps its meaning (an unscoped scan finished) and is
what a section with no flag of its own falls back to, so existing caches
read exactly as before and no version bump is needed. Discovery reports
which providers threw, and a provider that reported nothing because it
failed is never marked complete.
2026-09-12 04:00:40 -07:00
iamtoruk
62cb4f8630 test: isolate the real home on Windows and make the CLI suite pass there
os.homedir() reads USERPROFILE on Windows and never HOME, so the
isolation setup and every test that moved home were still resolving the
developer's real home directory. Fixtures came back empty, real sessions
leaked into counts, and the code under test wrote plans, receipts and a
sync ledger into the real ~/.config/codeburn during a run.

USERPROFILE now joins the redirected variables and a shared setHome()
helper moves HOME and USERPROFILE together; every spawn env carries both.
The remaining Windows-only failures were POSIX-shaped fixtures: chmod
tricks that only toggle a read-only attribute on NTFS, pid 1 as an
always-alive lock holder, splitting paths on '/', bare tar, mode-bit
expectations, and per-file timeouts too short for the cost of a tsx spawn.
Each gets a platform-aware equivalent so nothing is skipped on Windows and
the POSIX branches stay byte-for-byte what they were.

Full run on Windows: 267 files passed, 2 skipped, 3592 tests passed.
2026-09-02 13:43:57 -07:00
Resham Joshi
1d375ccf79
fix(tui): keep the dashboard responsive during the cold-start fill and exit on Ctrl+C (#1142)
The progressive cold start introduced by #1109 paints the dashboard from
recent files, then runs a background fill that parses the deferred
history and writes the per-file session cache. On a real 21k-file corpus
that fill's cache save publishes one JSON.stringify + fsync per provider
shard in a tight loop, and the whole save held the event loop long
enough that Ink's stdin reader never ran - a q or Ctrl+C pressed in the
post-paint window was swallowed, not buffered, and the user saw a
frozen terminal (#1139, #1141).

saveCache now yields to the event loop between every shard write, so
the Ink input handler fires between publishes. The yield sits BETWEEN
shards, never inside one, so each shard's temp+rename atomicity and the
lost-fence cleanup that owns the canonical envelope are preserved;
nothing marked seen without being parsed, the kill-safe resume
invariant from #1109 is intact, and the floored-parse memo key suffix
keeps warm reads byte-identical. The fill still parses every deferred
file, so money totals are unchanged.

The dashboard's useInput now exits on Ctrl+C through the same path as
q, so a raw 0x03 from the terminal lands as soon as the loop breathes.

Pinning tests:
- tests/save-cache-yields.test.ts: a setImmediate probe must interleave
  with saveCache's shard writes.
- tests/dashboard-exit.test.ts: a raw \x03 exits the dashboard the
  same way q does.
- tests/parser.test.ts: the copilot validated-clean re-parse test
  tightens its fake-timer scope to only Date, so the new event-loop
  yield inside saveCache still resolves under vitest's fake timers.

The #1109 cold-start one-shots (report, sessions, models, json/csv/
markdown, the desktop and menubar payloads) are unchanged.
2026-08-25 06:08:39 -07:00
iamtoruk
c30ebecff4 Merge origin/main into feat/copilot-session-store
Resolves the conflicts #946 accumulated while it was in validation. Eight
files conflicted; the session-store accounting is unchanged.

src/daily-cache.ts — version collision. This PR minted 25 when main was at
24; #1056 (`codex-auto-review` pricing) then spent 25 on main. The bump moves
to 26/MIN 26 and daily-cache.v26.json, with main's full comment ladder kept as
the foundation and this PR's paragraph rewritten to name 26 and record the
collision. PENDING_REDERIVE_PROVIDERS and the B1 migration semantics from
b6481c19 carry over intact, retargeted at 26.

src/models.ts, src/parser.ts, src/audit-report.ts, src/models-report.ts —
#1075/#1078 replaced the per-site "reasoning is already inside output" tests
with billableOutputTokens() and REASONING_INCLUDED_IN_OUTPUT. This PR had
added copilot to that case at three sites independently. Union: all three
sites take main's helper call verbatim, and copilot joins claude and codex in
the set — same accounting this PR shipped, now through main's single source of
truth. It also reaches parser.ts activeGeneratedTokens (a fourth site, from
#1079), which is the same correction: a copilot supplementary call carries
reasoning with output 0, so counting it as generated repeats the per-turn
output. The audit legend already said so on this side.

src/providers/copilot.ts — comment-only. #1054's lastEventTimestamp-first
shutdown fallback was derived from this branch, so the code was already
identical on both sides: the `shutdownTimestamp` expression and the
`copilot:<sid>:shutdown:<model>:<n>` key are byte-for-byte main's. Both
rationales are kept (leg-collapse on date, and residual anchoring).

src/session-cache.ts — PROVIDER_PARSE_VERSIONS.copilot takes this PR's
`-session-store-v3` suffix; main's #1051 note about why a fingerprint change
is expensive is kept above it. Codex keeps main's #1092 suffix chain untouched.

src/main.ts — #1067 deleted the unreachable live dailyMap fallback that this
PR had taught behavioral weight. Main's deletion wins; the now-unused
isBehavioralTurn import goes with it.

tests/parser.test.ts — import union.

Also: scripts/upgrade-path/run.mjs NEW_DAILY_CACHE -> daily-cache.v26.json,
CODEBURN_COPILOT_SESSION_STORE_DB added to the #1064 env-isolation CLEARED
list, and the CHANGELOG entry's stale "v21" corrected to v26.

Verified: tsc clean; 3132 tests pass across 223 files; test:locks 26/26;
verify:upgrade PASSED, re-deriving daily-cache.v26.json and holding durable
copilot history across the bump. #1054's regression ("keeps three stampless
shutdown legs as :n keys with lastEventTimestamp") passes on the merged tree.
Real-corpus A/B against origin/main over 2026-07-01..2026-08-22: codex, grok,
kimicode and opencode byte-identical in export, audit and models; claude drifts
only monotonically with run order (a live session writing transcripts, confirmed
by interleaving four runs). This machine has no copilot data, so the copilot
recovery semantics rest on the suites and the upgrade-path corpus.
2026-08-22 09:36:51 -07:00
iamtoruk
b6481c1933 fix(copilot): let a parse-version bump re-derive the durable calls it re-reads
@vidoluco's round-6 re-validation: a cache generated by 0.9.20, copied and then
migrated by this branch, dropped exactly ONE call a virgin cache serves -
mai-code-1-flash-picker, 324 input / 91,136 cache read / $0.01, two minutes
after the labelled compaction row in its session. Every other day matched.

Root cause is the durable union merge, not the reconciliation. `compactedAt` is
a capture-only field this branch adds to the shutdown-ROLLUP call, and 0.9.20
already cached that call under the same (deliberately stable) dedup key. The
union appended only UNSEEN keys, so the bump could never reach it. The migrated
cache then ran the compaction-anchored residual math with no anchor - the
pre-anchor behaviour exactly: the interval opens at -Infinity, the
PRE-compaction rows are subtracted from a rollup that never counted them, and
the residual clamps to zero and disappears.

The union now REPLACES a cached call with the freshly-derived one wherever the
key matches. Nothing is deleted: keys the re-read did not produce are exactly
the rows the source has pruned, and those still carry forward untouched with
the old parser's fields. Keys it did produce were just derived from the live
source in this very pass, so the fresh call is by construction the better one.

This is also what makes the round-5 compaction-output fix reach an existing
cache - that row's key is stable too, so append-only would have pinned it at
output 0 forever. Both directions are pinned by (c6).
2026-08-22 03:30:53 -07:00
iamtoruk
12c8cb7790 fix(copilot): re-derive migrated copilot day slices, count the compaction row's output
Two defects from @kelchm's two-machine validation of #946.

1. The daily-cache migration carried the copilot slice of every existing day
   entry verbatim instead of re-deriving it, so `overview` on a migrated cache
   kept serving pre-store numbers (2,980,804 tokens for a day whose fresh
   derivation is 74,811,412; 21,608 calls vs 16,326) while export/models/audit
   served the corrected ones off the same cache.

   Root cause is the partial-survival guard (e9d922ca), not the migration: it
   keeps a settled baseline slice whenever the fresh derivation reports FEWER
   calls, reading that as aged-out sources. This change's whole point is that
   copilot's supplementary accounting calls stop counting as api calls, so
   every store-era day shrinks in calls and got pinned to its pre-store value.
   The guard's own TRADE-OFF note predicted exactly this case.

   A migration from an older cache version now grants copilot ONE
   guarded-shrink-exempt re-derivation (`pendingRederive`), spent by the first
   COMPLETE parse. It fires only where that parse actually produced a slice for
   the (day, provider), so a day whose sources are gone still carries forward
   whole - never-lose is unchanged in both directions - and no other provider
   is exempt.

   The daily cache moves 21 -> 25: 21-24 landed on main during validation, and
   only a number the validators' own daily-cache.v21.json cannot claim gets
   their stale carried slices re-derived.

2. The `initiator='compaction'` store row was ingested with outputTokens 0. Its
   output has no assistant.message anywhere in events.jsonl, so that row is the
   only place those tokens exist - dropping them was the sole token discrepancy
   (-3,085) across a 30-session store-matched comparison. It is now read (on the
   same optional-select rung as `initiator`, which is what identifies the row),
   counted and priced; every other row keeps output 0 because a per-turn call
   owns it. The copilot parse version moves to session-store-v3 so cached v2
   rows re-parse; the dedup key is deliberately unchanged.

Settle window stays at 24 h - the three-machine evidence for tightening it is
recorded next to the constant, but shortening it is a product call.
2026-08-22 02:26:35 -07:00
Aditya Vikram Singh
9c8727d94f fix(copilot): prefer lastEventTimestamp for stampless shutdown calls
Maintainer review on #1054: two-journal collision is unreachable
through discovery. Keying by timestamp+journal collapsed a 3-leg
stampless journal onto one row, and the :n strip re-sent ledger
keys. Keep occurrence keys. Take lastEventTimestamp before
sessionStartTime for the call timestamp only. No migration.
2026-08-21 18:04:28 +05:30
iamtoruk
f77687a8b3 Merge origin/main, and act on vidoluco's real-Copilot findings
Merge resolution
- daily-cache: main shipped v20 with #1040, so 21 is now the first FREE number
  rather than one dodging an unmerged head. Both version notes kept, ours on
  top. The re-derivation seed stays one below current (20), now the shipped
  predecessor rather than a draft.
- upgrade-path: expected daily cache filename stays v21.

The rollup is cumulative across resume legs — already handled, now pinned
Measured on a real 3-leg session (CLI 1.0.80): every counter in a leg includes
the legs before it, and the last leg of a complete session equals its store-row
total to the nano. The parser has always emitted `cumulative - previous
cumulative` per model, so the interval arithmetic downstream already consumes
per-leg claims; it was the PR body and the provider doc that described the raw
journal as per-leg and misled the reader. Doc corrected, and (c3) pins it end
to end: a complete 3-leg session serves 900 (its last leg / store total), not
1,400 (the sum of its legs); an uncovered one serves 900 exactly once; and a
leg reporting LESS than its predecessor is taken as a fresh epoch rather than
clamped to a negative delta, so an older per-leg CLI never loses a leg.

initiator = 'compaction' replaces the timestamp heuristic where it exists
The summarization request does write its own assistant_usage_events row, and
newer stores label it. The label is now read — schema-adaptively, and the
enrichment select is a graduated chain so a store carrying the billing columns
but not `initiator` keeps its billing metadata instead of falling all the way
back to the base select. Two uses: the row is subtracted from the leg it
belongs to even though it commits before the compaction stamp, and it is kept
out of per-turn pairing since it has no assistant.message to pair with.
Optional twice over — absent on older stores, NULL on 1,504 of 2,509 rows on a
real one — so (c5) pins the labelled path at 350, the identical UNLABELLED
fixture at 400 (the documented one-request-per-compaction over-serve, which no
timestamp rule can close: the request that triggered the compaction completes
immediately before it too), and a compaction row never stealing a pairing
partner.

Attribution invariant, standing guard for the 1.8x report
(c4) runs `codeburn audit`'s own two numbers — attributed vs recomputed —
over all four combinations of the three representations a session can be
written in, at the magnitudes of a real reported day (gpt-5.6-terra, 146 rows,
input 17,792 / cache write 501,395 / cache read 12,097,364 / output 63,344 /
reasoning 24,831, billed $4.47). Every shape reconciles at 1.000, and the
covered case lands on $4.4687 = what the tokens price at = what GitHub billed,
with output 63,344 rather than the pre-fix 88,175 that re-priced reasoning on
top of itself.

upgrade-path: a day KEY is not the unit of never-lose
A parse change that re-dates a call to its true day legitimately empties one
day and fills its neighbour, token for token — observed on a real cache where
2026-08-08's single call moved to 08-07 exactly. The aging step now compares a
+/-1 day WINDOW rather than the day alone, so only a window that shrinks is a
loss, and an emptied day key is reported rather than failed.

RECONCILE_SETTLE_MS keeps 24h, with the measurement that argues against it
recorded at the constant and in the provider doc: across 91 real sessions zero
rows landed after shutdown (median -0.1s, max -0.0s). One machine, one CLI
version — the number to beat is seconds, not hours, once a second machine
agrees.
2026-08-21 02:14:14 -07:00
Aditya Vikram Singh
a3ea24859e fix(copilot): journal-scope shutdown keys without a fingerprint bump
Extra High HOLD on #1054: timestamp-only keys still collided across
journals, and the provider-wide rollup-ts-v1 bump dropped present
OTel sources, erasing conversations already pruned from the DB.
Identity is now (session, model, timestamp, journal basename).
Legacy :n keys migrate on that JSONL file only.
2026-08-20 09:52:56 +05:30
iamtoruk
dbeae9b48c fix(sync,parser): review B — symmetric shape freeze, honest settle, stable residual keys
B-1 The freeze was one-directional and left the reverse hole wide open. Rows
sync; at the 90-day durable age-out the cached rows are pruned, the rollup
stops being dropped and serves again under a key that was never sent; the
session is long past settle, so it pushes an aggregate on top of the
per-request spans the receiver already holds.

Reframed around the thing that actually matters: a copilot session's
input/cache leaves this machine in one of two SHAPES — the raw rollup
(`copilot:<sid>:shutdown:`), or reconciled output (rows plus
`:shutdown-residual:`, which are disjoint by construction and together are
exactly the rollup re-expressed). The receiver must never hold both. Whichever
shape a session was first synced in, it stays in, and the other is frozen for
that session permanently — in both directions. Growth WITHIN the sent shape is
untouched, because same-shape output is additive, never substitutive: a
resumed session's new rows and residuals still push if rows were sent, a new
leg's rollup still pushes if rollups were.

This also corrects the previous commit, which classed a residual as aggregate
and so froze it for a session whose rows had gone out — the residual is
reconciled output and belongs with the rows.

B-2 A session with an unparseable timestamp was held forever while the CLI
promised it would "push once it settles". Nothing could ever settle it. Settle
now reads the newest moment a session can be SHOWN active; a session with
nothing datable at all is sent rather than held. Mixed sessions still settle on
their datable stamps.

B-3 A stamp implausibly far in the future is broken data, not evidence the
session is live, so it no longer counts toward that newest moment. One
year-ahead row can no longer hold a month-old session hostage. Ordinary clock
skew is absorbed by a one-hour grace in both directions and still reads as
live.

B-4 The residual dedup key carried the leg's POSITION. Legs sort across every
cached file for a session, so an earlier leg arriving later renumbers every
residual after it — and a renamed key is a span the receiver takes a second
time, since there is no retraction for a usage span. Keyed by the leg's own
instant instead: append-only files mean it never moves, and the equal-timestamp
coalescing above makes it unique per leg. Residuals are new in this change and
have never shipped, so no CACHE_VERSION concern.

B-5 The dry-run "already synced" count now subtracts frozen too, matching the
nothing-to-push line.

Tests: symmetric freeze from the rows side (rollup frozen, residual and new
rows still sent, per-turn untouched); all-unparseable sent, partly-unparseable
still held; a year-ahead stamp ignored while a one-minute skew still holds;
residual keys named after their leg instant with the positional names gone. The
key-prefix pin picks up the residual's new tail. (c2) pins the property rather
than one insertion scenario, because a second file's leg currently collides on
the rollup's own dedup key before it can reach the residual sweep — the
reachable repro would prove nothing about the next one, and that is stated at
the test.

Flakes: (f2) and (sc) re-bucket each run 10x isolated and 10x with four busy
loops pinning cores — 40/40 clean. Neither has clock or ordering dependence at
its margins: afterEach removes both tmpHome and the cache dir, and the two
relative timestamps in the re-bucket test are 35 days apart so they cannot
share a month. The likeliest cause of the transient failures is this branch's
own commits rewriting src/parser.ts under a concurrent vitest. (f2) gains a
self-check that its fingerprint sentinel really differs from the computed one,
which is the one way it could have passed while exercising nothing.
2026-08-19 13:25:15 -07:00
iamtoruk
02d8c90007 fix(copilot): anchor a rollup leg's row interval at its compaction
The CLI's session.shutdown rollup RESETS its counters at a successful
in-session compaction, so a leg containing one describes only its
post-compaction requests. Running the store-row subtraction from the previous
leg cancelled that leg's usage against the whole pre-compaction conversation
and left the residual short by exactly that much. With a complete store the
floor hid it; with a partial snapshot it was a permanent undercount, and a day
sealed on it never corrected. That is the half of the reviewer's blocker 2 the
read-ordering fence could not reach: no timing rule helps when the rows are
simply not there yet, and rollup > rows-in-interval is indistinguishable from
legitimate partial coverage.

Anchor at the compaction instead. Pre-compaction rows still SERVE; they just
stop cancelling usage the rollup never claimed. The maintainer's repro (row A
100 pre-compaction, sole rollup 200 for post-compaction request B, B's row
absent) now totals 300 instead of 200, and still totals 300 once B lands.

Event names taken from @github/copilot 1.0.80, not guessed:
session.compaction_start (nothing needed from it) and
session.compaction_complete, whose `success` is read and every other field
ignored — the payload also carries the summarization call's own usage, token
counts, model and trigger. Only success:true anchors; failed or absent falls
back to the previous leg; last compaction in a leg wins. The stamp rides on
the cached rollup call as `compactedAt` (add-only, validated optional).

Accepted, bounded over-serve, documented at the doc site: if the summarization
call writes its own store row it sits just before the compaction stamp, so it
falls outside the interval while the post-reset rollup counts it — one request
per compaction can serve twice. Not subtracted from compactionTokensUsed,
because the request that TRIGGERED the compaction also completes immediately
before it: any grace window wide enough to catch the summarization row also
catches a real request and turns an over-serve into a loss. The row writer is
native, so the bundle cannot settle it; a real store can.

(c1) pins all four shapes: the repro at 300 (and again through the cache), no
double once the row lands, a failed compaction ignored, and a leg with no
compaction unchanged. The existing 107 copilot tests stay green.
2026-08-19 13:01:43 -07:00
iamtoruk
fa294146b3 fix(copilot): read the session store after the journals, fence it when stale
A session.shutdown rollup is reconciled against the store rows written up to
it, and the two live in different files. A session that shuts down mid-pass
appends its rollup to events.jsonl and its last row to the store, so whichever
we read first is the one that can be short — and the short direction
under-serves a compacted session, whose sole rollup covers only its
post-compaction requests while the missing row's usage has nowhere else to
come from. A day finalized on that pass keeps the undercount, because a sealed
day is only ever re-derived by a version bump.

Two halves, one rule: the durable record is read last.

- Changed stores parse after every other changed source of their provider, so
  the row set is a superset of anything a rollup we read can claim. The short
  direction becomes unreachable; only the harmless one is left (a row whose
  journal partner has not landed, which serves its own tokens and pairs next
  pass).
- A store served from cache is never re-read, so ordering cannot help it. If
  it moves between its classification stat and the end of the pass, the rows
  the journals were just reconciled against are stale — hold the daily
  watermark instead of sealing. It lifts by itself on the next refresh, which
  sees the store as changed and re-reads it last.

(f3) pins both: the flagged source parses last whatever the discovery order,
and a cache-served store mutated from inside another source's parse reports
partial hydration, then clears.

Not covered, deliberately: a store that is behind its journal for reasons no
local evidence can show (rows pruned, a backdated fixture). rollup > rows in a
leg's interval is indistinguishable from legitimate partial coverage — the
store adopted mid-session has exactly that shape permanently — so fencing on
it would hold the watermark forever. Anchoring a leg's interval at the
compaction that reset it is the real fix and needs the compaction event's name
from a machine that has Copilot data.
2026-08-19 12:38:46 -07:00
iamtoruk
22a8405d28 fix(cache): keep durable history when a bump meets an extant, pruned source
A parse-version or env-fingerprint change rebuilds the provider section, and
the durable carry-forward only kept entries whose source PATH had disappeared.
A durable SQLite source does not disappear: copilot's agent-traces.db and
session-store.db keep their file while the CLI prunes rows out of them. So
"the path exists" was being read as "the source can re-derive this", and the
bump deleted, permanently, exactly the history only the cache still held.

Carry every durable entry forward. For a source still on disk, park an
impossible fingerprint on it so the new parse version re-reads it in full; the
durable union merge then appends only turns whose dedup keys are not already
cached, which adds what the source still holds without duplicating it or
deleting what it has lost. The persisted `durable` flag is carried too, so the
orphan bootstrap does not have to wait for the section to be re-stamped.

The contract this rests on is stated at the site: a parse-version bump must not
re-key calls the source can still re-derive, or the union counts them twice.

(f2) pins both directions on the reviewer's shape — a pruned-but-extant DB
keeps its history across a bump, served AND persisted, and an intact DB
re-read under the same bump does not double.
2026-08-19 12:32:39 -07:00
iamtoruk
f060a2d806 Merge origin/main into feat/copilot-session-store-landing
Resolved against main's newer invariants:

- daily-cache: DAILY_CACHE_VERSION/MIN_SUPPORTED -> 21. Main shipped 19
  (#1015 grok); 18 was burned by this branch's earlier public head and 20 is
  claimed by the unmerged #1040, and isMigratableCache would adopt either as
  finalized without re-deriving. Header keeps main's v19 note and adds v21's.
- session-cache: PROVIDER_PARSE_VERSIONS keeps main's grok/dsh entries and
  appends this branch's `-session-store-v2` to main's copilot value.
- parser: durable age-out takes main's orphan-only rule (#992).
  retainWhilePresent is now redundant under it rather than load-bearing.
- audit-report: both imports (isBehavioralCall + sanitizeModelForDisplay).
- tests/parser (e), (l), (sc): re-pinned to the orphan-only age-out. A
  still-discovered >90d copilot events.jsonl now keeps serving its per-turn
  output alongside the store rows; the (sc) age-out subject is now an orphan.
- daily-cache-version-rederivation: seed 20 so the adjacent-draft-version
  case is what the test pins.
- scripts/upgrade-path: expected daily cache filename -> v21.
2026-08-19 12:29:33 -07:00
iamtoruk
2d35c8fa24 test(parser): cover durable retention through a month-scoped refresh 2026-08-18 08:52:43 -07:00
iamtoruk
b2d0f29b6c Merge origin/main into pr992 2026-08-18 08:49:13 -07:00
Matthew Kelch
a5a0ddaeac
feat(copilot): read per-request input/cache from session-store.db
The Copilot CLI and the GitHub Copilot desktop app both write
~/.copilot/session-store.db unconditionally; its assistant_usage_events
table holds one row per API request. Until now input/cache tokens for
these surfaces came only from the session.shutdown rollups in
events.jsonl, which are written only on clean shutdown (a crash loses
the whole leg's input/cache accounting) and lump each session leg into
one per-model total. The rollup also RESETS its counters at in-session
compaction (traced on a clean single-process 107-request session whose
sole rollup covered exactly its five post-compaction requests), so even
cleanly-closed long sessions were truncated; on a long-history machine
the store recovered ~35% of real Copilot spend lost to crashes and
compaction resets. The DB rows are per-request, crash-proof, and carry
real timestamps.

The store's input_tokens is cache-INCLUSIVE (input + cache_read +
cache_write), the same convention as the shutdown rollups — verified
against each row's token_details_json and by reconciling per-session
sums against the CLI's own footers and rollups across two machines
(1,380+ rows, 8 models, CLI 1.0.70–1.0.79, schema_version 6): every
divergence was a rollup gap. Emitted calls mirror the shutdown-call
contract: input/cache/reasoning only, output 0 — per-turn output stays
owned by the events.jsonl assistant.message calls.

Rollup-vs-store precedence is RECONCILED at serve time, per
(session, model), and only there. Both representations always parse and
cache; parseProviderSources aggregates the cached calls and, wherever
store rows exist for a (session, model), drops the rollup calls and
serves the rows plus per-leg RESIDUAL calls: each rollup leg subtracts
only the rows in its own interval — rows commit strictly before their
leg's shutdown line, so a leg at time T covers exactly the rows in
(previous leg's T, T] — and any remainder (per token component, floored
at zero) serves once at that leg's own timestamp. A store missing
requests a leg covered — adopted mid-session, rows pruned before ever
being read — therefore still serves that tail exactly once ON THAT
LEG'S DAY, a crash-tail row the rollup never saw can never cancel it,
and a complete store serves pure per-request granularity with every
residual retired to zero. The decision reads only cached contents, never discovery:
deleting or resetting the store changes nothing served, so finalized
daily history can never flip on an absence epoch; cached rows of a
deleted store remain the record until the 90-day orphan age-out (which
exempts still-discovered paths). The serve set is the one coherent
snapshot — nothing a writer does between discovery and a parse can
change what one pass sees — and read-time precedence heals persisted
duplication (stale epochs, runtimes without node:sqlite, restored
files) instead of preserving it, following the buildDurablePeriod
pattern.

Store rows and rollups carry supplementary accounting weight. A rollup
(or its residual) is aggregate accounting, never a request: zero
api-call/model-call/turn weight, tokens and cost fully retained. A
store row is one real request, but when it pairs with a served per-turn
call it is supplementary too; rows pair with same-model per-turn calls
by timestamp adjacency (monotone matching, tight 2-minute window — the
two are written at the same completion moment, and a wide window would
let a crash-only row pair against a neighbor whose own row is missing),
computed once over the FULL serve set so a date-range boundary that
separates a row from its call cannot double the request across adjacent
day queries. Only the unpaired rows — store-only requests, exactly
where crash-lost requests sit — count. Supplementary-only turns fold
into the nearest behavioral turn within 30 minutes; with no behavioral
turn to fold into they stay separate weightless turns, each on its own
day, with apiCalls 0 — and the session emission gate admits
usage-bearing zero-call sessions. The weight
propagates into the daily cache: aggregateProjectsIntoDays applies the
same rule to every calls counter and category-turn count it seals, so
v19 history and live summaries can never disagree about what was a
request.

A changed source whose read defers on the busy shape (locked, EACCES,
corrupt mid-replace — discovery still emits the source; only true
absence or a schema mismatch reads as absent) now marks session
hydration incomplete, so the daily backfill holds its watermark instead
of finalizing a day the deferred rows never reached; an unchanged
unreadable store defers nothing. The verdict travels with its result —
the 180s memo and the serve burst-reuse restore the hydration verdict
their cached data was parsed under, so a memoized partial parse cannot
inherit a later parse's complete — and a discovered source whose
FINGERPRINT cannot be read (EACCES on a present file) defers instead of
silently skipping, while a genuinely deleted file stays a silent skip. Copilot reasoning tokens are no longer
double-billed at the report layer: they are a subset of the output the
per-turn calls already price, and copilot joins claude in the
reasoning-inside-output case of the query-time cost recompute.

Store dedup keys are content-discriminated —
copilot-store:<sid>:<rowId>:<fnv1a64(created_at|tokens|model)> —
because AUTOINCREMENT prevents id reuse only within one database
lifetime: a same-path DB reset reusing row ids now mints new keys
instead of the durable union swallowing the new usage, while a
byte-identical re-insert still collapses (64-bit: 32-bit FNV
collisions between plausible token tuples are constructible). Every
call of a session serves under one project label resolved at serve
time — the session-state-derived label when the serve set knows it,
else the store rows' own — so neither rows cached before events.jsonl
existed nor an events.jsonl orphaned by a session-state prune can
split the session across two grouping keys.

CODEBURN_COPILOT_SESSION_STORE_DB is read but deliberately NOT
fingerprinted, per the #927 ruling (any copilot fingerprint change
drops cached entries whose path still exists, destroying pruned history
only the cache holds); the read is allowlisted in the #927 guard, and
serve-time reconciliation makes repointing safe without a fingerprint —
the new store's rows parse on sight and the old path's entries persist
as durable orphans. The copilot parse version appends session-store-v2
and the daily cache bumps v17 → v19: per-day attribution, call counts
and costs all change against pre-store builds. 19, not 18: an earlier
pushed head of this PR already claimed v18 under different accounting,
and the carry-forward would adopt those days as finalized without
re-deriving them.

Verified by A/B on snapshots of two real stores, a live SIGKILL crash
test (row present, no rollup, tokens recovered exactly), live resumes
whose warm-cache deltas matched new rows to the token, upgrade-healing
at 4,800-session scale, and serve-level regressions pinning every
maintainer finding from six review rounds: the rows-then-shutdown race,
stale-cache healing, age-out exemption, absence-epoch identity,
progressive row landing with residual retirement, behavioral weight
across all four pinned scenarios, the hydration fence, project
unification in both directions, the same-path reset, mixed
coverage (crash tail vs covered-leg gap), multi-leg residual day
attribution, range-invariant pairing, memo-scoped hydration verdicts,
and the fingerprint-failure fence.
2026-08-17 17:52:01 -04:00
iamtoruk
0bfdfa372a perf(cache): shard the session cache per provider
A warm launch rewrote the entire session cache whenever any provider
appended a few KB: on a 6 GB corpus that is a 155 MB stringify + fsync
every run. The on-disk cache is now a version-suffixed directory holding
one shard per provider plus a small envelope, and a save rewrites only
the providers marked dirty.

- Dirtiness is tracked per provider (markCacheDirty) instead of one
  global flag, so an appended Claude session no longer republishes
  Codex, Copilot and the rest.
- Shards carry a nonce in their filename and the envelope is renamed
  last, so a save is published at a single point: readers never see a
  half-updated set, and a writer that loses the refresh ownership fence
  leaves the canonical shards untouched.
- A shard that fails validation is treated as an absent provider rather
  than rejecting the whole cache, so one malformed turn costs one
  provider's re-parse instead of every provider's history.
- v7 migrates losslessly: the blob is re-laid-out into shards and
  removed only once that save publishes. Nothing re-parses.
- Cold-parse progress saves now trigger every N files parsed rather than
  every 5s, so a slow cold parse no longer rewrites the growing cache on
  a wall clock.
2026-08-16 19:03:12 -07:00
Aditya Vikram Singh
d841ea59d7 fix: retain discovered durable history 2026-08-13 20:44:11 +05:30
ozymandiashh
a95a2c5bf8 fix(desktop): close cache and lifecycle review gaps 2026-08-12 20:31:17 +03:00
iamtoruk
555a1a89f5 perf(serve): event-driven parse reuse — a no-change fetch costs nothing
The remaining warm-serve cost was the per-request discovery sweep
(stat-ing thousands of session files) plus re-aggregation, even when
nothing on disk had changed. Serve now watches every provider's
probeRoots() via fs.watch (FSEvents-backed recursive watches on macOS)
and injects a quiet-since validator into the parser: while the watched
roots are quiet, a previous parse stays reusable past the burst window,
and an output-level memo returns identical panel queries verbatim - so a
fetch with no data changes skips the sweep AND the aggregation.

Safety rails, in order: a parse is validated-reusable only if the
watchers were armed before it ran; any filesystem event ends reuse
instantly; a 5-minute hard cap self-heals a missed event; a root that
fails to watch just goes uncovered (shorter reuse, never staleness);
outside serve the validator is never installed and behavior is
byte-identical. During an active AI session the session roots fire
constantly, so reuse correctly stays inside the 10s burst window - the
extended reuse serves the idle-browsing case it was built for.

The one watched path inside the cache dir is antigravity's statusline
file specifically, so serve's own cache writes never self-invalidate.
2026-08-10 14:04:01 -07:00
iamtoruk
d78ab77d96 perf: resident serve process for the desktop app — panel fetches in milliseconds
Every CLI spawn on a large corpus pays seconds of fixed cost before any
query work: node boot, a 100MB+ session-cache JSON.parse, the discovery +
fingerprint sweep, and serve-time classification. The desktop app spawns
one CLI per panel fetch, so it pays that cost per panel.

codeburn serve --stdio is the same CLI kept warm: the app holds one child,
sends {id, args} per line, and gets the command's stdout back. Three layers
make it fast, each disabled outside serve so one-shot runs stay byte-exact:

- loadCache memo (session-cache.ts): the parsed cache object is reused
  while a stat() shows the file unchanged; saveCache updates it
  write-through. A rewrite by another process still forces a fresh read.
- burst reuse (parser.ts, CODEBURN_PARSE_BURST_MS, serve sets 10s): panel
  bursts anchor their range ends at their own new Date(), so the exact-key
  memo never hits in real traffic; within the window a re-anchored range is
  served by trimming the previous parse instead of re-running discovery.
- fresh commander program per request (main.ts buildProgram factory),
  because commander option state is sticky across parses.

The server allows only the app's read queries (status/overview/models/
sessions/compare/yield/spend/optimize/audit), refuses everything else
(client falls back to a spawn), serializes requests, and converts
process.exit into a caught signal. The app starts the child once at
startup; requests route through it only when warm, cold-start keeps the
spawn path with its progress events, any serve failure falls back to a
spawn, and three child deaths disable serve for the app run.

Measured on a real 17B-token corpus: panel fetches drop from ~7.4s per
spawn to 5-900ms warm (sessions/spend 5ms, status 898ms). One-shot CLI
output verified byte-identical against the pre-branch baseline.
2026-08-10 09:43:44 -07:00
Matthew Kelch
3536a1d3ac
fix(copilot): classify CLI sessions by source provenance, not producer (#945)
Some checks failed
CI / semgrep (push) Has been cancelled
Tests / test (push) Has been cancelled
Fixes #944.
2026-08-09 04:51:44 +03:00
ozymandiashh
bcf1155255 test: fix the five environment-sensitive failures the first ubuntu run exposed
- cli-durable-totals: the live fixture session was stamped at noon today,
  so every before-noon run saw it in the future; the provider-filtered
  path drops future instants while the all-provider path keeps the whole
  day, failing the parity assertion. Relative-and-clamped timestamps,
  the same fix project-filter-durable-totals got in 1596220.
- parser (copilot, 2 cases): the fixture's fixed 2026-05-01 dates crossed
  copilot's durable 90-day age-out on 2026-07-30, so the first parse
  pruned the freshly-cached session. Relative timestamps.
- parser-incremental-append: unlink-then-create let ext4 hand the freed
  inode straight back, breaking the new-inode premise. The replacement
  is now created beside the original and renamed over it.
- parser-proxy-pricing: normalizeProxyPath folds case only on darwin and
  win32, deliberately; the test now asserts the platform-correct
  behavior on both kinds of filesystem instead of hardcoding macOS.
- cli-status-menubar: the config-source filter case does real multi-parse
  work and needs more than the 5s default on shared runners; 30s cap.
2026-08-04 03:38:18 +03:00
ozymandiashh
6b903a7777 fix(parser): stop --provider filters from re-surfacing cached claude sessions
Claude is scanned via scanProjectDirs instead of parseProviderSources, and
that call had no provider-filter guard. On a --provider <other> run
discoverAllSessions correctly returns no claude sources, so claudeDirs is
empty, but scanProjectDirs still ran: its orphan pass reads the whole cached
claude section and treats every file as no-longer-discovered, re-injecting
PR-bearing entries (and in read-only mode every cached entry) into the result.

The headline stayed correct because it comes from the provider-sliced daily
cache, so only the live-parse panels were wrong. By Model then listed
Anthropic models under --provider cursor while the total showed cursor alone.

Guard the scan with claudeInScope, mirroring the guard the durable-orphan
loop already applies. Deliberately not a claudeDirs.length check: when claude
is in scope but every transcript has been pruned, the orphan pass is what
keeps PR-attributed spend from vanishing.
2026-08-02 19:45:16 +03:00
ozymandiashh
56e836fa9e fix(classifier): attribute skills to skillBreakdown regardless of turn category
Skill attribution was gated on the turn classifying as 'general', but
classifyByToolPattern checks hasSkill last, so any turn invoking the Skill
tool alongside Read/Bash/Edit/MCP got a real category and its skill was
silently dropped. Skill turns virtually always carry other tools, which is
why skills came back empty in all 25 usage snapshots (#741) while mcpServers
populated.

subCategory is now set for any turn with Skill tool_use; category assignment
is untouched. The claude parse version is bumped so cached session summaries
(which lack skill attribution) re-parse on upgrade.
2026-07-18 22:54:49 +03:00
iamtoruk
300e949951 feat(app): refresh cadence, instant switching, cache coherence, telemetry v1
Performance and coherence:
- Settings > General 'Refresh every' (Manual/30s/1m/3m/5m/10m), live via
  RefreshCadenceContext; Manual polls only on demand
- usePolled memoKey LRU: provider/period switches paint the last-good
  result instantly with a switching hairline while refreshing quietly
- quota TTL 5min + honest rate-limited copy on 429 backoff
- version-suffixed cache files (session-cache.v5.json, daily-cache.v12,
  auto-minted on future bumps); legacy files never written or deleted,
  adopted once when versions match: old and new binaries coexist
  without clobbering (field-observed menubar-vs-desktop ping-pong)
- advisory hydration lock: concurrent cold starts share one scan
  (wait-then-read-warm), stale/dead locks self-heal, never a
  correctness gate

Telemetry v1 + onboarding (desktop only, per owner decisions):
- first-launch onboarding (3 feature screens + consent); region-split
  default (EU/EEA/UK/CH off, elsewhere on, unknown off); nothing sends
  before consent completion or while off; dev builds never send
- anonymous install UUID, rotated on opt-out; day-granularity events,
  cost buckets only; whitelisted names; 200-event queue, 5min flush
- Settings > Privacy live toggle replaces the static claim

Zero computed-number changes. App 316/316, root 1805.

Wire contract targets api.codeburn.app/v1/telemetry (Worker follows).
2026-07-16 11:47:05 -07:00
Resham Joshi
98051b4263
fix(copilot): bill CLI input and cache tokens from session.shutdown (#684)
* fix(copilot): bill CLI input and cache tokens from session.shutdown

Copilot CLI sessions (~/.copilot/session-state/*/events.jsonl) recorded
only outputTokens, because the parser read assistant.message events and
ignored the session.shutdown rollup. That rollup is the sole on-disk
source of a CLI session's input, cache-read and cache-write tokens, so
cost was underreported (in the reported case 35k+ cache-read tokens were
dropped entirely).

Handle session.shutdown in the CLI (non-transcript) JSONL path and emit
one supplementary call per model from modelMetrics.<model>.usage. The
per-turn assistant.message events keep owning output tokens, tools and
turn structure; the shutdown call contributes only the input and cache
tokens they lack, with output excluded, so no dimension double-counts and
the combined cost equals the full CLI-measured cost. usage.inputTokens is
cache-inclusive (input + cache_read + cache_write), so the cache
components are subtracted to recover the uncached input calculateCost
expects. These are real counts written by the CLI, so costIsEstimated is
false. Sessions without session.shutdown (crashed or still running) keep
the exact previous output-only behavior, and the VS Code, JetBrains and
OTel paths are untouched.

Bump the copilot entry in PROVIDER_PARSE_VERSIONS so already-cached CLI
sessions re-parse and pick up the corrected tokens and cost.

* fix(copilot): never emit an empty-timestamp shutdown call

A session.shutdown without its own timestamp and without sessionStartTime
produced a call with an empty timestamp, which the date-range filters in
parser.ts drop silently, erasing exactly the tokens this feature bills.
Fall back to the last stamped event in the session.

* fix(cache): durable orphans survive provider fingerprint changes

Bumping a durable provider's PROVIDER_PARSE_VERSIONS changed its env
fingerprint, and getOrCreateProviderSection replaced the whole section on
mismatch. For copilot OTel orphans the discarded cache entry is the only
remaining record of pruned spans, so the first post-upgrade run silently
and permanently erased month-to-date history. Carry forward exactly the
entries whose source no longer exists; present sources still drop and
re-parse under the new fingerprint, which is the point of a bump.

Regression test forges a pre-upgrade fingerprint on a pruned-source
orphan and asserts it stays counted and cached across the transition.
2026-07-16 10:37:58 -07:00
Tiago Santos
75c32e6d65
fix: fix and improve test isolation and collision with environment (#530)
* fix: fix and improve test isolation and collision with environment

* docs: remove unnecessary comment

* test(env-isolation): clear CODEBURN_FORCE_MACOS_MAJOR and pin TZ

Two env vars read in src/ were not isolated: CODEBURN_FORCE_MACOS_MAJOR
(now cleared so it cannot leak between tests) and TZ (now pinned to UTC,
since clearing it falls back to the OS zone and would shift date buckets
versus a clean CI runner).

---------

Co-authored-by: AgentSeal <hello@agentseal.org>
2026-06-20 13:42:10 +02:00
steelp02
d68a214c76
fix(copilot): Implement durable sources for OTel DB parsing.
Signed-off-by: steelp02 <pieter.steel@pfizer.com>
2026-06-17 21:42:40 +02:00