codeburn/mac/Sources/CodeBurnMenubar/Data/ServeConnection.swift
Resham Joshi 2fc62f7876
0.9.25: correctness, energy, menu bar and installer fixes (#1511)
* 0.9.25: correctness + provider fixes, energy guard, logo, default DMG

CLI/engine fixes: local-model savings count reasoning; ZCode reasoning billed;
codex-credits version-boundary tiering; GLM alias repointed to z.ai discounted rate;
openclaw/pi/qwen timestamp mtime fallback; hermes delta timestamp;
sqlite/warp/hermes busy+permission reclassified as failed (no sealed empty);
codex-cache dev+ino; cursor-cache WAL; antigravity server TTL; quota timeout+
grokbot footer+claude near-expiry; serve RSS guard env+hysteresis; CSV export
subfolder; providers pane fixed window; Warp credit-based cost.
Desktop/menubar: unified flame logo; default DMG; docs signed+notarized.

* menubar: one running instance, self-relaunch on language change

The single-instance guard only retired strictly older copies, so two login
items starting in the same instant both survived, and a copy whose bundle
identity did not survive install was never seen at all. The decision is now
total (newest launch wins, ties break on pid) and peers are matched on bundle
id or executable name. --keep-both opts one launch out.

Language changes no longer have the desktop app quit and reopen the menu bar,
which made macOS attribute the new process to the desktop app and re-ask for
the data-access grant. The menu bar relaunches itself via the existing remote
command key; older menu bars that do not watch the key fall back to the old
path after 5s.

* desktop: export toast shows the saved path; resource buckets on app_close

The export toast showed the folder that was picked, not the dated subfolder
the files land in. The handler already parsed the CLI's saved line; it now
returns that path and the toast uses it.

app_close carries cpu/mem (and serveCpu/serveMem when serve reported) as bucket
labels, sampled on the existing flush interval. serve adds cpuSec and peak rssMb
to its answer frames. Unmeasured figures are omitted, never sent as zero.

* menubar guard never leaves zero instances; installer replaces in place; JSON export into folder

Guard: an unreadable launch date on either side could make two copies retire
each other. Unreadable peers now count only when their pid is lower, and a copy
that cannot read its own date retires lower pids only, so a disagreement keeps
at least one instance.

Installer: an existing bundle in /Applications or ~/Applications is replaced
where it lives instead of a second copy being added. The old bundle is moved
aside and restored on any failure; a cross-volume move falls back to a copy that
is re-verified with codesign. An unwritable location is never escalated: the
install goes to ~/Applications and the leftover is named.

Export: -o pointing at an existing folder writes the JSON inside it. Short
sessions omit the cpu buckets. setLanguage reopens the menu bar if its own
relaunch never comes back. AppRelaunch passes the bundle path as an argument.

* menubar guard orders by kernel start time; installer opens the copy that exists and recovers a killed install

Guard: launch dates could be unreadable and pid order can disagree with start
order after pid reuse, so two copies could still retire each other. Ordering now
uses the kernel-reported process start time, which both sides read identically.
A copy whose start time cannot be read is left out, so the worst case is two
menu bars, never none.

Installer: with the only copy in an unwritable /Applications, a plain run tried
to open a path that did not exist. It now opens the installed copy. A placement
killed mid-way is recovered on the next run, a concurrent install is refused,
and a failed restore names where the previous bundle is. Process matching uses
the exact executable name.

Export: a trailing separator names a folder. Telemetry cpu is measured from
telemetry start.

* installer: a recycled pid can no longer block installs; language change waits for the replacement process

An aside left by an earlier install counted as an active install whenever its
pid was live, so once that pid was reused every install was refused for good.
It now also has to be recent (ctime within 10 minutes), a pid we may not signal
is not ours, the refusal names the file, and an orphan that cannot be cleared is
reported instead of aborting the install.

setLanguage confirmed the relaunch with a matcher the dying process still
satisfied, so the reopen fallback never fired. It now waits for a pid that was
not running before.

* desktop: only the headline refreshes live; act, yield and the optimize scan move to slower tiers

An open window ran act report (a fresh CLI process each time), yield and an
optimize scan on every refresh. The headline poll now passes --no-optimize and
takes the snapshot path; act report refreshes on mount, on manual refresh and
every 10 minutes, yield every 5. The optimize scan runs at most once a day per
query scope, is stored in userData with its time, and is forced fresh by the
Optimize page or a manual refresh. Figures that are not live show their age,
and no stored scan means no savings clause rather than a zero.

On battery the live interval doubles; the chosen cadence stays the base.

* desktop: a stored optimize scan is only valid on the day it was computed; cadence changes no longer refetch

The scan cache is keyed on the argv, which says 'today' or '30days', not a date,
so yesterday's scan was served as today's. It is now reused only on the same
local day. Stored findings in Signals carry their age, a failed scan shows the
error instead of scanning forever, and the cold-boot label uses the time the app
received the payload.

usePolled fetched immediately whenever its interval changed, so every power
transition (and launching on battery) refetched every live hook and could spawn
an extra act report. Fetching and timer arming are now separate effects.
Optimize's yield is on the same 5 minute tier as Overview.

* desktop: re-ask for the optimize scan on the first render of a new local day

An app left open across midnight kept yesterday's stored savings under Today,
because the scan was only requested on mount, scope change or manual refresh.
The local day is now one of the hook's triggers; the previous day's figure is
dropped while the new scan runs.

* serve: a menubar poll is never answered from the status snapshot

Inside its settle window the status snapshot returns the pre-change payload on
purpose. The resident serve process then memoized that answer as clean, so after
a single write followed by quiet the headline stayed on the old figure until the
next write or the five minute memo cap. The snapshot is for one-shot runs; the
resident process already holds the parsed state, so it now skips it.

* tests stop writing the real serve pid file; superseded caches are swept; installer notes reach the desktop card

ServeConnection takes its pid file as a parameter, so the Swift tests no longer
share ~/.cache/codeburn/menubar-serve.pid with a running menu bar, and the
reaper only signals a record that is a serve --stdio command.

Superseded session-cache, codex-results and cursor-results files (100MB+ each)
are removed once untouched for 14 days, at most once a day. daily-cache and the
hermes ledger are never swept: they carry sealed days a version bump adopts.

The menu bar installer's leftover-copy advice is emitted as a marked line under
CODEBURN_PROGRESS and shown on the desktop card. The skipped-refresh NSLog is
debug only. The app's vitest workers turn off Node's built-in web storage.

* menubar: anonymous usage statistics, under the same consent as the desktop and the Windows tray

The Swift menu bar sent nothing, so every codeburn-menubar row was a Windows
tray. It now follows the desktop's telemetry.v1.json when that exists (read
only: same install id, same on/off, onboarded required) and otherwise keeps its
own state with the same region default (off in EU/EEA/UK/CH and when the region
is unknown). Settings gains one Privacy toggle, disabled when the desktop owns
the choice. Events use the tray's names; the daily usage snapshot is the CLI's
block passed through a sanitizer and is left to the desktop when it is the
consent source. Debug builds never send.

A resident CLI that gets disabled for the run now says so in the log.

* menubar statistics: private network session, abandoned batch on opt-out, a desktop No survives the desktop going away

The sender used URLSession.shared, so a server or CDN cookie and the default
User-Agent rode along and would have outlived an opt-out. It now owns an
ephemeral session with no cookies, no cache, a fixed agent string and no
redirects. A batch that is out when the toggle goes off is dropped rather than
re-queued under the fresh id. The desktop's answer is recorded locally, so a
missing or damaged state file never falls back to the region default, and the
standalone id is persisted when minted. The toggle stays usable as a local off
switch under desktop consent. Copy states what this app actually sends.

* vercel gateway spend is shown on its own row and kept out of totals by default

The gateway reports daily aggregates per model, with nothing to match against
the local tools that were pointed at it, so every headline counted that traffic
twice. Totals now leave the gateway slice out when all providers are shown; the
row keeps its full amount and is marked not in total, --provider vercel-gateway
is unchanged, and codeburn gateway-totals include folds it back in. The daily
cache still stores the slice, so the choice applies retroactively. Reports with
no gateway credential are byte-identical.

The gateway's request_count now feeds the call count instead of each daily row
counting as one call. Hermes: a mid-session price change lands on the day it is
observed and sealed days are never rewritten; stated in the code and changelog.

* gateway exclusion applies to every report through the parse, and the detail panels follow the headline

The first pass narrowed only the day totals, so By tool, top models, top
projects, activities, status json projects and any project-filtered table still
carried gateway spend next to a headline that did not, and By tool shares could
pass 100%. The rule now lives in parseAllSessions: with all providers in scope
and the opt-in off, aggregate-only sessions are dropped after the memo, and the
callers that derive cached days opt out so stored slices are unchanged.
day.projects is reduced like models. The gateway stays visible once, as a
labelled not-in-total row with no share, and standalone reports note the
excluded amount on stderr so machine output is unchanged.

* dashboard: the background history index follows the gateway exclusion too

The index keeps the whole corpus so it can fill the daily cache, but its
per-period projection handed that unfiltered list to the project, model and
activity panels, so they regained gateway spend once the index landed.

* ci: a hung lock suite fails in 90s and retries once with a warning; slow Windows tests get measured limits

The cache-lock step failed 6 of the last 60 runs with no output for five minutes
after the vitest banner; passing runs take 12-18s on every Node version. Each
attempt is now capped at 90s and retried once, with a warning whenever that
happens. On windows-latest a single CLI spawn takes 1.4-12s, so the four-spawn
status snapshot test and the c6 parser suite get limits taken from those runs,
each with its reason. Both test jobs print their ten slowest cases.

* desktop: Privacy & data uses the same rows as the rest of Settings

The pane mixed two static claims with two controls in its own layout. It now
uses the about-row pattern every other pane uses: title and hint on the left,
one control on the right. The claims became the subtitle. Rows: anonymous usage
statistics (toggle, link to what is sent), device sharing state with Manage,
local report snapshots with Clear, export with Open export.

* desktop: Privacy & data keeps one bounded column and rows that never squeeze

The card stretched with the window, stranding each control hundreds of pixels
from its text. This pane's heading, subtitle and card now share a 760px column
and one text measure. Rows stack on their own width, because the window can
never be narrow enough for the old viewport rule to fire; row buttons do not
wrap. The what-is-sent link sits on its own line, and the snapshot and export
descriptions fit their measure and say this machine.

* menubar tests clean up their scratch defaults and leave the real Keychain alone

Four test files never removed their UserDefaults suites and the rest removed the
domain but left the plist, so thousands of files piled up in
~/Library/Preferences. A shared helper forgets a suite and deletes its file, and
every test that makes a suite now uses it. cfprefsd still rewrites an empty plist
when the test process exits, so a run leaves about a hundred 42-byte files with
no test data in them. The real-Keychain round trip only runs with
CODEBURN_TEST_REAL_KEYCHAIN=1.

* desktop: the resident parser retires after 15 idle minutes and restarts on demand

The renderer stops polling while the window is hidden, but the serve child stayed
resident with its parsed cache, one to two GB on a large corpus. After 15 minutes
without a request it is stopped the way a settings restart stops it, without
spending the unexpected-death budget, and the next request starts a new resident
through the existing lazy path, never a one-shot. CODEBURN_SERVE_IDLE_MS tunes or
disables it.

* desktop: every settings pane keeps a bounded column, and dark buttons have an edge

Cards stretched with the window, so on a wide display controls sat up to 1300px
from their labels and intro lines ran past 140 characters. All panes now share
one left-aligned column (760px, 980px for projects, aliases and pricing, which
hold wide rows) and one text measure. Rows stack on their own width instead of a
viewport rule the 900px minimum window could never reach, and a long path in
Projects no longer overflows the pane.

In dark mode the secondary button matched or sat below its surface, and was
invisible on the plugin install dialog. It gets a face and a border from two new
tokens; light is unchanged. The onboarding consent label matches Settings.

* desktop: this device shows its fingerprint on its own line

The local name and the fingerprint are two adjacent inline spans, so they read as
one run of text.

* menubar: the resident parser retires after 15 idle minutes and restarts on demand

The menu bar kept its serve child, and the parsed cache with it, for the whole
run, even with the popover closed and nothing changing. After 15 minutes without
a request the child is retired without counting as a death, and the next request
starts a new resident through ensureStarted, never the one-shot fallback, whose
snapshot settle window would make the figure lag a whole tick. The window sits
above the longest gap between real requests, so a child stays warm while work is
happening. CodeBurnServeIdleSeconds tunes or disables it.

* serve: a poll re-checks only what the watcher saw change, and shard writes are coalesced

With a session being written, every poll re-listed about 2,400 directories and
re-fingerprinted about 6,000 transcripts, then rewrote the whole current-month
shard (49MB on a large corpus) and read it back on the next poll. The resident
process now reuses the previous listing and fingerprints for paths its root
watcher proves untouched, and falls back to the full sweep whenever the watcher
is missing, unhealthy, overflowed, armed after the sweep, does not cover the
path, the memo is older than 4.5 minutes, or the process was suspended. Shard
publication is held for up to 90s and flushed when stdin closes; a held write is
dropped rather than written over something another process published. One-shot
runs are unchanged. Active poll cost 0.71 to 0.33 CPU-s at a 30s cadence;
time-to-reflect unchanged at about 1.2s across append, new dir, new file, delete,
rename-over, in-flight, watcher overflow and unhealthy watcher.

Changelog: state the telemetry default as it is in this version.

* a retired parser gets to publish its held shards, and durable providers are never held

Both apps retired the serve child with SIGTERM before its stdin-close flush could
run, so every retire discarded the coalesced window. They now close stdin, give
the child a bounded grace to publish and exit, and signal it only if it is still
running; a settings restart does the same. Measured: flushed 0/5 before, 5/5
after, for both clients.

copilot's cache entries are the only record once Copilot prunes its own files, so
a held window that was dropped (retire, crash, kill -9) before the source was
pruned lost that spend for good. A window that a durable provider dirtied is
published on the poll that parsed it, as before coalescing existed. The RSS guard
publishes a held window before it clears the caches instead of discarding it.

* serve: release the lifetime parse after the fill, and make the collection real

The RSS guard dropped its memos and then called globalThis.gc, which exists only
under --expose-gc; no launcher passes it, so the collection had never run and the
freed pages never went back. On a cold start the resident idled at 2.2GB with
364MB of live heap: clearing every memo moved RSS by 0MB, a collection alone by
700MB.

After the background fill has sealed its summaries the process now publishes any
held shard window, drops the memos it can re-read from disk (the cold start had
loaded the cache with scope 'all', which then served every later poll), and
collects; it collects again a minute later for as long as a pass returns 64MB,
at most four times. Cold idle 2198MB to 700MB, warm 900MB to 781MB. Reports are
leaf-identical cold and warm, time-to-reflect unchanged, a released month scope
reloads in 0.3-0.5s.

* warp: the billed cost reaches reports and survives the cache

The provider derived a dollar cost from Warp's own billing record (provider cents,
charged usage, or credits), but warp was missing from the reported-cost
pass-through, so the cached call kept no cost and every report, cold or warm,
priced the conversation from its tokens. A fixture billed at $1.22 reported
$0.15. A call whose cost came from a billing record now keeps it; a call that
only has the token floor stores none and is still re-priced when prices change.

Warp entries re-parse once (provider parse version), and the daily cache moves to
v34 so days whose Warp source still exists re-derive; call counts do not move, so
nothing is double counted or dropped.

* openclaw and pi: a reported cost survives the cache like every other reported cost

Both providers read the cost their tool recorded for a message, and both were
missing from the reported-cost pass-through, so reports priced every call from
tokens instead. Fixtures reported $0.011 against $0.056 (openclaw) and $0.013
against $0.410 (pi). A call with a reported cost now keeps it; a call with none,
or an explicit zero, stores none and is still re-priced when prices change. omp
had been added to the list in #1150 and pi, the same parser, was never revisited.

The provider list is one named set, the unreachable providerCallToTurn is gone,
and both providers re-parse once; the v34 daily cache re-derives their days with
call counts unchanged.

* test: pin parse count per report and per resident poll
2026-09-21 00:16:21 +02:00

872 lines
39 KiB
Swift

import Darwin
import Foundation
/// A resident `codeburn serve --stdio` child, held so payload fetches skip the
/// per-spawn cost (node boot + a 100MB+ session-cache parse on large corpora,
/// seconds per fetch at the CLI level). Requests are JSON lines `{id, args}`;
/// replies are `{id, ok, output}`. Mirrors the desktop app's client contract:
///
/// - Only `status` payload queries route here; anything else spawns as before.
/// - The first real status request is also the warm-up. It may be written
/// before the child announces READY; the pipe buffers it until serve reads
/// stdin, avoiding a second one-shot process that parses the same cache.
/// - Transport/protocol failures fall back to the spawn path for that call;
/// resource-policy failures remain terminal. Three child deaths disable
/// serve for this app run.
/// - The child's stdin closing (app quit, even SIGKILL) ends the server loop
/// on the CLI side, so no orphan survives the menubar.
/// - A child with nothing to do for `idleSeconds` retires and gives its memory
/// back; the next request respawns it lazily through `ensureStarted()`.
actor ServeConnection {
static let shared = ServeConnection()
typealias ProcessFactory = ([String], QualityOfService) -> Process
typealias TimeoutSleep = @Sendable (UInt64) async throws -> Void
private struct QueuedRequest {
let token: Int
let args: [String]
let continuation: CheckedContinuation<Data, Error>
}
private struct ActiveRequest {
let token: Int
let id: Int
let args: [String]
let child: Process
}
private var process: Process?
private var stdinHandle: FileHandle?
/// Permanent refusal (explicit shutdown, or a binary that cannot be spawned
/// at all). Distinct from a spent death budget, which recovers with time.
private var disabled = false
private var lastDeathAt: Date?
private var nextId = 1
private var nextRequestToken = 1
private var queuedRequests: [QueuedRequest] = []
private var activeRequest: ActiveRequest?
private var pending: [Int: CheckedContinuation<Data, Error>] = [:]
private var requestTimeouts: [Int: Task<Void, Never>] = [:]
private var requestCeilings: [Int: Task<Void, Never>] = [:]
/// Silence window of the in-flight request, so any frame carrying its id can
/// re-arm the same budget. See `CLIWatchdog`.
private var requestSilence: [Int: UInt64] = [:]
private var timeoutOwners: [Int: Process] = [:]
private var responseBytes: [Int: Int] = [:]
private var deaths = 0
private var buffer = Data()
private var receivedTerminalResponse = false
private var outputTasks: [ObjectIdentifier: Task<Void, Never>] = [:]
private var terminationTasks: [ObjectIdentifier: Task<Void, Never>] = [:]
/// At most one armed idle-retire timer. Re-armed after every quiet moment,
/// cancelled the instant a request is admitted.
private var idleTask: Task<Void, Never>?
private let makeProcess: ProcessFactory
private let timeoutSleep: TimeoutSleep
private let terminationGraceSleep: TimeoutSleep
private let idleSleep: TimeoutSleep
private let idleSeconds: Double
private let responseLimitBytes: Int
private let pidFile: URL
private static let maxDeaths = 3
static let maxResponseBytes = 16 * 1024 * 1024
private static let stdoutReadChunkBytes = 64 * 1024
private static let terminationGraceNanoseconds = nanoseconds(CLIWatchdog.killGraceSeconds)
/// Absolute backstop per request: a child that heartbeats forever without
/// ever answering is still reaped.
private static let requestCeilingNanoseconds = nanoseconds(CLIWatchdog.ceilingSeconds)
/// The death budget is spent by a transient cause as easily as a permanent
/// one (a cold-cache pile-up spends all three in a minute). Let a later tick
/// try again instead of leaving the resident dead for the whole app run.
private static let deathBudgetResetSeconds: TimeInterval = 300
/// Quiet period after which the resident is retired and its ~1GB RSS given
/// back; the next request respawns it through `ensureStarted()`.
///
/// Deliberately much longer than the slowest background tick (300s, and
/// `UsageDataChangeGuard` can skip ticks for up to 5 minutes). At a shorter
/// window every background tick during a coding session would be a cold
/// respawn — ~1.7s and ~2.5 CPU-seconds each — which trades energy for
/// memory. At 900s ticks keep the child warm while work is happening (the worst
/// gap between real requests is about 600s in Low Power Mode) and it
/// only goes away once the user has genuinely stopped.
static let defaultIdleSeconds: Double = 900
/// Calibration knob for a signed build: 0 or negative keeps the child
/// resident forever (the pre-retire behavior).
static let idleSecondsDefaultsKey = "CodeBurnServeIdleSeconds"
static func configuredIdleSeconds() -> Double {
let defaults = UserDefaults.standard
guard defaults.object(forKey: idleSecondsDefaultsKey) != nil else { return defaultIdleSeconds }
return defaults.double(forKey: idleSecondsDefaultsKey)
}
private static func nanoseconds(_ seconds: Double) -> UInt64 {
UInt64(seconds * 1_000_000_000)
}
struct ServeUnavailable: Error {}
enum FailureReason: Sendable, Equatable {
case generic
case outputTooLarge
}
struct ServeRequestFailed: Error, Sendable {
let message: String
let reason: FailureReason
init(message: String, reason: FailureReason = .generic) {
self.message = message
self.reason = reason
}
}
init(
// First so tests, which all pass `makeProcess`, cannot silently inherit
// the real user cache directory.
pidFile: URL = ServeOrphanReaper.pidFileURL(),
makeProcess: @escaping ProcessFactory = CodeburnCLI.makeProcess,
timeoutSleep: @escaping TimeoutSleep = { nanoseconds in
try await Task<Never, Never>.sleep(nanoseconds: nanoseconds)
},
terminationGraceSleep: @escaping TimeoutSleep = { nanoseconds in
try await Task<Never, Never>.sleep(nanoseconds: nanoseconds)
},
responseLimitBytes: Int = ServeConnection.maxResponseBytes,
idleSeconds: Double = ServeConnection.configuredIdleSeconds(),
idleSleep: @escaping TimeoutSleep = { nanoseconds in
try await Task<Never, Never>.sleep(nanoseconds: nanoseconds)
}
) {
self.pidFile = pidFile
self.makeProcess = makeProcess
self.timeoutSleep = timeoutSleep
self.terminationGraceSleep = terminationGraceSleep
self.idleSeconds = idleSeconds
self.idleSleep = idleSleep
precondition(responseLimitBytes > 0)
self.responseLimitBytes = responseLimitBytes
}
static func isEligible(_ subcommand: [String]) -> Bool {
subcommand.first == "status"
}
/// Kick the child off (idempotent). Called from app startup and again by
/// the first request in case the startup task has not run yet.
///
/// Re-entry: a spent death budget is a cooldown, not a life sentence. The
/// budget's job is to stop a crash loop from spawning a child per tick; once
/// `deathBudgetResetSeconds` have passed with no new death, the next tick
/// gets a fresh budget instead of leaving every fetch on the one-shot path
/// for the rest of the app run.
func ensureStarted() {
guard !disabled, process == nil else { return }
if deaths >= Self.maxDeaths {
guard let lastDeathAt,
Date().timeIntervalSince(lastDeathAt) >= Self.deathBudgetResetSeconds else { return }
deaths = 0
}
// Before adding a child, clear the one a crashed previous run (or an
// earlier generation of this one) left behind. Idempotent: the recorded
// pid is only signalled if it is still that exact serve command.
ServeOrphanReaper.reap(at: pidFile)
// This single resident serves both background and user-visible status
// requests. Its cold hydration replaces the old interactive one-shot,
// so keep the child at the same user-initiated QoS as visible fetches.
let child = makeProcess(["serve", "--stdio"], .userInitiated)
// Progress frames are what re-arm this connection's no-output watchdog
// during a cold hydration; without them serve is silent for minutes.
child.environment = CLIWatchdog.withProgressHeartbeat(child.environment)
let stdinPipe = Pipe()
let stdinWriter = stdinPipe.fileHandleForWriting
// Suppress SIGPIPE only for this connection's write end. A process-wide
// SIG_IGN leaks into unrelated libraries and children; F_SETNOSIGPIPE
// keeps a closed child stdin on the normal throwable EPIPE path.
guard Darwin.fcntl(stdinWriter.fileDescriptor, F_SETNOSIGPIPE, 1) == 0 else {
disabled = true
NSLog("CodeBurn: resident CLI disabled for this app run — could not suppress SIGPIPE on the child's stdin; every refresh now spawns a cold CLI")
return
}
let stdoutPipe = Pipe()
let stdoutReader = stdoutPipe.fileHandleForReading
child.standardInput = stdinPipe
child.standardOutput = stdoutPipe
child.standardError = FileHandle.nullDevice
do {
try child.run()
} catch {
disabled = true // spawn path can't produce the binary either better than makeProcess did
// Domain and code only: a Process launch error carries the path it
// tried, and the unified log is not the place for it.
let failure = error as NSError
NSLog("CodeBurn: resident CLI disabled for this app run — the serve child failed to launch (%@ %ld); every refresh now spawns a cold CLI", failure.domain, failure.code)
return
}
process = child
stdinHandle = stdinWriter
ServeChildRegistry.shared.add(child)
// Record the argv WITHOUT the `/usr/bin/env --` prefix: that is the part
// exec rewrites away before `ps` can see it. See serveCommandMatches.
ServeOrphanReaper.record(
at: pidFile,
pid: child.processIdentifier,
command: (child.arguments ?? []).drop(while: { $0 == "--" }).joined(separator: " ")
)
let generation = ObjectIdentifier(child)
// One blocking reader owns this generation's stdout. It never reads a
// second bounded chunk until the actor has consumed the first, giving
// the 16 MiB protocol limit real backpressure instead of accumulating
// an unbounded callback/AsyncStream backlog. EOF is observed only after
// the pipe's final bytes, so child death cannot overtake a split reply.
outputTasks[generation] = Task.detached { [weak self] in
var bytes = [UInt8](repeating: 0, count: Self.stdoutReadChunkBytes)
while !Task.isCancelled {
let count = Darwin.read(stdoutReader.fileDescriptor, &bytes, bytes.count)
if count > 0 {
guard let self else { break }
await self.consume(Data(bytes[0..<count]), from: child)
} else if count == -1, errno == EINTR {
continue
} else {
break
}
}
await self?.outputStreamEnded(for: child)
await self?.outputStreamFinished(for: child)
}
// Covers the eager start at app launch: a child nobody ever asks
// anything of must still be able to go away.
armIdleRetireIfQuiet()
}
/// Send the first real payload through the resident child. A request does
/// not need to wait for the READY frame: stdin is safe to write as soon as
/// Process.run() succeeds, and serve serializes it after initialization.
func request(args: [String]) async throws -> Data {
try Task.checkCancellation()
ensureStarted()
guard process != nil else { throw ServeUnavailable() }
let token = nextRequestToken
nextRequestToken += 1
let response = try await withTaskCancellationHandler {
try await withCheckedThrowingContinuation { continuation in
queuedRequests.append(QueuedRequest(
token: token,
args: args,
continuation: continuation
))
startNextRequestIfPossible()
}
} onCancel: {
Task { await self.cancelRequest(token: token) }
}
try Task.checkCancellation()
return response
}
func shutdown() {
disabled = true
deaths = Self.maxDeaths
cancelIdleRetire()
for task in terminationTasks.values { task.cancel() }
terminationTasks.removeAll()
cancelAllTimeouts()
failAllRequests()
try? stdinHandle?.close()
process = nil
stdinHandle = nil
buffer = Data()
receivedTerminalResponse = false
// Deliberately harder than every other kill path, and covering RETIRED
// generations too: quit has no budget to wait a SIGTERM grace out, and
// the grace task dies with the app, so a child that defers SIGTERM (a
// node process mid-synchronous-parse does) would outlive us. A refresh
// lock left behind by SIGKILL self-heals through the next parse's
// dead-pid takeover (src/cache-refresh-lock.ts).
ServeChildRegistry.shared.reapAll()
}
/// Test seam: from outside, a retired generation and one that was never
/// started look identical (no child), so the retire tests need this.
var residentChildForTesting: Process? { process }
// MARK: - internals
private func startNextRequestIfPossible() {
guard activeRequest == nil, !queuedRequests.isEmpty else { return }
ensureStarted()
guard let stdinHandle, let child = process else {
failQueuedRequests(error: ServeUnavailable())
return
}
// A Process can report not-running just before its termination callback
// reaches the ordered event stream. Keep the request queued for that
// event instead of writing to a generation which is already exiting.
guard child.isRunning else { return }
let request = queuedRequests.removeFirst()
let id = nextId
nextId += 1
let line: Data
do {
line = try JSONSerialization.data(withJSONObject: ["id": id, "args": request.args])
} catch {
request.continuation.resume(throwing: error)
startNextRequestIfPossible()
return
}
// The previous response can resume its caller just before EOF reaches
// this actor. Avoid admitting a successor to an already-reaped child;
// the reader's ordered EOF path will start it on a replacement.
guard child.isRunning else {
queuedRequests.insert(request, at: 0)
outputStreamEnded(for: child)
return
}
// Select and arm the timeout only when this request becomes the sole
// protocol request in flight. A queued request must not spend its own
// budget while its predecessor is still hydrating or draining. The
// budget bounds SILENCE: every frame carrying this id restarts it.
let timeoutNanoseconds = Self.nanoseconds(CLIWatchdog.silenceWindow(warm: receivedTerminalResponse))
cancelIdleRetire()
activeRequest = ActiveRequest(
token: request.token,
id: id,
args: request.args,
child: child
)
pending[id] = request.continuation
responseBytes[id] = 0
do {
try stdinHandle.write(contentsOf: line + Data("\n".utf8))
armTimeout(id: id, child: child, nanoseconds: timeoutNanoseconds)
} catch {
// The previous terminal frame can resume its caller just before
// EOF detaches that generation. Preserve this never-admitted
// request and retry it on the replacement instead of surfacing a
// transient EPIPE to the UI.
pending.removeValue(forKey: id)
responseBytes.removeValue(forKey: id)
activeRequest = nil
queuedRequests.insert(request, at: 0)
outputStreamEnded(for: child)
}
}
private func cancelRequest(token: Int) {
if let index = queuedRequests.firstIndex(where: { $0.token == token }) {
let request = queuedRequests.remove(at: index)
request.continuation.resume(throwing: CancellationError())
return
}
guard let activeRequest, activeRequest.token == token,
let continuation = pending.removeValue(forKey: activeRequest.id) else { return }
continuation.resume(throwing: CancellationError())
// Caller cancellation abandons only this response. The serialized serve
// child may still be doing the expensive first hydration, and killing it
// here lets tab switches and UI watchdogs restart that work indefinitely.
// Its independent request timeout remains armed: a command that never
// returns is still reaped, so it cannot wedge every later serialized call.
}
private func armTimeout(id: Int, child: Process, nanoseconds: UInt64) {
timeoutOwners[id] = child
requestSilence[id] = nanoseconds
armSilenceTimer(id: id, nanoseconds: nanoseconds)
// Deliberately a real sleep, not `timeoutSleep`: the ceiling is a
// 15-minute backstop behind the injected silence budget, and routing it
// through the same seam would make every test's timeout ledger carry it.
// Its arithmetic is covered by CLIWatchdog.verdict.
requestCeilings[id] = Task.detached { [weak self] in
do {
try await Task<Never, Never>.sleep(nanoseconds: Self.requestCeilingNanoseconds)
} catch {
return
}
await self?.requestTimedOut(id: id)
}
}
private func armSilenceTimer(id: Int, nanoseconds: UInt64) {
let sleep = timeoutSleep
requestTimeouts[id]?.cancel()
requestTimeouts[id] = Task.detached { [weak self] in
do {
try await sleep(nanoseconds)
} catch {
return
}
await self?.requestTimedOut(id: id)
}
}
/// Any frame carrying this request's id is proof of life: restart its
/// watchdog so a long cold parse that heartbeats progress is never killed
/// mid-flight. The absolute ceiling is untouched.
private func touchRequest(id: Int) {
guard let nanoseconds = requestSilence[id] else { return }
armSilenceTimer(id: id, nanoseconds: nanoseconds)
}
private func requestTimedOut(id: Int) {
guard let child = timeoutOwners.removeValue(forKey: id) else { return }
requestTimeouts.removeValue(forKey: id)
responseBytes.removeValue(forKey: id)
if let continuation = pending.removeValue(forKey: id) {
continuation.resume(throwing: ServeRequestFailed(message: "serve timeout"))
}
// The waiter may already have been abandoned by caller cancellation.
// Timeout ownership is deliberately independent of that continuation:
// kill only the exact generation that received the timed-out request.
guard process === child else {
if activeRequest?.id == id { activeRequest = nil }
startNextRequestIfPossible()
// Only a stale generation timed out; the live child survived this
// failure and is now idle again.
armIdleRetireIfQuiet()
return
}
// Retire the timed-out generation synchronously. Its stdout may never
// reach EOF (for example, a stuck child can ignore SIGTERM or a
// descendant can retain the pipe), so waiting for the reader would also
// spend every queued caller's timeout before it can even be admitted.
retireCurrentGeneration()
if activeRequest?.id == id { activeRequest = nil }
cancelTimeouts(ownedBy: child)
terminateTimedOutChild(child)
// The waiter was removed above and cannot be requeued by stale EOF.
// A queued read starts on a replacement immediately, subject to the
// ordinary three-death budget.
startNextRequestIfPossible()
}
private func cancelTimeout(id: Int) {
timeoutOwners.removeValue(forKey: id)
requestTimeouts.removeValue(forKey: id)?.cancel()
requestCeilings.removeValue(forKey: id)?.cancel()
requestSilence.removeValue(forKey: id)
responseBytes.removeValue(forKey: id)
}
private func cancelTimeouts(ownedBy child: Process) {
let ids = timeoutOwners.compactMap { id, owner in owner === child ? id : nil }
for id in ids { cancelTimeout(id: id) }
}
private func cancelAllTimeouts() {
for task in requestTimeouts.values { task.cancel() }
requestTimeouts.removeAll()
for task in requestCeilings.values { task.cancel() }
requestCeilings.removeAll()
requestSilence.removeAll()
timeoutOwners.removeAll()
responseBytes.removeAll()
}
/// Detach the current generation. Closing our end of its stdin is the app's
/// half of "stdin closing ends the server loop": dropping the FileHandle is
/// NOT enough, because the Pipe stays alive inside `child.standardInput` for
/// as long as this generation's reader or termination task holds the
/// Process, so the write end stays open and the retired child never sees
/// EOF. That is how a retired-but-alive serve child becomes an orphan.
private func retireCurrentGeneration(countsAsDeath: Bool = true) {
try? stdinHandle?.close()
process = nil
stdinHandle = nil
buffer = Data()
receivedTerminalResponse = false
guard countsAsDeath else { return }
deaths += 1
lastDeathAt = Date()
}
/// Hand the resident's memory back after `idleSeconds` of no work. Not a
/// death: the budget and its cooldown are untouched, so retiring all day
/// never costs the connection a life or delays a real crash's recovery.
private func retireForIdle() {
idleTask = nil
guard activeRequest == nil, queuedRequests.isEmpty, let child = process else { return }
retireCurrentGeneration(countsAsDeath: false)
cancelTimeouts(ownedBy: child)
flushThenTerminate(child)
}
/// Let a deliberately retired child exit on its own first. `retireCurrentGeneration`
/// has just closed our end of its stdin, and EOF is what makes `serve --stdio`
/// publish its coalesced shard window before exiting. SIGTERM in the same hop
/// kills that flush: the CLI catches it only to unlink its cache lock and then
/// re-raises (src/session-cache.ts). So wait out one grace and signal only a
/// child that is still running — reusing `terminationTasks`, so `shutdown()`
/// cancels this exactly like the termination grace it replaces and stays bounded.
private func flushThenTerminate(_ child: Process) {
guard child.isRunning else {
ServeChildRegistry.shared.remove(child)
return
}
let generation = ObjectIdentifier(child)
let sleep = terminationGraceSleep
terminationTasks[generation] = Task.detached { [weak self] in
do {
try await sleep(Self.terminationGraceNanoseconds)
} catch {
// Cancelled: either shutdown (which must not orphan a child that
// is ignoring EOF) or the child already died and its stream
// finished. Escalate either way; the isRunning guard inside makes
// the dead-child case a no-op.
await self?.forceKillAfterGrace(child)
return
}
await self?.terminateAfterFlushGrace(child)
}
}
private func terminateAfterFlushGrace(_ child: Process) {
terminationTasks.removeValue(forKey: ObjectIdentifier(child))
// Exited on its own inside the grace: it flushed, nothing left to signal.
guard child.isRunning else {
ServeChildRegistry.shared.remove(child)
return
}
terminateTimedOutChild(child)
}
/// Re-arm the idle window after every moment the connection falls quiet.
/// One task at a time; a sleeping Task costs nothing, a polling timer would.
private func armIdleRetireIfQuiet() {
cancelIdleRetire()
guard idleSeconds > 0, !disabled, process != nil,
activeRequest == nil, queuedRequests.isEmpty else { return }
let sleep = idleSleep
let nanoseconds = Self.nanoseconds(idleSeconds)
idleTask = Task { [weak self] in
do {
try await sleep(nanoseconds)
} catch {
return
}
// A cancellation that lands after the sleep has already returned is
// caught by retireForIdle's own quiet check, not here.
await self?.retireForIdle()
}
}
private func cancelIdleRetire() {
idleTask?.cancel()
idleTask = nil
}
private func outputStreamFinished(for child: Process) {
outputTasks.removeValue(forKey: ObjectIdentifier(child))
if !child.isRunning {
terminationTasks.removeValue(forKey: ObjectIdentifier(child))?.cancel()
ServeChildRegistry.shared.remove(child)
}
}
private func terminateTimedOutChild(_ child: Process) {
guard child.isRunning else { return }
child.terminate()
let generation = ObjectIdentifier(child)
let sleep = terminationGraceSleep
terminationTasks[generation] = Task.detached { [weak self] in
do {
try await sleep(Self.terminationGraceNanoseconds)
} catch {
// Cancellation means the owner stopped waiting: either shutdown
// (which must not orphan a SIGTERM-ignoring generation) or the
// child already died and the stream finished. Escalate either
// way; the isRunning guard makes the dead-child case a no-op.
await self?.forceKillAfterGrace(child)
return
}
await self?.forceKillAfterGrace(child)
}
}
private func forceKillAfterGrace(_ child: Process) {
terminationTasks.removeValue(forKey: ObjectIdentifier(child))
guard child.isRunning else {
ServeChildRegistry.shared.remove(child)
return
}
_ = Darwin.kill(child.processIdentifier, SIGKILL)
ServeChildRegistry.shared.remove(child)
}
private func outputStreamEnded(for child: Process) {
guard process === child else { return }
// EOF/read failure is a transport death even if the process has not
// reaped yet. Terminate that exact generation so a child which closed
// stdout cannot survive after the actor starts its replacement, and
// escalate: SIGTERM alone is deferred by a node child that is inside a
// synchronous parse when it arrives.
childDied(child)
terminateTimedOutChild(child)
}
// Internal so the generation guard can be exercised deterministically by
// tests without relying on Foundation callback scheduling at process exit.
func consume(_ data: Data, from child: Process) {
// A readability callback can already have queued its actor Task when the
// old process exits. If a replacement starts first, those late bytes must
// not repopulate the shared line buffer or mark the new child as warm.
guard process === child else { return }
var remaining = data[data.startIndex..<data.endIndex]
while !remaining.isEmpty {
if let newline = remaining.firstIndex(of: UInt8(ascii: "\n")) {
let fragment = remaining[remaining.startIndex..<newline]
guard fragment.count <= responseLimitBytes - buffer.count else {
outputOverflowed(child)
return
}
buffer.append(contentsOf: fragment)
let lineData = buffer
buffer = Data()
consumeLine(lineData, from: child)
guard process === child else { return }
remaining = remaining[remaining.index(after: newline)..<remaining.endIndex]
} else {
guard remaining.count <= responseLimitBytes - buffer.count else {
outputOverflowed(child)
return
}
buffer.append(contentsOf: remaining)
return
}
}
}
private func consumeLine(_ lineData: Data, from child: Process) {
guard !lineData.isEmpty,
let object = try? JSONSerialization.jsonObject(with: lineData) as? [String: Any] else { return }
if object["ready"] as? Bool == true {
return
}
guard let id = object["id"] as? Int,
responseBytes[id] != nil else { return }
// Desktop asks serve to stream cold-scan stderr as progress frames.
// Menubar has no progress UI, but must leave the request pending
// until the terminal response arrives if such a frame is emitted.
if let progress = object["progress"] as? String {
// Proof of life for a request still being worked on: restart its
// silence window. A terminal frame does not need this - it
// cancels the timers outright a few lines down.
touchRequest(id: id)
guard accountResponseBytes(Data(progress.utf8).count, id: id, child: child) else { return }
return
}
let succeeded = object["ok"] as? Bool == true
let payload = succeeded
? (object["output"] as? String)
: (object["error"] as? String)
if let payload {
guard accountResponseBytes(Data(payload.utf8).count, id: id, child: child) else { return }
}
// A refused/failed command can finish before any cache hydration.
// Only a successful terminal proves the resident is warm. Keep
// this before the waiter lookup so a successful orphan response
// still records the child as warm without resuming anything.
if succeeded { receivedTerminalResponse = true }
cancelTimeout(id: id)
let continuation = pending.removeValue(forKey: id)
if activeRequest?.id == id { activeRequest = nil }
if let continuation {
if succeeded, let output = object["output"] as? String {
continuation.resume(returning: Data(output.utf8))
} else {
let message = object["error"] as? String ?? "serve request failed"
continuation.resume(throwing: ServeRequestFailed(message: message))
}
}
// This also advances after an orphan terminal response whose caller
// was cancelled: cancellation removes only the waiter, not the
// active protocol lifecycle.
startNextRequestIfPossible()
armIdleRetireIfQuiet()
}
private func accountResponseBytes(_ count: Int, id: Int, child: Process) -> Bool {
guard let current = responseBytes[id],
count <= responseLimitBytes - current else {
outputOverflowed(child)
return false
}
responseBytes[id] = current + count
return true
}
private func outputOverflowed(_ child: Process) {
guard process === child else { return }
// Detach this exact generation before terminating it. Its eventual exit
// and any already-scheduled stdout callbacks are then stale and cannot
// consume a second death or corrupt a replacement generation.
retireCurrentGeneration()
cancelTimeouts(ownedBy: child)
failAllRequests(error: ServeRequestFailed(
message: "serve output exceeded \(responseLimitBytes) bytes",
reason: .outputTooLarge
))
terminateTimedOutChild(child)
}
private func childDied(_ child: Process) {
guard process === child else { return }
retireCurrentGeneration()
cancelTimeouts(ownedBy: child)
if let activeRequest, activeRequest.child === child {
if let continuation = pending.removeValue(forKey: activeRequest.id) {
// Only read-only status requests enter this connection. If a
// generation exits after admission but before its terminal
// reply, retain the waiter and retry on the replacement rather
// than racing it into a one-shot fallback. A timed-out or
// cancelled waiter is already absent and is never retried.
queuedRequests.insert(QueuedRequest(
token: activeRequest.token,
args: activeRequest.args,
continuation: continuation
), at: 0)
}
self.activeRequest = nil
}
// Requests which were never written survive an ordinary child crash.
// They begin on a replacement only after this ordered death event.
startNextRequestIfPossible()
}
private func failAllRequests(
error: Error = ServeRequestFailed(message: "serve exited")
) {
for (_, continuation) in pending {
continuation.resume(throwing: error)
}
pending.removeAll()
activeRequest = nil
failQueuedRequests(error: error)
}
private func failQueuedRequests(error: Error) {
let requests = queuedRequests
queuedRequests.removeAll()
for request in requests {
request.continuation.resume(throwing: error)
}
}
}
/// Best-effort reap of a serve child orphaned by a previous run: a crash leaves
/// no one to close the child's stdin and no `applicationWillTerminate` to reap
/// it. Port of `reapOrphanServe` in app/electron/cli.ts (#1096).
enum ServeOrphanReaper {
static func pidFileURL() -> URL {
URL(fileURLWithPath: CodeBurnCacheDirectory.resolve())
.appendingPathComponent("menubar-serve.pid", isDirectory: false)
}
static func record(at url: URL = pidFileURL(), pid: pid_t, command: String) {
try? FileManager.default.createDirectory(
at: url.deletingLastPathComponent(),
withIntermediateDirectories: true
)
let body = ["pid": Int(pid), "cmd": command] as [String: Any]
guard let data = try? JSONSerialization.data(withJSONObject: body) else { return }
try? data.write(to: url) // reaping is best-effort
}
/// Reads the pid recorded by the previous run and — because pids are
/// recycled — signals it only after `ps` confirms the process is still that
/// exact serve child. SIGTERM, never SIGKILL: the orphan may be holding the
/// cache refresh lock and can release it on the way out.
static func reap(at url: URL = pidFileURL()) {
defer { try? FileManager.default.removeItem(at: url) }
guard let data = try? Data(contentsOf: url),
let record = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
let pid = record["pid"] as? Int,
let cmd = record["cmd"] as? String,
pid > 1, pid != Int(ProcessInfo.processInfo.processIdentifier),
cmd.hasSuffix(serveArgvSuffix),
serveCommandMatches(recorded: cmd, observed: commandLine(of: pid_t(pid)))
else { return }
_ = Darwin.kill(pid_t(pid), SIGTERM)
}
/// The tail every `codeburn serve` argv ends with. A record that does not
/// carry it was not written for a serve child and is never signalled.
static let serveArgvSuffix = "serve --stdio"
/// Suffix match on the full recorded argv, not a keyword sniff: any looser
/// test signals whatever unrelated process inherited the pid.
///
/// Electron can compare exactly because it spawns its own binary. We spawn
/// through `/usr/bin/env`, and the CLI is a shebang script, so by the time
/// `ps` sees the process the argv has been rewritten twice: `env -- <cli>
/// serve --stdio` becomes `node <cli> serve --stdio`. The interpreter
/// prefix is the only part that varies, so the recorded `<cli> serve
/// --stdio` has to match as a suffix rather than in full.
static func serveCommandMatches(recorded: String, observed: String?) -> Bool {
guard let observed else { return false }
let normalize = { (value: String) -> String in
value.split(whereSeparator: { $0 == " " || $0 == "\t" || $0 == "\n" || $0 == "\"" })
.joined(separator: " ")
}
let wanted = normalize(recorded)
return !wanted.isEmpty && normalize(observed).hasSuffix(wanted)
}
/// `-ww` defeats ps's width truncation; the pid is an integer before it is
/// passed, and the call never goes through a shell.
private static func commandLine(of pid: pid_t) -> String? {
let process = Process()
process.executableURL = URL(fileURLWithPath: "/bin/ps")
process.arguments = ["-ww", "-o", "command=", "-p", String(pid)]
let pipe = Pipe()
process.standardOutput = pipe
process.standardError = FileHandle.nullDevice
guard (try? process.run()) != nil else { return nil }
// readToEnd returns at EOF, which is this short-lived child exiting; no
// waitUntilExit, which would park the calling thread (see DataClient).
let data = (try? pipe.fileHandleForReading.readToEnd()) ?? nil
let text = String(data: data ?? Data(), encoding: .utf8)?
.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
return text.isEmpty ? nil : text
}
}
/// Every serve child this app run has started and not yet seen exit, readable
/// WITHOUT an actor hop.
///
/// `applicationWillTerminate` runs `Task { await shutdown() }`, and the app can
/// exit before that task is ever scheduled — so the actor's own reaping is not
/// reachable on the path that matters most. This registry is, and a Process that
/// still reports `isRunning` has not been waited on, so its pid is still ours
/// and cannot have been recycled under us.
final class ServeChildRegistry: @unchecked Sendable {
static let shared = ServeChildRegistry()
private let lock = NSLock()
private var children: [ObjectIdentifier: Process] = [:]
func add(_ child: Process) {
lock.lock()
children[ObjectIdentifier(child)] = child
lock.unlock()
}
func remove(_ child: Process) {
lock.lock()
children.removeValue(forKey: ObjectIdentifier(child))
lock.unlock()
}
func reapAll() {
lock.lock()
let all = Array(children.values)
children.removeAll()
lock.unlock()
for child in all where child.isRunning {
_ = Darwin.kill(child.processIdentifier, SIGKILL)
}
}
}