mirror of
https://github.com/AgentSeal/codeburn.git
synced 2026-11-13 09:53:29 +00:00
* 0.9.25: correctness + provider fixes, energy guard, logo, default DMG CLI/engine fixes: local-model savings count reasoning; ZCode reasoning billed; codex-credits version-boundary tiering; GLM alias repointed to z.ai discounted rate; openclaw/pi/qwen timestamp mtime fallback; hermes delta timestamp; sqlite/warp/hermes busy+permission reclassified as failed (no sealed empty); codex-cache dev+ino; cursor-cache WAL; antigravity server TTL; quota timeout+ grokbot footer+claude near-expiry; serve RSS guard env+hysteresis; CSV export subfolder; providers pane fixed window; Warp credit-based cost. Desktop/menubar: unified flame logo; default DMG; docs signed+notarized. * menubar: one running instance, self-relaunch on language change The single-instance guard only retired strictly older copies, so two login items starting in the same instant both survived, and a copy whose bundle identity did not survive install was never seen at all. The decision is now total (newest launch wins, ties break on pid) and peers are matched on bundle id or executable name. --keep-both opts one launch out. Language changes no longer have the desktop app quit and reopen the menu bar, which made macOS attribute the new process to the desktop app and re-ask for the data-access grant. The menu bar relaunches itself via the existing remote command key; older menu bars that do not watch the key fall back to the old path after 5s. * desktop: export toast shows the saved path; resource buckets on app_close The export toast showed the folder that was picked, not the dated subfolder the files land in. The handler already parsed the CLI's saved line; it now returns that path and the toast uses it. app_close carries cpu/mem (and serveCpu/serveMem when serve reported) as bucket labels, sampled on the existing flush interval. serve adds cpuSec and peak rssMb to its answer frames. Unmeasured figures are omitted, never sent as zero. * menubar guard never leaves zero instances; installer replaces in place; JSON export into folder Guard: an unreadable launch date on either side could make two copies retire each other. Unreadable peers now count only when their pid is lower, and a copy that cannot read its own date retires lower pids only, so a disagreement keeps at least one instance. Installer: an existing bundle in /Applications or ~/Applications is replaced where it lives instead of a second copy being added. The old bundle is moved aside and restored on any failure; a cross-volume move falls back to a copy that is re-verified with codesign. An unwritable location is never escalated: the install goes to ~/Applications and the leftover is named. Export: -o pointing at an existing folder writes the JSON inside it. Short sessions omit the cpu buckets. setLanguage reopens the menu bar if its own relaunch never comes back. AppRelaunch passes the bundle path as an argument. * menubar guard orders by kernel start time; installer opens the copy that exists and recovers a killed install Guard: launch dates could be unreadable and pid order can disagree with start order after pid reuse, so two copies could still retire each other. Ordering now uses the kernel-reported process start time, which both sides read identically. A copy whose start time cannot be read is left out, so the worst case is two menu bars, never none. Installer: with the only copy in an unwritable /Applications, a plain run tried to open a path that did not exist. It now opens the installed copy. A placement killed mid-way is recovered on the next run, a concurrent install is refused, and a failed restore names where the previous bundle is. Process matching uses the exact executable name. Export: a trailing separator names a folder. Telemetry cpu is measured from telemetry start. * installer: a recycled pid can no longer block installs; language change waits for the replacement process An aside left by an earlier install counted as an active install whenever its pid was live, so once that pid was reused every install was refused for good. It now also has to be recent (ctime within 10 minutes), a pid we may not signal is not ours, the refusal names the file, and an orphan that cannot be cleared is reported instead of aborting the install. setLanguage confirmed the relaunch with a matcher the dying process still satisfied, so the reopen fallback never fired. It now waits for a pid that was not running before. * desktop: only the headline refreshes live; act, yield and the optimize scan move to slower tiers An open window ran act report (a fresh CLI process each time), yield and an optimize scan on every refresh. The headline poll now passes --no-optimize and takes the snapshot path; act report refreshes on mount, on manual refresh and every 10 minutes, yield every 5. The optimize scan runs at most once a day per query scope, is stored in userData with its time, and is forced fresh by the Optimize page or a manual refresh. Figures that are not live show their age, and no stored scan means no savings clause rather than a zero. On battery the live interval doubles; the chosen cadence stays the base. * desktop: a stored optimize scan is only valid on the day it was computed; cadence changes no longer refetch The scan cache is keyed on the argv, which says 'today' or '30days', not a date, so yesterday's scan was served as today's. It is now reused only on the same local day. Stored findings in Signals carry their age, a failed scan shows the error instead of scanning forever, and the cold-boot label uses the time the app received the payload. usePolled fetched immediately whenever its interval changed, so every power transition (and launching on battery) refetched every live hook and could spawn an extra act report. Fetching and timer arming are now separate effects. Optimize's yield is on the same 5 minute tier as Overview. * desktop: re-ask for the optimize scan on the first render of a new local day An app left open across midnight kept yesterday's stored savings under Today, because the scan was only requested on mount, scope change or manual refresh. The local day is now one of the hook's triggers; the previous day's figure is dropped while the new scan runs. * serve: a menubar poll is never answered from the status snapshot Inside its settle window the status snapshot returns the pre-change payload on purpose. The resident serve process then memoized that answer as clean, so after a single write followed by quiet the headline stayed on the old figure until the next write or the five minute memo cap. The snapshot is for one-shot runs; the resident process already holds the parsed state, so it now skips it. * tests stop writing the real serve pid file; superseded caches are swept; installer notes reach the desktop card ServeConnection takes its pid file as a parameter, so the Swift tests no longer share ~/.cache/codeburn/menubar-serve.pid with a running menu bar, and the reaper only signals a record that is a serve --stdio command. Superseded session-cache, codex-results and cursor-results files (100MB+ each) are removed once untouched for 14 days, at most once a day. daily-cache and the hermes ledger are never swept: they carry sealed days a version bump adopts. The menu bar installer's leftover-copy advice is emitted as a marked line under CODEBURN_PROGRESS and shown on the desktop card. The skipped-refresh NSLog is debug only. The app's vitest workers turn off Node's built-in web storage. * menubar: anonymous usage statistics, under the same consent as the desktop and the Windows tray The Swift menu bar sent nothing, so every codeburn-menubar row was a Windows tray. It now follows the desktop's telemetry.v1.json when that exists (read only: same install id, same on/off, onboarded required) and otherwise keeps its own state with the same region default (off in EU/EEA/UK/CH and when the region is unknown). Settings gains one Privacy toggle, disabled when the desktop owns the choice. Events use the tray's names; the daily usage snapshot is the CLI's block passed through a sanitizer and is left to the desktop when it is the consent source. Debug builds never send. A resident CLI that gets disabled for the run now says so in the log. * menubar statistics: private network session, abandoned batch on opt-out, a desktop No survives the desktop going away The sender used URLSession.shared, so a server or CDN cookie and the default User-Agent rode along and would have outlived an opt-out. It now owns an ephemeral session with no cookies, no cache, a fixed agent string and no redirects. A batch that is out when the toggle goes off is dropped rather than re-queued under the fresh id. The desktop's answer is recorded locally, so a missing or damaged state file never falls back to the region default, and the standalone id is persisted when minted. The toggle stays usable as a local off switch under desktop consent. Copy states what this app actually sends. * vercel gateway spend is shown on its own row and kept out of totals by default The gateway reports daily aggregates per model, with nothing to match against the local tools that were pointed at it, so every headline counted that traffic twice. Totals now leave the gateway slice out when all providers are shown; the row keeps its full amount and is marked not in total, --provider vercel-gateway is unchanged, and codeburn gateway-totals include folds it back in. The daily cache still stores the slice, so the choice applies retroactively. Reports with no gateway credential are byte-identical. The gateway's request_count now feeds the call count instead of each daily row counting as one call. Hermes: a mid-session price change lands on the day it is observed and sealed days are never rewritten; stated in the code and changelog. * gateway exclusion applies to every report through the parse, and the detail panels follow the headline The first pass narrowed only the day totals, so By tool, top models, top projects, activities, status json projects and any project-filtered table still carried gateway spend next to a headline that did not, and By tool shares could pass 100%. The rule now lives in parseAllSessions: with all providers in scope and the opt-in off, aggregate-only sessions are dropped after the memo, and the callers that derive cached days opt out so stored slices are unchanged. day.projects is reduced like models. The gateway stays visible once, as a labelled not-in-total row with no share, and standalone reports note the excluded amount on stderr so machine output is unchanged. * dashboard: the background history index follows the gateway exclusion too The index keeps the whole corpus so it can fill the daily cache, but its per-period projection handed that unfiltered list to the project, model and activity panels, so they regained gateway spend once the index landed. * ci: a hung lock suite fails in 90s and retries once with a warning; slow Windows tests get measured limits The cache-lock step failed 6 of the last 60 runs with no output for five minutes after the vitest banner; passing runs take 12-18s on every Node version. Each attempt is now capped at 90s and retried once, with a warning whenever that happens. On windows-latest a single CLI spawn takes 1.4-12s, so the four-spawn status snapshot test and the c6 parser suite get limits taken from those runs, each with its reason. Both test jobs print their ten slowest cases. * desktop: Privacy & data uses the same rows as the rest of Settings The pane mixed two static claims with two controls in its own layout. It now uses the about-row pattern every other pane uses: title and hint on the left, one control on the right. The claims became the subtitle. Rows: anonymous usage statistics (toggle, link to what is sent), device sharing state with Manage, local report snapshots with Clear, export with Open export. * desktop: Privacy & data keeps one bounded column and rows that never squeeze The card stretched with the window, stranding each control hundreds of pixels from its text. This pane's heading, subtitle and card now share a 760px column and one text measure. Rows stack on their own width, because the window can never be narrow enough for the old viewport rule to fire; row buttons do not wrap. The what-is-sent link sits on its own line, and the snapshot and export descriptions fit their measure and say this machine. * menubar tests clean up their scratch defaults and leave the real Keychain alone Four test files never removed their UserDefaults suites and the rest removed the domain but left the plist, so thousands of files piled up in ~/Library/Preferences. A shared helper forgets a suite and deletes its file, and every test that makes a suite now uses it. cfprefsd still rewrites an empty plist when the test process exits, so a run leaves about a hundred 42-byte files with no test data in them. The real-Keychain round trip only runs with CODEBURN_TEST_REAL_KEYCHAIN=1. * desktop: the resident parser retires after 15 idle minutes and restarts on demand The renderer stops polling while the window is hidden, but the serve child stayed resident with its parsed cache, one to two GB on a large corpus. After 15 minutes without a request it is stopped the way a settings restart stops it, without spending the unexpected-death budget, and the next request starts a new resident through the existing lazy path, never a one-shot. CODEBURN_SERVE_IDLE_MS tunes or disables it. * desktop: every settings pane keeps a bounded column, and dark buttons have an edge Cards stretched with the window, so on a wide display controls sat up to 1300px from their labels and intro lines ran past 140 characters. All panes now share one left-aligned column (760px, 980px for projects, aliases and pricing, which hold wide rows) and one text measure. Rows stack on their own width instead of a viewport rule the 900px minimum window could never reach, and a long path in Projects no longer overflows the pane. In dark mode the secondary button matched or sat below its surface, and was invisible on the plugin install dialog. It gets a face and a border from two new tokens; light is unchanged. The onboarding consent label matches Settings. * desktop: this device shows its fingerprint on its own line The local name and the fingerprint are two adjacent inline spans, so they read as one run of text. * menubar: the resident parser retires after 15 idle minutes and restarts on demand The menu bar kept its serve child, and the parsed cache with it, for the whole run, even with the popover closed and nothing changing. After 15 minutes without a request the child is retired without counting as a death, and the next request starts a new resident through ensureStarted, never the one-shot fallback, whose snapshot settle window would make the figure lag a whole tick. The window sits above the longest gap between real requests, so a child stays warm while work is happening. CodeBurnServeIdleSeconds tunes or disables it. * serve: a poll re-checks only what the watcher saw change, and shard writes are coalesced With a session being written, every poll re-listed about 2,400 directories and re-fingerprinted about 6,000 transcripts, then rewrote the whole current-month shard (49MB on a large corpus) and read it back on the next poll. The resident process now reuses the previous listing and fingerprints for paths its root watcher proves untouched, and falls back to the full sweep whenever the watcher is missing, unhealthy, overflowed, armed after the sweep, does not cover the path, the memo is older than 4.5 minutes, or the process was suspended. Shard publication is held for up to 90s and flushed when stdin closes; a held write is dropped rather than written over something another process published. One-shot runs are unchanged. Active poll cost 0.71 to 0.33 CPU-s at a 30s cadence; time-to-reflect unchanged at about 1.2s across append, new dir, new file, delete, rename-over, in-flight, watcher overflow and unhealthy watcher. Changelog: state the telemetry default as it is in this version. * a retired parser gets to publish its held shards, and durable providers are never held Both apps retired the serve child with SIGTERM before its stdin-close flush could run, so every retire discarded the coalesced window. They now close stdin, give the child a bounded grace to publish and exit, and signal it only if it is still running; a settings restart does the same. Measured: flushed 0/5 before, 5/5 after, for both clients. copilot's cache entries are the only record once Copilot prunes its own files, so a held window that was dropped (retire, crash, kill -9) before the source was pruned lost that spend for good. A window that a durable provider dirtied is published on the poll that parsed it, as before coalescing existed. The RSS guard publishes a held window before it clears the caches instead of discarding it. * serve: release the lifetime parse after the fill, and make the collection real The RSS guard dropped its memos and then called globalThis.gc, which exists only under --expose-gc; no launcher passes it, so the collection had never run and the freed pages never went back. On a cold start the resident idled at 2.2GB with 364MB of live heap: clearing every memo moved RSS by 0MB, a collection alone by 700MB. After the background fill has sealed its summaries the process now publishes any held shard window, drops the memos it can re-read from disk (the cold start had loaded the cache with scope 'all', which then served every later poll), and collects; it collects again a minute later for as long as a pass returns 64MB, at most four times. Cold idle 2198MB to 700MB, warm 900MB to 781MB. Reports are leaf-identical cold and warm, time-to-reflect unchanged, a released month scope reloads in 0.3-0.5s. * warp: the billed cost reaches reports and survives the cache The provider derived a dollar cost from Warp's own billing record (provider cents, charged usage, or credits), but warp was missing from the reported-cost pass-through, so the cached call kept no cost and every report, cold or warm, priced the conversation from its tokens. A fixture billed at $1.22 reported $0.15. A call whose cost came from a billing record now keeps it; a call that only has the token floor stores none and is still re-priced when prices change. Warp entries re-parse once (provider parse version), and the daily cache moves to v34 so days whose Warp source still exists re-derive; call counts do not move, so nothing is double counted or dropped. * openclaw and pi: a reported cost survives the cache like every other reported cost Both providers read the cost their tool recorded for a message, and both were missing from the reported-cost pass-through, so reports priced every call from tokens instead. Fixtures reported $0.011 against $0.056 (openclaw) and $0.013 against $0.410 (pi). A call with a reported cost now keeps it; a call with none, or an explicit zero, stores none and is still re-priced when prices change. omp had been added to the list in #1150 and pi, the same parser, was never revisited. The provider list is one named set, the unreachable providerCallToTurn is gone, and both providers re-parse once; the v34 daily cache re-derives their days with call counts unchanged. * test: pin parse count per report and per resident poll
1823 lines
80 KiB
Swift
1823 lines
80 KiB
Swift
import Darwin
|
|
import Foundation
|
|
import Testing
|
|
@testable import CodeBurnMenubar
|
|
|
|
private let ignoredSIGPIPEHandlerBits = unsafeBitCast(SIG_IGN, to: UInt.self)
|
|
private let coldTimeoutNanoseconds: UInt64 = 10 * 60 * 1_000_000_000
|
|
private let warmTimeoutNanoseconds: UInt64 = 45 * 1_000_000_000
|
|
private let terminationGraceNanoseconds: UInt64 = 5 * 1_000_000_000
|
|
|
|
/// Serve children are recorded to a pid file so a later run can reap an orphan.
|
|
/// Tests get a scratch path; the production default is the user's cache directory.
|
|
private func scratchPidFile() -> URL {
|
|
URL(fileURLWithPath: NSTemporaryDirectory())
|
|
.appendingPathComponent("serve-connection-pids", isDirectory: true)
|
|
.appendingPathComponent(UUID().uuidString)
|
|
}
|
|
|
|
private func currentSIGPIPEHandlerBits() -> UInt {
|
|
var action = sigaction()
|
|
_ = sigaction(SIGPIPE, nil, &action)
|
|
return unsafeBitCast(action.__sigaction_u.__sa_handler, to: UInt.self)
|
|
}
|
|
|
|
private actor TimeoutRecorder {
|
|
private var values: [UInt64] = []
|
|
|
|
func recordAndSleep(_ nanoseconds: UInt64) async throws {
|
|
values.append(nanoseconds)
|
|
// Cold timers stay pending until the fake child replies and the
|
|
// connection cancels them. The warm timer returns immediately to exercise
|
|
// the timeout path without a real one-minute wait.
|
|
if nanoseconds == warmTimeoutNanoseconds { return }
|
|
try await Task.sleep(nanoseconds: 5 * 1_000_000_000)
|
|
}
|
|
|
|
func recordAndWait(_ nanoseconds: UInt64) async throws {
|
|
values.append(nanoseconds)
|
|
// This recorder verifies timeout selection without firing the timeout.
|
|
// The response must deterministically win, then cancel this sleeper.
|
|
try await Task.sleep(nanoseconds: 5 * 1_000_000_000)
|
|
}
|
|
|
|
func snapshot() -> [UInt64] { values }
|
|
}
|
|
|
|
private actor FallbackRecorder {
|
|
private var calls = 0
|
|
|
|
func record() { calls += 1 }
|
|
func snapshot() -> Int { calls }
|
|
}
|
|
|
|
/// A cancellation-aware timeout clock that tests can advance explicitly. This
|
|
/// keeps the regression independent of the production ten-minute cold budget.
|
|
private actor ManualTimeoutClock {
|
|
private struct Waiter {
|
|
let nanoseconds: UInt64
|
|
let continuation: CheckedContinuation<Void, Error>
|
|
}
|
|
|
|
private var nextToken = 0
|
|
private var waiters: [Int: Waiter] = [:]
|
|
private var recorded: [UInt64] = []
|
|
private var fired = 0
|
|
|
|
func sleep(_ nanoseconds: UInt64) async throws {
|
|
let token = nextToken
|
|
nextToken += 1
|
|
recorded.append(nanoseconds)
|
|
try await withTaskCancellationHandler {
|
|
try await withCheckedThrowingContinuation { (continuation: CheckedContinuation<Void, Error>) in
|
|
if Task.isCancelled {
|
|
continuation.resume(throwing: CancellationError())
|
|
} else {
|
|
waiters[token] = Waiter(nanoseconds: nanoseconds, continuation: continuation)
|
|
}
|
|
}
|
|
} onCancel: {
|
|
Task { await self.cancel(token) }
|
|
}
|
|
}
|
|
|
|
func snapshot() -> [UInt64] {
|
|
waiters.keys.sorted().compactMap { waiters[$0]?.nanoseconds }
|
|
}
|
|
|
|
func history() -> [UInt64] { recorded }
|
|
|
|
/// How many timeouts this clock has let fire.
|
|
func firedCount() -> Int { fired }
|
|
|
|
func fireOldest() {
|
|
guard let token = waiters.keys.min(), let waiter = waiters.removeValue(forKey: token) else { return }
|
|
fired += 1
|
|
waiter.continuation.resume()
|
|
}
|
|
|
|
private func cancel(_ token: Int) {
|
|
guard let waiter = waiters.removeValue(forKey: token) else { return }
|
|
waiter.continuation.resume(throwing: CancellationError())
|
|
}
|
|
}
|
|
|
|
private final class QualityOfServiceRecorder: @unchecked Sendable {
|
|
private let lock = NSLock()
|
|
private var values: [QualityOfService] = []
|
|
|
|
func record(_ value: QualityOfService) {
|
|
lock.lock()
|
|
values.append(value)
|
|
lock.unlock()
|
|
}
|
|
|
|
func snapshot() -> [QualityOfService] {
|
|
lock.lock()
|
|
defer { lock.unlock() }
|
|
return values
|
|
}
|
|
}
|
|
|
|
private final class ProcessQueue: @unchecked Sendable {
|
|
private let lock = NSLock()
|
|
private var processes: [Process]
|
|
|
|
init(_ processes: [Process]) {
|
|
self.processes = processes
|
|
}
|
|
|
|
func take(qualityOfService: QualityOfService) -> Process {
|
|
lock.lock()
|
|
let child = processes.removeFirst()
|
|
lock.unlock()
|
|
child.qualityOfService = qualityOfService
|
|
return child
|
|
}
|
|
|
|
var remainingCount: Int {
|
|
lock.lock()
|
|
defer { lock.unlock() }
|
|
return processes.count
|
|
}
|
|
}
|
|
|
|
/// Real-time bound on waiting for an event. It is a hang guard for a
|
|
/// regression, never the assertion: every fixture that relies on it either
|
|
/// blocks until the test releases it or never finishes on its own, so a runner
|
|
/// starved by concurrently scheduled suites makes the wait longer without
|
|
/// changing the outcome (#1333).
|
|
private let hangGuardNanoseconds: UInt64 = 30 * 1_000_000_000
|
|
|
|
/// Polls a condition the test can only observe from outside the process, such
|
|
/// as a file a fixture child writes, until it holds or the hang guard expires.
|
|
private func eventually(_ condition: () async throws -> Bool) async rethrows -> Bool {
|
|
let deadline = DispatchTime.now().uptimeNanoseconds + hangGuardNanoseconds
|
|
while DispatchTime.now().uptimeNanoseconds < deadline {
|
|
if try await condition() { return true }
|
|
try? await Task.sleep(nanoseconds: 10_000_000)
|
|
}
|
|
return try await condition()
|
|
}
|
|
|
|
/// Awaits `operation`. If it is still pending when the hang guard expires,
|
|
/// `onExpiry` releases whatever it is blocked on, so a regression fails the
|
|
/// assertions that follow instead of hanging the test run.
|
|
private func withHangGuard<T>(
|
|
onExpiry: @escaping @Sendable () async -> Void,
|
|
_ operation: () async throws -> T
|
|
) async rethrows -> T {
|
|
let expiry = Task {
|
|
do {
|
|
try await Task.sleep(nanoseconds: hangGuardNanoseconds)
|
|
} catch {
|
|
return
|
|
}
|
|
await onExpiry()
|
|
}
|
|
defer { expiry.cancel() }
|
|
return try await operation()
|
|
}
|
|
|
|
private let idleWindowSeconds: Double = 600
|
|
private let idleWindowNanoseconds: UInt64 = 600 * 1_000_000_000
|
|
|
|
/// A serve fixture that answers every request line and records its pid. It
|
|
/// ignores SIGTERM on purpose, so a test that sees it exit cleanly knows the
|
|
/// connection closed its stdin rather than signalling it.
|
|
private func makeEchoFixture(pidsFile: String) -> Process {
|
|
let child = Process()
|
|
child.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
child.arguments = ["-c", """
|
|
trap '' TERM
|
|
printf '%s\n' "$$" >> "$1"
|
|
while IFS= read -r line; do
|
|
id=$(printf '%s' "$line" | sed -E 's/.*"id":([0-9]+).*/\\1/')
|
|
printf '{"id":%s,"ok":true,"output":"reply-%s"}\n' "$id" "$id"
|
|
done
|
|
""", "serve-fixture", pidsFile]
|
|
return child
|
|
}
|
|
|
|
/// A serve fixture that RECORDS every SIGTERM it receives into `signalsFile`
|
|
/// instead of dying from one, and exits on stdin EOF. `ignoreEOF` keeps it alive
|
|
/// past EOF, modelling a child that only a signal can end.
|
|
private func makeSignalRecordingFixture(pidsFile: String, signalsFile: String, ignoreEOF: Bool = false) -> Process {
|
|
let child = Process()
|
|
child.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
let tail = ignoreEOF ? "while :; do sleep 0.1; done" : "exit 0"
|
|
child.arguments = ["-c", """
|
|
trap 'printf T >> "$2"' TERM
|
|
printf '%s\n' "$$" >> "$1"
|
|
while IFS= read -r line; do
|
|
id=$(printf '%s' "$line" | sed -E 's/.*"id":([0-9]+).*/\\1/')
|
|
printf '{"id":%s,"ok":true,"output":"reply-%s"}\n' "$id" "$id"
|
|
done
|
|
\(tail)
|
|
""", "serve-fixture", pidsFile, signalsFile]
|
|
return child
|
|
}
|
|
|
|
private func recordedPids(_ pidsFile: String) throws -> [Substring] {
|
|
try String(contentsOfFile: pidsFile, encoding: .utf8).split(separator: "\n")
|
|
}
|
|
|
|
private func killIfRunning(_ processes: [Process]) {
|
|
for child in processes where child.isRunning {
|
|
_ = Darwin.kill(child.processIdentifier, SIGKILL)
|
|
}
|
|
}
|
|
|
|
/// An idle-window clock the test releases by hand. Deliberately NOT
|
|
/// cancellation-aware: the connection cancels its idle task whenever a request
|
|
/// is admitted, and releasing the sleep anyway is the only way to drive the
|
|
/// retire path into a busy connection.
|
|
private actor ReleaseGate {
|
|
private var continuations: [Int: CheckedContinuation<Void, Never>] = [:]
|
|
private var nextToken = 0
|
|
private var armed = 0
|
|
|
|
func wait(_ nanoseconds: UInt64) async {
|
|
armed += 1
|
|
let token = nextToken
|
|
nextToken += 1
|
|
await withCheckedContinuation { (continuation: CheckedContinuation<Void, Never>) in
|
|
continuations[token] = continuation
|
|
}
|
|
}
|
|
|
|
func armedCount() -> Int { armed }
|
|
|
|
/// The most recently armed window, i.e. the one the connection considers
|
|
/// live. Earlier ones belong to already-cancelled tasks.
|
|
func releaseNewest() {
|
|
guard let token = continuations.keys.max() else { return }
|
|
continuations.removeValue(forKey: token)?.resume()
|
|
}
|
|
|
|
func releaseAll() {
|
|
let all = Array(continuations.values)
|
|
continuations.removeAll()
|
|
for continuation in all { continuation.resume() }
|
|
}
|
|
}
|
|
|
|
@Suite("ServeConnection", .serialized)
|
|
struct ServeConnectionTests {
|
|
@Test("the resident child starts at user-initiated QoS")
|
|
func residentChildUsesInteractiveQoS() async {
|
|
let recorder = QualityOfServiceRecorder()
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
recorder.record(qualityOfService)
|
|
let child = Process()
|
|
child.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
child.arguments = ["-c", "while IFS= read -r line; do :; done"]
|
|
child.qualityOfService = qualityOfService
|
|
return child
|
|
})
|
|
|
|
await connection.ensureStarted()
|
|
|
|
#expect(recorder.snapshot() == [.userInitiated])
|
|
await connection.shutdown()
|
|
}
|
|
|
|
@Test("cancelling a hung request returns promptly")
|
|
func cancellationUnblocksPendingContinuation() async throws {
|
|
let dir = NSTemporaryDirectory() + "serve-connection-cancel-test-" + UUID().uuidString
|
|
try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true)
|
|
defer { try? FileManager.default.removeItem(atPath: dir) }
|
|
let requestMarker = dir + "/request-read"
|
|
let clock = ManualTimeoutClock()
|
|
|
|
// The child reads the request, then neither answers nor exits until its
|
|
// stdin closes, and the request's timeout fires only when the test fires
|
|
// it. Cancellation is the only event left that can resume the caller, so
|
|
// "promptly" is asserted as "before the timeout", not as a wall-clock
|
|
// budget a starved runner cannot keep (#1333). A child that exited on
|
|
// its own would instead be retried on replacements until the death
|
|
// budget surfaced ServeUnavailable.
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
let child = Process()
|
|
child.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
child.arguments = [
|
|
"-c", "IFS= read -r line; : > \"$1\"; while IFS= read -r line; do :; done",
|
|
"serve-fixture", requestMarker,
|
|
]
|
|
child.qualityOfService = qualityOfService
|
|
return child
|
|
},
|
|
timeoutSleep: { nanoseconds in
|
|
try await clock.sleep(nanoseconds)
|
|
}
|
|
)
|
|
|
|
let request = Task {
|
|
try await connection.request(args: ["status", "--format", "menubar-json"])
|
|
}
|
|
let childReadRequest = await eventually { FileManager.default.fileExists(atPath: requestMarker) }
|
|
#expect(childReadRequest)
|
|
let timeoutArmed = await eventually { await clock.snapshot() == [coldTimeoutNanoseconds] }
|
|
#expect(timeoutArmed)
|
|
|
|
request.cancel()
|
|
do {
|
|
// Firing the timeout on expiry turns a connection that ignores
|
|
// cancellation into a failed assertion below rather than a hang.
|
|
_ = try await withHangGuard(onExpiry: { await clock.fireOldest() }) {
|
|
try await request.value
|
|
}
|
|
#expect(Bool(false), "cancelled request unexpectedly succeeded")
|
|
} catch {
|
|
#expect(error is CancellationError)
|
|
}
|
|
#expect(await clock.firedCount() == 0)
|
|
await connection.shutdown()
|
|
}
|
|
|
|
@Test("a request queued during cancelled hydration completes on the same child")
|
|
func cancellationKeepsQueuedRequestOnResidentChild() async throws {
|
|
let dir = NSTemporaryDirectory() + "serve-connection-cancel-overlap-test-" + UUID().uuidString
|
|
try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true)
|
|
defer { try? FileManager.default.removeItem(atPath: dir) }
|
|
let pidsFile = dir + "/pids"
|
|
let eventsFile = dir + "/events"
|
|
let releaseMarker = dir + "/release-first"
|
|
let clock = ManualTimeoutClock()
|
|
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
let child = Process()
|
|
child.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
child.arguments = ["-c", """
|
|
printf '%s\n' "$$" >> "$1"
|
|
IFS= read -r first
|
|
first_id=$(printf '%s' "$first" | sed -E 's/.*"id":([0-9]+).*/\\1/')
|
|
printf 'first-read\n' >> "$2"
|
|
while [ ! -f "$3" ]; do sleep 0.01; done
|
|
printf '{"id":%s,"ok":true,"output":"late-%s"}\n' "$first_id" "$first_id"
|
|
printf 'late-first\n' >> "$2"
|
|
IFS= read -r second
|
|
second_id=$(printf '%s' "$second" | sed -E 's/.*"id":([0-9]+).*/\\1/')
|
|
printf 'second-read\n' >> "$2"
|
|
printf '{"id":%s,"ok":true,"output":"live-%s"}\n' "$second_id" "$second_id"
|
|
printf 'second-replied\n' >> "$2"
|
|
""", "serve-fixture", pidsFile, eventsFile, releaseMarker]
|
|
child.qualityOfService = qualityOfService
|
|
return child
|
|
},
|
|
timeoutSleep: { nanoseconds in
|
|
try await clock.sleep(nanoseconds)
|
|
}
|
|
)
|
|
|
|
let first = Task {
|
|
try await connection.request(args: ["status", "--request", "first"])
|
|
}
|
|
let firstRead = await eventually {
|
|
((try? String(contentsOfFile: eventsFile, encoding: .utf8)) ?? "").contains("first-read\n")
|
|
}
|
|
#expect(firstRead)
|
|
#expect(try String(contentsOfFile: eventsFile, encoding: .utf8) == "first-read\n")
|
|
// The timer is armed by a detached task, so it can register after the
|
|
// child has already read the line.
|
|
let coldTimeoutArmed = await eventually { await clock.history() == [coldTimeoutNanoseconds] }
|
|
#expect(coldTimeoutArmed)
|
|
|
|
first.cancel()
|
|
do {
|
|
_ = try await withHangGuard(onExpiry: {
|
|
Issue.record("cancellation was not observed while the child was still hydrating")
|
|
_ = FileManager.default.createFile(atPath: releaseMarker, contents: Data())
|
|
}) {
|
|
try await first.value
|
|
}
|
|
#expect(Bool(false), "cancelled request unexpectedly succeeded")
|
|
} catch {
|
|
#expect(error is CancellationError)
|
|
}
|
|
|
|
// Submit the next request while the child is still blocked hydrating
|
|
// the cancelled first one. It stays client-side queued: neither its
|
|
// stdin line nor its own timeout may begin yet. Nothing observable
|
|
// happens while it waits, so this settle can only miss a regression,
|
|
// never fail a correct run; the warm timeout asserted after the release
|
|
// is the deterministic proof that its line was written only after the
|
|
// late first reply.
|
|
let second = Task {
|
|
try await connection.request(args: ["status", "--request", "second"])
|
|
}
|
|
try await Task.sleep(nanoseconds: 100_000_000)
|
|
#expect(await clock.history() == [coldTimeoutNanoseconds])
|
|
#expect(try String(contentsOfFile: eventsFile, encoding: .utf8) == "first-read\n")
|
|
|
|
_ = FileManager.default.createFile(atPath: releaseMarker, contents: Data())
|
|
let secondPayload = try await withHangGuard(onExpiry: { await connection.shutdown() }) {
|
|
try await second.value
|
|
}
|
|
|
|
#expect(String(decoding: secondPayload, as: UTF8.self) == "live-2")
|
|
let warmTimeoutArmed = await eventually {
|
|
await clock.history() == [coldTimeoutNanoseconds, warmTimeoutNanoseconds]
|
|
}
|
|
#expect(warmTimeoutArmed)
|
|
let pids = try String(contentsOfFile: pidsFile, encoding: .utf8)
|
|
.split(separator: "\n")
|
|
#expect(pids.count == 1)
|
|
// The child logs "second-replied" after writing the reply that resumed
|
|
// `second`, so wait for the log rather than racing the child to it.
|
|
let secondReplied = await eventually {
|
|
((try? String(contentsOfFile: eventsFile, encoding: .utf8)) ?? "").contains("second-replied\n")
|
|
}
|
|
#expect(secondReplied)
|
|
let events = try String(contentsOfFile: eventsFile, encoding: .utf8)
|
|
.split(separator: "\n")
|
|
#expect(events == ["first-read", "late-first", "second-read", "second-replied"])
|
|
await connection.shutdown()
|
|
}
|
|
|
|
@Test("a cancelled never-returning request retains a timeout owner and cannot wedge later work")
|
|
func cancelledHungRequestIsEventuallyReaped() async throws {
|
|
let dir = NSTemporaryDirectory() + "serve-connection-cancel-timeout-test-" + UUID().uuidString
|
|
try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true)
|
|
defer { try? FileManager.default.removeItem(atPath: dir) }
|
|
let firstReadMarker = dir + "/first-read"
|
|
let pidsFile = dir + "/pids"
|
|
let clock = ManualTimeoutClock()
|
|
let graceClock = ManualTimeoutClock()
|
|
|
|
let stuckChild = Process()
|
|
stuckChild.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
stuckChild.arguments = ["-c", """
|
|
trap '' TERM
|
|
printf '%s\n' "$$" >> "$1"
|
|
IFS= read -r line
|
|
: > "$2"
|
|
while :; do :; done
|
|
""", "serve-fixture", pidsFile, firstReadMarker]
|
|
|
|
let replacement = Process()
|
|
replacement.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
replacement.arguments = ["-c", """
|
|
printf '%s\n' "$$" >> "$1"
|
|
while IFS= read -r line; do
|
|
id=$(printf '%s' "$line" | sed -E 's/.*"id":([0-9]+).*/\\1/')
|
|
printf '{"id":%s,"ok":true,"output":"replacement-%s"}\n' "$id" "$id"
|
|
done
|
|
""", "serve-fixture", pidsFile]
|
|
|
|
let children = ProcessQueue([stuckChild, replacement])
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
children.take(qualityOfService: qualityOfService)
|
|
},
|
|
timeoutSleep: { nanoseconds in
|
|
try await clock.sleep(nanoseconds)
|
|
},
|
|
terminationGraceSleep: { nanoseconds in
|
|
try await graceClock.sleep(nanoseconds)
|
|
}
|
|
)
|
|
defer {
|
|
if stuckChild.isRunning { _ = Darwin.kill(stuckChild.processIdentifier, SIGKILL) }
|
|
}
|
|
|
|
let abandoned = Task {
|
|
try await connection.request(args: ["status", "--request", "stuck"])
|
|
}
|
|
for _ in 0..<200 where !FileManager.default.fileExists(atPath: firstReadMarker) {
|
|
try await Task.sleep(nanoseconds: 10_000_000)
|
|
}
|
|
#expect(FileManager.default.fileExists(atPath: firstReadMarker))
|
|
#expect(await clock.snapshot() == [coldTimeoutNanoseconds])
|
|
|
|
abandoned.cancel()
|
|
do {
|
|
_ = try await abandoned.value
|
|
#expect(Bool(false), "cancelled request unexpectedly succeeded")
|
|
} catch {
|
|
#expect(error is CancellationError)
|
|
}
|
|
|
|
// The caller is gone, but the independently-owned cold timeout must
|
|
// remain armed. This assertion is the red-before regression: the old
|
|
// task-group race cancelled the only timeout along with the caller.
|
|
#expect(await clock.snapshot() == [coldTimeoutNanoseconds])
|
|
let successor = Task {
|
|
try await connection.request(args: ["status", "--request", "after-cancel"])
|
|
}
|
|
try await Task.sleep(nanoseconds: 100_000_000)
|
|
#expect(await clock.snapshot() == [coldTimeoutNanoseconds])
|
|
#expect(children.remainingCount == 1)
|
|
|
|
await clock.fireOldest()
|
|
for _ in 0..<200 where children.remainingCount > 0 {
|
|
try await Task.sleep(nanoseconds: 10_000_000)
|
|
}
|
|
let replacementStartedBeforeOldEOF = children.remainingCount == 0
|
|
#expect(replacementStartedBeforeOldEOF)
|
|
// Keep the red-before run finite: the old implementation waits for EOF
|
|
// forever because this fixture deliberately ignores SIGTERM.
|
|
if !replacementStartedBeforeOldEOF {
|
|
_ = Darwin.kill(stuckChild.processIdentifier, SIGKILL)
|
|
for _ in 0..<200 where children.remainingCount > 0 {
|
|
try await Task.sleep(nanoseconds: 10_000_000)
|
|
}
|
|
}
|
|
|
|
// The retired child ignores SIGTERM, yet its stale stdout remains open.
|
|
// The queued successor must already run on a replacement; it cannot wait
|
|
// for either old-generation EOF or the force-kill grace period.
|
|
let payload = try await successor.value
|
|
#expect(String(decoding: payload, as: UTF8.self) == "replacement-2")
|
|
#expect(await clock.snapshot().isEmpty)
|
|
#expect(await clock.history() == [coldTimeoutNanoseconds, coldTimeoutNanoseconds])
|
|
#expect(await graceClock.snapshot() == [terminationGraceNanoseconds])
|
|
#expect(stuckChild.isRunning)
|
|
#expect(try String(contentsOfFile: pidsFile, encoding: .utf8).split(separator: "\n").count == 2)
|
|
|
|
await graceClock.fireOldest()
|
|
for _ in 0..<200 where stuckChild.isRunning {
|
|
try await Task.sleep(nanoseconds: 10_000_000)
|
|
}
|
|
#expect(!stuckChild.isRunning)
|
|
#expect(stuckChild.terminationReason == .uncaughtSignal)
|
|
#expect(stuckChild.terminationStatus == SIGKILL)
|
|
await connection.shutdown()
|
|
}
|
|
|
|
@Test("shutdown during the termination grace force-kills the SIGTERM-ignoring generation")
|
|
func shutdownDuringGraceKillsStubbornChild() async throws {
|
|
let dir = NSTemporaryDirectory() + "serve-connection-shutdown-grace-test-" + UUID().uuidString
|
|
try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true)
|
|
defer { try? FileManager.default.removeItem(atPath: dir) }
|
|
let firstReadMarker = dir + "/first-read"
|
|
let clock = ManualTimeoutClock()
|
|
let graceClock = ManualTimeoutClock()
|
|
|
|
let stuckChild = Process()
|
|
stuckChild.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
stuckChild.arguments = ["-c", """
|
|
trap '' TERM
|
|
IFS= read -r line
|
|
: > "$1"
|
|
while :; do :; done
|
|
""", "serve-fixture", firstReadMarker]
|
|
|
|
let children = ProcessQueue([stuckChild])
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
children.take(qualityOfService: qualityOfService)
|
|
},
|
|
timeoutSleep: { nanoseconds in
|
|
try await clock.sleep(nanoseconds)
|
|
},
|
|
terminationGraceSleep: { nanoseconds in
|
|
try await graceClock.sleep(nanoseconds)
|
|
}
|
|
)
|
|
defer {
|
|
if stuckChild.isRunning { _ = Darwin.kill(stuckChild.processIdentifier, SIGKILL) }
|
|
}
|
|
|
|
let request = Task {
|
|
try await connection.request(args: ["status", "--request", "stuck"])
|
|
}
|
|
for _ in 0..<200 where !FileManager.default.fileExists(atPath: firstReadMarker) {
|
|
try await Task.sleep(nanoseconds: 10_000_000)
|
|
}
|
|
#expect(FileManager.default.fileExists(atPath: firstReadMarker))
|
|
#expect(await clock.snapshot() == [coldTimeoutNanoseconds])
|
|
|
|
// Time out the request: the generation is retired and SIGTERM'd, and the
|
|
// SIGKILL escalation parks on the injected grace clock.
|
|
await clock.fireOldest()
|
|
do {
|
|
_ = try await request.value
|
|
#expect(Bool(false), "timed-out request unexpectedly succeeded")
|
|
} catch let error as ServeConnection.ServeRequestFailed {
|
|
#expect(error.message == "serve timeout")
|
|
}
|
|
for _ in 0..<200 where await graceClock.snapshot().isEmpty {
|
|
try await Task.sleep(nanoseconds: 10_000_000)
|
|
}
|
|
#expect(await graceClock.snapshot() == [terminationGraceNanoseconds])
|
|
#expect(stuckChild.isRunning) // SIGTERM ignored; escalation still pending
|
|
|
|
// Shutdown must not merely cancel the escalation. The retired generation
|
|
// is already detached from `process`, so nothing else will reap it; the
|
|
// grace task's cancellation path has to SIGKILL it or it outlives the app.
|
|
await connection.shutdown()
|
|
for _ in 0..<200 where stuckChild.isRunning {
|
|
try await Task.sleep(nanoseconds: 10_000_000)
|
|
}
|
|
#expect(!stuckChild.isRunning)
|
|
#expect(stuckChild.terminationReason == .uncaughtSignal)
|
|
#expect(stuckChild.terminationStatus == SIGKILL)
|
|
}
|
|
|
|
@Test("timed-out generations consume one death each and stop at the resident budget")
|
|
func timeoutDeathBudgetIsExact() async throws {
|
|
let dir = NSTemporaryDirectory() + "serve-connection-timeout-budget-test-" + UUID().uuidString
|
|
try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true)
|
|
defer { try? FileManager.default.removeItem(atPath: dir) }
|
|
let readsFile = dir + "/reads"
|
|
let clock = ManualTimeoutClock()
|
|
let processes = (0..<3).map { _ in
|
|
let child = Process()
|
|
child.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
child.arguments = ["-c", """
|
|
trap '' TERM
|
|
IFS= read -r line
|
|
printf r >> "$1"
|
|
while :; do :; done
|
|
""", "serve-fixture", readsFile]
|
|
return child
|
|
}
|
|
let children = ProcessQueue(processes)
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
children.take(qualityOfService: qualityOfService)
|
|
},
|
|
timeoutSleep: { nanoseconds in
|
|
try await clock.sleep(nanoseconds)
|
|
},
|
|
terminationGraceSleep: { _ in }
|
|
)
|
|
defer {
|
|
for child in processes where child.isRunning {
|
|
_ = Darwin.kill(child.processIdentifier, SIGKILL)
|
|
}
|
|
}
|
|
|
|
for attempt in 0..<3 {
|
|
let request = Task {
|
|
try await connection.request(args: ["status", "--attempt", String(attempt)])
|
|
}
|
|
for _ in 0..<200 {
|
|
let reads = (try? String(contentsOfFile: readsFile, encoding: .utf8).count) ?? 0
|
|
if reads == attempt + 1, await clock.snapshot().count == 1 { break }
|
|
try await Task.sleep(nanoseconds: 10_000_000)
|
|
}
|
|
#expect((try? String(contentsOfFile: readsFile, encoding: .utf8).count) == attempt + 1)
|
|
await clock.fireOldest()
|
|
do {
|
|
_ = try await request.value
|
|
Issue.record("timeout \(attempt) unexpectedly succeeded")
|
|
} catch let error as ServeConnection.ServeRequestFailed {
|
|
#expect(error.message == "serve timeout")
|
|
}
|
|
}
|
|
|
|
#expect(children.remainingCount == 0)
|
|
do {
|
|
_ = try await connection.request(args: ["status", "--after-budget"])
|
|
Issue.record("resident restarted after three timed-out generations")
|
|
} catch {
|
|
#expect(error is ServeConnection.ServeUnavailable)
|
|
}
|
|
for _ in 0..<200 where processes.contains(where: \.isRunning) {
|
|
try await Task.sleep(nanoseconds: 10_000_000)
|
|
}
|
|
#expect(processes.allSatisfy { !$0.isRunning })
|
|
#expect(processes.allSatisfy {
|
|
$0.terminationReason == .uncaughtSignal && $0.terminationStatus == SIGKILL
|
|
})
|
|
await connection.shutdown()
|
|
}
|
|
|
|
@Test("external cancellations keep one child and safely discard late replies")
|
|
func cancellationsKeepResidentChildAlive() async throws {
|
|
let dir = NSTemporaryDirectory() + "serve-connection-cancel-reuse-test-" + UUID().uuidString
|
|
try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true)
|
|
defer { try? FileManager.default.removeItem(atPath: dir) }
|
|
let pidsFile = dir + "/pids"
|
|
let requestsFile = dir + "/requests"
|
|
let lateRepliesFile = dir + "/late-replies"
|
|
let clock = ManualTimeoutClock()
|
|
|
|
// Ids 1-3 are answered only once the test has created release-<id>, so
|
|
// the late reply cannot beat the cancellation it is meant to follow
|
|
// (#1333).
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
let child = Process()
|
|
child.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
child.arguments = ["-c", """
|
|
printf '%s\n' "$$" >> "$1"
|
|
while IFS= read -r line; do
|
|
printf r >> "$2"
|
|
id=$(printf '%s' "$line" | sed -E 's/.*"id":([0-9]+).*/\\1/')
|
|
if [ "$id" -le 3 ]; then
|
|
while [ ! -f "$4/release-$id" ]; do sleep 0.01; done
|
|
printf '{"id":%s,"ok":true,"output":"late-%s"}\n' "$id" "$id"
|
|
printf l >> "$3"
|
|
else
|
|
printf '{"id":%s,"ok":true,"output":"live-%s"}\n' "$id" "$id"
|
|
fi
|
|
done
|
|
""", "serve-fixture", pidsFile, requestsFile, lateRepliesFile, dir]
|
|
child.qualityOfService = qualityOfService
|
|
return child
|
|
},
|
|
timeoutSleep: { nanoseconds in
|
|
try await clock.sleep(nanoseconds)
|
|
}
|
|
)
|
|
|
|
for attempt in 0..<3 {
|
|
let request = Task {
|
|
try await connection.request(args: ["status", "--attempt", String(attempt)])
|
|
}
|
|
let requestRead = await eventually {
|
|
((try? String(contentsOfFile: requestsFile, encoding: .utf8))?.count ?? 0) >= attempt + 1
|
|
}
|
|
#expect(requestRead)
|
|
let releaseMarker = dir + "/release-\(attempt + 1)"
|
|
request.cancel()
|
|
do {
|
|
_ = try await withHangGuard(onExpiry: {
|
|
Issue.record("cancellation of attempt \(attempt) was not observed before its reply was released")
|
|
_ = FileManager.default.createFile(atPath: releaseMarker, contents: Data())
|
|
}) {
|
|
try await request.value
|
|
}
|
|
#expect(Bool(false), "cancelled request unexpectedly succeeded")
|
|
} catch {
|
|
#expect(error is CancellationError)
|
|
}
|
|
|
|
// Only now does the fake child emit the orphaned response. It must
|
|
// be ignored without double-resuming anything, and the same
|
|
// resident child must remain available for the next id.
|
|
_ = FileManager.default.createFile(atPath: releaseMarker, contents: Data())
|
|
let lateReplyWritten = await eventually {
|
|
((try? String(contentsOfFile: lateRepliesFile, encoding: .utf8))?.count ?? 0) >= attempt + 1
|
|
}
|
|
#expect(lateReplyWritten)
|
|
let replies = (try? String(contentsOfFile: lateRepliesFile, encoding: .utf8).count) ?? 0
|
|
#expect(replies == attempt + 1)
|
|
}
|
|
|
|
let finalPayload = try await connection.request(args: ["status", "--attempt", "final"])
|
|
#expect(String(decoding: finalPayload, as: UTF8.self) == "live-4")
|
|
let pids = try String(contentsOfFile: pidsFile, encoding: .utf8)
|
|
.split(separator: "\n")
|
|
#expect(pids.count == 1)
|
|
#expect(try String(contentsOfFile: requestsFile, encoding: .utf8) == "rrrr")
|
|
#expect(try String(contentsOfFile: lateRepliesFile, encoding: .utf8) == "lll")
|
|
await connection.shutdown()
|
|
}
|
|
|
|
@Test("cancelling a queued request never writes it or arms its timeout")
|
|
func queuedCancellationNeverReachesChild() async throws {
|
|
let dir = NSTemporaryDirectory() + "serve-connection-queued-cancel-test-" + UUID().uuidString
|
|
try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true)
|
|
defer { try? FileManager.default.removeItem(atPath: dir) }
|
|
let requestsFile = dir + "/requests"
|
|
let releaseMarker = dir + "/release"
|
|
let recorder = TimeoutRecorder()
|
|
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
let child = Process()
|
|
child.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
child.arguments = ["-c", """
|
|
count=0
|
|
while IFS= read -r line; do
|
|
count=$((count + 1))
|
|
printf '%s\n' "$line" >> "$1"
|
|
if [ "$count" -eq 1 ]; then
|
|
while [ ! -f "$2" ]; do sleep 0.01; done
|
|
fi
|
|
id=$(printf '%s' "$line" | sed -E 's/.*"id":([0-9]+).*/\\1/')
|
|
printf '{"id":%s,"ok":true,"output":"served-%s"}\n' "$id" "$id"
|
|
done
|
|
""", "serve-fixture", requestsFile, releaseMarker]
|
|
child.qualityOfService = qualityOfService
|
|
return child
|
|
},
|
|
timeoutSleep: { nanoseconds in
|
|
try await recorder.recordAndWait(nanoseconds)
|
|
}
|
|
)
|
|
|
|
let first = Task { try await connection.request(args: ["status", "first"]) }
|
|
for _ in 0..<200 where !(FileManager.default.fileExists(atPath: requestsFile)) {
|
|
try await Task.sleep(nanoseconds: 10_000_000)
|
|
}
|
|
let cancelled = Task { try await connection.request(args: ["status", "cancelled"]) }
|
|
let third = Task { try await connection.request(args: ["status", "third"]) }
|
|
try await Task.sleep(nanoseconds: 100_000_000)
|
|
cancelled.cancel()
|
|
do {
|
|
_ = try await cancelled.value
|
|
Issue.record("queued cancellation unexpectedly succeeded")
|
|
} catch {
|
|
#expect(error is CancellationError)
|
|
}
|
|
#expect(await recorder.snapshot() == [coldTimeoutNanoseconds])
|
|
|
|
_ = FileManager.default.createFile(atPath: releaseMarker, contents: Data())
|
|
#expect(String(decoding: try await first.value, as: UTF8.self) == "served-1")
|
|
#expect(String(decoding: try await third.value, as: UTF8.self) == "served-2")
|
|
let requests = try String(contentsOfFile: requestsFile, encoding: .utf8)
|
|
#expect(requests.contains("first"))
|
|
#expect(requests.contains("third"))
|
|
#expect(!requests.contains("cancelled"))
|
|
#expect(await recorder.snapshot() == [coldTimeoutNanoseconds, warmTimeoutNanoseconds])
|
|
await connection.shutdown()
|
|
}
|
|
|
|
@Test("shutdown fails the active request and every client-side queued request")
|
|
func shutdownDrainsClientQueue() async throws {
|
|
let dir = NSTemporaryDirectory() + "serve-connection-shutdown-queue-test-" + UUID().uuidString
|
|
try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true)
|
|
defer { try? FileManager.default.removeItem(atPath: dir) }
|
|
let requestMarker = dir + "/request-read"
|
|
let recorder = TimeoutRecorder()
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
let child = Process()
|
|
child.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
child.arguments = ["-c", "IFS= read -r line; : > \"$1\"; sleep 5", "serve-fixture", requestMarker]
|
|
child.qualityOfService = qualityOfService
|
|
return child
|
|
},
|
|
timeoutSleep: { nanoseconds in
|
|
try await recorder.recordAndWait(nanoseconds)
|
|
}
|
|
)
|
|
|
|
let active = Task { try await connection.request(args: ["status", "active"]) }
|
|
for _ in 0..<200 where !FileManager.default.fileExists(atPath: requestMarker) {
|
|
try await Task.sleep(nanoseconds: 10_000_000)
|
|
}
|
|
let queued = Task { try await connection.request(args: ["status", "queued"]) }
|
|
try await Task.sleep(nanoseconds: 100_000_000)
|
|
#expect(await recorder.snapshot() == [coldTimeoutNanoseconds])
|
|
await connection.shutdown()
|
|
|
|
for request in [active, queued] {
|
|
do {
|
|
_ = try await request.value
|
|
Issue.record("shutdown request unexpectedly succeeded")
|
|
} catch {
|
|
#expect(error is ServeConnection.ServeRequestFailed)
|
|
}
|
|
}
|
|
}
|
|
|
|
@Test("late stdout from a replaced child cannot corrupt or warm its replacement")
|
|
func staleGenerationStdoutIsDiscarded() async throws {
|
|
let oldChild = Process()
|
|
oldChild.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
oldChild.arguments = ["-c", "IFS= read -r line; sleep 0.1; exit 1"]
|
|
|
|
let newChild = Process()
|
|
newChild.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
newChild.arguments = ["-c", """
|
|
while IFS= read -r line; do
|
|
id=$(printf '%s' "$line" | sed -E 's/.*"id":([0-9]+).*/\\1/')
|
|
printf '{"id":%s,"ok":true,"output":"new-%s"}\n' "$id" "$id"
|
|
done
|
|
"""]
|
|
|
|
let children = ProcessQueue([oldChild, newChild])
|
|
let recorder = TimeoutRecorder()
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
children.take(qualityOfService: qualityOfService)
|
|
},
|
|
timeoutSleep: { nanoseconds in
|
|
try await recorder.recordAndWait(nanoseconds)
|
|
}
|
|
)
|
|
|
|
// The admitted read survives the old generation's crash and retries
|
|
// on the replacement. Request id 1 belonged to the old child; the
|
|
// replacement receives id 2.
|
|
let retried = try await connection.request(args: ["status", "--generation", "old"])
|
|
#expect(String(decoding: retried, as: UTF8.self) == "new-2")
|
|
|
|
// Model both harmful trailing shapes after the replacement owns the
|
|
// connection: a complete terminal would incorrectly select the warm
|
|
// timeout, while a fragment would corrupt the replacement's first line.
|
|
await connection.consume(
|
|
Data("{\"id\":1,\"ok\":true,\"output\":\"late-old\"}\n".utf8),
|
|
from: oldChild
|
|
)
|
|
await connection.consume(Data("{\"id\":1".utf8), from: oldChild)
|
|
|
|
let payload = try await connection.request(args: ["status", "--generation", "new"])
|
|
|
|
#expect(String(decoding: payload, as: UTF8.self) == "new-3")
|
|
#expect(await recorder.snapshot() == [
|
|
coldTimeoutNanoseconds,
|
|
coldTimeoutNanoseconds,
|
|
warmTimeoutNanoseconds,
|
|
])
|
|
#expect(children.remainingCount == 0)
|
|
await connection.shutdown()
|
|
}
|
|
|
|
@Test("queued requests arm their warm timeout only after cold hydration finishes")
|
|
func coldAndWarmTimeoutSelection() async throws {
|
|
let dir = NSTemporaryDirectory() + "serve-connection-timeout-test-" + UUID().uuidString
|
|
try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true)
|
|
defer { try? FileManager.default.removeItem(atPath: dir) }
|
|
let releaseMarker = dir + "/release-cold-responses"
|
|
let recorder = TimeoutRecorder()
|
|
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
let child = Process()
|
|
child.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
child.arguments = ["-c", """
|
|
IFS= read -r first
|
|
while [ ! -f "$1" ]; do sleep 0.01; done
|
|
for slot in first second third; do
|
|
if [ "$slot" = first ]; then line="$first"; else IFS= read -r line; fi
|
|
id=$(printf '%s' "$line" | sed -E 's/.*"id":([0-9]+).*/\\1/')
|
|
printf '{"id":%s,"ok":true,"output":"served"}\\n' "$id"
|
|
done
|
|
""", "serve-fixture", releaseMarker]
|
|
child.qualityOfService = qualityOfService
|
|
return child
|
|
},
|
|
timeoutSleep: { nanoseconds in
|
|
try await recorder.recordAndWait(nanoseconds)
|
|
}
|
|
)
|
|
|
|
let first = Task { try await connection.request(args: ["status", "--request", "one"]) }
|
|
let second = Task { try await connection.request(args: ["status", "--request", "two"]) }
|
|
try await Task.sleep(nanoseconds: 100_000_000)
|
|
#expect(await recorder.snapshot() == [coldTimeoutNanoseconds])
|
|
|
|
_ = FileManager.default.createFile(atPath: releaseMarker, contents: Data())
|
|
let firstPayload = try await first.value
|
|
let secondPayload = try await second.value
|
|
#expect(String(decoding: firstPayload, as: UTF8.self) == "served")
|
|
#expect(String(decoding: secondPayload, as: UTF8.self) == "served")
|
|
|
|
let thirdPayload = try await connection.request(args: ["status", "--request", "three"])
|
|
#expect(String(decoding: thirdPayload, as: UTF8.self) == "served")
|
|
let allSelections = await recorder.snapshot()
|
|
#expect(allSelections == [
|
|
coldTimeoutNanoseconds,
|
|
warmTimeoutNanoseconds,
|
|
warmTimeoutNanoseconds,
|
|
])
|
|
await connection.shutdown()
|
|
}
|
|
|
|
@Test("a failed terminal response does not mark the resident child warm")
|
|
func failedTerminalResponseKeepsColdTimeout() async throws {
|
|
let recorder = TimeoutRecorder()
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
let child = Process()
|
|
child.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
child.arguments = ["-c", """
|
|
count=0
|
|
while IFS= read -r line; do
|
|
count=$((count + 1))
|
|
id=$(printf '%s' "$line" | sed -E 's/.*"id":([0-9]+).*/\\1/')
|
|
if [ "$count" -eq 1 ]; then
|
|
printf '{"id":%s,"ok":false,"error":"cold failure"}\\n' "$id"
|
|
else
|
|
printf '{"id":%s,"ok":true,"output":"served-%s"}\\n' "$id" "$count"
|
|
fi
|
|
done
|
|
"""]
|
|
child.qualityOfService = qualityOfService
|
|
return child
|
|
},
|
|
timeoutSleep: { nanoseconds in
|
|
try await recorder.recordAndWait(nanoseconds)
|
|
}
|
|
)
|
|
|
|
do {
|
|
_ = try await connection.request(args: ["status", "--request", "failed"])
|
|
#expect(Bool(false), "failed response unexpectedly succeeded")
|
|
} catch {
|
|
#expect(error is ServeConnection.ServeRequestFailed)
|
|
}
|
|
|
|
let second = try await connection.request(args: ["status", "--request", "cold-success"])
|
|
let third = try await connection.request(args: ["status", "--request", "warm-success"])
|
|
|
|
#expect(String(decoding: second, as: UTF8.self) == "served-2")
|
|
#expect(String(decoding: third, as: UTF8.self) == "served-3")
|
|
#expect(await recorder.snapshot() == [
|
|
coldTimeoutNanoseconds,
|
|
coldTimeoutNanoseconds,
|
|
warmTimeoutNanoseconds,
|
|
])
|
|
await connection.shutdown()
|
|
}
|
|
|
|
@Test("an actual stdout flood is bounded and the next generation stays healthy")
|
|
func oversizedFrameTerminatesOnlyItsGeneration() async throws {
|
|
let oldChild = Process()
|
|
oldChild.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
oldChild.arguments = ["-c", """
|
|
IFS= read -r line
|
|
dd if=/dev/zero bs=1024 count=1 2>/dev/null | tr '\\0' x
|
|
sleep 5
|
|
"""]
|
|
|
|
let replacement = Process()
|
|
replacement.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
replacement.arguments = ["-c", """
|
|
IFS= read -r line
|
|
id=$(printf '%s' "$line" | sed -E 's/.*"id":([0-9]+).*/\\1/')
|
|
printf '{"id":%s,"ok":true,"output":"replacement"}\\n' "$id"
|
|
"""]
|
|
|
|
let children = ProcessQueue([oldChild, replacement])
|
|
let recorder = TimeoutRecorder()
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
children.take(qualityOfService: qualityOfService)
|
|
},
|
|
timeoutSleep: { nanoseconds in
|
|
try await recorder.recordAndWait(nanoseconds)
|
|
},
|
|
responseLimitBytes: 128
|
|
)
|
|
|
|
do {
|
|
_ = try await connection.request(args: ["status", "--oversized"])
|
|
#expect(Bool(false), "oversized resident frame unexpectedly succeeded")
|
|
} catch let error as ServeConnection.ServeRequestFailed {
|
|
#expect(error.reason == .outputTooLarge)
|
|
}
|
|
|
|
await connection.ensureStarted()
|
|
let payload = try await connection.request(args: ["status", "--replacement"])
|
|
#expect(String(decoding: payload, as: UTF8.self) == "replacement")
|
|
#expect(children.remainingCount == 0)
|
|
await connection.shutdown()
|
|
}
|
|
|
|
@Test("an unterminated frame and cumulative progress cannot bypass the resident limit")
|
|
func partialAndCumulativeFramesAreBounded() async throws {
|
|
for mode in ["partial", "progress"] {
|
|
let child = Process()
|
|
child.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
child.arguments = ["-c", "IFS= read -r line; sleep 5"]
|
|
let recorder = TimeoutRecorder()
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
child.qualityOfService = qualityOfService
|
|
return child
|
|
},
|
|
timeoutSleep: { nanoseconds in
|
|
try await recorder.recordAndWait(nanoseconds)
|
|
},
|
|
responseLimitBytes: 128
|
|
)
|
|
let request = Task { try await connection.request(args: ["status", "--mode", mode]) }
|
|
for _ in 0..<200 {
|
|
if await recorder.snapshot().count == 1 { break }
|
|
try await Task.sleep(nanoseconds: 10_000_000)
|
|
}
|
|
|
|
if mode == "partial" {
|
|
await connection.consume(Data(repeating: UInt8(ascii: "x"), count: 129), from: child)
|
|
} else {
|
|
let progress = String(repeating: "p", count: 70)
|
|
let frame = Data("{\"id\":1,\"progress\":\"\(progress)\"}\n".utf8)
|
|
#expect(frame.count < 128)
|
|
await connection.consume(frame, from: child)
|
|
await connection.consume(frame, from: child)
|
|
}
|
|
|
|
do {
|
|
_ = try await request.value
|
|
#expect(Bool(false), "\(mode) overflow unexpectedly succeeded")
|
|
} catch let error as ServeConnection.ServeRequestFailed {
|
|
#expect(error.reason == .outputTooLarge)
|
|
}
|
|
await connection.shutdown()
|
|
}
|
|
}
|
|
|
|
@Test("a cancelled request keeps its cumulative progress bound until the child finishes")
|
|
func cancelledRequestStillBoundsOrphanProgress() async throws {
|
|
let oldChild = Process()
|
|
oldChild.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
oldChild.arguments = ["-c", "IFS= read -r line; sleep 5"]
|
|
|
|
let replacement = Process()
|
|
replacement.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
replacement.arguments = ["-c", """
|
|
IFS= read -r line
|
|
id=$(printf '%s' "$line" | sed -E 's/.*"id":([0-9]+).*/\\1/')
|
|
printf '{"id":%s,"ok":true,"output":"healthy"}\\n' "$id"
|
|
"""]
|
|
|
|
let children = ProcessQueue([oldChild, replacement])
|
|
let recorder = TimeoutRecorder()
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
children.take(qualityOfService: qualityOfService)
|
|
},
|
|
timeoutSleep: { nanoseconds in
|
|
try await recorder.recordAndWait(nanoseconds)
|
|
},
|
|
responseLimitBytes: 128
|
|
)
|
|
|
|
let abandoned = Task { try await connection.request(args: ["status", "--abandoned"]) }
|
|
for _ in 0..<200 {
|
|
if await recorder.snapshot().count == 1 { break }
|
|
try await Task.sleep(nanoseconds: 10_000_000)
|
|
}
|
|
abandoned.cancel()
|
|
do {
|
|
_ = try await abandoned.value
|
|
Issue.record("cancelled request unexpectedly succeeded")
|
|
} catch {
|
|
#expect(error is CancellationError)
|
|
}
|
|
|
|
let progress = String(repeating: "p", count: 70)
|
|
let frame = Data("{\"id\":1,\"progress\":\"\(progress)\"}\n".utf8)
|
|
await connection.consume(frame, from: oldChild)
|
|
await connection.consume(frame, from: oldChild)
|
|
|
|
await connection.ensureStarted()
|
|
#expect(children.remainingCount == 0)
|
|
let payload = try await connection.request(args: ["status", "--replacement"])
|
|
#expect(String(decoding: payload, as: UTF8.self) == "healthy")
|
|
await connection.shutdown()
|
|
}
|
|
|
|
@Test("each overflow consumes exactly one resident death")
|
|
func overflowDeathBudgetIsExact() async throws {
|
|
let processes = (0..<3).map { _ in
|
|
let child = Process()
|
|
child.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
child.arguments = ["-c", "IFS= read -r line; sleep 5"]
|
|
return child
|
|
}
|
|
let children = ProcessQueue(processes)
|
|
let recorder = TimeoutRecorder()
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
children.take(qualityOfService: qualityOfService)
|
|
},
|
|
timeoutSleep: { nanoseconds in
|
|
try await recorder.recordAndWait(nanoseconds)
|
|
},
|
|
responseLimitBytes: 64
|
|
)
|
|
|
|
for attempt in 0..<3 {
|
|
let request = Task { try await connection.request(args: ["status", "--attempt", "\(attempt)"]) }
|
|
for _ in 0..<200 {
|
|
if await recorder.snapshot().count == attempt + 1 { break }
|
|
try await Task.sleep(nanoseconds: 10_000_000)
|
|
}
|
|
await connection.consume(Data(repeating: UInt8(ascii: "x"), count: 65), from: processes[attempt])
|
|
do {
|
|
_ = try await request.value
|
|
Issue.record("overflow \(attempt) unexpectedly succeeded")
|
|
} catch let error as ServeConnection.ServeRequestFailed {
|
|
#expect(error.reason == .outputTooLarge)
|
|
}
|
|
}
|
|
|
|
#expect(children.remainingCount == 0)
|
|
do {
|
|
_ = try await connection.request(args: ["status", "--after-budget"])
|
|
Issue.record("resident restarted after exhausting its death budget")
|
|
} catch {
|
|
#expect(error is ServeConnection.ServeUnavailable)
|
|
}
|
|
await connection.shutdown()
|
|
}
|
|
|
|
@Test("output overflow is not eligible for a one-shot fallback")
|
|
func outputOverflowIsTerminalForDataClient() async {
|
|
let overflow = ServeConnection.ServeRequestFailed(
|
|
message: "too large",
|
|
reason: .outputTooLarge
|
|
)
|
|
let fallback = FallbackRecorder()
|
|
do {
|
|
_ = try await DataClient.runCLI(
|
|
subcommand: ["status", "--format", "menubar-json"],
|
|
serveRequest: { _ in throw overflow },
|
|
spawnFallback: {
|
|
await fallback.record()
|
|
return DataClient.ProcessResult(stdout: Data(), stderr: "", exitCode: 0)
|
|
}
|
|
)
|
|
Issue.record("output overflow unexpectedly fell back or succeeded")
|
|
} catch DataClientError.outputTooLarge {
|
|
// Expected: the one-shot closure must remain untouched.
|
|
} catch {
|
|
Issue.record("unexpected terminal error: \(error)")
|
|
}
|
|
#expect(await fallback.snapshot() == 0)
|
|
|
|
let ordinary = ServeConnection.ServeRequestFailed(message: "serve exited")
|
|
do {
|
|
let result = try await DataClient.runCLI(
|
|
subcommand: ["status", "--format", "menubar-json"],
|
|
serveRequest: { _ in throw ordinary },
|
|
spawnFallback: {
|
|
await fallback.record()
|
|
return DataClient.ProcessResult(stdout: Data("fallback".utf8), stderr: "", exitCode: 0)
|
|
}
|
|
)
|
|
#expect(String(decoding: result.stdout, as: UTF8.self) == "fallback")
|
|
} catch {
|
|
Issue.record("ordinary serve failure did not use fallback: \(error)")
|
|
}
|
|
#expect(await fallback.snapshot() == 1)
|
|
}
|
|
|
|
@Test("a verified-fresh request can bypass the resident worker")
|
|
func verifiedFreshRequestBypassesResident() async throws {
|
|
let resident = FallbackRecorder()
|
|
let fallback = FallbackRecorder()
|
|
|
|
let result = try await DataClient.runCLI(
|
|
subcommand: ["status", "--format", "menubar-json", "--provider", "hermes"],
|
|
bypassResident: true,
|
|
serveRequest: { _ in
|
|
await resident.record()
|
|
return Data("stale".utf8)
|
|
},
|
|
spawnFallback: {
|
|
await fallback.record()
|
|
return DataClient.ProcessResult(stdout: Data("fresh".utf8), stderr: "", exitCode: 0)
|
|
}
|
|
)
|
|
|
|
#expect(String(decoding: result.stdout, as: UTF8.self) == "fresh")
|
|
#expect(await resident.snapshot() == 0)
|
|
#expect(await fallback.snapshot() == 1)
|
|
}
|
|
|
|
@Test("the first real request is the only cold-start query")
|
|
func firstRequestIsTheWarmup() async throws {
|
|
let dir = NSTemporaryDirectory() + "serve-connection-test-" + UUID().uuidString
|
|
try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true)
|
|
defer { try? FileManager.default.removeItem(atPath: dir) }
|
|
let requestLog = dir + "/requests.log"
|
|
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
let child = Process()
|
|
child.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
child.arguments = ["-c", """
|
|
while IFS= read -r line; do
|
|
printf 'request\\n' >> "$1"
|
|
id=$(printf '%s' "$line" | sed -E 's/.*\"id\":([0-9]+).*/\\1/')
|
|
printf '{\"id\":%s,\"progress\":\"scanning\"}\\n' "$id"
|
|
printf '{\"id\":%s,\"ok\":true,\"output\":\"served\"}\\n' "$id"
|
|
# Emit READY after the terminal response. The client must
|
|
# register and complete the first real request without it.
|
|
printf '{\"ready\":true,\"pid\":1}\\n'
|
|
done
|
|
""", "serve-fixture", requestLog]
|
|
child.qualityOfService = qualityOfService
|
|
return child
|
|
})
|
|
|
|
await connection.ensureStarted()
|
|
let payload = try await connection.request(args: ["status", "--format", "menubar-json"])
|
|
|
|
#expect(String(decoding: payload, as: UTF8.self) == "served")
|
|
let requests = try String(contentsOfFile: requestLog, encoding: .utf8)
|
|
.split(separator: "\n")
|
|
#expect(requests.count == 1)
|
|
await connection.shutdown()
|
|
}
|
|
|
|
@Test("split terminal bytes are drained before child death and the next generation stays clean")
|
|
func finalStdoutDrainPrecedesTermination() async throws {
|
|
let first = Process()
|
|
first.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
first.arguments = ["-c", """
|
|
IFS= read -r line
|
|
id=$(printf '%s' "$line" | sed -E 's/.*"id":([0-9]+).*/\\1/')
|
|
printf '{"id":%s,"ok":true,' "$id"
|
|
printf '"output":"final-drain"}\n'
|
|
"""]
|
|
|
|
let replacement = Process()
|
|
replacement.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
replacement.arguments = ["-c", """
|
|
while IFS= read -r line; do
|
|
id=$(printf '%s' "$line" | sed -E 's/.*"id":([0-9]+).*/\\1/')
|
|
printf '{"id":%s,"ok":true,"output":"replacement"}\n' "$id"
|
|
done
|
|
"""]
|
|
|
|
let children = ProcessQueue([first, replacement])
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
children.take(qualityOfService: qualityOfService)
|
|
})
|
|
|
|
let drained = try await connection.request(args: ["status", "drain"])
|
|
#expect(String(decoding: drained, as: UTF8.self) == "final-drain")
|
|
let next = try await connection.request(args: ["status", "next"])
|
|
#expect(String(decoding: next, as: UTF8.self) == "replacement")
|
|
#expect(children.remainingCount == 0)
|
|
await connection.shutdown()
|
|
}
|
|
|
|
@Test("a child that closes stdin fails the request without terminating the app")
|
|
func closedChildStdinDoesNotRaiseSIGPIPE() async throws {
|
|
let dir = NSTemporaryDirectory() + "serve-connection-sigpipe-test-" + UUID().uuidString
|
|
try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true)
|
|
defer { try? FileManager.default.removeItem(atPath: dir) }
|
|
let closedMarker = dir + "/stdin-closed"
|
|
let sigpipeHandlerBefore = currentSIGPIPEHandlerBits()
|
|
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
let child = Process()
|
|
child.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
child.arguments = ["-c", "exec 0<&-; : > \"$1\"; sleep 2", "serve-fixture", closedMarker]
|
|
child.qualityOfService = qualityOfService
|
|
return child
|
|
})
|
|
|
|
await connection.ensureStarted()
|
|
#expect(currentSIGPIPEHandlerBits() == sigpipeHandlerBefore)
|
|
#expect(currentSIGPIPEHandlerBits() != ignoredSIGPIPEHandlerBits)
|
|
for _ in 0..<200 where !FileManager.default.fileExists(atPath: closedMarker) {
|
|
try await Task.sleep(nanoseconds: 10_000_000)
|
|
}
|
|
#expect(FileManager.default.fileExists(atPath: closedMarker))
|
|
|
|
var requestFailed = false
|
|
do {
|
|
_ = try await connection.request(args: ["status", "--format", "menubar-json"])
|
|
} catch {
|
|
requestFailed = true
|
|
}
|
|
#expect(requestFailed)
|
|
await connection.shutdown()
|
|
}
|
|
|
|
// MARK: - idle retire
|
|
|
|
@Test("an idle resident retires and the next request respawns through the start path")
|
|
func idleWindowRetiresAndRespawnsResident() async throws {
|
|
let dir = NSTemporaryDirectory() + "serve-connection-idle-retire-test-" + UUID().uuidString
|
|
try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true)
|
|
defer { try? FileManager.default.removeItem(atPath: dir) }
|
|
let pidsFile = dir + "/pids"
|
|
let idleClock = ManualTimeoutClock()
|
|
let requestClock = ManualTimeoutClock()
|
|
// Never fired: with SIGTERM trapped in the fixture, the only thing that
|
|
// can end a child is the retire closing its stdin.
|
|
let graceClock = ManualTimeoutClock()
|
|
let first = makeEchoFixture(pidsFile: pidsFile)
|
|
let second = makeEchoFixture(pidsFile: pidsFile)
|
|
let children = ProcessQueue([first, second])
|
|
defer { killIfRunning([first, second]) }
|
|
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
children.take(qualityOfService: qualityOfService)
|
|
},
|
|
timeoutSleep: { nanoseconds in try await requestClock.sleep(nanoseconds) },
|
|
terminationGraceSleep: { nanoseconds in try await graceClock.sleep(nanoseconds) },
|
|
idleSeconds: idleWindowSeconds,
|
|
idleSleep: { nanoseconds in try await idleClock.sleep(nanoseconds) }
|
|
)
|
|
|
|
let firstPayload = try await connection.request(args: ["status", "--first"])
|
|
#expect(String(decoding: firstPayload, as: UTF8.self) == "reply-1")
|
|
|
|
let armed = await eventually { await idleClock.snapshot() == [idleWindowNanoseconds] }
|
|
#expect(armed)
|
|
await idleClock.fireOldest()
|
|
|
|
let retired = await eventually { await connection.residentChildForTesting == nil }
|
|
#expect(retired)
|
|
let firstExited = await eventually { !first.isRunning }
|
|
#expect(firstExited)
|
|
// A clean exit proves stdin was closed: the fixture ignores SIGTERM and
|
|
// the escalation grace never elapses.
|
|
#expect(first.terminationReason == .exit)
|
|
|
|
let secondPayload = try await connection.request(args: ["status", "--second"])
|
|
#expect(String(decoding: secondPayload, as: UTF8.self) == "reply-2")
|
|
#expect(await connection.residentChildForTesting === second)
|
|
#expect(children.remainingCount == 0)
|
|
#expect(try recordedPids(pidsFile).count == 2)
|
|
// A retire resets warmth with the generation, so the respawned child's
|
|
// first request gets the COLD silence allowance, not the warm 45s.
|
|
let coldTwice = await eventually {
|
|
await requestClock.history() == [coldTimeoutNanoseconds, coldTimeoutNanoseconds]
|
|
}
|
|
#expect(coldTwice)
|
|
await connection.shutdown()
|
|
}
|
|
|
|
@Test("an idle retire closes stdin first and signals nothing that exits on EOF")
|
|
func idleRetireClosesStdinBeforeSignalling() async throws {
|
|
let dir = NSTemporaryDirectory() + "serve-connection-retire-stdin-test-" + UUID().uuidString
|
|
try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true)
|
|
defer { try? FileManager.default.removeItem(atPath: dir) }
|
|
let pidsFile = dir + "/pids"
|
|
let signalsFile = dir + "/signals"
|
|
let idleClock = ManualTimeoutClock()
|
|
let graceClock = ManualTimeoutClock()
|
|
// EOF is what makes `serve --stdio` publish its coalesced shard window,
|
|
// and SIGTERM is caught only to unlink the cache lock before being
|
|
// re-raised — so a signal in the same hop kills the flush.
|
|
let first = makeSignalRecordingFixture(pidsFile: pidsFile, signalsFile: signalsFile)
|
|
let second = makeSignalRecordingFixture(pidsFile: pidsFile, signalsFile: signalsFile)
|
|
let children = ProcessQueue([first, second])
|
|
defer { killIfRunning([first, second]) }
|
|
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in children.take(qualityOfService: qualityOfService) },
|
|
timeoutSleep: { nanoseconds in try await ManualTimeoutClock().sleep(nanoseconds) },
|
|
terminationGraceSleep: { nanoseconds in try await graceClock.sleep(nanoseconds) },
|
|
idleSeconds: idleWindowSeconds,
|
|
idleSleep: { nanoseconds in try await idleClock.sleep(nanoseconds) }
|
|
)
|
|
|
|
_ = try await connection.request(args: ["status", "--first"])
|
|
#expect(await eventually { await idleClock.snapshot() == [idleWindowNanoseconds] })
|
|
await idleClock.fireOldest()
|
|
|
|
let exited = await eventually { !first.isRunning }
|
|
#expect(exited)
|
|
#expect(first.terminationReason == .exit)
|
|
#expect(first.terminationStatus == 0)
|
|
// The grace was armed but never fired, and nothing was signalled.
|
|
#expect(await graceClock.firedCount() == 0)
|
|
#expect((try? String(contentsOfFile: signalsFile, encoding: .utf8)) == nil)
|
|
|
|
// The next request still gets a fresh resident.
|
|
_ = try await connection.request(args: ["status", "--second"])
|
|
#expect(await connection.residentChildForTesting === second)
|
|
await connection.shutdown()
|
|
}
|
|
|
|
@Test("a retired child that ignores EOF is still terminated after the grace")
|
|
func idleRetireTerminatesAChildThatIgnoresEOF() async throws {
|
|
let dir = NSTemporaryDirectory() + "serve-connection-retire-stuck-test-" + UUID().uuidString
|
|
try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true)
|
|
defer { try? FileManager.default.removeItem(atPath: dir) }
|
|
let pidsFile = dir + "/pids"
|
|
let signalsFile = dir + "/signals"
|
|
let idleClock = ManualTimeoutClock()
|
|
let graceClock = ManualTimeoutClock()
|
|
let stuck = makeSignalRecordingFixture(pidsFile: pidsFile, signalsFile: signalsFile, ignoreEOF: true)
|
|
let children = ProcessQueue([stuck])
|
|
defer { killIfRunning([stuck]) }
|
|
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in children.take(qualityOfService: qualityOfService) },
|
|
timeoutSleep: { nanoseconds in try await ManualTimeoutClock().sleep(nanoseconds) },
|
|
terminationGraceSleep: { nanoseconds in try await graceClock.sleep(nanoseconds) },
|
|
idleSeconds: idleWindowSeconds,
|
|
idleSleep: { nanoseconds in try await idleClock.sleep(nanoseconds) }
|
|
)
|
|
|
|
_ = try await connection.request(args: ["status", "--first"])
|
|
#expect(await eventually { await idleClock.snapshot() == [idleWindowNanoseconds] })
|
|
await idleClock.fireOldest()
|
|
|
|
// The flush grace is armed; the child is still alive inside it.
|
|
#expect(await eventually { await graceClock.snapshot().count == 1 })
|
|
#expect(stuck.isRunning)
|
|
#expect((try? String(contentsOfFile: signalsFile, encoding: .utf8)) == nil)
|
|
|
|
// Grace elapses: only now is it signalled.
|
|
await graceClock.fireOldest()
|
|
let signalled = await eventually {
|
|
(try? String(contentsOfFile: signalsFile, encoding: .utf8)) == "T"
|
|
}
|
|
#expect(signalled)
|
|
await connection.shutdown()
|
|
}
|
|
|
|
@Test("idle retires are not deaths and never spend the resident budget")
|
|
func idleRetiresDoNotConsumeDeathBudget() async throws {
|
|
let dir = NSTemporaryDirectory() + "serve-connection-idle-budget-test-" + UUID().uuidString
|
|
try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true)
|
|
defer { try? FileManager.default.removeItem(atPath: dir) }
|
|
let pidsFile = dir + "/pids"
|
|
let idleClock = ManualTimeoutClock()
|
|
let graceClock = ManualTimeoutClock()
|
|
let fixtures = (0..<6).map { _ in makeEchoFixture(pidsFile: pidsFile) }
|
|
let children = ProcessQueue(fixtures)
|
|
defer { killIfRunning(fixtures) }
|
|
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
children.take(qualityOfService: qualityOfService)
|
|
},
|
|
terminationGraceSleep: { nanoseconds in try await graceClock.sleep(nanoseconds) },
|
|
idleSeconds: idleWindowSeconds,
|
|
idleSleep: { nanoseconds in try await idleClock.sleep(nanoseconds) }
|
|
)
|
|
|
|
for attempt in 1...5 {
|
|
let payload = try await connection.request(args: ["status", "--attempt", String(attempt)])
|
|
#expect(String(decoding: payload, as: UTF8.self) == "reply-\(attempt)")
|
|
let armed = await eventually { await idleClock.snapshot() == [idleWindowNanoseconds] }
|
|
#expect(armed)
|
|
await idleClock.fireOldest()
|
|
let retired = await eventually { await connection.residentChildForTesting == nil }
|
|
#expect(retired)
|
|
}
|
|
|
|
// Three deaths disable the resident for the app run, so a sixth live
|
|
// request is the proof that five retires cost nothing.
|
|
let sixth = try await connection.request(args: ["status", "--sixth"])
|
|
#expect(String(decoding: sixth, as: UTF8.self) == "reply-6")
|
|
#expect(children.remainingCount == 0)
|
|
await connection.shutdown()
|
|
}
|
|
|
|
@Test("the idle window firing while work is active or queued retires nothing")
|
|
func idleRetireIsInertWhileRequestsAreOutstanding() async throws {
|
|
let dir = NSTemporaryDirectory() + "serve-connection-idle-busy-test-" + UUID().uuidString
|
|
try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true)
|
|
defer { try? FileManager.default.removeItem(atPath: dir) }
|
|
let pidsFile = dir + "/pids"
|
|
let eventsFile = dir + "/events"
|
|
let releaseMarker = dir + "/release-second"
|
|
let gate = ReleaseGate()
|
|
|
|
// Blocks after reading request 2, so request 3 stays client-side queued
|
|
// while the test fires the idle window.
|
|
let child = Process()
|
|
child.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
child.arguments = ["-c", """
|
|
trap '' TERM
|
|
printf '%s\n' "$$" >> "$1"
|
|
while IFS= read -r line; do
|
|
id=$(printf '%s' "$line" | sed -E 's/.*"id":([0-9]+).*/\\1/')
|
|
printf '%s\n' "read-$id" >> "$2"
|
|
if [ "$id" = "2" ]; then
|
|
while [ ! -f "$3" ]; do sleep 0.01; done
|
|
fi
|
|
printf '{"id":%s,"ok":true,"output":"reply-%s"}\n' "$id" "$id"
|
|
done
|
|
""", "serve-fixture", pidsFile, eventsFile, releaseMarker]
|
|
let replacement = makeEchoFixture(pidsFile: pidsFile)
|
|
let children = ProcessQueue([child, replacement])
|
|
defer { killIfRunning([child, replacement]) }
|
|
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
children.take(qualityOfService: qualityOfService)
|
|
},
|
|
idleSeconds: idleWindowSeconds,
|
|
idleSleep: { nanoseconds in await gate.wait(nanoseconds) }
|
|
)
|
|
|
|
let firstPayload = try await connection.request(args: ["status", "--first"])
|
|
#expect(String(decoding: firstPayload, as: UTF8.self) == "reply-1")
|
|
let armed = await eventually { await gate.armedCount() == 2 }
|
|
#expect(armed)
|
|
|
|
let second = Task { try await connection.request(args: ["status", "--second"]) }
|
|
let secondRead = await eventually {
|
|
((try? String(contentsOfFile: eventsFile, encoding: .utf8)) ?? "").contains("read-2\n")
|
|
}
|
|
#expect(secondRead)
|
|
let third = Task { try await connection.request(args: ["status", "--third"]) }
|
|
// Nothing observable happens while a request waits its turn, so this
|
|
// settle can only miss a regression, never fail a correct run.
|
|
try await Task.sleep(nanoseconds: 100_000_000)
|
|
#expect(try String(contentsOfFile: eventsFile, encoding: .utf8) == "read-1\nread-2\n")
|
|
|
|
// The idle task fires with one request active and one queued. The gate
|
|
// is deliberately not cancellation-aware, so the connection's own
|
|
// quiet check is what has to keep this inert.
|
|
await gate.releaseNewest()
|
|
try await Task.sleep(nanoseconds: 100_000_000)
|
|
#expect(await connection.residentChildForTesting === child)
|
|
#expect(child.isRunning)
|
|
|
|
_ = FileManager.default.createFile(atPath: releaseMarker, contents: Data())
|
|
let secondPayload = try await withHangGuard(onExpiry: { await connection.shutdown() }) {
|
|
try await second.value
|
|
}
|
|
let thirdPayload = try await withHangGuard(onExpiry: { await connection.shutdown() }) {
|
|
try await third.value
|
|
}
|
|
#expect(String(decoding: secondPayload, as: UTF8.self) == "reply-2")
|
|
#expect(String(decoding: thirdPayload, as: UTF8.self) == "reply-3")
|
|
#expect(try recordedPids(pidsFile).count == 1)
|
|
#expect(children.remainingCount == 1)
|
|
await gate.releaseAll()
|
|
await connection.shutdown()
|
|
}
|
|
|
|
@Test("a late EOF from a retired child cannot disturb its replacement")
|
|
func lateEOFFromRetiredChildIsInert() async throws {
|
|
let dir = NSTemporaryDirectory() + "serve-connection-idle-late-eof-test-" + UUID().uuidString
|
|
try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true)
|
|
defer { try? FileManager.default.removeItem(atPath: dir) }
|
|
let pidsFile = dir + "/pids"
|
|
let exitMarker = dir + "/let-first-exit"
|
|
let idleClock = ManualTimeoutClock()
|
|
let graceClock = ManualTimeoutClock()
|
|
|
|
// Holds its stdout open after its stdin closes, so the test decides when
|
|
// the retired generation's EOF reaches the connection.
|
|
let first = Process()
|
|
first.executableURL = URL(fileURLWithPath: "/bin/sh")
|
|
first.arguments = ["-c", """
|
|
trap '' TERM
|
|
printf '%s\n' "$$" >> "$1"
|
|
while IFS= read -r line; do
|
|
id=$(printf '%s' "$line" | sed -E 's/.*"id":([0-9]+).*/\\1/')
|
|
printf '{"id":%s,"ok":true,"output":"reply-%s"}\n' "$id" "$id"
|
|
done
|
|
while [ ! -f "$2" ]; do sleep 0.01; done
|
|
""", "serve-fixture", pidsFile, exitMarker]
|
|
let second = makeEchoFixture(pidsFile: pidsFile)
|
|
let children = ProcessQueue([first, second])
|
|
defer { killIfRunning([first, second]) }
|
|
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
children.take(qualityOfService: qualityOfService)
|
|
},
|
|
terminationGraceSleep: { nanoseconds in try await graceClock.sleep(nanoseconds) },
|
|
idleSeconds: idleWindowSeconds,
|
|
idleSleep: { nanoseconds in try await idleClock.sleep(nanoseconds) }
|
|
)
|
|
|
|
_ = try await connection.request(args: ["status", "--first"])
|
|
let armed = await eventually { await idleClock.snapshot() == [idleWindowNanoseconds] }
|
|
#expect(armed)
|
|
await idleClock.fireOldest()
|
|
let retired = await eventually { await connection.residentChildForTesting == nil }
|
|
#expect(retired)
|
|
|
|
let replacementPayload = try await connection.request(args: ["status", "--second"])
|
|
#expect(String(decoding: replacementPayload, as: UTF8.self) == "reply-2")
|
|
#expect(await connection.residentChildForTesting === second)
|
|
|
|
// The retired child's EOF only now reaches the reader, with a live
|
|
// replacement in place.
|
|
_ = FileManager.default.createFile(atPath: exitMarker, contents: Data())
|
|
let firstExited = await eventually { !first.isRunning }
|
|
#expect(firstExited)
|
|
try await Task.sleep(nanoseconds: 100_000_000)
|
|
|
|
#expect(await connection.residentChildForTesting === second)
|
|
let afterEOF = try await connection.request(args: ["status", "--third"])
|
|
#expect(String(decoding: afterEOF, as: UTF8.self) == "reply-3")
|
|
#expect(children.remainingCount == 0)
|
|
await connection.shutdown()
|
|
}
|
|
|
|
@Test("shutdown cancels the armed idle window")
|
|
func shutdownCancelsIdleRetire() async throws {
|
|
let dir = NSTemporaryDirectory() + "serve-connection-idle-shutdown-test-" + UUID().uuidString
|
|
try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true)
|
|
defer { try? FileManager.default.removeItem(atPath: dir) }
|
|
let pidsFile = dir + "/pids"
|
|
let idleClock = ManualTimeoutClock()
|
|
let child = makeEchoFixture(pidsFile: pidsFile)
|
|
let children = ProcessQueue([child])
|
|
defer { killIfRunning([child]) }
|
|
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
children.take(qualityOfService: qualityOfService)
|
|
},
|
|
idleSeconds: idleWindowSeconds,
|
|
idleSleep: { nanoseconds in try await idleClock.sleep(nanoseconds) }
|
|
)
|
|
|
|
_ = try await connection.request(args: ["status", "--only"])
|
|
let armed = await eventually { await idleClock.snapshot() == [idleWindowNanoseconds] }
|
|
#expect(armed)
|
|
|
|
await connection.shutdown()
|
|
let cancelled = await eventually { await idleClock.snapshot().isEmpty }
|
|
#expect(cancelled)
|
|
#expect(await idleClock.firedCount() == 0)
|
|
}
|
|
|
|
@Test("a non-positive idle window keeps the resident forever")
|
|
func disabledIdleWindowNeverRetires() async throws {
|
|
let dir = NSTemporaryDirectory() + "serve-connection-idle-disabled-test-" + UUID().uuidString
|
|
try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true)
|
|
defer { try? FileManager.default.removeItem(atPath: dir) }
|
|
let pidsFile = dir + "/pids"
|
|
let idleClock = ManualTimeoutClock()
|
|
let child = makeEchoFixture(pidsFile: pidsFile)
|
|
let children = ProcessQueue([child])
|
|
defer { killIfRunning([child]) }
|
|
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
children.take(qualityOfService: qualityOfService)
|
|
},
|
|
idleSeconds: 0,
|
|
idleSleep: { nanoseconds in try await idleClock.sleep(nanoseconds) }
|
|
)
|
|
|
|
_ = try await connection.request(args: ["status", "--first"])
|
|
try await Task.sleep(nanoseconds: 100_000_000)
|
|
#expect(await idleClock.history().isEmpty)
|
|
#expect(await connection.residentChildForTesting === child)
|
|
#expect(child.isRunning)
|
|
await connection.shutdown()
|
|
}
|
|
|
|
@Test("a request racing the idle window is served by exactly one live child")
|
|
func requestRacingIdleRetireKeepsOneChild() async throws {
|
|
let dir = NSTemporaryDirectory() + "serve-connection-idle-race-test-" + UUID().uuidString
|
|
try FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true)
|
|
defer { try? FileManager.default.removeItem(atPath: dir) }
|
|
let pidsFile = dir + "/pids"
|
|
let gate = ReleaseGate()
|
|
let first = makeEchoFixture(pidsFile: pidsFile)
|
|
let second = makeEchoFixture(pidsFile: pidsFile)
|
|
let children = ProcessQueue([first, second])
|
|
defer { killIfRunning([first, second]) }
|
|
|
|
let connection = ServeConnection(
|
|
pidFile: scratchPidFile(),
|
|
makeProcess: { _, qualityOfService in
|
|
children.take(qualityOfService: qualityOfService)
|
|
},
|
|
terminationGraceSleep: { _ in },
|
|
idleSeconds: idleWindowSeconds,
|
|
idleSleep: { nanoseconds in await gate.wait(nanoseconds) }
|
|
)
|
|
|
|
_ = try await connection.request(args: ["status", "--first"])
|
|
let armed = await eventually { await gate.armedCount() == 2 }
|
|
#expect(armed)
|
|
|
|
// Whichever of the two reaches the actor first, the request must be
|
|
// served by a live child: a retire that wins is respawned through
|
|
// ensureStarted, and one that loses is a no-op.
|
|
async let released: Void = gate.releaseNewest()
|
|
// A retire that swallowed the racing request would strand it forever,
|
|
// so the guard turns that regression into a failure, not a hang.
|
|
let payload = try await withHangGuard(onExpiry: { await connection.shutdown() }) {
|
|
try await connection.request(args: ["status", "--racing"])
|
|
}
|
|
await released
|
|
#expect(String(decoding: payload, as: UTF8.self) == "reply-2")
|
|
|
|
let settled = await eventually {
|
|
[first, second].filter(\.isRunning).count == 1
|
|
}
|
|
#expect(settled)
|
|
// One original plus at most one replacement: the race never spawns two
|
|
// children for one request.
|
|
#expect(try recordedPids(pidsFile).count <= 2)
|
|
await gate.releaseAll()
|
|
await connection.shutdown()
|
|
}
|
|
}
|