mirror of
https://github.com/NeuralNomadsAI/CodeNomad.git
synced 2026-10-26 00:01:57 +00:00
f400d5beb9
A session whose history holds image tool results (for example `read` on PNG files) never opened over a slow link (#1113): the 200-message page was 16.8 MB, 15.7 MB of it base64 in five tool results. It took 20 s at 6.5 Mbit/s, and a reverse proxy cut the transfer, so the UI showed "Failed to fetch". Every open, reload and history window paid for every image, shown or not. Transcript pages now carry large tool images by reference. The UI's native message list reads (createInstanceFetch) and history windows send the `x-codenomad-media: deferred` opt-in. For those reads only, the server replaces each tool image whose data URI exceeds 32 KiB with a `/api/workspaces/<id>/session-media/<session>/<message>/<part>/<index>/ <sha256>` path, in the instance proxy (forwardRuntimeRequest gains an optional whole-JSON transform for successful JSON responses) and in the history window route. toolImageSource resolves that path on the API base, so the existing lazy `<img>` loads only the images that are shown, as binary rather than base64. The new session-media route checks that the workspace owns the session, reads the native message, and refuses a session that moved during the read and messages at or after an undo boundary. It serves the bytes only when the image still matches the digest, as an immutable, nosniff, sandboxed image; anything else falls back to "Unable to display this image". What stays whole: - Single-message reads (`session.message.get`), so pruning revisions keep hashing native content. - Live events, user attachments, small images, non-image files and error responses. - Requests without the opt-in. The opt-in header is never forwarded to OpenCode. On the reported session, the page drops from 16.8 MB to about 1.1 MB. Validation: - session-media.test.ts: only large renderable tool images change, and only their URI. - routes/session-media.test.ts: exact bytes and headers; invalid references, digest and position mismatches, the undo boundary, ownership and a move during the read. - proxy.test.ts: the transform applies to successful JSON only and releases once. - instance-proxy.test.ts: an opted-in list read is deferred, the opt-in is not forwarded, and plain list or single-message reads stay whole. - session-navigation.test.ts: history windows. - sdk-manager.test.ts: only list reads opt in, keeping other headers. - tool-content.test.ts: path resolution, forged paths rejected; it is now listed in the PR build. - tool-images.test.ts: a deferred image decodes from the route in the real ToolCall renderer, and a missing one shows the fallback. The UI own-tsconfig, server and root typechecks pass, as do: - the server suite (2,467 passed; one index.test.ts process test timed out under load and passes alone, as on dev); - the UI unit step of the PR build (564 tests); - seven browser suites (278 tests).
17 lines
6.2 KiB
Markdown
17 lines
6.2 KiB
Markdown
# Transcript rendering
|
|
|
|
Message content visibility and presets, token lines, tool renderers, native session lifecycle signals and transcript re-projection.
|
|
|
|
## Agent conventions
|
|
|
|
- The message-content popup and Chat settings share `components/transcript-visibility.ts` and the semantic icons from `components/message-content-icons.ts`; tool presentation metadata lives independently of renderers in `components/tool-call/tool-presentation.ts`. Popup styles live in `styles/components/transcript-filters.css`.
|
|
- Transcript presets are Nothing (hides every row, including all tools), Minimal, Balanced and Everything (`TRANSCRIPT_PRESETS`); saved Detailed choices load as Custom with their effective tool column. Chat settings shows the four as buttons; the popup shows the same choice as a native select under its title, where a disabled Custom entry appears only after a manual row change. Escape closes an open list before the popup. Preset names and descriptions are localized in every locale; covered by `tests/browser/transcript-presets.test.ts`.
|
|
- Status Tokens are independent of message token visibility. A collapsed section uses the non-interactive `renderSummary` slot for one ellipsized session input/output/cost line; expanded content shows session metrics then descendant-inclusive totals as separate wrapping groups. Keep this layout in `styles/panels/status-tokens.css`.
|
|
- Message token disclosures keep one row in both states: collapsed shows the input/output/cost summary chips on a single ellipsized line; expanded swaps the same row to all six chips beside the chevron, which wrap onto further lines when narrow instead of truncating (expansion exists to read every value). Keep the metric renderer local and chip/row styles in `styles/messaging/message-usage.css`.
|
|
- Tool-result images retain native `state.content` and render through the shared `components/tool-call/output-images.tsx` surface, including MCP and specialized tools. Keep image bytes out of text projections (copy/search/speech); image styles live in `styles/messaging/tool-call/images.css`.
|
|
- Transcript pages carry large tool images by reference: the UI's native message list reads (`lib/sdk-manager.ts`) and history windows (`fetchHistoryWindow`) send the `x-codenomad-media: deferred` opt-in (`server/src/opencode/session-media-contract.ts`), and the server replaces each tool image data URI over `INLINE_MEDIA_LIMIT` (32 KiB) with a `/api/workspaces/<id>/session-media/<session>/<message>/<part>/<index>/<sha256>` path (`opencode/session-media.ts`). `toolImageSource` resolves that path on the API base, so the lazy `<img>` loads only shown images; `routes/session-media.ts` serves the bytes of the owned session's native message, pinned by the digest and refused at or after an undo boundary, as an immutable sandboxed image. Single-message reads (`session.message.get`: pruning revisions, previews), live events, user attachments and requests without the opt-in stay whole, so revisions hash native content.
|
|
- Code Mode reads native `metadata.toolCalls` in `tool-call/renderers/execute-data.ts`; keep call-list styling in `styles/messaging/tool-call/execute.css`, with native image/output ownership in the shared tool shell.
|
|
- Native websearch text is parsed conservatively by `tool-call/renderers/websearch-data.ts`; unknown/oversized responses use the shared raw renderer. Recognized provider-consent Forms retain the generic schema/answer path, with layout in `styles/messaging/tool-call/websearch.css`.
|
|
- Session lifecycle features use OpenCode V2's published signals: one `session.execution.{succeeded,failed,interrupted}` per busy period (`lib/native-execution-events.ts`) and `session.retry.scheduled` for provider backoff. The schema-declared `session.idle`/`session.status` are never published; never wire a feature only to them. A terminal event is a hint until `session.active` confirms settlement, which alone announces "A session finished", only for root sessions ending in success, failure or an inactivity stop. Transcript reducers retire at settlement, deferred while prompts stay parked in their inbox or a user shell or unsettled tool call still needs reconciling there. Token/cost lines (`components/message-usage.tsx`) appear once per agent message: the row of an assistant message's final text (the one carrying "delete tools and reasoning") renders, once that message has native usage, the sum (`sumMessageUsage`) of its own usage and of every message holding a tool or reasoning part that cleanup selects (`technicalCleanupParts`). Reuse that selection rather than a separate projection. The line renders inside the content row, so it never shifts `<Index>` items or splits a technical group (whose Delete needs two tools); tool-only steps get no line. A message whose text is followed by tools also counts in the next message's line, as its tools belong to that selection. Sums cover the loaded transcript window only, like cleanup. Capture real streams with `scripts/test-session-lifecycle-native.mjs`.
|
|
- Native transcript events reduce only through `stores/opencode-data.ts` (never add another partial-part reducer), which publishes bounded snapshots through `hydrateMessages` (message-v2/instance-store.ts); these per-event re-projections must not invalidate unchanged messages: `hydrateMessages` reuses the resident `MessageRecord` object whenever content, explicit timestamps and prompt-display metadata are unchanged, and an identical whole snapshot returns before any store write or session-revision bump. Part/info comparisons use allocation-free structural equality (`partDataEqual`), never `JSON.stringify`. The SDK reduce mutates rows in place, so reference identity can never prove a session unchanged — never add a reference-anchored projection skip. A session's reducer is replaced (authoritative resync after an overflow, rotation failure or reconnect; pruning invalidation) only through `replaceTranscriptEntry`, which bumps the transcript's `entryRevision`; reactive readers of the reducer, such as the inbox behind queued badges (`getOpenCodeSessionInbox`), track it so they rebind instead of keeping the disposed reducer's last state (`opencode-inbox-rebinding.test.ts`, `tests/browser/session-rendering.test.ts`).
|
|
- Shared speech byte/MediaSource transport lives in `lib/audio-utils.ts`; playback ownership and cancellation stay with its callers.
|