CodeNomad/dev-docs/BROWSER_AUTOMATION.md

Browser Automation

CodeNomad exposes codenomad.browser for controlling the visible web preview attached to the current OpenCode session. It supports open, navigate, snapshot, screenshot and the same page actions as codenomad.act: click, type, select, hover, drag, scroll, press and wait.

Integration and shared bridge

Browser previews are an ordinary desktop feature. They must work in user projects outside the CodeNomad source checkout and do not require Developer Mode.

• One bundled V2 plugin, codenomad.automation, is provisioned through native OpenCode plugin discovery in the selected host/WSL daemon namespace. Its definitions follow backend presence. Loading it does not start a browser or perform an action.
• Provisioning follows the authenticated connection, including reconnects. config.get supplies the first (global) discovery directory; neither the backend environment nor CLI debug paths identifies an existing daemon's roots. OpenCode recursively watches that whole root: a lease mtime update beneath it reloads every loaded location. New bundles and leases live in the sibling .codenomad/<SHA-256 of discovery root>/ namespace. Existing outside-root storage remains valid; inside-root storage migrates and the plugin reads both old and new presence directories so older backends remain represented. Only the new directory receives new heartbeats. WSL translates native paths to the selected distro's UNC path only for filesystem access. Missing authoritative discovery information is reported rather than replaced with a guessed directory.
• The plugin exposes browser, inspect, act, and screenshot while a desktop backend is present. Tool availability is separate from plugin loading; there is no Developer Mode access gate.
• Browser and developer calls share the backend's loopback registration and private native transport. Native instrumentation is prepared at normal startup, so tool access does not require a toggle or an additional restart. Execution still verifies the selected backend, supported native capabilities and invoking session.
• Developer inspection targets the native window, independently of its selected project/conversation (including empty views). The invoking session still identifies an owning backend; it need not be visible. Actions retain the inspected window/run and document-scoped accessibility references, so agents can navigate back themselves. Session/project selection changes do not invalidate window automation. Reloads, replaced pages and changed native targets require fresh inspection. Browser-preview tools retain their separate per-session attachment rules.
• packages/server/src/server/routes/automation-plugin.ts serves the existing /api/opencode-plugin/automation endpoint with the existing per-process token.
• browser-claim verifies that exactly one CodeNomad instance owns the session before an open action asks the UI to create a preview.
• browser-probe also requires a visible native preview registered to that session.
• browser-execute validates the browser action and forwards browser.probe or browser.execute over the existing fenced native-parent request channel.

The bridge remains loopback-only, token-authenticated, bounded, and session-location gated. The removed V1 plugin runtime and browser-specific bridge routes are not part of this design.

Page engine

codenomad.inspect / codenomad.act (the CodeNomad window) and codenomad.browser (the session preview) run on the same page engine in packages/server/src/automation/page/, which only needs a DevTools protocol channel: snapshot.ts renders the accessibility tree, actions.ts performs actions, input.ts dispatches trusted pointer, wheel and keyboard input, keys.ts maps key chords, parse.ts validates tool input on both sides of the bridge, protocol.ts holds the shared protocol primitives and refs.ts names DOM nodes. The app uses DeveloperCdp; previews use automation/browser-page.ts. The preview's snapshot({ ref, query }) has the same outline and scoping as the app's inspect.

• Inspection is an indented outline with refs, read as plain text. Wrappers, unnamed images and text repeating its parent's name are dropped; fields show their value instead of their inner nodes; native selects list their options with selected/disabled. A character budget (32 000 by default) is shared among siblings: small parts stay whole while large ones collapse in place with (+N hidden) or (+N more), keeping their first and last items, so nothing is cut silently. ref expands one subtree and query keeps elements whose name, value or description contains it, or whose role is exactly it, with their contents and ancestors.
• A ref names one node within one target, run and navigation epoch. It keeps its name and stays usable for the life of that page generation, whatever is inspected later; a node that left the page fails its action explicitly.
• type focuses the field and refuses one that cannot take focus (hidden, disabled, behind a modal dialog), because native text insertion goes to the focused element. It then replaces the field's text by default (select, then native insertion, so framework input handlers fire); clear: false appends. Date, time, range and color inputs are validated before anything changes, then set with input/change, keeping the browser's normalization; email and number inputs append as a value. select picks native <select> options by label, else value, and dispatches input/change only when the choice changes; clicking an option ref picks that exact option. Results return the field value (passwords masked, rich text with its line breaks) or the chosen labels.
• click is a real pointer click at the centre of the element's visible part, refused when another element would receive it (hit testing respects pointer-events: none, and a label of the target counts as the target); an element without a visible box is activated programmatically. click on a native select asks for select instead of opening its out-of-page popup. hover and scroll (wheel on a ref or the page) use the same aiming. drag requires both ends already visible, without scrolling under a saved start point; it uses stepped pointer moves for pointer-based drag libraries (HTML drag-and-drop is not emulated). press dispatches a key or chord (Enter, Shift+Tab, Control+K, Mod+K = Meta on macOS) to the focused element, optionally focusing a ref first and refusing it unless focus actually moved there, and returns the newly focused element. Failed input sequences best-effort release held buttons/keys on their original channel without replaying the action. wait polls (≤ 15 s) for text in the rendered page or in labels, titles, placeholders and values, or for its disappearance. Native application-menu accelerators handled by the host may not see dispatched keys.
• Trust rule: in the CodeNomad window page functions run under __codenomadAutomationDepth, and every trusted DevTools input (pointer, wheel, keys, text insertion) under the short __codenomadAutomationInputUntil lease in performance.now() units. Ending it retains its end timestamp: lib/automation-input.ts checks the originating event's timeStamp, not when a stalled renderer handles it. Delayed automated keys and their same-turn implicit submit clicks remain automated after expiry. form-answer-interaction.ts refuses human provenance for those events, so automation can never produce the trusted dock click or key press a human Form answer requires. Never add unmarked trusted input to the application window, and never expose arbitrary script evaluation there.
• An input timeout has an unknown outcome, not a proven failure: a busy renderer can deliver an already admitted key or click later, including when cleanup releases its held input. Cleanup still attempts release when its marker write times out, but never repeats the press. The tool explicitly instructs inspection before retrying; never automatically retry these inputs.
• The marker returns the page's own epoch timestamp for CDP mouse/key events. Cleanup retains the original press timestamp even if its marker renewal times out, so delayed releases remain within the original provenance interval independently of command-timeout settings. Missing/invalid provenance timestamps refuse before input dispatch.
• When a Preferences or remote window has focus, window automation targets the most recently used application window, as the update flow does; the Preferences window itself is not an automation target.
• packages/ui/tests/browser/automation-forms.test.ts drives a real Chromium page and the real Mission editor through DeveloperCdp; automation-input.test.ts covers trusted input and proves that automation clicks and Enter in the real InterruptionDock never carry the human-answer header, while a real user click does.

Guarded preview channel

The native hosts retain guest ownership, visibility, URL policy, storage, history, emulation, navigation and screenshots. They no longer implement page snapshots, refs or form input. The backend's BrowserPage serializes page operations per session and uses only the private native-parent browser.page.begin/send/check/end channel, never renderer IPC or Tauri commands.

Electron maps the channel's Input.insertText to WebContents.insertText and Input.dispatchKeyEvent to WebContents.sendInputEvent (including the character event when the key produces text): direct CDP text/key input can be silently ignored by a newly opened embedded/background guest until its first pointer click. These remain native inputs (no value setter or framework-specific action), require the same current lease, and never show or raise the application window.

begin binds one expiring operation lease to the exact visible registered guest and document generation. Every send is bounded and allowlisted to the protocol methods the engine consumes; every check revalidates visibility, registration and document identity. Responses are fenced again after native I/O. A changed, hidden, ambiguous or missing target fails rather than switching guests. The backend owns refs per session and generation; it issues no refs from an unconfirmed read and never retries a failed mutation. end cleans up that exact lease even after cancellation or navigation, without detaching someone else's emulation overrides. Missing channel support in an older host fails explicitly, with no legacy implementation fallback.

Only fixed engine functions and protocol arguments cross this channel: models cannot choose protocol methods or evaluate scripts. This does not grant guest pages any application/native capability. Preview pages do not receive the application's human-input provenance variables. tests/browser/automation-preview.test.ts exercises both production ends against a real isolated Electron guest, with the window hidden to avoid desktop focus changes.

September 2026 architecture decision

The original beta-era MCP limitation is obsolete: current OpenCode V2 MCP documentation specifies _meta.sessionID for session-backed calls, including Code Mode. This metadata is correlation, not authorization. MCP is now a viable integration choice.

For this in-application preview, use one bundled V2 automation plugin rather than add another protocol server, MCP configuration, discovery and cancellation implementation. The user explicitly chose unified distribution for browser and developer tools, with all tools available as the initial baseline rather than introducing a mode-based permission system. CodeNomad already needs the authenticated backend-to-native transport, and the existing pruning integration supplies normal native plugin discovery and backend presence mechanics. The plugin uses OpenCode's invoking tool context; session ownership is re-read through the shared connection and validated worktree inventory at execution. Neither a plugin location nor a caller-provided session ID alone grants access.

This decision concerns distribution and transport, not a replacement for the native controllers: OpenCode does not own CodeNomad's visible Electron/WebView2 preview, its window, storage partition or permissions. Keep those responsibilities native, preserve the independently owned OpenCode daemon, and keep the current connection-scoped compatibility adapter for V2 location/API differences.

The pinned @opencode/plugin 2.0.4 Promise tool context has no signal. Browser execution currently has native/HTTP deadlines rather than a guarantee of immediate cancellation when an agent turn is interrupted. Do not invent tool.signal or claim that switching to MCP alone cancels already-dispatched native work. Any stronger cancellation contract needs end-to-end propagation and native tests.

Native Hosts

Electron uses hardened <webview> tabs sharing one persistent browser partition (persist:codenomad-browser-shared-v1) per native CodeNomad channel/config profile. Attachment verifies guest ownership, allows only credential-free HTTP(S) URLs, denies permissions and downloads, and uses Chromium accessibility/CDP commands for snapshots and actions.

Windows Tauri uses a child webview owned by the local application window. Commands, capabilities, navigation checks, bounds, visibility, and storage are managed by browser_controller.rs. Application capabilities select primary local-* webviews, never their parent-window wildcard: Tauri combines window/webview matching with OR, so a window grant would also authorize an untrusted child. Full primary-renderer navigation disposes its preview children. Final-window decisions count native windows, including multi-webview windows. Other Tauri platforms continue to use the existing iframe preview and do not advertise native browser automation.

The UI routes an autonomous open request to the one loaded instance containing the session, claims the request for one local window, selects that session, and opens its preview. Hidden or duplicate targets are rejected rather than selected implicitly.

Each session has its own preview record, scoped by project folder and session ID. Returning to chat, selecting another session, viewing Info or hiding the project keeps explicitly opened browser guests mounted, preserving their native history and document state. Guest DOM order remains stable; inactive transcripts and composers still unmount under the existing memory policy. Electron unregisters hidden guests from automation without destroying them; Tauri hides its existing child. The preview toolbar's X also returns to chat without destroying the browser or its history. Older project-wide URLs are adopted once by the first restoring session rather than copied into every conversation.

Closing the project/window or restarting the renderer ends the live document. The saved URL history and cursor are restored only when the user explicitly opens that session's browser; neither session activation nor startup creates hidden browsers. Only its current URL loads at first. Back/Forward reuse owned native CDP entry IDs when available; a saved entry without a live ID loads only that selected URL, never replays intermediate pages. URL-only restoration does not preserve DOM, unsent forms, scroll, JavaScript memory or POST bodies. Existing per-session cookies/site storage remain persistent. URLs must be credential-free HTTP(S), and history is limited to 32 entries/session within the existing 4 KB aggregate preview layout value; history is trimmed before saved previews are evicted. This is a navigation journal, not a disk image of Chromium sessions.

Shared profile and upgrade

Native preview tabs share cookies, sign-ins, localStorage and other site storage across sessions and projects within the same CodeNomad host/profile. Windows Tauri selects the common browser/shared-v1 WebView2 data directory. This browser profile remains separate from the trusted application renderer, remote-window profiles, other CodeNomad config/channel profiles and the other desktop engine. Documents, forms, sessionStorage, native navigation stacks, persisted history cursors and automation attachments remain tab/session-owned. Logging out of a site in one tab logs out the others, just as in a normal browser.

Existing per-session browser profiles are retained on disk, but are not automatically merged: they can contain conflicting accounts and database state. The new shared profile may require one initial sign-in. Iframe fallbacks retain their opaque-origin capability sandbox and are not advertised as a full shared native browser profile.

Focused Validation

• Server: automation-plugin.test.ts, automation/desktop-plugin.test.ts, routes/automation-plugin.test.ts, and the shared pruning-presence regressions
• Native discovery: build with npm run build:automation --workspace @neuralnomads/codenomad, then run node scripts/test-automation-native.mjs <absolute-cli-path>. The fixture uses an isolated daemon/config/database and exercises ordinary-project discovery, all tools without activation, execution fences, independent leases, shutdown and reopening. CI crosses legacy and current V2 contract families, including 2.0.7.
• Electron: browser-controller.test.ts, browser-webview-security.test.ts, preload tests
• Tauri: browser_controller.rs and browser_controller_regressions.rs tests, including actual capability resolution, primary-renderer cleanup, window counting, lock ordering and expired dispatch
• UI: browser-frame-security.test.ts, lib/native/browser.test.ts and tests/browser/browser-frame-native.test.ts (real Solid/Electron guests, insecure HTTP, registration disposal and Tauri IPC failure recovery). Linux Electron rendering requires a display; CI runs the browser suite under Xvfb.
• Session lifetime: tests/browser/session-browser-history.test.ts exercises the real instance shell/store, a real Electron guest and the Tauri IPC boundary across chat/session/project/Info transitions, the preview's X, retained Back history and legacy URL migration.
• Shared site state: the same suite tests real Electron HttpOnly-cookie/localStorage sharing, independent documents/history, app-profile isolation and shared logout; Rust regressions verify Tauri's common persistent browser directory.