mirror of
https://github.com/NeuralNomadsAI/CodeNomad.git
synced 2026-11-16 02:42:17 +00:00
merge: integrate WebKit automation and mobile profiles
This commit is contained in:
commit
ef69b40858
125 changed files with 6870 additions and 641 deletions
|
|
@ -52,7 +52,7 @@ only a qualified path is retained across calls.
|
|||
|
||||
## Page engine
|
||||
|
||||
`codenomad.inspect` / `codenomad.act` (the CodeNomad window) and `codenomad.browser` (the session preview) run on the same page engine in `packages/server/src/automation/page/`, which only needs a DevTools protocol channel: `snapshot.ts` renders the accessibility tree, `actions.ts` performs actions, `input.ts` dispatches trusted pointer, wheel and keyboard input, `keys.ts` maps key chords, `parse.ts` validates tool input on both sides of the bridge, `protocol.ts` holds the shared protocol primitives and `refs.ts` names DOM nodes. The app uses `DeveloperCdp`; previews use `automation/browser-page.ts`. The preview's `snapshot({ ref, query })` has the same outline and scoping as the app's `inspect`.
|
||||
`codenomad.inspect` / `codenomad.act` (the CodeNomad window) and `codenomad.browser` (the session preview) run on the same page engine in `packages/server/src/automation/page/`. `protocol.ts` defines the semantic `PageDriver`: normalized tree, stable node/handle resolution, predefined function calls with JSON arguments, geometry, viewport, hit testing, native input, handle release and checkpoints. `cdp-driver.ts` adapts that contract to DevTools; `native-driver.ts` uses the fenced parent channel. `snapshot.ts` retains the outline renderer and budgets, `actions.ts`/`input.ts` retain action orchestration, `keys.ts` maps chords, and `refs.ts` owns document-scoped references. The app uses `DeveloperCdp`; previews use `automation/browser-page.ts`. The preview's `snapshot({ ref, query })` has the same outline and scoping as the app's `inspect`.
|
||||
|
||||
- Inspection is an indented outline with refs, read as plain text. Wrappers, unnamed images and text repeating its parent's name are dropped; fields show their value instead of their inner nodes; native selects list their options with `selected`/`disabled`. A character budget (32 000 by default) is shared among siblings: small parts stay whole while large ones collapse in place with `(+N hidden)` or `(+N more)`, keeping their first and last items, so nothing is cut silently. `ref` expands one subtree and `query` keeps elements whose name, value or description contains it, or whose role is exactly it, with their contents and ancestors.
|
||||
- A ref names one node within one target, run and navigation epoch. It keeps its name and stays usable for the life of that page generation, whatever is inspected later; a node that left the page fails its action explicitly.
|
||||
|
|
@ -69,9 +69,23 @@ The native hosts own the guests: their visibility, URL policy, storage, history,
|
|||
|
||||
Electron maps the channel's `Input.insertText` to `WebContents.insertText` and `Input.dispatchKeyEvent` to `WebContents.sendInputEvent` (including the character event when the key produces text): direct CDP text/key input can be silently ignored by a newly opened embedded/background guest until its first pointer click. These stay native inputs, never a value setter or framework-specific action; they require the same current lease and never show or raise the application window. Key and pointer input waits until Chromium's display compositor runs: it lives in the GPU process, which exists even with GPU acceleration disabled, and in the browser process when Electron runs with `--in-process-gpu` or `--single-process`. Without it, as during a cold launch, the guest renders no frame and drops that input. At the request deadline the controller refuses the input unsent instead of losing it.
|
||||
|
||||
`begin` binds one expiring operation lease to the exact visible registered guest and document generation. Every `send` is bounded and allowlisted to the protocol methods the engine consumes; every `check` revalidates visibility, registration and document identity. Responses are fenced again after native I/O. A changed, hidden, ambiguous or missing target fails rather than switching guests. The backend owns refs per session and generation; it issues no refs from an unconfirmed read and never retries a failed mutation. `end` cleans up that exact lease even after cancellation or navigation, without detaching someone else's emulation overrides. A host without channel support fails explicitly.
|
||||
`begin` binds one expiring operation lease to the exact visible registered guest and document generation. Every `send` is bounded and allowlisted to semantic operations; every `check` revalidates visibility, registration and document identity. Responses are fenced again after native I/O. A changed, hidden, ambiguous or missing target fails rather than switching guests. The backend owns refs per session and generation; it issues no refs from an unconfirmed read and never retries a failed mutation. `end` cleans up that exact lease even after cancellation or navigation, without detaching someone else's emulation overrides. A host without channel support fails explicitly.
|
||||
|
||||
Only fixed engine functions and protocol arguments cross this channel: models cannot choose protocol methods or evaluate scripts. Never grant guest pages application/native capabilities through it, nor expose the application's human-input provenance variables to preview pages. `tests/browser/automation-preview.test.ts` exercises both production ends against a real isolated Electron guest, with the window hidden to avoid desktop focus changes.
|
||||
`begin` announces `driver: "cdp" | "webkit"` and `contractVersion: 1`. `send` accepts one closed `operation` union: tree, resolve, call, scroll-into-view, geometry, viewport, hit-test, pointer, wheel, keyboard, insert-text, release-handles and checkpoint. Both native hosts reject unknown operation fields, function IDs and argument shapes. Calls carry a registered function ID, opaque target handle and JSON arguments. The hosts obtain executable programs from their own shared fixed registry; the private wire carries operations rather than protocol methods or program text. WebKit implements these operations through its isolated program and target-local native input adapters. Public WebKit automation remains disabled.
|
||||
|
||||
WebKit's fixed isolated-world reader bundles `dom-accessibility-api` 0.7.1 (MIT, notice included in the program and `webkit/LICENSE.dom-accessibility-api`). It computes names, descriptions and roles and adapts native field states to the shared renderer. Password values use the shared read-value policy's maximum sixteen masking bullets. Hidden, inert and ARIA-hidden branches are excluded. Open and captured closed shadow roots and same-origin frames are traversed; cross-origin frames receive an opaque, non-actionable boundary marker. Chromium's main-frame `Accessibility.getFullAXTree` exposes closed-root controls and frame elements, with frame contents available through separate frame-tree requests.
|
||||
|
||||
A fixed document-start page-world hook weakly associates each host with its `attachShadow` root, preserving wrapper identity for the host's lifetime. A synchronous host-bound reply carries one undispatched PointerEvent in its predicted-event list. Its native `relatedTarget` retains the root without shadow retargeting or JavaScript-value serialization. These vendor-named, untrusted reference messages stay separate from native input delivery. Host-bound listeners support nested roots and document recreation. The isolated reader invokes native event getters, validates ShadowRoot brand, exact host, mode and owner document, and rewraps the root before traversal or ref resolution. Forged endpoints receive an explicit unavailable marker. The bridge consumes PointerEvent constructor prediction lists and `getPredictedEvents`. Declarative parser-created closed roots are opaque: parser attachment bypasses the JavaScript hook and consumes its template outside document mutation observations. Their rendered controls stay outside WebKit snapshots and refs.
|
||||
|
||||
Each WebKit document owns weak numeric node identities and opaque token-prefixed handles, bounded to 20,000 live nodes. Mutation observations retire removed identities even after reinsertion; document recreation and pagehide rotate the document token. Subframe pagehide retires that document's nodes. Lifecycle listeners are reinstalled after document streaming recreation. Fragment and History API changes preserve identity. Native reads pass JSON as JavaScript argument dictionaries in a retained named content world, with loading, registration, visibility, deadline and document checks around native callbacks. Hidden, closed and replaced guests revoke leases. Regenerate the program and native header with `node --import tsx scripts/build-webkit-read.ts`; run `node --import tsx scripts/compare-webkit-read.ts` for CDP/read-program comparisons on the shared Electron HTML fixtures.
|
||||
|
||||
WebKit aiming uses visible client rectangles, same-origin frame offsets/clipping and deep shadow/frame hit testing. Native wheel dispatch also requires an isolated hit test. Form preparation, special-value setters, select validation/events and result masking share the fixed registry. Contenteditable selection uses `Selection.setBaseAndExtent`, including editing hosts inside shadow roots. Wait searches the semantic tree plus field attributes/values, including captured roots and same-origin frames, with the shared fifteen-second polling bound.
|
||||
|
||||
Native input dispatch follows an isolated document preflight and registration/navigation/deadline checks on the UI thread and at completion. History sampling publishes URL metadata; native navigation/load callbacks fence documents. Each lease retains at most 64 pending key/button releases before admitting downs. `end` and expired-lease admission best-effort release that exact document; an outstanding release fenced by same-document navigation can renew its document check once. Completed releases and input mutations are never replayed. The adapters target the guest view directly, with native text insertion and explicit modifier transitions, preserving desktop focus and the physical pointer.
|
||||
|
||||
Background WKWebView has qualified input limits: an initial standalone hover after navigation can omit `pointerenter`; a Meta editing equivalent such as `Mod+A` selects through its native responder while consuming the main `A` keydown before DOM listeners. Meta down/up and the following Enter down/up are delivered in the real parent-channel action flows. Native application-menu shortcuts retain the host handling described above.
|
||||
|
||||
`functions.ts` owns the fixed program source and argument metadata. Electron uses it directly, and Tauri compiles `functions.generated.json`. Application input-provenance functions belong only to the trusted application's direct CDP driver. Never grant guest pages application/native capabilities through this channel, nor expose the application's human-input provenance variables to preview pages. `tests/browser/automation-preview.test.ts` exercises both production ends against a real isolated Electron guest, with the window hidden to avoid desktop focus changes.
|
||||
|
||||
## Design constraints
|
||||
|
||||
|
|
@ -85,13 +99,17 @@ Electron uses hardened `<webview>` tabs sharing one persistent browser partition
|
|||
|
||||
Tauri uses a child webview owned by the local application window: WebView2 on Windows, WKWebView on macOS 14+, and WebKitGTK on Linux. Pages load directly in the native view. `embedded_view.rs` owns native creation, geometry, visibility, closure, shared storage and mandatory credential-free HTTP(S) navigation, permission, download and new-window protections; its Windows adapter also installs native accelerators. `browser_controller.rs` owns session registrations, renderer generations, history, events, emulation and automation leases. Application capabilities select primary `local-*` webviews, never their parent-window wildcard: Tauri combines window/webview matching with OR, so a window grant would also authorize an untrusted child, including localhost content. Full primary-renderer navigation disposes its preview children. Final-window decisions count native windows, including multi-webview windows.
|
||||
|
||||
Native views follow the preview bounds and hide beneath local menus, lists and dialogs. Reshow applies the latest bounds before revealing the retained document. WebKit Back/Forward uses each view's native back-forward list, with registration-owned entry identities and a 32-entry journal; restored sessions load the selected saved URL on explicit opening. WebKit viewport presets change dimensions only. Element comments are unavailable in native previews.
|
||||
Native views follow the preview bounds and hide beneath local menus, lists and dialogs. Reshow applies the latest bounds before revealing the retained document. WebKit Back/Forward uses each view's native back-forward list, with registration-owned entry identities and a 32-entry journal; restored sessions load the selected saved URL on explicit opening. WebKit presets apply fitted native CSS viewports and per-view mobile identity, with density/input differences specified in `BROWSER_DEVICE_EMULATION.md`. Element comments are unavailable in native previews.
|
||||
|
||||
Local Tauri primary renderers receive `__CODENOMAD_BROWSER_CAPABILITIES__` during initialization, with independent `embedding`, `history`, `emulation` and `automation` flags. WebKit publishes `reason: "webkit"` for its unavailable emulation/automation; unsupported targets publish `reason: "unsupported-platform"`. UI host selection consumes this contract; registration/history readiness stays independent of device emulation. UI registrations request hidden construction and recheck `localOverlayOpen()`, clipped bounds and display scale before the first reveal, including asynchronous construction beneath a dialog.
|
||||
Local Tauri primary renderers receive `__CODENOMAD_BROWSER_CAPABILITIES__` during initialization, with independent `embedding`, `history`, `emulation` and `automation` flags. WebKit publishes embedding/history/emulation true, automation false and `reason: "webkit"`; unsupported targets publish `reason: "unsupported-platform"`. UI host selection consumes this contract; registration/history readiness stays independent of device emulation. UI registrations request hidden construction and recheck `localOverlayOpen()`, clipped bounds and display scale before the first reveal, including asynchronous construction beneath a dialog. WebKit registration/update/profile commands await native scale and configuration off the UI thread.
|
||||
|
||||
Agent browser automation (`codenomad.browser`, including open, probe and page actions) is unavailable on Tauri macOS/Linux and returns an explicit refusal. Open these previews manually, or use Electron or Tauri Windows for agent automation. Plain-browser and remote-view clients use the sandboxed iframe proxy.
|
||||
|
||||
The Tauri launcher sets `CODENOMAD_TAURI_WEBKIT` on each backend spawn after shell-environment resolution. The authenticated bridge uses this host marker to return the WebKit refusal after validating session ownership, including hosts whose native-parent transport is unavailable. Mission-only ownership claims retain their normal admission path.
|
||||
The Tauri launcher pipes backend stdin and sets `CODENOMAD_NATIVE_PARENT=1` after shell-environment resolution on Windows, macOS and Linux. The private channel fences requests and responses against the backend process generation and absolute deadline. WebKit's internal open, navigate, probe and visible-PNG operations use the common target/open-claim registry, with native navigation completion/failure callbacks and registration/document checks around capture. Native adapter code is compiled only for its target platform.
|
||||
|
||||
WebKit publishes `CODENOMAD_NATIVE_BROWSER_AUTOMATION=0` independently of the transport and retains `CODENOMAD_TAURI_WEBKIT` for host identity. The authenticated bridge returns the public WebKit refusal after session ownership validation; the UI's private open-request subscription follows embedding availability. Mission-only ownership claims retain their normal admission path. Desktop transport/capability markers are stripped from the shared daemon's startup environment.
|
||||
|
||||
Private navigation can recover an owned visible guest from an engine-generated error document by loading a validated HTTP(S) destination. Page reads and screenshots require a current credential-free HTTP(S) document. WebKit failure results survive the subsequent load-finished notification; fragment-only navigation completes from the confirmed native URL when document-load callbacks are absent. Capture responses retain the 16 MiB base64 limit, and a hide, close, replacement, navigation or expired backend generation fences the native callback result.
|
||||
|
||||
On macOS, a page reading the clipboard after a user gesture can display the system's native **Paste** button. That button performs user-initiated paste. Site permission requests, downloads and new windows are denied by the native preview policy.
|
||||
|
||||
|
|
@ -108,6 +126,8 @@ On macOS, a page reading the clipboard after a user gesture can display the syst
|
|||
|
||||
The UI routes an autonomous open request to the one loaded instance containing the session, claims the request for one local window, selects that session, and opens its preview. Hidden or duplicate targets are rejected rather than selected implicitly.
|
||||
|
||||
Native open claims retain their request deadline and backend-generation fence. The UI revalidates the claim after asynchronous preview-record creation and before publishing the preview; a rejected record's token is released. Native-parent stdin preserves UTF-8 across buffer boundaries, and timed-out response IDs stay retired independently of subsequent requests.
|
||||
|
||||
## Focused Validation
|
||||
|
||||
- Server: `automation-plugin.test.ts`, `automation/desktop-plugin.test.ts`, `routes/automation-plugin.test.ts`, and the shared pruning-presence regressions
|
||||
|
|
@ -115,11 +135,16 @@ The UI routes an autonomous open request to the one loaded instance containing t
|
|||
- Electron: `browser-controller.test.ts`, `browser-webview-security.test.ts`, preload tests
|
||||
- Tauri: `embedded_view_tests.rs`, `browser_controller.rs` and `browser_controller_regressions.rs` tests, including ownership, fail-closed child policy, creation rollback, replacement fencing, actual capability resolution, primary-renderer cleanup, window counting, lock ordering and expired dispatch
|
||||
- WebKitGTK history: `python3 packages/tauri-app/scripts/test-webkit-history-linux.py` in a graphical Linux session with GTK 3/WebKitGTK 4.1 development packages. Run with `GDK_BACKEND=wayland` and `GDK_BACKEND=x11`; the standalone native example includes the production adapters, bounded-window selection and handle allocator. It covers redirects, fragments, same-URL `pushState`, `replaceState`, native traversal, branching and the 32-entry bound.
|
||||
- WebKit private parent channel: `packages/tauri-app/scripts/test-browser-parent-webkit.py` builds the production controller, framing, native adapters and vendored runtime with an isolated owning-window inventory and renderer-claim fixture. The Node fixture uses the production `NativeParent` over real child stdout/stdin pipes. It covers autonomous open from an absent preview, redirects, fragments, failed loads/TLS, deadlines, PNG, hidden/duplicate targets, a fixture-renderer dialog, close/reopen and stale responses. Native PNG-completion tests revoke visibility or close the guest immediately before callback acceptance. The harness owns its fixture child and local servers; the shared daemon keeps its independent lifecycle. Build the Node fixture and run the graphical harness as described in `packages/tauri-app/tests/browser-parent/README.md`.
|
||||
- UI: `browser-frame-security.test.ts`, `lib/native/browser.test.ts` and `tests/browser/browser-frame-native.test.ts` (real Solid/Electron guests, insecure HTTP, registration disposal and Tauri IPC failure recovery). Linux Electron rendering requires a display; CI runs the browser suite under Xvfb.
|
||||
- Session lifetime: `tests/browser/session-browser-history.test.ts` exercises the real instance shell/store, a real Electron guest and the Tauri IPC boundary across chat/session/project/Info transitions, the preview's X, retained Back history and project-wide URL migration.
|
||||
- Shared site state: the same suite tests real Electron HttpOnly-cookie/localStorage sharing, independent documents/history, app-profile isolation and shared logout; Rust regressions verify Tauri's common persistent browser directory.
|
||||
|
||||
## Agent conventions
|
||||
|
||||
- After changing fixed page programs or their signatures, run `node --import tsx scripts/build-page-function-registry.ts` and the driver contract tests. Keep `functions.generated.json` derived from the authored registry. Bump the private driver contract version for incompatible operation/signature changes. Preserve the fixed-program hashes and snapshot goldens when behavior is unchanged. Never accept caller-supplied protocol names or JavaScript source in the page channel.
|
||||
|
||||
- Give `embedded_view.rs` child modules explicit `#[path = "embedded_view/..."]` attributes so the native qualification example can include the production layer directly. Keep fixture-only lifecycle controls inside the example executable.
|
||||
|
||||
- Native session previews behave as tabs in one browser: share the persistent preview profile's cookies/site storage across sessions and projects within a CodeNomad data profile/config pair, but never with the trusted app renderer, remote-window profiles or the other desktop engine. Keep documents, navigation stacks, history cursors and automation attachments session-owned. Electron uses `persist:codenomad-browser-shared-v1`; Windows/Linux Tauri uses `browser/shared-v1`; macOS Tauri uses its profile/config-derived identified WebKit store. Leave per-session preview profiles found on disk in place and never merge them into the shared profile: they can hold conflicting accounts. Iframe fallbacks keep their sandbox; never grant guests native capabilities.
|
||||
- Browser preview state is keyed by project folder and session ID. Keep explicitly opened guests mounted in stable DOM order across chat/session/Info/project visibility changes; the preview's X also hides it and returns to chat without discarding history. Only their lightweight session shells remain: inactive transcripts and composers still unmount and the transcript memory budget stays independent. Hidden guests must stop being automation targets (Electron unregisters, Tauri hides). Saved project-wide URLs migrate once to the first restoring session. Across renderer restart, restore URL history/cursor only on explicit browser opening; never eagerly visit saved pages or persist native pageState/forms/POST bodies. Reuse owned CDP history IDs for live traversal and load just the selected saved URL when its native entry no longer exists. History is bounded to 32 credential-free HTTP(S) URLs/session within the 4 KB aggregate layout budget; trim history before evicting saved previews.
|
||||
|
|
|
|||
|
|
@ -11,21 +11,71 @@ reloads the preview because applications can cache device detection at startup.
|
|||
Returning to any dimension preset clears all overrides and reloads again.
|
||||
|
||||
Electron and Windows Tauri apply overrides to the owned native preview only.
|
||||
Tauri macOS (WKWebView) and Linux (WebKitGTK) embed native previews with native
|
||||
Back/Forward history; all six presets change dimensions only. The sandboxed iframe
|
||||
fallback also changes dimensions only. A translated tooltip identifies dimensions-only
|
||||
previews. Native display, history, device emulation and agent automation are separate
|
||||
host capabilities; WebKit previews support manual browsing, while agent browser
|
||||
Tauri macOS (WKWebView) and Linux (WebKitGTK) apply all six CSS viewports with native
|
||||
view geometry and per-view page zoom. Mobile presets use a generic WebKit mobile
|
||||
user agent, identical in HTTP and `navigator.userAgent`; dimension presets restore
|
||||
the desktop identity. Profile/template changes and desktop reset reload the guest;
|
||||
an identical request is a no-op. The sandboxed iframe fallback changes dimensions
|
||||
only. Translated tooltips distinguish Chromium emulation, native WebKit profiles
|
||||
and dimensions-only previews. Native display, history, device emulation and agent
|
||||
automation are separate host capabilities; WebKit previews support manual browsing, while agent browser
|
||||
automation requires Electron or Tauri Windows. Never expose application
|
||||
privileges or a general CDP bridge to preview pages. Tauri's synchronous WebView2
|
||||
callbacks are awaited off the UI thread. Electron retains its debugger attachment
|
||||
while emulation is active so the overrides survive accessibility automation.
|
||||
|
||||
Canonical profile data lives in `packages/ui/src/lib/native/browser-emulation.json`.
|
||||
Native adapters use the installed Chromium version for the simulated user agent.
|
||||
These generic profiles are a reproducible Android/Chromium layout reference;
|
||||
Gecko/Waterfox, Safari, mobile OS text settings, keyboard occlusion and
|
||||
browser/system bars are outside their scope.
|
||||
Chromium adapters use the installed Chromium version for the simulated user agent.
|
||||
WebKit uses `Mozilla/5.0 (Mobile) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile Safari/605.1.15`.
|
||||
These generic profiles provide engine-specific layout references; mobile OS text
|
||||
settings, keyboard occlusion and browser/system bars are outside their scope.
|
||||
|
||||
## Native WebKit profile contract
|
||||
|
||||
The UI sends clipped application-CSS bounds with `presentation.css: true` and an
|
||||
optional fixed `presentation.viewport`; emulation commands carry the same optional
|
||||
`viewport`. The native host reads primary-renderer page zoom and monitor scale on
|
||||
the UI thread. It fits the child into that physical canvas and finds integer native
|
||||
logical dimensions with a common zoom interval yielding the requested integer CSS
|
||||
width/height on both engines. Responsive uses the available CSS canvas. This keeps
|
||||
effective viewport dimensions stable across app zoom and monitor DPI, including
|
||||
clipped/narrow canvases. Pointer positions and wheel deltas scale through guest zoom.
|
||||
|
||||
Profile and canvas state belong to the registration. Replacement, unregister and
|
||||
primary-renderer cleanup clear them; other previews and the application renderer
|
||||
retain their own UA, scale and input. Commands serialize off the UI thread, revoke
|
||||
page leases before native changes, fence publication against registration changes
|
||||
and restore the preceding native configuration on application failure. Dimension
|
||||
resizes retain the document; profile/template changes reload after native application.
|
||||
|
||||
| Property | Chromium mobile | WKWebView | WebKitGTK |
|
||||
| --- | --- | --- | --- |
|
||||
| Portrait / landscape CSS viewport | 390 × 844 / 844 × 390 | Same | Same |
|
||||
| HTTP / JS mobile identity | Android Chromium | Matching generic WebKit | Matching generic WebKit |
|
||||
| DPR | 2.75 | 2.75 with runtime-checked `_setOverrideDeviceScaleFactor:`; native density if absent | Monitor scale × fitted page zoom |
|
||||
| Screen / screen orientation | Profile dimensions, portrait-primary/0 or landscape-primary/90 | Native desktop screen/orientation | Native desktop screen; screen.orientation availability follows engine |
|
||||
| CSS orientation | Viewport | Viewport | Viewport |
|
||||
| pointer / hover media | coarse / none | Native fine / hover on mouse-equipped desktop | Native fine / hover on mouse-equipped desktop |
|
||||
| maxTouchPoints | 5 | Native 0 on desktop | Native 0 on mouse-only desktop |
|
||||
| Input | Native touch from mouse | Native mouse | Per-view GTK left-pointer → native GDK touch, with native compatibility mouse/click events |
|
||||
| Platform / mobile client hints | Linux armv8l / Android | Native desktop platform / engine hints | Native desktop platform / engine hints |
|
||||
|
||||
WKWebView's density selector is optional private SPI and checked before invocation;
|
||||
core viewport/UA behavior uses public native APIs. Reset clears its override to zero.
|
||||
GTK's page zoom changes DPR and viewport together; fixing DPR 2.75 would violate
|
||||
the fitted viewport contract. Pointer/hover media, touch constructors/handlers,
|
||||
maximum contacts and screen metadata retain engine values. Never shim JS getters.
|
||||
GTK mouse-to-touch handlers are view-owned, pass native compatibility events through,
|
||||
and cancel active sequences on profile changes. macOS desktop WebKit keeps mouse input.
|
||||
|
||||
Qualify with the shared `tests/browser-parent/mobile.html` reporting fixture via
|
||||
`mobile-fixture.ts` and `test-browser-parent-webkit.py` on macOS, GTK X11 and Wayland;
|
||||
repeat with `GDK_SCALE=2` and renderer zoom 1.5. `mobile-trial.ts` isolates density,
|
||||
page zoom, layout SPI and GDK touch probes. Run `node --import tsx
|
||||
scripts/compare-webkit-mobile.ts <artifact-directory>` for the production Chromium
|
||||
profile adapter against the same reporting page. The native harness asserts six
|
||||
viewports, HTTP/JS UA, reload/no-op/reset, invalid-profile refusal, trusted input,
|
||||
resize/replacement lifetime and isolation; detailed measurements belong in artifacts.
|
||||
|
||||
Main local and remote desktop windows use a 390 × 600 CSS px content minimum.
|
||||
Native logical minimums scale with application zoom (312 px wide at 80%, 390 at
|
||||
|
|
|
|||
|
|
@ -20,8 +20,8 @@ Native Vorschauen teilen Cookies, Anmeldungen und Websitespeicher über Projekte
|
|||
| --- | --- | --- | --- |
|
||||
| Electron auf unterstützten Desktop-Plattformen | Nativer Chromium-Gast | Für sichtbare zugehörige Sitzungsvorschau unterstützt | Android-/Chromium-Identität, Touch- und Dichteemulation |
|
||||
| Windows-Tauri | Natives WebView2-Unterfenster | Für sichtbare zugehörige Sitzungsvorschau unterstützt | Android-/Chromium-Identität, Touch- und Dichteemulation |
|
||||
| macOS-Tauri | Natives WKWebView-Unterfenster | Nicht verfügbar; verwenden Sie Electron oder Windows-Tauri | Nur Abmessungen |
|
||||
| Linux-Tauri | Natives WebKitGTK-Unterfenster | Nicht verfügbar; verwenden Sie Electron oder Windows-Tauri | Nur Abmessungen |
|
||||
| macOS-Tauri | Natives WKWebView-Unterfenster | Nicht verfügbar; verwenden Sie Electron oder Windows-Tauri | Mobile WebKit-Identität und CSS-Viewport; native Eingabegrenzen |
|
||||
| Linux-Tauri | Natives WebKitGTK-Unterfenster | Nicht verfügbar; verwenden Sie Electron oder Windows-Tauri | Mobile WebKit-Identität und CSS-Viewport; native Dichte-/Eingabegrenzen |
|
||||
| Normaler Browser/Remote-Ansicht | Iframe-Ersatz | Kein lokales natives Vorschauziel | Nur Abmessungen |
|
||||
|
||||
Unter Electron und Windows-Tauri wird native Automatisierung beim Desktop-Start vorbereitet und nutzt das sitzungsgebundene Werkzeug `codenomad.browser` zum Öffnen, Navigieren, Schnappschuss, Screenshot und für Seitenaktionen: Klicken, Tippen, Auswählen, Überfahren, Ziehen, Scrollen, Tastendrücke und Warten auf Text. Werkzeugverfügbarkeit autorisiert nicht jedes Fenster: Ausführung prüft zuständiges Backend, native Fähigkeiten, Sitzung und sichtbares Ziel. Eine Aktion, deren Seite inzwischen verborgen, ersetzt oder weiternavigiert wurde, wird abgelehnt statt an eine andere Seite gesendet, und eine Aktion wird nie automatisch wiederholt. Seiteninhalte bleiben nicht vertrauenswürdig. Ein iframe kann scheitern, weil das Ziel Einbettung verbietet; nutzen Sie einen externen Browser, statt Schutzmaßnahmen zu schwächen.
|
||||
|
|
@ -43,7 +43,7 @@ Tauri-Vorschauen unter macOS/Linux werden manuell geöffnet, behalten gemeinsame
|
|||
| Mobile Ansichtsgröße | 390 × 844 |
|
||||
| Mobiles Querformat | 844 × 390 |
|
||||
|
||||
Unter Electron und Windows-Tauri setzen Mobilvorlagen Android-/Chromium-Identität, Ausrichtung, Touch und DPR 2.75 nur auf dem zugehörigen Gast. Rückkehr zu responsiv/Desktop/Tablet entfernt diese Überschreibungen und lädt neu. Tauri macOS/Linux und der Iframe-Ersatz ändern nur die Größe. Dies sind reproduzierbare Layoutreferenzen, keine Safari-/Firefox-Emulation oder Simulation von Telefontastaturen, Browser-/Systemleisten oder mobilen OS-Texteinstellungen. Sichern Sie Seitenarbeit vor Vorlagenwechsel, da Neuladen ungesendeten Seitenzustand verwerfen kann.
|
||||
Unter Electron und Windows-Tauri setzen Mobilvorlagen Android-/Chromium-Identität, Ausrichtung, Touch und DPR 2.75 nur auf dem zugehörigen Gast. Rückkehr zu responsiv/Desktop/Tablet entfernt diese Überschreibungen und lädt neu. Tauri macOS/Linux verwenden eine generische mobile WebKit-Kennung für HTTP und JavaScript und passen CSS-Viewports unabhängig von App-Zoom/DPI ein; Vorlagenwechsel und Zurücksetzen laden neu. macOS nutzt DPR 2,75, soweit nativ verfügbar; GTK-Dichte folgt Einpassungszoom und Monitor. Bildschirm, Plattform, Zeiger-/Hover-Medien und maximale Touchpunkte behalten native Desktopwerte. Mobile GTK-Klicks erzeugen native Touch- und kompatible Mausereignisse; macOS behält Mauseingabe. Der Iframe-Ersatz ändert nur die Größe. Die Profile dienen als Layoutreferenz; Telefontastaturen, Systemleisten und mobile Texteinstellungen liegen außerhalb ihres Umfangs. Sichern Sie Seitenarbeit vor Vorlagenwechsel, da Neuladen ungesendeten Seitenzustand verwerfen kann.
|
||||
|
||||
## SideCar registrieren {#register-sidecar}
|
||||
|
||||
|
|
|
|||
|
|
@ -20,8 +20,8 @@ Native previews share cookies, sign-ins and site storage across projects/session
|
|||
| --- | --- | --- | --- |
|
||||
| Electron on supported desktop platforms | Native Chromium guest | Supported for the visible owned session preview | Android/Chromium identity, touch and density emulation |
|
||||
| Windows Tauri | Native WebView2 child | Supported for the visible owned session preview | Android/Chromium identity, touch and density emulation |
|
||||
| macOS Tauri | Native WKWebView child | Unavailable; use Electron or Windows Tauri | Dimensions only |
|
||||
| Linux Tauri | Native WebKitGTK child | Unavailable; use Electron or Windows Tauri | Dimensions only |
|
||||
| macOS Tauri | Native WKWebView child | Unavailable; use Electron or Windows Tauri | WebKit mobile identity and CSS viewport; native input limits |
|
||||
| Linux Tauri | Native WebKitGTK child | Unavailable; use Electron or Windows Tauri | WebKit mobile identity and CSS viewport; native density/input limits |
|
||||
| Plain browser/remote view | Iframe fallback | No local native preview target | Dimensions only |
|
||||
|
||||
On Electron and Windows Tauri, native automation is prepared at desktop startup and uses the session-bound `codenomad.browser` tool for open, navigate, snapshot, screenshot and page actions: click, type, select, hover, drag, scroll, key presses and waiting for text. Tool availability is not authorization for every window: execution checks the owning backend, native capabilities, session and visible target. An action whose page was hidden, replaced or navigated meanwhile is refused rather than sent to another page, and an action is never repeated automatically. Page contents remain untrusted. An iframe may fail because the target forbids embedding; use an external browser rather than weakening its protections.
|
||||
|
|
@ -43,7 +43,7 @@ Tauri macOS/Linux previews open manually, retain shared sign-ins across restarts
|
|||
| Mobile viewport | 390 × 844 |
|
||||
| Mobile landscape | 844 × 390 |
|
||||
|
||||
On Electron and Windows Tauri, mobile templates set Android/Chromium identity, orientation, touch and DPR 2.75 on the owned guest only. Returning to responsive/desktop/tablet clears those overrides and reloads. Tauri macOS/Linux and iframe fallback only resize. These are reproducible layout references, not Safari/Firefox emulation or simulations of phone keyboards, browser/system bars or mobile OS text settings. Save page work before changing templates because reload can discard unsent page state.
|
||||
On Electron and Windows Tauri, mobile templates set Android/Chromium identity, orientation, touch and DPR 2.75 on the owned guest only. Returning to responsive/desktop/tablet clears those overrides and reloads. Tauri macOS/Linux apply a generic WebKit mobile identity to HTTP and JavaScript and fit CSS viewports independently of app zoom/DPI, reloading on template/profile changes and reset. macOS uses native DPR 2.75 when available; GTK density follows the fitted zoom and monitor. Screen, platform, pointer/hover media and maximum touch points keep native desktop values. GTK mobile clicks deliver native touch with mouse compatibility events; macOS keeps mouse input. The iframe fallback resizes only. These profiles provide layout references; phone keyboards, system bars and mobile OS text settings stay outside their scope. Save page work before changing templates because reload can discard unsent page state.
|
||||
|
||||
## Register a SideCar {#register-sidecar}
|
||||
|
||||
|
|
|
|||
|
|
@ -20,8 +20,8 @@ Las vistas previas nativas comparten cookies, inicios de sesión y almacenamient
|
|||
| --- | --- | --- | --- |
|
||||
| Electron en plataformas de escritorio compatibles | Navegador invitado Chromium nativo | Compatible para la vista previa visible de la sesión propia | Emulación de identidad Android/Chromium, entrada táctil y densidad |
|
||||
| Tauri para Windows | Ventana hija nativa WebView2 | Compatible para la vista previa visible de la sesión propia | Emulación de identidad Android/Chromium, entrada táctil y densidad |
|
||||
| Tauri para macOS | Ventana hija nativa WKWebView | No disponible; usa Electron o Tauri para Windows | Solo dimensiones |
|
||||
| Tauri para Linux | Ventana hija nativa WebKitGTK | No disponible; usa Electron o Tauri para Windows | Solo dimensiones |
|
||||
| Tauri para macOS | Ventana hija nativa WKWebView | No disponible; usa Electron o Tauri para Windows | Identidad móvil WebKit y viewport CSS; límites nativos de entrada |
|
||||
| Tauri para Linux | Ventana hija nativa WebKitGTK | No disponible; usa Electron o Tauri para Windows | Identidad móvil WebKit y viewport CSS; límites nativos de densidad/entrada |
|
||||
| Navegador normal/vista remota | Alternativa iframe | Sin objetivo local de vista previa nativa | Solo dimensiones |
|
||||
|
||||
En Electron y Tauri para Windows, la automatización nativa se prepara al iniciar el escritorio y usa la herramienta `codenomad.browser` vinculada a la sesión para abrir, navegar, obtener una instantánea, capturar pantalla y realizar acciones en la página: hacer clic, escribir, seleccionar, pasar el cursor, arrastrar, desplazarse, pulsar teclas y esperar un texto. Que una herramienta esté disponible no autoriza todas las ventanas: la ejecución comprueba el backend propietario, las capacidades nativas, la sesión y el objetivo visible. Una acción cuya página se ocultó, se sustituyó o navegó mientras tanto se rechaza en lugar de enviarse a otra página, y una acción nunca se repite automáticamente. El contenido de las páginas sigue sin ser fiable. Un iframe puede fallar porque el destino prohíbe incrustarlo; usa un navegador externo en lugar de debilitar sus protecciones.
|
||||
|
|
@ -43,7 +43,7 @@ Las vistas previas de Tauri para macOS/Linux se abren manualmente, conservan los
|
|||
| Área visible móvil | 390 × 844 |
|
||||
| Móvil horizontal | 844 × 390 |
|
||||
|
||||
En Electron y Tauri para Windows, las plantillas móviles establecen identidad Android/Chromium, orientación, entrada táctil y DPR 2.75 solo en el navegador invitado propio. Volver a adaptable/escritorio/tableta borra esos valores y recarga. Tauri para macOS/Linux y la alternativa iframe solo cambian el tamaño. Son referencias de disposición reproducibles, no emulación Safari/Firefox ni simulaciones de teclados de teléfono, barras del navegador/sistema o ajustes de texto del sistema móvil. Guarda el trabajo de la página antes de cambiar plantillas porque la recarga puede descartar estado sin enviar.
|
||||
En Electron y Tauri para Windows, las plantillas móviles establecen identidad Android/Chromium, orientación, entrada táctil y DPR 2.75 solo en el navegador invitado propio. Volver a adaptable/escritorio/tableta borra esos valores y recarga. Tauri para macOS/Linux aplica una identidad móvil WebKit genérica en HTTP y JavaScript y ajusta el viewport CSS con independencia del zoom/DPI de la aplicación; cambiar o restablecer plantillas recarga. macOS usa DPR nativo 2,75 cuando está disponible; la densidad GTK sigue el zoom de ajuste y el monitor. Pantalla, plataforma, medios de puntero/hover y máximo de contactos conservan valores nativos de escritorio. Los clics móviles GTK generan eventos táctiles nativos y de compatibilidad de ratón; macOS conserva el ratón. La alternativa iframe solo redimensiona. Los perfiles son referencias de disposición; teclados de teléfono, barras del sistema y ajustes de texto móvil quedan fuera de su alcance. Guarda el trabajo de la página antes de cambiar plantillas porque la recarga puede descartar estado sin enviar.
|
||||
|
||||
## Registrar un SideCar {#register-sidecar}
|
||||
|
||||
|
|
|
|||
|
|
@ -20,8 +20,8 @@ Les aperçus natifs partagent cookies, connexions et stockage de site entre proj
|
|||
| --- | --- | --- | --- |
|
||||
| Electron sur les plateformes de bureau prises en charge | Invité Chromium natif | Prise en charge pour l’aperçu de session détenu visible | Émulation d’identité Android/Chromium, tactile et densité |
|
||||
| Tauri Windows | Enfant WebView2 natif | Prise en charge pour l’aperçu de session détenu visible | Émulation d’identité Android/Chromium, tactile et densité |
|
||||
| Tauri macOS | Vue enfant WKWebView native | Indisponible ; utilisez Electron ou Tauri Windows | Dimensions seules |
|
||||
| Tauri Linux | Vue enfant WebKitGTK native | Indisponible ; utilisez Electron ou Tauri Windows | Dimensions seules |
|
||||
| Tauri macOS | Vue enfant WKWebView native | Indisponible ; utilisez Electron ou Tauri Windows | Identité mobile WebKit et viewport CSS ; limites de saisie natives |
|
||||
| Tauri Linux | Vue enfant WebKitGTK native | Indisponible ; utilisez Electron ou Tauri Windows | Identité mobile WebKit et viewport CSS ; limites natives de densité/saisie |
|
||||
| Navigateur ordinaire/vue distante | Iframe de repli | Aucune cible locale d’aperçu natif | Dimensions seules |
|
||||
|
||||
Sur Electron et Tauri Windows, l’automatisation native est préparée au démarrage du bureau et utilise l’outil `codenomad.browser` lié à la session pour ouvrir, naviguer, capturer l’état, prendre une capture d’écran et agir sur la page : cliquer, saisir, sélectionner, survoler, glisser, faire défiler, appuyer sur des touches et attendre un texte. La disponibilité d’un outil n’autorise pas toutes les fenêtres : l’exécution vérifie le backend propriétaire, les capacités natives, la session et la cible visible. Une action dont la page a été masquée, remplacée ou a changé d’adresse entre-temps est refusée plutôt qu’envoyée à une autre page, et une action n’est jamais répétée automatiquement. Le contenu des pages reste non fiable. Une iframe peut échouer si la cible interdit l’intégration ; utilisez un navigateur externe plutôt que d’affaiblir ses protections.
|
||||
|
|
@ -43,7 +43,7 @@ Les aperçus Tauri macOS/Linux s’ouvrent manuellement, conservent les connexio
|
|||
| Zone mobile | 390 × 844 |
|
||||
| Mobile paysage | 844 × 390 |
|
||||
|
||||
Sur Electron et Tauri Windows, les modèles mobiles définissent identité Android/Chromium, orientation, tactile et DPR 2.75 sur l’invité détenu seulement. Revenir à adaptatif/bureau/tablette efface ces remplacements et recharge. Tauri macOS/Linux et l’iframe de repli changent seulement de taille. Ce sont des références de disposition reproductibles, pas une émulation Safari/Firefox ou des simulations de clavier téléphonique, barres navigateur/système ou réglages de texte mobile. Enregistrez le travail de page avant de changer de modèle, car le rechargement peut perdre l’état non envoyé.
|
||||
Sur Electron et Tauri Windows, les modèles mobiles définissent identité Android/Chromium, orientation, tactile et DPR 2.75 sur l’invité détenu seulement. Revenir à adaptatif/bureau/tablette efface ces remplacements et recharge. Tauri macOS/Linux appliquent une identité mobile WebKit générique en HTTP et JavaScript et ajustent les viewports CSS indépendamment du zoom/DPI de l’application, avec rechargement au changement ou à la réinitialisation. macOS utilise un DPR natif de 2,75 si disponible ; la densité GTK suit le zoom d’ajustement et le moniteur. Écran, plateforme, médias pointeur/survol et nombre maximal de contacts conservent leurs valeurs natives de bureau. Les clics mobiles GTK produisent du tactile natif avec événements souris de compatibilité ; macOS conserve la souris. L’iframe de repli redimensionne seulement. Ces profils servent de références de disposition ; claviers téléphoniques, barres système et réglages de texte mobile restent hors périmètre. Enregistrez le travail de page avant de changer de modèle, car le rechargement peut perdre l’état non envoyé.
|
||||
|
||||
## Enregistrer un SideCar {#register-sidecar}
|
||||
|
||||
|
|
|
|||
|
|
@ -20,8 +20,8 @@
|
|||
| --- | --- | --- | --- |
|
||||
| Electron בפלטפורמות שולחן עבודה נתמכות | אורח Chromium מקורי | נתמכת לתצוגת הסשן הגלויה שבבעלות היישום | הדמיית זהות Android/Chromium, מגע וצפיפות |
|
||||
| Tauri ב־Windows | חלון־בת מקורי של WebView2 | נתמכת לתצוגת הסשן הגלויה שבבעלות היישום | הדמיית זהות Android/Chromium, מגע וצפיפות |
|
||||
| Tauri ב־macOS | חלון־בת מקורי של WKWebView | אינה זמינה; השתמשו ב־Electron או ב־Tauri ב־Windows | ממדים בלבד |
|
||||
| Tauri ב־Linux | חלון־בת מקורי של WebKitGTK | אינה זמינה; השתמשו ב־Electron או ב־Tauri ב־Windows | ממדים בלבד |
|
||||
| Tauri ב־macOS | חלון־בת מקורי של WKWebView | אינה זמינה; השתמשו ב־Electron או ב־Tauri ב־Windows | זהות ניידת של WebKit ואזור תצוגת CSS; מגבלות קלט מקוריות |
|
||||
| Tauri ב־Linux | חלון־בת מקורי של WebKitGTK | אינה זמינה; השתמשו ב־Electron או ב־Tauri ב־Windows | זהות ניידת של WebKit ואזור תצוגת CSS; מגבלות צפיפות וקלט מקוריות |
|
||||
| דפדפן רגיל/תצוגה מרוחקת | חלופת iframe | ללא יעד תצוגה מקומית מקורית | ממדים בלבד |
|
||||
|
||||
ב־Electron וב־Tauri ב־Windows, האוטומציה המקורית מוכנה בהפעלת שולחן העבודה ומשתמשת בכלי `codenomad.browser` הקשור לסשן לפתיחה, לניווט, לתמונת מצב, לצילום מסך ולפעולות בדף: לחיצה, הקלדה, בחירה, ריחוף, גרירה, גלילה, הקשות מקשים והמתנה לטקסט. זמינות כלי אינה הרשאה לכל חלון: ההרצה בודקת את שרת הקצה הבעלים, היכולות המקוריות, הסשן והיעד הגלוי. פעולה שהדף שלה הוסתר, הוחלף או נווט בינתיים נדחית במקום להישלח לדף אחר, ופעולה לעולם אינה חוזרת אוטומטית. תוכן דפים נשאר לא מהימן. iframe עלול להיכשל כי היעד אוסר הטמעה; השתמשו בדפדפן חיצוני במקום להחליש את הגנותיו.
|
||||
|
|
@ -43,7 +43,7 @@
|
|||
| אזור תצוגה לנייד | 390 × 844 |
|
||||
| נייד לרוחב | 844 × 390 |
|
||||
|
||||
ב־Electron וב־Tauri ב־Windows, תבניות נייד מגדירות זהות Android/Chromium, כיוון, מגע ו־DPR 2.75 באורח שבבעלות היישום בלבד. חזרה לתצוגה מגיבה/שולחן עבודה/טאבלט מנקה את הדריסות האלה וטוענת מחדש. Tauri ב־macOS/Linux וחלופת iframe רק משנים גודל. אלו נקודות ייחוס לפריסה שניתנות לשחזור, לא הדמיית Safari/Firefox או סימולציות של מקלדות טלפון, סרגלי דפדפן/מערכת או הגדרות טקסט של מערכת הפעלה ניידת. שמרו עבודה בדף לפני החלפת תבניות כי טעינה מחדש עלולה להשליך מצב דף שלא נשלח.
|
||||
ב־Electron וב־Tauri ב־Windows, תבניות נייד מגדירות זהות Android/Chromium, כיוון, מגע ו־DPR 2.75 באורח שבבעלות היישום בלבד. חזרה לתצוגה מגיבה/שולחן עבודה/טאבלט מנקה את הדריסות האלה וטוענת מחדש. Tauri ב־macOS/Linux מחילים זהות ניידת כללית של WebKit ב־HTTP וב־JavaScript ומתאימים את אזור תצוגת CSS בלי תלות בזום/DPI של היישום; שינוי תבנית או איפוס טוענים מחדש. macOS משתמש ב־DPR מקורי 2.75 כאשר הוא זמין; צפיפות GTK תלויה בזום ההתאמה ובמסך. מסך, פלטפורמה, מדיית מצביע/ריחוף ומספר נקודות המגע שומרים ערכי שולחן עבודה מקוריים. לחיצות ניידות ב־GTK מפיקות מגע מקורי ואירועי תאימות עכבר; macOS שומר על קלט עכבר. חלופת iframe משנה גודל בלבד. הפרופילים משמשים נקודות ייחוס לפריסה; מקלדות טלפון, סרגלי מערכת והגדרות טקסט נייד מחוץ לתחומם. שמרו עבודה בדף לפני החלפת תבניות כי טעינה מחדש עלולה להשליך מצב דף שלא נשלח.
|
||||
|
||||
## רישום SideCar {#register-sidecar}
|
||||
|
||||
|
|
|
|||
|
|
@ -20,8 +20,8 @@
|
|||
| --- | --- | --- | --- |
|
||||
| 対応デスクトップ環境の Electron | ネイティブ Chromium ゲスト | 所有する表示中のセッションプレビューで対応 | Android/Chromium の識別、タッチ、密度のエミュレーション |
|
||||
| Windows Tauri | ネイティブ WebView2 子 | 所有する表示中のセッションプレビューで対応 | Android/Chromium の識別、タッチ、密度のエミュレーション |
|
||||
| macOS Tauri | ネイティブ WKWebView 子 | 利用不可。Electron または Windows Tauri を使用 | 寸法のみ |
|
||||
| Linux Tauri | ネイティブ WebKitGTK 子 | 利用不可。Electron または Windows Tauri を使用 | 寸法のみ |
|
||||
| macOS Tauri | ネイティブ WKWebView 子 | 利用不可。Electron または Windows Tauri を使用 | WebKit モバイル識別と CSS ビューポート。入力はネイティブ環境の制約あり |
|
||||
| Linux Tauri | ネイティブ WebKitGTK 子 | 利用不可。Electron または Windows Tauri を使用 | WebKit モバイル識別と CSS ビューポート。密度と入力はネイティブ環境の制約あり |
|
||||
| 通常のブラウザー/リモート表示 | 代替 iframe | ローカルのネイティブプレビュー対象なし | 寸法のみ |
|
||||
|
||||
Electron と Windows Tauri では、ネイティブ自動操作はデスクトップ起動時に準備され、セッションに紐づく `codenomad.browser` ツールで open、navigate、snapshot、screenshot と、ページ操作(クリック、入力、選択、ホバー、ドラッグ、スクロール、キー入力、テキストの出現待ち)を行います。ツールが使えることは、全ウィンドウの操作許可ではありません。実行時に、所有するバックエンド、ネイティブ機能、セッション、表示中の対象を確認します。その間にページが非表示、置き換え、移動された操作は、別のページへ送られずに拒否され、操作が自動的に繰り返されることもありません。ページ内容は引き続き信頼できない情報です。埋め込みを対象が禁止して iframe が失敗する場合は、保護を弱めず外部ブラウザーを使ってください。
|
||||
|
|
@ -43,7 +43,7 @@ macOS/Linux Tauri のプレビューは手動で開き、共有ログインを
|
|||
| モバイルビューポート | 390 × 844 |
|
||||
| モバイル横向き | 844 × 390 |
|
||||
|
||||
Electron と Windows Tauri では、モバイルテンプレートは、所有するゲストにだけ Android/Chromium の識別、向き、タッチ、DPR 2.75 を設定します。レスポンシブ/デスクトップ/タブレットへ戻ると上書きを解除し再読み込みします。macOS/Linux Tauri と代替 iframe はサイズを変えるだけです。再現可能なレイアウトの基準であり、Safari/Firefox のエミュレーションや、携帯のキーボード、ブラウザー/システムバー、モバイル OS の文字設定のシミュレーションではありません。再読み込みで未送信のページ状態を失う場合があるため、テンプレート変更前にページ上の作業を保存してください。
|
||||
Electron と Windows Tauri では、モバイルテンプレートは、所有するゲストにだけ Android/Chromium の識別、向き、タッチ、DPR 2.75 を設定します。レスポンシブ/デスクトップ/タブレットへ戻ると上書きを解除し再読み込みします。macOS/Linux Tauri は HTTP と JavaScript に汎用 WebKit モバイル識別を適用し、アプリのズームや DPI に依存せず CSS ビューポートを合わせ、テンプレートの変更やリセットで再読み込みします。macOS は利用可能ならネイティブ DPR 2.75 を使用し、GTK の密度は表示倍率とモニターに従います。画面、プラットフォーム、ポインター/ホバーのメディア、最大タッチ数はネイティブのデスクトップ値を保持します。GTK のモバイルクリックはネイティブのタッチと互換マウスイベントを生成し、macOS はマウス入力を保持します。代替 iframe はサイズのみ変更します。これらはレイアウトの基準で、携帯のキーボード、システムバー、モバイルの文字設定は対象外です。再読み込みで未送信のページ状態を失う場合があるため、テンプレート変更前にページ上の作業を保存してください。
|
||||
|
||||
## SideCar を登録する {#register-sidecar}
|
||||
|
||||
|
|
|
|||
|
|
@ -20,8 +20,8 @@
|
|||
| --- | --- | --- | --- |
|
||||
| समर्थित desktop प्लेटफर्ममा Electron | नेटिभ Chromium guest | देखिने स्वामित्वभित्रको session पूर्वावलोकनमा समर्थित | Android/Chromium पहिचान, touch र density emulation |
|
||||
| Windows Tauri | नेटिभ WebView2 child | देखिने स्वामित्वभित्रको session पूर्वावलोकनमा समर्थित | Android/Chromium पहिचान, touch र density emulation |
|
||||
| macOS Tauri | नेटिभ WKWebView child | उपलब्ध छैन; Electron वा Windows Tauri प्रयोग गर्नुहोस् | आयाम मात्र |
|
||||
| Linux Tauri | नेटिभ WebKitGTK child | उपलब्ध छैन; Electron वा Windows Tauri प्रयोग गर्नुहोस् | आयाम मात्र |
|
||||
| macOS Tauri | नेटिभ WKWebView child | उपलब्ध छैन; Electron वा Windows Tauri प्रयोग गर्नुहोस् | WebKit मोबाइल पहिचान र CSS भ्युपोर्ट; नेटिभ इनपुट सीमा |
|
||||
| Linux Tauri | नेटिभ WebKitGTK child | उपलब्ध छैन; Electron वा Windows Tauri प्रयोग गर्नुहोस् | WebKit मोबाइल पहिचान र CSS भ्युपोर्ट; नेटिभ घनत्व/इनपुट सीमा |
|
||||
| सामान्य browser/remote दृश्य | iframe fallback | स्थानीय native preview target छैन | आयाम मात्र |
|
||||
|
||||
Electron र Windows Tauri मा नेटिभ automation desktop startup मै तयार हुन्छ र open, navigate, snapshot, screenshot र page action (click, type, select, hover, drag, scroll, key press र text पर्खने) का लागि session-bound `codenomad.browser` tool प्रयोग गर्छ। tool उपलब्धता हरेक विन्डोको अनुमति होइन: execution ले स्वामी backend, native capability, session र visible target जाँच्छ। बीचमै पृष्ठ लुकेको, बदलिएको वा अर्कै ठाउँ नेभिगेट भएको action अर्को पृष्ठमा पठाइँदैन, अस्वीकार गरिन्छ; कुनै action स्वतः दोहोर्याइँदैन। पृष्ठ सामग्री अविश्वसनीय नै हुन्छ। target ले embedding निषेध गर्दा iframe असफल हुन सक्छ; सुरक्षा कमजोर पार्नुको सट्टा बाह्य browser प्रयोग गर्नुहोस्।
|
||||
|
|
@ -43,7 +43,7 @@ macOS/Linux Tauri पूर्वावलोकन म्यानुअल र
|
|||
| Mobile viewport | 390 × 844 |
|
||||
| Mobile landscape | 844 × 390 |
|
||||
|
||||
Electron र Windows Tauri मा mobile template ले स्वामित्वको guest मा मात्र Android/Chromium पहिचान, orientation, touch र DPR 2.75 सेट गर्छन्। responsive/desktop/tablet मा फर्कँदा override हटेर reload हुन्छ। macOS/Linux Tauri र iframe fallback ले आकार मात्र बदल्छन्। यी पुनरुत्पादनयोग्य layout सन्दर्भ हुन्, Safari/Firefox emulation वा phone keyboard, browser/system bar वा mobile OS text setting को simulation होइनन्। reload ले नपठाइएको page state हटाउन सक्ने भएकाले template बदल्नुअघि पृष्ठ काम बचत गर्नुहोस्।
|
||||
Electron र Windows Tauri मा mobile template ले स्वामित्वको guest मा मात्र Android/Chromium पहिचान, orientation, touch र DPR 2.75 सेट गर्छन्। responsive/desktop/tablet मा फर्कँदा override हटेर reload हुन्छ। macOS/Linux Tauri ले HTTP र JavaScript मा सामान्य WebKit मोबाइल पहिचान लागू गर्छन् र एपको zoom/DPI बाट स्वतन्त्र CSS भ्युपोर्ट मिलाउँछन्; template परिवर्तन वा reset गर्दा reload हुन्छ। उपलब्ध हुँदा macOS ले नेटिभ DPR 2.75 प्रयोग गर्छ; GTK घनत्व फिट गर्ने zoom र मनिटरअनुसार हुन्छ। screen, platform, pointer/hover media र अधिकतम touch point मा नेटिभ desktop मान रहन्छन्। GTK मोबाइल click ले नेटिभ touch र mouse compatibility event दिन्छ; macOS मा mouse input रहन्छ। iframe fallback ले आकार मात्र बदल्छ। यी layout सन्दर्भ हुन्; फोन keyboard, system bar र मोबाइल text setting दायराबाहिर छन्। reload ले नपठाइएको page state हटाउन सक्ने भएकाले template बदल्नुअघि पृष्ठ काम बचत गर्नुहोस्।
|
||||
|
||||
## SideCar दर्ता गर्नुहोस् {#register-sidecar}
|
||||
|
||||
|
|
|
|||
|
|
@ -20,8 +20,8 @@
|
|||
| --- | --- | --- | --- |
|
||||
| Electron на поддерживаемых настольных платформах | Нативный гость Chromium | Поддерживается для видимого принадлежащего сессии предпросмотра | Эмуляция идентичности Android/Chromium, касаний и плотности |
|
||||
| Windows Tauri | Нативный дочерний WebView2 | Поддерживается для видимого принадлежащего сессии предпросмотра | Эмуляция идентичности Android/Chromium, касаний и плотности |
|
||||
| macOS Tauri | Нативный дочерний WKWebView | Недоступна; используйте Electron или Windows Tauri | Только размеры |
|
||||
| Linux Tauri | Нативный дочерний WebKitGTK | Недоступна; используйте Electron или Windows Tauri | Только размеры |
|
||||
| macOS Tauri | Нативный дочерний WKWebView | Недоступна; используйте Electron или Windows Tauri | Мобильная идентификация WebKit и CSS-область; нативные ограничения ввода |
|
||||
| Linux Tauri | Нативный дочерний WebKitGTK | Недоступна; используйте Electron или Windows Tauri | Мобильная идентификация WebKit и CSS-область; нативные ограничения плотности/ввода |
|
||||
| Обычный браузер/удалённый вид | Резервный iframe | Нет локальной нативной цели предпросмотра | Только размеры |
|
||||
|
||||
В Electron и Windows Tauri нативная автоматизация подготавливается при запуске настольного приложения и использует связанный с сессией инструмент `codenomad.browser` для открытия, навигации, снимка структуры, снимка экрана и действий на странице: нажатия, ввода, выбора, наведения, перетаскивания, прокрутки, нажатия клавиш и ожидания текста. Доступность инструмента не разрешает доступ к любому окну: выполнение проверяет серверную часть-владельца, нативные возможности, сессию и видимую цель. Действие, страница которого тем временем была скрыта, заменена или перешла на другой адрес, отклоняется, а не отправляется на другую страницу, и действие никогда не повторяется автоматически. Контент страниц остаётся недоверенным. Iframe может не работать из-за запрета встраивания у цели; используйте внешний браузер, не ослабляйте защиту.
|
||||
|
|
@ -43,7 +43,7 @@
|
|||
| Мобильная область | 390 × 844 |
|
||||
| Телефон, альбомная | 844 × 390 |
|
||||
|
||||
В Electron и Windows Tauri мобильные шаблоны задают идентичность Android/Chromium, ориентацию, касания и DPR 2.75 только собственному гостю. Возврат к адаптивному/настольному/планшетному очищает переопределения и перезагружает. Tauri в macOS/Linux и резервный iframe только меняют размер. Это воспроизводимые ориентиры раскладки, не эмуляция Safari/Firefox, клавиатуры телефона, панелей браузера/системы или настроек текста мобильной ОС. Сохраните работу страницы перед сменой шаблона: перезагрузка может отбросить неотправленное состояние.
|
||||
В Electron и Windows Tauri мобильные шаблоны задают идентичность Android/Chromium, ориентацию, касания и DPR 2.75 только собственному гостю. Возврат к адаптивному/настольному/планшетному очищает переопределения и перезагружает. Tauri в macOS/Linux применяет общую мобильную идентификацию WebKit в HTTP и JavaScript и подгоняет CSS-область независимо от масштаба/DPI приложения; смена шаблона и сброс перезагружают страницу. macOS использует нативный DPR 2,75, когда доступен; плотность GTK зависит от масштаба подгонки и монитора. Экран, платформа, медиа указателя/наведения и максимум касаний сохраняют нативные настольные значения. Мобильные клики GTK создают нативные касания и совместимые события мыши; macOS сохраняет ввод мышью. Резервный iframe только меняет размер. Профили служат ориентирами раскладки; клавиатуры телефонов, системные панели и настройки мобильного текста остаются вне их области. Сохраните работу страницы перед сменой шаблона: перезагрузка может отбросить неотправленное состояние.
|
||||
|
||||
## Регистрация SideCar {#register-sidecar}
|
||||
|
||||
|
|
|
|||
|
|
@ -20,8 +20,8 @@ Native önizlemeler aynı masaüstü motoru ve profil/yapılandırma kapsamında
|
|||
| --- | --- | --- | --- |
|
||||
| Desteklenen masaüstü platformlarında Electron | Yerel Chromium konuğu | Görünür sahip olunan oturum önizlemesi için desteklenir | Android/Chromium kimliği, dokunma ve yoğunluk emülasyonu |
|
||||
| Windows Tauri | Yerel WebView2 alt öğesi | Görünür sahip olunan oturum önizlemesi için desteklenir | Android/Chromium kimliği, dokunma ve yoğunluk emülasyonu |
|
||||
| macOS Tauri | Yerel WKWebView alt öğesi | Kullanılamaz; Electron veya Windows Tauri kullanın | Yalnızca boyutlar |
|
||||
| Linux Tauri | Yerel WebKitGTK alt öğesi | Kullanılamaz; Electron veya Windows Tauri kullanın | Yalnızca boyutlar |
|
||||
| macOS Tauri | Yerel WKWebView alt öğesi | Kullanılamaz; Electron veya Windows Tauri kullanın | WebKit mobil kimliği ve CSS görünümü; yerel giriş sınırları |
|
||||
| Linux Tauri | Yerel WebKitGTK alt öğesi | Kullanılamaz; Electron veya Windows Tauri kullanın | WebKit mobil kimliği ve CSS görünümü; yerel yoğunluk/giriş sınırları |
|
||||
| Normal tarayıcı/uzak görünüm | Alternatif iframe görünümü | Yerel native önizleme hedefi yok | Yalnızca boyutlar |
|
||||
|
||||
Electron ve Windows Tauri'de yerel otomasyon, masaüstü başlangıcında hazırlanır ve açma, gezinme, anlık görüntü, ekran görüntüsü ve sayfa eylemleri için oturuma bağlı `codenomad.browser` aracını kullanır: tıklama, yazma, seçme, üzerine gelme, sürükleme, kaydırma, tuşa basma ve metin bekleme. Aracın kullanılabilirliği her pencere için yetkilendirme değildir: yürütme, sahip olunan arka ucu, yerel yetenekleri, oturumu ve görünür hedefi kontrol eder. Sayfası bu arada gizlenmiş, değiştirilmiş veya başka bir adrese gitmiş bir eylem başka bir sayfaya gönderilmek yerine reddedilir ve hiçbir eylem otomatik olarak tekrarlanmaz. Sayfa içerikleri güvenilmez durumda kalıyor. Hedef yerleştirmeyi yasakladığından iframe başarısız olabilir; korumalarını zayıflatmak yerine harici bir tarayıcı kullanın.
|
||||
|
|
@ -43,7 +43,7 @@ Tauri macOS/Linux önizlemeleri elle açılır, paylaşılan oturum açma bilgil
|
|||
| Mobil dikey | 390 × 844 |
|
||||
| Mobil yatay | 844 × 390 |
|
||||
|
||||
Electron ve Windows Tauri'de mobil şablonlar Android/Chromium kimliğini, yönelimini, dokunuşunu ve DPR 2.75'i yalnızca sahip olunan misafir için ayarlar. Duyarlı/masaüstü/tablete geri dönmek, bu geçersiz kılmaları temizler ve sayfayı yeniden yükler. Tauri macOS/Linux ve iframe geri dönüşü yalnızca yeniden boyutlandırılır. Bunlar tekrarlanabilir düzen referanslarıdır; Safari/Firefox emülasyonu veya telefon klavyelerinin, tarayıcı/sistem çubuklarının veya mobil işletim sistemi metin ayarlarının simülasyonları değildir. Yeniden yükleme gönderilmemiş sayfa durumunu atabileceği için şablonları değiştirmeden önce sayfa çalışmasını kaydedin.
|
||||
Electron ve Windows Tauri'de mobil şablonlar Android/Chromium kimliğini, yönelimini, dokunuşunu ve DPR 2.75'i yalnızca sahip olunan misafir için ayarlar. Duyarlı/masaüstü/tablete geri dönmek, bu geçersiz kılmaları temizler ve sayfayı yeniden yükler. Tauri macOS/Linux, HTTP ve JavaScript için genel WebKit mobil kimliğini uygular ve CSS görünümünü uygulama yakınlaştırması/DPI'dan bağımsız ayarlar; şablon değişimi ve sıfırlama sayfayı yeniden yükler. macOS desteklendiğinde yerel DPR 2,75 kullanır; GTK yoğunluğu sığdırma yakınlaştırmasına ve monitöre bağlıdır. Ekran, platform, işaretçi/üzerine gelme medyası ve en fazla dokunma noktası yerel masaüstü değerlerini korur. GTK mobil tıklamaları yerel dokunma ve fare uyumluluk olayları üretir; macOS fare girişini korur. iframe yalnızca boyutlandırılır. Profiller düzen referansıdır; telefon klavyeleri, sistem çubukları ve mobil metin ayarları kapsam dışındadır. Yeniden yükleme gönderilmemiş sayfa durumunu atabileceği için şablonları değiştirmeden önce sayfa çalışmasını kaydedin.
|
||||
|
||||
## SideCar kaydedin {#register-sidecar}
|
||||
|
||||
|
|
|
|||
|
|
@ -20,8 +20,8 @@
|
|||
| --- | --- | --- | --- |
|
||||
| 支持的桌面平台上的 Electron | 原生 Chromium 访客页面 | 支持可见且归属已确认的会话预览 | Android/Chromium 身份、触摸和像素密度仿真 |
|
||||
| Windows Tauri | 原生 WebView2 子窗口 | 支持可见且归属已确认的会话预览 | Android/Chromium 身份、触摸和像素密度仿真 |
|
||||
| macOS Tauri | 原生 WKWebView 子窗口 | 不可用;请使用 Electron 或 Windows Tauri | 仅尺寸 |
|
||||
| Linux Tauri | 原生 WebKitGTK 子窗口 | 不可用;请使用 Electron 或 Windows Tauri | 仅尺寸 |
|
||||
| macOS Tauri | 原生 WKWebView 子窗口 | 不可用;请使用 Electron 或 Windows Tauri | WebKit 移动标识和 CSS 视口;输入受原生限制 |
|
||||
| Linux Tauri | 原生 WebKitGTK 子窗口 | 不可用;请使用 Electron 或 Windows Tauri | WebKit 移动标识和 CSS 视口;密度和输入受原生限制 |
|
||||
| 普通浏览器/远程视图 | iframe 后备实现 | 无本地原生预览目标 | 仅尺寸 |
|
||||
|
||||
在 Electron 和 Windows Tauri 上,原生自动化在桌面启动时准备好,使用会话绑定的 `codenomad.browser` 工具进行打开、导航、快照、截图和页面操作:点击、输入、选择、悬停、拖动、滚动、按键以及等待文本出现。工具可用不代表对每个窗口都已获授权:执行会检查所属后端、原生能力、会话和可见目标。如果操作所针对的页面在此期间已被隐藏、替换或导航离开,该操作会被拒绝,而不会发送到其他页面;操作也绝不会被自动重复。页面内容仍不可信。目标禁止嵌入时 iframe 可能失败;请使用外部浏览器,不要削弱保护。
|
||||
|
|
@ -43,7 +43,7 @@ Tauri macOS/Linux 预览需手动打开,共享登录状态在重启后仍会
|
|||
| 手机视口 | 390 × 844 |
|
||||
| 手机横向 | 844 × 390 |
|
||||
|
||||
在 Electron 和 Windows Tauri 上,移动模板仅对所属访客页面设置 Android/Chromium 身份、方向、触摸和 DPR 2.75。返回响应式/桌面/平板会清除覆盖值并重新加载。Tauri macOS/Linux 和 iframe 后备实现只调整尺寸。这些是可重现的布局参考,而非 Safari/Firefox 仿真,也不模拟手机键盘、浏览器/系统栏或移动操作系统文字设置。切换模板前请保存页面工作,因为重新加载可能丢弃未发送页面状态。
|
||||
在 Electron 和 Windows Tauri 上,移动模板仅对所属访客页面设置 Android/Chromium 身份、方向、触摸和 DPR 2.75。返回响应式/桌面/平板会清除覆盖值并重新加载。Tauri macOS/Linux 在 HTTP 和 JavaScript 中应用通用 WebKit 移动标识,并独立于应用缩放/DPI 调整 CSS 视口;切换模板和重置会重新加载。macOS 在可用时使用原生 DPR 2.75;GTK 密度取决于适配缩放和显示器。屏幕、平台、指针/悬停媒体和最大触点数保留原生桌面值。GTK 移动点击产生原生触摸和兼容鼠标事件;macOS 保留鼠标输入。iframe 后备实现只调整尺寸。这些配置提供布局参考;手机键盘、系统栏和移动文字设置不在其范围内。切换模板前请保存页面工作,因为重新加载可能丢弃未发送页面状态。
|
||||
|
||||
## 注册 SideCar {#register-sidecar}
|
||||
|
||||
|
|
|
|||
8
package-lock.json
generated
8
package-lock.json
generated
|
|
@ -8916,6 +8916,12 @@
|
|||
"node": ">=8"
|
||||
}
|
||||
},
|
||||
"node_modules/dom-accessibility-api": {
|
||||
"version": "0.7.1",
|
||||
"resolved": "https://registry.npmjs.org/dom-accessibility-api/-/dom-accessibility-api-0.7.1.tgz",
|
||||
"integrity": "sha512-vdnCeZD+3wZ+8h8xXL/ZtBlvvoobOFyPzSiIfO6sGOZDqjFx4aLMAjZhl4rawj5xYz3UwP6Tgvyh0iH4IOCVnQ==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/dotenv": {
|
||||
"version": "9.0.2",
|
||||
"resolved": "https://registry.npmjs.org/dotenv/-/dotenv-9.0.2.tgz",
|
||||
|
|
@ -13134,6 +13140,7 @@
|
|||
}
|
||||
],
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"nanoid": "^3.3.16",
|
||||
"picocolors": "^1.1.1",
|
||||
|
|
@ -17435,6 +17442,7 @@
|
|||
"@opencode/client": "2.2.3",
|
||||
"@opencode/schema": "2.2.3",
|
||||
"commander": "^12.1.0",
|
||||
"dom-accessibility-api": "0.7.1",
|
||||
"effect": "4.0.6",
|
||||
"fastify": "^4.28.1",
|
||||
"fuzzysort": "^2.0.4",
|
||||
|
|
|
|||
|
|
@ -55,8 +55,23 @@ function createHarness(requestOpen: (sessionID: string, url: string, requestID:
|
|||
return { controller, owner, add, hide: () => { visible = false } }
|
||||
}
|
||||
|
||||
const page = (controller: BrowserController, method: string, params: object, deadline = Infinity) =>
|
||||
handleNativeBrowserRequest(controller, `browser.page.${method}`, params, deadline)
|
||||
// Existing lifecycle scenarios keep their raw-CDP fixture vocabulary; the
|
||||
// production transport receives only the new closed semantic operations.
|
||||
const page = (controller: BrowserController, method: string, params: any, deadline = Infinity) => {
|
||||
if (method !== "send" || !params.method) return handleNativeBrowserRequest(controller, `browser.page.${method}`, params, deadline)
|
||||
const {method: command, params: data = {}, ...identity} = params
|
||||
const operations: Record<string, unknown> = {
|
||||
"Accessibility.getFullAXTree": {op:"tree"}, "DOM.resolveNode": {op:"resolve",node:42},
|
||||
"DOM.scrollIntoViewIfNeeded": {op:"scroll-into-view",node:42}, "DOM.getContentQuads": {op:"geometry",node:42},
|
||||
"DOM.getNodeForLocation": {op:"hit-test",x:0,y:0}, "Page.getLayoutMetrics": {op:"viewport"},
|
||||
"Runtime.callFunctionOn": {op:"call",function:"focus",handle:"h",args:[]},
|
||||
"Runtime.evaluate": {op:"call",function:"input-acceptance",args:[]},
|
||||
"Runtime.releaseObjectGroup": {op:"release-handles"}, "Input.insertText": {op:"insert-text",text:data && typeof data === "object" && !Array.isArray(data) ? data.text ?? "" : null},
|
||||
"Input.dispatchMouseEvent": {op:data?.type === "mouseWheel" ? "wheel" : "pointer",event:data},
|
||||
"Input.dispatchKeyEvent": {op:"keyboard",event:data},
|
||||
}
|
||||
return handleNativeBrowserRequest(controller, `browser.page.${method}`, {...identity,operation:operations[command] ?? {op:command}}, deadline)
|
||||
}
|
||||
|
||||
async function begin(controller: BrowserController, sessionID = "session", operationID: string = randomUUID()) {
|
||||
const result = await page(controller, "begin", { sessionID, operationID }) as { generation: string; url: string }
|
||||
|
|
@ -186,11 +201,13 @@ describe("BrowserController private page channel", () => {
|
|||
await assert.rejects(page(h.controller, "send", { ...lease, method: "Input.insertText", params }), /parameters/)
|
||||
}
|
||||
assert.deepEqual(calls, ["Accessibility.enable"])
|
||||
for (const method of ["Accessibility.getFullAXTree", "DOM.resolveNode", "DOM.scrollIntoViewIfNeeded", "DOM.getContentQuads",
|
||||
"DOM.getNodeForLocation", "Page.getLayoutMetrics", "Runtime.callFunctionOn", "Runtime.releaseObjectGroup",
|
||||
"Input.insertText", "Input.dispatchMouseEvent"]) {
|
||||
assert.deepEqual(await page(h.controller, "send", { ...lease, method, params: {} }), { ok: true })
|
||||
}
|
||||
const valid = [
|
||||
[{op:"tree"},{nodes:[]}], [{op:"resolve",node:42},{}], [{op:"scroll-into-view",node:42},{ok:true}],
|
||||
[{op:"geometry",node:42},{ok:true}], [{op:"hit-test",x:0,y:0},{}], [{op:"viewport"},{}],
|
||||
[{op:"call",function:"focus",handle:"h",args:[]},{}], [{op:"release-handles"},{ok:true}],
|
||||
[{op:"insert-text",text:""},{}], [{op:"pointer",event:{type:"mouseMoved",x:1,y:1}},{ok:true}],
|
||||
]
|
||||
for (const [operation,expected] of valid) assert.deepEqual(await page(h.controller,"send",{...lease,operation}),expected)
|
||||
assert.deepEqual(await page(h.controller, "send", { ...lease, method: "Input.dispatchKeyEvent", params: { type: "keyUp", key: "a" } }), {})
|
||||
await assert.rejects(page(h.controller, "send", { ...lease, method: "Runtime.evaluate" }), /response is too large/)
|
||||
assert.equal(guest.debugger.isAttached(), false)
|
||||
|
|
@ -244,7 +261,7 @@ describe("BrowserController private page channel", () => {
|
|||
release()
|
||||
await Promise.all([tree, input])
|
||||
assert.deepEqual(counts(), { attachCount: 1, detachCount: 0 })
|
||||
await assert.rejects(page(h.controller, "send", { ...lease, method: "Input.dispatchMouseEvent" }), /no longer current/)
|
||||
await assert.rejects(page(h.controller, "send", { ...lease, operation: {op:"pointer",event:{type:"mouseMoved",x:1,y:1}} }), /no longer current/)
|
||||
assert.deepEqual(counts(), { attachCount: 1, detachCount: 1 })
|
||||
})
|
||||
|
||||
|
|
|
|||
|
|
@ -3,6 +3,9 @@ import { randomUUID } from "node:crypto"
|
|||
import type { BrowserWindow, WebContents } from "electron"
|
||||
import { isBrowserUrlAllowed } from "./browser-webview-security"
|
||||
import { setBrowserEmulation } from "./browser-emulation"
|
||||
import { executeCdpOperation } from "../../../server/src/automation/page/cdp-driver"
|
||||
import { parsePageOperation, pageEnvelope } from "../../../server/src/automation/page/operations"
|
||||
import { PAGE_DRIVER_VERSION } from "../../../server/src/automation/page/protocol"
|
||||
|
||||
const electron = () => createRequire(import.meta.url)("electron") as typeof import("electron")
|
||||
|
||||
|
|
@ -57,12 +60,6 @@ interface PageLease {
|
|||
timer: NodeJS.Timeout
|
||||
}
|
||||
|
||||
const PAGE_METHODS = new Set([
|
||||
"Accessibility.getFullAXTree", "DOM.resolveNode", "DOM.scrollIntoViewIfNeeded", "DOM.getContentQuads",
|
||||
"DOM.getNodeForLocation", "Page.getLayoutMetrics", "Runtime.callFunctionOn", "Runtime.evaluate",
|
||||
"Runtime.releaseObjectGroup", "Input.insertText", "Input.dispatchMouseEvent", "Input.dispatchKeyEvent",
|
||||
])
|
||||
|
||||
export class BrowserController {
|
||||
private readonly registrations = new Map<string, Registration>()
|
||||
private readonly observedGuests = new Map<number, number>()
|
||||
|
|
@ -330,6 +327,7 @@ export class BrowserController {
|
|||
|
||||
// Only handleNativeBrowserRequest (the private backend transport) calls this.
|
||||
async page(method: string, input: Record<string, unknown>, deadline: number): Promise<unknown> {
|
||||
pageEnvelope(method, input)
|
||||
const { sessionID, operationID } = input
|
||||
if (!validId(sessionID, 256) || typeof operationID !== "string"
|
||||
|| !/^[\da-f]{8}-[\da-f]{4}-[\da-f]{4}-[\da-f]{4}-[\da-f]{12}$/i.test(operationID)) {
|
||||
|
|
@ -363,7 +361,7 @@ export class BrowserController {
|
|||
if (!guest.debugger.isAttached()) { guest.debugger.attach("1.3"); lease.ownsDebugger = true }
|
||||
await withDeadline(() => guest.debugger.sendCommand("Accessibility.enable"), Math.min(deadline, expires))
|
||||
const url = this.checkLease(lease, deadline)
|
||||
return { generation: lease.generation, url }
|
||||
return { generation: lease.generation, url, driver: "cdp", contractVersion: PAGE_DRIVER_VERSION }
|
||||
})
|
||||
} catch (error) {
|
||||
this.releaseLease(lease)
|
||||
|
|
@ -379,27 +377,28 @@ export class BrowserController {
|
|||
try { return { url: this.checkLease(lease, deadline) } }
|
||||
catch (error) { this.releaseLease(lease); throw error }
|
||||
}
|
||||
if (typeof input.method !== "string" || !PAGE_METHODS.has(input.method)) throw new Error("Unsupported browser page command")
|
||||
const command = input.method
|
||||
const params = input.params === undefined ? {} : input.params
|
||||
if (!params || typeof params !== "object" || Array.isArray(params) || Buffer.byteLength(JSON.stringify(params)) > 256 * 1024) {
|
||||
throw new Error("Invalid or oversized browser page parameters")
|
||||
}
|
||||
const operation = parsePageOperation(input.operation)
|
||||
return this.enqueue(lease.registration.guest, async () => {
|
||||
try {
|
||||
this.checkLease(lease, deadline)
|
||||
if (command === "Input.dispatchKeyEvent" || command === "Input.dispatchMouseEvent") {
|
||||
await this.awaitCompositor(Math.min(deadline, lease.expires))
|
||||
const result = await executeCdpOperation({ send: async (command, params = {}) => {
|
||||
this.checkLease(lease, deadline)
|
||||
}
|
||||
const result = await withDeadline(
|
||||
if (command === "Input.dispatchKeyEvent" || command === "Input.dispatchMouseEvent") {
|
||||
await this.awaitCompositor(Math.min(deadline, lease.expires))
|
||||
this.checkLease(lease, deadline)
|
||||
}
|
||||
const response = await withDeadline(
|
||||
// Electron's native insertion targets the guest's focused frame even when
|
||||
// CDP Input.insertText is silently ignored by an embedded/background view.
|
||||
() => command === "Input.insertText" && typeof (params as { text?: unknown }).text === "string"
|
||||
? lease.registration.guest.insertText((params as { text: string }).text).then(() => ({}))
|
||||
: command === "Input.dispatchKeyEvent" ? sendGuestKey(lease.registration.guest, params as Record<string, unknown>)
|
||||
: lease.registration.guest.debugger.sendCommand(command, params), Math.min(deadline, lease.expires),
|
||||
)
|
||||
)
|
||||
this.checkLease(lease, deadline)
|
||||
if (Buffer.byteLength(JSON.stringify(response ?? null)) > 4 * 1024 * 1024) throw new Error("Browser page response is too large")
|
||||
return response as Record<string, unknown>
|
||||
} }, operation)
|
||||
this.checkLease(lease, deadline)
|
||||
if (Buffer.byteLength(JSON.stringify(result ?? null)) > 4 * 1024 * 1024) throw new Error("Browser page response is too large")
|
||||
return result
|
||||
|
|
|
|||
|
|
@ -37,6 +37,7 @@
|
|||
"@opencode/client": "2.2.3",
|
||||
"@opencode/schema": "2.2.3",
|
||||
"commander": "^12.1.0",
|
||||
"dom-accessibility-api": "0.7.1",
|
||||
"effect": "4.0.6",
|
||||
"fastify": "^4.28.1",
|
||||
"fuzzysort": "^2.0.4",
|
||||
|
|
|
|||
|
|
@ -1,12 +1,15 @@
|
|||
import assert from "node:assert/strict"
|
||||
import { test } from "node:test"
|
||||
import { BrowserPage } from "./browser-page"
|
||||
import { executeCdpOperation } from "./page/cdp-driver"
|
||||
import type { PageOperation } from "./page/protocol"
|
||||
|
||||
class Native {
|
||||
generation = "run/guest/document-1"
|
||||
hidden = false
|
||||
failMethod?: string
|
||||
calls: Array<{ method: string; input: Record<string, any> }> = []
|
||||
protocol: Array<{method: string; params: Record<string, unknown>}> = []
|
||||
gate?: () => Promise<void>
|
||||
leases = new Map<string, string>()
|
||||
nodes = [{ nodeId: "one", backendDOMNodeId: 42, role: { value: "button" }, name: { value: "Save" } }]
|
||||
|
|
@ -16,15 +19,18 @@ class Native {
|
|||
if (this.hidden) throw new Error("No visible browser")
|
||||
if (method === "browser.page.begin") {
|
||||
this.leases.set(input.operationID, this.generation)
|
||||
return { generation: this.generation, url: "https://example.test/" } as T
|
||||
return { generation: this.generation, url: "https://example.test/", driver: "cdp", contractVersion: 1 } as T
|
||||
}
|
||||
if (input.generation !== this.generation || this.leases.get(input.operationID) !== this.generation) throw new Error("Browser document changed")
|
||||
if (method === "browser.page.check") return { url: "https://example.test/" } as T
|
||||
if (input.method === this.failMethod) throw new Error("Native reply uncertain")
|
||||
if (input.method === "Accessibility.getFullAXTree") { await this.gate?.(); return { nodes: this.nodes } as T }
|
||||
if (input.method === "DOM.resolveNode") return { object: { objectId: "object-1" } } as T
|
||||
if (input.method === "Runtime.callFunctionOn") return { result: { value: null } } as T
|
||||
return {} as T
|
||||
return await executeCdpOperation({ send: async (method, params = {}) => {
|
||||
this.protocol.push({method, params})
|
||||
if (method === this.failMethod) throw new Error("Native reply uncertain")
|
||||
if (method === "Accessibility.getFullAXTree") { await this.gate?.(); return { nodes: this.nodes } }
|
||||
if (method === "DOM.resolveNode") return { object: { objectId: "object-1" } }
|
||||
if (method === "Runtime.callFunctionOn") return { result: { value: null } }
|
||||
return {}
|
||||
} }, input.operation as PageOperation) as T
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -36,11 +42,14 @@ test("preview operations use the shared outline/actions without a CodeNomad inpu
|
|||
assert.match(String(first.snapshot), /- button "Save" \[b1e1\]/)
|
||||
const ref = firstRef(first.snapshot)
|
||||
assert.deepEqual(await page.execute("session", { action: "click", ref }), { url: "https://example.test/" })
|
||||
const functions = native.calls.filter(call => call.input.method === "Runtime.callFunctionOn")
|
||||
const functions = native.protocol.filter(call => call.method === "Runtime.callFunctionOn")
|
||||
assert.ok(functions.length > 0)
|
||||
assert.ok(functions.every(call => !String(call.input.params.functionDeclaration).includes("__codenomadAutomation")),
|
||||
assert.ok(functions.every(call => !String(call.params.functionDeclaration).includes("__codenomadAutomation")),
|
||||
"a user's web page never receives the app's input provenance variables")
|
||||
assert.equal(native.leases.size, 0)
|
||||
assert.ok(native.calls.filter(call => call.method === "browser.page.send").every(call =>
|
||||
call.input.operation && !Object.hasOwn(call.input, "method") && !Object.hasOwn(call.input, "params")
|
||||
&& !JSON.stringify(call.input).includes("functionDeclaration")), "the private wire carries IDs/JSON operations, never program source")
|
||||
const starts = native.calls.filter(call => call.method === "browser.page.begin")
|
||||
assert.equal(starts.length, 2)
|
||||
assert.notEqual(starts[0].input.operationID, starts[1].input.operationID)
|
||||
|
|
@ -97,7 +106,7 @@ test("a failed native mutation is never replayed, and queue admission is bounded
|
|||
const ref = firstRef((await page.execute("a", { action: "snapshot" })).snapshot)
|
||||
native.failMethod = "Runtime.callFunctionOn"
|
||||
await assert.rejects(page.execute("a", { action: "click", ref }), /reply uncertain/)
|
||||
assert.equal(native.calls.filter(call => call.input.method === "Runtime.callFunctionOn").length, 1)
|
||||
assert.equal(native.protocol.filter(call => call.method === "Runtime.callFunctionOn").length, 1)
|
||||
assert.equal(native.leases.size, 0)
|
||||
native.failMethod = undefined
|
||||
let release!: () => void, entered!: () => void
|
||||
|
|
@ -112,3 +121,12 @@ test("a failed native mutation is never replayed, and queue admission is bounded
|
|||
assert.ok((await Promise.all([first, ...pending])).every(result => result instanceof Error))
|
||||
assert.equal(native.leases.size, 0)
|
||||
})
|
||||
|
||||
test("a failed native scroll seals its lease and preserves the original failure before any mutation", async () => {
|
||||
const native = new Native(), page = new BrowserPage(native)
|
||||
const ref = firstRef((await page.execute("a", { action: "snapshot" })).snapshot)
|
||||
native.failMethod = "DOM.scrollIntoViewIfNeeded"
|
||||
await assert.rejects(page.execute("a", { action: "type", ref, text: "unsent", clear: true }), /reply uncertain/)
|
||||
assert.equal(native.protocol.filter(call => call.method === "Runtime.callFunctionOn").length, 0)
|
||||
assert.equal(native.leases.size, 0)
|
||||
})
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@ import type { NativeParent } from "../native-parent"
|
|||
import { bindRefs, performAction, snapshotPage, type PageSession } from "./page/actions"
|
||||
import type { RefAction } from "./page/parse"
|
||||
import { RefTable } from "./page/refs"
|
||||
import { nativeDriver } from "./page/native-driver"
|
||||
|
||||
export type BrowserPageAction = RefAction | { action: "snapshot"; ref?: string; query?: string }
|
||||
const OPERATION_MS = 25_000
|
||||
|
|
@ -40,32 +41,35 @@ export class BrowserPage {
|
|||
|
||||
private async run(sessionID: string, action: BrowserPageAction, signal?: AbortSignal): Promise<Record<string, unknown>> {
|
||||
const operationID = randomUUID(), deadline = Date.now() + OPERATION_MS
|
||||
let failure: Error | undefined
|
||||
const current = () => {
|
||||
if (failure) throw failure
|
||||
signal?.throwIfAborted()
|
||||
if (this.closed) throw new Error("Browser automation is shutting down")
|
||||
if (Date.now() >= deadline) throw new Error("Browser operation timed out; inspect before another action")
|
||||
}
|
||||
const request = async <T>(method: string, params: Record<string, unknown>): Promise<T> => {
|
||||
current()
|
||||
const result = await this.native.request<T>(method, { sessionID, operationID, ...params }, Math.max(1, deadline - Date.now()))
|
||||
const result = await this.native.request<T>(method, { sessionID, operationID, ...params }, Math.max(1, deadline - Date.now())).catch(error => {
|
||||
failure = error instanceof Error ? error : new Error(String(error))
|
||||
throw failure
|
||||
})
|
||||
current()
|
||||
return result
|
||||
}
|
||||
try {
|
||||
const target = await request<{ generation: string; url: string }>("browser.page.begin", {})
|
||||
const target = await request<{ generation: string; url: string; driver: unknown; contractVersion: unknown }>("browser.page.begin", {})
|
||||
if (!target || typeof target.generation !== "string" || !target.generation || target.generation.length > 512
|
||||
|| typeof target.url !== "string") throw new Error("Invalid native browser target")
|
||||
const generation = target.generation
|
||||
let url = target.url
|
||||
const page: PageSession = {
|
||||
markAutomation: false,
|
||||
channel: { send: (method, params) => request("browser.page.send", { generation, method, params: params ?? {} }) },
|
||||
checkpoint: async () => {
|
||||
const checkpoint = async () => {
|
||||
const result = await request<{ url: string }>("browser.page.check", { generation })
|
||||
if (!result || typeof result.url !== "string") throw new Error("Invalid native browser target")
|
||||
url = result.url
|
||||
},
|
||||
}
|
||||
const driver = nativeDriver(target, operation => request("browser.page.send", { generation, operation }), checkpoint)
|
||||
const page: PageSession = { driver, markAutomation: false, checkpoint }
|
||||
const refs = this.table(sessionID)
|
||||
if (action.action === "snapshot") {
|
||||
const rootBackendNodeId = action.ref ? refs.resolve(action.ref, generation) : undefined
|
||||
|
|
|
|||
|
|
@ -1,12 +1,11 @@
|
|||
import { aim, clickAt, dragBetween, hoverAt, pressKeys, waitForText, wheelAt, type MouseButton } from "./input"
|
||||
import { callOn, markedInput, OBJECT_GROUP, releaseObjects, resolveElement, VALUE_LIMIT, type PageSession } from "./protocol"
|
||||
import { callOn, markedInput, releaseObjects, resolveElement, PageDriverUnsupportedError, type PageSession } from "./protocol"
|
||||
import { snapshotTree } from "./semantic-tree"
|
||||
import type { RefAction } from "./parse"
|
||||
import { renderSnapshot, type AxNode, type Snapshot, type SnapshotOptions } from "./snapshot"
|
||||
|
||||
export type { PageChannel, PageSession } from "./protocol"
|
||||
|
||||
interface Target { ref: string; backendNodeId: number }
|
||||
|
||||
export type PageAction =
|
||||
| Target & { kind: "click"; button?: MouseButton }
|
||||
| Target & { kind: "type"; text: string; clear: boolean }
|
||||
|
|
@ -16,20 +15,8 @@ export type PageAction =
|
|||
| { kind: "scroll"; target?: Target; direction: "up" | "down" | "left" | "right"; amount: number }
|
||||
| { kind: "press"; key: string; target?: Target }
|
||||
| { kind: "wait"; text: string; gone: boolean; timeoutMs: number }
|
||||
export interface PageActionResult { value?: string; selected?: string[]; focused?: string; found?: boolean; waitedMs?: number }
|
||||
|
||||
export interface PageActionResult {
|
||||
/** The field's value after typing; passwords are masked. */
|
||||
value?: string
|
||||
/** The select's chosen option labels. */
|
||||
selected?: string[]
|
||||
/** The element focused after a key press. */
|
||||
focused?: string
|
||||
/** Whether wait found its text, and after how long. */
|
||||
found?: boolean
|
||||
waitedMs?: number
|
||||
}
|
||||
|
||||
/** Resolves a tool's refs to DOM nodes before any page access; a stale ref fails the whole action. */
|
||||
export function bindRefs(action: RefAction, resolve: (ref: string) => number): PageAction {
|
||||
const target = (ref: string): Target => ({ ref, backendNodeId: resolve(ref) })
|
||||
switch (action.action) {
|
||||
|
|
@ -43,20 +30,15 @@ export function bindRefs(action: RefAction, resolve: (ref: string) => number): P
|
|||
case "wait": return { kind: "wait", text: action.text, gone: action.gone, timeoutMs: action.timeout }
|
||||
}
|
||||
}
|
||||
|
||||
export async function readAccessibilityTree(session: PageSession): Promise<AxNode[]> {
|
||||
const result = await session.channel.send("Accessibility.getFullAXTree")
|
||||
const nodes = await session.driver.tree()
|
||||
await session.checkpoint()
|
||||
return Array.isArray(result.nodes) ? result.nodes as AxNode[] : []
|
||||
return snapshotTree(nodes)
|
||||
}
|
||||
|
||||
export async function snapshotPage(session: PageSession, issue: () => (backendNodeId: number) => string,
|
||||
options: SnapshotOptions = {}): Promise<Snapshot> {
|
||||
export async function snapshotPage(session: PageSession, issue: () => (backendNodeId: number) => string, options: SnapshotOptions = {}): Promise<Snapshot> {
|
||||
const nodes = await readAccessibilityTree(session)
|
||||
// Refs are issued only after the tree was confirmed current.
|
||||
return renderSnapshot(nodes, issue(), options)
|
||||
}
|
||||
|
||||
export async function performAction(session: PageSession, action: PageAction): Promise<PageActionResult> {
|
||||
try {
|
||||
switch (action.kind) {
|
||||
|
|
@ -64,7 +46,7 @@ export async function performAction(session: PageSession, action: PageAction): P
|
|||
case "type": return await type(session, action)
|
||||
case "select": {
|
||||
const element = await resolveElement(session, action.ref, action.backendNodeId)
|
||||
return { selected: (await callOn(session, action.ref, element, SELECT, [action.values]) as unknown[]).map(String) }
|
||||
return { selected: (await callOn(session, action.ref, element, "select", [action.values]) as unknown[]).map(String) }
|
||||
}
|
||||
case "hover": {
|
||||
const point = await aim(session, action.ref, action.backendNodeId)
|
||||
|
|
@ -73,8 +55,6 @@ export async function performAction(session: PageSession, action: PageAction): P
|
|||
return {}
|
||||
}
|
||||
case "drag": {
|
||||
// Both ends must already be visible: scrolling to the second target would
|
||||
// invalidate the first point and could press a different row underneath it.
|
||||
const from = await aim(session, action.ref, action.backendNodeId, { scroll: false })
|
||||
const to = await aim(session, action.to.ref, action.to.backendNodeId, { scroll: false })
|
||||
if (!from || !to) throw new Error(`${!from ? action.ref : action.to.ref} has no visible box to drag ${!from ? "from" : "to"}`)
|
||||
|
|
@ -92,167 +72,41 @@ export async function performAction(session: PageSession, action: PageAction): P
|
|||
case "press": {
|
||||
if (action.target) {
|
||||
const element = await resolveElement(session, action.target.ref, action.target.backendNodeId)
|
||||
await callOn(session, action.target.ref, element, FOCUS, [])
|
||||
await callOn(session, action.target.ref, element, "focus", [])
|
||||
}
|
||||
await pressKeys(session, action.key)
|
||||
return { focused: await focusedElement(session) }
|
||||
}
|
||||
case "wait": return await waitForText(session, action.text, action.gone, action.timeoutMs)
|
||||
}
|
||||
} finally {
|
||||
await releaseObjects(session)
|
||||
}
|
||||
} finally { await releaseObjects(session) }
|
||||
}
|
||||
|
||||
async function click(session: PageSession, action: Target & { button?: MouseButton }): Promise<PageActionResult> {
|
||||
const element = await resolveElement(session, action.ref, action.backendNodeId)
|
||||
const selected = await callOn(session, action.ref, element, OPTION_CLICK, [])
|
||||
const selected = await callOn(session, action.ref, element, "option-click", [])
|
||||
if (Array.isArray(selected)) return { selected: selected.map(String) }
|
||||
const point = await aim(session, action.ref, action.backendNodeId)
|
||||
// An element without a visible box (zero-size or offscreen-only) is activated programmatically.
|
||||
if (!point) {
|
||||
if (action.button && action.button !== "left") throw new Error(`${action.ref} has no visible box to ${action.button}-click`)
|
||||
await callOn(session, action.ref, element, PROGRAMMATIC_CLICK, [])
|
||||
await callOn(session, action.ref, element, "programmatic-click", [])
|
||||
} else await clickAt(session, point, action.button)
|
||||
return {}
|
||||
}
|
||||
|
||||
async function type(session: PageSession, action: Target & { text: string; clear: boolean }): Promise<PageActionResult> {
|
||||
const element = await resolveElement(session, action.ref, action.backendNodeId)
|
||||
await session.channel.send("DOM.scrollIntoViewIfNeeded", { backendNodeId: action.backendNodeId }).catch(() => undefined)
|
||||
await session.driver.scrollIntoView(action.backendNodeId).catch(error => { if (error instanceof PageDriverUnsupportedError) throw error })
|
||||
await session.checkpoint()
|
||||
const field = await callOn(session, action.ref, element, FIELD, [], false) as string
|
||||
const mode = await callOn(session, action.ref, field, PREPARE, [action.clear])
|
||||
if (mode === "value" || mode === "append") await callOn(session, action.ref, field, SET_VALUE, [action.text, mode === "append"])
|
||||
else if (action.text) await markedInput(session, dispatch => dispatch("Input.insertText", { text: action.text }))
|
||||
else if (action.clear) await callOn(session, action.ref, field, DELETE_SELECTION, [])
|
||||
return { value: String(await callOn(session, action.ref, field, READ_VALUE, [])) }
|
||||
const field = await callOn(session, action.ref, element, "field", [], false) as string
|
||||
const mode = await callOn(session, action.ref, field, "prepare", [action.clear])
|
||||
if (mode === "value" || mode === "append") await callOn(session, action.ref, field, "set-value", [action.text, mode === "append"])
|
||||
else if (action.text) await markedInput(session, dispatch => dispatch("insert-text", { text: action.text }))
|
||||
else if (action.clear) await callOn(session, action.ref, field, "delete-selection", [])
|
||||
return { value: String(await callOn(session, action.ref, field, "read-value", [])) }
|
||||
}
|
||||
|
||||
async function focusedElement(session: PageSession): Promise<string | undefined> {
|
||||
const active = await session.channel.send("Runtime.evaluate", { expression: "document.activeElement", objectGroup: OBJECT_GROUP })
|
||||
const active = await session.driver.call("active-element", undefined, [])
|
||||
await session.checkpoint()
|
||||
const objectId = (active.result as { objectId?: unknown } | undefined)?.objectId
|
||||
if (typeof objectId !== "string") return undefined
|
||||
const described = await callOn(session, "the focused element", objectId, DESCRIBE, [])
|
||||
if (typeof active.handle !== "string") return undefined
|
||||
const described = await callOn(session, "the focused element", active.handle, "describe", [])
|
||||
return typeof described === "string" ? described : undefined
|
||||
}
|
||||
|
||||
const LABEL = `const label = option => (option.label || option.textContent || "").replace(/\\s+/g, " ").trim();`
|
||||
|
||||
/** Picks options like a user would: by visible label, else by value; change fires only on a change.
|
||||
* `:disabled` also covers disabled fieldsets and optgroups. */
|
||||
const CHOOSE = `${LABEL}
|
||||
const apply = (select, picked) => {
|
||||
if (select.matches(":disabled")) throw new Error("codenomad:disabled");
|
||||
for (const option of picked) if (option.matches(":disabled")) throw new Error("codenomad:option-disabled:" + label(option));
|
||||
if (!select.multiple && picked.length !== 1) throw new Error("codenomad:single");
|
||||
select.focus();
|
||||
const options = Array.from(select.options);
|
||||
if (options.some(option => option.selected !== picked.includes(option))) {
|
||||
for (const option of options) option.selected = picked.includes(option);
|
||||
select.dispatchEvent(new Event("input", { bubbles: true, composed: true }));
|
||||
select.dispatchEvent(new Event("change", { bubbles: true }));
|
||||
}
|
||||
return Array.from(select.selectedOptions).map(label);
|
||||
};
|
||||
const choose = (select, wanted) => {
|
||||
if (select.matches(":disabled")) throw new Error("codenomad:disabled");
|
||||
const options = Array.from(select.options), picked = [];
|
||||
for (const raw of wanted) {
|
||||
const target = String(raw).replace(/\\s+/g, " ").trim();
|
||||
const option = options.find(item => label(item) === target) || options.find(item => item.value === String(raw));
|
||||
if (!option) throw new Error("codenomad:no-option:" + target);
|
||||
if (!picked.includes(option)) picked.push(option);
|
||||
}
|
||||
return apply(select, picked);
|
||||
};`
|
||||
|
||||
/** A native select's popup is not part of the page: clicking an option picks that exact option,
|
||||
* and the select itself is changed with `select` rather than opened. */
|
||||
const OPTION_CLICK = `${CHOOSE}
|
||||
const option = element.closest("option"), select = option && option.closest("select");
|
||||
if (option && select) return apply(select, [option]);
|
||||
if (element instanceof HTMLSelectElement) throw new Error("codenomad:is-select");
|
||||
return null;`
|
||||
|
||||
const PROGRAMMATIC_CLICK = `if (element instanceof HTMLElement) element.click();
|
||||
else element.dispatchEvent(new MouseEvent("click", { bubbles: true, cancelable: true, composed: true }));
|
||||
return null;`
|
||||
|
||||
const SELECT = `${CHOOSE}
|
||||
const control = element instanceof HTMLLabelElement ? element.control : element;
|
||||
const select = control instanceof HTMLSelectElement ? control : control && control.closest("select");
|
||||
if (!select) throw new Error("codenomad:not-select");
|
||||
return choose(select, args[0]);`
|
||||
|
||||
const FOCUS = `if (element.matches(":disabled")) throw new Error("codenomad:disabled");
|
||||
if (element instanceof HTMLElement || element instanceof SVGElement) element.focus();
|
||||
if (element.getRootNode().activeElement !== element) throw new Error("codenomad:unfocusable");
|
||||
return null;`
|
||||
|
||||
const DESCRIBE = `const role = element.getAttribute("role") || element.tagName.toLowerCase();
|
||||
const name = (element.getAttribute("aria-label") || element.getAttribute("title") || element.getAttribute("placeholder")
|
||||
|| (element.labels && element.labels[0] && element.labels[0].textContent) || element.textContent || "").replace(/\\s+/g, " ").trim().slice(0, 80);
|
||||
return role + (name ? " " + JSON.stringify(name) : "");`
|
||||
|
||||
const NOT_TEXT = ["checkbox", "radio", "button", "submit", "reset", "file", "image", "hidden"]
|
||||
const VALUE_TYPES = ["date", "time", "datetime-local", "month", "week", "range", "color"]
|
||||
/** Types whose value sanitization empties an invalid value. */
|
||||
const PROBED_TYPES = ["date", "time", "datetime-local", "month", "week", "number"]
|
||||
const TEXT_FIELD = `const textField = item => item instanceof HTMLInputElement || item instanceof HTMLTextAreaElement
|
||||
|| (item instanceof HTMLElement && item.isContentEditable);`
|
||||
|
||||
/** The field a ref designates: itself, its labelled control, or its first editable descendant. */
|
||||
const FIELD = `${TEXT_FIELD}
|
||||
const control = element instanceof HTMLLabelElement ? element.control : element;
|
||||
const field = control && textField(control) ? control
|
||||
: element.querySelector('input:not([type="hidden"]), textarea, [contenteditable]:not([contenteditable="false"])');
|
||||
if (!field || !textField(field)) throw new Error("codenomad:not-editable");
|
||||
return field;`
|
||||
|
||||
/** Native text insertion goes to the focused element, so a field that cannot take focus
|
||||
* (hidden, disabled, behind a modal dialog) fails here instead of typing into another one. */
|
||||
const PREPARE = `const field = element, clear = args[0];
|
||||
if (field.matches(":disabled")) throw new Error("codenomad:disabled");
|
||||
if (field.readOnly) throw new Error("codenomad:readonly");
|
||||
if (field instanceof HTMLInputElement && ${JSON.stringify(NOT_TEXT)}.includes(field.type)) throw new Error("codenomad:not-text");
|
||||
field.focus();
|
||||
const active = field.getRootNode().activeElement;
|
||||
const editable = !(field instanceof HTMLInputElement || field instanceof HTMLTextAreaElement);
|
||||
if (active !== field && !(editable && active instanceof HTMLElement && active.isContentEditable && active.contains(field))) {
|
||||
throw new Error("codenomad:unfocusable");
|
||||
}
|
||||
if (field instanceof HTMLInputElement && ${JSON.stringify(VALUE_TYPES)}.includes(field.type)) return "value";
|
||||
if (editable) {
|
||||
const range = document.createRange();
|
||||
range.selectNodeContents(field);
|
||||
if (!clear) range.collapse(false);
|
||||
const selection = getSelection();
|
||||
selection.removeAllRanges();
|
||||
selection.addRange(range);
|
||||
} else if (clear) field.select();
|
||||
// Email and number inputs have no caret API: their text is appended as a value.
|
||||
else { try { field.setSelectionRange(field.value.length, field.value.length); } catch { return "append"; } }
|
||||
return "insert";`
|
||||
|
||||
/** Sets an input's value like a picker would. An invalid value is refused before anything changes;
|
||||
* a valid one keeps the browser's normalization (lowercase colors, snapped ranges). */
|
||||
const SET_VALUE = `const [text, append] = args, next = append ? element.value + text : text;
|
||||
// Ranges and colors have no empty value: the browser would substitute its default.
|
||||
const valid = element.type === "color" ? /^#[0-9a-f]{6}$/i.test(next)
|
||||
: element.type === "range" ? /^-?(?:\\d+(?:\\.\\d+)?|\\.\\d+)(?:[eE][+-]?\\d+)?$/.test(next)
|
||||
: next !== "" && ${JSON.stringify(PROBED_TYPES)}.includes(element.type) ? (() => { const probe = document.createElement("input"); probe.type = element.type; probe.value = next; return probe.value !== ""; })()
|
||||
: true;
|
||||
if (!valid) throw new Error("codenomad:invalid-value:" + next);
|
||||
if (next === element.value) return null;
|
||||
Object.getOwnPropertyDescriptor(HTMLInputElement.prototype, "value").set.call(element, next);
|
||||
element.dispatchEvent(new Event("input", { bubbles: true, composed: true }));
|
||||
element.dispatchEvent(new Event("change", { bubbles: true }));
|
||||
return null;`
|
||||
|
||||
const DELETE_SELECTION = `document.execCommand("delete"); return null;`
|
||||
|
||||
const READ_VALUE = `const value = element instanceof HTMLInputElement || element instanceof HTMLTextAreaElement ? element.value : element.innerText || element.textContent || "";
|
||||
if (element instanceof HTMLInputElement && element.type === "password") return "•".repeat(Math.min(value.length, 16));
|
||||
return value.length > ${VALUE_LIMIT} ? value.slice(0, ${VALUE_LIMIT - 1}) + "…" : value;`
|
||||
|
|
|
|||
49
packages/server/src/automation/page/cdp-driver.ts
Normal file
49
packages/server/src/automation/page/cdp-driver.ts
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
import { declaration, ELEMENT_FUNCTIONS, GLOBAL_FUNCTIONS, handleResult } from "./functions"
|
||||
import { OBJECT_GROUP, type PageChannel, type PageOperation } from "./protocol"
|
||||
import { semanticTree } from "./semantic-tree"
|
||||
import type { AxNode } from "./snapshot"
|
||||
import { operationDriver } from "./driver"
|
||||
import { parsePageOperation } from "./operations"
|
||||
const optional = (key: string, value: unknown): Record<string, unknown> => value === undefined ? {} : { [key]: value }
|
||||
|
||||
/** Used in the app's CDP connection and Electron's native guest dispatcher. */
|
||||
export async function executeCdpOperation(channel: PageChannel, raw: PageOperation, mark = false): Promise<Record<string, unknown>> {
|
||||
const operation = parsePageOperation(raw, mark)
|
||||
switch (operation.op) {
|
||||
case "tree": { const result = await channel.send("Accessibility.getFullAXTree"); return { nodes: semanticTree(Array.isArray(result.nodes) ? result.nodes as AxNode[] : []) } }
|
||||
case "resolve": { const result = await channel.send("DOM.resolveNode", { backendNodeId: operation.node, objectGroup: OBJECT_GROUP }); return optional("handle",(result.object as {objectId?: string})?.objectId) }
|
||||
case "call": {
|
||||
const id = operation.function
|
||||
const result = Object.hasOwn(ELEMENT_FUNCTIONS, id)
|
||||
? await channel.send("Runtime.callFunctionOn", { objectId: operation.handle,
|
||||
functionDeclaration: declaration(id as keyof typeof ELEMENT_FUNCTIONS, mark),
|
||||
arguments: operation.args.map(value => value && typeof value === "object" && Object.hasOwn(value, "handle") ? { objectId: (value as {handle: string}).handle } : { value }),
|
||||
returnByValue: !handleResult(id), objectGroup: OBJECT_GROUP })
|
||||
: await channel.send("Runtime.evaluate", { expression: GLOBAL_FUNCTIONS[id as keyof typeof GLOBAL_FUNCTIONS],
|
||||
...(handleResult(id) ? { objectGroup: OBJECT_GROUP } : { returnByValue: true }) })
|
||||
const failure = result.exceptionDetails as {exception?: {description?: unknown}; text?: unknown} | undefined
|
||||
if (failure) return { exception: String(failure.exception?.description ?? failure.text ?? "") }
|
||||
const value = result.result as {value?: unknown; objectId?: string} | undefined
|
||||
return handleResult(id) ? optional("handle",value?.objectId) : optional("value",value?.value)
|
||||
}
|
||||
case "scroll-into-view": return channel.send("DOM.scrollIntoViewIfNeeded", { backendNodeId: operation.node })
|
||||
case "geometry": return channel.send("DOM.getContentQuads", { backendNodeId: operation.node })
|
||||
case "viewport": {
|
||||
const metrics = await channel.send("Page.getLayoutMetrics")
|
||||
const viewport = (metrics.cssLayoutViewport ?? metrics.layoutViewport) as {clientWidth?: number; clientHeight?: number} | undefined
|
||||
return { ...optional("width",viewport?.clientWidth), ...optional("height",viewport?.clientHeight) }
|
||||
}
|
||||
case "hit-test": { const result = await channel.send("DOM.getNodeForLocation", { x: operation.x, y: operation.y, includeUserAgentShadowDOM: false, ignorePointerEventsNone: false }); return optional("node",result.backendNodeId) }
|
||||
case "pointer": case "wheel": return channel.send("Input.dispatchMouseEvent", operation.event as unknown as Record<string, unknown>)
|
||||
case "keyboard": return channel.send("Input.dispatchKeyEvent", operation.event as unknown as Record<string, unknown>)
|
||||
case "insert-text": return channel.send("Input.insertText", { text: operation.text })
|
||||
case "release-handles": return channel.send("Runtime.releaseObjectGroup", { objectGroup: OBJECT_GROUP })
|
||||
case "checkpoint": return {}
|
||||
}
|
||||
}
|
||||
export function cdpDriver(channel: PageChannel, checkpoint: () => Promise<void>, mark = false) {
|
||||
return operationDriver("cdp", operation => executeCdpOperation(channel, operation, mark), checkpoint, mark)
|
||||
}
|
||||
export function cdpSession(channel: PageChannel, checkpoint: () => Promise<void>, markAutomation: boolean) {
|
||||
return { driver: cdpDriver(channel, checkpoint, markAutomation), checkpoint, markAutomation }
|
||||
}
|
||||
40
packages/server/src/automation/page/contract.cases.json
Normal file
40
packages/server/src/automation/page/contract.cases.json
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
{
|
||||
"valid": [
|
||||
{"op":"tree"}, {"op":"viewport"}, {"op":"checkpoint"}, {"op":"release-handles"},
|
||||
{"op":"resolve","node":42}, {"op":"geometry","node":42}, {"op":"scroll-into-view","node":42},
|
||||
{"op":"hit-test","x":1,"y":2},
|
||||
{"op":"call","function":"select","handle":"h","args":[["é漢🙂"]]},
|
||||
{"op":"call","function":"field","handle":"h","args":[]},
|
||||
{"op":"call","function":"set-value","handle":"h","args":["",false]},
|
||||
{"op":"call","function":"receives","handle":"h","args":[{"handle":"other"}]},
|
||||
{"op":"call","function":"document-root","args":[]},
|
||||
{"op":"call","function":"active-element","args":[]},
|
||||
{"op":"call","function":"input-acceptance","args":[]},
|
||||
{"op":"pointer","event":{"type":"mousePressed","x":4,"y":5,"button":"left","buttons":1,"clickCount":1}},
|
||||
{"op":"wheel","event":{"type":"mouseWheel","x":4,"y":5,"deltaX":0,"deltaY":100}},
|
||||
{"op":"keyboard","event":{"type":"keyDown","key":" ","code":"Space","text":" ","windowsVirtualKeyCode":32,"modifiers":0}},
|
||||
{"op":"keyboard","event":{"type":"keyUp","key":"Control","buttons":0,"modifiers":0}},
|
||||
{"op":"insert-text","text":""}
|
||||
],
|
||||
"invalid": [
|
||||
null, [], "tree", {}, {"op":"evaluate","source":"alert(1)"},
|
||||
{"op":"tree","extra":true}, {"op":"viewport","extra":true}, {"op":"checkpoint","extra":true}, {"op":"release-handles","extra":true},
|
||||
{"op":"resolve","node":0}, {"op":"resolve","node":1.5}, {"op":"resolve","node":1,"extra":true},
|
||||
{"op":"call","function":"mark-input","args":[]}, {"op":"call","function":"unmark-input","args":[]},
|
||||
{"op":"call","function":"__proto__","args":[]}, {"op":"call","function":"constructor","args":[]},
|
||||
{"op":"call","function":"select","handle":"h","args":["label"]},
|
||||
{"op":"call","function":"focus","args":[]}, {"op":"call","function":"focus","handle":null,"args":[]},
|
||||
{"op":"call","function":"document-root","handle":null,"args":[]},
|
||||
{"op":"call","function":"focus","handle":"h","args":[],"source":"evil"},
|
||||
{"op":"call","function":"receives","handle":"h","args":[{"handle":"other","value":"evil"}]},
|
||||
{"op":"pointer","event":{"type":["mouseMoved"],"x":0,"y":0}},
|
||||
{"op":"pointer","event":{"type":"mouseMoved","x":0,"y":0,"button":["left"]}},
|
||||
{"op":"pointer","event":{"type":"mouseMoved","x":0,"y":0,"unknown":true}},
|
||||
{"op":"pointer","event":{"type":"mouseWheel","x":0,"y":0}},
|
||||
{"op":"pointer","event":{"type":"mousePressed","x":0,"y":0,"modifiers":16}},
|
||||
{"op":"keyboard","event":{"type":"keyDown","key":null}},
|
||||
{"op":"keyboard","event":{"type":"char","key":"a"}},
|
||||
{"op":"keyboard","event":{"type":"keyUp","modifiers":null}},
|
||||
{"op":"insert-text","text":null}, {"op":"insert-text","text":"ok","script":"evil"}
|
||||
]
|
||||
}
|
||||
12
packages/server/src/automation/page/driver.fixture.json
Normal file
12
packages/server/src/automation/page/driver.fixture.json
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
[
|
||||
{"nodeId":"root","role":{"value":"RootWebArea"},"childIds":["heading","button","field","check","select","group"]},
|
||||
{"nodeId":"heading","parentId":"root","backendDOMNodeId":10,"role":{"value":"heading"},"name":{"value":"Checkout"},"properties":[{"name":"level","value":{"value":2}}]},
|
||||
{"nodeId":"button","parentId":"root","backendDOMNodeId":11,"role":{"value":"button"},"name":{"value":"Save"},"childIds":["text"]},
|
||||
{"nodeId":"text","parentId":"button","role":{"value":"StaticText"},"name":{"value":"Save"}},
|
||||
{"nodeId":"field","parentId":"root","backendDOMNodeId":12,"role":{"value":"textbox"},"name":{"value":"Name"},"value":{"value":"é漢🙂"},"properties":[{"name":"focused","value":{"value":true}}]},
|
||||
{"nodeId":"check","parentId":"root","backendDOMNodeId":13,"role":{"value":"checkbox"},"name":{"value":"Remember"},"properties":[{"name":"checked","value":{"value":"mixed"}}]},
|
||||
{"nodeId":"select","parentId":"root","backendDOMNodeId":14,"role":{"value":"combobox"},"name":{"value":"Plan"},"childIds":["option"]},
|
||||
{"nodeId":"option","parentId":"select","backendDOMNodeId":15,"role":{"value":"option"},"name":{"value":"Pro"},"properties":[{"name":"selected","value":{"value":true}},{"name":"disabled","value":{"value":true}}]},
|
||||
{"nodeId":"group","parentId":"root","ignored":true,"childIds":["quantity"]},
|
||||
{"nodeId":"quantity","parentId":"group","backendDOMNodeId":16,"role":{"value":"spinbutton"},"name":{"value":"Quantity"},"value":{"value":3}}
|
||||
]
|
||||
85
packages/server/src/automation/page/driver.golden.json
Normal file
85
packages/server/src/automation/page/driver.golden.json
Normal file
|
|
@ -0,0 +1,85 @@
|
|||
{
|
||||
"functions": {
|
||||
"option-click": {
|
||||
"plain": "90863b0a65aa2a1c497b545245a8296ad2cfeeab813547ede451d490e2632b1d",
|
||||
"marked": "0a881e6b538aa589f0d895558513d6a2e1d94cbb793f1339d6f0751fefa2cfcf"
|
||||
},
|
||||
"programmatic-click": {
|
||||
"plain": "734dbde138679fda25063747d26b0ffa822061aea13db26ed32e8288dcb1980f",
|
||||
"marked": "80870581fdc8a5751f5da969309b20cbdf0ac425aefc5c9432f3afb9c14b35bd"
|
||||
},
|
||||
"select": {
|
||||
"plain": "7444c1f9021566b13311d45daf37c2ab866a1c74dbe5783529317d6491a4941c",
|
||||
"marked": "a141bc08c8d7c5bff6e6f019dc12ef0a75720e69f4e895437cb1d19b077b149a"
|
||||
},
|
||||
"focus": {
|
||||
"plain": "6fb37154d05f9387248ef22993ebcb95a850c49b561a5627ebaf36ad5854228f",
|
||||
"marked": "eb92152919a890f84e4073024c27ef00bbfd268f532208fcaa839f9bc6cc6683"
|
||||
},
|
||||
"describe": {
|
||||
"plain": "dd657dd6820c450f57e498a8a77f1ff0e8b6db40715891a65438462f1dff309e",
|
||||
"marked": "dd7eed3b51f7a978ef2794caa3a4fad7cf66d817e699de3d3e8dfef056889f39"
|
||||
},
|
||||
"field": {
|
||||
"plain": "1900b6d0121b51cf4a5d29fc605e8b6434250bc0cc77f65c57c2abe83c10e81d",
|
||||
"marked": "87931b63b8dc16dfd5d8b5679f39f49c053f6c7de4193349a987315032712c66"
|
||||
},
|
||||
"prepare": {
|
||||
"plain": "60813db3ea35c405aede8f2509f9d9da1142d1b319615f6fc5c72fa40c0db951",
|
||||
"marked": "7e204012fa0990c918217f010ed84209bfc6eef3c609bac9af12cdc2e4db661c"
|
||||
},
|
||||
"set-value": {
|
||||
"plain": "b942b3037f70886892efd5f5e79e8fb1a123f24983bf225cf90c72be25425873",
|
||||
"marked": "38b3b9535b7269be7352cec9b41603ac88ae9750cc7b58f9b428a4e459d4cd75"
|
||||
},
|
||||
"delete-selection": {
|
||||
"plain": "708ed8fe629bf30c23328f99a7f0c4f41189e709cde4c533465404797b1e6e9d",
|
||||
"marked": "e329fbbc7e98ad022fc462f3bf5c4782dec9ddc2dd28ec2133f82e5d55b63da2"
|
||||
},
|
||||
"read-value": {
|
||||
"plain": "5e2558dd3d0f32375fb371b7f7a9e3244fb4b51e78d601add2b4881d9ad88e5b",
|
||||
"marked": "6ea9a054e607c5153a2075b502c0518f22c4bbb8c4118d1b9d7fae2b143f96b6"
|
||||
},
|
||||
"receives": {
|
||||
"plain": "8a2c9c323905bc889595cb0382056a921f924deafc08c04e6935e35560d6fddf",
|
||||
"marked": "1ab158aba01124b393cb34daf6cb5dad7fc0a865d015eba4221986322cc4df53"
|
||||
},
|
||||
"contains": {
|
||||
"plain": "5d62cfc6187d07d0a964a1eb3db5f6a49a84f375c437d3f606d3e7b07460ed80",
|
||||
"marked": "7c9d1e411468befe4839f1c066446cc828617a04f0e3c8f7c4db01b2f74b5f8a"
|
||||
}
|
||||
},
|
||||
"globals": {
|
||||
"active-element": "07828a2500ef731867740e4ab46774880eac064611688e422bcecb71c6cdc849",
|
||||
"document-root": "59531dfec0550e8542be5b831f2a3e873ea10ea2ee64d665ffb016f2e2222394",
|
||||
"input-acceptance": "bb376d5fec62936887da6418209b9be305b372aff93cbd98adb2edaf013a0b67",
|
||||
"mark-input": "c14fcf467ef073e1b4f0619e60795fa63e3bb17a406f55dcbf75f04b4bc6fb90",
|
||||
"unmark-input": "c0280c55c747c4208020d8463caf9ce044e3ab36310b09f9ef1b2bb89897469c"
|
||||
},
|
||||
"snapshots": {
|
||||
"all": {
|
||||
"text": "- heading \"Checkout\" [b1e1] level=2\n- button \"Save\" [b1e2]\n- textbox \"Name\" [b1e3] value=\"é漢🙂\" focused\n- checkbox \"Remember\" [b1e4] checked=mixed\n- combobox \"Plan\" [b1e5]\n - option \"Pro\" [b1e6] selected disabled\n- spinbutton \"Quantity\" [b1e7] value=\"3\"",
|
||||
"collapsed": 0,
|
||||
"total": 7,
|
||||
"shown": 7
|
||||
},
|
||||
"query": {
|
||||
"text": "- combobox \"Plan\" [b1e1]\n - option \"Pro\" [b1e2] selected disabled",
|
||||
"collapsed": 0,
|
||||
"total": 2,
|
||||
"shown": 2
|
||||
},
|
||||
"scoped": {
|
||||
"text": "- combobox \"Plan\" [b1e1]\n - option \"Pro\" [b1e2] selected disabled",
|
||||
"collapsed": 0,
|
||||
"total": 2,
|
||||
"shown": 2
|
||||
},
|
||||
"bounded": {
|
||||
"text": "- heading \"Checkout\" [b1e1] level=2\n- button \"Save\" [b1e2]\n- (+2 more)\n- combobox \"Plan\" [b1e3] (+1 hidden)\n- spinbutton \"Quantity\" [b1e4] value=\"3\"",
|
||||
"collapsed": 2,
|
||||
"total": 7,
|
||||
"shown": 4
|
||||
}
|
||||
}
|
||||
}
|
||||
64
packages/server/src/automation/page/driver.test.ts
Normal file
64
packages/server/src/automation/page/driver.test.ts
Normal file
|
|
@ -0,0 +1,64 @@
|
|||
import assert from "node:assert/strict"
|
||||
import { test } from "node:test"
|
||||
import { readFileSync } from "node:fs"
|
||||
import { createHash } from "node:crypto"
|
||||
import { parsePageOperation, pageEnvelope } from "./operations"
|
||||
import { cdpSession, executeCdpOperation } from "./cdp-driver"
|
||||
import { nativeDriver } from "./native-driver"
|
||||
import { declaration, ELEMENT_FUNCTIONS, GLOBAL_FUNCTIONS, nativeFunctionRegistry } from "./functions"
|
||||
import { PageDriverUnsupportedError, type PageChannel } from "./protocol"
|
||||
import { renderSnapshot, type AxNode } from "./snapshot"
|
||||
import { snapshotPage } from "./actions"
|
||||
import { RefTable } from "./refs"
|
||||
const json = (file: string) => JSON.parse(readFileSync(new URL(file, import.meta.url), "utf8"))
|
||||
const golden = json("./driver.golden.json")
|
||||
|
||||
test("the shared fixed registry matches the qualified program hashes", () => {
|
||||
const hash = (source: string) => createHash("sha256").update(source).digest("hex")
|
||||
for (const id of Object.keys(ELEMENT_FUNCTIONS) as (keyof typeof ELEMENT_FUNCTIONS)[]) {
|
||||
assert.equal(hash(declaration(id, false)), golden.functions[id].plain, id)
|
||||
assert.equal(hash(declaration(id, true)), golden.functions[id].marked, `${id} application mark`)
|
||||
}
|
||||
for (const [id, source] of Object.entries(GLOBAL_FUNCTIONS)) assert.equal(hash(source), golden.globals[id], id)
|
||||
assert.deepEqual(json("./functions.generated.json"), nativeFunctionRegistry(), "regenerate the checked-in native resource when programs change")
|
||||
})
|
||||
|
||||
test("closed driver contract refuses unknown operations, params, programs and malformed arguments", () => {
|
||||
const corpus = json("./contract.cases.json")
|
||||
for (const operation of corpus.valid) assert.deepEqual(parsePageOperation(operation), operation)
|
||||
for (const operation of corpus.invalid) assert.throws(() => parsePageOperation(operation), /Unsupported or invalid/, JSON.stringify(operation))
|
||||
assert.throws(() => parsePageOperation({op:"insert-text",text:"x".repeat(256*1024)}), /parameters/)
|
||||
assert.throws(() => pageEnvelope("browser.page.send", {sessionID:"s",operationID:"o",generation:"g",method:"Runtime.evaluate",params:{expression:"evil()"}}))
|
||||
assert.throws(() => pageEnvelope("browser.page.begin", {sessionID:"s",operationID:"o",driver:"cdp"}))
|
||||
})
|
||||
|
||||
test("CDP snapshots, scoped refs and bounded rendering equal the pre-driver goldens", async () => {
|
||||
const nodes: AxNode[] = json("./driver.fixture.json")
|
||||
for (const [name, options] of Object.entries({all:{},query:{query:"plan"},scoped:{rootBackendNodeId:14},bounded:{budget:200}})) {
|
||||
let before = 0
|
||||
assert.deepEqual(renderSnapshot(nodes, () => `b1e${++before}`, options), golden.snapshots[name])
|
||||
const calls: string[] = []
|
||||
const channel: PageChannel = {send: async method => { calls.push(method); return {nodes} }}
|
||||
const session = cdpSession(channel, async () => {}, false)
|
||||
const refs = new RefTable("inspect", "b1e")
|
||||
assert.deepEqual(await snapshotPage(session, () => refs.issuer("document"), options), golden.snapshots[name])
|
||||
assert.deepEqual(calls, ["Accessibility.getFullAXTree"])
|
||||
const transport = nativeDriver({driver:"cdp",contractVersion:1}, operation => executeCdpOperation(channel, operation), async () => {})
|
||||
const other = new RefTable("inspect", "b1e")
|
||||
assert.deepEqual(await snapshotPage({driver:transport,markAutomation:false,checkpoint:async()=>{}}, () => other.issuer("document"), options), golden.snapshots[name])
|
||||
}
|
||||
})
|
||||
|
||||
test("native drivers propagate structured unsupported results without fallback", async () => {
|
||||
assert.throws(() => nativeDriver({driver:"cdp",contractVersion:2}, async()=>({}), async()=>{}), /contract/)
|
||||
assert.throws(() => nativeDriver({}, async()=>({}), async()=>{}), /contract/)
|
||||
const calls: unknown[] = []
|
||||
const driver = nativeDriver({driver:"webkit",contractVersion:1}, async operation => {
|
||||
calls.push(operation)
|
||||
return {error:{code:"unsupported",driver:"webkit",operation:operation.op}}
|
||||
}, async()=>{})
|
||||
await assert.rejects(driver.tree(), error => error instanceof PageDriverUnsupportedError && error.code === "unsupported" && error.driver === "webkit" && error.operation === "tree")
|
||||
await assert.rejects(driver.insertText("é漢🙂"), PageDriverUnsupportedError)
|
||||
assert.deepEqual(calls,[{op:"tree"},{op:"insert-text",text:"é漢🙂"}])
|
||||
await assert.rejects(driver.call("mark-input",undefined,[]), /parameters/)
|
||||
})
|
||||
27
packages/server/src/automation/page/driver.ts
Normal file
27
packages/server/src/automation/page/driver.ts
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
import { parsePageOperation } from "./operations"
|
||||
import { PAGE_DRIVER_VERSION, PageDriverUnsupportedError, type PageDriver, type DriverKind, type PageOperation, type SemanticNode } from "./protocol"
|
||||
|
||||
/** Shared typed surface for direct CDP and the fenced native operation transport. */
|
||||
export function operationDriver(kind: DriverKind, send: (operation: PageOperation) => Promise<Record<string, unknown>>, checkpoint: () => Promise<void>, application = false): PageDriver {
|
||||
const run = async (operation: PageOperation) => {
|
||||
const result = await send(parsePageOperation(operation, application))
|
||||
const error = result?.error as { code?: unknown; operation?: unknown } | undefined
|
||||
if (error?.code === "unsupported") throw new PageDriverUnsupportedError(kind, String(error.operation ?? operation.op))
|
||||
return result
|
||||
}
|
||||
return {
|
||||
kind, version: PAGE_DRIVER_VERSION,
|
||||
tree: async () => { const { nodes } = await run({ op: "tree" }); return Array.isArray(nodes) ? nodes as SemanticNode[] : [] },
|
||||
resolve: async node => (await run({ op: "resolve", node })).handle as string | undefined,
|
||||
call: async (functionId, handle, args) => run({ op: "call", function: functionId, ...(handle === undefined ? {} : { handle }), args }),
|
||||
scrollIntoView: async node => { await run({ op: "scroll-into-view", node }) },
|
||||
geometry: async node => { const { quads } = await run({ op: "geometry", node }); return Array.isArray(quads) ? quads as number[][] : [] },
|
||||
viewport: async () => { const result = await run({ op: "viewport" }); return { width: typeof result.width === "number" ? result.width : Infinity, height: typeof result.height === "number" ? result.height : Infinity } },
|
||||
hitTest: async (x, y) => (await run({ op: "hit-test", x, y })).node as number | undefined,
|
||||
pointer: async event => { await run({ op: "pointer", event }) },
|
||||
wheel: async event => { await run({ op: "wheel", event }) },
|
||||
keyboard: async event => { await run({ op: "keyboard", event }) },
|
||||
insertText: async text => { await run({ op: "insert-text", text }) },
|
||||
releaseHandles: async () => { await run({ op: "release-handles" }) }, checkpoint,
|
||||
}
|
||||
}
|
||||
103
packages/server/src/automation/page/functions.generated.json
Normal file
103
packages/server/src/automation/page/functions.generated.json
Normal file
|
|
@ -0,0 +1,103 @@
|
|||
{
|
||||
"option-click": {
|
||||
"scope": "element",
|
||||
"args": [],
|
||||
"handle": false,
|
||||
"source": "function (...args) { const element = this.nodeType === Node.ELEMENT_NODE ? this : this.parentElement;\n if (!(element instanceof Element) || !element.isConnected) throw new Error(\"codenomad:gone\"); const label = option => (option.label || option.textContent || \"\").replace(/\\s+/g, \" \").trim();\n const apply = (select, picked) => {\n if (select.matches(\":disabled\")) throw new Error(\"codenomad:disabled\");\n for (const option of picked) if (option.matches(\":disabled\")) throw new Error(\"codenomad:option-disabled:\" + label(option));\n if (!select.multiple && picked.length !== 1) throw new Error(\"codenomad:single\");\n select.focus();\n const options = Array.from(select.options);\n if (options.some(option => option.selected !== picked.includes(option))) {\n for (const option of options) option.selected = picked.includes(option);\n select.dispatchEvent(new Event(\"input\", { bubbles: true, composed: true }));\n select.dispatchEvent(new Event(\"change\", { bubbles: true }));\n }\n return Array.from(select.selectedOptions).map(label);\n };\n const choose = (select, wanted) => {\n if (select.matches(\":disabled\")) throw new Error(\"codenomad:disabled\");\n const options = Array.from(select.options), picked = [];\n for (const raw of wanted) {\n const target = String(raw).replace(/\\s+/g, \" \").trim();\n const option = options.find(item => label(item) === target) || options.find(item => item.value === String(raw));\n if (!option) throw new Error(\"codenomad:no-option:\" + target);\n if (!picked.includes(option)) picked.push(option);\n }\n return apply(select, picked);\n };\n const option = element.closest(\"option\"), select = option && option.closest(\"select\");\n if (option && select) return apply(select, [option]);\n if (element instanceof HTMLSelectElement) throw new Error(\"codenomad:is-select\");\n return null; }"
|
||||
},
|
||||
"programmatic-click": {
|
||||
"scope": "element",
|
||||
"args": [],
|
||||
"handle": false,
|
||||
"source": "function (...args) { const element = this.nodeType === Node.ELEMENT_NODE ? this : this.parentElement;\n if (!(element instanceof Element) || !element.isConnected) throw new Error(\"codenomad:gone\"); if (element instanceof HTMLElement) element.click();\n else element.dispatchEvent(new MouseEvent(\"click\", { bubbles: true, cancelable: true, composed: true }));\n return null; }"
|
||||
},
|
||||
"select": {
|
||||
"scope": "element",
|
||||
"args": [
|
||||
"strings"
|
||||
],
|
||||
"handle": false,
|
||||
"source": "function (...args) { const element = this.nodeType === Node.ELEMENT_NODE ? this : this.parentElement;\n if (!(element instanceof Element) || !element.isConnected) throw new Error(\"codenomad:gone\"); const label = option => (option.label || option.textContent || \"\").replace(/\\s+/g, \" \").trim();\n const apply = (select, picked) => {\n if (select.matches(\":disabled\")) throw new Error(\"codenomad:disabled\");\n for (const option of picked) if (option.matches(\":disabled\")) throw new Error(\"codenomad:option-disabled:\" + label(option));\n if (!select.multiple && picked.length !== 1) throw new Error(\"codenomad:single\");\n select.focus();\n const options = Array.from(select.options);\n if (options.some(option => option.selected !== picked.includes(option))) {\n for (const option of options) option.selected = picked.includes(option);\n select.dispatchEvent(new Event(\"input\", { bubbles: true, composed: true }));\n select.dispatchEvent(new Event(\"change\", { bubbles: true }));\n }\n return Array.from(select.selectedOptions).map(label);\n };\n const choose = (select, wanted) => {\n if (select.matches(\":disabled\")) throw new Error(\"codenomad:disabled\");\n const options = Array.from(select.options), picked = [];\n for (const raw of wanted) {\n const target = String(raw).replace(/\\s+/g, \" \").trim();\n const option = options.find(item => label(item) === target) || options.find(item => item.value === String(raw));\n if (!option) throw new Error(\"codenomad:no-option:\" + target);\n if (!picked.includes(option)) picked.push(option);\n }\n return apply(select, picked);\n };\n const control = element instanceof HTMLLabelElement ? element.control : element;\n const select = control instanceof HTMLSelectElement ? control : control && control.closest(\"select\");\n if (!select) throw new Error(\"codenomad:not-select\");\n return choose(select, args[0]); }"
|
||||
},
|
||||
"focus": {
|
||||
"scope": "element",
|
||||
"args": [],
|
||||
"handle": false,
|
||||
"source": "function (...args) { const element = this.nodeType === Node.ELEMENT_NODE ? this : this.parentElement;\n if (!(element instanceof Element) || !element.isConnected) throw new Error(\"codenomad:gone\"); if (element.matches(\":disabled\")) throw new Error(\"codenomad:disabled\");\n if (element instanceof HTMLElement || element instanceof SVGElement) element.focus();\n if (element.getRootNode().activeElement !== element) throw new Error(\"codenomad:unfocusable\");\n return null; }"
|
||||
},
|
||||
"describe": {
|
||||
"scope": "element",
|
||||
"args": [],
|
||||
"handle": false,
|
||||
"source": "function (...args) { const element = this.nodeType === Node.ELEMENT_NODE ? this : this.parentElement;\n if (!(element instanceof Element) || !element.isConnected) throw new Error(\"codenomad:gone\"); const role = element.getAttribute(\"role\") || element.tagName.toLowerCase();\n const name = (element.getAttribute(\"aria-label\") || element.getAttribute(\"title\") || element.getAttribute(\"placeholder\")\n || (element.labels && element.labels[0] && element.labels[0].textContent) || element.textContent || \"\").replace(/\\s+/g, \" \").trim().slice(0, 80);\n return role + (name ? \" \" + JSON.stringify(name) : \"\"); }"
|
||||
},
|
||||
"field": {
|
||||
"scope": "element",
|
||||
"args": [],
|
||||
"handle": true,
|
||||
"source": "function (...args) { const element = this.nodeType === Node.ELEMENT_NODE ? this : this.parentElement;\n if (!(element instanceof Element) || !element.isConnected) throw new Error(\"codenomad:gone\"); const textField = item => item instanceof HTMLInputElement || item instanceof HTMLTextAreaElement\n || (item instanceof HTMLElement && item.isContentEditable);\n const control = element instanceof HTMLLabelElement ? element.control : element;\n const field = control && textField(control) ? control\n : element.querySelector('input:not([type=\"hidden\"]), textarea, [contenteditable]:not([contenteditable=\"false\"])');\n if (!field || !textField(field)) throw new Error(\"codenomad:not-editable\");\n return field; }"
|
||||
},
|
||||
"prepare": {
|
||||
"scope": "element",
|
||||
"args": [
|
||||
"boolean"
|
||||
],
|
||||
"handle": false,
|
||||
"source": "function (...args) { const element = this.nodeType === Node.ELEMENT_NODE ? this : this.parentElement;\n if (!(element instanceof Element) || !element.isConnected) throw new Error(\"codenomad:gone\"); const field = element, clear = args[0];\n if (field.matches(\":disabled\")) throw new Error(\"codenomad:disabled\");\n if (field.readOnly) throw new Error(\"codenomad:readonly\");\n if (field instanceof HTMLInputElement && [\"checkbox\",\"radio\",\"button\",\"submit\",\"reset\",\"file\",\"image\",\"hidden\"].includes(field.type)) throw new Error(\"codenomad:not-text\");\n field.focus();\n const active = field.getRootNode().activeElement;\n const editable = !(field instanceof HTMLInputElement || field instanceof HTMLTextAreaElement);\n if (active !== field && !(editable && active instanceof HTMLElement && active.isContentEditable && active.contains(field))) {\n throw new Error(\"codenomad:unfocusable\");\n }\n if (field instanceof HTMLInputElement && [\"date\",\"time\",\"datetime-local\",\"month\",\"week\",\"range\",\"color\"].includes(field.type)) return \"value\";\n if (editable) {\n const selection = getSelection();\n selection.setBaseAndExtent(field, clear ? 0 : field.childNodes.length, field, field.childNodes.length);\n } else if (clear) field.select();\n // Email and number inputs have no caret API: their text is appended as a value.\n else { try { field.setSelectionRange(field.value.length, field.value.length); } catch { return \"append\"; } }\n return \"insert\"; }"
|
||||
},
|
||||
"set-value": {
|
||||
"scope": "element",
|
||||
"args": [
|
||||
"string",
|
||||
"boolean"
|
||||
],
|
||||
"handle": false,
|
||||
"source": "function (...args) { const element = this.nodeType === Node.ELEMENT_NODE ? this : this.parentElement;\n if (!(element instanceof Element) || !element.isConnected) throw new Error(\"codenomad:gone\"); if (element.matches(\":disabled\")) throw new Error(\"codenomad:disabled\");\n if (element.readOnly) throw new Error(\"codenomad:readonly\");\n const [text, append] = args, next = append ? element.value + text : text;\n // Ranges and colors have no empty value: the browser would substitute its default.\n const valid = element.type === \"color\" ? /^#[0-9a-f]{6}$/i.test(next)\n : element.type === \"range\" ? /^-?(?:\\d+(?:\\.\\d+)?|\\.\\d+)(?:[eE][+-]?\\d+)?$/.test(next)\n : next !== \"\" && [\"date\",\"time\",\"datetime-local\",\"month\",\"week\",\"number\"].includes(element.type) ? (() => { const probe = document.createElement(\"input\"); probe.type = element.type; probe.value = next; return probe.value !== \"\"; })()\n : true;\n if (!valid) throw new Error(\"codenomad:invalid-value:\" + next);\n if (next === element.value) return null;\n Object.getOwnPropertyDescriptor(HTMLInputElement.prototype, \"value\").set.call(element, next);\n element.dispatchEvent(new Event(\"input\", { bubbles: true, composed: true }));\n element.dispatchEvent(new Event(\"change\", { bubbles: true }));\n return null; }"
|
||||
},
|
||||
"delete-selection": {
|
||||
"scope": "element",
|
||||
"args": [],
|
||||
"handle": false,
|
||||
"source": "function (...args) { const element = this.nodeType === Node.ELEMENT_NODE ? this : this.parentElement;\n if (!(element instanceof Element) || !element.isConnected) throw new Error(\"codenomad:gone\"); document.execCommand(\"delete\"); return null; }"
|
||||
},
|
||||
"read-value": {
|
||||
"scope": "element",
|
||||
"args": [],
|
||||
"handle": false,
|
||||
"source": "function (...args) { const element = this.nodeType === Node.ELEMENT_NODE ? this : this.parentElement;\n if (!(element instanceof Element) || !element.isConnected) throw new Error(\"codenomad:gone\"); const value = element instanceof HTMLInputElement || element instanceof HTMLTextAreaElement ? element.value : element.innerText || element.textContent || \"\";\n if (element instanceof HTMLInputElement && element.type === \"password\") return \"•\".repeat(Math.min(value.length, 16));\n return value.length > 200 ? value.slice(0, 199) + \"…\" : value; }"
|
||||
},
|
||||
"receives": {
|
||||
"scope": "element",
|
||||
"args": [
|
||||
"handle"
|
||||
],
|
||||
"handle": false,
|
||||
"source": "function (...args) { const element = this.nodeType === Node.ELEMENT_NODE ? this : this.parentElement;\n if (!(element instanceof Element) || !element.isConnected) throw new Error(\"codenomad:gone\"); const hit = args[0] && (args[0].nodeType === Node.ELEMENT_NODE ? args[0] : args[0].parentElement);\n if (!hit || element === hit || element.contains(hit)) return null;\n const label = hit.closest(\"label\");\n if (label && (label.control === element || label.contains(element))) return null;\n const name = (hit.getAttribute(\"aria-label\") || hit.getAttribute(\"title\") || hit.innerText || \"\").replace(/\\s+/g, \" \").trim().slice(0, 60);\n return hit.tagName.toLowerCase() + (name ? \" \" + JSON.stringify(name) : \"\"); }"
|
||||
},
|
||||
"contains": {
|
||||
"scope": "element",
|
||||
"args": [
|
||||
"string"
|
||||
],
|
||||
"handle": false,
|
||||
"source": "function (...args) { const element = this.nodeType === Node.ELEMENT_NODE ? this : this.parentElement;\n if (!(element instanceof Element) || !element.isConnected) throw new Error(\"codenomad:gone\"); const needle = String(args[0]).replace(/\\s+/g, \" \").trim().toLowerCase();\n const root = element;\n const text = value => String(value || \"\").replace(/\\s+/g, \" \").toLowerCase();\n if (text(root.innerText || root.textContent).includes(needle)) return true;\n for (const item of root.querySelectorAll(\"[aria-label], [title], [placeholder], input, textarea\")) {\n if ([item.getAttribute(\"aria-label\"), item.getAttribute(\"title\"), item.getAttribute(\"placeholder\"), item.value].some(value => text(value).includes(needle))) return true;\n }\n return false; }"
|
||||
},
|
||||
"active-element": {
|
||||
"scope": "global",
|
||||
"args": [],
|
||||
"handle": true,
|
||||
"source": "document.activeElement"
|
||||
},
|
||||
"document-root": {
|
||||
"scope": "global",
|
||||
"args": [],
|
||||
"handle": true,
|
||||
"source": "document.documentElement"
|
||||
},
|
||||
"input-acceptance": {
|
||||
"scope": "global",
|
||||
"args": [],
|
||||
"handle": false,
|
||||
"source": "({ inputDelayMs: (() => {\n if (!/^https?:$/.test(location.protocol) || performance.getEntriesByType(\"paint\").length > 0) return 0;\n const navigation = performance.getEntriesByType(\"navigation\")[0];\n if (!navigation || !(navigation.domInteractive > 0)) return 50;\n return Math.max(0, Math.ceil(navigation.domInteractive + 500 - performance.now()));\n})() })"
|
||||
}
|
||||
}
|
||||
152
packages/server/src/automation/page/functions.ts
Normal file
152
packages/server/src/automation/page/functions.ts
Normal file
|
|
@ -0,0 +1,152 @@
|
|||
/** Fixed page programs. Arguments are values/handles, never interpolated source. */
|
||||
export const VALUE_LIMIT = 200
|
||||
export const INPUT_MARK_LEASE_MS = 15_000
|
||||
const LABEL = `const label = option => (option.label || option.textContent || "").replace(/\\s+/g, " ").trim();`
|
||||
const CHOOSE = `${LABEL}
|
||||
const apply = (select, picked) => {
|
||||
if (select.matches(":disabled")) throw new Error("codenomad:disabled");
|
||||
for (const option of picked) if (option.matches(":disabled")) throw new Error("codenomad:option-disabled:" + label(option));
|
||||
if (!select.multiple && picked.length !== 1) throw new Error("codenomad:single");
|
||||
select.focus();
|
||||
const options = Array.from(select.options);
|
||||
if (options.some(option => option.selected !== picked.includes(option))) {
|
||||
for (const option of options) option.selected = picked.includes(option);
|
||||
select.dispatchEvent(new Event("input", { bubbles: true, composed: true }));
|
||||
select.dispatchEvent(new Event("change", { bubbles: true }));
|
||||
}
|
||||
return Array.from(select.selectedOptions).map(label);
|
||||
};
|
||||
const choose = (select, wanted) => {
|
||||
if (select.matches(":disabled")) throw new Error("codenomad:disabled");
|
||||
const options = Array.from(select.options), picked = [];
|
||||
for (const raw of wanted) {
|
||||
const target = String(raw).replace(/\\s+/g, " ").trim();
|
||||
const option = options.find(item => label(item) === target) || options.find(item => item.value === String(raw));
|
||||
if (!option) throw new Error("codenomad:no-option:" + target);
|
||||
if (!picked.includes(option)) picked.push(option);
|
||||
}
|
||||
return apply(select, picked);
|
||||
};`
|
||||
const NOT_TEXT = ["checkbox", "radio", "button", "submit", "reset", "file", "image", "hidden"]
|
||||
const VALUE_TYPES = ["date", "time", "datetime-local", "month", "week", "range", "color"]
|
||||
const PROBED_TYPES = ["date", "time", "datetime-local", "month", "week", "number"]
|
||||
const TEXT_FIELD = `const textField = item => item instanceof HTMLInputElement || item instanceof HTMLTextAreaElement
|
||||
|| (item instanceof HTMLElement && item.isContentEditable);`
|
||||
|
||||
export const ELEMENT_FUNCTIONS = {
|
||||
"option-click": `${CHOOSE}
|
||||
const option = element.closest("option"), select = option && option.closest("select");
|
||||
if (option && select) return apply(select, [option]);
|
||||
if (element instanceof HTMLSelectElement) throw new Error("codenomad:is-select");
|
||||
return null;`,
|
||||
"programmatic-click": `if (element instanceof HTMLElement) element.click();
|
||||
else element.dispatchEvent(new MouseEvent("click", { bubbles: true, cancelable: true, composed: true }));
|
||||
return null;`,
|
||||
select: `${CHOOSE}
|
||||
const control = element instanceof HTMLLabelElement ? element.control : element;
|
||||
const select = control instanceof HTMLSelectElement ? control : control && control.closest("select");
|
||||
if (!select) throw new Error("codenomad:not-select");
|
||||
return choose(select, args[0]);`,
|
||||
focus: `if (element.matches(":disabled")) throw new Error("codenomad:disabled");
|
||||
if (element instanceof HTMLElement || element instanceof SVGElement) element.focus();
|
||||
if (element.getRootNode().activeElement !== element) throw new Error("codenomad:unfocusable");
|
||||
return null;`,
|
||||
describe: `const role = element.getAttribute("role") || element.tagName.toLowerCase();
|
||||
const name = (element.getAttribute("aria-label") || element.getAttribute("title") || element.getAttribute("placeholder")
|
||||
|| (element.labels && element.labels[0] && element.labels[0].textContent) || element.textContent || "").replace(/\\s+/g, " ").trim().slice(0, 80);
|
||||
return role + (name ? " " + JSON.stringify(name) : "");`,
|
||||
field: `${TEXT_FIELD}
|
||||
const control = element instanceof HTMLLabelElement ? element.control : element;
|
||||
const field = control && textField(control) ? control
|
||||
: element.querySelector('input:not([type="hidden"]), textarea, [contenteditable]:not([contenteditable="false"])');
|
||||
if (!field || !textField(field)) throw new Error("codenomad:not-editable");
|
||||
return field;`,
|
||||
prepare: `const field = element, clear = args[0];
|
||||
if (field.matches(":disabled")) throw new Error("codenomad:disabled");
|
||||
if (field.readOnly) throw new Error("codenomad:readonly");
|
||||
if (field instanceof HTMLInputElement && ${JSON.stringify(NOT_TEXT)}.includes(field.type)) throw new Error("codenomad:not-text");
|
||||
field.focus();
|
||||
const active = field.getRootNode().activeElement;
|
||||
const editable = !(field instanceof HTMLInputElement || field instanceof HTMLTextAreaElement);
|
||||
if (active !== field && !(editable && active instanceof HTMLElement && active.isContentEditable && active.contains(field))) {
|
||||
throw new Error("codenomad:unfocusable");
|
||||
}
|
||||
if (field instanceof HTMLInputElement && ${JSON.stringify(VALUE_TYPES)}.includes(field.type)) return "value";
|
||||
if (editable) {
|
||||
const selection = getSelection();
|
||||
selection.setBaseAndExtent(field, clear ? 0 : field.childNodes.length, field, field.childNodes.length);
|
||||
} else if (clear) field.select();
|
||||
// Email and number inputs have no caret API: their text is appended as a value.
|
||||
else { try { field.setSelectionRange(field.value.length, field.value.length); } catch { return "append"; } }
|
||||
return "insert";`,
|
||||
"set-value": `if (element.matches(":disabled")) throw new Error("codenomad:disabled");
|
||||
if (element.readOnly) throw new Error("codenomad:readonly");
|
||||
const [text, append] = args, next = append ? element.value + text : text;
|
||||
// Ranges and colors have no empty value: the browser would substitute its default.
|
||||
const valid = element.type === "color" ? /^#[0-9a-f]{6}$/i.test(next)
|
||||
: element.type === "range" ? /^-?(?:\\d+(?:\\.\\d+)?|\\.\\d+)(?:[eE][+-]?\\d+)?$/.test(next)
|
||||
: next !== "" && ${JSON.stringify(PROBED_TYPES)}.includes(element.type) ? (() => { const probe = document.createElement("input"); probe.type = element.type; probe.value = next; return probe.value !== ""; })()
|
||||
: true;
|
||||
if (!valid) throw new Error("codenomad:invalid-value:" + next);
|
||||
if (next === element.value) return null;
|
||||
Object.getOwnPropertyDescriptor(HTMLInputElement.prototype, "value").set.call(element, next);
|
||||
element.dispatchEvent(new Event("input", { bubbles: true, composed: true }));
|
||||
element.dispatchEvent(new Event("change", { bubbles: true }));
|
||||
return null;`,
|
||||
"delete-selection": `document.execCommand("delete"); return null;`,
|
||||
"read-value": `const value = element instanceof HTMLInputElement || element instanceof HTMLTextAreaElement ? element.value : element.innerText || element.textContent || "";
|
||||
if (element instanceof HTMLInputElement && element.type === "password") return "•".repeat(Math.min(value.length, 16));
|
||||
return value.length > ${VALUE_LIMIT} ? value.slice(0, ${VALUE_LIMIT - 1}) + "…" : value;`,
|
||||
receives: `const hit = args[0] && (args[0].nodeType === Node.ELEMENT_NODE ? args[0] : args[0].parentElement);
|
||||
if (!hit || element === hit || element.contains(hit)) return null;
|
||||
const label = hit.closest("label");
|
||||
if (label && (label.control === element || label.contains(element))) return null;
|
||||
const name = (hit.getAttribute("aria-label") || hit.getAttribute("title") || hit.innerText || "").replace(/\\s+/g, " ").trim().slice(0, 60);
|
||||
return hit.tagName.toLowerCase() + (name ? " " + JSON.stringify(name) : "");`,
|
||||
contains: `const needle = String(args[0]).replace(/\\s+/g, " ").trim().toLowerCase();
|
||||
const root = element;
|
||||
const text = value => String(value || "").replace(/\\s+/g, " ").toLowerCase();
|
||||
if (text(root.innerText || root.textContent).includes(needle)) return true;
|
||||
for (const item of root.querySelectorAll("[aria-label], [title], [placeholder], input, textarea")) {
|
||||
if ([item.getAttribute("aria-label"), item.getAttribute("title"), item.getAttribute("placeholder"), item.value].some(value => text(value).includes(needle))) return true;
|
||||
}
|
||||
return false;`,
|
||||
} as const
|
||||
|
||||
export const GLOBAL_FUNCTIONS = {
|
||||
"active-element": "document.activeElement",
|
||||
"document-root": "document.documentElement",
|
||||
"input-acceptance": `({ inputDelayMs: (() => {
|
||||
if (!/^https?:$/.test(location.protocol) || performance.getEntriesByType("paint").length > 0) return 0;
|
||||
const navigation = performance.getEntriesByType("navigation")[0];
|
||||
if (!navigation || !(navigation.domInteractive > 0)) return 50;
|
||||
return Math.max(0, Math.ceil(navigation.domInteractive + 500 - performance.now()));
|
||||
})() })`,
|
||||
"mark-input": `globalThis.__codenomadAutomationInputUntil = performance.now() + ${INPUT_MARK_LEASE_MS}; Date.now() / 1000`,
|
||||
"unmark-input": "globalThis.__codenomadAutomationInputUntil = Math.min(globalThis.__codenomadAutomationInputUntil || 0, performance.now()); true",
|
||||
} as const
|
||||
export type PageFunctionId = keyof typeof ELEMENT_FUNCTIONS | keyof typeof GLOBAL_FUNCTIONS
|
||||
export const FUNCTION_ARGS: Record<PageFunctionId, readonly string[]> = {
|
||||
"option-click": [], "programmatic-click": [], select: ["strings"], focus: [], describe: [], field: [], prepare: ["boolean"],
|
||||
"set-value": ["string", "boolean"], "delete-selection": [], "read-value": [], receives: ["handle"], contains: ["string"],
|
||||
"active-element": [], "document-root": [], "input-acceptance": [], "mark-input": [], "unmark-input": [],
|
||||
}
|
||||
export const handleResult = (id: PageFunctionId) => ["field", "active-element", "document-root"].includes(id)
|
||||
export const applicationOnly = (id: PageFunctionId) => id === "mark-input" || id === "unmark-input"
|
||||
export function declaration(id: keyof typeof ELEMENT_FUNCTIONS, mark: boolean): string {
|
||||
const body = ELEMENT_FUNCTIONS[id]
|
||||
const element = `const element = this.nodeType === Node.ELEMENT_NODE ? this : this.parentElement;
|
||||
if (!(element instanceof Element) || !element.isConnected) throw new Error("codenomad:gone");`
|
||||
return mark
|
||||
? `function (...args) { ${element} globalThis.__codenomadAutomationDepth = (globalThis.__codenomadAutomationDepth || 0) + 1;
|
||||
try { return (() => { ${body} })(); } finally { globalThis.__codenomadAutomationDepth--; } }`
|
||||
: `function (...args) { ${element} ${body} }`
|
||||
}
|
||||
|
||||
/** Compile-time native resource; generation is checked by the contract tests. */
|
||||
export function nativeFunctionRegistry() {
|
||||
return Object.fromEntries((Object.keys(FUNCTION_ARGS) as PageFunctionId[]).filter(id => !applicationOnly(id)).map(id => [id, {
|
||||
scope: Object.hasOwn(ELEMENT_FUNCTIONS, id) ? "element" : "global", args: FUNCTION_ARGS[id], handle: handleResult(id),
|
||||
source: Object.hasOwn(ELEMENT_FUNCTIONS, id) ? declaration(id as keyof typeof ELEMENT_FUNCTIONS, false) : GLOBAL_FUNCTIONS[id as keyof typeof GLOBAL_FUNCTIONS],
|
||||
}]))
|
||||
}
|
||||
|
|
@ -2,20 +2,17 @@ import assert from "node:assert/strict"
|
|||
import { test } from "node:test"
|
||||
import { clickAt, dragBetween, pressKeys } from "./input"
|
||||
import type { PageSession } from "./protocol"
|
||||
import { cdpSession } from "./cdp-driver"
|
||||
|
||||
test("failed pointer and key sequences release potentially held inputs without replaying the action", async () => {
|
||||
for (const kind of ["click", "drag", "keys"] as const) {
|
||||
const calls: Array<{ method: string; params: Record<string, unknown> }> = []
|
||||
let pressed = false, failed = false
|
||||
const session: PageSession = {
|
||||
markAutomation: true,
|
||||
channel: { send: async (method, params = {}) => {
|
||||
const session: PageSession = cdpSession({ send: async (method, params = {}) => {
|
||||
calls.push({ method, params })
|
||||
if (params.type === "mousePressed" || params.type === "rawKeyDown") pressed = true
|
||||
return method === "Runtime.evaluate" ? { result: { value: 1_700_000_000 } } : {}
|
||||
} },
|
||||
checkpoint: async () => { if (pressed && !failed) { failed = true; throw new Error("Page changed") } },
|
||||
}
|
||||
} }, async () => { if (pressed && !failed) { failed = true; throw new Error("Page changed") } }, true)
|
||||
await assert.rejects(kind === "click" ? clickAt(session, { x: 10, y: 20 })
|
||||
: kind === "drag" ? dragBetween(session, { x: 10, y: 20 }, { x: 50, y: 60 }) : pressKeys(session, "Control+K"), /Page changed/)
|
||||
const inputs = calls.filter(call => call.method.startsWith("Input."))
|
||||
|
|
@ -31,10 +28,7 @@ test("a frozen mark write cannot suppress release, and late input effects are re
|
|||
for (const recoversBeforeRelease of [true, false]) {
|
||||
const calls: Array<{ method: string; params: Record<string, unknown> }> = []
|
||||
let frozen = false, clicks = 0, queuedRelease: (() => void) | undefined
|
||||
const session: PageSession = {
|
||||
markAutomation: true,
|
||||
checkpoint: async () => {},
|
||||
channel: { send: async (method, params = {}) => {
|
||||
const session: PageSession = cdpSession({ send: async (method, params = {}) => {
|
||||
calls.push({ method, params })
|
||||
if (method === "Runtime.evaluate" && frozen) throw new Error("CDP mark timed out")
|
||||
if (params.type === "mousePressed") { frozen = true; throw new Error("CDP input timed out") }
|
||||
|
|
@ -43,8 +37,7 @@ test("a frozen mark write cannot suppress release, and late input effects are re
|
|||
else { queuedRelease = () => { clicks++ }; throw new Error("CDP release timed out") }
|
||||
}
|
||||
return method === "Runtime.evaluate" ? { result: { value: 1_700_000_000 } } : {}
|
||||
} },
|
||||
}
|
||||
} }, async () => {}, true)
|
||||
await assert.rejects(clickAt(session, { x: 10, y: 20 }), /outcome is unknown.*may still be delivered; inspect before retrying/)
|
||||
assert.equal(calls.filter(call => call.params.type === "mousePressed").length, 1)
|
||||
assert.equal(calls.filter(call => call.params.type === "mouseReleased").length, 1,
|
||||
|
|
@ -61,10 +54,10 @@ test("trusted input waits until a freshly navigated page accepts it, within a bo
|
|||
const calls: string[] = []
|
||||
const answers = [...delays]
|
||||
const started = Date.now()
|
||||
await pressKeys({ markAutomation: false, checkpoint: async () => {}, channel: { send: async (method, params = {}) => {
|
||||
await pressKeys(cdpSession({ send: async (method, params = {}) => {
|
||||
calls.push(method === "Runtime.evaluate" ? "ready?" : `${method}:${params.type}`)
|
||||
return method === "Runtime.evaluate" ? { result: { value: { inputDelayMs: answers.length > 1 ? answers.shift() : answers[0] } } } : {}
|
||||
} } }, "a")
|
||||
} }, async () => {}, false), "a")
|
||||
const elapsed = Date.now() - started
|
||||
const firstKey = calls.indexOf("Input.dispatchKeyEvent:keyDown")
|
||||
assert.ok(firstKey > 0 && calls.slice(0, firstKey).every(call => call === "ready?"), "no key precedes acceptance")
|
||||
|
|
@ -80,8 +73,7 @@ test("trusted input waits until a freshly navigated page accepts it, within a bo
|
|||
|
||||
test("input refuses when the page cannot supply a valid provenance timestamp", async () => {
|
||||
const methods: string[] = []
|
||||
await assert.rejects(pressKeys({ markAutomation: true, checkpoint: async () => {},
|
||||
channel: { send: async method => { methods.push(method); return { result: { value: null } } } },
|
||||
}, "Enter"), /Cannot establish automation input provenance/)
|
||||
await assert.rejects(pressKeys(cdpSession({ send: async method => { methods.push(method); return { result: { value: null } } } },
|
||||
async () => {}, true), "Enter"), /Cannot establish automation input provenance/)
|
||||
assert.ok(methods.every(method => method === "Runtime.evaluate"))
|
||||
})
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
import { keyChord } from "./keys"
|
||||
import { callOn, Handle, markedInput, OBJECT_GROUP, resolveElement, type PageSession } from "./protocol"
|
||||
import { callOn, Handle, markedInput, resolveElement, PageDriverUnsupportedError, type PageSession } from "./protocol"
|
||||
|
||||
/** Trusted pointer, wheel and keyboard input through DevTools, aimed like a user would:
|
||||
* at the element's visible centre, only when that point actually reaches the element. */
|
||||
|
|
@ -8,12 +8,10 @@ export interface Point { x: number; y: number }
|
|||
|
||||
/** The point a user would click, or undefined for an element without a visible box. */
|
||||
export async function aim(session: PageSession, ref: string, backendNodeId: number, options: { hitTest?: boolean; scroll?: boolean } = {}): Promise<Point | undefined> {
|
||||
const { channel } = session
|
||||
if (options.scroll !== false) await channel.send("DOM.scrollIntoViewIfNeeded", { backendNodeId }).catch(() => undefined)
|
||||
if (options.scroll !== false) await session.driver.scrollIntoView(backendNodeId).catch(error => { if (error instanceof PageDriverUnsupportedError) throw error })
|
||||
await session.checkpoint()
|
||||
const result = await channel.send("DOM.getContentQuads", { backendNodeId }).catch(() => ({} as Record<string, unknown>))
|
||||
const quads = await session.driver.geometry(backendNodeId).catch(error => { if (error instanceof PageDriverUnsupportedError) throw error; return [] })
|
||||
await session.checkpoint()
|
||||
const quads = Array.isArray(result.quads) ? result.quads as number[][] : []
|
||||
const viewport = await layoutViewport(session)
|
||||
const quad = quads.find(item => item.length === 8 && area(item) >= 1)
|
||||
if (!quad) return undefined
|
||||
|
|
@ -30,29 +28,18 @@ export async function aim(session: PageSession, ref: string, backendNodeId: numb
|
|||
/** Refuses a point that another element covers, instead of acting on whatever is on top. */
|
||||
async function assertReceives(session: PageSession, ref: string, backendNodeId: number, point: Point): Promise<void> {
|
||||
// Like real events, hit testing passes through `pointer-events: none` layers (toasts, overlays' gaps).
|
||||
const hit = await session.channel.send("DOM.getNodeForLocation", { x: Math.floor(point.x), y: Math.floor(point.y),
|
||||
includeUserAgentShadowDOM: false, ignorePointerEventsNone: false }).catch(() => undefined)
|
||||
const hitNode = await session.driver.hitTest(Math.floor(point.x), Math.floor(point.y)).catch(error => { if (error instanceof PageDriverUnsupportedError) throw error; return undefined })
|
||||
await session.checkpoint()
|
||||
const hitNode = hit?.backendNodeId
|
||||
if (typeof hitNode !== "number" || hitNode === backendNodeId) return
|
||||
const target = await resolveElement(session, ref, backendNodeId)
|
||||
const other = await resolveElement(session, ref, hitNode).catch(() => undefined)
|
||||
if (!other) return
|
||||
const covered = await callOn(session, ref, target, RECEIVES, [new Handle(other)])
|
||||
const covered = await callOn(session, ref, target, "receives", [new Handle(other)])
|
||||
if (typeof covered === "string") throw new Error(`${ref} is covered by ${covered}; close or move it first, then inspect again`)
|
||||
}
|
||||
|
||||
const RECEIVES = `const hit = args[0] && (args[0].nodeType === Node.ELEMENT_NODE ? args[0] : args[0].parentElement);
|
||||
if (!hit || element === hit || element.contains(hit)) return null;
|
||||
const label = hit.closest("label");
|
||||
if (label && (label.control === element || label.contains(element))) return null;
|
||||
const name = (hit.getAttribute("aria-label") || hit.getAttribute("title") || hit.innerText || "").replace(/\\s+/g, " ").trim().slice(0, 60);
|
||||
return hit.tagName.toLowerCase() + (name ? " " + JSON.stringify(name) : "");`
|
||||
|
||||
async function layoutViewport(session: PageSession): Promise<{ width: number; height: number }> {
|
||||
const metrics = await session.channel.send("Page.getLayoutMetrics").catch(() => ({} as Record<string, unknown>))
|
||||
const viewport = (metrics.cssLayoutViewport ?? metrics.layoutViewport) as { clientWidth?: number; clientHeight?: number } | undefined
|
||||
return { width: viewport?.clientWidth ?? Number.POSITIVE_INFINITY, height: viewport?.clientHeight ?? Number.POSITIVE_INFINITY }
|
||||
return session.driver.viewport().catch(error => { if (error instanceof PageDriverUnsupportedError) throw error; return { width: Infinity, height: Infinity } })
|
||||
}
|
||||
|
||||
function area(quad: number[]): number {
|
||||
|
|
@ -69,42 +56,42 @@ export type MouseButton = keyof typeof BUTTONS
|
|||
|
||||
export async function clickAt(session: PageSession, point: Point, button: MouseButton = "left"): Promise<void> {
|
||||
await markedInput(session, async dispatch => {
|
||||
await dispatch("Input.dispatchMouseEvent", { type: "mouseMoved", ...point })
|
||||
await dispatch("Input.dispatchMouseEvent", { type: "mousePressed", ...point, button, buttons: BUTTONS[button], clickCount: 1 })
|
||||
await dispatch("Input.dispatchMouseEvent", { type: "mouseReleased", ...point, button, buttons: 0, clickCount: 1 })
|
||||
await dispatch("pointer", { type: "mouseMoved", ...point })
|
||||
await dispatch("pointer", { type: "mousePressed", ...point, button, buttons: BUTTONS[button], clickCount: 1 })
|
||||
await dispatch("pointer", { type: "mouseReleased", ...point, button, buttons: 0, clickCount: 1 })
|
||||
})
|
||||
}
|
||||
|
||||
export async function hoverAt(session: PageSession, point: Point): Promise<void> {
|
||||
await markedInput(session, dispatch => dispatch("Input.dispatchMouseEvent", { type: "mouseMoved", ...point }))
|
||||
await markedInput(session, dispatch => dispatch("pointer", { type: "mouseMoved", ...point }))
|
||||
}
|
||||
|
||||
export async function wheelAt(session: PageSession, point: Point | undefined, delta: { x: number; y: number }): Promise<void> {
|
||||
const at = point ?? await viewportCentre(session)
|
||||
await markedInput(session, async dispatch => {
|
||||
await dispatch("Input.dispatchMouseEvent", { type: "mouseMoved", ...at })
|
||||
await dispatch("Input.dispatchMouseEvent", { type: "mouseWheel", ...at, deltaX: delta.x, deltaY: delta.y })
|
||||
await dispatch("pointer", { type: "mouseMoved", ...at })
|
||||
await dispatch("wheel", { type: "mouseWheel", ...at, deltaX: delta.x, deltaY: delta.y })
|
||||
})
|
||||
}
|
||||
|
||||
/** A pointer drag in small steps, so pointer-based drag libraries pass their activation distance. */
|
||||
export async function dragBetween(session: PageSession, from: Point, to: Point): Promise<void> {
|
||||
await markedInput(session, async dispatch => {
|
||||
await dispatch("Input.dispatchMouseEvent", { type: "mouseMoved", ...from })
|
||||
await dispatch("Input.dispatchMouseEvent", { type: "mousePressed", ...from, button: "left", buttons: 1, clickCount: 1 })
|
||||
await dispatch("pointer", { type: "mouseMoved", ...from })
|
||||
await dispatch("pointer", { type: "mousePressed", ...from, button: "left", buttons: 1, clickCount: 1 })
|
||||
const steps = 10
|
||||
for (let step = 1; step <= steps; step++) {
|
||||
await dispatch("Input.dispatchMouseEvent", { type: "mouseMoved", x: from.x + (to.x - from.x) * step / steps,
|
||||
await dispatch("pointer", { type: "mouseMoved", x: from.x + (to.x - from.x) * step / steps,
|
||||
y: from.y + (to.y - from.y) * step / steps, button: "left", buttons: 1 })
|
||||
}
|
||||
await dispatch("Input.dispatchMouseEvent", { type: "mouseReleased", ...to, button: "left", buttons: 0, clickCount: 1 })
|
||||
await dispatch("pointer", { type: "mouseReleased", ...to, button: "left", buttons: 0, clickCount: 1 })
|
||||
})
|
||||
}
|
||||
|
||||
export async function pressKeys(session: PageSession, chord: string): Promise<void> {
|
||||
const events = keyChord(chord)
|
||||
await markedInput(session, async dispatch => {
|
||||
for (const event of events) await dispatch("Input.dispatchKeyEvent", event)
|
||||
for (const event of events) await dispatch("keyboard", event)
|
||||
})
|
||||
}
|
||||
|
||||
|
|
@ -117,10 +104,10 @@ async function viewportCentre(session: PageSession): Promise<Point> {
|
|||
export async function waitForText(session: PageSession, text: string, gone: boolean, timeoutMs: number): Promise<{ found: boolean; waitedMs: number }> {
|
||||
const started = Date.now()
|
||||
for (;;) {
|
||||
const root = await session.channel.send("Runtime.evaluate", { expression: "document.documentElement", objectGroup: OBJECT_GROUP })
|
||||
const root = await session.driver.call("document-root", undefined, [])
|
||||
await session.checkpoint()
|
||||
const objectId = (root.result as { objectId?: unknown } | undefined)?.objectId
|
||||
const found = typeof objectId === "string" && await callOn(session, "the page", objectId, CONTAINS, [text]) === true
|
||||
const objectId = root.handle
|
||||
const found = typeof objectId === "string" && await callOn(session, "the page", objectId, "contains", [text]) === true
|
||||
const waitedMs = Date.now() - started
|
||||
if (found !== gone) return { found, waitedMs }
|
||||
if (waitedMs >= timeoutMs) {
|
||||
|
|
@ -129,12 +116,3 @@ export async function waitForText(session: PageSession, text: string, gone: bool
|
|||
await new Promise(resolve => setTimeout(resolve, Math.min(200, timeoutMs - waitedMs)))
|
||||
}
|
||||
}
|
||||
|
||||
const CONTAINS = `const needle = String(args[0]).replace(/\\s+/g, " ").trim().toLowerCase();
|
||||
const root = element;
|
||||
const text = value => String(value || "").replace(/\\s+/g, " ").toLowerCase();
|
||||
if (text(root.innerText || root.textContent).includes(needle)) return true;
|
||||
for (const item of root.querySelectorAll("[aria-label], [title], [placeholder], input, textarea")) {
|
||||
if ([item.getAttribute("aria-label"), item.getAttribute("title"), item.getAttribute("placeholder"), item.value].some(value => text(value).includes(needle))) return true;
|
||||
}
|
||||
return false;`
|
||||
|
|
|
|||
7
packages/server/src/automation/page/native-driver.ts
Normal file
7
packages/server/src/automation/page/native-driver.ts
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
import { operationDriver } from "./driver"
|
||||
import { PAGE_DRIVER_VERSION, type DriverKind, type PageOperation } from "./protocol"
|
||||
|
||||
export function nativeDriver(descriptor: {driver?: unknown; contractVersion?: unknown}, send: (operation: PageOperation) => Promise<Record<string, unknown>>, checkpoint: () => Promise<void>) {
|
||||
if (!descriptor || (descriptor.driver !== "cdp" && descriptor.driver !== "webkit") || descriptor.contractVersion !== PAGE_DRIVER_VERSION) throw new Error("Unsupported native browser page driver contract")
|
||||
return operationDriver(descriptor.driver as DriverKind, send, checkpoint)
|
||||
}
|
||||
89
packages/server/src/automation/page/operations.ts
Normal file
89
packages/server/src/automation/page/operations.ts
Normal file
|
|
@ -0,0 +1,89 @@
|
|||
import { applicationOnly, FUNCTION_ARGS, GLOBAL_FUNCTIONS, ELEMENT_FUNCTIONS, type PageFunctionId } from "./functions"
|
||||
import type { PageOperation } from "./protocol"
|
||||
|
||||
const fail = (): never => { throw new Error("Unsupported or invalid browser page operation parameters") }
|
||||
function object(value: unknown): Record<string, unknown> {
|
||||
if (!value || typeof value !== "object" || Array.isArray(value) || ![Object.prototype, null].includes(Object.getPrototypeOf(value))) return fail()
|
||||
return value as Record<string, unknown>
|
||||
}
|
||||
function keys(value: Record<string, unknown>, allowed: string[], required: string[] = []) {
|
||||
if (Object.keys(value).some(key => !allowed.includes(key)) || required.some(key => !Object.hasOwn(value, key))) fail()
|
||||
}
|
||||
const string = (value: unknown, limit = 512): value is string => typeof value === "string" && value.length > 0 && value.length <= limit
|
||||
const number = (value: unknown, max = 10_000_000): value is number => typeof value === "number" && Number.isFinite(value) && Math.abs(value) <= max
|
||||
const integer = (value: unknown, min: number, max: number): value is number => number(value, max) && Number.isSafeInteger(value) && value >= min
|
||||
function optionalNumber(value: Record<string, unknown>, key: string, max: number, integral = false) {
|
||||
if (Object.hasOwn(value, key) && !(integral ? integer(value[key], 0, max) : number(value[key], max))) fail()
|
||||
}
|
||||
/** Strict at both ends of the private channel; application provenance never crosses it. */
|
||||
export function parsePageOperation(raw: unknown, allowApplication = false): PageOperation {
|
||||
const value = object(raw)
|
||||
try { if (new TextEncoder().encode(JSON.stringify(value)).length > 256 * 1024) fail() } catch { fail() }
|
||||
if (!Object.hasOwn(value, "op") || typeof value.op !== "string") fail()
|
||||
switch (value.op) {
|
||||
case "tree": case "viewport": case "release-handles": case "checkpoint": keys(value, ["op"]); break
|
||||
case "resolve": case "geometry": case "scroll-into-view":
|
||||
keys(value, ["op", "node"], ["node"])
|
||||
if (!integer(value.node, 1, Number.MAX_SAFE_INTEGER)) fail()
|
||||
break
|
||||
case "hit-test":
|
||||
keys(value, ["op", "x", "y"], ["x", "y"])
|
||||
if (!number(value.x) || !number(value.y) || !Number.isInteger(value.x) || !Number.isInteger(value.y)) fail()
|
||||
break
|
||||
case "call": {
|
||||
keys(value, ["op", "function", "handle", "args"], ["function", "args"])
|
||||
if (typeof value.function !== "string" || !Object.hasOwn(FUNCTION_ARGS, value.function)) fail()
|
||||
const id = value.function as PageFunctionId
|
||||
if (applicationOnly(id) && !allowApplication) fail()
|
||||
if (Object.hasOwn(ELEMENT_FUNCTIONS, id) ? !string(value.handle) : Object.hasOwn(value, "handle")) fail()
|
||||
if (!Object.hasOwn(ELEMENT_FUNCTIONS, id) && !Object.hasOwn(GLOBAL_FUNCTIONS, id)) fail()
|
||||
const args = value.args
|
||||
if (!Array.isArray(args) || args.length !== FUNCTION_ARGS[id].length) return fail()
|
||||
for (const [index, rule] of FUNCTION_ARGS[id].entries()) {
|
||||
const argument = args[index]
|
||||
if (rule === "string" ? typeof argument !== "string"
|
||||
: rule === "boolean" ? typeof argument !== "boolean"
|
||||
: rule === "strings" ? !Array.isArray(argument) || argument.some(item => typeof item !== "string")
|
||||
: rule === "handle" ? !argument || typeof argument !== "object" || Array.isArray(argument) || Object.keys(argument).join() !== "handle" || !string(argument.handle)
|
||||
: true) fail()
|
||||
}
|
||||
break
|
||||
}
|
||||
case "pointer": case "wheel": case "keyboard": {
|
||||
keys(value, ["op", "event"], ["event"])
|
||||
const event = object(value.event)
|
||||
const common = ["type", "modifiers", "timestamp"]
|
||||
optionalNumber(event, "modifiers", 15, true)
|
||||
optionalNumber(event, "timestamp", Number.MAX_SAFE_INTEGER)
|
||||
if (value.op === "keyboard") {
|
||||
keys(event, [...common, "key", "code", "windowsVirtualKeyCode", "text", "unmodifiedText", "location", "buttons"], ["type"])
|
||||
if (typeof event.type !== "string" || !["keyDown", "rawKeyDown", "keyUp"].includes(event.type)) fail()
|
||||
for (const key of ["key", "code", "text", "unmodifiedText"]) if (Object.hasOwn(event, key) && !string(event[key], 256)) fail()
|
||||
optionalNumber(event, "windowsVirtualKeyCode", 65535, true)
|
||||
optionalNumber(event, "location", 3, true)
|
||||
optionalNumber(event, "buttons", 7, true)
|
||||
} else {
|
||||
const wheel = value.op === "wheel"
|
||||
keys(event, [...common, "x", "y", ...(wheel ? ["deltaX", "deltaY"] : ["button", "buttons", "clickCount"])], ["type", "x", "y", ...(wheel ? ["deltaX", "deltaY"] : [])])
|
||||
if (!number(event.x) || !number(event.y)) fail()
|
||||
if (wheel) { if (event.type !== "mouseWheel" || !number(event.deltaX) || !number(event.deltaY)) fail() }
|
||||
else {
|
||||
if (typeof event.type !== "string" || !["mouseMoved", "mousePressed", "mouseReleased"].includes(event.type)) fail()
|
||||
if (event.button !== undefined && (typeof event.button !== "string" || !["left", "right", "middle", "none"].includes(event.button))) fail()
|
||||
optionalNumber(event, "buttons", 7, true)
|
||||
optionalNumber(event, "clickCount", 3, true)
|
||||
}
|
||||
}
|
||||
break
|
||||
}
|
||||
case "insert-text": keys(value, ["op", "text"], ["text"]); if (typeof value.text !== "string") fail(); break
|
||||
default: fail()
|
||||
}
|
||||
return value as unknown as PageOperation
|
||||
}
|
||||
|
||||
export function pageEnvelope(method: string, input: Record<string, unknown>): void {
|
||||
keys(input, ["sessionID", "operationID", ...(method === "browser.page.send" ? ["generation", "operation"] : method === "browser.page.check" || method === "browser.page.end" ? ["generation"] : [])],
|
||||
["sessionID", "operationID", ...(method === "browser.page.send" ? ["generation", "operation"] : method === "browser.page.check" ? ["generation"] : [])])
|
||||
if (Object.hasOwn(input, "generation") && !string(input.generation)) fail()
|
||||
}
|
||||
|
|
@ -1,77 +1,76 @@
|
|||
/** Protocol primitives shared by the page engine's actions. */
|
||||
|
||||
/** One page's protocol connection, whatever carries it (DevTools socket or a native pipe). */
|
||||
export interface PageChannel {
|
||||
send(method: string, params?: Record<string, unknown>): Promise<Record<string, unknown>>
|
||||
}
|
||||
|
||||
export interface PageSession {
|
||||
channel: PageChannel
|
||||
/** Throws once the page changed since the operation began; runs between protocol steps. */
|
||||
checkpoint(): Promise<void>
|
||||
/** The trusted CodeNomad window: page-side automation and trusted input are marked so they
|
||||
* never count as a human answer (`form-answer-interaction.ts`). Never set this in a user's web page. */
|
||||
markAutomation: boolean
|
||||
}
|
||||
|
||||
/** An element handle passed to a page function, as opposed to a plain value. */
|
||||
export class Handle {
|
||||
constructor(readonly objectId: string) {}
|
||||
}
|
||||
import { VALUE_LIMIT, type PageFunctionId } from "./functions"
|
||||
export { VALUE_LIMIT, INPUT_MARK_LEASE_MS } from "./functions"
|
||||
|
||||
/** Low-level CDP transport, used only by the CDP adapter. */
|
||||
export interface PageChannel { send(method: string, params?: Record<string, unknown>): Promise<Record<string, unknown>> }
|
||||
export const PAGE_DRIVER_VERSION = 1
|
||||
export const OBJECT_GROUP = "codenomad-automation"
|
||||
export const VALUE_LIMIT = 200
|
||||
|
||||
export function gone(ref: string): Error {
|
||||
return new Error(`${ref} is no longer in the page; inspect again`)
|
||||
export type DriverKind = "cdp" | "webkit"
|
||||
export interface SemanticNode {
|
||||
id?: string; parent?: string; children?: string[]; ignored?: boolean; node?: number
|
||||
role?: unknown; name?: unknown; description?: unknown; value?: unknown
|
||||
properties?: Array<{ name?: string; value?: unknown }>
|
||||
}
|
||||
export interface PointerInput { type: "mouseMoved" | "mousePressed" | "mouseReleased"; x: number; y: number; button?: "left" | "right" | "middle" | "none"; buttons?: number; clickCount?: number; modifiers?: number; timestamp?: number }
|
||||
export interface WheelInput { type: "mouseWheel"; x: number; y: number; deltaX: number; deltaY: number; modifiers?: number; timestamp?: number }
|
||||
export interface KeyboardInput { type: "keyDown" | "rawKeyDown" | "keyUp"; key?: string; code?: string; windowsVirtualKeyCode?: number; modifiers?: number; text?: string; unmodifiedText?: string; location?: number; timestamp?: number; buttons?: number }
|
||||
export type JsonValue = string | number | boolean | null | JsonValue[] | { [key: string]: JsonValue }
|
||||
export type PageArgument = JsonValue
|
||||
export interface CallResult { value?: unknown; handle?: string; exception?: string }
|
||||
export type PageOperation =
|
||||
| { op: "tree" | "viewport" | "release-handles" | "checkpoint" }
|
||||
| { op: "resolve" | "scroll-into-view" | "geometry"; node: number }
|
||||
| { op: "hit-test"; x: number; y: number }
|
||||
| { op: "call"; function: PageFunctionId; handle?: string; args: PageArgument[] }
|
||||
| { op: "pointer"; event: PointerInput }
|
||||
| { op: "wheel"; event: WheelInput }
|
||||
| { op: "keyboard"; event: KeyboardInput }
|
||||
| { op: "insert-text"; text: string }
|
||||
|
||||
export async function resolveElement(session: PageSession, ref: string, backendNodeId: number): Promise<string> {
|
||||
let result: Record<string, unknown>
|
||||
try {
|
||||
result = await session.channel.send("DOM.resolveNode", { backendNodeId, objectGroup: OBJECT_GROUP })
|
||||
} catch {
|
||||
/** One semantic page; drivers own protocol/native adaptation, never arbitrary evaluation. */
|
||||
export interface PageDriver {
|
||||
readonly kind: DriverKind
|
||||
readonly version: typeof PAGE_DRIVER_VERSION
|
||||
tree(): Promise<SemanticNode[]>
|
||||
resolve(node: number): Promise<string | undefined>
|
||||
call(functionId: PageFunctionId, handle: string | undefined, args: PageArgument[]): Promise<CallResult>
|
||||
scrollIntoView(node: number): Promise<void>
|
||||
geometry(node: number): Promise<number[][]>
|
||||
viewport(): Promise<{ width: number; height: number }>
|
||||
hitTest(x: number, y: number): Promise<number | undefined>
|
||||
pointer(event: PointerInput): Promise<void>
|
||||
wheel(event: WheelInput): Promise<void>
|
||||
keyboard(event: KeyboardInput): Promise<void>
|
||||
insertText(text: string): Promise<void>
|
||||
releaseHandles(): Promise<void>
|
||||
checkpoint(): Promise<void>
|
||||
}
|
||||
export class PageDriverUnsupportedError extends Error {
|
||||
readonly code = "unsupported"
|
||||
constructor(readonly driver: DriverKind, readonly operation: string) { super(`${driver} page driver does not support ${operation}`) }
|
||||
}
|
||||
export interface PageSession { driver: PageDriver; checkpoint(): Promise<void>; markAutomation: boolean }
|
||||
export class Handle { constructor(readonly objectId: string) {} }
|
||||
export function gone(ref: string): Error { return new Error(`${ref} is no longer in the page; inspect again`) }
|
||||
|
||||
export async function resolveElement(session: PageSession, ref: string, node: number): Promise<string> {
|
||||
let handle: string | undefined
|
||||
try { handle = await session.driver.resolve(node) } catch (error) {
|
||||
if (error instanceof PageDriverUnsupportedError) throw error
|
||||
throw gone(ref)
|
||||
}
|
||||
await session.checkpoint()
|
||||
const objectId = (result.object as { objectId?: unknown } | undefined)?.objectId
|
||||
if (typeof objectId !== "string") throw gone(ref)
|
||||
return objectId
|
||||
if (typeof handle !== "string") throw gone(ref)
|
||||
return handle
|
||||
}
|
||||
|
||||
/** Runs a fixed page function on an element; inputs travel as protocol arguments, never as source.
|
||||
* `byValue: false` returns another element handle. */
|
||||
export async function callOn(session: PageSession, ref: string, objectId: string, body: string, args: unknown[], byValue = true): Promise<unknown> {
|
||||
const result = await session.channel.send("Runtime.callFunctionOn", {
|
||||
objectId,
|
||||
functionDeclaration: pageFunction(body, session.markAutomation),
|
||||
arguments: args.map(value => value instanceof Handle ? { objectId: value.objectId } : { value }),
|
||||
returnByValue: byValue,
|
||||
objectGroup: OBJECT_GROUP,
|
||||
})
|
||||
export async function callOn(session: PageSession, ref: string, handle: string, functionId: PageFunctionId, args: Array<PageArgument | Handle>, byValue = true): Promise<unknown> {
|
||||
const result = await session.driver.call(functionId, handle, args.map(value => value instanceof Handle ? { handle: value.objectId } : value))
|
||||
await session.checkpoint()
|
||||
const failure = result.exceptionDetails as { exception?: { description?: unknown }; text?: unknown } | undefined
|
||||
if (failure) throw pageError(ref, String(failure.exception?.description ?? failure.text ?? ""))
|
||||
const value = result.result as { value?: unknown; objectId?: unknown } | undefined
|
||||
if (!byValue) {
|
||||
if (typeof value?.objectId !== "string") throw gone(ref)
|
||||
return value.objectId
|
||||
}
|
||||
return value?.value
|
||||
}
|
||||
|
||||
export async function releaseObjects(session: PageSession): Promise<void> {
|
||||
await session.channel.send("Runtime.releaseObjectGroup", { objectGroup: OBJECT_GROUP }).catch(() => undefined)
|
||||
}
|
||||
|
||||
function pageFunction(body: string, mark: boolean): string {
|
||||
const element = `const element = this.nodeType === Node.ELEMENT_NODE ? this : this.parentElement;
|
||||
if (!(element instanceof Element) || !element.isConnected) throw new Error("codenomad:gone");`
|
||||
return mark
|
||||
? `function (...args) { ${element} globalThis.__codenomadAutomationDepth = (globalThis.__codenomadAutomationDepth || 0) + 1;
|
||||
try { return (() => { ${body} })(); } finally { globalThis.__codenomadAutomationDepth--; } }`
|
||||
: `function (...args) { ${element} ${body} }`
|
||||
if (result.exception !== undefined) throw pageError(ref, result.exception)
|
||||
if (!byValue) { if (typeof result.handle !== "string") throw gone(ref); return result.handle }
|
||||
return result.value
|
||||
}
|
||||
export async function releaseObjects(session: PageSession): Promise<void> { await session.driver.releaseHandles().catch(() => undefined) }
|
||||
|
||||
function pageError(ref: string, description: string): Error {
|
||||
const [code, detail = ""] = /codenomad:([a-z-]+)(?::(.*))?/.exec(description.split("\n", 1)[0])?.slice(1) ?? []
|
||||
|
|
@ -93,91 +92,66 @@ function pageError(ref: string, description: string): Error {
|
|||
}
|
||||
}
|
||||
|
||||
/** Marks trusted input (DevTools mouse/keyboard events) in the CodeNomad window. A lease, not a
|
||||
* counter: if the clearing call is lost with the connection, the mark still expires. */
|
||||
export async function markedInput<T>(session: PageSession, run: (dispatch: (method: string, params: Record<string, unknown>) => Promise<void>) => Promise<T>): Promise<T> {
|
||||
const held = new Map<string, { method: string; params: Record<string, unknown> }>()
|
||||
type InputKind = "pointer" | "wheel" | "keyboard" | "insert-text"
|
||||
async function sendInput(session: PageSession, kind: InputKind, params: Record<string, unknown>): Promise<void> {
|
||||
switch (kind) {
|
||||
case "pointer": return session.driver.pointer(params as unknown as PointerInput)
|
||||
case "wheel": return session.driver.wheel(params as unknown as WheelInput)
|
||||
case "keyboard": return session.driver.keyboard(params as unknown as KeyboardInput)
|
||||
case "insert-text": return session.driver.insertText(params.text as string)
|
||||
}
|
||||
}
|
||||
/** Provenance marking belongs only to the trusted application window. */
|
||||
export async function markedInput<T>(session: PageSession, run: (dispatch: (kind: InputKind, params: Record<string, unknown>) => Promise<void>) => Promise<T>): Promise<T> {
|
||||
const held = new Map<string, { kind: InputKind; params: Record<string, unknown> }>()
|
||||
let point = { x: 0, y: 0 }
|
||||
let attempted = false
|
||||
const dispatch = async (method: string, params: Record<string, unknown>) => {
|
||||
const dispatch = async (kind: InputKind, params: Record<string, unknown>) => {
|
||||
let timestamp: number | undefined
|
||||
if (session.markAutomation) {
|
||||
const result = await session.channel.send("Runtime.evaluate", { expression: MARK_INPUT, returnByValue: true })
|
||||
const stamp = (result.result as { value?: unknown } | undefined)?.value
|
||||
const stamp = (await session.driver.call("mark-input", undefined, [])).value
|
||||
if (typeof stamp !== "number" || !Number.isFinite(stamp) || stamp <= 0) throw new Error("Cannot establish automation input provenance")
|
||||
timestamp = stamp
|
||||
}
|
||||
const mouse = method === "Input.dispatchMouseEvent"
|
||||
const mouse = kind === "pointer" || kind === "wheel"
|
||||
if (mouse && typeof params.x === "number" && typeof params.y === "number") point = { x: params.x, y: params.y }
|
||||
const id = mouse ? `mouse:${params.button}` : `key:${params.code}`
|
||||
const down = mouse ? params.type === "mousePressed" : params.type === "keyDown" || params.type === "rawKeyDown"
|
||||
const up = mouse ? params.type === "mouseReleased" : params.type === "keyUp"
|
||||
if (down) held.set(id, { method, params: { ...params, type: mouse ? "mouseReleased" : "keyUp", buttons: 0, modifiers: 0,
|
||||
if (down) held.set(id, { kind, params: { ...params, type: mouse ? "mouseReleased" : "keyUp", buttons: 0, modifiers: 0,
|
||||
...(timestamp === undefined ? {} : { timestamp }) } })
|
||||
attempted = true
|
||||
await session.channel.send(method, { ...params,
|
||||
...(timestamp === undefined || !["Input.dispatchMouseEvent", "Input.dispatchKeyEvent"].includes(method) ? {} : { timestamp }) })
|
||||
await sendInput(session, kind, { ...params, ...(timestamp === undefined || kind === "insert-text" ? {} : { timestamp }) })
|
||||
if (up) held.delete(id)
|
||||
await session.checkpoint()
|
||||
}
|
||||
await awaitInputAcceptance(session)
|
||||
try {
|
||||
return await run(dispatch)
|
||||
} catch (error) {
|
||||
if (attempted && error instanceof Error && /timed out|timeout/i.test(error.message)) {
|
||||
throw new Error("Input timed out: its outcome is unknown. The page was busy and the input may still be delivered; inspect before retrying.")
|
||||
}
|
||||
try { return await run(dispatch) }
|
||||
catch (error) {
|
||||
if (attempted && error instanceof Error && /timed out|timeout/i.test(error.message)) throw new Error("Input timed out: its outcome is unknown. The page was busy and the input may still be delivered; inspect before retrying.")
|
||||
throw error
|
||||
} finally {
|
||||
// Failure may be after admission but before acknowledgement/checkpoint. Release
|
||||
// anything potentially held, without a failed checkpoint suppressing cleanup.
|
||||
for (const release of [...held.values()].reverse()) {
|
||||
try {
|
||||
// A frozen page can time out this write too. The release retains the
|
||||
// original press's timestamp, so provenance never depends on the timeout.
|
||||
if (session.markAutomation) await session.channel.send("Runtime.evaluate", { expression: MARK_INPUT, returnByValue: true }).catch(() => undefined)
|
||||
if (session.markAutomation) await session.driver.call("mark-input", undefined, []).catch(() => undefined)
|
||||
const { text: _text, unmodifiedText: _unmodified, ...params } = release.params
|
||||
await session.channel.send(release.method, { ...params, ...(release.method === "Input.dispatchMouseEvent" ? point : {}) })
|
||||
} catch { /* Best effort, exact original channel only; never retry the action. */ }
|
||||
await sendInput(session, release.kind, { ...params, ...(release.kind === "pointer" ? point : {}) })
|
||||
} catch { /* Exact original driver, best effort; never replay a press. */ }
|
||||
}
|
||||
if (session.markAutomation) await session.channel.send("Runtime.evaluate", { expression: UNMARK_INPUT, returnByValue: true }).catch(() => undefined)
|
||||
if (session.markAutomation) await session.driver.call("unmark-input", undefined, []).catch(() => undefined)
|
||||
}
|
||||
}
|
||||
|
||||
/** After an HTTP(S) navigation, Chromium's renderer defers compositor commits until the first
|
||||
* (contentful) paint, for at most about 500 ms after rendering starts, and drops key, click and
|
||||
* wheel input meanwhile (Blink paint holding: `LocalFrameView` and
|
||||
* `WidgetInputHandlerManager::DispatchEvent`). A hidden or software-composited page can take that
|
||||
* long to paint, so its first trusted input waits until the page has painted or roughly that limit
|
||||
* has passed, approximated from the moment it became interactive. The page's answer only delays
|
||||
* input, by at most `INPUT_ACCEPTANCE_MAX_WAIT_MS`. */
|
||||
async function awaitInputAcceptance(session: PageSession): Promise<void> {
|
||||
const started = Date.now()
|
||||
for (;;) {
|
||||
const result = await session.channel.send("Runtime.evaluate", { expression: INPUT_ACCEPTANCE_DELAY, returnByValue: true })
|
||||
.catch(() => undefined)
|
||||
const result = await session.driver.call("input-acceptance", undefined, []).catch(error => {
|
||||
if (error instanceof PageDriverUnsupportedError) throw error
|
||||
return undefined
|
||||
})
|
||||
await session.checkpoint()
|
||||
const delay = ((result?.result as { value?: { inputDelayMs?: unknown } } | undefined)?.value)?.inputDelayMs
|
||||
const remaining = INPUT_ACCEPTANCE_MAX_WAIT_MS - (Date.now() - started)
|
||||
const delay = (result?.value as { inputDelayMs?: unknown } | undefined)?.inputDelayMs
|
||||
const remaining = 1_000 - (Date.now() - started)
|
||||
if (typeof delay !== "number" || !(delay > 0) || remaining <= 0) return
|
||||
await new Promise(resolve => setTimeout(resolve, Math.min(delay, remaining, 50)))
|
||||
}
|
||||
}
|
||||
|
||||
const PAINT_HOLDING_LIMIT_MS = 500
|
||||
const INPUT_ACCEPTANCE_MAX_WAIT_MS = 1_000
|
||||
// Milliseconds until the page accepts input: 0 once it painted, for non-HTTP(S) documents (no
|
||||
// paint holding) and after the limit; a short poll while the document is still loading.
|
||||
const INPUT_ACCEPTANCE_DELAY = `({ inputDelayMs: (() => {
|
||||
if (!/^https?:$/.test(location.protocol) || performance.getEntriesByType("paint").length > 0) return 0;
|
||||
const navigation = performance.getEntriesByType("navigation")[0];
|
||||
if (!navigation || !(navigation.domInteractive > 0)) return 50;
|
||||
return Math.max(0, Math.ceil(navigation.domInteractive + ${PAINT_HOLDING_LIMIT_MS} - performance.now()));
|
||||
})() })`
|
||||
|
||||
export const INPUT_MARK_LEASE_MS = 15_000
|
||||
// CDP timestamps use epoch seconds; DOM event.timeStamp uses the page's monotonic
|
||||
// origin. Reading both in the page keeps delayed input within its original interval.
|
||||
const MARK_INPUT = `globalThis.__codenomadAutomationInputUntil = performance.now() + ${INPUT_MARK_LEASE_MS}; Date.now() / 1000`
|
||||
// Keep the interval's end for events queued before a stalled renderer recovered.
|
||||
const UNMARK_INPUT = "globalThis.__codenomadAutomationInputUntil = Math.min(globalThis.__codenomadAutomationInputUntil || 0, performance.now()); true"
|
||||
|
|
|
|||
15
packages/server/src/automation/page/semantic-tree.ts
Normal file
15
packages/server/src/automation/page/semantic-tree.ts
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
import type { SemanticNode } from "./protocol"
|
||||
import type { AxNode } from "./snapshot"
|
||||
|
||||
/** CDP fields end here; stable numeric node identities remain shared with RefTable. */
|
||||
export function semanticTree(nodes: readonly AxNode[]): SemanticNode[] {
|
||||
return nodes.map(node => ({ id: node.nodeId, parent: node.parentId, children: node.childIds, ignored: node.ignored,
|
||||
node: node.backendDOMNodeId, role: node.role?.value, name: node.name?.value, description: node.description?.value,
|
||||
value: node.value?.value, properties: node.properties?.map(property => ({ name: property.name, value: property.value?.value })) }))
|
||||
}
|
||||
/** Keep the established snapshot renderer and its budgets unchanged. */
|
||||
export function snapshotTree(nodes: readonly SemanticNode[]): AxNode[] {
|
||||
return nodes.map(node => ({ nodeId: node.id, parentId: node.parent, childIds: node.children, ignored: node.ignored,
|
||||
backendDOMNodeId: node.node, role: { value: node.role }, name: { value: node.name }, description: { value: node.description },
|
||||
value: { value: node.value }, properties: node.properties?.map(property => ({ name: property.name, value: { value: property.value } })) }))
|
||||
}
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
MIT License
|
||||
|
||||
Copyright (c) 2020 Sebastian Silbermann
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
130
packages/server/src/automation/page/webkit/calls.generated.ts
Normal file
130
packages/server/src/automation/page/webkit/calls.generated.ts
Normal file
|
|
@ -0,0 +1,130 @@
|
|||
// Generated by scripts/build-webkit-read.ts.
|
||||
// @ts-nocheck
|
||||
export const readCalls = {
|
||||
"option-click": function (...args) { const element=this.nodeType===1?this:this.parentElement; if (!element || !element.isConnected) throw new Error("codenomad:gone"); const document=element.ownerDocument; const { HTMLInputElement, HTMLTextAreaElement, HTMLLabelElement, HTMLSelectElement, HTMLElement, SVGElement, Node, Event, MouseEvent } = document.defaultView; const getSelection=()=>document.defaultView.getSelection(); const label = option => (option.label || option.textContent || "").replace(/\s+/g, " ").trim();
|
||||
const apply = (select, picked) => {
|
||||
if (select.matches(":disabled")) throw new Error("codenomad:disabled");
|
||||
for (const option of picked) if (option.matches(":disabled")) throw new Error("codenomad:option-disabled:" + label(option));
|
||||
if (!select.multiple && picked.length !== 1) throw new Error("codenomad:single");
|
||||
select.focus();
|
||||
const options = Array.from(select.options);
|
||||
if (options.some(option => option.selected !== picked.includes(option))) {
|
||||
for (const option of options) option.selected = picked.includes(option);
|
||||
select.dispatchEvent(new Event("input", { bubbles: true, composed: true }));
|
||||
select.dispatchEvent(new Event("change", { bubbles: true }));
|
||||
}
|
||||
return Array.from(select.selectedOptions).map(label);
|
||||
};
|
||||
const choose = (select, wanted) => {
|
||||
if (select.matches(":disabled")) throw new Error("codenomad:disabled");
|
||||
const options = Array.from(select.options), picked = [];
|
||||
for (const raw of wanted) {
|
||||
const target = String(raw).replace(/\s+/g, " ").trim();
|
||||
const option = options.find(item => label(item) === target) || options.find(item => item.value === String(raw));
|
||||
if (!option) throw new Error("codenomad:no-option:" + target);
|
||||
if (!picked.includes(option)) picked.push(option);
|
||||
}
|
||||
return apply(select, picked);
|
||||
};
|
||||
const option = element.closest("option"), select = option && option.closest("select");
|
||||
if (option && select) return apply(select, [option]);
|
||||
if (element instanceof HTMLSelectElement) throw new Error("codenomad:is-select");
|
||||
return null; },
|
||||
"programmatic-click": function (...args) { const element=this.nodeType===1?this:this.parentElement; if (!element || !element.isConnected) throw new Error("codenomad:gone"); const document=element.ownerDocument; const { HTMLInputElement, HTMLTextAreaElement, HTMLLabelElement, HTMLSelectElement, HTMLElement, SVGElement, Node, Event, MouseEvent } = document.defaultView; const getSelection=()=>document.defaultView.getSelection(); if (element instanceof HTMLElement) element.click();
|
||||
else element.dispatchEvent(new MouseEvent("click", { bubbles: true, cancelable: true, composed: true }));
|
||||
return null; },
|
||||
"select": function (...args) { const element=this.nodeType===1?this:this.parentElement; if (!element || !element.isConnected) throw new Error("codenomad:gone"); const document=element.ownerDocument; const { HTMLInputElement, HTMLTextAreaElement, HTMLLabelElement, HTMLSelectElement, HTMLElement, SVGElement, Node, Event, MouseEvent } = document.defaultView; const getSelection=()=>document.defaultView.getSelection(); const label = option => (option.label || option.textContent || "").replace(/\s+/g, " ").trim();
|
||||
const apply = (select, picked) => {
|
||||
if (select.matches(":disabled")) throw new Error("codenomad:disabled");
|
||||
for (const option of picked) if (option.matches(":disabled")) throw new Error("codenomad:option-disabled:" + label(option));
|
||||
if (!select.multiple && picked.length !== 1) throw new Error("codenomad:single");
|
||||
select.focus();
|
||||
const options = Array.from(select.options);
|
||||
if (options.some(option => option.selected !== picked.includes(option))) {
|
||||
for (const option of options) option.selected = picked.includes(option);
|
||||
select.dispatchEvent(new Event("input", { bubbles: true, composed: true }));
|
||||
select.dispatchEvent(new Event("change", { bubbles: true }));
|
||||
}
|
||||
return Array.from(select.selectedOptions).map(label);
|
||||
};
|
||||
const choose = (select, wanted) => {
|
||||
if (select.matches(":disabled")) throw new Error("codenomad:disabled");
|
||||
const options = Array.from(select.options), picked = [];
|
||||
for (const raw of wanted) {
|
||||
const target = String(raw).replace(/\s+/g, " ").trim();
|
||||
const option = options.find(item => label(item) === target) || options.find(item => item.value === String(raw));
|
||||
if (!option) throw new Error("codenomad:no-option:" + target);
|
||||
if (!picked.includes(option)) picked.push(option);
|
||||
}
|
||||
return apply(select, picked);
|
||||
};
|
||||
const control = element instanceof HTMLLabelElement ? element.control : element;
|
||||
const select = control instanceof HTMLSelectElement ? control : control && control.closest("select");
|
||||
if (!select) throw new Error("codenomad:not-select");
|
||||
return choose(select, args[0]); },
|
||||
"focus": function (...args) { const element=this.nodeType===1?this:this.parentElement; if (!element || !element.isConnected) throw new Error("codenomad:gone"); const document=element.ownerDocument; const { HTMLInputElement, HTMLTextAreaElement, HTMLLabelElement, HTMLSelectElement, HTMLElement, SVGElement, Node, Event, MouseEvent } = document.defaultView; const getSelection=()=>document.defaultView.getSelection(); if (element.matches(":disabled")) throw new Error("codenomad:disabled");
|
||||
if (element instanceof HTMLElement || element instanceof SVGElement) element.focus();
|
||||
if (element.getRootNode().activeElement !== element) throw new Error("codenomad:unfocusable");
|
||||
return null; },
|
||||
"describe": function (...args) { const element=this.nodeType===1?this:this.parentElement; if (!element || !element.isConnected) throw new Error("codenomad:gone"); const document=element.ownerDocument; const { HTMLInputElement, HTMLTextAreaElement, HTMLLabelElement, HTMLSelectElement, HTMLElement, SVGElement, Node, Event, MouseEvent } = document.defaultView; const getSelection=()=>document.defaultView.getSelection(); const role = element.getAttribute("role") || element.tagName.toLowerCase();
|
||||
const name = (element.getAttribute("aria-label") || element.getAttribute("title") || element.getAttribute("placeholder")
|
||||
|| (element.labels && element.labels[0] && element.labels[0].textContent) || element.textContent || "").replace(/\s+/g, " ").trim().slice(0, 80);
|
||||
return role + (name ? " " + JSON.stringify(name) : ""); },
|
||||
"field": function (...args) { const element=this.nodeType===1?this:this.parentElement; if (!element || !element.isConnected) throw new Error("codenomad:gone"); const document=element.ownerDocument; const { HTMLInputElement, HTMLTextAreaElement, HTMLLabelElement, HTMLSelectElement, HTMLElement, SVGElement, Node, Event, MouseEvent } = document.defaultView; const getSelection=()=>document.defaultView.getSelection(); const textField = item => item instanceof HTMLInputElement || item instanceof HTMLTextAreaElement
|
||||
|| (item instanceof HTMLElement && item.isContentEditable);
|
||||
const control = element instanceof HTMLLabelElement ? element.control : element;
|
||||
const field = control && textField(control) ? control
|
||||
: element.querySelector('input:not([type="hidden"]), textarea, [contenteditable]:not([contenteditable="false"])');
|
||||
if (!field || !textField(field)) throw new Error("codenomad:not-editable");
|
||||
return field; },
|
||||
"prepare": function (...args) { const element=this.nodeType===1?this:this.parentElement; if (!element || !element.isConnected) throw new Error("codenomad:gone"); const document=element.ownerDocument; const { HTMLInputElement, HTMLTextAreaElement, HTMLLabelElement, HTMLSelectElement, HTMLElement, SVGElement, Node, Event, MouseEvent } = document.defaultView; const getSelection=()=>document.defaultView.getSelection(); const field = element, clear = args[0];
|
||||
if (field.matches(":disabled")) throw new Error("codenomad:disabled");
|
||||
if (field.readOnly) throw new Error("codenomad:readonly");
|
||||
if (field instanceof HTMLInputElement && ["checkbox","radio","button","submit","reset","file","image","hidden"].includes(field.type)) throw new Error("codenomad:not-text");
|
||||
field.focus();
|
||||
const active = field.getRootNode().activeElement;
|
||||
const editable = !(field instanceof HTMLInputElement || field instanceof HTMLTextAreaElement);
|
||||
if (active !== field && !(editable && active instanceof HTMLElement && active.isContentEditable && active.contains(field))) {
|
||||
throw new Error("codenomad:unfocusable");
|
||||
}
|
||||
if (field instanceof HTMLInputElement && ["date","time","datetime-local","month","week","range","color"].includes(field.type)) return "value";
|
||||
if (editable) {
|
||||
const selection = getSelection();
|
||||
selection.setBaseAndExtent(field, clear ? 0 : field.childNodes.length, field, field.childNodes.length);
|
||||
} else if (clear) field.select();
|
||||
// Email and number inputs have no caret API: their text is appended as a value.
|
||||
else { try { field.setSelectionRange(field.value.length, field.value.length); } catch { return "append"; } }
|
||||
return "insert"; },
|
||||
"set-value": function (...args) { const element=this.nodeType===1?this:this.parentElement; if (!element || !element.isConnected) throw new Error("codenomad:gone"); const document=element.ownerDocument; const { HTMLInputElement, HTMLTextAreaElement, HTMLLabelElement, HTMLSelectElement, HTMLElement, SVGElement, Node, Event, MouseEvent } = document.defaultView; const getSelection=()=>document.defaultView.getSelection(); if (element.matches(":disabled")) throw new Error("codenomad:disabled");
|
||||
if (element.readOnly) throw new Error("codenomad:readonly");
|
||||
const [text, append] = args, next = append ? element.value + text : text;
|
||||
// Ranges and colors have no empty value: the browser would substitute its default.
|
||||
const valid = element.type === "color" ? /^#[0-9a-f]{6}$/i.test(next)
|
||||
: element.type === "range" ? /^-?(?:\d+(?:\.\d+)?|\.\d+)(?:[eE][+-]?\d+)?$/.test(next)
|
||||
: next !== "" && ["date","time","datetime-local","month","week","number"].includes(element.type) ? (() => { const probe = document.createElement("input"); probe.type = element.type; probe.value = next; return probe.value !== ""; })()
|
||||
: true;
|
||||
if (!valid) throw new Error("codenomad:invalid-value:" + next);
|
||||
if (next === element.value) return null;
|
||||
Object.getOwnPropertyDescriptor(HTMLInputElement.prototype, "value").set.call(element, next);
|
||||
element.dispatchEvent(new Event("input", { bubbles: true, composed: true }));
|
||||
element.dispatchEvent(new Event("change", { bubbles: true }));
|
||||
return null; },
|
||||
"delete-selection": function (...args) { const element=this.nodeType===1?this:this.parentElement; if (!element || !element.isConnected) throw new Error("codenomad:gone"); const document=element.ownerDocument; const { HTMLInputElement, HTMLTextAreaElement, HTMLLabelElement, HTMLSelectElement, HTMLElement, SVGElement, Node, Event, MouseEvent } = document.defaultView; const getSelection=()=>document.defaultView.getSelection(); document.execCommand("delete"); return null; },
|
||||
"read-value": function (...args) { const element=this.nodeType===1?this:this.parentElement; if (!element || !element.isConnected) throw new Error("codenomad:gone"); const document=element.ownerDocument; const { HTMLInputElement, HTMLTextAreaElement, HTMLLabelElement, HTMLSelectElement, HTMLElement, SVGElement, Node, Event, MouseEvent } = document.defaultView; const getSelection=()=>document.defaultView.getSelection(); const value = element instanceof HTMLInputElement || element instanceof HTMLTextAreaElement ? element.value : element.innerText || element.textContent || "";
|
||||
if (element instanceof HTMLInputElement && element.type === "password") return "•".repeat(Math.min(value.length, 16));
|
||||
return value.length > 200 ? value.slice(0, 199) + "…" : value; },
|
||||
"receives": function (...args) { const element=this.nodeType===1?this:this.parentElement; if (!element || !element.isConnected) throw new Error("codenomad:gone"); const document=element.ownerDocument; const { HTMLInputElement, HTMLTextAreaElement, HTMLLabelElement, HTMLSelectElement, HTMLElement, SVGElement, Node, Event, MouseEvent } = document.defaultView; const getSelection=()=>document.defaultView.getSelection(); const hit = args[0] && (args[0].nodeType === Node.ELEMENT_NODE ? args[0] : args[0].parentElement);
|
||||
if (!hit || element === hit || element.contains(hit)) return null;
|
||||
const label = hit.closest("label");
|
||||
if (label && (label.control === element || label.contains(element))) return null;
|
||||
const name = (hit.getAttribute("aria-label") || hit.getAttribute("title") || hit.innerText || "").replace(/\s+/g, " ").trim().slice(0, 60);
|
||||
return hit.tagName.toLowerCase() + (name ? " " + JSON.stringify(name) : ""); },
|
||||
"contains": function (...args) { const element=this.nodeType===1?this:this.parentElement; if (!element || !element.isConnected) throw new Error("codenomad:gone"); const document=element.ownerDocument; const { HTMLInputElement, HTMLTextAreaElement, HTMLLabelElement, HTMLSelectElement, HTMLElement, SVGElement, Node, Event, MouseEvent } = document.defaultView; const getSelection=()=>document.defaultView.getSelection(); const needle = String(args[0]).replace(/\s+/g, " ").trim().toLowerCase();
|
||||
const root = element;
|
||||
const text = value => String(value || "").replace(/\s+/g, " ").toLowerCase();
|
||||
if (text(root.innerText || root.textContent).includes(needle)) return true;
|
||||
for (const item of root.querySelectorAll("[aria-label], [title], [placeholder], input, textarea")) {
|
||||
if ([item.getAttribute("aria-label"), item.getAttribute("title"), item.getAttribute("placeholder"), item.value].some(value => text(value).includes(needle))) return true;
|
||||
}
|
||||
return false; }
|
||||
};
|
||||
57
packages/server/src/automation/page/webkit/geometry.ts
Normal file
57
packages/server/src/automation/page/webkit/geometry.ts
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
import { shadowRoot } from "./shadow"
|
||||
type Roots=WeakMap<Element,WeakRef<ShadowRoot>>
|
||||
const element=(node:Node)=>node.nodeType === 1 ? node as Element : node.parentElement
|
||||
function frame(doc:Document): HTMLIFrameElement | undefined { return doc === document ? undefined : doc.defaultView?.frameElement as HTMLIFrameElement ?? undefined }
|
||||
export function geometry(node:Node): number[][] {
|
||||
const item=element(node)
|
||||
if(!item) return []
|
||||
return Array.from(item.getClientRects()).filter(rect=>rect.width>0 && rect.height>0).flatMap(rect=>{
|
||||
let left=rect.left,top=rect.top,right=rect.right,bottom=rect.bottom,doc=item.ownerDocument
|
||||
for(let owner=frame(doc);owner;owner=frame(doc)) {
|
||||
const box=owner.getBoundingClientRect(),sx=box.width/owner.offsetWidth,sy=box.height/owner.offsetHeight
|
||||
if(Math.min(owner.clientWidth,right)<=Math.max(0,left) || Math.min(owner.clientHeight,bottom)<=Math.max(0,top)) return []
|
||||
left=box.left+(owner.clientLeft+Math.max(0,left))*sx
|
||||
right=box.left+(owner.clientLeft+Math.min(owner.clientWidth,right))*sx
|
||||
top=box.top+(owner.clientTop+Math.max(0,top))*sy
|
||||
bottom=box.top+(owner.clientTop+Math.min(owner.clientHeight,bottom))*sy
|
||||
doc=owner.ownerDocument
|
||||
}
|
||||
return [[left,top,right,top,right,bottom,left,bottom]]
|
||||
})
|
||||
}
|
||||
export function scrollIntoView(node:Node): void {
|
||||
const item=element(node)
|
||||
if(!item) return
|
||||
item.scrollIntoView({block:"center",inline:"center",behavior:"instant"})
|
||||
for(let doc=item.ownerDocument,owner=frame(doc);owner;owner=frame(doc)) {owner.scrollIntoView({block:"nearest",inline:"nearest",behavior:"instant"});doc=owner.ownerDocument}
|
||||
}
|
||||
export function hitTest(x:number,y:number,roots:Roots): Element | undefined {
|
||||
let root:Document|ShadowRoot=document,hit:Element|null=null
|
||||
const seen=new Set<Element>()
|
||||
for(;;) {
|
||||
const item=root.elementFromPoint(x,y)
|
||||
if(!item || seen.has(item)) break
|
||||
hit=item;seen.add(item)
|
||||
const shadow=shadowRoot(item,roots)
|
||||
if(shadow) {root=shadow;continue}
|
||||
if(item.localName === "iframe" || item.localName === "frame") {
|
||||
try {
|
||||
const owner=item as HTMLIFrameElement,doc=owner.contentDocument
|
||||
if(doc) {const box=owner.getBoundingClientRect();x=(x-box.left)*owner.offsetWidth/box.width-owner.clientLeft;y=(y-box.top)*owner.offsetHeight/box.height-owner.clientTop;root=doc;continue}
|
||||
} catch { /* opaque frame */ }
|
||||
}
|
||||
break
|
||||
}
|
||||
return hit ?? undefined
|
||||
}
|
||||
export function activeElement(roots:Roots): Element | null {
|
||||
let item=document.activeElement
|
||||
const seen=new Set<Element>()
|
||||
while(item && !seen.has(item)) {
|
||||
seen.add(item)
|
||||
const nested=shadowRoot(item,roots)?.activeElement ?? ((item as HTMLIFrameElement).contentDocument?.activeElement)
|
||||
if(!nested || nested === item) break
|
||||
item=nested
|
||||
}
|
||||
return item
|
||||
}
|
||||
30
packages/server/src/automation/page/webkit/hook.ts
Normal file
30
packages/server/src/automation/page/webkit/hook.ts
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
/** Fixed weak capture bridge; identity, traversal and handles stay isolated. */
|
||||
export const CLOSED_SHADOW_HOOK = `(() => {
|
||||
const original = Element.prototype.attachShadow;
|
||||
const apply = Reflect.apply;
|
||||
const setAttribute = Element.prototype.setAttribute;
|
||||
const roots = new WeakMap();
|
||||
const Reply = PointerEvent;
|
||||
const dispatch = EventTarget.prototype.dispatchEvent;
|
||||
const listen = EventTarget.prototype.addEventListener;
|
||||
const reply = function (host) {
|
||||
const root = roots.get(host);
|
||||
if (root) {
|
||||
// The nested carrier is never dispatched, preserving its native Node
|
||||
// reference without shadow retargeting or cross-world JS serialization.
|
||||
const carrier = new Reply("__codenomad_reference_v1", {relatedTarget:root});
|
||||
apply(dispatch,host,[new Reply("__codenomad_shadow_v1",{predictedEvents:[carrier]})]);
|
||||
}
|
||||
};
|
||||
const wrapped = function (options) {
|
||||
const root = apply(original, this, [options]);
|
||||
if (root.mode === "closed") {
|
||||
const host = this;
|
||||
roots.set(host, root);
|
||||
apply(listen,host,["__codenomad_request_shadow_v1",()=>reply(host)]);
|
||||
apply(setAttribute, this, ["data-codenomad-shadow-boundary", "closed"]);
|
||||
}
|
||||
return root;
|
||||
};
|
||||
Object.defineProperty(Element.prototype, "attachShadow", { ...Object.getOwnPropertyDescriptor(Element.prototype, "attachShadow"), value: wrapped });
|
||||
})()`
|
||||
26
packages/server/src/automation/page/webkit/read.generated.js
Normal file
26
packages/server/src/automation/page/webkit/read.generated.js
Normal file
File diff suppressed because one or more lines are too long
250
packages/server/src/automation/page/webkit/read.ts
Normal file
250
packages/server/src/automation/page/webkit/read.ts
Normal file
|
|
@ -0,0 +1,250 @@
|
|||
import { computeAccessibleName, computeAccessibleDescription, getRole, isInaccessible } from "dom-accessibility-api"
|
||||
import { readCalls } from "./calls.generated"
|
||||
import type { SemanticNode, PageOperation } from "../protocol"
|
||||
import { shadowRoot } from "./shadow"
|
||||
import { geometry, scrollIntoView, hitTest, activeElement } from "./geometry"
|
||||
|
||||
type State = {
|
||||
document: Document; token: string; next: number; ids: WeakMap<Node,number>; nodes: Map<number,WeakRef<Node>>
|
||||
observers: Set<{observer: MutationObserver; root: WeakRef<Node>}>; observed: WeakSet<Node>
|
||||
roots: WeakMap<Element,WeakRef<ShadowRoot>>
|
||||
hides: WeakMap<Document,()=>void>
|
||||
}
|
||||
const MAX_NODES = 20_000
|
||||
const KEY = "__codenomadWebkitReadV1"
|
||||
const token = () => `${Date.now().toString(36)}-${Math.random().toString(36).slice(2)}`
|
||||
const globalState = globalThis as typeof globalThis & { [KEY]?: State }
|
||||
function makeState(): State { return {document,token:token(),next:0,ids:new WeakMap(),nodes:new Map(),observers:new Set(),observed:new WeakSet(),roots:new WeakMap(),hides:new WeakMap()} }
|
||||
function parent(node: Node): Node | null { return node.parentNode ?? (node.nodeType === 11 ? (node as ShadowRoot).host : null) }
|
||||
function live(node: Node): boolean {
|
||||
const doc = node.nodeType === 9 ? node as Document : node.ownerDocument
|
||||
if (!doc || (node.nodeType !== 9 && !node.isConnected)) return false
|
||||
if (doc === document) return true
|
||||
try { const frame = doc.defaultView?.frameElement as HTMLIFrameElement | null; return !!frame && frame.contentDocument === doc && live(frame) } catch { return false }
|
||||
}
|
||||
function retire(state: State, node: Node) {
|
||||
const id = state.ids.get(node)
|
||||
if (id !== undefined) { state.nodes.delete(id); state.ids.delete(node) }
|
||||
for (const child of Array.from(node.childNodes)) retire(state, child)
|
||||
if (node.nodeType === 1) { const root=shadowRoot(node as Element,state.roots); if(root) retire(state,root) }
|
||||
if (node.nodeType === 1 && ["iframe","frame"].includes((node as Element).localName)) {
|
||||
try { const doc=(node as HTMLIFrameElement).contentDocument; if(doc) retire(state,doc) } catch { /* opaque frame */ }
|
||||
}
|
||||
}
|
||||
function watchDocument(state:State,doc:Document) {
|
||||
let hide=state.hides.get(doc)
|
||||
if(!hide) {hide=()=>{if(doc===state.document)state.token=token();else retire(state,doc)};state.hides.set(doc,hide)}
|
||||
doc.defaultView?.addEventListener('pagehide',hide)
|
||||
}
|
||||
function watch(state: State, root: Node) {
|
||||
if(root.nodeType===9) watchDocument(state,root as Document)
|
||||
if (state.observed.has(root)) return
|
||||
state.observed.add(root)
|
||||
const observer = new MutationObserver(records => retireRecords(state,records))
|
||||
observer.observe(root,{childList:true,subtree:true})
|
||||
state.observers.add({observer,root:new WeakRef(root)})
|
||||
}
|
||||
function retireRecords(state: State, records: MutationRecord[]) {
|
||||
for (const record of records) for (const removed of Array.from(record.removedNodes)) {
|
||||
if (removed.nodeType === 1 && (removed as Element).localName === "html" && record.target === document) {
|
||||
for (const {observer} of state.observers) observer.disconnect()
|
||||
Object.assign(state,makeState(),{hides:state.hides})
|
||||
watch(state,document)
|
||||
return
|
||||
}
|
||||
retire(state,removed)
|
||||
if (removed.nodeType === 1 && (removed as Element).localName === "html" && record.target.nodeType === 9) retire(state,record.target)
|
||||
}
|
||||
}
|
||||
function flush(state: State) {
|
||||
for (const entry of state.observers) {
|
||||
retireRecords(state,entry.observer.takeRecords())
|
||||
const root=entry.root.deref()
|
||||
if(root?.nodeType===9 && live(root)) watchDocument(state,root as Document)
|
||||
if (!root || !live(root)) {entry.observer.disconnect();state.observers.delete(entry);if(root)state.observed.delete(root)}
|
||||
}
|
||||
for (const [id, reference] of state.nodes) { const node = reference.deref(); if (!node || !live(node)) { if (node) state.ids.delete(node); state.nodes.delete(id) } }
|
||||
}
|
||||
function id(state: State,node: Node): number {
|
||||
const existing=state.ids.get(node)
|
||||
if (existing !== undefined) return existing
|
||||
if (state.nodes.size >= MAX_NODES) throw new Error("WebKit semantic identity table exceeds 20000 nodes")
|
||||
const value=++state.next; state.ids.set(node,value); state.nodes.set(value,new WeakRef(node)); return value
|
||||
}
|
||||
function lookup(state: State,value: number): Node | undefined { const node=state.nodes.get(value)?.deref(); return node && live(node) && state.ids.get(node) === value ? node : undefined }
|
||||
function handle(state: State,node: Node): string { return `${state.token}:${id(state,node)}` }
|
||||
function fromHandle(state: State,value: string): Node | undefined { const prefix=`${state.token}:`; return value.startsWith(prefix) ? lookup(state,Number(value.slice(prefix.length))) : undefined }
|
||||
function hidden(element: Element): boolean {
|
||||
if (isInaccessible(element)) return true
|
||||
for (let node: Node | null=element; node; node=parent(node)) {
|
||||
if (node.nodeType !== 1) continue
|
||||
const item=node as Element
|
||||
if (item.ownerDocument.defaultView!.getComputedStyle(item).display === "none") return true
|
||||
if (item.hasAttribute("inert") || item.getAttribute("aria-hidden") === "true") return true
|
||||
if (item.localName === "details" && !item.hasAttribute("open")) {
|
||||
const summary=Array.from(item.children).find(child=>child.localName === "summary")
|
||||
if (element !== item && (!summary || (element !== summary && !summary.contains(element)))) return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
const collapse=(text: string)=>text.replace(/\s+/g," ").trim()
|
||||
function composedText(root: Node,state:State): string {
|
||||
if (root.nodeType === 3) return root.textContent ?? ""
|
||||
if (root.nodeType === 1 && hidden(root as Element)) return ""
|
||||
const element=root.nodeType === 1 ? root as Element : undefined
|
||||
if (element?.localName === "slot") { const nodes=(element as HTMLSlotElement).assignedNodes({flatten:true}); if (nodes.length) return nodes.map(node=>composedText(node,state)).join(" ") }
|
||||
return Array.from((element && shadowRoot(element,state.roots))?.childNodes ?? root.childNodes).map(node=>composedText(node,state)).join(" ")
|
||||
}
|
||||
function role(element: Element): string {
|
||||
const tag=element.localName, type=(element as HTMLInputElement).type
|
||||
if (element.getAttribute("role")?.trim()) return getRole(element) ?? "generic"
|
||||
if (tag === "input") {
|
||||
if (type === "password") return "textbox"
|
||||
if (type === "date" || type === "month" || type === "week") return "Date"
|
||||
if (type === "datetime-local") return "DateTime"
|
||||
if (type === "time") return "InputTime"
|
||||
if (type === "color") return "ColorWell"
|
||||
}
|
||||
if (tag === "summary") return "DisclosureTriangle"
|
||||
if (tag === "iframe" || tag === "frame") return "Iframe"
|
||||
let value=getRole(element) ?? "generic"
|
||||
if (["html","body","label","legend"].includes(tag)) value=tag === "label" ? "LabelText" : "generic"
|
||||
if (value === "region" && !computeAccessibleName(element)) value="generic"
|
||||
if ((tag === "header" || tag === "footer") && element.parentElement?.closest("article,aside,main,nav,section")) value="generic"
|
||||
return value
|
||||
}
|
||||
function ariaDisabled(element: Element): boolean {
|
||||
for (let node: Node | null=element; node; node=parent(node)) {
|
||||
if (node.nodeType === 1 && (node as Element).getAttribute("aria-disabled") === "true") return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
function semanticTree(state: State): SemanticNode[] {
|
||||
const nodes: SemanticNode[]=[], seen=new Set<Node>()
|
||||
const add=(node: Node, parentId?: string): SemanticNode => {
|
||||
const key=id(state,node), result: SemanticNode={id:String(key),node:key,...(parentId ? {parent:parentId}:{}),children:[]}
|
||||
nodes.push(result); return result
|
||||
}
|
||||
const boundary=(owner: SemanticNode,name: string,description: string) => {
|
||||
const key=`boundary-${owner.id}`
|
||||
owner.children!.push(key)
|
||||
nodes.push({id:key,parent:owner.id,role:"note",name,description})
|
||||
}
|
||||
const visit=(node: Node,parentId?: string): SemanticNode | undefined => {
|
||||
if (seen.has(node)) return
|
||||
seen.add(node)
|
||||
if (node.nodeType === 3) {
|
||||
if (!collapse(node.textContent ?? "")) return
|
||||
const result=add(node,parentId); result.role="StaticText"; result.name=node.textContent ?? ""; return result
|
||||
}
|
||||
if (node.nodeType === 9) {
|
||||
const doc=node as Document; watch(state,doc)
|
||||
const result=add(node,parentId); result.role="RootWebArea"; result.name=doc.title
|
||||
if (doc.body) { const child=visit(doc.body,result.id); if (child) result.children!.push(child.id!) }
|
||||
return result
|
||||
}
|
||||
if (node.nodeType !== 1) return
|
||||
const element=node as Element
|
||||
if (["script","style","template","noscript","head"].includes(element.localName) || hidden(element)) return
|
||||
const result=add(node,parentId), currentRole=role(element)
|
||||
result.role=currentRole
|
||||
let name=computeAccessibleName(element)
|
||||
const shadow=shadowRoot(element,state.roots)
|
||||
if (!name && shadow && ["button","link","checkbox","tab","heading"].includes(currentRole)) name=collapse(composedText(element,state))
|
||||
result.name=name
|
||||
result.description=computeAccessibleDescription(element)
|
||||
const properties: NonNullable<SemanticNode['properties']>=[]
|
||||
const property=(name: string,value: unknown)=>properties.push({name,value})
|
||||
const attr=(name: string)=>element.getAttribute(`aria-${name}`)
|
||||
const input=element as HTMLInputElement, tag=element.localName
|
||||
if (["checkbox","radio","switch","menuitemcheckbox","menuitemradio"].includes(currentRole)) property("checked", attr("checked") ?? (input.indeterminate ? "mixed" : !!input.checked))
|
||||
if (attr("pressed") !== null) property("pressed",attr("pressed"))
|
||||
if (attr("expanded") !== null) property("expanded",attr("expanded") === "true")
|
||||
else if (tag === "select" && !(element as HTMLSelectElement).multiple && (element as HTMLSelectElement).size <= 1) property("expanded",false)
|
||||
else if (tag === "summary" && element.parentElement?.localName === "details") property("expanded",element.parentElement.hasAttribute("open"))
|
||||
if (attr("selected") !== null || tag === "option") property("selected",attr("selected") !== null ? attr("selected") === "true" : (element as HTMLOptionElement).selected)
|
||||
if (element.matches(":disabled") || ariaDisabled(element)) property("disabled",true)
|
||||
if (input.required || attr("required") === "true") property("required",true)
|
||||
if (input.readOnly || attr("readonly") === "true") property("readonly",true)
|
||||
if (tag === "select" && (element as HTMLSelectElement).multiple || attr("multiselectable") === "true") property("multiselectable",true)
|
||||
if ((element as HTMLElement).tabIndex >= 0) property("focusable",true)
|
||||
for (const state of ["busy","modal"]) if(attr(state) === "true") property(state,true)
|
||||
const root=element.getRootNode()
|
||||
if (element.ownerDocument.activeElement === element || root.nodeType === 11 && (root as ShadowRoot).activeElement === element) property("focused",true)
|
||||
if (attr("invalid") && attr("invalid") !== "false") property("invalid",attr("invalid"))
|
||||
else if (attr("invalid") === null && input.validity && !input.validity.valid) property("invalid","true")
|
||||
if (currentRole === "heading") property("level",Number(attr("level") ?? /^h([1-6])$/.exec(tag)?.[1] ?? 2))
|
||||
if (tag === "a" && element.hasAttribute("href")) property("url",(element as HTMLAnchorElement).href)
|
||||
const editable=(element as HTMLElement).isContentEditable
|
||||
if (editable) property("editable","richtext")
|
||||
if ((tag === "input" && !["checkbox","radio","button","submit","reset","image","file"].includes(input.type)) || tag === "textarea") result.value=input.type === "password" ? "•".repeat(Math.min(input.value.length,16)) : input.value
|
||||
else if (tag === "select" && !(element as HTMLSelectElement).multiple) result.value=Array.from((element as HTMLSelectElement).selectedOptions).map(option=>option.label||option.textContent||"").join(" ")
|
||||
else if (editable) result.value=(element as HTMLElement).innerText
|
||||
else if (attr("valuetext") !== null || attr("valuenow") !== null) result.value=attr("valuetext") ?? attr("valuenow")
|
||||
result.properties=properties
|
||||
if (tag === "iframe" || tag === "frame") {
|
||||
try {
|
||||
const child=(element as HTMLIFrameElement).contentDocument
|
||||
if (!child) throw new Error("cross-origin")
|
||||
const nested=visit(child,result.id); if (nested) result.children!.push(nested.id!)
|
||||
} catch { delete result.node; boundary(result,"Cross-origin frame (unavailable)","Frame contents cannot be traversed") }
|
||||
return result
|
||||
}
|
||||
if (!shadow && element.getAttribute("data-codenomad-shadow-boundary") === "closed") { boundary(result,"Closed shadow root (unavailable)","Closed shadow capture could not be validated"); return result }
|
||||
if (shadow) watch(state,shadow)
|
||||
const children=tag === "slot" && (element as HTMLSlotElement).assignedNodes({flatten:true}).length ? (element as HTMLSlotElement).assignedNodes({flatten:true}) : Array.from(shadow?.childNodes ?? element.childNodes)
|
||||
for (const child of children) { const nested=visit(child,result.id); if (nested) result.children!.push(nested.id!) }
|
||||
return result
|
||||
}
|
||||
visit(document)
|
||||
return nodes
|
||||
}
|
||||
|
||||
export function read(operation: PageOperation): {document: string; result: Record<string,unknown>} {
|
||||
if (globalState[KEY]?.document !== document) {
|
||||
globalState[KEY]=makeState()
|
||||
}
|
||||
const state=globalState[KEY]!
|
||||
watch(state,document); flush(state)
|
||||
let result: Record<string,unknown>
|
||||
switch (operation.op) {
|
||||
case "tree": result={nodes:semanticTree(state)}; break
|
||||
case "checkpoint": result={}; break
|
||||
case "release-handles": result={}; break // All entries are weak; numeric document identity remains stable.
|
||||
case "resolve": { const node=lookup(state,operation.node); result=node ? {handle:handle(state,node)} : {}; break }
|
||||
case "viewport": result={width:innerWidth,height:innerHeight}; break
|
||||
case "geometry": {const node=lookup(state,operation.node);result=node ? {quads:geometry(node)} : {};break}
|
||||
case "scroll-into-view": {const node=lookup(state,operation.node);if(node)scrollIntoView(node);result={};break}
|
||||
case "hit-test": {const node=hitTest(operation.x,operation.y,state.roots);result=node ? {node:id(state,node)} : {};break}
|
||||
case "call": {
|
||||
if (operation.function === "document-root" || operation.function === "active-element") {
|
||||
const node=operation.function === "document-root" ? document.documentElement : activeElement(state.roots)
|
||||
result=node ? {handle:handle(state,node)} : {}; break
|
||||
}
|
||||
if (operation.function === "input-acceptance") { result={value:{inputDelayMs:0}}; break }
|
||||
if (!Object.hasOwn(readCalls,operation.function)) { result={error:{code:"unsupported",driver:"webkit",operation:`call:${operation.function}`}}; break }
|
||||
const node=operation.handle ? fromHandle(state,operation.handle) : undefined
|
||||
if (!node) { result={exception:"codenomad:gone"}; break }
|
||||
if(operation.function === "contains") {
|
||||
const needle=collapse(String(operation.args[0])).toLowerCase()
|
||||
result={value:semanticTree(state).some(item=>{
|
||||
const target=item.node ? lookup(state,item.node) : undefined
|
||||
const element=target?.nodeType === 1 ? target as Element : undefined
|
||||
const attributes=element ? ['aria-label','title','placeholder'].map(name=>element.getAttribute(name)) : []
|
||||
if(element && ['input','textarea'].includes(element.localName)) attributes.push((element as HTMLInputElement).value)
|
||||
return [item.name,item.value,item.description,...attributes].some(value=>typeof value === "string" && collapse(value).toLowerCase().includes(needle))
|
||||
})}
|
||||
break
|
||||
}
|
||||
try {
|
||||
const args=operation.args.map(arg=>arg && typeof arg === "object" && Object.hasOwn(arg,"handle") ? fromHandle(state,(arg as {handle:string}).handle) : arg)
|
||||
const value=readCalls[operation.function as keyof typeof readCalls].call(node,...args)
|
||||
result=operation.function === "field" ? {handle:handle(state,value as Node)} : {value}
|
||||
} catch (error) { result={exception:String(error)} }
|
||||
break
|
||||
}
|
||||
default: result={error:{code:"unsupported",driver:"webkit",operation:operation.op}}
|
||||
}
|
||||
return {document:state.token,result}
|
||||
}
|
||||
31
packages/server/src/automation/page/webkit/shadow.ts
Normal file
31
packages/server/src/automation/page/webkit/shadow.ts
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
/** Page-world responses are hints: native DOM getters enforce brand/host/document. */
|
||||
export function shadowRoot(host: Element, roots: WeakMap<Element, WeakRef<ShadowRoot>>): ShadowRoot | undefined {
|
||||
if (host.shadowRoot) return host.shadowRoot
|
||||
const validate = (candidate: unknown): ShadowRoot | undefined => {
|
||||
try {
|
||||
const get = (prototype: object, key: string, value: unknown) => Object.getOwnPropertyDescriptor(prototype,key)!.get!.call(value)
|
||||
if (get(ShadowRoot.prototype,"host",candidate) !== host || get(ShadowRoot.prototype,"mode",candidate) !== "closed"
|
||||
|| get(Node.prototype,"ownerDocument",candidate) !== host.ownerDocument) return
|
||||
// Rewrap the native object in this isolated realm before consuming it.
|
||||
return Node.prototype.getRootNode.call(candidate) as ShadowRoot
|
||||
} catch { return undefined }
|
||||
}
|
||||
const cached=validate(roots.get(host)?.deref())
|
||||
if(cached) return cached
|
||||
if(host.getAttribute("data-codenomad-shadow-boundary") !== "closed") return
|
||||
let root: ShadowRoot | undefined
|
||||
const receive=(event:Event)=>{
|
||||
try {
|
||||
const carriers=PointerEvent.prototype.getPredictedEvents.call(event)
|
||||
if(carriers.length!==1) return
|
||||
const candidate=Object.getOwnPropertyDescriptor(MouseEvent.prototype,"relatedTarget")!.get!.call(carriers[0])
|
||||
const validated=validate(candidate)
|
||||
if(validated) root=validated
|
||||
} catch { /* Native brands reject arbitrary page replies. */ }
|
||||
}
|
||||
host.addEventListener("__codenomad_shadow_v1",receive)
|
||||
try { host.dispatchEvent(new Event("__codenomad_request_shadow_v1")) }
|
||||
finally { host.removeEventListener("__codenomad_shadow_v1",receive) }
|
||||
if(root) roots.set(host,new WeakRef(root))
|
||||
return root
|
||||
}
|
||||
|
|
@ -1,5 +1,6 @@
|
|||
import { WebSocket } from "undici"
|
||||
import { bindRefs, performAction, snapshotPage, type PageActionResult, type PageSession } from "./automation/page/actions"
|
||||
import { cdpSession } from "./automation/page/cdp-driver"
|
||||
import type { RefAction } from "./automation/page/parse"
|
||||
import { RefTable } from "./automation/page/refs"
|
||||
import { SnapshotScopeError } from "./automation/page/snapshot"
|
||||
|
|
@ -176,11 +177,8 @@ export class DeveloperCdp {
|
|||
}
|
||||
|
||||
private session(state: RunState, epoch: number, message: string): PageSession {
|
||||
return {
|
||||
channel: { send: (method, params) => this.command(state, method, params) },
|
||||
checkpoint: () => this.assertContextCurrent(state, epoch, message),
|
||||
markAutomation: true,
|
||||
}
|
||||
return cdpSession({ send: (method, params) => this.command(state, method, params) },
|
||||
() => this.assertContextCurrent(state, epoch, message), true)
|
||||
}
|
||||
|
||||
async screenshot(identity: DeveloperCdpIdentity): Promise<DeveloperCdpScreenshot> {
|
||||
|
|
|
|||
|
|
@ -2,15 +2,21 @@ import assert from "node:assert/strict"
|
|||
import test from "node:test"
|
||||
import { NativeParent, NATIVE_REQUEST_PREFIX, NATIVE_RESPONSE_PREFIX } from "./native-parent"
|
||||
|
||||
test("a supported native parent takes precedence over an inherited WebKit host marker", () => {
|
||||
test("WebKit's explicit public automation gate remains closed with an available private native transport", () => {
|
||||
const previous = process.env.CODENOMAD_TAURI_WEBKIT
|
||||
const previousCapability = process.env.CODENOMAD_NATIVE_BROWSER_AUTOMATION
|
||||
try {
|
||||
process.env.CODENOMAD_TAURI_WEBKIT = "1"
|
||||
delete process.env.CODENOMAD_NATIVE_BROWSER_AUTOMATION
|
||||
assert.equal(new NativeParent(undefined, true).browserAutomationUnavailable, false)
|
||||
process.env.CODENOMAD_NATIVE_BROWSER_AUTOMATION = "0"
|
||||
assert.equal(new NativeParent(undefined, true).browserAutomationUnavailable, true)
|
||||
assert.equal(new NativeParent(undefined, false).browserAutomationUnavailable, true)
|
||||
} finally {
|
||||
if (previous === undefined) delete process.env.CODENOMAD_TAURI_WEBKIT
|
||||
else process.env.CODENOMAD_TAURI_WEBKIT = previous
|
||||
if (previousCapability === undefined) delete process.env.CODENOMAD_NATIVE_BROWSER_AUTOMATION
|
||||
else process.env.CODENOMAD_NATIVE_BROWSER_AUTOMATION = previousCapability
|
||||
}
|
||||
})
|
||||
|
||||
|
|
@ -34,3 +40,17 @@ test("matches native responses to requests and rejects pending work on shutdown"
|
|||
parent.close()
|
||||
await assert.rejects(pending, /shutting down/)
|
||||
})
|
||||
|
||||
test("late native replies cannot settle a subsequent request", async () => {
|
||||
const lines: string[] = []
|
||||
const parent = new NativeParent({ write: line => { lines.push(String(line)); return true } }, true)
|
||||
const first = parent.request("browser.probe", {}, 5)
|
||||
const old = JSON.parse(lines[0].slice(NATIVE_REQUEST_PREFIX.length))
|
||||
await assert.rejects(first, /timed out/)
|
||||
const second = parent.request("browser.probe", {}, 1000)
|
||||
const current = JSON.parse(lines[1].slice(NATIVE_REQUEST_PREFIX.length))
|
||||
parent.handleLine(`${NATIVE_RESPONSE_PREFIX}${JSON.stringify({ v: 1, id: old.id, ok: true, result: "stale" })}`)
|
||||
parent.handleLine(`${NATIVE_RESPONSE_PREFIX}${JSON.stringify({ v: 1, id: current.id, ok: true, result: "current" })}`)
|
||||
assert.equal(await second, "current")
|
||||
parent.close()
|
||||
})
|
||||
|
|
|
|||
|
|
@ -26,7 +26,8 @@ export class NativeParent {
|
|||
constructor(
|
||||
private readonly output: Pick<NodeJS.WriteStream, "write"> = process.stdout,
|
||||
readonly available = process.env.CODENOMAD_NATIVE_PARENT === "1",
|
||||
readonly browserAutomationUnavailable = !available && process.env.CODENOMAD_TAURI_WEBKIT === "1",
|
||||
readonly browserAutomationUnavailable = process.env.CODENOMAD_NATIVE_BROWSER_AUTOMATION === "0"
|
||||
|| (!available && process.env.CODENOMAD_TAURI_WEBKIT === "1"),
|
||||
) {}
|
||||
|
||||
request<T>(method: string, params: unknown, timeoutMs = DEFAULT_TIMEOUT_MS): Promise<T> {
|
||||
|
|
|
|||
|
|
@ -147,9 +147,9 @@ test("browser works with an unavailable developer target and unrelated inventory
|
|||
available: true,
|
||||
request: async (method: string, params: any) => {
|
||||
nativeCalls.push({ method, params })
|
||||
if (method === "browser.page.begin") return { generation: "preview-document", url: "https://example.com/" }
|
||||
if (method === "browser.page.send" && params.method === "Accessibility.getFullAXTree") {
|
||||
return { nodes: [{ nodeId: "one", backendDOMNodeId: 42, role: { value: "button" }, name: { value: "Preview action" } }] }
|
||||
if (method === "browser.page.begin") return { generation: "preview-document", url: "https://example.com/", driver: "cdp", contractVersion: 1 }
|
||||
if (method === "browser.page.send" && params.operation?.op === "tree") {
|
||||
return { nodes: [{ id: "one", node: 42, role: "button", name: "Preview action" }] }
|
||||
}
|
||||
return method === "developer.status" ? { status: { state: "stopped" } }
|
||||
: method === "browser.probe" ? { available: true } : { url: "https://example.com/" }
|
||||
|
|
|
|||
|
|
@ -104,6 +104,23 @@ describe("CLI shutdown signal registration", () => {
|
|||
}
|
||||
})
|
||||
|
||||
it("preserves UTF-8 native response text split across stdin buffers", async () => {
|
||||
const source = new PassThrough()
|
||||
const lines: string[] = []
|
||||
const parent = new NativeParent({ write: line => { lines.push(String(line)); return true } }, true)
|
||||
installShutdownStdinHandler(source, async () => {}, line => parent.handleLine(line))
|
||||
try {
|
||||
const pending = parent.request("browser.probe", {})
|
||||
const { id } = JSON.parse(lines[0].slice(NATIVE_REQUEST_PREFIX.length))
|
||||
const response = Buffer.from(`${NATIVE_RESPONSE_PREFIX}${JSON.stringify({ v: 1, id, ok: true, result: "é漢🙂" })}\n`)
|
||||
for (const byte of response) source.write(Buffer.from([byte]))
|
||||
assert.equal(await pending, "é漢🙂")
|
||||
} finally {
|
||||
parent.close()
|
||||
source.destroy()
|
||||
}
|
||||
})
|
||||
|
||||
it("allows a real piped process to exit naturally after the shutdown command", { timeout: 5_000 }, async () => {
|
||||
const moduleUrl = new URL("./shutdown.ts", import.meta.url).href
|
||||
const child = spawn(process.execPath, ["--import", "tsx", "--input-type=module", "-e", `
|
||||
|
|
|
|||
|
|
@ -1,3 +1,5 @@
|
|||
import { StringDecoder } from "node:string_decoder"
|
||||
|
||||
type ShutdownLogger = Pick<import("./logger").Logger, "info" | "warn" | "error">
|
||||
type ShutdownOperation = () => void | Promise<void>
|
||||
|
||||
|
|
@ -30,9 +32,10 @@ export function installShutdownStdinHandler(
|
|||
handleLine?: (line: string) => boolean,
|
||||
): void {
|
||||
let buffer = ""
|
||||
const decoder = new StringDecoder("utf8")
|
||||
let requested = false
|
||||
const onData = (chunk: Buffer | string) => {
|
||||
buffer += chunk.toString()
|
||||
buffer += decoder.write(typeof chunk === "string" ? Buffer.from(chunk) : chunk)
|
||||
const lines = buffer.split(/\r?\n/)
|
||||
buffer = lines.pop() ?? ""
|
||||
let shutdownRequested = false
|
||||
|
|
|
|||
|
|
@ -29,13 +29,15 @@ describe("HostOpenCodeService", () => {
|
|||
|
||||
it("never hands the desktop profile scope or backend hand-offs to the shared daemon", () => {
|
||||
const keys = ["CODENOMAD_UPDATE_CHANNEL", "CODENOMAD_PROFILE_CONFIG_IDENTITY",
|
||||
"CODENOMAD_PROFILE", "CLI_CONFIG", "CODENOMAD_NATIVE_PARENT", "ELECTRON_RUN_AS_NODE", "ELECTRON_NO_ASAR", "OPENCODE_DB"]
|
||||
"CODENOMAD_PROFILE", "CLI_CONFIG", "CODENOMAD_NATIVE_PARENT", "CODENOMAD_NATIVE_BROWSER_AUTOMATION", "CODENOMAD_TAURI_WEBKIT", "ELECTRON_RUN_AS_NODE", "ELECTRON_NO_ASAR", "OPENCODE_DB"]
|
||||
const saved = Object.fromEntries(keys.map(key => [key, process.env[key]]))
|
||||
try {
|
||||
process.env.CODENOMAD_UPDATE_CHANNEL = "stable"
|
||||
process.env.CODENOMAD_PROFILE_CONFIG_IDENTITY = "owned-config.yaml"
|
||||
process.env.CLI_CONFIG = "/owned/config.yaml"
|
||||
process.env.CODENOMAD_NATIVE_PARENT = "1"
|
||||
process.env.CODENOMAD_NATIVE_BROWSER_AUTOMATION = "0"
|
||||
process.env.CODENOMAD_TAURI_WEBKIT = "1"
|
||||
process.env.ELECTRON_RUN_AS_NODE = "1"
|
||||
process.env.ELECTRON_NO_ASAR = "1"
|
||||
const environment = daemonProcessEnvironment({ FIXTURE: "kept", CODENOMAD_PROFILE: "team", OPENCODE_DB: "/x.db" })
|
||||
|
|
|
|||
|
|
@ -42,6 +42,8 @@ const DAEMON_EXCLUDED_ENVIRONMENT = [
|
|||
"CODENOMAD_PROFILE_CONFIG_IDENTITY",
|
||||
"CLI_CONFIG",
|
||||
"CODENOMAD_NATIVE_PARENT",
|
||||
"CODENOMAD_NATIVE_BROWSER_AUTOMATION",
|
||||
"CODENOMAD_TAURI_WEBKIT",
|
||||
"ELECTRON_RUN_AS_NODE",
|
||||
"ELECTRON_NO_ASAR",
|
||||
]
|
||||
|
|
|
|||
2
packages/tauri-app/Cargo.lock
generated
2
packages/tauri-app/Cargo.lock
generated
|
|
@ -534,6 +534,7 @@ version = "1.0.1"
|
|||
dependencies = [
|
||||
"anyhow",
|
||||
"base64 0.22.1",
|
||||
"cc",
|
||||
"dirs 5.0.1",
|
||||
"fs2",
|
||||
"gtk",
|
||||
|
|
@ -543,6 +544,7 @@ dependencies = [
|
|||
"objc2-web-kit",
|
||||
"open",
|
||||
"parking_lot",
|
||||
"pkg-config",
|
||||
"plist",
|
||||
"regex",
|
||||
"reqwest 0.12.28",
|
||||
|
|
|
|||
151
packages/tauri-app/scripts/test-browser-parent-webkit.py
Normal file
151
packages/tauri-app/scripts/test-browser-parent-webkit.py
Normal file
|
|
@ -0,0 +1,151 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Production native-parent/controller qualification using an isolated fixture child.
|
||||
|
||||
Build the TypeScript fixture with esbuild first; pass its .mjs path via --fixture.
|
||||
The owning-window inventory and renderer's open-claim callback are fixture-owned.
|
||||
"""
|
||||
import argparse
|
||||
import http.server
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import platform
|
||||
import shutil
|
||||
import ssl
|
||||
import subprocess
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
|
||||
|
||||
class Fixture(http.server.BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
if self.path.startswith('/mobile'):
|
||||
directory = Path(__file__).resolve().parents[1] / 'tests/browser-parent'
|
||||
body = (directory / 'mobile.html').read_text().replace('{{UA}}', json.dumps(self.headers.get('User-Agent','')).replace('<','\\u003c')).replace('{{LOAD}}',json.dumps(uuid.uuid4().hex)).encode('utf-8')
|
||||
self.send_response(200)
|
||||
self.send_header('Content-Type','text/html; charset=utf-8')
|
||||
self.send_header('Content-Length',str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
return
|
||||
if self.path.startswith('/read/'):
|
||||
directory = Path(__file__).resolve().parents[1] / 'tests/browser-parent'
|
||||
name = self.path.split('?',1)[0][len('/read/'):]
|
||||
fixtures = json.loads((directory / 'read-fixtures.generated.json').read_text())
|
||||
if name == 'advanced':
|
||||
body = (directory / 'read-advanced.html').read_text().replace('{{CROSS}}', f'http://localhost:{self.server.server_port}')
|
||||
elif name == 'frame':
|
||||
body = '<title>Child document</title><button onclick="document.body.dataset.clicked=event.isTrusted">Frame action</button><label>Frame field <input value="Child value"></label>'
|
||||
elif name == 'extra':
|
||||
body = (directory / 'action-extra.html').read_text()
|
||||
else:
|
||||
body = fixtures.get(name, '<button>New document</button>')
|
||||
programs = json.loads((directory / 'action-programs.generated.json').read_text())
|
||||
body = (body + '<script>' + programs['auditor'] + '</script>').encode('utf-8')
|
||||
else:
|
||||
body = b'<!doctype html><meta charset="utf-8"><title>Parent channel fixture</title><h1>Native parent fixture</h1><dialog>Overlay</dialog>'
|
||||
if self.path == '/redirect':
|
||||
self.send_response(302)
|
||||
self.send_header('Location', '/ok')
|
||||
self.end_headers()
|
||||
return
|
||||
if self.path == '/slow':
|
||||
time.sleep(.3)
|
||||
self.send_response(200)
|
||||
self.send_header('Content-Type', 'text/html; charset=utf-8')
|
||||
self.send_header('Content-Length', str(len(body)))
|
||||
self.end_headers()
|
||||
try:
|
||||
self.wfile.write(body)
|
||||
except BrokenPipeError:
|
||||
pass
|
||||
|
||||
def log_message(self, *_args):
|
||||
pass
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--fixture', type=Path, required=True)
|
||||
parser.add_argument('--output', type=Path, required=True)
|
||||
parser.add_argument('--target', type=Path, required=True)
|
||||
parser.add_argument('--node', default=shutil.which('node'))
|
||||
parser.add_argument('--nested-kde', action='store_true', help='Own a private KWin Wayland compositor alongside the current desktop')
|
||||
args = parser.parse_args()
|
||||
if not args.node:
|
||||
parser.error('Node executable required: use --node /absolute/path/to/node')
|
||||
root = Path(__file__).resolve().parents[1]
|
||||
output = args.output.resolve()
|
||||
output.mkdir(parents=True, exist_ok=True)
|
||||
env = os.environ.copy()
|
||||
env.update(CARGO_TARGET_DIR=str(args.target.resolve()), CARGO_PROFILE_DEV_DEBUG='0', CARGO_INCREMENTAL='0')
|
||||
if platform.system() == 'Darwin':
|
||||
env['MACOSX_DEPLOYMENT_TARGET'] = '14.0'
|
||||
else:
|
||||
session = subprocess.check_output(['systemctl', '--user', 'show-environment'], text=True)
|
||||
for line in session.splitlines():
|
||||
key, separator, value = line.partition('=')
|
||||
if separator and key in {'DISPLAY', 'WAYLAND_DISPLAY', 'XAUTHORITY', 'XDG_RUNTIME_DIR', 'DBUS_SESSION_BUS_ADDRESS'}:
|
||||
env.setdefault(key, value)
|
||||
with tempfile.TemporaryDirectory(prefix='codenomad-parent-build-') as directory:
|
||||
manifest = Path(directory) / 'Cargo.toml'
|
||||
source = root / 'src-tauri/examples/browser-parent-webkit.rs'
|
||||
native = root / 'src-tauri/src/embedded_view'
|
||||
manifest.write_text(
|
||||
'[package]\nname="codenomad-parent-qualification"\nversion="0.0.0"\nedition="2021"\n'
|
||||
'[dependencies]\ntauri={version="=2.12.5",features=["unstable","test"]}\nserde={version="1",features=["derive"]}\nserde_json="1"\nuuid={version="1",features=["v4"]}\nbase64="0.22"\nsha2="0.10"\n'
|
||||
'[target.\'cfg(target_os="linux")\'.dependencies]\ngtk="0.18"\nwebkit2gtk="=2.0.2"\n'
|
||||
'[target.\'cfg(target_os="macos")\'.dependencies]\nobjc2="0.6.4"\nobjc2-web-kit={version="0.3.2",default-features=false,features=["std","objc2-app-kit","WKWebView","WKBackForwardList","WKBackForwardListItem","WKNavigation"]}\n'
|
||||
'[build-dependencies]\ncc="1"\npkg-config="0.3"\n'
|
||||
f'[[bin]]\nname="browser-parent-webkit"\npath={json.dumps(str(source))}\n'
|
||||
'[patch.crates-io]\n' + ''.join(f'{name}={{path={json.dumps(str(root / "vendor" / folder))}}}\n' for name, folder in [('tao','tao-0.37.2'),('wry','wry-0.57.1'),('tauri-runtime-wry','tauri-runtime-wry-2.12.3')]),
|
||||
encoding='utf-8')
|
||||
(Path(directory) / 'build.rs').write_text(
|
||||
f'#[path={json.dumps(str(root / "src-tauri/build_webkit.rs"))}] mod build_webkit;\nfn main() {{ build_webkit::build(std::path::Path::new({json.dumps(str(native))})); let mut b=cc::Build::new(); b.file({json.dumps(str(root / "tests/browser-parent" / ("mobile-trial-macos.m" if platform.system()=="Darwin" else "mobile-trial-linux.c")))}); '
|
||||
+ ('b.flag("-fobjc-arc").flag("-fblocks").flag("-mmacosx-version-min=14.0");' if platform.system()=='Darwin' else 'b.includes(pkg_config::Config::new().probe("webkit2gtk-4.1").unwrap().include_paths);')
|
||||
+ 'b.compile("mobile_trial"); }\n', encoding='utf-8')
|
||||
shutil.copyfile(root / 'Cargo.lock', Path(directory) / 'Cargo.lock')
|
||||
subprocess.run(['cargo', 'build', '--offline', '--manifest-path', str(manifest)], env=env, check=True, timeout=300)
|
||||
certificate, key = Path(directory) / 'certificate.pem', Path(directory) / 'key.pem'
|
||||
subprocess.run(['openssl', 'req', '-x509', '-newkey', 'rsa:2048', '-nodes', '-days', '1',
|
||||
'-subj', '/CN=localhost', '-keyout', str(key), '-out', str(certificate)],
|
||||
check=True, timeout=20, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
with http.server.ThreadingHTTPServer(('127.0.0.1', 0), Fixture) as server, http.server.ThreadingHTTPServer(('127.0.0.1', 0), Fixture) as tls_server:
|
||||
tls = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
|
||||
tls.load_cert_chain(certificate, key)
|
||||
tls_server.socket = tls.wrap_socket(tls_server.socket, server_side=True)
|
||||
env['CN_FIXTURE_TLS_URL'] = f'https://127.0.0.1:{tls_server.server_port}/ok'
|
||||
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
tls_thread = threading.Thread(target=tls_server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
tls_thread.start()
|
||||
compositor = None
|
||||
try:
|
||||
if args.nested_kde:
|
||||
socket = f'codenomad-parent-{os.getpid()}'
|
||||
env.update(WAYLAND_DISPLAY=socket, GDK_BACKEND='wayland')
|
||||
with (output / 'kwin.log').open('w') as log:
|
||||
compositor = subprocess.Popen(['kwin_wayland', '--virtual', '--socket', socket, '--width', '1024', '--height', '768'],
|
||||
env={**env, 'QT_QPA_PLATFORM': 'offscreen', 'KWIN_COMPOSE': 'Q'}, stdout=log, stderr=subprocess.STDOUT)
|
||||
deadline = time.monotonic() + 15
|
||||
while not (Path(env['XDG_RUNTIME_DIR']) / socket).exists():
|
||||
assert compositor.poll() is None, (output / 'kwin.log').read_text()
|
||||
assert time.monotonic() < deadline, 'KWin socket startup timed out'
|
||||
time.sleep(.1)
|
||||
subprocess.run([str(args.target.resolve() / 'debug/browser-parent-webkit'), args.node,
|
||||
str(args.fixture.resolve()), f'http://127.0.0.1:{server.server_port}', str(output)],
|
||||
env=env, check=True, timeout=150)
|
||||
finally:
|
||||
if compositor:
|
||||
compositor.terminate()
|
||||
compositor.wait(timeout=10)
|
||||
server.shutdown()
|
||||
tls_server.shutdown()
|
||||
thread.join()
|
||||
tls_thread.join()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
119
packages/tauri-app/scripts/test-webkit-input.py
Normal file
119
packages/tauri-app/scripts/test-webkit-input.py
Normal file
|
|
@ -0,0 +1,119 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Compile and exercise the production native primitives on a graphical session.
|
||||
|
||||
Linux: GDK_BACKEND=wayland|x11 python3 packages/tauri-app/scripts/test-webkit-input.py
|
||||
macOS: python3 packages/tauri-app/scripts/test-webkit-input.py
|
||||
Artifacts remain in --output (default: a printed temporary directory).
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import platform
|
||||
import shlex
|
||||
import shutil
|
||||
import struct
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--output', type=Path)
|
||||
parser.add_argument('--nested-kde', action='store_true', help='Own a separate headless KWin Wayland compositor; leave the desktop session intact')
|
||||
args = parser.parse_args()
|
||||
root = Path(__file__).resolve().parents[1]
|
||||
output = args.output or Path(tempfile.mkdtemp(prefix='codenomad-webkit-input-'))
|
||||
output.mkdir(parents=True, exist_ok=True)
|
||||
output = output.resolve()
|
||||
source = root / 'src-tauri/src/embedded_view'
|
||||
test = root / 'tests/webkit-input'
|
||||
env = os.environ.copy()
|
||||
binary = output / 'native-input'
|
||||
if platform.system() == 'Darwin':
|
||||
command = ['clang', '-Wall', '-Wextra', '-Werror', '-fobjc-arc', '-fblocks', '-DCN_INPUT_SELF_TEST',
|
||||
'-mmacosx-version-min=14.0', '-framework', 'AppKit', '-framework', 'WebKit',
|
||||
'-framework', 'ApplicationServices', str(source / 'input_macos.m'), str(test / 'macos.m')]
|
||||
else:
|
||||
session = subprocess.run(['systemctl', '--user', 'show-environment'],
|
||||
check=True, capture_output=True, text=True).stdout
|
||||
for line in session.splitlines():
|
||||
key, separator, value = line.partition('=')
|
||||
if separator and key in {'DISPLAY', 'WAYLAND_DISPLAY', 'XAUTHORITY', 'XDG_RUNTIME_DIR', 'DBUS_SESSION_BUS_ADDRESS'}:
|
||||
env.setdefault(key, value)
|
||||
flags = subprocess.run(['pkg-config', '--cflags', '--libs', 'webkit2gtk-4.1'],
|
||||
check=True, capture_output=True, text=True).stdout
|
||||
command = ['cc', '-Wall', '-Wextra', '-Werror', str(source / 'input_linux.c'), str(test / 'linux.c'), *shlex.split(flags)]
|
||||
subprocess.run([*command, '-o', str(binary)], check=True, timeout=120)
|
||||
print(f'Artifacts: {output}', flush=True)
|
||||
compositor = None
|
||||
desktop_pointer = None
|
||||
if platform.system() == 'Linux' and env.get('DISPLAY') and shutil.which('xdotool'):
|
||||
# Observation only: synthetic input goes exclusively through the native adapter.
|
||||
desktop_pointer = subprocess.check_output(['xdotool', 'getmouselocation', '--shell'], env=env, text=True)
|
||||
try:
|
||||
if args.nested_kde:
|
||||
socket = f'codenomad-input-{os.getpid()}'
|
||||
env['WAYLAND_DISPLAY'] = socket
|
||||
env['GDK_BACKEND'] = 'wayland'
|
||||
env['CN_NESTED_KDE'] = '1'
|
||||
compositor_env = {**env, 'QT_QPA_PLATFORM': 'offscreen', 'KWIN_COMPOSE': 'Q'}
|
||||
with (output / 'kwin.log').open('w') as log:
|
||||
compositor = subprocess.Popen(['kwin_wayland', '--virtual', '--socket', socket,
|
||||
'--width', '1024', '--height', '768'],
|
||||
env=compositor_env, stdout=log, stderr=subprocess.STDOUT)
|
||||
deadline = time.monotonic() + 15
|
||||
while not (Path(env['XDG_RUNTIME_DIR']) / socket).exists():
|
||||
assert compositor.poll() is None, (output / 'kwin.log').read_text()
|
||||
assert time.monotonic() < deadline, 'KWin socket startup timed out'
|
||||
time.sleep(.1)
|
||||
run = subprocess.run([str(binary), str(test / 'fixture.html')], env=env, cwd=output,
|
||||
capture_output=True, text=True, timeout=30)
|
||||
(output / 'native.log').write_text(run.stdout + run.stderr)
|
||||
print(run.stdout + run.stderr, flush=True)
|
||||
run.check_returncode()
|
||||
finally:
|
||||
if compositor:
|
||||
compositor.terminate()
|
||||
compositor.wait(timeout=10)
|
||||
pointer_unchanged = True
|
||||
if desktop_pointer is not None:
|
||||
after = subprocess.check_output(['xdotool', 'getmouselocation', '--shell'], env=env, text=True)
|
||||
# WINDOW identifies the surface under a stationary pointer and can change on mapping.
|
||||
coordinates = lambda value: [line for line in value.splitlines() if line.startswith(('X=', 'Y='))]
|
||||
pointer_unchanged = coordinates(desktop_pointer) == coordinates(after)
|
||||
(output / 'desktop-pointer.json').write_text(json.dumps({'before': desktop_pointer, 'after': after}))
|
||||
checkpoint = json.loads((output / 'checkpoint.json').read_text())
|
||||
# innerText contains the pointer pad label followed by the fixture JSON.
|
||||
body = checkpoint['body']
|
||||
observations = json.loads(body[body.index('{'):])
|
||||
events = observations['observations']
|
||||
required = {'click', 'contextmenu', 'pointermove', 'pointerdown', 'pointerup', 'wheel', 'keydown', 'keyup', 'input', 'beforeinput'}
|
||||
png = (output / 'capture.png').read_bytes()
|
||||
dimensions = struct.unpack('>II', png[16:24])
|
||||
criteria = {
|
||||
'isolated_world': checkpoint['pageGlobal'] == 'undefined',
|
||||
'trusted_events': bool(events) and all(event['trusted'] for event in events),
|
||||
'event_types': required <= {event['type'] for event in events},
|
||||
'hover': any(event['type'] == 'pointermove' and event.get('buttons') == 0 and event.get('x') == 30 and event.get('y') == 160 for event in events),
|
||||
'hover_enter_css': all(any(event['type'] == kind and event.get('x') == 30 and event.get('y') == 160 and event['padHover'] for event in events) for kind in ('mouseover', 'mouseenter')),
|
||||
'drag_buttons': any(event['type'] == 'pointermove' and event.get('buttons') == 1 for event in events),
|
||||
'chord_event': any(event['type'] == 'keydown' and (event.get('ctrl') or event.get('meta')) for event in events),
|
||||
'chord_select_all': observations['input'] == 'remplacé漢🙂',
|
||||
'contenteditable_unicode': observations['editor'].startswith('éditeur漢🙂'),
|
||||
'contenteditable_enter': any(event['type'] == 'keydown' and event.get('key') == 'Enter' and event['target'] == 'DIV' for event in events),
|
||||
'contenteditable_enter_dom': any(event['type'] == 'input' and event.get('inputType') in {'insertParagraph', 'insertLineBreak'} and event['target'] == 'DIV' for event in events) and '<' in observations['editorHtml'],
|
||||
'contenteditable_enter_once': all(sum(event['type'] == kind and event.get('inputType') == 'insertParagraph' and event['target'] == 'DIV' for event in events) == 1 for kind in ('beforeinput', 'input')),
|
||||
'visible_png': png[:8] == b'\x89PNG\r\n\x1a\n' and all(0 < value <= 2000 for value in dimensions),
|
||||
'desktop_pointer_unchanged': pointer_unchanged,
|
||||
}
|
||||
result = {'passed': all(criteria.values()), 'criteria': criteria, 'events': len(events), 'png': dimensions,
|
||||
'platform': platform.platform(), 'native': run.stdout}
|
||||
(output / 'result.json').write_text(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
print(json.dumps(result, ensure_ascii=False))
|
||||
assert result['passed'], [name for name, passed in criteria.items() if not passed]
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
|
@ -6,6 +6,8 @@ license = "MIT"
|
|||
|
||||
[build-dependencies]
|
||||
tauri-build = { version = "2.7.4", features = [] }
|
||||
cc = "1"
|
||||
pkg-config = "0.3"
|
||||
|
||||
[dependencies]
|
||||
tauri = { version = "2.12.5", features = ["devtools", "unstable"] }
|
||||
|
|
|
|||
|
|
@ -1,4 +1,6 @@
|
|||
mod build_webkit;
|
||||
fn main() {
|
||||
build_webkit::build(std::path::Path::new("src/embedded_view"));
|
||||
let manifest_dir =
|
||||
std::env::var("CARGO_MANIFEST_DIR").expect("CARGO_MANIFEST_DIR is set by Cargo");
|
||||
let out_dir = std::env::var("OUT_DIR").expect("OUT_DIR is set by Cargo");
|
||||
|
|
|
|||
46
packages/tauri-app/src-tauri/build_webkit.rs
Normal file
46
packages/tauri-app/src-tauri/build_webkit.rs
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
pub fn build(directory: &std::path::Path) {
|
||||
let os = std::env::var("CARGO_CFG_TARGET_OS").unwrap_or_default();
|
||||
if os != "macos" && os != "linux" {
|
||||
return;
|
||||
}
|
||||
let source = directory.join(format!(
|
||||
"input_{os}.{}",
|
||||
if os == "macos" { "m" } else { "c" }
|
||||
));
|
||||
println!("cargo:rerun-if-changed={}", source.display());
|
||||
println!(
|
||||
"cargo:rerun-if-changed={}",
|
||||
directory.join("input.h").display()
|
||||
);
|
||||
println!(
|
||||
"cargo:rerun-if-changed={}",
|
||||
directory.join("page_read.generated.h").display()
|
||||
);
|
||||
let mut build = cc::Build::new();
|
||||
build.file(source);
|
||||
let mobile = directory.join(format!(
|
||||
"mobile_{os}.{}",
|
||||
if os == "macos" { "m" } else { "c" }
|
||||
));
|
||||
println!("cargo:rerun-if-changed={}", mobile.display());
|
||||
build.file(mobile);
|
||||
if os == "macos" {
|
||||
let navigation = directory.join("input_navigation_macos.m");
|
||||
println!("cargo:rerun-if-changed={}", navigation.display());
|
||||
build
|
||||
.file(navigation)
|
||||
.flag("-fobjc-arc")
|
||||
.flag("-fblocks")
|
||||
.flag("-mmacosx-version-min=14.0");
|
||||
for framework in ["AppKit", "WebKit", "ApplicationServices"] {
|
||||
println!("cargo:rustc-link-lib=framework={framework}");
|
||||
}
|
||||
} else {
|
||||
let webkit = pkg_config::Config::new()
|
||||
.atleast_version("2.40")
|
||||
.probe("webkit2gtk-4.1")
|
||||
.expect("WebKitGTK 4.1 development files");
|
||||
build.includes(webkit.include_paths);
|
||||
}
|
||||
build.compile("codenomad_webkit_input");
|
||||
}
|
||||
349
packages/tauri-app/src-tauri/examples/browser-parent-webkit.rs
Normal file
349
packages/tauri-app/src-tauri/examples/browser-parent-webkit.rs
Normal file
|
|
@ -0,0 +1,349 @@
|
|||
//! Real native guests and production parent framing with an isolated fixture child.
|
||||
#[cfg(not(windows))]
|
||||
#[path = "../src/browser_controller.rs"]
|
||||
mod browser_controller;
|
||||
#[cfg(not(windows))]
|
||||
#[path = "../src/embedded_view.rs"]
|
||||
mod embedded_view;
|
||||
#[cfg(not(windows))]
|
||||
#[path = "../src/identity/local_window.rs"]
|
||||
mod identity;
|
||||
#[cfg(not(windows))]
|
||||
#[path = "../src/native_request.rs"]
|
||||
mod native_request;
|
||||
|
||||
// Only the shell's owning-window inventory is fixture-supplied. Browser target
|
||||
// claims, generations, leases, native loads, visibility and captures are real.
|
||||
#[cfg(not(windows))]
|
||||
mod local_windows {
|
||||
#[derive(Clone)]
|
||||
pub(crate) struct Record {
|
||||
pub(crate) label: String,
|
||||
}
|
||||
pub(crate) struct LocalWindows(pub(crate) Vec<Record>);
|
||||
impl LocalWindows {
|
||||
pub(crate) fn records(&self) -> Vec<Record> {
|
||||
self.0.clone()
|
||||
}
|
||||
}
|
||||
}
|
||||
#[cfg(not(windows))]
|
||||
use browser_controller::{BrowserController, BrowserTargetRegistration, BrowserTargetUpdate};
|
||||
#[cfg(not(windows))]
|
||||
use serde_json::{json, Value};
|
||||
#[cfg(not(windows))]
|
||||
use std::{
|
||||
io::{BufRead, BufReader, Write},
|
||||
process::{Command, Stdio},
|
||||
sync::{
|
||||
atomic::{AtomicU64, Ordering},
|
||||
Arc, Mutex,
|
||||
},
|
||||
time::Duration,
|
||||
};
|
||||
#[cfg(not(windows))]
|
||||
use tauri::{Listener, Manager};
|
||||
#[cfg(not(windows))]
|
||||
const OWNER: &str = "local-11111111-1111-4111-8111-111111111111";
|
||||
#[cfg(not(windows))]
|
||||
const OTHER: &str = "local-22222222-2222-4222-8222-222222222222";
|
||||
#[cfg(not(windows))]
|
||||
const SESSION: &str = "fixture-session";
|
||||
#[cfg(not(windows))]
|
||||
#[path = "../../tests/browser-parent/mobile-native.rs"]
|
||||
mod mobile_fixture;
|
||||
|
||||
#[cfg(not(windows))]
|
||||
fn bounds() -> Value {
|
||||
json!({"x":10,"y":60,"width":600,"height":400})
|
||||
}
|
||||
#[cfg(not(windows))]
|
||||
fn create_primary(app: &tauri::App, label: &str, base: &str) {
|
||||
let window = tauri::WindowBuilder::new(app, label)
|
||||
.title("Native parent qualification")
|
||||
.inner_size(740.0, 560.0)
|
||||
.focused(false)
|
||||
.build()
|
||||
.unwrap();
|
||||
window
|
||||
.add_child(
|
||||
tauri::webview::WebviewBuilder::new(
|
||||
label,
|
||||
tauri::WebviewUrl::External(format!("{base}/renderer").parse().unwrap()),
|
||||
)
|
||||
.focused(false),
|
||||
tauri::LogicalPosition::new(0.0, 0.0),
|
||||
tauri::LogicalSize::new(740.0, 560.0),
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
#[cfg(not(windows))]
|
||||
fn register(
|
||||
app: &tauri::AppHandle,
|
||||
controller: &BrowserController,
|
||||
owner: &str,
|
||||
id: &str,
|
||||
url: &str,
|
||||
) -> Result<(), String> {
|
||||
let input: BrowserTargetRegistration = serde_json::from_value(json!({"sessionId":SESSION,"registrationId":id,"url":url,"bounds":bounds(),"visible":false})).unwrap();
|
||||
controller.register(&app.get_window(owner).ok_or("Fixture owner closed")?, input)?;
|
||||
update(app, controller, owner, id, true)
|
||||
}
|
||||
#[cfg(not(windows))]
|
||||
fn update(
|
||||
app: &tauri::AppHandle,
|
||||
controller: &BrowserController,
|
||||
owner: &str,
|
||||
id: &str,
|
||||
visible: bool,
|
||||
) -> Result<(), String> {
|
||||
let input: BrowserTargetUpdate =
|
||||
serde_json::from_value(json!({"registrationId":id,"bounds":bounds(),"visible":visible}))
|
||||
.unwrap();
|
||||
controller.update(app, owner, input)
|
||||
}
|
||||
|
||||
#[cfg(not(windows))]
|
||||
fn capture_at_revocation(
|
||||
app: &tauri::AppHandle,
|
||||
controller: &BrowserController,
|
||||
id: &str,
|
||||
close: bool,
|
||||
deadline_ms: u64,
|
||||
fence: native_request::RequestFence,
|
||||
) -> Result<Value, String> {
|
||||
use std::sync::atomic::AtomicUsize;
|
||||
let prefix = format!("browser-{id}-");
|
||||
let view = app
|
||||
.webviews()
|
||||
.into_values()
|
||||
.find(|view| view.label().starts_with(&prefix))
|
||||
.ok_or("Fixture preview missing")?;
|
||||
let controller = controller.clone();
|
||||
let app = app.clone();
|
||||
let id = id.to_string();
|
||||
let reads = AtomicUsize::new(0);
|
||||
// The third guard runs in the actual native PNG completion callback, after
|
||||
// admission and dispatch. Revoke authority at that acceptance boundary.
|
||||
embedded_view::input::read(
|
||||
&view,
|
||||
embedded_view::input::Read::Capture,
|
||||
std::time::Instant::now() + Duration::from_secs(15),
|
||||
move || {
|
||||
fence.check()?;
|
||||
if reads.fetch_add(1, Ordering::SeqCst) == 2 {
|
||||
if close {
|
||||
controller.unregister(&app, OWNER, &id)?;
|
||||
} else {
|
||||
update(&app, &controller, OWNER, &id, false)?;
|
||||
}
|
||||
}
|
||||
controller
|
||||
.handle_native(
|
||||
&app,
|
||||
"browser.probe",
|
||||
Some(json!({"sessionID":SESSION})),
|
||||
deadline_ms,
|
||||
fence.clone(),
|
||||
)
|
||||
.map(|_| ())
|
||||
},
|
||||
)
|
||||
.map(|_| Value::Null)
|
||||
}
|
||||
|
||||
#[cfg(windows)]
|
||||
fn main() {}
|
||||
#[cfg(not(windows))]
|
||||
fn main() {
|
||||
let args = std::env::args().skip(1).collect::<Vec<_>>();
|
||||
assert_eq!(args.len(), 4, "node fixture.mjs base-url profile");
|
||||
let controller = BrowserController::new(args[3].clone().into());
|
||||
let live = Arc::new(Mutex::new(String::new()));
|
||||
let generation = Arc::new(AtomicU64::new(1));
|
||||
tauri::Builder::default()
|
||||
.manage(local_windows::LocalWindows(vec![local_windows::Record {
|
||||
label: OWNER.into(),
|
||||
}]))
|
||||
.manage(embedded_view::StorageIdentity::new(
|
||||
"parent-fixture".into(),
|
||||
args[3].clone(),
|
||||
))
|
||||
.setup(move |app| {
|
||||
#[cfg(target_os = "linux")]
|
||||
{
|
||||
use gtk::glib::object::ObjectExt;
|
||||
eprintln!(
|
||||
"native-display={}",
|
||||
gtk::gdk::Display::default().unwrap().type_().name()
|
||||
);
|
||||
}
|
||||
create_primary(app, OWNER, &args[2]);
|
||||
create_primary(app, OTHER, &args[2]);
|
||||
let open_app = app.handle().clone();
|
||||
let open_controller = controller.clone();
|
||||
let open_live = live.clone();
|
||||
app.listen("browser-target:open", move |event| {
|
||||
let payload: Value = serde_json::from_str(event.payload()).unwrap();
|
||||
let request = payload["requestID"].as_str().unwrap().to_string();
|
||||
if !open_controller.claim_open(OWNER, &request) {
|
||||
return;
|
||||
}
|
||||
let app = open_app.clone();
|
||||
let controller = open_controller.clone();
|
||||
let live = open_live.clone();
|
||||
std::thread::spawn(move || {
|
||||
let id = format!("fixture-{}", uuid::Uuid::new_v4());
|
||||
let result = register(
|
||||
&app,
|
||||
&controller,
|
||||
OWNER,
|
||||
&id,
|
||||
payload["url"].as_str().unwrap(),
|
||||
);
|
||||
if result.is_ok() {
|
||||
*live.lock().unwrap() = id;
|
||||
} else {
|
||||
eprintln!("open fixture failed: {result:?}");
|
||||
controller.release_open(&app, OWNER, &request);
|
||||
}
|
||||
});
|
||||
});
|
||||
let app = app.handle().clone();
|
||||
std::thread::spawn(move || {
|
||||
// Allow the primary documents' initial loads to settle; no daemon starts.
|
||||
std::thread::sleep(Duration::from_millis(500));
|
||||
let mut child = Command::new(&args[0])
|
||||
.arg("--enable-source-maps")
|
||||
.arg(&args[1])
|
||||
.arg(&args[2])
|
||||
.arg(&args[3])
|
||||
.env("CODENOMAD_NATIVE_PARENT", "1")
|
||||
.env("CODENOMAD_TAURI_WEBKIT", "1")
|
||||
.env("CODENOMAD_NATIVE_BROWSER_AUTOMATION", "0")
|
||||
.stdin(Stdio::piped())
|
||||
.stdout(Stdio::piped())
|
||||
.stderr(Stdio::inherit())
|
||||
.spawn()
|
||||
.unwrap();
|
||||
let mut input = child.stdin.take().unwrap();
|
||||
let mut reader = BufReader::new(child.stdout.take().unwrap());
|
||||
let mut line = String::new();
|
||||
loop {
|
||||
line.clear();
|
||||
if reader.read_line(&mut line).unwrap() == 0 {
|
||||
break;
|
||||
}
|
||||
let Some(request) = native_request::parse(line.trim_end()) else {
|
||||
panic!("Unexpected child output: {line}");
|
||||
};
|
||||
let current = generation.clone();
|
||||
let fence = native_request::RequestFence::new(request.deadline, move || {
|
||||
current.load(Ordering::SeqCst) == 1
|
||||
});
|
||||
let response = native_request::dispatch(request, fence, |request, fence| {
|
||||
let id = live.lock().unwrap().clone();
|
||||
match request.method.as_str() {
|
||||
method if method.starts_with("fixture.mobile-") => mobile_fixture::handle(&app,&controller,&id,method,request.params.as_ref().unwrap_or(&Value::Null),fence),
|
||||
"fixture.action-probe" | "fixture.action-control" => {
|
||||
let programs:Value=serde_json::from_str(include_str!("../../tests/browser-parent/action-programs.generated.json")).unwrap();
|
||||
let kind=if request.method=="fixture.action-probe" {"observe"} else {request.params.as_ref().and_then(|value|value["kind"].as_str()).ok_or("Invalid fixture control")?};
|
||||
let script=programs["programs"][kind].as_str().ok_or("Invalid fixture control")?;
|
||||
let view=app.webviews().into_values().find(|view|view.label().starts_with(&format!("browser-{id}-"))).ok_or("Fixture preview missing")?;
|
||||
view.eval(script).map_err(|error|error.to_string())?;
|
||||
if request.method=="fixture.action-probe" {
|
||||
let bytes=embedded_view::input::read(&view,embedded_view::input::Read::Checkpoint,std::time::Instant::now()+Duration::from_secs(5),move ||fence.check())?;
|
||||
let checkpoint:Value=serde_json::from_slice(&bytes).map_err(|error|error.to_string())?;
|
||||
let text=checkpoint["body"].as_str().and_then(|body|body.split_once("__CN_PROBE__").map(|(_,value)|value)).ok_or("Fixture observation missing")?;
|
||||
let observed=serde_json::from_str(text.trim()).map_err(|error|error.to_string())?;
|
||||
let _=view.eval(programs["programs"]["remove-probe"].as_str().unwrap());
|
||||
Ok(observed)
|
||||
} else {
|
||||
if kind=="recreate-shadow" { std::thread::sleep(Duration::from_millis(100)); }
|
||||
Ok(Value::Null)
|
||||
}
|
||||
}
|
||||
"fixture.read-mutation" => {
|
||||
let script = match request.params.as_ref().and_then(|value| value["kind"].as_str()) {
|
||||
Some("pushstate") => "history.pushState({}, '', '?same-document')",
|
||||
Some("reinsert") => "{ const item=document.querySelector('#aria-button');item.remove();document.querySelector('main').append(item); }",
|
||||
Some("recreate") => "document.open();document.write('<button>Recreated document</button>');document.close()",
|
||||
_ => return Err("Invalid read fixture mutation".into()),
|
||||
};
|
||||
let view = app.webviews().into_values().find(|view| view.label().starts_with(&format!("browser-{id}-"))).ok_or("Fixture preview missing")?;
|
||||
view.eval(script).map_err(|error| error.to_string())?;
|
||||
// eval is queued; the next isolated read observes its completion.
|
||||
std::thread::sleep(Duration::from_millis(100));
|
||||
Ok(Value::Null)
|
||||
}
|
||||
"fixture.hide" => {
|
||||
update(&app, &controller, OWNER, &id, false).map(|()| Value::Null)
|
||||
}
|
||||
"fixture.overlay" => {
|
||||
app.get_webview(OWNER)
|
||||
.ok_or("Fixture renderer missing")?
|
||||
.eval("document.querySelector('dialog').showModal()")
|
||||
.map_err(|error| error.to_string())?;
|
||||
update(&app, &controller, OWNER, &id, false).map(|()| Value::Null)
|
||||
}
|
||||
"fixture.show" => {
|
||||
app.get_webview(OWNER)
|
||||
.ok_or("Fixture renderer missing")?
|
||||
.eval("document.querySelector('dialog').close()")
|
||||
.map_err(|error| error.to_string())?;
|
||||
update(&app, &controller, OWNER, &id, true).map(|()| Value::Null)
|
||||
}
|
||||
"fixture.duplicate" => register(
|
||||
&app,
|
||||
&controller,
|
||||
OTHER,
|
||||
"fixture-duplicate",
|
||||
&format!("{}/ok", args[2]),
|
||||
)
|
||||
.map(|()| Value::Null),
|
||||
"fixture.removeDuplicate" => controller
|
||||
.unregister(&app, OTHER, "fixture-duplicate")
|
||||
.map(|()| Value::Null),
|
||||
"fixture.close" => controller
|
||||
.unregister(&app, OWNER, &id)
|
||||
.map(|()| Value::Null),
|
||||
"fixture.capture-hide" | "fixture.capture-close" => {
|
||||
capture_at_revocation(
|
||||
&app,
|
||||
&controller,
|
||||
&id,
|
||||
request.method == "fixture.capture-close",
|
||||
request.deadline,
|
||||
fence,
|
||||
)
|
||||
}
|
||||
"fixture.late" => {
|
||||
std::thread::sleep(Duration::from_millis(100));
|
||||
Ok(json!({"stale":true}))
|
||||
}
|
||||
"fixture.retire" => {
|
||||
generation.store(2, Ordering::SeqCst);
|
||||
Ok(json!({"stale":true}))
|
||||
}
|
||||
_ => controller.handle_native(
|
||||
&app,
|
||||
&request.method,
|
||||
request.params.clone(),
|
||||
request.deadline,
|
||||
fence,
|
||||
),
|
||||
}
|
||||
});
|
||||
if let Some(response) = response {
|
||||
input.write_all(response.as_bytes()).unwrap();
|
||||
input.flush().unwrap();
|
||||
}
|
||||
}
|
||||
let status = child.wait().unwrap();
|
||||
app.exit(if status.success() { 0 } else { 1 });
|
||||
});
|
||||
Ok(())
|
||||
})
|
||||
.run(tauri::test::mock_context(tauri::test::noop_assets()))
|
||||
.unwrap();
|
||||
}
|
||||
|
|
@ -15,6 +15,11 @@ use tauri::{PhysicalPosition, PhysicalSize};
|
|||
#[cfg(windows)]
|
||||
#[path = "browser_emulation.rs"]
|
||||
mod emulation;
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
#[path = "browser_emulation_webkit.rs"]
|
||||
mod emulation;
|
||||
#[path = "browser_emulation_layout.rs"]
|
||||
mod viewport;
|
||||
|
||||
#[path = "browser_controller_page.rs"]
|
||||
mod page;
|
||||
|
|
@ -26,6 +31,9 @@ mod history_ids;
|
|||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
#[path = "browser_history_webkit.rs"]
|
||||
mod webkit_history;
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
#[path = "browser_navigation_webkit.rs"]
|
||||
mod webkit_navigation;
|
||||
|
||||
#[derive(Clone, Debug, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
|
|
@ -35,6 +43,7 @@ pub(crate) struct BrowserTargetRegistration {
|
|||
url: String,
|
||||
bounds: BrowserTargetBounds,
|
||||
visible: Option<bool>,
|
||||
presentation: Option<viewport::Presentation>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Deserialize)]
|
||||
|
|
@ -43,6 +52,7 @@ pub(crate) struct BrowserTargetUpdate {
|
|||
registration_id: String,
|
||||
bounds: Option<BrowserTargetBounds>,
|
||||
visible: Option<bool>,
|
||||
presentation: Option<viewport::Presentation>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Deserialize)]
|
||||
|
|
@ -53,6 +63,7 @@ pub(crate) struct BrowserTargetAction {
|
|||
url: Option<String>,
|
||||
preset: Option<String>,
|
||||
entry_id: Option<i64>,
|
||||
viewport: Option<viewport::Viewport>,
|
||||
}
|
||||
|
||||
#[derive(Clone, PartialEq, Eq)]
|
||||
|
|
@ -79,11 +90,14 @@ struct OpenClaim {
|
|||
url: String,
|
||||
owner: Option<String>,
|
||||
failed_owners: HashSet<String>,
|
||||
fence: Option<crate::native_request::RequestFence>,
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct Inner {
|
||||
emulation_profiles: HashMap<String, String>,
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
canvases: HashMap<String, viewport::Canvas>,
|
||||
registrations: HashMap<String, Registration>,
|
||||
published_generations: HashMap<String, (String, u64)>,
|
||||
renderer_versions: HashMap<String, u64>,
|
||||
|
|
@ -185,7 +199,9 @@ impl BrowserController {
|
|||
generation,
|
||||
);
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
let installed = webkit_history::install(&webview, self.clone(), registration.clone());
|
||||
let installed = webkit_history::install(&webview, self.clone(), registration.clone())
|
||||
.and_then(|()| webkit_navigation::install(&webview, self.clone(), registration.clone()))
|
||||
.and_then(|()| crate::embedded_view::input::install_read(&webview));
|
||||
if let Err(error) = installed {
|
||||
let _ = self
|
||||
.embedded_views
|
||||
|
|
@ -214,6 +230,24 @@ impl BrowserController {
|
|||
&previous.webview_label,
|
||||
);
|
||||
}
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
{
|
||||
let current = self.owned_registration(window.label(), &input.registration_id)?;
|
||||
if let Err(error) = self.resize_webkit(
|
||||
window.app_handle(),
|
||||
¤t,
|
||||
input.bounds,
|
||||
input.presentation,
|
||||
) {
|
||||
let _ = self.unregister_generation(
|
||||
window.app_handle(),
|
||||
window.label(),
|
||||
&input.registration_id,
|
||||
Some(generation),
|
||||
);
|
||||
return Err(error);
|
||||
}
|
||||
}
|
||||
if let Err(error) = webview.navigate(url) {
|
||||
let _ = self.unregister_generation(
|
||||
window.app_handle(),
|
||||
|
|
@ -301,6 +335,9 @@ impl BrowserController {
|
|||
(registration.window_label.clone(), registration.generation),
|
||||
);
|
||||
let previous = inner.registrations.insert(id.clone(), registration);
|
||||
inner.emulation_profiles.remove(&id);
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
inner.canvases.remove(&id);
|
||||
inner
|
||||
.page_leases
|
||||
.retain(|_, lease| lease.registration.registration_id != id);
|
||||
|
|
@ -342,6 +379,9 @@ impl BrowserController {
|
|||
let bounds = input
|
||||
.bounds
|
||||
.ok_or_else(|| "Visible browser target bounds are required".to_string())?;
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
self.resize_webkit(app, ®istration, bounds, input.presentation)?;
|
||||
#[cfg(not(any(target_os = "linux", target_os = "macos")))]
|
||||
self.embedded_views.set_bounds(
|
||||
app,
|
||||
window_label,
|
||||
|
|
@ -449,7 +489,8 @@ impl BrowserController {
|
|||
{
|
||||
return;
|
||||
}
|
||||
self.invalidate_page_for_generation(®istration.registration_id, registration.generation);
|
||||
// History sampling publishes URL metadata; navigation/load callbacks
|
||||
// own document fences, including while a new page lease is binding.
|
||||
if let Some(webview) = app.get_webview(®istration.window_label) {
|
||||
let _ = webview.emit(
|
||||
"browser-target:navigated",
|
||||
|
|
@ -470,6 +511,9 @@ impl BrowserController {
|
|||
.get_webview(®istration.webview_label)
|
||||
.ok_or("Browser preview is no longer available")?;
|
||||
match input.action.as_str() {
|
||||
"emulate" => self
|
||||
.emulate(app, window_label, &input)
|
||||
.map(|()| Value::Null),
|
||||
"history" => webkit_history::action(self, &webview, ®istration, None),
|
||||
"history-go" => webkit_history::action(
|
||||
self,
|
||||
|
|
@ -537,6 +581,8 @@ impl BrowserController {
|
|||
}
|
||||
let registration = inner.registrations.remove(registration_id).unwrap();
|
||||
inner.emulation_profiles.remove(registration_id);
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
inner.canvases.remove(registration_id);
|
||||
inner
|
||||
.page_leases
|
||||
.retain(|_, lease| lease.registration.registration_id != registration_id);
|
||||
|
|
@ -558,6 +604,13 @@ impl BrowserController {
|
|||
let Some(claim) = inner.open_claims.get_mut(request_id) else {
|
||||
return false;
|
||||
};
|
||||
if claim
|
||||
.fence
|
||||
.as_ref()
|
||||
.is_some_and(|fence| fence.check().is_err())
|
||||
{
|
||||
return false;
|
||||
}
|
||||
if claim
|
||||
.owner
|
||||
.as_deref()
|
||||
|
|
@ -586,6 +639,13 @@ impl BrowserController {
|
|||
fn release_claim(&self, window_label: &str, request_id: &str) -> Option<OpenClaim> {
|
||||
let mut inner = self.inner.0.lock().ok()?;
|
||||
let claim = inner.open_claims.get_mut(request_id)?;
|
||||
if claim
|
||||
.fence
|
||||
.as_ref()
|
||||
.is_some_and(|fence| fence.check().is_err())
|
||||
{
|
||||
return None;
|
||||
}
|
||||
if claim.owner.as_deref() != Some(window_label) {
|
||||
return None;
|
||||
}
|
||||
|
|
@ -637,6 +697,8 @@ impl BrowserController {
|
|||
inner.page_load_expectations.remove(&id);
|
||||
inner.registrations.remove(&id);
|
||||
inner.emulation_profiles.remove(&id);
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
inner.canvases.remove(&id);
|
||||
}
|
||||
let retries = inner
|
||||
.open_claims
|
||||
|
|
@ -673,14 +735,16 @@ impl BrowserController {
|
|||
method: &str,
|
||||
params: Option<Value>,
|
||||
deadline_ms: u64,
|
||||
fence: crate::native_request::RequestFence,
|
||||
) -> Result<Value, String> {
|
||||
#[cfg(not(windows))]
|
||||
#[cfg(not(any(windows, target_os = "linux", target_os = "macos")))]
|
||||
{
|
||||
let _ = (app, method, params, deadline_ms);
|
||||
let _ = (app, method, params, deadline_ms, fence);
|
||||
return Err(unsupported());
|
||||
}
|
||||
#[cfg(windows)]
|
||||
#[cfg(any(windows, target_os = "linux", target_os = "macos"))]
|
||||
{
|
||||
fence.check()?;
|
||||
let input = params.ok_or_else(|| "Invalid browser request".to_string())?;
|
||||
let deadline = request_deadline(deadline_ms)?;
|
||||
let session_id = input
|
||||
|
|
@ -689,9 +753,10 @@ impl BrowserController {
|
|||
.ok_or_else(|| "Invalid browser request".to_string())?;
|
||||
match method {
|
||||
"browser.page.begin" | "browser.page.send" | "browser.page.check"
|
||||
| "browser.page.end" => self.page_request(app, method, &input, deadline),
|
||||
| "browser.page.end" => self.page_request(app, method, &input, deadline, &fence),
|
||||
"browser.probe" => {
|
||||
self.resolve(app, session_id)?;
|
||||
let registration = self.resolve(app, session_id)?;
|
||||
self.check_target(app, ®istration, deadline, &fence)?;
|
||||
Ok(json!({ "available": true }))
|
||||
}
|
||||
"browser.execute" => self.execute(
|
||||
|
|
@ -702,19 +767,21 @@ impl BrowserController {
|
|||
.cloned()
|
||||
.ok_or_else(|| "Invalid browser request".to_string())?,
|
||||
deadline,
|
||||
&fence,
|
||||
),
|
||||
_ => Err(format!("Unsupported native method: {method}")),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(windows)]
|
||||
#[cfg(any(windows, target_os = "linux", target_os = "macos"))]
|
||||
fn execute(
|
||||
&self,
|
||||
app: &AppHandle,
|
||||
session_id: &str,
|
||||
command: Value,
|
||||
deadline: Instant,
|
||||
fence: &crate::native_request::RequestFence,
|
||||
) -> Result<Value, String> {
|
||||
let action = command
|
||||
.get("action")
|
||||
|
|
@ -722,30 +789,82 @@ impl BrowserController {
|
|||
.ok_or_else(|| "Invalid browser command".to_string())?;
|
||||
if action == "open" {
|
||||
let url = command_url(&command)?;
|
||||
return self.open(app, session_id, url, deadline);
|
||||
return self.open(app, session_id, url, deadline, fence);
|
||||
}
|
||||
let registration = self.resolve(app, session_id)?;
|
||||
if action == "navigate" {
|
||||
self.check_target_identity(app, ®istration, deadline, fence)?;
|
||||
} else {
|
||||
self.check_target(app, ®istration, deadline, fence)?;
|
||||
}
|
||||
let webview = app
|
||||
.get_webview(®istration.webview_label)
|
||||
.ok_or_else(|| "Browser preview is no longer available".to_string())?;
|
||||
match action {
|
||||
"navigate" => {
|
||||
let url = allowed_url(command_url(&command)?)?;
|
||||
self.navigate_and_wait(®istration, &webview, &url, deadline)?;
|
||||
self.navigate_and_wait(app, ®istration, &webview, &url, deadline, fence)?;
|
||||
self.check_target(app, ®istration, deadline, fence)?;
|
||||
Ok(json!({ "url": webview.url().map_err(|error| error.to_string())? }))
|
||||
}
|
||||
"screenshot" => {
|
||||
let result = cdp(
|
||||
&webview,
|
||||
"Page.captureScreenshot",
|
||||
json!({ "format": "png", "fromSurface": true }),
|
||||
deadline,
|
||||
)?;
|
||||
let document = self.document_version(®istration)?;
|
||||
#[cfg(windows)]
|
||||
let result = {
|
||||
let controller = self.clone();
|
||||
let expected = registration.clone();
|
||||
let app = app.clone();
|
||||
let fence = fence.clone();
|
||||
cdp_guarded(
|
||||
&webview,
|
||||
"Page.captureScreenshot",
|
||||
json!({ "format": "png", "fromSurface": true }),
|
||||
deadline,
|
||||
move || {
|
||||
controller.check_target(&app, &expected, deadline, &fence)?;
|
||||
if controller.document_version(&expected)? != document {
|
||||
return Err("Browser document changed during capture".into());
|
||||
}
|
||||
Ok(())
|
||||
},
|
||||
24 * 1024 * 1024,
|
||||
)?
|
||||
};
|
||||
#[cfg(windows)]
|
||||
let data = result
|
||||
.get("data")
|
||||
.and_then(Value::as_str)
|
||||
.filter(|data| data.len() <= 16 * 1024 * 1024)
|
||||
.ok_or_else(|| "Browser screenshot is unavailable or too large".to_string())?;
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
let data = {
|
||||
use base64::Engine;
|
||||
let controller = self.clone();
|
||||
let expected = registration.clone();
|
||||
let app = app.clone();
|
||||
let fence = fence.clone();
|
||||
let bytes = crate::embedded_view::input::read(
|
||||
&webview,
|
||||
crate::embedded_view::input::Read::Capture,
|
||||
deadline,
|
||||
move || {
|
||||
controller.check_target(&app, &expected, deadline, &fence)?;
|
||||
if controller.document_version(&expected)? != document {
|
||||
return Err("Browser document changed during capture".into());
|
||||
}
|
||||
Ok(())
|
||||
},
|
||||
)?;
|
||||
let encoded = base64::engine::general_purpose::STANDARD.encode(bytes);
|
||||
if encoded.len() > 16 * 1024 * 1024 {
|
||||
return Err("Browser screenshot is unavailable or too large".into());
|
||||
}
|
||||
encoded
|
||||
};
|
||||
self.check_target(app, ®istration, deadline, fence)?;
|
||||
if self.document_version(®istration)? != document {
|
||||
return Err("Browser document changed during capture".into());
|
||||
}
|
||||
Ok(
|
||||
json!({ "image": { "mime": "image/png", "data": data }, "url": webview.url().map_err(|error| error.to_string())? }),
|
||||
)
|
||||
|
|
@ -754,13 +873,14 @@ impl BrowserController {
|
|||
}
|
||||
}
|
||||
|
||||
#[cfg(windows)]
|
||||
#[cfg(any(windows, target_os = "linux", target_os = "macos"))]
|
||||
fn open(
|
||||
&self,
|
||||
app: &AppHandle,
|
||||
session_id: &str,
|
||||
raw_url: &str,
|
||||
deadline: Instant,
|
||||
fence: &crate::native_request::RequestFence,
|
||||
) -> Result<Value, String> {
|
||||
let url = allowed_url(raw_url)?;
|
||||
match self.resolve(app, session_id) {
|
||||
|
|
@ -768,7 +888,8 @@ impl BrowserController {
|
|||
let webview = app
|
||||
.get_webview(®istration.webview_label)
|
||||
.ok_or_else(|| "Browser preview is no longer available".to_string())?;
|
||||
self.navigate_and_wait(®istration, &webview, &url, deadline)?;
|
||||
self.navigate_and_wait(app, ®istration, &webview, &url, deadline, fence)?;
|
||||
self.check_target(app, ®istration, deadline, fence)?;
|
||||
return Ok(json!({ "url": webview.url().map_err(|error| error.to_string())? }));
|
||||
}
|
||||
Err(error) if error == no_browser_target_error() => {}
|
||||
|
|
@ -776,24 +897,17 @@ impl BrowserController {
|
|||
}
|
||||
|
||||
let request_id = uuid::Uuid::new_v4().to_string();
|
||||
{
|
||||
let mut inner = self.inner.0.lock().map_err(|error| error.to_string())?;
|
||||
inner.open_claims.insert(
|
||||
request_id.clone(),
|
||||
OpenClaim {
|
||||
session_id: session_id.to_string(),
|
||||
url: raw_url.to_string(),
|
||||
owner: None,
|
||||
failed_owners: HashSet::new(),
|
||||
},
|
||||
);
|
||||
}
|
||||
let claim = OpenClaim {
|
||||
session_id: session_id.to_string(),
|
||||
url: raw_url.to_string(),
|
||||
owner: None,
|
||||
failed_owners: HashSet::new(),
|
||||
fence: Some(fence.clone()),
|
||||
};
|
||||
{
|
||||
let mut inner = self.inner.0.lock().map_err(|error| error.to_string())?;
|
||||
inner.open_claims.insert(request_id.clone(), claim.clone());
|
||||
}
|
||||
Self::emit_open_request(app, &request_id, &claim);
|
||||
let result = (|| -> Result<Value, String> {
|
||||
let registration_deadline = deadline.min(Instant::now() + Duration::from_secs(10));
|
||||
|
|
@ -801,6 +915,7 @@ impl BrowserController {
|
|||
let mut guard = inner.lock().map_err(|error| error.to_string())?;
|
||||
loop {
|
||||
drop(guard);
|
||||
fence.check()?;
|
||||
match self.resolve(app, session_id) {
|
||||
Ok(registration) => {
|
||||
let webview = app
|
||||
|
|
@ -814,6 +929,7 @@ impl BrowserController {
|
|||
sequence,
|
||||
deadline,
|
||||
)?;
|
||||
self.check_target(app, ®istration, deadline, fence)?;
|
||||
return Ok(
|
||||
json!({ "url": webview.url().map_err(|error| error.to_string())? }),
|
||||
);
|
||||
|
|
@ -858,8 +974,9 @@ impl BrowserController {
|
|||
}
|
||||
}
|
||||
|
||||
#[cfg(windows)]
|
||||
#[cfg(any(windows, target_os = "linux", target_os = "macos"))]
|
||||
fn resolve(&self, app: &AppHandle, session_id: &str) -> Result<Registration, String> {
|
||||
validate_id(session_id, 256)?;
|
||||
let registrations = self
|
||||
.inner
|
||||
.0
|
||||
|
|
@ -890,18 +1007,27 @@ impl BrowserController {
|
|||
#[cfg(windows)]
|
||||
fn navigate_and_wait(
|
||||
&self,
|
||||
app: &AppHandle,
|
||||
registration: &Registration,
|
||||
webview: &tauri::Webview,
|
||||
url: &tauri::Url,
|
||||
deadline: Instant,
|
||||
fence: &crate::native_request::RequestFence,
|
||||
) -> Result<(), String> {
|
||||
self.check_target(app, registration, deadline, fence)?;
|
||||
self.invalidate_page(®istration.registration_id);
|
||||
let sequence = self.prepare_page_load(®istration.registration_id, url.as_str())?;
|
||||
let result = cdp(
|
||||
let controller = self.clone();
|
||||
let expected = registration.clone();
|
||||
let app = app.clone();
|
||||
let fence = fence.clone();
|
||||
let result = cdp_guarded(
|
||||
webview,
|
||||
"Page.navigate",
|
||||
json!({ "url": url.as_str() }),
|
||||
deadline,
|
||||
move || controller.check_target_identity(&app, &expected, deadline, &fence),
|
||||
24 * 1024 * 1024,
|
||||
)?;
|
||||
if let Some(error) = result
|
||||
.get("errorText")
|
||||
|
|
@ -929,6 +1055,60 @@ impl BrowserController {
|
|||
Ok(registration.clone())
|
||||
}
|
||||
|
||||
fn document_version(&self, expected: &Registration) -> Result<u64, String> {
|
||||
let inner = self.inner.0.lock().map_err(|error| error.to_string())?;
|
||||
if inner.registrations.get(&expected.registration_id) != Some(expected) {
|
||||
return Err("Browser target is no longer current".into());
|
||||
}
|
||||
inner
|
||||
.navigation_versions
|
||||
.get(&expected.registration_id)
|
||||
.copied()
|
||||
.ok_or_else(|| "Browser document is unavailable".into())
|
||||
}
|
||||
|
||||
#[cfg(any(windows, target_os = "linux", target_os = "macos"))]
|
||||
fn check_target(
|
||||
&self,
|
||||
app: &AppHandle,
|
||||
expected: &Registration,
|
||||
deadline: Instant,
|
||||
fence: &crate::native_request::RequestFence,
|
||||
) -> Result<(), String> {
|
||||
self.check_target_identity(app, expected, deadline, fence)?;
|
||||
let view = app
|
||||
.get_webview(&expected.webview_label)
|
||||
.ok_or("Browser preview is no longer available")?;
|
||||
allowed_url(view.url().map_err(|error| error.to_string())?.as_str())?;
|
||||
self.document_version(expected)?;
|
||||
fence.check()
|
||||
}
|
||||
|
||||
#[cfg(any(windows, target_os = "linux", target_os = "macos"))]
|
||||
fn check_target_identity(
|
||||
&self,
|
||||
app: &AppHandle,
|
||||
expected: &Registration,
|
||||
deadline: Instant,
|
||||
fence: &crate::native_request::RequestFence,
|
||||
) -> Result<(), String> {
|
||||
fence.check()?;
|
||||
if Instant::now() >= deadline {
|
||||
return Err("Native request expired before execution".into());
|
||||
}
|
||||
if self.resolve(app, &expected.session_id)? != *expected {
|
||||
return Err("Browser target is no longer current".into());
|
||||
}
|
||||
let view = app
|
||||
.get_webview(&expected.webview_label)
|
||||
.ok_or("Browser preview is no longer available")?;
|
||||
if view.window().label() != expected.window_label {
|
||||
return Err("Browser target belongs to another window".into());
|
||||
}
|
||||
self.document_version(expected)?;
|
||||
fence.check()
|
||||
}
|
||||
|
||||
#[cfg(any(test, target_os = "linux", target_os = "macos"))]
|
||||
fn check_history_registration(&self, expected: &Registration) -> Result<(), String> {
|
||||
let current = self.owned_registration(&expected.window_label, &expected.registration_id)?;
|
||||
|
|
@ -942,9 +1122,9 @@ impl BrowserController {
|
|||
|
||||
fn invalidate_page(&self, registration_id: &str) {
|
||||
if let Ok(mut inner) = self.inner.0.lock() {
|
||||
inner
|
||||
.page_leases
|
||||
.retain(|_, lease| lease.registration.registration_id != registration_id);
|
||||
inner.page_leases.retain(|_, lease| {
|
||||
lease.registration.registration_id != registration_id || lease.document.is_some()
|
||||
});
|
||||
if let Some(version) = inner.navigation_versions.get_mut(registration_id) {
|
||||
*version = version.wrapping_add(1);
|
||||
}
|
||||
|
|
@ -961,9 +1141,9 @@ impl BrowserController {
|
|||
{
|
||||
return;
|
||||
}
|
||||
inner
|
||||
.page_leases
|
||||
.retain(|_, lease| lease.registration.registration_id != registration_id);
|
||||
inner.page_leases.retain(|_, lease| {
|
||||
lease.registration.registration_id != registration_id || lease.document.is_some()
|
||||
});
|
||||
if let Some(version) = inner.navigation_versions.get_mut(registration_id) {
|
||||
*version = version.wrapping_add(1);
|
||||
}
|
||||
|
|
@ -1051,7 +1231,13 @@ impl BrowserController {
|
|||
== Some(generation)
|
||||
{
|
||||
if let Some(expectation) = inner.page_load_expectations.get_mut(registration_id) {
|
||||
if expectation.url == url && expectation.navigation_id.is_none() {
|
||||
if expectation
|
||||
.navigation_id
|
||||
.is_some_and(|id| id != navigation_id)
|
||||
{
|
||||
expectation.result =
|
||||
Some(Err("Browser preview navigation is no longer current".into()));
|
||||
} else if expectation.url == url && expectation.navigation_id.is_none() {
|
||||
expectation.navigation_id = Some(navigation_id);
|
||||
}
|
||||
}
|
||||
|
|
@ -1077,7 +1263,8 @@ impl BrowserController {
|
|||
return;
|
||||
}
|
||||
if let Some(expectation) = inner.page_load_expectations.get_mut(registration_id) {
|
||||
if expectation.navigation_id == Some(navigation_id) {
|
||||
if expectation.navigation_id == Some(navigation_id) && expectation.result.is_none()
|
||||
{
|
||||
expectation.result = Some(error.map_or(Ok(()), Err));
|
||||
}
|
||||
}
|
||||
|
|
@ -1188,7 +1375,7 @@ fn unsupported() -> String {
|
|||
"Agent browser automation is unavailable in Tauri on macOS and Linux. Open the web preview manually, or use Electron for agent browser automation.".to_string()
|
||||
}
|
||||
|
||||
#[cfg(windows)]
|
||||
#[cfg(any(windows, target_os = "linux", target_os = "macos"))]
|
||||
fn command_url(command: &Value) -> Result<&str, String> {
|
||||
command
|
||||
.get("url")
|
||||
|
|
@ -1426,7 +1613,8 @@ mod tests {
|
|||
let input: super::BrowserTargetRegistration = serde_json::from_value(serde_json::json!({
|
||||
"sessionId": "session", "registrationId": "registration", "url": "https://example.org",
|
||||
"bounds": { "x": 0, "y": 0, "width": 300, "height": 200 }, "visible": false
|
||||
})).unwrap();
|
||||
}))
|
||||
.unwrap();
|
||||
assert_eq!(input.visible, Some(false));
|
||||
}
|
||||
use super::*;
|
||||
|
|
@ -1465,6 +1653,7 @@ mod tests {
|
|||
url: "https://example.com/".to_string(),
|
||||
owner: None,
|
||||
failed_owners: HashSet::new(),
|
||||
fence: None,
|
||||
},
|
||||
);
|
||||
|
||||
|
|
@ -1531,6 +1720,8 @@ mod tests {
|
|||
7,
|
||||
Some("Browser navigation failed: ConnectionAborted".to_string()),
|
||||
);
|
||||
// WebKitGTK emits Finished after LoadFailed for the same load.
|
||||
controller.mark_page_finished("registration", 1, 7, None);
|
||||
assert_eq!(
|
||||
controller
|
||||
.wait_for_page_load(
|
||||
|
|
|
|||
|
|
@ -1,5 +1,10 @@
|
|||
//! Private backend-only protocol adapter. Never register these methods as Tauri commands.
|
||||
use super::*;
|
||||
#[path = "browser_page_operation.rs"]
|
||||
mod operations;
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
#[path = "browser_page_webkit.rs"]
|
||||
mod webkit;
|
||||
|
||||
#[derive(Clone)]
|
||||
pub(super) struct PageLease {
|
||||
|
|
@ -8,34 +13,9 @@ pub(super) struct PageLease {
|
|||
operation_id: String,
|
||||
pub(super) generation: String,
|
||||
pub(super) expires: Instant,
|
||||
}
|
||||
|
||||
fn page_method_allowed(method: &str) -> bool {
|
||||
matches!(
|
||||
method,
|
||||
"Accessibility.getFullAXTree"
|
||||
| "DOM.resolveNode"
|
||||
| "DOM.scrollIntoViewIfNeeded"
|
||||
| "DOM.getContentQuads"
|
||||
| "DOM.getNodeForLocation"
|
||||
| "Page.getLayoutMetrics"
|
||||
| "Runtime.callFunctionOn"
|
||||
| "Runtime.evaluate"
|
||||
| "Runtime.releaseObjectGroup"
|
||||
| "Input.insertText"
|
||||
| "Input.dispatchMouseEvent"
|
||||
| "Input.dispatchKeyEvent"
|
||||
)
|
||||
}
|
||||
|
||||
fn page_params(method: &str, params: Value) -> Result<Value, String> {
|
||||
if !page_method_allowed(method) {
|
||||
return Err("Unsupported browser page command".into());
|
||||
}
|
||||
if !params.is_object() || params.to_string().len() > 256 * 1024 {
|
||||
return Err("Invalid or oversized browser page parameters".into());
|
||||
}
|
||||
Ok(params)
|
||||
pub(super) document: Option<String>,
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
pub(super) held: crate::embedded_view::input::Held,
|
||||
}
|
||||
|
||||
impl BrowserController {
|
||||
|
|
@ -84,6 +64,9 @@ impl BrowserController {
|
|||
generation: self.page_generation(®istration, navigation),
|
||||
registration,
|
||||
expires: Instant::now() + Duration::from_secs(30),
|
||||
document: None,
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
held: Default::default(),
|
||||
};
|
||||
inner.page_leases.insert(operation_id.into(), lease.clone());
|
||||
Ok(lease)
|
||||
|
|
@ -121,12 +104,13 @@ impl BrowserController {
|
|||
})
|
||||
|| inner.registrations.get(&lease.registration.registration_id)
|
||||
!= Some(&lease.registration)
|
||||
|| !inner
|
||||
.navigation_versions
|
||||
.get(&lease.registration.registration_id)
|
||||
.is_some_and(|navigation| {
|
||||
self.page_generation(&lease.registration, *navigation) == lease.generation
|
||||
})
|
||||
|| (lease.document.is_none()
|
||||
&& !inner
|
||||
.navigation_versions
|
||||
.get(&lease.registration.registration_id)
|
||||
.is_some_and(|navigation| {
|
||||
self.page_generation(&lease.registration, *navigation) == lease.generation
|
||||
}))
|
||||
{
|
||||
return Err("Browser page lease is no longer current".into());
|
||||
}
|
||||
|
|
@ -176,13 +160,15 @@ impl BrowserController {
|
|||
}
|
||||
}
|
||||
|
||||
#[cfg(windows)]
|
||||
#[cfg(any(windows, target_os = "linux", target_os = "macos"))]
|
||||
fn check_page(
|
||||
&self,
|
||||
app: &AppHandle,
|
||||
lease: &PageLease,
|
||||
deadline: Instant,
|
||||
fence: &crate::native_request::RequestFence,
|
||||
) -> Result<tauri::Url, String> {
|
||||
fence.check()?;
|
||||
self.check_page_state(lease, deadline)?;
|
||||
if self.resolve(app, &lease.session_id)? != lease.registration {
|
||||
return Err("Browser page lease is no longer current".into());
|
||||
|
|
@ -197,17 +183,28 @@ impl BrowserController {
|
|||
return Err("Browser page lease is no longer current".into());
|
||||
}
|
||||
self.check_page_state(lease, deadline)?;
|
||||
fence.check()?;
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
if let Some(document) = &lease.document {
|
||||
let response =
|
||||
self.webkit_read(app, lease, &json!({"op":"checkpoint"}), deadline, fence)?;
|
||||
if response["document"].as_str() != Some(document.as_str()) {
|
||||
return Err("Browser page document changed; inspect again".into());
|
||||
}
|
||||
}
|
||||
Ok(url)
|
||||
}
|
||||
|
||||
#[cfg(windows)]
|
||||
#[cfg(any(windows, target_os = "linux", target_os = "macos"))]
|
||||
pub(super) fn page_request(
|
||||
&self,
|
||||
app: &AppHandle,
|
||||
method: &str,
|
||||
input: &Value,
|
||||
deadline: Instant,
|
||||
fence: &crate::native_request::RequestFence,
|
||||
) -> Result<Value, String> {
|
||||
operations::envelope(method, input)?;
|
||||
let session_id = input["sessionID"]
|
||||
.as_str()
|
||||
.ok_or("Invalid browser page identity")?;
|
||||
|
|
@ -219,85 +216,137 @@ impl BrowserController {
|
|||
return Err("Invalid browser page identity".into());
|
||||
}
|
||||
if method == "browser.page.end" {
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
self.webkit_cleanup(app, session_id, operation_id, deadline, fence);
|
||||
self.end_page(session_id, operation_id);
|
||||
return Ok(json!({}));
|
||||
}
|
||||
let begin = method == "browser.page.begin";
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
if begin {
|
||||
self.webkit_expire(app, deadline, fence);
|
||||
}
|
||||
let lease = if begin {
|
||||
self.reserve_page(self.resolve(app, session_id)?, operation_id)?
|
||||
} else {
|
||||
self.page_lease(input)?
|
||||
};
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
let mut lease = lease;
|
||||
let operation = if method == "browser.page.send" {
|
||||
Some(operations::parse(
|
||||
input
|
||||
.get("operation")
|
||||
.cloned()
|
||||
.ok_or("Invalid browser page operation")?,
|
||||
)?)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let result = (|| {
|
||||
let url = self.check_page(app, &lease, deadline)?;
|
||||
let url = self.check_page(app, &lease, deadline, fence)?;
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
if begin {
|
||||
let response =
|
||||
self.webkit_read(app, &lease, &json!({"op":"checkpoint"}), deadline, fence)?;
|
||||
let document = response["document"]
|
||||
.as_str()
|
||||
.ok_or("Missing WebKit document identity")?
|
||||
.to_string();
|
||||
self.check_page_state(&lease, deadline)?;
|
||||
lease.generation = format!(
|
||||
"{}:{}:{}",
|
||||
self.run_id, lease.registration.generation, document
|
||||
);
|
||||
lease.document = Some(document);
|
||||
self.inner
|
||||
.0
|
||||
.lock()
|
||||
.map_err(|error| error.to_string())?
|
||||
.page_leases
|
||||
.insert(operation_id.into(), lease.clone());
|
||||
}
|
||||
if method == "browser.page.check" {
|
||||
return Ok(json!({ "url": url }));
|
||||
}
|
||||
if !begin && method != "browser.page.send" {
|
||||
return Err("Invalid browser page method".into());
|
||||
}
|
||||
#[cfg(windows)]
|
||||
let command = if begin {
|
||||
"Accessibility.enable"
|
||||
Some(("Accessibility.enable", json!({})))
|
||||
} else {
|
||||
input["method"]
|
||||
.as_str()
|
||||
.ok_or("Invalid browser page command")?
|
||||
operation.as_ref().and_then(operations::Operation::cdp)
|
||||
};
|
||||
let params = if begin {
|
||||
#[cfg(windows)]
|
||||
let response = if let Some((command, params)) = command {
|
||||
let webview = app
|
||||
.get_webview(&lease.registration.webview_label)
|
||||
.ok_or("Browser preview is no longer available")?;
|
||||
let controller = self.clone();
|
||||
let guard_lease = lease.clone();
|
||||
let guard_app = app.clone();
|
||||
let guard_fence = fence.clone();
|
||||
cdp_guarded(
|
||||
&webview,
|
||||
command,
|
||||
params,
|
||||
deadline.min(lease.expires),
|
||||
move || {
|
||||
controller
|
||||
.check_page(&guard_app, &guard_lease, deadline, &guard_fence)
|
||||
.map(|_| ())
|
||||
},
|
||||
4 * 1024 * 1024,
|
||||
)?
|
||||
} else {
|
||||
json!({})
|
||||
} else {
|
||||
page_params(command, input.get("params").cloned().unwrap_or(json!({})))?
|
||||
};
|
||||
let webview = app
|
||||
.get_webview(&lease.registration.webview_label)
|
||||
.ok_or("Browser preview is no longer available")?;
|
||||
let controller = self.clone();
|
||||
let guard_lease = lease.clone();
|
||||
let guard_app = app.clone();
|
||||
let response = cdp_guarded(
|
||||
&webview,
|
||||
command,
|
||||
params,
|
||||
deadline.min(lease.expires),
|
||||
move || {
|
||||
controller
|
||||
.check_page(&guard_app, &guard_lease, deadline)
|
||||
.map(|_| ())
|
||||
},
|
||||
4 * 1024 * 1024,
|
||||
)?;
|
||||
let url = self.check_page(app, &lease, deadline)?;
|
||||
if begin {
|
||||
Ok(json!({ "generation": lease.generation, "url": url }))
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
let response = if let Some(_) = &operation {
|
||||
let response =
|
||||
self.webkit_read(app, &lease, &input["operation"], deadline, fence)?;
|
||||
if response["document"].as_str() != lease.document.as_deref() {
|
||||
return Err("Browser page document changed; inspect again".into());
|
||||
}
|
||||
response["result"].clone()
|
||||
} else {
|
||||
Ok(response)
|
||||
json!({})
|
||||
};
|
||||
let url = self.check_page(app, &lease, deadline, fence)?;
|
||||
if begin {
|
||||
Ok(
|
||||
json!({ "generation": lease.generation, "url": url, "driver": if cfg!(windows) {"cdp"} else {"webkit"}, "contractVersion":1 }),
|
||||
)
|
||||
} else {
|
||||
#[cfg(windows)]
|
||||
{
|
||||
let operation = operation.as_ref().ok_or("Invalid browser page operation")?;
|
||||
Ok(operation.result(response))
|
||||
}
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
{
|
||||
Ok(response)
|
||||
}
|
||||
}
|
||||
})();
|
||||
if result.is_err() {
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
if std::env::var_os("CN_INPUT_TRACE").is_some() {
|
||||
eprintln!(
|
||||
"native page {method} op={} function={}: {}",
|
||||
input["operation"]["op"],
|
||||
input["operation"]["function"],
|
||||
result.as_ref().unwrap_err()
|
||||
);
|
||||
}
|
||||
#[cfg(windows)]
|
||||
self.release_page(&lease);
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
if lease.held.lock().is_ok_and(|held| held.is_empty()) {
|
||||
self.release_page(&lease);
|
||||
}
|
||||
}
|
||||
result
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn protocol_allowlist_and_parameter_budget_are_fixed() {
|
||||
for method in [
|
||||
"Page.navigate",
|
||||
"Network.enable",
|
||||
"Browser.close",
|
||||
"DOM.getBoxModel",
|
||||
"Accessibility.enable",
|
||||
] {
|
||||
assert!(page_params(method, json!({})).is_err());
|
||||
}
|
||||
assert!(page_params("Input.insertText", json!({ "text": "x" })).is_ok());
|
||||
assert!(page_params(
|
||||
"Runtime.evaluate",
|
||||
json!({ "expression": "x".repeat(256 * 1024) })
|
||||
)
|
||||
.is_err());
|
||||
assert!(page_params("Runtime.evaluate", Value::Null).is_err());
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -156,6 +156,123 @@ fn controller_with_registration() -> (BrowserController, Registration) {
|
|||
(controller, registration)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replacing_a_registration_clears_its_emulation_profile() {
|
||||
let (controller, mut current) = controller_with_registration();
|
||||
controller
|
||||
.inner
|
||||
.0
|
||||
.lock()
|
||||
.unwrap()
|
||||
.emulation_profiles
|
||||
.insert(current.registration_id.clone(), "mobile".into());
|
||||
current.generation += 1;
|
||||
controller
|
||||
.publish_registration(current.clone(), 0, "https://example.com/")
|
||||
.unwrap();
|
||||
assert!(!controller
|
||||
.inner
|
||||
.0
|
||||
.lock()
|
||||
.unwrap()
|
||||
.emulation_profiles
|
||||
.contains_key(¤t.registration_id));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn capture_document_identity_changes_on_navigation_hide_and_replacement() {
|
||||
let (controller, expected) = controller_with_registration();
|
||||
let document = controller.document_version(&expected).unwrap();
|
||||
controller.invalidate_page(&expected.registration_id);
|
||||
assert_ne!(controller.document_version(&expected).unwrap(), document);
|
||||
controller
|
||||
.inner
|
||||
.0
|
||||
.lock()
|
||||
.unwrap()
|
||||
.registrations
|
||||
.get_mut(&expected.registration_id)
|
||||
.unwrap()
|
||||
.visible = false;
|
||||
assert!(controller.document_version(&expected).is_err());
|
||||
controller.inner.0.lock().unwrap().registrations.insert(
|
||||
expected.registration_id.clone(),
|
||||
Registration {
|
||||
generation: expected.generation + 1,
|
||||
..expected.clone()
|
||||
},
|
||||
);
|
||||
assert!(controller.document_version(&expected).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn native_open_claim_cannot_be_reclaimed_after_deadline_or_backend_retirement() {
|
||||
use std::sync::atomic::AtomicBool;
|
||||
let (controller, _) = controller_with_registration();
|
||||
let current = Arc::new(AtomicBool::new(true));
|
||||
let predicate = current.clone();
|
||||
let now = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.unwrap()
|
||||
.as_millis() as u64;
|
||||
controller.inner.0.lock().unwrap().open_claims.insert(
|
||||
"pending".into(),
|
||||
OpenClaim {
|
||||
session_id: "session".into(),
|
||||
url: "https://example.com/".into(),
|
||||
owner: None,
|
||||
failed_owners: HashSet::new(),
|
||||
fence: Some(crate::native_request::RequestFence::new(
|
||||
now + 10_000,
|
||||
move || predicate.load(Ordering::SeqCst),
|
||||
)),
|
||||
},
|
||||
);
|
||||
assert!(controller.claim_open(OWNER, "pending"));
|
||||
current.store(false, Ordering::SeqCst);
|
||||
assert!(!controller.claim_open(OWNER, "pending"));
|
||||
assert!(controller.release_claim(OWNER, "pending").is_none());
|
||||
controller
|
||||
.inner
|
||||
.0
|
||||
.lock()
|
||||
.unwrap()
|
||||
.open_claims
|
||||
.get_mut("pending")
|
||||
.unwrap()
|
||||
.fence = Some(crate::native_request::RequestFence::new(0, || true));
|
||||
assert!(!controller.claim_open(OWNER, "pending"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_different_native_navigation_supersedes_a_bound_load_even_if_it_finishes_late() {
|
||||
let (controller, expected) = controller_with_registration();
|
||||
let sequence = controller
|
||||
.prepare_page_load(&expected.registration_id, "https://example.com/wanted")
|
||||
.unwrap();
|
||||
controller.mark_page_started(
|
||||
&expected.registration_id,
|
||||
expected.generation,
|
||||
1,
|
||||
"https://example.com/wanted".into(),
|
||||
);
|
||||
controller.mark_page_started(
|
||||
&expected.registration_id,
|
||||
expected.generation,
|
||||
2,
|
||||
"https://example.com/user".into(),
|
||||
);
|
||||
controller.mark_page_finished(&expected.registration_id, expected.generation, 1, None);
|
||||
assert!(controller
|
||||
.wait_for_page_load(
|
||||
&expected.registration_id,
|
||||
sequence,
|
||||
Instant::now() + Duration::from_secs(1)
|
||||
)
|
||||
.unwrap_err()
|
||||
.contains("no longer current"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn history_requires_current_owner_and_generation_but_survives_hide() {
|
||||
let (controller, expected) = controller_with_registration();
|
||||
|
|
@ -304,6 +421,7 @@ fn hiding_revokes_page_authority_even_when_native_hide_fails() {
|
|||
registration_id: "preview".into(),
|
||||
bounds: None,
|
||||
visible: Some(false),
|
||||
presentation: None,
|
||||
}
|
||||
)
|
||||
.is_err());
|
||||
|
|
@ -486,6 +604,58 @@ fn page_generation_survives_end_but_not_navigation_registration_or_host_replacem
|
|||
assert_ne!(other.generation, first.generation);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn document_bound_page_leases_preserve_uri_changes_and_registration_fences() {
|
||||
let (controller, registration) = controller_with_registration();
|
||||
let operation = uuid::Uuid::new_v4().to_string();
|
||||
let mut lease = controller.reserve_page(registration, &operation).unwrap();
|
||||
lease.document = Some("isolated-document-token".into());
|
||||
controller
|
||||
.inner
|
||||
.0
|
||||
.lock()
|
||||
.unwrap()
|
||||
.page_leases
|
||||
.insert(operation.clone(), lease.clone());
|
||||
controller.invalidate_page_for_generation("preview", 1);
|
||||
controller.invalidate_page("preview");
|
||||
let deadline = Instant::now() + Duration::from_secs(1);
|
||||
assert!(controller.check_page_state(&lease, deadline).is_ok());
|
||||
// The native adapter checks the document token; URI callbacks keep the lease.
|
||||
assert!(controller.check_page_state(&lease, Instant::now()).is_err());
|
||||
controller
|
||||
.inner
|
||||
.0
|
||||
.lock()
|
||||
.unwrap()
|
||||
.registrations
|
||||
.get_mut("preview")
|
||||
.unwrap()
|
||||
.visible = false;
|
||||
assert!(controller.check_page_state(&lease, deadline).is_err());
|
||||
controller
|
||||
.inner
|
||||
.0
|
||||
.lock()
|
||||
.unwrap()
|
||||
.registrations
|
||||
.get_mut("preview")
|
||||
.unwrap()
|
||||
.visible = true;
|
||||
controller
|
||||
.inner
|
||||
.0
|
||||
.lock()
|
||||
.unwrap()
|
||||
.registrations
|
||||
.get_mut("preview")
|
||||
.unwrap()
|
||||
.generation += 1;
|
||||
assert!(controller.check_page_state(&lease, deadline).is_err());
|
||||
controller.end_page("session", &operation);
|
||||
assert!(controller.check_page_state(&lease, deadline).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn page_lease_identity_limits_expiry_and_cleanup_are_exact() {
|
||||
let (controller, registration) = controller_with_registration();
|
||||
|
|
|
|||
229
packages/tauri-app/src-tauri/src/browser_emulation_layout.rs
Normal file
229
packages/tauri-app/src-tauri/src/browser_emulation_layout.rs
Normal file
|
|
@ -0,0 +1,229 @@
|
|||
use super::{validate_bounds, BrowserTargetBounds};
|
||||
use serde::Deserialize;
|
||||
#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Eq)]
|
||||
pub(crate) struct Viewport {
|
||||
pub width: u32,
|
||||
pub height: u32,
|
||||
}
|
||||
#[derive(Clone, Copy, Debug, Default, Deserialize, PartialEq, Eq)]
|
||||
#[serde(rename_all = "camelCase", deny_unknown_fields)]
|
||||
pub(crate) struct Presentation {
|
||||
#[serde(default)]
|
||||
pub css: bool,
|
||||
pub viewport: Option<Viewport>,
|
||||
}
|
||||
#[derive(Clone, Copy, Debug)]
|
||||
pub(super) struct Canvas {
|
||||
pub bounds: BrowserTargetBounds,
|
||||
pub presentation: Presentation,
|
||||
}
|
||||
pub(super) struct Layout {
|
||||
pub bounds: BrowserTargetBounds,
|
||||
pub zoom: f64,
|
||||
}
|
||||
pub(super) fn validate(size: Viewport) -> Result<(), String> {
|
||||
if size.width == 0 || size.height == 0 || size.width > 16384 || size.height > 16384 {
|
||||
Err("Invalid browser CSS viewport".into())
|
||||
} else {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
pub(super) fn layout(
|
||||
canvas: Canvas,
|
||||
pixel_scale: f64,
|
||||
device_scale: f64,
|
||||
target: Option<Viewport>,
|
||||
) -> Result<Layout, String> {
|
||||
validate_bounds(canvas.bounds)?;
|
||||
if !pixel_scale.is_finite()
|
||||
|| !device_scale.is_finite()
|
||||
|| pixel_scale <= 0.0
|
||||
|| device_scale <= 0.0
|
||||
{
|
||||
return Err("Invalid native preview scale".into());
|
||||
}
|
||||
let mut area = canvas.bounds;
|
||||
if canvas.presentation.css {
|
||||
area = BrowserTargetBounds {
|
||||
x: (area.x as f64 * pixel_scale).round() as i32,
|
||||
y: (area.y as f64 * pixel_scale).round() as i32,
|
||||
width: (area.width as f64 * pixel_scale).round() as u32,
|
||||
height: (area.height as f64 * pixel_scale).round() as u32,
|
||||
};
|
||||
}
|
||||
validate_bounds(area)?;
|
||||
let target = target.or(canvas.presentation.viewport).or_else(|| {
|
||||
canvas.presentation.css.then_some(Viewport {
|
||||
width: canvas.bounds.width,
|
||||
height: canvas.bounds.height,
|
||||
})
|
||||
});
|
||||
let Some(size) = target else {
|
||||
return Ok(Layout {
|
||||
bounds: area,
|
||||
zoom: 1.0,
|
||||
});
|
||||
};
|
||||
validate(size)?;
|
||||
let aw = (area.width as f64 / device_scale).floor() as u32;
|
||||
let ah = (area.height as f64 / device_scale).floor() as u32;
|
||||
// Native GTK allocations are integer DIPs. Find a nearby aspect ratio whose
|
||||
// zoom interval gives the exact integer CSS dimensions on both axes.
|
||||
let max_width = aw.min((ah as f64 * size.width as f64 / size.height as f64).ceil() as u32 + 1);
|
||||
for width in (1..=max_width).rev() {
|
||||
let height = width as f64 * size.height as f64 / size.width as f64;
|
||||
for height in [height.floor() as u32, height.ceil() as u32] {
|
||||
if height == 0 || height > ah {
|
||||
continue;
|
||||
}
|
||||
let lower = (width as f64 / (size.width + 1) as f64)
|
||||
.max(height as f64 / (size.height + 1) as f64);
|
||||
let upper = (width as f64 / size.width as f64).min(height as f64 / size.height as f64);
|
||||
if lower >= upper {
|
||||
continue;
|
||||
}
|
||||
let exact = width as f64 == size.width as f64 * upper
|
||||
&& height as f64 == size.height as f64 * upper
|
||||
&& (upper * 64.0).fract() == 0.0;
|
||||
let zoom = if exact { upper } else { (lower + upper) / 2.0 };
|
||||
let w = (width as f64 * device_scale).round() as u32;
|
||||
let h = (height as f64 * device_scale).round() as u32;
|
||||
return Ok(Layout {
|
||||
bounds: BrowserTargetBounds {
|
||||
x: area
|
||||
.x
|
||||
.checked_add(((area.width - w) / 2) as i32)
|
||||
.ok_or("Invalid browser target bounds")?,
|
||||
y: area.y,
|
||||
width: w,
|
||||
height: h,
|
||||
},
|
||||
zoom,
|
||||
});
|
||||
}
|
||||
}
|
||||
Err("Browser preview is too small for this viewport".into())
|
||||
}
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
#[test]
|
||||
fn css_viewports_survive_native_scale_and_clipping() {
|
||||
for size in [
|
||||
Viewport {
|
||||
width: 1440,
|
||||
height: 900,
|
||||
},
|
||||
Viewport {
|
||||
width: 768,
|
||||
height: 1024,
|
||||
},
|
||||
Viewport {
|
||||
width: 1024,
|
||||
height: 768,
|
||||
},
|
||||
Viewport {
|
||||
width: 390,
|
||||
height: 844,
|
||||
},
|
||||
Viewport {
|
||||
width: 844,
|
||||
height: 390,
|
||||
},
|
||||
] {
|
||||
for (width, height, scale) in [
|
||||
(500, 600, 1.0),
|
||||
(750, 900, 1.0),
|
||||
(1000, 1200, 2.0),
|
||||
(585, 1266, 1.0),
|
||||
] {
|
||||
let result = layout(
|
||||
Canvas {
|
||||
bounds: BrowserTargetBounds {
|
||||
x: 10,
|
||||
y: 60,
|
||||
width,
|
||||
height,
|
||||
},
|
||||
presentation: Presentation::default(),
|
||||
},
|
||||
1.0,
|
||||
scale,
|
||||
Some(size),
|
||||
)
|
||||
.unwrap();
|
||||
let zoom = result.zoom as f32 as f64;
|
||||
assert_eq!(
|
||||
(result.bounds.width as f64 / scale / zoom).floor() as u32,
|
||||
size.width
|
||||
);
|
||||
assert_eq!(
|
||||
(result.bounds.height as f64 / scale / zoom).floor() as u32,
|
||||
size.height
|
||||
);
|
||||
assert!(result.bounds.width <= width && result.bounds.height <= height);
|
||||
}
|
||||
}
|
||||
for device in [1.0, 2.0] {
|
||||
for app_zoom in [0.8, 1.0, 1.25, 1.5] {
|
||||
let result = layout(
|
||||
Canvas {
|
||||
bounds: BrowserTargetBounds {
|
||||
x: 10,
|
||||
y: 60,
|
||||
width: 500,
|
||||
height: 400,
|
||||
},
|
||||
presentation: Presentation {
|
||||
css: true,
|
||||
viewport: None,
|
||||
},
|
||||
},
|
||||
device * app_zoom,
|
||||
device,
|
||||
None,
|
||||
)
|
||||
.unwrap();
|
||||
let zoom = result.zoom as f32 as f64;
|
||||
assert_eq!(
|
||||
(result.bounds.width as f64 / device / zoom).floor() as u32,
|
||||
500
|
||||
);
|
||||
assert_eq!(
|
||||
(result.bounds.height as f64 / device / zoom).floor() as u32,
|
||||
400
|
||||
);
|
||||
}
|
||||
}
|
||||
for (x, width) in [(0, u32::MAX), (i32::MAX, 500)] {
|
||||
assert!(layout(
|
||||
Canvas {
|
||||
bounds: BrowserTargetBounds {
|
||||
x,
|
||||
y: 0,
|
||||
width,
|
||||
height: 400
|
||||
},
|
||||
presentation: Presentation::default(),
|
||||
},
|
||||
1.0,
|
||||
1.0,
|
||||
Some(Viewport {
|
||||
width: 390,
|
||||
height: 844
|
||||
})
|
||||
)
|
||||
.is_err());
|
||||
}
|
||||
assert!(validate(Viewport {
|
||||
width: 0,
|
||||
height: 844
|
||||
})
|
||||
.is_err());
|
||||
assert!(validate(Viewport {
|
||||
width: 390,
|
||||
height: 16385
|
||||
})
|
||||
.is_err());
|
||||
}
|
||||
}
|
||||
172
packages/tauri-app/src-tauri/src/browser_emulation_webkit.rs
Normal file
172
packages/tauri-app/src-tauri/src/browser_emulation_webkit.rs
Normal file
|
|
@ -0,0 +1,172 @@
|
|||
use super::viewport::{Canvas, Presentation, Viewport};
|
||||
use super::*;
|
||||
const MOBILE_UA: &str =
|
||||
"Mozilla/5.0 (Mobile) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile Safari/605.1.15";
|
||||
fn profile(preset: &str) -> Result<(Option<Viewport>, f64), String> {
|
||||
if preset == "none" {
|
||||
return Ok((None, 0.0));
|
||||
}
|
||||
if !matches!(preset, "mobile" | "mobileLandscape") {
|
||||
return Err("Invalid browser emulation profile".into());
|
||||
}
|
||||
#[derive(Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
struct Profiles {
|
||||
mobile: Viewport,
|
||||
device_scale_factor: f64,
|
||||
}
|
||||
let data: Profiles = serde_json::from_str(include_str!(
|
||||
"../../../ui/src/lib/native/browser-emulation.json"
|
||||
))
|
||||
.map_err(|e| e.to_string())?;
|
||||
let size = if preset == "mobileLandscape" {
|
||||
Viewport {
|
||||
width: data.mobile.height,
|
||||
height: data.mobile.width,
|
||||
}
|
||||
} else {
|
||||
data.mobile
|
||||
};
|
||||
Ok((Some(size), data.device_scale_factor))
|
||||
}
|
||||
impl BrowserController {
|
||||
fn configure_webkit<R: tauri::Runtime>(
|
||||
&self,
|
||||
app: &AppHandle<R>,
|
||||
registration: &Registration,
|
||||
canvas: Canvas,
|
||||
preset: &str,
|
||||
) -> Result<(), String> {
|
||||
let primary = app
|
||||
.get_webview(®istration.window_label)
|
||||
.ok_or("Browser renderer is unavailable")?;
|
||||
let (scale, device) = crate::embedded_view::mobile::scale(&primary)?;
|
||||
let (target, density) = profile(preset)?;
|
||||
let layout = viewport::layout(canvas, scale, device, target)?;
|
||||
validate_bounds(layout.bounds)?;
|
||||
let view = app
|
||||
.get_webview(®istration.webview_label)
|
||||
.ok_or("Browser preview is unavailable")?;
|
||||
if self.owned_registration(®istration.window_label, ®istration.registration_id)?
|
||||
!= *registration
|
||||
{
|
||||
return Err("Browser target changed before emulation".into());
|
||||
}
|
||||
self.inner
|
||||
.0
|
||||
.lock()
|
||||
.map_err(|e| e.to_string())?
|
||||
.page_leases
|
||||
.retain(|_, lease| lease.registration.registration_id != registration.registration_id);
|
||||
self.invalidate_page_for_generation(®istration.registration_id, registration.generation);
|
||||
crate::embedded_view::mobile::profile(
|
||||
&view,
|
||||
if preset == "none" { "" } else { MOBILE_UA },
|
||||
layout.zoom,
|
||||
density,
|
||||
)?;
|
||||
self.embedded_views.set_bounds(
|
||||
app,
|
||||
®istration.window_label,
|
||||
®istration.webview_label,
|
||||
layout.bounds,
|
||||
)?;
|
||||
if self.owned_registration(®istration.window_label, ®istration.registration_id)?
|
||||
!= *registration
|
||||
{
|
||||
return Err("Browser target changed during emulation".into());
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
pub(super) fn resize_webkit<R: tauri::Runtime>(
|
||||
&self,
|
||||
app: &AppHandle<R>,
|
||||
registration: &Registration,
|
||||
bounds: BrowserTargetBounds,
|
||||
presentation: Option<Presentation>,
|
||||
) -> Result<(), String> {
|
||||
let _lock = self.emulation_lock.lock().map_err(|e| e.to_string())?;
|
||||
let canvas = Canvas {
|
||||
bounds,
|
||||
presentation: presentation.unwrap_or_default(),
|
||||
};
|
||||
let (preset, previous) = {
|
||||
let inner = self.inner.0.lock().map_err(|e| e.to_string())?;
|
||||
(
|
||||
inner
|
||||
.emulation_profiles
|
||||
.get(®istration.registration_id)
|
||||
.cloned()
|
||||
.unwrap_or_else(|| "none".into()),
|
||||
inner.canvases.get(®istration.registration_id).copied(),
|
||||
)
|
||||
};
|
||||
if let Err(error) = self.configure_webkit(app, registration, canvas, &preset) {
|
||||
if let Some(previous) = previous {
|
||||
let _ = self.configure_webkit(app, registration, previous, &preset);
|
||||
}
|
||||
return Err(error);
|
||||
}
|
||||
let mut inner = self.inner.0.lock().map_err(|e| e.to_string())?;
|
||||
if inner.registrations.get(®istration.registration_id) != Some(registration) {
|
||||
return Err("Browser target changed before viewport publication".into());
|
||||
}
|
||||
inner
|
||||
.canvases
|
||||
.insert(registration.registration_id.clone(), canvas);
|
||||
Ok(())
|
||||
}
|
||||
pub(super) fn emulate(
|
||||
&self,
|
||||
app: &AppHandle,
|
||||
owner: &str,
|
||||
input: &BrowserTargetAction,
|
||||
) -> Result<(), String> {
|
||||
let preset = input.preset.as_deref().unwrap_or("");
|
||||
profile(preset)?;
|
||||
if let Some(size) = input.viewport {
|
||||
viewport::validate(size)?;
|
||||
}
|
||||
let _lock = self.emulation_lock.lock().map_err(|e| e.to_string())?;
|
||||
let registration = self.owned_registration(owner, &input.registration_id)?;
|
||||
let (previous, canvas) = {
|
||||
let inner = self.inner.0.lock().map_err(|e| e.to_string())?;
|
||||
(
|
||||
inner
|
||||
.emulation_profiles
|
||||
.get(&input.registration_id)
|
||||
.cloned()
|
||||
.unwrap_or_else(|| "none".into()),
|
||||
*inner
|
||||
.canvases
|
||||
.get(&input.registration_id)
|
||||
.ok_or("Browser viewport is unavailable")?,
|
||||
)
|
||||
};
|
||||
let mut next = canvas;
|
||||
next.presentation.viewport = input.viewport;
|
||||
if previous == preset && canvas.presentation == next.presentation {
|
||||
return Ok(());
|
||||
}
|
||||
if let Err(error) = self.configure_webkit(app, ®istration, next, preset) {
|
||||
let _ = self.configure_webkit(app, ®istration, canvas, &previous);
|
||||
return Err(error);
|
||||
}
|
||||
let view = app
|
||||
.get_webview(®istration.webview_label)
|
||||
.ok_or("Browser preview is unavailable")?;
|
||||
if let Err(error) = view.reload() {
|
||||
let _ = self.configure_webkit(app, ®istration, canvas, &previous);
|
||||
return Err(error.to_string());
|
||||
}
|
||||
let mut inner = self.inner.0.lock().map_err(|e| e.to_string())?;
|
||||
if inner.registrations.get(&input.registration_id) != Some(®istration) {
|
||||
return Err("Browser target changed before emulation publication".into());
|
||||
}
|
||||
inner
|
||||
.emulation_profiles
|
||||
.insert(input.registration_id.clone(), preset.into());
|
||||
inner.canvases.insert(input.registration_id.clone(), next);
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
240
packages/tauri-app/src-tauri/src/browser_navigation_webkit.rs
Normal file
240
packages/tauri-app/src-tauri/src/browser_navigation_webkit.rs
Normal file
|
|
@ -0,0 +1,240 @@
|
|||
//! Native load completion/failure for the private backend browser channel.
|
||||
use super::*;
|
||||
use std::sync::mpsc;
|
||||
|
||||
#[cfg(target_os = "linux")]
|
||||
pub(super) fn install(
|
||||
view: &tauri::Webview,
|
||||
controller: BrowserController,
|
||||
registration: Registration,
|
||||
) -> Result<(), String> {
|
||||
use std::cell::Cell;
|
||||
use std::rc::Rc;
|
||||
use webkit2gtk::WebViewExt;
|
||||
let (sender, receiver) = mpsc::sync_channel(1);
|
||||
view.with_webview(move |native| {
|
||||
let token = Rc::new(Cell::new(0u64));
|
||||
let started_token = token.clone();
|
||||
let started = controller.clone();
|
||||
let expected = registration.clone();
|
||||
native
|
||||
.inner()
|
||||
.connect_load_changed(move |view, event| match event {
|
||||
webkit2gtk::LoadEvent::Started => {
|
||||
started_token.set(started_token.get().wrapping_add(1));
|
||||
if let Some(url) = view.uri().filter(|url| url.as_str() != "about:blank") {
|
||||
started.mark_page_started(
|
||||
&expected.registration_id,
|
||||
expected.generation,
|
||||
started_token.get(),
|
||||
url.to_string(),
|
||||
);
|
||||
}
|
||||
}
|
||||
webkit2gtk::LoadEvent::Finished => started.mark_page_finished(
|
||||
&expected.registration_id,
|
||||
expected.generation,
|
||||
started_token.get(),
|
||||
None,
|
||||
),
|
||||
_ => {}
|
||||
});
|
||||
native.inner().connect_load_failed(move |_, _, _, error| {
|
||||
controller.mark_page_finished(
|
||||
®istration.registration_id,
|
||||
registration.generation,
|
||||
token.get(),
|
||||
Some(format!("Browser navigation failed: {error}")),
|
||||
);
|
||||
false
|
||||
});
|
||||
let _ = sender.send(());
|
||||
})
|
||||
.map_err(|error| error.to_string())?;
|
||||
receiver
|
||||
.recv_timeout(Duration::from_secs(5))
|
||||
.map_err(|_| "Timed out configuring native navigation".to_string())
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
mod macos {
|
||||
use super::*;
|
||||
use std::ffi::{c_char, c_void, CStr};
|
||||
struct Observer {
|
||||
controller: BrowserController,
|
||||
registration: Registration,
|
||||
}
|
||||
extern "C" {
|
||||
fn cn_observe_load(
|
||||
view: *mut c_void,
|
||||
callback: unsafe extern "C" fn(*mut c_void, u64, i32, *const c_char, *const c_char),
|
||||
release: unsafe extern "C" fn(*mut c_void),
|
||||
context: *mut c_void,
|
||||
) -> i32;
|
||||
pub(super) fn cn_navigate(view: *mut c_void, url: *const c_char) -> u64;
|
||||
}
|
||||
unsafe extern "C" fn released(context: *mut c_void) {
|
||||
drop(unsafe { Box::from_raw(context.cast::<Observer>()) });
|
||||
}
|
||||
unsafe extern "C" fn changed(
|
||||
context: *mut c_void,
|
||||
token: u64,
|
||||
event: i32,
|
||||
url: *const c_char,
|
||||
error: *const c_char,
|
||||
) {
|
||||
let observer = unsafe { &*context.cast::<Observer>() };
|
||||
let registration = &observer.registration;
|
||||
if event == 0 {
|
||||
if !url.is_null() {
|
||||
let url = unsafe { CStr::from_ptr(url) }
|
||||
.to_string_lossy()
|
||||
.into_owned();
|
||||
if url != "about:blank" {
|
||||
observer.controller.mark_page_started(
|
||||
®istration.registration_id,
|
||||
registration.generation,
|
||||
token,
|
||||
url,
|
||||
);
|
||||
}
|
||||
}
|
||||
} else {
|
||||
let failure = (event == 2).then(|| {
|
||||
if error.is_null() {
|
||||
"Browser navigation failed".into()
|
||||
} else {
|
||||
format!(
|
||||
"Browser navigation failed: {}",
|
||||
unsafe { CStr::from_ptr(error) }.to_string_lossy()
|
||||
)
|
||||
}
|
||||
});
|
||||
observer.controller.mark_page_finished(
|
||||
®istration.registration_id,
|
||||
registration.generation,
|
||||
token,
|
||||
failure,
|
||||
);
|
||||
}
|
||||
}
|
||||
pub(crate) fn install(
|
||||
view: &tauri::Webview,
|
||||
controller: BrowserController,
|
||||
registration: Registration,
|
||||
) -> Result<(), String> {
|
||||
let (sender, receiver) = mpsc::sync_channel(1);
|
||||
view.with_webview(move |native| {
|
||||
let context = Box::into_raw(Box::new(Observer {
|
||||
controller,
|
||||
registration,
|
||||
}))
|
||||
.cast::<c_void>();
|
||||
let ok = unsafe { cn_observe_load(native.inner(), changed, released, context) };
|
||||
if ok == 0 {
|
||||
unsafe { released(context) };
|
||||
}
|
||||
let _ = sender.send(if ok == 0 {
|
||||
Err("Native navigation delegate is unavailable".to_string())
|
||||
} else {
|
||||
Ok(())
|
||||
});
|
||||
})
|
||||
.map_err(|error| error.to_string())?;
|
||||
receiver
|
||||
.recv_timeout(Duration::from_secs(5))
|
||||
.map_err(|_| "Timed out configuring native navigation".to_string())?
|
||||
}
|
||||
}
|
||||
#[cfg(target_os = "macos")]
|
||||
pub(super) use macos::install;
|
||||
|
||||
impl BrowserController {
|
||||
pub(super) fn navigate_and_wait(
|
||||
&self,
|
||||
app: &AppHandle,
|
||||
registration: &Registration,
|
||||
view: &tauri::Webview,
|
||||
url: &tauri::Url,
|
||||
deadline: Instant,
|
||||
fence: &crate::native_request::RequestFence,
|
||||
) -> Result<(), String> {
|
||||
self.check_target_identity(app, registration, deadline, fence)?;
|
||||
let previous = view.url().map_err(|error| error.to_string())?;
|
||||
let mut previous_base = previous.clone();
|
||||
previous_base.set_fragment(None);
|
||||
let mut target_base = url.clone();
|
||||
target_base.set_fragment(None);
|
||||
let fragment_only = previous != *url && previous_base == target_base;
|
||||
self.invalidate_page(®istration.registration_id);
|
||||
let sequence = self.prepare_page_load(®istration.registration_id, url.as_str())?;
|
||||
let expected = registration.clone();
|
||||
let controller = self.clone();
|
||||
let app = app.clone();
|
||||
let queued_fence = fence.clone();
|
||||
let target = url.clone();
|
||||
let (sender, receiver) = mpsc::sync_channel(1);
|
||||
view.with_webview(move |native| {
|
||||
let result: Result<(), String> = (|| {
|
||||
controller.check_target_identity(&app, &expected, deadline, &queued_fence)?;
|
||||
if controller.page_load_expectation(&expected.registration_id, target.as_str())?
|
||||
!= Some(sequence)
|
||||
{
|
||||
return Err("Browser preview navigation is no longer current".into());
|
||||
}
|
||||
#[cfg(target_os = "linux")]
|
||||
{
|
||||
use webkit2gtk::WebViewExt;
|
||||
native.inner().load_uri(target.as_str());
|
||||
}
|
||||
#[cfg(target_os = "macos")]
|
||||
{
|
||||
let target = std::ffi::CString::new(target.as_str())
|
||||
.map_err(|error| error.to_string())?;
|
||||
let token = unsafe { macos::cn_navigate(native.inner(), target.as_ptr()) };
|
||||
if token == 0 {
|
||||
return Err("Native browser navigation was refused".into());
|
||||
}
|
||||
let mut inner = controller
|
||||
.inner
|
||||
.0
|
||||
.lock()
|
||||
.map_err(|error| error.to_string())?;
|
||||
let expectation = inner
|
||||
.page_load_expectations
|
||||
.get_mut(&expected.registration_id)
|
||||
.filter(|entry| entry.sequence == sequence)
|
||||
.ok_or("Browser preview navigation is no longer current")?;
|
||||
// WKWebView can return a navigation object for a fragment
|
||||
// change without issuing document completion callbacks.
|
||||
if !fragment_only {
|
||||
expectation.navigation_id = Some(token);
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
})();
|
||||
let _ = sender.send(result);
|
||||
})
|
||||
.map_err(|error| error.to_string())?;
|
||||
receiver
|
||||
.recv_timeout(deadline.saturating_duration_since(Instant::now()))
|
||||
.map_err(|_| "Timed out dispatching native navigation".to_string())??;
|
||||
if fragment_only {
|
||||
loop {
|
||||
self.check_target(view.app_handle(), registration, deadline, fence)?;
|
||||
let current = view.url().map_err(|error| error.to_string())?;
|
||||
if current == *url {
|
||||
self.complete_same_document_navigation(
|
||||
registration,
|
||||
sequence,
|
||||
current.as_str(),
|
||||
)?;
|
||||
break;
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(10));
|
||||
}
|
||||
}
|
||||
self.wait_for_page_load(®istration.registration_id, sequence, deadline)?;
|
||||
self.check_target(view.app_handle(), registration, deadline, fence)
|
||||
}
|
||||
}
|
||||
445
packages/tauri-app/src-tauri/src/browser_page_operation.rs
Normal file
445
packages/tauri-app/src-tauri/src/browser_page_operation.rs
Normal file
|
|
@ -0,0 +1,445 @@
|
|||
//! Closed semantic operation schema; fixed programs come from the shared registry.
|
||||
use serde::Deserialize;
|
||||
use serde_json::{json, Map, Value};
|
||||
use std::sync::OnceLock;
|
||||
const GROUP: &str = "codenomad-automation";
|
||||
const INVALID: &str = "Unsupported or invalid browser page operation parameters";
|
||||
fn registry() -> &'static Value {
|
||||
static REGISTRY: OnceLock<Value> = OnceLock::new();
|
||||
REGISTRY.get_or_init(|| {
|
||||
serde_json::from_str(include_str!(
|
||||
"../../../server/src/automation/page/functions.generated.json"
|
||||
))
|
||||
.expect("generated fixed function registry")
|
||||
})
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
#[serde(tag = "op", rename_all = "kebab-case", deny_unknown_fields)]
|
||||
pub(super) enum Operation {
|
||||
Tree,
|
||||
Viewport,
|
||||
ReleaseHandles,
|
||||
Checkpoint,
|
||||
Resolve {
|
||||
node: u64,
|
||||
},
|
||||
ScrollIntoView {
|
||||
node: u64,
|
||||
},
|
||||
Geometry {
|
||||
node: u64,
|
||||
},
|
||||
HitTest {
|
||||
x: f64,
|
||||
y: f64,
|
||||
},
|
||||
Call {
|
||||
function: String,
|
||||
handle: Option<String>,
|
||||
args: Vec<Value>,
|
||||
},
|
||||
Pointer {
|
||||
event: Value,
|
||||
},
|
||||
Wheel {
|
||||
event: Value,
|
||||
},
|
||||
Keyboard {
|
||||
event: Value,
|
||||
},
|
||||
InsertText {
|
||||
text: String,
|
||||
},
|
||||
}
|
||||
fn keys(value: &Value, allowed: &[&str], required: &[&str]) -> Result<(), String> {
|
||||
let object = value.as_object().ok_or(INVALID)?;
|
||||
if object.keys().any(|key| !allowed.contains(&key.as_str()))
|
||||
|| required.iter().any(|key| !object.contains_key(*key))
|
||||
{
|
||||
return Err(INVALID.into());
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
fn number(value: &Value, max: f64) -> bool {
|
||||
value
|
||||
.as_f64()
|
||||
.is_some_and(|value| value.is_finite() && value.abs() <= max)
|
||||
}
|
||||
fn integer(value: &Value, max: u64) -> bool {
|
||||
value.as_u64().is_some_and(|value| value <= max)
|
||||
}
|
||||
fn handle(value: &Value) -> bool {
|
||||
value
|
||||
.as_str()
|
||||
.is_some_and(|value| !value.is_empty() && value.len() <= 512)
|
||||
}
|
||||
pub(super) fn envelope(method: &str, value: &Value) -> Result<(), String> {
|
||||
let mut allowed = vec!["sessionID", "operationID"];
|
||||
let mut required = allowed.clone();
|
||||
if method == "browser.page.send" {
|
||||
allowed.extend(["generation", "operation"]);
|
||||
required.extend(["generation", "operation"]);
|
||||
} else if method == "browser.page.check" {
|
||||
allowed.push("generation");
|
||||
required.push("generation");
|
||||
} else if method == "browser.page.end" {
|
||||
allowed.push("generation");
|
||||
}
|
||||
keys(value, &allowed, &required)?;
|
||||
if value.get("generation").is_some_and(|value| !handle(value)) {
|
||||
return Err(INVALID.into());
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(super) fn parse(value: Value) -> Result<Operation, String> {
|
||||
if value.to_string().len() > 256 * 1024 {
|
||||
return Err(INVALID.into());
|
||||
}
|
||||
let operation: Operation =
|
||||
serde_json::from_value(value.clone()).map_err(|_| INVALID.to_string())?;
|
||||
match &operation {
|
||||
Operation::Tree
|
||||
| Operation::Viewport
|
||||
| Operation::ReleaseHandles
|
||||
| Operation::Checkpoint => keys(&value, &["op"], &["op"])?,
|
||||
Operation::Resolve { node }
|
||||
| Operation::Geometry { node }
|
||||
| Operation::ScrollIntoView { node }
|
||||
if *node == 0 || *node > 9_007_199_254_740_991 =>
|
||||
{
|
||||
return Err(INVALID.into())
|
||||
}
|
||||
Operation::HitTest { x, y }
|
||||
if !x.is_finite()
|
||||
|| !y.is_finite()
|
||||
|| x.abs() > 10_000_000.0
|
||||
|| y.abs() > 10_000_000.0
|
||||
|| x.fract() != 0.0
|
||||
|| y.fract() != 0.0 =>
|
||||
{
|
||||
return Err(INVALID.into())
|
||||
}
|
||||
Operation::Call {
|
||||
function,
|
||||
handle: target,
|
||||
args,
|
||||
} => {
|
||||
let definition = registry().get(function).ok_or(INVALID)?;
|
||||
if definition["scope"] == "element" {
|
||||
if !target.as_ref().is_some_and(|id| handle(&json!(id))) {
|
||||
return Err(INVALID.into());
|
||||
}
|
||||
} else if value.get("handle").is_some() {
|
||||
return Err(INVALID.into());
|
||||
}
|
||||
let rules = definition["args"].as_array().ok_or(INVALID)?;
|
||||
if rules.len() != args.len() {
|
||||
return Err(INVALID.into());
|
||||
}
|
||||
for (rule, argument) in rules.iter().zip(args) {
|
||||
let valid = match rule.as_str() {
|
||||
Some("string") => argument.is_string(),
|
||||
Some("boolean") => argument.is_boolean(),
|
||||
Some("strings") => argument
|
||||
.as_array()
|
||||
.is_some_and(|items| items.iter().all(Value::is_string)),
|
||||
Some("handle") => {
|
||||
keys(argument, &["handle"], &["handle"]).is_ok()
|
||||
&& handle(&argument["handle"])
|
||||
}
|
||||
_ => false,
|
||||
};
|
||||
if !valid {
|
||||
return Err(INVALID.into());
|
||||
}
|
||||
}
|
||||
}
|
||||
Operation::Pointer { event }
|
||||
| Operation::Wheel { event }
|
||||
| Operation::Keyboard { event } => {
|
||||
let keyboard = matches!(&operation, Operation::Keyboard { .. });
|
||||
let wheel = matches!(&operation, Operation::Wheel { .. });
|
||||
if keyboard {
|
||||
keys(
|
||||
event,
|
||||
&[
|
||||
"type",
|
||||
"key",
|
||||
"code",
|
||||
"windowsVirtualKeyCode",
|
||||
"modifiers",
|
||||
"text",
|
||||
"unmodifiedText",
|
||||
"location",
|
||||
"timestamp",
|
||||
"buttons",
|
||||
],
|
||||
&["type"],
|
||||
)?;
|
||||
if !matches!(
|
||||
event["type"].as_str(),
|
||||
Some("keyDown" | "rawKeyDown" | "keyUp")
|
||||
) {
|
||||
return Err(INVALID.into());
|
||||
}
|
||||
for key in ["key", "code", "text", "unmodifiedText"] {
|
||||
if let Some(value) = event.get(key) {
|
||||
if !value
|
||||
.as_str()
|
||||
.is_some_and(|text| !text.is_empty() && text.len() <= 256)
|
||||
{
|
||||
return Err(INVALID.into());
|
||||
}
|
||||
}
|
||||
}
|
||||
} else {
|
||||
let allowed = if wheel {
|
||||
vec![
|
||||
"type",
|
||||
"x",
|
||||
"y",
|
||||
"deltaX",
|
||||
"deltaY",
|
||||
"modifiers",
|
||||
"timestamp",
|
||||
]
|
||||
} else {
|
||||
vec![
|
||||
"type",
|
||||
"x",
|
||||
"y",
|
||||
"button",
|
||||
"buttons",
|
||||
"clickCount",
|
||||
"modifiers",
|
||||
"timestamp",
|
||||
]
|
||||
};
|
||||
keys(
|
||||
event,
|
||||
&allowed,
|
||||
if wheel {
|
||||
&["type", "x", "y", "deltaX", "deltaY"]
|
||||
} else {
|
||||
&["type", "x", "y"]
|
||||
},
|
||||
)?;
|
||||
if !number(&event["x"], 10_000_000.0) || !number(&event["y"], 10_000_000.0) {
|
||||
return Err(INVALID.into());
|
||||
}
|
||||
if wheel {
|
||||
if event["type"] != "mouseWheel"
|
||||
|| !number(&event["deltaX"], 10_000_000.0)
|
||||
|| !number(&event["deltaY"], 10_000_000.0)
|
||||
{
|
||||
return Err(INVALID.into());
|
||||
}
|
||||
} else {
|
||||
if !matches!(
|
||||
event["type"].as_str(),
|
||||
Some("mouseMoved" | "mousePressed" | "mouseReleased")
|
||||
) {
|
||||
return Err(INVALID.into());
|
||||
}
|
||||
if let Some(button) = event.get("button") {
|
||||
if !matches!(button.as_str(), Some("left" | "right" | "middle" | "none")) {
|
||||
return Err(INVALID.into());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
for (key, max) in [
|
||||
("modifiers", 15),
|
||||
("buttons", 7),
|
||||
("clickCount", 3),
|
||||
("windowsVirtualKeyCode", 65535),
|
||||
("location", 3),
|
||||
] {
|
||||
if let Some(value) = event.get(key) {
|
||||
if !integer(value, max) {
|
||||
return Err(INVALID.into());
|
||||
}
|
||||
}
|
||||
}
|
||||
if let Some(value) = event.get("timestamp") {
|
||||
if !number(value, 9_007_199_254_740_991.0) {
|
||||
return Err(INVALID.into());
|
||||
}
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
Ok(operation)
|
||||
}
|
||||
|
||||
impl Operation {
|
||||
pub(super) fn cdp(&self) -> Option<(&'static str, Value)> {
|
||||
Some(match self {
|
||||
Self::Tree => ("Accessibility.getFullAXTree", json!({})),
|
||||
Self::Resolve { node } => (
|
||||
"DOM.resolveNode",
|
||||
json!({"backendNodeId":node,"objectGroup":GROUP}),
|
||||
),
|
||||
Self::ScrollIntoView { node } => {
|
||||
("DOM.scrollIntoViewIfNeeded", json!({"backendNodeId":node}))
|
||||
}
|
||||
Self::Geometry { node } => ("DOM.getContentQuads", json!({"backendNodeId":node})),
|
||||
Self::Viewport => ("Page.getLayoutMetrics", json!({})),
|
||||
Self::HitTest { x, y } => (
|
||||
"DOM.getNodeForLocation",
|
||||
json!({"x":*x as i64,"y":*y as i64,"includeUserAgentShadowDOM":false,"ignorePointerEventsNone":false}),
|
||||
),
|
||||
Self::Call {
|
||||
function,
|
||||
handle,
|
||||
args,
|
||||
} => {
|
||||
let definition = ®istry()[function];
|
||||
if definition["scope"] == "element" {
|
||||
(
|
||||
"Runtime.callFunctionOn",
|
||||
json!({"objectId":handle,"functionDeclaration":definition["source"],"arguments":args.iter().map(|arg| if arg.get("handle").is_some() { json!({"objectId":arg["handle"]}) } else { json!({"value":arg}) }).collect::<Vec<_>>(),"returnByValue":definition["handle"] != true,"objectGroup":GROUP}),
|
||||
)
|
||||
} else if definition["handle"] == true {
|
||||
(
|
||||
"Runtime.evaluate",
|
||||
json!({"expression":definition["source"],"objectGroup":GROUP}),
|
||||
)
|
||||
} else {
|
||||
(
|
||||
"Runtime.evaluate",
|
||||
json!({"expression":definition["source"],"returnByValue":true}),
|
||||
)
|
||||
}
|
||||
}
|
||||
Self::Pointer { event } | Self::Wheel { event } => {
|
||||
("Input.dispatchMouseEvent", event.clone())
|
||||
}
|
||||
Self::Keyboard { event } => ("Input.dispatchKeyEvent", event.clone()),
|
||||
Self::InsertText { text } => ("Input.insertText", json!({"text":text})),
|
||||
Self::ReleaseHandles => ("Runtime.releaseObjectGroup", json!({"objectGroup":GROUP})),
|
||||
Self::Checkpoint => return None,
|
||||
})
|
||||
}
|
||||
pub(super) fn result(&self, raw: Value) -> Value {
|
||||
match self {
|
||||
Self::Tree => {
|
||||
json!({"nodes":raw["nodes"].as_array().map(|nodes| nodes.iter().map(semantic_node).collect::<Vec<_>>()).unwrap_or_default()})
|
||||
}
|
||||
Self::Resolve { .. } => optional("handle", raw["object"].get("objectId")),
|
||||
Self::Call { function, .. } => {
|
||||
if let Some(error) = raw.get("exceptionDetails") {
|
||||
return json!({"exception":error["exception"]["description"].as_str().or_else(||error["text"].as_str()).unwrap_or("")});
|
||||
}
|
||||
if registry()[function]["handle"] == true {
|
||||
optional("handle", raw["result"].get("objectId"))
|
||||
} else {
|
||||
optional("value", raw["result"].get("value"))
|
||||
}
|
||||
}
|
||||
Self::Viewport => {
|
||||
let viewport = raw
|
||||
.get("cssLayoutViewport")
|
||||
.filter(|value| !value.is_null())
|
||||
.unwrap_or(&raw["layoutViewport"]);
|
||||
let mut result = optional("width", viewport.get("clientWidth"));
|
||||
if let Some(height) = viewport.get("clientHeight") {
|
||||
result["height"] = height.clone();
|
||||
}
|
||||
result
|
||||
}
|
||||
Self::HitTest { .. } => optional("node", raw.get("backendNodeId")),
|
||||
_ => raw,
|
||||
}
|
||||
}
|
||||
}
|
||||
fn optional(key: &str, value: Option<&Value>) -> Value {
|
||||
let mut result = Map::new();
|
||||
if let Some(value) = value {
|
||||
result.insert(key.into(), value.clone());
|
||||
}
|
||||
Value::Object(result)
|
||||
}
|
||||
fn semantic_node(raw: &Value) -> Value {
|
||||
let mut result = Map::new();
|
||||
for (key, source) in [
|
||||
("id", "nodeId"),
|
||||
("parent", "parentId"),
|
||||
("children", "childIds"),
|
||||
("ignored", "ignored"),
|
||||
("node", "backendDOMNodeId"),
|
||||
] {
|
||||
if let Some(value) = raw.get(source) {
|
||||
result.insert(key.into(), value.clone());
|
||||
}
|
||||
}
|
||||
for key in ["role", "name", "description", "value"] {
|
||||
if let Some(value) = raw[key].get("value") {
|
||||
result.insert(key.into(), value.clone());
|
||||
}
|
||||
}
|
||||
if let Some(properties) = raw["properties"].as_array() {
|
||||
result.insert(
|
||||
"properties".into(),
|
||||
Value::Array(
|
||||
properties
|
||||
.iter()
|
||||
.map(|property| {
|
||||
let mut result = optional("name", property.get("name"));
|
||||
if let Some(value) = property["value"].get("value") {
|
||||
result["value"] = value.clone();
|
||||
}
|
||||
result
|
||||
})
|
||||
.collect(),
|
||||
),
|
||||
);
|
||||
}
|
||||
Value::Object(result)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
#[test]
|
||||
fn closed_operations_refuse_scripts_unknown_fields_and_bad_args() {
|
||||
let cases: Value = serde_json::from_str(include_str!(
|
||||
"../../../server/src/automation/page/contract.cases.json"
|
||||
))
|
||||
.unwrap();
|
||||
for value in cases["valid"].as_array().unwrap() {
|
||||
assert!(
|
||||
parse(value.clone()).is_ok(),
|
||||
"valid operation rejected: {value}"
|
||||
);
|
||||
}
|
||||
for value in cases["invalid"].as_array().unwrap() {
|
||||
assert!(
|
||||
parse(value.clone()).is_err(),
|
||||
"invalid operation accepted: {value}"
|
||||
);
|
||||
}
|
||||
for bad in [
|
||||
json!({"op":"Runtime.evaluate","expression":"alert(1)"}),
|
||||
json!({"op":"tree","extra":true}),
|
||||
json!({"op":"call","function":"mark-input","args":[]}),
|
||||
json!({"op":"call","function":"select","handle":"h","args":["wrong"]}),
|
||||
json!({"op":"pointer","event":{"type":"mouseMoved","x":0,"y":0,"arbitrary":true}}),
|
||||
] {
|
||||
assert!(parse(bad).is_err());
|
||||
}
|
||||
let call = parse(
|
||||
json!({"op":"call","function":"set-value","handle":"h","args":["\";evil()",false]}),
|
||||
)
|
||||
.unwrap();
|
||||
let (_, params) = call.cdp().unwrap();
|
||||
assert_eq!(params["arguments"][0]["value"], "\";evil()");
|
||||
assert!(!params["functionDeclaration"]
|
||||
.as_str()
|
||||
.unwrap()
|
||||
.contains("evil()"));
|
||||
assert!(parse(json!({"op":"checkpoint"})).unwrap().cdp().is_none());
|
||||
}
|
||||
}
|
||||
166
packages/tauri-app/src-tauri/src/browser_page_webkit.rs
Normal file
166
packages/tauri-app/src-tauri/src/browser_page_webkit.rs
Normal file
|
|
@ -0,0 +1,166 @@
|
|||
//! Fixed isolated-world reads with registration/deadline fences on both sides.
|
||||
use super::*;
|
||||
|
||||
impl BrowserController {
|
||||
pub(super) fn webkit_read(
|
||||
&self,
|
||||
app: &AppHandle,
|
||||
lease: &PageLease,
|
||||
operation: &Value,
|
||||
deadline: Instant,
|
||||
fence: &crate::native_request::RequestFence,
|
||||
) -> Result<Value, String> {
|
||||
self.webkit_operation(app, lease, operation, deadline, fence, false)
|
||||
}
|
||||
fn webkit_operation(
|
||||
&self,
|
||||
app: &AppHandle,
|
||||
lease: &PageLease,
|
||||
operation: &Value,
|
||||
deadline: Instant,
|
||||
fence: &crate::native_request::RequestFence,
|
||||
cleanup: bool,
|
||||
) -> Result<Value, String> {
|
||||
let webview = app
|
||||
.get_webview(&lease.registration.webview_label)
|
||||
.ok_or("Browser preview is no longer available")?;
|
||||
let controller = self.clone();
|
||||
let expected = lease.clone();
|
||||
let app = app.clone();
|
||||
let fence = fence.clone();
|
||||
let native = matches!(
|
||||
operation["op"].as_str(),
|
||||
Some("pointer" | "wheel" | "keyboard" | "insert-text")
|
||||
);
|
||||
let version = self.document_version(&lease.registration)?;
|
||||
let read = if native {
|
||||
crate::embedded_view::input::Read::Input {
|
||||
document: lease
|
||||
.document
|
||||
.clone()
|
||||
.ok_or("Missing WebKit document identity")?,
|
||||
operation: operation.clone(),
|
||||
held: lease.held.clone(),
|
||||
}
|
||||
} else {
|
||||
crate::embedded_view::input::Read::Page(operation.to_string())
|
||||
};
|
||||
let bytes = crate::embedded_view::input::read(
|
||||
&webview,
|
||||
read,
|
||||
if cleanup {
|
||||
deadline
|
||||
} else {
|
||||
deadline.min(lease.expires)
|
||||
},
|
||||
move || {
|
||||
fence.check()?;
|
||||
if cleanup {
|
||||
if !controller
|
||||
.inner
|
||||
.0
|
||||
.lock()
|
||||
.map_err(|error| error.to_string())?
|
||||
.page_leases
|
||||
.get(&expected.operation_id)
|
||||
.is_some_and(|lease| {
|
||||
lease.generation == expected.generation
|
||||
&& lease.expires == expected.expires
|
||||
})
|
||||
{
|
||||
return Err("Browser page lease is no longer current".into());
|
||||
}
|
||||
} else {
|
||||
controller.check_page_state(&expected, deadline)?;
|
||||
}
|
||||
if native && controller.document_version(&expected.registration)? != version {
|
||||
return Err("Browser page changed during native input".into());
|
||||
}
|
||||
if controller.resolve(&app, &expected.session_id)? != expected.registration {
|
||||
return Err("Browser page lease is no longer current".into());
|
||||
}
|
||||
Ok(())
|
||||
},
|
||||
)?;
|
||||
let response: Value = serde_json::from_slice(&bytes).map_err(|error| error.to_string())?;
|
||||
if !response["document"]
|
||||
.as_str()
|
||||
.is_some_and(|token| !token.is_empty() && token.len() <= 128)
|
||||
|| !response["result"].is_object()
|
||||
{
|
||||
return Err("Invalid native semantic read response".into());
|
||||
}
|
||||
Ok(response)
|
||||
}
|
||||
|
||||
pub(super) fn webkit_cleanup(
|
||||
&self,
|
||||
app: &AppHandle,
|
||||
session: &str,
|
||||
operation: &str,
|
||||
deadline: Instant,
|
||||
fence: &crate::native_request::RequestFence,
|
||||
) {
|
||||
let lease = self.inner.0.lock().ok().and_then(|inner| {
|
||||
inner
|
||||
.page_leases
|
||||
.get(operation)
|
||||
.filter(|lease| lease.session_id == session)
|
||||
.cloned()
|
||||
});
|
||||
let Some(lease) = lease else { return };
|
||||
let releases = lease
|
||||
.held
|
||||
.lock()
|
||||
.map(|held| held.clone())
|
||||
.unwrap_or_default();
|
||||
for (identity, release) in releases.into_iter().rev() {
|
||||
let mut result = self.webkit_operation(app, &lease, &release, deadline, fence, true);
|
||||
// A same-document navigation can fence an unsent release. Renew its
|
||||
// document check once, only while that release is still outstanding.
|
||||
// Admitted mutations and completed releases are never replayed.
|
||||
if result
|
||||
.as_ref()
|
||||
.is_err_and(|error| error == "Browser page changed during native input")
|
||||
&& lease
|
||||
.held
|
||||
.lock()
|
||||
.is_ok_and(|held| held.iter().any(|(id, _)| id == &identity))
|
||||
{
|
||||
result = self.webkit_operation(app, &lease, &release, deadline, fence, true);
|
||||
}
|
||||
if let Err(error) = result {
|
||||
if std::env::var_os("CN_INPUT_TRACE").is_some() {
|
||||
eprintln!("native input cleanup: {error}");
|
||||
}
|
||||
}
|
||||
}
|
||||
if let Ok(mut held) = lease.held.lock() {
|
||||
held.clear();
|
||||
};
|
||||
}
|
||||
pub(super) fn webkit_expire(
|
||||
&self,
|
||||
app: &AppHandle,
|
||||
deadline: Instant,
|
||||
fence: &crate::native_request::RequestFence,
|
||||
) {
|
||||
let expired = self
|
||||
.inner
|
||||
.0
|
||||
.lock()
|
||||
.map(|inner| {
|
||||
inner
|
||||
.page_leases
|
||||
.values()
|
||||
.filter(|lease| lease.expires <= Instant::now())
|
||||
.cloned()
|
||||
.collect::<Vec<_>>()
|
||||
})
|
||||
.unwrap_or_default();
|
||||
for lease in expired {
|
||||
self.webkit_cleanup(app, &lease.session_id, &lease.operation_id, deadline, fence);
|
||||
self.end_page(&lease.session_id, &lease.operation_id);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1234,7 +1234,7 @@ impl CliProcessManager {
|
|||
#[cfg(not(windows))]
|
||||
let mut c = Command::new(&cmd.program);
|
||||
#[cfg(not(windows))]
|
||||
c.args(&cmd.args);
|
||||
c.args(&cmd.args).stdin(Stdio::piped());
|
||||
#[cfg(target_os = "linux")]
|
||||
if shell_env.is_none() {
|
||||
crate::appimage_environment::use_host_environment(&mut c);
|
||||
|
|
@ -1260,8 +1260,8 @@ impl CliProcessManager {
|
|||
"0"
|
||||
},
|
||||
);
|
||||
#[cfg(windows)]
|
||||
c.env("CODENOMAD_NATIVE_PARENT", "1");
|
||||
c.env("CODENOMAD_NATIVE_BROWSER_AUTOMATION", if cfg!(windows) { "1" } else { "0" });
|
||||
configure_spawn(&mut c);
|
||||
if let Some(ref cwd) = cwd {
|
||||
c.current_dir(cwd);
|
||||
|
|
@ -1499,24 +1499,19 @@ impl CliProcessManager {
|
|||
&& line.starts_with(crate::native_request::REQUEST_PREFIX)
|
||||
{
|
||||
if let Some(request) = crate::native_request::parse(line) {
|
||||
let now = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|duration| duration.as_millis() as u64)
|
||||
.unwrap_or(u64::MAX);
|
||||
let result = if now >= request.deadline {
|
||||
Err("Native request expired before execution".to_string())
|
||||
} else {
|
||||
let current = manager.clone();
|
||||
let fence = crate::native_request::RequestFence::new(request.deadline, move || current.is_current_generation(generation));
|
||||
if let Some(response) = crate::native_request::dispatch(request, fence, |request, fence| {
|
||||
crate::handle_native_request(
|
||||
app,
|
||||
&request.method,
|
||||
request.params,
|
||||
request.params.clone(),
|
||||
request.deadline,
|
||||
fence,
|
||||
)
|
||||
};
|
||||
manager.write_native_response(
|
||||
generation,
|
||||
&crate::native_request::response(&request.id, result),
|
||||
);
|
||||
}) {
|
||||
manager.write_native_response(generation, &response);
|
||||
}
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -6,11 +6,20 @@ use std::sync::{Arc, Mutex};
|
|||
use tauri::{AppHandle, Manager, PhysicalPosition, PhysicalSize, Runtime};
|
||||
|
||||
#[cfg(target_os = "linux")]
|
||||
#[path = "embedded_view/linux.rs"]
|
||||
mod linux;
|
||||
#[cfg(target_os = "macos")]
|
||||
#[path = "embedded_view/macos.rs"]
|
||||
mod macos;
|
||||
#[cfg(windows)]
|
||||
#[path = "embedded_view/windows.rs"]
|
||||
mod windows;
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
#[path = "embedded_view/input.rs"]
|
||||
pub(crate) mod input;
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
#[path = "embedded_view/mobile.rs"]
|
||||
pub(crate) mod mobile;
|
||||
|
||||
/// Full, normalized desktop identity. The caller resolves config aliases before startup.
|
||||
pub(crate) struct StorageIdentity {
|
||||
|
|
|
|||
23
packages/tauri-app/src-tauri/src/embedded_view/input.h
Normal file
23
packages/tauri-app/src-tauri/src/embedded_view/input.h
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
#pragma once
|
||||
#include <stddef.h>
|
||||
#include <stdint.h>
|
||||
|
||||
/* UI-thread only. Bytes passed to completion are borrowed until it returns.
|
||||
* The caller owns the exact guest, visibility/generation and deadline fences.
|
||||
* No function raises a window or injects global desktop input. */
|
||||
typedef void (*CNCompletion)(void *, const unsigned char *, size_t, const char *);
|
||||
enum CNPointer { CN_MOVE, CN_LEFT_DOWN, CN_LEFT_UP, CN_RIGHT_DOWN, CN_RIGHT_UP, CN_DRAG };
|
||||
enum CNModifier { CN_SHIFT = 1, CN_CONTROL = 2, CN_ALT = 4, CN_META = 8 };
|
||||
void cn_checkpoint(void *view, CNCompletion done, void *context);
|
||||
int cn_pointer(void *view, int action, double x, double y, unsigned modifiers);
|
||||
int cn_wheel(void *view, double x, double y, double dx, double dy, unsigned modifiers);
|
||||
int cn_key(void *view, const char *key, unsigned modifiers, int down);
|
||||
int cn_text(void *view, const char *text);
|
||||
void cn_capture(void *view, CNCompletion done, void *context);
|
||||
int cn_install_page_read(void *view);
|
||||
void cn_page_read(void *view, const char *request, unsigned timeout_ms, CNCompletion done, void *context);
|
||||
|
||||
/* Fixed isolated-world program. A page-global value must never leak into it.
|
||||
* The document fields also permit an external fixture to publish observations. */
|
||||
#define CN_CHECKPOINT "JSON.stringify({url:location.href,title:document.title,body:document.body?document.body.innerText:'',width:innerWidth,height:innerHeight,pageGlobal:typeof window.__codenomadPageOnly})"
|
||||
#define CN_WORLD "com.codenomad.preview.automation.v1"
|
||||
180
packages/tauri-app/src-tauri/src/embedded_view/input.rs
Normal file
180
packages/tauri-app/src-tauri/src/embedded_view/input.rs
Normal file
|
|
@ -0,0 +1,180 @@
|
|||
//! UI-thread native reads. Ownership/document fences belong to BrowserController.
|
||||
use std::ffi::{c_char, c_void, CStr, CString};
|
||||
use std::sync::{mpsc, Arc};
|
||||
use std::time::Instant;
|
||||
#[path = "input_native.rs"]
|
||||
mod native;
|
||||
pub(crate) use native::Held;
|
||||
|
||||
type Guard = Arc<dyn Fn() -> Result<(), String> + Send + Sync>;
|
||||
type Completion = unsafe extern "C" fn(*mut c_void, *const u8, usize, *const c_char);
|
||||
extern "C" {
|
||||
fn cn_checkpoint(view: *mut c_void, done: Completion, context: *mut c_void);
|
||||
fn cn_capture(view: *mut c_void, done: Completion, context: *mut c_void);
|
||||
fn cn_install_page_read(view: *mut c_void) -> i32;
|
||||
fn cn_page_read(
|
||||
view: *mut c_void,
|
||||
request: *const c_char,
|
||||
timeout_ms: u32,
|
||||
done: Completion,
|
||||
context: *mut c_void,
|
||||
);
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
pub(crate) enum Read {
|
||||
Checkpoint,
|
||||
Capture,
|
||||
Page(String),
|
||||
Input {
|
||||
document: String,
|
||||
operation: serde_json::Value,
|
||||
held: Held,
|
||||
},
|
||||
}
|
||||
|
||||
struct Pending {
|
||||
sender: mpsc::SyncSender<Result<Vec<u8>, String>>,
|
||||
guard: Guard,
|
||||
limit: usize,
|
||||
input: Option<(*mut c_void, String, serde_json::Value, Held)>,
|
||||
}
|
||||
|
||||
unsafe extern "C" fn completed(
|
||||
context: *mut c_void,
|
||||
bytes: *const u8,
|
||||
length: usize,
|
||||
error: *const c_char,
|
||||
) {
|
||||
// Native adapters invoke exactly once and lend their buffers until return.
|
||||
let pending = unsafe { Box::from_raw(context.cast::<Pending>()) };
|
||||
let result = (|| {
|
||||
(pending.guard)()?;
|
||||
if !error.is_null() {
|
||||
return Err(unsafe { CStr::from_ptr(error) }
|
||||
.to_string_lossy()
|
||||
.into_owned());
|
||||
}
|
||||
if length > pending.limit || (length != 0 && bytes.is_null()) {
|
||||
return Err("Native browser result exceeds its byte budget".into());
|
||||
}
|
||||
let bytes = if length == 0 {
|
||||
Vec::new()
|
||||
} else {
|
||||
unsafe { std::slice::from_raw_parts(bytes, length) }.to_vec()
|
||||
};
|
||||
if let Some((view, document, operation, held)) = &pending.input {
|
||||
let response: serde_json::Value =
|
||||
serde_json::from_slice(&bytes).map_err(|error| error.to_string())?;
|
||||
if response["document"].as_str() != Some(document.as_str()) {
|
||||
return Err("Browser page document changed before input".into());
|
||||
}
|
||||
if operation["op"] == "wheel" && response["result"]["node"].as_u64().is_none() {
|
||||
return Err("Native wheel point is unavailable".into());
|
||||
}
|
||||
unsafe { native::dispatch(*view, operation, held) }?;
|
||||
(pending.guard)()?;
|
||||
return serde_json::to_vec(&serde_json::json!({"document":document,"result":{}}))
|
||||
.map_err(|error| error.to_string());
|
||||
}
|
||||
Ok(bytes)
|
||||
})();
|
||||
let _ = pending.sender.send(result);
|
||||
}
|
||||
|
||||
pub(crate) fn read(
|
||||
webview: &tauri::Webview,
|
||||
operation: Read,
|
||||
deadline: Instant,
|
||||
guard: impl Fn() -> Result<(), String> + Send + Sync + 'static,
|
||||
) -> Result<Vec<u8>, String> {
|
||||
let guard: Guard = Arc::new(guard);
|
||||
guard()?;
|
||||
let (sender, receiver) = mpsc::sync_channel(1);
|
||||
let queued_guard = guard.clone();
|
||||
webview
|
||||
.with_webview(move |native| {
|
||||
if Instant::now() >= deadline {
|
||||
let _ = sender.send(Err("Native request expired before execution".into()));
|
||||
return;
|
||||
}
|
||||
if let Err(error) = queued_guard() {
|
||||
let _ = sender.send(Err(error));
|
||||
return;
|
||||
}
|
||||
#[cfg(target_os = "linux")]
|
||||
let view = {
|
||||
use gtk::glib::object::ObjectType;
|
||||
native.inner().as_ptr().cast::<c_void>()
|
||||
};
|
||||
#[cfg(target_os = "macos")]
|
||||
let view = native.inner();
|
||||
let context = Box::into_raw(Box::new(Pending {
|
||||
sender,
|
||||
guard: queued_guard,
|
||||
limit: match &operation {
|
||||
Read::Checkpoint => 1024 * 1024,
|
||||
Read::Capture => 16 * 1024 * 1024,
|
||||
Read::Page(_) => 4 * 1024 * 1024,
|
||||
Read::Input {..} => 4 * 1024 * 1024,
|
||||
},
|
||||
input:match &operation {Read::Input {document,operation,held}=>Some((view,document.clone(),operation.clone(),held.clone())),_=>None},
|
||||
}))
|
||||
.cast::<c_void>();
|
||||
unsafe {
|
||||
match operation {
|
||||
Read::Checkpoint => cn_checkpoint(view, completed, context),
|
||||
Read::Capture => cn_capture(view, completed, context),
|
||||
Read::Page(request) => {
|
||||
let request = CString::new(request).expect("serialized JSON request");
|
||||
cn_page_read(
|
||||
view,
|
||||
request.as_ptr(),
|
||||
deadline
|
||||
.saturating_duration_since(Instant::now())
|
||||
.as_millis()
|
||||
.clamp(1, u32::MAX as u128) as u32,
|
||||
completed,
|
||||
context,
|
||||
);
|
||||
}
|
||||
Read::Input {operation,..}=> {
|
||||
let preflight=if operation["op"]=="wheel" {serde_json::json!({"op":"hit-test","x":operation["event"]["x"].as_f64().unwrap_or(0.0).floor(),"y":operation["event"]["y"].as_f64().unwrap_or(0.0).floor()})} else {serde_json::json!({"op":"checkpoint"})};
|
||||
let request=CString::new(preflight.to_string()).expect("serialized input preflight");
|
||||
cn_page_read(view,request.as_ptr(),deadline.saturating_duration_since(Instant::now()).as_millis().clamp(1,u32::MAX as u128) as u32,completed,context);
|
||||
},
|
||||
}
|
||||
}
|
||||
})
|
||||
.map_err(|error| error.to_string())?;
|
||||
let result = receiver
|
||||
.recv_timeout(deadline.saturating_duration_since(Instant::now()))
|
||||
.map_err(|_| "Timed out reading the native browser".to_string())??;
|
||||
guard()?;
|
||||
Ok(result)
|
||||
}
|
||||
|
||||
pub(crate) fn install_read(webview: &tauri::Webview) -> Result<(), String> {
|
||||
let (sender, receiver) = mpsc::sync_channel(1);
|
||||
webview
|
||||
.with_webview(move |native| {
|
||||
#[cfg(target_os = "linux")]
|
||||
let view = {
|
||||
use gtk::glib::object::ObjectType;
|
||||
native.inner().as_ptr().cast::<c_void>()
|
||||
};
|
||||
#[cfg(target_os = "macos")]
|
||||
let view = native.inner();
|
||||
let installed = unsafe { cn_install_page_read(view) };
|
||||
let _ = sender.send(installed != 0);
|
||||
})
|
||||
.map_err(|error| error.to_string())?;
|
||||
if receiver
|
||||
.recv_timeout(std::time::Duration::from_secs(5))
|
||||
.map_err(|_| "Timed out installing native read hooks".to_string())?
|
||||
{
|
||||
Ok(())
|
||||
} else {
|
||||
Err("Native read hooks are unavailable".into())
|
||||
}
|
||||
}
|
||||
193
packages/tauri-app/src-tauri/src/embedded_view/input_linux.c
Normal file
193
packages/tauri-app/src-tauri/src/embedded_view/input_linux.c
Normal file
|
|
@ -0,0 +1,193 @@
|
|||
#include "input.h"
|
||||
#include "page_read.generated.h"
|
||||
#include <gtk/gtk.h>
|
||||
#include <webkit2/webkit2.h>
|
||||
#include <math.h>
|
||||
#include <string.h>
|
||||
|
||||
typedef struct { CNCompletion done; void *context; } Completion;
|
||||
|
||||
static void evaluated(GObject *object, GAsyncResult *result, gpointer data) {
|
||||
Completion *completion = data;
|
||||
GError *error = NULL;
|
||||
JSCValue *value = webkit_web_view_evaluate_javascript_finish(WEBKIT_WEB_VIEW(object), result, &error);
|
||||
char *text = value ? jsc_value_to_string(value) : NULL;
|
||||
completion->done(completion->context, (unsigned char *)text, text ? strlen(text) : 0,
|
||||
error ? error->message : (text ? NULL : "WebKit evaluation returned no value"));
|
||||
g_free(text);
|
||||
g_clear_object(&value);
|
||||
g_clear_error(&error);
|
||||
g_free(completion);
|
||||
}
|
||||
|
||||
void cn_checkpoint(void *view, CNCompletion done, void *context) {
|
||||
Completion *completion = g_new(Completion, 1);
|
||||
*completion = (Completion){done, context};
|
||||
webkit_web_view_evaluate_javascript(WEBKIT_WEB_VIEW(view), CN_CHECKPOINT, -1,
|
||||
CN_WORLD, "codenomad-checkpoint", NULL, evaluated, completion);
|
||||
}
|
||||
|
||||
typedef struct { CNCompletion done; void *context; GCancellable *cancel; guint timeout; } PageRead;
|
||||
static gboolean cancel_page_read(gpointer data) { g_cancellable_cancel(data); return G_SOURCE_REMOVE; }
|
||||
static void page_read_done(GObject *object, GAsyncResult *result, gpointer data) {
|
||||
PageRead *read = data;
|
||||
GError *error = NULL;
|
||||
JSCValue *value = webkit_web_view_call_async_javascript_function_finish(WEBKIT_WEB_VIEW(object), result, &error);
|
||||
char *text = value ? jsc_value_to_string(value) : NULL;
|
||||
const char *failure = error ? error->message : (webkit_web_view_is_loading(WEBKIT_WEB_VIEW(object)) ? "Native browser page is loading" : NULL);
|
||||
read->done(read->context, (unsigned char *)text, text ? strlen(text) : 0, failure);
|
||||
if (read->timeout && g_main_context_find_source_by_id(NULL, read->timeout)) g_source_remove(read->timeout);
|
||||
g_object_unref(read->cancel);
|
||||
g_free(text); g_clear_object(&value); g_clear_error(&error); g_free(read);
|
||||
}
|
||||
int cn_install_page_read(void *view) {
|
||||
/* Pin the named content world between async calls. WebKit otherwise releases
|
||||
* an unused world and its document identity table after each callback. */
|
||||
WebKitUserScript *world = webkit_user_script_new_for_world("void 0", WEBKIT_USER_CONTENT_INJECT_ALL_FRAMES, WEBKIT_USER_SCRIPT_INJECT_AT_DOCUMENT_START, CN_WORLD, NULL, NULL);
|
||||
webkit_user_content_manager_add_script(webkit_web_view_get_user_content_manager(WEBKIT_WEB_VIEW(view)), world);
|
||||
webkit_user_script_unref(world);
|
||||
WebKitUserScript *script = webkit_user_script_new(CN_PAGE_HOOK, WEBKIT_USER_CONTENT_INJECT_ALL_FRAMES, WEBKIT_USER_SCRIPT_INJECT_AT_DOCUMENT_START, NULL, NULL);
|
||||
webkit_user_content_manager_add_script(webkit_web_view_get_user_content_manager(WEBKIT_WEB_VIEW(view)), script);
|
||||
webkit_user_script_unref(script);
|
||||
return 1;
|
||||
}
|
||||
void cn_page_read(void *view, const char *request, unsigned timeout_ms, CNCompletion done, void *context) {
|
||||
if (webkit_web_view_is_loading(WEBKIT_WEB_VIEW(view))) { done(context,NULL,0,"Native browser page is loading"); return; }
|
||||
PageRead *read = g_new0(PageRead,1);
|
||||
read->done=done; read->context=context; read->cancel=g_cancellable_new();
|
||||
read->timeout=g_timeout_add_full(G_PRIORITY_DEFAULT, MAX(1,timeout_ms),cancel_page_read,g_object_ref(read->cancel),g_object_unref);
|
||||
GVariantBuilder args; g_variant_builder_init(&args,G_VARIANT_TYPE_VARDICT);
|
||||
g_variant_builder_add(&args,"{sv}","request",g_variant_new_string(request));
|
||||
GVariant *arguments=g_variant_ref_sink(g_variant_builder_end(&args));
|
||||
webkit_web_view_call_async_javascript_function(WEBKIT_WEB_VIEW(view),CN_PAGE_READ,-1,arguments,CN_WORLD,"codenomad-page-read",read->cancel,page_read_done,read);
|
||||
g_variant_unref(arguments);
|
||||
}
|
||||
|
||||
static GdkModifierType modifiers(unsigned flags) {
|
||||
return ((flags & CN_SHIFT) ? GDK_SHIFT_MASK : 0) |
|
||||
((flags & CN_CONTROL) ? GDK_CONTROL_MASK : 0) |
|
||||
((flags & CN_ALT) ? GDK_MOD1_MASK : 0) |
|
||||
((flags & CN_META) ? GDK_META_MASK : 0);
|
||||
}
|
||||
|
||||
static GdkEvent *event_for(GtkWidget *view, GdkEventType type, int keyboard) {
|
||||
GdkWindow *window = gtk_widget_get_window(view);
|
||||
if (!window || !gtk_widget_get_mapped(view)) return NULL;
|
||||
GdkSeat *seat = gdk_display_get_default_seat(gtk_widget_get_display(view));
|
||||
GdkDevice *device = keyboard ? gdk_seat_get_keyboard(seat) : gdk_seat_get_pointer(seat);
|
||||
if (!device) return NULL;
|
||||
GdkEvent *event = gdk_event_new(type);
|
||||
event->any.window = g_object_ref(window);
|
||||
/* WebKit accepts these as targeted native events on either display backend. */
|
||||
event->any.send_event = FALSE;
|
||||
gdk_event_set_device(event, device);
|
||||
return event;
|
||||
}
|
||||
|
||||
static int valid_point(GtkWidget *view, double x, double y) {
|
||||
return isfinite(x) && isfinite(y) && x >= 0 && y >= 0 &&
|
||||
x < gtk_widget_get_allocated_width(view) && y < gtk_widget_get_allocated_height(view);
|
||||
}
|
||||
|
||||
static int deliver(GdkEvent *event) {
|
||||
if (!event) return 0;
|
||||
gtk_main_do_event(event);
|
||||
gdk_event_free(event);
|
||||
return 1;
|
||||
}
|
||||
|
||||
int cn_pointer(void *opaque, int action, double x, double y, unsigned flags) {
|
||||
double zoom=webkit_web_view_get_zoom_level(WEBKIT_WEB_VIEW(opaque));x*=zoom;y*=zoom;
|
||||
GtkWidget *view = GTK_WIDGET(opaque);
|
||||
if (!valid_point(view, x, y) || action < CN_MOVE || action > CN_DRAG) return 0;
|
||||
GdkEventType type = (action == CN_MOVE || action == CN_DRAG) ? GDK_MOTION_NOTIFY :
|
||||
((action == CN_LEFT_DOWN || action == CN_RIGHT_DOWN) ? GDK_BUTTON_PRESS : GDK_BUTTON_RELEASE);
|
||||
GdkEvent *event = event_for(view, type, 0);
|
||||
if (!event) return 0;
|
||||
if (type == GDK_MOTION_NOTIFY) {
|
||||
event->motion.time = GDK_CURRENT_TIME;
|
||||
event->motion.x = x; event->motion.y = y;
|
||||
event->motion.state = modifiers(flags) | (action == CN_DRAG ? GDK_BUTTON1_MASK : 0);
|
||||
} else {
|
||||
event->button.time = GDK_CURRENT_TIME;
|
||||
event->button.x = x; event->button.y = y;
|
||||
event->button.button = (action == CN_RIGHT_DOWN || action == CN_RIGHT_UP) ? 3 : 1;
|
||||
event->button.state = modifiers(flags) | (type == GDK_BUTTON_RELEASE ?
|
||||
(event->button.button == 1 ? GDK_BUTTON1_MASK : GDK_BUTTON3_MASK) : 0);
|
||||
}
|
||||
return deliver(event);
|
||||
}
|
||||
|
||||
int cn_wheel(void *opaque, double x, double y, double dx, double dy, unsigned flags) {
|
||||
double zoom=webkit_web_view_get_zoom_level(WEBKIT_WEB_VIEW(opaque));x*=zoom;y*=zoom;dx*=zoom;dy*=zoom;
|
||||
GtkWidget *view = GTK_WIDGET(opaque);
|
||||
if (!valid_point(view, x, y) || !isfinite(dx) || !isfinite(dy)) return 0;
|
||||
GdkEvent *event = event_for(view, GDK_SCROLL, 0);
|
||||
if (!event) return 0;
|
||||
event->scroll.time = GDK_CURRENT_TIME;
|
||||
event->scroll.x = x; event->scroll.y = y;
|
||||
event->scroll.state = modifiers(flags);
|
||||
event->scroll.direction = GDK_SCROLL_SMOOTH;
|
||||
/* GDK smooth deltas are wheel steps (WebKit maps one step to 40 CSS px). */
|
||||
event->scroll.delta_x = dx / 40; event->scroll.delta_y = dy / 40;
|
||||
return deliver(event);
|
||||
}
|
||||
|
||||
int cn_key(void *opaque, const char *key, unsigned flags, int down) {
|
||||
GtkWidget *view = GTK_WIDGET(opaque);
|
||||
guint keyval = gdk_keyval_from_name(key);
|
||||
if (keyval == GDK_KEY_VoidSymbol && g_utf8_strlen(key, -1) == 1)
|
||||
keyval = gdk_unicode_to_keyval(g_utf8_get_char(key));
|
||||
if (keyval == GDK_KEY_VoidSymbol) return 0;
|
||||
GdkEvent *event = event_for(view, down ? GDK_KEY_PRESS : GDK_KEY_RELEASE, 1);
|
||||
if (!event) return 0;
|
||||
event->key.time = GDK_CURRENT_TIME;
|
||||
event->key.state = modifiers(flags);
|
||||
event->key.keyval = keyval;
|
||||
GdkKeymapKey *entries = NULL;
|
||||
gint count = 0;
|
||||
if (gdk_keymap_get_entries_for_keyval(gdk_keymap_get_for_display(gtk_widget_get_display(view)),
|
||||
keyval, &entries, &count) && count) {
|
||||
event->key.hardware_keycode = entries[0].keycode;
|
||||
event->key.group = entries[0].group;
|
||||
}
|
||||
g_free(entries);
|
||||
/* gtk_main_do_event routes keys by toplevel focus; dispatch directly to the
|
||||
* exact WebKit widget so another child cannot receive a background key. */
|
||||
gtk_widget_event(view, event);
|
||||
gdk_event_free(event);
|
||||
return 1;
|
||||
}
|
||||
|
||||
int cn_text(void *view, const char *text) {
|
||||
if (!text || !g_utf8_validate(text, -1, NULL) || !gtk_widget_get_mapped(GTK_WIDGET(view))) return 0;
|
||||
webkit_web_view_execute_editing_command_with_argument(WEBKIT_WEB_VIEW(view), "InsertText", text);
|
||||
return 1;
|
||||
}
|
||||
|
||||
static void captured(GObject *object, GAsyncResult *result, gpointer data);
|
||||
static cairo_status_t png_write(void *context, const unsigned char *data, unsigned length) {
|
||||
GByteArray *bytes = context;
|
||||
if (bytes->len + (size_t)length > 16 * 1024 * 1024) return CAIRO_STATUS_WRITE_ERROR;
|
||||
g_byte_array_append(bytes, data, length);
|
||||
return CAIRO_STATUS_SUCCESS;
|
||||
}
|
||||
static void captured(GObject *object, GAsyncResult *result, gpointer data) {
|
||||
Completion *completion = data;
|
||||
GError *error = NULL;
|
||||
cairo_surface_t *surface = webkit_web_view_get_snapshot_finish(WEBKIT_WEB_VIEW(object), result, &error);
|
||||
GByteArray *bytes = g_byte_array_new();
|
||||
cairo_status_t status = surface ? cairo_surface_write_to_png_stream(surface, png_write, bytes) : CAIRO_STATUS_NULL_POINTER;
|
||||
completion->done(completion->context, bytes->data, bytes->len,
|
||||
error ? error->message : (status == CAIRO_STATUS_SUCCESS ? NULL : cairo_status_to_string(status)));
|
||||
g_byte_array_unref(bytes);
|
||||
if (surface) cairo_surface_destroy(surface);
|
||||
g_clear_error(&error);
|
||||
g_free(completion);
|
||||
}
|
||||
void cn_capture(void *view, CNCompletion done, void *context) {
|
||||
Completion *completion = g_new(Completion, 1);
|
||||
*completion = (Completion){done, context};
|
||||
webkit_web_view_get_snapshot(WEBKIT_WEB_VIEW(view), WEBKIT_SNAPSHOT_REGION_VISIBLE,
|
||||
WEBKIT_SNAPSHOT_OPTIONS_NONE, NULL, captured, completion);
|
||||
}
|
||||
346
packages/tauri-app/src-tauri/src/embedded_view/input_macos.m
Normal file
346
packages/tauri-app/src-tauri/src/embedded_view/input_macos.m
Normal file
|
|
@ -0,0 +1,346 @@
|
|||
#import <AppKit/AppKit.h>
|
||||
#import <WebKit/WebKit.h>
|
||||
#include "input.h"
|
||||
#include "page_read.generated.h"
|
||||
#include <math.h>
|
||||
#include <objc/runtime.h>
|
||||
#include <stdatomic.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
|
||||
/* WebKit samples this class method while translating native mouse input.
|
||||
* Only the dispatching UI thread sees synthetic state. Other threads retain
|
||||
* the original behavior, and every exit path restores the implementation. */
|
||||
static _Thread_local unsigned synthetic_depth;
|
||||
static _Thread_local NSUInteger synthetic_buttons;
|
||||
static _Atomic(IMP) original_buttons;
|
||||
static NSUInteger pressed_buttons(id object, SEL selector) {
|
||||
if (synthetic_depth) return synthetic_buttons;
|
||||
return ((NSUInteger (*)(id, SEL))atomic_load(&original_buttons))(object, selector);
|
||||
}
|
||||
static void with_buttons(NSUInteger buttons, void (^dispatch)(void)) {
|
||||
NSCAssert(NSThread.isMainThread, @"Native input requires the UI thread");
|
||||
Method method = class_getClassMethod(NSEvent.class, @selector(pressedMouseButtons));
|
||||
NSUInteger previous = synthetic_buttons;
|
||||
BOOL outer = synthetic_depth == 0;
|
||||
IMP original = NULL;
|
||||
if (outer) {
|
||||
original = method_getImplementation(method);
|
||||
atomic_store(&original_buttons, original);
|
||||
method_setImplementation(method, (IMP)pressed_buttons);
|
||||
}
|
||||
synthetic_depth++;
|
||||
synthetic_buttons = buttons;
|
||||
@try { dispatch(); }
|
||||
@finally {
|
||||
synthetic_buttons = previous;
|
||||
synthetic_depth--;
|
||||
if (outer) method_setImplementation(method, original);
|
||||
}
|
||||
}
|
||||
static void trace(NSString *message) {
|
||||
if (getenv("CN_INPUT_TRACE")) fprintf(stderr, "%s\n", message.UTF8String);
|
||||
}
|
||||
static _Thread_local unsigned modifier_depth;
|
||||
static _Thread_local NSEventModifierFlags synthetic_modifiers;
|
||||
static _Atomic(IMP) original_modifiers;
|
||||
static NSEventModifierFlags modifier_flags(id object, SEL selector) {
|
||||
if (modifier_depth) return synthetic_modifiers;
|
||||
return ((NSEventModifierFlags (*)(id, SEL))atomic_load(&original_modifiers))(object, selector);
|
||||
}
|
||||
static void with_modifiers(NSEventModifierFlags flags, void (^dispatch)(void)) {
|
||||
NSCAssert(NSThread.isMainThread, @"Native input requires the UI thread");
|
||||
Method method = class_getClassMethod(NSEvent.class, @selector(modifierFlags));
|
||||
BOOL outer = modifier_depth == 0;
|
||||
NSEventModifierFlags previous = synthetic_modifiers;
|
||||
IMP original = NULL;
|
||||
if (outer) {
|
||||
original = method_getImplementation(method);
|
||||
atomic_store(&original_modifiers, original);
|
||||
method_setImplementation(method, (IMP)modifier_flags);
|
||||
}
|
||||
modifier_depth++;
|
||||
synthetic_modifiers = flags;
|
||||
@try { dispatch(); }
|
||||
@finally {
|
||||
synthetic_modifiers = previous;
|
||||
modifier_depth--;
|
||||
if (outer) method_setImplementation(method, original);
|
||||
}
|
||||
}
|
||||
#ifdef CN_INPUT_SELF_TEST
|
||||
void cn_test_scoped_buttons(void) {
|
||||
Method method = class_getClassMethod(NSEvent.class, @selector(pressedMouseButtons));
|
||||
IMP original = method_getImplementation(method);
|
||||
NSUInteger physical = NSEvent.pressedMouseButtons;
|
||||
@try {
|
||||
with_buttons(1, ^{
|
||||
NSCAssert(NSEvent.pressedMouseButtons == 1, @"synthetic button missing");
|
||||
with_buttons(2, ^{ NSCAssert(NSEvent.pressedMouseButtons == 2, @"nested button missing"); });
|
||||
NSCAssert(NSEvent.pressedMouseButtons == 1, @"nested state leaked");
|
||||
dispatch_semaphore_t observed = dispatch_semaphore_create(0);
|
||||
dispatch_async(dispatch_get_global_queue(QOS_CLASS_DEFAULT, 0), ^{
|
||||
NSCAssert(NSEvent.pressedMouseButtons == physical, @"synthetic state leaked to another thread");
|
||||
dispatch_semaphore_signal(observed);
|
||||
});
|
||||
NSCAssert(dispatch_semaphore_wait(observed, dispatch_time(DISPATCH_TIME_NOW, 5 * NSEC_PER_SEC)) == 0, @"background check timed out");
|
||||
@throw [NSException exceptionWithName:@"Qualification" reason:@"test cleanup" userInfo:nil];
|
||||
});
|
||||
} @catch (NSException *exception) {
|
||||
NSCAssert([exception.name isEqualToString:@"Qualification"], @"unexpected exception");
|
||||
}
|
||||
NSCAssert(method_getImplementation(method) == original, @"method implementation leaked");
|
||||
NSCAssert(NSEvent.pressedMouseButtons == physical, @"button state leaked");
|
||||
Method modifier_method = class_getClassMethod(NSEvent.class, @selector(modifierFlags));
|
||||
IMP modifier_original = method_getImplementation(modifier_method);
|
||||
NSEventModifierFlags physical_modifiers = NSEvent.modifierFlags;
|
||||
@try {
|
||||
with_modifiers(NSEventModifierFlagCommand, ^{
|
||||
NSCAssert(NSEvent.modifierFlags == NSEventModifierFlagCommand, @"synthetic modifier missing");
|
||||
with_modifiers(0, ^{ NSCAssert(NSEvent.modifierFlags == 0, @"zero modifier missing"); });
|
||||
NSCAssert(NSEvent.modifierFlags == NSEventModifierFlagCommand, @"nested modifier leaked");
|
||||
@throw [NSException exceptionWithName:@"Qualification" reason:@"modifier cleanup" userInfo:nil];
|
||||
});
|
||||
} @catch (NSException *exception) {
|
||||
NSCAssert([exception.name isEqualToString:@"Qualification"], @"unexpected exception");
|
||||
}
|
||||
NSCAssert(method_getImplementation(modifier_method) == modifier_original, @"modifier implementation leaked");
|
||||
NSCAssert(NSEvent.modifierFlags == physical_modifiers, @"modifier state leaked");
|
||||
}
|
||||
#endif
|
||||
|
||||
/* Public mouseMoved: is inherited from NSView on the qualified WebKit build.
|
||||
* This SPI is explicitly checked at runtime; its DOM behavior stays gated by
|
||||
* the strict hover assertion before application integration. */
|
||||
@interface WKWebView (CNTargetedMouseMove)
|
||||
- (void)_simulateMouseMove:(NSEvent *)event;
|
||||
@end
|
||||
#ifdef CN_INPUT_SELF_TEST
|
||||
int cn_test_direct_hover(void *opaque, double x, double y) {
|
||||
WKWebView *view = (__bridge WKWebView *)opaque;
|
||||
NSPoint local = NSMakePoint(x, view.isFlipped ? y : view.bounds.size.height - y);
|
||||
NSPoint point = [view convertPoint:local toView:nil];
|
||||
NSEvent *entered = [NSEvent enterExitEventWithType:NSEventTypeMouseEntered location:point
|
||||
modifierFlags:0 timestamp:NSProcessInfo.processInfo.systemUptime windowNumber:view.window.windowNumber
|
||||
context:nil eventNumber:1 trackingNumber:0 userData:NULL];
|
||||
NSEvent *moved = [NSEvent mouseEventWithType:NSEventTypeMouseMoved location:point
|
||||
modifierFlags:0 timestamp:NSProcessInfo.processInfo.systemUptime windowNumber:view.window.windowNumber
|
||||
context:nil eventNumber:2 clickCount:0 pressure:0];
|
||||
trace([NSString stringWithFormat:@"hover direct mouseEntered+mouseMoved css=%g,%g window=%g,%g flipped=%d keyWindow=%d", x, y, point.x, point.y, view.isFlipped, view.window.isKeyWindow]);
|
||||
[view mouseEntered:entered];
|
||||
[view mouseMoved:moved];
|
||||
return 1;
|
||||
}
|
||||
#endif
|
||||
|
||||
static void fail(CNCompletion done, void *context, NSString *error) {
|
||||
done(context, NULL, 0, error.UTF8String);
|
||||
}
|
||||
static WKContentWorld *read_world(void) {
|
||||
// WKContentWorld's named cache is weak. Keep the world alive between reads
|
||||
// so its per-document identity table survives separate native callbacks.
|
||||
static WKContentWorld *world;
|
||||
static dispatch_once_t once;
|
||||
dispatch_once(&once, ^{ world = [WKContentWorld worldWithName:@CN_WORLD]; });
|
||||
return world;
|
||||
}
|
||||
int cn_install_page_read(void *opaque) {
|
||||
WKWebView *view = (__bridge WKWebView *)opaque;
|
||||
WKUserScript *hook = [[WKUserScript alloc] initWithSource:[NSString stringWithUTF8String:CN_PAGE_HOOK] injectionTime:WKUserScriptInjectionTimeAtDocumentStart forMainFrameOnly:NO inContentWorld:WKContentWorld.pageWorld];
|
||||
[view.configuration.userContentController addUserScript:hook];
|
||||
return 1;
|
||||
}
|
||||
void cn_page_read(void *opaque, const char *request, unsigned timeout_ms, CNCompletion done, void *context) {
|
||||
(void)timeout_ms; // The Rust request fence rejects late WKWebView callbacks.
|
||||
WKWebView *view = (__bridge WKWebView *)opaque;
|
||||
if (view.loading) { fail(done,context,@"Native browser page is loading"); return; }
|
||||
NSDictionary *arguments = @{ @"request": [NSString stringWithUTF8String:request] };
|
||||
[view callAsyncJavaScript:[NSString stringWithUTF8String:CN_PAGE_READ] arguments:arguments inFrame:nil inContentWorld:read_world()
|
||||
completionHandler:^(id result, NSError *error) {
|
||||
if (error || view.loading || ![result isKindOfClass:NSString.class]) {
|
||||
fail(done,context,error.localizedDescription ?: @"Native browser page changed or is loading"); return;
|
||||
}
|
||||
NSData *bytes = [result dataUsingEncoding:NSUTF8StringEncoding];
|
||||
done(context,bytes.bytes,bytes.length,NULL);
|
||||
}];
|
||||
}
|
||||
void cn_checkpoint(void *opaque, CNCompletion done, void *context) {
|
||||
WKWebView *view = (__bridge WKWebView *)opaque;
|
||||
[view evaluateJavaScript:@CN_CHECKPOINT inFrame:nil
|
||||
inContentWorld:read_world()
|
||||
completionHandler:^(id result, NSError *error) {
|
||||
if (error || ![result isKindOfClass:NSString.class]) {
|
||||
fail(done, context, error.localizedDescription ?: @"WebKit evaluation returned no value");
|
||||
return;
|
||||
}
|
||||
NSData *bytes = [result dataUsingEncoding:NSUTF8StringEncoding];
|
||||
done(context, bytes.bytes, bytes.length, NULL);
|
||||
}];
|
||||
}
|
||||
static NSEventModifierFlags modifiers(unsigned flags) {
|
||||
return ((flags & CN_SHIFT) ? NSEventModifierFlagShift : 0) |
|
||||
((flags & CN_CONTROL) ? NSEventModifierFlagControl : 0) |
|
||||
((flags & CN_ALT) ? NSEventModifierFlagOption : 0) |
|
||||
((flags & CN_META) ? NSEventModifierFlagCommand : 0);
|
||||
}
|
||||
static BOOL target(WKWebView *view, double x, double y, NSPoint *point) {
|
||||
x*=view.pageZoom;y*=view.pageZoom;
|
||||
if (!view.window || view.hidden || !isfinite(x) || !isfinite(y) ||
|
||||
x < 0 || y < 0 || x >= view.bounds.size.width || y >= view.bounds.size.height) return NO;
|
||||
NSPoint local = NSMakePoint(x, view.isFlipped ? y : view.bounds.size.height - y);
|
||||
*point = [view convertPoint:local toView:nil];
|
||||
return YES;
|
||||
}
|
||||
int cn_pointer(void *opaque, int action, double x, double y, unsigned flags) {
|
||||
WKWebView *view = (__bridge WKWebView *)opaque;
|
||||
NSPoint point;
|
||||
if (!target(view, x, y, &point) || action < CN_MOVE || action > CN_DRAG) return 0;
|
||||
IMP inherited = class_getMethodImplementation(NSView.class, @selector(mouseMoved:));
|
||||
BOOL public_move = class_getMethodImplementation(view.class, @selector(mouseMoved:)) != inherited;
|
||||
if (action == CN_MOVE && !public_move && ![view respondsToSelector:@selector(_simulateMouseMove:)]) return 0;
|
||||
NSEventType types[] = {NSEventTypeMouseMoved, NSEventTypeLeftMouseDown,
|
||||
NSEventTypeLeftMouseUp, NSEventTypeRightMouseDown, NSEventTypeRightMouseUp,
|
||||
NSEventTypeLeftMouseDragged};
|
||||
static char sequence_key;
|
||||
NSInteger number = [objc_getAssociatedObject(view, &sequence_key) integerValue] + 1;
|
||||
objc_setAssociatedObject(view, &sequence_key, @(number), OBJC_ASSOCIATION_RETAIN_NONATOMIC);
|
||||
NSEvent *event = [NSEvent mouseEventWithType:types[action] location:point
|
||||
modifierFlags:modifiers(flags) timestamp:NSProcessInfo.processInfo.systemUptime
|
||||
windowNumber:view.window.windowNumber context:nil eventNumber:number
|
||||
clickCount:action == CN_MOVE ? 0 : 1
|
||||
pressure:(action == CN_LEFT_DOWN || action == CN_DRAG) ? 1 : 0];
|
||||
NSUInteger buttons = (action == CN_LEFT_DOWN || action == CN_DRAG) ? 1 : (action == CN_RIGHT_DOWN ? 2 : 0);
|
||||
with_buttons(buttons, ^{
|
||||
switch (action) {
|
||||
case CN_MOVE:
|
||||
trace([NSString stringWithFormat:@"hover css=%g,%g window=%g,%g flipped=%d", x, y, point.x, point.y, view.isFlipped]);
|
||||
if (public_move) { trace(@"hover public-mouseMoved"); [view mouseMoved:event]; }
|
||||
else { trace(@"hover SPI=_simulateMouseMove:"); [view _simulateMouseMove:event]; }
|
||||
break;
|
||||
case CN_LEFT_DOWN: [view mouseDown:event]; break;
|
||||
case CN_LEFT_UP: [view mouseUp:event]; break;
|
||||
case CN_RIGHT_DOWN: [view rightMouseDown:event]; break;
|
||||
case CN_RIGHT_UP: [view rightMouseUp:event]; break;
|
||||
case CN_DRAG: [view mouseDragged:event]; break;
|
||||
}
|
||||
});
|
||||
return 1;
|
||||
}
|
||||
int cn_wheel(void *opaque, double x, double y, double dx, double dy, unsigned flags) {
|
||||
WKWebView *view = (__bridge WKWebView *)opaque;
|
||||
NSPoint point;
|
||||
if (!target(view, x, y, &point) || !isfinite(dx) || !isfinite(dy)) return 0;
|
||||
/* Construct a native event value; never post it to the system queue. */
|
||||
CGEventRef cg = CGEventCreateScrollWheelEvent(NULL, kCGScrollEventUnitPixel, 2, (int32_t)(-dy*view.pageZoom), (int32_t)(-dx*view.pageZoom));
|
||||
if (!cg) return 0;
|
||||
CGEventSetFlags(cg, (CGEventFlags)modifiers(flags));
|
||||
CGEventSetLocation(cg, point);
|
||||
[view scrollWheel:[NSEvent eventWithCGEvent:cg]];
|
||||
CFRelease(cg);
|
||||
return 1;
|
||||
}
|
||||
static NSString *characters(const char *key, unsigned short *code) {
|
||||
NSString *text = [NSString stringWithUTF8String:key];
|
||||
NSDictionary *special = @{
|
||||
@"Return": @[@13, @36], @"Tab": @[@9, @48], @"Escape": @[@27, @53],
|
||||
@"BackSpace": @[@127, @51], @"Delete": @[@(NSDeleteFunctionKey), @117],
|
||||
@"Left": @[@(NSLeftArrowFunctionKey), @123], @"Right": @[@(NSRightArrowFunctionKey), @124],
|
||||
@"Down": @[@(NSDownArrowFunctionKey), @125], @"Up": @[@(NSUpArrowFunctionKey), @126],
|
||||
@"Home": @[@(NSHomeFunctionKey), @115], @"End": @[@(NSEndFunctionKey), @119],
|
||||
@"Page_Up": @[@(NSPageUpFunctionKey), @116], @"Page_Down": @[@(NSPageDownFunctionKey), @121],
|
||||
@"Insert": @[@(NSInsertFunctionKey), @114], @"Menu": @[@(NSMenuFunctionKey), @110]
|
||||
};
|
||||
NSArray *entry = special[text];
|
||||
if ([text hasPrefix:@"F"] && text.length<=3) {
|
||||
int number=[[text substringFromIndex:1] intValue];
|
||||
const unsigned short codes[]={122,120,99,118,96,97,98,100,101,109,103,111};
|
||||
if (number>=1 && number<=12) entry=@[@(NSF1FunctionKey+number-1),@(codes[number-1])];
|
||||
}
|
||||
if (entry) {
|
||||
unichar character = [entry[0] unsignedShortValue];
|
||||
*code = [entry[1] unsignedShortValue];
|
||||
return [NSString stringWithCharacters:&character length:1];
|
||||
}
|
||||
// ANSI keycodes serve native shortcuts; Unicode insertion uses the responder API.
|
||||
NSArray *keys = @[@"a", @"s", @"d", @"f", @"h", @"g", @"z", @"x", @"c", @"v", @"", @"b", @"q", @"w", @"e", @"r", @"y", @"t", @"1", @"2", @"3", @"4", @"6", @"5", @"=", @"9", @"7", @"-", @"8", @"0", @"]", @"o", @"u", @"[", @"i", @"p", @"", @"l", @"j", @"'", @"k", @";", @"\\", @",", @"/", @"n", @"m", @".", @"", @" ", @"`"];
|
||||
NSDictionary *shifted=@{@"~":@"`",@"!":@"1",@"@":@"2",@"#":@"3",@"$":@"4",@"%":@"5",@"^":@"6",@"&":@"7",@"*":@"8",@"(":@"9",@")":@"0",@"_":@"-",@"+":@"=",@"{":@"[",@"}":@"]",@"|":@"\\",@":":@";",@"\"":@"'",@"<":@",",@">":@".",@"?":@"/"};
|
||||
NSUInteger index = [keys indexOfObject:shifted[text] ?: text.lowercaseString];
|
||||
if (index == NSNotFound || !text.length) return nil;
|
||||
*code = (unsigned short)index;
|
||||
return text;
|
||||
}
|
||||
static void change_modifiers(WKWebView *view, unsigned before, unsigned after) {
|
||||
const unsigned bits[] = {CN_SHIFT, CN_CONTROL, CN_ALT, CN_META};
|
||||
const unsigned short codes[] = {56, 59, 58, 55};
|
||||
unsigned current = before;
|
||||
for (int i = 0; i < 4; i++) {
|
||||
if (!((before ^ after) & bits[i])) continue;
|
||||
current = (current & ~bits[i]) | (after & bits[i]);
|
||||
NSEvent *changed = [NSEvent keyEventWithType:NSEventTypeFlagsChanged location:NSZeroPoint
|
||||
modifierFlags:modifiers(current) timestamp:NSProcessInfo.processInfo.systemUptime
|
||||
windowNumber:view.window.windowNumber context:nil characters:@"" charactersIgnoringModifiers:@""
|
||||
isARepeat:NO keyCode:codes[i]];
|
||||
trace([NSString stringWithFormat:@"flagsChanged receiver=%@ code=%hu flags=0x%lx timestamp=%.6f window=%ld", view.class, codes[i], (unsigned long)changed.modifierFlags, changed.timestamp, (long)changed.windowNumber]);
|
||||
with_modifiers(changed.modifierFlags, ^{ [view flagsChanged:changed]; });
|
||||
}
|
||||
}
|
||||
int cn_key(void *opaque, const char *key, unsigned flags, int down) {
|
||||
WKWebView *view = (__bridge WKWebView *)opaque;
|
||||
if (!view.window || view.hidden) return 0;
|
||||
static char held_modifiers_key;
|
||||
unsigned held = [objc_getAssociatedObject(view, &held_modifiers_key) unsignedIntValue];
|
||||
NSString *name=[NSString stringWithUTF8String:key];
|
||||
if ([@[@"Shift_L",@"Control_L",@"Alt_L",@"Meta_L"] containsObject:name]) {
|
||||
change_modifiers(view,held,flags);
|
||||
objc_setAssociatedObject(view,&held_modifiers_key,@(flags),OBJC_ASSOCIATION_RETAIN_NONATOMIC);
|
||||
return 1;
|
||||
}
|
||||
unsigned short code = 0;
|
||||
NSString *text = characters(key, &code);
|
||||
if (!text) return 0;
|
||||
change_modifiers(view, held, flags);
|
||||
objc_setAssociatedObject(view, &held_modifiers_key, @(flags), OBJC_ASSOCIATION_RETAIN_NONATOMIC);
|
||||
NSEvent *event = [NSEvent keyEventWithType:down ? NSEventTypeKeyDown : NSEventTypeKeyUp
|
||||
location:NSZeroPoint modifierFlags:modifiers(flags)
|
||||
timestamp:NSProcessInfo.processInfo.systemUptime windowNumber:view.window.windowNumber
|
||||
context:nil characters:text charactersIgnoringModifiers:text isARepeat:NO keyCode:code];
|
||||
[view.window makeFirstResponder:view];
|
||||
trace([NSString stringWithFormat:@"%@ key=%@ flags=0x%lx timestamp=%.6f window=%ld held=0x%x", down ? @"keyDown" : @"keyUp", [NSString stringWithUTF8String:key], (unsigned long)event.modifierFlags, event.timestamp, (long)event.windowNumber, flags]);
|
||||
with_modifiers(event.modifierFlags, ^{
|
||||
NSDictionary *commands = @{@"a": @"selectAll:", @"c": @"copy:", @"v": @"paste:", @"x": @"cut:", @"z": @"undo:"};
|
||||
NSString *selector = flags == CN_META ? commands[text.lowercaseString]
|
||||
: (flags == (CN_META | CN_SHIFT) && [text.lowercaseString isEqualToString:@"z"]) ? @"redo:" : nil;
|
||||
SEL command = selector ? NSSelectorFromString(selector) : NULL;
|
||||
// Execute editing equivalents once on this view. Acknowledged native input
|
||||
// is never followed by a second command as an inactive-window fallback.
|
||||
if (down && command && [view respondsToSelector:command]) {
|
||||
[view tryToPerform:command with:nil];
|
||||
} else if (down) {
|
||||
trace([NSString stringWithFormat:@"key %@ keyDown", [NSString stringWithUTF8String:key]]);
|
||||
[view keyDown:event];
|
||||
} else { trace([NSString stringWithFormat:@"key %@ keyUp", [NSString stringWithUTF8String:key]]); [view keyUp:event]; }
|
||||
});
|
||||
return 1;
|
||||
}
|
||||
int cn_text(void *opaque, const char *text) {
|
||||
WKWebView *view = (__bridge WKWebView *)opaque;
|
||||
if (!view.window || view.hidden) return 0;
|
||||
[view.window makeFirstResponder:view];
|
||||
NSResponder *receiver = view.window.firstResponder;
|
||||
if (!view.window || view.hidden || ![receiver isKindOfClass:NSView.class] ||
|
||||
![(NSView *)receiver isDescendantOf:view] || ![receiver respondsToSelector:@selector(insertText:replacementRange:)]) return 0;
|
||||
[(id<NSTextInputClient>)receiver insertText:[NSString stringWithUTF8String:text] replacementRange:NSMakeRange(NSNotFound, 0)];
|
||||
return 1;
|
||||
}
|
||||
void cn_capture(void *opaque, CNCompletion done, void *context) {
|
||||
WKWebView *view = (__bridge WKWebView *)opaque;
|
||||
WKSnapshotConfiguration *config = [WKSnapshotConfiguration new];
|
||||
config.rect = view.bounds;
|
||||
[view takeSnapshotWithConfiguration:config completionHandler:^(NSImage *image, NSError *error) {
|
||||
if (error || !image) { fail(done, context, error.localizedDescription ?: @"WebKit snapshot failed"); return; }
|
||||
NSBitmapImageRep *bitmap = [NSBitmapImageRep imageRepWithData:image.TIFFRepresentation];
|
||||
NSData *png = [bitmap representationUsingType:NSBitmapImageFileTypePNG properties:@{}];
|
||||
if (!png || png.length > 16 * 1024 * 1024) { fail(done, context, @"WebKit snapshot exceeds PNG budget"); return; }
|
||||
done(context, png.bytes, png.length, NULL);
|
||||
}];
|
||||
}
|
||||
161
packages/tauri-app/src-tauri/src/embedded_view/input_native.rs
Normal file
161
packages/tauri-app/src-tauri/src/embedded_view/input_native.rs
Normal file
|
|
@ -0,0 +1,161 @@
|
|||
//! Semantic input mapped to the already-qualified, target-local native adapters.
|
||||
use serde_json::{json, Value};
|
||||
use std::{
|
||||
ffi::{c_char, c_void, CString},
|
||||
sync::{Arc, Mutex},
|
||||
};
|
||||
extern "C" {
|
||||
fn cn_pointer(view: *mut c_void, action: i32, x: f64, y: f64, modifiers: u32) -> i32;
|
||||
fn cn_wheel(view: *mut c_void, x: f64, y: f64, dx: f64, dy: f64, modifiers: u32) -> i32;
|
||||
fn cn_key(view: *mut c_void, key: *const c_char, modifiers: u32, down: i32) -> i32;
|
||||
fn cn_text(view: *mut c_void, text: *const c_char) -> i32;
|
||||
}
|
||||
pub(crate) type Held = Arc<Mutex<Vec<(String, Value)>>>;
|
||||
fn flags(event: &Value) -> u32 {
|
||||
let flags = event["modifiers"].as_u64().unwrap_or(0) as u32;
|
||||
((flags & 8) >> 3) | (flags & 2) | ((flags & 1) << 2) | ((flags & 4) << 1)
|
||||
}
|
||||
fn key(value: &str) -> &str {
|
||||
match value {
|
||||
"Enter" => "Return",
|
||||
"Backspace" => "BackSpace",
|
||||
"ArrowUp" => "Up",
|
||||
"ArrowDown" => "Down",
|
||||
"ArrowLeft" => "Left",
|
||||
"ArrowRight" => "Right",
|
||||
"PageUp" => "Page_Up",
|
||||
"PageDown" => "Page_Down",
|
||||
"Shift" => "Shift_L",
|
||||
"Control" => "Control_L",
|
||||
"Alt" => "Alt_L",
|
||||
"Meta" => "Meta_L",
|
||||
"ContextMenu" => "Menu",
|
||||
other => other,
|
||||
}
|
||||
}
|
||||
pub(crate) unsafe fn dispatch(
|
||||
view: *mut c_void,
|
||||
operation: &Value,
|
||||
held: &Held,
|
||||
) -> Result<(), String> {
|
||||
let event = &operation["event"];
|
||||
let kind = operation["op"].as_str().unwrap_or("");
|
||||
if kind == "pointer" && event["button"] == "middle" {
|
||||
return Err("Native middle-button input is unsupported".into());
|
||||
}
|
||||
let down = matches!(
|
||||
event["type"].as_str(),
|
||||
Some("mousePressed" | "keyDown" | "rawKeyDown")
|
||||
);
|
||||
let up = matches!(event["type"].as_str(), Some("mouseReleased" | "keyUp"));
|
||||
let identity = format!(
|
||||
"{kind}:{}",
|
||||
if kind == "pointer" {
|
||||
event["button"].as_str().unwrap_or("")
|
||||
} else {
|
||||
event["code"].as_str().unwrap_or("")
|
||||
}
|
||||
);
|
||||
{
|
||||
let mut held = held.lock().map_err(|error| error.to_string())?;
|
||||
if kind == "pointer" {
|
||||
for (_, release) in held.iter_mut().filter(|(_, item)| item["op"] == "pointer") {
|
||||
release["event"]["x"] = event["x"].clone();
|
||||
release["event"]["y"] = event["y"].clone();
|
||||
}
|
||||
}
|
||||
if down {
|
||||
if held.len() >= 64 && !held.iter().any(|(id, _)| id == &identity) {
|
||||
return Err("Too many held native inputs".into());
|
||||
}
|
||||
let mut release = operation.clone();
|
||||
release["event"]["type"] = json!(if kind == "pointer" {
|
||||
"mouseReleased"
|
||||
} else {
|
||||
"keyUp"
|
||||
});
|
||||
release["event"]["buttons"] = json!(0);
|
||||
release["event"]["modifiers"] = json!(0);
|
||||
if let Some(event) = release["event"].as_object_mut() {
|
||||
event.remove("text");
|
||||
event.remove("unmodifiedText");
|
||||
}
|
||||
held.retain(|(id, _)| id != &identity);
|
||||
held.push((identity.clone(), release));
|
||||
}
|
||||
}
|
||||
let number = |key: &str| event[key].as_f64().unwrap_or(0.0);
|
||||
let accepted = match kind {
|
||||
"pointer" => {
|
||||
let right = event["button"] == "right";
|
||||
let action = match event["type"].as_str() {
|
||||
Some("mousePressed") => {
|
||||
if right {
|
||||
3
|
||||
} else {
|
||||
1
|
||||
}
|
||||
}
|
||||
Some("mouseReleased") => {
|
||||
if right {
|
||||
4
|
||||
} else {
|
||||
2
|
||||
}
|
||||
}
|
||||
_ => {
|
||||
if event["buttons"].as_u64().unwrap_or(0) & 1 != 0 {
|
||||
5
|
||||
} else {
|
||||
0
|
||||
}
|
||||
}
|
||||
};
|
||||
unsafe { cn_pointer(view, action, number("x"), number("y"), flags(event)) }
|
||||
}
|
||||
"wheel" => unsafe {
|
||||
cn_wheel(
|
||||
view,
|
||||
number("x"),
|
||||
number("y"),
|
||||
number("deltaX"),
|
||||
number("deltaY"),
|
||||
flags(event),
|
||||
)
|
||||
},
|
||||
"keyboard" => {
|
||||
let value = CString::new(key(event["key"].as_str().unwrap_or("")))
|
||||
.map_err(|_| "Invalid native key")?;
|
||||
unsafe { cn_key(view, value.as_ptr(), flags(event), i32::from(down)) }
|
||||
}
|
||||
"insert-text" => {
|
||||
let value = CString::new(operation["text"].as_str().unwrap_or(""))
|
||||
.map_err(|_| "Native text contains a NUL character")?;
|
||||
unsafe { cn_text(view, value.as_ptr()) }
|
||||
}
|
||||
_ => return Err("Unsupported native input operation".into()),
|
||||
};
|
||||
if accepted == 0 {
|
||||
return Err("Native browser input was refused".into());
|
||||
}
|
||||
if up {
|
||||
held.lock()
|
||||
.map_err(|error| error.to_string())?
|
||||
.retain(|(id, _)| id != &identity);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
#[test]
|
||||
fn semantic_modifiers_and_keys_match_native_adapters() {
|
||||
for (cdp, native) in [(0, 0), (1, 4), (2, 2), (4, 8), (8, 1), (15, 15)] {
|
||||
assert_eq!(flags(&json!({"modifiers":cdp})), native);
|
||||
}
|
||||
assert_eq!(key("Enter"), "Return");
|
||||
assert_eq!(key("Meta"), "Meta_L");
|
||||
assert_eq!(key("ArrowDown"), "Down");
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,56 @@
|
|||
#import <WebKit/WebKit.h>
|
||||
#import <objc/runtime.h>
|
||||
#include <stdint.h>
|
||||
|
||||
typedef void (*CNLoad)(void *, uint64_t, int, const char *, const char *);
|
||||
typedef void (*CNRelease)(void *);
|
||||
@interface CNLoadObserver : NSObject <WKNavigationDelegate>
|
||||
@property (strong) id<WKNavigationDelegate> original;
|
||||
@property CNLoad callback;
|
||||
@property CNRelease releaseContext;
|
||||
@property void *context;
|
||||
@end
|
||||
@implementation CNLoadObserver
|
||||
- (BOOL)respondsToSelector:(SEL)selector {
|
||||
return [super respondsToSelector:selector] || [self.original respondsToSelector:selector];
|
||||
}
|
||||
- (id)forwardingTargetForSelector:(SEL)selector { return self.original; }
|
||||
- (void)webView:(WKWebView *)view didStartProvisionalNavigation:(WKNavigation *)navigation {
|
||||
if ([self.original respondsToSelector:_cmd]) [self.original webView:view didStartProvisionalNavigation:navigation];
|
||||
self.callback(self.context, (uint64_t)(uintptr_t)(__bridge void *)navigation, 0, view.URL.absoluteString.UTF8String, NULL);
|
||||
}
|
||||
- (void)webView:(WKWebView *)view didFinishNavigation:(WKNavigation *)navigation {
|
||||
if ([self.original respondsToSelector:_cmd]) [self.original webView:view didFinishNavigation:navigation];
|
||||
self.callback(self.context, (uint64_t)(uintptr_t)(__bridge void *)navigation, 1, view.URL.absoluteString.UTF8String, NULL);
|
||||
}
|
||||
- (void)webView:(WKWebView *)view didFailProvisionalNavigation:(WKNavigation *)navigation withError:(NSError *)error {
|
||||
if ([self.original respondsToSelector:_cmd]) [self.original webView:view didFailProvisionalNavigation:navigation withError:error];
|
||||
self.callback(self.context, (uint64_t)(uintptr_t)(__bridge void *)navigation, 2, view.URL.absoluteString.UTF8String, error.localizedDescription.UTF8String);
|
||||
}
|
||||
- (void)webView:(WKWebView *)view didFailNavigation:(WKNavigation *)navigation withError:(NSError *)error {
|
||||
if ([self.original respondsToSelector:_cmd]) [self.original webView:view didFailNavigation:navigation withError:error];
|
||||
self.callback(self.context, (uint64_t)(uintptr_t)(__bridge void *)navigation, 2, view.URL.absoluteString.UTF8String, error.localizedDescription.UTF8String);
|
||||
}
|
||||
- (void)dealloc { if (self.releaseContext) self.releaseContext(self.context); }
|
||||
@end
|
||||
|
||||
int cn_observe_load(void *opaque, CNLoad callback, CNRelease releaseContext, void *context) {
|
||||
WKWebView *view = (__bridge WKWebView *)opaque;
|
||||
if (!view.navigationDelegate) return 0;
|
||||
CNLoadObserver *observer = [CNLoadObserver new];
|
||||
observer.original = view.navigationDelegate;
|
||||
observer.callback = callback;
|
||||
observer.context = context;
|
||||
observer.releaseContext = releaseContext;
|
||||
static char key;
|
||||
objc_setAssociatedObject(view, &key, observer, OBJC_ASSOCIATION_RETAIN_NONATOMIC);
|
||||
view.navigationDelegate = observer;
|
||||
return 1;
|
||||
}
|
||||
|
||||
uint64_t cn_navigate(void *opaque, const char *url) {
|
||||
WKWebView *view = (__bridge WKWebView *)opaque;
|
||||
NSURL *target = [NSURL URLWithString:[NSString stringWithUTF8String:url]];
|
||||
WKNavigation *navigation = [view loadRequest:[NSURLRequest requestWithURL:target]];
|
||||
return (uint64_t)(uintptr_t)(__bridge void *)navigation;
|
||||
}
|
||||
50
packages/tauri-app/src-tauri/src/embedded_view/mobile.rs
Normal file
50
packages/tauri-app/src-tauri/src/embedded_view/mobile.rs
Normal file
|
|
@ -0,0 +1,50 @@
|
|||
use std::{
|
||||
ffi::{c_char, c_void, CString},
|
||||
sync::mpsc,
|
||||
time::Duration,
|
||||
};
|
||||
extern "C" {
|
||||
fn cn_preview_scale(view: *mut c_void, device: *mut f64) -> f64;
|
||||
fn cn_preview_profile(view: *mut c_void, ua: *const c_char, zoom: f64, density: f64) -> i32;
|
||||
}
|
||||
pub(crate) fn scale<R: tauri::Runtime>(view: &tauri::Webview<R>) -> Result<(f64, f64), String> {
|
||||
native(view, |ptr| {
|
||||
let mut device = 1.0;
|
||||
let scale = unsafe { cn_preview_scale(ptr, &mut device) };
|
||||
if !scale.is_finite() || scale <= 0.0 || !device.is_finite() || device <= 0.0 {
|
||||
return Err("Native preview scale is unavailable".into());
|
||||
}
|
||||
Ok((scale, device))
|
||||
})
|
||||
}
|
||||
pub(crate) fn profile<R: tauri::Runtime>(
|
||||
view: &tauri::Webview<R>,
|
||||
ua: &str,
|
||||
zoom: f64,
|
||||
density: f64,
|
||||
) -> Result<bool, String> {
|
||||
let ua = CString::new(ua).map_err(|_| "Invalid preview identity")?;
|
||||
native(view, move |ptr| {
|
||||
Ok(unsafe { cn_preview_profile(ptr, ua.as_ptr(), zoom, density) } != 0)
|
||||
})
|
||||
}
|
||||
fn native<R: tauri::Runtime, T: Send + 'static>(
|
||||
view: &tauri::Webview<R>,
|
||||
run: impl FnOnce(*mut c_void) -> Result<T, String> + Send + 'static,
|
||||
) -> Result<T, String> {
|
||||
let (sender, receiver) = mpsc::sync_channel(1);
|
||||
view.with_webview(move |native| {
|
||||
#[cfg(target_os = "macos")]
|
||||
let ptr = native.inner();
|
||||
#[cfg(target_os = "linux")]
|
||||
let ptr = {
|
||||
use gtk::glib::object::ObjectType;
|
||||
native.inner().as_ptr().cast::<c_void>()
|
||||
};
|
||||
let _ = sender.send(run(ptr));
|
||||
})
|
||||
.map_err(|e| e.to_string())?;
|
||||
receiver
|
||||
.recv_timeout(Duration::from_secs(5))
|
||||
.map_err(|_| "Native preview profile timed out".to_string())?
|
||||
}
|
||||
|
|
@ -0,0 +1,71 @@
|
|||
#include <gtk/gtk.h>
|
||||
#include <webkit2/webkit2.h>
|
||||
typedef struct { gboolean mobile, active, sending; double x, y; } TouchState;
|
||||
static void touch(GtkWidget *view, TouchState *state, GdkEventType type, double x, double y) {
|
||||
GdkWindow *window = gtk_widget_get_window(view);
|
||||
GdkSeat *seat = gdk_display_get_default_seat(gtk_widget_get_display(view));
|
||||
if (!window || !seat) return;
|
||||
GdkEvent *event = gdk_event_new(type);
|
||||
event->touch.window = g_object_ref(window);
|
||||
event->touch.send_event = FALSE;
|
||||
event->touch.time = GDK_CURRENT_TIME;
|
||||
event->touch.x = x; event->touch.y = y;
|
||||
event->touch.sequence = (GdkEventSequence *)state;
|
||||
event->touch.emulating_pointer = TRUE;
|
||||
int ox = 0, oy = 0;
|
||||
gdk_window_get_origin(window, &ox, &oy);
|
||||
event->touch.x_root = ox + x; event->touch.y_root = oy + y;
|
||||
gdk_event_set_device(event, gdk_seat_get_pointer(seat));
|
||||
state->sending = TRUE;
|
||||
gtk_widget_event(view, event);
|
||||
state->sending = FALSE;
|
||||
gdk_event_free(event);
|
||||
state->x = x; state->y = y;
|
||||
}
|
||||
static gboolean pressed(GtkWidget *view, GdkEventButton *event, gpointer data) {
|
||||
TouchState *state = data;
|
||||
if (state->sending || gdk_event_get_pointer_emulated((GdkEvent *)event)) return FALSE;
|
||||
if (!state->mobile || event->button != 1) return FALSE;
|
||||
// GTK also emits multi-click notifications after the ordinary press.
|
||||
if (event->type != GDK_BUTTON_PRESS) return TRUE;
|
||||
state->active = TRUE;
|
||||
touch(view, state, GDK_TOUCH_BEGIN, event->x, event->y);
|
||||
return TRUE;
|
||||
}
|
||||
static gboolean released(GtkWidget *view, GdkEventButton *event, gpointer data) {
|
||||
TouchState *state = data;
|
||||
if (state->sending || gdk_event_get_pointer_emulated((GdkEvent *)event)) return FALSE;
|
||||
if (!state->mobile || event->button != 1 || !state->active) return FALSE;
|
||||
state->active = FALSE;
|
||||
touch(view, state, GDK_TOUCH_END, event->x, event->y);
|
||||
return TRUE;
|
||||
}
|
||||
static gboolean moved(GtkWidget *view, GdkEventMotion *event, gpointer data) {
|
||||
TouchState *state = data;
|
||||
if (state->sending || gdk_event_get_pointer_emulated((GdkEvent *)event)) return FALSE;
|
||||
if (!state->mobile) return FALSE;
|
||||
if (state->active) touch(view, state, GDK_TOUCH_UPDATE, event->x, event->y);
|
||||
return TRUE;
|
||||
}
|
||||
double cn_preview_scale(void *opaque, double *device) {
|
||||
*device = gtk_widget_get_scale_factor(GTK_WIDGET(opaque));
|
||||
return webkit_web_view_get_zoom_level(WEBKIT_WEB_VIEW(opaque)) * *device;
|
||||
}
|
||||
int cn_preview_profile(void *opaque, const char *ua, double zoom, double density) {
|
||||
(void)density;
|
||||
GtkWidget *view = GTK_WIDGET(opaque);
|
||||
TouchState *state = g_object_get_data(G_OBJECT(view), "codenomad-mobile-touch");
|
||||
if (!state) {
|
||||
state = g_new0(TouchState, 1);
|
||||
g_object_set_data_full(G_OBJECT(view), "codenomad-mobile-touch", state, g_free);
|
||||
g_signal_connect(view, "button-press-event", G_CALLBACK(pressed), state);
|
||||
g_signal_connect(view, "button-release-event", G_CALLBACK(released), state);
|
||||
g_signal_connect(view, "motion-notify-event", G_CALLBACK(moved), state);
|
||||
}
|
||||
if (state->active) touch(view, state, GDK_TOUCH_CANCEL, state->x, state->y);
|
||||
state->active = FALSE; state->mobile = ua && *ua;
|
||||
WebKitWebView *web = WEBKIT_WEB_VIEW(view);
|
||||
webkit_settings_set_user_agent(webkit_web_view_get_settings(web), state->mobile ? ua : NULL);
|
||||
webkit_web_view_set_zoom_level(web, zoom);
|
||||
return 0;
|
||||
}
|
||||
|
|
@ -0,0 +1,19 @@
|
|||
#import <AppKit/AppKit.h>
|
||||
#import <WebKit/WebKit.h>
|
||||
@interface WKWebView (CNDeviceScale)
|
||||
- (void)_setOverrideDeviceScaleFactor:(CGFloat)scale;
|
||||
@end
|
||||
double cn_preview_scale(void *opaque,double *device) {
|
||||
WKWebView *view=(__bridge WKWebView *)opaque;
|
||||
*device=view.window.backingScaleFactor;
|
||||
return view.pageZoom * *device;
|
||||
}
|
||||
int cn_preview_profile(void *opaque,const char *ua,double zoom,double density) {
|
||||
WKWebView *view=(__bridge WKWebView *)opaque;
|
||||
view.customUserAgent=ua&&*ua?[NSString stringWithUTF8String:ua]:nil;
|
||||
view.pageZoom=zoom;
|
||||
if([view respondsToSelector:@selector(_setOverrideDeviceScaleFactor:)]) {
|
||||
[view _setOverrideDeviceScaleFactor:density];return 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
|
|
@ -3,7 +3,8 @@ use std::path::{Component, Path, PathBuf};
|
|||
use std::time::SystemTime;
|
||||
|
||||
pub(crate) const STABLE_IDENTIFIER: &str = "ai.neuralnomads.codenomad.client";
|
||||
pub(crate) const LOCAL_WINDOW_PREFIX: &str = "local-";
|
||||
mod local_window;
|
||||
pub(crate) use local_window::{local_window_id, local_window_label};
|
||||
const DEFAULT_CONFIG: &str = "~/.config/codenomad/config.json";
|
||||
|
||||
#[derive(Clone, Debug, Eq, PartialEq)]
|
||||
|
|
@ -18,24 +19,6 @@ pub(crate) struct IdentityScope {
|
|||
pub(crate) webview_data_directory: PathBuf,
|
||||
}
|
||||
|
||||
pub(crate) fn local_window_label(window_id: &str) -> Result<String, String> {
|
||||
let uuid =
|
||||
uuid::Uuid::parse_str(window_id).map_err(|_| "Invalid local window UUID".to_string())?;
|
||||
let normalized = uuid.to_string();
|
||||
if normalized != window_id {
|
||||
return Err("Local window UUID must be lowercase".to_string());
|
||||
}
|
||||
Ok(format!("{LOCAL_WINDOW_PREFIX}{normalized}"))
|
||||
}
|
||||
|
||||
pub(crate) fn local_window_id(label: &str) -> Result<String, String> {
|
||||
let id = label
|
||||
.strip_prefix(LOCAL_WINDOW_PREFIX)
|
||||
.ok_or_else(|| "Native operation is limited to local windows".to_string())?;
|
||||
local_window_label(id)?;
|
||||
Ok(id.to_string())
|
||||
}
|
||||
|
||||
#[derive(Debug, Eq, PartialEq)]
|
||||
pub(crate) struct LaunchProfile {
|
||||
pub(crate) key: String,
|
||||
|
|
@ -117,10 +100,18 @@ fn resolve_config_input(value: &str, cwd: &Path, home: &Path) -> PathBuf {
|
|||
|
||||
/** Keep the caller's original filename for backend config/legacy migration,
|
||||
* but remove dependence on the backend process's different working directory. */
|
||||
pub(crate) fn resolve_selected_config(raw: Option<&str>, cwd: &Path, home: &Path) -> Option<String> {
|
||||
pub(crate) fn resolve_selected_config(
|
||||
raw: Option<&str>,
|
||||
cwd: &Path,
|
||||
home: &Path,
|
||||
) -> Option<String> {
|
||||
raw.map(str::trim)
|
||||
.filter(|value| !value.is_empty())
|
||||
.map(|value| resolve_config_input(value, cwd, home).to_string_lossy().into_owned())
|
||||
.map(|value| {
|
||||
resolve_config_input(value, cwd, home)
|
||||
.to_string_lossy()
|
||||
.into_owned()
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) fn normalize_config_identity(raw: Option<&str>, cwd: &Path, home: &Path) -> String {
|
||||
|
|
@ -282,13 +273,25 @@ mod tests {
|
|||
for extension in ["json", "yaml"] {
|
||||
let raw = format!("alternate/../selected/custom.{extension}");
|
||||
let selected = resolve_selected_config(Some(&raw), &cwd, &home).unwrap();
|
||||
assert_eq!(PathBuf::from(&selected), cwd.join("selected").join(format!("custom.{extension}")));
|
||||
assert_eq!(
|
||||
PathBuf::from(&selected),
|
||||
cwd.join("selected").join(format!("custom.{extension}"))
|
||||
);
|
||||
if extension == "json" {
|
||||
assert_ne!(selected, normalize_config_identity(Some(&raw), &cwd, &home));
|
||||
}
|
||||
assert_eq!(
|
||||
resolve_selected_config(Some(&format!("~/selected/custom.{extension}")), &cwd, &home),
|
||||
Some(home.join("selected").join(format!("custom.{extension}")).to_string_lossy().into_owned())
|
||||
resolve_selected_config(
|
||||
Some(&format!("~/selected/custom.{extension}")),
|
||||
&cwd,
|
||||
&home
|
||||
),
|
||||
Some(
|
||||
home.join("selected")
|
||||
.join(format!("custom.{extension}"))
|
||||
.to_string_lossy()
|
||||
.into_owned()
|
||||
)
|
||||
);
|
||||
}
|
||||
assert_eq!(resolve_selected_config(None, &cwd, &home), None);
|
||||
|
|
|
|||
19
packages/tauri-app/src-tauri/src/identity/local_window.rs
Normal file
19
packages/tauri-app/src-tauri/src/identity/local_window.rs
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
pub(crate) const LOCAL_WINDOW_PREFIX: &str = "local-";
|
||||
|
||||
pub(crate) fn local_window_label(window_id: &str) -> Result<String, String> {
|
||||
let uuid =
|
||||
uuid::Uuid::parse_str(window_id).map_err(|_| "Invalid local window UUID".to_string())?;
|
||||
let normalized = uuid.to_string();
|
||||
if normalized != window_id {
|
||||
return Err("Local window UUID must be lowercase".to_string());
|
||||
}
|
||||
Ok(format!("{LOCAL_WINDOW_PREFIX}{normalized}"))
|
||||
}
|
||||
|
||||
pub(crate) fn local_window_id(label: &str) -> Result<String, String> {
|
||||
let id = label
|
||||
.strip_prefix(LOCAL_WINDOW_PREFIX)
|
||||
.ok_or_else(|| "Native operation is limited to local windows".to_string())?;
|
||||
local_window_label(id)?;
|
||||
Ok(id.to_string())
|
||||
}
|
||||
|
|
@ -11,7 +11,7 @@ const MAX_FOLDER_ATTEMPTS: u8 = 3;
|
|||
#[cfg(target_os = "windows")]
|
||||
const LOCAL_WINDOW_CONTEXT_SCRIPT: &str = "window.__CODENOMAD_RUNTIME_HOST__ = 'tauri'; window.__CODENOMAD_WINDOW_CONTEXT__ = 'local'; window.__CODENOMAD_BROWSER_CAPABILITIES__ = Object.freeze({embedding:true,history:true,emulation:true,automation:true});";
|
||||
#[cfg(any(target_os = "macos", target_os = "linux"))]
|
||||
const LOCAL_WINDOW_CONTEXT_SCRIPT: &str = "window.__CODENOMAD_RUNTIME_HOST__ = 'tauri'; window.__CODENOMAD_WINDOW_CONTEXT__ = 'local'; window.__CODENOMAD_BROWSER_CAPABILITIES__ = Object.freeze({embedding:true,history:true,emulation:false,automation:false,reason:'webkit'});";
|
||||
const LOCAL_WINDOW_CONTEXT_SCRIPT: &str = "window.__CODENOMAD_RUNTIME_HOST__ = 'tauri'; window.__CODENOMAD_WINDOW_CONTEXT__ = 'local'; window.__CODENOMAD_BROWSER_CAPABILITIES__ = Object.freeze({embedding:true,history:true,emulation:true,automation:false,reason:'webkit'});";
|
||||
#[cfg(not(any(target_os = "windows", target_os = "macos", target_os = "linux")))]
|
||||
const LOCAL_WINDOW_CONTEXT_SCRIPT: &str = "window.__CODENOMAD_RUNTIME_HOST__ = 'tauri'; window.__CODENOMAD_WINDOW_CONTEXT__ = 'local'; window.__CODENOMAD_BROWSER_CAPABILITIES__ = Object.freeze({embedding:false,history:false,emulation:false,automation:false,reason:'unsupported-platform'});";
|
||||
|
||||
|
|
|
|||
|
|
@ -290,13 +290,14 @@ pub(crate) fn handle_native_request(
|
|||
method: &str,
|
||||
params: Option<serde_json::Value>,
|
||||
deadline: u64,
|
||||
fence: native_request::RequestFence,
|
||||
) -> Result<serde_json::Value, String> {
|
||||
let state = app.state::<AppState>();
|
||||
match method {
|
||||
"browser.probe" | "browser.execute" | "browser.page.begin" | "browser.page.send"
|
||||
| "browser.page.check" | "browser.page.end" => state
|
||||
.browser_controller
|
||||
.handle_native(app, method, params, deadline),
|
||||
.handle_native(app, method, params, deadline, fence),
|
||||
"opencode.service.start" => native_service_start::start(params, deadline),
|
||||
"developer.status" => state.developer_mode.native_snapshot(app),
|
||||
"developer.restart" => state.developer_mode.request_restart(app),
|
||||
|
|
@ -514,20 +515,25 @@ async fn browser_target_register(
|
|||
) -> Result<(), String> {
|
||||
require_local_app_webview(&webview, &state)?;
|
||||
let window = webview.window();
|
||||
state.browser_controller.register(&window, payload)
|
||||
let controller = state.browser_controller.clone();
|
||||
tauri::async_runtime::spawn_blocking(move || controller.register(&window, payload))
|
||||
.await
|
||||
.map_err(|e| e.to_string())?
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
fn browser_target_update(
|
||||
async fn browser_target_update(
|
||||
webview: tauri::Webview,
|
||||
app: AppHandle,
|
||||
state: tauri::State<'_, AppState>,
|
||||
payload: browser_controller::BrowserTargetUpdate,
|
||||
) -> Result<(), String> {
|
||||
require_local_app_webview(&webview, &state)?;
|
||||
state
|
||||
.browser_controller
|
||||
.update(&app, webview.label(), payload)
|
||||
let controller = state.browser_controller.clone();
|
||||
let owner = webview.label().to_string();
|
||||
tauri::async_runtime::spawn_blocking(move || controller.update(&app, &owner, payload))
|
||||
.await
|
||||
.map_err(|e| e.to_string())?
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
|
|
|
|||
|
|
@ -1,5 +1,69 @@
|
|||
use serde::{Deserialize, Serialize};
|
||||
use serde_json::Value;
|
||||
use std::sync::Arc;
|
||||
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
|
||||
|
||||
#[derive(Clone)]
|
||||
pub(crate) struct RequestFence {
|
||||
deadline: Instant,
|
||||
deadline_ms: u64,
|
||||
current: Arc<dyn Fn() -> bool + Send + Sync>,
|
||||
}
|
||||
|
||||
impl RequestFence {
|
||||
pub(crate) fn new(
|
||||
deadline_ms: u64,
|
||||
current: impl Fn() -> bool + Send + Sync + 'static,
|
||||
) -> Self {
|
||||
let now = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|value| value.as_millis() as u64)
|
||||
.unwrap_or(u64::MAX);
|
||||
let deadline = Instant::now()
|
||||
.checked_add(Duration::from_millis(deadline_ms.saturating_sub(now)))
|
||||
.unwrap_or_else(Instant::now);
|
||||
Self {
|
||||
deadline,
|
||||
deadline_ms,
|
||||
current: Arc::new(current),
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn check(&self) -> Result<(), String> {
|
||||
if !(self.current)() {
|
||||
return Err("Native backend process is no longer current".into());
|
||||
}
|
||||
let now_ms = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|value| value.as_millis() as u64)
|
||||
.unwrap_or(u64::MAX);
|
||||
if Instant::now() >= self.deadline || now_ms >= self.deadline_ms {
|
||||
return Err("Native request expired before execution".into());
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
/// Shared by the desktop stream reader and native-engine regression harness.
|
||||
/// A replacement process never receives its predecessor's result.
|
||||
pub(crate) fn dispatch(
|
||||
request: NativeRequest,
|
||||
fence: RequestFence,
|
||||
handle: impl FnOnce(&NativeRequest, RequestFence) -> Result<Value, String>,
|
||||
) -> Option<String> {
|
||||
if !(fence.current)() {
|
||||
return None;
|
||||
}
|
||||
let result = fence.check().and_then(|()| handle(&request, fence.clone()));
|
||||
if !(fence.current)() {
|
||||
return None;
|
||||
}
|
||||
let result = match fence.check() {
|
||||
Ok(()) => result,
|
||||
Err(error) => Err(error),
|
||||
};
|
||||
Some(response(&request.id, result))
|
||||
}
|
||||
|
||||
pub(crate) const REQUEST_PREFIX: &str = "CODENOMAD_NATIVE_REQUEST:";
|
||||
pub(crate) const RESPONSE_PREFIX: &str = "CODENOMAD_NATIVE_RESPONSE:";
|
||||
|
|
@ -88,4 +152,53 @@ mod tests {
|
|||
"CODENOMAD_NATIVE_RESPONSE:{\"v\":1,\"id\":\"r1\",\"ok\":true,\"result\":{\"available\":true}}\n"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fences_expired_dispatch_retired_processes_and_late_results() {
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
let request = || NativeRequest {
|
||||
v: 1,
|
||||
id: "request".into(),
|
||||
method: "browser.probe".into(),
|
||||
params: None,
|
||||
deadline: u64::MAX,
|
||||
};
|
||||
let expired = RequestFence::new(0, || true);
|
||||
let response =
|
||||
dispatch(request(), expired, |_, _| panic!("expired work dispatched")).unwrap();
|
||||
assert!(response.contains("expired"));
|
||||
let current = Arc::new(AtomicBool::new(true));
|
||||
let predicate = current.clone();
|
||||
let fence = RequestFence::new(u64::MAX - 1, move || predicate.load(Ordering::SeqCst));
|
||||
// Use a representable deadline while keeping this test independent of clock changes.
|
||||
let fence = RequestFence {
|
||||
deadline: Instant::now() + Duration::from_secs(10),
|
||||
..fence
|
||||
};
|
||||
let queued = fence.clone();
|
||||
assert!(dispatch(request(), fence, |_, _| {
|
||||
current.store(false, Ordering::SeqCst);
|
||||
Ok(json!({"secret":"old process"}))
|
||||
})
|
||||
.is_none());
|
||||
assert!(queued.check().unwrap_err().contains("no longer current"));
|
||||
let late = RequestFence {
|
||||
deadline: Instant::now() + Duration::from_millis(5),
|
||||
deadline_ms: u64::MAX,
|
||||
current: Arc::new(|| true),
|
||||
};
|
||||
let response = dispatch(request(), late, |_, _| {
|
||||
std::thread::sleep(Duration::from_millis(10));
|
||||
Ok(json!({"stale":true}))
|
||||
})
|
||||
.unwrap();
|
||||
assert!(response.contains("expired"));
|
||||
assert!(!response.contains("stale"));
|
||||
let absolute = RequestFence {
|
||||
deadline: Instant::now() + Duration::from_secs(10),
|
||||
deadline_ms: 0,
|
||||
current: Arc::new(|| true),
|
||||
};
|
||||
assert!(absolute.check().unwrap_err().contains("expired"));
|
||||
}
|
||||
}
|
||||
|
|
|
|||
57
packages/tauri-app/tests/browser-parent/README.md
Normal file
57
packages/tauri-app/tests/browser-parent/README.md
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
# WebKit private parent-channel regression
|
||||
|
||||
Build the Node side from the checkout root:
|
||||
|
||||
```sh
|
||||
npx --no-install esbuild packages/tauri-app/tests/browser-parent/fixture.ts --bundle --platform=node --format=esm --outfile=.tmp/browser-parent-fixture.mjs
|
||||
```
|
||||
|
||||
On a macOS or Linux graphical session:
|
||||
|
||||
```sh
|
||||
python3 packages/tauri-app/scripts/test-browser-parent-webkit.py \
|
||||
--fixture /path/to/browser-parent-fixture.mjs \
|
||||
--output /path/to/artifacts --target /path/to/owned-cargo-target \
|
||||
--node /absolute/path/to/node
|
||||
```
|
||||
|
||||
Requirements: Rust, Node, OpenSSL, and the host's native development libraries (GTK 3 / WebKitGTK 4.1, version 2.40+ for isolated-world `evaluate_javascript`, on Linux). Linux inherits display/session variables from the user manager; `GDK_BACKEND=x11` or `wayland` selects the engine display backend. `--nested-kde` owns a separate virtual KWin Wayland compositor alongside an existing desktop and terminates only that compositor. The Cargo target is caller-owned and reusable. The initial offline build reuses the checkout lockfile's resolved versions and the production native build helper; it has a five-minute bound. Native execution is bounded to 150 seconds. Output includes `parent-result.json` and the native `parent-capture.png`.
|
||||
|
||||
The harness compiles production `BrowserController`, `EmbeddedViews`, local-window identity validation, native adapters and `native_request::dispatch`. Its child uses production `NativeParent`, real prefixed stdout requests and piped stdin responses. The owning-window inventory and renderer's open-claim callback are fixture-supplied; the existing application UI subscription has its own regression. The harness starts its own fixture child and HTTP/self-signed TLS servers. Keep its temporary data profile separate from application profiles. The public automation capability stays disabled while private requests are exercised directly.
|
||||
|
||||
The fixed fixture-only operations control visibility, a renderer dialog, duplicate ownership, close/reopen, response delay and backend retirement. Native capture-completion tests revoke authority at the callback guard's acceptance boundary, after the real engine has produced its result. The controller's native navigation and capture operations remain unchanged by the fixture. These operations belong only to this example executable, never renderer IPC or the application parent-method allowlist.
|
||||
|
||||
## Semantic reads
|
||||
|
||||
Regenerate the fixed reader and extracted Electron fixtures, then build the read child:
|
||||
|
||||
```sh
|
||||
node --import tsx scripts/build-webkit-read.ts
|
||||
npx --no-install esbuild packages/tauri-app/tests/browser-parent/read-fixture.ts --bundle --platform=node --format=esm --outfile=.tmp/read-fixture.mjs
|
||||
```
|
||||
|
||||
Pass that child as `--fixture` to the same native harness. It saves each fixture's tree and shared-renderer snapshot, and `read-result.json`. Checks include stable same-document refs, fragment/pushState preservation, removal/reinsertion, recreation, cross-document navigation, scope/query and visibility/closure fences. Fixture-only mutations use fixed scripts in the example. Read identities and semantic traversal stay inside the production isolated-world program. The document-start closed-root hook weakly associates roots with hosts; the isolated reader validates native reference-event endpoints against their exact host and document. Declarative closed roots are opaque to this public-DOM capture path.
|
||||
|
||||
Run `node --import tsx scripts/compare-webkit-read.ts <comparison-output> <native-artifact-parent> <optional-folder-prefix>` to assert rendered control parity and produce a side-by-side table for each fixture. The native parent contains `<prefix>macos`, `<prefix>x11` and `<prefix>wayland` folders; the default prefix is `p3-read-`. Frame contents are compared explicitly; native date-editor internals are filtered by the production snapshot renderer.
|
||||
|
||||
## Native actions
|
||||
|
||||
```sh
|
||||
npx --no-install esbuild packages/tauri-app/tests/browser-parent/action-fixture.ts --bundle --platform=node --format=esm --sourcemap=inline --outfile=.tmp/p4-action-fixture.mjs
|
||||
node --import tsx scripts/compare-webkit-actions.ts .tmp/p4-comparison
|
||||
```
|
||||
|
||||
Run the native harness with that action child as `--fixture` on macOS, Linux X11 and private Wayland. The shared `action-scenarios.ts` drives the production BrowserPage/PageDriver over real parent pipes and the direct CDP driver over Chromium. HTML input/form/preview fixtures are extracted from the Electron tests. Additional cases cover captured closed-root click/text/contenteditable/select, same-origin frame click/text, forged root endpoints, native ten-step drags, special input append/validation, modal/hidden/covered refusal, waits and declarative parser boundaries. Fixed observation/control programs belong only to the example host.
|
||||
|
||||
`action-result.json` records assertions, DOM event trust/modifiers and engine limitations. Native-only timeout cases cover both an expired unsent insertion and an admitted pointer whose handler blocks the page: each down appears once, exact-lease cleanup releases buttons/keys, and the insertion stays unsent. macOS traces can be enabled with `CN_INPUT_TRACE=1`; preserve the standalone-hover and Meta-equivalent event differences alongside the normalized state comparison.
|
||||
|
||||
## Device profiles
|
||||
|
||||
```sh
|
||||
npx --no-install esbuild packages/tauri-app/tests/browser-parent/mobile-fixture.ts --bundle --platform=node --format=esm --outfile=.tmp/mobile-fixture.mjs
|
||||
node --import tsx scripts/compare-webkit-mobile.ts .tmp/mobile-comparison
|
||||
```
|
||||
|
||||
Pass `mobile-fixture.mjs` to the graphical harness on macOS, GTK X11 and `--nested-kde` Wayland; repeat X11 with `GDK_SCALE=2`. `mobile-result.json` reports all six CSS viewports, HTTP/JS user-agent identity, screen/orientation, density, media and touch support. Assertions cover renderer zoom 1.5, native trusted click/touch, unchanged-profile no-op, invalid-profile refusal, resize/replacement lifetime, desktop reset and independent renderer/preview state. The Chromium comparison uses the same HTML and production Electron emulation adapter and records `chromium-mobile.json`.
|
||||
|
||||
Build `mobile-trial.ts` the same way for isolated page-zoom, DPR override, layout/view-scale and native GDK-touch experiments. Trial controls compile only into this qualification executable; profile application uses the production modules. The native WebKit limitations and optional macOS density selector are specified in `dev-docs/BROWSER_DEVICE_EMULATION.md`.
|
||||
10
packages/tauri-app/tests/browser-parent/action-extra.html
Normal file
10
packages/tauri-app/tests/browser-parent/action-extra.html
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
<!doctype html><meta charset="utf-8"><title>Action edge cases</title>
|
||||
<script>window.declarativeTemplates=0;new MutationObserver(records=>{for(const record of records)for(const node of record.addedNodes)if(node.nodeName==='TEMPLATE')window.declarativeTemplates++}).observe(document,{childList:true,subtree:true})</script>
|
||||
<div id="declarative"><template shadowrootmode="closed"><button>Declarative closed action</button></template></div>
|
||||
<label>Multiple <select id="multiple" multiple><option value="one" selected>One</option><option value="two">Two</option><optgroup label="Locked" disabled><option value="three">Three</option></optgroup></select></label>
|
||||
<label>Readonly <input id="readonly" readonly value="Kept"></label>
|
||||
<label>Password <input id="password" type="password" value="private"></label>
|
||||
<label>Clock <input id="clock" type="time"></label>
|
||||
<label>Focus locked clock <input id="focus-locked" type="time" onfocus="this.readOnly=true"></label>
|
||||
<label>Moment <input id="moment" type="datetime-local"></label>
|
||||
<div id="rich" contenteditable role="textbox" aria-label="Rich lines">Old rich</div>
|
||||
57
packages/tauri-app/tests/browser-parent/action-fixture.ts
Normal file
57
packages/tauri-app/tests/browser-parent/action-fixture.ts
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
import assert from 'node:assert/strict'
|
||||
import { randomUUID } from 'node:crypto'
|
||||
import { writeFileSync } from 'node:fs'
|
||||
import { join } from 'node:path'
|
||||
import { NativeParent } from '../../../server/src/native-parent'
|
||||
import { installShutdownStdinHandler } from '../../../server/src/shutdown'
|
||||
import { BrowserPage } from '../../../server/src/automation/browser-page'
|
||||
import { parsePageAction } from '../../../server/src/automation/page/parse'
|
||||
import { nativeDriver } from '../../../server/src/automation/page/native-driver'
|
||||
import { actionScenarios } from './action-scenarios'
|
||||
process.setSourceMapsEnabled(true)
|
||||
const parent=new NativeParent(process.stdout,true),browser=new BrowserPage(parent)
|
||||
installShutdownStdinHandler(process.stdin,async()=>{},line=>parent.handleLine(line))
|
||||
const [base,output]=process.argv.slice(2),engine=process.platform==='darwin'?'macos':output.includes('wayland')?'wayland':'x11'
|
||||
const request=<T=any>(method:string,params:object={},timeout=20_000)=>parent.request<T>(method,{sessionID:'fixture-session',...params},timeout)
|
||||
const control=(kind:string)=>request('fixture.action-control',{kind})
|
||||
const observe=()=>request('fixture.action-probe')
|
||||
async function main(){
|
||||
assert.equal(parent.browserAutomationUnavailable,true)
|
||||
await request('browser.execute',{command:{action:'open',url:`${base}/read/preview`}})
|
||||
const result=await actionScenarios({
|
||||
navigate:name=>{console.error(`action fixture: ${name}`);return request('browser.execute',{command:{action:'navigate',url:`${base}/read/${name}`}})},
|
||||
snapshot:async()=>(await browser.execute('fixture-session',{action:'snapshot'})).snapshot as string,
|
||||
act:action=>{console.error(`action: ${action.action}${action.key?` ${action.key}`:''}`);return browser.execute('fixture-session',parsePageAction(action,'snapshot')!)},observe,control,
|
||||
},engine)
|
||||
for(const admitted of [false,true]) {
|
||||
await request('browser.execute',{command:{action:'navigate',url:`${base}/read/inputs`}})
|
||||
const operationID=randomUUID(),descriptor=await request('browser.page.begin',{operationID})
|
||||
const send=(operation:any,timeout?:number)=>request('browser.page.send',{operationID,generation:descriptor.generation,operation},timeout)
|
||||
const driver=nativeDriver(descriptor,operation=>send(operation),async()=>{})
|
||||
const target=(await driver.tree()).find(node=>node.role==='button' && node.name==='Hover me')!
|
||||
const quad=(await driver.geometry(target.node!))[0],x=(quad[0]+quad[4])/2,y=(quad[1]+quad[5])/2
|
||||
const pointer={type:'mousePressed' as const,x,y,button:'left' as const,buttons:1,clickCount:1}
|
||||
if(admitted) {
|
||||
await control('arm-busy-click')
|
||||
await assert.rejects(send({op:'pointer',event:pointer},150),/expired|timed out|timeout/i)
|
||||
} else {
|
||||
await driver.pointer(pointer)
|
||||
const query=(await driver.tree()).find(node=>node.role==='textbox' && node.name==='Query')!
|
||||
await driver.call('focus',await driver.resolve(query.node!),[])
|
||||
await driver.keyboard({type:'rawKeyDown',key:'Shift',code:'ShiftLeft',modifiers:8})
|
||||
await control('busy')
|
||||
await assert.rejects(send({op:'insert-text',text:'MUST-NOT-REPLAY'},20),/expired|timed out|timeout/i)
|
||||
}
|
||||
await request('browser.page.end',{operationID})
|
||||
const observed=await observe();result.observations[admitted?'admittedTimeout':'queuedTimeout']=observed
|
||||
writeFileSync(join(output,'action-observations.json'),JSON.stringify(result,null,2))
|
||||
assert.equal(observed.audit.filter((event:any)=>event.type==='pointerdown').length,1)
|
||||
assert.equal(observed.audit.filter((event:any)=>event.type==='pointerup').length,1)
|
||||
if(!admitted) assert.ok(observed.audit.some((event:any)=>event.type==='keyup' && event.key==='Shift'))
|
||||
assert.ok(!JSON.stringify(observed.fields).includes('MUST-NOT-REPLAY'))
|
||||
}
|
||||
result.checks.push('native parent timeouts: queued mutation refused, admitted pointer not replayed; held pointer/key released on exact lease end')
|
||||
writeFileSync(join(output,'action-result.json'),JSON.stringify(result,null,2))
|
||||
console.error(JSON.stringify({passed:true,engine,checks:result.checks,limitations:result.limitations}))
|
||||
}
|
||||
main().then(()=>{parent.close();process.exit(0)},error=>{writeFileSync(join(output,'action-result.json'),JSON.stringify({passed:false,engine,error:String(error)},null,2));console.error(error);parent.close();process.exit(1)})
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
{
|
||||
"auditor": "(() => {\n window.__actionAudit=[];\n for(const type of ['keydown','keyup','pointerdown','pointerup','pointermove','click','contextmenu','wheel','beforeinput','input','change']) document.addEventListener(type,event=>{\n if(window.__actionAudit.length<1000) window.__actionAudit.push({type,id:event.target.id,key:event.key||'',inputType:event.inputType||'',trusted:event.isTrusted,buttons:event.buttons||0,meta:!!event.metaKey,ctrl:!!event.ctrlKey,shift:!!event.shiftKey});\n },true);\n document.addEventListener('contextmenu',event=>event.preventDefault());\n})()",
|
||||
"programs": {
|
||||
"observe": "(() => {\n const fields={};for(const item of document.querySelectorAll('input,textarea,select,[contenteditable]')) {const editable=item.hasAttribute('contenteditable')&&item.getAttribute('contenteditable')!=='false';fields[item.id||item.getAttribute('aria-label')||item.type]={value:item.type==='password'?'masked':editable?item.innerText:item.value,html:editable?item.innerHTML:undefined,checked:['checkbox','radio'].includes(item.type)?item.checked:undefined,selected:item.selectedOptions?Array.from(item.selectedOptions).map(option=>option.label):undefined};}\n const value={fields,audit:window.__actionAudit||[],log:window.log||[],dropped:!!window.dropped,closedClicks:window.closedClicks||0,closedTrusted:!!window.closedTrusted,nestedClicked:document.body.dataset.nestedClicked,recreatedClicked:document.body.dataset.recreatedClicked,frameClicked:document.querySelector('#same')?.contentDocument?.body.dataset.clicked,hover:document.querySelector('#hover-state')?.textContent,scroll:document.querySelector('#scroller')?.scrollTop||0,pageScroll:scrollY,declarative:{height:document.querySelector('#declarative')?.getBoundingClientRect().height||0,marker:document.querySelector('#declarative')?.getAttribute('data-codenomad-shadow-boundary'),open:!!document.querySelector('#declarative')?.shadowRoot,templates:window.declarativeTemplates||0}};\n value.capabilities={predictedEvents:typeof PointerEvent.prototype.getPredictedEvents};\n let output=document.getElementById('__cn_probe');if(!output){output=document.createElement('output');output.id='__cn_probe';output.style='position:fixed;right:0;bottom:0;font-size:1px;pointer-events:none';document.body.append(output)}output.textContent='__CN_PROBE__'+JSON.stringify(value);\n })()",
|
||||
"remove-probe": "document.getElementById('__cn_probe')?.remove()",
|
||||
"remove-ready": "document.querySelector('#later').textContent=''",
|
||||
"schedule-ready": "{document.querySelector('#later').textContent='';setTimeout(()=>document.querySelector('#later').textContent='Ready now',250)}",
|
||||
"hidden-open": "{const details=document.createElement('details');details.id='hidden-details';details.open=true;details.innerHTML='<summary>Target</summary><input aria-label=\"Hidden target\">';document.body.append(details)}",
|
||||
"hidden-close": "document.querySelector('#hidden-details').open=false",
|
||||
"long-rows": "{const section=document.createElement('section');section.innerHTML=Array.from({length:35},(_,index)=>'<button style=\"display:block;height:65px\" aria-label=\"Row '+(index+1)+'\">Row</button>').join('');document.body.append(section);scrollTo(0,0)}",
|
||||
"busy": "{const until=performance.now()+500;while(performance.now()<until){}}",
|
||||
"arm-busy-click": "document.getElementById('hover-target').addEventListener('pointerdown',()=>{const until=performance.now()+500;while(performance.now()<until){}},{once:true})",
|
||||
"remove-closed": "document.querySelector('#closed').remove()",
|
||||
"recreate-shadow": "document.open();document.write('<div id=\"recreated\"></div>');document.close();{const root=document.querySelector('#recreated').attachShadow({mode:'closed'});root.innerHTML='<button>Recreated closed action</button><input aria-label=\"Recreated closed field\">';root.querySelector('button').addEventListener('click',event=>document.body.dataset.recreatedClicked=event.isTrusted)}",
|
||||
"pagehide": "dispatchEvent(new PageTransitionEvent('pagehide',{persisted:true}))",
|
||||
"frame-pagehide": "document.querySelector('#same').contentWindow.dispatchEvent(new PageTransitionEvent('pagehide',{persisted:true}))",
|
||||
"show-modal": "document.querySelector('#modal').showModal()",
|
||||
"forge-shadow": "{const host=document.createElement('div');host.id='forged';host.setAttribute('data-codenomad-shadow-boundary','closed');document.body.append(host);host.addEventListener('__codenomad_request_shadow_v1',()=>{for(const candidate of [document.body,...(typeof closedRoot==='undefined'?[]:[closedRoot])])host.dispatchEvent(new PointerEvent('__codenomad_shadow_v1',{predictedEvents:[new PointerEvent('__codenomad_reference_v1',{relatedTarget:candidate})]}))})}"
|
||||
}
|
||||
}
|
||||
30
packages/tauri-app/tests/browser-parent/action-programs.ts
Normal file
30
packages/tauri-app/tests/browser-parent/action-programs.ts
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
/** Fixed fixture-only programs, consumed by both the CDP and native test hosts. */
|
||||
export const ACTION_AUDITOR=`(() => {
|
||||
window.__actionAudit=[];
|
||||
for(const type of ['keydown','keyup','pointerdown','pointerup','pointermove','click','contextmenu','wheel','beforeinput','input','change']) document.addEventListener(type,event=>{
|
||||
if(window.__actionAudit.length<1000) window.__actionAudit.push({type,id:event.target.id,key:event.key||'',inputType:event.inputType||'',trusted:event.isTrusted,buttons:event.buttons||0,meta:!!event.metaKey,ctrl:!!event.ctrlKey,shift:!!event.shiftKey});
|
||||
},true);
|
||||
document.addEventListener('contextmenu',event=>event.preventDefault());
|
||||
})()`
|
||||
export const ACTION_PROGRAMS={
|
||||
observe:`(() => {
|
||||
const fields={};for(const item of document.querySelectorAll('input,textarea,select,[contenteditable]')) {const editable=item.hasAttribute('contenteditable')&&item.getAttribute('contenteditable')!=='false';fields[item.id||item.getAttribute('aria-label')||item.type]={value:item.type==='password'?'masked':editable?item.innerText:item.value,html:editable?item.innerHTML:undefined,checked:['checkbox','radio'].includes(item.type)?item.checked:undefined,selected:item.selectedOptions?Array.from(item.selectedOptions).map(option=>option.label):undefined};}
|
||||
const value={fields,audit:window.__actionAudit||[],log:window.log||[],dropped:!!window.dropped,closedClicks:window.closedClicks||0,closedTrusted:!!window.closedTrusted,nestedClicked:document.body.dataset.nestedClicked,recreatedClicked:document.body.dataset.recreatedClicked,frameClicked:document.querySelector('#same')?.contentDocument?.body.dataset.clicked,hover:document.querySelector('#hover-state')?.textContent,scroll:document.querySelector('#scroller')?.scrollTop||0,pageScroll:scrollY,declarative:{height:document.querySelector('#declarative')?.getBoundingClientRect().height||0,marker:document.querySelector('#declarative')?.getAttribute('data-codenomad-shadow-boundary'),open:!!document.querySelector('#declarative')?.shadowRoot,templates:window.declarativeTemplates||0}};
|
||||
value.capabilities={predictedEvents:typeof PointerEvent.prototype.getPredictedEvents};
|
||||
let output=document.getElementById('__cn_probe');if(!output){output=document.createElement('output');output.id='__cn_probe';output.style='position:fixed;right:0;bottom:0;font-size:1px;pointer-events:none';document.body.append(output)}output.textContent='__CN_PROBE__'+JSON.stringify(value);
|
||||
})()`,
|
||||
"remove-probe":"document.getElementById('__cn_probe')?.remove()",
|
||||
"remove-ready":"document.querySelector('#later').textContent=''",
|
||||
"schedule-ready":"{document.querySelector('#later').textContent='';setTimeout(()=>document.querySelector('#later').textContent='Ready now',250)}",
|
||||
"hidden-open":"{const details=document.createElement('details');details.id='hidden-details';details.open=true;details.innerHTML='<summary>Target</summary><input aria-label=\"Hidden target\">';document.body.append(details)}",
|
||||
"hidden-close":"document.querySelector('#hidden-details').open=false",
|
||||
"long-rows":"{const section=document.createElement('section');section.innerHTML=Array.from({length:35},(_,index)=>'<button style=\"display:block;height:65px\" aria-label=\"Row '+(index+1)+'\">Row</button>').join('');document.body.append(section);scrollTo(0,0)}",
|
||||
busy:"{const until=performance.now()+500;while(performance.now()<until){}}",
|
||||
"arm-busy-click":"document.getElementById('hover-target').addEventListener('pointerdown',()=>{const until=performance.now()+500;while(performance.now()<until){}},{once:true})",
|
||||
"remove-closed":"document.querySelector('#closed').remove()",
|
||||
"recreate-shadow":`document.open();document.write('<div id="recreated"></div>');document.close();{const root=document.querySelector('#recreated').attachShadow({mode:'closed'});root.innerHTML='<button>Recreated closed action</button><input aria-label="Recreated closed field">';root.querySelector('button').addEventListener('click',event=>document.body.dataset.recreatedClicked=event.isTrusted)}`,
|
||||
pagehide:"dispatchEvent(new PageTransitionEvent('pagehide',{persisted:true}))",
|
||||
"frame-pagehide":"document.querySelector('#same').contentWindow.dispatchEvent(new PageTransitionEvent('pagehide',{persisted:true}))",
|
||||
"show-modal":"document.querySelector('#modal').showModal()",
|
||||
"forge-shadow":`{const host=document.createElement('div');host.id='forged';host.setAttribute('data-codenomad-shadow-boundary','closed');document.body.append(host);host.addEventListener('__codenomad_request_shadow_v1',()=>{for(const candidate of [document.body,...(typeof closedRoot==='undefined'?[]:[closedRoot])])host.dispatchEvent(new PointerEvent('__codenomad_shadow_v1',{predictedEvents:[new PointerEvent('__codenomad_reference_v1',{relatedTarget:candidate})]}))})}`,
|
||||
} as const
|
||||
184
packages/tauri-app/tests/browser-parent/action-scenarios.ts
Normal file
184
packages/tauri-app/tests/browser-parent/action-scenarios.ts
Normal file
|
|
@ -0,0 +1,184 @@
|
|||
import assert from "node:assert/strict"
|
||||
export interface Harness {
|
||||
navigate(name:string):Promise<unknown>
|
||||
snapshot():Promise<string>
|
||||
act(action:Record<string,unknown>):Promise<Record<string,unknown>>
|
||||
observe():Promise<any>
|
||||
control(kind:string):Promise<unknown>
|
||||
}
|
||||
export async function actionScenarios(h:Harness,engine:string) {
|
||||
const checks:string[]=[],limitations:string[]=[],observations:Record<string,unknown>={}
|
||||
const ref=async(pattern:RegExp)=>{const snapshot=await h.snapshot();const line=snapshot.split('\n').find(line=>pattern.test(line));assert.ok(line,`${pattern}: ${snapshot}`);return /\[([^\]]+)\]/.exec(line)![1]}
|
||||
const type=async(pattern:RegExp,text:string,clear=true)=>h.act({action:"type",ref:await ref(pattern),text,clear})
|
||||
const press=async(key:string,target?:string)=>h.act({action:"press",key,...target?{ref:target}:{}})
|
||||
const until=async(predicate:(value:any)=>boolean)=>{const deadline=Date.now()+3000;let value;do{value=await h.observe();if(predicate(value))return value;await new Promise(resolve=>setTimeout(resolve,30))}while(Date.now()<deadline);assert.ok(predicate(value));return value}
|
||||
await h.navigate('preview')
|
||||
assert.equal((await type(/textbox "Title"/,'Nouveau é🙂')).value,'Nouveau é🙂')
|
||||
assert.equal((await type(/textbox "Title"/,' appended',false)).value,'Nouveau é🙂 appended')
|
||||
await h.act({action:'select',ref:await ref(/combobox "Plan"/),values:['Exploration']})
|
||||
await h.act({action:'click',ref:await ref(/button "Confirm"/)})
|
||||
assert.match(await h.snapshot(),/Nouveau é🙂 appended:wayfinder/)
|
||||
observations.preview=await h.observe()
|
||||
assert.ok((observations.preview as any).audit.some((event:any)=>event.type==='click' && event.trusted))
|
||||
checks.push('automation-preview: Unicode clear/append, select, trusted click')
|
||||
|
||||
await h.navigate('forms')
|
||||
assert.equal((await type(/textbox "Notes"/,'New notes')).value,'New notes')
|
||||
assert.equal((await type(/textbox "Rich"/,'Rich é🙂')).value,'Rich é🙂')
|
||||
assert.equal((await type(/Date "Due"/,'2026-11-07')).value,'2026-11-07')
|
||||
const select=await ref(/combobox "Scenario"/)
|
||||
await h.act({action:'select',ref:select,values:['wayfinder']})
|
||||
const before=(await h.observe()).audit.filter((event:any)=>event.id==='scenario' && ['input','change'].includes(event.type)).length
|
||||
await h.act({action:'select',ref:select,values:['Exploration']})
|
||||
assert.equal((await h.observe()).audit.filter((event:any)=>event.id==='scenario' && ['input','change'].includes(event.type)).length,before)
|
||||
await assert.rejects(h.act({action:'select',ref:select,values:['Debugging']}),/disabled/)
|
||||
await h.act({action:'click',ref:await ref(/checkbox "Agree"/)})
|
||||
observations.forms=await h.observe();assert.equal((observations.forms as any).fields.agree.checked,true)
|
||||
checks.push('automation-forms: native/contenteditable/date, select value/label, unchanged events, disabled option')
|
||||
|
||||
await h.navigate('edges')
|
||||
await assert.rejects(type(/textbox "Locked"/,'Forbidden'),/disabled/)
|
||||
await type(/textbox "Mail"/,'x@y.test')
|
||||
assert.equal((await type(/textbox "Mail"/,'.ok',false)).value,'x@y.test.ok')
|
||||
await type(/spinbutton "Count"/,'42')
|
||||
assert.equal((await type(/spinbutton "Count"/,'3',false)).value,'423')
|
||||
await type(/ColorWell "Tint"/,'#123abc')
|
||||
await type(/slider "Volume"/,'60')
|
||||
await assert.rejects(type(/Date "Day"/,'invalid'),/not a valid/)
|
||||
assert.equal((await h.observe()).fields.day.value,'2026-01-02')
|
||||
const outside=await ref(/textbox "Outside"/);await h.control('show-modal')
|
||||
await assert.rejects(h.act({action:'type',ref:outside,text:'Forbidden'}),/cannot take focus/)
|
||||
observations.edges=await h.observe()
|
||||
checks.push('automation-forms edges: disabled, email/number/color/range setters and date validation')
|
||||
|
||||
await h.navigate('inputs')
|
||||
const query=await ref(/textbox "Query"/)
|
||||
await type(/textbox "Query"/,'hello')
|
||||
await press('Enter',query)
|
||||
let observed=await h.observe()
|
||||
observations.enter=observed
|
||||
if(!observed.log.includes('submit:hello') && engine==='macos') limitations.push('Enter in a background WKWebView input did not submit the form')
|
||||
else assert.ok(observed.log.includes('submit:hello'))
|
||||
await press('Tab')
|
||||
await press('Shift+Tab')
|
||||
await press('Mod+K',query)
|
||||
await type(/textbox "Query"/,'',false)
|
||||
await press('Shift+a',query);await press('+');await press('`')
|
||||
assert.equal((await h.observe()).fields.query.value,'helloA+`')
|
||||
await type(/textbox "Query"/,'hello')
|
||||
await h.act({action:'hover',ref:await ref(/button "Hover me"/)})
|
||||
observed=await h.observe()
|
||||
if(observed.hover!=='hovered' && engine==='macos') limitations.push('Initial standalone hover did not deliver pointerenter')
|
||||
else assert.equal(observed.hover,'hovered')
|
||||
await h.act({action:'click',ref:await ref(/button "Open menu"/)})
|
||||
await h.act({action:'click',ref:await ref(/menuitem "Menu item"/)})
|
||||
await h.act({action:'click',ref:await ref(/button "Open menu"/),button:'right'})
|
||||
await h.act({action:'scroll',ref:await ref(/region "Scroller"/),direction:'down',amount:500})
|
||||
observed=await until(value=>value.scroll>0)
|
||||
await h.act({action:'drag',ref:await ref(/button "Handle"/),to:await ref(/region "Drop zone"/)})
|
||||
observed=await until(value=>value.dropped)
|
||||
await assert.rejects(h.act({action:'click',ref:await ref(/button "Under"/)}),/covered/)
|
||||
await h.control('schedule-ready')
|
||||
assert.equal((await h.act({action:'wait',text:'Ready now',timeout:15000})).found,true)
|
||||
await h.control('remove-ready')
|
||||
assert.equal((await h.act({action:'wait',text:'Ready now',gone:true,timeout:15000})).found,false)
|
||||
await assert.rejects(h.act({action:'wait',text:'absent',timeout:50}),/did not appear/)
|
||||
observed=await h.observe()
|
||||
assert.ok(observed.audit.some((event:any)=>event.type==='contextmenu' && event.trusted))
|
||||
assert.ok(observed.audit.some((event:any)=>event.type==='wheel' && event.trusted))
|
||||
const textInputs=observed.audit.filter((event:any)=>event.type==='input' && event.id==='query')
|
||||
assert.ok(textInputs.length>0 && textInputs.every((event:any)=>event.trusted))
|
||||
assert.ok(observed.audit.some((event:any)=>event.type==='keydown' && event.key.toLowerCase()==='k' && (engine==='macos' ? event.meta : event.ctrl)))
|
||||
assert.ok(observed.audit.filter((event:any)=>['pointerdown','pointerup','pointermove','wheel','keydown','keyup','click'].includes(event.type)).every((event:any)=>event.trusted))
|
||||
observations.inputs=observed
|
||||
await h.control('hidden-open');const hidden=await ref(/textbox "Hidden target"/);await h.control('hidden-close')
|
||||
await assert.rejects(press('Enter',hidden),/cannot take focus/)
|
||||
await h.control('long-rows')
|
||||
await assert.rejects(h.act({action:'drag',ref:await ref(/button "Row 1"/),to:await ref(/button "Row 35"/)}),/no visible box/)
|
||||
await h.act({action:'scroll',direction:'down',amount:500})
|
||||
await until(value=>value.pageScroll>0)
|
||||
checks.push('automation-input: Enter/Tab, hover, left/right pointer, native element/page wheel, drag, covered/hidden/offscreen refusal, wait appear/disappear')
|
||||
|
||||
await h.navigate('advanced')
|
||||
await h.act({action:'click',ref:await ref(/button "Closed action"/)})
|
||||
assert.equal((await type(/textbox "Closed field"/,'Closed é🙂')).value,'Closed é🙂')
|
||||
assert.equal((await type(/textbox "Closed rich"/,'Closed rich é🙂')).value,'Closed rich é🙂')
|
||||
assert.equal((await type(/textbox "Closed rich"/,' appended',false)).value,'Closed rich é🙂 appended')
|
||||
await press('Enter',await ref(/textbox "Closed rich"/))
|
||||
assert.equal((await type(/textbox "Closed rich"/,'tail',false)).value,'Closed rich é🙂 appended\ntail')
|
||||
assert.deepEqual((await h.act({action:'select',ref:await ref(/combobox "Closed plan"/),values:['Two']})).selected,['Two'])
|
||||
assert.equal((await type(/textbox "Nested closed field"/,'Nested é🙂')).value,'Nested é🙂')
|
||||
await h.act({action:'click',ref:await ref(/button "Nested closed action"/)})
|
||||
assert.equal((await h.observe()).nestedClicked,'true')
|
||||
if(engine!=='chromium') assert.equal((await h.act({action:'wait',text:'Closed é🙂',timeout:1000})).found,true)
|
||||
assert.equal((await type(/textbox "Shadow field"/,'Open shadow')).value,'Open shadow')
|
||||
if(engine!=='chromium') {
|
||||
assert.equal((await type(/textbox "Frame field"/,'Frame é🙂')).value,'Frame é🙂')
|
||||
await h.act({action:'click',ref:await ref(/button "Frame action"/)})
|
||||
assert.equal((await h.observe()).frameClicked,'true')
|
||||
const frameRef=await ref(/textbox "Frame field"/);await h.control('frame-pagehide')
|
||||
await assert.rejects(h.act({action:'type',ref:frameRef,text:'stale frame'}),/no longer|snapshot/)
|
||||
assert.equal((await type(/textbox "Frame field"/,'Fresh frame')).value,'Fresh frame')
|
||||
}
|
||||
else observations.frame='Contents absent from Chromium main-frame AX tree'
|
||||
observed=await h.observe();assert.equal(observed.closedClicks,1);assert.equal(observed.closedTrusted,true)
|
||||
observations.closed=observed
|
||||
observations.closedSnapshot=await h.snapshot()
|
||||
assert.equal(observed.audit.filter((event:any)=>event.id==='closed' && event.type==='input' && event.inputType==='insertParagraph').length,1)
|
||||
await h.control('forge-shadow')
|
||||
const forged=await h.snapshot();assert.equal(forged.split('button "Closed action"').length-1,1)
|
||||
if(engine!=='chromium') assert.match(forged,/Closed shadow root \(unavailable\)/)
|
||||
const old=await ref(/textbox "Closed field"/);await h.control('remove-closed')
|
||||
await assert.rejects(h.act({action:'type',ref:old,text:'stale'}),/no longer|snapshot/)
|
||||
await h.control('recreate-shadow')
|
||||
assert.equal((await type(/textbox "Recreated closed field"/,'Recreated é🙂')).value,'Recreated é🙂')
|
||||
await h.act({action:'click',ref:await ref(/button "Recreated closed action"/)})
|
||||
assert.equal((await h.observe()).recreatedClicked,'true')
|
||||
if(engine!=='chromium') {
|
||||
const recreated=await ref(/textbox "Recreated closed field"/);await h.control('pagehide')
|
||||
await assert.rejects(h.act({action:'type',ref:recreated,text:'stale recreated'}),/stale|snapshot|changed/)
|
||||
}
|
||||
checks.push('closed/nested/open shadow: click/text/contenteditable/select, forged roots, stale/recreated refs; same-origin frame on WebKit')
|
||||
|
||||
await h.navigate('extra')
|
||||
const extraSnapshot=await h.snapshot()
|
||||
if(engine==='chromium') assert.match(extraSnapshot,/Declarative closed action/)
|
||||
else assert.ok(!extraSnapshot.includes('Declarative closed action'))
|
||||
await assert.rejects(type(/textbox "Readonly"/,'Forbidden'),/read-only/)
|
||||
assert.equal((await type(/textbox "Password"/,'secret')).value,'•'.repeat(6))
|
||||
assert.equal((await type(/textbox "Password"/,'')).value,'')
|
||||
await type(/InputTime "Clock"/,'12:34')
|
||||
await assert.rejects(type(/InputTime "Focus locked clock"/,'12:34'),/read-only/)
|
||||
assert.equal((await h.observe()).fields['focus-locked'].value,'')
|
||||
await type(/DateTime "Moment"/,'2026-11-07T12:34')
|
||||
const multiple=await ref(/listbox "Multiple"/)
|
||||
assert.deepEqual((await h.act({action:'select',ref:multiple,values:['one','Two']})).selected,['One','Two'])
|
||||
await assert.rejects(h.act({action:'select',ref:multiple,values:['Three']}),/disabled/)
|
||||
assert.deepEqual((await h.observe()).fields.multiple.selected,['One','Two'])
|
||||
assert.deepEqual((await h.act({action:'click',ref:await ref(/option "Two"/)})).selected,['Two'])
|
||||
await assert.rejects(h.act({action:'click',ref:multiple}),/native select/)
|
||||
await type(/textbox "Rich lines"/,'one')
|
||||
await press('Enter',await ref(/textbox "Rich lines"/))
|
||||
await type(/textbox "Rich lines"/,'two',false)
|
||||
const enterRich=await h.observe();observations.richEnter=enterRich
|
||||
assert.equal(enterRich.fields.rich.value,'one\ntwo')
|
||||
assert.equal(enterRich.audit.filter((event:any)=>event.id==='rich' && event.type==='input' && event.inputType==='insertParagraph').length,1)
|
||||
await press('Mod+A',await ref(/textbox "Rich lines"/));await press('Enter')
|
||||
observations.modThenEnter=await h.observe()
|
||||
assert.equal((observations.modThenEnter as any).audit.filter((event:any)=>event.id==='rich' && event.type==='input' && event.inputType==='insertParagraph').length,2)
|
||||
assert.ok(!((observations.modThenEnter as any).fields.rich.value as string).includes('one'),'Mod+A selects the editable text before Enter')
|
||||
assert.equal((observations.inputs as any).audit.filter((event:any)=>event.type==='pointermove' && event.id==='handle' && event.buttons===1).length,10)
|
||||
observations.declarative=observations.modThenEnter
|
||||
const declarative=(observations.modThenEnter as any).declarative
|
||||
assert.ok(declarative.height>0);assert.equal(declarative.marker,null);assert.equal(declarative.open,false);assert.equal(declarative.templates,0)
|
||||
if(engine==='macos') {
|
||||
const audit=(observations.modThenEnter as any).audit
|
||||
if(!audit.some((event:any)=>event.type==='keyup' && event.key==='Meta')) limitations.push('Mod+A: WKWebView did not expose the Meta keyup in DOM events')
|
||||
if(!audit.some((event:any)=>event.type==='keydown' && event.key.toLowerCase()==='a' && event.meta)) limitations.push('Mod+A selects through the native responder; WKWebView consumes the A keydown before DOM listeners')
|
||||
const enters=audit.filter((event:any)=>event.type==='keydown' && event.key==='Enter')
|
||||
if(enters.length<2) limitations.push('Enter following Mod+A was not delivered to contenteditable')
|
||||
if(!audit.some((event:any)=>event.type==='keyup' && event.key==='Enter')) limitations.push('Standalone Enter: WKWebView omitted DOM keyup')
|
||||
}
|
||||
checks.push('multiple select/disabled optgroup, readonly, password masking, time/datetime, contenteditable Enter and Mod chord')
|
||||
return {passed:true,engine,checks,limitations,observations}
|
||||
}
|
||||
85
packages/tauri-app/tests/browser-parent/fixture.ts
Normal file
85
packages/tauri-app/tests/browser-parent/fixture.ts
Normal file
|
|
@ -0,0 +1,85 @@
|
|||
import assert from "node:assert/strict"
|
||||
import { writeFileSync } from "node:fs"
|
||||
import { join } from "node:path"
|
||||
import { NativeParent } from "../../../server/src/native-parent"
|
||||
import { installShutdownStdinHandler } from "../../../server/src/shutdown"
|
||||
|
||||
const parent = new NativeParent(process.stdout, true)
|
||||
installShutdownStdinHandler(process.stdin, async () => { throw new Error("Unexpected fixture shutdown command") }, line => parent.handleLine(line))
|
||||
const [base, output] = process.argv.slice(2)
|
||||
const request = <T = unknown>(method: string, params: unknown = {}, timeout = 15_000) => parent.request<T>(method, { sessionID: "fixture-session", ...params as object }, timeout)
|
||||
const execute = <T = {url: string}>(action: string, url?: string, timeout?: number) => request<T>("browser.execute", { command: { action, url } }, timeout)
|
||||
const checks: string[] = []
|
||||
async function check(name: string, run: () => Promise<void>) { await run(); checks.push(name); console.error(`PASS ${name}`) }
|
||||
try {
|
||||
assert.equal(parent.browserAutomationUnavailable, true, "public automation gate remains closed")
|
||||
await check("no preview", async () => { await assert.rejects(request("browser.probe"), /No visible/ ) })
|
||||
await check("autonomous open and redirect", async () => {
|
||||
assert.equal((await execute("open", `${base}/redirect`)).url, `${base}/ok`)
|
||||
assert.deepEqual(await request("browser.probe"), { available: true })
|
||||
})
|
||||
await check("visible PNG", async () => {
|
||||
const screenshot = await execute<{ image: {mime: string; data: string}; url: string }>("screenshot")
|
||||
assert.equal(screenshot.image.mime, "image/png")
|
||||
assert.ok(screenshot.image.data.length <= 16 * 1024 * 1024)
|
||||
const png = Buffer.from(screenshot.image.data, "base64")
|
||||
assert.equal(png.subarray(0,8).toString("hex"), "89504e470d0a1a0a")
|
||||
writeFileSync(join(output, "parent-capture.png"), png)
|
||||
})
|
||||
await check("fragment navigation", async () => { assert.equal((await execute("navigate", `${base}/ok#fragment`)).url, `${base}/ok#fragment`) })
|
||||
await check("cross-document redirect", async () => { assert.equal((await execute("navigate", `${base}/redirect`)).url, `${base}/ok`) })
|
||||
await check("failed load", async () => { await assert.rejects(execute("navigate", "http://127.0.0.1:1/unavailable"), /failed|load|navigation/i) })
|
||||
await execute("navigate", `${base}/ok`)
|
||||
await check("TLS failure", async () => {
|
||||
assert.ok(process.env.CN_FIXTURE_TLS_URL)
|
||||
await assert.rejects(execute("navigate", process.env.CN_FIXTURE_TLS_URL), /failed|load|navigation/i)
|
||||
})
|
||||
await execute("navigate", `${base}/ok`)
|
||||
await check("navigation deadline", async () => {
|
||||
await assert.rejects(execute("navigate", `${base}/slow`, 30), /timed out|expired/i)
|
||||
})
|
||||
await execute("navigate", `${base}/ok`)
|
||||
await check("hidden target", async () => {
|
||||
await request("fixture.hide")
|
||||
await assert.rejects(request("browser.probe"), /No visible/)
|
||||
await assert.rejects(execute("screenshot"), /No visible/)
|
||||
await request("fixture.show")
|
||||
})
|
||||
await check("overlay hides authority", async () => {
|
||||
await request("fixture.overlay")
|
||||
await assert.rejects(execute("screenshot"), /No visible/)
|
||||
await request("fixture.show")
|
||||
})
|
||||
await check("duplicate target", async () => {
|
||||
await request("fixture.duplicate")
|
||||
await assert.rejects(request("browser.probe"), /Multiple visible/)
|
||||
await request("fixture.removeDuplicate")
|
||||
})
|
||||
await check("native capture callback after hide", async () => {
|
||||
await assert.rejects(request("fixture.capture-hide"), /No visible/)
|
||||
await request("fixture.show")
|
||||
})
|
||||
await check("native capture callback after close", async () => {
|
||||
await assert.rejects(request("fixture.capture-close"), /No visible/)
|
||||
assert.equal((await execute("open", `${base}/ok`)).url, `${base}/ok`)
|
||||
})
|
||||
await check("late reply is discarded", async () => {
|
||||
await assert.rejects(request("fixture.late", {}, 20), /timed out/)
|
||||
assert.deepEqual(await request("browser.probe"), { available: true })
|
||||
})
|
||||
await check("close and reopen", async () => {
|
||||
await request("fixture.close")
|
||||
await assert.rejects(request("browser.probe"), /No visible/)
|
||||
assert.equal((await execute("open", `${base}/ok`)).url, `${base}/ok`)
|
||||
await request("fixture.close")
|
||||
})
|
||||
await check("retired process gets no response", async () => { await assert.rejects(request("fixture.retire", {}, 50), /timed out/) })
|
||||
writeFileSync(join(output, "parent-result.json"), JSON.stringify({passed: true, checks}, null, 2))
|
||||
parent.close()
|
||||
process.exit(0)
|
||||
} catch (error) {
|
||||
writeFileSync(join(output, "parent-result.json"), JSON.stringify({passed: false, checks, error: String(error)}, null, 2))
|
||||
console.error(error)
|
||||
parent.close()
|
||||
process.exit(1)
|
||||
}
|
||||
74
packages/tauri-app/tests/browser-parent/mobile-fixture.ts
Normal file
74
packages/tauri-app/tests/browser-parent/mobile-fixture.ts
Normal file
|
|
@ -0,0 +1,74 @@
|
|||
import assert from 'node:assert/strict'
|
||||
import {writeFileSync} from 'node:fs'
|
||||
import {NativeParent} from '../../../server/src/native-parent'
|
||||
import {installShutdownStdinHandler} from '../../../server/src/shutdown'
|
||||
import {BrowserPage} from '../../../server/src/automation/browser-page'
|
||||
const parent=new NativeParent(process.stdout,true),page=new BrowserPage(parent)
|
||||
installShutdownStdinHandler(process.stdin,async()=>{},line=>parent.handleLine(line))
|
||||
const [base,output]=process.argv.slice(2),request=(method:string,input:object={})=>parent.request<any>(method,{sessionID:'fixture-session',...input},20000)
|
||||
const probe=(target?:string)=>request('fixture.mobile-probe',{target})
|
||||
async function until(check:(v:any)=>boolean,target?:string){const end=Date.now()+8000;let last;do{try{last=await probe(target);if(check(last))return last}catch{}await new Promise(r=>setTimeout(r,40))}while(Date.now()<end);throw new Error('Mobile observation did not settle: '+JSON.stringify(last))}
|
||||
async function main(){
|
||||
assert.equal(parent.browserAutomationUnavailable,true)
|
||||
await request('browser.execute',{command:{action:'open',url:base+'/mobile'}})
|
||||
await request('fixture.mobile-init',{url:base+'/mobile'})
|
||||
const primary=await until(v=>v.ua===v.httpUA,'primary'),other=await until(v=>v.ua===v.httpUA,'other'),baseline=await until(v=>v.ua===v.httpUA)
|
||||
const rows=[]
|
||||
for(const [name,width,height] of [['responsive',600,400],['desktop',1440,900],['tablet',768,1024],['tabletLandscape',1024,768],['mobile',390,844],['mobileLandscape',844,390]] as const){
|
||||
const viewport=name==='responsive'?undefined:{width,height}
|
||||
const preset=name==='mobile'||name==='mobileLandscape'?name:'none'
|
||||
await request('fixture.mobile-size',{width:600,height:400,presentation:{css:true}})
|
||||
const before=await until(v=>v.ua===v.httpUA)
|
||||
await request('fixture.mobile-profile',{preset,viewport})
|
||||
const value=await until(v=>v.inner[0]===width&&v.inner[1]===height&&v.ua===v.httpUA&&(name==='responsive'||v.load!==before.load))
|
||||
if(name!=='responsive')assert.notEqual(value.load,before.load,'changing viewport/profile reloads')
|
||||
assert.equal(value.ua.includes('(Mobile)'),preset!=='none')
|
||||
assert.deepEqual((await probe('primary')).inner,primary.inner);assert.equal((await probe('primary')).ua,primary.ua)
|
||||
assert.deepEqual((await probe('other')).inner,other.inner);assert.equal((await probe('other')).ua,other.ua)
|
||||
rows.push({name,...value});console.error(`PASS profile ${name}: ${value.inner} DPR ${value.dpr}`)
|
||||
}
|
||||
// Native renderer zoom is read by the host, rather than inferred from JS DPR.
|
||||
await request('fixture.mobile-trial',{target:'primary',zoom:1.5})
|
||||
await until(v=>v.inner[0]<primary.inner[0],'primary')
|
||||
await request('fixture.mobile-size',{width:600,height:400,presentation:{css:true,viewport:{width:390,height:844}}})
|
||||
const beforeZoomed=await probe()
|
||||
await request('fixture.mobile-profile',{preset:'mobile',viewport:{width:390,height:844}})
|
||||
const zoomed=await until(v=>v.inner[0]===390&&v.inner[1]===844&&v.ua===v.httpUA&&v.load!==beforeZoomed.load)
|
||||
const snapshot=await page.execute('fixture-session',{action:'snapshot'})
|
||||
const ref=/button "Touch target" \[([^\]]+)\]/.exec(snapshot.snapshot as string)![1]
|
||||
await page.execute('fixture-session',{action:'click',ref})
|
||||
const clicked=await until(v=>v.events.some((e:any)=>e.type==='click'))
|
||||
assert.equal(clicked.events.filter((e:any)=>e.type==='click').length,1)
|
||||
assert.ok(clicked.events.every((e:any)=>e.trusted))
|
||||
if(process.platform==='linux')assert.ok(clicked.events.some((e:any)=>e.type==='touchstart'))
|
||||
else assert.equal(clicked.events.some((e:any)=>e.type==='touchstart'),false)
|
||||
const currentLoad=clicked.load
|
||||
await request('fixture.mobile-profile',{preset:'mobile',viewport:{width:390,height:844}})
|
||||
assert.equal((await probe()).load,currentLoad,'identical profile is a no-op')
|
||||
await assert.rejects(request('fixture.mobile-profile',{preset:'arbitrary'}),/Invalid browser/)
|
||||
assert.equal((await probe()).load,currentLoad)
|
||||
await assert.rejects(request('fixture.mobile-profile',{preset:'none',viewport:{width:0,height:844}}),/Invalid browser/)
|
||||
assert.equal((await probe()).load,currentLoad)
|
||||
await request('fixture.mobile-size',{width:500,height:300,presentation:{css:true,viewport:{width:390,height:844}}})
|
||||
const resized=await until(v=>v.inner[0]===390&&v.inner[1]===844)
|
||||
assert.equal(resized.ua,clicked.ua);assert.equal(resized.load,currentLoad,'resizing retains the document/profile')
|
||||
await request('fixture.mobile-trial',{target:'primary',zoom:1})
|
||||
await request('fixture.mobile-size',{width:600,height:400,presentation:{css:true}})
|
||||
await request('fixture.mobile-profile',{preset:'none'})
|
||||
const reset=await until(v=>v.ua===baseline.ua&&v.httpUA===baseline.ua&&v.inner[0]===600&&v.inner[1]===400)
|
||||
assert.notEqual(reset.load,currentLoad)
|
||||
assert.equal((await probe('other')).ua,other.ua)
|
||||
const resetSnapshot=await page.execute('fixture-session',{action:'snapshot'})
|
||||
const resetRef=/button "Touch target" \[([^\]]+)\]/.exec(resetSnapshot.snapshot as string)![1]
|
||||
await page.execute('fixture-session',{action:'click',ref:resetRef})
|
||||
const resetClicked=await until(v=>v.events.some((e:any)=>e.type==='click'))
|
||||
assert.equal(resetClicked.events.some((e:any)=>e.type==='touchstart'),false,'reset restores mouse input')
|
||||
await request('fixture.mobile-profile',{preset:'mobile'})
|
||||
await until(v=>v.ua===clicked.ua&&v.httpUA===clicked.ua)
|
||||
await request('fixture.mobile-replace',{url:base+'/mobile'})
|
||||
const replaced=await until(v=>v.ua===baseline.ua&&v.httpUA===baseline.ua)
|
||||
assert.equal(replaced.dpr,baseline.dpr,'replacement clears profile density')
|
||||
const result={passed:true,platform:process.platform,rows,zoomed,clicked,resized,reset,resetClicked,replaced,primary,other}
|
||||
writeFileSync(output+'/mobile-result.json',JSON.stringify(result,null,2));console.error(JSON.stringify({passed:true,profiles:rows.length,zoomed:zoomed.inner,reset:reset.inner}));parent.close();process.exit(0)
|
||||
}
|
||||
main().catch(error=>{writeFileSync(output+'/mobile-result.json',JSON.stringify({passed:false,error:String(error)},null,2));console.error(error);parent.close();process.exit(1)})
|
||||
123
packages/tauri-app/tests/browser-parent/mobile-native.rs
Normal file
123
packages/tauri-app/tests/browser-parent/mobile-native.rs
Normal file
|
|
@ -0,0 +1,123 @@
|
|||
use super::*;
|
||||
use std::ffi::{c_char, c_void, CString};
|
||||
extern "C" {
|
||||
fn cn_mobile_trial(
|
||||
view: *mut c_void,
|
||||
ua: *const c_char,
|
||||
zoom: f64,
|
||||
dpr: f64,
|
||||
width: f64,
|
||||
height: f64,
|
||||
mode: i32,
|
||||
) -> i32;
|
||||
fn cn_mobile_touch(view: *mut c_void, phase: i32, x: f64, y: f64) -> i32;
|
||||
}
|
||||
pub(super) fn handle(
|
||||
app: &tauri::AppHandle,
|
||||
controller: &BrowserController,
|
||||
id: &str,
|
||||
method: &str,
|
||||
input: &Value,
|
||||
fence: native_request::RequestFence,
|
||||
) -> Result<Value, String> {
|
||||
fence.check()?;
|
||||
let prefix = if input["target"] == "other" {
|
||||
"browser-mobile-other-".into()
|
||||
} else {
|
||||
format!("browser-{id}-")
|
||||
};
|
||||
let view = if input["target"] == "primary" {
|
||||
app.get_webview(OWNER)
|
||||
} else {
|
||||
app.webviews()
|
||||
.into_values()
|
||||
.find(|view| view.label().starts_with(&prefix))
|
||||
}
|
||||
.ok_or("Mobile fixture view missing")?;
|
||||
if method == "fixture.mobile-init" {
|
||||
let url = input["url"].as_str().ok_or("Missing fixture URL")?;
|
||||
app.get_webview(OWNER)
|
||||
.ok_or("Missing renderer")?
|
||||
.navigate(url.parse().map_err(|_| "Invalid fixture URL")?)
|
||||
.map_err(|e| e.to_string())?;
|
||||
let registration=serde_json::from_value(json!({"sessionId":"mobile-other","registrationId":"mobile-other","url":url,"bounds":bounds(),"visible":false})).unwrap();
|
||||
controller.register(&app.get_window(OTHER).unwrap(), registration)?;
|
||||
update(app, controller, OTHER, "mobile-other", true)?;
|
||||
std::thread::sleep(Duration::from_millis(200));
|
||||
return Ok(Value::Null);
|
||||
}
|
||||
if method == "fixture.mobile-replace" {
|
||||
let registration=serde_json::from_value(json!({"sessionId":SESSION,"registrationId":id,"url":input["url"],"bounds":bounds(),"visible":true})).map_err(|e|e.to_string())?;
|
||||
controller.register(
|
||||
&app.get_window(OWNER).ok_or("Missing renderer")?,
|
||||
registration,
|
||||
)?;
|
||||
return Ok(Value::Null);
|
||||
}
|
||||
if method == "fixture.mobile-size" {
|
||||
let update=serde_json::from_value(json!({"registrationId":id,"visible":true,"presentation":input["presentation"],"bounds":{"x":10,"y":60,"width":input["width"],"height":input["height"]}})).map_err(|e|e.to_string())?;
|
||||
controller.update(app, OWNER, update)?;
|
||||
return Ok(Value::Null);
|
||||
}
|
||||
if method == "fixture.mobile-probe" {
|
||||
view.eval("report()").map_err(|e| e.to_string())?;
|
||||
let bytes = embedded_view::input::read(
|
||||
&view,
|
||||
embedded_view::input::Read::Checkpoint,
|
||||
std::time::Instant::now() + Duration::from_secs(5),
|
||||
move || fence.check(),
|
||||
)?;
|
||||
let value: Value = serde_json::from_slice(&bytes).map_err(|e| e.to_string())?;
|
||||
return serde_json::from_str(
|
||||
value["body"]
|
||||
.as_str()
|
||||
.and_then(|s| s.split_once("__CN_MOBILE__").map(|(_, v)| v))
|
||||
.ok_or("Mobile report missing")?
|
||||
.trim(),
|
||||
)
|
||||
.map_err(|e| e.to_string());
|
||||
}
|
||||
if method == "fixture.mobile-profile" {
|
||||
let action=serde_json::from_value(json!({"registrationId":id,"action":"emulate","preset":input["preset"],"viewport":input["viewport"]})).unwrap();
|
||||
return controller.action(app, OWNER, action);
|
||||
}
|
||||
let ua = CString::new(input["ua"].as_str().unwrap_or("")).map_err(|_| "Invalid trial UA")?;
|
||||
let zoom = input["zoom"].as_f64().unwrap_or(1.0);
|
||||
let dpr = input["dpr"].as_f64().unwrap_or(0.0);
|
||||
let mode = input["mode"].as_i64().unwrap_or(0) as i32;
|
||||
let width = input["cssWidth"].as_f64().unwrap_or(390.0);
|
||||
let height = input["cssHeight"].as_f64().unwrap_or(844.0);
|
||||
let phase = input["phase"].as_i64().unwrap_or(0) as i32;
|
||||
let (sender, receiver) = std::sync::mpsc::sync_channel(1);
|
||||
let touch = method == "fixture.mobile-touch";
|
||||
view.with_webview(move |native| {
|
||||
if let Err(error) = fence.check() {
|
||||
let _ = sender.send(Err(error));
|
||||
return;
|
||||
}
|
||||
#[cfg(target_os = "macos")]
|
||||
let ptr = native.inner();
|
||||
#[cfg(target_os = "linux")]
|
||||
let ptr = {
|
||||
use gtk::glib::object::ObjectType;
|
||||
native.inner().as_ptr().cast::<c_void>()
|
||||
};
|
||||
let result = unsafe {
|
||||
if touch {
|
||||
cn_mobile_touch(ptr, phase, 40.0, 140.0)
|
||||
} else {
|
||||
cn_mobile_trial(ptr, ua.as_ptr(), zoom, dpr, width, height, mode)
|
||||
}
|
||||
};
|
||||
let _ = sender.send(Ok(result));
|
||||
})
|
||||
.map_err(|e| e.to_string())?;
|
||||
let result = receiver
|
||||
.recv_timeout(Duration::from_secs(5))
|
||||
.map_err(|_| "Native mobile trial timed out")??;
|
||||
if !touch && input["reload"] != false {
|
||||
view.reload().map_err(|e| e.to_string())?;
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(150));
|
||||
Ok(json!({"native":result}))
|
||||
}
|
||||
23
packages/tauri-app/tests/browser-parent/mobile-trial-linux.c
Normal file
23
packages/tauri-app/tests/browser-parent/mobile-trial-linux.c
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
#include <gtk/gtk.h>
|
||||
#include <webkit2/webkit2.h>
|
||||
int cn_mobile_trial(void *opaque,const char *ua,double zoom,double dpr,double width,double height,int mode) {
|
||||
(void)dpr;(void)width;(void)height;(void)mode;
|
||||
WebKitWebView *view=WEBKIT_WEB_VIEW(opaque);
|
||||
webkit_settings_set_user_agent(webkit_web_view_get_settings(view),ua&&*ua?ua:NULL);
|
||||
webkit_web_view_set_zoom_level(view,zoom);
|
||||
return 0;
|
||||
}
|
||||
int cn_mobile_touch(void *opaque,int phase,double x,double y) {
|
||||
int emulating=phase<10;phase%=10;
|
||||
GtkWidget *view=GTK_WIDGET(opaque);
|
||||
GdkWindow *window=gtk_widget_get_window(view);
|
||||
GdkSeat *seat=gdk_display_get_default_seat(gtk_widget_get_display(view));
|
||||
if(!window||!seat)return 0;
|
||||
GdkEvent *event=gdk_event_new(phase==0?GDK_TOUCH_BEGIN:phase==1?GDK_TOUCH_UPDATE:GDK_TOUCH_END);
|
||||
event->touch.window=g_object_ref(window);event->touch.send_event=FALSE;
|
||||
event->touch.time=GDK_CURRENT_TIME;event->touch.x=x;event->touch.y=y;
|
||||
event->touch.sequence=(GdkEventSequence *)view;event->touch.emulating_pointer=emulating;
|
||||
int ox=0,oy=0;gdk_window_get_origin(window,&ox,&oy);event->touch.x_root=ox+x;event->touch.y_root=oy+y;
|
||||
gdk_event_set_device(event,gdk_seat_get_pointer(seat));
|
||||
gtk_widget_event(view,event);gdk_event_free(event);return 1;
|
||||
}
|
||||
21
packages/tauri-app/tests/browser-parent/mobile-trial-macos.m
Normal file
21
packages/tauri-app/tests/browser-parent/mobile-trial-macos.m
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
#import <AppKit/AppKit.h>
|
||||
#import <WebKit/WebKit.h>
|
||||
@interface WKWebView (CNMobileTrial)
|
||||
- (void)_setOverrideDeviceScaleFactor:(CGFloat)scale;
|
||||
- (void)_setLayoutMode:(NSUInteger)mode;
|
||||
- (void)_setFixedLayoutSize:(CGSize)size;
|
||||
- (void)_setViewScale:(CGFloat)scale;
|
||||
@end
|
||||
int cn_mobile_trial(void *opaque,const char *ua,double zoom,double dpr,double width,double height,int mode) {
|
||||
WKWebView *view=(__bridge WKWebView *)opaque;
|
||||
view.customUserAgent=ua&&*ua?[NSString stringWithUTF8String:ua]:nil;
|
||||
view.pageZoom=zoom;
|
||||
int flags=0;
|
||||
if([view respondsToSelector:@selector(_setOverrideDeviceScaleFactor:)]) {flags|=1;[view _setOverrideDeviceScaleFactor:dpr];}
|
||||
if([view respondsToSelector:@selector(_setLayoutMode:)] && [view respondsToSelector:@selector(_setFixedLayoutSize:)]) {
|
||||
flags|=2;[view _setLayoutMode:mode==1?1:0];[view _setFixedLayoutSize:CGSizeMake(width,height)];
|
||||
}
|
||||
if([view respondsToSelector:@selector(_setViewScale:)]) {flags|=4;[view _setViewScale:mode==2?0.5:1];}
|
||||
return flags;
|
||||
}
|
||||
int cn_mobile_touch(void *opaque,int phase,double x,double y) {(void)opaque;(void)phase;(void)x;(void)y;return 0;}
|
||||
25
packages/tauri-app/tests/browser-parent/mobile-trial.ts
Normal file
25
packages/tauri-app/tests/browser-parent/mobile-trial.ts
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
import { NativeParent } from '../../../server/src/native-parent'
|
||||
import { installShutdownStdinHandler } from '../../../server/src/shutdown'
|
||||
import { writeFileSync } from 'node:fs'
|
||||
const parent=new NativeParent(process.stdout,true)
|
||||
installShutdownStdinHandler(process.stdin,async()=>{},line=>parent.handleLine(line))
|
||||
const [base,output]=process.argv.slice(2)
|
||||
const request=(method:string,input:object={})=>parent.request<any>(method,{sessionID:'fixture-session',...input},20000)
|
||||
async function main(){
|
||||
await request('browser.execute',{command:{action:'open',url:base+'/mobile'}})
|
||||
await request('fixture.mobile-init',{url:base+'/mobile'})
|
||||
const rows=[]
|
||||
for(const [name,settings] of Object.entries({baseline:{},zoom:{zoom:1.5},density:{dpr:2.75},combined:{zoom:1.5,dpr:2.75},fixed:{mode:1,cssWidth:390,cssHeight:844},viewScale:{mode:2},mobileUA:{ua:'Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.0 Mobile/15E148 Safari/604.1'}})) {
|
||||
await request('fixture.mobile-size',{width:600,height:400})
|
||||
const native=await request('fixture.mobile-trial',settings)
|
||||
const row={name,native,guest:await request('fixture.mobile-probe'),primary:await request('fixture.mobile-probe',{target:'primary'}),other:await request('fixture.mobile-probe',{target:'other'})};rows.push(row)
|
||||
}
|
||||
await request('fixture.mobile-trial',{})
|
||||
for(const phase of [0,1,2]) await request('fixture.mobile-touch',{phase})
|
||||
rows.push({name:'touch',guest:await request('fixture.mobile-probe')})
|
||||
await request('fixture.mobile-trial',{})
|
||||
for(const phase of [10,11,12]) await request('fixture.mobile-touch',{phase})
|
||||
rows.push({name:'touchNoEmulation',guest:await request('fixture.mobile-probe')})
|
||||
writeFileSync(output+'/mobile-trial.json',JSON.stringify(rows,null,2));console.error(JSON.stringify(rows.map(row=>({name:row.name,inner:row.guest.inner,dpr:row.guest.dpr,events:row.guest.events}))));parent.close();process.exit(0)
|
||||
}
|
||||
main().catch(e=>{console.error(e);parent.close();process.exit(1)})
|
||||
9
packages/tauri-app/tests/browser-parent/mobile.html
Normal file
9
packages/tauri-app/tests/browser-parent/mobile.html
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
<!doctype html><meta name="viewport" content="width=device-width,initial-scale=1"><meta charset="utf-8"><title>Native device profile</title>
|
||||
<style>body{margin:0}#tap{position:absolute;left:20px;top:120px;width:100px;height:60px}output{font:10px monospace;overflow-wrap:anywhere}dialog{display:none}</style>
|
||||
<button id="tap">Touch target</button><dialog>Overlay</dialog><output id="report"></output>
|
||||
<script>
|
||||
window.httpUA={{UA}};window.loadId={{LOAD}};window.inputEvents=[];
|
||||
for(const type of ['touchstart','touchmove','touchend','pointerdown','pointerup','click']) document.addEventListener(type,event=>{inputEvents.push({type,trusted:event.isTrusted,pointer:event.pointerType||'',touches:event.touches?.length??0});report()}, {passive:true});
|
||||
function report(){document.querySelector('#report').textContent='__CN_MOBILE__'+JSON.stringify({load:loadId,inner:[innerWidth,innerHeight],screen:[screen.width,screen.height,screen.availWidth,screen.availHeight],orientation:screen.orientation?{type:screen.orientation.type,angle:screen.orientation.angle}:null,dpr:devicePixelRatio,ua:navigator.userAgent,httpUA,platform:navigator.platform,touch:{max:navigator.maxTouchPoints,constructor:typeof TouchEvent,handler:'ontouchstart'in window},media:Object.fromEntries(['pointer:coarse','pointer:fine','hover:none','hover:hover','any-pointer:coarse','any-hover:none','orientation:portrait','orientation:landscape'].map(value=>[value,matchMedia('('+value+')').matches])),events:inputEvents})}
|
||||
addEventListener('resize',report);report();
|
||||
</script>
|
||||
34
packages/tauri-app/tests/browser-parent/read-advanced.html
Normal file
34
packages/tauri-app/tests/browser-parent/read-advanced.html
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
<!doctype html><meta charset="utf-8"><title>Semantic boundaries</title>
|
||||
<main>
|
||||
<span id="first">First</span><span id="second"> choice</span><p id="description">Detailed help</p>
|
||||
<button id="aria-button" aria-labelledby="first second" aria-describedby="description">Ignored content</button>
|
||||
<label>Secret <input id="secret" type="password" value="never-expose-this-secret"></label>
|
||||
<label>Required <input id="required" required readonly value="Fixed"></label>
|
||||
<label>Checked <input id="checked" type="checkbox" checked></label>
|
||||
<div id="mixed" role="checkbox" aria-checked="mixed" aria-label="Mixed"></div>
|
||||
<button id="expanded" aria-expanded="true" aria-pressed="mixed">Expanded</button>
|
||||
<button id="disabled" aria-disabled="true">Disabled</button>
|
||||
<label>Multiple <select id="multiple" multiple><option selected>One</option><option>Two</option><optgroup disabled label="Locked"><option>Three</option></optgroup></select></label>
|
||||
<section hidden><button>Hidden attribute</button></section>
|
||||
<section style="display:none"><button>Hidden display</button></section>
|
||||
<section inert><button>Hidden inert</button></section>
|
||||
<section aria-hidden="true"><button>Hidden ARIA</button></section>
|
||||
<x-open id="open"></x-open><div id="closed"></div>
|
||||
<div id="disabled-shadow" aria-disabled="true"></div>
|
||||
<div id="hidden-shadow" style="display:none"></div>
|
||||
<label>Search secret <input type="password" role="searchbox" value="private"></label>
|
||||
<iframe id="same" title="Same-origin frame" src="/read/frame"></iframe>
|
||||
<iframe id="cross" title="Cross-origin frame" src="{{CROSS}}/read/frame"></iframe>
|
||||
</main>
|
||||
<script>
|
||||
window.__codenomadPageOnly=true;
|
||||
document.querySelector('#open').attachShadow({mode:'open'}).innerHTML='<span id="label">Shadow label</span><button id="shadow-button" aria-labelledby="label">Wrong text</button><label>Shadow field <input value="Shadow value"></label>';
|
||||
const closedRoot=document.querySelector('#closed').attachShadow({mode:'closed'});
|
||||
closedRoot.innerHTML='<button id="closed-button">Closed action</button><label>Closed field <input id="closed-field" value="Old closed"></label><div contenteditable role="textbox" aria-label="Closed rich">Old closed rich</div><label>Closed plan <select><option>One</option><option>Two</option></select></label><div id="nested"></div>';
|
||||
const nestedRoot=closedRoot.querySelector('#nested').attachShadow({mode:'closed'});
|
||||
nestedRoot.innerHTML='<button>Nested closed action</button><input aria-label="Nested closed field">';
|
||||
nestedRoot.querySelector('button').addEventListener('click',event=>{document.body.dataset.nestedClicked=event.isTrusted});
|
||||
closedRoot.querySelector('button').addEventListener('click',event=>{window.closedClicks=(window.closedClicks||0)+1;window.closedTrusted=event.isTrusted});
|
||||
document.querySelector('#disabled-shadow').attachShadow({mode:'open'}).innerHTML='<button>Disabled shadow action</button>';
|
||||
document.querySelector('#hidden-shadow').attachShadow({mode:'open'}).innerHTML='<button>Hidden shadow action</button>';
|
||||
</script>
|
||||
95
packages/tauri-app/tests/browser-parent/read-fixture.ts
Normal file
95
packages/tauri-app/tests/browser-parent/read-fixture.ts
Normal file
|
|
@ -0,0 +1,95 @@
|
|||
import assert from "node:assert/strict"
|
||||
import { randomUUID } from "node:crypto"
|
||||
import { writeFileSync } from "node:fs"
|
||||
import { join } from "node:path"
|
||||
import { NativeParent } from "../../../server/src/native-parent"
|
||||
import { installShutdownStdinHandler } from "../../../server/src/shutdown"
|
||||
import { BrowserPage } from "../../../server/src/automation/browser-page"
|
||||
import { snapshotTree } from "../../../server/src/automation/page/semantic-tree"
|
||||
import { renderSnapshot } from "../../../server/src/automation/page/snapshot"
|
||||
import type { SemanticNode, PageOperation } from "../../../server/src/automation/page/protocol"
|
||||
|
||||
const parent=new NativeParent(process.stdout,true)
|
||||
installShutdownStdinHandler(process.stdin,async()=>{},line=>parent.handleLine(line))
|
||||
const [base,output]=process.argv.slice(2), checks:string[]=[]
|
||||
const request=<T=Record<string,unknown>>(method:string,params:object={})=>parent.request<T>(method,{sessionID:"fixture-session",...params},20_000)
|
||||
const navigate=(url:string)=>request("browser.execute",{command:{action:"navigate",url}})
|
||||
const browser=new BrowserPage(parent)
|
||||
async function lease() {
|
||||
const operationID=randomUUID()
|
||||
const target=await request<{generation:string;driver:string}>("browser.page.begin",{operationID})
|
||||
assert.equal(target.driver,"webkit")
|
||||
return {
|
||||
generation:target.generation,
|
||||
send:<T=Record<string,unknown>>(operation:PageOperation)=>request<T>("browser.page.send",{operationID,generation:target.generation,operation}),
|
||||
end:()=>request("browser.page.end",{operationID}),
|
||||
check:()=>request("browser.page.check",{operationID,generation:target.generation}),
|
||||
}
|
||||
}
|
||||
async function main() {
|
||||
assert.equal(parent.browserAutomationUnavailable,true)
|
||||
await request("browser.execute",{command:{action:"open",url:`${base}/read/preview`}})
|
||||
for(const name of ["preview","forms","edges","inputs","advanced"]) {
|
||||
await navigate(`${base}/read/${name}`)
|
||||
if(name === 'inputs') await browser.execute('fixture-session',{action:'wait',text:'Ready now',gone:false,timeout:5000})
|
||||
const current=await lease()
|
||||
const {nodes}=await current.send<{nodes:SemanticNode[]}>({op:"tree"})
|
||||
assert.ok(nodes.length)
|
||||
assert.deepEqual((await current.send<{nodes:SemanticNode[]}>({op:"tree"})).nodes,nodes)
|
||||
const snapshot=renderSnapshot(snapshotTree(nodes),id=>`n${id}`).text
|
||||
writeFileSync(join(output,`${name}.json`),JSON.stringify({nodes,snapshot},null,2))
|
||||
writeFileSync(join(output,`${name}.txt`),snapshot)
|
||||
if(name === "advanced") {
|
||||
assert.ok(!JSON.stringify(nodes).includes("never-expose-this-secret"))
|
||||
for(const label of ["First choice","Shadow label","Frame action","Closed action","Closed field","Cross-origin frame (unavailable)"]) assert.ok(snapshot.includes(label),`${label}: ${snapshot}`)
|
||||
for(const hidden of ["Hidden attribute","Hidden display","Hidden inert","Hidden ARIA","Hidden shadow action"]) assert.ok(!snapshot.includes(hidden))
|
||||
const secret=nodes.find(node=>node.role === "textbox" && node.name === "Secret")!
|
||||
const secretHandle=(await current.send({op:"resolve",node:secret.node!})).handle as string
|
||||
assert.deepEqual(await current.send({op:"call",function:"read-value",handle:secretHandle,args:[]}),{value:"•".repeat(16)})
|
||||
assert.deepEqual(await current.send({op:"call",function:"field",handle:secretHandle,args:[]}),{handle:secretHandle})
|
||||
const button=nodes.find(node=>node.role === "button" && node.name === "First choice")!
|
||||
const handle=(await current.send({op:"resolve",node:button.node!})).handle
|
||||
assert.equal(typeof handle,"string")
|
||||
await navigate(`${base}/read/advanced#stable`)
|
||||
await current.check()
|
||||
assert.equal((await current.send({op:"resolve",node:button.node!})).handle,handle)
|
||||
await request("fixture.read-mutation",{kind:"pushstate"})
|
||||
await current.check()
|
||||
assert.equal((await current.send({op:"resolve",node:button.node!})).handle,handle)
|
||||
await request("fixture.read-mutation",{kind:"reinsert"})
|
||||
assert.equal((await current.send({op:"resolve",node:button.node!})).handle,undefined)
|
||||
assert.equal((await current.send({op:"call",function:"read-value",handle:handle as string,args:[]})).exception,"codenomad:gone")
|
||||
await request("fixture.read-mutation",{kind:"recreate"})
|
||||
await assert.rejects(current.check(),/document changed|no longer current/i)
|
||||
checks.push("refs: stable snapshots/fragments/pushState; removal/reinsertion/recreation refused")
|
||||
}
|
||||
await current.end()
|
||||
checks.push(`${name}: real native semantic tree`)
|
||||
}
|
||||
await navigate(`${base}/read/advanced`)
|
||||
const snapshot=await browser.execute("fixture-session",{action:"snapshot"})
|
||||
const ref=/button "First choice" \[([^\]]+)\]/.exec(snapshot.snapshot as string)![1]
|
||||
assert.match((await browser.execute("fixture-session",{action:"snapshot",ref})).snapshot as string,/Detailed help/)
|
||||
assert.match((await browser.execute("fixture-session",{action:"snapshot",query:"Detailed help"})).snapshot as string,/First choice/)
|
||||
const current=await lease()
|
||||
await navigate(`${base}/read/preview`)
|
||||
await assert.rejects(current.check(),/document changed|no longer current/i)
|
||||
await current.end()
|
||||
await assert.rejects(browser.execute("fixture-session",{action:"snapshot",ref}),/snapshot|page/i)
|
||||
const hidden=await lease()
|
||||
await request("fixture.hide")
|
||||
await assert.rejects(hidden.check(),/current|visible/i)
|
||||
await hidden.end()
|
||||
await request("fixture.show")
|
||||
const closed=await lease()
|
||||
await request("fixture.close")
|
||||
await assert.rejects(closed.check(),/current|visible/i)
|
||||
await closed.end()
|
||||
checks.push("shared renderer scope/query; navigation/hide/close refuse stale leases and refs")
|
||||
writeFileSync(join(output,"read-result.json"),JSON.stringify({passed:true,checks},null,2))
|
||||
console.error(JSON.stringify({passed:true,checks}))
|
||||
}
|
||||
main().then(()=>{parent.close();process.exit(0)},error=>{
|
||||
writeFileSync(join(output,"read-result.json"),JSON.stringify({passed:false,checks,error:String(error)},null,2))
|
||||
console.error(error);parent.close();process.exit(1)
|
||||
})
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
{
|
||||
"preview": "<!doctype html><main><form>\n <label>Title <input value=\"Prefilled\"></label>\n <label>Plan <select><option value=\"custom\">Flexible</option><option value=\"wayfinder\">Exploration</option></select></label>\n <button id=\"confirm\" type=\"button\">Confirm</button><button>Submit</button><output id=\"result\">Pending</output>\n <script>document.getElementById('confirm').addEventListener('click', () => {\n document.getElementById('result').textContent = document.querySelector('input').value + ':' + document.querySelector('select').value\n }); document.querySelector('input').addEventListener('keydown', event => {\n document.getElementById('result').textContent = 'Key:' + event.key\n }); let submitted = 0; document.querySelector('form').addEventListener('submit', event => {\n event.preventDefault(); document.getElementById('result').textContent = 'Submitted:' + (++submitted) + ':' + document.querySelector('input').value\n })</script></form></main>",
|
||||
"forms": "<!doctype html><html lang=\"en\"><body><main>\n <label>Title <input id=\"title\" value=\"Prefilled\"></label>\n <label>Notes <textarea id=\"notes\">Old notes</textarea></label>\n <div id=\"rich\" contenteditable=\"true\" role=\"textbox\" aria-label=\"Rich\">Old rich</div>\n <label>Scenario <select id=\"scenario\"><option value=\"custom\">Flexible mission</option><option value=\"wayfinder\">Exploration</option>\n <option value=\"debug\" disabled>Debugging</option></select></label>\n <label>Due <input id=\"due\" type=\"date\"></label>\n <label>Agree <input id=\"agree\" type=\"checkbox\"></label>\n</main><script>\n window.events = []\n for (const element of document.querySelectorAll(\"input, textarea, select, [contenteditable]\")) for (const type of [\"input\", \"change\"]) {\n element.addEventListener(type, () => window.events.push(type + \":\" + element.id + \":\" + (globalThis.__codenomadAutomationDepth || 0)))\n }\n</script></body></html>",
|
||||
"edges": "<!doctype html><html lang=\"en\"><body><main>\n <label>Outside <input id=\"outside\" value=\"Kept\"></label>\n <details><summary>More</summary><label>Hidden <input id=\"hidden\"></label></details>\n <fieldset disabled><label>Locked <input id=\"locked\"></label></fieldset>\n <label>Mail <input id=\"mail\" type=\"email\" value=\"ab\"></label>\n <label>Count <input id=\"count\" type=\"number\" value=\"12\"></label>\n <label>Tint <input id=\"tint\" type=\"color\"></label>\n <label>Volume <input id=\"volume\" type=\"range\" min=\"0\" max=\"100\" step=\"20\" value=\"40\"></label>\n <label>Day <input id=\"day\" type=\"date\" value=\"2026-01-02\"></label>\n <label>Twins <select id=\"twins\"><option value=\"s1\">Same</option><option value=\"s2\">Same</option></select></label>\n <div id=\"story\" contenteditable=\"true\" role=\"textbox\" aria-label=\"Story\"><div>one</div><div>two</div></div>\n <section aria-label=\"Mission editor\"><p>Mission</p></section>\n <dialog id=\"modal\"><label>Modal field <input id=\"modal-field\"></label></dialog>\n</main><script>\n window.events = []\n for (const element of document.querySelectorAll(\"input\")) for (const type of [\"input\", \"change\"]) element.addEventListener(type, () => window.events.push(type + \":\" + element.id))\n</script></body></html>",
|
||||
"inputs": "<!doctype html><html lang=\"en\"><body style=\"margin:0\"><main>\n <form id=\"search\"><label>Query <input id=\"query\"></label><button>Go</button></form>\n <button id=\"hover-target\">Hover me</button><output id=\"hover-state\">idle</output>\n <button id=\"menu-trigger\" aria-haspopup=\"menu\">Open menu</button><div id=\"menu\" role=\"menu\" hidden><button role=\"menuitem\" id=\"item\">Menu item</button></div>\n <div id=\"scroller\" role=\"region\" aria-label=\"Scroller\" style=\"height:100px;overflow:auto\"><div style=\"height:2000px\">tall</div></div>\n <div style=\"position:relative;height:200px\">\n <div id=\"handle\" role=\"button\" aria-label=\"Handle\" style=\"position:absolute;left:20px;top:20px;width:40px;height:40px;background:#888\"></div>\n <div id=\"zone\" role=\"region\" aria-label=\"Drop zone\" style=\"position:absolute;left:300px;top:20px;width:100px;height:100px;border:1px solid\"></div>\n </div>\n <div style=\"position:relative\"><button id=\"under\">Under</button>\n <div id=\"cover\" style=\"position:absolute;inset:0;background:#fff8\" aria-label=\"Cover\"></div></div>\n <p id=\"later\"></p>\n</main><script>\n const marked = () => Boolean(globalThis.__codenomadAutomationDepth) || (globalThis.__codenomadAutomationInputUntil || 0) >= performance.now()\n window.log = []\n for (const type of [\"keydown\", \"pointerdown\", \"click\"]) document.addEventListener(type, event => {\n window.log.push(type + \":\" + (event.target.id || event.target.tagName) + \":\" + (event.key || \"\") + \":\" + event.isTrusted + \":\" + marked())\n }, true)\n document.getElementById(\"search\").addEventListener(\"submit\", event => { event.preventDefault(); window.log.push(\"submit:\" + query.value) })\n document.getElementById(\"hover-target\").addEventListener(\"pointerenter\", () => { document.getElementById(\"hover-state\").textContent = \"hovered\" })\n // Opens on pointerdown only, like Kobalte menus: a programmatic click never opens it.\n document.getElementById(\"menu-trigger\").addEventListener(\"pointerdown\", () => { document.getElementById(\"menu\").hidden = false })\n document.getElementById(\"item\").addEventListener(\"click\", () => window.log.push(\"menu-item\"))\n let dragging = false\n const handle = document.getElementById(\"handle\"), zone = document.getElementById(\"zone\")\n handle.addEventListener(\"pointerdown\", event => { dragging = true; handle.setPointerCapture(event.pointerId) })\n handle.addEventListener(\"pointermove\", event => { if (dragging) { handle.style.left = (event.clientX - 20) + \"px\"; handle.style.top = (event.clientY - handle.offsetParent.getBoundingClientRect().top - 20) + \"px\" } })\n handle.addEventListener(\"pointerup\", event => {\n dragging = false\n const box = zone.getBoundingClientRect()\n window.dropped = event.clientX >= box.left && event.clientX <= box.right && event.clientY >= box.top && event.clientY <= box.bottom\n })\n setTimeout(() => { document.getElementById(\"later\").textContent = \"Ready now\" }, 600)\n</script></body></html>"
|
||||
}
|
||||
53
packages/tauri-app/tests/webkit-input/README.md
Normal file
53
packages/tauri-app/tests/webkit-input/README.md
Normal file
|
|
@ -0,0 +1,53 @@
|
|||
# Native WebKit input qualification
|
||||
|
||||
The harness compiles the native adapter sources in `src-tauri/src/embedded_view/input_{linux.c,macos.m}` directly. The application's target-gated native build uses the same sources; its private browser controller integrates visible PNG capture through the Rust read adapter. Public WebKit automation stays disabled, and input dispatch retains the qualification boundary below.
|
||||
|
||||
Run on the graphical Linux desktop:
|
||||
|
||||
```sh
|
||||
GDK_BACKEND=x11 python3 packages/tauri-app/scripts/test-webkit-input.py --output /path/to/artifacts/x11
|
||||
GDK_BACKEND=wayland python3 packages/tauri-app/scripts/test-webkit-input.py --output /path/to/artifacts/wayland
|
||||
```
|
||||
|
||||
With an X11 desktop in use, `--nested-kde` starts a separate virtual KWin Wayland compositor, gives its keyboard focus to a sentinel window, and terminates only its own compositor. The WebKit widget uses `GdkWaylandDisplay`. This exercises native Wayland client routing while preserving the desktop session. A full Plasma Wayland login remains a separate qualification target.
|
||||
|
||||
On macOS:
|
||||
|
||||
```sh
|
||||
python3 packages/tauri-app/scripts/test-webkit-input.py --output /path/to/artifacts/macos
|
||||
```
|
||||
|
||||
Requirements: a C compiler, Python 3, GTK 3 and WebKitGTK 4.1 development packages on Linux; Xcode command-line tools and AppKit/WebKit on macOS. `xdotool`, when present on the Linux desktop, observes pointer coordinates only. Synthetic input exclusively uses the adapter. The runner bounds compilation, compositor startup and execution. Each native executable checks background-window status throughout input dispatch.
|
||||
|
||||
Artifacts: `checkpoint.json` contains the fixed isolated-world callback result and page event observations; `capture.png` is the visible native viewport; `native.log` records window and pointer state; `result.json` lists each acceptance check. Linux also records desktop-pointer observations when X11 observation is available. Wayland's GDK coordinates describe its own compositor, so the desktop observer supplies the external pointer check in nested runs.
|
||||
|
||||
The fixture checks left/right click, hover, drag with a held button, wheel, key down/up, the platform select-all chord, Unicode insertion into input/contenteditable, and Enter in contenteditable. Every observed input must be trusted. A page-only global must be absent from the isolated world. The program is fixed in the adapter; callers supply neither script text nor an evaluation expression.
|
||||
|
||||
## Qualification boundary
|
||||
|
||||
The AppKit candidate uses targeted NSEvent/responder APIs. Its UI-thread dispatch scopes the native `pressedMouseButtons` implementation to the synthetic button state, delegates other threads to the original implementation, and restores it in `@finally`. The native self-check exercises nested dispatch, another thread and exception cleanup; the harness also verifies method restoration after every action. Its drag path must produce `pointermove.buttons == 1`; a trusted event alone is insufficient. Select-all must replace the entire input value, and Enter must reach contenteditable. Keep these assertions strict when investigating an alternative native path.
|
||||
|
||||
`CN_INPUT_TRACE=1` records key-equivalent, native editing fallback and hover dispatch paths. Hover first checks for a public `mouseMoved:` implementation on the view and otherwise checks availability of the narrow `_simulateMouseMove:` WebKit SPI. An initial native snapshot establishes the paint callback before input starts. Keyboard delivery targets the exact view; an inactive native editing fallback remains subject to qualification of page cancellation and single execution before integration.
|
||||
|
||||
The macOS harness primes hover at `(5,5)`, then targets `(30,160)` on the next timer tick, with tracking areas installed and mouse-moved events enabled. The trace records both CSS and native window coordinates. Chords send native `flagsChanged:` transitions for each modifier, the key equivalent, its matching key-up, and modifier release. Synthetic class modifier state is UI-thread scoped with exception-safe restoration; the native self-check covers nested modifier state and exception cleanup. Fresh key-event timestamps, window numbers and flags are recorded for sequence comparison.
|
||||
|
||||
`CN_INPUT_NO_CHORD=1` is a diagnostic that omits only the chord down/up steps. The acceptance assertions remain enabled, so this diagnostic fails the chord criteria by design. Compare its contenteditable keyboard events against the complete sequence; use the complete sequence for acceptance.
|
||||
|
||||
`CN_INPUT_DIRECT_HOVER=1` exercises public `mouseEntered:` then `mouseMoved:` directly on the exact WKWebView at the priming and target positions. This variant lives in the native self-test build. The trace includes each tracking area's options and key-window/always-active bits. Keep the window inactive throughout qualification.
|
||||
|
||||
## macOS candidate limitations
|
||||
|
||||
- A standalone Enter uses WebKit's native `keyDown:` text-input path and produces exactly one `insertParagraph` beforeinput/input pair. The harness counts both events independently to catch duplicate execution. Its DOM trace contains Enter key-down; Enter key-up is missing in this diagnostic.
|
||||
- In the complete Cmd+A sequence, the native select-all fallback replaces the input value. DOM delivery includes Meta key-down, while the Meta release and following Enter keyboard events are missing. The native trace records direct WKWebView `flagsChanged:` down/up, chord key-up and zero modifiers on the following Enter. After this chord, the agent's subsequent Enter also lacks its native paragraph edit. Keep keyboard actions following a Mod chord gated by this limitation.
|
||||
- The initial hover lacks `pointermove` at the requested target while the window is inactive, including after priming at a different in-view location. Public direct enter/move and the checked `_simulateMouseMove:` SPI are covered by separate harness variants. Mouse-enter/CSS hover observations occur with subsequent clicks. The agent's initial standalone hover consequently cannot establish the expected pointer-move behavior. The harness preserves background-window operation; tracking-area option logging is diagnostic evidence rather than a key-window positive control.
|
||||
|
||||
The strict complete-sequence assertions retain these failures. The private transport integrates navigation and capture; pointer and keyboard dispatch retain the documented macOS limitations and their separate capability gate.
|
||||
|
||||
WebKit's `WebEventFactory::createWebMouseEvent` obtains ordinary NSEvent button state from `WebCore::currentlyPressedMouseButtons`, which reads `[NSEvent pressedMouseButtons]`. The automation event path has distinct handling. Source references for investigation:
|
||||
|
||||
- <https://github.com/WebKit/WebKit/blob/main/Source/WebKit/Shared/mac/WebEventFactory.mm>
|
||||
- <https://github.com/WebKit/WebKit/blob/main/Source/WebCore/platform/mac/PlatformEventFactoryMac.mm>
|
||||
- <https://github.com/WebKit/WebKit/blob/main/Source/WebKit/UIProcess/Automation/mac/WebAutomationSessionMac.mm>
|
||||
- <https://github.com/WebKit/WebKit/blob/main/Source/WebKit/UIProcess/API/Cocoa/_WKAutomationSession.h>
|
||||
|
||||
An alternative based on WebKit automation SPI requires explicit contract qualification and runtime capability checks. Its background-window behavior must pass this harness before application integration. Never use global CGEvent posting, unscoped event-state overrides, XTest, DOM-event substitutes, or broad script evaluation to satisfy a failed assertion.
|
||||
18
packages/tauri-app/tests/webkit-input/fixture.html
Normal file
18
packages/tauri-app/tests/webkit-input/fixture.html
Normal file
|
|
@ -0,0 +1,18 @@
|
|||
<!doctype html><meta charset="utf-8"><title>Native input qualification</title>
|
||||
<style>body{margin:0}input, [contenteditable]{position:absolute;left:10px;width:280px;height:40px}input{top:10px}[contenteditable]{top:70px;background:#eee}#pad{position:absolute;top:140px;left:10px;width:400px;height:180px;background:#bdf}output{position:absolute;top:340px;white-space:pre-wrap}</style>
|
||||
<input aria-label="input"><div contenteditable="true" aria-label="editor"></div><div id="pad">pointer target</div><output></output>
|
||||
<style>#pad:hover{background:rgb(10,200,30)}</style>
|
||||
<script>
|
||||
window.__codenomadPageOnly = true;
|
||||
const observations = [];
|
||||
for (const type of ['pointerdown','pointerup','pointermove','mousemove','mouseover','mouseenter','click','contextmenu','wheel','keydown','keyup','beforeinput','input']) {
|
||||
document.addEventListener(type, event => {
|
||||
observations.push({type, trusted:event.isTrusted, button:event.button, buttons:event.buttons,
|
||||
x:event.clientX, y:event.clientY, key:event.key, ctrl:event.ctrlKey, meta:event.metaKey, data:event.data, inputType:event.inputType,
|
||||
padHover:document.querySelector('#pad').matches(':hover'), target:event.target.tagName});
|
||||
document.querySelector('output').textContent = JSON.stringify({observations,
|
||||
input:document.querySelector('input').value, editor:document.querySelector('[contenteditable]').textContent, editorHtml:document.querySelector('[contenteditable]').innerHTML});
|
||||
if(type === 'contextmenu' || type === 'wheel') event.preventDefault();
|
||||
}, {passive:false,capture:true});
|
||||
}
|
||||
</script>
|
||||
61
packages/tauri-app/tests/webkit-input/linux.c
Normal file
61
packages/tauri-app/tests/webkit-input/linux.c
Normal file
|
|
@ -0,0 +1,61 @@
|
|||
#include <gtk/gtk.h>
|
||||
#include <webkit2/webkit2.h>
|
||||
#include "sequence.h"
|
||||
|
||||
static GtkWidget *window, *view;
|
||||
static int step = -4;
|
||||
static gint initial_x, initial_y;
|
||||
static GdkDevice *pointer;
|
||||
static gboolean tick(gpointer unused) {
|
||||
(void)unused;
|
||||
if (step >= 0) assert(!gtk_window_is_active(GTK_WINDOW(window)));
|
||||
if (step >= 0 && step <= 19) sequence(view, step);
|
||||
if (step == 21) cn_checkpoint(view, write_result, "checkpoint.json");
|
||||
if (step == 22) cn_capture(view, write_result, "capture.png");
|
||||
if (step == 26) {
|
||||
gint x, y;
|
||||
gdk_device_get_position(pointer, NULL, &x, &y);
|
||||
printf("window_active=%d pointer_before=%d,%d pointer_after=%d,%d backend=%s\n",
|
||||
gtk_window_is_active(GTK_WINDOW(window)), initial_x, initial_y, x, y,
|
||||
G_OBJECT_TYPE_NAME(gtk_widget_get_display(window)));
|
||||
assert(!gtk_window_is_active(GTK_WINDOW(window)));
|
||||
assert(x == initial_x && y == initial_y);
|
||||
gtk_main_quit();
|
||||
return G_SOURCE_REMOVE;
|
||||
}
|
||||
step++;
|
||||
return G_SOURCE_CONTINUE;
|
||||
}
|
||||
static void loaded(WebKitWebView *webview, WebKitLoadEvent event, gpointer unused) {
|
||||
(void)webview; (void)unused;
|
||||
if (event == WEBKIT_LOAD_FINISHED) g_timeout_add(250, tick, NULL);
|
||||
}
|
||||
int main(int argc, char **argv) {
|
||||
assert(argc == 2);
|
||||
gtk_init(&argc, &argv);
|
||||
window = gtk_window_new(GTK_WINDOW_TOPLEVEL);
|
||||
gtk_window_set_title(GTK_WINDOW(window), "CodeNomad input qualification (no activation)");
|
||||
gtk_window_set_focus_on_map(GTK_WINDOW(window), FALSE);
|
||||
gtk_window_set_default_size(GTK_WINDOW(window), 700, 600);
|
||||
view = webkit_web_view_new();
|
||||
gtk_container_add(GTK_CONTAINER(window), view);
|
||||
g_signal_connect(view, "load-changed", G_CALLBACK(loaded), NULL);
|
||||
gtk_widget_show_all(window);
|
||||
if (g_getenv("CN_NESTED_KDE")) {
|
||||
/* A new private compositor initially focuses its first surface. Give
|
||||
* keyboard focus to a sentinel before sending background guest input. */
|
||||
GtkWidget *sentinel = gtk_window_new(GTK_WINDOW_TOPLEVEL);
|
||||
gtk_window_set_title(GTK_WINDOW(sentinel), "Qualification focus sentinel");
|
||||
gtk_window_set_default_size(GTK_WINDOW(sentinel), 200, 100);
|
||||
gtk_widget_show_all(sentinel);
|
||||
}
|
||||
pointer = gdk_seat_get_pointer(gdk_display_get_default_seat(gtk_widget_get_display(window)));
|
||||
gdk_device_get_position(pointer, NULL, &initial_x, &initial_y);
|
||||
char *html = NULL;
|
||||
assert(g_file_get_contents(argv[1], &html, NULL, NULL));
|
||||
webkit_web_view_load_html(WEBKIT_WEB_VIEW(view), html, "http://127.0.0.1/");
|
||||
g_free(html);
|
||||
gtk_main();
|
||||
gtk_widget_destroy(window);
|
||||
return 0;
|
||||
}
|
||||
94
packages/tauri-app/tests/webkit-input/macos.m
Normal file
94
packages/tauri-app/tests/webkit-input/macos.m
Normal file
|
|
@ -0,0 +1,94 @@
|
|||
#import <AppKit/AppKit.h>
|
||||
#import <WebKit/WebKit.h>
|
||||
#include "sequence.h"
|
||||
#include <objc/runtime.h>
|
||||
extern void cn_test_scoped_buttons(void);
|
||||
extern int cn_test_direct_hover(void *, double, double);
|
||||
@interface WKWebView (CNInputDrain)
|
||||
- (void)_doAfterProcessingAllPendingKeyEvents:(void (^)(void))completion;
|
||||
- (void)_doAfterProcessingAllPendingMouseEvents:(void (^)(void))completion;
|
||||
@end
|
||||
|
||||
@interface Runner : NSObject <WKNavigationDelegate>
|
||||
@property NSWindow *window;
|
||||
@property WKWebView *view;
|
||||
@property NSInteger step;
|
||||
@property NSPoint pointer;
|
||||
@end
|
||||
static void painted(void *context, const unsigned char *bytes, size_t length, const char *error) {
|
||||
(void)bytes; (void)length;
|
||||
assert(!error);
|
||||
Runner *runner = (__bridge Runner *)context;
|
||||
[NSTimer scheduledTimerWithTimeInterval:0.25 target:runner selector:@selector(tick:) userInfo:nil repeats:YES];
|
||||
}
|
||||
@implementation Runner
|
||||
- (void)webView:(WKWebView *)view didFinishNavigation:(WKNavigation *)navigation {
|
||||
(void)view; (void)navigation;
|
||||
self.pointer = NSEvent.mouseLocation;
|
||||
self.step = -4;
|
||||
cn_capture((__bridge void *)self.view, painted, (__bridge void *)self);
|
||||
}
|
||||
- (void)tick:(NSTimer *)timer {
|
||||
void *view = (__bridge void *)self.view;
|
||||
if (self.step >= 0) assert(!self.window.isKeyWindow && !NSApp.active);
|
||||
if (self.step == -1) {
|
||||
self.window.acceptsMouseMovedEvents = YES;
|
||||
[self.view updateTrackingAreas];
|
||||
fprintf(stderr, "hover variant=prime-5,5-then-target trackingAreas=%lu acceptsMouseMoved=%d\n", (unsigned long)self.view.trackingAreas.count, self.window.acceptsMouseMovedEvents);
|
||||
for (NSTrackingArea *area in self.view.trackingAreas) {
|
||||
fprintf(stderr, "tracking owner=%s options=0x%lx activeInKeyWindow=%d activeAlways=%d\n", NSStringFromClass([area.owner class]).UTF8String, (unsigned long)area.options, !!(area.options & NSTrackingActiveInKeyWindow), !!(area.options & NSTrackingActiveAlways));
|
||||
}
|
||||
if (getenv("CN_INPUT_DIRECT_HOVER")) assert(cn_test_direct_hover(view, 5, 5));
|
||||
else assert(cn_pointer(view, CN_MOVE, 5, 5, 0));
|
||||
}
|
||||
IMP buttons = class_getMethodImplementation(object_getClass(NSEvent.class), @selector(pressedMouseButtons));
|
||||
IMP modifiers = class_getMethodImplementation(object_getClass(NSEvent.class), @selector(modifierFlags));
|
||||
BOOL omit_chord = getenv("CN_INPUT_NO_CHORD") && (self.step == 12 || self.step == 13);
|
||||
BOOL direct_hover = getenv("CN_INPUT_DIRECT_HOVER") && self.step == 0;
|
||||
if (direct_hover) assert(cn_test_direct_hover(view, 30, 160));
|
||||
if (self.step >= 0 && self.step <= 19 && !omit_chord && !direct_hover) sequence(view, (int)self.step);
|
||||
assert(class_getMethodImplementation(object_getClass(NSEvent.class), @selector(pressedMouseButtons)) == buttons);
|
||||
assert(class_getMethodImplementation(object_getClass(NSEvent.class), @selector(modifierFlags)) == modifiers);
|
||||
if (self.step >= 0 && self.step <= 19 && getenv("CN_INPUT_TRACE")) {
|
||||
NSInteger current = self.step;
|
||||
if ([self.view respondsToSelector:@selector(_doAfterProcessingAllPendingKeyEvents:)])
|
||||
[self.view _doAfterProcessingAllPendingKeyEvents:^{ fprintf(stderr, "key-drained step=%ld\n", (long)current); }];
|
||||
if ([self.view respondsToSelector:@selector(_doAfterProcessingAllPendingMouseEvents:)])
|
||||
[self.view _doAfterProcessingAllPendingMouseEvents:^{ fprintf(stderr, "mouse-drained step=%ld\n", (long)current); }];
|
||||
}
|
||||
if (self.step == 21) cn_checkpoint(view, write_result, "checkpoint.json");
|
||||
if (self.step == 22) cn_capture(view, write_result, "capture.png");
|
||||
if (self.step == 26) {
|
||||
NSPoint pointer = NSEvent.mouseLocation;
|
||||
printf("window_key=%d app_active=%d pointer_before=%g,%g pointer_after=%g,%g\n",
|
||||
self.window.isKeyWindow, NSApp.active, self.pointer.x, self.pointer.y, pointer.x, pointer.y);
|
||||
assert(!self.window.isKeyWindow && !NSApp.active);
|
||||
assert(NSEqualPoints(self.pointer, pointer));
|
||||
[timer invalidate];
|
||||
fflush(stdout);
|
||||
exit(0);
|
||||
}
|
||||
self.step++;
|
||||
}
|
||||
@end
|
||||
int main(int argc, char **argv) {
|
||||
assert(argc == 2);
|
||||
@autoreleasepool {
|
||||
[NSApplication sharedApplication];
|
||||
[NSApp setActivationPolicy:NSApplicationActivationPolicyAccessory];
|
||||
cn_test_scoped_buttons();
|
||||
Runner *runner = [Runner new];
|
||||
runner.window = [[NSWindow alloc] initWithContentRect:NSMakeRect(30, 30, 700, 600)
|
||||
styleMask:NSWindowStyleMaskTitled backing:NSBackingStoreBuffered defer:NO];
|
||||
runner.window.title = @"CodeNomad input qualification (no activation)";
|
||||
runner.view = [[WKWebView alloc] initWithFrame:NSMakeRect(0, 0, 700, 600)];
|
||||
runner.window.contentView = runner.view;
|
||||
runner.view.navigationDelegate = runner;
|
||||
[runner.window orderFront:nil];
|
||||
NSString *html = [NSString stringWithContentsOfFile:[NSString stringWithUTF8String:argv[1]]
|
||||
encoding:NSUTF8StringEncoding error:nil];
|
||||
assert(html);
|
||||
[runner.view loadHTMLString:html baseURL:[NSURL URLWithString:@"http://127.0.0.1/"]];
|
||||
[NSApp run];
|
||||
}
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue