feat(desktop): authorize coordinate actions from pixel snapshots #156
- IA
`desktop_inspect` with `mode: "pixels"` now returns a single-use snapshot that authorizes the existing coordinate actions: screenshot-point clicks, point scrolls and drags, in the exact observed window. Part of #8, slice 5; pointer authority from #68.
- The client allocates a producer-bound ID with the existing `createSnapshot()` and pins it as `output.snapshotID`, so the signed observation binding's check stays unchanged. On failure it makes a best-effort `cleanSnapshot`. - The ID is exposed only after the signed canonical `getDetectionResult` projection checks out: zero elements, no focused control, an exact-window receipt equal to the observation's (pid, generation, window, bounds), and the canonical capture context with coordinate authority. - The GUI allowlists the existing `createSnapshot` / `cleanSnapshot`. There is no new bridge operation, store, or vendor patch. - Element, value, selection, key and insertion requests on pixel snapshots refuse naturally, with no invented focus. The post-action refresh stays in accessibility mode, so click-to-edit is unchanged. Receipts, single-use leases, refused/unknown outcomes and unsafe replay are unchanged.
## Validation
Detailed evidence, original failures and paths: [PR comment](https://github.com/githubnext/ace2/pull/156#issuecomment-6025492650).
- `bun types`, `bun run ci` and GitHub CI pass. - The signed Ace-dev build of `769438a` passes strict deep signature verification, in an isolated fresh profile. - **Native**, on a disposable AppKit fixture: - Six refusals before dispatch, plus a refused consumed snapshot. - A point click, a point scroll and a drag each took effect. - **Moved window**: a stale point click returned signed `unknown` with no input received. An Accessibility-snapshot control on a separate fixture gave the same result. This is pre-existing click classification, tracked in #155. - **Two real pi/Anthropic runs** used fresh pixel snapshots for every mutation, and clicks and scrolls took effect in both. Run 1's drag was accepted but had no effect on the inactive view. Run 2's prerequisite `desktop_focus` refused with the #106 timeout. Neither run is a full three-effect workflow; a successful drop was verified only by the direct native checks. - **Reopen**: a distinct worker replayed identical durable results for both runs, with unchanged usage and no input. - **Cleanup**: all owned processes, ports, socket, channels, projects, profile data and fixtures were cleaned and independently verified.
## Limits
- Native single left point clicks still use the existing Accessibility hit-test/press route. An AX-capable fixture proves screenshot-only authority, not the absence of native AX requirements. General canvas click delivery remains open. - GUI artifact copies outlive a GUI quit, as they already do for Accessibility snapshots. - No multi-display, hosted or second-machine validation.
### Built in Ace
The implementation, signed build, validation, issues and this PR were done by the pi agent in an Ace channel lane, using Anthropic after the channel's default OpenAI invocation failed with HTTP 401 before any work. Codex coordinated the requested work and performed read-only review outside Ace, because external harness integration is #9. The lane's changes view showed stale base `6e925f3` with unrelated merged work. That came from this channel's source worker (PID 20884), launched from the stale `/Users/natebutler/code/ace2` checkout with the Canary data profile: the old runtime form of #96, not the installed Ace binary. Review therefore used `git diff 6dc0f1c..769438a` directly.
iamnbutlerreferenced this in [Meta] Dogfood Ace end to end #5last month on GitHub
iamnbutlerreferenced this in Focus and paste into WebKit composers with native desktop tools #106last month on GitHub
iamnbutlerreferenced this in [Meta] Complete native computer use in Ace #8last month on GitHub
iamnbutlerreferenced this in Finish native pointer delivery and validation #68last month on GitHub
- IA
Full evidence record for #156 at `769438a`, preserved before the PR body was shortened. Original failures and private evidence paths are kept here.
Pixel inspection now publishes a single-use snapshot that authorizes existing **coordinate** actions: screenshot-point clicks, point scrolls, and drags in the exact observed window. Part of #8, slice 5; pointer authority from #68.
## Change
- **Client (`client.swift`)**: `pixels` mode allocates a producer-bound ID with the existing `createSnapshot()`. It passes that ID as `output.snapshotID` with `saveSnapshot: true`, so the signed observation binding's `request.snapshotID` check applies unchanged. Screenshot-only results have no detection ID, so `saveSnapshot` alone would fail that binding. On failure the client makes a best-effort `cleanSnapshot` of the allocated ID. The ID is exposed only after the bridge's signed `getDetectionResult` projection of the stored canonical screenshot passes all of these checks: - zero elements and no focused control - exact-window identity equal to the observation receipt (pid, process generation, window ID, captured bounds) - canonical `CaptureCoordinateContext` equal to the capture metadata - the same coordinate-authority checks pointer actions use - **Actions (`actions.swift`)**: the existing exact-window and coordinate-authority checks move into two shared helpers with no behavior change. Pixel snapshots naturally refuse element, value, selection, key and insertion requests: there is no element, or "no exact focused control". No focus is invented. - **GUI (`desktop.swift`)**: allowlists the existing `createSnapshot` / `cleanSnapshot` lifecycle operations. There is no new bridge operation, snapshot store, or vendor patch. - The automatic observation after a completed action stays in accessibility mode, so the working click-to-edit path is unchanged. If it fails, the result keeps the completed action and the agent can inspect with `pixels`. - Tool description, host failure guidance, `desktop-tools.md` and `architecture.md` describe point-only authority. Peekaboo's existing `copyArtifactsOnStore` keeps a GUI-owned screenshot copy after the host deletes its temporary capture.
Receipt authenticity, exact generation/window/bounds/capture authority, single-use leases, refused/unknown outcomes and unsafe replay are unchanged.
## Validation
Signed Ace-dev build of exact head `769438a` from a clean lane. Bundle passes `codesign --verify --deep --strict`; client identifier `dev.ace.desktop.dev.cda78631.desktop-client`, team `8S43JJMN6B`. Isolated fresh profile `cda78631`, opened through the dev LaunchServices launcher. The fixture was a disposable signed AppKit pointer app. Independent audit matched the binary hashes, profile, and kernel executable paths (dev-launcher, launcher, GUI, fixture).
- `bun types` and `bun run ci` pass. - **Native, through the real host adapter (`native-2`)**: - Six requests on one pixel snapshot were refused before dispatch, each with the exact receipt and zero fixture events. Element click, type, select and element scroll refused with "not in this snapshot"; key and insert refused with "no exact focused control". The clipboard change count was unchanged. - Point click on Activate: `completed`, count 0→1. - Reusing the consumed snapshot: refused as stale. - Point scroll: origin 0→687.5. - Drag: one down, 20 drags, one up; dropped. - Each completed action returned a fresh Accessibility observation with image. - Each pixel inspection created a GUI-owned artifact copy that survived the host's temp removal. - **Moved window**: a stale pixel-snapshot point click after a verified `desktop_move` returned signed **`unknown`** / `may_have_dispatched`, not `refused`. The original verifier failure is preserved. A fresh observation found the same window and generation at the moved bounds, with no input received. An Accessibility-snapshot control on a separate fresh fixture gave the identical result. Source review traced this to the existing ClickService/bridge classification of an untyped pre-dispatch stale error, independent of snapshot type. Tracked in #155; not changed here. - **Real pi/Anthropic (`claude-opus-5-5`)**, two independent runs on fresh fixtures. Every mutation used a fresh pixels-mode snapshot: - **Run 1** (341,628 tokens): click (count 1) and scroll (origin 458.5) took effect. The drag returned `completed` with `dispatched_unverified` / `delivery_accepted` (23 units), but the inactive fixture recorded **no** events: accepted delivery, no drop, as documented for inactive views. My harness then stopped the run on the model's ordinary `channel` summary call, so there is no final reply. The run is recorded as failed. - **Run 2** (204,385 tokens): click and scroll took effect. The documented `desktop_focus` step before dragging returned a signed pre-dispatch refusal, "Timeout while waiting for condition", the known focus failure in #106. The model stopped without dragging or retrying. Recorded as failed. - Neither run is a full three-effect model workflow. A successful drop from a pixel snapshot was verified only by the direct native checks above. - **Distinct-worker reopen** for both runs: the original workers (47095, 48234) exited, and new workers (47751, 48696) replayed identical durable message/tool/result entries with images (8/8/7 and 6/6/4). Usage was unchanged, no run started, and fixture state was unchanged. Independent audit confirmed the reopened and cleanup workers (48871, 48881) executed no models or tools.
## Limits
- Pixel snapshots remove Ace's Accessibility *observation* requirement only. Single left point clicks still use the existing native route, including the Accessibility hit test and positional press. On an AX-capable AppKit fixture this proves screenshot-only authority, not the absence of native AX requirements. General canvas single-click delivery remains a separate native transport gap. - Allocated pixel snapshots rely on existing bounded in-memory pruning if the client dies mid-inspection. GUI artifact copies, for pixel and Accessibility snapshots alike, outlive a GUI quit in the OS temp dir. This validation removed the five it could attribute. - No multi-display, hosted-workspace or second-machine validation.
## Cleanup
Verified independently: the coordinator, host, launcher, application/GUI, all six workers and four fixtures exited; ports 47415/48415 are closed and the socket is absent. Both model channels and projects were deleted. The fresh profile home and settings were removed after copying logs into private evidence. Nothing was stored in the profile keychain service; the key was passed only through the process environment. Fixture bundles were removed. The installed apps, permissions and clipboard contents were not touched. Private evidence: `/tmp/ace-pixel-authority/run.8JpTz3`.
### Built in Ace
The implementation, signed build, native/model validation, issues and this PR were done by the pi agent in an Ace channel, in its lane, using Anthropic after the channel's default OpenAI invocation failed with HTTP 401 before doing any work. Codex coordinated the requested work and performed read-only source and evidence review outside Ace, because external harness integration is still #9. Ace's changes view for this lane reported the stale base `6e925f3` and included unrelated merged work. That came from this channel's source worker (PID 20884), launched from the stale `/Users/natebutler/code/ace2` checkout with the Canary data profile, so it is the old runtime form of #96 and not the installed Ace binary. Codex therefore reviewed `git diff 6dc0f1c..769438a` directly. Shell use: building, `open` to launch fixtures, `kill` for owned fixture processes during cleanup, and a `swift` one-liner reading only the pasteboard change count, with no clipboard contents.
iamnbutlermerged 783aec34c13dlast month on GitHub
iamnbutlerreferenced this in Inspect Chrome windows with Accessibility when the page title changes #113last month on GitHub