Finish native pointer delivery and validation
iamnbutler opened ·
**Status (Oct 6):** clicks, scrolling and atomic drag are merged (#82, #70, #85), and #156 now gives pixels-only snapshots exact-window coordinate authority for point clicks, scrolls and drags. Remaining evidence gaps: one full model workflow using #156 pixel snapshots that includes an effective drag (an earlier real pi/Anthropic run with Accessibility snapshots completed click, scroll and drop; see below), worker crash and in-flight Stop during input, self-Ace scrolling, multiple displays/negative origins/hotplug, and a second machine. Moved-window stale-click classification is #155.
Continue #8's complete-input slice with native pointer operations through Ace's embedded bridge and existing pi tools.
- [x] Extend observed-control clicks with useful click variants and exact-window screenshot-coordinate clicks. - [x] Add targeted scrolling to an observed control or screenshot point, with explicit direction and bounded native scroll amount. - [x] Add an atomic drag with one complete press/move/release sequence, bounded duration, and cleanup on cancellation. Do not expose unowned cross-call held input. - Resolve screenshot points through Peekaboo's stored capture coordinate authority. Account for Ace image resizing, Retina scale, multiple/negative-origin displays, and moved windows. Reject stale target identities/bounds rather than silently remapping against a new window. - Reuse exact process/window receipts, single-use snapshots, native coordination, durable completed/refused/unknown outcomes, and the channel owner's desktop-tools setting (see #8). Make supported background versus foreground operations explicit; no silent global-input fallback.
Done when real native and pi/model workflows can click by visual coordinates, scroll to initially offscreen content, and complete a real drag-and-drop, with resulting state verified and interruption cleanup/no-replay validated. Include the hosted workspace path and clearly track unverified multi-display or second-machine behavior in #8.
Click variants and normalized screenshot-coordinate clicks landed in [PR #82](https://github.com/githubnext/ace2/pull/82), merge `e5618c1`. Real native callbacks covered single/double/right/middle/triple behavior and resized screenshot coordinates; consumed snapshots were refused. The real pi/Anthropic check on exact PR82 head `5f98d94` performed one coordinate click with exactly one intended callback and unchanged unrelated state. Exact tool text/images and usage survived reopening the same pi store without repeated input.
Targeted scrolling landed in [PR #70](https://github.com/githubnext/ace2/pull/70), merge `44312423ce66443ab38cd1ba6666bec32904e068`, from reviewed head `f1682d028a9f810c93c103286949c1ad69c8ef13`. It accepts an observed element or normalized screenshot point, four directions, and 1–20 native units. Final types, formatting/lint, Swift parsing, diff checks, and independent source review passed. The extracted scroll implementation is unchanged from signed candidate `d9b0c869`; the extraction/base integration was not rebuilt into another signed app.
On that signed candidate, five real AppKit scrolls passed: element down, point down revealing previously offscreen content, element up, point right, and element left. A subsequent real pi/Anthropic run performed inspect → element up1 → inspect, moving the actual vertical origin from 2063 to 1834 while horizontal origin 68 and unrelated pointer state stayed unchanged. Exact native receipts and tool text/images survived completed-store reopen with unchanged usage and no repeated input. This does not claim worker-crash or hosted behavior.
The earlier delivered-but-unknown scroll was traced to omitted snapshot focus in the request, despite that focus being retained in the signed native receipt. The landed fix carries the original focus into scroll's expected-window target without relaxing receipt validation or coordinate authority. No uncertain input was replayed.
Atomic dragging landed in [PR #85](https://github.com/githubnext/ace2/pull/85), merge `68b4a3f641858526d80de352de84260a7f3e8f5a`, from reviewed head `bf930584700306ec379036c320dd1c61be4f6db9`. The native code is unchanged from signed candidate `2531311`; final changes were tool/documentation guidance, with fresh type/lint checks. Native signatures, compilation, and independent source review passed.
Real focused left/right drags each produced one press, twenty moves and one release, moved the tile and completed the drop. Invalid endpoints/stationary gestures/durations refused without input. A new, verified window move made its fresh prior observation stale, and a drag using it refused with the receiver unchanged. Cancelling a five-second gesture only after actual mouse-down returned unknown, delivered exactly one release, and drained every native call without changing scroll position.
A real pi/Anthropic run in local Workers/Durable Objects then performed inspect → Activate click → one scroll → drag → inspect through the actual workspace adapter and signed native bridge. The fixture recorded one click, scroll from 0 to 229, and one completed tile drop. Native text and images crossed the workspace unchanged. Restarting the same persistence with the workspace offline replayed all five exact results/images with unchanged receiver state and provider usage. This is completed-run replay, not a crash during input.
The earlier inactive receiver accepted dispatch but recorded no input or movement. Separate explicit activation/focus before the successful cases establishes the supported workflow; delivery acceptance alone never proves a drop. One Retina display (native scale 2) and Ace's resized 1600×1053 image were exercised.
Remaining coverage stays open here and in #8: worker-crash/in-flight Stop cases, self-Ace scrolling, multiple displays/negative origins/hotplug, and another physical machine. Pixels-only observations could not authorize pointer actions without canonical capture authority; #156 resolved that. No broader hardware or inactive-view delivery claim is made.
iamnbutlerreferenced this in [Meta] Complete native computer use in Ace #8last month on GitHub
iamnbutlerreferenced this in Add native targeted scrolling #70last month on GitHub
- IA
Draft implementation: #70, stacked on the keyboard work in #67. Current pointer head: `ce87272ab69e3c8f2b7a8f7b0cef9f7a9a2f3f94`.
Implemented screenshot-normalized click variants, element/point scrolling, and atomic left/right drags through the existing native receipt, snapshot lease, unsafe replay, outcome, and cancellation paths. Coordinates use Peekaboo's retained capture authority; scroll amounts are bounded native units rather than pixels.
Passed: type checks, formatting/lint, signed desktop build, strict deep signature verification, independent source review, and review through Ace's own changes/patch APIs. A signed disposable AppKit fixture and temporary smoke script are prepared and compile.
**Not yet run for this pointer slice:** native fixture execution, real pi/model and history replay, hosted workspace checks, interruption cleanup after delivered mouse-down, and self-Ace scrolling. Multiple/negative-origin displays, Retina behavior, and a second physical machine remain unverified. The PR stays a draft and this issue remains open until the required runtime evidence exists.
Pure pixels-only captures also remain a tracked limit: the Bridge currently exposes retained Accessibility detection results but no canonical receipt retrieval for screenshot-only snapshots. This implementation requires the former and does not invent Accessibility metadata or silently switch to global input. Observation availability in #64 remains relevant to the upcoming validation.
iamnbutlerreferenced this in Add targeted click variants and screenshot points #82last month on GitHub
- IA
Merged #82 on `feat/github-cache` as `e5618c1`: click variants and normalized screenshot-coordinate clicks. Six native click groups passed; the exact-PR82 pi/Anthropic run then made one coordinate click, verified the intended callback incremented once, and reopened identical tool results/images with unchanged usage and no repeated input. Evidence: `/tmp/ace-pointer-fixture.TNcgBS/click-model-KfkLfz/result.json`.
#70 continues scrolling and atomic dragging. #68 remains open for that scope and its remaining interruption/hosted/coordinate validation. No new canary has been published.
iamnbutlerreferenced this in Add atomic native drags #85last month on GitHub
iamnbutlerreferenced this in Land desktop inspection/actions and tab API from feat/github-cache #89last month on GitHub
iamnbutlerreferenced this in Focus and paste into WebKit composers with native desktop tools #106last month on GitHub
iamnbutlerreferenced this in feat(app): open GitHub items on GitHub or in a new channel #109last month on GitHub
iamnbutlerreferenced this in Report moved-window coordinate clicks rejected before dispatch as refused #155last month on GitHub
iamnbutlerreferenced this in feat(desktop): authorize coordinate actions from pixel snapshots #156last month on GitHub
iamnbutlerreferenced this in [Meta] Dogfood Ace end to end #5last month on GitHub
- IA
#156 resolves this issue's tracked limit "Pixels-only observations still cannot authorize pointer actions without canonical capture authority": pixel inspection now publishes a single-use snapshot whose stored canonical capture grants exact-window coordinate authority for clicks, scrolls and drags. Native checks on the signed `769438a` build passed a point click, a scroll (0→687.5) and a drag (one down, one up, dropped) from pixel snapshots. A moved-window stale point click returned `unknown` / `may_have_dispatched` rather than `refused`, with no input received; the same happens with Accessibility snapshots. That pre-existing classification is tracked in #155. Real pi/Anthropic runs and distinct-worker reopen results, with their limits, are in #156 and #8.
iamnbutlerparent-issue-addedlast month on GitHub
iamnbutlerrenamed Add native pointer clicks, scrolling, and atomic drag to Finish native pointer delivery and validationlast month on GitHub
iamnbutlerreferenced this in fix(desktop): report clicks on an already moved window as refused #187last month on GitHub
iamnbutlerreferenced this in fix(ui): add a dedicated top-edge window drag strip #190last month on GitHub
- IA
During #190 validation, one Ace-dev top-edge drag was accepted (window_targeted_events, dispatched_unverified) with no observed geometry change: window 10684 stayed at [160,120,1100,760]. desktop_focus had refused with operation_unsupported / timeout (#106). No replay or global-input fallback; app quit normally. Actual drag behavior remains manually unverified.
iamnbutlerreferenced this in Make the desktop window reliably draggable across its top edge #189last month on GitHub