Add atomic native drags #85
- IA
Adds `desktop_drag`: one bounded, straight-line press/move/release between distinct normalized screenshot points in the observed window, with left or right button. It reuses capture coordinates, single-use snapshots, native gesture ownership/release cleanup, and pi's unsafe-tool outcomes. The request preserves the snapshot's original focus evidence so it matches the complete capture-owned receipt; receipt validation remains strict.
Part of #68 and #8. No button remains held across calls, and gestures cannot cross windows.
Validation:
- Types, formatting/lint, Swift parsing, signed native build/signatures, and independent source review passed. Native candidate `2531311999293133809e845a47522ebcb090a4d2` is unchanged by final source `bf930584700306ec379036c320dd1c61be4f6db9`, which adds only reviewed tool/documentation guidance and passed types/lint again. - Real focused AppKit left/right drags moved and dropped the tile with exactly one press and release each. Invalid endpoints, stationary gestures, and invalid duration refused without input. One Retina display (`native_scale: 2`) and native 960×632 capture coordinates with Ace's resized 1600×1053 image were exercised. - The remaining direct check passed at `bf930584`: a freshly moved window refused its prior observation with a specific stale/bounds diagnostic and no receiver change. Cancelling only after an actual mouse-down returned unknown and delivered exactly one release; all native calls drained and the prior scroll positions remained unchanged. Evidence: `drag-current-FkQMdR` in the immutable `ace-pointer-remaining-fixture.5mF516` directory. - A real pi/Anthropic run in the local Workers/Durable Objects runtime passed inspect → Activate click → one scroll → drag → inspect through the actual workspace adapter and signed native bridge. It made exactly five tool calls and three fixture mutations after separate explicit app activation/window focus setup. Receiver callbacks verified one click, scroll from 0 to 229, and one completed tile drop. Native outcomes and images crossed the workspace unchanged. Restarting the same persisted channel with the workspace offline replayed all five exact results/images with unchanged receiver state and provider usage. Evidence: `hosted-model-YHsStf` under the fresh `ace-pointer-hosted-fixture.CPa0AR` fixture.
Accepted delivery does not prove a drop: an inactive custom view ignored an earlier delivered gesture. After explicit focus and fresh inspection, separate left/right cases worked. The tool tells the model to verify the effect and explicitly focus then inspect again before deciding on another action; it adds no automatic foreground fallback or blind replay. A Move window setup changed its bounds and truthfully returned unknown after final identity validation, so its successful effect was preserved without replaying it.
Multi-display behavior, another physical machine, and crash-during-gesture recovery remain unverified. Pixels-only observations do not authorize pointer input.
### Built in Ace
Git/GitHub, source-diff review, static checks, signed builds, and signature verification ran in dedicated Ace terminals. Native checks use Ace's signed bridge and actual receiver callbacks; the hosted check uses the real channel/workspace route and provider.
Codex external file tools edited source and prepared/read temporary validation artifacts because Ace lacks a file-editing surface (#22) and external-harness integration (#9). Codex computer-use launched the disposable fixture because a native application-launch tool is still pending in #8, and read-only observation checked fixture visibility and the desktop lock. No input success is inferred from those observations or compilation.
iamnbutlerreferenced this in [Meta] Complete native computer use in Ace #8last month on GitHub
iamnbutlerreferenced this in Finish native pointer delivery and validation #68last month on GitHub
iamnbutlerready-for-reviewlast month on GitHub
iamnbutlermerged 68b4a3f64185last month on GitHub
iamnbutlerreferenced this in Finish native window, menu, and dialog workflows #73last month on GitHub
iamnbutlerreferenced this in Land desktop inspection/actions and tab API from feat/github-cache #89last month on GitHub
iamnbutlerreferenced this in Move and resize exact native windows #86last month on GitHub
iamnbutlerreferenced this in Verify a deployed hosted channel with a local workspace #176last month on GitHub
iamnbutlerreferenced this in [Meta] Run hosted channels in the cloud with local workspaces #13last month on GitHub