githubnext/ace2Public

Ace: a collaborative coding environment on the team's own machines

feat(desktop): add targeted native actions #63

Mergediamnbutlermerged 2 commits intofeat/github-cachefromrefs/pull/63/head
  1. IA

    Ace's native tools can inspect windows but cannot yet complete a click-and-type workflow. This adds `desktop_click`, `desktop_type`, and `desktop_key` to pi for the first mutating slice of #8.

    Actions use single-use observations bound to the application instance, exact window, and observed control. Text replaces a field's entire value; keys are limited to Return, Tab, Escape, deletion, and arrows. Successful actions return fresh bounded Accessibility/image evidence when available. Outcomes distinguish completed, refused, and unknown delivery, and pi never automatically replays input after interruption. Hosted workspace calls also have generation-specific IDs so a late reply cannot settle a new runtime's action.

    The signed embedded bridge owns input and macOS permissions. Ace's existing collaborator-agent switch still controls every tool together. Settings now report Keyboard input permission. A pinned Peekaboo source patch moves semantic AXPress off MainActor while retaining operation ownership, and prevents an ambiguous native error from falling back to another click (#61).

    Validation completed:

    - Type checks, formatting/lint, React Doctor (100/100), signed macOS build and signature verification, and workspace reply correlation across runtime generations. - Real signed desktop field replacement, arrow keys, and a Rename button press that closed the dialog and persisted the new channel name. Foreign observations and nonliteral element IDs were refused; simultaneous reuse of one snapshot delivered one action and refused the duplicate. - A real pi/Anthropic run inspected the window, clicked and edited the field, moved the caret, pressed Rename, and verified the result. Exact text, errors, and image results replayed from persisted history after worker restart. - Stop and worker-restart cases each withheld a real native response after the input had completed. Both preserved an unknown interrupted result and delivered the action exactly once; the restarted model reinspected without repeating it. - The local Workers/Durable Objects path passed collaborator-switch rejection, offline refusal before native dispatch, and workspace loss after launching the native request (unknown delivery, no retry, drained handler and cleaned temporary files). This disconnect test does not establish that its AX mutation landed.

    The final hosted workflow also passed through a real pi/Anthropic run against a signed disposable AppKit form: click, replace, key, and button activation changed the visible result and persisted the intended value. Restarting the local Workers/Durable Objects runtime without a workspace replayed all ten completed/refused/unknown results with identical text, errors, and images. Hosted validation used the local Workers runtime, not a second physical machine.

    During validation, Ace-dev's test window became unavailable to native inventory/Accessibility despite remaining readable through Codex's controlled view. That separate availability/diagnostics problem is tracked in #64; the successful hosted form run does not establish recovery from it. Wider application/key coverage remains follow-up work in #8.

    Fixes #61.

    ### Built in Ace

    Builds, checks, runtime scripts, Git/GitHub work, and diff retrieval use Ace's terminal and channel APIs. Codex and its subagents edited and reviewed source and prepared smoke scripts; external harness integration is tracked in #9 and Ace's file editor in #22. Codex computer use prepared the disposable desktop target because the missing native action surface is the work in #8. Tests used a separate Ace profile/project and a disposable signed AppKit form, preserving existing channels. Dogfooding found the button completion problem recorded in #61.

  2. iamnbutlerreferenced this in [Meta] Complete native computer use in Ace #8last month on GitHub

  3. iamnbutlerreferenced this in Native button click can take effect but time out waiting for completion #61last month on GitHub

  4. iamnbutlerreferenced this in Explain native inspection failure when the desktop or target window becomes unavailable #64last month on GitHub

  5. iamnbutlerready-for-reviewlast month on GitHub

  6. iamnbutlermerged d9f4c5816dfalast month on GitHub

  7. iamnbutlerclosedlast month on GitHub

  8. iamnbutlerreferenced this in Validate native text insertion on hosted channels and under interruption #65last month on GitHub

  9. iamnbutlerreferenced this in [Meta] Dogfood Ace end to end #5last month on GitHub

  10. iamnbutlerreferenced this in Add native text selection and keyboard shortcuts #81last month on GitHub

  11. iamnbutlerreferenced this in Land desktop inspection/actions and tab API from feat/github-cache #89last month on GitHub