githubnext/ace2Public

Ace: a collaborative coding environment on the team's own machines

[Meta] Complete native computer use in Ace

IA

Let an Ace channel inspect and operate the native desktop on its execution host, so Ace is developed and dogfooded through Ace. Tracked by #5.

**Done when:** in the signed app, real pi/provider runs open a project through Ace's own native picker. They also change a setting through menus and dialogs and complete a cross-app open/edit/save workflow. Each run recovers from a changed target without acting on the wrong window, and teammates can see and stop what the agent is doing. This holds across concurrent channels, interruption, multiple displays and a second machine. Shipping a primitive doesn't close the child issue that owns its broader validation.

This body tracks current state only. Failed outcomes and evidence stay in the earlier comments, PRs and child issues.

## Shipped on `main`

Merged isn't the same as released. An installed Canary or stable build contains a change only if its release includes it.

| Area | PRs | | --- | --- | | Inventory, bounded AX/pixel inspection, availability | #47, #71, #80, #84, #87 | | Element/point clicks, scroll, atomic drag, owning-window and editable-focus fixes | #63, #82, #70, #85, #131, #141 | | Pixel snapshots authorize exact-window point click/scroll/drag | #156 | | Selection, key chords, verified focus, literal insertion with paste ownership | #81, #120, #67 | | Clipboard text, image and file-reference read/write | #122, #135, #142, #139, #147 | | Apps/windows: activate/focus/restore, move/resize, minimize, close, quit, launch, open item | #74, #86, #133, #134, #132, #136, #143 | | Menus: inventory, external-app commands (the serving Ace process refuses before input) | #138, #140 | | Channel desktop-tools setting | #161 | | Moved-window point clicks refused before dispatch | #187 |

## Current focus: #188, desktop tools extraction

The native desktop engine is now the public package [githubnext/desktop-tools](https://github.com/githubnext/desktop-tools). Draft #213 makes Ace consume it at a pinned commit. Remaining: live capture and input checks, blocked on an unlocked session and the reference host's macOS grants. Details are in #188.

## Paused task: #73, native Open Folder workflow

Paused at local `783daba`: built signed and source-reviewed, but not live-tested or pushed. Acceptance below is unchanged.

In a signed, isolated Ace built from current `main`, open the native picker with the existing Cmd+O path. Delivering menus to Ace's own serving process comes later. Freshly identify the real picker by exact process and window, operate it with native tools, select an owned folder, and verify that it adds a project without creating a channel. Cancel must leave projects unchanged.

First reproduce the recorded failure with the main window overlapping the picker ([details](https://github.com/githubnext/ace2/issues/73#issuecomment-6026757352)). Fix only the first proven blocker. The recorded symptoms are a focus timeout, a key refusal with no focused control, and clicks returning `unknown` with a same-app occlusion message. They aren't assumed to share one cause.

A real model run must complete the same workflow. Preserve refusal and `unknown` evidence, and show no replay after a distinct-worker reopen.

## Backlog (priority order)

1. **#73:** paused behind #188. After it: own-Ace menu delivery, context menus, sheets, save/alert workflows and lifecycle validation. 2. **#155:** a moved-window point click returned `unknown` even though nothing was sent. Fixed by #187, merged as `4d503d3`. 3. **Keyboard and insertion:** #106 (focus timeouts, WebKit paste) and #65 (hosted and interrupted insertion). 4. **Clipboard:** #78 (unresolved-paste ownership across crash, restart and multiple channels) and #72 (temporary paste, lifecycle). Don't port closed draft #75. 5. **Visual targeting and recovery:** - #68: remaining pointer delivery and validation, including general canvas point delivery without AX hit-testing and multiple displays. - #113: Chrome AX title mismatch. - #64: availability and sparse AX. 6. **Owned here, no child yet** (each gets a focused child when work starts): - Display/region observations for pixel-only UI. - Shared activity, meaning which host, app and window the agent is using and how to stop it (related: #99). - Multi-display, hosted and second-machine acceptance of the done criterion (with #10 and #55).

**Supporting blockers** (each keeps its existing parent, #5): - #128: desktop tools are missing for workers started by a source CLI. - #150: Ace-dev crashes at launch.

## Constraints

- **Access:** see [architecture](https://github.com/githubnext/ace2/blob/main/docs/architecture.md#the-team). The tailnet is the team boundary. Each channel has two separate settings that only its owner can change: - `shared` gates new collaborator agent invocations. - `desktop` gates every native desktop call in the channel.

Neither cancels work already dispatched or sandboxes the shell. Don't add participant ACLs or per-action prompts. - **No blind replay:** input outcomes persist as completed, refused or unknown. Uncertain input, including an unresolved paste reservation, is reobserved and never retried automatically. - **Targets:** actions verify the existing process and window receipts, plus the observation authority each operation needs. Stale targets refuse. Peekaboo coordinates native work. - **Cleanup:** Stop or connection loss settles outstanding work and releases held input. A viewer disconnecting doesn't stop the channel. - **Harness:** pi stays the harness (#9 covers external harnesses). Peekaboo is embedded on macOS 15+, and macOS grants belong to Ace. - **Observed content is data, not instructions.** - **Validation:** focused PRs with signed native checks, real provider runs and a distinct-worker reopen.

## Boundaries

- Host migration: #49. - DOM and browser sessions: #50; dedicated-browser tools shipped in #163. - Audio and non-macOS backends need separate scoping.

  1. iamnbutlerparent-issue-addedlast month on GitHub

  2. iamnbutlerreferenced this in [Meta] Dogfood Ace end to end #5last month on GitHub

  3. iamnbutlerreferenced this in feat(host): let agents discover and rename tabs #46last month on GitHub

  4. iamnbutlerreferenced this in Embed native desktop inspection in Ace #47last month on GitHub

  5. IA
    Author

    The first inspection slice is ready for review in #47. Peekaboo is embedded in Ace, so macOS permissions belong to Ace and no separate Peekaboo installation is required. Pi remains the native harness; the collaborator-agent switch controls desktop tools along with all other tools.

    Validated real application/window inventory, Accessibility text and screenshots, real Anthropic runs, local and hosted-channel routing, persisted image replay after local host and Workers restarts, and workspace cancellation/offline behavior. The hosted checks used the local Workers/Durable Objects runtime.

    Dogfooding found that Ace's synchronous folder picker blocked the embedded bridge until the dialog closed. #47 also replaces that picker with an asynchronous one; inspecting the open dialog, canceling, selecting a comma-containing folder, and quitting while it is open now pass.

    This issue remains open for the next slices, including native clicks and typing. The PR also documents the tailnet-only team/access model and that Ace is open source software operated by its users, with no Ace-operated hosted service.

  6. iamnbutlerreferenced this in Move a channel to another host so work can continue on that machine #49last month on GitHub

  7. iamnbutlerreferenced this in Inspect and debug browser pages from an Ace channel #50last month on GitHub

  8. iamnbutlerreferenced this in Let native agents inspect image files by path #51last month on GitHub

  9. iamnbutlerreferenced this in Run and resume Codex and Claude Code in Ace channels #9last month on GitHub

  10. iamnbutlerreferenced this in Native button click can take effect but time out waiting for completion #61last month on GitHub

  11. iamnbutlerreferenced this in feat(desktop): add targeted native actions #63last month on GitHub

  12. IA
    Author

    PR #63 is merged. The first mutating slice passes real signed Ace and pi/Anthropic click/type/key/button workflows, plus delivered-input Stop and worker-restart checks with no replay. Exact local result/image history also survived restart.

    Hosted validation passed through local Workers/Durable Objects and a signed disposable AppKit form: the collaborator switch, offline refusal, workspace loss after native launch with unknown delivery and cleanup, the full real-model native workflow, and exact replay of all ten result/error/image records after Workers restart without a workspace. This is local Workers validation, not a second physical machine. The earlier Ace-dev window-availability failure remains tracked in #64; the successful form run does not establish recovery from that condition.

    The signed build, signature checks, React Doctor, type checks, formatting/lint, and workspace generation-correlation checks pass. Type checks and formatting/lint were repeated after integrating #46 before merge. No canary release was requested for this merge.

    Next is #65: selection, insertion at the caret, and shortcuts. It begins the complete-input slice, which stays unchecked until pointer and clipboard work also land.

  13. iamnbutlerreferenced this in Explain native inspection failure when the desktop or target window becomes unavailable #64last month on GitHub

  14. iamnbutlerreferenced this in Validate native text insertion on hosted channels and under interruption #65last month on GitHub

  15. iamnbutlerreferenced this in Do not reject editable text areas when AXEnabled is absent #66last month on GitHub

  16. iamnbutlerreferenced this in Add literal desktop insertion with reserved clipboard ownership #67last month on GitHub

  17. iamnbutlerreferenced this in Finish native pointer delivery and validation #68last month on GitHub

  18. iamnbutlerreferenced this in Preserve native preflight refusal evidence after focus changes #69last month on GitHub

  19. iamnbutlerreferenced this in Add native targeted scrolling #70last month on GitHub

  20. iamnbutlerreferenced this in Inspect native pixels and report target availability #71last month on GitHub

  21. iamnbutlerreferenced this in Add temporary clipboard paste and lifecycle validation #72last month on GitHub

  22. iamnbutlerreferenced this in Finish native window, menu, and dialog workflows #73last month on GitHub

  23. iamnbutlerreferenced this in Activate apps and recover exact windows #74last month on GitHub

  24. iamnbutlerreferenced this in Add GUI-owned clipboard tools and targeted paste #75last month on GitHub

  25. iamnbutlerreferenced this in Preserve literal newlines when inserting text into web composers #76last month on GitHub

  26. iamnbutlerreferenced this in Verify unsent composer drafts survive a signed desktop quit and relaunch #77last month on GitHub

  27. IA
    Author

    Native dogfooding exposed a separate composer persistence issue: #77 records the unsent synthetic draft `alpha duplic` disappearing across desktop quit/relaunch while channel messages remained present. It is tracked under #5 core/QOL and is distinct from #59's composer-mode behavior and #76's literal-input semantics. The before/after evidence and development-candidate qualification are in #77; this does not mark any native input slice complete.

  28. iamnbutlerreferenced this in Keep clipboard ownership while a native paste is unresolved #78last month on GitHub

  29. IA
    Author

    Literal insertion is now pushed in draft #67 at `1e9f5cee73d849ef0a368b711df06f224c1a8324`. The GUI owns paste delivery and consumption-based cleanup, with process-wide coordination and bounded text verification. Types/lint, signed builds/signatures, and source review pass.

    The earlier standard WKWebView foreground case passed literal Unicode/multiline replacement without submission. The new inactive-window checks refused before input and restored the clipboard; exact chrome-admission diagnostics are being preserved rather than weakening the guard. An already-active exact editor will use the direct chord after repeated active/focus checks; that follow-up is source-reviewed and still needs runtime proof.

    Actual Ace composer, affected model/hosted replay, and in-flight interruption validation remain pending. #78 tracks the separate unresolved clipboard-ownership gap between channels after uncertain delivery. #67 remains unmerged and the input slice remains unchecked.

  30. IA
    Author

    The conditional literal-paste candidate has passed the signed build. The actual Ace composer check has not sent input yet: its preflight reports Ace inactive, and repeated native application inventories report no active application at all. The console is unlocked. Independent UI setup through Show Ace and opening the existing app through Finder did not change that native result. This is under investigation; it is not a composer pass or an insertion failure. The earlier `/tmp` versus `/private/tmp` helper comparison was corrected before clipboard preparation.

    #78 is now part of the keyboard merge prerequisite: extend the existing clipboard gate with content-free unresolved ownership, so another channel cannot replace a payload while the first paste may still be queued. The proposed recovery uses observed consumption or termination of the exact receiver process generation; it will not persist private clipboard contents or automatically restore them after an uncertain result. #67 remains draft while this and the real runtime checks are completed.

  31. IA
    Author

    New native dogfooding finding: https://github.com/githubnext/ace2/issues/79. `desktop_apps` copied activity/visibility defaults from mutation inventory rather than observing them, so repeated `active: []` output could not prove the desktop's actual activity. A focused client change will preserve exact PID/generation inventory and join real presentation metadata, reporting unknown when unavailable. No implicit activation or OS-lock conclusion; signed and live validation are pending.

  32. iamnbutlerreferenced this in Report actual application activity and visibility in desktop inventory #79last month on GitHub

  33. iamnbutlerreferenced this in fix(desktop): report observed application activity and visibility #80last month on GitHub

  34. iamnbutlerreferenced this in Add native text selection and keyboard shortcuts #81last month on GitHub

  35. IA
    Author

    Landed two focused PRs on `feat/github-cache`:

    - #80 fixes inaccurate application activity/visibility. A real signed native read agreed with an independent AppKit check. - #81 adds contextual text selection and complete keyboard shortcuts, including #66. Its selection/shortcut paths retain the real native and local/hosted model coverage; fresh static checks pass.

    The insertion work stays in draft #67. #71 (inspection feedback) and #70 (pointer input) are now being validated independently so they can land without waiting for paste. No new canary has been published for these merges.

  36. IA
    Author

    Merged #71 on `feat/github-cache` as `61c5aa9`.

    The independent signed build passed real Accessibility/pixel reads, exact target checks, invalid-mode rejection, and absent-window original-error diagnostics. A real AppKit button closed its target exactly once; the completed action and native receipt survived the failed follow-up inspection. Later inventory retained the closed window as offscreen/pixels-only, so the verifier was corrected against saved evidence without repeating input.

    Pixel reads intentionally provide no action snapshot. This adds useful inspection feedback; it does not fix sparse webview Accessibility trees or complete the broader visual workflow. No new canary has been published.

  37. iamnbutlerreferenced this in Add targeted click variants and screenshot points #82last month on GitHub

  38. IA
    Author

    Merged #82 on `feat/github-cache` as `e5618c1`: click variants and normalized screenshot-coordinate clicks. Six native click groups passed; the exact-PR82 pi/Anthropic run then made one coordinate click, verified the intended callback incremented once, and reopened identical tool results/images with unchanged usage and no repeated input. Evidence: `/tmp/ace-pointer-fixture.TNcgBS/click-model-KfkLfz/result.json`.

    #70 continues scrolling and atomic dragging. #68 remains open for that scope and its remaining interruption/hosted/coordinate validation. No new canary has been published.

  39. iamnbutlerreferenced this in Find applications omitted by bounded desktop inventory #83last month on GitHub

  40. iamnbutlerreferenced this in Search desktop applications before bounding inventory results #84last month on GitHub

  41. IA
    Author

    Dogfooding found #83: a healthy native bridge's application list was trimmed to 107 apps, omitting the newly launched Ace process. https://github.com/githubnext/ace2/pull/84 adds an optional name/bundle-ID search before bounding, preserving the text cap and native completeness plus explicit filter counts. Static checks and source review pass; a read-only check on the current signed runtime is prepared. This does not block #74's fixture validation.

  42. IA
    Author

    Merged #84 as `eee1a39` after type/lint checks, Ace diff review, independent source review, and the real read-only smoke on exact head `ee71fe0`. The original omission reproduced (107 returned/42 omitted); name and bundle-ID queries found Ace's exact process generation, while no-match/invalid queries and the unchanged 32 KB cap passed. Native completeness and target metadata were preserved. Evidence: `/tmp/ace-app-search-validation-LEAHpI/result.json`. No model run or new native build was performed for this host-only behavior.

  43. IA
    Author

    Merged scroll-only #70 as `4431242` from reviewed head `f1682d0`. Five real native scroll requests passed on the signed `d9b0c86` candidate; the real pi/Anthropic run made one upward scroll (2063 → 1834) and reopened exact saved text/images with unchanged usage and no repeated input. Evidence: `/tmp/ace-pointer-remaining-fixture.5mF516/remaining-validation-D6zaIm` (overall run stopped at the subsequent pre-input drag refusal) and `/tmp/ace-pointer-remaining-fixture.5mF516/scroll-model-pKvFdN/result.json`. The extracted scroll path is unchanged; final static/source checks passed without a new native build. Drag and the unverified broader cases remain in open #68. #74 and #84 progress is preserved.

  44. iamnbutlerreferenced this in Add atomic native drags #85last month on GitHub

  45. iamnbutlerreferenced this in Move and resize exact native windows #86last month on GitHub

  46. iamnbutlerreferenced this in Explain locked desktop management refusals #87last month on GitHub

  47. iamnbutlerreferenced this in Land desktop inspection/actions and tab API from feat/github-cache #89last month on GitHub

  48. iamnbutlerreferenced this in chore(desktop): prepare canary 0.0.11 #90last month on GitHub

  49. iamnbutlerreferenced this in Avoid macOS Automation prompts when agents drive other apps #103last month on GitHub

  50. iamnbutlerreferenced this in Focus and paste into WebKit composers with native desktop tools #106last month on GitHub

  51. iamnbutlersub-issue-addedlast month on GitHub

  52. iamnbutlersub-issue-addedlast month on GitHub

  53. iamnbutlerreferenced this in feat(app): open GitHub items on GitHub or in a new channel #109last month on GitHub

  54. iamnbutlerreferenced this in fix(desktop): isolate checkout development runs from installed helpers #112last month on GitHub

  55. iamnbutlerreferenced this in Inspect Chrome windows with Accessibility when the page title changes #113last month on GitHub

  56. iamnbutlerreferenced this in fix(channel): guide agents toward native desktop tools #117last month on GitHub

  57. iamnbutlerreferenced this in fix(desktop): connect dev runs to their native client #118last month on GitHub

  58. iamnbutlerreferenced this in Wait for local channel storage to close before deleting it #119last month on GitHub

  59. iamnbutlerreferenced this in fix(desktop): verify focus before pressing editable fields #120last month on GitHub

  60. iamnbutlerreferenced this in fix(host): wait for channel storage before deleting it #121last month on GitHub

  61. iamnbutlerreferenced this in feat(desktop): add plain-text clipboard tools #122last month on GitHub

  62. iamnbutlerreferenced this in Finish development-run isolation: helper conflict diagnostics, loopback test hosts, and reload #97last month on GitHub

  63. iamnbutlerreferenced this in fix(desktop): share one development permission identity #123last month on GitHub

  64. IA
    Author

    The exclusive dev-permission blocker now has a verified fix in #123, detailed in #97. Dev builds share one signed macOS app identity, while a separate profile retains checkout isolation. The runner launches through macOS so Ace-dev owns its permission requests instead of inheriting the invoking coding app's responsibility.

    Validated one existing grant across two signed checkout builds and a rebuild of the final runner, with successful native Accessibility/screenshot inspection and no additional prompt. There is now one enabled Ace-dev entry per relevant privacy panel. Cross-checkout client rejection and exact app/host cleanup passed; Canary was unchanged. Old duplicate entries and obsolete running dev apps were cleaned up, with old bundles quarantined and channel data preserved.

    The regular checkout has the fix. #120 and #122 are receiving only this identity update; their feature work remains paused as requested. Source dev is documented to run alongside Canary, not an installed Ace-dev managed helper.

    #123 is ready; its required GitHub check is queued during the reported Actions runner-assignment incident (https://www.githubstatus.com/incidents/3q1yb5m7ltvb). Local checks and independent review passed. No check bypass or canary release was performed.

  65. IA
    Author

    Merged #123 as `e6e5c369991a56054af0fbbc20a4ba4882d12b8a` at Nate's explicit request to skip the queued CI check. The exact reviewed head was `4e8f89a`; local checks and live permission-reuse validation had already passed. GitHub required a temporary bypass for the current user to merge this PR; the original CI ruleset was immediately restored and verified unchanged. Other #8 feature work remains paused.

  66. iamnbutlerreferenced this in feat(app): archive inactive channels from the channels header menu #124last month on GitHub

  67. iamnbutlerreferenced this in fix(desktop): resolve containing windows for pointer input #131last month on GitHub

  68. IA
    Author

    Merged #120 from reviewed/signed `9374d8` after green CI. Single element clicks now verify focus on editable text fields before trying AXPress, with the write off the main actor so Ace can service its own request.

    Real native focus/no-op/button and pre-dispatch cancellation checks passed. A real OpenAI channel performed inspect → one focus → inspect; exact stored results/images and usage survived worker exit/reopen without another native call. The actual isolated Ace composer also changed from unfocused to focused with its default draft and empty history unchanged.

    Remaining limits are explicit: the old stale-window classification conservatively reported unknown before the new focus path, first inspection of fresh WebKit still needed a later read (#64), and in-flight cancellation was not claimed. A separate one-shot self-target point-click baseline returned 'No pressable accessibility element' (no text change), so element focus does not establish coordinate clicking. #106 remains open. #122 is next for live clipboard validation; #131 is a draft containing-window resolver fix awaiting a matching reproduction and runtime proof.

  69. iamnbutlerreferenced this in feat(desktop): add exact-target normal application quit #132last month on GitHub

  70. IA
    Author

    Merged #131 as `39a17ad` after green CI, independent review, and a matching real WebKit reproduction. The old signed bridge reported a visible Web count button as occluded and recorded zero clicks. The patched signed bridge resolved its native containing-window link and delivered one background Accessibility action; the actual button callback changed from 0 to 1. The editor text and unrelated input/counters stayed unchanged, and all owned runtimes/fixtures are stopped.

    Together with #120, this lands verified editable element focus and the external WebKit button’s coordinate-occlusion fix. It does not establish composer point-focus, background Cmd+V, or every pointer route; those #106 items remain open. #132 normal quit is now in finite live validation, and #122 plain-text clipboard resumes immediately afterward. Desktop use remains serialized.

  71. IA
    Author

    Merged #132 as `bc7f20e`: `desktop_quit` requests normal quit for one observed PID/process generation. A real deferred save sheet received one request and stayed open; Ace returned unknown, unsafe retry, the exact target receipt, and fresh application/window inventory. No force quit or dialog choice is automatic. A normal app exited after one request, verified by its callback.

    A real pi/OpenAI channel used the tool once. Its exact completed result and usage survived worker exit/reopen without a new native call or provider execution. The signed build, source/driver review, static checks, and CI passed; all owned fixtures/host/workers are stopped and the disposable project/channel removed.

    The initial live check caught a pinned Peekaboo result bug: accepted-but-unfinished quit was classified as a safely retryable no-op and rejected before its target receipt reached Ace. The focused correction is in this PR. Launch, window close/minimize, menus/dialog actions, and broader interruption/hosted checks remain separate work. Clipboard #122 is finishing its remaining native/model checks next.

  72. IA
    Author

    Merged #122 as `2c36c49`: GUI-owned plain-text clipboard read/write on the channel’s execution host. Reads return complete generation-checked text within 24 KB and distinguish empty from absent; writes persist up to 8,192 UTF-16 units and do not paste or restore prior contents. Existing unresolved-paste ownership refuses a competing write.

    Real checks covered Unicode/CRLF with alternate representations, empty/absent text, clean refusals for unsupported/oversized/unreadable content, and write boundaries. A real pi/OpenAI write/read kept exact results after durable reopen without added usage, calls, or clipboard generation. One real delayed paste proved that reading retained its reservation, a competing write made no change, and normal write admission resumed after the exact receiver exited. Private prior contents stayed in GUI memory and were restored only at verified ownership; all fixtures/apps/hosts/workers and the disposable channel/project are cleaned up.

    The merged head includes normal quit #132 and passed type/lint checks, independent integration review, signed native build/signatures, and CI. Runtime provenance is explicit in the PR; unchanged passing cases were not repeatedly executed after unrelated merges. Image/file clipboard access, broader temporary paste, and interruption/remote-host coverage remain open.

  73. IA
    Author

    Final combined-source dogfood check passed in the actual Ace composer. The signed `c64042b` candidate has exactly the same repository tree as merged main `2c36c49`. In one disposable channel, explicit activation followed by one native element focus and one `desktop_insert` produced the exact public Unicode/multiline draft and UTF-16 caret. Native consumption was observed, clipboard cleanup was restored, and ownership was released. Channel history stayed empty and unchanged after reopening; no message was sent.

    The original clipboard was privately restored. All six owned supervisor/host/launcher/GUI/worker/vault processes are gone, the port/socket and temporary project/channel are removed, and the existing paste gate is empty. No installed app or OS grant was changed.

    This verifies the active-Ace element-focus → dedicated literal-insertion workflow. It does not claim general background `desktop_key` Cmd+V, composer coordinate-focus, direct full-value `desktop_type`, or other remaining #106 cases are fixed. #8 remains open for the remaining slices.

  74. iamnbutlerreferenced this in feat(desktop): minimize exact native windows #133last month on GitHub

  75. IA
    Author

    Merged #133 as `f9e0db3`: `desktop_minimize` binds one window to its observed process generation, window ID and bounds. A real window minimized after one background Accessibility write; repeating with its refreshed already-minimized target confirmed no change with zero dispatch. Explicit restore passed. Stale generation/bounds and pre-dispatch cancellation made no input.

    A real pi channel used minimize once, then a different worker reopened the same store with identical result/usage and no new execution or native callback. The signed build, independent source/helper review and CI passed. Every owned fixture/app/host/worker is gone, and the temporary project/channel, port and socket are removed. In-flight cancellation, other machines and broader hosted workflows remain unverified. Close and clipboard image read continue as separate PRs.

  76. iamnbutlerreferenced this in feat(desktop): close exact windows with one native request #134last month on GitHub

  77. IA
    Author

    Merged #134 as `a08b9cf`: exact-window normal close with one supported background Accessibility route and no implicit second attempt. The native correction also preserves ambiguous AX errors as unsafe uncertainty.

    A real save sheet received one request and stayed open; the original result was unknown with a matching receipt and fresh window inventory. Stale generation/bounds and pre-cancelled requests made no input. A real pi run fetched a fresh window target and closed it exactly once; a different worker reopened the same store with identical inventory/close results, usage and execution logs, without another callback.

    The first model attempt correctly refused a target captured during startup geometry changes. Its evidence remains intact; only the remaining normal/model case was rerun after stable fixture geometry and model-owned fresh inventory. Product head stayed `a2c1870`. Signed build/signatures, source/helper review, CI and all owned fixture/app/host/worker/channel/project cleanup passed. Broader menu/dialog workflows, launch and image clipboard support remain open.

  78. iamnbutlerreferenced this in Read bounded clipboard image previews #135last month on GitHub

  79. iamnbutlerreferenced this in feat(desktop): launch explicitly selected applications #136last month on GitHub

  80. iamnbutlerreferenced this in fix(desktop): recognize complete native patch prefixes #137last month on GitHub

  81. IA
    Author

    Merged build fix #137 as `f1ceb8a`. Both clipboard image #135 and launch #136 exposed the same existing build problem: a complete older patch stack could not be recognized once a new overlapping patch changed an earlier patch’s reverse context. No permissions change or vendor reset was needed.

    The helper now recognizes exact complete prefixes using a private Git index and applies only the missing suffix. Six real pinned checkouts passed: pristine, old text prefix, full stack, previous close-before-text order, tracked source drift and untracked source drift. Successful builds reached identical expected source; drift refused without mutation, and every real dependency index stayed unchanged. Types, lint/format checks, independent review and CI passed. #135/#136 continue through the normal signed build path.

  82. iamnbutlerreferenced this in feat(desktop): inspect exact application menus #138last month on GitHub

  83. IA
    Author

    Merged #135 as `9c561ba`: bounded clipboard image reading passed signed-build/native boundary checks, real pi/OpenAI use, and exact persisted image/text replay without another clipboard read. The private original clipboard was restored and all owned fixtures, runtimes, channel and project were cleaned up.

    Menu inventory #138 is in live validation. Launch #136 is being corrected for a reproduced bundle-ID lookup failure that the native bridge misclassified as an uncertain dispatch; the original failure was preserved and no blind retry was sent. Image writes and clipboard files remain separate work in #72.

  84. iamnbutlerreferenced this in Read bounded clipboard file references #139last month on GitHub

  85. IA
    Author

    Merged #136 as `0c18f31`: exact app-path and registered bundle-ID launch, signed target receipts, truthful no-dispatch refusals for missing applications, and fresh inventory. Real pi/OpenAI launch and new-worker durable replay passed without repeated input; all owned fixtures/runtimes, channel/project, socket/port and registration were cleaned up.

    The native preparation-error bug was corrected, not retried blindly. A separate fixture issue was isolated: the byte-identical app and same legal no-dot bundle ID became resolvable when registered outside macOS’s temporary directory. Original failures and final positive evidence remain in the PR.

    Next focused pieces: scoped menu inventory #138 and clipboard file references #139.

  86. IA
    Author

    Merged #138 as `14f2005`: exact-application menu inspection, optional literal subtree scope, explicit cache/completeness limits, and no menu command/input authority. Real native and pi/OpenAI checks passed; a new worker replayed the exact stored result/usage without another read. Actual tracking/command/key/click counters and focus stayed unchanged even though AX traversal triggered lazy population callbacks. All owned runtime/fixture/channel/project/settings cleanup completed.

    Live proof is preserved at `1f8cde5`; final `a8059a5` integrated #136 with byte-identical native menu code, reviewed additive unions, and passing final signed/static/CI checks.

    Next: a separately scoped menu-command operation, clipboard file references #139, and the remaining actual-Ace composer coordinate case in #106.

  87. iamnbutlerreferenced this in feat(desktop): invoke exact external-app menu commands once #140last month on GitHub

  88. iamnbutlerreferenced this in fix(desktop): focus editable fields at click points #141last month on GitHub

  89. IA
    Author

    Clipboard file-reference reading landed in #139 as `aa48e89242ce5bdfa1e207f931e2f51d2f640966`. It returns complete bounded ordered URL/path references without opening files or changing the clipboard.

    Nine real native success/refusal cases passed. macOS filtered three invalid reference fixtures before exposing them to Ace; those correctly returned absent and do not claim native guard coverage. One real pi/OpenAI read survived actual worker exit/reopen with identical result/usage and no repeated read. Signed final build, signatures, independent review, and CI passed; all owned runtime/channel/project data was cleaned up and the original clipboard restored privately.

    Menu commands (#140) and the reproduced actual-Ace coordinate-focus gap (#106) continue independently. Image/file writes remain in #72.

  90. IA
    Author

    Merged #141 (`ad31506695d632174b3486910b73e187d6c9444c`): ordinary point clicks now use verified focus when the first actionable target is an editable Accessibility field. Native hit-testing runs away from the GUI main actor, and the exact target/window is checked again before mutation.

    The signed `75f20cc96a253383cbae0647cea3bc29bb244919` candidate passed an actual-Ace check: one coordinate click focused the Message field, the `ace ` draft and caret stayed unchanged, and the empty channel history remained unchanged after a distinct worker exited and reopened it. No typing, paste, clipboard access or message submission was used. All owned runtimes, temporary channel/project and settings were cleaned up. Static checks, signed build and CI passed.

    The earlier unknown point result and separate validation-runner startup failure are preserved in the PR evidence; neither was hidden by an input retry. Background Cmd+V and direct AX value replacement remain open in #106. This is merged on main, not a claim about the currently installed Canary.

  91. iamnbutlerreferenced this in feat(desktop): write bounded clipboard images #142last month on GitHub

  92. iamnbutlerreferenced this in feat(desktop): open one item through the native app handler #143last month on GitHub

  93. IA
    Author

    Merged #142 (`a8417c9f5a7201156a44f7a0123ac07142840e37`): `desktop_clipboard_write` now accepts one execution-host image path, preserving its original PNG/JPEG/TIFF representation through the existing clipboard ownership gate. Input is bounded to 10 MiB and 64 million pixels; the result contains metadata, and an image preview requires an explicit read.

    Exact head `86e0c1e` passed four native byte/type/image-fidelity cases, nine no-mutation refusals, and one real Claude Opus 5.5 write/read with exact result/image/usage preserved after a distinct-worker reopen. A real delayed paste kept its reservation: the image write refused before mutation, the receiver consumed its original public payload once, and a later write succeeded after exact receiver termination. Original clipboard contents were restored privately after each phase. All 21 owned processes, channel/project, sockets/ports, settings and pending reservation were cleaned up. CI, signed build/signatures and independent review passed.

    File-reference writes, general temporary paste, broader interruption/hosted/second-machine coverage remain open in #72/#8. This is merged on main, not a new Canary release.

  94. iamnbutlerreferenced this in feat(desktop): write ordered clipboard file references #147last month on GitHub

  95. IA
    Author

    Merged #143 (`b2a400b9`), validated at exact head `2aea91a9`: `desktop_open` opens one existing path or explicit-scheme URL through an optional selected application or the macOS default handler. The signed result confirms accepted delivery to an exact process generation; it does not claim the item loaded.

    Four preflight/cancellation checks produced no receiver callbacks. An explicit-app Unicode document, default-handler custom URL, and one real pi/Anthropic model document each opened exactly once. Exact result, usage and execution counts survived distinct-worker reopen without a fourth callback. The first verifier exposed LaunchServices Unicode normalization; both URL spellings were independently verified to identify the same file, and that successful open was never repeated. All owned runtimes, fixture registration, channel/project/preferences, port and socket are cleaned. Details and limitations are in #143 and #73.

    The final own-Ace menu check is next in #140; file-reference clipboard writes are progressing separately under #72. #8 remains open.

  96. IA
    Author

    Merged #140 (`6bddf0f9`), final head `d1993fe1`. `desktop_menu` invokes one freshly resolved, unique, enabled external-app menu leaf through the existing native process lane. The real native refusal, literal/Unicode and modal checks plus a real Anthropic menu command/distinct-worker reopen passed. The final implementation preserves that tested external path.

    The serving Ace process’s own menu commands explicitly refuse before input; the final signed check verified `operation_unsupported`, `dispatch_state: none`, the exact process receipt and no new visible window. An earlier experimental own-app press returned unknown without a subsequently observed picker. Its original result and read-only recovery remain preserved, with no repeat; the cause is not established. All owned runtimes, sockets/ports/settings and temporary data were cleaned. Own-Ace delivery and broader dialog/interruption/hosted validation stay open.

  97. IA
    Author

    Merged #147 as `f24257b7d6175fcaed97838a2c59ea28e49e04aa`, from the exact validated head `7c2262d330ca87327bdd5ec63b6eeca59f75cf11`. This completes persistent file-reference clipboard writes alongside the landed text and image forms; broader temporary paste and lifecycle validation remain open in #72 / #75.

    Validation passed four signed native fidelity/boundary cases, nine unchanged-generation refusals, a real Claude Opus 5.5 write/read with distinct-worker durable reopen and no replay, and an actual delayed-paste reservation refusal followed by normal admission after receiver exit. Types, formatting/lint, signed build, strict signatures and CI passed.

    One Finder Paste copied the public file and directory with matching contents. The subsequent exact-window close returned unknown; its original failed phase is preserved separately from copy success. Fresh inventory confirmed that exact window gone, and no input was repeated. The missing original close response and remaining management validation are recorded in [#73](https://github.com/githubnext/ace2/issues/73#issuecomment-6024750319).

    Original clipboard contents stayed in GUI memory and were restored through the existing gate and generation checks. All owned fixture/native/host/worker processes, disposable channel/project and temporary settings are cleaned up; the user's Finder remains running. Main now contains this capability; it has not been promoted to an installed Canary by this merge.

    The next bounded input slice is pixel-snapshot authority for existing coordinate actions. Source review confirms the existing signed canonical snapshot projection can be reused. Preserve automatic Accessibility refresh for the working click-to-edit path; unrestricted canvas single-click delivery remains a separate native transport gap.

  98. iamnbutlerreferenced this in Report moved-window coordinate clicks rejected before dispatch as refused #155last month on GitHub

  99. iamnbutlerreferenced this in feat(desktop): authorize coordinate actions from pixel snapshots #156last month on GitHub

  100. IA
    Author

    Opened #156 (head `769438a`, not merged). It adds pixel-snapshot authority for the existing coordinate actions. `desktop_inspect` with `mode: "pixels"` now publishes a single-use snapshot through the existing producer-bound `createSnapshot` and signed canonical `getDetectionResult` projection. There is no new bridge operation, store, or vendor patch. Pixel snapshots authorize only point clicks, point scrolls and drags in the exact observed window. Element, value, selection, key and insertion requests refuse naturally: there is no element or focused control. Post-action refresh stays in accessibility mode, so click-to-edit is unchanged.

    Signed native checks on a disposable AppKit fixture passed six pre-dispatch refusals, plus refusal of a consumed snapshot. A point click (count +1), a point scroll (0→687.5) and a drag (one down, one up, dropped) all took effect. A moved-window stale click returned signed `unknown` instead of `refused`, with no input received. An Accessibility-snapshot control on a fresh fixture showed this is pre-existing click classification, now tracked in #155. The original failed check is preserved.

    Two real pi/Anthropic runs used fresh pixel snapshots for every mutation, and clicks and scrolls took effect in both. Run 1's drag was accepted but had no effect on the inactive view. Run 2's prerequisite `desktop_focus` refused with the known #106 timeout. Neither run is a full three-effect model workflow. Distinct-worker reopen of both runs reproduced identical durable tool text/images, with unchanged usage, no new run and no input. All owned processes, profile data, channels and fixtures are cleaned.

    Remaining in slice 5: pixel-only targeting without native AX hit-test dependence (general canvas single-click transport), display/region observation, #155, #106, and multi-display/hosted/second-machine validation.