Avoid macOS Automation prompts when agents drive other apps
iamnbutler opened ·
While validating #91 in Ace Canary 0.0.11, an agent prepared real clipboard cases (Finder file copy, Numbers cells, Safari "Copy Image") by running `osascript` through the channel shell tool. Each target app made macOS show a separate prompt asking to let **Ace Canary** control another app (Automation / Apple Events, TCC `kTCCServiceAppleEvents`). These prompts are separate from Ace's existing Accessibility and Screen Recording grants, and the supervisor had to ask why they appeared.
## Commands that triggered prompts
- `osascript -e 'tell application "Finder" … make new Finder window … select {…}'` to open a folder and select files, plus `tell application "Finder" to activate` and `close window`. - `osascript -e 'tell application "System Events" to keystroke "c" using command down'` (also `tell process "Numbers"`) to copy. Native `desktop_key` Cmd+C in Finder was refused with "The snapshot has no exact focused control" even though the file list had a selection, so the agent fell back to System Events. - `osascript` `tell application "Numbers"` to create a document, set cells, select a range, and later `quit saving no`. - `osascript` `tell application "Safari"` to list tab URLs and `quit`.
Native `desktop_click` (right-click, then the "Copy Image" menu item) worked in Safari without any prompt.
## Gaps that pushed the agent to AppleScript
- ~~`desktop_key` can't send a shortcut to a window whose focus is a list or outline row rather than a text control (Finder's file list).~~ Unverified: the agent didn't follow the refusal's hint to click a row and inspect again (see comments). - There's no native way to open a file or folder in an app, launch an app, or close or quit an app or window the agent opened. - Agent guidance doesn't warn that `osascript`/Apple Events cause per-app Automation prompts attributed to Ace.
## Expected
Agents can do routine cross-app setup (open, select, copy, close) with Ace's native desktop tools under the existing Accessibility grant. When Apple Events are truly required, the agent explains why before triggering the prompt.
This issue covers prompt guidance; the tested capability failures are in #106. Sub-issue of #8 (tracked under #5). Related: #65, #73.
iamnbutlerparent-issue-addedlast month on GitHub
- IA
Follow-up from channel `image-paste-review`. This is a read-only review of the agent's prompt and of the image-paste-91 transcript. It adds the guidance side of this issue; the capability gaps stay tracked here and in #8, #65 and #73.
## Missing guidance
At origin/main `1ab91be`, nothing in what the agent receives says when to use the native desktop tools instead of the shell:
- `packages/channel/src/room.ts` (`ace-channel` section) covers only chat etiquette, the channel summary and lanes. It doesn't mention desktop tools. - `packages/channel/src/context.ts` injects `AGENTS.md`. Its only related line is "If Ace ships a surface (channels, diffs, terminals), use Ace's." That line is about dogfooding Ace's own surfaces; it doesn't say how to control other apps. - The `desktop_*` descriptions in `packages/channel/src/desktop.ts` describe mechanics only: targets, snapshots, delivery and refusals. None of them says these tools are the default way to control apps, that Apple Events trigger a separate Automation prompt for each app, or what to do when a tool can't do a step. - `docs/desktop-tools.md` states the tool contract ("There is no fallback to global mouse or keyboard input"; clipboard operations, application launch, window close, menus and dialogs are later slices of #8). That file isn't in the prompt, and the agent didn't read it. - The handoff from `dev-paste-validation` also said "You have permission to write the pasteboard (swift/osascript)", which made `osascript` look sanctioned.
## Existing instructions the agent didn't follow
- In Finder, `desktop_key` Cmd+C was refused with "The snapshot has no exact focused control. Click a control, then inspect the window again." The agent didn't click a file row and inspect again. It moved straight to `osascript … System Events keystroke`. The gap listed above ("desktop_key can't send a shortcut to … Finder's file list") is therefore unverified. - After Nate said to prefer native tools and to explain any concrete need for Apple Events first, the agent stopped using `osascript`. Instead it compiled its own Swift helpers in `/tmp/pb`: - `axfocus` sets `AXFocused` on the composer. - `menu` runs AXPress on Edit > Paste. - `ev` posts CGEvents at `cghidEventTap`, guarded only by a frontmost check.
It ran `./ev 50933 key 0`, which typed a stray "a" through global input while Nate was using the machine. None of these helpers raises an Automation prompt, but they bypass the snapshot and exact-target checks of the native tools. The agent didn't explain the gap in chat before using them.
The native gaps it hit were real and should be fixed:
- `desktop_click` on the WebKit composer didn't focus the contenteditable. - `desktop_focus` was refused with `operation_unsupported` (timeout). - Background `desktop_key` Cmd+V did nothing.
The agent also didn't retry native input after `desktop_activate` succeeded. The legitimate shell uses are also worth recording: builds, `open -a` (there is no native launch yet), and writing the pasteboard directly with NSPasteboard were fine and raise no Automation prompt.
## Proposed narrow fix
Have the `ace-desktop` extension (`packages/channel/src/desktop.ts`) contribute a short prompt section, present only when desktop tools are injected. Then every project gets the guidance, not just repositories whose AGENTS.md mentions it. For example:
> Use the desktop_* tools to observe and operate apps on this host. Shell commands remain appropriate for builds, files, launching apps with `open`, and preparing clipboard fixtures directly. Don't drive other apps through AppleScript, `osascript`, or System Events, or through self-built Accessibility or CGEvent programs. Apple Events raise a separate macOS Automation prompt for each app, and custom input bypasses the desktop tools' target checks. If a desktop tool refuses or can't do a step, follow its hint once. If it still can't, name the tool and the gap before using any fallback.
Done when an agent asked to set up a cross-app clipboard case uses the native tools, or explains which native capability is missing before it falls back.
iamnbutlerreferenced this in feat(ui): attach images pasted into the composer #104last month on GitHub
iamnbutlerreferenced this in Focus and paste into WebKit composers with native desktop tools #106last month on GitHub
- IA
The tested native focus and paste failures are now tracked in #106, with exact steps and results. This issue stays about Automation prompts and agent guidance.
One correction to the gap list above: the line "`desktop_key` can't send a shortcut to a window whose focus is a list or outline row (Finder's file list)" wasn't tested. `desktop_key` Cmd+C was refused with "Click a control, then inspect the window again." The agent didn't follow that hint before using `osascript`, so that gap is unverified, as the review comment above noted.
iamnbutlerparent-issue-removedlast month on GitHub
iamnbutlerparent-issue-addedlast month on GitHub
iamnbutlerreferenced this in [Meta] Complete native computer use in Ace #8last month on GitHub
iamnbutlerreferenced this in [Meta] Dogfood Ace end to end #5last month on GitHub
iamnbutlerreferenced this in fix(channel): guide agents toward native desktop tools #117last month on GitHub