Focus and paste into WebKit composers with native desktop tools
iamnbutler opened ·
While validating PR #104 (#91) in Ace Canary 0.0.11, Ace's desktop tools couldn't place focus in the Ace-dev (Electrobun/WKWebView) composer or paste into it. This issue records the tested failures only. #103 keeps the Automation-prompt and agent-guidance problem.
**October 5 progress:** #120 fixes verified editable element focus; #131 fixes the reproduced external WebKit button’s false coordinate occlusion. A combined signed-main check also passed activation → element focus → one dedicated literal insertion in Ace’s actual composer, with exact Unicode/multiline draft/caret, unchanged chat history, and clipboard restoration. #141 additionally fixes actual-Ace editable coordinate focus: one normalized point click gave the Message field verified keyboard focus, left the existing draft unchanged, and preserved the empty history across worker exit/reopen. Background `desktop_key` Cmd+V, direct AX value replacement, and the other remaining cases below are not claimed fixed.
Targets were Ace-dev builds from the #104 branch: - PID 50933, window 4742, 1100x760 at (160,120) - PID 56752, window 4821, same bounds
The Message composer is a ProseMirror contenteditable, exposed as an AXTextArea "Message".
## Tested steps and results
1. **Point click, window in the background.** `desktop_click` at (0.5, 0.32), on the composer, was refused: "Point (710, 363) is occluded for the pinned target window 4742: accessibility hit-testing resolved an element in a different window of the same app." A CGWindowList check at that point showed Ace-dev window 4742 as the topmost layer-0 window. Electrobun also creates invisible strip windows (2560x30 at y=0 and 1512x33 at y=1440); none of them contains that point. 2. **`desktop_focus`, then the same point click.** Focus reported `confirmed_change`, but the click was refused again with the same occlusion error. On 56752, after `desktop_activate` reported the app active, a point click at (820, 804) on the composer was also refused as occluded. 3. **Element click on the textarea.** `desktop_click` on the AXTextArea "Message" was accepted (`accessibility_action`, `dispatched_unverified`). The next inspect still reported the AXWindow as the focused element, and the textarea had `isFocused=false`. This was seen on both PIDs, the second time right after `desktop_activate`. 4. **`desktop_select`** with `cursor_before` on the textarea returned `confirmed_no_change`, and focus stayed on the AXWindow. 5. **`desktop_type`** writing "x" returned "The accessibility value write was accepted, but its requested result could not be verified." Nothing appeared in the composer. 6. **Background Cmd+V.** Setting `AXFocused=true` on the textarea outside the tools made the next inspect report the AXTextArea as focused. `desktop_key` Cmd+V, delivered as background `window_targeted_events`, then changed nothing while Ace-dev wasn't frontmost. A plain-text clipboard didn't paste either, so this isn't specific to images. 7. **`desktop_focus` refused.** Once, while the user was working in another app, `desktop_focus` was refused with "Timeout while waiting for condition", `operation_unsupported`.
**What worked:** with Ace-dev frontmost after `desktop_activate`, setting `AXFocused` on the textarea and pressing the app's Edit > Paste menu item through Accessibility pasted reliably. Native `desktop_key` Cmd+V wasn't retried in the frontmost state because steps 2–4 never produced a focused control.
## Not tested (corrected assumption)
In Finder, `desktop_key` Cmd+C was refused: "The snapshot has no exact focused control. Click a control, then inspect the window again." The agent didn't follow that hint (click a file row, inspect again) before switching to `osascript`. Whether the native tools can copy from Finder's file list is therefore **unverified**. #103 originally listed it as a gap; the review comment there corrected that.
## Expected
- `desktop_click` on a visible web control in Electrobun windows isn't refused as occluded. - An element or point click on a WKWebView contenteditable gives it keyboard focus. - `desktop_key` delivers Cmd+V to that control, or explains why it can't.
This issue covers capability bugs; prompt guidance is in #103. Sub-issue of #8 (slices 3–5, tracked under #5). Related: #65, #68, #73. Found during #91 / PR #104.
iamnbutlerparent-issue-addedlast month on GitHub
iamnbutlerreferenced this in Avoid macOS Automation prompts when agents drive other apps #103last month on GitHub
iamnbutlerreferenced this in Attach images pasted into the composer #91last month on GitHub
iamnbutlerreferenced this in feat(ui): attach images pasted into the composer #104last month on GitHub
iamnbutlerparent-issue-removedlast month on GitHub
iamnbutlerparent-issue-addedlast month on GitHub
iamnbutlerreferenced this in [Meta] Complete native computer use in Ace #8last month on GitHub
iamnbutlerreferenced this in [Meta] Dogfood Ace end to end #5last month on GitHub
iamnbutlerreferenced this in fix(desktop): verify focus before pressing editable fields #120last month on GitHub
- IA
First focused candidate: draft #120, verified focus before `AXPress` for writable editable fields. Normal button and coordinate paths retain their behavior. It also moves the native focus write off MainActor while keeping its lane owned, and preserves ambiguous completion so it cannot fall through to a second input.
Final source `4ec5802` passed types/lint, Swift parsing, independent source review, patch application/reversal and a signed build with strict app/client signatures. Real native inventory found the fresh WebKit fixture, then `listWindows` refused with `permissionDenied`. That operation requires Accessibility, so this establishes the isolated app's missing Accessibility grant; Screen Recording was not reached or tested. No focus/input was sent, no grants were requested or changed, and the fixture/runner are stopped.
The reusable validation checkout is `desktop-app-state/ace2`, app identity `dev.ace.desktop.dev.4472239c`. Reusing that checkout across future native candidates avoids adding another app identity for every PR. The bounded fixture is ready to verify actual focus, an already-focused no-op, a normal button, stale-bounds refusal and pre-dispatch cancellation. This PR remains draft until those live checks pass. Coordinate occlusion, type/paste and the window-focus timeout in this issue remain open.
iamnbutlerreferenced this in fix(desktop): resolve containing windows for pointer input #131last month on GitHub
- IA
Merged #120 from reviewed/signed `9374d8` after green CI. Single element clicks now verify focus on editable text fields before trying AXPress, with the write off the main actor so Ace can service its own request.
Real native focus/no-op/button and pre-dispatch cancellation checks passed. A real OpenAI channel performed inspect → one focus → inspect; exact stored results/images and usage survived worker exit/reopen without another native call. The actual isolated Ace composer also changed from unfocused to focused with its default draft and empty history unchanged.
Remaining limits are explicit: the old stale-window classification conservatively reported unknown before the new focus path, first inspection of fresh WebKit still needed a later read (#64), and in-flight cancellation was not claimed. A separate one-shot self-target point-click baseline returned 'No pressable accessibility element' (no text change), so element focus does not establish coordinate clicking. #106 remains open. #122 is next for live clipboard validation; #131 is a draft containing-window resolver fix awaiting a matching reproduction and runtime proof.
- IA
Merged #131 as `39a17ad` after green CI, independent review, and a matching real WebKit reproduction. The old signed bridge reported a visible Web count button as occluded and recorded zero clicks. The patched signed bridge resolved its native containing-window link and delivered one background Accessibility action; the actual button callback changed from 0 to 1. The editor text and unrelated input/counters stayed unchanged, and all owned runtimes/fixtures are stopped.
Together with #120, this lands verified editable element focus and the external WebKit button’s coordinate-occlusion fix. It does not establish composer point-focus, background Cmd+V, or every pointer route; those #106 items remain open. #132 normal quit is now in finite live validation, and #122 plain-text clipboard resumes immediately afterward. Desktop use remains serialized.
- IA
Final combined-source dogfood check passed in the actual Ace composer. The signed `c64042b` candidate has exactly the same repository tree as merged main `2c36c49`. In one disposable channel, explicit activation followed by one native element focus and one `desktop_insert` produced the exact public Unicode/multiline draft and UTF-16 caret. Native consumption was observed, clipboard cleanup was restored, and ownership was released. Channel history stayed empty and unchanged after reopening; no message was sent.
The original clipboard was privately restored. All six owned supervisor/host/launcher/GUI/worker/vault processes are gone, the port/socket and temporary project/channel are removed, and the existing paste gate is empty. No installed app or OS grant was changed.
This verifies the active-Ace element-focus → dedicated literal-insertion workflow. It does not claim general background `desktop_key` Cmd+V, composer coordinate-focus, direct full-value `desktop_type`, or other remaining #106 cases are fixed. #8 remains open for the remaining slices.
iamnbutlerreferenced this in fix(desktop): focus editable fields at click points #141last month on GitHub
- IA
Merged #141 (`ad31506695d632174b3486910b73e187d6c9444c`): ordinary point clicks now use verified focus when the first actionable target is an editable Accessibility field. Native hit-testing runs away from the GUI main actor, and the exact target/window is checked again before mutation.
The signed `75f20cc96a253383cbae0647cea3bc29bb244919` candidate passed an actual-Ace check: one coordinate click focused the Message field, the `ace ` draft and caret stayed unchanged, and the empty channel history remained unchanged after a distinct worker exited and reopened it. No typing, paste, clipboard access or message submission was used. All owned runtimes, temporary channel/project and settings were cleaned up. Static checks, signed build and CI passed.
The earlier unknown point result and separate validation-runner startup failure are preserved in the PR evidence; neither was hidden by an input retry. Background Cmd+V and direct AX value replacement remain open in #106. This is merged on main, not a claim about the currently installed Canary.
iamnbutlerreferenced this in feat(app): add reasoning effort to the composer #148last month on GitHub
- IA
During native validation of #148, desktop_type on the Ace-dev ProseMirror Message AXTextArea returned: value write accepted but requested result could not be verified. A fresh pixel inspection confirmed the composer remained unchanged. No input replay or fallback injection was used; the effort menu itself was successfully exercised through AX clicks.
iamnbutlerreferenced this in feat(desktop): authorize coordinate actions from pixel snapshots #156last month on GitHub
- IA
Another real `desktop_focus` timeout refusal, from the #156 pixel-authority model validation (signed Ace-dev `769438a`, disposable AppKit pointer fixture). Before a planned drag, pi/Anthropic called `desktop_windows`, then `desktop_focus` with the window's unchanged target. The call returned `refused` (`dispatch_state: none`, `refusal_reason: operation_unsupported`, `retry_safety: safe`) with message "Timeout while waiting for condition" and hint "Use a window and host that support focus window". The model stopped without dragging or retrying. In an earlier run the drag was not preceded by focus: it was accepted (`dispatched_unverified`) but recorded no fixture events, consistent with the documented inactive-view limit. The focus cause is not established here.
iamnbutlerreferenced this in Finish native window, menu, and dialog workflows #73last month on GitHub
iamnbutlerreferenced this in fix(ui): add a dedicated top-edge window drag strip #190last month on GitHub
iamnbutlerreferenced this in Finish native pointer delivery and validation #68last month on GitHub
- IA
Observed again while native-verifying the queue UI for #24 in a signed source Ace-dev build (main 4d503d3 + queue changes). `desktop_inspect` exposed the Message AXTextArea as value-settable, but `desktop_type` returned unknown: "The accessibility value write was accepted, but its requested result could not be verified." A fresh pixel inspection showed the composer still contained only the Ace mention. I did not replay the input; used Ace CLI admissions for the real queued-message visual fixture instead.
iamnbutlerreferenced this in Restore queued-message visibility above the composer #193last month on GitHub
iamnbutlerreferenced this in Make the desktop window reliably draggable across its top edge #189last month on GitHub