Explain native inspection failure when the desktop or target window becomes unavailable
iamnbutler opened ·
## Status (Oct 6)
Shipped: pixels-mode inspection and target availability reporting (#71), and typed `target_unavailable` refusal for a known locked session with unlock-and-refresh guidance (#87, [details](https://github.com/githubnext/ace2/issues/64#issuecomment-5995277331)). Remaining: a reliable distinction and recovery path for an offscreen/other-desktop target versus an unavailable desktop versus incomplete Accessibility, sparse initial WKWebView Accessibility trees (seen in [#73](https://github.com/githubnext/ace2/issues/73#issuecomment-6026757352)), and the real recovery check below. The Chrome window-title mismatch is tracked separately in #113.
## Original report
During real desktop-action dogfooding for #8 / #63, native inspection became unavailable after previously passing against the same live Ace-dev process and window. The agent received an incomplete Accessibility-tree error without an actionable explanation of target availability.
Observed on the signed macOS development app:
- The real local pi click/type/key/button workflow and delivered-input Stop/restart checks had passed. - A subsequent local Workers/Durable Objects run passed its seed native inspection, then two model `desktop_inspect` calls failed with `DESKTOP_ERROR: Warning: AX tree incomplete at incomplete accessibility read`. - Fresh native inventory still returned the same process instance/window and bounds, with `is_minimized: false`, but `is_on_screen: false`, `observation_capability: pixels_only`, and `observation_reason: no_matching_accessibility_window`. Inventory itself reported complete. - Codex computer use could still read and capture that test window. Its Raise and Ace's Show Ace action did not restore native availability in the subsequent inventory. - No mutation ran in the failed hosted workflow. No evidence establishes whether a Space/desktop/controlled-surface change or another macOS condition caused the availability change; this is not yet a confirmed hosted-transport defect.
Expected: give the agent a bounded, useful explanation when native window/Accessibility availability changes, retain the exact target, and require a fresh observation when the target is available again. Do not silently switch to another window, steal focus, or replay uncertain input as recovery.
Investigate which native state reliably distinguishes an offscreen/other-desktop target, a locked/unavailable desktop, and incomplete Accessibility. Add a real recovery check under #8's visual targeting and longer-workflow slice. Keep the unavailable-desktop limitation explicit until that check passes.
iamnbutlerreferenced this in feat(desktop): add targeted native actions #63last month on GitHub
iamnbutlerreferenced this in [Meta] Dogfood Ace end to end #5last month on GitHub
iamnbutlerreferenced this in [Meta] Complete native computer use in Ace #8last month on GitHub
- IA
The hosted connection and history checks are now unblocked using a second real, signed AppKit form: a pi/Anthropic run clicked/typed/pressed a key/submitted it, changed its persisted value, and replayed all results/images after Workers restart. PR #63 is merged.
The original Ace-dev availability problem is still open. Restarting only that disposable app produced a brief visible window with five chrome-only AX elements, followed by another offscreen/no-matching-Accessibility-window inventory. The second app remained available long enough to complete the hosted workflow. This narrows the validation failure but does not establish the cause or prove recovery for Ace-dev. No activation or alternate-window fallback was added.
- IA
Further #65 validation reproduced the same availability failure in a standalone, signed AppKit form, so this is not established as an Ace WKWebView-specific problem. Six real checks passed, including contextual selection and Unicode/multiline insertion. A subsequent focus click completed as confirmed-no-change, but its fresh observation and the next explicit inspection both failed with the incomplete Accessibility read.
Afterward, inventory still found the same process/window with `is_minimized: false`, `is_on_screen: false`, `pixels_only`, and `no_matching_accessibility_window`. One bounded read-only retry failed too; no keyboard action was retried. Native keyboard validation is paused at that availability failure while checking the desktop state. The cause remains unproven.
iamnbutlerreferenced this in Add literal desktop insertion with reserved clipboard ownership #67last month on GitHub
iamnbutlerreferenced this in Validate native text insertion on hosted channels and under interruption #65last month on GitHub
iamnbutlerreferenced this in Do not reject editable text areas when AXEnabled is absent #66last month on GitHub
- IA
The draft keyboard PR is #67. A fresh disposable AppKit window with `canJoinAllSpaces` and `fullScreenAuxiliary` also failed validation, this time because the captured target no longer matched the resolved generation/window/owner/bounds. The run stopped before shortcut input; no identity check was relaxed and no input was retried. That setup changes only test-window presentation, not Ace's production behavior, and does not establish the cause or recovery.
For the earlier unavailable form, fresh inventory showed the app was not hidden. The console was unlocked and display sleep was prevented when checked. Those observations do not explain the offscreen/Accessibility transition. Remaining runtime checks for #67 are explicitly pending in #8 and #65.
iamnbutlerreferenced this in Add native targeted scrolling #70last month on GitHub
iamnbutlerreferenced this in Finish native pointer delivery and validation #68last month on GitHub
iamnbutlerreferenced this in Inspect native pixels and report target availability #71last month on GitHub
- IA
Draft implementation: #71, stacked on #67.
Added explicit read-only `desktop_inspect` pixels mode with exact process/window-generation and capture-digest verification. It returns no reusable action snapshot, element IDs, or fabricated Accessibility metadata. Native inspection failures preserve their original error and add a bounded later app/window inventory; those later readings are context, not proof of the failure's cause. Completed input retains its outcome when the following observation fails.
Real signed-client validation against the current disposable AppKit fixture passed normal Accessibility inspection, pixel capture with no action authority, exact generation checks, invalid empty-mode refusal, and rejection of a window absent from the live inventory with its original error and complete inventory diagnostics preserved. Type checks, formatting/lint, signed desktop build, signature verification, and Ace diff review also passed.
This does not yet establish why living windows intermittently become offscreen or lose their Accessibility tree. The native passive capture path already retries a receipt mismatch once. Keep this issue open; the draft still needs the completed-input/failed-reinspection case and broader model flow. Explicit app/window recovery and canvas mutation support remain separate #8 work.
iamnbutlerreferenced this in Activate apps and recover exact windows #74last month on GitHub
- IA
New dogfooding evidence from the development Ace app: a native inspection returned a screenshot of the full interface but only five Accessibility nodes (the window, group, and three window-chrome buttons). This was a sparse successful observation, distinct from the earlier incomplete-read errors.
The exact target remained PID `7944`, window `2533`, process generation `1791170764047349`. An independent, read-only CUA `getApp(path)` inspection then exposed the full WKWebView tree without sending input. The next native inspection of that same target returned 68 AX nodes plus nine OCR text labels; `Message` was a real `AXTextArea`, not an OCR-derived editable control. There was no rebuild or process/window restart between these observations.
Local evidence:
- Before: `/tmp/ace-actions-validation-xaLuQq/keyboard-webview-Xlf1Xh/observation.json` and `screenshot.jpg`. - After: `/tmp/ace-actions-validation-xaLuQq/keyboard-webview-TnvO1M/observation.json`.
This sequence does not prove that CUA was necessary or caused the recovery. WebKit's lazy Accessibility enablement or remote Accessibility token initialization is a candidate explanation for this subtree symptom. Compare repeated native-only reads on a fresh process before concluding causation; this also does not explain all of the earlier standalone AppKit failures. PR #71 provides diagnostics and explicit read-only pixel inspection, not a fix for this missing subtree.
Separately, a native AX click on `Message` truthfully returned `dispatched_unverified` but did not focus the field. CUA focus is explicit setup for the ongoing keyboard-only checks; it does not validate Ace's complete pointer-to-keyboard flow. That pointer work remains pending in #70 and should stay distinct from the subtree-availability investigation. Keep #64 open.
iamnbutlerreferenced this in Preserve literal newlines when inserting text into web composers #76last month on GitHub
- IA
Controlled native-only follow-up for the sparse WKWebView subtree: after a fresh development GUI launch, the validation used only native application/window inventory and native inspection. The first inspection returned five AX nodes; the second native-only inspection returned 74 AX nodes plus nine OCR labels, including a real `Message` text field. No CUA app read or other external AX inspection ran between the fresh launch and these two observations.
Both observations identify PID `20701`, window `2679`, process generation `1791197308099437`. Local evidence:
- First: `/tmp/ace-actions-validation-xaLuQq/keyboard-webview-a1GJnR/observation.json`. - Second: `/tmp/ace-actions-validation-xaLuQq/keyboard-webview-tNcEkQ/observation.json`.
This reproduced recovery shows that an external CUA read was not necessary in this case. It does not establish why the first subtree was sparse, prove a general recovery strategy, or fix the earlier incomplete/offscreen AppKit failures. The build was the reviewed `d04c8c4` keyboard parent plus the uncommitted #76 candidate, so this is runtime evidence rather than a clean release validation. No input was retried as part of these two observations. PR #71 remains diagnostics/read-only pixel support; keep #64 open.
- IA
PR #71 is now a focused change on the merged application-state and selection/shortcut base, independent of unfinished literal insertion. It adds explicit pixel inspection without action authority and bounded later inventory diagnostics while preserving the original inspection error and any completed action outcome. Its diagnostic projection also retains `is_active_known` from #79.
The current candidate passes type checks, formatting/lint, its own signed desktop build, deep/strict signature verification, and diff checks. A fresh read-only native helper is ready; current-candidate runtime follow-ups are still pending. This does not establish or fix the cause of sparse/incomplete Accessibility observations, so this issue remains open.
- IA
Merged #71 on `feat/github-cache` as `61c5aa9`.
The independent signed build passed real Accessibility/pixel reads, exact target checks, invalid-mode rejection, and absent-window original-error diagnostics. A real AppKit button closed its target exactly once; the completed action and native receipt survived the failed follow-up inspection. Later inventory retained the closed window as offscreen/pixels-only, so the verifier was corrected against saved evidence without repeating input.
Pixel reads intentionally provide no action snapshot. This adds useful inspection feedback; it does not fix sparse webview Accessibility trees or complete the broader visual workflow. No new canary has been published.
iamnbutlerreferenced this in Explain locked desktop management refusals #87last month on GitHub
- IA
Landed focused locked-desktop feedback in [PR #87](https://github.com/githubnext/ace2/pull/87), merge `c89f87ba0450a6b05455da914f2909739ebeda3d` on `feat/github-cache`.
A known locked session now produces typed `target_unavailable`, no-dispatch evidence, and guidance to unlock then refresh inventory before choosing another action. This fixes the observed focus failure that previously looked like an unsupported Accessibility operation. Request-shape checks still precede the lock preflight; false/unknown lock state follows the existing path.
Static checks, independent review, isolated signed-client build/signature verification, and real locked-session activate/focus/restore checks passed. All three returned `dispatch_state: none` and kept the disposable receiver unchanged. The requests deliberately used stale process generations, preventing input even if the desktop unlocked between observation and execution. This verifies the real explicit-locked branch, not unlocked/unknown or mid-operation lock behavior. No desktop bundle was replaced or launched for this check.
This broader issue remains open for its remaining scope; #8 tracks subsequent focused landings.
iamnbutlerreferenced this in Land desktop inspection/actions and tab API from feat/github-cache #89last month on GitHub
- IA
Observed while validating #109 in Ace Canary (2026-10-05, about 16:20Z). `desktop_inspect` in accessibility mode on Google Chrome (pid 2175) window 474 failed twice in a row:
``` DESKTOP_ERROR: The desktop observation provider returned inconsistent response evidence: window-context window title.. ```
Target availability, read after the failure, reported the window as present, on screen, key, not minimized, and `combined_eligible`, with its app not active. Just before, `desktop_windows` listed the window with a truncated active-tab title, `Add JSDoc comments to all expor… #5 · iamnbutler/tasks-fixture`. The tab had changed moments earlier: the window's title had moved through several tasks-fixture pages in the preceding minute. A title that changes, or is truncated differently between the window inventory and the Accessibility context, may explain the mismatch. That is unconfirmed. No input was dispatched and no snapshot was returned, so the agent couldn't verify the tab title before Cmd+W, and closing the test tabs needed a manual exception (see the Built in Ace section of #109). Not investigated further.
iamnbutlerreferenced this in feat(app): open GitHub items on GitHub or in a new channel #109last month on GitHub
iamnbutlerreferenced this in Open GitHub items on GitHub or in a new channel from the Issues and PRs lists #107last month on GitHub
iamnbutlerreferenced this in fix(desktop): verify focus before pressing editable fields #120last month on GitHub
iamnbutlerreferenced this in Focus and paste into WebKit composers with native desktop tools #106last month on GitHub
iamnbutlerreferenced this in fix(desktop): resolve containing windows for pointer input #131last month on GitHub
iamnbutlerparent-issue-addedlast month on GitHub