githubnext/ace2Public

Ace: a collaborative coding environment on the team's own machines

Inspect native pixels and report target availability #71

Mergediamnbutlermerged 1 commit intofeat/github-cachefromrefs/pull/71/head
  1. IA

    Failed native window inspections retain their original error and add a bounded, later read of target availability. Explicit `desktop_inspect` pixels mode returns a screenshot with exact target and image-digest checks, without a reusable action snapshot. Failed reinspection also preserves an already completed action's outcome.

    Relates to #64 and #8. These diagnostics do not establish or fix the cause of intermittent offscreen or incomplete Accessibility observations.

    This focused PR is based on merged application-state and selection/shortcut changes (`9418c8e`); it no longer depends on #67. Availability diagnostics retain the new `is_active_known` flag.

    Validation: type checks, formatting/lint, the independent signed desktop build, deep/strict signature verification, diff checks, and independent source review pass. Fresh native calls against that signed candidate passed exact app/window generation, Accessibility and pixel inspection, absent-window original-error diagnostics, unknown-state flags, and invalid-mode rejection. Pixel results have a screenshot but no action snapshot, element IDs, or detection metadata. Ace's initial webview AX tree was sparse; this does not claim that separate issue is fixed.

    A real AppKit Close Alpha button ran exactly once and closed its target while another fixture window remained open. The returned action retained its original completed outcome/native receipt and separately reported the failed follow-up inspection. Callback evidence confirmed the actual close; no input was replayed. The initial verifier incorrectly expected the closed window to disappear from inventory; macOS retained an offscreen pixels-only row. Rechecking the saved result against the correct contract passed without repeating the click. Existing pi result/history transport is unchanged; broader visual workflows remain tracked in #8.

    A real pi/Anthropic run using this PR's desktop tool schema and host adapter passed exactly three read-only inspections: pixels, Accessibility, and an absent window. The stored tool results retained the native text, screenshots, and error flags exactly. Reopening the same pi SQLite store returned identical results and images with unchanged model usage and no repeated native calls. Evidence: `/tmp/ace-availability-model-odVTLq/result.json`.

    ### Built in Ace

    Ace's terminal ran installation, builds, checks, Git/GitHub operations, and prior native validation. Codex coordinated and read/edited source while Ace's external-harness support (#9) and file editor (#22) remain unfinished. Native checks ran through Ace’s terminal against the isolated signed app. External CUA was used only to quit the prior validation build and launch the disposable AppKit fixture while Ace’s application-lifecycle support remains incomplete under #8.

  2. iamnbutlerreferenced this in Explain native inspection failure when the desktop or target window becomes unavailable #64last month on GitHub

  3. iamnbutlerreferenced this in [Meta] Complete native computer use in Ace #8last month on GitHub

  4. iamnbutlerreferenced this in Validate native text insertion on hosted channels and under interruption #65last month on GitHub

  5. iamnbutlerhead-ref-force-pushed a99b28421aa1last month on GitHub

  6. iamnbutlerbase-ref-changedlast month on GitHub

  7. iamnbutlerready-for-reviewlast month on GitHub

  8. iamnbutlermerged 61c5aa9b1d54last month on GitHub

  9. iamnbutlerclosedlast month on GitHub

  10. IA
    Author

    The real pi/Anthropic inspection smoke passed: pixels, Accessibility, and absent-window diagnostics. Exact tool text, images, and error flags survived reopening pi's store with unchanged model usage and no repeated native calls. Evidence: `/tmp/ace-availability-model-odVTLq/result.json`. Validation details are recorded in the PR body and #8.

  11. iamnbutlerreferenced this in Activate apps and recover exact windows #74last month on GitHub

  12. iamnbutlerreferenced this in Finish native window, menu, and dialog workflows #73last month on GitHub

  13. iamnbutlerreferenced this in Land desktop inspection/actions and tab API from feat/github-cache #89last month on GitHub