githubnext/chopinPublic

Let's compose rich plans together

Add the AI SDK harness contract suite #186

Mergedlavaman131merged 2 commits intomainfromrefs/pull/186/head
  1. LA

    Part of GitHub stack #202 (#181 harness).

    Part of #181. This is slice 1 of 5 stacked PRs for the pluggable agent harness RFC. It covers RFC phase 1 (spike) and phase 2 (contract suite and fake). Phase 9, the author's own runtime adapter, is deferred.

    Based on `main` (#185, the Bun 1.4.2 upgrade it was stacked on, has merged). This diff contains only the slice 1 commits.

    ## What changes

    - **Pinned harness dependencies.** A new `harness` catalog in the root `package.json` pins `@ai-sdk/harness` 1.0.126, `@ai-sdk/mcp` 2.0.60, `ai` 7.0.116 and `zod` 4.4.3. `apps/server` declares them, and `bun.lock` is updated. Nothing imports them at runtime yet. The Copilot SDK adapter in slice 3 is the first runtime consumer. - **`apps/server/src/harness/contract.ts`.** `harnessContract(name, createHarness, createSandboxSession?)` is a reusable `bun:test` suite. It drives any `HarnessV1` through `HarnessAgent` and asserts four guarantees: - **Active tool set:** only the host tools are active. The suite checks the adapter's built-in filtering declaration and the tool names it actually emits, not scripted call IDs. An adapter with no built-ins passes. - **Structured output:** `output` returns an object that matches the schema. Invalid output fails. - **Abort:** aborting stops the turn. - **Destroy:** only a successful `doDestroy` counts as destroying the session. - **`apps/server/src/harness/contract.test.ts`.** Runs the suite against scripted fake `HarnessV1`s, one that filters built-ins and one with no built-ins (6/6 pass). Each guarantee also has a negative variant (approvals-only, leaky built-in, invalid output, ignored abort, and a destroy that throws), and each one fails its targeted assertion. Later adapters (Copilot SDK in slice 3, Pi in slice 5) run this same suite.

    Commits: - `b6dc496` Add AI SDK harness contract suite - `af3d549` Fix harness contract boundary checks (review round: accept adapters with no built-ins, and assert on filtering declarations and emitted tool names instead of call IDs)

    ## Phase 1 spike results

    The throwaway scripts ran outside the repository and are not committed. Findings:

    1. `@ai-sdk/harness` and `@ai-sdk/mcp` install and run on Bun 1.4.2, which supersedes the RFC's 1.3.2. A fake turn, structured output, abort and destroy all worked. 2. Pi with only host tools active: `doStart` received `builtinToolFiltering: {mode: "allow", toolNames: []}`. No real model turn ran. Slice 5 proves runtime visibility. 3. GitHub remote MCP `tools({ schemas })` returned exactly `list_pull_requests,pull_request_read` with a gh OAuth user token. Still open for later slices: the GitHub App user token path and an account without Copilot. Neither is proven yet. 4. A host-process Copilot SDK stub implemented `HarnessV1.doStart` and mapped deltas, idle, abort and disconnect. The live transport is slice 3 work.

    One version note: `HarnessAgentSettings.sandbox` is deprecated in 1.0.126, so the suite uses `createSession({ sandboxSession })`. None of the results invalidates an RFC decision.

    ## Temporary pieces

    None. The contract suite and pins are permanent. Later slices add adapters that run against the suite.

    ## Interactive Edge verification

    - After `b6dc496`, I signed in to a local Chopin build in Microsoft Edge with agent-browser, opened the `golden-wind` document and sent `@chopin` a message. The Planner replied "Hello!" (screenshot `planner-reply.png`). - After `af3d549`, a new Planner turn could not run because the GitHub Next SSO authorization gate blocked the OAuth flow (screenshot `review-sso-gate.png`). An 85-second recording captured the gate, not a Planner reply. The review fix does not touch runtime code, so the earlier reply is the only interactive evidence.

    Evidence: the Planner answering `@chopin` in Edge on this branch.

    ![Planner replies Hello! to @chopin in the golden-wind document](https://github.com/user-attachments/assets/023cf0f8-afc9-42b7-9846-f4193addd379)

    Assistant-model: GPT-6 Sol, Claude Opus 5.5 Assistant-workflow: harness-stack (run f6f676b9-6506-489b-93e2-2616ea55c6a8 announced over Intercom) Assistant-duration: 46m converged, estimated 50m Assistant-verification: bun run ci passed: $ dprint check && oxlint && bun scripts/check-tokens.ts | Found 0 warnings and 0 errors. | Finished in 80ms on 486 files with 120 rules using 16 threads. | tokens ok — 227 defined across 21 stylesheets Assistant-verification: bun run types passed: @chopin/visuals types: Exited with code 0 | @chopin/editor types: Exited with code 0 | @chopin/server types: Exited with code 0 | @chopin/web types: Exited with code 0 Assistant-verification: bun test passed: 2 skip | 0 fail | 6447 expect() calls | Ran 1531 tests across 173 files. [10.65s] Assistant-verification: bun run e2e passed: Container chopin-harness-1-db-e2e-fixtures-1 Removing | Container chopin-harness-1-db-e2e-1 Removing | Container chopin-harness-1-db-e2e-1 Removed | Container chopin-harness-1-db-e2e-fixtures-1 Removed Assistant-verification: Bun contract unit passed: 6/6 across filtered and no-builtins fakes Assistant-verification: Bun contract negative variants passed: approvals-only, leaky-builtin, invalid-output, ignored-abort, and destroy-throws each failed their targeted guarantee Assistant-verification: agent-browser Edge manual scenario passed: @chopin reply "Hello!" on golden-wind at b6dc496; screenshot planner-reply.png kept locally Assistant-verification: agent-browser Edge manual scenario unavailable: GitHub Next SSO authorization gate blocked a new Planner turn at af3d549; screenshot review-sso-gate.png kept locally Assistant-verification: agent-browser recording passed: 85.3s webm captures the OAuth SSO gate, not a Planner reply Assistant-verification: GitHub MCP spike passed: gh OAuth user token tools({ schemas }) returned list_pull_requests,pull_request_read (App token and non-Copilot account unverified) Co-authored-by: @lavaman131

  2. chopinmentionedlast month on GitHub

  3. chopinsubscribedlast month on GitHub

  4. lavaman131mentionedlast month on GitHub

  5. lavaman131subscribedlast month on GitHub

  6. lavaman131referenced this in Port Chopin tools to AI SDK tools #187last month on GitHub

  7. lavaman131added-to-stacklast month on GitHub

  8. lavaman131removed-from-stacklast month on GitHub

  9. lavaman131added-to-stacklast month on GitHub

  10. lavaman131convert-to-draftlast month on GitHub

  11. lavaman131referenced this in Run the Planner on HarnessAgent with a Copilot SDK adapter #192last month on GitHub

  12. lavaman131referenced this in Move workers to structured output and add harness deployment modes #194last month on GitHub

  13. lavaman131referenced this in Add the Pi harness adapter and E2E harness coverage #195last month on GitHub

  14. lavaman131ready-for-reviewlast month on GitHub

  15. MaggieAppletonautomatic-base-change-failedlast month on GitHub

  16. MA

    No review summary.

  17. lavaman131removed-from-stacklast month on GitHub

  18. lavaman131head-ref-force-pushed 8701a31c440blast month on GitHub

  19. lavaman131base-ref-changedlast month on GitHub

  20. lavaman131added-to-stacklast month on GitHub

  21. lavaman131merged 5c1433d008b5last month on GitHub

  22. lavaman131closedlast month on GitHub

  23. lavaman131head-ref-deletedlast month on GitHub