Add the AI SDK harness contract suite #186
- LA
Part of GitHub stack #202 (#181 harness).
Part of #181. This is slice 1 of 5 stacked PRs for the pluggable agent harness RFC. It covers RFC phase 1 (spike) and phase 2 (contract suite and fake). Phase 9, the author's own runtime adapter, is deferred.
Based on `main` (#185, the Bun 1.4.2 upgrade it was stacked on, has merged). This diff contains only the slice 1 commits.
## What changes
- **Pinned harness dependencies.** A new `harness` catalog in the root `package.json` pins `@ai-sdk/harness` 1.0.126, `@ai-sdk/mcp` 2.0.60, `ai` 7.0.116 and `zod` 4.4.3. `apps/server` declares them, and `bun.lock` is updated. Nothing imports them at runtime yet. The Copilot SDK adapter in slice 3 is the first runtime consumer. - **`apps/server/src/harness/contract.ts`.** `harnessContract(name, createHarness, createSandboxSession?)` is a reusable `bun:test` suite. It drives any `HarnessV1` through `HarnessAgent` and asserts four guarantees: - **Active tool set:** only the host tools are active. The suite checks the adapter's built-in filtering declaration and the tool names it actually emits, not scripted call IDs. An adapter with no built-ins passes. - **Structured output:** `output` returns an object that matches the schema. Invalid output fails. - **Abort:** aborting stops the turn. - **Destroy:** only a successful `doDestroy` counts as destroying the session. - **`apps/server/src/harness/contract.test.ts`.** Runs the suite against scripted fake `HarnessV1`s, one that filters built-ins and one with no built-ins (6/6 pass). Each guarantee also has a negative variant (approvals-only, leaky built-in, invalid output, ignored abort, and a destroy that throws), and each one fails its targeted assertion. Later adapters (Copilot SDK in slice 3, Pi in slice 5) run this same suite.
Commits: - `b6dc496` Add AI SDK harness contract suite - `af3d549` Fix harness contract boundary checks (review round: accept adapters with no built-ins, and assert on filtering declarations and emitted tool names instead of call IDs)
## Phase 1 spike results
The throwaway scripts ran outside the repository and are not committed. Findings:
1. `@ai-sdk/harness` and `@ai-sdk/mcp` install and run on Bun 1.4.2, which supersedes the RFC's 1.3.2. A fake turn, structured output, abort and destroy all worked. 2. Pi with only host tools active: `doStart` received `builtinToolFiltering: {mode: "allow", toolNames: []}`. No real model turn ran. Slice 5 proves runtime visibility. 3. GitHub remote MCP `tools({ schemas })` returned exactly `list_pull_requests,pull_request_read` with a gh OAuth user token. Still open for later slices: the GitHub App user token path and an account without Copilot. Neither is proven yet. 4. A host-process Copilot SDK stub implemented `HarnessV1.doStart` and mapped deltas, idle, abort and disconnect. The live transport is slice 3 work.
One version note: `HarnessAgentSettings.sandbox` is deprecated in 1.0.126, so the suite uses `createSession({ sandboxSession })`. None of the results invalidates an RFC decision.
## Temporary pieces
None. The contract suite and pins are permanent. Later slices add adapters that run against the suite.
## Interactive Edge verification
- After `b6dc496`, I signed in to a local Chopin build in Microsoft Edge with agent-browser, opened the `golden-wind` document and sent `@chopin` a message. The Planner replied "Hello!" (screenshot `planner-reply.png`). - After `af3d549`, a new Planner turn could not run because the GitHub Next SSO authorization gate blocked the OAuth flow (screenshot `review-sso-gate.png`). An 85-second recording captured the gate, not a Planner reply. The review fix does not touch runtime code, so the earlier reply is the only interactive evidence.
Evidence: the Planner answering `@chopin` in Edge on this branch.

Assistant-model: GPT-6 Sol, Claude Opus 5.5 Assistant-workflow: harness-stack (run f6f676b9-6506-489b-93e2-2616ea55c6a8 announced over Intercom) Assistant-duration: 46m converged, estimated 50m Assistant-verification: bun run ci passed: $ dprint check && oxlint && bun scripts/check-tokens.ts | Found 0 warnings and 0 errors. | Finished in 80ms on 486 files with 120 rules using 16 threads. | tokens ok — 227 defined across 21 stylesheets Assistant-verification: bun run types passed: @chopin/visuals types: Exited with code 0 | @chopin/editor types: Exited with code 0 | @chopin/server types: Exited with code 0 | @chopin/web types: Exited with code 0 Assistant-verification: bun test passed: 2 skip | 0 fail | 6447 expect() calls | Ran 1531 tests across 173 files. [10.65s] Assistant-verification: bun run e2e passed: Container chopin-harness-1-db-e2e-fixtures-1 Removing | Container chopin-harness-1-db-e2e-1 Removing | Container chopin-harness-1-db-e2e-1 Removed | Container chopin-harness-1-db-e2e-fixtures-1 Removed Assistant-verification: Bun contract unit passed: 6/6 across filtered and no-builtins fakes Assistant-verification: Bun contract negative variants passed: approvals-only, leaky-builtin, invalid-output, ignored-abort, and destroy-throws each failed their targeted guarantee Assistant-verification: agent-browser Edge manual scenario passed: @chopin reply "Hello!" on golden-wind at b6dc496; screenshot planner-reply.png kept locally Assistant-verification: agent-browser Edge manual scenario unavailable: GitHub Next SSO authorization gate blocked a new Planner turn at af3d549; screenshot review-sso-gate.png kept locally Assistant-verification: agent-browser recording passed: 85.3s webm captures the OAuth SSO gate, not a Planner reply Assistant-verification: GitHub MCP spike passed: gh OAuth user token tools({ schemas }) returned list_pull_requests,pull_request_read (App token and non-Copilot account unverified) Co-authored-by: @lavaman131
chopinmentionedlast month on GitHub
chopinsubscribedlast month on GitHub
lavaman131mentionedlast month on GitHub
lavaman131subscribedlast month on GitHub
lavaman131referenced this in Port Chopin tools to AI SDK tools #187last month on GitHub
lavaman131added-to-stacklast month on GitHub
lavaman131removed-from-stacklast month on GitHub
lavaman131added-to-stacklast month on GitHub
lavaman131convert-to-draftlast month on GitHub
lavaman131referenced this in Run the Planner on HarnessAgent with a Copilot SDK adapter #192last month on GitHub
lavaman131referenced this in Move workers to structured output and add harness deployment modes #194last month on GitHub
lavaman131referenced this in Add the Pi harness adapter and E2E harness coverage #195last month on GitHub
lavaman131ready-for-reviewlast month on GitHub
MaggieAppletonautomatic-base-change-failedlast month on GitHub
- MA
No review summary.
lavaman131removed-from-stacklast month on GitHub
lavaman131head-ref-force-pushed 8701a31c440blast month on GitHub
lavaman131base-ref-changedlast month on GitHub
lavaman131added-to-stacklast month on GitHub
lavaman131merged 5c1433d008b5last month on GitHub
lavaman131head-ref-deletedlast month on GitHub