Return Pi structured output from a terminating extension tool #196
- LA
Part of GitHub stack #202 (#181 harness).
Part of #181. This is slice 6 of the stacked harness PRs, on top of #195 (`feat/harness-5-pi-e2e`). It replaces the Pi structured-output workaround from #195 with Pi's supported extension pattern, following Pi's [structured-output example](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/examples/extensions/structured-output.ts). It resolves the `setTimeout(0)` caveat listed in #195.
## What changes
- **Terminating extension tool instead of a host tool.** `@ai-sdk/harness-pi` 1.0.128 rejects `responseFormat`, so #195 gave Pi a host result tool and answered it from Chopin one event-loop turn later, after Pi registered the pending result. The wrapper now registers `chopin_submit_pi_result` through the adapter's public `extensionFactories` setting. Its `execute` runs inside Pi and returns `{ details, terminate: true }`, so Pi ends the turn on that call. That means: - no host round trip, no `submitToolResult`, and no timing assumption; - one fewer model request per summary or research turn. - **Offered only on structured turns.** Each Pi session gets its own `createPi()` settings with a per-session extension state. Before each turn, the session wrapper sets that state from the turn's `responseFormat`. On `before_agent_start` the extension reads the state and calls `setActiveTools` to add or remove the tool, and a `tool_call` hook blocks the tool on every other turn, including before the first turn. Prompt text never switches the tool on: an earlier version matched the system prompt, which quotes earlier chat (fixed in `89c13a7` after review). Chopin never resumes a Pi session, so giving up in-process reattach across `doStart` calls is safe. The wrapper also keeps the result tool's parts out of the outer stream, so the Planner's boundary check never sees them. - **Schema in the instructions.** The extension tool is registered once per session, so its parameters are a generic object. Each structured turn's instructions carry that turn's JSON schema, and `HarnessAgent` still validates the parsed output. - **Built-in filtering unchanged.** With an inline extension, the adapter switches Pi to `noTools: 'builtin'`. Pi's native tools stay off, and the contract suite still passes its host-tools-only and no-built-ins checks. - **Dependencies.** `@earendil-works/pi-coding-agent` 0.85.1 (for the extension types) and `typebox` 1.3.7 are now pinned in the `harness` catalog. The lockfile already resolved both; the lock diff only re-hoists existing versions. - **Docs.** `docs/self-hosting.md` describes the extension approach in place of the timing caveat.
The Pi `AGENTS.md` caveat from #195 is unchanged.
## Verification
The real Pi agent loop (`createPi()` against a local OpenAI-completions stub) now proves three things: 1. A structured turn makes exactly one model request, and that request offers the result tool. 2. A plain turn's request doesn't offer it. 3. A result-tool call on a plain turn is blocked, the model continues to its answer, and no result-tool parts reach the stream.
Run against the previous adapter, (1) failed with 2 requests instead of 1, and (3) failed with 3 leaked parts.
Assistant-model: Claude Opus 5.5 Assistant-workflow: inline Assistant-duration: 10m converged, estimated 1-2h Assistant-verification: Bun Pi contract passed: 6/6 against the real createPi() agent loop and a local model stub Assistant-verification: Bun Pi contract red check passed: against the previous adapter, a structured turn made 2 model requests instead of 1, and a plain-turn result-tool call leaked 3 parts Assistant-verification: Bun Pi adapter unit passed: 9/9, including extension gating and blocking with a fake ExtensionAPI Assistant-verification: bun run ci passed: dprint, oxlint, tokens Assistant-verification: bun run types passed: all workspaces and E2E Assistant-verification: bun test passed: 1598 pass, 2 PostgreSQL skips, 0 fail Assistant-verification: Playwright E2E passed: CI=1 locked bun run e2e with no mise on PATH, 229 passed and 2 flaky that passed on retry (responsive-workspace.e2e.ts:110, :186) User-preference: Prefer an upstream-supported extension point over working around third-party adapter races Co-authored-by: @lavaman131
lavaman131mentionedlast month on GitHub
lavaman131subscribedlast month on GitHub
lavaman131added-to-stacklast month on GitHub
lavaman131ready-for-reviewlast month on GitHub
lavaman131referenced this in Add the Pi harness adapter and E2E harness coverage #195last month on GitHub
- MAapps/server/src/harness/pi/adapter.ts:676a8fde0
[Earlier chat messages are included in that prompt](https://github.com/githubnext/chopin/blob/6a8fde021f02f924666a205d729e686d4b43c095/apps/server/src/chat/service.ts#L815-L831). If someone copies the sentence into chat, Pi could get the ‘submit result’ tool during a normal reply and stop before answering. [Those tool calls are hidden here](https://github.com/githubnext/chopin/blob/6a8fde021f02f924666a205d729e686d4b43c095/apps/server/src/harness/pi/adapter.ts#L113-L119), so it would be hard to spot. Could we add a test with the sentence in earlier chat?
- MA
A sentence in earlier chat can switch on Pi’s special ‘submit result’ tool during a normal reply. Chopin should switch it on directly, only when that turn needs it.
lavaman131head-ref-force-pushed 89c13a7a3aedlast month on GitHub
- LA
No review summary.
- LAapps/server/src/harness/pi/adapter.ts:676a8fde0
Good catch, confirmed: `plannerInstructions()` puts the chat backscroll into Pi's system prompt, so quoting the sentence enabled the tool. Fixed in 89c13a7. Each Pi session now gets its own extension state, and the session wrapper sets it from the turn's `responseFormat` before the turn starts. Prompt text no longer affects it.
Added the test you asked for: `pi.contract.test.ts` runs a plain turn through the real Pi loop with `PI_RESULT_INSTRUCTION` quoted as earlier chat and checks that the tool isn't offered and the turn still answers. Against the old adapter it fails with an empty reply (the turn ended on the hidden tool call). There's also a unit test for the per-turn state. I rebased onto #195's d330c0f.
Assistant-workflow: inline Assistant-verification: bun test contract passed: plain turn with the instruction in earlier chat is answered normally and never offered the tool Assistant-verification: red check passed: same test fails against the previous adapter Assistant-verification: bun test passed: 1605 pass, 2 skip, 0 fail; bun run types and bun run ci passed
lavaman131referenced this in Move workers to structured output and add harness deployment modes #194last month on GitHub
lavaman131referenced this in Share worker session setup and teardown between background jobs #198last month on GitHub
lavaman131removed-from-stacklast month on GitHub
lavaman131base-ref-force-pushedlast month on GitHub
lavaman131head-ref-force-pushed fadb3e69b287last month on GitHub
lavaman131added-to-stacklast month on GitHub
lavaman131review-dismissedlast month on GitHub
lavaman131merged 4cf49aae7897last month on GitHub
lavaman131head-ref-deletedlast month on GitHub