# Working on Chopin Chopin is an experimental collaborative authoring system: several people and a hosted agent share one rich, repository-connected document. Plans are one document workflow, not the product boundary. Read [README.md](README.md) for the product framing and [Architecture](docs/architecture.md) before changing cross-package behavior. Before changing UI or interface copy, read the [design guidance](apps/web/DESIGN.md). Design jigs are temporary local artifacts. Keep them out of commits and pull requests. The most useful technical references are: - [Authentication and authorization](docs/authentication.md) - [Repository channels](docs/channels.md) - [Storage and persistence](docs/storage.md) - [Hosted agent (Planner)](docs/hosted-agent.md) - [Background jobs and workers](docs/background-jobs.md) - [Experimental implementation lifecycle](docs/implementation-lifecycle.md) - [Self-hosting](docs/self-hosting.md) ## Commands ```bash bun install # install the workspace bun run dev # Vite and Bun development supervisor bun run dev:exe # exe.dev proxy and HMR mode bun run db:up # start the local PostgreSQL service bun run db:down # tear down the local Compose project bun run migrate # apply PostgreSQL migrations bun test # unit, domain, and memory-adapter tests bun run test:postgres # PostgreSQL contract and lifecycle tests bun run e2e # Chromium system integration suite bun run e2e:ui # Playwright UI mode bun run e2e:browsers # install Chromium once bun run types # TypeScript checks across packages and E2E bun run ci # dprint, oxlint, and token checks bun run fix # format and apply safe lint fixes bun run build # build the web client bun run start # start the server; build and migrate first bun run docker:up # build and start app plus PostgreSQL locally bun run docker:down # tear down the local Compose project ``` The repository pins Bun 1.4.2. `mise.toml` selects that version automatically for contributors who use [mise](https://mise.jdx.dev); run `mise install` once. Keep the package metadata, `mise.toml`, Docker image, CI, and documentation synchronized when changing it. `bun run e2e` starts four disposable PostgreSQL services, migrates them, builds the client, starts applications on ports 8788 and 8789 with `AGENT=off`, and an isolated `AGENT=on` fake-harness server on 8792 (with its fake GitHub MCP server on 8797) alongside them, then runs Chromium. The local-auth test starts its own application on port 8791 against the fourth database. Set `E2E_SKIP_BUILD=1` only when the existing client build is known to match the checkout. CI has three independent jobs: validation, browser integration, and a Docker image build. A documentation-only change should still pass `bun run ci`. ## Repository map | Area | Responsibility | Internal workspace dependencies | | --------------------- | --------------------------------------------------------- | ------------------------------------------------------------------------------ | | `packages/dialect` | Restricted MDX, MDAST, and Lexical schema | none | | `packages/protocol` | WebSocket declarations and addressing helper | none | | `packages/question` | Questionnaire definitions and shared drafts | `protocol` | | `packages/draft` | Bounded collaborative plain-text drafts | none | | `packages/viewport` | Browser geometry and subscriptions | none | | `packages/diagrams` | Bounded diagram rendering and scoped React viewing | none (React peer) | | `packages/experiment` | Investigation result schemas, selections and native views | `diagrams` (React peer) | | `apps/connector` | Local ACP client and run-scoped MCP bridge | `experiment` | | `packages/editor` | Collaborative editor, decisions, comments, and widgets | `diagrams`, `dialect`, `experiment`, `question`, `protocol`, `viewport` | | `apps/server` | Auth, channels, rooms, storage, Planner, MCP, tasks | `diagrams`, `dialect`, `draft`, `experiment`, `question`, `protocol` | | `apps/web` | Repository picker, navigation, conversation, workspace | `dialect`, `diagrams`, `draft`, `editor`, `experiment`, `protocol`, `viewport` | | `e2e` | Browser and system integration harness | may import server internals as fixtures | Runtime workspace packages do not depend on an application. E2E and skill contract tests may deliberately import server internals; do not treat those test harnesses as runtime package boundaries. ## Runtime model The browser edits Lexical bound to Yjs. One WebSocket multiplexes session, document (`plan:*` on the wire), conversation, questions, comments, and implementation lifecycle messages; the Projects sidebar's own `/ws/sidebar` socket carries only decision counts. The server keeps each open channel as an authoritative Y.Doc with a headless Lexical mirror so it can validate and serialize the document without trusting a browser. Human updates are grouped for 5 ms, applied, projected to canonical MDX, validated, and committed with sidecar state before acknowledgement or relay. An invalid Yjs batch cannot be undone; the room rebuilds the latest known-good state under a fresh epoch and clients reopen. The Planner reads a plan revision and edits through structural block operations. Operations stage against MDAST, pass dialect and Lexical round-trip validation, reconcile into the live tree, and produce one Yjs delta. Unchanged or moved MDAST object identity preserves existing Lexical nodes, selections, and undo history. PostgreSQL is the only runtime storage adapter. One renewable `chopin:writer` lease permits one application process per database. Hosted browser sessions use encrypted credentials in PostgreSQL and restore on presentation of the browser cookie. Startup clears Planner ownership and metadata-only local sessions; collaborative state and external implementation runs are durable. ## Authority and security - **The dialect is an allowlist.** Document MDX is parsed and rendered, never evaluated. Keep imports, exports, expressions, raw HTML, and unknown JSX out. - **Records own decisions.** Question answers and accepted comment decisions live in sidecar records. Their document components are projections. Browser CRDT validation allows unchanged projections to move but rejects their creation, removal or alteration. Domain operations update records and projections together; do not treat the projection as authority. - **Admission is not authorization.** Optional user and organization lists admit an identity. Browser routes, sockets, and Planner tools separately recheck the App installation and repository role. - **MCP has a different credential boundary.** `/mcp` checks its caller-supplied GitHub bearer directly, does not require the App installation, and can mutate data for callers with push or administration access. - **Planner ownership is process-bound.** The first eligible Planner or model-backed research request supplies its GitHub App token and Copilot entitlement. The database stores only a token-free owner reference and durable context. - **Planner tools are repository-fixed under `copilot-sdk` and `pi`.** Those Planners have no shell, checkout, host filesystem, skills, plugins, or arbitrary GitHub access. `HARNESS=atomic` deliberately runs every Planner session, local or hosted, as a full Atomic session with the operator's Atomic tools and resources, including shell and filesystem access as the server process's user. Its working directory is a checkout verified from `invoke_planner` and remembered per channel in memory, or else an empty per-channel directory. Its summary and research workers stay isolated. There is no flag for this; the harness is the choice. - **Repository node IDs are authoritative.** Owner and repository names resolve GitHub requests but never replace the stored node identity. - **Persistence should precede publication.** Do not acknowledge or broadcast a domain mutation before its fenced durable commit. ## Conversation and Planner addressing The web composer uses `packages/protocol/address.ts` to translate `@chopin` into `Chat.Send.to = "planner"`; ordinary messages use `to = "room"`. The wire destination is authoritative on the server. Do not describe `@chopin` as a server authorization boundary: a custom write-authorized client can send an explicit Planner destination without a mention. `instruction()` strips the mention before model input. Recent room messages that did not address the Planner still enter a bounded backscroll for the next turn. An accepted comment also starts an explicit Planner turn after it commits. An MCP `invoke_planner` call posts its instruction as the caller's member message and runs under the channel's existing Planner owner; only a caller with a live browser login can claim an unowned channel. Under `copilot-sdk` and `pi` the Planner is a custom agent, not a general coding agent; under `atomic` it is a full Atomic session. Either way its turn runs through `HarnessAgent.stream()` from `@ai-sdk/harness`; the conversation stays active until the returned stream finishes. An interrupted turn is never replayed automatically because it may already have made durable tool changes. ## Questions, comments, and anchors Question submission claims the shared draft with `claimSubmit()`, then stages the document, authoritative record, and copied question maps inside the room lock. Publish the candidate only after its fenced commit and invoke the returned `Store.stage()` finalizer afterward. A failed commit releases the claim and keeps the accepted draft available for retry. Planner asks and cancellation follow the same persistence-before-publication ordering. A decision's definition is frozen except for appended options. Any writer may send `question:option` to add one while the question is open (at most 10 options, no case-insensitive duplicate label). The server changes the sidecar record, the open entry, and the plan `Option` projection together under the plan queue, commits, and only then acknowledges and broadcasts `question:option-added`. The client's `key` makes a retry return the same option. Shared drafts need no rewrite: an option has a register only once someone selects it. New free-text custom answers are not offered; an existing `custom` draft or answer still renders and derives. Anchors combine Yjs relative positions with canonical block digests. A position survives surrounding edits; a digest can recover one unique block after a move or epoch replacement. Ambiguous matches must orphan rather than guess. The safe ordering is to rebase against the old document before a server-authored edit. Scoped decision-prose jobs follow that ordering. The ordinary `edit_plan` tool still reconciles first and remains a recovery gap. The browser starts a comment from a bounded quote locator, selected length, offset hint, and block indices, not an unbounded copy of selected text. The server resolves the locator and mints relative positions. It handles stale or ambiguous passages conservatively, but does not yet enforce every client-side size and ordering bound on custom wire input. Question and comment relationships have four deliberate states: pending, linked, deliberately empty, and orphaned. Preserve the distinction. Empty means the Planner reviewed the decision and intentionally linked no prose; orphaned means a former target can no longer be identified safely. ## Research requests and child documents Conversation research offers are distinct from research requests. With `CONVERSATION_PLAN=on`, decision and research classifiers process saved messages independently. The research offer's original source and execution identity never change when its Chat placement moves. Human brief edits are shared CRDT patches; Start seals the latest server-accepted brief and persists its delivery intent before publication. See [Research offers from Chat](docs/conversation-research.md). Typing `/research` starts one parent-scoped durable request from the exact brief. A pending request is an inline card, not a channel: it has no URL, sidebar row, Conversation, or Decisions. Failure, cancellation, and explicit retry preserve the request identity; observational reads must not restart terminal work. Successful reconciliation validates the complete report and atomically creates an initialized ordinary child channel plus the request link. Publication must be idempotent and persistence must precede the ready card or nested navigation row. A child owns ordinary document, Conversation, and Decisions state. V1 does not offer child research, Background Work, implementation/tasks, or grandchildren. The server retains `research_workspaces`, turns, messages, and historical chat references as staging and compatibility internals. Do not present them as a standalone report, thread, navigation child, or confirmation flow. ## Implementation graphs The hosted Planner's graph tools remain technically available in any channel after plan readiness checks pass. That does not make child implementation a supported product workflow: the child surface exposes no tasks or implementation destination. The supported MCP read path exposes a graph only for an MCP-created document. The Planner cannot approve, lock, or start a graph. Approval exists in the domain but has no current production UI or route; the workflow is experimental. An approved graph binds one plan revision, graph version, and graph revision. `start_implementation` atomically claims those values and locks the graph. Task, pull-request, blocker, revision, and verification transitions persist before publication. Active implementation prevents plan and decision mutations that would invalidate the claim. The run claim is logical, not authorization-bound to the original coding agent. Any admitted repository writer who knows the run ID can submit lifecycle transitions. Do not describe the claimant as an exclusive security principal. Keep graph counters separate from Yjs epoch, document sequence, plan revision, and storage revision. See [Experimental implementation lifecycle](docs/implementation-lifecycle.md). ## Editor invariants ### Tables - The first row is always the header row. A table has no independent header flag. - Import normalizes merged cells into the rectangular subset supported by the editor. Export remains valid even if Lexical temporarily exposes a span. - Tables are limited to 100 rows and 20 columns in the shared dialect. - Empty paragraphs in cells render as the visible placeholder, so a new cell is immediately editable. - Row and column rails use viewport-fixed overlays outside the editor's clipped container. Keep the overlay tree pointer-transparent except for controls. - A move through a merged span may normalize to the original rectangle and is a valid no-op, not a failed operation. ### Code and diff blocks - Syntax and diff previews are derived React views beside Lexical's canonical code node. Do not mount a second editor or write highlighted DOM into Lexical. - An unnamed fence is plain text. Never guess a language from content. - Invalid `patch` content temporarily falls back to plain code rendering without changing the stored language. Repairing the text restores the diff view. - The preview is non-editable derived UI. The current Enter handler requires a Lexical range selection inside the canonical code block; do not assume a key event focused only on the preview will return to the source. ### Change marks and presence - Planner changes broadcast only after the Yjs update that created their target nodes. - Added and moved marks attach to live block elements. Removed marks use a gap between surviving blocks; no empty tombstone node is inserted into the document. - A Planner cursor and a change mark are separate. The cursor points after the final changed block even if that block is outside the current viewport. - Change marks are broadcast decoration, not durable server history. A client retains at most 50 unseen marks until they enter the viewport, the editor is cleared, or the epoch changes. - Refresh Awareness with `setLocalState`, not only a field mutation, when the visible user profile changes after connection. ### Related prose Hover highlights every related block. Click scrolls to the first related block and pins the set briefly. Only one pin may exist, and clearing a pin must remove the exact highlight object that created it. CSS Highlights are shared by name, so merge ranges from all mounted editors before replacing a registry entry. ## Failure traps ### Collaboration - A Lexical node present in the dialect but missing from server collaboration registration can throw inside a Lexical listener and silently stop sync. Current `registry.test.ts` coverage is narrower than an all-node headless Lexical/Yjs round trip; add that regression coverage when extending the registry. - Lexical may report listener failures without throwing from the transaction. Capture editor errors explicitly when correctness depends on detecting them. - Open the document on provider connection, not only editor mount. Otherwise a late initial socket connection never requests state. - A draft edit must not leave the queue until its acknowledgement. Dropping it after `send()` loses edits on disconnect. - Rebuilds and reconnects must replay unacknowledged updates only when the epoch is still compatible. - Room opening and closing share the document lifecycle lock. Opening waits for an active close; eviction forgets the room only after runtime shutdown and persistence complete. Preserve that ordering. - Sidecar restoration currently drops an invalid optional implementation graph instead of rejecting the whole sidecar. Do not generalize that fail-open behavior to other durable fields. ### Harness and Planner - Chat consumes `HarnessAgent.stream()` as AI SDK `TextStreamPart`s; `translate()` in `apps/server/src/chat/service.ts` projects each part. A `tool-call` naming a tool outside the Planner's active tool set aborts the turn and logs a boundary failure rather than trusting adapter-internal filtering alone. - Permission and approval refusals arrive as `tool-output-denied` or `tool-approval-request` stream parts, not a separately shaped event; render the refusal from that part rather than a guessed top-level field. - The Copilot SDK adapter (`apps/server/src/harness/copilot-sdk/adapter.ts`) still reads raw Copilot SDK events and keeps its own live tool check against session metadata. Treat that as adapter-internal behavior, not a property every harness in `HARNESS` shares. - The GitHub MCP tools reached through `apps/server/src/harness/github-tools.ts` come from a remote server that can change its offered tool count. Diagnose by required names and denied capabilities, not literal counts. - `HARNESS_AUTH` values that fall back to a host-logged-in subscription (for example `auto`) are refused at startup unless the server binds only to a loopback interface, so an operator's own subscription is never lent to every admitted writer on a public deployment. - `AGENT=off` prevents hosted agent turns but does not disable local MCP and does not currently remove every Planner label from the UI. - A success callback for persisted sidecar work is not optional. Calling it after persistence prevents durable transcript state from being dropped. ### Browser and editor - Keep one React owner for derived code or diff previews. Mounting a second owner over the same node causes race-driven cleanup and remounting. - Native `