Visual selection assessment

This is a bounded local comparison of three ways to choose an explanatory visual for engineering discussion. It does not change the hosted Planner or document routing.

Plan and frozen inputs

1. Pin a small SeeCode catalogue and the same generator instructions for all strategies.
2. Verify the frozen development registry and one checkpoint per source discussion.
3. Probe one case, then run one complete baseline before changing prompts or choices.
4. Review disagreements against the source and report counts, latency, and available usage.

cases.json selects seven distinct real development discussions, one checkpoint each. Its registry hash is 5cdc086840041f9b5c2140d90f9baa8355c7604b8b6fa46d991f09c9f3b889cb from the development-only loader at 0c560ea126181bd3c6911a318f9804a029f0c85a. The loader verifies every selected checkpoint hash and cutoff before sending source text. It sends only event identity, authorship, timing, body, and process fields. Source links stay inert. Candidate interpretations, later events, synthetic examples, validation, and reserved data are not in these model inputs. This first batch has discussions, not standalone plans.

The pinned outcomes are dependency, sequence, state, flowchart, table, and prose. The first four are actual SeeCode renderer types. The set covers directed relationships, ordered interactions, state changes, explicit branches, comparisons, and appropriate abstention. It deliberately excludes chart types that require absent measurements.

Strategies

Strategy	Selection	Filling
Direct	Model chooses from all six	Same model and generator instructions
Jev one	Jev's top choice	Same model, constrained to that choice
Jev shortlist	Jev's top three probabilities	Same model chooses among those three and fills

The two Jev strategies reuse one validated probability distribution per case. Count its latency and usage once for the executed batch; either deployed Jev strategy would incur one Jev call per case. The generator uses an isolated, tool-free HarnessAgent session, not an actual hosted Planner turn. The exact requested harness model is recorded; if a provider does not expose its resolved version or token usage, report those fields as unavailable.

The runner writes one JSONL record per Jev call and strategy attempt. It never retries a model result silently. A later mechanical repair, if needed, must be a separate run with a new output path, limited to failed attempts and reported apart from this baseline. At most 20 cases and 60 strategy attempts are permitted. Results belong under ignored data/ or another private location, never in a source commit.

Rubric

Evaluate each completed output against its checkpoint only. More than one choice may be defensible. A matching label by itself earns no credit. Use 0, 1, or 2 for each judgment:

Criterion	0	1	2
Useful choice	Distracts or obscures	Defensible but weaker	Makes the source easier to understand
Source faithfulness	Invents or contradicts	Mostly grounded with material omission	Claims and relationships trace to source
Data sufficiency	Required inputs absent	Partial, with visible gaps	Enough stated information for this form
Readability	Hard to parse	Workable	Clear at ordinary document size
Abstention	Wrongly draws or declines	Debatable	Correctly uses prose/table or draws

Separately record parse validity, renderer validation problems, successful render, failures, and whether evidence IDs exist. Human reviewers inspect the actual wording and rendered SVG for fabricated relationships, false certainty, missing alternatives, unreadable labels, and useful diagram placement. Automated and agent judgments are provisional until reviewed. Keep an explicit small queue of disagreements and failures rather than treating the scores as calibrated gold. Summarize by source cluster, not by individual message or checkpoint.

Commands

From the repository root, with the frozen development directory available locally:

bun test apps/server/src/visual-selection-eval
bun apps/server/src/visual-selection-eval/run.ts --jev-probe \
  --case astro-production-deps --registry data/visual-selection/frozen-development/registry.json \
  --jev-env /Users/maggieappleton/Projects/chopin/.env \
  --output data/visual-selection/jev-probe.jsonl
bun apps/server/src/visual-selection-eval/run.ts \
  --registry data/visual-selection/frozen-development/registry.json \
  --jev-env /Users/maggieappleton/Projects/chopin/.env \
  --model gpt-6-luna --output data/visual-selection/baseline.jsonl
bun apps/server/src/visual-selection-eval/review.ts \
  --registry data/visual-selection/frozen-development/registry.json \
  --results data/visual-selection/baseline.jsonl \
  --output data/visual-selection/review.md

The default Copilot harness reads a GitHub user token from GITHUB_TOKEN, GH_TOKEN, or the existing gh keyring without printing it. Use an existing supported pi or atomic harness with --harness, --auth, and a model available to that provider if Copilot is unavailable. Run one case first with --case astro-production-deps and a separate output file. Do not tune after that smoke check; complete the pinned seven-case baseline first.