Experiment: visual selection comparison harness #408
- MA
## Status: experiment — hold from merge
This PR adds a **local visual-selection evaluator**, not a feature that makes the hosted Planner create diagrams. Its first seven-discussion comparison cannot decide whether direct selection or Jev works better. I recommend keeping it as an experiment for now.
## What happened
The first run used 21 model attempts and seven Jev calls. It recorded zero successfully rendered diagrams, but that number is **not a finding that the model cannot make diagrams**. The evaluator asked for a JSON-object diagram spec while its response schema required a string. Failed baseline responses were not saved in full, so their individual causes cannot all be assigned. In one separate mechanical retry pass, five responses returned JSON-object sequence specs that the old schema rejected; all five saved specs passed SeeCode rendering and cited existing source event IDs. The retry is not part of the original baseline.
The discussion passages often warranted prose or a table, and no blinded human ratings were collected. The current completion counts cannot rank the three strategies.
## What this PR contains
- A pinned six-choice catalogue, a seven-discussion manifest, source hash and cutoff checks, and local runners for direct choice, Jev top choice, and Jev shortlist. - A corrected JSON-object output contract, renderer validation, raw attempt records, and a human review rubric. - No production routing or document changes. Source bytes and raw results remain in ignored local data, so the evaluation is not independently reproducible from this PR alone.
## Flow

## Before reconsidering a merge
Decide whether a maintained evaluator belongs in the repository, make its public inputs reproducible, and run a fresh cohort with the corrected contract and human source/output review. The follow-up [PR #411](https://github.com/githubnext/chopin/pull/411) is a separate three-passage diagnostic, not a replacement for this PR.
All GitHub validation, browser, and container checks passed. Those checks verify the code, not the experimental conclusion.
MaggieAppletonreferenced this in Experiment: composed Jev visual selection #411last month on GitHub
MaggieAppletonrenamed Add bounded visual selection evaluation harness to Experiment: visual selection comparison harnesslast month on GitHub
MaggieAppletonconvert-to-draftlast month on GitHub