Contents
On 14 November 2026 a local Qwen 3.8 Flash builder on my DGX Spark wrote a 477-line YAML loader for this site. It passed all 475 tests in the suite, and I still did not trust it. The gap a lead review found: a bar chart whose highlighted label is misspelled was silently ignored instead of rejected, because the validation code sat after an early return. The frozen test for that gap took 1,125.42 seconds and 207 tool calls before the builder found the return itself.
The method behind that merge is simple to say. I write what a change must do as tests the builder cannot touch. The builder writes the code. A reviewer that did not write the code runs mutations. I verify by hand. The person who owns the method is closer to an architect than an author, and the numbers below are what each layer actually caught.
The deal: frozen tests
The site is Astro 7.4.1 on Node 22.23.0, tested with vitest. Every delegated card carries lead-written frozen tests, and the file header says so in the builder’s face: // FROZEN (lead-written; the builder never edits this file). The contract types are written by hand before any implementation exists. This is Pre-registration applied to software: the decision rules are committed before the code they judge.
Five pull requests merged into the site repository that Saturday, the first merge at 12:44 and the last at 13:14. They ran this way; the first of them changed no product code at all.
Data
| PR | Merged (local) | Title |
|---|---|---|
| 1 | 12:44 | G17-0: stop pinning the real content in tests |
| 2 | 12:44 | G17-A: adaptive phone title limit |
| 3 | 12:52 | G17-B1: figures as data (loader + registry) |
| 4 | 13:12 | G17-B2: figures as data (renderers) |
| 5 | 13:14 | G17-0b: two more tests stop assuming today’s content |
Source: the merged pull requests of 14 November 2026 in the site repository
Data
| Commit | Who | Lines added |
|---|---|---|
| 8ec6ecd | Lead | 283 |
| 6d20a16 | Builder | 708 |
| a094bae | Builder | 120 |
| 9fb065b | Builder | 54 |
| 6fb5a3c | Lead | 1 |
The loader card is a fair cross-section: the figure has one bar per commit, every line a sum of git show --numstat on that commit. Before the work I wrote 185 lines of frozen tests, 184 at 8ec6ecd alongside the hand-written contract types and a renderer stub, and 1 more at 6fb5a3c, the review finding. The builder’s three commits added 708 lines (the 477-line loader at 6d20a16, with index wiring, extra tests and the docs section), 120 (docs plus extra render tests at a094bae) and 54 (the highlight fix, 41 loader lines plus 13 new tests, at 9fb065b). The builder’s tests always sit in a separate file with its own header, and the frozen file never moved except when I added a case it had missed.
The phone-title card went the same way and showed what the frozen tests are for. The feature: the phone atlas labels the newest N pieces, and the build picks the largest N in 6..0 that passes every phone check. The lead review record for that PR says vitest 445/445, astro check 0 errors, frozen tests untouched, and one mutation run: the reviewer changed the fitter to always return limit 6, and the outliers test caught it. A builder that hard-codes the old constant dies to a test written before the feature existed.
What the builders did
The transcripts of the delegated runs sit in one folder, each with a start line, one tool call per line, and a final line with the printed duration. Three of them cover the two figure cards, the ones where the method earned its keep.
Data
| Run | Tool calls | Wall clock (s) |
|---|---|---|
| First renderer attempt | 11 | 67.3 |
| Renderers | 49 | 715.49 |
| Highlight fix | 207 | 1,125.42 |
Source: the three delegation transcripts in my delegation cache
The first renderer delegation read the card, the frozen tests, the spec types and the drawing helpers: 11 tool calls, duration 67.3 seconds, and no code. The record does not say whether it stopped itself or was cut short; a fresh delegation started 16 seconds later. The second wrote the three renderers in 49 calls and 715.49 seconds. The third, the highlight fix, needed 207 calls and 1,125.42 seconds for 41 lines of validation, because the bug was invisible: the check ran after a return, so the frozen test never reached it.
The transcript of the fix makes the wrong turns readable. The builder first blamed the environment: it grepped vitest’s and vite’s caches for its own probe strings, cleared them, and suspected a concurrent process was rewriting the file. It instrumented the checker with probes at every early-return point; only then did it see that checkBar returns early on the no-whisker path and its tail never ran. Its own summary line is blunt: “The reason was not a cache (I burned a long debug session ruling out vitest/vite caches, stale modules, duplicate files)”. The point is that the builder could not see its own blind spot, and the frozen test would not let it ship one.
The other episode was smaller. At 12:57:55 of the second run the suite failed with Cannot find module '../src/figures/data-loader.ts': the frozen loader test imports a file that the sibling card was still writing. The builder found that file in a commit on the sibling card’s branch, 6d20a16, and pulled it in with git checkout 6d20a16 -- src/figures/data-loader.ts; the suite then read 1 failed | 492 passed (493), then 493 passed (493). Its final report, as my session record has it, lists the file as restored by checkout from the sibling branch’s commit and flags it as outside the card’s file list; the delegation transcript I kept is cut off before that report. The live transcript does not carry that report in full, so I am quoting what the delegation returned to me. I dropped the borrowed file on rebase, because it collided with the version already merged from the loader card. My review record for that PR is one clause: “rebased on main (dropped builder’s copy of data-loader)”.
What the reviewers caught
A reviewer that did not write the code does two things no test can: it reads the spec against the implementation, and it mutates the code to see which tests notice. Two independent review cards, on an unrelated Python repo, give the counts.
The first reviewed a retry-and-registry module. Full suite: 93 passed, baseline main was 66. The reviewer ran 12 mutations and 11 died: the exhausted path keeping the first attempt instead of the last, the attempt counter hard-coded, a dropped error prefix, the registry allowing duplicates. So did the subtle one, a blocked result after failures skipping the usage sum. One survived: nothing tested that the fake runner copies the caller’s script list, so a regression removing the defensive copy “would pass the whole suite” (the reviewer’s words). The reviewer lists that copy among the contract’s sentences it verified against the code, and calls it “defensive copy … works (but untested)”. A written rule, no test pinned it, and the builder had passed every test while skipping it. The review’s frozen additions took the suite from 93 to 100 tests.
The second reviewed a circuit breaker: 137 passed, 19 mutants, 17 caught. The two survivors were behavioural: a backdated trip that shortened an active cooldown (probe: cooldown_until moving from 111 to 60, a banned backend back in service early), and float truncation at the trip boundary. Both became MEDIUM findings, and the review’s closing line is the honest kind: approve, with a follow-up card.
Data
| Reviewed module | Caught | Survived |
|---|---|---|
| Retry-and-registry module | 11 | 1 |
| Circuit breaker | 17 | 2 |
Source: the two independent review cards
The site cards had their own reviewer pass, which is the human kind. On the loader card, the lead review record reads: “found and fixed one gap (highlight labels matching no row were silently ignored: frozen test added, builder fixed)”. The mutation run on the phone card was mine. Between them, every finding about the builders’ code came from a mutation or a hand-read, not from a failing test. On those cards, every builder problem the records show was caught before its merge.
What the end-to-end run caught
The last PR of the morning was not about the builders at all, and it is the one that justifies the whole procedure. After the loader and renderer cards merged, I ran the full build with a published note plus a YAML figure file, exactly what the feature promises an author can write without code. Everything passed except two tests that quietly assumed today’s content: the loader test expected src/figures/data/ to be empty forever (“no yaml today”), and the phone-fit test expected a note named outliers not to fit at 6 titles. Both were frozen tests written before any content existed, both green at their merge times, and both went red when the content came. Both now assert invariants over whatever content exists, with the phone-fit check pinned against an independent largest-passing-limit reference.
A test that pins the present fails in the future. The method’s own tests were the last thing that needed fixing, and only an end-to-end run, the lead’s job, found it.
What this does not show
One codebase. One operator, who wrote the specs, did the lead reviews and ran the end-to-end check himself. Five merged cards, two reviewer cards, three delegation transcripts. The two reviewer cards are from a different repository and a different kind of module (a runner contract and a circuit breaker in Python) than the site’s figure cards, so their mutation counts show what the reviewer role catches, not what it caught here. No control group: I have not run the same cards with the builder writing its own tests, so I cannot say what the frozen tests alone contribute versus the reviewer or the hand check. The durations are wall-clock of delegated runs on one GB10 box, not effort, and the 207-call fix happened once. Small n, no generalisation.
What I take from it is about where thinking goes, not who types. The spec is the work: writing a frozen test for the misspelled highlight label is worth more than the 41-line fix, and the mutation that kills the hard-coded limit is worth more than the 121-line feature. The builder is fast and literal: it will look for the shortest way to a green suite, including a file from another branch, and say so afterwards. The tests and the mutations are what make literal a good property. The open question, untested so far: how much of the review value is me, and how much would survive a reviewer agent that reads the spec the way the two cards’ reviewers did.