A renamed export is the easiest break in a codebase. Seven places still asked for the old name. A parser found six of them. A language model found six. The two lists did not match.
That was the easy category.
The problem
A front-end monorepo holds a hundred packages leaning on each other. You change one, and something in another stops working.
Your typechecker covers part of that. It misses a hook registered under a string name, a key in a JSON manifest, a default value moving from null to 50. Rename the string and nothing complains. The code compiles. The behaviour is gone.
You have a feeling about how big that gap runs. I wanted the number.
The method
Crash-test labs do not wait for accidents. They cause one under conditions they control, then measure the damage.
- Take a real merged pull request from a real repository, at the state someone merged it.
- Break one thing inside it, on purpose, in a shape I know exactly.
- Ask four tools to find it, starting with the free one the repository already runs.
- Count what each one caught and what it cost.
The real pull request does more work than it looks. It carries hundreds of lines of legitimate change alongside my break, including other lawful edits to the same kind of thing. Strip that away and the task collapses into “point at the only edit in the file”, where every tool scores full marks for nothing.
How the scoring works
No opinion grades anything here.
Every tool, the language model included, hands back the same object: a list of pairs saying this package broke because that package changed. Then I look up whether the pair I broke on purpose sits in the list.
Let a model grade another model’s prose and the judge’s leniency sets the score. So I removed the judge.
Anything a tool reports that I did not break counts against it. I run each tool on untouched code first, then score it against its own clean run. Never against another tool’s.
Counted before I broke anything
How much is covered
One question deserved an answer before any experiment ran. Of everything crossing a package boundary in this codebase, how much can the type system see?
- Codebase
- WordPress/gutenberg
- Packages
- 102, indexed in full
- Toolchain as configured
- tsc --build (in CI)
- Breaks injected
- 7, across four kinds
Of the cross-package surface carries type information
Is plain JavaScript with none
Connections made through a bare string, which no typechecker reads
Three hundred and twenty-six places where one package reaches another by naming it in quotation marks. No compiler checks those. That count paid for the rest of the work.
Live: the seven breaks, and who found them
Try it here
Four kinds of damage, ordered by distance from anything a compiler reaches. Pick one to see what I changed and who noticed.
The kind of break
What I changed
The plainest break there is. Something exported under one name now exports under another, and seven places still ask for the old one.
The change itself
packages/components
- export { ThemeProvider }
+ export { ThemeProviderV2 }
7 places still ask for ThemeProvider.Who found it
S1 Nothing found all seven. The parser walked past a wildcard import. The model caught that one and walked past a different place the parser had. On the easiest category on the board, they disagree about one case in seven.
Bars show how many of the broken places each tool reported. Every figure comes from a saved run.
The ladder: seven breaks, twenty-two broken places
Where it stops paying
Each rung costs more than the one above it. I wanted to know what the extra money buys.
Catches renamed exports inside the typed three-quarters of the codebase. Blind to every string connection.
Adds the untyped quarter and all 326 string channels. The largest jump on the ladder, and it costs nothing.
Adds changed default values by reading them out of the function signature. Still free, though it flags two callers that never broke.
Catches two semantic changes the parser cannot see, but misses three hook consumers the parser found — because its prompt did not include the right files. Below the parser in total recall.
With the answer baked into the prompt, the model scores higher — but two of those points came from a leaked framing, not from reading the code. The gap between these two rows is the finding.
Read it top to bottom and the shape is unusual: nothing here is redundant. Each rung reaches a class of damage the rung above it structurally cannot. That is a far more useful result than one tool beating another.
The design I built this to test
Piecework started from one idea. Split the codebase into pieces. Give each piece its own agent. Let each agent report only on what it can see. Find the contradictions by comparing reports, so nobody needs the whole picture.
I like that design, and it happens to be the fashionable answer.
It lost every round I measured. On the breaks it could reach it matched the free parser’s recall, raised two false alarms the parser did not raise, and spent 133,000 tokens getting there. On the last two categories — defaults and semantics — I ran the seat agents and they scored zero for seven, because their inventory does not carry defaults or body-level logic. The inventories before and after the mutation are identical. There is nothing for the join to collide.
One defence remained. Splitting should win at scale, once a codebase stops fitting inside a single request. So I starved the single model of context and handed it the changed file alone. Recall held at 4 of 4. It recovered the affected packages from what it already knew about the codebase. Precision dropped to 4 of 6.
133,000 tokens for the split-agent design. 247,000 for one model with a clean prompt. Zero for the parser that beat both of them in total recall.
Where the model earns its money
Once, and less cleanly than expected.
A parser costing nothing handles three of the four categories. The fourth will never fall to one. A function used to return http:// and now returns https://. A guard used to accept a number and now rejects it. No signature moved. No name went missing, no key vanished, no default changed. You have to read the code to see it.
With a clean prompt, the model caught two of the four semantic breaks. It missed two consumers of the protocol change, judging them unaffected. With the answer baked into the prompt — “all consumers expect this function to add http://” — it caught all four. That delta is itself a finding: two of the model’s four semantic detections were confirmation, not discovery.
It also did something no list comparison can reach. On one string rename — but only when the prompt described the guard logic — it worked out that the rename drives the system into infinite recursion, because the guard and the thing it guards no longer share a name. Without the hint, it found the same broken consumers but missed the consequence.
The model isn’t worth paying to find out what broke. It may be worth paying to find out what happens next — but only if the prompt gives it enough to reason from, and that prompt is itself a source of error.
The mistake, and why it stays in
My first table put the repository’s own toolchain at zero. Nothing at all. It flattered every number under it.
I had run tsc --noEmit, which never writes the type files that let one package see into another. The repository’s CI runs tsc --build, which writes them. Corrected, the toolchain catches 3 of the 7 export-rename breaks, all of them inside typed packages.
That correction shrank everything below it. It stays in the write-up because you should not believe a benchmark where the author’s tool wins and the incumbent scores zero. I did not believe mine until I found the bug.
What it can’t tell you
Is seven breaks enough?
No. Seven injected breaks in one repository give you worked cases, not a statistic. The design calls for two hundred across two codebases. Read everything above as what happened these seven times.
Does the parser catch default-value changes?
Yes, without discrimination. It flags every caller of the changed function, two of which pass their own value and never break. The model reads those two and discards them. Same recall, two false alarms, and the table scores it that way.
Is multi-agent review a bad idea?
It lost this comparison, on this task, against a parser I wrote for the same job. The claim stops there. What travels is the method: price the cheap tool before you price the expensive one.
Would this work on your codebase?
The measurement would. The figures would not. They describe how much of one codebase carries types and how much of it talks through strings, and yours will differ.
What changed since publication
This page is a record of runs, so a re-run rewrites it. Each revision stays listed rather than being quietly absorbed into the figures above.
The model arm split in two. A clean prompt and a hinted prompt now score separately, because two of the model’s four semantic detections turned out to be confirmation rather than discovery. 19 of 22 at 192,000 tokens became 16 of 22 clean and 18 of 22 hinted.
The split-agent design was measured on the last two categories. It scored zero for seven: its inventory carries neither defaults nor body-level logic, so the join has nothing to collide. The earlier draft declined to guess. Now there is a number.
The toolchain baseline was corrected. The repository’s own typechecker had been run in a mode that never writes the files letting one package see into another, which scored it at zero. Corrected, it catches 3 of the 7 export-rename breaks. The section above tells that story in full.
Piecework runs against public repositories. Every input is open source, every run is saved, and you can trace each figure above to a file in the repository it came from.
Before you price a language model into a workflow, find out what a parser would have done for free. Three times out of four here, it did the job.