A renamed export is the easiest break in a codebase. Seven places still asked for the old name. A parser found six of them. A language model found six. The two lists did not match.
That was the easy category.
The problem
A front-end monorepo holds a hundred packages leaning on each other. You change one, and something in another stops working.
Your typechecker covers part of that. It misses a hook registered under a string name, a key in a JSON manifest, a default value moving from null to 50. Rename the string and nothing complains. The code compiles. The behaviour is gone.
You have a feeling about how big that gap runs. I wanted the number.
The method
Crash-test labs do not wait for accidents. They cause one under conditions they control, then measure the damage.
- Take a real merged pull request from a real repository, at the state someone merged it.
- Break one thing inside it, on purpose, in a shape I know exactly.
- Ask four tools to find it, starting with the free one the repository already runs.
- Count what each one caught and what it cost.
The real pull request does more work than it looks. It carries hundreds of lines of legitimate change alongside my break, including other lawful edits to the same kind of thing. Strip that away and the task collapses into “point at the only edit in the file”, where every tool scores full marks for nothing.
How the scoring works
No opinion grades anything here.
Every tool, the language model included, hands back the same object: a list of pairs saying this package broke because that package changed. Then I look up whether the pair I broke on purpose sits in the list.
Let a model grade another model’s prose and the judge’s leniency sets the score. So I removed the judge.
Anything a tool reports that I did not break counts against it. I run each tool on untouched code first, then score it against its own clean run. Never against another tool’s.
Counted before I broke anything
How much is covered
One question deserved an answer before any experiment ran. Of everything crossing a package boundary in this codebase, how much can the type system see?
- Codebase
- WordPress/gutenberg
- Packages
- 102, indexed in full
- Toolchain as configured
- tsc --build (in CI)
- Breaks injected
- 7, across four kinds
Of the cross-package surface carries type information
Is plain JavaScript with none
Connections made through a bare string, which no typechecker reads
Three hundred and twenty-six places where one package reaches another by naming it in quotation marks. No compiler checks those. That count paid for the rest of the work.
Live: the seven breaks, and who found them
Try it here
Four kinds of damage, ordered by distance from anything a compiler reaches. Pick one to see what I changed and who noticed.
The kind of break
What I changed
The plainest break there is. Something exported under one name now exports under another, and seven places still ask for the old one.
The change itself
packages/components
- export { ThemeProvider }
+ export { ThemeProviderV2 }
7 places still ask for ThemeProvider.Who found it
S1 Nothing found all seven. The parser walked past a wildcard import. The model caught that one and walked past a different place the parser had. On the easiest category on the board, they disagree about one case in seven.
Bars show how many of the broken places each tool reported. Every figure comes from a saved run.
The ladder: seven breaks, twenty-two broken places
Where it stops paying
Each rung costs more than the one above it. I wanted to know what the extra money buys.
Catches renamed exports inside the typed three-quarters of the codebase. Blind to every string connection.
Adds the untyped quarter and all 326 string channels. The largest jump on the ladder, and it costs nothing.
Adds changed default values by reading them out of the function signature. Still free, though it flags two callers that never broke.
Buys one thing nothing below it reaches at any price: changes where the shape holds still and the meaning moves. It clears the two false alarms and explains consequences.
Read it top to bottom and you find no redundancy anywhere. Each rung reaches damage the rung above it cannot see, so none of them replaces another.
The design I built this to test
Piecework started from one idea. Split the codebase into pieces. Give each piece its own agent. Let each agent report only on what it can see. Find the contradictions by comparing reports, so nobody needs the whole picture.
I like that design, and it happens to be the fashionable answer.
It lost every round I measured. On the breaks it could reach it matched the free parser’s recall, raised two false alarms the parser did not raise, and spent 133,000 tokens getting there.
I never ran it on the last two categories. My reason for expecting nothing: comparing inventories cannot surface a change of meaning, because both inventories come back identical. That reasoning has no measurement behind it, and I would rather say so than let the table imply one.
One defence remained. Splitting should win at scale, once a codebase stops fitting inside a single request. So I starved the single model of context and handed it the changed file alone. Recall held at 4 of 4. It recovered the affected packages from what it already knew about the codebase. Precision dropped to 4 of 6.
133,000 tokens for the split-agent design. 192,000 for one model reading the whole change. Zero for the parser that matched both of them on three categories out of four.
Where the model earns its money
Once, and the once matters.
A parser costing nothing handles three of the four categories. The fourth will never fall to one. A function used to return http:// and now returns https://. A guard used to accept a number and now rejects it. No signature moved. No name went missing, no key vanished, no default changed. You have to read the code to see it.
The model caught all four of those. Nothing else caught one.
It did something a list comparison cannot reach, too. On one string rename it reported the broken connection, then worked out that this particular rename drives the system into infinite recursion, because a guard and the thing it guards no longer share a name. Three free tools already tell you what broke. The model tells you what the breakage does.
The mistake, and why it stays in
My first table put the repository’s own toolchain at zero. Nothing at all. It flattered every number under it.
I had run tsc --noEmit, which never writes the type files that let one package see into another. The repository’s CI runs tsc --build, which writes them. Corrected, the toolchain catches 3 of the 7 export-rename breaks, all of them inside typed packages.
That correction shrank everything below it. It stays in the write-up because you should not believe a benchmark where the author’s tool wins and the incumbent scores zero. I did not believe mine until I found the bug.
What it can’t tell you
Is seven breaks enough?
No. Seven injected breaks in one repository give you worked cases, not a statistic. The design calls for two hundred across two codebases. Read everything above as what happened these seven times.
Does the parser catch default-value changes?
Yes, without discrimination. It flags every caller of the changed function, two of which pass their own value and never break. The model reads those two and discards them. Same recall, two false alarms, and the table scores it that way.
Is multi-agent review a bad idea?
It lost this comparison, on this task, against a parser I wrote for the same job. The claim stops there. What travels is the method: price the cheap tool before you price the expensive one.
Would this work on your codebase?
The measurement would. The figures would not. They describe how much of one codebase carries types and how much of it talks through strings, and yours will differ.
Piecework runs against public repositories. Every input is open source, every run is saved, and you can trace each figure above to a file in the repository it came from.
Before you price a language model into a workflow, find out what a parser would have done for free. Three times out of four here, it did the job.