The don’t version got broken three and a half times as often. Then I tried it on a second project and got nothing at all, which turns out to be the more useful half of the story.
The problem
Instruction files for AI assistants are full of the word never. Never commit unless asked. Don’t add comments. Do not create files unless you have to.
There is an old belief that models are bad at negatives. Tell one not to do X, the theory goes, and you have just put X in its head. People repeat this a lot. Nobody had checked it on real work, with a real instruction file, using a count that needs nobody’s opinion.
If the belief holds, plenty of instruction files are written backwards and fixing them is free.
The method
Take a rule someone really wrote, in a repo people really use. Say it forwards instead of backwards. Change nothing else.
- Pull every don’t-style rule out of a real instruction file, and keep the ones a script can check.
- Rewrite one of them to say the same thing, pointed forwards.
- Give an agent the same coding job 40 times under each version of the zod rule, then 20 more with the rule deleted, and 8 on the maintainer’s file untouched.
- Count the broken rules with the repo’s own checker, not one I wrote.
That last step carries more weight than it looks. The rule I picked is one the maintainers cared about enough to write a checker for. It is a real program that reads the structure of the code, it shipped long before I started, and you can run it yourself. I never had to decide what counted as breaking the rule.
The question everything else rests on
What makes a rewrite fair
This is where a test like this usually falls apart.
A do-this rule tends to cover more ground than the don’t rule it replaces. “Never stack prose across comment lines” says nothing about whether you write comments at all. “Write single-line comments” quietly tells you to write some. Compare those two and you are testing a new rule, not the same rule reworded.
So I gave up on making the two lines mean the same thing. You cannot do that in English, and nobody could check it if I claimed I had. I tied both lines to the checker instead.
Two lines are a fair swap when the checker marks the same code as broken for both of them. Nothing more than that.
That reverses the order of the work, which is the whole trick. You do not write the do version by rewording the don’t version, because that leaves the wording up to my judgement, and my judgement is the thing under suspicion. You freeze the checker first. Then you write the do version as a plain reading of what the checker already looks for. Neither line comes from the other. Both describe the same program.
Then four checks, and you can do all four by eye. The do version cannot ban anything the don’t version allowed. It cannot allow anything the don’t version banned. It cannot need the checker touched. And it has to match on length, position and shape, with no negative words in it. Any rule that fails one of those gets dropped, and how often that happens is worth reporting on its own.
The don’t version, in the repo’s own words
Never stack prose across consecutive `//` lines.
7 words. One clause. Untouched.
The do version, the only line I wrote
Write prose comments as one long `//` line.
8 words. One clause. Same spot in the file. No negative word in it.
Everything else in that paragraph stays identical, character for character, in both versions: the reasoning, the list of exceptions, the mention of the checker.
What the real rule taught me
My plan said the don’t version would be the repo’s own words, so that only one of the two lines was mine. The real rule would not cooperate.
Written out, the maintainer’s paragraph runs like this: a don’t, then the reasoning, then his own do version of the same rule (write one long line instead), then the exceptions. Both directions are already sitting in there.
So a word-for-word don’t version is not a don’t version. It already holds the do version inside it, and the comparison dies before it starts. I cut that one sentence out of both versions, the same way in each, so direction is the only thing left that differs.
I would guess this is normal rather than rare. People writing rules by hand put both directions in, because they are trying to be understood, not measured.
Counted before anything ran
How many rules can even be checked
Before measuring anything, a smaller question. Of the don’t-style rules in a real instruction file, how many can a script check at all?
- Instruction file
- colinhacks/zod · AGENTS.md
- Lines with a negative
- 68, sorted one by one
- The checker
- scripts/check-comments.ts
- Runs on zod
- 125 · one model, two jobs
Of the 68 lines are real rules. The other 32 describe things (27) or already point forwards (5); the 68th restates a rule counted here
Of those 35 can be checked by a script in this setup, which is under a quarter
Of the 35 need a person, or another model, to say whether they were broken at all
Half the don’t rules in a working instruction file are beyond any script. “Avoid redundant assertions.” “Don’t hide behind passive voice.” “Length matches substance.” Real maintainers lean on these, and you cannot count them without someone forming an opinion. I left them out instead of grading them loosely.
Live: four ways this test could have printed a clean zero
Try it here
Each of these prints the same thing. No violations, in every version, which looks just like a model behaving itself. Two were in my plan. I found the other two by building the thing.
The trap
What happens
The repo runs formatters on its own before a commit finishes. One of them joins stacked comment lines back together. Count after committing and the evidence is already gone.
The mechanism
.husky/pre-commit
check:comments <- the rule
lint-staged
biome format --write
biome lint --write
--write means it fixes in place.What the count would have said
T1 In the plan, and the reason nothing here is ever committed. The agent gets no shell access at all, which is blunter than switching the hooks off and leaves nothing to remember. Everything is counted in the working files.
Bars compare what the count would have said against what was really there. Every figure comes from a saved run.
The near miss
The fourth trap came close to handing me a tidy, confident, wrong answer.
The plan said this: before changing anything, run the untouched instruction file a few times and watch how much the count moves about. If it moves a lot, one rule’s effect gets lost in the noise, and that is the finding. I ran it nine times on the untouched file. Zero violations, nine times out of nine. Then eight more with the rule deleted. Zero again. Seventeen runs of the first job, and not one of them moved.
A count that never moves. On paper that is the best thing you can get, a measurement that holds perfectly steady. It would have backed a clean claim that the wording makes no difference at all.
It was worthless. The job I had given the agent produced one or two comments per run. You cannot stack prose across several comment lines if you only ever write one comment. The rule had no chance to be broken. I was measuring my own job, not the model.
A steady count is not enough. It has to be steady and above zero. A measurement pinned at zero looks the same as no measurement at all.
The fix was a job that needs explaining: write three functions whose obvious versions are wrong, and say why. That gets about seven comments per run, which gives the rule something to bite on. Every count in this write-up sits next to the number of chances the rule had.
Before the number
How far this reaches
What follows is one rule, in one project, on one job, with one model. It is not a finding about instruction files in general.
I ran the same test on a second project and got nothing. The two wordings landed on top of each other and the small difference pointed the wrong way. That section is further down, and it is not a footnote.
One account fits both. Wording seems to matter in proportion to how much work the rule is already doing. The first rule cut breakages by 91% at its best. The second cut them by 29%. The rule that was doing real work showed a gap between its two wordings; the rule that was barely landing did not.
So the number below is what turning one working rule around was worth on that rule. It is not a rate to carry anywhere else.
zod, second job · 108 runs · 814 comments · $100 reconstructed from the session logs
The result
The run is the unit. The treatment is applied per run: one run gets one version of the line, and the comments inside a run come from a single generation, so they are not independent of each other.
Inside a run I count comments, not comment lines. One comment is one place where the agent stopped to explain something. Counting lines would be the wrong move, because an obedient run writes one long line where a disobedient one writes three, and that would bury the effect.
So there are two rates, and they are not interchangeable. The headline figure is the per-run rate: take the share of that run’s comments that broke the rule, then average across runs. The per-comment rate pools all 814 comments from these 108 runs and takes the share that broke the rule. That figure is descriptive. It treats each comment as if it stood on its own, and it does not. The significance test permutes whole runs, not comments.
With no rule at all, more than half the comments break it. This is the scale that makes the other rows readable. Without it, a small gap between two wordings could mean a small effect or a dead rule.
The don’t version, in the repo’s own words. Cuts breakages by about two thirds against having no rule, so it works. 23 of the 40 runs came back clean.
The same rule pointed forwards. Same checker, same job, same spot in the file, and the same number of chances to break it: 7.6 comments per run against 7.4. 28 of the 40 runs came back clean.
Never meant to be part of the test — this one was the baseline. It is the maintainer’s untouched paragraph, which holds both directions at once, and it behaves like the do version. I did not write a word of it.
Bars are per-run rates against the worst arm on the board. Longer is worse.
The two middle rows are the test. The top and bottom rows are what make them readable. Delete the rule and the average run breaks it on 55.0% of its comments, so the rule earns its place in both wordings. The gap between the two wordings is about a quarter of what the rule does in total.
Per run, the don’t version got broken 3.4 times as often as the do version covering the same ground: 17.2% against 5.1%, a gap of 12.1 points. Pooled across all 814 comments from these 108 runs, the same comparison is 15.8% against 5.0%, a ratio of 3.2. The per-run gap is the one that carries the claim, and it holds up after correcting for every other comparison on this page.
One thing worth ruling out. Maybe the do version just made the model do less, and a quieter run breaks fewer rules by writing less. The price per run says otherwise. The two versions cost within 2.5% of each other, 99 cents against 96, while the breakage rate moved by 239%. The do version is one word longer and produced 9% less output. Nobody bought the extra obedience with extra work.
The result I did not plan
One row on that chart was never meant to be part of the test. The bottom row is the maintainer’s file exactly as it ships, the version with both directions in it, which I had run only to see how much the count moved.
It sits with the do version and clearly apart from the don’t version. Sitting with the do version is as far as it goes: I cannot tell the two apart, and I would not try. That row is 8 runs, 63 comments and 2 violations, against 40-run arms either side of it. Two violations in either direction move it across the do version.
What survives the small sample is the distance from the don’t version, and who wrote it. I did not write a word of that paragraph. Taking the maintainer’s own do sentence out and leaving the don’t behind made the model obey less. A person wrote that line with none of this in mind, and it points forwards.
The runs I threw away
Partway through, an account limit killed the sweep. Thirty-three runs came back with an error, and twenty-eight of those had written no code at all.
A run that writes nothing breaks nothing. It scores zero and sits in the table as the best-behaved run of the batch. Left in, those runs would have dragged both versions toward looking obedient and shrunk the gap I was trying to measure.
I deleted all thirty-three and ran them again. Every number here comes from a run that finished and wrote at least one comment. The scoring script now marks a run that edits nothing as dead at the moment it writes the record, so nothing further down the line has to remember.
The second project
Where it failed to happen again
One result in one project is an anecdote, so I ran the whole thing again on a second open-source project, remeda. The gap did not show up. That is the honest headline of this section and I am not going to bury it.
The second project turned out to be a better test than I expected, for a reason I did not plan. remeda writes this rule forwards: prefer this over type assertions. zod wrote its rule backwards. So this time the project supplied the do version and I wrote the don’t version. Authorship flipped.
That matters, because it tests the most obvious objection to the first result. In zod, the do version was the one sentence I wrote. Maybe my sentence was just clearer, and clarity rather than direction is what moved the number. Flipping who writes which side puts that to the test.
54 runs on this rule, 18 per version. remeda took 71 runs in all: these 54, and 17 on a different rule, one banning date-based expiry on TODO comments. I dropped that rule before running any comparison. With the rule deleted the agents never wrote a date-based TODO at all, so no wording of it had anything to reduce.
The difference between the two wordings is 0.11, and it points the wrong way. Nothing there.
This is not a case of too few runs. That is the usual excuse for a result that fails to repeat, and it does not apply. The first project’s threefold gap sits outside the range this second test could plausibly have missed. A gap that size would have shown up. It is not that I could not see it. It was not there.
One explanation fits both projects. The first rule was doing a lot of work: at its best it cut breakages by 91%. The second rule barely works at all, cutting them by 29%. Wording seems to matter in proportion to how much the rule is doing in the first place. A rule that is barely landing has little room for its phrasing to matter. Counting the rule I dropped, which had no room at all, that makes three rules, and the phrasing only moved the number on the one doing real work.
The agents did read the second rule. They doubled their use of the approved alternative when the rule was present. They just carried on doing the forbidden thing at the same rate under either wording.
What it can’t tell you
Is one rule enough?
No, and the second one settled that. Two rules, two projects, one gap and one null. What survives is a result about the first rule, plus a reason the second behaved differently. What does not survive is any general claim that prohibitions get obeyed less.
Could a different rewrite have changed the answer?
Yes, and it is still the soft spot. The second project was meant to close it by flipping who wrote which side, but a result of nothing cannot tell you whether the first result came from direction or from my sentence being clearer. That question is open. Closing it needs a project whose own prohibition is strong and whose positive version is written by someone who is not me.
So should you delete the word never?
Not quite, and the reason is the interesting part. The reasoning attached to this rule, the bit explaining why stacked comments are bad, is written around the thing that goes wrong. That is how explanations work. It sits in both versions. For any rule that comes with a reason, a fully forwards version might not be buildable at all, which makes 3.4x a floor rather than a ceiling.
Why not let a model judge the violations?
Then the judge’s mood sets the result. Half the rules in this file would need exactly that, which is why they are out. The rule carrying this write-up got picked because its author had already written the checker.
Everything here runs on a public repo, a public instruction file and a public job. The checker belongs to the maintainers, not to me. Both versions of the line are printed above, and they differ by their direction and one word of length.
Both projects together come to 196 runs and about $137 on Sonnet, if you want to repeat it. zod is 125 runs and $110, averaging 88 cents each; remeda is 71 runs and $26.95. The two figures are not measured the same way: remeda’s costs were captured as each run finished, while zod’s were reconstructed afterwards from the per-run session logs, because the script that ran zod recorded only exit code and wall clock. Almost all of that is the repo being read into context again on every run.
Turning a rule around is free to try, and on one of the two rules here it was worth about a threefold difference in whether the rule got followed. On the other it was worth nothing. Try it on a rule you already know is doing work, and measure rather than assume.