I expected the cheap models to forget first. I wrote that down before the first run. None of the four forgot: each one obeyed the rule as often behind 256 KB of code as it did with nothing in the way.
Instruction files sit at the top of the context. The work happens at the bottom, after the agent has read dozens of source files. If a “never” rule has a half-life, you would see it there, so I built that gap on purpose and measured across four Claude price tiers.
The rule
I took it word for word from vitest’s own AGENTS.md, line 61, and put it alone in an otherwise empty CLAUDE.md.
vitest-dev/vitest · AGENTS.md:61 · 2899eb3
When writing tests, AVOID using `toContain` for validation. Prefer using `toMatchInlineSnapshot` to include the test error and its stack. If snapshot is failing, update the snapshot instead of reverting it to `toContain`.
The task, which never mentions the rule
Write unit tests for `packages/utils/src/helpers.ts`. Reply with the complete test code in fenced code blocks.
A script decides every verdict. It reads the fenced TS and JS blocks in the answer. One .toContain( breaks the run. A run obeys when it has at least one expect( and no toContain. A run with no test code at all scores “no chance”, and I never count that as obeyed.
- The rule
- vitest-dev/vitest · AGENTS.md:61
- The task
- packages/utils/src/helpers.ts
- Padding
- 0 or 256 KB of vitest source, rule-neutral
- Runs
- 20 per point · 240 on this rule
- Spend
- $105.98 at list prices
The padding
Between the rule and the task I placed real files from vitest/packages/*/src, at the same commit. Each file had to pass one test first: the checker finds zero chances in it. No expect(, no toContain, no snapshots, no vitest imports. The padding shows the model nothing about the rule, for or against.
Every run went through headless claude -p --tools "" with an isolated config, in a scratch folder outside my home directory, with no session saved.
Live: four models, three conditions each
Flat on every tier
Pick a model. The top rail is the same model with the rule deleted, so you can see how much work the rule does. The bottom two are the rule with no padding and the rule behind 256 KB.
The model
Runs that obeyed
95% bands: deleted 0–16%, 0 KB 18–57%, 256 KB 18–57%. The 256 KB prompt ran to ~93k tokens. Inline snapshots used: 20/20 → 18/20.
Each run at 256 KB: toContain calls / expect calls
0/159 0/245 7/165 12/144 0/139 4/142 2/150 26/227 6/187 0/167 1/118 5/144 1/160 0/167 2/165 3/191 5/151 0/188 2/175 0/152
Haiku 4.5 Same 7 of 20 at both ends. The rule never held well on Haiku, and 93k tokens of padding did not make it worse. Without the rule, every run reached for toContain.
20 runs per condition. Rails show runs that obeyed. Tiers sort by list price per input token; parameter counts are not public, so price stands in for size.
On all four tiers, obedience at 256 KB matches obedience at zero. Haiku 4.5 sits at 7 of 20 at both ends. Sonnet, Opus and Fable obey every run that wrote tests.
The tiers differ in where they start. Haiku follows the rule a third of the time, and the 5.x models follow it in every run that writes tests. Haiku is also the only 4.5-generation model on the board, so I can’t pin that gap on price. My pre-registered guess, that cheaper tiers forget faster, found no support.
What this doesn’t show
Is this about rules in general?
No. One rule. A second, ruff’s “Don’t prefix tests with test_”, failed screening, and a third, tldraw’s @param rule, needs a person to judge it.
What about longer contexts?
I stopped at 256 KB: about 93k tokens for Haiku 4.5 and 115k to 117k for the 5.x models. Past that I have no data.
Is Haiku worse because it is smaller?
I can’t say. Haiku is the only 4.5-generation model here, so its tier and its generation move together. Parameter counts are not public either, so price is the only size proxy.
Why only two depths?
The plan had eight. Both ends of the price ladder came back flat, so I cut the sweep and logged the reason in the pre-registration.
Does this cover long conversations?
No. Each run is one user message. Depth here means bytes of code between the rule and the task, not turns of back-and-forth.
Prior art
- arXiv 2604.20911
- “Never” rules decay with conversation depth while “do” rules hold. 12 models, synthetic tool-schema padding, no same-family size comparison.
- arXiv 2605.10039
- CLAUDE.md compliance in Claude Code drops as a session writes more functions. Sonnet 4.6 and Opus 4.6.
- arXiv 2402.10962
- Instruction drift within eight turns, on LLaMA2-chat-70B and GPT-3.5.
In those papers the context grows with turns of conversation or with the model’s own output. Here it grows with inert repo code inside one message, and nothing decays. Both results can hold at once.
Found along the way
Three things I didn’t plan to learn
A shipped rule that does nothing on 5.x models. ruff’s AGENTS.md says “Don’t prefix tests with test_.” With that rule deleted, Haiku named its tests test_* in 20 of 20 runs. Sonnet, Opus and Fable did it in 0 of 20 each. The newer models lack the habit the rule exists to stop, so I couldn’t measure it fading.
claude -p leaks your own instructions into a benchmark. Claude Code reads .claude/CLAUDE.md in every parent directory. Run it from a folder under your home directory and it loads ~/.claude/CLAUDE.md, even with --setting-sources project. A canary test caught it. The fix: a scratch folder outside home and its own CLAUDE_CONFIG_DIR.
--tools "" doesn’t stop tool calls. With no tools to call, Haiku wrote its tests inside fake tool-call XML in 9 of 10 pilot runs, and a checker that reads fenced code sees no answer there. One format sentence in the task fixed it. Fable still drifted into fake tool calls in 4 of 20 runs at 256 KB, and in none with no padding.
240 runs on the vitest rule, plus 80 screening runs on ruff’s, all through headless Claude Code. Total spend $105.98 at list prices. I pre-registered the design, and every change to it, before the first run.
The rule and the padding come from vitest-dev/vitest at 2899eb3. Harness 1c771b3, data built 14 October 2026.