A master's-thesis preprint scores circuit-based explanations in GPT-2 Small. They read well and are sufficient, but cover only about 22% of the causal picture.
A recent preprint in mechanistic interpretability asks whether the plain-English explanations we attach to a language model’s internal circuits are actually true to what the model does. On a small test, the answer is a qualified no: the explanations read well, but they cover only a fraction of the causal story. The work is an unreviewed master’s thesis on one model and one task, so it is best treated as a clean demonstration of a problem, not a verdict.
The paper, by Ajay Pravin Mahale, targets mechanistic interpretability in GPT-2 Small and tries to measure something rarely quantified: whether a circuit-based explanation is faithful. Below is how the experiment works, what the numbers say, and what to be careful about.
The task: finding the indirect object
The test bed is Indirect Object Identification, or IOI. A prompt like “When Mary and John went to the store, John gave a drink to” should be completed with “Mary.” The model has to find the two names, work out which one repeated, and pick the other.
IOI became a standard puzzle after a 2022 paper, “Interpretability in the Wild” (Wang, Variengien, Conmy, Shlegeris and Steinhardt). That work proposed a circuit of 26 attention heads in seven classes that explain GPT-2 small’s behavior on this task, and judged it on faithfulness, completeness and minimality. The authors themselves noted remaining gaps in understanding. The new preprint starts from that tradition and asks a narrower question about the explanation step.
The experimental setup
The thesis uses GPT-2 Small (124 million parameters, 12 layers of 12 heads) through the TransformerLens library. The dataset is small by design: 50 prompts built from 25 name pairs and two sentence templates, one set in a store and one in a park. The model completes all 50 correctly.
For each prompt the author builds a corrupted twin by swapping the name positions. The core measurement is the logit difference, the model’s score for the correct name minus its score for the other name. Averaged over the dataset it is about 3.36 on clean prompts and about -3.53 on corrupted ones, so the gap to recover is large.
The method is activation patching: copy one attention head’s output from the clean run into the corrupted run and see how much of that gap comes back. A head that restores a lot is causally important for this behavior.
Which heads matter
Patching ranks six heads at the top. Their roles line up with what the 2022 work described.
| Head | Role | Share of the effect |
|---|---|---|
| L9H9 | Name mover | 17.4% |
| L8H10 | Subject inhibition | 12.3% |
| L7H3 | Subject inhibition | 10.3% |
| L10H6 | Backup name mover | 8.9% |
| L9H6 | Name mover | 6.3% |
| L10H0 | Backup name mover | 6.2% |
Together they account for 61.4% of the logit difference. The other 38.6% is spread over heads outside the set. “L9H9” means layer 9, head 9. Name movers copy the right name to the output, subject-inhibition heads suppress the repeated one, and backup movers take over when the main ones are disrupted.
Turning circuits into sentences
The paper’s pipeline then writes an explanation of each prediction, in two ways: filling a fixed template, or prompting a language model with the circuit data. Explanations were scored from 0 to 1 on five checks: names specific heads, gives actual attention percentages, names the correct prediction, names the indirect object, and names the subject.
Templates averaged 60% and never included real percentages or the subject. LLM-written explanations averaged 99%, always gave percentages, named both people in 97% of cases, and ran about 47 words against 21 for templates (n = 30). The abstract rounds this to “64% better.”
That is a readability result, not a truth result. A fluent paragraph with the right numbers in it is easy to produce. Whether those numbers describe what the model needs is a separate question, and the paper tests it next.
Sufficiency and comprehensiveness
Borrowing the ERASER vocabulary from text-classification research, the author defines two faithfulness measures. Sufficiency asks whether the cited heads alone carry the prediction. Comprehensiveness asks whether removing the cited heads changes it. A good explanation should do well on both.
| Method | Sufficiency | Comprehensiveness | F1 |
|---|---|---|---|
| Circuit (the paper’s method) | 100.0% | 22.0% (±17.3) | 36.0% |
| Attention-based | 16.7% | 26.6% | 20.6% |
| Random heads | 46.7% | 28.8% | 35.6% |
Read the last column carefully. The circuit explanation is perfectly sufficient but only 22% comprehensive, and its combined score of 36.0% is barely above the random baseline’s 35.6%. The author attributes the weak comprehensiveness to backup mechanisms: delete the cited heads and the model reroutes the computation through others, so the output barely moves.
One thing the summary I could access does not explain is how six heads that together recover 61.4% of the logit gap score 100% on sufficiency. The definition given, the summed contribution of the cited heads divided by the clean logit difference, does not obviously produce that figure. It may reflect a normalization detail in the full paper, so treat the 100% with some caution until the code is released.
Where the explanations break
The paper sorts prompts by comprehensiveness: 34% fall below 15%, 30% sit between 15% and 25%, and 36% exceed 25%. It names three failure categories.
- Distributed computation. Many heads each contribute a modest amount and none dominates, so a top-six list misses most of the work.
- A missing head. Head L10H10 appears in 82% of failure cases but is not in the top six. Adding it lowers F1 from 36.0% to 34.4%, which the author reads as redundancy rather than a missing causal part.
- Redundant activity. Heads are active, yet including them adds no causal coverage.
The finding with the sharpest practical edge is the last one: model confidence and explanation faithfulness are essentially uncorrelated, with r = 0.009. When the model is sure of its answer, that tells you nothing about whether the circuit story fits. An auditor cannot use confidence as a shortcut for deciding which explanations to trust.
Why backup heads make faithfulness hard
The backup-head result deserves a closer look, because it explains the low comprehensiveness and is a known feature of these circuits. In the setup above, ablating a head and watching the output is the natural test of whether the head matters. If another head quietly takes over the job, that test understates the original head’s role. The head is important when it is present and nearly invisible when it is removed.
This puts explanation writers in a bind. An explanation that names only the heads doing the work in the ordinary case is sufficient, because those heads produce the answer. It is not comprehensive in the ablation sense, because the system has spare capacity. Neither number is wrong; they measure different properties, and a single score can hide the difference. The thesis reports both for that reason.
A checklist for reading interpretability claims
Papers in this area often present a tidy diagram of heads and arrows. A few questions help separate a solid result from a suggestive one.
- How many prompts, and how varied? Fifty prompts from two templates test a narrow slice of language.
- Is there a random or trivial baseline? Here the random baseline nearly ties the circuit on F1, which is exactly the sort of comparison that keeps a result honest.
- Are both directions tested? Keeping the cited components and removing them answer different questions.
- Does the explanation get scored on truth or on style? The 99% quality score measures whether the text contains the right ingredients, not whether the model works that way.
- Could the finding survive a second model or task? This work does not claim to.
What would strengthen the case
The author’s own list of limitations points toward the follow-up work. Per-prompt importance rather than dataset averages would show whether the distributed-computation failures are a property of certain prompts. Larger models would test whether backup mechanisms grow more common with scale, which is an open question the 2022 circuit paper also flagged when it said scaling understanding to bigger models remains unresolved. Human studies would show whether a reader given the explanation predicts the model’s behavior better than a reader without it. That last test is the one that matters for anyone who wants to use these tools for audits.
How much weight this can bear
The limits are stated in the paper and they are big. It uses one task and one model, only 50 prompts, and a fixed top-six set whose importance is averaged across the dataset rather than computed per prompt. There is no human evaluation of whether the explanations help anyone, and no comparison against gradient-based attribution methods such as Integrated Gradients or SHAP. The code was not yet released at the time of the listing, and the work is an MSc thesis from Hochschule Trier rather than a peer-reviewed article.
There is also a bookkeeping flag. The arXiv listing gives a submission date of 13 February 2026, yet the identifier (2603.09988) is the format arXiv assigns to March 2026 submissions. Check the listing history before citing a date.
Within those limits the contribution is real. It shows a way to put a number on how well a mechanistic story captures a model’s behavior, and the first numbers are unflattering. A circuit can be a correct description of part of a computation and still be a poor explanation of the whole. For readers who work with interpretability tools on production models, the lesson is to ask what share of the behavior an explanation covers, not just whether the cited components matter.
Sources
