II · THE IDEA · ARTIFICIAL INTELLIGENCE
Mechanistic Interpretability of Induction Heads
▶ Listen · narrated
A model completes “When Mary and John went to the store, John gave a drink to” with Mary. The path that produces that name can be traced head by head.
At a glance
- Model
- GPT-2 small
- Task
- Indirect object identification in short English prompts
- Method
- Activation and path patching to isolate causal components
- Finding
- A sparse attention-head circuit implements in-context name copy
Think of a short play script in which two characters are named, then one is mentioned again, and you must fill in the other. The model does that fill-in without an external memory of the story: everything it needs is inside the prompt.
Inside the network, information moves in a shared stream. Attention heads are small readers that look back at earlier words and write what they find into that stream. Some heads behave like a finger tracing a repeated pattern—useful for copying. Others favour one name or damp the other. By the last layer, the stream has been nudged so the missing name scores highest.
Researchers discover which heads matter by swapping pieces of the computation between a normal prompt and a carefully broken one, then watching whether the answer name flips. Heads that flip the answer when swapped are part of the circuit; heads that do nothing can be ignored for this task.
IOI in GPT-2 small is treated as a causal circuit discovery problem over residual-stream writes. Let a clean prompt p contain names A and B with B repeated as subject so the indirect object is A; a corrupted prompt p′ alters names or structure so the preferred completion differs. Activation patching replaces selected activations z(p′) with z(p) (or the reverse) and measures Δlogit = logit(A) − logit(B) at the final position.
Path patching restricts the intervention to particular edges—commonly a head’s contribution through its OV pathway into later queries, keys, or values—so composition can be attributed rather than only endpoint activations. Under these tests, a sparse set of heads accounts for most of the clean IOI behaviour: earlier previous-token and induction-like heads move token information; S-inhibition heads reduce attention to the repeated subject; name-mover heads attend to the indirect object and write in a direction that raises its unembedding logit. Backup name movers and opposing heads appear when primary movers are ablated, which is why single-head knockouts understate redundancy.
Limitations are sharp. The circuit is task- and model-specific; patching assumes a chosen corruption family; and head role names are behavioural summaries, not guaranteed monosemantic labels. The result is still strong evidence that in-context copy on this distribution is implemented by identifiable attention composition rather than by an undifferentiated whole-network effect.
Look closer
Duplicate names force a choice
In the IOI setup, two names appear, then one is repeated as the subject of a later clause. The model must write the name that is not repeated—the indirect object. That forces the network to suppress the more recently mentioned name and promote the other, which makes the internal competition visible to patching experiments.
Induction-like heads move tokens
Some heads behave like induction heads: they attend from a token back to earlier positions where a similar prefix occurred, then contribute to copying what followed. In IOI they help route name information so later heads can write the correct token into the residual stream at the final position.
Name movers and inhibitors
Later heads, often called name-mover heads in this literature, attend strongly to the correct name and increase its logit. Other heads work against them—attending to the wrong name or reducing the correct one—so the circuit is not a single clean pathway but a small bundle of cooperating and opposing components.
The story
Mechanistic interpretability tries to open a trained transformer and say, of a particular behaviour, which components cause it. The indirect-object-identification work on GPT-2 small is a concrete case: the behaviour is narrow enough to test, yet rich enough that the model must track entities across a sentence rather than emit a fixed string.
The prompts look like ordinary narrative fragments. Two people are introduced; one is mentioned again as doing something; the sentence ends where the other person’s name should appear. Humans find the answer trivial. The model usually gets it right as well. The research question is not whether it can do the task, but which attention heads and residual-stream directions implement the solution.
The method is causal rather than purely correlational. Activation patching replaces internal activations from a clean run with those from a carefully altered run (or the reverse) and measures how much the output name changes. Path patching refines this by restricting the patch to particular routes through the computation graph, so that an effect can be attributed to a head’s query, key, or value pathway rather than to the head as an opaque block.
What emerges is a sparse circuit. Early heads gather and move token-level information, including patterns that resemble induction: a head attends to prior occurrences of a cue and helps recover what came next. Mid-layer heads assemble candidate names and their roles. Later name-mover heads write the favoured name into the final residual stream, raising its logit. A smaller set of heads does the opposite—backing the repeated name or damping the correct one—so the successful output reflects a balance, not a single triumphant component.
Induction heads matter here because in-context copying is exactly what the IOI task demands. The model is not retrieving a memorised fact about Mary or John; it is reading the prompt as a short prefix-completion problem and reusing a name that already appeared. The same family of attention patterns shows up in broader work on how small transformers learn to complete repeated structures from context alone.
The circuit is not claimed to be the whole of linguistic competence inside GPT-2 small. It is a traced subgraph sufficient for this task family, discovered under a particular patching methodology. Other tasks, prompt formats, or model scales may recruit different heads or different residual directions. The value of the result is that the subgraph is specific enough to name, ablate, and argue about—an existence proof that at least one non-trivial behaviour in a real language model can be reverse-engineered at the level of attention heads rather than left as a black box.
Why it mattered then
When this circuit work appeared, large language models were already impressive on prompts that looked like reasoning, yet almost nothing was known about which internal pieces implemented even simple behaviours. Indirect object identification was chosen because it is easy to state, easy to score, and hard to dismiss as pure memorisation: the answer is determined by the prompt’s own structure. Showing that a sparse set of heads in GPT-2 small could be isolated with path patching gave the field a worked example—complete with failure modes, opposing heads, and a vocabulary for talking about copy behaviour—rather than another accuracy table.
Why it matters now
In-context copying still sits under many behaviours people care about: following a format shown in the prompt, reusing entities in a dialogue, and completing patterns without weight updates. Circuits built around induction-like attention are therefore not a historical curiosity; they are a template for asking whether a model is doing the task the way we think it is. The same patching habits—corrupt a prompt, restore one path, measure the logit difference—transfer to other open-weight models when someone needs evidence beyond “the output looked right.”
The surprising detail
The circuit is not purely cooperative. Some heads actively push probability toward the wrong name, and the model’s correct answer is what remains after that internal disagreement. Removing or patching only the “helpful” heads therefore misreads the mechanism: the behaviour is a small politics of attention, not a single dedicated wire from subject to object.
What is disputed
The named circuit is tied to GPT-2 small, to the IOI prompt distribution used in the study, and to the patching tests applied there. Head labels such as name-mover or induction-like describe observed roles on this task; they are not proof that the same heads play only those roles elsewhere, or that larger models implement IOI with the same subgraph.
Remember this
In GPT-2 small, in-context name copying on IOI is implemented by a sparse, partly adversarial bundle of attention heads—not by a single specialised module.
Test yourself
A patching experiment restores only the outputs of late name-mover heads from a clean IOI run into a corrupted run, and the correct name’s logit largely recovers. What does that result show, and what does it still not show?
It shows those heads are sufficient, on that path, to carry much of the clean-run signal that favours the correct name. It does not show they are the whole mechanism: earlier heads may have prepared the residual stream they read, and opposing heads may still be active. Sufficiency of a patch is not the same as a complete account of every contributing component.
Go deeper
- [2211.00593] Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small · arxiv.org
- [2208.11895] Magnetoresistive behaviour of ternary Cu-based materials processed by high-pressure torsion · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.