Ablation: Reading Function by Removing Parts

A trained neural network is a very large collection of numbers. Nobody wrote those numbers by hand: training set them, and nobody knows in advance which of them do which jobs. Interpretability is the effort to find out. The hopeful picture is that small groups of parts — single neurons, or attention heads, the sub-modules that let one word in a sentence pull in information from another — each perform one recognisable step of the model's work. If that picture is right, it can be tested directly. Switch a part off and see what the model can no longer do. That test is called ablation, and its logic is the logic of removal: if the part was needed for a job, the job should fail without it.

The procedure has five steps. First, choose a candidate part: a neuron, an attention head, or a direction in the model's internal numbers — one particular pattern in the values flowing through it. Second, neutralise it. There is more than one way to do this, and the choice matters. You can force the part's output to 0, which deletes its signal entirely. You can replace its output with the average value it takes across other data, which removes its message about this particular input while leaving something typical in its place. You can mask an attention head so it passes nothing on. Or you can subtract out one direction, erasing a single pattern while leaving every other pattern intact. These are different operations, and they can give different answers. Third, run the same inputs through the damaged model that you ran through the intact one. Fourth, compare the two sets of outputs. Fifth — the hard step — decide what the comparison means.

If accuracy on one task falls sharply while other tasks hold steady, that is evidence the removed part was doing something specific to that task. If nothing changes, the part was not necessary — for those inputs, removed in that particular way. Both conclusions are hedged, and the hedges are not politeness. They mark real limits on what the experiment can show.

Networks resist the removal test for a plain reason: they are not machines assembled from parts with one job each. They have far more capacity than their tasks require. They are trained all at once rather than built piece by piece, so nothing forces a function to live in one place. And they are full of duplicate routes to the same answer. Cut one route and another may quietly carry the load, so the output barely moves and the cut looks harmless. Subtract one direction that tracks a concept and the concept may still be recoverable from everything left behind. In each case the experiment reports, truthfully, what the altered model does. Turning that report into a claim about how the original model works is a further step, and it is where mistakes happen.

The clearest warning comes from BERT, a language model whose internal representations have been studied heavily. Researchers found subspaces — sets of directions in the model's internal numbers — that looked, by several standard interpretability tests at once, like the place where a particular piece of grammar was stored. The tests agreed with each other, which is normally reassuring. Then the researchers did the obvious next thing: they intervened on those directions. If the grammar really lived there, erasing the directions should have erased the model's use of it. It did not, at least not reliably. The model went on using the concept. The authors named the pattern an interpretability illusion: several apparently independent criteria can agree about where a concept sits and still be wrong about what the model actually uses.

This is exactly what ablation is for. It is not a complete method for reading out a circuit, and a successful ablation does not prove the removed piece was the circuit. What it does is put pressure on stories built from correlation alone. A neuron that fires on French text, and whose removal leaves the model's French intact, was never a sufficient account of how French is handled — the firing was real, but it was not the mechanism. A head whose removal collapses one narrow behaviour is a candidate worth studying further. A candidate, not a finished explanation: the cut may have severed one link in a larger chain rather than removed the chain itself.

So the discipline is bookkeeping rather than drama. Name the part. State exactly how you switched it off, because zeroing, averaging and subtracting a direction mean different things. Report what changed and what did not, on which inputs, by which measure. Try several related cuts, to see whether an effect is stable or an artefact of one particular incision. And note what remains unknown: how far the BERT illusion generalises to other architectures, including today's larger models, is not settled by that work alone. What the field does know is the shape of the trap. Seeing structure is not the same as testing function, and ablation — used carefully, stated precisely — is how the second gets done.