II · THE IDEA · ARTIFICIAL INTELLIGENCE
Causal Abstraction via Interchange Interventions
▶ Listen · narrated
If a group of neurons truly encodes a concept, swapping their values between two inputs should transfer that concept's effect on the output — a direct causal test, not a correlation.
At a glance
- What it tests
- Whether a neuron group plays the causal role of a named concept
- The edit
- Replace chosen activations from a base input with those from a source
- Success look
- Output shifts as if the high-level concept itself had been swapped
- Failure look
- Little change, or change the abstract story cannot explain
- Stricter than
- Probe accuracy or other purely correlational readouts
Think of the model as a factory line and a concept as a labelled crate that is supposed to sit on one particular shelf. Correlation is walking past and noticing that whenever the crate should be red, that shelf looks red. An interchange intervention is different: you take the crate from a second run of the line and physically put it on the shelf of the first run, then watch whether the finished product changes as if the crate's label had changed.
If the product changes in exactly the way the label predicts, the shelf was doing the job the story claimed. If nothing much happens, the red appearance was a clue, not the mechanism. That is the whole idea in plain terms — replace a suspected internal variable between two inputs and demand that the output follow the high-level story.
An interchange intervention compares three forward passes. Let x_b and x_s be base and source inputs. Cache activations z_b = f_{1:k}(x_b) and z_s = f_{1:k}(x_s) at a chosen locus (a set of neurons, channels or residual-stream subspaces at one or more layers). Form an intervened state z' that equals z_b everywhere outside the locus and z_s on the locus. Continue the forward pass from z' through f_{k+1:L} and read the output.
The hypothesis under test is an alignment between that locus and a high-level variable V in an abstract causal model. For input pairs that differ on V, the intervened output should match the abstract counterfactual in which V takes its source value and all other abstract variables keep their base values. Agreement across a distribution of pairs is evidence that the locus plays the causal role of V on that distribution; systematic disagreement is evidence against that particular alignment.
The test is strictly stronger than probe accuracy at the same locus. A probe only establishes that V is decodable. Interchange establishes that overwriting the locus is sufficient to transfer V's effect on downstream computation. Limitations are real: the locus may be incomplete, the abstract model may be misspecified, effects may be non-linear across multiple sites, and positive results need not globalise beyond the tested input family. Activation patching without a pre-stated abstract counterfactual is a related edit but answers a weaker question.
Look closer
Only the candidate variables move
The intervention is local on purpose. Two forward passes are recorded; then a third run uses the base input everywhere except at a chosen set of neurons, where the source input's activations are written in. The rest of the network is left as it was. What you are testing is whether that thin slice of state is enough to carry the concept's causal effect — not whether the whole activation pattern of the source can be pasted in.
The verdict lives in the output counterfactual
The interesting object is not the swapped tensor itself but the change in behaviour. If the high-level story is right, the intervened model should act as though the abstract variable had taken the source's value while everything else stayed with the base. Agreement with that prediction supports the claim that those neurons implement the variable. Disagreement weakens it, even when the same neurons look informative under a passive probe.
Detection is not the same as use
A group of neurons can correlate strongly with a human label and still fail interchange. The network may be carrying a shadow or a side-effect of the concept rather than routing computation through that locus. The method is built to expose exactly that gap: readable is cheaper than causally responsible, and the surgery is there to separate the two.
The story
Interpretability often begins with correlation. A neuron fires when a concept is present; a direction in activation space lines up with a human label. Those measurements are useful and easy to over-read. Causal abstraction aims at a stricter claim: that some internal variables the network actually uses are, under intervention, the same variables a human description would name.
An interchange intervention is one way to press that claim. Take two inputs — a base and a source. Run the model on both and record activations at a chosen set of neurons, treated as a candidate high-level variable. Then run the model again on the base input, but at those neurons write in the activations that came from the source. Everything outside the chosen set stays as it was for the base. The question is whether the output moves in the way the high-level story predicts.
If the chosen neurons really implement the concept you have in mind, the surgical swap should transfer that concept's effect. The model should behave as if the abstract variable had taken the source's value. If they do not implement it, the output may barely move, or it may move in ways the abstract story cannot explain. The test is therefore about causal role inside the computation, not about whether a probe can read the concept from those neurons after the fact.
The method is deliberately local and comparative. It does not require a full causal model of the network. It asks only whether a particular partition of the activation state is interchangeable in the way the abstraction claims. That is a higher bar than linear separability. A concept can be decodable from a layer and still not be the variable the rest of the network is listening to. Interchange is one of the tools that try to catch that distinction in the act.
In practice the procedure is repeated across many base–source pairs. A single successful swap is weak evidence; a pattern of output counterfactuals that match the high-level predictions is stronger. Failures matter as much as successes. They may mean the wrong neurons were chosen, that the concept is distributed more widely than the candidate set, or that the human label does not line up with any compact internal variable at all. Each of those is a different scientific conclusion, and the method alone does not decide which one applies.
The approach sits inside a broader family of activation-patching and causal-mediation ideas. What marks the interchange framing is the explicit link to abstraction: the claim under test is not merely that some edit changes the output, but that a named high-level variable has been successfully localised and manipulated. That is why the predicted counterfactual is stated in advance in the language of the concept, rather than discovered after the fact by staring at whatever the output did.
Why it mattered then
As models grew large enough that their internal state no longer fitted in a single glance, purely correlational tools became both more common and more dangerous. Probes, saliency maps and activation atlases could suggest that a concept lived somewhere without showing that the network used that somewhere to compute. Interchange-style tests were a response to that gap: a way to force the interpretation to make a falsifiable prediction about what should happen under a controlled internal edit. The point was not theatrical surgery for its own sake, but a standard of evidence closer to the one used when people talk about mechanisms rather than about readable features.
Why it matters now
Open-weight models make this kind of experiment routine rather than exceptional. Anyone who can run a forward pass with hooks can record a locus, overwrite it, and compare outputs. That accessibility is double-edged. It is easy to run an interchange, declare victory when the output moves, and under-specify the high-level counterfactual that was supposed to be tested. The method still matters because deployment and alignment arguments increasingly rest on claims about what internal variables a model is using. Those claims need a sharper test than correlation, and interchange remains one of the clearer ones available without a full mechanistic account of every layer.
The surprising detail
A locus can look excellent under a probe and still fail interchange almost completely. The network may be storing a readable trace of the concept while routing the actual decision through a different path. The reverse pattern also appears: a set of neurons that looks messy to a simple decoder can nonetheless carry the causal effect cleanly when swapped. The lesson is uncomfortable but useful — interpretability metrics that never touch the forward computation can rank the wrong candidates first.
What is disputed
Success under interchange supports a causal-role claim for the tested inputs and the stated abstraction; it does not by itself prove that the same neurons implement the concept for all inputs, or that the abstraction is the only faithful one. Negative results are also ambiguous without further localisation work. Treat the method as a demanding test of a specific hypothesis, not as a complete map of the network.
Remember this
Interchange asks whether swapping a candidate locus transfers a named concept's effect — a causal test that correlation alone cannot pass or fail.
Test yourself
A linear probe reads a binary concept from layer L with high accuracy. An interchange intervention at the same neurons, using base and source inputs that differ only on that concept, leaves the model's output almost unchanged. What are two distinct explanations consistent with both results?
First, the concept may be represented at L in a readable way but not used from there — a side channel or echo rather than the variable downstream layers read. Second, the true causal variable may be distributed more widely than the probed neurons, so overwriting only that subset leaves enough residual signal for the base behaviour to survive. A third possibility is that the probe is exploiting a correlate that travels with the label in the dataset but is not the concept the task actually depends on. In all three cases the correlational evidence stands, and the causal claim does not.
Go deeper
- [2103.06613] On the approximation error for approximating convex bodies using multiobjective optimization · arxiv.org
- [2210.01807] TripleE: Easy Domain Generalization via Episodic Replay · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.