II · THE IDEA · ARTIFICIAL INTELLIGENCE

Looking Inside the Model

Reasoning and architecture · Mechanistic interpretability · Active research, partial results

▶ Listen · narrated

We can measure a model's behaviour on any input we choose. We still cannot open it and read a legible account of what it has learned, written out step by step.

At a glance

Probing: Train a small classifier on activations to see if a concept is linearly represented
Activation patching: Swap activations between two runs to isolate which parts of the network matter for a behaviour
Sparse autoencoders: Decompose superposed features into interpretable directions, published by Anthropic in 2023
Current scope: Small models and toy tasks mostly understood; frontier models remain largely opaque

Imagine you are trying to understand how a clockwork mechanism works, but you are only allowed to observe the clock face. You can measure when the hands move, test whether winding the spring changes the speed, and compare different clocks to see what they have in common. Mechanistic interpretability does something similar with neural networks. The model is a black box, but researchers have developed tools to look inside while it runs. Probing checks whether a particular piece of information — say, whether a sentence is a question — is present in the activations at a given layer. Activation patching goes further: it swaps in activations from a different input to see if that changes the output in a predictable way, which tells you whether those activations were actually responsible for the behaviour. Sparse autoencoders try to reverse-engineer the features the model has learned, by finding a larger set of interpretable concepts that can reconstruct what each layer is doing. The tools work well on small models and toy problems. Frontier models remain mostly opaque.

Look closer

Probing asks whether information is present, not whether it is used
A linear probe is a simple classifier trained to predict some property — say, whether a sentence is grammatical — from the activation vector at a particular layer. If the probe succeeds, the information was linearly accessible at that point. But the model itself may never consult it. The probe tells you the information is there in principle; it does not tell you the information participates in the computation that produces the output. A representation can be recoverable and yet causally inert.
Activation patching swaps in a piece from a different forward pass
Run the model on input A, save the activations at some layer. Run it again on input B, but this time replace the activations at that layer with the ones you saved from A. If the output changes in a specific way — say, the model now gives the answer it gave for A — you have evidence that those activations carry the information responsible for that behaviour. The technique is also called causal tracing or interchange intervention. It is more expensive than probing because it requires multiple forward passes, and interpreting the results requires care: patching can introduce distribution shift, and a null result does not prove the layer is irrelevant, only that swapping it in isolation was not sufficient.
Superposition means one neuron can encode many features
Toy Models of Superposition demonstrated that networks learn to pack more features into fewer dimensions than they have neurons, storing them as overlapping directions in activation space. A single neuron's activation is then a mixture of many semantic features, which is why inspecting individual neurons rarely yields clean interpretations. Sparse autoencoders attempt to undo this: they are trained to reconstruct a layer's activations using a much larger set of features, most of which are zero for any given input. The hope is that these learned features correspond to interpretable concepts. Anthropic's 2023 work identified features in a one-layer model that respond to specific topics, grammatical structures and even individual entities, but scaling the technique to frontier models remains an open problem.

The story

A trained neural network is a list of numbers. The numbers encode something — the model demonstrably knows facts, follows instructions, generates coherent text — but they do not encode it in a form we can read. The weights are not a lookup table. They are the residue of a learning process, and the knowledge is distributed across millions or billions of parameters in a way that resists straightforward inspection.

Mechanistic interpretability is the project of reverse-engineering that encoding. It asks not just what a model does, but how: which circuits of neurons implement which behaviours, which representations emerge at which layers, and whether any of it can be expressed in terms a human can verify.

Three techniques have produced the most legible results so far, though none scales easily to frontier models.

Probing trains a separate small classifier to predict some property from a layer's activations. If a linear probe can tell whether a sentence is grammatical by looking at the activation vector at layer eight, then the information about grammaticality is present in that vector in a linearly accessible form. The technique is cheap and widely used, but it has a limitation: it tells you the information is there, not that the model uses it. A probe might succeed even when the model's own computation ignores that representation entirely.

Activation patching is more expensive but more causal. You run the model on one input, save the activations at some layer, then run it on a different input and replace that layer's activations with the saved ones. If the output changes in a predictable way — if the model now behaves as it did for the first input — you have evidence that those activations were responsible for the behaviour. The technique is also called causal tracing or interchange intervention. It requires multiple forward passes and careful interpretation, because patching can create distribution shifts, and a null result does not prove irrelevance, only that the intervention in isolation was insufficient.

Sparse autoencoders address a deeper problem: superposition. Toy Models of Superposition, published by Anthropic researchers in 2022, demonstrated that networks pack more features into a layer than the layer has dimensions. They store features as overlapping directions in activation space, which means a single neuron's firing is usually a weighted mixture of many unrelated concepts. Inspecting individual neurons therefore rarely yields clean interpretations.

A sparse autoencoder is trained to reconstruct a layer's activations using a much larger dictionary of features, most of which must be zero for any given input. The sparsity constraint encourages the autoencoder to learn features that correspond to coherent concepts. In 2023, Anthropic published results from applying this technique to a one-layer transformer model. They identified learned features that activate for specific topics, grammatical structures, and even individual entities. Some features were highly interpretable: one responded to descriptions of the Golden Gate Bridge, another to code involving Base64 encoding.

The work is careful to note its limitations. The model studied was small, and the features were identified by a combination of automated search and manual inspection, not by a process that guarantees completeness. Scaling the technique to models with hundreds of layers and hundreds of billions of parameters remains an open problem. The computational cost grows steeply, and it is not yet clear whether the features learned by larger models will remain interpretable or fracture into something more diffuse.

Why it mattered then

Mechanistic interpretability emerged as a response to a specific failure mode: models that performed well on benchmarks but failed unpredictably on inputs that seemed minor variations of tasks they could handle. Understanding behaviour through input-output testing alone was proving insufficient, particularly as models began to be deployed in settings where a failure could have consequences. The field drew on neuroscience's tradition of lesion studies and single-neuron recordings, but adapted those methods to systems where every parameter could be inspected and every forward pass could be repeated identically. Early work focused on small vision models and toy tasks where ground truth was available — models trained to add numbers, or to classify simple shapes — because those were the only settings where a claim to have understood a circuit could be verified exhaustively. The techniques developed in that setting are now being applied to language models, where the task is vastly harder and the notion of ground truth is less clear.

Why it matters now

The practical motivation has intensified. Models are now used to generate code that runs in production, to summarise medical records, to answer questions where a wrong answer is not just unhelpful but dangerous. Regulators in multiple jurisdictions are beginning to ask whether a system can be audited, and what it would mean to audit a neural network. Mechanistic interpretability is one of the few research programmes that attempts to answer that question with something other than statistical correlation. It is also motivated by a longer-term concern: if models become capable enough to pursue goals in the world, we would like to know what goals they are pursuing, and whether those goals remain aligned with the ones we intended. That question cannot be answered by observing behaviour alone, because behaviour is consistent with many underlying goal structures, and some of them are catastrophically misaligned. The field remains far from that level of interpretability, but the work being done now on smaller models is building the methods that might one day apply to systems where the stakes are higher.

The surprising detail

Sparse autoencoders sometimes learn features that are more interpretable than anything a human labeller would have annotated. In Anthropic's 2023 work, one feature activated strongly for text involving the Golden Gate Bridge, but also for other contexts involving San Francisco landmarks, descriptions of suspension bridges in general, and even metaphorical uses of the phrase "bridging the gap". The feature was not simply a keyword detector; it appeared to track a cluster of related concepts that the model had learned to treat as a coherent category during training. Another feature responded to Base64-encoded strings in code, but also to discussions of encoding schemes in natural language and to garbled text that resembled encoding errors. The features cut across the categories a human might have chosen, and in some cases revealed structure in the model's representations that would have been difficult to hypothesise in advance.

What is disputed

The claim that sparse autoencoder features correspond to interpretable concepts rests on a combination of automated metrics and manual inspection. There is not yet a formal definition of interpretability that would allow the claim to be verified mechanically, and it is possible that features which appear interpretable on the examples inspected would prove less coherent on a broader distribution of inputs. The published work acknowledges this, and treats the results as suggestive rather than conclusive.

Remember this

We can now identify some features in small models. Frontier models remain largely opaque, and the gap is not just a matter of scale.

Test yourself

A probe trained on layer twelve can predict with high accuracy whether a sentence is a question or a statement. Does this prove that the model uses that representation when deciding how to continue the text?

Go deeper

Image: Original diagram, The Daily Triptych. Licence: Original work. Source.

← Back to day 72