Probing for Linguistic Knowledge

A modern language model does not store words as words. It turns each word of a sentence into a vector — a long list of numbers — and it revises that list at every layer, which means at every stage of processing between the sentence going in and the prediction coming out. Whatever the model knows about a word at a given stage is carried in those numbers and nowhere else. Probing asks a question that is easy to state and hard to answer well: is grammatical information written into those numbers plainly enough to be read straight off?

The test works like this. Take a sentence, run it through a trained model, and record the vectors at one chosen layer. Do not change the model at all — every one of its weights stays frozen. Then train a second, much smaller program on the recorded vectors: a classifier, a routine that learns to sort inputs into categories from examples with known answers. Feed it vectors for words whose part of speech is already known, and see whether it learns to tell the nouns from the verbs. Because the big model is frozen, it cannot learn anything new during the test. If the small classifier succeeds, the credit must belong to the vectors themselves. The information was already sitting there; the probe merely read it out.

The model most famously put through this procedure is BERT, a widely used encoder trained on a fill-in-the-blank exercise: hide some words in a sentence and make the model guess them from the words around them. That is the entire training signal. No one ever shows BERT a grammar rule, a parse tree — a diagram of how a sentence's phrases fit together — or a label saying which noun a pronoun points back to.

The edge-probing suite then puts a battery of grammatical questions to each of BERT's layers in turn. Every question concerns one or two marked stretches of a sentence, called spans. What part of speech is this span? Do these words form a single grammatical phrase? How does this word depend grammatically on that one? In the event the sentence describes, which span names the doer and which the thing done to? And do two spans — say, "the doctor" and a later "she" — refer to the same person? One design decision matters more than any other: the same kind of probe is used at every layer. That keeps the comparison fair. If the probe reads part of speech more accurately from an early layer than from a late one, the difference must lie in the layers, not in the test.

Run layer by layer, the scores form an orderly map. Local grammatical facts, such as part of speech, are easiest to read off the early layers. Grammatical links between words — which word depends on which — firm up in the middle. Questions that need meaning gathered across a longer stretch of the sentence, such as who did what to whom and which pronoun matches which name, are easiest to read off the later layers. Nobody built this order in. Earlier language software was assembled by hand as a pipeline: first tag parts of speech, then group words into phrases, then work out grammatical links, then meaning. BERT was trained only to fill in blanks, yet a similar order appeared inside it on its own — hence the paper's title, that BERT "rediscovers" the classical NLP pipeline.

The order is a tendency, not a law. It is an average across tasks and datasets, and it shifts with the task; no single sentence marches through a rigid sequence of stages. The honest reading is that different kinds of information become easiest to read at different depths, on average.

Here the second paper enters, because a high probe score can mean 3 different things. The feature may genuinely be written into the vectors. The probe may be powerful enough to reconstruct the feature from weaker hints that are present. Or the probe may have learned nothing from the vectors at all. That last case is worth spelling out. A probe with many adjustable parts can build what amounts to a lookup table: this exact word carried this label in training, so answer the same way again. A probe passing by lookup tells you about the probe's memory, not about the model.

Control tasks were invented to separate these cases. A control task keeps the same words and the same kind of probe, but throws away the real grammatical labels and assigns each word a random label instead. Now there is no structure left to read; the only route to a good score is memory. So the probe's score on the random labels measures exactly how well it can cheat. The number that matters is therefore not the raw score but the gap: accuracy on the real task minus accuracy on the random one. That gap is called selectivity.

Seen this way, restraint pays. A linear probe — one that can only separate categories with a straight dividing line through the space of numbers — is a poor memoriser. When it scores well on the real task and badly on the random one, that wide gap is hard to explain unless the structure sits plainly in the vectors. A deep probe that scores well on both has proved only that it can pass from memory, so its real-task score settles nothing. The choice of probe is part of the scientific claim, not a technical footnote.

Two limits remain. Showing that information can be read out of a layer does not show that the model consults that information when it predicts the next word; presence and use are different claims. And how much power a probe may have before its results stop being about the model is still argued over — control tasks narrow that argument without ending it. What the two papers leave behind is a procedure rather than a headline. Freeze the model. Attach a small classifier. Compare layers. Then check, every time, against random labels. It turns a stack of opaque numbers into something that can be scored against grammatical categories linguists already agree on.