II · THE IDEA · ARTIFICIAL INTELLIGENCE
Distillation
▶ Listen · narrated
You cannot afford to run the large model in production. You train a smaller one on its outputs instead, hoping to capture what it learned without the cost.
At a glance
- What transfers
- Patterns in how the teacher assigns probability across classes
- What does not
- The teacher's full representational capacity and reasoning depth
- DistilBERT result
- Retained 97% of BERT's performance at 60% of the size
- Typical speed gain
- 40-60% faster inference, depending on architecture and task
Imagine you learned to cook by watching a master chef. You do not just see what dish they made; you watch them taste, adjust, hesitate, choose between two techniques. Distillation works the same way. A large model — the teacher — processes examples and outputs not just an answer but a probability for every possible answer. A smaller model — the student — trains on those probability distributions instead of the original data. It learns the teacher's sense of what is likely, what is close, what is implausible. The student runs faster and costs less, but it never quite matches the teacher's performance. The last few percentage points resist transfer, because the student's smaller architecture cannot hold everything the teacher learned. You have compressed the knowledge successfully, but some of it required more capacity than you gave the student.
Distillation trains a student model to minimise the divergence between its output distribution and a teacher model's output distribution, typically using cross-entropy loss with the teacher's softmax outputs as targets. Hinton's formulation introduces a temperature parameter T that softens both distributions before computing the loss: the softmax is computed as exp(z_i / T) / Σ exp(z_j / T), where higher T spreads probability more evenly across classes. During training, the student typically optimises a weighted combination of two losses: one matching the teacher's soft targets at high temperature, one matching the true hard labels at T=1. The student architecture is smaller — fewer layers, narrower hidden dimensions, or both — and trains from scratch; no weights are copied. What transfers is the teacher's learned similarity structure: which inputs are close in the teacher's representation space, which outputs co-occur, which classes are confusable. What does not transfer is representational capacity. If the teacher's decision boundary requires composing features across more layers than the student has, or holding more hypotheses in parallel than the student's width permits, the student will plateau before reaching parity. DistilBERT demonstrated this: six layers retained 97% of twelve-layer BERT's GLUE performance, but the gap persisted across tasks and training duration, suggesting an architectural rather than optimisation-related limit. The method generalises across architectures — the student need not match the teacher's structure — which implies that much of the transferred knowledge exists at the level of input-output mappings rather than internal representations.
Look closer
The student trains on soft targets, not hard labels
A conventional model sees only the correct class: cat, not dog, not car. The teacher model outputs a probability distribution — perhaps 0.87 for cat, 0.09 for dog, 0.03 for lynx, 0.01 elsewhere. That distribution, called a soft target, encodes similarity structure. The student learns from it, discovering that lynx is closer to cat than car is, even when lynx is never the right answer. Hinton introduced a temperature parameter that softens the distribution further, spreading probability to more classes and making the dark knowledge — the teacher's nuanced uncertainty — more visible to the student.
The student model is architecturally smaller
Distillation is not pruning or quantisation. The student is a distinct, smaller network, often with fewer layers or narrower hidden dimensions. DistilBERT halved BERT's layer count from twelve to six. The student trains from scratch, but instead of learning from ground-truth labels alone, it learns to match the teacher's output distribution. Sometimes the loss function blends both: a term that matches the teacher's probabilities and a term that matches the true labels. The balance between them is a design choice, and the optimal mix varies by task.
Performance plateaus well before parity
The student reliably captures much of the teacher's behaviour, but the last few percentage points resist transfer. DistilBERT retained 97% of BERT's performance on the GLUE benchmark, but that remaining 3% gap persisted across tasks. The plateau appears to reflect a hard limit: the student's smaller architecture cannot represent everything the teacher learned, no matter how long it trains. Patterns that require deep composition or subtle context discrimination may simply exceed the student's capacity. Distillation compresses well-learned regularities effectively, but it does not create capacity that was never there.
The story
A large model takes hours to label a dataset, costs hundreds of dollars per day to serve, and will not fit on the hardware you can afford. A smaller model trains faster and runs cheaper, but when you train it normally on the original data, it performs far worse. Distillation offers a middle path: train the small model not on the data's labels, but on the large model's predictions.
The idea rests on the observation that a trained model's outputs contain more information than the ground-truth labels do. When a classifier sees an image of a cat, the correct label is simply "cat". But the model outputs a probability for every class. It might assign 0.87 to cat, 0.09 to dog, 0.03 to lynx, and tiny fractions to everything else. That distribution is a compressed summary of what the model learned: cats resemble lynxes more than they resemble cars, even though neither lynx nor car is correct here.
Hinton and his co-authors called this the dark knowledge in the model — knowledge that is present but hidden when you look only at the winning class. A student model trained to match these soft probability distributions, rather than the hard one-hot labels, learns not just what is correct but what is almost correct, what is plausible, what is wildly wrong. It learns the teacher's internal geometry.
The student is architecturally smaller. DistilBERT, for instance, has six layers where BERT has twelve. It trains from scratch, but its loss function penalises disagreement with the teacher's output distribution. Sometimes a second term is added that also penalises disagreement with the true labels, and the relative weight of the two terms becomes a tuning parameter. The student never sees the teacher's internal activations or weights directly; it learns only by imitation of behaviour.
What transfers is pattern. The teacher has learned which features co-occur, which inputs cluster together, which outputs are mutually exclusive. Much of that structure can be captured by a smaller model, because the structure itself is often simpler than the data it was extracted from. What does not transfer is capacity. If the teacher's reasoning depends on composing representations across many layers, or holding many hypotheses in parallel, the student may lack the representational space to do the same.
The result is a consistent empirical pattern: distilled models close much of the gap quickly, then plateau. DistilBERT retained 97% of BERT's performance on a standard benchmark, but the final 3% resisted further training. The student had learned everything its architecture permitted, and the rest required a larger model.
Why it mattered then
Hinton's 2015 paper arrived at a moment when neural networks were growing rapidly in size and computational cost. Ensembles of networks had been shown to improve performance, but deploying them was impractical. Distillation offered a way to compress an ensemble's collective knowledge into a single deployable model, or to transfer a large model's capabilities to a smaller one that could run on constrained hardware. The method was not entirely new — earlier work had explored training one model on another's outputs — but Hinton's formulation, particularly the temperature parameter for softening distributions, gave it a clear theoretical framing and demonstrated its effectiveness across multiple domains. It also opened a conceptual door: if a model's knowledge could be transferred without transferring its structure, then knowledge was in some sense independent of the architecture that produced it, a claim with implications for how we think about what models learn.
Why it matters now
Distillation has become a standard tool for deploying models in production, where latency and cost matter as much as accuracy. The DistilBERT result — 97% of performance at 60% of size and substantially faster inference — established a trade-off that many applications accept willingly. The technique is now used not only to compress single models but to transfer capabilities from proprietary models to open ones, from English-trained models to multilingual ones, and from general models to task-specific ones. It also exposes a deeper question that remains unresolved: why does the plateau occur where it does? The answer seems to involve the student's architectural capacity, but we lack a precise theory of what aspects of the teacher's behaviour require what kinds of capacity to reproduce. The 3% gap is not random noise; it represents something the teacher learned that the student cannot hold, and understanding what that something is would tell us more about what models learn and how they represent it.
The surprising detail
Hinton's original paper showed that distillation works even when the student has a completely different architecture from the teacher. You can distill a deep convolutional network into a shallow one, or a recurrent network into a feedforward one, as long as they operate on the same input and output spaces. This implies that much of what the teacher learned is not locked into its particular architectural choices — the knowledge exists at the level of input-output mappings, not internal structure. It also means you can use distillation to translate across architectural paradigms, which has been exploited to move capabilities from older architectures to newer, more efficient ones without retraining from scratch on the original data.
Remember this
The student learns the teacher's uncertainty, not just its answers. That transfers well, but only up to the student's architectural limit.
Test yourself
You distill a large model into a smaller one and the student reaches 95% of the teacher's accuracy, then stops improving. You have more of the teacher's outputs available. Will generating more synthetic training data from the teacher reliably close the remaining 5% gap?
No. The plateau reflects the student's architectural capacity, not the amount of training data. If the student's network is too shallow or too narrow to represent the patterns the teacher uses for the hardest examples, more data will not create the capacity that is missing. The student has likely already learned everything its architecture permits. Closing the gap would require a larger student model, not more training examples from the same teacher. This is why distillation results consistently report a residual performance gap that persists across datasets and training regimes — it is a capacity limit, not a data limit.
Go deeper
- Distilling the Knowledge in a Neural Network · arXiv · Geoffrey Hinton et al. · 2015-03-09
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter · arXiv · Victor Sanh et al. · 2019-10-02
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.