II · THE IDEA · ARTIFICIAL INTELLIGENCE
Knowledge Distillation from Multiple Teachers
▶ Listen · narrated
Running an ensemble at serving time is accurate but expensive. Distillation folds those teachers into one smaller student that approximates their combined judgement without carrying every parameter.
At a glance
- Core idea
- Train a compact student on soft outputs from one or more teachers
- Signal used
- Class probability distributions, not only hard labels
- Practical gain
- Ensemble-like behaviour with a single forward pass at inference
- Control knob
- Softmax temperature that smooths or sharpens teacher distributions
Think of several experienced markers grading the same exam. Instead of only writing the final grade, each marker also notes how plausible every other grade seemed. You average those fuller mark sheets and train a single junior marker to reproduce the average sheet, not just the final grade. After that training you only need the junior marker.
That is knowledge distillation with multiple teachers. The teachers are larger or specialised models. Their soft predictions—probability distributions over classes—are combined and used as training targets for a smaller student. A temperature parameter softens those distributions during training so fine differences stay visible. At serving time the student runs alone at normal sharpness, carrying much of the committee’s judgement without paying the committee’s cost.
Distillation trains a student network to match teacher predictive distributions. For input x, each teacher produces logits z_i; a temperature T > 1 yields soft targets p_i = softmax(z_i / T). With multiple teachers the p_i are aggregated (for example by averaging) into a single target distribution p̄. The student, with logits v, is trained with a distillation loss such as KL(p̄ || softmax(v / T)) scaled by T², often plus a standard cross-entropy term against the true label at T = 1.
The T² factor keeps gradient magnitudes stable as T changes. High T flattens distributions so that logit differences among low-probability classes contribute to the gradient; at inference the student uses T = 1. The method transfers “dark knowledge” in the relative probabilities of incorrect classes. Limitations include student capacity bottlenecks, mismatch when teachers disagree sharply, sensitivity to T and loss weighting, and no guarantee of full ensemble fidelity—only that a compact model can approximate much of the teachers’ generalisation at one forward pass.
Look closer
Soft targets carry relative structure
A hard label says only which class won. A teacher’s full probability vector says how the remaining probability is shared among the losers. That residual structure is useful training signal: two classes that the teacher confuses are treated as more alike than two classes it separates cleanly, and the student can absorb that similarity without ever seeing an explicit similarity matrix.
Temperature reshapes the teaching signal
Raising the softmax temperature flattens the teacher’s distribution so that small logits become visible as non-negligible probabilities. The student is usually trained on these softened targets, then run at temperature one at inference. Too low a temperature collapses back toward hard labels; too high washes out the ranking the teacher had learned.
Many teachers, one student
When several specialised models each contribute predictions, the student can be trained against an aggregate of those soft outputs. The ensemble never needs to run in production: only the student does. The editorial payoff is collective behaviour retained in a single compact network, with inference cost closer to one model than to the full set of teachers.
The story
Knowledge distillation starts from a simple mismatch between training and deployment. At training time it is acceptable, even desirable, to use large models or ensembles: they have capacity, they can be specialised, and wall-clock cost is paid once. At inference time every extra parameter and every extra forward pass is paid again for each request. The aim of distillation is to move the useful behaviour of the expensive side onto a cheaper network that will actually be served.
The transfer does not copy weights. It copies a response pattern. A teacher, or a set of teachers, produces a probability distribution over classes for each training example. Those distributions—soft targets—are richer than the original one-hot labels because they encode which alternatives the teacher considers plausible. A student network is then trained to match those distributions, typically with a loss on the softened probabilities, often alongside an ordinary supervised loss on the true labels. Matching the teacher’s full output teaches the student not only what to predict but how the teacher spreads uncertainty.
Temperature is the practical handle on how much structure is visible in those soft targets. In the softmax, dividing logits by a temperature greater than one flattens the distribution. Information that would otherwise sit near zero becomes large enough to influence gradients. After training, the student reverts to temperature one so that its deployed predictions are sharp again. The same temperature is applied consistently when comparing teacher and student so that the loss measures a meaningful divergence between similarly scaled distributions.
With multiple teachers the same idea extends naturally. Each teacher may be specialised—different data, different architectures, different inductive biases—and each emits its own soft prediction. Those predictions can be combined into a single teaching distribution for the student. The student never has to host the teachers’ parameters; it only has to imitate the aggregate behaviour on the examples it sees. Once training is finished, inference is a single forward pass through the student.
What is preserved is not guaranteed to be everything the ensemble knew. Distillation is an approximation. Capacity limits, differences in architecture, and the particular mix of soft-target loss versus hard-label loss all shape how faithfully the student follows the teachers. The method’s claim, supported by the original distillation work, is more modest and more useful: a well-trained student can capture a large fraction of the teachers’ generalisation behaviour at far lower serving cost than running the ensemble itself.
Why it mattered then
Large ensembles improved accuracy but made production serving awkward: latency, memory, and energy all scaled with the number of models. Distillation offered a way to keep much of that accuracy while returning to a single network at inference. Soft targets mattered because labelled data was finite; the teacher’s full distribution extracted more training signal from each example than a hard class index alone. That mattered in settings where an expensive model or committee could be trained offline and a compact student had to run under tighter constraints.
Why it matters now
The same asymmetry still defines modern deployment. Training budgets and serving budgets diverge. Specialised models are routinely combined in research and offline pipelines, yet product surfaces often need one fast network. Distilling several teachers into one student remains a direct response: pay for breadth during training, ship a single compact artefact. Temperature, soft targets, and a blend of distillation loss with ordinary supervision are still the basic controls when open-weight models are compressed or specialised for on-device or high-throughput use.
The surprising detail
The valuable part of the teacher’s output is often not the winning class but the small probabilities on the other classes. Those faint values encode similarities the hard label never states. Raising the softmax temperature is what makes them audible to the student; without it, distillation collapses toward ordinary supervised training and much of the transfer disappears.
What is disputed
How much of an ensemble’s accuracy a student retains depends on capacity, architecture match, temperature, and the mix of soft and hard losses. The original work shows strong transfer in the regimes studied; it does not claim that every ensemble compresses equally well for every student size.
Remember this
Distillation moves an ensemble’s behaviour into one student by training on soft predictions, so inference pays for a single model rather than the full committee.
Test yourself
You can train a student on hard labels alone, or on a blend of hard labels and high-temperature soft outputs from several specialised teachers. In the second case, what extra information is the student receiving, and what deployment cost does that avoid compared with running the teachers as an ensemble?
The soft outputs supply the teachers’ full class distributions—how probability is shared among non-winning classes—so the student learns similarity structure and uncertainty patterns that one-hot labels omit. At deployment only the student runs, so you avoid the memory, latency and energy of executing every teacher on each request while still approximating much of their combined behaviour.
Go deeper
- [1503.02531] Distilling the Knowledge in a Neural Network · arxiv.org
- [1805.06309] A note on the permutation behaviour of the polynomial $g_{n,q}$ · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.