II · THE IDEA · ARTIFICIAL INTELLIGENCE
The Information Bottleneck
▶ Listen · narrated
On this account, most of training is not learning the answers. The answers come early. The long remainder is spent forgetting: shedding detail about the inputs that the task never needed.
At a glance
- Core quantity
- Mutual information: uncertainty about one thing that knowing another removes
- The trade-off
- Keep few bits about the input, keep many bits about the label
- The plot
- Information plane: input information on one axis, label information on the other
- Claimed phases
- A short fitting phase, then a much longer compression phase
- Evidence base
- Small networks, a synthetic task, activations binned to estimate the bits
Think of someone taking notes in a lecture they will be examined on. A word-for-word transcript keeps everything: every cough, every false start, every aside. The answer to any exam question is in there somewhere, but so is a great deal that will never be asked about. Good notes are shorter. They have dropped the coughs and kept the argument.
The information bottleneck view says a network's layers do something similar, and that you can score each layer with two measurements. First: how much does this layer still reveal about the exact input it came from? Second: how much does it reveal about the right answer? You want the first number small and the second large.
The 2017 experiments added a claim about timing. Early in training, layers get better at the answer, and they also grow more detailed about the input, more like a transcript. Then, in those runs, the direction reverses for a long stretch: the layers keep the answer but shed the input detail, more like notes. Most of training, on this account, is the note-making, not the listening. And the shedding is claimed to be why the network copes with new examples: a layer that has forgotten the quirks of its training examples cannot lean on them, so it must rely on what truly signals the answer.
One caveat. Neither measurement can be taken directly; both must be estimated, and the estimate involves a judgement call by the researchers. So how general the pattern is remains an open question.
Treat a network as a Markov chain from input X through hidden representations T1, T2, ... to the output, with label Y. For each Ti you can in principle compute two mutual informations: I(X; Ti), the bits Ti carries about the input, and I(Y; Ti), the bits it carries about the label. Because information can only be lost along the chain, no layer can carry more of either quantity than the layer before it. The information bottleneck objective asks for representations that minimise I(X; T) while keeping I(Y; T) high, which traces an optimal trade-off curve. Tishby and Zaslavsky's 2015 paper proposes reading deep networks against that curve, layer by layer.
The diagnostic is the information plane: plot each layer as a point, I(X; T) on the horizontal axis, I(Y; T) on the vertical. Plot the same layer at successive training steps and you get a trajectory. Shwartz-Ziv and Tishby's 2017 experiments report that these trajectories have two regimes: a brief fitting phase in which both quantities rise, then a much longer compression phase in which I(X; T) falls while I(Y; T) holds roughly steady. Their reading ties the compression phase to generalisation: the less input information a representation retains, the less it has available to overfit with.
The limitations carry weight. For deterministic layers with continuous activations, I(X; T) is not straightforward to estimate. The 2017 work discretises activations into bins and computes information on the resulting discrete variables, so every result is conditional on the bin width and on the network being small enough for the empirical joint distribution to be populated; the task is synthetic for the same reason. The reported phase structure should therefore be read as an observation about particular networks under a particular estimator, not as a law of training, and the compression phase in particular has been disputed. What is robust and portable is the pair of axes: I(X; T) and I(Y; T) are well-posed questions to ask of any representation, and asking them separately tells you more than asking whether a layer is useful.
Look closer
Where the layers land
Plot one point per layer. Its horizontal position is how much that layer still tells you about the raw input; its vertical position is how much it tells you about the label. In the 2017 experiments the deeper layers sit further to the left, holding less of the input, while staying nearly as high up, having kept most of the signal that predicts the answer. That combination is the whole claim in one picture: successive layers shed input detail and hold on to task-relevant detail.
The turn in the trajectory
Watch a single layer across training rather than only at the end, and it traces a path. First it moves up and to the right: it is getting better at the label, and along the way it is also carrying more information about the particular input. Then, in the reported runs, it turns and drifts left for a long stretch, losing input information while the label information stays roughly flat. That turn is the boundary between the two phases the authors name, and it arrives early, well before training stops.
The axis that must be estimated
Neither axis can be read off the network directly. Mutual information between a layer of continuous activations and anything else has to be estimated, and in the 2017 work this was done by chopping each neuron's output range into bins and treating the resulting pattern of bins as a discrete symbol. The bin width is a free parameter chosen by the analyst. So the position of the points, and arguably whether the leftward drift shows up at all, can depend on a decision made during the analysis rather than on anything the network did.
The story
The idea starts with a plain question. A network's job is to turn an input into a label, the answer the task wants. So what should the layers in between actually hold? The obvious reply is: whatever helps predict the label, and nothing else. In 2015, Naftali Tishby and Noga Zaslavsky took that reply seriously enough to turn it into a way of scoring every layer of a deep network.
Their measuring stick is mutual information. Mutual information between two things is how much your uncertainty about one shrinks when you learn the other, counted in bits. If a layer's activations, the numbers the layer outputs, tell you nothing about the label, the two share no information. If knowing the layer pins the label down completely, the mutual information equals all the uncertainty there was to remove. The same ruler can be turned round to face the input: how much does the layer reveal about which exact example produced it?
Those two numbers pull against each other, and the pull is the bottleneck. A layer that simply copies its input keeps everything useful about the label, but it also keeps every irrelevant detail: the lighting, the slant of the handwriting, the background. A layer that discards too much destroys the label along with the clutter. The information bottleneck says a good representation sits at the best available compromise: keep as few bits about the input as possible while keeping as many bits about the label as possible. Each layer of a deep network, on this view, is one attempt at that compromise. And because each layer can only work with what the previous layer passed on, information can only be lost as you go deeper, never recovered.
Two numbers per layer means you can draw them. Put input information along the bottom, label information up the side, and each layer becomes one dot. The chart is called the information plane, and it is what made these papers travel. A network stops being an unreadable pile of weights and becomes a handful of dots you can read: early layers on the right, still close to the raw input; deeper layers further left, having discarded more, and ideally no lower down, having lost little of the answer.
The 2017 follow-up, by Ravid Shwartz-Ziv and Naftali Tishby, did something bolder than plotting the finished network. It plotted the layers repeatedly during training, so each layer traces a path across the chart. The reported path has two stages. In the first, brief stage, a layer moves up and to the right: it is getting better at the label, and while doing so it also soaks up more detail about each particular input. Then the path turns. Label information levels off, and input information starts to fall, and keeps falling for far longer than the first stage lasted. The authors call these the fitting phase and the compression phase.
If that picture is right, it changes what most of training is for. The part we watch, the error dropping until the answers are right, is the short part. The long stretch afterwards, when the error barely moves and it is tempting to think nothing is happening, would be the network forgetting: stripping out the features of each example that do not bear on the answer. And the forgetting, on this account, is why the network then works on new examples. A representation that no longer remembers the incidental quirks of the training examples cannot memorise its way to the right answers; it is forced to rely on what genuinely predicts the label, and that is exactly what carries over to data it has never seen.
It matters what was actually shown, because the framing is broad and the demonstration narrow. The 2017 experiments used small networks on an artificial task, chosen partly so the information quantities could be estimated at all. Mutual information between continuous numbers cannot be read off a network directly; it has to be estimated. Here the estimate came from chopping each neuron's output range into bins, which is to say rounding every value to the nearest notch on a scale, and the width of those bins was a choice made by the analysts. Whether the two phases appear in large modern networks, on real data, under other estimators, is not something these two papers settle, and the compression phase has been contested rather than accepted as established.
What survives either way is the pair of questions. Ask of any representation: how much does it still hold about the input, and how much about the task? You can ask that without believing in phases, and it is a sharper question than asking whether a layer is good.
Why it mattered then
In 2015 the standard complaint about deep networks was that they worked and nobody could say why. Most explanations were local: this filter finds edges, that unit lights up for a shape. What was missing was a quantity you could compute for any layer of any network, whether or not a human could see what the layer was for. The information bottleneck offered exactly that. Just two numbers per layer, both counted in bits, both defined for any architecture, and both borrowed from information theory, the field that had already given communication engineering its limits and its vocabulary. The 2017 paper mattered for a different reason. It offered an explanation for something practitioners saw every day and could not account for: the long stretch of training after the error has essentially stopped falling. If that stretch is a compression phase, then a familiar and slightly boring part of the routine turns out to be where the ability to handle new data is earned. The claim was concrete enough to test, which is why it drew both enthusiasm and immediate scrutiny.
Why it matters now
The pair of questions has aged better than the specific experiments. Whenever you build a representation and reuse it, you are choosing, deliberately or not, how much of the input to keep, and the two ways of getting it wrong cost you differently. Keep too much, and the representation carries incidental detail the next stage can latch onto, including detail you would rather it never saw. Keep too little, and the signal the task needs has been destroyed before the model that needs it gets a look. That trade-off appears wherever one model's output feeds another: embeddings reused across tasks, features frozen and shipped, intermediate activations logged or probed. It also gives a clean way to think about privacy, because information about the input is precisely what a representation might be leaking beyond what its task requires. The cautionary lesson is just as durable. Neither axis of the information plane is a reading taken off the machine; both are estimates, and the estimation involves choices. A result that appears or disappears when you change a bin width is telling you about your measurement procedure at least as much as about the network. That habit of suspicion is worth carrying into any claim about what is happening inside a model.
The surprising detail
The most striking part of the 2017 report is how the time divides. The phase that produces correct answers is the short one; the phase in which the network appears to be discarding information about its inputs runs far longer. If that is right, then for most of a training run the error curve sits nearly still while the representation changes substantially, and what is changing is not what the network knows about the answer but what it has stopped knowing about the examples. It is an odd sort of claim, because it describes progress the loss curve cannot show. It is also the part that has drawn the most argument, since the whole reading rests on an estimate of information whose resolution was chosen by the people doing the estimating.
What is disputed
The theoretical trade-off and the empirical claim should be judged separately. That a good representation balances input bits against label bits is a framing, and a defensible one. That training reliably splits into a short fitting phase and a long compression phase is a specific empirical claim, demonstrated here on small networks and a synthetic task, using an estimate of mutual information that depends on a hand-chosen binning resolution. It has been contested rather than settled, and nothing in these two papers establishes that the phases appear in large networks on real data.
Remember this
Score a layer with two numbers: bits kept about the input, bits kept about the answer. A good layer forgets as much of the first as it can without losing the second.
Test yourself
A colleague repeats the information plane experiment on a larger network and reports no compression phase at all: input information never falls. Before concluding the theory is wrong, what should you check, and why?
Check how the information was estimated. Mutual information between continuous activations and anything else is not read off the network; in the 2017 work it was estimated by binning each neuron's output range and treating the pattern of bins as a discrete symbol. Bin width is a choice, and a coarse or fine choice can change where the points sit and whether any leftward drift is visible. Then check that the comparison is fair: the original result came from small networks on a synthetic task, measured per layer over the whole of training, not just at the end. The deeper lesson is that both axes of the information plane are estimates with free parameters, so a disagreement about phases may be a disagreement about measurement rather than about what the network did.
Go deeper
- [1503.02406] Deep Learning and the Information Bottleneck Principle · arxiv.org
- [1703.00810] Opening the Black Box of Deep Neural Networks via Information · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.