Feature Learning: The Lazy-to-Rich Transition

Take a network with a single hidden layer and make it very wide. Train it with gradient descent — the standard procedure that nudges every weight a little, each step, in whichever direction reduces the error. The loss falls and the network fits its training data. Then look inside. The hidden layer is almost exactly as it was before training began. Its features — the patterns each hidden unit responds to — were fixed by the random starting weights and were never revised. All the fitting happened in the final layer, which simply chose how strongly to count each of the random features it was handed for free.

This is called the lazy regime, and the cause is arithmetic rather than mystery. It helps to walk through it in steps. The network's output is a sum over all its hidden units. Keep adding units without adjusting anything and that sum grows without bound, so the standard scaling divides it by a factor that grows with the width. The wider the network, the smaller each unit's share of the output. Now recall what gradient descent does: it moves each weight in proportion to that weight's effect on the error. A weight with a tiny share of the output has a tiny effect on the error, so it receives a tiny nudge. Here is the step that makes the whole thing work. Multiply a tiny nudge by an enormous number of units and the output still moves plenty — many small contributions add up, which is why the loss falls. But inspect any single unit and almost nothing has happened to it. Push the width to infinity, under this scaling, and the features freeze exactly. The network then behaves like linear regression — a straight weighted sum — over a fixed set of random functions. The function the network computes changes a great deal; the parts that build it barely move at all.

None of this shows up in the loss curve, which is why the distinction is easy to miss. The instrument that does work is the hidden layer's similarity structure: for every pair of training inputs, record how alike their hidden representations are, giving a table of pairwise similarities. Compute that table twice — once at initialisation, when the weights are still random, and once after training. In a lazy run the 2 tables come out close to identical, even though the loss has dropped a long way. In a rich run they differ substantially, because the layer no longer describes the inputs the way random weights did. To know which happened, you have to look inside.

The frozen limit is convenient for theory, which is why it drew so much attention. A network whose features never move is a linear model in disguise, and linear models can be reasoned about: training becomes a well-understood fitting problem over a fixed set of basis functions, and whole chapters of classical mathematics apply. It gave the field its first rigorous handle on what gradient descent does to a heavily overparameterised network — one with far more weights than data points.

The trouble is that feature learning was supposed to be the point. A network beats a hand-designed feature map because it can decide, from the data, what its intermediate layers should describe. A model that keeps its initial random features and only reweights them has given up the very property that made deep learning interesting. So the lazy limit is both an achievement — a genuinely tractable description of a wide network — and a warning that the description may fit something other than the networks people actually train.

The other end is the rich regime. Here the hidden layers reorganise. Feed the network the same inputs before and after training and their internal representations will have shifted: inputs the task treats as equivalent come to sit closer together, and inputs it must tell apart move further apart. The similarity table changes, and it changes to reflect the labels rather than the random starting draw. This is what people usually mean by learning features.

What decides where a given run lands? Not width alone. The size of the starting weights matters just as much. Start them smaller and the network begins far from any fit, so the weights must travel a long way before the loss can fall — and travelling reorganises features. An output multiplier does similar work: divide the network's output by a larger constant and the internal weights must move more to produce the same predictions. The learning rate interacts with all of this; a rate tuned at one width can put a network of another width into a different regime. Depth complicates the arithmetic further. In a deep network the right update for one layer depends on what every other layer is currently doing, and that mutual dependence — each layer's correct move shifting as its neighbours move — is exactly what the single-hidden-layer analysis assumes away. Width alone does not obviously suppress it. Two networks of identical width, trained on identical data, can therefore sit at opposite ends purely because of how their weights were scaled at the start.

The honest summary is that lazy and rich are the 2 ends of a continuum with unmapped ground between them. Runs in the middle move their features somewhat. Some layers of a single network can be lazier than others. Whether the successful networks people actually deploy sit near the rich end, the lazy end, or somewhere between is not settled — there is no agreed threshold where one regime becomes the other, and the answer likely depends on architecture, data and training schedule. What is firm is the mechanism: the standard scaling makes each weight's movement shrink as width grows, and anything that forces the weights to travel further before the loss can fall pushes a network out of laziness.