The Lottery Ticket Hypothesis

A neural network stores everything it learns in its weights — the numbers attached to each connection between its units. Before training, those numbers are set at random. Training nudges them, step by step, until the network does its job. Afterwards, it turns out that most of the weights were barely contributing. Pruning is the practice of deleting them: train a large network, remove the connections that mattered least, and keep the rest. Done carefully, this can cut more than 90 per cent of the weights without hurting accuracy, which is why pruning is the standard way to make a finished model smaller and faster to run.

That raises an obvious question. If a tenth of the network was enough in the end, why not build the small network in the first place and skip the expensive part, which is training the big one? Practitioners had tried. Take the sparse shape that pruning leaves behind — the same connections, nothing more — give it fresh random starting values, and train it from scratch. It trains poorly and ends up less accurate. The small shape works after pruning but not before, and nobody could say why.

Frankle and Carbin changed exactly one thing. Before training their dense network, they wrote down every one of its random starting values. Then they trained it, pruned it, and kept the record of which connections survived. Now the unusual step: instead of giving those survivors fresh random values, they reset each one to the exact random number it had held before training began. The deleted connections stayed deleted. Then they trained this sparse network, from that particular starting point, for no more steps than the original had used.

It worked. In their experiments — fully-connected and convolutional feed-forward networks, tested on the MNIST and CIFAR10 image datasets — the sparse network matched the accuracy of the dense one it came from. The subnetworks they found held fewer than 10 to 20 per cent of the original weights, and above that size they did better than match: they learned faster and reached higher test accuracy than the full network. So the sparse shape was never the problem. The problem was the starting values. A pruned network trains well only when its surviving connections begin from the same random numbers they held in the original network.

The authors call such a subnetwork a winning ticket, and the name explains their reading of the result. Setting up a large network at random is like buying a great many lottery tickets at once: each possible combination of connections and starting values is one ticket. Training the whole network, then pruning it, reveals which combination happened to win. Their hypothesis, stated carefully, says a randomly initialised dense network contains some subnetwork that, kept at its original starting values and trained on its own, matches the full network's accuracy within the same number of training steps. Every clause is load-bearing. Not any subnetwork — one that exists in there somewhere. Not any starting values — the ones it was born with. Not unlimited training — the same budget.

The method is honest about its own limit. You only learn which connections mattered by training the dense network and pruning it. So the procedure saves no training cost. It proves that a small trainable network was in there; it does not tell you how to find one without doing the expensive run first. Most of the follow-up work lives in that gap.

Two years later, Malach and colleagues proved something stronger on paper. For a random network with enough spare capacity, they showed, you can find a subnetwork that approximates a target network with no training at all — you delete connections and never adjust the survivors. Their title puts it plainly: pruning is all you need. Read that way, a large random network is not a blank surface waiting for training to write on it. It already contains, by chance, an approximation of the function you want, and training is one way of uncovering it.