II · THE IDEA · ARTIFICIAL INTELLIGENCE

Pruning

Reasoning and architecture · Weight removal after training · 50–90% of parameters

▶ Listen · narrated

Training a billion-parameter model costs millions. Removing eighty per cent of the weights after training costs hours, and the model still works. The economics look miraculous until you try to serve it.

At a glance

What it is: Setting a large fraction of trained weights to zero, permanently
Unstructured pruning: Remove individual weights anywhere; pattern is irregular
Structured pruning: Remove entire rows, columns, attention heads or layers; pattern is regular
Reported tolerance: SparseGPT demonstrated 60% sparsity on large models with minimal accuracy loss

Imagine a thick book where half the pages are blank. You can make the book thinner by removing the blank pages, and it weighs less and fits in a smaller bag. That is structured pruning: you remove entire pages, and the book becomes genuinely smaller. Now imagine the blank pages are scattered throughout, one paragraph here, one sentence there. You can still count them and say half the content is missing, but the book is still the same thickness because the pages are still there. That is unstructured pruning. In a neural network, weights are numbers that get multiplied during inference. Many of them are close to zero and contribute almost nothing. Set them to exactly zero and the output barely changes. But if those zeros are scattered throughout the weight matrices, the computer still has to check every position, and the work is nearly the same. Only when you remove entire rows or columns does the matrix shrink and the computation speed up.

Pruning sets a subset of trained weights to zero, either by magnitude threshold or by learned importance scores. Unstructured pruning removes individual weights anywhere in the network, producing irregular sparsity patterns. The resulting sparse matrices can be stored in compressed formats like CSR, reducing memory footprint by fifty per cent or more. However, inference on GPUs using standard dense BLAS operations sees negligible speedup, because memory access patterns remain unchanged and bandwidth is the bottleneck. Sparse matrix kernels exist but add overhead and are not widely supported in production inference frameworks.

Structured pruning removes entire rows, columns, channels, attention heads or layers, producing smaller dense matrices. A linear layer with weight matrix W of shape (m, n) might become (m, n/2) by removing half the output dimensions. This is directly compatible with standard dense operations and yields proportional reductions in FLOPs and memory bandwidth. The cost is accuracy: structured constraints mean you cannot select weights individually, and importance scores must be aggregated over groups. Empirically, structured pruning degrades performance faster than unstructured pruning at equivalent sparsity.

SparseGPT introduced a layer-wise pruning method that processes each layer independently using a second-order approximation to minimise output error. This allows pruning very large models in a few GPU-hours without retraining. The method achieves 60% unstructured sparsity on models with tens of billions of parameters with minimal perplexity increase, but the sparsity does not translate to faster inference without sparse kernels.

The Lottery Ticket Hypothesis posits that dense networks contain sparse subnetworks that, when trained in isolation from the same initialisation, reach comparable accuracy. Finding these subnetworks at scale remains difficult, but the hypothesis reframes pruning as discovering which weights matter rather than compressing a trained model.

Look closer

Most weights do very little
In a trained network, the distribution of weight magnitudes is highly uneven. Many weights sit close to zero, and zeroing them changes the output barely at all. Others carry substantial signal. Pruning exploits this: identify the weights with the smallest apparent contribution, set them to zero, and the model's behaviour shifts only slightly. The Lottery Ticket Hypothesis went further, suggesting that sparse subnetworks capable of training to full accuracy exist inside the dense network from initialisation, though finding them reliably remains difficult.
Unstructured sparsity is invisible to hardware
If you remove individual weights scattered throughout a matrix, the matrix is still the same shape. A GPU multiplying that matrix by a vector must still load every position, check whether it is zero, and perform the same number of memory operations. The arithmetic may be cheaper, but memory bandwidth — the usual bottleneck — is unchanged. Sparse matrix formats exist, but they add overhead and most production inference stacks do not use them. So a 60% sparse unstructured model occupies less disk space and uses less memory, but runs at nearly the same speed as the dense original.
Structured sparsity changes the shape
Remove entire rows or columns and the matrix becomes smaller. A layer that was 4096 by 4096 might become 4096 by 2048, and standard dense matrix operations now do half the work. This is genuinely faster on ordinary hardware. The cost is that you cannot choose weights individually: you must delete them in groups, and the groups are chosen by the architecture rather than by their contribution. Structured pruning therefore tends to degrade accuracy more quickly than unstructured pruning at the same sparsity level.

The story

Pruning begins with a trained model. You have spent days or weeks adjusting billions of weights so that the network produces useful outputs. Now you examine those weights and discover that a large fraction of them contribute almost nothing. Set them to zero and the model still works.

The simplest approach is magnitude-based: sort all weights by absolute value and delete the smallest fifty per cent. This is unstructured pruning, because the deleted weights are scattered throughout the network wherever they happen to be small. The pattern is irregular. SparseGPT, described by Frantar and others, demonstrated that models with tens of billions of parameters tolerate 60% unstructured sparsity with minimal loss in accuracy, and the pruning itself can be done in a few hours on a single machine by processing the network layer by layer.

The result is a sparse network: most entries in the weight matrices are now zero. You can store it more compactly, because you need only record the non-zero values and their positions. You can load it into memory using less space. But when you run inference, you discover that the model is barely faster than before you pruned it. On a GPU performing dense matrix multiplication, a zero still occupies a position in memory. The hardware still loads it, still processes it, still writes the result. Unstructured sparsity saves space but not time.

Structured pruning offers a different trade. Instead of removing individual weights, you remove entire structural units: a complete attention head, an entire row of a weight matrix, a whole layer. The resulting network is smaller in every dimension. A matrix that was 4096 by 4096 becomes 4096 by 2048, and the GPU now does half the work with standard operations. This is genuinely faster. The cost is accuracy. When you must delete weights in groups rather than individually, you cannot be as selective, and performance degrades more quickly as sparsity increases.

The Lottery Ticket Hypothesis, proposed by Frankle and others, suggested something stranger: that the sparse networks we find by pruning might have been trainable from the start, if only we had known which weights to keep. The hypothesis is that a large network contains many smaller subnetworks, and some of them — the winning tickets — could have reached full accuracy if trained in isolation with the right initialisation. Finding these tickets reliably remains an open problem, but the hypothesis reframed pruning as a search problem rather than a compression problem.

Why it mattered then

Pruning emerged as a practical response to the cost of deploying large models. Training a billion-parameter network required substantial resources, but once trained, much of that capacity appeared redundant. Early pruning work showed that networks tolerate substantial weight removal, and the Lottery Ticket Hypothesis in 2018 suggested that the redundancy might be fundamental rather than incidental. SparseGPT extended these ideas to models with tens of billions of parameters, demonstrating that very large language models could be pruned in a few hours without retraining. The appeal was economic: if a model could be made smaller and cheaper to serve without losing capability, the same hardware could support more users or the same service could run on cheaper hardware.

Why it matters now

Pruning remains relevant because model size continues to grow faster than hardware efficiency. A 70-billion-parameter model occupies more than 140 gigabytes in half precision, and serving it requires expensive accelerators. Unstructured pruning can reduce storage and memory by half or more, which matters for organisations running many models or deploying to edge devices. Structured pruning offers actual speed improvements, though at a steeper accuracy cost. The tension between the two approaches has not been resolved. Most production systems still run dense models, because the tooling is mature and the performance is predictable. Sparse inference remains a research area with commercial interest but limited deployment. The Lottery Ticket Hypothesis continues to generate work on training efficiency, asking whether we could skip the dense phase entirely and train only the subnetwork that matters.

The surprising detail

The Lottery Ticket Hypothesis implies that most of the weights in a large network may never have been necessary. If a sparse subnetwork can train to full accuracy given the right initialisation, then the dense network's main role may be to make the search for that subnetwork easier, not to provide representational capacity. This is not proven for very large models — finding the winning ticket at scale remains impractical — but the hypothesis has held for smaller networks across multiple domains. It suggests that overparameterisation is a search strategy rather than a storage strategy, which reframes how we think about model size.

Remember this

Unstructured pruning saves space but rarely saves time. Structured pruning saves both, but costs more accuracy.

Test yourself

You prune a model to 50% sparsity using an unstructured method and measure inference latency. It is almost unchanged. Your colleague suggests pruning to 50% structured sparsity instead. What changes, and what stays the same?

Go deeper

Image: Original diagram, The Daily Triptych. Licence: Original work. Source.

← Back to day 71