II · THE IDEA · ARTIFICIAL INTELLIGENCE
Quantisation
▶ Listen · narrated
The difference between a model you can run locally and one you cannot is often just the bits per weight. Fewer bits mean smaller files and faster loading—with subtler trade-offs.
At a glance
- What it is
- Representing each learned weight with fewer bits than the original training precision
- Common formats
- 16-bit float (training default), 8-bit integer, 4-bit integer
- Memory saving
- 4-bit uses one quarter the space of 16-bit; an order of magnitude for large models
- Where loss appears
- Outlier features, rare tokens, fine distinctions in long-tail distributions
Think of each weight as a measurement on a ruler. A 16-bit weight can distinguish between positions a fraction of a millimetre apart. A 4-bit weight has only sixteen marks on the entire ruler, so many positions that were distinct round to the same mark. For most weights this does not matter — the model's behaviour comes from billions of them working together, and small errors average out. But a few weights carry unusual importance, and rounding them costs more. The trick is to identify those outliers and keep them at higher precision while compressing everything else. That is what methods like LLM.int8() do: mixed precision, targeted where it matters. The result is a model that takes up far less space and runs faster, with output that is usually close enough to the original that the difference is hard to spot.
Quantisation maps a continuous range of floating-point weight values onto a discrete set of integer levels. In uniform quantisation, you compute the minimum and maximum weight in a tensor, divide that range into 2^n equal intervals (256 for 8-bit, 16 for 4-bit), and round each weight to the nearest level. The quantised weight is stored as an integer, and a scale factor and zero-point are stored per tensor (or per channel, in finer-grained schemes) to reconstruct an approximation of the original value during inference. GPTQ improves on naive rounding by solving for the quantisation parameters that minimise reconstruction error against a calibration set, layer by layer, using a second-order approximation to avoid retraining. LLM.int8() takes a different approach: it identifies weights with absolute values above a threshold (typically the 99.9th percentile) and keeps those in 16-bit, processing them separately in a mixed-precision matmul, while quantising the rest to 8-bit. The outlier weights are few enough that the memory overhead is small, but their contribution to the output is large enough that preserving their precision recovers most of the quality loss. Both methods are post-training: the model is trained normally in high precision, and quantisation is applied afterward as a form of lossy compression. The quantised model is smaller and faster to load, and inference requires less memory bandwidth, but the outputs are approximations, not bit-identical reproductions.
Look closer
Not all weights degrade equally
Dettmers and colleagues observed that a small fraction of weight values — often less than 0.1 per cent — carry disproportionate importance. These outliers appear systematically in certain layers and certain dimensions, and quantising them aggressively causes measurable perplexity jumps even when the bulk of the model tolerates it well. LLM.int8() keeps these outlier features in higher precision and quantises everything else, a mixed-precision scheme that recovers most of the quality loss at modest additional cost.
Post-training quantisation is a lossy compression step
The model was trained with 16-bit or 32-bit floats. Quantisation happens afterward, mapping a continuous range onto a small set of discrete levels — 256 levels for 8-bit, sixteen levels for 4-bit. GPTQ chooses the mapping carefully, layer by layer, minimising the error against a small calibration dataset. It is not retraining; it is finding the least-damaging way to round. The result is a smaller file that produces similar — not identical — outputs.
Four bits became the practical default, not the theoretical optimum
At 4-bit, a 70-billion-parameter model fits in the unified memory of a high-end laptop or a single consumer GPU. At 8-bit it does not, and at 2-bit the quality loss becomes harder to ignore in ordinary use. The threshold is as much about what hardware people own as about what the mathematics permits. GPTQ and similar methods made 4-bit quantisation reliable enough that the trade-off — half the memory of 8-bit, quality still close to the original — became the default for local inference.
The story
A large language model stores tens of billions of learned weights, each a number that encodes part of what the model knows. During training, these weights are typically represented as 16-bit floating-point numbers, a format that gives enough precision to make stable gradient updates. Once training finishes, that precision is often no longer necessary. Quantisation is the process of storing those weights in fewer bits — 8, 4, or even fewer — shrinking the model's memory footprint without retraining it from scratch.
The simplest form of quantisation is uniform rounding: map the range of weight values onto a smaller set of discrete levels and round each weight to the nearest one. With 8 bits you have 256 levels; with 4 bits, sixteen. The immediate benefit is storage: a 4-bit model occupies one quarter the disk space and memory of the 16-bit original. For a 70-billion-parameter model, that is the difference between 140 gigabytes and 35, which is the difference between requiring a data-centre GPU and fitting on a laptop.
The cost is precision. Weights that were finely distinguished in 16-bit become identical in 4-bit if they round to the same level. For most weights this matters little. The model's behaviour emerges from billions of parameters working together, and small rounding errors in the majority wash out. But a small fraction of weights — the outliers that Dettmers and colleagues identified in LLM.int8() — carry outsize influence. Quantise those aggressively and quality degrades measurably, even when the rest of the model tolerates it.
GPTQ, published by Frantar and colleagues, approaches quantisation as an optimisation problem. For each layer, it finds the mapping from high-precision weights to low-precision integers that minimises error against a small calibration dataset. The method is post-training: no gradients, no backpropagation, just careful rounding informed by how the weights actually behave on real data. The result is a 4-bit model that produces outputs close to the 16-bit original, close enough that the difference is often hard to detect in ordinary use.
Four bits became the practical default not because it is the theoretical optimum but because it is the point where the trade-offs align with the hardware people own. At 4-bit, large models fit in consumer memory. At 8-bit they often do not, and at 2-bit the quality loss becomes harder to ignore. The threshold is as much social as technical.
Why it mattered then
LLM.int8() and GPTQ arrived in 2022 and 2023, at a moment when open-weight models were becoming large enough to be useful and too large to run on most personal hardware. The methods made it possible to take a 30-billion or 70-billion parameter model and run it on a single GPU or a high-end laptop, without access to a cluster or a cloud account. That shift mattered because it moved inference out of the data centre and into environments where privacy, cost and control were different. Quantisation was not new — the idea of storing weights in fewer bits is older than deep learning — but the specific techniques that made 4-bit quantisation reliable for billion-parameter language models were new, and they arrived just as the models themselves were becoming widely available.
Why it matters now
Quantisation is now the default path for anyone running open-weight models locally. The 4-bit versions are often the first ones published alongside the original weights, and the tooling assumes you will use them unless you have a reason not to. The trade-off — smaller, faster, slightly less precise — has been tested widely enough that the risks are understood. For most tasks, most users cannot reliably distinguish 4-bit output from 16-bit output, which means the quality loss is real but tolerable. The practical consequence is that models that would have required institutional resources two years ago now run on hardware individuals own, which changes who can deploy them, where, and under what terms.
The surprising detail
The outlier features that make mixed-precision quantisation necessary are not evenly distributed across the model. Dettmers and colleagues found them concentrated in specific layers and specific dimensions, and their presence is consistent across different model families. Why those particular dimensions develop outsize importance during training is not fully explained. The features are not rare in the sense of being unused — they activate frequently — but their weight values sit far from the mean, and quantising them to the same grid as everything else causes disproportionate damage. LLM.int8() treats them specially, keeping them in higher precision while quantising the rest, which suggests that the model's learned structure is less uniform than the training objective might lead you to expect.
Remember this
Quantisation is lossy compression applied after training. Four bits became the default because it fits the models people want to run onto the hardware they own.
Test yourself
You quantise a 70-billion-parameter model to 4-bit and notice that it handles common queries well but struggles with rare proper nouns and technical jargon. Explain why quantisation might degrade performance unevenly across the vocabulary.
Rare tokens and unusual vocabulary were underrepresented during training, so their associated weights and embeddings are less well-tuned and may already sit in noisier regions of the parameter space. Quantisation adds rounding error uniformly, but that error is more damaging where the signal was already weak. Additionally, rare tokens may depend more heavily on the outlier features that carry outsize influence — if those features are quantised aggressively, the model loses precisely the distinctions it needs for edge cases. Common words, by contrast, are overdetermined: many weights contribute to them, so small errors in any one weight matter less. The result is that quantisation often degrades the long tail of the distribution faster than the head, which is why the loss shows up first in proper nouns, technical terms and other low-frequency vocabulary.
Go deeper
- LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale · arXiv · Tim Dettmers et al. · 2022-08-15
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers · arXiv · Elias Frantar et al. · 2022-10-31
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.