II · THE IDEA · ARTIFICIAL INTELLIGENCE
Extreme Low-Bit Post-Training Quantization
▶ Listen · narrated
A finished model need not stay at full precision forever. Extreme low-bit post-training quantisation aims to keep most of its behaviour while storing each weight in only two to four bits.
At a glance
- What it is
- Reducing trained weights to 2–4 bits without retraining the model
- When applied
- After training has finished, on an already-learned set of weights
- Main levers
- Advanced rounding choices and calibration on representative data
- Stated aim
- Strong compression with only minimal loss of accuracy
- Why hard
- At two to four bits, naive rounding destroys useful signal quickly
Think of each weight as a precise measurement written with many decimal places. Extreme low-bit post-training quantisation is the act of rewriting every measurement with only a handful of allowed values—four values if you have two bits, sixteen if you have four—after the model has already been trained.
If you always snap each number to the nearest allowed value, the model often gets worse quickly, because small snaps stack up through the layers. Better methods choose the snaps more carefully and use a short rehearsal on sample inputs (calibration) to see which choices keep the model’s answers closest to the original. You do not train the whole model again; you only decide how to round and how to scale within that tiny set of levels.
So the headline is not merely “use fewer bits.” It is “use fewer bits, and spend the effort on rounding and calibration so the compressed model still behaves.”
Extreme low-bit post-training quantisation (PTQ) maps trained full-precision weights onto a grid with 2–4 bits per weight after optimisation has finished. The forward approximation is typically of the form ŵ = s · clip(round(w/s), range), or a close variant, where scale s, clip range and the rounding operator are chosen without gradient-based retraining of w itself.
At 8 bits, uniform range fitting and round-to-nearest are often tolerable. At 4, 3 and especially 2 bits, the same pipeline frequently collapses accuracy because per-weight rounding error correlates across layers and interacts with activation distributions. Advanced PTQ therefore separates three concerns: (1) range and scale estimation (per-tensor or per-channel), (2) the rounding decision (not necessarily nearest, and sometimes informed by local error or reconstruction objectives on calibration activations), and (3) a calibration dataset used to measure reconstruction or task error while those choices are set.
Because the stage is post-training, compute cost is dominated by a modest number of forward passes plus the search or heuristic that assigns discrete levels—not by full backprop through the original loss. The matching limitation is statistical: quantisation parameters are only as good as the calibration distribution. A method can report strong fidelity on a calibration batch and still degrade under domain shift. Bit width fixes the cardinality of the codebook; rounding and calibration decide which code is used and whether the network’s functional behaviour remains close to the full-precision reference.
Look closer
The bit width is not the whole story
Two, three and four bits per weight define how many distinct levels a weight may take. That grid is coarse. What matters as much as the grid itself is which full-precision value is mapped to which level. Simple nearest-level rounding ignores how a small error in one weight can cascade through later layers. Advanced rounding treats that mapping as a decision that can be improved, not as a fixed rule applied once in isolation.
Calibration stands in for retraining
Because the model is not trained again, the method needs another way to see what the low-bit weights will do. Calibration passes a modest set of representative inputs through the network and watches activations and errors. Those observations guide scale factors, clipping ranges and rounding choices. The procedure is cheaper than training, but it inherits whatever biases sit in the calibration set: a narrow set can leave the quantised model brittle on inputs it never saw during that pass.
Post-training means the weights were never low-bit
The original training optimised full-precision parameters. Extreme low-bit post-training quantisation therefore starts from a solution that was not designed for a two-to-four-bit grid. That is both the appeal and the difficulty: no long retraining run is required, yet every compression choice must protect behaviour the optimiser never had to defend under such severe discretisation.
The story
Neural networks store their learned knowledge largely as weight values. In ordinary training those values are kept at high numerical precision so that tiny gradient updates can accumulate smoothly. Once training is over, that precision is often more than the deployed model needs. Quantisation replaces each weight with a value drawn from a small set of allowed levels, and records which level was chosen. Extreme low-bit post-training quantisation pushes that idea hard: each weight is represented with only two, three or four bits, and the conversion is done after training rather than by training a low-bit model from scratch.
At such widths the set of representable levels is tiny. Four bits give sixteen levels; two bits give four. Mapping a continuous weight onto so few choices is lossy by definition. If every weight is simply rounded to the nearest level, errors stack across layers and accuracy can collapse. The promise of this family of methods is that the collapse is not inevitable. Better rounding—rounding that takes account of how a weight is used, or of the error it introduces relative to neighbouring decisions—can preserve far more behaviour than nearest-level rounding alone. Calibration supplies the observations those decisions need: a batch of representative data is run through the still-precise model, scales and ranges are fitted, and rounding is adjusted so that the network’s outputs stay close to the full-precision reference on that data.
The procedure is deliberately cheaper than training. There is no long optimisation over the original loss, and the bulk of the learned structure is treated as fixed. What changes is the discrete code assigned to each weight, plus a handful of quantisation parameters such as step sizes and clip bounds. Because those choices are made with the finished model in hand, they can be layer-aware and data-aware in ways that a uniform cast to low precision cannot. The cost of that thrift is sensitivity: if the calibration inputs do not resemble deployment traffic, the chosen scales and rounding may look excellent on the calibration batch and still hurt accuracy elsewhere.
Extreme low bit-widths magnify every design choice. At eight bits, many networks tolerate fairly blunt quantisation; at four, three or two bits, the same bluntness is often fatal. That is why the literature on neural-network quantisation treats rounding and calibration as first-class topics rather than afterthoughts. The bit budget sets the size of the alphabet; rounding and calibration decide which message, among the ones that alphabet can spell, the compressed model will actually carry.
Nothing in the approach requires the original training recipe to change. That is the practical point. A model that was expensive to train can be compressed afterwards for memory, bandwidth or specialised hardware, provided the post-training stage is careful enough that the behaviour worth keeping survives the grid.
Why it mattered then
As networks grew, the gap between training precision and deployment need became hard to ignore. Full-precision weights are convenient for optimisation, yet they are bulky to store and to move. Post-training quantisation offered a route to smaller footprints without repeating the full training run. Pushing that route down to two–four bits was an attempt to keep the convenience of a finished checkpoint while approaching compression ratios that matter for tight memory and edge hardware. Rounding and calibration became the levers because, once training is off the table, they are almost the only levers left.
Why it matters now
Open-weight models and local deployment make storage and memory bandwidth everyday constraints, not only data-centre concerns. A method that starts from an already-trained checkpoint and aims for extreme compression without a full retrain still matches how many people actually receive and run models. The same caveats remain: calibration data must stand in for real use, and at two–four bits the difference between careless and careful rounding is the difference between a usable model and a broken one. The topic stays current because the pressure to shrink weights has not eased, and because post-training workflows fit the way finished models circulate.
The surprising detail
The severe limit is not only how few bits each weight receives, but that those bits are chosen without revisiting the original training loss. The model is asked to survive a brutal discretisation using only a calibration pass and smarter rounding—almost as if the network must be re-spelled in a tiny alphabet while nobody is allowed to rewrite the essay. That constraint is what makes advanced rounding feel disproportionate to its description: it is doing work that training would otherwise have done, with far less freedom.
What is disputed
Public discussion of extreme low-bit post-training quantisation spans many specific rounding schemes and calibration recipes. Their relative strength depends on architecture, bit width and calibration data, and is not settled by a single rule. Treat rounding and calibration as the decisive levers, and treat any particular recipe’s headline accuracy as conditional on the setting in which it was measured.
Remember this
At two to four bits, compression quality hinges on rounding and calibration, not on bit width alone.
Test yourself
A team quantises a finished model to three bits per weight with careful rounding, but builds the calibration set from a single narrow domain. On that domain the quantised model matches the original closely; on a broader mix of real traffic it does not. What went wrong, and why is retraining not required to explain the failure?
The calibration pass is the only place the method observes how quantisation error affects outputs. Scales, clip ranges and rounding decisions were fitted to a narrow slice of inputs, so they protected behaviour on that slice and under-protected it elsewhere. The failure is in the stand-in for training signal, not in the absence of a full retrain: post-training quantisation never claimed to re-optimise the model for every domain—it only claims to preserve behaviour visible during calibration and rounding.
Go deeper
- [2106.08295] A White Paper on Neural Network Quantization · arxiv.org
- [2210.05459] Unsupervised classification of the spectrogram zeros · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.