Activation Compression via Quantized Training

When people first meet model compression they usually meet weight quantisation: store each parameter in fewer bits, shrink the file, speed up multiply-accumulate units that prefer integer arithmetic. That story is real, and it is incomplete. In training, and often in high-throughput inference, the dominant moving parts are activations — the intermediate tensors produced layer by layer. They must be written out, retained for back-propagation, and read back when gradients are computed. Their volume scales with batch size, sequence or spatial dimensions, and depth. Compressing only the weights leaves most of that traffic untouched.

Activation compression via quantised training attacks that traffic directly. Instead of waiting until a full-precision network has finished training and then attempting to round its activations, the training procedure itself exposes the network to low-bit integer activations. In practice this often means inserting quantisation and dequantisation into the forward pass so that each activation tensor is forced through a small discrete set of levels — typically integer codes with a scale, and sometimes a zero-point — before the next layer sees it. The rest of the stack may still accumulate in wider precision; the point is that what must be stored and moved can be the narrow integer form.

Why train with that noise rather than bolt it on afterwards? Networks trained entirely in wide floating point learn to rely on fine gradations in intermediate values. A sudden post-hoc quantisation of those values can destroy accuracy because the weights never adapted to the clipped, rounded signals. By making the forward path already discrete during training, the optimiser adjusts weights under the same constraint the deployed system will impose. Gradients with respect to the discrete step are commonly approximated (a straight-through style treatment), so parameter updates remain possible even though the forward values are integer-valued after scaling.

The literature on efficient inference for deep convolutional networks treats quantisation as a system choice that must be validated end to end, not only as a storage trick. Parallel work on mixed-precision networks argues that the decisive ingredient is often the parametrisation: how scales are represented, which layers keep higher precision, and how the discrete codes are tied to continuous parameters the optimiser can move. A uniform bit-width policy across every tensor is rarely optimal; some layers tolerate aggressive activation quantisation, others do not. Getting that assignment right is part of the method, not an afterthought.

The payoff is concrete. Lower-bit activations shrink the working set that must stay on-device between forward and backward passes, reduce bandwidth pressure on memory hierarchies, and can unlock larger batches or deeper models on the same hardware. The same machinery that trains under quantised activations also prepares the model for integer-friendly inference, so the compression is not a one-sided training hack. It is a joint decision about how the network represents intermediate state for its entire life cycle.

None of this removes the need for care. Clipping ranges must track activation distributions; poor range estimation wastes the few bits available. Some operators and some network families are more brittle under activation quantisation than others. The whitepaper tradition is explicit that efficient inference is an engineering stack — numerics, scheduling, and model structure — rather than a single switch. Quantised training of activations is one of the more consequential switches in that stack, because it changes the volume of data the hardware must move on every step.