II · THE IDEA · ARTIFICIAL INTELLIGENCE
GPTQ, AWQ and K-Quants
▶ Listen · narrated
Quantisation lets you run large models locally, but GPTQ, AWQ and the K-quant family make different trade-offs about where to spend their limited precision budget and what calibration data to trust.
At a glance
- What they compress
- Model weights, from 16-bit floating-point down to 4-bit integers
- Memory saving
- Roughly 75% reduction in model size on disk and in RAM
- GPTQ approach
- Layer-by-layer reconstruction with Hessian weighting of errors
- AWQ approach
- Protect weights with high activation magnitudes, quantise the rest more aggressively
- K-quant approach
- Mixed precision within each layer, keeping some weights at higher bit-widths
Think of a model's weights as a long list of decimal numbers, each one a knob that adjusts how the model behaves. At full precision, each number is stored with enough detail to capture tiny differences — sixteen bits, or about five decimal places. That takes a lot of memory. Quantisation rounds those numbers to a coarser grid, using only four bits — sixteen possible values instead of 65,536. The model shrinks by three-quarters, but you lose detail. The clever part is choosing which numbers to round gently and which to round hard. GPTQ looks at how much each weight affects the model's output and protects the sensitive ones. AWQ looks at which weights get multiplied by large numbers during use and protects those. K-quants keep some weights at higher precision and round the rest more. All three methods try to throw away information that matters least, but they disagree about what "matters least" means.
GPTQ applies layer-wise quantisation with optimal brain quantisation (OBQ) informed by the Hessian of the reconstruction loss. For each layer, it quantises weights sequentially, updating the remaining weights to compensate for the error introduced. The Hessian diagonal approximates the curvature of the loss with respect to each weight, so weights with high curvature — those where small changes cause large output shifts — are quantised more carefully. The method uses a calibration set to compute the Hessian and measure reconstruction error, and it processes one layer at a time to keep memory usage tractable. Group-wise scaling divides each weight matrix into blocks of 128 or 256 weights, each with its own scale factor, which reduces quantisation error for weights with non-uniform distributions.
AWQ identifies salient weights by measuring activation magnitudes on a calibration set. Weights that are consistently multiplied by large activations contribute disproportionately to the output, so AWQ protects them by scaling them up before quantisation or keeping them at higher precision. The remaining weights are quantised to 4 bits with per-group scaling. The method is faster than GPTQ because it does not require Hessian computation or iterative compensation, and it often achieves better quality on tasks where salient features are stable across inputs. The calibration set should be representative of inference-time activations, or the protected weights may not align with actual importance.
K-quants use mixed precision within each layer, assigning different bit-widths to different weight groups based on empirical sensitivity. Q4_K_M, for example, keeps roughly half the weights at 6 bits and the rest at 4, with the higher-precision budget spent on weights that empirical testing shows matter most — often attention projections and the first and last layers. Q5_K_S uses 5-bit quantisation more uniformly but reduces precision in less critical layers. The schemes evolved through community testing in llama.cpp, guided by perplexity on standard benchmarks, and they are faster to apply than GPTQ or AWQ because they do not require calibration-driven optimisation. The trade-off is that they are less adaptive to specific tasks or datasets.
Look closer
Calibration data shapes the final model
Both GPTQ and AWQ require a small calibration dataset — typically a few hundred sequences — to measure either reconstruction error or activation magnitude. The choice of data matters. If you calibrate on code and then use the model for creative writing, the protected weights may not be the ones that matter for your task. The calibration step is fast, but it bakes in assumptions about what the model will be asked to do.
Per-group scales let different parts of a layer use different ranges
A naive quantiser maps the full range of weights in a layer to 0–15. GPTQ and AWQ divide each weight matrix into groups — commonly 128 weights per group — and give each group its own scale factor. A group containing weights clustered near zero uses a narrow scale; a group with outliers uses a wider one. The group size is a tunable parameter, and smaller groups preserve more detail at the cost of slightly more metadata.
K-quants are a family, not a single method
The llama.cpp project introduced a naming scheme — Q4_K_M, Q5_K_S — where the number is the average bit-width and the suffix indicates a strategy. K_M keeps some weights at 6 bits, K_S saves memory by using fewer high-precision weights, and K_L spends more bits on attention layers. There is no formal paper; the methods evolved through community experimentation, guided by perplexity benchmarks rather than a unified theory.
The story
When a model is trained, its weights are stored as 16-bit or 32-bit floating-point numbers. A 70-billion-parameter model at 16 bits occupies roughly 140 gigabytes. Most consumer hardware cannot hold that in RAM, and even professional cards struggle. Quantisation compresses the weights by representing them with fewer bits — often four — which brings the same model down to around 40 gigabytes. That difference is the boundary between a model you can run and one you cannot.
The challenge is that not all weights matter equally. Some are large and influential, others are small and contribute little to any given output. A uniform quantisation scheme treats them all the same and loses information where it hurts. GPTQ, AWQ and the K-quant methods all try to be selective, but they choose different strategies.
GPTQ, published in 2022 by Frantar and others, works layer by layer. It quantises the weights in one layer, measures how much that changes the layer's output on a small calibration dataset, then adjusts the remaining weights to compensate. The adjustment is guided by the Hessian — a matrix of second derivatives that tells you which weights, if perturbed, will cause the largest errors downstream. Weights with high curvature are treated carefully; weights in flat regions can be rounded more aggressively. The process is expensive — quantising a large model can take hours — but it happens once, and the result is a compressed model that tries to behave as the original did on the calibration data.
AWQ, published in 2023 by Lin and others, takes a different view. It observes that a small fraction of weights — around one per cent — are consistently multiplied by large activation values during inference. Those products dominate the output, so AWQ protects those weights by keeping them at higher precision or scaling them carefully before quantisation. The rest are quantised more aggressively. The method is faster than GPTQ and often preserves quality better, especially on tasks where certain features fire reliably. The trade-off is that it depends on the calibration data capturing the activations that will matter in production. If your use case is far from the calibration distribution, the protected weights may not be the right ones.
The K-quant methods, developed within the llama.cpp project, are more pragmatic. They use mixed precision within each layer: some weights stay at 5 or 6 bits, others drop to 4 or even 2. The exact mix is determined by heuristics and community testing rather than a formal optimisation. Q4_K_M, for instance, keeps roughly half the weights at 6 bits and the rest at 4, targeting a balance between size and quality. Q5_K_S uses 5-bit quantisation more widely but spends fewer bits on less critical layers. The naming convention is dense — the suffixes S, M and L stand for small, medium and large, referring to how much precision is preserved — but the underlying idea is consistent: spend your bit budget where perplexity measurements say it matters.
All three approaches require you to choose a calibration dataset. GPTQ typically uses a few hundred samples from a general corpus; AWQ benefits from data that matches your intended use. The K-quant methods are less prescriptive, and many quantised models in circulation were calibrated on datasets the end user never sees documented. That opacity is a practical problem. A model quantised on English Wikipedia may behave differently on legal text or on a language that was rare in the calibration set, and you will not know until you test it.
Why it mattered then
GPTQ and AWQ were published at a moment when open-weight models had crossed into the tens of billions of parameters — large enough to be useful, too large to run on consumer hardware without compression. The papers demonstrated that careful quantisation could preserve most of the quality while cutting memory requirements by three-quarters. That made local inference viable for individuals and small organisations, shifting the economics of deployment. Before these methods, running a 70-billion-parameter model meant renting cloud GPUs or buying enterprise hardware. After them, it meant a gaming PC with 64 gigabytes of RAM. The K-quant methods emerged slightly later, as the llama.cpp community iterated on mixed-precision schemes that were faster to apply and easier to distribute than full GPTQ or AWQ quantisation.
Why it matters now
Quantised models are now the default for local inference. Most models shared on repositories like Hugging Face are available in multiple quantised formats, and users choose between them based on the hardware they have and the quality they need. GPTQ remains common for models where reconstruction quality is critical; AWQ is preferred when activation-aware protection matters; K-quants dominate in llama.cpp and its derivatives because they are fast, flexible and well-tested by a large community. The methods also matter for edge deployment — running models on phones, embedded systems or in environments where bandwidth and power are constrained. A 4-bit model is not just smaller in RAM; it is faster to download, faster to load and cheaper to serve. The choice of quantisation method is now a routine part of model distribution, and understanding the trade-offs helps you pick the right one for your hardware and task.
The surprising detail
The K-quant methods have no formal paper and no single inventor. They emerged from incremental changes in the llama.cpp codebase, guided by perplexity benchmarks and user reports rather than theoretical analysis. The naming scheme — Q4_K_M, Q5_K_S — is documentation after the fact, describing strategies that were already in use. This is unusual in machine learning, where most techniques arrive with a preprint and a set of experiments. The K-quants evolved in public, in a repository where the goal was to make models run fast on consumer hardware, and the validation was whether people kept using them. That pragmatic, community-driven process produced methods that are now as widely used as the academic ones, and in some contexts more so.
Remember this
Not all 4-bit quantisation is equivalent. The method determines which weights are protected, which are degraded, and what calibration data shaped those decisions.
Test yourself
You have quantised the same model with GPTQ and AWQ, both targeting 4 bits per weight, and both using the same calibration dataset. On one task the GPTQ version performs better; on another the AWQ version does. Explain one specific structural reason this could happen, beyond randomness or measurement error.
GPTQ minimises reconstruction error across all weights, guided by how sensitive the layer's output is to each weight's perturbation. It protects weights with high curvature in the loss landscape. AWQ protects weights that are multiplied by large activations during calibration, regardless of curvature. If the first task depends on preserving subtle relationships across many weights — something that reconstruction error would capture — GPTQ may do better. If the second task depends on a few high-magnitude features that fire reliably — something activation-aware protection would preserve — AWQ may win. The methods optimise for different proxies of importance, and tasks differ in which proxy predicts their quality.
Go deeper
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers · arXiv · Elias Frantar et al. · 2022-10-31
- AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration · arXiv · Ji Lin et al. · 2023-06-01
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.