II · THE IDEA · ARTIFICIAL INTELLIGENCE
VRAM: The Number That Decides Everything
▶ Listen · narrated
The number on the box — 8GB, 24GB, 80GB — is not marketing. It is the hard limit that decides which models you can run locally, and at what precision.
At a glance
- Base formula
- Parameters × bytes per parameter = memory for weights
- 16-bit precision
- 2 bytes per parameter
- 8-bit precision
- 1 byte per parameter
- Context overhead
- Adds memory proportional to sequence length and model depth
Think of a model's parameters as a large reference book the system must keep open on its desk. A book with seven billion entries, where each entry is a number. If you write each number with high precision — say, sixteen binary digits — the book is thick and heavy: fourteen gigabytes. If you round each number to eight digits, the book is half as thick. Round to four digits and it halves again. The desk is your graphics card's memory, and it has a fixed size. A bigger book than the desk can hold will not fit, no matter how useful it might be. Quantisation is deciding how much precision you can sacrifice to make the book thin enough to fit on the desk you have.
VRAM consumption for a transformer model has two main components: weights and activations. Weights are stored as tensors of floating-point or quantised integer values. At FP16, each parameter is two bytes; at INT8, one byte; at 4-bit quantisation schemes like GPTQ or GGML Q4, half a byte per parameter. The memory for weights is params × bytes_per_param, and it is allocated once when the model loads.
Activations include the KV cache, which stores the key and value tensors produced by each attention layer for every token in the context. For a model with L layers, hidden dimension d, and context length n, the cache is approximately 2 × L × n × d × bytes_per_element. A 7B model typically has 32 layers and a hidden dimension of 4096. At FP16, a 2048-token context adds roughly 2 × 32 × 2048 × 4096 × 2 bytes ≈ 1GB. Double the context and you double this term. Batch size multiplies it again, because each sequence in the batch maintains its own cache.
Quantisation reduces weight memory linearly but does not eliminate the cache overhead unless you quantise activations too, which some engines support with further quality trade-offs. The practical result is that VRAM budgeting requires summing the quantised weight size and the projected cache size for your workload, then ensuring the total stays below the card's capacity with some margin for framework overhead and fragmentation.
Look closer
The weights are the largest fixed cost
A 7-billion-parameter model at 16-bit precision is 7,000,000,000 parameters × 2 bytes = 14,000,000,000 bytes, or roughly 14 gigabytes. That memory is occupied the moment you load the model, before you process any text. A 13-billion-parameter model at the same precision needs 26GB. A 70-billion model needs 140GB. The arithmetic is linear and unforgiving.
Quantisation trades precision for capacity
Storing each parameter as an 8-bit integer instead of a 16-bit float halves the memory requirement: the same 7B model now fits in 7GB. Four-bit quantisation halves it again, to 3.5GB. Quality degrades as precision falls, but the loss is often modest enough that a larger quantised model outperforms a smaller full-precision one. The llama.cpp project documents quantisation schemes in detail, and reports that many tasks tolerate 4-bit weights with little measurable harm.
Context and activations add a variable cost
The KV cache — which stores attention keys and values for every token in the context — grows with sequence length and model depth. For a 7B model with 32 layers processing 2048 tokens, the cache may add another 1–2GB. Larger contexts or deeper models increase this substantially. Batch size also matters: processing multiple sequences in parallel multiplies the activation memory by the batch size, which is why inference servers carefully tune batching against available VRAM.
The story
The question is always the same: will it fit? A model's parameter count and the precision you store them at determine a fixed memory cost, and your card's VRAM is a hard ceiling. Cross it and the model will not load, or will spill to system RAM and slow to a crawl.
The base calculation is straightforward. Each parameter is a number. At 16-bit precision, that number occupies two bytes. Multiply parameter count by two and you have the memory required for the weights. A 7-billion-parameter model is 14 gigabytes. A 13-billion model is 26. A 70-billion model is 140. There is no cleverness that removes this cost; the weights must be in memory for the model to run.
Quantisation offers a trade. Store each parameter as an 8-bit integer and the memory halves. Store it as a 4-bit value and it halves again. The 7B model that needed 14GB at full precision now needs 3.5GB at 4-bit. The loss in representational fidelity is real — fewer bits mean coarser distinctions — but empirical testing suggests that many tasks tolerate 4-bit quantisation with surprisingly little degradation. The llama.cpp documentation, which has become a reference for local inference, reports that 4-bit and 5-bit quantised models often perform nearly as well as their full-precision counterparts on common benchmarks, and that the difference is harder to detect than the raw bit count would suggest.
The weights are the largest cost, but not the only one. The KV cache, which stores the keys and values computed by the attention mechanism for every token in the context window, grows with sequence length and with model depth. For a 7-billion-parameter model with 32 layers processing a 2048-token context, the cache might add 1–2 gigabytes. Double the context length and you double that overhead. Deeper models with more layers pay more per token. Batch size multiplies it again: process four sequences in parallel and the activation memory grows by a factor of four, which is why inference servers tune batch size carefully against available headroom.
The practical consequence is that VRAM becomes a budget you allocate. A 24GB card can hold a 13B model at 16-bit precision with room left for a moderate context, or a 70B model quantised to 4-bit with a tight context, or a 7B model at high precision with a very large context window. Each choice trades one constraint against another. The formula does not care what you want to do; it only enforces what will fit.
Why it mattered then
VRAM capacity has been the binding constraint on local inference since the first experiments with running large models outside datacentres. Early adopters quickly discovered that parameter count translated directly into memory demand, and that consumer cards with 8 or 12 gigabytes could not hold models that research labs ran routinely. The release of LLaMA in early 2023 made the arithmetic urgent for a much wider audience, because the weights were suddenly available but the hardware requirements were not negotiable. Quantisation techniques, particularly the work documented in llama.cpp, emerged as the primary method for fitting capable models onto cards that individuals could afford. The constraint was not artificial; it was and remains a direct consequence of how transformers store and retrieve their learned knowledge.
Why it matters now
The arithmetic still governs everything. A wave of open-weight models has made capable systems available for local inference, but VRAM remains the gate. The newest cards from NVIDIA — the RTX 4090 with 24GB, the professional A6000 with 48GB — set the practical ceiling for most individuals, and the models that fit on them are chosen by the same multiplication. Quantisation has improved, and newer schemes compress more gracefully, but the base formula has not changed. Anyone running models locally budgets VRAM first, then selects model size and precision to fit within it. The number on the card still decides what is possible.
The surprising detail
The KV cache can exceed the size of the model weights if the context is long enough. A 7B model at 4-bit quantisation occupies roughly 3.5GB, but processing a 32,000-token context with a 32-layer model can require 8–10GB for the cache alone. This inversion surprises people who assume the weights will always dominate. It also explains why some inference engines offer options to offload the cache to system RAM or to recompute keys and values on demand rather than storing them — trading speed for capacity when the context grows large enough that memory becomes the bottleneck again.
Remember this
Parameter count times bytes per parameter gives you the fixed cost. Everything else is overhead, and it grows with context length.
Test yourself
You have a 16GB card and want to run a 13-billion-parameter model. At 16-bit precision it needs 26GB and will not fit. You quantise it to 8-bit, bringing it down to 13GB. You load it successfully, but inference fails after a few hundred tokens. What is the likely cause, and what are two ways to fix it?
The KV cache is consuming the remaining 3GB faster than expected, probably because the context length or batch size is larger than the headroom allows. First fix: reduce the maximum context length so the cache stays within budget. Second fix: reduce batch size to one if you are processing multiple sequences in parallel. A third option, if the engine supports it, is to enable cache offloading or recomputation, which trades speed for memory by not storing the entire cache on the GPU.
Go deeper
- llama.cpp/docs/build.md at master · ggml-org/llama.cpp · GitHub · github.com
- GPU · Hugging Face · huggingface.co
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.