II · THE IDEA · ARTIFICIAL INTELLIGENCE
Prefill Versus Decode
▶ Listen · narrated
Every time you watch a model pause, then suddenly stream words, you are seeing the boundary between prefill and decode — two stages with completely different performance limits.
At a glance
- Prefill phase
- Process entire prompt in parallel, compute all attention once
- Decode phase
- Generate one token at a time, each requiring full attention recalculation
- Prefill bottleneck
- Compute-bound: limited by matrix multiplication throughput
- Decode bottleneck
- Memory-bandwidth-bound: limited by loading model weights repeatedly
Imagine you are reading a long document aloud, then answering a question about it. Reading the document is prefill: you do it once, as fast as you can read, and the time depends on how long the document is. Answering the question is decode: you produce one word at a time, and each word requires you to remember everything you have said so far. The reading is limited by how fast your brain can process text. The answering is limited by how fast you can retrieve the relevant memories — even if your brain is capable of thinking faster, you spend most of the time fetching what you need from memory. In a language model, prefill is limited by how fast the hardware can multiply matrices, and decode is limited by how fast it can load the model's weights from memory. The two stages have opposite bottlenecks, so speeding up one does not necessarily speed up the other.
Prefill performs a single forward pass over the entire input sequence. Attention is computed for all token pairs simultaneously, producing a matrix of size sequence_length × sequence_length. This is dense linear algebra: large matrix multiplications that saturate the GPU's compute units if the batch or sequence is large enough. Arithmetic intensity — FLOPs per byte of memory traffic — is high, so prefill is compute-bound on modern accelerators. Throughput scales with TFLOPS and parallelises across devices cleanly.
Decode runs the model autoregressively. After prefill, each iteration appends one generated token to the context, computes attention over the now-longer sequence, and produces a distribution for the next token. The KV cache stores keys and values from previous tokens to avoid recomputing them, so attention cost per step is linear in context length rather than quadratic. However, each decode step must load the full model weights from HBM into SRAM, perform a small amount of arithmetic relative to those loads, then write results back. Arithmetic intensity collapses. Pope et al. measured that decode becomes memory-bandwidth-bound at batch sizes where prefill still has compute headroom. Weight quantisation — reducing precision from FP16 to INT8 or INT4 — cuts memory traffic and can double decode throughput without increasing FLOPS. Batching helps prefill far more than decode, because decode's memory bottleneck worsens as you add requests competing for bandwidth.
The KV cache itself becomes a memory management problem. Kwon et al. showed that naive allocation fragments GPU memory and limits batch size. PagedAttention, which pages the cache in fixed-size blocks, eliminates fragmentation and allows 2× higher throughput on decode-heavy workloads. The cache grows linearly with context length and batch size, so long-context models can exhaust memory during decode even when prefill fit comfortably.
Look closer
Prefill processes everything at once
When you submit a prompt, the model does not read it token by token. It loads the entire sequence into memory and computes attention across all positions in a single forward pass. This is highly parallel work — thousands of values being multiplied and added simultaneously — so it saturates the GPU's compute units. A thousand-token prompt takes barely longer than a hundred-token one, because the work scales with sequence length but runs in parallel. The output of prefill is a single probability distribution: what token should come next.
Decode runs the model once per output token
After prefill, the model enters a loop. It samples one token from the distribution, appends it to the context, then runs another forward pass to predict the next token. This repeats until it emits a stop token or hits a length limit. Each pass must attend to the entire growing context, but it produces only one new token, so the arithmetic intensity — the ratio of computation to memory access — drops sharply. The GPU spends most of its time waiting for weights to arrive from memory, not multiplying them. Pope et al. measured that decode can be bottlenecked by memory bandwidth even when compute units sit mostly idle.
The ratio determines which optimisations matter
If your workload is short prompts and long completions, you spend nearly all your time in decode, and memory bandwidth is the constraint to relieve. Quantisation, which shrinks weights, helps enormously. If your workload is long prompts with short answers — retrieval-augmented generation, for instance — prefill dominates, and you want faster matrix engines. The same hardware will feel fast or slow depending entirely on this ratio. Kwon et al. noted that efficient memory management during decode, such as reusing cached key-value blocks, can double throughput on memory-bound workloads without changing the model at all.
The story
When you send a prompt to a model, two distinct things happen in sequence, and they have almost nothing in common from a performance perspective.
The first stage is prefill. The model receives your entire prompt — ten tokens or ten thousand — and processes it in one forward pass. Every token attends to every other token simultaneously. This is dense, parallel work. The GPU's tensor cores multiply enormous matrices together, and if the batch is large enough or the sequence long enough, those cores stay busy. Prefill is compute-bound: the bottleneck is how fast the hardware can perform the arithmetic, not how fast it can fetch the weights from memory. On modern accelerators, prefill throughput scales nearly linearly with compute capacity. A prompt that would take ten seconds on one GPU might take five on two, because the work parallelises cleanly.
Prefill ends when the model has computed a probability distribution over the vocabulary for the next token. It samples one token from that distribution — or picks the most probable, depending on the sampling settings — and appends it to the context. Now the second stage begins.
Decode is a loop. The model runs another forward pass with the context now one token longer, attends to everything again, and produces another distribution. Sample, append, attend, predict. Repeat until the model emits a stop token or you hit a length limit. Each iteration produces exactly one token, and each iteration must load the entire model's weights from memory into the compute units, multiply them against the hidden states, then write the results back. The arithmetic itself is trivial compared to the memory traffic. Pope et al. found that decode becomes bottlenecked by memory bandwidth long before the GPU's compute capacity is saturated. The tensor cores sit idle, waiting for data.
This is why the two stages feel so different. Prefill happens in a short burst: you wait, then the first token appears. Decode is a visible drip: one token, slight pause, next token, pause. The pauses are memory latency made visible. If you quantise the model — represent weights in eight bits instead of sixteen — you have just halved the memory traffic per decode step, and throughput can double even though you have not added any compute. Conversely, throwing more compute at decode yields diminishing returns unless you also widen the memory bus.
The ratio between prefill and decode time depends entirely on your workload. A chatbot that receives short questions and writes long answers spends most of its life in decode. A retrieval system that processes thousand-token documents and emits three-word answers is prefill-dominated. The same hardware, the same model, but the performance profile inverts. Kwon et al. demonstrated that memory management during decode — specifically, how you store and reuse the key-value cache that holds attention states for all previous tokens — can matter more than the speed of the arithmetic itself. Efficient paging of that cache, they found, doubled throughput on decode-heavy workloads without changing a single weight.
Why it mattered then
The distinction mattered as soon as Transformers moved from research to production. Early deployments treated inference as a single monolithic operation, optimising for throughput across the entire forward pass. But user-facing applications revealed the problem: people do not experience average throughput, they experience latency, and latency has two components that respond to completely different interventions. A system that felt fast during internal testing — where prompts and completions were similar lengths — felt sluggish in production, where users wrote three sentences and expected paragraphs in return. The decode loop, invisible in the averaged benchmarks, dominated the wall-clock time. Recognition of the two-phase structure led directly to the optimisations that made real-time generation feasible: quantisation to relieve memory bandwidth, KV cache management to avoid redundant computation, and speculative decoding, which tries to predict multiple tokens ahead and verify them in parallel. None of these techniques make sense until you see prefill and decode as separate problems.
Why it matters now
The split still explains almost every latency number you will encounter. If a model feels slow to start but then streams quickly, prefill is the bottleneck — the prompt is long or the compute is inadequate. If the first token appears instantly but subsequent ones trickle, decode is memory-bound. Cloud providers now report prefill and decode throughput separately, because a single number obscures which resource is constrained. The rise of long-context models has made the distinction sharper, not softer: a model with a 128,000-token context window can spend tens of seconds in prefill on a full window, then generate at exactly the same tokens-per-second as a model with an 8,000-token window, because decode cost scales with model size and context length, not with the length of the original prompt once it has been processed. Optimisations continue to target the two phases independently. Prefill benefits from tensor parallelism and faster interconnects between GPUs. Decode benefits from weight compression, flash attention, and anything that reduces the bytes moved per token. The hardware itself is diverging: some accelerators now ship with asymmetric memory configurations, wide and slow for prefill, narrow and fast for decode. Understanding the split is not historical context. It is the prerequisite for reading any inference benchmark, choosing any serving stack, or predicting whether your workload will be cheap or ruinously expensive at scale.
The surprising detail
The same model on the same hardware can show a tenfold difference in throughput depending solely on the prompt-to-completion ratio, even though the total number of tokens processed is identical. A system that generates one hundred tokens after a ten-token prompt runs almost entirely in decode and is memory-bound. A system that generates ten tokens after a hundred-token prompt is prefill-heavy and compute-bound. The work is not the same: attention cost scales quadratically with sequence length during prefill but linearly during decode, because decode reuses cached keys and values rather than recomputing them. This means that batching strategies that work beautifully for one workload can collapse throughput for another, and it means that the same optimisation — say, increasing batch size — can speed up prefill but slow down decode by starving it of memory bandwidth. The two phases are so different that some serving systems now run them on different hardware entirely.
Remember this
Prefill is parallel and compute-bound; decode is sequential and memory-bound. Every inference optimization targets one or the other, rarely both.
Test yourself
You have a fixed hardware budget and two workloads: one generates long summaries from short documents, the other generates short answers from long documents. You can afford either four GPUs with high compute and moderate memory bandwidth, or two GPUs with moderate compute and very high memory bandwidth. Which workload gets which hardware, and why?
The long-summary workload — short prompts, long completions — spends most of its time in decode, so it is memory-bandwidth-bound. Give it the two GPUs with high bandwidth. The short-answer workload — long prompts, short completions — is prefill-dominated and compute-bound, so it benefits from the four GPUs with high compute, which can parallelise the large attention operations across the long input sequence. The mismatch is a common production mistake: people assume more GPUs always means faster inference, but if your workload is decode-heavy, adding compute without adding memory bandwidth can actually reduce throughput per dollar, because the extra GPUs sit idle waiting for memory.
Go deeper
- Efficient Memory Management for Large Language Model Serving with PagedAttention · arXiv · Woosuk Kwon et al. · 2023-09-12
- Efficiently Scaling Transformer Inference · arXiv · Reiner Pope et al. · 2022-11-09
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.