II · THE IDEA · ARTIFICIAL INTELLIGENCE
The KV Cache
▶ Listen · narrated
A thousand-token prompt and a ten-token reply do not cost the same as a ten-token prompt and a thousand-token reply, even though both conversations contain 1,010 tokens. The asymmetry lives in the cache.
At a glance
- What it stores
- Key and value vectors for every token processed so far, at every layer
- Why it exists
- Avoids recomputing attention over all previous tokens when generating each new one
- Memory cost
- Grows linearly with sequence length and number of layers
- Typical bottleneck
- VRAM fills with KV cache long before model weights become the constraint
Imagine you are reading a long document aloud, and every time you read a new sentence, you must glance back at every previous sentence to check whether it is still relevant. That would be exhausting and slow. The KV cache is a set of notes you make while reading the first time through, so that when you start speaking, you can glance at your notes instead of rereading everything. The notes grow as you read more, and they take up space on your desk, but they make speaking much faster. In a language model, those notes are key and value vectors — intermediate calculations from the attention mechanism — and they are stored in memory so the model does not have to recompute them for every new word it generates. The trade-off is explicit: memory for speed. A long conversation fills memory with notes, and eventually you run out of desk space. That is why very long conversations slow down or require the model to forget early parts.
The KV cache stores the key and value projections computed during attention, separately for each layer and each head. During prefill, the model processes the input sequence and computes Q, K and V matrices via learned linear projections. The keys and values are stored; the queries are used immediately and discarded. During autoregressive decode, each new token generates a new query vector, which attends to all cached keys via scaled dot-product attention, then uses the attention weights to aggregate cached values. The new key and value are appended to the cache, and generation continues. Without the cache, each decode step would require recomputing K and V for the entire sequence up to that point, making generation O(n²) in sequence length. With the cache, each step is O(n) in memory reads but O(1) in computation per token, because only the new token's projections are computed. The memory cost is 2 × layers × heads × sequence_length × head_dim × bytes_per_element. In a 7B parameter model with 32 layers, 32 heads, 128-dimensional heads and float16 precision, a 2,048-token cache occupies roughly 4 GB. Grouped-query attention reduces this by sharing K and V across multiple Q heads, cutting memory proportionally. Multi-query attention uses a single K and V per layer, reducing cache size by a factor equal to the number of heads, at a small cost in model quality.
Look closer
The cache grows in two dimensions
Each new token adds one key vector and one value vector per attention head per layer. In a model with forty layers and thirty-two heads per layer, a single token contributes 2,560 vectors to the cache. Those vectors are not small — in a model with 4,096-dimensional hidden states, each vector is 4,096 floating-point numbers. The cache for a 2,000-token conversation in such a model occupies multiple gigabytes, and that figure is per conversation. Serving ten users concurrently means ten separate caches.
Prefill is different from decode
When the model first sees your prompt, it processes all tokens in parallel, computing attention in one pass. This is called the prefill phase, and it populates the cache. After that, generation enters the decode phase: one token at a time, autoregressively, with each new token attending to everything already in the cache. Prefill is bound by compute — how fast can the accelerator perform matrix operations on the entire prompt. Decode is bound by memory bandwidth — how fast can key and value vectors be read from VRAM for every layer, every head, every token generated. The two phases stress different parts of the hardware.
Eviction is a real problem
Once the cache fills the available memory, something must give. Early systems simply failed or truncated. More recent work treats the cache as a paging problem: Kwon and colleagues introduced PagedAttention, which splits the cache into fixed-size blocks and manages them the way an operating system manages virtual memory. Blocks can be swapped, shared across requests that have common prefixes, or evicted according to a policy. This raises the utilization of expensive hardware substantially, but it also means that what gets remembered and what gets forgotten is now an engineering decision with quality implications, not just a hard limit.
The story
Transformers generate text one token at a time, and each token depends on every token before it. When the model produces the fifth word in a sentence, it must attend to the first four. When it produces the hundredth, it must attend to all ninety-nine predecessors. Attention is not a cheap operation — it involves scoring the new token against every prior token, across every head, in every layer. If the model recomputed those scores from scratch for each new token, generation would be quadratic in the length of the sequence, and a 2,000-token conversation would involve nearly two million redundant attention calculations per layer.
The KV cache eliminates that redundancy. During the prefill phase, when the model first processes your prompt, it computes a key vector and a value vector for every token at every layer. Those vectors are the intermediate results of the attention mechanism, and they do not change once computed. When the model generates the next token, it does not need to recompute keys and values for the prompt or for any previously generated tokens. It computes only the key and value for the new token, then retrieves all the cached vectors and performs attention. The new key and value are added to the cache, and the process repeats.
This is why the cost structure is asymmetric. A long prompt is expensive during prefill — the model must process every token, compute attention across the growing sequence, and populate the cache. But once prefill is complete, each generated token has roughly the same cost regardless of prompt length, because it reads from the cache rather than recomputing. A short prompt is cheap to prefill but generates tokens at the same rate. The cache converts a quadratic problem into a linear one, at the cost of memory that grows with every token retained.
The memory cost is not trivial. Ainslie and colleagues note that in standard multi-head attention, every head maintains its own key and value cache. A model with thirty-two heads and forty layers stores sixty-four vectors per token per layer, and those vectors are large. Grouped-query attention reduces this by sharing key and value vectors across multiple query heads, cutting cache size substantially without major quality loss. Multi-query attention takes this further, using a single key and value per layer rather than per head. The trade-off is memory for flexibility, and in serving contexts where batch size and sequence length matter more than marginal quality, the trade is often worth making.
Why it mattered then
The KV cache was implicit in the original Transformer architecture — attention requires keys and values, and caching them during autoregressive generation is the obvious optimization. But it became a visible engineering concern only as models grew larger and serving them at scale became expensive. Kwon's work on PagedAttention, published in 2023, treated the cache as a first-class memory management problem rather than an implementation detail. That shift reflected the practical reality of serving: in production systems, the cache often occupies more VRAM than the model weights, and managing it efficiently is the difference between serving eight concurrent users and eighty.
Why it matters now
The cache is now a primary constraint in inference system design. Techniques like grouped-query attention and multi-query attention are chosen during training specifically to reduce cache size at serving time. Quantization schemes increasingly target the cache as well as the weights. PagedAttention and similar systems have enabled much higher hardware utilization by treating cache memory as a shared, managed resource rather than a per-request allocation. As context windows grow — some models now support hundreds of thousands of tokens — the cache becomes the dominant memory consumer, and optimizing it is not optional.
The surprising detail
The cache enables an unexpected form of efficiency: prefix sharing. If ten users all start their prompts with the same system message or document, a smart serving system can compute and cache that shared prefix once, then fork separate caches only where the prompts diverge. This is not hypothetical — production systems like vLLM implement it. The cache structure makes this possible because keys and values depend only on the tokens processed so far, not on what comes after. A shared prefix produces identical cache entries regardless of what follows, so those entries can be reused across requests. The memory savings compound with the number of concurrent users who share structure.
Remember this
The cache is why generating a thousand tokens after a short prompt is fast, but prefilling a thousand-token prompt is slow. It trades memory for speed, and in serving systems it often occupies more VRAM than the model itself.
Test yourself
You are serving a model with 40 layers and 32 heads per layer. A user submits a 500-token prompt. Roughly how much VRAM does the KV cache for that single request consume, and what happens to that memory after the response is complete?
Each token stores 2 vectors (key and value) per head per layer, so 2 × 32 × 40 = 2,560 vectors per token. At 500 tokens, that is 1.28 million vectors. If the model uses 4,096-dimensional hidden states and 16-bit floats, each vector is 8 KB, so the cache occupies roughly 10 GB. After the response completes, that memory can be freed immediately if the conversation is not continued — but in a stateful system where the user might send a follow-up, the cache is often retained for some time, because discarding it means re-prefilling the entire history if the user returns. This is why serving systems implement eviction policies: memory is too valuable to hold indefinitely, but recomputation is too expensive to do casually.
Go deeper
- Efficient Memory Management for Large Language Model Serving with PagedAttention · arXiv · Woosuk Kwon et al. · 2023-09-12
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints · arXiv · Joshua Ainslie et al. · 2023-05-22
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.