II · THE IDEA · ARTIFICIAL INTELLIGENCE
KV Cache Compression for Long Context
▶ Listen · narrated
A long input rarely defeats a model because its weights are too big. It defeats it because the model stores a note about every token it reads, and those notes fill memory first.
At a glance
- What it is
- Making the stored keys and values that attention reuses smaller
- Why it grows
- Each new token adds a key and a value in every layer
- How it scales
- Cache memory rises with sequence length; the weights do not
- One named method
- Sliding-window attention with a rolling buffer cache (Mistral 7B)
- The levers
- Share heads, bound the window, use fewer bits, or drop tokens
In a long meeting, a good note-taker does not replay the recording every time a new point comes up. They keep a running sheet of notes and glance back at it. A language model does something similar. Once it has processed a token, it files away 2 things about it — a key, which says what that token holds, and a value, which is the information itself — so it never has to redo that work. Those filed notes are the key-value cache.
The trouble is that the sheet never stops filling. Every new token adds another line, in every layer of the model. The model's own weights take a fixed amount of space, set when you load them. The cache keeps growing with the text. So on a very long input, it is the cache, not the model, that runs out of room first.
There are 4 ways to shrink the sheet. Let several readers share 1 set of notes instead of each keeping their own. Agree that nobody will ever look back further than a fixed distance, so old lines can be wiped as new ones are written. Write the numbers with fewer digits. Or decide some lines never mattered and bin them. Each works, and each costs something. Sharing must be arranged before the model is trained. Fewer digits blur the detail. And wiping old lines means the distant past can only reach the present second-hand, passed along by everything in between.
The key-value cache stores, for each processed position, the projected key and value tensors at every layer, so that autoregressive decoding attends against cached history instead of re-projecting the full prefix at each step. Its footprint is a product: sequence length × layer count × key-value heads × head dimension × 2 (keys and values) × bytes per element. Weights are constant; the cache is linear in sequence length and multiplies across concurrent requests, which is why long-context serving is usually cache-bound rather than compute- or weight-bound.
4 levers follow from that product. First, reduce key-value heads. Grouped-query attention maps several query heads onto a shared key-value head, dividing cached entries by the group size. The Mistral 7B report (arXiv 2310.06825) lists it among its mechanisms for faster inference. It is fixed at training time; it cannot be retrofitted to weights trained with per-head keys and values.
Second, bound the attention span. Sliding-window attention, also in that report, restricts each position to a fixed number of preceding positions. No live query can reach beyond the window, so older entries are unreachable and need not be retained. The cache becomes a fixed-size rolling buffer that overwrites its oldest slot, and peak cache size becomes independent of total sequence length. Receptive field beyond the window survives only indirectly, because each layer's window composes over the one below; direct attention to distant tokens is lost.
Third, reduce bytes per element by storing cache entries at lower precision. This is the only lever generally available at runtime with no architectural change, and it scales the footprint linearly in bit width. Its accuracy cost is empirical, varies by model and task, and should be measured rather than assumed.
Fourth, evict: retain a subset of positions and discard the rest. A bounded window is the degenerate, purely positional case. More selective policies exist, but the sources cited here do not establish which positions are safely droppable; treat that as open, not settled practice.
The practical upshot: usable context length is a memory budget, not a model constant. Identical weights served under different cache precisions, window settings or concurrency levels expose different effective limits. 2 of the 4 levers — head grouping and window bounding — must be chosen before training; precision and eviction are deployment-time decisions.
Look closer
The weights hold still; the cache does not
Load a model and watch the memory. The weights claim their space once and then sit there, the same size whether you feed the model a sentence or a novel. The key-value cache behaves differently. It is the store of intermediate values — one key and one value per token, per layer — that attention keeps so it need not recompute the whole history at every step. It starts near nothing and climbs steadily as tokens arrive. On a long input the cache ends up competing with the weights for the same limited memory, and it is usually the cache that runs out of room first.
A window with a rolling buffer
The Mistral 7B report describes sliding-window attention: each token attends only to a fixed number of previous positions rather than to all of them. The consequence for memory is direct. If a token can never look further back than the window, keys and values older than the window need not be kept, so the cache can be a rolling buffer — a fixed-size array that overwrites its oldest entry as each new one arrives. Cache size stops growing once the sequence passes the window length. Information from earlier tokens can still travel forward indirectly, because each layer's window sits on top of the one below it, but the direct link is gone.
Grouped key-value heads
Attention runs in parallel heads. If every head keeps its own keys and values, the cache multiplies by the head count. The alternative is to let several query heads share one set of keys and values, so the number of stored entries falls by whatever the sharing ratio is. The Mistral 7B report lists grouped-query attention among its choices for faster inference. What matters practically is that this saving is fixed at training time. It is not a switch you can flip on a model that was trained without it.
The story
A language model produces text 1 token at a time, and it chooses each token by looking back at everything it has already read. The looking is done by a mechanism called attention. At every step, the current position issues a query — a short description of what it wants to know right now. Every earlier position holds a key — a description of what that position contains — and a value — the information actually stored there. Attention compares the query against every key. Wherever the match is strong, that position's value is mixed into the result.
The comparison needs a key and a value for every earlier token, in every layer of the model. Computing them fresh at each step would mean redoing the same arithmetic thousands of times over, so implementations compute them once and keep them. When a token is processed, its key and value are written into memory at every layer, and they sit there for the rest of the generation. That store is the key-value cache.
The cache, not the model, is what limits long inputs, because the cache grows and the weights do not. The weights — the numbers learned in training — take a fixed amount of memory, claimed once when the model loads. The cache is a running cost. 10,000 tokens in, each layer holds 10,000 keys and 10,000 values. 100,000 tokens in, it holds 10 times as many. Nothing about the model has changed; its notes on the conversation have simply outgrown the room set aside for them. Serving many users at once makes this worse, because the weights can be shared between requests but each request needs its own cache.
The cache's size is a plain multiplication: the number of tokens, times the number of layers, times the number of key-value heads, times the size of each entry, times the bits used per number. Shrink any factor and the whole product shrinks. That observation gives 4 levers.
The first lever is sharing heads. Attention runs as several heads in parallel — independent copies of the query-key-value comparison, each free to look for something different. If every head stores its own keys and values, the cache multiplies by the head count. Grouped-query attention cuts this down: a group of query heads reads from 1 shared set of keys and values, so the number of stored entries falls by the size of the group. The Mistral 7B report names this among its choices for faster inference. The saving is decided before training and is baked into the released weights; a model trained with separate heads cannot have sharing switched on afterwards.
The second lever is bounding how far back attention may look. Under sliding-window attention, also described in the Mistral 7B report, each position may see only a fixed number of the positions just before it. The memory saving follows as cause and effect. If no position will ever look at a token again, there is no reason to keep its key and value. So the cache can be a rolling buffer: a fixed-size block of memory in which each new entry overwrites the oldest one. Once the sequence is longer than the window, the cache stops growing at all. The cost is direct access to the distant past. A token outside the window can no longer be looked at; its influence can still reach the present, but only relayed forward, because each layer's window rests on the output of the layer below, and that output already carries traces of earlier tokens.
The third lever is precision. The cache holds ordinary numbers, and a number can be written with fewer bits if you accept a coarser version of it. Halve the bits per number and you halve the cache. This is the only lever that needs no retraining and no change to the model's design, which is why inference engines offer it first. The price is fidelity: every stored key and value becomes a rougher copy of itself, and how much that shifts the model's output depends on the model and the task. There is no general rule; it has to be measured.
The fourth lever is eviction: keeping the notes on some tokens and discarding the rest, on the theory that not every position in a long document is worth remembering. Sliding-window attention is the simplest case, because its rule is purely positional — oldest out first. More selective schemes exist, but which tokens can be dropped without harm is a question the sources here do not settle. It should be treated as open.
None of the 4 levers makes the model more capable. Each makes its memory of the conversation cheaper, and each charges in a different currency: a commitment fixed before training, direct sight of the distant past, numerical fidelity, or retained detail. 2 of the levers — shared heads and the bounded window — belong to whoever designs the model. The other 2 belong to whoever runs it. Choosing among them is choosing which loss you can best afford.
Why it mattered then
When long-context models became something people ran rather than read about, the arithmetic of inference changed. A model with 7 billion parameters is a fixed, known quantity: you can check whether it fits in your memory before you start. A conversation of unknown length is not, because its cache keeps growing while it runs. Serving many users at once sharpens the problem. The weights can be shared across every request; each request's cache cannot. That is the setting for the Mistral 7B report. Grouped-query attention and sliding-window attention appear there together, and both are aimed squarely at inference: faster generation, and longer sequences without the memory bill rising in step with the token count. The striking thing is where these choices sit. They are not settings applied to a finished model at run time. They are design decisions, taken before training, on the assumption that the model would spend its life answering queries rather than sitting in a benchmark. Cheap serving had become part of how a model is designed, not a problem left for whoever deploys it.
Why it matters now
Context windows are advertised in tokens, and the figure sounds like a property of the model. In practice it is a property of the memory available. A model that could, in principle, attend across a very long input will still refuse the job if there is no room to store the keys and values attention needs. That is why 2 deployments of identical weights can offer different usable lengths, and why the length you get may shrink when a service is busy: each concurrent request holds its own cache, and they all draw on the same memory. It also explains behaviour that otherwise looks arbitrary. If the cache is stored at reduced precision, output can drift from what full precision would have produced — no error, no warning, just coarser numbers doing the same job. If attention is windowed, a fact stated at the start of a long document is no longer being looked at directly by the end; it survives only if the layers in between carried it forward. Neither is a bug. Each follows from a memory decision made somewhere in the stack — sometimes at training time, sometimes by a flag in an inference engine. For anyone running open weights on their own hardware, the practical point is that cache size is a budget you can inspect and control: through window length where the architecture allows it, and through precision almost always.
The surprising detail
The cheapest saving is the one nobody computes. Under sliding-window attention, the keys and values of tokens that have fallen out of every window are not compressed, summarised or approximated — they are never kept at all, because the rule guarantees no position will ever ask for them again. The cache becomes a rolling buffer that overwrites its own oldest entry, and its size stops depending on sequence length entirely. Note how different this is from the other levers. Reduced precision trades accuracy for space; eviction gambles on which tokens mattered. A bounded window changes the question instead, so the memory was never owed in the first place. The bill lands elsewhere: distant context must now be passed forward through the positions in between, rather than read directly.
What is disputed
The sources here describe grouped-query attention and sliding-window attention as design choices aimed at faster inference and longer sequences; they do not settle how much quality any given compression costs. Claims about reduced-precision caches, and about which tokens can be evicted without harm, are model- and task-dependent and should be measured on your own workload rather than assumed. The relative bar lengths in the diagram are schematic, illustrating direction of change only.
Remember this
A model's weights are a fixed cost; its key-value cache grows with every token it reads. Long-context inference is largely the work of shrinking that cache — and every way of shrinking it charges something in return.
Test yourself
A colleague halves the memory a long-context deployment uses in two different ways: on one server by storing cache entries at lower precision, on another by switching to a model trained with sliding-window attention. Both fit twice the input. Why are these not interchangeable choices?
They fail differently, and they are available at different moments. Lower precision keeps every token's key and value — attention can still look directly at any position — but each stored number is coarser, so outputs may drift from the full-precision result in ways that are hard to predict and produce no error message. It is also a runtime setting, applicable to weights you already have. A bounded window keeps its numbers exact but discards entries once no live window can reach them, so a detail stated early in a long input is no longer looked at directly; it survives only if intervening layers carried it forward. That is an architectural property, fixed before training, so it comes with the model rather than being switched on later. One degrades fidelity everywhere and uniformly; the other removes direct access to the distant past while leaving what remains untouched.
Go deeper
- [2310.06825] Mistral 7B · arxiv.org
- [2404.16849] Smart Grids Secured By Dynamic Watermarking: How Secure? · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.