II · THE IDEA · ARTIFICIAL INTELLIGENCE
Long Context Is Not Uniform Attention
▶ Listen · narrated
A model advertises a context length in tokens, not in reliability. Performance on information buried mid-sequence can degrade sharply, even when the window has room to spare.
At a glance
- What it is
- Measured drop in retrieval accuracy for facts placed in the middle of a long context
- Where it peaks
- Information at the start and end of the context is recalled more reliably
- Observed in
- Multi-document question answering, needle-in-haystack retrieval tasks
- Implication
- A large advertised window is not a guarantee of uniform use
Imagine you are reading a very long list of facts, and someone later asks you a question that depends on one of them. If that fact was near the top or bottom of the list, you will probably remember it. If it was buried in the middle, you are more likely to have skimmed past it. Language models show a similar pattern, but for a different reason. They do not forget the middle; they pay less attention to it during processing. Attention is the mechanism that decides which parts of the input matter most for the current task, and in long contexts, attention tends to concentrate near the boundaries. The information in the middle is still present, but its signal is weaker by the time the model generates an answer. This matters because many real applications—legal document review, code analysis, multi-document question answering—depend on the model consulting a detail that may appear anywhere in a long prompt. If you know the middle is weak, you can design around it: put the most important material near the start or end, or repeat key facts in multiple locations.
The degradation is measurable and reproducible. Liu and colleagues tested models on multi-document question answering, where the input consists of a question followed by ten or more documents, only one of which contains the answer. Accuracy was highest when the relevant document appeared in the first or last position and lowest when it appeared in the middle. The effect held across architectures—decoder-only transformers, encoder-decoder models—and across scales from seven billion to seventy billion parameters. Hsieh's RULER benchmark extends the test to four task types: single-needle retrieval, multi-needle aggregation, multi-hop tracing, and variable tracking. Single-needle tasks show the mildest degradation; multi-needle tasks, which require combining information from several mid-context locations, show the steepest. The U-shape flattens somewhat in models trained on longer contexts or fine-tuned with better positional encodings, but it does not disappear. Attention scores, computed as softmax over the dot product of queries and keys, tend to be higher for boundary tokens, and this bias compounds across layers. Some evidence suggests the effect arises because training corpora place salient information near document boundaries more often than in the middle, so the model learns that statistical prior. Other evidence points to positional encodings that lose precision or generalise poorly at extreme indices. Both factors likely contribute, and interventions that address either one—data augmentation, rotary embeddings, ALiBi—reduce but do not eliminate the effect.
Look closer
The U-shape appears across architectures
Liu and colleagues tested models on tasks requiring retrieval of a single relevant document from a set of many. Accuracy was highest when the key document appeared at the very beginning or the very end of the input, and lowest when it sat in the middle. The pattern held across different model families and sizes, suggesting it is not an artefact of one particular design but a more general behaviour of how attention is distributed in long contexts.
RULER tests four distinct needle types
Hsieh and colleagues designed a benchmark that goes beyond simple retrieval. It asks models to find single facts, to aggregate information from multiple locations, to answer questions that require consulting widely separated passages, and to track frequencies across the entire context. Performance degrades differently depending on the task: single-needle retrieval holds up better than multi-needle aggregation, and the middle remains the weakest zone in nearly every condition tested.
Longer training context helps, but does not eliminate the effect
Models trained on longer sequences show flatter degradation curves than those trained on shorter ones, but the U-shape persists. Even a model fine-tuned specifically for extended context will show some preference for material near the boundaries. The effect diminishes with better positional encoding and more exposure to long documents during training, but it does not vanish entirely. Architecture and training regime both matter.
The story
When a model advertises a context window of a hundred thousand tokens, it is stating a technical limit: the maximum number of tokens it can process in a single forward pass without running out of memory or violating the constraints of its positional encoding. It is not promising that all hundred thousand positions are equally useful.
Liu and colleagues demonstrated this gap in 2023 by constructing a straightforward test. They gave models a question and a set of documents, only one of which contained the answer. They then varied where in the sequence that relevant document appeared. When it was first or last, models retrieved it reliably. When it was buried in the middle of a long list, accuracy dropped sharply. The effect was consistent across architectures and model scales, suggesting it reflects something about how attention is allocated rather than a quirk of one particular system.
The pattern resembles findings from human memory research, where primacy and recency effects are well documented, but the mechanisms are different. A transformer does not forget the middle; it attends to it less strongly, and that weaker signal propagates through every subsequent layer. By the time the model generates an answer, the information from mid-context has been diluted by many rounds of attention that favoured the edges.
Hsieh and colleagues extended this line of inquiry with RULER, a benchmark designed to test four distinct capabilities across long contexts. Single-needle retrieval asks the model to find one planted fact. Multi-needle retrieval requires aggregating information from several locations. Multi-hop tracing demands following a chain of references across distant passages. Variable tracking asks the model to count or summarise patterns that recur throughout the entire sequence. Performance degrades differently depending on the task, but the middle remains the weakest zone in nearly every condition.
The effect is not immutable. Models trained on longer sequences during pre-training or fine-tuning show flatter degradation curves. Better positional encodings help. Architectural changes that redistribute attention more evenly across the sequence can reduce the U-shape. But none of these interventions eliminate it entirely. Even state-of-the-art long-context models show some measurable preference for information near the boundaries, and that preference matters when the task depends on a detail that happens to sit in the wrong place.
Why it mattered then
The findings mattered immediately because they arrived just as models began advertising context windows in the tens or hundreds of thousands of tokens. The marketing emphasis was on capacity: how much you could fit in. Liu's paper demonstrated that capacity and usability are not the same thing. A model that can technically process a hundred thousand tokens may still perform poorly on tasks that require consulting information from the middle of that range. The research gave practitioners a reason to test their own workloads rather than trust the advertised number, and it prompted developers to investigate why attention was distributed unevenly and whether architectural changes could flatten the curve.
Why it matters now
The effect remains relevant because long-context models are now widely deployed in settings where uniform attention matters. Retrieval-augmented generation systems stuff dozens of documents into a prompt and assume the model will weigh them equally. Legal and medical applications depend on models consulting specific clauses or results that may appear anywhere in a long record. Code assistants are given entire repositories and asked to reason about dependencies scattered across many files. In all these cases, position-dependent recall is not an academic curiosity; it is a reliability problem. Knowing that the middle is weak lets you design around it: you can reorder documents by relevance, repeat key facts near the boundaries, or break a long context into smaller chunks and aggregate the results. The benchmarks also matter for evaluation. A model that scores well on short-context tasks may degrade sharply on long-context ones, and a model that handles long contexts may still fail when the critical information is poorly positioned. RULER and similar tests are now standard tools for characterising that behaviour.
The surprising detail
The U-shaped curve is reminiscent of human memory effects, but the mechanisms are entirely different. Humans exhibit primacy and recency because of the way episodic memory consolidates over time and the way working memory prioritises recent input. A transformer has no memory in that sense; every token is processed in parallel during the forward pass. The degradation happens because attention scores, computed from queries and keys, tend to be higher for tokens near the boundaries, and those higher scores compound across layers. The model is not forgetting the middle; it is attending to it less strongly from the start, and that weaker signal propagates. Some researchers have speculated that the effect arises because training data often places important information near the beginning or end of documents—titles, conclusions, first sentences—and the model learns that statistical regularity. Others point to positional encodings that may not generalise uniformly across very long sequences. The cause is likely a combination of both, and it is not yet fully resolved.
Remember this
A large context window is a capacity limit, not a promise of uniform attention. Information buried mid-sequence is recalled less reliably.
Test yourself
You are building a retrieval-augmented system that stuffs twenty documents into a prompt and asks the model to answer a question. The most relevant document is ranked fifth by your retriever. Name two strategies to mitigate position-dependent recall.
First, reorder the documents so the most relevant one appears at the start or end of the sequence, where recall is strongest. Many systems now place high-confidence results at the boundaries deliberately. Second, include the key fact or passage more than once, at different positions, so the model encounters it in a strong-attention zone even if it misses the original placement. A third option, more costly but sometimes necessary: break the task into multiple passes, each with a shorter context, and aggregate the answers. This avoids the long-context degradation entirely but requires more inference calls.
Go deeper
- Lost in the Middle: How Language Models Use Long Contexts · arXiv · Nelson F. Liu et al. · 2023-07-06
- RULER: What's the Real Context Size of Your Long-Context Language Models? · arXiv · Cheng-Ping Hsieh et al. · 2024-04-09
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.