II · THE IDEA · ARTIFICIAL INTELLIGENCE
The Context Window
▶ Listen · narrated
You assume the model recalls what you said ten minutes ago. It does not. Unless your client re-sent that text, the model never saw it this turn.
At a glance
- What it is
- The maximum number of tokens the model process in one inference call
- Contains
- System prompt, conversation history, current input — everything you want the model to consider
- When it fills
- Oldest messages are dropped, or the call fails, depending on the client
- Attention cost
- Scales quadratically with context length in standard transformers
Imagine a desk with room for exactly fifty pages. Every time you ask the model a question, you must put all the pages you want it to read on that desk: the instruction manual, the previous conversation, your new question. If you have sixty pages, ten must be removed. The model reads only what is on the desk, and when it finishes answering, the desk is cleared. Next time, you start again. If you want the model to remember the earlier conversation, you must put those pages back on the desk yourself. The model has no filing cabinet. The desk is everything.
The context window is the maximum sequence length the model's positional encoding and attention mechanism can handle in a single forward pass. It is fixed during training: if a model is trained with sinusoidal or learned positional embeddings for positions 0 to 8,191, you cannot simply pass it a sequence of length 16,000 without modifying the embeddings, and doing so often degrades performance. During inference, all input — system prompts, chat history, user message, retrieved documents — is concatenated into a single token sequence. If the total exceeds the window size, the client must truncate, summarise, or reject the call. The model itself is stateless between calls. Attention is computed over the provided context only, with no access to previous turns unless they are explicitly included in the current input. In standard Transformer attention, each token attends to all others in the context, giving O(n²) complexity in sequence length. Variants like Longformer use sparse or local attention patterns to reduce this to O(n), enabling longer windows at the cost of some representational capacity. The Liu et al. finding that models attend less reliably to tokens in the middle of long contexts suggests that simply increasing window size does not uniformly improve performance, and that position within the context affects how information is weighted during generation.
Look closer
There is no memory outside the window
If your chat interface feels continuous, that is the interface doing work, not the model. Each time you send a message, the client bundles your new text with as much prior conversation as will fit, then submits the whole package. The model sees a single block of tokens and produces a reply from that alone. If earlier turns have been truncated to make room, the model has no access to them. It is not summarising them in the background or storing a gist. They are gone.
Position in the window affects use
Liu and colleagues tested models on a task requiring them to extract one relevant document from a set of distractors, then varied where in the context that document appeared. Performance was highest when the document sat at the very beginning or the very end, and worst in the middle. The U-shaped curve held across models and context lengths. The paper's title, Lost in the Middle, has become shorthand for the finding that long context and effective use of long context are not the same thing.
Extending the window is not free
Attention in the original Transformer architecture compares every token to every other token, so doubling the context length quadruples the computation. Beltagy and colleagues introduced Longformer, which replaces full attention with a mixture of local windows and a small number of global tokens, reducing the cost to linear. Other architectures use sliding windows, sparse patterns or hierarchical schemes. Each trades some modelling flexibility for the ability to handle longer inputs, and the trade-offs differ. A model trained with one attention pattern cannot trivially adopt another.
The story
The context window is a budget, not a feature. It sets the maximum number of tokens the model can attend to when producing a response, and that maximum is fixed when the model is trained. If the architecture was trained with 8,192-position embeddings, you cannot later ask it to handle 16,384 tokens without retraining or interpolation tricks that may degrade quality.
Everything the model considers must fit inside that budget. The system prompt that sets behaviour, the conversation so far, the user's new message, any documents you want the model to reference — all of it counts against the same limit. When you paste a long document into a chat interface and ask a question about it, you are not giving the model a separate memory. You are filling the context window, and the model's answer is generated from that single, flat span of tokens.
If the total exceeds the limit, something must be removed. Some clients truncate the oldest messages silently. Others return an error and require you to shorten the input manually. A few attempt to summarise earlier turns and substitute the summary, but that summary is itself produced by a model with a context window, and it necessarily discards detail. There is no lossless compression here.
The window is also where computational cost lives. In the standard Transformer attention mechanism, every token attends to every other token, which means the number of comparisons grows with the square of the context length. Double the context, quadruple the work. For a model with an 8,192-token window, that is more than 67 million token-pair comparisons per layer. Scale to 128,000 tokens and the cost becomes prohibitive without architectural changes.
Longformer, introduced by Beltagy and colleagues, replaced global attention with a combination of local sliding windows and a small set of tokens that attend globally. The result is linear scaling rather than quadratic, making much longer contexts tractable. Other approaches include sparse attention patterns, where only certain token pairs are computed, and hierarchical schemes that process the input in chunks. Each method trades some representational power for efficiency, and the trade-offs are not uniform across tasks.
Extending a trained model's context window after the fact is difficult. Position embeddings are learned during training for a specific maximum length. Techniques exist to interpolate or extrapolate those embeddings, but they often degrade performance, particularly on tasks that require precise positional reasoning. A model trained on 4,096 tokens will struggle if you simply feed it 8,000, even if the architecture could theoretically handle it.
Liu and colleagues documented another limitation: models do not use long contexts uniformly well. They tested retrieval tasks where a relevant document was buried among distractors and found that performance dropped sharply when the relevant information appeared in the middle of the context. The effect held across multiple models and context lengths. The paper's title, Lost in the Middle, has since become a shorthand for this behaviour. A large context window is not the same as effective use of that window.
Why it mattered then
The context window was a practical necessity before it was a design parameter. Early recurrent models processed sequences one token at a time and maintained a hidden state, but that state compressed everything seen so far into a fixed-size vector, and information decayed. The Transformer's self-attention mechanism allowed every token to attend to every other token directly, which solved the compression problem but introduced a new one: cost scaled quadratically. The context window became the point where you drew the line. It was the maximum span over which full attention was feasible, and it defined what the model could consider in one pass. For the original Transformer in 2017, that was 512 tokens. For GPT-2 in 2019, 1,024. For GPT-3 in 2020, up to 2,048 for the largest variant. Each step required more memory and more computation, and each step was constrained by what hardware could handle during training.
Why it matters now
The context window remains one of the most visible specifications when a new model is announced, and the numbers have grown rapidly. Models with 32,000-token windows are common, and some now claim 128,000 or more. But the Liu finding — that models lose track of information in the middle of long contexts — has not been solved by making the window larger. It has made the engineering problem more urgent. Systems now routinely include retrieval layers that fetch only the relevant portions of a document, or re-ranking steps that move important tokens closer to the edges of the window where the model attends more reliably. The window is large enough that the bottleneck has shifted from whether you can fit the information in, to whether the model will actually use it. That is a different kind of problem, and it is not yet clear that architecture changes alone will resolve it. The context window is no longer just a limit. It is a space that must be managed.
The surprising detail
The context window is emptied after every response. This is not a limitation of current systems that future versions will fix; it is intrinsic to how inference works. The model does not maintain state between calls. If your chat application feels continuous, that continuity is being constructed by the client, which saves your conversation to disk or memory and re-submits it with each new message. The model itself is stateless. Some research has explored ways to cache intermediate representations so that repeated context does not need to be reprocessed from scratch, but even with caching, the model is not remembering in the sense that you are. It is recomputing or retrieving, and the distinction matters when you try to understand why a conversation went a particular way.
Remember this
The context window is everything the model sees this turn. If something is not in it, the model has no access to it, no matter what happened earlier.
Test yourself
A client submits a 10,000-token context to a model with a 8,192-token window. The client does not receive an error. What are two different ways the system might have handled the excess, and what information would the model lose in each case?
First, the client may have truncated the oldest tokens, keeping the most recent 8,192. The model loses the beginning of the conversation or document, which may contain important framing, definitions, or earlier references that later text assumes you remember. Second, the client may have summarised the early portion and replaced it with a shorter version, then appended the recent tokens. The model loses whatever detail the summary omitted, and it also loses the exact wording, which can matter for tasks like legal or technical analysis where phrasing is significant. A third, less common approach is to split the context into chunks and process them separately, then combine the results, but that is not a single inference call and the model never sees the chunks together, so cross-reference between them is impossible.
Go deeper
- Lost in the Middle: How Language Models Use Long Contexts · arXiv · Nelson F. Liu et al. · 2023-07-06
- Longformer: The Long-Document Transformer · arXiv · Iz Beltagy et al. · 2020-04-10
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.