II · THE IDEA · ARTIFICIAL INTELLIGENCE
Sliding Window Attention for Infinite Context
▶ Listen · narrated
Context windows feel like a hard wall until attention itself is redesigned. Restrict what each token may see, and the quadratic cost that enforced the wall no longer applies.
At a glance
- Core idea
- Each token attends only inside a fixed local window
- Cost shape
- Linear in sequence length instead of quadratic
- Memory
- Stays roughly constant as the sequence grows
- Longformer
- Sliding window, dilated window, and global attention
- RoFormer
- Rotary position embeddings for relative position
Think of reading a long scroll through a narrow moving frame. You never see the whole scroll at once. You only see the part under the frame, and you slide the frame along as you go. Each moment of reading is cheap because the frame never gets wider, no matter how long the scroll is.
Sliding window attention works like that frame. In a normal transformer, every word looks at every other word. That gets expensive fast as the text grows. With a sliding window, each word looks only at its neighbours inside a fixed-size neighbourhood. Far-away words are out of view in that layer. If you stack many layers, news from farther away can pass along the chain, hop by hop. A few special words can be allowed to look at the whole text so the model still has somewhere to hang big, document-wide facts.
The win is simple: the amount of attention work grows steadily with length instead of exploding. The cost is also simple: the model is not truly staring at everything at once.
Dense self-attention computes, for sequence length n and head dimension d, interactions between all query and key positions, with time and memory dominating at O(n²) for the attention map (before value aggregation). Sliding window attention restricts position i to keys in [i−w, i+w] (or a causal half-window), yielding O(n·w) attention cost with w fixed at design time. The mask is a band matrix; implementation may use blocked sparse kernels or explicit neighbourhood gathers rather than materialising a full n×n matrix.
Longformer combines three patterns: (1) sliding window local attention; (2) dilated sliding windows that attend every d-th token inside an extended span, increasing receptive field without raising the number of attended positions per token; (3) global attention on a designated subset of positions that attend to all tokens and are attended by all, restoring limited full-sequence pathways while preserving linear scaling if the global set is O(1) or otherwise small relative to n. Stacking L layers expands the theoretical receptive field roughly as O(L·w) under plain windows, larger under dilation, analogous to stacked convolutions.
RoFormer replaces or augments absolute position encodings with rotary position embeddings (RoPE): query and key vectors are rotated by position-dependent angles so that the dot product is a function of relative offset. That relative structure suits long sequences and pairs cleanly with local windows, which are themselves defined in relative coordinates. Limitations remain: information outside the multi-layer receptive field must travel via hidden-state compression or global tokens; naive windowing can hurt tasks that need dense long-range binding; and length generalisation still depends on training distribution and position scheme, not only on the mask.
Look closer
The attention matrix becomes a band
In ordinary self-attention every query may look at every key, so the attention matrix is dense. Under a sliding window the allowed connections form a band along the diagonal: token i may attend only to tokens inside a neighbourhood of fixed width. Cells outside that band are masked. The diagram of the pattern is almost the whole idea — sparsity is not a side-effect but the design.
Stacking widens the receptive field
A single layer sees only its window. Stack several layers and information can hop across windows, much as a convolutional stack builds a larger effective field from small kernels. Dilated windows, used in Longformer, skip tokens inside the window so the same budget of attended positions reaches farther without filling the band solid.
A few tokens still see everything
Pure local attention struggles with tasks that need a document-level signal. Longformer therefore lets selected tokens use global attention: they attend to the full sequence, and the full sequence may attend back to them. Those positions act as hubs. The rest of the sequence stays inside the sliding window, so the overall cost remains linear in length if the number of global tokens is kept small.
The story
Standard transformer self-attention asks every token to compare itself with every other token in the sequence. That comparison is powerful and simple, but the work and the memory both scale with the square of the length. Double the document and you roughly quadruple the attention cost. Past a few thousand tokens the bill becomes the binding constraint on what the model can read in one pass.
Sliding window attention changes the contract. Each token is allowed to attend only to a fixed neighbourhood — a window of recent and nearby positions rather than the entire history. The attention matrix is no longer dense; it is a narrow band. Because the window width is chosen by the architect and does not grow with the document, the cost per layer grows roughly in proportion to sequence length, not its square. Memory for the attention pattern can stay effectively constant as more tokens stream through, which is what the phrase “infinite context” is gesturing at in practice: not magic, but the removal of the quadratic wall.
Longformer set out this family of patterns for long documents. The basic ingredient is the sliding window. A dilated variant spaces the attended positions so the same number of connections spans a wider range — useful when depth is limited but span still matters. On top of the local pattern, a small set of tokens may be marked global. Those tokens attend broadly and are attended to broadly, giving the model places to park document-level information without restoring a full dense matrix.
Position information still has to be supplied somehow. Absolute position indices that were trained only up to a fixed maximum do not automatically behave well when the sequence grows past that maximum. RoFormer proposed rotary position embeddings, which rotate query and key vectors as a function of position so that the attention score depends on relative offset. Relative structure of that kind pairs naturally with long or open-ended contexts, because what matters to the sliding window is how far apart two tokens are, not which absolute index they hold in a pretrained table.
The resulting picture is deliberately modest. The model does not truly “see” an unbounded past in one glance. It sees a local band at each layer, optionally a few global hubs, and a position scheme that speaks in offsets. Information from far away must travel through the stack or through those hubs. That is a real limitation, and it is also the reason the memory footprint no longer explodes. For long documents, sparse local attention is a trade: less all-to-all communication per layer, in exchange for length that full attention could not afford.
Why it mattered then
When Longformer appeared, transformer models were already the default for language, yet most practical systems still truncated inputs hard because dense self-attention could not be paid for at document scale. Work on long documents — full papers, books, multi-document settings — either chopped text into isolated chunks or accepted severe length caps. A pattern that kept the transformer block intact while making attention scale linearly opened a route to train and run models on sequences that denser attention simply priced out. RoFormer’s rotary embeddings addressed a companion problem: how to encode position so that longer sequences remain meaningful rather than extrapolating clumsily from a short trained range. Together, local sparse attention and relative position schemes marked a shift from “make the context window slightly larger” toward “change the cost structure so length is no longer the enemy.”
Why it matters now
Context length is still one of the first specifications people compare when choosing a model, and the bill for dense attention has not gone away. Sliding-window and other local patterns remain a standard tool in the efficiency kit: they appear in long-context architectures, in streaming setups, and wherever memory on a single device must stay bounded while tokens keep arriving. The same intuition — attend nearby by default, spend full attention only where it is worth it — shows up in later sparse and hybrid designs. Rotary position embeddings, or close relatives, are now commonplace in open-weight models precisely because relative position handles stretch and shift more gracefully than a fixed absolute table. Understanding the sliding window is still the clearest way to see why “longer context” is sometimes an attention-pattern problem rather than only a hardware problem.
The surprising detail
“Infinite context” here does not mean the model holds an infinite past in active attention. It means the incremental cost of another token need not grow with how many tokens came before. The past is still filtered through a fixed-width aperture at each layer. Far-away detail survives only if it has been compressed into the hidden state, carried by stacked receptive fields, or written into a global token. The phrase describes a memory scaling regime, not omniscience.
What is disputed
How far stacked local windows transfer information in practice depends on depth, window width, dilation, and training data; the papers propose the patterns and show long-document gains, but they do not establish a single universal window size or prove that local attention matches dense attention on every task. Claims of unbounded context describe asymptotic memory behaviour under the sparse pattern, not measured quality at arbitrary length.
Remember this
A sliding window turns dense all-to-all attention into a fixed local band, so sequence length can grow while attention memory stays bounded.
Test yourself
A document is far longer than one attention window. After several stacked sliding-window layers, why can a token at the end still be influenced by material near the start — and when might that path still fail?
Each layer lets information move by roughly half a window (or farther with dilation). Across depth, signals hop along the sequence, so the effective receptive field grows with the number of layers. The path fails when the stack is too shallow for the distance, when dilation and window width leave gaps that never connect, or when important document-level facts were never placed on global tokens or compressed into states that travel. Local attention alone does not guarantee end-to-end visibility; it only guarantees a route whose length depends on depth and window design.
Go deeper
- [2004.05150] Longformer: The Long-Document Transformer · arxiv.org
- [2104.09864] RoFormer: Enhanced Transformer with Rotary Position Embedding · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.