II · THE IDEA · ARTIFICIAL INTELLIGENCE
Retentive Networks for Long Sequences
▶ Listen · narrated
Key-value caches grow with every token you keep. Retention keeps a bounded state instead, so the work per step need not rise as the context lengthens.
At a glance
- Core swap
- Retention in place of multi-head attention
- Three forms
- Parallel, recurrent, and chunkwise recurrent
- Training mode
- Parallel over the sequence, as in Transformers
- Decode mode
- Recurrent update of a fixed-size state
- Stated aim
- Strong length scaling with constant incremental cost
Think of reading a book while writing a short running summary on one card. Each new sentence updates the card; you never carry every previous page under your arm. Training, though, is different: the teacher can lay many pages on the table at once and mark them together. Retention tries to be both of those workflows with the same underlying rule. In parallel form, the whole sequence is mixed at once so training can use big accelerator batches. In recurrent form, generation feeds one token at a time into a fixed-size state and writes a new state out. Chunkwise form processes a block in parallel, then passes the state to the next block. Multi-scale retention means several heads keep summaries with different fade rates—some cling to the near past, others to a longer trace—much as multi-head attention looks at the sequence in several learned ways.
RetNet swaps multi-head attention for multi-scale retention inside an otherwise familiar deep stack. Retention combines content-dependent projections with an exponential decay along positions, yielding a causal mixer. Algebraically the operator admits three implementations intended to realise the same mapping: a parallel form (sequence-wise matrix computation for training), a recurrent form (fixed-size state update for O(1) incremental decoding relative to history length), and a chunkwise recurrent form (intra-chunk parallel, inter-chunk state carry) for long sequences. Heads differ by decay scale, giving a multi-scale receptive structure without relying solely on absolute position embeddings. Practical deployment hinges on numerical parity between the parallel path used at train time and the recurrent path used at serve time; gating and normalisation in the block are part of keeping that parity. Relative to Transformer decoding, the recurrent view removes the growing key-value cache. Relative to classical RNNs, the parallel train-time layout is the point. Limitations follow from the state: capacity is finite, decay attenuates history, and long-range precise recall is not guaranteed merely because step cost is constant. Published results in arXiv 2307.08621 support competitiveness under the authors’ settings; they do not by themselves settle scaling laws or replace attention in every regime.
Look closer
One maths, three layouts
Retention is written so the same recurrence can be unrolled into a parallel matrix form for training, run step-by-step as a recurrent state for decoding, or hybridised into chunkwise recurrence. The parallel layout fills accelerators; the recurrent layout keeps memory and work per token from growing with context already seen.
Multi-scale retention
Heads are not identical copies. Different heads use different decay scales, so some emphasise recent tokens while others retain a longer trace. That multi-scale block is the direct counterpart to multi-head attention inside the layer stack.
State instead of a growing cache
In the recurrent view, each new token updates a fixed-size state and produces an output. There is no key-value list that lengthens with every step. The authors therefore describe inference cost as constant per token and context as unbounded in principle—qualified, in practice, by what that finite state can usefully hold.
The story
Transformers made large language models practical by letting every training token attend to every earlier one in parallel. The same design is awkward at deployment. Autoregressive decoding either recomputes attention over a lengthening context or stores a key-value cache that grows with every token kept. Memory and bandwidth become the constraint long before arithmetic does.
Retentive Network, proposed in 2023, attacks that mismatch at the mechanism level. It replaces multi-head attention with retention: a content-dependent mixing rule equipped with an exponential decay along the sequence. Retention is deliberately formulated so that three implementations compute the same family of functions. In parallel form, the full sequence is present and the work resembles a masked attention-style matrix—suited to training on GPUs. In recurrent form, a fixed-size state is updated one token at a time—suited to generation. Chunkwise recurrence sits between them, processing blocks in parallel while carrying state across block boundaries, which helps when sequences are long even during training.
Inside the block, multi-scale retention plays the structural role of multi-head attention. Heads differ by decay scale rather than only by learned projections, so the layer can mix short-range and longer-range traces without a separate absolute position embedding doing all the positional work. Gating and normalisation choices in the paper are there to keep the parallel and recurrent paths stable and aligned, because a mechanism that is only fast in theory is useless if the two layouts drift numerically.
The editorial claim in the paper’s title—a “successor to Transformer”—is the authors’ framing, not a settled consensus. What the design actually contributes is clearer and narrower: a sequence mixer that preserves parallel training while offering recurrent inference with a bounded state. That is the same ideal pursued by other linear-time and state-space lines of work. RetNet’s particular bet is that retention, with multi-scale decay and carefully matched dual forms, can sit in a Transformer-like stack and still scale.
On the lengths and sizes reported in the paper, the authors show competitive language-modelling results and favourable inference behaviour relative to Transformer baselines under their experimental settings. Readers should treat those comparisons as evidence from one study, not as a closed verdict on the architecture family. The deeper point for practitioners is architectural: if the mixer’s recurrent view really does keep a fixed-size state, then “infinite context” in the systems sense means constant memory and work per new token—not that every distant token remains equally recoverable from that state.
Why it mattered then
By 2023 the industry had already felt the inference tax of long contexts. Training could throw FLOPs at full attention; serving could not afford caches that grew without bound, especially for multi-user, long-session workloads. RetNet arrived as a direct proposal to keep the training recipe that had worked—deep stacks, parallel sequence mixing, large-batch GPU layouts—while changing the mixer so decoding no longer inherited quadratic cost or linear cache growth. It mattered in that moment because it joined a live argument: whether the next default backbone would still be attention with engineering patches, or a different recurrence-friendly operator with dual parallel and recurrent forms.
Why it matters now
Context windows advertised in products keep stretching, and the bill still shows up in memory bandwidth, cache management, and latency under load. Architectures that train in parallel yet decode with bounded state remain one of the few structural answers to that bill. Retention is part of the broader family of ideas—linear attention, state-space models, and other recurrent reformulations—that ask whether “infinite context” should be a systems property of the update rule rather than a larger and larger cache. Even where teams stay with Transformers, the RetNet framing clarifies what any successor must deliver: unchanged training throughput, matched numerical behaviour across layouts, and an inference path whose cost does not track history length.
The surprising detail
The same retention operator is meant to be exactly rearrangeable into parallel, recurrent, and chunkwise forms. That is a stronger claim than “we have a fast approximate mode.” The engineering burden shifts onto proving that training in one layout and serving in another do not silently disagree—hence the paper’s emphasis on formulation and on multi-scale decay as part of the block, not as a bolt-on positional trick.
What is disputed
The paper presents RetNet as a successor to the Transformer; that is an authorial claim. Comparative quality, stability at larger scales, and behaviour on tasks that need precise long-range retrieval remain matters for broader evaluation beyond a single study. “Infinite context” here refers to constant incremental inference cost with a bounded state, not guaranteed recovery of arbitrary distant detail.
Remember this
Retention keeps three layouts of one mixer: parallel for training, recurrent for constant-cost decoding, chunkwise when sequences are long.
Test yourself
A team wants longer conversations without growing a key-value cache. In RetNet’s recurrent form, what replaces that cache, and what limit does that replacement still impose on “infinite” context?
A fixed-size state updated at each step replaces the lengthening key-value cache, so memory and work per new token need not grow with history. Context is unbounded only in the systems sense; information still has to fit through that finite state, and older content can fade under the decay, so usable memory is not the same as perfect recall of every past token.
Go deeper
- [2307.08621] Retentive Network: A Successor to Transformer for Large Language Models · arxiv.org
- [2402.07597] Trustworthy SR: Resolving Ambiguity in Image Super-resolution via Diffusion Models and Human Feedback · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.