II · THE IDEA · ARTIFICIAL INTELLIGENCE
Retentive Networks (RetNet)
▶ Listen · narrated
Long context costs a transformer twice: heavy training compute, then a key-value cache that grows with every new token. RetNet was offered as a way to cut that second bill.
At a glance
- What it is
- A neural architecture proposed as a successor to the transformer for large language models
- Inference memory
- O(1) in sequence length — constant rather than growing with context
- Training mode
- Parallel over the sequence, unlike classical recurrent networks
- Design aim
- Efficient handling of long sequences without giving up train-time parallelism
Think of reading a long letter aloud. A transformer keeps a growing pile of sticky notes — one for every word so far — so it can look anything up again. The pile gets heavier as the letter gets longer. A classical recurrent model keeps only one small notepad, updating it after each word, but it has to read the letter strictly in order when learning, which slows training on big computers.
RetNet is aimed at both needs at once. While the model is learning, it can still look at the whole sequence in parallel, in the way training hardware prefers. While it is generating text, the memory it carries forward stays a fixed size instead of growing with every new token. The point is not a smaller sticky-note pile; it is replacing the pile with a constant-size state without giving up parallel training.
RetNet is proposed as a drop-in sequence architecture for large language models that targets the transformer’s inference-memory scaling while preserving train-time parallelism. In a standard decoder-only transformer, self-attention over past positions is supported at decode time by a key-value cache whose storage grows as O(sequence length) per layer and per batch element. That linear dependence dominates device memory for long prompts and long generations.
RetNet’s stated contract is O(1) inference memory in sequence length: the recurrent state advanced from token t to token t+1 is fixed-size with respect to context length. Separately, the architecture is specified so that training can still be expressed as a parallel computation over the sequence, avoiding the purely sequential scan that makes classical RNNs inefficient on modern accelerators.
For implementers, the evaluation checklist is therefore twofold. First, measure resident state per token at decode time and confirm it does not scale with context length. Second, confirm that the training graph still parallelises across positions rather than forcing a time-major serial loop. Either property alone is familiar; the design claim is their conjunction inside one block intended for large-language-model stacks. Limitations and head-to-head quality at a given parameter count are empirical and outside the bare architectural brief.
Look closer
Two pressures, one stack
Language-model training wants full-sequence parallelism so hardware stays busy. Deployment wants memory that does not climb with every extra token of context. Transformers satisfy the first demand well and pay for the second with a cache that scales with length. RetNet is framed as a single design that tries to meet both pressures at once, rather than bolting a separate efficient decoder onto a parallel trainer.
What O(1) inference memory buys
Constant inference memory means the working state carried from one generated token to the next does not grow as the prompt or the completion lengthens. That is a different cost shape from attention with a key-value cache, where stored keys and values accumulate over the sequence. For long contexts the difference is not a small constant factor; it is whether memory is bounded by model width or by context length.
Parallel training still required
Classical recurrent models already offer compact step-by-step state, but they train sequentially along time, which is awkward on modern accelerators. RetNet’s editorial pitch is not recurrence alone: it is recurrence-like inference cost paired with training that can still run in parallel over the sequence. Without that second property, constant memory would be an old answer with a familiar training bottleneck.
The story
Retentive Networks, or RetNet, arrive in the literature under a direct claim: a successor to the transformer for large language models. The practical problem they address is easy to state even when internal mechanisms are left aside. Training a large language model rewards architectures that can see a whole sequence at once and map it across parallel hardware. Serving that model, especially with long prompts or long generations, rewards architectures whose memory footprint does not keep a growing ledger of past tokens.
The transformer resolved the training side with self-attention and became the default stack. Its inference side carries a well-known price. Keys and values from earlier positions are stored so later positions need not recompute them; that cache is linear in sequence length. For short chats the overhead is tolerable. For long documents, multi-turn histories, or batch serving under tight memory budgets, the cache becomes a first-class systems constraint, not a footnote.
RetNet is positioned against that trade-off. The design goal stated for it is efficient long sequences, achieved by holding inference memory to O(1) in sequence length while still allowing parallel training. Those two properties are doing different jobs. Constant inference memory is about what must be kept alive between decoding steps. Parallel training is about whether the loss over a full sequence can be computed without stepping token-by-token during optimisation. An architecture that only had the first property would resemble classical recurrence. An architecture that only had the second would resemble the transformer. The claim worth examining is the joint one.
That joint claim is why RetNet is discussed as an alternative rather than as a minor variant. If inference state truly does not grow with context, long-sequence serving stops being a memory-scaling problem in the same way. If training remains parallel, the path from research prototype to large-scale pre-training does not immediately reintroduce the sequential bottleneck that made pure recurrent language models hard to scale on modern chips. Whether any given implementation fully delivers both properties in practice is a separate empirical question; the architecture’s stated brief is to hold them together.
For someone reading the proposal in its own terms, the useful mental model is therefore not “faster attention” and not “just an RNN”. It is a redesign of the sequence block so that the representation carried forward at inference time stays fixed in size, while the training graph can still expose the whole sequence to parallel compute. Everything else in the surrounding system — tokenisers, optimisers, data pipelines — can stay familiar. The bet is concentrated in how sequence mixing is formulated.
Why it mattered then
By the time RetNet was proposed, large language models had largely standardised on the transformer, and the costs of that standard were no longer theoretical. Training runs already assumed dense parallel attention over long contexts. Deployment teams were simultaneously discovering that the key-value cache, harmless at modest lengths, dominated memory once prompts and generations stretched. The moment called for architectures that did not force a choice between train-time hardware efficiency and serve-time memory growth. RetNet’s contribution in that moment was to put a concrete alternative on the table under an explicit successor claim, aimed at long sequences rather than at a narrow academic benchmark.
Why it matters now
Long context remains a live product and systems problem. Models are asked to condition on documents, codebases, and multi-turn histories that make linear memory growth expensive, while training budgets still favour architectures that parallelise cleanly. RetNet matters as one clear formulation of the joint requirement: constant inference state and parallel training in a single design. Even when practitioners stay with transformers, the RetNet brief is a useful yardstick. It asks whether a proposed efficiency trick preserves both sides of the ledger, or only shifts cost from one phase to the other.
The surprising detail
The paper’s title does not hedge. It calls RetNet a successor to the transformer for large language models, not a variant, a hybrid, or a complementary block. That wording is unusually direct for an architecture paper and is part of why the work drew attention: it framed the problem as replacement under a specific efficiency contract — O(1) inference memory plus parallel training — rather than as a marginal improvement to attention.
What is disputed
“Successor to the transformer” is the authors’ framing in the paper title, not an established consensus. Public discussion of RetNet rests on that proposal and its stated efficiency goals; independent, long-running adoption at the largest scales is a separate question the supplied facts do not settle.
Remember this
RetNet’s brief is joint: train in parallel like a transformer, yet keep inference memory constant as sequences grow.
Test yourself
A team says they have “solved long context” because their new block uses a fixed-size state at generation time. What second property must you still check before treating it as a RetNet-style alternative for large-scale language modelling, and why?
Whether training can still run in parallel over the sequence. A fixed-size inference state alone is compatible with classical recurrence, which is awkward to scale on modern accelerators because it steps through time during training. RetNet’s claim is the conjunction: constant inference memory and parallel training. Without the second property you may have cheap decoding but a training bottleneck that the transformer had already removed.
Go deeper
- [2307.08621] Retentive Network: A Successor to Transformer for Large Language Models · arxiv.org
- [2310.05214] Resonant optical trapping of Janus nanoparticles in plasmonic nanoaperture · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.