II · THE IDEA · ARTIFICIAL INTELLIGENCE
The Recurrent Era and Its Bottleneck
▶ Listen · narrated
A recurrent network could translate short sentences well enough. Then researchers tried longer paragraphs, and the first words started vanishing from memory before the last ones arrived.
At a glance
- Processing order
- Sequential — one token per step, left to right
- Memory mechanism
- Hidden state vector updated at each step
- Training constraint
- Each step depends on the previous one, preventing parallelisation
- Long-range problem
- Information from early tokens degrades as the sequence continues
Imagine you are summarising a long article for a friend, but you can only remember one sentence at a time. You read the first paragraph and compress it into a sentence. Then you read the second paragraph, combine it with your one-sentence summary, and compress that into a new single sentence. By the tenth paragraph, your summary still fits in one sentence, but details from the first paragraph have been compressed and re-compressed so many times that much of the original information is lost. That is the recurrent network's problem: it maintains a fixed-size memory that must summarise everything so far, and early information fades as the sequence grows. The second problem is speed. If you must read and summarise each paragraph in strict order — you cannot start paragraph five until you have finished paragraph four — then you cannot work faster by recruiting more help. Recurrent networks had this constraint during training: each step depended on the previous one, so the process could not be parallelised even on hardware designed for parallel work.
A recurrent neural network processes a sequence by iterating over positions, maintaining a hidden state vector h_t that is updated at each step as a function of the current input x_t and the previous hidden state h_(t-1). Formally, h_t = f(W_h · h_(t-1) + W_x · x_t + b), where f is a nonlinearity and W_h, W_x are learned weight matrices. LSTMs extend this with gating mechanisms — forget, input, and output gates — that modulate what information is retained or discarded, which mitigates but does not eliminate the degradation of long-range dependencies. The hidden state remains a fixed-dimension vector, so the model's capacity to represent information about the sequence is bounded regardless of sequence length. During training, backpropagation through time requires computing gradients sequentially from the final position back to the first, because each hidden state depends on the previous one. This creates a critical path of length T for a sequence of length T, preventing parallelisation across the time dimension. Batch parallelism is possible — you can process multiple sequences simultaneously — but within a single sequence, the computation is inherently serial. This makes training time scale linearly with sequence length in a way that cannot be mitigated by additional compute resources, a limitation that became prohibitive as researchers moved toward longer contexts and larger datasets.
Look closer
The hidden state is a fixed-size summary
At each step, the network reads one token and the hidden state from the previous step, then produces a new hidden state to pass forward. That vector — typically a few hundred or thousand numbers — must somehow encode everything relevant from the entire sequence so far. Early tokens leave traces that weaken with each update. By the time a model has processed fifty words, the influence of the first word on the current hidden state is indirect, filtered through dozens of successive transformations.
LSTMs added gates to control forgetting
A plain recurrent network updates its memory at every step with no mechanism to decide what to keep. The Long Short-Term Memory architecture introduced gates: small neural networks that learn to open or close, controlling what information flows into the hidden state, what gets forgotten, and what gets passed to the output. This helped substantially with sequences of moderate length, but the fundamental bottleneck remained. The hidden state was still a fixed-size vector, and every piece of information still had to pass through it.
Training required processing the entire sequence in order
To compute gradients during training, you need the hidden state at each position. But the hidden state at position fifty depends on the state at forty-nine, which depends on forty-eight, and so on back to the beginning. You cannot compute them in parallel. A modern GPU can perform thousands of operations simultaneously, but a recurrent network forced it to wait at each step for the previous step to finish. Training on long sequences was therefore slow in a way that more hardware could not fix.
The story
Recurrent neural networks process sequences by maintaining a hidden state: a vector of numbers that gets updated at each step. You feed in the first token and an initial hidden state, and the network produces a new hidden state. Then you feed in the second token and that new hidden state, producing another state. The process continues, one token at a time, until the sequence ends.
This design mirrors something intuitive about how we experience language. You read a sentence from left to right, and at any moment you are holding in mind some summary of what has come before. The hidden state is meant to be that summary, compressed into a few hundred or thousand numbers.
The compression is the problem. Suppose you are translating a paragraph from English to French. The network reads the English sentence word by word, updating its hidden state as it goes. When it reaches the end, that final hidden state must contain enough information to generate the entire French translation. Every proper noun, every verb tense, every relationship between clauses — all of it must be encoded in a single fixed-size vector.
For short sequences, this works tolerably well. For longer ones, information begins to degrade. The hidden state from thirty steps ago has been transformed thirty times, and the signal from the original token has been diluted at each step. Sutskever and colleagues, working on sequence-to-sequence learning, found that reversing the order of the source sentence improved translation quality, apparently because it reduced the distance between corresponding words in the two languages. That helped, but it was a workaround, not a solution.
LSTMs improved matters by adding gating mechanisms. Instead of updating the hidden state indiscriminately at every step, the network learned to control what information to keep, what to discard, and what to let through to the output. Empirical evaluations showed that LSTMs handled longer dependencies better than plain recurrent networks. But the core constraint remained: everything still had to flow through a single hidden state vector.
The second problem was parallelisation. Training a neural network involves computing gradients, and for a recurrent network, the gradient at step fifty depends on the hidden state at step fifty, which depends on step forty-nine, and so on. You cannot skip ahead. This sequential dependency meant that even with powerful hardware, training was bound by the length of the sequence. A graphics card capable of thousands of parallel operations was forced to process each token in series, waiting for the previous step to complete.
These two limitations — the fixed-size memory bottleneck and the inability to parallelise training — defined the recurrent era. The architecture could handle language, but it struggled with long-range dependencies and trained slowly. When attention mechanisms arrived, they addressed both problems directly: memory became dynamic rather than fixed, and all positions could be computed simultaneously.
Why it mattered then
Recurrent networks represented the first architecture that could genuinely process sequences of arbitrary length. Before them, neural networks required fixed-size inputs. The ability to maintain a hidden state and update it step by step meant you could, in principle, handle a sentence of any length, or a paragraph, or a document. Sutskever and colleagues demonstrated in 2014 that a sequence-to-sequence model built from LSTMs could translate between languages without any hand-engineered rules about grammar or word order. The network learned the structure of both languages from data alone. That was a meaningful achievement, even though the translations degraded noticeably on longer sentences. The recurrent architecture proved that end-to-end learning on language was possible. It also made the bottleneck visible: researchers could see exactly where and how the models failed, and that clarity helped direct the search for better designs.
Why it matters now
Recurrent networks are largely absent from modern large language models. Transformers replaced them because attention solved both the memory bottleneck and the parallelisation problem more effectively. But the recurrent era clarified what the problems were. The hidden state bottleneck is still a useful concept when thinking about context windows and memory in any architecture. And the sequential-versus-parallel distinction matters whenever you are choosing between processing strategies: streaming versus batch, autoregressive versus non-autoregressive, online learning versus offline. Understanding why recurrence became a limitation helps explain why attention was designed the way it was, and why training large models became feasible only after architectures allowed true parallelisation across sequence positions.
The surprising detail
Reversing the input sequence improved translation quality in the Sutskever et al. experiments, apparently because it shortened the path between corresponding words in the source and target languages. If you are translating 'The cat sat on the mat' to French, and you feed the English words in reverse, 'mat' arrives shortly before you need to generate 'tapis', rather than having travelled through the entire hidden state chain from the beginning of the sentence. This worked well enough to be adopted in practice, but it is a striking admission of the architecture's limitation: the solution to a memory problem was to rearrange the input so the model did not have to remember as far back.
Remember this
Recurrent networks processed sequences one step at a time, compressing everything into a fixed-size hidden state. That bottleneck limited long-range memory, and the step-by-step dependency made training slow.
Test yourself
A researcher trains two models on the same translation task: one recurrent, one attention-based. Both reach similar accuracy on short sentences, but the recurrent model's loss stops improving after sequences exceed forty tokens, while the attention model continues to improve. Training the attention model also takes half the wall-clock time despite having more parameters. Explain both observations.
The recurrent model's plateau on long sequences reflects the hidden state bottleneck. Information from early tokens has degraded by the time the model processes token forty, so adding more tokens does not improve the representation — the model cannot make effective use of context it can barely remember. The attention model avoids this because every token can attend directly to every other token, so long-range dependencies do not have to pass through a long chain of hidden states. The training time difference comes from parallelisation. The recurrent model must process tokens sequentially during both the forward and backward pass, so it cannot exploit the GPU's parallel capacity. The attention model computes representations for all positions simultaneously, which means more operations overall but far better hardware utilisation. Wall-clock time is determined by the longest serial dependency, not by the total number of operations.
Go deeper
- Sequence to Sequence Learning with Neural Networks · arXiv · Ilya Sutskever et al. · 2014-09-10
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling · arXiv · Junyoung Chung et al. · 2014-12-11
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.