II · THE IDEA · ARTIFICIAL INTELLIGENCE
The Transformer Architecture Paper
▶ Listen · narrated
For years, translating a sentence meant processing it one token at a time. The Transformer discarded that serial path and still improved the numbers that mattered.
At a glance
- Paper
- Attention Is All You Need (Vaswani et al., 2017)
- Core claim
- Sequence transduction without recurrence or convolution
- Mechanism
- Multi-head self-attention plus positional encodings
- En–De BLEU
- 28.4 on WMT 2014 English-to-German
- En–Fr BLEU
- 41.8 on WMT 2014 English-to-French
- Training note
- Big model trained in 3.5 days on eight GPUs
Think of an old assembly line where each worker can only pass a note to the next worker along the belt. By the time a message from the first word of a sentence reaches the last word, it has been handed on many times and training has to wait for that chain. The Transformer replaces the belt with a meeting room: every word can look at every other word in one go, decide how much to listen, and write a summary. Stack those meetings a few times, add a mark so the model still knows word order, and you can translate without the chain.
In plain terms, the 2017 paper builds an encoder and a decoder only from attention and small feed-forward networks. Attention scores how relevant each token is to each other token, mixes their values, and does this with several heads at once so different kinds of links can be learned together. Positions are added as extra patterns on the input. On standard English–German and English–French news-translation tests, this setup improved BLEU and trained with full parallelism across the sentence length.
The Transformer is an encoder–decoder for sequence transduction with no RNN or CNN layers. Input tokens are embedded and added to positional encodings (fixed sinusoids of varying frequency; learned embeddings performed similarly on the paper’s MT tasks). The encoder stacks N=6 identical layers, each residual block containing multi-head self-attention then a position-wise MLP (two linear maps with ReLU, inner width 2048 in base). The decoder stacks N=6 layers with masked self-attention, cross-attention into the encoder memory, and the same MLP. Base width d_model=512 with h=8 heads so d_k=d_v=64; the big model uses d_model=1024, h=16, and larger feed-forward width.
Scaled dot-product attention is softmax(QK^T / sqrt(d_k)) V. Scaling counters the growth of dot products with dimension. Multi-head attention uses separate projections per head, concatenates, and projects back. Decoder self-attention masks future positions to preserve auto-regressivity. The authors compare complexity: self-attention is O(n² · d) per layer with O(1) maximum path length between positions, versus O(n) path length for recurrent layers. That constant path length is the stated reason long-range dependencies are easier to learn. On WMT 2014 the big model reports 28.4 BLEU (En–De) and 41.8 BLEU (En–Fr), trained 3.5 days on eight GPUs. Limitation carried forward: full self-attention’s quadratic cost in n, accepted for sentence MT but central to later efficient-attention work.
Look closer
Attention as the only path between positions
In a recurrent encoder, information from the first token reaches the last only by passing through every step in between. The paper replaces that chain with scaled dot-product attention: each position computes a weighted sum over all positions in a single step. The maximum path length between any two tokens becomes constant, which is the architectural reason long-range dependencies become cheaper to learn.
Eight heads, not one
A single attention distribution can collapse onto one kind of relation. The Transformer runs several attention heads in parallel — eight in the base configuration — each with its own projections, then concatenates and re-projects the results. Different heads can specialise on different linkages; the paper reports that this multi-head design works better than one wider head with the same total compute.
Order injected by sine and cosine
Without recurrence or convolution, the model has no built-in sense of sequence order. The authors add positional encodings to the input embeddings: fixed sinusoids at different frequencies, one pair per dimension. They also tried learned positional embeddings and found nearly identical results on the reported translation tasks, keeping the sinusoidal version for its potential to extrapolate beyond training lengths.
The story
By 2017, the dominant pattern for sequence-to-sequence work was an encoder–decoder built from recurrent networks, often LSTMs or GRUs, with attention bolted on as a bridge between the two stacks. Convolutional alternatives existed, but they still required stacked layers to grow the receptive field. Both families processed or combined positions in ways that limited parallel computation across the sequence length. Training large models on long sentences was slow for structural reasons, not only for lack of hardware.
Attention Is All You Need proposed a cleaner cut. The Transformer keeps the encoder–decoder framing familiar from machine translation, but removes recurrence and convolution entirely. Each encoder layer has two sub-layers: multi-head self-attention over the encoder positions, then a position-wise feed-forward network. Each decoder layer has three: masked self-attention over previous decoder outputs, multi-head attention into the encoder stack, and another feed-forward network. Residual connections and layer normalisation wrap every sub-layer. The base model uses six identical layers on each side, model dimension 512, feed-forward width 2048, and eight attention heads.
The attention itself is scaled dot-product attention. Queries, keys and values are linear projections of the incoming representations. Compatibility between a query and a key is their dot product, divided by the square root of the key dimension so that large depths do not push the softmax into regions with vanishing gradients. The softmax over those scores produces weights that mix the values. Multi-head attention repeats this with different learned projections and concatenates the heads. In the decoder’s self-attention, future positions are masked so that generation remains auto-regressive.
Because the architecture no longer walks the sequence step by step, order must be supplied explicitly. The authors add sinusoidal positional encodings to the token embeddings; they also report that learned positional embeddings perform nearly as well on the translation benchmarks they measured. Stacked above those summed embeddings, the self-attention layers can in principle draw a direct connection between any pair of positions in a single step — a constant path length, in contrast to the linear path length of a recurrent net or the logarithmic growth of stacked convolutions.
On WMT 2014 English-to-German, the big Transformer reached 28.4 BLEU. On English-to-French it reached 41.8 BLEU. The big configuration was trained for 3.5 days on eight GPUs. The paper also reports that the models trained faster than the contemporary recurrent and convolutional baselines they compared against, under the setups described. A later line of work, including Generating Wikipedia by Summarizing Long Sequences, took the decoder-only and encoder-focused variants of the same attention stack into longer-document tasks, showing that the architecture was not tied to bilingual translation alone.
What the paper did not claim is that attention is free. Self-attention is quadratic in sequence length, a cost the authors state plainly when they compare layer types. For the sentence lengths common in news translation in 2017 that trade-off was acceptable. The lasting move was simpler: once recurrence was no longer required for competitive transduction, the field could train larger models with far more parallelism across positions.
Why it mattered then
Sequence transduction in 2017 was still organised around recurrence. State-of-the-art translation systems relied on multi-layer LSTMs or GRUs, sometimes with attention as an add-on, and training them at scale meant paying for sequential dependence along the length of every sentence. Convolutional sequence models offered more parallelism but needed depth to connect distant tokens. The Transformer arrived as a direct challenge to that settlement. It kept the encoder–decoder task framing that translation researchers already understood, yet stripped out the recurrent backbone and showed stronger BLEU on standard WMT 2014 English–German and English–French benchmarks, with training times measured in a few days on eight GPUs for the big model. The result mattered immediately because it attacked both quality and wall-clock training cost at once, under public benchmarks the community already used to keep score.
Why it matters now
Almost every large language model in wide use descends from this stack or from a decoder-only simplification of it. The paper’s separation of attention, feed-forward blocks, residuals and positional information is still the skeleton under modern training runs, even where later work has changed norms, activations, positional schemes or sparse attention patterns. The same design also explains present constraints. Quadratic self-attention, the need to inject position somehow, and the value of many heads rather than one are not folklore; they are engineering facts first laid out in this architecture. Reading the original paper remains a practical way to see which pieces were load-bearing from the start and which were open choices later generations revised.
The surprising detail
The title’s claim is almost literal: the architecture drops recurrence and convolution and still beats the systems built around them on the reported translation tasks. Equally striking is how small the positional experiment was. Sinusoidal encodings and learned embeddings performed nearly the same on those benchmarks; the authors kept the waves mainly for possible length extrapolation. A decision that later spawned a whole literature of rotary, relative and learned variants began as a near-tie on a side comparison.
What is disputed
BLEU gains and training-time claims are those reported by the authors on WMT 2014 under their experimental setup. Exact comparisons to concurrent systems depend on implementation details and hardware that later replications do not always match line for line. The paper’s own tables are the source for the figures given here.
Remember this
The Transformer showed that multi-head attention plus positions could replace recurrence for sequence transduction — and raised the translation numbers while training in parallel across tokens.
Test yourself
A recurrent encoder and a Transformer encoder both read a 40-token sentence. In terms of path length through the network, what is the essential difference in how information at token 1 can affect the representation at token 40, and why did that difference matter for training speed?
In the recurrent encoder the signal must pass through the intervening steps, so the path length grows with distance. In the Transformer, self-attention connects any two positions in a single step, so the maximum path length is constant (one attention operation, plus the depth of stacked layers). Because positions no longer depend on a left-to-right hidden state, computation across the sequence can proceed in parallel, which is why the architecture trained faster on long sentences under the setups the paper reports.
Go deeper
- [1706.03762] Attention Is All You Need · arxiv.org
- [1801.10198] Generating Wikipedia by Summarizing Long Sequences · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.