Transformer Neural Machine Translation

Think of translating a sentence as organising a table of people who must all pass notes. In an older recurrent model the notes travel along a line: person one writes to person two, who writes to person three, and so on. Distant people wait on everyone in between.

The Transformer gives everyone a broadcast channel at once. In each layer, every position publishes a query (what am I looking for?), keys (what do I contain?) and values (what do I pass along?). Matching queries to keys decides how much of each value to mix in. Doing this several times in parallel—multi-head attention—lets the model track different kinds of link at once, such as syntax in one head and nearby wording in another.

Because that broadcast does not know left from right by itself, the model adds a positional pattern to each token’s embedding so order is still visible. Stack six such layers for reading the source sentence and six for writing the translation, let the writer’s layers also consult the reader’s final notes, and you have the architecture from the 2017 paper. On standard translation benchmarks it beat heavier recurrent systems while training in a long working day on eight GPUs.

The Transformer is an encoder–decoder transducer whose layers contain no recurrence and no convolution. Let d_model be the residual stream width (512 base, 1024 big). Token embeddings are scaled by sqrt(d_model) and added to positional encodings PE(pos, 2i) = sin(pos / 10000^{2i/d_model}), PE(pos, 2i+1) = cos(pos / 10000^{2i/d_model}), then fed to a stack of N = 6 identical encoder layers.

Each encoder layer is: x ← LayerNorm(x + MultiHead(x, x, x)) x ← LayerNorm(x + FFN(x)) with residual connections around both sub-layers. Multi-head attention uses h = 8 heads (base). For each head, projections W^Q_i, W^K_i, W^V_i map to d_k = d_v = d_model/h = 64. Scaled dot-product attention is softmax(QK^T / sqrt(d_k)) V. Heads are concatenated and projected by W^O. The position-wise FFN is max(0, xW_1 + b_1)W_2 + b_2 with inner width 2048 (base).

The decoder mirrors this stack but inserts a third sub-layer: cross-attention over encoder outputs (queries from the decoder, keys and values from the encoder). Decoder self-attention adds a causal mask, setting future logits to −∞ before the softmax, so position i cannot attend to j > i. Training minimises token-level cross-entropy with label smoothing ε_ls = 0.1. Optimisation uses Adam with β1 = 0.9, β2 = 0.98, ε = 10^{-9} and the paper’s learning-rate schedule: linear warmup for 4000 steps then inverse-square-root decay in step number. Regularisation includes residual dropout P_drop = 0.1 (base) and attention dropout.

On WMT 2014 En–De (shared ~37k BPE vocab) the big model reports 28.4 BLEU; on En–Fr (~32k word-piece vocab) 41.8 BLEU. Base En–De training is given as approximately 12 hours on 8× P100. Ablations in the paper vary head count, depth, and positional encoding type; a learned positional embedding performs nearly identically to sinusoids on their translation tasks. Complexity tables contrast per-layer path lengths: self-attention O(1) versus recurrent O(n) and convolutional O(log_k n) for kernel width k.

Limitations explicit in the design include O(n² d) attention cost in sequence length, the need for an external positional signal, and auto-regressive decoding that remains sequential at inference even though training parallelises across positions.