II · THE IDEA · ARTIFICIAL INTELLIGENCE
The Transformer Block, Assembled
▶ Listen · narrated
The architecture that changed machine learning in 2017 is simpler than it first appears: one block, refined until it worked, then repeated until it scaled.
At a glance
- Core components
- Multi-head attention, feedforward network, layer normalisations, residual connections
- Repetition
- The same block structure stacked N times; original paper used 6, modern models use dozens
- Parameter growth
- Each block has its own learned weights; deeper stack means more parameters
- Information flow
- Each token's representation passes through every block in sequence, refined at each step
Imagine a factory assembly line where each station does one job, then passes the product to the next station, which does a different job. A transformer block is like one station that always does two tasks: first, check what is happening at every other position on the line and adjust your work based on that context; second, apply a learned transformation to your own work independently. Then pass everything forward. The whole model is this two-task station, repeated dozens of times. Each repetition refines the work a bit more. The trick that makes it trainable is that at every station, you also keep a copy of what came in and add it back to what comes out, so if a station is not helping, the original signal can bypass it. That bypass is the residual connection, and without it, deep chains of stations would lose the signal.
A transformer block is a sequence of two sublayers, each wrapped in a residual connection and followed by layer normalisation. The first sublayer is multi-head self-attention: the input sequence is projected into queries, keys and values, attention scores are computed as softmax over scaled dot products of queries and keys, and the output is a weighted sum of values. Multiple heads perform this in parallel with different learned projections, and their outputs are concatenated and projected again. The second sublayer is a position-wise feedforward network, typically two linear layers with a nonlinearity (ReLU or GELU) between them, with an intermediate dimension several times larger than the model dimension. Both sublayers use residual addition: the input is added to the sublayer output before normalisation. The normalisation is either post-sublayer (LayerNorm after addition) or pre-sublayer (LayerNorm before the operation, then add the unnormalised residual), with pre-norm now more common in very deep models. The entire block is a pure function from sequence to sequence, preserving shape. Parameters are not shared across blocks, so each block has its own attention and feedforward weights. Stacking N blocks gives O(N) depth in parameter count and O(N) serial dependency in computation per token during autoregressive generation, though attention within each block still has O(L²) cost over sequence length L.
Look closer
The residual connection bypasses the operation it wraps
Before the attention sublayer transforms the input, the input is copied. After attention finishes, that original copy is added back to the result. The same pattern repeats around the feedforward sublayer. This means the block is not learning a transformation directly — it is learning a delta, a correction to apply on top of what was already there. If a particular layer contributes nothing useful, gradient descent can push its output toward zero and the residual path carries the signal through unharmed. Without this, deep networks struggle to train.
Layer normalisation happens twice per block, in a specific order
Each sublayer output is normalised before being added to the residual stream. The original paper applied normalisation after the addition; many implementations since have moved it before the sublayer instead, a variant called pre-norm that appears to train more stably in very deep networks. Both orders work, but they are not equivalent — the location of the normalisation changes what gradients see during backpropagation, and therefore what the model learns. The fact that this detail matters is a reminder that architecture choices interact in non-obvious ways.
The feedforward network operates on each position independently
While attention mixes information across tokens, the feedforward sublayer does not. It applies the same two-layer network to every token position separately, in parallel. The first layer expands the representation to a higher dimension, typically four times wider than the model dimension, then applies a nonlinearity. The second layer projects back down. This is where much of the parameter count lives, and where the model appears to store factual associations, though the mechanisms are still being investigated.
The story
A transformer block is a short, rigid recipe. It takes a sequence of vectors — one per token — and returns a sequence of the same shape, modified. The recipe has four steps, always in the same order.
First, multi-head attention. Each token looks at every other token in the sequence, decides how much to attend to each one, and updates its own representation by taking a weighted mix of what it saw. The weighting is learned, and different attention heads learn different weighting patterns. The output is a new set of vectors, one per token, each now informed by the context around it.
Second, add and normalise. The input to the attention sublayer is added back to its output — the residual connection — and the result is normalised so that each vector has controlled mean and variance. This stabilises training and allows gradients to flow backward through many layers without vanishing.
Third, a feedforward network. Each token's vector is passed through a small neural network independently of the others. The network has two layers with a nonlinearity between them, and it expands to a higher dimension in the middle before projecting back down. This is where much of the block's parameter count lives.
Fourth, add and normalise again. The feedforward output gets its own residual connection and normalisation, using the same pattern as the attention sublayer.
That is one block. The output is a sequence of vectors, the same length as the input, each one refined by attention and then by the feedforward transformation. Those vectors become the input to the next block, which applies the same four steps with its own learned parameters. The process repeats.
The original transformer encoder in Attention Is All You Need stacked six blocks. The decoder also had six, with an extra attention sublayer in each that looked at the encoder's output. Modern language models are decoder-only and much deeper — dozens of blocks, sometimes over a hundred — but the block structure is recognisably the same.
The architecture's power comes partly from this repetition. Each block refines the representation a little further. Early blocks might learn syntax and word relationships; later blocks might learn more abstract patterns or task-specific behaviour. The residual connections mean that information from the input can flow directly to any layer, and gradients from the loss can flow directly back, which makes the deep stack trainable.
Nothing in the block design is obvious from first principles. The order matters: attention before feedforward, normalisation in a particular place, residuals around both sublayers. These choices were tested empirically, and they worked. Variations exist — different normalisation schemes, different activation functions, different dimensions for the feedforward expansion — but the four-step structure has proven robust enough that most transformer models since 2017 are variations on this theme rather than departures from it.
Why it mattered then
The transformer block solved a problem that had limited neural network depth for years: how to train a very deep model without gradients vanishing or exploding. Residual connections, introduced in computer vision in 2015, provided one answer, and the transformer applied them systematically. The 2017 paper demonstrated that this block design, stacked only six layers deep, could match or exceed the performance of recurrent models that had dominated sequence tasks for a decade, while training much faster because every position could be processed in parallel. The architecture was not the first to use attention, but it was the first to show that attention plus feedforward, repeated in a stable stack, was sufficient — that recurrence was not necessary. That result opened the door to scaling, because parallel operations across positions meant that adding more compute improved speed rather than just accuracy.
Why it matters now
The transformer block is now the default unit of scale in machine learning. When a model is described as having 70 billion parameters, most of those parameters live in the feedforward sublayers of dozens of repeated blocks. When a laboratory announces a new model, the architecture description is often just the number of blocks, the model dimension, the number of attention heads, and the feedforward expansion ratio — because the block structure itself is assumed. Understanding the block means understanding where the parameters are, why depth matters, and what changes when a model goes from 12 layers to 96. It also explains certain limitations: the quadratic cost of attention is paid once per block, so a 48-layer model does 48 separate attention operations over the sequence, each one scanning the entire context. The block is not just a historical artifact; it is the recurring cost in every forward pass of every modern large language model.
The surprising detail
The feedforward sublayer, which seems like the simpler of the two operations, is where most of the parameters live. In a typical configuration, the feedforward network has roughly twice as many parameters as the attention mechanism, because it expands to a much higher intermediate dimension. Recent work has found evidence that these feedforward layers store factual knowledge in a way that can be located and even edited: specific facts appear to be encoded in specific parameter subsets, and changing those parameters can change what the model believes without retraining. The attention sublayer, which gets more theoretical attention, may be doing something closer to retrieval and routing, while the feedforward network does storage. The architecture does not make this division explicit — both sublayers are just learned transformations — but the functional split has emerged from training.
Remember this
A transformer is one block, repeated. Attention mixes information across tokens; feedforward transforms each token independently; residuals and norms keep the stack stable.
Test yourself
You double the number of transformer blocks in a model while keeping the model dimension and all other hyperparameters the same. Name two distinct ways this affects inference cost, beyond the obvious doubling of FLOPs.
First, memory usage increases, because each block's activations must be stored during the forward pass if you want to compute gradients later, and even at inference you may need intermediate states depending on the implementation. Second, latency increases more than throughput does: if you are generating one token at a time, each token must pass through every block in sequence, so twice as many blocks means roughly twice the wall-clock time per token, even if you have enough parallelism to keep the hardware busy. Throughput for large batches may scale better because you can still process many tokens in parallel, but per-token latency is inherently serial through the stack. A third effect worth noting: the model's effective context window in terms of what it can use may not double just because you doubled depth — but its memory footprint for storing attention caches across all those blocks does grow, which can force smaller batch sizes.
Go deeper
- Attention Is All You Need · arXiv · Ashish Vaswani et al. · 2017-06-12
- The Annotated Transformer · nlp.seas.harvard.edu
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.