Residual Connections and Normalisation

In 2015, researchers at Microsoft trained a network with over a hundred layers and won the ImageNet competition. The achievement was not the depth for its own sake — it was that the network trained at all. A year earlier, the same team had found that adding more layers made performance worse, not better. The problem was not overfitting. The problem was that the network could not learn in the first place.

The gradients were vanishing. During backpropagation, the learning signal has to flow backward through every layer to update the early weights. Each layer multiplies the gradient by its own derivatives, and if those derivatives are small — which they often are — the gradient shrinks exponentially with depth. By the time the signal reaches the first few layers, it has decayed to nothing. The early layers stop learning, and the network as a whole gets stuck.

The solution was the residual connection. Instead of computing y = f(x), the layer computes y = f(x) + x. The input is added back to the output. This seems trivial, but it changes the gradient behaviour completely. When the gradient flows backward through the addition, it splits: one copy goes through the layer's computation, where it may shrink, and one copy passes straight through unchanged. That second path — the identity shortcut — means the gradient can always reach the early layers, no matter how deep the network is.

The improvement was immediate and dramatic. Networks with fifty layers trained faster and better than shallower networks without residual connections. Networks with over a hundred layers became practical. The technique transferred to every domain: vision, speech, language. By 2017, the transformer architecture had absorbed the same idea, and every attention block and feedforward block in every modern language model uses residual connections.

Normalisation was the other half of the solution. Even with residual connections, activations can drift to extreme values during training, and extreme values make gradients unstable. Layer normalisation rescales the activations at each layer to have stable statistics — typically zero mean and unit variance, though Root Mean Square normalisation drops the mean-centring and keeps only the variance rescaling. The effect is that every layer receives inputs in a predictable range, no matter what the previous layers did.

Transformers use pre-norm: the input is normalised before it enters the attention or feedforward computation, not after. This makes training more stable because the normalisation does not depend on the layer's weights, which are changing rapidly early in training. The residual connection then adds the layer's output to the original, unnormalised input. The combination — normalise, compute, add back — appears in every transformer block, twice per layer, and it is the reason transformers can train at all at the scales we now use.

The residual stream, as it is sometimes called, is the vector that flows through the network, accumulating contributions from every layer. It is not rewritten by each layer; it is edited. Early layers add token embeddings and positional information. Middle layers add syntactic structure and semantic relationships. Late layers add task-specific adjustments. The final output is the sum of all these edits, and because the additions are linear, you can in principle isolate what each layer contributed by subtracting its output from the stream.