II · THE IDEA · ARTIFICIAL INTELLIGENCE
Backpropagation
▶ Listen · narrated
Training a network means changing its weights to reduce error. The question is which weights to change, and by how much. Backpropagation answers both at once.
At a glance
- What it is
- Automatic differentiation applied to neural networks, computing gradients for every parameter
- Core mechanism
- Chain rule of calculus, evaluated recursively from output back to input
- Enables
- Gradient descent in networks with arbitrary depth and millions of parameters
- Computational cost
- Roughly twice the cost of a forward pass through the same network
Imagine you are adjusting the temperature on a shower. Turn it too far and the water is scalding; not far enough and it is cold. You want to know how much turning the dial affects the temperature. Now imagine the dial controls a valve, the valve controls a pipe, the pipe controls another valve, and that valve finally controls the temperature. To know how much to turn the dial, you need to know how each piece in the chain responds to the one before it. Backpropagation does this for neural networks. The network is a chain of calculations, and backpropagation figures out how much each weight in the chain affects the final error. It works backward, starting at the error and moving back through each layer, multiplying together the sensitivities at each step. By the time it reaches the first layer, it knows exactly how much each weight contributed to the error, which tells it how much to adjust each one.
Backpropagation is reverse-mode automatic differentiation applied to the computational graph of a neural network. During the forward pass, the network computes ŷ = fₙ(fₙ₋₁(...f₁(x, θ₁)..., θₙ₋₁), θₙ), where each fᵢ is a layer parameterised by θᵢ. The forward pass stores all intermediate activations aᵢ. During the backward pass, the algorithm computes ∂L/∂θᵢ for every parameter by applying the chain rule recursively. It begins with ∂L/∂ŷ, computed from the loss function. For each layer i in reverse order, it computes ∂L/∂θᵢ = ∂L/∂aᵢ · ∂aᵢ/∂θᵢ and ∂L/∂aᵢ₋₁ = ∂L/∂aᵢ · ∂aᵢ/∂aᵢ₋₁, where ∂aᵢ/∂θᵢ and ∂aᵢ/∂aᵢ₋₁ are the local Jacobians of layer i. The gradient ∂L/∂aᵢ₋₁ becomes the input to the backward pass of layer i−1. Each layer's backward pass is essentially a vector-Jacobian product, which is computationally efficient because the Jacobian is never materialised explicitly. The algorithm requires O(n) memory to store activations and O(n) time for the backward pass, where n is the number of operations in the forward pass. Frameworks implement this by constructing a dynamic or static computation graph during the forward pass, then traversing it in reverse topological order during the backward pass, accumulating gradients at each parameter node.
Look closer
The forward pass builds a graph of dependencies
When you feed data through a network, each layer's output depends on the previous layer's output and on that layer's weights. The network is really a long composition of functions: output = f₃(f₂(f₁(input, w₁), w₂), w₃). The forward pass computes the final value and stores the intermediate results. Those stored values are what make the backward pass efficient, because they let you reuse computation rather than recalculating everything from scratch. Without them, backpropagation would be intractably slow.
The backward pass applies the chain rule one layer at a time
Suppose the loss is L and a weight deep in the network is w. You want ∂L/∂w, the rate at which loss changes with respect to that weight. The chain rule says you can compute this by multiplying together the derivatives along the path from L back to w. Backpropagation starts at the output, computes ∂L/∂(output), then moves one layer backward and computes ∂L/∂(hidden layer) by multiplying the gradient from above by the local derivative of the layer's operation. It repeats this, layer by layer, until every weight has a gradient. Each layer receives a gradient from the layer above and passes a gradient to the layer below.
A two-layer example in concrete numbers
Input x = 2. First layer: h = x · w₁ = 2 · 3 = 6. Second layer: y = h · w₂ = 6 · 0.5 = 3. True target is 5, so loss L = (y − 5)² = 4. Forward pass done. Now backward: ∂L/∂y = 2(y − 5) = −4. For w₂: ∂L/∂w₂ = ∂L/∂y · ∂y/∂w₂ = −4 · h = −4 · 6 = −24. For the hidden layer: ∂L/∂h = ∂L/∂y · ∂y/∂h = −4 · w₂ = −4 · 0.5 = −2. For w₁: ∂L/∂w₁ = ∂L/∂h · ∂h/∂w₁ = −2 · x = −2 · 2 = −4. Both weights now have gradients, computed in two steps backward. Scale this reasoning to a hundred layers and a billion weights and you have modern backpropagation.
The story
Neural networks learn by adjusting weights to reduce the difference between their output and the correct answer. The difficulty is that a network with even modest depth has thousands or millions of weights, each influencing the output through a long chain of intermediate calculations. Backpropagation solves this by computing, for every weight in the network, the derivative of the loss with respect to that weight — a number that tells you both the direction and magnitude of the adjustment needed.
The method rests on the chain rule from calculus. If you have a composition of functions, the derivative of the whole is the product of the derivatives of the parts. A neural network is exactly such a composition: each layer is a function, and the output is the result of applying them in sequence. During the forward pass, you feed data through the network and compute the final output, storing the intermediate values along the way. During the backward pass, you start at the output, compute how the loss changes with respect to that output, then work backward through each layer in turn, computing how the loss changes with respect to that layer's inputs and weights.
Consider a minimal two-layer network. An input value passes through the first layer, producing a hidden value, which then passes through the second layer to produce the final output. Each layer has a weight. The forward pass computes the output and compares it to the target, yielding a loss. The backward pass begins at that loss. You compute the derivative of the loss with respect to the output, then use the chain rule to compute the derivative with respect to the second layer's weight by multiplying by the derivative of the output with respect to that weight. Next you compute the derivative of the loss with respect to the hidden value by multiplying by the derivative of the output with respect to the hidden value. Finally you compute the derivative with respect to the first layer's weight using the gradient you just computed for the hidden value. Two layers, two weights, two applications of the chain rule, and every weight has a gradient.
The efficiency comes from reusing intermediate results. When you compute the gradient for the hidden layer, you are computing something that every weight in the earlier layers will need. You compute it once and pass it backward. Each layer receives a gradient from the layer above, computes its own weight gradients, then passes a new gradient to the layer below. The computation flows backward through the same graph that the data flowed forward through, but now carrying derivatives instead of activations.
The cost is roughly double the forward pass. You must store all the intermediate activations from the forward pass, which is why training uses more memory than inference. You must also compute the local derivative at each operation, which is typically cheap — the derivative of a multiplication or a ReLU is a simple expression. The overall algorithm scales linearly with the number of parameters, which is why networks with billions of weights remain trainable. Without backpropagation, computing gradients by finite differences or symbolic manipulation would be prohibitively expensive for anything beyond toy examples.
Why it mattered then
Backpropagation was described in various forms in the 1960s and 1970s, but it did not become the standard training method for neural networks until the mid-1980s, when Rumelhart, Hinton and Williams published a clear presentation of the algorithm and demonstrated its effectiveness on problems that earlier methods had struggled with. Before backpropagation, training multi-layer networks was difficult. The perceptron learning rule worked only for single-layer networks. Researchers knew in principle that gradient descent could train deeper networks, but computing the gradients by hand or by symbolic differentiation was impractical for anything beyond trivial architectures. Backpropagation made gradient computation automatic and efficient, which meant researchers could experiment with deeper architectures and larger datasets without having to derive new update rules for each network design. It turned neural networks from a theoretical curiosity into a practical tool, though the computational resources of the time still limited the scale of what could be trained.
Why it matters now
Backpropagation remains the training method for essentially all neural networks in production today. Every major deep learning framework — PyTorch, TensorFlow, JAX — implements automatic differentiation using backpropagation as its core mechanism. The algorithm itself has not changed, but the scale has. Networks now have billions of parameters, trained on datasets that would have been unimaginable in the 1980s, and backpropagation scales to handle them. Modern refinements address practical concerns — mixed precision training, gradient checkpointing to reduce memory use, distributed backpropagation across multiple machines — but the underlying algorithm is the same recursive application of the chain rule. Understanding backpropagation is necessary for diagnosing training failures, because many problems — vanishing gradients, exploding gradients, dead neurons — are visible only when you understand how gradients flow backward through the network. It is also the reason why network architecture matters: a poorly designed architecture can impede gradient flow, making the network difficult or impossible to train even when the capacity is theoretically sufficient.
The surprising detail
Backpropagation was independently discovered several times, in different fields, under different names. Researchers in control theory, numerical analysis and machine learning all arrived at the same algorithm without initially realising they were solving the same problem. The version used in neural networks today is most directly descended from the 1986 paper by Rumelhart, Hinton and Williams, but earlier work by Werbos in 1974, Linnainmaa in 1970, and even Kelley in 1960 described mathematically equivalent procedures. The repeated rediscovery suggests the algorithm is in some sense inevitable — it is the natural way to compute gradients in a computational graph — but it also meant that the method took longer to become widely known than it might have if the different research communities had been in closer contact.
Remember this
Backpropagation is the chain rule applied recursively, layer by layer, to compute how every weight in a network affects the final loss.
Test yourself
A network has ten layers. During backpropagation, the gradient of the loss with respect to a weight in the third layer depends on values computed in which other layers, and in what order?
It depends on values from every layer between the third layer and the output — layers four through ten — and they must be computed in reverse order, from the output back toward layer three. The gradient at layer three is the product of the local gradient at layer three and the gradient passed back from layer four, which itself depends on the gradient from layer five, and so on. You cannot compute the gradient for layer three until you have computed the gradient for layer four, which means backpropagation must proceed strictly from output to input. This is why it is called backpropagation: the direction of computation is the reverse of the forward pass.
Go deeper
- mlp.html · deeplearningbook.org
- Automatic differentiation in machine learning: a survey · arXiv · Atilim Gunes Baydin et al. · 2015-02-20
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.