II · THE IDEA · ARTIFICIAL INTELLIGENCE

Gradient Descent

Foundations · Iterative optimisation · Billions of parameters

▶ Listen · narrated

Every competent answer from a model traces back to one repeating step: measure how wrong the output was, nudge billions of numbers downhill. This is gradient descent in unvisualizable space.

At a glance

What it does: Adjusts parameters to reduce a loss function, one small step at a time
The gradient: A vector of partial derivatives showing which direction is downhill for each parameter
Learning rate: A multiplier controlling step size, typically very small (0.001 or less)
Variants: Adam, SGD, RMSprop — differing in how they choose step size and direction

Imagine you are blindfolded on a hillside and told to reach the lowest point. You feel the ground under your feet to determine which direction slopes downward, then take a small step that way. You repeat: feel the slope, step downhill. Eventually you reach a valley floor. Gradient descent works the same way, except the hill has billions of dimensions — one for each parameter in the model — and the algorithm measures the slope by calculating how the error changes when each parameter shifts infinitesimally. It steps opposite the gradient, moving all the parameters at once in whatever direction reduces the loss. Over millions of steps, the model improves.

Look closer

The landscape metaphor breaks immediately
We say the algorithm walks downhill on an error surface, and diagrams show a ball rolling into a valley. This is a fiction. A model with ten billion parameters has a loss function defined over a ten-billion-dimensional space. There is no hill, no valley, no surface you could stand on. The gradient is not a slope you could measure with a spirit level; it is a vector with ten billion components, each one the rate at which loss changes if you nudge that one parameter and hold all the others still. The three-dimensional sketch is a teaching convenience, not a scaled-down version of the real thing.
Stochastic means it uses a sample, not the whole dataset
Computing the true gradient would require running the model on every training example, summing the errors, then backpropagating through all of them. For a dataset with billions of tokens, this is prohibitively slow. Stochastic gradient descent picks a small batch — perhaps 32 or 512 examples — calculates the gradient from that sample alone, and takes a step. The gradient is noisy, an estimate rather than the true direction, but it is fast to compute and the noise averages out over many steps. Most modern training uses this stochastic variant, not the original batch method.
Adam adapts the learning rate for each parameter individually
Plain gradient descent uses one learning rate for every parameter. Adam, now the dominant variant, maintains a moving average of past gradients and past squared gradients for each parameter separately, then scales the step size accordingly. Parameters that have been changing rapidly get smaller steps; parameters with consistent gradients get larger ones. This per-parameter adaptation often converges faster and more reliably than a single global rate, which is why Adam has become the default in much of the literature since its publication in 2014.

The story

Training begins with parameters set to small random values. The model is useless. You feed it a batch of examples, record how far its predictions miss the target, and compute a single number called the loss. High loss means poor performance.

The question is: which way should you adjust each parameter to make the loss smaller? Trying random changes would take longer than the age of the universe. Instead, you calculate the gradient — the vector of partial derivatives of the loss with respect to every parameter. Each component tells you whether increasing that parameter would increase or decrease the loss, and by how much, assuming an infinitesimal step and holding everything else constant.

You then move each parameter a small distance in the direction opposite its gradient component. This is one update step. The loss usually drops, though not always; the gradient is a local measurement and the function is not a smooth bowl. You repeat this thousands or millions of times, each iteration using a fresh batch of training data.

The learning rate controls how far you step. Too large and you overshoot, bouncing around or even diverging to infinity. Too small and training creeps forward, requiring impractical amounts of time and compute. Choosing it well is part craft, part search. Many practitioners start with a standard value like 0.001 and adjust based on whether the loss curve is stable.

The stochastic variant, which uses small batches rather than the full dataset per step, introduces noise but also a useful kind of exploration. The path wanders more than the true gradient would allow, sometimes escaping shallow local minima that might trap a deterministic walk. This noisiness is generally considered beneficial, though the theory behind why it helps remains an active research area.

Adam and similar adaptive methods add another layer: they track how each parameter has been behaving over recent steps and adjust its effective learning rate accordingly. A parameter whose gradient keeps pointing the same direction gets a confidence boost and takes larger steps. One whose gradient sign keeps flipping gets reined in. This per-parameter tuning reduces the need for manual learning rate schedules and often accelerates convergence, which is why Adam has become a default choice despite being more complex than plain stochastic gradient descent.

The process ends when the loss stops improving, or when you run out of patience or budget. The final set of parameters is your trained model. Everything it knows is encoded in the particular location in parameter space that gradient descent managed to reach.

Why it mattered then

Gradient descent itself dates to the nineteenth century, but applying it to train neural networks required waiting for backpropagation, which made gradient calculation feasible, and for computers fast enough to iterate millions of times. By the late 1980s the method was established, but training remained slow and results were mixed. The stochastic variant, using small batches, was formalised in the 1950s but became standard practice in neural network training only as datasets grew too large for batch methods. Adam was published in 2014, at a moment when deep learning was already producing striking results but training remained finicky and hyperparameter-sensitive. Its adaptive per-parameter learning rates made training more robust and less dependent on manual tuning, which mattered as models grew and the number of researchers without years of optimisation experience expanded rapidly. The method's rapid adoption reflected a field that was scaling faster than its collective expertise.

Why it matters now

Gradient descent and its variants remain the only practical way to train models at the scale now routine. Every frontier model — every system with hundreds of billions of parameters — has been trained by iterating this same basic step millions of times. The cost of training is dominated by the cost of computing gradients and applying updates, which is why so much engineering effort goes into making those operations faster. The choice of optimiser, learning rate schedule, and batch size are still among the most consequential decisions in a training run, capable of changing whether a model converges at all. Adam remains the most common choice, though variants and competitors continue to appear. The method's simplicity is deceptive; it works, but we still lack a complete theory of why it works as well as it does in high-dimensional non-convex spaces, or why certain learning rates and schedules succeed where others fail. It is the foundation of the field, empirically proven and theoretically still partly mysterious.

The surprising detail

The loss landscape of a large neural network is not convex — it is full of saddle points, local minima, and flat regions where the gradient is nearly zero. Classical optimisation theory offers few guarantees in such spaces, and yet gradient descent reliably finds parameters that work. Why it succeeds remains incompletely understood. One hypothesis is that in very high dimensions, local minima are rare and most critical points are saddle points, which stochastic gradient descent can escape due to noise. Another observation is that many different parameter settings achieve similarly low loss, so the algorithm does not need to find one global minimum, just any point in a large basin of good solutions. The theory lags behind the practice, and much of what we know about training at scale comes from empirical trial rather than proof.

Remember this

Gradient descent adjusts billions of parameters by stepping opposite the direction that increases error. The landscape metaphor is a cartoon; the real space has as many dimensions as the model has parameters.

Test yourself

A training run is unstable: the loss drops for a while, then suddenly spikes to infinity. You halve the learning rate and try again. This time it trains smoothly. Explain in terms of the gradient what likely happened in the first run, and why the smaller learning rate fixed it.

Go deeper

Image: Original diagram, The Daily Triptych. Licence: Original work. Source.

← Back to day 5