The Neural Tangent Kernel

A neural network learns by adjusting its parameters — the millions of internal numbers that determine what answer it gives — a little at a time, until its answers on the training examples come out right. Gradient descent, the standard method, works by repeated small corrections: measure how wrong the current answer is, work out which direction each parameter should move to shrink the error, and take a small step that way. What all those steps add up to is usually hard to say in advance. It depends on the data, on the random numbers the parameters started from, and on the order in which the examples arrive. The 2018 paper that introduced the neural tangent kernel found one circumstance in which the whole process becomes predictable.

Start with the object itself. Pick an input and ask, for each parameter in turn: if this parameter were nudged slightly, how much would the network's output on this input shift? Each answer is a single number, called a gradient. Collect one for every parameter and you have a long list attached to that input — one entry per parameter, millions of entries in a large network.

Now take two inputs and their two lists. Multiply the lists entry by entry and add up all the products. The single number that comes out is the neural tangent kernel for that pair of inputs, and it answers a concrete question about side effects. When gradient descent adjusts the parameters to fix the network's answer on the first input, the same parameters serve every other input too, so the answer on the second input moves as well. The kernel value says how far. A large value means fixing one answer drags the other a long way. A value near zero means the two inputs are handled almost independently.

That number exists for any network of any size. The trouble is that it normally will not sit still. The gradients depend on where the parameters currently are; the parameters move at every step of training; so the kernel keeps turning into something different as training proceeds. The side effects that governed the early steps are not the side effects that govern the later ones, and the whole process resists analysis.

The 2018 result identifies the escape. Make every layer wider — more units side by side, more parameters — and let the width grow without limit. In that limit the kernel converges to one fixed, deterministic function. It no longer depends on which random values the parameters happened to start from. And it stops drifting during training: the same table of side effects holds from the first step to the last.

Once the kernel is fixed, training lands in territory statisticians had already mapped. Kernel methods are a family of techniques, older than deep learning, in which you never work with the inputs directly — only with a similarity score between pairs of them — and a prediction on a new input is a weighted combination of the training labels, weighted by how similar the new input is to each training example. Gradient descent on a very wide network, in this account, performs something very close to that kind of regression, with the neural tangent kernel supplying the similarity score. Questions about convergence, meaning whether training reaches a good fit, and generalisation, meaning what the network does on inputs it never saw, can then be asked in a language that already had answers.

The 2019 paper turned the limit into something checkable at real, finite width. Its route is linearisation. Suppose the parameters end up travelling only a short distance from their random starting point. Then the network's output is well approximated by a simple rule: take the output at the start, and for each parameter add its displacement multiplied by its gradient at the start. That rule is linear in the parameters — double every displacement and the correction doubles — though still nonlinear in the input. And a linear model trained by gradient descent under squared loss, where the error is the squared gap between prediction and target, is a solved problem. Taking gradient flow, which is gradient descent with infinitesimally small steps, the paper writes down a closed-form expression for the model's predictions at every moment of training, including the end.

The striking part is the check. Train a real wide network. Separately, evaluate the formula, which requires no training at all. Plot both prediction curves against training time. The paper reports that the two curves track each other closely — for fully connected networks of any depth, and for convolutional networks — provided two conditions hold. The network must be wide enough. And the learning rate, the size of each training step, must stay below a critical threshold the paper identifies. Push past that threshold and the match breaks, for a reason the framework itself explains: large steps carry the parameters far from their starting point, and the linear approximation was built around that starting point.

The honest reading is therefore narrow. The neural tangent kernel does not explain deep learning as practised. It explains one regime — wide layers, small steps, parameters staying near their random initial values — in which deep learning becomes mathematically transparent, and it states exactly what that transparency costs. In this regime there is almost no feature learning. A deep network is usually valued for discovering new internal representations of its data; here, a fixed kernel means a fixed set of features, chosen by chance at initialisation, and training merely reweights them. Whether the successes of practical networks — trained with large learning rates, for a long time, with parameters wandering far from where they began — are mostly kernel-like or mostly not, neither paper settles.