II · THE IDEA · ARTIFICIAL INTELLIGENCE
Neural Tangent Kernel Analysis
▶ Listen · narrated
Training a deep network looks hopelessly nonlinear. Yet when every layer is made arbitrarily wide, the path of learning straightens into something that can be written down in closed form.
At a glance
- Core object
- The neural tangent kernel (NTK), fixed by architecture and initialisation
- Key limit
- Infinite width at every layer
- Dynamics
- Gradient descent becomes linear in function space
- Consequence
- Training resembles kernel regression with a frozen kernel
- Scope
- Wide networks of any finite depth, under gradient descent
Think of a complex machine with millions of adjustable knobs. At the start, you note how a tiny twist of each knob changes the machine’s answers on your training examples. That table of sensitivities is like a fixed recipe for learning.
If the machine is enormously “wide” — vastly many units in every layer — those sensitivities barely change while you train. Following the recipe is then enough: each step reduces errors the way ordinary kernel methods do, and you can predict the path from the starting recipe alone.
The machine can still give nonlinear answers to new inputs. What has simplified is not the answers themselves but the way training moves them. Infinite width freezes the recipe; gradient descent becomes linear in that frozen description. For ordinary, narrower machines the recipe keeps shifting, and the simple picture is only an approximation.
Consider a network f(θ, x) with parameters θ drawn from a standard random initialisation. Gradient flow on a loss L yields ∂t f(θ(t), ·) = −∇θ f(θ(t), ·) ∇θ f(θ(t), ·)⊤ ∇f L, i.e. motion in function space driven by the neural tangent kernel Θ(x, x′) = ⟨∂θ f(θ, x), ∂θ f(θ, x′)⟩.
At finite width Θ evolves with θ. In the infinite-width limit, under the parameterisations studied in the NTK literature, Θ concentrates on a deterministic kernel Θ∞ fixed by the architecture and the initialisation distribution, and ∂t Θ → 0 during training. The dynamics on the training set close as a linear ODE: ∂t f_t = −Θ∞ ∇f L(f_t). For least-squares this is kernel gradient descent and admits an explicit exponential solution in the eigenbasis of Θ∞.
Jacot, Gabriel and Hongler define this kernel and analyse convergence and generalisation in the resulting regime. Lee et al. show that wide networks of any finite depth evolve as their linearisations under gradient descent: the nonlinear network and the model f_lin(θ) = f(θ0) + ∇θ f(θ0)·(θ−θ0) remain close for longer as width → ∞.
Limitations: the result is a limit statement; finite width reintroduces kernel drift and feature learning. Standard NTK parameterisation is a “lazy” regime in which relative parameter movement vanishes with width. Other parameterisations (e.g. mean-field / μP) can retain feature learning even at large width. The NTK describes training dynamics in function space; it does not claim the predictor is linear in x.
Look closer
The tangent, not the network
At initialisation a network defines a map from parameters to functions. The NTK is the Gram matrix of that map's Jacobian: how much the output at one input moves when parameters are nudged to fit another. In the infinite-width limit this kernel stops changing during training. The network may still look deeply nonlinear as a circuit, but the trajectory of its predictions is the trajectory of a linear model in that fixed tangent space.
Width freezes the kernel
For finite width the kernel evolves as weights move, so the linear picture is only an approximation. As width grows, relative parameter changes shrink and the kernel concentrates around its expectation. At infinite width the evolution vanishes: the NTK at step zero is the NTK at every later step. That is the hinge on which predictability turns.
Depth is allowed; width is not optional
The linearised description is not restricted to shallow models. The second source establishes that wide networks of any depth still evolve as linear models under gradient descent in this limit. Depth shapes which kernel you get; width is what makes the kernel stay put long enough for the linear analysis to close.
The story
A neural network is usually treated as a nonlinear object trained by a nonlinear procedure. Change a weight and the effect on the output depends on every other weight; take a gradient step and the landscape itself shifts. That picture is accurate for the networks people actually run. It is also, in a precise limiting regime, replaceable by something far simpler.
The neural tangent kernel analysis begins from a change of viewpoint. Instead of watching parameters, watch the function the network computes. Gradient descent on a loss induces a velocity field in function space. The operator that converts a residual at the training points into that velocity is built from the Jacobian of the network with respect to its parameters. The inner product of Jacobian rows at two inputs is an entry of the neural tangent kernel. Once that kernel is known, the continuous-time training dynamics on the training set become an ordinary linear differential equation.
For a finite network the kernel is itself a moving target: as parameters drift, the Jacobian changes, and so does the operator driving learning. The central observation of the NTK programme is that this motion can be suppressed by width. When every layer is taken to infinite width, under standard parameterisations and random initialisation, the kernel concentrates and ceases to evolve. Training then proceeds exactly as kernel gradient descent with a kernel fixed at initialisation. In that sense the infinite-width network, for the duration of gradient descent, behaves as a linear model in its tangent space — linear in the parameters around their starting point, even though the architecture that produced the tangent was nonlinear.
Two papers frame the result. Jacot, Gabriel and Hongler introduced the neural tangent kernel and showed convergence and generalisation behaviour for neural networks in this kernel regime. Lee and co-authors showed that wide neural networks of any depth evolve as linear models under gradient descent: the linearised dynamics track the full nonlinear network increasingly tightly as width grows. Together they turn an apparently intractable optimisation path into something that can, in the limit, be written in closed form from the architecture and the initial draw of weights.
The practical reading is careful rather than triumphant. Infinite width is not a switch one flips on a laptop. Finite networks retain a moving kernel; the NTK description is an idealisation that becomes sharper with width, not a certificate for every trained model. What the analysis supplies is a regime in which training is predictable, a language for asking when a real network is still near that regime, and a reminder that “nonlinear architecture” and “nonlinear training dynamics” are not the same claim.
Why it mattered then
Before this line of work, the success of gradient descent on deep networks was largely an empirical fact without a matching dynamical theory. Mean-field and kernel perspectives existed in pieces, but the NTK gave a concrete object — a kernel determined by architecture and initialisation — through which infinite-width training could be analysed as ordinary kernel regression. That mattered because it separated two confounded ideas: the expressivity of a deep circuit and the path that optimisation actually takes. In the limit the path is the path of a linear model in tangent space, which made questions about convergence rates and generalisation newly tractable with classical tools.
Why it matters now
Most networks in use are finite and often far from the NTK regime, where feature learning and kernel evolution are essential rather than negligible. The analysis still matters as a baseline. It tells you what training looks like when representation learning is switched off by width, and therefore what must be happening when a finite network beats its own tangent kernel. Debates about lazy training, about when fine-tuning moves features, and about how much of a model’s behaviour is fixed at initialisation all sit downstream of that distinction. The infinite-width linear picture remains the clean limiting case against which those richer regimes are measured.
The surprising detail
The network does not become a linear function of its inputs. It becomes linear in its parameters, in a neighbourhood of initialisation, as a map into function space. The architecture can still compute highly nonlinear input–output maps; what freezes is the training trajectory’s operator. People often hear “the network becomes linear” and picture a linear classifier. The claim is narrower and stranger: the geometry of learning flattens while the circuit’s input nonlinearity remains.
What is disputed
The exact rate at which finite-width networks approach NTK dynamics depends on architecture, parameterisation and depth; the sources establish the infinite-width limit and linearised evolution, not a universal finite-width error bound for every practical model. Whether a given trained network is “in the NTK regime” is an empirical question the pure limit does not settle.
Remember this
In the infinite-width limit the NTK freezes at initialisation, and gradient descent trains a linear model in that fixed tangent space.
Test yourself
A colleague says infinite-width NTK theory shows that deep networks are merely linear classifiers. What exactly has been misunderstood, and what does the theory actually linearise?
The theory does not say the network’s input–output map is linear. It says that, at infinite width, training dynamics are linear in the parameters around initialisation: the model evolves as kernel gradient descent with a frozen neural tangent kernel. The architecture may still implement a highly nonlinear function of the inputs; what becomes linear and predictable is the path of learning in function space, not the form of the predictor applied to raw features.
Go deeper
- [1806.07572] Neural Tangent Kernel: Convergence and Generalization in Neural Networks · arxiv.org
- [1902.06720] Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.