II · THE IDEA · ARTIFICIAL INTELLIGENCE
Neural Tangent Kernel Theory for Wide Networks
▶ Listen · narrated
Train a network that is wide enough and gradient descent stops looking mysterious. The path of the network function is governed by a fixed kernel that never changes.
At a glance
- Core claim
- Infinite-width nets under GD equal kernel regression with a fixed kernel
- The kernel
- Neural Tangent Kernel (NTK), constant during training in the limit
- Dynamics
- Network function follows a linear ODE driven by the NTK
- Computation
- Infinite-width NTK can be computed exactly for standard architectures
Think of the network as a huge panel of fixed random dials. At the start, each dial contributes a little random pattern over the inputs. Training does not redesign the patterns; it only turns the dials by tiny amounts. When there are infinitely many dials, those tiny turns are enough to match any training labels, and the rule for how the output moves is the same at every step — that unchanging rule is the Neural Tangent Kernel.
So gradient descent, which usually looks like wandering through a complicated landscape, becomes something much simpler: ordinary kernel fitting with a kernel you could have written down before training began. The network is powerful because the random patterns were already extremely rich, not because training invented new ones.
Consider a neural network f(θ, x) with parameters θ scaled in the NTK parametrisation. The tangent kernel is Θ(x, x'; θ) = ⟨∇_θ f(θ, x), ∇_θ f(θ, x')⟩. Jacot et al. prove that as hidden widths → ∞, Θ converges in probability to a deterministic kernel Θ_∞ that depends on architecture and depth but not on the random draw of θ, and that under gradient flow Θ remains equal to Θ_∞ for all training time. Consequently ḟ_t = −Θ_∞ · ∇_f L, so for square loss on a finite dataset the training outputs obey an explicit linear ODE whose solution is exponential convergence along the eigenbasis of the empirical NTK matrix. The induced predictor on test points is kernel regression with Θ_∞.
Arora et al. give a dual formulation and a finite recursive algorithm that computes Θ_∞ exactly for fully connected ReLU networks (and extensions to convolutions), enabling direct evaluation of the infinite-width trained predictor without sampling wide nets. Limitations: the result is parametrisation-dependent; mean-field and other scalings allow feature evolution; finite width reintroduces kernel drift of order 1/√width; and the NTK regime often underperforms feature-learning regimes on tasks where representation adaptation matters.
Look closer
The kernel does not move
In a finite network the tangent kernel — the Gram matrix of parameter gradients of the network outputs — changes as weights move. In the infinite-width limit that object converges in probability to a deterministic kernel and, crucially, stays put for the whole of training. Gradient descent therefore never leaves the linearised model defined at initialisation.
Training becomes an ODE
Once the NTK is fixed, the evolution of the network function under continuous-time gradient descent is ordinary kernel gradient descent. The solution is an explicit linear dynamical system: outputs on the training set converge exponentially toward the labels at rates set by the eigenvalues of the kernel matrix.
Exact, not only asymptotic
For fully connected and certain convolutional architectures the infinite-width NTK admits a closed-form recursive computation. That makes the limiting predictor something one can evaluate on real data without ever instantiating an infinitely wide net, and then compare against finite networks of growing width.
The story
A neural network is usually treated as a highly non-linear function of its parameters. Gradient descent on that function looks intractable to analyse: the loss landscape changes shape as the weights move, and every step alters the geometry of the next. The Neural Tangent Kernel result of Jacot, Gabriel and Hongler shows that this picture simplifies dramatically when every hidden layer is taken to infinite width.
At any finite width one can form the tangent kernel: the matrix of inner products, over parameters, of the gradients of the network outputs. This kernel describes the instantaneous linearisation of the model. Under gradient descent the network function moves in the direction that kernel dictates. In ordinary networks the kernel itself evolves, because the gradients change as parameters change. The infinite-width analysis shows two things at once. First, at initialisation the tangent kernel concentrates on a deterministic limit that depends only on the architecture and the input distribution, not on the particular random draw of weights. Second, that limiting kernel remains constant throughout training. Parameter updates become infinitesimal relative to the scale of the network, so the linearisation never has to be refreshed.
The practical consequence is equivalence to kernel gradient descent. Continuous-time gradient flow on the network parameters induces ordinary differential equation dynamics on the function values, driven by the fixed NTK. On a finite training set the solution is explicit: the network outputs decay toward the training labels along the eigenmodes of the kernel matrix, each mode at its own exponential rate. Generalisation outside the training set is then ordinary kernel regression with that same NTK. In this regime a neural network is not discovering new features during training; it is fitting labels inside a fixed reproducing kernel Hilbert space chosen by the architecture and the initialisation.
Arora and colleagues later showed that this limiting object is not merely theoretical. For standard fully connected networks, and for certain convolutional ones, the infinite-width NTK can be computed exactly by a finite recursion. One can therefore run the corresponding kernel regression on real datasets and obtain the precise infinite-width predictor, then watch finite networks approach that predictor as width grows. The theory thus supplies both a characterisation of training dynamics and a concrete computational baseline.
The result is specific to a particular scaling of initialisation and learning rate, sometimes called the NTK parametrisation, in which the network output changes by an amount that stays order one while individual weights barely move. Other scalings can produce different infinite-width limits in which features do evolve. Within the NTK regime, however, the equivalence is sharp: infinite width turns gradient descent on a neural network into kernel gradient descent with a frozen kernel.
Why it mattered then
Before this work, infinite-width limits were mainly associated with Gaussian process behaviour at initialisation, not with what happens during training. The NTK papers closed that gap. They gave a regime in which the entire optimisation trajectory of a neural network could be written down, convergence to zero training loss could be proved under mild conditions on the kernel, and generalisation could be read off from classical kernel theory. That mattered because it turned a heuristic practice — train a very wide net with gradient descent — into something that admitted theorems rather than only experiments.
Why it matters now
Most networks used in practice are wide but finite, and many are trained in regimes where features do move. The NTK limit is therefore not a literal description of every modern model. It remains a reference point: a solvable baseline against which feature learning, finite-width effects and alternative parametrisations can be measured. When a training phenomenon survives in the NTK regime, one knows it does not require representation learning; when it disappears, one knows it does. Exact NTK computation also still serves as a diagnostic tool for architecture design and for checking how closely a given finite network sits to its infinite-width shadow.
The surprising detail
In the infinite-width NTK limit the parameters barely move, yet the network function can still fit arbitrary labels on the training set. Learning happens almost entirely in function space, through the accumulated effect of tiny weight updates filtered by a kernel that was fixed at initialisation. The network does not need to reshape its features to interpolate; the random features it started with were already rich enough, once width is infinite.
What is disputed
The constant-kernel equivalence holds in the infinite-width limit under NTK scaling of initialisation and step size. Finite networks only approximate it, and other parametrisations can yield infinite-width limits in which the kernel evolves and features are learned. How closely any particular practical network tracks NTK behaviour is an empirical question, not a theorem.
Remember this
At infinite width under gradient descent, a neural network becomes kernel regression with a frozen Neural Tangent Kernel whose dynamics admit a closed form.
Test yourself
In the NTK regime the network can fit training labels even though individual weights change only infinitesimally. What, then, is actually doing the learning, and what does that imply about whether the model is discovering new features during training?
Learning occurs in function space: many infinitesimal parameter steps accumulate into a finite change in the network outputs, steered by the fixed NTK. Because the kernel never updates, the model stays inside the linearisation chosen at initialisation and does not discover new features; it reweights the random features it already had. Fitting is kernel regression, not representation learning.
Go deeper
- [1806.07572] Neural Tangent Kernel: Convergence and Generalization in Neural Networks · arxiv.org
- [1904.11955] On Exact Computation with an Infinitely Wide Neural Net · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.