Optimization Theory for Adaptive Gradient Methods

Adaptive gradient methods scale each coordinate by a running estimate of gradient magnitude. Adam does this with two exponential moving averages: one for the gradient itself and one for its square. The second average supplies a per-coordinate normaliser. The practical appeal is obvious — less hand-tuning of a global learning rate, faster progress on sparse or ill-scaled features — and for several years the method was treated as essentially settled.

The convergence paper shows that the settlement was premature. The original regret analysis for Adam rested on an assumption that does not always hold: that the second-moment terms behave so that a certain telescoping sum stays controlled. When that assumption fails, the proof does not go through. More sharply, there exist simple convex online problems on which Adam's average regret stays bounded away from zero. The algorithm can keep taking steps that are too large on the wrong coordinates because the exponential average is allowed to forget earlier large gradients. Forgetting is useful for non-stationary signals; it is harmful when the theory needs the effective step sizes to diminish.

AMSGrad is the direct response. Instead of using the current second-moment estimate alone, one keeps a running maximum of those estimates and divides by the square root of that maximum (plus the usual epsilon). The adaptive scale on each coordinate can only tighten or stay put; it cannot loosen again. With that monotonicity restored, a standard style of regret bound becomes available again. The modification is easy to state and easy to implement, which is part of why the result travelled quickly: it named a real failure mode and offered a one-line change rather than a wholesale redesign.

Hyperparameter sensitivity sits next to the convergence gap. The moments use decay coefficients usually written β1 and β2. End-of-proof constants and the severity of the counter-examples depend on how close those coefficients sit to one, and on the interplay with the learning-rate schedule. A setting that looks stable on one problem can violate the conditions that keep the effective steps under control on another. The theory does not ban Adam; it says that its guarantees are narrower than the original claims, and that β2 in particular is not a free knob.

A second line of work connects adaptive methods to learning-rate schedules that grow or decay exponentially. Under weight normalisation and related reparameterisations, an exponentially increasing learning rate can be shown to interact cleanly with the scale of the weights, and can be rearranged into forms that resemble what practitioners already do with decay. The point for adaptive methods is comparative: once the per-coordinate scale is itself a moving estimate, the global schedule and the adaptive denominator are no longer independent dials. An exponential schedule that is safe for plain SGD can push Adam into the regime where the second-moment estimate and the step-size interact badly — or, with care, can be tuned so that the combination stays stable. The schedule paper supplies a precise setting in which those interactions can be analysed rather than only observed.

Taken together, the results shift the standing of Adam-like algorithms from “default with proof” to “default with known failure modes and known patches.” Stable training still depends on the learning rate, on β1 and β2, on epsilon, and on whether the second-moment estimate is allowed to decrease. The theory does not prescribe a single recipe for every architecture. It does say which pieces of the update are load-bearing when something diverges, and which small changes restore the arguments that justify using the method at all.