II · THE IDEA · ARTIFICIAL INTELLIGENCE
Layer-wise Adaptive Rate Scaling (LARS)
▶ Listen · narrated
Push the batch size high enough and a carefully tuned learning rate suddenly fails. The problem is not one rate, but many layers that need different ones.
At a glance
- What it is
- A per-layer learning-rate scale from ||weights|| / ||gradients||
- Problem
- One global rate misfits layers when batches grow large
- Control knob
- A trust coefficient caps how far the local rate may move
- Later form
- LAMB applies the same idea inside adaptive-moment updates
Think of a team of hikers roped together on different slopes. One whistle (the global learning rate) tells everyone how long a stride to take. On gentle ground that works; on mixed ground the people on steep scree overshoot and the people on flat rock barely move.
LARS gives each hiker a personal stride length. It looks at how big that person already is (the weight norm) and how hard the slope is pulling (the gradient norm), then sets their stride from the ratio, with a safety limit called the trust coefficient so nobody is told to leap a canyon. The whole team still follows one route and one timetable; only the stride is local.
That is why large batches help. A big batch is like a louder whistle—you are tempted to order longer strides for everyone at once. Without per-layer adjustment, the mismatched slopes become obvious. With LARS, the loud whistle is reinterpreted quietly for each slope.
LARS computes, for each layer ℓ with weights w_ℓ and gradient g_ℓ, a local learning rate η_ℓ = η · τ · (||w_ℓ|| / ||g_ℓ||), where η is the global learning rate and τ is a trust coefficient (often combined with a weight-decay term in the denominator in practical implementations). The layer update then uses η_ℓ in place of η, typically inside a momentum SGD step in the original formulation.
The ratio ||w|| / ||g|| is a dimensionless scale correction: it asks how large a relative change the raw gradient would induce in that layer, and renormalises so that layers with very different absolute scales take comparably cautious relative steps. Large-batch training tends to raise η (linear scaling with batch size is the usual baseline); without the per-layer factor, layers whose ||w|| / ||g|| is small relative to others absorb too large a fractional update and destabilise.
LAMB reuses the same layer-wise trust ratio but applies it to the update direction produced by adaptive moments (the Adam-style preconditioned step) rather than to the raw gradient. That is the form used for the reported large-batch BERT training run that finished in 76 minutes in the 2019 paper’s setup. Limitations: the method assumes layer boundaries are the right granularity; it adds hyperparameters (τ, and interplay with weight decay); and it does not by itself fix other large-batch issues such as generalisation gaps that are not caused by layer scale mismatch.
Look closer
The ratio, not the gradient alone
LARS does not merely shrink or grow steps by gradient magnitude. For each layer it forms the ratio of the norm of that layer’s weights to the norm of its gradient, then multiplies the base learning rate by that ratio (and by a trust coefficient). A layer whose weights are large relative to its gradient receives a larger local rate; a layer already taking bold steps relative to its scale is held back. The geometry of the layer, not a hand-chosen schedule per block, drives the correction.
Why large batches expose the mismatch
With small batches, noisy gradients and modest base rates often hide layer imbalance. As batch size rises, practitioners raise the global learning rate to keep the same amount of progress per example. That single larger rate is applied everywhere. Layers whose weight and gradient scales differ sharply then diverge: some overshoot, some crawl. LARS keeps the global schedule but reinterprets it locally, so the same large-batch recipe need not be retuned layer by layer.
From LARS to LAMB
LARS was framed for large-batch training of convolutional networks with momentum-style updates. The later LAMB optimiser keeps the layer-wise norm ratio but places it on top of adaptive moment estimates, the family of updates used for models such as BERT. The second paper’s headline result—training BERT in 76 minutes—comes from that combination of large batches and layer-wise scaling inside an adaptive-moment method, not from LARS alone.
The story
Training with a larger batch is attractive for a simple reason: more examples per step means better hardware utilisation and fewer steps to see the dataset. The usual accompanying move is to raise the learning rate, often roughly in proportion to the batch size, so that each example still contributes a comparable nudge. On many convolutional workloads that linear scaling rule works for a while and then fails. Accuracy drops, or the run diverges, even though the small-batch baseline was solid.
The failure is not mysterious once you look inside the network. Different layers live at different scales. Weight norms vary; gradient norms vary; the ratio between them varies more still. A global learning rate is a single scalar asked to serve every one of those regimes. When batches are small and rates are modest, the mismatch is often tolerable. When both batch and rate grow, layers that were slightly over- or under-stepped become clearly so.
Layer-wise Adaptive Rate Scaling, introduced by You, Gitman and Ginsburg for large-batch convolutional training, attacks that mismatch directly. After the gradient is computed, each layer receives its own local learning rate. The local rate is the global rate multiplied by a trust coefficient and by the ratio of the layer’s weight norm to its gradient norm. In effect the update asks: given how large this layer’s weights already are, how large a step does this gradient imply, and should we trust that step fully?
The trust coefficient is a deliberate damper. Without it the ratio alone can propose aggressive local rates. With it, the optimiser still adapts per layer but stays within a band the practitioner is willing to accept. Weight decay and momentum continue to play their usual roles; LARS is a rescaling of the step, not a replacement for the rest of the training recipe.
That design was later carried into adaptive-moment methods. LAMB keeps the same layer-wise norm logic but applies it on top of the moment-based update direction used for large Transformer training. The 2019 large-batch BERT result—training completed in 76 minutes in the reported setup—relies on that combination: big batches for throughput, and layer-wise scaling so the raised learning rate does not treat every block as if it had the same geometry.
What LARS does not claim is that every large-batch failure is a learning-rate geometry problem, or that one trust coefficient suits every architecture. It is a targeted correction for a specific, observable failure mode: layers whose weight and gradient scales refuse to share a single rate once batches become large.
Why it mattered then
By 2017, distributed training made very large batches practical on convolutional networks, but the linear scaling rule for the learning rate was hitting a wall. Practitioners faced a choice between leaving hardware idle with smaller batches or accepting unstable, lower-accuracy runs. LARS offered a middle path that preserved a global schedule while admitting that layers are not interchangeable. It mattered because it turned an empirical dead-end—scale the batch, scale the rate, watch training break—into something adjustable with one extra coefficient and a norm ratio already cheap to compute. The later extension to adaptive-moment updates mattered for the same reason in a different regime: pretraining language models where batch size was again a lever for wall-clock time, and a single global rate was again a poor fit across depth.
Why it matters now
Large-batch training remains the default route to shorter wall-clock runs on modern clusters. Whenever a recipe multiplies batch size and learning rate together, the old mismatch between layers can reappear—especially in deep stacks where early and late blocks differ sharply in scale. LARS and LAMB are not the only answers, but the underlying observation still guides practice: stability at scale is often a per-layer problem dressed up as a global hyperparameter problem. Understanding the weight-to-gradient ratio gives a concrete way to read training failures instead of only lowering the rate and hoping.
The surprising detail
The same idea—rescale the step by ||weights|| / ||gradients|| per layer—migrated from momentum-style CNN training into the adaptive-moment world almost unchanged. LAMB is not a reinvention of the ratio; it is the ratio bolted onto a different update direction. The headline BERT timing result therefore rests on a geometric correction first written down for convolutional networks, not on a Transformer-specific trick.
What is disputed
The original LARS paper and the LAMB follow-on report strong large-batch results on the workloads they studied; they do not establish that norm-ratio scaling is necessary for every architecture or that a single trust coefficient transfers unchanged across models. Treat the method as a well-motivated correction for layer scale mismatch, not as a universal guarantee.
Remember this
LARS replaces one global learning rate with per-layer rates scaled by each layer’s weight norm over its gradient norm, so large batches need not break layers that live at different scales.
Test yourself
A run uses a large batch and a raised global learning rate. Early layers have weight norms much larger than their gradient norms; late layers reverse that pattern. With a single global rate the late layers diverge. What does LARS change in each group of layers, and what role does the trust coefficient play?
LARS multiplies the global rate by ||weights|| / ||gradients|| in each layer, so early layers (high ratio) receive a larger local rate and late layers (low ratio) a smaller one—reducing the overshoot that was breaking the late layers. The trust coefficient further scales that ratio so the local rate cannot run away even when the raw norm ratio is extreme; it is the practitioner’s cap on how far layer-wise adaptation may go.
Go deeper
- [1708.03888] Large Batch Training of Convolutional Networks · arxiv.org
- [1904.00962] Large Batch Optimization for Deep Learning: Training BERT in 76 minutes · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.