II · THE IDEA · ARTIFICIAL INTELLIGENCE
Loss Landscapes: Flat Minima and Sharp Ones
▶ Listen · narrated
Two networks can reach the same error and still differ. What matters is the shape of the dip they landed in: a narrow crack, or a wide floor that forgives small knocks.
At a glance
- Core problem
- Loss is a function of millions of parameters; the surface cannot be plotted directly
- Common tool
- A 2D slice through parameter space, centred on a trained point
- Sharp minima
- Narrow basins where small parameter steps raise loss quickly
- Flat minima
- Wide basins where loss stays low over a larger neighbourhood
- Caveat
- Unnormalised directions can make the same surface look sharp or flat at will
Think of the network's error as height, and the network's settings as a position on the ground. Lower ground means fewer mistakes, and training is a walk downhill.
You cannot draw that ground. A real network has millions of settings, so the ground has millions of directions. What you can do is cut a flat slice through it: stand at the finished network, choose two directions, walk a short way in each, and note the height as you go. That produces a small map you can actually look at.
The map answers one question — how wide is the dip the network stopped in? Sometimes it is a narrow crack, where one short step sends the error jumping. Sometimes it is a broad shallow bowl, where the same step changes almost nothing.
Wide is usually safer, for a plain reason. The network never stays exactly where it stopped. It gets jostled: by the random handfuls of data used during training, by the rounding a computer performs, by real inputs that differ a little from the practice ones. A wide bowl absorbs those shoves. A narrow crack does not.
One catch matters more than the others. Make your two directions longer and every dip in the drawing looks steeper, even though nothing about the network changed. So the directions are rescaled first, to match the size of the network's own parts. Only then is it fair to hold two maps side by side.
Let θ* ∈ R^n be the reference parameters — typically a trained network — and let δ, η ∈ R^n be two direction vectors. The plotted surface is f(α, β) = L(θ* + αδ + βη), evaluated on a grid of (α, β). This is a two-dimensional affine slice through the loss rather than a projection of it, and it carries no information about directions outside span(δ, η).
Filter-wise normalisation exists to remove a scale artefact. Because f depends on the magnitude of δ and η, rescaling those vectors rescales apparent curvature: sharpness read off the plot is not invariant to how the plot was parameterised. The standard remedy replaces each filter block of δ with a copy whose norm matches the corresponding block of θ*, and likewise for η. Curvature is then at least comparable across networks of the same architecture.
Sharpness along the slice is read from how quickly f grows with ‖(α, β)‖, and flatness from a wide region where f stays near f(0,0). These are slice-local quantities, not the Hessian eigenspectrum of L at θ*, although the two are related when δ and η align with high-curvature eigenvectors.
Trajectories are drawn by expressing θ_t − θ* in the basis (δ, η), by least squares when δ and η are not orthonormal. The component orthogonal to the plane is discarded, so planar distance understates true distance in R^n unless that residual is zero.
The limitations are structural rather than incidental. A single plane can miss barriers, saddles and alternate basins; competing flatness measures disagree with one another; and a reparameterisation of the network can change measured sharpness while leaving the function the network computes identical. Any claim linking slice geometry to generalisation therefore has to survive changes of direction choice, normalisation scheme, architecture and task before it counts as evidence.
Look closer
A slice is not the whole surface
Any plot of this kind is a two-dimensional cut through a vastly higher-dimensional space. Two random or otherwise chosen directions define a plane; loss is evaluated on a grid in that plane and drawn as contours or a height map. Structure that lives outside the plane is invisible. Apparent convexity, ridges, or chaotic contours therefore describe that slice, not the full geometry the optimiser actually travels.
Scale of the axes decides the story
If the two direction vectors are left at arbitrary scale, a network with large weights can look artificially flat and one with small weights artificially sharp. Filter-wise normalisation rescales each direction so that it matches the magnitude of the corresponding filters in the reference network. Only after that step do side-by-side landscapes become comparable across architectures and training runs.
Trajectories and basins are different objects
Plotting the path of optimisation means projecting successive parameter iterates into the same plane used for the contours. The path may cross ridges, skirt plateaus, or settle in a basin that the final 2D contour map makes look smooth. Convergence in the plot does not prove the basin is globally flat; it only shows that, along those two directions, the neighbourhood of the final point is relatively gentle or steep.
The story
Training a neural network means adjusting a very long list of numbers — the parameters — until the network's error falls. Call that error the loss: a single number saying how wrong the network currently is.
Here is the useful picture. Every possible setting of the parameters produces some loss. So the loss is a surface stretched over all possible settings, and training is a walk downhill across it. With two parameters you could draw that surface as a landscape and watch a marble roll into a valley. A modern network has millions of parameters, so its surface has millions of directions and cannot be drawn at all.
The workaround is to draw a slice. Take a finished network as the centre of the picture. Choose two directions to walk in. Step out along those two directions, measure the loss at each point, and plot the result as hills and valleys. That gives a flat cut through a million-dimensional object, in the way a cross-section is a flat cut through a solid.
What the cut shows is the width of the dip the network settled into. A sharp minimum is a narrow crack: step a little away from the finished settings and the loss climbs steeply. A flat minimum is a broad, shallow bowl: the same step barely changes the loss.
That width has a practical consequence, and the reason is concrete. The parameters never stay exactly where training left them. Training uses small random batches of data, numbers are rounded to fit the hardware, and real inputs drift away from the examples used in training. Each of those knocks the network slightly off the point it stopped at. In a wide bowl a knock costs almost nothing. In a narrow crack the same knock sends the error climbing.
There is a trap in the picture, and it is the part most worth knowing. The apparent steepness depends on how far one step along your chosen directions actually travels. Stretch those directions and every dip looks sharp; shrink them and every dip looks flat — same network, same weights, same losses. The remedy is to scale the plotting directions to match the size of the network's own weights, filter by filter, before drawing anything. Only then does a comparison between two plots carry meaning.
The route training took can be drawn on the same map. Each step of training is a point in the full parameter space, and describing it in terms of the two chosen directions casts a shadow of that route onto the flat picture. The shadow shows whether training approached its resting place smoothly or bounced about first. It remains a shadow rather than the path: two points that look close on the map may be far apart along the directions nobody drew.
That limit deserves stating plainly, because a single slice can miss almost anything. A wall between two valleys, a way around that wall, a second and better basin entirely — none of them appear unless the two chosen directions happen to pass through them. So the honest use of these figures is narrow and comparative: holding the normalisation and the recipe fixed, does this architecture settle somewhere wider than that one? A slice can answer that. What the whole surface looks like, it cannot.
Why it mattered then
A loss curve — error plotted against training time — tells you whether a run worked, not why. As networks grew deeper and their parameter counts climbed, that gap turned into a problem: two designs could finish at the same error while one trained dependably and the other collapsed or generalised poorly. Researchers wanted to examine the ground around a finished solution without pretending they could draw millions of directions at once. Normalised two-dimensional slices supplied a picture people could argue over. Once the plotting directions were tied to the scale of the network's own weights, papers could set two landscapes side by side and make a claim that survived scrutiny: this change widens the basin, that one narrows it. Part of the conversation shifted from the final score toward the shape of the place training actually stopped.
Why it matters now
The question now surfaces in deployment rather than in research. A model squeezed into smaller numbers for cheaper inference — quantisation — has its parameters deliberately rounded, which is precisely the knock a narrow basin cannot absorb. Two fine-tuning runs ending at the same loss can behave very differently once the data drifts a little. When engineers choose a learning-rate schedule or a regulariser, they are deciding, with limited visibility, where in that surface the run will come to rest. Loss slices remain a research instrument rather than a routine dashboard, but the underlying question is an everyday one: is this solution sitting somewhere wide enough to survive the rounding, the noise and the drift it is about to meet?
The surprising detail
The same trained network can be made to look as though it sits in a dramatically sharp minimum or a dramatically flat one, without a single weight changing. All it takes is stretching or shrinking the two directions chosen for the plot. Same parameters, same losses, two opposite conclusions — and both are printable. The geometry only becomes arguable once the axes are pinned to the scale of the network's own filters, which is a reminder that the figure is an instrument somebody built rather than a photograph of the loss.
What is disputed
Apparent flatness or sharpness in a 2D slice need not hold in other directions. Empirical links between flat minima and generalisation are suggestive in many settings but not settled as a universal law; different measures of flatness can disagree. Whitening and iterative normalisation improve conditioning under stated assumptions, yet their effect on the global loss geometry is indirect and architecture-dependent.
Remember this
A loss landscape plot is a two-dimensional cut through a surface of millions of dimensions, drawn along two directions someone chose. Flat and sharp mean something only once those directions are scaled to the network's own weights.
Test yourself
Two papers show contour plots around trained networks. Paper A’s minimum looks needle-sharp; paper B’s looks wide and flat. Before concluding that B found a better basin, name two checks you would demand of both figures.
First, that both used comparable direction normalisation — ideally filter-wise normalisation against each network’s own weights — so axis scale is not inventing the difference. Second, that the slice directions and the contour ranges are reported clearly enough to compare like with like; a zoomed-in window around a narrow well can look as dramatic as a zoomed-out window around a wide valley. Even then the conclusion is only about those planes, not the full high-dimensional surface.
Go deeper
- [1712.09913] Visualizing the Loss Landscape of Neural Nets · arxiv.org
- [1904.03441] Iterative Normalization: Beyond Standardization towards Efficient Whitening · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.