II · THE IDEA · ARTIFICIAL INTELLIGENCE
Model Soups: Weight Averaging for Improved Robustness
▶ Listen · narrated
Ensembling many fine-tunes raises accuracy and also multiplies inference cost. Weight averaging keeps the gain and leaves the bill unchanged: one forward pass, one model.
At a glance
- What it is
- Element-wise average of weights from several fine-tuned models
- Payoff
- Higher accuracy than the separate ingredients
- Inference cost
- Same as a single model, not an ensemble
- Ingredients
- Models fine-tuned from a shared initialisation
Think of several cooks who all started from the same basic sauce and then seasoned it slightly differently. Instead of tasting each sauce in turn and voting, you pour them into one pan and stir. What you serve is still a single bowl — not five bowls — yet the blend can taste more balanced than any one cook’s version.
That is a model soup. You fine-tune the same starting model several times, average the learned numbers inside it, and deploy the average as one model. You do not run the separate models at answer time. You pay for one.
Model soups construct a single parameter vector θ̄ by averaging the weights of K fine-tuned models θ₁…θ_K that share architecture and initialisation θ₀:
θ̄ = Σᵢ wᵢ θᵢ, with wᵢ = 1/K for a uniform soup, or wᵢ chosen by a held-out selection rule for greedy or learned recipes.
Unlike an ensemble, which computes (1/K) Σᵢ f(x; θᵢ) at inference, a soup evaluates f(x; θ̄) once. Memory, latency and throughput therefore match a solitary fine-tune. The method relies on the empirical observation that fine-tunes from a common pre-trained point often lie in a shared low-error basin in which linear interpolation does not cross a high-loss barrier. When that basin assumption fails — highly divergent fine-tunes, different pre-trained bases, or training from scratch with uncorrelated seeds — the average can leave the basin and accuracy falls. Selective soups mitigate this by admitting a candidate only when it improves validation error, at the cost of a small held-out search rather than any change to serving.
Look closer
The average is in weight space, not output space
A classical ensemble runs every model and combines predictions. A soup never does that. The fine-tuned parameter tensors are averaged once, offline, into a single new tensor. At serving time only that tensor is loaded. Accuracy moves; latency and memory stay those of one network.
Shared starting point matters
The method is built on models that begin from the same pre-trained initialisation and then diverge under different fine-tuning runs. Averaging unrelated trained networks is a different, less reliable act. The soup inherits the geometry of one optimisation basin explored several times, not a blend of distant solutions.
Uniform is not the only recipe
A uniform average of every fine-tune is the simplest soup. Selective recipes keep only some ingredients, or weight them unequally, when a held-out slice of data suggests that certain runs pull the average the wrong way. The finished object is still one model, not a committee at inference.
The story
Fine-tuning the same pre-trained network several times is ordinary practice. Learning rate, data order, augmentation strength and the random seed all shift the final weights a little. Each run is a usable model; some are better than others on a given test set. The usual ways to spend that diversity are to keep the single best checkpoint, or to keep them all and ensemble their predictions.
Model soups take a third route. The weights themselves are averaged. If several fine-tunes share an architecture and a common initialisation, their parameters can be combined element-wise into one new parameter set. That set is the soup. It is stored, loaded and executed exactly like any other checkpoint of the same architecture.
The practical distinction from ensembling is sharp. An ensemble’s cost grows with the number of members: each input is forwarded through every network before the outputs are merged. A soup’s cost does not. After the average has been taken, inference is a single forward pass. The central empirical claim is that this averaged model often exceeds the accuracy of the individual fine-tunes that went into it, and does so without the inference penalty an ensemble would incur.
Why averaging can help at all is tied to how those fine-tunes relate. Starting from one pre-trained point, moderate fine-tuning tends to land in a region of parameter space where the loss surface is relatively well behaved. Different runs explore slightly different paths inside that region. Their average can sit in a flatter, more central place than any single endpoint, which shows up as better accuracy and, in reported experiments, improved robustness under distribution shift. None of that requires changing the architecture or the serving stack.
Recipes vary in how much care they take over the ingredients. The uniform soup simply averages every fine-tune with equal weight. Greedier procedures add models one at a time when doing so improves held-out accuracy, and skip those that do not. In either case the product is still a single weight tensor. The hyperparameter search that produced the fine-tunes is no longer a list of discarded runners-up; it becomes the material from which the deployed model is mixed.
Why it mattered then
By the early 2020s it had become cheap to launch many fine-tuning runs from one strong pre-trained checkpoint, and expensive to serve all of them. Practitioners already kept tables of validation scores and threw most checkpoints away. Model soups gave those discarded runs a second use: instead of choosing a single winner or paying for a full ensemble, the laboratory could fold the search into one artefact whose inference cost matched the cheapest option. That mattered wherever accuracy under shift was valued and latency budgets were fixed.
Why it matters now
Fine-tuning remains the default way to specialise open-weight and closed models, and teams still produce more checkpoints than they can afford to serve. Weight averaging turns that surplus into a free accuracy lever: no new architecture, no extra GPUs at inference, no change to the tokenizer or the API shape. Wherever several fine-tunes already exist from a shared base, a soup is a low-ceremony experiment that can be tried before more elaborate distillation or routing schemes.
The surprising detail
The method’s thrift is almost blunt. Nothing clever happens at runtime. The ingenuity is entirely in refusing to treat fine-tuning diversity as either a leaderboard or a committee, and in noticing that an arithmetic mean of weights can inherit the strengths of several endpoints while remaining one file on disk.
What is disputed
Reported gains assume fine-tunes that share an initialisation and remain in a compatible region of weight space. Averaging models trained from scratch with different seeds, or fine-tunes that have diverged sharply, is not guaranteed to help and can degrade accuracy. The word “consistently” in the editorial angle should be read as “often, under the paper’s training regime,” not as a universal law.
Remember this
A model soup averages fine-tuned weights into one network, keeping ensemble-like gains without ensemble inference cost.
Test yourself
You have five fine-tunes of the same pre-trained model and a hard latency budget that allows only one forward pass. Why might a uniform soup still underperform the single best fine-tune, and what change to the recipe addresses that without returning to a multi-model ensemble?
If one or more runs landed in a poorer region, equal averaging drags the soup toward them and can erase the best run’s advantage. A selective or greedy recipe, which adds a fine-tune only when held-out accuracy rises, keeps the single-model budget while dropping ingredients that hurt the average.
Go deeper
- [2203.05482] Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time · arxiv.org
- [2210.07545] Hypergraphs for multiscale cycles in structured data · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.