II · THE IDEA · ARTIFICIAL INTELLIGENCE
Model Merging via Weight Interpolation
▶ Listen · narrated
Fine-tune the same starting model several times and you get several sets of numbers. Average those numbers, position by position, and the single blended model can beat every one of them.
At a glance
- What it is
- Coordinate-wise average of several fine-tuned weight sets
- Extra training
- None after the individual fine-tunes exist
- Inference
- One model; time and memory match a single network
- Reported effect
- Accuracy can rise relative to the separate fine-tunes
- Name in source
- Model soups
Several cooks each adjust the same family recipe slightly differently. You could serve all their stews at dinner and let people taste every one, which is what an ensemble does and which takes far more time and washing-up. You could pick your favourite cook and pour the others away. Or you could stir equal parts of every finished stew into one pot and serve that. The one pot is the soup: one meal, one helping of effort.
In the network, each stew is the full list of numbers the model learned during fine-tuning. Stirring means averaging those lists position by position, so the first number is averaged with the first, the second with the second, and so on down the line. No further cooking happens. You average once, load the result, and run the model exactly as you would run any other.
It works because every cook began from the same base recipe, the shared pretrained model. Their versions are variations on one dish. Mixing stews that started from unrelated recipes gives you something nobody wants to eat.
A model soup builds one parameter vector θ̄ = (1/k) Σᵢ θᵢ from k fine-tuned checkpoints θ₁…θₖ. All k must share an architecture and, crucially, a common pretrained initialisation. The average is coordinate-wise across the tensors, or across a chosen subset of them. No gradient step is ever applied to θ̄, so the merge is not training; it is a single pass of arithmetic over the saved weights.
At inference only θ̄ is loaded. Arithmetic operations and memory therefore match a single model. Contrast an output ensemble, which runs k forward passes and averages logits or softmax outputs, multiplying both. Reported accuracy for the soup can exceed the mean of the ingredients and sometimes the best of them, which happens when the fine-tunes settle in nearby low-loss regions joined by a straight path along which loss stays low.
The limitations follow directly from that geometry. If the checkpoints sit too far apart, because of different pretraining, incompatible fine-tuning recipes, or tasks that drag parameters in opposing directions, the interpolated point falls in a high-loss region and the soup performs worse than its ingredients. The basic recipe assigns every checkpoint the same 1/k share. Greedy variants that add or reject candidates one at a time exist in the wider literature, but they still act directly on parameters rather than learning a mixing function. Anything wrapped around the weights, including the tokeniser, the configuration and the numeric type, must match across ingredients, or the averaged tensors describe nothing coherent.
Look closer
The blend is in parameter space, not output space
An ensemble averages predictions from several models at run time. A weight soup averages the parameters once, offline, and then discards the separate checkpoints. What remains is a single set of weights. That is why inference time does not grow with the number of models that went into the soup, whereas an ensemble’s cost does.
The ingredients share a starting point
The method described in the source averages multiple fine-tuned models. In practice that only stays coherent when those fine-tunes begin from a common pretrained initialisation and remain in a region of weight space where linear interpolation is meaningful. Averaging unrelated networks trained from scratch is a different, and generally destructive, operation; the soup recipe does not claim to handle that case.
Uniform average, no learned mixing coefficients
In the basic form, each fine-tuned checkpoint contributes equally to every parameter: the merged weight is the arithmetic mean of the corresponding weights. There is no second training loop to learn how much of each model to keep. The editorial appeal is precisely that simplicity—blend, then serve—while still reporting accuracy gains over the individual fine-tunes.
The story
A neural network is, in the end, a long list of numbers. Those numbers are called weights, and they are what training changes. Fine-tuning means taking a network that has already been trained on a large body of data and continuing to train it a little further for a particular job. Do that several times from the same starting network, varying the random seed, the settings, or the order the data arrives in, and you finish with a handful of finished models. Each one is complete and usable.
The usual choices are both unsatisfying. You can test them all, keep the best, and delete the rest, which throws away the work in every other run. Or you can keep them all and run them together at answer time, averaging what they predict. That second approach is called an ensemble, and it works, but it costs what it sounds like it costs: five models means five times the computation and five times the memory, every time a question is asked.
Weight interpolation offers a third route, and the mechanism is simpler than the name. Line the models up. Take the first number in each one and average them. Take the second number in each one and average them. Keep going through every position. The result is a new list of numbers exactly as long as any single ingredient. Load it into the same network shape and you have one ordinary model again. The paper that introduced this called the result a model soup, and reported that the averaged model can be more accurate than any of the fine-tunes that went into it, while taking no longer to run.
Notice what is absent. There is no further training on the blend. No loss is measured, no gradient is computed, no data is passed through. The expensive part was already paid for when the separate fine-tunes were run. The merge is arithmetic, and it happens once, offline.
The arithmetic only works because of a condition that is easy to miss. Averaging two lists of numbers is the same as walking in a straight line from one to the other and stopping halfway. That walk has to stay over ground where the model still works. Fine-tunes that all began from the same pretrained network stay close enough together for the halfway point to be sensible; they are variations on one model rather than strangers. Two networks trained from scratch, from different random starting numbers, sit far apart, and the point midway between them is usually a model that does nothing useful at all. The soup recipe does not claim to handle that case.
Once the soup exists, using it is completely unremarkable. There is no routing machinery deciding which ingredient to consult, no extra copies held in memory, no need to keep the original checkpoints on disk to serve traffic. Storage and response time match one model, because it is one model. The gain reported in the source arrives for free at that point, having been bought entirely with fine-tuning runs you were going to do anyway.
Why it mattered then
Before this, there were two accepted ways to wring more accuracy out of fine-tuning. You could search carefully through training settings and keep the single best result, discarding everything else. Or you could keep several results and run them side by side, averaging their answers. The second was reliably better and reliably more expensive, because every extra model added its full cost to every request served. Averaging the weights instead used the variety already sitting in those multiple runs, but paid for it only once, in advance. At a moment when the same pretrained models were being fine-tuned again and again for benchmarks and products, a step that raised accuracy and left serving costs untouched was useful the day it was published.
Why it matters now
Fine-tuned open-weight models pile up fast: different seeds, small adapter modules folded back into the main weights, separate checkpoints for closely related jobs. Weight interpolation is still the least ceremonious way to fold that pile into one file that fits an ordinary single-model serving slot. It also underlies the more elaborate merging recipes people reach for when they want one model that covers several skills without running a fresh joint training job. The original finding, that a plain average can beat its own ingredients, is what those later recipes are built on.
The surprising detail
The basic soup never learns how to blend. Each merged number is a flat arithmetic mean of the numbers in that position across the ingredients, weighted equally. There is no second training stage, no small network deciding which ingredient to trust, no separate ratio tuned for each layer. It is the same average a child would compute. The source still reports accuracy above the individual fine-tunes, with running costs identical to one model.
What is disputed
Gains depend on the fine-tunes being comparable and related; the cited work studies averaging multiple fine-tunes of a shared initialisation. Results on arbitrary, unrelated networks are not what that evidence covers, and not every soup beats every ingredient on every metric.
Remember this
Average the weights of fine-tunes that share a starting point, and you may get one stronger model that still costs a single pass to run.
Test yourself
You have three fine-tunes of the same pretrained checkpoint and want better accuracy at serving time without paying three forward passes. Why might a uniform weight average be preferable to keeping the single best fine-tune, and what assumption does that average rely on?
The average can outperform any one ingredient by combining complementary gains from the separate runs, while still loading as one set of weights—so latency and memory match a single model. It relies on the fine-tunes sitting in a region of parameter space where linear interpolation stays well-behaved, which shared initialisation usually provides. Unrelated models trained from different starts are outside that assumption.
Go deeper
- [2203.05482] Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time · arxiv.org
- [2306.14887] Efficient tensor network simulation of IBM's Eagle kicked Ising experiment · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.