II · THE IDEA · ARTIFICIAL INTELLIGENCE
Dense Versus Sparse Models
▶ Listen · narrated
The number in the model name tells you how much you must store, not how much you must compute. That gap is the difference between a model you can run and one you cannot.
At a glance
- Dense model
- Every parameter active for every token
- Sparse model (MoE)
- Only a subset of parameters active per token, routed dynamically
- Memory requirement
- Both must hold all parameters in memory
- Compute requirement
- Sparse uses far fewer FLOPs per token
Imagine a large reference library where every book must remain on the shelf, but you only read two books per visit. A mixture-of-experts model works this way. It contains many sub-networks called experts—often eight or sixteen of them—but for each token it processes, it activates only one or two. The rest sit idle. You still need a building large enough to hold all the books, but you only spend time reading a small fraction of them. That is why a sparse model with 400 billion parameters can run as fast as a dense model with far fewer, even though both need similar amounts of memory. The total size determines what fits on your hardware; the active size determines how quickly it runs.
In a mixture-of-experts architecture, the feedforward sublayer of each transformer block is replaced with multiple parallel feedforward networks (the experts) and a gating mechanism that routes each token to a subset of them—typically one or two out of eight or sixteen. The gating network is usually a learned linear projection followed by a softmax, and it operates on the token's hidden state at that layer. The output is a weighted combination of the chosen experts' outputs. Unchosen experts contribute zero to the forward pass for that token.
The total parameter count is the sum of all experts plus shared layers (attention, embeddings, layer norms). The active parameter count per token is the sum of only the activated experts plus shared layers. For Mixtral 8x7B, that is roughly 13B active out of 47B total. For Switch Transformers with one expert active out of 2048, the ratio is far more extreme.
Memory usage is determined by the total parameter count, because all experts must remain resident for the routing decision to be made at runtime. Compute cost per token scales with the active parameter count. Training cost scales with total parameters times the number of tokens, but the per-token cost is much lower than a dense model of the same total size. Deployment cost depends on whether you are constrained by memory (favouring dense models of smaller total size) or by compute and latency (favouring sparse models with lower active counts). The routing mechanism adds negligible compute overhead compared to the expert networks themselves, but it introduces training challenges around load balancing, which both Mixtral and Switch Transformers address with auxiliary loss terms that penalise uneven expert utilisation.
Look closer
The routing decision happens per token
In a mixture-of-experts architecture, the model contains multiple expert networks—often eight or sixteen—and a small gating network chooses which one or two to activate for each token. The choice is made afresh every time, so two tokens in the same sentence may be routed to different experts. The unchosen experts contribute nothing to that token's forward pass. Mixtral 8x7B has eight experts of seven billion parameters each, plus shared layers, totalling roughly 47 billion parameters, but only about 13 billion are active for any given token.
Memory holds all experts, always
Even though most experts are idle for a given token, they must all remain loaded in GPU memory, because the routing decision is data-dependent and unpredictable. You cannot page an expert in from disk mid-inference without destroying throughput. The memory footprint is therefore determined by the total parameter count, not the active count. A 400-billion-parameter sparse model needs roughly the same memory as a 400-billion-parameter dense model, even though it computes far less per token.
The advantage is in compute, not capacity
Sparse models let you fit more learned capacity—more total parameters—into the same compute budget. Training and inference are faster per token because fewer parameters are touched, but the model is not smaller in memory or on disk. This matters when your constraint is FLOPs or latency, not when your constraint is VRAM. If you have 80GB of memory, a dense 70B model and a sparse 400B model may both fit, but the sparse one will generate tokens faster.
The story
A dense model is the simpler case. Every parameter participates in every forward pass. If the model has 175 billion parameters, then generating one token means performing roughly 175 billion multiply-accumulate operations, plus activations and a few other steps. The parameter count and the active parameter count are the same number.
A sparse model, specifically a mixture-of-experts model, breaks that equality. The parameters are divided into groups called experts, and a small routing network—sometimes just a single linear layer with a softmax—decides which expert or experts to use for each token. The decision is made dynamically, based on the token's hidden state at that layer. Most experts do nothing for that token. Their weights sit idle in memory.
Mixtral 8x7B is a clear example. It contains eight experts, each roughly the size of a 7B dense model, plus shared attention and embedding layers. The total parameter count is around 47 billion. But the router activates only two experts per token, so the active parameter count per token is closer to 13 billion. The compute cost per token is therefore much closer to a 13B dense model than a 47B one, even though the memory footprint is that of the full 47 billion.
Switch Transformers took this further, building models with over a trillion parameters by using up to 2048 experts per layer, with only one expert active per token. The total parameter count becomes enormous, but the active count per token stays manageable. The paper reports that Switch-C, with 1.6 trillion parameters, achieves better performance than a dense model four times its training compute cost, because sparsity lets you grow capacity without proportionally growing compute.
The routing itself adds complexity. The gating network must learn which expert is best for which input, and during training the system must balance load across experts so that all of them receive enough examples to learn effectively. If routing concentrates all tokens on a few experts, the others atrophy. Both Mixtral and Switch Transformers describe auxiliary loss terms designed to encourage balanced routing, though the details differ.
The memory constraint remains absolute. If your hardware cannot hold all the experts, the model will not run, regardless of how few are active at once. This is why a 400B sparse model and a 400B dense model have similar deployment requirements even though their compute profiles are entirely different. The sparsity saves you time and electricity, not memory.
Why it mattered then
Mixture-of-experts architectures are older than the transformer. They appeared in the 1990s as a way to divide a problem among specialised sub-networks. But they became practically important at scale only when transformer models grew large enough that compute, not memory, became the primary bottleneck during training. Switch Transformers, published in 2021, demonstrated that sparsity could scale to over a trillion parameters while keeping training cost manageable. The paper showed a clear trade: by activating only a tiny fraction of parameters per token, you could train a model with far more total capacity in the same wall-clock time. The result was better performance on many tasks, particularly those requiring specialised knowledge, because different experts could specialise in different domains or linguistic patterns. Mixtral, released in 2023, brought the same principles to an open-weight model that individuals and small labs could actually run. It made the memory-versus-compute distinction visceral: here was a model with 47 billion parameters that generated tokens as fast as a 13B dense model, yet still required the VRAM to hold all 47 billion. The architecture was no longer a research curiosity; it was a deployment decision.
Why it matters now
The distinction between total parameters and active parameters now determines what you can afford to run. Cloud providers price by compute, so a sparse model costs less per token even though it occupies the same memory. If you are running models locally, the calculation reverses: you pay once for the VRAM, and the sparse model gives you more capacity for the same memory budget, plus faster generation. Mixture-of-experts models are also becoming the default at the frontier. Many recent large models use some form of sparsity, because the alternative—scaling dense models to trillions of parameters—requires proportionally scaling compute for every token, which becomes prohibitively expensive. Sparsity is a way to keep growing model capacity without growing the cost per token at the same rate. There are still open questions. Routing can be unstable, and load balancing remains a tuning problem. Some tasks benefit more from sparsity than others, and it is not always clear in advance which will. But the fundamental trade is well understood now: sparsity exchanges memory for compute efficiency. You must still pay the memory cost, but you get the compute savings in return, and at current scales that trade is usually worth making.
The surprising detail
Switch Transformers reported training a 1.6-trillion-parameter model using the same compute budget as a 10-billion-parameter dense model, and achieving better downstream performance. The gap between total capacity and active capacity was so large—only one expert active per token out of hundreds—that the effective compute cost per token stayed tractable even as the parameter count grew by two orders of magnitude. The result suggests that much of what we think of as model capacity is latent: it does not need to be active all the time to be useful, as long as the routing network learns to activate the right subset when it matters.
What is disputed
The exact active parameter counts for Mixtral and similar models depend on how shared layers (attention, embeddings, normalisation) are counted, and different sources report slightly different figures. The principle—that active count is much lower than total count—is consistent, but the specific ratios vary by architecture and by how the calculation is done.
Remember this
Sparse models save compute, not memory. The parameter count tells you what will fit; the active parameter count tells you how fast it will run.
Test yourself
You have 80GB of VRAM and a choice between a dense 70B model and a sparse 8x22B model (176B total parameters, roughly 22B active per token). Both fit in memory. For a long document summarisation task where you must process a 50,000-token input, which constraint matters more, and which model does it favour?
The constraint that matters more is compute time, not memory, because both models fit. Processing 50,000 tokens through 70 billion parameters takes longer than processing the same tokens through 22 billion active parameters, even though the sparse model holds 176 billion in memory. The sparse model will finish the task faster, assuming the routing overhead is small compared to the compute savings. If the task were instead to fit the largest possible model into a fixed memory budget and you did not care about speed, the calculation would be different—but here, with both fitting and speed mattering, the sparse model wins.
Go deeper
- Mixtral of Experts · arXiv · Albert Q. Jiang et al. · 2024-01-08
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity · arXiv · William Fedus et al. · 2021-01-11
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.