II · THE IDEA · ARTIFICIAL INTELLIGENCE
Mixture of Experts: Sparse Routing at Scale
▶ Listen · narrated
A model can hold far more knowledge than it spends compute on. The trick is a gate that sends each token to a few specialists and leaves the rest untouched.
At a glance
- Core idea
- Route each input to a small subset of expert sub-networks
- Gate
- A trainable network chooses which experts run
- Sparsity
- Most experts stay inactive for any single example
- Payoff
- Parameter count can rise without a matching rise in FLOPs
- Classic paper
- Shazeer et al., Outrageously Large Neural Networks (2017)
Think of a hospital with many specialists and a triage desk. Every patient does not see every doctor. The triage desk reads the symptoms and sends the patient to one or two specialists; the others stay free for someone else. The hospital can employ a huge staff—its total expertise is large—while each visit only costs the time of the doctors who were actually called.
A mixture-of-experts layer works in a similar way inside a neural network. The “doctors” are expert sub-networks, usually ordinary feed-forward blocks. The “triage desk” is a gating network that scores those experts for the current input and keeps only a small top set. The chosen experts run; their outputs are combined; the rest of the pool does no work on that step. That is why parameter count can grow faster than per-example compute: most parameters are capacity on standby, not labour on every pass.
A sparsely-gated mixture-of-experts layer replaces a dense sub-layer (commonly a position-wise feed-forward block) with a pool of N expert networks and a gating network. For an input vector x, the gate produces scores over the N experts—classically via a linear projection, optional noise for exploration during training, and a top-k selection that zeros all but k experts. The layer output is a sparse mixture, typically the weighted sum of the selected experts’ outputs using normalised gate values as coefficients.
Because only k experts run, multiply-add cost scales with k times the cost of one expert, not with N, while the parameter count scales with N. Training is joint: experts and gate receive gradients through the mixture. A known failure mode is gate collapse, in which a few experts dominate and the remainder starve. Auxiliary load-balancing losses (or equivalent constraints) encourage batch-level utilisation across experts so that sparse capacity remains in use. At system level, efficient implementations must also handle the routing traffic—dispatching different batch elements to different experts—without letting communication and load imbalance erase the theoretical FLOP saving.
The 2017 Shazeer et al. formulation is the reference point for this pattern at large scale in language modelling and translation: conditional computation as a first-class, trainable layer rather than a hand-built cascade.
Look closer
The gate is part of the model
Routing is not a fixed heuristic. A gating network scores the experts for the current input and typically keeps only the top few. Those scores are trained jointly with the experts, so the system learns both what each specialist should do and when to call it. The unused experts contribute parameters and capacity, but almost no multiply-adds on that step.
Experts are usually ordinary blocks
In the sparsely-gated design that made the approach practical at scale, the experts are feed-forward sub-networks sitting behind the gate. The surrounding architecture can remain a familiar sequence model; the MoE layer is a drop-in way to thicken capacity at selected depths rather than a wholly new backbone.
Load balance is a real constraint
If the gate collapses onto a handful of favourites, most experts starve and the extra parameters buy little. Training therefore needs pressure—through auxiliary losses or related mechanisms—so that examples are spread across experts. Sparse capacity only helps when the gate actually uses it.
The story
Dense scaling is simple and expensive. Every new parameter participates in every forward pass, so memory and compute climb together. Mixture-of-experts scaling loosens that coupling. The model still contains a large pool of parameters, but they are partitioned into expert sub-networks. A gating network inspects the incoming representation and activates only a sparse subset of those experts. The rest do no work for that input.
The 2017 sparsely-gated mixture-of-experts layer made this pattern workable for very large nets. Earlier mixture-of-experts ideas existed; the practical contribution was a gate that stays sparse during training and inference, combined with engineering that keeps the routing and the expert computation efficient on real hardware. The result is conditional computation: capacity is paid for in parameters and in the occasional use of each expert, not in a full dense multiply on every step.
On a single token or example, the path through an MoE layer looks almost ordinary. The gate produces scores, a small number of experts run, and their outputs are combined—often as a weighted sum using the gate values. Zoom out across a batch or a corpus and the picture changes. Different inputs light up different specialists. In principle the model can grow by adding experts whose skills cover rarer patterns, while the per-token compute stays tied to the number of experts that actually fire, not to the total headcount.
That separation is the point of the architecture, and also its fragility. Gating must remain sparse enough to save compute, yet balanced enough that experts receive useful gradient signal. Poor balance wastes capacity; overly sharp or unstable routing can hurt quality. So an MoE model is not merely a dense model with extra weights. It is a dense scaffold plus a learned traffic system that decides, at each step, which parts of the scaffold earn their keep.
The 2017 work demonstrated the idea at what was then an extreme scale for neural language models and translation systems, showing that sparsely-gated layers could absorb huge parameter counts while keeping the activated computation manageable. The headline is not that sparsity is free—routing, communication, and load balance all cost something—but that parameter growth and per-example FLOPs need not move in lockstep. That is the scaling lever the architecture offers.
Why it mattered then
By the mid-2010s, gains from simply widening or deepening dense networks were increasingly bought with compute. Conditional computation had long been an appealing alternative on paper: run only the pieces that matter for this input. The sparsely-gated MoE layer turned that appeal into a trainable, hardware-conscious design. It mattered because it offered a route to models whose total parameter count could be described as outrageously large without demanding a matching outrage in FLOPs on every example. For language modelling and machine translation, that was a concrete way to keep scaling capacity when dense scaling alone was becoming painful.
Why it matters now
The same tension still structures large-model design. Memory budgets, serving cost, and training FLOPs pull in different directions. Sparse expert routing remains one of the few architectural patterns that deliberately separates how much a model can store from how much it must compute per token. Whenever a system advertises a vast parameter count alongside a smaller active path, it is replaying the MoE bet: specialists on call, most of them quiet, a gate deciding the roster. Understanding that bet—its load-balance obligations as well as its savings—is part of reading modern architecture claims without confusing capacity with cost.
The surprising detail
The gate is trained, not hand-written, and nothing forces it to use the expert pool evenly. Without an explicit load-balancing pressure, a rational gate can collapse onto a few strong experts and leave the rest almost unused—so the model quietly becomes dense and small again while still carrying the memory burden of a large one. Sparse capacity is therefore something the training recipe must defend, not a property that appears automatically once experts exist.
What is disputed
Public discussion often treats expert specialisation as neatly semantic—one expert for code, one for dialogue, and so on. The 2017 sparsely-gated work establishes trainable sparse routing and scale, not a settled map of what each expert represents. How cleanly experts carve the input space remains dependent on architecture, data, and training recipe; it should not be assumed from the name alone.
Remember this
Mixture-of-experts scaling buys parameters on the shelf and spends compute only on the few experts the gate opens.
Test yourself
A team doubles the number of experts in an MoE layer but leaves the gate’s top-k unchanged and does not improve load balancing. Which costs and benefits should move, and which might barely move at all?
Stored parameters and memory footprint should rise with the larger expert pool. Per-token compute should stay roughly tied to k, so FLOPs per example need not double. Quality and effective capacity, however, may barely improve if the gate still routes most traffic to the same favourites—the new experts must actually receive examples and gradients, or they remain unused baggage rather than specialists.
Go deeper
- [1701.06538] Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer · arxiv.org
- [2304.14233] Large Language Models are Strong Zero-Shot Retriever · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.