II · THE IDEA · ARTIFICIAL INTELLIGENCE
Sparse Mixture of Experts Routing
▶ Listen · narrated
A network can hold far more knowledge than it spends on any single input. The trick is a router that leaves most of the weights idle for each token.
At a glance
- What it is
- Per-token routing into a few expert subnetworks, not the full layer
- The gate
- A small network that scores experts and keeps only the top-k
- Experts
- Usually feed-forward blocks; only the chosen ones run
- Payoff
- Parameter capacity rises without a matching rise in compute per token
- Hard part
- Keeping expert load balanced so a few do not monopolise traffic
Think of a hospital with many specialists and a triage desk. Every patient does not see every doctor. The desk reads the symptoms and sends the patient to one or two specialists; the others stay free for someone else. The hospital can employ a large staff without every arrival costing a full round of every clinic.
A sparse mixture-of-experts layer works in a similar way. The experts are repeated blocks inside the network, usually feed-forward subnetworks. A small gating network looks at each token and scores the experts. Only the top-scoring few run. Their outputs are blended according to the gate’s weights. The unused experts do no work for that token.
That is why the model can hold a large number of parameters while each token still pays for only a thin slice of them. Training has to push the gate to share patients across doctors; otherwise a few experts take all the traffic and the rest never learn.
A sparsely-gated MoE layer replaces a dense position-wise feed-forward block with N expert FFNs plus a gating network. For input x the gate computes scores g = Gate(x), retains the top-k experts, and forms y = Σ_i g'_i · E_i(x) over the selected set, where g' is the renormalised gate mass on those experts. Non-selected experts contribute neither matmuls nor activation memory for that token.
Parameter count therefore scales with N, while FLOPs per token scale with k and the expert width, not with N. The 2017 sparsely-gated MoE paper established this conditional-computation pattern; GShard applied it at large scale for multilingual translation and combined it with automatic sharding so experts live on different devices and tokens are dispatched to the devices that hold their chosen experts.
Training pathologies are structural: without noise, auxiliary load-balancing losses, and per-expert capacity limits, the gate’s discrete top-k choice concentrates on a few experts. Overloaded experts hit capacity and drop or reroute tokens; underused experts receive weak gradients and fail to specialise. Implementing MoE in practice is therefore as much a systems and load-balancing problem as a pure architecture choice—communication of tokens to sharded experts, capacity factor tuning, and stability of the gate all sit on the critical path.
Look closer
Routing is decided token by token
In a sparsely-gated mixture-of-experts layer, each token is scored against a pool of experts by a gating network. Only the highest-scoring experts—commonly the top one or two—are allowed to run for that token. The layer output is a weighted combination of those selected experts. Neighbouring tokens in the same sequence may therefore awaken entirely different blocks.
Capacity and compute part company
A dense feed-forward layer charges you for every parameter on every token. A sparse MoE layer can hold far more parameters because most experts stay idle for any given token. The 2017 sparsely-gated MoE work and the later GShard system both exploit this gap: model capacity scales with the expert count, while the floating-point work per token scales mainly with how many experts the gate actually opens.
The gate must be trained to share work
Left alone, a gate tends to favour a small subset of experts and starve the rest. Both lines of work treat load balancing as a first-class training concern—through noisy gating, auxiliary losses, and capacity limits that stop a popular expert from accepting unbounded tokens. Without those constraints the sparsity pattern collapses and the extra experts stop earning their keep.
The story
A conventional neural layer applies the same weights to every input that reaches it. Sparse mixture-of-experts routing breaks that habit. The layer is built from many expert blocks—typically feed-forward networks of the same shape—plus a gating network that sees each token and decides which few experts are worth running.
The gate produces a score for every expert. A top-k selection keeps only the winners; the rest do no computation for that token. The selected experts process the token, and their outputs are combined using the gate’s normalised weights. Everything else in the layer stays cold. That is conditional computation in the sense used by the sparsely-gated MoE layer and by GShard: which parameters fire depends on the input, not only on the architecture diagram.
The practical motive is scale. Adding experts increases the number of parameters the model can store, and therefore the variety of patterns it can specialise for, without multiplying the work done on each token by the same factor. A token that would once have passed through one large feed-forward block now passes through a handful of smaller ones chosen for it. The rest of the expert pool remains available for other tokens, other languages, or other corners of the distribution.
GShard took this pattern into large multilingual translation models and paired it with automatic sharding, so that experts could be spread across devices while the gate still routed each token to the machines that held its chosen experts. The same sparse rule that saves compute also becomes a distribution rule: only the devices that own the selected experts need to do heavy work for that token.
None of this is free of friction. If the gate collapses onto a few favourites, the unused experts waste memory and the model behaves like a much smaller dense network. Training therefore includes mechanisms—noise in the gate, auxiliary load-balancing losses, per-expert capacity limits—so that tokens spread across the pool. When those mechanisms work, the layer keeps its bargain: a large parameter store, a small active subset per token, and a router that decides the subset on the fly.
Why it mattered then
By the late 2010s, simply widening or deepening every layer had become an expensive way to buy capacity. The sparsely-gated mixture-of-experts layer offered a different lever: grow the number of specialist blocks while keeping per-token computation roughly fixed by activating only a few of them. GShard then showed how to combine that idea with automatic sharding for giant translation models, turning conditional computation into something that could be placed across many devices rather than only described on a whiteboard. In its moment, the method mattered because it separated two quantities that had been locked together—how much the model could store, and how much work each token demanded.
Why it matters now
Sparse expert routing remains one of the main architectural answers when parameter count is allowed to outrun the compute budget of a single forward pass. Any system that advertises a large total weight count but activates only a subset of blocks per token is working in this lineage. The same design tensions still apply: the gate must be trained not to collapse, load must be balanced across devices, and capacity limits still decide what happens when too many tokens pick the same expert. Understanding the original routing idea clarifies why those systems are fast in one sense and awkward in another—they are large in storage and selective in use.
The surprising detail
The gate is not a fixed schedule or a hand-written rule. It is a learned network, trained jointly with the experts, so the model discovers for itself which inputs belong together. That freedom is also the failure mode: without balancing pressure the gate can learn a near-constant preference for a few experts, and the remaining blocks never specialise because they almost never see traffic. The sparsity pattern is therefore something the system can quietly abandon unless training actively defends it.
What is disputed
Published MoE systems differ in k, gating noise, balancing losses and capacity factors. The shared claim is conditional top-k expert activation; exact recipes are not universal across papers.
Remember this
Sparse MoE routing grows capacity by adding experts, then spends compute only on the few the gate opens for each token.
Test yourself
A dense feed-forward layer and a sparse MoE layer are given the same per-token compute budget. The MoE layer has many more total parameters. What must be true about how the MoE layer spends that budget, and what training failure would erase the advantage?
Only a small top-k subset of experts may run for each token; the rest stay idle so total parameters can exceed active compute. If the gate collapses onto a few favourites—and load balancing fails—most experts never activate, the layer behaves like a much smaller dense network, and the extra parameters stop buying useful capacity.
Go deeper
- [2006.16668] GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding · arxiv.org
- [1701.06538] Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.