II · THE IDEA · ARTIFICIAL INTELLIGENCE
Mixture of Experts Routing
▶ Listen · narrated
Parameter count and compute need not rise together. Mixture-of-experts layers keep only a few sub-networks active per token, leaving the rest dark until selected.
At a glance
- Core idea
- A gate routes each token to a small subset of expert networks
- Sparsity
- Only the chosen experts run; the rest add parameters, not FLOPs
- 2017 layer
- Sparsely-gated MoE with learned gating and top-k selection
- Switch change
- Route each token to a single expert, simplifying the design
- Typical home
- Expert FFNs replacing dense feed-forward blocks in a Transformer
Think of a hospital with many specialists and a triage desk. Every patient does not see every doctor. The desk reads the symptoms and sends the patient to one or two doctors; the others stay free for someone else. The hospital can employ a huge staff without every patient paying for every salary on every visit.
A mixture-of-experts layer works in a similar way inside a neural network. The “doctors” are expert sub-networks — usually feed-forward blocks with their own weights. The “triage desk” is a small gating network. For each token of input, the gate scores the experts and wakes only the winners. Those experts process the token; the rest do not run. The model can therefore store a very large total number of weights while doing maths only with a thin active slice.
The 2017 sparsely-gated design could wake several experts and blend their answers. Switch Transformers often wake just one per token, which is simpler to implement at scale. In both cases trainers must stop the gate from always picking the same favourites, or the unused experts never learn and the extra capacity is wasted.
In a Transformer block, sparse MoE typically replaces the position-wise FFN. Attention stays dense and shared. For token representation x, a gating network — commonly a linear projection followed by softmax or a noisy top-k variant in the 2017 layer — produces scores over N experts. Only the selected experts {E_i} are evaluated; their outputs are combined with gate weights (mixture for top-k; effectively a single weighted expert for Switch-style top-1).
Computational cost per token scales with k · cost(E) plus the cheap gate, not with N · cost(E). Parameter count scales with N. That is the efficiency claim behind large sparsely-gated MoE layers and Switch Transformers’ trillion-parameter-scale discussion.
Training requires explicit pressure toward balanced routing. Auxiliary load-balancing losses penalise uneven expert assignment. Switch also describes a capacity factor: each expert accepts only a limited number of tokens per batch, which bounds compute and communication but can drop overflow tokens. Dispatch and combine steps dominate the systems problem on multi-device setups, because tokens must be shuffled to the devices holding their assigned experts.
Limitations follow from the same mechanics: routing is discrete and can be unstable early in training; specialisation is emergent rather than labelled; and quality depends on gate quality and capacity settings, not merely on raising N.
Look closer
The gate is itself a learned network
For each incoming token the gating network produces a score over the expert pool, then keeps only the top-scoring experts — top-k in the 2017 sparsely-gated layer, top-1 in Switch Transformers. Those scores become the mixture weights for the experts that actually run. Everything else is skipped for that token, which is how a model can hold far more parameters than it multiplies on any one forward pass.
Experts specialise without being told how
The experts are usually copies of the same feed-forward architecture, distinguished only by their weights. Nothing in the loss labels an expert as “syntax” or “rare words”. Specialisation emerges because the gate repeatedly sends similar tokens to the same experts, and those experts then adapt. The papers report useful load patterns and capacity gains from this arrangement, without claiming a fixed semantic map of what each expert “means”.
Load balance is a first-class training problem
If the gate collapses onto a few favourites, most experts starve and the extra parameters stop earning their keep. Both lines of work add auxiliary load-balancing terms so tokens spread more evenly, and Switch discusses a capacity factor that caps how many tokens an expert may accept in a batch. Routing is therefore not only about choosing the best expert; it is about keeping the pool usable under real batch dynamics.
The story
A dense feed-forward layer applies the same large matrix multiply to every token. A mixture-of-experts layer offers a pool of such sub-networks — the experts — and a gate that decides, token by token, which of them should run. The unselected experts contribute parameters to the model but almost no compute on that step. That separation is the point: capacity can rise without a matching rise in FLOPs per token.
The 2017 sparsely-gated mixture-of-experts layer made this practical at large scale for neural sequence models. A trainable gating network scores the experts for each input; only the top-k are evaluated, and their outputs are combined with weights from the gate. The rest of the pool stays dark. Because different tokens may activate different experts, a batch still utilises many experts overall, while any single token pays only for its chosen few.
Switch Transformers took the same sparse-routing idea into the Transformer stack and simplified the routing decision itself. Instead of mixing several experts per token, Switch routes each token to exactly one expert. The paper argues that this top-1 choice reduces routing complexity and communication while still delivering the parameter-efficiency of mixture-of-experts layers. In the architectures studied, the experts sit where a dense feed-forward block would sit: attention remains shared; the position-wise MLP becomes a routed pool of MLPs.
Neither design is free of training hazards. A gate that always prefers the same experts wastes the pool. Both papers therefore treat load balancing as part of the learning problem, not an afterthought: auxiliary losses encourage a more even spread of tokens, and Switch adds an explicit capacity limit so a popular expert cannot absorb unlimited traffic in a batch. Tokens that would overflow may be dropped or deferred under those rules — a practical compromise between ideal routing and stable hardware utilisation.
The editorial payoff is straightforward. Once routing is sparse and balanced, adding experts mainly adds parameters and memory for weights that are only sometimes touched. Compute per token tracks the active subset, not the full roster. That is how the Switch line could discuss models at trillion-parameter scale while keeping the per-token multiply cost closer to a much smaller dense network. The gate is small; the specialists are large; only a few specialists answer each knock.
Why it mattered then
By 2017, simply widening or deepening dense networks was already an expensive way to buy quality. The sparsely-gated MoE layer offered a different lever: grow the parameter count by adding experts, but charge each token only for the experts it actually uses. That mattered on the hardware and cluster budgets of the time, where memory for weights and FLOPs per step were both scarce. Switch Transformers, a few years later, pressed the same idea into the Transformer FFN slot and showed that even top-1 routing — the simplest useful sparse policy — could support very large sparse models with a cleaner systems story. In that moment, MoE routing was less a curiosity than a path past the wall where dense scaling alone looked increasingly costly.
Why it matters now
Open-weight and production language models still face the same tension: users want more capacity, operators want bounded latency and energy. Sparse expert routing remains one of the few architectural patterns that deliberately decouples stored parameters from active compute. Whenever a model card cites dozens or hundreds of experts and a small number active per token, it is reusing this lineage — gate, top-k or top-1 selection, load balance, expert FFNs. Understanding the routing step explains why two models with similar active FLOPs can differ wildly in total size, why serving them stresses interconnect and memory layout differently from dense peers, and why training stability still hinges on not letting the gate collapse.
The surprising detail
The “mixture” in the original sparsely-gated layer is literal: several experts can run for one token and their outputs are weighted together. Switch’s headline simplification is almost austere by comparison — one expert per token, no blend at the expert outputs for that step. The claim is not that mixing was useless, but that the systems and optimisation overhead of multi-expert routing could be traded away while still harvesting most of the sparse-capacity benefit. The name stayed; the mixture sometimes shrank to a singleton.
What is disputed
Published MoE and Switch results depend on particular gating functions, auxiliary losses, capacity factors and cluster implementations. The papers support sparse routing as an efficient scaling method under those setups; they do not establish a single universal recipe for how many experts, what k, or how specialisation will look in every domain.
Remember this
MoE routing spends full parameters only on the experts a token actually activates — capacity without proportional FLOPs.
Test yourself
A dense FFN and an MoE FFN pool are both described as having “billions of parameters,” yet the MoE model claims far lower compute per token. What has to be true about routing and load balance for that claim to hold in practice, and what failure mode quietly destroys it?
Only a small subset of experts may run per token (top-k or top-1), so multiplies track the active experts rather than the full pool. Load must also stay spread: if the gate collapses onto a few favourites, almost all tokens pay for the same busy experts while the idle majority never earns its parameter cost — and you are back to something closer to a smaller dense model wearing a larger disguise.
Go deeper
- [1701.06538] Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer · arxiv.org
- [2101.03961] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.