II · THE IDEA · ARTIFICIAL INTELLIGENCE
Sparse Mixture of Experts Inference Routing
▶ Listen · narrated
Dense layers grow cost with every new parameter. Sparse mixture-of-experts models break that link: capacity rises while the work per token stays closer to that of a much smaller network.
At a glance
- What it is
- Sending each token to specialised expert sub-networks rather than one shared dense layer
- How chosen
- A learned gating mechanism scores experts and routes the token
- Why sparse
- Only the selected experts run, so most parameters stay idle for that token
- Design aim
- Massive total parameter counts with modest computation per token
- Named work
- Switch Transformers: simple, efficient sparsity at very large scale
Think of a hospital with many specialists and a triage desk. Every patient (token) does not see every doctor. The triage desk (the gating mechanism) reads the symptoms and sends the patient to one specialist department (an expert network). The hospital can employ a huge staff overall—its “parameter count”—but the work done for any one patient is only what that specialist performs.
A sparse mixture-of-experts layer works the same way inside a model. Instead of one giant shared network that always runs in full, there is a bank of expert networks and a learned gate. The gate routes each token to the expert that should handle it. The other experts stay unused for that token. That is how the model can be very large in total size while still doing a smaller amount of computation on each step. Switch Transformers are a well-known line of work that emphasises making this sparse pattern simple and efficient, including at extreme scales talked about in trillion-parameter terms.
A sparse mixture-of-experts (MoE) layer replaces a dense position-wise feed-forward block with N expert sub-networks and a gating network. For an input token representation x, the gate computes scores g = Gate(x) over the experts. A routing rule maps g to a small active set A (often a top-k selection; Switch-style designs stress simple, efficient sparsity, commonly associated with very low k). Only experts e ∈ A are evaluated. Their outputs are combined—typically as a gate-weighted sum when |A| > 1—to form the layer output.
Parameters scale with N times the size of one expert (plus the gate), but FLOPs per token scale with |A| times expert cost, not with N. Different tokens in a sequence or batch may select different experts, so the active parameter set is dynamic across the forward pass. At inference, performance hinges on the router implementation: grouping tokens by assigned expert, handling capacity limits when an expert is over-subscribed, and avoiding pathological gate collapse where only a few experts ever receive mass.
Switch Transformers frame this as a path to trillion-parameter scale under sparse activation: grow the expert bank for capacity, keep per-token work closer to a dense model whose width matches the active experts rather than the full bank. The architectural claim is precisely the decoupling of stored capacity from executed compute via learned token routing.
Look closer
The gate sits in front of the experts
Before any expert network sees a token, a small gating network reads a representation of that token and produces a score for each expert. Routing is the act of turning those scores into a choice: which expert, or which small set of experts, will actually run. The unused experts contribute parameters to the model’s total size but do not contribute multiply-accumulate work for that token. The quality of the whole system depends as much on this choice as on the experts themselves.
Sparsity is per token, not per model
From the outside the model is enormous. From the inside of a single forward pass, only a slice is alive. Different tokens in the same sequence can take different routes, so the set of active parameters shifts as the sequence is processed. That is what “dynamically routing tokens” means in practice: the computational graph is not fixed for the batch in the way a dense feed-forward layer is.
Switch-style simplicity
Switch Transformers are associated with making this sparsity simpler and more efficient at scale, including regimes described in terms of trillion-parameter models. The editorial point is not a particular leaderboard score; it is the architectural bet that a lean routing rule plus specialised experts can grow capacity without a matching growth in cost per token. How many experts fire, and how ties and load are handled, are design choices that shape both speed and stability.
The story
In a conventional transformer block, every token that reaches a feed-forward layer is processed by the same large dense network. Grow that network and you grow both the model’s capacity and the floating-point work required for every token. Sparse mixture-of-experts layers try to separate those two curves. They replace one shared dense map with a bank of expert networks and a gate. The gate’s job is to read the token and decide which expert—or which small handful of experts—should handle it. Only those experts run. The others stay cold for that token.
The parameters still exist. They still have to be stored, moved, and kept consistent across training and deployment. What they do not have to do is fire on every example. That is the sense in which routing enables massive parameter counts with modest computational cost: cost tracks the active path, while capacity tracks the full collection of experts.
At inference time the picture is concrete. A token representation enters the MoE layer. The gating mechanism produces scores over the expert pool. A routing rule turns scores into assignments. The token is sent along the chosen path, the expert computes its output, and—when more than one expert is used—the results are combined according to the gate. Tokens that look different to the gate can land on different experts inside the same batch and the same layer. Specialisation is not hard-coded by the designer in advance; it is whatever division of labour the learned gate and the trained experts settle into.
Switch Transformers sit in this family with an emphasis on simple and efficient sparsity, and with scaling language that reaches trillion-parameter models. The “switch” idea is part of that simplicity story: make the routing decision cheap and decisive enough that the sparse pattern remains practical when both the expert count and the total parameter count become extreme. The details of load balancing, capacity factors, and training stability matter enormously in real systems, but they are supporting machinery for the same core loop—score, route, compute, merge.
What you notice as a practitioner is the new failure modes that come with the new freedom. If the gate collapses onto a few favourites, some experts starve and capacity is wasted. If routing is noisy, specialised behaviour never forms. If the infrastructure cannot gather tokens by expert efficiently, the theoretical saving never appears on the wall clock. Sparse MoE inference routing is therefore not only an algorithmic idea; it is a claim about how computation should be organised when most of the model is deliberately left unused on any given step.
Why it mattered then
As models scaled, the dense link between parameter count and per-token compute became a hard constraint on what could be trained and served. Mixture-of-experts routing offered a different contract: keep adding specialised capacity, but pay mainly for the experts that actually run. Switch Transformers pushed that contract toward simpler sparsity and toward scales described in trillion-parameter terms, making the routing layer a central object of architecture design rather than a curiosity on the side.
Why it matters now
Serving large models is still dominated by memory, communication, and the cost of every active parameter along the path. Sparse MoE routing remains one of the few widely used ways to hold more parameters than you are willing to compute with on each token. Anyone reading modern architecture diagrams—clouds of experts, a gate, a router—is looking at this same separation of capacity from work, whether or not the implementation matches any single paper.
The surprising detail
The idle parameters are not a bug; they are the point. A sparse mixture-of-experts model is allowed to be far larger than the compute budget would suggest, because the gate’s refusal to call most experts is what keeps inference tractable. The model’s headline size and its per-token cost are deliberately measuring different things.
What is disputed
The supplied sources name Switch Transformers and the general sparse-MoE routing idea; they do not fix a single routing cardinality, load-balancing rule, or measured efficiency number for every deployment. Real systems differ in whether one expert or several fire, and in how uneven expert load is corrected. Treat “modest computational cost” as the design aim of sparse routing, not as a guarantee independent of implementation.
Remember this
In a sparse MoE layer the gate chooses a path; capacity is the full expert bank, but cost follows only the experts that run for each token.
Test yourself
A dense feed-forward layer and a sparse MoE layer are both described as having a huge number of parameters. At inference, for a single token, what fundamental difference determines how much compute you actually spend—and what role does the gating mechanism play in that difference?
In the dense layer every parameter in that layer participates in the computation for the token, so cost scales with the full width of the layer. In the sparse MoE layer only the expert networks selected for that token run; the other experts’ parameters remain inactive for the step, so cost scales with the active path rather than with total parameter count. The gating mechanism is what makes the selection: it scores experts from the token’s representation and thereby decides which specialised sub-networks will absorb the work.
Go deeper
- [2101.03961] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity · arxiv.org
- [2203.06091] Convection Reconciles the Difference in Efficiencies Between Low-Mass and High-Mass Common Envelopes · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.