II · THE IDEA · ARTIFICIAL INTELLIGENCE
Mixture of Experts
▶ Listen · narrated
You can run a model too large for your machine's memory if most of it stays dormant at any moment. Mixture of Experts makes that conditional activation the architecture, not a trick.
At a glance
- What it replaces
- A single dense feedforward block in each transformer layer
- What it adds
- A router network and multiple parallel expert blocks, typically 8 to 64
- Typical routing
- Top-1 or top-2: each token is sent to one or two experts, ignoring the rest
- Parameter efficiency
- Total parameters can be ten times active parameters, or more
Imagine a large hospital with many specialist doctors. When a patient arrives, a triage nurse decides which two specialists they need to see, and those two examine the patient while the others remain free for other cases. The hospital has the capacity of all its doctors, but each patient only occupies two at a time, so more patients can be seen in parallel without hiring more staff.
A mixture-of-experts model works similarly. It contains many expert networks, but for each token only a small number are activated. A router — a small learned network — looks at the token and decides which experts should process it. Those experts do their work, the others stay idle, and the token moves forward. The model's total size is large, because it contains all the experts, but the computation per token is small, because most experts are not used. That makes it possible to run a very large model on hardware that could not handle a traditional model of the same total size, as long as you have enough memory to store all the experts even though you are only computing with a few at a time.
In a mixture-of-experts layer, the feedforward block is replaced by a router and N expert networks, each structurally identical to a standard feedforward block. For each token, the router computes a score for every expert, typically using a linear projection followed by a softmax. In top-k routing, the k experts with the highest scores are selected, their outputs are computed, and the results are combined using the normalised router scores as weights. The other N-k experts are not evaluated for that token.
The forward pass cost is therefore proportional to k, not N, while the parameter count and memory footprint scale with N. If k=2 and N=32, you activate one-sixteenth of the expert capacity per token, but you must hold all thirty-two experts in memory. This is beneficial when memory bandwidth and capacity are less constrained than compute, which is increasingly true for inference on modern accelerators.
Training introduces two complications. First, the router must learn a useful policy. Its input is the token representation before the feedforward block, and its output is a probability distribution over experts. Gradients flow back through the selected experts and through the router's scoring function, so the router learns to assign tokens to experts that reduce the loss. Second, without intervention the router can collapse, sending most tokens to a small subset of experts. To prevent this, an auxiliary load-balancing loss is added, penalising uneven distribution of tokens across experts within a batch. The exact form varies: Shazeer et al. used a differentiable approximation of load variance; Switch Transformers used a simpler auxiliary term based on the fraction of tokens routed to each expert and the mean router probability for that expert.
The load-balancing loss is weighted by a hyperparameter, and setting it is a trade-off. Too little and some experts are undertrained; too much and the router's preferences are overridden, harming quality. Capacity factor is another parameter: if more than k tokens in a batch select the same expert, some implementations drop the excess, others buffer them, and others allow the expert to process more than its nominal capacity. Each choice has implications for training stability and throughput.
In practice, mixture-of-experts models achieve strong performance with significantly fewer active parameters than dense models of comparable quality, but the total parameter count is higher, and the memory and bandwidth requirements reflect that. For deployment, this means a mixture-of-experts model may be preferable when inference cost is dominated by FLOPs rather than memory access, or when the hardware can hold the full model but cannot compute through a dense equivalent quickly enough.
Look closer
The router is itself a small learned network
For each token, the router produces a score for every expert. In a top-2 system, the two highest-scoring experts process that token; the others see nothing. The router's weights are trained alongside everything else, so it learns during training which kinds of input belong with which expert. The scores are normalised, often with a softmax, so they can be interpreted as a probability distribution — though the router is making a discrete choice, not blending outputs from all experts.
Experts specialise, but not in ways you can always label
Early intuition suggested experts might divide by topic or language: one for code, one for French, one for mathematics. Inspection shows the reality is more textured. Some experts do show a preference for certain domains or syntactic roles, but the specialisation is learned implicitly from the loss signal, not imposed, and it is rarely as clean as a human category. An expert may activate more often for formal register, or for tokens that appear mid-sentence, or for reasons that resist simple description. The router's job is to minimise loss, not to build an interpretable taxonomy.
Load balancing is enforced with an auxiliary loss term
Left to its own devices, the router can collapse: it sends nearly all tokens to a few experts, leaving the others undertrained and ignored. To prevent this, training adds a penalty that encourages roughly equal use of all experts across a batch. The exact mechanism varies — the original Shazeer paper used a differentiable load-balancing loss; Switch Transformers introduced a simpler auxiliary term — but the goal is the same. This does not guarantee that every expert sees every kind of input, only that no expert is starved of data. The tension between router preference and enforced balance is a recurring theme in the literature.
The story
A standard transformer layer contains a self-attention block followed by a feedforward block, and the feedforward block is applied identically to every token. If the model has twelve billion parameters and a thousand tokens in the context, all twelve billion parameters are active for all thousand tokens. Mixture of Experts changes that.
Instead of one feedforward block, the layer contains a small routing network and many parallel expert blocks — typically eight, sixteen, or more. For each token, the router examines the token's representation and assigns it to a small number of experts, often just one or two. Those chosen experts process the token; the others remain inactive. The result passes forward, and the next layer repeats the process with its own router and its own set of experts.
The immediate consequence is that total parameters and active parameters become different numbers. A model might contain forty billion parameters but use only four billion per token. That ten-to-one ratio is not fixed — it depends on how many experts exist and how many are active — but the principle holds across designs. You pay the cost of the active parameters in computation and the cost of the total parameters in storage and memory bandwidth.
For local inference, this is transformative. A model with forty billion total parameters and four billion active parameters needs enough memory to hold forty billion, but each forward pass does the arithmetic of a four-billion-parameter dense model. If your machine has sufficient RAM or VRAM to load the weights, inference becomes feasible even when training or running an equivalently sized dense model would not be. The bottleneck shifts: you are limited by how much you can store and transfer, not by how much you can compute per token.
The routing decision happens independently for each token, and a token in position five might visit different experts than a token in position fifty. This means the active set of parameters changes across the sequence, and the model's behaviour is more dynamic than a dense architecture of the same total size. It also means the architecture is harder to reason about. In a dense model, every parameter sees every token; in a mixture of experts, participation is conditional, and the training signal for a given expert depends on how often the router sends work its way.
The idea itself is older than the transformer. The 2017 Shazeer paper introduced a sparsely gated mixture of experts as a layer type for recurrent and convolutional models, and the core insight — that you can build capacity without requiring all of it to activate — carried forward. Switch Transformers, published in 2021, simplified the design by routing each token to exactly one expert and demonstrated that the approach scaled to over a trillion parameters. Mixtral, released in 2024, brought mixture of experts to an open-weight model with strong performance and a practical size for local deployment, using eight experts and top-2 routing.
There is a training cost. The router must learn a useful assignment policy, the load-balancing penalty adds complexity to the loss function, and distributed training must handle the fact that different experts may reside on different devices. The efficiency gains are real, but they are not free, and the engineering required to train these models well is more involved than for a dense architecture of comparable active size.
Why it mattered then
The original 2017 paper from Shazeer and colleagues was motivated by a scaling problem. Training larger models improved performance, but the computational cost grew prohibitively. Mixture of Experts offered a way to increase model capacity — the total number of parameters available to represent patterns — without increasing the arithmetic required per example in proportion. That conditional computation was not new in principle, but making it work reliably within a neural network that trained with backpropagation required solving the load-balancing problem and ensuring the router's gradients were useful. The timing mattered. Hardware was becoming faster, but not fast enough to keep pace with the desired model sizes, and researchers were looking for architectural changes that could deliver better scaling properties. The mixture-of-experts layer provided one such change: it decoupled capacity from computation in a way that dense layers did not. The 2017 work showed a 1000-fold improvement in efficiency on certain tasks, though those numbers came with caveats about task choice and measurement. The broader point held: you could build a model that was large in parameters but efficient in FLOPs, and that opened a design space that had been impractical before.
Why it matters now
Mixture of Experts has moved from a research curiosity to a deployment architecture. Mixtral, an open-weight model with eight experts and forty-seven billion total parameters, runs on consumer hardware that would struggle with a dense model of the same total size. The active parameter count per token is roughly seven billion, which makes inference feasible on a machine with enough memory to hold the weights but not enough compute to process a forty-seven-billion-parameter dense model at reasonable speed. This changes what is possible locally. A developer or researcher with a single high-end GPU can now run a model whose total capacity rivals systems that previously required a cluster. The quality-per-active-parameter is competitive, and the memory-versus-computation trade-off favours local deployment in a way that dense architectures do not. The result is that mixture-of-experts models are appearing in open-weight releases, not just in papers, and they are being used for applications where inference cost and hardware availability are constraints. The architecture also matters for hosted services. A provider can serve a mixture-of-experts model with lower per-token cost than a dense model of equivalent quality, because the FLOPs per forward pass are smaller. That cost saving can be passed to users or retained as margin, and it makes certain applications economically viable that would not be otherwise. The trade-off is memory bandwidth and the complexity of managing which experts are active, but for many workloads that trade-off is favourable.
The surprising detail
The load-balancing penalty is essential, but it works against the router's primary objective. The router is trained to minimise the language modelling loss, which means sending each token to the expert that will process it most effectively. The load-balancing term penalises the router for doing exactly that if it results in uneven expert usage. The final routing policy is therefore a compromise: not the assignment that would give the lowest loss if all experts were equally trained, and not the perfectly balanced assignment that would ignore quality. The auxiliary loss coefficient is a hyperparameter, and tuning it is part of the art of training these models. Set it too low and some experts become dead weight; set it too high and the router's learned preferences are overridden, and performance suffers. The fact that this tension exists at all, and that it must be managed with a manually chosen penalty weight, is a reminder that the system is not learning a single coherent objective but balancing two.
Remember this
Total parameters determine memory; active parameters determine computation. That split makes large models feasible on modest hardware.
Test yourself
A mixture-of-experts model with thirty-two experts and top-2 routing has twelve billion total parameters. You want to estimate the active parameter count per token. What else do you need to know, and why is the answer not simply twelve billion divided by sixteen?
You need to know how the parameters are distributed. The router itself has parameters, and the self-attention blocks (which are not sparse in a typical mixture-of-experts design) also contribute. Only the feedforward blocks are replaced by experts, so only those parameters are subject to sparse activation. If the feedforward blocks account for two-thirds of the model's parameters, and top-2 routing activates two of thirty-two experts, then two-sixteenths (one-eighth) of those feedforward parameters are active, plus all of the attention and router parameters. The active count is therefore roughly one-third (attention and router) plus one-eighth of two-thirds (active feedforward), which is about forty-two per cent of the total, not six per cent. The ratio of total to active parameters is not the same as the ratio of experts to active experts, because not all parameters participate in the sparse routing.
Go deeper
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer · arXiv · Noam Shazeer et al. · 2017-01-23
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity · arXiv · William Fedus et al. · 2021-01-11
- Mixtral of Experts · arXiv · Albert Q. Jiang et al. · 2024-01-08
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.