The paper in four sentences
A mixture-of-experts layer contains many separate small networks (experts) plus a gating network, or router, that looks at each input and picks a handful of experts to handle it.
Only the chosen experts run, so you can add more experts, and with them more knowledge, without making each input more expensive to process. The paper calls this conditional computation.
Making it work needed engineering: noise and extra training terms so the router spreads work evenly instead of favouring a few experts, and clever batching across many GPUs so each expert gets enough inputs at once to run efficiently.
With up to 137 billion parameters in one layer, these models beat the best published language-modelling and translation results with less computation per word. The same design powers many of today's largest language models.
Idea · §1, §1.1
Size helps, but cost grows with it
By 2016 it was clear that neural networks improve with more data and more parameters (the adjustable numbers inside them). The problem is that in an ordinary network every parameter takes part in processing every input. Double the parameters and you double the work per word. Training cost grows roughly with model size times data size, which quickly becomes unaffordable.
An old idea offered a way out: conditional computation, where only part of the network is active for any particular input. A sentence about chemistry might use different parts from a sentence about football. Then the network could be enormous while each input still uses only a small slice. The idea had been proposed before but never delivered big gains in practice, mainly because computer hardware is built for large, uniform blocks of arithmetic, not for "this input goes here, that one goes there".
TakeawayOrdinary networks pay for every parameter on every input. Conditional computation would let capacity grow without the cost per input growing with it.
Idea · §2 · Fig. 1
Experts and a router
The paper's layer has n experts, each a small, ordinary feed-forward network, all with the same shape but their own parameters. A gating network looks at the input and gives each expert a score. Only the top k scorers (the paper mostly uses k = 4, or 2 at each level of a two-level version) are run. Their outputs are combined as a weighted average, with weights from the scores. The other experts do nothing for this input.
The layer sits between the layers of a recurrent language model and is applied at every word position separately, so every word can be sent to a different set of experts. In the toy below, each word has a fixed (made-up) description, and the router scores 16 experts for it.
TakeawayA router scores every expert, keeps the top few, and mixes their outputs; the rest of the layer stays idle for that input.
Check yourself
- With 4,096 experts and k = 4, what fraction of the layer's experts does one word use?
- 4/4,096, about 0.1%. The layer's parameters grow with the number of experts; the work per word grows only with k.
Idea · §2.1, §4 · Appendix A
Stopping the rich from getting richer
Left alone, the router tends to collapse onto a few favourites. Experts that get chosen early receive more training, so they get better, so they get chosen more. Soon most experts sit idle: wasted parameters, and the busy ones become a bottleneck.
The paper uses two fixes. First, it adds a little random noise to the router's scores during training, so near-ties are broken in different ways and more experts get a turn. Second, it adds an extra training term that penalises imbalance: it measures how unequal the experts' total shares of the work are across a batch and nudges the router to even them out. (A second, similar term targets the number of inputs each expert receives.)
The simulation below shows the effect. Tokens arrive from eight different topics. Each time an expert is chosen it improves a little, and the router learns to favour better experts. Switch the balancing term on or off and watch the share of tokens each expert handles.
TakeawayRouters naturally collapse onto a few experts; noise and a penalty on uneven usage keep all experts busy and learning.
Check yourself
- Why is collapse self-reinforcing?
- Chosen experts receive gradient updates and improve; unchosen ones don't. The router then sees the chosen ones performing better and picks them even more often.
Idea · §3, §5 · Table 1 · Figs. 2–3
Making it fast, and what it achieved
Sparsity creates a practical problem: if a batch of examples is split among thousands of experts, each expert sees only a few, and a computer chip doing arithmetic on tiny batches wastes most of its power. The authors fix this by spreading the experts across many GPUs and having every copy of the rest of the model send its inputs to the same shared experts, so each expert collects a decent-sized batch from all of them at once.
Results, on a standard one-billion-word language-modelling benchmark: models with about 4 billion parameters beat the previous best published result, even the cheapest using only a fraction of its computation per word. On a far larger 100-billion-word news corpus, a model with 65,536 experts (about 68 billion parameters in the expert layer) reached 39% lower perplexity (a measure of prediction error) than a model with the same computation per word but no experts. Bigger still, 131,072 experts and 137 billion parameters, was slightly worse, which the authors suspect was too much sparsity. In machine translation, mixture-of-experts models beat Google's production system of the time on English to French and English to German, with less computation per word and less training time.
TakeawayWith the batching tricks in place, adding experts bought large quality gains at almost constant cost per word, up to a point.
Check yourself
- Why do sparse experts create a "shrinking batch" problem?
- A batch is divided among many experts, so each expert gets only about k/n of it, and small batches use hardware inefficiently.
Abstract, restated
A network's capacity to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, was proposed in theory as a way to increase capacity dramatically without a proportional increase in computation, but faced significant algorithmic and performance challenges. This work addresses them and realises its promise, showing greater than 1000× improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. It introduces a Sparsely-Gated Mixture-of-Experts layer of up to thousands of feed-forward sub-networks, with a trainable gating network choosing a sparse combination of experts for each example, and applies it to language modelling and machine translation, where capacity is critical. MoE layers with up to 137 billion parameters are applied convolutionally between stacked LSTM layers; on large language-modelling and translation benchmarks the models achieve significantly better results than the state of the art at lower computational cost. ICLR 2017.
Section · §2–2.1 · Eqs. 1–5
The MoE layer and noisy top-k gating
Given experts E1, …, En (feed-forward networks with identical architecture and separate parameters) and gating network G with sparse n-dimensional output:
Experts with G(x)i = 0 are not computed. A dense softmax gate Gσ(x) = Softmax(x · Wg) (Eq. 2) is the classical choice; the paper adds sparsity and tunable noise:
The noise scale is input-dependent and learned through Wnoise; it helps load balancing (Appendix A). Training is plain backpropagation: the gate values of the selected experts are differentiable functions of Wg and Wnoise, so with k > 1 the router learns which of its chosen experts to weight more. The authors note the top-k discontinuity was not a problem in practice, and contrast this with earlier boolean-gate approaches trained with REINFORCE. Step through the computation:
Hierarchical MoE (Appendix B). For very large n, a primary gate selects a sparse set of expert groups, each with its own secondary gate; this two-level design was used for the largest models.
Section · §3
Performance engineering
The shrinking batch problem (§3.1). With batch size b, each expert receives about kb/n ≪ b examples. The fixes:
- Mixing data and model parallelism. Standard layers and gates are replicated data-parallel over d devices, but each expert exists once; all replicas send their routed examples to it, so each expert sees about kbd/n examples, a factor d larger. Growing the number of devices then grows the expert batch.
- Taking advantage of convolutionality. In a language model the same MoE is applied to every time step of the previous LSTM layer, so all time steps of the batch can be routed together, multiplying the batch by the sequence length. (A recurrently applied MoE would lose this.)
Network bandwidth (§3.2). Experts are stationary; only their inputs and outputs travel. The ratio of an expert's computation to its input and output size equals its hidden-layer width, so wide hidden layers (the paper uses ReLU hidden layers of large width) keep the arithmetic-to-communication ratio high.
Section · §4 · Appendix A · Eqs. 6–11
Balancing expert utilisation
The gate tends to converge to a state where it always gives large weights to the same few experts, an imbalance that is self-reinforcing because favoured experts are trained faster. Rather than hard constraints, the paper adds soft losses. The importance of an expert over a batch X is its summed gate value:
where CV is the coefficient of variation (standard deviation ÷ mean) across experts. Equal importance does not imply equal numbers of examples (one expert could get a few large weights, another many small ones), which matters for memory and speed on distributed hardware. Appendix A therefore adds Lload = wload · CV(Load(X))2, where Load is a smooth estimator of the number of examples per expert: for each example and expert, the probability (under the gating noise, using the normal CDF Φ) that the expert's noisy score stays in the top k if only its own noise is resampled (Eqs. 8–11). This is differentiable even though the count itself is not. Wg and Wnoise are initialised to zero so the start is balanced and noisy. In the language-model experiments, wimportance = wload = 0.1 (Appendix C). The paper's ablation (Table 6) finds that using no balancing loss gives clearly worse perplexity than using either loss, and that the load loss reduces the maximum-to-mean load ratio.
Check yourself
- Why square the coefficient of variation rather than use the variance?
- CV is scale-free: it does not depend on the batch size or on the overall magnitude of gate values, so one weight w works across settings. Squaring makes it smooth with a simple gradient.
- Why is a separate load loss needed when importance is balanced?
- Importance sums gate weights, which can be balanced while example counts are not; hardware cost depends on counts.
Section · §5 · Tables 1–5 · Figs. 2–3
Experiments
One Billion Word benchmark (§5.1). MoE layers between two LSTM layers; experts are feed-forward networks with a hidden layer of 1,024 ReLU units (about a million parameters each). At about 8 million operations per time step, perplexity falls steadily as experts are added; the 4,096-expert hierarchical model is 24% lower than the computationally matched baseline (Fig. 2, left). Higher-budget models (Table 1):
| Model (Table 1) | Test perplexity, 10 epochs | ops / timestep | Params excl. embedding & softmax | Training | TFLOPS / GPU |
|---|---|---|---|---|---|
| Best published (Jozefowicz et al., 2016) | 34.7 | 151M | 151M | 59 h, 32 K40s | 1.09 |
| Low-budget MoE | 34.1 | 8.9M | 4,303M | 15 h, 16 K40s | 0.74 |
| Medium-budget MoE | 31.3 | 33.8M | 4,313M | 17 h, 32 K40s | 1.22 |
| High-budget MoE | 28.0 | 142.7M | 4,371M | 47 h, 32 K40s | 1.56 |
The low-budget model beats the best published result with about 6% of its computation per time step. The TFLOPS column measures how efficiently the GPUs were used, against a quoted peak of 4.29 TFLOPS for the K40.
100 Billion Word Google News corpus (§5.2, Fig. 3). At about 8M ops per timestep, quality keeps improving with experts up to 65,536 (about 68B parameters), at 39% lower perplexity than the matched baseline after 100B words of training; 131,072 experts (about 137B parameters) did slightly worse, possibly from too much sparsity. Efficiency at 65,536 experts was 0.72 TFLOPS/GPU with 99.994% of the layer inactive per example.
Machine translation (§5.3–5.4). A GNMT-style model with fewer LSTM layers plus MoE layers in encoder and decoder, up to 2,048 experts (about 8.7B parameters):
| Task | MoE BLEU | GNMT BLEU | MoE ops/timestep | GNMT ops/timestep | Training |
|---|---|---|---|---|---|
| WMT'14 En→Fr (Table 2) | 40.56 (longer training; 40.35 standard) | 39.22 | 85M | 214M | MoE 3 days on 64 K40s vs GNMT 6 days on 96 K80s |
| WMT'14 En→De (Table 3) | 26.03 | 24.91 | 85M | 214M | MoE 1 day on 64 K40s vs GNMT 1 day on 96 K80s |
On Google's production English→French data the MoE gained about 1 BLEU on the test set after one sixth of GNMT's training time (Table 4). A single multilingual MoE trained on 12 language pairs had 19% lower perplexity than multilingual GNMT and beat it on 11 of 12 pairs, losing on English→Korean, which the authors attribute to overtraining on rare, oversampled pairs (Table 5). An appendix table shows individual experts specialising by syntax and meaning.
Section · §6 and later work
Limitations and lineage
- Systems complexity. The approach depends on custom parallelism, all-to-all communication and per-device batching. Hardware efficiency (0.3–1.56 TFLOPS/GPU) was well below peak, and the 137B model was limited partly by not scaling the batch with the number of GPUs.
- Heuristic balancing. Auxiliary losses trade off against the main objective and need tuning; uneven load still forces capacity limits or dropped tokens in later systems.
- Memory, not compute. Parameters must all be stored and served even though few are used per token, which shifts the bottleneck to memory and communication.
- Architecture. The MoE sat between LSTM layers; its natural home turned out to be the feed-forward block of a transformer.
Origins: the paper builds on mixtures of experts from Jacobs, Jordan, Nowlan and Hinton (1991) and Jordan and Jacobs (1994), and Eigen, Ranzato and Sutskever's (2013) stacked deep MoEs, adding sparsity and scale.
Check yourself
- A modern MoE transformer has 8 experts per layer and routes each token to 2. Roughly what fraction of the expert parameters does a token use, and why is total parameter count misleading about its cost?
- 2/8 = 25% of the expert parameters (plus all shared attention parameters). Cost per token follows the active parameters, while memory follows the total.
Reference
Glossary
- Expert
- One feed-forward sub-network inside the MoE layer.
- Gating network / router
- Computes G(x), the sparse weights over experts.
- Top-k
- Keep the k highest-scoring experts per input; others get weight 0.
- Conditional computation
- Activating only part of a network per example.
- Importance
- An expert's summed gate weight over a batch (Eq. 6).
- Load
- (Smoothed) number of examples routed to an expert over a batch.
- CV
- Coefficient of variation, standard deviation ÷ mean.
- Shrinking batch
- Each expert receives only about kb/n of a batch of size b.
- ops/timestep
- Multiply-adds per word position in the forward pass, excluding the softmax.