The paper in four sentences
In an ordinary transformer every word (token) passes through the same feed-forward layer, so adding parameters always adds computation.
A Switch Transformer replaces that layer with many copies, called experts, plus a small "router" that sends each token to just one of them, so the parameter count grows with the number of experts while the work per token stays the same.
Routing to a single expert, instead of the two or more earlier work used, is simpler and cheaper, and three practical tricks (a balancing penalty, higher numerical precision in the router, and smaller starting weights) made it train reliably.
Using the same computing budget per token as Google's T5 models, Switch models reached the same quality up to seven times faster, helped in all 101 languages tested, and were scaled to 1.6 trillion parameters.
Idea · §1–2
More parameters without more work
Bigger language models are better, as the scaling-law papers showed, but in a normal ("dense") model every parameter is used for every token, so a model ten times bigger costs ten times as much to run.
A mixture of experts breaks that link. Take one of the transformer's feed-forward layers (the part of each block that processes each token on its own) and make many copies of it, each with its own weights. These copies are the experts. A small router looks at each token and decides which expert should handle it. Each token only visits its chosen expert, so the work per token stays about the same no matter how many experts exist. The model's total knowledge, held in all those parameters, can grow almost for free. This is called sparse computation: only a small part of the model is active for any one token.
TakeawayExperts plus a router let the number of parameters grow while the computation per token stays fixed.
Idea · §2.1–2.2 · Figs. 2–3
One expert per token, and a capacity limit
Earlier mixture-of-experts models sent each token to at least two experts and blended their answers, believing the router could not learn which expert is good otherwise. The Switch paper simply sends each token to its single highest-scoring expert (and multiplies that expert's output by the router's confidence, which is what lets the router learn). This halves the work of routing and the amount of data that has to move between chips.
There is a catch. The experts sit on different chips, and each chip must reserve a fixed amount of room in advance. So each expert gets a capacity: the number of tokens it can take per batch, equal to the fair share (tokens ÷ experts) times a capacity factor. If too many tokens choose the same expert, the extras are dropped: they skip the expert and pass through unchanged on the layer's shortcut ("residual") connection. A bigger capacity factor drops fewer tokens but wastes room when experts are not full.
TakeawaySwitch routes each token to one expert; each expert has a fixed capacity, and tokens over the limit skip the layer.
Check yourself
- Why can't experts simply take however many tokens arrive?
- The hardware (TPUs) needs tensor sizes fixed before running, so each expert's buffer must be set in advance. Anything beyond it is overflow.
- What happens to a dropped token? Is it lost?
- Not entirely: it skips this one expert layer and continues through the residual connection, so it still reaches the next layer, just without that layer's expert processing.
Idea · §2.2 · Eq. 4
Keeping the experts evenly busy
Left alone, a router tends to fall into favourites: a few experts get most tokens, overflow and drop them, while others sit idle and never learn. Switch adds a small extra term to the training loss, the load-balancing loss, which is smallest when every expert gets the same share of tokens and the router spreads its confidence evenly. It is multiplied by a small weight (0.01) so it nudges the router without overriding the main task.
Toggle the router in the figure above between unbalanced and balanced and watch the dropped tokens and the balancing loss fall.
TakeawayA gentle penalty for uneven routing keeps all experts in use and few tokens dropped.
Idea · §2.4, §3 · Figs. 4–6
Stable training, and how much faster
Sparse models had a reputation for crashing during training. The authors found three cheap fixes. Do the router's arithmetic in full 32-bit precision even though the rest of the model uses 16-bit numbers (the router's exponentials are sensitive to rounding). Start the weights ten times smaller than usual. And when fine-tuning on small tasks, use much stronger dropout inside the experts, which have so many parameters they otherwise overfit.
The payoff: the parameter count rises with each extra expert, while the computation per token stays flat. Compared with T5-Base on the same hardware and with the same computation per token, a Switch-Base model with 64 experts reached T5-Base's quality about seven times faster.
TakeawayWith a few stability tricks, adding experts adds parameters for free per token, and that bought large speed-ups over dense models.
Idea · §4–5 · Table 9
Trillion parameters, and what it led to
The authors combined data, model and expert parallelism to spread models across many chips, and built Switch-C with 2,048 experts and about 1.6 trillion parameters, which reached a fixed quality four times faster than the 11-billion-parameter T5-XXL. The gains carried over to fine-tuned tasks and to 101 languages. Because sparse models are awkward to deploy, they also distilled them into small dense models, which kept about 30% of the quality gain.
Not everything worked: the largest models were sometimes unstable, and the trillion-parameter model was not the best on every downstream task.
TakeawaySwitch showed sparse experts could scale past a trillion parameters with simple top-1 routing; later models kept the idea and refined routing and stability.
Abstract, restated
Mixture-of-experts models select different parameters for each input, giving sparsely activated models with outrageous parameter counts at constant compute, but complexity, communication costs and training instability have limited adoption. The Switch Transformer simplifies MoE routing, reduces communication and compute, and introduces training techniques that address instability, showing that large sparse models can be trained for the first time in lower-precision (bfloat16) formats. Built on T5-Base and T5-Large, it achieves up to 7× pre-training speed-ups with the same computational resources, gains over mT5-Base in all 101 languages, and, using the Colossal Clean Crawled Corpus (C4), trillion-parameter models with a 4× speed-up over T5-XXL. Fedus, Zoph & Shazeer, JMLR 23 (2022) (arXiv 2101.03961, 2021).
Section · §2.1 · Eqs. 1–2
The Switch layer
Following Shazeer et al. (2017), a router with weights Wr produces logits h(x) = Wr·x for a token representation x, normalised over the N experts:
with 𝒯 the top-k experts. Prior work argued k ≥ 2 was needed for non-trivial gradients to the router. Switch uses k = 1: y = pi*(x)Ei*(x), where the gate value pi* still provides a gradient to the router. Claimed benefits: router computation is reduced, each expert's batch (capacity) can be at least halved since each token goes to one expert, and the implementation and communication are simplified. The Switch layer replaces the dense FFN sub-layer of a transformer block; experts are FFNs with independent weights, and attention layers are shared.
Section · §2.2 · Eqs. 3–6
Capacity and the auxiliary load-balancing loss
Overflowing tokens are not computed by any expert in that layer and are passed on via the residual connection. Empirically, the paper reports low drop rates (typically under 1%) with the balancing loss. For a batch ℬ of T tokens and N experts, the auxiliary loss is
fi, the fraction of tokens dispatched to expert i, is not differentiable; Pi, the average router probability, is. Under uniform routing both equal 1/N and the loss is α (the factor N makes it invariant to the number of experts). The gradient flows through Pi and is largest for experts that are already over-used, pushing their probability down. α = 10−2 was chosen from a sweep over 10−1 to 10−5. The loss is added at every Switch layer.
Simulate it below: tokens are routed to experts whose popularity follows a skewed distribution; the readout gives the drop rate, wasted capacity and Eq. 4.
Section · §2.3–2.4 · Tables 1–4
Comparisons and stability techniques
| Model (Table 1; 128 experts) | Capacity factor | Neg. log perp. @100k steps | Time to threshold | Examples / s |
|---|---|---|---|---|
| T5-Base | n/a | −1.731 | not reached | 1600 |
| T5-Large | n/a | −1.550 | 131.1 h | 470 |
| MoE-Base | 2.0 | −1.547 | 68.7 h | 840 |
| Switch-Base | 2.0 | −1.554 | 72.8 h | 860 |
| MoE-Base | 1.25 | −1.559 | 80.7 h | 790 |
| Switch-Base | 1.25 | −1.553 | 65.0 h | 910 |
| MoE-Base | 1.0 | −1.572 | 80.1 h | 860 |
| Switch-Base | 1.0 | −1.561 | 62.8 h | 1000 |
| Switch-Base+ | 1.0 | −1.534 | 67.6 h | 780 |
Switch beats top-2 MoE at equal speed, and does especially well at low capacity factors; Switch-Base+ is a larger Switch model matched to MoE's speed. All runs on 32 TPUv3 cores, trained on C4 with the T5 span-corruption objective.
- Selective precision (Table 2). Router input cast to float32 locally, softmax computed in float32, outputs cast back to bfloat16 before dispatch: quality −1.716 at 1,390 examples/s, vs −1.718 at 1,160 for full float32 and divergence (−3.780) for full bfloat16. The exponentials in the router softmax are the sensitive part.
- Initialisation (Table 3). Truncated normal with σ = √(s/n), scale s reduced from 1.0 to 0.1: average quality −2.72 (std 0.01) vs −3.60 (std 0.68) after 3.5k steps over three seeds.
- Expert dropout (Table 4). Fine-tuning with dropout 0.1 on non-expert layers and 0.4 inside experts gave the best results (GLUE 85.2, CNNDM 19.6, SQuAD 83.7, SuperGLUE 73.0, vs T5-Base 82.9, 19.6, 83.5, 72.4).
Section · §3–5 · Figs. 4–8 · Tables 5–9
Scaling, downstream, distillation, parallelism
- Step scaling (§3.1, Fig. 4). At fixed FLOPs per token, going from the dense T5-Base (223M) through 2 to 256 experts (14.7B parameters) monotonically improves loss at a fixed step count. Switch-Base with 64 experts reaches T5-Base's step-450k quality at step 60k, a 7.5× step speed-up.
- Wall clock (§3.2–3.3, Figs. 5–6). About 7× faster than T5-Base to the same quality; Switch-Base is also 2.5× faster than T5-Large despite T5-Large using 3.5× more FLOPs per token.
- Fine-tuning (§4.1, Table 5). SuperGLUE: Switch-Base 79.5 vs T5-Base 75.1; Switch-Large 84.7 vs T5-Large 82.7. Winogrande and knowledge-heavy closed-book QA benefit strongly; ARC is a mixed case.
- Distillation (§4.2, Tables 6–8). Initialising the dense student with the teacher's non-expert weights and using a 0.25 teacher-probability / 0.75 ground-truth loss mix keeps about 30% of the quality gain of a 3.8B Switch-Base teacher in a 223M student; at 99% compression (14.7B teacher) 28% remains.
- Multilingual (§4.3, Figs. 7–8). mSwitch-Base improves over mT5-Base on all 101 languages, mean step speed-up 5×, with 91% of languages at least 4×.
- Parallelism (§5, Fig. 9). Data, model and expert parallelism are combined in Mesh-TensorFlow; expert parallelism uses all-to-all communication to send tokens to the device holding their expert.
- Trillion scale (§5.6, Table 9). Switch-XXL: 395B parameters, 64 experts, FLOP-matched to T5-XXL. Switch-C: 1,571B parameters, 2,048 experts, expert parallelism only, fewer FLOPs per token. Switch-C reaches T5-XXL's quality 4× faster; Switch-XXL, which has the same FLOPs per token as T5-XXL, leads it by about 0.087 in negative log perplexity at 500k steps (Table 9), but was sometimes unstable.
Section · §6–8 · lineage
Limitations and what came next
- Stability at scale. Switch-XXL remained sometimes unstable; the stability tricks were sufficient for Base/Large but not fully for the largest FLOP-matched models (§8).
- Pre-training ≠ fine-tuning. Switch-C, with better pre-training perplexity, scored 87.7 on SQuAD vs Switch-XXL's 89.6; the relationship between upstream and downstream quality for sparse models is not straightforward.
- Memory. Parameters are cheap in FLOPs but not in memory or communication; serving needs every expert resident, which motivated distillation and later expert offloading.
- Experts in attention. Tried in Appendix A and improved quality, but unstable in bfloat16; left for future work.
- Successors. GShard (Lepikhin et al., 2020) preceded Switch with top-2 routing and capacity; ST-MoE (Zoph et al., 2022) added the router z-loss and a careful fine-tuning study; Expert Choice routing (Zhou et al., 2022) lets experts pick tokens, guaranteeing balance; Mixtral 8x7B uses top-2 of 8; DeepSeekMoE uses fine-grained plus shared experts, and DeepSeek-V3 balances load with learned per-expert biases instead of an auxiliary loss.
Check yourself
- Why does the load-balancing loss multiply fi by Pi rather than just penalising the variance of f?
- f comes from an argmax and has no gradient. Multiplying by the differentiable Pi gives a gradient that lowers router probability most for experts with large fi.
- Why is top-1 routing cheaper in communication than top-2?
- Each token is sent across devices to one expert instead of two, halving the all-to-all traffic and the expert capacity needed.
Reference
Glossary
- Expert
- One of N FFN sub-networks in a Switch layer.
- Router
- Linear layer plus softmax choosing an expert per token.
- Capacity factor
- Multiplier on the even share of tokens per expert that sets each expert's buffer.
- Token dropping
- Overflow tokens skip the expert and pass through the residual.
- Aux loss
- αNΣfiPi, encouraging uniform routing.
- Expert parallelism
- Placing different experts on different devices, with all-to-all token exchange.
- Neg. log perplexity
- The paper's quality metric; higher (closer to 0) is better.