Study guide · Neural networks · 2021

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

William Fedus, Barret Zoph, Noam Shazeer · JMLR

Can a model have far more parameters without costing more to run? A "mixture of experts" model does this by holding many alternative sub-networks and sending each word to only a few of them, but earlier versions were complicated and unstable. The Switch Transformer simplifies the idea to its extreme: send each token to exactly one expert. With a few fixes for stability, this trained up to seven times faster than a comparable dense model on the same hardware and scaled to 1.6 trillion parameters.

The paper in four sentences

In an ordinary transformer every word (token) passes through the same feed-forward layer, so adding parameters always adds computation.

A Switch Transformer replaces that layer with many copies, called experts, plus a small "router" that sends each token to just one of them, so the parameter count grows with the number of experts while the work per token stays the same.

Routing to a single expert, instead of the two or more earlier work used, is simpler and cheaper, and three practical tricks (a balancing penalty, higher numerical precision in the router, and smaller starting weights) made it train reliably.

Using the same computing budget per token as Google's T5 models, Switch models reached the same quality up to seven times faster, helped in all 101 languages tested, and were scaled to 1.6 trillion parameters.

Idea · §1–2

More parameters without more work

Bigger language models are better, as the scaling-law papers showed, but in a normal ("dense") model every parameter is used for every token, so a model ten times bigger costs ten times as much to run.

A mixture of experts breaks that link. Take one of the transformer's feed-forward layers (the part of each block that processes each token on its own) and make many copies of it, each with its own weights. These copies are the experts. A small router looks at each token and decides which expert should handle it. Each token only visits its chosen expert, so the work per token stays about the same no matter how many experts exist. The model's total knowledge, held in all those parameters, can grow almost for free. This is called sparse computation: only a small part of the model is active for any one token.

Everyday pictureA hospital with one general doctor sees every patient with the same person. A hospital with sixty specialists and a receptionist who sends each patient to one of them can know far more medicine, yet each patient still has just one appointment.

TakeawayExperts plus a router let the number of parameters grow while the computation per token stays fixed.

Idea · §2.1–2.2 · Figs. 2–3

One expert per token, and a capacity limit

Earlier mixture-of-experts models sent each token to at least two experts and blended their answers, believing the router could not learn which expert is good otherwise. The Switch paper simply sends each token to its single highest-scoring expert (and multiplies that expert's output by the router's confidence, which is what lets the router learn). This halves the work of routing and the amount of data that has to move between chips.

There is a catch. The experts sit on different chips, and each chip must reserve a fixed amount of room in advance. So each expert gets a capacity: the number of tokens it can take per batch, equal to the fair share (tokens ÷ experts) times a capacity factor. If too many tokens choose the same expert, the extras are dropped: they skip the expert and pass through unchanged on the layer's shortcut ("residual") connection. A bigger capacity factor drops fewer tokens but wastes room when experts are not full.

Twelve tokens, four experts. Each token goes to its highest-scoring expert (lines). Each expert has room for capacity = 12 ÷ 4 × capacity factor tokens (empty boxes are wasted room). Tokens that arrive after an expert is full are dropped (red, dashed) and skip to the next layer unchanged. "Balanced" shows a router after the balancing penalty has done its job; "top-2" shows the older approach where each token visits two experts, doubling the traffic, so it needs twice the capacity to avoid drops. Router scores are made up for illustration.

TakeawaySwitch routes each token to one expert; each expert has a fixed capacity, and tokens over the limit skip the layer.

Check yourself
Why can't experts simply take however many tokens arrive?
The hardware (TPUs) needs tensor sizes fixed before running, so each expert's buffer must be set in advance. Anything beyond it is overflow.
What happens to a dropped token? Is it lost?
Not entirely: it skips this one expert layer and continues through the residual connection, so it still reaches the next layer, just without that layer's expert processing.

Idea · §2.2 · Eq. 4

Keeping the experts evenly busy

Left alone, a router tends to fall into favourites: a few experts get most tokens, overflow and drop them, while others sit idle and never learn. Switch adds a small extra term to the training loss, the load-balancing loss, which is smallest when every expert gets the same share of tokens and the router spreads its confidence evenly. It is multiplied by a small weight (0.01) so it nudges the router without overriding the main task.

Toggle the router in the figure above between unbalanced and balanced and watch the dropped tokens and the balancing loss fall.

TakeawayA gentle penalty for uneven routing keeps all experts in use and few tokens dropped.

Idea · §2.4, §3 · Figs. 4–6

Stable training, and how much faster

Sparse models had a reputation for crashing during training. The authors found three cheap fixes. Do the router's arithmetic in full 32-bit precision even though the rest of the model uses 16-bit numbers (the router's exponentials are sensitive to rounding). Start the weights ten times smaller than usual. And when fine-tuning on small tasks, use much stronger dropout inside the experts, which have so many parameters they otherwise overfit.

The payoff: the parameter count rises with each extra expert, while the computation per token stays flat. Compared with T5-Base on the same hardware and with the same computation per token, a Switch-Base model with 64 experts reached T5-Base's quality about seven times faster.

Switch-Base with 2k experts. Parameter counts are approximate, interpolated between the two sizes the paper gives for this family (223M for the dense T5-Base, 14.7B with 256 experts, so about 57M per extra expert). Computation per token does not change, because each token still visits one expert per layer. The paper's Fig. 4 shows the matching result: at a fixed number of training steps, the test loss keeps falling as experts are added.

TakeawayWith a few stability tricks, adding experts adds parameters for free per token, and that bought large speed-ups over dense models.

Idea · §4–5 · Table 9

Trillion parameters, and what it led to

The authors combined data, model and expert parallelism to spread models across many chips, and built Switch-C with 2,048 experts and about 1.6 trillion parameters, which reached a fixed quality four times faster than the 11-billion-parameter T5-XXL. The gains carried over to fine-tuned tasks and to 101 languages. Because sparse models are awkward to deploy, they also distilled them into small dense models, which kept about 30% of the quality gain.

Not everything worked: the largest models were sometimes unstable, and the trillion-parameter model was not the best on every downstream task.

Later refinementMixture-of-experts layers became mainstream in large language models. ST-MoE (2022) added a "router z-loss" for stability. Many later models went back to routing each token to two or more experts, but with Switch-style capacity and balancing: Mixtral 8x7B (2024) uses top-2 of 8 experts, and DeepSeek's models use many small experts plus always-on "shared" experts, later with a balancing method that needs no extra loss. The core Switch idea, sparse experts to decouple parameters from compute, is now standard.

TakeawaySwitch showed sparse experts could scale past a trillion parameters with simple top-1 routing; later models kept the idea and refined routing and stability.

Follow the idea

Built on: Mixture of experts (2017)