Study guide · Neural networks · 2017

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Noam Shazeer et al. · ICLR 2017

Bigger neural networks get better results, but normally every parameter is used for every input, so compute grows as fast as size. Shazeer and colleagues build a layer of thousands of small "expert" networks with a learned router that sends each input to only a few of them. The model can hold over a hundred billion parameters while each word still costs only a small, fixed amount of computation, and it beat the best language-modelling and translation results of the time.

The paper in four sentences

A mixture-of-experts layer contains many separate small networks (experts) plus a gating network, or router, that looks at each input and picks a handful of experts to handle it.

Only the chosen experts run, so you can add more experts, and with them more knowledge, without making each input more expensive to process. The paper calls this conditional computation.

Making it work needed engineering: noise and extra training terms so the router spreads work evenly instead of favouring a few experts, and clever batching across many GPUs so each expert gets enough inputs at once to run efficiently.

With up to 137 billion parameters in one layer, these models beat the best published language-modelling and translation results with less computation per word. The same design powers many of today's largest language models.

Idea · §1, §1.1

Size helps, but cost grows with it

By 2016 it was clear that neural networks improve with more data and more parameters (the adjustable numbers inside them). The problem is that in an ordinary network every parameter takes part in processing every input. Double the parameters and you double the work per word. Training cost grows roughly with model size times data size, which quickly becomes unaffordable.

An old idea offered a way out: conditional computation, where only part of the network is active for any particular input. A sentence about chemistry might use different parts from a sentence about football. Then the network could be enormous while each input still uses only a small slice. The idea had been proposed before but never delivered big gains in practice, mainly because computer hardware is built for large, uniform blocks of arithmetic, not for "this input goes here, that one goes there".

Everyday pictureA large hospital employs hundreds of specialists, but each patient sees only two or three of them, chosen by a triage nurse. The hospital's total expertise can grow far faster than the time any one patient spends being treated.

TakeawayOrdinary networks pay for every parameter on every input. Conditional computation would let capacity grow without the cost per input growing with it.

Idea · §2 · Fig. 1

Experts and a router

The paper's layer has n experts, each a small, ordinary feed-forward network, all with the same shape but their own parameters. A gating network looks at the input and gives each expert a score. Only the top k scorers (the paper mostly uses k = 4, or 2 at each level of a two-level version) are run. Their outputs are combined as a weighted average, with weights from the scores. The other experts do nothing for this input.

The layer sits between the layers of a recurrent language model and is applied at every word position separately, so every word can be sent to a different set of experts. In the toy below, each word has a fixed (made-up) description, and the router scores 16 experts for it.

Top: the router's score for each of 16 experts for the chosen word; the top k (amber) are selected. Bottom: the chosen experts' weights in the final mix (they add to 1). Experts not chosen are skipped entirely. Similar words tend to pick overlapping experts. Toy router with random weights; noise is explained in the next idea.

TakeawayA router scores every expert, keeps the top few, and mixes their outputs; the rest of the layer stays idle for that input.

Check yourself
With 4,096 experts and k = 4, what fraction of the layer's experts does one word use?
4/4,096, about 0.1%. The layer's parameters grow with the number of experts; the work per word grows only with k.

Idea · §2.1, §4 · Appendix A

Stopping the rich from getting richer

Left alone, the router tends to collapse onto a few favourites. Experts that get chosen early receive more training, so they get better, so they get chosen more. Soon most experts sit idle: wasted parameters, and the busy ones become a bottleneck.

The paper uses two fixes. First, it adds a little random noise to the router's scores during training, so near-ties are broken in different ways and more experts get a turn. Second, it adds an extra training term that penalises imbalance: it measures how unequal the experts' total shares of the work are across a batch and nudges the router to even them out. (A second, similar term targets the number of inputs each expert receives.)

The simulation below shows the effect. Tokens arrive from eight different topics. Each time an expert is chosen it improves a little, and the router learns to favour better experts. Switch the balancing term on or off and watch the share of tokens each expert handles.

Bars: the share of recent tokens each of 16 experts handled (dashed line: a perfectly even share, 1/16 of all choices). Each batch of 64 tokens is routed with k = 2 and noisy gating; chosen experts gain "skill" (which slowly fades when unused), and the router is rewarded for picking skilled experts: the rich-get-richer loop. The balancing term adds the gradient of a squared coefficient of variation of each expert's total router probability over the batch, a smooth stand-in for the paper's importance and load losses. Illustrative toy, not the paper's training setup.

TakeawayRouters naturally collapse onto a few experts; noise and a penalty on uneven usage keep all experts busy and learning.

Check yourself
Why is collapse self-reinforcing?
Chosen experts receive gradient updates and improve; unchosen ones don't. The router then sees the chosen ones performing better and picks them even more often.

Idea · §3, §5 · Table 1 · Figs. 2–3

Making it fast, and what it achieved

Sparsity creates a practical problem: if a batch of examples is split among thousands of experts, each expert sees only a few, and a computer chip doing arithmetic on tiny batches wastes most of its power. The authors fix this by spreading the experts across many GPUs and having every copy of the rest of the model send its inputs to the same shared experts, so each expert collects a decent-sized batch from all of them at once.

Results, on a standard one-billion-word language-modelling benchmark: models with about 4 billion parameters beat the previous best published result, even the cheapest using only a fraction of its computation per word. On a far larger 100-billion-word news corpus, a model with 65,536 experts (about 68 billion parameters in the expert layer) reached 39% lower perplexity (a measure of prediction error) than a model with the same computation per word but no experts. Bigger still, 131,072 experts and 137 billion parameters, was slightly worse, which the authors suspect was too much sparsity. In machine translation, mixture-of-experts models beat Google's production system of the time on English to French and English to German, with less computation per word and less training time.

Later refinementThe Switch Transformer (2021) put mixture-of-experts layers inside transformers and routed each token to just one expert, simplifying the system and scaling past a trillion parameters. Today many large language models, including Mixtral and DeepSeek's models, are mixtures of experts.

TakeawayWith the batching tricks in place, adding experts bought large quality gains at almost constant cost per word, up to a point.

Check yourself
Why do sparse experts create a "shrinking batch" problem?
A batch is divided among many experts, so each expert gets only about k/n of it, and small batches use hardware inefficiently.

Follow the idea

Led to: Switch Transformer (2021)