Study guide · Language models & AI · 2023

LLaMA: Open and Efficient Foundation Language Models

Hugo Touvron et al. · arXiv

Do you need a giant, secret model trained on private data to get state-of-the-art results? Meta's LLaMA paper trained four models of 7 to 65 billion parameters only on publicly available text, and trained them on far more data than was usual for their size. The 13-billion-parameter model beat the 175-billion-parameter GPT-3 on most benchmarks, the 65B model rivalled the best models of the day, and the weights were given to researchers, setting off the open-model ecosystem.

The paper in four sentences

The best language models of early 2023 (GPT-3, PaLM, Chinchilla) were either closed or trained partly on data nobody else could get.

LLaMA uses only public sources (filtered web pages, GitHub code, Wikipedia, public-domain books, arXiv papers and Stack Exchange) and an architecture assembled from known improvements.

The key choice was to train relatively small models on a lot of text, up to 1.4 trillion tokens, because a smaller model is cheaper every time someone uses it.

LLaMA-13B outperforms GPT-3 on most benchmarks despite being more than ten times smaller and able to run on a single GPU; LLaMA-65B is competitive with Chinchilla-70B and PaLM-540B, and the models were released to the research community.

Idea · §1

Train smaller, but longer

A language model's size is measured in parameters, the numbers adjusted during training. The amount it reads is measured in tokens, small chunks of text (about three-quarters of a word on average in English). Training cost grows with both: roughly, parameters times tokens.

In 2022 the Chinchilla paper worked out the best split for a fixed training budget, and found earlier models such as GPT-3 were too big for the amount of text they read. LLaMA's authors made a further point: training happens once, but the model is then used millions of times, and every use costs in proportion to its size. So it can be worth training a smaller model for much longer than the "optimal" training budget suggests: for example, Chinchilla's analysis recommends a 10B model on 200B tokens, but LLaMA found a 7B model still improving after 1 trillion tokens.

Play with the trade-off below. Slide the amount of use, and compare total computing cost over the model's life.

Training compute ≈ 6 × parameters × training tokens; using the model ≈ 2 × parameters per generated token. These are standard rough estimates, not figures from the paper; model sizes and token counts are the real ones (GPT-3: 175B parameters, 300B tokens; LLaMA: 13B on 1T and 65B on 1.4T). The slider is a power of ten: 11 means 1011 tokens generated. Blue = training, amber = use; the axis rescales as you slide.
Everyday pictureBuilding a car. A bigger engine is quick to build to a given speed, but you pay for its fuel every day. Spending longer engineering a smaller, efficient engine costs more up front and saves on every journey after.

TakeawayIf a model will be used a lot, it pays to train a smaller one on more data than the training-optimal amount.

Check yourself
Why isn't the "compute-optimal" model size from Chinchilla always the right choice?
It minimises training cost for a given quality, but ignores the cost of running the model afterwards; a smaller model trained longer can reach the same quality and is cheaper at every use.

Idea · §2.1 · Table 1

Only public data

Earlier top models used private datasets, such as unpublished book collections or proprietary web crawls, so nobody outside could reproduce them. LLaMA's training mix was built only from sources others could obtain:

  • Web pages from CommonCrawl (67%), cleaned of duplicates, filtered to English and kept only if they resembled pages cited by Wikipedia; plus the already-cleaned C4 web collection (15%).
  • GitHub code under permissive licences (4.5%), Wikipedia in 20 languages (4.5%), books from Project Gutenberg and the Books3 collection (4.5%), scientific papers from arXiv (2.5%) and Stack Exchange question-and-answer sites (2%).

About 1.4 trillion tokens in total. High-quality sources such as Wikipedia and books were read about twice; most of the rest about once.

Later refinement"Publicly available" is not the same as "free of problems". Books3 turned out to consist of pirated books and became the subject of copyright lawsuits, and later Llama releases disclosed much less about their training data. "Open" for these models came to mean open weights, not open data.

TakeawayCompetitive models can be trained from public data alone, which made the recipe reproducible by others.

Idea · §3 · Tables 3, 9

A 13B model beats the 175B GPT-3

The models were tested on standard benchmarks without further training: common-sense questions ("which is a better way to dry wet shoes?"), sentence completion, science exam questions, and the MMLU exam covering 57 subjects. Pick a benchmark to compare LLaMA with the large closed models of the time.

Real numbers: zero-shot accuracy from Table 3 (common-sense reasoning) and 5-shot MMLU from Table 9. Missing bars are scores the paper does not report. Parameters in brackets. LLaMA-13B beats GPT-3 on most of these despite being about 13 times smaller.

The 65B model was not best at everything. On MMLU, a test of broad knowledge, it scored 63.4 against Chinchilla's 67.5 and PaLM's 69.3. The authors suggest the reason: their mix contains comparatively little book and academic text (177 GB), while those models trained on up to 2 TB of books.

TakeawayData and training length can substitute for size: a model a tenth the size of GPT-3 matched or beat it on most tests.

Check yourself
Why does it matter that LLaMA-13B runs on a single GPU?
Researchers and small companies with one graphics card could run a model at GPT-3's level themselves, instead of paying for access to a closed one.

Idea · §2.2

A transformer with three borrowed upgrades

LLaMA's design is the standard transformer with three changes, each borrowed from an earlier model:

  • Normalise inputs, with RMSNorm. Inside the network, numbers are rescaled to a standard size before each layer to keep training stable (RMSNorm, a simpler and cheaper version of layer normalisation).
  • A gated activation (SwiGLU). The layer that processes each word's information uses a "gate" that lets the network choose how much of each signal to pass.
  • Rotary positions (RoPE). Word order is encoded by rotating internal vectors by an angle that depends on position, so the model naturally attends to how far apart two words are.

None of these was new. The contribution was a clean combination, efficient training code, and the data recipe, which together became the standard template for open models.

TakeawayLLaMA's architecture is a well-chosen bundle of known improvements; most open models since have copied it.

Idea · §5–6 · Release

Released to researchers, then to everyone

Meta offered the weights to researchers under a non-commercial licence, on application. Within about a week they had leaked publicly online. Whatever the intent, the effect was that a strong model was suddenly in everyone's hands, and a wave of work followed: cheap instruction-tuned versions (Stanford's Alpaca, Vicuna), ways to run it on laptops, and memory-saving fine-tuning methods such as QLoRA, which used LLaMA-65B.

The paper itself reports the usual risks honestly: the models reproduce biases from web text, their tendency to produce toxic text rises with size, and they often give untruthful answers. It also reports the carbon cost: training the 65B model took about a million GPU-hours and an estimated 173 tonnes of CO₂-equivalent.

Later refinementLlama 2 (July 2023) added chat-tuned versions and a licence allowing most commercial use; Llama 3 (2024) pushed "train small, long" much further, training an 8B model on about 15 trillion tokens. Other open-weight families (Mistral, Qwen, DeepSeek) adopted essentially the same architecture.

TakeawayLLaMA turned top-tier language models from a closed service into something researchers could download, study and modify.

Follow the idea

Built on: RMSNorm (2019), GPT-3 (2020), RoPE (2021)