The paper in four sentences
The best language models of early 2023 (GPT-3, PaLM, Chinchilla) were either closed or trained partly on data nobody else could get.
LLaMA uses only public sources (filtered web pages, GitHub code, Wikipedia, public-domain books, arXiv papers and Stack Exchange) and an architecture assembled from known improvements.
The key choice was to train relatively small models on a lot of text, up to 1.4 trillion tokens, because a smaller model is cheaper every time someone uses it.
LLaMA-13B outperforms GPT-3 on most benchmarks despite being more than ten times smaller and able to run on a single GPU; LLaMA-65B is competitive with Chinchilla-70B and PaLM-540B, and the models were released to the research community.
Idea · §1
Train smaller, but longer
A language model's size is measured in parameters, the numbers adjusted during training. The amount it reads is measured in tokens, small chunks of text (about three-quarters of a word on average in English). Training cost grows with both: roughly, parameters times tokens.
In 2022 the Chinchilla paper worked out the best split for a fixed training budget, and found earlier models such as GPT-3 were too big for the amount of text they read. LLaMA's authors made a further point: training happens once, but the model is then used millions of times, and every use costs in proportion to its size. So it can be worth training a smaller model for much longer than the "optimal" training budget suggests: for example, Chinchilla's analysis recommends a 10B model on 200B tokens, but LLaMA found a 7B model still improving after 1 trillion tokens.
Play with the trade-off below. Slide the amount of use, and compare total computing cost over the model's life.
TakeawayIf a model will be used a lot, it pays to train a smaller one on more data than the training-optimal amount.
Check yourself
- Why isn't the "compute-optimal" model size from Chinchilla always the right choice?
- It minimises training cost for a given quality, but ignores the cost of running the model afterwards; a smaller model trained longer can reach the same quality and is cheaper at every use.
Idea · §2.1 · Table 1
Only public data
Earlier top models used private datasets, such as unpublished book collections or proprietary web crawls, so nobody outside could reproduce them. LLaMA's training mix was built only from sources others could obtain:
- Web pages from CommonCrawl (67%), cleaned of duplicates, filtered to English and kept only if they resembled pages cited by Wikipedia; plus the already-cleaned C4 web collection (15%).
- GitHub code under permissive licences (4.5%), Wikipedia in 20 languages (4.5%), books from Project Gutenberg and the Books3 collection (4.5%), scientific papers from arXiv (2.5%) and Stack Exchange question-and-answer sites (2%).
About 1.4 trillion tokens in total. High-quality sources such as Wikipedia and books were read about twice; most of the rest about once.
TakeawayCompetitive models can be trained from public data alone, which made the recipe reproducible by others.
Idea · §3 · Tables 3, 9
A 13B model beats the 175B GPT-3
The models were tested on standard benchmarks without further training: common-sense questions ("which is a better way to dry wet shoes?"), sentence completion, science exam questions, and the MMLU exam covering 57 subjects. Pick a benchmark to compare LLaMA with the large closed models of the time.
The 65B model was not best at everything. On MMLU, a test of broad knowledge, it scored 63.4 against Chinchilla's 67.5 and PaLM's 69.3. The authors suggest the reason: their mix contains comparatively little book and academic text (177 GB), while those models trained on up to 2 TB of books.
TakeawayData and training length can substitute for size: a model a tenth the size of GPT-3 matched or beat it on most tests.
Check yourself
- Why does it matter that LLaMA-13B runs on a single GPU?
- Researchers and small companies with one graphics card could run a model at GPT-3's level themselves, instead of paying for access to a closed one.
Idea · §2.2
A transformer with three borrowed upgrades
LLaMA's design is the standard transformer with three changes, each borrowed from an earlier model:
- Normalise inputs, with RMSNorm. Inside the network, numbers are rescaled to a standard size before each layer to keep training stable (RMSNorm, a simpler and cheaper version of layer normalisation).
- A gated activation (SwiGLU). The layer that processes each word's information uses a "gate" that lets the network choose how much of each signal to pass.
- Rotary positions (RoPE). Word order is encoded by rotating internal vectors by an angle that depends on position, so the model naturally attends to how far apart two words are.
None of these was new. The contribution was a clean combination, efficient training code, and the data recipe, which together became the standard template for open models.
TakeawayLLaMA's architecture is a well-chosen bundle of known improvements; most open models since have copied it.
Idea · §5–6 · Release
Released to researchers, then to everyone
Meta offered the weights to researchers under a non-commercial licence, on application. Within about a week they had leaked publicly online. Whatever the intent, the effect was that a strong model was suddenly in everyone's hands, and a wave of work followed: cheap instruction-tuned versions (Stanford's Alpaca, Vicuna), ways to run it on laptops, and memory-saving fine-tuning methods such as QLoRA, which used LLaMA-65B.
The paper itself reports the usual risks honestly: the models reproduce biases from web text, their tendency to produce toxic text rises with size, and they often give untruthful answers. It also reports the carbon cost: training the 65B model took about a million GPU-hours and an estimated 173 tonnes of CO₂-equivalent.
TakeawayLLaMA turned top-tier language models from a closed service into something researchers could download, study and modify.
Abstract, restated
LLaMA is a collection of foundation language models from 7B to 65B parameters, trained on trillions of tokens using publicly available datasets exclusively, showing that state-of-the-art models can be trained without proprietary or inaccessible data. LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. All models were released to the research community. Touvron et al., Meta AI, arXiv 2302.13971, February 2023.
Section · §1 · §2.1 · Table 1
Objective and data
The stated objective is the best performance at various inference budgets rather than the best performance for a training budget. Hoffmann et al. recommend training a 10B model on 200B tokens; the authors find a 7B model continues to improve after 1T tokens. Hence 7B and 13B are trained on 1.0T tokens and 33B and 65B on 1.4T.
| Source | Sampling | Epochs | Disk | Processing |
|---|---|---|---|---|
| CommonCrawl | 67.0% | 1.10 | 3.3 TB | Five dumps 2017–2020, CCNet pipeline: line dedup, fastText language ID, n-gram LM quality filter, linear classifier for pages resembling Wikipedia references |
| C4 | 15.0% | 1.06 | 783 GB | Dedup, language ID, heuristic quality filters |
| GitHub | 4.5% | 0.64 | 328 GB | Apache, BSD and MIT licences; heuristic filters; file-level exact dedup |
| Wikipedia | 4.5% | 2.45 | 83 GB | June–August 2022 dumps, 20 languages |
| Books | 4.5% | 2.23 | 85 GB | Gutenberg and Books3; dedup of books with >90% content overlap |
| ArXiv | 2.5% | 1.06 | 92 GB | LaTeX; preamble and bibliography removed |
| StackExchange | 2.0% | 1.03 | 78 GB | 28 largest sites; answers sorted by score |
Tokenisation is BPE via SentencePiece, splitting all numbers into individual digits and falling back to bytes for unknown UTF-8 characters. Epochs are for the 1.4T-token run; apart from Wikipedia and Books, each token is seen about once.
Section · §2.2 · Table 2
Architecture and shapes
- Pre-normalisation (from GPT-3): normalise the input of each sub-layer, using RMSNorm (Zhang & Sennrich, 2019), which drops LayerNorm's mean-centring and bias.
- SwiGLU (from PaLM; Shazeer, 2020) replaces ReLU, with hidden dimension 2/3 · 4d instead of 4d, which keeps the three-matrix feed-forward at the parameter count of the two-matrix one.
- Rotary embeddings (from GPTNeo; Su et al., 2021) at every layer instead of absolute positional embeddings.
| Params | d | Heads | Layers | LR | Batch | Tokens |
|---|---|---|---|---|---|---|
| 6.7B | 4096 | 32 | 32 | 3.0e−4 | 4M | 1.0T |
| 13.0B | 5120 | 40 | 40 | 3.0e−4 | 4M | 1.0T |
| 32.5B | 6656 | 52 | 60 | 1.5e−4 | 4M | 1.4T |
| 65.2B | 8192 | 64 | 80 | 1.5e−4 | 4M | 1.4T |
Below, compare the two normalisations on a five-number activation vector. RMSNorm only rescales; LayerNorm also subtracts the mean. When the mean is near zero, as in many trained activations, they nearly coincide, which is why the cheaper one suffices.
Section · §2.3–2.4
Optimisation and efficiency
AdamW (Loshchilov & Hutter) with β1 = 0.9, β2 = 0.95; cosine learning-rate schedule ending at 10% of the maximum; weight decay 0.1; gradient clipping 1.0; 2,000 warm-up steps; batch 4M tokens for all sizes.
Efficiency: a causal multi-head attention implementation from the xformers library that neither stores attention weights nor computes masked key/query scores (inspired by Rabe & Staats and the backward pass of FlashAttention); activation checkpointing with a hand-written transformer backward pass, saving expensive activations such as linear-layer outputs; model and sequence parallelism; overlapping computation with GPU communication. The 65B model processes about 380 tokens/s/GPU on 2,048 A100 80GB GPUs, so 1.4T tokens take about 21 days.
Section · §3–6 · Tables 3–15
Results, instruction tuning, bias and carbon
- Common sense (Table 3, zero-shot). LLaMA-65B beats Chinchilla-70B on all reported benchmarks except BoolQ, and PaLM-540B everywhere except BoolQ and WinoGrande. LLaMA-13B beats GPT-3 on most.
- Closed-book QA (Tables 4–5). LLaMA-65B: NaturalQuestions 23.8 / 31.0 / 35.0 / 39.9 exact match at 0 / 1 / 5 / 64 shots; TriviaQA 68.2 / 71.6 / 72.6 / 73.0.
- Maths (Table 7). GSM8k 50.9, or 69.7 with majority voting over k samples, which beats Minerva-62B with voting (68.5) but not Minerva-540B (68.5 without voting, 78.5 with), and without any maths-specific fine-tuning.
- Code (Table 8). HumanEval pass@1 23.7 vs PaLM-540B's 26.2.
- MMLU (Table 9, 5-shot). GPT-3 43.9, Gopher 60.0, LLaMA-13B 46.9, LLaMA-33B 57.8, LLaMA-65B 63.4, Chinchilla 67.5, PaLM 69.3; attributed to only 177 GB of books and academic text.
- Training dynamics (§3.7). Performance improves steadily during training and tracks training perplexity, except on SIQA and WinoGrande.
- Instruction tuning (§4, Table 10). A brief fine-tune on instruction data gives LLaMA-I 65B 68.9 on MMLU, above other moderate-size instruction-tuned models (Flan-PaLM-cont 62B 66.1) but below code-davinci-002's 77.4.
- Bias and truthfulness (§5). Toxicity on RealToxicityPrompts rises with size (65B: 0.128 basic, 0.141 respectful); CrowS-Pairs average 66.6 (GPT-3 67.2, OPT 69.5); TruthfulQA truthful 0.57 and truthful×informative 0.53 for 65B, versus 0.28 / 0.25 for GPT-3, still prone to hallucination.
- Carbon (§6, Table 15). 65B: 1,022,362 A100 GPU-hours, 449 MWh, 173 tCO₂eq (US-average carbon intensity); all models developed about 2,638 MWh and 1,015 tCO₂eq.
Section · Limitations and lineage
Limitations and what followed
- Not compute-optimal by design. Over-training trades training efficiency for inference efficiency; the paper does not fit a scaling law for this trade-off.
- Data provenance. "Publicly available" included Books3, later at the centre of copyright litigation; the corpus itself was not released.
- Evaluation. Mostly zero/few-shot multiple-choice benchmarks, some with known contamination risk; little evaluation of long-context, multilingual or dialogue ability.
- Base model only. No safety tuning; only a short instruction-tuning experiment. The paper's own bias and toxicity measurements show the costs.
Check yourself
- Why is SwiGLU's hidden size 8d/3 rather than 4d?
- SwiGLU uses three weight matrices instead of two; shrinking the hidden size by 2/3 keeps parameters and compute equal to the standard 4d ReLU block (3 · d · 8d/3 = 8d2).
- What does RMSNorm give up relative to LayerNorm?
- Re-centring (shift invariance) and the bias term; it keeps scale invariance, which the RMSNorm paper argued is what matters, at lower cost.
Reference
Glossary
- Token
- A unit of text from the BPE tokenizer, typically a word piece.
- Inference budget
- The compute spent using a model after training.
- Pre-normalisation
- Normalising each sub-layer's input rather than its output.
- RMSNorm
- Normalisation by root-mean-square only, without mean subtraction.
- SwiGLU
- Gated feed-forward: Swish(xW) ⊙ xV, then a projection.
- RoPE
- Rotary positional embedding: rotate query/key pairs by position-dependent angles.
- Over-training
- Training on more tokens than the compute-optimal amount for the model size.
- tCO₂eq
- Tonnes of CO₂-equivalent emissions.