Study guide · Machine learning & training · 2021

LoRA: Low-Rank Adaptation of Large Language Models

Edward Hu et al. · ICLR 2022

Adapting a giant language model to a new task normally means retraining all of its billions of numbers and storing a full copy for every task. LoRA asks whether the change really needs that many degrees of freedom, and answers no: freeze the original model and learn the change as the product of two thin matrices. On GPT-3 this cut the trainable parameters by about 10,000 times and the memory needed by three times, matched full fine-tuning in quality, and adds no delay at all when the model is used.

The paper in four sentences

Fine-tuning a pretrained model means nudging all its weights for a new task, which for a 175-billion-parameter model like GPT-3 needs enormous memory and a 350 GB copy for every task.

The authors bet that the change needed for a task is simple, in a precise sense called "low rank": it can be written as one tall thin matrix times one short wide one.

So LoRA keeps the original weights frozen and trains only these two small matrices beside some of them; when training ends, their product can be added straight into the frozen weights, so the adapted model runs exactly as fast as the original.

On GPT-3 and smaller models this matched or beat full fine-tuning on several benchmarks while training about 10,000 times fewer parameters, and it became the standard way to customise large models.

Idea · §1–2

The cost of fine-tuning everything

A large language model is first pretrained on huge amounts of text, then usually fine-tuned: trained a bit more on examples of one task, such as answering questions about a database or summarising conversations. Ordinary fine-tuning adjusts every one of the model's parameters.

For GPT-3, that is 175 billion numbers. Training them needs the model, its gradients and the optimiser's bookkeeping all in GPU memory, about 1.2 terabytes in the paper's setup. And each task produces a new 350-gigabyte copy of the model. A company serving a hundred customised versions would need a hundred full copies.

Earlier shortcuts existed. Adapters insert small extra layers into the network and train only those, but the extra layers have to run one after another, which slows the model down when answering one request at a time. Prefix tuning trains a few special input vectors, but those use up part of the model's input length and turned out to be hard to optimise.

TakeawayFull fine-tuning of a giant model is expensive to train and to store; earlier cheap alternatives cost speed or quality.

Idea · §4.1 · Fig. 1 · Eq. 3

Learn the change as two thin matrices

Inside a transformer, most of the work is done by big square grids of numbers called weight matrices. One of GPT-3's is 12,288 rows by 12,288 columns: about 151 million numbers. Fine-tuning changes each matrix W to W + ΔW, where ΔW (say "delta W") is the change.

LoRA writes the change as a product of two thin matrices, B × A. B is tall and thin (12,288 × r), A is short and wide (r × 12,288), and r, the rank, is small, like 1, 4 or 8. Multiplied together they make a full-size 12,288 × 12,288 change, but they contain only 2 × 12,288 × r numbers. With r = 4 that is about 98,000 numbers instead of 151 million.

During training only A and B change; the original W is frozen. B starts at zero, so at the start the model behaves exactly like the original.

Left: the frozen d × d matrix W. Right: LoRA's B (d × r) and A (r × d), drawn to the same scale (but never thinner than a couple of pixels so you can see them). 12,288 is GPT-3's model width. The count below is for one matrix; the paper adapts two matrices in each of GPT-3's 96 layers.
Everyday pictureImagine adjusting the colour of a huge photo. You could edit every pixel separately, or apply a few global sliders such as "warmer" and "more contrast". Each slider changes every pixel, but you only had to choose a few numbers. LoRA bets that adapting a model to a task is more like moving a few sliders than repainting the picture.

TakeawayLoRA freezes the original weights and learns each change as B × A, which has a tiny fraction of the numbers but still affects the whole matrix.

Check yourself
Why start B at zero rather than starting both matrices random?
So that B × A = 0 at the start, and the adapted model is exactly the pretrained model before training. Training then moves smoothly away from a known good starting point.
If you double r, how does the number of trainable numbers change?
It doubles, because it is 2 × d × r. The full matrix's size, d × d, does not depend on r.

Idea · §4.1 · §7

Why a low-rank change is enough

"Rank" measures how many independent patterns a matrix contains. A rank-1 matrix is just one pattern, a column times a row, like a multiplication table. A rank-4 matrix is four such patterns added together. A full-rank 12,288-wide matrix could contain 12,288 patterns.

Earlier work had found that big pretrained models can be fine-tuned well even when the change is squeezed into a small random subspace (they have a low "intrinsic dimension"). The LoRA authors guessed the same holds for the change to each weight matrix: it has low intrinsic rank. Their experiments supported this strongly. For GPT-3, a rank of 1 or 2 was nearly as good as 64 on the tasks they tested, and the important directions found at rank 8 were mostly the same as those found at rank 64.

Below, a made-up "change" matrix is built from a few patterns plus a little noise. Slide the rank and see how much of it a rank-r product can capture.

Left: a 16 × 16 "update" made of three patterns of decreasing strength plus random noise (amber positive, blue negative). Middle: the best possible rank-r version (computed by singular value decomposition). Right: what is left over. Three patterns are captured by rank 3; beyond that you are only fitting noise. Illustrative; real updates are far larger, but the paper's analysis (§7.2) finds the same kind of concentration in a few directions.

TakeawayThe change a task needs seems to live in a handful of directions, so a very low rank captures almost all of it.

Idea · §4.1–4.2 · Table 4

No slowdown, tiny files, same quality

Because B × A is the same shape as W, after training you can simply add it in: W + BA is one ordinary matrix again. The adapted model has exactly the same size and speed as the original. To switch tasks, subtract one product and add another.

For GPT-3, adapting two matrices per layer with rank 4 made each task's saved changes about 35 megabytes instead of 350 gigabytes, a ten-thousandth of the size. Training needed about a third of the memory (350 GB instead of 1.2 TB) and ran about 25% faster, because no gradients are needed for the frozen weights. On tests such as turning questions into database queries (WikiSQL), recognising whether one sentence follows from another (MultiNLI) and summarising chat conversations (SAMSum), LoRA matched or beat full fine-tuning.

Later refinementLoRA became the default way to customise open models, and many variants followed. QLoRA (2023) stores the frozen model in 4-bit precision, letting a 65-billion-parameter model be fine-tuned on a single 48 GB GPU. DoRA (2024) splits each weight into a size and a direction and trains them separately. Later studies found LoRA tends to learn less than full fine-tuning on very different domains such as code and maths, but also forgets less of what the model already knew.

TakeawayLoRA updates merge into the original weights, so they cost nothing at run time, take megabytes to store, and matched full fine-tuning on the paper's tests.

Check yourself
Why do adapters slow a model down while LoRA does not?
Adapters are extra layers that must be computed in sequence on every request. LoRA's change has the same shape as an existing weight matrix, so it can be added into that matrix once and disappear.
What is the catch of merging?
A merged model serves one task. To mix requests for different tasks in a single batch, you keep B and A separate and pay a small extra cost.

Follow the idea

Led to: QLoRA (2023)