The paper in four sentences
Fine-tuning a pretrained model means nudging all its weights for a new task, which for a 175-billion-parameter model like GPT-3 needs enormous memory and a 350 GB copy for every task.
The authors bet that the change needed for a task is simple, in a precise sense called "low rank": it can be written as one tall thin matrix times one short wide one.
So LoRA keeps the original weights frozen and trains only these two small matrices beside some of them; when training ends, their product can be added straight into the frozen weights, so the adapted model runs exactly as fast as the original.
On GPT-3 and smaller models this matched or beat full fine-tuning on several benchmarks while training about 10,000 times fewer parameters, and it became the standard way to customise large models.
Idea · §1–2
The cost of fine-tuning everything
A large language model is first pretrained on huge amounts of text, then usually fine-tuned: trained a bit more on examples of one task, such as answering questions about a database or summarising conversations. Ordinary fine-tuning adjusts every one of the model's parameters.
For GPT-3, that is 175 billion numbers. Training them needs the model, its gradients and the optimiser's bookkeeping all in GPU memory, about 1.2 terabytes in the paper's setup. And each task produces a new 350-gigabyte copy of the model. A company serving a hundred customised versions would need a hundred full copies.
Earlier shortcuts existed. Adapters insert small extra layers into the network and train only those, but the extra layers have to run one after another, which slows the model down when answering one request at a time. Prefix tuning trains a few special input vectors, but those use up part of the model's input length and turned out to be hard to optimise.
TakeawayFull fine-tuning of a giant model is expensive to train and to store; earlier cheap alternatives cost speed or quality.
Idea · §4.1 · Fig. 1 · Eq. 3
Learn the change as two thin matrices
Inside a transformer, most of the work is done by big square grids of numbers called weight matrices. One of GPT-3's is 12,288 rows by 12,288 columns: about 151 million numbers. Fine-tuning changes each matrix W to W + ΔW, where ΔW (say "delta W") is the change.
LoRA writes the change as a product of two thin matrices, B × A. B is tall and thin (12,288 × r), A is short and wide (r × 12,288), and r, the rank, is small, like 1, 4 or 8. Multiplied together they make a full-size 12,288 × 12,288 change, but they contain only 2 × 12,288 × r numbers. With r = 4 that is about 98,000 numbers instead of 151 million.
During training only A and B change; the original W is frozen. B starts at zero, so at the start the model behaves exactly like the original.
TakeawayLoRA freezes the original weights and learns each change as B × A, which has a tiny fraction of the numbers but still affects the whole matrix.
Check yourself
- Why start B at zero rather than starting both matrices random?
- So that B × A = 0 at the start, and the adapted model is exactly the pretrained model before training. Training then moves smoothly away from a known good starting point.
- If you double r, how does the number of trainable numbers change?
- It doubles, because it is 2 × d × r. The full matrix's size, d × d, does not depend on r.
Idea · §4.1 · §7
Why a low-rank change is enough
"Rank" measures how many independent patterns a matrix contains. A rank-1 matrix is just one pattern, a column times a row, like a multiplication table. A rank-4 matrix is four such patterns added together. A full-rank 12,288-wide matrix could contain 12,288 patterns.
Earlier work had found that big pretrained models can be fine-tuned well even when the change is squeezed into a small random subspace (they have a low "intrinsic dimension"). The LoRA authors guessed the same holds for the change to each weight matrix: it has low intrinsic rank. Their experiments supported this strongly. For GPT-3, a rank of 1 or 2 was nearly as good as 64 on the tasks they tested, and the important directions found at rank 8 were mostly the same as those found at rank 64.
Below, a made-up "change" matrix is built from a few patterns plus a little noise. Slide the rank and see how much of it a rank-r product can capture.
TakeawayThe change a task needs seems to live in a handful of directions, so a very low rank captures almost all of it.
Idea · §4.1–4.2 · Table 4
No slowdown, tiny files, same quality
Because B × A is the same shape as W, after training you can simply add it in: W + BA is one ordinary matrix again. The adapted model has exactly the same size and speed as the original. To switch tasks, subtract one product and add another.
For GPT-3, adapting two matrices per layer with rank 4 made each task's saved changes about 35 megabytes instead of 350 gigabytes, a ten-thousandth of the size. Training needed about a third of the memory (350 GB instead of 1.2 TB) and ran about 25% faster, because no gradients are needed for the frozen weights. On tests such as turning questions into database queries (WikiSQL), recognising whether one sentence follows from another (MultiNLI) and summarising chat conversations (SAMSum), LoRA matched or beat full fine-tuning.
TakeawayLoRA updates merge into the original weights, so they cost nothing at run time, take megabytes to store, and matched full fine-tuning on the paper's tests.
Check yourself
- Why do adapters slow a model down while LoRA does not?
- Adapters are extra layers that must be computed in sequence on every request. LoRA's change has the same shape as an existing weight matrix, so it can be added into that matrix once and disappear.
- What is the catch of merging?
- A merged model serves one task. To mix requests for different tasks in a single batch, you keep B and A separate and pay a small extra cost.
Abstract, restated
As models grow, full fine-tuning, which retrains all parameters, becomes less feasible; deploying independent fine-tuned instances of GPT-3 175B is prohibitively expensive. Low-Rank Adaptation freezes the pretrained weights and injects trainable rank-decomposition matrices into each layer of the transformer, greatly reducing the number of trainable parameters for downstream tasks. Compared with GPT-3 175B fine-tuned with Adam, LoRA reduces trainable parameters by 10,000× and GPU memory by 3×. It performs on par with or better than fine-tuning on RoBERTa, DeBERTa, GPT-2 and GPT-3, despite fewer trainable parameters, higher training throughput and, unlike adapters, no additional inference latency. An empirical study of rank deficiency in adaptation sheds light on its efficacy. Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang & Chen (Microsoft), ICLR 2022 (arXiv 2106.09685).
Section · §2 · Eqs. 1–2
Problem statement
Full fine-tuning of an autoregressive model PΦ(y|x) on context–target pairs 𝒵 = {(xi, yi)} solves
starting from Φ0 and learning ΔΦ with |ΔΦ| = |Φ0|. LoRA instead encodes the increment as ΔΦ = ΔΦ(Θ) with |Θ| ≪ |Φ0|:
§3 argues existing methods have drawbacks: adapter layers (Houlsby et al., 2019; Lin et al., 2020) add sequential depth whose latency matters at small batch sizes (Table 1: on GPT-2 medium with batch 1 and sequence length 128, +20.7% to +30.3% forward latency), and prefix tuning is hard to optimise, with performance changing non-monotonically in the number of trainable parameters, and it reserves part of the sequence length.
Section · §4.1–4.2 · Eq. 3
The reparametrisation
For a pretrained weight W0 ∈ ℝd×k, constrain its update to ΔW = BA with B ∈ ℝd×r, A ∈ ℝr×k, r ≪ min(d, k). W0 receives no gradient. The forward pass becomes
Initialisation. A random Gaussian, B = 0, so ΔW = 0 at the start. Scaling. ΔWx is scaled by α/r; with Adam, tuning α is roughly like tuning the learning rate, so the authors set α to the first r they try and do not tune it, which reduces the need to retune when r changes. Generality. As r approaches the rank of W0, LoRA recovers the expressiveness of full fine-tuning; adapter- and prefix-based methods instead converge to a smaller model or one that cannot take long inputs. No inference latency. Deploy W = W0 + BA; switch tasks by subtracting BA and adding B′A′.
Where to apply it (§4.2). Only self-attention weights Wq, Wk, Wv, Wo are considered, each treated as a single dmodel × dmodel matrix even though the output is split into heads; MLP layers, LayerNorm and biases are frozen. Most experiments adapt only Wq and Wv. The trainable count is |Θ| = 2 × L̂LoRA × dmodel × r, with L̂LoRA the number of adapted matrices (§5.1).
Section · §5 · Tables 2–4
Results
GLUE (Table 2). Average scores: RoBERTa-base FT 86.4 vs LoRA (0.3M trainable) 87.2; RoBERTa-large FT 88.9 vs LoRA (0.8M) 89.0; DeBERTa XXL FT 91.1 vs LoRA (4.7M) 91.3. GPT-2 medium and large on E2E NLG, WebNLG and DART (Table 3) generally favour LoRA over adapters and prefix methods at similar or larger budgets.
| GPT-3 175B (Table 4) | Trainable | WikiSQL acc. | MNLI-m acc. | SAMSum R1/R2/RL |
|---|---|---|---|---|
| Fine-tune | 175,255.8M | 73.8 | 89.5 | 52.0/28.0/44.5 |
| BitFit | 14.2M | 71.3 | 91.0 | 51.3/27.4/43.5 |
| PreEmbed | 3.2M | 63.1 | 88.6 | 48.3/24.2/40.5 |
| PreLayer | 20.2M | 70.1 | 89.5 | 50.8/27.3/43.5 |
| AdapterH | 7.1M | 71.9 | 89.8 | 53.0/28.9/44.8 |
| AdapterH | 40.1M | 73.2 | 91.5 | 53.2/29.0/45.1 |
| LoRA | 4.7M | 73.4 | 91.7 | 53.8/29.8/45.9 |
| LoRA | 37.7M | 74.0 | 91.6 | 53.4/29.2/45.1 |
Efficiency (§4.2): training VRAM 1.2 TB → 350 GB; checkpoint ≈ 350 GB → 35 MB (r = 4, Wq, Wv); about 25% faster training on GPT-3 175B (43.1 vs 32.5 tokens/s per V100). The paper also notes that prefix-based methods degrade when given too many special tokens (Fig. 2), whereas LoRA scales well with more trainable parameters.
Section · §7 · Tables 5–7 · Figs. 3–4
Understanding the low-rank updates
- Which matrices (Table 5). With a fixed budget of 18M parameters on GPT-3: putting it all in Wq (r = 8) gives 70.4 WikiSQL / 91.0 MNLI; Wk 70.0 / 90.8; Wv 73.0 / 91.0; Wo 73.2 / 91.3; Wq + Wv (r = 4) 73.7 / 91.3; all four (r = 2) 73.7 / 91.7. Spreading rank across more matrices beats concentrating it.
- Which rank (Table 6). Wq + Wv on WikiSQL: 73.4, 73.3, 73.7, 73.8, 73.5 for r = 1, 2, 4, 8, 64; MNLI 91.3, 91.4, 91.3, 91.6, 91.4. Adapting Wq alone benefits more from higher rank.
- Subspace similarity (Eq. 4, Fig. 3). With UAi the top-i right singular vectors of a learned A, φ(Ar=8, Ar=64, i, j) = ‖Ui⊤A8UjA64‖F2/min(i, j) ∈ [0, 1]. The top singular direction overlaps strongly (φ > 0.5), the rest much less, so the extra directions at r = 64 are mostly noise. Across random seeds (Fig. 4), ΔWq shares more directions than ΔWv, suggesting a higher intrinsic rank for ΔWq.
- Relation to W (Table 7). Project Wq onto the r-dimensional subspace of ΔWq, ‖U⊤WqV⊤‖F. At layer 48 with r = 4 this is 0.32, versus 21.67 for Wq's own top-4 directions and 0.02 for random ones, while ‖ΔWq‖F = 6.91 and ‖Wq‖F = 61.95. So ΔW is correlated with W more than chance, but amplifies directions that W does not emphasise, by a factor of about 6.91/0.32 ≈ 21.5. The authors read this as LoRA amplifying task-specific features already latent in the pretrained model.
The figure below reproduces the qualitative rank result on a toy regression: a frozen random W0, a target W* = W0 + Δ with a true rank-3 Δ, and LoRA trained with B = 0, Gaussian A, Adam and α/r scaling.
Section · §4.2, §8 · lineage
Limitations and what came next
- Batching tasks. Merged weights serve one task; batching inputs for different tasks with different A, B in one forward pass is not straightforward (§4.2). Unmerged multi-adapter serving systems (e.g. S-LoRA, Punica) later addressed this.
- Heuristic choices. Which matrices to adapt and which rank to use are set by hand; the paper only studies attention matrices and leaves MLP, LayerNorm and bias adaptation to future work (§8). Practice later moved to adapting all linear layers, including the MLP.
- Scaling. α/r makes the update magnitude shrink with r at fixed learning rate; rsLoRA (Kalajdzievski, 2023) argues for α/√r so higher ranks actually help. LoRA+ (Hayou et al., 2024) uses a larger learning rate for B than A.
- Capacity. Biderman et al. (2024, "LoRA learns less and forgets less") found LoRA underperforms full fine-tuning on large continued-pretraining-style shifts (code, maths), with full fine-tuning learning updates of much higher rank, while LoRA better preserves the base model's other abilities.
- Descendants. AdaLoRA (adaptive rank allocation), QLoRA (4-bit NF4 frozen base, paged optimisers), DoRA (magnitude/direction decomposition), VeRA (shared random matrices with trainable scalings).
Check yourself
- Why does LoRA reduce GPU memory by about 3× rather than by 10,000×?
- The frozen weights and the activations still have to be in memory for forward and backward passes. What disappears is the optimiser state and gradients for the frozen weights, which for Adam are several times the size of the weights themselves.
- Table 5 shows Wq + Wv at r = 4 beating Wq alone at r = 8 for the same budget. What does that suggest?
- That a small rank per matrix is already enough, so the budget is better spent touching more weight types than adding rank to one.
Reference
Glossary
- Rank r
- Inner dimension of BA; upper bound on the rank of ΔW.
- α
- Constant in the α/r scaling of the update.
- Intrinsic dimension
- Smallest subspace dimension in which fine-tuning reaches a target performance (Aghajanyan et al., 2020).
- Adapter
- Small bottleneck layer inserted into each block; adds sequential computation.
- Prefix tuning
- Training continuous vectors prepended to the input or every layer's activations.
- BitFit
- Fine-tuning only bias vectors.
- Merge
- Computing W0 + BA once so inference uses a single matrix.