The paper in four sentences
Fine-tuning a 65-billion-parameter model the ordinary way needs more than 780 GB of GPU memory, far beyond any single card.
QLoRA freezes the original model, squeezes each of its numbers into 4 bits using a clever set of 16 allowed values ("NormalFloat"), and trains only small LoRA adapters bolted on top, which stay at higher precision.
Two extra tricks, compressing the compression's own bookkeeping ("double quantization") and spilling memory spikes over to ordinary computer RAM ("paged optimizers"), bring the 65B model under 48 GB.
Models trained this way match 16-bit fine-tuning on the benchmarks tested, and the resulting chatbot, Guanaco, scored 99.3% of ChatGPT's level on the Vicuna benchmark after about a day on one GPU.
Idea · §1–2
The memory wall
A language model is, at heart, a huge list of numbers called parameters (or weights). LLaMA-65B has about 65 billion of them. Each one is normally stored in 16 bits, which is 2 bytes, so just holding the model takes about 130 GB.
Fine-tuning means continuing to train a model on new examples so it learns a new skill, such as following instructions politely. Ordinary fine-tuning needs much more than the model itself: for every parameter you also store its gradient (which way to nudge it) and the optimiser's running averages. The paper puts the total for the 65B model at more than 780 GB. The biggest single GPUs of the time had 80 GB.
So fine-tuning big models was something only well-funded labs could do. QLoRA's goal is to make it possible on one card.
TakeawayFine-tuning needs several times the memory of the model itself. For a 65B model that meant hundreds of gigabytes, which QLoRA cuts to under 48.
Check yourself
- Why does fine-tuning need much more memory than just running the model?
- Because training also stores, for every trainable parameter, its gradient and the optimiser's extra state, and keeps intermediate results from the forward pass for the backward pass.
Idea · §2 "Low-rank Adapters"
Freeze the giant, train a small add-on
QLoRA builds on LoRA (2021). Instead of changing all 65 billion numbers, LoRA freezes them and adds a pair of thin extra matrices next to each layer. Only those small "adapters" are trained. Their output is simply added to the frozen layer's output.
Because the adapters hold well under 1% as many numbers as the model, their gradients and optimiser state are tiny. That removes most of the 780 GB. But one big cost remains: the frozen model itself still has to sit in GPU memory, and at 16 bits that is 130 GB for LLaMA-65B. QLoRA's contribution is shrinking that frozen part.
One finding surprised the authors (§4). The original LoRA paper often added adapters only to two of the attention matrices. QLoRA found that to match full fine-tuning you need adapters on every linear layer of every transformer block, while the adapters' size (their "rank") barely mattered.
TakeawayLoRA already made the training part small. What was left was the frozen model's own size, and that is what QLoRA attacks.
Check yourself
- If the base model is frozen, why can it be stored at low precision?
- Because it is never updated. You only need to read it during the forward and backward passes, so small rounding errors are a fixed, one-off cost rather than something that accumulates during training. The adapters, which are trained, stay at 16-bit precision.
Idea · §3 "4-bit NormalFloat" · Appendix E
Sixteen well-placed values
Storing a number in 4 bits means it can only take one of 24 = 16 possible values. This is called quantization: every real weight gets rounded to the nearest allowed value. The question is which 16 values to allow.
The simple choice is evenly spaced values, like marks on a ruler. But the weights of a trained network are not spread evenly. Most are close to zero and only a few are large, in a bell-shaped curve (a normal distribution). Evenly spaced marks waste many of their 16 slots on the rare large values and leave the crowded middle under-served.
QLoRA's 4-bit NormalFloat (NF4) places the 16 values so that each one is responsible for roughly the same number of weights, assuming the bell shape. They are packed tightly near zero and spaced out at the edges. Exactly one of them is zero, so a weight that is truly zero stays exactly zero.
Each group ("block") of 64 weights is first divided by its largest absolute value, so it fits between −1 and 1, then rounded. Try both rulers below.
TakeawayNF4 spends its 16 values where the weights actually are, so 4-bit rounding loses less information than evenly spaced 4-bit numbers.
Check yourself
- Why does NF4 put its values close together near zero?
- Because that is where most weights are. Closely spaced values there mean most weights are rounded by only a small amount.
- Why scale each block of 64 weights to [−1, 1] first?
- So one fixed set of 16 values can serve every block, whatever its overall size. Only one scale number per block has to be stored to undo it.
Idea · §3 "Double Quantization", "Paged Optimizers"
Compressing the bookkeeping, and an overflow shelf
Each block of 64 weights carries one scale number, stored at 32 bits. That works out to an extra half a bit per weight, which sounds small but is about 4 GB for a 65B model. Double quantization applies the same trick to the scale numbers themselves: they are rounded to 8 bits in groups of 256, with one 32-bit scale per group. The overhead falls from 0.5 to about 0.127 bits per weight, saving roughly 3 GB on the 65B model.
The second trick deals with sudden spikes. Training on an unusually long example can briefly need more memory than the GPU has, which normally crashes the run. Paged optimizers use an NVIDIA feature called unified memory to move the optimiser's state out to the computer's ordinary RAM when the GPU fills up, and bring it back when needed. The authors report this rarely kicks in, and when it doesn't, training runs at full speed.
TakeawaySmall savings on the scale numbers and a safety valve for memory spikes are what push the 65B model from "almost fits" to "fits" on a 48 GB card.
Idea · §4–5 · Tables 1, 4, 6
Does it lose quality? And Guanaco
The authors trained over 1,000 models to check. On standard tests, 4-bit QLoRA with NF4 scored the same as 16-bit fine-tuning: for example, on the MMLU knowledge test across LLaMA sizes from 7B to 65B, the averages were 53.1 for NF4 versus 53.0 for 16-bit (Table 4). They did not run the full 16-bit comparison at 33B and 65B because it was too expensive, and they say so.
They then trained a family of chatbots called Guanaco on OASST1, a dataset of only about 9,000 human-written conversations. With GPT-4 acting as judge, Guanaco-65B scored 99.3% of ChatGPT's level on the Vicuna benchmark (Table 6), using 41 GB of memory.
Two lessons came out of this. First, data quality beat data quantity: the 9,000 carefully written OASST1 examples made a much better chatbot than 450,000 examples from FLAN v2. Second, the benchmarks disagreed: FLAN v2 gave better MMLU scores but worse chat. The authors warn that chatbot benchmarks of the time were not fully trustworthy.
Takeaway4-bit storage plus adapters matched 16-bit fine-tuning on the tests run, and a small, high-quality dataset produced a chatbot close to ChatGPT on that era's benchmark.
Check yourself
- What does it mean that FLAN v2 scored better on MMLU but worse as a chatbot?
- That a model's score on one benchmark does not predict how good it is at another kind of task, so "best model" depends on how you measure.
Abstract, restated
QLoRA backpropagates gradients through a frozen, 4-bit quantized pretrained model into Low-Rank Adapters, reducing memory enough to fine-tune a 65B-parameter model on a single 48 GB GPU while preserving full 16-bit fine-tuning task performance. Three innovations save memory without sacrificing performance: 4-bit NormalFloat (NF4), an information-theoretically motivated data type for normally distributed weights; Double Quantization of the quantization constants; and Paged Optimizers for memory spikes. Over 1,000 models are fine-tuned across 8 instruction datasets, LLaMA and T5 families and scales up to 65B. The Guanaco family reaches 99.3% of ChatGPT on the Vicuna benchmark after 24 hours on one GPU. Findings: small high-quality data can give state-of-the-art results; GPT-4 evaluation is a cheap, reasonable alternative to human evaluation; current chatbot benchmarks are unreliable. Dettmers, Pagnoni, Holtzman, Zettlemoyer, NeurIPS 2023 (arXiv 2305.14314).
Section · §2
Background: block-wise quantization, LoRA, memory
Block-wise absmax quantization. Quantizing a 32-bit tensor to Int8 rescales by its absolute maximum:
A single outlier in the tensor wastes most of the quantization bins, so the flattened tensor is split into n = (b×h)/B contiguous blocks, each with its own constant ci.
LoRA. For a projection XW with X ∈ ℝb×h, W ∈ ℝh×o, LoRA adds a scaled low-rank path with L1 ∈ ℝh×r, L2 ∈ ℝr×o:
Where memory goes. The paper's 7B accounting (batch 1, sequence 512, gradient checkpointing): the 4-bit base model takes 5,048 MB, LoRA parameters 26 MB, and LoRA input gradients 567 MB, which drops to about 18 MB per sequence with checkpointing. The point: once the base model is 4-bit, activation gradients rather than adapter parameters dominate the remaining training memory, so the number of adapters can be increased almost for free.
Section · §3 · Eq. 4 · Appendix E
4-bit NormalFloat
NF4 builds on quantile quantization, which is information-theoretically optimal in the sense that each bin receives an equal expected number of values. Exact quantiles are expensive to estimate per tensor, but pretrained weights are approximately zero-centred normal with some σ. Rescaling each block to [−1, 1] lets a single fixed codebook, built from the quantiles of N(0, 1), serve all blocks:
Here QX is the standard normal quantile function. A symmetric k-bit construction would have no exact zero. NF4 is therefore built asymmetrically: 2k−1 = 8 quantiles for the negative half and 2k−1 + 1 = 9 for the positive half, unified with one duplicate zero removed, giving 16 values with an exact 0 (useful for padding and other zero elements). The resulting codebook from Appendix E is:
−1.0000 −0.6962 −0.5251 −0.3949 −0.2844 −0.1848 −0.0911 0.0000 0.0796 0.1609 0.2461 0.3379 0.4407 0.5626 0.7230 1.0000
The same machinery explains why block-wise scaling matters. A block's absmax sets its grid; one large outlier stretches the grid and every other value in the block is rounded coarsely. Smaller blocks isolate outliers but store more constants. Explore the trade-off:
Evidence. Mean Pile Common Crawl perplexity over OPT, BLOOM, LLaMA and Pythia models from 125M to 13B (Table 2):
| Data type | Mean perplexity |
|---|---|
| Int4 | 34.34 |
| Float4 (E2M1) | 31.07 |
| Float4 (E3M0) | 29.48 |
| NFloat4 + DQ | 27.41 |
Mean zero-shot accuracy on Winogrande, HellaSwag, PiQA, Arc-Easy and Arc-Challenge (Fig. 3) tells the same story: NF4 beats FP4 and double quantization costs nothing measurable.
Section · §3 · Eqs. 5–6
Double quantization, paged optimizers, and the QLoRA layer
Double Quantization (DQ). With block size 64 and FP32 constants, the constants cost 32/64 = 0.5 bits per parameter. DQ treats the first-level constants c2FP32 as a new tensor and quantizes them to 8-bit floats with block size 256, subtracting their mean first (they are positive) so symmetric quantization can be used. This yields second-level constants c1FP32. The cost becomes 8/64 + 32/(64·256) = 0.127 bits per parameter, a saving of 0.373 bits per parameter, about 3 GB for a 65B model.
Paged Optimizers. Optimizer states are allocated in NVIDIA unified memory, which pages automatically between CPU RAM and GPU when the GPU runs out, for instance on a mini-batch with long sequences while using gradient checkpointing. The paper reports no hard timing measurements but notes that for 65B models on 48 GB GPUs at batch size 16 the paged optimizers ran at the same speed as regular ones, since paging is rare.
The QLoRA layer. With storage type NF4 and computation type BFloat16:
Weights are dequantized on the fly, block by block, for both forward and backward passes. Gradients are computed with respect to the adapter weights only (∂E/∂Li); ∂E/∂W is never needed, although backpropagating to earlier layers still passes through the dequantized W. Quantization error in W is therefore a fixed perturbation that the adapters can partially compensate for during fine-tuning.
# Guanaco settings (Appendix B, OASST1 rows)
LoRA on all linear layers of every transformer block
r = 64, α = 16, dropout 0.1 (7B, 13B) / 0.05 (33B, 65B)
lr 2e-4 (7B, 13B) / 1e-4 (33B, 65B), constant schedule
batch 16, 1,875 steps, Adam β2 = 0.999, max grad norm 0.3
Check yourself
- Why is the overhead after DQ 8/64 + 32/(64·256) rather than 8/64?
- Each 64-weight block now stores an 8-bit constant (8/64 bits per weight), and every 256 such constants share one FP32 second-level constant, which is spread over 64·256 weights.
- Why does NF4 need an asymmetric construction?
- An even number of symmetric quantiles has no value at exactly 0. Using 8 negative and 9 positive quantiles, sharing one zero, gives 16 values including an exact zero.
Section · §4–5 · Tables 3–6, Table 1
Experiments
Matching 16-bit fine-tuning (§4). Two hyperparameter findings come first. Adapters only on query and value projections, as in many LoRA setups, did not match full fine-tuning for large base models; adapters on all linear transformer-block layers did, and the rank r had no clear effect. Then, with that setup, Table 3 compares against full fine-tuning on GLUE (RoBERTa-large) and Super-NaturalInstructions (T5 80M–11B). For example, RougeL for T5-3B is 54.3 (full BF16), 55.4 (LoRA BF16) and 55.3 (QLoRA NF4 + DQ). Table 4, 5-shot MMLU after fine-tuning LLaMA on Alpaca or FLAN v2:
| Data type | 7B | 13B | 33B | 65B | Mean |
|---|---|---|---|---|---|
| BFloat16 | 38.4 / 45.6 | 47.2 / 50.6 | 57.7 / 60.5 | 61.8 / 62.5 | 53.0 |
| Float4 | 37.2 / 44.0 | 47.3 / 50.0 | 55.9 / 58.5 | 61.3 / 63.3 | 52.2 |
| NFloat4 + DQ | 39.0 / 44.5 | 47.5 / 50.7 | 57.3 / 59.2 | 61.8 / 63.9 | 53.1 |
Each cell is Alpaca / FLAN v2. Note that the 16-bit baseline in Table 4 is 16-bit LoRA-style fine-tuning of the same configuration, not full fine-tuning; the full-fine-tuning equivalence was established at smaller scale only.
Guanaco (§5). Trained on OASST1 (about 9k samples). Vicuna benchmark, GPT-4 judged, % of ChatGPT score with 95% CI (Table 6): Guanaco 65B 99.3 ± 4.4 (41 GB), 33B 97.8 ± 4.4 (21 GB), 13B 90.4 ± 5.2 (10 GB), 7B 87.0 ± 5.4 (5 GB), against Vicuna 13B 94.9 ± 4.5 (26 GB, 16-bit) and Bard 94.8 ± 4.1. Elo with GPT-4 as judge (Table 1): GPT-4 1348, Guanaco-65B 1022, Guanaco-33B 992, Vicuna-13B 974, ChatGPT 966, Guanaco-13B 916, Bard 902, Guanaco-7B 879.
Data over size. On the Vicuna benchmark, LLaMA fine-tuned on FLAN v2 scores 44.8%, 47.0% and 38.0% at 7B, 13B and 33B, versus 87.0%, 90.4% and 97.8% with OASST1; yet FLAN v2 is better on MMLU at every size (e.g. 44.5 vs 36.6 at 7B, Table 5). Dataset suitability, not size, determines chatbot quality, and MMLU and chatbot performance are largely decoupled.
Evaluation. The authors compare GPT-4 judgements with human raters, find moderate agreement at the system level but notable disagreements at the example level, and observe a position bias in GPT-4 (favouring the first response shown).
Section · §8 and later work
Limitations and lineage
- Scale of the equivalence claim. The authors did not establish that QLoRA matches full 16-bit fine-tuning at 33B and 65B, because of cost; equivalence is shown at smaller scale and against 16-bit adapter baselines at large scale.
- Benchmarks. Evaluation covers MMLU, the Vicuna benchmark and OA; BigBench, RAFT and HELM were not tested. Performance appears to depend on how similar the fine-tuning data is to the benchmark.
- Responsible AI. Only limited evaluation. On CrowS bias, Guanaco-65B scores lower (better) than raw LLaMA-65B on average (43.5 vs 66.6), but other bias types were not assessed.
- Untested variants. Other bit widths (e.g. 3-bit) and other adapter methods were not evaluated.
- Speed. Dequantizing on every pass makes QLoRA steps slower in practice than 16-bit LoRA steps on the same hardware; the method trades compute for memory.
Check yourself
- QLoRA saves memory. What does it cost?
- Extra compute per step to dequantize weights block by block, a fixed quantization error in the base model, and (at the paper's time) the absence of a large-scale check against full fine-tuning.
Reference
Glossary
- Quantization
- Mapping values to a small set of representable values (a codebook) to save memory.
- Absmax
- Scaling a block by its largest absolute value so it fits the codebook's range.
- Block size B
- Number of contiguous values sharing one quantization constant (64 for weights in QLoRA).
- NF4
- 4-bit NormalFloat: a 16-value codebook from normal quantiles with an exact zero.
- DQ
- Double Quantization: quantizing the quantization constants (8-bit, block 256).
- Storage vs compute type
- NF4 in memory, dequantized to BF16 for matrix multiplication.
- LoRA rank r
- Inner dimension of the adapter factors L1L2.
- Paged optimizer
- Optimizer state in unified memory that can spill to CPU RAM under memory pressure.
- Vicuna benchmark
- 80 prompts; responses rated relative to ChatGPT by GPT-4.