Study guide · Machine learning & training · 2023

QLoRA: Efficient Finetuning of Quantized LLMs

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer · NeurIPS 2023

Fine-tuning a large language model normally needs a rack of expensive GPUs, because every number in the model has to sit in memory at full precision. QLoRA stores the frozen model in just 4 bits per number and trains only small add-on "adapters", so a 65-billion-parameter model can be fine-tuned on a single 48 GB GPU with no measurable loss in quality on the benchmarks the authors tested.

The paper in four sentences

Fine-tuning a 65-billion-parameter model the ordinary way needs more than 780 GB of GPU memory, far beyond any single card.

QLoRA freezes the original model, squeezes each of its numbers into 4 bits using a clever set of 16 allowed values ("NormalFloat"), and trains only small LoRA adapters bolted on top, which stay at higher precision.

Two extra tricks, compressing the compression's own bookkeeping ("double quantization") and spilling memory spikes over to ordinary computer RAM ("paged optimizers"), bring the 65B model under 48 GB.

Models trained this way match 16-bit fine-tuning on the benchmarks tested, and the resulting chatbot, Guanaco, scored 99.3% of ChatGPT's level on the Vicuna benchmark after about a day on one GPU.

Idea · §1–2

The memory wall

A language model is, at heart, a huge list of numbers called parameters (or weights). LLaMA-65B has about 65 billion of them. Each one is normally stored in 16 bits, which is 2 bytes, so just holding the model takes about 130 GB.

Fine-tuning means continuing to train a model on new examples so it learns a new skill, such as following instructions politely. Ordinary fine-tuning needs much more than the model itself: for every parameter you also store its gradient (which way to nudge it) and the optimiser's running averages. The paper puts the total for the 65B model at more than 780 GB. The biggest single GPUs of the time had 80 GB.

So fine-tuning big models was something only well-funded labs could do. QLoRA's goal is to make it possible on one card.

Bars use a log scale. "Full 16-bit fine-tuning" assumes about 12 bytes per parameter, which reproduces the paper's figure of over 780 GB for 65B. "LoRA, 16-bit model" counts only the frozen 16-bit weights (2 bytes per parameter), so its real total is a bit higher. The QLoRA bars are the paper's own measured totals (Appendix G, Fig. 6: 6.9, 11.3, 24.7 and 45.0 GB). Dashed lines mark common GPU sizes.
Everyday pictureYou want to annotate a 20-volume encyclopaedia, but your desk only fits two volumes. QLoRA's answer: shrink the encyclopaedia into a pocket edition you will never write in, and keep your notes on a thin pad beside it.

TakeawayFine-tuning needs several times the memory of the model itself. For a 65B model that meant hundreds of gigabytes, which QLoRA cuts to under 48.

Check yourself
Why does fine-tuning need much more memory than just running the model?
Because training also stores, for every trainable parameter, its gradient and the optimiser's extra state, and keeps intermediate results from the forward pass for the backward pass.

Idea · §2 "Low-rank Adapters"

Freeze the giant, train a small add-on

QLoRA builds on LoRA (2021). Instead of changing all 65 billion numbers, LoRA freezes them and adds a pair of thin extra matrices next to each layer. Only those small "adapters" are trained. Their output is simply added to the frozen layer's output.

Because the adapters hold well under 1% as many numbers as the model, their gradients and optimiser state are tiny. That removes most of the 780 GB. But one big cost remains: the frozen model itself still has to sit in GPU memory, and at 16 bits that is 130 GB for LLaMA-65B. QLoRA's contribution is shrinking that frozen part.

One finding surprised the authors (§4). The original LoRA paper often added adapters only to two of the attention matrices. QLoRA found that to match full fine-tuning you need adapters on every linear layer of every transformer block, while the adapters' size (their "rank") barely mattered.

TakeawayLoRA already made the training part small. What was left was the frozen model's own size, and that is what QLoRA attacks.

Check yourself
If the base model is frozen, why can it be stored at low precision?
Because it is never updated. You only need to read it during the forward and backward passes, so small rounding errors are a fixed, one-off cost rather than something that accumulates during training. The adapters, which are trained, stay at 16-bit precision.

Idea · §3 "4-bit NormalFloat" · Appendix E

Sixteen well-placed values

Storing a number in 4 bits means it can only take one of 24 = 16 possible values. This is called quantization: every real weight gets rounded to the nearest allowed value. The question is which 16 values to allow.

The simple choice is evenly spaced values, like marks on a ruler. But the weights of a trained network are not spread evenly. Most are close to zero and only a few are large, in a bell-shaped curve (a normal distribution). Evenly spaced marks waste many of their 16 slots on the rare large values and leave the crowded middle under-served.

QLoRA's 4-bit NormalFloat (NF4) places the 16 values so that each one is responsible for roughly the same number of weights, assuming the bell shape. They are packed tightly near zero and spaced out at the edges. Exactly one of them is zero, so a weight that is truly zero stays exactly zero.

Each group ("block") of 64 weights is first divided by its largest absolute value, so it fits between −1 and 1, then rounded. Try both rulers below.

3,200 random bell-shaped weights in blocks of 64, each block scaled to [−1, 1]. Grey bars: how the weights are spread. Vertical lines: the allowed values (NF4 uses the 16 values listed in the paper's Appendix E; Int4 uses 15 evenly spaced steps of 1/7). The coloured strip at the bottom shows how many weights round to each value. The error is measured after scaling back, and NF4's is consistently lower.
Everyday pictureA shoe shop can only stock 16 sizes. Stocking sizes evenly from tiny to enormous means lots of empty shelves at the extremes and long queues in the middle. Stocking more sizes near the common ones serves far more customers well.

TakeawayNF4 spends its 16 values where the weights actually are, so 4-bit rounding loses less information than evenly spaced 4-bit numbers.

Check yourself
Why does NF4 put its values close together near zero?
Because that is where most weights are. Closely spaced values there mean most weights are rounded by only a small amount.
Why scale each block of 64 weights to [−1, 1] first?
So one fixed set of 16 values can serve every block, whatever its overall size. Only one scale number per block has to be stored to undo it.

Idea · §3 "Double Quantization", "Paged Optimizers"

Compressing the bookkeeping, and an overflow shelf

Each block of 64 weights carries one scale number, stored at 32 bits. That works out to an extra half a bit per weight, which sounds small but is about 4 GB for a 65B model. Double quantization applies the same trick to the scale numbers themselves: they are rounded to 8 bits in groups of 256, with one 32-bit scale per group. The overhead falls from 0.5 to about 0.127 bits per weight, saving roughly 3 GB on the 65B model.

The second trick deals with sudden spikes. Training on an unusually long example can briefly need more memory than the GPU has, which normally crashes the run. Paged optimizers use an NVIDIA feature called unified memory to move the optimiser's state out to the computer's ordinary RAM when the GPU fills up, and bring it back when needed. The authors report this rarely kicks in, and when it doesn't, training runs at full speed.

Everyday pictureDouble quantization is like noticing that your shipping labels take up a lot of space and printing them smaller too. Paged optimizers are an overflow shelf in the hallway for the odd day a delivery is too big for the desk.

TakeawaySmall savings on the scale numbers and a safety valve for memory spikes are what push the 65B model from "almost fits" to "fits" on a 48 GB card.

Idea · §4–5 · Tables 1, 4, 6

Does it lose quality? And Guanaco

The authors trained over 1,000 models to check. On standard tests, 4-bit QLoRA with NF4 scored the same as 16-bit fine-tuning: for example, on the MMLU knowledge test across LLaMA sizes from 7B to 65B, the averages were 53.1 for NF4 versus 53.0 for 16-bit (Table 4). They did not run the full 16-bit comparison at 33B and 65B because it was too expensive, and they say so.

They then trained a family of chatbots called Guanaco on OASST1, a dataset of only about 9,000 human-written conversations. With GPT-4 acting as judge, Guanaco-65B scored 99.3% of ChatGPT's level on the Vicuna benchmark (Table 6), using 41 GB of memory.

Two lessons came out of this. First, data quality beat data quantity: the 9,000 carefully written OASST1 examples made a much better chatbot than 450,000 examples from FLAN v2. Second, the benchmarks disagreed: FLAN v2 gave better MMLU scores but worse chat. The authors warn that chatbot benchmarks of the time were not fully trustworthy.

Real numbers from the paper. Elo is a chess-style rating built from many head-to-head comparisons judged by GPT-4 (averaged over 10,000 random orderings). Guanaco models are amber. Note how the judge matters: the paper also ran human raters, and both are imperfect measures.
Later refinementNF4 and QLoRA became a default in open-source tools (the bitsandbytes and Hugging Face PEFT libraries), which is how many hobbyists and small labs fine-tune models today. Later work such as LoftQ and QA-LoRA tried to start the adapters in a way that compensates for the rounding error, or to make the final merged model itself 4-bit.

Takeaway4-bit storage plus adapters matched 16-bit fine-tuning on the tests run, and a small, high-quality dataset produced a chatbot close to ChatGPT on that era's benchmark.

Check yourself
What does it mean that FLAN v2 scored better on MMLU but worse as a chatbot?
That a model's score on one benchmark does not predict how good it is at another kind of task, so "best model" depends on how you measure.

Follow the idea

Built on: LoRA (2021)