LoRA: Low-Rank Fine-Tuning

A neural network is, underneath everything, a long list of numbers called weights, stored in rectangular grids called matrices. When the network reads a sentence, the words are turned into lists of numbers, and those lists are multiplied through the grids, layer after layer, until an answer comes out. Everything the model knows lives in the values of those weights. Fine-tuning — teaching a pre-trained model a new task — traditionally meant letting the training process adjust every one of them.

That works, but the cost is larger than it first appears, and the cost is the point of this story. Training a weight means computing a gradient for it: a number saying which direction to nudge it. The optimiser — the routine that decides the actual step sizes — keeps one or two more bookkeeping numbers per weight on top of that. So for every weight it trains, the computer holds two or three extra values in memory. A model with 175 billion weights, the size of GPT-3, therefore needs room for hundreds of billions of extra values before it has learned from a single example. And when training ends, you own a complete second copy of the model that is good at exactly one task. Train it for ten tasks and you own ten copies.

LoRA — low-rank adaptation, introduced by Hu and colleagues — begins from an observation about what fine-tuning actually produces. Call the pre-trained grid W0, with d rows and k columns. Full fine-tuning turns it into some new grid; the difference between new and old is the update, ΔW, a grid of exactly the same shape. LoRA's premise is that this update, though it has room for d times k independent numbers, does not use that room. The measure of how much genuine freedom a grid of numbers exercises is called its rank, and the premise is that the update's effective rank is low — its useful content can be squeezed into a far thinner form.

Here is the mechanism, step by step. Instead of learning ΔW directly, LoRA learns two thin grids: B, with d rows and only r columns, and A, with r rows and k columns. The number r is the rank, chosen by hand before training starts, and it is small — perhaps 8, against grid widths in the thousands. Multiplying B by A gives back a grid of the full d by k shape, and that product stands in for the update. Now count what is actually trained: not d times k numbers, but r times (d plus k). For small r that is a tiny fraction of the original.

During training, W0 does not move at all. It receives no gradients, and the optimiser stores no bookkeeping for it — which is exactly where the memory saving comes from. When an input x passes through the layer, it is multiplied by the frozen W0 and, separately, by the thin pair; the two results are added together, position by position. Written out, the layer's output is W0x + BAx. The starting values matter too. A begins as random noise drawn from a Gaussian — a bell-curve distribution — while B begins as all zeros. Anything multiplied by zeros gives zeros, so the product BA starts at exactly zero, and at the first training step the model behaves precisely as the pre-trained model did. The adapter then grows away from zero only as far as the new task demands.

In the original paper the thin pairs are not attached everywhere. They go mainly on the attention projections — the four matrices in each attention layer, called query, key, value and output, which govern how the model weighs earlier words when processing the current one. The rest of the network stays untouched. The rank r is a setting the researcher picks, not something the model discovers.

The reported results are what made the method spread. On GPT-3 with 175 billion parameters, the paper reports up to roughly 10,000 times fewer trainable parameters than full fine-tuning, and about 3 times less GPU memory, while matching or beating full fine-tuning quality on the RoBERTa, DeBERTa, GPT-2 and GPT-3 experiments run.

There is a closing trick, and the algebra gives it away for free. The layer computes W0x + BAx, and that is the same value as (W0 + BA)x — adding the two grids first and multiplying once gives an identical answer. So before serving the model you may add BA into W0 permanently. What ships is then one ordinary dense matrix of the usual shape. The adapter has vanished into it; nothing extra runs when the model answers a question, so there is no added delay. Or you may skip the merge, keep a single frozen base model in memory, and change what the model does by swapping in a different small A and B pair. Several specialised behaviours can then share one copy of the expensive base weights.

LoRA belongs to a family of methods called parameter-efficient fine-tuning, or PEFT. A separate study compared that family with in-context learning — teaching a model a task by pasting a few worked examples into the prompt itself. On the tasks reported, the efficient updates were both more accurate and cheaper to run. The reason is structural. Pasted examples must be re-read, token by token, on every single request; a trained adapter is paid for once and then costs nothing more.

Two honest limits deserve their own sentences. The low-rank premise is an empirical working assumption — well supported on the models and tasks measured, but not a theorem. A task that genuinely needs a high-rank change, or a change in layers left untouched, will not be captured. And the striking ratios — 10,000 times, 3 times — belong to the specific configurations the paper measured, not to every model at every scale. What the method claims is narrower, and stronger for it: on everything tested, the adjustment a new task needed fitted in a thin slice, and training only that slice matched rewriting the whole.