II · THE IDEA · ARTIFICIAL INTELLIGENCE
LoRA and Parameter-Efficient Fine-Tuning
▶ Listen · narrated
Fine-tuning used to mean renting a cluster and waiting days. LoRA made it possible to adapt a seven-billion-parameter model on hardware you can buy, in an afternoon, without touching the original weights.
At a glance
- Full name
- Low-Rank Adaptation
- What it trains
- Two small matrices (A and B) per adapted layer, not the original weights
- Typical rank
- 8 to 64, much smaller than the layer's actual dimensions
- Memory saving
- QLoRA reported fine-tuning a 65B model on a single 48GB GPU
Imagine you have a large reference book that you cannot write in, but you are allowed to insert thin sheets of corrections between the pages. LoRA works like those sheets. The original model is the book, frozen and unchanged. The sheets are the small matrices you train, and they contain only the adjustments you need for your specific task. When the model runs, it reads the original page and the correction sheet together, so the output reflects both. Because the sheets are thin, you can train them quickly and store dozens of them without needing much space. The rank is how much information each correction sheet can hold: a higher rank means more capacity, but also more to train and store.
LoRA decomposes a weight update into the product of two low-rank matrices. For a pretrained weight matrix W₀ of dimension d × d, the adapted forward pass computes h = W₀x + BAx, where B is d × r, A is r × d, and r is the rank. During training, W₀ remains frozen and only A and B receive gradient updates. The number of trainable parameters per adapted layer is 2dr, compared to d² for full fine-tuning. For a 4096-dimensional layer with rank 8, that is 65,536 parameters instead of 16,777,216, a reduction by a factor of 256. Matrix A is initialised randomly and B is initialised to zero, so BA starts at zero and the adapted model begins identical to the base model. The technique can be applied selectively: the original paper adapted only the query and value projection matrices in the attention layers, leaving the feedforward weights and other projections untouched. The rank is a hyperparameter that trades off expressiveness against parameter count, and the optimal value depends on the task and the amount of training data. At inference, BA can be merged into W₀ offline, so the adapted model runs at the same speed as the base model. QLoRA extends this by quantising W₀ to 4-bit precision and using paged optimisers to handle memory spikes, which together reduce the memory footprint enough to fine-tune a 65B model on a single 48GB GPU. The quantised weights are dequantised on the fly during the forward pass, and gradients flow only through the LoRA matrices, which remain in higher precision.
Look closer
The matrices are low-rank by design
If a weight matrix is 4096 × 4096, a full update would require training sixteen million parameters. LoRA instead trains two matrices: A is 4096 × r and B is r × 4096, where r is the rank, typically 8, 16 or 64. The product BA has the same shape as the original weight, but contains far fewer learned degrees of freedom. During inference, BA is added to the frozen weight, so the adapted model runs at the same speed as the base model.
The original weights stay frozen
LoRA does not update the pretrained parameters at all. The base model remains intact, and the adaptation is stored separately as a small file, often a few megabytes or tens of megabytes. You can load the same base model once and swap between multiple LoRA adapters without reloading anything large. This makes it practical to maintain dozens of task-specific adaptations of a single foundation model.
Not every layer needs adapting
The original LoRA paper applied the technique only to the attention projection matrices, leaving the feedforward layers untouched. Later work experimented with adapting more layers, but the selective approach remains common. Adapting fewer layers means fewer parameters to train, and in many cases the performance loss is small enough that the trade-off is worth it.
The story
Before LoRA, fine-tuning meant updating every parameter in the network so that it learned your specific task or domain. For a model with seven billion parameters, that meant storing and updating seven billion floating-point numbers, which required hardware most individuals and small teams could not afford. The training run itself needed enough memory to hold the model, the optimizer state, the gradients, and the activations for a batch of examples, which together could exceed a hundred gigabytes.
LoRA changed the arithmetic. Instead of updating the original weight matrices, it trains a pair of much smaller matrices for each layer you want to adapt. If the original weight is d × d, the two new matrices are d × r and r × d, where r is the rank and is chosen to be much smaller than d. A rank of 8 in a layer with dimension 4096 means training 65,536 parameters instead of sixteen million. The product of the two small matrices is added to the frozen original weight, so the model still computes the same shape of output, but almost all the parameters never change.
The technique rests on an assumption about the geometry of fine-tuning: that the update you would make to a weight matrix during adaptation is low-rank, meaning it lies in a space of much lower dimension than the full matrix. This is not a theorem, but it appears to hold empirically across a wide range of tasks. The original paper showed that rank 8 or even rank 4 often sufficed to match the performance of full fine-tuning on natural language tasks, and later work confirmed similar results in other domains.
QLoRA extended the idea by combining LoRA with quantisation. Instead of storing the frozen base model in full precision, QLoRA quantises it to 4-bit integers, which cuts the memory footprint by a factor of eight. The LoRA matrices are still trained in higher precision, but because they are so much smaller, the overall memory requirement drops enough that a 65-billion-parameter model can be fine-tuned on a single consumer GPU with 48 gigabytes of memory. The paper reported that the quality loss from quantisation was negligible when combined with LoRA, making the technique practical for researchers and developers who do not have access to clusters.
Why it mattered then
The LoRA paper appeared in 2021, at a point when large language models were growing rapidly but fine-tuning them remained expensive. Most academic labs and small companies could not afford the hardware or the cloud bills required to adapt a model with tens of billions of parameters. LoRA offered a way to make adaptation affordable without sacrificing much performance, which mattered because the pretrained models were general-purpose and many real applications required domain-specific behaviour. The method was adopted quickly, in part because it was simple to implement and could be added to existing codebases without rebuilding the training infrastructure. QLoRA followed in 2023 and pushed the accessibility further, making it possible to fine-tune models that had previously required institutional resources.
Why it matters now
LoRA has become the default method for fine-tuning open-weight models. Most fine-tuning libraries and platforms support it, and many publicly shared adaptations are distributed as LoRA weights rather than full model checkpoints. The technique also enables a model-sharing ecosystem: a single base model can support hundreds of task-specific adapters, each a few megabytes, which users can download and swap as needed. This matters for deployment as well as training, because serving multiple adapted versions of a model becomes practical when the adapters are small. The method is not limited to language models; it has been applied to diffusion models for image generation and to other domains where large pretrained models are adapted to specific tasks. The low-rank assumption does not hold universally, and there are tasks where LoRA underperforms full fine-tuning, but for many applications the trade-off between cost and quality has proved acceptable.
The surprising detail
The rank required for good performance is often surprisingly low. The original paper found that rank 8 matched full fine-tuning on several benchmarks, and rank 4 sometimes sufficed. This suggests that the effective dimensionality of fine-tuning updates is much smaller than the dimensionality of the weight matrices themselves, which is not obvious from first principles. It also means that the choice of rank is an important hyperparameter: too low and performance degrades, too high and you train more parameters than necessary without gaining much. There is no single correct rank, and practitioners often tune it empirically for each task.
Remember this
LoRA trains two small matrices per layer instead of updating the full weights, making fine-tuning cheap enough for a single GPU.
Test yourself
You are fine-tuning a model with LoRA and you have a choice: use rank 8 and adapt only the attention layers, or use rank 64 and adapt the attention layers plus the feedforward layers. Both fit in your memory budget. What are two reasons you might prefer the first option, and one reason you might prefer the second?
The first option trains far fewer parameters, so training will be faster and the saved adapter file will be smaller, which matters for storage and for swapping adapters at inference time. It may also generalise better if your training data is limited, because a lower-capacity adapter is less likely to overfit. The second option gives the model more expressive power to capture the adaptation, which may be necessary if the task requires changes that cannot be represented well by low-rank updates to attention alone, or if the domain shift is large enough that adapting more of the network is worth the cost.
Go deeper
- LoRA: Low-Rank Adaptation of Large Language Models · arXiv · Edward J. Hu et al. · 2021-06-17
- QLoRA: Efficient Finetuning of Quantized LLMs · arXiv · Tim Dettmers et al. · 2023-05-23
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.