LoRA

The parameterisation#

For a frozen pretrained weight W₀ ∈ R^{d_out × d_in}, LoRA constrains the update to a low-rank factorisation:

h = W₀ x  +  ΔW x ,        ΔW = (α / r) · B A

A ∈ R^{r × d_in}      initialised  A ~ N(0, σ²)
B ∈ R^{d_out × r}     initialised  B = 0
r ≪ min(d_in, d_out)

r is the rank, α a fixed scaling constant, and α/r the effective magnitude of the update. B = 0 gives ΔW = 0 at initialisation, so the adapted model is exactly the pretrained model at step zero. Exactly one factor may be zero-initialised, or the gradient of both vanishes identically.

Trainable parameters per adapted matrix fall from d_in · d_out to r(d_in + d_out). For d_in = d_out = 4096 and r = 16 that is 16.8M down to 131k, a factor of 128.

What is actually saved#

The saving is in optimizer state and gradients, not in activations.

full fine-tune, Adam, mixed precision   ≈ 16 bytes per parameter
LoRA                                    ≈ 2 bytes per frozen parameter (weights only)
                                          + 16 bytes per adapter parameter

Activation memory for the backward pass is unchanged: gradients must still flow through the frozen layers to reach adapters in earlier layers, so their inputs are still stored. Batch size and sequence length remain the binding constraints on long-context fine-tuning.

Hu et al. (2021) report 10,000× fewer trainable parameters and roughly 3× less GPU memory for GPT-3 175B compared with full fine-tuning under Adam, at equal or better quality on their benchmarks. Checkpoints shrink by the same factor as the parameter count, which is what makes per-task adapters practical to store and ship.

Why low rank is a reasonable prior#

Aghajanyan, Zettlemoyer and Gupta (2020), Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning, measured the smallest random subspace in which fine-tuning still reaches 90% of full performance. For many tasks it is in the hundreds or low thousands of dimensions, against hundreds of millions of parameters. Larger pretrained models had lower intrinsic dimension, which is the empirical result LoRA is built on.

The assumption is a prior, not a theorem. It holds well for adaptation to a task the base model can nearly already do, and degrades when the target genuinely requires new capability.

Variants that matter#

QLoRA (Dettmers et al., 2023). Three contributions, all load-bearing: NF4, a 4-bit data type that is information-theoretically optimal for normally distributed weights; double quantisation, quantising the quantisation constants themselves for roughly 0.37 bits per parameter more; and paged optimizers, using unified memory to survive gradient-checkpointing memory spikes. Enabled 65B fine-tuning on a single 48GB card with 16-bit full-fine-tuning quality on their evaluations.

rsLoRA (Kalajdzievski, 2023). Scaling by α/r causes gradients to shrink as r grows, so high-rank adapters underperform their capacity. Scaling by α/√r fixes the collapse and makes rank behave monotonically.

DoRA (Liu et al., 2024). Decomposes the weight into magnitude and direction, applying LoRA only to the direction. Closes much of the remaining gap to full fine-tuning at low rank, for a small extra cost.

LoRA+ (Hayou et al., 2024). A and B have different roles under the zero-initialisation, and a single learning rate is suboptimal. Setting B's learning rate several times higher than A's improves convergence at no cost.

Serving many adapters#

Because ΔW = BA is separable, a single base model in memory can serve many adapters concurrently by batching their side paths. S-LoRA (Sheng et al., 2023) and Punica demonstrate thousands of adapters on one GPU with unified paging for the adapter weights. Merging is the opposite choice: fold ΔW into W₀ for zero inference overhead, at the cost of losing the ability to switch.

Merging is exact only in the precision you merge at. Merging a bf16 adapter into a 4-bit base requires dequantising, merging, and requantising, which is lossy and a real source of quality regressions between training and deployment.

Known limitations#

Biderman et al. (2024) trained matched LoRA and full fine-tuning runs on code and mathematics. LoRA underperformed on target-domain gains, particularly on continued pretraining with large token budgets, and simultaneously forgot less of the base model's general ability. They also found LoRA acts as a stronger regulariser than weight decay or dropout, and that full fine-tuning learns perturbations of far higher rank than typical LoRA settings — 10 to 100 times higher.

Shuttleworth et al. (2024) identify intruder dimensions: singular vectors in the adapted weight matrix that are near-orthogonal to anything in the pretrained spectrum. Full fine-tuning does not produce them. Models with intruder dimensions match on the target task but generalise worse out of distribution and forget more of the pretraining distribution, and higher rank with rank stabilisation reduces the effect.

The practical reading: LoRA is the right default for task adaptation on a fixed budget, and the wrong tool for teaching a model a genuinely new domain at scale.

Papers#

• Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, 2021 — arxiv.org/abs/2106.09685
• Aghajanyan et al., Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning, 2020 — arxiv.org/abs/2012.13255
• Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs, 2023 — arxiv.org/abs/2305.14314
• Kalajdzievski, A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA, 2023 — arxiv.org/abs/2312.03732
• Liu et al., DoRA: Weight-Decomposed Low-Rank Adaptation, 2024 — arxiv.org/abs/2402.09353
• Hayou et al., LoRA+: Efficient Low Rank Adaptation of Large Models, 2024 — arxiv.org/abs/2402.12354
• Biderman et al., LoRA Learns Less and Forgets Less, 2024 — arxiv.org/abs/2405.09673
• Sheng et al., S-LoRA: Serving Thousands of Concurrent LoRA Adapters, 2023 — arxiv.org/abs/2311.03285

What to learn next#

• Fine-tuning — the decision that comes before choosing a method.
• Hugging Face — where base models and adapters are published.
• What is RAG? — the alternative when the problem is missing knowledge.