Adapters beyond LoRA

The low-rank hypothesis and its descendants#

LoRA (Hu et al., 2022) reparameterises the weight update as

$$ W = W_0 + \Delta W = W_0 + \frac{\alpha}{r} BA, \qquad B \in \mathbb{R}^{d\times r},\ A \in \mathbb{R}^{r\times k} $$

with $W_0$ frozen, $B$ initialised to zero and $A$ to a random Gaussian. The claim is that the update has low intrinsic rank even when $W_0$ does not — supported by Aghajanyan et al., 2021 (Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning), who showed a few hundred to a few thousand parameters in a random subspace can reach 90% of full fine-tuning performance on GLUE.

rsLoRA (Kalajdzievski, 2023) points out the $\alpha/r$ scaling makes gradients vanish as $r$ grows, so higher ranks stop helping for the wrong reason. Replacing it with $\alpha/\sqrt r$ fixes the scaling and restores monotonic improvement with rank.

DoRA (Liu et al., 2024) decomposes the weight into magnitude and direction:

$$ W = m \cdot \frac{W_0 + BA}{\lVert W_0 + BA \rVert_c} $$

where $m \in \mathbb{R}^{1\times k}$ is a learned per-column magnitude and $\lVert\cdot\rVert_c$ is the column-wise norm. Their analysis shows full fine-tuning changes magnitude and direction largely independently, while LoRA couples them; decoupling closes most of the low-rank gap. The cost is $k$ extra parameters per layer — the 1,024 in the table.

LoRA+ (Hayou et al., 2024) shows the optimal learning rates for $A$ and $B$ differ by a factor that grows with width, and that setting $\eta_B = \lambda \eta_A$ with $\lambda \gg 1$ improves both speed and final quality. This is a one-line change most practitioners never make.

Smaller than low-rank#

IA3 (Liu et al., 2022, Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning) learns three vectors per block, rescaling keys, values and the feed-forward intermediate:

$$ \text{attn} = \mathrm{softmax}!\left(\frac{Q(l_k \odot K)^\top}{\sqrt{d}}\right)(l_v \odot V), \qquad \text{FFN} = (l_{ff}\odot \gamma(W_1 x))W_2 $$

It uses roughly 0.01% of parameters and, in their few-shot setting, beat in-context learning at a fraction of the inference cost.

VeRA (Kopiczko et al., 2024) freezes a single pair of random matrices $A, B$ shared across all layers and learns only two scaling vectors $b, d$ per layer:

$$ \Delta W = \Lambda_b B \Lambda_d A $$

Parameter count becomes $O(d + r)$ per layer instead of $O(r(d+k))$, reported at roughly 10× smaller than LoRA at comparable quality on GLUE and E2E. The relevant use case is serving thousands of adapters.

BitFit (Ben Zaken et al., 2022) trains only bias terms — about 0.09% of parameters — and is competitive on small-data GLUE tasks. It is the strongest evidence that a large part of task adaptation is a shift rather than a rotation.

Structure-preserving methods#

OFT and BOFT (Qiu et al., 2023; Liu et al., 2024) multiply rather than add:

$$ W = R\,W_0, \qquad R\ \text{orthogonal, block-diagonal} $$

Because $R$ is orthogonal, the singular values of $W_0$ are preserved exactly — the transformation is a rotation, hence norm-preserving and information-preserving in a precise sense. This is why OFT was first adopted in text-to-image work, where additive adapters visibly damage the base model's generative quality. BOFT adds a butterfly factorisation to raise expressiveness at fixed parameter count.

Prompt-space methods#

Prefix tuning (Li and Liang, 2021) prepends trainable key–value vectors to every attention layer. Prompt tuning (Lester et al., 2021) prepends trainable embeddings to the input only, and shows the gap to full fine-tuning closes as model size grows, vanishing around 10B parameters. P-tuning v2 restores the per-layer prefixes for smaller models.

These consume context length, which is their practical drawback: a 100-token soft prompt costs 100 tokens of window on every request forever.

What adapters cost in quality#

Biderman et al., 2024 (LoRA Learns Less and Forgets Less) is the most careful public comparison, on continued pretraining and instruction tuning in code and maths. Their findings:

• On continued pretraining with 10B–20B tokens of new domain data, LoRA falls substantially short of full fine-tuning. The gap is largest for code.
• On instruction tuning with smaller datasets, LoRA is close to full fine-tuning.
• LoRA forgets less — it better preserves performance outside the target domain — and acts as a stronger regulariser than classic weight decay or dropout.
• Full fine-tuning finds updates whose rank is 10–100× higher than typical LoRA configurations, which explains the first two points directly.

The practical rule follows from the ranking: adapters for shaping behaviour on modest data, full fine-tuning or continued pretraining for installing large new knowledge.

Serving many adapters#

The property that made LoRA an infrastructure decision rather than a training trick: $B A$ can be computed as a separate small matmul on a shared frozen base, so many adapters can be served in one batch. S-LoRA (Sheng et al., 2024) and Punica implement this with custom kernels, serving thousands of adapters on a single GPU with the base weights loaded once. Merging is the alternative and gives up multi-tenancy.

Papers#

• Houlsby et al., Parameter-Efficient Transfer Learning for NLP, ICML 2019 — arxiv.org/abs/1902.00751
• Li and Liang, Prefix-Tuning, ACL 2021 — arxiv.org/abs/2101.00190
• Lester et al., The Power of Scale for Parameter-Efficient Prompt Tuning, EMNLP 2021 — arxiv.org/abs/2104.08691
• Aghajanyan et al., Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning, ACL 2021 — arxiv.org/abs/2012.13255
• Ben Zaken et al., BitFit, ACL 2022 — arxiv.org/abs/2106.10199
• Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, ICLR 2022 — arxiv.org/abs/2106.09685
• Liu et al., Few-Shot PEFT is Better and Cheaper than In-Context Learning (IA3), NeurIPS 2022 — arxiv.org/abs/2205.05638
• Dettmers et al., QLoRA, NeurIPS 2023 — arxiv.org/abs/2305.14314
• Qiu et al., Controlling Text-to-Image Diffusion by Orthogonal Finetuning (OFT), NeurIPS 2023 — arxiv.org/abs/2306.07280
• Kalajdzievski, A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA, 2023 — arxiv.org/abs/2312.03732
• Kopiczko et al., VeRA: Vector-based Random Matrix Adaptation, ICLR 2024 — arxiv.org/abs/2310.11454
• Liu et al., DoRA: Weight-Decomposed Low-Rank Adaptation, ICML 2024 — arxiv.org/abs/2402.09353
• Hayou et al., LoRA+, ICML 2024 — arxiv.org/abs/2402.12354
• Biderman et al., LoRA Learns Less and Forgets Less, TMLR 2024 — arxiv.org/abs/2405.09673
• Sheng et al., S-LoRA: Serving Thousands of Concurrent LoRA Adapters, MLSys 2024 — arxiv.org/abs/2311.03285

What to learn next#

• Catastrophic forgetting — the damage adapters are partly protecting you from.
• LoRA — the method this lesson builds on, in full.
• Merging model weights — combining several of these into one model.