The LoRA Family: A Mathematically Precise Guide to Parameter-Efficient Fine-Tuning

1. The Foundational Insight: Why Full Fine-Tuning Is Wasteful

The central claim of Hu et al. (arXiv:2106.09685) is not immediately obvious — it requires a detour through the geometry of high-dimensional weight spaces.

During full fine-tuning of a pretrained language model, each weight matrix W₀ ∈ ℝ^{d×d} accumulates an update ΔW across gradient steps. The hypothesis Hu et al. test — and confirm — is that these updates exhibit low intrinsic rank: the learned ΔW, despite living in a d×d parameter space, has meaningful structure concentrated in a subspace of dimension r ≪ d.

This is not an accident. A pretrained model already occupies a high-density region of a well-structured loss landscape. Task-specific adaptation steers the model within that landscape via updates whose gradient flow concentrates along a small number of dominant singular directions. The remaining singular values are numerically negligible — a manifestation of the same compressibility that makes SVD useful for image compression, but occurring naturally in the optimization dynamics.

The parameterisation that follows from this observation:

$\Delta W = B A, \quad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times d},\quad r \ll d$

During training, W₀ is frozen — no gradient flows through it. Only A and B are trainable. A is initialized from a random Gaussian; B is initialized to zero so that ΔW = BA = 0 at the start of training, preserving the pretrained model’s output exactly at step zero.

The parameter reduction is dramatic. A standard attention projection with d = 4096 has d² = 16,777,216 parameters. LoRA with r = 8 substitutes 2 × d × r = 65,536 — a reduction of 256× for that layer. Across all four attention projections (Q, K, V, O) in a 70B model with ~8192 embedding dimension, full fine-tuning touches ~268M parameters per layer; LoRA at r = 8 touches ~524K — a ~512× reduction per layer, accumulating to roughly 10,000× fewer trainable parameters across the full model at representative rank choices.

2. The Forward Pass: Algebra, Scaling, and Why α/r Is Not Cosmetic

The LoRA forward pass for an input x through a modified projection is:

$h = W_0 x + \frac{\alpha}{r} \cdot B A x$

where α is a scalar hyperparameter, fixed (not trained). The scaling factor α/r appears simple but carries precise meaning.

Why not just absorb α into the learning rate? Because LoRA adapters are routinely transferred across ranks. If you train at r = 8 and inference requires r = 16 (say, for a merged-then-re-split use case), the magnitude of BA changes: a rank-16 decomposition of the same ΔW produces entries roughly √2 larger in expectation under random initialization. The α/r normalization ensures that for fixed α, the effective update magnitude is invariant to rank at initialization — so hyperparameter sweeps over r remain comparable. Concretely: doubling r at fixed α halves the scale of the adapter contribution, exactly counteracting the increased capacity. This makes α/r a genuine architectural constant, not a redundant knob.

Practical defaults: Hu et al. recommend α = r as the neutral choice (α/r = 1), recovering unscaled ΔW. Many practitioners find α = 16 with r = 8 (α/r = 2) provides a mild amplification that accelerates early task specialization without instability. The key is that once α is set, it should remain fixed across rank ablations to preserve comparability.

At inference, the adapter can be merged into the base weight at zero cost:

$W_{\text{merged}} = W_0 + \frac{\alpha}{r} B A$

This produces a standard weight matrix with no runtime overhead — a crucial property for production deployment.

3. Which Layers to Adapt: Empirical Rules and Mechanistic Reasoning

The original LoRA paper focuses on the attention projection matrices — specifically Q, K, V, and the output projection O. The MLP blocks are often left untouched in the original formulation. Why?

Mechanistically: Attention heads are the primary locus of task-relevant routing in transformers. Query and Key projections govern where the model attends; Value and Output projections govern what it extracts and writes. Adapting these four matrices gives direct control over the model’s information-routing behaviour — the axis most sensitive to task-specific fine-tuning signals — while leaving the MLP’s knowledge-storing function intact.

Empirically: Hu et al. ablate adapter placement across Q, K, V, O, and combinations thereof on GPT-2 and GPT-3 scale models. Adapting Q and V alone at rank 4 matches or exceeds adapting all four projections at rank 1, suggesting the value-side is the highest-leverage intervention. The finding is robust but not universal — code generation tasks show stronger signal from MLP adaptation than dialogue tasks.

Contemporary practice (post-LLaMA era):

For instruction-following and chat fine-tuning on LLaMA-class models, the broadly validated configuration is:

• Q, K, V, O at r = 8–32 as the baseline
• gate_proj, up_proj, down_proj (MLP) added when the target task requires factual knowledge injection or domain vocabulary shift, not purely behavioural change
• r = 8 for tasks with abundant data (>100K examples); r = 4 for scarce-data regimes to prevent adapter overfitting; r = 64–128 when target capability diverges substantially from the pretrained distribution (e.g., reasoning format shift)

The embed and lm_head layers are almost never adapted via LoRA — their parameter counts are large but their role is primarily tokenisation, not task reasoning.

4. QLoRA: Making 70B Fine-Tuning Accessible on a Single GPU

Dettmers et al. (arXiv:2305.14314) identify the binding constraint on LoRA adoption: even with 10,000× fewer trainable parameters, the frozen base model still occupies GPU memory. A 70B parameter model in BF16 requires ~140 GB — more than any single consumer or even prosumer GPU can hold.

QLoRA’s solution is a precise composition of three innovations:

4.1 NF4: NormalFloat Quantization

Standard int4 quantization maps values to uniformly spaced bins. But pretrained neural network weights are not uniform — they are approximately normally distributed (centred near zero, with tails). NF4 (4-bit NormalFloat) constructs bins that are information-theoretically optimal for normally distributed values: the quantization levels are set at the quantiles of the standard normal distribution, not uniform intervals. For a 4-bit representation (16 levels), NF4 places 16 values at the quantile boundaries of N(0, 1), minimizing expected squared quantization error under the assumption that weights are normally distributed. Empirically, NF4 reduces perplexity degradation by 0.25–0.5 points relative to int4 at the same bit-width — a meaningful gap at model scale.

4.2 Double Quantization

Every quantization scheme requires quantization constants (scale factors) per block of weights. In standard 8-bit block-wise quantization with 64-parameter blocks, the scale constants themselves consume approximately 0.5 bits per parameter — non-trivial overhead at 70B scale. QLoRA applies a second quantization pass to these scale constants (quantizing them to 8-bit from FP32), recovering approximately 0.37 bits per parameter — roughly 3.3 GB on a 70B model.

4.3 Paged Optimizers

The remaining memory spike occurs during the backward pass when gradient checkpoints are processed: optimizer state (Adam momentum terms, second moments) can momentarily demand GPU memory that peaks above steady-state capacity. QLoRA adopts NVIDIA’s unified memory to page optimizer state to CPU RAM on-demand, preventing out-of-memory crashes from transient spikes without requiring permanent CPU offloading.

The result: A 70B model quantized to NF4 occupies approximately 35 GB of GPU memory (roughly 0.5 bytes per parameter). With paged optimizer state and LoRA adapters at r = 64 (approximately 200M trainable parameters), fine-tuning fits within a single A100 80 GB or RTX 3090/4090 48 GB GPU — a reduction from ~400 GPU-hours on 8×A100 to single-GPU accessibility.

The key theoretical claim — that quantization error in the frozen base and LoRA adapter gradients are independent — means NF4 quantization of W₀ does not corrupt the adapter’s learning signal. In practice, QLoRA recovers ~99% of full BF16 fine-tuning performance on standard benchmarks (MMLU, BigBench), with the remaining gap often within task noise.

5. DoRA: Decomposing Weight Updates into Magnitude and Direction

The persistent gap between LoRA-tuned and fully fine-tuned models — roughly 1–4% on held-out reasoning tasks even at high rank — motivated Liu et al. (arXiv:2402.09353) to ask a structural question: what is LoRA failing to represent?

Their analysis begins with a weight decomposition borrowed from the optimization literature:

$W = \underbrace{\|W\|_c}_{m \in \mathbb{R}} \cdot \underbrace{\frac{W}{\|W\|_c}}_{V \in \mathbb{R}^{d \times k}}$

where ‖·‖_c is the column-wise norm and m is the resulting magnitude vector (one scalar per output dimension). The unit-norm matrix V encodes directional structure.

The DoRA insight: Full fine-tuning modifies both magnitude and direction simultaneously and with independent degrees of freedom. LoRA, by construction, applies a rank-r additive perturbation that couples magnitude and directional changes — a rank-r update to V inherently also modifies column norms, but not with the freedom of an independent magnitude parameter.

DoRA decouples them:

$W' = m \cdot \frac{W_0 + \Delta W}{\|W_0 + \Delta W\|_c} = m \cdot \frac{W_0 + BA}{\|W_0 + BA\|_c}$

Here:

• m is a trainable magnitude vector (d parameters — negligible cost)
• BA is the standard LoRA low-rank directional update
• The column normalisation ensures V’ = (W₀ + BA)/‖W₀ + BA‖_c is unit-norm, isolating directional adaptation

What this buys: By granting an independent degree of freedom to magnitude scaling, DoRA allows the model to separately ask “should this direction’s influence grow or shrink?” (m) and “should this direction rotate?” (BA). Full fine-tuning naturally represents both; LoRA conflates them. DoRA restores the independence at the cost of d additional trainable scalars — essentially zero overhead relative to the BA parameters.

Empirically, Liu et al. show that DoRA at r = 8 matches or exceeds LoRA at r = 16 across commonsense reasoning, instruction following, and image-text tasks. On the LLaMA-2 7B → 13B scale, DoRA at r = 8 closes approximately 80% of the remaining gap between LoRA and full fine-tuning, without the memory cost of higher-rank LoRA. The improvement is most pronounced on tasks requiring structured output and multi-step reasoning — precisely where directional and magnitude adaptation are most independent in the underlying gradient flow.

DoRA is backward compatible with QLoRA: the magnitude vector m and the BA adapter both operate over the NF4-quantized W₀, so QDoRA inherits QLoRA’s memory efficiency while narrowing the quality gap.

6. The Practical Decision Tree

Given the family of methods above, the decision path for a practitioner is determinate:

START
  │
  ├─ Budget: multiple A100/H100 nodes, unlimited time?
  │      └─ YES → Full Fine-Tuning
  │                (highest fidelity; needed for extreme domain shift
  │                 or new modalities; use gradient checkpointing)
  │
  ├─ Budget: 1–8 × 80 GB GPUs, BF16 base?
  │      └─ YES → Standard LoRA
  │                (r = 8–32 on Q, K, V, O; add MLP if knowledge injection needed)
  │                (α = r or 2r; train with learning rate ~2e-4)
  │
  ├─ Quality gap vs. full FT is unacceptable at LoRA r = 32?
  │      └─ YES → DoRA (drop-in replacement; same GPU budget)
  │
  └─ Single GPU or ≤ 48 GB?
         └─ YES → QLoRA (NF4 + double quantisation + paged optimizers)
                   (r = 64 recommended to compensate for quantisation noise)
                   If quality still insufficient → QDoRA

Rank selection heuristics:

One failure mode to internalize: LoRA adapters trained at rank r cannot be naively merged with adapters trained at different ranks without scaling adjustments. The α/r factor normalises this at inference, but when composing multiple adapters (e.g., LoRAHub or model merging), the user must ensure consistent α/r conventions across adapters or renormalise explicitly.

7. The Unifying View

The LoRA family is, at its core, a sequence of answers to the same question: what is the minimum parameterisation that preserves the expressive power of the update ΔW?

LoRA answers: rank. QLoRA adds: precision of the base. DoRA adds: the independence of magnitude and direction. Each innovation attacks a different binding constraint — trainable parameter count, GPU memory, and representational expressivity respectively — while preserving the core low-rank structure that makes parameter-efficient fine-tuning viable.

The roofline intuition from kernel programming applies here too. Just as the H100’s compute throughput is overprovisioned relative to HBM bandwidth — making memory movement, not arithmetic, the binding constraint — a pretrained language model’s representational capacity is overprovisioned relative to the low-rank subspace that task adaptation actually requires. LoRA, QLoRA, and DoRA are each exploiting a different form of that overprovisioning.

References

Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.

Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314.

Liu, S., Zhu, C., Liao, Q., & Bansal, M. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation. arXiv:2402.09353.

Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019). Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv:1909.08053.

Rajbhandari, S., Rasley, J., Ruwase, O., & He, Y. (2020). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. arXiv:1910.02054.

@article{fp4-2606020,
  title   = {The LoRA Family: A Mathematically Precise Guide to Parameter-Efficient Fine-Tuning},
  author  = {fp4 editorial desk},
  year    = {2026},
  url     = {https://fp4.dev/algorithm/lora-family/},
  journal = {fp4}
}