# LoRA in one page of math

By @omar · Published Nov 2, 2026 · From the roadmap [Large Language Models, End to End](https://ornix.io/u/omar/large-language-models-end-to-end)

Full fine-tuning updates every weight, which means storing gradients and optimizer state for every weight: about 16 bytes per parameter with Adam in mixed precision. For a 6.7B model that is roughly 108 GB before activations. LoRA gets most of the benefit by training a tiny correction and leaving the model frozen.

## The idea

A linear layer computes $y = Wx$ with $W \in \mathbb{R}^{d \times k}$. LoRA freezes $W$ and learns the update as a product of two thin matrices:

$$y = Wx + \frac{\alpha}{r} B A x,\qquad B \in \mathbb{R}^{d \times r},\ A \in \mathbb{R}^{r \times k},\ r \ll \min(d, k)$$

The bet is that the *change* a task needs lives in a low-dimensional subspace, even though $W$ itself is full rank.

```mermaid
%% ornix:id=lora0006
flowchart LR
    X["x"] --> W["W (frozen)"]
    X --> A["A: r x k (trainable)"]
    A --> B["B: d x r (trainable, starts at 0)"]
    B --> S["scale alpha / r"]
    W --> P(("+"))
    S --> P
    P --> Y["y"]
```

## A tiny example

Let $d = k = 3$, $r = 1$, $\alpha = r$ so the scale is 1:

$$B = \begin{pmatrix}1\\0\\2\end{pmatrix},\quad A = \begin{pmatrix}1&2&0\end{pmatrix},\quad BA = \begin{pmatrix}1&2&0\\0&0&0\\2&4&0\end{pmatrix}$$

For $x = (1,1,1)^{\top}$: $Ax = 3$, so the LoRA branch outputs $B \cdot 3 = (3, 0, 6)^{\top}$. The update matrix has rank 1 however large $d$ and $k$ are.

## Counting parameters

Full update: $dk$ parameters. LoRA: $r(d + k)$.

For one $4096 \times 4096$ projection with $r = 16$: full is $16{,}777{,}216$, LoRA is $16 \times 8192 = 131{,}072$, which is $0.78\%$.

For Llama 7B with LoRA ($r = 8$) on only the query and value projections: per matrix $8 \times 8192 = 65{,}536$; two matrices in each of 32 layers gives $4{,}194{,}304$ trainable parameters, about 0.062% of the 6.74B total. This is the number PEFT prints for that configuration.

## Why it starts safely

$B$ is initialised to zero and $A$ to small random values, so $BA = 0$ at step 0 and the model begins exactly as the pretrained one. Let $s = \alpha / r$, $g = \partial L / \partial y$:

$$\frac{\partial L}{\partial B} = s\, g\, (Ax)^{\top},\qquad \frac{\partial L}{\partial A} = s\, B^{\top} g\, x^{\top}$$

At step 0, $B = 0$ makes the second gradient zero, so $B$ moves first and $A$ follows once $B$ is non-zero. After training you can merge: $W' = W + sBA$. The merged model has the same size and latency as the base model. Keeping adapters separate lets you swap many tasks over one base.

## QLoRA

QLoRA keeps the frozen base weights in 4-bit NormalFloat (a data type matched to the roughly normal distribution of weights), quantizes the quantization constants again to save more ("double quantization"), and pages optimizer state to CPU when GPU memory spikes. The adapters stay in higher precision. For the 6.7B model, 4-bit weights take about 3.4 GB instead of 13.5 GB.

## Choosing rank and alpha

Treat $r$ as capacity and $\alpha$ as a learning-rate-like multiplier. A common starting point is $r = 16$ with $\alpha = 2r$ (so the scale is 2), applied to every linear layer. If the fine-tune underfits, raise the rank or the learning rate before anything else. If it forgets general behaviour, lower the learning rate, train for fewer steps, or mix some general instruction data back in. Keep one number fixed while you sweep the other, or you will not know which one mattered.

## What the evidence says

From the QLoRA and LoRA literature and Thinking Machines' "LoRA Without Regret":

- On small to medium instruction-style datasets, LoRA can match full fine-tuning.
- Apply it to every linear layer, including the MLP, not only attention. Attention-only LoRA underperforms.
- The best learning rate is often around ten times higher than for full fine-tuning.
- LoRA loses when the dataset holds more information than the adapter has capacity for.
- Rank matters less than people fear in the small-data regime. Sweep 4, 16, 64 and look.

## Code

```python
import math
import torch
import torch.nn as nn


class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r: int = 16, alpha: int = 32):
        super().__init__()
        self.base = base
        for p in self.base.parameters():
            p.requires_grad = False
        self.A = nn.Parameter(torch.empty(r, base.in_features))
        self.B = nn.Parameter(torch.zeros(base.out_features, r))
        nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
        self.scale = alpha / r

    def forward(self, x):
        return self.base(x) + (x @ self.A.T @ self.B.T) * self.scale

    @torch.no_grad()
    def merge(self):
        self.base.weight += self.scale * (self.B @ self.A)
        return self.base


layer = nn.Linear(64, 64)
lora = LoRALinear(layer, r=4, alpha=8)
x = torch.randn(3, 64)
assert torch.allclose(lora(x), layer(x))
lora.B.data.normal_()
y = lora(x)
merged = lora.merge()
assert torch.allclose(y, merged(x), atol=1e-4)
```

## Common mistakes

- Forgetting to freeze the base, so you train everything and wonder about memory.
- Using the wrong scale. The factor is $\alpha / r$, not $\alpha$.
- Computing loss on prompt tokens too. For SFT, mask the prompt so the loss covers only the response.
- Initialising $B$ randomly, which perturbs the model before any training.
- Merging adapters into a quantized base without dequantizing first.
- Judging by training loss. Hold out examples and a general-text perplexity check for forgetting.

## Check yourself

- How many trainable parameters does rank 8 add to a $2048 \times 8192$ layer?
- Why does zero-initialising $B$ (and not $A$) give a clean start while still allowing learning?
- When would you prefer to keep adapters separate rather than merge them?
- Name two situations where LoRA is likely to lose to full fine-tuning.

---

Roadmap: https://ornix.io/u/omar/large-language-models-end-to-end (Markdown: https://ornix.io/u/omar/large-language-models-end-to-end.md)

Canonical page: https://ornix.io/u/omar/large-language-models-end-to-end/lora-in-one-page-of-math
