# Attention, one head, by hand

By @omar · Published Nov 2, 2026 · From the roadmap [Deep Learning from Scratch: backprop to diffusion](https://ornix.io/u/omar/deep-learning-from-scratch)

Every token in a sequence emits three vectors. A **query** says what it is looking for, a **key** says what it contains, and a **value** says what it will hand over if chosen. Attention scores every query against every key, turns the scores into weights, and returns the weighted average of the values. It is a differentiable dictionary lookup that returns a blend instead of one entry.

## The formula, then the numbers

$$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{Q K^{\top}}{\sqrt{d_k}} + M\right) V$$

$M$ is the causal mask: $0$ on and below the diagonal, $-\infty$ above it. Three tokens with $d_k = 2$:

$$Q = K = \begin{pmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{pmatrix}, \qquad V = \begin{pmatrix} 1 & 2 \\ 3 & 4 \\ 5 & 6 \end{pmatrix}$$

The raw scores $QK^{\top}$ are $\begin{pmatrix}1&0&1\\0&1&1\\1&1&2\end{pmatrix}$. Divide by $\sqrt 2 = 1.414$ and apply the mask. Row by row:

- **Token 1** can only see itself: weights $[1, 0, 0]$, output $[1, 2]$.
- **Token 2** sees tokens 1 and 2 with scores $[0, 0.707]$. Softmax gives $e^0 = 1$ and $e^{0.707} = 2.03$, so the weights are $[0.330, 0.670]$. Output: $0.330\,[1, 2] + 0.670\,[3, 4] = [2.34, 3.34]$.
- **Token 3** sees all three with scores $[0.707, 0.707, 1.414]$, weights $[0.248, 0.248, 0.503]$ and output $[3.51, 4.51]$.

Notice what masking did: the weights in each row sum to 1 over the allowed positions only. The masked scores were set to $-\infty$ **before** the softmax, so they contribute $e^{-\infty} = 0$ to the denominator.

```mermaid
%% ornix:id=attn0002
flowchart LR
    X["x: B x T x C"] --> QKV["one Linear to Q, K, V"]
    QKV --> Q["Q: B x h x T x d"]
    QKV --> K["K: B x h x T x d"]
    QKV --> V["V: B x h x T x d"]
    Q --> S["scores = Q K^T / sqrt(d)"]
    K --> S
    S --> M["mask future, softmax"]
    M --> O["weights @ V"]
    V --> O
    O --> P["merge heads, output Linear"]
```

## Why the square root

Suppose $q$ and $k$ have independent entries with zero mean and unit variance. Then $q \cdot k = \sum_{m=1}^{d_k} q_m k_m$ has variance $d_k$. For $d_k = 64$ the standard deviation of a score is 8; a quick simulation of 100,000 random pairs gives a variance of 63.5. A softmax over scores like $[8, 0, -8]$ returns $[0.9997, 0.0003, 0.0000001]$, almost one-hot, and its gradient $p(1 - p)$ for the winning entry is about $3 \times 10^{-4}$. Training stalls. Dividing by $\sqrt{d_k}$ brings the score variance back to 1, where the softmax still has room to move.

## Heads and shapes

One attention head can only learn one pattern of "who looks at whom". Multi-head attention runs $h$ heads in parallel, each with $d_k = d_{model}/h$, concatenates their outputs and mixes them with an output projection. It costs the same as one full-width head and lets different heads specialise, for example one following the previous token and another following a matching bracket.

The cost that matters is the score matrix: $T \times T$ per head, so compute and memory grow with the **square** of the sequence length. Everything about long-context engineering comes from this fact.

Attention on its own has no notion of order: permute the tokens and the outputs permute the same way. Position information has to be added, either as learned or sinusoidal vectors added to the embeddings, or as rotations of $q$ and $k$ in modern models. The causal mask does leak some order implicitly, but explicit positions are far more reliable.

## A block you can test

```python
import torch
import torch.nn as nn


class CausalSelfAttention(nn.Module):
    def __init__(self, d_model, n_heads):
        super().__init__()
        self.h = n_heads
        self.qkv = nn.Linear(d_model, 3 * d_model)
        self.proj = nn.Linear(d_model, d_model)

    def forward(self, x):
        B, T, C = x.shape
        q, k, v = self.qkv(x).chunk(3, dim=-1)
        q, k, v = (t.view(B, T, self.h, C // self.h).transpose(1, 2) for t in (q, k, v))
        att = (q @ k.transpose(-2, -1)) / (C // self.h) ** 0.5
        keep = torch.tril(torch.ones(T, T, dtype=torch.bool, device=x.device))
        att = att.masked_fill(~keep, float("-inf")).softmax(dim=-1)
        y = (att @ v).transpose(1, 2).reshape(B, T, C)
        return self.proj(y)


m = CausalSelfAttention(32, 4)
x = torch.randn(1, 5, 32)
x2 = x.clone()
x2[:, 4] += 1.0
print(torch.allclose(m(x)[:, :4], m(x2)[:, :4], atol=1e-6))
```

The last line is the most useful test in this roadmap: changing a future token must not change any earlier output. It should print `True`. If it prints `False` your mask is in the wrong place, and your language model will report a validation loss that is too good to be true.

## Common mistakes

- **Masking after the softmax.** The future scores are then inside the normalizer, so the weights still depend on the future and no longer sum to 1.
- **Forgetting the transpose before merging heads.** `view` after `transpose` either errors or silently scrambles the heads; use `reshape` or `contiguous`.
- **Dropping the scale.** Training still starts, then plateaus at a worse loss.
- **Reading attention maps as explanations.** A weight says where information came from, not that the model "used" it for the answer.

## Check yourself

1. Compute the output for token 2 in the example if the mask were applied **after** the softmax. Do the weights sum to 1?
2. Show that $\mathrm{Var}(q \cdot k) = d_k$ for unit-variance independent entries.
3. What shapes do $Q$, $K$ and the score matrix have for batch 8, 12 heads, 128 tokens and $d_{model} = 768$?
4. Why does a transformer need positional information while an LSTM does not?

---

Roadmap: https://ornix.io/u/omar/deep-learning-from-scratch (Markdown: https://ornix.io/u/omar/deep-learning-from-scratch.md)

Canonical page: https://ornix.io/u/omar/deep-learning-from-scratch/attention-one-head-by-hand
