Attention, one head, by hand

Every token in a sequence emits three vectors. A query says what it is looking for, a key says what it contains, and a value says what it will hand over if chosen. Attention scores every query against every key, turns the scores into weights, and returns the weighted average of the values. It is a differentiable dictionary lookup that returns a blend instead of one entry.

The formula, then the numbers

$$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}!\left(\frac{Q K^{\top}}{\sqrt{d_k}} + M\right) V$$

$M$ is the causal mask: $0$ on and below the diagonal, $-\infty$ above it. Three tokens with $d_k = 2$:

$$Q = K = \begin{pmatrix} 1 & 0 \ 0 & 1 \ 1 & 1 \end{pmatrix}, \qquad V = \begin{pmatrix} 1 & 2 \ 3 & 4 \ 5 & 6 \end{pmatrix}$$

The raw scores $QK^{\top}$ are $\begin{pmatrix}1&0&1\0&1&1\1&1&2\end{pmatrix}$. Divide by $\sqrt 2 = 1.414$ and apply the mask. Row by row:

  • Token 1 can only see itself: weights $[1, 0, 0]$, output $[1, 2]$.
  • Token 2 sees tokens 1 and 2 with scores $[0, 0.707]$. Softmax gives $e^0 = 1$ and $e^{0.707} = 2.03$, so the weights are $[0.330, 0.670]$. Output: $0.330,[1, 2] + 0.670,[3, 4] = [2.34, 3.34]$.
  • Token 3 sees all three with scores $[0.707, 0.707, 1.414]$, weights $[0.248, 0.248, 0.503]$ and output $[3.51, 4.51]$.

Notice what masking did: the weights in each row sum to 1 over the allowed positions only. The masked scores were set to $-\infty$ before the softmax, so they contribute $e^{-\infty} = 0$ to the denominator.

Why the square root

Suppose $q$ and $k$ have independent entries with zero mean and unit variance. Then $q \cdot k = \sum_{m=1}^{d_k} q_m k_m$ has variance $d_k$. For $d_k = 64$ the standard deviation of a score is 8; a quick simulation of 100,000 random pairs gives a variance of 63.5. A softmax over scores like $[8, 0, -8]$ returns $[0.9997, 0.0003, 0.0000001]$, almost one-hot, and its gradient $p(1 - p)$ for the winning entry is about $3 \times 10^{-4}$. Training stalls. Dividing by $\sqrt{d_k}$ brings the score variance back to 1, where the softmax still has room to move.

Heads and shapes

One attention head can only learn one pattern of "who looks at whom". Multi-head attention runs $h$ heads in parallel, each with $d_k = d_{model}/h$, concatenates their outputs and mixes them with an output projection. It costs the same as one full-width head and lets different heads specialise, for example one following the previous token and another following a matching bracket.

The cost that matters is the score matrix: $T \times T$ per head, so compute and memory grow with the square of the sequence length. Everything about long-context engineering comes from this fact.

Attention on its own has no notion of order: permute the tokens and the outputs permute the same way. Position information has to be added, either as learned or sinusoidal vectors added to the embeddings, or as rotations of $q$ and $k$ in modern models. The causal mask does leak some order implicitly, but explicit positions are far more reliable.

A block you can test

import torch
import torch.nn as nn


class CausalSelfAttention(nn.Module):
    def __init__(self, d_model, n_heads):
        super().__init__()
        self.h = n_heads
        self.qkv = nn.Linear(d_model, 3 * d_model)
        self.proj = nn.Linear(d_model, d_model)

    def forward(self, x):
        B, T, C = x.shape
        q, k, v = self.qkv(x).chunk(3, dim=-1)
        q, k, v = (t.view(B, T, self.h, C // self.h).transpose(1, 2) for t in (q, k, v))
        att = (q @ k.transpose(-2, -1)) / (C // self.h) ** 0.5
        keep = torch.tril(torch.ones(T, T, dtype=torch.bool, device=x.device))
        att = att.masked_fill(~keep, float("-inf")).softmax(dim=-1)
        y = (att @ v).transpose(1, 2).reshape(B, T, C)
        return self.proj(y)


m = CausalSelfAttention(32, 4)
x = torch.randn(1, 5, 32)
x2 = x.clone()
x2[:, 4] += 1.0
print(torch.allclose(m(x)[:, :4], m(x2)[:, :4], atol=1e-6))

The last line is the most useful test in this roadmap: changing a future token must not change any earlier output. It should print True. If it prints False your mask is in the wrong place, and your language model will report a validation loss that is too good to be true.

Common mistakes

  • Masking after the softmax. The future scores are then inside the normalizer, so the weights still depend on the future and no longer sum to 1.
  • Forgetting the transpose before merging heads. view after transpose either errors or silently scrambles the heads; use reshape or contiguous.
  • Dropping the scale. Training still starts, then plateaus at a worse loss.
  • Reading attention maps as explanations. A weight says where information came from, not that the model "used" it for the answer.

Check yourself

  1. Compute the output for token 2 in the example if the mask were applied after the softmax. Do the weights sum to 1?
  2. Show that $\mathrm{Var}(q \cdot k) = d_k$ for unit-variance independent entries.
  3. What shapes do $Q$, $K$ and the score matrix have for batch 8, 12 heads, 128 tokens and $d_{model} = 768$?
  4. Why does a transformer need positional information while an LSTM does not?