Attention, one matrix at a time
Attention is a soft dictionary lookup, and most of what makes it feel hard is bookkeeping about shapes. The cure is to do it once with numbers small enough to hold in your head.
The idea in one paragraph
Every token produces three vectors from its embedding $x$: a query $q = xW_Q$ (what am I looking for?), a key $k = xW_K$ (what do I contain?) and a value $v = xW_V$ (what do I hand over if you pick me?). The output for token $i$ is a weighted average of the values of the tokens it is allowed to see, where the weight is how well $q_i$ matches each $k_j$.
$$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}!\left(\frac{Q K^{\top}}{\sqrt{d_k}} + M\right) V$$
$M$ is the causal mask: $0$ where token $i$ may look at $j \le i$, and $-\infty$ elsewhere.
One matrix at a time
Take three tokens and $d_k = 2$:
$$Q=\begin{pmatrix}1&0\0&1\1&1\end{pmatrix},\quad K=\begin{pmatrix}1&0\1&1\0&1\end{pmatrix},\quad V=\begin{pmatrix}1&0\0&2\3&1\end{pmatrix}$$
Step 1, scores. Row $i$ of $QK^{\top}$ is $q_i \cdot k_j$ for every $j$:
$$QK^{\top}=\begin{pmatrix}1&1&0\0&1&1\1&2&1\end{pmatrix}$$
Step 2, scale by $\sqrt{2} \approx 1.414$:
$$\begin{pmatrix}0.707&0.707&0\0&0.707&0.707\0.707&1.414&0.707\end{pmatrix}$$
Step 3, mask. Token 0 sees only itself, token 1 sees tokens 0 and 1, token 2 sees all three. The forbidden entries become $-\infty$.
Step 4, softmax each row. Row 0 is $[1, 0, 0]$. Row 1 is $[e^{0}, e^{0.707}]/(1 + 2.028) = [0.330, 0.670]$. Row 2 is $[2.028, 4.113, 2.028]/8.170 = [0.248, 0.504, 0.248]$.
Step 5, mix the values.
- Token 0: $[1, 0]$. It can only copy itself.
- Token 1: $0.330,[1,0] + 0.670,[0,2] = [0.330,\ 1.340]$
- Token 2: $0.248,[1,0] + 0.504,[0,2] + 0.248,[3,1] = [0.993,\ 1.255]$
Notice that token 2 put the most weight on token 1 because its query $(1,1)$ lined up best with key $(1,1)$. That is the entire mechanism. Learning is just finding $W_Q, W_K, W_V$ so that useful tokens line up.
Why divide by $\sqrt{d_k}$
If entries of $q$ and $k$ have zero mean and unit variance, then $q \cdot k$ is a sum of $d_k$ products and has variance $d_k$. At $d_k = 64$ the scores have a standard deviation of 8, the softmax becomes nearly one-hot, and its gradients vanish. Dividing by $\sqrt{d_k}$ puts the variance back at 1.
What attention does not do
Attention alone has no idea where a token sits: shuffle the input tokens and, without position information, each token's output is the same up to the shuffle. That is why GPT-2 adds a learned position embedding to the token embedding, and why newer models rotate queries and keys by position (RoPE). Attention also only mixes information between tokens with a weighted average. The per-token nonlinear computation happens in the MLP that follows, which is where most of the parameters are.
Mask before the softmax, never after
Setting the forbidden scores to $-\infty$ before softmax gives them weight exactly 0 and renormalises the rest. Zeroing weights after the softmax leaves rows that no longer sum to 1, and the surviving weights still depended on tokens from the future in the denominator.
Multi-head, and what it costs
Run $h$ smaller attentions in parallel and concatenate. For GPT-2 small, $d_{model} = 768$, $h = 12$, $d_{head} = 64$. Parameters in the attention sublayer: four $768 \times 768$ matrices ($W_Q$, $W_K$, $W_V$ and the output projection), $4 \times 589{,}824 = 2{,}359{,}296$, plus biases.
The score matrix is $T \times T$ per head. At $T = 1024$ in fp16, one layer stores $1024^2 \times 12 \times 2 \approx 25$ MB per sequence just for attention weights. This quadratic memory is what FlashAttention (week 6) avoids materialising.
Code
import math
import torch
import torch.nn as nn
import torch.nn.functional as F
class CausalSelfAttention(nn.Module):
def __init__(self, d_model: int, n_head: int):
super().__init__()
assert d_model % n_head == 0
self.n_head = n_head
self.qkv = nn.Linear(d_model, 3 * d_model)
self.proj = nn.Linear(d_model, d_model)
def forward(self, x):
B, T, C = x.shape
q, k, v = self.qkv(x).split(C, dim=2)
q, k, v = (t.view(B, T, self.n_head, C // self.n_head).transpose(1, 2) for t in (q, k, v))
scores = q @ k.transpose(-2, -1) / math.sqrt(k.size(-1))
keep = torch.tril(torch.ones(T, T, device=x.device, dtype=torch.bool))
scores = scores.masked_fill(~keep, float("-inf"))
out = F.softmax(scores, dim=-1) @ v
out = out.transpose(1, 2).contiguous().view(B, T, C)
return self.proj(out)
x = torch.randn(2, 5, 32)
attn = CausalSelfAttention(32, 4)
y = attn(x)
x2 = x.clone()
x2[:, -1] += 1.0
assert torch.allclose(y[:, :-1], attn(x2)[:, :-1], atol=1e-6)
The last two lines are the test that would have caught most of my early bugs: change the last token and every earlier output must stay identical.
Common mistakes
- Scaling by $\sqrt{d_{model}}$ instead of $\sqrt{d_{head}}$.
- Softmax over the wrong dimension. It must normalise over keys (the last axis of the score matrix).
- Calling
.viewafter.transposewithout.contiguous(). - Applying the mask after the softmax.
- Forgetting that heads are independent: a bug that mixes heads usually shows up as a model that trains but plateaus higher.
Check yourself
- For the example, what would token 1's output be without the mask? Compute it.
- Why is the cost of the score matrix quadratic in $T$ but the cost of the projections linear?
- What changes in the output if you permute the input tokens and use no position information?
- How many parameters does one GPT-2 small block have, and which sublayer owns two thirds of them?