Reading a loss curve
A loss curve is the cheapest instrument you own. Read it well and it tells you whether the model is learning, whether the code is wrong, and which knob to turn next. Most training time is lost to not looking.
Start with the number at step zero
A model that knows nothing assigns equal probability to every token, so its loss is $\ln V$ for vocabulary size $V$:
- Character model, $V = 65$: $\ln 65 = 4.17$ nats.
- GPT-2 vocabulary, $V = 50{,}257$: $\ln 50257 = 10.82$ nats.
If your first printed loss is close to that, initialisation is sane. If it is 4.9 on a 65-character model, the output logits are too large at init and the model starts confidently wrong. If it is 0.1, labels are leaking into the input.
Conversions you will use constantly. Loss in nats $\ell$ becomes perplexity $e^{\ell}$ and bits per token $\ell / \ln 2$. A loss of 1.5 nats is perplexity 4.48 (as if the model were choosing among about 4.5 equally likely tokens) and 2.16 bits per character.
The curve, phase by phase
Fast initial drop. The first hundred steps learn unigram and bigram statistics: spaces, common letters, common words. The drop is large and means little.
Slow power-law decline. The long middle of a healthy run is a roughly straight line on log-log axes. This is where scaling laws live.
Plateaus followed by drops. Sometimes loss sits flat and then falls fast. In small transformers this often marks a capability appearing, such as induction heads that copy earlier text (week 11).
Spikes. A single spike that recovers is usually a bad batch. Spikes that grow are a learning rate that is too high, missing gradient clipping, or lower-precision overflow.
Train versus validation
For tiny Shakespeare, the corpus is about 1.1M characters and a 10M-parameter model can memorise it. The tell is a gap: train loss keeps falling while validation turns upward. A small gap (a few hundredths) is normal. A widening gap means you are fitting noise.
If validation loss is lower than training loss early on, check that dropout is off in eval and that train loss is not an average over a changing window.
A worked reading
Suppose a run logs, every 250 steps (train / val):
| step | train | val |
|---|---|---|
| 0 | 4.19 | 4.19 |
| 250 | 2.45 | 2.48 |
| 1000 | 1.62 | 1.78 |
| 2500 | 1.18 | 1.60 |
| 5000 | 0.74 | 1.95 |
Step 0 is within 0.02 of $\ln 65 = 4.17$: init is fine. Through step 1000 the gap is 0.16 and healthy. By step 2500 the gap is 0.42 and validation has reached its minimum around 1.60. By step 5000 validation rose to 1.95 while train kept falling: classic overfitting. The right move is to keep the step-2500 checkpoint and add dropout or reduce model size, not to train longer.
Code
import torch
@torch.no_grad()
def estimate_loss(model, get_batch, eval_iters=50):
model.eval()
out = {}
for split in ("train", "val"):
losses = torch.zeros(eval_iters)
for i in range(eval_iters):
x, y = get_batch(split)
_, loss = model(x, y)
losses[i] = loss.item()
out[split] = losses.mean().item()
model.train()
return out
def ema(values, beta=0.98):
smooth, avg = [], values[0]
for v in values:
avg = beta * avg + (1 - beta) * v
smooth.append(avg)
return smooth
Always evaluate on many batches (50 here). A single validation batch gives noise of the same size as the effects you are trying to see. Plot the raw curve faintly under the smoothed one so you can still see spikes.
The debugging order
- Print the loss at step 0 and compare with $\ln V$.
- Overfit one batch to near zero. If you cannot, the bug is in the model or the loss, not the data.
- Train on a small slice and confirm validation tracks train.
- Only now sweep the learning rate. It is the most sensitive knob: try 3x steps in each direction.
- Change one thing per run and write a line in your log.
Common mistakes
- Comparing train and validation losses computed differently (one averaged over a window, the other a fresh batch).
- Smoothing so heavily that spikes vanish.
- Judging a run by the last value rather than the best checkpoint.
- Changing the data split between runs, then comparing validation losses.
Check yourself
- What loss should a uniform predictor get over a 4,096-token vocabulary?
- Why does a perfect model not reach loss 0 on natural text?
- Your validation loss is flat but training loss falls. Name two causes and a test for each.
- Why evaluate in
model.eval()mode underno_grad?