@omarUpdated Nov 3, 2026

Large Language Models, End to End

For developers who know Python and PyTorch: 12 weeks to understand how LLMs work and are trained. You build a tokenizer, a GPT, a LoRA fine-tune, an eval harness and a tiny pretrain, tune and evaluate pipeline.

Following gives you your own copy, dated from today. Free to start.

4 phases · 12 weeks

About this roadmap

For developers who can write Python and a PyTorch training loop and want to know what is actually inside an LLM. Twelve weeks, four phases, five things built by hand. Six to eight hours a week (ten in the capstone): learn early in the week, build late in it.

This is about how language models work and how they are trained. It is not about wiring a model into an app; the agents and RAG roadmap covers that.

What you will have built

  • Week 1: a byte-level BPE tokenizer that round-trips any Unicode string and reports its own compression.
  • Week 4: a GPT written from scratch, trained on tiny Shakespeare, with loss curves you can read.
  • Week 7: a LoRA fine-tune of a small open model, with a held-out before/after evaluation and a forgetting check.
  • Week 10: an eval harness that scores any checkpoint with perplexity, multiple-choice accuracy, a contamination check and a calibrated LLM judge.
  • Week 12: a tiny model pretrained, tuned and evaluated end to end, with a report.

How the weeks work

  1. The first half of the week is for watching, reading and deriving. The second half is for building. If you did not build it, you did not learn it.
  2. Every phase ends in a project week with a deliverable and a definition of done.
  3. Do the arithmetic yourself: parameter counts, FLOPs, bytes of memory. Most of the intuition in this field is back-of-envelope.
  4. Keep a lab notebook of runs. One line per run: what changed, what the loss did, what you think it means.

Failure modes to watch

  • Watching instead of typing. A lecture you did not code along with teaches you the vocabulary, not the system.
  • Reading a benchmark number as a fact. Ask what the model saw in training, how it was prompted and how big the error bars are.
  • Skipping the small experiment. Everything in weeks 5 to 10 can be reproduced at toy scale in minutes. Do that before you trust a paper's claim.
  • Tangents. Mixture of experts, multimodality and long-context tricks are real topics. They go on a reading list, not into this week.

Map

Every week on one canvas. Open a week to see its goals and resources.

What’s inside

12 weeks · 60 goals

Phase 1 · Text becomes a modelWeeks 1–4 · 20 goals
WK 1Tokenization and BPE5 goals

Learn why models read subword pieces, then build a byte-level BPE tokenizer and find where it breaks.

Goals

  • Write the pair-count and merge loop from memory and train 256 merges on a text file
  • Implement encode (apply merges in rank order) and decode (bytes back to UTF-8)
  • Pass a round-trip test on emoji, accents, newlines and a 1 MB file
  • Plot bytes per token against vocab size and compare with tiktoken's gpt2 encoding
  • List five tokenizer quirks (numbers, spaces, languages) and say which hurt a model
WK 2Embeddings and the LM objective5 goals

Turn next-token prediction into a number you can minimise, from a count table to a neural model, scored with cross-entropy and perplexity.

Goals

  • Derive cross-entropy with a one-hot target and perplexity = exp(loss) on paper
  • Build the count bigram and the one-layer neural bigram; match their average NLL
  • Add an embedding table and a context of 8 characters; beat the bigram on validation loss
  • Convert your best loss to bits per character and perplexity, and say what each means
  • Check that your first-step loss is close to ln(vocab size) and explain why
WK 3Attention and the transformer block5 goals

Derive scaled dot-product attention, then assemble heads, MLP, residuals and layer norm into the block every LLM repeats.

Goals

  • Work the 3-token attention example by hand, with the causal mask, before running code
  • Implement single-head causal attention and match F.scaled_dot_product_attention
  • Wrap it in multi-head attention with one fused QKV projection and write every tensor shape
  • Build a pre-norm block (LayerNorm, attention, residual, LayerNorm, MLP, residual)
  • Count the parameters of one GPT-2 small block by hand and confirm with sum(p.numel())
WK 4Project: a GPT from scratch5 goals

Assemble the pieces into a trainable GPT, train it on tiny Shakespeare and learn to read what the loss curve is telling you.

Goals

  • Assemble embeddings, N blocks, a final LayerNorm and a tied output head
  • Overfit a single batch to near-zero loss before touching the full dataset
  • Train with AdamW, warmup and cosine decay; log train and val loss every 250 steps
  • Write the sampling loop with temperature and top-k; save three samples at different temperatures
  • Diff your model against nanoGPT and note every difference and its reason
Phase 2 · Bigger, faster, specialisedWeeks 5–7 · 15 goals
WK 5Pretraining at scale5 goals

See how real pretraining runs are sized and run: compute budgets, scaling laws, data, mixed precision and the parallelism that spreads one model over many GPUs.

Goals

  • Derive 6ND and the 16 bytes per parameter rule on paper
  • Compute FLOPs, GPU-hours and memory for GPT-2 small on 10B tokens and compare with the lecture
  • Train four model widths at two token budgets and log final validation loss
  • Fit the power law on log-log axes and report the exponent with a rough error bar
  • Write the run planner (time and memory) and test it against a real run of yours
WK 6Inference: sampling and the KV cache5 goals

See why generation is memory-bound, then make it fast with sampling controls, the KV cache, FlashAttention, quantization and speculative decoding.

Goals

  • Implement temperature, top-k and top-p sampling and compare outputs at three settings
  • Add a KV cache and assert cached logits match a full recompute over 200 tokens
  • Benchmark tokens per second with and without the cache at lengths 64, 256 and 1024
  • Quantize linear weights to int8 per channel and measure the validation perplexity change
  • Compute KV-cache bytes per token for a 7B model and the speculative-decoding speedup at 80% acceptance
WK 7Project: fine-tune with SFT and LoRA5 goals

Adapt a pretrained model to one narrow task with supervised fine-tuning and low-rank adapters, then prove it helped and measure what it broke.

Goals

  • Pick one narrow task, build 2,000 to 5,000 training examples and 200 held-out ones
  • Write LoRALinear (frozen W, trainable A and B, scale alpha/r) and match PEFT's output
  • Train with loss masked to the response tokens only; log train and held-out loss
  • Sweep rank 4, 16 and 64 and plot held-out score against trainable parameters
  • Measure general-text perplexity before and after to quantify forgetting
Phase 3 · Teaching it to behaveWeeks 8–10 · 15 goals
WK 8Alignment: RLHF and DPO5 goals

Understand how a base model becomes an assistant: reward models, KL-regularised RL, and DPO, which reaches the same optimum without a reward model.

Goals

  • Derive the DPO loss from the KL-regularised objective and the Bradley-Terry model
  • Compute one DPO loss and margin by hand for the example pair, then in code
  • Implement dpo_loss on sequence log-probs and match TRL on one batch
  • Train on 2,000 preference pairs; plot margin, accuracy and KL to the reference
  • Compare 20 prompts before and after; write down verbosity and tone drift
WK 9Reasoning and test-time compute5 goals

Learn why spending more tokens at inference helps, and how reinforcement learning on verifiable rewards trains models to think before answering.

Goals

  • Write the GRPO advantage (reward minus group mean over group std) and why no critic is needed
  • Evaluate greedy accuracy on 200 GSM8K problems with a fixed prompt and answer parser
  • Sample n = 4 and 16 answers per problem; compute majority-vote accuracy
  • Plot accuracy against samples and against total tokens; mark where it saturates
  • Implement grpo_advantages with a unit test on a hand-computed group
WK 10Project: evaluation you can trust5 goals

Measure models properly: perplexity, benchmarks and how they leak, human preference, and LLM judges that you calibrate against your own labels.

Goals

  • Implement held-out perplexity and a log-likelihood multiple-choice scorer
  • Write an n-gram overlap contamination check and run it between a benchmark and your training data
  • Build a pairwise judge with position swapping; label 50 pairs yourself
  • Report agreement and bootstrap confidence intervals for every metric
  • Write the three biggest ways this harness could still mislead you
Phase 4 · Open the box, then shipWeeks 11–12 · 10 goals
WK 11Interpretability: features, circuits5 goals

Look inside: the residual stream, induction heads, superposition, and sparse autoencoders that pull readable features out of activations.

Goals

  • Explain the residual stream as a shared bus that each layer reads and writes
  • Train the toy superposition model and plot the learned feature directions at three sparsities
  • Run GPT-2 small in TransformerLens and detect induction behaviour on repeated random tokens
  • Train an SAE on residual activations; report L0, reconstruction error and loss recovered
  • Inspect top-activating contexts for five features and label them honestly, including the unclear ones
WK 12Capstone: pretrain, tune, evaluate5 goals

Run the whole pipeline once at small scale: pretrain a tiny model, fine-tune it, evaluate every stage with your own harness and write up what you found.

Goals

  • Write a one-page plan: size, token budget, predicted loss from your scaling fit, evals
  • Prepare the corpus: tokenize with your BPE or a reference tokenizer and shard to disk
  • Pretrain to the token budget with checkpoints and logged curves
  • Fine-tune a copy and run the week-10 harness on both checkpoints
  • Write the report: curves, predicted versus actual loss, samples, evals, one probe, one change