Large Language Models, End to End
For developers who know Python and PyTorch: 12 weeks to understand how LLMs work and are trained. You build a tokenizer, a GPT, a LoRA fine-tune, an eval harness and a tiny pretrain, tune and evaluate pipeline.
Following gives you your own copy, dated from today. Free to start.

About this roadmap
For developers who can write Python and a PyTorch training loop and want to know what is actually inside an LLM. Twelve weeks, four phases, five things built by hand. Six to eight hours a week (ten in the capstone): learn early in the week, build late in it.
This is about how language models work and how they are trained. It is not about wiring a model into an app; the agents and RAG roadmap covers that.
What you will have built
- Week 1: a byte-level BPE tokenizer that round-trips any Unicode string and reports its own compression.
- Week 4: a GPT written from scratch, trained on tiny Shakespeare, with loss curves you can read.
- Week 7: a LoRA fine-tune of a small open model, with a held-out before/after evaluation and a forgetting check.
- Week 10: an eval harness that scores any checkpoint with perplexity, multiple-choice accuracy, a contamination check and a calibrated LLM judge.
- Week 12: a tiny model pretrained, tuned and evaluated end to end, with a report.
How the weeks work
- The first half of the week is for watching, reading and deriving. The second half is for building. If you did not build it, you did not learn it.
- Every phase ends in a project week with a deliverable and a definition of done.
- Do the arithmetic yourself: parameter counts, FLOPs, bytes of memory. Most of the intuition in this field is back-of-envelope.
- Keep a lab notebook of runs. One line per run: what changed, what the loss did, what you think it means.
Failure modes to watch
- Watching instead of typing. A lecture you did not code along with teaches you the vocabulary, not the system.
- Reading a benchmark number as a fact. Ask what the model saw in training, how it was prompted and how big the error bars are.
- Skipping the small experiment. Everything in weeks 5 to 10 can be reproduced at toy scale in minutes. Do that before you trust a paper's claim.
- Tangents. Mixture of experts, multimodality and long-context tricks are real topics. They go on a reading list, not into this week.
Map
Every week on one canvas. Open a week to see its goals and resources.
What’s inside
12 weeks · 60 goals
Phase 1 · Text becomes a modelWeeks 1–4 · 20 goals
WK 1Tokenization and BPE5 goals
Learn why models read subword pieces, then build a byte-level BPE tokenizer and find where it breaks.
Goals
- Write the pair-count and merge loop from memory and train 256 merges on a text file
- Implement encode (apply merges in rank order) and decode (bytes back to UTF-8)
- Pass a round-trip test on emoji, accents, newlines and a 1 MB file
- Plot bytes per token against vocab size and compare with tiktoken's gpt2 encoding
- List five tokenizer quirks (numbers, spaces, languages) and say which hurt a model
WK 2Embeddings and the LM objective5 goals
Turn next-token prediction into a number you can minimise, from a count table to a neural model, scored with cross-entropy and perplexity.
Goals
- Derive cross-entropy with a one-hot target and perplexity = exp(loss) on paper
- Build the count bigram and the one-layer neural bigram; match their average NLL
- Add an embedding table and a context of 8 characters; beat the bigram on validation loss
- Convert your best loss to bits per character and perplexity, and say what each means
- Check that your first-step loss is close to ln(vocab size) and explain why
Resources
WK 3Attention and the transformer block5 goals
Derive scaled dot-product attention, then assemble heads, MLP, residuals and layer norm into the block every LLM repeats.
Goals
- Work the 3-token attention example by hand, with the causal mask, before running code
- Implement single-head causal attention and match F.scaled_dot_product_attention
- Wrap it in multi-head attention with one fused QKV projection and write every tensor shape
- Build a pre-norm block (LayerNorm, attention, residual, LayerNorm, MLP, residual)
- Count the parameters of one GPT-2 small block by hand and confirm with sum(p.numel())
WK 4Project: a GPT from scratch5 goals
Assemble the pieces into a trainable GPT, train it on tiny Shakespeare and learn to read what the loss curve is telling you.
Goals
- Assemble embeddings, N blocks, a final LayerNorm and a tied output head
- Overfit a single batch to near-zero loss before touching the full dataset
- Train with AdamW, warmup and cosine decay; log train and val loss every 250 steps
- Write the sampling loop with temperature and top-k; save three samples at different temperatures
- Diff your model against nanoGPT and note every difference and its reason
Phase 2 · Bigger, faster, specialisedWeeks 5–7 · 15 goals
WK 5Pretraining at scale5 goals
See how real pretraining runs are sized and run: compute budgets, scaling laws, data, mixed precision and the parallelism that spreads one model over many GPUs.
Goals
- Derive 6ND and the 16 bytes per parameter rule on paper
- Compute FLOPs, GPU-hours and memory for GPT-2 small on 10B tokens and compare with the lecture
- Train four model widths at two token budgets and log final validation loss
- Fit the power law on log-log axes and report the exponent with a rough error bar
- Write the run planner (time and memory) and test it against a real run of yours
WK 6Inference: sampling and the KV cache5 goals
See why generation is memory-bound, then make it fast with sampling controls, the KV cache, FlashAttention, quantization and speculative decoding.
Goals
- Implement temperature, top-k and top-p sampling and compare outputs at three settings
- Add a KV cache and assert cached logits match a full recompute over 200 tokens
- Benchmark tokens per second with and without the cache at lengths 64, 256 and 1024
- Quantize linear weights to int8 per channel and measure the validation perplexity change
- Compute KV-cache bytes per token for a 7B model and the speculative-decoding speedup at 80% acceptance
Resources
- The Curious Case of Neural Text Degenerationpaper
- Transformer Inference Arithmeticblog
- Coding LLaMA 2 from scratch in PyTorch (KV cache, GQA, RoPE, RMSNorm)video
- Quantization explained with PyTorchvideo
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesspaper
- Fast Inference from Transformers via Speculative Decodingpaper
WK 7Project: fine-tune with SFT and LoRA5 goals
Adapt a pretrained model to one narrow task with supervised fine-tuning and low-rank adapters, then prove it helped and measure what it broke.
Goals
- Pick one narrow task, build 2,000 to 5,000 training examples and 200 held-out ones
- Write LoRALinear (frozen W, trainable A and B, scale alpha/r) and match PEFT's output
- Train with loss masked to the response tokens only; log train and held-out loss
- Sweep rank 4, 16 and 64 and plot held-out score against trainable parameters
- Measure general-text perplexity before and after to quantify forgetting
Phase 3 · Teaching it to behaveWeeks 8–10 · 15 goals
WK 8Alignment: RLHF and DPO5 goals
Understand how a base model becomes an assistant: reward models, KL-regularised RL, and DPO, which reaches the same optimum without a reward model.
Goals
- Derive the DPO loss from the KL-regularised objective and the Bradley-Terry model
- Compute one DPO loss and margin by hand for the example pair, then in code
- Implement dpo_loss on sequence log-probs and match TRL on one batch
- Train on 2,000 preference pairs; plot margin, accuracy and KL to the reference
- Compare 20 prompts before and after; write down verbosity and tone drift
Resources
- Deep Dive into LLMs like ChatGPTvideo
- Training language models to follow instructions with human feedbackpaper
- Direct Preference Optimizationpaper
- Illustrating Reinforcement Learning from Human Feedbackblog
- Direct Preference Optimization explained (Umar Jamil)video
- Reward Hacking in Reinforcement Learningblog
WK 9Reasoning and test-time compute5 goals
Learn why spending more tokens at inference helps, and how reinforcement learning on verifiable rewards trains models to think before answering.
Goals
- Write the GRPO advantage (reward minus group mean over group std) and why no critic is needed
- Evaluate greedy accuracy on 200 GSM8K problems with a fixed prompt and answer parser
- Sample n = 4 and 16 answers per problem; compute majority-vote accuracy
- Plot accuracy against samples and against total tokens; mark where it saturates
- Implement grpo_advantages with a unit test on a hand-computed group
Resources
WK 10Project: evaluation you can trust5 goals
Measure models properly: perplexity, benchmarks and how they leak, human preference, and LLM judges that you calibrate against your own labels.
Goals
- Implement held-out perplexity and a log-likelihood multiple-choice scorer
- Write an n-gram overlap contamination check and run it between a benchmark and your training data
- Build a pairwise judge with position swapping; label 50 pairs yourself
- Report agreement and bootstrap confidence intervals for every metric
- Write the three biggest ways this harness could still mislead you
Resources
- Measuring Massive Multitask Language Understandingpaper
- Rethinking Benchmark and Contamination for Language Models with Rephrased Samplespaper
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenapaper
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencepaper
- EleutherAI/lm-evaluation-harnesscode
- Your AI Product Needs Evalsblog
Phase 4 · Open the box, then shipWeeks 11–12 · 10 goals
WK 11Interpretability: features, circuits5 goals
Look inside: the residual stream, induction heads, superposition, and sparse autoencoders that pull readable features out of activations.
Goals
- Explain the residual stream as a shared bus that each layer reads and writes
- Train the toy superposition model and plot the learned feature directions at three sparsities
- Run GPT-2 small in TransformerLens and detect induction behaviour on repeated random tokens
- Train an SAE on residual activations; report L0, reconstruction error and loss recovered
- Inspect top-activating contexts for five features and label them honestly, including the unclear ones
WK 12Capstone: pretrain, tune, evaluate5 goals
Run the whole pipeline once at small scale: pretrain a tiny model, fine-tune it, evaluate every stage with your own harness and write up what you found.
Goals
- Write a one-page plan: size, token budget, predicted loss from your scaling fit, evals
- Prepare the corpus: tokenize with your BPE or a reference tokenizer and shard to disk
- Pretrain to the token budget with checkpoints and logged curves
- Fine-tune a copy and run the week-10 harness on both checkpoints
- Write the report: curves, predicted versus actual loss, samples, evals, one probe, one change
Resources
- karpathy/nanochatcode
- TinyStories: How Small Can Language Models Be and Still Speak Coherent English?paper
- SmolLM2: When Smol Goes Bigpaper
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalepaper
- TinyStories datasetdataset
- Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingpaper
Notes
8 published notes
Superposition and sparse autoencoders
Nov 2, 2026- Nov 2, 2026
- Nov 2, 2026
Why scaling laws bend (Chinchilla)
Nov 2, 2026- Nov 2, 2026
Attention, one matrix at a time
Nov 2, 2026- Nov 2, 2026