# Large Language Models, End to End

By @omar · 12 weeks

> For developers who know Python and PyTorch: 12 weeks to understand how LLMs work and are trained. You build a tokenizer, a GPT, a LoRA fine-tune, an eval harness and a tiny pretrain, tune and evaluate pipeline.

## About

**For developers who can write Python and a PyTorch training loop and want to know what is actually inside an LLM.** Twelve weeks, four phases, five things built by hand. Six to eight hours a week (ten in the capstone): learn early in the week, build late in it.

This is about how language models work and how they are trained. It is not about wiring a model into an app; the agents and RAG roadmap covers that.

### What you will have built

- **Week 1:** a byte-level BPE tokenizer that round-trips any Unicode string and reports its own compression.
- **Week 4:** a GPT written from scratch, trained on tiny Shakespeare, with loss curves you can read.
- **Week 7:** a LoRA fine-tune of a small open model, with a held-out before/after evaluation and a forgetting check.
- **Week 10:** an eval harness that scores any checkpoint with perplexity, multiple-choice accuracy, a contamination check and a calibrated LLM judge.
- **Week 12:** a tiny model pretrained, tuned and evaluated end to end, with a report.

### How the weeks work

1. The first half of the week is for watching, reading and deriving. The second half is for building. If you did not build it, you did not learn it.
2. Every phase ends in a project week with a deliverable and a definition of done.
3. Do the arithmetic yourself: parameter counts, FLOPs, bytes of memory. Most of the intuition in this field is back-of-envelope.
4. Keep a lab notebook of runs. One line per run: what changed, what the loss did, what you think it means.

### Failure modes to watch

- **Watching instead of typing.** A lecture you did not code along with teaches you the vocabulary, not the system.
- **Reading a benchmark number as a fact.** Ask what the model saw in training, how it was prompted and how big the error bars are.
- **Skipping the small experiment.** Everything in weeks 5 to 10 can be reproduced at toy scale in minutes. Do that before you trust a paper's claim.
- **Tangents.** Mixture of experts, multimodality and long-context tricks are real topics. They go on a reading list, not into this week.

## Text becomes a model

### Week 1: Tokenization and BPE

Focus: Learn why models read subword pieces, then build a byte-level BPE tokenizer and find where it breaks.

Goals:

- Write the pair-count and merge loop from memory and train 256 merges on a text file
- Implement encode (apply merges in rank order) and decode (bytes back to UTF-8)
- Pass a round-trip test on emoji, accents, newlines and a 1 MB file
- Plot bytes per token against vocab size and compare with tiktoken's gpt2 encoding
- List five tokenizer quirks (numbers, spaces, languages) and say which hurt a model

Resources:

- [Let's build the GPT Tokenizer](https://www.youtube.com/watch?v=zduSFxRajkE) (video, Whole lecture, code along)
- [karpathy/minbpe](https://github.com/karpathy/minbpe) (code, Read after you have written your own)
- [Neural Machine Translation of Rare Words with Subword Units](https://arxiv.org/abs/1508.07909) (paper, Section 3.2, byte pair encoding)
- [Byte-Pair Encoding tokenization (Hugging Face LLM course)](https://huggingface.co/learn/llm-course/chapter6/5) (course)
- [SentencePiece: a simple and language independent subword tokenizer](https://arxiv.org/abs/1808.06226) (paper, Skim the introduction and system overview)
- [openai/tiktoken](https://github.com/openai/tiktoken) (code, Use it as the reference to compare against)

### Week 2: Embeddings and the LM objective

Focus: Turn next-token prediction into a number you can minimise, from a count table to a neural model, scored with cross-entropy and perplexity.

Goals:

- Derive cross-entropy with a one-hot target and perplexity = exp(loss) on paper
- Build the count bigram and the one-layer neural bigram; match their average NLL
- Add an embedding table and a context of 8 characters; beat the bigram on validation loss
- Convert your best loss to bits per character and perplexity, and say what each means
- Check that your first-step loss is close to ln(vocab size) and explain why

Resources:

- [The spelled-out intro to language modeling: building makemore](https://www.youtube.com/watch?v=PaCmpygFfXo) (video)
- [Building makemore Part 2: MLP](https://www.youtube.com/watch?v=TCH_1BHY58I) (video)
- [Transformers, the tech behind LLMs (3Blue1Brown, chapter 5)](https://www.youtube.com/watch?v=wjZofJX0v4M) (video)
- [A Neural Probabilistic Language Model](https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf) (paper, Sections 1-3)
- [Speech and Language Processing: N-gram Language Models](https://web.stanford.edu/~jurafsky/slp3/3.pdf) (book, The perplexity section)
- [The Illustrated Word2vec](https://jalammar.github.io/illustrated-word2vec/) (blog)

### Week 3: Attention and the transformer block

Focus: Derive scaled dot-product attention, then assemble heads, MLP, residuals and layer norm into the block every LLM repeats.

Goals:

- Work the 3-token attention example by hand, with the causal mask, before running code
- Implement single-head causal attention and match F.scaled_dot_product_attention
- Wrap it in multi-head attention with one fused QKV projection and write every tensor shape
- Build a pre-norm block (LayerNorm, attention, residual, LayerNorm, MLP, residual)
- Count the parameters of one GPT-2 small block by hand and confirm with sum(p.numel())

Resources:

- [Attention in transformers, step-by-step (3Blue1Brown, chapter 6)](https://www.youtube.com/watch?v=eMlx5fFNoYc) (video)
- [The Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/) (blog)
- [Attention Is All You Need](https://arxiv.org/abs/1706.03762) (paper, Sections 3.1-3.3)
- [Let's build GPT: from scratch, in code, spelled out](https://www.youtube.com/watch?v=kCc8FmEb1nY) (video, First half, up to the full self-attention block)
- [Understanding and Coding Self-Attention from Scratch](https://sebastianraschka.com/blog/2023/self-attention-from-scratch.html) (blog)
- [The Transformer Family Version 2.0](https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/) (blog, Skim: what changed since 2017)

### Week 4: Project: a GPT from scratch

Focus: Assemble the pieces into a trainable GPT, train it on tiny Shakespeare and learn to read what the loss curve is telling you.

Goals:

- Assemble embeddings, N blocks, a final LayerNorm and a tied output head
- Overfit a single batch to near-zero loss before touching the full dataset
- Train with AdamW, warmup and cosine decay; log train and val loss every 250 steps
- Write the sampling loop with temperature and top-k; save three samples at different temperatures
- Diff your model against nanoGPT and note every difference and its reason

Deliverable: A repo with model.py, train.py, sample.py, a config, loss-curve plots, a run log and a README that reports parameter count, final train and validation loss, and a sample.

Resources:

- [Let's build GPT: from scratch, in code, spelled out](https://www.youtube.com/watch?v=kCc8FmEb1nY) (video, Whole lecture, code along)
- [karpathy/nanoGPT](https://github.com/karpathy/nanoGPT) (code, model.py and train.py, after your own version works)
- [Language Models are Unsupervised Multitask Learners (GPT-2)](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) (paper, Section 2)
- [A Recipe for Training Neural Networks](https://karpathy.github.io/2019/04/25/recipe/) (blog)
- [Tiny Shakespeare](https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt) (dataset)
- [rasbt/LLMs-from-scratch](https://github.com/rasbt/LLMs-from-scratch) (code, Chapter 4 as a second reference implementation)

## Bigger, faster, specialised

### Week 5: Pretraining at scale

Focus: See how real pretraining runs are sized and run: compute budgets, scaling laws, data, mixed precision and the parallelism that spreads one model over many GPUs.

Goals:

- Derive 6ND and the 16 bytes per parameter rule on paper
- Compute FLOPs, GPU-hours and memory for GPT-2 small on 10B tokens and compare with the lecture
- Train four model widths at two token budgets and log final validation loss
- Fit the power law on log-log axes and report the exponent with a rough error bar
- Write the run planner (time and memory) and test it against a real run of yours

Resources:

- [Let's reproduce GPT-2 (124M)](https://www.youtube.com/watch?v=l8pRSuU81PU) (video, Optimisation and systems chapters)
- [Scaling Laws for Neural Language Models](https://arxiv.org/abs/2001.08361) (paper, Section 1 summary and figure 1)
- [Training Compute-Optimal Large Language Models](https://arxiv.org/abs/2203.15556) (paper, Sections 1-3)
- [Transformer Math 101](https://blog.eleuther.ai/transformer-math/) (blog)
- [The Ultra-Scale Playbook](https://huggingface.co/spaces/nanotron/ultrascale-playbook) (article, Data parallelism, ZeRO, tensor and pipeline parallelism)
- [Stanford CS336: Language Modeling from Scratch](https://cs336.stanford.edu/) (course, Scaling-law lectures (9 and 11) and the parallelism lectures (7 and 8))

### Week 6: Inference: sampling and the KV cache

Focus: See why generation is memory-bound, then make it fast with sampling controls, the KV cache, FlashAttention, quantization and speculative decoding.

Goals:

- Implement temperature, top-k and top-p sampling and compare outputs at three settings
- Add a KV cache and assert cached logits match a full recompute over 200 tokens
- Benchmark tokens per second with and without the cache at lengths 64, 256 and 1024
- Quantize linear weights to int8 per channel and measure the validation perplexity change
- Compute KV-cache bytes per token for a 7B model and the speculative-decoding speedup at 80% acceptance

Resources:

- [The Curious Case of Neural Text Degeneration](https://arxiv.org/abs/1904.09751) (paper)
- [Transformer Inference Arithmetic](https://kipp.ly/transformer-inference-arithmetic/) (blog)
- [Coding LLaMA 2 from scratch in PyTorch (KV cache, GQA, RoPE, RMSNorm)](https://www.youtube.com/watch?v=oM4VmoabDAI) (video, KV cache and grouped-query attention sections)
- [Quantization explained with PyTorch](https://www.youtube.com/watch?v=0VdNflU08yA) (video, Post-training quantization sections)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://arxiv.org/abs/2205.14135) (paper, Sections 1-3)
- [Fast Inference from Transformers via Speculative Decoding](https://arxiv.org/abs/2211.17192) (paper, Sections 1-3)

### Week 7: Project: fine-tune with SFT and LoRA

Focus: Adapt a pretrained model to one narrow task with supervised fine-tuning and low-rank adapters, then prove it helped and measure what it broke.

Goals:

- Pick one narrow task, build 2,000 to 5,000 training examples and 200 held-out ones
- Write LoRALinear (frozen W, trainable A and B, scale alpha/r) and match PEFT's output
- Train with loss masked to the response tokens only; log train and held-out loss
- Sweep rank 4, 16 and 64 and plot held-out score against trainable parameters
- Measure general-text perplexity before and after to quantify forgetting

Deliverable: A repo with your LoRALinear, the training script, a before/after table on 200 held-out examples, a rank sweep, the adapter size, and a paragraph on what the fine-tune made worse.

Resources:

- [Deep Dive into LLMs like ChatGPT](https://www.youtube.com/watch?v=7xTGNNLPyMI) (video, Post-training and supervised fine-tuning sections)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685) (paper, Sections 1-4)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://arxiv.org/abs/2305.14314) (paper, Sections 1-3)
- [LoRA Without Regret](https://thinkingmachines.ai/blog/lora/) (blog)
- [PEFT documentation](https://huggingface.co/docs/peft) (article)
- [TRL: SFT Trainer](https://huggingface.co/docs/trl/sft_trainer) (article)

## Teaching it to behave

### Week 8: Alignment: RLHF and DPO

Focus: Understand how a base model becomes an assistant: reward models, KL-regularised RL, and DPO, which reaches the same optimum without a reward model.

Goals:

- Derive the DPO loss from the KL-regularised objective and the Bradley-Terry model
- Compute one DPO loss and margin by hand for the example pair, then in code
- Implement dpo_loss on sequence log-probs and match TRL on one batch
- Train on 2,000 preference pairs; plot margin, accuracy and KL to the reference
- Compare 20 prompts before and after; write down verbosity and tone drift

Resources:

- [Deep Dive into LLMs like ChatGPT](https://www.youtube.com/watch?v=7xTGNNLPyMI) (video, Reinforcement learning sections)
- [Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155) (paper, Sections 2-3)
- [Direct Preference Optimization](https://arxiv.org/abs/2305.18290) (paper, Sections 3-4)
- [Illustrating Reinforcement Learning from Human Feedback](https://huggingface.co/blog/rlhf) (blog)
- [Direct Preference Optimization explained (Umar Jamil)](https://www.youtube.com/watch?v=hvGa5Mba4c8) (video)
- [Reward Hacking in Reinforcement Learning](https://lilianweng.github.io/posts/2024-11-28-reward-hacking/) (blog)

### Week 9: Reasoning and test-time compute

Focus: Learn why spending more tokens at inference helps, and how reinforcement learning on verifiable rewards trains models to think before answering.

Goals:

- Write the GRPO advantage (reward minus group mean over group std) and why no critic is needed
- Evaluate greedy accuracy on 200 GSM8K problems with a fixed prompt and answer parser
- Sample n = 4 and 16 answers per problem; compute majority-vote accuracy
- Plot accuracy against samples and against total tokens; mark where it saturates
- Implement grpo_advantages with a unit test on a hand-computed group

Resources:

- [Self-Consistency Improves Chain of Thought Reasoning](https://arxiv.org/abs/2203.11171) (paper)
- [Let's Verify Step by Step](https://arxiv.org/abs/2305.20050) (paper)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://arxiv.org/abs/2408.03314) (paper, Sections 1-4)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://arxiv.org/abs/2501.12948) (paper, Sections 1 and 2)
- [Why We Think](https://lilianweng.github.io/posts/2025-05-01-thinking/) (blog)
- [GSM8K](https://huggingface.co/datasets/openai/gsm8k) (dataset)

### Week 10: Project: evaluation you can trust

Focus: Measure models properly: perplexity, benchmarks and how they leak, human preference, and LLM judges that you calibrate against your own labels.

Goals:

- Implement held-out perplexity and a log-likelihood multiple-choice scorer
- Write an n-gram overlap contamination check and run it between a benchmark and your training data
- Build a pairwise judge with position swapping; label 50 pairs yourself
- Report agreement and bootstrap confidence intervals for every metric
- Write the three biggest ways this harness could still mislead you

Deliverable: An eval repo and a report scoring three checkpoints (base, SFT, DPO) with confidence intervals, a contamination table, and judge-versus-human agreement on 50 labelled pairs.

Resources:

- [Measuring Massive Multitask Language Understanding](https://arxiv.org/abs/2009.03300) (paper)
- [Rethinking Benchmark and Contamination for Language Models with Rephrased Samples](https://arxiv.org/abs/2311.04850) (paper)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://arxiv.org/abs/2306.05685) (paper, Sections 3-4, judge biases)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://arxiv.org/abs/2403.04132) (paper, Skim: how pairwise votes become ratings)
- [EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) (code, How multiple-choice tasks are scored)
- [Your AI Product Needs Evals](https://hamel.dev/blog/posts/evals/) (blog)

## Open the box, then ship

### Week 11: Interpretability: features, circuits

Focus: Look inside: the residual stream, induction heads, superposition, and sparse autoencoders that pull readable features out of activations.

Goals:

- Explain the residual stream as a shared bus that each layer reads and writes
- Train the toy superposition model and plot the learned feature directions at three sparsities
- Run GPT-2 small in TransformerLens and detect induction behaviour on repeated random tokens
- Train an SAE on residual activations; report L0, reconstruction error and loss recovered
- Inspect top-activating contexts for five features and label them honestly, including the unclear ones

Resources:

- [A Mathematical Framework for Transformer Circuits](https://transformer-circuits.pub/2021/framework/index.html) (paper, Introduction and the residual stream sections)
- [In-context Learning and Induction Heads](https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html) (paper, Summary of results)
- [Toy Models of Superposition](https://transformer-circuits.pub/2022/toy_model/index.html) (paper)
- [Towards Monosemanticity: Decomposing Language Models With Dictionary Learning](https://transformer-circuits.pub/2023/monosemantic-features) (paper)
- [Scaling Monosemanticity](https://transformer-circuits.pub/2024/scaling-monosemanticity/) (paper, Skim: the feature examples and the evaluation of SAEs)
- [TransformerLens](https://github.com/TransformerLensOrg/TransformerLens) (code)

### Week 12: Capstone: pretrain, tune, evaluate

Focus: Run the whole pipeline once at small scale: pretrain a tiny model, fine-tune it, evaluate every stage with your own harness and write up what you found.

Goals:

- Write a one-page plan: size, token budget, predicted loss from your scaling fit, evals
- Prepare the corpus: tokenize with your BPE or a reference tokenizer and shard to disk
- Pretrain to the token budget with checkpoints and logged curves
- Fine-tune a copy and run the week-10 harness on both checkpoints
- Write the report: curves, predicted versus actual loss, samples, evals, one probe, one change

Deliverable: A reproducible repo and a report of at most three pages: loss curves, predicted versus actual loss from your scaling fit, samples at each stage, eval tables with confidence intervals, one interpretability probe, and five sentences on what you would change with ten times the compute.

Resources:

- [karpathy/nanochat](https://github.com/karpathy/nanochat) (code, The speedrun script and the training stages it calls)
- [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://arxiv.org/abs/2305.07759) (paper)
- [SmolLM2: When Smol Goes Big](https://arxiv.org/abs/2502.02737) (paper, Data and training sections)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://arxiv.org/abs/2406.17557) (paper, Filtering and deduplication sections)
- [TinyStories dataset](https://huggingface.co/datasets/roneneldan/TinyStories) (dataset)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://arxiv.org/abs/2304.01373) (paper, Skim: what to log across a training run)

## Notes

- [Superposition and sparse autoencoders](https://ornix.io/u/omar/large-language-models-end-to-end/superposition-and-sparse-autoencoders.md)
- [RLHF vs DPO](https://ornix.io/u/omar/large-language-models-end-to-end/rlhf-vs-dpo.md)
- [LoRA in one page of math](https://ornix.io/u/omar/large-language-models-end-to-end/lora-in-one-page-of-math.md)
- [The KV cache and why inference is memory-bound](https://ornix.io/u/omar/large-language-models-end-to-end/kv-cache-memory-bound-inference.md)
- [Why scaling laws bend (Chinchilla)](https://ornix.io/u/omar/large-language-models-end-to-end/why-scaling-laws-bend.md)
- [Reading a loss curve](https://ornix.io/u/omar/large-language-models-end-to-end/reading-a-loss-curve.md)
- [Attention, one matrix at a time](https://ornix.io/u/omar/large-language-models-end-to-end/attention-one-matrix-at-a-time.md)
- [BPE by hand](https://ornix.io/u/omar/large-language-models-end-to-end/bpe-by-hand.md)

Follow this roadmap on Ornix: https://ornix.io/u/omar/large-language-models-end-to-end
