# Deep Learning from Scratch: backprop to diffusion

By @omar · 12 weeks

> For programmers who know Python and a little ML: 12 weeks, 7 to 10 hours a week, from hand-written autograd to a diffusion model you train yourself. Four projects, no black boxes.

## About

**For programmers who want to build neural networks, not just call them.** Twelve weeks, four phases, four things you will have built. Plan on 7 to 10 hours a week: learn early in the week, build late in it. The capstone week is heavier.

### What you will have built

- **Week 3:** a reproducible training recipe: an MLP and a small CNN on Fashion-MNIST, one `train.py`, learning curves and an ablation table.
- **Week 6:** an LSTM cell written by hand that matches PyTorch, a character-level model, and a plot of exactly where recurrence runs out of memory.
- **Week 9:** SimCLR-lite: a ResNet pretrained with no labels on CIFAR-10, judged by a linear probe against a random-weights baseline.
- **Week 12:** a class-conditional diffusion model trained from scratch, with fast sampling and a classifier-free guidance sweep.

Along the way you also write micrograd, batch norm, convolution via im2col, a ResNet, a transformer block and a Vision Transformer.

### How the weeks work

1. Watch and read in the first half of the week, build in the second. If you did not type it, you did not learn it.
2. Derive one thing on paper every week. Each one is named in the week's goals. Do not skip them; they are the point.
3. Every phase ends in a project week with a definition of done. Ship it, write three paragraphs about it, move on.
4. Eight deep notes sit on the weeks where intuition matters most. Read them after the lecture, before you code.
5. Flashcards are for the ideas that should live in your head, not in a notebook.

### What this roadmap does not cover

LLM pretraining at scale, alignment, RAG and agents. Week 7 builds the transformer block and stops there. The language-model and agent roadmaps pick up from it.

### Failure modes to watch

- **Tutorial treadmill.** Watching Karpathy code is not the same as coding. Close the video and rebuild the file.
- **Framework first.** Hand-write each idea once (autograd, batch norm, conv, an LSTM cell, attention). After that, use the library.
- **Silent bugs.** A model that trains is not a model that is right. Check shapes, overfit one batch, look at the initial loss, plot activation statistics.
- **Chasing numbers.** Hitting 94% on CIFAR-10 matters less than being able to say why the 91% run was worse.

## Gradients all the way down

### Week 1: Backprop from scratch

Focus: Build a scalar autograd engine and see a neural network as a differentiable expression graph.

Goals:

- Watch 3Blue1Brown chapters 1 to 4 and sketch the chain of derivatives for one neuron
- Code along with micrograd: Value with +, *, tanh, exp, pow and a topological backward()
- Pass a central-difference gradient check on every op and on a small MLP
- Train a two-layer MLP on four points to loss below 0.01 and plot the loss curve
- Derive on paper why backward() accumulates with += (try b = a * a) and write it in five sentences

Resources:

- [But what is a neural network?](https://www.youtube.com/watch?v=aircAruvnKk) (video, Chapter 1, the picture)
- [Gradient descent, how neural networks learn](https://www.youtube.com/watch?v=IHZwWFHWa-w) (video, Chapter 2)
- [Backpropagation calculus](https://www.youtube.com/watch?v=tIeHLnjs5U8) (video, Chapter 4, pause and derive)
- [The spelled-out intro to neural networks and backpropagation: building micrograd](https://www.youtube.com/watch?v=VMj-3S1tku0) (video, Whole lecture, code along)
- [Dive into Deep Learning: Forward and backward propagation](https://d2l.ai/chapter_multilayer-perceptrons/backprop.html) (book, Section 5.3)
- [Lecture 6: Backpropagation (EECS 498, Justin Johnson)](https://www.youtube.com/watch?v=dB-u77Y5a6A) (video, After micrograd: the vector and matrix case)

### Week 2: Init, activations and batch norm

Focus: Make a deep MLP trainable: initialization, saturating activations, normalization, and reading activation statistics.

Goals:

- Watch makemore parts 2 and 3; pause on the initial loss and compute what it should be
- Rebuild the part 3 network and fix a bad init so the first loss is within 0.05 of ln(27)
- Plot per-layer activation histograms and gradient-to-data ratios before and after Kaiming init
- Write BatchNorm1d by hand, including running stats for eval mode, and compare loss with and without it
- Derive on paper why ReLU layers want weight variance 2/fan_in

Resources:

- [Building makemore Part 2: MLP](https://www.youtube.com/watch?v=TCH_1BHY58I) (video, Skim if embeddings are familiar)
- [Building makemore Part 3: Activations & Gradients, BatchNorm](https://www.youtube.com/watch?v=P6sfmUTpUmc) (video, Whole lecture, code along)
- [Delving Deep into Rectifiers (He initialization)](https://arxiv.org/abs/1502.01852) (paper, Section 2.2)
- [Batch Normalization: Accelerating Deep Network Training](https://arxiv.org/abs/1502.03167) (paper, Section 3)
- [Dive into Deep Learning: Numerical Stability and Initialization](https://d2l.ai/chapter_multilayer-perceptrons/numerical-stability-and-init.html) (article)
- [Building makemore Part 4: Becoming a Backprop Ninja](https://www.youtube.com/watch?v=q8SA3rM6ckI) (video, Optional: the manual-backprop exercises)

### Week 3: PyTorch loops and a training recipe

Focus: Own the training loop: tensors, autograd, optimizers, schedules and the debugging habits that save days.

Goals:

- Read Karpathy's recipe end to end and write your own eight-line checklist
- Port the MLP to nn.Module, DataLoader and AdamW with seeding, eval, and checkpointing
- Overfit a single batch to near-zero loss, and log the initial loss, before touching the full dataset
- Add a small CNN baseline and a cosine or one-cycle schedule with warmup
- Run six ablations, write the recipe README, then reproduce the result from a clean clone

Deliverable: A repo with train.py and a recipe write-up: MLP at 88% or better and a small CNN at 91% or better on the Fashion-MNIST test set, learning curves, an ablation table (learning rate, weight decay, batch size) and a one-command reproduction.

Resources:

- [A Recipe for Training Neural Networks](https://karpathy.github.io/2019/04/25/recipe/) (blog)
- [PyTorch: Learn the Basics, quickstart](https://docs.pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html) (course)
- [Adam: A Method for Stochastic Optimization](https://arxiv.org/abs/1412.6980) (paper, Section 2, the algorithm)
- [Decoupled Weight Decay Regularization (AdamW)](https://arxiv.org/abs/1711.05101) (paper, Sections 1 to 3)
- [Deep Learning Tuning Playbook](https://github.com/google-research/tuning_playbook) (code, Skim: learning rate and batch size)
- [Fashion-MNIST](https://github.com/zalandoresearch/fashion-mnist) (dataset)

## Structure in the data

### Week 4: Convolutions as weight sharing

Focus: Derive convolution as a constrained linear layer, count parameters and receptive fields, and write conv and pooling by hand.

Goals:

- Watch the convolution lecture; hand-compute output shapes and parameter counts for three conv layers
- Write conv2d with four loops, then with unfold and a matmul, and match F.conv2d to 1e-5
- Write max pooling with a backward pass that routes gradient to the argmax
- Train a three-layer CNN on CIFAR-10 to 70% or better and plot the first-layer filters
- Compute the receptive field of a five-layer stack by hand and confirm it with a gradient probe

Resources:

- [But what is a convolution?](https://www.youtube.com/watch?v=KuXjwB4LzSA) (video)
- [Lecture 7: Convolutional Networks (EECS 498, Justin Johnson)](https://www.youtube.com/watch?v=ANyxBVxmdZ0) (video, Whole lecture)
- [CS231n: Convolutional Neural Networks for Visual Recognition](https://cs231n.github.io/convolutional-networks/) (article, Convolutional layer and pooling)
- [Dive into Deep Learning: From Fully Connected Layers to Convolutions](https://d2l.ai/chapter_convolutional-neural-networks/why-conv.html) (article)
- [A guide to convolution arithmetic for deep learning](https://arxiv.org/abs/1603.07285) (paper, Sections 1 and 2)
- [Dive into Deep Learning: LeNet](https://d2l.ai/chapter_convolutional-neural-networks/lenet.html) (book, The LeNet-5 architecture, in code)

### Week 5: ResNets and training deeper nets

Focus: See why depth fails without skip connections, build a ResNet, and train it past 90% on CIFAR-10.

Goals:

- Watch the architectures lecture and state the degradation problem in two sentences
- Read ResNet and identity-mappings; derive the gradient through y = x + F(x) for a stack of blocks
- Implement BasicBlock and a ResNet-18-style net for 32x32 inputs; check shapes and parameter count
- Train on CIFAR-10 with crop and flip augmentation and a one-cycle schedule to above 90% test accuracy
- Ablate: remove the skips, log per-layer gradient norms at init for both, and plot them

Resources:

- [Lecture 8: CNN Architectures (EECS 498, Justin Johnson)](https://www.youtube.com/watch?v=XaZIlVrIO-Q) (video)
- [Deep Residual Learning for Image Recognition](https://arxiv.org/abs/1512.03385) (paper, Sections 1 to 4.1)
- [Identity Mappings in Deep Residual Networks](https://arxiv.org/abs/1603.05027) (paper, Sections 1 to 3)
- [Dive into Deep Learning: Residual Networks (ResNet)](https://d2l.ai/chapter_convolutional-modern/resnet.html) (article)
- [Visualizing the Loss Landscape of Neural Nets](https://arxiv.org/abs/1712.09913) (paper, Introduction and the skip-connection figure)
- [A ConvNet for the 2020s](https://arxiv.org/abs/2201.03545) (paper, Section 2, the modernization roadmap)

### Week 6: Sequence models and why they struggle

Focus: Build an RNN and an LSTM, watch gradients vanish over time, and see why attention was the fix.

Goals:

- Read Karpathy's RNN post and Olah's LSTM post, then redraw the LSTM cell from memory
- Watch the recurrent networks lecture and write dh_T/dh_0 as a product of Jacobians
- Hand-write an LSTMCell and match nn.LSTMCell outputs and gradients to 1e-5
- Train a character LSTM on Tiny Shakespeare and sample 500 characters at three temperatures
- Run the copy-at-lag experiment for RNN and LSTM, plot accuracy against lag, and write one paragraph

Deliverable: An LSTM cell that matches PyTorch to 1e-5 in outputs and gradients, a character LSTM that samples Shakespeare-like text at three temperatures, and one plot (accuracy against dependency distance, RNN against LSTM) with a paragraph explaining it.

Resources:

- [The Unreasonable Effectiveness of Recurrent Neural Networks](https://karpathy.github.io/2015/05/21/rnn-effectiveness/) (blog)
- [Understanding LSTM Networks](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) (blog)
- [Lecture 12: Recurrent Networks (EECS 498, Justin Johnson)](https://www.youtube.com/watch?v=dUzLD91Sj-o) (video)
- [On the difficulty of training Recurrent Neural Networks](https://arxiv.org/abs/1211.5063) (paper, Section 2)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473) (paper, Sections 1 to 3)
- [Visualizing a Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention)](https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/) (blog)

## Attention and representations

### Week 7: Attention and the transformer block

Focus: Derive attention as a soft lookup, then build a decoder-only transformer block with multi-head attention, residuals and LayerNorm.

Goals:

- Watch the attention chapter; derive attention as a soft dictionary lookup on paper
- Write single-head causal attention with loops, then batched matmul; assert rows sum to 1 and nothing leaks from the future
- Build multi-head attention, MLP, residuals and pre-LN LayerNorm as one block with shape asserts
- Train a four-layer character GPT on Tiny Shakespeare and compare validation loss to the week 6 LSTM
- Show numerically that q . k has variance d_k for unit-variance entries, and explain the 1/sqrt(d_k)

Resources:

- [Attention in transformers, step-by-step](https://www.youtube.com/watch?v=eMlx5fFNoYc) (video, Chapter 6)
- [Let's build GPT: from scratch, in code, spelled out](https://www.youtube.com/watch?v=kCc8FmEb1nY) (video, Through the transformer block; skip the scaling-up discussion)
- [Attention Is All You Need](https://arxiv.org/abs/1706.03762) (paper, Sections 3.1 to 3.3 and 3.5)
- [The Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/) (blog)
- [The Annotated Transformer](https://nlp.seas.harvard.edu/annotated-transformer/) (article, Attention and the encoder-decoder blocks)
- [karpathy/nanoGPT](https://github.com/karpathy/nanoGPT) (code, Read model.py after you have built your own)

### Week 8: Vision Transformers

Focus: Treat an image as a sequence of patches, build a ViT, and learn when it beats a CNN and when it does not.

Goals:

- Watch the walkthrough and read the ViT method; write the shape of every tensor from image to logits
- Implement PatchEmbed, CLS token, learned position embeddings and the head on top of your week 7 block
- Train ViT-tiny on CIFAR-10 with AdamW, warmup and the week 5 augmentation; log curves next to the ResNet
- Ablate: no position embeddings, and patch sizes 2, 4 and 8; one table
- Plot CLS attention for eight test images and write three sentences comparing the ViT and the ResNet

Resources:

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (Paper Explained)](https://www.youtube.com/watch?v=TrdevFK_am4) (video)
- [An Image is Worth 16x16 Words (ViT)](https://arxiv.org/abs/2010.11929) (paper, Section 3 and the scaling results)
- [Better plain ViT baselines for ImageNet-1k](https://arxiv.org/abs/2205.01580) (paper)
- [Training data-efficient image transformers and distillation through attention (DeiT)](https://arxiv.org/abs/2012.12877) (paper, Skim: augmentation and regularization)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://arxiv.org/abs/2108.08810) (paper, Introduction and conclusions)
- [lucidrains/vit-pytorch](https://github.com/lucidrains/vit-pytorch) (code, Read vit.py after you have built your own)

### Week 9: Representation learning

Focus: Learn features without labels: autoencoders, word2vec, then contrastive learning with SimCLR and the idea behind CLIP.

Goals:

- Read Olah, the autoencoder section and the word2vec negative-sampling idea; say why a bottleneck forces features
- Train a 2-D bottleneck autoencoder on Fashion-MNIST and plot the latent space coloured by class
- Implement NT-Xent with a toy-batch test, add SimCLR augmentations and pretrain a ResNet-18 on CIFAR-10
- Linear-probe and kNN-evaluate frozen features against a random-init network and report both
- Read the CLIP method and write one page on what changes versus SimCLR

Deliverable: SimCLR-lite on CIFAR-10: a ResNet-18 pretrained with NT-Xent, a unit test for the loss on a toy batch with a known answer, a linear-probe table (pretrained against random init, plus kNN), and a one-page note on how CLIP changes the positive pairs.

Resources:

- [Neural Networks, Manifolds, and Topology](https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/) (blog)
- [From Autoencoder to Beta-VAE](https://lilianweng.github.io/posts/2018-08-12-vae/) (blog, The autoencoder section only; VAEs are next phase)
- [Distributed Representations of Words and Phrases and their Compositionality](https://arxiv.org/abs/1310.4546) (paper, Skim: negative sampling)
- [Contrastive Representation Learning](https://lilianweng.github.io/posts/2021-05-31-contrastive/) (blog)
- [A Simple Framework for Contrastive Learning of Visual Representations (SimCLR)](https://arxiv.org/abs/2002.05709) (paper, Sections 2 and 3)
- [Learning Transferable Visual Models From Natural Language Supervision (CLIP)](https://arxiv.org/abs/2103.00020) (paper, Section 2 and the pseudocode figure)

## From noise to images

### Week 10: Variational autoencoders

Focus: Derive the ELBO, implement the reparameterization trick, and train a VAE you can sample from and interpolate in.

Goals:

- Read Altosaar and Kingma-Welling section 2; derive the ELBO from log p(x) = ELBO + KL(q || p(z|x))
- Derive the closed-form KL between N(mu, sigma^2) and N(0, 1) and check it against torch.distributions
- Train a convolutional VAE (z = 16) on Fashion-MNIST; plot reconstruction and KL separately per epoch
- Make latent traversals, an interpolation grid and prior samples; compare beta = 0.5, 1 and 4
- Force posterior collapse with a huge beta and explain what the per-dimension KL shows

Resources:

- [Tutorial: What is a variational autoencoder?](https://jaan.io/what-is-variational-autoencoder-vae-tutorial/) (article)
- [Auto-Encoding Variational Bayes](https://arxiv.org/abs/1312.6114) (paper, Sections 2 and 3)
- [From Autoencoder to Beta-VAE](https://lilianweng.github.io/posts/2018-08-12-vae/) (blog, The VAE and Beta-VAE sections)
- [Stanford CS236: Variational Autoencoder notes](https://deepgenerativemodels.github.io/notes/vae/) (article)
- [Lecture 19: Generative Models I (EECS 498, Justin Johnson)](https://www.youtube.com/watch?v=Q3HU2vEhD5Y) (video, The VAE portion)
- [PyTorch examples: VAE](https://github.com/pytorch/examples/tree/main/vae) (code, Read after you have written your own)

### Week 11: Diffusion models

Focus: Understand the forward noising process, the noise-prediction loss and the score view; sample with DDPM and add classifier-free guidance.

Goals:

- Watch the Welch Labs video and read Weng's forward process; derive q(x_t | x_0) from the one-step recursion
- Read DDPM sections 2 and 3; show why the simplified loss is MSE on epsilon and write both algorithms
- Train an epsilon-network on a 2-D toy dataset, sample with DDPM and plot noise-to-data trajectories
- Add class conditioning with 10% label dropout and sweep the guidance scale
- Explain the score view (epsilon-hat as a scaled score) and why DDIM sampling is deterministic

Resources:

- [But how do AI images and videos actually work? (Guest video by Welch Labs)](https://www.youtube.com/watch?v=iv-5mZ_9CPY) (video)
- [What are Diffusion Models?](https://lilianweng.github.io/posts/2021-07-11-diffusion-models/) (blog, Through the training and sampling sections)
- [Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2006.11239) (paper, Sections 2 and 3)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://yang-song.net/blog/2021/score/) (blog)
- [Classifier-Free Diffusion Guidance](https://arxiv.org/abs/2207.12598) (paper, Sections 2 and 3)
- [MIT 6.S184: Flow Matching and Diffusion Models, Lecture 1](https://www.youtube.com/watch?v=9eJQQVrUUoI) (video, Second pass, after the DDPM paper)

### Week 12: Capstone: diffusion from scratch

Focus: Train a small class-conditional U-Net diffusion model on a small image dataset and ship it with samples, a guidance sweep and an honest write-up.

Goals:

- Read the annotated diffusion post; list every U-Net component and where the timestep enters
- Build the U-Net with a sinusoidal time embedding, class conditioning and EMA; overfit one batch first
- Train on Fashion-MNIST or CIFAR-10; save a sample grid every N epochs from both raw and EMA weights
- Add a DDIM sampler (20 to 50 steps), a guidance-scale sweep grid and a nearest-training-neighbour check
- Write the README: loss curve, sample grids, what failed, and what you would change next

Deliverable: A repo that trains a class-conditional U-Net diffusion model on Fashion-MNIST or CIFAR-10 in one command, with EMA, DDIM sampling in 50 steps or fewer, a classifier-free guidance sweep grid, a nearest-training-neighbour memorization check, and a README with an honest failure analysis.

Resources:

- [The Annotated Diffusion Model](https://huggingface.co/blog/annotated-diffusion) (blog, The U-Net and the training loop)
- [Denoising Diffusion Implicit Models](https://arxiv.org/abs/2010.02502) (paper, Sections 1 to 4)
- [Improved Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2102.09672) (paper, Section 3: cosine schedule and learned variance)
- [MIT 6.S184: Flow Matching and Diffusion Models, Lecture 3B: Classifier-free Guidance](https://www.youtube.com/watch?v=8oWZ1bHwyRI) (video)
- [huggingface/diffusion-models-class](https://github.com/huggingface/diffusion-models-class) (code, Unit 1: training from scratch)
- [Scalable Diffusion Models with Transformers (DiT)](https://arxiv.org/abs/2212.09748) (paper, Stretch: sections 1 to 3)

## Notes

- [Diffusion: noise in, image out](https://ornix.io/u/omar/deep-learning-from-scratch/diffusion-noise-in-image-out.md)
- [The ELBO without tears](https://ornix.io/u/omar/deep-learning-from-scratch/elbo-without-tears.md)
- [Attention, one head, by hand](https://ornix.io/u/omar/deep-learning-from-scratch/attention-one-head-by-hand.md)
- [Why LSTMs gate](https://ornix.io/u/omar/deep-learning-from-scratch/why-lstms-gate.md)
- [Residual connections and gradient flow](https://ornix.io/u/omar/deep-learning-from-scratch/residual-connections-gradient-flow.md)
- [Convolution as weight sharing](https://ornix.io/u/omar/deep-learning-from-scratch/convolution-as-weight-sharing.md)
- [Why initialization matters: the variance math](https://ornix.io/u/omar/deep-learning-from-scratch/why-initialization-matters.md)
- [Backprop is the chain rule with bookkeeping](https://ornix.io/u/omar/deep-learning-from-scratch/backprop-chain-rule-bookkeeping.md)

Follow this roadmap on Ornix: https://ornix.io/u/omar/deep-learning-from-scratch
