Deep Learning from Scratch: backprop to diffusion
For programmers who know Python and a little ML: 12 weeks, 7 to 10 hours a week, from hand-written autograd to a diffusion model you train yourself. Four projects, no black boxes.
Following gives you your own copy, dated from today. Free to start.

About this roadmap
For programmers who want to build neural networks, not just call them. Twelve weeks, four phases, four things you will have built. Plan on 7 to 10 hours a week: learn early in the week, build late in it. The capstone week is heavier.
What you will have built
- Week 3: a reproducible training recipe: an MLP and a small CNN on Fashion-MNIST, one
train.py, learning curves and an ablation table. - Week 6: an LSTM cell written by hand that matches PyTorch, a character-level model, and a plot of exactly where recurrence runs out of memory.
- Week 9: SimCLR-lite: a ResNet pretrained with no labels on CIFAR-10, judged by a linear probe against a random-weights baseline.
- Week 12: a class-conditional diffusion model trained from scratch, with fast sampling and a classifier-free guidance sweep.
Along the way you also write micrograd, batch norm, convolution via im2col, a ResNet, a transformer block and a Vision Transformer.
How the weeks work
- Watch and read in the first half of the week, build in the second. If you did not type it, you did not learn it.
- Derive one thing on paper every week. Each one is named in the week's goals. Do not skip them; they are the point.
- Every phase ends in a project week with a definition of done. Ship it, write three paragraphs about it, move on.
- Eight deep notes sit on the weeks where intuition matters most. Read them after the lecture, before you code.
- Flashcards are for the ideas that should live in your head, not in a notebook.
What this roadmap does not cover
LLM pretraining at scale, alignment, RAG and agents. Week 7 builds the transformer block and stops there. The language-model and agent roadmaps pick up from it.
Failure modes to watch
- Tutorial treadmill. Watching Karpathy code is not the same as coding. Close the video and rebuild the file.
- Framework first. Hand-write each idea once (autograd, batch norm, conv, an LSTM cell, attention). After that, use the library.
- Silent bugs. A model that trains is not a model that is right. Check shapes, overfit one batch, look at the initial loss, plot activation statistics.
- Chasing numbers. Hitting 94% on CIFAR-10 matters less than being able to say why the 91% run was worse.
Map
Every week on one canvas. Open a week to see its goals and resources.
What’s inside
12 weeks · 60 goals
Phase 1 · Gradients all the way downWeeks 1–3 · 15 goals
WK 1Backprop from scratch5 goals
Build a scalar autograd engine and see a neural network as a differentiable expression graph.
Goals
- Watch 3Blue1Brown chapters 1 to 4 and sketch the chain of derivatives for one neuron
- Code along with micrograd: Value with +, *, tanh, exp, pow and a topological backward()
- Pass a central-difference gradient check on every op and on a small MLP
- Train a two-layer MLP on four points to loss below 0.01 and plot the loss curve
- Derive on paper why backward() accumulates with += (try b = a * a) and write it in five sentences
Resources
- But what is a neural network?video
- Gradient descent, how neural networks learnvideo
- Backpropagation calculusvideo
- The spelled-out intro to neural networks and backpropagation: building microgradvideo
- Dive into Deep Learning: Forward and backward propagationbook
- Lecture 6: Backpropagation (EECS 498, Justin Johnson)video
WK 2Init, activations and batch norm5 goals
Make a deep MLP trainable: initialization, saturating activations, normalization, and reading activation statistics.
Goals
- Watch makemore parts 2 and 3; pause on the initial loss and compute what it should be
- Rebuild the part 3 network and fix a bad init so the first loss is within 0.05 of ln(27)
- Plot per-layer activation histograms and gradient-to-data ratios before and after Kaiming init
- Write BatchNorm1d by hand, including running stats for eval mode, and compare loss with and without it
- Derive on paper why ReLU layers want weight variance 2/fan_in
Resources
- Building makemore Part 2: MLPvideo
- Building makemore Part 3: Activations & Gradients, BatchNormvideo
- Delving Deep into Rectifiers (He initialization)paper
- Batch Normalization: Accelerating Deep Network Trainingpaper
- Dive into Deep Learning: Numerical Stability and Initializationarticle
- Building makemore Part 4: Becoming a Backprop Ninjavideo
WK 3PyTorch loops and a training recipe5 goals
Own the training loop: tensors, autograd, optimizers, schedules and the debugging habits that save days.
Goals
- Read Karpathy's recipe end to end and write your own eight-line checklist
- Port the MLP to nn.Module, DataLoader and AdamW with seeding, eval, and checkpointing
- Overfit a single batch to near-zero loss, and log the initial loss, before touching the full dataset
- Add a small CNN baseline and a cosine or one-cycle schedule with warmup
- Run six ablations, write the recipe README, then reproduce the result from a clean clone
Phase 2 · Structure in the dataWeeks 4–6 · 15 goals
WK 4Convolutions as weight sharing5 goals
Derive convolution as a constrained linear layer, count parameters and receptive fields, and write conv and pooling by hand.
Goals
- Watch the convolution lecture; hand-compute output shapes and parameter counts for three conv layers
- Write conv2d with four loops, then with unfold and a matmul, and match F.conv2d to 1e-5
- Write max pooling with a backward pass that routes gradient to the argmax
- Train a three-layer CNN on CIFAR-10 to 70% or better and plot the first-layer filters
- Compute the receptive field of a five-layer stack by hand and confirm it with a gradient probe
Resources
- But what is a convolution?video
- Lecture 7: Convolutional Networks (EECS 498, Justin Johnson)video
- CS231n: Convolutional Neural Networks for Visual Recognitionarticle
- Dive into Deep Learning: From Fully Connected Layers to Convolutionsarticle
- A guide to convolution arithmetic for deep learningpaper
- Dive into Deep Learning: LeNetbook
WK 5ResNets and training deeper nets5 goals
See why depth fails without skip connections, build a ResNet, and train it past 90% on CIFAR-10.
Goals
- Watch the architectures lecture and state the degradation problem in two sentences
- Read ResNet and identity-mappings; derive the gradient through y = x + F(x) for a stack of blocks
- Implement BasicBlock and a ResNet-18-style net for 32x32 inputs; check shapes and parameter count
- Train on CIFAR-10 with crop and flip augmentation and a one-cycle schedule to above 90% test accuracy
- Ablate: remove the skips, log per-layer gradient norms at init for both, and plot them
Resources
WK 6Sequence models and why they struggle5 goals
Build an RNN and an LSTM, watch gradients vanish over time, and see why attention was the fix.
Goals
- Read Karpathy's RNN post and Olah's LSTM post, then redraw the LSTM cell from memory
- Watch the recurrent networks lecture and write dh_T/dh_0 as a product of Jacobians
- Hand-write an LSTMCell and match nn.LSTMCell outputs and gradients to 1e-5
- Train a character LSTM on Tiny Shakespeare and sample 500 characters at three temperatures
- Run the copy-at-lag experiment for RNN and LSTM, plot accuracy against lag, and write one paragraph
Resources
- The Unreasonable Effectiveness of Recurrent Neural Networksblog
- Understanding LSTM Networksblog
- Lecture 12: Recurrent Networks (EECS 498, Justin Johnson)video
- On the difficulty of training Recurrent Neural Networkspaper
- Neural Machine Translation by Jointly Learning to Align and Translatepaper
- Visualizing a Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention)blog
Phase 3 · Attention and representationsWeeks 7–9 · 15 goals
WK 7Attention and the transformer block5 goals
Derive attention as a soft lookup, then build a decoder-only transformer block with multi-head attention, residuals and LayerNorm.
Goals
- Watch the attention chapter; derive attention as a soft dictionary lookup on paper
- Write single-head causal attention with loops, then batched matmul; assert rows sum to 1 and nothing leaks from the future
- Build multi-head attention, MLP, residuals and pre-LN LayerNorm as one block with shape asserts
- Train a four-layer character GPT on Tiny Shakespeare and compare validation loss to the week 6 LSTM
- Show numerically that q . k has variance d_k for unit-variance entries, and explain the 1/sqrt(d_k)
WK 8Vision Transformers5 goals
Treat an image as a sequence of patches, build a ViT, and learn when it beats a CNN and when it does not.
Goals
- Watch the walkthrough and read the ViT method; write the shape of every tensor from image to logits
- Implement PatchEmbed, CLS token, learned position embeddings and the head on top of your week 7 block
- Train ViT-tiny on CIFAR-10 with AdamW, warmup and the week 5 augmentation; log curves next to the ResNet
- Ablate: no position embeddings, and patch sizes 2, 4 and 8; one table
- Plot CLS attention for eight test images and write three sentences comparing the ViT and the ResNet
Resources
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (Paper Explained)video
- An Image is Worth 16x16 Words (ViT)paper
- Better plain ViT baselines for ImageNet-1kpaper
- Training data-efficient image transformers and distillation through attention (DeiT)paper
- Do Vision Transformers See Like Convolutional Neural Networks?paper
- lucidrains/vit-pytorchcode
WK 9Representation learning5 goals
Learn features without labels: autoencoders, word2vec, then contrastive learning with SimCLR and the idea behind CLIP.
Goals
- Read Olah, the autoencoder section and the word2vec negative-sampling idea; say why a bottleneck forces features
- Train a 2-D bottleneck autoencoder on Fashion-MNIST and plot the latent space coloured by class
- Implement NT-Xent with a toy-batch test, add SimCLR augmentations and pretrain a ResNet-18 on CIFAR-10
- Linear-probe and kNN-evaluate frozen features against a random-init network and report both
- Read the CLIP method and write one page on what changes versus SimCLR
Resources
- Neural Networks, Manifolds, and Topologyblog
- From Autoencoder to Beta-VAEblog
- Distributed Representations of Words and Phrases and their Compositionalitypaper
- Contrastive Representation Learningblog
- A Simple Framework for Contrastive Learning of Visual Representations (SimCLR)paper
- Learning Transferable Visual Models From Natural Language Supervision (CLIP)paper
Phase 4 · From noise to imagesWeeks 10–12 · 15 goals
WK 10Variational autoencoders5 goals
Derive the ELBO, implement the reparameterization trick, and train a VAE you can sample from and interpolate in.
Goals
- Read Altosaar and Kingma-Welling section 2; derive the ELBO from log p(x) = ELBO + KL(q || p(z|x))
- Derive the closed-form KL between N(mu, sigma^2) and N(0, 1) and check it against torch.distributions
- Train a convolutional VAE (z = 16) on Fashion-MNIST; plot reconstruction and KL separately per epoch
- Make latent traversals, an interpolation grid and prior samples; compare beta = 0.5, 1 and 4
- Force posterior collapse with a huge beta and explain what the per-dimension KL shows
WK 11Diffusion models5 goals
Understand the forward noising process, the noise-prediction loss and the score view; sample with DDPM and add classifier-free guidance.
Goals
- Watch the Welch Labs video and read Weng's forward process; derive q(x_t | x_0) from the one-step recursion
- Read DDPM sections 2 and 3; show why the simplified loss is MSE on epsilon and write both algorithms
- Train an epsilon-network on a 2-D toy dataset, sample with DDPM and plot noise-to-data trajectories
- Add class conditioning with 10% label dropout and sweep the guidance scale
- Explain the score view (epsilon-hat as a scaled score) and why DDIM sampling is deterministic
Resources
- But how do AI images and videos actually work? (Guest video by Welch Labs)video
- What are Diffusion Models?blog
- Denoising Diffusion Probabilistic Modelspaper
- Generative Modeling by Estimating Gradients of the Data Distributionblog
- Classifier-Free Diffusion Guidancepaper
- MIT 6.S184: Flow Matching and Diffusion Models, Lecture 1video
WK 12Capstone: diffusion from scratch5 goals
Train a small class-conditional U-Net diffusion model on a small image dataset and ship it with samples, a guidance sweep and an honest write-up.
Goals
- Read the annotated diffusion post; list every U-Net component and where the timestep enters
- Build the U-Net with a sinusoidal time embedding, class conditioning and EMA; overfit one batch first
- Train on Fashion-MNIST or CIFAR-10; save a sample grid every N epochs from both raw and EMA weights
- Add a DDIM sampler (20 to 50 steps), a guidance-scale sweep grid and a nearest-training-neighbour check
- Write the README: loss curve, sample grids, what failed, and what you would change next
Resources
- The Annotated Diffusion Modelblog
- Denoising Diffusion Implicit Modelspaper
- Improved Denoising Diffusion Probabilistic Modelspaper
- MIT 6.S184: Flow Matching and Diffusion Models, Lecture 3B: Classifier-free Guidancevideo
- huggingface/diffusion-models-classcode
- Scalable Diffusion Models with Transformers (DiT)paper
Notes
8 published notes
Diffusion: noise in, image out
Nov 2, 2026- Nov 2, 2026
- Nov 2, 2026
- Nov 2, 2026
Residual connections and gradient flow
Nov 2, 2026- Nov 2, 2026
Backprop is the chain rule with bookkeeping
Nov 2, 2026