@omarUpdated Nov 3, 2026

Deep Learning from Scratch: backprop to diffusion

For programmers who know Python and a little ML: 12 weeks, 7 to 10 hours a week, from hand-written autograd to a diffusion model you train yourself. Four projects, no black boxes.

Following gives you your own copy, dated from today. Free to start.

4 phases · 12 weeks

About this roadmap

For programmers who want to build neural networks, not just call them. Twelve weeks, four phases, four things you will have built. Plan on 7 to 10 hours a week: learn early in the week, build late in it. The capstone week is heavier.

What you will have built

  • Week 3: a reproducible training recipe: an MLP and a small CNN on Fashion-MNIST, one train.py, learning curves and an ablation table.
  • Week 6: an LSTM cell written by hand that matches PyTorch, a character-level model, and a plot of exactly where recurrence runs out of memory.
  • Week 9: SimCLR-lite: a ResNet pretrained with no labels on CIFAR-10, judged by a linear probe against a random-weights baseline.
  • Week 12: a class-conditional diffusion model trained from scratch, with fast sampling and a classifier-free guidance sweep.

Along the way you also write micrograd, batch norm, convolution via im2col, a ResNet, a transformer block and a Vision Transformer.

How the weeks work

  1. Watch and read in the first half of the week, build in the second. If you did not type it, you did not learn it.
  2. Derive one thing on paper every week. Each one is named in the week's goals. Do not skip them; they are the point.
  3. Every phase ends in a project week with a definition of done. Ship it, write three paragraphs about it, move on.
  4. Eight deep notes sit on the weeks where intuition matters most. Read them after the lecture, before you code.
  5. Flashcards are for the ideas that should live in your head, not in a notebook.

What this roadmap does not cover

LLM pretraining at scale, alignment, RAG and agents. Week 7 builds the transformer block and stops there. The language-model and agent roadmaps pick up from it.

Failure modes to watch

  • Tutorial treadmill. Watching Karpathy code is not the same as coding. Close the video and rebuild the file.
  • Framework first. Hand-write each idea once (autograd, batch norm, conv, an LSTM cell, attention). After that, use the library.
  • Silent bugs. A model that trains is not a model that is right. Check shapes, overfit one batch, look at the initial loss, plot activation statistics.
  • Chasing numbers. Hitting 94% on CIFAR-10 matters less than being able to say why the 91% run was worse.

Map

Every week on one canvas. Open a week to see its goals and resources.

What’s inside

12 weeks · 60 goals

Phase 1 · Gradients all the way downWeeks 1–3 · 15 goals
WK 1Backprop from scratch5 goals

Build a scalar autograd engine and see a neural network as a differentiable expression graph.

Goals

  • Watch 3Blue1Brown chapters 1 to 4 and sketch the chain of derivatives for one neuron
  • Code along with micrograd: Value with +, *, tanh, exp, pow and a topological backward()
  • Pass a central-difference gradient check on every op and on a small MLP
  • Train a two-layer MLP on four points to loss below 0.01 and plot the loss curve
  • Derive on paper why backward() accumulates with += (try b = a * a) and write it in five sentences
WK 2Init, activations and batch norm5 goals

Make a deep MLP trainable: initialization, saturating activations, normalization, and reading activation statistics.

Goals

  • Watch makemore parts 2 and 3; pause on the initial loss and compute what it should be
  • Rebuild the part 3 network and fix a bad init so the first loss is within 0.05 of ln(27)
  • Plot per-layer activation histograms and gradient-to-data ratios before and after Kaiming init
  • Write BatchNorm1d by hand, including running stats for eval mode, and compare loss with and without it
  • Derive on paper why ReLU layers want weight variance 2/fan_in
WK 3PyTorch loops and a training recipe5 goals

Own the training loop: tensors, autograd, optimizers, schedules and the debugging habits that save days.

Goals

  • Read Karpathy's recipe end to end and write your own eight-line checklist
  • Port the MLP to nn.Module, DataLoader and AdamW with seeding, eval, and checkpointing
  • Overfit a single batch to near-zero loss, and log the initial loss, before touching the full dataset
  • Add a small CNN baseline and a cosine or one-cycle schedule with warmup
  • Run six ablations, write the recipe README, then reproduce the result from a clean clone
Phase 2 · Structure in the dataWeeks 4–6 · 15 goals
WK 4Convolutions as weight sharing5 goals

Derive convolution as a constrained linear layer, count parameters and receptive fields, and write conv and pooling by hand.

Goals

  • Watch the convolution lecture; hand-compute output shapes and parameter counts for three conv layers
  • Write conv2d with four loops, then with unfold and a matmul, and match F.conv2d to 1e-5
  • Write max pooling with a backward pass that routes gradient to the argmax
  • Train a three-layer CNN on CIFAR-10 to 70% or better and plot the first-layer filters
  • Compute the receptive field of a five-layer stack by hand and confirm it with a gradient probe
WK 5ResNets and training deeper nets5 goals

See why depth fails without skip connections, build a ResNet, and train it past 90% on CIFAR-10.

Goals

  • Watch the architectures lecture and state the degradation problem in two sentences
  • Read ResNet and identity-mappings; derive the gradient through y = x + F(x) for a stack of blocks
  • Implement BasicBlock and a ResNet-18-style net for 32x32 inputs; check shapes and parameter count
  • Train on CIFAR-10 with crop and flip augmentation and a one-cycle schedule to above 90% test accuracy
  • Ablate: remove the skips, log per-layer gradient norms at init for both, and plot them
WK 6Sequence models and why they struggle5 goals

Build an RNN and an LSTM, watch gradients vanish over time, and see why attention was the fix.

Goals

  • Read Karpathy's RNN post and Olah's LSTM post, then redraw the LSTM cell from memory
  • Watch the recurrent networks lecture and write dh_T/dh_0 as a product of Jacobians
  • Hand-write an LSTMCell and match nn.LSTMCell outputs and gradients to 1e-5
  • Train a character LSTM on Tiny Shakespeare and sample 500 characters at three temperatures
  • Run the copy-at-lag experiment for RNN and LSTM, plot accuracy against lag, and write one paragraph
Phase 3 · Attention and representationsWeeks 7–9 · 15 goals
WK 7Attention and the transformer block5 goals

Derive attention as a soft lookup, then build a decoder-only transformer block with multi-head attention, residuals and LayerNorm.

Goals

  • Watch the attention chapter; derive attention as a soft dictionary lookup on paper
  • Write single-head causal attention with loops, then batched matmul; assert rows sum to 1 and nothing leaks from the future
  • Build multi-head attention, MLP, residuals and pre-LN LayerNorm as one block with shape asserts
  • Train a four-layer character GPT on Tiny Shakespeare and compare validation loss to the week 6 LSTM
  • Show numerically that q . k has variance d_k for unit-variance entries, and explain the 1/sqrt(d_k)
WK 8Vision Transformers5 goals

Treat an image as a sequence of patches, build a ViT, and learn when it beats a CNN and when it does not.

Goals

  • Watch the walkthrough and read the ViT method; write the shape of every tensor from image to logits
  • Implement PatchEmbed, CLS token, learned position embeddings and the head on top of your week 7 block
  • Train ViT-tiny on CIFAR-10 with AdamW, warmup and the week 5 augmentation; log curves next to the ResNet
  • Ablate: no position embeddings, and patch sizes 2, 4 and 8; one table
  • Plot CLS attention for eight test images and write three sentences comparing the ViT and the ResNet
WK 9Representation learning5 goals

Learn features without labels: autoencoders, word2vec, then contrastive learning with SimCLR and the idea behind CLIP.

Goals

  • Read Olah, the autoencoder section and the word2vec negative-sampling idea; say why a bottleneck forces features
  • Train a 2-D bottleneck autoencoder on Fashion-MNIST and plot the latent space coloured by class
  • Implement NT-Xent with a toy-batch test, add SimCLR augmentations and pretrain a ResNet-18 on CIFAR-10
  • Linear-probe and kNN-evaluate frozen features against a random-init network and report both
  • Read the CLIP method and write one page on what changes versus SimCLR
Phase 4 · From noise to imagesWeeks 10–12 · 15 goals
WK 10Variational autoencoders5 goals

Derive the ELBO, implement the reparameterization trick, and train a VAE you can sample from and interpolate in.

Goals

  • Read Altosaar and Kingma-Welling section 2; derive the ELBO from log p(x) = ELBO + KL(q || p(z|x))
  • Derive the closed-form KL between N(mu, sigma^2) and N(0, 1) and check it against torch.distributions
  • Train a convolutional VAE (z = 16) on Fashion-MNIST; plot reconstruction and KL separately per epoch
  • Make latent traversals, an interpolation grid and prior samples; compare beta = 0.5, 1 and 4
  • Force posterior collapse with a huge beta and explain what the per-dimension KL shows
WK 11Diffusion models5 goals

Understand the forward noising process, the noise-prediction loss and the score view; sample with DDPM and add classifier-free guidance.

Goals

  • Watch the Welch Labs video and read Weng's forward process; derive q(x_t | x_0) from the one-step recursion
  • Read DDPM sections 2 and 3; show why the simplified loss is MSE on epsilon and write both algorithms
  • Train an epsilon-network on a 2-D toy dataset, sample with DDPM and plot noise-to-data trajectories
  • Add class conditioning with 10% label dropout and sweep the guidance scale
  • Explain the score view (epsilon-hat as a scaled score) and why DDIM sampling is deterministic
WK 12Capstone: diffusion from scratch5 goals

Train a small class-conditional U-Net diffusion model on a small image dataset and ship it with samples, a guidance sweep and an honest write-up.

Goals

  • Read the annotated diffusion post; list every U-Net component and where the timestep enters
  • Build the U-Net with a sinusoidal time embedding, class conditioning and EMA; overfit one batch first
  • Train on Fashion-MNIST or CIFAR-10; save a sample grid every N epochs from both raw and EMA weights
  • Add a DDIM sampler (20 to 50 steps), a guidance-scale sweep grid and a nearest-training-neighbour check
  • Write the README: loss curve, sample grids, what failed, and what you would change next