# Diffusion: noise in, image out

By @omar · Published Nov 2, 2026 · From the roadmap [Deep Learning from Scratch: backprop to diffusion](https://ornix.io/u/omar/deep-learning-from-scratch)

A diffusion model learns to undo a very small amount of noise, and generates an image by undoing it many times in a row. Take an image and add a little Gaussian noise. Repeat a thousand times and nothing is left but static. Because each step was tiny, reversing a single step is an easy problem: look at a noisy image and guess the noise that was added. Train a network to do that, then start from pure noise and walk backward.

```mermaid
%% ornix:id=diff0001
flowchart LR
    X0["x0 (image)"] -->|"add a little noise"| XT["x_t (noisier)"]
    XT -->|"add a little noise"| XN["x_T (pure noise)"]
    XN -->|"network removes a little noise"| Y1["x_(t-1)"]
    Y1 -->|"repeat T times"| Y0["new image"]
```

## The forward process has a shortcut

Each noising step is $x_t = \sqrt{1-\beta_t}\,x_{t-1} + \sqrt{\beta_t}\,\epsilon$ with a small variance $\beta_t$. Let $\alpha_t = 1 - \beta_t$ and $\bar\alpha_t = \prod_{s \le t}\alpha_s$. Composing Gaussians gives a closed form that jumps straight to any step:

$$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1 - \bar\alpha_t}\,\epsilon, \qquad \epsilon \sim \mathcal{N}(0, I)$$

For the original linear schedule, $\beta$ from $10^{-4}$ to $0.02$ over $T = 1000$ steps, $\bar\alpha_{100} = 0.90$, $\bar\alpha_{250} = 0.52$, $\bar\alpha_{500} = 0.079$ and $\bar\alpha_{1000} = 4.0 \times 10^{-5}$. By the end only $\sqrt{4 \times 10^{-5}} = 0.006$ of the image is left. That is pure noise for practical purposes, which is what lets you start sampling from $\mathcal{N}(0, I)$.

Worked example with a single pixel $x_0 = 0.8$ at a step where $\bar\alpha = 0.25$ and the noise draw is $\epsilon = 1$: $x_t = 0.5 \cdot 0.8 + 0.866 \cdot 1 = 1.266$. If the network predicts $\epsilon$ exactly, you recover $x_0 = (x_t - 0.866\,\epsilon)/0.5 = 0.8$. Predicting the noise and predicting the clean image are the same problem, written two ways.

## Training is regression on noise

Pick a random image, a random $t$ and a random $\epsilon$. Build $x_t$ with the formula above. Ask the network $\epsilon_\theta(x_t, t)$ for the noise and minimize

$$\mathcal{L} = \mathbb{E}\,\big\|\epsilon - \epsilon_\theta(x_t, t)\big\|^2$$

This is the simplified DDPM objective. It comes from a variational bound, like the ELBO of last week, with the weights dropped. It is plain mean squared error, with no adversary, no mode collapse and no tricks.

## Sampling walks backward

Start from $x_T \sim \mathcal{N}(0, I)$ and repeat

$$x_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left(x_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\,\epsilon_\theta(x_t, t)\right) + \sigma_t z, \qquad z \sim \mathcal{N}(0, I)$$

with $\sigma_t^2 = \beta_t$ and no noise ($z = 0$) in the last step. The bracket removes the predicted noise; the added $\sigma_t z$ keeps the chain stochastic so it samples a distribution rather than collapsing to one point.

```python
import torch
import torch.nn.functional as F

T = 1000
betas = torch.linspace(1e-4, 0.02, T)
alphas = 1.0 - betas
abar = torch.cumprod(alphas, dim=0)


def train_step(model, x0, opt):
    t = torch.randint(0, T, (x0.size(0),))
    eps = torch.randn_like(x0)
    a = abar[t].view(-1, *([1] * (x0.dim() - 1)))
    xt = a.sqrt() * x0 + (1 - a).sqrt() * eps
    loss = F.mse_loss(model(xt, t), eps)
    opt.zero_grad()
    loss.backward()
    opt.step()
    return loss.item()


@torch.no_grad()
def sample(model, shape):
    x = torch.randn(shape)
    for t in reversed(range(T)):
        eps = model(x, torch.full((shape[0],), t))
        mean = (x - betas[t] / (1 - abar[t]).sqrt() * eps) / alphas[t].sqrt()
        x = mean + (betas[t].sqrt() * torch.randn_like(x) if t > 0 else 0)
    return x
```

Twenty lines for training and sampling. The network (a U-Net, later a transformer) is where the engineering lives.

## The score view

Differentiate the log of the forward Gaussian with respect to $x_t$:

$$\nabla_{x_t}\log q(x_t \mid x_0) = -\frac{x_t - \sqrt{\bar\alpha_t}\,x_0}{1-\bar\alpha_t} = -\frac{\epsilon}{\sqrt{1-\bar\alpha_t}}$$

So predicting the noise is the same as predicting the **score**, the direction in which the density of noisy images increases fastest: $\epsilon_\theta \approx -\sqrt{1-\bar\alpha_t}\,\nabla_x\log p_t(x)$. Sampling is following that arrow uphill, plus noise. Replace the noise with nothing and you get DDIM, a deterministic ODE that uses the same trained network but can skip steps, so 20 to 50 steps replace 1000.

## Guidance

For a conditional model, train $\epsilon_\theta(x_t, t, c)$ with the condition $c$ dropped (replaced by a null token) about 10% of the time, so one network learns both the conditional and the unconditional prediction. At sampling time extrapolate:

$$\tilde\epsilon = \epsilon_\theta(x_t, t, \varnothing) + w\,\big(\epsilon_\theta(x_t, t, c) - \epsilon_\theta(x_t, t, \varnothing)\big)$$

With $w = 1$ you get the plain conditional model; $w > 1$ moves further along the direction that distinguishes the condition from its absence. Samples become more faithful to the label and less diverse. The original paper parameterizes the same thing as $(1+w)\epsilon_c - w\epsilon_\varnothing$, so check which convention a codebase uses before comparing numbers.

## Common mistakes

- **Off-by-one in the schedule.** Index $\bar\alpha$ with $t$ in $[0, T-1]$ consistently in training and sampling.
- **Not scaling data to $[-1, 1]$.** The noise assumes roughly unit-scale data.
- **Sampling with raw weights.** Keep an exponential moving average of the weights and sample with it; the difference in sample quality is large.
- **Believing the loss curve.** Diffusion loss is noisy and flattens early. Look at samples.
- **Mixing up guidance conventions.** A scale of 7 means different things under the two formulas.

## Check yourself

1. Starting from $x_t = \sqrt{\alpha_t}\,x_{t-1} + \sqrt{1-\alpha_t}\,\epsilon$, derive $\bar\alpha_t$ and the closed form for $x_t$ given $x_0$.
2. For $\bar\alpha = 0.09$, $x_0 = 1$ and $\epsilon = -1$, what is $x_t$?
3. Why is the loss a plain MSE when the objective came from a variational bound?
4. What does classifier-free guidance cost at sampling time compared with an unconditional model?

---

Roadmap: https://ornix.io/u/omar/deep-learning-from-scratch (Markdown: https://ornix.io/u/omar/deep-learning-from-scratch.md)

Canonical page: https://ornix.io/u/omar/deep-learning-from-scratch/diffusion-noise-in-image-out
