Diffusion: noise in, image out

A diffusion model learns to undo a very small amount of noise, and generates an image by undoing it many times in a row. Take an image and add a little Gaussian noise. Repeat a thousand times and nothing is left but static. Because each step was tiny, reversing a single step is an easy problem: look at a noisy image and guess the noise that was added. Train a network to do that, then start from pure noise and walk backward.

The forward process has a shortcut

Each noising step is $x_t = \sqrt{1-\beta_t},x_{t-1} + \sqrt{\beta_t},\epsilon$ with a small variance $\beta_t$. Let $\alpha_t = 1 - \beta_t$ and $\bar\alpha_t = \prod_{s \le t}\alpha_s$. Composing Gaussians gives a closed form that jumps straight to any step:

$$x_t = \sqrt{\bar\alpha_t},x_0 + \sqrt{1 - \bar\alpha_t},\epsilon, \qquad \epsilon \sim \mathcal{N}(0, I)$$

For the original linear schedule, $\beta$ from $10^{-4}$ to $0.02$ over $T = 1000$ steps, $\bar\alpha_{100} = 0.90$, $\bar\alpha_{250} = 0.52$, $\bar\alpha_{500} = 0.079$ and $\bar\alpha_{1000} = 4.0 \times 10^{-5}$. By the end only $\sqrt{4 \times 10^{-5}} = 0.006$ of the image is left. That is pure noise for practical purposes, which is what lets you start sampling from $\mathcal{N}(0, I)$.

Worked example with a single pixel $x_0 = 0.8$ at a step where $\bar\alpha = 0.25$ and the noise draw is $\epsilon = 1$: $x_t = 0.5 \cdot 0.8 + 0.866 \cdot 1 = 1.266$. If the network predicts $\epsilon$ exactly, you recover $x_0 = (x_t - 0.866,\epsilon)/0.5 = 0.8$. Predicting the noise and predicting the clean image are the same problem, written two ways.

Training is regression on noise

Pick a random image, a random $t$ and a random $\epsilon$. Build $x_t$ with the formula above. Ask the network $\epsilon_\theta(x_t, t)$ for the noise and minimize

$$\mathcal{L} = \mathbb{E},\big|\epsilon - \epsilon_\theta(x_t, t)\big|^2$$

This is the simplified DDPM objective. It comes from a variational bound, like the ELBO of last week, with the weights dropped. It is plain mean squared error, with no adversary, no mode collapse and no tricks.

Sampling walks backward

Start from $x_T \sim \mathcal{N}(0, I)$ and repeat

$$x_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left(x_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}},\epsilon_\theta(x_t, t)\right) + \sigma_t z, \qquad z \sim \mathcal{N}(0, I)$$

with $\sigma_t^2 = \beta_t$ and no noise ($z = 0$) in the last step. The bracket removes the predicted noise; the added $\sigma_t z$ keeps the chain stochastic so it samples a distribution rather than collapsing to one point.

import torch
import torch.nn.functional as F

T = 1000
betas = torch.linspace(1e-4, 0.02, T)
alphas = 1.0 - betas
abar = torch.cumprod(alphas, dim=0)


def train_step(model, x0, opt):
    t = torch.randint(0, T, (x0.size(0),))
    eps = torch.randn_like(x0)
    a = abar[t].view(-1, *([1] * (x0.dim() - 1)))
    xt = a.sqrt() * x0 + (1 - a).sqrt() * eps
    loss = F.mse_loss(model(xt, t), eps)
    opt.zero_grad()
    loss.backward()
    opt.step()
    return loss.item()


@torch.no_grad()
def sample(model, shape):
    x = torch.randn(shape)
    for t in reversed(range(T)):
        eps = model(x, torch.full((shape[0],), t))
        mean = (x - betas[t] / (1 - abar[t]).sqrt() * eps) / alphas[t].sqrt()
        x = mean + (betas[t].sqrt() * torch.randn_like(x) if t > 0 else 0)
    return x

Twenty lines for training and sampling. The network (a U-Net, later a transformer) is where the engineering lives.

The score view

Differentiate the log of the forward Gaussian with respect to $x_t$:

$$\nabla_{x_t}\log q(x_t \mid x_0) = -\frac{x_t - \sqrt{\bar\alpha_t},x_0}{1-\bar\alpha_t} = -\frac{\epsilon}{\sqrt{1-\bar\alpha_t}}$$

So predicting the noise is the same as predicting the score, the direction in which the density of noisy images increases fastest: $\epsilon_\theta \approx -\sqrt{1-\bar\alpha_t},\nabla_x\log p_t(x)$. Sampling is following that arrow uphill, plus noise. Replace the noise with nothing and you get DDIM, a deterministic ODE that uses the same trained network but can skip steps, so 20 to 50 steps replace 1000.

Guidance

For a conditional model, train $\epsilon_\theta(x_t, t, c)$ with the condition $c$ dropped (replaced by a null token) about 10% of the time, so one network learns both the conditional and the unconditional prediction. At sampling time extrapolate:

$$\tilde\epsilon = \epsilon_\theta(x_t, t, \varnothing) + w,\big(\epsilon_\theta(x_t, t, c) - \epsilon_\theta(x_t, t, \varnothing)\big)$$

With $w = 1$ you get the plain conditional model; $w > 1$ moves further along the direction that distinguishes the condition from its absence. Samples become more faithful to the label and less diverse. The original paper parameterizes the same thing as $(1+w)\epsilon_c - w\epsilon_\varnothing$, so check which convention a codebase uses before comparing numbers.

Common mistakes

  • Off-by-one in the schedule. Index $\bar\alpha$ with $t$ in $[0, T-1]$ consistently in training and sampling.
  • Not scaling data to $[-1, 1]$. The noise assumes roughly unit-scale data.
  • Sampling with raw weights. Keep an exponential moving average of the weights and sample with it; the difference in sample quality is large.
  • Believing the loss curve. Diffusion loss is noisy and flattens early. Look at samples.
  • Mixing up guidance conventions. A scale of 7 means different things under the two formulas.

Check yourself

  1. Starting from $x_t = \sqrt{\alpha_t},x_{t-1} + \sqrt{1-\alpha_t},\epsilon$, derive $\bar\alpha_t$ and the closed form for $x_t$ given $x_0$.
  2. For $\bar\alpha = 0.09$, $x_0 = 1$ and $\epsilon = -1$, what is $x_t$?
  3. Why is the loss a plain MSE when the objective came from a variational bound?
  4. What does classifier-free guidance cost at sampling time compared with an unconditional model?