The ELBO without tears

A VAE is a generative model with a hidden variable $z$: draw $z$ from a simple prior, then decode it into $x$. Training it means maximizing $\log p(x)$, and that is where the trouble is. The evidence $p(x) = \int p(x \mid z),p(z),dz$ is an integral over every possible $z$, and there is no closed form for a neural-network decoder. The ELBO is a bound we can compute and optimize instead.

The bound, derived two ways

Introduce any distribution $q(z \mid x)$ over the latent, the encoder. First derivation, with Jensen's inequality ($\log$ is concave):

$$\log p(x) = \log \mathbb{E}{q}!\left[\frac{p(x, z)}{q(z \mid x)}\right] \ge \mathbb{E}{q}!\left[\log p(x, z) - \log q(z \mid x)\right] = \mathrm{ELBO}$$

Second derivation, an identity with no inequality in it:

$$\log p(x) = \mathrm{ELBO} + \mathrm{KL}\big(q(z \mid x),|,p(z \mid x)\big)$$

The second form tells you more. The gap between the bound and the truth is the KL divergence from your encoder to the true posterior, which is never negative, so ELBO $\le \log p(x)$, with equality exactly when $q$ equals the posterior. Maximizing the ELBO does two things at once: it raises the likelihood and it pulls the encoder toward the true posterior.

Split $p(x, z) = p(x \mid z),p(z)$ and the same quantity reads

$$\mathrm{ELBO} = \underbrace{\mathbb{E}{q(z \mid x)}\big[\log p(x \mid z)\big]}{\text{reconstruction}} - \underbrace{\mathrm{KL}\big(q(z \mid x),|,p(z)\big)}_{\text{keep the code simple}}$$

The first term wants the decoder to rebuild $x$ from $z$. The second wants the encoder's distributions to stay close to the prior, so that sampling $z$ from the prior later lands in places the decoder has seen.

The KL has a closed form

With $q = \mathcal{N}(\mu, \mathrm{diag},\sigma^2)$ and prior $\mathcal{N}(0, I)$, the KL for one latent dimension is

$$\mathrm{KL} = \tfrac12\left(\mu^2 + \sigma^2 - \ln\sigma^2 - 1\right)$$

Worked example, $\mu = 0.5$ and $\sigma = 0.8$: $\ \tfrac12(0.25 + 0.64 + 0.446 - 1) = 0.168$ nats. The encoder pays a small price for being off-centre and narrower than the prior. If $\mu = 0$ and $\sigma = 1$ the KL is exactly 0, the encoder has said nothing about $x$, and that dimension is unused. Sum over dimensions to get the total.

The reparameterization trick

The reconstruction term is an expectation over $z \sim q_{\phi}(z \mid x)$. You cannot backpropagate through "sample from this distribution". The trick moves the randomness into a fixed input:

$$z = \mu_\phi(x) + \sigma_\phi(x) \odot \epsilon, \qquad \epsilon \sim \mathcal{N}(0, I)$$

Now $z$ is a deterministic, differentiable function of $\phi$ given $\epsilon$, so gradients flow into the encoder's $\mu$ and $\sigma$. One sample per example is enough in practice.

The loss in code

import torch
import torch.nn.functional as F


def sample_latent(mu, logvar):
    eps = torch.randn_like(mu)
    return mu + (0.5 * logvar).exp() * eps


def vae_loss(x, x_logits, mu, logvar, beta=1.0):
    n = x.size(0)
    recon = F.binary_cross_entropy_with_logits(x_logits, x, reduction="sum") / n
    kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp()) / n
    return recon + beta * kl, recon, kl

Predicting logvar rather than $\sigma$ keeps the variance positive without a constraint. The reconstruction is summed over pixels and the KL over latent dimensions, and both are averaged over the batch. Mixing a mean over pixels with a sum over latents changes the effective $\beta$ by a factor of hundreds, which is a classic way to get a model that either ignores its latent or ignores its input.

What the knobs do

  • $\beta$ above 1 pushes harder on the KL: smoother, more regular latents, blurrier reconstructions.
  • $\beta$ near 0 gives a plain autoencoder: sharp reconstructions, a latent space with holes, and prior samples that look like junk.
  • Posterior collapse. If the decoder is strong enough to model $x$ alone, the cheapest way to cut the KL is $q = p(z)$ in every dimension. Watch the per-dimension KL: dimensions at 0 are dead.
  • Blurriness. A Gaussian or Bernoulli likelihood averages over plausible outputs. This is a property of the objective, not of your code, and part of the motivation for diffusion models two weeks from now.

Common mistakes

  • Using $\sigma$ where the code holds $\log \sigma^2$. The two formulas look alike and differ by a factor.
  • Sampling at test time. For reconstruction, decode $\mu$. For generation, decode a draw from the prior.
  • Judging the model only by the total loss. A falling total can hide a KL that went to zero.

Check yourself

  1. Which term in $\log p(x) = \mathrm{ELBO} + \mathrm{KL}(q ,|, p(z \mid x))$ is intractable and why?
  2. Compute the KL for $\mu = 0$, $\sigma = 0.5$ in one dimension.
  3. Why does $z = \mu + \sigma\epsilon$ allow backpropagation when "sample $z$" does not?
  4. What does a latent dimension with KL equal to 0 tell you?