# The ELBO without tears

By @omar · Published Nov 2, 2026 · From the roadmap [Deep Learning from Scratch: backprop to diffusion](https://ornix.io/u/omar/deep-learning-from-scratch)

A VAE is a generative model with a hidden variable $z$: draw $z$ from a simple prior, then decode it into $x$. Training it means maximizing $\log p(x)$, and that is where the trouble is. The evidence $p(x) = \int p(x \mid z)\,p(z)\,dz$ is an integral over every possible $z$, and there is no closed form for a neural-network decoder. The ELBO is a bound we can compute and optimize instead.

## The bound, derived two ways

Introduce any distribution $q(z \mid x)$ over the latent, the encoder. **First derivation**, with Jensen's inequality ($\log$ is concave):

$$\log p(x) = \log \mathbb{E}_{q}\!\left[\frac{p(x, z)}{q(z \mid x)}\right] \ge \mathbb{E}_{q}\!\left[\log p(x, z) - \log q(z \mid x)\right] = \mathrm{ELBO}$$

**Second derivation**, an identity with no inequality in it:

$$\log p(x) = \mathrm{ELBO} + \mathrm{KL}\big(q(z \mid x)\,\|\,p(z \mid x)\big)$$

The second form tells you more. The gap between the bound and the truth is the KL divergence from your encoder to the **true posterior**, which is never negative, so ELBO $\le \log p(x)$, with equality exactly when $q$ equals the posterior. Maximizing the ELBO does two things at once: it raises the likelihood and it pulls the encoder toward the true posterior.

Split $p(x, z) = p(x \mid z)\,p(z)$ and the same quantity reads

$$\mathrm{ELBO} = \underbrace{\mathbb{E}_{q(z \mid x)}\big[\log p(x \mid z)\big]}_{\text{reconstruction}} - \underbrace{\mathrm{KL}\big(q(z \mid x)\,\|\,p(z)\big)}_{\text{keep the code simple}}$$

The first term wants the decoder to rebuild $x$ from $z$. The second wants the encoder's distributions to stay close to the prior, so that sampling $z$ from the prior later lands in places the decoder has seen.

```mermaid
%% ornix:id=elbo0001
flowchart LR
    X["image x"] --> ENC["encoder"]
    ENC --> MU["mu"]
    ENC --> LV["log sigma^2"]
    MU --> Z["z = mu + sigma * eps"]
    LV --> Z
    EPS["eps ~ N(0, I)"] --> Z
    Z --> DEC["decoder"]
    DEC --> XH["x-hat"]
    XH --> REC["reconstruction loss vs x"]
    MU --> KL["KL(q || N(0, I))"]
    LV --> KL
```

## The KL has a closed form

With $q = \mathcal{N}(\mu, \mathrm{diag}\,\sigma^2)$ and prior $\mathcal{N}(0, I)$, the KL for one latent dimension is

$$\mathrm{KL} = \tfrac12\left(\mu^2 + \sigma^2 - \ln\sigma^2 - 1\right)$$

Worked example, $\mu = 0.5$ and $\sigma = 0.8$: $\ \tfrac12(0.25 + 0.64 + 0.446 - 1) = 0.168$ nats. The encoder pays a small price for being off-centre and narrower than the prior. If $\mu = 0$ and $\sigma = 1$ the KL is exactly 0, the encoder has said nothing about $x$, and that dimension is **unused**. Sum over dimensions to get the total.

## The reparameterization trick

The reconstruction term is an expectation over $z \sim q_{\phi}(z \mid x)$. You cannot backpropagate through "sample from this distribution". The trick moves the randomness into a fixed input:

$$z = \mu_\phi(x) + \sigma_\phi(x) \odot \epsilon, \qquad \epsilon \sim \mathcal{N}(0, I)$$

Now $z$ is a deterministic, differentiable function of $\phi$ given $\epsilon$, so gradients flow into the encoder's $\mu$ and $\sigma$. One sample per example is enough in practice.

## The loss in code

```python
import torch
import torch.nn.functional as F


def sample_latent(mu, logvar):
    eps = torch.randn_like(mu)
    return mu + (0.5 * logvar).exp() * eps


def vae_loss(x, x_logits, mu, logvar, beta=1.0):
    n = x.size(0)
    recon = F.binary_cross_entropy_with_logits(x_logits, x, reduction="sum") / n
    kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp()) / n
    return recon + beta * kl, recon, kl
```

Predicting `logvar` rather than $\sigma$ keeps the variance positive without a constraint. The reconstruction is summed over pixels and the KL over latent dimensions, and both are averaged over the batch. Mixing a mean over pixels with a sum over latents changes the effective $\beta$ by a factor of hundreds, which is a classic way to get a model that either ignores its latent or ignores its input.

## What the knobs do

- **$\beta$ above 1** pushes harder on the KL: smoother, more regular latents, blurrier reconstructions.
- **$\beta$ near 0** gives a plain autoencoder: sharp reconstructions, a latent space with holes, and prior samples that look like junk.
- **Posterior collapse.** If the decoder is strong enough to model $x$ alone, the cheapest way to cut the KL is $q = p(z)$ in every dimension. Watch the per-dimension KL: dimensions at 0 are dead.
- **Blurriness.** A Gaussian or Bernoulli likelihood averages over plausible outputs. This is a property of the objective, not of your code, and part of the motivation for diffusion models two weeks from now.

## Common mistakes

- **Using $\sigma$ where the code holds $\log \sigma^2$.** The two formulas look alike and differ by a factor.
- **Sampling at test time.** For reconstruction, decode $\mu$. For generation, decode a draw from the prior.
- **Judging the model only by the total loss.** A falling total can hide a KL that went to zero.

## Check yourself

1. Which term in $\log p(x) = \mathrm{ELBO} + \mathrm{KL}(q \,\|\, p(z \mid x))$ is intractable and why?
2. Compute the KL for $\mu = 0$, $\sigma = 0.5$ in one dimension.
3. Why does $z = \mu + \sigma\epsilon$ allow backpropagation when "sample $z$" does not?
4. What does a latent dimension with KL equal to 0 tell you?

---

Roadmap: https://ornix.io/u/omar/deep-learning-from-scratch (Markdown: https://ornix.io/u/omar/deep-learning-from-scratch.md)

Canonical page: https://ornix.io/u/omar/deep-learning-from-scratch/elbo-without-tears
