Superposition and sparse autoencoders

A neuron is not a concept. If it were, a layer with 768 neurons could represent 768 things, and language models plainly know far more than that. Interpretability work since 2022 explains the gap with one idea, superposition, and extracts what is stored there with sparse autoencoders.

Superposition

Suppose you want to store $n$ features in an $m$-dimensional activation, with $n > m$. If features were always active together you could not. But most features are sparse: "this token is part of a Python traceback" is false almost everywhere. If two features are rarely on at once, they can share dimensions and the interference rarely matters.

Anthropic's Toy Models of Superposition makes this concrete. Features $x \in \mathbb{R}^n$ are compressed to $h = Wx \in \mathbb{R}^m$ and decoded with a ReLU:

$$\hat{x} = \mathrm{ReLU}(W^{\top} h + b)$$

Train it with an importance-weighted reconstruction loss while varying how sparse the features are. At low sparsity the model keeps only the most important $m$ features and drops the rest. As sparsity rises it packs in more. With 5 features and 2 dimensions the learned directions form a pentagon, 72 degrees apart.

Why is that tolerable? Adjacent directions have $\cos 72^\circ = 0.31$, so activating one produces a small positive bleed into its neighbours. The ReLU and bias clip that noise. Non-adjacent pairs have $\cos 144^\circ = -0.81$, a negative bleed the ReLU removes. If features rarely co-occur, the noise rarely lands on a feature that is also active.

High-dimensional space helps further. Random vectors in $m$ dimensions are nearly orthogonal, and the number of almost-orthogonal directions grows exponentially in $m$.

From theory to a tool: sparse autoencoders

If activations are a sparse combination of many more directions than neurons, then recover those directions. An SAE learns an overcomplete dictionary:

$$f = \mathrm{ReLU}\big(W_{enc}(x - b_{dec}) + b_{enc}\big),\qquad \hat{x} = W_{dec} f + b_{dec}$$

$$\mathcal{L} = \lVert x - \hat{x} \rVert_2^2 + \lambda \lVert f \rVert_1$$

The L1 term pushes most entries of $f$ to zero. Each column of $W_{dec}$ is a feature direction; $f_i$ says how strongly feature $i$ fires on this token.

Sizes: for GPT-2 small's residual stream, $d = 768$. An 8x dictionary has $768 \times 8 = 6144$ features, and the encoder plus decoder hold $2 \times 768 \times 6144 \approx 9.4$M parameters. Anthropic's Scaling Monosemanticity trained SAEs with 1M, 4M and 34M features on a production model and found interpretable features such as the Golden Gate Bridge and code vulnerabilities, some of which could be turned up or down to change model behaviour.

How to judge an SAE

  • L0: average number of active features per token. Tens, not hundreds.
  • Reconstruction error, as fraction of variance unexplained.
  • Loss recovered: splice the reconstruction back into the model and measure the language-modelling loss. The honest test: it asks whether the SAE preserves what the model uses, not only what it stores.
  • Dead features (never fire) and feature splitting (a bigger dictionary splits "Python" into "Python, loops", "Python, imports"...).
  • Human readability: look at the top-activating contexts for features, and also at random activations, not only the best ones.

Code

import torch
import torch.nn as nn
import torch.nn.functional as F


class SAE(nn.Module):
    def __init__(self, d_model: int, expansion: int = 8):
        super().__init__()
        d_hidden = d_model * expansion
        self.W_enc = nn.Parameter(torch.randn(d_model, d_hidden) * 0.02)
        self.b_enc = nn.Parameter(torch.zeros(d_hidden))
        self.W_dec = nn.Parameter(torch.randn(d_hidden, d_model) * 0.02)
        self.b_dec = nn.Parameter(torch.zeros(d_model))

    def forward(self, x):
        f = F.relu((x - self.b_dec) @ self.W_enc + self.b_enc)
        x_hat = f @ self.W_dec + self.b_dec
        return x_hat, f

    @torch.no_grad()
    def normalise_decoder(self):
        self.W_dec /= self.W_dec.norm(dim=1, keepdim=True)


def step(sae, x, opt, l1=5e-3):
    x_hat, f = sae(x)
    mse = F.mse_loss(x_hat, x)
    loss = mse + l1 * f.abs().sum(-1).mean()
    opt.zero_grad()
    loss.backward()
    opt.step()
    sae.normalise_decoder()
    return mse.item(), (f > 0).float().sum(-1).mean().item()

Keep decoder rows at unit norm, otherwise the model can cheat the L1 penalty by shrinking f and growing the decoder. The function returns reconstruction error and L0 so you can watch both.

Caveats worth holding on to

  • SAE features are a lens, not ground truth. A feature that looks clean can still be a bundle, and reconstruction is never perfect.
  • L1 shrinks activations, which biases what you see. Newer variants (TopK, JumpReLU) change the penalty to reduce this.
  • Interpretable does not mean causal. To claim a feature matters, intervene: clamp it and watch what the model does.

Common mistakes

  • Training on too few tokens, then reading dead and duplicated features as insight.
  • Reporting only reconstruction error and not loss recovered.
  • Looking only at top-activating examples, which flatters every feature.
  • Not normalising activations before training, so the L1 coefficient means different things at different layers.

Check yourself

  • Why do sparse features make superposition possible, while dense ones do not?
  • With 3 features in 2 dimensions at equal spacing, what is the cosine between neighbours?
  • What does the L1 term trade away?
  • Why is "loss recovered" a stronger test than reconstruction error?