# Why LSTMs gate

By @omar · Published Nov 2, 2026 · From the roadmap [Deep Learning from Scratch: backprop to diffusion](https://ornix.io/u/omar/deep-learning-from-scratch)

Recurrent networks reuse one set of weights at every time step, so the gradient from step $T$ back to step 0 passes through the same transformation $T$ times. Whatever that transformation does to the gradient, it does it $T$ times in a row. LSTMs exist to make that product behave.

## What goes wrong in a vanilla RNN

The state update is $h_t = \tanh(W h_{t-1} + U x_t + b)$. Backpropagation through time needs

$$\frac{\partial h_T}{\partial h_0} = \prod_{t=1}^{T} \mathrm{diag}(1 - h_t^2)\,W$$

Each factor has norm at most $\gamma\,\|W\|$ with $\gamma \le 1$ coming from the tanh slope. If the largest singular value is below 1, the product shrinks exponentially in $T$ and the model cannot learn dependencies more than a few steps apart. If it is above 1 the product can explode, and the standard remedy is gradient clipping. With a spectral radius of $0.9$ and a perfect slope of 1, fifty steps give $0.9^{50} = 0.005$; with $1.1$ they give $117$. There is a narrow stable band, and training has to find it.

## The gated cell

An LSTM keeps a separate **cell state** $c_t$ and controls it with three sigmoid gates, each a learned function of $(x_t, h_{t-1})$:

$$f_t = \sigma(W_f[h_{t-1}, x_t] + b_f), \quad i_t = \sigma(W_i[h_{t-1}, x_t] + b_i), \quad o_t = \sigma(W_o[h_{t-1}, x_t] + b_o)$$

$$g_t = \tanh(W_g[h_{t-1}, x_t] + b_g)$$

$$c_t = f_t \odot c_{t-1} + i_t \odot g_t, \qquad h_t = o_t \odot \tanh(c_t)$$

Read the cell state update in plain words: **keep a fraction $f$ of the old memory, and write a fraction $i$ of a new candidate $g$**. The output gate decides how much of the memory to show the rest of the network.

```mermaid
%% ornix:id=lstm0001
flowchart LR
    C0["c(t-1)"] --> MF(("x f"))
    F["forget gate f"] --> MF
    MF --> ADD(("+"))
    I["input gate i"] --> MI(("x i"))
    G["candidate g (tanh)"] --> MI
    MI --> ADD
    ADD --> C1["c(t)"]
    C1 --> TH["tanh"]
    TH --> MO(("x o"))
    O["output gate o"] --> MO
    MO --> H["h(t)"]
```

## Why this fixes the gradient

Look at the path along the cell state only. Since $c_t = f_t \odot c_{t-1} + \dots$,

$$\frac{\partial c_t}{\partial c_{t-1}} = \mathrm{diag}(f_t)$$

(ignoring the indirect routes through $h_{t-1}$). The gradient over $T$ steps along this path is a product of forget gates, a number between 0 and 1 **that the network chooses**, not a weight matrix norm it has to land on by luck. Hochreiter and Schmidhuber called this path the constant error carousel. If the model learns $f \approx 1$ for a feature, that feature's gradient passes through untouched.

The forgetting is a per-dimension, per-step decision, so the numbers are worth seeing:

| Forget gate value | After 50 steps |
|---|---|
| 0.99 | 0.61 |
| 0.95 | 0.077 |
| 0.731 (this is $\sigma(1)$, the common bias init of 1) | $1.5 \times 10^{-7}$ |
| 0.5 | $8.9 \times 10^{-16}$ |

A forget bias initialized to 1 starts the gates mostly open without being saturated, which is why it is a widely used default.

It is the same idea as a residual connection. Both replace a **multiplicative** update ($h \leftarrow \tanh(Wh)$) with an **additive** one ($c \leftarrow f c + \dots$ or $x \leftarrow x + F(x)$) so that a clean path survives depth, or time.

## A cell you can check

```python
import torch
import torch.nn as nn


class LSTMCell(nn.Module):
    def __init__(self, n_in, n_hid):
        super().__init__()
        self.W = nn.Linear(n_in + n_hid, 4 * n_hid)

    def forward(self, x, state):
        h, c = state
        i, f, g, o = self.W(torch.cat([x, h], dim=-1)).chunk(4, dim=-1)
        c = torch.sigmoid(f) * c + torch.sigmoid(i) * torch.tanh(g)
        h = torch.sigmoid(o) * torch.tanh(c)
        return h, c


ref = nn.LSTMCell(8, 16)
mine = LSTMCell(8, 16)
with torch.no_grad():
    mine.W.weight.copy_(torch.cat([ref.weight_ih, ref.weight_hh], dim=1))
    mine.W.bias.copy_(ref.bias_ih + ref.bias_hh)

x, h, c = torch.randn(4, 8), torch.randn(4, 16), torch.randn(4, 16)
h1, c1 = mine(x, (h, c))
h2, c2 = ref(x, (h, c))
print((h1 - h2).abs().max().item(), (c1 - c2).abs().max().item())
```

PyTorch orders the four gates as input, forget, cell, output, and keeps two bias vectors that always appear as a sum. Both facts are needed to make the comparison come out to about $10^{-7}$.

## What gating does not fix

The LSTM still reads the sequence one step at a time and squeezes everything into a fixed-size state. In sequence-to-sequence models the whole source sentence had to fit in the encoder's last $h$. Bahdanau attention removed that bottleneck by letting the decoder look back at every encoder state, and the transformer then dropped the recurrence entirely. Next week starts there.

## Common mistakes

- **Never clipping gradients.** Gating helps with vanishing but the explosion side remains. Clip the global norm.
- **Carrying the graph across batches.** In truncated BPTT, detach the hidden state between chunks or memory grows without bound.
- **Mixing up (seq, batch) and (batch, seq).** `nn.LSTM` defaults to sequence-first.
- **Initializing the forget bias to 0.** It works, but long dependencies take much longer to appear.

## Check yourself

1. Write $\partial h_T / \partial h_0$ for a vanilla RNN and explain what the largest singular value of $W$ controls.
2. What is $\partial c_t / \partial c_{t-1}$ and why does it make long-range credit assignment possible?
3. A forget gate sits at 0.9 for 100 steps. How much of the original memory is left?
4. In what sense is the LSTM cell state update the same idea as a residual connection?

---

Roadmap: https://ornix.io/u/omar/deep-learning-from-scratch (Markdown: https://ornix.io/u/omar/deep-learning-from-scratch.md)

Canonical page: https://ornix.io/u/omar/deep-learning-from-scratch/why-lstms-gate
