Why LSTMs gate
Recurrent networks reuse one set of weights at every time step, so the gradient from step $T$ back to step 0 passes through the same transformation $T$ times. Whatever that transformation does to the gradient, it does it $T$ times in a row. LSTMs exist to make that product behave.
What goes wrong in a vanilla RNN
The state update is $h_t = \tanh(W h_{t-1} + U x_t + b)$. Backpropagation through time needs
$$\frac{\partial h_T}{\partial h_0} = \prod_{t=1}^{T} \mathrm{diag}(1 - h_t^2),W$$
Each factor has norm at most $\gamma,|W|$ with $\gamma \le 1$ coming from the tanh slope. If the largest singular value is below 1, the product shrinks exponentially in $T$ and the model cannot learn dependencies more than a few steps apart. If it is above 1 the product can explode, and the standard remedy is gradient clipping. With a spectral radius of $0.9$ and a perfect slope of 1, fifty steps give $0.9^{50} = 0.005$; with $1.1$ they give $117$. There is a narrow stable band, and training has to find it.
The gated cell
An LSTM keeps a separate cell state $c_t$ and controls it with three sigmoid gates, each a learned function of $(x_t, h_{t-1})$:
$$f_t = \sigma(W_f[h_{t-1}, x_t] + b_f), \quad i_t = \sigma(W_i[h_{t-1}, x_t] + b_i), \quad o_t = \sigma(W_o[h_{t-1}, x_t] + b_o)$$
$$g_t = \tanh(W_g[h_{t-1}, x_t] + b_g)$$
$$c_t = f_t \odot c_{t-1} + i_t \odot g_t, \qquad h_t = o_t \odot \tanh(c_t)$$
Read the cell state update in plain words: keep a fraction $f$ of the old memory, and write a fraction $i$ of a new candidate $g$. The output gate decides how much of the memory to show the rest of the network.
Why this fixes the gradient
Look at the path along the cell state only. Since $c_t = f_t \odot c_{t-1} + \dots$,
$$\frac{\partial c_t}{\partial c_{t-1}} = \mathrm{diag}(f_t)$$
(ignoring the indirect routes through $h_{t-1}$). The gradient over $T$ steps along this path is a product of forget gates, a number between 0 and 1 that the network chooses, not a weight matrix norm it has to land on by luck. Hochreiter and Schmidhuber called this path the constant error carousel. If the model learns $f \approx 1$ for a feature, that feature's gradient passes through untouched.
The forgetting is a per-dimension, per-step decision, so the numbers are worth seeing:
| Forget gate value | After 50 steps |
|---|---|
| 0.99 | 0.61 |
| 0.95 | 0.077 |
| 0.731 (this is $\sigma(1)$, the common bias init of 1) | $1.5 \times 10^{-7}$ |
| 0.5 | $8.9 \times 10^{-16}$ |
A forget bias initialized to 1 starts the gates mostly open without being saturated, which is why it is a widely used default.
It is the same idea as a residual connection. Both replace a multiplicative update ($h \leftarrow \tanh(Wh)$) with an additive one ($c \leftarrow f c + \dots$ or $x \leftarrow x + F(x)$) so that a clean path survives depth, or time.
A cell you can check
import torch
import torch.nn as nn
class LSTMCell(nn.Module):
def __init__(self, n_in, n_hid):
super().__init__()
self.W = nn.Linear(n_in + n_hid, 4 * n_hid)
def forward(self, x, state):
h, c = state
i, f, g, o = self.W(torch.cat([x, h], dim=-1)).chunk(4, dim=-1)
c = torch.sigmoid(f) * c + torch.sigmoid(i) * torch.tanh(g)
h = torch.sigmoid(o) * torch.tanh(c)
return h, c
ref = nn.LSTMCell(8, 16)
mine = LSTMCell(8, 16)
with torch.no_grad():
mine.W.weight.copy_(torch.cat([ref.weight_ih, ref.weight_hh], dim=1))
mine.W.bias.copy_(ref.bias_ih + ref.bias_hh)
x, h, c = torch.randn(4, 8), torch.randn(4, 16), torch.randn(4, 16)
h1, c1 = mine(x, (h, c))
h2, c2 = ref(x, (h, c))
print((h1 - h2).abs().max().item(), (c1 - c2).abs().max().item())
PyTorch orders the four gates as input, forget, cell, output, and keeps two bias vectors that always appear as a sum. Both facts are needed to make the comparison come out to about $10^{-7}$.
What gating does not fix
The LSTM still reads the sequence one step at a time and squeezes everything into a fixed-size state. In sequence-to-sequence models the whole source sentence had to fit in the encoder's last $h$. Bahdanau attention removed that bottleneck by letting the decoder look back at every encoder state, and the transformer then dropped the recurrence entirely. Next week starts there.
Common mistakes
- Never clipping gradients. Gating helps with vanishing but the explosion side remains. Clip the global norm.
- Carrying the graph across batches. In truncated BPTT, detach the hidden state between chunks or memory grows without bound.
- Mixing up (seq, batch) and (batch, seq).
nn.LSTMdefaults to sequence-first. - Initializing the forget bias to 0. It works, but long dependencies take much longer to appear.
Check yourself
- Write $\partial h_T / \partial h_0$ for a vanilla RNN and explain what the largest singular value of $W$ controls.
- What is $\partial c_t / \partial c_{t-1}$ and why does it make long-range credit assignment possible?
- A forget gate sits at 0.9 for 100 steps. How much of the original memory is left?
- In what sense is the LSTM cell state update the same idea as a residual connection?