Backprop is the chain rule with bookkeeping

A neural network is a big arithmetic expression. Backpropagation is the chain rule applied to that expression in reverse order, with each intermediate value cached so nothing is computed twice. That is the whole algorithm. The rest is bookkeeping, and the bookkeeping is where the bugs live.

One rule, applied everywhere

Every node in the graph knows exactly one thing: how its output changes when its own inputs change. If $y = f(x)$ and the loss $L$ depends on $y$, then

$$\frac{\partial L}{\partial x} = \frac{\partial L}{\partial y}\cdot\frac{\partial y}{\partial x}$$

The first factor arrives from downstream. The second is the node's local derivative. Multiply and pass the result upstream. A node never needs to know what the rest of the network looks like.

A worked example with real numbers

One neuron: $n = x_1 w_1 + x_2 w_2 + b$ and $o = \tanh(n)$. Take $x_1 = 2$, $x_2 = 0$, $w_1 = -3$, $w_2 = 1$, $b = 6.8814$. Forward: $n = 0.8814$ and $o = 0.7071$.

Backward, starting from $\partial o / \partial o = 1$:

  • $\partial o/\partial n = 1 - o^2 = 0.5$. This is the only place tanh matters.
  • Addition just copies the gradient, so $\partial o/\partial b = 0.5$ and each product term also receives $0.5$.
  • A product swaps its inputs: $\partial o/\partial w_1 = x_1 \cdot 0.5 = 1.0$ and $\partial o/\partial x_1 = w_1 \cdot 0.5 = -1.5$.
  • $\partial o/\partial w_2 = x_2 \cdot 0.5 = 0$ because $x_2 = 0$, and $\partial o/\partial x_2 = w_2 \cdot 0.5 = 0.5$.

Read the result: nudging $w_1$ up by a small amount raises the output by that amount, and $w_2$ has no influence at all for this input. A central-difference check, $(f(\theta + \epsilon) - f(\theta - \epsilon)) / 2\epsilon$ with $\epsilon = 10^{-6}$, reproduces every number to about six digits.

Why the gradient accumulates

Let $b = a \cdot a$. Node $a$ feeds the multiplication twice, through two edges. Each edge contributes its own term and they add: $\partial b/\partial a = a + a = 2a$. If backward assigned instead of added, the second edge would overwrite the first and you would get $a$. This is the multivariable chain rule: the gradient at a node is the sum over every path from that node to the loss. In code it is always +=.

The same fact explains why you call zero_grad() before each step in PyTorch. Gradients accumulate by design, so nothing resets them for you.

Why reverse, not forward

Forward-mode differentiation pushes one input's derivative through the graph, so it costs one pass per parameter. Reverse mode pushes the loss's derivative back once and gets every parameter's gradient. A network has millions of parameters and one scalar loss, so reverse mode costs about two to three times a forward pass. The price is memory: the forward pass must keep its intermediate values until the backward pass uses them.

The core in 35 lines

import math


class Value:
    def __init__(self, data, parents=()):
        self.data = data
        self.grad = 0.0
        self._parents = parents
        self._backward = lambda: None

    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, (self, other))

        def _backward():
            self.grad += out.grad
            other.grad += out.grad

        out._backward = _backward
        return out

    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, (self, other))

        def _backward():
            self.grad += other.data * out.grad
            other.grad += self.data * out.grad

        out._backward = _backward
        return out

    def tanh(self):
        t = math.tanh(self.data)
        out = Value(t, (self,))

        def _backward():
            self.grad += (1 - t * t) * out.grad

        out._backward = _backward
        return out

    def backward(self):
        order, seen = [], set()

        def visit(v):
            if v not in seen:
                seen.add(v)
                for p in v._parents:
                    visit(p)
                order.append(v)

        visit(self)
        self.grad = 1.0
        for v in reversed(order):
            v._backward()

Run the neuron above through it and you should print $x_1$ grad $-1.5$, $w_1$ grad $1.0$, $w_2$ grad $0$ and $b$ grad $0.5$. The topological order is not decoration: a node's _backward may only run after every node downstream of it has finished adding into its .grad.

The matrix version

For a linear layer $Y = XW$ with $X \in \mathbb{R}^{B \times n}$ and $W \in \mathbb{R}^{n \times m}$, the same rule reads

$$\frac{\partial L}{\partial W} = X^{\top},\frac{\partial L}{\partial Y}, \qquad \frac{\partial L}{\partial X} = \frac{\partial L}{\partial Y},W^{\top}$$

You never need to remember these. The shapes force them: $\partial L/\partial W$ must be $n \times m$, and the only product of $X^{\top}$ ($n \times B$) with $\partial L/\partial Y$ ($B \times m$) that fits is the one above.

Common mistakes

  • Assigning instead of accumulating. Gives the right answer on a chain and the wrong one the first time a value is used twice.
  • Skipping the gradient check. Compare every new op to a central difference with $\epsilon \approx 10^{-6}$ in float64. Float32 is too noisy for this.
  • In-place edits. Overwriting a tensor the backward pass still needs silently corrupts gradients. PyTorch raises an error for many of these; hand-written engines do not.
  • Forgetting to reset. Gradients from the previous step leak into this one.

Check yourself

  1. Why is the gradient of $b = a \cdot a$ equal to $2a$, and what does that say about the node's two outgoing edges?
  2. Backprop through $o = \tanh(n)$ multiplies by $1 - o^2$. What happens to learning when $|n|$ is large?
  3. For $Y = XW$, derive the shape of $\partial L / \partial X$ without looking at the formula.
  4. Why is reverse mode cheap for a loss and expensive for a Jacobian with many outputs?