# Why cross-entropy and not squared error

By @omar · Published Nov 2, 2026 · From the roadmap [Machine Learning from First Principles](https://ornix.io/u/omar/machine-learning-first-principles)

The first time you see logistic regression, the loss looks like it fell out of the sky: $-[y\log p + (1-y)\log(1-p)]$. Why not just square the error between the probability and the label, as you did in regression? Because the squared version quietly stops learning exactly when your model is most wrong. The cross-entropy loss is what maximum likelihood gives you, and its gradient is the cleanest in machine learning.

## Where the loss comes from

A logistic model says $p = P(y = 1 \mid x) = \sigma(z)$ with $z = w^{\top}x + b$ and $\sigma(z) = 1/(1 + e^{-z})$. A label is a coin flip with that bias, so the probability of one observed label is

$$P(y \mid x) = p^{\,y}(1-p)^{1-y}$$

For independent examples, multiply. Maximizing the product is the same as minimizing the negative log of it:

$$\ell = -\frac{1}{n}\sum_{i=1}^{n}\Big[y_i \log p_i + (1 - y_i)\log(1 - p_i)\Big]$$

That is cross-entropy. It is not a design choice, it is the Bernoulli negative log-likelihood. In the language of information theory, $H(q, p) = H(q) + \mathrm{KL}(q \,\Vert\, p)$: with the true labels $q$ fixed, minimizing cross-entropy minimizes the KL divergence between the data and your model.

```mermaid
%% ornix:id=mlk1d002
flowchart LR
    X["Features x"] --> Z["Score z = w.x + b"]
    Z --> P["Probability p = sigmoid of z"]
    P --> L["Loss: cross-entropy vs label y"]
    L --> G["Gradient wrt z is p - y"]
    G --> W["Update w, b by gradient descent"]
```

## The gradient is almost embarrassingly simple

Use $\sigma'(z) = \sigma(z)(1 - \sigma(z)) = p(1-p)$. For one example with $y = 1$, the loss is $-\log p$, so

$$\frac{\partial \ell}{\partial z} = -\frac{1}{p}\cdot p(1-p) = -(1 - p) = p - y$$

The same result holds when $y = 0$. So for the whole dataset

$$\nabla_w \ell = \frac{1}{n} X^{\top}(p - y)$$

Compare with linear regression, $\frac{1}{n}X^{\top}(\hat y - y)$. Same shape: features times the residual. The sigmoid and the log cancel each other exactly.

## What goes wrong with squared error

Try $\ell_{\text{sq}} = (p - y)^2$ instead. The chain rule gives $\partial \ell_{\text{sq}}/\partial z = 2(p - y)\,p(1-p)$, and now there is an extra factor $p(1-p)$ that goes to zero when the sigmoid saturates.

Take a true label $y = 1$ and a confidently wrong score $z = -6$, so $p = \sigma(-6) \approx 0.00247$:

- Cross-entropy loss: $-\ln(0.00247) \approx 6.00$. Gradient wrt $z$: $p - y \approx -0.9975$.
- Squared-error loss: $(1 - 0.00247)^2 \approx 0.995$. Gradient wrt $z$: $2(-0.9975)(0.00247)(0.9975) \approx -0.0049$.

The cross-entropy gradient is about 200 times larger. The model that is certain and wrong gets a loud correction under cross-entropy and a whisper under squared error. A second problem: through a sigmoid, squared error is not convex in $w$, so gradient descent can stall on plateaus. Cross-entropy with a linear score is convex, so every local minimum is global.

```python
import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def log_loss(y, z):
    # stable form of -[y log p + (1-y) log(1-p)] written in terms of the score z
    return np.mean(np.logaddexp(0, z) - y * z)

def grad_w(X, y, w, b):
    p = sigmoid(X @ w + b)
    return X.T @ (p - y) / len(y), np.mean(p - y)

z, y = -6.0, 1.0
p = sigmoid(z)
print("cross-entropy:", float(np.logaddexp(0, z) - y * z), "grad:", p - y)
print("squared error:", (p - y) ** 2, "grad:", 2 * (p - y) * p * (1 - p))
```

The line using `np.logaddexp(0, z)` is $\log(1 + e^z)$ computed without overflow. Writing `np.log(1 - p)` directly returns `-inf` once $p$ rounds to exactly 1.

## Sanity checks

- At initialization with $w = 0$ every $p = 0.5$, so the loss must be $\ln 2 \approx 0.693$. If you see something else, you have a bug.
- A classifier that always predicts the class rate $\bar y$ scores $-[\bar y \ln \bar y + (1-\bar y)\ln(1-\bar y)]$. That is the baseline your model must beat.
- Multiclass works the same way. Softmax gives probabilities, and the gradient on each logit is again $p_k - y_k$.

## Common mistakes

- **Clipping probabilities instead of using the stable form.** Clipping hides the saturation you want to see.
- **Forgetting the average.** With a sum instead of a mean, the right learning rate scales with the dataset size.
- **Labels in $\{-1, +1\}$ with a $\{0, 1\}$ loss.** The two conventions have different formulas. Convert once, at the boundary.
- **Reading log-loss like accuracy.** A model can improve log-loss while its accuracy stays flat, because it is becoming better calibrated.

## Check yourself

- Derive $\partial \ell / \partial z = p - y$ without looking.
- A model predicts $p = 0.9$ for a positive example and $p = 0.9$ for a negative one. What are the two losses?
- Why does the loss at $w = 0$ equal $\ln 2$ whatever the labels are?
- Which of the two losses would you use for a regression target, and why is the answer different?

---

Roadmap: https://ornix.io/u/omar/machine-learning-first-principles (Markdown: https://ornix.io/u/omar/machine-learning-first-principles.md)

Canonical page: https://ornix.io/u/omar/machine-learning-first-principles/why-cross-entropy-not-squared-error
