# Why scaling laws bend (Chinchilla)

By @omar · Published Nov 2, 2026 · From the roadmap [Large Language Models, End to End](https://ornix.io/u/omar/large-language-models-end-to-end)

Scaling laws are the most useful empirical result in the field, and the most often misquoted. The useful version is not "bigger is better". It is a budget rule: **given a fixed amount of compute, how should you split it between model size and training data?**

## Compute, in one line

For a transformer with $N$ parameters trained on $D$ tokens, training FLOPs are about

$$C \approx 6ND$$

Each token costs $2N$ FLOPs on the forward pass (one multiply and one add per parameter) and about $4N$ on the backward pass, which computes gradients with respect to both activations and weights. For GPT-2 small ($N \approx 124$M) on $D = 10$B tokens: $6 \times 1.24\times10^{8} \times 10^{10} = 7.4\times10^{18}$ FLOPs. A GPU sustaining 40% of 312 TFLOP/s bf16 does that in roughly 16.6 hours.

## The two results

**Kaplan et al. (2020)** found that loss falls as a smooth power law in model size, data and compute, each limited by the other two. Their recommendation was to spend extra compute mostly on model size: $N_{opt} \propto C^{0.73}$. Models of that era were large and undertrained.

**Hoffmann et al. (2022, "Chinchilla")** retrained over 400 models and fitted

$$L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}$$

The paper's reported constants for this fit are $E = 1.69$, $A = 406.4$, $B = 410.7$, $\alpha = 0.34$, $\beta = 0.28$. The three terms mean: $E$ is the irreducible loss (the entropy of text itself), $A/N^{\alpha}$ is the penalty for a model that is too small, $B/D^{\beta}$ is the penalty for too little data.

Minimising $L$ subject to $6ND = C$ gives $N_{opt}$ and $D_{opt}$ that both grow roughly as $C^{0.5}$. Their headline: for compute-optimal training, **parameters and tokens should scale together, about 20 tokens per parameter.** The authors say Kaplan's setup, which used one learning-rate schedule for all runs, penalised models trained on fewer tokens and so pushed the answer towards bigger models.

```mermaid
%% ornix:id=scal0004
flowchart LR
    C["Compute budget C"] --> R["C = 6 N D"]
    R --> N["Choose N"]
    R --> D["Choose D = C / (6N)"]
    N --> L["Loss L(N, D) = E + A/N^a + B/D^b"]
    D --> L
    L --> O{"Minimise over N"}
    O --> P["Compute-optimal N and D"]
    O --> Q["Inference-aware: smaller N, much larger D"]
```

## A worked example

Gopher had 280B parameters trained on 300B tokens. Chinchilla had 70B parameters trained on 1.4T tokens, with roughly the same compute budget ($6ND$ gives $5.0\times10^{23}$ for Gopher and $5.9\times10^{23}$ for Chinchilla). Plugging the constants in:

- Chinchilla: $A/N^{\alpha} = 0.083$, $B/D^{\beta} = 0.163$, so $L = 1.69 + 0.083 + 0.163 = 1.937$.
- Gopher: the data term dominates, giving $L \approx 1.993$.

Similar compute, a predicted gap of about 0.056 nats in favour of the smaller, longer-trained model. The paper reports Chinchilla beating Gopher on downstream tasks too.

## Why the laws bend

A power law is a straight line on log-log axes. In practice the line bends, for several honest reasons.

- **The floor.** $E$ is a constant. As $N$ and $D$ grow, the reducible part shrinks towards zero and the curve flattens against $E$.
- **Data runs out.** The law assumes fresh tokens. Muennighoff et al. found that repeating data for up to about four epochs costs little, and returns diminish quickly after that.
- **Training is not the whole bill.** If a model will serve billions of requests, inference cost dominates. A smaller model trained far past the compute-optimal point is cheaper to serve. Llama 3 8B saw about 15T tokens, roughly 1,900 tokens per parameter, almost 100 times the Chinchilla ratio.
- **The fit is fragile.** An independent replication attempt (Besiroglu et al., 2024) re-derived the parametric fit from the paper's data and found the published constants hard to reconcile with the paper's own other estimates. Use scaling laws for planning ranges, not for decimals.

There is a quieter inconsistency worth computing yourself: with the constants above, $D_{opt}/N_{opt}$ is not constant. It drifts upward slowly with compute, because $\alpha \ne \beta$. The "20 to 1" rule is a rounded summary.

## Code

```python
import numpy as np

E, A, B, alpha, beta = 1.69, 406.4, 410.7, 0.34, 0.28


def loss(N, D):
    return E + A / N**alpha + B / D**beta


def compute_optimal(C):
    Ns = np.logspace(6, 12, 6000)
    Ds = C / (6 * Ns)
    L = loss(Ns, Ds)
    i = L.argmin()
    return Ns[i], Ds[i], L[i]


for C in (1e19, 1e21, 1e23):
    N, D, L = compute_optimal(C)
    print(f"C={C:.0e}  N={N:.2e}  D={D:.2e}  D/N={D / N:.0f}  L={L:.3f}")
```

Run it, then replace the constants with ones you fitted in the week-5 build and see how far your own optimum moves.

## What to take away

- Plan runs with $C = 6ND$ and a target tokens-per-parameter ratio, not with a parameter count alone.
- Compute-optimal is not the same as cost-optimal. Decide whether you are minimising training cost or total cost of ownership.
- Fit your own small-scale law before a big run. The shape of the curve transfers better than the constants.

## Common mistakes

- Counting embedding parameters in $N$ for small models, where they are a large share, then comparing with a paper that did not.
- Fitting a power law without a floor, then extrapolating to a loss below the entropy of the data.
- Comparing runs with different learning-rate schedules and attributing the difference to scale.
- Treating 20 tokens per parameter as a law rather than an estimate with wide error.

## Check yourself

- Derive $6ND$ from the forward and backward costs.
- If you double compute, by how much should you scale $N$ and $D$ under Chinchilla's result?
- Why would a company train a model at 1,000 tokens per parameter?
- What does a gap between a power-law fit and the data at large $N$ usually indicate?

---

Roadmap: https://ornix.io/u/omar/large-language-models-end-to-end (Markdown: https://ornix.io/u/omar/large-language-models-end-to-end.md)

Canonical page: https://ornix.io/u/omar/large-language-models-end-to-end/why-scaling-laws-bend
