# The KV cache and why inference is memory-bound

By @omar · Published Nov 2, 2026 · From the roadmap [Large Language Models, End to End](https://ornix.io/u/omar/large-language-models-end-to-end)

Training a transformer is limited by compute. Generating from one is limited by memory. If you remember one idea from the inference week, make it this one: **at batch size 1, every generated token has to read all of the model's weights from memory, and the arithmetic hardly matters.**

## Why a cache exists

Autoregressive generation produces one token at a time. To produce token $t+1$ you need attention over keys and values for tokens $1..t$. Those do not change when a new token arrives, because attention is causal. So store them.

Without a cache, step $t$ re-runs the whole prefix. Generating 512 tokens means $1 + 2 + \dots + 512 = 131{,}328$ token-forwards instead of 512, a 256x difference in compute.

```mermaid
%% ornix:id=kvca0005
sequenceDiagram
    participant U as Prompt
    participant M as Model
    participant C as KV cache
    U->>M: Prefill: all prompt tokens at once
    M->>C: Store K and V for every layer
    M-->>U: First new token
    loop each new token
        M->>C: Read all cached K and V
        M->>M: Attend with ONE new query
        M->>C: Append the new K and V
        M-->>U: Next token
    end
```

Two phases follow. **Prefill** processes the prompt in parallel: big matrix products, compute-bound, the GPU is busy. **Decode** processes one token per step: tiny matrix-vector products, memory-bound, the GPU mostly waits for bytes.

## How big is the cache

Per token, per layer, you store one key and one value vector for each KV head:

$$\text{bytes per token} = 2 \times n_{layers} \times n_{kv\_heads} \times d_{head} \times \text{bytes per element}$$

For Llama 2 7B (32 layers, 32 heads of size 128, fp16): $2 \times 32 \times 32 \times 128 \times 2 = 524{,}288$ bytes, half a MiB per token. A 4,096-token sequence needs 2 GiB. Batch 32 and the cache alone is 64 GiB, more than the 13.5 GB of weights.

**Grouped-query attention (GQA)** shares each key/value head across several query heads. Llama 2 70B uses 8 KV heads with 80 layers: $2 \times 80 \times 8 \times 128 \times 2 = 327{,}680$ bytes per token, 320 KiB. With full multi-head attention at 64 heads it would be 2.5 MiB per token, eight times more.

## Why decode is memory-bound

Take a 7B model in fp16: about 13.5 GB of weights. A100 80GB has roughly 2.0 TB/s of memory bandwidth. At batch 1:

- Time to read the weights once: $13.5 / 2000 \approx 6.7$ ms. That caps you near 150 tokens per second.
- Compute per token: about $2N = 13.5$ GFLOPs. At 312 TFLOP/s that is 0.04 ms.

Reading takes about 150 times longer than computing. The arithmetic intensity is about 1 FLOP per byte, while the A100's ridge point is $312 \times 10^{12} / 2.0 \times 10^{12} \approx 150$ FLOPs per byte. The chip is idle more than 99% of the time.

This explains the whole bag of inference tricks:

- **Batching** serves many users per weight read. Intensity grows with batch size, until the KV cache fills memory.
- **Quantization** shrinks the bytes: int4 weights are about 3.4 GB, so the same read is up to 4x faster.
- **GQA/MQA** shrinks the cache.
- **Paged attention** (vLLM) stops fragmentation from wasting cache memory.
- **Speculative decoding** turns several sequential steps into one parallel verification.

## Speculative decoding, with numbers

A small draft model proposes $\gamma$ tokens; the large model scores all of them in one forward pass (one weight read) and keeps the longest prefix it agrees with. If each proposal is accepted independently with probability $a$, the expected number of tokens per large-model pass is

$$\frac{1 - a^{\gamma + 1}}{1 - a}$$

With $a = 0.8$ and $\gamma = 4$ this is $(1 - 0.8^5) / 0.2 = 3.36$ tokens per pass instead of 1. The output distribution is exactly the large model's: rejected tokens are resampled with a correction, so quality is unchanged.

## Code

```python
import math
import torch
import torch.nn.functional as F


class CachedAttention(torch.nn.Module):
    def __init__(self, d_model, n_head):
        super().__init__()
        self.n_head, self.dh = n_head, d_model // n_head
        self.qkv = torch.nn.Linear(d_model, 3 * d_model)
        self.proj = torch.nn.Linear(d_model, d_model)

    def forward(self, x, cache=None):
        B, T, C = x.shape
        q, k, v = self.qkv(x).split(C, dim=2)
        q, k, v = (t.view(B, T, self.n_head, self.dh).transpose(1, 2) for t in (q, k, v))
        if cache is not None:
            k = torch.cat([cache[0], k], dim=2)
            v = torch.cat([cache[1], v], dim=2)
        scores = q @ k.transpose(-2, -1) / math.sqrt(self.dh)
        if T > 1:
            S = k.size(2)
            keep = torch.ones(T, S, dtype=torch.bool, device=x.device).tril(S - T)
            scores = scores.masked_fill(~keep, float("-inf"))
        out = (F.softmax(scores, dim=-1) @ v).transpose(1, 2).reshape(B, T, C)
        return self.proj(out), (k, v)
```

During decode `T == 1`, so no mask is needed: the single query may see every cached key. The `tril(S - T)` handles prefill with a non-empty cache.

## Common mistakes

- Position bugs: with absolute or rotary positions, the new token's position is the cache length, not 0.
- Caching after the output projection instead of keys and values.
- Letting the cache grow without a limit, then running out of memory at long context.
- Comparing speed at batch 1 and claiming it transfers to a serving system with batching.
- Testing only the first generated token. Compare logits for 200 steps.

## Check yourself

- Compute the KV bytes per token for a model with 24 layers, 8 KV heads, head size 64 in bf16.
- Why does prefill saturate the GPU while decode does not?
- Why does int4 quantization speed up decode more than it speeds up prefill?
- Speculative decoding with $a = 0.5$, $\gamma = 4$: how many tokens per pass? Is it worth it?

---

Roadmap: https://ornix.io/u/omar/large-language-models-end-to-end (Markdown: https://ornix.io/u/omar/large-language-models-end-to-end.md)

Canonical page: https://ornix.io/u/omar/large-language-models-end-to-end/kv-cache-memory-bound-inference
