Budgeting tokens and latency

An agent is a loop, and every turn of the loop re-reads everything that came before. That one fact explains most agent bills and most agent slowness. This note gives you the arithmetic so you can predict both before you ship.

Three prices, one clock

A model call has three kinds of tokens. Input tokens are cheap and processed in parallel (the prefill). Output tokens cost several times more per token and are generated one at a time. Cached input tokens are a prefix you have sent recently, billed at a fraction of the normal rate. On the major APIs a cache read costs about 0.1 times the base input price and a five-minute cache write about 1.25 times. Check your provider's pricing page: the exact multipliers differ by model and change over time.

Latency splits the same way:

$$T \approx T_{\text{first token}} + \frac{n_{\text{out}}}{r}$$

Here $n_{\text{out}}$ is the number of output tokens and $r$ is the generation speed in tokens per second. A long prompt mostly shows up in time to first token. A long answer shows up in $n_{\text{out}}/r$, and you pay that serially.

The loop is quadratic in steps

Let the fixed prefix (system prompt plus tool schemas) be $S$ tokens, and let each step add $\Delta$ tokens (the model's message plus the tool result). At step $k$ the prompt holds $S + k\Delta$ tokens, and you pay for all of it again. Over $N$ steps:

$$\text{input tokens} = \sum_{k=1}^{N} (S + k\Delta) = NS + \Delta,\frac{N(N+1)}{2}$$

Worked example with invented round numbers: $S = 3000$, $\Delta = 1500$, $N = 20$, 300 output tokens per step, and prices of 3.00 per million input, 15.00 per million output, 0.30 per million cache read, 3.75 per million cache write.

  • Input tokens: $20 \cdot 3000 + 1500 \cdot 210 = 60{,}000 + 315{,}000 = 375{,}000$.
  • Uncached cost: $375{,}000 \cdot 3.00/10^6 = 1.125$, plus output $6000 \cdot 15/10^6 = 0.09$, so 1.215 in total.
  • With the prefix cached, each step reads everything from the previous step and writes only the new $\Delta$. Reads total 342,000 tokens and writes total 33,000: 0.103 + 0.124 + 0.09 = 0.316.

Same agent, same answers, 3.8 times cheaper, and the input bill is no longer the dominant term. The output cost did not move at all.

Latency for the same run, with 1.0 s to first token and 60 tokens per second: each step is $1.0 + 300/60 = 6$ s, so 20 steps take about 120 s before you count tool time. The two numbers that matter are the step count and the output length per step.

The levers, biggest first

  1. Fewer steps. Compose tools so one call does the job of three. Allow parallel tool calls. Give the agent a clear stop condition. Every step you remove saves a full re-read.
  2. Smaller $\Delta$. Truncate and paginate tool output, return the fields the model needs and not the whole record, and clear stale tool results (week 5).
  3. Cache the prefix. Order the prompt stable first: tools, then system prompt, then history, with anything volatile at the end. A timestamp at the top of the system prompt changes the prefix on every call and silently turns your cache off.
  4. Right-size per step. Routing, extraction and classification steps rarely need your biggest model.
  5. Cap output. Set max_tokens, ask for terse structured output, and do not let the model narrate what the tool already returned.
  6. Enforce a budget. A per-run cap checked inside the loop turns a runaway into a clean failure.

In code

from dataclasses import dataclass


@dataclass
class Prices:
    inp: float = 3.00
    out: float = 15.00
    read: float = 0.30
    write: float = 3.75


def loop_cost(S, d, n, out_per_step, p=Prices(), cached=False):
    if not cached:
        tokens_in = n * S + d * n * (n + 1) // 2
        return (tokens_in * p.inp + n * out_per_step * p.out) / 1e6
    reads = (n - 1) * S + d * n * (n - 1) // 2
    writes = S + n * d
    return (reads * p.read + writes * p.write + n * out_per_step * p.out) / 1e6


class Budget:
    def __init__(self, max_usd, p=Prices()):
        self.max_usd, self.p, self.spent = max_usd, p, 0.0

    def charge(self, usage):
        u = lambda name: getattr(usage, name, 0) or 0
        self.spent += (
            u("input_tokens") * self.p.inp
            + u("output_tokens") * self.p.out
            + u("cache_read_input_tokens") * self.p.read
            + u("cache_creation_input_tokens") * self.p.write
        ) / 1e6
        if self.spent > self.max_usd:
            raise RuntimeError(f"budget exceeded: {self.spent:.2f} > {self.max_usd:.2f}")


print(round(loop_cost(3000, 1500, 20, 300), 3))
print(round(loop_cost(3000, 1500, 20, 300, cached=True), 3))

The two prints give 1.215 and 0.316. Call Budget.charge(response.usage) after every model call. Cache reads and writes are reported in their own usage fields, apart from plain input tokens, so charge them separately or your meter will lie.

Common mistakes

  • Pricing a single call and multiplying by the number of runs, forgetting the loop re-reads its own history.
  • Putting per-request values (dates, user names, request ids) at the top of the prompt and wondering why the cache hit rate is zero.
  • Measuring cost per call instead of cost per task. The number your business cares about is dollars per successful task.
  • Optimizing the prompt for length while tool results are 90 percent of the tokens. Look at a real trace before you cut anything.
  • No max_tokens and no budget, so one confused run spends a day's allowance.

Check yourself

  • Derive the input-token total for $N$ steps with growth $\Delta$ and prefix $S$.
  • Why does caching help more as $N$ grows?
  • Name two things that would break your prompt cache without changing the meaning of the prompt.
  • If output tokens cost five times input tokens, which lever do you pull first for a chatty agent, and which for a long-running one?