Evals you can trust

"It seems better" is the most expensive sentence in agent engineering. An eval turns it into a number you can defend. But a bad eval is worse than none, because it gives you confidence for free. This note is about building evals you can trust.

Vocabulary that keeps you honest

  • A task is one input plus what success means.
  • A trial is one run of the agent on a task. Agents are stochastic, so you run several.
  • A transcript (or trace) is everything that happened: messages, tool calls, results.
  • The outcome is the state of the world at the end. A transcript may say "booked" while the database says otherwise. Grade the outcome whenever you can.
  • A grader scores a trial. Three kinds: code (cheap, exact, brittle), model (flexible, needs calibration), human (the reference you calibrate against).
  • Capability evals ask "can it do this at all?" and start low. Regression evals ask "does it still do everything it used to?" and should sit near 100 percent. A capability task that your agent now passes reliably graduates to the regression suite.

Start with error analysis, not metrics

Do not begin with a list of metrics like "helpfulness" and "tone". Begin by reading 50 to 100 real traces, writing a one-line note about each failure, and grouping the notes. The groups are your failure taxonomy: wrong tool, bad arguments, gave up early, hallucinated an id, ignored a constraint. Each group becomes a test you write, and the biggest groups tell you what to fix first. This is Hamel Husain's central advice and it is right: the evals worth having come from your data.

Prefer code graders

Check things you can check deterministically: did it call the expected tool, avoid the forbidden one, stay under 8 steps and 0.25 in cost, return valid JSON, put the order id in the answer? These graders are free, fast and never drift. Reserve a model judge for what code cannot see (is this explanation correct and complete?).

Trusting an LLM judge

A judge is a classifier, so measure it like one. Label 30 to 50 outputs yourself as pass or fail, run the judge on the same outputs, and compute two rates:

  • TPR: of the truly good outputs, the share the judge passed.
  • TNR: of the truly bad outputs, the share the judge failed.

Judges lean lenient and have known biases (they favor longer answers and the first option). If you only report the judge's raw pass rate, the bias goes straight into your number. With the observed pass rate $p_{\text{obs}}$, you can correct it:

$$\theta = \frac{p_{\text{obs}} + \text{TNR} - 1}{\text{TPR} + \text{TNR} - 1}$$

Worked example: TPR $= 0.90$, TNR $= 0.50$, observed pass rate $0.80$. Then $\theta = (0.80 + 0.50 - 1)/(0.90 + 0.50 - 1) = 0.30/0.40 = 0.75$. The judge says 80 percent; the truth is about 75. Check the forward direction: $0.75 \cdot 0.90 + 0.25 \cdot 0.50 = 0.80$. Also ask for a reason before the verdict, give the judge a binary rubric, and never grade an output with the same model and prompt that wrote it.

Small samples lie

With $n$ cases and pass rate $\hat{p}$, a 95 percent interval is roughly

$$\hat{p} \pm 1.96\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$$

For $\hat{p} = 0.80$ and $n = 40$: $\sqrt{0.16/40} = 0.063$, so $\pm 0.124$. Your 40-case suite cannot tell 0.80 from 0.72. To gate changes, compare the old and new version on the same cases (a paired comparison), run several trials per case, and treat anything inside the noise as "no evidence".

Reliability needs its own metric. If one trial passes with probability $p$, then $k$ independent trials all pass with probability $p^k$. This is the pass-hat-$k$ idea from the tau-bench paper. At $p = 0.9$ and $k = 5$ it is $0.59$: an agent that is right 90 percent of the time fails at least once in five tries four times in ten. If users run the same flow repeatedly, that is the number to track.

In code

import json
import math


def load_cases(path):
    with open(path) as f:
        return [json.loads(line) for line in f]


def grade(case, result):
    called = set(result["tools_called"])
    return {
        "used_expected_tool": case["expected_tool"] in called,
        "no_forbidden_tool": not called & set(case.get("forbidden", [])),
        "answer_contains": all(s.lower() in result["answer"].lower() for s in case["must_include"]),
        "under_budget": result["cost_usd"] <= case.get("max_usd", 0.25),
        "under_steps": result["steps"] <= case.get("max_steps", 8),
    }


def summarize(rows):
    n = len(rows)
    p = sum(all(r.values()) for r in rows) / n
    half = 1.96 * math.sqrt(p * (1 - p) / n)
    return {"n": n, "pass_rate": round(p, 3), "ci95": (round(p - half, 3), round(p + half, 3))}


def corrected_rate(p_obs, tpr, tnr):
    return (p_obs + tnr - 1) / (tpr + tnr - 1)


print(corrected_rate(0.80, 0.90, 0.50))

The last line prints 0.75. Wire summarize into CI and fail the build when the new pass rate is below the stored baseline by more than the interval, or when any regression-suite case newly fails.

Observability is the other half

An eval tells you what failed. A trace tells you why. Record, for every run: the full message list, each tool call with arguments, result size and latency, token usage and cost per step, the prompt version and model id. OpenTelemetry's generative AI conventions give you shared names for these fields, and tools like Langfuse or Inspect can store and replay them. Your first eval cases should come from the traces of runs that went wrong.

Common mistakes

  • Starting from generic metric names instead of reading traces.
  • Grading the transcript when you could grade the outcome.
  • One judge prompt, never compared against human labels.
  • A 20-case suite and a decision based on a two-point difference.
  • Training on the test set: iterating the prompt against the same cases you report. Keep a held-out set you rarely touch.
  • Letting the suite rot. When production finds a new failure, it becomes a case that day.

Check yourself

  • Your judge has TPR 0.95 and TNR 0.70, and passes 0.85 of outputs. Estimate the true rate.
  • Why does a regression suite want near 100 percent while a capability suite does not?
  • At $p = 0.95$ per trial, what is the chance of five passes in a row?
  • What would you log to debug an agent that "sometimes" picks the wrong tool?