Reading a confusion matrix and choosing a threshold
A classifier outputs a score. A decision needs a yes or a no. The step in between, the threshold, is where a model's accuracy turns into real costs and benefits, and the default of 0.5 is almost never the right choice. This note builds up a confusion matrix, shows why accuracy can mislead, and then derives the threshold from costs.
The four boxes
Every prediction at a given threshold lands in one of four boxes:
- True positive (TP): predicted yes, truly yes.
- False positive (FP): predicted yes, truly no. A false alarm.
- False negative (FN): predicted no, truly yes. A miss.
- True negative (TN): predicted no, truly no.
The metrics are just ratios of these:
$$\text{precision} = \frac{TP}{TP + FP} \qquad \text{recall} = \frac{TP}{TP + FN} \qquad \text{accuracy} = \frac{TP + TN}{n}$$
Precision answers "when it says yes, how often is it right?" Recall answers "of the real yeses, how many did it find?"
Why accuracy lies on imbalanced data
Suppose 1,000 patients and 50 have the disease. A model produces TP = 40, FN = 10, FP = 76, TN = 874 (an illustrative example).
- Accuracy is $(40 + 874)/1000 = 91.4%$.
- A model that always says "healthy" scores $950/1000 = 95%$ accuracy and finds nobody.
- Recall is $40/50 = 0.80$ and precision is $40/116 = 0.345$. Two of every three alarms are false.
Is that model good? Accuracy cannot tell you. It depends on what a miss costs against a false alarm.
Deriving the threshold from costs
Let a false positive cost $c_{FP}$ and a false negative cost $c_{FN}$. Assume the model's score $p$ is a calibrated probability: among cases scored 0.3, about 30 percent are positive. For one case, the expected cost of each action is
- Predict positive: wrong with probability $(1 - p)$, so expected cost $(1-p),c_{FP}$.
- Predict negative: wrong with probability $p$, so expected cost $p,c_{FN}$.
Predict positive when $(1 - p)c_{FP} < p, c_{FN}$, which rearranges to
$$p > \frac{c_{FP}}{c_{FP} + c_{FN}}$$
If a miss is ten times worse than a false alarm, the threshold is $1/11 \approx 0.091$, not 0.5. You do not tune it by feel. You write down the costs and compute it.
Checking it with a simulation
To see this work, simulate 200,000 cases with a perfectly calibrated model: 8.3 percent are positive, and the score is the true probability. Costs: a miss is 10, a false alarm is 1.
import numpy as np
rng = np.random.default_rng(0)
n = 200_000
x = rng.normal(size=n)
p = 1 / (1 + np.exp(-(-3.2 + 1.5 * x)))
y = rng.random(n) < p
c_fn, c_fp = 10, 1
for t in [0.5, 0.2, 0.0909, 0.05]:
pred = p >= t
tp = (pred & y).sum(); fp = (pred & ~y).sum()
fn = (~pred & y).sum(); tn = (~pred & ~y).sum()
cost = (c_fn * fn + c_fp * fp) / n
print(t, round(tp / (tp + fp), 3), round(tp / (tp + fn), 3),
round((tp + tn) / n, 3), round(cost, 3))
Output (columns: threshold, precision, recall, accuracy, cost per case):
- 0.5: precision 0.620, recall 0.124, accuracy 0.921, cost 0.730
- 0.2: precision 0.350, recall 0.479, accuracy 0.883, cost 0.504
- 0.0909: precision 0.223, recall 0.745, accuracy 0.765, cost 0.425
- 0.05: precision 0.166, recall 0.869, accuracy 0.628, cost 0.470
Always predicting negative has accuracy 0.917 and cost 0.826. Look at what happened. The default threshold of 0.5 has the highest accuracy and the worst cost of the four, finding only 12 percent of the positives. The cost-optimal threshold, 0.09, gives up a lot of accuracy (0.765) and has the lowest cost. A sweep over thresholds put the minimum cost at about 0.086, as the formula predicts. Moving past it, to 0.05, adds more false alarms than it saves misses, and cost goes back up.
Ranking versus deciding
ROC-AUC measures how well the scores rank positives above negatives, over all thresholds at once. It says nothing about which threshold to use and it is not changed by calibration. For this simulation AUC is 0.835. When positives are rare, a precision-recall curve is more informative than the ROC curve, because ROC's false positive rate divides by the huge number of negatives and hides many false alarms.
Common mistakes
- Choosing the threshold on the test set. Choose it on validation data or by cross-validation, then freeze it before the final test.
- Applying the cost formula to uncalibrated scores. A random forest's "0.3" is not a 30 percent probability. Calibrate first with isotonic regression or Platt scaling.
- Optimizing F1 by habit. F1 weights misses and false alarms equally. If your costs are not equal, F1 optimizes the wrong thing.
- Ignoring prevalence shift. If the share of positives in production differs from training, precision changes even though the model did not.
Check yourself
- Compute the F1 score for the 1,000-patient example above.
- A miss costs 5 and a false alarm costs 20. What is the optimal threshold for a calibrated model?
- Why can accuracy go down while expected cost goes down?
- Why does ROC-AUC not depend on the threshold, and why can that be a problem?