# Data leakage: five ways to fool yourself

By @omar · Published Nov 2, 2026 · From the roadmap [Machine Learning from First Principles](https://ornix.io/u/omar/machine-learning-first-principles)

Data leakage is when information that will not exist at prediction time sneaks into training or evaluation. It is the most common reason a model that scored 95 percent in a notebook scores 60 in production. It never throws an error. It only makes your numbers look better than the truth, which is exactly why nobody goes looking for it. Here are five ways to fool yourself, each with the habit that prevents it.

```mermaid
%% ornix:id=mlk1d007
flowchart LR
    subgraph Wrong["Leaky: preprocess first"]
      W1["All data"] --> W2["Scale or select features"] --> W3["Split"] --> W4["Train and score"]
    end
    subgraph Right["Honest: split first"]
      R1["All data"] --> R2["Split"] --> R3["Fit scaler or selector on train only"] --> R4["Apply to validation, then score"]
    end
```

## 1. Fitting preprocessing on all the data

Scaling, imputing, one-hot encoding, PCA and feature selection all learn something from the data. If they learn it from rows that later land in the validation fold, the validation fold has already influenced the model. Feature selection is the worst offender, because it uses the labels.

Here is a demonstration that should make you uncomfortable. Make 100 rows with 5,000 features of pure random noise and random labels. No model can do better than 50 percent. Select the 20 features most correlated with the label, then cross-validate:

```python
import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(0)
X = rng.normal(size=(100, 5000))
y = rng.integers(0, 2, 100)
cv = StratifiedKFold(5, shuffle=True, random_state=0)

X_leaky = SelectKBest(f_classif, k=20).fit_transform(X, y)
print("leaky:", cross_val_score(LogisticRegression(), X_leaky, y, cv=cv).mean())

honest = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression())
print("honest:", cross_val_score(honest, X, y, cv=cv).mean())
```

In my run (seed 0) the leaky version reported about 0.87 accuracy and the honest pipeline reported 0.50. The leaky score is not a bug in cross-validation. The selector looked at every label before the folds were cut, so the "held-out" fold had already voted on which features to keep. The fix is structural: everything that learns from data lives inside a `Pipeline`, and cross-validation refits the whole pipeline on each training fold.

## 2. Related rows on both sides of the split

If one patient, user, device or product appears in many rows, a random split puts some of their rows in training and some in validation. The model learns to recognize the person and gets rewarded for it. Near-duplicate images, augmented copies and repeated sensor readings do the same thing. Split by the entity, not by the row: `GroupKFold` with the patient or user id as the group.

## 3. A feature that is secretly the answer

Predicting whether a customer will churn, and one column is "cancellation_reason". Predicting loan default, and one column is "days_overdue". Predicting a hospital readmission with a column that is filled in only after readmission. These columns can have huge feature importance and perfect validation scores. Ask of every feature: would I know this value at the moment I make the prediction? If the answer is "only afterwards", delete it. A single feature that dominates the importance ranking is a smell, not a gift.

## 4. Using the future to predict the past

On time-ordered data a random split lets the model train on tomorrow and be tested on yesterday. Seasonal patterns, trends and one-off events all leak. Rolling features can leak as well: a 7-day average that includes the target day contains the target. Validate forward in time: train on the past, test on the next block, using `TimeSeriesSplit`, and leave a gap if your features look ahead.

## 5. Peeking at the test set until it agrees

You train, look at the test score, tweak something, look again. After twenty rounds the test set has trained your choices, and the score you report is optimistic by an amount you cannot measure. The same thing happens with cross-validation: the best of 200 hyperparameter settings has an inflated CV score, because it was selected as the maximum of noisy estimates. The habits that prevent it:

- Touch the test set once, at the very end, and report whatever it says.
- Use nested cross-validation if you must report a tuned model's score from CV alone.
- Keep a written log of every experiment you ran, including the ones you dropped.

## A short audit you can run on any project

- Is every transformation inside a pipeline that is fit on training folds only?
- Does any entity appear in more than one fold?
- For each feature, can I name the moment it becomes known?
- Is the split by time if the data has an order?
- How many times have I looked at the test set?
- Is the score surprisingly good? Then assume a leak until you have shown otherwise.

## Common mistakes

- **Fixing the leak by making the model weaker.** The score drops because the leak is gone, not because the model got worse.
- **Checking leakage only for the target.** Leaks also come from statistics computed on the whole dataset: means, vocabularies, category encodings, target encoding.
- **Trusting a pipeline you built by hand.** If `fit_transform` is called on validation data anywhere, it is a leak.

## Check yourself

- Why does selecting features before cross-validation inflate the score even when the features are noise?
- When should you use `GroupKFold` and when `TimeSeriesSplit`?
- A feature has 90 percent of the importance in a tree ensemble. What is your first question?
- Why is the best CV score from a 200-setting grid search biased upward?

---

Roadmap: https://ornix.io/u/omar/machine-learning-first-principles (Markdown: https://ornix.io/u/omar/machine-learning-first-principles.md)

Canonical page: https://ornix.io/u/omar/machine-learning-first-principles/data-leakage-five-ways-to-fool-yourself
