A model doing better than you expected is not good news
Most analysis mistakes look like mistakes. Code stops, a number comes out impossible, a chart goes wrong. Eventually somebody notices.
Leakage is not like that. When a pipeline leaks, the model fits better. Validation accuracy rises, AUC rises, the slide deck improves. Nobody questions it. The problem surfaces after deployment.
| When this goes wrong | Does the code stop? | What happens to the validation score | When you find out |
|---|---|---|---|
| An ordinary mistake — a typo, a bad join | it stops | it gets worse | immediately |
| Data leakage | it does not stop | it gets better | after deployment |
The danger in the bottom row lives in the third column. A mistake that raises your score never gives anyone a reason to go looking for it.
Leakage is what you have when information unavailable at prediction time gets into the model. In the training table that information sits there quite legitimately, so nothing complains. At the moment a prediction is actually needed, the cell is empty.
The rules that kept recurring in earlier lessons — fit on training rows only, do it inside the fold, split on time — were all defences against this one problem. Here they come together.
The key question
Do I actually have this information at the moment I have to predict?
Six routes in
Target leakage
the answer is a columnA field that only exists once the outcome has happened.
Train–test contamination
the split is porousThe same rows, or their statistics, appear on both sides.
Temporal leakage
training on the futureRows from after the prediction point are in training.
| Kind | Where it enters | How to catch it |
|---|---|---|
| Target leakage | A feature that encodes the outcome | Ask whether the value exists at prediction time |
| Train–test contamination | Preprocessing fitted on everything | Put every fitted step inside a pipeline |
| Temporal leakage | A random split on ordered data | Split on time and check the maximum dates |
| Group leakage | One entity on both sides | Split by the entity key |
| Duplicate leakage | The same row in both sets | De-duplicate before splitting |
| Selection leakage | Features or thresholds chosen on all data | Do the choosing inside each fold |
The first three are the main families and the last three are variations. What they share is that the evaluation data contributed, somehow, to the model it is grading.
Once a feature carries the outcome, the model looks at nothing else. And it almost always arrives as a by-product of the outcome — something that looks like a cause and is in fact a consequence.
| Predicting | Feature | Exists at prediction time? | Fix |
|---|---|---|---|
| Churn next month | Cancellation reason | ✗ | Drop it entirely |
| Churn next month | Support tickets last 30 days | ✗ | Re-anchor to the prediction date |
| Loan default | Recovery amount | ✗ | Drop it entirely |
| Loan default | Credit score at application | ✓ | Keep — but freeze the as-of date |
| Order value | Sales tax | ✗ | It is the target times a constant |
| Machine failure | Maintenance ticket opened | ✗ | The ticket follows the failure |
| Machine failure | Vibration in the previous hour | ✓ | Keep |
The second row is the subtle one. "Support tickets in the last 30 days" is a perfectly reasonable feature. The problem is that "last" is anchored to today. In the training table those 30 days are the 30 days before cancellation, and customers who have decided to leave file a burst of tickets on the way out.
Anchor every window to the prediction date
Time aggregates have to be defined as "the 30 days before the date the prediction is made". Anywhere a spec says "last 30 days", find out what it is anchored to — it is usually wrong.
It shows up as a good result, not an error
Normally something going wrong makes performance worse, which is why worse performance triggers investigation. Leakage exploits exactly that habit.
| An ordinary bug | Data leakage | |
|---|---|---|
| Execution | Crashes or warns | Completes cleanly |
| Validation score | Falls | Rises |
| First reaction | "Something is broken" | "Better than I expected" |
| Found | During development | After deployment |
| Cost to fix | A few hours | Rebuild or abandon the model |
| Credibility | Untouched | Damaged — the number was already reported |
Which leaves exactly one workable rule: treat "better than expected" as a warning. Before celebrating, go and find the feature that already knows the answer.
A validation score on its own cannot tell you anything. 0.88 could be an honest 0.88 or a leaked one. The verdict comes from the distance between validation and production.
When you cannot wait for production
Before deployment there is no production figure. Instead, hold out the most recent period by time and evaluate on that; the gap will expose much of the leakage. If a random split and a temporal split disagree substantially, that disagreement is the size of your temporal leak.
This creates an organisational problem. Finding and removing a leak reduces the reported performance. If 0.92 has already been presented, revising it to 0.78 is politically hard.
So leakage audits have to happen before results are shared. A number that has been announced is difficult to withdraw, and while it stands, decisions get made on it.
One question and one pipeline stop most of it
Open the feature list and go down it one at a time. "At the moment I need a prediction, does this value exist?" That single question catches most target leakage.
Preprocessing leakage is not prevented by being careful; it is prevented structurally. Put every fitted step into a pipeline object and cross-validation refits it per fold without being asked.
| Step | In the pipeline | Because |
|---|---|---|
| Imputation | Required | It learns a mean or median |
| Scaling | Required | It learns μ and σ |
| Target encoding | Required | It learns per-category target means |
| Feature selection | Required | It learns which features survive |
| Outlier thresholds | Required | It learns quantiles |
| Dimension reduction (PCA) | Required | It learns component directions |
| Type casting, de-duplication | Not needed | Nothing is learned |
| Check | What a bad answer looks like | What to do |
|---|---|---|
| Validation minus production | More than a few points apart | Assume leakage until shown otherwise |
| Single-feature performance | One feature alone scores near the full model | Check its timing before anything else |
| Correlation with the target | Above 0.95 | It probably contains the target |
| Score by fold | One fold far above the rest | Look for duplicates or a group straddling it |
| Maximum date per split | Training extends past the test start | Re-split on time |
| Perfect scores | AUC of 1.00, accuracy of 100% | This is never a modelling success |
Row two is the most efficient in practice. Train a model on each feature alone and record the score: an implausibly strong single feature stands out immediately. Anything approaching the full model on its own is nearly always a leak.
SMOTE goes after the split
On imbalanced data, oversampling must be applied after splitting and to the training rows only. Applied first, an original row and the synthetic points generated from it end up on opposite sides, and the validation score inflates sharply.
Fixing the leaks made the score go down
A churn model validated at 0.94 AUC and ran at 0.76 in production. Here is the record of tracking down that 0.18.
| Change | Validation AUC | Production AUC | Gap |
|---|---|---|---|
| Starting point | 0.940 | 0.760 | 0.180 |
| Dropped cancel_reason | 0.860 | 0.761 | 0.099 |
| Re-anchored the ticket window | 0.812 | 0.762 | 0.050 |
| Re-split by customer | 0.789 | 0.762 | 0.027 |
| Moved scaling into the pipeline | 0.775 | 0.762 | 0.013 |
| Re-split on time | 0.766 | 0.763 | 0.003 |
Validation fell from 0.940 to 0.766 at every step. Production went from 0.760 to 0.763 — essentially unchanged. Over six changes, the model got no better.
What changed is what was known about the model. The team started out believing it had a 0.94 model and ended up knowing it had a 0.766 one. The second state is far more useful than the first.
The lesson here
Removing leakage is a measurement fix, not a performance fix. Unless that distinction is explained in advance, whoever lowers the number looks like the person who broke it.
An equipment failure model returned a validation AUC of 1.00 — perfect separation. It looks like a triumph, and there is no version of this that is legitimate.
Tracking it down
Single-feature scores put maintenance_ticket_opened at AUC 0.998 on its own
That ticket is raised by the maintenance team after the failure occurs.
So it was not a feature predicting failures but a feature recording them. With it removed the AUC came to 0.81, which is the real number.
An AUC of 1.00 or 100% accuracy is not a sign of a well-built model but a sign that the answer is in the inputs. There is effectively no exception.
Misconception 1
❌ Leakage is a beginner's mistake.
It is a structural problem, not a carelessness problem. Without a pipeline, an experienced practitioner gets the ordering wrong too. It is found in winning competition solutions on a regular basis.
Misconception 2
❌ As long as the target itself is not a feature, it is fine.
Leakage usually arrives as a by-product of the target. Cancellation reasons, recovery amounts and maintenance tickets are none of them the target, and none of them exist without it.
Misconception 3
❌ A high validation score means a good model.
A validation score alone decides nothing. The gap between validation and production is the real metric, and where the gap is wide, the high score is itself the evidence of a problem.
Misconception 4
❌ Preprocessing does not change the data, so order does not matter.
Scaling, imputation and encoding all learn numbers from data. Fitted before the split, the test set has helped set the standard it is then judged against.
Misconception 5
❌ We audited it once.
Every new feature and every new join opens another route. A leakage audit is something you run every time, immediately before sharing a result.
Switch leaks on and compare the two bars
Same model, same data. Turn on each of the six leaks and watch which bar moves. The dashed line is the true performance.
Switch leaks on one at a time. Watch which bar moves.
Leaks active
0
Validation − production
0.000
Verdict
trustworthy
With no leaks the two bars sit together at 0.762. That is what a clean pipeline looks like: the validation number is a prediction of the production number.
Leakage never announces itself as an error. It announces itself as a good result.
Learning points
Leakage only raises the validation bar — production stays put or drifts down.
A small leak opens a gap of around 0.03, which is easy to miss.
Which is why the test is not the height of the bars but the distance between them.
It looks like success
Leakage never raises an error
One question finds most of it
Does this value exist at prediction time?
The gap is the test
Compare validation against production
The question is not "what did it score?"
It is "will that score survive deployment?" Leakage arrives disguised as a good result, so unless you go looking for it, it is never found.
A leakage audit is where every rule in this section converges. In SKARI you can check the following alongside.
Leakage checks
Leakage Check, Single-Feature AUC, Timing Audit
Split validation
Group Leakage Check, Temporal Order Check
Pipelines
Pipeline, CV Fold Isolation, Nested CV
Monitoring
Drift Detection, Production vs Validation
Once this clicks, you can answer questions like these.