The same customer was in the training set and the test set at once
A recommendation model validated at 0.94 accuracy. Deployed, it ran at 0.71. The cause was that each customer appeared in the data as several rows.
Split those rows at random and a customer lands in training and in test at the same time. The model has already memorised that person's taste during training, and validation just asks it to recognise them again. In production it meets customers it has never seen.
| How the split was made | Can one customer land on both sides? | Does the validation score predict production? |
|---|---|---|
| Shuffle the rows and cut | yes | no — it said 0.94, production gave 0.71 |
| Keep each customer entirely on one side | no | yes |
The only thing that changed is where the line was drawn. Same model, same features, same 70/30.
Splitting data is not a mechanical 70/30 step. It is a claim about the situation the model will actually face — and when the claim is wrong, the validation score stops predicting anything at all.
The key question
Does this split imitate the situation the model will meet in production?
Sampling and splitting are the same problem
The same ideas appear twice: once when drawing data from a population, and again when dividing the data you have into training and evaluation. The ways bias creeps in are identical.
Simple random
every row equally likelyThe default, and fine when the rows are exchangeable.
Stratified
same share from each stratumKeeps the class or segment balance exact.
Cluster
whole groups at a timeCheaper to collect; the effective n is smaller.
Systematic
every k-th rowSimple, and dangerous if the data has a cycle.
How systematic sampling fails
Take every 24th row of hourly sales and you have sampled the same hour every day. The mean you get is that hour's mean and says nothing about the day. If you do not know whether the data has a period, do not sample systematically.
Train, validation and test exist because they do different jobs. With only two, you end up reporting performance measured on the data you tuned against, and that number is optimistic.
| Set | Job | How often you look |
|---|---|---|
| Training | The model fits its parameters | Constantly |
| Validation | You choose hyperparameters and models | Every experiment |
| Test | One final verdict on performance | Exactly once |
A test set gets used up a little every time you look at it. The moment you adjust a model in response to its score, it is no longer unseen data. After twenty peeks it has effectively become a second validation set.
When random is right, and when it is not
A random split is correct only when the rows are exchangeable — when it genuinely does not matter which rows land where. Real data usually fails that condition.
| If the data has | Split by | What random splitting breaks |
|---|---|---|
| Nothing special | Random | Nothing — random is correct here |
| A rare class | Stratified | The test set can end up with almost no positives |
| Several rows per person | Group | The same person appears on both sides |
| A time order | Time | The model trains on the future |
| Both groups and time | Time, then group | Both of the above at once |
| Nested structure (schools, stores) | The outer level | Within-group similarity inflates the score |
Data where one entity generates many rows is everywhere: several orders per customer, several tests per patient, several days per store. Split at random and the same entity appears on both sides.
Same data, two splits
Random split: validation 0.94, production 0.71
Split by customer: validation 0.73, production 0.71
The second predicted production almost exactly. The 0.94 measured "how well it re-identifies customers it has already seen", a situation production never provides.
If the model has to predict for new entities, the split must be by entity. If it predicts the next action of an existing customer, having that customer on both sides is fine — and then the split should be by time instead.
Build a model to predict the future, split at random, and the model sees the future while predicting the past. Validation is then optimistic without exception.
With both structures, time comes first
When the data has customers and a time order, cut on time first and preserve entity boundaries inside that if you can. Violating the time order is the more fundamental error.
Match the structure, then size the split
"70/30" is a habit from when datasets ran to a few thousand rows. It is better to think in absolute counts: how many rows does the evaluation set need before it can be trusted?
| Rows available | Suggested split | Note |
|---|---|---|
| < 1,000 | Cross-validation, no fixed test set | A held-out set this small is too noisy to trust |
| 1,000 – 10,000 | 60 / 20 / 20 | The textbook case |
| 10,000 – 100,000 | 70 / 15 / 15 | The validation set is already large enough |
| > 1,000,000 | 98 / 1 / 1 | 1% is 10,000 rows — plenty |
| Rare class, few positives | Size the split by positives, not rows | Aim for at least a few hundred positives in test |
The last row catches people most often. On data with a 3% churn rate, a 1,000-row test set contains 30 positives. At that point the uncertainty on the estimate swamps any conclusion. Size the split by positives, not by rows.
Below about a thousand rows, carving off a fixed test set costs more than it buys. You lose data, and the noise from a single split exceeds the differences you are trying to detect.
| Scheme | Use when | Watch out for |
|---|---|---|
| k-fold | The default, exchangeable rows | Wrong for time series and grouped data |
| Stratified k-fold | Classification, especially imbalanced | Stratify on the target, not on a feature |
| Group k-fold | Repeated measurements per entity | Fold sizes become uneven |
| Time series split | Anything with a time order | Folds grow, so early folds are small |
| Nested CV | Tuning and evaluating at once | Expensive — k × k fits |
| Leave-one-out | Very small samples | High variance and n fits |
The thing to get right with cross-validation is not reporting only the mean. If the spread across folds is wide, the model's performance is not "0.78" but "0.78 ± 0.09", and the second is the honest statement.
Getting the split right does not help if preprocessing happens outside it. The rule that has come up repeatedly in earlier lessons collects here in one place.
| Step | Outside the fold | Correct place |
|---|---|---|
| Missing-value imputation | The held-out fold's mean enters training | Inside |
| Scaling | Same problem | Inside |
| Target encoding | The held-out fold's labels leak in | Inside (out-of-fold) |
| Feature selection | Features chosen by looking at the held-out fold | Inside |
| Outlier thresholds | The held-out fold helps set the cut-off | Inside |
| De-duplication, type casting | No problem | Anywhere |
If changing the seed changes the conclusion
That is not a seed problem, it is a signal that you do not have enough data. Rather than picking a seed you like, switch to cross-validation and report the spread.
The 0.94 that a random split invented
A store-level demand forecast. The data has store structure (many rows per store) and time structure (two years of daily records). Only the split changed between runs.
| Split | Validation | Production | Gap | Read |
|---|---|---|---|---|
| Random | 0.91 | 0.68 | 0.23 | Both store and future leak |
| By store | 0.79 | 0.68 | 0.11 | The future still leaks |
| By time | 0.72 | 0.69 | 0.03 | Nearly accurate |
| By time, then store | 0.70 | 0.69 | 0.01 | The honest one |
The lowest validation score belongs to the best configuration. Production performance is essentially identical in all four; only the validation number moves, from 0.70 to 0.91, purely as a function of how the data was cut.
Which means changing the split never improves the model. What it changes is how accurately you know what you have.
The lesson here
Changing the split to raise a validation score is swapping the thermometer to bring down a fever. The goal is not a high number but a number that matches production.
A defect model on data with a 1.2% defect rate — 120 positives in 10,000 rows. Twenty per cent was held out at random.
Positives landing in the test set
Expected: 24
Observed across seeds: 14 to 36
Measure recall on a test set holding 14 positives and every single miss moves it by 7 points. If models A and B differ by 3 points, that comparison carries no information at all.
Stratifying pins the count at 24 — and 24 is still not many. The real fix is a larger evaluation share, or cross-validation so that every positive gets evaluated once.
Misconception 1
❌ A random split is the fairest one.
Only when the rows are independent of one another. With repeated customers, repeated stores or a time order, random splitting produces the most optimistic score available.
Misconception 2
❌ The split with the best validation score is the best split.
The reverse. A good split is the one whose score lands closest to production, and that is usually the lowest number on the list.
Misconception 3
❌ 80/20 is a reasonable default.
The criterion is absolute count, not proportion. At a million rows 1% is plenty; at 500 rows 20% is not enough. With a rare class, count positives rather than rows.
Misconception 4
❌ You can look at the test set more than once.
Every look spends a little of it. Once you start adjusting the model in response, it has become a second validation set and the final number is optimistic again.
Misconception 5
❌ Run several seeds and keep the good one.
That is not choosing a split, it is choosing a result. If the conclusion moves with the seed, switch to cross-validation and report the variation.
The same 200 rows, split four ways
Two hundred rows in time order, four rows per customer, 6% positive class. Watch what each split preserves and what it breaks.
200 rows in time order, four rows per customer, 6% positive class. Filled cells are the test set; a ring marks a positive.
Random — every row independently
Positive rate train / test
9.3 / 2.0%
Customers split across sides
35
Test rows before the train cut-off
50
This shuffle left 1 positive in the test set — 2.0% against 9.3% in training. It also splits 35 customers across both sides and puts 50 test rows before the end of training. Three problems in one split.
A split is a claim about how the model will be used. Make it match.
Learning points
Each split solves one problem and leaves the others standing.
The time split removes temporal leakage at the cost of an unstable class ratio.
Where the data has several structures, the splits have to be combined.
Match the use
The split should imitate deployment
Three sets, three jobs
Train, tune, and judge once
Cross-validate small data
One split is one noisy estimate
The question is not "what proportion should I hold out?"
It is "does this split imitate production?" When the split and the real use diverge, the validation score predicts nothing.
The split is decided before any preprocessing, and getting it wrong makes every number after it wrong too. In SKARI you can check the following alongside.
Splits
Random, Stratified, Group, Time Series Split
Cross-validation
k-fold, Stratified k-fold, Nested CV
Sample design
Sample Size, Power Analysis, Stratification
Leakage checks
Group Leakage Check, Temporal Order Check
Once this clicks, you can answer questions like these.