Cross-Validation

Split your data once into train and test, and the score you get depends on which rows happened to land in the test set. Get an easy split and the model looks great; get a hard one and it looks weak. Neither number is trustworthy on its own.

How K-Fold Works

Split the data into k equal folds. Train on k−1 of them and validate on the one held out. Rotate so every fold serves as validation exactly once, then average the k scores. Laid out, the rotation looks like this:

Fold 1   [ VALID ][ train ][ train ][ train ][ train ]
Fold 2   [ train ][ VALID ][ train ][ train ][ train ]
Fold 3   [ train ][ train ][ VALID ][ train ][ train ]
Fold 4   [ train ][ train ][ train ][ VALID ][ train ]
Fold 5   [ train ][ train ][ train ][ train ][ VALID ]

• k = 5 or 10 is the common choice
• Every row is used for both training and validation, across different folds
• The spread of scores tells you how stable the model is

Why the Average Beats a Single Split

A worked example makes the point immediately. Run 5-fold cross-validation and you get five scores, one per fold:

A single split might have handed you the 91% fold and left you thinking the model is excellent — or the 84% fold and left you thinking it's weak. The averaged 88.2% is the number you can actually defend, and the spread (84% to 91%) tells you how much the model wobbles.

Variants Worth Knowing

• Stratified k-fold: keeps class balance in each fold — essential for imbalanced data
• Group k-fold: keeps related rows (same user, same site) together
• Time-series split: trains on the past, validates on the future — never shuffle time
• Leave-One-Out (LOOCV): k equals the number of rows — thorough but expensive, used mainly on small datasets

The Leakage Trap

Cross-validation only tells the truth if every preprocessing step happens inside each fold. Scale or impute using the whole dataset first, and information from the validation fold leaks into training — inflating every score.

Cross-Validation in the SKARI Model Lab

The Model Lab validates models with proper cross-validation built in — so the accuracy you see is the averaged, leakage-free estimate, not a single optimistic split.

This is what makes SKARI's Auto Compare trustworthy: every model is scored under the same cross-validation, so the comparison is apples-to-apples.

• k-fold validation applied consistently across every model
• Preprocessing fit within folds to avoid leakage
• Score spread shown so you see stability, not just the mean

Frequently Asked Questions

How many folds?

5 or 10 balances stability and cost. More folds give a smoother estimate but take longer to run.

Do I still need a test set?

Yes — hold out a final test set for the very end. Cross-validation is for model selection; the test set is the last, untouched check.

What about time-series data?

Never shuffle it. Use a forward-chaining split so you always train on the past and validate on the future.

How does cross-validation relate to Grid Search?

Cross-validation is how you evaluate a model; Grid Search compares many hyperparameter combinations by scoring each one with cross-validation, and picks the best setting. In other words, Grid Search runs cross-validation many times over.

What is LOOCV?

Leave-One-Out Cross-Validation sets k to the number of rows — each row is validated once against a model trained on all the others. It's exhaustive but slow, so it's mostly used on small datasets.

Conclusion

Cross-validation is how you turn a shaky single score into a number you can defend. Rotate the folds, keep preprocessing inside them, and your model comparison rests on solid ground.