Cross-validation

Suppose you built a machine learning model that uses data on 1,000 customers to predict whether they will cancel a service. You split the data into 800 customers for training and 200 for testing, and the accuracy came out at 90%. Based on this result alone, can you expect 90% accuracy for new customers too?

Not necessarily. The result can change depending on which customers end up in the training data and which end up in the test data. If, by chance, many easy-to-predict customers landed in the test data, performance could look better than it really is.

A standard method used to reduce this problem is cross-validation (Cross-Validation).

Cross-validation is a method for evaluating a model's predictive performance by splitting the data into several parts and repeating training and validation. Instead of looking only at the score from a single data split, it combines several validation results to help judge how stably the model will work on new data.

Cross-validation can be applied to many kinds of predictive models, including logistic regression, decision trees, random forests, gradient boosting and neural networks (MLP). It also plays an important role when comparing several algorithms or choosing hyperparameters.

In this article we cover the basic idea of cross-validation, K-fold cross-validation, stratified, group and time series cross-validation, a real-world example, evaluation metrics, data leakage, and how to select models and evaluate final performance.

In typical machine learning, data are split into training data (Training Data) and test data (Test Data). You build the model on the training data and check how well it predicts new cases in the test data.

But the result of a single split can be affected by which observations ended up in the test data. Especially when the data are small or a particular class has few observations, performance can differ considerably from split to split. Cross-validation performs training and validation several times to ease this problem.

For example, if the data are split into five parts, you train on four parts and validate on the remaining one. Then you change the part used for validation and repeat the same process. This way every observation gets a chance to be used for validation, and you can combine predictive performance measured several times.

The core of cross-validation is that instead of evaluating one model once, you repeat the same training procedure on several data configurations to check predictive performance and its variability.

Why is cross-validation needed?

The main reason cross-validation is needed is that predictive performance can be affected by how the data are split.

For example, suppose you built a customer churn prediction model and evaluated it by splitting the data at random. The accuracy might come out at 92% in the first split, 86% in the second and 89% in the third. The algorithm and settings are the same, but the customers included in training and in validation changed.

Judging a model's performance from the score of a single split like this risks picking a result that was good by chance. With cross-validation, you can compare performance across several splits and check that the result does not depend too heavily on one particular split.

Cross-validation is also useful for comparing several algorithms. If you evaluate logistic regression and a random forest on the same cross-validation splits, you can compare the two algorithms' performance under more consistent conditions.

However, cross-validation does not perfectly guarantee performance on future data either. If the data distribution in training differs from that in the real operating environment, or if there is data leakage, cross-validation scores can also differ from real performance.

K-fold splits the whole dataset into K parts and repeats training and validation K times. For example, with K=5 the data are split into five parts, four parts are used for training at a time and the remaining part is used for validation. This is repeated five times in total so that each part serves as validation data once.

When the 5 validations are done, combine the performance metrics calculated in each round. For example, if accuracy in the 5 folds was 85%, 87%, 86%, 89% and 88%, the mean accuracy is 87%. Looking at the differences in accuracy between rounds also shows how much predictive performance changes with the data split.

However, K-fold validation scores are not fully independent measurements, because the training data in different rounds overlap considerably. So the standard deviation of the fold scores should not be interpreted directly as a confidence interval for generalization performance in the population.

How does cross-validation actually proceed?

Suppose you are predicting whether customers will cancel a service using data on 1,000 customers.

Of these, 800 are used for model development and 200 are set aside for final evaluation. Applying 5-fold cross-validation to the 800 development customers puts 160 customers in the validation data of each fold. In each round, the model is trained on 640 customers and performance is calculated on the remaining 160.

In this process, the model is trained from scratch five times in total. The model trained in the first round is not reused as-is for validation in the second round. In each round a new model is built on the designated training data, and the score is calculated on that round's validation data.

When the five validations are done, you can check the mean score and the differences in performance between rounds. Once you have chosen a model and settings, you can retrain the final model on all 800 development customers.

Then you apply the final model to the 200 customers set aside from the start, checking performance on data that were not used in model development.

In practice, 5-fold and 10-fold are widely used. A small K puts relatively more data in each validation, but less data is used for training in each round.

Conversely, a large K lets you use more data in each training run, but the validation data become smaller and the number of times the model is trained increases.

You can generally consider 5-fold or 10-fold first, but the same K is not the best choice for all data. Especially when the data are very small or the model is expensive to train, the choice of K can matter more.

Also, in classification problems with severe class imbalance, what matters more than simply setting K large is whether each validation fold contains enough observations of the class of interest. So K should be decided considering the sample size, class distribution, training time and purpose of the analysis together.

For example, suppose that of 1,000 customers, 100 canceled the service and 900 stayed. The overall churn rate is 10%.

With ordinary random K-fold, some validation folds may contain relatively few churned customers. Especially if churned customers are very rare, a particular fold may have almost no positive cases, or none at all.

Applying stratified 5-fold, you can split so that each fold contains about 20 churned customers and about 180 retained customers.

This reduces the problem of the proportion of the class of interest differing too much in a particular fold. Stratified cross-validation is useful in classification problems with imbalanced class proportions, such as customer churn, credit risk and fraud. However, if the class of interest has fewer than K observations, it is impossible to include that class in every fold.

Also, stratification only balances class proportions; it does not solve differences from the real operating environment or data leakage due to repeated records per customer.

But chance differences can also arise in how the folds are formed. Repeated cross-validation creates new data splits several times and repeats K-fold validation. For example, repeating 5-fold cross-validation 3 times gives 15 rounds of training and validation in total.

Each repetition uses a different random split, so you can check performance on several data configurations.

Repeated cross-validation helps check whether the result from a single fold split is strongly affected by a chance configuration. However, training time increases as the number of repetitions grows.

Also, the results of repeated folds are not independent of one another, so performing 30 validations should not be interpreted as having obtained 30 independent external test sets.

For example, suppose you collected usage records for 500 customers, and each customer has several transaction records.

Transaction records from the same customer may not be independent of one another. With ordinary random K-fold, some of a customer's transaction records may go into the training data while the rest go into the validation data.

In this case, because the model has already learned that customer's characteristics, the results may be more optimistic than predictive performance for new customers. Group K-fold splits the data by customer ID so that records from the same customer are never in training and validation at the same time. This method is useful for data such as the following.

• Repeated transaction records from the same customer
• Multiple test results from the same patient
• Repeated assessment scores from the same student
• Multiple observations from the same company
• Repeated measurements collected from the same piece of equipment

However, whether to use group cross-validation depends on what new situation you want to evaluate. For example, predicting future transactions of customers you already know and predicting completely new customers may require different evaluation designs.

So what matters is not the fact of separating groups itself, but splitting the data in a way that reflects the real prediction environment.

For example, suppose you want to predict next month's sales using monthly sales data from 2023 to 2026. Splitting the data at random could lead to training the model on 2026 data and validating it on 2024 data.

In real forecasting, however, you do not use future data to predict the past; you use past data to predict the future. So cross-validation that preserves time order is needed.

With the expanding window approach, the past data used for training grow over time. With the rolling window (Rolling Window) approach, by contrast, the window moves forward in time while the length of the training period stays fixed.

Which approach is appropriate depends on the characteristics of the data. For example, if old past information is still useful, you can consider the expanding window approach. Conversely, if the market environment changes often and recent data matter more, the rolling window approach may be appropriate.

Also, when the prediction target spans several periods or the information periods of observations overlap, you may need to leave a gap between the training and validation periods to prevent leakage.

For example, with 100 observations, the first round trains on 99 and validates on 1. The second round uses a different single observation for validation, and this process is repeated 100 times in total. LOOCV has the advantage of using almost all observations for training in each round.

But with large data or models that are expensive to train, computation time can increase considerably. Also, LOOCV does not necessarily give a more stable performance estimate than ordinary 5-fold or 10-fold.

In particular, because each round's validation data contain a single observation, classification metrics such as accuracy or recall for individual rounds may be hard to interpret or impossible to calculate. So LOOCV can be considered when the sample is small and training is cheap enough, but it is not the method to use first for every problem.

Of all customers, 650 canceled the service and the remaining 4,350 kept it. The churn rate is 13%. Data on 4,000 customers were used for model development, and the remaining 1,000 were set aside for final evaluation.

Suppose stratified 5-fold cross-validation was applied to the development data to compare the two algorithms. PR-AUC was chosen as the evaluation metric to account for class imbalance.

Cross-validation results

In these results, the mean PR-AUC was 0.41 for logistic regression and about 0.49 for the random forest. Evaluated on the same folds, the random forest had the higher PR-AUC all five times.

So in this example, the random forest can be said to have provided a more useful prediction ranking than logistic regression for distinguishing customers who cancel. But scoring higher in all five folds does not guarantee that it will necessarily perform better in other populations or for future customers.

After choosing the final model, you should check its performance on the test data set aside separately.

For example, suppose two models had the following validation accuracies.

Model A has a mean accuracy of 86%, but its accuracy varied widely across folds, from 75% to 95%. Model B, on the other hand, has a mean accuracy of 88%, and its fold accuracies were relatively steady, between 86% and 90%.

From these results, you can say that model B showed higher mean performance and smaller score variation on these cross-validation splits. But small differences between fold scores do not let you conclude that the model is stable in every environment.

If the folds have similar data distributions, they may not adequately reflect changes that occur in the real operating environment. Also, the standard deviation of cross-validation scores shows the variation in scores between folds; it is not the same as a rigorous confidence interval for the final model's prediction error.

So refer to the mean and standard deviation together, but interpret them in light of the actual data structure and the purpose of the evaluation.

What evaluation metrics are used in cross-validation?

Cross-validation itself does not mean any particular performance metric.

Cross-validation is a way of training and validating models, and which score to calculate in each round is decided by the purpose of the analysis.

Evaluation metrics for classification problems

When the event of interest is rare, as with customer churn or fraud, using accuracy alone may not be appropriate. In such cases it is good to check PR-AUC, precision, recall and so on together.

Evaluation metrics for regression problems

For example, if you are predicting customers' purchase amounts, you can run cross-validation based on RMSE. However, for metrics such as RMSE or R² that involve a nonlinear transformation or a denominator, the average of the fold values and the value calculated by pooling all validation predictions can differ.

So in a report, it is good to make clear whether you averaged the fold scores or calculated the metric by pooling all validation predictions.

For example, suppose you replace missing values in customer data with the mean and then run 5-fold cross-validation. If you calculate the mean of the whole dataset first, information from each validation fold can be reflected in that fold's training process. The same goes for standardization and variable selection.

If you calculate variables' means and standard deviations on the whole dataset, or select important variables using the whole dataset, and then run cross-validation, the validation data do not stay fully independent. So in proper cross-validation, preprocessing parameters must be estimated on the training data only, in each round.

To handle this process consistently, combining preprocessing and the model into a single pipeline (Pipeline) is widely used. Especially when you use imputation of missing values, standardization, one-hot encoding, variable selection, dimensionality reduction, resampling and so on, set things up so that each step is learned inside the cross-validation.

Hyperparameters are settings specified before model training begins. For example, in a random forest the number of trees, maximum depth and minimum number of samples per leaf are hyperparameters. Comparing cross-validation scores while changing these settings shows which combination gives better validation performance.

Example of cross-validation by hyperparameter

In these results, the mean ROC-AUC is highest, at 0.85, with a maximum depth of 8. So you can choose a maximum depth of 8 based on the cross-validation results on the development data. But the highest cross-validation score alone does not guarantee that the actual generalization performance of that setting is exactly 0.85.

This is because, while comparing many settings, a combination that happens to fit the validation data well can be selected by chance. So the cross-validation score used for choosing hyperparameters must be kept separate from the final evaluation of generalization performance.

What is nested cross-validation (Nested Cross-Validation)?

Nested cross-validation is an evaluation method that separates the process of choosing model settings from the process of estimating final performance.

It generally consists of two levels: inner cross-validation (Inner Cross-Validation) and outer cross-validation (Outer Cross-Validation). The inner cross-validation chooses the hyperparameters. The outer cross-validation evaluates how well the model selected in the inner process applies to new data.

For example, consider using an outer 5-fold and an inner 3-fold. First, four parts of the outer data are used for model development and the remaining part is used for evaluation. Within the four development parts, 3-fold cross-validation is run again to choose the most suitable settings.

Next, the outer training data are retrained with the selected settings, and performance is calculated on the outer test fold. This process is repeated for each outer fold. Nested cross-validation reduces the selection bias that can arise when comparing many hyperparameters, helping to estimate the generalization performance of the model selection procedure.

However, computational cost can be high, and scores from the outer test folds also do not guarantee exact performance in the same population. Also, when building a single final model for real use, after evaluating with nested cross-validation you may need to choose the settings again on all development data and train the model.

If you compared several models and settings using cross-validation, the scores obtained in the process are validation performance that was used for model selection. Final test data are useful for checking how well the selected model applies to data not used during development. For example, suppose the following results were obtained.

In these results, the final test score was 0.04 lower than the cross-validation mean. Such a difference can arise from various factors, including chance variation in the data composition, the model selection process, the size of the training data or differences in the actual data distribution.

However, the mere fact that the cross-validation mean is higher than the final test score does not prove that data leakage or overfitting must exist. If cross-validation is properly designed, performance can also be estimated without separate final test data, and in studies with little data especially, nested cross-validation and similar approaches can be used.

Things to watch out for when interpreting cross-validation results

Cross-validation is an important evaluation method in machine learning, but you need to understand some of its limits.

More folds are not always better.

Setting K large gives more training data in each round, but the validation data get smaller and computational cost rises. Especially in classification problems that predict a rare class, check that each validation fold contains enough events of interest.

Do not judge the stability of performance from the mean score alone.

Models with the same mean score can still differ in how much their fold scores vary. So it is good to check the per-round scores, standard deviation, data composition and so on along with the mean score.

Consider the independence of observations.

Data collected repeatedly from the same customer, patient or company may be related. Ignoring this structure and applying ordinary random K-fold can make performance look more optimistic than it really is. Use group-based cross-validation where necessary.

Do not ignore time order.

In problems that predict future outcomes, training on past data and validating on later data may better match the real operating situation. Especially in problems where change over time matters, such as finance, sales and demand forecasting, consider time series cross-validation.

Cross-validation is also affected by data leakage.

If imputation of missing values, standardization, variable selection and so on are applied to the whole dataset first, information from the validation data can be reflected in the training process. So preprocessing must be done within each training fold.

Cross-validation cannot fully anticipate changes in the real operating environment.

Even if the training and validation data are similar, customer behavior, market conditions, product mix and so on can change in real operation. So you need to keep monitoring predictive performance even after the model is put into use.

Using the customer churn example, it could be written as follows.

In a real analysis, it is good to present not only the mean score but also the variation in performance across folds and the results on the final test data. Recording the random seed used for cross-validation, the data splitting scheme, the main preprocessing steps, the criteria for choosing hyperparameters and so on also improves the reproducibility of the analysis.

In particular, if you used group or time series cross-validation, explain clearly how the groups were defined and how the training and validation periods were split.

Where can cross-validation be used?

Cross-validation is not an analysis that applies only to particular algorithms.

It is used in many kinds of machine learning analyses that need to evaluate a model's predictive performance or compare several settings.

Evaluating classification models

You can compare the classification performance of logistic regression, decision trees, random forests, gradient boosting, MLP and so on.

Evaluating regression models

You can compare the prediction errors of linear regression, ridge, lasso, elastic net, random forest regression, MLP regression and so on.

Choosing hyperparameters

You can compare the performance of various settings, such as tree depth, regularization strength, hidden layer structure and learning rate, and choose an appropriate combination.

Variable selection

You can evaluate different combinations of variables and compare which variables give better predictive performance on new data. Here, too, the variable selection process must be carried out inside the cross-validation.

Model comparison and automated machine learning

Cross-validation can also be used in AutoML, which evaluates many algorithms and settings automatically. However, the same cross-validation scheme is not appropriate for every algorithm. Choose the evaluation scheme considering the independence of the data, its time structure, class distribution and so on.

The most standard method is K-fold cross-validation, and depending on the characteristics of the data you can use stratified, repeated, group or time series cross-validation.

Cross-validation lets you compare several validation scores without relying too heavily on the result from a single data split. It also helps you compare the performance of different machine learning algorithms and choose appropriate hyperparameters.

But simply having run cross-validation does not make a model's performance trustworthy. You need to use a splitting method that suits the structure of the data, prevent data leakage during preprocessing, and keep model selection separate from final evaluation.

In particular, be careful not to apply ordinary random K-fold as-is to repeated observations of the same customer or to data with a time order.