Voting and Stacking

If, for the same house, linear regression predicts $410,000, a random forest predicts $430,000 and gradient boosting predicts $425,000, which value should you trust? Instead of picking one model, can you use the judgments of several models together?

Methods that combine the results of several predictive models to produce a final prediction are called voting and stacking. Voting averages the models’ predictions, or takes a weighted average, while stacking trains a separate model that takes the base models’ predictions as its input.

Random forests and gradient boosting also use many learners, but what sets voting and stacking apart is that they can combine different kinds of models that have already been built. Still, building an ensemble does not by itself guarantee better performance than the individual models.

They are useful for problems that predict continuous values, such as home prices, sales, demand and customer behavior. In this lesson we use a home sale price example to cover how voting and stacking work, how predictions are combined, preventing data leakage, comparing performance, how they differ from other analyses, and how to report the results.

In regression problems, voting produces the final prediction by taking the average, or a weighted average, of the numbers predicted by several models. Stacking, by contrast, trains a meta-model that uses the predictions made by each base model as new explanatory variables. To distinguish it from majority voting in classification, regression voting is usually described as averaging.

Linear regression expresses linear relationships between variables fairly clearly but can miss complex nonlinearity. A random forest reduces variability by averaging many trees, and gradient boosting corrects the errors of earlier models one after another. When different errors are mixed together, an ensemble may reduce the error, but if the models are wrong in the same direction, the benefit is limited.

① Regression voting: average the predictions

If the models predict 410, 430 and 425 ($ thousands), the simple average is about 421.7 ($ thousands). You can also use a weighted average that gives more weight to better models, but the weights must be set during training without looking at the evaluation data.

② Stacking: learn how to combine

In stacking, the predictions of the three base models are used as inputs to a meta-model. The meta-model can learn in which situations to trust which model’s predictions more.

③ Don’t reselect models during the final evaluation

Decide on the base models, the combining rule and the hyperparameters within the training data, and then check performance just once at the end on independent evaluation data. Repeatedly looking at the evaluation score and making adjustments can make the results overly optimistic.

We prepared 1,000 hypothetical home listings. The sale price is predicted from floor area, number of rooms, year built, neighborhood, distance to a transit station, and so on. 800 listings are used for training and cross-validation, and 200 are set aside for the final evaluation.

The values in the results table are hypothetical numbers designed to explain the method. They should not be interpreted as performance obtained by actually training and evaluating models.

① Split off the training data and the final evaluation data

Model development is carried out only on the 800 listings, excluding the 200 held out for the final evaluation. If the actual sale dates need to be taken into account, a time-based split may be more appropriate than a random split.

② Train the three base models

Compare linear regression, a random forest and gradient boosting on the same training data, and choose their settings through cross-validation.

③ Set up the voting model

Take a simple average of the base models’ predictions, or combine them with weights chosen from cross-validation results on the training data.

④ Train the stacking meta-model

Collect the predictions (OOF) for each training row from base models trained on the folds that exclude that row. Training the meta-model on these predictions ensures it does not simply use the overly good predictions of models that were trained on those same rows.

⑤ Compare performance on the same evaluation data

Apply the five models that have completed the selection process to the 200 listings in the final evaluation data and compare their prediction errors.

A model with a large weight is not necessarily the better model. The correlation of errors between models also affects the combined result. With real data, it is best to check the simple average first and then compare more complex methods.

In the hypothetical results, stacking has the lowest MAE of the five models, at $26,000. But it differs from voting’s $27,000 by only $1,000. Further review is needed to tell whether this difference is due to the particular evaluation sample, and whether it matters enough in practice to justify the added model complexity.

MAE is the average of the absolute differences between the predicted prices and the actual sale prices. An MAE of $26,000 means the average absolute error in the evaluation data was about that much. RMSE gives more weight to large errors, so it helps you see whether the price was badly off for some houses.

Are the base models’ errors too similar?

If the three models are wrong in similar ways for the same houses, averaging them may improve things only a little. Check the correlations between their predictions and residuals.

Is there data leakage in the stacking?

If the meta-model is trained on the base models’ direct predictions on their own training data, overfitting is likely. Use OOF predictions, and fit all preprocessing within the training folds of the cross-validation.

Has performance improved enough to justify the complexity?

If the improvement is small, you might choose a simpler model or voting, taking into account training and operating costs, explainability and inference latency.

Does it generalize across neighborhoods and time periods?

If listings from the same nearby development or sales from adjacent time periods end up in both datasets, performance can be rated higher than it would be for genuinely new listings.

In the report, state the sample size, the input variables, how the data were split, how missing values and categorical variables were handled, the settings of each model, the voting weights, the fold setup and meta-model for stacking, the evaluation metrics, and the units of the home prices.

Stacking is not the answer in every field. What matters is confirming the improvement through independent evaluation and judging whether the complexity is manageable in operation.

In the home sale price example, linear regression, a random forest and gradient boosting predicted different prices for the same listing. Voting averaged the three values to produce a final price, and stacking additionally learned a combining rule from the base models’ predictions. In the hypothetical results stacking had the lowest MAE, but we cannot assume that the performance differences between the individual models and the ensembles would be reproduced with real data.

For stacking in particular, a design that prevents data leakage while training the meta-model is essential. Rather than feeding the meta-model direct predictions on the training rows, use cross-validated OOF predictions. The final evaluation data should not be used repeatedly to choose models or tune weights. If you consider maintenance costs and explainability as well as predictive accuracy, then when the improvement is small, simple voting or a single model that works well may be more appropriate.