Random Forest

Suppose you built a decision tree to predict whether customers will cancel a service. Splitting customers by tenure, contract type and number of customer-service contacts gave fairly good results. But when the customers used for training were changed slightly, even the tree's first split changed, and the predictions changed a lot.

This is a classic problem with decision trees. A single tree makes it easy to explain the rules that split the data, but its structure can change a great deal with even small changes in the training data.

A widely used machine learning algorithm for reducing this instability is the random forest.

A random forest is an ensemble learning method that trains many decision trees and then combines each tree's predictions. The key idea is to make predictions more stable by pooling the results of many trees instead of relying on the judgment of a single tree.

Because random forests can learn complex nonlinear relationships and interactions between variables, they are used in many fields, including customer churn prediction, financial risk analysis, medical research and manufacturing quality control.

In this article we cover random forests from the basic concept through bagging and the bootstrap, random feature selection, classification and regression predictions, a worked example, OOB evaluation, variable importance, and the main settings and cautions.

For example, suppose we use data on 5,000 customers to predict whether they will cancel a service. One decision tree might choose tenure as the most important condition, while another tree might choose contract type or service usage first.

A random forest builds many such trees and combines their judgments. In classification problems it can use majority voting or the average of predicted probabilities; in regression problems the usual approach is to average each tree's predicted value.

The important point is that a random forest does not simply build many identical trees. Each tree is trained on a different sample of the data, and during splitting it also chooses a random subset of the variables.

Creating differences between trees in this way reduces the effect that the unstable judgment of any single tree has on the final prediction.

That said, a random forest does not always outperform a single decision tree. Results depend on the size of the data, the structure of the variables and the main settings, so performance must be checked on separate evaluation data.

Why use many trees instead of one decision tree

A decision tree repeatedly chooses the most effective split in the training data.

But when several candidate splits perform about equally well, even a small change in the data can change which condition is chosen.

For example, suppose that in a certain customer group both tenure and contract type are related to cancellation. Depending on the training sample, one tree might choose tenure as its first condition, while another chooses contract type first.

The problem is that once the first condition changes, the rest of the split structure can also change a lot. Because of this, a single decision tree is known as a model with high prediction variance.

A random forest eases this variability by averaging or combining the predictions of many trees. Even if one tree overreacts to some observations, its influence is reduced if the other trees have learned different patterns. This is similar to the statistical approach of combining several estimates to reduce instability.

However, if the trees' predictions are too similar to one another, combining many trees may bring only limited improvement. That is why creating diversity among the trees is important in a random forest.

Bagging is short for bootstrap aggregating. It is an ensemble method that generates many samples from the original training data, trains a model on each sample, and then combines the predictions.

For example, suppose the training data contain records for 1,000 customers. In bagging, we can draw 1,000 observations from these data with replacement to create the first training sample.

Generating a second and third sample in the same way gives each sample a different mix of customers. A separate decision tree is trained on each of these samples. Finally, combining the trees' predictions gives a single final prediction.

The core of bagging is to reduce prediction variability by combining models built from different training samples. A random forest builds on bagging, with the added feature that it also randomly chooses the variables used at each split.

How does the bootstrap work?

The bootstrap is a way to create new samples from the original data by sampling with replacement.

Sampling with replacement means that after an observation is selected, it is put back into the original data so it can be selected again in the next draw. For example, suppose we have data on 5 customers.

Drawing 5 from these data with replacement might produce samples like the following.

In Sample 1, customer B was selected twice and customer C was not selected. In Sample 2, customer A was selected twice. In this way, a bootstrap sample can contain the same observation several times, and some observations may not be included at all.

In general, when you sample with replacement as many times as the original sample size, one bootstrap sample contains about 63.2% of the original observations as distinct observations. The remaining 36.8% or so are never selected for that sample. These unselected observations can be used for the OOB evaluation described later.

For example, suppose we use the following variables to predict customer churn.

• Tenure
• Monthly fee
• Contract type
• Service usage count
• Customer-service contacts
• Used a discount

An ordinary decision tree can consider every candidate variable at each node. A random forest, however, looks for the best split at each node among only a randomly chosen subset of variables.

For example, a node in the first tree might consider tenure, monthly fee and customer-service contacts, while another tree considers contract type, service usage count and whether a discount was used. This reduces the tendency for every tree to always pick the same variable first.

If one variable has very strong predictive power, in ordinary bagging many trees may split on that same variable in similar ways. Random feature selection in a random forest helps lower this correlation between trees. As a result, combining the trees' predictions reduces variance more effectively.

The way predictions are combined differs between classification and regression problems.

Classification problems

Suppose we are predicting whether a customer will cancel a service, and five trees made the following predictions.

Three of the five trees predicted cancellation, so under majority voting the final result is cancellation. But not every random forest implementation uses only majority voting. Some implementations average the class probabilities output by each tree and decide the final class based on that average.

For example, if the trees' cancellation probabilities are 0.80, 0.40, 0.70, 0.65 and 0.30, the average cancellation probability is 0.57. With a threshold of 0.5, the final predicted class is cancellation.

Regression problems

In regression problems, each tree predicts a continuous value. For example, if five trees predict a customer's purchase amount next month as $200, $240, $220, $180 and $260, the average is $220. A random forest regression model can use this average as its final prediction.

Of all customers, 650 canceled and the remaining 4,350 stayed. The overall cancellation rate is 13%. The variables in the model are tenure, monthly fee, contract type, recent service usage and number of customer-service contacts. The research question is:

Suppose the data are split into 4,000 customers for training and 1,000 for final evaluation. The final evaluation data contain 130 customers who canceled and 870 who stayed. Settings such as the number of trees and maximum depth were chosen by cross-validation within the training data, and then a single decision tree was compared with a random forest.

Predictive performance by model

These figures do not represent a general ranking of the two algorithms.

In this example, the random forest outperformed the single decision tree on every main evaluation metric. In particular, PR-AUC rose from 0.32 to 0.47. Given that 13% of customers in the data canceled, the random forest likely provided a more useful prediction ranking for identifying customers who cancel.

But this result alone does not show that random forests beat decision trees on all data. Model performance depends on the characteristics of the data, the hyperparameter settings, the evaluation method and more.

Observations not included in a given tree's training sample are called out-of-bag (OOB) data. For example, suppose customer A was not included in Tree 1's bootstrap sample but was included in the samples for Trees 2 and 3.

In this case, the OOB prediction for customer A uses only the predictions of trees that did not use customer A in training. Combining, for each customer, the predictions of the trees that did not learn from that customer lets you compute OOB-based predictive performance.

OOB evaluation helps estimate a random forest's generalization performance without setting aside separate validation data.

Also, for data where time order or repeated observations per customer matter, a simple bootstrap-based OOB evaluation may not adequately reflect the real prediction setting.

The two most common methods are impurity-based importance and permutation importance.

Impurity-based variable importance

Impurity-based importance calculates how much impurity was reduced when a given variable was used for a split in each tree. Aggregating these values across trees gives an importance for each variable. For example, suppose a customer churn model produced the following results.

In these results, tenure and contract type accounted for a relatively large share of the random forest's splitting. But an importance of 0.34 for tenure does not mean that tenure explains 34% of the causes of actual cancellations.

Also, impurity-based importance can favor variables with many possible split points or continuous variables, so interpret it with care.

Permutation importance

Permutation importance checks how much the model's predictive performance drops when the values of a given variable are randomly shuffled. For example, if PR-AUC fell from 0.47 to 0.37 after contract type was shuffled, the original contract type information likely made a substantial contribution to the model's predictive performance.

However, if some variables are highly correlated with one another, similar information may remain in other variables even after one variable is shuffled. So permutation importance should not be interpreted as an indicator of a variable's causal effect or absolute importance either.

The main hyperparameters are as follows.

The names above are the common parameter names used in scikit-learn; names and defaults may differ by software.

Increasing the number of trees generally tends to reduce the variability of the ensemble prediction and stabilize performance. Beyond a certain point, however, further gains may be small, while training time and memory use increase.

Maximum depth and minimum samples per leaf control the complexity of each tree. If trees are too deep or leaves too small, they can pick up even fine-grained patterns in the training data.

The number of split features limits the candidate variables chosen at each node. Setting it small can increase diversity among trees, but individual trees may miss important split variables and predict less well. So the setting should be tuned to the purpose of the analysis and the characteristics of the data.

Are more trees always better?

The number of trees is one of the important settings in a random forest.

With few trees, the final result can vary quite a lot depending on the predictions of individual trees. Increasing the number of trees, on the other hand, combines more predictions and stabilizes the result. For example, consider the following results.

In this example, the performance gains gradually shrank once the number of trees passed 100. Adding trees to a random forest does not, as in boosting, fit the training data more and more closely in sequence. Generally, once enough trees are added, the predictions converge toward a stable result.

So it is hard to argue that simply increasing the number of trees causes the kind of overfitting seen in typical boosting. However, if the data that individual trees learn from have problems, or the model as a whole is biased, adding more trees will not fix it.

Also, if the data are large or there are many variables, consider the computational cost of adding trees.

For example, if you need to give a customer manager a simple set of rules, a single decision tree may be a better fit. If instead it matters more to make stable predictions using many customer characteristics, consider a random forest.

But a single tree is not always a good prediction model just because it is easy to explain, and a random forest is not impossible to interpret just because it is complex. So consider predictive performance, explainability and the operating environment together.

How do random forests and gradient boosting differ?

Random forests and gradient boosting are both ensemble methods that combine many decision trees.

But they train and combine the trees differently. A random forest usually trains its trees independently of one another and then combines their predictions by averaging or voting. Gradient boosting, by contrast, adds trees sequentially, each one built to make up for the models before it.

For example, in a random forest each tree is trained using a different data sample and different split candidates. In gradient boosting, by contrast, later trees successively make up for what the first tree's predictions missed.

Both algorithms can learn complex nonlinear relationships and interactions between variables, but neither can be said to always be more accurate. So with real data it is best to compare their performance under the same evaluation conditions.

For example, suppose 95% of all customers keep the service and only 5% cancel. Predicting every customer as staying gives 95% accuracy. But it would not find a single customer who cancels, so it would be of little practical use as a churn model.

In such cases, check precision, recall, the F1 score, PR-AUC and so on in addition to accuracy. During training you can also adjust class weights or apply resampling.

Class weights give larger weight to errors in the less common class. Resampling increases or decreases the number of samples in a given class to adjust the class mix of the training data. However, resampling must be done only within the training data.

This is because applying oversampling to the full data before splitting into training and evaluation data can put observations that are similar to or duplicates of training observations into the evaluation data.

Also, applying class weights or resampling can change how well the predicted probabilities are calibrated, so a separate calibration check is needed if the probabilities will be used for real decisions.

But that does not make overfitting disappear entirely. For example, if the training data have few observations and very many variables, or include future information that would not be available at prediction time, a random forest can also show higher training performance than is real.

Likewise, making individual trees too complex, or tuning settings while repeatedly checking validation performance, can give results more optimistic than the true generalization performance. To reduce overfitting, you can adjust maximum depth and minimum samples per leaf, and use cross-validation or OOB evaluation within the training data.

Final performance is best checked on evaluation data not used for model selection. Especially when the data distribution changes over time, consider training on past data and evaluating on data from a later period.

How does data leakage occur in a random forest?

Because random forests can learn complex combinations of variables, you need to watch out for data leakage.

For example, suppose you predict whether a customer will cancel next month while using next month's cancellation request records as an input variable. The model may show very high predictive performance, but it is using information that would not be available at the actual time of prediction.

So every variable used in training must be information available at the actual time of prediction. Leakage can also happen during preprocessing. For example, imputing missing values or selecting variables on the full data before splitting into training and evaluation data can let information from the evaluation data leak into training.

To prevent this, split the data first and compute preprocessing rules such as missing-value imputation and encoding from the training data only. Also, if there are repeated records for the same customer, it may be appropriate to split the data by customer.

But you can interpret the model's predictions using a range of methods such as variable importance, SHAP, partial dependence plots (PDP) and individual conditional expectation (ICE).

SHAP

SHAP explains how much each variable contributed to a particular prediction. For example, if a particular customer's predicted cancellation probability is high, you can look at how much tenure, contract type, customer-service contacts and so on pushed the prediction up or down.

However, the interpretation of SHAP values can depend on the scale of the model output, the baseline prediction, the background data and so on.

Partial dependence plots (PDP)

A PDP shows how the model's average prediction changes as the value of a given variable changes. For example, you can see how the average predicted cancellation probability changes as tenure increases to 3, 6, 12 and 24 months.

However, when a variable is strongly correlated with other variables, a PDP may include combinations of values that are rare in reality, so interpret it with care.

Individual conditional expectation (ICE)

ICE shows how the prediction changes with a given variable's value for each individual observation. Where a PDP shows the overall average relationship, ICE helps you see how the pattern of change in predictions differs from customer to customer.

These methods are useful for understanding the prediction structure of a random forest, but they do not carry the causal meaning that changing a variable's value would change the actual outcome in the same way.

Using the customer churn example above, a report might read as follows.

In actual studies, it is good to also report the final number of trees, maximum depth, minimum samples per leaf, number of split features and so on. And if you explain which variables were important for prediction, state which variable importance method you used.

Especially for data with class imbalance, reporting the proportion of the class of interest and the baseline performance alongside the results helps readers understand the evaluation.

Marketing and customer analytics

They can be used to predict customer churn, purchase likelihood, repeat purchase amounts, customer lifetime value and more.

Finance and economics

They can be used for loan delinquency risk, credit risk, fraud detection, forecasting corporate financial performance and more.

Health and medical research

They can be used in studies that predict specific health outcomes from multiple clinical indicators or test results. However, actual use in clinical settings requires external validation and safety evaluation.

Education research

They can predict students' academic achievement, likelihood of dropping out, completion of education programs and more.

Manufacturing and quality control

They can be used to predict equipment failure risk, product defects, production quality, maintenance needs and more.

Environmental and weather data analysis

They can predict numeric outcomes or particular conditions using a variety of environmental measurements and weather-related variables. In this way, random forests can be used not only for classification problems that predict categorical outcomes but also for regression problems that predict continuous values.

Each tree is trained on a bootstrap sample and chooses a random subset of variables during splitting. This creates diversity among the trees, and averaging or combining their many predictions eases the instability of a single tree.

OOB evaluation also lets you estimate generalization performance within the training data, and methods such as variable importance and SHAP let you examine the factors that contributed to the predictions.

But a random forest does not perform best on all data. You need to set the number of trees, maximum depth, number of split features and so on appropriately, and check performance on evaluation data that reflect the real environment of use.

Especially in classification problems where the event of interest is rare, it is important to look at precision, recall, PR-AUC and so on, not just accuracy.