Decision Trees

What information should you check to predict whether a customer will cancel a service next month? Are customers with a short tenure more likely to cancel? Even with the same tenure, does the chance of cancelling differ by contract type or the number of customer service inquiries?

The method that splits data by applying several conditions in turn and predicts the outcome in this way is the decision tree.

A decision tree has a structure similar to the way a person reaches a judgment by answering questions. For example, it starts with the question 'Is the tenure 12 months or less?' and then checks conditions such as 'Is it a monthly contract?' and 'Has the customer contacted customer service 3 or more times?' In the end, it predicts the chance that customers matching each set of conditions will cancel the service.

Thanks to this structure, decision trees are known as one of the machine learning algorithms whose prediction process is relatively easy to understand. They are used in many fields, such as customer churn prediction, loan risk analysis, predicting students' academic achievement and detecting defective products.

In this article we look at the basic concept of decision trees, the actual splitting process, the difference between classification trees and regression trees, Gini impurity and entropy, pruning, interpreting predictions, and points to watch in the analysis.

Supervised learning is a way of training a prediction model using data whose correct answers are known. For example, if you have existing customers' usage records and whether they actually cancelled, a decision tree can learn which customer characteristics were associated with many cancellations.

Then, when you enter a new customer's information, it follows the conditions created during training to find the group the customer belongs to and gives the prediction for that group.

A distinctive feature of decision trees is that, instead of expressing the complex relationships among many variables as a single formula, they learn rules that split the data step by step according to conditions.

For example, even a customer with a short tenure may be unlikely to cancel if they signed a long-term contract, and among customers who all have monthly contracts, those with many customer service inquiries may be more likely to cancel. Decision trees can learn such combinations of conditions.

However, the learned splitting rules represent patterns in the observed data. Just because a condition is related to the prediction, you can't conclude that the condition directly causes the outcome.

It starts at the top, checks a condition, and moves down according to the result of the condition until it reaches a final prediction. The main components are as follows.

For example, suppose a tree that predicts customer churn is built as follows.

If a new customer has a tenure of 8 months, uses a monthly contract and has made 4 customer service inquiries, they follow the conditions on the left and reach the leaf node with a 60% churn rate. In this case, the model can estimate the chance of cancellation based on the training cases in that leaf.

On the other hand, a customer with a tenure of 24 months moves right at the first condition and reaches the leaf with a 5% churn rate. In this way, a decision tree has the advantage that you can see which conditions a prediction went through.

For example, suppose you used tenure, monthly fee, contract type, number of service uses and number of customer service inquiries to predict whether customers cancel the service.

The model examines a variety of splitting conditions for these variables, such as splitting tenure at 6, 12 or 18 months, or splitting the number of inquiries at 2, 3 or 4. It then compares how well the outcome categories are separated when the data are split by each condition.

For example, if splitting tenure at 12 months puts many cancelling customers in one group and many retained customers in the other, that condition can be a useful split. A decision tree repeats this process to gradually form groups with different characteristics.

However, typical decision tree algorithms use a greedy approach that chooses the currently most favorable split at each step. So there is no guarantee that the resulting tree is the globally best tree among all possible structures.

The standard criterion used for this is Gini impurity. Gini impurity shows how mixed the categories are within a node. For example, suppose a group contains 50 customers who cancelled the service and 50 who stayed. Because the two categories are mixed in equal proportions, the impurity is high.

Conversely, if 95 of 100 customers cancelled and only 5 stayed, one category makes up most of the node, so the impurity is low. Gini impurity is calculated as follows.

Here pᵢ is the proportion of each category in the node.

Worked calculation example

In binary classification, Gini impurity is highest when each of the two categories is 50%, and it is 0 when only one category is present.

A decision tree prefers conditions that lower the weighted average impurity of the nodes created by the split. For example, if a single mixed group of customers can be separated into a group with mostly cancelling customers and a group with mostly retained customers, that can be judged a useful split.

However, low Gini impurity does not necessarily mean high predictive performance on new data. If a node contains too few samples, it may reflect chance patterns in the training data.

What are entropy and information gain?

Another standard splitting criterion used in classification trees is entropy.

Entropy is a measure of the uncertainty of a category distribution. When different categories appear in similar proportions in a node, the uncertainty is high; when one category makes up most of it, the uncertainty is low. Entropy is generally calculated as follows.

Terms with a probability of 0 contribute 0. In binary classification, if the two categories are each 50%, the entropy is 1. On the other hand, if all observations belong to one category, the entropy is 0. A decision tree can calculate information gain by subtracting the weighted average entropy of the nodes after the split from the entropy before the split.

A large information gain means that splitting the data by that condition greatly reduced the uncertainty about the categories.

The difference between Gini impurity and entropy

The two criteria can produce similar splits on many data sets, but they don't always create the same tree. So rather than deciding that one of Gini impurity and entropy is better for every problem, you can compare predictive performance on your data.

Of all customers, 650 cancelled the service and the remaining 4,350 kept it. The proportion of cancelling customers is 13%. The following variables were used in the analysis.

The research question is as follows.

Suppose training the decision tree formed the following customer groups.

Churn rate by main leaf node

In these results, the customer group with a short tenure, a monthly contract and many customer service inquiries had the highest churn rate. On the other hand, the churn rate of the group with a tenure over 12 months was about 9.1%, lower than the overall average.

In this way, a decision tree can separate groups with different levels of risk according to combinations of several conditions.

Classification tree

A classification tree is used when the dependent variable is categorical. For example, you can predict whether a customer cancels a service, whether a loan becomes delinquent or whether a product is defective. In a classification tree, the distribution of categories in each leaf node can be used to calculate the predicted category or predicted probability.

Regression tree

A regression tree is used when the dependent variable is a continuous number. For example, you can predict a customer's purchase amount next month, a home price or a production volume. In a typical squared-error-based regression tree, the mean of the observations in a leaf node is used as that node's prediction.

Comparing classification trees and regression trees

Both methods share the principle of repeatedly splitting the data according to conditions, but they differ in the outcome they predict and the splitting criterion.

How does a regression tree actually make predictions?

Suppose an online store wants to predict customers' purchase amounts next month.

You can train a regression tree using customers' number of recent visits, previous purchase amounts, length of membership and so on. Suppose training produced the following rules.

If a new customer visited the store 5 times in the past month and their previous purchases were $300, the customer reaches the third leaf node. In this case, the regression tree would predict a purchase amount of $360 for next month.

In this way, a regression tree gives different predictions depending on the conditions of the variables. But because customers in the same leaf node generally get the same prediction, the predictions can take a step shape. Also, a basic regression tree is limited in its ability to extrapolate trends beyond the range of the training data smoothly, the way a straight line would.

For example, if you need to predict a very high purchase amount not observed in the existing data, using the leaf means alone may not adequately represent values in a new range.

Typical ones include maximum depth, minimum samples to split, minimum samples per leaf and maximum number of leaf nodes.

For example, setting the maximum depth to 3 can keep the tree from becoming overly fragmented. On the other hand, setting the maximum depth to 20 lets it learn more complex combinations of conditions, but the risk of overfitting can also go up.

The minimum samples per leaf matters too. If a leaf contains only 2 customers, a new customer's chance of cancelling is estimated from the outcomes of those two customers alone.

Conversely, requiring each leaf to contain at least 100 customers can reduce unstable predictions based on very small groups. But it may also miss the patterns of small customer groups that really matter. So rather than setting these values arbitrarily large or small, it is best to choose them through cross-validation.

What is pruning?

A decision tree can become more and more complex as it keeps splitting on conditions.

Looking at the training data alone, prediction error often decreases as the tree gets deeper. But on new data, a complex tree can actually perform worse. The method used to reduce this problem is pruning.

Pruning is a way of simplifying a tree by removing branches that are unnecessarily complex relative to their predictive value. It can be divided into two broad approaches.

Pre-pruning

Limits further splitting while the tree is being trained. This works by setting the maximum depth, minimum samples per leaf, minimum samples to split and so on.

Post-pruning

First builds a relatively large tree and then removes unnecessary branches. The best-known example is cost-complexity pruning. This method decides the level of pruning by considering both the tree's prediction error and its complexity.

Both approaches can ease overfitting and make the results easier to understand, but a tree that is too simple can miss important patterns. So when applying pruning, rather than focusing only on shrinking the tree, check its predictive performance on validation data.

Because decision trees can keep splitting until they separate very small groups, you need to watch out for overfitting. For example, suppose you built a tree to predict cancellation for 5,000 customers.

Example of predictive performance by tree depth

With a maximum depth of 4, both training accuracy and validation accuracy were relatively high. But as the depth increased, training accuracy kept rising while validation accuracy actually fell. In particular, the tree with no depth limit classified the training data perfectly, but its accuracy on the validation data dropped to 79%.

These results show that the model may have fit the detailed patterns of the training data too closely. However, accuracy alone can't fully tell you whether there is overfitting. In problems with severe class imbalance, also check other evaluation metrics such as precision, recall and PR-AUC.

Evaluation metrics for classification trees

For example, if 90% of customers keep the service and only 10% cancel, predicting that every customer stays still gives 90% accuracy. So with data where the category of interest is rare, like customer churn, you should not evaluate a model by accuracy alone.

Evaluation metrics for regression trees

For a regression tree that predicts customers' purchase amounts, you can use MAE or RMSE to check the difference between actual and predicted purchase amounts. You can also compare performance on training and validation data to check for possible overfitting.

How do you interpret predicted probabilities and classification thresholds?

In a classification tree, the proportions of categories in each leaf node can be used to calculate predicted probabilities.

For example, if a leaf node contains 100 customers and 60 of them cancelled the service, a basic classification tree can estimate a 60% probability of cancellation for a new customer who reaches that leaf. But to check whether 60% of such customers will really cancel, you need to evaluate the calibration of the predicted probabilities on separate data.

Probability estimates can be especially unstable when a node contains few samples.

Classification threshold

To turn predicted probabilities into the categories cancel or stay, you can use a classification threshold. For example, with a threshold of 0.5, customers with a predicted cancellation probability of 50% or more are classified as at-risk customers. But lowering or raising the threshold can change classification performance, such as precision and recall.

So it is best to set the threshold by considering which prediction errors matter more in practice.

For example, if missing a customer who will cancel is costly, you can adjust the threshold to increase recall. Conversely, if giving unnecessary perks to customers who would not have cancelled is costly, you can choose a threshold that increases precision.

However, it is appropriate to decide such thresholds during training and validation, and to use the final evaluation data to check the performance of the chosen threshold.

The standard measure used for this is feature importance. Typical impurity-based feature importance is calculated from how much each variable reduced impurity during the splitting process. For example, suppose the following results were obtained.

In these results, tenure had the highest importance at 0.38. This means that in the trained decision tree, splits using tenure contributed relatively much to the total reduction in impurity. But an importance of 0.38 does not mean that tenure has a 38% causal effect on customer cancellation.

Also, impurity-based importance can be biased, for example by favoring variables with many possible split values. So when interpreting feature importance, you can also review other methods such as permutation importance.

Logistic regression expresses the relationship between the independent variables and the log odds of an event as a mathematical model. A decision tree, by contrast, repeatedly splits the data according to conditions on the variables.

For example, if you want to explain how the number of customer service inquiries relates to the probability of cancellation in terms of odds ratios, logistic regression can be useful. On the other hand, if the relationship between the number of inquiries and cancellation is clear only among customers with a short tenure, a decision tree may express such a conditional pattern more easily.

However, logistic regression can also express complex relationships if you add interaction or nonlinear terms. So you can't say that either method is always better.

How do decision trees and random forests differ?

A decision tree is a method that trains a single tree.

A random forest, on the other hand, is an ensemble method that trains many decision trees and combines their results. A single decision tree makes it easy to draw the whole split structure, so it is good for seeing which conditions an individual prediction went through.

A random forest, by contrast, combines the predictions of many trees, so it often gives more stable predictions than a single tree.

For example, if you need a simple classification rule that can be applied to customers, a decision tree can be useful. On the other hand, if higher predictive performance and stability matter, you can also compare ensemble methods such as random forests.

However, random forests don't give better results than a single decision tree on every data set. Actual predictive performance should be checked on separate validation data.

Don't let the tree get too deep.

If a tree gets too deep, it ends up separating very small groups and can overfit the training data. So control its complexity with the maximum depth, minimum samples per leaf, pruning and so on.

Be careful with predictions from small leaf nodes.

If a leaf contains only 5 customers and 4 of them cancelled the service, the churn rate in the training data is 80%. But because the sample is so small, it can be hard to estimate a new customer's real chance of cancelling reliably. So check the number of samples in each leaf together with the reliability of the predicted probabilities.

The tree structure can change even if the data change only slightly.

A decision tree can be sensitive to small changes in the data when choosing particular split conditions. For example, adding or removing some customers' records could change the split variable chosen first. So don't interpret an individual tree's split structure as an absolute rule.

Check how categorical variables are handled.

Decision trees can use categorical as well as numeric variables, but different software handles categorical variables differently. Some implementations can handle categorical variables directly, while others require a transformation such as one-hot encoding.

So check which data formats the library you plan to use supports.

Prevent data leakage.

A model that predicts future outcomes must not include information that is only known after the time of prediction. For example, if you predict whether customers will cancel next month while using next month's cancellation request records as an input variable, performance can look much better than it really is.

Also, the rules for preprocessing such as missing-value imputation and variable encoding should be estimated from the training data only.

Distinguish statistical association from causation.

Just because the condition 'tenure 12 months or less' was chosen as a main split in the decision tree does not mean that tenure itself directly causes cancellation. Other factors such as contract type, customer characteristics and service satisfaction may be at work too, so splitting rules should be interpreted as data patterns used for prediction.

Using the customer churn prediction example above, you could write it as follows.

In actual prediction research, you should report not only these splitting results on the training data but also performance metrics obtained on separate evaluation data, such as accuracy, precision, recall, ROC-AUC and PR-AUC.

Also, stating the final tree's maximum depth, minimum samples per leaf, pruning settings and so on helps other researchers reproduce the analysis.

Marketing and customer analytics

You can predict whether customers will cancel a service, how likely they are to buy, whether they will buy again and so on. They are especially useful for separating customer groups with particular behaviors or usage characteristics.

Finance and credit risk analysis

They can be used for the chance of loan delinquency, whether a borrower defaults, fraud detection and more. However, when applying them to real financial decisions, fairness, explainability and relevant regulations should also be considered.

Education research

You can predict whether students pass an exam, their chance of dropping out, their academic achievement and so on. They can also be used to explore which groups of students with certain learning conditions show particular outcomes more often.

Health and medical research

They can be used in research that predicts categories of a particular health condition or continuous outcomes from patients' test results or health indicators. However, applying them in clinical settings requires sufficient external validation and safety evaluation.

Manufacturing and quality control

You can predict whether products are defective, production quality grades, the chance of equipment failure and so on.

HR and organizational research

You can build prediction models related to whether employees leave, whether they complete training programs, job performance and so on. In this way, decision trees are widely used for problems that use numeric and categorical variables and aim to predict outcomes through combinations of several conditions.

If the outcome variable is categorical you can use a classification tree, and if it is a continuous number, a regression tree. Classification trees use Gini impurity, entropy and the like as splitting criteria, while regression trees use squared error, variance reduction and so on.

The biggest strength of decision trees is that the prediction process is relatively intuitive. You can see which conditions led to a case being classified into a particular group, which helps you understand patterns in complex data.

But if a tree gets too deep, overfitting can occur, and the split structure can change even with small changes in the data. So it is important to control the tree's complexity and validate its performance with separate training and evaluation data.