Decision Trees Explained

A decision tree splits the data one question at a time: is income above a threshold? Is the account older than a year? Each answer sends the row down a branch, until a leaf hands back a prediction — as the diagram above shows.

A Worked Example

The easiest way to grasp a tree is to trace one. Here's a loan-approval tree deciding whether to approve an applicant:

Income > 50,000?
├─ No  ->  Reject
└─ Yes
      Credit score > 700?
      ├─ Yes ->  Approve
      └─ No  ->  Reject

Every applicant follows one path from the top down to a decision. That readability — you can point to the exact rules behind any prediction — is why trees are trusted in regulated fields like lending and healthcare.

The Anatomy: Root, Internal, and Leaf Nodes

• Root node: the very first split, at the top of the tree
• Internal node: any further split along the way
• Leaf node: an end point with no more splits — it holds the prediction

Knowing these terms makes every other tree article (and every diagram) easier to read.

How a Split Is Chosen

At each node, the tree tries every feature and threshold and keeps the split that best separates the classes. "Best" is measured by impurity: how mixed the groups are before and after the split.

• Gini impurity: the chance of mislabeling a random pick from a node
• Entropy / information gain: the reduction in disorder from a split
• For regression trees, the split minimizes variance instead

Feature Importance: Trees as an X-ray

In practice, many people use a tree less to predict than to see which variables matter. Every split reduces impurity, and summing those reductions per feature yields a feature-importance score.

Read it directly: Income is the strongest driver of the prediction, and Region barely moves it. That single table is often the most actionable output a tree produces.

The Overfitting Trap

Keep splitting and a tree will carve out a leaf for nearly every row — perfect on training data, useless on new data. Depth is the dial: too shallow underfits, too deep memorizes.

• Limit max depth or minimum samples per leaf
• Prune branches that don't improve validation performance
• Prefer an ensemble — a random forest or boosting — over one deep tree

Pruning: Grow, Then Cut Back

Pruning is the classic cure for an overgrown tree. You first grow a large tree, then remove the branches that don't improve validation performance — keeping only the splits that genuinely help. The result is a smaller, more general tree that no longer memorizes noise.

From One Tree to a Forest

One tree is unstable — shift a few rows and its splits change. A random forest grows many trees on random subsets and averages them, trading a little interpretability for a large gain in accuracy and stability. Boosting goes further, building trees in sequence.

Decision Trees in the SKARI Model Lab

The Model Lab trains decision trees and forests without code: pick the target, and SKARI fits the model, shows feature importance, and reports validation metrics so you see whether it generalizes.

Because a single tree is rarely the strongest model, SKARI's Auto Compare runs the whole tree family on the same data and ranks them for you:

Decision Tree  ->  Random Forest  ->  XGBoost  ->  LightGBM  ->  CatBoost  ->  compare

• Decision tree, random forest, and boosted models side by side
• Feature importance to see what drives predictions
• Train/validation metrics that flag overfitting

Frequently Asked Questions

Gini or entropy?

They usually agree; Gini is slightly faster and the common default. The choice rarely changes the tree much.

Do trees need scaling?

No — splits are based on thresholds, so trees are unaffected by feature scale, unlike regression or clustering.

Tree or forest?

Use a single tree when interpretability matters most; a forest or boosting when accuracy matters most.

What is CART?

CART (Classification and Regression Trees) is the algorithm behind most tree implementations, including scikit-learn's. It builds binary splits using Gini impurity (for classification) or variance reduction (for regression).

Conclusion

Decision trees are intuitive and transparent, but their appetite for depth makes overfitting the default. Constrain the depth, prune, or ensemble — and you keep the readability without the memorization.