A single shallow tree barely beats a coin flip on a hard problem. Boosting's trick is to build trees one after another, each trained on the errors the previous trees still make. Add enough of them and the residual error shrinks round after round — as the chart above shows.
Note
How Boosting Works
- Start with a simple prediction (often the average)
- Fit a small tree to the residuals — what the model got wrong
- Add a fraction of that tree, scaled by the learning rate
- Repeat, each tree chipping away at the remaining error
Each tree is deliberately weak; the strength comes from combining hundreds of them.
A Worked Example: Learning the Residual
"Each tree learns the previous errors" is abstract until you trace numbers through it. Suppose the true value is 120:
Tree 1 -> predicts 100 (actual = 120, error = +20) Tree 2 -> learns the +20 (correction brings it to 118) Tree 3 -> fixes the last +2 (now ~120) ... each tree shrinks the leftover error
Every new tree doesn't predict the target directly — it predicts what's still missing. Stack enough of those corrections and the ensemble homes in on the right answer.
Gradient Boosting vs XGBoost vs LightGBM vs CatBoost
The most common beginner question is "how is XGBoost different from gradient boosting?" The answer: they're the same core algorithm, with XGBoost and its cousins adding engineering on top.
| Model | What it adds |
|---|---|
| Gradient Boosting (GBM) | The base algorithm — trees fit to residuals |
| XGBoost | Regularization, parallel processing, built-in missing-value handling |
| LightGBM | Speed and strong performance on large datasets |
| CatBoost | Native, well-handled categorical variables |
All four win tabular-data competitions and power countless production models. Which one is best is an empirical question — the reason you compare them rather than guess.
The Hyperparameters That Matter
| Parameter | Controls |
|---|---|
| n_estimators | Number of trees |
| learning_rate | How much each tree corrects at a time |
| max_depth | Complexity of each tree (3–6 is the norm) |
| subsample | Fraction of rows used per tree |
| colsample_bytree | Fraction of features used per tree |
Watch out
Boosted Models in the SKARI Model Lab
Since which boosted model wins is empirical, SKARI's Auto Compare trains the whole roster on the same data and ranks them — no code, no hand-tuning:
Logistic Regression -> Decision Tree -> Random Forest
-> XGBoost -> LightGBM -> CatBoost -> Auto Compare- Boosted models compared against simpler baselines on the same validation split
- Feature importance and validation metrics for every model
- Sensible defaults with tuning when you want it
Takeaway
Frequently Asked Questions
Boosting or random forest?
Boosting is usually more accurate but touchier to tune; forests are more forgiving. Try both — SKARI makes that easy.
Why a small learning rate?
Small steps let the ensemble correct itself gradually, which generalizes better — at the cost of needing more trees.
Does boosting need scaling?
No — like all tree models, it splits on thresholds and is unaffected by feature scale.
How do I explain a boosted model's predictions?
Feature importance shows which variables matter overall; to explain an individual prediction — why this row was scored the way it was — SHAP values are the common tool, attributing the prediction across features.
Conclusion
Gradient boosting turns a crowd of weak trees into one of the strongest models for tabular data. Keep the learning rate small, stop early, and it rewards you with accuracy that's hard to beat.
Takeaway
Decision Trees Explained
The building block of boosting
Hyperparameter Tuning
Dialing in learning rate and depth
Cross-Validation
Scoring boosted models honestly