Hyperparameter Tuning: Grid, Random, and the Validation Curve

A model learns its parameters — the coefficients, the split points — from the data. But its hyperparameters — tree depth, learning rate, regularization strength — you set beforehand. Those choices can make or break performance, and finding good ones is tuning.

Note

Parameters are learned; hyperparameters are chosen. Tuning is the search for the hyperparameters that let the model generalize best.

The Validation Curve

Plot performance as you dial a hyperparameter, and a familiar shape appears — as in the chart above. Training error keeps falling as the model grows more flexible, but validation error bottoms out and then rises. That bottom is the sweet spot.

Too little flexibility → underfitting, high error on both
Too much → overfitting, low train error but rising validation error
The best hyperparameter sits at the validation minimum

Grid vs Random Search

Method	How it searches
Grid search	Every combination in a defined grid — thorough but expensive
Random search	Random samples across ranges — finds good values faster
Bayesian / smart search	Uses past results to focus the next trials

Random search usually beats grid search for the same budget, because a few hyperparameters matter far more than the rest, and random sampling explores those more efficiently.

Don't Tune on the Test Set

Every hyperparameter you try and score peeks at the validation data. Do it enough and you overfit the validation set itself — the tuning looks great, then fails on truly new data.

Watch out

Score tuning with cross-validation, and keep a final test set untouched until the very end. Tuning against your only holdout quietly inflates the result.

Tuning in the SKARI Model Lab

The Model Lab ships sensible defaults and tunes key hyperparameters with cross-validated search — so you get a well-configured model without hand-running a grid, and without leaking your test set.

Cross-validated hyperparameter search built in
Sensible defaults so you're not starting from scratch
A clean final test score, separate from the tuning

Takeaway

You get a tuned model and an honest score — the search and the final evaluation kept properly apart.

Frequently Asked Questions

Grid or random search?

Random search for most cases — it finds strong values faster. Use grid search only for a small, well-understood set of options.

Which hyperparameters matter most?

Usually just a few — learning rate and depth for boosting, regularization strength for linear models. Focus your budget there.

How do I avoid overfitting the tuning?

Score with cross-validation and reserve a separate, untouched test set for the final number.

Conclusion

Tuning is finding the flexibility that generalizes best — the bottom of the validation curve. Search efficiently, score with cross-validation, and protect a final test set, and your tuned model earns the score it reports.

Takeaway

Aim for the validation minimum, not the training minimum — that's where the model does best on data it hasn't seen.

Cross-Validation

The scoring loop tuning relies on

Overfitting & Regularization

What the validation curve reveals

Gradient Boosting & XGBoost

A model where tuning pays off most