A model that nails every training row but stumbles on new data hasn't learned the pattern — it has memorized the noise. That's overfitting (오버피팅), and it's the central problem in machine learning.
Watch out
The Bias-Variance Tradeoff
- High bias (underfitting): the model is too simple and misses real structure
- High variance (overfitting): the model is too flexible and chases noise
- The goal is the sweet spot between them — the dip in the validation curve
The bias-variance tradeoff is the whole game: reduce variance without letting bias grow too much. Regularization is the main dial for doing exactly that.
How Regularization Helps
Regularization adds a penalty for complexity, discouraging the model from leaning too hard on any one feature. Instead of fitting large coefficients that chase every wiggle, it shrinks them — trading a little training accuracy for much better generalization. The picture below is the fastest way to see it.
Coefficient Shrinkage
Ridge shrinks every coefficient toward zero; Lasso pushes some all the way to zero, dropping those variables.
| Method | Penalty | Effect on coefficients |
|---|---|---|
| Ridge (L2) | Sum of squared coefficients | Shrinks all coefficients toward zero |
| Lasso (L1) | Sum of absolute coefficients | Shrinks some to exactly zero — feature selection |
| Elastic Net | Blend of L1 and L2 | Balance of shrinkage and selection |
The Role of Lambda (λ)
Every regularized model has one key knob: the penalty strength, written λ (lambda). It sets how hard the model is pushed toward simpler coefficients.
λ = 0 -> no penalty (ordinary OLS, may overfit) λ small -> gentle shrinkage λ large -> strong shrinkage, coefficients approach zero (may underfit)
Tip
When to Use Which
The choice between Ridge, Lasso, and Elastic Net comes down to your predictors.
| Situation | Use |
|---|---|
| Most predictors are relevant — keep them all | Ridge |
| You also want automatic feature selection | Lasso |
| Many correlated predictors | Elastic Net |
Tip
Other Cures
- More data: the simplest fix — harder to memorize a larger, richer set
- A simpler model: fewer parameters, less room to overfit
- Cross-validation: rotate the validation set to get an honest estimate
- Early stopping: halt training once validation error starts rising
In the SKARI Model Lab
The Model Lab lets you run the whole comparison in one flow, so you don't just take regularization on faith — you see it beat plain OLS on validation data.
Same data -> OLS -> Ridge -> Lasso -> Elastic Net -> compare
- Fit OLS, Ridge, Lasso, and Elastic Net on the same data without code
- Cross-validated metrics that make the train/validation gap visible
- See exactly which coefficients Lasso drives to zero — automatic feature selection
- Pick λ by cross-validation, not by guesswork
Takeaway
Frequently Asked Questions
Ridge or Lasso?
Ridge when you want to keep all features but tame them; Lasso when you also want it to drop irrelevant ones. Elastic Net blends both, and is safest when predictors are correlated.
If Lasso zeros a variable, does that mean the variable is useless?
It means the variable contributes little to prediction in this dataset. It may still matter in another dataset or domain, so don't treat a Lasso zero as an absolute verdict — it's dataset-specific.
How do I know I'm overfitting?
Compare training and validation scores. A large gap — strong on training, weak on validation — is the signature.
Does regularization need scaling?
Yes. Without standardized features, the penalty falls unevenly. See the Data Normalization guide.
Conclusion
Overfitting is memorizing instead of learning. Watch the validation curve, tune λ with cross-validation, and use Ridge, Lasso, or Elastic Net to keep complexity — and the bias-variance tradeoff — in check.
Takeaway
Regression Analysis
The model regularization tames
Cross-Validation
How you pick λ honestly
Data Normalization
Why regularization needs scaling