Why can a model fit the training data well yet predict new data poorly?
With many correlated predictors, ordinary regression can chase noise and produce large, unstable coefficients. Training error may be small while validation error is much worse.
Regularization adds a penalty for coefficient size, trading a little training fit for a simpler model that can generalize better.
Key question
How much coefficient shrinkage improves prediction without erasing useful signal?
What does regularization add?
Penalized objective
Loss + λ × penalty
λ controls the strength of the penalty. At λ=0 the model returns to the unpenalized fit; larger λ shrinks the coefficients more strongly.
Shrinking the coefficients
λ ↑As the penalty grows, every coefficient is pulled toward zero.
Lasso drops variables
L1The corner of the L1 penalty lets some coefficients land exactly on zero.
Where λ should sit
test errorToo little penalty overfits and too much underfits, so the error curve has a floor.
How do Ridge and Lasso differ?
| Method | Penalty | Typical behavior |
|---|---|---|
| Ridge | sum of squared coefficients (L2) | keeps correlated predictors but shrinks them toward zero |
| Lasso | sum of absolute coefficients (L1) | can remove predictors by setting coefficients to exactly zero |
Correlated predictors
Lasso may select one variable from a correlated group somewhat arbitrarily, while Ridge tends to share weight across the group.
What does Elastic Net combine?
Elastic Net is useful when predictors are numerous and correlated. Its mixing parameter controls the balance between Lasso-like sparsity and Ridge-like group shrinkage.
| Setting | Behavior |
|---|---|
| Mostly L2 | more Ridge-like shrinkage |
| Mixed L1/L2 | selection with improved stability among correlated variables |
| Mostly L1 | more Lasso-like sparsity |
How is λ selected?
Cross-validation fits the model on earlier folds and evaluates prediction on held-out folds. The best λ is usually the value with the lowest average validation error, or a slightly simpler value within one standard error of the minimum.
Standardize first
Because the penalty depends on coefficient magnitude, predictors measured in different units are usually standardized before regularization.
Change penalty strength
Increase λ and compare coefficient shrinkage, the number of retained predictors, training error, and validation error.
Retained coefficients
5/5
λ
0.0
Illustrative validation error
1.52
The validation error is an illustrative U-shaped example, not a result estimated from uploaded data.
What to watch
Training error usually rises as the model is constrained. Validation error may first fall and then rise, revealing the bias–variance tradeoff.