Regularized Regression

Suppose you use many variables, such as age, income, number of visits, purchase frequency, and whether discount coupons were used, to predict how much customers spend. Does using more variables in the analysis make more accurate predictions possible?

Not necessarily. In practice, as unimportant variables get included in the model or variables carrying similar information pile up, predictive performance on new data can drop. This problem of a model fitting its training data too closely is called overfitting.

A classic method for reducing overfitting and controlling model complexity is regularized regression.

Regularized regression adds regularization, which limits the size of the regression coefficients, to ordinary regression to keep the model from becoming too complex. The main types are ridge regression, lasso regression, and elastic net regression.

In this article we cover the basic concepts of regularized regression, overfitting and multicollinearity, the three main methods, a worked example, setting the regularization strength, and points to watch out for when interpreting the results.

Ordinary linear regression estimates the coefficients so as to minimize the error between observed and predicted values. But when there are many independent variables or they are highly correlated with each other, the coefficients can become unstable and predictive performance on new data can be poor.

To reduce this problem, regularized regression keeps the coefficients from growing too large.

For example, consider a situation where you use 30 variables to predict how much customers spend. Some variables are closely related to the amount spent, but others may be nearly unrelated or contain overlapping information.

Ordinary regression estimates the coefficients with all of these variables included. Regularized regression, by contrast, can simplify the model by shrinking the coefficients or setting some of them to 0. This reduces the problem of fitting the training data too closely and can improve predictive performance on new data.

Using regularized regression does not, however, improve predictive performance in every case. The results depend on the regularization strength and the characteristics of the data.

Why is regularized regression needed?

The main reasons regularized regression is needed are overfitting and multicollinearity.

Overfitting

Overfitting is when a model learns not only the general patterns in the training data but also chance fluctuations and noise.

For example, suppose you built a regression model with 50 variables using purchase data from 100 customers. This model may show very high explanatory power on the training data but perform much worse when predicting how much new customers will spend.

Regularized regression helps reduce unnecessary model complexity by limiting the size of the regression coefficients.

Multicollinearity

Multicollinearity is the problem of high correlations among the independent variables. For example, suppose you include variables that are closely related to one another at the same time, such as a customer’s monthly spending, annual spending, and total number of purchases.

If these variables carry similar information, ordinary regression may have trouble estimating the individual coefficients stably. Regularized regression can help reduce the variability of the estimates and make the model more stable by shrinking the coefficients.

Regularized regression does not, however, remove the cause of multicollinearity or change the actual relationships among the independent variables.

Regularized regression adds a penalty that depends on the size of the regression coefficients.

Regularized regression looks for a balance between reducing the model’s error and limiting the size of the coefficients. In the process, it can accept some bias in exchange for lower variance in the estimates. This balance is called the bias–variance tradeoff.

If regularization is applied too weakly, overfitting may not be adequately addressed. Conversely, if it is applied too strongly, even important relationships can be shrunk too much, resulting in underfitting.

What types of regularized regression are there?

The main regularized regression methods are ridge, lasso, and elastic net.

All three penalize the regression coefficients, but they differ in how they limit the coefficients.

Ridge focuses on shrinking coefficients while keeping many variables. Lasso has a variable selection feature that drops some variables from the model, and elastic net combines the features of both. Which method is more appropriate depends on the number of variables, the correlations among them, and the research goal.

It controls model complexity by keeping coefficients from growing too large. The objective function of ridge regression can generally be written as follows.

Here λ (lambda) is the value that determines the strength of regularization. As λ grows, a stronger penalty is applied to the coefficients, and their absolute values tend to get smaller. Ridge shrinks coefficients toward 0 but generally does not set them exactly to 0.

Worked example

Suppose you use the following variables to predict how much customers spend.

• Average visits per month
• Total number of purchases
• Average monthly spending
• Number of discount coupons used
• Membership length

If some of these variables are highly related, such as number of visits and number of purchases, coefficient estimates can be unstable in ordinary regression. Ridge regression shrinks each variable’s coefficient, which reduces the chance of giving an excessively large weight to a particular variable.

Characteristics of ridge regression

Ridge regression can be useful when many variables contribute to predicting the outcome and the variables are highly correlated. But because it tends to keep every variable, lasso or elastic net may be more appropriate if you want to build a simple model with only the important variables.

Like ridge regression, it shrinks the coefficients, but there is an important difference.

Variables whose coefficients become 0 drop out of the prediction equation, so lasso is used as a method that performs regularization and variable selection at the same time. The objective function of lasso regression can generally be written as follows.

As λ grows, a stronger penalty is applied and some coefficients become more likely to be 0.

Worked example

Suppose you use 20 variables to predict how much customers spend on repeat purchases. Some of these variables are related to repeat purchase spending, but others may contribute little to the prediction. Applying lasso regression sets some variables’ coefficients to 0, giving a model with fewer variables.

For example, only 7 of the original 20 variables might remain with nonzero coefficients. This does not mean, however, that the variables lasso selected are necessarily the most statistically important or have causal effects. The selection can change depending on the correlations among variables, the regularization strength, and the makeup of the sample.

By using both penalties together, it can shrink coefficients while also setting some variables’ coefficients to 0. The objective function of elastic net generally has the following structure.

In elastic net, you control the overall regularization strength and the mixing ratio of the L1 and L2 penalties. Many implementations express the regularization strength as alpha or λ and the mixing ratio as l1_ratio or another symbol. Parameter names and the scaling of the objective function can differ across software, so check them.

Worked example

Suppose you use the following variables to predict customers’ purchasing behavior.

• Visits in the last 30 days
• Visits in the last 90 days
• Purchases in the last 30 days
• Purchases in the last 90 days
• Number of discount coupons used
• Membership length

These include highly related variables, such as visits in the last 30 and 90 days. Lasso may select only some of a set of highly correlated variables, but elastic net, depending on its settings, can keep related variables together while dropping some unnecessary ones.

So it can be a useful choice for data with many highly correlated variables.

The variables used in the analysis are as follows.

• Recent number of visits
• Total number of purchases
• Average purchase amount
• Membership length
• Number of discount coupons used
• Number of customer service contacts

Suppose ordinary linear regression, ridge, lasso, and elastic net were each trained and their predictive performance compared on a separate test set.

Predictive performance by model

RMSE is assumed to be in the same units as the purchase amount.

RMSE (root mean squared error) is a measure of the difference between actual and predicted values; smaller values mean smaller prediction errors on average. Ordinary linear regression had the lowest RMSE on the training data but the largest error on the test data.

The regularized models, on the other hand, had slightly larger errors on the training data but better predictive performance on the test data. This example suggests that ordinary linear regression may have fit the fluctuations in the training data too closely. Elastic net in particular had the lowest test RMSE, 20.9.

With real data, though, ordinary linear regression, ridge, or lasso could perform better. So rather than assuming one algorithm is always superior, compare performance under the same evaluation conditions.

How do the coefficients of ridge, lasso, and elastic net differ?

The differences among the three methods also show up in the regression coefficients.

The following is an example that assumes different regularized regression models were applied to the same data.

Assumes the same standardization was used.

In ridge regression, every variable’s coefficient moved closer to 0, but none became exactly 0. In lasso regression, the coefficients of membership length and coupons used became 0, so those variables were dropped from the model. Elastic net shrank the coefficients while setting the coefficient of coupons used to 0.

In this way, the size of the coefficients and the variables ultimately used can differ by type of regularization. But a coefficient of 0 does not mean the variable has no effect in reality. It means that, given the data, the other variables, and the regularization strength, the variable did not remain in the prediction model.

The regularization strength determines the size of the penalty placed on the coefficients. In general, a small regularization strength puts weak limits on the coefficients, and a large one shrinks them more strongly.

If you set the regularization strength too small, you may not reduce overfitting enough. Conversely, if you set it too large, even truly important relationships can be shrunk, lowering predictive performance. For this reason, the regularization strength is often chosen by cross-validation.

For example, you can set several candidate λ values, calculate the cross-validation error for each, and choose the value with the lowest average validation error. To evaluate the performance of the chosen regularization strength objectively, however, you need test data that were not used for model selection.

For example, a customer’s age might range from 20 to 70, while annual spending might range from a few hundred dollars to several thousand dollars. If you use such variables in their original units, the size of the coefficients depends on the units, so even when the same penalty is applied, its effect can differ from variable to variable.

So with ridge, lasso, and elastic net, it is generally a good idea to standardize the independent variables before the analysis. A standard method is z-score standardization.

Standardization transforms each variable to have a mean of 0 and a standard deviation of 1. This lets regularization be applied more consistently to variables measured in different units.

So it is good practice to build the standardization and regression model into a single pipeline and estimate the preprocessing parameters only within the training data.

Variable selection (feature selection) is the process of choosing the variables to use in a model. Regularized regression, by contrast, is a method that controls model complexity by penalizing the regression coefficients. Ridge regression generally keeps every variable, so it focuses on shrinking coefficients rather than on direct variable selection.

Lasso and elastic net can set some coefficients to 0, so they can also perform variable selection. But the set of variables lasso selects does not always stay stable. Especially when there are highly correlated variables, the selected variables can change with small changes in the data sample or the regularization strength.

So in studies where the stability of variable selection matters, it is good to check how consistently the selection appears, for example through repeated resampling or cross-validation.

But because ridge, lasso, and elastic net use a penalty when estimating the coefficients, the p-values and confidence intervals of ordinary least squares regression are hard to apply as they are. Lasso in particular selects variables based on the data, so running ordinary statistical tests without accounting for the selection process can lead to wrong conclusions.

The main goals of regularized regression are generally to improve predictive performance and control model complexity. So when evaluating the results, the key things to check are cross-validation performance, prediction error on separate test data, the stability of the coefficients, and so on.

Of course, there are separate methods for performing statistical inference on regularized regression coefficients. These methods, however, should be kept distinct from calculating p-values in ordinary regression.

For regression models that predict a continuous dependent variable, you can use the following measures.

For example, if you predict how much customers will spend next month, you can use MAE or RMSE to evaluate the difference between the actual and predicted amounts. You can use cross-validation to choose the regularization strength, and it is best to check final performance on holdout data that were not used in model selection.

If you compared several regularization strengths and algorithms, be careful about judging final generalization performance using only the validation data used for that comparison. Also, regularization can be applied not only to continuous outcomes but also to other models such as logistic regression, so choose the actual evaluation metrics to fit the dependent variable and the purpose of the analysis.

Stronger regularization is not always better.

As the regularization strength increases, coefficients get smaller, but even important relationships can be shrunk too much. In that case predictive performance may actually drop, so choose an appropriate regularization strength through cross-validation or similar methods.

A selected variable is not necessarily a cause.

Just because lasso selected a particular variable, you cannot conclude that the variable has a causal effect on the outcome. Regularized regression is a method for finding combinations of variables useful for prediction based on observed data. Claiming causation requires a separate research design and causal inference.

Multicollinearity does not disappear completely.

Regularized regression can help stabilize coefficient estimates for highly correlated variables, but it does not remove the actual correlations among the variables themselves. Especially in studies that aim to interpret the effects of individual variables, examine the relationships among variables separately.

Prevent data leakage.

If you apply standardization, missing value imputation, variable selection, and so on to the entire dataset first, information from the test data can leak into the training process. So build the preprocessing and the regularized regression model into a single pipeline, and during cross-validation, estimate the preprocessing parameters only on each training fold.

Regularized regression does not handle every nonlinear relationship.

Basic ridge, lasso, and elastic net are generally based on a linear prediction equation. If the actual relationships among variables are complex and nonlinear, you may need to add polynomial or interaction terms or consider other models.

When should you choose ridge, lasso, or elastic net?

It is hard to say that any one of the three methods is always better.

The appropriate method can depend on the characteristics of the data and the purpose of the analysis.

For example, if you have hundreds of variables and expect only some of them to be useful for prediction, you can consider lasso. On the other hand, if most variables each contribute a little to the prediction and are highly correlated with one another, ridge may be a better fit.

If there are many highly correlated variables and you also want to drop some variables, you can consider elastic net. But these guidelines are general reference points; in practice it is best to compare performance using the same data split and evaluation metrics.

Using the customer purchase amount example above, you could write it as follows.

In a real analysis, it is good to also report the data split ratio, the cross-validation method, the selected regularization strength, the elastic net mixing ratio, and so on. And if variable selection is a main goal, you should also check which variables were included in the final model and how stable the selection is.

Marketing and customer analytics

You can predict customers’ spending, repeat purchase amounts, service usage, and so on, or select the variables useful for prediction among many customer behavior variables.

Finance and economics

You can build prediction models involving many economic variables, such as corporate financial performance, asset returns, and consumer spending.

Health and medical research

You can predict continuous health indicators from many test results or measured variables, or explore useful variables among many candidates.

Education research

You can build models that predict students’ academic achievement from many learning behavior and educational environment variables.

Social science research

You can use many survey items or socioeconomic variables to predict an outcome while reducing overfitting.

Manufacturing and quality control

You can predict product quality and performance using many sensor measurements or production conditions. In this way, regularized regression is useful for controlling the complexity of prediction models in data with many predictors or overlapping information.

The main types are ridge regression, lasso regression, and elastic net, which differ in how they shrink coefficients or drop some variables. Ridge can be useful for shrinking the coefficients of many variables together, lasso for performing variable selection and shrinkage at the same time, and elastic net for combining the features of both.

But a higher regularization strength does not always produce a better model. Use cross-validation and separate test data to choose an appropriate regularization strength and check actual predictive performance. Also take care not to interpret the variables selected by regularized regression directly as causal factors.