Robust Regression

Does a used car sell for less the more miles it has on it? But what happens to the results if the data include a rare car sold far above market value or a car sold unusually cheaply?

As in this case, a method that estimates a relationship so that a few unusual observations don't have an outsized influence on the overall regression results is called robust regression.

Ordinary least squares regression squares large residuals, so a handful of extreme sales can move the regression line considerably. Depending on the estimation method chosen, robust regression softens the influence of large residuals so you can look at the overall relationship more stably. That does not mean it automatically fixes outliers or data entry errors.

It is useful for data in which extreme observations can occur, such as transaction prices, income, or measurements from a manufacturing process. In this article we use the relationship between used-car mileage and sale price as an example to look at how the estimation works, how it compares with ordinary regression, how to interpret the regression lines and results table, checking influential points, and reporting the results.

Linear regression expresses, as a regression coefficient, how much the outcome changes on average when a predictor increases by one unit. Ordinary least squares (OLS) minimizes the sum of squared residuals, the differences between observed and predicted values.

Robust regression uses a loss function or estimation approach that softens the influence of large residuals when they occur. For example, Huber regression applies squared loss when a residual is small and a loss that grows linearly once it exceeds a certain threshold.

Suppose that for 14 ordinary passenger cars, the sale price falls steadily as mileage increases, but one car with high mileage sold at a high price because of special collector value. Conversely, a car with an accident history might sell at a very low price.

When special sales like these are mixed in, the OLS regression line can shift toward those observations to reduce the large errors. If what the researcher wants to know is the general trend in used-car prices, it helps to compare OLS with robust estimates rather than presenting OLS alone.

However, it does not automatically handle high-leverage observations or data entry errors. You should check the characteristics of the data and the diagnostic results before and after applying robust regression.

① Specify a regression equation for the relationship

Set sale price as the outcome and mileage as the predictor. The basic model is sale price = intercept + slope × mileage + error.

② Compute an initial regression line and residuals

Find the difference between the price the model predicts and the actual sale price. The larger the residual, the farther that sale is from the general trend.

③ Adjust the influence of large residuals

Huber M-estimation gives relatively lower weights to observations with large standardized residuals so that extreme values don't move the coefficients too much.

④ Estimate iteratively

Obtain the final estimates through repeated calculations that update the coefficients and weights.

⑤ Compare the results with ordinary regression

Compare the slopes, residuals, and strongly influential observations from OLS and robust estimation to see how sensitive the results are.

Suppose we observed 16 hypothetical used-car sales. Prices are shown in dollars and mileage in units of 10,000 km. These are example data for learning that include some abnormal sale prices.

We estimated the example data with OLS and with Huber robust regression. Both methods show prices falling as mileage increases, but the size of the drop differs.

OLS estimates the price decrease per 10,000 km of additional mileage at about $1,694, and Huber regression at about $1,942. This does not mean all unusual sales should be removed; it means the estimated trend can change depending on how much weight the large residuals of a few sales are given.

In the Huber results, the regression coefficient for mileage is −1,942. This means a linear relationship was estimated in which the sale price decreases by about $1,942 when mileage increases by 10,000 km. The intercept of $59,876 is the model's predicted price at zero mileage and does not represent the actual price of a new car.

Coefficients and confidence intervals

For statistical reporting, you need to check not only the point estimates of the coefficients but also how the standard errors and confidence intervals were computed. In particular, robust regression p-values should not be compared with OLS p-values as if they were computed the same way.

Weights and highly influential observations

In Huber regression, observations with large residuals may receive low weights. However, high-leverage observations whose predictor values themselves are extreme may not be controlled sufficiently by residual-based weights alone.

OLS is not always wrong, and robust regression is not always more accurate. If unusual sales are an important part of the market you are actually studying, you should consider whether giving those values low weights fits the purpose of the research.

If a price was stored as $600,000 instead of $60,000 because of a typo, you should first correct the data. Conversely, if it is a rare car that really sold at that price, decide on an analysis strategy according to the purpose of the research, such as whether to include the case or treat it as a separate market.

Check erroneous data first.

Robust regression does not automatically distinguish mistyped values or different currency units. First check what the unusual observations actually mean.

Diagnose high-leverage observations separately.

Cases that are extreme on the predictor side can strongly influence the regression line. Robust estimation based only on residuals may not be enough.

Check for nonlinear relationships and omitted variables.

If the relationship between mileage and price is curved, or if model year, brand, accident history, and so on matter, a straight-line model itself may not be appropriate.

Report the method and settings.

The Huber loss threshold, estimation algorithm, convergence criteria, and so on can differ by implementation, so state clearly which method you used.

Consider the sample and the purpose of the research.

This example is a 16-case dataset for illustration. Predicting actual sale prices or generalizing about market prices requires more representative sales data and validation.

Huber M-estimation

Applies squared loss to small residuals and a softened loss to large residuals. This is the method used in this example.

Tukey's biweight

Uses a loss function designed so that the influence of observations with very large residuals is reduced even more.

Least trimmed squares (LTS)

Reduces the influence of outliers by estimating the coefficients mainly from the subset of observations with small squared residuals.

Difference from robust standard errors

There is also an approach that leaves the OLS coefficients as they are and computes only standard errors that are robust to heteroskedasticity and the like. This is different from re-estimating the regression coefficients themselves robustly.

In a paper or analysis report, rather than writing only “robust regression was used,” name the specific method, such as Huber M-estimation. Also, presenting the characteristics of important observations and the comparison with OLS makes the rationale for your analytic choices clear.

The sentences above are an example describing coefficient results from learning data. In real research, complete the report after checking standard errors and confidence intervals, model diagnostics, and whether other variables were controlled for.

Real estate and used-car price analysis

You can check whether the general price trend is shaken by especially expensive sales or special sale conditions.

Economic and financial research

You can check how extreme returns or large swings in some observation periods affect the regression coefficients.

Environmental and weather data

You can examine whether a few very high measurements substantially change the relationship between pollutants and weather factors.

Medical and health research

You can look at how sensitive the estimates are to a few extreme cases in the measurements.

Operations and industrial data

You can estimate relationships while accounting for large measurement deviations that occur rarely in machine sensors or production processes.

Robust regression helps soften situations in which a few unusual observations change the regression coefficients too much. In the used-car sales example, we compared ordinary least squares and robust regression lines to see how unusual sale prices affect the overall trend.

But robust regression is not resistant to every kind of outlier. You need to distinguish large residuals from high leverage and consider the possibility of entry errors, omitted variables, and nonlinear relationships. So it is important to interpret the results after checking both the estimation method and what the observations mean.