Interactive·

Multiple Regression & Multicollinearity

How can we separate the effect of advertising from store size and customer traffic?

High-revenue stores may spend more on advertising, operate larger locations, and receive more visitors. A regression with advertising alone can mix the advertising relationship with store size and customer traffic.

Multiple regression includes several predictors at once to estimate the relationship between one predictor and the outcome while holding the included predictors constant.

Key question

If store size and customer traffic are the same, how much does average revenue differ when advertising spending is $1,000 higher?

Concept

What does multiple regression separate?

Several predictors help separate relationships that were previously mixed together

Multiple regression equation

ŷ = b₀ + b₁x₁ + b₂x₂ + … + bₖxₖ

Revenue model

Predicted revenue ($1,000s) = 20 + 4.5×advertising ($1,000s) + 1.2×visitors (hundreds)

The advertising coefficient of 4.5 means that, among stores with the same visitor count, $1,000 more in advertising is associated with $4,500 higher revenue on average. It does not mean every store increases by exactly that amount.

Holding the rest fixed

control

Each coefficient is the effect of one predictor with the others held still.

Multicollinearity

overlap

When two predictors overlap, the shared part belongs to neither alone.

Unstable coefficients

VIF

The estimate stays near the truth while its interval balloons.

Why It Matters

How are coefficients interpreted?

Each coefficient is a conditional relationship holding the other included predictors fixed

Term	Interpretation
b₁	average difference associated with a one-unit increase in x₁ when the other predictors are held constant
b₀	calculated baseline when every predictor equals zero
R²	proportion of outcome variation explained jointly by all predictors in the sample

Adjustment is not proof of causation

Omitted variables, measurement error, and reverse causality can remain. A coefficient should not automatically be read as a causal effect.

How It Works

Why does multicollinearity occur?

When predictors contain nearly the same information, their individual effects are difficult to separate

Advertising spending and ad impressions, for example, may be so similar that the model cannot stably decide how much of the shared relationship belongs to each predictor. This is multicollinearity.

•
Coefficient signs and magnitudes may change substantially across samples.
•
Standard errors can grow, making individual p-values larger.
•
Overall prediction may remain reasonable even while individual coefficients are unstable.
•
A high correlation is not by itself a reason to automatically delete every variable.

Example

How should VIF be used?

VIF describes how much overlapping predictor information inflates coefficient uncertainty

VIF	Common interpretation
1	little linear overlap with the other predictors
1–5	some overlap; interpret in context
5 or more	coefficient instability deserves closer examination
10 or more	a conventional warning level, not an absolute rule

Possible responses

Combine redundant predictors, remove one when substantively justified, or consider Ridge and Lasso when prediction is the main goal.

Interactive

Change correlation among predictors

Change the correlation between predictors

Begin with low correlation and inspect the coefficients and standard errors. Then increase the correlation and compare the VIF and coefficient instability.

VIF

1.04

β₁

2.13

SE(β₁)

0.20

β₂

1.39

Correlation between predictors0.20Small sample change0.5

As predictors become more correlated, the model has more difficulty separating their individual effects. Standard errors rise even when overall prediction changes little.

What to watch

Prediction may change little as correlation rises, while the standard errors of individual coefficients can increase substantially.

Key takeaways

•
Multiple regression estimates conditional relationships while holding the other included predictors constant.
•
Coefficients must be interpreted in the context of the included variables and their measurement units.
•
Multicollinearity usually harms the stability of individual coefficients before it harms overall prediction.
•
Do not remove variables from VIF alone; consider the research goal and the meaning of each variable.

Next: Regression diagnostics, R², and residuals