Regression Analysis

How much higher do exam scores get as study time increases? How are customer satisfaction and length of service use related to repurchase intent? If you spend more on advertising, how much do sales change?

The go-to statistical method for analyzing how a particular outcome changes with one or more variables like this is regression analysis.

Regression analysis expresses the relationship between variables as a mathematical model and estimates how the outcome changes on average when a predictor changes. With it, you can check the direction and size of relationships between variables or predict outcomes for new data.

It is used in many fields, including business, economics, education, the social sciences and public health, to test research hypotheses and understand relationships in data. In this article we'll cover the basic ideas of regression, what regression coefficients and the coefficient of determination mean, a worked example, the main types of regression, and what to watch out for when interpreting the results.

An independent variable is one used to explain or predict the outcome, and the dependent variable is the outcome you are trying to explain or predict.

For example, suppose you surveyed 100 students on their average daily study time and exam scores. You could treat study time as the independent variable and exam score as the dependent variable.

Regression lets you check whether students who study longer tend to score higher. Going further, it lets you estimate how many points the expected exam score changes on average for each additional hour of study.

However, calling something an independent variable in regression does not mean it causes the dependent variable. To claim causation, you need to consider additional conditions such as the study design, confounders and the time order of events.

How does regression work?

The most basic form of regression is simple linear regression.

Simple linear regression analyzes the linear relationship between one independent variable and one continuous dependent variable. The general regression equation is:

Here is what each symbol means.

For example, suppose the regression equation for study time and exam score was estimated as follows.

In this equation the intercept is 55 and the coefficient for study time is 5. This means there is a linear relationship in which the expected exam score rises by 5 points on average for each additional hour of study. A student who studies 4 hours has an expected exam score of 55 + 5 × 4 = 75 points.

But this is a predicted value from the regression model. It doesn't mean every student who studies 4 hours actually scores 75. An individual student's exam score can be affected by many factors not in the model, such as learning ability, learning environment and prior knowledge.

Regression coefficient (B)

The regression coefficient is the key measure of the relationship between the independent and dependent variables. For example, a coefficient of 5 for study time means the expected exam score increases by 5 points on average for each additional hour of study.

Conversely, a coefficient of -3 means that the expected value of the dependent variable decreases by 3 units on average for each one-unit increase in the independent variable. In multiple regression, each variable's coefficient is interpreted holding the other independent variables constant.

p-value

The p-value is the probability, under the null hypothesis that the coefficient is 0, of getting a test statistic as extreme as the one observed or more extreme. With the usual significance level of 0.05, a p-value below 0.05 is taken to indicate a statistically significant relationship.

But a small p-value does not mean the relationship is strong or practically important. Consider the size of the coefficient, its confidence interval and the research context as well.

Coefficient of determination (R²)

The coefficient of determination shows how much of the variation in the dependent variable the regression model explains. For example, R² = 0.64 means that, in an ordinary least squares linear regression with an intercept, about 64% of the sample variation in the dependent variable is explained by the model.

A high R², however, does not mean the model has strong causal explanatory power or that it will predict well on new data.

The research question is:

The variables in the analysis are average daily study time and final exam score.

Regression results

Suppose the model's coefficient of determination came out as R² = 0.376.

How do you interpret the results?

The coefficient for study time was 5.00. This means the expected exam score rises by 5 points on average for each additional hour of study. The p-value for study time is below 0.001, so you can say there is a statistically significant positive linear relationship.

R² is about 0.376, so the regression model using study time explains about 37.6% of the sample variation in exam scores.

Simple linear regression

Simple linear regression uses one independent variable to analyze its linear relationship with the dependent variable, for example, explaining exam scores with study time alone. It is useful for getting a handle on the basic relationship between two variables or building a simple prediction model.

Multiple linear regression

Multiple linear regression uses two or more independent variables together to explain the dependent variable. For example, you might consider study time, attendance rate and pretest score together as variables that could affect exam scores. The regression equation can then be written as Y = β₀ + β₁X₁ + β₂X₂ + β₃X₃ + ε.

Multiple regression has the advantage of analyzing the relationships between several variables and the outcome at once. In particular, it can estimate the relationship for one variable while statistically adjusting for the other independent variables.

Including control variables, however, does not remove all confounding. If there are unmeasured variables or the model is misspecified, the results can be biased.

How do you interpret multiple regression results?

Suppose you are analyzing the factors that affect service satisfaction among 300 customers.

The dependent variable is overall service satisfaction, and the independent variables are service quality, price satisfaction and ease of use. Suppose the results were as follows.

All three variables showed a statistically significant positive relationship with overall service satisfaction. For example, a coefficient of 0.42 for service quality means that, holding price satisfaction and ease of use constant, each 1-point increase in the service quality score is associated with an average 0.42-point increase in expected overall satisfaction.

Be careful, though, about declaring which variable matters most from coefficient size alone. If the variables differ in measurement units and variance, unstandardized coefficients can be hard to compare directly. In that case you can also look at standardized coefficients, confidence intervals and the theoretical importance of each variable.

Many different regression models can be used depending on the structure of the data and the research goal.

All of these methods fall under regression, but they differ in the relationships they analyze and the data structures they handle.

For example, suppose income doesn't keep rising with age but rises up to a certain age and then declines. A single straight line may not capture such a relationship well. Polynomial regression adds squared or cubed terms of the independent variable to represent a curved relationship.

A typical second-order polynomial regression equation is Y = β₀ + β₁X + β₂X² + ε, where X² is the squared term of the independent variable. Polynomial regression represents a curved relationship, but it can still be considered a model that is linear in its coefficients.

However, if the polynomial degree is set too high, the model can overfit, fitting the training data too closely, so the model's complexity needs to be kept in check.

But an average relationship doesn't apply equally to every observation. Suppose, for example, you are analyzing the relationship between education and wages. How much wages change on average for each additional year of education matters, but you may also want to know whether the relationship differs for low-wage and high-wage workers.

In that case you can use quantile regression. Quantile regression estimates relationships for the conditional median or a specific quantile of the dependent variable. For example, you could analyze quantiles such as these:

Quantile regression lets you see how the relationship between education and wages differs between low and high conditional wage quantiles. However, the fact that coefficients differ across quantiles should not be read to mean that the same change would occur for an individual moving from one quantile of the income distribution to another.

In real research, though, data are often grouped into clusters, or the same subjects are measured several times.

Suppose, for example, you are studying the academic achievement of students across several schools. Students in the same school may share a similar learning environment and the same teachers, so they may not be fully independent of one another. For data like these, you can consider a mixed-effects model.

A mixed-effects model includes both fixed effects and random effects to analyze differences between groups or the correlation structure from repeated measurements. For example, while analyzing the relationship between students' study time and exam scores, the model can reflect the fact that average levels differ from school to school.

If needed, you can also analyze whether the relationship between study time and exam scores differs from school to school.

Ordinary regression vs. mixed-effects models

Mixed-effects models are closely related to multilevel models and hierarchical linear models (HLM). Not every mixed-effects model is the same kind of multilevel model, however, and how the model is built can vary with the data structure.

For example, a correlation of 0.6 between study time and exam scores tells you there is a positive linear relationship between them. Regression, on the other hand, estimates how many points the expected exam score changes on average for each additional hour of study.

Multiple regression can also include several independent variables at once to analyze the conditional relationship between a particular variable and the outcome. Rather than substitutes, the two methods are tools you can use together depending on the research goal.

Linearity

Check whether the relationship between the predictors and the conditional mean of the dependent variable is properly captured by the regression equation you specified. For example, if the true relationship is curved and you use a straight-line model, you may miss an important pattern. In that case you can consider polynomial terms or other nonlinear structures.

Independence of errors

Check whether the errors are independent of each other. In time series or repeated-measures data in particular, the errors can be correlated. Ignoring that structure can make the standard errors and test results inaccurate.

Homoscedasticity

This assumption is that the variance of the errors is constant across values of the predictors. For example, if spending becomes more variable as income rises, there may be heteroscedasticity. When heteroscedasticity is present, you can consider robust standard errors or similar remedies.

Normality of residuals

In ordinary linear regression, the assumption of normally distributed errors can matter if you want small-sample t tests or F tests to rest on exact distributions. But the residuals not being normally distributed doesn't automatically mean the coefficient estimates are wrong.

The impact depends on the sample size, the purpose of the analysis and the error structure, so look at residual plots and statistical diagnostics together.

Multicollinearity

In multiple regression, check whether the independent variables are too strongly related to one another. For example, if service quality and an overall service rating carry almost the same information, it can be hard to estimate each coefficient stably. You can check for this problem with the variance inflation factor (VIF) and similar measures.

High multicollinearity inflates the standard errors of the coefficients, which can make the relationships for individual variables hard to interpret.

Some observations in particular can change the coefficients a lot or shift the regression line. Common diagnostics include the following.

For example, if one customer's purchase amount is far larger than everyone else's, that observation can have a large effect on the regression results. But finding an outlier is no reason to delete it automatically. First check whether it is a data entry error, a genuinely possible observation, or an important case relevant to the research goal.

If needed, you can run a sensitivity analysis to see how much the results change with and without that observation.

Statistical significance does not imply causation.

A positive relationship between advertising spending and sales doesn't let you conclude that increasing ad spending will definitely raise sales. The company may have increased ad spending in periods when sales were expected to be high anyway, and other factors such as seasonality or economic conditions may have played a role.

To analyze causal relationships, you need to consider experimental designs or appropriate causal inference methods as well.

A large coefficient doesn't necessarily mean an important variable.

When independent variables are measured in different units, the sizes of their unstandardized coefficients are hard to compare directly. For example, the coefficient on income will be a different number depending on whether income is measured in dollars or in thousands of dollars. So when judging how important a variable is, consider the measurement units, standardized effect sizes and the research goal together.

A high R² doesn't make a good model.

Even with a high R², the model may be overfit or predict poorly on new data. If prediction is the goal, you need to evaluate predictive performance on data not used for fitting. And even a low R² can come with the discovery of an important relationship, depending on the research goal.

Adding more predictors isn't always better.

Adding predictors indiscriminately makes the model complex and can cause multicollinearity or overfitting. So it's best to choose variables based on your research hypotheses, theoretical grounds and the characteristics of the data.

Regression line plot

In simple linear regression, you can draw the estimated regression line on a scatterplot. This shows at a glance which direction the dependent variable moves as the independent variable increases.

Coefficient plot

In multiple regression, you can plot each independent variable's coefficient and confidence interval. This makes it easy to compare which variables show positive or negative relationships and how uncertain the estimates are. Be careful, though, when comparing unstandardized coefficients for variables with different measurement units on a single axis.

Residual plot

A residual plot visualizes the differences between the model's predicted values and the actual observations. It helps you spot possible problems with the model, such as nonlinear patterns, heteroscedasticity and outliers.

Using the study time and exam score example from earlier, you could write it up like this:

For multiple regression, it's good to report each independent variable's coefficient and confidence interval along with the overall model fit. Depending on the research goal, you can also present standardized coefficients, adjusted R² and multicollinearity diagnostics.

Business and marketing

Analyze relationships such as advertising spending and sales, service quality and customer satisfaction, or price and purchase amount.

Education research

Check how study time, attendance rate, prior achievement and so on relate to exam scores.

Social science research

Analyze relationships such as income and life satisfaction, work environment and organizational commitment, or social support and psychological well-being.

Health and medical research

Analyze relationships between exercise time and health indicators, lifestyle habits and body measurements, or treatment-related factors and continuous outcomes.

Finance and economics

Analyze relationships among economic variables such as income and consumption, interest rates and investment, or company characteristics and financial performance.

HR and organizational research

Check how the work environment, training hours, work experience and so on relate to job performance or satisfaction. In these ways, regression is widely used to put numbers on the relationships between variables and outcomes, test research hypotheses and build prediction models.

There are many methods, including simple and multiple linear regression, polynomial regression, quantile regression and mixed-effects models, and you should choose the model that fits the data structure and research goal. When interpreting regression results, it's important to look not only at the coefficients and p-values but also at confidence intervals, R² and residual diagnostics.

And a statistically significant relationship should not be taken to mean that a causal relationship necessarily exists.