Analysis of Covariance (ANCOVA)

Which teaching method raises exam scores more? Even if students in the hands-on class had the highest final scores, what if many of them were strong students from the start? Would the same difference remain after taking their pre-class level into account?

Analysis of covariance (ANCOVA) is a method that compares mean differences between groups while also taking into account a continuous variable that can affect the outcome. It goes one step beyond a simple comparison of means and looks at group differences on a common baseline.

One-way ANOVA compares group means directly, whereas ANCOVA adjusts for the influence of a covariate related to the outcome, such as a pretest score. Statistically controlling for a covariate, however, does not remove every bias that can arise in a study.

It is used in evaluating educational effects, health research, psychology experiments, marketing research, and more. In this article we use three teaching methods and pretest scores as an example to cover adjusted means, homogeneity of regression slopes, interpreting the results table, follow-up comparisons, and points to watch out for.

Analysis of covariance (ANCOVA) is a method that compares mean differences between groups while statistically adjusting for the influence of a continuous covariate. Beyond each group’s raw mean, it compares the adjusted mean, the mean when the covariate is set to the same reference value.

Even if students in the hands-on class have higher final scores, they may have been higher achievers from the start. If you don’t take pretest scores into account, you can mistake this difference for an effect of the teaching method.

ANCOVA estimates the relationship between pretest and final scores, then compares the final scores you would expect if every group had the same pretest score. Statistical adjustment alone, however, does not remove every bias that arose from how groups were assigned.

As a covariate, it is common to use a pre-existing characteristic that is related to the outcome and measured before the teaching method could have affected it. Carelessly controlling for a variable that was influenced and changed by instruction can distort the interpretation.

Suppose 120 hypothetical college students were assigned, 40 each, to lecture, discussion, and hands-on classes. Pretest scores were recorded before the classes started and final exam scores after they ended. The values below are hypothetical summaries for illustration, not the results of an actual analysis used to compute test statistics.

When the three groups’ pretest means differ (60, 64, and 68 points), there are limits to comparing the final means of 70, 78, and 84 points as they are. ANCOVA usually sets a common reference, such as the covariate mean of the whole sample, and computes each group’s predicted final score at that value.

The most basic model uses the final score as the dependent variable and enters teaching method and pretest score together. You can think of it as final score = intercept + teaching method effect + regression effect of pretest score + error.

The effect of each teaching method shows how much final scores differ at the same pretest level. For this, the assumption that the slope on pretest score is the same in all three groups is important.

Basic ANCOVA assumes that in every group the slope by which the final score rises for each 1-point increase in pretest score is the same. You can first examine the teaching method × pretest score interaction to see whether the slopes differ substantially.

If the slopes differ, it may be hard to speak of a constant difference in adjusted means between groups at every pretest score. In that case, change the interpretation, for example by using a regression model that includes the interaction or by comparing groups at specific pretest levels.

In the results table, focus on the F value, degrees of freedom, p-value, and effect size for teaching method, controlling for the covariate. Also present the regression effect of the pretest score and each group’s adjusted mean with its confidence interval.

Because F values, p-values, and adjusted means cannot be computed from hypothetical group means alone, no arbitrary test values are given.

If the teaching method effect in ANCOVA is significant, you can compare the adjusted mean differences for lecture–discussion, lecture–hands-on, and discussion–hands-on. Present each difference’s 95% confidence interval along with multiple-comparison-adjusted results.

Independence, normal residuals, equal variances

Check whether the observations are independent, whether the distribution of model residuals is not too asymmetric, and whether residual variances differ greatly between groups. Also check whether outliers have an excessive influence on the regression slope.

Linear relationship between covariate and outcome

Check whether pretest and final scores have a roughly linear relationship.

Overlap in covariate ranges across groups

If you compare groups whose pretest score distributions barely overlap, adjusted means may be extrapolated into ranges that were never observed.

Measuring and choosing the covariate

If the pretest score has large measurement error, or if a variable affected by instruction is used as the covariate, interpretation can run into problems.

ANCOVA is less a separate system of calculation than one form of a linear model that includes a grouping variable and a covariate. It is also not the same as simply calculating the difference before and after instruction.

First present the sample size of each group and descriptive statistics for pretest and posttest scores, then report the definition of the covariate and the results of the homogeneity-of-slopes check. After that, describe in order the adjusted means, 95% confidence intervals, the F value, p-value, and effect size for the group effect, and any post hoc comparisons needed.

Once you have actual results, report them like F(2, [error df]) = [value], p = [value], partial η² = [value]. The sample summaries in this example alone cannot determine the test values.

In randomized studies, adjusting for baseline values can improve precision; in observational studies, unmeasured confounders can remain even after controlling for covariates.

When you compare final scores by teaching method and the pretest scores differ, simple means alone make it hard to explain differences between teaching approaches. ANCOVA estimates each teaching method’s adjusted mean by taking into account the relationship between pretest and final scores. So you need to check both how the group means and adjusted means differ and what the basis for comparison is.

Before interpreting, look at the relationship between the covariate and the outcome, the homogeneity of regression slopes, the residual assumptions, and whether the covariate distributions of the groups overlap enough. If the difference in adjusted means between groups is significant, you can proceed with follow-up comparisons, but with observational data, covariate adjustment alone does not establish a causal effect.