Multivariate Analysis of Covariance (MANCOVA)

Do students taking the same class end up with different levels of comprehension and problem-solving skill? Do the two outcomes change together depending on the teaching method? And if students’ levels differed before the class started, how should we take that difference into account?

Multivariate analysis of covariance (MANCOVA) is the method that analyzes group differences in several outcome variables at once while statistically adjusting for covariates such as pretest scores.

Whereas ANOVA and ANCOVA focus on a single outcome variable, MANCOVA handles related outcome variables jointly. And unlike MANOVA, it analyzes group differences while including continuous covariates that may affect the outcomes.

It is useful when comparing several outcomes together in education, psychology, health and organizational research. In this lesson we’ll look at the concept and principle of MANCOVA, an example comparing teaching methods, interpreting multivariate tests and adjusted means, the key assumptions, how it differs from other analyses, and how to report it in a paper.

Multivariate analysis of covariance (MANCOVA) handles two or more continuous outcome variables in a single analysis and tests group differences while taking into account covariates such as pretest scores or age.

Comprehension scores and problem-solving scores are separate outcomes, but they can be related. Running a separate ANCOVA for each doesn’t directly test the overall multivariate difference and also increases the number of tests. MANCOVA first tests the group effect while taking the covariance structure of the outcome variables into account.

Also, if prior achievement differs between groups, comparing only posttest scores can reflect differences in initial levels. Statistically adjusting for covariates can partly ease this problem, but in an observational study it doesn’t remove all confounding or prove a causal relationship.

① Specify the outcome variables together

Specify posttest comprehension scores and posttest problem-solving scores as joint dependent variables.

② Specify the group factor and the covariate

Teaching method (discussion vs. lecture) is the group factor, and the pretest learning score is the covariate.

③ Estimate the adjusted group means

Compare the expected mean of each outcome variable when the covariate is held at the same reference value.

④ Run the multivariate test, then follow-up analyses

First evaluate the joint difference between groups with a multivariate test such as Pillai’s Trace, then look at individual ANCOVAs and multiple comparisons according to the research question and testing plan.

This is a hypothetical example for learning. 120 college students took part in a discussion-based class (60 students) or a lecture-based class (60 students). Suppose a pretest of learning was given before the class, and comprehension and problem-solving scores were measured after it.

In the example, we assumed that comprehension means adjusted for pretest scores were 77 points for discussion and 72 points for lecture, and problem-solving means were 74 and 69 points respectively. The difference on each measure is 5 points, but whether it is statistically significant depends on the variances and covariances in the actual data and on the sample size.

It’s good to decide in advance on one multivariate test as the main result and report it. Pillai’s Trace is a commonly used choice, but its robustness to violated assumptions isn’t guaranteed in every situation. Because there are no raw data for the example, no p-values or statistics were made up for the table.

If the multivariate test shows a group difference, you can look at covariate-adjusted ANCOVA results for comprehension and for problem solving. Significance of the multivariate test alone doesn’t let you conclude that both outcome variables are significant.

Consider a multiple-comparison correction such as Holm’s, based on the number of follow-up tests and the research hypotheses.

Box’s M test and residual diagnostics can be used as reference information, but a single test value shouldn’t decide whether the analysis is valid.

The usual common-slope MANCOVA model assumes that the relationship between the covariate and each outcome variable is similar across groups. Examining the teaching method × pretest score interaction lets you see whether this assumption is appropriate.

If the interaction turns out to be meaningful, it may be more appropriate to interpret group differences at different pretest levels from a model that includes the interaction, instead of adjusted means that force a common slope.

Check when the pretest was measured

Indiscriminately adding a variable affected by the treatment as a covariate can distort the effect of interest.

Check how the outcome variables are related

If the outcome variables are entirely different concepts, you need to explain why they are analyzed together. You should also avoid including nearly identical measures redundantly.

Distinguish multivariate significance from individual significance

Even if the joint test is significant, the difference in each individual outcome variable has to be examined separately.

Review group assignment and confounders

When teaching methods are compared without random assignment, controlling for one covariate isn’t enough to call the result a causal effect.

If you are dealing with repeated measures or longitudinal observations, consider repeated-measures MANOVA or a multivariate mixed model that fits the correlation structure between measurements.

First state the sample size per group and the definitions of the covariates and outcome variables. Then report the assumption checks and the results of the chosen multivariate test (Pillai’s Trace or another), and, if there are follow-up analyses, present the adjusted means, mean differences and confidence intervals for each variable and the multiple-comparison correction used.

In an actual paper, report the results computed by your analysis software, with exact degrees of freedom, p-values and effect sizes, instead of example figures.

What matters isn’t simply putting in several outcome variables; it’s having a research question that requires evaluating these variables jointly.

MANCOVA statistically adjusts for the influence of covariates such as pretest scores and then tests the group differences across several outcome variables all at once. If you compare comprehension and problem solving by teaching method, check the overall multivariate test and then look at each outcome’s adjusted means and follow-up analyses to see where the differences appeared.

Adjusted means don’t mean that every student actually started at the same point. You need to check the relationship between the covariate and the outcomes, the regression slopes by group, the residual distribution and the covariance structure. And controlling for covariates in observational data doesn’t prove a causal effect of the teaching method.

In the teaching-method example, we compared comprehension and problem-solving scores for discussion-based and lecture-based classes together. Because prior learning levels may differ, the key is to include them as a covariate and look at each group’s adjusted means on the same basis. Still, the mean differences given in the example alone don’t let you conclude that the effect is statistically significant.

In an actual analysis, check the joint difference in the outcome variables with a multivariate test and then, if needed, use follow-up analyses of individual outcome variables to explain which outcomes showed differences. Also check assumptions such as homogeneity of regression slopes, the covariance matrices, outliers and missing data, and don’t confuse the fact that covariates were adjusted for with causal evidence.