Multivariate Analysis of Variance (MANOVA)

Does students' understanding differ by teaching method? Does their problem-solving ability differ as well? If the two outcomes are related to each other, is it enough to compare each one separately?

In this way, the method that tests group differences while considering several continuous outcome variables at the same time is multivariate analysis of variance (MANOVA). Its distinguishing feature is that you can look at several outcomes in a single analysis.

Whereas one-way ANOVA compares mean differences in a single outcome variable, MANOVA also takes into account the relationships among the outcome variables. You first check the overall multivariate difference and then, where needed, look at specific differences with follow-up tests on the individual outcome variables.

It is used in studies that evaluate several outcomes at once, in fields such as education, psychology, health and business. In this article we use an example of understanding and problem solving by teaching method to cover how the analysis works, how to interpret the results table, checking assumptions, how it differs from related analyses, and how to report it in a paper.

Multivariate analysis of variance (MANOVA) is a method that compares two or more continuous outcome variables at the same time across one or more categorical grouping variables. Instead of testing each outcome variable separately, it evaluates group differences while also taking into account the correlations among the outcome variables.

Understanding and problem solving usually move together to some degree. If you run several separate ANOVAs, the number of tests increases, and it is hard to directly evaluate the pattern the two outcomes share.

MANOVA examines whether there is a difference that separates the groups through a linear combination of the outcome variables. However, a significant MANOVA result does not mean there is a difference in every outcome variable.

In the analysis data, each row holds one student, and that student's understanding and problem-solving scores are recorded in separate columns. You must not match scores from different students as if they were a pair.

Suppose 135 hypothetical college students were assigned 45 each to three teaching methods, and after the class ended their understanding and problem solving were each measured on a 100-point scale. The figures below are learning results computed from reproducible simulated data.

The hands-on group has higher means on both, and the discussion group has a higher understanding mean than the lecture group, but its difference in problem-solving mean is relatively small. Whether these differences go beyond chance sampling variation is examined with a multivariate test.

Whereas one-way ANOVA compares the between-group and within-group sums of squares, MANOVA compares matrices that hold not only the variation of each outcome variable but also the covariances between the outcome variables. Because it considers the direction that best separates the outcomes, the result is not simply the sum or average of ANOVA p-values.

These statistics do not test different questions, but they differ in how they are calculated and how sensitive they are to assumption violations. It is clearest to choose one and report it as the main criterion of the analysis.

In the example, the combined difference in understanding and problem solving by teaching method is statistically significant. However, this test alone cannot determine which teaching method differs or in which outcome variable the difference appeared.

The test results are actual calculated values for the simulated data generated earlier and do not represent effect sizes that are typically observed.

Follow-up one-way ANOVAs on the two outcomes also show differences between teaching methods. But to know exactly which pairs of groups differ, you need post hoc comparisons. When there are several follow-up tests, manage the multiple testing problem with a method such as Holm's.

To make the results easy for learners to understand, you can first show a bar chart of the means for each outcome variable and, if needed, also present a scatterplot or group centroids with the two outcome variables as axes. A graph of means alone does not let you directly check MANOVA's multivariate test.

In an actual analysis, you should also check the distribution in each group, outliers and the correlation between the two outcome variables.

Independence of observations

Observations from different participants must be independent. If the data are repeated measurements of the same students or are clustered in classes, consider a different model that fits the study design.

Multivariate normality and outliers

Examine the joint distribution of the outcome variables and multivariate outliers in each group. Severe outliers can affect covariance estimation and the test.

Homogeneity of covariance matrices

Check whether the covariance structure of the outcome variables is similar across groups. Box’s M test can be used for reference, but it is sensitive, so also consider graphs and sample size.

Relationships among outcome variables

There should be a theoretical reason to analyze the outcome variables together, and the analysis can become unstable if there is perfect redundancy or severe linear dependence.

If, in the example, you want to adjust for pre-class scores as a covariate and analyze the two learning outcomes together, consider MANCOVA rather than MANOVA.

In an actual paper, report the mean and standard deviation of each group, the chosen multivariate test statistic, degrees of freedom, p-value and, where needed, effect sizes and confidence intervals as well.

Rather than choosing MANOVA just because there are several outcome variables, first check how each outcome is measured, the research question, the relationships among the variables and the statistical assumptions required.

MANOVA does not stop at comparing understanding or problem solving one at a time; it lets you see how related outcome variables differ as a whole. It is especially helpful when a single study needs to compare several learning outcomes or psychological scales together.

However, a significant overall test does not mean that differences arise in every outcome variable. Check the study design and assumptions, and go through follow-up tests and post hoc comparisons to explain which group differences appeared in which outcomes.