Analysis of Variance (ANOVA)

Which of three teaching methods gives students better exam results? Does customer satisfaction differ by age group? Does consumers’ purchase intention change when different advertising strategies are used?

The standard method for comparing the means of several groups like this and checking whether the differences between groups are statistically significant is analysis of variance (ANOVA).

ANOVA is used in many fields, including education, psychology, business, the social sciences and public health, to analyze differences between groups or the effects of experimental conditions. It is especially useful in studies that compare the means of three or more groups or look at how two or more factors affect an outcome together.

In this article we’ll go from the basic idea of ANOVA through how it works, the main types of analysis, a worked example, post hoc tests and what to watch out for when interpreting results.

For example, suppose we compare the effects of three study methods in 90 students.

• Group A: traditional lecture-based learning
• Group B: online learning
• Group C: discussion-based learning

Thirty students are assigned to each group, and exam scores are measured after the course. Suppose the three groups’ mean scores turn out to be 72, 78 and 85 points. Looking only at the means, group C scored highest, but we don’t yet know whether this difference is actually statistically significant.

ANOVA judges whether the mean differences are statistically significant by considering both the differences between the group means and the variation of scores within each group. The important point is that ANOVA is not simply a way of ranking which group has the higher mean.

It’s an analysis that compares means, so why is it called “analysis of variance” rather than “analysis of means”? The reason is that it compares the variability in the data in order to judge differences in means. ANOVA splits the total variation in the data into two main parts.

Between-group variation

Shows how far each group’s mean is from the grand mean. The more the group means differ from one another, the larger the between-group variation.

Within-group variation

Shows how far the observations in the same group are from that group’s mean. For example, if the exam scores of students who used the same study method differ a lot from one another, the within-group variation is large. ANOVA compares the relative sizes of these two kinds of variation.

If the differences between group means are large enough compared with the differences in scores within groups, that can be evidence against the hypothesis that all the population means are equal.

The F value is the between-group mean square divided by the within-group mean square.

Here a mean square is the sum of squares for that source of variation divided by its degrees of freedom. In a one-way ANOVA, a large F value means the differences between group means are relatively large compared with the variation within groups.

Conversely, an F value close to 1 can be taken to mean that the between-group and within-group variation are of similar size. But a large F value is not automatically statistically significant. Because the F distribution depends on the degrees of freedom, you need to check the p-value calculated from the F value and the degrees of freedom as well.

ANOVA output typically includes the following measures.

Ninety students were assigned to three groups of 30. The research question is as follows.

Descriptive statistics by group

Comparing the three group means, group C, which used discussion-based learning, was highest at 85 points. Group B, which used online learning, scored 78, and group A, which used lecture-based learning, scored 72. But the differences in means alone cannot tell us about statistical significance.

One-way ANOVA results

Calculated using the group standard deviations as sample standard deviations; figures are rounded.

How do you interpret the results?

The ANOVA gave an F value of about 10.35 and a p-value below 0.001. At the usual significance level of 0.05, we can reject the null hypothesis that the population means of all three groups are equal. In other words, we can conclude that there is a statistically significant difference in mean exam scores across the three study methods.

But there is something to be careful about here.

So to find out which groups differ, you need to run a post hoc test.

But it does not tell you directly which groups differ from which. A post hoc test compares the groups with one another to find out specifically which groups differ. For example, if the ANOVA for the three study methods was significant, you could make the following comparisons.

• Group A vs. group B
• Group A vs. group C
• Group B vs. group C

Simply running several t-tests increases the chance of a Type I error as the number of comparisons grows. Post hoc tests are designed to take this multiple-comparisons problem into account. Common post hoc methods include the following.

For example, running a Tukey HSD post hoc test on the study-method example above shows which study methods differ in mean. The post hoc method should be chosen with homogeneity of variance, group sample sizes and the purpose of the comparisons in mind, however.

If you planned specific group comparisons before starting the study, planned contrasts may be more appropriate than post hoc tests.

Beyond simply comparing the means of three or more groups, it can also analyze the effects of several factors, repeated measurements, the influence of covariates and more.

One-way ANOVA

A method that analyzes differences in the means of several groups defined by one independent variable. For example, you can compare the exam scores of three groups based on the single factor of study method. There is only one independent variable, but it can have three or more levels.

Two-way ANOVA

A method that analyzes the effects of two factors on an outcome at the same time. For example, suppose we analyze how study method and class format affect exam scores. A two-way ANOVA lets you check the following three things.

• Main effect of study method
• Main effect of class format
• Interaction effect of study method and class format

Here an interaction means that the effect of one factor changes depending on the level of another factor. For example, if the effect of online learning is small in in-person classes but large in remote classes, you might consider a possible interaction.

Analysis of covariance (ANCOVA)

ANCOVA compares group means while statistically adjusting for another numeric variable that could affect the outcome. For example, when comparing the effects of different study methods, you can include students’ pretest scores as a covariate.

This lets you compare results across study methods after statistically adjusting for differences related to pretest scores. Including a covariate does not remove all differences between groups or automatically establish causation, however.

Multivariate analysis of variance (MANOVA)

MANOVA analyzes two or more dependent variables at the same time. For example, you can look at how study method affects both exam scores and satisfaction with learning. Unlike analyzing each dependent variable separately, it tests group differences while also taking into account the relationships among the outcome variables.

Multivariate analysis of covariance (MANCOVA)

MANCOVA is MANOVA with covariates added. It lets you analyze group differences in several dependent variables while statistically adjusting for related covariates. For example, when analyzing differences in exam scores and learning satisfaction by study method, you can include prior academic achievement as a covariate.

Repeated measures ANOVA

Repeated measures ANOVA compares measurements taken on the same subjects at several times or under several conditions. For example, suppose the exam scores of 30 students were measured before the course, right after it and three months later. Because scores from the same student are not independent of one another, the repeated-measures structure has to be taken into account.

Repeated measures ANOVA lets you check whether the mean score changes across measurement times.

Mixed design ANOVA

Mixed design ANOVA analyzes a between-subjects factor and a repeated-measures factor together. For example, suppose exam scores were measured before and after the course for students in two different education programs. In this case you can look at the following effects.

• Between-group difference by education program
• Time effect from before to after the course
• Interaction effect, where the before–after change differs by program

It is used especially in studies that compare how different groups change over time.

Main effect

A main effect is the effect a particular factor shows on the dependent variable when the levels of the other factor are averaged over. For example, when study method and class format are analyzed together, you can look at the score differences by study method averaged across class formats.

Interaction effect

An interaction effect shows whether the effect of one factor changes depending on the level of another factor. For example, suppose study method A works well in in-person classes but not in remote classes, while study method B shows the opposite pattern.

In this case the effect of study method depends on class format, so an interaction may be present. When an interaction effect is important, interpreting main effects from the overall means alone can miss the real differences between groups.

So it’s a good idea to look at the specific differences through simple main effects analysis, interaction plots and the like.

But they differ in the study designs they can handle and in what they compare.

For example, if you are comparing the means of two different groups, you can use an independent-samples t-test. If you want to compare three or more study methods, or look at the effects of study method and class format at the same time, ANOVA may be more appropriate.

That said, when comparing just two independent groups under the usual equal-variance assumption, Student’s t-test and a one-way ANOVA give the same significance test result. In that case the F value equals the square of the t value.

So rather than working on completely separate principles, the t-test and ANOVA can be seen as related methods within the same statistical framework for analyzing differences in means.

The main assumptions are as follows.

Independence of observations

In an ordinary ANOVA of independent groups, different observations must be independent. For example, if data from measuring the same students several times are analyzed as if they came from independent groups, the correlation between observations is ignored. In that case you should consider repeated measures ANOVA, a mixed-effects model or similar.

Normality

An ordinary ANOVA assumes that the errors in each group follow a normal distribution. In practice, it’s more important to check the distribution of the model residuals than the raw data itself. If the sample is large enough and group sizes are similar, ANOVA can work fairly robustly even with some departure from normality.

However, extreme outliers or heavily skewed distributions can affect the results.

Homogeneity of variance

An ordinary one-way ANOVA assumes the population variances are equal across groups. Homogeneity of variance can be checked with Levene’s test and similar tests. But rather than deciding mechanically on the analysis based only on the p-value of a test of equal variances, it’s better to look at the group standard deviations, sample sizes and distributions together.

If the equal-variance assumption is not appropriate, you can consider Welch’s ANOVA.

Sphericity

In repeated measures ANOVA, the sphericity assumption can be important. Sphericity is the assumption that the difference scores between the repeated conditions all have the same variance. Violations of sphericity can be a problem when there are three or more repeated levels, and you can check Mauchly’s test of sphericity and similar tests.

If the sphericity assumption is not met, corrections such as Greenhouse-Geisser or Huynh-Feldt can be applied.

The alternative should be chosen based on which assumption is violated and what the data are like.

For example, if the data from three independent groups are heavily skewed and have many extreme outliers, you might consider the Kruskal-Wallis test. Note, though, that the Kruskal-Wallis test does not test exactly the same thing as an ordinary ANOVA.

So it is not appropriate to switch to a nonparametric test automatically just because the normality assumption isn’t met.

Common effect size measures include eta squared (η²), partial eta squared (partial η²) and omega squared (ω²).

Eta squared (η²)

In a one-way ANOVA, eta squared is the proportion of the total variation explained by the grouping. For example, η² = 0.16 means about 16% of the total variation is associated with differences between groups. For the study-method example above, it can be calculated as follows.

In other words, about 19.2% of the total variation in exam scores is associated with the grouping by study method. This does not mean the causal contribution of study method is 19.2%, however.

Partial eta squared

Partial eta squared expresses the effect size relative to the variation of a particular effect plus its error term. It is often used when several effects are analyzed together, as in a two-way ANOVA or repeated measures ANOVA. In a one-way ANOVA, eta squared and partial eta squared are the same, but in analyses with several factors they can differ.

Omega squared (ω²)

Omega squared is an effect size measure adjusted for the fact that eta squared calculated from a sample can overestimate the effect in the population. Depending on the situation, it can be used to estimate the proportion explained in the population.

In this way, effect sizes help you understand not only whether an ANOVA result is statistically significant but also how large the differences between groups are.

Using the study-method example above, you could write it up like this.

This result shows that the means of the three groups are not all equal. To explain specifically which groups differ significantly, however, you need to also report the actual post hoc test results. If no post hoc test was run, you should not write that all groups differ significantly based only on the order of the means.

Education research

You can compare the effects of several study methods or analyze change before and after taking part in an education program.

Marketing and business

You can analyze differences in satisfaction by age group, purchase intention by type of advertising, the interaction between product type and price level, and more.

Social science research

You can examine differences in life satisfaction or perceptions by occupation type, education level, income group and so on.

Health and medical research

You can analyze differences in mean measurements across different intervention programs or changes across repeated measurement times.

Psychology research

You can examine differences in behavior across experimental conditions, interactions between stimulus type and individual characteristics, and more.

Product development and quality control

You can analyze how different production conditions or experimental factors affect product quality and performance. As this shows, ANOVA is used widely, from simple group comparisons to experimental studies in which several factors act together.

There are many forms, including one-way ANOVA, two-way ANOVA, ANCOVA, repeated measures ANOVA and mixed design ANOVA, and the right method should be chosen for the study design. Also, a significant ANOVA result does not mean that every pair of groups differs.

Where needed, run post hoc tests and also check the mean differences, effect sizes and data distributions. In studies with several factors in particular, it’s important to interpret main effects and interaction effects separately.