Two-Way ANOVA

How much do exam scores differ by study method? Even with the same method, are results different when studying in a quiet classroom versus a noisy setting? Is a particular study method especially effective in a certain setting?

In cases like these, the method that compares the effects of two factors on a single outcome variable together is two-way ANOVA. It tests not only each factor’s mean differences but also the interaction that appears when the two factors are combined.

Where a one-way ANOVA checks differences in group means across one factor, a two-way ANOVA separates the main effects of two factors from their interaction. What is especially important is that it checks whether the effect of one factor changes depending on the level of the other factor.

It is used in education, psychology, marketing, product experiments and other research where a combination of two conditions can affect the outcome. In this article we’ll use study method and study setting as an example to go through the principle of the analysis, interaction plots, the results table, follow-up comparisons and cautions, in that order.

Two-way ANOVA analyzes mean differences across two categorical independent variables. It tests not only how each factor affects the outcome but also whether the outcome changes depending on the combination of the two factors.

A one-way ANOVA compares only one factor, such as study method. A two-way ANOVA handles study method and study setting at the same time, checking the mean differences for each factor and also looking at whether which method works better depends on the setting.

Main effect of study method

Looks at whether scores differ among lecture, discussion and hands-on learning when the quiet and noisy settings are averaged.

Main effect of study setting

Looks at whether scores differ between the quiet and noisy settings when the three study methods are averaged.

Study method × study setting interaction

If the difference between settings varies from one study method to another, there is an interaction. For example, discussion-based learning might show a large score difference in the noisy setting while hands-on learning does not.

Study method was divided into three types (lecture, discussion and hands-on) and study setting into two (quiet and noisy). We assume a hypothetical experiment (120 students in total) in which 20 different students were assigned to each condition. The values below are example means for explaining the structure of the analysis, not actual test results.

For lecture, the mean in the noisy setting is 6 points higher; for discussion it is 14 points higher; and for hands-on it is 2 points higher. The size of the setting difference varies by method. This is a pattern that suggests an interaction, but statistical significance cannot be judged from the plot alone.

Even if the lines in an interaction plot don’t cross, an interaction can be present if their slopes differ. It must be confirmed with an F test that takes into account the sample size and score variation in each cell.

It decomposes the variation in each student’s score into differences due to study method, differences due to study setting, the interaction of the two factors, and unexplained error. For each factor, the F value is obtained by comparing its mean square with the appropriate error mean square.

Degrees of freedom for a complete, balanced 3×2 design with 20 students per cell. Test statistics can only be calculated from actual observed scores.

An actual results table reports the sum of squares (SS), degrees of freedom (df), mean square (MS), F value, p-value and effect size for each of the three effects: study method, setting and method × setting.

If the interaction is statistically significant, compare the three study methods within each study setting, or compare the quiet and noisy settings within each study method. Because several comparisons are made, consider a multiple-comparison correction such as Holm’s.

Presenting the mean differences of the simple effects with their 95% confidence intervals makes it easier to understand in what direction and how strongly the interaction actually appeared.

Independence

Each student belongs to one condition, and the observations must be independent of one another. If the same students were measured repeatedly under several conditions, the design differs from a simple independent-groups two-way ANOVA.

Normality of residuals and outliers

Examine the distribution of residuals in each condition and how extreme values affect the analysis. Normality is assessed mainly from the model residuals rather than from all the raw scores.

Homogeneity of variance

Check that the score variances in the six conditions are not very different. You can use Levene’s test and residual plots, and if the assumption is violated, consider robust methods or a suitable alternative model.

When the sample sizes of the combinations differ, interpreting the main-effect means and the interaction becomes more complicated than in a balanced design. You then need to check the calculation basis, such as Type II or Type III sums of squares, the contrast coding and how the estimated marginal means are defined. It’s important to decide in advance which hypotheses are being tested in a model that includes the interaction.

If missing data are concentrated in particular conditions, or the group variances differ greatly, the results can be distorted. Rather than simply deleting missing values, check why the data are missing and report the analyzable sample and the limitations.

Both factors in a two-way ANOVA are categorical. Having two factors does not by itself make a design a mixed design; the key distinction is whether the same participants were measured repeatedly.

First present the 3×2 factorial design and each condition’s sample size, mean and standard deviation. Then report the assumption checks and the F value, degrees of freedom, p-value and effect size for the two main effects and the interaction. If the interaction is significant, add the simple effects analysis and the corrected follow-up comparisons.

After the actual analysis, report the interaction result in a format such as F(2, 114) = [value], p = [value], partial η² = [value]. The bracketed values must be replaced with the actual results.

Two-way ANOVA is useful when two factors need to be analyzed together. The scope of causal interpretation depends, however, on whether the design is a randomized experiment or an observational study.

Comparing study method and study setting together goes beyond simply asking which method has the higher mean: it shows that the same method can give different results depending on the setting. In this hypothetical example, the mean difference between settings varied by study method. But to say the difference is actually meaningful, you need to run an interaction test on the individual data.

When reporting the results, distinguish the two main effects from the interaction, and if there is a significant interaction, use simple effects and follow-up comparisons to explain under which conditions the differences arise. Accurate interpretation also requires checking each condition’s sample size, the residuals and homogeneity of variance, and the study design.