Mixed-Design Analysis of Variance (ANOVA)

How much does the confidence of people who took part in a training program rise after the training? Does the change last over time? Is the size of the change different compared with people who did not receive the training?

As in these questions, a method that compares different groups while also analyzing repeated measurements of the same participants is called mixed-design analysis of variance (mixed-design ANOVA). One factor varies between participants, and the other is repeated within the same participant.

Independent-groups ANOVA focuses on differences between groups, and repeated-measures ANOVA focuses on differences over time within the same participants. Mixed-design ANOVA handles both at once and, in particular, tests whether the change over time depends on the group as an interaction effect.

It can be used to compare the effects of training programs, follow up treatment and control groups, evaluate products before and after use, and more. In this article we use a training group and a comparison group to look at how the analysis works, the three effects, how to read the graph and results table, checking assumptions, post hoc comparisons, how it differs from other analyses, and how to report it.

Mixed-design analysis of variance is an ANOVA that includes a between-subjects factor and a within-subjects factor. For example, whether someone received training is a group factor fixed for each participant, while before, right after, and 4 weeks after training is a time factor measured repeatedly on the same participant.

Even if the training group scores higher right after training, the interpretation changes if the groups already differed at the start. Conversely, if the groups were similar at first but only one group improved substantially after training, the pattern of change itself becomes an important result. Mixed-design ANOVA evaluates these group differences and changes over time simultaneously.

① Define the group factor

Divide participants into a training group and a comparison group. Each participant belongs to only one group.

② Specify the repeated time points

Measure each participant's confidence before training, right after training, and 4 weeks later.

③ Separate the three effects

The group main effect is the difference between groups averaged over all time points, the time main effect is the difference between time points averaged over all groups, and the interaction means whether the change over time differs by group.

④ Run F tests that match the error structure

Account for the correlation among repeated measurements within participants and compute F statistics using the appropriate error term for each effect.

This is a hypothetical example for learning. Thirty people who took part in a training program and 30 who did not were surveyed before, right after, and 4 weeks after the training. The means below are examples for explaining the concepts, not results tested on real observed data.

In this case, the difference between the two groups is −1 point before training, +9 points right after, and +14 points 4 weeks later. Because the size of the group difference changes over time, you can see visually that the training group may have improved more than the comparison group.

In an actual analysis, you should present each effect's F value, degrees of freedom, p-value, and effect size. Because only means are shown here, we have not attached made-up test statistics or p-values. In particular, if the interaction is significant, prioritize follow-up comparisons of the change over time within each group.

If sphericity is violated, you can consider a Greenhouse–Geisser or Huynh–Feldt correction. If there is a lot of missing data or the measurement intervals are uneven, a linear mixed-effects model may be more appropriate.

If the group × time interaction is significant, analyze the simple effects. You can check whether scores changed across time points in the training group and whether there was any change in the comparison group, or compare the two groups at each time point.

When comparing several pairs, apply a multiple-comparison correction such as Holm or Bonferroni and check the adjusted p-values and confidence intervals.

In mixed-design ANOVA you can report effect sizes such as partial η² for each effect. But a single value alone can hardly explain how important the actual change is. It is better to also present confidence intervals for the score differences and the changes.

In the hypothetical example, the training group's increase from before training to 4 weeks later is 18 points, and the comparison group's is 3 points. The between-group difference in change is 15 points, but whether this is more than a chance difference requires estimation and testing with observed data.

Check whether group assignment was random

If the training and comparison groups were not randomly assigned, preexisting differences and confounders may remain. A significant interaction cannot be taken directly as a causal effect.

Keep participants independent of each other

Repeated measurements within the same participant are linked, but observations across participants must be independent. Data clustered by class or organization must account for the clustering.

Check for missing data and dropout

If participants who did not respond at 4 weeks are concentrated in one group, the results can be distorted.

Choose the analysis according to assumption violations

Review sphericity, outliers, and the variance structure, and use an appropriate correction or a mixed-effects model if needed.

Mixed-design ANOVA can be understood as repeated-measures ANOVA with a group factor added. In practice, depending on the time intervals, missing data, and group structure, a more flexible longitudinal model may be needed.

First describe the study design, the sample size of each group, and the time points, and present means and standard deviations. Then report assumption checks such as sphericity and whether corrections were applied, and the F value, degrees of freedom, p-value, and effect size for the three effects; if the interaction is significant, describe the simple effects and corrected post hoc comparisons.

It fits when change over time in different groups is the center of the research question. However, you don't use a mixed design just because there are repeated measurements; the group division and measurement structure must match the actual study design.

In the training program example, the training group's mean confidence rose from 61 to 79 points, while the comparison group went from 62 to 65 points. Rather than the simple difference in final scores, the important question is how different the two groups' amounts of change were from start to finish. The interaction test in mixed-design ANOVA evaluates exactly this difference.

When interpreting the results, distinguish the group and time main effects from the interaction, and if the interaction is significant, explain what change occurred using simple effects and post hoc comparisons. You should also check sphericity, missing data, and how groups were assigned, and it is important not to confuse a statistical association with a causal effect.