Repeated Measures Analysis of Variance (ANOVA)

Does the confidence of people who took part in a training program go up right after the training? Does the change last after 4 weeks? If you measured the same people several times, is it fine to simply compare the means of three groups?

In this way, the method for analyzing mean differences in measurements obtained several times from the same participants is repeated measures analysis of variance (Repeated-Measures ANOVA). Because each person serves as their own baseline for comparison, the analysis can take into account the differences each participant already had.

Whereas one-way ANOVA is commonly used to compare the means of independent groups, repeated measures ANOVA reflects the fact that measurements from the same person are related to each other. It is useful for checking change across three or more time points or mean differences across several experimental conditions.

It is used in pre-post follow-up studies in psychology, education and health, in user experience experiments, and in lab studies with repeated measurements. In this article we use a training program example to look at how the analysis works, how to interpret the means and the F test, the sphericity assumption, post hoc comparisons, how it differs from mixed-design analysis, and how to report the results.

Repeated measures ANOVA measures one continuous dependent variable three or more times on the same participants and tests whether the mean differs by time point or condition. It differs from independent-groups ANOVA in that it evaluates the time effect while taking into account each participant's own mean level.

The same person may score low before training and higher right after training and 4 weeks later. If you analyze these data as if they were scores from different groups, you throw away the information that the values from one person are linked. Repeated measures analysis reflects this correlation in a way that fits the study design.

① Measure the same participants at every time point

The confidence scores of 36 participants are measured before training, right after training and 4 weeks later. The data can be arranged so that each row is one participant and each column is one time point.

② Find the mean differences between time points

Calculate the mean at each of the three time points, then check how far each is from the grand mean. These differences form the basis of the variation that represents the time effect.

③ Account for each participant's own starting level

Differences in mean level between participants are separated out. So the main test compares the variation remaining within the same participant with the variation due to time point.

④ Test the overall time effect with the F statistic

The F value is obtained by comparing the variation between time points with the residual variation. If the result is significant, you can say that the mean at at least one time point differs from the others, but finding which pairs of time points differ requires further comparisons.

In this hypothetical example for learning, the confidence scores (0–100) of 36 participants in a job training program were recorded three times. The values below were calculated from generated, reproducible hypothetical data and do not represent the effect of any real training program.

The mean was higher right after training than before, and 4 weeks later it was at an even higher level. However, the fact that the means look different does not tell you whether the change is statistically significant.

Even if the group mean increases, not every participant improves by the same amount. Some may increase a lot and some may barely change. Looking at individual change trajectories as well lets you see the variation that the overall mean hides.

In these hypothetical data, the effect of time point was F(2, 70) = 188.51, p < .001. Partial η² = 0.843 is an effect size calculated from the time effect and the residual variation. This test result means that the mean at at least one time point differs, not that every pair of time points differs from each other.

Sphericity is the assumption that, with three or more time points, the variances of the difference scores for all pairs of time points are equal. For example, the variances of the before–right after difference, the before–4 weeks later difference and the right after–4 weeks later difference must be the same. It does not mean that the variances of the scores at each time point must all be the same.

Sphericity is checked with Mauchly's test and similar methods. If the assumption is violated, you can correct the test's degrees of freedom with the Greenhouse–Geisser or Huynh–Feldt method, or consider a suitable alternative model. If there are only two time points, the sphericity problem does not arise.

If the overall F test is significant, you can carry out paired comparisons of before–right after, before–4 weeks later and right after–4 weeks later. Because the scores come from the same people, you must use paired-sample comparisons, and since you are comparing several times, apply a multiple comparison correction such as Holm or Bonferroni.

The t values above are comparisons on the hypothetical data for each pair of time points and are statistics before correction. In an actual report where a multiple comparison correction is needed, present adjusted p-values together with confidence intervals.

Even if a difference is statistically significant, you need to look at the score difference and the effect size together to judge whether it is a practically important change. In repeated measures ANOVA, partial η² is often reported, but depending on the sample and design, generalized η² and others may also be examined.

In this example, the mean changed from 62.4 to 75.6 points across time points, so a change of about 13.2 points on the score scale was observed. Present this score difference together with the effect size, but if participants were not randomly assigned, do not conclude that it is the causal effect of the program itself.

Independence of observations between participants

Repeated measurements from one participant are linked to each other, but in the basic model observations from different participants must be independent of each other. If the data are clustered in the same team or school, you can account for the additional cluster structure.

Normality and extreme changes

Check the error structure of the basic ANOVA and the distribution of the differences between time points. Also see whether extreme score changes drive the results.

Missing data and dropout

If some participants did not answer the post-training survey, an analysis using only complete cases can be biased. Review the reasons for and patterns of dropout and, if needed, consider a suitable method such as a mixed-effects model.

Sphericity and time intervals

With three or more time points, check the sphericity assumption. If the time points are irregular or the change trajectory itself is the research question, a different longitudinal model may be more suitable.

Because this example observed one group three times, repeated measures ANOVA is a suitable starting point. If you were comparing the difference in change over time between a group that received the training and a group that did not, you could consider a mixed-design ANOVA.

In reporting results, first present the number of participants and time points, and the mean and standard deviation at each time point. Next, report the sphericity test and whether a correction was applied if needed, the F value, degrees of freedom, p-value and effect size. Finally, describe the post hoc comparison results with a multiple comparison correction applied, and the specific mean differences.

The key is a design that measures the same participants under different conditions. For data with a time order, consider learning effects, order effects and maturation effects in addition to simple time-point differences.

In the training program example, the mean confidence scores of the 36 participants were 62.4 before training, 69.3 right after and 75.6 four weeks later. Because the data come from measuring the same participants several times, the observations must not be treated as independent groups. Repeated measures ANOVA evaluates mean differences across time points while taking into account each participant's own score level. When explaining the results, present the F test, the effect size and any necessary post hoc comparisons along with the change in means.

When applying the analysis, look together at the number of time points, independence between participants, sphericity, and missing data and dropout. Even if the overall time effect is significant, you need to check separately which time points differ, and the mere fact that change appeared over time does not let you conclude a causal effect of the training program. To compare patterns of change between groups as well, a mixed-design ANOVA may be more suitable; if the number of measurements or the correlation structure is complex, a mixed-effects model may be.