A significant ANOVA says that at least one mean differs. How do we find the groups responsible?
An ANOVA comparing three delivery methods produced a statistically significant result.
| Delivery method | Mean delivery time |
|---|---|
| Standard delivery | 42 hours |
| Next-day delivery | 35 hours |
| Local hub delivery | 28 hours |
We know that the equal-means model does not fit well, but ANOVA does not identify the groups that differ.
Key question
How can several groups be compared without allowing the false-positive rate to grow unchecked?
Why do repeated p-values create a problem?
One test at the 0.05 level has a 5% false-positive rate under the null. Across several tests, the probability that at least one crosses the threshold by chance becomes much larger.
| Number of comparisons | Probability of at least one false positive |
|---|---|
| 1 | 5% |
| 3 | About 14% |
| 6 | About 26% |
| 10 | About 40% |
Familywise error rate
The probability of at least one false positive across a family of comparisons is called the familywise error rate.
Which post hoc method should you use?
Consider whether every pair matters, whether one group is a control, and whether group variances are similar.
All pairs
Tukey HSD
A common choice when every pair of means matters
Unequal variances
Games–Howell
Useful when variances or sample sizes differ
Selected comparisons
Bonferroni
Simple and conservative for a small comparison set
One control
Dunnett
Compares several groups with one reference group
A common choice
After a standard one-way ANOVA, Tukey HSD is common for all pairwise comparisons. When variances or sample sizes differ substantially, Games–Howell is usually safer.
What does adjustment change?
The difference remains the same. The adjusted p-value, confidence interval, and evidence threshold change.
Adjustment can change the conclusion without changing the difference
When all three pairs are tested, raw p-values should not be read in isolation.
| Comparison | Mean difference | Raw p | Adjusted p | Conclusion |
|---|---|---|---|---|
| Standard − next-day | 7 hours | 0.048 | 0.144 | Not significant |
| Standard − local hub | 14 hours | < 0.001 | < 0.001 | Significant |
| Next-day − local hub | 7 hours | 0.048 | 0.144 | Not significant |
The tradeoff
Stronger control of false positives can make real differences harder to detect. Match the method to the planned comparisons.
How should the results be read?
Post hoc results
Tukey HSD · familywise error rate 5%
Reporting example
Post hoc comparisons showed that local hub delivery was 14 hours faster than standard delivery (adjusted 95% CI [5.5, 22.5], adjusted p < .001). The 7-hour difference from next-day delivery was not significant after adjustment (adjusted 95% CI [−1.5, 15.5], adjusted p = .144).
Change the number of comparisons and adjustment
The significance threshold, adjusted p-value, and false-positive risk update together.
Comparisons
3
Raw p-value
0.031
Adjusted p-value
0.093
Conclusion
Not significant
Per-comparison threshold: p < 0.0167
Key lesson
Adjustment is not a way to make p-values smaller. It is a way to reduce overconfidence created by repeated comparisons.
Post hoc tests answer “which groups differ?” while accounting for the fact that several comparisons were examined.