Choosing a test to compare groups
From a simple two-group difference to comparisons across many conditions, find the test that fits your study design and your data.
When you want to know whether groups differ, a t-test or an ANOVA is usually the first thing that comes to mind. But the right test depends on whether you're comparing two groups or three, and on whether you measured the same people more than once.
This guide won't walk you through the arithmetic of each test or how to read its output. Instead, it lays out how to decide which test fits which question. How each method works, and how to interpret it, is covered in that method's own chapter.
Check four things before you pick a test
Start with your research question and the structure of your data, not with test names.
Don't let the p-value from a normality test decide on its own between a parametric and a nonparametric test. Settle the study design and level of measurement first, then weigh the shape of the distribution, the sample size, any outliers, and what you're trying to learn.
A decision map for comparing groups
Follow the number of groups and how they were measured, and the list of candidates gets short fast.
These paths are the most common starting points, not the last word. A real study may need something different depending on the type of outcome, the pattern of missing data, how participants were recruited, or how strongly the repeated measurements are correlated.
Comparing one group or two
First decide which case you have: a comparison with a reference value, two independent groups, or paired measurements.
One group's mean vs. a reference value
Say you want to know whether your students' average test score differs from a benchmark of 70.
Two separate groups
Say you're comparing the grades of two different sets of students, one taught in Section A and the other in Section B. To compare means on a continuous outcome, reach for the independent-samples t-test; if a rank-based approach makes more sense, consider the Mann–Whitney U test.
The same people measured twice, or matched pairs
If you scored the same students before and after a training program, each student's two scores belong together, and the analysis has to keep them paired. The paired-samples t-test works with the mean of those within-pair differences; the Wilcoxon signed-rank test works with their ranks.
If you'd like to weigh the evidence for competing hypotheses from a Bayesian point of view, instead of relying only on a classical t-test, a Bayesian t-test is worth a look.
Comparing three or more groups
The key question: are the groups independent, or did you measure the same subjects several times?
| Study design | Typical method | What decides it |
|---|---|---|
| Means of three or more independent groups | One-way ANOVA | A single factor defines the groups |
| Rank-based comparison of three or more independent groups | Kruskal–Wallis | Compares the ranks of continuous or ordinal values |
| The same subjects under three or more conditions | Repeated-measures ANOVA | Mean differences across time points or conditions |
| The same subjects under three or more conditions, compared by ranks | Friedman | Uses each subject's ranking of the conditions |
| Evaluating mean differences from a Bayesian perspective | Bayesian ANOVA | Takes priors and the evidence for each model into account |
A significant overall test doesn't tell you which pairs of groups differ. Follow it with post hoc comparisons that suit your design, plus a correction for multiple comparisons. Each method's own chapter shows how.
Complex designs call for different choices
Cases where the number of groups alone won't settle it.
Two factors: two-way ANOVA
For example, you can look at how teaching method (online vs. in person) and grade level (9th, 10th, 11th) together affect test scores, and check whether the two factors interact.
Accounting for pretest scores and the like: ANCOVA
Compare posttest scores between students who joined a program and those who didn't, while statistically adjusting for their pretest scores. Keep in mind that adding a covariate doesn't turn a nonrandomized study into a causal analysis.
Group differences and change over time: mixed-design ANOVA
Measure two separate groups before and after training, then test whether the two groups changed in different ways.
Several outcome variables: MANOVA and MANCOVA
These compare groups on several related continuous outcomes at once, such as customer satisfaction and loyalty (intent to return). If you also need to adjust for a covariate, look at MANCOVA.
Suppose you measured test scores before and after a tutoring program, both for students who took part and for students who didn't. Running two separate independent-samples t-tests is tempting, but a mixed-design analysis that tests the group × time interaction directly is usually a better match for the question you're actually asking.
What should you check before running the analysis?
Picking the test isn't the last step.
| What to check | Why it matters |
|---|---|
| Independence and pairing | Treating different people as if they were the same people, or repeated measurements as if they were independent samples, can change your results. |
| Level of measurement | Scores, ordinal ratings, and multiple outcome variables each call for different methods. |
| Distribution and outliers | Especially with small samples, check how extreme values and heavy skew affect the means and the test. |
| Variance and covariance assumptions | Depending on the method, check equal variances, sphericity for repeated measures, or the covariance structure. |
| Missing data and sample size | Missing values in repeated measures and small groups both limit what the analysis can do and how far you can trust it. |
| Post hoc plan | If you're comparing several groups, decide in advance which pairs you'll compare and how you'll correct for multiple testing. |
And nonparametric tests aren't assumption-free. Each one still has requirements of its own, such as independent observations, the right paired structure, or similarly shaped distributions, and you need to check them.
All 16 group-comparison methods at a glance
One line each here; the details are on each method's own page.
Two groups or a reference value
Three or more groups and repeated measures
Covariates, complex designs, and multiple outcomes
Note: These 16 are the core group-comparison methods, not an exhaustive list. Related approaches, such as design of experiments (DOE), linear mixed models, and Welch's t-test, may also be worth considering, depending on your goals.
Pin down your study design first, and choosing a test gets much easier.
Comparing groups isn't about picking the most famous test. It's about understanding exactly how the things you're comparing are structured. Check, in order: ① how many groups there are, ② whether they're independent or paired, ③ how many outcome variables you have and what type they are, and ④ the assumptions of each candidate method. Two studies can ask what sounds like the same question and still need different analyses if their designs differ. Once you've chosen a method, turn to its own chapter for the assumptions, how to interpret the results, and post hoc comparisons.