Group Comparison

Almost every experiment ends with the same question: are these groups actually different, or did the numbers just fall that way? Version B converted a little better. Plan C's users spend a little more. The new process shaved a few seconds. Each of those is a group comparison — and the entire family of methods exists to tell a real difference from a lucky sample.

The catch is that "compare the groups" is not one test. It's a decision tree. Two groups or many? Independent or paired? Normal or skewed? Each branch leads to a different method, and choosing the wrong branch produces a p-value that looks authoritative and means nothing. This guide walks the whole tree.

The Three Questions That Pick the Test

Before any test runs, three questions determine which one is valid. Answer them in order and the choice makes itself.

1. 1How many groups? Two points you toward a t-test; three or more toward ANOVA.
2. 2Are the groups independent or paired? Before-and-after on the same people is paired; separate cohorts are independent.
3. 3Is the data roughly normal, or skewed with outliers? Normal supports the classic parametric tests; skewed points to nonparametric alternatives.

Two Groups: The t-test

With exactly two groups, the t-test weighs the gap between the means against the spread within each group. A big difference between two noisy groups can be nothing; a small difference between two tight groups can be real.

Use the paired test whenever the two measurements come from the same people or items — it removes person-to-person variation and is far more powerful. When variances differ, Welch's correction is the safe default. The dedicated t-test guide covers the assumptions and reading the result in depth.

Three or More Groups: ANOVA

The instinct with several groups is to t-test every pair. Don't — each test carries a false-positive risk, and six comparisons stack them up. ANOVA (Analysis of Variance) tests all the groups at once by comparing the variation between group means to the variation within groups.

When the Data Isn't Normal: Nonparametric Tests

The t-test and ANOVA assume roughly normal data. When it's heavily skewed, has strong outliers, is measured on a rating scale, or the sample is tiny, the mean stops being a fair summary. Rank-based tests compare the order of values instead, and stay honest where the mean can't.

Each swaps a mean-based comparison for a rank-based one, answering the same question without the normality assumption. They're the honest fallback — but when the data really is normal, the parametric test is more powerful, so don't reach for ranks by default. See the nonparametric tests guide for the trade-offs.

Significance Is Not Size

Every comparison returns a p-value, but a p-value only answers 'is the difference real?' It says nothing about 'does it matter?' With a large enough sample, a difference too small to care about turns "significant."

• Effect size (Cohen's d, eta-squared) reports how big the difference actually is
• A confidence interval gives the plausible range for the true gap
• Report both alongside the p-value — significance without size is half an answer

The Multiple-Comparison Trap

Every test carries a chance of a false positive. Run twenty comparisons at the 0.05 threshold and, on average, one "significant" result appears by pure chance. Testing many pairs, peeking repeatedly, or trying variables until one lands is how noise gets published as signal.

A Decision Table

This table is the whole family in miniature: read down to your row, across to your distribution, and the valid test is the cell where they meet.

Group Comparison in the SKARI Statistical Lab

SKARI's Statistical Lab treats comparison as its own family — the largest, with the full set of tests — and does the selection and assumption-checking for you.

• One-sample, independent, and paired t-tests, with Welch's correction
• One-way, two-way, and repeated-measures ANOVA, with post-hoc comparisons
• Nonparametric tests — Mann-Whitney, Wilcoxon, Kruskal-Wallis — offered when normality fails
• Bayesian comparison methods for when you want a probability, not just a p-value
• Automatic normality and variance checks, effect size, and a plain-language read

Frequently Asked Questions

Can I just run t-tests on every pair?

No — with three or more groups that inflates false positives. Use ANOVA with a post-hoc test, which controls the overall error rate.

Paired or independent?

Paired when the two measurements are on the same subjects (before/after); independent when the groups are separate cohorts.

How do I know if my data is normal enough?

Check a histogram or a normality test. SKARI runs the check automatically and points you to a nonparametric test when it fails.

What if I only care about the size of the difference?

Report the effect size and its confidence interval. Significance tells you it's real; the effect size tells you whether it's worth acting on.

Key Takeaways

Group comparison is less about the math than the choice. Count the groups, check for pairing, look at the distribution, and the right test follows — then read significance alongside effect size, and correct when you run many. Do that, and 'these groups are different' becomes a claim you can defend.