When two averages differ, how can we tell whether the gap reflects a real pattern?
A distribution center compares two packing methods. Similar orders are assigned to each method, and the time required to finish packing is recorded.
| Packing method | Orders | Mean time | Standard deviation |
|---|---|---|---|
| A | 40 | 42 min | 11 min |
| B | 40 | 36 min | 10 min |
Method B is 6 minutes faster on average. But a different allocation of orders would produce different sample means. Method B may simply have received more straightforward orders in this sample.
A t-test compares the 6-minute gap with the amount of variation and uncertainty in the data.
Key question
Is a 6-minute mean difference large enough to stand out from ordinary sampling variation?
What does a t-test compare?
The same 6-minute gap can tell very different stories. If packing times cluster tightly around each group mean, the methods are easier to distinguish. If times vary widely from order to order, the distributions overlap and the observed gap is less convincing.
Score distributions for two groups
Compare the mean difference with the spread within each group.
A t-test compares the mean difference with its standard error. The standard error describes how much the estimated difference would vary across repeated samples.
The t-statistic is difference ÷ uncertainty
For the same mean difference, a smaller standard error makes the result more distinct.
Mean difference
6 min
Standard error
2 min
t = 6 ÷ 2
3.00
How to read the t-statistic
A larger absolute t-statistic means the observed difference is large relative to its uncertainty. The final inference, however, comes from the p-value calculated from the t-statistic and its degrees of freedom.
Which t-test should you use?
Two averages do not always call for the same t-test. First determine whether the observations come from separate groups or from repeated measurements on the same subjects.
Compare with a reference
One-sample t-test
One group mean vs a fixed value
Compare separate groups
Independent-samples t-test
Group A vs Group B
Compare matched measurements
Paired-samples t-test
Before vs after
| Data structure | Appropriate test |
|---|---|
| Average product weight vs the 500 g target | One-sample t-test |
| Separate packing teams A and B | Independent-samples t-test |
| The same workers before and after a process change | Paired-samples t-test |
A common mistake
If the same workers are measured twice but the data are analyzed as independent, the analysis ignores the fact that some workers are consistently faster or slower. A paired t-test focuses on each worker's change.
A p-value is not enough
When the mean difference and spread remain the same, a larger sample produces a smaller standard error. The same 6-minute gap can therefore lead to a larger absolute t-statistic and a smaller p-value.
The same 6-minute gap, different sample sizes
Larger samples reduce uncertainty around the estimated mean difference.
| Case | Mean difference | Sample size | p-value | Interpretation |
|---|---|---|---|---|
| 1 | 0.4 min | 5,000 per group | < 0.001 | Significant, but probably trivial |
| 2 | 9 min | 10 per group | 0.09 | A large observed gap with substantial uncertainty |
A useful result therefore includes the mean difference, confidence interval, and effect size. For two independent groups, Cohen's d is a common standardized effect-size measure.
Different questions
p-value: How unusual is this result under the null hypothesis?
Effect size: How far apart are the two means?
What should you check first?
For two independent groups, Welch's t-test is usually safer when the variances differ or the sample sizes are substantially unequal.
| Situation | Recommended approach |
|---|---|
| Similar variances and similar sample sizes | The pooled t-test may be acceptable |
| Unequal variances or uncertainty about equality | Welch's t-test |
| Strong skew, small samples, or influential outliers | Consider a robust or nonparametric method |
How to read p = 0.16
❌ The two methods have the same mean packing time.
✓ The current data do not provide sufficient evidence that the two population means differ.
Change the means, spread, and sample size
Adjust the mean difference, standard deviation, and sample size for the two groups. The distributions, t-statistic, p-value, and confidence interval update together.
Two packing methods. Move the three sliders and watch the curves, the t value, the p-value and the interval move together.
Mean difference
6.0
Standard error
2.35
t
2.56
p
0.0125
95% CI for the difference [1.33, 10.67]
A difference of 6.0 minutes against a standard error of 2.35 gives t = 2.56 and p = 0.0125. The interval says the true difference is plausibly between 1.33 and 10.67 minutes.
The same difference can be significant or not. The spread and the sample size decide it.
Key lesson
A t-test result is not determined by the mean difference alone. The difference, spread, and sample sizeall shape the result.
Difference
How far apart are the two means?
Spread
How widely do values vary within each group?
Sample size
How precisely is the mean difference estimated?
Conclusion
Interpret the p-value, confidence interval, and effect size together.
A t-test asks not only whether two means differ, but whether the difference is large relative to its uncertainty.
Analysis results
Independent-samples t-test · Welch
Mean difference
-6.0 min
t-statistic
-2.43
p-value
0.018
Cohen's d
-0.54
A useful results view brings the following information together.
Descriptive statistics
Sample size · mean · standard deviation
Test result
t-statistic · degrees of freedom · p-value
Magnitude
Mean difference · Cohen's d
Uncertainty
Standard error · 95% confidence interval