Is a new design truly better, or did it only look better in this sample?
An app redesigns its checkout screen. The current version converts 8.1% of visitors, while the new version converts 9.4%.
Current conversion
8.1%
New conversion
9.4%
p-value
0.012
The observed difference is 1.3 percentage points. We need to determine whether it reflects a real improvement or ordinary sampling variation.
Key question
If there were no real difference, how unusual would a gap of 1.3 percentage points or more be?
What is a p-value?
The p-value is the probability of obtaining a result at least as extreme as the observed one, assuming the null hypothesis is true.
| p-value | Interpretation |
|---|---|
| 0.37 | A result this large is common under H₀ |
| 0.012 | A result this large is relatively unusual |
| 0.001 | A very unusual result under H₀ |
What is the significance level?
The significance level α defines the rule for rejecting H₀. A common choice is 0.05, but it is a convention rather than a natural law.
| Comparison | Decision |
|---|---|
| p < α | Reject H₀ |
| p ≥ α | Do not reject H₀ |
p = 0.049 and p = 0.051 represent nearly the same strength of evidence. A hard threshold should not erase that continuity.
Small does not mean large
| Case | Effect | Sample size | p-value |
|---|---|---|---|
| 1 | 0.15 pp | 1,000,000 per group | < 0.001 |
| 2 | 4 pp | 25 per group | 0.64 |
The p-value depends on effect size, sample size, and variability. Report the observed difference and confidence interval as well.
A commonly misread number
For p = 0.02
❌ There is a 2% probability that H₀ is true.
❌ There is a 98% probability that the redesign is better.
✓ If H₀ were true, a result this extreme would occur 2% of the time.
Change the sample size
Two checkout screens. Move the gap and the number of visitors, and watch the p-value follow both.
Gap
1.30%p
Standard error
0.516%p
p
0.0117
A gap of 1.30 percentage points across 6,000 visitors per screen gives p = 0.0117. The gap on the screen never changed — only how much data stands behind it.
Enough visitors make any gap significant. That is why the p-value is not the size of the effect.
A p-value answers “How unusual is this result under H₀?” not “What is the probability that this claim is correct?”