Almost every data question is really the same question: is what I'm seeing real, or could it have happened by chance? Conversions ticked up, one group scored higher, a metric moved — but samples wobble on their own. Hypothesis testing is the disciplined way to separate a genuine effect from ordinary noise.
Note
The Null and Alternative
Every test starts with two competing statements. The null hypothesis is the boring default: nothing is going on — the groups are equal, the change did nothing. The alternative is what you suspect: there is a difference.
| Hypothesis | Says |
|---|---|
| Null (H₀) | No effect — any difference is chance |
| Alternative (H₁) | A real effect exists |
The logic is deliberately conservative: you assume the null is true and ask whether the data is too surprising to believe it. You never "accept" the null — you either reject it or fail to reject it.
What a p-value Actually Means
The p-value is the probability of seeing data at least as extreme as yours if the null hypothesis were true. A small p-value means your result would be unlikely under "no effect," so the no-effect story starts to look implausible.
| p-value | Interpretation |
|---|---|
| p < 0.05 | Statistically significant by the common convention |
| p ≈ 0.05 | Borderline — treat with caution |
| p > 0.05 | Not enough evidence to reject the null |
Watch out
Significance Isn't Size
A tiny, meaningless difference can be "significant" with a big enough sample, and a large, important one can miss significance in a small study. That's why a p-value alone is never the whole answer.
- Effect size: how big the difference actually is, in real units
- Confidence interval: the plausible range for the true effect
- Sample size: large samples make even trivial effects significant
Tip
Choosing the Test
| Situation | Test |
|---|---|
| Two group means | t-test |
| Three or more group means | ANOVA |
| Two categorical variables | Chi-square |
| Skewed data / small samples | Nonparametric (Mann-Whitney, Kruskal-Wallis) |
Each test also carries assumptions — normality, equal variances, independence. Break them and the p-value is unreliable, which is why checking assumptions is part of the test, not an afterthought.
Two Ways to Be Wrong
| Error | Meaning |
|---|---|
| Type I (false positive) | Reject a true null — see an effect that isn't there |
| Type II (false negative) | Miss a real effect that is there |
Watch out
Hypothesis Testing in the SKARI Statistical Lab
SKARI's Statistical Lab runs the right test and guards the assumptions: pick your variables and it selects, checks, and interprets — so you get significance, effect size, and a plain-language read together.
- One-sample, independent, and paired t-tests; one-way and two-way ANOVA
- Crosstab and chi-square for categorical relationships
- Automatic normality and variance checks, with nonparametric alternatives (Mann-Whitney, Kruskal-Wallis) when they fail
- Effect size and interpretation alongside the p-value
Takeaway
Frequently Asked Questions
Is p < 0.05 a law?
No — it's a convention. The right threshold depends on the cost of a false positive; some fields use 0.01 or stricter.
Does a big p-value prove no effect?
No — it means you lack evidence for one. Absence of evidence isn't evidence of absence, especially in small samples.
What's the single most important habit?
Decide your hypothesis and test before seeing the data, and always report effect size beside the p-value.
Key Takeaways
Assumes
H₀
no effect
Measures
p-value
surprise under H₀
Also needs
Effect
does it matter?
Threshold
0.05
a convention
Hypothesis testing keeps you from mistaking noise for signal. Assume nothing is happening, measure how surprising your data would be, and pair significance with effect size before you call anything a win.
Takeaway
The t-test
Comparing two means
Confidence Intervals
The range behind the estimate
ANOVA
Three or more groups at once