A/B Testing: Design, Significance, and the Lift

Version B converted at 5.3%, version A at 4.1%. Ship B? Not yet. An A/B test is a controlled experiment, and whether that lift is real depends entirely on how you ran it.

Note

Random assignment is what turns a comparison into causation. Because users are split randomly, a difference in outcome can be credited to the change itself.

How to Design One

State one hypothesis and one primary metric before you start
Randomly assign users to A (control) and B (variant)
Compute the sample size you need up front — for the effect you care about
Run until you hit that size, then evaluate — once

Reading the Result

A raw lift isn't enough. A significance test (often a chi-square on conversions, or a proportion test) tells you whether the gap is bigger than chance would produce. Pair it with a confidence interval to see the plausible range of the true lift.

Tip

Report the lift with its confidence interval. "5.3% vs 4.1%, and the interval excludes zero" is a decision; "5.3% vs 4.1%" alone is a guess.

The Traps That Fake a Win

Peeking: checking every day and stopping the moment it looks significant inflates false positives
Stopping early: small samples swing wildly — commit to your sample size
Too many metrics: test enough and one will look 'significant' by chance
Novelty effect: a new design gets a temporary bump that fades

Watch out

Peeking is the classic killer. Decide the sample size in advance and evaluate once — or use a method built for continuous monitoring.

A/B Testing in SKARI

SKARI supports A/B comparison in its survey and analysis tools: split responses into variants, compare the metric, and get the significance test and interval — no manual stats.

Compare conversion or score across two variants
Significance test and confidence interval on the difference
Plain-language verdict so you know whether to ship

Takeaway

You get the lift, the significance, and the interval together — a real decision, not a hopeful one.

Frequently Asked Questions

How long should I run a test?

Until you reach the sample size your power calculation requires — and through full weekly cycles to avoid day-of-week bias.

Why not stop as soon as it's significant?

Because "peeking" massively inflates false positives. Fix the sample size in advance, or use sequential/Bayesian methods designed for it.

Can I test several changes at once?

Yes, but you can't attribute the result to any single change. Use an A/B/n or a factorial design if you need to isolate effects.

Conclusion

A/B testing is experimentation you can actually run — but only randomization, a pre-set sample size, and honest significance make the result trustworthy.

Takeaway

Randomize, size it up front, and evaluate once — then a winning variant really is winning.

Hypothesis Testing Basics

The inference behind the test

Confidence Intervals

Put a range on the lift

Correlation ≠ Causation

Why randomization matters so much