A/B Testing

This is where the whole course comes together. An A/B test (a randomized controlled experiment) is the single most powerful tool a data scientist has for answering "does this change actually work?" — and it runs on every concept you've built: populations and samples, randomization, sampling distributions, confidence intervals, hypothesis tests, p-values, power, and effect sizes. If you can design and read an A/B test correctly, you can be trusted with real decisions.

We'll go in the order a real experiment does: design it before you collect a single row, analyze it once it's done, and decide whether to ship — then catalog the pitfalls that quietly invalidate otherwise-fine tests.

Design comes first (and is where most tests are won or lost)

The analysis is the easy part. A test that wasn't designed properly can't be rescued by clever statistics afterward. Four decisions, all made before data collection:

1. A single, falsifiable hypothesis. "The new checkout button increases the purchase-conversion rate." Not "the new button is better" (better at what?) — name the exact change and the exact outcome.

2. One primary metric, chosen in advance. Pick the number the decision hinges on — say, conversion rate — and commit to it. Tracking ten metrics and celebrating whichever one moves is the multiple-comparisons trap from Statistical Fallacies wearing a business suit.

3. Randomization. Assign each user to control (A) or treatment (B) at random. This is the load-bearing wall of the whole method, so it gets its own section below.

4. Sample size / power, up front. Decide how many users you need before starting, so the test can actually detect an effect worth caring about. More on this two sections down.

Why randomization licenses causal claims

Here is the deep reason A/B tests can say "X caused Y" when ordinary observational data (from Statistical Fallacies) cannot. Randomly assigning who gets the treatment makes the two groups statistically identical on everything — measured and unmeasured — except the change you're testing. Age, device, loyalty, the weather, mood, and every confounder you never thought of are, on average, balanced across the two arms. So if B outperforms A by more than chance allows, the only systematic difference between the groups is the treatment. That severs the confounder's arrow and earns you a causal conclusion.

Power and sample size, decided before you start

You met statistical power in Errors and Power: the probability your test detects a real effect of a given size. Run a test on too few users and even a genuine, worthwhile lift can come back non-significant — you "found nothing" only because you lacked the data to find it. So you size the experiment up front, around the smallest effect worth detecting (the minimum detectable effect, MDE).

The four knobs trade off against each other:

Analyze: simulate and fully read one experiment

Let's run a complete analysis. We'll simulate an experiment with a known true effect (so we can check our work), then analyze it the way you would real data: compute each arm's conversion rate, the observed lift, a confidence interval on the difference, a formal test, the p-value, and an effect size.

For a conversion (yes/no) metric, the natural test compares two proportions. We'll use a two-proportion z-test computed directly, and cross-check it against a chi-square test on the 2×2 table — they answer the same question.

Look at the full picture, not just the p-value. You get a point estimate of the lift, a confidence interval that shows how precise it is, a p-value for "is it distinguishable from no effect," and an effect size. That's the reporting trio from Effect Sizes, applied.

A chart makes the comparison and its uncertainty legible. We'll show each arm's conversion rate with an error bar (the CI half-width).

Decide: ship or don't ship

A significant p-value is permission to look at the effect size, not an order to ship. The decision weighs two axes together — exactly the significance-vs-importance distinction from Effect Sizes.

Walk the branches:

• Not significant. No green light. But check power first — a null result on a tiny sample means "we couldn't tell," not "no effect."
• Significant but tiny. The classic trap. With enough traffic a 0.05pp lift is "significant" and worthless. Look at the effect size and whether the whole confidence interval clears the threshold you'd care about.
• Significant and meaningful. Ship — and ideally confirm the CI's lower bound is still above the level that justifies the rollout cost.

Pitfalls that quietly invalidate a test

Even a well-designed test can be ruined by a handful of classic mistakes. Each one inflates false positives or biases the estimate.

• Peeking and early stopping. Watching the p-value daily and stopping the instant it dips below 0.05 dramatically inflates your false-positive rate. The p-value wanders; if you stop on the first lucky dip, you stop on noise. Decide n (or use a proper sequential-testing method) and run to it.
• Multiple metrics / multiple variants. Every extra metric or arm is another shot at a false positive. Correct for it (e.g., Bonferroni) or pre-commit to one primary metric.
• Novelty effect. Users click the new thing because it's new; the bump fades. Run long enough to see whether the lift persists past the novelty window.
• Simpson's paradox across segments. An overall win can hide a loss in every segment (or vice versa) if your traffic mix shifted — straight from Statistical Fallacies. Sanity-check key segments.
• Unequal / broken groups (assignment bias). If the arms differ in size or composition more than randomization predicts — a "sample ratio mismatch" — your randomization is broken and the comparison is suspect.

Practice

Check your understanding

QuestionSelect one

Why does randomly assigning users to A and B let an A/B test support a causal claim, when ordinary observational data usually can't?

Because randomization guarantees the two groups have identical sample sizes

Because randomization balances confounders — known and unknown — across the two groups, so the treatment is the only systematic difference

Because randomization increases the sample size

Because randomized tests never produce false positives

QuestionSelect one

A test reaches p = 0.04 after 3 days. Your teammate wants to stop and ship immediately. What's the statistical problem with stopping the moment it crosses 0.05?

Nothing — once p < 0.05 the result is locked in and valid

Peeking and stopping at the first significant moment inflates the false-positive rate well above the stated alpha

The p-value is too high to stop

You should switch to a one-sided test to justify stopping

QuestionSelect one

With 5 million users, a new layout shows a statistically significant lift in conversion of 0.02 percentage points (from 8.00% to 8.02%). The 95% CI is [0.005pp, 0.035pp]. Should you ship?

Yes — it's statistically significant, and significance means ship

Probably not on this alone — the effect is real but so small it likely isn't worth the cost; weigh the tiny effect size against the rollout cost

Yes, because the confidence interval excludes zero

No, because the result must be a false positive

QuestionSelect one

Your team's experiment tracks 12 metrics and declares victory because one of them improved with p = 0.03. What's the flaw?

A t-test was the wrong choice for 12 metrics

Testing 12 metrics multiplies the chance that at least one looks significant by luck; without correction or a pre-declared primary metric, the "win" may be noise

p = 0.03 is not significant enough

They should have used a larger sample

QuestionSelect one

An A/B test shows the treatment winning overall, but when you split by device, the treatment loses on both mobile and desktop. What's the most likely culprit?

The treatment is genuinely better, so ignore the segments

The test had too few users

Simpson's paradox — the traffic mix differs between arms, so the pooled result reverses what happens within each device

The metric was continuous instead of binary