A/B Testing Done Properly
Work out how many visitors a fair test needs, run a simulation that shows how peeking inflates false wins, and learn to plan, stop and read a test honestly.
In this lesson you will
- Compute the visitors a test needs
- Explain why peeking inflates false wins
- Write a stopping rule in advance
- Choose tests your traffic can support
About 26 minutes
Noor stops the test when it looks good
Noor compares two versions of her product page. After three days, version B is ahead and the dashboard shows a happy green number, so she stops the test and switches everyone to B. A month later her sales have not moved. What went wrong is among the most common mistakes in testing: she decided when to stop by watching the results, and results that bounce around will always look good at some moment. Lesson 8 introduced ranges. This lesson shows how to run a fair test: decide the size in advance, look at the result once, and understand what peeking does.
How many visitors do you need?
The answer depends on three choices. The size of the difference you hope to detect, how often you accept a false alarm, and how often you want to catch a real difference. A common convention is a 5 per cent false alarm rate and 80 per cent power, which means catching a real difference of that size four times in five. The smaller the difference, the more visitors you need, and the need grows fast. The program below uses a standard formula for comparing two rates to work out the visitors needed on each page.
import math, random
from statistics import NormalDist
nd = NormalDist()
def per_arm(p1, p2, alpha=0.05, power=0.80):
za, zb = nd.inv_cdf(1 - alpha / 2), nd.inv_cdf(power)
pb = (p1 + p2) / 2
num = za * math.sqrt(2 * pb * (1 - pb)) + zb * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))
return math.ceil((num / (p2 - p1)) ** 2)
print("visitors per page, 3% to 4%:", per_arm(0.03, 0.04))
print("visitors per page, 3% to 5%:", per_arm(0.03, 0.05))
print("visitors per page, 3% to 3.5%:", per_arm(0.03, 0.035))
def z_stat(a, na, b, nb):
p = (a + b) / (na + nb)
se = math.sqrt(p * (1 - p) * (1 / na + 1 / nb))
return (b / nb - a / na) / se if se else 0
random.seed(7)
runs, looks, step, p = 2000, 10, 100, 0.03
end_win = any_win = 0
for _ in range(runs):
a = b = na = nb = 0
seen = False
for k in range(looks):
a += sum(random.random() < p for _ in range(step)); na += step
b += sum(random.random() < p for _ in range(step)); nb += step
sig = abs(z_stat(a, na, b, nb)) > 1.96
seen = seen or sig
end_win += sig
any_win += seen
print("tests where nothing differs:", runs)
print("looked once at the end, false wins:", round(100 * end_win / runs, 1), "%")
print("stopped at the first win, false wins:", round(100 * any_win / runs, 1), "%")
visitors per page, 3% to 4%: 5301
visitors per page, 3% to 5%: 1506
visitors per page, 3% to 3.5%: 19743
tests where nothing differs: 2000
looked once at the end, false wins: 5.0 %
stopped at the first win, false wins: 19.1 %
To detect a rise from 3 per cent to 4 per cent you need about 5301 visitors on each page. For 3 to 5 per cent, a bigger gap, you need only 1506. For a tiny gain from 3 to 3.5 per cent you need 19743. For a small shop with a few hundred visitors a week, that last test could take years, which means it is not worth running. Instead test bold changes that might produce a big difference, or combine pages to get more traffic.
What peeking does
The second half of the program is a simulation. It runs 2000 imaginary tests where the two pages are exactly the same, so every win is a false alarm. In each test, 1,000 visitors arrive per page, in ten batches, and the test is checked after each batch. If you look once at the end, as planned, false wins happen about 5.0 per cent of the time, which is what a 5 per cent threshold promises. But if you stop at the first moment the result looks significant, false wins jump to about 19.1 per cent. That is nearly one test in five declaring a winner when nothing is different. Peeking does not make the data wrong. It gives the data many chances to look right by luck.
- Decide the smallest difference worth detecting, and compute the visitors needed before you start.
- Write the stopping rule down: a number of visitors, or an end date.
- Do not stop early because it looks good, and do not extend because it does not.
- Check the result once at the planned end, with a range as in lesson 8.
- If there is no clear winner, accept that the change did not matter much, and move on to a bigger idea.
Running many tests and reporting the winners. If you test twenty ideas, about one will look significant by luck alone. Keep a list of everything you tested, and treat a surprising win as something to repeat.
There are methods that allow looking at results during a test, called sequential methods, but they are built so that the false alarm rate stays controlled. They are not the same as watching a dashboard. If your testing tool offers one, read its explanation. This lesson uses simple, fixed-size tests and made-up numbers.
Practice for this lesson
Check what you learned
Five quick questions. Pick an answer and you will see straight away whether it is right and why.
What does a larger planned difference do to the visitors needed?
- It increases them
- It does not change
- It makes them random
- It reduces them
Bigger differences are easier to detect, so fewer visitors are needed.
What is peeking?
- Looking once at the planned end
- Stopping when the result first looks significant
- Counting visitors
- Making a report
Checking repeatedly and stopping at the first win gives luck many chances.
What happened to false wins with peeking in the simulation?
- They fell to zero
- They stayed at 5%
- They disappeared
- They jumped to about one in five
Repeated looks multiplied the chance of a false win.
What should you do before starting a test?
- Decide the visitors needed and the stopping rule
- Pick a winner
- Change the page daily
- Hide the data
A fixed plan protects against fooling yourself.
What if your traffic is too small for a tiny change?
- Run it anyway for a day
- Stop all testing
- Test bolder changes instead
- Peek more
Bold changes produce bigger differences that small traffic can detect.
Summary
- Plan the sample size and the stopping rule before you start.
- Smaller differences need many more visitors.
- Stopping when it first looks good inflates false wins.
- Test bolder changes when traffic is small.
Questions
What does power mean?
It is the chance a test catches a real difference of the size you planned for. A common choice is 80 per cent, meaning it would be caught four times in five.
Is the simulation real data?
It is a real computer simulation with random numbers and a fixed seed, using made-up rates. It shows what peeking does in general.
Can my testing tool stop early for me?
Some tools use sequential methods designed for it. Read the tool's explanation. A tool that simply shows a winner as soon as it appears may be vulnerable to peeking.
