Experiments and Testing on a Small Blog
Test changes fairly on a small blog: work out how many visitors a test needs, check a result with a shuffle test for luck, avoid peeking and keep an experiment log.
In this lesson you will
- Run a fair one-change test
- Estimate the visitors a test needs
- Check a result for luck
- Avoid peeking and keep a log
About 24 minutes
Priya tries two signup forms
Priya wonders whether a shorter signup form would get more subscribers. She tries form A for a while, then form B, and B seems better. But was it really? Perhaps the weather changed, or a popular post brought a different crowd. A blog can run experiments, and it can also fool itself. This lesson shows how to test fairly on a small site and how to avoid believing a lucky result.
Test one change, at the same time
A fair test compares two versions at the same time, shown to similar visitors, with only one thing different. Running A in October and B in November mixes the change with everything else that happened. Many form and popup tools can split visitors at random. If yours cannot, alternate by week as a second best, and write down anything unusual that happened.
How many visitors do you need?
Small blogs often lack the traffic to detect small changes. The program below uses a standard sample-size formula with a 5 per cent chance of a false alarm and an 80 per cent chance of spotting a real difference, which are conventional choices. Moving a signup rate from 3 to 4 per cent needs 5301 visitors per version. From 3 to 5 per cent needs 1506, and from 3 to 6 per cent only 748. With 400 visitors a week split between two forms, a 3 to 5 per cent change would take about 7.5 weeks.
import random
from statistics import NormalDist
z = NormalDist().inv_cdf
def per_version(p1, p2, alpha=0.05, power=0.80):
pbar = (p1 + p2) / 2
top = z(1 - alpha / 2) * (2 * pbar * (1 - pbar)) ** 0.5 + z(power) * (p1 * (1 - p1) + p2 * (1 - p2)) ** 0.5
return top ** 2 / (p1 - p2) ** 2
for a, b in ((0.03, 0.04), (0.03, 0.05), (0.03, 0.06)):
n = round(per_version(a, b))
print(f"{a:.0%} to {b:.0%}: {n} visitors a version")
weekly = 400
print("weeks at 400 a week:", round(2 * per_version(0.03, 0.05) / weekly, 1))
# MADE-UP result: signup form A 18 of 400, form B 31 of 420
a_yes, a_n, b_yes, b_n = 18, 400, 31, 420
print(f"A: {a_yes / a_n:.1%} B: {b_yes / b_n:.1%}")
people = [1] * (a_yes + b_yes) + [0] * (a_n + b_n - a_yes - b_yes)
observed = b_yes / b_n - a_yes / a_n
rng = random.Random(7)
count, trials = 0, 20000
for _ in range(trials):
rng.shuffle(people)
diff = sum(people[:b_n]) / b_n - sum(people[b_n:]) / a_n
if abs(diff) >= abs(observed):
count += 1
p = count / trials
print(f"as big a gap by luck: {p:.2f} of shuffles")
print("clear win:", p < 0.05)
3% to 4%: 5301 visitors a version
3% to 5%: 1506 visitors a version
3% to 6%: 748 visitors a version
weeks at 400 a week: 7.5
A: 4.5% B: 7.4%
as big a gap by luck: 0.10 of shuffles
clear win: False
The lesson from those numbers is that small improvements cannot be seen on a small blog. A change from 3 to 4 per cent would need 5301 visitors per version, which is about 26.5 weeks of all your traffic at 400 a week, just to learn about one small difference. So test big changes, such as a different offer, and avoid agonising over button colours.
Was it luck?
The last part checks Priya’s made-up result. Form A got 18 signups from 400 visitors, which is 4.5 per cent, and form B got 31 from 420, which is 7.4 per cent. B looks much better. The check is a permutation test, which is easy to picture. Take all the 49 signups and all the visitors, shuffle who got which form 20,000 times at random, and count how often chance alone produces a gap as large as the one seen. It happened in 0.10 of the shuffles. A common line for calling a result convincing is 0.05, so 0.10 is not convincing. B may truly be better, but the evidence is not strong enough to say so. The honest answer is ‘probably, keep testing’.
Two cautions. First, a seeded shuffle gives a slightly different figure each time you change the seed, so treat it as approximate. Second, 0.05 is a convention and not a law of nature. A result just below it is not proof, and one just above it is not nothing.
Avoid the common traps
- Peeking. Checking every day and stopping when you see a win inflates false alarms. Decide the sample size first.
- Many tests, one winner. Try twenty ideas and one will look good by luck. Repeat any surprising win.
- Changing two things. You will not know which one mattered.
- Ignoring size. A tiny gain that is real may not be worth the work.
- Forgetting your readers. Do not trick people into signing up. Test honest wording only.
Keep an experiment log
- Write the question and what you expect, before you start.
- Choose one change and the number of visitors you will wait for.
- Run both versions together, and note anything odd that happens.
- Compare the results with a check for luck, and write the answer.
- Decide: keep, drop or repeat. Record it so you do not run the same test twice.
Declaring victory after three days because the numbers look good. Early results swing wildly on small counts. Wait for the sample you planned.
The rates and counts are made up. A test on one blog says nothing certain about other blogs. Statistical checks measure luck and not importance, and they cannot fix a badly designed test.
Practice for this lesson
Check what you learned
Five quick questions. Pick an answer and you will see straight away whether it is right and why.
What makes a test fair?
- Running A then B in different months
- Changing everything
- Stopping early
- Both versions run at the same time, with one change
Same time and one change keeps the comparison clean.
Why can small blogs not see small improvements?
- They need thousands of visitors per version
- Small blogs never improve
- Tests are banned
- Readers refuse
Small differences need large samples.
What did the luck check say about form B?
- B was proven best
- A was proven best
- The gap was not convincing yet
- Nothing was measured
About one in ten shuffles matched the gap, above the 0.05 line.
What is peeking?
- Reading comments
- Looking at photos
- A kind of link
- Checking early and stopping when you see a win
Peeking raises the chance of false winners.
What should you test on a small blog?
- Tiny colour changes
- Big changes such as a different offer
- Nothing at all
- Ten ideas at once
Bigger effects can be detected with less traffic.
Summary
- Test one change, at the same time, with similar visitors.
- Small improvements need thousands of visitors to detect.
- A shuffle test shows how often luck gives the same gap.
- Decide the sample first and keep a log.
Questions
Can I trust a result from a few hundred visitors?
Only for big differences. Small differences are drowned by luck, as the sample sizes show.
Do I need special software?
No. A spreadsheet or a short program can count and shuffle. Many form tools also include their own testing.
Is 0.05 a rule?
It is a convention. Treat it as a guide, repeat surprising wins and judge by usefulness as well as numbers.
