Comparing Two Groups: Is the Gap Luck? A Permutation Test
Measure a gap between two groups and test it by shuffling labels 10,000 times, then read the result with care.
In this lesson you will
- Measure a gap between two groups
- Run a permutation test by shuffling labels
- Read a p-value without overclaiming
- Avoid testing many comparisons and picking one
About 30 minutes
Is the gap real?
Sam is sure that gear repairs take far longer than chain repairs. Ola, the other mechanic, says it depends on the day. Leila turns to the data and finds a gap in the average times. But any two groups of numbers will differ a little, even if nothing real separates them. The question that matters is whether this gap is bigger than the gaps you would see from pure luck. A method for answering it is called a significance test, and the simplest honest one to understand is the permutation test.
Shuffling the labels
Imagine that the repair type does not affect the time at all. Then the labels “gears” and “chain” are meaningless, and shuffling them among the visits should give gaps about as big as the real one. So you do exactly that: pool all the times, shuffle them into two random groups of the same sizes, and measure the gap. Repeat many times. If real gaps are rarely as large as the one you observed, the observed gap is hard to explain by luck. If shuffled gaps are often as large, the gap tells you nothing.
- Measure the real gap between the two group averages.
- Pool the values and shuffle them into two groups of the original sizes.
- Measure the gap in the shuffled groups, and note whether it is at least as large as the real one in either direction.
- Repeat 10,000 times, with a fixed seed so the result can be repeated.
- Count how many shuffled gaps matched or beat the real one.
import pandas as pd, numpy as np
df = pd.read_csv("clean.csv")
def compare(a, b, seed, shuffles=10000):
a = np.asarray(a, dtype=float); b = np.asarray(b, dtype=float)
observed = a.mean() - b.mean()
pooled = np.concatenate([a, b])
rng = np.random.default_rng(seed)
hits = 0
for _ in range(shuffles):
rng.shuffle(pooled)
if abs(pooled[:len(a)].mean() - pooled[len(a):].mean()) >= abs(observed):
hits += 1
return observed, hits
def say(hits, shuffles=10000):
return "fewer than 1 shuffle in 10000" if hits == 0 else str(hits) + " shuffles in 10000"
gears = df.loc[df["repair"] == "gears", "minutes"].dropna()
chain = df.loc[df["repair"] == "chain", "minutes"].dropna()
diff, hits = compare(gears, chain, 16)
print("gears:", len(gears), "visits | chain:", len(chain), "visits")
print("mean minutes, gears minus chain:", round(diff, 1))
print("gaps this big from shuffling:", say(hits))
hard = df["repair"].isin(["wheel", "brakes"])
a = df.loc[hard, "rating"].dropna()
b = df.loc[~hard, "rating"].dropna()
diff2, hits2 = compare(a, b, 17)
print("wheel or brakes:", len(a), "ratings | others:", len(b), "ratings")
print("mean rating, wheel or brakes minus others:", round(diff2, 2))
print("gaps this big from shuffling:", say(hits2))
gears: 10 visits | chain: 8 visits
mean minutes, gears minus chain: 19.1
gaps this big from shuffling: fewer than 1 shuffle in 10000
wheel or brakes: 11 ratings | others: 22 ratings
mean rating, wheel or brakes minus others: -0.5
gaps this big from shuffling: 2287 shuffles in 10000
For the first comparison, gears has 10 visits and chain has 8. The gears visits took 19.1 minutes longer on average, and the program says that gaps this large came from fewer than 1 shuffle in 10000. In plain words, shuffling the labels almost never produces a gap like that, so luck is a very poor explanation. The gap is real, at least for this workshop’s sample. It also makes sense: gear work is fiddly.
The second comparison asks whether the harder repairs, wheel and brakes, get lower ratings than the others. The wheel and brakes group has 11 ratings, and the other repairs have 22. The harder repairs rated 0.5 lower on average, which sounds meaningful. Yet in 2287 of the 10,000 shuffles the gap was at least as large, which is more than one in five. A gap of that size turns up by luck fairly often in samples this small. The honest conclusion is “we cannot tell”, not “the harder repairs are rated the same”.
Reading the result carefully
The share of shuffles that matched the real gap is usually called a p-value. It is a statement about the shuffled world only: “if nothing real were going on, gaps this big would turn up this often”. It is not the probability that your idea is true, and it is not a measure of how important the gap is. A very small p-value on a tiny gap, or on a huge sample, may be real but useless. A large p-value on a small sample may hide a real gap that you could not detect.
- Look at the size of the gap first. Nineteen minutes is large against an average of thirty. Ask whether the gap matters for the decision before asking whether it is real.
- Say how many values are in each group. The test result is only as good as the samples.
- Decide the comparison before looking. If you test twenty comparisons and report the one that looks good, a lucky result is almost certain.
- Do not treat a line at five percent as magic. A p-value of 0.04 and one of 0.06 carry nearly the same message.
The permutation test needs no formula and no assumption about the shape of the data, which is why it is a good teacher. Other tests do the same job by formula and are quicker on large data, but the logic of “how often would luck match this?” is the same.
Reading “not significant” as “no difference”. It means the data could not rule out luck. With few ratings, a real gap can easily hide.
The times and ratings are made up. The results describe this small sample and would not carry over to a real workshop. The random shuffling uses a fixed seed, so you will get the same counts if you run the program.
Practice for this lesson
Check what you learned
Five quick questions. Pick an answer and you will see straight away whether it is right and why.
What does shuffling the labels imitate?
- A world with no data
- A larger sample
- A cleaner table
- A world where the groups make no real difference
If labels are meaningless, shuffled gaps show what luck can do.
What did the gears versus chain comparison show?
- Shuffled gaps almost never matched the real gap
- Shuffled gaps often matched it
- No gap existed
- The groups were the same size
Luck is a poor explanation for that gap.
Why can we not say the harder repairs are rated lower?
- The gap was zero
- The ratings were deleted
- More than one in five shuffled gaps were as large
- The shuffle was not run
Gaps that size appear by luck fairly often in small samples.
What does a p-value tell you?
- The probability that your idea is true
- The size of the gap
- The number of rows
- How often luck alone would match the gap
It is a statement about the shuffled world only.
Why decide on comparisons before looking at the data?
- It makes the file smaller
- Testing many comparisons makes a lucky result almost certain
- It removes duplicates
- It sorts the groups
Pick the question first, then test it.
Summary
- A gap between two groups is only interesting if luck rarely matches it.
- Shuffling labels shows what luck alone can produce.
- A p-value describes the shuffled world, not the truth of your idea.
- Not significant does not mean no difference.
Questions
Is a permutation test as good as a t-test?
For simple comparisons of two averages they usually agree. The permutation test makes fewer assumptions and is easy to explain; the t-test is a quick formula. Neither fixes a biased sample.
What if I only have a handful of values in a group?
The test still runs, but it can only detect large gaps. Report the group sizes and treat a non-significant result as unclear, not as proof of no difference.
Why set a seed?
Shuffling uses random numbers. A fixed seed makes the result repeatable, so a reader can check your work and get the same counts.
