AdvancedLesson 25 of 306 min read

Many Comparisons and Effect Sizes: Do Not Fool Yourself

See why ten tests produce lucky results, use exact permutation p-values and a stricter line, and report effect sizes.

In this lesson you will

  • Explain why many tests give lucky results
  • Run exact permutation tests
  • Apply a stricter line for many tests
  • Report effect sizes with p-values

About 30 minutes

The one that looked special

Leila’s team checks whether any repair type is rated differently from the others. There are five repair types, and comparing each pair gives ten comparisons. One of them looks interesting. If you report only that one, you are telling a story that leaves out nine attempts. This is one of the most common ways that honest people mislead themselves, and the fix is simple to state: the more questions you ask of the same data, the more likely one of them looks special by luck.

The arithmetic of many tests

If each test has a five percent chance of a lucky result, and the tests are independent, then the chance that at least one of ten gives a false alarm is one minus the chance that none does: 1 minus 0.95 to the power of ten. That is about 40 percent. Ask twenty questions and it is about 64 percent. The tests here are not completely independent, so the figure is approximate, but the lesson stands: with enough questions, something will always seem to appear.

import pandas as pd, itertools
df = pd.read_csv("clean.csv")
def exact_p(a, b):
    pool = a + b
    n, k = len(pool), len(a)
    total = sum(pool)
    observed = abs(sum(a) / len(a) - sum(b) / len(b))
    hits = count = 0
    for idx in itertools.combinations(range(n), k):
        s = sum(pool[i] for i in idx)
        gap = abs(s / k - (total - s) / (n - k))
        count += 1
        if gap >= observed - 1e-9:
            hits += 1
    return hits / count
groups = {name: part["rating"].dropna().tolist() for name, part in df.groupby("repair")}
results = []
for a, b in itertools.combinations(sorted(groups), 2):
    results.append((exact_p(groups[a], groups[b]), a, b))
results.sort()
print("comparisons:", len(results))
for p, a, b in results[:3]:
    print("smallest:", a, "vs", b, "p =", round(p, 3))
print("under 0.05:", sum(1 for p, _, _ in results if p < 0.05))
print("chance of one or more by luck:", round((1 - 0.95 ** len(results)) * 100), "percent")
print("stricter line, 0.05 divided by", len(results), "=", 0.05 / len(results))
print("under the stricter line:", sum(1 for p, _, _ in results if p < 0.05 / len(results)))
gm = df.loc[df["repair"] == "gears", "minutes"].dropna(); cm = df.loc[df["repair"] == "chain", "minutes"].dropna()
pooled = (((len(gm) - 1) * gm.var() + (len(cm) - 1) * cm.var()) / (len(gm) + len(cm) - 2)) ** 0.5
print("effect size, gears vs chain minutes:", round((gm.mean() - cm.mean()) / pooled, 1))
comparisons: 10
smallest: brakes vs puncture p = 0.047
smallest: brakes vs wheel p = 0.097
smallest: brakes vs chain p = 0.147
under 0.05: 1
chance of one or more by luck: 40 percent
stricter line, 0.05 divided by 10 = 0.005
under the stricter line: 0
effect size, gears vs chain minutes: 4.1

The program runs 10 pairwise tests of the ratings and prints the three smallest p-values. It uses an exact permutation test: instead of shuffling at random, it goes through every possible way of splitting the ratings into two groups of the right sizes, so the answer is the same every time and there is no random seed to worry about. The smallest p-value belongs to brakes vs puncture, at 0.047. By the usual line of 0.05, 1 of the ten would be called significant. A report that stopped there would sound exciting.

But ten comparisons were made, and the program reports the chance of one or more false alarms: 40 percent. A simple correction, named after the mathematician Bonferroni, divides the line by the number of tests, so the new line is 0.05 divided by 10, or 0.005. Under the stricter line, 0 comparisons pass. The honest summary is that the data shows no reliable difference in ratings between repair types, and the one near-miss is the kind of result that ten tries would produce by chance.

The Bonferroni correction is blunt and cautious. There are gentler methods, and statisticians argue about when to use them. For this course, the habit matters more than the method: count your comparisons and say how many you made.

How big is the gap? Effect sizes

A p-value says whether a gap is unlikely to be luck. It does not say whether the gap is large. For that you need an effect size. A common one is the gap between two averages measured in standard deviations: the difference in means divided by the typical spread inside the groups. The last line of the program compares gears and chain repairs in minutes and gets an effect size of 4.1. Rough rules of thumb call 0.2 small, 0.5 medium and 0.8 large, so four is enormous. It is large because this made-up workshop was built with gear repairs far longer than chain repairs. Real differences are rarely so clear.

Reporting both pieces is the mark of a careful analyst: “gear repairs took about nineteen minutes longer (an effect of about four standard deviations), and shuffling the labels almost never reproduces a gap like that”. The size tells the reader whether to care, and the p-value tells them whether to believe.

Habits against fooling yourself

  • Write down your questions before you look. A short list in advance is a plan. Questions invented after the data are a hunt.
  • Report every comparison you made, not only the ones that worked.
  • Prefer effect sizes and ranges to bare verdicts of significant or not.
  • Check a surprising result on fresh data. If a finding is real, it should appear again in next month’s visits.
Common mistake: the lucky one out of many

Running many comparisons and reporting only the one with the smallest p-value. Count your tries. A single lucky result among ten is expected.

Luck by construction

The ratings are made up, and they were generated with no link to the repair type. So any difference found is luck by construction, which makes this a safe place to practise. In real data you do not know the truth in advance.

Imagine you test twenty ideas about your own habits against some data. How many would you expect to look significant by luck alone at the five percent line?

Practice for this lesson

Check what you learned

Five quick questions. Pick an answer and you will see straight away whether it is right and why.

Why does asking many questions of one data set cause trouble?

  1. The data runs out
  2. The answers become negative
  3. Something is likely to look special by luck
  4. The tests change the data

About 40 percent for ten tests at the five percent line.

What does an exact permutation test avoid?

  1. Random shuffling and any need for a seed
  2. All calculation
  3. Group sizes
  4. Comparing means

It goes through every possible split, so the answer is the same each time.

What does the Bonferroni correction do?

  1. Multiplies the p-value by the mean
  2. Divides the line by the number of tests
  3. Removes the smallest p-value
  4. Adds more data

Ten tests turn a line of 0.05 into 0.005.

What does an effect size add to a p-value?

  1. Whether the data is clean
  2. The number of tests
  3. The date
  4. How large the gap is, in units of spread

The p-value says whether to believe it, the size says whether to care.

What is a good defence against a lucky finding?

  1. Test it again on the same data
  2. Check it on fresh data
  3. Delete the other tests
  4. Round the p-value

A real finding should appear again in new data.

Summary

  • The more questions you ask, the likelier a lucky result.
  • Count and report all comparisons, not only the winner.
  • A stricter line, such as dividing by the number of tests, cuts false alarms.
  • Report effect size and p-value, and check on fresh data.

Next lesson

Lesson 26 looks ahead: simple forecasts and how to be honest about them.

Questions

Is the Bonferroni correction always the right choice?

It is simple and cautious, so it can miss real effects. There are gentler methods. The habit that matters most is to count your tests and to say how many you ran.

What is an exact test?

One that works through every possible outcome instead of sampling some of them at random. It is only practical for small groups, but it gives the same answer every time.

Can a result be significant and unimportant?

Yes, especially with large samples. A tiny gap can be reliably real and still too small to matter. That is why the effect size belongs beside the p-value.

Progress is saved in this browser.