Sampling and Uncertainty: Standard Errors and the Bootstrap
Find a range around an average and a share, understand sampling variation, and use a fixed-seed bootstrap as a second opinion.
In this lesson you will
- Explain samples, populations and sampling variation
- Calculate a standard error and a rough range
- Run a bootstrap with a fixed seed
- Give a range for a share
About 30 minutes
“Is 3.88 good?”
Leila reads a number from the first course: the mean rating is 3.88. A volunteer asks her, “Is that better than last year’s 4.1?”. Leila hesitates. The ratings come from a small group of customers on a few days, and a different few days might have given a different average. A number from a sample is an estimate of something bigger: how all customers, past and future, would rate the workshop. The honest question is not just “what is the number?” but “how far could it be from the truth?”.
Samples and populations
The population is everyone you care about, such as every customer the workshop will ever serve. The sample is the part you actually measured. If another group of customers had been asked, the average would have come out a little different. This natural wobble, called sampling variation, is not a mistake in the data. It is built into using a sample, and it shrinks as the sample grows.
The standard error
The spread of the ratings tells you how much customers disagree. Dividing that spread by the square root of the number of ratings gives the standard error: roughly, how much the average itself would wobble from sample to sample. A rough rule says the true average is within about two standard errors of the sample average most of the time. Two ideas are packed into the formula: more disagreement means more wobble, and more ratings mean less wobble, but only with the square root, so quadrupling the ratings only halves the error.
import pandas as pd, numpy as np
df = pd.read_csv("clean.csv")
r = df["rating"].dropna().to_numpy()
n = len(r)
mean = r.mean()
se = r.std(ddof=1) / n ** 0.5
print("ratings:", n, "| mean:", round(mean, 2))
print("standard error:", round(se, 2))
print("mean plus or minus 2 se:", round(mean - 2 * se, 2), "to", round(mean + 2 * se, 2))
rng = np.random.default_rng(15)
boot = np.array([rng.choice(r, size=n, replace=True).mean() for _ in range(5000)])
lo, hi = np.percentile(boot, [2.5, 97.5])
print("bootstrap 95 percent range:", round(lo, 2), "to", round(hi, 2))
k = int((r >= 4).sum())
p = k / n
z = 1.96
centre = (p + z * z / (2 * n)) / (1 + z * z / n)
half = z * ((p * (1 - p) / n + z * z / (4 * n * n)) ** 0.5) / (1 + z * z / n)
print("happy share:", k, "of", n, "=", round(p * 100, 1), "percent")
print("range for the share:", round((centre - half) * 100), "to", round((centre + half) * 100), "percent")
print("with 4 times as many ratings, se would be about", round(se / 2, 2))
ratings: 33 | mean: 3.88
standard error: 0.19
mean plus or minus 2 se: 3.5 to 4.26
bootstrap 95 percent range: 3.48 to 4.24
happy share: 24 of 33 = 72.7 percent
range for the share: 56 to 85 percent
with 4 times as many ratings, se would be about 0.09
The program counted 33 ratings with a mean of 3.88. The standard error is 0.19, so the rough range runs from 3.5 to 4.26. That range comfortably includes last year’s 4.1, so Leila cannot say the rating has fallen. The data is consistent with no change.
The bootstrap
There is a second way to find the range that needs no formula and works for almost any statistic. The bootstrap treats the sample as a stand-in for the population. It draws a new sample of the same size from your ratings, with replacement, which means a rating can be picked more than once, and it calculates the mean of that new sample. Repeating this 5,000 times gives 5,000 plausible means, and the middle 95 percent of them form a range. The program uses a fixed seed so that the answer is repeatable. The bootstrap range is 3.48 to 4.24, very close to the formula’s range. When two different methods agree, you can trust the answer more.
The bootstrap looks like magic, but the logic is plain. If your sample is a fair miniature of the population, then resampling from it imitates what would happen if you could sample again from the real world. It fails when the sample is tiny, biased or missing important kinds of customer. No clever method can recover information that was never collected.
Shares have uncertainty too
A share of customers needs its own range. The program found 24 happy ratings out of 33, which is 72.7 percent. The formula used for the range is the Wilson interval, a standard choice that behaves well for small samples. The range runs from 56 to 85 percent. A reader who sees “73 percent” alone might feel certain. A reader who sees “somewhere between about 56 and 85 percent” understands that the workshop cannot yet claim much.
What the range does and does not say
- It describes sampling wobble only. It does not cover a biased sample, such as only asking happy customers.
- It is not the chance that the truth lies inside this one range. It is a statement about how the method behaves over many samples.
- A wide range is useful information. It says, “collect more before deciding”.
- Do not stretch one range into a verdict. Show the range beside the estimate and let it temper your language.
The last line of the program looks ahead. With four times as many ratings, the standard error would be about 0.09, half as large. If Leila wanted to detect a small change in rating, she would need many more ratings than she has now.
Quoting a single average as if it were exact. With 33 ratings, a difference of a few tenths between two averages is well inside the noise. Show the range.
The ratings are made up, and the last year figure of 4.1 is an invented comparison for the story. The ranges here describe sampling wobble in this one sample, not real customers.
Practice for this lesson
Check what you learned
Five quick questions. Pick an answer and you will see straight away whether it is right and why.
What is a sample?
- The whole population
- A cleaned table
- The part of a population that was actually measured
- A chart type
The 33 ratings are a sample of all possible ratings.
What does the standard error describe?
- How much an average would wobble from sample to sample
- The number of rows
- The largest rating
- The date of the file
It combines the spread of ratings with the sample size.
How does quadrupling the number of ratings change the standard error?
- It quarters it
- It roughly halves it
- It doubles it
- It does not change it
The error shrinks with the square root of the sample size.
In the bootstrap, what does with replacement mean?
- Ratings are replaced by zero
- The file is replaced
- Each rating is used exactly once
- The same rating can be picked more than once
Each resample has the same size as the original.
What does a range around an average not cover?
- The wobble of random sampling
- A biased sample, such as asking only happy customers
- The size of the sample
- The spread of the ratings
The range covers only random sampling variation.
Summary
- A sample estimate wobbles from sample to sample.
- The standard error shrinks with the square root of the sample size.
- The bootstrap resamples your own data to find a range.
- Always show a range beside an estimate; it does not cover bias.
Questions
Why 5,000 resamples?
It is a common round number that is large enough for a stable range. Ten thousand would give a slightly steadier answer. With a fixed seed, the result is repeatable.
Is the bootstrap better than the formula?
Neither is always better. The formula is quick when its assumptions hold. The bootstrap needs fewer assumptions and works for statistics that have no simple formula. Agreement between them is reassuring.
How big a sample do I need?
It depends on how much the values vary and how small a difference matters. A wide range from your pilot data is a good sign that you need more, and the standard error formula shows roughly how much.
