IntermediateLesson 17 of 305 min read

Correlation and a Simple Line Fit, with Cautions

Measure how two numbers move together, fit a line, predict with a typical miss, and see how a confounder fools a correlation.

In this lesson you will

  • Calculate and read a correlation
  • Fit a line and read its slope
  • Report a typical prediction miss
  • Spot a confounder inside groups

About 28 minutes

Do longer repairs cost more?

Omar has noticed that the long repairs seem to cost more, and he wonders whether the workshop could estimate a price from the time a repair takes. When two numbers rise and fall together, they are correlated. The strength of the relationship is measured by the correlation coefficient, written r, which runs from minus one to plus one. Plus one means the points lie exactly on a rising line, zero means no straight-line relationship, and minus one means they lie on a falling line.

Fitting a line

A line of best fit summarises the relationship as a rule: cost equals a starting amount plus a rate for each minute. It is chosen so that the total of the squared misses between the line and the points is as small as possible, which is why it is also called least squares. The slope tells you the average change in cost for one more minute. The square of r, called r squared, says what share of the ups and downs in cost is accounted for by the line.

import pandas as pd, numpy as np
df = pd.read_csv("clean.csv").dropna(subset=["minutes"])
x, y = df["minutes"].to_numpy(), df["cost"].to_numpy()
r = np.corrcoef(x, y)[0, 1]
slope, intercept = np.polyfit(x, y, 1)
print("visits used:", len(df))
print("correlation r:", round(r, 2), "| r squared:", round(r * r, 2))
print("line: cost = " + str(round(intercept, 2)) + " + " + str(round(slope, 3)) + " x minutes")
print("30 minutes predicts a cost of", round(intercept + slope * 30, 2))
for name in ["gears", "brakes"]:
    part = df[df["repair"] == name]
    print("inside " + name + ": " + str(len(part)) + " visits, r = " + str(round(part["minutes"].corr(part["cost"]), 2)))
top = df["minutes"].idxmax()
without = df.drop(top)
print("r without the longest visit:", round(without["minutes"].corr(without["cost"]), 2))
resid = y - (intercept + slope * x)
print("typical miss:", round(float(np.sqrt((resid ** 2).mean())), 2))
visits used: 37
correlation r: 0.71 | r squared: 0.51
line: cost = 2.36 + 0.368 x minutes
30 minutes predicts a cost of 13.4
inside gears: 10 visits, r = -0.1
inside brakes: 8 visits, r = 0.07
r without the longest visit: 0.69
typical miss: 4.22

The program used 37 visits, leaving out the one visit whose minutes were removed as a typo. The correlation is r = 0.71, which is strong and positive, and r squared is 0.51, so about half of the variation in cost lines up with the minutes. The fitted line is cost = 2.36 + 0.368 x minutes, so each extra minute goes with roughly 0.368 sample pounds more. A repair of thirty minutes predicts a cost of 13.4, with a typical miss of 4.22.

That miss matters. A prediction of 13 with a typical error of 4 is a rough guide, not a price list. A line is an average, and individual visits scatter around it.

The warning: what lies behind the line

Now read the two lines in the middle of the output. Inside the gears repairs alone, the correlation between minutes and cost is -0.1, and inside the brakes repairs it is 0.07, essentially none. The strong overall correlation does not come from longer repairs costing more within a type. It comes from the repair types themselves: wheel and gears jobs are both long and expensive, while punctures are short and cheap. The repair type is a confounder, a third thing that drives both numbers. Within one type of job, an extra ten minutes does not add to the price, because the price follows the job and not the stopwatch.

This is why the old warning matters: correlation is not causation. Spending longer on a repair did not cause the higher price, and charging more would not make a repair take longer. A pattern in the data can be real and still mislead you about what to change.

Other ways a correlation misleads

  • One extreme point. A single unusual visit can pull the line. The program recalculates r without the longest visit, and it moves only a little to 0.69, which shows that this pattern does not hang on one point.
  • A curve, not a line. r measures straight-line patterns only. Two numbers can be tightly linked in a curve and still have r near zero.
  • A narrow range. A line fitted to visits of 20 to 50 minutes says nothing reliable about a visit of 200.
  • Pure chance. With a handful of points, strong-looking correlations appear by luck. The shuffle idea from the last lesson works here too.

Good habits

  1. Draw the points first, even as a rough text sketch. A number alone hides curves and outliers.
  2. Calculate r, but also look inside the groups, as the program did.
  3. Say what the slope means in the units of the problem: pounds per minute.
  4. Report the typical miss, so the reader sees how rough a prediction is.
  5. Use words like goes with, not causes, unless you have an experiment or a strong reason.
Common mistake: correlation as cause

Saying “longer repairs cause higher prices” from a correlation. A third factor, the type of repair, drives both. Look inside groups before you claim a cause.

Made-up data with a built-in confounder

The visits are made up. Because the cost was generated from the repair type, the within-group correlation is close to zero by design, a deliberate teaching example of a confounder.

Think of two things in your life that rise together, like ice cream sales and sunburn. Name a third thing that could drive both.

Practice for this lesson

Check what you learned

Five quick questions. Pick an answer and you will see straight away whether it is right and why.

What does a correlation of plus one mean?

  1. There is no relationship
  2. The points lie on a falling line
  3. The points lie exactly on a rising line
  4. The data is missing

Minus one is the falling line, and zero means no straight-line relationship.

What does r squared tell you?

  1. The share of the variation in cost that the line accounts for
  2. The size of the file
  3. The slope of the line
  4. The number of visits

Here about half of the variation lines up with minutes.

Why was the correlation inside gears close to zero?

  1. Gears has no cost
  2. The line was wrong
  3. The strong overall pattern came from repair types, not minutes within a type
  4. The file had no minutes

Repair type was a confounder.

What is a confounder?

  1. A third factor that drives both numbers
  2. A missing value
  3. A duplicate row
  4. A kind of chart

The type of repair drove both time and price.

Why report the typical miss of a fitted line?

  1. It changes the slope
  2. It shows how rough a prediction is
  3. It cleans the data
  4. It reduces r

Individual visits scatter around the line.

Summary

  • r runs from minus one to plus one and measures straight-line patterns.
  • A fitted line predicts an average; visits scatter around it.
  • Check inside groups: a confounder can create a pattern.
  • Correlation is not causation.

Next lesson

Lesson 18 turns to words: coding free-text survey comments into themes.

Questions

What is a strong correlation?

There is no single cutoff, and it depends on the field. What matters is whether the pattern is useful for the decision, whether it holds inside groups, and how many points support it.

Can I use a line to predict outside the data I have?

Be very careful. A line fitted to a narrow range can fail badly outside it. Predictions beyond the data are guesses.

Does a zero correlation mean the two things are unrelated?

Not necessarily. r only measures straight-line patterns. Two numbers can be linked in a curve and still have r near zero, so look at the points as well.

Progress is saved in this browser.