AdvancedLesson 22 of 306 min read

Testing Your Analysis Code: Small Tests Catch Silent Bugs

Write small tests with answers you can check by hand, run them against a good and a buggy function, and add total checks.

In this lesson you will

  • Write tests with answers worked out by hand
  • Cover boundaries and odd inputs
  • Catch a bug that real data hides
  • Add total checks to results

About 30 minutes

A number that looked right

Sam writes a short function that counts visits per week. It runs without errors, and it prints a table of numbers that look sensible. Three weeks later Leila notices that the Sunday session at an event is counted with the wrong week. Nobody saw it, because the numbers were plausible. This is the worst kind of bug in analysis: it does not crash. It produces an answer, and the answer is wrong. The defence is testing: asking small questions with known answers, before you trust the function with questions whose answers you do not know.

What a test is

A test has three parts: a small input you made up, the answer you expect, and a check that the function gives that answer. The input is small enough that you can work out the answer in your head, which is what makes it trustworthy. You do not test that the function is clever. You test the cases where mistakes hide: the edges.

Where bugs hide Example test
Smallest input One visit gives one week
Boundaries A Sunday and the Monday after it
Rules you rely on Every week starts on a Monday
Totals The counts add up to the rows

The program below holds a good function and a buggy version, five tests, and a tiny runner that applies the tests to either. The buggy version starts its week on Sunday instead of Monday, a one-line difference that is easy to type and hard to see.

import pandas as pd
def weekly_counts(dates):
    dates = pd.to_datetime(pd.Series(dates))
    start = dates - pd.to_timedelta(dates.dt.weekday, unit="D")
    return start.value_counts().sort_index()
def weekly_counts_buggy(dates):
    dates = pd.to_datetime(pd.Series(dates))
    start = dates - pd.to_timedelta((dates.dt.weekday + 1) % 7, unit="D")
    return start.value_counts().sort_index()
TESTS = []
def test(name):
    def add(fn):
        TESTS.append((name, fn))
        return fn
    return add
@test("one visit gives one week")
def t1(f): return list(f(["2026-11-03"]).values) == [1]
@test("two visits in one week are counted together")
def t2(f): return list(f(["2026-11-03", "2026-11-07"]).values) == [2]
@test("a Sunday belongs to the week before")
def t3(f): return list(f(["2026-11-01", "2026-11-02"]).values) == [1, 1]
@test("weeks start on a Monday")
def t4(f): return all(d.weekday() == 0 for d in f(["2026-09-05", "2026-10-20"]).index)
@test("counts add up to the rows")
def t5(f):
    days = ["2026-09-05", "2026-09-08", "2026-09-12", "2026-11-01"]
    return int(f(days).sum()) == len(days)
def run_all(fn):
    failed = []
    for name, check in TESTS:
        try:
            ok = check(fn)
        except Exception:
            ok = False
        if not ok:
            failed.append(name)
    return len(TESTS) - len(failed), failed
passed, failed = run_all(weekly_counts)
print("good version: passed", passed, "of", len(TESTS))
passed, failed = run_all(weekly_counts_buggy)
print("buggy version: passed", passed, "of", len(TESTS))
for name in failed:
    print("  FAILED:", name)
df = pd.read_csv("clean.csv")
print("real file, rows:", len(df), "| weekly total:", int(weekly_counts(df["date"]).sum()))
good version: passed 5 of 5
buggy version: passed 3 of 5
  FAILED: a Sunday belongs to the week before
  FAILED: weeks start on a Monday
real file, rows: 38 | weekly total: 38

The good function passed 5 of 5 tests. The buggy version passed only 3, and the runner names the two that failed: the Sunday rule and the Monday rule. Notice what the tests did not find. The buggy function gives the right totals, and it would give the right weekly counts for the real visits file, because the workshop is not open on Sundays or Mondays. A test on real data alone would have passed it. The tests with invented Sunday and Monday dates caught a bug that real data cannot show yet, but a future Sunday event would have.

The last line applies the good function to the real clean file and checks that its weekly total equals the number of rows: 38 rows, 38 visits counted. A total check like this is cheap and catches a whole family of mistakes, such as dropped or doubled rows.

How to write useful tests

  • Make the expected answer by hand, never by running the function and copying what it says. A test that agrees with the code by construction proves nothing.
  • Name each test as a sentence, as the program does. When a test fails, the name should tell you what rule broke.
  • Cover the empty and the odd. What happens with no rows, one row, a missing date or a value on a boundary?
  • Add a test for every bug you find. The bug that surprised you once will try again. A test is a memory that does not fade.
  • Keep tests fast and small, so that you run them every time you change anything.

Testing the whole analysis

Functions are not the only thing to test. You can also check the results of an analysis against facts that must be true. The weekly counts must add up to all rows. A share must lie between zero and one hundred. A cleaned table must have no duplicate visit numbers. Such checks are cheap to write, and they run on every new file, which makes them a safety net for the data as well as the code.

There are libraries that organise tests, such as the testing tools that come with Python, and they report failures neatly. The idea does not need them. Even a few lines of plain checks, as here, are far better than none.

Common mistake: only real data

Testing only with the real data. If the real data never contains a Sunday, a Sunday bug will pass every time until the day it matters. Invent the awkward cases yourself.

Invented test cases

The dates and the buggy function are made up for this lesson. The workshop is not really open on Sundays. The point is that a test can use any date you invent.

Pick a calculation you rely on, such as a weekly total. Write three tiny inputs where you know the answer, including one awkward edge case.

Practice for this lesson

Check what you learned

Five quick questions. Pick an answer and you will see straight away whether it is right and why.

What is a test, in this lesson?

  1. A chart of the results
  2. A small input with a known answer and a check that the function agrees
  3. A copy of the real data
  4. A cleaning step

The input is small enough to work out by hand.

Why can plausible numbers still be wrong?

  1. Numbers are always right
  2. Tests change the numbers
  3. A bug can produce an answer without crashing
  4. Plausible means checked

The Sunday bug gave a sensible-looking table.

Why did the buggy version pass on the real visits?

  1. The workshop is never open on Sundays or Mondays, so the real data hides the bug
  2. The real data was empty
  3. The tests were wrong
  4. The bug was in the file

Invented awkward cases caught what real data could not.

Where should the expected answer in a test come from?

  1. The function itself
  2. A random number
  3. Worked out by hand, not copied from the function
  4. The log

Copying the function output would agree by construction.

What should you do each time you find a bug?

  1. Add a test for it
  2. Delete the function
  3. Hide the data
  4. Ignore it

A test is a memory that does not fade.

Summary

  • A test is a small input with a known answer.
  • Silent bugs give plausible but wrong numbers.
  • Invent awkward cases that real data does not contain.
  • Check totals, ranges and uniqueness on every run.

Next lesson

Lesson 23 handles odd values: outliers, robust summaries and what not to delete.

Questions

Do I need a testing library?

Not to start. A list of small checks and a loop, as in the lesson, gives most of the benefit. A library helps when you have many tests and want tidy reports.

How many tests are enough?

Enough to cover the rules you rely on and the awkward edges. A handful of well-chosen tests beats a hundred copies of the easy case.

Can tests prove that code is correct?

No. They show that it works on the cases you thought of. That is still valuable, because most bugs live in cases you can think of if you try.

Progress is saved in this browser.