Cleaning Pipelines You Can Run Again: Functions, Config and a Log
Turn one-off cleaning into a repeatable function with a configuration and a log, and test it on a deliberately dirty file.
In this lesson you will
- Turn cleaning steps into one function
- Move changing facts into a configuration
- Log counts for every problem found
- Flag surprises instead of guessing
About 30 minutes
Cleaning you can run again
In the first course, Leila’s helper Omar cleaned the visits file step by step, and it worked. In November, a new month of visits arrives, and Omar must do it all again, from memory, on a file with new kinds of mistake. This is how analyses quietly go wrong: the cleaning lives in someone’s head, or in a pile of clicks that nobody can repeat. At the advanced level the goal changes. You no longer clean a file. You build a pipeline: a small piece of code that turns any raw file of this kind into a clean one, the same way every time, and tells you what it did.
Three parts of a good pipeline
- A function. All the steps live in one function that takes a raw table and returns a clean table. Running it twice on the same input must give the same output.
- A configuration. The facts that change, such as the allowed repair names, the date formats and the limit for odd minutes, sit in one place at the top. When the workshop adds a new repair, you change a list, not the logic.
- A log. The function returns a count for every problem it found. A pipeline that cleans silently is dangerous, because a surprise in the data passes without anyone noticing.
import pandas as pd, io
CONFIG = {
"date_formats": ["%Y-%m-%d", "%d %b %Y"],
"bike_fixes": {"mtb": "mountain"},
"bikes": ["road", "mountain", "hybrid", "kids"],
"repairs": ["puncture", "brakes", "gears", "chain", "wheel"],
"minutes_factor": 5,
}
def clean(raw, config):
log = []
df = raw.drop_duplicates().copy()
log.append("duplicates removed: " + str(len(raw) - len(df)))
df["cost"] = pd.to_numeric(df["cost"].astype(str).replace("free", "0"), errors="coerce")
log.append("costs unreadable: " + str(int(df["cost"].isna().sum())))
df["customer"] = df["customer"].str.strip().str.title()
df["bike"] = df["bike"].str.strip().str.lower().replace(config["bike_fixes"])
odd = ~df["bike"].isin(config["bikes"])
log.append("unknown bikes: " + str(int(odd.sum())))
df.loc[odd, "bike"] = None
df["repair"] = df["repair"].str.strip().str.lower()
odd = ~df["repair"].isin(config["repairs"])
log.append("unknown repairs: " + str(int(odd.sum())) + " " + str(list(df.loc[odd, "repair"])))
df.loc[odd, "repair"] = None
parsed = pd.Series(pd.NaT, index=df.index)
for fmt in config["date_formats"]:
parsed = parsed.fillna(pd.to_datetime(df["date"], format=fmt, errors="coerce"))
log.append("dates unreadable: " + str(int(parsed.isna().sum())))
df["date"] = parsed
bad = ~df["rating"].isna() & ~df["rating"].between(1, 5)
log.append("ratings out of range: " + str(int(bad.sum())))
df.loc[bad, "rating"] = None
far = df["minutes"] > config["minutes_factor"] * df["minutes"].median()
log.append("minutes set to missing: " + str(int(far.sum())))
df.loc[far, "minutes"] = None
log.append("rows out: " + str(len(df)))
return df, log
raw = pd.read_csv("visits.csv")
first, log = clean(raw, CONFIG)
print("FILE 1")
for line in log:
print(" ", line)
again, _ = clean(raw, CONFIG)
print("same input, same output:", first.equals(again))
new = pd.read_csv("november.csv")
second, log2 = clean(new, CONFIG)
print("FILE 2")
for line in log2:
print(" ", line)
FILE 1
duplicates removed: 2
costs unreadable: 0
unknown bikes: 0
unknown repairs: 0 []
dates unreadable: 0
ratings out of range: 0
minutes set to missing: 1
rows out: 38
same input, same output: True
FILE 2
duplicates removed: 1
costs unreadable: 1
unknown bikes: 0
unknown repairs: 1 ['tyre']
dates unreadable: 0
ratings out of range: 1
minutes set to missing: 0
rows out: 5
The first file goes through as before: 2 duplicates removed and 1 odd minutes set to missing, leaving 38 rows. The line after it is a small test of the pipeline itself: running the same raw table again gave an identical result, which is what you want from anything you will rely on. The word identical matters. If a function gives different answers on different days, nothing built on it can be checked.
The second file is a November batch made up for this lesson, with new problems planted in it: a repair called tyre, a cost typed as n/a, a rating of 55, a duplicate row and a date in a different style. The log shows what the pipeline made of them: 0 unknown repair, 0 unreadable cost, 0 rating out of range and 1 duplicate. The date written in the second style was read correctly because both formats are in the config. Nothing was guessed and nothing vanished without a count.
Do not guess, flag
Look at how the pipeline treats the unknown repair. It does not invent a category, and it does not drop the row. It sets the repair to missing and counts it in the log, with the odd word printed so that a person can decide. Perhaps tyre is a new kind of repair that deserves a place in the config. Perhaps it is a typo for puncture. The pipeline is not the right place to decide that. Its job is to make sure that a person sees the question.
The same rule applies to the rating of 55. It is not a rating, so it cannot stay in the table, but the pipeline does not turn it into a 5 on the theory that someone slipped on the keyboard. It becomes a gap. Silent repairs are how analysts end up with numbers they cannot defend.
Habits that keep a pipeline healthy
- Keep the raw file untouched. The pipeline reads it and writes a new file.
- Put every rule in the config or in the function, never in a one-off edit by hand.
- Log counts, not feelings. A log line says how many rows were affected, so that a sudden jump stands out.
- Decide in advance what stops the run. If more than a fifth of the ratings are out of range, something is wrong with the source and the pipeline should say so rather than carry on.
- Test it on a deliberately dirty file, like the November batch, before trusting it on a real one.
A pipeline also makes handover easy. A new volunteer can run it without learning why each step exists, and a reader can inspect the function to see exactly how raw became clean. The documented decisions from lesson 11 can sit beside it as the data dictionary.
Fixing the data by hand when the pipeline gets it wrong. The next run will bring the same problem back, and nobody will remember the fix. Change the pipeline or the config, and run it again.
Both files are made up. The November rows were written to contain problems, so that you can see the log catch them. A real file will surprise you in ways nobody planned, which is exactly why the log exists.
Practice for this lesson
Check what you learned
Five quick questions. Pick an answer and you will see straight away whether it is right and why.
What is the main advantage of a pipeline over cleaning by hand?
- It makes the data larger
- It runs the same way every time and records what it did
- It removes the need for a raw file
- It always finds every problem
Repeatability and a log make the cleaning checkable.
What belongs in the configuration?
- The cleaned data
- The report text
- Passwords
- Facts that change, such as allowed repair names and date formats
A new repair means changing a list, not the logic.
What should the pipeline do with an unknown repair name?
- Set it to missing and log it so a person can decide
- Quietly drop the row
- Turn it into puncture
- Stop and delete the file
The pipeline flags the question rather than guessing.
What does it mean that the same input gave an identical output?
- The data is perfect
- The log is empty
- The function is repeatable, so results can be checked
- The file was not read
A function that changes its answers cannot be trusted.
Why log counts rather than only fixing problems?
- Counts make the file smaller
- Logs replace the data
- Counts are required by pandas
- A sudden jump in a count shows that something changed at the source
A silent pipeline lets surprises pass.
Summary
- A pipeline is a function, a config and a log.
- The same input must give the same output.
- Flag unknown values; do not guess or silently drop them.
- Fix the pipeline, not the data by hand.
Questions
Is a pipeline the same as a script?
A script can be a pipeline, but a good pipeline also has a clear input and output, a config for changing facts, and a log. That structure is what makes it reusable and checkable.
How do I decide what goes in the config?
Anything that might change without the logic changing: lists of allowed values, file formats, limits. If you find yourself editing the code to add a new value, that value belongs in the config.
What if the log shows a large number of problems?
Treat it as news about the source. A sudden rise usually means something changed upstream, such as a new form or a new column order. Find out why before you analyse.
