Logs, Monitoring and Alerts
Read a run log, work out failure rates and median times and set alerts that tell you when a workflow breaks, including when it goes silent.
In this lesson you will
- Decide what to log and what to leave out
- Work out failure rates and median times
- Use timing clues to guess causes
- Design alerts people will act on
About 22 minutes
Finding out on Friday that it broke on Tuesday
Ravi’s sorter had been quietly failing for three days. Some emails were not labelled, and no error message told him. He found out when a customer asked why a quote request had been ignored. Automations run when nobody is looking, which is their whole point, so they need ways to tell you when something is wrong. Those ways are logging, monitoring and alerts.
What to log
A log is a record of what happened, one line per event. Each line should say when, which step, whether it worked and how long it took. Keep out the private details such as full email text or contact information; an identifier is enough to look the item up elsewhere. Most platforms keep a run history for you. Learn where yours is and how long it keeps entries, because history that is gone cannot help.
Read a log with a short program
Here is a made-up log of twenty lines covering ten runs of two steps, a classify step and a draft step. The program splits each line, then works out, for each step, how many runs there were, how many failed, the middle time of the successful runs, and whether the failure rate is high enough to raise an alert. The alert rule is a made-up one: raise an alarm when 20 per cent or more of runs fail.
from statistics import median
# MADE-UP log: time, step, status, milliseconds
log = """
09:00 classify ok 410
09:00 draft ok 900
09:05 classify ok 380
09:05 draft ok 950
09:10 classify fail 5000
09:10 draft ok 870
09:15 classify ok 420
09:15 draft fail 1200
09:20 classify ok 400
09:20 draft ok 910
09:25 classify fail 5000
09:25 draft ok 880
09:30 classify ok 390
09:30 draft ok 940
09:35 classify ok 430
09:35 draft ok 890
09:40 classify fail 5000
09:40 draft ok 920
09:45 classify ok 395
09:45 draft ok 930
""".strip().splitlines()
rows = [l.split() for l in log]
alerts = 0
for step in ("classify", "draft"):
mine = [r for r in rows if r[1] == step]
fails = sum(r[2] == "fail" for r in mine)
rate = fails / len(mine)
ok_ms = [int(r[3]) for r in mine if r[2] == "ok"]
slow = sum(int(r[3]) > 3 * median(ok_ms) for r in mine)
flag = rate >= 0.20
alerts += flag
print(f"{step}: {len(mine)} runs, {fails} failed ({rate:.0%})")
print(f" median ok time: {median(ok_ms):.0f} ms, slow runs: {slow}")
print(" alert:", "yes" if flag else "no")
print("alerts raised:", alerts)
classify: 10 runs, 3 failed (30%)
median ok time: 400 ms, slow runs: 3
alert: yes
draft: 10 runs, 1 failed (10%)
median ok time: 910 ms, slow runs: 0
alert: no
alerts raised: 1
The classify step failed 3 times out of ten, a failure rate of 30 per cent, over the line, so it raised an alert. The draft step failed 1 time, 10 per cent, under the line, so it did not. The median time for a successful classify run was 400 milliseconds. The 3 slow runs the program counted are the failed ones, which took 5000 milliseconds, since they ran until a timeout. That detail is a clue: classify failures are slow failures, which suggests the step waited for something and gave up, so the cause may be a busy service, not a bad email. The program raised 1 alert in total. Logs do not tell you the answer, but they point your questions in the right direction.
Why the median and not the average
The median is the middle value. It resists a few extreme values. If one run in ten takes five seconds while the rest take about 400 milliseconds, the average climbs noticeably, but the median stays near 400. For typical behaviour, use the median, and look separately at the slowest runs.
Alerts people will not ignore
- Alert on things that need action, such as a failure rate over the line, no runs at all in a day or a step that stopped working.
- Send alerts to a person who will look, by a channel they actually read.
- Say in the alert what failed, since when and what to try first.
- Avoid constant false alarms, because people learn to ignore a noisy alert.
- Also alert on silence. A workflow that stopped running produces no errors at all.
Add a regular look too. Five minutes each week with the run history is often enough to notice a creeping problem, such as a step that is getting slower, before it becomes a failure.
Having no alerts and relying on customers to tell you. By the time they do, the damage is days old. Set at least one alert on failure rate and one on silence.
The log lines, times and alert line are made up. Platforms offer run histories and alerts with their own names, retention and limits, which depend on tool and plan, so check yours. Keep personal details out of logs, and check privacy rules that apply to you. This is an outline and not legal advice.
Practice for this lesson
Check what you learned
Five quick questions. Pick an answer and you will see straight away whether it is right and why.
What is a log?
- A type of workflow
- A customer list
- A password file
- A record of what happened, one line per event
Each line says when, which step, whether it worked and how long it took.
Why did the classify step raise an alert?
- It was the slowest step
- Its failure rate was over the 20 per cent line
- It ran too few times
- It had no log
Three failures in ten runs is above the line.
Why use the median time?
- It is always bigger
- It includes failures
- It is the fastest run
- A few extreme values do not distort it
The middle value shows typical behaviour.
What did the five-second failures suggest?
- The step waited for something and gave up
- The email was empty
- The draft step was broken
- The log was wrong
Slow failures hint at a timeout rather than bad input.
Why alert on silence as well as on errors?
- Silence is always fine
- Errors never happen
- A stopped workflow produces no errors
- Alerts are free
A workflow that stops running leaves nothing to fail.
Summary
- Logs say when, which step, whether it worked and how long it took.
- A failure-rate line turns a log into an alert.
- Medians show typical behaviour.
- Alert on silence as well as errors.
Questions
How long should I keep logs?
Long enough to investigate problems, and no longer than you need, especially if logs touch personal data. Check your platform's retention and privacy rules.
Is a 20 per cent line right for everything?
No, it is a made-up example. Choose lines to suit how costly a failure is for each step.
Do I need a program to read logs?
No. Many platforms show filters and counts. The program shows the logic behind those screens.
