Beginner Project: Sort the Print Shop Inbox
Assemble redaction, routing, a confidence line and human review into one small sorter and test it end to end on ten saved emails.
In this lesson you will
- Combine the beginner ideas in one small system
- Test with saved examples that have known answers
- Read wrong-but-caught and wrong-and-sent separately
- Plan a safe launch
About 25 minutes
Putting the pieces together
Ravi now knows the parts: triggers and actions, a mapped process, fields passed between steps, a tidy data mapper, a cautious AI step with a confidence line, a validator, retries, limits and a person in front of risky actions. In this project lesson he assembles them into one small system and tests it end to end on ten saved emails. The aim is not to build something grand. It is to practise the habit of building a little, testing with saved examples and deciding what a person still does.
The plan on one page
- Trigger. A new email arrives in the shop inbox.
- Redact. Replace email addresses in the text before anything else sees it.
- Route. Junk words go straight to junk. Otherwise classify by keywords and compute a confidence.
- Decide. A confidence of 0.70 or more goes through automatically. Anything lower goes to a person.
- Record. Label the email, and keep the label and the route in the log.
Only labelling happens automatically. Nothing is sent to a customer. That keeps the first version safe, as lesson 9 advised.
Test it on ten saved emails
The program below holds ten made-up emails together with the label a person would give each. The route function applies the plan above. For every email we compare the machine’s label with the person’s label and note whether the email went through automatically or to review. This kind of check, using saved examples with known answers, is the most valuable habit in this whole course.
import re
# MADE-UP saved inbox with the label a person would give: (text, hand label)
inbox = [
("How much for 300 flyers?", "sales"),
("Invoice 120 is overdue", "billing"),
("Where is my order? It is late", "support"),
("Price for a banner please", "sales"),
("Where is my invoice? It is late", "billing"),
("My order arrived broken", "support"),
("You won a prize, click here", "junk"),
("Thanks, the cards look great", "other"),
("Can you do a quote and a price for the banner?", "sales"),
("Hello", "other"),
]
words = {"sales": ["quote", "price", "how much", "flyers", "banner"],
"billing": ["invoice", "payment", "overdue", "receipt"],
"support": ["order", "late", "broken", "missing", "where is"]}
def redact(t):
return re.sub(r"[\w.+-]+@[\w-]+\.[\w.]+", "[EMAIL]", t)
def route(text):
low = text.lower()
if "prize" in low or "click here" in low:
return "junk", 1.0, "auto"
score = {k: sum(w in low for w in v) for k, v in words.items()}
total = sum(score.values())
if total == 0:
return "other", 0.0, "review"
best = max(score, key=score.get)
conf = score[best] / total
return best, conf, "auto" if conf >= 0.7 else "review"
right = auto = review = 0
wrong, wrong_auto = [], 0
for text, hand in inbox:
label, conf, how = route(redact(text))
auto += how == "auto"
review += how == "review"
if label == hand:
right += 1
else:
wrong.append(text[:20])
wrong_auto += how == "auto"
print("emails:", len(inbox))
print("handled automatically:", auto)
print("sent to review:", review)
print("labels matching a person:", right, "of", len(inbox))
print("misses:", wrong)
print("wrong and sent out automatically:", wrong_auto)
print("wrong but caught by review:", len(wrong) - wrong_auto)
print("share handled by machine:", f"{auto / len(inbox):.0%}")
print("review minutes a day (made up, 2 each, 20 a day):", round(review / len(inbox) * 20 * 2))
emails: 10
handled automatically: 7
sent to review: 3
labels matching a person: 9 of 10
misses: ['Where is my invoice?']
wrong and sent out automatically: 0
wrong but caught by review: 1
share handled by machine: 70%
review minutes a day (made up, 2 each, 20 a day): 12
There were 10 emails. The machine handled 7 automatically and sent 3 to review, so it handled 70% on its own. Its labels matched the person’s in 9 of 10 cases, and the one miss began with the words ‘Where is my invoice’. The most important lines come next. The number of wrong labels that went out automatically was 0, and the number that were wrong but caught by review was 1. That is the confidence line doing its job: the one message mixing an invoice with a late delivery was uncertain, so it landed in front of a person instead of in the wrong folder. At a pace of 20 emails a day, the review pile would cost about 12 minutes a day, with two minutes each. That is a made-up figure, but notice how much smaller it is than sorting everything by hand.
Why ten examples are not enough
Be honest about this result. The rules were written by looking at these very ten emails, so of course they do well on them. That is called testing on the data you trained on, and it flatters you. Before relying on the system, collect fifty or a hundred real emails, with names and numbers removed, label them by hand, and run the same program. Expect a worse score and read the misses, because they show you which rules to add. Then keep watching: customers change how they write, and a sorter that worked in spring may slip by autumn.
Before you switch it on
- Test on examples the rules have never seen.
- Run it in labelling-only mode for a week, and compare with your own sorting.
- Choose a person who looks at the review pile every day.
- Set a retry limit, a spending limit and an alert for failed runs.
- Write down the way to switch it off, and who may do it.
When these are in place, you can consider adding drafts and then, much later, sending. The Intermediate level of this course shows how to build workflows like this with real tools, how to connect AI steps with care and how to measure whether they are working.
Judging the system only on the examples used to build it. A high score on familiar data tells you little. Always test on fresh examples with known answers.
The ten emails, the labels and the minutes are made up for teaching, the classifier is a stand-in for an AI step and nothing here contacts a real service. Real results will differ, and no promises about savings are made.
Practice for this lesson
Check what you learned
Five quick questions. Pick an answer and you will see straight away whether it is right and why.
Why does the project only label emails and send nothing?
- A safe first version keeps risky actions with people
- Sending is impossible
- Labels are paid
- Customers dislike email
Labelling is easy to undo, sending is not.
What is the most valuable habit shown in the project?
- Skipping tests
- Testing on saved examples with known answers
- Using more tools
- Hiding errors
Known answers let you measure the system.
What did the confidence line do with the invoice and late delivery email?
- Sent it to junk
- Deleted it
- Replied to it
- Sent it to a person because it was uncertain
The wrong label was caught by review.
Why is 9 of 10 not strong evidence?
- Ten is too many
- The rules were written by looking at those same ten emails
- Labels do not matter
- People never label emails
Testing on the data you built from flatters the result.
What should you do before relying on the system?
- Switch on sending
- Remove the review pile
- Raise the confidence line to zero
- Test on fresh labelled examples and run it in labelling-only mode first
Fresh data and a trial period reveal real weaknesses.
Summary
- Build small, label only, and keep people in charge of sending.
- Test with saved examples that have known answers.
- Count wrong labels that went out automatically.
- Test on fresh data before you rely on it.
Questions
Can I build this without code?
Yes. Most platforms let you build the same steps with forms and buttons. The code here is a model so that you understand the logic and can test it.
Why compare against labels a person gave?
Because you cannot judge a machine without a right answer to compare with. Saved, hand-labelled examples give you that.
Are these results real?
No. The emails, labels and minutes are made up, and the classifier is a stand-in. Real results will differ.
