Prompts for Automations: Structured Replies and Checks
Ask an AI step for a strict format and check every reply with a validator before the next step trusts it.
In this lesson you will
- Write a strict prompt for a workflow step
- Read a JSON reply and check it
- Repair harmless slips and reject the rest
- Route failed replies to a person
About 22 minutes
When the reply is not what the next step expected
Ravi’s workflow now has an AI step that returns a label. The next step, which files the email, expects exactly one word such as billing. On Tuesday the AI step replies, ‘Sure! Here you go: billing. Hope that helps.’ The filing step finds no folder with that name and the email vanishes into a default pile. The AI did nothing wrong by its own lights, since it was being friendly. The workflow was the problem, because it trusted free text. The fix is to ask for a structured reply and to check it before using it.
Prompts for automations are different
When you chat with an AI, a loose answer is fine because you read it yourself. In a workflow, the next step is a machine. So an automation prompt should be strict and short about the shape of the answer. A good one states the job, gives the allowed values, names the exact format and forbids extra text. Here is an example of such a prompt.
You sort emails for a print shop.
Reply with JSON only, with exactly two keys:
label: one of sales, billing, support, other
confidence: a number from 0 to 1
Do not add any other text.
Email: Where is my order? It is late.
Even with a prompt this clear, an AI step sometimes adds a greeting, drops a key or invents a label. Prompts reduce these slips but do not remove them. So we add a second layer: a validator, a few lines that check the reply before anything else uses it.
A validator for JSON replies
JSON is a simple text format for labelled data, like {“label”: “sales”}. Most platforms and programs read it easily. The program below holds six saved example replies, written by hand to show common problems, and checks each one. It first tries to read the reply as JSON. If that fails, it hunts for a curly-bracket block inside the text and tries to read that, which repairs the ‘Sure! Here you go’ case. Then it checks that the two expected keys are present, that the label is one of the allowed values and that the confidence is a number between 0 and 1.
import json, re
allowed = {"sales", "billing", "support", "other"}
# SAVED example replies (made up) from an AI step asked for JSON
saved = [
'{"label": "sales", "confidence": 0.92}',
'Sure! Here you go: {"label": "billing", "confidence": 0.81} Hope that helps.',
'{"label": "support"}',
'{"label": "complaint", "confidence": 0.7}',
'{"label": "other", "confidence": 1.4}',
'The email is about an order.',
]
def check(text):
try:
data, how = json.loads(text), "clean"
except json.JSONDecodeError:
m = re.search(r"\{.*\}", text, re.S)
if not m:
return "rejected", "no JSON found"
try:
data, how = json.loads(m.group(0)), "repaired"
except json.JSONDecodeError:
return "rejected", "broken JSON"
if set(data) != {"label", "confidence"}:
return "rejected", "wrong keys"
if data["label"] not in allowed:
return "rejected", "unknown label"
if not isinstance(data["confidence"], (int, float)) or not 0 <= data["confidence"] <= 1:
return "rejected", "bad confidence"
return how, "ok"
tally = {}
for i, text in enumerate(saved, 1):
verdict, why = check(text)
tally[verdict] = tally.get(verdict, 0) + 1
print(f"reply {i}: {verdict} ({why})")
print("tally:", dict(sorted(tally.items())))
print("usable:", tally.get("clean", 0) + tally.get("repaired", 0), "of", len(saved))
reply 1: clean (ok)
reply 2: repaired (ok)
reply 3: rejected (wrong keys)
reply 4: rejected (unknown label)
reply 5: rejected (bad confidence)
reply 6: rejected (no JSON found)
tally: {'clean': 1, 'rejected': 4, 'repaired': 1}
usable: 2 of 6
Of the six replies, 1 was clean, 1 was repaired and 4 were rejected, so 2 of 6 were usable. The second reply had friendly words around valid JSON, and the repair step rescued it. The others failed for four different reasons: a missing key, a label that was not on the allowed list, a confidence of 1.4 which is not between 0 and 1 and a plain sentence with no structure at all. A rejected reply should never be guessed at. It should go to a person or be retried once, and then to a person.
What to do with a rejected reply
- Retry once, since a second try often fixes a slip.
- If it fails again, send the item to a person with the raw reply attached.
- Count rejections each week. A rising count means the prompt or the input has changed.
- Never fill in a missing label yourself with a default that hides the failure.
Many platforms offer an option to request structured output or to parse JSON in a step. Where you have it, use it, and keep the validator anyway, since it protects the later steps from anything unexpected.
Writing a prompt that asks for ‘a short answer’ and expecting a single word. Say exactly which keys or words are allowed, and check the reply every time.
The six replies are saved examples written by hand for teaching, not output from any real service. A real AI step will not make exactly these slips on demand, and its wording will differ. Features for structured output depend on your tool, plan and region.
Practice for this lesson
Check what you learned
Five quick questions. Pick an answer and you will see straight away whether it is right and why.
Why should an automation prompt be strict about the answer shape?
- Machines like long answers
- It makes replies slower
- It removes the need for a workflow
- The next step is a machine that expects exact input
A friendly extra sentence can break a later step.
What does a validator do?
- Checks a reply before anything else uses it
- Writes the prompt
- Sends the email
- Deletes bad replies silently
It confirms the reply has the right keys, labels and ranges.
Why was the reply with a confidence of 1.4 rejected?
- It had too many letters
- It was clean JSON
- A confidence must be between 0 and 1
- It was repaired
The validator checked the range of the number.
What is a sensible response to a rejected reply?
- Guess the label
- Use a default label silently
- Ignore the email
- Retry once, then send to a person
Never hide the failure behind a guessed value.
Are the six saved replies real output from a service?
- Yes, from the latest model
- No, they are hand-written examples for teaching
- Yes, from a competitor
- They are legal documents
They show common slips and are not real service output.
Summary
- Automation prompts state allowed values and an exact format.
- A validator protects the steps that follow.
- Repair harmless slips and reject everything else.
- Rejected replies go to a person, never to a guess.
Questions
What is JSON?
It is a plain text way to write labelled data, such as a label and a confidence, which both people and programs can read.
Will a clearer prompt remove all bad replies?
It reduces them but never removes them. That is why a check after the AI step matters.
Does my platform check replies for me?
Some offer options for structured output. Check its current documentation, and keep your own check even when it does.
