AdvancedLesson 24 of 305 min read

Reliable chains: validation, retries and fallbacks

Turn prompt chains into dependable systems with validation rules, retry loops that feed errors back, human checkpoints, logging and a failure plan.

In this lesson you will

  • Apply the validate, retry, fall back pattern
  • Choose validation rules
  • Feed errors back into retries
  • Plan for failure and logging

About 18 minutes

A chain is only as strong as its weakest link

Danish connects three steps: read an email, extract the order details as JSON, then add a row to a spreadsheet. It works perfectly in his tests. In the first week of real use, one email makes the AI add a friendly sentence before the JSON, the second step fails to read it, and the order silently disappears. In lesson 11 you learned to split jobs into steps. This lesson adds what turns a chain from a demo into something dependable: validation after every step, retries with feedback, and a safe fallback to a person.

The reliable-step pattern

  1. Ask for output in a fixed shape (lesson 12).
  2. Validate it with rules: valid format, required fields, allowed values, sensible ranges.
  3. Retry if it fails, giving the model the exact error and asking for a corrected version.
  4. Limit retries to a small number, such as 2 or 3.
  5. Fall back to a human review queue if all tries fail. Never silently drop or guess.

What to validate

Check Example
Format Is it valid JSON? Does it have the right columns?
Required fields Does every order have a name, item and quantity?
Allowed values Is the label one of PRAISE, COMPLAINT, QUESTION?
Ranges Is the quantity between 1 and 100? Is the date in the future?
Cross-checks Does the total equal quantity times price?
Grounding Is every quoted phrase really in the source text?
Safety Does the output contain forbidden words, links or private data?

The loop in code

The program below shows the loop. A fake model stands in for the real one: it fails twice (once with chatty text, once with single quotes) and then succeeds. No AI service is called. The retry logic and output are real:

import json

# A fake model that fails twice and then succeeds, to demonstrate the loop (no real AI is called)
attempts = iter([
    "Sure! Here is the JSON you asked for: {\"label\": \"COMPLAINT\"}",
    "{'label': 'COMPLAINT'}",
    "{\"label\": \"COMPLAINT\", \"confidence\": 0.9}",
])
def fake_model(prompt):
    return next(attempts)

ALLOWED = {"PRAISE", "COMPLAINT", "QUESTION"}
def validate(raw):
    data = json.loads(raw)
    if data.get("label") not in ALLOWED:
        raise ValueError("label not allowed")
    return data

def classify(message, max_tries=4):
    prompt = f"Return only JSON like {{\"label\": \"PRAISE\"}} for: {message}"
    for n in range(1, max_tries + 1):
        raw = fake_model(prompt)
        try:
            result = validate(raw)
            print(f"try {n}: valid -> {result}")
            return result
        except (json.JSONDecodeError, ValueError) as e:
            print(f"try {n}: rejected ({type(e).__name__}); asking again with the error")
            prompt += f"\nYour last answer was invalid: {e}. Return only valid JSON."
    print("giving up; send to a person")
    return None

classify("My parcel is late")
try 1: rejected (JSONDecodeError); asking again with the error
try 2: rejected (JSONDecodeError); asking again with the error
try 3: valid -> {'label': 'COMPLAINT', 'confidence': 0.9}

Look at what each retry does. It adds the error to the prompt so the next attempt knows what to fix. This is more effective than repeating the same request. After the limit, a real system would route the item to a person.

Feeding errors back

Your last answer could not be used because: invalid JSON (expecting property name in double quotes). Return the corrected output only, in this exact shape:
{"label": "PRAISE" | "COMPLAINT" | "QUESTION"}

Where to put human checkpoints

  • Before anything leaves your organisation (emails, posts, invoices).
  • When validation fails after retries.
  • When confidence is low or the model says it is unsure.
  • For a random sample of passed items, so you can catch silent errors.
  • When money, health, legal or personal data is involved.

Designing for failure

Failure Response
Invalid format Retry with the error, then human queue
Missing required field Ask the source for it or mark UNKNOWN, never invent
Service down or slow Queue the item and try later; tell the user
Model refuses Log it and route to a person
Output looks fine but is wrong Sample checks and test sets (next lesson)

Logging

Keep a log of inputs, outputs, validation results and retries, with personal data removed or protected. When something goes wrong, the log tells you which step failed and how often. It is also the best source of new test cases.

Cost and speed

Each retry costs time and, in paid tools, money. Set a retry limit, keep prompts short, and prefer fixing the prompt when the same error repeats. If 30 percent of items need retries, the prompt is the problem, not the loop.

Practice

Pick a chain you use or can imagine: for example, summarise an email, extract a due date, and add a calendar reminder. Write down, for each step, the output shape, three validation rules and what happens on failure. Then list which step you would put a human check on and why.

Practice answer

The human check usually belongs before the action that has consequences (adding the reminder or sending a message) and on any item where a required value, such as the date, is UNKNOWN. Validation rules for the date step could be: valid date format, not in the past, within the next year.

Common mistake

Retrying forever. A loop with no limit can burn time and money on an input that will never work. Always cap retries and route failures to a person.

Check what you learned

Five quick questions. Pick an answer and you will see straight away whether it is right and why.

What should happen after a step’s output fails validation?

  1. Ignore the failure
  2. Delete the input
  3. Retry with the error, then fall back to a person
  4. Guess the value

Feedback helps; a human is the safety net.

Why feed the error back into the retry?

  1. To make it longer
  2. To hide the failure
  3. To change tone
  4. The model learns what to fix

Specific errors beat repeating the request.

Why cap the number of retries?

  1. To make answers shorter
  2. To avoid wasted time and cost
  3. To skip validation
  4. To avoid logging

Some inputs never work.

What is a good use of a random sample check?

  1. Replace validation
  2. Delete logs
  3. Catch silent errors in passed items
  4. Skip humans

Valid-looking output can still be wrong.

If 30 percent of items need retries, the problem is likely…

  1. The prompt
  2. The loop
  3. The screen
  4. The keyboard

Fix the prompt, not just the retry.

Summary

  • Validate every step’s output.
  • Retry with the exact error, a limited number of times.
  • Fall back to a person; never silently drop or guess.
  • Log and sample-check results.

Next lesson

Next: evaluation sets and regression testing for prompts.

Questions

Do I need code to validate output?

Simple checks can be done by eye or in a spreadsheet. Code helps when volume is high.

How many retries are sensible?

Two to three. More rarely helps and wastes time.

What if validation passes but the answer is wrong?

Use sample checks, grounding rules and test sets to catch it.

Progress is saved in this browser.