AdvancedLesson 25 of 305 min read

Evaluation sets and regression testing

Build a saved test set, choose checks, run it after every change with a regression runner and read failures to improve prompts safely.

In this lesson you will

  • Build an evaluation set
  • Choose rule, reference and quality checks
  • Detect regressions with a runner
  • Avoid misleading metrics

About 18 minutes

Did the change help, or just move the problem?

Ume improves her customer-reply prompt so that it handles complaints better. She tries two complaints and both replies look great. A week later a colleague notices that the new version has started promising refunds the shop does not offer. Her improvement fixed one thing and broke another. Software engineers call this a regression, and they prevent it with regression testing: a saved set of test cases that is re-run after every change. You can do exactly the same for prompts, and it is the habit that separates careful prompt work from guessing.

Building an evaluation set

  1. Collect real cases. Use anonymised real inputs wherever you can: typical ones, awkward ones, and every failure you meet from now on.
  2. Cover the edges. Include very short, very long, angry, confusing, off-topic and trick inputs.
  3. Decide the checks. For each case, what makes an answer acceptable? Write it down before you see the output.
  4. Store it. A spreadsheet works: ID, input, expected behaviour, notes.
  5. Grow it. Each time something goes wrong in real use, add it as a new case.

Types of checks

Type Example How
Rule checks Under 60 words; contains next step; no forbidden promise Automatic (code or formulas)
Reference checks Key facts match the source Automatic or by eye
Quality ratings Tone, helpfulness, clarity Human rubric, 1 to 5
Safety checks No private data, no harmful instructions Automatic plus human sampling
Comparison Is v2 better than v1 on the same cases? Side-by-side scoring

A regression runner in code

The program below checks three test cases against the outputs of two prompt versions. The outputs are pretend, written for the demo, but the checking logic is real. It reports the pass count for each version and lists regressions: cases that passed before but fail now. The output is real:

test_cases = [
    {"id": "T1", "input": "Late parcel", "must_contain": ["sorry"], "max_words": 60, "forbidden": ["refund guaranteed"]},
    {"id": "T2", "input": "Wrong colour", "must_contain": ["replace"], "max_words": 60, "forbidden": ["refund guaranteed"]},
    {"id": "T3", "input": "Where is my order?", "must_contain": ["order"], "max_words": 40, "forbidden": []},
]
# Pretend outputs from two prompt versions (illustrative, written for this demo)
outputs = {
    "v1": {"T1": "We are sorry your parcel is late. Refund guaranteed within 2 days.",
           "T2": "Thanks for telling us. We will replace the candle.",
           "T3": "Your order is on its way."},
    "v2": {"T1": "We are sorry your parcel is late. We are checking with the courier now.",
           "T2": "We will replace the candle at no cost and send a new one this week.",
           "T3": "Your order is packed and the courier has it."},
}
def check(case, text):
    problems = []
    for word in case["must_contain"]:
        if word not in text.lower():
            problems.append(f"missing '{word}'")
    if len(text.split()) > case["max_words"]:
        problems.append("too long")
    for bad in case["forbidden"]:
        if bad in text.lower():
            problems.append(f"forbidden '{bad}'")
    return problems

results = {}
for version, outs in outputs.items():
    results[version] = {c["id"]: check(c, outs[c["id"]]) for c in test_cases}
    passed = sum(1 for p in results[version].values() if not p)
    print(f"{version}: {passed}/{len(test_cases)} passed")
    for cid, probs in results[version].items():
        if probs:
            print("   ", cid, "->", ", ".join(probs))
regressions = [c for c, p in results["v2"].items() if p and not results["v1"][c]]
print("Regressions (passed in v1, fail in v2):", regressions or "none")
v1: 2/3 passed
    T1 -> forbidden 'refund guaranteed'
v2: 3/3 passed
Regressions (passed in v1, fail in v2): none

Version 2 passes all three cases and has no regressions, while version 1 fails one case because it promised a guaranteed refund. Without the saved cases you might never have noticed the promise.

Good to know

Real model outputs vary between runs. For important prompts, run each test case two or three times and look at the worst result, not only the best.

Metrics that mislead

  • One average. A high average can hide a disastrous failure in one case. Look at the worst cases and any safety fails separately.
  • Tiny test sets. Five cases can flatter you. Aim for 20 to 50 on important prompts.
  • Testing on what you tuned on. Keep a few unseen cases aside as a final check.
  • Length bias. Longer answers often look better to people and to AI judges. Include a length limit.
  • Changing the answer key to fit. Decide expected behaviour first.

When to run the tests

  • Before and after every prompt change.
  • When you switch to a different model or tool version, since behaviour can change.
  • On a regular schedule for live systems, because services and models are updated over time.
  • After adding or changing facts, policies or templates.

Reading failures

A failed test is information. Group failures by cause: missing context, ambiguous instruction, format slip, invented fact, tone. If most failures share a cause, one prompt change may fix many. If failures are scattered and random, the prompt may be unstable, and you need constraints, examples or a stricter format.

Practice

Take a prompt you rely on. Write eight test cases: three typical, three awkward and two traps (for example a request for something your policy forbids). For each, write the check before running. Run, score and list failures. Change one thing, re-run all eight, and check for regressions.

Practice answer

A common finding is that fixing the awkward cases makes a typical case worse, usually because a new rule conflicts with an old one. That is a regression, and it is exactly why you re-run every case after every change.

Common mistake

Testing only the cases that failed last time. A fix for one case can break another. Always re-run the whole set.

Check what you learned

Five quick questions. Pick an answer and you will see straight away whether it is right and why.

What is a regression?

  1. A new feature
  2. A longer prompt
  3. A case that passed before but fails after a change
  4. A faster reply

Improvement in one place broke another.

When should you decide the expected result?

  1. Before seeing the output
  2. After picking favourites
  3. Never
  4. Only for failures

It avoids bending the key to fit.

Why include trap cases?

  1. To confuse users
  2. To check rules hold when pushed
  3. To lengthen the set
  4. To skip checks

Policy and safety rules need testing.

Why keep unseen cases aside?

  1. To save space
  2. To hide failures
  3. To shorten the set
  4. To avoid tuning only to the test

They give an honest final check.

A high average score can hide…

  1. Nothing
  2. A serious failure in one case
  3. All errors
  4. A long prompt

Look at the worst cases separately.

Summary

  • Keep a saved set of real, awkward and trap cases.
  • Decide expected results first.
  • Re-run everything after every change to catch regressions.
  • Study failures by cause.

Next lesson

Next: context management, cost and speed.

Questions

How big should the evaluation set be?

Start with 20 to 50 cases for important prompts and grow it with every failure you meet.

Can AI generate test cases?

It can suggest ideas, but real cases from real use are more valuable. Review any generated cases yourself.

Why re-test after a model update?

Tools and models change over time, and behaviour can change with them.

Progress is saved in this browser.