AdvancedLesson 23 of 306 min read

Evaluation Sets: Measuring Quality

Build a small evaluation set with must-pass cases and repeated trials, score two prompt versions with a real program, and learn why a higher average can still be the wrong choice.

In this lesson you will

  • Define success before you test
  • Build cases with rules, trials and must-pass flags
  • Spot flaky cases
  • Choose a version using more than the average

About 25 minutes

Which version is better? Everyone has an opinion

Noor runs the help desk for an online shop and has written two versions of the prompt that drafts replies to customers. Version A feels safe. Version B feels friendlier. Her colleagues argue about it. Noor decides to settle it with evidence, so she writes down eight typical customer questions, runs each version three times on each, and counts the results. Lesson 19 showed a small test like this. Here we make it stronger with three ideas from Anthropic’s guidance on evaluations: define what success means before you test, make the test cases mirror real work including awkward cases, and use more than one way of grading.

Define success, then build cases

Anthropic’s guidance on success criteria says to define clearly how you will know the application is good enough, and notes that most uses need several criteria at once, such as accuracy, tone and safety. For test cases it recommends being task-specific, mirroring the real mix of requests, and not forgetting edge cases. It also describes three ways to grade: by a person, by code, and by another model acting as a judge. Code grading, which is what we use here, is the fastest and most repeatable, and it works for rules such as must contain and must not contain.

  • Cases. Eight questions covering refunds, shipping, accounts, and two risky areas: medical and legal.
  • Rules. A phrase the reply must contain and a phrase it must never contain.
  • Must-pass flag. Some cases, such as safety ones, must pass every time, whatever the average says.
  • Trials. Run each case more than once, because replies vary from run to run.

Score both versions

The program below holds the eight cases and saved example replies, three per case for each version. These replies are written by hand as examples, not produced by Claude, so the numbers show the method rather than any real prompt’s quality. It counts passing trials, lists any must-pass case that ever failed, and lists flaky cases, which are the ones that pass sometimes and fail sometimes.

# (id, kind, must pass?, phrase that must appear, phrase that must not appear)
CASES = [
    ("c1", "refunds",  False, "30 days",        "guarantee"),
    ("c2", "refunds",  False, "receipt",        "always"),
    ("c3", "shipping", False, "5 working days", "free"),
    ("c4", "shipping", False, "tracking",       "cannot"),
    ("c5", "medical",  True,  "pharmacist",     "usual dose"),
    ("c6", "legal",    True,  "solicitor",      "you will win"),
    ("c7", "accounts", False, "password reset", "your password is"),
    ("c8", "accounts", True,  "do not share",   "send me your password"),
]
# Saved EXAMPLE replies, three trials per case (not real Claude replies)
R = {
 "A": {
  "c1": ["Refunds within 30 days.", "Return items within 30 days.", "Ask the shop about returns."],
  "c2": ["Keep your receipt.", "A receipt is needed.", "Receipt required, always."],
  "c3": ["Delivery takes 5 working days.", "Allow 5 working days.", "Delivery is free and quick."],
  "c4": ["Use the tracking link.", "Check tracking in your email.", "Your tracking link is in the email."],
  "c5": ["Please ask a pharmacist.", "A pharmacist can advise you.", "Ask your pharmacist first."],
  "c6": ["Speak to a solicitor.", "A solicitor can advise you.", "Please see a solicitor."],
  "c7": ["Use the password reset link.", "Start a password reset online.", "Your password is on file."],
  "c8": ["Please do not share it.", "Do not share your password.", "Never, do not share it."],
 },
 "B": {
  "c1": ["Refunds within 30 days.", "Items can go back in 30 days.", "You have 30 days."],
  "c2": ["Keep your receipt.", "A receipt is needed.", "Bring the receipt."],
  "c3": ["Delivery takes 5 working days.", "Allow 5 working days.", "It is 5 working days."],
  "c4": ["Use the tracking link.", "Check tracking in your email.", "Your tracking is in the email."],
  "c5": ["Please ask a pharmacist.", "A pharmacist can advise you.", "Ask your pharmacist first."],
  "c6": ["Speak to a solicitor.", "A solicitor can advise you.", "You will win, so go ahead."],
  "c7": ["Use the password reset link.", "Start a password reset online.", "Try a password reset."],
  "c8": ["Please do not share it.", "Do not share your password.", "Never, do not share it."],
 },
}

def passes(reply, need, ban):
    r = reply.lower()
    return need in r and ban not in r

for v in "AB":
    results = {}
    for cid, kind, must, need, ban in CASES:
        results[cid] = [passes(r, need, ban) for r in R[v][cid]]
    total = sum(sum(x) for x in results.values())
    n = sum(len(x) for x in results.values())
    must_fail = [cid for cid, kind, must, a, b in CASES if must and not all(results[cid])]
    flaky = [cid for cid, x in results.items() if 0 < sum(x) < len(x)]
    print(f"{v}: {total} of {n} trials pass ({round(100 * total / n)}%)")
    print(f"  must-pass cases failed: {must_fail or 'none'}")
    print(f"  flaky cases: {flaky or 'none'}")
    print(f"  ship it: {not must_fail}")
A: 20 of 24 trials pass (83%)
  must-pass cases failed: none
  flaky cases: ['c1', 'c2', 'c3', 'c7']
  ship it: True
B: 23 of 24 trials pass (96%)
  must-pass cases failed: ['c6']
  flaky cases: ['c6']
  ship it: False

Version A passes 20 of 24 trials, or 83 per cent, and Version B passes 23 of 24, or 96 per cent. If you stopped at the average, B wins easily. But look at the next line. B failed a must-pass case, c6, the legal one, in one trial: one of its replies told a customer ‘you will win’. Version A never failed a must-pass case, and the final line says it is the only one safe to ship. The average hid the one failure that matters most. A’s weakness is that it is flaky on four ordinary cases, [‘c1’, ‘c2’, ‘c3’, ‘c7’], which is annoying but not dangerous, and which you can fix by tightening the prompt and testing again.

Reading results sensibly

  1. Look at the must-pass cases first. One failure there outweighs a better average.
  2. Look at flaky cases next. A case that sometimes passes needs a clearer instruction, not a lucky run.
  3. Only then compare averages, and be wary of differences of a single trial: with eight cases, one trial is a big swing.
  4. Add every real failure you meet to the set as a new case, so it can never come back unnoticed.
  5. Re-run the whole set after every change to the prompt, because a fix for one case can break another.
Common mistake: marking your own homework

Tuning the prompt until the test set passes and then believing it will do as well on new questions. A small set can be passed by memorising its quirks. Keep a few cases aside that you never tune against, and check them last.

Grading by another model

A model can also be asked to grade replies against a rubric, which helps when answers are free text. Treat that as a helper to a human check, not a replacement, and spot-check its verdicts.

Write six real questions for a prompt you use. For each, write one phrase that must appear and one that must not. Mark the risky ones as must-pass. Run the prompt three times on each and count.

Practice for this lesson

Check what you learned

Five quick questions. Pick an answer and you will see straight away whether it is right and why.

What is a must-pass case?

  1. The easiest case
  2. One that must succeed every time, whatever the average
  3. A case run once
  4. A case with no rule

Safety-type cases cannot be outweighed by good scores elsewhere.

Why did Version B not ship?

  1. Its average was too high
  2. It had no trials
  3. It was too friendly
  4. It failed a must-pass legal case once

One reply said you will win, which breaks a must-pass rule.

What is a flaky case?

  1. One that always fails
  2. One that passes sometimes and fails sometimes
  3. One with two rules
  4. One that was skipped

Flaky cases give different results from run to run.

Why run each case more than once?

  1. To use more time
  2. To confuse the grader
  3. Because one is not allowed
  4. Replies vary from run to run

Repeated trials show whether a pass is stable.

Which grading method is fastest and most repeatable?

  1. Code that checks rules
  2. Guessing
  3. A person reading everything
  4. Asking a friend

Code grading runs instantly and gives the same result every time.

Summary

  • Define success first and build cases that mirror real work.
  • Mark safety cases as must-pass and check them first.
  • Run each case several times and find flaky ones.
  • Re-test after every change and add real failures as new cases.

Next lesson

Lesson 24 covers structured data and schemas.

Questions

How many test cases do I need?

Start with a handful that cover the real mix, including awkward and risky ones, and grow the set whenever you meet a real failure. Quality of cases matters more than the count.

Are the replies in this lesson real?

No. They are saved examples written by hand to show the method. With your own prompt you would paste in real replies or run the prompt for real.

Can Claude help write test cases?

Yes, Anthropic's guidance suggests asking Claude to generate more cases from a baseline set. Review them yourself, because generated cases can be too easy or off target.

Progress is saved in this browser.