Testing and Improving Your Prompts
Learn to treat a prompt like something you can test: a small test set, countable checks, one change at a time and a log, with a real harness that scores two prompt versions.
In this lesson you will
- Explain why one reply is not evidence
- Build a small test set with countable checks
- Compare versions fairly
- Keep a log of changes
About 22 minutes
Which prompt is actually better?
Anil has two versions of the prompt he uses for the monthly newsletter, and he cannot decide which is better. Version B feels more specific. Version A feels simpler. Every time he asks a question he gets a slightly different reply, so one lucky or unlucky answer proves nothing. He is about to choose by gut. This lesson shows a better way: treat a prompt like something you can test, with a small set of examples and a few clear checks, so that changes become improvements you can measure.
Why one reply is not evidence
Claude’s replies vary from one try to the next, so a single reply tells you little. A prompt that looks excellent on one example may fail on the next. Anthropic’s own guidance on strengthening prompts encourages defining what success looks like and building evaluations. At a personal scale you can do the same, simply: choose a handful of realistic inputs, decide what a good answer must do, and check every answer the same way.
A small test set
- Choose four to ten real inputs. Include an easy one, a hard one and an odd one.
- Write down the checks. Make them clear and countable: under 25 words, includes a date, no filler phrases.
- Run both versions on the same inputs. Save every reply.
- Score each reply against the checks. Count passes.
- Change one thing at a time. If you change the whole prompt, you will not know what helped.
- Re-run the tests after every change, so you notice when a fix breaks something else.
- Read some replies yourself. Checks catch what you can count, and your eyes catch what you cannot.
The program below is a tiny test harness. It holds four test inputs and, for each of two prompt versions, a saved reply. These replies are labelled examples written for this lesson, since Claude cannot be called from the lesson’s program. In your own testing, they would be real replies you saved. Three checks are applied to every reply: it must be 25 words or fewer, it must mention a date or a day, and it must avoid filler phrases like ‘lots of’ and ‘great fun’. The harness then adds up the passes by version and by check.
import re
tests = ["bake sale email", "library newsletter", "school fair poster", "volunteer thank-you"]
replies = { # saved example replies for two prompt versions (labelled examples)
"A": ["Dear parents, we are holding a bake sale on Saturday. Please come along and bring friends.",
"The library newsletter this month covers lots of news and several events and many other things that are happening across the whole village and beyond for everyone to enjoy.",
"Come to the school fair! Great fun for all.",
"Thank you for helping. It was great."],
"B": ["Dear parents, our bake sale is on Saturday 10 June, 10am to 2pm. Please bake or help for an hour.",
"This month: a new reading club on Wednesdays, a book sale on 10 June, and a quiet hour for studying.",
"School fair, Saturday 10 June, 11am. Games, tea and a raffle. All welcome.",
"Thank you for giving your time on Saturday. Your help raised money for the library."],
}
MAX_WORDS = 25
def checks(text):
return {
"short": len(text.split()) <= MAX_WORDS,
"has a date or day": bool(re.search(r"\b(saturday|wednesdays|june|monday)\b", text, re.I)),
"no filler": not re.search(r"\b(lots of|many other things|great fun)\b", text, re.I),
}
for version, rs in replies.items():
results = [checks(r) for r in rs]
total = sum(sum(c.values()) for c in results)
print(f"version {version}: {total} of {3 * len(rs)} checks passed")
for name in results[0]:
n = sum(c[name] for c in results)
print(f" {name}: {n} of {len(rs)}")
a = sum(sum(checks(r).values()) for r in replies["A"])
b = sum(sum(checks(r).values()) for r in replies["B"])
print()
print("better version:", "B" if b > a else "A" if a > b else "tie")
version A: 6 of 12 checks passed
short: 3 of 4
has a date or day: 1 of 4
no filler: 2 of 4
version B: 12 of 12 checks passed
short: 4 of 4
has a date or day: 4 of 4
no filler: 4 of 4
better version: B
Version A passes 6 of 12 checks, and version B passes 12. The detail shows why. Version A’s replies were short enough in 3 of 4 cases, mentioned a date or day in only 1 of 4, and avoided filler in 2 of 4. Version B passed every check on every input. So the numbers back up Anil’s gut feeling, and they also tell him what is better: version B, which asks for specific details, stops the replies from being vague. That is the real prize of testing. You learn not only which version wins, but which instruction is doing the work, so you can keep it and reuse it.
A caution: four examples are a start, not a proof. Better to test ten, and to keep your set growing as you meet new cases, especially ones that went wrong. A reply that failed once is the most valuable test of all, so add it.
Common improvements to try
- Add an example of a good answer.
- State the limit with a number, such as under 25 words.
- List what must be included, such as the date, the place and one action.
- List what must not appear, such as filler or jargon.
- Say what to do when information is missing, such as use square brackets.
- Split a big task into two prompts, one to plan and one to write.
You will write a one-line newsletter item. Rules: 25 words or fewer; include the day or date; no filler such as "lots of" or "great fun". If I have not given you a date, put [DATE] in its place.
Item: the school fair on Saturday 10 June, games, tea and a raffle.
School fair, Saturday 10 June, 11am. Games, tea and a raffle. All welcome.
Changing many things at once, or judging by one lucky reply. If you rewrite the prompt completely and it improves, you do not know why. If it gets worse, you do not know why either. One change, one test.
Keep your prompts and results in a simple table: version, change, score, notes. After a few weeks, the table becomes the best guide you have to what works for your tasks.
Practice for this lesson
Check what you learned
Five quick questions. Pick an answer and you will see straight away whether it is right and why.
Why is one reply poor evidence about a prompt?
- Claude only replies once
- Replies are always identical
- Prompts cannot change
- Replies vary from try to try
Because replies differ, you need several examples to judge a prompt.
Which check is countable?
- Sounds nice
- Under 25 words
- Feels warm
- Is interesting
Clear, countable checks can be scored the same way every time.
What did the test show about version B?
- It failed everything
- It was the same as A
- It passed every check on every input
- It was longer
Version B passed all twelve checks, while version A passed fewer.
Why change one thing at a time?
- So you know what caused any difference
- To save time
- To lose track
- To avoid testing
One change per test shows which change helped or hurt.
Which test input is most valuable to add?
- Only easy ones
- The same one again
- One where a reply failed before
- A random word
Past failures catch the cases most likely to fail again.
Summary
- Replies vary, so test prompts on several inputs.
- Use clear checks you can count, and change one thing at a time.
- Testing shows which instruction is doing the work.
- Add past failures to the test set and keep a log.
Questions
How many test inputs do I need?
Start with four to ten realistic ones, including an easy, a hard and an odd case. Add more as you meet new failures.
Can I test prompts automatically?
Yes, with a program like the one here, if you save the replies. Developers can also test through Anthropic's API, but that is beyond this lesson.
Do tests replace reading the replies?
No. Checks catch what you can count, and your own reading catches what you cannot.
