AdvancedLesson 27 of 306 min read

Troubleshooting Bad Answers

Diagnose a bad reply by naming the symptom, link it to a likely cause, change one thing at a time, and let a real program do the first check for length, format, missing items and unsupported numbers.

In this lesson you will

  • Name the symptom of a bad reply
  • Link symptoms to likely causes and fixes
  • Change one thing at a time
  • Check length, format, missing items and numbers by program

About 24 minutes

The reply was wrong and Priya does not know why

Priya asks Claude for a twelve-word summary as bullet points. She gets a long paragraph. She asks again and gets a better format but a missing date. The third time the date is back but the budget figure is wrong. She is changing the prompt at random and getting nowhere. The skill in this lesson is diagnosis: look at the bad reply, name the symptom, link it to a likely cause, and make one change at a time. It is the same skill as fixing a leaking tap: you find where the water comes from before you reach for the spanner.

Symptoms, causes and fixes

  • Too long or too short. Cause: the length was vague, or at the end of a crowded prompt. Fix: give a hard number and say what to cut first.
  • Wrong format. Cause: the format was described loosely. Fix: show one example of the exact shape, and say reply with nothing else if you need clean output.
  • Something missing. Cause: the requirement was buried or only implied. Fix: list each required item by name, or turn the request into a short checklist.
  • A figure or fact not in the source. Cause: Claude filled a gap from general knowledge. Fix: ground it in the source, and allow not stated as an answer.
  • Off topic or wrong tone. Cause: the audience or purpose was not stated. Fix: name who it is for and what they will do with it.
  • Different every time. Cause: the task leaves room for many answers. Fix: tighten the task, add an example, and test with several runs as in lesson 23.

Let a program do the first look

Many symptoms can be spotted mechanically, which is quicker and more honest than a hunch. The program below holds five saved example replies, each with the rules the prompt asked for: a word limit, a format, required items and a source text. It checks the length, tests the format, looks for each required item, and flags any number that is not in the source. Then it counts the problems and suggests the first fix for the most common one. The replies are written by hand as examples, not produced by Claude.

import json, re

SOURCE = "The fair is on Saturday 17 June. The budget is 400 pounds."
# (what was asked, what came back); saved EXAMPLES, not real replies
CASES = [
    ({"max_words": 12, "format": "bullets", "include": ["17 June"]},
     "- Fair on 17 June\n- Budget 400 pounds"),
    ({"max_words": 12, "format": "bullets", "include": ["17 June"]},
     "The fair takes place on Saturday 17 June and the budget for the whole day is 400 pounds, as agreed."),
    ({"max_words": 30, "format": "json", "include": ["budget"]},
     "Sure! Here you go: budget is 400 pounds."),
    ({"max_words": 30, "format": "prose", "include": ["17 June", "budget"]},
     "The fair is on 17 June."),
    ({"max_words": 30, "format": "prose", "include": ["budget"]},
     "The budget is 450 pounds."),
]
FIX = {
    "too long": "state a hard limit and say what to cut",
    "wrong format": "show one example of the format",
    "missing": "name each required item in the prompt",
    "number not in source": "ground in the source, allow 'not stated'",
}

def diagnose(want, reply):
    found = []
    if len(reply.split()) > want["max_words"]:
        found.append("too long")
    if want["format"] == "json":
        try:
            json.loads(reply)
        except ValueError:
            found.append("wrong format")
    if want["format"] == "bullets" and not all(ln.startswith("- ") for ln in reply.splitlines()):
        found.append("wrong format")
    if any(k.lower() not in reply.lower() for k in want["include"]):
        found.append("missing")
    if any(n not in SOURCE for n in re.findall(r"\d+(?:\.\d+)?", reply)):
        found.append("number not in source")
    return found

counts = {}
for i, (want, reply) in enumerate(CASES, 1):
    f = diagnose(want, reply)
    print(f"case {i}: " + (", ".join(f) if f else "ok"))
    for x in f:
        counts[x] = counts.get(x, 0) + 1
print("most common:", max(counts, key=counts.get))
print("first fix to try:", FIX[max(counts, key=counts.get)])
case 1: ok
case 2: too long, wrong format
case 3: wrong format
case 4: missing
case 5: number not in source
most common: wrong format
first fix to try: show one example of the format

Case 1 is fine. Case 2 is too long and the wrong format, because bullets were requested and a paragraph came back. Case 3 should have been JSON and began with the chatty line ‘Sure! Here you go’, which breaks a parser. Case 4 left out the budget. Case 5 gave a budget of 450 pounds, while the source says 400, a number that is not in the source. The most common problem was the wrong format, so the first fix to try is: show one example of the format. Notice that the program never asked why the reply failed. It just told you what, and the fix list does the rest. Changing one thing and running the test again tells you whether you found the true cause.

A calm troubleshooting routine

  1. Keep the bad reply. Do not delete it, because it is your evidence.
  2. Name the symptom in one phrase, using the list above.
  3. Pick the matching fix and change only that.
  4. Run the same input again, ideally several times, and compare.
  5. If it is fixed, add the case to your test set. If not, try the next likely cause.
Common mistake: changing everything at once

Rewriting the whole prompt after one bad reply. You change ten things, the reply improves, and you do not know which change helped. The next failure will be a mystery again.

Ask Claude for ideas

You can also ask Claude to help diagnose: paste the prompt and the bad reply and ask what in the prompt could have led to this. Treat the answer as a suggestion to test, not as a fact about how it works.

Find a reply that disappointed you recently. Name the symptom. Choose the matching fix. Change only that and run the same input once more.

Practice for this lesson

Check what you learned

Five quick questions. Pick an answer and you will see straight away whether it is right and why.

What is the first step when a reply is wrong?

  1. Delete it
  2. Rewrite the whole prompt
  3. Keep it and name the symptom
  4. Ask again at random

The bad reply is evidence, and naming the symptom points to a fix.

Why change only one thing at a time?

  1. So you know which change helped
  2. It is faster to type
  3. Claude requires it
  4. It saves paper

One change per test shows what actually mattered.

A number that is not in the source suggests what cause?

  1. The prompt was too short
  2. The format was JSON
  3. Claude filled a gap from general knowledge
  4. The reply was too long

The fix is to ground in the source and allow not stated.

Which fix helps a wrong format?

  1. Show one example of the exact shape
  2. Ask for more words
  3. Remove the source
  4. Say please

An example removes doubt about the shape you want.

Why add a fixed case to the test set?

  1. To make the set longer
  2. So it cannot come back unnoticed
  3. To change the prompt
  4. To save time

Every real failure becomes a permanent test.

Summary

  • Keep the bad reply and name the symptom.
  • Match the symptom to a likely cause and a fix.
  • Change one thing at a time and re-run.
  • Add real failures to your test set.

Next lesson

Lesson 28 covers using Claude in professional life.

Questions

What if the same prompt gives different answers each time?

That is normal, because replies vary. If it matters, tighten the task, add an example, and judge with several runs rather than one.

Can Claude diagnose its own bad answer?

It can offer ideas about what in the prompt may have caused it, which are worth testing. It cannot see its own workings, so test each idea.

Are the replies in the program real?

No. They are saved examples written by hand to show each symptom. Your own replies will differ.

Progress is saved in this browser.