Answering From Your Own Notes: Lookup and Handing Over
Build the idea behind retrieval: find the best note for a question, show the source and hand over to a person when nothing matches.
In this lesson you will
- Explain retrieval before generation
- Match questions to notes by shared words
- Choose a threshold that favours handing over
- Write notes that make answers checkable
About 25 minutes
The question Ravi answers forty times
Ravi’s customers ask the same questions again and again. How long does delivery take? Can I change my order? When do you close? He has answers written down in a handful of short notes, but he still types them out, message after message. An automated assistant could look up the right note and reply. But what if it makes something up? A shop that promises a delivery time it cannot meet is worse off than one that answers slowly. The safe design is one where the assistant answers only from your own notes, shows which note it used and says ‘I will ask a person’ when it cannot find one.
Lookup before writing
This design has a name: retrieval-augmented generation, often shortened to RAG. The idea has three parts: retrieve the notes that look relevant to the question, add them to the request, then generate an answer grounded in them. The grounding is what lets you show the source and lets a person check. Real systems often use clever search that understands meaning, not just words. To keep everything runnable and visible, the program below uses the simplest possible search: it counts the shared words between the question and each note.
A lookup you can read
The program holds five short notes and six saved questions, each with the note a person would choose, or none when no note answers it. Common little words such as ‘the’ and ‘is’ are ignored. A note is only accepted if it shares at least two words with the question. Below that, the assistant hands over to a person, the safest default.
import re
docs = {
"d1": "Delivery takes three working days after you approve the proof.",
"d2": "We print flyers on thick or thin paper. Thick paper costs more.",
"d3": "You can change an order before the proof is approved. After that, changes start a new order.",
"d4": "Our shop opens at nine and closes at five on weekdays.",
"d5": "Pay by card or bank transfer. Invoices are due in fourteen days.",
}
stop = set("a an the is are do does did how i you we our my to of in on at or and by for it can what when after before more long".split())
def words(t):
return {w for w in re.findall(r"[a-z]+", t.lower()) if w not in stop}
def ask(q, need=2):
mine = words(q)
scores = {d: len(mine & words(t)) for d, t in docs.items()}
best = max(scores, key=scores.get)
if scores[best] < need:
return None, scores[best]
return best, scores[best]
# saved questions with the document a person would pick (None = no answer exists)
tests = [
("How long is delivery once I approve the proof?", "d1"),
("Is thick paper dearer than thin paper?", "d2"),
("Can I change my order after the proof is approved?", "d3"),
("When does the shop close?", "d4"),
("Can I pay by bank transfer or card?", "d5"),
("Do you print on glass?", None),
]
right = 0
for i, (q, want) in enumerate(tests, 1):
got, n = ask(q)
right += got == want
print(f"q{i}: {got or 'ask a person'} ({n} shared)")
print("matches a person:", right, "of", len(tests))
print("answerable but missed:", sum(ask(q)[0] is None and w is not None for q, w in tests))
print("wrong answers given:", sum(ask(q)[0] not in (None, w) for q, w in tests))
q1: d1 (3 shared)
q2: d2 (3 shared)
q3: d3 (4 shared)
q4: ask a person (1 shared)
q5: d5 (4 shared)
q6: ask a person (1 shared)
matches a person: 5 of 6
answerable but missed: 1
wrong answers given: 0
The assistant chose the same note as a person in 5 of 6 cases and gave 0 wrong answers. Question 3 matched its note on 4 words, a strong match. Question 6, about printing on glass, correctly found nothing useful and went to a person. Look more carefully at question 4, ‘when does the shop close?’. The note says the shop ‘closes’ at five, but the question says ‘close’, and the program treats those as different words. The assistant said ‘ask a person’ after finding only 1 shared word, below the line. So there was 1 answerable question it missed. This is the weakness of word matching: it does not know that close and closes mean the same. Smarter search methods, often built on numbers that capture meaning, can cope with that, but they have their own failures too, and you test them the same way you test this one.
Why the miss is acceptable, and the wrong answer is not
Notice which kind of mistake happened. The assistant failed to answer a question it could have answered, so a person had to step in. That costs a minute. The opposite mistake, a confident wrong answer, costs trust. A good design therefore leans towards handing over. A higher word threshold makes the assistant more cautious. A lower one answers more but risks wrong notes. You choose by looking at saved examples, which lesson 17 returns to.
Rules for an assistant that answers from notes
- Keep the notes short, current and written in the words customers use.
- Show the note used, so a person can check the answer.
- When no note matches well enough, say so and pass the question to a person.
- Never let the assistant answer about prices, deadlines or refunds from memory. Put them in a note or leave them to a person.
- Review the unanswered questions each week and write new notes for the common ones.
If an AI model writes the final wording, ask it to use only the supplied note, and check its reply with the output checks from lesson 16. An instruction in a prompt reduces invention but cannot prevent it.
Letting the assistant answer anything, using the AI’s general knowledge. It will sound confident about your delivery times and refund rules even though it has never seen them.
The notes, questions and rules are made up for a pretend shop. The search here is a deliberately simple teaching model. Real search and AI services differ by tool, plan and region, and none are quoted or run here.
Practice for this lesson
Check what you learned
Five quick questions. Pick an answer and you will see straight away whether it is right and why.
What does retrieval-augmented generation mean in simple terms?
- Answer from memory only
- Delete old notes
- Find relevant notes first, then answer using them
- Translate every question
Retrieve, add to the request, then generate a grounded answer.
Why show the note the assistant used?
- So a person can check the answer
- To make the reply longer
- To hide errors
- To save time on typing
A visible source makes checking possible.
Why did the shop-closing question miss its note?
- The note was empty
- The question said close and the note said closes
- It was a rush order
- The threshold was zero
Word matching treats close and closes as different words.
Which mistake is cheaper?
- Giving a confident wrong answer
- Inventing a delivery time
- Promising a refund
- Handing over an answerable question to a person
A person can answer in a minute; a wrong answer costs trust.
What should the assistant do when no note matches well?
- Guess
- Say so and pass the question to a person
- Use general knowledge
- Ignore the customer
Handing over is the safe default.
Summary
- Answer only from your own notes, and show the source.
- Word matching is simple and misses variations.
- Handing over costs less than a confident wrong answer.
- Review unanswered questions and write new notes.
Questions
Is this the same as what AI search tools do?
The idea is the same: look up relevant material, then answer from it. Real tools use more advanced search, and they can still be wrong.
How many notes can I keep?
As many as you can maintain. Short, current notes beat long, stale ones. Check your tool for any size limits.
Can the assistant make up an answer anyway?
An AI model can. That is why you show the source, check replies and hand over when no note matches.
