Coding Free-Text Survey Comments into Themes
Turn customer comments into countable themes with word rules, check them against a person, and report how well they agree.
In this lesson you will
- Define themes for free text
- Write and improve word rules
- Measure agreement with a person
- Report counts honestly with their limits
About 28 minutes
What do customers actually say?
At the end of each visit, Omar leaves a card for a comment. Over a few weeks, twenty comments have piled up in the cash box. Numbers like ratings are easy to count, but comments hold the reasons. Reading all of them is possible with twenty, and impossible with twenty thousand. The skill is to turn free text into a few themes that can be counted, without losing the sense of what people said. This is called coding the comments, and researchers do it by hand with great care.
Rules, and a person to check them
A simple way to start is a set of keyword rules: if a comment contains a word from the price list, it is about cost. If it contains a word about waiting, it is about speed. The danger is that rules are crude and sound more precise than they are. So never trust the rules until a person has coded some comments independently, and the two have been compared. The program below does exactly that: a person’s labels for all twenty comments, a first set of word rules, a count of how often they agree, and a second set of rules after the misses are studied.
- Read the comments and agree a short list of themes, such as people, cost, speed and opening hours.
- Have a person label each comment, ideally two people, and keep the labels as the standard.
- Write the word rules, run them, and compare with the person.
- Study every miss. Add words, change the order or accept the limit.
- Count themes only after the rules agree well enough for the purpose, and say how well they agree.
comments = [
"friendly volunteers and a fair price", "waited too long on a Tuesday", "kind people, fixed my brakes", "cheap and quick",
"wish you opened on Sundays", "the volunteer taught me to fix a puncture", "queue was slow", "free repair, thank you",
"can you open later in the evening", "patient and helpful", "too slow but a lovely welcome", "good value donation box",
"opening hours are tricky", "fast service", "they explained everything", "long wait for parts",
"great price", "open on more days please", "nice team", "no complaints",
]
truth = ["people", "speed", "people", "cost", "hours", "people", "speed", "cost", "hours", "people",
"speed", "cost", "hours", "speed", "people", "speed", "cost", "hours", "people", "none"]
rules = {"cost": ["price", "cheap", "free", "value"],
"speed": ["wait", "slow", "quick", "queue"],
"people": ["friendly", "kind", "volunteer", "patient"],
"hours": ["open", "evening", "hours"]}
def code(text, rules):
for theme, words in rules.items():
if any(w in text for w in words):
return theme
return "none"
first = [code(c, rules) for c in comments]
agree = sum(a == b for a, b in zip(first, truth))
print("first rules agree with a person:", agree, "of", len(comments))
for c, a, b in zip(comments, first, truth):
if a != b:
print("miss: " + a + " vs " + b + " | " + c)
rules["speed"] += ["long", "fast"]
rules["people"] += ["helpful", "team", "welcome"]
rules["hours"] += ["days"]
second = [code(c, rules) for c in comments]
agree2 = sum(a == b for a, b in zip(second, truth))
print("after adding words:", agree2, "of", len(comments))
for c, a, b in zip(comments, second, truth):
if a != b:
print("still wrong: " + a + " vs " + b + " | " + c)
from collections import Counter
for theme, n in Counter(second).most_common():
print(theme + ":", n)
first rules agree with a person: 16 of 20
miss: cost vs people | friendly volunteers and a fair price
miss: none vs speed | fast service
miss: none vs people | they explained everything
miss: none vs people | nice team
after adding words: 18 of 20
still wrong: cost vs people | friendly volunteers and a fair price
still wrong: none vs people | they explained everything
cost: 5
speed: 5
people: 4
hours: 4
none: 2
The first rules agreed with the person on 16 of 20 comments. The four misses teach something. Fast service and nice team had no word on any list, so the rules found nothing. They explained everything is about people, but contains none of the people words. And friendly volunteers and a fair price mentions two themes. The rules pick whichever theme comes first in the list, here cost, while the person chose people.
After adding a few words, agreement rose to 18 of 20. Two misses remain, and they are the instructive ones. The comment about explaining everything still has no keyword, because the idea is expressed in a way that no short word list captures. The comment with two themes cannot be solved by one label at all. Real survey work lets a comment carry more than one theme, so that this one counts as both people and cost.
How well is good enough?
Agreement of 18 out of 20 is 90 percent. Whether that is acceptable depends on what you are doing. For a quick look at what customers mention most, it is plenty. For a report that says “exactly 5 comments were about cost”, it is not, because each miss shifts the counts. A fair report says something like “about a quarter of comments mention cost, from word rules that agreed with a person on 18 of 20”.
The final counts are cost 5, speed 5, people 4, hours 4 and none 2. Look at the pattern, not the exact numbers: cost and speed are mentioned a little more often than people and hours, but with twenty comments one comment moves a share by five points, so the differences are small.
Cautions about words
- Comments come from the people who write them. Customers who bother to write are often the very happy or the very unhappy. Silence is not agreement.
- Do not hunt for the quote that proves your idea. Choose representative comments, and show at least one that does not fit your story.
- Protect people. Comments can name individuals. Remove names and details before you share them beyond the team.
- Check the rules on new comments. Rules built on twenty comments will fail on the twenty-first. A fresh set shows how well they really work.
- Tools that read meaning, such as language models, can code comments too. The same discipline applies: compare with a person’s labels on a sample and report the agreement.
Trusting keyword rules because the output looks tidy. A neat table of counts hides every miss. Always check against a person and report the agreement.
The twenty comments and the person’s labels are made up for the lesson. A real project would have two people label the comments independently and compare their labels before judging the rules.
Practice for this lesson
Check what you learned
Five quick questions. Pick an answer and you will see straight away whether it is right and why.
What does coding comments mean here?
- Writing a computer program
- Hiding names
- Sorting by date
- Turning free text into a few themes that can be counted
Themes make words countable.
Why compare the rules with a person?
- People are always right
- Rules are crude and can be wrong without showing it
- Rules are illegal
- It makes the file smaller
The person gives you a standard to measure the rules against.
Why did one comment keep being coded wrongly even after improving the rules?
- It had no text
- It was a duplicate
- It was too short
- It mentions two themes, and the rules pick only one
Comments can carry more than one theme.
Why can comment counts mislead about all customers?
- Only some customers write comments, often the very happy or unhappy
- Comments are always rounded
- Customers lie in comments
- Comments are random
The writers are not a fair sample of all customers.
How should you report counts from word rules?
- As exact facts
- Without the number of comments
- With how well they agreed with a person
- Only as percentages
Say how many comments and how well the rules agreed.
Summary
- Coding turns free text into countable themes.
- Check rules against a person before trusting counts.
- A comment can carry more than one theme.
- Report the agreement and remember who writes comments.
Questions
Is it better to code comments by hand or with rules?
By hand is more accurate and slow. Rules or language tools are fast and imperfect. A good approach is to code a sample by hand, then use that sample to measure how well the faster method works.
How many people should code the comments?
Two, working separately, is a common good practice. Where they disagree, discuss and agree a rule. That also shows how clear your themes are.
Can a comment have more than one theme?
Yes, and it often should. Allowing several themes per comment gives a truer picture. Here a single label was used to keep the lesson simple.
