Pensées · UX & AI
How many people should test your AI?
Why five users miss the mistakes an AI makes once in twenty, and how to plan research that follows people longer and repeats every test.
You test a new AI assistant with five people. All five sessions go well. The answers are clear, the tone is pleasant, nobody gets lost. You launch on Monday. On Thursday, a customer receives a refund rule that does not exist, and nobody in the team has ever seen the assistant say it.
Nobody tested badly. The team tested an AI the way we test a website. A website gives the same answer every time you ask. An AI may not. Even with its most stable settings, the same question can come back with a different answer, and its worst mistakes may appear in one conversation out of twenty, or fewer.
A plant you cannot judge in one visit
An amaryllis has a year in several acts. It flowers, then grows long leaves for months. To make it flower again, many gardeners give it a rest in autumn. They stop feeding it, water it much less, and keep it somewhere cooler for eight to ten weeks. The plant looks finished. Then it is woken up with water and warmth, and it flowers again.
Visit it once during its rest, and you would think it is dying. Visit it once in bloom, and you would think it is perfect. Both visits are true. Both are misleading.
An AI is the same. One session shows one moment of its behaviour. To know it, you have to follow it longer, and watch it many times.
Why five users worked
In 2000, Jakob Nielsen explained why testing with five users is usually enough. His model counts on problems that many people meet: in his projects, the typical problem was found by 31% of users, about one in three. A problem that common is hard to miss. With five people, at least one will very likely stumble on it, and five users find about 85% of the problems.
He never meant a single round, though. He advised three studies of five users rather than one study of fifteen. Test, fix, and test again.
Why it breaks with AI
AI changes that assumption. Its most serious mistakes are often rare: an invented policy, a confident answer to a question it should refuse, personal data shown to the wrong person. Rare does not mean harmless.
Here is what five sessions can catch, depending on how often a mistake happens.
- About 1 time in 3: five sessions will very likely show it (about 85%).
- 1 time in 10: a 41% chance of seeing it. You need 29 sessions for a 95% chance.
- 1 time in 20: a 23% chance. You need 59 sessions.
- 1 time in 100: a 5% chance. You need 299 sessions.
The maths is simple. If a mistake happens one time in twenty, each session has a 95% chance of missing it. Five sessions in a row miss it 77% of the time.
Agents make this harder. When an agent acts, a task has several steps: search, choose, fill in, pay, confirm. If each step works 95% of the time, a task of ten steps works only about 60% of the time. Test whole tasks, and not only single answers.
Researchers measured this inconsistency in 2024. The team behind τ-bench gave customer-service agents the same tasks several times. GPT-4o, one of the leading models at the time, solved a task for an online shop in about 61% of attempts, but solved the same task eight times in a row only about 25% of the time.
Here is how I would plan the research instead, in four steps.
1. Name the mistakes you cannot afford
Before choosing how many people to test, list what must never happen. Sort the mistakes by how serious they are, not by how often you have seen them: an invented price, a wrong deadline, a refusal that leaves someone stuck, private data shown to the wrong person.
For each one, set a limit you can check, such as “fewer than one conversation in a hundred”. That limit tells you how many tests you need.
2. Go deeper with people
People tell you why a mistake hurts, and how they live with an AI over time. A few deep sessions teach more than many short ones.
- Longer sessions, several tasks. Give each person three or four real tasks.
- The same question, twice. Ask people to repeat a question. Watch what they do when the answer changes.
- The same people, over a week or two. A diary study shows how trust grows or breaks. The first session shows curiosity, not habit.
- People under pressure. In a hurry, worried, angry or lost.
- How they check. Note when people verify an answer, and when they simply believe it.
3. Repeat every test many times
The AI’s own variation needs numbers, and numbers cost less than people.
- Replay each scenario many times. Write test scenarios from real questions, and run each one again and again. A script can replay a scenario a hundred times overnight, then a person reads a sample.
- Size the runs with the list above. To catch a mistake that happens one time in twenty, plan about sixty runs of the scenarios where it could appear.
- Read zero with care. No mistake in 60 runs is still no proof. Statisticians use the “rule of three”: the true rate is probably below 3 divided by the number of runs, here 5%.
- For a metric, count on about 40 people. For a success rate or a satisfaction score with real people, the Nielsen Norman Group recommends 40 participants in most cases.
- Test again after every change. A new model version, a new instruction or a new document can change the answers. Keep your scenarios, and run them all again.
4. Keep watching after launch
The rarest mistakes only show up with real use. With people’s consent, read a sample of real conversations every week. Track the mistakes from step 1, and fix their cause (the instructions, the information or the tools), as I describe in my note on AI agents. Keep as little personal data as you can, and remove what identifies people.
A note for those who build it
A simple test sheet is enough to start. One row per run: the scenario, the run number, the date, the model version, “right answer?”, “sounded like itself?” and “what went wrong”. Note the model version, because the same scenario can change when the model changes. After a few weeks, you can see whether a mistake is rare, frequent or gone.
To remember
- Five users find the problems that many people meet. AI’s worst mistakes can be rare.
- One session shows a moment, not the behaviour.
- Go deeper with people: longer, over time, under pressure.
- Repeat every test: the same scenario, many times.
- No mistake in a few runs proves very little.
- Keep watching after launch.
Look closer
My amaryllis has a season when it looks finished. I water it less, and I let it rest. If I judged it on that day, I would throw it away. Judged over the year, it is only preparing its next flower.
An AI needs the same kind of look: long enough, and often enough, before you decide what it is.
If you plan research for an AI feature this way, I’d be glad to hear what you found.
Sources (5)
- Anthropic, Messages API reference, on the temperature setting: “even with temperature of 0.0, the results will not be fully deterministic”.
- Jakob Nielsen, Why You Only Need to Test with 5 Users, Nielsen Norman Group (2000).
- Shunyu Yao, Noah Shinn, Pedram Razavi and Karthik Narasimhan, τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024).
- James A. Hanley and Abby Lippman-Hand, If nothing goes wrong, is everything all right? Interpreting zero numerators, JAMA (1983).
- Raluca Budiu and Kate Moran, How Many Participants for Quantitative Usability Studies, Nielsen Norman Group (2021).
