Usability Testing That Ends the Argument

How many participants do you need for usability testing?

Around five per distinct user group surfaces the majority of significant usability problems. Problems repeat quickly across participants, so the return on the sixth, seventh and eighth session drops sharply.

That number applies to finding problems. It does not apply to measuring anything. If the question is "what proportion of users prefer this" or "did the conversion rate change", five people cannot answer it and no amount of careful analysis makes them able to.

Where there are genuinely different user groups — a buyer and an administrator, say — each needs its own five. Testing five people drawn from two different groups gives you a thin sample of each rather than a good sample of either.

Should we ask users what they think?

Only after watching what they do, and even then treat the answers cautiously. Stated preference and actual behavior diverge routinely, and people are generally poor at predicting their own future actions.

The specific failure is politeness. Ask someone whether they like a design and most will say something encouraging, particularly to the person who made it. Ask them to complete a task and their difficulty is visible whatever they say about it.

So the session is built around tasks. Opinion questions come at the end, framed as "what did you expect to happen there" rather than "did you like it" — which produces something usable, because it is about a specific moment they just experienced.

When is the best time to run user testing?

On a prototype, before build. That is where a finding costs a day to act on rather than a sprint, and it is consistently the highest-return moment to test.

Testing after launch is still worth doing — you have real users and real data — but the economics are worse. A structural problem found in a shipped product competes for developer time against everything else in the backlog, and often loses.

The pattern that works is testing twice: once on the prototype to catch structural problems while they are cheap, and once after launch to catch what only appears with real content, real data and real device variety.

What is the difference between usability testing and A/B testing?

Usability testing tells you why something fails. A/B testing tells you which of two options performs better at scale. They answer different questions and neither substitutes for the other.

A/B testing needs traffic — enough conversions on both variants to reach significance. Below that threshold the results are noise, which is why most small sites cannot A/B test meaningfully however much they would like to.

Usability testing works at any scale and explains causes. The efficient pattern for sites with traffic is to use analytics to find where people drop out, usability testing to understand why, and A/B testing to confirm the fix. Skipping the middle step means testing variants of a guess.

How do you write tasks that do not lead?

By describing a goal in the participant's language and leaving out every word that appears in the interface.

The classic mistake is naming the mechanism. "Use the filter to find a red one under fifty dollars" tests whether someone can operate a filter they have just been told about. "Find something you would actually consider buying, in red, for under fifty" tests whether they discover the filter at all — which is the real question.

Tasks should also have a reason attached. A goal with context — you are buying this as a gift, it needs to arrive by Friday — produces more natural behavior than an abstract instruction, because people can bring judgment to it.

And a task needs a definable end so both of you know when it is done. Open-ended browsing produces pleasant sessions and very little you can act on afterwards.

Should you ask people what they think?

After the task, not during it, and with the understanding that the answer is much weaker evidence than what you just watched.

People are unreliable narrators of their own behavior, and not through any fault of theirs. They rationalize after the fact, they are polite to whoever is running the session, and they genuinely do not have access to why they clicked one thing rather than another.

What opinions are good for is surfacing expectation and emotion. "I assumed that would show me the price" is valuable, because it reveals a mental model. "I think the design is clean" is not, and it is the kind of feedback that fills reports without changing anything.

During the task, the useful prompts are neutral and few: what are you looking at, what do you expect will happen, what would you do next. Anything more turns the session into a conversation and stops you observing the thing you came to observe.

What is the difference between this and A/B testing?

Usability testing tells you why something does not work, with a few people, in depth. A/B testing tells you which of two versions performs better, with many people, and does not tell you why.

They answer different questions and work best in sequence. Usability testing generates the hypothesis — people miss this because it looks like a heading rather than a button. A/B testing verifies that changing it produces a real difference at scale.

A/B testing alone tends to produce a series of small wins with no understanding underneath them, and it needs traffic volume many sites simply do not have. Running an experiment that cannot reach significance is a way to spend weeks producing noise.

Usability testing alone risks confidence without proof — a change that clearly helped five people may not move anything measurable. Neither is complete, and the choice between them should follow from whether you need explanation or verification.

What makes a session go wrong?

The moderator helping. It is the most common failure and the hardest to resist, because watching someone struggle with something you understand is genuinely uncomfortable.

The moment you say "it is at the top" the session is over as evidence. Whatever you learn afterwards is about a person who has been told the answer. Silence is the technique, and it takes practice.

The second failure is testing the wrong person. A participant who does not resemble your users will produce confident data about a population you do not serve, and it is worse than no data because it carries the appearance of evidence.

The third is a broken prototype. If the thing under test fails technically, you spend the session apologising and learn about your prototype rather than your design. Everything gets walked through once before anyone is recruited.

What happens with the findings?

They get ranked, assigned and verified. A finding that is reported and not fixed is a cost with no return.

Ranking is by severity and frequency together. Something that blocks the primary task for most participants is different in kind from a cosmetic irritation one person mentioned, and a flat list obscures that difference completely.

Each finding should carry a concrete suggestion rather than only a diagnosis. "People do not recognize this as clickable" is a finding; "give it the standard button treatment used elsewhere in the product" is something someone can do on Tuesday.

Then the fix gets checked. Retesting the specific tasks that failed is quick, because the script exists and the tasks are known — and it is the only way to know whether a change helped, made no difference, or moved the problem somewhere else.

How do you run a session remotely?

Much as you would in person, with more attention to setup and a lower tolerance for anything that could go technically wrong.

Remote moderated sessions are now the default for most projects, and the quality difference from in-person is smaller than expected for anything screen-based. What is lost is peripheral observation — hesitation, body language, whether they reached for a phone — and that matters more for some studies than others.

The setup is where sessions fail. Screen sharing that the participant cannot start, permissions not granted, and audio that does not work consume the first ten minutes and put people on edge before anything begins. A short technical check beforehand prevents almost all of it.

Unmoderated testing is cheaper again and answers narrower questions. You get task completion and a recording, and you cannot ask why. It suits verifying a specific known issue and does not suit exploring an unfamiliar problem.

The thing to preserve either way is stakeholders watching. Remote makes that easier rather than harder, and a colleague who has seen one person struggle is more persuaded than one who reads the report.