I’ve often received questions about how testing with as few as five participants can possibly lead to statistically significant results. This question is based on the assumption, especially common among engineering or KPI-driven organizations, that usability testing is quantitative. In fact, usability testing can be used as either a quantitative or a qualitative method. In this post, I’ll explain the basic differences and provide a few tips about how to apply this information to your research practices.

Most usability testing is qualitative

The most prevalent form of usability testing is qualitative. It aims to learn more about users’ opinions and observe challenging or interesting points in their experience with a design. Because the object is simply to reveal these issues to the researcher, it is not necessarily important how many participants experience the same issue; the researchers and designers use their expertise to make judgments about which issues should be addressed.

Since qualitative usability studies do not aim for statistical significance, it is okay to make changes to your design between test sessions, as we have recommended in our post on running remote unmoderated tests.

Testing can be quantitative, too

While number-minded stakeholders may be prone to discount qualitative research, many UX researchers do not make enough use of quantitative testing. When usability testing is designed to obtain mathematical results, it enables benchmarking and tracking metrics over time. Metrics can be task-specific, like completion times or success rates, or they can look at experience more generally through questionnaires like the System Usability Scale. They can also be used to make comparisons between competitors, as we have done in our car rental and hotel booking website usability studies.

Tracking metrics is especially useful as a way of demonstrating the return on investment of UX work. This is especially true today, as an increasing number of start-ups use metrics as the basis for all important business decisions. Following suit and becoming experts in metrics can only earn a UX team more respect and buy-in.

Which should you use?

Quantitative and qualitative usability testing are complementary; a good UX research program will make use of both. Which method to use primarily depends on where you are in the design process.

Qualitative testing is often useful while you are still in the formative phases of design, as it will help you validate that your concepts make sense and are appealing to your target users.

Quantitative testing is needed a little later, when a design is already in use and you want to track how any changes you make affect its usability.

However, quantitative testing can only show you if metrics are going up or down; it cannot tell you why. Thus, further qualitative testing is needed to understand why metrics change.

A few more tips

We’ve talked about the importance of clearly defining your goals and research questions when planning a study. As part of this process, it’s essential that you identify whether your study will be qualitative, quantitative, or a combination of both.

For one thing, the type of study you are running will affect how you should write your tasks. While tasks for qualitative studies can be open-ended, tasks in quantitative studies should have only one correct solution and should be written so that the user has every detail needed to reach that solution.

You will also need to keep in mind that quantitative testing does require larger numbers of users — usually 30 or more.

In addition, the think-aloud protocol is generally not used for quantitative usability tests, as it adds a variability factor (and anyway, only contributes qualitative data). If you choose to combine both qualitative and quantitative research questions in one study, though, you may still want to use the think-aloud protocol.

In conclusion

Understanding the objectives of quantitative and qualitative usability testing further demonstrates the need for continuous, iterative usability testing. UX research is an ongoing process of learning and improvement and must be conducted as a cycle in order to gain its full benefits. In a subsequent post, we will dig deeper into the how and why of setting up a regular usability testing program.