Choosing an Optimal Formative Usability Test Sample Size

Though the scope of a formative usability test is up to an individual medical device, IVD or combination product manufacturer, when going through the effort of arranging a usability test, you want to feel confident you have included an optimal number of participants. Without specific regulatory guidance and few guidelines on formative usability test sample size, you still want to yield the most value in terms of understanding what is working well with your product, and what might need improvement. If including too few participants, you might not uncover an opportunity for improvement or see important trends in interaction challenges. If including too many participants, you might be “wasting time” by seeing repeated findings you have already identified as a trend (i.e., not uncovering anything new) or “wasting participants” in the case of rare participants you might be better off reserving to participate in future rounds of usability testing.

General guidelines for adequate test sample sizes

Conventional wisdom from a commonly cited paper on usability test sample size postulates that to reveal 80% to 90% of interaction challenges, you should include five to eight participants per user group, respectively. Although this is a valuable guideline, there are some key factors you should consider to tailor this guideline appropriately:

• Number and types of user groups. If your product has only one user group, you might have the resources to include more than the aforementioned guideline recommends (e.g., 10 participants) to gain deeper insights into your product. If you have multiple user groups, cumulatively you will have more participants, so you might opt toward the lower end of the guideline, or perhaps even below if some user groups interact with the product in similar ways (e.g., people with diabetes and lay caregivers of people with diabetes).
• Test scope. If there are many key interaction areas you would like to investigate, you might consider going toward the higher end of the five to eight participant guideline, noting that you might not have time to take a deep dive on every topic with every participant. Conversely, if there is a limited focus for the test (e.g., one key area of interest) then you could skew towards the lower end of the guideline.
• Design iteration speed. If you have the capability to quickly iterate on the product’s design, you could consider conducting several rounds of testing with smaller sample sizes. If design iteration is a lengthier process, then you’ll likely want to use a more substantial sample size to capture a more comprehensive set of findings for each test.  
• Critical task composition. Particularly as you approach finalizing your product’s design and progress towards human factors (HF) validation testing, you’ll want to ensure you thoroughly evaluate your product’s critical tasks in your formative usability tests. Consider the quantity of critical tasks, and the severity of harm associated with the critical tasks. The “riskier” your product is to interact with from a user harm perspective, the more participants you should include in your formative usability tests to ensure you’ve done everything you can to discover and mitigate against potentially harmful interaction problems.
• Product complexity. Though one could argue that product complexity goes hand-in-hand with the previously mentioned test scope and critical task composition factors, do generally consider your product’s complexity as it relates to formative usability test sample size. The more possible interactions users can have with your product, the more participants you should include in your formative usability tests to feel confident you’ve comprehensively evaluated all possible interactions.  
• Number of planned formative usability tests. If you’ve planned for several formative usability tests, it might be sufficient to skew towards lower sample sizes noting cumulatively across your tests you will have the opportunity for many participants to interact with your product at various stages. Conversely, if you’ll be conducting fewer formative usability tests you will likely want to use more substantial sample sizes.

As these key factors for consideration have reinforced, there is not a “one size fits all” approach to formative usability test sample size. As with the product’s design, the beauty is that you don’t have to get the sample size exactly right the first time if you’ve made time for multiple rounds of testing. Just as you iterate on the product’s design, you can iterate on your approach to sample size as you conduct subsequent formative usability tests.

Stephanie Larsen is Managing Human Factors Specialist at Emergo by UL’s Human Factors Research & Design division.

Additional human factors engineering and usability resources from Emergo by UL:

• HFE user research support for medical devices, IVDs and combination products
• Human factors design and prototype development support