Are US Residents Representative Usability Test Participants for the Entire World?

You are probably aware that the US Food and Drug Administration (FDA) requires human factors (HF) validation testing to be conducted with residents of the US. By contrast, neither the European Medical Devices Regulation (MDR) nor the International Electrotechnical Commission (IEC) 62366-1 standard require that HF testing is carried out in a specific region or with residents of a specific region. As such, while the FDA will not typically accept data with non-US resident participants, European notified bodies and other regulatory authorities have shown themselves to be open to accepting data collected from US residents. Consequently, companies wanting to market in multiple regions including the US often only conduct their HF validation test in the US. 

The resulting data is then often used to support European (and other) submissions, but can we be certain that such data is truly representative of the local intended users? The short answer is maybe. US residents might be representative in some cases but not in others. For the purposes of this blog, we will focus on Europe, but the considerations would be the same for other markets. 

There are some major differences that might influence the way users interact with products between the US and Europe--here are a few that should be considered: 

• Healthcare systems and clinical practice. Healthcare systems and clinical practice can differ widely between the US and Europe. For example, the accessibility (cost and distance), the clinical ways of working (e.g., disinfecting an injection site for subcutaneous injections), the staff involved or availability of certain procedures might differ between countries.  

• Environments (size, age, layouts). How comparable are the use environments?  Houses, schools, and hospitals in the US are often more spacious than many in Europe. Moreover, power plugs, light switches and window opening mechanisms are different. 

• Culture (language, education). Language is the obvious differentiator here, and often translations of labelling and instructions can be inaccurate if not adequately evaluated. Literacy and level of education may vary between regions, which could impact how well individuals can understand instructions and labelling. And what about using the metric versus the imperial system? Considering most medication strengths are presented using the metric system, this data might be easier to understand for individuals from regions that use the metric system.  

In addition to the topics listed above, there are more regional differences that influence the experiences someone has that would shape their expectations of, and interaction with, a product. Simplistically, this could just be the types of products that the end user is currently exposed to. For example, there are many more advertisements for medications and medical devices in the US than there are in Europe. Additionally, there are various products and designs (medical and non-medical) that are available in the US that are not in Europe, and vice versa. 

Could any of these (or other) differences lead to a real use-related issue? Again, the answer is, maybe. This is up to the manufacturer to determine, and part of the question is: Do you want to know before or after you marketed your product whether there are differences in use that could result in use-related risks? 

Not being aware of potential differences in use before launch could result in customer dissatisfaction, complaints and possibly product recalls, which can be costly and time-consuming to remediate. 

The interesting thing is that a lot of these differences and concerns could be minimized with improved understanding and consideration of the proposed markets early in product development.  A well-considered plan to identify issues and demonstrate participant equivalence through research and formative evaluation might help to justify a singular US validation study. 

So, what type of assessments can one perform to determine whether US residents are representative of other markets for the product?   

• User and use environment research. Research the users and use environments of different regions, identify differences in backgrounds, and relevant characteristics. 

• Early research. Investigate during early usability research whether there are any conflicting design requirements between different markets. 

• Formative usability studies. When conducting formative usability tests in Europe and the US, determine whether there are any differences in use-related issues that might be rooted in regional differences. 

Data from such activities can help determine the level of equivalence between markets and help justify using data from an HF validation test in the US for a European submission. 

But, what if all the work has already been done: the product’s design is finalized and the HF validation test completed. Is it possible to compare user characteristics in order to build confidence? It might be--a thorough review of the user characteristics, behaviors, environments and existing clinical practices might help to demonstrate retrospectively that the participants included in validation testing are representative. However, there is no guarantee that one would be able to provide sufficient data to sustain such a claim. 

So, in the end, it all comes down to knowing your users in the different regions. Our team has experience conducting usability research in multiple regions with a wide variety of users and devices, and we will go into further detail on this topic in future blogs. 

Jenny Collinson is Research Manager and Tamara Will is Managing Human Factors Specialist at Emergo by UL's Human Factors Research & Design division.