Top 5 Dos and Don’ts - Human factors validation testing of medical devices for FDA market access

By Allison Strochlic

Although there are dozens of valuable human factors engineering (HFE) activities a manufacturer should conduct throughout medical device development, human factors (HF) validation testing is arguably one of the most critical. The US Food and Drug Administration (FDA) often requires HF validation testing to demonstrate that a device can be used safely and effectively for the intended users, uses and use environments. Based on our extensive hands-on experience conducting HF validation tests, we have identified 5 things you absolutely should – and 5 things you shouldn’t - do when conducting an HF validation test.

Top 5 tips for conducting robust HF validation testing for the US FDA

1. Evaluate a production-equivalent device. For HF validation testing, you need to use a production-equivalent version of the product, meaning that it looks like, acts like and is packaged and labeled like, the device you intend to commercialize. You generally need the full device user interface – including hardware, software, packaging, labeling and training – to be commercial-ready to confirm your test represents the expected real-world use of your device.
    
2. Design your usability test’s evaluation activities based on critical tasks. It is not enough to invite participants to simulate performing tasks assumed to be the most important or frequent, such as set up, treatment or dosing and disposal or storage. Instead, you should perform a comprehensive use-related risk analysis (URRA) to identify critical tasks. Critical tasks are use steps that, if performed incorrectly or not performed at all, could lead to harm or compromised medical care (for combination products) or “serious harm” for the patient or user (for medical devices). HF validation test participants should have the opportunity to perform all critical tasks, presented via representative, naturalistic scenarios, which we refer to as evaluation activities. For additional information on identifying critical tasks, read more here on key URRA steps and here on pitfalls to avoid when conducting a URRA.
    
3. Include at least 15 representatives of each distinct user group. When it comes to HF validation test sample sizes that comply with the FDA’s final HFE guidance, you don’t want to fall short of the “magic number” of 15 per user group. Defining your distinct user groups is definitely an art form, which at the highest level requires thinking about your end-users’ characteristics and how they may influence interactions with the product, including but not limited to age, relevant education and training, past experience with a given device and whether someone is a layperson or professional user.
    
4. Record use errors and issues and interview participants about them. Primary HF validation test data includes use errors and other interaction difficulties – namely, close calls and difficulties – that occur when participants complete the evaluation activities. Test personnel should observe unobtrusively as the participant works with the device and record all such events. Then, toward the end of the test session, the moderator needs to tactfully and unbiasedly interview the participants about any use issues that occurred. It is important to collect not only objective performance data, but also participants’ subjective assessments of the root cause of any use issues. The HF validation test report should reflect this mix of objective and subjective data, as well as a robust root cause analysis performed by test personnel.
    
5. Conduct testing in a representative use environment. HF validation testing is all about being representative – including the participants, evaluation activities and the device itself, as mentioned above. However, the test environment should also be representative. Generally, you do not need to conduct usability testing in the actual use environment, but you should simulate the key characteristics, such as lighting and sound levels, potential distractions and other personnel and equipment present during use of the subject device. Testing in such an environment helps put device use, as well as any interaction difficulties, in the proper context.

Top 5 things NOT to do when conducting HF validation testing for the US FDA

1. Do not include company employees or Key Opinion Leaders (KOLs) as HF validation test participants. In most cases, employees and KOLs are not typical, representative end-users and, therefore, should be excluded as study participants. The only exception would be in the relatively rare case that the end-users are exclusively company employees, such as if you are testing a complex device installation and calibration process that is performed only by manufacturers’ representatives.
    
2. Do not conduct testing outside of the U.S. When generating HF validation test data for the U.S. FDA, except in special circumstances and with explicit pre-approval via a pre-submission review, all participants must be U.S .residents per the FDA’s final HFE guidance. That said, do conduct a subset of your user research and formative usability testing outside of the U.S. if you will ultimately seek to commercialize your product globally.
    
3. Do not simply administer a survey or questionnaire to collect subjective feedback. HF validation testing must include simulated, hands-on use of a given device, supplemented by a live, open-ended interview to collect participants’ subjective feedback regarding their device interactions. This involves asking questions that cannot be answered with a simple “yes/no” or “true/false” response but rather invites the participant to share their subjective impressions.
    
4. Do not interview participants about their “likes.” FDA’s HF reviewers are focused on confirming a device can be used safely and effectively, not whether a participant likes a given aspect of a device or prefers your proposed device over an existing one. While there is value in understanding this side of participants’ subjective feedback from a commercial success perspective, such data will not serve as evidence of a device’s use-safety.
    
5. Do not just report the raw data. An HF validation test report is more than just scanned checklists and interview transcripts. Rather, test personnel – ideally, experienced HF specialists – must review and analyze the data, which includes categorizing and tallying like findings, describing the context of use issues and performing rigorous root cause analysis that goes far beyond blaming the user.

Contact our team to learn more about how we can help you plan, conduct and report the findings of a comprehensive HF validation test that meets the US FDA’s expectations and generates data to support your regulatory submission. You can also sign up for a complimentary account with OPUS, our team’s software platform that provides training, tools and templates related to various aspects of HF validation testing.

Allison Strochlic is a senior research director at Emergo by UL.