When to Verify Risk Control Measures as Part of Human Factors Engineering Activities

By Petra Boeree  

We all know the chicken or the egg causality dilemma, often introduced as “what came first, the chicken or the egg?” You could pose a similar question for risk control measures referenced in HF validation tests: What comes first, the verification of use-related risk control measures or the HF validation test? 

Regulatory perspective 

Regulators require manufacturers to conduct a use-related risk analysis (URRA) to identify critical tasks that users should perform correctly for medical device use to be safe and effective. The URRA not only contains all use-related hazards and resulting potential harms and severity levels but should also include all risk control measures that have been implemented to eliminate or reduce the risk. This input is then used to evaluate all use-related risks during human factors (HF) validation testing. 

In HF validation testing, the goal is to demonstrate that the device can be used by the intended users without serious user errors or problems, for the intended uses and under the expected use conditions (i.e., confirming that the device can be used safely and effectively). 

The FDA’s 2016 guidance indicates that the device user interface should represent the final design during HF validation testing. The guidance states that HF validation testing is used to assess the effectiveness of risk control measures. This implies that risk control measures are evaluated during the HF validation test.  

However, HF validation tests are costly and by no means the preferred point in time to learn about bugs in the device’s software or faulty needle safety shields that do not deploy. And although such risk control measures are generally of a technical nature and can therefore more easily be verified to determine whether they “work” or not, it is more difficult to determine whether a use-related risk control measure is truly effective in preventing harm. 

Terminology 

To understand when and how risk control measures should be evaluated, it is important to understand the difference between “verification” and “validation.” These two related, albeit distinct, activities are sometimes used interchangeably, leading to confusion. The activities are defined as: 

• Verification. Confirmation, through the provision of objective evidence (i.e., data supporting the existence or verity of something), that specified requirements have been fulfilled, as defined per ISO 14971:2019 (Medical devices – Application of risk management to medical devices). 

• Validation. The act of confirming something as true or correct (i.e., confirming that the risk control measures achieve their intended objectives and reduce the risks to an acceptable level). 

Notably, ISO 14971 distinguishes between two verifications: the first verification to make sure that the risk control measure has been implemented in the final design of the medical device or in the manufacturing process, and the second verification to ensure the risk control measure (including information for safety) as implemented actually reduces the risk. As such, the second verification described in ISO 14971 overlaps with what is oftentimes referred to as validation. 

Both ISO 14971 and FDA’s 2016 guidance state that in some instances HF validation testing can be used to assess the effectiveness of risk control measures. In other words, the second verification may be performed during the HF validation test by observing whether the risk control measure prevents test participants from making mistakes (e.g., by eliminating the risk completely or providing test participants with the opportunity to correct their mistake).  

Types of risk control measures 

Whether a use-related risk control measure will be verified during an HF validation test will largely depend on which category the risk control measure falls into:  

• risk control measures with user touchpoint, or  

• risk control measures without user touchpoint. 

A user touchpoint is any instance a user interacts with the device. Risk control measures that do not have user touchpoints (e.g., electrical safety features in your system) might not need to be evaluated during the HF validation test. These risk control measures can more readily be verified in parallel with or after the HF validation usability test.  

There is one caveat; if during verification it turns out that changes to the system are required (e.g., to the design of the system or instructions), further human factors activities might be required. Specifically, the FDA could consider the system used during the HF validation test to not be “production equivalent” (i.e., representative of the final marketed product) and request additional testing. Alternatively, if verification of risk control measures goes smoothly, there is no need for human factors follow-up activities. 

Conclusion 

In conclusion, consider the following to determine when and how to verify risk control measures: 

• Identify all risk control measures, especially for critical tasks, prior to conducting HF validation testing. 

• Perform the first verification of risk control measures with user touchpoints before HF validation testing (i.e., ensuring that risk control measures have been implemented in the final design). 

• Perform the second verification of risk control measures with user touchpoints before and/or during HF validation testing (i.e., ensuring that the risk control measures are effective). 

• Verify risk control measures without user touchpoints before, during, or after HF validation testing. Although verifying risk control measures after HF validation testing is acceptable, this comes with some level of risk if verification leads to design changes that might need to be re-validated in a supplemental HF validation test. 

Contact our team to learn more about verifying risk control measures and HF validation testing. Or, sign up for a complimentary account with OPUS, our team’s software platform developed to empower companies with HFE training, tools and templates. 

Petra Boeree is a Senior Human Factors Specialist at Emergo by UL.