Out-of-the-Box Usability Testing for Medical Devices | Emergo by UL

By Michael Wiklund

Human factors specialists might opt to conduct an “out of the box” usability test of a medical device, such as for a home-use blood pressure monitor. Such a test might even be necessary from a regulatory standpoint, serving to determine if a product is safe and effective when people use it for the first time. Such a test might also be motivated by the commercial goal of bringing to market a product that is initially easy to use and popular. In any case, it’s a great way to assess the user experience from the very start of product interactions. 

Use-scenario for out-of-the-box usability testing of medical devices 

During an “out-of-the box” usability test in our lab at Emergo by UL of a blood pressure monitor, we might pose the following scenario: 

Imagine you just returned from a trip to the local pharmacy to purchase a blood pressure monitor. You bought the monitor because your doctor wants you to record your blood pressure in the morning, afternoon, and evening to help determine if you need medication. Please proceed to use the product, just as if you were at home, to measure your blood pressure. 

From this point forward, the test participant will be on their own to use the device if we are trying to validate the product’s use-safety for regulatory submission purposes. If we are evaluating a prototype, we will probably ask the test participant to “think aloud” while they work through tasks. This helps us identify the product’s strengths and opportunities for improvement to guide the next stage of product design. 

Importance of out-of-the-box usability testing for medical devices 

As you can imagine, “out-of-the box” usability tests are a good way to evaluate not only over-the-counter medical devices, but also prescriptions devices, such as a CPAP machine. Such devices must work well for untrained and self-trained users/consumers. Therefore, what amounts to an initial ease-of-use test is quite relevant. 

The products that tend to perform the best in these tests are those reflecting a solid investment in human factors that leads to a solution people can immediately use safely and effectively based on (1) their prior experience with such products or ones with similar features, (2) their innate problem-solving skills, and (3) their intuition that is likely shaped by the product’s user interface cues. In most cases, a product that is initially intuitive to use also affords long-term ease of use, particularly because its simplicity is not a barrier to efficient use. 

“Out-of-the box” usability tests are best suited to products that do not require training. So, we can set aside dialysis machines, insulin pumps, and spinal cord nerve stimulators used outside of clinical settings. But the evaluation method is quite suitable for evaluating pen-injectors, glucose meters, nebulizers, inhalers, AEDs, pulse oximeters, digital thermometers, and the previously cited blood pressure monitor.  

Expectations for out-of-the-box usability testing of medical devices 

The one thing you can expect from an “out-of-the box” test is the unexpected. For example, I have seen someone struggle at length with a device – it wouldn’t turn on – because they did not remove the insulating pull tabs from the batteries that come in the device. Also, I have seen someone maintain physical contact with a simulated patient when using an AED to deliver a shock. This use error would give the caregiver an unpleasant jolt and possibly trigger a heart arrythmia. Notably, one study suggests that emergency medical technicians give themselves an unintended shock at a rate of 0.1% (1 in one thousand), and it is likely to be a more common occurrence among laypersons. 

Many of the features that make a product excel in an “out-of-the box” usability test are straightforward. Here are ten such features. 

1. Clear package labeling that enables consumers to select the right product. For some devices, it is critical that people can self-select a device that is suitable for their use versus not. 

2. Labels on key product features, even if it means creating product variants and different stock keeping units (SKUs) for different markets. 

3. Clear and conspicuous warnings that prevent potential use errors that could lead to harm. 

4. Visually highlighting parts that warrant special attention, such as removing bright yellow insulating tabs from the batteries prior to powering-up the device. 

5. Visual differentiation of interactive elements from static ones. 

6. A package organized to present the user with the instructions before the device. 

7. Designing instructions for use (IFUs) for medical devices that feature simple language, complementary graphics, and differentiate operational steps from supporting information that exactly match the packaged device versus many varying models. 

8. A user interface that provides good feedback in response to key user actions, rather than leaving users guessing about what’s going on. 

9. Contact information enabling users to call the manufacturer for help. 

10. A link to a video explaining and demonstrating how to use the device safely and effectively.  

Oh yes, there is one more tip to share to help ensure a successful human factors validation test: conduct one or more formative tests (preliminary tests) ahead of the final one. Early testing identifies opportunities for design improvement and reduces the chance that users will struggle in a final, “out-of-the box” test.  

Michael Wiklund is Business Development Director, Human Factors Research & Design (HFR&D) at Emergo by UL.