Don’t Let Software Bugs Stand in Your Way

Software engineers have it tough when it comes to developing the software used to control medical devices, especially complex devices such as infusion pumps, robotic-assisted systems (e.g., surgical, bronchoscopy), and dialysis equipment. Each line of code might call other lines of code. Each function might be impacted by varying system parameters. And, each subroutine might be influenced by user inputs. Because of these variables, it is no surprise that software, when first being written, is often rife with bugs. Even when software is stress tested as part of a good verification process, actual use of the software by representative users always seems to find those hard-to-detect defects. Sometimes these bugs are discovered right before a human factors (HF) validation test (i.e., summative usability test). At other times, they are encountered by users during the usability test. My experience has been that no matter the completion date on the project schedule, software bugs have a habit of rearing their ugly heads during those final tests and often result in project delays.

The ”production equivalent device” challenge

These bugs often pose a challenge to human factors engineers. Most often, we hear that we must conduct the HF validation test using the “production equivalent device” (or system). In other words, the device used in testing must be the same as that which will be released to the market; that it is the final version. And, we often take this to mean, when considering software, that there shall be no software defects. No manufacturer should be releasing a device to the market if it contains software bugs. So, the question is: “How are we supposed to conduct an HF validation test if we have to wait for all the bugs to be worked out so that we have a production equivalent device?”

But this is where many of us get it wrong. The guidance and standards do not indicate you must use a “production equivalent device” during HF validation testing. The US Food and Drug Administration states in their 2016 human factors guidance, Applying Human Factors and Usability Engineering to Medical Devices, that HF validation testing should be designed such that “the device user interface represents the final design.” And IEC 62366-1:2015+AMD1:2020 Application of usability engineering to medical devices states “the manufacturer shall perform a summative evaluation… on the final or production equivalent user interface.” In other words, for HF validation purposes, any aspect of the device that a user might interact with—the software interface, hardware controls, labeling—needs to represent the final design. Therefore, software that impacts any part of the user interface’s look or behavior must be bug free to the extent that the user experience is correct.

However, what about embedded software that does not affect the user experience? While some software code might impact the reliability or performance of a device, and therefore needs to be defect-free before market release, as long as any bugs don’t affect the user interface, it might be acceptable to use a non-final software version as part of the your HF validation effort.

Pre-testing

To help alleviate any concerns in proceeding with HF validation, plan to pre-test the device to verify that any existing bugs do not cause the device to malfunction or quit unexpectedly and confirm there is no impact to the user experience. Specifically, I recommend that you conduct your own stress test of the device’s software by running through the usability test’s use scenarios anticipating common, and perhaps uncommon, user interactions. Make sure that, even if the test participants perform unanticipated actions, the software will not crash. At a minimum, conduct a pre-validation usability test following the defined validation protocol to gauge the likelihood of completing the test without any device freezes or system crashes.

Effects on device user interfaces

Also, be sure to record the state of the software at the time of the test. It will be important to document that the software defects do not have any impact on user performance or the user interface. Furthermore, as the bugs are eliminated and the software changes, note whether any of the software fixes impacted the user interface. (They shouldn’t if the bugs were not related to the user interface; but you never know.) Of course, should any part of the user interface change, a subsequent HF validation test might be necessary.

In conclusion, while bugs may persist up to the point of HF validation testing (and beyond), proceed with testing as long as the device has a properly functioning, “production-equivalent user interface.” Just remember to document any post-test modifications to the software and how they do not impact the user experience.

Merrick Kossack is Research Director at Emergo by UL’s Human Factors Research & Design division.

Learn more about human factors engineering and usability issues for medical devices:

• HFE user research for medical devices, IVDs and combination products
• Human factors analysis for medical devices
• Medical device and healthcare product evaluation support