Medical Device Evaluation and Usability Testing

Expert reviews

Frequently, we are called upon to evaluate a product’s design by conducting either an expert review or a heuristic analysis. In these cases, we apply established design principles and our judgment to identify user interface design strengths and opportunities for improvement. Such evaluations can usually be done quickly and at low cost yet produce valuable findings.

Examples:

• Perform a heuristic analysis of the screens associated with a fitness app to be used on a smartphone by comparing the user interface design to recognized design standards.
• Review and identify opportunities to improve a product’s packaging and labeling, seeking to maximize clarity and information presentation.

Cognitive walkthrough

A cognitive walkthrough is another technique often used to evaluate early design concepts.

This approach involves representative users reviewing product sketches, physical models and prototypes, or a series of static images or screens and thinking through and explaining how they might perform a task with the actual product.

Formative usability testing

We often conduct iterative, formative usability tests of products. Such tests can be quick to deliver findings for a design team that might be in the middle of a development sprint and could even follow a rapid iteration approach, in which one or two days of usability testing is followed by a quick product, labeling, and/or package redesign, followed up again by additional testing days. We can conduct formative usability tests on full products and systems, as well as smaller-scale evaluations focused on a subset of features or workflows.

Examples:

• Conduct a six-participant formative usability test of two potential concepts for the homepage dashboard and sample alerts within a web-based application designed for healthcare professionals to track patients’ dialysis treatments and kidney health.
• Conduct an eight-participant formative usability test of an exoskeleton used to help people suffering from stroke regain their stride.
• Conduct a 15-participant pre-validation, i.e., final formative usability test of a dialysis machine, following a test protocol intended for use in the subsequent HF validation (summative) usability test.

HF validation (summative) testing

We have conducted HF validation tests with a wide range of participant sample sizes, ranging up to 200+ participants. Our test protocols are routinely reviewed and tacitly accepted by the FDA. We conduct tests in our own laboratories, at customer sites, in medical simulation facilities and in other research facilities worldwide. Our comprehensive test reports thoroughly describe the usability testing approach, test findings and root causes of any use error and other interaction problems. We base our testing and reporting approaches on available guidance from multiple sources, e.g., FDA, AAMI and IEC, as well as best practices and our many years of experience conducting authoritative tests as a capstone HFE activity to product development.

Examples:

• Conduct a 75-participant, HF validation test of an insulin pen injector to generate data demonstrating that the product can be used safely and effectively by adolescents, adults, seniors, caregivers and healthcare providers.
• Oversaw and conducted HF validation testing in China involving team use of a surgical system.
• Conduct an HF validation test of a heart-lung machine used by perfusionists to sustain patients undergoing open-heart surgery.
• Conduct an HF validation test of a mobile application used to track diabetes management and insulin delivery.