What is a cognitive walkthrough, and why are they beneficial to medical device development?

By Harrison Preusse and Ashley Carroll

Addressing potential usability issues early in the human factors engineering (HFE) process can save your team time and money over the course of development while providing evidence of a robust human factors process to regulatory agencies. Cognitive walkthroughs are a powerful method to identify potential usability issues in early development stages when initial decisions about design directions are being made.

What are cognitive walkthroughs?

During a cognitive walkthrough, researchers typically present users with a low-fidelity model of the product. For example, a physical device might be represented by an early prototype or design sketches, while a graphical user interface can be modelled by a series of static screens. During the evaluation, users are asked to describe how they might use the actual product to accomplish specific tasks, “thinking aloud” to share their thought process in the moment. The researcher might also simulate aspects of interacting with the actual product by explaining how the device would respond to a user action or swapping between screens when a user states they would click a button. The walkthrough is followed by a debrief conversation where the user and researcher talk through any specific challenges encountered during their simulated use. This can even include brainstorming potential solutions.

Key benefits of cognitive walkthroughs

The primary goal of a cognitive walkthrough is to identify usability challenges and opportunities for innovation early in the design phase before significant resources are committed to more detailed design and development efforts. This facilitates efficient feedback that manufacturers can consider when iteratively improving their product to minimize errors, support safety and enhance the overall user experience.

From a logistical standpoint, perhaps the most significant benefit of a cognitive walkthrough is that they are typically less costly and quicker to execute than a formative usability test. This can be a powerful option during the early design phase, providing rapid insight to help steer iterative rounds of prototyping.

Leveraging cognitive walkthroughs for FDA submissions

The FDA recognizes cognitive walkthroughs as one of several preliminary evaluation methods that can contribute to a comprehensive HFE process. Cognitive walkthroughs are uniquely well-suited to accomplishing many early-stage goals defined by regulators, including identifying user needs and potential use errors.

Cognitive walkthroughs can provide more in-depth insights into user needs than other research-based methods, such as interviews or observations, because the user interacts with an early model of the device being developed during a walkthrough. Feedback from later stage evaluations, such as a formative usability test with a functioning device, will be constrained by your overall development approach, as this device will have already been designed to meet a specific set of user needs. Walkthroughs enable you to explore other potential designs that might better address existing needs or reveal that your current understanding of user needs might be incomplete before key design and/or engineering commitments have been made.

Cognitive walkthroughs can also reveal usability issues with a preliminary design, which you can incorporate into your use-related risk analysis (URRA) and begin developing mitigations to address.

Documenting design decisions made in response to cognitive walkthrough findings in an HFE report also serves as an early chapter in the comprehensive story of how the human factors process guided development of your product.

Tips for conducting cognitive walkthroughs

Cognitive walkthroughs incorporate aspects of other common preliminary usability activities, such as user interviews and contextual inquiries, with the added benefit of learning how users interact with your proposed product rather than merely learning about their current workflows.

Due to their early-stage nature and use of low-fidelity models, you can be flexible in your approach to cognitive walkthroughs based on your team’s specific goals. Here are some tips to keep in mind when planning any cognitive walkthrough:

Preparation

• Favor low-fidelity prototypes. Presenting a few alternative low-fidelity prototypes can be more beneficial than committing to a single, higher-fidelity prototype. The range of options can prompt users to consider a wider range of possibilities in their responses.
• Develop realistic use scenarios based on tasks of interest. First, identify the tasks you are most interested in simulating with users. Then, develop a scenario which provides enough context for a user to begin thinking their way through accomplishing a task, without providing explicit guidance towards the solution.
• Draft a comprehensive, yet flexible, script. In addition to providing adequate context to the user, the script should provide direction to the researcher on when and how to simulate feedback from the device. Develop a set of initial questions for a post-walkthrough debrief discussion, while leaving room for the user to guide the conversation.

Conducting the walkthrough

• Prompt users to “think aloud” while working. By verbalizing their thoughts, users provide direct insight into their expectations, initial reactions and ongoing thought process while using a device for the first time.
• Observe and record user performance. In addition to tracking whether users perform steps as intended or not, pay attention to moments where users work through those steps with particular ease or difficulty. Key questions to consider include:
  • Is the user trying to achieve the correct task?
  • Does the user notice that the intended action is possible?
  • Does the user associate the intended action with the task they’re trying to achieve?
  • Can the user see that their action has made progress towards achieving their overall task (i.e., does the system provide adequate feedback)?
• Encourage users to work through impasses. Users might feel stuck at certain points of the walkthrough. Rather than immediately directing them back on track, it can be useful to observe how users troubleshoot difficulties or recover from an initial mistaken action. Encourage users to continue working through for a few minutes. If they are still stuck, provide gradually increasing levels of assistance.
• Inquire about context of use. Encourage users to reflect on how using the product might play out in a real-world use environment. What would it be like to use this product in their workplace or at home? Are there any characteristics of the user (or their patient) which might affect their interaction?
• Thoroughly debrief on the walkthrough. After completing the walkthrough, first invite the user to discuss what stood out most about their experience with device. Afterwards, talk through any observations that you would like to further explore with them. Many best practices of debriefing on usability findings also apply to discussing a cognitive walkthrough. However, you might also dedicate a section for more participatory design discussion about potential improvements and/or alternative solutions.
• Remain open to unexpected directions. Users might struggle with steps you expected to be straightforward or propose unanticipated alternative designs. Approach these moments as opportunities to learn more about how actual users’ mental models might differ from your initial assumptions and pre-conceived designs. In other words, you might spend more time in the moment discussing alternative ideas and design expectations than you would in a formative usability test, for example.

Reporting

• Synthesize key findings. Identify trends in observed usability issues and user feedback on the product. Pay special to attention to any feedback which updated your understanding of user needs, potential use-related risks and/or design mitigations.
• Draw a throughline between findings and subsequent design directions. Findings during a cognitive walkthrough often highlight strengths and shortcomings across multiple concepts. As such, document recommendations in a manner that enables the team to freely leverage the feedback toward an optimal design, even if that design becomes a hybrid of features. As you develop the next round of the design, highlight design decisions made in response to specific findings to provide traceability through the early-stage HFE process.
• Update other HFE documentation as relevant. Add newly identified risks to your use-related risk analysis. Apply lessons learned from working through a use scenario in the walkthrough to future usability study protocols.

Optimizing HFE processes with cognitive walkthroughs

Cognitive walkthroughs provide an opportunity to proactively identify usability issues and promising mitigations before committing significant time and resources to a product design direction. Cognitive walkthroughs also serve as one method of meeting regulatory expectations to conduct preliminary HF analyses of a new device. Ultimately, conducting a cognitive walkthrough can make your overall HFE process more cost-effective and efficient by pinpointing usability issues and revealing promising directions for your product’s design at an early stage of development.

Contact our team to learn more about cognitive walkthroughs. Or, sign up for a complimentary account with OPUS, our team’s software platform that provides HFE training, tools and templates.

Harrison Preusse is a Senior Human Factors Specialist and Ashley Carroll is a Human Factors Specialist at Emergo by UL.