Usability Testing of Medical Devices Used by Teams

By Laura Birmingham and Alix Dorfman

Conducting usability testing of medical devices most often involves one participant at a time – the intended user of the device. However, there are some products that require multiple individuals to work together to perform tasks in a safe and effective manner. For example, a robotic-assisted surgical system might require a surgeon, a scrub nurse, a circulating nurse, and maybe more, all working collaboratively to support the safe use of the system.

When conducting usability testing for medical device systems, you would likely test the different intended users not as individuals, but rather as a team. While the approach for conducting these “team testing” sessions might be similar to the approach with individual participants, there are a number of variables that should be given special consideration. Below, we summarize several variables to consider when conducting a team-based usability test.

Recruiting for team-based usability testing

How many usability study staff members should conduct a team session?

It is important to consider how many staff members are needed to conduct team testing to ensure all participants’ interactions are effectively observed and captured. When conducting a study that only includes a single participant, our standard approach is that one moderator and one analyst (sometimes referred to as the “note taker”) collaborate to conduct the sessions. This approach enables the analyst to focus on capturing detailed observations of the session in real time, while the moderator focuses on leading the session and interacting with the participant.

For team testing, it usually works well to have at least one study staff member (i.e., moderator or analyst) per participant from the participant team. So, if you have a usability study involving a surgical team of three participants who collaborate to use a product, then you might consider assigning three staff members to conduct the study (i.e., two analysts and one moderator). This enables at least one staff member to observe each participant to ensure no key interactions go unnoticed. This staffing tactic can be particularly suitable if participants tend to perform highly collaborative, serial tasks (e.g., working as a pair to lift and install equipment). 

That said, if you expect that participants will be performing many tasks in parallel and independent of one another, then you might consider including one additional staff member to ensure there is at least one analyst devoted to each participant. For example, if you have three participants working in parallel, then you might consider staffing the test with three analysts in addition to the moderator. Sample tasks that fit this bill might include a scenario in which one participant assembles a component, another prepares the surgical field, and another interacts with a graphical user interface (GUI) to select an optimal surgical route for an upcoming procedure.

Who do you recruit for a team-based usability test?

As is often the answer when it comes to human factors engineering of healthcare and medical technology: it depends! Here, it depends on what constitutes the most representative nature of your intended users. If you expect that your system will be used by individuals who typically work together, then it would be best to replicate this level of familiarity among your recruited participant teams (i.e., recruit participants that currently work together, as teams). This will ensure that the participant team has a representative dynamic, including an established communication style and an understanding of each team member’s expected role and responsibility. Notably, it is helpful to have one team member refer or recruit their team members for the study, rather than needing to recruit each team member for the usability test individually.

If, on the other hand, your system will be used together by a rotating list of staff members who might be collaborating for the first time, you might want to recruit at least some individuals who have not worked together in the past (i.e., recruit participants individually to work together as a team for the first time during the study). That said, it is more common that team members will be familiar with one another, and efforts should be made to recruit as such.

Team-based usability testing: How to debrief with the participants

When debriefing your participants to learn about any usability issues encountered during the test session, you have another decision point: either debrief all participant team members together or debrief each separately (or in sub-groups).

Conducting an interview with participants as a team provides the opportunity to benefit from participants’ collaborative responses. For example, one participant might better understand why they had a certain assumption while performing a task if they are privy to hearing their teammate’s take on the situation. In other words, root causes beget other root causes.

However, discussing the test session as a group might limit researchers’ insights due to real — or perceived — chain of command. Simply put, an assistant might not feel comfortable reporting that a doctor committed a mistake due to their established hierarchy in the workplace. As such, participants might not be as “open” and forthcoming as needed to sufficiently glean participant-reported root causes.

In some instances, it might also be preferable to debrief participants as a group and then individually (or vice-versa) to benefit from both debriefing approaches. For example, you might choose to interview the participants as a team initially, and then follow up with separate individual discussions. Or you could divide the debrief discussions by relative roles. Considering again an example of a surgical team, you might choose to debrief the surgeon separately but interview the rest of the team (e.g., the support staff) together. That said, some of these configurations might require a bit more time (and/or space), so it is important to consider these logistical limitations as well. 

Conclusion: Conducting efficient team-based usability tests

There are certainly multiple ways to approach team-based usability testing. What is most important is ensuring sufficient consideration of all of these aspects and tradeoffs before conducting a study. This will enable manufacturers and researchers to make informed decisions when designing their test method to maximize representative participant interactions and thoughtful insights collected.

Laura Birmingham is an Associate Research Director and Alix Dorfman is a Managing Human Factors Specialist at Emergo by UL.