Knowledge Tasks, Product Labeling Design and FDA Guidelines: Webinar Debrief

By Andrea Dwyer and Laura Birmingham

On July 26, 2023, our Human Factors (HF) team’s Andrea Dwyer and Laura Birmingham delivered a webinar about how to design and administer knowledge tasks that meet the US Food and Drug Administration's (FDA) expectations and help ensure instructions and warnings are clear and effective. They discussed the purpose of knowledge tasks broadly; the FDA’s formal guidance and informally communicated expectations regarding knowledge tasks; the design, administration, and analysis of knowledge tasks in a two-phase approach; and key considerations to facilitate effective critical task evaluation via knowledge tasks. 

What are knowledge tasks in usability testing? 

Knowledge tasks serve to assess users’ understanding of important use-related information that is often located within a product’s labelling, including the instructions for use (IFU) and packaging. Knowledge tasks are a component of usability testing. During usability test sessions, moderators present evaluation activities and conduct subjective feedback interviews; knowledge tasks occur during the evaluation activities portion of a usability test session. Knowledge tasks are most frequently included as part of HF validation testing performed for FDA purposes. However, you might choose to evaluate knowledge tasks during a pre-summative or formative study, or for regulatory bodies other than FDA. 

Understanding FDA’s guidance on knowledge tasks in usability testing 

FDA’s written guidance provides some information related to their expectations for knowledge tasks.  

In the guidance, the Center for Devices and Radiological Health (CDRH) acknowledges that, “Some critical tasks involve users’ understanding of information, which is difficult to ascertain by observing user behavior. . .This knowledge can be tested by questioning the test participants.” This excerpt indicates FDA anticipates that there will be some situations that cannot be easily observed and that they expect that study administrators will collect the necessary data by asking questions instead.  

FDA’s Center for Drug Evaluation Research (CDER) also explains the goal of knowledge task studies in their guidance, in which they state, “In comparison to other types of HF studies in which critical task performance is assessed by observing user interaction with the product, Knowledge Task studies focus on the understanding and interpretation of important information in the user interface that will be applied to make use related decisions.” FDA also explains how they expect the users’ understanding of the labeling to be evaluated, “by questioning test participants and assessing whether the information has been understood.” Here, FDA is reiterating the need for knowledge tasks to evaluate what test participants know and understand, which cannot be easily observed. 

Regarding knowledge tasks, FDA’s written guidance communicates the following: 

• Knowledge tasks are an anticipated component of usability testing. 

• Knowledge tasks should be used for tasks that cannot be adequately observed when usability test participants perform hands-on tasks. 

• The labeling components that contribute to participants’ knowledge evaluated in knowledge tasks include Instructions for Use, Quick Start Guide, device labeling, and training.  

• Example situations when knowledge tasks might be used include when users might need to understand critical contraindications and warning information.  

FDA’s guidance does not provide as much practical advice, such as writing and preparing knowledge tasks or how a study administrator should administer knowledge tasks during usability test sessions. Further, the formal guidance does not indicate the FDA’s expectations for proper analysis of knowledge task data. Fortunately, there are additional means to understand the latest regulatory expectations, including comments and feedback from public forums that FDA attends and inputs we see from our clients’ communications with FDA. 

Overview of knowledge tasks and the two-phase approach 

Based on FDA’s guidance from these various sources, we developed a two-phase knowledge task approach. Phase 1 (the knowledge check) is intended to broadly assess participants’ knowledge and/or how they would seek to answer questions intended to evaluate critical tasks. They might or might not choose to use the IFU or other labeling when answering these questions. Phase 2 (reading comprehension) involves directing participants to read and interpret specific IFU or other labeling content that serve as mitigations to critical tasks that otherwise cannot be evaluated effectively during the hands-on tasks. This ensures that participants also see these mitigations to evaluate them.  

Note that these knowledge tasks generally will occur after participants perform all hands-on tasks. We perform this two-phase knowledge task after the hands-on tasks (or use scenarios) rather than before to make sure that we are not biasing participants to use the IFU when performing the use scenarios. 

Check out our webinar and receive practical guidance from our human factors experts on the following topics:  

• Writing protocols: developing comprehensive knowledge tasks 

• Conducting usability test sessions: administering effective knowledge tasks 

• Analyzing data: evaluating knowledge task results 

Contact our team to learn more about conducting knowledge tasks to meet FDA’s expectations. Or, sign up for a complimentary account with OPUS, our team’s software platform that provides HFE training, tools, and templates.    

Andrea Dwyer and Laura Birmingham are Associate Research Directors at Emergo by UL's Human Factors Research & Design division.