Essential Differences Between Heuristic Analysis and Expert Review

To understand how users in the real world will interact with a product in development, there is no substitute for testing with real users. However, usability testing is not always feasible due to constraints on timeline, budget, and access to representative users, as well as unique circumstances such as the current COVID-19 situation. Fortunately, performing other evaluative activities prior to usability testing can help you maximize your time with users to yield the most valuable and relevant findings.

What are heuristic analyses and expert reviews?

Two such evaluative activities are heuristic analyses and expert reviews. Both are common interface assessment methods that involve one or more usability professionals conducting a review of a product to identify interactive strengths and shortcomings. However, there are key differences between these two evaluation methods. Understanding these differences can help you determine which approach is most appropriate to achieve your evaluation goals.

To begin, let's define each method.

• To conduct a heuristic analysis, reviewers evaluate an interface with respect to a predetermined list of design principles or "rules of thumb" (i.e., heuristics).*
• To conduct an expert review, reviewers evaluate an interface based on their professional judgement (i.e., prior experience and education), as well as established human factors engineering (HFE) guidelines.

What are the differences between these approaches to evaluating a product design?

How do these methods differ in practice? To demonstrate, imagine two reviewers have one hour to evaluate a new product. Reviewer A will perform a heuristic analysis and Reviewer B will perform an expert review. Reviewer A comes to the evaluation with a checklist, while Reviewer B shows up with a blank notebook. To perform the review, Reviewer A goes through the checklist, which consists of a set of usability-related criteria, or heuristics, and assesses whether the product fulfills each criterion on the list. On the other hand, Reviewer B takes notes on a blank page, drawing upon professional experience and knowledge of established guidelines to determine which aspects of the product to assess and how to make these assessments.

By following their respective methods, both reviewers will identify strengths, shortcomings, and opportunities for improvement to the product’s interface. In fact, it would not be surprising for there to be significant overlap between their findings: Because professional judgement is grounded in best practice, Reviewer B is likely to assess the product with respect to many of the heuristics on Reviewer A's checklist. However, each review is also likely to generate some unique findings. Because the heuristics are determined before becoming familiar with the product, Reviewer A’s checklist might not anticipate aspects of the interface that Reviewer B finds noteworthy. Additionally, in some cases, the heuristics might be deliberately limited in scope to focus on specific interface features or qualities. On the other hand, Reviewer B might not identify aspects of the interface related to the checklist’s heuristics as particularly notable strengths, shortcomings, or opportunities.

Which approach is best for my medical device, combination product, or IVD?

So, how should you choose which approach to take? Review the following considerations to help determine whether a heuristic analysis or an expert review is the best fit for your evaluation goals.

• Scope. If your goal is to assess and improve specific features or qualities (e.g., a device’s ergonomics, an instructional document’s readability), consider conducting a heuristic analysis using a set of heuristics catered to your desired focus areas. For a broader evaluation to assess your product as a whole, consider an expert review.
• Timeline. A heuristic analysis’s “checklist” approach can facilitate a quicker evaluation. The predetermined heuristics set a clear road map for the review, enabling an efficient process. On the other hand, the open-endedness of an expert review means that its scope could expand over time as the reviewer(s) identify more findings of interest. As such, consider a heuristic analysis for tight timelines, and an expert review if you have more flexibility.
• Development stage. Conducting a heuristic analysis on an early-stage concept can be difficult if many of the interface’s details are not yet fully defined. If your product is still in an early-concept phase, consider conducting an expert review, supplemented with recommendations for interface aspects that have yet to be defined.

In summary, both heuristic analyses and expert reviews can uncover potential interaction problems, enabling you to implement design mitigations before exposing your product to users. While you are likely to derive value from either approach, with a greater understanding of the differences between them, you can select the best fit for your product and goals.

*To select heuristics for your analysis, you can choose an established list, such as Jakob Nielsen’s widely used 10 Usability Heuristics for User Interface Design. Alternatively, consider leveraging such a list as a starting point while also drawing from other guidelines and standards to develop a list catered to your goals.

Hannah Miller is Human Factors Associate in Emergo by UL’s Human Factors Research & Design division.

Learn more about HFE and product evaluation for medical devices:

• HFE user research support for medical devices, IVDs, and combination products
• Medical device and product evaluation
• Webinar: Human factors engineering for medical devices