The Benefits of Running Pilot Tests for Usability Studies

By Aurélie Debeyre 

Why pilot tests are essential for usability studies 

Developing a methodology for usability studies can involve uncertainty and introduce questions such as “How long should the test session last?,” “How can I improve my setup?” or “Is this prompt clear enough?” Running pilot test sessions prior to formal testing can help you find answers to these questions and increase the data quality of your upcoming study. 

Efficient planning for pilot test success 

Pilot testing can be considered for all types of usability studies from preliminary research interviews to human factors validation tests. However, this step is particularly valuable while preparing for your product’s validation test as the stakes are usually higher, and you do not want to conduct a re-validation test due to issues in methodology or device readiness. The data collected during the pilot test sessions will help to identify ways to improve your methodology and ensure smoother test sessions. 

One or two pilot test sessions are usually sufficient to assess the methodology. Whenever possible, consider including at least one representative of each distinct user group. 

When scheduling a pilot test, remember to allow sufficient time to make any changes to the methodology after having finished pilot testing and before proceeding with the formal test. Determine the time required between the pilot sessions and the formal test based on the type of study. For instance, you may need a few hours for a formative study, a few days for a validation study or even weeks if it is logistically difficult to organize a session (for example, due to a difference in location).  

Key areas to optimize through pilot testing 

Running pilot test sessions can serve to improve the following aspects of your usability test methodology: 

• Test duration and scope. In some cases, it may be difficult to estimate the length of a test session and the training duration (if any). One or two pilot tests can be conducted to determine the appropriate duration of the session and adjust the test scope or training content if necessary. Assessing the test duration before starting the formal study will prevent you from modifying the schedule during the study or skipping tasks because the session duration is too short. 

• Device readiness. Depending on the type of device and the stage of product development, it may be useful to conduct a pilot test to confirm that the test material is working properly. Some products may also be damaged in transit, and it is preferable to verify that they are functioning as intended by completing all the tasks specified in the study. This ensures that any potential bugs or product malfunctions can be corrected before formal testing. 

• Use scenarios and task prompts. Assessing your protocol with a participant will enable you to identify if the prompts and questions are written in a clear and unbiased way and adjust them if needed. It will also enable you to confirm a natural flow of the session, and reorganize tasks and scenarios, if necessary, after the pilot sessions. 

• Test environment/Setup. During a pilot test, you might receive participant feedback regarding missing test items or unrealistic aspects of your test environment which will help you to adjust these details prior to formal testing. It is also the opportunity for you to evaluate your setup, especially your AV equipment and camera positioning to make sure you don't miss out on viewing the things you are evaluating. 

Cost savings, better data, and enhanced methodology 

Besides improving the methodology and test set-up as described above, the main benefits of pilot testing lie in: 

• Saving cost. Although there are exceptions, using a solid, approved methodology when conducting a usability test reduces the risk of missing data and, consequently, the need for further testing. Also, conducting pilot test sessions can help determine if your device is not ready yet for validation based on the number of findings or the necessity of further design changes after only two sessions rather than fifteen participants.  

• Getting better data. Giving your team the opportunity to run a few sessions before formal testing will enable them to start testing smoother and be better prepared for the potential hazards they may face during testing. It will also allow you to obtain better data and therefore lead to a better test report. 

While running one or two pilot sessions are already valuable for usability testing, it can sometimes be appropriate to conduct a larger pilot test. For example, a pre-validation test serves as a pilot that consists of including a small sample of participants per user group to assess the methodology on a larger scale. Recognizing the usual large investment that validation testing brings to manufacturers, conducting these pilot sessions are particularly beneficial. 

Using pilot testing to streamline usability studies and improve outcomes 

Contact us to learn more about the benefits of running pilot tests prior to conducting usability studies. Or, sign up for a complimentary account with OPUS, our team’s software platform that provides HFE training, tools and templates.

Aurélie Debeyre is a Senior Human Factors Specialist at Emergo by UL.