BLOG

Want to prevent poor usability? Conduct usability tests!

July 7, 2022

7-minute read

We encounter usability every day: from coffee machines to smartphone apps to ticket vending machines. Usability is usually translated as “ease of use,” and it is precisely this that is the decisive factor in the user experience during human-machine interactions. In the worst-case scenario, users are confronted with negative emotions. For example, anyone looking to buy a transit ticket in Washington will likely find themselves standing in front of a ticket machine like this one:

Source: Twitter@RianVDM

Due to the large number of haphazardly arranged labels and markings, this system is anything but intuitive to use. It’s safe to assume that inexperienced users will have difficulty purchasing the correct ticket and will also experience negative emotions such as frustration or stress. However, there is a simple way to avoid poor user experiences: usability testing. The goal of a usability test is to identify problems and weaknesses in the use of software or a device. To do this, a test subject performs typical use cases (tasks) using the test subject. Just five rounds with one test subject each are sufficient to identify approximately 85% of all usability problems in the test subject (Nielsen, 2000). Usability tests with fewer participants are more likely to yield incomplete results, while usability tests with more participants usually only lead to duplicate observations. The amount of work involved is therefore manageable compared to the impact of poor usability.

Proper Planning

When beginning to plan a usability test, the first step should be to clarify the goal of the study. The most important questions to consider are:

Which test item should be examined, and for what purpose?
Which user groups are relevant for the test?

Once these questions have been clarified, a suitable testing setting is selected. The quickest method is the hallway test, also known as the guerrilla test. In this approach, people who happen to be nearby—usually employees from the company itself or people on the street—are interviewed spontaneously and briefly. The classic usability test in a lab requires significantly more preparation. Here, the “lab” refers to a room with a prepared workstation. For this type of test, test participants are recruited in advance and appointments for the usability test are scheduled. Throughout the entire test, the test participant and the interviewer sit together at the prepared workstation, while observers watch through a one-way mirror or via livestream. This form of usability testing can also be conducted remotely, e.g., when test participants are geographically dispersed. In this case, the interviewer and the test participant join a web conference together, and the use cases are worked through via screen sharing. There are now also providers offering remote usability testing services. However, since there is no interviewer present who can ask follow-up questions in case of problems, this form of usability testing is only recommended to a limited extent.

The exact procedure for the planned usability test is documented in an interview guide. Such a guide is also helpful in ensuring that all project participants develop the same expectations regarding the test and its content in advance. The interview guide lists all questions and tasks and sets out the schedule. During the test, the interviewer follows the guide to ensure nothing is overlooked.

The following structure for the interview guide has proven effective:

Introduction and Instructions
Preliminary Interview
Processing the use cases)
Follow-up Survey

Implementation

Once the dates have been set, the interview guide has been finalized, and the technical setup is ready, we can begin conducting the test. Since usability tests have recently only been possible remotely—and this format will likely remain popular in the future—we would like to use this testing setup as our best practice.

At the start of the scheduled appointment, the interviewer and the test subject meet via Microsoft Teams, Zoom, or another suitable web conferencing tool. With the test subject’s consent, the interviewer begins recording the screen and audio. The recorded video is particularly helpful for reviewing critical points later during the evaluation. It also allows you to edit the individual usability test videos into a highlight reel, which provides all project participants with a quick overview of the results. Such videos are particularly effective when eye tracking or mouse tracking is also used during the usability test. However, for usability tests conducted remotely, these testing methods can only be implemented with considerable additional effort.
Once the recording has started, the interviewer and the test subject begin the usability test. A preliminary interview usually takes place first. This serves to gather important information about the typical usage context and also acts as a sort of warm-up for the test subject.

After the preliminary interview, work on the use cases begins. The interviewer presents tasks that the test subject is expected to complete without assistance. It is essential that the test subject think aloud and voice their thoughts. This makes it possible to identify areas of confusion and potential for improvement. It is also important that the test subject interacts with the test object as naturally as possible—just as they would in a real-world usage context. Meanwhile, the interviewer asks context-appropriate questions from the guide. The interviewer may also ask their own spontaneous questions, which are often very useful. These should be phrased simply, as closed-ended questions, and should not be hypothetical or leading, so as not to influence the test participant. Once the use cases have been completed, a debriefing usually takes place. During this debriefing, the interviewer asks questions about the test subject’s previous interaction with the prototype. The goal is to revisit any usability hurdles and ask the test subject about their overall impression. The debriefing marks the conclusion of the usability test with this test subject. The interviewer then thanks the test participant for their cooperation, ends the recording of the test session, and says goodbye. This process is repeated until all test participants have taken part. The usability test is thus complete, and the evaluation of the results can begin.

Analyze observations

To evaluate a usability test, it is necessary to observe what happens during the test. Typically, this role is filled by project participants or other UX team members. Observation can take place directly during the tests via a livestream or by reviewing a recording of the session.

Observations should be noted on sticky notes. It is important to assign each test subject a specific color and to use a separate sticky note for each observation. It does not matter whether the sticky notes are physical or virtual. In a remote setting, we recommend using a digital collaboration tool such as Miro or Google Docs. There, all observers can easily collect digital sticky notes on a virtual board. If the observation takes place in person at the office rather than remotely, sticky notes can alternatively be written on and attached to the walls.

Once the sticky notes have been collected in one place, they can be sorted and clustered. We recommend first grouping together sticky notes with the same or similar observations. Duplicate sticky notes (same observation, same color) can be safely discarded. By clustering the sticky notes, you can identify areas where problems have occurred repeatedly and also recognize problem patterns and causes. The colors quickly show how many test participants are affected by a particular problem. This makes it clear which interactions more frequently lead to problems and which observations are more likely to be isolated incidents.

Once the clustering is complete, a list of all observations can be created from the sorted sticky notes. The individual problems should then be prioritized based on how frequently they occurred and the severity of the limitation they caused for the test subject. Afterward, initial solutions can be added to the list for each problem. This results in an overview at the end of the usability test that includes all problems, their severity, and possible solutions. Based on this list, a decision can then be made as to which of the identified problems should be resolved and what approaches are available to address them.

Properly conducted usability tests offer a relatively simple way to identify the majority of usability issues that arise. This helps prevent negative user experiences and significantly enhances the product’s appeal. Good usability has become indispensable for modern products and, through positive user experiences, also contributes significantly to a company’s reputation. It paves the way for a barrier-free future.