Five Questions About Inspection Methods
What they are; why, how, and when to use them in UX research; plus a bonus question
Inspection methods are a powerful and inexpensive set of tools that any UX team can add to its research toolkit. Many researchers are familiar with at least one inspection method, and most UX teams report conducting them, even if only rarely.
Yet there are a number of points on which professionals might be confused or disagree. What exactly is an inspection method? When should you use one, and why? What are the specific methods to actually do them? And what place do they have in our arsenal of research methods today?
Taking a cue from a classic paper about one specific inspection method, let’s unpack the story of inspection methods — past, present, and future — in five questions.
1. What are inspection methods?
Let’s begin by defining terms. You may know the group of inspection methods by other names, including analytic methods, or expert reviews (although, some contend an expert review is a specific instance of inspection method).
These methods serve as a contrast to empirical or observational methods, like usability testing, which recruit representative users as participants for study. Rather than observe user behavior, inspection methods primarily involve the judgments of evaluators who review the product in a systematic way. These reviews are based on established principles of behavior and design, and the experiences of the reviewers who, having experience with observational studies, can make deductions from the steps of the task itself and anticipate some of the common issues to be encountered.
The reviewers’ expertise is typically in the domain of user experience and related fields. Having other expertise can be helpful as well, such as the domain of the experience under review. For instance, a health-tech startup reviewing products for doctors might involve a healthcare practitioner. Even non-experts — often, users themselves — have been employed from time to time, though typically as a secondary evaluator.
Inspection methods generally produce a similar output: a set of findings about the experience reviewed, including usability issues uncovered, potential strengths, and recommendations for improvement. For this reason, multiple evaluators are always helpful to get more exposure to more potential issues and to strengthen the validity of the output, provided that they work independently and only come together afterwards to reconcile findings.
Depth is brought to you by Drill Bit Labs, a UX research and digital strategy consulting firm. We solve complex user experience issues for digital product teams.
2. Why do inspection methods?
Inspection methods are particularly useful whenever traditional observational methods with users or test participants are prohibitively expensive. This may be due to the logistics of recruitment, the cost of incentivizing participants, the timeline required, or internal obstacles (such as process or regulatory compliance).
Since you don't have to go through the recruitment process — which involves developing a screener, administering it, and identifying candidates — you can simply access the system and conduct the evaluation within your team, greatly improving your speed to insights. As long as the evaluators are trained in these methods and have some familiarity with them, they can be completed quickly.
In short, in situations where empirical methods might otherwise leave you unable to provide stakeholders with needed information for decision-making — perhaps due to an urgent deadline, or unforeseen logistical obstacles emerging during a traditional usability study — inspection methods make for a great supplement or a viable alternative.
3. When might you use inspection methods?
Although inspection methods may be used at virtually any point in the design process, they are typically used early in the process, during the pre-development phase of designs. You get the most “bang for the buck” when you find issues to fix ahead of launch, perhaps even before it’s possible to get the experience in the hands of potential users.
Another common use case is to employ inspection methods prior to an observational study, like a usability test. Since evaluators can often identify some of the most significant usability issues through inspection, those improvements can be made ahead of a test with representative users — empowering your team to focus on deeper and less common issues.
Further, I suggest using them both early and often. When inspection methods are used frequently in an organization, the benefits compound.
Research teams will find efficiencies with a method simply by virtue of its familiarity. Regular exposure to inspection methods educates stakeholders on what to expect. In contrast, if a team conducts a heuristic evaluation a few times a year, there will be some cost of inertia from switching from typical observational methods and re-familiarizing with the process. Creating a culture comfortable with inspection methods can yield significant benefits.
4. How do you conduct an inspection method?
There are a number of options, including: guideline review, heuristic evaluation, task analysis, and cognitive walkthrough, among others.
Guideline review
Organizations develop guidelines for their own design standards, which evaluators can use to review a system for compliance. These detailed reviews were most common in the days before heuristic evaluation (see below), although there are more generally-applicable guidelines in contemporary use, including MeasuringU’s Calibrated Evaluator’s Guide, the (archived) Usability.gov Guidelines, or Baymard’s proprietary guidelines.
How it’s done: Practitioners take a set of guidelines and go sequentially and thoroughly through a flow on the experience. Each guideline is reviewed for applicability, and reviewers note violations.
Although less common than heuristic evaluation today, teams that want a more exhaustive review tailored to their experience’s domain should consider them. This level of relevant detail can potentially help to uncover more issues and offer more specific recommendations than other inspection methods. However, guideline reviews are perhaps the most time-consuming inspection method — a cost that should be weighed against alternatives.
Heuristic evaluation
An original set of heuristics was drafted in 1989 by Jakob Nielsen and Rolf Molich based on their own judgments about usability at the time. But they were later refined based upon a statistical analysis of 249 usability problems, using a total of 101 competing heuristics or guidelines to “explain” each issue. This data-driven approach yielded a final set of 10 heuristics in 1994, which remain in usage today. They have since remained common, and have had an enduring appeal.
How it’s done: Multiple evaluators review each step in a task using the experience being inspected using the 10 heuristics developed by Jakob Nielsen. Reviewers note any violations of the heuristics, give a severity rating, and may also provide potential recommendations for improvement.
Part of the enduring appeal of heuristic evaluation is its simplicity. Practitioners with some familiarity with usability testing and principles may be quickly trained and ramped up on the method, the 10 heuristics, rating severity, and reconciling findings. The amount of issues then uncovered by heuristic evaluation can generate financially-valuable usability improvements in a very short time (and thus, at very low cost in labor).
Somewhat ironically, the heuristics have been adapted for use in specific domains, including virtual reality, ambient displays, and video games, which offsets its potential disadvantage relative to guideline review.
Nevertheless, there are some downsides to heuristic evaluation. Its popularity and simplicity can lead to potential abuse by evaluators conducting the method more informally and without the necessary rigor. Further, the validity and reliability of heuristic evaluations have been extensively studied — and even under the best of circumstances, multiple studies note that heuristic evaluations typically identify only 30–50% of the issues an observational method might uncover. Additionally, about 34% of issues identified are “false alarms” that few if any actual users encounter.
Task analysis
A task analysis is a family of methods that take a set of actions with a clear start and end point, define it, and break it down into the discrete elements (such as sub-goals or specific steps) needed to achieve a desired outcome. Since we have written extensively about task analysis in UX work, we will focus on one instance of task analysis: PURE (Pragmatic Usability Rating by Experts), developed by Christian Rohrer and Jeff Sauro, which quantifies the friction present in a task.
How it’s done: A first step in PURE is decomposing the task into its discrete steps along the ideal or “happy path.” Evaluators independently review each step for issues or friction, and assign the step a severity rating on a color-coded scale from 1 to 3. An overall score is summed across the steps, which then inherits the color of the most severe step. This may be repeated across multiple key tasks in an experience to produce a holistic PURE score, as in the example above.
I think of PURE as the most executive-friendly version of task analysis. Its output creates a visually appealing but also immediately understandable description of the complexity of a task or set of tasks, showing precisely where the areas of greatest friction occur.
Cognitive walkthrough
The cognitive walkthrough was developed in the early 90s by a set of HCI researchers including Clayton Lewis, John Rieman, and Cathleen Wharton, among others. It’s intended to gauge how well a complete novice could learn to use any system they’ve approached without specific training or experience, such as an ATM kiosk or self-checkout machine in a grocery store. Although best suited for these “walk-up-and-use” experiences, it has been applied more broadly in the years since.
How it’s done: A set of evaluators working together in a workshop setting, review each step in a task, asking four key questions:
Will users try to achieve the right result?
Will users notice that the correct action is available?
Will users associate the correct action with the result they're trying to achieve?
After the action is performed, will users see that progress has been made toward the goal?
Cognitive walkthroughs are better suited than other inspection methods to assess its specific goal: the learnability of “walk-up-and-use” experiences. Nevertheless, going through four questions for each step in a task can become time-consuming. For this reason, Rick Spencer, a researcher at Microsoft, developed a streamlined version for use in industry settings.
The list goes on
Over the years, a number of books, papers, and blog posts have been written about inspection methods. If you dive deeper into the literature, you will encounter others not mentioned here — some lesser-used, some verging on obsolete, and some variants of the above.
It’s also possible to combine inspection methods in a single project. For example, using both task analysis and heuristic evaluation can serve to make the heuristic evaluation much more thorough. Or, using both cognitive walkthrough and KLM could give both learnability issues and estimates of task completion times. This can enhance the strengths of each method while offsetting disadvantages.
5. Whither inspection methods?
To understand what the future might hold for inspection methods, it may be helpful to examine its past and present. You may have noticed that most of these methods were developed in earlier days of HCI and usability — the 80’s and 90’s — when conducting observational methods was labor-intensive and expensive, before modern forms of automated participant recruitment, and most often in a dedicated laboratory setting. This is no coincidence. Organizations needed to know where the usability issues were, and inspection methods are an attractive, low-cost tool to find them.
At one point, heuristic evaluation was the second most commonly used method after usability testing. In recent years, it appears in only 7% of UX Researcher job descriptions. What happened?
In the years since inspection methods were first developed, the field of UX, including our tools and processes, has changed considerably — and observational methods have become drastically cheaper. We take platforms for remote testing for granted today, but they have made it much easier to get feedback from representative users.
Nevertheless, the field continues to change. In recent years, the tech industry in general and UX in specific has been subject to repeated layoffs and a tight labor market, leaving many teams squeezed for resources. Today’s UX teams might look to inspection methods as a tool to get more from less.
And although the rise of next-generation AI tools, including LLMs like ChatGPT has accelerated aspects of UX research, it also creates concerns about the data quality from remote tests. The old practice of combining inspection and observational methods might today be used to improve both.
Drill deeper
Depth is produced by Drill Bit Labs, a leading UX and digital strategy consulting firm working side-by-side with UX and product design leaders to elevate their UX strategy, delight their users, and outperform business goals. Ways we can work together:
User research to inform confident design decisions and improve digital experiences
Live training courses that teach teams research skills — including inspection methods
Advisory services to improve UX processes and strategy.
Let’s connect to discuss your goals and how we can help.