The researcher who tried it two years ago might dismiss it as producing generic, unusable output. The one experimenting today might find it surprisingly helpful for initial structure and probing questions. By next year, both perspectives could be obsolete, as indeed, LLMs continue to quickly evolve, with capabilities roughly doubling every seven months by one estimate.

A deeper challenge is that traditional evaluation methods fall short. Even if one conducted rigorous studies comparing AI-assisted versus human-only research processes, those findings would be outdated by the next frontier model release. This reality demands a different approach than the typical “best practices” playbook. We can’t rely on checklists, curated prompt libraries, or even broad mental models that cast AI as a “research assistant” or “brainstorming partner.” These frameworks become brittle too quickly.

Instead, let’s focus on something more stable: how humans interact with these tools. While LLMs rapidly change, human cognition is the product of a slow evolutionary process, and is unlikely to shift too quickly. Studies are finally starting to reveal patterns in how human cognition and artificial intelligence interact.

Introducing a recent MIT study

A June 2025 study with the memorable title “Your Brain on ChatGPT” by MIT scholars Nataliya Kosmyna and colleagues has garnered much attention. You can read the full 206 page preprint manuscript, but let’s briefly summarize what they did and found.

The authors recruited a total of 54 adult participants in the Boston area, and randomly assigned them into one of three groups: one that used LLMs, one that used search engines, and one that worked without outside tools (humorously termed Brain-only in the manuscript). Participants in each group were asked to write on different general interest essay topics over four sessions of 20 minutes each. In the first three sessions, although the topics would vary, participants could use only those tools described in their group assignment. The last session, which was optional (and had only 18 participants) swapped the members of the LLM group to Brain-only and vice versa, and prompted them to revisit a topic they had written on in a previous session. Below is a figure developed by the authors representing the study design.

This study has two interesting elements that contrast it with similar studies to date. First, they used EEG (electroencephalography) to measure brain signals. Although any measurement has its tradeoffs — EEG’s spatial resolution is limited — this gives us some indication of how LLM usage affects brain activity, such as cognitive effort. Second, it took place over four months. While a relatively short period of time, this longitudinal element provides a window into how patterns of brain activation change over time with repeated behavior.

What were the findings?

The most pertinent results come from comparisons between the LLM and Brain-only groups.

Compared with Brain-only participants, LLM participants’ essays were longer, more similarly structured, and less diverse in ideas and content. That should come as no surprise to anyone who’s spent much time on LinkedIn lately, which promotes AI-generated posts within the interface, and where the feed often feels gratingly uniform in style and tone.

In survey and interview responses, LLM participants felt less of a sense of ownership than the Brain-only group. This seems appropriate, even expected, given that LLM participants’ were largely using the outputs given them, instead of working hard to produce them. By the third session, authors summarize LLM participants’ essays as “mostly copy-paste [with] minimal editing”. This further showed up in EEG measurements, with lower activation and cognitive effort over the first three sessions. But this lower effort came at the cost of being able to recall specific quotes from the co-written text, which LLM participants had a much harder time doing.

So much for the first three sessions. What happened in the final session when the groups were switched?

Having revisited an essay topic they had already spent significant cognitive effort writing on, the Brain-to-LLM group’s EEG measurements showed a huge boost in cognitive activation suggesting “high levels of cognitive integration, memory reactivation, and top-down control.” Although they hadn’t used an LLM in previous sessions, their prompts were actually more sophisticated, i.e. more “information-seeking.” And even with the assistance of an LLM, they maintained a higher recall of the produced text than the previous LLM-only group.

As for the LLM-to-Brain group, their neural connectivity and engagement remained low. Nevertheless, their essays were rated highly by human judges — perhaps because they apparently reused phrasing and structures learned from previous interactions with LLMs on the topics.

Implications for user researchers

What might this mean for user researchers — who are under increasing pressure these days to deliver insights more quickly or to fall in line with company initiatives to go “AI first”?

For many reasons, we mustn’t be quick to over-interpret or generalize the results of a single study, even if some of its results have been seen in studies in other contexts. For one thing, this is a preprint on arXiv — which means it hasn’t yet been subjected to peer review, the standard for good science. A relatively small sample of only 18 participants completed the final and most interesting “switching” session. Further, artificial tasks in a laboratory with an EEG monitor on your head don’t exactly resemble the real world, and some of those differences may be important. That said, there’s a logical plausibility to the findings, and I (and I imagine many readers who have experimented with LLMs at work will) recognize some similarity in the overall pattern of results with personal experience.

Takeaway 1: The performance benefits of LLMs likely come with cognitive tradeoffs. To date, most studies of LLM usage in the workplace focus on performance outcomes, namely how quickly tasks are completed (i.e., efficiency) and how well the tasks are done (i.e. quality). Most — though not all — studies of this kind point to enormous benefits. Likewise, in the MIT study, the LLM group’s essays were judged as “deeper” and technically superior, even if one judge called them “soulless.” Interestingly, the most highly-rated essays were those produced in Session 4 by participants without LLMs who had previously written on the topic with an LLM’s help, suggesting they had learned to imitate the style and structure of LLM outputs.

Nevertheless, these benefits aren’t without costs. Many have already pointed out societal costs, whether the energy demands and ecological impacts of AI datacenters, or the ethical quandaries of using models trained on human-authored works without consent or compensation. But few, if any, studies have yet looked at the drawbacks to human cognition, and it turns out there isn’t a free lunch here either.

Takeaway 2: Outsourcing thinking likely means outsourcing learning and comprehension. One of the most commonly-touted benefits of LLMs is that they solve the “blank page” problem of getting a first draft going. And in my own experience, I have found LLMs excel in low stakes situations like drafting one-off communications such as an email asking for a refund. These are things I just want to get off my plate, that I have no desire to become an expert in doing. But when it comes to those things we want expertise in, going “AI first” will backfire.

Researchers, like other tech professionals and knowledge workers broadly, often have more tasks than time in which to do them. Allocating equally-intense cognitive resources to every task isn’t a winning strategy, if it’s even possible. So using LLMs can be one tool to offload some of that effort. But if these results are generalizable, asking an LLM to take the first pass at any deliverable relevant to our job function (whether a discussion guide or a report of findings) will leave us understanding it in an only superficial way. When we need to speak for those deliverables with stakeholders, we’ll be less authoritative and less persuasive as collaborators. It may even hamstring us later if we decide or need to do those tasks ourselves.

Takeaway 3: There may be an “optimal sequence” for LLM usage. Contrary to some popular accounts of the study, the point isn’t that LLM usage was unequivocally bad for the brain. (The authors specifically warn against drawing the conclusion that LLMs make people “dumber.”) The Brain-to-LLM group, which was given the chance to revisit (with an LLM) a topic previously written upon (without assistance), showed deeper engagement with the task and more nuanced usage of the tool.

In other words, you can potentially offset the downsides of LLM outsourcing by largely doing the work yourself first. In his comments on this study, Ethan Mollick, a University of Pennsylvania professor who has published extensively on the performance benefits of LLMs, plainly states that one should “become disciplined” and generate their own ideas and drafts before consulting with these tools.

Virtually every tool and SaaS product built to support user and market research work has in recent years added AI-powered functionalities — whether for drafting study questions, moderating interviews, analyzing data, etc. But for any professional wishing to develop expertise in research (and its component skills), turning first to these features may be counterproductive. At best (setting aside open questions about the reliability and biases of these tools), they may be helpful for non-specialists who have no wish to learn research. Otherwise, specialists should do the work first and (if desired) later compare with what these tools can generate to tweak and improve it.

I say “if desired” because — for now, at least, and based on these results — the performance benefit of using an LLM once you’ve already done the work is less clear. This isn’t to say there are none, but it wouldn’t show up in the efficiency savings observed in previous studies, and it didn’t emerge in human judgments of quality in this study.

The bottom line

Past studies on how people use LLMs at work have pointed to their upside with performance benefits and their downside with reliability issues (like hallucinations). On that latter point, since we’re still seeing rapid technical improvements in LLMs from week to week, there remains cautious optimism that present issues might be solved.

Nevertheless, human cognition changes slowly over many generations, so the effects of LLM use on our intelligence is unlikely to change. A recent study from MIT shows that over-reliance on LLMs effectively stunts learning and memory related to the task, which should make professionals interested in developing research expertise think twice about going to them first.

We shouldn’t consider the MIT study definitive — there’s more work to be done — but I’d argue it warrants our attention. For now, the best strategy may be to do research-related work yourself first before comparing against the output of either a general purpose LLM like ChatGPT or a research-tuned AI tool. I hope future studies will take a closer look at the benefits of alternative approaches. For instance, there may be effective more granular sequences, such as outlining with an LLM’s help before drafting the text yourself. If nothing else, this study highlights the need for a more nuanced look at how artificial and human intelligence interact.

“Strategic research” is often too narrowly defined as the foundational and discovery methods that occur early in the product design lifecycle. That limited view of research’s connection to product strategy undercuts its impact.

Research doesn’t just inform what to build (i.e., strategy formation); it lends to prioritization, ensures execution aligns with vision, de-risks launch decisions, and measures performance. In high-functioning product orgs, research powers strategy across the entire product design lifecycle (PDLC).

When research is embedded across the PDLC, strategy becomes clearer, execution improves, and product moves faster with less risk and better results. That’s not just good for researchers; it’s a competitive advantage for any organization serious about building the right things and making them work.

What we really mean by “product strategy” and how research fits into the loop

There’s a problem with the way practitioners talk about product strategy and research. Too often, foundational discovery methods are called “strategic,” implying that other methods are inherently not-strategic?

First, let’s get concrete about product strategy: Depending on who you go to, they’ll define product strategy a little bit differently, but generally, it is a comprehensive plan for a digital product that includes how the product will achieve a business’s goals, deliver user value, and do so feasibly. That’s not just a roadmap or a prioritized set of features; it is a set of decisions (e.g., defining clear goals, understanding target audiences, choosing the right digital channels, prioritizing initiatives that align user experience with business objectives) that help a digital product compete, deliver value, and evolve over time.

Other authors have argued that common distinctions between “strategic” and “tactical” research are artificial boundaries and that we ought to frame the strategic value of research by its scope rather than its method.

I agree with them, and I’ll take it a step further: These labelings and misconceptions overlook the fact that neither product research nor product strategy is a distinct phase; both exist in harmony throughout the product development lifecycle (PDLC).

Generally, we can break the PDLC into three broad phases:

1. Discovery - Gain a deep understanding of the problem space, frame opportunities, and identify what to build.

2. Design - Design solutions based on information from discovery. Test and iterate to deliver experiences that are effective, desirable, and viable.

3. Measure - Evaluate the effectiveness of solutions based on data, feedback, and real-world use to optimize the experience and inform future builds.

Product strategy decisions and product research that empower this decision-making occur at all three stages of this development lifecycle.

Discovery: Clarifying the problem space and defining opportunity

The discovery phase of the product development lifecycle focuses on gaining a deep understanding of the problem space and identifying what to build.

In terms of product strategy, these activities fall under what we’d call strategy formation. This is where we define the foundation of a digital strategy: shaping the positioning of a product, clarifying our problem space, aligning business goals to user needs, identifying market gaps, and assessing competitive advantages.

Discovery phase research identifies user needs and pain points, explores potential solutions, and informs prioritization decisions, all of which shape strategy formation. Research-led strategy formation comes with the benefits of validating that there are real problems worth solving, uncovering unmet needs or underserved segments, and setting early strategic direction based on evidence, not assumptions.

To achieve this research-led approach, teams typically employ methods like in-depth interviews, surveys, ethnography, social listening, prioritization studies (e.g., Kano), diary studies, and key driver analyses. For example, imagine a hotel chain in a discovery phase. They might conduct diary studies with frequent travelers to understand the end-to-end travel experience, from booking to check-in, and even the return home. Insights from this research might highlight that one of the top moments-that-matter for business travelers is check-in. This finding might help define a key pillar in the product’s strategic direction.

Design: Keeping execution aligned with vision

After discovery comes the design phase, where we build product solutions, translating strategic intent into specific designs, all while balancing ambition with feasibility. This phase typically includes frequent cycles of iteration, creating and testing design ideas to see how they hold up when presented to real users.

While the discovery phase is where we formulate product strategy, the design phase is where it is executed. But things don’t always go as planned; the iterative nature of this phase gives teams the opportunity to adjust direction and course-correct strategy. This is where research plays a critical role.

Research in the design phase powers strategy by helping teams identify what’s working and what isn’t. It tests whether strategy holds up under real conditions, de-risks the development process by catching issues before they go into code, and produces insights and recommendations for iteration.

Teams often rely on several different methods here, including usability testing, card sorting, tree testing, concept tests, accessibility reviews, first-click tests, five-second tests, and heuristic evaluations.

Imagine an enterprise SaaS platform that has historically focused solely on operations managers. Their new strategic objective is to expand adoption within existing accounts by supporting a second key user type: finance analysts, who need visibility into operations data for budget forecasting. The design team builds an interactive prototype of a new “Finance View” tailored to analysts’ needs. Research can reveal many different findings that would steer strategy:

• If finance users find the tools clear and useful, that may signal that the strategic expansion can work.

• If analysts express confusion, skepticism, or mistrust in the data, the team may need to revisit assumptions or even question whether this user type is a viable strategic target.

• If operations managers feel the changes disrupt their workflow, the team might revise how the new finance functionality is surfaced, for fear of alienating their core.

In short, design research doesn’t just tweak UI elements; it keeps execution strategically aligned.

Measure: Tracking performance and learning what’s next

Once product changes are live, we enter the measurement phase, where we evaluate how well solutions fare in the real world. In this phase, we assess product effectiveness based on feedback and usage data to optimize the current experience and inform future product direction.

Strategically, the measurement phase serves two key purposes:

1. Assessing results against strategic goals. Through product feedback and usage data, we can evaluate whether the outcomes we’re seeing align with what the strategy set out to achieve.

2. Determining what’s next. Measurement activities help shape both short-term updates and long-term product direction, including new opportunity areas that will drive the next cycle of discovery and strategy formation.

Measurement research gives teams the visibility into product performance needed to make these strategic decisions. To support this, teams rely on methods such as usability benchmarking, A/B testing, competitive UX analysis, live intercept surveys, and in-product behavioral data.

Imagine a travel booking company that conducts a competitive usability benchmark of its booking flow. The findings reveal that a key competitor excels in one area: pricing transparency. Their design communicates costs, fees, and taxes upfront, significantly increasing user trust and loyalty. This insight leads to immediate product optimizations, such as redesigning the pricing summary in the booking flow. But it also informs broader strategic direction; the company may decide to revisit pricing policies and make transparency a core pillar of its value proposition.

In short, measurement research powers strategy by helping track performance and shape what’s next.

The bottom line

Research powers product strategy throughout the entire product development lifecycle, shaping what to build, how to build it, and how to evolve it. Organizations that embed research across all phases of the PDLC have a significant competitive advantage, building the right things faster with less risk and better performance.

• Researchers should learn to articulate the strategic value of their work at every stage of the PDLC

• Product teams should invest in research across the entire lifecycle to ensure that strategy is formulated, executed, and assessed on evidence

• Executives should champion research in strategic decision points, requiring evidence-based backing for product bets from kickoff through key milestones.

Compared with designers (who shape the interface and interactions) and engineers (who build the technical infrastructure that powers it all), researchers focus on the users themselves. We want to understand what their needs and expectations are, and where interactions cause confusion or errors. But to truly influence product and digital strategy, we must be comfortable translating that into at least a high-level outline of potential solutions.

This article will explore how to decide when (and at what level) to give recommendations, how to effectively situate and prioritize them within a report deliverable, and how to summarize them for actionability.

Should researchers write recommendations?

This is a common and understandable question. Although user researchers often lack a design or engineering background and the precise context to dictate exact implementation or final form, they are experts in the users, their needs, and their pain points. This gives them a unique and valuable perspective on how solutions might manifest within the product or experience itself.

A recommendation is simply a practical instruction guiding a product development team on how to address a research finding, whether by making an improvement or fixing an issue. They generally live within a report deliverable, regardless of its specific format. For example, they might appear as a designated text box at the bottom of a finding slide in a presentation deck, as illustrated below.

The very name “recommendation” suggests a certain degree of tentativeness. It isn’t a command, nor is it a requirement.

You’re simply giving stakeholders something to consider. The recipient is then free to implement it, adapt it, or even disregard it entirely, based on their best judgment. Consequently, in most situations, there's little downside to providing them. At the very least, recommendations offer the significant upside of demonstrating that you've thoughtfully considered the implications of your findings.

Of course, not all recommendations are equal. What works for one researcher, one team, or one company might look totally different elsewhere. Let’s look at how some of those variables influence the recommendation.

Three approaches to recommendation depth

The appropriate level of detail for your recommendations depends on the established working relationships and communication norms with your report’s intended audience.

Recommendations can manifest in one of three primary forms: no recommendations at all, low-resolution recommendations, or high-resolution recommendations. In practice, these latter two categories exist along a continuous spectrum rather than as rigid choices.

No Recommendation: On some teams, there may be a desire to independently digest findings and determine what actions should follow. This approach might stem from an operating model that mandates discrete hand-offs, assigning recommendation responsibility to another specific role. Corporate culture can also play a part. For instance, an engineering team might historically interpret recommendations more literally than intended.

Regardless of the underlying reason, the optimal strategy here is to refrain from providing explicit recommendations. If you believe there are good reasons to challenge this policy, the most effective course of action is to start a discussion in an appropriate forum, rather than trying to sneak recommendations into your next findings report. Nevertheless, even here, clearly articulating your findings will help stakeholders formulate their own actionable next steps.

Low Resolution Recommendations: Researchers sometimes find themselves new to a team or engaging with unfamiliar problem spaces. For example, as an internal (or external) consultant brought in for a specific research engagement, you might lack prior product experience, and it may be unlikely that you’ll work on it again. As such, you’re unfamiliar with technical constraints and design decisions that team members take for granted.

While there are times and places for questioning those decisions or constraints, you lack the context to make those judgments effectively. Given this lack of context, the most appropriate approach is to offer directional recommendations. These describe how the user experience should feel or function, rather than providing detailed, prescriptive instructions. For instance, instead of dictating “add a search bar to the navigation,” a low-resolution recommendation might propose, “Users require a more efficient method to locate specific items on the page,” implying a solution without explicitly prescribing it.

High Resolution Recommendations: For researchers who are deeply embedded within a product space — who thoroughly understand the constraints and design conventions, have longstanding relationships with key stakeholders, and perhaps even possess a design background themselves — you are in a much better position to gauge the feasibility of proposed solutions.

Such researchers are able to offer highly detailed instructions regarding the interface itself, outlining its appearance, even down to precise pixel-level specifications. Recommendations of this nature frequently incorporate visual aids such as figures, mockups, or even interactive prototypes, presenting a high-fidelity example of the intended improvement.

When time permits and your audience is inclined to active participation, you may choose to frame low-resolution recommendations as How Might We statements. This approach shifts the onus onto stakeholders, challenging them to collaboratively brainstorm and develop solutions that deliver the outlined experience, effectively guiding them toward an outcome closer to a high-resolution recommendation.

Once you’ve set the dial to the right level of detail, the next task is figuring out the best way to package and present the recommendation.

Structure recommendations for impact

Good recommendations don’t exist in a vacuum. Rather, they are ideally presented within a structured communication sequence (FIRE, for short) designed to foster understanding, build credibility, and ultimately drive action.

1. Describe the Finding simply and provide any necessary context, such as where it occurs or under what circumstances. Since stakeholders aren’t often direct collaborators in the research, you need to make what happened clear to them. Visual aids like videos and annotated screenshots (using markups such as boxes, circles, and arrows) can be helpful to illustrate.

2. Detail the core problem and, more importantly, its Implication for the user. What negative effect is it having on the experience? If you are aware of business goals or KPIs that the product team is responsible for, here’s a great place to draw a connection between the issue the user is facing and those metrics. For example, a user’s inability to complete a purchase directly impacts revenue, while expressions of frustration could adversely affect customer loyalty or satisfaction. Include supporting evidence here as well, such as videos or descriptive statistics (e.g., frequencies).

3. Having articulated the finding clearly and concisely and given a rationale for its importance to the product, you then state the Recommendation, ensuring it aligns with the appropriate level of resolution discussed previously. Provide stakeholders with a suggested action designed to address the issues you’ve uncovered.

4. Follow that up by painting a picture of the ultimate Expectation — that is, the anticipated outcome or benefit that implementing the recommendation would generate. For instance, if a simple tweak would likely help users complete an upgrade that was previously extremely difficult, state that explicitly.

Within the context of the finding, its implication for the product or business, and expected outcomes for solving it, a recommendation is more likely to be understood and much harder to ignore. You can further “light a fire” for action by prioritizing which recommendations should be addressed first.

Help stakeholders choose what to fix first

Since resources are finite and not all problems can be addressed immediately (or even at all, within any given timeframe), we should do our stakeholders a favor and give them guidance on what to tackle and in what order. They may ultimately disagree with what seems most pertinent to address, but you’re still doing them a kindness by “solving the blank-page problem” and creating an initial draft of a plan that they can react to or adjust.

Traditionally, there are several ways to prioritize usability issues. One is simply the frequency of problem occurrence. If an issue affects a majority of participants in a small qualitative sample, it’s likely to affect a substantial number of real-world users.

While sensible, this frequency-based approach provides only a partial picture. An issue might affect nearly everyone but be so minor that it barely registers, causing at most a negligible delay or slight irritation. Conversely, certain issues may occur very seldom during a usability test yet qualify as “show-stoppers” — meaning a user encountering such a problem would be entirely unable to resolve or recover from it, leading to a critical failure with severe negative consequences.

This is why we consider both frequency and severity. Even rare, highly consequential issues often warrant higher priority.

It’s wise to combine both frequency and severity when making judgments about issue priority. Nevertheless, the final determination is less about punching these values into a calculator, and more about blending them with your intuition. These judgments are part of your unique contribution to the team.

Create scannable recommendation summaries

Whatever prioritization method you use, you should find a crisp and simple visual way of communicating the priority of issues in your report deliverables. One tried-and-true approach is a table with columns for a description of the finding, the implication, the recommendation, and its priority. This final column might use something like a “red, yellow, green” color coding, a star rating system, or even both for multiple encodings.

A sensible place for these tables is near the end of your report deliverable, just ahead of summaries or concluding remarks. Consider also distributing this table separately (e.g., as an embedded image in Slack or an email, or as a standalone PDF) so that stakeholders can pull them up easily for quick access and action.

The bottom line

Recommendations are researchers’ suggestions that stakeholders can accept, modify, or reject to address the negative consequences of a research finding. Aside from a few prescribed circumstances in which researchers should avoid providing them, they may be given at a level of resolution or specificity appropriate for the working dynamics of the team and the researcher’s level of technical expertise regarding their feasibility.

One rhetorically effective way of presenting them is within the FIRE context; that is, after describing the Finding and its Implications, and before describing the Expected outcomes or benefits. Since a single study often generates multiple recommendations, it’s important to prioritize them, considering both the frequency and the severity of issues. Stakeholders are more likely to act upon them when given a summary table with prioritization symbols.

No researcher wants their findings to go un-acted upon. Therefore, given all the advice discussed here, we suggest that you implement these changes to how you present recommendations in research reports. Your stakeholders will more quickly understand why you’re recommending what you’ve recommended and what the impact will be. They’ll also be much more likely to put your suggestions into action.1

User research rightly emphasizes testing products in realistic scenarios and contexts. However, in some cases, selectively testing isolated interface elements can be a quick way to uncover issues with early designs.

This article outlines when isolated testing is beneficial and how to structure those tests effectively, including three examples: icons, copy, and menus. When used responsibly, isolated testing can be a valuable part of a broader research approach, providing quick feedback and enabling iteration on early design elements without abandoning the importance of context.

Why context matters in user research

Testing interfaces in realistic contexts produces more valid research findings, resulting in better design decisions and better products.

Well-planned product research accounts for context, ideally across three dimensions: users, tasks, and scenarios.

• Representative Users
  We want to make sure we’re testing with representative users, meaning people who would actually be the ones using the product. They bring a specific mindset, expectations, behaviors, attitudes, beliefs, and domain knowledge shaped by their experience. Testing with stand-ins, whether that’s a teammate or a gen-pop participant who doesn’t match the product’s actual target user, can lead to misleading results or overlook critical ways real users would interpret or interact with the product.

• Representative Tasks
  Users don’t just click around randomly; they interact with products to achieve specific goals, which we refer to as tasks. Well-designed research is often task-based to ensure the product supports goal-driven interaction and decision-making.

• Representative Scenarios
  Scenarios refer to the situation in which the user engages with the product: where they are, what’s going on around them, how they’re accessing it, and what their mental state might be. Environmental and situational factors, like distractions, urgency, or lighting, can influence both comprehension and usability. For example, a feature that’s perfectly usable in a quiet office may become confusing or go unnoticed by someone on a noisy subway.

This trifecta -representative users, tasks, and scenarios- captures the real-world complexity that digital products must be designed to accommodate. Effective research aims to account for all three.

However, there are times when it makes sense to run quick, early tests of specific interface elements outside of a full task flow or realistic scenario.

The case for isolated testing

Isolated testing of interface components can be valuable early in the design process as a precursor to in-context user research.

Just like a builder might inspect their materials before constructing a wall, designers can validate interface elements before assembling a complete experience. In the early design phases, while you're still working with sketches, wireframes, or low-fidelity components, it can be useful to get quick feedback on basic comprehension, recognition, or clarity. Doing this early helps surface issues before investing significant time or effort into higher-fidelity design.

Not everything needs to be tested out of context, but in some situations, it’s useful to test individual components first, then follow up with task-based, in-context testing once the full flow is ready.

A two-step approach is often best.

1. Isolated testing. This is useful for early-stage feedback before full designs are ready. You might validate components for clarity, recognition, or effectiveness using methods like first-click tests, comprehension studies, targeted surveys, or card sorting. The goal is fast iteration and early issue detection.

2. In-context research. Once components are placed in realistic flows, you can evaluate whether they come together to support the user’s goals through task-based research with representative users and scenarios. This step uncovers system-level usability issues, unclear visual hierarchy, or interactions between elements that weren’t apparent when tested in isolation.

Whenever isolated testing is used, it should always be followed by contextual testing. Interactions between elements, environmental factors, and task context all influence how users understand and navigate the experience. Isolated testing is not a substitute for in-context research; it’s a supplement. Used selectively, this two-step approach can lead to a more efficient design process and ultimately more user-friendly products.

Let’s look at three examples of when this approach makes sense: icons, copy, and menus.

Example 1: Icons

It can be useful to evaluate early illustrations of icons to ensure they are recognizable and interpreted correctly by users before they go into a fully designed task flow. At this stage, we can test for recognition, distinctiveness, and meaning, even while the broader experience is still being designed.

For example, imagine we are working on the order customization flow of a pizza delivery app. A designer has created a set of draft buttons, with each one using an icon to depict a different topping.

We want to make sure users can easily recognize these icons, so we set up a simple survey:

• Half of the participants are shown the icons individually and asked, through an open-ended question, what each one represents.

• The other half is shown the full set of unlabeled icons and asked which one they would tap to select a particular topping in a first-click test.

We then analyze the results by coding the correctness of the open-ended responses and reviewing the accuracy of the first-click selections. We might learn that our onion and pepperoni icons are not particularly recognizable and need refinement before we move forward with integrating them into the ordering flow.

Example 2: Menus

Testing menus and organization outside the context of an interface can help assess how well the information architecture’s structure and grouping align with user mental models and support finding information quickly. These are commonly assessed in tree tests and card sorts, where IA elements are presented in plain text rather than fully-fledged interfaces.

Imagine you were redesigning a product’s account settings menu (e.g., Profile, Notifications, Settings, Security, etc.). Before implementing the revised menu, you might choose to set up a tree test to evaluate how well users can locate functionality within the hierarchy using intent-based tasks (e.g., where would you expect to click to change your password). We can analyze overall success rates and directness to inspect the findability and clarity of the new structure.

Example 3: Copy

Finally, testing copy in isolation can help validate that users clearly understand the message before it's placed within the larger context of an interface. By doing so, we can detect issues like confusing phrasing, missing context, or misaligned tone before those issues are embedded in a full design.

There are many different ways to test copy, depending on what you want to evaluate. You might focus on comprehension to check whether users grasp the key points. Or recall, which assesses whether they can remember the message after a short delay. You can also evaluate tone and sentiment to ensure the message comes across as intended, such as being helpful, reassuring, or aligned with your brand voice.

For example, imagine we’re working on a system outage banner. You’ve drafted a message intended to keep users informed about which features are affected, when the service will be restored, and how to obtain help. While the team begins designing the notification UI, you decide to test the copy for comprehension.

We can set up a five-second test where we briefly show the message to users, then ask them to elaborate, in their own words, what the message is communicating. We can then code their responses (e.g., correct, partially correct, or incorrect) to see which parts of the message may need editing for clarity before later implementing it into a prototype for further in-context testing.

The bottom line

Testing UI elements in isolation can be an effective precursor to contextual, task-based research. Early in the design process, isolated tests can be used to uncover issues like icons with recognition problems, unclear copy, or poorly organized menus before integrating these elements into full task flows, and later in-context testing to ensure holistic usability. When used selectively, this two-step process can be an efficient use of time and resources and ultimately produce better product experiences.

You've worked hard to schedule research sessions, and the big week has arrived. Facing back-to-back, hour-long sessions across several days, it's easy to get overwhelmed by the sheer volume of data. What's important? What isn't? What constitutes a meaningful theme? And how will you organize everything into a coherent, insightful report by next week?

Poorly taken notes—or even excessively detailed ones—can leave you struggling to craft a compelling report for your stakeholders. What you need are effective, efficient note-taking strategies. Entering research sessions with a clear approach to collecting and organizing information helps you quickly identify emerging findings and themes, accelerates your synthesis process, and keeps stakeholders informed and engaged throughout.

This article covers essential note-taking tips and tricks for UX Researchers, whether you're new to the field or a seasoned professional refining your methods. It combines time-tested best practices with contemporary approaches based on practical experience.

Choosing the right tool for note-taking

Let's take a look at the most common tools used for note-taking during research sessions, and why the humble spreadsheet is our preferred option.

First, the original: pen and paper—it can work in a pinch and is still the go-to when you're taking your own notes during a session. It's helpful as a prop that demonstrates to participants that you're engaged in active listening. However, it requires a lot of manual transposition of handwritten notes into a format that can later be manipulated during analysis.

Workshop and facilitation tools like Miro or Figma are another option we avoid, although they're often popular with designers and other UX professionals. The main reason being: notes are seldom the final deliverable, and further synthesis and manipulation of the data will need to take place somewhere. Spreadsheets naturally facilitate this, and the outputs of synthesis can later be translated back into Miro if desired.

Finally, we generally prefer not to use note-taking tools completely integrated into your UX Research SaaS platform for the simple reason that these have low transportability over time. Unfortunately, valuable insights will get lost as SaaS companies get acquired, sunset features, raise their prices (prompting your organization to switch), or do any number of things that mean you lose access to the data. Even when these notes can be exported, you lose the richness which was the selling point for maintaining them inside the platform in the first place (such as labels or direct connections with video clips). Better to use a spreadsheet which can then be integrated into any repository tool that you use.

So how does one actually take notes in Excel? One tried and true approach is the rainbow spreadsheet method, popularized by the late Tomer Sharon: list your discussion guide questions down rows, and assign each participant a column. Within each cell, record data such as observed behaviors or relevant quotes. While the classical rainbow method uses colors as data, we prefer numbers or simple binary indicators (1/0) which can be used in calculations. This creates a matrix that allows you to quickly spot patterns horizontally (by question) and vertically (by participant).

Get a copy of the sample spreadsheet

Let the discussion guide drive the order for note-taking

Some have pointed out a distinction between chronological and topical note-taking—organizing notes by when they occur during the session, or by high-level topic and its relevance to research questions. Both approaches have merit, but neither is ideal.

Instead, start with a logically-flowing discussion guide that’s at least semi-structured around the types of questions you're going to ask and in what order. This makes it easier to organize your notes, while making the session more logically coherent for the participant.

Though rainbow spreadsheets are a natural fit for this, you’ll have to start with a strong discussion guide. That’s the topic of another article, but one classic and effective way to go about this: starting with the research questions and using a grid to make sure that your discussion guide is addressing each of the high-level research questions from multiple angles.

Selecting the appropriate tool forms the foundation of effective note-taking—but equally important is determining who will use them during research sessions

Ideal setup: facilitator with dedicated note-taker

Wherever possible, the ideal is to have a dedicated note-taker supporting a facilitator in real time.

Aside from serving many practical advantages (like offering technical support as needed), dedicated note-takers create an inherent efficiency by making sure notes are documented as soon as the session is over. This facilitates speed to insight and analysis time because otherwise, a solo researcher will need to take minimalistic notes during the session and then go back and watch the recorded session (effectively doubling analysis time).

Further, a dedicated note-taker is a second set of eyes with different perspectives, interpretations, and expectations, which helps to reduce bias and increase the reliability and validity of findings. If two independent individuals reach similar conclusions, it's more likely to be an accurate representation of reality than if one person comes to the conclusion by themselves. Confusing moments that might leave a solo researcher stuck re-watching a video later on instead get captured in the moment by a note-taker.

Of course, this isn't always practical or possible for every team or for every research project. If you’re a solo researcher or freelancer coming from the outside, this could be a great opportunity to take a first dip of the toe into the “pool” of democratization. First, identify an engaged stakeholder or colleague, such as a designer or product manager. Give them a high-level briefing on what the note-taker duty requires (perhaps by sending them this article), and—with some training—they can do this for you. This is a very low-stakes and high-reward way to further scale the research resources on your team. On the other hand, if you’re already on a team with multiple research resources, work it into your process that each project will have a dedicated note-taker.

While having a dedicated note-taker creates the ideal setup for capturing comprehensive research data, the question remains: exactly what, and how much, should be documented during a session?

Goldilocks note-taking: neither too thorough nor too sparse

Your goal as a note-taker is not to thoroughly document every single behavior or verbalization observed or heard during the session. That’s what a recording is for.

Consequently, a best practice is to always get the participant's consent to make a recording for internal purposes that will not be shared publicly. This way, you can refer back to it for anything that might have been missed. Even for in-person sessions, a proper set-up can broadcast to a virtual meeting for remote observers and recording services.

Regardless, when starting out as a note-taker, err on the side of documenting more rather than less. You'll gradually get a feel for what’s important over time. One key lens for this is going back to the research questions and thinking about the ultimate output of these insights—i.e., the report or deliverable that will be used by stakeholders to make decisions.

Some of the most common categories of data recorded in notes include: quotes, behaviors, keywords, patterns, concerns, frustrations, surprises, positive moments, gaps in knowledge, missing information, body language, emotional tone, or reactions.

If circumstances require you to take cursory notes (e.g., a solo researcher without a note-taker), marginalia or labeling may be helpful. Developing some sort of system where a color highlight or symbol in the margins has a specific meaning can act as a shorthand, making quickly jotted ideas more thorough. For instance: “!” or yellow highlighting could indicate a “Wow” moment, “☹” or red highlighting could indicate a frustration, “Q” or green highlighting could indicate a great participant quote.

Bear in mind also that note-takers can and should paraphrase verbatim quotes. No report needs to include the "ums" and "uhs" and other verbal pauses that naturally arise as people articulate their thoughts.

Finding the right balance in manual note-taking is essential, but technology offers additional layers of support that can complement your note-taking strategy.

Record and transcribe, but don't rely on AI note-takers

Most modern video conferencing tools such as Zoom, Google Meet, and WebEx now offer AI-powered transcription and summarization tools, which further help to document everything that is stated. The key benefit is that these generate searchable, timestamped transcripts, freeing note-takers to focus on meaningful highlights or key words instead of transcribing quotes by hand.

Another advantage of AI transcription and note-taking tools is the ability to tag and highlight moments in the session in real-time as they happen. For instance, Fathom1 has a bookmark feature that a facilitator may use during a session to note a key quote that should be used as a highlight reel clip in the final deliverable.

Clips from recordings are a powerful way to add ethos, or emotional engagement, to your final deliverables. These may be anonymized with post-processing tools, and if it doesn't make sense to make a video, you can use tools like Descript to pull out just the audio and create an engaging shareable.

Nevertheless, the AI-generated summary these tools provide should not be considered an alternative to enlisting a dedicated note-taker, since these tools work by collapsing or mechanically shortening a larger text rather than organizing the information and synthesizing it according to a structure like the spreadsheet that we've been discussing. We expect these tools to get better over time, but perhaps a more durable limitation will be their ability to note behaviors and other non-verbal cues.

While recordings and transcripts provide a comprehensive record of what happened during research sessions, they don't inherently reveal what matters most to your project. Rather, this must emerge from active interpretation of findings.

Front-load UX research synthesis as you go

You’ll need to synthesize notes into insights for stakeholders—and getting a jump-start on the process can be augmented with two strategies.

The first is summarizing the high-level themes right after each session, which can be done individually—quickly jotting down what stood out as the most important insights or revelations relevant to the research questions directly after the session—or collaboratively, in a huddle between the facilitator and dedicated note-taker.

The second (which may be combined with the first), is a debrief, which occurs with stakeholders, clients, and/or other non-researchers, either after each session or on a daily/weekly basis. This helps process observations and findings with the team that will be using the information as quickly as possible, keeping it fresh. When you learn what was surprising or interesting to a stakeholder, it may be completely different from what was interesting or surprising to you. This valuable signal gives you an idea of what to listen for and what to focus on in whatever sessions you have left.

Both these ideas will front-load synthesis and help you to expedite the final deliverable writing since you’ll know the key answers to your research questions as they happen.

The bottom line

Effective note-taking is about capturing what matters, in a format that accelerates insight discovery. With a structured approach—whether choosing the right tools to front-loading your synthesis—you'll transform the often overwhelming process of mixed-methods research into a streamlined workflow that delivers value faster.

Your team’s system should evolve with your needs. Experiment with different aspects of these techniques, assess what works in your specific context, and refine over time. Whether you're working solo or on a larger team, the ultimate goal remains consistent: less time managing notes and more time generating the insights behind meaningful design decisions.

There is a lot of discussion about AI and its place in UX right now — in terms of both how we work (i.e., integrating AI into workflows and processes) and what we work on (i.e., how can we craft experiences that meet real user needs as companies put pressure on product teams to ship AI features quickly).

This leaves many wondering if AI is a passing trend or a shift in the expectations for their roles. We analyzed 2,983 user research job descriptions to examine how often AI is mentioned, how that has changed over time, and the contexts in which it appears.

This article summarizes our findings and what they signal about artificial intelligence’s place in the industry.

Analyzing 2,983 user research job descriptions for AI mentions

For this analysis, we used a sample of 2,983 user research job descriptions collected between January 2024 and April 2025. Job postings are a great source of data to address questions about role expectations because they are publicly available in large enough quantities to support longitudinal analyses and include skill and qualification requirements.

We flagged the job descriptions and their metadata for any mention of AI across several related terms (e.g., AI, artificial intelligence, generative AI, ChatGPT, Copilot, etc.) and analyzed the frequency of mentions over time. Additionally, we performed thematic analyses, coding where and why it appeared in the job posting.

AI in user research job descriptions: What the data shows

How often AI appears in user research job descriptions

Starting at the broadest level, we can look at how frequently AI comes up in user research job postings. Our data show that out of the 2,983 job posts sampled, 334 (11.2% of the total) mentioned AI for one reason or another.

So, most user research job posts don’t mention AI, but about 1-in-9 do, which is a fair number when you consider the overall volume of jobs across the entire industry. We should also note that job descriptions are notorious for inconsistent formatting and excluding key details, so we should treat this 11.2% as a rough directional measure.

What’s probably more noteworthy than the overall number is how it’s changing over time. When comparing the data from 2024 to what we have collected so far in 2025, we see an increase from 9.5% to 16.4% — that roughly 7% YoY increase is statistically significant (z = -5.07, p < .0001).

It probably doesn’t surprise many that this change shows up in the data. Anecdotally, in conversations with researchers across many types of companies and industries, we hear that most organizations are scrambling to find applications for new AI technologies and asking their product teams to deliver AI-driven features.

How user research job listings reference AI

Taking the 334 job postings that mentioned AI, we performed thematic coding and found three primary reasons why AI was referenced:

1. The role would involve research to shape AI products or features (39.8%)

2. AI was a part of the required experience or skills for the role (40.1%)

3. The posting included disclosures about AI usage within the hiring and interview process (24.3%)

Each of those deserves a closer look.

Of the job postings that mention AI, nearly 40% do so because they explicitly state that the role will focus on research in support of developing AI features and experiences. For example, one posting said, "You will play a crucial role in shaping the user experience of our artificial intelligence-enabled assistant and digital products." This is probably unsurprising, given the broader focus on AI across the industry. Many researchers report that their teams have been tasked with delivering AI-driven experiences.

Similarly, roughly 40% of the job postings that reference AI list it within required experience or skills; diving into that reveals some interesting nuance.

• 65% of the time, the posting asks for prior experience designing AI products or features. For example, it’s relatively common to see something like, "3+ years of experience in user research, preferably with a focus on AI and chatbot technology."

• 25% of the time, the posting was asking for skills or expertise using AI as a part of the user research process. What’s particularly noteworthy is that these requirements were often described in relatively broad or generic terms.
  For instance,

  • "Knowledge of artificial intelligence and how it can be applied to UX practices," or

  • "Exposure to AI as part of product development, executing research, and/or increasing personal productivity."

• An additional 2% of postings referenced both, seeking candidates with experience designing AI products as well as using AI in their research processes.

These data have some important implications. First, we can see that when a job asks for prior experience with AI, it's much more commonly about creating AI products, experiences, and features. It could be a strong differentiator for a job seeker to highlight prior experience working on these types of digital experiences in their resume or portfolio. Second, it was far less common to see a job description mention skills applying AI to research work and processes. Where it was mentioned, it tended to be described at a relatively high level, which reflects broader trends: AI adoption today is largely driven by individuals, and most applications are relatively simple in scope.

We were surprised to find that 24% of the postings that mentioned AI did so in a disclosure about the hiring or interview process.

Some companies have started adding a blanket statement to all of their job posts, like this example: “Please note that [redacted company] may leverage artificial intelligence and machine learning technologies in connection with applications for employment.“

On the other hand, some postings mention a new issue in hiring: candidates using ChatGPT during live interviews to generate answers on the spot and reading them verbatim. For example, one company stated, “We are committed to ensuring a fair and equitable interview process for all candidates. As part of this commitment, the use of artificial intelligence (AI) tools to generate or assist with responses during interviews is not permitted. This policy is in place to maintain the integrity and authenticity of the interview process."

Commentary on how AI is changing the nature of hiring and job-seeking is somewhat outside the scope of this particular analysis, but it may be worth revisiting in the future. In the meantime, both candidates and hiring managers should be broadly aware of these emerging implications for the current hiring environment.

The bottom line

Mentions of AI in user research job postings are becoming more common, increasing from 9.5% in 2024 to 16.4% in 2025 (YTD). This shift reflects current organizational priorities to ship AI features.

Most often, AI is cited because the role will involve researching AI features and/or because it requires prior experience performing research on AI products. Mentions of using AI for accomplishing user research tasks are less common and tend to be described in broad, non-specific terms.

In light of these data, user researchers should consider:

1. Gaining experience working on AI-focused products and highlighting them in their resumes or portfolios.

2. Experimenting with AI tools (within reason) to improve their personal workflows and efficiency, even if expectations about AI-assisted research are loosely defined.

Portfolios and case studies have become a central artifact in the UX Research job search—yet they’re not something we’re trained to produce, nor are they part of our regular day-to-day responsibilities. They become urgent only when we’re seeking our next opportunity. And in today’s volatile and competitive job market, where many researchers have been impacted by layoffs, the stakes feel especially high.

Researchers are doing their best to stand out, but often without clear guidance—especially when they’re not landing interviews and receiving little or no feedback from hiring managers. When feedback does come, it typically comes from colleagues or mentors: well-meaning, but sometimes contradictory.

This isn’t just a problem for candidates. It’s a problem for hiring teams too. In past research with hiring managers and candidates, we found that work sample reviews are often conducted with no consistent structure or shared standards—leading to subjective, inconsistent evaluations.

This article aims to fill that gap. What follows is a flexible rubric template that UX Researchers can use to self-evaluate their portfolios—and that hiring teams can adapt to bring more structure, fairness, and clarity to their review process.

Where this rubric came from

The rubric draws on three primary sources:

1. Research on the UX Research hiring cycle conducted by Drill Bit Labs, including interviews with hiring managers and candidates about portfolio practices, pain points, and what makes a portfolio stand out.

2. Direct experience on hiring teams, reviewing candidates’ work samples and seeing firsthand how difficult it is to evaluate portfolios without a shared framework.

3. Portfolio review experience, drawn from dozens of sessions with mentees, during which common themes and repeated advice began to emerge.

To refine and validate an early version of this rubric, we conducted five 45-minute portfolio review sessions with UX Researchers at various career stages—from entry-level to staff and principal roles. Feedback from those sessions helped shape the dimensions, language, and scoring guidelines.

How the rubric works (and how to use it well)

What we’ve produced is an evaluation rubric designed to be useful across the full spectrum of seniority, offering clear standards whether you’re just starting out or bringing years of experience to the table.

It assesses portfolios across five key dimensions: Clarity, Rigor, Impact, Engagement, and Growth. Each dimension is rated on a scale from 1 (Poor) to 5 (Exceptional). The rubric includes detailed criteria for each level, making it easier for candidates to self-assess. There’s also space for comments to note suggested changes and areas for development.

The scoring and letter grades are non-standard, so you might be surprised to see a 72 of 100 (for example) rated as a B. This is intentional. Consistent performance at 4 (Strong) across all five dimensions yields a 90, which gives you a solid portfolio for casting a wide net, and getting invited to a phone screen. A few additional ratings of 5 (Exceptional) will push the score into A+ or bonus territory, which indicates a strong potential fit for later-round interviews when you are presenting your work.

Use of this rubric is based on two key assumptions:

1. This may seem obvious, but you’ll need a portfolio—or at least a few case studies in progress—to evaluate.

2. You should be pursuing a targeted job search. For example, you might be focusing on Senior or Lead roles at small to mid-size companies, rather than applying indiscriminately to any opening with “Researcher” in the title.

The five dimensions that make or break a portfolio

To assess yourself or someone else’s work, you first need to understand the five dimensions.

1. Clarity

Definition: How well the portfolio communicates the problem, context, and relevance to the targeted role.

Strong clarity means the work is well-scoped, the goals are articulated, and the hiring team can immediately see how the work connects to the job. Weak clarity often shows up in generic case studies with vague goals or only superficial alignment with job requirements.

For example, one candidate was applying for Staff-level roles, which often emphasize non-technical competencies like mentoring and influencing strategy. However, all of the case studies presented were focused on technical execution. A key opportunity was to highlight leadership, influence, and mentorship activities that were more relevant to those target roles.

2. Rigor

Definition: The appropriateness and depth of the presented activities—often, but not exclusively, research methods—and the reasoning behind them.

Strong rigor goes beyond listing what you did. It involves explaining why those activities or methods were chosen, how they were executed, and/or how they addressed the project goals. The best examples convey this even in standalone form, without needing a candidate’s voiceover to fill in the gaps.

Above, this candidate’s portfolio took a beat to explain the chosen method (in this case, a special application of task analysis) and its specific strengths in relation to the questions being explored. This kind of explicit reasoning sets the candidate up as an expert practitioner.

3. Impact

Definition: How well the portfolio connects activities or research insights to business or user outcomes.

Impact may take the form of product decisions, strategic direction, user value, or even return to the business.

This can be a tricky one to demonstrate, since we often have little control over how our findings and recommendations ultimately get used. Projects might get shelved, the team might pivot, or the fruits of an early discovery effort may not materialize for years. Even so, if you drill deeper, you may find meaningful signals of impact: perhaps your work influenced a future line of research, led to a process improvement, or helped shape internal conversations about product direction.

This example shows how one candidate was able to artfully show impact. Although unable to demonstrate tangible impact from the findings of the study itself, the case study is nevertheless careful to show how the work was grounded in business outcomes (in this case, revenue and market share) from the earliest stages of planning. Under less than ideal circumstances, this indirect approach can be effective.

4. Engagement

Definition: The overall polish and flow of the portfolio, i.e., how well it communicates effectively and holds the viewer’s attention.

This isn’t about visual design, per se—after all, we’re researchers, not designers. But a good portfolio should avoid distracting the viewer with generic or sloppy templates, or an incoherent layout. It should communicate in a way that feels purposeful and easy to follow. This includes effective visualizations and figures, logical flow, and scannability.

One common question about the rubric is how to balance detail with brevity. How do you include enough information to score well in Clarity, Rigor, and Impact while keeping your portfolio concise? Balancing these with Engagement can take many forms. For instance, well-structured portfolios might include an executive summary, clear sections and headers, or even a hyperlinked table of contents to let viewers dig in wherever they choose.

The example above is representative of how many portfolios struggle to toe the line between comprehensiveness and brevity. The minimal detail makes it hard to follow how key insights emerged from research activities. It also assumes a level of context and knowledge that non-research stakeholders may not have.

5. Growth

Definition: Evidence of reflection, learning, and evolution across projects.

Growth is often the most underrepresented dimension—which makes sense. Reflecting on “failure” can be uncomfortable. But in reality, things rarely go exactly as planned. Strong portfolios show what the researcher took away from those bumps in the road, how they adapted, and how it shaped future decisions.

During our portfolio validation sessions, one candidate shared their experience working in a new domain: healthcare, which is a regulated environment shaped by U.S. HIPAA laws. They reflected on the challenges of participant recruitment and research planning in such a context, showing thoughtful adaptation and professional growth.

How candidates and hiring teams can use this rubric

For candidates, start with a representative or target job description. This could be a real role you’re preparing for, a position you applied to and felt well-suited to, or one related to a panel presentation you’ve been asked to give. Some candidates mentioned leveraging AI tools here: one can gather three to five job descriptions that reflect your target role and ask an LLM to synthesize them into a composite.

From there, this tool is most powerful when used in conversation. We all suffer from the Curse of Knowledge—we’re so close to our own work that we forget what’s obvious to us might not be obvious to someone else. So wherever possible, get out of your head and review your portfolio alongside a mentor or trusted colleague. Share your materials along with this rubric and article, have each person assess independently, and then compare notes. Use the comments column to flag areas for improvement and identify concrete action items. Iterate and revise accordingly to strengthen your application materials.

For hiring teams, one of the biggest issues we identified in our research on the UXR hiring cycle was a lack of structure. Too often, panel presentations are evaluated based on gut feel or first impressions. A more reliable and equitable approach is to systematically assess candidates based on the specific responsibilities of the role.

While this rubric is intentionally broad to accommodate a variety of UXR roles, think of it as a template that can and should be adapted for your team’s needs. Consider customizing the dimensions or criteria to align with your org’s expectations of a specific role. Further, we suggest sharing what you’re looking for, at least in broad terms, with the candidate in advance. Even a general signal of what you’re looking for helps them tailor their presentation and helps you assess more effectively.

The bottom line

Drawing upon original research, hiring experience, and dozens of portfolio reviews, we created this rubric as a resource for candidates and a template for hiring teams. It provides structured criteria to evaluate portfolios across five key dimensions: Clarity, Rigor, Impact, Engagement, and Growth.

Interested in using it? Get your copy of the rubric in Google Drive. It’s free to use, but if you adapt or share it, please credit this source.

Special thanks to our portfolio review session participants, whose experiences, comments, and questions have improved this resource.

When they have little to no past research, a repository can seem unnecessary. On the other hand, teams with a large archive of past research face a steep implementation challenge in structuring everything into a useful repository.

This creates the feeling of a paradox: it never feels like the right time to build a repository. But the truth is that the amount of past research you have shouldn’t dictate whether or not you set up a repository. Whether you have a little research or a lot, you can still reap the benefits of a repository. The key is adjusting your approach to fit your starting point.

Why teams put off establishing a research repository

In theory, the more past research insights a repository holds, the more valuable it becomes. This creates friction when deciding when to implement one. A team that just completed its first user research study could establish a simple tagging and storage system fairly easily, but they would see little immediate value. Conversely, a large company with years of past studies could benefit immensely from organizing, tagging, and making them searchable, but the investment of time, money, and sheer effort required can feel overwhelming.

We call this dilemma the Repository Paradox. On one end of the spectrum, teams may choose to wait until they have more research before setting up a repository. On the other end, the perceived hurdle of organizing years of studies can make them hesitate to start at all. This leads to the insights management equivalent of technical debt. Other challenges, like the need to add new tools and a focus on generating new research to address incoming questions, can make the idea of dedicating resources to implementation even more daunting. However, the longer a team delays, the harder it becomes to get started.

When should a UX team set up a research repository?

The truth is that it’s never too late, or too early, to start a research repository. Regardless of whether you have many or few past studies, building an insights repository creates a single place where anyone can quickly search for and access findings. It helps prevent duplicated research, increases the discoverability of insights, and extends their shelf life. Delaying the decision to build a repository only postpones these benefits and makes implementation more challenging down the line.

The Repository Paradox isn’t a problem of optimization—where at some magical point, the value and effort curves intersect to justify the investment. Instead, the real question is whether your team faces certain challenges that a repository can solve. If you can answer "yes" to these three questions, then it’s the right time to start a repository:

1. Do stakeholders frequently ask about users, their behaviors, or attitudes?

2. Is your research data scattered or not centralized in a single location?

3. Do you want to democratize access to user insights across your organization?

If the answer to all three is yes, it’s time to start—whether you have 10, 100, or 1,000 studies in your archive. When building a repository, there are many considerations, such as:

• The type of software you’ll use—whether a purpose-built UX research tool or a more general database.

• The repository format—whether a document library or an atomic insights database.

• The process and steps involved in structuring and maintaining it.

These are worthy topics but outside the scope of this article. Instead, we’ll focus on how to approach building a repository based on where your research practice stands today—whether you’re starting fresh with only a few studies or working with a vast archive of past research.

What to do if you have little to no past research

It can be easy for a young research practice to procrastinate on building an insights repository. Despite the temptation to wait, if your team meets the key criteria outlined earlier (i.e., frequent stakeholder questions, scattered research, and a desire to democratize insights), now is the time to start organizing and managing your insights. Instead of thinking about a repository as something that only provides value once it has a critical mass of information, think of it as a habit of organizing your work and an infrastructure that will scale alongside your research team.

When starting an insight repository at this stage, the key is to set yourself up for success and scale without over-complicating things:

1. Identify the minimum viable repository: A repository doesn’t need to be highly structured with a complex tagging system from day one. Over-engineering your repository and its organization places unnecessary friction and pressure, making setup much more overwhelming to get started than it should be. Ask yourself: “What is the simplest possible solution that would be useful today?” This could be as simple as ensuring that all of your insights are in a single, searchable location with a simple set of broad tags or basic organization.

2. Avoid the urge to go high-maintenance: Nielsen Norman Group's research shows that one of the key reasons why repositories fail is that researchers find it takes too much work to contribute to them. So when building your repository early on, ask yourself how you can reduce the friction for researchers who are adding their studies. Decide on the bare minimum information that makes an entry useful, while keeping the effort relatively low. Additionally, consider how adding something to the repository can be incorporated into existing processes, such as tying it to project retrospectives or other regular workflows.

3. Use a flexible schema: The way you organize your repository today doesn’t need to be perfect, but it does need to be flexible enough to accommodate the future. Your repository should scale with your research team. Failing to build in flexibility now can lead to a situation down the road where adapting it is so difficult that you have to rebuild from scratch, effectively undoing the investment made in the beginning. Ask yourself: “How can our work be sorted today?”, “How might that change in the future?” and, most importantly, “How can we build things now to accommodate those future needs?” Your team's needs will evolve, but you might not know exactly how, so your repository should be structured to adapt accordingly. Starting with broad categories and a flexible tagging system today allows you to grow and refine the structure over time.

By approaching building a repository this way, even a team with little to no existing research can quickly build an insights repository that provides immediate value and has room to evolve as their research team and projects change.

What to do if you have a lot of past research

With a huge backlog of insights, building a repository can feel like an insurmountable effort. In the face of such an effort, the easy temptation is to put things off and wait until your team has the capacity to handle a project of that scale. In practice, that moment rarely comes.

Again, research teams in this situation should build a repository if they regularly receive questions from stakeholders about users, have disorganized past findings, and desire to democratize access to those insights. However, it is wise to scope things down, turning the rollout into a more manageable project focused on the highest-importance items rather than attempting to restructure all past projects completely.

1. Make it smaller with a gradual rollout: Focus your initial repository rollout on the highest-impact materials. That could mean prioritizing past research by specific methods, projects, products, or document types. Additionally, you might decide to only include research from a certain timeframe—for example, the past 18 months. A thorough review of the research projects that have been most frequently referenced will help you identify the most critical studies for decision-making within your organization. You could even create a scoring system to prioritize the most valuable projects. By the end of this process, you might produce something like this example from Marianne Carpentier, to illustrate what is and isn’t in scope for the initial rollout.

   

2. Consider how automation can assist you: Some tools offer automated tagging, which can further reduce the manual work involved in organizing and categorizing past research. Consider how these tools might help, and review options to see which are the most appropriate for your team. Automated tagging will not be perfect, but it will help you produce something useful.

3. Pilot and build as you go: After completing your initial rollout, treat the next few months as a pilot period. Use this phase to assess the scope and effectiveness of your repository. You might track requests for materials or information that aren’t in the repository yet and create an updated list of things to add. Focus on adding new research and insights within the scope of your repository. After assessing if your team has more bandwidth, you can consider adding second-order priorities.

Starting a repository when you already have a significant archive might seem daunting, but you can manage the challenge by taking things in stages. Focus on adding your highest trafficked and most impactful work first, making use of automation where appropriate, then expand on your pilot as demand reveals itself.

The bottom line

Research repositories are essential for making UX insights accessible across an organization, but teams often delay setting them up. A team with very little past research may not see immediate value in building a repository; on the other hand, a large company with years of past studies may see immense potential in organizing its information, but the scope of that effort can feel daunting. This often leads to the Repository Paradox—a feeling that it’s never the right time to roll out an insights repository.

The truth is, as long as you have frequent questions from stakeholders about users, the desire to democratize access to insights, and a current system that is either disorganized or not centralized, you can and should start building your repository. This applies equally to teams at all stages of their practice:

• Teams with little to no past research can begin with a minimum viable repository designed for scalability, using a flexible schema that grows with their team and ensuring that the process of adding new insights remains simple and low-maintenance.

• Teams with a large archive of past projects should start by narrowing the scope of their repository rollout, prioritizing the highest-impact and most frequently referenced studies for the initial release. Afterward, they should pilot the repository for a few months to assess its effectiveness before expanding its scope.

This article was originally published on the Great Question blog: Link to post

Inspection methods are a powerful and inexpensive set of tools that any UX team can add to its research toolkit. Many researchers are familiar with at least one inspection method, and most UX teams report conducting them, even if only rarely.

Yet there are a number of points on which professionals might be confused or disagree. What exactly is an inspection method? When should you use one, and why? What are the specific methods to actually do them? And what place do they have in our arsenal of research methods today?

Taking a cue from a classic paper about one specific inspection method, let’s unpack the story of inspection methods — past, present, and future — in five questions.

1. What are inspection methods?

Let’s begin by defining terms. You may know the group of inspection methods by other names, including analytic methods, or expert reviews (although, some contend an expert review is a specific instance of inspection method).

These methods serve as a contrast to empirical or observational methods, like usability testing, which recruit representative users as participants for study. Rather than observe user behavior, inspection methods primarily involve the judgments of evaluators who review the product in a systematic way. These reviews are based on established principles of behavior and design, and the experiences of the reviewers who, having experience with observational studies, can make deductions from the steps of the task itself and anticipate some of the common issues to be encountered.

The reviewers’ expertise is typically in the domain of user experience and related fields. Having other expertise can be helpful as well, such as the domain of the experience under review. For instance, a health-tech startup reviewing products for doctors might involve a healthcare practitioner. Even non-experts — often, users themselves — have been employed from time to time, though typically as a secondary evaluator.

Inspection methods generally produce a similar output: a set of findings about the experience reviewed, including usability issues uncovered, potential strengths, and recommendations for improvement. For this reason, multiple evaluators are always helpful to get more exposure to more potential issues and to strengthen the validity of the output, provided that they work independently and only come together afterwards to reconcile findings.

2. Why do inspection methods?

Inspection methods are particularly useful whenever traditional observational methods with users or test participants are prohibitively expensive. This may be due to the logistics of recruitment, the cost of incentivizing participants, the timeline required, or internal obstacles (such as process or regulatory compliance).

Since you don't have to go through the recruitment process — which involves developing a screener, administering it, and identifying candidates — you can simply access the system and conduct the evaluation within your team, greatly improving your speed to insights. As long as the evaluators are trained in these methods and have some familiarity with them, they can be completed quickly.

In short, in situations where empirical methods might otherwise leave you unable to provide stakeholders with needed information for decision-making — perhaps due to an urgent deadline, or unforeseen logistical obstacles emerging during a traditional usability study — inspection methods make for a great supplement or a viable alternative.

3. When might you use inspection methods?

Although inspection methods may be used at virtually any point in the design process, they are typically used early in the process, during the pre-development phase of designs. You get the most “bang for the buck” when you find issues to fix ahead of launch, perhaps even before it’s possible to get the experience in the hands of potential users.

Another common use case is to employ inspection methods prior to an observational study, like a usability test. Since evaluators can often identify some of the most significant usability issues through inspection, those improvements can be made ahead of a test with representative users — empowering your team to focus on deeper and less common issues.

Further, I suggest using them both early and often. When inspection methods are used frequently in an organization, the benefits compound.

Research teams will find efficiencies with a method simply by virtue of its familiarity. Regular exposure to inspection methods educates stakeholders on what to expect. In contrast, if a team conducts a heuristic evaluation a few times a year, there will be some cost of inertia from switching from typical observational methods and re-familiarizing with the process. Creating a culture comfortable with inspection methods can yield significant benefits.

4. How do you conduct an inspection method?

There are a number of options, including: guideline review, heuristic evaluation, task analysis, and cognitive walkthrough, among others.

Guideline review

Organizations develop guidelines for their own design standards, which evaluators can use to review a system for compliance. These detailed reviews were most common in the days before heuristic evaluation (see below), although there are more generally-applicable guidelines in contemporary use, including MeasuringU’s Calibrated Evaluator’s Guide, the (archived) Usability.gov Guidelines, or Baymard’s proprietary guidelines.

How it’s done: Practitioners take a set of guidelines and go sequentially and thoroughly through a flow on the experience. Each guideline is reviewed for applicability, and reviewers note violations.

Although less common than heuristic evaluation today, teams that want a more exhaustive review tailored to their experience’s domain should consider them. This level of relevant detail can potentially help to uncover more issues and offer more specific recommendations than other inspection methods. However, guideline reviews are perhaps the most time-consuming inspection method — a cost that should be weighed against alternatives.

Heuristic evaluation

An original set of heuristics was drafted in 1989 by Jakob Nielsen and Rolf Molich based on their own judgments about usability at the time. But they were later refined based upon a statistical analysis of 249 usability problems, using a total of 101 competing heuristics or guidelines to “explain” each issue. This data-driven approach yielded a final set of 10 heuristics in 1994, which remain in usage today. They have since remained common, and have had an enduring appeal.

How it’s done: Multiple evaluators review each step in a task using the experience being inspected using the 10 heuristics developed by Jakob Nielsen. Reviewers note any violations of the heuristics, give a severity rating, and may also provide potential recommendations for improvement.

Part of the enduring appeal of heuristic evaluation is its simplicity. Practitioners with some familiarity with usability testing and principles may be quickly trained and ramped up on the method, the 10 heuristics, rating severity, and reconciling findings. The amount of issues then uncovered by heuristic evaluation can generate financially-valuable usability improvements in a very short time (and thus, at very low cost in labor).

Somewhat ironically, the heuristics have been adapted for use in specific domains, including virtual reality, ambient displays, and video games, which offsets its potential disadvantage relative to guideline review.

Nevertheless, there are some downsides to heuristic evaluation. Its popularity and simplicity can lead to potential abuse by evaluators conducting the method more informally and without the necessary rigor. Further, the validity and reliability of heuristic evaluations have been extensively studied — and even under the best of circumstances, multiple studies note that heuristic evaluations typically identify only 30–50% of the issues an observational method might uncover. Additionally, about 34% of issues identified are “false alarms” that few if any actual users encounter.

Task analysis

A task analysis is a family of methods that take a set of actions with a clear start and end point, define it, and break it down into the discrete elements (such as sub-goals or specific steps) needed to achieve a desired outcome. Since we have written extensively about task analysis in UX work, we will focus on one instance of task analysis: PURE (Pragmatic Usability Rating by Experts), developed by Christian Rohrer and Jeff Sauro, which quantifies the friction present in a task.

How it’s done: A first step in PURE is decomposing the task into its discrete steps along the ideal or “happy path.” Evaluators independently review each step for issues or friction, and assign the step a severity rating on a color-coded scale from 1 to 3. An overall score is summed across the steps, which then inherits the color of the most severe step. This may be repeated across multiple key tasks in an experience to produce a holistic PURE score, as in the example above.

I think of PURE as the most executive-friendly version of task analysis. Its output creates a visually appealing but also immediately understandable description of the complexity of a task or set of tasks, showing precisely where the areas of greatest friction occur.

Cognitive walkthrough

The cognitive walkthrough was developed in the early 90s by a set of HCI researchers including Clayton Lewis, John Rieman, and Cathleen Wharton, among others. It’s intended to gauge how well a complete novice could learn to use any system they’ve approached without specific training or experience, such as an ATM kiosk or self-checkout machine in a grocery store. Although best suited for these “walk-up-and-use” experiences, it has been applied more broadly in the years since.

How it’s done: A set of evaluators working together in a workshop setting, review each step in a task, asking four key questions:

1. Will users try to achieve the right result?

2. Will users notice that the correct action is available?

3. Will users associate the correct action with the result they're trying to achieve?

4. After the action is performed, will users see that progress has been made toward the goal?

Cognitive walkthroughs are better suited than other inspection methods to assess its specific goal: the learnability of “walk-up-and-use” experiences. Nevertheless, going through four questions for each step in a task can become time-consuming. For this reason, Rick Spencer, a researcher at Microsoft, developed a streamlined version for use in industry settings.

The list goes on

Over the years, a number of books, papers, and blog posts have been written about inspection methods. If you dive deeper into the literature, you will encounter others not mentioned here — some lesser-used, some verging on obsolete, and some variants of the above.

It’s also possible to combine inspection methods in a single project. For example, using both task analysis and heuristic evaluation can serve to make the heuristic evaluation much more thorough. Or, using both cognitive walkthrough and KLM could give both learnability issues and estimates of task completion times. This can enhance the strengths of each method while offsetting disadvantages.

5. Whither inspection methods?

To understand what the future might hold for inspection methods, it may be helpful to examine its past and present. You may have noticed that most of these methods were developed in earlier days of HCI and usability — the 80’s and 90’s — when conducting observational methods was labor-intensive and expensive, before modern forms of automated participant recruitment, and most often in a dedicated laboratory setting. This is no coincidence. Organizations needed to know where the usability issues were, and inspection methods are an attractive, low-cost tool to find them.

At one point, heuristic evaluation was the second most commonly used method after usability testing. In recent years, it appears in only 7% of UX Researcher job descriptions. What happened?

In the years since inspection methods were first developed, the field of UX, including our tools and processes, has changed considerably — and observational methods have become drastically cheaper. We take platforms for remote testing for granted today, but they have made it much easier to get feedback from representative users.

Nevertheless, the field continues to change. In recent years, the tech industry in general and UX in specific has been subject to repeated layoffs and a tight labor market, leaving many teams squeezed for resources. Today’s UX teams might look to inspection methods as a tool to get more from less.

And although the rise of next-generation AI tools, including LLMs like ChatGPT has accelerated aspects of UX research, it also creates concerns about the data quality from remote tests. The old practice of combining inspection and observational methods might today be used to improve both.

Conjoint analysis, while relatively popular in market research, is not particularly common in user research. Our analysis found that only 1% of user research job descriptions included conjoint as a required or preferred skill.

Yet, there is clear interest in the method. In a recent LinkedIn poll of UX professionals—admittedly, non-scientific—over 50% said they had never used conjoint analysis but would like to learn more about it. This makes sense, as market researchers and UXRs often seek to answer similar questions like: “Which product improvements would have the greatest impact on satisfaction?“ or “Which features matter most to our users?“

Despite this interest, many researchers don’t know when to use conjoint analysis. This article will help by outlining three specific scenarios where the method can be highly effective, including prioritizing a set of features, identifying the best combination of features for a single product, and building a suite of products that cover diverse user needs.

Conjoint analyses: A quick Primer

Conjoint analyses help answer a foundational yet challenging question in product design: What is the combination of product features that maximizes user satisfaction? Conjoint surveys typically present users with a set of products (usually 2-4) and ask them to indicate their preferred option. Each product consists of attributes and levels:

• Attributes are features or characteristics of a product or interface

• Levels are a specific variant of an attribute

For example, imagine you were purchasing a car – color might be an attribute, while blue would be a level. Below you can see an example of a typical conjoint study from OpinionX.

It’s worth noting that there are many variants of conjoint studies, often broadly referred to as “discrete choice analyses.” But they all generally follow this pattern of presenting the respondent with a set of products or features and asking them to make a discrete choice to indicate their preference.

By presenting several product combinations to an appropriate sample size (typically in the range of n= 150-300 or more), we can analyze how users make tradeoff decisions between features. Through these quantitative analyses, we can rank the importance of attributes, identify user preferences for levels within those attributes, determine the most preferred feature combination, or even estimate preference share to gauge how a product might perform in the market. These insights allow product teams to optimize packages, plans, and pricing. With this foundational understanding, let’s explore three scenarios where conjoint can be useful.

Conjoint analyses: Use cases in user research

1. Identify the best combination of features for a single product

A common use of conjoint analysis is to determine which specific combination of attributes and levels results in the single most preferred product. You might be in this scenario if your stakeholders have questions like, "What combination of features will maximize satisfaction, purchase intent, or subscriptions?"

To answer this, you would define the attributes and levels for your test, listing the key features and their variations. Then, create a conjoint survey where respondents are presented with combinations of the attributes and levels and asked to select their preference.

After collecting responses, you can analyze the data to examine the relative importance of each attribute and the preferences for levels within each attribute. However, you’ll also examine how specific concepts (i.e., a specific combination of attributes/levels) rank relative to each other. For example, a concept report—like the one from OpinionX below—ranks every possible product and assigns each a score.

To illustrate this use case, imagine a tech company is developing a new laptop and wants to maximize consumer appeal. They are considering multiple attributes, including:

• Screen size: 13", 15", 17"

• Processor speed: i5, i7, i9

• Battery life: 8, 12, or 24 hours

• Touchscreen capabilities: Touchscreen or no touchscreen

• Price points: $1,000, $1,500, or $2,000

A conjoint analysis would help determine which combination of these features creates the most desirable laptop for their users. The result might show that a 15-inch laptop with an i7 processor, 12-hour battery life, no touchscreen, and a $1,500 price point is the most preferred combination.

You can use this analysis to determine which product concept is most compelling and maximizes user preferences. However, keep in mind that your “winning” concept may not be a runaway winner; the difference between the top-ranking options might be within a margin of error, meaning either could be a strong choice.

2. Prioritize a set of features

You might find yourself in a situation where you want to assess the relative importance of different features rather than identifying the best combination of attributes and levels for a single product. In those cases, you could use a MaxDiff analysis, a variation of conjoint that helps you rank features (or attributes) by importance, providing a clear sense of user preferences. This type of analysis is useful when you’re answering questions like:

• Which features matter most to our users?

• Which features should we invest in first?

• Are there any features that we could de-prioritize without negatively impacting satisfaction?

A MaxDiff survey presents respondents with a subset of features and asks them to choose which is the most important and which is the least important. Analyzing the results of these responses quantifies the importance of each feature in your test. For example, below, you can see the results from a MeasuringU MaxDiff analysis on the most and least important features for hotels.

Continuing the example from earlier, imagine that the tech company is in the early phases of developing concepts for their new laptop and wants to understand where they should focus their efforts. A MaxDiff study can help them determine which features matter most for their next-generation laptop. They might test:

• Battery life

• Screen size

• Processor speed

• Storage capacity

• Touchscreen functionality

• Weight

The results might show that processor speed, screen size, and battery life are significantly more important to customers than other features, while touchscreen functionality ranks lowest. Based on this insight, the company could prioritize improvements in performance and portability rather than investing in touchscreen technology.

MaxDiff is a relatively simple but powerful discrete choice analysis. It helps you understand the relative importance of features but does not determine their optimal combination. Use MaxDiff when you need a clear, ranked list of feature priorities to align stakeholders quickly. However, if you need to determine the best combination of features for a single product, the ideal price point, or the optimal suite of products for a portfolio, MaxDiff isn’t the right fit.

3. Optimize a set of products to cover more user needs

What if, rather than developing a single product, you wanted to make a product lineup? This is a relatively common scenario — companies often create a suite of offerings that meet the varied needs of different user groups rather than designing just one product. When creating a product portfolio, a team might have questions like:

• How can we develop a set of 2-3 distinct offerings that cover a majority of users’ needs while minimizing overlap?

• How can we design tiered product plans?

The methodological setup for this question is the same as the first use case we reviewed (i.e., define attributes and levels and run a conjoint survey). However, additional analysis methods are required to answer these research questions. The challenge is that you can’t simply select the top i ranked concepts to develop a product portfolio of size n= i. The top-ranking concepts may all appeal to the same type of user, leading to significant preference overlap.

Specialized analysis is needed to select a product portfolio that minimizes this overlap. There are a couple of approaches that accomplish this:

1. Total unduplicated reach and frequency (TURF) - This calculation identifies the concepts that maximize the coverage of user preferences for a given product portfolio size (n= i).

2. Genetic algorithmic optimization - As described by Chapman and Alford in their 2010 conference presentation, another approach is to use genetic algorithms to generate random sets of product portfolios and iterate until the user preference share percentage doesn’t improve.

Both methods help determine the optimal number of products for a portfolio and identify the specific products to include, ensuring broader market coverage and better alignment with diverse user needs.

For example, instead of designing a single laptop, imagine we wanted to create a product line with varying features and price points. We could use TURF or genetic algorithms to determine the optimal catalog of three laptop models. The results might suggest that an ideal portfolio would include:

1. A premium laptop with long battery life, a lightweight design, and a high-end processor

2. A gaming laptop with a high-resolution display, a larger screen, and a powerful graphics processor

3. A budget-friendly laptop with decent battery life, affordability, and portability

These techniques are some of the most advanced applications of conjoint analysis. If you have access to tools with these capabilities built-in, it can be relatively straightforward. If you’re comfortable with elegant quantitative analyses, you might be able to do it manually yourself. However, in many cases, you might need specialized software or the help of a data scientist to run these analyses effectively.

The bottom line

Conjoint analyses aren’t commonly used in UX research, but they have remarkable potential for informing product decisions. This article explored three use cases for where these methods add value:

1. Optimizing the feature set for a single product

2. Prioritizing a set of features by relative importance

3. Designing a suite of products that meet diverse user needs without overlap

Beyond their standalone value, these methods serve as a strong quantitative complement to exploratory methods like interviews or ethnography by validating qualitative findings and measuring user preferences. When used together, these techniques can inform product roadmaps with greater confidence.

With this foundation, UX researchers can recognize when to use these techniques and use the provided reference materials and tools to implement them in their work.

“By the end of that meeting, I was so mad that I literally set my laptop on fire.”

What about that statement stands out to you? If you bristled at “literally” being used to mean its opposite (i.e., figuratively), you might be a prescriptivist.

Language is alive. It evolves through usage. For any given term, there’s the prescriptive sense found in formal definitions, and a descriptive sense that reflects how people use it. The problem is that, in a way, both are correct! For example, “literally” has been used in the figurative sense widely enough that it now has a corresponding secondary definition.

It’s no surprise then that this also applies to the terms we use in UX. Whatever our understanding of words, our stakeholders — or at times, even our colleagues — might have a very different definition. So from time to time, we leave meetings confident we’ve reached consensus, only to later discover we walked away with opposite conclusions. That’s enough to make you want to set your laptop on fire (figuratively, of course).

Miscommunication happens in many ways, but most can be avoided by defining our terms. In this article, we’ll look at some of the most commonly confused words and phrases used in UX research, so you can recognize them, clarify them, and avoid misunderstandings.

A/B Testing

How it’s often used: Any user research study that compares two variants (or options, e.g. A vs. B). This might include comparative usability studies, preference tests with concepts or prototypes, or similar methods. For example…

Researcher: “We recruited 10 participants and did an A/B test of the two prototypes, and found that users preferred B in 4 out of 5 dimensions.”

A more precise understanding: A quantitative test that exposes thousands (or even millions) of users, usually without their knowledge, to different design variants in a live product. The winner is determined based on statistical analysis of concrete behaviors, such as conversion or open rates.

Product and engineering teams will often be familiar with the latter, as they tend to collaborate with analytics and data science teams to develop, run, and analyze A/B tests of this kind. It’s a great way to quickly settle debates on smaller, incremental changes to a live product when a typical research study might not be worth the time and hassle.

On the other hand, smaller sample user research studies are great for finding the strengths and weaknesses of designs at an early stage of development. In fact, a comparative usability study can make a later A/B test more effective by identifying what are the most valuable variants to test, and why.

Survey

How it’s often used: Any form of research or user feedback, whether formal or informal. For example…

Stakeholder: “We came to your team because we know we need a survey.”

A more precise understanding: A set of standardized questions administered to a sample of online research participants, usually using tools like Google Forms, Qualtrics, or Tally. It’s intended to collect a large volume of data in a relatively short time and at a relatively low cost. It can be contrasted with other remote, unmoderated methods, such as unmoderated usability testing. Further, surveys can be the vehicle for sophisticated methodologies such as Kano, conjoint, and Key Drivers Analysis.

When “survey” is used as a catch-all term for research, it may signal a limited understanding of research methodologies. Think about it: even if you don’t know much about research methods, you’ve probably seen, taken, or even designed a survey. But you’ve also likely encountered poorly-designed surveys with leading questions and incomplete response options. Though surveys are both common and powerful tools, designing an effective one can be surprisingly nuanced.

In short, don’t take a stakeholder’s request for a survey at face value. A survey may be the right tool, but there may be a better alternative.

Interview

How it’s often used: Any form of scheduled research session with participants or representative users, such as usability study. For example…

Stakeholder: “What did you learn from those interviews last week?”

Researcher: “Wait, was someone on the team running interviews?”

A more precise understanding: A one-on-one session with a participant, primarily consisting of a structured or semi-structured conversation designed to answer specific research questions. See also: in-depth interviews (IDIs), or user interviews

Like “survey," the term “interview” is often used more broadly by non-researchers than by researchers. Because researchers have a deeper understanding of methodologies, we tend to use more precise terminology. If the session primarily involves interacting with stimuli (i.e., a prototype, website, or app), we would consider it a usability study. However, many non-researchers might be hard-pressed to define the word stimuli, so the catch-all term “interview” seems appropriate.

This can lead to confusion when stakeholders request or describe “interviews.” If in doubt, ask follow-up questions or clarify with examples.

Quantitative

How it’s often used: As a synonym for reliable or trustworthy, implying that data from these studies is inherently more robust because its methodology (e.g., A/B tests, surveys, remote unmoderated studies) allows for large sample sizes. For example…

Stakeholder: “It’s interesting that only 1 of the 5 participants finished the task… but we have quantitative data showing that it works fine.”

A more precise understanding: Numerical data about users’ attitudes and behaviors. Common methodologies for collecting such data include A/B tests, surveys, and remote unmoderated studies. Quantitative research is most useful for tracking trends over time, comparing data points, or measuring the scale and impact of an issue or finding.

Although “quantitative” is often associated with reliability, the term itself says nothing about the trustworthiness of the data. A remote unmoderated usability study with over 1,000 participants could still be unreliable due to issues such as incomplete or inauthentic responses, insufficient statistical power (e.g., dividing participants across 50 variants), or poor study design.

Likewise, we know that if only one out of five participants completes a task, it doesn’t mean exactly 20% of real-world users will succeed. But even with this small sample, we can calculate 90% confidence intervals and estimate that fewer than 58% would complete the task. If we replicated the study with a larger sample — say, 500 participants, where 100 successfully complete the task — the upper bound of our confidence interval would drop to 23%, much closer to the observed 20%.

So while it’s true that a well-executed quantitative study provides more reliable estimates, that reliability depends on more than just the sample size. When precision is necessary and resources allow, quantitative research can be invaluable. But it isn’t inherently superior to other research methods and shouldn’t be pursued for its own sake.

Intuitive

How it’s often used: Describes a design that is easy to use, generally pleasing, or subjectively “good.” For example…

Stakeholder: “The main thing we want to know is: is the new experience intuitive for users?”

A more precise understanding: “Familiar, or … uses readily transferred skills.” This definition comes from Jef Raskin, an early Apple employee and HCI expert, who explored this concept at length in a well-known paper (and highly recommended read). Even in 1994, Raskin noted that the most common request in design was for an “intuitive” experience.

Raskin pointed out that, when understood as “familiar,” an intuitive design cannot be significantly different from what people already know. This creates a paradox: since the biggest improvements are often drastically different, a substantial improvement may be rejected on the grounds that it doesn’t feel intuitive. Thus, one reason to get clear about what we mean by “intuitive” is to recognize its implications for our product.

Another reason to define “intuitive” more concretely—by operationalizing it—is that it makes measurement possible. Instead of using it as a vague, aspirational synonym for “good,” we can specifically assess how well a design reflects familiar systems and tools that users already know.

Insight

How it’s often used: As a synonym for data or findings, particularly when they are actionable. According to a literature review by researchers Battle and Ottley, it’s also common for scholars to use the term to refer to either “summary observations about data” or “descriptions of data that support a specific conclusion.”

Stakeholder: “We were hoping you can give us some insights about how people use the customization tool.”

A more precise understanding: Merriam-Webster defines insight in two ways: “1. penetration, or the act of seeing into a situation; 2. the act or result of apprehending the inner nature of things or of seeing intuitively.” In other words, it’s seeing clearly both the trees (the details, data, and findings) and the forest (the big picture, meaning, or implications).

A good illustration of this definition comes from the early study of the term in the context of problem-solving by Gestalt psychologists. While many problems can be solved with incremental, step-by-step progress, others — called insight problems —may appear unsolvable for a time. However, a sudden shift in perspective can reveal the solution.

One famous example is the candle problem, in which the solver is given only a candle, a matchbook, and a box of thumbtacks. The task is to mount the lit candle on the wall in such a way that melted wax won’t drip onto the floor. If you’re unfamiliar, take a minute to try to solve it yourself before reading the solution.

In an era when product teams are under increasing pressure to get insights faster, this definition helps to underscore our key value proposition as professional researchers. It’s true that an LLM can summarize (or, to be more exact, mechanically shorten) an interview transcript. But these tools struggle to truly connect the dots. Unlike human researchers, they lack the domain knowledge and contextual understanding required to interpret not just what was said, but why it matters.

Task Analysis

How it’s often used: Any method using the task as a fundamental unit of analysis, such as task-based usability studies, or diagramming user flows. For example…

UX Pro: “While we’ve been finalizing the journey maps, the design team has been doing a task analysis of the payment flow.”

A more precise understanding: A set of formal approaches that defines a task and decomposes it into more fundamental elements, such as sub-steps or sub-goals, to identify strengths and opportunities. It comes in a variety of flavors, such as a GOMS-based task analysis or a hierarchical task analysis.

In other words, while it may sound like a generic approach, task analysis has a specific meaning in the context of human factors, which is one of the forebears and sister disciplines of user experience. Before you dismiss it is a purely academic method, note that it is one of the most commonly used human factors techniques in applied settings. And although it’s underutilized in UX, it may be put to fruitful use in the work we do.

ROI (Return on Investment)

How it’s often used: The financial benefit of a product enhancement made as the result of UX involvement, such as cost savings. For example…

Researcher: “Our research suggests that the auto-enroll feature, if implemented, would produce an ROI of nearly $25,000 per month in additional revenue.”

A more precise understanding: The financial benefit of product enhancements expressed as a function of the investment required to produce them.

To illustrate, imagine you’re running a stand selling lemonade at 25¢ per glass. Your revenue, or return, would be determined by the quantity of glasses sold. However, the ROI would also account for the cost to produce each glass. If it costs more than 25¢ to make a glass, you would have a negative ROI!

Often return is often discussed without considering the required investment. This may be a simple oversight, as many UX practitioners are not business owners or accountants familiar with these calculations. It might also be because investment figures can be difficult to obtain, especially for individual contributors without purchasing authority or visibility into budgets.

Nonetheless, there are good reasons to overcome these obstacles. Greater business acumen allows UX professionals to understand how the different components of an organization contribute to its success, making them more valuable partners. And even if investment figures are not precise, using estimates can help to create a more compelling picture of the value added by UX efforts.

If you’d like to learn more, Depth previously published an article that offers further explanation and examples for making calculating ROI based on your scenario and data.

Let’s be good communicators

Jargon is inevitable in any specialized field, but as UX professionals, we must strive for clarity. Clear language is the product of clear thinking. If we know what we’re saying, it’s a good sign that we also know what we’re doing. On the other hand, vague or muddled language can hurt our credibility.

We also can’t assume that others understand the terms in the same way we do. Instead, we should take the time to clarify meanings and check for mutual understanding. Simple follow-ups like, “Could you tell me more about what you envision in this survey?” or “What kind of outcome are you hoping for with an intuitive design?” can prevent confusion down the line.

This is not an exhaustive list of problematic terms in UX research. Some would argue that the very name of our field — user experience — is the thorniest. And since we as a discipline love to refresh and rebrand our terminology, a phenomenon Jakob Nielsen refers to as “vocabulary inflation,” misunderstandings can grow worse over time if we’re not careful.

That’s why it’s essential to recognize commonly misinterpreted terms and use them as opportunities to establish a shared understanding. By doing so, we strengthen our communication and, ultimately, our impact as UX professionals.

Intake forms help UX teams streamline the process of collecting user research requests, building a backlog of potential projects, and prioritizing them into a roadmap. But they need to strike a delicate balance.

On the one hand, collecting detailed information allows the UX team to wrap their heads around a research request and the context behind it. On the other hand, every additional question included in the form increases the friction for the stakeholder making the request.

An overly long request form is unlikely to be used by our stakeholders — they may give up halfway through, avoid using the form altogether, and form the misperception that UX research is an overly burdensome process. When creating an intake form, UX practitioners should prioritize the questions included to get just enough information to understand the ask, its scope, and timelines.

Building a UX research intake form that stakeholders will use

The first step in building a form that stakeholders will use is deciding where to create and host it. There are many tools for creating a form, but generally, you want it to be easy for stakeholders to find and use whenever needed. Ideally, you would co-locate your form with something your stakeholders already use or a system they’re familiar with. For example, this could be:

• A custom webpage, like this example from Columbia University, could be particularly useful if your stakeholders are frequently on a company intranet.

• A work management system like Trello, Google Forms and Sheets, Monday.com, or Notion. These platforms can fit into existing project management workflows and include helpful integrations and automations that make organizing requests simple. Here, you can see Odette Jansen's Notion request form.

• A simple template document. If stakeholders in your organization frequently use tools like Google Docs for project planning, a request template like Nikki Anderson's might be useful.

Once you decide where to host the form, the next primary consideration is determining what information to collect, ensuring you gather enough background without creating unnecessary friction for stakeholders.

What to include in your UX research intake form

User research intake forms can capture a variety of project information. It’s helpful to organize this information into categories based on priority and relevance to decide what to include in your request form. Some questions are essential and will be used in every form, while others are useful but optional. Certain details may only be valuable in specific contexts, and some are better left out entirely. Let’s break these different groups down:

The essentials are the foundational pieces of information every intake form needs to provide clarity and context while keeping the process efficient for stakeholders. The six essentials are:

• Set context for the requester — A good intake form should clarify what happens after submission, including when and how you will respond, expected timelines, and how you prioritize requests. Setting this context helps stakeholders understand the form’s purpose and what to expect next. Once they understand the context, you can begin to ask for information.

• A project name — Giving the request a project name makes it easier to reference and track the request.

• Research goals or objectives — Ask the requester what they want to learn, the potential impact of that information, and how they plan to use the findings. This ensures you understand their underlying objectives and what they hope to achieve.

• Background — Ask for any additional information that explains why the research is being requested that can help you interpret their goals and objectives more effectively.

• Important dates — Stakeholders usually request research to support specific initiatives, such as a product release or a key decision. Knowing the ideal timing, or even the last date where findings would be useful, allows you to plan accordingly.

• Name & Contact — This is straightforward but critical—you need to know who submitted the request and how to follow up with them.

If you wanted, you could stop with these six items. However, you could also carefully consider adding 1-2 of these Useful additions — information that is typically useful but not essential and adds to the length of your form. Only include items from this list if you feel they will add substantial value to your scoping and prioritization of requests. Every additional question should be weighed against its potential to deter stakeholders from completing the form, so even if you include them, it is best to mark them as optional.

• User details — Information about the users involved, such as personas, user groups, demographics, or other key characteristics. This is particularly helpful if you have multiple different user groups.

• Needs or Jobs to be Done — The specific user need or task the request relates to.

• File uploads — A space for stakeholders to share relevant documents related to their requests.

• Product line or area — For organizations with multiple products or large websites.

• Success criteria — Ask what success looks like. For example, are they aiming to increase feature adoption or a specific KPI?

Finally, you might consider if any Context-dependent extras are needed. These questions might be helpful in specific situations but often depend on familiarity with user research and UX outputs and should only be included if the rest of your form is sufficiently brief.

• Research method & Details — If your stakeholders are familiar with various types of user research and have specific methods in mind, you could ask about their preferences. However, asking stakeholders to select methods may lead to uninformed choices.

• Assumptions — When planning research, there are often underlying assumptions. If your stakeholder has research experience, they may be able to share these assumptions, which could help shape the project.

• Desired deliverables — If your stakeholders are familiar with user research outputs they might request something specific. So, you could consider providing options to help them indicate what they are looking for (e.g., raw data, PowerPoint report, topline summary, charts, or a specific UX artifact like a journey map)

• Level of support — Depending on your organization, you might offer different levels of research involvement. This is particularly relevant if research democratization is a priority on your team. Consider allowing stakeholders to indicate whether they need help with planning, recruitment, conducting the research, analysis, or any combination of these.

• Budget — This is very context-dependent, but for some teams, it will make sense to ask if a project has received funding during the request phase, so they don’t prioritize it before budget is in place.

Drawing from the three categories above should be sufficient for any research intake form. Anything beyond this we can consider as part of “The don’t list.” In theory, the set of things we shouldn’t include in request forms is infinite (e.g., the stakeholder’s coffee order). In practice, there are two types of questions commonly found on intake forms that should be excluded:

• Decisions that should be left to the research team, like sample sizes, proposed participant incentive amounts, and research tool preferences.

• Information that can be determined at project kickoff, like preferences for meeting times and preferred channels for project communications.

To keep your request form concise and effective, include the essentials and select only a few useful additions or context-dependent extras, as needed—while leaving out anything from the “don’t list.”

Once you have a draft of the form, test it with your UX team first. Aim for a clear goal, like ensuring the form can be completed in 10 minutes or less. After finalizing your draft, release it for a pilot by promoting it to stakeholders through multiple channels (e.g., email announcements, team meetings, workshops, or chat threads). Use feedback and observations from your pilot period to further refine the form, ensuring it provides the information your team needs while respecting your stakeholders’ time and effort.

The bottom line

Research intake forms can help a UX team effectively triage incoming requests, but they need to strike the right balance to be effective. Too short, and they result in time-wasting back-and-forth communications to fill in gaps. On the other hand, overly long intake forms create friction for stakeholders and may lead to the misperception that the process of user research is too time-consuming.

When building your research request form, prioritize the information you collect, including the must-haves from the list above, adding other questions only as they are necessary. Focus on collecting the most essential information needed to triage user research requests.

In 2013, Google unveiled a groundbreaking wearable device called Glass. Sleek and lightweight, resembling a pair of eyeglasses, Glass had a heads-up display, internet connectivity, and the ability to capture photos and videos at any moment without the need to pull out a smartphone. Though it garnered a small and enthusiastic following among tech enthusiasts, Glass was quietly shelved a few years later after failing to find a broader market niche.

It’s a familiar story in the tech industry. Every few years, a new innovation emerges and captures the hype cycle, whether it’s wearable devices, virtual reality, blockchain and NFTs, or more recently, next-generation AI. Companies invest millions, hoping to capitalize on the excitement. But without a compelling use case, they risk developing “a solution in search of a problem” that flops in the marketplace.

Ideally, we develop our solutions based on a deep understanding of the people who will use the tool and their specific needs. But in practice, product teams are often given an initiative from higher-ups to build according to certain specifications. When this happens, how can UX teams incorporate user needs and help build something valuable to them?

This is where matchmaking, a structured framework for identifying and evaluating potential use cases for new technologies, comes into play. By working backwards from the technology to discover user needs that align, we can establish a basic compatibility as a starting point for building more promising future designs.

What is matchmaking?

Matchmaking was developed by two human-computer interaction (HCI) researchers, Sara Bly and Elizabeth Churchill, in 1999. They described it as a four-step process, which we’ve adapted below:

1. Identify the technology’s key capabilities or characteristics.

2. Map capabilities to the broad user activities they support.

3. Identify relevant potential work domains or contexts.

4. Evaluate the fit and feasibility of the matches.

When product development teams follow the traditional development lifecycle, any number of bread and butter UX research methods can be used to identify user needs. For instance, two common viable options include in-depth user interviews or early-stage prototype concept testing.

However, if the team already possesses a “solution” or technical capability in development and is unsure about the most valuable specific use case or circumstances, the matchmaking framework can be a valuable approach.

We’ll take a closer look at each step in that process as Bly and Churchill practiced it using the then-emerging technology of virtual environments. We’ll also describe a more recent adaptation of the process, as used by Yildirim and colleagues in 2023, to develop novel AI use-cases. We’ll compare the examples and describe how UX teams might apply each step in their own contexts.

Step 1: Identify the technology’s key capabilities

First, we must establish the scope of the matchmaking exercise by defining our technology and its key characteristics.

In 1999, virtual environments were still in their infancy. Bly and Churchill described the technology as simple, two-dimensional, and using a combination of graphics and text. These distinctions helped set it apart from similar technologies like instant messaging tools or virtual reality. By clearly articulating these specifics, we can constrain the boundaries for future steps in the matchmaking process.

Identifying what the technology can do may be done at the level most appropriate to your team’s goals. Drawing from their experience using and researching the technology within a single company, Bly and Churchill identified at a high level what they felt were the most important qualities of virtual environments of the time: continuous accessibility, computational efficiency, the ability for synchronous and asynchronous communications, among a few others.

On the other hand, Yildirim and team wanted to develop resources applicable across many AI-relevant contexts and companies. Consequently, they were more exhaustive. Their first step involved compiling a list of dozens of examples of AI-enabled features integrated into existing products across a wide variety domains.

More importantly, they wanted to look under the hood of each feature to understand its technical underpinnings. To achieve this, they thoroughly reviewed secondary sources, including research papers, news articles, press releases, product descriptions, and API documentation. Reviewing each example in greater detail, they were able to simplify the AI’s capabilities by focusing on the inference it made in relation to the available data. This process ultimately resulted in the table presented below, outlining AI’s broad capabilities (as of 2023).

As UX professionals, we’re seldom domain experts on the technical aspects of the products we support. More often, this knowledge sits with our colleagues on the product and engineering teams. At this stage, when we need to understand the technical capabilities of our solution, we may interview them or another subject-matter expert. Alternatively, or as a supplementary approach, we may be able to gather this information through secondary research.

Step 2: Map capabilities to the user activities they support

With a clear understanding of what our solution is and what it can do, we must align those capabilities with broad descriptions of user activities and requirements.

Having already reviewed a combination of existing primary and secondary research, Bly and Churchill were able to identify common scenarios where the specific capabilities they had found useful for users of virtual environments would be beneficial. For instance, having a continuously accessible tool proved helpful in ongoing work, while asynchronous communications proved useful when collaborators were separated by time.

Yildirim and colleagues wanted to instead identify opportunities for brand-new AI-powered applications, so existing research would be of limited help. Instead, they facilitated workshops where they helped different teams of specialists ideate using different lenses. One key approach provided participants with an example-rich version of the AI-features table above, so that they could build ideas on the basis of what the technology is well-suited to do. In contrast to the typical workshop ideation approach which primarily considers personas and user journeys, this feature-led approach kept things focused on viability from the start.

In these examples, the teams of researchers used different approaches to identify potential areas where the technology’s capabilities might benefit users. For many technologies, some combination of primary and secondary research will help UX teams to form those connections, as in the case of Bly and Churchill working on virtual environments. But for wide-open problem spaces with little existing knowledge, such as novel AI use cases, facilitated exercises with other product team members may be a more effective starting point.

Step 3: Identify relevant potential work domains

With a broad list of user activities that the capabilities might support, it’s time to narrow to specific contexts, domains, or user groups, where those activities are common.

Bly and Churchill, who were, at the time of writing, a consultant and a researcher at Xerox PARC, respectively, were supporting a large organization with a diverse range of working streams. Further narrowing down the focus was key. Having some knowledge of the organization, they identified a few promising examples of teams where the activities described were frequent, and the capabilities identified might be put to productive use.

Each team was trying to coordinate work across dispersed members using a variety of communication methods, both new and traditional. These included a math and computer science research team, a sales and consulting team, a magnetic fusion science lab, and a team of middle-management in software development. We’ll revisit these teams in the next step.

In contrast, Yildirim and their colleagues had already limited ideation to domains where their colleagues had prior experience. For instance, they conducted one facilitated ideation workshop with a group of HCI specialists. They asked these specialists to consider familiar applications, such as ride-sharing services like Lyft and Uber, and vacation rental services like Airbnb and Vrbo. In another session, they collaborated with a group of data scientists and nurses to develop ideas for applications in intensive care units.

Depending on your context, you may need to adapt this step in the process as well. For instance, strategic UX teams working across a large variety of workstreams in a mature organization may have the flexibility to narrow down to a few promising groups, as did Bly and Churchill. However, if you’re working on a smaller team or within a more narrowly defined space, you may need to constrain the process from an earlier stage, as did Yildirim and colleagues.

Regardless, by the end of this step, you’ll have a sense of where the solution under development might be more valuable for specific populations.

Step 4: Evaluate the fit of the matches

In this final step, we aim to evaluate how well the specific domain’s activities and requirements align with the capabilities previously described.

Having identified four potential teams as contenders for whom virtual environments might be useful, Bly and Churchill conducted interviews and observations with each of these teams to understand their needs across the identified user activities. They summarized the match between their activities and the capabilities in a table, partially shown below.

Two of the four teams were large and involved in ongoing scientific research with frequent, lightweight interactions. They were selected as the most promising potential users, and Bly and Churchill focused their subsequent iterative design efforts with these teams.

After their ideation sessions, Yildirim and their colleagues used two different collaborative methods to evaluate the potential of their proposed use cases as a working group. The first method, known as the impact-effort matrix, is a widely used prioritization tool that plots the impact of a solution against the effort required to develop it. However, they also developed a unique method tailored to the challenges they identified in developing viable AI-based solutions. This method involves plotting the level of expertise required for a human to accomplish a task against the level of AI performance on that task.

The ultimate goal of this step is to identify the most promising and feasible match between users in a specific context and the technical solution. Depending on your technology and context, you may rely on conventional UX research methods such as interviews and observations, or you may facilitate collective judgments using tools like the impact/effort matrix or something more tailored to your specific needs.

From this point forward, the match has been made! Your team can focus continued design and research efforts in the areas identified.

The bottom line

When product teams build from a technology-first rather than a user-first foundation, they risk developing solutions in search of a problem. Nevertheless, hope is not lost: matchmaking, a structured framework developed and practiced by HCI researchers for over 20 years, offers a way to work backwards from cutting-edge solutions to those users whose needs align most closely.

By following its four steps—identifying key capabilities, mapping them to user activities, narrowing down to relevant domains, and evaluating the fit—UX professionals can help their partners focus efforts where they are most likely to succeed.

Many UX teams find it challenging to handle incoming user research requests while prioritizing and completing projects with limited resources. To address these difficulties, some teams introduce UX research intake forms, but they are just one of many possible solutions. Misapplied, intake forms can create unnecessary friction or fail to deliver desired results. Whether or not a request form is the right solution for their practice will depend on many factors.

In this article, we will explore scenarios where using an intake form can increase a UXR team’s efficiency and situations where it is less likely to be a viable solution. UX researchers can use this guidance to determine the most suitable approach for their team and plan accordingly.

What are UX research intake forms?

UX teams use intake forms to manage requests for user research within their organization. Typically, they are online forms shared with stakeholders who request user research. These forms collect information like project name, background, research questions, and relevant user groups that help a UX team quickly understand the stakeholder’s context and objectives. They provide a structured method for collecting the information a team needs to understand, plan, and prioritize research efforts.

Beyond helping collect the information UX teams need to wrap their heads around research requests, intake forms also help build a research backlog — potential research efforts your team could be working on that aren’t ongoing or scheduled. By using intake forms, UX teams hope to achieve benefits including:

• Operational efficiency - In theory, an intake form should simplify the process of requesting UX research services and reduce the need for time-wasting back-and-forth communication by collecting all the essential information upfront.

• Roadmap prioritization - Generating a backlog of user research needs can provide a UX team with a starting point for building a flexible roadmap.

• Standardization, Consistency, and Transparency - By providing a standardized intake process, a form promotes consistency in how research needs are articulated by stakeholders and evaluated by UX researchers. It also creates documentation that can be an important aspect of providing visibility into how research priorities are set.

Intake forms are not a one-size-fits-all approach. Many factors will impact whether or not a UX research practice will achieve the intended benefits of using an intake form. To decide whether an intake form suits your team’s needs, let’s review the contexts where they are most beneficial and where they might be less useful.

When to use a UX research intake form

Generally speaking, intake forms may be more effective for teams with these characteristics:

1. Centralized team models - In a centralized model, all UX professionals work within a unified team that ultimately reports to a single UX manager or executive. Teams in this model work on an as-needed basis, managing projects from various departments across the organization. Intake forms provide a valuable centralized location for stakeholders to submit their requests.

   

2. Stakeholders proactively request user research - The most important factor in determining whether introducing an intake form will be successful is whether stakeholders are already asking for user research. If they aren’t currently making requests, introducing a form is unlikely to change that behavior.

3. There is a surplus of research needs - Many mature UX teams have more research requests than available resources to execute them. In this context, roadmapping is crucial for triaging which projects to prioritize and which to defer. An intake form helps create a centralized backlog of research needs that can be reviewed and prioritized.

4. Increased standardization is desired - Some teams aim to make their UX research practices more consistent, so they introduce standardization to their processes. If this is the case in your UX org, a request form can be a part of standardizing the earliest stages of the research process by ensuring that all research needs are articulated and assessed in the same format.

5. Stakeholders have some baseline research literacy - Finally, you’ll want to consider if the stakeholders who engage with UX research can articulate their needs. Filling out an intake form requires basic familiarity with research. If your stakeholders understand research concepts, they can likely complete the form independently.

After reviewing the considerations above, you might feel that an intake form is a good fit for your team, in which case your next step should be deciding how to build it. We will cover this more in an article next month.

When to skip a UX research intake form

On the other hand, introducing an intake form might not deliver the intended results in situations like these:

1. Embedded team models - Where intake forms make sense for centralized UX teams, they might not be necessary for embedded teams. In an embedded UX organization, UX staff are distributed within specific product teams. Any given UX researcher or designer will consistently work on the same product, with the same developers, project managers, and product owners. In these instances, informal ongoing communication or specific review meetings often replace the need for formal research requests.

2. Nobody directly asks for user research - It should be said again, that if stakeholders don’t proactively ask for user research, an intake form won’t be a viable method for creating a backlog of requests. In these situations, UXR should take a more proactive role in identifying areas where research is needed.

3. When stakeholders need extra support to define research needs - Teams unfamiliar with UXR might find forms intimidating or confusing, leading to poorly defined or no requests at all. In these situations, a more collaborative and direct approach is generally more effective. You might consider holding live meetings with stakeholders to walk through the process of articulating their research needs together.

4. A culture that strongly resists formal process - You might work in an anti-bureaucracy — a workplace that values speed and flexibility over process, common in early-stage companies. In companies like this, any added process might be perceived as a blocker. Consider whether introducing an intake form will face resistance and hinder research efforts.

5. Research plans face frequent pivots - If you find that your research plans and priorities are constantly changing, introducing an intake form might add unnecessary workload, requiring constant change to form submissions to stay current with shifting goals. Before implementing a request form, determine why plans change so frequently and consider process improvements and change management to reduce the need for rework in the planning stages.

If you determine that an intake form isn’t the right fit for your current situation, there are alternative means for building your organization’s research backlog. These include documenting future study ideas at the close-out of every project, keeping up regular check-ins with key partners to uncover potential research gaps, and actively participating in planning sessions to identify research opportunities for upcoming releases. Additionally, you might consider hosting research office hours for stakeholders to discuss their needs or reviewing product analytics, support tickets, and customer reviews to uncover pain points and spot areas where further research may be needed. Ultimately, a combination of these methods is usually more effective than relying on just one.

The bottom line

A UX research intake form can be a valuable tool for collecting and managing research requests across an organization, but it’s not a one-size-fits-all solution. There are a variety of context-dependent factors that will determine how effective a request form will be for a given team. Alternatives like stakeholder meetings, project closeout procedures, regular check-ins, or even office hours can also help UX teams document research needs.

Evaluate your team’s current situation against the set of considerations above. If you feel an intake form could work, start by designing a simple version for a pilot period and iterate based on feedback and observations. If intake forms don’t fit your needs, there are plenty of other methods for capturing your user research backlog. Regardless of whether you use an intake form, using a combination of these methods will yield the most comprehensive results.

UX research teams often behave like order-takers, reacting to an endless stream of urgent, ill-defined requests from stakeholders.

To address this, some teams take a proactive approach by creating a UX research roadmap. Leaders collaborate with their teams, investing significant time and energy crafting a detailed, eye-catching artifact. But as product roadmaps shift, these roadmaps end up gathering dust, creating the sense that you’re chasing moving targets.

Ideally, a roadmap will adapt to reflect the ever-changing realities of your organization. The goal is a living document that evolves with your organization’s changing priorities. In this article, we’ll discuss a flexible approach for creating one.

1. Fill a backlog of research ideas

A backlog comprises all of the potential research efforts your team could be working on that aren’t ongoing or scheduled. This list serves as the foundation to any roadmap.

If you already have one, you can skip ahead. If not (or if it’s looking sparse), here are a few ideas that could help you get started:

Create an intake form. Start directing your stakeholders to a form that documents their problems and needs. This forces them to think through and clarify their requests, and will often preempt unnecessary meetings.

While you spin up a more robust roadmap, your intake form can serve as the basis of at least the most rudimentary form of one. Call it the “Deli Shop model”: stakeholders place their “order”, get a “number”, and requests are handled in chronological order.

Document future study ideas. Every project raises new questions. A “good hygiene” practice is to end each deliverable with proposed follow-up topics. You should document these in a centralized location (see, for example, Odette Jansen’s template). Over time, you can review these ideas, keeping the most relevant while discarding the rest.

Keep up regular check-ins with key partners. Don’t expect these conversations to automatically yield fully developed research ideas—you may need to ask probing questions to understand where there may be gaps in their understanding that need to be filled. At a minimum, being aware of upcoming initiatives will get you thinking ahead of future conversations.

Participate in planning sessions. These can be goldmines for backlog opportunities. Keep your ears open for any upcoming features that might benefit from user research; chances are these opportunities will jump out at you more quickly than any non-UX colleagues.

By combining these strategies, you’ll keep a healthy backlog of ideas and never lack for potential work to do.

2. Estimate time and resources required

For each of your backlog items, you’ll want a rough sense of size and scope. Time and resources are always limited, so tally up the cost for each project idea. Some studies will be smaller, quick wins; others will require significant investment.

In no particular order, here are three elements to consider:

Estimate time. Here’s a rule of thumb: research projects often take 3–6 weeks (though timelines can range from a week to six months depending on the method and scope). Assign each project a ballpark timeline and work backwards from deadlines to establish the latest start date.

Determine who will execute the work. Another rule of thumb: a dedicated UX researcher can handle 2–4 projects simultaneously (depending on complexity). Are your researchers available, and equipped with the right skills? If not, consider alternatives: hire additional staff, engage a vendor, or involve experienced designers or PMs who would be willing and able to pitch in.

Evaluate your budget. Projects come with costs, from tooling to participant incentives. Check what’s covered in your annual budget and identify any gaps. If additional funding is needed, document these requests for consideration. By the end of this process, you may be able to build a compelling case for stakeholder support.

Having considered the time, resources, and costs, you can start to arrange the pieces in time.

3. Rough out the sequence of events

You can’t do everything at once. Provisionally spreading these efforts out over time in a logical sequence makes the list more manageable and less daunting. Here are three considerations to help you do that:

The phase of development. Different research activities are better suited for different stages of the product development lifecycle. For example, benchmarking aligns best with measurement, while discovery research is common in strategy phases. If your team supports multiple products, consider the differing phases they might be in and plan methods accordingly.

Existing product roadmaps. Your stakeholders likely have their own timelines for initiatives. Your research won’t be useful if it doesn’t deliver insights ahead of milestones. Be mindful of key dates that aren’t likely to shift; for instance, ecommerce teams often institute year-end code freezes to avoid disruptions during peak periods.

Dependencies and timing. Some studies will rely on the results of others, so sequence these accordingly. Other efforts might align with seasonal or strategic milestones, such as preparing benchmarks ahead of annual planning or syncing data collection with a major release cycle.

With these considerations in mind, using a simple “Now, Next, Later” framework to organize the backlog items will suffice in most cases. Concrete weeks and months are nice to add where you have them. Naturally, you’ll have more certainty about nearer-term items, while anything too far out will be more tentative and subject to change.

By arranging the backlog into a logical sequence, you can start to identify conflicts or overcommitments, giving you a clearer picture of where prioritization is needed.

4. Prioritize research efforts

So far, we haven’t had to make any tough choices. If you’re looking at your list and thinking your team has the resources to handle it all, consider yourself one of the lucky ones. The rest of us must prioritize carefully to ensure we’re focusing what matters most.

One straightforward and effective tool for this is the Impact-Effort Matrix. Plot each project across two dimensions: high or low impact, and high or low effort. You’ve already estimated effort in step 2, so now it’s time to evaluate impact.

Evaluating impact means considering two things:

1. Return on Investment (ROI): Which projects deliver the biggest business benefits relative to their costs?

2. Alignment with KPIs: Does the work contribute to the goals your team or organization is judged by? Ideally, ROI and KPIs align, but if not, individual leaders should determine what best serves their long-term interests.

If you’re looking for stronger justification for judgments about high or low impact items, you may be able to bring user research into the discussion. For instance, some teams will have relevant quantitative research that provides more reliable estimates of what matters most to users, such as the Kano model or a Key Drivers study. You wouldn’t want to rush these complex methods if the planning cycle has already begun—but you might pencil them in to inform the next one.

Filled out, the matrix will guide your choices: focus on high-impact, low-effort projects, and cut low-impact, high-effort ones.

5. Prepare an artifact you can socialize

With decisions made about what to prioritize and when, it’s time to put it together in a way that people can react to it and use it.

Whatever format you choose, keep it functional. Avoid over-investing in a polished roadmap deliverable; instead, think about how you can leverage the tools your team uses daily. Whether that’s Miro, a Kanban board, Trello or some other project management software, or even a simple spreadsheet, what matters most is that the roadmap lives in a place where it’s actively used.

Now that you have a working draft, it’s important to seek feedback and perspectives from anyone who might have a stake in it, including your leaders, peers, and team members. They may have pieces of the puzzle that you’re missing—valuable context or information that can refine, improve, and keep your roadmap relevant.

As the artifact matures, it may be appropriate to disseminate it more widely in a larger setting, like org-wide town halls or the aforementioned planning sessions. Presenting your vision to a broader audience can raise your team’s profile, opening the door to new partnerships and opportunities. Make sure to share links or attachments to provide ongoing (even view-only) access, enabling stakeholders to revisit the roadmap whenever needed.

6. Keep it relevant over time

The heart of this approach is to create a living document that’s resilient to whatever shifts and changes come along, as inevitably, they will—whether that’s a new C-suite directive, a market disruption, or some other unforeseen event.

That starts with regular hygiene habits in day-to-day use. For instance, the project management software you use may support automations or “if-then” workflows that automatically clean up items with tags and dates. Even if you’re using a basic solution, you can dedicate time each day or week to review and revise items to keep them up to date.

Additionally, you should have occasional dedicated reviews, often on a monthly or quarterly basis, throughout the life of the roadmap. These sessions involve taking a step back to evaluate the plan holistically and assess its alignment with the current state and reality of the organization and its goals.

By treating your roadmap as dynamic and flexible, you’ll ensure that it remains relevant and actionable, even in the face of unexpected circumstances.

The bottom line

If you’re frustrated by roadmaps that quickly fall out of sync with reality, consider moving to a more adaptable system. Begin with a backlog of potential research initiatives, estimate the scope and logical order of each, and use prioritization frameworks to focus on high-value work. Then, house your roadmap in a system that’s easy to update and accessible to stakeholders.

This method won’t eliminate every last-minute pivot or unforeseen demand, but it gives your team the tools to manage unpredictability with confidence. By adopting a living, flexible roadmap, you’ll stay ahead of incoming work and establish your team as a trusted strategic partner for your stakeholders.

Most UX professionals have never tried to estimate the ROI of their work. Very few in UX have formal education in finance, which can make concepts like ROI seem out of reach. Unfortunately, this leaves them at a disadvantage when discussing the value of UX with stakeholders who are more financially focused. Without quantifiable impact, UX practitioners miss opportunities to secure buy-in and budget for their initiatives.

Fortunately, calculating return on investment isn’t as intimidating as it seems on the surface. Once you grasp their basic structure and the specific scenarios where they can be used, nearly anyone can do an ROI calculation.

This article will illustrate the importance and relevance of ROI calculations in UX research practices, helping you identify and understand three specific scenarios where UX professionals should calculate ROI.

Calculating ROI for UX Research: Importance and challenges

Return on investment (ROI) is a standard demonstration of the financial efficiency of an initiative. As its name implies, ROI calculates the profitability of an investment relative to its costs. By calculating ROI, UX practitioners can provide tangible demonstrations of the value their work generates. These demonstrations can help:

• Appeal to financially-focused stakeholders - Many stakeholders in business are driven by dollars and cents. Linking UX initiatives to outcomes like boosting customer lifetime value or reducing development costs helps persuade them that user experience is good for business.

• Secure buy-in and budget - Decision-makers want assurance that their investments will pay off. Budget proposals supported by ROI provide clear and persuasive benefits for our work, bolstering a more compelling case for funding.

• Elevate the strategic importance of UX in an organization - Demonstrating financial and operational outcomes can help shift the view of UX as a support function to a mission-critical business driver. Linking UX outcomes to top-level goals showcases its strategic value. For example, if a business is focused on customer acquisition, a UX team can demonstrate how its work leads to increased product adoption.

Despite these benefits, few UX practitioners incorporate ROI calculations in their work. While third-party examples like Forrester’s case study claiming 9,900% ROI on UX investments are commonly cited, few UX teams apply similar calculations to their own projects. A few common barriers get in the way.

First, most UX professionals enter the field because they are passionate about improving how technology is designed by focusing on the people who use it, not financial modeling. While we might focus on the human benefit of our work, demonstrating ROI can help win over stakeholders who are more focused on monetary returns. The good news is that you don’t need any formal training in finance or accounting to start making basic return on investment calculations.

Second, ROI can feel daunting at first. In particular, many worry about the accuracy of their calculations. It’s natural to have these concerns, given how many variables influence business metrics. But rest assured that ROI is meant to be an estimate of the isolated effect of your work under controlled assumptions, not an exact figure. It’s a practical, theoretical demonstration of your impact, not a formal financial report.

Finally, time is one of the biggest constraints for UX practitioners. Performing ROI calculations can seem like adding one more thing to a long list of to-dos. Fortunately, ROI estimates can be surprisingly quick, only taking a few minutes after metrics are gathered. With this in mind, let’s look at how we make these calculations and go through a few examples.

How to calculate return on investment

On the surface, ROI calculations can seem intimidating, but they are rather simple when you break them down. Every ROI calculation uses this formula:

Return is the financial benefit that an initiative achieves, which can be an increase in profit (e.g., an improved conversion rate on an e-comm site leads to more sales) or a decrease in cost (e.g., improved usability on a booking page means fewer travelers need to call support for help with their flight, reducing call center costs). Investment refers to the expenses that were required to achieve the Return. In UX, this often includes participant recruitment, agency fees, software licenses, and time spent by designers, researchers, or developers.

Many articles on ROI focus solely on the Return part of the equation. Calculating the benefits of your UX projects alone is worthwhile because it quantifies the value of your work. Return-only calculations offer quick wins for demonstrating impact, but only tell part of the story. Calculating the complete ROI is even better, providing a full picture of your project's financial efficiency. However, estimating Investment can be difficult, especially for an individual contributor who might not have visibility into all the budget details for a project. However, to be a true ROI, you need to have estimates for both Return and Investment.

Once you have estimates for the Return and the Investment, enter them into the equation. For example, if a project had a budget of $10,000 and we estimated it increased revenue by $15,000, the calculation would be:

Let’s look at three in-depth examples of when you can leverage return-on-investment calculations.

1. Demonstrate the return on investment (ROI) of past UX projects

The easiest scenario to calculate ROI is for a completed project. To do so, you will need product analytics before and after design changes to calculate the Return.

Imagine we work at a software company with a subscription model offering the first month free. Our KPI is the percentage of users who convert to paying customers after their free trial. Our UX team works on a project to improve the onboarding process so users can set up and use the product more easily. This project leads to an increase in the conversion rate, directly increasing company revenue.

Imagine that we collected the following metrics:

• Conversion rate pre-redesign: 2.5%

• Conversion rate post-redesign: 7.5%

• Average trials per month: 1,000

• Subscription cost: $100

• Average retention: 14 months

• Total project budget: $300,000

To calculate the Return of our project, we could estimate that the increase in revenue follows this equation:

Our estimated Return factors together the increased conversion rate, the number of users who sign up every month, subscription cost, and how long the average user renews their subscription. The resulting return calculation would be:

This change in onboarding adds $70,000 in customer lifetime value per month. As we mentioned earlier, showing the Return alone can be sufficient, especially if you don’t have access to budget details. However, let’s imagine you have access to budget information for the project.

We can enter our estimated Return and project budget into the standard ROI equation. Since our redesign creates ongoing benefits, while the project budget was a one-time expense, it is logical to calculate ROI over time. In this example, we’ll multiply the monthly return by 12 to estimate ROI over a year.

Through this example, we easily demonstrated the ROI of a past project by using analytics to track key outcomes before and after redesigns. In this case, we can say our redesign adds $70,000 in LTV per month and is estimated to yield 180% ROI over the first year.

2. Estimate the potential business outcome of proposed projects

In addition to calculating the ROI for past projects, UX practitioners can estimate the ROI for a proposed project. By demonstrating the potential business outcomes of a user experience project, you can make a stronger case for its approval and funding.

The main difference in these a priori calculations is that they are speculative estimates. When calculating the return on investment of a past project, we can refer to changes in product analytics. On the other hand, when determining the potential outcomes of a future project, we have to forecast results using reasonable estimates from public case studies or your team’s prior work.

Imagine an example where we’re working for a hotel chain facing frequent customer support calls from users struggling to book rooms online. These calls, resulting from a poorly designed booking flow, cost the company a significant amount of money. This is a clear opportunity to redesign the booking experience, which could both help our users and save the company money. Our UX team wants to include an ROI estimate in our project proposal.

To produce our estimation, we collect the following metrics:

• Current booking success rate: 75%

• Expected ∆success rate, based on previous case study: +12%

• Traffic volume: 80,000 users/month

• Probability of calling support: 18% (of users who fail to book)

• Average support call cost: $8

• Project budget: $35,000

To estimate the Return, we’d compute the projected cost reduction from the decrease in customer support calls from users who fail to book online. That might look like:

After entering our values we get the expected Return — we estimate the changes will reduce customer support costs by $13,824 per month.

Now, using this estimate and the project budget, we can compute ROI. Again, it makes sense to illustrate the ROI for a year, as the budget is a one-time cost while the usability improvements will produce lasting effects.

Through this estimation, we can demonstrate that our proposed project reduces customer support costs by $13.8k/month and yields a 374% return on investment within 12 months.

3. Make the case for investment in tools and software

Finally, it is useful to demonstrate the return for a tool or software that your team is considering purchasing. In isolation, software costs look like expenses that a company wants to reduce. However, when put in the context of an ROI calculation, tools can be framed as strategic investments that help save time, reduce costs, and increase a team’s ability to deliver results.

Let’s imagine that our team is considering a new subscription to a participant recruitment tool. Our researchers spend hours recruiting, scheduling, and sending reminders to participants for their usability studies. Our goal in purchasing this tool is to streamline recruitment, allowing them to focus on other impactful tasks. So, our team runs a proof-of-concept (POC) with a prospective vendor and collects these metrics:

• Current time per study: 4 hours

• Time per study with new tool: 1/2 hour

• Average hourly cost of UXR: $90

• Studies per year: 175

• Avg participants per study: n= 8

• Avg cost per participant, current process: $90

• Avg cost per participant, via new tool: $125

• Software license cost: $20,000

We can estimate the Return by factoring the time saved per study with the value of our researcher’s time, and the total volume of studies over one year.

Entering the values we collected produces the following estimate:

Now, we estimate the total cost of the tool. Factoring both the fixed cost of the license and the estimated variable costs of participant recruitment.

Note we’re looking at the difference in cost per participant, as we’d be recruiting participants either way. After entering our figures, we calculate the estimated year-long investment as:

After plugging in the Return and Investment, we find a negative ROI for the software. At the currently quoted price, this software doesn’t produce financial benefits for our team.

Although the estimation didn’t show positive returns, that’s useful information. Now, we know that we’d need to negotiate the price of the software license or the cost per participant to see positive economic returns.

The bottom line

Return on investment calculations are a standard method for demonstrating the financial efficiency of an investment. UX practitioners can leverage ROI calculations to demonstrate the value of their work, secure buy-in from financially motivated stakeholders, and elevate the strategic importance of user experience in their organizations.

While many in UX don’t calculate the ROI of their work, it’s quite easy to get started. By using a standard equation, and making a few estimates, UX practitioners can demonstrate the business impact of UX improvements, estimate the potential impact of a proposed project, or even make the case for new software or tools that will make their work easier.

In just about any context, human behavior and decision-making can be complex, and it rarely hinges upon a single factor.

Consider something as seemingly straightforward as purchase intent. Is it all about the price? Does easy website navigation influence the decision? Or maybe a product feature stands out? Most likely, it’s a blend of these and other factors. Without a clear understanding, it’s easy to waste resources, focusing on aspects that may not significantly impact user behavior.

Key Drivers Analysis (KDA) is a specific technique that quantifies the elements of your product or service that most impact outcomes like purchases, satisfaction, or retention. By quantifying the influence of each factor, it takes the guesswork out of deciding where to focus development, marketing, and sales resources.

In this article, we’ll introduce KDA, explain its purpose, and compare and contrast it with similar methods. You’ll learn how to conduct a KDA, and how to visualize and interpret the results. By the end, you’ll add KDA as an essential tool in your toolkit for making informed, data-driven decisions.

What is a Key Drivers Analysis?

KDA aims to uncover the independent variables—called drivers—that most strongly influence a desired outcome.

In product development, this means identifying which features or attributes have the greatest impact on key performance indicators like customer satisfaction, perceived effort, purchase intent, churn likelihood, or loyalty. KDA helps answer essential questions such as:

• What drives users’ perception of trust while using our website or app? Is it the content, usability, or loading speed that matters most?

• What factors influence the decision to purchase or subscribe? Is it price, product features, or the quality of customer support?

The concept behind KDA is straightforward. Imagine you want to predict whether customers will recommend your product. Your team is aware of several potential factors—such as price, product quality, or ease of use—that could influence their decision. KDA uses statistical techniques, often through a method called multiple linear regression, to analyze these factors and determine which are most influential. In simple terms, KDA reveals how much each factor impacts the overall outcome, which gives teams much more confidence in their product decisions.

For those less familiar with quantitative methods, KDA may seem similar to MaxDiff and conjoint, which are also popular for measuring customer preferences. However, each method serves a distinct purpose, addressing related but unique questions. KDA’s strength lies in its ability to quantify how much each factor influences a specific outcome. Here’s a quick comparison with these other two common methods:

By understanding these differences, you can choose the right method—or combination of methods—to gain the clearest insights into what drives your customers and where to focus your efforts.

How to conduct a Key Drivers Analysis

A KDA may sound complex, but breaking it down into simple steps can make it manageable, provided that you have some confidence with inferential statistics. Here’s an overview of how KDA is done:

1. Define the outcome variable and potential driver variables

Start by selecting a single outcome variable (e.g., customer satisfaction or purchase intent) and potential driver variables (e.g., product features or usability factors) that your company has some influence over. Your choices should align with business objectives and be based on prior research efforts (if you have any), or at least, solid theoretical grounding. At a high level, that means making sure the predictors logically precede or influence the outcome. Also avoid any obviously irrelevant variables to prevent noise from clouding your results.

2. Collect data

The next step is to collect data that includes both outcome and driver variables. While this can come from a range of sources—such as unmoderated UX benchmarks, intercept surveys, or live product analytics—often, the best approach is to design a Key Drivers survey specifically for this purpose. For example, if your outcome variable is Customer Satisfaction (CSAT), you’ll ask the standard CSAT question alongside other questions targeting attitudes and behaviors that may influence satisfaction.

A note on sample size: the ideal target will depend on factors such as the number of drivers you’re assessing, the expected strength of their impact, and whether you’re comparing different participant groups. Generally, sample sizes between 50 and 150 are common for meaningful results.

3. Prepare a statistical tool for the analysis

Once your data is collected and cleaned, import it into a robust statistical tool. Options include commercial packages like SPSS or SAS, or free tools like R, Python, JASP, or Jamovi. These tools allow you to process the data and run the necessary calculations to determine key drivers.

4. Ensure data meets assumptions

Review the data for potential issues like collinearity, non-linear relationships, or homoscedasticity.

Although a full treatment of these considerations is beyond the scope of this overview, we can consider collinearity, when two or more factors are closely related, as an example. This common issue can threaten any meaningful interpretation of your results — ideally, you would head this off in Step 1 by defining clearly distinct variables. Nevertheless, there are a few statistical approaches to help you identify or avoid multicollinearity problems, such as by calculating the Variance Inflation Factor (VIF; values above 10 typically indicate a problem), or averaging over orderings.

5. Calculate the importance of each factor

In most cases, you will next run a multiple linear regression analysis. This step identifies how different factors relate to your outcome and, specifically, which factors have the most significant influence.

Calculate how much of the outcome your factors explain and how much remains unexplained. Use standardized beta weights to express each factor’s influence in comparable units. Then, use the R-squared value (or proportion of total variance explained) to convert these weights into percentages to rank which factors matter most.

6. Visualize and apply the insights

Finally, generate visualizations to bring your results to life. Visuals will help illustrate which factors are most important and how they compare with each other. In the next section, we’ll explore common visualizations for presenting KDA results and discuss how to use these insights to inform better business decisions.

What does a Key Drivers output look like?

The output of a KDA may be presented through a variety of visuals, each offering a unique perspective on the data.

One of the most common and effective ways to display results is a scatter plot combined with a 2×2 matrix. In this figure, each factor is plotted along two key dimensions: importance (how much it influences the outcome) and performance (how well the factor is currently doing). The visual is then divided into four quadrants:

• Improve (high importance, low performance): These are critical areas that need urgent attention because they have a significant impact on the desired outcome but are currently underperforming.

• Grow (high importance, high performance): These are your strengths, and you should further capitalize on them.

• Maintain (low importance, high performance): These areas are performing well but have a lower impact, so you can focus efforts on keeping them where they are.

• Defer (low importance, low performance): These factors matter least, and so should be the lowest priority for investment.

Another visual output to consider is a proportion chart. This rectangular chart, resembling a mix between a pie chart and a treemap, visually represents the relative contribution of each key factor as a score or percentage of the overall variance in the outcome. Each “slice” or section of the chart illustrates how much of the outcome variability is explained by each driver.

One feature that might stand out in this chart is the unexplained variance. This reflects the portion of the outcome that cannot be explained by the factors in your current model. Even the best models will leave a significant amount of variance unexplained, as human behavior and customer decisions are often influenced by unpredictable or unmeasured factors. You’ll need to use your discretion when discussing unexplained variance with non-researchers, however, as this could backfire and undermine the credibility of your results.

Substantial unexplained variance can also suggest that there are other important drivers missing from the current analysis or that more data is needed to better capture customer behavior—an opportunity for further exploration.

The bottom line

Understanding what drives your customers’ behaviors and judgments can better inform important business decisions. Key Drivers Analysis (KDA) helps you identify and quantify those most influential factors, allowing you to focus resources on what truly matters.

• KDA uncovers the most influential drivers behind outcomes like customer loyalty, purchase decisions, or service interactions.

• Compared with other methods like conjoint and MaxDiff, KDA provides a precise measure of influence, highlighting not only what matters but how much it matters.

• Conducting a KDA involves defining outcome and driver variables, collecting and cleaning data, running multiple linear regression, and developing visual outputs that inform.

• Visualizations such as scatter plots/2×2 matrices, and proportion charts provide a clear picture of which areas to improve, grow, maintain, or deprioritize.

KDA equips you with actionable insights that guide where to focus your efforts for maximum return—whether that means refining your product features, improving trust, or boosting customer satisfaction.

Recruiting representative users is crucial, as it ensures that our findings reflect our target audience's behaviors, needs, and attitudes. So, one of the most frustrating pain points we encounter in user research is the inability to recruit users for our studies.

In particular, the two biggest roadblocks that get in the way are:

1. Low Budgets — Insufficient funding limits the ability to offer attractive incentives, reducing participant interest and participation.

2. Difficult to find users — Certain types of users, whether due to being from a rare population or their general unwillingness to participate in research, are difficult to find and recruit.

Depending on their circumstances, a research team may experience one, both, or neither of these issues. Each of these scenarios influences how research teams should approach recruitment and study design. In this article, we'll explore how to navigate the challenges associated with each of the four possible scenarios.

Low budgets & Difficulty finding participants

By far, the two most common pain points we hear about recruiting participants for UX research are low budgets and difficult-to-find users. Let’s go over what each of those means.

When discussing participants being difficult to find, it’s helpful to frame the conversation around incidence rates (IR); the proportion of people in the broader population who meet our requirements to participate. In the table below, you can see that if our target audience was for a mass-market application that anyone uses or even a broad segment of the population (e.g., people who use online banking), that would fall into the High to Medium tiers of IR.

On the other hand, if our target participants were from a more narrow demographic or interest group (e.g., optometrists, people who are currently pregnant, 3D developers, consumers of a niche product, etc.), that would fall into the low or very low IR ranges.

We find that most teams’ research tends to predominantly fall on one side of the spectrum; we’ll consider teams who do a majority of their studies with Medium-High IRs as primarily looking for ‘easy to find’ participants, while those focused on Low-Very Low-IRs as primarily seeking ‘difficult to find’ participants.

Defining what constitutes a low or high budget is less straightforward than measuring recruiting difficulty by IR. It is also somewhat situation-dependent; what is a generous budget for one team might not fulfill the basic needs of another. Generally speaking, if your team doesn’t have enough budget to source and compensate participants across all the research studies it plans or needs to do, we would consider that a “low or no budget” scenario.

When considering the various IR and budget scenarios, we get a 2x2 matrix that looks like this:

Regardless of which scenario you find yourself in, good research is possible — it’s a matter of making the most of your resources and leveraging techniques to minimize these difficulties and their consequences.

Scenario I - High IRs and Low Budget

Starting with this first scenario: a research team primarily recruits participants who are easy to find (high-IR), but they find themselves limited by low (or no) budget for recruitment. Budget constraints can be frustrating and restrictive, but groups in this situation can find success by using flexible and cost-effective solutions to engage their high-IR users:

• Reduce costs with DIY recruitment - Using a recruitment agency or panel company can save a lot of time, but you pay for the convenience through commission fees. Techniques and channels like guerilla recruitment, social media, forums, going to a busy location your users frequently visit, and implementing in-product nudges are more time-consuming but often save significantly on costs.

• Turn one participant into many - Reduce the need to recruit from scratch for every project by establishing a participant list. Do so by including an option to opt-in to future research at the end of every study you run; over time this reduces the costs and time needed to find willing participants. Additionally, consider the snowball technique by asking participants if they know another person who would be interested in participating.

• Right-size monetary incentives - IR and participant honorariums have an inverse relationship. Fortunately for teams in this scenario, high-IR participants generally come with the most modest honorariums. Still, you’ll want to make sure you’re paying participants a fair rate. Consider modifying your research methods (e.g., use more unmoderated methods, shorter study durations, etc.) for cost efficiency.

• Consider alternate incentives - For teams with particularly low budgets, it might be wise to consider a raffle-style drawing or non-monetary incentives like swag. If you opt for these incentive methods, relentlessly reduce the time it takes for a participant to complete your study — it’s a lot easier to get someone to answer 2-3 questions than join an hour-long call if you’re only offering a free shirt.

These techniques help teams maximize their budgets by reducing the costs associated with locating participants at the expense of a researcher’s time. Over time, you may be able to move from Scenario I to Scenario II by showing value through discount methods and articulating the potential impact of an expanded budget.

Scenario II - High IRs and High Budget

In this scenario, neither pain point is present; target users are easy to find (high-IR), and there is ample recruitment budget. This is the most manageable of the four situations and allows for the greatest research flexibility. Teams in this situation should maximize the returns on their budgets and consider:

• Third-party panels - Unlike Scenario I, teams in Scenario II can make the cost-time tradeoff associated with employing panels. With high-IR participants, most reputable panel companies and tools with built-in recruitment functionality are viable options.

• Use funds wisely & Negotiate - When using third-party panels and recruitment vendors, take advantage of bulk discounts or more favorable terms in ongoing service agreements. Small negotiations can yield large differences in what you get for your money here.

• Set appropriate incentives - Test different incentive levels to find what your target audience finds most appealing. Often, high-IR users engage well with modest compensation, so be fair, but don’t overspend.

Without the pain points of budget and sourcing difficulty, most research methods are feasible; even very high-sample size rolling research programs are doable given how easy it is to find and compensate participants in large numbers.

One last caution: carefully monitor your data quality — use best practices in your study screeners and exercise due diligence in reviewing the participants sourced by third parties.

Scenario III - Low IRs and Low Budget

In Scenario III, we have a team with both of the most common pain points; their users are rare (low-IR), and they receive little budget to allocate to recruitment. Again, good research is possible here, but teams have to use a variety of strategies, invest a good deal of time, and sometimes make a few concessions:

• Invest in building a panel - Third-party panels are most likely out of the question; lower-IRs have significantly higher costs per participant (in terms of both the incentive paid to users and the commission the sourcing agency charges). Many teams in this scenario turn to building and managing their own panel of users who have agreed to participate in research. This solution is an up-front investment that helps reduce the ongoing costs of doing research.

• Tap existing contacts - A wise way to fill an internal panel (at almost no cost) is to coordinate with other teams who regularly interact with users in their day-to-day roles, like sales or CS in SaaS businesses. Use these natural touchpoints as a mechanism to invite users to your panel.

• Individualized recruitment - Given the scarcity of target users, you may need to contact potential participants individually. That can involve searching for your target criteria and sending individual messages on platforms like LinkedIn. This is a time-consuming approach, so be sure to pair it with opt-ins for future research and snowball sampling.

• Appeal with unique benefits - With limited funds, you may need to attract these participants with non-monetary incentives. Many users may be motivated by the opportunity to help shape the direction of products they use frequently for work. Others may find discount codes or free subscriptions a great alternative to a cash incentive.

• Make participation easy - In addition to setting the right form of incentive, you will want to make it as effortless as possible for people to participate. That might include using unmoderated techniques more often, shortening study durations by relentlessly prioritizing questions and tasks included in the study, and even setting any moderated sessions outside normal working hours.

While the combination of these two pain points presents significant challenges, teams can still conduct meaningful user research that drives product strategy — but there might be a few methodological concessions. For example, high sample sizes are unlikely in this scenario, so you may primarily conduct qualitative methods. Using secondary research and inspection methods at the start of any project can be a helpful supplement that will prepare you to make the most of available test sessions.

Scenario IV - Low IRs and High Budget

In this final scenario, teams face the challenge of recruiting difficult-to-find (low-IR) users, but they receive the financial resources (high budget) to recruit participants effectively. A few things these teams should keep in mind:

• Specialized panels - Though expensive, panels that have the established ability to reach your niche target are a time-saving solution that can be very attractive when you have the budget to employ them.

• Cost-benefit analysis for panel building - Assess the long-term financial implications of using a specialized participant recruitment vendor vs. the investment required to build an internal panel. Some key variables you’ll want to consider are participant incentive costs, sourcing fees, ongoing maintenance expenses, panel attrition, and your overall research throughput. You might find it makes sense to use both, engaging a niche recruitment vendor and supplementing with your own participants.

• Competitive incentives - To attract the right users to your studies higher-than-average incentives will be necessary. However, this comes with two implications:

  • Watch for bad actors - High incentives are attractive to fraudulent participants. Watch for bad actors who game your screeners and consider alternative/supplementary methods of verifying qualification.

  • Must-show ROI - This type of research is expensive, so there is extra pressure to show the ongoing return on investment. Without showing the value of research, teams can move from Scenario IV to III very quickly.

While a generous budget can facilitate niche recruitment, teams should still allocate funds carefully and demonstrate the impact of their research. An additional implication for teams in Scenario IV is that quantitative methods are still feasible, but the likelihood of achieving extremely high sample sizes on an ongoing basis is reduced.

The bottom line

Recruiting representative users for studies is a core research operations consideration for UX and product teams. Two barriers commonly impede this task: 1) Low budgets & 2) Difficult to find (low-IR) participants.

Depending on their circumstances, teams can have one, both, or neither of these pain points. This results in four unique scenarios, each with its own set of strategies and viable recruitment solutions. Using the framework provided in this article, you can determine which scenario best describes your team and use the suggested solutions to address your specific recruitment challenges.

Recent Podcast: Making Sense of UX Research Job Leveling

Lawton Pybus and I joined Amy Santee on the What is Wrong with Hiring podcast to drill into our research on UXR leveling frameworks, from IC to manager roles. Listen to the episode for a live discussion exploring the implications of our recent Data-Driven Career Ladder series.

You can go a long way with the classic “discount” UX methods like in-depth user interviews, usability testing, and well-designed surveys. These perennial approaches are popular for good reason—they deliver valuable insights without requiring a lot of resources.

But when it comes to showing the long-term impact of UX efforts, teams are often asked to provide more concrete evidence. While some argue that other departments don’t face the same pressure to prove their value, we often say our contributions are important for business outcomes—and it’s only fair to back those claims up.

This is where UX benchmarking can come in. As a quantitative method, benchmarking uses larger sample sizes to provide more precise estimates of key metrics. It uses a more tightly-defined protocol, which makes the findings and comparisons more reliable over time than those generated by qualitative methods alone. And when properly done, benchmarking can help to show the return on investment (ROI) of your team’s UX efforts over time.

But challenges can arise in the first few steps—take, for example, selecting the right tasks. What should you measure? How many can you include? In this article, we’ll guide you through the considerations that will help you craft effective UX benchmarks. We’ll cover how to identify tasks that align with your business objectives, measure up against the competition, ensure feasibility and repeatability, and create a seamless participant experience.

List the tasks that matter to your users

A task is one action or set of actions that a user must take to achieve a specific outcome. For example, for a fitness app, a key task could be logging a workout or setting a fitness goal. For an e-commerce site, it might be applying a discount code at checkout.

Before you can identify your top tasks, you need a comprehensive list of what users are doing, or could be doing, with your product. Established UX teams often already have an understanding of this. But if you’re starting from scratch or just beginning to chart a poorly-defined problem space, there are several ways to begin gathering this information:

• Explore your app, website, or sitemap to identify key actions users can take.

• Investigate competitor products to see what users might accomplish there.

• Review customer support tickets, app store reviews, and feedback from social media.

• Analyze web analytics data and common search queries.

You’ll likely compile a long list of potential tasks, so narrowing down is key. Unlike traditional usability testing, where participants often dedicate an hour, online UX benchmarking requires brevity. Best practices suggest you can expect around 10 to 20 minutes of engagement from remote participants, many of whom are fitting your study into busy schedules. This translates into roughly 4 to 8 tasks, if each takes about 2 or 3 minutes to complete.

To narrow down your task list, focus on two main factors: frequency and importance. What do users do most often, and what matters most to them? Keep in mind that what’s important to your users may not always align with your business priorities—that’s a separate consideration we’ll address in the next step.

If you have existing product analytics, use that data to identify frequently performed tasks. If not, consider collecting user data through methods like top-tasks surveys, stack ranking, or pairwise comparisons. These techniques allow users to weigh in on the relative importance and frequency of the actions they take with your product, giving you a clear picture of where to focus.

Select tasks that matter to the business

As UX teams, we represent the voice of our users, particularly in the ways their needs and goals overlap with our organizational goals. So it’s not enough to choose tasks that users perform frequently—the tasks should also align with what drives business success.

After narrowing your list of frequent and important tasks, the next step is to assess their value to the business. In broad terms, these are the tasks that ultimately result in either generated revenue or reduced costs. A few examples that lead to a sale might include:

• Locate pricing comparison page

• Complete a purchase

• Reach out for a quote

And here are a few others that reduce burden on customer support channels:

• Find FAQs and common problems

• Locate community support forum

• Start a conversation with a chatbot

To identify which tasks are most closely tied to business objectives, start by listing the company’s key goals and performance metrics (or KPIs). These are often shared during all-hands meetings, goal-setting sessions, or found in investor relations materials such as annual reports.

Next, for each of the tasks in your shortlist, determine which KPI it relates to, if any, and how strongly it influences that KPI. This simple method will help you prioritize the tasks that matter most to both users and the company.

Consider competitive tasks

Your product isn’t the only one your users will interact with. Competitive benchmarking places your data in the broader context of your industry, helping you see how well your experience holds up against both direct and indirect competitors.

Your competitors likely cover much of the same core functionality as your product, but they may also offer services or features you haven’t considered. Every competitor has unique value propositions, whether competing on price, features, or specific customer segments.

Most teams will begin with a baseline benchmark focused solely on their own product. Competitive benchmarking adds complexity, but it’s worth at least thinking about how to build flexibility into your study design for future iterations. Even if you’re not ready to implement competitive comparisons right away, having a plan will prepare you to answer the inevitable question from stakeholders: “How do we compare to the rest of the industry?”

While industry benchmarks are sometimes available from third-party services, they often lack the specificity needed to inform your product decisions. The most valuable data comes from directly benchmarking your competitors, using tasks and metrics tailored to your product’s unique goals. This approach will produce far more actionable findings than generic industry data.

Competitive benchmarking reveals where your competitors may excel and where you might have an advantage. More importantly, it reveals opportunities to improve the user experience that might otherwise be missed. For example, if a competitor incentivizes automatic payments with a monthly discount, that could inspire a quick solution to boost adoption of your own payment services.

By collecting this data, you empower your team to make informed, strategic decisions that improve both your product and your competitive edge.

Present tasks clearly to participants

Once you’ve prioritized the tasks, the next step is structuring their presentation in the study. This includes drafting clear task scenarios, determining start and end points, and success criteria.

In unmoderated sessions, participants are on their own—there’s no facilitator on hand to clarify instructions or troubleshoot technical issues. For this reason, stress-testing your tasks is essential. Every word matters, and instructions should be vetted by pilot testers (even, if necessary, colleagues) who are unfamiliar with the study’s goals. This makes sure the tasks are clear for all participants, even those with no prior knowledge of the product or task.

For comparison’s sake, it’s important that all participants begin and end in similar places. For example, you might direct participants to the homepage of your website (the task start) and ask them to find specific information that resides on only one page (the task end). Participants would then need to navigate the site to the task end to locate the required information.

You can then verify success in a few ways. One option is to use your testing platform to confirm that participants reached the correct URL. Alternatively, if your platform doesn’t support tracking (or the task isn’t tied to a unique URL), you can ask participants to identify the information they found in a multiple-choice question with distractors. This allows them to recognize the correct answer rather than having to write it down or rely on memory.

Whether through URL tracking or knowledge-based questions, validating success ensures an apples-to-apples comparison across participants, regardless of how they accomplish the task.

Make sure the tasks are feasible

At this point in the process, take care that your task and study design are actually possible within the constraints of your chosen study platform or tool.

For example, not all platforms allow webcam or microphone recordings for thinking aloud during the task. Even among those that do, some impose participant limits or additional fees. Being aware of these limitations early will save time and prevent headaches later on in the process.

Complex experimental designs—such as assigning participants to different conditions or segmenting users based on specific criteria—are often important for the integrity of the study. While some platforms support these designs seamlessly, others may require clunky workarounds, such as creating multiple studies. You’ll need to carefully plan for any platform limitations to avoid introducing sample bias or reducing the accuracy of your results.

Once your study is built, thoroughly test it to ensure that all logic, conditions, and task flows are working properly. For example, one common use of study logic is to ask an open-ended follow-up question only when a participant rates a task’s difficulty below a certain threshold (such as less than 6 on a 7-point scale)—an effort towards improved data quality. If you’ve built in conditions such as these, you’ll want to try each possibility and confirm the platform is recording data as intended.

By focusing on feasibility from the outset, you create a smoother, more reliable benchmarking study.

Select today’s tasks with tomorrow’s in mind

When designing your first benchmark, think beyond the immediate study. Consider the second and third benchmarks—which could take place several years from now, depending on your chosen cadence. Selecting tasks with repeatability in mind ensures that your future benchmarks will remain relevant and offer comparable data points.

Incorporating tasks for features that are still in development or slated for redesigns is another way to future-proof your benchmarks. These tasks often support broader strategic initiatives for the product’s long-term success. Examples of forward-thinking tasks might include:

• A wireless provider expanding into home internet services might test whether customers can find a waitlist to sign up for the offering. This task could become subscription-focused in future efforts.

• A financial services firm adding comparison tools for retail investors could benchmark how users currently research securities on (or even outside of) their platform. This task could examine the new research tools in later versions.

With an eye to the future from day one, you’ll make sure your benchmark results have a long shelf life and remain valuable points of comparison in future iterations.

The bottom line

Benchmarking is a powerful tool to demonstrate the long-term impact of your UX efforts, but it requires careful planning and thoughtful execution… beginning with selecting the tasks. Here are some of the key considerations to keep in mind:

• Identify top tasks: Start by understanding what your users are trying to accomplish and narrow your list based on frequency and importance.

• Align with business objectives: Prioritize tasks that directly contribute to revenue generation or cost reduction.

• Consider the competitive context: Keep in mind that your product is judged against competing alternatives, shaping user expectations and preferences.

• Focus on the participant experience: Ensure tasks are clear, easy to understand, and allow for consistent comparisons in an unmoderated environment.

• Stay focused on feasibility: Make sure your tasks and study design are compatible with the testing tools you’re using.

• Build for repeatability: Create tasks that can be repeated across future benchmarks to track long-term progress.

Benchmarking not only helps show the value of your UX work but also gives immediate insights for improving your product and staying competitive in the market. By aligning tasks with both user needs and business objectives, and ensuring repeatability, you can see how your team’s contributions improve the experience over time.

How can we better understand users’ tasks? How can we create more useful designs that support users in efficiently and effectively accomplishing their goals? 

This article will introduce you to task analysis, a powerful technique for inspecting user goals and informing design decisions early in the product design lifecycle. By the end, you’ll understand what task analysis is, when to use it in your projects, and how to do it. 

Task analysis in UX projects

Task analysis is a technique where UX practitioners break down tasks to better understand and design for user goals. “Tasks” are goal-oriented user actions with a clear start and end point that achieve a desired outcome – examples include booking a flight, buying a soda from a vending machine, canceling your meal kit delivery service, etc. “Analysis” refers to breaking a larger object into smaller elements to better understand it. So, task analysis takes tasks, defines them, and breaks them down into elements like sub-goals, and specific steps needed to accomplish the user’s end goal. 

This analysis process serves to clarify a user’s goals, define the steps they take, and evaluate how effectively an interface enables task completion. Task analysis can uncover hidden complexities even in seemingly simple tasks; below you can see an example of a task analysis for buying a soda from a vending machine. By decomposing these steps, we can begin to understand where users might make mistakes or experience issues with an interface. 

Task analysis increases our understanding of the user’s goal and is a valuable guide for informing design decisions, so it is most useful when applied early in the design process. During discovery and concept generation, UX practitioners might use a task analysis to inform design concepts or narrow down which of many concepts is the best candidate for prototyping. In the prototyping stage, task analysis helps quickly evaluate and refine designs before further testing and higher-fidelity prototyping. 

How to conduct a task analysis

Task analysis is a general approach with a family of specific methods that range in complexity. One method, Task Description (example below), is as simple as listing out the steps a user may execute to perform a task. Although this method is straightforward, its simplicity lacks depth and isn’t suited for non-linear tasks.

On the other end of the extreme, GOMS (example below) is a complex method that involves breaking tasks into goals, operators, methods, and selection rules. GOMS produces quantitative time-on-task predictions that evaluate how well an interface supports expert users in achieving their goals, but it requires significant expertise to apply correctly. 

Hierarchical task analysis (HTA) hits the Goldilocks zone; it is generally easy to learn but also provides a lot of value as a diagnostic tool for evaluating a user interface. Consequently, it is the most widely used task analysis method in UX, and we advise starting with this approach when teaching our task analysis courses.

To conduct an HTA there are five steps:

Step 1: Collect information about the task

Before starting to build your HTA, you’ll want to make sure you understand who your users are, what they are trying to do, and how they do it. You might get this information through previous primary or secondary research, direct observation, or even product instructions. In addition to collecting what you know about users and their tasks, make sure you have access to the interface that you’ll be evaluating.

Step 2: Describe the user’s goal

The next step of the HTA process is to describe the user’s goal. To do this you’ll answer: 

• What is the user trying to accomplish?

• What is their starting point? 

• What is the end point? 

After identifying the user’s goal, this is placed at the top of the hierarchy diagram. As illustrated in the example above, the user’s goal—'check savings bank account balance via mobile'—is shown at the top of the task hierarchy, labeled as 0 (HTA are often labeled with numbers for easy reference). 

Step 3: Split the user’s goal into sub-goals

The next step of hierarchical task analysis is to decompose the user’s goal into a set of smaller sub-goals. These sub-goals should be specific, actionable sequences that comprise the overall task. In the example above, for a user to successfully check their account balance, they must find the mobile app on their phone, log in, and then locate the appropriate account and balance. Splitting up the main user goal into its component sub-goals helps decompose the task and forms the second layer of the HTA.

Step 4: Break each sub-goal into a sequence of steps

After identifying sub-goals, the next step is to break each one down into a sequence of mental and physical actions required to complete it. Each step is noted below the sub-goal in the hierarchy. 

In some cases, you might encounter non-linear steps or discover multiple approaches to completing a sub-goal. In cases like this, it may be worth noting a “Plan” on your HTA. A plan simply indicates where a user might decide to perform the task in a specific way or in a different order. In the banking example here, users might be able to use Face ID to log in. We could simply represent this option using a separate branch and note the plan (below). 

Step 5: Inspect the HTA for design opportunities 

Finally, once you’ve completed decomposing the user’s task into a hierarchy, it can be inspected to uncover opportunities to improve the design. Generally, these notes fall into three categories: 

1. Errors: Making note of where errors might happen and how a new design might prevent them. Ask yourself: Where might users make a mistake? How likely is this mistake? How critical is this mistake? How might a new design prevent this mistake? 

2. Efficiency: Making note of where the design can be made more efficient for the end-user. Ask yourself: What steps seem unnecessary or redundant? Where could we make this quicker and easier? How might we design for increased efficiency? 

3. Opportunities: Making note of general design opportunities. Ask yourself: In what ways would an ideal design support a user in this task, sub-goal, or step? 

This final step of inspection is where we get the most value out of doing task analysis; identifying design improvements that we use to formulate recommendations and effectively guide redesigns. In our example below, we’ve added a note to step 3.2 that the user may fail to recognize the “Accounts” menu item — the interface we evaluated represents it with a piggy bank icon that lacks strong, established associations with accounts functionality. During the inspection step, a UX practitioner would thoroughly review the HTA several times and add many notes like these to document likely errors, efficiency issues, and design opportunities.

The bottom line

Task analysis breaks a user’s task into its sub-goals and detailed steps. This systematic approach helps UX practitioners define user goals, describe tasks, assess how well an interface supports task completion, and identify design opportunities for improved efficiency and effectiveness. Begin by integrating hierarchical task analyses early in your design process to inform, evaluate, and improve design concepts and early-stage prototypes.