Five Mixed Methods Study Designs You Need to Know
A Mixed Methods Primer and 5 Practical Study Designs for UX Research
It is common for researchers to start their careers specializing in either qualitative or quantitative methods. Through education and training, they may develop proficiency with quantitative usability metrics, data, and statistics; or cultivate deep experience with ethnography, interviews, and thematic analyses. As a researcher's career progresses, they typically develop complementary skills, with the end goal of being a “mixed methods” researcher.
What exactly is mixed methods research? When should you consider a mixed methods approach? How can you get started? This issue will brief you on mixed methods, review practical considerations, and provide five mixed methods study designs you can apply to your research.
What you need to know about mixed methods
Mixed methods (MM) is the combination of multiple methods, including qualitative and quantitative, to address the research question(s) related to a single project.
“Mixed methods research is defined as research in which the investigator collects and analyzes data, integrates the findings, and draws inferences using both qualitative and quantitative approaches or methods in a single study or program of inquiry.”
There are a few reasons why a practitioner may opt for a mixed methods study design (MMSD).
First, MMSDs are particularly effective at addressing complex research questions that a single method cannot answer. While a mixed methods design may be excessive for a simple evaluative study (e.g., identifying the usability issues with a prototype, or evaluating whether a button should say ‘Checkout’ or ‘Continue’), it can be an excellent approach for discovery projects (e.g., understanding the attitudes, behaviors, and unmet needs of a new user group to prioritize design efforts of new products or features) and validating bigger bets (e.g., large-scale changes to an established experience, like a complete menu overhaul, or rebranding).
Second, mixed methods balance the limitations of any one study type. Each method has its pros & cons. By leveraging MMSDs, researchers can balance each method’s limitations.
Finally, mixed methods approaches provide the opportunity to triangulate data. If multiple sources of information point to the same answer, we have strong evidence to support our findings and can be more confident in the insights we report.
It is important to remember that mixed methods approaches are not a one-size-fits-all solution for research projects. To determine whether this approach suits a project, you may want to consider two factors.
Foremost is feasibility. Do you have access to the resources you need to conduct each method? Before committing to a mixed methods approach, the savvy UXR will ensure they have the time, tools, people, and participants to accomplish their plans. While mixed methods study designs are not always resource-intensive, it is crucial to account for each method in the planning phase. In short, it might not always be pragmatic to conduct multiple methods.
Second, while MMSDs offer the benefit of triangulation and convergent findings, they also have the potential for conflicting results. Complex questions resist simple answers, and multiple data sources may appear to tell different stories. You should be prepared to spend time reconciling discrepancies and communicate nuanced findings in an easily understandable way for your stakeholders.
When deciding whether or not to use a mixed methods design in your next research project, it might be useful to ask yourself these three questions:
Does a single method fully address my research questions?
Is there a benefit and added insight from another source of data?
Have I considered all the logistics?
Components of mixed methods study designs
So, how does one mix methods? Apart from selecting the appropriate qualitative and quantitative methods, there are four key components to mixed methods study designs:
Priority - Relatively speaking, is qualitative or quantitative information the main focus? Or are they equally important?
Timing - When will the qualitative and quantitative methods happen? Should they run simultaneously or sequentially to build upon previous findings?
Integration Point - When and where will the qualitative and quantitative methods come together? The integration point is where you mix your qualitative and quantitative studies. Integration may happen during data collection (i.e., qual & quant data are collected together), analysis (i.e., you integrate your qual & quant data sets and analyze them in tandem), or during interpretation (i.e., qual & quant data are analyzed separately, then findings are compared).
Preplanned vs. Emergent - How much of the study design should be preplanned? Not every aspect of mixed methods study designs needs to be predetermined; it might be beneficial to leave room for flexibility. For example, you could preplan a sequence of methods, like an interview followed by a survey. In this case, you may benefit from determining the topic and aim of the survey after interview themes are gathered.
The five mixed methods designs you should know
Fortunately, you do not need to create a mixed methods design from scratch; there are several established study designs. Think of these as templates for conducting mixed methods studies.
Here are five mixed methods designs that you can put into your UX research practice:
1. Convergent Parallel
The most simple and obvious mixed methods design. In the convergent parallel design, you conduct qualitative and quantitative methods separately but simultaneously. Then, you integrate their data and findings afterward.
This design is simple and effective. Consider using this approach if running both studies at once is of value and if the qualitative and quantitative data are of relatively the same importance to you.
An example applied to UXR:
Conducting a quantitative tree test to determine which items are difficult to find on your website, and at the same time analyze customer support tickets to identify qualitative trends in what users are searching for.
2. Explanatory Sequential
The explanatory sequential design involves a sequence of quantitative first, and qualitative second.
First, data is collected and analyzed from a quantitative study. Then data is collected and analyzed from a qualitative study. Finally, the quantitative results are revisited in light of the new information obtained from the follow-up qualitative study. This approach is effective at providing additional context and explanation for the results of a quantitative study.
An example applied to UXR:
An unmoderated quantitative usability benchmark of ten key user journeys, followed by a moderated usability study focused on the 2-3 key journeys with the lowest completion rate to explain their low scores.
3. Exploratory Sequential
The exploratory sequential design is essentially the opposite of the explanatory sequential design.
Qualitative data is collected, analyzed, and followed by a quantitative study and analysis. The quantitative study can be used to assess if the qualitative themes generalize to larger samples, estimate frequencies, and identify trends. This study design is ideal for exploratory projects, where the goal is to understand users and their needs, and prioritize which issues to address.
An example applied to UXR:
Semi-structured interviews to discover users’ unmet needs and pain points, followed by a survey with questions about the frequency & severity of those pain points.
4. The Embedded Design
In the embedded design both qualitative and quantitative data are collected in the same protocol. Integration happens during or immediately after data collection.
There are various reasons why you might use an embedded design. For instance, it is pragmatic when you have limited access to users or time constraints (e.g., a contextual inquiry with only one day for data collection). Additionally, this method works well for teams of researchers (e.g., one researcher can focus on quant while the other takes qual).
An example applied to UXR:
An on-site contextual inquiry with two researchers. One focuses on qualitative observations (e.g., how the users accomplish their tasks, whether they use any shortcuts, etc.), and the other records quantitative data (e.g., how long things take, how frequently users engage in certain behaviors, etc.).
5. Multiphase Design
The multiphase design involves combining three or more phases of research, with at least one qualitative and one quantitative study, conducted over a period of time to achieve an overall project objective.
The multiphase design is effective if one or more of the following conditions are present:
Single-method studies or other mixed methods study designs are not sufficient for addressing your research questions.
Sequential learning is needed (i.e., results of one study inform subsequent studies).
You have multiple objectives that each independently require a sequence of studies.
An example applied to UXR:
Tasked with a mobile app redesign, you might plan a sequence of research that includes two tracks
One track of studies for improving the usability of current functionality: a baseline benchmark → qualitative usability testing redesigns → A/B test redesign vs. old design ahead of launch
A second track of methods for identifying opportunities for new functionality: interviews with current users to identify unmet needs→ KANO survey to test and prioritize new feature ideas → rapid iterative usability testing new feature prototypes
The bottom line…
Mixed methods research combines multiple methods, including at least one qualitative and quantitative, to address research question(s) related to a single project. These study designs, when selectively applied and carefully planned, are especially powerful at addressing complex research objectives. By understanding and effectively applying these mixed methods approaches, UXRs can enhance their research capabilities and elevate the insights of their strategic projects.
See you at UXPA
We're a little less than a month away from UXPA 2023. Drop me a line if you'll be in Austin!
I'm looking forward to presenting with
. We'll provide findings and takeaways from our recent discovery research on UXR hiring and interview processes.Until next time.
Cheers,
Thomas