Mixed methods. User research

1.

Mixed Methods
User Research

2.

• Mixed Methods Research is defined as a type of user
research that combines qualitative and quantitative
methods into a single study. Companies like Spotify,
Airbnb and Lyft are using Mixed Methods Research to
combine rich user insights with actionable statistics for
deeper user insights.
Qualitative
Mixed
Quantitative

3.

Where has Mixed Methods User
Research come from?
• The rules of user research have changed. The
distinction between qualitative and quantitative
research is becoming blurry as research at scale
becomes more common. Fast-growing companies
are prioritising researchers fluent in both qualitative
and quantitative methods that can adapt to any
research challenge. We have entered the era of
Mixed Methods UX Research.
• Traditionally these qualitative and quantitative
research methods were viewed in opposition. Each
side would proudly stand and proclaim that their
methods were more important for the company's
success. The researcher conducted the study using
either a qual or quant research method and
interpreted the findings on their own.

4.

• Then it all started to change. The importance of user
experience became increasingly important as
companies adopted product-led and lean startup
growth strategies. Founders initially took on this
responsibility for user research. As companies grew,
this work was passed down to technically-minded
product managers. Product managers quickly
learned that big data could tell them what users
were doing but it couldn't tell them why.

5.

•Mixed Methods Research was the ideal
toolkit. Technical product teams were
already using quantitative research to
justify decisions and qualitative research
became the natural fit to inform their
quant studies. Before long the
combination of the two made Mixed
Methods Research part of the natural
vocabulary at scaling companies.
•And then we arrive at the present day.
Mixed Methods User Research has started
to find its footing outside of young
startups and is quickly emerging on its
own as a field of expertise. Let's dive in
and learn more about what Mixed
Methods Research is all about and the
many advantages it offers.

6.

What makes Mixed Methods
Research unique?
Quantitative research methods are remarkably useful for gathering hard
data to measure, validate and inform crucial decisions. This quant data is
the currency of internal decision-making in product-led companies. Stats
from this research can answer questions like 'Which feature do users like
the most?' or 'Which design do users prefer?'.
However, quantitative data will likely surface new questions that can only
be answered with qualitative research. Qual insights can help you to
understand why questions such as 'Why do my users like to use my
product?', 'Why do they churn?' and 'Why do they behave this way?'.
Additionally, qualitative research provides an opportunity for users to share
insights you never would have thought to ask about.

7.

Mixed Methods Research is
unique because the researcher
can address multiple research
objectives in one project —
understanding the pains and
motivations
driving
user
behaviour as well as the scale of
their impact on the total
userbase.

8.

Why do Product Leaders love
Mixed Method Researchers?
It is only in more recent years that user researchers
and product teams have joined the mixed methods
movement. The view on hiring user researchers
has changed since product teams have adopted
Mixed Methods Research. A researcher specialised
in a single area will require a complimentary
researcher, which many of the most ambitious
early stage companies just can't afford.
This is why product leaders love to hire
researchers that can turn their hand to any research
project at the drop of a hat. These are the people
that be called upon for any challenge. They
uniquely fit the research needs of fast-moving and
scaling companies. The demand for such people
has given rise to the Full-Stack Researcher.

9.

Why Is Mixed Methods Research
Becoming Of Rising Importance?
• The default strategy for most companies over the past decade has been a salesdriven approach that targets execs with cost saving and efficiency messaging.
The salesperson was the driver of this exec-focused strategy. As companies
adopt Product-Led Growth, their messaging shifts to focus on unmet needs and
pains of the end user. In this Product-Led Growth model, the product and UX
functions are the ones that interact with the end user and it is therefore the
responsibility of these teams to understand the user's motivation to purchase.
• Understanding unmet needs, pains and undiscovered motivations requires a
different mindset than the mindset of a salesperson or traditional market
researcher. The oldschool qualitative researcher's objective is to craft detailed
and relatable customer personas. They act reactively to the needs of the product
team and work in the world of perspective rather than numbers. In the world of
Product-Led Growth, a researcher must do so much more than that. Research
must shift to become a continuous function that isn't confined to predetermined project timelines. Building and shipping quickly is the lifeblood of
the company so researchers must be responsive and adaptive. Identifying the
core pains driving user behaviour must be proactive instead of a reactive
response to the output of big data.

10.

The need for user
researchers to be
skilled internal
networkers grows
as companies
continue to
become more
integrated crossfunctionally.
They become the
go-to resource
for everyone,
from the
marketing and
sales teams to
the product and
design functions.
Each of these
teams speak a
different
language and
value different
types of data
for decisionmaking.
The increased
focus that Mixed
Methods Research
puts on
understanding user
experience gives
researchers the
flexibility to
address the needs
of every internal
stakeholder.
However, in order
to deliver the output
of those projects,
user researchers
must become
multilingual
influencers capable
of wielding any type
of data in order to
convert internal
skeptics to the view
of the end user.

11.

The Three Research Designs You Need
To Master Mixed Methods Research
• The highly sought Full-Stack Researchers we see today have
mastered these three Mixed Methods Research designs:

12.

1. Exploratory: Qual → Quant
Exploratory Mixed Methods Research puts the
qualitative step first. This typically involves a
user interview sprint but can also take the form
of an online survey with open-ended questions.
The findings of this first step inform the
quantitative research that follows. This usually
means taking the insights from your qual step
and using them to inform a quantitative project,
such as a quant survey or unmoderated usability
testing. The results of your quant step help you
to quantify the significance of your qualitative
insights.
This approach is great when you are working on
a research problem with a lot of unknowns and
you don't know what you should ask in your
quant study. Qualitative insights will help you to
form a hypothesis which can then be validated
or invalidated during the quantitative step.

13.

Exploratory Mixed Methods - Example
In 2018, Alison was tasked with conducting research for Slingshot, an
application that helps competitive foosball players to improve how they
practice. To kick off her research, Alison needed to get a basic
understanding of the opinions and perspectives that foosball players held.
This led her to conduct exploratory interviews with foosball players. It
was through these interviews that Alison was first able to understand the
frustrations that players felt.
After discovering some of the common foosball frustrations, Alison felt
like she had gained enough insights to develop the product features that
would be most useful for players. However, rather than jumping in and
building features based on anecdotal evidence from a handful of
interviews, she instead brought these insights to a quantitative study to
measure their validity and to see how much impact each feature would
really have on the player's practice.
By using mixed methods, Alison was able to distinguish between
consensus insights and those that were held only by a minority of users.
As a result, Slingshot was able to prioritise features that were meaningful
to the most amount of players. Following this exploratory mixed methods
process meant the team could avoid technical debt by only building
solutions that would add the most value for customers.

14.

2. Explanatory: Quant → Qual
Explanatory user research starts with collecting and
analysing quantitative data. This step often starts by
looking at user product behaviour at scale using big data
tools like Segment or MixPanel. You can then bring
your insights to a qualitative deep dive to learn more
about the context behind the figures through focus
groups, in-depth interviews or online community
discussions. Explanatory research integrates all of your
findings for a broader and deeper understanding of your
users.
This research design is especially useful when there is a
need to explain and interpret the quantitative findings.

15.

Explanatory Mixed Methods - Example
• Prior to Blend, Clement was working at a CRM startup called Movoto that struggled with
high user churn. Clement noticed that the most engaged users interacted with the app very
differently than the disengaged cohort. He decided to segment the user base by
engagement levels to pinpoint the specific events and actions that were indicators of which
users were the most and least engaged.
• Clement started with quantitative research. He analysed the user base to determine their
top quartile of "highly engaged" users (with a median login rate of 31.7 times per month)
and their bottom quartile of "highly disengaged" users (with a median login rate of 4.9
times per month). Comparing these two groups, they found that highly engaged users
regularly reassigned leads to teammates, rejected leads and set vacations in the app.
Clement was very surprised that highly engaged users would reassign leads to their
teammates — this insight directly contradicted their hypothesis of what a highly engaged
user was.
• Confused by these results, Clement's team turned to qualitative user research to find the
root cause behind the behaviors they saw in their analysis. Through 1:1 interviews with
highly engaged and highly disengaged users, they unearthed a range of insights that they
used to improve their UX. They increased the visibility of key features, introduced new
user views to make it easier for highly engaged users to perform their most important tasks
and launched an experiment to algorithmically determine CRM lead value for their users.
These changes increased median logins from 10.6 times per month to 18.3 times per
month, drove a reduced churn rate and created an increased referral rate — boosting both
top-line revenue and bottom-line profits.

16.

17.

3. Dynamic: Qual + Quant
Dynamic research methods blend qualitative and quantitative data together at the same
time within one research study. Unlike exploratory and explanatory research, an entire
Mixed Methods Research project can be carried out in just one step using a dynamic
mixed methods research tool.
A number of core principles define dynamic user research methods. They are
fundamentally user-driven. By putting the user at the centre, the researcher takes more of
an observation role. This means that dynamic research is often unmoderated or reactively
controlled so that participants can shape the evolving data themselves. Dynamic research
is particularly suited for teams that are using a Product-Led Growth strategy as the user's
ability to share their experience is prioritised over the researcher's interference.
The most common method of dynamic research is unmoderated usability testing at scale.
At an individual level, this is inherently qualitative and observational. However at scale,
quantitative data begins to surface trends in user behaviour.

18.

Dynamic Mixed Methods Example
• When COVID first hit back in March 2020, a group of Irish
entrepreneurs launched a nonprofit called Feed the Heroes to
provide a way for ordinary people stuck at home to support
frontline workers. The project was a rapid success and reached
over €1M in donations, mainly from individuals, within a few
weeks. The Feed the Heroes team were keen to understand what
supporters' core motivations were for donating to their cause.
• The Feed the Heroes team had drafted a traditional multiple
choice survey but they felt that a significant amount of people
would end up falling into the "Other" box. They also worried that
they would end up missing out on some really rich insights from
passionate donors that way. Instead, they launched an OpinionX
survey. They added over 30 interesting sample opinions that were
directly from donors tweets, emails and testimonials to kickstart
the survey. The final thing to do was to share their survey link
with their 17,000-strong donor base.