Workshop: How to Take a Quantitative Approach to UX Research

I’ll be teaching a one-day tutorial on “Measuring the User Experience: How to Take a Quantitative Approach to UX Research” at the UXPA 2018 conference in San Juan, Puerto Rico, on June 25, 2018. Here are some details about the tutorial:

Session Overview

The UX field is maturing, moving out of the usability labs, towards defining and implementing UX strategy. A key ingredient of any UX strategy is quantitative data. While many UX professionals have experience with qualitative research techniques, few are comfortable with these techniques. The goal of this tutorial is to provide UX professionals with a foundation for how to carry out basic quantitative research. In this tutorial we will review common, as well as lesser known UX metrics. We will discuss the advantages and limitations of different metrics, when to use (and not use) them, and how to present metrics in effective ways. Participants will have an opportunity to get their hands dirty by playing with different types of UX data, learning some statistical tests, and experimenting with different visualizations. Participants will walk away with a practical understanding of how to incorporate quantitative data into their UX toolkit.

Tutorial Outline

Introduction to Measuring the User Experience (1.5 hours)
The tutorial will begin with introductions for both the instructor and the participants, as well as laying out the agenda for the day. Following the introductions, there will be an overview discussion of UX metrics. The overview discussion will define usability metrics and the importance of using metrics as part of everyday usability research. There will also be an introduction to the different types of usability metrics, and different approaches to collecting metrics.

Performance Metrics (2 hours)
Five main types of performance metrics will be covered in this section: success rates, task time, errors, efficiency, and learnability. For each type of performance metric, we will discuss how to measure and analyze the data, and the most effective ways to present it. Also, there will be many examples, as well as a discussion of some of the strengths, limitations, and issues around using each metric. Special attention will be given to task success, since it is one of the most widely used usability metrics. Exercise: Analysis of performance data from an online study

Self-Reported Metrics (2 hours)
The goal of this discussion is to expose participants to a wide variety of ways to collect and analyze self-reported metrics (feedback coming directly from users). Several common self-reported metrics will be covered, such as: rating scales, expectation measures, standard questionnaires such as SUS, USE, QUIS, ACSI, and NPS, Microsoft’s Product Reaction Cards, and verbatim comments. For each type of self-reported metric, instructors will discuss the strengths and limitations, and how to present the data in an interesting and compelling way. Special attention will be devoted towards explaining how self-reported metrics are fundamentally different than performance metrics. Exercise: Analysis of self-reported data from an online study.

Special Topics and Other Metrics (1 hour)
Some special topics such as how to analyze card-sorting data and live-site A/B test data will be covered. Several other types of metrics will be grouped together in an “other” category. Other metrics include: derived (or combined), web navigation, and behavioral & physiological metrics. Derived metrics includes combining more than one type of metric to come up with a single usability score such as SUM. Web navigation metrics include a “Lostness” metric. Behavioral & physiological metrics are based on different kinds of observations or measurements of user behavior. We will review metrics such as eye-tracking, facial expressions, physiological responses (skin conductivity), and stress measures.

Bringing it All Together (1.5 hours)
The group exercise will focus on developing a usability research plan utilizing specific metrics. Participants will form into groups of 5 – 7 people. Each group be handed a problem statement. Each problem statement will describe a situation in which they have been asked to solve a series of usability issues for a business sponsor. They will need to come up a list of metrics they want to collect to answer their particular questions. They will need to figure out how they will collect the data and how they are going to analyze the data. Each group is going to be given a different situation. For some groups, their problem statement may focus on solving for low success and high abandonment rates. Other groups may need to solve for low satisfaction rates, while other groups may need to solve for low efficiency. After each group has developed their research plan, the instructor will lead a discussion focusing on which usability metrics were selected by each group and why. The outcome of this exercise is that participants will understand the types of questions they need to answer to come up with the optimal research plan given a variety of situations.

Intended Audience

Entry level – Appropriate for UX professionals new to the topic. Some degree of familiarity with Excel is assumed.

Goals

There are five specific goals for this course. Participants of this course will be able to:
• Describe different ways to measure the user experience.
• Identify the most appropriate metrics to use for a wide variety of situations.
• Understand the strengths and limitations of each metric covered.
• Measure and analyze a wide variety of data.
• Analyze and present UX metrics in a clear and persuasive way.

Here is pricing information for the conference, including the pre-conference courses such as this one.