Duration: 6 months
Overview
Peter Drucker a founding father of modern management studies once commented that, “If you can’t measure it, you can’t improve it”. This principle is particularly relevant to user experience (UX). Effective measurement of UX is crucial for assessing how a product is performing. At Redgate Software, I led a 6-month initiative to improve product UX metrics and measurement.
Empowering teams to be more data informed
Product metrics and measurement is a broad subject. It was therefore important to define the focus and goals of the project. The goals were agreed as:
Create a shared definition of what great UX means for Redgate products, along with mechanisms for designers and product teams to easily measure, benchmark and report the user experience (UX) for their products.
The ultimate aim was to empower product teams to take more data informed product decisions by equipping them with richer UX related information and metrics. Key outputs of the project were:
- A framework for measuring and benchmarking the UX of Redgate products.
- Guidance and tooling to capture quantitative and qualitative UX related data.
- Richer UX related information and metrics for product teams e.g. Dashboards.
Kicking off the initiative
I planned and ran an in-person half day workshop to kick off the project. The workshop was important in order to build a shared understanding of the goals of the project, to identify desired outputs, to identify key stakeholders and to agree an initial backlog of activities. To ensure that stakeholders who were unable to attend the kick off had good visibility of what was discussed I shared the outputs, along with agreed actions.
Reviewing UX frameworks
To help explore existing frameworks we could build on I organised and contributed to a review of existing UX frameworks, including the Google HEART framework and BASIC UX framework. This highlighted some common themes, such as the importance of considering usability, effectiveness and satisfaction.
Defining key UX measures
Following the review, I led a number of iterations to define the UX measures to use (see below). Key UX measures were:
- Effectiveness for completing tasks
- Ease of use
- Emotional response
These key UX measures were mapped out for each stage of the user’s product journey: onboarding, adoption & retention and advocacy.
Defining UX measures for each product
Having agreed a set of UX measures I led the identification of key measurements to be captured for each Redgate product. In addition to the UX measures, I used the Google HEART framework to help teams to consider potential measurements to capture.
Starting with onboarding
To help determine the more product specific metrics to capture (such as signals that a product had been successfully set-up) I worked with product teams to create an onboarding experience map for their product. Onboarding was chosen as an initial focus for measuring because it’s the first stage of the user’s journey using a product, and because it’s key for driving a good overall user experience. I ran a number of workshops to review and validate onboarding experience maps with key stakeholders who work very closely with customers, such as the customer success team and support team at Redgate.
Capturing in-product feedback
It’s important to not just capture metrics within a product, but also direct feedback from users. This helps to build a good understanding of not only what a user is doing, but their experience doing it as well. To help explore different mechanisms for capturing in-product feedback I co-ordinated and contributed to a review of well used in-product feedback design patterns. This included patterns, such as footers, pop-ups and dialog boxes.
I created mock ups for different design patterns to capture in-product feedback and reviewed these with product designers and product managers.
A/B testing different feedback designs
A/B testing was carried out to find out which design pattern led to more user feedback. For example, a footer design was A/B tested with a modal dialogue.
Results
The project has led to a much better understanding of the user experience delivered by Redgate products. Adding more product usage metrics, along with in-product feedback (such as satisfaction ratings as show below) have allowed product teams to better understand the user experience delivered by their products, and to identify areas for improvements. Product teams can take more data informed decisions and can see the impact their actions have on measurements such as satisfaction and NPS.
Learnings
- Everyone loves data! – Access to dashboards and satisfaction scores allowed teams to see how their efforts impacted the product, fostering a more data-driven culture.
- Measure what happens, and why – In-product feedback provided context to the quantitative metrics, giving deeper insights into user experience.
- Measurement is too often an afterthought – Retrofitting measurement frameworks is time-consuming and challenging. Defining metrics early on means that product decisions can be data informed from the start.
- Don’t try to measure everything – The initial approach to UX measurement included many potential metrics, but not all were relevant or actionable. Focusing on the most important aspects of UX helped to identify the key interactions and events to measure.