如何进行BENCHMARK研究

Select competitors to benchmark against
Define target audience
Identity common site activities/tasks
•
Conduct research using task-based testing
•
Use the quantitative data to test for statistically significant differences between means or proportions
•
Research at least twice a year ( or even quarterly) to measure design changes and trends in expectations
•
Use qualitative data to support your findings and understand the why
1
UX research methods & tools
2
Recruiting participants
3
4 must_dos with UX Benchmarking
Plan ahead
1
Conduct
2
Gather
3
Again
4
• • • •
Define measurable & comparable UX-related KPIs
Tasks
Pre Task Post Tasks
Go through the entire process of booking a flight
Tasks Locate the total price for checking in two bags
Post-task
Wrap Up
Self-reported success
Likelihood to recommend (Net Promoter Score )
Participant Profile
Participants were required to book their travel online in order to qualify. Age Range 14% 8% 31% How often do you fly yearly(for business or leisure)?
2.Against Competitors within your industry;
What is UX Benchmarking?
•
The process of comparing a website’s (or any digital product) performance, specifically in relation to UX metrics, including among others effectiveness, efficiency, overall satisfaction and quality perception, to another that is widely considered to be an industry standard benchmark or best practice.
14% 27% 27% 31%
-27% 0% -30% -10%
42% 26% 44% 36%
10% 40% 24% 22%
-32% +14% -20% -14%
Task 1: Find a round-trip flight
Your task: Find a round-up flight departing May 1st and returning May 6th that you would consider booking on American Airlines/Delta Air Lines website. You are traveling from San Francisco, CA to New York, NY. After that, please go through the entire process of booking a flight until you get to the ‘Purchase’ button.
How satisfied were you with the process of booking a flight?
Ease of Use Post-tasks
Ease of navigation
Cisfaction with amount of time to complete task
Date:April 24 - April 25, 2012
Panel
100 participants(50 per site)
Must fly for business or leisure more than twice a year
Must use websites to make travel arrangements
Study background
Study
PANEL
Competitive benchmark of American Airlines VS. Delta Air Lines websites
Using task-based surveys, or Remote Unmoderated Usability Testing
Study design
AA
Task 1
Task 2
Screener
Pre-task q’s
Follow-up q’s
DELTA
Task 1
Task 2
• • • • •
Large sample of participants take the study simultaneously
Natural context
Panel source: Survey Sampling International(SSI) STUDY TASKS
Tasks
Go through the entire process of booking a flight
Locate the total price for checking in two bags
More competition, more maturity in the online marketplace;
You can’t really manage what you can’t measure. UX Pros must deliver more measured value. MGMT teams LOVE metrics
American Airlines
Task Success 6%
Delta Airlines
Task Success 4%
94%
96%
Not Success Success
Median Success Time
3:43
Median Success Time
3:56
On both studies the majority of participants self-reported being able to successfully complete the booking process. There was not significant difference in success rate or completion time(90% Confidence Level).
UX research methods & tools
• Remote Unmoderated Usability Testing (or task-based survey tools)
imperium Keynote web effective
• Remote Unmoderated Usability Testing (or task-based survey tools)
2.Private Mailing List;
3.Intercept visitors;
4.Invite relatives or friends;
5.Colleagues;
6.Wechat Official Accounts;
7.UI;
Case study
• Benchmark Study to compare vs
In order to Benchmark, first you need to MEASURE。
Jeff Sauro
Quantifying
the User Experience
Measuring U
3 pillars of UX Benchmarking
3 PILLARS
Plans & Goals
Task 1: Find a round-trip flight
Post-Task Questionnaire
Top Box Scores*
How satisfied were you with the process of booking the flights?
AA
55%
DELTA
50%
Booking the flight was easy
10 + 5 - 10
27%
20%
2-4 1
Gender
0
17.5
35
52.5
70
45% 55%
Brand Perception
AA
Pre
Friendly Customer Focused Safe Good Valve