In UI/UX Design
A story about one qualitative unmoderated use research
Principal UX Consultant
What do you do when you run a research study, and you're not really sure what the results tell you? I saw it happen and the experience really stuck with me.
A long time ago i worked with a team that wanted to test a navigation prototype and their theory was that, if they tested with lots of users, they would get better statistical significance on the test and get a clear winner.
So they set up a quantitative test that was unmoderated with hundreds of users, and they decided: "we don't need any qualitative information because it wouldn't be feasible at that scale".
When they were done with the test, neither nav A nor nav B was a clear winner. Then they started digging down into the tree data of the navigation test to see what was happening, but users were just splintered all over the place, and it was hard to really tell or make any conclusions from that test.
Instead of reducing noise, the high volume of sessions in the test actually just created more confusion, and it left them wondering a lot of "why" which they never did ask the user.
Ask "why"
What did they learn in this test? Well, more data is great, but don't forget to ask your users "why", especially if you have a lot of uncertainty in your hypotheses.
If you're going to run unmoderated quantitative usability tests, consider platforms that let you do a hybrid, where you can also get user commentary or, maybe, run two stages where you start with something moderated, and then you verify some of your results with quantitative tests.
Starting unmoderated tests
Second lesson is that unmoderated tests are really hard to adapt mid-study if you find anything unexpected. That's another reason why, if you have any uncertainty about your hypotheses, it might be good just to start with a moderated test, and then start to verify later with larger volume unmoderated tests.
When you do decide to pursue an unmoderated quantitative test, be sure to give yourself plenty of time to finesse and stress test the prototype, because if something goes wrong during the test, it's really hard to shift.