Tom Tullis
User Experience Magazine: Volume 7, Issue 3, 2008
Publication year: 2008

In our user experience team at Fidelity Investments, we’ve conducted over forty unmoderated remote usability tests over the past five years. We use them as an adjunct to traditional lab tests and remote, moderated usability tests. We’ve found that unmoderated remote tests reveal usability variations between different design solutions that typical lab tests generally don’t detect. The advantage of the unmoderated remote tests lies in the sheer number of participants. We usually have at least 500 participants in just a few days when we can use our own employees as participants in these tests, and it’s not uncommon to have over 1,000 participants. When performing evaluations with panels of our customers, we commonly have at least 200 participants in a week. These numbers provide tremendous data. We routinely get statistically significant differences in task completion rates, task times, and subjective ratings when comparing alternative designs. Even what appears to be a minor design difference (e.g., a different phrase to describe a single link on a website) can yield significant differences in usability measures.

Leave a Reply

Your email address will not be published. Required fields are marked *