Hi All: I am not a mathematician but I have conducted many usability tests. Sometimes clients have demanded large samples in tests that have spanned multiple days. In my experience, this was not productive. I generally felt that I learned everything I could from the first 6 or so users.
I've thought a lot about why this might be and would like to offer the following thoughts... Generally in usability testing (at least in formative testing) we are not looking for statistical significance. Rather we are looking for problems to address. We don't particularly care, for example, if 40% vs. 60% of users make a particular error -- what is important is that we are seeing that a problem exists so we can address it. As a designer, I benefit most from the qualitative aspects of usability testing. Often, I find the metrics less useful. Though they do play well with management though. As I practice them, usability tests are deep structured interviews during which I can observe behaviors against a controlled set of tasks and really learn a lot about the user's mental models and where they clash with the design. With this perspective I learn a lot from 6 users and usually test 8-10 just to make certain. But by the end of the tests I am hearing the same things over and over again. Similar debates have been part of social science for a long time. Much scientific research is statistical (nomothetic) and relies on finding the shared characteristics of a group. This is great for assessing the outcomes of treatments but does not generate a lot of in-depth information. The other alternative is the case-study approach (ideographic research) which probes individuals in-depth. I suspect that a lot of metrically-inclined people will disagree with me but I find that thinking of usability testing as case-studies yields the most information. I might take a different position for a summative test whose purpose is to demonstrate the usability of an entire product and not as a design tool. Best, Charlie ============================ Charles B. Kreitzberg, Ph.D. CEO, Cognetics Corporation ============================ -----Original Message----- From: discuss-boun...@lists.interactiondesigners.com [mailto:discuss-boun...@lists.interactiondesigners.com] On Behalf Of Chauncey Wilson Sent: Thursday, October 01, 2009 4:17 PM To: Chris Ryan Cc: disc...@ixda.org Subject: Re: [IxDA Discuss] Article on Number of Usability Test Participants Laura Faulkner has written a reasoned article on sample size. You can find a copy at: http://www.geocities.com/faulknerusability/Faulkner_BRMIC_Vol35.pdf The number of participants issue depends on a number of issues including the risk inherent in the product, the number of distinct user groups, whether you are using the sample in many rounds of iterative evaliuation designed to filter out problems over the course of the design cycle (formative versus summative), the complexity of the UI, the number of paths possible, ..... If you look in the ACM Digital Library, you will find a number of articles related to the number of participants. Chauncey On Thu, Oct 1, 2009 at 2:05 PM, Chris Ryan <chris.r...@visioncritical.com> wrote: > I have been looking, unsuccessfully, through back issues of interactions magazine for an article, published a few years back, written I believe by someone from Microsoft as part of a debate about statistical significance in usability testing. There was something of a debate about testing with large numbers of users, and this article, as I recall, made an eloquent case for sticking to six to eight participants. Does anyone remember this? Perhaps I'm wrong in recalling that it was in interactions. > ________________________________________________________________ > Welcome to the Interaction Design Association (IxDA)! > To post to this list ....... disc...@ixda.org > Unsubscribe ................ http://www.ixda.org/unsubscribe > List Guidelines ............ http://www.ixda.org/guidelines > List Help .................. http://www.ixda.org/help > ________________________________________________________________ Welcome to the Interaction Design Association (IxDA)! To post to this list ....... disc...@ixda.org Unsubscribe ................ http://www.ixda.org/unsubscribe List Guidelines ............ http://www.ixda.org/guidelines List Help .................. http://www.ixda.org/help ________________________________________________________________ Welcome to the Interaction Design Association (IxDA)! To post to this list ....... disc...@ixda.org Unsubscribe ................ http://www.ixda.org/unsubscribe List Guidelines ............ http://www.ixda.org/guidelines List Help .................. http://www.ixda.org/help
