A t-test is possible, assuming you're doing a within-subjects analysis
(classifying every person separately). But it's not what I prefer. One
reason is that we're often on the edge of what parametric tests can
handle (number of subjects, distributions, dependencies, etc.). Another
is that a t-test isn't quite focused on what I want to know: I want to
know if the average accuracy is greater than what we'd get with random
images, which is what's tested with a well-designed permutation test.
For example, imagine your subjects had very similar accuracies just
above chance (0.52, 0.53, etc.). Under the right conditions this could
turn out as significant with a t-test, but probably shouldn't be
considered important.
As a practical matter, I sometimes calculate t-test p-values in the
early stages of analysis because they're so fast, then calculate
permutation tests for the final p-values. In some datasets the p-values
from the two methods are close, in others they've been far apart,
sometimes with the t-test p-values much less significant.
Jo
ps: always good to plot data to eyeball normality, not just run tests. :)
On 5/18/2011 4:03 PM, Vadim Axel wrote:
Thank you a lot for your advices!
I have one more related question:
How reliable in your opinion would be to test significance of the
classification using t-test vs.0.5, where my vector of classification
results contain subjects results? In other words, subject's A prediction
was 0.6, B - 0.52, C - 0.55 etc. I take all those values as an input to
t-test. The values are independent and the normality condition is also
fulfilled (I can check it using Lilie-test).
_______________________________________________
Pkg-ExpPsy-PyMVPA mailing list
[email protected]
http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa
