Hi Jo and all other experts, Thanks for your reply! Indeed that was the concern for me. For my first try, I simply transformed the accuracy by normalization taking into account of the trail numbers. For example, if the accuracy is 70%, I would convert it by *(0.7-0.5)/sqrt(0.5*0.5/n)*, where n = the number of total testing examples.
I hope this is valid? I tried the Lee 2012 mean correction method, and yields very similar results. Have not tried the Logit mixed models yet, though. Best, Frances On Mon, May 11, 2015 at 10:04 AM, J.A. Etzel <[email protected]> wrote: > To inject a slightly different topic: while I and many others have done > t-tests on accuracies (equivalently, error rates) for group-level analyses > and gotten sensible-looking results, this is probably not ideal: accuracies > are bounded by 0 and 1, which violates the assumptions of t-tests (and > similar statistics). > > Logit mixed models may be a better parametric test for the group level. > I've found some readable introductions to these issues in the linguistics > literature, such as this one: > > http://dx.doi.org/10.1016%2Fj.jml.2007.11.007 > Categorical Data Analysis: Away from ANOVAs (transformation or not) and > towards Logit Mixed Models > T. Florian Jaeger J Mem Lang. 2008 Nov; 59(4): 434–446. > > I haven't yet started playing with these for MVPA datasets; anyone tried? > > I don't want to imply that logit mixed models (or t-tests) are better than > permutation methods; permutation-based tests are almost certainly > preferable, particularly with cross-validated statistics. However, it is > sometimes useful to have a quick parametric statistic as well; but this > should perhaps not be a t-test. > > Jo > > > > > On 5/9/2015 11:49 AM, Jingwen Jin wrote: > >> >> Hi MVPA experts, >> >> I have a general question about conducting group-level analysis on the >> subjects' classification accuracy maps. Let's say I am doing a >> one-sample t-test to find the voxels that have high classification >> accuracy across subjects. Essentially, I am doing a t-test on percentage >> numbers (SVM classification accuracy measured as percentage correct). >> Since percentage is highly affected by the testing example numbers, and >> in general would probably not meet the normal distribution assumption >> for t-test. >> >> So my question is if people adjust for testing trial numbers or any sort >> of transformation? For example, I converted each voxel's percentage >> number to a z score at the individual subject's classification map >> level, and then do group-level t-test on these z score maps. I wonder if >> this is valid? >> >> Thank you very much! >> >> Best, >> Frances >> -- >> Jingwen Frances Jin >> Department of Psychology, PhD candidate in Clinical area >> Stony Brook University >> >> >> _______________________________________________ >> Pkg-ExpPsy-PyMVPA mailing list >> [email protected] >> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa >> >> > _______________________________________________ > Pkg-ExpPsy-PyMVPA mailing list > [email protected] > http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa > -- Jingwen Frances Jin Department of Psychology Stony Brook University
_______________________________________________ Pkg-ExpPsy-PyMVPA mailing list [email protected] http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
