Thank you for the link, I will look into it carefully. Sorry for not being clear, yes I have 4 acquisitions from each participant. I calculate an accuracy for each participant by calculating the mean across all cross validation folds. Then I take the this calculated mean from each participant and run a t test in which each participant contributes with a single number.
Regards, Raul On Mon, Nov 26, 2018 at 4:13 PM Etzel, Jo <[email protected]> wrote: > I agree with Patil that consistent below-chance accuracy is a sign that > something is not working properly. > > I collected some thoughts in > http://mvpa.blogspot.com/2013/04/below-chance-classification-accuracy.html > (and a few other posts tagged "below-chance"). > > Also, be careful with terminology; I assume by "leave-one-run-out > cross-validation on 4 acquisitions" you mean each person completed four > scanning runs (each with the same fMRI acquisition parameters)? And a > t-test can be fine for a quick significance test, but it should be done > at the group level, testing if the subjects' accuracies are above chance > (i.e., each person contributing one number to the t-test), not on the > cross-validation folds within each person. > > Jo > > > On 11/26/2018 7:05 AM, Raúl Hernández wrote: > > I also consider that option, but when I try the very same thing with a > > different, region (not related to the task). I get accuracies of 50%. So > > that makes me think that it is affected by the task, but I don't know > > what to think of it. > > > > Regards > > > > On Mon, Nov 26, 2018 at 1:34 PM Kaustubh Patil <[email protected] > > <mailto:[email protected]>> wrote: > > > > I suspect that there might be something wrong in the code/how the > > data is handled. > > > > If you 30% accuracy then that would mean that you will get 70% if > > you use a simple rule to predict the "other class" after your > > classifier. This is a sign that something is not right in data > > handling/evaluation. > > > > Best > > > > On Mon, Nov 26, 2018 at 1:27 PM Raúl Hernández <[email protected] > > <mailto:[email protected]>> wrote: > > > > No, it is balanced. It has the same number of observations for > > each class. > > > > On Mon, Nov 26, 2018 at 12:52 PM Kaustubh Patil > > <[email protected] <mailto:[email protected]>> > wrote: > > > > Just for clarification. > > > > Is that data imbalanced, i.e. many more observations from > > one class? > > > > Best, > > Kaustubh > > > > On Mon, Nov 26, 2018 at 12:50 PM Raúl Hernández > > <[email protected] <mailto:[email protected]>> wrote: > > > > Dear PyMVPA community, > > > > I'm doing classification in ROI's, I'm performing a > > simple 2 way classification using LSVM, and a > > leave-one-run-out cross-validation on 4 acquisitions. On > > some ROI's, I get a good accuracy for the number of > > participants (60%), but in others I get consistently bad > > accuracy (30%). To test whether the performance is above > > chance, I use a one sample t test (I know that it is not > > the best test for this type of data, I just use it as > > quick overview). When I test the bad accuracies, those > > are also significant. > > > > What does it mean a consistently bad accuracy? > > > > Regards, > > > > Raul > > _______________________________________________ > > Pkg-ExpPsy-PyMVPA mailing list > > [email protected] > > <mailto:[email protected]> > > > https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa > > > > _______________________________________________ > > Pkg-ExpPsy-PyMVPA mailing list > > [email protected] > > <mailto:[email protected]> > > > https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa > > > > _______________________________________________ > > Pkg-ExpPsy-PyMVPA mailing list > > [email protected] > > <mailto:[email protected]> > > > https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa > > > > _______________________________________________ > > Pkg-ExpPsy-PyMVPA mailing list > > [email protected] > > <mailto:[email protected]> > > > https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa > > > > > > _______________________________________________ > > Pkg-ExpPsy-PyMVPA mailing list > > [email protected] > > > https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa > > > _______________________________________________ > Pkg-ExpPsy-PyMVPA mailing list > [email protected] > https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
_______________________________________________ Pkg-ExpPsy-PyMVPA mailing list [email protected] https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
