I think that it could be a registration problem or a normalization/detrending problem. I used to have below-chance accuracy in across-subject analyses, but this is not your case!
Bests, R On Mon, 26 Nov 2018 at 16:29, Raúl Hernández <r...@lafuentelab.org> wrote: > Thank you for the link, I will look into it carefully. > Sorry for not being clear, yes I have 4 acquisitions from each > participant. I calculate an accuracy for each participant by calculating > the mean across all cross validation folds. Then I take the this calculated > mean from each participant and run a t test in which each participant > contributes with a single number. > > Regards, > > Raul > > On Mon, Nov 26, 2018 at 4:13 PM Etzel, Jo <jet...@wustl.edu> wrote: > >> I agree with Patil that consistent below-chance accuracy is a sign that >> something is not working properly. >> >> I collected some thoughts in >> http://mvpa.blogspot.com/2013/04/below-chance-classification-accuracy.html >> (and a few other posts tagged "below-chance"). >> >> Also, be careful with terminology; I assume by "leave-one-run-out >> cross-validation on 4 acquisitions" you mean each person completed four >> scanning runs (each with the same fMRI acquisition parameters)? And a >> t-test can be fine for a quick significance test, but it should be done >> at the group level, testing if the subjects' accuracies are above chance >> (i.e., each person contributing one number to the t-test), not on the >> cross-validation folds within each person. >> >> Jo >> >> >> On 11/26/2018 7:05 AM, Raúl Hernández wrote: >> > I also consider that option, but when I try the very same thing with a >> > different, region (not related to the task). I get accuracies of 50%. >> So >> > that makes me think that it is affected by the task, but I don't know >> > what to think of it. >> > >> > Regards >> > >> > On Mon, Nov 26, 2018 at 1:34 PM Kaustubh Patil < >> kaustubh.pa...@gmail.com >> > <mailto:kaustubh.pa...@gmail.com>> wrote: >> > >> > I suspect that there might be something wrong in the code/how the >> > data is handled. >> > >> > If you 30% accuracy then that would mean that you will get 70% if >> > you use a simple rule to predict the "other class" after your >> > classifier. This is a sign that something is not right in data >> > handling/evaluation. >> > >> > Best >> > >> > On Mon, Nov 26, 2018 at 1:27 PM Raúl Hernández < >> r...@lafuentelab.org >> > <mailto:r...@lafuentelab.org>> wrote: >> > >> > No, it is balanced. It has the same number of observations for >> > each class. >> > >> > On Mon, Nov 26, 2018 at 12:52 PM Kaustubh Patil >> > <kaustubh.pa...@gmail.com <mailto:kaustubh.pa...@gmail.com>> >> wrote: >> > >> > Just for clarification. >> > >> > Is that data imbalanced, i.e. many more observations from >> > one class? >> > >> > Best, >> > Kaustubh >> > >> > On Mon, Nov 26, 2018 at 12:50 PM Raúl Hernández >> > <r...@lafuentelab.org <mailto:r...@lafuentelab.org>> wrote: >> > >> > Dear PyMVPA community, >> > >> > I'm doing classification in ROI's, I'm performing a >> > simple 2 way classification using LSVM, and a >> > leave-one-run-out cross-validation on 4 acquisitions. On >> > some ROI's, I get a good accuracy for the number of >> > participants (60%), but in others I get consistently bad >> > accuracy (30%). To test whether the performance is above >> > chance, I use a one sample t test (I know that it is not >> > the best test for this type of data, I just use it as >> > quick overview). When I test the bad accuracies, those >> > are also significant. >> > >> > What does it mean a consistently bad accuracy? >> > >> > Regards, >> > >> > Raul >> > _______________________________________________ >> > Pkg-ExpPsy-PyMVPA mailing list >> > Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net >> > <mailto:Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net> >> > >> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa >> > >> > _______________________________________________ >> > Pkg-ExpPsy-PyMVPA mailing list >> > Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net >> > <mailto:Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net> >> > >> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa >> > >> > _______________________________________________ >> > Pkg-ExpPsy-PyMVPA mailing list >> > Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net >> > <mailto:Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net> >> > >> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa >> > >> > _______________________________________________ >> > Pkg-ExpPsy-PyMVPA mailing list >> > Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net >> > <mailto:Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net> >> > >> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa >> > >> > >> > _______________________________________________ >> > Pkg-ExpPsy-PyMVPA mailing list >> > Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net >> > >> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa >> > >> _______________________________________________ >> Pkg-ExpPsy-PyMVPA mailing list >> Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net >> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa > > _______________________________________________ > Pkg-ExpPsy-PyMVPA mailing list > Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net > https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa -- Ing. Roberto Guidotti, PhD. PostDoc Fellow Institute for Advanced Biomedical Technologies - ITAB Department of Neuroscience and Imaging University of Chieti "G. D'Annunzio" Via dei Vestini, 33 66013 Chieti, Italy tel: +39 0871 3556919 e-mail: r.guido...@unich.it; rguido...@acm.org linkedin: http://it.linkedin.com/in/robertogui/ twitter: @robbisg github: https://github.com/robbisg
_______________________________________________ Pkg-ExpPsy-PyMVPA mailing list Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa