Thank you a lot for your advices! I have one more related question: How reliable in your opinion would be to test significance of the classification using t-test vs.0.5, where my vector of classification results contain subjects results? In other words, subject's A prediction was 0.6, B - 0.52, C - 0.55 etc. I take all those values as an input to t-test. The values are independent and the normality condition is also fulfilled (I can check it using Lilie-test).
On Wed, May 18, 2011 at 10:54 PM, Yaroslav Halchenko <[email protected]>wrote: > > On Wed, 18 May 2011, J.A. Etzel wrote: > > I agree; I would be worried if the *middle* of the permutation > > distribution was around 0.6, but a wide distribution such that 0.6 is in > > the top 0.05 can happen. > > yeap -- and even could be heavy tails at 0.7 and 0.8 and even 0.9 -- > everything depends, especially on number of trials ;) > > > >permute truly independent (must be in the correct design) items: > > >sequences of trials across runs: i.e. take sequence of labels from > > >run 1, and place it into run X, and so across all runs. That should > > >account for possible inter-trial dependencies within runs, and thus I > > >would expect that distribution would get even slightly wider (than if > > >permuted within each run) > > Not sure I follow ... you mean taking the order of trials from one > > run and copying it to another, then partitioning on the runs? > > I guess "yes", if "partitioning on the runs" means "splitting into > training and testing sets for cross-validation". > > > >please correct me if I am wrong -- under permutation of samples > > >labels, those must differ regardless of block structure, simple due > > >to the change of number of trials (just compare binomial > > >distributions for 2 trials vs 4 ;) ) > > Yes, the change in the variance of the permutation distribution > > could be just from the smaller number of samples. But I can imagine > > setting up dodgy classifications of individual trials from block > > designs that could also make the permutation distributions change > > (not that Vadim did that!), so wanted to mention double-checking the > > not-averaged partitioning scheme. > > yeap ;) > > -- > =------------------------------------------------------------------= > Keep in touch www.onerussian.com > Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic > > _______________________________________________ > Pkg-ExpPsy-PyMVPA mailing list > [email protected] > http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa >
_______________________________________________ Pkg-ExpPsy-PyMVPA mailing list [email protected] http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa
