Hi Francisco, Great that you followed up -- could you please clarify for me if indeed in one of your publications you did some power/ROC analysis of such permutation scheme (keeping testing set labels assignment) against "classical" (permute all independent assignments)? I have vague memory that you did but I could be wrong.
NB A will argue with Michael in reply to his post ;) On Tue, 05 Feb 2013, Francisco Pereira wrote: > I'm catching up with this long thread and all I can say is I fully > concur with Michael, in particular: > On Tue, Feb 5, 2013 at 3:11 AM, Michael Hanke <[email protected]> wrote: > > Why are we doing permutation analysis? Because we want to know how > > likely it is to observe a specific prediction performance on a > > particular dataset under the H0 hypothesis, i.e. how good can a > > classifier get at predicting our empirical data when the training > > did not contain the signal of interest -- aka chance performance. > Permuting the test set might make sense, perhaps, if you wanted to > make a statement about the result variability over all possible test > sets of that size if H0 was true. -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik _______________________________________________ Pkg-ExpPsy-PyMVPA mailing list [email protected] http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
