Dear Experts, besides reporting accuracies for whole-brain binary classification I would like show some 'pattern maps'. One might go with the classical searchlight approach, but I personally don't like the idea that the pattern map derived from searchlight is not really driving the whole-brain classifier accuracy (comparing searchlight maps and sensitivity maps showed some (more or less) considerable differences in my case). One big advantage of the searchlight (besides 'smooth' maps) is the possibility to calculate z-/t-maps and apply statistical-testing quite easy.
So, the question is how to handle the sensitivity maps. There may be a lot of different approaches (e.g. bin weights and to calculate overlap over subjects, bootstrap as suggested by Pereira et al. (2009), etc.). I would like to go with the following procedure and I would like to ask your advice if this might be valid. The Idea is to calculate a sensitivity map plus an amount of sensitivity maps with permuted labels like in MCS. The p-value would be the number of permutations where clf weight is greater than or equal to observed clf weight (respectively = max((number_of samples>alpha)/number_of_permutations, 1/number_of_permutations as Yarik pointed out here: http://groups.google.com/group/mvpa-toolbox/browse_thread/thread/1059ece7da64e360) . My questions are: 1. Would it be valid to use MCS with clf weights? 2. Would this approach be also valid for sparse classifiers (like e.g. SMLR, ENET etc.)? Due to the sparsity a lot of features get values of zero which might be not "strict" enough for H0-testing. 3. Would it be necessary (for any reason I don't see yet) to L1/L2 norm the pattern weights over permutations? 4. Would it be valid to use absolute weights for MCS for clfs where high positive & low negative weights have huge influence on the decision (I don't think so)? 5. To deal with the issue of multiple comparison I would go with FDR since clustering might ignore the nature of this multivariate pattern approach. Do you agree? I would really appreciate to hear your thoughts on this, Matthias _____________________________________________ Matthias Ekman, Dipl.-Psych. Max Planck Institute for Neurological Research Gleueler Strasse 50 50931 Cologne, Germany E-mail: [email protected] Phone: +49-(0)221-4726-255 _____________________________________________ _______________________________________________ Pkg-ExpPsy-PyMVPA mailing list [email protected] http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa
