Hi Nynke, Could you help me about the question that you have asked?
“How could I perform the permutation test for each searchlight and let the searchlight output also the p values for testing against the null distribution?” I want to do the same thing and have you solved this? Best! Xiangzhen On Tue, Aug 2, 2011 at 8:00 PM, < [email protected]> wrote: > Send Pkg-ExpPsy-PyMVPA mailing list submissions to > [email protected] > > To subscribe or unsubscribe via the World Wide Web, visit > > http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa > > or, via email, send a message with subject or body 'help' to > [email protected] > > You can reach the person managing the list at > [email protected] > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Pkg-ExpPsy-PyMVPA digest..." > > > Today's Topics: > > 1. Searchlight + cross validation & significance (permutation > test) (Nynke van der Laan) > 2. Re: Searchlight + cross validation & significance > (permutation test) (Francisco Pereira) > 3. Re: Searchlight + cross validation & significance > (permutation test) (J.A. Etzel) > 4. Searchlight + cross validation & significance (permutation > test) (Nynke van der Laan) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 1 Aug 2011 16:34:03 +0200 > From: Nynke van der Laan <[email protected]> > To: [email protected] > Subject: [pymvpa] Searchlight + cross validation & significance > (permutation test) > Message-ID: > <CADHpHBOUFyc4nfboeYjBOngHry=m8jv5cmg0w_75cwq6ncr...@mail.gmail.com > > > Content-Type: text/plain; charset=ISO-8859-1 > > Hello all, > > I have a question about the following: I am performing a searchlight > analysis (v04) with a crossvalidation (leave one subject out, 20 > samples per subject, 2 labels (10 samples label A, 10 samples label > B)). > I have adapted the example scripts to generate a map with the mean > classification error over de NFold crossvalidation. Now, in addition I > would like to be able to say something about the significance of the > accuracies that the analysis yielded. > In the module reference > ( > http://v04.pymvpa.org/modref/mvpa.measures.searchlight.html#module-measures-searchlight > ) > I noticed the null_prob and null_T are available variables. Null_prob > is enabled as default. However, the analysis now only results in the > searchlight errormap. How can I acces the null_prob information? Is > this averaged over the NFolds? > > cv = CrossValidatedTransferError( > TransferError(LinearCSVMC()), > NFoldSplitter()) > sl = Searchlight(cv, radius=6) > sl_map = sl(dataset_total) > > An additional question I have is whether there is any easy way to > combine a searchlight analysis as described above with a permutation > test. In a previous discussion I read about the NullhypothesisTest > ( > http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2008q2/000031.html > ) > but this does not seem to be present in the v.04 version anymore (?). > I predict that this analysis will be very costly in time as the > permutation is done in each searchlight, but I wouldn't know any > good/valid alternatives. Does anyone else have a suggestion? > > Thanks a lot! > > Best, > Nynke > > > > ------------------------------ > > Message: 2 > Date: Mon, 1 Aug 2011 10:47:35 -0400 > From: Francisco Pereira <[email protected]> > To: Development and support pf PyMVPA > <[email protected]> > Subject: Re: [pymvpa] Searchlight + cross validation & significance > (permutation test) > Message-ID: > <cafdubx0hlplfkftoqkzcg7vizjuq0ecai7hhnpkhxxptwt8...@mail.gmail.com > > > Content-Type: text/plain; charset=ISO-8859-1 > > On Mon, Aug 1, 2011 at 10:34 AM, Nynke van der Laan > <[email protected]> wrote: > > > > An additional question I have is whether there is any easy way to > > combine a searchlight analysis as described above with a permutation > test. > > For a particular classifier (GNB) it's possible to do this fast > enough, if I recall correctly this is the PyMVPA implementation > > http://www.pymvpa.org/generated/mvpa.measures.gnbsearchlight.html > > cheers, > Francisco > > > > ------------------------------ > > Message: 3 > Date: Mon, 01 Aug 2011 09:58:19 -0500 > From: "J.A. Etzel" <[email protected]> > To: [email protected] > Subject: Re: [pymvpa] Searchlight + cross validation & significance > (permutation test) > Message-ID: <[email protected]> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > My usual strategy is to split up the permutations and run the jobs on a > cluster; with luck your university will have access to one. Then you can > have dozens of computers working on the job at the same time, for > example one node doing permutations 1 to 5, another 6 to 10, etc. > > Jo > > > > On 8/1/2011 9:47 AM, Francisco Pereira wrote: > > On Mon, Aug 1, 2011 at 10:34 AM, Nynke van der Laan > > <[email protected]> wrote: > >> > >> An additional question I have is whether there is any easy way to > >> combine a searchlight analysis as described above with a permutation > test. > > > > For a particular classifier (GNB) it's possible to do this fast > > enough, if I recall correctly this is the PyMVPA implementation > > > > http://www.pymvpa.org/generated/mvpa.measures.gnbsearchlight.html > > > > cheers, > > Francisco > > > > _______________________________________________ > > Pkg-ExpPsy-PyMVPA mailing list > > [email protected] > > > http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa > > > > ------------------------------ > > Message: 4 > Date: Tue, 2 Aug 2011 11:55:21 +0200 > From: Nynke van der Laan <[email protected]> > To: [email protected] > Subject: [pymvpa] Searchlight + cross validation & significance > (permutation test) > Message-ID: > <CADHpHBPh=6jy0bsyobzi08avy0i0+ixgwb4btox7yf84sen...@mail.gmail.com > > > Content-Type: text/plain; charset=ISO-8859-1 > > Thanks for your replies! > Francisco, great suggestion, I will absolutely consider that > classifier for my next study. For the present study/additional > analysis though, I would like to stick with the v4.0 version and the > linear SVM as this showed good results (and I preferably don't switch > to a different version in between analyses/revisions). > Jo, using multiple computers is indeed a good option, > > However, in the documentation I read that if there is a strong > assumptiom of the data being normally/gaussian distributed > (http://v04.pymvpa.org/examples/permutation_test.html), 30 > permutations would also suffice. As my data indeed approaches a normal > distribution, I would like to try this first. But, unfortunately I did > not manage to implement this into the searchlight. In the example, it > is clearly described how this permutation test would be done in a > 'normal' analysis, but how exactly to combine this with the > searchlight is unclear to me. I assume the null_dist = MCNullDist... > should be included in the TransferError in the script below, but the > searchlight now only returns the error scores for each voxel and no > other results. How could I perform the permutation test for each > searchlight and let the searchlight output also the p values for > testing against the null distribution? > > cv = CrossValidatedTransferError( > TransferError(LinearCSVMC()), > NFoldSplitter()) > sl = Searchlight(cv, radius=6) > sl_map = sl(dataset_total) > > > > ------------------------------ > > _______________________________________________ > Pkg-ExpPsy-PyMVPA mailing list > [email protected] > http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa > > End of Pkg-ExpPsy-PyMVPA Digest, Vol 42, Issue 1 > ************************************************ > -- ----------------------------------------------------------------------------------------- Kong Xiangzhen State Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University, Beijing, China, 100875.
_______________________________________________ Pkg-ExpPsy-PyMVPA mailing list [email protected] http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
