> >no, not at all ;) for that case I was just curious to have a look at the > >values from training data themselves. > But your asking actually lead me to a question about cross-validation output > esp. for leave-one-out method. > I noticed the harvest_attribs option in CrossValidatedTransferError, and was > wondering whether the sensitivities or > other measures harvested there should always be used in place of those from > full dataset without cross-validation. depends on your assumptions, goals, and what kind of sensitivity is at hands ;-)
> For sensitivity particularly, CrossValidatedTransferError gives a set of sens > values for each run, and I'm not sure > how could they be summed up. In SMLR for instance, I guess there might be > tiny shifts of selected voxels in each > leave-one-out run. what kinds of shifts? in SMLR there is another tricky point -- it does feature selection, so any kind of analysis of sensitivities across splits might need to take that into account. look at our 2nd paper though: http://frontiersin.org/neuroinformatics/paper/10.3389/neuro.11/003.2009/ taking sensitivities across splits allow to judge on the significance of the values if you allow youself to consider them as independent samples of sensitivities drawn from some distribution... so you are obtaining error-margin on their mean across the splits > Would a mean across runs still be a valid sensitivity? Any suggestions on > that? anything you do is valid, if you state you prior assumptions ;-) > I found a NFoldSplitter() can be added in SMLRWeights(SMLR() ) and give > a sensitivity vector. not sure where what is added... just cut-paste source snippet > This is however apparently not the mean of the harvest_attribs one, > as the number of selected voxels are much smaller for the former. So it seems > the voxel > shift issues across CV runs is dealt with already. not clean what 'voxel shift' we are talking about? the one which we hope is addressed during preprocessing motion-correction stage? but iirc in your case you work on anatomicals, ie there is not time-sequence but different subjects, which are resliced into some common space; ofcause there is variability but... > Is this the one that should be preferred > to the one from full dataset (without cross-validation)? sensitivity mean might be more stable and less noisy imho, so depending on what your goals are once again > Is there a general form of > clf().getSensitivityAnalyzer() with a NFoldSplitter() option? kinda... iirc it is SplitFeaturewiseDatasetMeasure just look at its constructor help -- it is pretty much as simple as SplitFeaturewiseDatasetMeasure(splitter, measure) > I know I need to go to documentation / source code and read more carefully. I > guess > for now a simple hint about what you would choose /chose on this for a paper > would be > helpful enough. Thanks! you are welcome! yeah -- documentation reading session would help, but I would advise to get through both our papers (they are shortish) and them glance over the code in supplementals materials of the 2nd paper -- I bet you would feel more comfortable with pymvpa after that. > >Thanks again for your response! > >Best, Frank > > Yeah the dot_prod is not that important. I just tried to get a little more > > idea about how it works. > > Have a nice weekend! > U2 ;-) > > Best, Frank -- Yaroslav Halchenko Research Assistant, Psychology Department, Rutgers-Newark Student Ph.D. @ CS Dept. NJIT Office: (973) 353-1412 | FWD: 82823 | Fax: (973) 353-1171 101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102 WWW: http://www.linkedin.com/in/yarik _______________________________________________ Pkg-ExpPsy-PyMVPA mailing list [email protected] http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa
