and would it be possible to get predicted probabilities instead of just labeled predictions? Something similar to 'predict_prob' from SVS in scikit-learn?
On Fri, Jan 31, 2014 at 9:44 PM, Arman Eshaghi <[email protected]>wrote: > Thanks so much! > > All the best, > Arman > > > On Fri, Jan 31, 2014 at 9:22 PM, Yaroslav Halchenko <[email protected] > > wrote: > >> >> On Fri, 31 Jan 2014, Arman Eshaghi wrote: >> >> > MyData is structural MRI data coming from fmri_dataset function. >> There are >> > two chunks, and similar to clf.predictions (in tutorial), I'm >> wondering >> > whether I can get each predicted label, because I want to compare >> AUC in >> >> so each sample is a subject. ok >> cvte.stats.sets would have sets of original targets and their >> predictions per each cross-validation split. >> >> also if you set your errorfx=None I guess you would also get raw >> predictions (and possibly original targets) in your results... yeap: >> >> In [2]: cv = CrossValidation(kNN(), HalfPartitioner(attr='chunks'), >> errorfx=None, enable_ca=['stats']) >> >> In [3]: from mvpa2.testing.datasets import datasets as tdatasets >> >> In [4]: results = cv(tdatasets['uni2small']) >> >> In [5]: results >> Out[5]: <Dataset: 24x1@|S2, <sa: cvfolds,targets>> >> >> In [6]: print results.targets, results.samples >> ['L0' 'L0' 'L0' 'L0' 'L0' 'L0' 'L1' 'L1' 'L1' 'L1' 'L1' 'L1' 'L0' 'L0' >> 'L0' >> 'L0' 'L0' 'L0' 'L1' 'L1' 'L1' 'L1' 'L1' 'L1'] [['L0'] >> ['L0'] >> ['L0'] >> ['L0'] >> ['L0'] >> ['L0'] >> ['L1'] >> ['L1'] >> ['L1'] >> ['L1'] >> ['L1'] >> ['L1'] >> ['L0'] >> ['L0'] >> ['L0'] >> ['L0'] >> ['L0'] >> ['L0'] >> ['L0'] >> ['L1'] >> ['L1'] >> ['L1'] >> ['L1'] >> ['L1']] >> >> *In [8]: print cv.ca.stats.sets >> [(array(['L0', 'L0', 'L0', 'L0', 'L0', 'L0', 'L1', 'L1', 'L1', 'L1', 'L1', >> 'L1'], >> dtype='|S2'), array(['L0', 'L0', 'L0', 'L0', 'L0', 'L0', 'L1', >> 'L1', 'L1', 'L1', 'L1', >> 'L1'], >> dtype='|S2'), [{'L0': 1.0, 'L1': 1.0}, {'L0': 1.0, 'L1': 1.0}, >> {'L0': 2.0, 'L1': 0.0}, {'L0': 2.0, 'L1': 0.0}, {'L0': 2.0, 'L1': 0.0}, >> {'L0': 2.0, 'L1': 0.0}, {'L0': 0.0, 'L1': 2.0}, {'L0': 0.0, 'L1': 2.0}, >> {'L0': 0.0, 'L1': 2.0}, {'L0': 0.0, 'L1': 2.0}, {'L0': 0.0, 'L1': 2.0}, >> {'L0': 0.0, 'L1': 2.0}]), (array(['L0', 'L0', 'L0', 'L0', 'L0', 'L0', 'L1', >> 'L1', 'L1', 'L1', 'L1', >> 'L1'], >> dtype='|S2'), array(['L0', 'L0', 'L0', 'L0', 'L0', 'L0', 'L0', >> 'L1', 'L1', 'L1', 'L1', >> 'L1'], >> dtype='|S2'), [{'L0': 2.0, 'L1': 0.0}, {'L0': 2.0, 'L1': 0.0}, >> {'L0': 2.0, 'L1': 0.0}, {'L0': 2.0, 'L1': 0.0}, {'L0': 2.0, 'L1': 0.0}, >> {'L0': 2.0, 'L1': 0.0}, {'L0': 2.0, 'L1': 0.0}, {'L0': 0.0, 'L1': 2.0}, >> {'L0': 0.0, 'L1': 2.0}, {'L0': 0.0, 'L1': 2.0}, {'L0': 0.0, 'L1': 2.0}, >> {'L0': 0.0, 'L1': 2.0}])] >> >> and here are some snippets for you for AUC (you need a classifier which >> would provide estimates, not just final decisions): >> >> *In [10]: print cv.ca.stats.stats['AUC'] >> [nan, nan] >> >> *In [11]: cv = CrossValidation(SMLR(enable_ca=['estimates']), >> HalfPartitioner(attr='chunks'), errorfx=None, enable_ca=['stats']) >> >> In [12]: results = cv(tdatasets['uni2small']) >> >> In [13]: print cv.ca.stats.stats['AUC'] >> [1.0, 1.0] >> >> In [14]: tdatasets['uni2small'].samples += >> np.random.normal(size=tdatasets['uni2small'].shape)*0.5 >> >> In [15]: results = cv(tdatasets['uni2small']) >> >> In [16]: print cv.ca.stats.stats['AUC'] >> [0.81944444444444442, 0.81944444444444442] >> >> *In [17]: results = cv(tdatasets['uni4small']) >> >> In [18]: print cv.ca.stats.stats['AUC'] >> [1.0, 1.0, 1.0, 1.0] >> >> *In [19]: tdatasets['uni4small'].samples += >> np.random.normal(size=tdatasets['uni4small'].shape)*0.5 >> >> In [20]: results = cv(tdatasets['uni4small']) >> >> In [21]: print cv.ca.stats.stats['AUC'] >> [0.64814814814814814, 0.68518518518518512, 0.76388888888888884, >> 0.55092592592592593] >> >> >> -- >> Yaroslav O. Halchenko, Ph.D. >> http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org >> Senior Research Associate, Psychological and Brain Sciences Dept. >> Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 >> Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 >> WWW: http://www.linkedin.com/in/yarik >> >> _______________________________________________ >> Pkg-ExpPsy-PyMVPA mailing list >> [email protected] >> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa >> > >
_______________________________________________ Pkg-ExpPsy-PyMVPA mailing list [email protected] http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
