Attached an output of one such pathological case (completely real data). I even do not talk about splits. It's sufficient that for half subjects you have 0.8/0.4 prediction and for other half you have 0.4/0.8...
For more than two classes it really becomes hardly maintainable. I recently had 5 classes experiment and I ended up reporting each classes separately. On Mon, May 23, 2011 at 11:01 PM, Yaroslav Halchenko <deb...@onerussian.com>wrote: > it is a good thesis indeed, especially for the case of multiclass > classification where people make claims about unraveling complex > categorical structure, whenever it is only few categories which get > "significantly" well classified. > > And your illustration goes even further than your verbal description -- > at first I thought that there is an error, since I expected at least one > class to be significant when "average" accuracy becomes significant. > But indeed it might be not the case, e.g. if a classifier favors one > class over another across splits, thus none of the classes come out with > a consistently "significant" performance while mean accuracy does (could > you check if that is indeed the case by looking on per split > diagonals?). Cool. I always thought that digging in the mud is > very entertaining ;) > > On Mon, 23 May 2011, Vadim Axel wrote: > > > Attached an illustration for my thesis. > > The average classification rate can be considered significant, while we > > clearly see that it is not exactly true... > > -- > .-. > =------------------------------ /v\ ----------------------------= > Keep in touch // \\ (yoh@|www.)onerussian.com > Yaroslav Halchenko /( )\ ICQ#: 60653192 > Linux User ^^-^^ [175555] > > > > _______________________________________________ > Pkg-ExpPsy-PyMVPA mailing list > Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org > http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa >
yaroslav.xls
Description: MS-Excel spreadsheet
_______________________________________________ Pkg-ExpPsy-PyMVPA mailing list Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa