On Aug 14, 2014, at 12:45 AM, Hanson, Gavin Keith <[email protected]> wrote:
> I’m just wondering if anyone has any advice on some ways to deal with > evaluating classifier performance on a 4-way problem. > I’ve been using the BayesConfusionHypothesis tool which works quite well, but > I just wondered what else was out there by way of quantitative evidence to > insure that classifier accuracy isn’t being driven by perfect classification > between 2 labels, and confusion between the other 2, or whatever. Just > glancing at the confusion matrices can give us a good idea about what ROIs > are confusing certain conditions, but a more objective solution would be > nice. I don't think there's much else out there besides the BCH tool. If you have a specific hypothesis to test, e.g. effects of classification of classes w, x, y, z is driven by perfect classification between w and x only, you can do a contrast at each subject. If A(p,q) is the classification accuracy between classes p and q, then compute, for example, c=A(w,x) - (A(w,y)+A(w,z)+A(x,y)+A(x,z)+A(y,z))/5. You can use standard group-analysis (random effect) analyses to see if this is reliable across participants. _______________________________________________ Pkg-ExpPsy-PyMVPA mailing list [email protected] http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
