Common approaches are: 1) One-versus-rest: gives as many weight sets as classes and one overall accuracy 2) One-versus-one: gives as many weight sets as possible pairs and one overall accuracy
In both appeoaches, the binary classifier is applied internally to obtain three-way classification outcomes Whether a classifier with native capacity to distinguish three classes is „better“ than the above schemes with a two-class-only classifier is an epistemologically challenging question that may be hard to decide without overfitting the dataset at hand. Cheers, Danilo On Sun 23. Sep 2018 at 13:00, < [email protected]> wrote: > Send Pkg-ExpPsy-PyMVPA mailing list submissions to > [email protected] > > To subscribe or unsubscribe via the World Wide Web, visit > > https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa > > or, via email, send a message with subject or body 'help' to > [email protected] > > You can reach the person managing the list at > [email protected] > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Pkg-ExpPsy-PyMVPA digest..." > > > Today's Topics: > > 1. Re: comparing accuracies of a 3-way classifier and a 2-way > classifier (Richard Dinga) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 22 Sep 2018 15:50:01 +0200 > From: Richard Dinga <[email protected]> > To: Development and support of PyMVPA > <[email protected]> > Subject: Re: [pymvpa] comparing accuracies of a 3-way classifier and a > 2-way classifier > Message-ID: > <CABbjURB= > [email protected]> > Content-Type: text/plain; charset="utf-8" > > Are your 3 classes ordered? > > On Fri, Sep 21, 2018, 18:28 Michael Bannert <[email protected]> > wrote: > > > dear pymvpa users, > > > > i have predictions from a 3-way classification and a 2-way > > classification that i would like to compare with one another. how could > > i do this? > > > > 1) i could subtract the chance level from each accuracy score, i.e., > > subtract 1/3 from the 3-way classification accuracy and 1/2 from 2-way > > classification. not ideal because percentage changes above chance are > > not directly comparable anymore. but the approach is pretty intuitive > > and permutation inference against chance levels would still be valid. > > > > 2) use a different performance metric like (adjusted) mutual information > > maybe? methodologically more appropriate probably but maybe confusing > > for the readers. > > > > 3) but perhaps there are even better ways to do this. for example > > examine the 3-by-3 and 2-by-2 confusion matrices and compare > > main-diagonal with off-diagonal entries? > > > > any other ideas? > > > > thank you, > > michael > > > > _______________________________________________ > > Pkg-ExpPsy-PyMVPA mailing list > > [email protected] > > > https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://alioth-lists.debian.net/pipermail/pkg-exppsy-pymvpa/attachments/20180922/911ee8df/attachment-0001.html > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Pkg-ExpPsy-PyMVPA mailing list > [email protected] > https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa > > ------------------------------ > > End of Pkg-ExpPsy-PyMVPA Digest, Vol 125, Issue 3 > ************************************************* >
_______________________________________________ Pkg-ExpPsy-PyMVPA mailing list [email protected] https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
