dear all,
thanks for your replies, danilo, richard, and nick. and sorry for the
slow reply.
@danilo: i'm using lda so it directly performs three-way classification
(i guess it's very much like nn classification on mahalanobis
distances). svm could use one of your approaches. this gave me the idea
that i could turn the three-way classification problem into 3 pair-wise
classification problems and then take the average. chance would be 50 %
again and i could compare the results accuracy in a binary classification.
@richard: ordered in the sense of scale of measurement? no, they're nominal.
@nick: yes, that is almost forever ago. still very good times though in
my opinion :)
the number of predictions are identical between the two classifications.
hm, the conversion into z-scores sounds like a good idea. so for each
participant i would convert the accuracay to a p value using number of
bernoulli trials and chance level. and the p value would give me a z
score, which i would then analyze at the group level. that does sound
straightforward.
thanks for your suggestions!
best,
michael
On 23.09.18 13:32, Danilo Bzdok RWTH wrote:
Common approaches are:
1) One-versus-rest: gives as many weight sets as classes and one overall
accuracy
2) One-versus-one: gives as many weight sets as possible pairs and one
overall accuracy
In both appeoaches, the binary classifier is applied internally to
obtain three-way classification outcomes
Whether a classifier with native capacity to distinguish three classes
is „better“ than the above schemes with a two-class-only classifier is
an epistemologically challenging question that may be hard to decide
without overfitting the dataset at hand.
Cheers,
Danilo
On Sun 23. Sep 2018 at 13:00,
<[email protected]
<mailto:[email protected]>> wrote:
Send Pkg-ExpPsy-PyMVPA mailing list submissions to
[email protected]
<mailto:[email protected]>
To subscribe or unsubscribe via the World Wide Web, visit
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
or, via email, send a message with subject or body 'help' to
[email protected]
<mailto:[email protected]>
You can reach the person managing the list at
[email protected]
<mailto:[email protected]>
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Pkg-ExpPsy-PyMVPA digest..."
Today's Topics:
1. Re: comparing accuracies of a 3-way classifier and a 2-way
classifier (Richard Dinga)
----------------------------------------------------------------------
Message: 1
Date: Sat, 22 Sep 2018 15:50:01 +0200
From: Richard Dinga <[email protected] <mailto:[email protected]>>
To: Development and support of PyMVPA
<[email protected]
<mailto:[email protected]>>
Subject: Re: [pymvpa] comparing accuracies of a 3-way classifier and a
2-way classifier
Message-ID:
<CABbjURB=bdoverfzzwphto3o7upu9-i1lg0ay2lwzkofs4x...@mail.gmail.com
<mailto:[email protected]>>
Content-Type: text/plain; charset="utf-8"
Are your 3 classes ordered?
On Fri, Sep 21, 2018, 18:28 Michael Bannert
<[email protected] <mailto:[email protected]>>
wrote:
> dear pymvpa users,
>
> i have predictions from a 3-way classification and a 2-way
> classification that i would like to compare with one another. how
could
> i do this?
>
> 1) i could subtract the chance level from each accuracy score, i.e.,
> subtract 1/3 from the 3-way classification accuracy and 1/2 from
2-way
> classification. not ideal because percentage changes above chance are
> not directly comparable anymore. but the approach is pretty intuitive
> and permutation inference against chance levels would still be valid.
>
> 2) use a different performance metric like (adjusted) mutual
information
> maybe? methodologically more appropriate probably but maybe confusing
> for the readers.
>
> 3) but perhaps there are even better ways to do this. for example
> examine the 3-by-3 and 2-by-2 confusion matrices and compare
> main-diagonal with off-diagonal entries?
>
> any other ideas?
>
> thank you,
> michael
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> [email protected]
<mailto:[email protected]>
>
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://alioth-lists.debian.net/pipermail/pkg-exppsy-pymvpa/attachments/20180922/911ee8df/attachment-0001.html>
------------------------------
Subject: Digest Footer
_______________________________________________
Pkg-ExpPsy-PyMVPA mailing list
[email protected]
<mailto:[email protected]>
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
------------------------------
End of Pkg-ExpPsy-PyMVPA Digest, Vol 125, Issue 3
*************************************************
_______________________________________________
Pkg-ExpPsy-PyMVPA mailing list
[email protected]
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
_______________________________________________
Pkg-ExpPsy-PyMVPA mailing list
[email protected]
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
