What follows in this message and a few others is some discussion regarding a supervised method of evaluation that was used in the sense induction task and proposed in the paper mentioned below.
This is not what we do in SenseClusters, but it raises some interesting issues. Agirre E., Lopez de Lacalle Lekuona O., Martinez D., Soroa A. 2006. Two graph-based algorithms for state-of-the-art WSD. Procceedings of EMNLP 2006. http://ixa.si.ehu.es/Ixa/Argitalpenak/Artikuluak/1149260582/publikoak/emnlp.pdf These messages are forwarded here as the discussion is at a pretty intuitive level and might be useful. I'll eventually try and summarize the differences between this supervised method of evaluation and what we do in SenseClusters and then the more classical methods of evaluating clustering like purity and entropy... Read on....understanding that you are jumping into the middle of things here. The discussion is between me and Aitor, one of the task organizers. Ted ---------- Forwarded message ---------- From: Aitor Soroa Etxabe <[EMAIL PROTECTED]> Date: Apr 20, 3:17 am Subject: results tables question / question on supervisedscoring To: senseinduction On 2007/04/19, [EMAIL PROTECTED] wrote : > Greetings Aitor, > Me again. :) ;-) > [...] > Anyway, my understanding is that the results of clustering on the > training data are stored in a matrix, essentially a confusion matrix. > Suppose the true senses as shown by the gold standard are S1, S2, and > S3, and that we discover 3 clusters, C1, C2, C3. > C1 C2 C3 > S1 0 10 5 > S2 10 5 5 > S3 5 5 5 > [...] > Now, I think these counts are converted into probabilities.... > C1 C2 C3 > S1 0 .66 .33 > S2 .5 .25 .25 > S3 .33 .33 .33 Yes, your analysis is right. This is the way to create what we call a "mapping matrix" Now, suppose the system returned a cluster C2 to an instance of the test corpus. We interpret this assigment by means of what we call a "cluster score vector", which in this case will be Csv = (0, 1, 0)^{T}. So, to obtain the sense we multiply the mapping matrix with the cluster score vector, which gives a "sense score vector" Ssv: Ssv = M*Csv And then we choose the sense with maximum score. In case of ties, we take one sense arbitrarily (but ties don't occur very frequently). In this case, Ssv = (.66, .25, .33)^{T}, so we choose S1. Note that this procedure allows assigning more than a cluster to an instance (like a soft clustering). Suppose we assign the cluster score vector Csv = (.9, .3, .6)^{T}, i.e., C1 has a weight of 0.9, C2 has 0.3 and C3 0.6. Multiplying it with the matrix, we obtain Ssv = (0.396, 0.675, 0.505) so sense S2 will be assigned. I hope the explanation helps understanding the sup. evaluation, but if you have more questions feel free to ask on the list. best, aitor -- Ted Pedersen http://www.d.umn.edu/~tpederse ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ senseclusters-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/senseclusters-users
