On 3 Mar 2000 11:36:25 -0800, [EMAIL PROTECTED] (Marie Elaine Rump)
wrote:
...
>
> We are in the middle of a study that compares 16 clinicians
> answers to a questionnaire (answers selected from 0,1,2,3) and
> would like to use weighted kappa to analyse our intra and inter
> rater results. For inter rater analysis the 16 raters produce 256
> pairings. We are looking for a some advice/program that might be
> able to help us.
< SNIP >
Are these clinicians each rating the same set of patients, and if so,
how many? The "reliability" that you will figure is going to be an
assessment over *this sample*, as is always the case.
If the clinicians are not rating patients, then I don't yet know what
your design is about; you will have to give more detail; and kappa
might be quite irrelevant.
Kappa is best for 2x2 tables, anyway. You probably should be
interested in the Pearson correlation, plus the paired t-test. See my
stats-FAQ for a bit more discussion.
What are your hypotheses? Are you looking for accuracy, or are you
looking for styles of rating? If you are looking for styles, then you
might want to do some sort of factor analysis across some key scores.
For accuracy: you will vastly reduce your complexity if you designate
a modal score, or a "gold standard" of a correct score. Then you will
have 16 comparisons instead of 120 (or, does order matter?)
--
Rich Ulrich, [EMAIL PROTECTED]
http://www.pitt.edu/~wpilib/index.html
===========================================================================
This list is open to everyone. Occasionally, less thoughtful
people send inappropriate messages. Please DO NOT COMPLAIN TO
THE POSTMASTER about these messages because the postmaster has no
way of controlling them, and excessive complaints will result in
termination of the list.
For information about this list, including information about the
problem of inappropriate messages and information about how to
unsubscribe, please see the web page at
http://jse.stat.ncsu.edu/
===========================================================================
