On Thu, 22 Mar 2001 08:23:54 -0500, Bruce Weaver
<[EMAIL PROTECTED]> wrote:
>
> On 21 Mar 2001, Awahab El-Naggar wrote:
>
> > Dear Colleagues
> > I have been using "test-retest" method for calculating reliability by
> > applying the Pearson Product Moment (PPM) analysis. However, I have been
> > told that this not the right way to calculate reliability, and I should use
> > the ANOVA to calculate the reliability. Would you comment and advise me.
> > Many Thanks.
> > A'Wahab
> >
>
> Here are a couple sites that may provide some useful information:
>
> http://www.nyu.edu/acf/socsci/Docs/correlate.html
> http://www.nyu.edu/acf/socsci/Docs/intracls.html
Awahab,
= what is in your data =
If you want to know what you have in your data, you were doing it the
right way. To be complete, you do want to look at the paired *t-test*
to check for systematic differences; and you want to confirm that the
variances are not too different. If you have multiple raters, you
usually want to know about oddities for any single rater.
You can find other comments about reliability in my stats-FAQ.
= publishing a single number =
If you want to publish a simple, single number, then editors have been
trained to ask for an IntraClass Correlation (ICC) of some sort.
The ICC reference Bruce W. cites above tells how SPSS now offers
10 different ICCs, following some over-used, much-cited studies.
The most common ICC (between two raters) does a simple job
of confounding the Pearson correlation with the mean difference
(by assuming the means are equal), instead of inviting you look at
those two dimensions separately. It can look pretty good, even when
a t-test would give you a warning. That's why I think of an ICC
as a summary that "only an editor can love."
Once you have confirmed that you have good reliability, then you
might want to do the ANOVA to get the ICC that an editor wants.
But a wise editor or reviewer should be pleased with suitable reports
of Pearson r and tests of means.
= ICC for special purposes =
I have seen a study planned, where 3-rater estimates of some X
would be used, in order to increase the precision of X and reduce
the number of cases. The estimate of the eventual sample size
used one particular species of ICC, from the many that are possible.
That's the legitimate reason for computing a special ICC (however,
I do have doubts about its accuracy). Over the last thirty years, I
remember seeing that once.
--
Rich Ulrich, [EMAIL PROTECTED]
http://www.pitt.edu/~wpilib/index.html
=================================================================
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=================================================================
