I think I would consider using generalizability theory for this problem.
Shavelson and Webb have a good book out on the subject, published by Sage.
On Thu, 27 Apr 2000, Robert McGrath wrote:
> I am looking for a formula for kappa that applies for very special
> circumstances:
>
> 1) Two raters rated each event, but the raters varied across event.
> 2) The study involved 100 subjects, each of whom generated app. 17 events,
> so multiple events were generated by the same subject.
>
> I know Fleiss has developed a formula for kappa that allows for multiple
> sets of raters, but is there a formula that is appropriate for the
> circumstance I have described? Thanks for your help!
>
> Bob
>
> -
>
> Robert McGrath, Ph.D.
> School of Psychology T110A
> Fairleigh Dickinson University, Teaneck NJ 07666
> voice: 201-692-2445 fax: 201-692-2304
>
> - Original Message -
> From: "Bob Wheeler" <[EMAIL PROTECTED]>
> To: <[EMAIL PROTECTED]>
> Sent: Thursday, April 27, 2000 3:15 PM
> Subject: Sample size and distributions programs
>
>
> > I have uploaded two programs that some may find of
> > use:
> >
> > (1) Tables. A Windows program written quite a few
> > years ago. It treats 42 distributions extensively
> > including plots and technical documentation.
> > (2) SSize. A sample size program for the Palm
> > devices. It treats linear models for several
> > distributions: normal, binomial, Poisson, and
> > chi-squared. ANOVA, t-tests, logistic, etc. There
> > is a fairly extensive documentation in pdf format.
> > This is a new program, so there are undoubtedly
> > bugs.
> > I would greatly appreciate hearing about them.
> >
> > They are at http://www.bobwheeler.com/stat/
> >
> >
> > --
> > Bob Wheeler --- (Reply to: [EMAIL PROTECTED])
> > ECHIP, Inc.
> >
> >
> >
> ===
> > This list is open to everyone. Occasionally, less thoughtful
> > people send inappropriate messages. Please DO NOT COMPLAIN TO
> > THE POSTMASTER about these messages because the postmaster has no
> > way of controlling them, and excessive complaints will result in
> > termination of the list.
> >
> > For information about this list, including information about the
> > problem of inappropriate messages and information about how to
> > unsubscribe, please see the web page at
> > http://jse.stat.ncsu.edu/
> >
> ===
> >
>
>
>
> ===
> This list is open to everyone. Occasionally, less thoughtful
> people send inappropriate messages. Please DO NOT COMPLAIN TO
> THE POSTMASTER about these messages because the postmaster has no
> way of controlling them, and excessive complaints will result in
> termination of the list.
>
> For information about this list, including information about the
> problem of inappropriate messages and information about how to
> unsubscribe, please see the web page at
> http://jse.stat.ncsu.edu/
> ===
>
===
This list is open to everyone. Occasionally, less thoughtful
people send inappropriate messages. Please DO NOT COMPLAIN TO
THE POSTMASTER about these messages because the postmaster has no
way of controlling them, and excessive complaints will result in
termination of the list.
For information about this list, including information about the
problem of inappropriate messages and information about how to
unsubscribe, please see the web page at
http://jse.stat.ncsu.edu/
===