Interrater reliability
Stat folks, I have an interrater reliability dilemma. We are examining a 3-item scale (each item scored 1 to 5) used to rate compliance behavior of patients. Two separate raters have used the scale to rate patients' behavior, and we now want to calculate the interrater agreement for the scale. Two problems: 1) The majority of patients are compliant, and receive either a 4 or 5 for each of the three items from both of the raters. While this is high agreement, values for ICC are very low due to the limited range of scores. Are there any indexes that would reflect the high agreement of the raters under these conditions? Perhaps something that accounts for the full range of the scale (1 to 5)? 2) The dataset contains a total of about 100 observations, but there are multiple observations on the same patients at different times, probably about 5 to 6 observations per patient. Does this repeated assessment need to be accounted for in the interrater agreement, or can each observation be treated as independent for the purpose of interrater agreement? Any suggestions or references addressing this problem would be appreciated. Thanks. Allen Cornelius
RE: Interrater reliability
Allen, You might refer to this paper. Burry-Stock, J. A., Shaw, D. G., Laurie, C., & Chissom, B. S. (1996). Rater agreement indexes for performance assessment. Educational & Psychological Measurement, 56, 251-262. Peter Chen -Original Message- From: Allen E Cornelius [SMTP:[EMAIL PROTECTED]] Sent: Wednesday, January 19, 2000 11:22 AM To: [EMAIL PROTECTED] Subject:Interrater reliability Stat folks, I have an interrater reliability dilemma. We are examining a 3-item scale (each item scored 1 to 5) used to rate compliance behavior of patients. Two separate raters have used the scale to rate patients' behavior, and we now want to calculate the interrater agreement for the scale. Two problems: 1) The majority of patients are compliant, and receive either a 4 or 5 for each of the three items from both of the raters. While this is high agreement, values for ICC are very low due to the limited range of scores. Are there any indexes that would reflect the high agreement of the raters under these conditions? Perhaps something that accounts for the full range of the scale (1 to 5)? 2) The dataset contains a total of about 100 observations, but there are multiple observations on the same patients at different times, probably about 5 to 6 observations per patient. Does this repeated assessment need to be accounted for in the interrater agreement, or can each observation be treated as independent for the purpose of interrater agreement? Any suggestions or references addressing this problem would be appreciated. Thanks. Allen Cornelius
Re: Interrater reliability
I am in the process of determining inter and intrarater reliabilities with a 70 category coding procedure with 3 different coders we are using a ³Cohen¹s kappa² to determine the reliability amongst coders (a more stringent standard ?) - have not finished yet a simple interrater agreement ratio can probably give you a higher score (number of agreements divided by total agreements + disagreements) In article <[EMAIL PROTECTED]>, [EMAIL PROTECTED] (Allen E Cornelius) wrote: > I have an interrater reliability dilemma. We are examining a 3-item >scale (each item scored 1 to 5) used to rate compliance behavior of >patients. Two separate raters have used the scale to rate patients' >behavior, and we now want to calculate the interrater agreement for the >scale. Two problems: -- Robert Kujda Purdue University Division of Art and Design - Department of Visual and Performing Arts Curriculum and Instruction CA-1 West Lafayette, IN 47907 765 494-3058 [EMAIL PROTECTED] have an Aesthetic Experience /\__/| (\ \ , , \__ _ \\_/ __ = _ o= _ o/ /o
