Re: Standardized Confidence Intervals
On 15 Oct 2001 07:44:33 -0700, [EMAIL PROTECTED] (Warren) wrote: Dear group, It seems to me that the one issue here is that when we measure something, then that measure should have some meaning that is relevant to the study hypotheses. And that meaning should be interpretable so that the width of the CI does have meaning...why would you want to estimate the mean if it is meaningless? This reminds me that data analysts sometimes can help to make outcomes more transparent. Scaled scores of Likert-like items are intelligible when presented as Item-averages, instead of being Scale-totals for varying numbers of items. You can describe the relevant anchors for 2.5 versus 3.0, for group contrasts, change scores, or contrasts between different scales -- and not get into confusion of how many items were added for each total. I do think that standardized outcomes are usually appropriate to communicate the size of the outcome effect -- even when the measure is fairly well known. This discussion leads me to conclude that you absolutely need to describe your sample *more* thoroughly when you are stuck with using standardized measures to describe all your outcomes. How variable is this group? How extreme is this group and sample (if it is a clinical trial)? A small N is especially problematic, since you do want to show how narrowly (or not) it was selected. -- Rich Ulrich, [EMAIL PROTECTED] http://www.pitt.edu/~wpilib/index.html = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Standardized Confidence Intervals
In article [EMAIL PROTECTED], dennis roberts [EMAIL PROTECTED] wrote: At 03:45 PM 10/9/01 -0400, Wuensch, Karl L wrote: Some of those who think that estimation of the size of effects is more important than the testing of a nil hypothesis of no effect argue that we would be better served by reporting a confidence interval for the size of the effect. Such confidence intervals are, in my experience, most often reported in terms of the original unit of measure for the variable involved. When the unit of measure is arbitrary, those who are interested in estimating the size of effects suggest that we do so with standardized estimates. It seems to me that it would be useful to present confidence intervals in standardized units. why? you only get further away from the original data scale/units you are working with ... in what sense ... is ANY effect size indicator anything BUT arbitrary? i don't see how trying to standardize it ... or any confidence interval ... makes it anything other than still being in arbitrary units ... i would argue that whatever the scale is you start off using ... that is as CLOSE as you can get to the real data ... even if the scale does not have any natural or intuitive kind of meaning standardizing an arbitrary variable does NOT make it more meaningful ... just like converting raw data to a z score scale does NOT make the data more meaningful standardizing a variable may have useful properties but, imputing more meaning into the raw data is not one of them Furthermore, standardizing on almost anything makes it impossible to compare different populations, where the location and scale, or other relevant parameters, may be different. It also makes asymptotic theory much more complicated, as the effects of non-normality are usually much greater if the standardization is used. -- This address is for information only. I do not claim that these views are those of the Statistics Department or of Purdue University. Herman Rubin, Dept. of Statistics, Purdue Univ., West Lafayette IN47907-1399 [EMAIL PROTECTED] Phone: (765)494-6054 FAX: (765)494-0558 = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
Re: Standardized Confidence Intervals
At 03:45 PM 10/9/01 -0400, Wuensch, Karl L wrote: Some of those who think that estimation of the size of effects is more important than the testing of a nil hypothesis of no effect argue that we would be better served by reporting a confidence interval for the size of the effect. Such confidence intervals are, in my experience, most often reported in terms of the original unit of measure for the variable involved. When the unit of measure is arbitrary, those who are interested in estimating the size of effects suggest that we do so with standardized estimates. It seems to me that it would be useful to present confidence intervals in standardized units. why? you only get further away from the original data scale/units you are working with ... in what sense ... is ANY effect size indicator anything BUT arbitrary? i don't see how trying to standardize it ... or any confidence interval ... makes it anything other than still being in arbitrary units ... i would argue that whatever the scale is you start off using ... that is as CLOSE as you can get to the real data ... even if the scale does not have any natural or intuitive kind of meaning standardizing an arbitrary variable does NOT make it more meaningful ... just like converting raw data to a z score scale does NOT make the data more meaningful standardizing a variable may have useful properties but, imputing more meaning into the raw data is not one of them == dennis roberts, penn state university educational psychology, 8148632401 http://roberts.ed.psu.edu/users/droberts/drober~1.htm = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
RE: Standardized Confidence Intervals
Title: RE: Standardized Confidence Intervals Dennis..yes, the effect size index may be arbitrary, but for argument sake, say I have a measure of 'self-esteem', a 10 item measure (each item a 5-pt. Likert scale) that has a range of 10-50; sample1 has a 95% CI of [23, 27] whereas a comparison sample2 has CI of [22, 29]. Thus, by maintaining the CI in its own unit of measurement, we can observe that there is more error/wider interval for sample1 than sample2 (for now assuming equal 'n' for each sample). However, it is problematic, given the inherent subjectivity of measuring self-esteem, to claim what is too wide of an interval for this type of phenomenon. How do we know, especially with self-report measures, where indeed the scaling may be arbitrary, if the margin of error is of concern? It would seem that by standardizing the CI, as Karl suggests, then we may be able to get a better grasp of the dimensions of error...at least I know the differences between .25 SD vs. 1.00 SD in terms of magnitude..or is this just a stretch?!!! Dale N. Glaser, Ph.D. Pacific Science Engineering Group 6310 Greenwich Drive; Suite 200 San Diego, CA 92122 Phone: (858) 535-1661 Fax: (858) 535-1665 http://www.pacific-science.com -Original Message- From: dennis roberts [mailto:[EMAIL PROTECTED]] Sent: Tuesday, October 09, 2001 1:52 PM To: Wuensch, Karl L; edstat (E-mail) Subject: Re: Standardized Confidence Intervals At 03:45 PM 10/9/01 -0400, Wuensch, Karl L wrote: Some of those who think that estimation of the size of effects is more important than the testing of a nil hypothesis of no effect argue that we would be better served by reporting a confidence interval for the size of the effect. Such confidence intervals are, in my experience, most often reported in terms of the original unit of measure for the variable involved. When the unit of measure is arbitrary, those who are interested in estimating the size of effects suggest that we do so with standardized estimates. It seems to me that it would be useful to present confidence intervals in standardized units. why? you only get further away from the original data scale/units you are working with ... in what sense ... is ANY effect size indicator anything BUT arbitrary? i don't see how trying to standardize it ... or any confidence interval ... makes it anything other than still being in arbitrary units ... i would argue that whatever the scale is you start off using ... that is as CLOSE as you can get to the real data ... even if the scale does not have any natural or intuitive kind of meaning standardizing an arbitrary variable does NOT make it more meaningful ... just like converting raw data to a z score scale does NOT make the data more meaningful standardizing a variable may have useful properties but, imputing more meaning into the raw data is not one of them == dennis roberts, penn state university educational psychology, 8148632401 http://roberts.ed.psu.edu/users/droberts/drober~1.htm = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
RE: Standardized Confidence Intervals
At 03:04 PM 10/9/01 -0700, Dale Glaser wrote: Dennis..yes, the effect size index may be arbitrary, but for argument sake, say I have a measure of 'self-esteem', a 10 item measure (each item a 5-pt. Likert scale) that has a range of 10-50; sample1 has a 95% CI of [23, 27] whereas a comparison sample2 has CI of [22, 29]. Thus, by maintaining the CI in its own unit of measurement, we can observe that there is more error/wider interval for sample1 than sample2 (for now assuming equal 'n' for each sample). However, it is problematic, given the inherent subjectivity of measuring self-esteem, to claim what is too wide of an interval for this type of phenomenon. i did not know that CIs could tell you this ... under any circumstance ... i don't see that standardizing it will solve this problem ... supposedly, CIs tell you something about the parameter values ... and nothing else ... i don't think it is within the capacity of ANY statistic ... to tell you if some CI is too wide or too narrow ... WE have to judge that ... given what we consider in our heads ... is too much error or what we are willing to tolerate as precision of our estimates How do we know, especially with self-report measures, where indeed the scaling may be arbitrary, if the margin of error is of concern? It would seem that by standardizing the CI, as Karl suggests, then we may be able to get a better grasp of the dimensions of error...at least I know the differences between .25 SD vs. 1.00 SD in terms of magnitude..or is this just a stretch?!!! you do this ahead of time ... BEFORE data are collected ... perhaps with some pilot work as a guide to what sds you might get ... and then you design it so you try to work withIN some margin of error ... i think the underlying problem here is trying to make sense of things AFTER the fact ... without sufficient PREplanning to achieve some approximate desired result after the fact musings will not solve what should have been dealt with ahead of time ... and certainly, IMHO of course, standardizing things won't solve this either karl was putting regular CIs (and effect sizes) and standardized CIs (or effect sizes) in juxtaposition to those not liking null hypothesis testing but, to me, these are two different issues ... i think that CIs and/or effect sizes are inherently more useful than ANY null hypothesis test ... again, IMHO ... thus, brining null hypothesis testing into this discussion seems not to be of value ... of course, i suppose that debating to standardize or not standardize effect sizes and/or CIs ... is a legitimate matter to deal with ... even though i am not convinced that standardizing these things will really gain you anything of value we might draw some parallel between covariance and correlation ... where, putting the linear relationship measure on a 'standardized' dimension IS useful ... so that the boundaries have some fixed limits ... which covariances do not ... but, i am not sure that the analog for effect sizes and/or CIs ... is equally beneficial Dale N. Glaser, Ph.D. Pacific Science Engineering Group 6310 Greenwich Drive; Suite 200 San Diego, CA 92122 Phone: (858) 535-1661 Fax: (858) 535-1665 http://www.pacific-science.comhttp://www.pacific-science.com -Original Message- From: dennis roberts [mailto:[EMAIL PROTECTED]mailto:[EMAIL PROTECTED]] Sent: Tuesday, October 09, 2001 1:52 PM To: Wuensch, Karl L; edstat (E-mail) Subject: Re: Standardized Confidence Intervals dennis roberts, penn state university educational psychology, 8148632401 http://roberts.ed.psu.edu/users/droberts/drober~1.htm = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
RE: Standardized Confidence Intervals
At 03:04 PM 10/9/01 -0700, Dale Glaser wrote: It would seem that by standardizing the CI, as Karl suggests, then we may be able to get a better grasp of the dimensions of error...at least I know the differences between .25 SD vs. 1.00 SD in terms of magnitude well, yes, 1 sd means about 4 times as much spread (in sd units that is) than .25 sd (whether it be error or anything else) ... but, UNLess you know what the underlying scale is ... what the raw units mean ... have some feel for the metric you started with ... i don't see that this really makes it instantaneously more understandable i would like to see a fully worked out example ... where we have say: regular effect sizes next to standardized effect sizes ... and/or regular CIs next to standardized CIs ... and then try to make the case that standardized values HELP one to UNDERSTAND the data better ... or the inference under examination ... we might even do some study on this ... experimental ... where we vary the type of information ... then either ask Ss to elaborate on what they think the data mean ... or, answer some mc items about WHAT IS POSSIBLE to infer from the data ... and see if standardizing really makes a difference my hunch is that it will not ..or is this just a stretch?!!! Dale N. Glaser, Ph.D. Pacific Science Engineering Group 6310 Greenwich Drive; Suite 200 San Diego, CA 92122 Phone: (858) 535-1661 Fax: (858) 535-1665 http://www.pacific-science.comhttp://www.pacific-science.com == dennis roberts, penn state university educational psychology, 8148632401 http://roberts.ed.psu.edu/users/droberts/drober~1.htm = Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =
