Re: MA MCAS statistical fallacy

[Paul R Swank]

Regression toward the mean occurs when the pretest is used to form the groups, which it appears is the case here.

At 08:31 AM 1/11/01 -0400, you wrote:
>
>
>Gene Gallagher wrote:
>>
>> Those familiar with "regression to the mean" know what's coming next.
>> The poor schools, many in urban centers like Boston, met their
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> improvement "targets," while most of the state's top school districts
>> failed to meet their improvement targets.
>
> Wait one... Regression to the mean occurs because of the _random_
>component in the first measurement. Being in an urban center is not part
>of the random component - those schools' grades didn't improve because
>some of them woke up one day and found that their school had moved to a
>wealthier district.
> If the effect of nonrandom components such as this is large enough
>(as I can well believe) to justify the generalization highlighted above,
>and if there was a strong pattern of poor-performing schools meeting
>their
>targets and better-performing schools not doing so, we are looking at
>something else - what, I'll suggest later
>
>
>> The Globe article describes how superintendents of high performing
>> school districts were outraged with their failing grades, while the
>> superintendent of the Boston school district was all too pleased with
>> the evaluation that many of his low-performing schools had improved:
>>
>> [Brookline High School, for example, with 18 National Merit Scholarship
>> finalists and the highest SAT scores in years, missed its test-score
>> target - a characterization blasted by Brookline Schools Superintendent
>> James F. Walsh, who dismissed the report.
>
> There *is* a problem here,but it's not (entirely) regression
>to the mean. If I recall correctly, Brookline High School is
>internationally
>known as an excellent school, on the basis of decades of excellent
>teaching.
>If it couldn't meet its target, it's not because its presence among the
>top
>schools was a fluke in the first measurement - it's probably because the
>targets for the top schools were unrealistic.
>
> Was there any justification for the assumption voiced by the Boston
>superintendant that the top-performing schools were in fact not
>performing at
>their capacity and would be "smug" if they assumed that their present
>per-
>formance was acceptable? The targets described seem to imply that no
>school
>in the entire state - not one - was performing satisfactorily, even the
>top
>ones. Perhaps this was felt to be true, or perhaps it was politically
>more
>acceptable to say "you all need to pull your socks up" than to say "the
>following schools need to pull their socks up; the rest of you, steady
>as she goes."
>
> As a reductio ad absurdum, if this policy were followed repeatedly,
>it would be mathematically impossible for any school to meet its target
> every year. That - and not regression to the mean - is the problem
>here, I
>think.
>
> -Robert Dawson
>
>
>=================================================================
>Instructions for joining and leaving this list and remarks about
>the problem of INAPPROPRIATE MESSAGES are available at
> http://jse.stat.ncsu.edu/
>=================================================================
>
------------------------------------
Paul R. Swank, PhD.
Professor & Advanced Quantitative Methodologist
UT-Houston School of Nursing
Center for Nursing Research
Phone (713)500-2031
Fax (713) 500-2033

================================================================= Instructions for joining and leaving this list and remarks about the problem of INAPPROPRIATE MESSAGES are available at http://jse.stat.ncsu.edu/ =================================================================