>The school results are presented in a very odd fashion, making it
>difficult to assess the patterns.
>http://www.doe.mass.edu/ata/ratings00/SPRPDistribTables.html
They are that. Let's try.
These data don't look at all like the newspaper story. Here they are,
with outcomes given as proportions of each group.
Failed App Met Exceeded N
(Very high 0% 50% 33% 17% 6)
High 25% 13% 26% 36% 140
Moderate 43% 17% 23% 17% 471
LOw 60% 14% 18% 8% 545
VeryLow 76% 8% 14% 4% 287
Critical 91% 4% 3% 1% 91
Overall - regression to the mean would cause a NW-SE ridge in this
table - and the newspaper story suggested this. What we see is a NE-SW
ridge. Whatever is causing that ridge is much stronger than regression
to the mean.
It might be just the demands for more improvement from the "low"
schools.
To check this, put all the groups onto one quantile plot (I have
omitted the tiny "very high" group; others are labelled as Critical,
Very low, Low, Medium, or High by initial):
Proportion of schools improved by fewer than X pts
0 1 2 3 4 5 6 7 8
1 v c
c
.9 c l
vm
.8
vl
.7
h
.6 lm
.5
m
.4 h
.3
h
.2
.1
0
0 1 2 3 4 5 6 7 8
Bingo - one common curve, as near as we can tell. We have different
groups in different parts of the plot because of the funny way the data
were presented, but one curve seems to fit nicely.
This suggests that - far from better schools being penalized for
regression to the mean, or poorer schools being rewarded for it - the
ability of schools to improve on a once-off basis was roughly constant
across the spectrum, and historically poorer-performing schools are
being penalized by unreasonably-high goals.
What would be reasonable goals? On a once-off basis, it suggests that
about 50% of schools can improve by 2 points, about 75% can at least
hold their own, and (assuming approximate symmetry in the ogive) very
few schools "in control" would drop by more than about 2 points. A
possible system, then, would be to give - across the board - a major pat
on the back for an improvement of more than 4 points, a minor pat on the
back for an improvement of more than 2 points, and an investigation for
a drop of more than 2 points. Also, a complementary system based on raw
performance.
-Robert Dawson
=================================================================
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=================================================================
