Professor Gallagher and All --
Let me first of all absolutely commend your professional interest, time,
and dedication to fair testing practice! With high stakes testing
apparently the fad of this particular few years in public education,
fairness is -- if posible -- an even more important concern than usual.
I have downloaded the newspaper article and your analysis I have not yet
had a chance to read your criticisms, but I did notice something odd in the
newspaper report.
It would appear that neither the "appeal systems" nor a claim of
"technical adequacy" would be a response to your concern about bad
questions. The claim of technical adequacy, i.e. "that good students tend
to answer them correctly anyway, but poor students don't" does not, to my
mind, constitute technical adequacy. Presuming that the aforementioned
claim is that the questions have positive discrimination, one is constrained
to point out that an item may have good discrimination and still not be a
valid question. For example, if you are correct that item X is in the
11th-12th grade curriculum, but given to 10th-12th graders, it would
presumably have high discrimination, but still not be "fair" to 10th
graders.
Secondly, the "appeal system" seems to miss your concerns also. As
quoted in the article, a technical appeal may be made if (1) the test was
graded incorrectly, and (2) if competence can be established by other means.
"[Scoring accurately" (Nelhaus quote) seems to mean having two readers for
the 10th grade exams. If this is in response to (1), the validities of the
individual questions are still unadressed -- it is not generally considered
progress if you give a cannibal and knife AND a fork, to paraphrase the old
epigram.
It would seem to me the only place to address issues of validity is by
tightening up on the test at the writing stage, not at the "Oops" stage.
Again, my thanks for addressing this problem -- keep on it!
-- Chris
Chris Olsen
George Washington High School
2205 Forest Drive SE
Cedar Rapids, IA
(319)-398-2161
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, October 04, 2001 9:33 PM
> To: [EMAIL PROTECTED]
> Subject: MCAS, statistics and other math problems
>
>
> During the last week in August, there was a lengthy thread on
> sci.stat.edu
> about problems with the probability and statistics questions
> in MCAS, the high
> stakes test required for graduating from a MA public high school.
>
> Shortly after participating in that thread, I wrote up my
> analyses of 6 of the
> 41 questions on the 2001 exam and 4 of the 42 questions on
> the 2000 exam. I
> mentioned this group and several people who posted on that
> thread in my
> commentary. Four of the 2000 & 2001 questions were in the
> area of statistics
> and probability. The most severe problems appear to be
> testing on concepts
> intended only for 11th & 12th graders - a violation of the
> basic principle of
> MCAS. My commentary includes cross-references to these
> learning standards.
>
> I sent my critique to the Dept. of Education in Aug. so that
> they could delete
> the flawed questions from the scores on which failure
> decisions are based. As
> described in the newspaper article linked below, there are no
> appeals now
> possible in MCAS for failing because of invalid questions.
> Students must pass
> this MCAS math test, or one of 4 retakes, to graduate in
> 2003. 45-50% of MA
> 10th graders failed the tests offered from 1998 through 2000.
>
> The MA Dept of Education's Press spokesman, Heidi Perlman, is
> quoted in a
> Berkshire Eagle story that DOE stands behind all of the
> questions I discussed:
> "But the Department of Education dismisses the
> professor's analysis of the
> test and stands by all its questions. In fact, the department
> changed its
> scoring policy on the 10th-grade tests this year to safeguard
> against errors,
> officials said. .. "Some of the questions could've been
> slightly better worded,
> but in total none of them are invalid questions," said
> department spokeswoman
> Heidi Perlman. She said it was reviewed by department staff,
> but officials did
> not agree with Gallagher's analysis. "We don't believe his
> criticisms are
> valid," Perlman said.
>
> Ms Perlman's comments can can be read in the following news
> article from the
> Berkshire Eagle, by Jennifer Fenn:
> http://search.newschoice.com/ArchiveDisplay.asp?story=d:\index
\newsarchive
s\ne\nebe\fpg\20011002\1122674_mcaserrors.txt&source=www%2Eberkshireeagle%
2Ecom&puid=2269&paper=Berkshire+Eagle+Online
One reporter told me that he wasn't sure how to handle my analysis of these
questions because DOE says the questions are all valid, and I say that they
aren't. He said that he's not a mathematician and can't judge who is
correct.
I said publish the questions, and let the mathematically literate readers
respond.
I would appreciate what the members of this group think about the questions,
especially about the statistics and probability questions. I realize that
my
28-p commentary is a bit much, but the issue is very important to the Class
of
2003 and other citizens of MA.
The math/stats questions, with my analysis can be viewed at:
http://www.es.umb.edu/edg/MCAS/mcasproblems.pdf
Gene Gallagher
UMASS/Boston
=================================================================
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=================================================================
=================================================================
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=================================================================
