At 11:13 AM 8/28/01 -0300, Robert J. MacG. Dawson wrote:
> If indeed the scores are being reduced by hiding the easy questions
>among the harder ones, then I would say yes, this is a defect of the
>current system, and should be changed. It may be that the questions
>themselves ought to be more difficult; but the difficulty ought to be
>intrinsic to the questions, not an artifact of the test format. What is
>at issue here is essentially signal-to-noise ratio.
>
> -Robert Dawson
however, we have to consider a countering factor too ... content
many tests sort of group their items in clusters ... that is ... items
about stat together ... general math together ... etc. and, there is some
usefulness to this FOR the examinees ... rather than mixing everything up
in that format ... difficulty probably spirals ... that is, within a
cluster ... you will have easier and harder items ... next cluster ... same
thing
so, in this case ... content is more or less constant (across n items) but,
difficulty varies
however, when we order items OVERALL from easy to hard (assuming you have
good and sufficient data to be ABLE to do this) ... the content keeps
mixing up ... so, you have the back and forth phenomenon of having to
constantly switch gears ... which could defeat the purpose of ordering by
difficulty
here, difficulty is relatively constant across groups of n items ... but
content varies
much of this depends on the TIME you have to work on the test ... if the
time limit is generous ... so everyone has sufficient time ... these
factors play much less (if any) of a role ... but, when the time is tight
... then anything we can do to get examinees through all the items ... to
at least have a look and see if they have any idea about how to answer them
... the better
the main thing we have to guard against is ... having examinees start the
test with a couple of lulu items ... and then think ... well, if
these first couple are doozies ... the rest of the items must be IMpossible!
the problem we face is helping examinees adopt good test taking strategies
... but, we know that once we let em loose on the test ... we have no
control over how they move through the test ... we would like to believe
that our instruction to "not spend too much time on any one item" will be
headed but, we know that it won't (some examinees will get "stuck" on an
item and NOT move on) ... so, if we can ARRANGE items in a way to help
optimize their performance ... we are getting better estimates of what they
know ...
and that should be our main goal
_________________________________________________________
dennis roberts, educational psychology, penn state university
208 cedar, AC 8148632401, mailto:[EMAIL PROTECTED]
http://roberts.ed.psu.edu/users/droberts/drober~1.htm
=================================================================
Instructions for joining and leaving this list and remarks about
the problem of INAPPROPRIATE MESSAGES are available at
http://jse.stat.ncsu.edu/
=================================================================
