On Thu, 9 Sep 1999, Alan & Susan Mead wrote:
> >We have definitely been discussing written tests. I think there is a lot
> of
> >interest in psychometricians and others in more performance-based testing
> >and a lot of certification folks (i.e., sys admins) have mentioned it in
> >discussions but I don't know anyone that is doing it validly and cost
> >effectively.
Alan message touches upon all of the major limitations of using "hands-on"
assessment instead of traditional testing. It all boils down to this: It
might appear obvious that hands-on assessment, where for example the
examinee actually sits in front of a live Linux box and manipulates it, is
a better measure of competency that regular testing, but research does not
back this assumption up. Most studies that have been done comparing the
two methodologies find that there is a stronger correlation between test
score and future job performance than between evaluation of hands-on
experience and future job performance. The truth is we
(psychometricians) have not figured out how to extract reliable and valid
measures from performance tests. Until (and unless) advances are made
that permit this, we are better off sticking with traditional tests.
One of the greatest benefits of traditional Q&A testing over hands-on
experiences is that the former allows broader and deeper content sampling
than the latter. By this, I mean that it is possible to cover much more
material on a traditional test than on a hands-on assessment (assuming
time is held constant.) For example, in 2 hours candidates could answer as
many as 90 questions, covering a broad range of topics. But if hands-on
assessment were used, they would be limited to just a few different
assessments, and it would be difficult to draw general conclusions about
their overall level of competence.
As a result of these factors, and other mentioned by Alan, we have
decided to use tests rather than hands-on assessments for our basic
certification levels. The key to making this work is to include many
different kinds of questions, particularly those that require the examinee
to apply high level problem solving skills.
------------------------------------------------------------------------
Scott Murray Director of Exam Development / Psychometrician
[EMAIL PROTECTED] Linux Professional Institute http://www.lpi.org/
------------------------------------------------------------------------
________________________________________________________________________
This message was sent by the linux-cert-program mailing list. To unsubscribe:
echo unsubscribe | mail -s '' [EMAIL PROTECTED]
