> I would like to make a plea for a validation suite for Haskell.
Hear hear.
> One of the best specified procedural languages is Pascal, which is similar in
> size to Haskell. The Pascal standard defines Pascal rather tighter than the
> Haskell report defines Haskell, but in practice the limiting cases are defined
> by a suite of several hundred (mostly pathological) test programs.
The Pascal validation suite and its followers (CHILL, Ada etc) have a
variety of tests:
1. For each occurrence of 'shall' in the standard, a test to ensure that
the feature is implemented.
2. For each occurrence of 'shall not', a test to ensure that the thing
forbidden is in fact forbidden.
3. "Gut feeling" tests where you guess an invalid implementation strategy
and deduce a test which reveals its existence (a GOTO into a FOR loop
which then prints the value of the loop variable is a good one).
4. "Evaluation" tests which check out the limits of the compiler (MAXINT,
maximum length variable names etc).
5. "Pathological" tests whose result is unclear. These illustrate problems
in the standard and don't properly belong in the validation suite (as
you can't expect the compiler to know what to do with them).
6. "Quality" tests to check out goodness of the implementation e.g. compile
and run-time error handling.
7. Really nasty tests whose result is defined but they don't go in the
validation suite because compiler writers would have to expend too
much effort to fix their systems.
Moral: the pathological tests are useful for language definers to focus
their attention on problems; the other sorts are more useful for compiler
writers. However, the pathological ones are far more interesting to write.
I would suggest that a Haskell suite consists of whatever tests people want
to submit, with the following provisos:
1. The test should state which bit of the Haskell report it is testing.
2. It should test exactly one feature and be as short as is sensible.
You might make an exception for tests which every compiler should pass
easily e.g. one test might check correct reading in of all possible escaped
characters in strings (one compiler, which shall remain nameless, used
to get this wrong).
3. It should say what the expected result is.
4. If at all possible it should be self-checking (i.e. print out PASS or
FAIL at the end). This makes regression testing easier.
Nick North
