This proposal is the result of several days of IRC talks and research
into how tests are handled by Parrot and Pugs, and how to reorganize
the pugs test suite to make it more amenable to testing to other
compilers.
Proposal:
Each test in the test suite should be broken down into functional or
semantic chunks, possibly in a sub-directory that is feature specific.
Givens:
1) Most people think that a unified test suite is the best way to go
forward with Perl6 testing.
IRC channel talks on #parrot, #perl6
2) The pugs test suite has been declared as the official test suite.
Perl 6 Design Meeting Notes
3) For the most part, pugs tests are monolithic, testing all options for
a given feature.
4) All perl6 compilers are at different stages of completeness, especially
with respect to compiling code. It isn't safe to assume that new compilers
will follow any given roadmap for implementing features.
Example:
Currently, the substr test:
t/builtins/strings/substr.t
contains 2-argument substr calls, 3-argument substr calls,
substr string method calls, lvalue calls, unicode tests, and disabled
sections in both POD for a junctive subtr call and a 4-argument substr,
all in one file.
If a compiler which cannot compile all those syntaxes (like parrot in
July 2007) tries to run that test, you won't get partial output, you get a
syntax error on the first unimplemented feature (which is currently 2-arg
substr in Parrot). In order for the test to have value to a developer,
the test will be broken into a smaller chunks:
t/builtings/strings/substr/:
2arg.t
3arg.t
method.t
lvalue.t
unicode.t
junctive.t
[I've picked a sub-directory here just to keep the t/builtins/strings
directory cleaner, but it's arbitrary.]
Now, compiler developers can pick and choose from the test suite which
files they want to run against their compiler. This is more difficult
than just saying runtests, but has an additional side benefit.
For a moment, assume that the list of tests to be run is in a file,
compiler_tests. If you were to take the compiler_tests file for all
compilers, you can very quickly feature check them against each other.
Instead of also running N tests for each feature, it would be simple to
take the compiler_tests file and rebuild compiler specific monolithic
tests for the sake of speed.
Also, instead of a file of tests, the test master (runtests,
etc.) could look inside of each test file for a marker similar to
the Perl6 TODO syntax:
http://perlcabal.org/syn/S02.html#line_271
and generate the lists of tests to run.
