Two sorts of testing - a) compiler, b) modules.
Each category has different environment and function.
a) compiler.
An official test suite is defined (and probably will be added to as
corner cases / ambiguities are discovered and disambiguation decided, so
presumably some standardisation of testing must exist, either implicitly
or explicitly).
Hence tests are canonical. Compiler is changed to conform to tests.
Function of test suite - validate compiler implementation as perl6.
b) modules.
Valid compiler is assumed.
Tests and module are changed to conform to compiler.
Function of test suite - validate that a module works in an expected
manner on a valid complier.
Clearly, the functionality in Test.pm (for modules) could/should be a
superset of the functionality in test_compiler.pm, and can assume a
valid compiler.
The interesting question - I think - is how practically and elegantly to
ensure the tests in the compiler validation suite achieve their task.
Two alternatives it seems are possible:
a) Define the code for test_compiler.pm which is guaranteed to run all
the test suite. This make test_compiler.pm a part of the language, viz.,
implementation as specification.
Tests will be needed in the official test suite to ensure that
test_compiler.pm running under the new compiler conforms to
specification (eg., an error in the compiler reverses the meaning of the
test functions).
b) Define the core functions for testing the compiler as part of the
language, allowing no others in the official test suite. Alternative
implementations of test_compiler.pm are possible (though a prototype
could be provided).
Tests needed in the official suite to validate conformity of
implementations of the test functions to specification.
Whatever, it does seem to me that if perl6 is to be defined by its
specification and its test suite, then the testing process has also to
be defined within the language. If the testing process is not defined at
the same level as the language itself, then at some point the validity
of a compiler of the language will be dependent on the implementation of
a part of the testing process.
Patrick R. Michaud wrote:
On Tue, Sep 02, 2008 at 12:32:49PM -0700, Darren Duncan wrote:
Patrick R. Michaud wrote:
I think it's good to have a prototype Test.pm that we can point to as
a reference, but I don't think we need to try to designate it as being
"official".
[...]
2. The Perl 6 language spec itself would specify a basic set of test
routines built-in to the language, in a Test namespace, much as it
defines collections of routines now for such as numbers and arrays and
standard I/O. And so the basic test routines would be formally defined
in a Synopsis document.
I disagree. The testing we're likely want to do as part of the language
test suite may be substantially different from what we want to provide
to module writers for testing. In particular, I think that the test
suite harness should require only a minimal Perl 6 implementation
(note I said "harness", not the tests themselves), whereas it's much
more reasonable that a testing system used by module writers could/should
assume a fully working Perl 6 implementation.
It's a difference of "bootstrapping" versus "running environment".
I also don't see the possibility of our "getting it wrong" in the design
to be such a big deal, since the odds are anything we think of now will
work well for many years, as Test.pm/Test::More has been fairly stable
already and meanwhile Perl 6 is versioned now, so we could make an
incompatible change to the Test related language spec in the future, and
as long as users say "use Perl-6.0.0" their code relying on the
older/current Test.pm like interface won't break.
"Perl 6 is versioned now" is a misnomer. The *spec* calls for a versioned
Perl 6, but I'm not aware that any of the implementations do much with that.
At any rate, relying on handling multiple versions of Perl 6 to run Test.pm
is exactly one of those things I'd like to avoid in the official test suite.
Pm
This mail was received via Mail-SeCure System.
