Allison Randal wrote:
> Okay, languages/perl6 is failing all its tests.
[...]
> Will or Jerome, could you take a minute to explain the direction
> you're taking Parrot::Test? I'll patch P6C::TestCompiler to match the
> new setup, but I want to do an "architecture sanity check" first.
I'm sorry, I've been very busy for 2 weeks, and did not have the time to
continue things regarding this topic. :-(
I hope to have some more time this week-end, but I'm not even sure...
So, very quickly:
1. The main goal is to have unified languages tests, that is, make
language-tests would get a list of all the test files for all of the
languages, and then run only _one_ harness with all those files. This
would allow for one nice overview of all the tests, and tinderboxen
would benefit from this too: they were reporting success as long as the
last language was reporting succes.
2. In order for this to work, all the languages should have commit to
having a t/harness file in their directory, responsible to get back a
list of all the test files when called with a -files argument. (Except
if we consider that *.t files are test files, but this may be too
strict) If the language implementer wants to use this harness to also
launch test for the current language only, this is of course welcome
3. Will and I thought that it would be nice for tests to just "use
Parrot::Test" and then have a new function:
language_output_is($language, $code, $expected, $comment)
4. Will worked on a patch for Parrot::Test, with some (little) help from
me to debug it. (Perl 6 tests did not even run because it seems Perl 6
test suite has core knowledge of Parrot::Test)
5. With Dan's agreement, I applied it since Will does not have checkin
rights on the whole tree.
6. The next thing to do was to work on the subject in order to make all
languages aware of this scheme.
7. The patch has some, erm, "action at distance" problems, such as the
pasm files put in /tmp
This is for what happened.
Now, here's my view for the future of languages testing:
1. I really think the scheme one language_output_is function and one
global harness is the way to go. This does not forbid having one
harness per language: indeed, each language still _needs_ one harness
script to fetch the list of test files.
2. Will's patch needs to be updated in (at least) two ways: a) put back
pasm files in test directories and b) remove the need for a
Parrot::Test::<Language> module. It should _allow_ it, but should also
provide a convenient default for languages that do not need fancy
testing things. (This should be the case of all the "little" languages
where running a $parrot languages/$lang/$lang.pasm $code_file should be
enough)
3. All the languages needs to be updated to take this scheme into
account once Parrot::Test is modified accordingly.
That's it, I don't know if I made myself clear? (tricky subject to work
on - and thus to explain :o) )
As we saw, it seems that Perl 6's test suite does weird things. Can you
explain what those weird things are, so we can take them into account
in the Plan?
Also, what do you think of the scheme? (and when I say you, I also mean
the p6i list) Knowing that the scheme will allow (currently it
_forces_) you to extend Parrot::Test with Parrot::Test::Perl6 (or
whatever language)
I'll try to work on it saturday, but maybe we should currently revert
the patch on Parrot::Test so I can have some more time re-working the
languages testing stuff, without annoying Leo and you. (at least)
Once again, sorry for lack of time that led to those problems,
Jerome
--
[EMAIL PROTECTED]
