{
Test::Harness just runs the tests you give it. Simplest thing to do is
to just write a little script that has the necessary logic to determine what
set of tests to run and feed that file list to runtests().
}
However, I notice, looking through the Makefiles for the various languages, that it's a real hodge-podge.
Some of them use Test::Harness in the Makefile to run their code, some have their own harness, some eschew a harness and run their tests individually.
Some just run examples, not tests at all (Bad!).
Attached, find a first pass at a "testall" script that does the best it can to pull out the information from the various Makefiles and do a single test harness. Put the script in languages/, and edit languages/Makefile (or config/gen/makefiles/languages.in if you want it to stick around) so that test: runs "$(PERL) testall" and doesn't actually depend on any of the ".test" targets.
It currently sort of allows the following (though my logic isn't perfect, it mostly works in my current local copy.) ways to specify tests
+ calling Test::Harness with a glob() pattern.
+ specifying foo.t files somewhere in the test rule.
+ calling your own harness script. (t/harness) (must have a glob pattern in there.)
As language authors move over to use the various Test modules, this script /might/ add them to the test run transparently.
Probably the best thing is to agree on a standard way for languages to run tests, however, that supports both an individual and combined harness.
My best suggestion along those lines at the moment is to have everyone's "make test" do "$(PERL) t/harness", which should use Test::Harness to run their tests. Our global languages "make test" will call "$(PERL) testall", which will call "$(PERL) t/harness -files" to get a complete list of all test files required. It will then run its own Test::Harness.
This approach requires that (a) people use the perl testing harness, and (b) that their tests be agnostic about the directory they're run from. (tcl, for example, currently fails every test when run from "testall" because it does care.)
Feedback?
Oh, and btw:
Failed 77/85 test scripts, 9.41% okay. 195/196 subtests failed, 0.51% okay.
(I think the astonishing part there is that something actually passed! =-)
testall
Description: Binary data
On Friday, March 26, 2004, at 02:17 PM, Dan Sugalski wrote:
Another job for the intrepid configure and/or makefile hacker.--
Right now, there's a languages-test target in the top level makefile. This is good.
Unfortunately, the way it works is... sub-good. What it does is do a "make test" in the languages directory, and that target runs each language test in turn. Not bad in itself, but it has two problems:
1) tinderbox clients pick up the language test as OK, even when its not. (cf the sprite/languages tinder build)
2) The language tests stop on the first language that dies hard, which at the moment is perl 6
So, what I'd like is for someone to figure a way to get all the languages test harnesses together under a single master harness, so all the languages will have their tests run all the time, and so tinderbox is happy with the results. (Get the first part done and I'll work out the tinder part--that's doable)
--
Dan
--------------------------------------"it's like this"-------------------
Dan Sugalski even samurai
[EMAIL PROTECTED] have teddy bears and even
teddy bears get drunk
Will "Coke" Coleda will at coleda dot com
