On 7 Mar 2007, at 19:39, Eric Hacker wrote:
Now, I know you are thinking about exit status on test scripts and I'm thinking individual tests (of which running another test script might be an instance), but in the distributed functional testing space, one really can't rely on independent test scripts to do much.
OK, I see where you're going. We may be talking at cross purposes slightly. TAP already has syntax to capture the status of an individual test. As far as TAP is concerned tests are binary - they succeed or fail. There may be additional diagnostic information that'll help someone to work out why they failed but in terms of the test harness' view of what happened they either pass or fail.
Now if all TAP can handle is Perl, or even just code output, then those of us pushing the envelope will eventually migrate to some other test output format.
TAP is a simple protocol which has implementations in C, Python, Javascript, PHP.
It sounds as if you're doing monitoring rather than testing though. Although they're related the requirements are quite different. You need real time capture of as much information as you can get; the tests we're talking about aren't time dependent and it's quite acceptable to run any test multiple times until the source of the problem is found. The ability to capture more nuanced information about the outcome of a particular test would burden the tests with producing that information.
In summary I think you're saying "don't just tell me it failed, tell me /why/ it failed dammit" and we're saying pretty much the opposite: "just tell me if it worked or not and leave it to me to find out why".
Functional testing is years behind, especially testing distributed applications like network security. I'm trying to leverage what I can to get something useful working. Perl and Test::Harness seem to be good tools to try.
I'd say not actually. Perl's certainly an excellent tool for your application but I don't think Perl's testing framework is a good match for what you're doing. Use the testing framework to prove that your code works of course but don't try to make it do something it wasn't intended to do and then blame it when it fights back :)
-- Andy Armstrong, hexten.net
