On 3/7/07, Andy Armstrong <[EMAIL PROTECTED]> wrote:
It sounds as if you're doing monitoring rather than testing though.
Although they're related the requirements are quite different.
Poor explaining on my part then. Monitoring has similar needs, but us
usually much more shallow. Consider a web app test (simplified):
# Set tail before running test
$database->ok("tail test add $data");
$client->ok("submit $data to url http://www.example.com/form.cgi";);
Monitoring would check that the http server is up, and check that the
database server is up, but this is functional testing. Does the
application work, end to end.
Now stick some new web app security thingy in from of the web server,
and a database security tool in between the web app server and the
database. We need to run a regression test on the changes, but how.
This is not monitoring. This is scripted testing.
You
need real time capture of as much information as you can get; the
tests we're talking about aren't time dependent and it's quite
acceptable to run any test multiple times until the source of the
problem is found. The ability to capture more nuanced information
about the outcome of a particular test would burden the tests with
producing that information.
That's what Test::More does. Otherwise, everything would just be Ok.
In summary I think you're saying "don't just tell me it failed, tell
me /why/ it failed dammit" and we're saying pretty much the opposite:
"just tell me if it worked or not and leave it to me to find out why".
is_deeply and cmp_ok start to say why.
Now why is a matter of degree, and if a test could tell that the
developer meant && instead of || or that the new HTTP firewall was
dropping all POSTS with "hacker" in the data, then we'd be out of
work. I thought the spirit was to present as much info as practical.
> Functional testing is years behind, especially testing distributed
> applications like network security. I'm trying to leverage what I can
> to get something useful working. Perl and Test::Harness seem to be
> good tools to try.
I'd say not actually. Perl's certainly an excellent tool for your
application but I don't think Perl's testing framework is a good
match for what you're doing. Use the testing framework to prove that
your code works of course but don't try to make it do something it
wasn't intended to do and then blame it when it fights back :)
I really don't care too much, because I am already making what's there
work for me now. But if you are asking what to improve... :)
I took a look at http://www.testobsessed.com the other day. Check out
the functional test tools next generation dream. That makes status
codes look like Lego Duplos.
