On Wednesday 02 July 2003 00:25, Adrian Howard wrote:
> On Monday, June 30, 2003, at 02:07 pm, Fergal Daly wrote:
> The footer doesn't indicates that the correct number of tests ran
> (that's the plan's job), it just shows that a test script completed
> without error.
I think that would be useful.
> So a noplan test might look like
>
> ok 1
> ok 2
> 1..2
> # fin
>
> a planned test with an extension might look like
>
> 1..2
> ok 1
> ok 2
> 3..3
> ok 3
> # fin
but what if your script outputs
1..2
ok 1
ok 2
# fin
there is no way to know that there's an extension missing. It might sound
unlikely that this would happen but take the Test::NoWarnings module and say
it works by extending the plan at the end and say for some reason it's buggy
or even that you forgot the use it. There'd be no way to know that the END
block didn't run.
There's a good chance that when writing scripts people will naturally bundle
the plan extension and the tests together as a unit, possibly even in a
subroutine which makes it even easier for the whole things to be silently
skipped.
The purpose of the plan is to declare in advance how many tests will run, so
that if an error in your script or some exceptional circumstances cut the
test short, you will find out. The reason plans are effective is because they
are declared very early, before anything has a chance to go wrong. If the
plan is changeable then when the script finishes you have know way of knowing
if the plan was changed the correct number of times.
The current situation is bullet proof, if a test is missed the plan will be
wrong, end of story The only way to make extensions just as bullet proof is
to predeclare your extensions too, so your plan becomes x tests and y
extensions.
> To be honest, I'm not sure that having a global plan at the test script
> level gives you that much benefit.
>
> I like planned tests, but a manually maintained global plan is a
> maintainence nightmare if you have a large number of tests, or if the
> number of tests is runtime dependant.
Absolutely, that's the big problem with plans they are too difficult. That's
the problem arbitrarily nested sub blocks (each with their own plan) solve. A
global plan is hard to maintain because the plan is far away from the tests.
By using blocks, each with their own plan, things are much easier, if you add
a test to a block, you just need to update the plan for the block, that's it,
no changes required anywhere else.
It allows the testing equivalent of structured programming, local changes do
not require global mainteance. I rarely use plans at the moment because I
write a lot of scripts like this
foreach my $thing (@things)
{
do_some_comlicated_test($thing);
}
run_some_other_test();
sub do_some_complicated_test
{
is_this($thing);
is_that($thing);
is_the_other($thing);
}
so the plan is number of elements in @things X number of tests in the
subroutine + other tests. A royal pain to maintain. If I could do this
use Test::More tests => 3;
{
block(tests => 20);
foreach my $thing (@things)
{
do_some_comlicated_test($thing);
}
}
a_test(@things);
another_test(@things);
sub do_some_complicated_test
{
block(tests => 3);
is_this($thing);
is_that($thing);
is_the_other($thing);
}
that makes my plan a doddle to maintain.
> I've not noticed any problems now that I use Test::Class for most of
> my test code. I still plan the number of tests executed - I just do it
> at the method-level rather than the test-script level.
I'm not familiar with it although it looks nice. I'm guessing that Test::Class
calculates your global plan for you? One advantage to the block approach over
extensions or autoplanners is that you will know exactly which block ran too
many or too few tests, whereas with a global plan, you just know that
somewhere something went wrong, maybe you go hunting for it, maybe you "fix"
the plan.
> That would work, but you face the problem of having an evil
> internal-error causing your script to exit early without getting any
> output from earlier tests. You also lose user feedback for long test
> sequences (you only find out that test 2 failed after all 276000 test
> run).
Absolutely, that idea was just thrown out for discussion. There's another one
percolating but it's not quite there yet, it goes something like
plan 1..3
plan .1 1..5
ok .1.1
ok .1.2
ok .1.3
plan .1.4 1..2
ok .1.4.1
ok .1.4.2
ok .1.5
ok .2
ok .3
1..8
Which is compatible with the current TH protocol (which is why all the test
numbers are preceeded by .s) but a newer TH could figure it out. It doesn't
depend on indentation, so tests can be carried out in any old order, allowing
the cross-cutting AOP tests you mentioned and it also caters for wacky
threaded scripts too.
> Being able to extend the plan seems a simpler solution to me.
Much simpler, no doubt about it.
Actually I just thought of another problem with extensions and threads, you'd
have to make it possible to declare another extension before the current one
has finished. This means that if part of your script misses a test and a
later part runs 1 extra, everything looks fine.
F
