On 19/09/06, Michael G Schwern <[EMAIL PROTECTED]> wrote:
Fergal Daly wrote:
> But there's nothing here for the author to do so it should not be
> marked TODO.
Sure there is. They now know Fribble works and can once again start relying on
it.
To be clear, the scenario is that Foo uses Fribble which used to work
fine but recent versions are broken? This seems to be what we're
considering (from the "once again"). Also, Fribble does not provide
essnetial funcationality to Foo, if it did then we shouldn't SKIP or
TODO, we should just fail.
You seem to be saying that when the author of Foo finds out about this
he should wrap the Fribble tests in a TODO block and release a new
version of Foo on CPAN.
The TODO item is to remove the TODO from the test suite when a new
unbroken version of Fribble arrives.
The result of this is that users who still have an old working Fribble
get a bunch of TODO tests that should actually have been real tests.
Best case scenario is that they get confusing messages about
unexpected success, worst case is that the TODO hides failing tests
that really should have passed.
Ditto when a new working version of Fribble is released and Foo's
author has not yet released the un-TODOd version.
Also, consider when Fribble is fixed and the TODO is removed, anyone
with a broken Fribble will now start getting real fails when they
install Foo. So if you're going to manage this by inserting and
removing TODOs then you MUST update your MakeMaker dependencies, no
matter how unimportant Fribble is.
The only time when everything is behaving as it should is when a user
has a broken Fribble and a TODO Foo or a good Fribble and an un-TODOd
Foo. The other 2 cases are broken.
The SKIP method detailed below seems to be far better.
> This should be a skip wrapped in something to test if the installed
> version of Fribble works or not. Otherwise, if $o->fribble starts
> failing because it's genuinely broken, this becomes a false positive
> (which is the worst thing you can have from a test),
I think we're talking about two different ways to accomplish the same thing.
I'm talking about...
TODO: {
local $TODO = 'Fribble.pm is busted';
ok( $o->fribble, 'fribble' );
}
You're (I think) talking about:
SKIP: {
skip "Fribble.pm is busted" unless $o->fribble;
...tests which rely on Fribble.pm working...
}
Not quite, more like
SKIP: {
skip "Fribble.pm is busted" unless FribbleIsWorking();
...tests which rely on Fribble.pm working...
}
sub FribbleIsWorking {
#(some direct test of Fribble's functionality
}
In this case, Foo's Fribble parts get properly tested if the user has
a good version and get skipped if it's bad. Foo's author doesn't need
to do anything once Fribble starts working and users get no confusing
messages and no risk of false positives.
Foo's author doesn't even _need_ notification that Fribble is working
again. However if Fribble is fixed in a way that breaks backward
compatibility, Foo's author will know straight away because his
Fribble tests will start running and his Foo-Fribble tests will start
failing loudly.
If Foo's author wants to up the MakeMaker deps he can but if Fribble
really is that unimportant he doesn't have to.
Finally the FribbleIsWorking test could actually be included in the
module and used a load time to replace all the Fribble dependent
routines with warnings or errors or whatever.
Test::More :
You don't skip tests which are failing because there's a bug in your
program, or for which you don't yet have code written. For that you
use TODO. Read on.
Conversely, you DO skip tests which are failing because there's a bug
OUTSIDE your program. TODO is for code that's not yet written or for
bugs in in _your_ code,
F
