--- demerphq <[EMAIL PROTECTED]> wrote: > On 7/30/07, Ovid <[EMAIL PROTECTED]> wrote: > > Tests should *only* fail when there is a clear, unequivocal reason > to > > believe that the code will not function appropriately on someone's > > machine. Having '=head0' or a '=back' without '=over' should not > be > > such a failure. It's taken a lot of grief while working on > TAP::Parser > > to realize how terribly wrong I was about this in the past and it's > a > > mistake I would like to better understand and rectify in the > future. > > Thats a very interesting comment that Id love to hear more about. Can > you expand on your experiences with TAP::Parser in this respect?
It's not TAP::Parser *per se* which has led to this. It's the creation of TAP::Parser and learning more about how toolchains work. For example, I thought that bonus tests (TODO tests which unexpectedly succeed) should really be a failure. However, that would break the CPAN toolchain dramatically. I have no way of quantifying things, so let's play "proctologist with a flashlight" and guess that there are maybe 100,000 people who regularly use cpan or cpanplus. Now let's say that I release a "Super::WunderModule" that fully 5% of those users want to install and on an average of two boxes apiece. In "Super::WunderModule", I have dependencies on Test::Class, Test::Exception, and PadWalker. Let's assume that only PadWalker is needed for the *functionality*, so we can safely say that a failure to install that is really a problem. So let's consider Test::Exception. It has a dependency on 5.6.1, but we'll pretend the code does, too. That's OK if you really need that. We also have dependencies on Sub::Uplevel (a notorious source of build problems), Test::Builder::Tester and we'll just pretend that the other modules are already installed. >From Test::Class, we'll just assume that only Devel::Symdump was not installed. Without continuing to follow dependencies, we have four completely optional modules to install, so our unecessary points of failure can be: $number_of_users * $percent_who_install * $average_number_of_boxes_installed_on * $modules_to_install So for this trivial example, we have about 40,000 potentially unneeded installs and possible failures. Of course, that's a rough and stupid calculation and those numbers have NO MEANING, but it shows how quickly this could balloon. Now try and calculate this for all CPAN modules. You can see why false negatives are a fact of life and anything which avoids this would be wonderful. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/
