Andreas J. Koenig wrote:
>>>>>> On Fri, 14 Dec 2007 15:49:32 -0800, Michael G Schwern <[EMAIL
>>>>>> PROTECTED]> said:
> > We would seem to be agreeing. If the goal of the test suite is not to
> protect
> > against spoofing, and if it doesn't accomplish that anyway, why put a
> > signature check in there?
>
> Of course we are agreeing 99%. But I'm citing the Michael Schwern
> saying that is dearer to me than the above paragraph: tests are there
> to find bugs.
I say lots of apparently contradictory things. The trick is knowing when one
rule wins out over the other.
Something to keep in mind is that I'm talking about one very specific test.
Don't let this discussion get tangled up in the "author tests" brouhaha that
often brews up around here.
> > [...] But if the test failed because it's a bad test,
>
> Clearly a strawman's argument. It's impossible to contradict you on
> this. Thou shalt not write bad tests. Period.
That was supposed to come out more like "if the test failed because of a
mistake in the test suite". You know the sort of thing. Like when you write:
like $error, qr/your shit is broke at $0 line \d+\.\n/;
and it blows up on Windows because you forgot about the backslashes in Windows
path names. The test failure indicates a bug in the test, not the code.
Thus, the failure has a cost and no benefit.
> > The
> > signature test is not actually indicating a failure in Test::More, so
> it's of
> > no benefit to me or the users, and the bug has already been reported to
> > Module::Signature.
>
> See above. Once the bug is reported there is no justification to keep
> the test around. In this case I prefer a skip over a removal because
> the test apparently once was useful.
Buuuut skipped tests don't get run so it's effectively deleted, except a
permanently skipped test sits around cluttering things up. Smells like
commenting out code that maybe someday you might want to use again in the
future. Just adds clutter.
If I want to bring a test (or code) back from the dead that's what version
control is for.
> > The threading test is indicating a perl bug that's very difficult to
> detect
> > [2], only seems to exist in vendor patched perls, I can't do anything
> about
> > and is unlikely to effect anyone since there's so few threads users. It's
> > already been reported to the various vendors but it'll clear up as soon as
> > they stop mixing bleadperl patches into 5.8.
>
> > In short, I'm paying for somebody else's known bugs. I get nothing.
> > Test::More gets nothing. The tools get nothing. Cost with no benefit.
> So
> > why am I incurring these costs? Maybe the individual users find out their
> > tools are broken, but it's not my job to tell them that.
>
> During smoking CPAN I often find bugs in one module revealed by a test
> in another one... Only because David Golden tests so hard his tests were
> well suited to reveal a bug in Test::Harness. I'm glad he doesn't ask
> if it is his job or not. Just a few RT headlines of the past year:...
> Catalyst:Plugin:Authorization:Roles found a bug in C:P:Authentication.
> DBI 1.601 broke Exception::Class::DBI. HTML-TreeBuilder-XPath 0.09
> broke Web::Scraper. Test::Distribution 1.29 broke Lingua::Stem.
> Math-BigInt-FastCalc broke Convert::ASN1. Test:Harness 3.0 broke POE.
> DBM-Deep-1.0006 broke IPC::PubSub. DateTime-Locale-0.35 broke
> Strptime. Data::Alias 1.06 breaks Devel:EvalContext. Class::Accessor
> breaks Class::Accessor::Class. DBIx-DBSchema-0.33 breaks Jifty::DBI.
> File::chdir 0.08 breaks Module::Depends 0.12. Lingua:Stem:It 0.02
> breaks the Lingua:Stem testsuite. SVN-Notify-0.26 breaks
> SVN::Notify::Config (and others). Heap 0.80 breaks Graph. DBI-1.53's
> NUM_OF_FIELDS change breaks DBD-Sybase 1.07. Getopt-Long 2.36 breaks
> Verilog::Language. And so on.
I agree that's all very useful. Interlocking dependent test suites ferret out
bugs the original authors wouldn't find. However, there is a very important
difference between the list above and Test::More's signature test.
On a quick scan, all of those modules have direct dependencies. DBD::Sybase
uses DBI, Lingua::Stem uses Test::Distribution, etc... so it's natural that
their tests would test their dependencies. If a dependency breaks, they
break. I'm sure most of the authors above did not set out with the intention
to test their dependencies, it's all inherent in the testing of their own code.
Test::More doesn't actually use Module::Signature, so why is it testing it?
It would be like if DBI decided to add a test to make sure MakeMaker can read
the MANIFEST. Sure it's useful to know that part of the toolchain works and
that the MANIFEST can be read, but why is that in DBI? One can argue that DBI
depends on the good functioning of ExtUtils::Manifest to install, so it should
test it. Ok, then what about all the other things DBI depends on to install?
Should it test that MakeMaker can make a valid Makefile? Should it test that
tar and gzip work? Should I check that CPAN.pm can properly download and
install modules? Should it check that Test::Harness can parse TAP? That
Test::More is outputting correct TAP?
At some point you have to draw the line in how much of the universe you're
going to be responsible for. I like to put that border at the edge of the
module's functionality, it has a nice, sharp edge and it lets a module author
do their thing without going insane. The signature test goes past that line
so I'm going to clip it off.
> Maybe you want to borrow a bit from my Module::Signature test in
> CPAN.pm. It didn't find a bug in Module::Signature but it is well
> suited to find other bugs that would be really relevant for me and my
> users. I'll keep mine in and I hope others do likewise. And I hope
> nobody tells me to remove working tests.
CPAN.pm is different. It relies on Module::Signature for it's features. So
it makes perfect sense for CPAN.pm to test Module::Signature.
PS Irony of ironies, just after I wrote my previous reply I got a bug report
about that fucking thread test again.
--
Hating the web since 1994.
