>>>>> On Fri, 14 Dec 2007 15:49:32 -0800, Michael G Schwern <[EMAIL PROTECTED]> >>>>> said:
>> Asking the wrong question. None of our testsuites is there to protect >> against spoof or attacks. That's simply not the goal. Same thing for >> 00-signature.t > We would seem to be agreeing. If the goal of the test suite is not to protect > against spoofing, and if it doesn't accomplish that anyway, why put a > signature check in there? Of course we are agreeing 99%. But I'm citing the Michael Schwern saying that is dearer to me than the above paragraph: tests are there to find bugs. >> Yupp. And testing the signature in a test is better than not >> testing it because a bug in a signature or in crypto software is >> as alarming as a bug in perl or a module. > I believe this to be outside the scope of a given module's tests. It's not > the responsibility of every CPAN module to make sure that your crypto software > is working. Or perl. Or the C compiler. Or make. That's the job of the > toolchain modules which more directly use them (CPAN, Module::Signature, > MakeMaker, Module::Build, etc...). [1] You're barking up the wrong tree. Stick to your older wisdom that bugs are there to find tests, err, forgive me, tests are there to find bugs, and when a test found a bug it is a hero forever. > At some point you have to trust that the tools work, you can't test the whole > universe. You simply don't have the time. I agree totally. But you also do not have the wisdom that you can predict which tests will find bugs in which software. Your test found a bug in Module::Signature? Great, so it was a good test. Of course you should not annoy people with a test once the bug is known and reported. Make a new release, skip the test with the skip statement 'Module::Signature 123.45 known bug see RT #54321' and wait for the next time the test finds a bug once Module::Signature gets upgraded. Besides that I agree with the rest of your musings. Great writup. Some guidelines about cost/benefit relations are worthwhile but one should take care not to make them too narrow. The last thing we want to have is a new kind of CPAN police dictating people to remove tests. > [...] But if the test failed because it's a bad test, Clearly a strawman's argument. It's impossible to contradict you on this. Thou shalt not write bad tests. Period. > Let's look at the example of Test::More. The last release has 120 passes and > just 4 failures. > http://cpantesters.perl.org/show/Test-Simple.html#Test-Simple-0.74 > What are those four failures? Three are due to a threading bug in certain > vendor patched versions of perl, one is due to the broken signature test. > Look at the previous gamma release, 0.72. 256 passes, 9 failures. > 5 due to the threading bug, 4 from the signature test. > 0.71: 73 passes, 2 failures. 1 signature, 1 threads > 0.70: 221 passes, 12 failures. 3 signature, 9 threads > And so on. That's nine months with nothing but false negatives. The bug lies in the time span of nine months. Bad tests are bugs, insidious bugs and need to be fixed and shall not be kept for 9 months. > The > signature test is not actually indicating a failure in Test::More, so it's of > no benefit to me or the users, and the bug has already been reported to > Module::Signature. See above. Once the bug is reported there is no justification to keep the test around. In this case I prefer a skip over a removal because the test apparently once was useful. > The threading test is indicating a perl bug that's very difficult to detect > [2], only seems to exist in vendor patched perls, I can't do anything about > and is unlikely to effect anyone since there's so few threads users. It's > already been reported to the various vendors but it'll clear up as soon as > they stop mixing bleadperl patches into 5.8. > In short, I'm paying for somebody else's known bugs. I get nothing. > Test::More gets nothing. The tools get nothing. Cost with no benefit. So > why am I incurring these costs? Maybe the individual users find out their > tools are broken, but it's not my job to tell them that. During smoking CPAN I often find bugs in one module revealed by a test in another one. Only because David Golden tests so hard his tests were well suited to reveal a bug in Test::Harness. I'm glad he doesn't ask if it is his job or not. Just a few RT headlines of the past year: Catalyst:Plugin:Authorization:Roles found a bug in C:P:Authentication. DBI 1.601 broke Exception::Class::DBI. HTML-TreeBuilder-XPath 0.09 broke Web::Scraper. Test::Distribution 1.29 broke Lingua::Stem. Math-BigInt-FastCalc broke Convert::ASN1. Test:Harness 3.0 broke POE. DBM-Deep-1.0006 broke IPC::PubSub. DateTime-Locale-0.35 broke Strptime. Data::Alias 1.06 breaks Devel:EvalContext. Class::Accessor breaks Class::Accessor::Class. DBIx-DBSchema-0.33 breaks Jifty::DBI. File::chdir 0.08 breaks Module::Depends 0.12. Lingua:Stem:It 0.02 breaks the Lingua:Stem testsuite. SVN-Notify-0.26 breaks SVN::Notify::Config (and others). Heap 0.80 breaks Graph. DBI-1.53's NUM_OF_FIELDS change breaks DBD-Sybase 1.07. Getopt-Long 2.36 breaks Verilog::Language. And so on. > I've kept in the threading test because the perl bug it's tickling > does have a direct effect on Test::More, and it could indicate > future threading issues, but lacking any way to resolve it I'm > tempted to pull it. > The signature test, otoh, does not indicate anything that effects > Test::More. Sure it does. There is a symbiotic relation between modules on CPAN. Weigh that in and we're fine. > The ability or inability to check the signature has nothing to do > with the operation of Test::More. So why am I checking it? The > Test::More test suite isn't a full service gas station. It's not > going to wash the windows and check the oil and give you > directions. It makes sure Test::More works and that's that. > As you can see, this is a considered analysis. In general, > redundant tests are ok. They're often not truly redundant but just > have a large overlap. And this all assumes the tests have some > benefit. > And often it's more trouble than it's worth to ferret out test > redundancies. Worse yet is the creeping mental attitude of "should > I write this test? Can I justify the cost?" This is like the > related attitude of "can I justify cleaning up this code?" > Unchecked, both lead to paralysis. > But in this case it's a test which has no benefit to the module, > [for the signature test] indicates no failure of module > functionality, is reporting on known bugs in other tools and is a > test that should be and is done by other modules more directly. > Cost and redundancy with no benefit. Maybe you want to borrow a bit from my Module::Signature test in CPAN.pm. It didn't find a bug in Module::Signature but it is well suited to find other bugs that would be really relevant for me and my users. I'll keep mine in and I hope others do likewise. And I hope nobody tells me to remove working tests. -- andreas
