Re: What's the point of a SIGNATURE test?
Andreas J. Koenig wrote: On Fri, 14 Dec 2007 15:49:32 -0800, Michael G Schwern [EMAIL PROTECTED] said: We would seem to be agreeing. If the goal of the test suite is not to protect against spoofing, and if it doesn't accomplish that anyway, why put a signature check in there? Of course we are agreeing 99%. But I'm citing the Michael Schwern saying that is dearer to me than the above paragraph: tests are there to find bugs. I say lots of apparently contradictory things. The trick is knowing when one rule wins out over the other. Something to keep in mind is that I'm talking about one very specific test. Don't let this discussion get tangled up in the author tests brouhaha that often brews up around here. [...] But if the test failed because it's a bad test, Clearly a strawman's argument. It's impossible to contradict you on this. Thou shalt not write bad tests. Period. That was supposed to come out more like if the test failed because of a mistake in the test suite. You know the sort of thing. Like when you write: like $error, qr/your shit is broke at $0 line \d+\.\n/; and it blows up on Windows because you forgot about the backslashes in Windows path names. The test failure indicates a bug in the test, not the code. Thus, the failure has a cost and no benefit. The signature test is not actually indicating a failure in Test::More, so it's of no benefit to me or the users, and the bug has already been reported to Module::Signature. See above. Once the bug is reported there is no justification to keep the test around. In this case I prefer a skip over a removal because the test apparently once was useful. Bt skipped tests don't get run so it's effectively deleted, except a permanently skipped test sits around cluttering things up. Smells like commenting out code that maybe someday you might want to use again in the future. Just adds clutter. If I want to bring a test (or code) back from the dead that's what version control is for. The threading test is indicating a perl bug that's very difficult to detect [2], only seems to exist in vendor patched perls, I can't do anything about and is unlikely to effect anyone since there's so few threads users. It's already been reported to the various vendors but it'll clear up as soon as they stop mixing bleadperl patches into 5.8. In short, I'm paying for somebody else's known bugs. I get nothing. Test::More gets nothing. The tools get nothing. Cost with no benefit. So why am I incurring these costs? Maybe the individual users find out their tools are broken, but it's not my job to tell them that. During smoking CPAN I often find bugs in one module revealed by a test in another one... Only because David Golden tests so hard his tests were well suited to reveal a bug in Test::Harness. I'm glad he doesn't ask if it is his job or not. Just a few RT headlines of the past year:... Catalyst:Plugin:Authorization:Roles found a bug in C:P:Authentication. DBI 1.601 broke Exception::Class::DBI. HTML-TreeBuilder-XPath 0.09 broke Web::Scraper. Test::Distribution 1.29 broke Lingua::Stem. Math-BigInt-FastCalc broke Convert::ASN1. Test:Harness 3.0 broke POE. DBM-Deep-1.0006 broke IPC::PubSub. DateTime-Locale-0.35 broke Strptime. Data::Alias 1.06 breaks Devel:EvalContext. Class::Accessor breaks Class::Accessor::Class. DBIx-DBSchema-0.33 breaks Jifty::DBI. File::chdir 0.08 breaks Module::Depends 0.12. Lingua:Stem:It 0.02 breaks the Lingua:Stem testsuite. SVN-Notify-0.26 breaks SVN::Notify::Config (and others). Heap 0.80 breaks Graph. DBI-1.53's NUM_OF_FIELDS change breaks DBD-Sybase 1.07. Getopt-Long 2.36 breaks Verilog::Language. And so on. I agree that's all very useful. Interlocking dependent test suites ferret out bugs the original authors wouldn't find. However, there is a very important difference between the list above and Test::More's signature test. On a quick scan, all of those modules have direct dependencies. DBD::Sybase uses DBI, Lingua::Stem uses Test::Distribution, etc... so it's natural that their tests would test their dependencies. If a dependency breaks, they break. I'm sure most of the authors above did not set out with the intention to test their dependencies, it's all inherent in the testing of their own code. Test::More doesn't actually use Module::Signature, so why is it testing it? It would be like if DBI decided to add a test to make sure MakeMaker can read the MANIFEST. Sure it's useful to know that part of the toolchain works and that the MANIFEST can be read, but why is that in DBI? One can argue that DBI depends on the good functioning of ExtUtils::Manifest to install, so it should test it. Ok, then what about all the other things DBI depends on to install? Should it test that MakeMaker can make a valid Makefile? Should it test that tar and gzip work? Should I check that CPAN.pm can properly
Re: Milton Keynes PM coding collaboration
On Tue, 11 Dec 2007, Edwardson, Tony wrote: Anyone written any CPAN modules for which the testing coverage needs to be improved ? Oh, yes. Want someone else to sort this out for you ? Yes. Milton Keynes PM want to start working together to contribute to the Perl development effort and as we need to do something that beginners and experts can both work on, we thought that improving the testing coverage of popular CPAN Modules would be a good initial project. So, we are looking for popular CPAN modules which could do with a better test suite and the Author is willing to let others work in it on their behalf. Log::Dispatch might be a possibility. The tests are old and crufty, and probably don't cover lots of corner cases and features. I'd be happy to see the test suite refactored a bit into multiple files. It could also use better mocking so we could test things like syslog and email. -dave /*=== VegGuide.Orgwww.BookIRead.com Your guide to all that's veg. My book blog ===*/
Re: What's the point of a SIGNATURE test?
On Saturday 15 December 2007 01:34:37 Michael G Schwern wrote: Bt skipped tests don't get run so it's effectively deleted, except a permanently skipped test sits around cluttering things up. Smells like commenting out code that maybe someday you might want to use again in the future. Just adds clutter. If I want to bring a test (or code) back from the dead that's what version control is for. Some people (not me; I think they're slightly NUTS) believe that it should be possible to fork a CPAN module from the CPAN sources only, and thus every tarball of every distribution should contain everything necessary to take over maintainership of a module. I say that people who want to know what the silly POD-checking tests look like should be able to find a thousand verbatim instances on the CPAN, and anyone who can't isn't fit to take over maintainership of someone else's module. -- c
[ANNOUNCE] TAP::Harness::Archive 0.03
The uploaded file
TAP-Harness-Archive-0.03.tar.gz
has entered CPAN as
file: $CPAN/authors/id/W/WO/WONKO/TAP-Harness-Archive-0.03.tar.gz
size: 5995 bytes
md5: 5a92a2ae5b43469229a68146703d
The biggest change in this release is the removal of Archive::Builder and
Archive::Extract as dependencies since they had s many other dependencies
that our chance of failure on install we greater.
SYNOPSIS
use TAP::Harness::Archive;
my $harness = TAP::Harness::Archive-new(\%args);
$harness-runtests(@tests);
DESCRIPTION
This module is a direct subclass of TAP::Harness and behaves in exactly
the same way except for one detail. In addition to outputting a running
progress of the tests and an ending summary it can also capture all of
the raw TAP from the individual test files or streams into an archive
file (.tar or .tar.gz).
METHODS
All methods are exactly the same as our base TAP::Harness except for the
following.
new
In addition to the options that TAP::Harness allow to this method, we
also allow the following:
archive
This is the name of the archive file to generate. We use
Archive::Tar in the background so we only support .tar and
.tar.gz archive file formats.
runtests
Takes the same arguments as TAP::Harness's version and returns the same
thing (a TAP::Parser::Aggregator object). The only difference is that in
addition to the normal test running and progress output we also create
the TAP Archive when it's all done.
aggregator_from_archive
This class method will return a TAP::Parser::Aggregator object when
given a TAP Archive to open and parse. It's pretty much the reverse of
creating a TAP Archive from using new and runtests.
It takes a hash of arguments which are as follows:
archive
The path to the archive file. This is required.
parser_callbacks
This is a hash ref containing callbacks for the TAP::Parser objects
that are created while parsing the TAP files. See the TAP::Parser
documentation for details about these callbacks.
made_parser_callback
This callback is executed every time a new TAP::Parser object is
created. It will be passed the new parser object and the name of the
file to be parsed.
meta_yaml_callback
This is a subroutine that will be called if we find and parse a YAML
file containing meta information about the test run in the archive.
The structure of the YAML file will be passed in as an argument.
my $aggregator = TAP::Harness::Archive-aggregator_from_archive(
{
archive = 'my_tests.tar.gz',
parser_callbacks = {
plan= sub { warn Nice to see you plan ahead... },
unknown = sub { warn Your TAP is bad! },
},
}
);
--
Michael Peters
Developer
Plus Three, LP
Re: What's the point of a SIGNATURE test?
On Fri, 14 Dec 2007 15:49:32 -0800, Michael G Schwern [EMAIL PROTECTED] said: Asking the wrong question. None of our testsuites is there to protect against spoof or attacks. That's simply not the goal. Same thing for 00-signature.t We would seem to be agreeing. If the goal of the test suite is not to protect against spoofing, and if it doesn't accomplish that anyway, why put a signature check in there? Of course we are agreeing 99%. But I'm citing the Michael Schwern saying that is dearer to me than the above paragraph: tests are there to find bugs. Yupp. And testing the signature in a test is better than not testing it because a bug in a signature or in crypto software is as alarming as a bug in perl or a module. I believe this to be outside the scope of a given module's tests. It's not the responsibility of every CPAN module to make sure that your crypto software is working. Or perl. Or the C compiler. Or make. That's the job of the toolchain modules which more directly use them (CPAN, Module::Signature, MakeMaker, Module::Build, etc...). [1] You're barking up the wrong tree. Stick to your older wisdom that bugs are there to find tests, err, forgive me, tests are there to find bugs, and when a test found a bug it is a hero forever. At some point you have to trust that the tools work, you can't test the whole universe. You simply don't have the time. I agree totally. But you also do not have the wisdom that you can predict which tests will find bugs in which software. Your test found a bug in Module::Signature? Great, so it was a good test. Of course you should not annoy people with a test once the bug is known and reported. Make a new release, skip the test with the skip statement 'Module::Signature 123.45 known bug see RT #54321' and wait for the next time the test finds a bug once Module::Signature gets upgraded. Besides that I agree with the rest of your musings. Great writup. Some guidelines about cost/benefit relations are worthwhile but one should take care not to make them too narrow. The last thing we want to have is a new kind of CPAN police dictating people to remove tests. [...] But if the test failed because it's a bad test, Clearly a strawman's argument. It's impossible to contradict you on this. Thou shalt not write bad tests. Period. Let's look at the example of Test::More. The last release has 120 passes and just 4 failures. http://cpantesters.perl.org/show/Test-Simple.html#Test-Simple-0.74 What are those four failures? Three are due to a threading bug in certain vendor patched versions of perl, one is due to the broken signature test. Look at the previous gamma release, 0.72. 256 passes, 9 failures. 5 due to the threading bug, 4 from the signature test. 0.71: 73 passes, 2 failures. 1 signature, 1 threads 0.70: 221 passes, 12 failures. 3 signature, 9 threads And so on. That's nine months with nothing but false negatives. The bug lies in the time span of nine months. Bad tests are bugs, insidious bugs and need to be fixed and shall not be kept for 9 months. The signature test is not actually indicating a failure in Test::More, so it's of no benefit to me or the users, and the bug has already been reported to Module::Signature. See above. Once the bug is reported there is no justification to keep the test around. In this case I prefer a skip over a removal because the test apparently once was useful. The threading test is indicating a perl bug that's very difficult to detect [2], only seems to exist in vendor patched perls, I can't do anything about and is unlikely to effect anyone since there's so few threads users. It's already been reported to the various vendors but it'll clear up as soon as they stop mixing bleadperl patches into 5.8. In short, I'm paying for somebody else's known bugs. I get nothing. Test::More gets nothing. The tools get nothing. Cost with no benefit. So why am I incurring these costs? Maybe the individual users find out their tools are broken, but it's not my job to tell them that. During smoking CPAN I often find bugs in one module revealed by a test in another one. Only because David Golden tests so hard his tests were well suited to reveal a bug in Test::Harness. I'm glad he doesn't ask if it is his job or not. Just a few RT headlines of the past year: Catalyst:Plugin:Authorization:Roles found a bug in C:P:Authentication. DBI 1.601 broke Exception::Class::DBI. HTML-TreeBuilder-XPath 0.09 broke Web::Scraper. Test::Distribution 1.29 broke Lingua::Stem. Math-BigInt-FastCalc broke Convert::ASN1. Test:Harness 3.0 broke POE. DBM-Deep-1.0006 broke IPC::PubSub. DateTime-Locale-0.35 broke Strptime. Data::Alias 1.06 breaks Devel:EvalContext. Class::Accessor breaks Class::Accessor::Class. DBIx-DBSchema-0.33 breaks Jifty::DBI. File::chdir 0.08 breaks Module::Depends 0.12. Lingua:Stem:It 0.02 breaks the
Re: What's the point of a SIGNATURE test?
* chromatic [EMAIL PROTECTED] [2007-12-15 19:10]: thus every tarball of every distribution should contain everything necessary to take over maintainership of a module. It is a reasonable position, really, so long as you don’t stretch it to absurd lengths. If there is something *unusual* involved in the construction of a release of the module, then the code that performs that task absolutely should be part of the distribution. I say that people who want to know what the silly POD-checking tests look like should be able to find a thousand verbatim instances on the CPAN My first act in office as new maintainer (be it ownership change or fork) is often to throw out its toolchain plumbing and install the infrastructure I prefer. Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: Milton Keynes PM coding collaboration
On Dec 11, 2007, at 12:27 AM, Edwardson, Tony wrote: Anyone written any CPAN modules for which the testing coverage needs to be improved ? Sure - one that particularly bothers me is Apache2::AuthCookieDBI - back when I took over maintenance of that module I didn't know how to test it, but it certainly could have unit tests, and currently has none. http://search.cpan.org/dist/Apache2-AuthCookieDBI Also, another one would be Perl::Metrics::Simple which has decent test coverage but I would love to see it have 100% branch coverage (a reported by Devel::Cover) http://search.cpan.org/dist/Perl-Metrics-Simple/ -Matisse --- Matisse Enzer [EMAIL PROTECTED] http://www.matisse.net/ - http://www.eigenstate.net/
