Re: Milton Keynes PM coding collaboration

[Matisse Enzer]

On Dec 11, 2007, at 12:27 AM, Edwardson, Tony wrote:

Anyone written any CPAN modules for which the testing coverage needs  
to be

improved ?


Sure - one that particularly bothers me is Apache2::AuthCookieDBI -  
back when I took over maintenance of that module I didn't know how to  
test it, but it certainly could have unit tests, and currently has none.

http://search.cpan.org/dist/Apache2-AuthCookieDBI

Also, another one would be Perl::Metrics::Simple which has decent test  
coverage but I would love to see it have 100% branch coverage (a  
reported by Devel::Cover)

http://search.cpan.org/dist/Perl-Metrics-Simple/

-Matisse

---
Matisse Enzer <[EMAIL PROTECTED]>
http://www.matisse.net/  - http://www.eigenstate.net/

Re: What's the point of a SIGNATURE test?

[A. Pagaltzis]

* chromatic <[EMAIL PROTECTED]> [2007-12-15 19:10]:
> thus every tarball of every distribution should contain
> everything necessary to take over maintainership of a module.

It is a reasonable position, really, so long as you don’t stretch
it to absurd lengths. If there is something *unusual* involved in
the construction of a release of the module, then the code that
performs that task absolutely should be part of the distribution.

> I say that people who want to know what the silly POD-checking
> tests look like should be able to find a thousand verbatim
> instances on the CPAN

My first act in office as new maintainer (be it ownership change
or fork) is often to throw out its toolchain plumbing and install
the infrastructure I prefer.

Regards,
-- 
Aristotle Pagaltzis // 

Re: What's the point of a SIGNATURE test?

[Andreas J. Koenig]

> On Fri, 14 Dec 2007 15:49:32 -0800, Michael G Schwern <[EMAIL PROTECTED]> 
> said:

 >> Asking the wrong question. None of our testsuites is there to protect
 >> against spoof or attacks. That's simply not the goal. Same thing for
 >> 00-signature.t

  > We would seem to be agreeing.  If the goal of the test suite is not to 
protect
  > against spoofing, and if it doesn't accomplish that anyway, why put a
  > signature check in there?

Of course we are agreeing 99%. But I'm citing the Michael Schwern
saying that is dearer to me than the above paragraph: tests are there
to find bugs.

 >> Yupp. And testing the signature in a test is better than not
 >> testing it because a bug in a signature or in crypto software is
 >> as alarming as a bug in perl or a module.

  > I believe this to be outside the scope of a given module's tests.  It's not
  > the responsibility of every CPAN module to make sure that your crypto 
software
  > is working.  Or perl.  Or the C compiler.  Or make.  That's the job of the
  > toolchain modules which more directly use them (CPAN, Module::Signature,
  > MakeMaker, Module::Build, etc...). [1]

You're barking up the wrong tree. Stick to your older wisdom that bugs
are there to find tests, err, forgive me, tests are there to find
bugs, and when a test found a bug it is a hero forever.

  > At some point you have to trust that the tools work, you can't test the 
whole
  > universe.  You simply don't have the time.

I agree totally. But you also do not have the wisdom that you can
predict which tests will find bugs in which software. Your test found
a bug in Module::Signature? Great, so it was a good test.

Of course you should not annoy people with a test once the bug is
known and reported. Make a new release, skip the test with the skip
statement 'Module::Signature 123.45 known bug see RT #54321' and wait
for the next time the test finds a bug once Module::Signature gets
upgraded.

Besides that I agree with the rest of your musings. Great writup. Some
guidelines about cost/benefit relations are worthwhile but one should
take care not to make them too narrow. The last thing we want to have
is a new kind of CPAN police dictating people to remove tests.

  > [...] But if the test failed because it's a bad test,

Clearly a strawman's argument. It's impossible to contradict you on
this. Thou shalt not write bad tests. Period.

  > Let's look at the example of Test::More.  The last release has 120 passes 
and
  > just 4 failures.
  > http://cpantesters.perl.org/show/Test-Simple.html#Test-Simple-0.74

  > What are those four failures?  Three are due to a threading bug in certain
  > vendor patched versions of perl, one is due to the broken signature test.

  > Look at the previous gamma release, 0.72.  256 passes, 9 failures.
  > 5 due to the threading bug, 4 from the signature test.

  > 0.71:  73 passes, 2 failures.  1 signature, 1 threads

  > 0.70:  221 passes, 12 failures.  3 signature, 9 threads

  > And so on.  That's nine months with nothing but false negatives.

The bug lies in the time span of nine months. Bad tests are bugs,
insidious bugs and need to be fixed and shall not be kept for 9
months.

  > The
  > signature test is not actually indicating a failure in Test::More, so it's 
of
  > no benefit to me or the users, and the bug has already been reported to
  > Module::Signature.

See above. Once the bug is reported there is no justification to keep
the test around. In this case I prefer a skip over a removal because
the test apparently once was useful.

  > The threading test is indicating a perl bug that's very difficult to detect
  > [2], only seems to exist in vendor patched perls, I can't do anything about
  > and is unlikely to effect anyone since there's so few threads users.  It's
  > already been reported to the various vendors but it'll clear up as soon as
  > they stop mixing bleadperl patches into 5.8.

  > In short, I'm paying for somebody else's known bugs.  I get nothing.
  > Test::More gets nothing.  The tools get nothing.  Cost with no benefit.  So
  > why am I incurring these costs?  Maybe the individual users find out their
  > tools are broken, but it's not my job to tell them that.

During smoking CPAN I often find bugs in one module revealed by a test
in another one. Only because David Golden tests so hard his tests were
well suited to reveal a bug in Test::Harness. I'm glad he doesn't ask
if it is his job or not. Just a few RT headlines of the past year:
Catalyst:Plugin:Authorization:Roles found a bug in C:P:Authentication.
DBI 1.601 broke Exception::Class::DBI. HTML-TreeBuilder-XPath 0.09
broke Web::Scraper. Test::Distribution 1.29 broke Lingua::Stem.
Math-BigInt-FastCalc broke Convert::ASN1. Test:Harness 3.0 broke POE.
DBM-Deep-1.0006 broke IPC::PubSub. DateTime-Locale-0.35 broke
Strptime. Data::Alias 1.06 breaks Devel:EvalContext. Class::Accessor
breaks Class::Accessor::Class. DBIx-DBSchema-0.33 breaks Jifty::DBI.
File::chdir 0.08 breaks Mod

[ANNOUNCE] TAP::Harness::Archive 0.03

[Michael Peters]

The uploaded file

TAP-Harness-Archive-0.03.tar.gz

has entered CPAN as

  file: $CPAN/authors/id/W/WO/WONKO/TAP-Harness-Archive-0.03.tar.gz
  size: 5995 bytes
   md5: 5a92a2ae5b43469229a68146703d

The biggest change in this release is the removal of Archive::Builder and
Archive::Extract as dependencies since they had s many other dependencies
that our chance of failure on install we greater.

SYNOPSIS
use TAP::Harness::Archive;
my $harness = TAP::Harness::Archive->new(\%args);
$harness->runtests(@tests);

DESCRIPTION
This module is a direct subclass of TAP::Harness and behaves in exactly
the same way except for one detail. In addition to outputting a running
progress of the tests and an ending summary it can also capture all of
the raw TAP from the individual test files or streams into an archive
file (".tar" or ".tar.gz").

METHODS
All methods are exactly the same as our base TAP::Harness except for the
following.

  new
In addition to the options that TAP::Harness allow to this method, we
also allow the following:

archive
This is the name of the archive file to generate. We use
Archive::Tar in the background so we only support ".tar" and
".tar.gz" archive file formats.

  runtests
Takes the same arguments as TAP::Harness's version and returns the same
thing (a TAP::Parser::Aggregator object). The only difference is that in
addition to the normal test running and progress output we also create
the TAP Archive when it's all done.

  aggregator_from_archive
This class method will return a TAP::Parser::Aggregator object when
given a TAP Archive to open and parse. It's pretty much the reverse of
creating a TAP Archive from using "new" and "runtests".

It takes a hash of arguments which are as follows:

archive
The path to the archive file. This is required.

parser_callbacks
This is a hash ref containing callbacks for the TAP::Parser objects
that are created while parsing the TAP files. See the TAP::Parser
documentation for details about these callbacks.

made_parser_callback
This callback is executed every time a new TAP::Parser object is
created. It will be passed the new parser object and the name of the
file to be parsed.

meta_yaml_callback
This is a subroutine that will be called if we find and parse a YAML
file containing meta information about the test run in the archive.
The structure of the YAML file will be passed in as an argument.

my $aggregator = TAP::Harness::Archive->aggregator_from_archive(
{
archive  => 'my_tests.tar.gz',
parser_callbacks => {
plan=> sub { warn "Nice to see you plan ahead..." },
unknown => sub { warn "Your TAP is bad!" },
},
}
);

-- 
Michael Peters
Developer
Plus Three, LP

Re: What's the point of a SIGNATURE test?

[chromatic]

On Saturday 15 December 2007 01:34:37 Michael G Schwern wrote:

> Bt skipped tests don't get run so it's effectively deleted, except a
> permanently skipped test sits around cluttering things up.  Smells like
> commenting out code that maybe someday you might want to use again in the
> future.  Just adds clutter.
>
> If I want to bring a test (or code) back from the dead that's what version
> control is for.

Some people (not me; I think they're slightly NUTS) believe that it should be 
possible to fork a CPAN module from the CPAN sources only, and thus every 
tarball of every distribution should contain everything necessary to take 
over maintainership of a module.

I say that people who want to know what the silly POD-checking tests look like 
should be able to find a thousand verbatim instances on the CPAN, and anyone 
who can't isn't fit to take over maintainership of someone else's module.

-- c

Re: Milton Keynes PM coding collaboration

[Dave Rolsky]

On Tue, 11 Dec 2007, Edwardson, Tony wrote:


Anyone written any CPAN modules for which the testing coverage needs to be
improved ?


Oh, yes.


Want someone else to sort this out for you ?


Yes.


Milton Keynes PM  want to start working together to contribute to the Perl
development effort and as we need to do something that beginners and experts
can both work on, we thought that improving the testing coverage of popular
CPAN Modules would be a good initial project.

So, we are looking for popular CPAN modules which could do with a better
test suite and the Author is willing to let others work in it on their
behalf.


Log::Dispatch might be a possibility. The tests are old and crufty, and 
probably don't cover lots of corner cases and features. I'd be happy to 
see the test suite refactored a bit into multiple files. It could also use 
better mocking so we could test things like syslog and email.



-dave

/*===
VegGuide.Orgwww.BookIRead.com
Your guide to all that's veg.   My book blog
===*/

Re: Parsing TAP into TAP

[Andy Armstrong]

On 15 Dec 2007, at 02:05, Ricardo SIGNES wrote:


* Andy Armstrong <[EMAIL PROTECTED]> [2007-12-10T11:52:33]

On 10 Dec 2007, at 16:49, Ovid wrote:

Seems Ricardo Signes likes this idea, too:

http://use.perl.org/~Ovid/journal/35076



Who? :)


:'(

I like this idea so much that if you point me at a repo and failing  
tests, I

will try to be a little helpful. :)



You have commit (mailed offlist).

A few minutes after each commit a bunch of automated test results show  
up on


http://hexten.net/mailman/listinfo/tapx-commit

and discussion happens here

http://hexten.net/mailman/listinfo/tapx-dev

Outputting TAP /should/ just be a case of plugging in a different  
formatter. See TAP::Formatter::Console and friends for info.


--
Andy Armstrong, Hexten

Re: What's the point of a SIGNATURE test?

[Michael G Schwern]

Andreas J. Koenig wrote:
>> On Fri, 14 Dec 2007 15:49:32 -0800, Michael G Schwern <[EMAIL 
>> PROTECTED]> said:
>   > We would seem to be agreeing.  If the goal of the test suite is not to 
> protect
>   > against spoofing, and if it doesn't accomplish that anyway, why put a
>   > signature check in there?
> 
> Of course we are agreeing 99%. But I'm citing the Michael Schwern
> saying that is dearer to me than the above paragraph: tests are there
> to find bugs.

I say lots of apparently contradictory things.  The trick is knowing when one
rule wins out over the other.

Something to keep in mind is that I'm talking about one very specific test.
Don't let this discussion get tangled up in the "author tests" brouhaha that
often brews up around here.


>   > [...] But if the test failed because it's a bad test,
> 
> Clearly a strawman's argument. It's impossible to contradict you on
> this. Thou shalt not write bad tests. Period.

That was supposed to come out more like "if the test failed because of a
mistake in the test suite".  You know the sort of thing.  Like when you write:

like $error, qr/your shit is broke at $0 line \d+\.\n/;

and it blows up on Windows because you forgot about the backslashes in Windows
path names.  The test failure indicates a bug in the test, not the code.
Thus, the failure has a cost and no benefit.


>   > The
>   > signature test is not actually indicating a failure in Test::More, so 
> it's of
>   > no benefit to me or the users, and the bug has already been reported to
>   > Module::Signature.
> 
> See above. Once the bug is reported there is no justification to keep
> the test around. In this case I prefer a skip over a removal because
> the test apparently once was useful.

Bt skipped tests don't get run so it's effectively deleted, except a
permanently skipped test sits around cluttering things up.  Smells like
commenting out code that maybe someday you might want to use again in the
future.  Just adds clutter.

If I want to bring a test (or code) back from the dead that's what version
control is for.


>   > The threading test is indicating a perl bug that's very difficult to 
> detect
>   > [2], only seems to exist in vendor patched perls, I can't do anything 
> about
>   > and is unlikely to effect anyone since there's so few threads users.  It's
>   > already been reported to the various vendors but it'll clear up as soon as
>   > they stop mixing bleadperl patches into 5.8.
> 
>   > In short, I'm paying for somebody else's known bugs.  I get nothing.
>   > Test::More gets nothing.  The tools get nothing.  Cost with no benefit.  
> So
>   > why am I incurring these costs?  Maybe the individual users find out their
>   > tools are broken, but it's not my job to tell them that.
> 
> During smoking CPAN I often find bugs in one module revealed by a test
> in another one... Only because David Golden tests so hard his tests were
> well suited to reveal a bug in Test::Harness. I'm glad he doesn't ask
> if it is his job or not. Just a few RT headlines of the past year:...
> Catalyst:Plugin:Authorization:Roles found a bug in C:P:Authentication.
> DBI 1.601 broke Exception::Class::DBI. HTML-TreeBuilder-XPath 0.09
> broke Web::Scraper. Test::Distribution 1.29 broke Lingua::Stem.
> Math-BigInt-FastCalc broke Convert::ASN1. Test:Harness 3.0 broke POE.
> DBM-Deep-1.0006 broke IPC::PubSub. DateTime-Locale-0.35 broke
> Strptime. Data::Alias 1.06 breaks Devel:EvalContext. Class::Accessor
> breaks Class::Accessor::Class. DBIx-DBSchema-0.33 breaks Jifty::DBI.
> File::chdir 0.08 breaks Module::Depends 0.12. Lingua:Stem:It 0.02
> breaks the Lingua:Stem testsuite. SVN-Notify-0.26 breaks
> SVN::Notify::Config (and others). Heap 0.80 breaks Graph. DBI-1.53's
> NUM_OF_FIELDS change breaks DBD-Sybase 1.07. Getopt-Long 2.36 breaks
> Verilog::Language. And so on.

I agree that's all very useful.  Interlocking dependent test suites ferret out
bugs the original authors wouldn't find.  However, there is a very important
difference between the list above and Test::More's signature test.

On a quick scan, all of those modules have direct dependencies.  DBD::Sybase
uses DBI, Lingua::Stem uses Test::Distribution, etc... so it's natural that
their tests would test their dependencies.  If a dependency breaks, they
break.  I'm sure most of the authors above did not set out with the intention
to test their dependencies, it's all inherent in the testing of their own code.

Test::More doesn't actually use Module::Signature, so why is it testing it?

It would be like if DBI decided to add a test to make sure MakeMaker can read
the MANIFEST.  Sure it's useful to know that part of the toolchain works and
that the MANIFEST can be read, but why is that in DBI?  One can argue that DBI
depends on the good functioning of ExtUtils::Manifest to install, so it should
test it.  Ok, then what about all the other things DBI depends on to install?
 Should it test that MakeMaker can make a valid Makefile?