Re: Bogus PASS from test.pl
On Fri, Apr 18, 2008 at 02:42:05PM +0100, Barbie wrote: > Have you seen the TAP discussions laterly? Yes. Unfortunately. -- David Cantrell | Minister for Arbitrary Justice When a man is tired of London, he is tired of life -- Samuel Johnson
Re: Bogus PASS from test.pl
On Thu, Apr 17, 2008 at 06:15:28PM -0400, David Golden wrote: > On Thu, Apr 17, 2008 at 6:05 PM, David Cantrell <[EMAIL PROTECTED]> wrote: > > I take it that we would say that the test.pl is faulty here, in that it > > exits with a 0 status despite a test failing. However, I betcha that > > there are *lots* of test.pl files out there that suffer from this - I > > know mine did when I used them - so would it be worthwhile for > > CPAN::Reporter etc to check whether it really is a pass? There ARE lots and many are for old releases that have been updated in a long time. > I would rather see test.pl added as a negative Kwalitee score and > manage it that way. DG++ I think there was some discussion regards this very point a while back, but I don't think it ever got implemented. > Trying to "guess" at the intent of tests is a mess. Nail on the head. > We have enough heuristics as is -- there is a > well defined -- if poorly understood -- convention that test.pl should > report success or failure with its exit code. We should leave it at > that. Agreed. We should recommend people use the Test::Simple and friends, with a t/ directory, but second guessing what should be the result is going to be a headache and a half. Have you seen the TAP discussions laterly? Cheers, Barbie. -- Birmingham Perl Mongers - http://birmingham.pm.org Memoirs Of A Roadie - http://barbie.missbarbell.co.uk
Re: Bogus PASS from test.pl
On Thu, Apr 17, 2008 at 6:05 PM, David Cantrell <[EMAIL PROTECTED]> wrote: > I take it that we would say that the test.pl is faulty here, in that it > exits with a 0 status despite a test failing. However, I betcha that > there are *lots* of test.pl files out there that suffer from this - I > know mine did when I used them - so would it be worthwhile for > CPAN::Reporter etc to check whether it really is a pass? > > This could be done with a simple heuristic - if: > * test.pl exists and t/ doesn't; and > * the exit status was 0; and > * /^not ok /; > then turn the test result into UNKNOWN or discard it. No -- I don't want to turn CPAN::Reporter into a parser. What if someone does "^not ok #skip" or whatever the right TAP is? It's actually worse that you think -- Makefile.PL just runs test.pl and takes its exit code, but Build.PL runs test.pl through Test::Harness. I would rather see test.pl added as a negative Kwalitee score and manage it that way. Trying to "guess" at the intent of tests is a mess. Should CPAN::Reporter check for "bad plan" since so many things were broken by a Test::More change? We have enough heuristics as is -- there is a well defined -- if poorly understood -- convention that test.pl should report success or failure with its exit code. We should leave it at that. David
Bogus PASS from test.pl
One of my herd of killer robots wrote: > Running make test > PERL_DL_NONLAZY=1 /usr/local/home/david/cpantesting/perl-5.10.0/bin/perl > "-Iblib/lib" "-Iblib/arch" test.pl > 1..4 > ok 1 > ok 2 > not ok 3 > ok 4 > (/usr/local/bin/make test exited with 0) > CPAN::Reporter: Test result is 'pass', 'make test' no errors. > CPAN::Reporter: preparing a CPAN Testers report for Rstat-Client-2.2 > CPAN::Reporter: sending test report with 'pass' to [EMAIL PROTECTED] > ISAACSON/Rstat-Client-2.2.tar.gz > /usr/local/bin/make test -- OK I take it that we would say that the test.pl is faulty here, in that it exits with a 0 status despite a test failing. However, I betcha that there are *lots* of test.pl files out there that suffer from this - I know mine did when I used them - so would it be worthwhile for CPAN::Reporter etc to check whether it really is a pass? This could be done with a simple heuristic - if: * test.pl exists and t/ doesn't; and * the exit status was 0; and * /^not ok /; then turn the test result into UNKNOWN or discard it. I have also notified the author of this bogus pass, and asked him to CC this 'ere list if he replies. -- David Cantrell | top google result for "topless karaoke murders" I hate baby seals. They get asked to all the best clubs.
