Re: Test skip()'ing weirdness
- Original Message From: Pedro Figueiredo m...@pedrofigueiredo.org Reformatting: Test::Harness 2.56: 1/1 skipped: meh All tests successful, 1 subtest skipped. Test::Harness 2.64: 1/1 skipped: various reasons All tests successful, 1 subtest skipped. Test::Harness 3.14: All tests successful. So Test::Harness 2.56 provided the most useful information. 2.64 had a bug where it said various reasons even if there was only one reason, but at least noted that there were skipped tests. 3.14 drops the ball entirely. I've also verified this problem on 3.01 (since it advertises better skip parsing), so it looks like the series 3 TH has been doing this all the while. I'll try to fix this if I get some tuits. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Tech blog- http://use.perl.org/~Ovid/journal/ Twitter - http://twitter.com/OvidPerl Official Perl 6 Wiki - http://www.perlfoundation.org/perl6
Re: Test skip()'ing weirdness
chromatic wrote: On Thursday 19 February 2009 04:05:37 Ovid wrote: Properly, if we want to report SKIPs for each test (presumably with numbers), then we want to report failing TODOs with each test for consistency's sake. I don't like that. You can already get this behavior with the --directives flag from prove, and it subverts the point of the TODO directive: run the test, but expect it to fail and don't make a big deal of it. If you report failing TODOs (sorry, *passing* TODOs), you might as well report passing oks. I agree. The same argument can be made for skips, they are passing tests and passing tests should be quiet. Which is what I suspect happened sometime in the past and now we've forgotten that decision was made. OTOH normal todo tests have no user serviceable parts. The author expects them to fail. They fail. There's nothing the user can do about that. Informing the average user leads to no new action. It's just clutter. Skipped tests can sometimes be turned on by the user. For example... $ prove -l t/is_deeply_with_threads.t t/is_deeply_with_threadsskipped: many perls have broken threads. Enable with AUTHOR_TESTING. Or $ prove -l t/pod.t t/pod_coverageskipped: Install Pod::Coverage to run Those messages are informative for the user and suggest an action to be taken. Currently prove shows the skip message when the whole test is skipped, but not when individual tests are skipped. If there's a consistency problem, that's it. OTGH I have had more than one novice Perl user wonder what to do about those skip messages. Which brings us back to the principle that passing tests should be quiet. -- ...they shared one last kiss that left a bitter yet sweet taste in her mouth--kind of like throwing up after eating a junior mint. -- Dishonorable Mention, 2005 Bulwer-Lytton Fiction Contest by Tami Farmer
Re: Test skip()'ing weirdness
Michael G Schwern wrote: Skipped tests can sometimes be turned on by the user. For example... $ prove -l t/is_deeply_with_threads.t t/is_deeply_with_threadsskipped: many perls have broken threads. Enable with AUTHOR_TESTING. this is exactly what i do in these skips, check for an envvar. now, the user didn't actually see the skip message (because the skip wasn't reported) but luckily he found it odd that i mentioned it in the readme and yet the tests still reported ok despite him not having set said envvar. cheers, pedro
Test skip()'ing weirdness
Hi,
I've had a report from a user regarding some tests under Darwin (10.5.6,
Leopard, I have no idea if it happens on earlier versions too). I've
since noticed the behaviour under 5.10 on Linux is not what I expected
either.
This is the test code:
#!perl -T
use strict;
use Test::More tests = 1;
SKIP: {
skip( meh, 1 );
ok( 1, dud );
}
Darwin (10.5.6, Leopard)
perl 5.8.8:
$ prove foo.t
foook
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.02 cusr
0.00 csys = 0.04 CPU)
Result: PASS
$ /usr/bin/perl -MTest::More -le 'print $Test::More::VERSION'
0.86
perl 5.10.0:
$ prove foo.t
foook
1/1 skipped: various reasons
All tests successful, 1 subtest skipped.
Files=1, Tests=1, 0 wallclock secs ( 0.02 cusr + 0.00 csys = 0.02 CPU)
$ /usr/local/bin/perl -MTest::More -le 'print $Test::More::VERSION'
0.86
Still not what I expected, but better than 5.8.8.
Linux (Ubuntu Server 8.04)
perl 5.8.8:
$ prove foo.t
foook
1/1 skipped: meh
All tests successful, 1 subtest skipped.
Files=1, Tests=1, 0 wallclock secs ( 0.05 cusr + 0.00 csys = 0.05 CPU)
$ /usr/bin/perl -MTest::More -le 'print $Test::More::VERSION'
0.62
perl 5.10.0:
$ prove foo.t
foook
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.05 usr 0.01 sys + 0.04 cusr
0.00 csys = 0.10 CPU)
Result: PASS
$ /usr/local/bin/perl -MTest::More -le 'print $Test::More::VERSION'
0.86
5.8.8 is the default system Perl in both Darwin and Linux, 5.10.0 was
installed manually by me. Can anyone please shed some light on this?
Cheers,
Pedro
Re: Test skip()'ing weirdness
On Wednesday 18 February 2009 10:14:14 Pedro Figueiredo wrote: 5.8.8 is the default system Perl in both Darwin and Linux, 5.10.0 was installed manually by me. Can anyone please shed some light on this? Check the version of Test::Harness installed; that's what interprets and reports the TAP emitted by Test::More. -- c
Re: Test skip()'ing weirdness
Pedro Figueiredo wrote: I've had a report from a user regarding some tests under Darwin (10.5.6, Leopard, I have no idea if it happens on earlier versions too). I've since noticed the behaviour under 5.10 on Linux is not what I expected either. My orbital mind reading laser got hit by an Iridium satellite. What did you expect? -- 60. The Giant Space Ants are not at the top of my chain of command. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Re: Test skip()'ing weirdness
Actually, I think he was fairly clear. For test runs with skipped tests, you get different outputs: No mention of skipped tests: All tests successful. Mention of skipped tests, but not listing the reasons: 1/1 skipped: various reasons All tests successful, 1 subtest skipped. Mention of skipped tests and listing the reason: 1/1 skipped: meh All tests successful, 1 subtest skipped. So chromatic is right. We need to know which versions of Test::Harness are installed. This looks like buggy behavior to me. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Tech blog- http://use.perl.org/~Ovid/journal/ Twitter - http://twitter.com/OvidPerl Official Perl 6 Wiki - http://www.perlfoundation.org/perl6 - Original Message From: Michael G Schwern schw...@pobox.com To: Pedro Figueiredo m...@pedrofigueiredo.org Cc: perl-qa@perl.org Sent: Wednesday, 18 February, 2009 20:20:50 Subject: Re: Test skip()'ing weirdness Pedro Figueiredo wrote: I've had a report from a user regarding some tests under Darwin (10.5.6, Leopard, I have no idea if it happens on earlier versions too). I've since noticed the behaviour under 5.10 on Linux is not what I expected either. My orbital mind reading laser got hit by an Iridium satellite. What did you expect? -- 60. The Giant Space Ants are not at the top of my chain of command. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Re: Test skip()'ing weirdness
Here are the versions of Test::Harness for each case. On Wed, Feb 18, 2009 at 6:14 PM, Pedro Figueiredo m...@pedrofigueiredo.org wrote: Darwin (10.5.6, Leopard) perl 5.8.8: $ prove foo.t foook All tests successful. Files=1, Tests=1, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.02 cusr 0.00 csys = 0.04 CPU) Result: PASS $ /usr/bin/perl -MTest::More -le 'print $Test::More::VERSION' 0.86 $ /usr/bin/perl -MTest::Harness -le 'print $Test::Harness::VERSION' 3.14 perl 5.10.0: $ prove foo.t foook 1/1 skipped: various reasons All tests successful, 1 subtest skipped. Files=1, Tests=1, 0 wallclock secs ( 0.02 cusr + 0.00 csys = 0.02 CPU) $ /usr/local/bin/perl -MTest::More -le 'print $Test::More::VERSION' 0.86 $ /usr/local/bin/perl -MTest::Harness -le 'print $Test::Harness::VERSION' 2.64 Still not what I expected, but better than 5.8.8. Linux (Ubuntu Server 8.04) perl 5.8.8: $ prove foo.t foook 1/1 skipped: meh All tests successful, 1 subtest skipped. Files=1, Tests=1, 0 wallclock secs ( 0.05 cusr + 0.00 csys = 0.05 CPU) $ /usr/bin/perl -MTest::More -le 'print $Test::More::VERSION' 0.62 $ /usr/bin/perl -MTest::Harness -le 'print $Test::Harness::VERSION' 2.56 perl 5.10.0: $ prove foo.t foook All tests successful. Files=1, Tests=1, 0 wallclock secs ( 0.05 usr 0.01 sys + 0.04 cusr 0.00 csys = 0.10 CPU) Result: PASS $ /usr/local/bin/perl -MTest::More -le 'print $Test::More::VERSION' 0.86 $ /usr/local/bin/perl -MTest::Harness -le 'print $Test::Harness::VERSION' 3.14 Cheers, Pedro -- http://pedrofigueiredo.org/ you don't code php. you merely edit it until it works. - merlyn
