Re: TODO tests and test::harness
On Wed, Apr 19, 2006 at 07:22:33AM +0200, demerphq wrote:
On 4/19/06, Andy Lester [EMAIL PROTECTED] wrote:
BTW, the patch only shows TODO pass status when no failures occur.
Oh and obviously all of Test::Harness'es tests pass. :-)
This patch doesn't apply against my latest dev version of
Test::Harness. I'm going to have to massage it manually.
But I like the idea. Thanks.
You're welcome. If it helps It was against Test-Harness-2.56.
Maybe I'm thinking too hard, or maybe the results reported aren't
exactly as clear as they probably should be. Here's an example test and
its results as reported by Test::Harness with the TODO changes.
#!perl -w
use strict;
use Test::More qw(no_plan);
TODO: {
local $TODO = TODO testing;
is(1, 2, A failing test);
is(1, 1, A passing test);
}
[EMAIL PROTECTED]:~/smoke/perl-current/t$ ./perl harness th_test.t
th_testok
1/2 unexpectedly succeeded
TODO PASSED tests 1-2
All tests successful (1 subtest UNEXPECTEDLY SUCCEEDED).
Passed Test Stat Wstat Total Pass Passed List of Passed
---
th_test.t 21 50.00% 1-2
Files=1, Tests=2, 0 wallclock secs ( 0.11 cusr + 0.01 csys = 0.12
CPU)
The line starting TODO PASSED shows all TODO tests, not those that
unexpectedly succeeded, which confused me a bit. Also, the final
results show that one test passed, but then the list of passed is 1-2
instead of just 2 which is the unexpected success. Is there a way to
have the list of passed just show the unexpected successes?
Steve Peters
[EMAIL PROTECTED]
signature.asc
Description: Digital signature
Re: TODO tests and test::harness
On 4/20/06, Steve Peters [EMAIL PROTECTED] wrote:
On Wed, Apr 19, 2006 at 07:22:33AM +0200, demerphq wrote:
On 4/19/06, Andy Lester [EMAIL PROTECTED] wrote:
BTW, the patch only shows TODO pass status when no failures occur.
Oh and obviously all of Test::Harness'es tests pass. :-)
This patch doesn't apply against my latest dev version of
Test::Harness. I'm going to have to massage it manually.
But I like the idea. Thanks.
You're welcome. If it helps It was against Test-Harness-2.56.
Maybe I'm thinking too hard, or maybe the results reported aren't
exactly as clear as they probably should be.
I think thats probably true.
Here's an example test and its results as reported by Test::Harness with the
TODO changes.
#!perl -w
use strict;
use Test::More qw(no_plan);
TODO: {
local $TODO = TODO testing;
is(1, 2, A failing test);
is(1, 1, A passing test);
}
[EMAIL PROTECTED]:~/smoke/perl-current/t$ ./perl harness th_test.t
th_testok
1/2 unexpectedly succeeded
TODO PASSED tests 1-2
All tests successful (1 subtest UNEXPECTEDLY SUCCEEDED).
Passed Test Stat Wstat Total Pass Passed List of Passed
---
th_test.t 21 50.00% 1-2
Files=1, Tests=2, 0 wallclock secs ( 0.11 cusr + 0.01 csys = 0.12
CPU)
The line starting TODO PASSED shows all TODO tests, not those that
unexpectedly succeeded, which confused me a bit. Also, the final
results show that one test passed, but then the list of passed is 1-2
instead of just 2 which is the unexpected success. Is there a way to
have the list of passed just show the unexpected successes?
I have to admit im flummoxed on this one. The first number '2' is the
number of tests in the file. The next number '1' is the number of
problematic results. The %50.00 shows the percentage that are
problematic. So everything checks out up till there. But then the list
of failures is wrong. Which I dont get at all.
I had a look at the results when i did the patch i definately didnt
see this result. So far i dont see the cause, but i do see a subtle
bug that I hadnt noticed before.
The line that says:
failed = $test{bonus},
should probably read
failed = scalar @{$test{todo_pass}}
But i still dont see why the list is wrong. Ill have to investigate
further later.
Cheers,
Yves
--
perl -Mre=debug -e /just|another|perl|hacker/
Re: TODO tests and test::harness
On Thu, Apr 20, 2006 at 10:36:08PM +0200, demerphq wrote:
On 4/20/06, Steve Peters [EMAIL PROTECTED] wrote:
Maybe I'm thinking too hard, or maybe the results reported aren't
exactly as clear as they probably should be. Here's an example test and
its results as reported by Test::Harness with the TODO changes.
#!perl -w
use strict;
use Test::More qw(no_plan);
TODO: {
local $TODO = TODO testing;
is(1, 2, A failing test);
is(1, 1, A passing test);
}
[EMAIL PROTECTED]:~/smoke/perl-current/t$ ./perl harness th_test.t
th_testok
1/2 unexpectedly succeeded
TODO PASSED tests 1-2
All tests successful (1 subtest UNEXPECTEDLY SUCCEEDED).
Passed Test Stat Wstat Total Pass Passed List of Passed
---
th_test.t 21 50.00% 1-2
Files=1, Tests=2, 0 wallclock secs ( 0.11 cusr + 0.01 csys = 0.12
CPU)
The line starting TODO PASSED shows all TODO tests, not those that
unexpectedly succeeded, which confused me a bit. Also, the final
results show that one test passed, but then the list of passed is 1-2
instead of just 2 which is the unexpected success. Is there a way to
have the list of passed just show the unexpected successes?
Attached patch should fix it up. Both in terms of making it clearer
and of fixing the list. So your test file would look like:
All tests successful (1 subtest UNEXPECTEDLY SUCCEEDED), 37 subtests skipped.
Passed Todo Stat Wstat Todos Pass Passed List of Passed
---
t/demerphq.t21 50.00% 3
Files=19, Tests=572, 8 wallclock secs ( 0.00 cusr + 0.00 csys = 0.00 CPU)
Hopefully thats clearer. The Todos column shows how many todos there
are in the file.
Excellent! It seems to be working more as I was hoping to see. This
patch was applied as change #27925.
Steve Peters
[EMAIL PROTECTED]
signature.asc
Description: Digital signature
Re: TODO tests and test::harness
On 4/20/06, Steve Peters [EMAIL PROTECTED] wrote:
Maybe I'm thinking too hard, or maybe the results reported aren't
exactly as clear as they probably should be. Here's an example test and
its results as reported by Test::Harness with the TODO changes.
#!perl -w
use strict;
use Test::More qw(no_plan);
TODO: {
local $TODO = TODO testing;
is(1, 2, A failing test);
is(1, 1, A passing test);
}
[EMAIL PROTECTED]:~/smoke/perl-current/t$ ./perl harness th_test.t
th_testok
1/2 unexpectedly succeeded
TODO PASSED tests 1-2
All tests successful (1 subtest UNEXPECTEDLY SUCCEEDED).
Passed Test Stat Wstat Total Pass Passed List of Passed
---
th_test.t 21 50.00% 1-2
Files=1, Tests=2, 0 wallclock secs ( 0.11 cusr + 0.01 csys = 0.12
CPU)
The line starting TODO PASSED shows all TODO tests, not those that
unexpectedly succeeded, which confused me a bit. Also, the final
results show that one test passed, but then the list of passed is 1-2
instead of just 2 which is the unexpected success. Is there a way to
have the list of passed just show the unexpected successes?
Attached patch should fix it up. Both in terms of making it clearer
and of fixing the list. So your test file would look like:
All tests successful (1 subtest UNEXPECTEDLY SUCCEEDED), 37 subtests skipped.
Passed Todo Stat Wstat Todos Pass Passed List of Passed
---
t/demerphq.t21 50.00% 3
Files=19, Tests=572, 8 wallclock secs ( 0.00 cusr + 0.00 csys = 0.00 CPU)
Hopefully thats clearer. The Todos column shows how many todos there
are in the file.
Cheers,
Yves
--
perl -Mre=debug -e /just|another|perl|hacker/
Only in Test-Harness: Makefile.old
diff -wurd Test-Harness-2.57_05/lib/Test/Harness.pm Test-Harness/lib/Test/Harness.pm
--- Test-Harness-2.57_05/lib/Test/Harness.pm 2006-04-19 07:25:51.0 +0200
+++ Test-Harness/lib/Test/Harness.pm 2006-04-20 22:30:18.102615400 +0200
@@ -39,6 +39,7 @@
=cut
$VERSION = 2.57_05;
+$VERSION = eval $VERSION;
# Backwards compatibility for exportable variable names.
*verbose = *Verbose;
@@ -352,7 +353,7 @@
# state of the current test.
my @failed = grep { !$results{details}[$_-1]{ok} }
[EMAIL PROTECTED];
-my @todo_pass = grep { $results{details}[$_-1]{ok}
+my @todo_pass = grep { $results{details}[$_-1]{actual_ok}
$results{details}[$_-1]{type} eq 'todo' }
[EMAIL PROTECTED];
@@ -362,6 +363,7 @@
max = $results{max},
failed = [EMAIL PROTECTED],
todo_pass = [EMAIL PROTECTED],
+todo= $results{todo},
bonus = $results{bonus},
skipped = $results{skip},
skip_reason = $results{skip_reason},
@@ -384,14 +386,14 @@
push(@msg, $test{skipped}/$test{max} skipped: $test{skip_reason})
if $test{skipped};
if ($test{bonus}) {
-my ($txt, $canon) = _canondetail($test{max},$test{skipped},'TODO passed',
+my ($txt, $canon) = _canondetail($test{todo},0,'TODO passed',
@{$test{todo_pass}});
$todo_passed{$tfile} = {
canon = $canon,
-max = $test{max},
+max = $test{todo},
failed = $test{bonus},
name= $tfile,
-percent = 100*$test{bonus}/$test{max},
+percent = 100*$test{bonus}/$test{todo},
estat = '',
wstat = '',
};
@@ -568,7 +570,7 @@
if (_all_ok($tot)) {
$out .= All tests successful$bonusmsg.\n;
if ($tot-{bonus}) {
-my($fmt_top, $fmt) = _create_fmts(Passed,$todo_passed);
+my($fmt_top, $fmt) = _create_fmts(Passed Todo,$todo_passed);
# Now write to formats
for my $script (sort keys %{$todo_passed||{}}) {
my $Curtest = $todo_passed-{$script};
@@ -593,7 +595,7 @@
$tot-{max} - $tot-{ok}, $tot-{max},
$percent_ok;
-my($fmt_top, $fmt1, $fmt2) = _create_fmts(Failed,$failedtests);
+my($fmt_top, $fmt1, $fmt2) = _create_fmts(Failed Test,$failedtests);
# Now write to formats
for my $script (sort keys %$failedtests) {
@@ -767,12 +769,13 @@
sub _create_fmts {
-my $type = shift;
+my $failed_str = shift;
my $failedtests = shift;
+my ($type) = split /\s/,$failed_str;
my $short =
Re: TODO tests and test::harness
On 4/19/06, Andy Lester [EMAIL PROTECTED] wrote: BTW, the patch only shows TODO pass status when no failures occur. Oh and obviously all of Test::Harness'es tests pass. :-) This patch doesn't apply against my latest dev version of Test::Harness. I'm going to have to massage it manually. But I like the idea. Thanks. You're welcome. If it helps It was against Test-Harness-2.56. Yves -- perl -Mre=debug -e /just|another|perl|hacker/
Re: TODO tests and test::harness
BTW, the patch only shows TODO pass status when no failures occur. Oh and obviously all of Test::Harness'es tests pass. :-) This patch doesn't apply against my latest dev version of Test::Harness. I'm going to have to massage it manually. But I like the idea. Thanks. xoa -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
