Re: UNKNOWN despite only failing tests -- how come?
On Dec 4, 2007 6:37 PM, Michael G Schwern [EMAIL PROTECTED] wrote: Andreas J. Koenig wrote: Bug in CPAN::Reporter and/or Test::Harness and/or CPAN.pm? http://www.nntp.perl.org/group/perl.cpan.testers/796974 http://www.nntp.perl.org/group/perl.cpan.testers/825449 All tests fail but Test::Harness reports NOTESTS and CPAN::Reporter concludes UNKNOWN and CPAN.pm then installs it. Test::Harness bug where it concludes NOTESTS if it sees no test output, as is the case when every test dies. I'll see about fixing it. [Original thread sent to perl-qa -- I'm adding tapx-dev] The fix in Test::Harness 3.05 now gives FAIL on no test output even if the test exits without error. Previously, no failure and no test output was NOTESTS. Andreas reported this to me on the RT queue for CPAN::Reporter as it breaks CPAN::Reporter tests. I initially would have said this new behavior -- while definitely a change -- is fine and I just need to update what CPAN::Reporter expects, but as I dug into the test cases, I realized why it may not be the right thing to do. ExtUtils::MakeMake and Module::Build treat test.pl differently. EU::MM just runs it and uses the exit code. M::B passes it to Test::Harness. Thus, a test.pl with no output (i.e no plan) has different behavior under each: * test.pl, no output, exit 0 - EU::MM says pass, M::B says unknown * test.pl, no output, exit 1 - EU::MM says fail, M::B says unknown (This is what t/54_test_report_split.t in CPAN::Reporter checks for.) Now, I know that test.pl is increasingly rare, but someone once pointed out to me that h2xs in 5.6.2 still generates test.pl (and with Test.pm, no less). So at least until 5.6 is declared dead, I'm trying to keep CPAN::Reporter able to deal with test.pl whether by itself or in combination with *.t files and under EU::MM or M::B. Overall, I'd prefer if Test::Harness releases that changed the definition of failure were released in a developer version first so we could try to catch and discuss these things earlier. Running CPAN::Reporter tests against T::H dev releases should help catch any behavior changes -- I've tried to capture as many test/output scenarios as possible. (See t/dist/ in CPAN::Reporter.) In any case, right now T::H 3.05 breaks CPAN::Reporter installation. I can change what CPAN::Reporter expects from T::H for 3.05+, but I don't want to do that until I'm sure the new T::H behavior is correct and stable. Thoughts from the crowd welcome. Regards, David
Re: UNKNOWN despite only failing tests -- how come?
On 4 Dec 2007, at 20:24, Andreas J. Koenig wrote: Bug in CPAN::Reporter and/or Test::Harness and/or CPAN.pm? http://www.nntp.perl.org/group/perl.cpan.testers/796974 http://www.nntp.perl.org/group/perl.cpan.testers/825449 All tests fail but Test::Harness reports NOTESTS and CPAN::Reporter concludes UNKNOWN and CPAN.pm then installs it. Your opinions welcome, I guess it's because Test::Harness says NOTESTS at the end. Hmm. Yeah, I guess it should fail in that case... -- Andy Armstrong, Hexten
Re: UNKNOWN despite only failing tests -- how come?
On 4 Dec 2007, at 23:37, Michael G Schwern wrote: Test::Harness bug where it concludes NOTESTS if it sees no test output, as is the case when every test dies. I'll see about fixing it. Is done. -- Andy Armstrong, Hexten
Re: UNKNOWN despite only failing tests -- how come?
On 4 Dec 2007, at 20:24, Andreas J. Koenig wrote: Bug in CPAN::Reporter and/or Test::Harness and/or CPAN.pm? http://www.nntp.perl.org/group/perl.cpan.testers/796974 http://www.nntp.perl.org/group/perl.cpan.testers/825449 All tests fail but Test::Harness reports NOTESTS and CPAN::Reporter concludes UNKNOWN and CPAN.pm then installs it. Will be fixed in 3.05. Thanks for the sanity Andreas :) -- Andy Armstrong, Hexten
Re: UNKNOWN despite only failing tests -- how come?
CPAN::Reporter says explicitly that UNKNOWN will return success to CPAN.pmand not prevent installation. I think this is a Test::Harness bug. David On Dec 4, 2007 3:24 PM, Andreas J. Koenig [EMAIL PROTECTED] wrote: Bug in CPAN::Reporter and/or Test::Harness and/or CPAN.pm? http://www.nntp.perl.org/group/perl.cpan.testers/796974 http://www.nntp.perl.org/group/perl.cpan.testers/825449 All tests fail but Test::Harness reports NOTESTS and CPAN::Reporter concludes UNKNOWN and CPAN.pm then installs it. Your opinions welcome, -- andreas
UNKNOWN despite only failing tests -- how come?
Bug in CPAN::Reporter and/or Test::Harness and/or CPAN.pm? http://www.nntp.perl.org/group/perl.cpan.testers/796974 http://www.nntp.perl.org/group/perl.cpan.testers/825449 All tests fail but Test::Harness reports NOTESTS and CPAN::Reporter concludes UNKNOWN and CPAN.pm then installs it. Your opinions welcome, -- andreas
Re: UNKNOWN despite only failing tests -- how come?
Andreas J. Koenig wrote: Bug in CPAN::Reporter and/or Test::Harness and/or CPAN.pm? http://www.nntp.perl.org/group/perl.cpan.testers/796974 http://www.nntp.perl.org/group/perl.cpan.testers/825449 All tests fail but Test::Harness reports NOTESTS and CPAN::Reporter concludes UNKNOWN and CPAN.pm then installs it. Test::Harness bug where it concludes NOTESTS if it sees no test output, as is the case when every test dies. I'll see about fixing it. -- On error resume stupid
