On Dec 4, 2007 6:37 PM, Michael G Schwern <[EMAIL PROTECTED]> wrote: > Andreas J. Koenig wrote: > > Bug in CPAN::Reporter and/or Test::Harness and/or CPAN.pm? > > > > http://www.nntp.perl.org/group/perl.cpan.testers/796974 > > http://www.nntp.perl.org/group/perl.cpan.testers/825449 > > > > All tests fail but Test::Harness reports NOTESTS and CPAN::Reporter > > concludes UNKNOWN and CPAN.pm then installs it. > > Test::Harness bug where it concludes "NOTESTS" if it sees no test output, as > is the case when every test dies. I'll see about fixing it.
[Original thread sent to perl-qa -- I'm adding tapx-dev] The fix in Test::Harness 3.05 now gives FAIL on no test output even if the test exits without error. Previously, no failure and no test output was "NOTESTS". Andreas reported this to me on the RT queue for CPAN::Reporter as it breaks CPAN::Reporter tests. I initially would have said this new behavior -- while definitely a change -- is fine and I just need to update what CPAN::Reporter expects, but as I dug into the test cases, I realized why it may not be the right thing to do. ExtUtils::MakeMake and Module::Build treat test.pl differently. EU::MM just runs it and uses the exit code. M::B passes it to Test::Harness. Thus, a test.pl with no output (i.e no plan) has different behavior under each: * test.pl, no output, exit 0 -> EU::MM says pass, M::B says unknown * test.pl, no output, exit 1 -> EU::MM says fail, M::B says unknown (This is what t/54_test_report_split.t in CPAN::Reporter checks for.) Now, I know that test.pl is increasingly rare, but someone once pointed out to me that h2xs in 5.6.2 still generates test.pl (and with Test.pm, no less). So at least until 5.6 is declared dead, I'm trying to keep CPAN::Reporter able to deal with test.pl whether by itself or in combination with *.t files and under EU::MM or M::B. Overall, I'd prefer if Test::Harness releases that changed the definition of failure were released in a developer version first so we could try to catch and discuss these things earlier. Running CPAN::Reporter tests against T::H dev releases should help catch any behavior changes -- I've tried to capture as many test/output scenarios as possible. (See t/dist/ in CPAN::Reporter.) In any case, right now T::H 3.05 breaks CPAN::Reporter installation. I can change what CPAN::Reporter expects from T::H for 3.05+, but I don't want to do that until I'm sure the new T::H behavior is correct and stable. Thoughts from the crowd welcome. Regards, David
