> > * test.pl, no output, exit 0 -> EU::MM says pass, M::B says unknown > > * test.pl, no output, exit 1 -> EU::MM says fail, M::B says unknown
For clarity, let me add the following cases (where T::H < 3.05 includes 2.XX): * foo.t, no output, exit 0 -> T::H < 3.05 says unknown, T::H 3.05 says fail * foo.t, no output, exit 1 -> T::H < 3.05 says unknown, T::H 3.05 says fail The last one is the bug that T::H 3.05 fixed. That's a good fix, I think, but it also hit the exit 0 case, depending on whether one thinks that should be unknown or fail. > T::H currently works on the premise that if you run a test and it > outputs nothing it's failed. I guess we could special-case test.pl but > that's messy - would be good if we could avoid that. I think it's too late for that. I've done some more testing and I think we have three options: (a) Keep T::H 3.05 behavior -- CPAN::Reporter adds special-case tests for T::H < 3.05 and T::H >= 3.05 (b) Change T::H to say "unknown" for the no output, exit 0 case (regardless of whether it's test.pl or not) -- CPAN::Reporter adds special-case tests for T::H < 3.05, = 3.05, > 3.05. (c) Change T::H to say "unknown" for the no output, exit 0 case, but only for test.pl -- CPAN::Reporter adds special-case tests for T::H < 3.05, = 3.05, > 3.05. So at this point, I've written a new test file for T::H version-specific outcomes, since that seems inevitable thanks to the legitimate 3.05 bug fix. I just need to know whether the Perl collective thinks that (b) or (c) make sense so I know whether to add the appropriate logic for T::H > 3.05. > > Overall, I'd prefer if Test::Harness releases that changed the > > definition of failure were released in a developer version first so we > > I'm really sorry about that David. It was a dumb thing to do - no > excuses. I'll make sure it doesn't happen again. That's OK. When I've got bug fixes, I tend to want to get them out the door too, and CPAN::Reporter tests take a while. On the positive side, I realized that I needed to add a test case for foo.t, no output, exit 1. I'm writing this while I test the CPAN::Reporter trunk on T::H 3.05, T::H 3.04 and T::H 2.64. (I've had do that enough, and under each of M::B and EU::MM that I really need to automate it.) If all goes well, I'll check in the code and you can pull the latest for testing from the repository: http://dagolden.googlecode.com/svn/CPAN-Reporter/trunk/ If it has t/56_test_report_harness_versions.t then you've got the version with the fixes. If you don't see that, then I'm still working on it. David
