Test::Builder2: running coverage fails to display key modules
When I run Devel::Cover over the Test::Builder2 test suite (v1.005000_001-193-g00d4609), I get no coverage reports for blib/lib/Test/Builder2.pm or blib/lib/Test/More.pm. See: http://thenceforward.net/test-more/coverage/coverage.html Any thoughts on why that is happening? Thank you very much. Jim Keenan
Re: Test::Builder2: running coverage fails to display key modules
On Sat, Aug 04, 2012 at 08:54:27AM -0400, James E Keenan wrote: When I run Devel::Cover over the Test::Builder2 test suite (v1.005000_001-193-g00d4609), I get no coverage reports for blib/lib/Test/Builder2.pm or blib/lib/Test/More.pm. See: http://thenceforward.net/test-more/coverage/coverage.html Any thoughts on why that is happening? I'm not sure. I just ran 00d4609 (which I had as 1.005000_005) through my development copy of Devel::Cover (which will shortly become a release) and got somewhat different results. I ran: $ cover -test -report html_basic and got these results: http://pjcj.sytes.net/tmp/test-more/cover_db/coverage.html which includes both Test/Builder2.pm and Test/More.pm as well as some other modules, but doesn't include Makefile.PL. How did you generate your coverage report? -- Paul Johnson - p...@pjcj.net http://www.pjcj.net
Re: Test::Builder2: running coverage fails to display key modules
On 8/4/12 10:54 AM, Paul Johnson wrote: How did you generate your coverage report? Main part of my shell script: cd $SOURCEDIR \ cover -delete $COVERAGEDIR \ PERL5OPT=-MDevel::Cover=-db,$COVERAGEDIR/ \ perl Makefile.PL make make test \ cover $COVERAGEDIR \ -coverage statement \ -coverage branch \ -coverage condition \ -coverage subroutine \ -ignore_re '^t\/' \ -ignore_re '^\/usr\/local\/bin' \ -ignore_re '^\/tmp' \ chmod 0755 $COVERAGEDIR/
Re: Test::Builder2: running coverage fails to display key modules
On 8/4/12 11:16 AM, James E Keenan wrote: On 8/4/12 10:54 AM, Paul Johnson wrote: How did you generate your coverage report? Main part of my shell script: cd $SOURCEDIR \ cover -delete $COVERAGEDIR \ PERL5OPT=-MDevel::Cover=-db,$COVERAGEDIR/ \ perl Makefile.PL make make test \ cover $COVERAGEDIR \ -coverage statement \ -coverage branch \ -coverage condition \ -coverage subroutine \ -ignore_re '^t\/' \ -ignore_re '^\/usr\/local\/bin' \ -ignore_re '^\/tmp' \ chmod 0755 $COVERAGEDIR/ Data point: When I call this command, the first thing I get is a couple of dozen warnings like this: Devel::Cover: Warning: can't open ro-accessor for object_id (.../test-more/blib/lib/TB2/Mouse.pm) for MD5 digest: No such file or directory ... even though blib/lib/TB2/Mouse.pm clearly exists. jimk
Re: Test::Builder2: running coverage fails to display key modules
On 8/4/12 11:34 AM, James E Keenan wrote: On 8/4/12 11:16 AM, James E Keenan wrote: On 8/4/12 10:54 AM, Paul Johnson wrote: How did you generate your coverage report? Main part of my shell script: cd $SOURCEDIR \ cover -delete $COVERAGEDIR \ PERL5OPT=-MDevel::Cover=-db,$COVERAGEDIR/ \ perl Makefile.PL make make test \ cover $COVERAGEDIR \ -coverage statement \ -coverage branch \ -coverage condition \ -coverage subroutine \ -ignore_re '^t\/' \ -ignore_re '^\/usr\/local\/bin' \ -ignore_re '^\/tmp' \ chmod 0755 $COVERAGEDIR/ Data point: When I call this command, the first thing I get is a couple of dozen warnings like this: Devel::Cover: Warning: can't open ro-accessor for object_id (.../test-more/blib/lib/TB2/Mouse.pm) for MD5 digest: No such file or directory ... even though blib/lib/TB2/Mouse.pm clearly exists. And a further data point: When I call a command as simple as: cover $COVERAGEDIR -summary ... I fail to get reporting on the three packages mentioned earlier, though all tests pass. This suggests that for some reason data is not being recorded in the Devel::Cover DB.
Re: Test::Builder2: running coverage fails to display key modules
On Sat, Aug 04, 2012 at 12:19:18PM -0400, James E Keenan wrote: On 8/4/12 11:34 AM, James E Keenan wrote: On 8/4/12 11:16 AM, James E Keenan wrote: On 8/4/12 10:54 AM, Paul Johnson wrote: How did you generate your coverage report? Main part of my shell script: cd $SOURCEDIR \ cover -delete $COVERAGEDIR \ PERL5OPT=-MDevel::Cover=-db,$COVERAGEDIR/ \ perl Makefile.PL make make test \ I think the root of the problem lies here. If I'm not mistaken, I think that $PERL5OPT will only have this value for the call of perl Makefile.PL and not for make and make test. cover $COVERAGEDIR \ And if PERL5OPT was set here you would get am error message that cover shouldn't be run with coverage turned on. -coverage statement \ -coverage branch \ -coverage condition \ -coverage subroutine \ -ignore_re '^t\/' \ -ignore_re '^\/usr\/local\/bin' \ -ignore_re '^\/tmp' \ chmod 0755 $COVERAGEDIR/ Might I suggest something like this? cd $SOURCEDIR \ cover -delete $COVERAGEDIR \ perl Makefile.PL \ HARNESS_PERL_SWITCHES=-MDevel::Cover=-db,$COVERAGEDIR,-coverage,statement,branch,condition,subroutine make test \ cover $COVERAGEDIR Or, perhaps better still, cd $SOURCEDIR \ perl Makefile.PL \ DEVEL_COVER_OPTIONS=-db,$COVERAGEDIR,-coverage,statement,branch,condition,subroutine cover -test $COVERAGEDIR Data point: When I call this command, the first thing I get is a couple of dozen warnings like this: Devel::Cover: Warning: can't open ro-accessor for object_id (.../test-more/blib/lib/TB2/Mouse.pm) for MD5 digest: No such file or directory ... even though blib/lib/TB2/Mouse.pm clearly exists. Mouse.pm exists, but ro-accessor for object_id (.../test-more/blib/lib/TB2/Mouse.pm) doesn't. That's the made-up filename that that Mouse has pretended was the location of that code. Recent versions of Devel::Cover should suppress that warning. And a further data point: When I call a command as simple as: cover $COVERAGEDIR -summary ... I fail to get reporting on the three packages mentioned earlier, though all tests pass. This suggests that for some reason data is not being recorded in the Devel::Cover DB. Hopefully I've explained why above. What's confusing me is why you're getting as much coverage as you are. -- Paul Johnson - p...@pjcj.net http://www.pjcj.net
Test::Builder::Tester considered harmful. Use Test::Tester.
Executive Summary: If you're using Test::Builder::Tester to test your Test module, switch to Test::Tester. It will make the transition to Test::Builder 1.5 smoother and avoid future breakage due to TAP formatting changes. A lot of the work done fixing CPAN Test::* modules to be compatible with Test::Builder 1.5 has been to do with small changes to the TAP format. The TAP version header and things like the casing of # SKIP. Stupid little things. Testing a test library currently involves comparing the output TAP. This sucks, even with Test::Builder::Tester. The whole idea of testing test modules by looking at the formatted output sucks. Test::Builder::Tester is a gallant effort, Test::Builder 1.5 is event driven, so there's no need to look at the formatted output, you can look at the events directly and get a complete view of what's going on. For example, this test... ok( 1 ); is( 23, 42 ); done_testing; Generates something like this: # TB2::Event::TestStart # TB2::Result ok( 1 ); # TB2::Result # TB2::Event::Log (for the failure diagnostics) is( 23, 42 ); # TB2::SetPlan # TB2::Event::TestEnd done_testing; There's a couple of interfaces to test via events. Test::Tester is an existing one that works with both 0.x and 1.5, so that would be the preferred tool to use now. TB2::Tester is the one specifically written for Test::Builder 1.5. My plan is to mark Test::Builder::Tester as discouraged and focus on making it work as a transitional tool. That means emulating as many 0.x quirks as possible to keep existing TBT-based tests working, but that also means sacrificing improving it. IMO its approach is dead, so that's ok. What say? -- s7ank: i want to be one of those guys that types s/jjd//.^$ueu*///djsls/sm. and it's a perl script that turns dog crap into gold.
fork and tests: making it easier
Executive Summary: I propose Test::Builder 1.5 makes writing tests using fork as easy as writing tests using threads is. Test::Builder will handle the coordination for you. Downside: this breaks existing behavior. Rebutal: if you're testing with fork your tests are probably broken with Test::Builder 1.5 anyway. Currently, Test::Builder has little knowledge of forking. This makes forking in a test problematic if you expect to run tests in both forks. As people who have done it know, it involves a lot of fiddling with Test::Builder internals and yuck. Test::SharedFork solves this, but its deeply in bed with Test::Builder and fixing it for Test::Builder 1.5 is difficult. Test::TCP depends on Test::SharedFork. Plack depends on Test::TCP. So it's important to make it work. Turns out its easier to just do it inside Test::Builder 1.5. The state of the test is encapsulated in a single object and its a not-so-SMOP to freeze and restore that between processes. This is what geistteufel and I were working on at the QA Hackathon. So I propose having Test::Builder 1.5 coordinate the state of the test like it does with threads. It'll take care of the test counter and all that for you. There will be a switch to turn coordination off. It won't hijack fork or anything. Upsides: * Makes writing tests with forks and doing things in parallel much easier. * Eliminates a lot of Test::Builder hacking which often relies on undocumented behaviors and is difficult to support. Downsides: * It breaks existing behavior. * ...but tests which fork are probably broken with 1.5 anyway. * It involves file locking, which introduces filesystem complications * ...but it will only trigger when a fork runs a test. So most tests will be unaffected. * ...maybe it can detect if file locking is hosed and error early? * ...file locking will be in the temp directory which is usually a sane filesystem. * You might want to fork and run separate tests. * ...but this is less common than forking and wanting coordination * ...there will be a switch to turn coordination off. What do you think? -- Who invented the eponym?
