Re: Should we dump the perl5 t/ tests?
On Fri, Sep 01, 2000 at 01:06:24AM -0500, Brent LaVelle wrote: > A long time ago on p5p I suggested dumping the current test scheme in > favor of a general purpose testing system where you could test things > programs like: I remember it, or at least I selectively remember parts of it. But I thought the idea was to build an extra level of testing to compliment rather than usurp the current t/*.t tests. The current tests try to test simple parts of the language first, before moving on to other more complicated parts. That's why some of them are written in a fairly strange fashion. I don't think we can (yet) assume that there will be an older, working perl available for our use. But I do think that once perl has passed its t/*.t tests, we can then use it for further testing purposes. At the time you first mentioned this I offered some testing code I had, subject to it being released under an appropriate licence. That has now happened (was is late in 1998 or 1997 when you first mentioned this?) and so I make the offer again. This time though, I will hopefully have a little time to make it happen if this idea is deemed sound, and I expect that it will be, generally. I will make a few comments, with particular reference to the code which I have, not that I expect it to totally suitable, but with the idea that it could be used as a base upon which to build, or just to get ideas about what is and isn't a good idea. The code I have is all Perl. It runs on multiple platforms across a network, well a few Unices and NT anyway. It runs a set of tests, each of which may have a number of steps. The tests and steps in each test can be decided upon at runtime to allow for platform and other environment differences. The tests are written as Perl data structures which allows subroutines to be embedded in the tests, the output of these subroutines determining the nature of the test. Test output can be checked against golden results, with allowances made for certain differences, and at the end you get a pretty HTML report giving you all sorts of information about the tests, what passed, what the exit statuses were, how long it took... > The test system should be able to collect metrics. What tests are good > for what. > > The metrics should be used to slice and dice the tests. Based on [ snip ] > When I proposed this before nobody seemed to get fired up, they where > worried about the expensive fork/exec that it added. I'm worried about > 5 lines of Perl code that test the same 20% of the C (or whatever) > code that Perl is written in but take 100 times longer than the 500 > lines needed to test that 20%. Aha. Are you suggesting code coverage be integrated? Code coverage might also be useful at the Perl level to test Perl modules. If it can be done right, this whole package might be valuable to module authors in the same way that t/*.t is now. > Below the bottom line: the test system should be aware of the Perl > version but be and independent product that produces a set of tests for > Perl, t/ tests. This test system would be a general purpose tool and > take Perl testing to a new level. I agree, although I also still see a place for the t/*.t tests. -- Paul Johnson - [EMAIL PROTECTED]
Re: Bug tracking and todo tests
On Mon, Feb 12, 2001 at 11:58:19PM -0500, [EMAIL PROTECTED] wrote:
> bugs which have yet to be fixed. The syntax is alittle weird, but
> that can still be fixed as its undocumented. Basically, it looks (or
I'm not sure whether you're talking about the API or the textual output
here. The API looks pretty well documented to me.
BEGIN { plan tests => 14, todo => [3,4] }
It's been there for a couple of years at least I think.
For the textual output, does it matter what it looks like? (Which is
neither an argument for or against changing it.)
--
Paul Johnson - [EMAIL PROTECTED]
http://www.pjcj.net
Re: Bug tracking and todo tests
On Wed, Feb 14, 2001 at 07:27:10PM +, [EMAIL PROTECTED] wrote:
> >BEGIN { plan tests => 14, todo => [3,4] }
>
> Which requires you to got edit the numbers when you add tests.
> Not wanting to do that is why I started use'ing Test.
Same here. But you have to edit the number of tests anyway, and I think
you _should_ have to.
Of course, editing the todo list is a lot harder, but I wouldn't want to
go naming all my tests. Perhaps they could have an optional name?
--
Paul Johnson - [EMAIL PROTECTED]
http://www.pjcj.net
Re: Towards solid coverage and profiling tools.
On Thu, Feb 15, 2001 at 05:44:04PM -0500, [EMAIL PROTECTED] wrote: > Now, it strikes me that a profiler is just a coverage tool that also > records times, and a coverage tool is just a profiler that doesn't. Well, I think a real coverage tool is bit more than that really. All the world is not statement coverage. In fact, although it is much better than nothing, statement coverage is a fairly weak coverage metric. Ideally we would have branch coverage at least, if not some of the more powerful metrics. However, I suspect that we are in a similar position to Pure Software or whatever they're called, in that their coverage tool fell fairly easily out of Quantify, but can only do statement coverage. I'm not sure how or if we could use the debugger to provide other coverage information. That sort of thing really requires some navel gazing - we would want to analyse our own parse tree or something. Hopefully that sort of thing will be possible in Perl 6, if not sooner. The other option is code instrumentation, but that requires parsing of the code. Could be done I suppose, but I wouldn't really want to. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Towards solid coverage and profiling tools.
On Thu, Feb 15, 2001 at 11:30:07PM +, Simon Cozens wrote: > > Depends what sort of instrumentation you want. If you want instrumentation > about how perl sees the code, you probably want my "mangle" patch. If you want > instrumentation which can be produced by a Perl program, see > http://perltidy.sourceforge.net/ Your recent efforts did pass through my mind, and I looked at perltidy when you mentioned it on p5p. What is needed is a way to add callbacks at strategic points in the code. Without thinking too hard, the start of most blocks and a few other places where code branches would be a minimum requirement. Ideally, it should be done with the help of perl (as with your patch) rather than by parsing (as with perltidy) because the parsing will get it wrong eventually and will probably be harder and less efficient too. There was talk at the start of Perl 6 about being able to modify the parse tree or byte code or something on the fly. That is really what's wanted. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Towards solid coverage and profiling tools.
On Thu, Feb 15, 2001 at 06:24:06PM -0500, [EMAIL PROTECTED] wrote: > I'm fairly ignorant about anything beyond basic coverage, as I'm sure > most everyone else on this list is as well. Could you whip out a > summary of useful coverage techniques? Certainly, but not tonight (this morning) I'm afraid. There may well be information by following Brent's links to gct. Be warned that many similar techniques have different names though. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Towards solid coverage and profiling tools.
ectify any problems thrown up. Finally, if the overhead is too great it won't get used either. So there's a basic tutorial on code coverage, or at least my version of it. Typing a few of these terms into google will probably provide a basis for future research. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Suggestions about Testing.pm
On Wed, Mar 14, 2001 at 07:45:29AM -0500, barries wrote: > On Wed, Mar 14, 2001 at 02:57:42AM +, Michael G Schwern wrote: > > On Fri, Mar 09, 2001 at 09:56:37AM -0500, barries wrote: > > > You may want to settle on using underscores everywhere or nowhere in the > > > interests of consistency. > > > > Yeah. Underscores read better but type slower. Hmmm... > > Looking up inconsistent ids is slowest :-). I don't find the extra char > '_' causes noticable impact. I optimize for maintainability and put in > '_'. YMMV a lot, though. I know it's religious, but since this is Perl for testing Perl, and will be in the standard distribution (I suppose), we should probably stick as close as possible to what's in man perlstyle. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Stuck on Testing::skip()
On Sat, Mar 17, 2001 at 05:20:23PM +, Piers Cawley wrote:
> Michael G Schwern <[EMAIL PROTECTED]> writes:
>
> > Ok, that's great. But then Piers Cawley cried out "HALTING PROBLEM!"
> > from the front row (and get your feet off the stage).
>
> My heckles have always been of the highest quality.
>
> > > From: Piers Cawley <[EMAIL PROTECTED]>
> > > Subject: Re: [FWP] Stumper
> > > To: Michael G Schwern <[EMAIL PROTECTED]>
> > > Cc: [EMAIL PROTECTED]
> > >
> > >
> > > Isn't this a halting problem issue?
> > >
> > > foo { bar() if bar() && bar() };
> > >
> > > How many times does bar get run?
> >
> > To which my reply is basically, "Oh crap."
>
> Ah well, better to find the problem early.
I've been mostly following all this without having had the time to
follow it in detail, and I keep thinking about the parenthetical comment
Joshua Pritikin put in Test.pm, viz "(Your test code should be simpler
than the code it is testing, yes?)".
OK, so this isn't the user's test code, but it seems to me that getting
anywhere near the halting problem means that we've taken a wrong turn
somewhere. Some of the other suggestions on FWP (admittedly provided
without full knowledge of the use to which they might be put) included
use of the B modules and temporary redefinition of ok(). These also
strike me as being a little to complicated for test code.
I have gone back and read the discussion so far. Is "better late than
never" still true? Feel free to withhold judgement
If we're allowed to mess with the interface, my counter proposal would
be to simplify it rather than complicate or extend it. Here are some
thoughts:
Planning.
I understand the motivation behind the ideas discussed here, but I don't
see any problem with the current method whereby the number of tests
needs to be declared at the start of the run. OK, so we all run the
script, see how many tests there were and then fill in the number. This
still strikes me as safer than not having the number there. I could
fairly easily be persuaded that having the number specified at the end
is probably OK, but I wouldn't want to recommend it as general practice.
ok()
May I propose the following interface?
ok($scalar, $test_name, $reason);
I suspect that the semantics are fairly obvious. If $scalar is true the
test passes, otherwise it fails. $test_name needs no explanation, and
it need not be present. $reason is ignored if present. It is only
there to match todo() and skip(). Details below.
My motivation for this simple interface is that I see little benefit to
me as a module author in being able to write
ok($x, 17);
rather than
ok($x == 17);
If anything, the latter is more explicit. Think of it as removing all
special cases :-)
ok($x =~ /qaz/);
seems preferable to
ok($x, qr/qaz/);
At a stroke, all the problems associated with eq/ne/lt/... and similar
groups have been solved.
I don't even see the need for passing a subroutine reference to be run.
Why not run it yourself, and pass the result?
todo()
Changing the todo() interface has merit I think, but I would suggest
that changing a test from todo() to ok() should simply be a matter of
changing the name of the function. In other words, I would like todo()
and ok() to be identical, except for the expectation of success or
failure. For todo(), $reason is, of course, the reason the test should
fail.
skip()
Using the same interface as ok() and todo(), skip() skips the test for
$reason. If $reason changes (eg Windows supports file locking) the test
can just change to ok(). If it changes from skip() to todo(), $reason
may need to be changed, but if it wasn't it would probably still be
somewhat appropriate.
I see the motivation for having a parameter to distinguish between
skipping and running the test, but how often does this happen in
practice? I wouldn't be opposed to a function which called either ok()
or skip() based on a parameter if people feel strongly about it.
Prototypes.
I would suggest not prototyping the functions. This will make writing a
wrapper function easier if that is needed.
ONFAIL.
I don't think anyone has mentioned this yet. I've never seen it used,
but I wouldn't want to just chuck it out. It must have scratched some
itch.
That's it. Basically, KISS. We really don't want bugs in the test
code. I'm willing to be told that I've oversimplified, but I'd like to
see real code that has to jump through hoops to cope with this
interface.
I was about to write a detailed critique of Michael Schwern's original
proposal, but most of the comments would now be redundant, based on what
I have written above. I'll still do so, if necessary, but I think I'll
wait to see if anyone has any comments about my ideas.
--
Paul Johnson - [EMAIL PROTECTED]
http://www.pjcj.net
Re: Stuck on Testing::skip()
On Sat, Mar 17, 2001 at 01:19:02PM -1000, Tim Jenness wrote: > On Sat, 17 Mar 2001, Paul Johnson wrote: > > > My motivation for this simple interface is that I see little benefit to > > me as a module author in being able to write > > > > ok($x, 17); > > > > rather than > > > > ok($x == 17); > > > > If anything, the latter is more explicit. Think of it as removing all > > special cases :-) > > This is fine but the 2 arg version allows the test module to tell you > *why* the two args did not match. One of the useful features of the test > module is that it tells you both $x and 17 when they are not equal so that > you can see the resaaon for the failure without having to add a print > statemtnt to the test. The single argument ok() will just tell you a > failure but not what the failure was. This is an excellent point, and you are quite right that this is extremely useful in this most common case. I return to ponder mode for a while. (Well sleep actually.) > I think the skip() function should at least add "Skip" to the reason when > printing it - the user of Test.pm should not need to know the format of a > skip message required by Test::Harness. Quite. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
[ANNOUNCE] Devel::Cover 0.01
I've started writing a code coverage module for Perl. Yes, another one. Why? Well, I felt a different approach was needed to get sufficient information to report on different coverage criteria. Hooking into the debugger can provide information for statement coverage, but little else. >From the README: Code coverage data are collected using a plugable runops subroutine which counts how many times each op is executed. These data are then mapped back to reality using the B compiler modules. At the moment, only statement coverage information is reported. Coverage data for other metrics are collected, but not reported. Coverage data for some metrics are not yet collected. Requirements: Perl 5.6.1 or bleadperl. The ability to compile XS extensions. Those requirements are real - not even 5.6.0 will work. I don't know about the 5.6.1 trials. "Release early, release often", right? Well this one is early, but it seems to work reasonably well. I've run it on my Gedcom module on CPAN. I haven't fully analysed the results, but the output looks realistic. Get it from my homepage http://www.pjcj.net. I'll chuck it on CPAN in a while, pending any comments which are, of course, very welcome. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Descriptive strings?
On Wed, Jun 20, 2001 at 02:38:14PM -0400, Kirrily Robert wrote: > On Tue, Jun 19, 2001 at 08:42:29PM -0400, Michael G Schwern wrote: > | > | I assume you're talking about "make test"? Test::Harness in > | non-verbose mode (ie. "make test") won't display any of that info. If > | you set $verbose = 1 you'll see all the test output. For failed tests > | it will just report their numbers. > > Where do I set $verbose? make test TEST_VERBOSE=1 -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: confused by Devel::Cover
On Mon, Aug 06, 2001 at 12:09:45AM -0400, Kirrily 'Skud' Robert wrote: > This one's for Paul or anyone else who happens to know :) Sounds like my cue > After hearing about Devel::Cover at YAPC::E's CPANTS session, I thought > I'd pull it down and take a look at it. Wow. Does that mean I'm famous? :-) > Well, I don't understand it. This is probably because I've never used > such a thing before. So here are some questions and, I guess, > a request for more (or rather, clearer) documentation of the usage and > features of the module. Yes. That's certainly a requirement, but I'm still sorting out functionality. I know, "write the docs first", but where's the fun in that? > Firstly, let me say that I've read the perldoc and also did a Google > search and found http://www.bullseye.com/coverage.html. The general > concepts are clear enough to me now. Where I have the problem is in > relation to usage. I'm totally confused wrt finding out how much of a > module Foo::Bar is covered by the tests in t/foobar.t > > I tried a simple "perl -MDevel::Cover foobar.t" and it told me I had > 100% statement and condition coverage. It looked to me as if it was > just looking through foobar.t and not actually getting down into > Foo::Bar. > > Can anyone please clarify this a bit? First, I wouldn't count on the accuracy of the condition coverage just yet. It still needs lots of work. I was just looking at it last night. Unfortunately, optrees can get very complicated. Still, at least I have B::Deparse to crib from. But the statement coverage should be quite good, modulo Perl messing with the optree and making some things look a little funny. With respect to exercising modules, this should all work. Did the module pass its own tests? Actually, there's only one test at the moment, but it does include a module. What you did should have worked. You should have got a report about t/foobar.t and Foo::Bar.pm. I've run some Gedcom.pm tests, and they seemed to give reasonable results (from the point of view of being believable, rather than having good coverage.) I'm happy to take it as a bug report if you'd like to send it to me. There's some reasonably hairy code in there to locate all the modules used, and look through their optrees, so I wouldn't be surprised if things weren't quite right there. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: ANNOUNCE Devel::Cover 0.05
On Thu, Aug 09, 2001 at 10:27:04AM +0200, Arthur Bergman wrote: > 01-08-09 01.50, skrev Paul Johnson på [EMAIL PROTECTED] följande: > > > This release fixes two problems: > > > > - Try harder to get the correct line number, even when perl has > > optimised the line away. This is similar to the fix I sent in for > > the core - in fact, this one came first. > > Shouldn't we just do coverage analysis on unoptimized code? Ideally we would but, as you know, some of the work done by the optimisation routines is more along the lines of essential optree manipulation rather than optimisation, especially where threads are concerned, I think. I did try commenting out peep(), or returning from it early, or something, but I quickly decided that was getting outside the scope of my project ;-) Now, if we could separate the optimisation from the essential work that would be great, but I think that's one for some intrepid explorer on p5p. Some parts, such as constant subroutine inlining for example, might not be too hard but others would be. Did I hear that you had nearly finished the threading work and were looking round for a new challenge? :-) In any case, it's not _that_ bad. B::Deparse somehow manages. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
ANNOUNCE Devel::Cover 0.09
Two main advancements since 0.06 (yes, you blinked and missed a couple): - The coverage database now has an API so it will be easier to write reporting tools. - I think I've sorted out the @Inc stuff so you can now control which files you want reports on reasonably easily. The defaults should mean that TEST_HARNESS_PERL='-MDevel::Cover' make test does what you would hope it would. I tested it on my Gedcom.pm module, and surprised myself with how complete the testsuite was (or at least it wasn't as bad as I thought it might have been ;-) So, it's even starting to get a little useful now -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
ANNOUNCE Devel::Cover 0.10
The main improvement here is the introduction of a program, cover, to generate reports. The most exciting report is a snazzy HTML one. Well, more functional than snazzy actually. Anyone who wants to add snaz is welcome to. This can also function as an example for any other reports that people might like to write. Available from CPAN soon, or now from my homepage. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
ANNOUNCE Devel::Cover 0.11
Oooh. A shiny new release. The most exciting thing here is the integration with Pod::Coverage to provide pod coverage(!) along with the other coverage metrics. I've also put a proper API on the coverage database and enhanced the html output somewhat. Available from CPAN soon, or now from my homepage. Still lots to do though -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Preliminary test coverage analysis
On Thu, Oct 18, 2001 at 10:42:33PM -0400, Michael G Schwern wrote: > On Thu, Oct 18, 2001 at 12:07:26AM -0600, chromatic wrote: > > > - Of those covered, a smidge less than half have < 75% statement coverage. - " > > >" " , 20% have < 50% statement coverage. > > > > Is it possible to get an update on this? Writing a new test from > > scratch is generally harder than adding to an existing test. (It's > > probably the blank page thing that plagues authors.) > > Unfortunately, no. Devel::Coverage is fundamentally broken and > Devel::Cover is gagging on the core test suite. > > I'll see what I can do about getting Devel::Cover working. I started with the intention of looking at this last night but got sidetracked by the AUTOLOAD problem which is stopping Devel::Cover working with bleadperl. I'll try to look further. By the way, I got rid of perl-qa-metrics. It's basically dead, right? -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Preliminary test coverage analysis
On Thu, Oct 18, 2001 at 10:42:33PM -0400, Michael G Schwern wrote: > On Thu, Oct 18, 2001 at 12:07:26AM -0600, chromatic wrote: > > > - Of those covered, a smidge less than half have < 75% statement coverage. - " > > >" " , 20% have < 50% statement coverage. > > > > Is it possible to get an update on this? Writing a new test from > > scratch is generally harder than adding to an existing test. (It's > > probably the blank page thing that plagues authors.) > > Unfortunately, no. Devel::Coverage is fundamentally broken and > Devel::Cover is gagging on the core test suite. I ran this last night, and supply the results here without comment save: - Devel::Cover is still alpha code and there are obviously bugs. - 11/316 tests failed, 2 are obviously due to bugs in Devel::Cover. - A few tests ended up in different directories and are not reported here. - The URL is to my home machine so apologies if it is down or slow. - I'll leave the data there for a while, but not for ever. - This gives me plenty to work on http://pjcj.sytes.net/cover.12493/cover_db/cover_db.html username: perl password: cover -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Preliminary test coverage analysis
On Sun, Oct 21, 2001 at 07:31:20PM +0200, Tels wrote: > On 21-Oct-01 Paul Johnson tried to scribble about: > > On Thu, Oct 18, 2001 at 10:42:33PM -0400, Michael G Schwern wrote: > >> On Thu, Oct 18, 2001 at 12:07:26AM -0600, chromatic wrote: > >> > > - Of those covered, a smidge less than half have < 75% statement > >> > > coverage. - " > >> > >" " , 20% have < 50% statement coverage. > > I ran this last night, and supply the results here without comment save: > > Good work! (I see my modules ;) > > Some hints, though: > > You could leave of *.t files. The test's own coverage is not very > interesting, since most testsuites probably are supposed to not be covered > 100%. F.i. bigintpm.t runs some tests different based on your system, and I > bet others do so, too. So you can't achieve 100% anyway. Certainly there is a lot that could do with cleaning up, but I wanted to throw out the first pass so people could get a feel for it. You're right that platform specific cod is going to skew the figures somewhat. [ I liked that typo so much I decided to keep it :-) ] > Of course, somebody might be interested in why some test files have poor > coverage ;) This can be useful on occasion. > I tried to view my modules, but your server is currently melting and can't > deliver the HUGE html documents :/ I keep getting hit by things that think I am running Micros~1 stuff :-( Looks like you have it now, but let me know if that's not the case. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Preliminary test coverage analysis
On Sun, Oct 21, 2001 at 07:40:11PM +0200, Tels wrote: > Thank you for your work! I looked at BigFloat.pm, noticed a untested line > in facmp(), added tests and lo and behold, there was a bug! (bacmp(5,+inf) > returned 1 but should return -1 ;) Hah! It was all worth it :-) > Wow ;) Indeed. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
ANNOUNCE Devel::Cover 0.12
Hey! It's 0.12! Now with improved integration to Pod::Coverage _and_ time coverage, aka profiling. Plus a bunch of internal stuff you don't care about unless you feel the urge to hack on it. Try it out, tell me what's wrong, or fix it yourself! What an offer! Coming soon to a CPAN mirror near you, or get it now from my home page. This announcement brought to you by the ! key. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: ANNOUNCE Devel::Cover 0.12
On Sun, Oct 14, 2001 at 05:19:28PM +0200, Paul Johnson wrote: > Hey! It's 0.12! Better yet get 0.13. It's had some marginal quality control :-) -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Untested modules update: There's more than we thought
On Sat, Dec 15, 2001 at 11:46:29PM +, Nicholas Clark wrote: > On Sat, Dec 15, 2001 at 05:59:22PM -0500, Michael G Schwern wrote: > > On Sat, Dec 15, 2001 at 11:48:29PM +0100, Tels wrote: > > > Te"One day Math::Big* will have more tests than Perl;)"ls > > > > One day Perl will have more tests than Bit::Vector. 67255 at last count. > > Ilya chose not to attempt that for 5.6, by make op/numconvert.t only report > 1 ok (or not) for a cluster of 15 numbers passed through the same conversions. > We could always change op/numconvert.t to report 15 times as many tests > (about another 20,000 I think) > > But that's cheating, isn't it? My Gedcom module has 7671 tests. Most of these are from checking a 1476 line file five times. I could easily make that file bigger ... Although really I'd prefer to make it smaller. It used to take about 15 mins to run on my old Dec. Nowadays that's not such a priority :-) > More on topic - I like Test::More. > I like test.pl in the core. > But I fear that we aren't bootstrapping our tests carefully enough. > In that I'm pretty sure we shouldn't be using Test::More until we're > sure we've tested all the ops, and we shouldn't be using test.pl until > we've tested all the ops it uses (and be careful to limit the ops > and the complexity of ops it uses) > > TEST starts: > > # This is written in a peculiar style, since we're trying to avoid > # most of the constructs we'll be testing for. > > Maybe it should print that out on each run, to remind everyone :-) I agree. That is, I share your fears. I also wonder about TEST. Last time I did a patch for it I was surprised at just what constructs it did include. Maybe all the constructs it uses are tested earlier, but after all the fingers that have been in the pie, it's probably time for someone to take a good look at what gets tested where to make sure that the various test programs do only use previously tested constructs as far as possible. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Compiled programs to keep BEGIN blocks? (was Re: [RFC] Switch to make Test::Builder output even if $^C ( for B::C ))
On Tue, Jan 15, 2002 at 05:30:09PM -0500, [EMAIL PROTECTED] wrote:
> On Tue, Jan 15, 2002 at 08:51:06AM +, Piers Cawley wrote:
> > > I don't understand. Why do we have to deal with them? Just translate
> > > the Perl code in the BEGIN block to C , dump it and make
> > > sure it gets run first. Right?
> > >
> > > I think I'm missing something very vital here.
Yeah, I think so ;-)
[snip]
> Guess? Its a BEGIN block, it gets run once!
Oh good, because you seemed to be suggesting otherwise.
> Ok, there's some really fundemental breakdown of understanding here,
> and I think its me. Let's step back a second. Why is anything
> guessing when and how many times BEGIN blocks are run?
Well, you seemed to be suggesting that some BEGIN blocks would be run at
runtime. Some BEGIN blocks have to run at compile time. If they're
only to be run once then you have to decide when that will be.
My boss recently gave me a fairly tricky problem to solve. The Java
boys weren't making much headway, so he asked if I could do it in PERL.
After (a) little thought I came up with an elegant modular solution.
Since the program contains proprietary intellectual property and also
because it has to run as fast as C, I decided to run perlcc on it.
Here's my program. Please don't break it.
ONE.pm
==
package ONE;
# I might obfuscate this eval statement later in case any of our
# competitors ever see this code.
sub import { eval '$main::ONE--' }
1
one.plx
===
#!/usr/bin/perl
# Copyright Paul Johnson, 2002
# This program is mine. You can't have it.
# Program to print 1
BEGIN { $ONE = 2 } # Initialisation.
use ONE;# Abstracted away main logic.
print $ONE # Output.
__END__
The fundamental problem here is that BEGIN blocks have to run when they
are compiled. This is documented. You can try to work out whether the
BEGIN blocks can be rearranged and run at different times, but as Piers
has pointed out, this is impossible in the general case. So if the
BEGIN blocks are run at compile time, there's no point having them
around at run time. This is the raison d'être of INIT blocks.
This problem cannot be solved. But if you do solve it millions will
rejoice. Well, thousands anyway. Still, I wonder whether you shouldn't
work up to it by perhaps finding a polynomial time solution to the
travelling salesman problem for example.
PS. Parts of this message are moderated by :-) where required by law.
--
Paul Johnson - [EMAIL PROTECTED]
Re: gcov/xs trickery.
'' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=4, prototype=define Linker and Libraries: ld='gcc', ldflags =' -L/usr/local/lib' libpth=/usr/local/lib /lib /usr/lib libs=-lnsl -lndbm -lgdbm -ldl -lm -lc -lcrypt -lutil perllibs=-lnsl -ldl -lm -lc -lcrypt -lutil libc=/lib/libc-2.2.2.so, so=so, useshrplib=false, libperl=libperl.a Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic' cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib' Characteristics of this binary (from libperl): Compile-time options: DEBUGGING USE_LARGE_FILES Locally applied patches: DEVEL14574 Built under linux Compiled at Feb 9 2002 02:01:34 @INC: /sw/packages/bleadperl/lib/5.7.2/i686-linux /sw/packages/bleadperl/lib/5.7.2 /sw/packages/bleadperl/lib/site_perl/5.7.2/i686-linux /sw/packages/bleadperl/lib/site_perl/5.7.2 /sw/packages/bleadperl/lib/site_perl/5.7.1/i686-linux /sw/packages/bleadperl/lib/site_perl/5.7.1 /sw/packages/bleadperl/lib/site_perl . -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
[ANNOUNCE] Devel::Cover 0.14
Spurred on by Tels' bug reports and the fixing of a bug in bleadperl, Devel::Cover 0.14 is escaping into the wild. It's just a bug fix release really, but it also includes the gcov2perl script that was alluded to a few messages ago. Currently it's only on my homepage until I can keep my network connection up long enough to get it on CPAN. Enjoy, -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: make cover
On Thu, Feb 28, 2002 at 07:23:43PM +0100, Tels wrote: > Anyway, it is not so easy to run your entire testsuite trough Devel::Cover. > Well, sort it is: > > PERL_DL_NONLAZY=1 /usr/local/bin/perl -Iblib/arch -Iblib/lib -e 'use > Test::Harness qw(&runtests $switches $verbose); $switches="-w > - -MDevel::Cover"; $verbose=0; runtests @ARGV;' t/*.t I normally go with: PERL5OPT='-MDevel::Cover' make test just don't try it on the Devel::Cover testsuite itself :-) > takes care of that. Only problem is that the cover_db is created locally to > each testfile, and that can be a problem if your testfiles chdir to > different dirs. Hopefully specifying -db,/full/path/to/db takes care of that. Yes, this information should be in the docs. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Devel::Cover and v5.8.0
On Wed, Sep 04, 2002 at 11:25:30PM +0200, Tels wrote: > null:/home/te/perl/perl-5.8.0/lib # perl -MDevel::Cover -I. Math/Complex.t > [snip] > Can't locate object method "START" via package "B::SPECIAL" at > /usr/local/lib/perl5/site_perl/5.8.0/i686-linux/Devel/Cover.pm line 147. > END failed--call queue aborted. > > B.pm contains B::SPECIAL in only one place and no "START" AFAIS. > Devel::Cover seems to die always with this error. > > Did anybody use Devel::Cover with 5.8.0 yet? Me :-) Actually, I have a shiny new release all ready to go. It might even happen tonight if I can stay awake ... I just want to test it a little ;-) -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Devel::Cover and v5.8.0
On Thu, Sep 05, 2002 at 11:28:26PM +0200, Tels wrote: > > On Wed, Sep 04, 2002 at 11:25:30PM +0200, Tels wrote: > > > >> null:/home/te/perl/perl-5.8.0/lib # perl -MDevel::Cover -I. > >> Math/Complex.t > >> [snip] > >> Can't locate object method "START" via package "B::SPECIAL" at > >> /usr/local/lib/perl5/site_perl/5.8.0/i686-linux/Devel/Cover.pm line 147. > >> END failed--call queue aborted. > >> > >> B.pm contains B::SPECIAL in only one place and no "START" AFAIS. > >> Devel::Cover seems to die always with this error. > >> > >> Did anybody use Devel::Cover with 5.8.0 yet? > > > > Me :-) > > > > Actually, I have a shiny new release all ready to go. It might even > > happen tonight if I can stay awake ... > > > > I just want to test it a little ;-) > > But, does it contain the B::SPECIAL fix? :) Well, that bit of code has been completely reworked anyway. Do you have a small example I could test with? There are still bugs, but I think I should probably release it soon. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
[ANNOUNCE] Devel::Cover 0.15
Here's fun! Oh, hold on. Wrong list. Here's something really serious. As threatened, Devel::Cover 0.15 has escaped into the wild, and is currently marauding around CPAN, using up computing resources around the entire world. What's new? Loads. For a start, it should mostly work. Statement cover was mostly there anyway, but now it is joined by branch and condition coverage. The real stuff, mind you. Not what I fobbed you of with before. There's a bunch of other changes too, but if you're interested you'll read the CHANGES file, I'm sure. There are also some tests. Well, there were some before, but these are better and should be portable too. These massive advances convinced me that a 0.01 increase in the version number was in order. HTML output still looks pretty dodgy, but now it's generated from templates, so if you don't like it you can fix it :-) Speaking of templates, because this is a specialised application I wrote my own template system, which I'll release to CPAN separately. Wait a minute. No I didn't. I used TT2. That's OK then. There is at least one problem with the templates though. Where should I put them, and how should I get them there? Advice sought. Remember, this is still alpha code, so don't expect too much and you won't be too disappointed. It's on my homepage if you can't wait for CPAN. Enjoy. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: [ANNOUNCE] Devel::Cover 0.15
On Fri, Sep 06, 2002 at 04:40:40PM +0900, Tatsuhiko Miyagawa wrote: > At Fri, 6 Sep 2002 01:59:36 +0200, Paul Johnson wrote: > > > Here's something really serious. As threatened, Devel::Cover 0.15 has > > escaped into the wild, and is currently marauding around CPAN, using up > > computing resources around the entire world. > > Does anyone have a fascinating plan to integrate this great module > with Test::Builder and/or MakeMaker into Test::Coverage or something > like that? Well, I don't really know about Test::Builder, but at some point, when Devel::Cover works well, I think that a make cover target might be useful. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
[ANNOUNCE] Devel::Cover 0.16
Just some bug fixes really. Oh, and the HTML output routine should be able to find the templates now :-) On CPAN and at my homepage, as usual. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
[ANNOUNCE] Devel::Cover 0.18
This release adds complete condition coverage, as long as I haven't forgotten anything. New is coverage for xor, &&= and ||=. I also try to be a bit more lenient about constructs such as $val = shift || "???" Hmm. I suppose I'll have to add support for // now. Usual caveats apply. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
[ANNOUNCE] Devel::Cover 0.19
If you liked that release, you're going to love this one ;-) It's just the same, but without those annoying "Use of uninitialized value" warnings. Well, most of them anyway. To get rid of them all you'll need bleadperl. It's good to see that QA is alive and well and working in the wee hours. At least I didn't make two releases on the same day :-) -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
[ANNOUNCE] Devel::Cover 0.20
Mostly this is a bugfix release. You'll need it if you are using AIX or threads. It also contains the start of cpancover, which will hopefully turn into something similar to cpansmoke. At the moment though it mostly just gives me more tests and more bugs to fix :-) http://www.pjcj.net/testing_and_code_coverage/ shows what cpancover does. Just don't expect too much (yet) and remember that the output may say more about Devel::Cover than the module it is testing. Some of the results look interesting, nevertheless. You can also see my slides and paper from YAPC::Europe at the same address. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Test::Class - comments wanted
On Mon, Oct 14, 2002 at 05:46:38PM -0400, Michael G Schwern wrote: > OTOH, my thinking recently is that the explicit plan has become obsolescent. > [1] > [1] This thinking makes me nervous, so I'm open to someone convincing me > otherwise. My take on this is that tests should failsafe. I don't know that I can think of all the ways that one of my tests could fail to print the expected output, so I prefer to explicitly state how many tests I am going to run. Though as Tels points out, this is probably not the biggest testing problem facing most modules. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Testing failure in the core
Recently I reported a bug which caused perl to segv. http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2002-10/msg00346.html I thought the solution might be more complex than it turned out to be, and so I included a patch to the test suite to add a TODO test using fresh_perl_is(). Rafael was quite rightly concerned about this. When the bug is fixed we don't want unneeded fresh_perl_is() tests lying around. What is the correct solution here? I was planning to submit another patch when the fix was done. I suppose the passing TODO test would be a reminder to do something. Is this the best way to handle this, or have I missed something? In this case, Rafael fixed the bug and added passing tests, so there was no problem, but in general what is the solution? Now I've asked the same question three times I'll wait for enlightenment :-) -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Problem with coverage
[ Please note followups set to perl-qa, perl-qa-metrics is pretty much dead ] On Tue, Dec 03, 2002 at 07:45:04PM +0530, Rajanikanth Dandamudi wrote: > Paul, > > I had downloaded the Devel::Cover module form the URL > http://search.cpan.org/author/PJCJ/Devel-Cover-0.20/ and trying to get > the coverage for a program. I had installed the Devel::Cover module > and here is what I had done: > > * I am using the perl version 5.8.0 . > > The command I had given is : > > perl -MDevel::Cover=-db,program_1,-coverage,statement,time program_1.pl > > The file program_1.pl contains: > = > #!/usr/local/bin/perl > > print "Hello World\n"; > > = > > The above command creates a directory and by name program_1 and > creates a file by name cover.4 in it. > > Now when I run the command : > > cover program_1 -report text > > I get the following output : > > Reading database from program_1 > > > -- -- -- > File stmt time total > -- -- -- > program_1.pl 0.00n/a 0.00 > Total 0.00n/a 0.00 > -- -- -- > > > program_1.pl > > line err stmt time code > 1 #!/usr/local/bin/perl > 2 > 3 *** 0 print "Hello World\n"; > 4 > > Paul, I am not able to understand why is it showing the coverage as 0. Neither am I. I went through exactly the same procedure, except with bleadperl, and everything worked as expected. Did all of the tests pass? Is this a 64bit perl? Has anyone ever run Devel::Cover with a 64 bit perl? Maybe it's time for me to plug in my Tru64 box. Could you send your perl -V and program_1/cover.4? > Thanks and Regards, > Rajanikanth > > > Paul Johnson wrote: > > > Rajanikanth Dandamudi said: > > > Hai > > > > > > I am planning to get the coverage of a file by name program_1.pl . This > > > program_1.pl dynamically loads a c module defined in program_1.so . This > > > program_1.so is a 64 bit ELF shared library and the perl available in > > > the standard location is a 32-bit perl. > > > > > > So I had downloaded the perl v5.8.0 from cpan and compiled to obtain > > > 64-bit perl. The module Devel/Coverage.pm which I generally use for > > > obtaining coverage is not part of the perl distribution. Hence I > > > downloaded the package(Devel-Coverage-0.2.tar.gz) and installed the > > > Devel/Coverage.pm module. > > > > > > Now when I run the following command to obtain the coverage : > > > > > > perl -d:Coverage program_1.pl > > > > > > I get the following error message: > > > > > > No DB::DB routine defined at > > > /proj/dite/WorkArea/Raja/perl/install/lib/site_perl/5.8.0/Devel/Coverage.pm > > > line 4. > > > BEGIN failed--compilation aborted at > > > /proj/dite/WorkArea/Raja/perl/install/lib/site_perl/5.8.0/Devel/Coverage.pm > > > line 4. > > > Compilation failed in require. > > > BEGIN failed--compilation aborted. > > > > > > > > > Can someone helpme in understanding what the above problem is and how to > > > overcome this? > > > > This is a known problem. Randy, the author, has said that he will provide > > an updated version when he can. I think the solution involves declaring > > the DB::DB subroutine earlier, but I am not sure. Your other options are > > to go back to 5.6.1, or to try Devel::Cover. > > > > Devel::Cover should work on a 64 bit perl, but feel free to send me > > feedback if you try it. Randy, do you have any thoughts about this? -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Problem with coverage
Rajanikanth Dandamudi said:
> Paul,
>
> perl -V on my machine gives :
>
> Summary of my perl5 (revision 5.0 version 8 subversion 0) configuration:
> Platform:
> osname=solaris, osvers=2.7, archname=sun4-solaris
I can get hold of a Solaris 2.8 machine ...
> uname='sunos apdsparc136 5.7 generic_106541-17 sun4u sparc '
> config_args=''
> hint=recommended, useposix=true, d_sigaction=define
> usethreads=undef use5005threads=undef useithreads=undef
> usemultiplicity=undef useperlio=define d_sfio=undef
> uselargefiles=define usesocks=undef use64bitint=define
> use64bitall=define uselongdouble=undef
> usemymalloc=n, bincompat5005=undef
> Compiler:
> cc='/apps/CC50/SUNWspro/SC5.0/bin/cc', ccflags ='-xtarget=ultra
> -xarch=v9
> -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -xarch=v9
> -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
but not Sun's cc, nor can I easily get hold of an appropriate version
of gcc to compile with -Duse64bitall.
> optimize='-O',
> cppflags='-xtarget=ultra -xarch=v9 -D_FILE_OFFSET_BITS=64
> -D_LARGEFILE64_SOURCE'
>
> ccversion='WorkShop Compilers 5.0 98/12/15 C 5.0', gccversion='',
> gccosandvers=''
> intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=87654321
> d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
> ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
> lseeksize=8 alignbytes=8, prototype=define
> Linker and Libraries:
> ld='/apps/CC50/SUNWspro/SC5.0/bin/cc', ldflags ='-xtarget=ultra
> -xarch=v9
> -xarch=v9 '
> libpth=/usr/lib/sparcv9
> libs=-lsocket -lnsl -ldl -lm -lc
> perllibs=-lsocket -lnsl -ldl -lm -lc
> libc=/usr/lib/sparcv9/libc.so, so=so, useshrplib=false,
> libperl=libperl.a gnulibc_version=''
> Dynamic Linking:
> dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' '
> cccdlflags='-KPIC', lddlflags=' -G -xarch=v9'
>
>
> Characteristics of this binary (from libperl):
> Compile-time options: USE_64_BIT_INT USE_64_BIT_ALL USE_LARGE_FILES
> Built under solaris
> Compiled at Nov 29 2002 10:54:30
> %ENV:
> PERL5DB="BEGIN { require
> '/vobs/pyramid/perl/distribution/sun7/perl5.005_03/lib/5.00503/perl5db.pl'}"
I don't suppose this has any effect.
> PERL5LIB="/usr/local/lib/perl5"
Is there anything special in here?
> @INC:
> /usr/local/lib/perl5
> /proj/dite/WorkArea/Raja/perl/install/lib/5.8.0/sun4-solaris
> /proj/dite/WorkArea/Raja/perl/install/lib/5.8.0
> /proj/dite/WorkArea/Raja/perl/install/lib/site_perl/5.8.0/sun4-solaris
> /proj/dite/WorkArea/Raja/perl/install/lib/site_perl/5.8.0
> /proj/dite/WorkArea/Raja/perl/install/lib/site_perl
>
> and
>
> program_1/cover.4 is :
>
> $db = {'collected' => ['statement','time'],'cover' => {'program_1.pl' =>
> {'statement' => {3 => [[[0]]]}}},'indent' => 0};
Looks about right for what you have reported.
Did you run make test on Devel::Cover? Did the tests pass?
Has anyone had any success with Devel::Cover on a 64 bit machine?
--
Paul Johnson - [EMAIL PROTECTED]
http://www.pjcj.net
Re: Graphically depicting coverage vs. test results
On Tue, Dec 10, 2002 at 12:48:29PM -0500, Barrie Slaymaker wrote: > Here's an interesting way of depicting the statements that are likely > to have cause test failures: > > http://www.cc.gatech.edu/aristotle/Tools/tarantula/ > > Tarantula displays each source code statement using color models > that reflect its relative success rate of its execution by the test > suite. Roughly, statements that are executed by a failed test case > become more red, and statements that are executed by a passed test > case become more green. I had been thinking about ways to make code coverage information more useful, and had considered something along these lines. I also had thoughts along the lines of all tests not being equal. Generally the earlier a test is run, the more important it is. I also have plans for providing an ordering for tests such that tests which give the greatest increase in coverage are run earlier. But all this is for the future. For me anyway. Here's something I've been working on recently. It's ugly, but that's the level of my artistic competence. It's also inaccurate. That, I'm working on. http://pjcj.sytes.net/cover/cpancover/cpancover.html -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Graphically depicting coverage vs. test results
On Mon, Dec 16, 2002 at 04:49:46PM +, Tony Bowden wrote: > On Fri, Dec 13, 2002 at 12:16:53AM +0100, Paul Johnson wrote: > > I also had thoughts along the lines of all tests not being equal. > > Generally the earlier a test is run, the more important it is. > > This isn't necessarily true. > > Test::Class, for example, runs tests in alphabetical order ... Well, yes, I think this is true for most (all?) of Perl's core and CPAN modules, but a developer is free to impose an order by playing with the names of the tests, similar to the way things work with rc scripts. Maybe I should have said that tests for basic functionality should probably be run before tests for more advanced functionality, as happens in the perl core. Fortunately, "basic" comes fairly early in the alphabet :-) -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Graphically depicting coverage vs. test results
On Tue, Dec 17, 2002 at 11:35:38AM +, Adrian Howard wrote: > Ah. Confusion of vocab. You're talking about the order of the test > scripts rather than the order of tests run by those scripts. Yes? Ah, Yes. > I think Tony was talking about the order of tests. OK. I think I'm hampered by never having used Test::Class. But I suspect that the same technique could be used within Test::Class if desired. In any case, it seems that sometimes the decision to add tests to an existing script or to create a new script is sometimes fairly arbitrary. > While key test scripts tend to be run earlier in some setups, I'm not > sure that's it going to be a useful distinction all of the time. There > are certainly lots of test setups where it's not true for me. Agreed. It was really just a half baked idea of mine that may or may not have any practical benefit in the future :-) -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: pls suggest me huge package in perl with testsuites
[EMAIL PROTECTED] said: > On Mon, Feb 10, 2003 at 07:21:39PM +0400, Vlad Harchev wrote: >> But anyway, they more big packages will be recommended, the better, so >> keep >> suggesting them :) > > Off the top of my head, some well tested modules: > Math::BigInt > Test::More > some of the recent stuff by Abigail > Test::Harness Feel free to take a peek at my cpancover project, which should give you a (very) rough idea about the size of various modules and how well they are tested. http://pjcj.sytes.net/cover/cpancover/cpancover.html -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: WWW::Mechanize 0.37 released
Andy Lester said: > There's a new version of WWW::Mechanize for you people who do automated > testing with it. It adds a title() and is_html() method, and removes > the dependency on Clone, which I hope will fix stability issues on > Windows. Nice, thanks. I installed it on a machine at work, and a large number of tests failed because the machine has no access to the outside world. Fixing that is probably more effort than it is worth, but you might want to keep it in mind for the next major rewrite. On the other hand, I think Perl is almost unique in that as a rule software is distributed with tests to be run at installation time, and if the software is being installed in an environment where those tests cannot easily be run, maybe just skipping them is an appropriate response. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Devel::cover 0.20 and daemon, show not covered lines of code
On Tue, Mar 11, 2003 at 03:20:17PM +0100, [EMAIL PROTECTED] wrote:
> Cover works fine for my client client.pl script.
> But when I run a SOAP daemon, I can't get a cover_db produced: When I kill
> the daemon, the Cover is stopped too.
>
> perl -MDevel::Cover start_daemon.pl
>
> where start_daemon.pl contains something like
>
> my $daemon = SOAP:Transport::HTTP::Daemon
> -> new (...)
> ->dispatch_to("my_modul.pm");
>
> $daemon->handle;
>
> When I send a kill -1 or kill -10 or kill -15 signal to the daemon, the
> daemon stops, but Cover stops too.
> How do I get a cover_db when running a daemon?
The trouble is that the DB is not written until your program ends. Do
you have any way of stopping the deamon without killing it? Otherwise,
you could try calling Devel::Cover::report() at some point from your
deamon.
> Another question is:
> How can I see which statements and branches are covered and which not?
> Because currently I know that for example 80% are covered, but not which
> branches are not covered.
Have you run the "cover" program? That will give you a detailed textual
or html report.
--
Paul Johnson - [EMAIL PROTECTED]
http://www.pjcj.net
Re: error message from Devel::Cover 0.20
[EMAIL PROTECTED] said: > I get the following error message from Devel::Cover: > > Can't locate object method "find_cv" via package "B::SPECIAL" (perhaps you > forgot to load "B::SPECIAL"?) at /usr/lib/perl5/5.6.1/i586-linux/B.pm line > 213. > Do I make something wrong or is it a bug? > I am using a Suse Linux 7.3 standard perl installation version 5.6.1. > The make, make test and make install worked without any complainments. That'll be a bug, then :-/ I suspect that your program is producing an optree that I hadn't anticipated, or that I am messing up in some way. If you are able to produce a minimal example and send it to me, I would be grateful. (I'd be even more grateful if you include a patch, too :-) I know that getting a small example might be tricky, so just send me what you can, if you can. If you are able to test against 5.8.0 or bleadperl, that would also be helpful - otherwise I'll do that myself. Thanks, -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Devel::Coverage warning about POSIX.pm
Danny Faught said: > A "use POSIX" statement is causing Devel::Cover to issue a warning. > I've seen it both on Linux (RH7.3/Perl 5.6.1) and Cygwin (Win2K/Perl > 5.8.0). Has anyone investigated it? Hey, a harmless warning is the least of your worries ;-) I had noted the message. but not done anything about it until now. The warning comes because of the way POSIX.pm is using AutoSplit. I'm not convinced that the ../../lib/POSIX.pm filename is terribly useful to anyone, but that's what's in the .al files for POSIX. In any case, I've put in a workaround specifically for POSIX, which seems to be the only core module with this particular affliction. I'm not sure whether this is a general problem liable to affect other modules - I never use AutoSplit. If the problem is general I'll need a general solution. The fix will be in the next release. The cheque is in the post. Actually, I'm currently delaying the next release because bleadperl has recently changed the optree for foreach loops (a change for the better as far as Devel::Cover is concerned). I'm waiting for the dust to settle, and trying to decide the best way to handle all Devel::Cover's tests now failing. I'd welcome any feedback and ideas for improvements. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Renaming modules (was Re: [ANNOUNCE] Test::Warn::None 0.02)
On Sat, Jun 28, 2003 at 10:13:06PM +0100, Fergal Daly wrote: > Is there a way to > know if Makefile.PL is being run by CPAN.pm? Not as far as I know, but Jos tells me that there should be some way in the next version of CPANPLUS. My personal itch that this would scratch is to be able to specify prerequisites if they can be easily satisfied, and just to output a message if the installation is manual and the prerequisites are not essential. (Yes, I know that doesn't make sense.) -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: passing arguments to tests
On Sun, Jul 13, 2003 at 03:22:43PM +0100, Fergal Daly wrote:
> Hi,
> is it possible with Test::Harness and MakeMaker to pass arguments to my test
> scripts? I think it's not but I just want to check for sure. The module I'm
> working on is getting a new "optimised" mode so I'd like to be able to run
> all the tests twice, once with and once without the optimiser.
>
> One possibilty is to have a file called run_opt which is something like
>
> #! /usr/bin/perl
>
> ($script, $optimise) = $0 =~ /(.*?)(_opt)?/;
>
> system($script, "otimise");
>
> then do
>
> ln -s run_opt script1_opt
> ln -s run_opt script2_opt
> ln -s run_opt script3_opt
> ln -s run_opt script4_opt
>
> But that's a bit of a pain. What will happen to the symbolic links on non-unix
> platforms? I suppose you could do a cp rather a ln -s.
>
> Anyway, it'd be nice to be able to somehow tell Test::Harness to run all these
> scripts multiple times in these different ways. Any chance of that happening?
>
> How about allowing run_tests to accept not just strings but also objects like
>
> {
> script_name => 'script1',
> environments => [
> {
> perl_switches => '-w -MOptimise',
> arguments => [1, 'wibble'],
> env_vars => { BLAH => 1 },
> setup_script => 'setup.pl',
> teardown_script => 'teardown.pl',
> },
> {
> perl_switches => '-w',
> arguments => [0, 'wobble'],
> env_vars => { BLAH => 2 },
> }
> ]
> }
>
> I'm not that bothered about env_vars and setup/teardown stuff but the switches
> and args would be nice. I'll make a patch if there's any chance it would be
> accepted,
The way I deal with this is to make a module that does all the heavy
lifting, give it an appropriate import sub, and then a test reduces to
something like:
use TestX ( opt1 => "x", opt2 => "y" )
That covers everything except the perl switches, but although I've never
needed that functionality, I suppose it could be quite important.
--
Paul Johnson - [EMAIL PROTECTED]
http://www.pjcj.net
Re: passing arguments to tests
On Sun, Jul 13, 2003 at 04:13:07PM +0100, Fergal Daly wrote: > On Sunday 13 July 2003 15:53, Paul Johnson wrote: > > The way I deal with this is to make a module that does all the heavy > > lifting, give it an appropriate import sub, and then a test reduces to > > something like: > > > > use TestX ( opt1 => "x", opt2 => "y" ) > > > > That covers everything except the perl switches, but although I've never > > needed that functionality, I suppose it could be quite important. > > Do you make separate .t files for every test and combination of parameters? Yes > That's what I'm hoping to avoid, But I look on it as a feature. It means that my test interface is standard and it is easy to run a single test or a subset of tests. That is not something I would give up lightly. But I have nothing against improving the standard test interface. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
[ANNOUNCE] Devel::Cover 0.21
I've finally found no compelling reasons not to release Devel::Cover 0.21. Highlights of this release are: - spiffy new HTML backend from Michael Carman - pod coverage is working again - time coverage is much more accurate - works with 5.8.1 (to be) and current 5.9.0 - bug fixes - ... and many more It should be on CPAN now, or get it from my homepage if you have a slow mirror. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Devel::Cover and Module::Build
On Thu, Jul 31, 2003 at 05:31:53PM +0200, Tels wrote:
> On Thursday 31 July 2003 12:47, Richard Clamp wrote:
> > On Thu, Jul 31, 2003 at 11:39:48AM +0100, Tony Bowden wrote:
> > > Has anyone added a "./Build cover" target for Module::Build yet? I'm
> > > trying to move some of my stuff from MakeMaker to Build, and can't
> > > really work out how to pull my "make cover" equivalent across...
> >
> > From Siesta::Build, Siesta's Module::Build subclass:
> >
> > sub ACTION_cover {
> > my $self = shift;
> > $self->depends_on('build');
> > system qw( rm -rf cover_db );
>
> Not very portable :)
"cover -delete" would be the recommended way to do that. Portable, and
should still work when/if the database format changes.
--
Paul Johnson - [EMAIL PROTECTED]
http://www.pjcj.net
Re: About Devel::Cover
On Sun, Jul 27, 2003 at 07:07:01PM +0200, alian wrote: > First thanks for your talk on YAPC 2003 at Paris on this subject. > The first thing i've do when I was come back was to test your module. > > Here feedback. Thanks very much. I think most of your points are addressed in the latest release, and what isn't is on the TODO list. > In the README: > > Requirements: > - Perl 5.6.1 or 5.7.1. > * May be 5.8.0|1 ? ;-) > * Or 5.6.1 or greater ? Actually, it was that way because 5.7.0 was not up to the job, and 5.8.0 didn't exist. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: [ANNOUNCE] Devel::Cover 0.21
On Tue, Sep 02, 2003 at 01:43:06PM +0200, Paul Johnson wrote: > I've finally found no compelling reasons not to release Devel::Cover > 0.21. And hot on its heels comes 0.22. This is a bugfix release. The first line of DESTROY blocks and overload subs (and possibly some other things) was not being covered. (This was a bug in previous versions too.) -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
[ANNOUNCE] Devel::Cover 0.23
On Tue, Sep 02, 2003 at 11:17:00PM +0200, Paul Johnson wrote: > On Tue, Sep 02, 2003 at 01:43:06PM +0200, Paul Johnson wrote: > > > I've finally found no compelling reasons not to release Devel::Cover > > 0.21. > > And hot on its heels comes 0.22. You wait for ages, and then three come along at once. This release places condition coverage associated with elsif and else on a better line (can't get it on the right line, unfortunately), makes the "all" coverage value work and makes it the default, and adds subroutine coverage, which is even weaker then statement coverage. Oh, and the TODO list has grown. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Devel::Cover can't find loaded modules
On Fri, Oct 03, 2003 at 10:44:24AM -0700, Ovid wrote: > Hi all, > > I was running some test code with Devel::Cover and I've had no problem > using it or generating pretty reports that make coworkers "ooh" and > "ahh". Unfortunately, I started running my test suite on a different > set of tests and started getting some strange errors (formatted for > clarity): > > Devel::Cover: Can't find file "blib/lib/Text/CSV.pm": ignored. > Can't locate object method "find_cv" via package "B::SPECIAL" > (perhaps you forgot to load "B::SPECIAL"?) at > /usr/local/stow/ASperl-5.6.1-629/lib/5.6.1/i686-linux/B.pm line 213. > > To narrow down the problem, I tried running this on a smaller set of > tests, only to get the above output, but with the following line > prepended: > > Devel::Cover: Can't find file "blib/lib/Storable.pm": ignored. > > I'm not sure how running fewer tests is generating more errors, but in > digging through Devel::Cover, I can't even find a reference to > Text::CSV The message here is trying to say that Devel::Cover is wanting to provide coverage information about Text::CSV and Storable, but is unable to locate these modules. That it is looking in blib/lib would suggest when you ran the test perl picked up those modules from a blib directory. That seems a little strange to me, but I'm sure you have your reasons. It is likely that when this message is given you are in a different directory and since the path is relative the module cannot be found. Do you actually want coverage on these modules? Normally you would not, unless you were running their own test suites. If you do not want the coverage information you can simply ignore the warning, or better, use -ignore +-inc and -select to narrow down the list of modules for which you want coverage information. This has the bonus of speeding up runtime by not collecting data you don't want. >nor has digging through B.pm shed any light (though I may > require therapy now). The find_cv error is a real bug. It will be caused either my me using the core B modules incorrectly (not handling the op tree correctly), or by me using B::Deparse in a way it was not expecting. > Googling has also been fruitless for me. > > Has anyone seen any errors like this before and can you give me some > guidance on what I can do to alleviate them? I'm trying to create a > small test case, but this has been pesky to narrow down. In the > meantime, I'm going to try to add signal handlers to dump stack traces > and maybe even symlink a 'blib/lib' to the appropriate libraries (what > a nasty hack!). Right. I don't think the blib problem is related to the find_cv one, so either ignore it or deal with it as I mentioned above. If you are able to narrow the bug down at all I'd be very grateful for a test case. And a patch if you fancy. Otherwise I'm slowly working through CPAN modules using cpancover and trying to fix bugs like this as I go. CPAN is my test suite ;-) If you just want to get some results from the thing, you could try determining which of the coverage criteria is causing the problem, though it may well be common to all the criteria. You could also try putting an eval around the deparse calls at the end of get_cover() in Devel::Cover.pm, though this will likely reduce the reported coverage where the error has occurred. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Devel::Cover can't find loaded modules
On Sat, Oct 04, 2003 at 07:23:09AM +1000, Andrew Savige wrote: > Ovid wrote: > > I was running some test code with Devel::Cover and I've had no problem using > > it or generating pretty reports that make coworkers "ooh" and "ahh". > > Unfortunately, I started running my test suite on a different set of > > tests and started getting some strange errors... > > I have no experience with Devel::Cover yet, but remember this post: > http://nntp.x.perl.org/group/perl.qa/2213 > which states that Devel::Cover cannot be run from the 't' directory I don't think there is anything special about the t directory that upsets Devel::Cover. The problem is related to relative paths and changing directories within the tests/test suite. As a partial solution to the problem you can try giving an absolute path to the database when Devel::Cover is run, eg perl -MDevel::Cover=-db,$PWD/cover_db ... This particular problem seems to have hit the top of the most wanted list so I'll see what I can do with it for the next release. Who are these people who change directories and don't put things back as they found them? ;-) -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Devel::Cover - what do the numbers mean ?
On Sat, Oct 04, 2003 at 11:25:27PM +0200, Gabor Szabo wrote: > On Sat, 4 Oct 2003 [EMAIL PROTECTED] wrote: > > > On Sat, Oct 04, 2003 at 11:00:33PM +0200, Gabor Szabo wrote: > > > But what are those numbers and colors in the columns ? > > > I could not find in the documentation. Please point me to the > > > description if it is available somewhere. > > > > Percentage of the code covered, I think. Right, in general. > That's what I thought and recalled from the presentation of Paul > and I understand the red 50 in branch and the red 33 in cond but > I don't understand the green 33 or the green 1 in stmt. > And the best is the green 108 stmt. Since a statement is either 0% covered or 100% covered I don't waste space by reporting the coverage as a percentage. Instead the statement column tells you the number of times the statement was exectued. Thus, anything over 0 is OK, and coloured green. The colours are just eye candy, they map to the numbers. The idea being that the red stuff is what you want to look at first. I suspect that you are right, and this is probably not documented anywhere. > Sure I need some hand holding on this. Sounds like you're basically there to me. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
[ANNOUNCE] Devel::Cover 0.24
This is mostly a bug fix release. Hopefully, the problem with tests which chdir has been fixed. If anyone still sees problems in this area, normally manifest with cover_db database directories springing up in unexpected places, please let me know. Devel::Cover wouldn't run on Windows systems. From what I see on the ActiveState site I've tried to fix that, but I can't test it. If anyone has built their own perl on Windows and would like to try building Devel::Cover, I'd be very pleased to get a report on how far Devel::Cover gets now. I've made a few changes to cpancover too. http://pjcj.sytes.net/cpancover/ Enjoy. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Devel::Cover - what do the numbers mean ?
On Mon, Oct 06, 2003 at 12:14:55PM +0100, Tim Bunce wrote: > Changing statement column header to Statement Executions and removing the > colour from that column, or changing it to a very pale green, would help. I'll be attacking the output a little in the next release (0.25). I'll see what I can do on this front. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
[ANNOUNCE] Devel::Cover 0.26
Yes, you blinked and missed 0.25. I managed to put out a release that didn't work with 5.6.1, except under the conditions with which I tested it, so 0.25 came out rapidly to fix that. This release adds a couple of other enhancements that were only half implemented in 0.25. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Phalanx & Devel::Cover
On Sun, Oct 12, 2003 at 01:08:39PM +, Mark Stosberg wrote:
> I'm excited to see that the Phalanx project is happening.
>
> On the website I see this unfiled item:
>
> "Use Devel::Cover and gconv".
>
> One way that seems useful to use Devel::Cover is to have an automated coverage
> testing system that would test the 100 module periodically.
That would be useful.
> All the phalanx page, an extra column on the module page could like to the
> coverage results. This could be another way to track the test the progress.
>
> One sticking point I've noticed is that a different syntax is needed for
> modules that use Module::Build. That seems surmountable, though.
And it has already been surmounted. I sent a patch last week and Ken
Williams told me it has been applied in CVS and will be in the next
release, which will hopefully be before Hugo assimilates the module.
If you need it sooner:
--- Module/Build/Base.pm.org2003-10-07 17:12:31.0 +0200
+++ Module/Build/Base.pm2003-10-08 22:09:07.0 +0200
@@ -972,9 +972,9 @@
$self->depends_on('code');
# Do everything in our power to work with all versions of Test::Harness
- local ($Test::Harness::switches,
-$Test::Harness::Switches,
- $ENV{HARNESS_PERL_SWITCHES}) = ($p->{debugger} ? '-w -d' : '') x 3;
+ local ($Test::Harness::switches, $Test::Harness::Switches) = ('', '');
+ local $ENV{HARNESS_PERL_SWITCHES} = $ENV{HARNESS_PERL_SWITCHES} .
+ ($p->{debugger} ? ' -w -d' : '');
local ($Test::Harness::verbose,
$Test::Harness::Verbose,
> A second sticking point could be that code is OS-specific, so it's not ever going
> to get testing by just one build machine.
Correct, as is Tony's observation about different databases. Then
there's threads, filesystems, other modules which may or may not be
installed, external programs which may or may not be available, blah
blah blah, und so weiter und so fort etc. Fortunately, after running
Devel::Cover on these disparate systems, you can merge the databases to
collect the combined coverage.
It is also going to be a problem just getting these 100 modules
installed. I speak from some bitter experience, having spent some time
last week trying to do just that. An easily automated task it is not.
However, if you can install the modules and run the tests on appropriate
systems I think running Devel::Cover should be almost trivial in
comparison :-)
The current state of the art is http://pjcj.sytes.net/cpancover/ which
is just the modules I could (reasonably) easily install and test on my
system.
--
Paul Johnson - [EMAIL PROTECTED]
http://www.pjcj.net
Re: Devel::Cover bug in ActiveState Perl for *Linux*
On Mon, Oct 13, 2003 at 01:49:58PM -0700, Ovid wrote: > After much research, I've managed to reduce a Devel::Cover bug down to > a one-liner which only fails on one box. This box is unique because > it's ActiveState Perl for Linux (not my choice!). However, that might > be a red herring. My 'perl -V' is below the test case. Does anyone > have a similar setup which they can check this against? I think the key is that it is perl version 5.6.1. I can reproduce the problem on a stock 5.6.1 on linux (well, hacked just enough to get it to build). Is there any chance that we can just say it is a bug in 5.6.1 that has been fixed? I think that is actually the case. Devel::Cover uses the B modules which were still a little immature in 5.6.1. The test works (in that the bug does not show) for 5.8.0, 5.8.1, maintperl and bleadperl. Actually, I wonder whether 5.6.1 should be supported or not. It generally works, but then you get cases like this. The results are also different between different versions of perl, and often it is 5.6.1 and the rest. But this is a problem which will only get worse. In any case, thanks a lot for tracking this down. I'm sure it wasn't trivial. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: No more code coverage
On Mon, Oct 20, 2003 at 12:24:14PM -0700, Ovid wrote:
> --- Tels <[EMAIL PROTECTED]> wrote:
> > Well, still it would be good to improve the speed of Devel::Cover,
> > on my box (lowly 500 Mhz, yes, I could upgrade to 2 Ghz AMD or so :)
> > a particulary project takes 33 minutes for one run... not exactly
> > what I call "interactive"
> > :)
>
> Frankly, I think some major work could be done here.
I agree. Without more details it is hard to tell, but in my experience
the majority of the resources used on reasonably sized examples are
expended in "database" manipulation, specifically in reading, merging
and writing databases.
I wrote "database" in quotes because currently we are talking about a
flat file, written using Data::Dumper and eval'd in. I have considered
other options - specifically YAML and Storable. I have found YAML to be
even slower and too buggy, and Storable to less reliable. (I never
tracked down specific problems.) This in an area that needs to be
addressed.
> In an attempt to
> eliminate the problem I had, I wrote a quick program that recursed
> through directories and ran the tests that it found there. The core
> of the loop was something like this:
>
> my $count = 1;
> my $total = @test_dirs;
> foreach my $dir (@test_dirs) {
> printf "\nProcessing %3d out of %3d: $dir\n\n", $count, $total;
> chdir $dir;
> my $command = 'perl -MDevel::Cover=-db,/home/ovid/cover_db /usr/bin/rtk_test";
> system($command);
> $count++;
> }
I would suggest that the first thing to do would be to limit coverage
collected to statement cover. Until that's at a reasonable level the
rest of the code coverage metrics are of little use.
> There were 92 test directories in @test_dirs. By the time I got to
> the fourth directory, I had a 45 megabyte coverage database and the
> output of top was:
>
> PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
> 22348 cxp 18 0 1133M 1.1G 19280 R99.9 55.5 30:29 perl
At least you haven't run out of memory yet ;-)
> Needless to say, I killed the tests. I've never dug into the
> internals of Devel::Cover so I don't know how easy it would be to fix
> this (and maybe there's something really silly that I've overlooked).
> Unless I get very lucky, I think the code coverage project here is
> dead.
Try limiting coverage to statement cover and see if that salvages
anything.
Another possibility is to write separate databases, and see if any of
them are exceptionally large, then use the cover program to merge them.
Don't give up _just_ yet. I'd like to try to help you get some value
from Devel::Cover. But do remember, I've never called it anything other
than alpha.
--
Paul Johnson - [EMAIL PROTECTED]
http://www.pjcj.net
Re: Devel::Cover and large test suites
On Mon, Oct 20, 2003 at 08:41:12PM +0100, Tony Bowden wrote: > On Mon, Oct 20, 2003 at 07:02:56PM +, [EMAIL PROTECTED] wrote: > > Practically, the only way to do this is to save the results of each > > test in a seperate cover_db and then selectively merge them to see > > whether or not your coverage changed. Even then, finding the minimal > > set of tests that provides "full" coverage is a trial and error, > > brute force approach. Actually, I think it's equivilant to the > > knapsack problem. But you can get an awful long way with heuristics. First pick the test that provides the greatest coverage. Then pick the test which when merged with previously selected tests provides the greatest coverage. Continue until you reach some limit (possibly no more tests). Skew this somewhat with the time taken to run the test, if desired. In practise this can provide some pretty good results. The coverage can be represented as bitmaps which are or'd together and the whole thing can be fairly efficient, certainly enough to make it worthwhile if your tests are taking an hour to run. This has always been on the TODO list. Or rather, having checked, this has never been on the TODO list, but I've always planned on doing it. > Yes, but that's not really what I'm looking for. > > As I said, I know that this is a rather coarse grained approach. I'm > not looking for something that can automatically reduce my test suite. > I'm looking for something that can raise a large flag and say - "Oi! > This lot of tests over here seem to be doing almost exactly the same > stuff as this stuff over here. Is that deliberate?" Or maybe for a single test in the heuristics above substitute set of tests. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: No more code coverage
On Mon, Oct 20, 2003 at 09:34:38PM +0100, Tony Bowden wrote: > On Mon, Oct 20, 2003 at 10:16:40PM +0200, Paul Johnson wrote: > > I wrote "database" in quotes because currently we are talking about a > > flat file, written using Data::Dumper and eval'd in. I have considered > > other options - specifically YAML and Storable. I have found YAML to be > > even slower and too buggy, and Storable to less reliable. (I never > > tracked down specific problems.) This in an area that needs to be > > addressed. > > Have you considered SQLite? Initially I wanted something with few, or better yet no dependencies. I also wanted something that required little or no work when I changed the internal data structures. I'll compromise on both of these, and especially the latter, for something that is efficient and reliable. I'll look into SQLite. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: No more code coverage
On Mon, Oct 20, 2003 at 02:10:51PM -0700, Ovid wrote: > I'm trying to talk my boss into letting me poke into the internals of > Devel::Cover more, but I doubt that will be approved. If it is, maybe > I can work on this. If you or anyone else does get to look at this the code should all be localised to Devel::Cover::DB.pm and, unless you change the internal representation, the changes will be in the 33 lines of code which make up the read() and write() methods. Of course, I'd expect that the change will be increasing that number ;-) -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: PATCH: (unofficial) Make Devel::Cover use Storable
Tim Bunce said: > On Thu, Oct 23, 2003 at 09:17:24PM -0500, Michael Carman wrote: >> I've included a patch for Devel/Cover/DB.pm in case someone else wants >> to try it. It has *not* been sanctioned by Paul (though he's welcome to >> it) so use at your own risk. Obviously, the file format for the database >> is incompatible with the current version. Thanks very much Michael. > Has anyone tried this yet? Paul, will you be adopting it? About the time this all started I skipped off to Miami for a week (Let's Go Marlins!), so that's why I haven't chipped in at all. The results so far look pretty good. It will be interesting to test it with something that gives the database a really good workout, such as TT2. I'm still a little concerned about hitting problems with Storable, so I think I'll apply the patch and then run cpancover on a bunch of modules and compare the output to that of Data::Dumper/eval. If all goes well I'll keep the patch in for the next release. I'm considering whether it would be useful to keep read only support for Data::Dumper databases, but I suspect that at the moment that is not necessary. Still, I may do something like that, if only because it helps me during debugging. I'm suffering from sleep deprivation at the moment, but I'll try this as soon as I can. Thanks again to all concerned. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: PATCH: (unofficial) Make Devel::Cover use Storable
[EMAIL PROTECTED] said: > I ran some more tests, some of which might be more significant: > >time(sec) db size (kB)peak RAM (MB) > no coverage 15 --- ~ 10 > Data::Dumper+eval246 245 ~ 23.4 > Storable 190 60 ~ 19.7 > no storage 184 --- ~ 18 > > The 'no coverage' run is to provide a baseline. > > For the 'no storage' test, I ran using Devel::Cover, but modified the > read() and write() methods to be essentially no-ops. I did this to > isolate the time overhead of coverage itself, as opposed to the time > spent reading and writing the db. Thanks. This is interesting. Was this using all the coverage criteria? I suspect so. At some point I need to do some tests to determine the average overhead of different criteria but in general running subroutine and statement coverage will give the lowest overhead, adding branch coverage will bump up the overhead significantly, and putting condition coverage on top will get you to the 12x seen above. All dependent on the actual code being covered, of course. There is an overhead in the time needed to collect coverage, the memory required whilst it is being collected, the disk space required to store it, and the size of the (html) report files. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: reducing size of the Devel::Cover html report
Michael Carman said: > On 10/31/2003 8:08 AM, Andy Lester wrote: >> >> Personally, I find the POD useful when I'm going thru a Devel::Cover run. >> I interleave POD with functions and I like to refer to my docs. > > Well, since POD is intended to be documentation (not code or even comments) > it seems reasonable to omit it from a coverage report. That said, if I can > find a clean way to make it an option, I'll do so. That's what the -option option of cover is for, options which are specific to a particular report. Whatever is in there gets passed in to the report() method of the reporter module. None of the reports use that feature at the moment, so it is just possible that things might not work exactly as they should, but the infrastructure is in place. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
[ANNOUNCE] Devel::Cover 0.27
The main change in this release is that I have applied Michael Carman's patch to use Storable as the database format, so I'd be particularly interested in any effects that might have, good or bad. Storable is now a prerequisite, but this should only affect 5.6.1 and 5.6.2, when it comes out. I have other patches waiting to be applied and other bug reports to investigate, but I wanted to get this release out. This release has been tested on 5.6.2-RC1 and 5.8.2 as well as more mature platforms and bleadperl. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: reducing size of the Devel::Cover html report
On Sat, Oct 25, 2003 at 01:27:41PM +0200, Gabor Szabo wrote: > The HTML files generated by Devel::Cover are huge. > I ran coverage on HTML::Template and while the source code > is ~100K the HTML report is 2.5Mb > > So I think some work should be done to reduce this size. > I made a simple change - removing leading spaces from the > templates - this reduced the file size by about 60% for HTML::Template. > > The resulting file looks the same in Mozilla, please check it with other > browsers, eg. IE. With no dissenting voices, I have applied this patch, or something very similar. (What do you mean, you didn't test it?) > It is still huge ~870K but I could not find any easy way to remove > further large chunks without changing the resulting page. > > What do you think about removing > 1) the lines from the source code which are empty > 2) removing the lines that display parts of the documentation ? Michael said he's working on this so we'll see how it looks, but I think I'd prefer that the default be for the source code to be left in its original form. > --- Html_subtle.pm.original Fri Oct 24 21:23:58 2003 > +++ Html_subtle.pm Fri Oct 24 21:43:50 2003 Thanks very much. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
[ANNOUNCE] Devel::Cover 0.28
Changes in this release: - Remove leading whitespace from HTML templates (Gabor Szabo). - Remove obsolete indent option. - Add MD5 checksums (Michael Carman). - Add Html_minimal.pm (Michael Carman) (Obsoleting Gabor's patch before it was released). - Pass unknown cover options to the formatter and remove -option. - Specify the output directory for HTML. - Search up directory trees for modules. The most important change is Micheal's new HTML backend which I have made the default - the others are still available. Thanks Micheal! This makes the -pod, -data and -unified options available to fine tune the report. The report has not changed a great deal by default, though it is a lot smaller. You can see what it looks like at http://pjcj.sytes.net/cover/cpancover.5/ Andy is planning to add an option to prove to test using Devel::Cover, so if you have any suggestions concerning this please followup here or mail him directly. There are some fairly significant changes in this release, so please let me know if anything doesn't seem right. Available from CPAN soon or my homepage now. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
[ANNOUNCE] Devel::Cover 0.29
On Mon, Dec 01, 2003 at 01:45:22AM +0100, Paul Johnson wrote: People seem to be using this module. I keep getting bug reports, suggestions and even patches ;-) Changes in this release: - Merge data from files with identical MD5 checksums (Arthur Bergman). - Add do test. - Handle $x || return. - Keep cover -delete happy when there is no existing database. - In cover, make -file a glob and add -exclude. - Watch for coverage options being set in cover (PERL5OPT set?). - Fix up html_basic and html_subtle. - Make 5.6.x builds a bit quieter. - Clean up time routines in XS code. The last change means that this release might have a better chance of working on Win32 systems. If anyone is able to try that, and optionally fix any problems, I'd be grateful. As usual, available from CPAN soon or my homepage now. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
[ANNOUNCE] Devel::Cover 0.30
Nothing to see here, unless you use Windows, in which case this might have a chance of working. Bug reports welcome. No, wait. I take that back. Enjoy. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
[ANNOUNCE] Devel::Cover 0.31
On Mon, Dec 22, 2003 at 03:44:32AM +0100, Paul Johnson wrote: > Nothing to see here, unless you use Windows, in which case this might > have a chance of working. And here's the version without debugging output. Bah! -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
[ANNOUNCE] Devel::Cover 0.32
This release fixes a few bugs and introduces the concept of runs in the database whereby data for tests are stored separately and merged later when needed. I'm hoping this will speed things up somewhat. - Actually include do test. - Create run concept in database. - Belatedly remove check for Template. - Add branch_return_sub test. - Add finalise_conditions() to collect previously missed coverage. - Fix incorrect coverage results associated with "and" conditions. - Add all_versions utility script. - Put /usr/bin/perl on all shebang lines. A couple of tests fail on Win32, but they are just rounding errors in the percentages. I'll look at that later. Enjoy, -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
[ANNOUNCE] Devel::Cover 0.33
This is a fairly important release because it fixes some errors in the coverage reported. - Get things working on paths with spaces in them. - Documentation clarifications (Andy Lester). - Fix coverage for simple if, elsif and unless conditionals. - Add if test. - Ensure runs are merged in the order they were created. - Don't report multiple data from dynamically created subs. - Add dynamic_subs test. - Fix and document Devel::Cover::Op. - Redo subroutine coverage so anon subs are covered correctly in 5.6.x. Enjoy, -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: [ANNOUNCE] Devel::Cover 0.33
On Tue, Jan 13, 2004 at 10:22:52PM +0100, Paul Johnson wrote: > This is a fairly important release because it fixes some errors in the > coverage reported. And introduces some others :-( Hold off on this release - I'll upload a new one as soon as possible. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
[ANNOUNCE] Devel::Cover 0.34
On Tue, Jan 13, 2004 at 10:39:02PM +0100, Paul Johnson wrote: > On Tue, Jan 13, 2004 at 10:22:52PM +0100, Paul Johnson wrote: > > > This is a fairly important release because it fixes some errors in the > > coverage reported. > > And introduces some others :-( > > Hold off on this release - I'll upload a new one as soon as possible. This one should be OK. Sorry for the noise. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: changes to T::H to enable continuous testing
On Fri, Feb 06, 2004 at 11:03:42AM -0600, Scott Bolte wrote: > I'd like to propose an addition to the Test::Harness parsing > rules to support dependency analysis. That, in turn, allows > monitoring for file changes and selective, immediate > re-execution of test files. Is this the right forum for > that discussion? > > Building on the mini_harness.plx example from > Test::Harness::Straps, I added checks for declarations like > the following: > > DEPENDS_ON "file" # implicit test file dependency > "test_file" DEPENDS_ON "module_file" > "test_file" DEPENDS_ON "data_file" In my development copy of Devel::Cover I have code which records which tests trigger the various coverage criteria. Such information should allow these dependencies to be generated automatically, at least as far as code dependencies are concerned, and would also allow more detailed dependencies to be generated if required. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
[ANNOUNCE] Devel::Cover 0.35
>From the CHANGES file: - Change Text2 to pick up version changes. - Minor documentation updates. - Minor changes to Devel::Cover::Op. - Add outputfile option to HTML output (David Wheeler). - Document -silent option to Devel::Cover. - Add -silent option to cover (David Wheeler). - Make Devel::Cover taint safe, or tolerant at least. - Only add versioned golden results to tests that need them. - Add trivial, md5 and module_no_inc tests. - Increase flexibility of testing system to accomodate md5 test. - Add mani, all_gold and all_test Makefile targets. - Make all_versions skip non-existent platforms. - Add DB option to make dump target. Probably the most important change is that Devel::Cover will now work with programs that turn on tainting. I gather this is important for Phalanx. Hi Andy! Available now from my homepage or from CPAN soon. Enjoy, -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
[ANNOUNCE] Devel::Cover 0.36
This release fixes up a problem with adding coverage to an existing database, and also fixes ... On Mon, Mar 08, 2004 at 11:12:55PM +0100, Philippe 'BooK' Bruhat wrote: > Does Devel::Cover work with programs/modules that fork a lot? It didn't before. With 0.35 it did with a strong following wind. With 0.36 it should, but I've only tested it with one program that forks a little. In other words, I had done most of the work, but hadn't quite finished it. > I tried to use Devel::Cover to have a peek at how my test suite covered > my module's code. I just did what the documentation said: > > cover -delete > HARNESS_PERL_SWITCHES=-MDevel::Cover make test > cover > > But I did get an empty report. Did I forget something? > > I tried that with Devel::Cover 0.31 (I think) and HTTP::Proxy version > 0.12. I'm sorry I didn't try yet with a newer version of Devel::Cover. Back then there were probably multiple processes trying to write to the database at the same time. That shouldn't happen now. But note that any coverage before a fork will get counted twice. I'd be interested to know whether the results look correct to you. I'm not sure how we stand with threads. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Devel::Cover - require 5.8?
I am considering dropping support for Perl 5.6 in Devel::Cover. Whilst Devel::Cover basically works with Perl 5.6.1 and Perl 5.6.2, there are many parts which are difficult or impossible to implement, leading to certain constructs which cannot be covered. I realise that some people, especially those in a corporate environment may not be able to simply upgrade. Until recently I was working somewhere where /usr/bin/perl was Perl 4. I just wanted to get opinions on this move. If too many people make too many good cases for keeping 5.6 support, I'll see what I can do, but I'm finding that supporting 5.6 is taking a lot of time that I would rather be spending on other things. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Devel::Cover - require 5.8?
On Fri, Apr 02, 2004 at 09:45:24PM -0500, Randy W. Sims wrote: > > Paul Johnson wrote: > >I am considering dropping support for Perl 5.6 in Devel::Cover. Whilst > >Devel::Cover basically works with Perl 5.6.1 and Perl 5.6.2, there are > >many parts which are difficult or impossible to implement, leading to > >certain constructs which cannot be covered. > > > >I realise that some people, especially those in a corporate environment > >may not be able to simply upgrade. Until recently I was working > >somewhere where /usr/bin/perl was Perl 4. I just wanted to get opinions > >on this move. If too many people make too many good cases for keeping > >5.6 support, I'll see what I can do, but I'm finding that supporting 5.6 > >is taking a lot of time that I would rather be spending on other things. > > It might be nice if it could operate in a degraded mode for Perls < 5.8. OK - this is the option I've taken. I'm now supporting 5.6 in the sense that Devel::Cover should run, and produce some results, but some coverage data might be missing. I'm recommending the use of 5.8.1 or greater. Thanks for the opinions. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
[ANNOUNCE] Devel::Cover 0.41
This is a pretty important release. It fixes a nasty bug in the last release and improves the coverage collection. To test it I have rerun cpancover. You can see the results at http://pjcj.sytes.net/cpancover/ If you have had problems recently I suggest upgrading. Changes since the last announcement: Release 0.37 - 10th March 2004 - Fix up pod test golden results. - Add limitation documentation (Michael Carman). Release 0.38 - 12th March 2004 - Allow coverage summary title to be changed (David Wheeler). - More care generating pod golden results. - Small Devel::Cover::Op output fix. - Handle "my $x = shift || []" and friends nicely. - Add default_param test. - Provide summary output to one decimal place. - Update gcov2perl. Release 0.39 - 22nd March 2004 - Major database rework to store runs. - Add Devel::Cover::DB::Structure.pm. - Check for Test::Differences in Makefile.PL. - Test with perl5.9.2. - Skip fork test on MSWin32. Release 0.40 - 24th March 2004 - Remove DB structure for unwanted files. - Identify ops based on address and OP contents, except for op_ppaddr, which we modify. - Overhaul of coverage collection. - Additions to tests. Release 0.41 - 29th April 2004 - Correct time coverage percentages. - Collect data for BEGIN, CHECK, INIT and END blocks in the main program, and INIT and END blocks in modules. - Ensure our END block is always the last run. - Don't clean up Pending_conditionals data. - Untaint @INC after using blib. - Be silent if called via HARNESS_PERL_SWITCHES. - Test against perl-5.8.4. - Store cwd from when each module was required in order to find them again. - Refactor and tidy XS code. - Get cpancover running again with the DB changes. - Normalise filenames. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: [ANNOUNCE] Devel::Cover 0.41
On Thu, Apr 29, 2004 at 08:05:35PM +0200, Tels wrote:
> On Friday 30 April 2004 00:40, Gabor Szabo wrote:
> > Where would we be without your work ?
> >
> > Out in the desert to fall prey to any bug ?
> > At least we can run and get some cover.
(!)
> > Thank you !
>
> Exactly! Especially the ability to look at
>
> http://pjcj.sytes.net/cpancover/Math-BigRat-0.12/lib-Math-BigRat-pm.html
>
> without having the "hassle" of downloading/installing Devel::Cover is a big
> plus! I just saw that BigRat misses tests for as_bin() and as_hex() :-)
>
> Thank you!
You're welcome. But be aware that cpancover is probably best used as a
guide. You'll want to run the tests yourself and generate your own
coverage so you can be sure the tests pass and that you understand the
environment in which they are being run. (Tels, I know you do that
anyway.)
> Tels
>
> PS: One stupid question:
>
> In Calc.pm (part of BigInt):
>
> 1350 149 50 408if ($cx->[0] == 0)
> 1351 {
> 1352 0 0 $zero_elements ++; shift @$cx;
>
> If the branch is taken 50% of the time (I think that is what 50 means in the
> second column, then why is the line "$zero_elements++" etc not covered at
> all? Bug in Devel::Cover or did I get something wrong?
The latter ;-)
When branch coverage is 50% that means that 50% of the branches have
been taken at least once, and thus 50% of the branches have never been
taken. If you follow the 50 link you will get to the detailed branch
coverage report which tells you that the true branch (the body of the
if statement) was never taken, but the false branch was (the missing
else). So one out of two, or 50% branch coverage. This fits in with
the statement coverage for the statements in the body of the if
statement being zero.
In order to get 100% branch coverage here you will need a test in which
$cx->[0] == 0 is true.
--
Paul Johnson - [EMAIL PROTECTED]
http://www.pjcj.net
Re: Devel::Cover: completing $x{foo} ||= 1 conditions
On Thu, May 20, 2004 at 04:34:39PM -0400, Geoffrey Young wrote:
> hi paul.
>
> I've found that in a statement like
>
> $x{foo} ||= 1;
>
> I can't ever satisfy the first condition in the "condition coverage" matrix
> (0,0) since 1 is always true. is it desirable to remove fixed truth values
> like this from the truth table?
Well, not to completely remove them, but certainly to downgrade then.
There is already code in place to do that for cases such as
C but the different op tree in this case fooled
me.
This is unlikely to be the only case in which I have not fully
understood the subtleties of the op tree, and so I am grateful for
reports such as this.
The following patch should fix it, and will be in the next release,
hopefully coming next week:
--- lib/Devel/Cover.pm.org 2004-05-18 01:31:08.0 +0200
+++ lib/Devel/Cover.pm 2004-05-21 00:55:13.0 +0200
@@ -638,7 +638,9 @@ sub add_condition_cover
if ($type eq "or")
{
-my $name = $op->first->sibling->name;
+my $r = $op->first->sibling;
+my $name = $r->name;
+$name = $r->first->name if $name eq "sassign";
if ($name eq "const" || $name eq "srefgen")
{
$c = [ $c->[3], $c->[1] + $c->[2] ];
> I tried taking a look at adding the condition to tests/cond_or but I really
> couldn't grok the test suite at first glance :)
It's not one of the more simple test suites out there, that's true. In
part this is because testing requires running the test program in a new
process under Devel::Cover, then running F to generate the
results, then checking the results are correct. To do this, I use
golden results.
So the process is something like:
- Add the code to the test program in tests/cond_or
- Run make text TEST=cond_or and check the results are correct
- Run make gold TEST=cond_or
- Do this for perl5.6.1, 562, 580, 581, 582, 583, 594 and 592,
threaded and unthreaded
- Take appropriate shortcuts by examining the golden results in
test_output/cover/cond_or.* and using all_versions
- Try very hard not to mess up
I've not really expected anyone else to do this up to now. Medium term,
I'd like to improve this process, and I really want to find a way to run
Devel::Cover on itself.
Thanks again for the report.
--
Paul Johnson - [EMAIL PROTECTED]
http://www.pjcj.net
Re: Devel::Cover: completing $x{foo} ||= 1 conditions
On Thu, May 20, 2004 at 08:08:19PM -0400, Geoffrey Young wrote: > well, I was trying to be a good citizen and provide a useful test case > (rather than a tarball that merely exhibited the issue). I did add And I appreciate it. > something to cond_or and ran 'make test TEST_FILES=t/acond_or.t' but I got > more than just one failure so I figured there was more to it :) If you install Test::Differences you should get better output from that command. Without it, you run into the problem that adding or deleting lines from the test means that the remainder of the report has shifted, and so all the comparisons fail. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Style question
On Tue, May 25, 2004 at 07:58:21PM +0200, Francisco Olarte Sanz. wrote: > 1st of all, thanks everyone for the prompt response regarding my > previous question about return values. > > Now a style question. I'm doing a database oriented module, and I have > rouhly the following tests: > > 1.- Test wether module can load ( just a require inside an eval ). > 2.- Test a bunch of functions for string processing (escaping) etc.. > which don't need server connections. > 3.- Test connection to the database. > 4.- Create a test database, connect to it > 5.- Create test tables, populate them, testing insertion functions on > the way. > 6.- Lots of tests which need the test database. > 7.- drop test database. > > I have 1 in a file, 2 split in several. 3 can be put in it's own file, > but the problems arrive with steps 4-7. Steps 5 and 6 are lengthy, and > I'd like to split them into several files, giving files 4, 5a, 5b, 5c, > 6a, 6b, 6c..6z, 7 but in this case steps 4-7 should be run to > completion. This goes against the recomendation of having tests which > can be run separately ( and takes slightly longer as i need to > reconnect on each test, but tests don't need to be so efficient). So > I'm now putting 4- in a single file, but it's becoming huge ( hundreds > of tests ). My question is.. > ¿ Which aproach is better, have a single independent huge test file or > several interdependent smaller ones ( w/ notes in the readme stating > test dependence ) ? In general, I prefer the convenience of independent tests, but I have also written tests which have an ordering where that makes more sense. But remember that tests are just programs. If they are unwieldy, bring standard software development techniques to bear. There is no reason not to include modules in your distribution which are only required as part of the test suite. Then again, I like Joshua Pritikin's parenthetical question in Test.pm, "Your test code should be simpler than the code it is testing, yes?". Hmmm. I'm not sure whether that will help you make a decision or not. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Devel::Cover: completing $x{foo} ||= 1 conditions
On Fri, May 21, 2004 at 12:50:55PM -0400, Geoffrey Young wrote:
>
> >> I might like to signal Devel::Cover that func() has a constant return (or
> >> lack thereof).
> >
> > I don't know if I would like this feature. To me it would allow you to
> > falsely skew the results of the Devel::Cover output. I look at
> > Devel::Cover as a harshly objective analysis of my test-code coverage,
> > anything destroying that objectivity IMO would lessen the value of the
> > tool.
>
> however, in the process of development we are required to analyze any of the
> inevitable gaps and decide whether the unhit condition is valid. if it is
> we write a test for it. if it represents a condition we would explain away
> (D::C limitation, or whatnot) then it would be nice to have some way to
> track it within the tool itself. partially to appease management with heavy
> greens, but more to save development cycles chasing down issues I (or other
> developers) have analyzed before.
I enjoyed this discussion and the various opinions and suggestions.
It has always been my plan to include something along these lines. The
TODO list (which seems to keep growing) mentions "Marking of unreachable
code - commandline tool and gui." I think this is useful for exactly
the reasons Geoff gave. Hacking the figures to appease management seems
to me to be quite a long way down a slippery slope, but (unfortunately)
I can understand the motivation.
For me, <100% coverage has never necessarily indicated an error, which
is why, internally, each item of coverage has associated with it the
number of times it was covered, the total number of times that it could
have been covered, the percentage that represents and, crucially,
whether or not that indicates an error.
I have just released Devel::Cover 0.45 and I've included some code which
is working towards a solution in this area. The code seems to work, but
there is no (reasonable) user interface for it, and it is not documented
at all. You'll need to grep the source for "uncoverable" if you want to
play with it. With this, the textual report can report no errors even
when the coverage is not 100%.
Of course, this was not obvious the Michael when he wrote the html
output backend, and so the colours there are dictated by the percentage,
but if you use my original (and ugly) html_basic backend you can see how
you can get green everywhere without 100% coverage. The halfway house
of html_subtle is, fittingly, halfway between in this regard too ;-)
Michael has some ideas for backends and interfaces to the uncoverable
code, which I'll let him talk about or work on as he sees fit.
Actually, I've just noticed that the test I shipped is broken, which
illustrates the fragility of the interface. Or the testing framework.
Or the code. Or something.
Other features of this release:
- Cope with spaces in build path on Windows (Max Maischein).
- Allow Devel::Cover to be used under mod_perl (Philippe M. Chiasson).
- Handle $x ||= 1 and friends nicely, including subs and *foo{THING}.
Enjoy,
--
Paul Johnson - [EMAIL PROTECTED]
http://www.pjcj.net
Re: ANNOUNCE: New Module::Build beta on CPAN
Cross-posted to module-build-general, p5p and perl-qa. Please trim CCs
if your reply isn't relevant to all the lists. I'm not subscribed to
module-build-general.
On Wed, Jun 02, 2004 at 12:09:01AM -0500, Ken Williams wrote:
> I've just released a new beta of Module::Build to CPAN. This is a
> sanity-check release on my way to 0.26. Changes since 0.25:
> - Added the 'testcover' action, which runs a test suite using
>Devel::Cover. [Dave Rolsky]
Nice. I'd been planning for something like that when I was happy with
Devel::Cover. But I imagine there will likely always be something more
I want to do, and that Devel::Cover is probably in a fairly usable state
at the moment.
A few questions about the code though, which I have reproduced below:
] sub ACTION_testcover {
] my ($self) = @_;
]
] unless ($self->find_module_by_name('Devel::Cover', [EMAIL PROTECTED])) {
] warn("Cannot run testcover action unless Devel::Cover is installed.\n");
] return;
] }
]
] $self->add_to_cleanup('coverage', 'cover_db');
"cover_db" is the default coverage database, but what is the file
"coverage"? I don't think that that is a file produced by Devel::Cover.
] local $Test::Harness::switches=
] local $Test::Harness::Switches=
] local $ENV{HARNESS_PERL_SWITCHES} = "-MDevel::Cover";
I think it is probably most useful for a coverage run to start with a
clean database by default. This would be accomplished, I think, by
$self->do_system('cover -delete');
] $self->depends_on('test');
] $self->do_system('cover');
] }
And now to the name, "testcover". Up to now, I have been using the
target "cover" for this purpose, and I know a couple of other projects
which have been doing the same. I also know other people have been
using Devel::Cover, but I don't know whether they have created a target
for this, and if so, what they have called it.
I'm not particularly attached to any one name, but I would like to
ensure consistency, so now seems like the right time to pick a good
target and stick to it.
I'd be interested in any opinions on this.
And then, at some point, add a similar rule to MakeMaker?
--
Paul Johnson - [EMAIL PROTECTED]
http://www.pjcj.net
Re: New Devel::Cover Noise
On Thu, Jun 17, 2004 at 09:31:48AM -0700, David Wheeler wrote: > I checked in a bunch of changes to my module last night, and then the > nightly process that runs cover on it output these lines: > > Devel::Cover: ignoring extra subroutine > Devel::Cover: ignoring extra statement > Devel::Cover: ignoring extra statement > Devel::Cover: ignoring extra subroutine > Devel::Cover: ignoring extra statement > Devel::Cover: ignoring extra statement > > Anyone know what they mean? What extra statements and subroutines? Good Question! It's a shame you checked in a bunch of changes rather than one small change, otherwise you could have told me ;-) But I can explain what is happening at a low level. For the first forty or so releases Devel::Cover would collect coverage data for each run by noting which ops were executed and then, at the end of the run, it would look through each subroutine and match the constructs in the subroutine with the coverage of the ops to produce the metrics for that run. These data would be stored for each run, and when the cover program was eventually run they would be merged to produce the final report. This worked well, but had a couple of drawbacks, so a little while ago I changed things rather fundamentally. Now, at the end of a run there are two sets of files created. One, which contains the structure of the program - where the subroutines, statements, branches and conditions are, and the other which contains data about how many times each construct was exercised. But the structure data only needs to be created if it does not already exist, so subsequent runs can benefit by not have to create and store these data. The main advantage, however, which is as yet unrealised, is that the data for different runs can be directly compared, which should yield some interesting information. But there is a downside to all this. It's called Perl. Perl is so dynamic that the structure of one run may differ from the structure of another. I suppose this will happen when a subroutine is created at run time in one run but not in another. This would ultimately be through an eval, but might be hidden behind an AutoLoaded function or something similar. What happens then is that on attempting to add collected data to the run information the (first) structure is searched, but the construct is not found, and this warning is emitted. In practise this means that those constructs will not be considered as far as coverage is concerned. This is a bug, and I've just added it to the TODO list. If anyone is able to reproduce this with a small example I'd be very grateful to receive it, otherwise I know of a number of larger modules which also exhibit this problem. Thanks for reading this far. I must stop now as half time is over, so it's back to the football ... -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Removing Tests from Devel::Cover results
On Thu, Jun 17, 2004 at 11:58:16AM -0400, Geoffrey Young wrote: > Tony Bowden wrote: > > Is there any simple way to remove the test files themselves from the > > Devel::Cover results? i.e. just see the coverage analysis of the files > > being tested rather than all the t/* files as well? > > > > We want to monitor the total coverage over time on a project, and it > > would be nice to just grab that from the bottom line of the HTML report. > > But, from what I can see, we can make that number get larger purely by > > adding more and more lines to test files that don't actually test > > anything, as running those lines will increase the coverage of the > > project as a whole! If there were a flag that would -exclude (or > > -ignore, both seem to be documented?), the actual test files, this might > > be slightly nicer. > > ignore and +-inc do the trick for me. my current 'make cover' target starts > like this: > > [EMAIL PROTECTED]::Cover=-ignore,\.t\$$, \ > -ignore,apache_test_config.pm,+inc,$(TOPDIR)/lib, \ > +inc,$(TOPDIR)/Apache-Test\\ This is generally the best way, as Devel::Cover will try to collect as little information as it can while the tests are running. If you have already collected the coverage the -file and -exclude options to cover will allow you to choose the files on which to report. I should probably change those to be -select and -ignore to match. Thank goodness I'm still on alpha versions ;-) -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: testing for unsuccessful require - mocking require ?
On Sat, Jun 19, 2004 at 09:11:43PM -0200, Gabor Szabo wrote:
>
> I would like to test a module for unsuccessful "require" while the
> required module is installed. That is I'd like ot test how my code would
> work if Foo.pm was not present.
>
> In the middle of a module I have code such as
>
> eval {
>require Foo;
> };
> if ($@) {
>foo();
> } else {
>my_own_foo();
> }
>
> I already installed Foo so I can test the behaviour of the code
> with Foo. How can I test now the behaviour without Foo ?
> Removing Foo.pm is not an option.
>
> I have some ideas outlined below, but I'd like to get your recommendation.
[snip]
> In general I'd like to get your opinion on how to mock unsuccessful
> require.
local *CORE::GLOBAL::require = sub { die "require $_[0] failed" };
Goes back to before 5.6. I think it came in in 5.004.
--
Paul Johnson - [EMAIL PROTECTED]
http://www.pjcj.net
Re: Devel::Cover bug
On Thu, Jun 24, 2004 at 12:45:26PM -0400, Vsevolod (Simon) Ilyushchenko wrote:
> Hi,
Hello,
> I've run into "Can't call method "add_statement" on an undefined value"
> running Devel::Cover. Apologies if this was reported before, but the
> list archive is not searchable. I am using perl 5.8.4 and Devel::Cover 0.46.
I think the list is archived in a number of places. I suppose the
official archive is at http://www.nntp.perl.org/group/perl.qa/
And this is the first time that I've come across this bug, so thanks
for reporting it.
> To reproduce the bug, run
>
> /opt/perl/bin/perl -MDevel::Cover -MFooBar -e "FooBar->new->test_foo"
>
> The files FooBar.pm and CodeRef.pm are attached.
>
> The bug occurs while calling $sub in CodeRef::to_string. It is probably
> related to using B::Deparse, but I was not able to minimize the code
> further and still reproduce the error.
Absolutely correct. I was able to reduce the code to:
require B::Deparse;
B::Deparse->new->coderef2text(sub {})
The problem occurs because Devel::Cover overrides some of B::Deparse's
subs, but when you go calling them in a program it gets upset. The
solution is to only override the subs for as long as is necessary. The
change is in my development copy and will be in the next release.
Thanks again for reporting the problem and producing a small test case.
--
Paul Johnson - [EMAIL PROTECTED]
http://www.pjcj.net
Re: Devel::Cover and nested subroutines
On Fri, Jun 25, 2004 at 11:07:06AM -0400, Geoffrey Young wrote: > hi paul :) > > I recently discovered an issue with nested subroutines while using > Devel::Cover with Parse::Yapp. the basic issue is that some subroutines are > not discovered by Devel::Cover and thus no metrics are generated. > > there are two files in the tarball. Foo.pm is a minimal test case showing > that two subroutines are missing (one defined and one anonymous). Parser.pm > is an autogenerated file based on Parser.yp that illustrates the same thing > - note that the _missing() subroutine is missing from coverage results (as > well as a handful of anonymous nested subroutines I think). > > the only interesting thing is that the behavior of my minimal test case and > Parser.pm seem to be opposite - in the former the missing subroutine is the > one that contains the anonymous sub, while in the latter is is the named > subroutine that occurs after a nested anonymous sub. > > anyway, let me know if there is anything else I can do - I realize this is > kind of obscure, so no pressure :) > > oh, and I tested using 0.46 and both 5.8.4 and 5.8.0. Thanks a lot for the test cases. I think there are two separate bugs here, but I'm only going to take responsibility for one ;-) First, mine. The problem with Foo.pm (the minimal test case) is that completely empty subroutines (that is subs which contain no statements at all) are ignored as far as subroutine coverage is concerned. That is the case for both named and anonymous subs. The op tree for an empty sub didn't contain the structure I was looking for, so it wasn't recognised as a sub. I have put in a fix for this, but it only works with Perl 5.8.2 and later versions. I've not gone trying to get it to work with earlier versions, since it is pretty obscure and I prefer to keep the code reasonably clean. Or maybe I'm just too lazy. In any case, let me know if this is going to cause anyone problems. I have documented it as a bug though, and upped the recommended version to use from 5.8.1 to 5.8.2. Then the second bug. The problem here is that if you lie to perl it will bite your bottom ;-) Parse::Yapp appears to take portions of the grammar in Parser.yp and insert them into Parser.pm. In doing so, it quite reasonably wants to reference the original grammar which means that perl will report errors in the grammar where they appear in the grammar. Parse::Yapp does this by using the #line directive. This is all well and good, and Devel::Cover will honour that by reporting the coverage in the original grammar. But ... This means that: 1. The coverage will not be reported in the Parser.pm module. 2. Devel::Cover needs to be able to find Parser.yp. In the example the filename given is Parser.yp, but the file is actually at lib/My/Parser.yp and so Devel::Cover can't find it. Changing the example to give the full filename, or putting in a link from the current directory fixes the problem. I'm not sure if this is actually a problem with Parse::Yapp or just a result of the way you packaged up the testcase. 3. Parse::Yapp doesn't clean up after itself by setting the line number and filename back to what it actually is, which means that subsequent code coverage is reported on in the grammar file even though it didn't come from there, which can be somewhat surprising. So I'm afraid there's not much I can do about this one - it will need to be fed to the author of Parse::Yapp and he can decide if he wants to do anything about it. In any case, the first fix will be in the next release, and thanks again for the great test cases. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Devel::Cover bug
On Sat, Jun 26, 2004 at 08:41:08PM -0400, Vsevolod (Simon) Ilyushchenko wrote: > >The problem occurs because Devel::Cover overrides some of B::Deparse's > >subs, but when you go calling them in a program it gets upset. The > >solution is to only override the subs for as long as is necessary. The > >change is in my development copy and will be in the next release. > Muchas gracias! I am looking formard to using Devel::Cover on my testing > classes (the offending code occured in the Test::Unit framework). Sounds good. I've just released 0.47 with this fix in it, so let me know if there are any other problems. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Devel::Cover and nested subroutines
On Mon, Jun 28, 2004 at 01:55:09PM -0400, Geoffrey Young wrote: > > 2. Devel::Cover needs to be able to find Parser.yp. In the example > > the filename given is Parser.yp, but the file is actually at > > lib/My/Parser.yp and so Devel::Cover can't find it. Changing the > > example to give the full filename, > > you mean changing the #line directive? Yes, sorry, that's exactly what I mean. > > or putting in a link from the > > current directory fixes the problem. I'm not sure if this is > > actually a problem with Parse::Yapp or just a result of the way > > you packaged up the testcase. > > well, the packaging is pretty much the way the code I'm testing is layed out > - Foo.yp in the same directory as Foo.pm, and all under some lib/ someplace. > fairly standard I'd think. Hmm. If the name was My/Parser.yp you could argue that Devel::Cover should search @INC to find it, but as it stands I don't see a sensible and general way to locate the file with the information provided. > but the symlink works - linking Foo.yp to the top-level directory, along > side cover_db. > > > > > 3. Parse::Yapp doesn't clean up after itself by setting the line > > number and filename back to what it actually is, which means that > > subsequent code coverage is reported on in the grammar file even > > though it didn't come from there, which can be somewhat > > surprising. > > hmph. > > > > > So I'm afraid there's not much I can do about this one - it will need to > > be fed to the author of Parse::Yapp and he can decide if he wants to do > > anything about it. > > I suppose I could do that, but it seems kinda strange to ask him to change > stuff around just so we can have good test metrics. but, per your > suggestion, at least there is a simple workaround - thanks for that. Yes, though you could argue that any warnings or errors coming from that code will be confusing anyway. > > In any case, the first fix will be in the next release, > > excellent, thanks. Which is now out. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
