Devel::Cover and large test suites
Hi all, After much agony, I finally managed to get our test suite running with 5.8.1. Unfortunately, the suite of 15,000 tests takes about an hour to run. After it finished, Devel::Cover proceeded to use more and more memory until finally someone on our night crew killed the process. I'm wondering if coverage results are cumulative? If I have tests in four different directories, can I run the tests separately, specifying the same coverage db for each and thereby *gradually* accumulate this information? Are other workarounds available? Cheers, Ovid = Silence is Evilhttp://users.easystreet.com/ovid/philosophy/indexdecency.htm Ovid http://www.perlmonks.org/index.pl?node_id=17000 Web Programming with Perl http://users.easystreet.com/ovid/cgi_course/ __ Do you Yahoo!? The New Yahoo! Shopping - with improved product search http://shopping.yahoo.com
Re: Devel::Cover and reading docs :/
--- Ovid <[EMAIL PROTECTED]> wrote: > I'm wondering if coverage results are cumulative? If I have tests in four different > directories ... Of course they are, if the -merge option is on. I missed that. You may now return to your regularly scheduled programming. Whoops, Ovid = Silence is Evilhttp://users.easystreet.com/ovid/philosophy/indexdecency.htm Ovid http://www.perlmonks.org/index.pl?node_id=17000 Web Programming with Perl http://users.easystreet.com/ovid/cgi_course/ __ Do you Yahoo!? The New Yahoo! Shopping - with improved product search http://shopping.yahoo.com
Re: Taint mode testing and project Phalanx
-BEGIN PGP SIGNED MESSAGE- Moin, On Monday 20 October 2003 04:07, Andrew Savige wrote: > I noticed in Test::Tutorial: > "Taint mode is a funny thing. It's the globalest of all global features. > Once you turn it on it effects all code in your program and all modules > used (and all the modules they use). If a single piece of code isn't > taint clean, the whole thing explodes. With that in mind, it's very > important to ensure your module works under taint mode." > > I don't have much practical experience with taint mode testing, > however, and was wondering how important taint mode testing is > to project Phalanx. Hm, good idea. I have a large project at work, which almost exclusively runs under Taint (it's a server/client thingy). However, the testsuite largely tests under non-taint mode. Hm. Should basically all modules be tested under taind and non-taint, even though they usually are not developed with this in mind? E.g. running their one suite twice, w/ & w/o -T? Best wishes, Tels - -- Signed on Mon Oct 20 18:28:35 2003 with key 0x93B84C15. Visit my photo gallery at http://bloodgate.com/photos/ PGP key on http://bloodgate.com/tels.asc or per email. "Ivonna Humpalot" -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.2-rc1-SuSE (GNU/Linux) Comment: When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl. iQEVAwUBP5QNQncLPEOTuEwVAQGjNQf7BosSLPwq9XPqzcgaJhH12NRtoQxATieU r2KpnwGUvK+zEg8mxiMxKY4s7HKg2AyhDZoJD0WLkg2TjWEm5CAg0HOs3RYdUjOS 4dmq6gLpq9z8o0adD4i2LKDVIim40IOYFOzm8hFTQ5fnvTgT/QAKG3pNhJw2pv5N f3xzTcKuouH8yfk7fvSiC6mLWA0o9p1fqGs1onV4UnWyJ6Efn0yOsb3Vi73sbE34 RmeQp4UQLbToqXKGi3v1Aa+np09A3Us0ADbZcsnxoVuH6t9zK8FaT1UWKnsVBN5A YoDQ4t6rAn31Wc1zfp2FGII4cQu0bZN1ZDrkvBogD3fB4gw0GoutXA== =PBTb -END PGP SIGNATURE-
Re: Devel::Cover and large test suites
On Mon, Oct 20, 2003 at 08:34:50AM -0700, Ovid wrote: > the suite of 15,000 tests takes about an hour to run. Devel::Cover issues aside, has anyone else looked at interesting ways of making test suites run faster? We insist on a full regression test on checkin, so having the test suite take a long time to run puts people off making small changes. :( On one project a full regression test takes about 8 minutes, which is at least an order of magnitude longer than I'd like. I couldn't dream of having it take an hour! Interestingly, a lot of this time is due to the fact that we front-load a lot of our stuff to compile time (dynamic class generation etc.) as it's a web-app. This works well under mod_perl, but not so well under testing as just loading the modules to be tested takes a while - which is noticable when it has to happen again for each different test file. So having some way to have that not need to happen for each test file would be nice. Alternatively we looked at running all the test files in parallel rather than sequentially. This was looking promising, but we never got it finished. I'm curious as to whether anyone else is doing anything interesting here... Tony
Re: Devel::Cover and large test suites
-BEGIN PGP SIGNED MESSAGE- Moin, On Monday 20 October 2003 18:35, Tony Bowden wrote: > On Mon, Oct 20, 2003 at 08:34:50AM -0700, Ovid wrote: > > the suite of 15,000 tests takes about an hour to run. > > Devel::Cover issues aside, has anyone else looked at interesting ways of > making test suites run faster? > > We insist on a full regression test on checkin, so having the test suite > take a long time to run puts people off making small changes. :( > > On one project a full regression test takes about 8 minutes, which is at > least an order of magnitude longer than I'd like. I couldn't dream of > having it take an hour! > > Interestingly, a lot of this time is due to the fact that we front-load > a lot of our stuff to compile time (dynamic class generation etc.) as > it's a web-app. This works well under mod_perl, but not so well under > testing as just loading the modules to be tested takes a while - which > is noticable when it has to happen again for each different test file. > > So having some way to have that not need to happen for each test file > would be nice. > > Alternatively we looked at running all the test files in parallel > rather than sequentially. This was looking promising, but we never got > it finished. > > I'm curious as to whether anyone else is doing anything interesting > here... Basically I do: * "optimize" the tests as to not run so many ok() (e.g. run 1000 tests, output on ok(), or the internal test number that failed), although I avoid this "optimization" since it is not so important * optimize the code to be faster (this is a winner) * run the tests in the background, e.g. run them, while working in the foreground and later see if they broke * run the tests only a few times a day. After all, it only matters if the daily build has them running ok or not, not if each change I do breaks the test or not. Some people believe that each individually change must be regression tested, but where do you draw the line? Chaning a file, a line, a word, a comma, a character? I tend to retest after implementing a "feature", e.g. anything that takes minutes to hours to complete, but is still done before the daily build get's pushed out of the door. Oh, I only "update" the dailys build if all tests pass, meanign I can stop working anytime and still have the "latest" daily build which passes all test. Ah, ignore my ramblings... Cheers, Tels - -- Signed on Mon Oct 20 18:51:24 2003 with key 0x93B84C15. Visit my photo gallery at http://bloodgate.com/photos/ PGP key on http://bloodgate.com/tels.asc or per email. "Man, I'm hot." - "Thirsty?" - "No, I mean good looking." -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.2-rc1-SuSE (GNU/Linux) Comment: When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl. iQEVAwUBP5QSjXcLPEOTuEwVAQFAIAf9FTuz6isjW4n9W+N/5JRif3V9W5LFcKsv pKY++5DiV7PpwyvJwvrqMtk7bMTRpFUAQeX6G6uDM79Kk9gUKJ3CnGR2DTnwIt+R 4eGmyhcr3uZxvURxFFPi0fRYI13If6tqJwbKC0ATHUCrWFnUozJfQlm2HoVD2/Yq wXk2Xw3xHAsrgPLAnJgz3dpoglKLXPdUP1JDLVytRsZIdoTyibWm0b6EtGVlqvE8 gjAvmPkvjVPQV6WQEv5i3mY0ZsHz7cw5E4CdChXnk19yIiQ6/b1w+v8c9SbhFoSj z4LK/DYHksvgfEWgzV2aFcGkEVe4N7x7KvvDe+NmGThma468IveI9Q== =rzUK -END PGP SIGNATURE-
Re: Devel::Cover and reading docs :/
-BEGIN PGP SIGNED MESSAGE- Moin, On Monday 20 October 2003 18:26, Ovid wrote: > --- Ovid <[EMAIL PROTECTED]> wrote: > > I'm wondering if coverage results are cumulative? If I have tests in > > four different directories ... > > Of course they are, if the -merge option is on. I missed that. > > You may now return to your regularly scheduled programming. Well, still it would be good to improve the speed of Devel::Cover, on my box (lowly 500 Mhz, yes, I could upgrade to 2 Ghz AMD or so :) a particulary project takes 33 minutes for one run... not exactly what I call "interactive" :) Best wishes, Tels - -- Signed on Mon Oct 20 18:36:35 2003 with key 0x93B84C15. Visit my photo gallery at http://bloodgate.com/photos/ PGP key on http://bloodgate.com/tels.asc or per email. "Some spammers have this warped idea that their freedom of speech is guaranteed all the way into my hard drive, but it is my firm belief that their rights end at my firewall." -- Nigel Featherston -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.2-rc1-SuSE (GNU/Linux) Comment: When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl. iQEVAwUBP5QRVncLPEOTuEwVAQHdqgf+IMjXV3cgUr9BX2zfXOy2PhgTXINVJPXf cXN0tY1bQjCk62rpdLZl8vziPJy870CqPBjrYGfUhmrYEx98Yqpom4p++tKKNGYV EsfH23A5dUYPUN5SckkFzSkl1n2OqRpM+Gq91CUMCyn14MOXQd4wESLd+g/EJA4m CofxhUtaRYFq7lH42auUenwbJskIib/imX0KHvvsK4krKVX1YLD3A15jPSkDtiTM t1DfYAmt5EeDBfHb/qxFAs2rht3Q9HzSKD7hlLW1lCXnjEFvRvrnL1fXPFWlloX7 ILtotLrcZKFIY9P4+gbFzETlczs65n1HUG0W9gvK19B1hIMWqCbgKg== =Df3t -END PGP SIGNATURE-
When Good Practices Go Bad (was: re: Devel::Cover...)
--- Tony Bowden <[EMAIL PROTECTED]> wrote:
> We insist on a full regression test on checkin, so having the test suite
> take a long time to run puts people off making small changes. :(
>
> On one project a full regression test takes about 8 minutes, which is at
> least an order of magnitude longer than I'd like. I couldn't dream of
> having it take an hour!
Unfortunately, that's something we seem to be stuck with here. At my previous job, we
had one
test suit with a similar number of tests, but it took -- at most -- about 3 minutes to
run. The
problem is that test driven development started here before the current testing
craze. As a
result, my fellow programmers have some curious notions about testing. Amongst other
things, each
file is a package that has several TEST subroutines in it. These tend to have the
following
characteristics:
sub TEST__WE_CAN_ADD_FOO_TO_BAR
{
make_up_test_data();
add_test_data_to_database();
my $foo = function_to_test();
assert($foo, $some_made_up_val);
}
Now, before that function is run, we automatically have a "setup_test" function that
runs. After
that function runs, a "teardown_test" function runs (which typically does a rollback).
So we have to make up a bunch of dummy data, insert this data, test this data, and
roll back this
data FOR EVERY TEST. Further, because of cut-n-paste testing, much of the dummy data
that is
inserted has nothing to do with a particular test and therefore is useless overhead.
Not only
does this slow the test suite to a crawl (and this is with Oracle, no less), I've
asked some of my
fellow programmers if they see a problem with testing made-up data and all I get are
blank stares.
It's similar to why programmers should not do their own acceptance testing: two
minutes later
they send an email saying "yup, it does what I programmed it to do."
To make matters worse, I've also asked "have you ever considered testing the data that
you have in
production?"
The answer, from every programmer I've talked to, has been "how can we test the data
if we don't
know what it is?" The programmes working here haven't worked with any other way of
testing so it
never occurs to them that there might be another way.
This company is doing some great work, but sometimes I want to scream.
Cheers,
Ovid
=
Silence is Evilhttp://users.easystreet.com/ovid/philosophy/indexdecency.htm
Ovid http://www.perlmonks.org/index.pl?node_id=17000
Web Programming with Perl http://users.easystreet.com/ovid/cgi_course/
__
Do you Yahoo!?
The New Yahoo! Shopping - with improved product search
http://shopping.yahoo.com
Re: Devel::Cover and large test suites
> Devel::Cover issues aside, has anyone else looked at interesting ways of > making test suites run faster? One of the drums I beat heavily in my discussion of testing of large projects is that you shouldn't care how long your tests take, so long as they run within the smokebot window. The way we do it is having an hourly cron job that checks out the main CVS project and runs a full blown make test on it. This means that we don't get immediate feedback if a check-in causes a failure, but getting feedback within 59 minutes is close enough to immediate for me. Also, smokebot failures are mailed to the entire department (8 people), so that helps raise the visibility. We've also had to have a 2nd smokebot running, this time hourly on the half hour, to check out and smoke the branch for the parallel project we're running. Also, as an FYI, one of the things I'm going to be adding to Test::Harness soon (Where are those tuits?) is the ability to run the .t files in random order, in the hopes of exciting bugs that are for some reason dependent on order of execution. xoxo, Andy -- Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance
Re: When Good Practices Go Bad (was: re: Devel::Cover...)
At 10:00 AM 10/20/2003 -0700, Ovid wrote:
--- Tony Bowden <[EMAIL PROTECTED]> wrote:
> We insist on a full regression test on checkin, so having the test suite
> take a long time to run puts people off making small changes. :(
>
> On one project a full regression test takes about 8 minutes, which is at
> least an order of magnitude longer than I'd like. I couldn't dream of
> having it take an hour!
Unfortunately, that's something we seem to be stuck with here. At my
previous job, we had one
test suit with a similar number of tests, but it took -- at most -- about
3 minutes to run. The
problem is that test driven development started here before the current
testing craze. As a
result, my fellow programmers have some curious notions about
testing. Amongst other things, each
file is a package that has several TEST subroutines in it. These tend to
have the following
characteristics:
sub TEST__WE_CAN_ADD_FOO_TO_BAR
{
make_up_test_data();
add_test_data_to_database();
my $foo = function_to_test();
assert($foo, $some_made_up_val);
}
Now, before that function is run, we automatically have a "setup_test"
function that runs. After
that function runs, a "teardown_test" function runs (which typically does
a rollback).
So we have to make up a bunch of dummy data, insert this data, test this
data, and roll back this
data FOR EVERY TEST. Further, because of cut-n-paste testing, much of the
dummy data that is
inserted has nothing to do with a particular test and therefore is useless
overhead. Not only
does this slow the test suite to a crawl (and this is with Oracle, no
less), I've asked some of my
fellow programmers if they see a problem with testing made-up data and all
I get are blank stares.
It's similar to why programmers should not do their own acceptance
testing: two minutes later
they send an email saying "yup, it does what I programmed it to do."
To make matters worse, I've also asked "have you ever considered testing
the data that you have in
production?"
The answer, from every programmer I've talked to, has been "how can we
test the data if we don't
know what it is?" The programmes working here haven't worked with any
other way of testing so it
never occurs to them that there might be another way.
This company is doing some great work, but sometimes I want to scream.
Cheers,
Ovid
There is another side to the "made-up" data issue. There are instances
where you legally can _not_ use live data. Consider a Hospital or
Insurance company, for example ...
What we did was take a proper subset of the live data and sanitize it to
remove identifiable things (name, SSN, Medical Number, address, you get the
idea). It took a lot of work to get right; but it did give us a feel for
what real data looked like. We rebuilt the test DB every three months, so
trends in data quality could be factored in.
This meant that running tests did not have to do a setup/tear-down unless
they were exercising Objects not yet in production. It also gave us heads
up on code that was doing excessive searches or otherwise abusing the
database and the User's patience.
B
Bob Goolsby
[EMAIL PROTECTED]
(408) 853-2055
Re: Devel::Cover and large test suites
-BEGIN PGP SIGNED MESSAGE- Moin, On Monday 20 October 2003 19:10, Andy Lester wrote: > > Devel::Cover issues aside, has anyone else looked at interesting ways of > > making test suites run faster? > > One of the drums I beat heavily in my discussion of testing of large > projects is that you shouldn't care how long your tests take, so long as > they run within the smokebot window. > > The way we do it is having an hourly cron job that checks out the main > CVS project and runs a full blown make test on it. This means that we > don't get immediate feedback if a check-in causes a failure, but getting > feedback within 59 minutes is close enough to immediate for me. Also, > smokebot failures are mailed to the entire department (8 people), so > that helps raise the visibility. Good ideas... (but when I start running cronjobs that email me the results of the testsuite I will be officially declared mad. I mean, I am the only developer on the project.. but still, I like the idea *hohum* > We've also had to have a 2nd smokebot running, this time hourly on the > half hour, to check out and smoke the branch for the parallel project > we're running. > > Also, as an FYI, one of the things I'm going to be adding to > Test::Harness soon (Where are those tuits?) is the ability to run the .t > files in random order, in the hopes of exciting bugs that are for some > reason dependent on order of execution. Good idea, except that some testsuites actually depend on the order of the test. I know I have at least written one that did "setup and compile" some stuff in test 00, and then runs the rest, to make sure that it is always run with the latest code - and not manually using the old one.. (Me == bad) Best wishes, Tels - -- Signed on Mon Oct 20 19:30:42 2003 with key 0x93B84C15. Visit my photo gallery at http://bloodgate.com/photos/ PGP key on http://bloodgate.com/tels.asc or per email. "Remember: If the game let's you do it, it's not cheating." -- Xarax -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.2-rc1-SuSE (GNU/Linux) Comment: When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl. iQEVAwUBP5QbwncLPEOTuEwVAQFMiQf9HDAjfd0MSQ6p0no6fVXQTUTJMJxv52Mm oAFcqKLQgOda9+XbK0NLv1O3v9m0YhRgKTl6SKY7YEeDh+PJY/6TiMQEdg6NHG9c meeHJ7857qNoIdNAEEsolBPFjBKnc7v88dUZLxW6HoPhT99mHVnWu9i25HNgBttE zTOCgOjbVQW8MvJIyzZgs5JoCwdXMN54GdsLBreOZDiTStuPUSiVvXDaUCJGic2x C8u6RmBp0VnTQPmeC37JDGm3Y3oY73P+VecL4A8/kZSF/KXwG4Wapd+EV+HYUC2b d4X/8zT12DpTZ2GT91oJIQgnLHhc/U/JVn54m+/bou8jblvSrXLu8Q== =6QCN -END PGP SIGNATURE-
Re: When Good Practices Go Bad (was: re: Devel::Cover...)
On Mon, Oct 20, 2003 at 10:25:17AM -0700, Bob Goolsby (bogoolsb) wrote: > There is another side to the "made-up" data issue. There are instances > where you legally can _not_ use live data. Consider a Hospital or > Insurance company, for example ... In Europe it's pretty much illegal for any company to use live data for testing if you can identify a person from it: http://www.out-law.com/php/page.php?page_id=systemtestingwith1065526767 Tony
Re: Devel::Cover and large test suites
On Mon, Oct 20, 2003 at 12:10:33PM -0500, Andy Lester wrote: > One of the drums I beat heavily in my discussion of testing of large > projects is that you shouldn't care how long your tests take, so long as > they run within the smokebot window. This doesn't work so well with aegis, which has a mentality of making it imposible for anyone to ever break the build... It does lots of nice things related to testing like ensuring that any change that is marked as a bugfix has a new test added that fails with the code that existed before the change and passes with the new code. Tony
Re: Devel::Cover and large test suites
--- Andy Lester <[EMAIL PROTECTED]> wrote: > One of the drums I beat heavily in my discussion of testing of large > projects is that you shouldn't care how long your tests take, so long as > they run within the smokebot window. Sorry Andy, I can't quite agree with you on this one. Technically, you're 100% correct. The reality, though, is that when I'm trying to convince other programmers that they want to run the entire test suite before they check things in, I don't want to see them wince when I say that. The suite that I mentioned that takes over an hour to run is for *one* of our projects. We have many projects. Guess what happens when someone changes code that's shared with all projects? We are talking about many hours of tests. Programmers who "just know" that there change didn't break anything are faced with the prospect of several hours of tests, just to find out that they broke a couple of things, need to go in and fix it and then run several *more* hours of tests. When they made a 15 minute change and are looking at a couple of workdays to get it moved into production, they skip the tests. And yes, we've been bit by this :) The way we're doing tests is simply foolish and we could experience some significant productivity gains if we were to improve the test performance. Of course, I think it's fair to point out that much of the performance is due to a poorly thought out testing strategy. If the tests were designed better, I'd have less room to complain about how long they take to run. Cheers, Ovid = Silence is Evilhttp://users.easystreet.com/ovid/philosophy/indexdecency.htm Ovid http://www.perlmonks.org/index.pl?node_id=17000 Web Programming with Perl http://users.easystreet.com/ovid/cgi_course/ __ Do you Yahoo!? The New Yahoo! Shopping - with improved product search http://shopping.yahoo.com
Re: Devel::Cover and large test suites
On Mon, Oct 20, 2003 at 10:47:02AM -0700, Ovid wrote: > The way we're doing tests is simply foolish and we could experience some significant > productivity > gains if we were to improve the test performance. Of course, I think it's fair to > point out that > much of the performance is due to a poorly thought out testing strategy. If the > tests were > designed better, I'd have less room to complain about how long they take to run. One other interesting idea I had, although I have no idea how practical it is at all really, it to have some way of telling from the Devel::Cover reports (or otherwise) what tests aren't actually adding any value as the stuff they're testing is already adequately tested elsewhere. Now, yes, I know this is pretty much impossible as you need to take into account that they may be testing with different values of data that might be important, but I can't help feeling there's _something_ that could be worked out. Even if all it's doing is giving a pointer to potential areas that could be re-examined... Tony
Re: Devel::Cover and large test suites
Tony Bowden wrote:
>
> One other interesting idea I had, although I have no idea how
> practical it is at all really, it to have some way of telling from the
> Devel::Cover reports (or otherwise) what tests aren't actually adding
> any value as the stuff they're testing is already adequately tested
> elsewhere.
Practically, the only way to do this is to save the results of each test
in a seperate cover_db and then selectively merge them to see whether or
not your coverage changed. Even then, finding the minimal set of tests
that provides "full" coverage is a trial and error, brute force
approach. Actually, I think it's equivilant to the knapsack problem.
>From a testing perspective, this is secondary anyway. There are two
distinct types of testing: functional and structural. Functional testing
proves that the code does what it's supposed to. Structural testing
shows that the code was exercised to some arbitrary level.
What's important to remember is that functional and structural coverage
are *different* things. Having full functional coverage doesn't mean you
have full structural coverage, and vice-versa. e.g. take this code
if ($foo < 10) {
do_a();
#...
do_b();
}
with this test:
$foo = 0;
# test effects of do_a()
$foo = 1;
# test effects of do_a()
You'll get full structural coverage, but that's incidental. You haven't
shown that sub do_b() does what it's supposed to.
I would question whether it's necessary to gather structural coverage
every time the test suite is run. However much you might optimize it,
there will always be a lot of overhead associated with doing that. It
would seem reasonable for the regression tests to be run without
collecting structural coverage for the normal, day-to-day checks. Then,
before each release, rerun the test suite with it to ensure that you've
properly exercised the code.
Further, you could figure out the minimum set of tests that provide full
structural coverage and run just those, but there's a tradeoff. If it
takes longer to figure out which tests to run than it does to run the
tests, it's not worth the effort. Fully automated test suites also tend
to make it not worth the effort. (Who cares if the test suite takes 10
minutes or 10 hours if you let it run overnight?)
If the code has been properly designed (i.e. factored into separate
pieces without lots of ugly interdependancies) it should be possible to
run the module-level[1] test for that piece when it changes, not the full-
out test suite for the entire product. Again, that could be saved for
the release process.
[1] By "module" I don't mean a Perl module, I mean a logical component
of the overall product. That _might_ be a single Perl module, but it
could be none, or ten.
-mjc
No more code coverage
--- Tels <[EMAIL PROTECTED]> wrote:
> Well, still it would be good to improve the speed of Devel::Cover, on my box
> (lowly 500 Mhz, yes, I could upgrade to 2 Ghz AMD or so :) a particulary
> project takes 33 minutes for one run... not exactly what I call "interactive"
> :)
Frankly, I think some major work could be done here. In an attempt to eliminate the
problem I
had, I wrote a quick program that recursed through directories and ran the tests that
it found
there. The core of the loop was something like this:
my $count = 1;
my $total = @test_dirs;
foreach my $dir (@test_dirs) {
printf "\nProcessing %3d out of %3d: $dir\n\n", $count, $total;
chdir $dir;
my $command = 'perl -MDevel::Cover=-db,/home/ovid/cover_db /usr/bin/rtk_test";
system($command);
$count++;
}
There were 92 test directories in @test_dirs. By the time I got to the fourth
directory, I had a
45 megabyte coverage database and the output of top was:
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
22348 cxp 18 0 1133M 1.1G 19280 R99.9 55.5 30:29 perl
Needless to say, I killed the tests. I've never dug into the internals of
Devel::Cover so I don't
know how easy it would be to fix this (and maybe there's something really silly that
I've
overlooked). Unless I get very lucky, I think the code coverage project here is dead.
Cheers,
Ovid
=
Silence is Evilhttp://users.easystreet.com/ovid/philosophy/indexdecency.htm
Ovid http://www.perlmonks.org/index.pl?node_id=17000
Web Programming with Perl http://users.easystreet.com/ovid/cgi_course/
__
Do you Yahoo!?
The New Yahoo! Shopping - with improved product search
http://shopping.yahoo.com
Re: Devel::Cover and large test suites
On Mon, Oct 20, 2003 at 07:02:56PM +, [EMAIL PROTECTED] wrote: > Practically, the only way to do this is to save the results of each test > in a seperate cover_db and then selectively merge them to see whether or > not your coverage changed. Even then, finding the minimal set of tests > that provides "full" coverage is a trial and error, brute force > approach. Actually, I think it's equivilant to the knapsack problem. Yes, but that's not really what I'm looking for. As I said, I know that this is a rather coarse grained approach. I'm not looking for something that can automatically reduce my test suite. I'm looking for something that can raise a large flag and say - "Oi! This lot of tests over here seem to be doing almost exactly the same stuff as this stuff over here. Is that deliberate?" > Fully automated test suites also tend to make it not worth the > effort. (Who cares if the test suite takes 10 minutes or 10 hours if > you let it run overnight?) The two sentences here don't necessarily go together! :) We have a fully automated test suite. It takes too long to run, and we want to run it at each integration - not just overnight. > If the code has been properly designed (i.e. factored into separate > pieces without lots of ugly interdependancies) it should be possible to > run the module-level[1] test for that piece when it changes, not the full- > out test suite for the entire product. Again, that could be saved for > the release process. Surely this is only true if no other module in your system interfaces to this in any way. No matter how well factored the code is something is going to depend on it functioning. And so any changes will need to ensure that the code depending on it still works. You can layer the code such that changes to code at the "top level" don't need to run the tests on "lower" code, but changes to that "lower" code will always need to be checked by re-testing the "higher" level code. Tony
Re: No more code coverage
On Mon, Oct 20, 2003 at 12:24:14PM -0700, Ovid wrote:
> --- Tels <[EMAIL PROTECTED]> wrote:
> > Well, still it would be good to improve the speed of Devel::Cover,
> > on my box (lowly 500 Mhz, yes, I could upgrade to 2 Ghz AMD or so :)
> > a particulary project takes 33 minutes for one run... not exactly
> > what I call "interactive"
> > :)
>
> Frankly, I think some major work could be done here.
I agree. Without more details it is hard to tell, but in my experience
the majority of the resources used on reasonably sized examples are
expended in "database" manipulation, specifically in reading, merging
and writing databases.
I wrote "database" in quotes because currently we are talking about a
flat file, written using Data::Dumper and eval'd in. I have considered
other options - specifically YAML and Storable. I have found YAML to be
even slower and too buggy, and Storable to less reliable. (I never
tracked down specific problems.) This in an area that needs to be
addressed.
> In an attempt to
> eliminate the problem I had, I wrote a quick program that recursed
> through directories and ran the tests that it found there. The core
> of the loop was something like this:
>
> my $count = 1;
> my $total = @test_dirs;
> foreach my $dir (@test_dirs) {
> printf "\nProcessing %3d out of %3d: $dir\n\n", $count, $total;
> chdir $dir;
> my $command = 'perl -MDevel::Cover=-db,/home/ovid/cover_db /usr/bin/rtk_test";
> system($command);
> $count++;
> }
I would suggest that the first thing to do would be to limit coverage
collected to statement cover. Until that's at a reasonable level the
rest of the code coverage metrics are of little use.
> There were 92 test directories in @test_dirs. By the time I got to
> the fourth directory, I had a 45 megabyte coverage database and the
> output of top was:
>
> PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
> 22348 cxp 18 0 1133M 1.1G 19280 R99.9 55.5 30:29 perl
At least you haven't run out of memory yet ;-)
> Needless to say, I killed the tests. I've never dug into the
> internals of Devel::Cover so I don't know how easy it would be to fix
> this (and maybe there's something really silly that I've overlooked).
> Unless I get very lucky, I think the code coverage project here is
> dead.
Try limiting coverage to statement cover and see if that salvages
anything.
Another possibility is to write separate databases, and see if any of
them are exceptionally large, then use the cover program to merge them.
Don't give up _just_ yet. I'd like to try to help you get some value
from Devel::Cover. But do remember, I've never called it anything other
than alpha.
--
Paul Johnson - [EMAIL PROTECTED]
http://www.pjcj.net
Re: No more code coverage
On Mon, Oct 20, 2003 at 10:16:40PM +0200, Paul Johnson wrote: > I wrote "database" in quotes because currently we are talking about a > flat file, written using Data::Dumper and eval'd in. I have considered > other options - specifically YAML and Storable. I have found YAML to be > even slower and too buggy, and Storable to less reliable. (I never > tracked down specific problems.) This in an area that needs to be > addressed. Have you considered SQLite? Tony
Re: Devel::Cover and large test suites
On Mon, Oct 20, 2003 at 08:41:12PM +0100, Tony Bowden wrote: > On Mon, Oct 20, 2003 at 07:02:56PM +, [EMAIL PROTECTED] wrote: > > Practically, the only way to do this is to save the results of each > > test in a seperate cover_db and then selectively merge them to see > > whether or not your coverage changed. Even then, finding the minimal > > set of tests that provides "full" coverage is a trial and error, > > brute force approach. Actually, I think it's equivilant to the > > knapsack problem. But you can get an awful long way with heuristics. First pick the test that provides the greatest coverage. Then pick the test which when merged with previously selected tests provides the greatest coverage. Continue until you reach some limit (possibly no more tests). Skew this somewhat with the time taken to run the test, if desired. In practise this can provide some pretty good results. The coverage can be represented as bitmaps which are or'd together and the whole thing can be fairly efficient, certainly enough to make it worthwhile if your tests are taking an hour to run. This has always been on the TODO list. Or rather, having checked, this has never been on the TODO list, but I've always planned on doing it. > Yes, but that's not really what I'm looking for. > > As I said, I know that this is a rather coarse grained approach. I'm > not looking for something that can automatically reduce my test suite. > I'm looking for something that can raise a large flag and say - "Oi! > This lot of tests over here seem to be doing almost exactly the same > stuff as this stuff over here. Is that deliberate?" Or maybe for a single test in the heuristics above substitute set of tests. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Devel::Cover and large test suites
Tels wrote: test. I know I have at least written one that did "setup and compile" some stuff in test 00, and then runs the rest, Theres something of a nascent convention here; 00_*.t and 99_*.t could be 'reserved' to be 1st and last respectively, with others randomized. begin_* and end_* are reasonable synonyms, but one could take this too far.
Re: Devel::Cover and large test suites
> Theres something of a nascent convention here; > 00_*.t and 99_*.t could be 'reserved' to be 1st and last > respectively, with others randomized. That's pretty much what I was thinking about doing. [0-4]*.t = startup, in that order, and [5-9]*.t = shutdown, in that order. The rest are shuffled. But nothing definite yet. xoa -- Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance
Re: No more code coverage
On Mon, Oct 20, 2003 at 09:34:38PM +0100, Tony Bowden wrote: > On Mon, Oct 20, 2003 at 10:16:40PM +0200, Paul Johnson wrote: > > I wrote "database" in quotes because currently we are talking about a > > flat file, written using Data::Dumper and eval'd in. I have considered > > other options - specifically YAML and Storable. I have found YAML to be > > even slower and too buggy, and Storable to less reliable. (I never > > tracked down specific problems.) This in an area that needs to be > > addressed. > > Have you considered SQLite? Initially I wanted something with few, or better yet no dependencies. I also wanted something that required little or no work when I changed the internal data structures. I'll compromise on both of these, and especially the latter, for something that is efficient and reliable. I'll look into SQLite. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: No more code coverage
--- Paul Johnson <[EMAIL PROTECTED]> wrote: > Initially I wanted something with few, or better yet no dependencies. I > also wanted something that required little or no work when I changed the > internal data structures. Well, SQLite fails the first and *might* fail on the second. On the other hand, it's ridiculously fast, is easy to install and I'd cheerfully vote for it! I'm trying to talk my boss into letting me poke into the internals of Devel::Cover more, but I doubt that will be approved. If it is, maybe I can work on this. Cheers, Ovid = Silence is Evilhttp://users.easystreet.com/ovid/philosophy/indexdecency.htm Ovid http://www.perlmonks.org/index.pl?node_id=17000 Web Programming with Perl http://users.easystreet.com/ovid/cgi_course/ __ Do you Yahoo!? The New Yahoo! Shopping - with improved product search http://shopping.yahoo.com
Re: No more code coverage
On Mon, Oct 20, 2003 at 02:10:51PM -0700, Ovid wrote: > I'm trying to talk my boss into letting me poke into the internals of > Devel::Cover more, but I doubt that will be approved. If it is, maybe > I can work on this. If you or anyone else does get to look at this the code should all be localised to Devel::Cover::DB.pm and, unless you change the internal representation, the changes will be in the 33 lines of code which make up the read() and write() methods. Of course, I'd expect that the change will be increasing that number ;-) -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
Re: Devel::Cover and large test suites
On 10/20/2003 2:41 PM, Tony Bowden wrote: > On Mon, Oct 20, 2003 at 07:02:56PM +, [EMAIL PROTECTED] wrote: >> >> If the code has been properly designed (i.e. factored into separate >> pieces without lots of ugly interdependancies) it should be >> possible to run the module-level test for that piece when it >> changes, not the full- out test suite for the entire product. > > Surely this is only true if no other module in your system interfaces > to this in any way. Okay, perhaps the bias of my background is showing through. Most of my work is in embedded systems (not Perl) where code modules are indeed largely independant. We do module level testing for most changes, and then later (at release time) do a higher-level integration test to ensure that everything works together. Even without that situation, I don't think that you necessarily have to test everything that has an interface to the modified code. For example, the changes being tested could be a new faster algorithm that doesn't (shouldn't) have any externally visible effect. *If* you have a good module-level test suite, you should be able to run just that and verify that all your tests still pass and you haven't changed the observable behavior. Of course, if the change is a bugfix in a library module, I agree with you completely. > You can layer the code such that changes to code at the "top level" > don't need to run the tests on "lower" code, but changes to that > "lower" code will always need to be checked by re-testing the > "higher" level code. I hate that ripple effect. :) -mjc
Re: Taint mode testing and project Phalanx
On Mon, 20 Oct 2003, Andrew Savige wrote: > I noticed in Test::Tutorial: > "Taint mode is a funny thing. It's the globalest of all global features. > Once you turn it on it effects all code in your program and all modules > used (and all the modules they use). If a single piece of code isn't > taint clean, the whole thing explodes. With that in mind, it's very > important to ensure your module works under taint mode." Not to mention that it's buggy as hell. For example, in various versions of Perl I've used there have been rather serious bugs in the regex engine when taint mode is on, even when dealing with untainted variables! I for one think taint mode is rather hopeless until taint mode itself gets better testing in the Perl core. > What are general opinions re taint tests for CPAN modules? > (i.e. a test with #!perl -Tw as the first line of the test). See above. -dave /*=== House Absolute Consulting www.houseabsolute.com ===*/
Re: Taint mode testing and project Phalanx
On Tue, Oct 21, 2003 at 12:24:03AM -0500, Dave Rolsky wrote: > On Mon, 20 Oct 2003, Andrew Savige wrote: > > I noticed in Test::Tutorial: > > "Taint mode is a funny thing. It's the globalest of all global features. > > Once you turn it on it effects all code in your program and all modules > > used (and all the modules they use). If a single piece of code isn't > > taint clean, the whole thing explodes. With that in mind, it's very > > important to ensure your module works under taint mode." > > Not to mention that it's buggy as hell. For example, in various versions > of Perl I've used there have been rather serious bugs in the regex engine > when taint mode is on, even when dealing with untainted variables! I've never hit anything like this. Do you have examples? -- Michael G Schwern[EMAIL PROTECTED] http://www.pobox.com/~schwern/ Loon.
