Tony Bowden wrote:
>
> One other interesting idea I had, although I have no idea how
> practical it is at all really, it to have some way of telling from the
> Devel::Cover reports (or otherwise) what tests aren't actually adding
> any value as the stuff they're testing is already adequately tested
> elsewhere.
Practically, the only way to do this is to save the results of each test
in a seperate cover_db and then selectively merge them to see whether or
not your coverage changed. Even then, finding the minimal set of tests
that provides "full" coverage is a trial and error, brute force
approach. Actually, I think it's equivilant to the knapsack problem.
>From a testing perspective, this is secondary anyway. There are two
distinct types of testing: functional and structural. Functional testing
proves that the code does what it's supposed to. Structural testing
shows that the code was exercised to some arbitrary level.
What's important to remember is that functional and structural coverage
are *different* things. Having full functional coverage doesn't mean you
have full structural coverage, and vice-versa. e.g. take this code
if ($foo < 10) {
do_a();
#...
do_b();
}
with this test:
$foo = 0;
# test effects of do_a()
$foo = 1;
# test effects of do_a()
You'll get full structural coverage, but that's incidental. You haven't
shown that sub do_b() does what it's supposed to.
I would question whether it's necessary to gather structural coverage
every time the test suite is run. However much you might optimize it,
there will always be a lot of overhead associated with doing that. It
would seem reasonable for the regression tests to be run without
collecting structural coverage for the normal, day-to-day checks. Then,
before each release, rerun the test suite with it to ensure that you've
properly exercised the code.
Further, you could figure out the minimum set of tests that provide full
structural coverage and run just those, but there's a tradeoff. If it
takes longer to figure out which tests to run than it does to run the
tests, it's not worth the effort. Fully automated test suites also tend
to make it not worth the effort. (Who cares if the test suite takes 10
minutes or 10 hours if you let it run overnight?)
If the code has been properly designed (i.e. factored into separate
pieces without lots of ugly interdependancies) it should be possible to
run the module-level[1] test for that piece when it changes, not the full-
out test suite for the entire product. Again, that could be saved for
the release process.
[1] By "module" I don't mean a Perl module, I mean a logical component
of the overall product. That _might_ be a single Perl module, but it
could be none, or ten.
-mjc
