I debated whether or not to post this here for a long time, because I
gather that deferred plans are somewhat of a hot topic on this group.
But finally I decided that I just needed to understand some history
and make sure there's nothing I'm missing here.
First, if you'll bear with me, a little background: When I first
discovered test-first programming, via the original XP book, I thought
it was brilliant, and I started employing it on many of my projects.
However, I confess that (somewhat foolishly) I did not use Test::More
and friends to do my tests. Instead, I rolled my own system. Since I
have never (yet) released any of my code to CPAN, using Test::More
(and friends) was not already a habit for me. And I thought to myself
that the testing process for installation of a module was quite
different from the testing process for developing code. Now that I'm
considering finally getting off my lazy ass and putting some stuff out
to CPAN, I realized that I had to deal with installation testing, and
I started reading everything I could about the testing frameworks of
standard Perl, TAP, etc. My conclusions are that I still think that
installation testing is a _little_ different from development testing,
but certainly not enough to justify having my own little world of
testing libraries, most of which do exactly what Test::More et al do,
only without the benefit of nearly as much aggregated programmer time
put into making sure they're correct. So I set out to convert my
stuff to the standard stuff. And I only hit two real snags.
The first is that _during development_, I want to stop my tests on the
first failure. This is the minor difference, IMHO, between
installation testing and development testing. For installation, it
obviously makes no sense to do this. But for development testing, it
makes sense to me. (I understand that others may not feel so, but
I've also noticed that some on this list agree.) When I first began
this effort, there wasn't really a solution for me on CPAN, but now
that Ovid has released the excellent Test::Most, that particular itch
is scratched, so I don't really want to open that can of worms again.
The second sticking point is the concept of plans, and here is where I
really want to understand the reasoning behind it. My understanding
is that a plan has one primary function: to insure that if a test
script dies before all the tests are run, that is recognized as an
error. Now, in my own test scripts, I achieved this via a very simple
methodology: all my scripts ended with a simple print statement: "ALL
DONE". I ran the script, and if I saw my ALL DONE at the end, the
script was fine. If I didn't, I obviously had a problem. Very
simple, very clean. My implementation was a bit simplistic, perhaps,
but no matter how much I've read about testing in Perl, I still can't
see why the idea itself isn't sound.
This idea is similar to what Ovid has been calling "deferred plans",
except that he has been suggesting that the _number_ of tests be
submitted to the test harness at the end, whereas I'm just talking
about a success flag. Now, in the original discussion back in
November, Adrian Howard asked:
> I don't get this. Why is saying "I know this test script outputs test
> results" at the start better than saying it at the end?
And Aristotle answered:
> So you'd defer the plan in cases where the number of tests is
> predetermined but maybe hard to precompute, or where it's
> variable. So in both cases you are calculating the number at run
> time, which is immediately subject to more bugs than providing a
> constant.
>
> Additionally, it's more likely for a bug in the calculation to
> line up with a bug in the corresponding test code, so that you
> end up with a plan that matches the number of tests run even
> though you *intended* to run fewer/more tests.
>
> And lastly, even a runtime-calculated predeclared plan separates
> the test code and calculation code at least in time (while
> running) and probably also in space (in the source code).
> Therefore it seems to me that bugs are somewhat less likely
> to line up.
Now, I have a great respect for Aristotle (actually, I have a great
respect for all the people on this list with whose work I have any
previous experience), but it seems to me that all these objections
relate to presenting a number at the end of the tests. My method
inherently avoids those issues.
And I don't want to defer the plan because the number of tests is hard
to predetermine, or because it's indeterminate. I just think that
having code where every time you change things in one place, you have
to make sure you change something somewhere else isn't a good thing.
In any other programming scenario, I think most everyone would agree
with me. But when it comes to testing, doing this in terms of tests
is not only okay, it's considered best practice.
So I guess I'm looking for enlightenment. I can't help but feel like
there's something I'm missing. If the only function of a test plan is
to make sure all the tests are run, my method handles that, and it's
simpler and requires less bookkeeping. And if there are other
functions of the test plan, I'm just not seeing what they are. But
I'm not so arrogant as to believe that no one's ever considered my
method before either. So I'm hoping that someone here can show me the
light.
Not criticizing, not claiming my method is better, just looking for
any reasons why this wouldn't work. And, JIC there's some agreement
that it _would_ work, I've already put together a patch for Test::Most
that does it. That is, at the top of your script, you put this:
use Test::Most 'defer_plan';
and at the bottom of your script, you put this:
all_done();
and it just DTRT. The implementation is perhaps not as clean as I'd
like it to be, but it's not a horrific hack either. I'm going to
forward it to Ovid through normal CPAN channels.
Thoughts?
-- Buddy
