Chris Dolan <[EMAIL PROTECTED]> wrote:
> There is a class of tests that module authors perform that end users
> are not expected to run. For example code coverage tests, spelling
> tests, coding style tests, etc. These tests are either prohibitively
> expensive or complicated or unpredictable for end users to run. I
> call these "private tests" or author tests.
I really like this idea. But as you pointed out, it's not just
authors that need to worry about running these tests, it's packagers
(ppm/deb/etc), automated testers (cpants/testers.cpan.org/etc), and hackers.
I'd suggest we call these "exhaustive" tests.
Spelling tests, code coverage tests, testing interaction with
CPAN.pm, etc. are things that I think should be done by automated testers,
and by people packaging your module for distribution inside another larger
system (eg, PPMs or debian packages). So maybe it would make sense if there
were certain tests that only run if AUTOMATED_TESTING is set, and/or if the
"disttest" action is being run.
That said, it would be easy to whip together a Test:: module that
only allows these tests to run under these conditions, especially if
Module::Build, say, set a "DIST_TEST" environment variable when the disttest
action was being run. I already check for AUTOMATED_TESTING to manipulate my
test environments to be more exhaustive. Both Module::Build, and perhaps
future versions of ExtUtils::MakeMaker (if there will ever be a future
version) could depend on this Test:: module (whether or not they use it
themselves) so that it's already on everybody's system. It'd be nice and
small so I don't see anybody complaining too loudly.
Maybe we could call it "Test::Exhaustive"? It's mandate being to
SKIP_ALL of a test if you havent somehow (AUTOMATED_TESTING, disttest, etc)
made it clear that you wish to exhaustively test this module?
> Roughly, I propose that we adopt a standard file extension for author
> tests, like "t/*.ta", and add an ACTION_authortest to M::B that runs
> both t/*.t and t/*.ta (which ones first??)
If they were named differently (i don't think they have to be), I'd
want to see them run in lexical order like the current tests, so it'd go
"01foo.t", "02bar.ta", "03baz.t".
Cheers,
Tyler
