Re: Fixing the damage caused by has_test_pod
* David Golden [EMAIL PROTECTED] [2007-07-31 13:20]: On 7/31/07, Eric Wilhelm [EMAIL PROTECTED] wrote: * Test::Pod::Coverage Why does that matter? Does it care where its .t file lives? The synopsis suggests t/pod-coverage.t. Wow, now that is *incredible* inertia right there. Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: Fixing the damage caused by has_test_pod
* David Cantrell [EMAIL PROTECTED] [2007-07-31 13:20]: A. Pagaltzis wrote: Disagree. The tarball should contain the full source necessary to recreate everything the author did. Stuff that’s not relevant to end-user installation of the module should simply be kept out of the way. I ever-so-respectfully disagree. I am not going to include the original data that I downloaded from the UK telecoms regulator for a particular build of Number::Phone::UK::Data (although I do include a shell script that downloads the current version of that data), nor the list of exchanges for Number::Phone::UK::DetailedLocations. Bear in mind that these two modules are already 2.5MB and 7.5MB *compressed*, 18 and 36MB uncompressed. I said “the tarball should contain everything necessary to recreate everything the author did”, to which you respond “I include a shell script for downloading the un-munged data I used to build the module.” Can you explain to me where the disagreement is? :-) (Yes, I know there’s technically a difference. But what I *meant* is that someone else should be able to losslessly take over or fork the module, and what you’re doing is perfectly in line with that.) Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: Fixing the damage caused by has_test_pod
* Andy Lester [EMAIL PROTECTED] [2007-07-30 22:30]: On Jul 30, 2007, at 2:46 PM, A. Pagaltzis wrote: Certainly, the intent is to have boilerplate.t NOT get shipped with the module. Disagree. The tarball should contain the full source necessary to recreate everything the author did. Stuff that’s not relevant to end-user installation of the module should simply be kept out of the way. You can't disagree. It's a statement of fact that RJBS and I did not intend for boilerplate.t to actually get shipped with the module. I can’t disagree with the fact that this is what you intended, but I can and do disagree with what you intended. If the original author feels the boilerplate test is no longer necessary, then not shipping the file is an automatic consequence of deleting the file from his own source tree. If the original author feels the boilerplate test might be useful again in the future, and hence does not delete it from his own source tree, then it should ship with the distro as an author test. There is no good reason to special-case the boilerplate test. Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: Fixing the damage caused by has_test_pod
Eric Wilhelm wrote: # from David Cantrell Let us assume that I write a module that, if you have a particular module installed will do some extra stuff. This is very common. ... Skipping tests because you correctly identify that the optional module isn't available is, of course, counted as passing. Test::Pod is *not* optional to PERL_AUTHOR_TESTING. If your intent is to test the pod (here, I am taking PERL_AUTHOR_TESTING to imply that we're trying to prevent bad pod), you must have the module that tests it. That is, to test the pod, you have to test the pod. To put it another way, the pod is not tested unless the pod tests have been run. If the pod tests didn't get run, the pod hasn't been tested. A pod test which skips due to 'no Test::Pod' has not tested the pod. To test the pod, you must run the pod tests. Seeing that you obviously think I'm an idiot, there's probably not much point continuing. -- David Cantrell
Re: Fixing the damage caused by has_test_pod
Eric Wilhelm wrote: # from David Cantrell No. If AUTHOR_TESTING, fail miserably unless the pod and coverage both 1. gets tested and 2. passes. That means the Test::Pod::* module in question must load. Wrong. If AUTHOR_TESTING then surely all tests *must* pass both with and *without* the optional module! What good does it do to skip the test if AUTHOR_TESTING? That just gets us back into the same situation as the current one where the author didn't have Test::Pod and pod which fails the tests got shipped. Let us assume that I write a module that, if you have a particular module installed will do some extra stuff. This is very common. Before uploading my module to the CPAN I need to make sure that it passes its tests both with and without the optional module. In fact, ideally I need to test with all possible combinations of optional modules including none of them. It makes no difference whether the optional module is Test::Pod or DateTime. Skipping tests because you correctly identify that the optional module isn't available is, of course, counted as passing. -- David Cantrell
Re: Fixing the damage caused by has_test_pod
Adam Kennedy wrote: Salve J Nilsen wrote: Anyhow, I think being able to recreate the entire distribiution may be a good thing, but perhaps not necessarily from a CPAN-distributed tarball. I'd certainly expect that from a source repository checkout, though. CPAN-distributed tarballs are the primary source of the source code, any tests that the author feels should be run as part of release testing or automated testing should be included in the CPAN distribution. Yes, of course. Although with recreate the entire distribution, I should perhaps have also said including any custom steps to generate modules from third-party sources I was thinking of modules that have some generated parts, where the datasources for those parts aren't publically available for some reason. (Not exactly the common case, obviously.) I would have liked to use Geo::Postcodes::NO as an example, but I see Arne already has included his tools for automatically updating the module postcode index. :-P Repositories have this nasty habit of being private, out of date, on servers that die, not a version control you like, not a product you can legally use, or are not used at all by the author. Yeah, not much one can do other than relate to the CPAN version then. But if at all possible, it's still a better practice to try to work on the repository version. If at all possible. :) - Salve
Re: Fixing the damage caused by has_test_pod
On 7/31/07, Eric Wilhelm [EMAIL PROTECTED] wrote: * Test::Pod::Coverage Why does that matter? Does it care where its .t file lives? The synopsis suggests t/pod-coverage.t. David
Re: Fixing the damage caused by has_test_pod
A. Pagaltzis wrote: Disagree. The tarball should contain the full source necessary to recreate everything the author did. Stuff that’s not relevant to end-user installation of the module should simply be kept out of the way. I ever-so-respectfully disagree. I am not going to include the original data that I downloaded from the UK telecoms regulator for a particular build of Number::Phone::UK::Data (although I do include a shell script that downloads the current version of that data), nor the list of exchanges for Number::Phone::UK::DetailedLocations. Bear in mind that these two modules are already 2.5MB and 7.5MB *compressed*, 18 and 36MB uncompressed. -- David Cantrell
Re: Fixing the damage caused by has_test_pod
Eric Wilhelm wrote:
# from David Golden
# pod-coverage.t
use Test::More;
plan skip_all = Skipping author tests if not $ENV{AUTHOR_TESTING};
...
No. If AUTHOR_TESTING, fail miserably unless the pod and coverage both
1. gets tested and 2. passes. That means the Test::Pod::* module in
question must load.
Wrong. If AUTHOR_TESTING then surely all tests *must* pass both with
and *without* the optional module!
I'm still stuck on thinking up a decent name for this:
http://drhyde.cvs.sourceforge.net/drhyde/perlmodules/No/lib/No.pm?revision=1.2view=markup
which will make those sort of tests much easier. Yes, I do know that
there are other modules that do some of what this does. But only some
of it. Also before uploading it to the CPAN I really need a volunteer
to sanity-check it on VMS.
--
David Cantrell
Re: Fixing the damage caused by has_test_pod
On 31 Jul 2007, at 12:19, David Cantrell wrote: A. Pagaltzis wrote: Disagree. The tarball should contain the full source necessary to recreate everything the author did. Stuff that’s not relevant to end-user installation of the module should simply be kept out of the way. I ever-so-respectfully disagree. I am not going to include the original data that I downloaded from the UK telecoms regulator for a particular build of Number::Phone::UK::Data (although I do include a shell script that downloads the current version of that data), nor the list of exchanges for Number::Phone::UK::DetailedLocations. Bear in mind that these two modules are already 2.5MB and 7.5MB *compressed*, 18 and 36MB uncompressed. That's something of a special case, no? As you say there's a shell script to download the data anyway - so no need to distribute it. -- Andy Armstrong, hexten.net
Re: Fixing the damage caused by has_test_pod
On 31 Jul 2007, at 12:27, David Cantrell wrote: Wrong. If AUTHOR_TESTING then surely all tests *must* pass both with and *without* the optional module! I'm still stuck on thinking up a decent name for this: http://drhyde.cvs.sourceforge.net/drhyde/perlmodules/No/lib/No.pm? revision=1.2view=markup As per my post of a few seconds ago, how about making it a pragma: 'unavailable'? :) -- Andy Armstrong, hexten.net
Re: Fixing the damage caused by has_test_pod
On 31 Jul 2007, at 12:50, Steffen Mueller wrote: I'm still stuck on thinking up a decent name for this: http://drhyde.cvs.sourceforge.net/drhyde/perlmodules/No/lib/No.pm? revision=1.2view=markup As per my post of a few seconds ago, how about making it a pragma: 'unavailable'? :) Does this do what you want? http://search.cpan.org/~corion/Test-Without-Module-0.09/ Looks like it, thanks :) -- Andy Armstrong, hexten.net
Re: Fixing the damage caused by has_test_pod
A. Pagaltzis wrote: * Andy Lester [EMAIL PROTECTED] [2007-07-30 21:41]: On Jul 30, 2007, at 1:58 PM, Eric Wilhelm wrote: Thus, I posit that the quality of the module is generally lower if 'boilerplate.t', 'pod-coverage.t', and 'pod.t' *exist*. Certainly, the intent is to have boilerplate.t NOT get shipped with the module. Disagree. The tarball should contain the full source necessary to recreate everything the author did. Stuff that’s not relevant to end-user installation of the module should simply be kept out of the way. boilerplate.t is obviously a very special case, only useful when creating new files in a distribution, in order to warn forgetful programmers to update these with something less boilerplate-ish. :) Anyhow, I think being able to recreate the entire distribiution may be a good thing, but perhaps not necessarily from a CPAN-distributed tarball. I'd certainly expect that from a source repository checkout, though. - Salve
Re: Fixing the damage caused by has_test_pod
Hi Andy, Andy Armstrong schrieb: On 31 Jul 2007, at 12:27, David Cantrell wrote: Wrong. If AUTHOR_TESTING then surely all tests *must* pass both with and *without* the optional module! I'm still stuck on thinking up a decent name for this: http://drhyde.cvs.sourceforge.net/drhyde/perlmodules/No/lib/No.pm?revision=1.2view=markup As per my post of a few seconds ago, how about making it a pragma: 'unavailable'? :) Does this do what you want? http://search.cpan.org/~corion/Test-Without-Module-0.09/ Steffen
Re: Fixing the damage caused by has_test_pod
Steffen Mueller wrote: Does this do what you want? http://search.cpan.org/~corion/Test-Without-Module-0.09/ Doesn't do what I want, as I want to be able to infect any perl processes that might get exec()ed by the one from which I've hidden Some::Module. -- David Cantrell
Re: Fixing the damage caused by has_test_pod
Adriano Ferreira wrote: PERL5LIB=MDevel::Hide=Module::ToHide process_that_fork_other_processes.pl works as far as I know. PERL5OPT. Yes, that will do the job, but I want it to be built in to the module-hiding module rather than something extra that the user needs to do himself. Would you accept a patch to make that an option? -- David Cantrell
Re: Fixing the damage caused by has_test_pod
On 7/31/07, David Cantrell [EMAIL PROTECTED] wrote: Steffen Mueller wrote: Does this do what you want? http://search.cpan.org/~corion/Test-Without-Module-0.09/ Doesn't do what I want, as I want to be able to infect any perl processes that might get exec()ed by the one from which I've hidden Some::Module. PERL5LIB=MDevel::Hide=Module::ToHide process_that_fork_other_processes.pl works as far as I know. -- David Cantrell
Re: Fixing the damage caused by has_test_pod
David Golden wrote: CPAN is a core strength of Perl because it makes developer's lives easier by not reinventing the wheel. This is not necesarily the case. Every language has ways of providing additional libraries that make developers lives easier by not reinventing the wheel. CPAN is a core strength of Perl for the same reason .deb packages are a core strength of debian. Not only do they fulfil the not reinventing the wheel role, but they do so in a way that makes the procurement and installation of those package dramatically easier than the alternatives. And the ease of procurement of a piece of software is twice as important to success than the quality of the piece of software. The person installing something from CPAN has ONLY two reasonable outcomes. 1. The module or application, and all the dependencies, install correctly. 2. The installation process fails, and the end-user cannot use that software. Additionally, dealing with recursion-based issues is paramount. One thing I keep repeating over and over is that for MOST cases, nobody will ever use the modules they are installing, nobody understands what they do, and nobody will ever read the documentation. To use two examples here, Jifty developers install 50-100 packages that are Jifty dependencies, and primary use exactly two. Jifty and Jifty::DBI. That's a couple of percent at best. For a more extreme example, JSAN users often come from the Ruby of Python worlds. They aren't even Perl coders AT ALL. They just want to install the JSAN client. It is installation that has made CPAN successful, and the prioritisation of installation over all other use cases is paramount. More usage of modules means more installs and points of dependency failure. Modules which use lots of dependencies that might fail to install will themselves fail to be installed more often and may be passed over and the world goes on. So let's not start a dependencies are bad bandwagon -- let the market figure out what dependencies are worth the risk. A market is a bad analogy for CPAN. Markets require choice, and arbitrary layers of recursive dependencies makes for very little choice. In my view, testing modules that make modules faster to develop and or lead to better diagnostic output when failures do happen can make author's lives easier, which leads to more development, which is a good thing. If that means some extra installs, well, OK. Balance the benefits to you against the likelihood that some people won't put up with the extra dependencies. Benefit: Improved quality of error reports for factors that do not impact the functional operation of a module. Cost: A percentage of your users can cannot use the module at all, give up in disgust and avoid it in the future. I see this balance as undermining the core functionality of CPAN for the benefit of the author in catching minor nits. It escalates a minor non-critical issue for an author into a critical issue for installers. It's selfish and wrong. Dealing with errors in non-functional areas is what we have CPAN Testers for. Functionality and installation always comes first. Adam K
Re: Fixing the damage caused by has_test_pod
chromatic wrote: On Monday 30 July 2007 10:01:25 Eric Wilhelm wrote: I think the important bit is that `make test` only runs tests which verify the module's functionality. ... and if my POD coverage somehow mysteriously changed between the time I bundled a new dist and some user downloaded that bundle from the CPAN, there are bigger problems with that downloaded dist than failing POD coverage tests. Or alternatively, there might have been a new release of Test::Pod with a slight change in behaviour that causes failures. Adam K
Re: Fixing the damage caused by has_test_pod
Salve J Nilsen wrote: Anyhow, I think being able to recreate the entire distribiution may be a good thing, but perhaps not necessarily from a CPAN-distributed tarball. I'd certainly expect that from a source repository checkout, though. CPAN-distributed tarballs are the primary source of the source code, any tests that the author feels should be run as part of release testing or automated testing should be included in the CPAN distribution. Repositories have this nasty habit of being private, out of date, on servers that die, not a version control you like, not a product you can legally use, or are not used at all by the author. From the point of view of continuity, only the CPAN exists. There are repositories. Adam K
Re: Fixing the damage caused by has_test_pod
Eric Wilhelm wrote: I'm thinking the installer should have the option to scan the tests and skip any that use qr/Test::Pod(?:::Coverage)?/. This wouldn't work. It would just mean that the installer would skip ALL the tests for Test::Pod :) There's many cases when testing the Test:: modules you want to run tests that you might otherwise want to skip for regular modules. Adam K
Re: Fixing the damage caused by has_test_pod
Eric Wilhelm wrote: # from Adam Kennedy Thus, I would like to propose the following. 1. That the running of POD-related tests by end users is considered harmful. +20 -40 I don't see that it's harmful at all. It's not even particularly harmful (merely annoying) for an author to add Test::Pod etc as pre-requisites. And if you have a religious objection to installing them, you can always run the rest of the tests for a module, see that it passes all the functional tests, and force install it. 2. That the test_pod and test_pod_coverage tests by modified ... I propose that they be deleted. The build system There is no the build system. contains all of the needed info to accomplish the same thing. ./Build testpod ./Build testpodcoverage $ make testpod make: *** No rule to make target `testpod'. Stop. Running a kwalitee check is also part of the pre-release process, but I omit these 'has a file $foo' metrics because I believe they *hinder* quality. I wouldn't go that far, but I don't think they help much. -- David Cantrell
Re: Fixing the damage caused by has_test_pod
--- David Cantrell [EMAIL PROTECTED] wrote: Eric Wilhelm wrote: # from Adam Kennedy Thus, I would like to propose the following. 1. That the running of POD-related tests by end users is considered harmful. +20 -40 I don't see that it's harmful at all. It's not even particularly harmful (merely annoying) for an author to add Test::Pod etc as pre-requisites. And if you have a religious objection to installing them, you can always run the rest of the tests for a module, see that it passes all the functional tests, and force install it. Nope. Adam's right. Anything which causes the build to fail for unnecessary reasons is bad. Very bad. Every time we say oh, we'll just allow one more tiny thing, we soon get a steaming pile of tiny things (core modules shipped with perl are exhibit A). I've been guilty of missing this point in the past and I'd like to avoid it in the future. As a result, I've often forced users to install Test::Differences when really Test::More::is_deeply is a suitable fallback. Test::Differences merely gives me, the author, better diagnostics. What it gives the end user is another point of failure. Multiply those points of failure by the number of end users and the number of systems they might install the code on and you begin to get an idea of the magnitude of the problem. Tests should *only* fail when there is a clear, unequivocal reason to believe that the code will not function appropriately on someone's machine. Having '=head0' or a '=back' without '=over' should not be such a failure. It's taken a lot of grief while working on TAP::Parser to realize how terribly wrong I was about this in the past and it's a mistake I would like to better understand and rectify in the future. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/
Re: Fixing the damage caused by has_test_pod
* David Cantrell [EMAIL PROTECTED] [2007-07-30 12:04]: Eric Wilhelm wrote: # from Adam Kennedy Thus, I would like to propose the following. 1. That the running of POD-related tests by end users is considered harmful. +20 -40 I don't see that it's harmful at all. It's not even particularly harmful (merely annoying) for an author to add Test::Pod etc as pre-requisites. And if you have a religious objection to installing them, you can always run the rest of the tests for a module, see that it passes all the functional tests, and force install it. Sysadmins’ hairs grow grey over stuff like this. Hapless users are completely overwhelmed with it. Tests should NEVER EVER fail without a really good reason. The only ones who can deal with it relatively well are the Perl developers who themselves write modules and understand the mechanics well enough to be able to make a sound call. Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: Fixing the damage caused by has_test_pod
--- demerphq [EMAIL PROTECTED] wrote: On 7/30/07, Ovid [EMAIL PROTECTED] wrote: Tests should *only* fail when there is a clear, unequivocal reason to believe that the code will not function appropriately on someone's machine. Having '=head0' or a '=back' without '=over' should not be such a failure. It's taken a lot of grief while working on TAP::Parser to realize how terribly wrong I was about this in the past and it's a mistake I would like to better understand and rectify in the future. Thats a very interesting comment that Id love to hear more about. Can you expand on your experiences with TAP::Parser in this respect? It's not TAP::Parser *per se* which has led to this. It's the creation of TAP::Parser and learning more about how toolchains work. For example, I thought that bonus tests (TODO tests which unexpectedly succeed) should really be a failure. However, that would break the CPAN toolchain dramatically. I have no way of quantifying things, so let's play proctologist with a flashlight and guess that there are maybe 100,000 people who regularly use cpan or cpanplus. Now let's say that I release a Super::WunderModule that fully 5% of those users want to install and on an average of two boxes apiece. In Super::WunderModule, I have dependencies on Test::Class, Test::Exception, and PadWalker. Let's assume that only PadWalker is needed for the *functionality*, so we can safely say that a failure to install that is really a problem. So let's consider Test::Exception. It has a dependency on 5.6.1, but we'll pretend the code does, too. That's OK if you really need that. We also have dependencies on Sub::Uplevel (a notorious source of build problems), Test::Builder::Tester and we'll just pretend that the other modules are already installed. From Test::Class, we'll just assume that only Devel::Symdump was not installed. Without continuing to follow dependencies, we have four completely optional modules to install, so our unecessary points of failure can be: $number_of_users * $percent_who_install * $average_number_of_boxes_installed_on * $modules_to_install So for this trivial example, we have about 40,000 potentially unneeded installs and possible failures. Of course, that's a rough and stupid calculation and those numbers have NO MEANING, but it shows how quickly this could balloon. Now try and calculate this for all CPAN modules. You can see why false negatives are a fact of life and anything which avoids this would be wonderful. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/
Re: Fixing the damage caused by has_test_pod
On 7/30/07, Ovid [EMAIL PROTECTED] wrote:
We also have dependencies on Sub::Uplevel (a notorious source of build
problems),
Excuse me? CPAN::Testers for Sub::Uplevel 0.14: PASS 165 FAIL 0
There was a time when Test::Exception had things in build_requires
that should have been in requires. Test::Exception also exposed bugs
in nesting uplevel() calls, but this only affected modules that used
Sub::Uplevel and tested code with Test::Exception as well. In those
cases, Sub::Uplevel wasn't just a build requirement anyway and test
failure were quite helpful in finding and fixing the upstream bug.
So for this trivial example, we have about 40,000 potentially unneeded
installs and possible failures.
Of course, that's a rough and stupid calculation and those numbers have
NO MEANING, but it shows how quickly this could balloon. Now try and
calculate this for all CPAN modules. You can see why false negatives
are a fact of life and anything which avoids this would be wonderful.
sarcasmAnd if usage of Perl doubled, we'd have 80,000 unneeded
installs! So clearly the answer is to reduce the number of people
using Perl to reduce the number of unneeded installs./sarcasm
So, yes, that's a rough and stupid calculation.
CPAN is a core strength of Perl because it makes developer's lives
easier by not reinventing the wheel. More usage of modules means more
installs and points of dependency failure. Modules which use lots of
dependencies that might fail to install will themselves fail to be
installed more often and may be passed over and the world goes on. So
let's not start a dependencies are bad bandwagon -- let the market
figure out what dependencies are worth the risk. (I.e. your chance of
install failure is increased by the install failure chance of all your
upstream dependencies)
In my view, testing modules that make modules faster to develop and or
lead to better diagnostic output when failures do happen can make
author's lives easier, which leads to more development, which is a
good thing. If that means some extra installs, well, OK. Balance the
benefits to you against the likelihood that some people won't put up
with the extra dependencies.
The issue at hand is really *release* testing (i.e. did I bump the
version, did I test my Pod, do I use good style, etc.) being mixed
with *functional* testing -- and the corresponding push for release
testing modules being included as requirements.
For that, I blame -- among other things -- Module::Starter for
including pod.t and pod-coverage.t and with a default setting to run
tests. Better would have been to skip tests unless
$ENV{AUTHOR_TESTING} or some similar flag was set.
Yes, there's good defensive code to make sure Test::Pod and
T::P::Coverage are installed, but that just encourages people just
putting them in their prereqs as it's easy to imply that they should
be run if at all possible.
(There's also a tricky problem with T::P::C being happy with any
version of P::C, too: http://perlmonks.org/index.pl?node_id=508109 )
If some author wants to bear the risk of forcing them as a dependency,
fine. But I do think that the community shouldn't be actively
encouraging that through Kwality, Module::Starter and so on.
Regards,
David
Re: Fixing the damage caused by has_test_pod
--- David Golden [EMAIL PROTECTED] wrote: On 7/30/07, Ovid [EMAIL PROTECTED] wrote: We also have dependencies on Sub::Uplevel (a notorious source of build problems), Excuse me? CPAN::Testers for Sub::Uplevel 0.14: PASS 165 FAIL 0 There was a time when Test::Exception had things in build_requires that should have been in requires snip Yes, you're right. That was a unfair of me. I was rushing off to the pub and wanted to get the email out quickly. I shouldn't have cast aspersions on a great module. The Test::Exception/Sub::Uplevel problem causing false negatives was always a bit frustrating for me and it came out in the last email. I'm sorry if I offended you :( So, yes, that's a rough and stupid calculation. CPAN is a core strength of Perl because it makes developer's lives easier by not reinventing the wheel. More usage of modules means more installs and points of dependency failure. The calculation was not intended to be a serious attempt to put forward numbers. It was a serious attempt to show how minor, avoidable failures can quickly become a major headache. So let's not start a dependencies are bad bandwagon Dependences are not bad and I never said that. Dependencies which cause false negatives are bad. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/
Re: Fixing the damage caused by has_test_pod
On 7/30/07, Ovid [EMAIL PROTECTED] wrote: aspersions on a great module. The Test::Exception/Sub::Uplevel problem causing false negatives was always a bit frustrating for me and it came out in the last email. I'm sorry if I offended you :( Apology accepted. It frustrated me so much I wound up adopting Sub::Uplevel, so I definitely get where you're coming from. Dependences are not bad and I never said that. Dependencies which cause false negatives are bad. That's pretty much the whole toolchain, then, and all the major testing modules. Where do you draw the line? As I suggested, functional test module dependencies are OK, but release test module dependencies are not. In general, in managing dependencies, I think you've got (at least) two choices for anything non-standard. * List it as a dependency, and suffer any upstream test failures that prevent installation * Bundle it in t/inc (as many toolchain modules seem to do) and suffer the risk that the version you bundled contains a bug I get the sense that many application deployments follow the second approach, prefering the evil they know to the evil they don't. David
Re: Fixing the damage caused by has_test_pod
# from David Golden
# on Monday 30 July 2007 05:34 am:
The issue at hand is really *release* testing (i.e. did I bump the
version, did I test my Pod, do I use good style, etc.) being mixed
with *functional* testing -- and the corresponding push for release
testing modules being included as requirements.
Yes. We want to know that the author tested the code and went through
the checklist before the release. We would like to be able to verify
it ourselves, but we don't want all of that to get in the way of
installation.
For that, I blame -- among other things -- Module::Starter for
including pod.t and pod-coverage.t and with a default setting to run
tests. Better would have been to skip tests unless
$ENV{AUTHOR_TESTING} or some similar flag was set.
Again, yes. Though I'm going to stick with delete them.
I think the important bit is that `make test` only runs tests which
verify the module's functionality.
Anything else needs to be a separate test target or out-of-band tool.
The fact that this is ugly, opaque, and error-prone should tell us
something:
AUTHOR_TESTING=1 \
TEST_THE_NETWORK=1 \
TEST_THE_HARDWARE=1 \
ADD_THOSE_OTHER_TESTS=1 \
TEST_EVERYTHING_REALLY_I_MEAN_IT=1 \
make test
Compare to something as simple as:
prove t_author t_network t_hardware t_other t_more t
Or
./Build testall
--Eric
--
Don't worry about what anybody else is going to do. The best way to
predict the future is to invent it.
--Alan Kay
---
http://scratchcomputing.com
---
Re: Fixing the damage caused by has_test_pod
On Monday 30 July 2007 10:01:25 Eric Wilhelm wrote: I think the important bit is that `make test` only runs tests which verify the module's functionality. ... and if my POD coverage somehow mysteriously changed between the time I bundled a new dist and some user downloaded that bundle from the CPAN, there are bigger problems with that downloaded dist than failing POD coverage tests. -- c
Re: Fixing the damage caused by has_test_pod
* Christopher H. Laco [EMAIL PROTECTED] [2007-07-29 17:05]: A failing pod test is the first sign pod2* generation is broken. It's rare, but I've been in the situation where for some reason (mistmatching dist requirements, failed upgrades, bad dist packages, broken troff, etc) the creation of man pages/html from the dist pod fails outright. Pod tests assure the user that they will be able to perldoc/man the thing they are installing. That's doubly true if you're using pod2usage to expose help in scripts. Is that actually true? Test::Pod uses Pod::Simple, but that’s far from the only game in POD town. Even the perldoc utility included with versions of perl that ship with Pod::Simple uses its own formatter by default. And Pod::Simple has not always shipped with perl. So if Pod::Test says you’re fine, that doesn’t necessarily mean things will work for the user (though if it says you’re broken then things are likely to break for the user as well, since Pod::Simple is very solid). Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: Fixing the damage caused by has_test_pod
Eric Wilhelm wrote:
# from David Golden
# on Monday 30 July 2007 05:34 am:
The issue at hand is really *release* testing (i.e. did I bump the
version, did I test my Pod, do I use good style, etc.) being mixed
with *functional* testing -- and the corresponding push for release
testing modules being included as requirements.
Yes. We want to know that the author tested the code and went through
the checklist before the release. We would like to be able to verify
it ourselves, but we don't want all of that to get in the way of
installation.
Ok, I can agree with that.
For that, I blame -- among other things -- Module::Starter for
including pod.t and pod-coverage.t and with a default setting to run
tests. Better would have been to skip tests unless
$ENV{AUTHOR_TESTING} or some similar flag was set.
Again, yes. Though I'm going to stick with delete them.
I think the important bit is that `make test` only runs tests which
verify the module's functionality.
Anything else needs to be a separate test target or out-of-band tool.
I don't agree. What runs when I do 'make test' is up to me, and if I
want to litter it up with 'author' tests, then that's my business; right
or wrong. Don't like it, then don't use my modules. (I still think all
author tests should not run by default...)
What needs to be unencumbered is:
perl Makefile.PL (should not reference non-functional modules in prereq)
make install (just install it please)
Any notion that make test needs to be altered, or should or shouldn't
run some and not other 'tests' is silly imho.
Putting Test::Foo in PREREQ=Bad. Fine. I understand.
Bitching about how many and what tests get run during 'make test' is absurd.
Of course, I could be missing the point of it all.
signature.asc
Description: OpenPGP digital signature
Re: Fixing the damage caused by has_test_pod
# from Christopher H. Laco # on Monday 30 July 2007 11:14 am: I don't agree. What runs when I do 'make test' is up to me, and if I want to litter it up with 'author' tests, then that's my business; right or wrong. Don't like it, then don't use my modules. (I still think all author tests should not run by default...) This is not about what happens when *you* do `make test`, it's about what happens when the end-user does `make test`. The default module-starter setup creates this t/pod.t #!perl -T use Test::More; eval use Test::Pod 1.14; plan skip_all = Test::Pod 1.14 required for testing POD if $@; all_pod_files_ok(); If *you* don't have Test::Pod, and *I* do, *I* cannot install your module if the pod doesn't pass. This could possibly fail as part of a dependency of a dependency of a dependency. This makes Perl harder to use, which is bad. Thus, I posit that the quality of the module is generally lower if 'boilerplate.t', 'pod-coverage.t', and 'pod.t' *exist*. Kwalitee is supposed to be an approximation of quality, not the opposite of it. --Eric -- If the collapse of the Berlin Wall had taught us anything, it was that socialism alone was not a sustainable economic model. --Robert Young --- http://scratchcomputing.com ---
Re: Fixing the damage caused by has_test_pod
On Jul 30, 2007, at 1:58 PM, Eric Wilhelm wrote: Thus, I posit that the quality of the module is generally lower if 'boilerplate.t', 'pod-coverage.t', and 'pod.t' *exist*. Certainly, the intent is to have boilerplate.t NOT get shipped with the module. -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
Re: Fixing the damage caused by has_test_pod
* Andy Lester [EMAIL PROTECTED] [2007-07-30 21:41]: On Jul 30, 2007, at 1:58 PM, Eric Wilhelm wrote: Thus, I posit that the quality of the module is generally lower if 'boilerplate.t', 'pod-coverage.t', and 'pod.t' *exist*. Certainly, the intent is to have boilerplate.t NOT get shipped with the module. Disagree. The tarball should contain the full source necessary to recreate everything the author did. Stuff that’s not relevant to end-user installation of the module should simply be kept out of the way. Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: Fixing the damage caused by has_test_pod
On 30 Jul 2007, at 20:46, A. Pagaltzis wrote: * Andy Lester [EMAIL PROTECTED] [2007-07-30 21:41]: On Jul 30, 2007, at 1:58 PM, Eric Wilhelm wrote: Thus, I posit that the quality of the module is generally lower if 'boilerplate.t', 'pod-coverage.t', and 'pod.t' *exist*. Certainly, the intent is to have boilerplate.t NOT get shipped with the module. Disagree. The tarball should contain the full source necessary to recreate everything the author did. Stuff that’s not relevant to end-user installation of the module should simply be kept out of the way. I agree. If I'm going to produce a patch for someone I want to run the author tests before I submit it. -- Andy Armstrong, hexten.net
Re: Fixing the damage caused by has_test_pod
On 7/30/07, Andy Armstrong [EMAIL PROTECTED] wrote:
I agree. If I'm going to produce a patch for someone I want to run
the author tests before I submit it.
Including the tests but leaving them disabled is my preferred
approach. It is trivial to subclass Module::Build to set the
appropriate environment variables in a Build disttest action if you
prefer that instead of just leaving it set.
In case anyone is inspired to update the Module::Starter (or other)
boilerplate, I'm attaching my standard pod-coverage.t and recommending
it as a more robust approach than the existing boilerplate. Set
minimum module versions as necessary for your documentation style
and/or bug fixes in T::P::C and P::C.
Regards,
David
# pod-coverage.t
use Test::More;
plan skip_all = Skipping author tests if not $ENV{AUTHOR_TESTING};
my $min_tpc = 1.08;
eval use Test::Pod::Coverage $min_tpc;
plan skip_all = Test::Pod::Coverage $min_tpc required for testing
POD coverage
if $@;
my $min_pc = 0.18;
eval use Pod::Coverage $min_pc;
plan skip_all = Pod::Coverage $min_pc required for testing POD coverage
if $@;
all_pod_coverage_ok();
__END__
use Test::Pod::Coverage; # Fake CPANTS
Re: Fixing the damage caused by has_test_pod
Eric Wilhelm wrote: # from Christopher H. Laco # on Monday 30 July 2007 11:14 am: I don't agree. What runs when I do 'make test' is up to me, and if I want to litter it up with 'author' tests, then that's my business; right or wrong. Don't like it, then don't use my modules. (I still think all author tests should not run by default...) This is not about what happens when *you* do `make test`, it's about what happens when the end-user does `make test`. The default module-starter setup creates this t/pod.t #!perl -T use Test::More; eval use Test::Pod 1.14; plan skip_all = Test::Pod 1.14 required for testing POD if $@; all_pod_files_ok(); If *you* don't have Test::Pod, and *I* do, *I* cannot install your module if the pod doesn't pass. Huh? In that code, no Test::POD = skip all tests. How does that equate to an install failure for anyone...either the author, the dist maint, or the end user? signature.asc Description: OpenPGP digital signature
Re: Fixing the damage caused by has_test_pod
On Jul 30, 2007, at 2:46 PM, A. Pagaltzis wrote: Certainly, the intent is to have boilerplate.t NOT get shipped with the module. Disagree. The tarball should contain the full source necessary to recreate everything the author did. Stuff that’s not relevant to end-user installation of the module should simply be kept out of the way. You can't disagree. It's a statement of fact that RJBS and I did not intend for boilerplate.t to actually get shipped with the module. -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
Re: Fixing the damage caused by has_test_pod
On Monday 30 July 2007 13:19:56 Christopher H. Laco wrote: Eric Wilhelm wrote: If *you* don't have Test::Pod, and *I* do, *I* cannot install your module if the pod doesn't pass. Huh? In that code, no Test::POD = skip all tests. How does that equate to an install failure for anyone...either the author, the dist maint, or the end user? Eric's case is where user who just wants to install the module has Test::POD but there are POD errors, so the tests fail and he or she has to examine the test output to see if forcing the installation is a good idea. It's not about not having Test::POD installed. If the distribution with the failing POD happens to be way up the dependency chain, I think we can safely call the situation A big bucket of suck. -- c
Re: Fixing the damage caused by has_test_pod
On Jul 30, 2007, at 3:06 PM, David Golden wrote: In case anyone is inspired to update the Module::Starter (or other) boilerplate, I'm attaching my standard pod-coverage.t and recommending it as a more robust approach than the existing boilerplate. Set minimum module versions as necessary for your documentation style and/or bug fixes in T::P::C and P::C. Better fora for patches exist. Best would be the RT queue. -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
Re: Fixing the damage caused by has_test_pod
On 30 Jul 2007, at 21:26, Andy Lester wrote: You can't disagree. It's a statement of fact that RJBS and I did not intend for boilerplate.t to actually get shipped with the module. Bah - sucked me in too :) Everything you'd use if you were the developer of the module should be shipped - because the person to whom it is shipped may turn out to be an additional developer of the module. -- Andy Armstrong, hexten.net
Re: Fixing the damage caused by has_test_pod
On Mon, Jul 30, 2007 at 10:01:12PM +0100, Andy Armstrong ([EMAIL PROTECTED]) wrote: Everything you'd use if you were the developer of the module should be shipped - because the person to whom it is shipped may turn out to be an additional developer of the module. Right, and once boilerplate.t passes, it never needs to exist again. xoxo, Andy -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
Re: Fixing the damage caused by has_test_pod
Eric Wilhelm wrote: # from Christopher H. Laco # on Monday 30 July 2007 11:14 am: I don't agree. What runs when I do 'make test' is up to me, and if I want to litter it up with 'author' tests, then that's my business; right or wrong. Don't like it, then don't use my modules. (I still think all author tests should not run by default...) This is not about what happens when *you* do `make test`, it's about what happens when the end-user does `make test`. The default module-starter setup creates this t/pod.t #!perl -T use Test::More; eval use Test::Pod 1.14; plan skip_all = Test::Pod 1.14 required for testing POD if $@; all_pod_files_ok(); If *you* don't have Test::Pod, and *I* do, *I* cannot install your module if the pod doesn't pass. Oh, duh. *smacks forehead*. I get it. You have it installed, the tests run, and fail, then you can't install. I say that's fine. If it fails and you can't install it, then don't. Arguments about whether the tests should or shouldn't be run [or included at all] is irrelevant. Tests failed. Don't install. File RT. If the author made a choice to have them run always and piss people off, or restrict the user base, then that's the authors prerogative. Now, I won't argue that it would be better if the author put in a skip check for TEST_AUTHOR or other variants, but that's up to the author and not something that should be enforced or needs to be regulated. If I ship a module and test coverage fails, that's still a bug. The end user still has no documentation on what the hell the 'naked' method is supposed to do. Sure, Test::Pod::Coverage could fail for some reason other than it's main purpose of checking coverage and finding a naked method, but so then can Test::More and any other test module. Failed Tests = Don't Install. Any arguments over what tests failed and why, or if they should even be run or shipped is moot in my book. signature.asc Description: OpenPGP digital signature
Re: Fixing the damage caused by has_test_pod
On 7/30/07, Christopher H. Laco [EMAIL PROTECTED] wrote: Sure, Test::Pod::Coverage could fail for some reason other than it's main purpose of checking coverage and finding a naked method, but so then can Test::More and any other test module. Ironically, that just happened today with Statistics::Bedford: http://cpantesters.perl.org/show/Statistics-Benford.html There was a coverage failure on a hidden ._Bedford.pm. I'm guessing that was some editor swap file, but somehow, it got packaged and distributed. David
Re: Fixing the damage caused by has_test_pod
On 30 Jul 2007, at 23:22, David Golden wrote: There was a coverage failure on a hidden ._Bedford.pm. I'm guessing that was some editor swap file, but somehow, it got packaged and distributed. Mac OS resource fork I think. -- Andy Armstrong, hexten.net
Re: Fixing the damage caused by has_test_pod
# from David Golden
# on Monday 30 July 2007 01:06 pm:
# pod-coverage.t
use Test::More;
plan skip_all = Skipping author tests if not $ENV{AUTHOR_TESTING};
my $min_tpc = 1.08;
eval use Test::Pod::Coverage $min_tpc;
plan skip_all = Test::Pod::Coverage $min_tpc required for testing
No. If AUTHOR_TESTING, fail miserably unless the pod and coverage both
1. gets tested and 2. passes. That means the Test::Pod::* module in
question must load.
While you're at it, put it somewhere out of the way (as Aristotle
said), like at/ and forget this environment variable sillyness.
at/pod-coverage.t
use Test::More;
use Test::Pod::Coverage 1.08;
all_pod_coverage_ok();
at/pod.t
use Test::More;
use Test::Pod 1.14;
all_pod_files_ok();
Who knows when you'll run into a box in the publishing industry which
just happens to be setup for AUTHOR_TESTING some other system. Or,
simply the hardcore CPAN author who put it in their .bashrc and forgot
about it until some broken pod appeared in the middle of a big
dependency stack. Environment variables are as global as they get.
I think the skipped more often than run use-case should also imply
something. I keep my winter clothes in the back of the closet during
the summer and all-that.
--Eric
--
It is impossible to make anything foolproof because fools are so
ingenious.
--Murphy's Second Corollary
---
http://scratchcomputing.com
---
Re: Fixing the damage caused by has_test_pod
# from chromatic # on Monday 30 July 2007 01:39 pm: On Monday 30 July 2007 13:19:56 Christopher H. Laco wrote: Eric Wilhelm wrote: If *you* don't have Test::Pod, and *I* do, *I* cannot install your module if the pod doesn't pass. Huh? In that code, no Test::POD = skip all tests. How does that equate to an install failure for anyone...either the author, the dist maint, or the end user? Eric's case is where user who just wants to install the module has Test::POD but there are POD errors, so the tests fail and he or she has to examine the test output to see if forcing the installation is a good idea. It's not about not having Test::POD installed. Well, it is _all about_ whether it is installed because that's the only conditional in the test. If Test::Pod *is not* on the author system, then pod errors get through. Then, if it *is* installed on the destination system, test fails, game over. --Eric -- Don't worry about what anybody else is going to do. The best way to predict the future is to invent it. --Alan Kay --- http://scratchcomputing.com ---
Re: Fixing the damage caused by has_test_pod
# from Christopher H. Laco # on Monday 30 July 2007 01:35 pm: If it fails and you can't install it, then don't. Arguments about whether the tests should or shouldn't be run [or included at all] is irrelevant. Tests failed. Don't install. I think you're looking at this as oh, I'll try whiz-bang module foo. Sure, you can just dismiss it, but that's really not the concern. It is quite relevant when we get 2-3 levels deep in dependencies and somebody made a slight slip or added a new dependency which fails for this reason. Then we end up digging through a customer e-mail trying to sort through what went wrong and finding that a lot of time has been lost on something very simple and dismissable. And then, it turns out that this is a situation which the perl qa community has encouraged, condoned, and rewarded even. Gah! Where was I during that vote? File RT. Won't happen. Not ~95% of the time anyway. If the author made a choice to have them run always and piss people off, or restrict the user base, then that's the authors prerogative. In most cases, it is not the author's deliberate choice that these things happen, instead, it is believed to be some sort of mandate that they be in the distro because Module::Starter put them there or CPANTS says I should have them. That is, the vast majority of authors who have them have not worked through the logic of what damage these tests do on the install side. --Eric -- Any sufficiently advanced incompetence is indistinguishable from malice. --The Napoleon-Clarke Law --- http://scratchcomputing.com ---
Re: Fixing the damage caused by has_test_pod
On 7/30/07, Eric Wilhelm [EMAIL PROTECTED] wrote: Who knows when you'll run into a box in the publishing industry which just happens to be setup for AUTHOR_TESTING some other system. Or, simply the hardcore CPAN author who put it in their .bashrc and forgot about it until some broken pod appeared in the middle of a big dependency stack. Environment variables are as global as they get. s/AUTHOR_TESTING/PERL_AUTHOR_TESTING/ And hardcore CPAN authors aren't the audience that this discussion is trying to help. David
Re: Fixing the damage caused by has_test_pod
# from David Golden # on Monday 30 July 2007 05:03 pm: Or, simply the hardcore CPAN author who put it in their .bashrc and forgot about it ^-- who will it bite first, will it be him, her, or me? until some broken pod appeared in the middle of a big dependency stack. Environment variables are as global as they get. s/AUTHOR_TESTING/PERL_AUTHOR_TESTING/ And hardcore CPAN authors aren't the audience that this discussion is trying to help. I admire your skillful rhetoric and valiant attempt to avoid typing `mkdir at`, but I fail to understand the resistance. I refer again to the skipped more often than run issue and the supposition that residents of warm climates tend not to keep their ski equipment next to the front door. --Eric -- Consumers want choice, consumers want openness. --Rob Glaser --- http://scratchcomputing.com ---
Re: Fixing the damage caused by has_test_pod
On Monday 30 July 2007 19:39:55 David Golden wrote: I admire your desire to engineer a better solution, but there is a lot of inertia in the existing tools. And, yes, they can be customized, but few people do. Easy solution: STOP REWARDING PEOPLE TO SHIP AND RUN THE STUPID POD TESTS WITH POINTS OF KWALITEE. Even better solution: PENALIZE people for forcing the stupid POD tests by removing Kwalitee points. I was wrong; the POD tests should be part of CPANTS. We just had the wrong sign on the point value. -- c
Re: Fixing the damage caused by has_test_pod
# from David Golden # on Monday 30 July 2007 07:39 pm: On 7/30/07, Eric Wilhelm [EMAIL PROTECTED] wrote: `mkdir at`, but I fail to understand the resistance. * CPANTS Kwalitee * Module::Starter Both need correcting anyway. * Test::Pod::Coverage Why does that matter? Does it care where its .t file lives? * ExtUtils::MakeMaker Not affected. prove at/ Or, runtests at/ * Module::Build Not affected. ./Build testpod ./Build testpodcoverage Or, of course runtests at/ --Eric -- To succeed in the world, it is not enough to be stupid, you must also be well-mannered. --Voltaire --- http://scratchcomputing.com ---
Re: Fixing the damage caused by has_test_pod
Adam Kennedy wrote: For background on this email, see the following entry in my journal. http://use.perl.org/~Alias/journal/33893 Lately, I've noticed a series of modules that are cargo-culting the use of test_pod and test_pod_coverage in their tests. Not only are we seeing spurious failures for users, but worse some authors are putting Test::Pod and Test::Pod::Coverage as dependencies, and in extreme cases, Perl::Critic as well. Now, one argument is that it's the author's fault for not appropriately skipping these tests if the Test::Pod module is not available. However, in most of these cases the author would not have made these errors AT ALL if not for the fact that Kwalitee says they should have these tests. The CPANTS Game just makes this pressure worse, by making it competive. That said, I do see the value of these tests EXISTING in the distribution, as their provide a guarentee that any new maintainers for the modules will continue to comply with the principles those tests implement. BUT for most of these tests, it is BAD that the end-user EVER run them. Thus, I would like to propose the following. 1. That the running of POD-related tests by end users is considered harmful. I can never agree with this. A failing pod test is the first sign pod2* generation is broken. It's rare, but I've been in the situation where for some reason (mistmatching dist requirements, failed upgrades, bad dist packages, broken troff, etc) the creation of man pages/html from the dist pod fails outright. Pod tests assure the user that they will be able to perldoc/man the thing they are installing. That's doubly true if you're using pod2usage to expose help in scripts. Should they be run automatically, all the time? Maybe. Maybe not. Should they skip if the test modules aren't installed? Absolutely. Should they require the Test* modules in prereq? OF course not. But I don't think saying pod tests are something the end user should never run is wrong. Then again, I'm never on the popular side of these discussions. Personally, part of me wants to say: stop worrying about what tests I decide to ship and enable with my dists. -=Chris signature.asc Description: OpenPGP digital signature
Re: Fixing the damage caused by has_test_pod
# from Adam Kennedy # on Saturday 28 July 2007 01:52 pm: Unfortunately, the pod coverage tests requires the module to be compiled, so CPANTS can never safely run it, and thus can never run it at all. :) No. *Pod::Coverage* requires the module to be compiled. Testing the coverage of pod does not. For those that want to nit-pick. Yes, I know methods can be defined via loops/accessors at compile-time. (I'm not documenting --set_foo() anyway.) They can also be defined at run-time (via AUTOLOAD, and possibly even driven by HTTP or GPIO), so I'll happily draw a line at static analysis and call it good. --Eric -- Ignorance more frequently begets confidence than does knowledge. -- Charles Darwin --- http://scratchcomputing.com ---
Re: Fixing the damage caused by has_test_pod
# from Christopher H. Laco # on Sunday 29 July 2007 08:00 am: It's rare, but I've been in the situation where for some reason (mistmatching dist requirements, failed upgrades, bad dist packages, broken troff, etc) the creation of man pages/html from the dist pod fails outright. That reinforces the point that running pod tests on the install side is a waste of time. Maybe the build tools need to make more noise and/or refuse to install if pod2man/html goes horribly wrong. After all, we abort *before testing* if compilation of code fails, so why not treat compilation of docs the same way? But I don't think saying pod tests are something the end user should never run is wrong. So, it would be correct to say that end users should never run pod tests then? I can totally agree with that. ;-) Personally, part of me wants to say: stop worrying about what tests I decide to ship and enable with my dists. I'm thinking the installer should have the option to scan the tests and skip any that use qr/Test::Pod(?:::Coverage)?/. Speaking of which, I just noticed that Module::CPANTS::Uses doesn't even check whether the Test::Pod(::Coverage) modules are used in the *tests* -- it is happy if they get mentioned anywhere. Further, since Module::ExtractUse doesn't understand strings, you can simply put this in one of your tests to game the kwalitee: my $q = 'use Test::Pod::Coverage; use Test::Pod;'; --Eric -- It works better if you plug it in! --Sattinger's Law --- http://scratchcomputing.com ---
Fixing the damage caused by has_test_pod
For background on this email, see the following entry in my journal.
http://use.perl.org/~Alias/journal/33893
Lately, I've noticed a series of modules that are cargo-culting the use
of test_pod and test_pod_coverage in their tests.
Not only are we seeing spurious failures for users, but worse some
authors are putting Test::Pod and Test::Pod::Coverage as dependencies,
and in extreme cases, Perl::Critic as well.
Now, one argument is that it's the author's fault for not appropriately
skipping these tests if the Test::Pod module is not available.
However, in most of these cases the author would not have made these
errors AT ALL if not for the fact that Kwalitee says they should have
these tests.
The CPANTS Game just makes this pressure worse, by making it competive.
That said, I do see the value of these tests EXISTING in the
distribution, as their provide a guarentee that any new maintainers for
the modules will continue to comply with the principles those tests
implement.
BUT for most of these tests, it is BAD that the end-user EVER run them.
Thus, I would like to propose the following.
1. That the running of POD-related tests by end users is considered harmful.
2. That the test_pod and test_pod_coverage tests by modified, such that
these tests check for the mentioning of $ENV{AUTOMATED_TESTING} in the
tests scripts, ensuring that the tests not only exist, but exist in a
form that is otherwise benign for end users.
This change to the tests is targetted, in the sense that people who do
NOT have the test_pod and test_pod_coverage tests will not be further
penalized, while the people that are doing cargo-culted damaging test
scripts are the ones most likely to care about the CPANTS game point loss.
Thoughts? Alternative?
Adam K
Re: Fixing the damage caused by has_test_pod
On Saturday 28 July 2007 09:38:14 Adam Kennedy wrote:
For background on this email, see the following entry in my journal.
http://use.perl.org/~Alias/journal/33893
Lately, I've noticed a series of modules that are cargo-culting the use
of test_pod and test_pod_coverage in their tests.
Not only are we seeing spurious failures for users, but worse some
authors are putting Test::Pod and Test::Pod::Coverage as dependencies,
and in extreme cases, Perl::Critic as well.
Now, one argument is that it's the author's fault for not appropriately
skipping these tests if the Test::Pod module is not available.
However, in most of these cases the author would not have made these
errors AT ALL if not for the fact that Kwalitee says they should have
these tests.
The CPANTS Game just makes this pressure worse, by making it competive.
That said, I do see the value of these tests EXISTING in the
distribution, as their provide a guarentee that any new maintainers for
the modules will continue to comply with the principles those tests
implement.
BUT for most of these tests, it is BAD that the end-user EVER run them.
Thus, I would like to propose the following.
1. That the running of POD-related tests by end users is considered
harmful.
2. That the test_pod and test_pod_coverage tests by modified, such that
these tests check for the mentioning of $ENV{AUTOMATED_TESTING} in the
tests scripts, ensuring that the tests not only exist, but exist in a
form that is otherwise benign for end users.
This change to the tests is targetted, in the sense that people who do
NOT have the test_pod and test_pod_coverage tests will not be further
penalized, while the people that are doing cargo-culted damaging test
scripts are the ones most likely to care about the CPANTS game point loss.
Thoughts? Alternative?
I use Module::Build (don't jerk your knee yet, Adam) so that I can easily
override 'make disttest' to run my author tests. I keep them in t/author/.
Build.PL for Test::MockObject should make it a little clearer:
http://search.cpan.org/src/CHROMATIC/Test-MockObject-1.08/Build.PL
There's no reason that someone who isn't me couldn't write something similar
for MakeMaker.
I prefer this approach because I don't have to remember any environment
variables to run all of my tests, and because my normal workflow before a
release runs the distribution tests.
Let me also suggest #3: revise the appropriate CPANTS tests to add Kwalitee
only if the static analysis tests do *not* run by default.
-- c
Re: Fixing the damage caused by has_test_pod
# from Adam Kennedy
# on Saturday 28 July 2007 09:38 am:
Thus, I would like to propose the following.
1. That the running of POD-related tests by end users is considered
harmful.
+20
2. That the test_pod and test_pod_coverage tests by modified, such
that these tests check for the mentioning of $ENV{AUTOMATED_TESTING}
in the tests scripts, ensuring that the tests not only exist, but
exist in a form that is otherwise benign for end users.
I propose that they be deleted.
The build system already contains all of the needed info to accomplish
the same thing.
./Build testpod
./Build testpodcoverage
That's part of my pre-release process, thus my dists will never contain
these files.
Running a kwalitee check is also part of the pre-release process, but I
omit these 'has a file $foo' metrics because I believe they *hinder*
quality. Perhaps the dist should contain a kwalitee.yml file stating
which checks were run and which metrics are being protested. That's
conveying the information: Yes, I care about quality and I have
imposed these standards on myself before releasing.
CPANTS should *just run* testpod and testpodcoverage -- report the
results, not the presence of some file which may or may not work.
--Eric
--
Chicken farmer's observation: Clunk is the past tense of cluck.
---
http://scratchcomputing.com
---
Re: Fixing the damage caused by has_test_pod
chromatic wrote: I use Module::Build (don't jerk your knee yet, Adam) so that I can easily override 'make disttest' to run my author tests. I keep them in t/author/. Build.PL for Test::MockObject should make it a little clearer: http://search.cpan.org/src/CHROMATIC/Test-MockObject-1.08/Build.PL There's no reason that someone who isn't me couldn't write something similar for MakeMaker. I prefer this approach because I don't have to remember any environment variables to run all of my tests, and because my normal workflow before a release runs the distribution tests. Let me also suggest #3: revise the appropriate CPANTS tests to add Kwalitee only if the static analysis tests do *not* run by default. Me? Knee-jerk? NEVER! :) As long as the following is true for tests unrelated to functionality, I'm fine with any and every solution. The mechanism itself is probably not important, we just want the following behaviour. 1. The author runs the tests 2. Automated testing systems run the tests 3. End-users installing the modules do NOT run the tests Adam K
Re: Fixing the damage caused by has_test_pod
The POD testing I can mostly agree with.
Unfortunately, the pod coverage tests requires the module to be
compiled, so CPANTS can never safely run it, and thus can never run it
at all. :)
Adam K
Eric Wilhelm wrote:
# from Adam Kennedy
# on Saturday 28 July 2007 09:38 am:
Thus, I would like to propose the following.
1. That the running of POD-related tests by end users is considered
harmful.
+20
2. That the test_pod and test_pod_coverage tests by modified, such
that these tests check for the mentioning of $ENV{AUTOMATED_TESTING}
in the tests scripts, ensuring that the tests not only exist, but
exist in a form that is otherwise benign for end users.
I propose that they be deleted.
The build system already contains all of the needed info to accomplish
the same thing.
./Build testpod
./Build testpodcoverage
That's part of my pre-release process, thus my dists will never contain
these files.
Running a kwalitee check is also part of the pre-release process, but I
omit these 'has a file $foo' metrics because I believe they *hinder*
quality. Perhaps the dist should contain a kwalitee.yml file stating
which checks were run and which metrics are being protested. That's
conveying the information: Yes, I care about quality and I have
imposed these standards on myself before releasing.
CPANTS should *just run* testpod and testpodcoverage -- report the
results, not the presence of some file which may or may not work.
--Eric
Re: Fixing the damage caused by has_test_pod
* chromatic [EMAIL PROTECTED] [2007-07-28 19:10]: Let me also suggest #3: revise the appropriate CPANTS tests to add Kwalitee only if the static analysis tests do *not* run by default. Maybe deduct points for depencies on obvious author-testing distros? (Although that could get tricky when rating such an author test module itself.) Regards, -- Aristotle Pagaltzis // http://plasmasturm.org/
Re: Fixing the damage caused by has_test_pod
On 7/28/07, A. Pagaltzis [EMAIL PROTECTED] wrote: * chromatic [EMAIL PROTECTED] [2007-07-28 19:10]: Let me also suggest #3: revise the appropriate CPANTS tests to add Kwalitee only if the static analysis tests do *not* run by default. Maybe deduct points for depencies on obvious author-testing distros? Just be sure to deduct more points than they get for having pod tests, so it's a net loss. :-) David
