Which Modules Does Your Distro Need?
Imagine this: ./Build testdeps That would be a custom action which would load a special Test::More wrapper which, at the end of each test program, cache %INC. At the end of the test run, it would load up the cached versions of %INC and build a report of all loaded modules. Sample header: Module Core Prereq Loaded --- -- --- Attribute::Handlers 5.007003 no 0.78_01 SUPERn/a 1.161.14 Test::Class n/a no 0.24 In other words, it would tell you all of the modules loaded, when (or if) they were added to the core and whether or not they're listed as a preref. In the simple report above, we can see three modules loaded, when they were released in core, what version, if any, is listed as a prereq and what version, if any, is loaded (might be good, if differing versions somehow got loaded by different test programs, to track that, too). We can see right away from the report above that Test::Class will cause many folk's installs to fail since it's not listed as a prereq and it's possible that SUPER doesn't need as high a version as is listed, though investigation into the history for a version mismatch is warranted. Thoughts? Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/
Re: Which Modules Does Your Distro Need?
On 31 Jul 2007, at 11:56, Ovid wrote: We can see right away from the report above that Test::Class will cause many folk's installs to fail since it's not listed as a prereq and it's possible that SUPER doesn't need as high a version as is listed, though investigation into the history for a version mismatch is warranted. Thoughts? I like it. At some point in the future we could spew that information out as part of the TAP stream (as YAMLish metadata) and gather and analyse it in the harness. -- Andy Armstrong, hexten.net
Re: Which Modules Does Your Distro Need?
On 31 Jul 2007, at 11:56, Ovid wrote: [snip] We can see right away from the report above that Test::Class will cause many folk's installs to fail since it's not listed as a prereq and it's possible that SUPER doesn't need as high a version as is listed, though investigation into the history for a version mismatch is warranted. Thoughts? [snip] Would the modules that get lists that are not your prerequisites, but are the prerequisites of the prerequisites of some of your prerequisites be problematical? Adrian
Re: Which Modules Does Your Distro Need?
--- Andy Armstrong <[EMAIL PROTECTED]> wrote: > I like it. At some point in the future we could spew that information > out as part of the TAP stream (as YAMLish metadata) and gather and > analyse it in the harness. Sounds great, but that we consume the test data, not produce it. The reason I was thinking about a Test::More wrapper is that this could ensure that the code picks it up. Oh, duh. When we run the perl process, internally we could do something like: perl -Isome/lib -MCapture::Inc t/test_prog.t That could work. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/
Re: Which Modules Does Your Distro Need?
--- Adrian Howard <[EMAIL PROTECTED]> wrote: > Would the modules that get lists that are not your prerequisites, but > are the prerequisites of the prerequisites of some of your > prerequisites be problematical? That's why this would just be a report and not a test. Manual investigation would be needed to find out unless you hooked this into a minicpan installation or something similar in order to figure out what prereqs are listed there. Even then, that would be difficult. I figure that running this on something like HTML::TokeParser::Simple would be trivial. Running it on Jifty would be painful :) Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/
Re: Which Modules Does Your Distro Need?
Adrian Howard wrote: > > On 31 Jul 2007, at 11:56, Ovid wrote: > [snip] >> We can see right away from the report above that Test::Class will cause >> many folk's installs to fail since it's not listed as a prereq and it's >> possible that SUPER doesn't need as high a version as is listed, though >> investigation into the history for a version mismatch is warranted. >> >> Thoughts? > [snip] > > Would the modules that get lists that are not your prerequisites, but > are the prerequisites of the prerequisites of some of your prerequisites > be problematical? I was about to say that. I declare LWP as a prereq, but loading LWP loads tons more modules. I don't have to declare all those as prereqs, in fact I should not as its violating LWP's encapsulation. I don't know an easy way to give it the necessary smarts.
Re: Fixing the damage caused by has_test_pod
On 7/31/07, Eric Wilhelm <[EMAIL PROTECTED]> wrote: > >* Test::Pod::Coverage > > Why does that matter? Does it care where its .t file lives? The synopsis suggests t/pod-coverage.t. David
Re: Fixing the damage caused by has_test_pod
A. Pagaltzis wrote: Disagree. The tarball should contain the full source necessary to recreate everything the author did. Stuff that’s not relevant to end-user installation of the module should simply be kept out of the way. I ever-so-respectfully disagree. I am not going to include the original data that I downloaded from the UK telecoms regulator for a particular build of Number::Phone::UK::Data (although I do include a shell script that downloads the current version of that data), nor the list of exchanges for Number::Phone::UK::DetailedLocations. Bear in mind that these two modules are already 2.5MB and 7.5MB *compressed*, 18 and 36MB uncompressed. -- David Cantrell
Re: Which Modules Does Your Distro Need?
On 31 Jul 2007, at 12:16, Michael G Schwern wrote: I was about to say that. I declare LWP as a prereq, but loading LWP loads tons more modules. I don't have to declare all those as prereqs, in fact I should not as its violating LWP's encapsulation. I don't know an easy way to give it the necessary smarts. Presumably there's a non-easy way that involves adding some runops instrumentation to Perl and using that to find out where modules are included from? Anything not included directly from the code under test is uninteresting to us. Slightly tangential but related: is there currently a module that simplifies simulating the non-availability of selected modules? use unavailable qw/LWP::UserAgent/; use Module::That::Needs::LWP::UserAgent; -- Andy Armstrong, hexten.net
Re: Fixing the damage caused by has_test_pod
Eric Wilhelm wrote:
# from David Golden
# pod-coverage.t
use Test::More;
plan skip_all => "Skipping author tests" if not $ENV{AUTHOR_TESTING};
...
No. If AUTHOR_TESTING, fail miserably unless the pod and coverage both
1. gets tested and 2. passes. That means the Test::Pod::* module in
question must load.
Wrong. If AUTHOR_TESTING then surely all tests *must* pass both with
and *without* the optional module!
I'm still stuck on thinking up a decent name for this:
http://drhyde.cvs.sourceforge.net/drhyde/perlmodules/No/lib/No.pm?revision=1.2&view=markup
which will make those sort of tests much easier. Yes, I do know that
there are other modules that do some of what this does. But only some
of it. Also before uploading it to the CPAN I really need a volunteer
to sanity-check it on VMS.
--
David Cantrell
Re: Fixing the damage caused by has_test_pod
On 31 Jul 2007, at 12:19, David Cantrell wrote: A. Pagaltzis wrote: Disagree. The tarball should contain the full source necessary to recreate everything the author did. Stuff that’s not relevant to end-user installation of the module should simply be kept out of the way. I ever-so-respectfully disagree. I am not going to include the original data that I downloaded from the UK telecoms regulator for a particular build of Number::Phone::UK::Data (although I do include a shell script that downloads the current version of that data), nor the list of exchanges for Number::Phone::UK::DetailedLocations. Bear in mind that these two modules are already 2.5MB and 7.5MB *compressed*, 18 and 36MB uncompressed. That's something of a special case, no? As you say there's a shell script to download the data anyway - so no need to distribute it. -- Andy Armstrong, hexten.net
Re: Fixing the damage caused by has_test_pod
On 31 Jul 2007, at 12:27, David Cantrell wrote: Wrong. If AUTHOR_TESTING then surely all tests *must* pass both with and *without* the optional module! I'm still stuck on thinking up a decent name for this: http://drhyde.cvs.sourceforge.net/drhyde/perlmodules/No/lib/No.pm? revision=1.2&view=markup As per my post of a few seconds ago, how about making it a pragma: 'unavailable'? :) -- Andy Armstrong, hexten.net
Re: Which Modules Does Your Distro Need?
--- Andy Armstrong <[EMAIL PROTECTED]> wrote:
> use unavailable qw/LWP::UserAgent/;
> use Module::That::Needs::LWP::UserAgent;
>
Can you just do the following?
BEGIN { $INC{'LWP/UserAgent.pm'} = 1 }
use Module::That::Needs::LWP::UserAgent;
Should be trivial to write, unless I have (as usual) missed glaringly
obvious corner cases.
Cheers,
Ovid
--
Buy the book - http://www.oreilly.com/catalog/perlhks/
Perl and CGI - http://users.easystreet.com/ovid/cgi_course/
Personal blog - http://publius-ovidius.livejournal.com/
Tech blog - http://use.perl.org/~Ovid/journal/
Re: Which Modules Does Your Distro Need?
On 31 Jul 2007, at 12:34, Ovid wrote:
Can you just do the following?
BEGIN { $INC{'LWP/UserAgent.pm'} = 1 }
use Module::That::Needs::LWP::UserAgent;
Should be trivial to write, unless I have (as usual) missed glaringly
obvious corner cases.
That doesn't cause it to die immediately - because it just thinks it
already has LWP::UserAgent.
--
Andy Armstrong, hexten.net
Re: Which Modules Does Your Distro Need?
On 31 Jul 2007, at 12:24, Andy Armstrong wrote: On 31 Jul 2007, at 12:16, Michael G Schwern wrote: I was about to say that. I declare LWP as a prereq, but loading LWP loads tons more modules. I don't have to declare all those as prereqs, in fact I should not as its violating LWP's encapsulation. I don't know an easy way to give it the necessary smarts. Presumably there's a non-easy way that involves adding some runops instrumentation to Perl and using that to find out where modules are included from? Anything not included directly from the code under test is uninteresting to us. Hmmm... you could probably hack a 90% solution to that with the old codref on @INC trick... Slightly tangential but related: is there currently a module that simplifies simulating the non-availability of selected modules? use unavailable qw/LWP::UserAgent/; use Module::That::Needs::LWP::UserAgent; Test::Without::Module ? Adrian
Re: Which Modules Does Your Distro Need?
On 31 Jul 2007, at 12:16, Ovid wrote: I figure that running this on something like HTML::TokeParser::Simple would be trivial. Running it on Jifty would be painful :) Devel::TraceLoad seems to be on roughly the right track although it's rather rough around the edges. It works by replacing CORE::GLOBAL::require. I assume it's been abandoned though - last release in 2001, tests fail. [12:55] andy $ perl -MDevel::TraceLoad examples/simple.pl +strict.pm [from: examples/simple.pl 3] +warnings.pm [from: examples/simple.pl 4] +HTML/Tiny.pm [from: examples/simple.pl 5] +strict.pm[from: /Library/Perl/5.8.6/HTML/Tiny.pm 3] +Carp.pm[from: /Library/Perl/5.8.6/HTML/Tiny.pm 4] +vars.pm[from: /Library/Perl/5.8.6/HTML/Tiny.pm 6] +strict.pm[from: /Library/Perl/5.8.6/HTML/Tiny.pm 21] Looks like a reasonable starting point. Can anyone think of anything wrong with the approach of replacing require with an instrumented version? If the consensus is that that's a sensible approach I'll play with tidying up the code. [1] http://search.cpan.org/~pverd/Devel-TraceLoad-0.07/ -- Andy Armstrong, hexten.net
Re: Fixing the damage caused by has_test_pod
On 31 Jul 2007, at 12:50, Steffen Mueller wrote: I'm still stuck on thinking up a decent name for this: http://drhyde.cvs.sourceforge.net/drhyde/perlmodules/No/lib/No.pm? revision=1.2&view=markup As per my post of a few seconds ago, how about making it a pragma: 'unavailable'? :) Does this do what you want? http://search.cpan.org/~corion/Test-Without-Module-0.09/ Looks like it, thanks :) -- Andy Armstrong, hexten.net
Re: Which Modules Does Your Distro Need?
--- Andy Armstrong <[EMAIL PROTECTED]> wrote: > Can anyone think of anything wrong with the approach of replacing > require with an instrumented version? If the consensus is that that's > a sensible approach I'll play with tidying up the code. You know, that solves an annoying problem I've had with some code which attempts to rebuild a module's source from what's loaded (very handy for debugging). I couldn't figure out how to tell what module another module has loaded, but now it looks like I can :) Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/
Devel::Hide (was Re: Which Modules Does Your Distro Need?)
On 7/31/07, Andy Armstrong <[EMAIL PROTECTED]> wrote: > Slightly tangential but related: is there currently a module that > simplifies simulating the non-availability of selected modules? > > use unavailable qw/LWP::UserAgent/; > use Module::That::Needs::LWP::UserAgent; > >From the synopsis of Devel::Hide: use Devel::Hide qw(Module/ToHide.pm); require Module::ToHide; # fails use Devel::Hide qw(Test::Pod Test::Pod::Coverage); require Test::More; # ok require Test::Pod 1.18; # fails Regards, David
Re: Which Modules Does Your Distro Need?
On 7/31/07, Andy Armstrong <[EMAIL PROTECTED]> wrote: > Can anyone think of anything wrong with the approach of replacing > require with an instrumented version? If the consensus is that that's > a sensible approach I'll play with tidying up the code. This is the approach I thought of. I think you'll need to keep a persistent file of output to capture require calls across each test file and then summarize. (I.e. so you can set it as a command line option to the harness.) Regards, David
Re: Which Modules Does Your Distro Need?
On 31 Jul 2007, at 13:39, David Golden wrote: On 7/31/07, Andy Armstrong <[EMAIL PROTECTED]> wrote: Can anyone think of anything wrong with the approach of replacing require with an instrumented version? If the consensus is that that's a sensible approach I'll play with tidying up the code. This is the approach I thought of. I think you'll need to keep a persistent file of output to capture require calls across each test file and then summarize. (I.e. so you can set it as a command line option to the harness.) Yup, I'm just playing with it now. Seems that it'll do everything we need of it. Lovely :) -- Andy Armstrong, hexten.net
Re: Fixing the damage caused by has_test_pod
A. Pagaltzis wrote: * Andy Lester <[EMAIL PROTECTED]> [2007-07-30 21:41]: On Jul 30, 2007, at 1:58 PM, Eric Wilhelm wrote: Thus, I posit that the quality of the module is generally lower if 'boilerplate.t', 'pod-coverage.t', and 'pod.t' *exist*. Certainly, the intent is to have boilerplate.t NOT get shipped with the module. Disagree. The tarball should contain the full source necessary to recreate everything the author did. Stuff that’s not relevant to end-user installation of the module should simply be kept out of the way. boilerplate.t is obviously a very special case, only useful when creating new files in a distribution, in order to warn forgetful programmers to update these with something less boilerplate-ish. :) Anyhow, I think being able to recreate the entire distribiution may be a good thing, but perhaps not necessarily from a CPAN-distributed tarball. I'd certainly expect that from a source repository checkout, though. - Salve
Re: Fixing the damage caused by has_test_pod
Hi Andy, Andy Armstrong schrieb: On 31 Jul 2007, at 12:27, David Cantrell wrote: Wrong. If AUTHOR_TESTING then surely all tests *must* pass both with and *without* the optional module! I'm still stuck on thinking up a decent name for this: http://drhyde.cvs.sourceforge.net/drhyde/perlmodules/No/lib/No.pm?revision=1.2&view=markup As per my post of a few seconds ago, how about making it a pragma: 'unavailable'? :) Does this do what you want? http://search.cpan.org/~corion/Test-Without-Module-0.09/ Steffen
Re: Fixing the damage caused by has_test_pod
Steffen Mueller wrote: Does this do what you want? http://search.cpan.org/~corion/Test-Without-Module-0.09/ Doesn't do what I want, as I want to be able to "infect" any perl processes that might get exec()ed by the one from which I've hidden Some::Module. -- David Cantrell
Re: Summarizing the pod tests thread
Michael G Schwern wrote: Long threads scare me and, I'm sure, other people. Can people sum up what useful things have been said in that long thread? Skimming it so far it seems to be: * POD tests should be run by the user. * POD tests should not be run by the user. Anything productive come out of it? No. In fact, this argument is ludicrus, and here's why: 1. We're playing the Open Source Development game here, of which the prime directive is "Many Eyes Make All Bugs Shallow". By denying end-users to partake in this game (by not giving them hints of something wrong) we're not only denying them a chance to inform themselves of the state of the software they're about to use, but we're also missing out on feedback that may lead to the improvement of the software, dependencies, distribution system, distribution format, tests, documentation or ANY other thing which may be the source of a failed test. 2. We need to keep ALL the following "users" happy: - "This is MY module"-developers - "I'd like to improve this module somehow"-developers - "I have free time and like to see if this works"-developers - "I've agreed to test this for CPANTS"-developers - "I need to make a RPM/DEB package for distribution"-developers - "I just need this for my job, please don't bother me"-developers - "I just want to install this, and don't know Perl"-sysadmins So this shouldn't be a "To test POD or not to test POD" question. This should rather be a question of what the Best Practices for CPAN module authoring are, and how we should share these with everyone. Perhaps the arguments should just be summarized on the wiki. http://perl-qa.hexten.net/wiki/index.php/Testing_POD I'd rather see a "Best Practices for testing" wiki page, followed up by arguments, and reflected by the tests and modules produces by skeleton module-generating tools like Module::Starter, ExtUtils::ModuleMaker. http://perl-qa.hexten.net/wiki/index.php/Best_Practices_for_Testing - Salve
Re: Fixing the damage caused by has_test_pod
Adriano Ferreira wrote: PERL5LIB=MDevel::Hide=Module::ToHide process_that_fork_other_processes.pl works as far as I know. PERL5OPT. Yes, that will do the job, but I want it to be built in to the module-hiding module rather than something extra that the user needs to do himself. Would you accept a patch to make that an option? -- David Cantrell
Re: Which Modules Does Your Distro Need?
In article <[EMAIL PROTECTED]>, Ovid <[EMAIL PROTECTED]> wrote: > Imagine this: > > ./Build testdeps > > That would be a custom action which would load a special Test::More > wrapper which, at the end of each test program, cache %INC. I named mine Test::Prereq. :)
Re: Summarizing the pod tests thread
On Tuesday 31 July 2007 10:44:07 Salve J Nilsen wrote: > In fact, this argument is ludicrus, and here's why: > > 1. We're playing the Open Source Development game here, of which the prime > directive is "Many Eyes Make All Bugs Shallow". By denying end-users to > partake in this game (by not giving them hints of something wrong) we're > not only denying them a chance to inform themselves of the state of the > software they're about to use, but we're also missing out on feedback that > may lead to the improvement of the software, dependencies, distribution > system, distribution format, tests, documentation or ANY other thing which > may be the source of a failed test. Please explain to me, in detail sufficient for a three year old, precisely how: 1) POD can possibly behave any differently on my machine versus anyone else's machine, being non-executed text and not executed code 2) "Failures" in POD have any bearing on the use of the distribution, especially if an end-user has installed the distribution merely as a dependency and not as a developer 3) False negatives are EVER acceptable in tests -- c
Re: Fixing the damage caused by has_test_pod
On 7/31/07, David Cantrell <[EMAIL PROTECTED]> wrote: > Steffen Mueller wrote: > > > Does this do what you want? > > http://search.cpan.org/~corion/Test-Without-Module-0.09/ > > Doesn't do what I want, as I want to be able to "infect" any perl > processes that might get exec()ed by the one from which I've hidden > Some::Module. PERL5LIB=MDevel::Hide=Module::ToHide process_that_fork_other_processes.pl works as far as I know. > -- > David Cantrell >
Re: Which Modules Does Your Distro Need?
Hi Ovid, Ovid schrieb: --- Andy Armstrong <[EMAIL PROTECTED]> wrote: Can anyone think of anything wrong with the approach of replacing require with an instrumented version? If the consensus is that that's a sensible approach I'll play with tidying up the code. You know, that solves an annoying problem I've had with some code which attempts to rebuild a module's source from what's loaded (very handy for debugging). I couldn't figure out how to tell what module another module has loaded, but now it looks like I can :) Seems like I'm getting into the habit of recommending modules on perl-qa :) Perhaps http://search.cpan.org/~chromatic/Devel-TraceUse-1.00/ helps here? I think Eric Wilhelm was hacking on something which extends on this concept. You know, the MOdule::ScanDeps compile-time and execution-time dependency scanning could use a make-over. Best regards, Steffen
Re: Fixing the damage caused by has_test_pod
# from David Cantrell # on Tuesday 31 July 2007 04:27 am: >> No. If AUTHOR_TESTING, fail miserably unless the pod and coverage >> both 1. gets tested and 2. passes. That means the Test::Pod::* >> module in question must load. > >Wrong. If AUTHOR_TESTING then surely all tests *must* pass both with >and *without* the optional module! What good does it do to skip the test if AUTHOR_TESTING? That just gets us back into the same situation as the current one where the author didn't have Test::Pod and pod which fails the tests got shipped. To accomplish author testing, you must actually *run the tests*. You can't do that if you don't have the module available. If the tests didn't run, well ... it's untested. If untested pod ships with the possibility of the tests actually running (and quite likely failing) on the install system, the error appears at the wrong end. The advocacy of "check your pod" and "cover your pod" is fine, but please apply the clue-stick *before* the dist gets shipped to CPAN. --Eric -- The opinions expressed in this e-mail were randomly generated by the computer and do not necessarily reflect the views of its owner. --Management --- http://scratchcomputing.com ---
Re: Summarizing the pod tests thread
On 7/31/07, chromatic <[EMAIL PROTECTED]> wrote:
> Please explain to me, in detail sufficient for a three year old, precisely
> how:
>
> 1) POD can possibly behave any differently on my machine versus anyone else's
> machine, being non-executed text and not executed code
What version of Pod::Simple do you have? What version does everyone
else have? Will POD parsed on your machine always parse the same
everywhere?
"Should you care?" is really your second question:
> 2) "Failures" in POD have any bearing on the use of the distribution,
> especially if an end-user has installed the distribution merely as a
> dependency and not as a developer
Personally, I think it has little bearing, but not zero -- but your
point is a good one.
> 3) False negatives are EVER acceptable in tests
Which way do you mean this?
False negative meaning a the test is negative for the unwanted
condition when it should be true? (E.g. pass tests when the module has
a bug or you test negative for cancer but really have a tumor; etc)
Or false negative meaning the result shows an undesirable ("negative")
outcome when the real situation is a desirable one? (E.g. Fail module
tests when it shouldn't; test positive for cancer when you actually
don't have it; etc.)
Either way, I recall that it's usually hard to minimize both "false
negative" and "false positive" errors in certain types of testing --
you often have to trade off one for the other.
David
Re: Summarizing the pod tests thread
On Tuesday 31 July 2007 12:35:06 David Golden wrote: > On 7/31/07, chromatic <[EMAIL PROTECTED]> wrote: > > 1) POD can possibly behave any differently on my machine versus anyone > > else's machine, being non-executed text and not executed code > What version of Pod::Simple do you have? What version does everyone > else have? Will POD parsed on your machine always parse the same > everywhere? POD being a mostly-human readable markup language, the possible problem is that the POD renders incorrectly. In this case, the documentation is still available, whether by reading around code incorrectly appearing in the documentation or reading the documentation directly, not through a POD translator/transliterator. This is not a functional failure; it has nothing to do with whether the *code* will behave correctly on a user's system. > "Should you care?" is really your second question: > > 2) "Failures" in POD have any bearing on the use of the distribution, > > especially if an end-user has installed the distribution merely as a > > dependency and not as a developer > > Personally, I think it has little bearing, but not zero -- but your > point is a good one. What does it have to do with the question "Will this code work on a user's system?" > > 3) False negatives are EVER acceptable in tests > > Which way do you mean this? Reporting that a distribution may not work on a user's system (by having failing tests) is a false negative when a failing POD checking test has nothing to do with the functional behavior of the distribution. We might as well make installers also run the disttest action alongside the test action, for all the good this will do them. -- c
Re: Summarizing the pod tests thread
chromatic wrote: > On Tuesday 31 July 2007 12:35:06 David Golden wrote: > >> On 7/31/07, chromatic <[EMAIL PROTECTED]> wrote: > >>> 1) POD can possibly behave any differently on my machine versus anyone >>> else's machine, being non-executed text and not executed code > >> What version of Pod::Simple do you have? What version does everyone >> else have? Will POD parsed on your machine always parse the same >> everywhere? > > POD being a mostly-human readable markup language, the possible problem is > that the POD renders incorrectly. In this case, the documentation is still > available, whether by reading around code incorrectly appearing in the > documentation or reading the documentation directly, not through a POD > translator/transliterator. > > This is not a functional failure; it has nothing to do with whether the > *code* > will behave correctly on a user's system. Unless that render failure also means that --help/pod2usage in a script doesn't work. Then it's a functional failure. signature.asc Description: OpenPGP digital signature
Re: Summarizing the pod tests thread
On Tuesday 31 July 2007 12:54:05 Christopher H. Laco wrote: > > This is not a functional failure; it has nothing to do with whether the > > *code* will behave correctly on a user's system. > Unless that render failure also means that --help/pod2usage in a script > doesn't work. Then it's a functional failure. Such a distribution needs a function test for that behavior then. Test::POD* will not cover it. -- c
Re: Which Modules Does Your Distro Need?
# from Andy Armstrong
# on Tuesday 31 July 2007 05:55 am:
>> This is the approach I thought of. I think you'll need to keep a
>> persistent file of output to capture require calls across each test
>> file and then summarize. (I.e. so you can set it as a command line
>> option to the harness.)
>
>Yup, I'm just playing with it now. Seems that it'll do everything we
>need of it. Lovely :)
Yep. My thinking for Devel::TraceDeps was that it would be enabled with
a PERL5OPT=-MDevel::TraceDeps=somedir so-as to catch the subprocesses.
It would need to basically create a subpath-keyed (YAML?) file in
somedir for every $0 that hits import.
The post-processing then allows you to filter-out the "test that runs a
program" while still allowing you to process data from a "test that
runs other tests".
The trouble with basically [EMAIL PROTECTED] is that it is always
first-come first-serve and you never get a poke for the latter
require()s.
use Foo; # uses warnings, strict, Carp
use warnings; # got it already, no callback
use strict; # ditto
use Carp; # ditto
With END {} looking at %INC, you're only seeing "everything that
everyone used anywhere."
And to get complete coverage (e.g. for PAR to work), we also need do()
for things like 'utf8_heavy.pl does unicore/lib/SpacePer.pl' and etc.
Plus, of course, the .dll/.so/.dylibs.
Also note that "-d" tends to segfault in Wx usage (at window creation or
thereabouts) and maybe elsewhere.
Would be nice to note $^S as well as far as caring whether an
eval()-wrapped load succeeds or fails.
http://scratchcomputing.com/svn/Devel-TraceDeps/trunk
--Eric
--
"I've often gotten the feeling that the only people who have learned
from computer assisted instruction are the authors."
--Ben Schneiderman
---
http://scratchcomputing.com
---
Re: Summarizing the pod tests thread
On 7/31/07, chromatic <[EMAIL PROTECTED]> wrote: > This is not a functional failure; it has nothing to do with whether the *code* > will behave correctly on a user's system. That's not the question you posed. You asked why POD would "behave differently". Ask a silly question, get a silly answer. I *agree* with you that Pod and Pod coverage testing should die -- and have held that view for a long time. So instead of contributing to more pedantry, here's my summation of potential practical steps: * CPANTS -- eliminating or penalizing Kwalitee for pod/pod-coverage tests would seem to be up to Thomas Klausner; I don't think I've seen him weigh in on this recently. So people need to lobby him or fork the project. * Module::Starter -- Andy wants these tests run by default and he doesn't seem to be swayed; So people need to fork this and start advocating for an alternative. Ditto other module generators. (And mea culpa for my own.) * Alternatives -- Eric Wilhelm suggests putting these tests in at/ and running them with prove. Likewise, Module::Build and ExtUtils::MakeMaker could have targets added that run tests in an "at/" directory as another way to encourage the practice. (Module authors who want such targets can upgrade without ever affecting end-users) Lobby Ken Williams and Schwern. * Advocacy -- As alternatives are developed, people should start writing articles for perl.com, Perl Monks and should start giving local .pm group talks, etc. demonstrating and advocating those approaches. Regards, David
Re: Summarizing the pod tests thread
On Jul 31, 2007, at 3:43 PM, David Golden wrote: * Module::Starter -- Andy wants these tests run by default and he doesn't seem to be swayed; So people need to fork this and start advocating for an alternative. Or maybe it's just not that big a deal. -- Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance
Re: Summarizing the pod tests thread
On Tue, Jul 31, 2007 at 04:43:08PM -0400, David Golden wrote: > * CPANTS -- eliminating or penalizing Kwalitee for pod/pod-coverage > tests would seem to be up to Thomas Klausner; I don't think I've seen > him weigh in on this recently. So people need to lobby him or fork > the project. He's somewhere in Outer Mongolia, and won't be back for about 2 weeks: http://use.perl.org/~domm/journal/33801 http://use.perl.org/~domm/journal/33878 He then might be rather busy until the end of YAPC::EU Don't be hasty with the f word. Nicholas Clark
STOP! Its just not that big a deal! (was Re: Summarizing the pod tests thread)
Andy Lester wrote: > > On Jul 31, 2007, at 3:43 PM, David Golden wrote: > >> * Module::Starter -- Andy wants these tests run by default and he >> doesn't seem to be swayed; So people need to fork this and start >> advocating for an alternative. > > Or maybe it's just not that big a deal. Once again, thank you Andy for stabbing at the heart of the matter. The two sides have butted heads for a while, its clear they're not going to budge. Further restating of the contrary positions isn't going to do anything but produce anger and claw marks. Back off. One side can go and paint their bikeshed blue and one side can go and paint their bikeshed red. And they can write all about the virtues of their color scheme they want on the wiki so that others can decide for themselves. I don't want to hear any more about it on this list for at least a week. This issue is not worth the bad blood and energy. There's been, oh let's say, 70 or 80 posts over the span of half a week consuming pretty much all our energy. Only SJN has touched the wiki. And hey, how's that Test::Harness 3.0 testing coming?
Re: Module::Starter's pod tests
On Jul 31, 2007, at 4:08 PM, Eric Wilhelm wrote: With the current situation[1], unaware users won't realize that they are shipping failing tests[2]. [1] Unless you add Test::Pod and Test::Pod::Coverage as prereqs to Module::Starter. Aha! Prereqs to Module::Starter ITSELF. Got it. I hadn't thought of that situation. That would be a bummer. -- Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance
Re: STOP! Its just not that big a deal! (was Re: Summarizing the pod tests thread)
In fact, when a topic degrades to this level on the list go yell about it on IRC. Email is the worst possible form of communication for this, its optimized for big, detached speeches. At least IRC is more like a real time conversation and the five people arguing won't scare off the hundred other subscribed to this list. irc://irc.perl.org/#perl-qa -- When it gets to the Shlomi level of "I don't like this, I'm going to fork it!" it's time to back off.
Re: Fixing the damage caused by has_test_pod
David Golden wrote: CPAN is a core strength of Perl because it makes developer's lives easier by not reinventing the wheel. This is not necesarily the case. Every language has ways of providing additional libraries that make developers lives easier by not reinventing the wheel. CPAN is a core strength of Perl for the same reason .deb packages are a core strength of debian. Not only do they fulfil the "not reinventing the wheel" role, but they do so in a way that makes the procurement and installation of those package dramatically easier than the alternatives. And the ease of procurement of a piece of software is twice as important to success than the quality of the piece of software. The person installing something from CPAN has ONLY two reasonable outcomes. 1. The module or application, and all the dependencies, install correctly. 2. The installation process fails, and the end-user cannot use that software. Additionally, dealing with recursion-based issues is paramount. One thing I keep repeating over and over is that for MOST cases, nobody will ever use the modules they are installing, nobody understands what they do, and nobody will ever read the documentation. To use two examples here, Jifty developers install 50-100 packages that are Jifty dependencies, and primary use exactly two. Jifty and Jifty::DBI. That's a couple of percent at best. For a more extreme example, JSAN users often come from the Ruby of Python worlds. They aren't even Perl coders AT ALL. They just want to install the JSAN client. It is installation that has made CPAN successful, and the prioritisation of installation over all other use cases is paramount. > More usage of modules means more installs and points of dependency failure. Modules which use lots of dependencies that might fail to install will themselves fail to be installed more often and may be passed over and the world goes on. So let's not start a "dependencies are bad" bandwagon -- let the market figure out what dependencies are worth the risk. A market is a bad analogy for CPAN. Markets require choice, and arbitrary layers of recursive dependencies makes for very little choice. In my view, testing modules that make modules faster to develop and or lead to better diagnostic output when failures do happen can make author's lives easier, which leads to more development, which is a good thing. If that means some extra installs, well, OK. Balance the benefits to you against the likelihood that some people won't put up with the extra dependencies. Benefit: Improved quality of error reports for factors that do not impact the functional operation of a module. Cost: A percentage of your users can cannot use the module at all, give up in disgust and avoid it in the future. I see this balance as undermining the core functionality of CPAN for the benefit of the author in catching minor nits. It escalates a minor non-critical issue for an author into a critical issue for installers. It's selfish and wrong. Dealing with errors in non-functional areas is what we have CPAN Testers for. Functionality and installation always comes first. Adam K
Re: Fixing the damage caused by has_test_pod
chromatic wrote: On Monday 30 July 2007 10:01:25 Eric Wilhelm wrote: I think the important bit is that `make test` only runs tests which verify the module's functionality. ... and if my POD coverage somehow mysteriously changed between the time I bundled a new dist and some user downloaded that bundle from the CPAN, there are bigger problems with that downloaded dist than failing POD coverage tests. Or alternatively, there might have been a new release of Test::Pod with a slight change in behaviour that causes failures. Adam K
Re: Fixing the damage caused by has_test_pod
Salve J Nilsen wrote: Anyhow, I think being able to recreate the entire distribiution may be a good thing, but perhaps not necessarily from a CPAN-distributed tarball. I'd certainly expect that from a source repository checkout, though. CPAN-distributed tarballs are the primary source of the source code, any tests that the author feels should be run as part of release testing or automated testing should be included in the CPAN distribution. Repositories have this nasty habit of being private, out of date, on servers that die, not a version control you like, not a product you can legally use, or are not used at all by the author. From the point of view of continuity, only the CPAN exists. There are repositories. Adam K
Re: Summarizing the pod tests thread
David Golden wrote: * Module::Starter -- Andy wants these tests run by default and he doesn't seem to be swayed; So people need to fork this and start advocating for an alternative. Ditto other module generators. (And mea culpa for my own.) You don't need to fork Module::Starter. You simply need to use ExtUtils::ModuleMaker, which by default does *not* include these tests in its initial setup of the directories/files needed for a CPAN-ready module, but which does offer them as options. jimk
Re: Fixing the damage caused by has_test_pod
Christopher H. Laco wrote: I say that's fine. If it fails and you can't install it, then don't. Arguments about whether the tests should or shouldn't be run [or included at all] is irrelevant. Tests failed. Don't install. File RT. Filing RT requires understand what failed and why. This requires some level of skill to diagnose problems. You cannot assume that people using the CPAN shell actually know Perl. If the author made a choice to have them run always and piss people off, or restrict the user base, then that's the authors prerogative. Now, I won't argue that it would be better if the author put in a skip check for TEST_AUTHOR or other variants, but that's up to the author and not something that should be enforced or needs to be regulated. I'm not proposing we regulate of enforce it. I'm simply proposing that we discourage it via the Kwalitee mechanism, since it was in part the Kwalitee mechanism that created this problem in the first place. > If I ship a module and test coverage fails, that's still a bug. The end user still has no documentation on what the hell the 'naked' method is supposed to do. In 95%+ of cases, the end user will never ever read the documentation, and never use the module they are installing, and any problems with either the POD syntax or the thoroughness of the API will be utterly moot. Because 95% of modules are installed due to recursive dependencies. Sure, Test::Pod::Coverage could fail for some reason other than it's main purpose of checking coverage and finding a naked method, but so then can Test::More and any other test module. This is exactly the reason that we discourage dependencies, and that wherever it is possible to do so, people should only use testing dependencies for cases where the tests are used to determine that the module is functionally correct on the host. Adam K
Re: Fixing the damage caused by has_test_pod
Eric Wilhelm wrote: I'm thinking the installer should have the option to scan the tests and skip any that use qr/Test::Pod(?:::Coverage)?/. This wouldn't work. It would just mean that the installer would skip ALL the tests for Test::Pod :) There's many cases when testing the Test:: modules you want to run tests that you might otherwise want to skip for regular modules. Adam K
Re: Summarizing the pod tests thread
David Golden wrote: * Alternatives -- Eric Wilhelm suggests putting these tests in at/ and running them with prove. Likewise, Module::Build and ExtUtils::MakeMaker could have targets added that run tests in an "at/" directory as another way to encourage the practice. (Module authors who want such targets can upgrade without ever affecting end-users) Lobby Ken Williams and Schwern. A small nit, if we end up converging around the use of a relatively standard directory name, can we at least use something relatively self-evident? For example "author/"? Adam K
Re: Summarizing the pod tests thread
chromatic said:
>
> Please explain to me, in detail sufficient for a three year old,
> precisely how:
>
> 1) POD can possibly behave any differently on my machine versus anyone
> else's machine, being non-executed text and not executed code
I'll assume the three-year-old is a particularly bright kid. :)
Let's say Joe Sysadmin wants to install the author's (a.k.a. "your")
module Useful::Example, and during the test-phase one of the POD tests
fail. There could be any number of reasons for this failure, including
but not limited to...
1) Joe's POD-analyzing module has a different/early/buggy notion of how
the POD syntax is supposed to be. This can be fixed by you in several
possible ways:
- The author ("you") can require a newer/better version of the
POD-analyzing module to be installed (So you're not keeping up with the
development on that front? Good thing your users can give you hints
about changes!)
- If there's no newer, you can contact the author of the POD-analyzer
and see if he can update his software (So your module is helping other
projects by finding edge cases or inconsistencies? Nice! Good thing your
users update the POD analyzer software quicker than you.)
2) The module actually has broken POD! POD converters scream in pain
everywhere! The generated HTML on search.cpan.org will be wrong! Google
will index the mistakes! Users searching for the functionality the
module supplies won't find it because the Google indexed the broken
documentation! OHNOEZ! What to do?
- Fix it, and release a new version.
3) Joe's POD-analyzing module is buggy (as in #1), but Joe cannot
upgrade that module for some reason. What to do?
- Make sure your POD tests ONLY run when a good version of the
POD-analyzer is installed, otherwise SKIP.
4) There's something wrong with your test. What to do?
- Fix it and release.
5) The POD specification is wrong. Or unclear about your use case. What
to do?
- Make the relevant author(s) aware of this bug, and tell them if you
think it's a bug in the documentation, the specification or something
else.
6) There's no bug!
- Yeah, rght. ;-)
So it's not the POD that behaves differently, but /the surrounding
modules interpreting the POD/ that behave differently. But even if the
author's influence over Joe Sysadmin's installation is rather limited,
it's still the author's duty to make sure Joe can know (as best as
possible) that the module is in good shape. Now if Joe doesn't care
enough about Useful::Module to send a bug report (or send a failure
report to CPANTS), one can still hope someone else does it.
But you can be sure that if you _don't_ let the users run the tests,
then there's an even smaller chance you'll get that bugreport.
The short version
- All code has bugs - and if it has none now, someone will find (or
create, or define) them soon.
- The tests are code.
- The modules used by your tests are code.
- Build.PL and Makefile.PL is code
- The requirements sections in B.PL and M.PL are part of this code.
- It's a bug when build_requires for a module should say "0.42", but
instead says "0.41"
- Even your data and documentation (in whatever format you chose to
keep it) touches some sort of code - to parse it, convert it, analyze
it, check it or even to specify how it's supposed to look.
- In fact, we might as well treat everything as code, since we tend to
treat code somewhat respectfully (at least compared to how we treat
documentation.)
- So when we check if some documentation syntax is correct, we're
actually checking if some "code" is correct - not much unlike "use
warnings;" enables checks on the developer's use of Perl.
- In other words, "turning off" syntax checks (like Pod::Checker) in
order to spare the user of warnings/failures is much like turning off
"use warnings;" because the warnings make the developer think something
is wrong.
The super-short version :)
Turning off syntax checking of your POD is comparable to not turning on
warnings in your code. Now would you publish code developed without "use
warnings;"?
> 2) "Failures" in POD have any bearing on the use of the distribution,
> especially if an end-user has installed the distribution merely as a
> dependency and not as a developer
If that was the only kind of end-user, I'd agree - the bearing on usage
would be negligent. But it's not the only kind of end-user. I'll even be
brash and postulate that the non-developer end-user is one of the LESS
important ones, since he most likely won't interested in or capable of
taking part in the development community for that module.
But that doesn't mean the author should make it difficult to install
their modules cleanly... :)
> 3) False negatives are EVER acceptable in tests
If there's a false negative in a test, that's still a sign of a bug
somewhere. Maybe in the test itself? Or the dependencies? Or the build
system? Or some third-party module? Wherever it is, it's the author's
job to fix it,
Re: Summarizing the pod tests thread
On Tuesday 31 July 2007 20:25:15 Salve J. Nilsen wrote: > Turning off syntax checking of your POD is comparable to not turning on > warnings in your code. Now would you publish code developed without "use > warnings;"? Now that's just silly. I really have nothing more to say in this thread. Wow. -- c
Re: Summarizing the pod tests thread
On Tue, July 31, 2007 9:56 pm, chromatic wrote: > On Tuesday 31 July 2007 20:25:15 Salve J. Nilsen wrote: >> Turning off syntax checking of your POD is comparable to not turning on >> warnings in your code. Now would you publish code developed without >> "use >> warnings;"? > > Now that's just silly. Is it? Comparable. Capable of being compared. 1913 Webster. > I really have nothing more to say in this thread. Wow. It's one of those threads that I didn't even bother reading till now, since I felt pretty confident that I wasn't going to change my opinion (and that most others would not as well). Pity I broke down.
Re: Summarizing the pod tests thread
# from Yitzchak Scott-Thoennes # on Tuesday 31 July 2007 10:19 pm: >On Tue, July 31, 2007 9:56 pm, chromatic wrote: >> On Tuesday 31 July 2007 20:25:15 Salve J. Nilsen wrote: >>> Turning off syntax checking of your POD is comparable to not >>> turning on warnings in your code. Now would you publish code >>> developed without "use >>> warnings;"? >> >> Now that's just silly. > >Is it? Comparable. Capable of being compared. 1913 Webster. You can compare them, but it is silly. An uninitialized value might strike the user at runtime, so we leave warnings on when we ship. The pod will be unchanged, so we *test it*, and *then* ship it. I like to put my socks on before my shoes, but after my pants. Sometimes I put socks on before my pants, but shoes are always last. --Eric -- Turns out the optimal technique is to put it in reverse and gun it. --Steven Squyres (on challenges in interplanetary robot navigation) --- http://scratchcomputing.com ---
Re: Summarizing the pod tests thread
Salve J. Nilsen wrote:
Let's say Joe Sysadmin wants to install the author's (a.k.a. "your")
module Useful::Example, and during the test-phase one of the POD tests
fail.
Joe Sysadmin doesn't use modules, lets try the following.
"Joe Sysadmin wants to install the JSAN client, because his
HTML/JavaScript guys want to use some of the JavaScript modules. Joe
Sysadmin doesn't know Perl. He does not know what POD is, and has never
heard of CPANTS. He will never need to read the documentation for any
dependencies of the JSAN client."
1) Joe's POD-analyzing module has a different/early/buggy notion of how
the POD syntax is supposed to be. This can be fixed by you in several
possible ways:
Joe Sysadmin runs "sudo cpan --install JSAN". 10,000 lines of text
scrolls down the screen for about 10 minutes. 9 minutes and 8,500 lines
in, the POD tests in a utility module 6 layers of recursive dependencies
up the chain fails.
Installation of that module fails, and as the CPAN recursion unwinds
another 5 modules recursively fail.
The final summary lists 6 modules which did not install. The original
reason is 1,500 lines above this summary, at the top of many many
details about failed tests due to the missing dependencies.
Joe Sysadmin has no idea why the installation failed, he scrolls up
through last 1000 lines of output, before giving up, and just running
"sudo cpan --install JSAN" again. It still fails, with 2,000 lines of
output.
At this point, MOST people that are not Perl developers are utterly
lost. I've seen it several times in the JSAN IRC channels, as quite
competant JavaScript, Python, Ruby and ASP.Net coders come in to ask for
help because their CPAN installation fails.
> - The author ("you") can require a newer/better version of the
> POD-analyzing module to be installed (So you're not keeping up with the
> development on that front? Good thing your users can give you hints
> about changes!)
>
> - If there's no newer, you can contact the author of the POD-analyzer
> and see if he can update his software (So your module is helping other
> projects by finding edge cases or inconsistencies? Nice! Good thing your
> users update the POD analyzer software quicker than you.)
The author has no idea it has failed for the user, because the user does
not know how to report the fault.
Likewise, not only does the user not know HOW to blame the pod analyzer,
but often does not even know what POD is.
2) The module actually has broken POD! POD converters scream in pain
everywhere! The generated HTML on search.cpan.org will be wrong! Google
will index the mistakes! Users searching for the functionality the
module supplies won't find it because the Google indexed the broken
documentation! OHNOEZ! What to do?
- Fix it, and release a new version.
3) Joe's POD-analyzing module is buggy (as in #1), but Joe cannot
upgrade that module for some reason. What to do?
- Make sure your POD tests ONLY run when a good version of the
POD-analyzer is installed, otherwise SKIP.
Unfortunately, what the author THINKS is a good version is actually not.
But the author will not find this out because the end user doesn't
know how to report it.
> But even if the
author's influence over Joe Sysadmin's installation is rather limited,
it's still the author's duty to make sure Joe can know (as best as
possible) that the module is in good shape.
Surely the best way to do this is simply to not have failing tests for
things that aren't related to the actual functionality of the module.
Now if Joe doesn't care
enough about Useful::Module to send a bug report (or send a failure
report to CPANTS), one can still hope someone else does it.
Joe has no what the problem is, and after screwing around for half an
hour, gives up in frustration and just manually unzips the JSAN modules.
But you can be sure that if you _don't_ let the users run the tests,
then there's an even smaller chance you'll get that bugreport.
Again, we have the problem that most people can't diagnose errors in the
installation toolchain well enough to report them.
Which leaves most of the users with a failing install because of a typo
in the POD 6 levels of dependencies up the chain, in exchange for the
small number of cases in which a user catches an error that the
collected CPAN Testers missed.
The super-short version :)
Turning off syntax checking of your POD is comparable to not turning on
warnings in your code. Now would you publish code developed without "use
warnings;"?
Yes. Absolutely.
Every single module I write ships with use warnings disabled in the module.
I've had a number of instances in the past where spurious warnings in a
web application overflowed the log files. Nobody ever knew the warnings
were happening, and there was very little they could do about it anyway.
However, I ALWAYS run all the tests with warnings enabled and where
possible with warnings fatal.
2) "Failures" in PO
