Salve J. Nilsen wrote:
Let's say Joe Sysadmin wants to install the author's (a.k.a. "your")
module Useful::Example, and during the test-phase one of the POD tests
fail.
Joe Sysadmin doesn't use modules, lets try the following.
"Joe Sysadmin wants to install the JSAN client, because his
HTML/JavaScript guys want to use some of the JavaScript modules. Joe
Sysadmin doesn't know Perl. He does not know what POD is, and has never
heard of CPANTS. He will never need to read the documentation for any
dependencies of the JSAN client."
1) Joe's POD-analyzing module has a different/early/buggy notion of how
the POD syntax is supposed to be. This can be fixed by you in several
possible ways:
Joe Sysadmin runs "sudo cpan --install JSAN". 10,000 lines of text
scrolls down the screen for about 10 minutes. 9 minutes and 8,500 lines
in, the POD tests in a utility module 6 layers of recursive dependencies
up the chain fails.
Installation of that module fails, and as the CPAN recursion unwinds
another 5 modules recursively fail.
The final summary lists 6 modules which did not install. The original
reason is 1,500 lines above this summary, at the top of many many
details about failed tests due to the missing dependencies.
Joe Sysadmin has no idea why the installation failed, he scrolls up
through last 1000 lines of output, before giving up, and just running
"sudo cpan --install JSAN" again. It still fails, with 2,000 lines of
output.
At this point, MOST people that are not Perl developers are utterly
lost. I've seen it several times in the JSAN IRC channels, as quite
competant JavaScript, Python, Ruby and ASP.Net coders come in to ask for
help because their CPAN installation fails.
> - The author ("you") can require a newer/better version of the
> POD-analyzing module to be installed (So you're not keeping up with the
> development on that front? Good thing your users can give you hints
> about changes!)
>
> - If there's no newer, you can contact the author of the POD-analyzer
> and see if he can update his software (So your module is helping other
> projects by finding edge cases or inconsistencies? Nice! Good thing your
> users update the POD analyzer software quicker than you.)
The author has no idea it has failed for the user, because the user does
not know how to report the fault.
Likewise, not only does the user not know HOW to blame the pod analyzer,
but often does not even know what POD is.
2) The module actually has broken POD! POD converters scream in pain
everywhere! The generated HTML on search.cpan.org will be wrong! Google
will index the mistakes! Users searching for the functionality the
module supplies won't find it because the Google indexed the broken
documentation! OHNOEZ! What to do?
- Fix it, and release a new version.
3) Joe's POD-analyzing module is buggy (as in #1), but Joe cannot
upgrade that module for some reason. What to do?
- Make sure your POD tests ONLY run when a good version of the
POD-analyzer is installed, otherwise SKIP.
Unfortunately, what the author THINKS is a good version is actually not.
But the author will not find this out because the end user doesn't
know how to report it.
> But even if the
author's influence over Joe Sysadmin's installation is rather limited,
it's still the author's duty to make sure Joe can know (as best as
possible) that the module is in good shape.
Surely the best way to do this is simply to not have failing tests for
things that aren't related to the actual functionality of the module.
Now if Joe doesn't care
enough about Useful::Module to send a bug report (or send a failure
report to CPANTS), one can still hope someone else does it.
Joe has no what the problem is, and after screwing around for half an
hour, gives up in frustration and just manually unzips the JSAN modules.
But you can be sure that if you _don't_ let the users run the tests,
then there's an even smaller chance you'll get that bugreport.
Again, we have the problem that most people can't diagnose errors in the
installation toolchain well enough to report them.
Which leaves most of the users with a failing install because of a typo
in the POD 6 levels of dependencies up the chain, in exchange for the
small number of cases in which a user catches an error that the
collected CPAN Testers missed.
---- The super-short version :) ----
Turning off syntax checking of your POD is comparable to not turning on
warnings in your code. Now would you publish code developed without "use
warnings;"?
Yes. Absolutely.
Every single module I write ships with use warnings disabled in the module.
I've had a number of instances in the past where spurious warnings in a
web application overflowed the log files. Nobody ever knew the warnings
were happening, and there was very little they could do about it anyway.
However, I ALWAYS run all the tests with warnings enabled and where
possible with warnings fatal.
2) "Failures" in POD have any bearing on the use of the distribution,
especially if an end-user has installed the distribution merely as a
dependency and not as a developer
If that was the only kind of end-user, I'd agree - the bearing on usage
would be negligent. But it's not the only kind of end-user. I'll even be
brash and postulate that the non-developer end-user is one of the LESS
important ones, since he most likely won't interested in or capable of
taking part in the development community for that module.
But that doesn't mean the author should make it difficult to install
their modules cleanly... :)
Unfortunately, this is exactly what having end users run the POD (and
similar tests does). It makes it more difficult to install their modules
cleanly.
3) False negatives are EVER acceptable in tests
If there's a false negative in a test, that's still a sign of a bug
somewhere. Maybe in the test itself? Or the dependencies? Or the build
system? Or some third-party module? Wherever it is, it's the author's
job to fix it, but he can't do that unless he first learns about it. :)
So it comes down to this. If the user can't use a module because of a
non-critical bug, and the user can't report that bug to the author
because they don't know how to diagnose the problem, you have a catch-22.
Additionally, it's fallacious to assume that authors are either helpful,
proactive, competant, or even (unfortunately in some cases) alive.
It's extremely common for bugs to sit in RT queues for years, especially
for minor things like a POD nit.
In the mean time, the user still can't use the module, or anything that
depends on it.
The ONLY time that a test fails during installation, it should be for a
reason that is serious enough to justify denying the use of the module
AT ALL from the user.
Because failing tests deny the use of the module for that user.
Adam K
