On 5 Jul 2002, Andreas Marcel Riechert wrote:
> Brian and cpantesters,
>
> Personally, in the beginning I had problems with "na" vs "fail".
> Therefore my first draft on grades in germanic english. Please
> proofread and correct:
This looks very good ....
>
> ----------------------------------------------------------------
> x.) Grades
> There are four different grades you can report.
>
> pass
> The built will go fine and all supplied tests passed.
> If you get some non fatal warnings you should cc the
> output of the command to the author as well.
>
> fail
> Some or all tests failed. If you get a fatal error during
> the built process ("perl Makefile.PL", "make") you should
> grade the distribution as "fail", too. The only exception
> is when "na" applies. In the case of a fail its necessary
> to supply the output of the failed command in your report
> as well.
> You should cc your report to the author.
I'd also suggest that a fail grade not be given if the package
requires a non-standard program/library/whatever (eg, MySQL,
Apache) which your system doesn't have. For such cases it's
probably better not to report anything, as (as you mention below)
"na" isn't applicable, either.
> na
> You should only give a na if the distribution is not supposed
> to run on your specific platform (OS). Therefore e.g Macintosh
> or Windows specific modules will get a na on Linux.
> You won't give a na if a Module requires MySQL but you haven't
> installed it.
>
> unknown
> You judge a distribution as unknown if the distribution did not
> include any tests. It may also approbriate to give a unknown if
> the tests output does not show any "ok" or other information which
> let you decide wether the test passed or not.
> You should cc to the author noting something like "No tests defined."
> or "No test defined. It would be nice if you could supply at least
> some load test as supplied by h2xs".
This is a bit of a grey area for packages using external
programs, for which writing a "real" test suite can be quite
involved. If "real" tests aren't supplied, it may be better not
to supply any tests at all, as a simple load test may lull the
user into thinking things are fine. Furthermore, for some such
modules a simple load test may not even be possible, as they may
require the environment of the external program. So I wouldn't
bother suggesting to the author about supplying a load test, as
perhaps he/she has thought about it and decided against it.
Also, some packages in general supply some non-standard tests
that must be run and evaluated manually - although it's a bit of
a judgement call, I think such tests can reasonably lead to a
"pass" grade.
>
> ------------------------------------------------------------------
>
> Missing stuff:
> FAIL -- The "Prerequisite not set in Makefile.PL" Problem
> Should I give a fail or not if I get a missing module warning/error
> during "make test", but not during "perl Makefile.PL" ?
>
> IMHO, if we cannot decide on this issue this problem still should be
> mentioned.
I don't think we can come to a consensus on this ... CPANPLUS is
strict about failing all dependencies missing in a PREREQ_PM,
whereas manual testers cannot be - they may on chance have a
missing dependency already installed, or they may install a
dependency from some indication other than a PREREQ_PM (eg,
Makefile.PL dialogue, reading an INSTALL file, etc.). So as you
say, probably just mention this issue, and leave the decision to
the tester.
best regards,
randy
