Note -- I'm dropping Ask and David C from the CC list -- they can read it on CPAN testers if they wish.
Overall, I see the logic in what you suggest. Please document it more fully on the wiki. When I was writing CPAN::Reporter, it was extremely hard to find any documentation. At best, I was reading source for CPANPLUS and trying to understand what it was doing. On 7/4/07, Barbie <[EMAIL PROTECTED]> wrote:
Okay I see what you mean. In this case the report should be abandoned. If the user is forcing the test regardless, the results are not a fair result. The author has said these prereqs need installing and is basically saying that carrying on regardless is likely to cause errors.
Agreed.
> In such cases, CPAN::Reporter > "downgrades" the detected test grade from FAIL to NA -- indicating > that the report results are meaningless. Since NA reports by default > aren't sent to the list or the author, it also avoids sending out such > useless results. I'm a little concerned by this. If you are downgrading to NA so the result doen't get sent, does this mean that CPAN::Reporter never sends a NA report? If so, there are likely to be valuable NA reports that you are throwing away. As I mentioned previously, NA reports have a purpose and ignoring them devalues that. The reports above shouldn't be downgraded, but ignored completely.
By default CPAN::Reporter throws away NA reports. I couldn't find the definition of NA anywhere and questions on perl-qa about unsupported platforms suggested that NA results shouldn't be sent.
> Additionally, there is no well-defined way that I've ever seen > documented for a distribution to specify what platform it runs on or > to gracefully bail out if it's on the wrong platform. See File::FDPasser, it's the example I use in my Preparing For CPAN talk. There are others, but this seems to handle it better than most. This was the reason I was prompted to patch CPANPLUS for "OS Unsupported" or "No support for OS", so that authors could die appropriately if they knew their module wouldn't work with that OS.
This needs to be much better documented -- e.g. in EU::MakeMaker and Module::Build, probably in the PAUSE FAQ, etc. If it works today, it's probably because of cargo cult repetition of what someone saw in some other module.
If they bail out of a test with the string "OS Unsupported", then that is a valid NA report. Anything else is unknown and should result in the report being abandoned, or possibly sending only to the author so they can at least understand why it bailed.
So is "UNKNOWN" really unknown or "NOTESTS"? To me, UNKNOWN is supposed to be exactly that -- an unknown result.
I feel quite strongly about this, as I know companies who take the time to look at what perls/platforms a distribution works on, and adding misleading NA reports or not sending them altogether doesn't help those people.
Overall, I get the sense that you (and/or perhaps the original CPAN Testers creators) intended to have "NA" and "UNKNOWN" have very specific meanings that differ from what those terms mean colloquially. That's unfortunate. More generally, I think it's a design flaw to not have a grade to capture things that don't fall cleanly into one or another category. Too much of what happens in determining grades is heuristics -- just discarding things that don't fit make the whole system brittle. I'll adjust CPAN::Reporter behavior on prereqs and NA in a future version, but I'd like to discuss a common and comprehensive approach for the other issues and see if we can't come up with something that is more robust and better defined and documented. (Probably on the wiki -- if that's to be the "canonical" documentation for things.) Regards, David
