On Wed, Jul 04, 2007 at 10:08:59AM -0400, DAGOLDEN wrote: > On 7/4/07, Barbie <[EMAIL PROTECTED]> wrote: > >Hi David, > > > >> - UNKNOWN results when prereqs are missing are downgraded to NA (just > >> like FAIL reports with missing prereqs already were) > > > >This is wrong. If a distribution has prereqs missing from > >Makefile.PL/Build.PL then it should receive a FAIL report. If the tool > >chain fails to install the prereq, then the report should be abandoned > >and not sent at all. > > I don't think it's wrong -- but we are using the phrase "prereqs are > missing" differently. CPAN::Reporter checks to see if any of the > prereqs specified in the Makefile.PL or Build.PL are missing (i.e. not > installed). This can happen if the tests were run anyway despite > missing prereqs (e.g. "force test").
Okay I see what you mean. In this case the report should be abandoned. If the user is forcing the test regardless, the results are not a fair result. The author has said these prereqs need installing and is basically saying that carrying on regardless is likely to cause errors. > In such cases, CPAN::Reporter > "downgrades" the detected test grade from FAIL to NA -- indicating > that the report results are meaningless. Since NA reports by default > aren't sent to the list or the author, it also avoids sending out such > useless results. I'm a little concerned by this. If you are downgrading to NA so the result doen't get sent, does this mean that CPAN::Reporter never sends a NA report? If so, there are likely to be valuable NA reports that you are throwing away. As I mentioned previously, NA reports have a purpose and ignoring them devalues that. The reports above shouldn't be downgraded, but ignored completely. > In 0.45, UNKNOWN results also benefit from this downgrade behavior. I don't see this as a benefit. The report should be abandoned. > These results do get sent out, unlike NA reports CPAN::YACSmoke uses UNGRADED and ABORTED internally, which results in the report not getting sent. Anything else gets sent. If you are parsing the output to see whether the grade is bogus, that's fine, but please don't corrupt the value of NA reports. > >NA is reserved for when the distribution doesn't work on a particular > >version of Perl or platform. Adding broken prereqs to the mix is > >confusing and misleading. > > I would disagree on the first part and suggest that the definition be > broadened to include any condition where the test result should just > be ignored. Plenty of tests get run when they shouldn't -- NA should > be the grade for all of those cases. Originally the NA was used to highlight a distribution didn't work on a specific platform. This was extended to a version of Perl when 5.8 started causing problems for users of 5.6 and earlier. It has a specific meaning to users now. It doesn't mean a test can be ignored. If tests get run when they shouldn't, then abandon the run and/or report as CPAN::YACSmoke does with ABORTED. Polluting the NA grade is going to lead to just as much confusion as with sending these reports as FAIL or UNKNOWN. > Additionally, there is no well-defined way that I've ever seen > documented for a distribution to specify what platform it runs on or > to gracefully bail out if it's on the wrong platform. See File::FDPasser, it's the example I use in my Preparing For CPAN talk. There are others, but this seems to handle it better than most. This was the reason I was prompted to patch CPANPLUS for "OS Unsupported" or "No support for OS", so that authors could die appropriately if they knew their module wouldn't work with that OS. > Some people > avoid creating a Makefile, which may work or may get picked up as a > Makefile.PL failure. Some people bail out of test scripts (or just > let tests fail). Detecting a "platform" NA is thus a poorly defined > heuristic. If they bail out of a test with the string "OS Unsupported", then that is a valid NA report. Anything else is unknown and should result in the report being abandoned, or possibly sending only to the author so they can at least understand why it bailed. I feel quite strongly about this, as I know companies who take the time to look at what perls/platforms a distribution works on, and adding misleading NA reports or not sending them altogether doesn't help those people. Barbie. -- Birmingham Perl Mongers - http://birmingham.pm.org Miss Barbell Productions - http://www.missbarbell.co.uk
