Re: Request for Comments: Package testing results analysis, result codes
On Sun, Feb 19, 2006 at 10:22:20PM +1100, Adam Kennedy wrote: > > 2. Incompatible packaging. > Packaging unwraps, but missing files for the testing scheme. You may want to split this into a result that contains no test suite at all (UNKNOWN) and one that has missing files according to the MANIFEST. Currently the second only gets flagged to the tester, and is usually only sent to the author, if the tester makes a point of it, or the CPAN testing FAILs. > 5. Installer missing dependency. > For the installer itself. > > 6. Installer waits forever. > Intended to cover the "non-skippable interactive question" This needs to cover misuse of fork() and alarm. Too many distributions assume these work on Windows. They don't. At least not on any Windows platform I've ever used from Win95 to WinXP. They usually occur either in the Makefile.PL or the test suite, rarely in the actual module code. They may also fail on other OSs. Although they could potentially be covered in point 5. > 12. System is incompatible with the package. > Linux::, Win32::, Mac:: modules. Irreconcilable differences. Not sure how you would cover this, but point 12 seems to possibly fit. POSIX.pm is created for the platform it's installed on. A recent package I was testing, File::Flock (which is why I can't install PITA) attempted to use the macro EWOULDBLOCK. Windows doesn't support this, and there doesn't seem to be a suitable way to detect this properly. This is just one example, I've come across others during testing. Also note that the name alone does not signify it will not work on other platforms. There are some Linix:: distros that work on Windows. > 14. Tests run, and some/all tests fail. > The normal FAIL case due to test failures. Unless you can guarantee the @INC paths are correct when testing, this should be split into two. The first is simply the standard FAIL result. The second is the result of the fact that although the distribution states the minimum version of a dependancy, the installer has either failed to find it or found the wrong version. This is a problem currently with CPANPLUS, and is unfortunately difficult to track down. It's part of the "bogus dependancy" checks. > Installation and Completion > --- > > 16. All Tests Pass, did not attempt install. > > 17. All Tests Pass, but installation failed. > > 18. All Tests Pass, installation successful. And another one, test suite reports success, but failures occurred. This is usually the result of the use of Test.pm. I've been told that due to legacy systems, test scripts using Test.pm must always pass, even if there are failures. There are still a few distributions that are submitted to CPAN like this. Barbie.
Re: Request for Comments: Package testing results analysis, result codes
Moin, On Monday 20 February 2006 04:20, Adam Kennedy wrote: > (Andreas J. Koenig) wrote: > >> On Sun, 19 Feb 2006 22:22:20 +1100, Adam Kennedy <[EMAIL PROTECTED]> > >> said: > > > > > > 1. Broken or corrupt packaging. > > > A bad tarball, MANIFEST files missing. > > > > Make sure you verify that all files in the distro are readable. > > Reject if the permissions are bogus. Recently we had an increasig > > number of distros that had absurd permissions. > > That's a 1. > > > Also there is the rule that it must unpack into a single top level > > directory and must not clobber the working directory. > > That's a 2. > > > Besides that, we reject distros that contain a blib/ directory. This > > seems arbitrary usually just catches a trivial error that might cause > > grief later. > > That one's more interesting... the packaging itself is ok, and > technically it's a valid "perl5", "perl5.make" or "perl5.build" > package, but it has failed something more like a compulsory consistency > check... > > So maybe 3. Package failed compulsory quality check Can you do warnings as well? Warn if any of these is missing: SIGNATURE META.yml Makefile.PL ? If the SIGNATURE is there, but fails to check, then reject outright. Best wishes, Tels -- Signed on Mon Feb 20 12:29:40 2006 with key 0x93B84C15. Visit my photo gallery at http://bloodgate.com/photos/ PGP key on http://bloodgate.com/tels.asc or per email. "My glasses, my glasses. I cannot see without my glasses." - "My glasses, my glasses. I cannot be seen without my glasses." pgpjcG7ZOo9VW.pgp Description: PGP signature
Re: Request for Comments: Package testing results analysis, result codes
On Mon, Feb 20, 2006 at 11:36:27AM +, Barbie wrote:
> > 12. System is incompatible with the package.
> > Linux::, Win32::, Mac:: modules. Irreconcilable differences.
>
> Not sure how you would cover this, but point 12 seems to possibly fit.
> POSIX.pm is created for the platform it's installed on. A recent package
> I was testing, File::Flock (which is why I can't install PITA) attempted
> to use the macro EWOULDBLOCK. Windows doesn't support this, and there
> doesn't seem to be a suitable way to detect this properly.
use POSIX "EWOULDBLOCK"; $ok = eval { EWOULDBLOCK(); 1 }; print $ok
Re: Request for Comments: Package testing results analysis, result codes
Regarding the blow, I may have been a little unclear on the layout of the points. The first line is the name of the error. Following lines are meant to provide details to help clarify what it means. Barbie wrote: On Sun, Feb 19, 2006 at 10:22:20PM +1100, Adam Kennedy wrote: 2. Incompatible packaging. Packaging unwraps, but missing files for the testing scheme. You may want to split this into a result that contains no test suite at all (UNKNOWN) and one that has missing files according to the MANIFEST. Currently the second only gets flagged to the tester, and is usually only sent to the author, if the tester makes a point of it, or the CPAN testing FAILs. If the package type supports a MANIFEST, and there are files missing from it, then the package is broken or corrupt, see 1. As for not testing suite at all, I'm not sure that's cause for an UNKNOWN, probably something else. 5. Installer missing dependency. For the installer itself. 6. Installer waits forever. Intended to cover the "non-skippable interactive question" This needs to cover misuse of fork() and alarm. Too many distributions assume these work on Windows. They don't. At least not on any Windows platform I've ever used from Win95 to WinXP. They usually occur either in the Makefile.PL or the test suite, rarely in the actual module code. I have absolutely no idea how we would go about testing that though. Certainly in the installer, which is expected to run on ALL platforms, having a fork/alarm would be possibly something of a no no. But how to test it? Almost impossible. It might be more of a Kwalitee element perhaps? And I believe fork does work on Windows, it's just that it's simulated with a thread or something yes? They may also fail on other OSs. Although they could potentially be covered in point 5. 12. System is incompatible with the package. Linux::, Win32::, Mac:: modules. Irreconcilable differences. Not sure how you would cover this, but point 12 seems to possibly fit. POSIX.pm is created for the platform it's installed on. A recent package I was testing, File::Flock (which is why I can't install PITA) attempted to use the macro EWOULDBLOCK. Windows doesn't support this, and there doesn't seem to be a suitable way to detect this properly. Presumably because it failed tests and thus failed to install? :) To use an analogy, tt's not up to the testing infrastructure to stop you putting the noose around your neck, only to tell everybody that you hanged yourself. It shouldn't have to do problem analysis, only identify that there is a problem, identify the point at which it failed and in what class of general problem, and record everything that happened so you can fix the problem yourself. BTW, the top level PITA package is not intended to be compatible with Windows, although I do go out of my way to generally try to be platform neutral wherever possible. The key one that requires cross-platform support is PITA::Image and PITA::Scheme which will sit inside the images and so need to work on everything. This is just one example, I've come across others during testing. Also note that the name alone does not signify it will not work on other platforms. There are some Linix:: distros that work on Windows. That's fine, I don't mean to test by name and reject on that basis. I expect the Makefile.PL to somehow inform me that the platform is NA. 14. Tests run, and some/all tests fail. The normal FAIL case due to test failures. Unless you can guarantee the @INC paths are correct when testing, this should be split into two. The first is simply the standard FAIL result. The second is the result of the fact that although the distribution states the minimum version of a dependancy, the installer has either failed to find it or found the wrong version. This is a problem currently with CPANPLUS, and is unfortunately difficult to track down. It's part of the "bogus dependancy" checks. The second half is a clarification. I mean it to cover all the various things that current generate FAIL... when tests run and some or all fail. As for the CPANPLUS, etc problems, I imagine we can fix those over time. Hopefully the results we can provide will have enough granularity so they don't just all get dumped into "FAIL". Installation and Completion --- 16. All Tests Pass, did not attempt install. 17. All Tests Pass, but installation failed. 18. All Tests Pass, installation successful. And another one, test suite reports success, but failures occurred. This is usually the result of the use of Test.pm. I've been told that due to legacy systems, test scripts using Test.pm must always pass, even if there are failures. There are still a few distributions that are submitted to CPAN like this. Um, what? Can you explain this in a bit more detail? Is this documented somewhere? Adam K
Re: Request for Comments: Package testing results analysis, result codes
Barbie wrote: > > 12. System is incompatible with the package. > > Linux::, Win32::, Mac:: modules. Irreconcilable differences. > > Not sure how you would cover this, but point 12 seems to possibly fit. > POSIX.pm is created for the platform it's installed on. A recent package > I was testing, File::Flock (which is why I can't install PITA) attempted > to use the macro EWOULDBLOCK. Windows doesn't support this, and there > doesn't seem to be a suitable way to detect this properly. > > This is just one example, I've come across others during testing. > > Also note that the name alone does not signify it will not work on other > platforms. There are some Linix:: distros that work on Windows. Same with some Mac:: and Win32:: modules that perfectly work on other platforms because they're pure Perl modules. -- Sébastien Aperghis-Tramoni Close the world, txEn eht nepO.
Re: Request for Comments: Package testing results analysis, result codes
Adam Kennedy <[EMAIL PROTECTED]> wrote: > Firstly is that it might turn an otherwise normal result into something > else, with no clear rule. It makes a judgement call that some level of > testing is good or bad, which isn't really the place of an installer to > call. > > The reason Kwalitee has metrics like this is that it's not important in > the scheme of things, it's only looking for indicators which may well be > wrong (hence the name Kwalitee). The Kwalitee of a module does not > prevent it being installed. What makes 79 skips different from 80 skips? > You need some clear distinction between the two states, not just > something arbitrary (be it 50 or 80 or something else). I was speaking in percentages, not actual numbers, but I do see your point: part of the problem is that while we can send a message to the screen about why a test was skipped, there's no convention for making a *computer* understand why. If there were flags or tags that were standard convention (eg; "need_config", "missing_module", etc) then this would be a lot easier. > Now 100% skips, THAT could potentially be interesting, or maybe TODOs. But > then I don't necesarily know why it would be worthy of a different result > code. Is there metadata stored apart from these result codes? If so it might be useful to just store the statistics on skips. Assuming this database was searchable, it'd be easy to identify modules that could be tested better by, say, sorting by % of tests skipped and then looking at the top curplits' log output manually. That actually brings up another point: Currently testers.cpan.org only provides verbose log output for failures. In cases like this, (or cases where you're driving yourself mad trying to understand why platform X always passes whereas platform Y which doesn't seem too different always fails), it'd be handy for the module author to have access to full logs of the test output for both passes *and* fails. > The second issue is that it has the potential to override the package > author. If the author felt that the tests weren't critical, and could be > skipped quite happily, who are we to make a call that he is wrong and that > it's a probably, or anything other than 100% ok. I'd still like such a thing to be visible in some way. Of course you're going to happily skip tests that require a database if you don't have DBI_DSN set. (I am toying with the idea of marshalling a whitebox for the purpose of doing CPAN testers specifically for database and mod_perl-based packages... that'll probably be a project in itself, hacking YACsmoke to only pick up those packages, add "-mysql" or whatever to the uname that it posts with, etc. :) Cheers, Tyler
Re: Request for Comments: Package testing results analysis, result codes
Now 100% skips, THAT could potentially be interesting, or maybe TODOs. But then I don't necesarily know why it would be worthy of a different result code. Is there metadata stored apart from these result codes? If so it might be useful to just store the statistics on skips. Assuming this database was searchable, it'd be easy to identify modules that could be tested better by, say, sorting by % of tests skipped and then looking at the top curplits' log output manually. That actually brings up another point: Currently testers.cpan.org only provides verbose log output for failures. In cases like this, (or cases where you're driving yourself mad trying to understand why platform X always passes whereas platform Y which doesn't seem too different always fails), it'd be handy for the module author to have access to full logs of the test output for both passes *and* fails. Yep. One of the general concepts is that we capture in as high a resolution as possible while we are inside the testing instance, and then do further analysis later, outside the testing instance. So yes, you get the complete output of everything, in as much details as I can pull out. And analysis will (sort of) be pluggable or extendable in some way. No details or specifics on analysis yet, as it's not on the critical development path. There will be ways to do what you want. The second issue is that it has the potential to override the package author. If the author felt that the tests weren't critical, and could be skipped quite happily, who are we to make a call that he is wrong and that it's a probably, or anything other than 100% ok. I'd still like such a thing to be visible in some way. Of course you're going to happily skip tests that require a database if you don't have DBI_DSN set. Not necesarily... it all depends on how important it is to you. I see some potential cases where you'd rather abort the install if you can't do proper testing. Adam K
Re: Request for Comments: Package testing results analysis, result codes
Adam Kennedy <[EMAIL PROTECTED]> wrote: > > I'd still like such a thing to be visible in some way. Of course > >you're going to happily skip tests that require a database if you don't > >have DBI_DSN set. > Not necesarily... it all depends on how important it is to you. I see > some potential cases where you'd rather abort the install if you can't > do proper testing. Fair enough. So both uses cases exist and it sounds like you're gearing PITA to be able to handle them in some fashion. I'm looking forward to seeing it. :) Cheers, Tyler
Re: Network Testing
On Feb 17, 2006, at 7:57 AM, David Steinbrunner wrote: ... that give the ability to ask for the current kb/s or the like. I think you'll have to roll your own, but you might get help from the various NetPacket::* classes, such as NetPacket::TCP --- Matisse Enzer <[EMAIL PROTECTED]> http://www.matisse.net/ - http://www.eigenstate.net/
Re: Request for Comments: Package testing results analysis, result codes
On Feb 19, 2006, at 7:13 AM, Andreas J. Koenig wrote: Make sure you verify that all files in the distro are readable. Reject if the permissions are bogus. Recently we had an increasig number of distros that had absurd permissions. This reminds me - it doesn't seem like Module::Build allows one to specify the installed permissions of files, for example, if i want my executables to be installed as 550, or 4555 it doesn't look like I can do that. I could be wrong about that though. --- Matisse Enzer <[EMAIL PROTECTED]> http://www.matisse.net/ - http://www.eigenstate.net/
Re: Network Testing
Matisse Enzer wrote: > > On Feb 17, 2006, at 7:57 AM, David Steinbrunner wrote: > >> ... that give the ability to ask for the current kb/s or the like. > > I think you'll have to roll your own, but you might get help from the > various NetPacket::* classes, such as >NetPacket::TCP Thanks for the NetPacket pointer. Those modules look like they could come in handy. -- David Steinbrunner
