Request for Comments: Package testing results analysis, result codes
I'm starting to get a bit closer (waiting on a test images and some last testing to be done) to finishing the initial PITA test cycle (and thus be able to do an initial release) and so I'm starting to do some prep work now for the next stage, which is to start to assemble some infrastructure around it. One of the questions that I don't really have clear answers for yet and doesn't really seem to be answered yet properly relates to the testing return codes. That is, if we run a testing instance, what is the _result_ (PASS, FAIL, etc) of that test run. Some caveats and context, to keep the conversation on track and prevent musing and side-tracking... :) I understand that there is going to be some logic relating to recursiveness of results, but I'm not worried about that yet. I also understand that these eventually need to map to the current CPAN Testers PASS/FAIL/NA/UNKNOWN, but I'm not worried about how to do that yet. Please also note that analysis is done _separately_ in some cases from results collection, so there isn't necessarily a hard need for any conclusions to be reached only with the resources available inside the testing run itself. And on that same point, these codes are looking for the root cause, without necessarily assigning fixed blame or conclusions, as this will vary depending on the context in which the results are used. Again, my starting point for this analysis work is only the specific result value that is emitted out of a test instance (or that is derived from testing output outside the instance) Without these being final names or codes, I have the following result cases so far that might need to be taken into account by an arbitrary testing scheme (Makefile.PL, Build.PL, Java Ant, Configure, etc) Package Container catagory (covering tarballs, contents of the package, etc) - 1. Broken or corrupt packaging. A bad tarball, MANIFEST files missing. 2. Incompatible packaging. Packaging unwraps, but missing files for the testing scheme. Installer catagory (covering Makefile.PL/Build.PL/Configure) - 3. Installer unknown result. Something happened, we just aren't sure what. 4. Installer crash/failure. Installer fails to complete, and doesn't give a reason. 5. Installer missing dependency. For the installer itself. 6. Installer waits forever. Intended to cover the "non-skippable interactive question" Resources and Dependencies. All the below would refer to compulsory deps, not optional. All the below class 'unable to locate' as 'missing' even if the dependency is actually installed. 6. Package is missing a build-time dependency within installer scope. Doesn't have Test::More, missing headers in the C case. 7. Package missing a run-time dependency within installer scope. Missing a Perl module dependency, missing a .so in the C case. 8. Package missing a build-process resource. C-compiler, other required build tools missing. Does not include testing tools? They are in 10? Comments? 9. Package missing an external resource. System .h for Perl-with-C, missing run-time bins like 'cvs', libraries in other languages outside the normal scope of the installer (so non-CPAN things, possibly even Perl things) 10. Package missing an external build-time resource. Defining "external" is a little fuzzy, and should there be a seperate build-external? Does build-process cover this? How do we split $make from $test within "build". Can we do this? 11. System is missing a required resource. Missing a bit of hardware, missing something else structural, that could be resolvable. 12. System is incompatible with the package. Linux::, Win32::, Mac:: modules. Irreconcilable differences. Building and Testing 11. Compilation and/or transformation of files fails. Includes XS, C, pod2man, Java compiling, etc. Basically covers the scope of "make". 12. Compilation and/or transformation of files hangs. Covers "non-skippable interactive question". Might also cover other things. 13. Tests exist, but fail to be executed. There is tests, but the tests themselves aren't failing. It's the build-process that is failing. 14. Tests run, and some/all tests fail. The normal FAIL case due to test failures. 15. Tests run, but hang. Covers "non-skippable interactive question". Covers infinite attempts to connect network sockets. Covers other such cases, if detectable. Installation and Completion --- 16. All Tests Pass, did not attempt install. 17. All Tests Pass, but installation failed. 18. All Tests Pass, installation successful. =
Re: Request for Comments: Package testing results analysis, result codes
> On Sun, 19 Feb 2006 22:22:20 +1100, Adam Kennedy <[EMAIL PROTECTED]> said: > 1. Broken or corrupt packaging. > A bad tarball, MANIFEST files missing. Make sure you verify that all files in the distro are readable. Reject if the permissions are bogus. Recently we had an increasig number of distros that had absurd permissions. Also there is the rule that it must unpack into a single top level directory and must not clobber the working directory. Besides that, we reject distros that contain a blib/ directory. This seems arbitrary usually just catches a trivial error that might cause grief later. I have only 1/3 of a cent today:/ -- andreas
Re: Request for Comments: Package testing results analysis, result codes
Adam, I have one more edgey case I'd like to see on this list: > > 13. Tests exist, but fail to be executed. > There is tests, but the tests themselves aren't failing. > It's the build-process that is failing. > > 14. Tests run, and some/all tests fail. > The normal FAIL case due to test failures. > > 15. Tests run, but hang. > Covers "non-skippable interactive question". > Covers infinite attempts to connect network sockets. > Covers other such cases, if detectable. Tests run, but >50% (or maybe >80%?) are skipped. From what I've seen, the most common cause of this is that the package is untestable with the current build configuration. Eg; you needed to specify a webserver or database or something to get these tests to run. Apache::Test provides some hooks for autotesters to configure themselves to test packages using it, IMHO setting DBI_DSN etc should be enough for packages that test against a database. I've been thinking a lot about how to properly autotest database-driven packages lately. So far the best I've got is: - If a package depends on a particular DBD:: driver, and you have the matching database server available, specify a DBI_DSN that uses that driver. - If a package depends on DBI, but no particular driver, try testing it with every driver you can use, starting with SQLite since it does not need a "live" database server. In the case where a package supports multiple, but not all, database driver backends, they would probably depend on DBI, but "reccomend" each DBD:: driver in the META.yml, which would be covered by the first bullet. Cheers, Tyler
Re: Request for Comments: Package testing results analysis, result codes
Tyler MacDonald <[EMAIL PROTECTED]> wrote: > Adam, > I have one more edgey case I'd like to see on this list: > Tests run, but >50% (or maybe >80%?) are skipped. > > From what I've seen, the most common cause of this is that the > package is untestable with the current build configuration. Eg; you needed > to specify a webserver or database or something to get these tests to run. This scenario (many skipped tests) can also happen with modules that come bundled with many drivers. One driver might be required, but the rest might be optional. Tests for optional drivers that are missing their prerequisites will be skipped. I have several modules[1] like this, and for each of them, I wrote the test suite so that each test script re-runs all of its tests with every available driver. In the case of C::A::P::AnyTemplate, five templating systems are supported, but only HTML::Template is required[2]. So if the user has no other supported templating modules besides HTML::Template installed (i.e. the minimum supported configuration), then 243 out of 326 tests will be skipped, i.e. 75%. If, at some point in the future, I add support for another templating system, then the number of tests skipped by default will increase. I'm not really sure if this is the best way to manage my optional tests (one disavantage is that users tend to get freaked out by so many skipped tests). I'm just pointing out a couple of modules where a lot of skips are the norm. Michael [1] http://search.cpan.org/dist/Config-Context/ http://search.cpan.org/dist/CGI-Application-Plugin-Config-Context/ http://search.cpan.org/dist/CGI-Application-Plugin-AnyTemplate/ [2] HTML::Template is required because it's already a prerequisite of CGI::Application -- Michael Graham <[EMAIL PROTECTED]>
Re: Request for Comments: Package testing results analysis, result codes
Tyler MacDonald wrote: Tests run, but >50% (or maybe >80%?) are skipped. From what I've seen, the most common cause of this is that the package is untestable with the current build configuration. Eg; you needed to specify a webserver or database or something to get these tests to run. Apache::Test provides some hooks for autotesters to configure themselves to test packages using it, IMHO setting DBI_DSN etc should be enough for packages that test against a database. Note that there is also the case where the module need administrator privileges to execute parts of its test suite and the author put some effort to skip all the tests that can't be run because the user running the installation doesn't have the required rights (I'm thinking very loud about Net::Pcap if you haven't guessed :) Sébastien Aperghis-Tramoni -- - --- -- - -- - --- -- - --- -- - --[ http://maddingue.org ] Close the world, txEn eht nepO
Re: Request for Comments: Package testing results analysis, result codes
While an interesting idea, I forsee two challenges to doing this... Firstly is that it might turn an otherwise normal result into something else, with no clear rule. It makes a judgement call that some level of testing is good or bad, which isn't really the place of an installer to call. The reason Kwalitee has metrics like this is that it's not important in the scheme of things, it's only looking for indicators which may well be wrong (hence the name Kwalitee). The Kwalitee of a module does not prevent it being installed. What makes 79 skips different from 80 skips? You need some clear distinction between the two states, not just something arbitrary (be it 50 or 80 or something else). Now 100% skips, THAT could potentially be interesting, or maybe TODOs. But then I don't necesarily know why it would be worthy of a different result code. The second issue is that it has the potential to override the package author. If the author felt that the tests weren't critical, and could be skipped quite happily, who are we to make a call that he is wrong and that it's a probably, or anything other than 100% ok. An interesting idea, but maybe one that is better as a Kwalitee metric? Adam K Tyler MacDonald wrote: Adam, I have one more edgey case I'd like to see on this list: 13. Tests exist, but fail to be executed. There is tests, but the tests themselves aren't failing. It's the build-process that is failing. 14. Tests run, and some/all tests fail. The normal FAIL case due to test failures. 15. Tests run, but hang. Covers "non-skippable interactive question". Covers infinite attempts to connect network sockets. Covers other such cases, if detectable. Tests run, but >50% (or maybe >80%?) are skipped. From what I've seen, the most common cause of this is that the package is untestable with the current build configuration. Eg; you needed to specify a webserver or database or something to get these tests to run. Apache::Test provides some hooks for autotesters to configure themselves to test packages using it, IMHO setting DBI_DSN etc should be enough for packages that test against a database. I've been thinking a lot about how to properly autotest database-driven packages lately. So far the best I've got is: - If a package depends on a particular DBD:: driver, and you have the matching database server available, specify a DBI_DSN that uses that driver. - If a package depends on DBI, but no particular driver, try testing it with every driver you can use, starting with SQLite since it does not need a "live" database server. In the case where a package supports multiple, but not all, database driver backends, they would probably depend on DBI, but "reccomend" each DBD:: driver in the META.yml, which would be covered by the first bullet. Cheers, Tyler
Re: Request for Comments: Package testing results analysis, result codes
(Andreas J. Koenig) wrote: On Sun, 19 Feb 2006 22:22:20 +1100, Adam Kennedy <[EMAIL PROTECTED]> said: > 1. Broken or corrupt packaging. > A bad tarball, MANIFEST files missing. Make sure you verify that all files in the distro are readable. Reject if the permissions are bogus. Recently we had an increasig number of distros that had absurd permissions. That's a 1. Also there is the rule that it must unpack into a single top level directory and must not clobber the working directory. That's a 2. Besides that, we reject distros that contain a blib/ directory. This seems arbitrary usually just catches a trivial error that might cause grief later. That one's more interesting... the packaging itself is ok, and technically it's a valid "perl5", "perl5.make" or "perl5.build" package, but it has failed something more like a compulsory consistency check... So maybe 3. Package failed compulsory quality check Adam K
