I was going to reply to each comment, but figured that by doing that, I'd set myself up to get bogged down in details, and the FAQ would never evolve. I remain completely open to criticism, so please don't stop.
Factors guiding my decisions at this point are simplicity and speed. I want something that will help new testers get started, and I want it to be ready quickly. The points some have made about alternatives to testing by hand are well taken - and I intend to include those topics, just not right away. Once a basic set of questions and answers is assembled and available to new testers, I'll add more advanced stuff, and more setup-intensive stuff. What follows is the second draft, based on feedback from Andreas Marcel Riechert, Andy Lester, sherzod and Randy Kobes. Please send your comments, corrections and additions, either to me directly, or to the list. Either way, I'll have at least one more revision in the next week or so. Here it is: =head1 NAME cpantestfaq - Frequently Asked Questions Aboout Testing for CPAN ($Date: 2002/07/11 00:23:36 $) =head1 DESCRIPTION This FAQ addresses testing CPAN modules and reporting your test results. It points out how simple it is to begin testing, and tries to help you get started. It also offers tips for more advanced or automated testing with such tools as CPAN and CPANPLUS. =head1 Before You Begin The first, and most important thing, to understand about testing for CPAN is that it is very, very easy to get started. There's always more that you I<can> do, but there's very little that you I<must> do in order to get started and be helpful. If you're looking for a way to contribute something to the Perl community, but don't know where to begin, this just might be the place. =head1 After You Begin =head2 How do I know which modules I should test? =over 4 =item 1 Test whatever modules you download to install and use. This is the easiest, most straightforward way to test. You're already running the tests, so why not report the results? If your results match what's already on CPAN for your platform, you needn't report them again. If results for your platform are different from what's on CPAN for your platform, report it. If you can explain the differences, please do so in your report. =item 2 Test whatever is announced on the CPAN-Testers mailing list. In the days following an announcement that a new module (or a new version of an existing one) has been posted, most authors are alert for problem reports. This is the best time for you to get your results in. =item 3 Test any modules you find that do not have results noted for the platform you use. Even if a module has not been updated for quite some time - if you find there are no results for that module on your platform, test it and report your results to cpan-testers and to the author. =item 4 Any time a new version of perl is released, every module on the CPAN should be tested again on every platform. See http://testers.cpan.org/search?request=by-config for more about this. =back =head2 How can I tell quickly which modules don't have testing results from my platform? =over 4 =item * Check the module's page on CPAN at http://search.cpan.org/ =item * Use CPANPLUS to check. (Details to be added) =back =head2 Should I ignore the line in the module posting message that says no action is required? Yes, you should. B<NOTE TO REVIEWERS> - I<if anyone knows the purpose of that phrase, please fill me in> =head2 How do I test the module distributions? Install them by hand with perl Makefile.PL; make; make test; Notice the results shown after make test. Run cpantest to submit the results from your platform. If anything other than PASS, add notes saying what failed and, if you can, why. If you can fix it, fix it and include the fix with your report if it's small. If it's not small, ask the author how to submit it. Always send a copy of your message to the author. Not every one of them monitors the CPAN-Testers mailing list as closely as you might like. =head2 What alternatives are there to testing by hand? If you have the CPAN module installed and configured you can do C<perl -MPCAN -e 'test Some::Module'> If you have CPANPLUS installed, you can run C<t Some::Module> in the default shell. Or you can write your own interface. CPANPLUS is an excellent tool for interacting with the Comprehensive Perl Archive Network. If you're interested in automating your tests, and don't mind a little setup work and a touch of uncertainty. It has not reached version 1.0 as of this writing (2002-07-09) and the authors are not working to keep each version backward-compatible until version 1.0 is released. Many of the testers you see posting lots of results are not doing all that work by hand. =head2 How do I report results? Use the C<cpantest> script, available from http://testers.cpan.org/. If you just run it as 'C<cpantest>' it will give you usage notes. Generally, you should run it from the directory in which the module unpacked. Unless you ask it not to, C<cpantest> will prompt you for the name and version of the module you're testing, using the name of the current directory as the default answer. C<cpantest> generates the subject line of the message you'll send, and nearly every automated part of the process is based on that subject line. It's very important that it follow the conventions that C<cpantest> encourages you to use. The message is sent by deafult to the cpan-testers mailing list. You should always send a copy to the module's author, as well, no matter what result you report. When reporting results, you choose from one of four grades: PASS, FAIL, UNKNOWN or NA. They are defined as follows: B<pass> The build went smoothly and every test included with the package either passed or was skipped. B<NOTE TO REVIEWERS> I<Some have suggested that a PASS grade can be reported if the module passes tests other than the usual 'test.pl' or those in the ./t directory. I disagree - I think we want to have a standard, and that standard seems to be the tests that run with C<make test>. Are there many cases where that's not possible? Is there some commonly-accepted alternative? How should we explain it?> B<fail> Some or all tests failed. If you get a fatal error during the build process (C<perl Makefile.PL> or C<make>) you should grade the distribution as "fail", too. The only failures that should not be reported with B<fail> are those resulting from unfulfilled dependencies. Some modules require that other modules be installed in order to function. Some modules require that you have certain software on your system - the DBD modules are the clearest example of this. If DBD::MySQL fails because you don't have MySQL installed, that's not the module's fault. Actually, that depends on I<how> it fails. If it complains during C<perl Makefile.PL> that a dependency is unfulfilled, then the module distribution is behaving properly. You should not report any result. Your system does not meet the requirements that the module distribution laid out, and is therefore not a valid testing platform right now. You may choose to satisfy the dependency by installing whatever the module asked for. If you do that, start the installation over. If C<perl Makefile.PL> does not complain, but something fails during C<make test>, then you should report B<fail>. If you can tell that it's simply an unfulfilled dependency that was not noticed during C<perl Makefile.PL>, say so in your error report so that the author can adjust the makefile to check next time. There is some disagreement on this point. I think module packages should check to see if the bridge is fully built before closing their eyes and stepping on the gas. Others feel differently. B<NOTE TO REVIEWERS -> I<it would be great if we could reach consensus on this one. In case Randy is right, though, and we can't, please provide an argument for whatever differing point of view you take. Randy made some good points earlier, I just haven't digested them entirely yet...> B<unknown> No tests were supplied in the module distribution. Modules are supposed to be distributed with tests - it is an error (but not a failure) if there are none. This grade is not seen very often, so take a minute to include a note in your result asking the author to please include some tests with the next version. Just in case the author doesn't know what I<unknown> means. B<na> The module was specifically designed not to function on the platform you're running. Typically, these are packages that take advantage of a particular operating system, such as those in the WIN32 or Mac namespaces. Unless there's a good reason for it, all modules should work on all platforms. =head2 What do I do if tests fail? Do what you'd want someone to do for you. Report the problems with as much detail as the author will need in order to fix - or at least understand - the problem. Most important is to send the output of the command were the failure occurred. Second most important is to pay attention to your e-mail and do your best to reply intelligently if the author has follow-up questions. =head1 AUTHOR This document is copyright 2002, Brian King. Please send updates, questions or criticism to Brian King <[EMAIL PROTECTED]>
