Talk: Why You Really Want To Write Tests
I just got back from JOIN (Jornadas de Informatica) 2005, a little student run conference at the University of Minho in Portugal. Why this might be of interest is its the same place and largely the same folks who will be running YAPC::Europe. http://natura.di.uminho.pt/join2005/out/news_en.html I was asked to give something about testing to an audience of undergraduate informatics students, largely Haskell and maybe some Java. What I finally came up with is this: http://www.pobox.com/~schwern/talks/Why_Test/ I had 45 minutes to say something about testing and decided all I could do was expose them to the subject, since its not part of their coursework, and try to get them interested to so some work on their own. The audience being inexperienced at programming and not Perl programmers meant I couldn't go into much detail. Additionally since they're students and/or academics it doesn't help much to talk about business reasons for testing. So I went for the most basic, convincing reasons to write tests: how it will benefit the lone wolf programmer. Selfish reasons. The reasons most likely to convince someone who would otherwise just look at testing as just more work. There's really nothing there which hasn't been said before, not the best talk in the universe, but its nice to have it all layed out if nothing else than to help convince your co-workers. So, what's wrong with it? While I thought about what I wanted to say for months I ultimately wrote the slides the night before the talk. :) I forgot one important argument: testing lets you optimize/redesign without fear. The whole fear based programming problem. There's almost no pictures. There's not nearly enough references, particularly when I expect the audience to go out and work things out on their own. I still can't think of a decent testing book nor tutorial to recommend. Test::Tutorial leaves the reader at a dead end without referencing further works on, say, perl.com. I don't know the JUnit community to recommend anything there. There's not nearly enough examples. I avoided code to avoid confusing the audience with a bunch of Perl and to keep things focused on ideas rather than implementation. I only had 45 minutes. From the questions I got I believe this left people without a clear concept of what a test was. Or how big a test should be or how much a single test should cover. One question was how tests should be divided up for which I used the WWW::Mechanize test suite as an example. During the break after the talk, but while I still had my laptop plugged into the projector, I was demonstrating Devel::Cover's test coverage ability and noticed quite a number of folks in the audience were rather interested in whatever it was I was doing. I don't know if it was the pretty colors on the HTML page or what but it seems to be a fairly powerful demonstration. Anyhow, there it is.
Re: Phalanxing MakeMaker
On Thu, Feb 24, 2005 at 11:47:31AM -0600, Ken Williams wrote: Better would be to snark it from ExtUtils::CBuilder, which has a has_compiler() method. How portable is this? I see there's some minimal VMS and OS/2 code in there, will it work? If so then the MakeMaker tests can just use have_compiler() to determine if they should run the XS tests if ExtUtils::CBuilder is installed.
Re: Talk: Why You Really Want To Write Tests
Michael G Schwern wrote: I was asked to give something about testing to an audience of undergraduate informatics students, largely Haskell and maybe some Java. What I finally came up with is this: http://www.pobox.com/~schwern/talks/Why_Test/ Liked the emphasis on version control. Had to learn Subversion as part of working in Phalanx project -- and had to learn Test::More much better. Glad I did both. In slide011.html: :%s/repetetive/repetitive/g jimk
Test::Output 0.05
Test::Output 0.05 has been released. The main difference in this version is a patch from chromatic (thank you). His patch fixed a bug where all tests printed error diags even if they passed. Additional tests were added to ensure that no output is sent from diag in successful tests. While releasing this version I realized I never wrote anything about version 0.04 which saw some major changes: Added fuctions: output_like and output_unlike. stdout_from, stderr_from, and output_from were removed from @EXPORT and moved to @EXPORT_OK as suggested by Schwern. Renamed test comment to description. Refactoring and additional tests. --- Shawn Sorichetti
Re: Test::Output 0.05
Shawn Sorichetti wrote: Test::Output 0.05 has been released. The main difference in this version is a patch from chromatic (thank you). His patch fixed a bug where all tests printed error diags even if they passed. Additional tests were added to ensure that no output is sent from diag in successful tests. While releasing this version I realized I never wrote anything about version 0.04 which saw some major changes: Added fuctions: output_like and output_unlike. stdout_from, stderr_from, and output_from were removed from @EXPORT and moved to @EXPORT_OK as suggested by Schwern. Renamed test comment to description. Refactoring and additional tests. --- Shawn Sorichetti It occurs to me that the Test::* and ExtUtils::* modules should be the first stop for the Phalanx project. :) Incidentally, is it acceptable for an author to Phalanx-review his own module, or does that defeat the peer-reviewed-ness of it? I'm about to upload my first module, and if Phalanx has some kind of check list that must be satisfied to have the Phalanx-approval-seal (is there one?), then I'd be happy to go through it. Random thought: Could Devel::Cover be automatically run against all modules in CPAN, with ratings posted on cpan.org right next to the usual test results? -ofer
Re: Test::Output 0.05
On Fri, Mar 04, 2005 at 04:27:07PM -0800, Ofer Nave wrote: Random thought: Could Devel::Cover be automatically run against all modules in CPAN, with ratings posted on cpan.org right next to the usual test results? It still chokes on certain not uncommon constructs like threads.
Re: Test::Output 0.05
Michael G Schwern wrote: On Fri, Mar 04, 2005 at 04:27:07PM -0800, Ofer Nave wrote: Random thought: Could Devel::Cover be automatically run against all modules in CPAN, with ratings posted on cpan.org right next to the usual test results? It still chokes on certain not uncommon constructs like threads. Is it possible to tell quasi-reliably if the code in question uses threads and skip it? On that note, is forking also problematic? I think it would be a powerful addition to CPAN. If you go to the distribution page for any module - say, for example, Class::DBI (http://search.cpan.org/~tmtm/Class-DBI/) - you can see that all tests passed on 29 platforms and failed on 6, but it doesn't tell you how comprehensive the tests are. Worst case, it could be a single test hard-coded to return true, and you'll never know without download and examing the tarball. Imagine if instead of: CPAN Testers PASS (29) FAIL (6) [ View ] it said: CPAN Testers PASS (29) FAIL (6) COVERAGE (42%) [ View ] -ofer
Re: Test::Output 0.05
On Fri, Mar 04, 2005 at 04:44:42PM -0800, Ofer Nave wrote: Is it possible to tell quasi-reliably if the code in question uses threads and skip it? On that note, is forking also problematic? Not really. Anywhere within the test or any code called by the test might use threads. Devel::Cover however, is aware when it hits threads and dies with an error message. So it would be possible to at least know when a coverage run is not complete due to threading problems. I think it would be a powerful addition to CPAN. If you go to the distribution page for any module - say, for example, Class::DBI (http://search.cpan.org/~tmtm/Class-DBI/) Trouble right there. Now search.cpan.org has to run untrusted code so a jail would have to be constructed. Additionally it has to have all the optional modules necessary for testing the module available else the coverage run will appear low due to skipped tests. CDBI is a perfect example of this. Finally, any external dependencies or platform specific code causing tests to be skipped (libwin32, DBD::Pg, etc...) will produce inaccurately low coverage reports.
Test::* Modules, Devel::Cover, Phalanx [was Test::Output 0.05]
Ofer Nave wrote: Shawn Sorichetti wrote: Test::Output 0.05 has been released. Renaming thread since only the OP referred to Test::Output. It occurs to me that the Test::* and ExtUtils::* modules should be the first stop for the Phalanx project. :) At least as originally proposed by project leader Andy Lester, Phalanx was supposed to focus on *non-core*, but widely used distributions. The rationale for this approach, IIRC, was that, for better or worse, there already was a peer-review approach to those modules distributed as core. I know there are core modules whose test coverage I would like to see improved, and there's nothing stopping anyone from applying tools such as Devel::Cover to core modules and submitting patches to their authors or p5p or whomever is appropriate. Until I hear different from Andy, I'm going to proceed on the assumption that Phalanx is for non-core modules. (Parenthetical rant: As I have found from my own work both in a Phalanx context and from submitting patches to other authors' modules, the single most difficult thing has nothing to do with code, tests or coverage; it's getting the author to give you feedback once you've submitted your proposed revisions.) Incidentally, is it acceptable for an author to Phalanx-review his own module, or does that defeat the peer-reviewed-ness of it? I'm about to upload my first module, and if Phalanx has some kind of check list that must be satisfied to have the Phalanx-approval-seal (is there one?), then I'd be happy to go through it. There is no such seal of approval. You have done well by getting feedback on your idea on the local list, perl.module-authors and this list before uploading it. Write some tests, get it up there and let the world take a crack at it. Random thought: Could Devel::Cover be automatically run against all modules in CPAN, with ratings posted on cpan.org right next to the usual test results? As others have already pointed out, this is not a good idea, because a module's overall coverage rating conceals as much as it reveals. Among other things, unless you write your documentation in Pod::Coverage's default format (which is used by Devel::Cover in its default setting), you will get a lousy POD coverage rating, which in turn will bring down your distro's overall rating. Text::Template, to cite one example, is well documented but not written in the Pod::Coverage-correct format, so its documentation gets rated zilch. Devel::Cover -- like coverage analysis in general -- is a diagnostic tool. That's why it's in the Devel:: TLNS. It's not like testers.cpan.org, which is more focused on the question of whether a distro successfully built or not on a wide variety of OSes. Jim Keenan
Re: Test::* Modules, Devel::Cover, Phalanx [was Test::Output 0.05]
At least as originally proposed by project leader Andy Lester, Phalanx was supposed to focus on *non-core*, Not exactly. I just figured it's easier for teams to submit tests back to individual authors than back to Perl itself. Also, those patches are more likely to get out into the mainstream, than having to get put into bleadperl and then into 5.8.x. the single most difficult thing has nothing to do with code, tests or coverage; it's getting the author to give you feedback once you've submitted your proposed revisions.) It's funny, because the FIRST most difficult thing was getting anyone to write any code. That was a year ago. Now, your observation is unfortunately so. For example, Chicago.pm did some great work on HTML::Tree (I think) and gave the patches to Sean Burke, but Sean has yet to do anything with them. As an author who has lots of modules that he maintains, I totally understand why that might be. xoa -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
