On Sat, Feb 21, 2004 at 01:13:30AM -0500, sungo wrote: > On (02/07 15:53), Rocco Caputo wrote: > > > I think the tests should be generated at "make test" time. We get a > > smaller distribution this way. > > you've never been terribly concerned about a really small distribution > before. we've shipped 10K of examples until really recently. i'd much > rather ship 10K of real honest to god easy to read and maintain tests.
POE still ships with about 200K of samples, and it will continue to until they're replaced by cookbook recipes or tutorials. The plans have been on display for months at http://poe.perl.org/?POE_RFCs/Deprecate_Sample_Program > > Once the generation phase is done, Test::Harness executes specific and > > generated tests as usual. > > what happens when test generation fails? how do we include tests for the > test generation framework? As I see it, test generation is an exercise in templating. The proposal suggests combining a pool of perhaps 20 reusable test programs---one for each of POE::Kernel's reusable APIs---against each available runtime environment. The result would be maybe a hundred small programs that look virtually identical except that they use different modules and require different tests. This idea is based on POE's current t/27_poll.t. If IO::Poll is available, it's loaded so POE::Loop::Poll will be activated. Then it tests the select() API, which exercises the loop's filehandle watchers. If you've never seen it: #!/usr/bin/perl -w # $Id: 27_poll.t,v 1.10 2004/01/31 06:58:30 rcaputo Exp $ # Rerun t/04_selects.t but with IO::Poll instead. use strict; use lib qw(./mylib ../mylib ../lib ./lib); use TestSetup; BEGIN { eval 'use IO::Poll'; test_setup(0, "IO::Poll is needed for these tests") if length($@) or not exists $INC{'IO/Poll.pm'}; test_setup(0, "IO::Poll 0.05 or newer is needed for these tests") if $IO::Poll::VERSION < 0.05; } require 't/04_selects.t'; exit; So, what do we do if templating fails? Well... 1. If POE's favorite templating language isn't available... POE does not need a full-featured, general-purpose templating library. The test generator should include some basic, line-based tag replacement and be done with it. No problem. 2. If the test generator contains one or more bugs... A large number of the test files will come out broken. Like the Balrog, or constipation, they shall not pass. In theory, the release technician will be bright enough notice something is wrong while testing a distribution before releasing it. We can always hire a new release technician if the current one's excessively stupid or careless. 3. If there's not enough disk space to write the test files... The test generator reports a "disk full" error, just like tar or unzip would. 4. If the filesystem doesn't like the filenames used... The test generator reports a "cannot open file: $!" error, just like tar or unzip would. Fixing a generator is probably easier than maintaining a hundred discrete files. 5. If Perl is broken... It happens. The tests will probably fail either way, so POE's neither better nor worse off. 6. If a god smites the machine... The bad disk block would have been there for tar or unzip to extract a file into. The meteorite would have hit the server farm anyway. POE is neither better nor worse off. 7. If something unanticipated happens... Sorry, I just don't know. That's the problem with unanticipated failure modes. Are there failure modes I'm missing? > > The major difference between revisions is that Sean pointed out some > > test environments that I missed. So we have a potential for many more > > test files, but we also test more of POE. > > then i completely misread it. or it changed drastically from the time > you showed it to me in irc to the time you sent it to the list. It's probably changed since you last read it. There's this iterative cycle of release, gather feedback, edit, and re-release going on. It happens even when no one's paying attention to the list. > my opinion is still that you are suggesting a massive new complexity of > autogenerating self-building tests that isnt really necessary. this > level of complexity will form a roadblock to long term maintenance and > developer understand. unless of course you're planning on the only > person doing long term maintenance or developing on this being you. if > that's what you're thinking, go for it. do whatever you want. I don't see the massive complexity you do. The proposal is for a program that combines a set of test libraries (such as "selects.t" and "alarms.t") with a set of event loops (Gtk, Tk, Event, IO::Poll, select), generating from them: #!/usr/bin/perl -w use strict; use lib qw(./mylib ../mylib ../lib ./lib); use Gtk; require "selects.t"; # and a second one with "alarms.t" exit; and #!/usr/bin/perl -w use strict; use lib qw(./mylib ../mylib ../lib ./lib); use Tk; require "selects.t"; # and a second one with "alarms.t" exit; and #!/usr/bin/perl -w use strict; use lib qw(./mylib ../mylib ../lib ./lib); use Event; require "selects.t"; # and a second one with "alarms.t" exit; and #!/usr/bin/perl -w use strict; use lib qw(./mylib ../mylib ../lib ./lib); use IO::Poll; require "selects.t"; # and a second one with "alarms.t" exit; and #!/usr/bin/perl -w use strict; use lib qw(./mylib ../mylib ../lib ./lib); require "selects.t"; # and a second one with "alarms.t" exit; and perhaps is smart enough to not generate the Event tests if C<eval "use Event"> fails. Perhaps I'm off-base, and it'll be easier to write and maintain them by hand. It would be useful to get a third opinion on the matter. -- Rocco Caputo - [EMAIL PROTECTED] - http://poe.perl.org/
