The biggest time suck in developing MakeMaker, and to a lesser extent Test::More, is running the tests. Why? Because they need to be run on lots of different platforms with lots of different versions of Perl. Currently, I do this by hand. And we all know manual testing sucks. Its time consuming and you tend to avoid it. I can't run the tests on every platform at every patch so I often wind up breaking something and not realizing it for a while.
So what I need is some way to set up a network of test servers such that I can say "test this module for me" and my testing client would ship it to as many test servers as it can find and get the results back all in just a few minutes. Here's a simple sketch of what I'd imagine the protocol would look like. I'm not much on networking, so I'm going to keep it simple. HTTP is simple. I'd also like to make running a test server as simple and portable as possible. Web servers are simple and portable. Client: GETs master.test.server.list containing what OS / versions of Perl they're testing with and their availability. The client then builds an OS / Perl version matrix and contacts each server to necessary to fill the matrix, rather than simply all available servers. Client: POSTs to some.test.server saying "Here's my credentails and the URL where you can get the source code I'd like you to test" Server: Returns "ok, I'll run this for you with $these versions of Perl. Pick up your results at this URL" or "sorry, I can't run this right now" Client: If the server is willing to run the code, we poll the results URL until we get results or a timeout. If the server times out or won't run the tests, we move onto the next available server that can cover this OS/version combo. Loop until the matrix is filled or we run out of servers to contact or timeout. And that's about it. The results can be as simple as the verbose T::H output plus a simple boolean indicating overall pass/fail. Here's the hard problems: Since we're basically setting up a big network to run arbitrary code, it cannot be made available to just anyone with a CPAN id. Credentials become a problem. There has to be access control on the users of the system. We also must make sure that someone can't attach a test server which harvests user's credentials. I also want to prevent every test server from contacting the master server for each user wanting to run tests to avoid undue load. Finally, I want to allow the system to still work without the master server (clients can cache the test server list). The last two may be asking too much. A strong sandbox for the test servers to run the code in would be nice, however since the users of the network are authenticated we don't need to guard too strongly anymore than, say, CPANPLUS does. Since its difficult to do this in a cross-platform manner, I'm tempted to push the responsibility for setting up the sandbox off to the individual test server admin. About the only thing our code would do is install module dependencies in its own directory outside the normal @INC. The most important thing to remember about a project like this is KISS. Keep It Simple, Stupid. Features can rapidly spiral out of control, and it'll never get done. But if we keep it simple something workable can probably be knocked together in a few days. I'm putting this up on the MakeMaker Wiki for collaborative refinement. http://www.makemaker.org/wiki/?AutomatedTestDistribution -- Michael G Schwern [EMAIL PROTECTED] http://www.pobox.com/~schwern/ If at first you don't succeed, call in an air strike.