The biggest time suck in developing MakeMaker, and to a lesser extent
Test::More, is running the tests. Why? Because they need to be run on
lots of different platforms with lots of different versions of Perl.
Currently, I do this by hand. And we all know manual testing sucks.
Its time consuming and you tend to avoid it. I can't run the tests on
every platform at every patch so I often wind up breaking something and
not realizing it for a while.
So what I need is some way to set up a network of test servers such that
I can say "test this module for me" and my testing client would ship it
to as many test servers as it can find and get the results back all in
just a few minutes.
Here's a simple sketch of what I'd imagine the protocol would look like.
I'm not much on networking, so I'm going to keep it simple. HTTP is
simple. I'd also like to make running a test server as simple and
portable as possible. Web servers are simple and portable.
Client: GETs master.test.server.list containing what OS / versions of
Perl they're testing with and their availability.
The client then builds an OS / Perl version matrix and contacts each
server to necessary to fill the matrix, rather than simply all available
servers.
Client: POSTs to some.test.server saying "Here's my credentails and the
URL where you can get the source code I'd like you to test"
Server: Returns "ok, I'll run this for you with $these versions of Perl.
Pick up your results at this URL"
or "sorry, I can't run this right now"
Client: If the server is willing to run the code, we poll the results URL
until we get results or a timeout. If the server times out or
won't run the tests, we move onto the next available server that
can cover this OS/version combo.
Loop until the matrix is filled or we run out of servers to contact or
timeout.
And that's about it. The results can be as simple as the verbose T::H
output plus a simple boolean indicating overall pass/fail.
Here's the hard problems:
Since we're basically setting up a big network to run arbitrary code, it
cannot be made available to just anyone with a CPAN id. Credentials become
a problem. There has to be access control on the users of the system. We
also must make sure that someone can't attach a test server which harvests
user's credentials. I also want to prevent every test server from contacting
the master server for each user wanting to run tests to avoid undue load.
Finally, I want to allow the system to still work without the master server
(clients can cache the test server list). The last two may be asking too
much.
A strong sandbox for the test servers to run the code in would be nice,
however since the users of the network are authenticated we don't need to
guard too strongly anymore than, say, CPANPLUS does. Since its difficult to
do this in a cross-platform manner, I'm tempted to push the responsibility
for setting up the sandbox off to the individual test server admin. About
the only thing our code would do is install module dependencies in its own
directory outside the normal @INC.
The most important thing to remember about a project like this is KISS.
Keep It Simple, Stupid. Features can rapidly spiral out of control, and
it'll never get done. But if we keep it simple something workable can
probably be knocked together in a few days.
I'm putting this up on the MakeMaker Wiki for collaborative refinement.
http://www.makemaker.org/wiki/?AutomatedTestDistribution
--
Michael G Schwern [EMAIL PROTECTED] http://www.pobox.com/~schwern/
If at first you don't succeed, call in an air strike.
