Re: Distributed testing idea
On Sun, Mar 14, 2004 at 10:31:44PM -0600, Scott Bolte wrote: Just for the record, I've abandoned the HTTP::Daemon changes that supported using two unidirectional pipes. Given what I've learned, I believe the HTTP protocol precludes that mode. The problem is reading some types of response messages. When using sockets, the server closes the socket after a sending a response without a Content-Length field. The resulting EOF allows the client to detect that the message is complete. With persistent, unidirectional pipes an EOF is not an option. Now it might be possible to rely on a blank line to indicate the header is complete. However, I see no way to enforce that so I am not going to try. Doesn't HTTP 1.1 with persistent connections avoid that problem? Scott P.S. Btw, I've switched to Net::Jabber for this IPC problem. Thanks to whoever mentioned it, it rocks! Okay. Thanks for the update. Tim. On Sun, 22 Feb 2004 19:05:02 -0600, Scott Bolte wrote: On Sun, 22 Feb 2004 16:13:01 +, Tim Bunce wrote: Now I do agree the HTTP protocol is worth using for managing the flow of data. I run HTTP over ssh myself. (I modified HTTP::Daemon to use two unidirectional pipes instead of a single, bidirectional socket.) Has (will) that be included in future releases? Yes. I plan on submitting the changes back after the bits have a chance to dry. I want to advance my current project to the point where I can stress-test the HTTP::Daemon and UserAgent modifications before I submit the changes. I'll subscribe to the libwww mailing list since that seems to be the right forum for posting the changes. If you want a copy of the HTTP code as-is just drop me a line. Scott
Re: Distributed testing idea
On Mon, 15 Mar 2004 10:30:30 +, Tim Bunce wrote: ... The problem is reading some types of response messages. When using sockets, the server closes the socket after a sending a response without a Content-Length field. The resulting EOF allows the client to detect that the message is complete. With persistent, unidirectional pipes an EOF is not an option. Now it might be possible to rely on a blank line to indicate the header is complete. However, I see no way to enforce that so I am not going to try. Doesn't HTTP 1.1 with persistent connections avoid that problem? Good question. Even with Keep-Alive set, the LWP code will force the connection to be closed after returning an error. I did not study it well enough to see how running over HTTPS changed that. As I worked on the problem it keep feeling like I was forcing a square peg into a round hole. So while I wanted the simplicity of invoke-http-daemon-as-necessary-with-ssh model, I have to admit that jabber solves problems I'm tackling very well. Scott
Re: Distributed testing idea
Just for the record, I've abandoned the HTTP::Daemon changes that supported using two unidirectional pipes. Given what I've learned, I believe the HTTP protocol precludes that mode. The problem is reading some types of response messages. When using sockets, the server closes the socket after a sending a response without a Content-Length field. The resulting EOF allows the client to detect that the message is complete. With persistent, unidirectional pipes an EOF is not an option. Now it might be possible to rely on a blank line to indicate the header is complete. However, I see no way to enforce that so I am not going to try. Scott P.S. Btw, I've switched to Net::Jabber for this IPC problem. Thanks to whoever mentioned it, it rocks! On Sun, 22 Feb 2004 19:05:02 -0600, Scott Bolte wrote: On Sun, 22 Feb 2004 16:13:01 +, Tim Bunce wrote: Now I do agree the HTTP protocol is worth using for managing the flow of data. I run HTTP over ssh myself. (I modified HTTP::Daemon to use two unidirectional pipes instead of a single, bidirectional socket.) Has (will) that be included in future releases? Yes. I plan on submitting the changes back after the bits have a chance to dry. I want to advance my current project to the point where I can stress-test the HTTP::Daemon and UserAgent modifications before I submit the changes. I'll subscribe to the libwww mailing list since that seems to be the right forum for posting the changes. If you want a copy of the HTTP code as-is just drop me a line. Scott
Re: Distributed testing idea
Le lundi 23 février 2004 à 14:06, Michael G Schwern écrivait: I'm going with just straight, unsecured socket communications and an ad hoc protocol. At this point, encryption is not necessary. There's nothing worth encrypting. To see why, look at the example protocol conversation at http://www.makemaker.org/wiki/?AutomatedTestDistribution Its really that simple. Here are a few questions and proposals: * didn't you swap server and client in your example? * I plan to set up several boxes for testing (when I get one of those round tuits ;-), but I only have one IP. It would be great if the protocol could let the server be a front end to several smoke boxes. - the easiest way would be for the server to act as a client for the other smoke boxes (but that implies the test would be run on all available platforms) - or maybe the protocol could support having the server inform the client of what's available, and let the client decide There's still a need for server-to-server communication... - maybe it's not to much to ask that the various test platforms can access the web via a proxy. So that URLs could be passed around easily. * couldn't the work be done asynchronously? - so this part of the protocol would be useless: Server: Great. Hang around. Client: Waiting. Server: Please hold. Client: Waiting. ... - information about what went right or wrong would be sent back via email. Since we are talking of smoke boxes, the clients should be able to fetch the emails via POP3 or whatever. And get the useful information from them. - a client would connect to a test server and ask: Client: what test platform do you manage? Server: X1/Y1/Z1 X2/Y1/Z2 ... Client: (checks its matrice) Great. Test the code at this URL on the following platforms X2/Y1/Z2 X3/Y3/Z3 Server: OK - since you talk about a central repository of public keys, and testing platforms, the client does not really need to ask what's available, since the information should be cached in the central repository. So ideally, a script could check all the available test platforms from the central repository and contact only the server it wants. - the central repository would then just be a client that regularly ask for the available test platform on each of the server it knows and trust. * the reports could be send by email, but also be made available from the server. - For example, the client would say: Client: Test the code at this URL on the following platforms X2/Y1/Z2 X3/Y3/Z3 Server: X2/Y1/Z2 will be report number 1234 X3/Y3/Z3 will be report number 1235 - So the client could also fetch the report from the server by asking for its number. Or all the reports could also be stored in a central place (think Cc:), but then we need a scheme for unique report ids. - the central repository could also fetch the test reports regularly, rather than waiting for email (for example if we don't trust emails). The conversation could then be something like: Client: Any reports after xxyyzzz-1234? Server: Yes. I have: xxyyzzz-1235 xxyyzzz-1236 Client: Gimme xxyyzzz-1235 Server: Here goes: ... Client: Gimme xxyyzzz-1236 Server: Here goes: ... If you need help for coding parts of your client/server/protocol (no matter how much you like or dislike my proposals), I'd love to help. Small chunks, if possible, so that I can deliver on time. :-) -- Philippe BooK Bruhat Wisdom cannot be bought. It can, at best, be rented until you can find your own. (Moral from Groo The Wanderer #66 (Epic))
Re: Distributed testing idea
On Mon, Mar 01, 2004 at 05:37:39PM +0100, Philippe 'BooK' Bruhat wrote: I'm going with just straight, unsecured socket communications and an ad hoc protocol. At this point, encryption is not necessary. There's nothing worth encrypting. To see why, look at the example protocol conversation at http://www.makemaker.org/wiki/?AutomatedTestDistribution Its really that simple. Here are a few questions and proposals: * didn't you swap server and client in your example? From what I originally posted? Yes. I've ruthlessly chopped the hell out of it to get down to just what I need to get something useful off the ground. The scheme is now that users wishing to help simply run the testing client which connects to my server and waits for instructions, similar to Distributed.net. * I plan to set up several boxes for testing (when I get one of those round tuits ;-), but I only have one IP. It would be great if the protocol could let the server be a front end to several smoke boxes. Since your tesing machines are clients connecting to a server it doesn't matter if they're behind NAT. Another simplification afforded by making it a straight client/server connection. * couldn't the work be done asynchronously? No. Time from initiating tests to getting the results back would be too slow. Asynch is too complicated to manage and I'd have to parse email in order to fully automate it. Yuck. Futhermore, there's no point in complicating it. Its a little client daemon that idles in the background waiting for instructions from the server exactly like you'd run a distributed.net client. One client per server per Perl configuration. - since you talk about a central repository of public keys This unnecessary complication has been eliminted. Individual authors looking for testing machines are on their own for the time being. -- Michael G Schwern[EMAIL PROTECTED] http://www.pobox.com/~schwern/ They just don't make any good porn music anymore, do they? - WXDX DJ refering to More, More, More
Re: Distributed testing idea
Michael G Schwern sent the following bits through the ether: So what I need is some way to set up a network of test servers such that I can say test this module for me and my testing client would ship it to as many test servers as it can find and get the results back all in just a few minutes. The scheme that you propose sounds awfully like a messaging system. Leon -- Leon Brocard.http://www.astray.com/ scribot.http://www.scribot.com/ ... But I don't *like* ponies!
Re: Distributed testing idea
Le lundi 01 mars 2004 à 16:53, Leon Brocard écrivait: Michael G Schwern sent the following bits through the ether: So what I need is some way to set up a network of test servers such that I can say test this module for me and my testing client would ship it to as many test servers as it can find and get the results back all in just a few minutes. The scheme that you propose sounds awfully like a messaging system. Which means the server and client could communicate as IRC bots, with Net::IRC or a similar module? The client would say test the code at this URL for me in public, and all the servers listening on the public channel would fetch the package, test it and /msg the result reports to the client? -- Philippe BooK Bruhat When you create a climate of peace, you have only fair weather. But where the climate is one of violence, it can only rain blood. (Moral from Groo The Wanderer #120 (Epic))
Re: Distributed testing idea
On Mar 1, 2004, at 9:12 AM, Philippe 'BooK' Bruhat wrote: Which means the server and client could communicate as IRC bots, with Net::IRC or a similar module? Jabber. David
Re: Distributed testing idea
On Sun, Feb 22, 2004 at 12:35:03PM +, Nick Ing-Simmons wrote: See above. Yes, ssh is not portable enough. Well runs on Linux/Unix and Win32 has at least a client so which platforms is VMS the problem? VMS and Windows. While it exists for Windows, it typically isn't a simple command line utility like OpenSSH. For example, I don't know if PuTTY is scriptable. Anyhow, encryption is not necessary. There's nothing secret passing back and forth. The main thing to defend against is a man-in-the-middle attack which is handled by the simple public/private key authentication I mentioned earlier. Thing is who is going to give access to their machine(s) via some ad. hoc. scheme? I am not for a start. Why would the method of network transport figure into that decision? -- Michael G Schwern[EMAIL PROTECTED] http://www.pobox.com/~schwern/ Do not try comedy at home! Milk Cheese are advanced experts! Attempts at comedy can be dangerously unfunny!
Re: Distributed testing idea
On Sun, Feb 22, 2004 at 06:36:22AM -0600, Scott Bolte wrote: On Sun, 22 Feb 2004 01:07:38 -0500, Michael G Schwern wrote: See above. Yes, ssh is not portable enough. Where is the gap? I have OpenSSH on every Unix platform I use and, with cygwin's help, all the windows based systems too. I can even log *into* a w98 host with ssh. Anything which is not Unix is the gap. Namely VMS and non-Cygwin Windows. On the Windows end, I'd rather not depend that they have Cygwin installed. Also, if the client runs in the Cygwin environment it executes the Cygwin Perl. It will be difficult to test as straight Windows. Resolving the client-side ssh installation problem is not worth the trouble. I'm going with just straight, unsecured socket communications and an ad hoc protocol. At this point, encryption is not necessary. There's nothing worth encrypting. To see why, look at the example protocol conversation at http://www.makemaker.org/wiki/?AutomatedTestDistribution Its really that simple. -- Michael G Schwern[EMAIL PROTECTED] http://www.pobox.com/~schwern/ Ya gotta figure whatever you come up with is gonna be about as portable as a flypress. -- Brian Ingerson on advanced make usage
Re: Distributed testing idea
Michael G Schwern [EMAIL PROTECTED] writes: On Thu, Feb 19, 2004 at 08:35:28AM +, Nick Ing-Simmons wrote: Michael G Schwern [EMAIL PROTECTED] writes: One thing to keep in mind is portability. In order for this to be useful it has to run on pretty much all platforms. Unix, Windows, VMS, etc... So I'm trying to keep it as simple as possible. *snip* HTTPS might be overkill, we don't need to encrypt the communications, just identify the server. A simple thing to do would be for my server to have a public/private key pair. How about layering it on ssh then? See above. Yes, ssh is not portable enough. Well runs on Linux/Unix and Win32 has at least a client so which platforms is VMS the problem? Thing is who is going to give access to their machine(s) via some ad. hoc. scheme? I am not for a start.
Re: Distributed testing idea
On Sun, 22 Feb 2004 01:07:38 -0500, Michael G Schwern wrote: See above. Yes, ssh is not portable enough. Where is the gap? I have OpenSSH on every Unix platform I use and, with cygwin's help, all the windows based systems too. I can even log *into* a w98 host with ssh. Now I do agree the HTTP protocol is worth using for managing the flow of data. I run HTTP over ssh myself. (I modified HTTP::Daemon to use two unidirectional pipes instead of a single, bidirectional socket.) The twist is each remote HTTP 'server' is started on-demand and is dedicated to the task at hand. The how/when start-up problem is eliminated. One major advantage is it works equally well to create local or remote servers. Another advantage is security. By using ssh forced commands, I have both authentication and authorization covered. With HTTPS you only get authentication. Scott
Re: Distributed testing idea
On Sun, Feb 22, 2004 at 06:36:22AM -0600, Scott Bolte wrote: On Sun, 22 Feb 2004 01:07:38 -0500, Michael G Schwern wrote: See above. Yes, ssh is not portable enough. Where is the gap? I have OpenSSH on every Unix platform I use and, with cygwin's help, all the windows based systems too. I can even log *into* a w98 host with ssh. Now I do agree the HTTP protocol is worth using for managing the flow of data. I run HTTP over ssh myself. (I modified HTTP::Daemon to use two unidirectional pipes instead of a single, bidirectional socket.) Has (will) that be included in future releases? Tim.
Re: Distributed testing idea
On Sun, 22 Feb 2004 16:13:01 +, Tim Bunce wrote: Now I do agree the HTTP protocol is worth using for managing the flow of data. I run HTTP over ssh myself. (I modified HTTP::Daemon to use two unidirectional pipes instead of a single, bidirectional socket.) Has (will) that be included in future releases? Yes. I plan on submitting the changes back after the bits have a chance to dry. I want to advance my current project to the point where I can stress-test the HTTP::Daemon and UserAgent modifications before I submit the changes. I'll subscribe to the libwww mailing list since that seems to be the right forum for posting the changes. If you want a copy of the HTTP code as-is just drop me a line. Scott
Re: Distributed testing idea
On Wed, Feb 18, 2004 at 06:49:19PM -0600, Ken Williams wrote: 1) In order to be convenient for the code author, he/she should be able to poll for available clients before submitting a job. My inclination would be to make this a simple inetd on the client, rather than any persistent connection between client server. I think if there were daemons on the client, or if the client had to connect to a daemon on the server, lots of clients would forget to do it when they rebooted. Its just as easy to make the client start on reboot as it is to add it to your inetd config. -- Michael G Schwern[EMAIL PROTECTED] http://www.pobox.com/~schwern/ Cuius rei demonstrationem mirabilem sane detexi hanc subscriptis exiguitas non caperet.
Re: Distributed testing idea
On Thu, Feb 19, 2004 at 08:35:28AM +, Nick Ing-Simmons wrote: Michael G Schwern [EMAIL PROTECTED] writes: One thing to keep in mind is portability. In order for this to be useful it has to run on pretty much all platforms. Unix, Windows, VMS, etc... So I'm trying to keep it as simple as possible. *snip* HTTPS might be overkill, we don't need to encrypt the communications, just identify the server. A simple thing to do would be for my server to have a public/private key pair. How about layering it on ssh then? See above. Yes, ssh is not portable enough. -- Michael G Schwern[EMAIL PROTECTED] http://www.pobox.com/~schwern/ I've just gone through a lung-crushing breakup with my blender and I don't think I should screw my forehead alone tonight.
Re: Distributed testing idea
On Thu, 19 Feb 2004 08:35:28 +, Nick Ing-Simmons wrote: How about layering it on ssh then? That has done all authentication stuff already. I agree. With empty passphrases and/or ssh-agent it is very easy to set up automated commands. Add forced commands (an under-appreciated feature of ssh) to restrict what can be done on the remote host and you've excellent security control over the entire environment. Scott
Re: Distributed testing idea
Michael G Schwern [EMAIL PROTECTED] writes: One thing to keep in mind is portability. In order for this to be useful it has to run on pretty much all platforms. Unix, Windows, VMS, etc... So I'm trying to keep it as simple as possible. On Wed, Feb 18, 2004 at 05:29:49PM +, Adrian Howard wrote: -If this is going to be run by paranoid people everything would have to be over https to prevent man-in-the-middle attacks on the code being transported HTTPS might be overkill, we don't need to encrypt the communications, just identify the server. A simple thing to do would be for my server to have a public/private key pair. How about layering it on ssh then? That has done all authentication stuff already. The clients ship with or they download my public key. Then later when they communicate they encrypt a little string with my public key, I decrypt it and send it back for verification. Simple.
Re: Distributed testing idea
On Wednesday, February 11, 2004, at 09:24 pm, Michael G Schwern wrote: The biggest time suck in developing MakeMaker, and to a lesser extent Test::More, is running the tests. Why? Because they need to be run on lots of different platforms with lots of different versions of Perl. Currently, I do this by hand. And we all know manual testing sucks. Its time consuming and you tend to avoid it. I can't run the tests on every platform at every patch so I often wind up breaking something and not realizing it for a while. So what I need is some way to set up a network of test servers such that I can say test this module for me and my testing client would ship it to as many test servers as it can find and get the results back all in just a few minutes. [interesting outline implementation snipped] Random comments. I have zero available tuits to help implementation so feel free to ignore and/or laugh: - Nice idea - If this is going to be run by paranoid people everything would have to be over https to prevent man-in-the-middle attacks on the code being transported - I've done a vaguely related task in the past. Instead of distributing one test suite over several machines, I was distributing bits of a single test suite over several machines (so I could run the test scripts in parallel and decrease the overall runtime of the whole test suite). Would be nice if we had something flexible enough to cope with both scenarios - but that might make it too complex to implement in a day or so :-) - I solved my problem with SSH rather than HTTP since I had the infrastructure for it in place on the machines I was playing with. Might be worth considering as an alternative to HTTP[S] - Would you want to deal with cannot test (e.g. because the test server didn't have the necessary prerequisite modules rather than the test itself timing out or there being a communication problem) as well as pass/fail? - Some mechanism to automatically gather/report the base platforms would seem to be a good idea. Otherwise you are going to have people forgetting to keep the central list up to date when then update their box. Cheers, Adrian
Re: Distributed testing idea
On 2/12/2004 9:07 PM, Randy W. Sims wrote: How about a model that allows people to volunteer when they can (vs always). Say you have a server. You would be an author-client. You'd say I have this software that needs to be tested. The server would take it and mark it available for testing. Then test-clients could log-on (or more probably set a cron job or login script). The server would say to clients that login, Ok, I have these jobs that need testing. The problem is turn around time. In order for this to be useful, I need to be able to make a change to the code and within the span of five or ten minutes get results back from the system. The distributed testing would be used pretty much like 'make test'. The other problem is I don't see that always volunteering is inconvenient. You run a client program that connects to the test server and idles in the background until contacted by the server with work. Then it does an automated build/test with no intervention from the user. No human required. Actually, why not just setup your own CPAN. Testers can point CPAN++ at your CPAN server and it's mostly done. You just need a script to drive CPAN++ for testers to use to make things easier for them. Easy, simple, and effective if you can get people testing. Turn around time too slow and requires a human. -- Michael G Schwern[EMAIL PROTECTED] http://www.pobox.com/~schwern/ purl Hey Schwern! honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk, honk!
Re: Distributed testing idea
On 2/16/2004 5:25 PM, Michael G Schwern wrote: On 2/12/2004 9:07 PM, Randy W. Sims wrote: How about a model that allows people to volunteer when they can (vs always). Say you have a server. You would be an author-client. You'd say I have this software that needs to be tested. The server would take it and mark it available for testing. Then test-clients could log-on (or more probably set a cron job or login script). The server would say to clients that login, Ok, I have these jobs that need testing. The problem is turn around time. In order for this to be useful, I need to be able to make a change to the code and within the span of five or ten minutes get results back from the system. The distributed testing would be used pretty much like 'make test'. The idea (which, admittedly is far from ideal) is to have enough volunteers that a machine would always be available. Systems would be coming on and going off, but *hopefully* there will always a system online for testing. The other problem is I don't see that always volunteering is inconvenient. You run a client program that connects to the test server and idles in the background until contacted by the server with work. Then it does an automated build/test with no intervention from the user. No human required. The problem that I was trying to overcome with my suggestion is that, for examle, I have several computers at my house running Debian GNU/Linux and Windows 2000 (and soon will have one running OpenBSD), but I don't keep those systems up all the time - they generate too much heat. I could make them available part-time, so that when they're available they could poll a server to see if anything needs to be tested. If it does it downloads, runs the tests, and sends off a test report - which is what CPAN++ was made for. Actually, why not just setup your own CPAN. Testers can point CPAN++ at your CPAN server and it's mostly done. You just need a script to drive CPAN++ for testers to use to make things easier for them. Easy, simple, and effective if you can get people testing. Turn around time too slow and requires a human. The polling, downloading, testing, reporting would all be scripted. No human intervention. It may not be the best solution (although, I don't see a down side), but it would likely be the easiest to implement, and I think it completely side-steps any security issues. Randy.
Re: Distributed testing idea
Some thoughts: 1) In order to be convenient for the code author, he/she should be able to poll for available clients before submitting a job. My inclination would be to make this a simple inetd on the client, rather than any persistent connection between client server. I think if there were daemons on the client, or if the client had to connect to a daemon on the server, lots of clients would forget to do it when they rebooted. 2) I'd be very hesitant to let anyone do this on my machine. I don't worry [much] about malicious code, but when I'm testing non-ready-for-CPAN code, I have done some pretty dumb things. At the very least, I'd create a new user account for the testing processes and give it very low priorities. I also think I'd want to set it up so that I only give access to certain code authors, and they *MUST* be properly signed with a SIGNATURE file. 3) I'd sure be interested in it for Module::Build. -Ken
Re: Distributed testing idea
One thing to keep in mind is portability. In order for this to be useful it has to run on pretty much all platforms. Unix, Windows, VMS, etc... So I'm trying to keep it as simple as possible. On Wed, Feb 18, 2004 at 05:29:49PM +, Adrian Howard wrote: - If this is going to be run by paranoid people everything would have to be over https to prevent man-in-the-middle attacks on the code being transported HTTPS might be overkill, we don't need to encrypt the communications, just identify the server. A simple thing to do would be for my server to have a public/private key pair. The clients ship with or they download my public key. Then later when they communicate they encrypt a little string with my public key, I decrypt it and send it back for verification. Simple. Not something I'm going to worry about in the beginning, but definately later. - I've done a vaguely related task in the past. Instead of distributing one test suite over several machines, I was distributing bits of a single test suite over several machines (so I could run the test scripts in parallel and decrease the overall runtime of the whole test suite). Would be nice if we had something flexible enough to cope with both scenarios - but that might make it too complex to implement in a day or so :-) Can I get a pony with that? Thanks. ;) - I solved my problem with SSH rather than HTTP since I had the infrastructure for it in place on the machines I was playing with. Might be worth considering as an alternative to HTTP[S] Perl has native HTTP libraries. Perl doesn't have native ssh libraries. From a portability standpoint, the choice is clear. ssh also requires that the clients have accounts on the server machine. Extra complexity and security issues. In reality I'll probably forgo either of them and just write a very simple network daemon to speak a simple text protocol. Very little information needs to be passed back and forth. - Would you want to deal with cannot test (e.g. because the test server didn't have the necessary prerequisite modules rather than the test itself timing out or there being a communication problem) as well as pass/fail? Part of the client's logic is to be able to download and install the prereqs, probably just assume there's a working CPAN.pm installation. They'd be installed in a seperate directory so as not to mess with the system Perl libraries. - Some mechanism to automatically gather/report the base platforms would seem to be a good idea. Otherwise you are going to have people forgetting to keep the central list up to date when then update their box. This I don't understand. There's nothing for the user to keep up to date by hand. -- Michael G Schwern[EMAIL PROTECTED] http://www.pobox.com/~schwern/ HA HA HA You're all so rediculous! But thanks for the money!
Re: Distributed testing idea
Le mercredi 11 février 2004 à 13:24, Michael G Schwern écrivait: Here's a simple sketch of what I'd imagine the protocol would look like. I'm not much on networking, so I'm going to keep it simple. HTTP is simple. I'd also like to make running a test server as simple and portable as possible. Web servers are simple and portable. How do you see this server? A collection of CGI scripts? A simple HTTP::Daemon based script? Server: Returns ok, I'll run this for you with $these versions of Perl. Pick up your results at this URL or sorry, I can't run this right now Maybe the server could send back an email when the test run is finished, so that the client does not request the server for naught. The URL would be given in the email. Client: If the server is willing to run the code, we poll the results URL until we get results or a timeout. If the server times out or won't run the tests, we move onto the next available server that can cover this OS/version combo. Avoid polling the servers for nothing, maybe? The advantage of email over the web for this part, is that it's asynchronous, that the client does not need to contact the server until it knows the resultat data is ready. Would the test results be pushed back into cpan-testers? Since we're basically setting up a big network to run arbitrary code, it cannot be made available to just anyone with a CPAN id. Credentials become a problem. There has to be access control on the users of the system. We also must make sure that someone can't attach a test server which harvests user's credentials. I also want to prevent every test server from contacting the master server for each user wanting to run tests to avoid undue load. Since a PAUSE id is not enough, credentials could be handled we public key systems. So that a server knows that the person is authorized to run tests, but cannot harvest the credentials themselves. A list of public keys could be made available on the master server, and a test server would need to fetch it only when someone new contacts it. Finally, I want to allow the system to still work without the master server (clients can cache the test server list). The last two may be asking too much. Where are the tests results stored? On the master server? -- Philippe BooK Bruhat The surest path to chaos is too much organization. (Moral from Groo #4 (Image))
Re: Distributed testing idea
On 2/11/2004 4:24 PM, Michael G Schwern wrote: The biggest time suck in developing MakeMaker, and to a lesser extent Test::More, is running the tests. Why? Because they need to be run on lots of different platforms with lots of different versions of Perl. Currently, I do this by hand. And we all know manual testing sucks. Its time consuming and you tend to avoid it. I can't run the tests on every platform at every patch so I often wind up breaking something and not realizing it for a while. So what I need is some way to set up a network of test servers such that I can say test this module for me and my testing client would ship it to as many test servers as it can find and get the results back all in just a few minutes. How about a model that allows people to volunteer when they can (vs always). Say you have a server. You would be an author-client. You'd say I have this software that needs to be tested. The server would take it and mark it available for testing. Then test-clients could log-on (or more probably set a cron job or login script). The server would say to clients that login, Ok, I have these jobs that need testing. The client would take any module (or might possibly have a list of preferred modules to test), test it, and send results. I'd suggest using the cpan-testers backend for this; that way, the jobs mostly done. All you have to do is write a cgi script to accept uploads from authors and give clients an appropriate link to retrieve the module for testing. Instant compiler-farm. Randy.
Re: Distributed testing idea
On 2/12/2004 9:07 PM, Randy W. Sims wrote: How about a model that allows people to volunteer when they can (vs always). Say you have a server. You would be an author-client. You'd say I have this software that needs to be tested. The server would take it and mark it available for testing. Then test-clients could log-on (or more probably set a cron job or login script). The server would say to clients that login, Ok, I have these jobs that need testing. The client would take any module (or might possibly have a list of preferred modules to test), test it, and send results. I'd suggest using the cpan-testers backend for this; that way, the jobs mostly done. All you have to do is write a cgi script to accept uploads from authors and give clients an appropriate link to retrieve the module for testing. Instant compiler-farm. Actually, why not just setup your own CPAN. Testers can point CPAN++ at your CPAN server and it's mostly done. You just need a script to drive CPAN++ for testers to use to make things easier for them. Easy, simple, and effective if you can get people testing. Regards, Randy.
Distributed testing idea
The biggest time suck in developing MakeMaker, and to a lesser extent Test::More, is running the tests. Why? Because they need to be run on lots of different platforms with lots of different versions of Perl. Currently, I do this by hand. And we all know manual testing sucks. Its time consuming and you tend to avoid it. I can't run the tests on every platform at every patch so I often wind up breaking something and not realizing it for a while. So what I need is some way to set up a network of test servers such that I can say test this module for me and my testing client would ship it to as many test servers as it can find and get the results back all in just a few minutes. Here's a simple sketch of what I'd imagine the protocol would look like. I'm not much on networking, so I'm going to keep it simple. HTTP is simple. I'd also like to make running a test server as simple and portable as possible. Web servers are simple and portable. Client: GETs master.test.server.list containing what OS / versions of Perl they're testing with and their availability. The client then builds an OS / Perl version matrix and contacts each server to necessary to fill the matrix, rather than simply all available servers. Client: POSTs to some.test.server saying Here's my credentails and the URL where you can get the source code I'd like you to test Server: Returns ok, I'll run this for you with $these versions of Perl. Pick up your results at this URL or sorry, I can't run this right now Client: If the server is willing to run the code, we poll the results URL until we get results or a timeout. If the server times out or won't run the tests, we move onto the next available server that can cover this OS/version combo. Loop until the matrix is filled or we run out of servers to contact or timeout. And that's about it. The results can be as simple as the verbose T::H output plus a simple boolean indicating overall pass/fail. Here's the hard problems: Since we're basically setting up a big network to run arbitrary code, it cannot be made available to just anyone with a CPAN id. Credentials become a problem. There has to be access control on the users of the system. We also must make sure that someone can't attach a test server which harvests user's credentials. I also want to prevent every test server from contacting the master server for each user wanting to run tests to avoid undue load. Finally, I want to allow the system to still work without the master server (clients can cache the test server list). The last two may be asking too much. A strong sandbox for the test servers to run the code in would be nice, however since the users of the network are authenticated we don't need to guard too strongly anymore than, say, CPANPLUS does. Since its difficult to do this in a cross-platform manner, I'm tempted to push the responsibility for setting up the sandbox off to the individual test server admin. About the only thing our code would do is install module dependencies in its own directory outside the normal @INC. The most important thing to remember about a project like this is KISS. Keep It Simple, Stupid. Features can rapidly spiral out of control, and it'll never get done. But if we keep it simple something workable can probably be knocked together in a few days. I'm putting this up on the MakeMaker Wiki for collaborative refinement. http://www.makemaker.org/wiki/?AutomatedTestDistribution -- Michael G Schwern[EMAIL PROTECTED] http://www.pobox.com/~schwern/ If at first you don't succeed, call in an air strike.
