Hi Adam,
this is a good idea, my experience with gump is also that the updates are taking ages.
Cheers,
Antoine
Adam R. B. Jack wrote:
I took a break from the studying I ought be doing, in order to tinker with multi-threading. Heck, I've suffered all this pain w/ Python (my own doing, no doubt) so I might as well get some fun out of it. I like the results.
Since cvs|svn|whatever are typically network latency/IO bound, there is quite a benefit (in terms of length of run) to having a few of these running simultaneously. Since we update to a different directory per module, and then sync to their separate working directories, with no overlaps, we have a perfect chore for multithreading.
Basically the list of modules to update is generated from the list (in order) of projects to build. We have a pool of worker threads (I made it 5, but that may be too many, could be too few on Brutus) and this pool work in addition to the main thread. The main thread works through the project list and when it needs a module checked out, it either waits for the workers to do it, or does it itself in it's own thread. [Python seems to have nice simple lock and semaphore code. Look at gump.treads.tools if you like.]
I've added the following to the workspace so we can experiment with this:
<!-- Numbers of *extra* threads, besides the main one. --> <threads updaters="5" builders="0" />
The 'risk' with this change that we are running fair chunks of code simultaneously & (currently) the few places the code could overlap (the run lists) are not synchronized (since Python has no such concept on objects, locks are separate). I think we are pretty lucky with this (considering) because we have so few points of overlap, but *maybe* also Python is reasonably thread-safe. I'm not sure yet, and over time I'll work on locking more.
I'm not quite ready to do the same with building (there is more interaction there, and thread work detection is a little trickier). I hope to get there, primarily 'cos Brutus has two CPUs to play on.
I will leave this feature in on gump.try for a while, and give it some runs in order to get comfortable with how reliable it is.
http://gump.try.sybase.com/buildLog.html
Feedback welcomed.
regards,
Adam
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]