I took a break from the studying I ought be doing, in order to tinker with
multi-threading. Heck, I've suffered all this pain w/ Python (my own doing,
no doubt) so I might as well get some fun out of it. I like the results.
Since cvs|svn|whatever are typically network latency/IO bound, there is
quite a benefit (in terms of length of run) to having a few of these running
simultaneously. Since we update to a different directory per module, and
then sync to their separate working directories, with no overlaps, we have a
perfect chore for multithreading.
Basically the list of modules to update is generated from the list (in
order) of projects to build. We have a pool of worker threads (I made it 5,
but that may be too many, could be too few on Brutus) and this pool work in
addition to the main thread. The main thread works through the project list
and when it needs a module checked out, it either waits for the workers to
do it, or does it itself in it's own thread. [Python seems to have nice
simple lock and semaphore code. Look at gump.treads.tools if you like.]
I've added the following to the workspace so we can experiment with this:
<!-- Numbers of *extra* threads, besides the main one. -->
<threads updaters="5" builders="0" />
The 'risk' with this change that we are running fair chunks of code
simultaneously & (currently) the few places the code could overlap (the run
lists) are not synchronized (since Python has no such concept on objects,
locks are separate). I think we are pretty lucky with this (considering)
because we have so few points of overlap, but *maybe* also Python is
reasonably thread-safe. I'm not sure yet, and over time I'll work on locking
more.
I'm not quite ready to do the same with building (there is more interaction
there, and thread work detection is a little trickier). I hope to get there,
primarily 'cos Brutus has two CPUs to play on.
I will leave this feature in on gump.try for a while, and give it some runs
in order to get comfortable with how reliable it is.
http://gump.try.sybase.com/buildLog.html
Feedback welcomed.
regards,
Adam
--
Experience the Unwired Enterprise:
http://www.sybase.com/unwiredenterprise
Try Sybase: http://www.try.sybase.com
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
