For our project one of the top goals was minimizing build times; so
being able to invoke compilers/linkers in parallel to take advantage of
multiple-core computers was a necessity.
(Since finishing my implementation, I have learned about drake. I intend
to take a look at that as a possible alternative approach.)
The user specifies "j=N" on the command line, similar to make's -jN.
When N>1, tasks are created as MultiTasks. Since FileTasks aren't
MultiTasks, when there's a library or application, a MultiTask is
created that has all the object files as prerequisistes; then the
library/app includes that MultiTask in its prerequisites. This allows
the object files to compile in parallel.
Since MultiTasks run all their prereqs at the same time, I made a custom
version of the 'sh' method which synchronizes using a semaphore; that
semaphore is initialized with the j=N value that the user specified.
That way, only N commands will be invoked at once.
Recently, I tried updating to ruby 1.9.1. My MultiTask approach fell
down because it created too many threads. 1.9 uses 'real' threads which
consume more resources than 1.8's 'fake' (but very lightweight) threads.
When I modified MultiTask to spawn threads in batches to limit their
number, it worked but it was many times slower than before, probably due
to the additional overhead of creating/joining lots of OS threads.
So we're sticking with ruby 1.8 for now. As I said, perhaps drake's
approach will be better for 1.9, I'll try that out sometime.
-heath
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
_______________________________________________
Rake-devel mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/rake-devel
