Sean, et al.,
You do seem to understand my system correctly: dmucs is a wrapper around distcc which is a wrapper around gcc.
Note that we here at marconi use pcons, not make to build our tree. pcons is smarter than make because it calculates the entire tree of build dependencies before beginning to compile even the first file. So, it can know which things to do in which order. Now, in my opinion, pcons isn't quite smart enough, because it doesn't parallelize compiles across multiple directories -- that would REALLY speed up the compile, I think.
I think distcc should be able to have the option, where instead of compiling immediately, it can store what the build will be in a file that is a sorted tree of dependencies. A very simple wrapper for make could take care of this by replacing gcc with distcc and the specified option, then running distcc on the tree file. This way, builds can be in parallel even in different directories.
I have thought of other ways to do this, with a make wrapper, executing "make -n" then running every command, and if a given command fails (returns 1), move on to the next one, if a command fails, then move on to the next looping around until everything is built, or everything left fails. I know this isn't pretty, but it should be a lot easier to implement.
The nice things about dmucs are:
o if other engineers are using the compile machines for other tasks, the loadavg daemons will communicate the loads to the dmucs server, and those machines will be used less often for compiles -- so my system takes into account not just loadavg based on number of compiles on a machine, but on other factors as well.
Distcc already does this to some degree, if you send a job to a heavily loaded computer, it will take a long time to compile. More jobs will be sent to computers that are not loaded (in the same amount of time) If the computer is at 100% cpu this would still be a major problem if distccd is run with a lower priority (nice)
I think if a slow computer is working on the last file, and other computers are ready, it should try to compile it on other computers as well, and finish when it gets the compiled result from anyone.
o machines can be added and deleted from the compilation farm dynamically. For example, a powerful desktop machine on an engineers desktop can be added to the compilation farm when it goes into screen saver mode. When the eningeer comes back to his/her desk, the machine can be immediately removed from the farm. I've tried this, and it works well.
o machines with differing "power" can all contribute to the compilation farm. Each machine is assigned a "power index", and the most powerful, available, and not-overloaded machines will be given out for compiles. If these machines become overloaded, then less powerful machines will be given out. This provides the best compilation times possible whenever the engineers does the compile.
This sounds like a good start, maybe the computers running should have a way to feedback and update their "power index"
o the system scales really well. I've tried it with 6 compilations going simultaneously, and the time to compile our code based went up from 44 minutes to 56 minutes -- an increase of 27%. That's not bad for a six-fold increase in the number of compilation requests happening.
So it takes 44 minutes to compile on one computer, and 56 minutes to compile 6 times using 6 computers? Can you test the difference without your wrapper but still using distcc? Ideally we will get this down to 44 minutes and 44 minutes :-P
o ideally, it would be very nice to be able to know how long different files take to compile, and what order to do them in to produce the very best use of the compilation farm. But that is probably an NP-complete problem, and I needed to bite off something I could chew.
I agree, but maybe we can set things up now so it won't be hard to do that later?
Can we download and test it? I would like to see if it really speeds things up, it's obvious it adds useful features.
-sean
__ distcc mailing list http://distcc.samba.org/ To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/distcc
