Re: How Do Parallel Builds Scale?

2012-11-18 Thread NightStrike
On Sat, Nov 17, 2012 at 7:30 AM, Bob Friesenhahn
 wrote:
> I suspect that this anaysis has been mentioned on the Automake list before
> but (if so) it is worth looking at again.  It seems that few packages
> benefit significantly from parallel builds.  Many packages use Automake, but
> they use it in a very inefficient way.  Here is a web page which shows the
> current pitiful state of parallel builds:
>
>   http://hubble.gforge.inria.fr/parallel-builds.html
>
> Given that even desktop and laptop computers support 4 CPU cores and that
> affordable server type hardware is available with as many as 64 CPU cores,
> it is useful if the build process of packages is redone so that it benefits
> from compilation on modern hardware.

http://mingw-w64.sf.net/ makes excessive use of massive parallelism.
In fact, we can build every single file in parallel, thanks to a
strong focus on non-recursive make.  At one point in our build
process, we have to build several thousand libX.a files from
corresponding X.def files.  This can be parallelized down to 1 job per
library.



How Do Parallel Builds Scale?

2012-11-17 Thread Bob Friesenhahn
I suspect that this anaysis has been mentioned on the Automake list 
before but (if so) it is worth looking at again.  It seems that few 
packages benefit significantly from parallel builds.  Many packages 
use Automake, but they use it in a very inefficient way.  Here is a 
web page which shows the current pitiful state of parallel builds:


  http://hubble.gforge.inria.fr/parallel-builds.html

Given that even desktop and laptop computers support 4 CPU cores and 
that affordable server type hardware is available with as many as 64 
CPU cores, it is useful if the build process of packages is redone so 
that it benefits from compilation on modern hardware.


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/