On Mon, Nov 28, 2011 at 12:13 AM, Pandu Poluan <pa...@poluan.info> wrote: > On Nov 28, 2011 11:38 AM, "Michael Mol" <mike...@gmail.com> wrote: >> On Sun, Nov 27, 2011 at 7:54 PM, Pandu Poluan <pa...@poluan.info> wrote: >> > On Nov 28, 2011 6:24 AM, "Neil Bothwick" <n...@digimed.co.uk> wrote: >> >> On Mon, 28 Nov 2011 00:56:17 +0700, Pandu Poluan wrote: >> >> >> >> > I don't know where the 'blame' lies, but I've found myself >> >> > standardizing on MAKEOPTS=-j3, and PORTAGE_DEFAULT_OPTS="--jobs >> >> > --load-average=<1.6*num_of_vCPU>" >> >> > >> >> > (Yes, no explicit number of jobs. The newer portages are smart enough >> >> > to >> >> > keep starting new jobs until the load number is reached) >> >> >> >> The problem I found with that is the ebuilds load the system lightly to >> >> start with, before they enter the compile phase, to portage starts >> >> dozens >> >> of parallel ebuilds, then the system gets completely bogged down when >> >> they start compiling. >> >> >> > >> > Yes, sometimes that would happen if at the beginning there are >> > network-bound >> > ebuilds all downloading their respective distfiles. The load stays low >> > until >> > they all start ./configure-ing roughly at the same time. Then all hell >> > breaks loose. >> > >> > I successfully mitigate such "load-explosion" by doing a --fetchonly >> > step >> > first, and keeping MAKEOPTS at low -j (which, in my case, is actually >> > required). >> > >> > Just to add more info: I use USE=graphite (with some CFLAGS, uh, >> > 'enhancements') with gcc-4.5.3. IIRC, I could push MAKEOPTS up to -j5 >> > (and >> > even more, but I ran out of cores) when I was still using gcc-4.4.x and >> > no >> > USE=graphite. >> > >> > Won't file a bug report, though. I have a feeling that my bug report re: >> > emerge failure will be marked WONTFIX thanks to the 'ricer special' >> > CFLAGS >> >> As I noted, "-l" in MAKEOPTS takes care of the load explosion very nicely. > > Most likely so. I am not aware of -l in MAKEOPTS before, so what I posted > was my workaround to prevent load explosion. Thanks to your very useful tip, > I now no longer have to worry about load explosion :-) > > (I still like doing pre-fetchonly-ing, though. But now for a different main > reason :-)
The explosion of information in this thread is going to make for a *great* followup blog post. :) Now I just wish there were a way to get Portage and Make to watch CPU usage and raise or lower the load-average threshold depending on how much CPU was going to 'sys', 'user' and 'wait'; Lower -l if a great deal of time is spent in 'sys'; you're likely burning cycles in context switches. Raise -l if a great deal of time is spent waiting on I/O. It'd also be helpful to be able to give keystone[1] packages and Make recipes a more favorable NICE value than those less important, to induce the scheduler to favor the important packages over less-important packages when we've got more ready work than cores. I don't think Make *can* have the smarts for that, but Portage could conceivably do it for its own parallelization. [1] Where many things depend on them, either directly or indirectly. Getting these out of the way means more parallel-buildable packages being available at the same time. -- :wq
