On 9/11/2012 9:54 AM, Ian Stakenvicius wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On 11/09/12 12:43 PM, Zac Medico wrote:
On 09/11/2012 09:36 AM, viv...@gmail.com wrote:
Dunno where to place this request, but if we go for something
like EJOBS could we also make it phase specific? So compile,
install and test could have a different number of jobs running.
Possibly three different variables that override a predefined
EJOBS.
Per-phase sounds a little to fine-grained. Instead, I'd suggest to
add an ELOADAVG variable that's analogous to make's --load-average
option. That should be enough to compensate for any differences
between phases.
I personally wonder about why this would be necessary from the
perspective of the user; if the user's system at emerge time can
handle X concurrent processes per emerge-job , i don't see why it
would matter what phase these jobs would be launched from.
At the ebuild level, certainly, but that's one of the reasons for
EJOBS in the first place, so that it can be overridden consistently
within a phase, if necessary for the ebuild (regardless of build
system type), right?
I would hope so. There are definitely real-world reasons to want to
restrict jobs -- usually to just one -- in a particular phase... several
ebuilds, notably several in @system, do this already by injecting -j1
somewhere.
To say it's all about performance seems to be forgetting a major reason
somebody might want to do this as a user or an ebuild author.
Hmm...
I should preface this by saying that it really pains me to say this,
because now this just starts to seem so fucking complicated -- but now
that I think about it, this really seems to highlight a kind of semantic
discrepancy that this thread has mostly dusted under the rug.
There are really TWO things we seem to care about -- one is some kind of
global build-parallelism frob-set, and the other is the ability to turn
off all parallelism for certain parts of an ebuild and those are
actually kind-of orthogonal.
In other words, imagine if we had some kind of "global parallellism
arbiter" thingy that looked at EJOBS or ELOADAVG or maybe some other
things we haven't thought of yet, and decided things like:
o should portage start a new parallel emerge process?
o What is the appropriate "MAKEOPTS" for the "emake"
that foo-1.0.ebuild just issued?
or even, one could imagine, such things as:
o should portage decide that parallelism has gotten out of control
somehow and suspend or kill a running ebuild for subsequent
resuming or restarting?
o should portage re-nice some of its processes?
Note that, effectively, we have this already, and it's called "portage".
But one could certainly make a case for modularizing it better, since,
in truth, we are talking about a very common, very abstract problem here
which portage shares with any number of batch-build systems.
Such an engine could very well do exactly the right thing if it were
faced with a constraint that a certain part of a certain build needed to
proceed without parallelism due to limitations coming from the build.
Also, there are very large parts of most builds -- configure comes to
mind -- that don't parallelize even if, perhaps, they should. In such
cases, a really smart global parallelism arbiter could easily respond by
spawning more jobs from other builds.
Not sure what I'm suggesting we do about it, exactly, but just pointing
out that maybe a completely "correct" solution requires a much more
elaborate implementation than just a bunch of syntactic sugar around
what we currently call MAKEOPTS. Whether or not Gentoo wants to take
that all on, right now, as part of the next EAPI, is certainly debatable
-- in fact, I'm inclined to say maybe it's not the best idea.
Perhaps we should really be asking: is the status-quo really so
problematic/inelegant that it needs fixing? -- before we decide how best
to fix it.
-gmt
