On Sat, 2007-03-10 at 18:41 +0100, Phil Sutter wrote: > On Sat, Mar 10, 2007 at 05:57:49PM +0100, Lothar Gesslein wrote: > > Another idea that i want to evaluate and eventually implement is > > building in several instances, as an addition/alternative to make -j. So > > not only compile on _one_ packet at a time, but build several packages > > (that do not depend on each other) in parallel. This avoids "gaps" in > > the processor usage, because even while some packet is downloading or > > unpacking, there is another one compiling. > > If each package is built using more than one job, problems might occur > with certain packages not building cleanly. If parallel building is done > inside the ADK itself, so as you said certain packages are built at the > same time, but each with a single job, we should gain a performance > improvement without destabilising the whole process too much.
Well, i wrote "addition/alternative", i wanted to provide the make -j as an option, maybe with a blacklist, so packages known to break can be excluded. You can choose for yourself if you want it as an addition or only as an alternative. But clearly, building whole packages in parallel is a more elegant solution. > > If you have more ideas on stressing the avaliable ressouces as much as > > possible, or in general, building as fast as the machine can, please > > tell me. > > What about parallel fetching as being able for Gentoo's portage? So a > single job fetches all tarballs in background, while the build process > waits for it if necessary. Yeah, this is part of the plan i have ... but everything is theoretical for now. At first, dependencies are "resolved", so we know what to build at first, and what has to wait. We start downloading, and as soon as the first package finished, it gets to the unpack-queue (which is empty at this point, so it doesn't have to wait) and then to the compile-queue (dito). When the second download is ready, it also gets to the unpack- and then to the compile-queue and so on. At every point, the first <n> slots of the queue gets processed (configureable in menuconfig), all other packages have to wait. Maybe the processing of the unpack-queue should be dependend on the size of the compile-queue, so if there are e.g. already 5 packages waiting in the compile queue, do not unpack anymore but wait. And perhaps it's better not to unpack in parallel but only one packet at a time. Well, i'll evaluate this when i'm at the point... and this could take quite some time ;) Of course, the whole queueing and managing stuff should not generate much overhead, because we want the ressouces to be used for building, not for managing the build. > > And yes, i know some developer(s) don't like make -j, ccache (and > > possibly neither distcc). They'll become optional, non-default, > > features. So no need to flame ;) > > I do! :) I'm happy to get a response at all, so you're welcome. :P Bye, Lothar _______________________________________________ freewrt-developers mailing list [email protected] https://www.freewrt.org/lists/listinfo/freewrt-developers
