Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
Nikos Chantziaras wrote: > I'm not aware of any package system that supports this. Briegel does this. It can even build the same package (maybe with different feature flags) in parallel. Basicly it walks the dependency tree from leaves to root, builds binpkg's yet missing (at the point in it reaches them in the graph) and puts them into an archive. Perhaps important to note that everything happens in an sysroot, so there eg. cannot be any issues w/ upgrading some deps in the middle (along w/ other common issues kicked of of the game). cu -- -- Enrico Weigelt, metux IT service -- http://www.metux.de/ phone: +49 36207 519931 email: weig...@metux.de mobile: +49 151 27565287 icq: 210169427 skype: nekrad666 -- Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme --
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
* Dale wrote: > If one instance of emerge doesn't know what the other instance has > already done, then the second one could emerge it again. Doesn't emerge > do all the calculating at the beginning and runs with that until the end? That's also one of my questions - does parallel emerge instances know if some dependency had already been built/installed in the middle ? > I am using the -j option for the first time now. I'm updating KDE. It > seems to work fine. It doesn't scroll all the stuff like with a regular > emerges but this new rig is so fast, I can't read it anyway. I did have > a package to fail and it spit out the error for me to read. BTW: does -j / --jobs also cause a parallel make (-j parameter to gmake) ? cu -- -- Enrico Weigelt, metux IT service -- http://www.metux.de/ phone: +49 36207 519931 email: weig...@metux.de mobile: +49 151 27565287 icq: 210169427 skype: nekrad666 -- Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme --
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
J. Roeleveld wrote:
On Thursday 27 January 2011 23:53:04 Dale wrote:
I just used -j with no number. It worked fine and I played Solitaire
and checked my email while it was running. The only thing I noticed was
it using swap. That could slow things down but otherwise, it worked fine.
I do have the nice and ionice settings in make.conf tho. That helps a
lot for sure. Without those, it would likely slow down a lot.
Which settings do you use for that?
--
Joost
This is my settings at the moment:
PORTAGE_NICENESS=5
PORTAGE_IONICE_COMMAND="ionice -c -3 -p \${PID}"
MAKEOPTS="-j6"
FEATURES="buildpkg sandbox fixpackages parallel-fetch --keep-going"
EMERGE_DEFAULT_OPTS="--with-bdeps y --backtrack=30"
That works very well here for me. Rig, AMD 4 core CPU at 3.2Ghz, 4Gbs
of ram at DDR3 1600 and SATA 3.0Gbs/sec. Your mileage may vary tho.
Dale
:-) :-)
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
On Thursday 27 January 2011 23:53:04 Dale wrote: > Nikos Chantziaras wrote: > > On 01/27/2011 09:41 PM, Dale wrote: > >> I noticed the same thing with mine. It used a LOT of ram. I have 4Gbs > >> and it was up to about 3Gbs at one point and using some swap as well. > >> I'm hoping to max out to 16Gbs as soon as I can. May upgrade to a 6 core > >> CPU too. > >> > >> I wonder how much faster it would be if the work directory is put on > >> tmpfs? With 16Gbs, that should work even for OOo. > > > > Btw, if you're using more instances than the amount of CPUs, the > > result will be slow-down. > > > > With the default kernel scheduler, best if amount of CPUs + 1. (On a > > 4-core, that's -j5). > > I just used -j with no number. It worked fine and I played Solitaire > and checked my email while it was running. The only thing I noticed was > it using swap. That could slow things down but otherwise, it worked fine. > > I do have the nice and ionice settings in make.conf tho. That helps a > lot for sure. Without those, it would likely slow down a lot. Which settings do you use for that? -- Joost
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
On Thursday 27 January 2011 23:59:24 Mick wrote: > I'm running i7 Q 720 (4 cores, hyperthreaded) and have MAKEOPTS="-j9" > without any slowdown. One or two packages (like OpenOffice) will fail and > need -j=1 to emerge. Otherwise no noticeable drop in desktop > responsiveness. > > I have not set up portage niceness so it runs with default value. > > Given the above what shall I set --load-average as? Not sure, I tested last night with a value of 12 and my system was still responsive. It depends, I think, on how well your system handles higher load. The best suggestion I can give is to actually test with values and see how well it responds. -- Joost
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
Nikos Chantziaras wrote: On 01/27/2011 09:41 PM, Dale wrote: I noticed the same thing with mine. It used a LOT of ram. I have 4Gbs and it was up to about 3Gbs at one point and using some swap as well. I'm hoping to max out to 16Gbs as soon as I can. May upgrade to a 6 core CPU too. I wonder how much faster it would be if the work directory is put on tmpfs? With 16Gbs, that should work even for OOo. Btw, if you're using more instances than the amount of CPUs, the result will be slow-down. With the default kernel scheduler, best if amount of CPUs + 1. (On a 4-core, that's -j5). I just used -j with no number. It worked fine and I played Solitaire and checked my email while it was running. The only thing I noticed was it using swap. That could slow things down but otherwise, it worked fine. I do have the nice and ionice settings in make.conf tho. That helps a lot for sure. Without those, it would likely slow down a lot. Dale :-) :-)
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
On Thursday 27 January 2011 22:18:22 J. Roeleveld wrote: > On Thursday 27 January 2011 23:05:22 Paul Hartman wrote: > > On Thu, Jan 27, 2011 at 3:46 PM, J. Roeleveld wrote: > > >> Once, when building my kernel, I accidentally forgot to specify the > > >> number of makes and ran "make -j all". That was a really bad idea, the > > >> system became totally unresponsive for quite a long time, much longer > > >> than normal kernel build time, but it did eventually finish! > > > > > > I have found that multi-core systems with sufficient memory can handle > > > "-j" (no value) a lot better then sindle-core systems. I do on occasion > > > do it with the kernel and can still continue using the system. (For > > > comparison, my desktop is a 4-core AMD64 with 8GB memory) > > > > Strange, in my case it was an i7 920 (4 cores, hyperthreaded, appears > > as 8 CPUs to Linux) with 12GB of RAM. Maybe if I prefixed it > > with"nice" it would not have brought my computer to its knees... or > > maybe related to the schedulers and other kernel voodoo that I don't > > understand. I might try it again someday :) > > That is strange, unless your harddrive is really underperforming? > Or do you have all the options in the kernel selected? > > Btw, HyperThreading doesn't work too well when you have a lot of identical > tasks. In that case, you might end up with lesser performance as there are > no "usable unused" parts in your cores, but the CPU-schedules (the > hardware one for HT) is looking for things to fill those last few bits > with. I'm running i7 Q 720 (4 cores, hyperthreaded) and have MAKEOPTS="-j9" without any slowdown. One or two packages (like OpenOffice) will fail and need -j=1 to emerge. Otherwise no noticeable drop in desktop responsiveness. I have not set up portage niceness so it runs with default value. Given the above what shall I set --load-average as? -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
On Thursday 27 January 2011 23:05:22 Paul Hartman wrote: > On Thu, Jan 27, 2011 at 3:46 PM, J. Roeleveld wrote: > > On Thursday 27 January 2011 21:25:02 Paul Hartman wrote: > >> On Thu, Jan 27, 2011 at 2:15 PM, Nikos Chantziaras wrote: > >> > On 01/27/2011 09:41 PM, Dale wrote: > >> >> YoYo Siska wrote: > >> >>> Yes. > >> >>> It might not be perfect, but mostly it works pretty well. > >> >>> Once make started 10 or so process, which ate all my ram, because I > >> >>> forgot to reenable swap, when I was playing with something before > >> >>> that > >> >>> > >> >>> :) > >> >>> > >> >>> yoyo > >> >> > >> >> I noticed the same thing with mine. It used a LOT of ram. I have 4Gbs > >> >> and it was up to about 3Gbs at one point and using some swap as well. > >> >> I'm hoping to max out to 16Gbs as soon as I can. May upgrade to a 6 > >> >> core CPU too. > >> >> > >> >> I wonder how much faster it would be if the work directory is put on > >> >> tmpfs? With 16Gbs, that should work even for OOo. > >> > > >> > Btw, if you're using more instances than the amount of CPUs, the > >> > result will be slow-down. > >> > > >> > With the default kernel scheduler, best if amount of CPUs + 1. (On a > >> > 4-core, that's -j5). > >> > >> Once, when building my kernel, I accidentally forgot to specify the > >> number of makes and ran "make -j all". That was a really bad idea, the > >> system became totally unresponsive for quite a long time, much longer > >> than normal kernel build time, but it did eventually finish! > > > > I have found that multi-core systems with sufficient memory can handle > > "-j" (no value) a lot better then sindle-core systems. I do on occasion > > do it with the kernel and can still continue using the system. (For > > comparison, my desktop is a 4-core AMD64 with 8GB memory) > > Strange, in my case it was an i7 920 (4 cores, hyperthreaded, appears > as 8 CPUs to Linux) with 12GB of RAM. Maybe if I prefixed it > with"nice" it would not have brought my computer to its knees... or > maybe related to the schedulers and other kernel voodoo that I don't > understand. I might try it again someday :) That is strange, unless your harddrive is really underperforming? Or do you have all the options in the kernel selected? Btw, HyperThreading doesn't work too well when you have a lot of identical tasks. In that case, you might end up with lesser performance as there are no "usable unused" parts in your cores, but the CPU-schedules (the hardware one for HT) is looking for things to fill those last few bits with. -- Joost
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
On Thursday 27 January 2011 22:06:30 YoYo Siska wrote: > On Thu, Jan 27, 2011 at 08:18:34PM +0100, J. Roeleveld wrote: > > On Thursday 27 January 2011 19:56:23 Allan Gottlieb wrote: > > > On Thu, Jan 27 2011, Neil Bothwick wrote: > > > > On Thu, 27 Jan 2011 17:09:27 +0200, Nikos Chantziaras wrote: > > > >> > So on a 20 package world update, only 19 are faster while the 20th > > > >> > runs at the same speed? Where's the loss there? Even if the last > > > >> > were slower, it would be worth it. > > > >> > > > >> Given the amount of time unpack/configure/install of most packages > > > >> needs (very short), my observation is that it would not be worth it. > > > > > > > > Even if that were true, how much time would you have to save to > > > > justify adding -j 2 to EMERGE_DEFAULTS in make.conf? > > > > > > > > But it's not true, large packages spend a lot of time on these phases > > > > of the install. > > > > > > OK I'm convinced since I know that those phases do take noticeable > > > time. > > > > > > I have a "4" processor i7 model 620 (2 cores, doubled for > > > hyperthreading) and have set MAKEOPTS="-j5". > > > If I add -jobs=2 to EMERGE_DEFAULT_OPTS, should I lower > > > MAKEOPTS to 3 (to 4)? > > > > > > thanks, > > > allan > > > > You could, as if you leave it at -j5, you can end up with 2 * 5 = 10 > > processed, eg: similar as if running with MAKEOPTS="-j10" > > > > I think the option that YoYo came with is a good compromise: > > # MAKEOPTS="-j -l5" emerge -j --load-average=5 > > > > Next time I am doing a big upgrade, I'm going to test that to see how it > > behaves. > > I was just building the whole system for my notebook in a chroot on my > desktop machine ( I use FEATURES=buildpkg to build binary packages in > the chroot on a fast desktop machine and then upgrade the notebook with > the binary packages) and I used exactly that (-j -l5 for bot make and > emerge). Can't say if it really is better or not ;) but most of the time > all four cores were busy, though sometimes I saw even 6 or 7 gcc-s > simultaneously in top ;) > emerge was running 3 to 4 jobs most of the time, sometimes dropping to 1 > and once I saw it emerging about 10 parallel packages ;) (mostly small > things, which I guess were doing a lot of > unpacking/configuring/installing but almost nothing of compiling ;) > > > btw, just now I got this error from dev-lang/v8: > > SCons error: option -j: invalid integer value: '-l4' > > seems scons honors MAKEOPTS, but doesn't understand the "loadaverage" > version (-j -l4) > > > yoyo Doing an update now, just had this: *** top - 23:09:47 up 15:12, 4 users, load average: 5.92, 3.62, 1.87 Tasks: 321 total, 35 running, 285 sleeping, 0 stopped, 1 zombie Cpu(s): 89.5%us, 10.5%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 8114712k total, 5266456k used, 2848256k free, 162600k buffers Swap: 12582904k total,0k used, 12582904k free, 2812008k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 24196 root 20 0 63756 39m 6932 R 15 0.5 0:00.45 cc1plus 24092 root 20 0 65532 40m 6896 R 15 0.5 0:00.64 cc1plus 24158 root 20 0 60572 36m 6832 R 15 0.5 0:00.50 cc1plus 24187 root 20 0 61296 36m 6904 R 15 0.5 0:00.45 cc1plus 24202 root 20 0 59404 34m 6832 R 14 0.4 0:00.43 cc1plus 24096 root 20 0 61272 36m 6868 R 14 0.5 0:00.56 cc1plus 24245 root 20 0 43364 20m 6372 R 12 0.3 0:00.36 cc1 24240 root 20 0 41636 18m 6340 R 12 0.2 0:00.35 cc1
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
On Thu, Jan 27, 2011 at 3:46 PM, J. Roeleveld wrote: > On Thursday 27 January 2011 21:25:02 Paul Hartman wrote: >> On Thu, Jan 27, 2011 at 2:15 PM, Nikos Chantziaras wrote: >> > On 01/27/2011 09:41 PM, Dale wrote: >> >> YoYo Siska wrote: >> >>> Yes. >> >>> It might not be perfect, but mostly it works pretty well. >> >>> Once make started 10 or so process, which ate all my ram, because I >> >>> forgot to reenable swap, when I was playing with something before that >> >>> >> >>> :) >> >>> >> >>> yoyo >> >> >> >> I noticed the same thing with mine. It used a LOT of ram. I have 4Gbs >> >> and it was up to about 3Gbs at one point and using some swap as well. >> >> I'm hoping to max out to 16Gbs as soon as I can. May upgrade to a 6 core >> >> CPU too. >> >> >> >> I wonder how much faster it would be if the work directory is put on >> >> tmpfs? With 16Gbs, that should work even for OOo. >> > >> > Btw, if you're using more instances than the amount of CPUs, the result >> > will be slow-down. >> > >> > With the default kernel scheduler, best if amount of CPUs + 1. (On a >> > 4-core, that's -j5). >> >> Once, when building my kernel, I accidentally forgot to specify the >> number of makes and ran "make -j all". That was a really bad idea, the >> system became totally unresponsive for quite a long time, much longer >> than normal kernel build time, but it did eventually finish! > > I have found that multi-core systems with sufficient memory can handle "-j" > (no value) a lot better then sindle-core systems. I do on occasion do it with > the kernel and can still continue using the system. (For comparison, my > desktop is a 4-core AMD64 with 8GB memory) Strange, in my case it was an i7 920 (4 cores, hyperthreaded, appears as 8 CPUs to Linux) with 12GB of RAM. Maybe if I prefixed it with"nice" it would not have brought my computer to its knees... or maybe related to the schedulers and other kernel voodoo that I don't understand. I might try it again someday :)
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
On Thursday 27 January 2011 22:06:30 YoYo Siska wrote: > On Thu, Jan 27, 2011 at 08:18:34PM +0100, J. Roeleveld wrote: > > On Thursday 27 January 2011 19:56:23 Allan Gottlieb wrote: > > > On Thu, Jan 27 2011, Neil Bothwick wrote: > > > > On Thu, 27 Jan 2011 17:09:27 +0200, Nikos Chantziaras wrote: > > > >> > So on a 20 package world update, only 19 are faster while the 20th > > > >> > runs at the same speed? Where's the loss there? Even if the last > > > >> > were slower, it would be worth it. > > > >> > > > >> Given the amount of time unpack/configure/install of most packages > > > >> needs (very short), my observation is that it would not be worth it. > > > > > > > > Even if that were true, how much time would you have to save to > > > > justify adding -j 2 to EMERGE_DEFAULTS in make.conf? > > > > > > > > But it's not true, large packages spend a lot of time on these phases > > > > of the install. > > > > > > OK I'm convinced since I know that those phases do take noticeable > > > time. > > > > > > I have a "4" processor i7 model 620 (2 cores, doubled for > > > hyperthreading) and have set MAKEOPTS="-j5". > > > If I add -jobs=2 to EMERGE_DEFAULT_OPTS, should I lower > > > MAKEOPTS to 3 (to 4)? > > > > > > thanks, > > > allan > > > > You could, as if you leave it at -j5, you can end up with 2 * 5 = 10 > > processed, eg: similar as if running with MAKEOPTS="-j10" > > > > I think the option that YoYo came with is a good compromise: > > # MAKEOPTS="-j -l5" emerge -j --load-average=5 > > > > Next time I am doing a big upgrade, I'm going to test that to see how it > > behaves. > > I was just building the whole system for my notebook in a chroot on my > desktop machine ( I use FEATURES=buildpkg to build binary packages in > the chroot on a fast desktop machine and then upgrade the notebook with > the binary packages) and I used exactly that (-j -l5 for bot make and > emerge). Can't say if it really is better or not ;) but most of the time > all four cores were busy, though sometimes I saw even 6 or 7 gcc-s > simultaneously in top ;) > emerge was running 3 to 4 jobs most of the time, sometimes dropping to 1 > and once I saw it emerging about 10 parallel packages ;) (mostly small > things, which I guess were doing a lot of > unpacking/configuring/installing but almost nothing of compiling ;) > > > btw, just now I got this error from dev-lang/v8: > > SCons error: option -j: invalid integer value: '-l4' > > seems scons honors MAKEOPTS, but doesn't understand the "loadaverage" > version (-j -l4) You might be able to avoid this by using the "long" version in the MAKEOPTS for "-l"? Eg: MAKEOPTS="-j --load-average=5" emerge -j --load-average=5 -- Joost
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
On Thursday 27 January 2011 21:25:02 Paul Hartman wrote: > On Thu, Jan 27, 2011 at 2:15 PM, Nikos Chantziaras wrote: > > On 01/27/2011 09:41 PM, Dale wrote: > >> YoYo Siska wrote: > >>> Yes. > >>> It might not be perfect, but mostly it works pretty well. > >>> Once make started 10 or so process, which ate all my ram, because I > >>> forgot to reenable swap, when I was playing with something before that > >>> > >>> :) > >>> > >>> yoyo > >> > >> I noticed the same thing with mine. It used a LOT of ram. I have 4Gbs > >> and it was up to about 3Gbs at one point and using some swap as well. > >> I'm hoping to max out to 16Gbs as soon as I can. May upgrade to a 6 core > >> CPU too. > >> > >> I wonder how much faster it would be if the work directory is put on > >> tmpfs? With 16Gbs, that should work even for OOo. > > > > Btw, if you're using more instances than the amount of CPUs, the result > > will be slow-down. > > > > With the default kernel scheduler, best if amount of CPUs + 1. (On a > > 4-core, that's -j5). > > Once, when building my kernel, I accidentally forgot to specify the > number of makes and ran "make -j all". That was a really bad idea, the > system became totally unresponsive for quite a long time, much longer > than normal kernel build time, but it did eventually finish! I have found that multi-core systems with sufficient memory can handle "-j" (no value) a lot better then sindle-core systems. I do on occasion do it with the kernel and can still continue using the system. (For comparison, my desktop is a 4-core AMD64 with 8GB memory) -- Joost
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
On Thu, Jan 27, 2011 at 2:15 PM, Nikos Chantziaras wrote: > On 01/27/2011 09:41 PM, Dale wrote: >> >> YoYo Siska wrote: >>> >>> Yes. >>> It might not be perfect, but mostly it works pretty well. >>> Once make started 10 or so process, which ate all my ram, because I >>> forgot to reenable swap, when I was playing with something before that >>> :) >>> >>> yoyo >> >> I noticed the same thing with mine. It used a LOT of ram. I have 4Gbs >> and it was up to about 3Gbs at one point and using some swap as well. >> I'm hoping to max out to 16Gbs as soon as I can. May upgrade to a 6 core >> CPU too. >> >> I wonder how much faster it would be if the work directory is put on >> tmpfs? With 16Gbs, that should work even for OOo. >> > > Btw, if you're using more instances than the amount of CPUs, the result will > be slow-down. > > With the default kernel scheduler, best if amount of CPUs + 1. (On a > 4-core, that's -j5). Once, when building my kernel, I accidentally forgot to specify the number of makes and ran "make -j all". That was a really bad idea, the system became totally unresponsive for quite a long time, much longer than normal kernel build time, but it did eventually finish!
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
On 01/27/2011 12:15 PM, Nikos Chantziaras wrote: > > Btw, if you're using more instances than the amount of CPUs, the result > will be slow-down. > > With the default kernel scheduler, best if amount of CPUs + 1. (On a > 4-core, that's -j5). And if you use emerge's "--jobs 2", each of those jobs will get the MAKEOPTS values sent to it. So, if you have MAKEOPTS="-s -j4" and you use "--jobs 2" with emerge, then you'll get two jobs running "gcc -s -j4". Enjoy.
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
On Thu, Jan 27, 2011 at 08:18:34PM +0100, J. Roeleveld wrote: > On Thursday 27 January 2011 19:56:23 Allan Gottlieb wrote: > > On Thu, Jan 27 2011, Neil Bothwick wrote: > > > On Thu, 27 Jan 2011 17:09:27 +0200, Nikos Chantziaras wrote: > > >> > So on a 20 package world update, only 19 are faster while the 20th > > >> > runs at the same speed? Where's the loss there? Even if the last were > > >> > slower, it would be worth it. > > >> > > >> Given the amount of time unpack/configure/install of most packages > > >> needs (very short), my observation is that it would not be worth it. > > > > > > Even if that were true, how much time would you have to save to justify > > > adding -j 2 to EMERGE_DEFAULTS in make.conf? > > > > > > But it's not true, large packages spend a lot of time on these phases of > > > the install. > > > > OK I'm convinced since I know that those phases do take noticeable time. > > > > I have a "4" processor i7 model 620 (2 cores, doubled for > > hyperthreading) and have set MAKEOPTS="-j5". > > If I add -jobs=2 to EMERGE_DEFAULT_OPTS, should I lower > > MAKEOPTS to 3 (to 4)? > > > > thanks, > > allan > > You could, as if you leave it at -j5, you can end up with 2 * 5 = 10 > processed, eg: similar as if running with MAKEOPTS="-j10" > > I think the option that YoYo came with is a good compromise: > # MAKEOPTS="-j -l5" emerge -j --load-average=5 > > Next time I am doing a big upgrade, I'm going to test that to see how it > behaves. I was just building the whole system for my notebook in a chroot on my desktop machine ( I use FEATURES=buildpkg to build binary packages in the chroot on a fast desktop machine and then upgrade the notebook with the binary packages) and I used exactly that (-j -l5 for bot make and emerge). Can't say if it really is better or not ;) but most of the time all four cores were busy, though sometimes I saw even 6 or 7 gcc-s simultaneously in top ;) emerge was running 3 to 4 jobs most of the time, sometimes dropping to 1 and once I saw it emerging about 10 parallel packages ;) (mostly small things, which I guess were doing a lot of unpacking/configuring/installing but almost nothing of compiling ;) btw, just now I got this error from dev-lang/v8: SCons error: option -j: invalid integer value: '-l4' seems scons honors MAKEOPTS, but doesn't understand the "loadaverage" version (-j -l4) yoyo
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
YoYo Siska wrote: Yes. It might not be perfect, but mostly it works pretty well. Once make started 10 or so process, which ate all my ram, because I forgot to reenable swap, when I was playing with something before that :) yoyo I noticed the same thing with mine. It used a LOT of ram. I have 4Gbs and it was up to about 3Gbs at one point and using some swap as well. I'm hoping to max out to 16Gbs as soon as I can. May upgrade to a 6 core CPU too. I wonder how much faster it would be if the work directory is put on tmpfs? With 16Gbs, that should work even for OOo. Dale :-) :-)
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
On Thursday 27 January 2011 19:56:23 Allan Gottlieb wrote: > On Thu, Jan 27 2011, Neil Bothwick wrote: > > On Thu, 27 Jan 2011 17:09:27 +0200, Nikos Chantziaras wrote: > >> > So on a 20 package world update, only 19 are faster while the 20th > >> > runs at the same speed? Where's the loss there? Even if the last were > >> > slower, it would be worth it. > >> > >> Given the amount of time unpack/configure/install of most packages > >> needs (very short), my observation is that it would not be worth it. > > > > Even if that were true, how much time would you have to save to justify > > adding -j 2 to EMERGE_DEFAULTS in make.conf? > > > > But it's not true, large packages spend a lot of time on these phases of > > the install. > > OK I'm convinced since I know that those phases do take noticeable time. > > I have a "4" processor i7 model 620 (2 cores, doubled for > hyperthreading) and have set MAKEOPTS="-j5". > If I add -jobs=2 to EMERGE_DEFAULT_OPTS, should I lower > MAKEOPTS to 3 (to 4)? > > thanks, > allan You could, as if you leave it at -j5, you can end up with 2 * 5 = 10 processed, eg: similar as if running with MAKEOPTS="-j10" I think the option that YoYo came with is a good compromise: # MAKEOPTS="-j -l5" emerge -j --load-average=5 Next time I am doing a big upgrade, I'm going to test that to see how it behaves. -- Joost
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
On Thu, Jan 27 2011, Neil Bothwick wrote: > On Thu, 27 Jan 2011 17:09:27 +0200, Nikos Chantziaras wrote: > >> > So on a 20 package world update, only 19 are faster while the 20th >> > runs at the same speed? Where's the loss there? Even if the last were >> > slower, it would be worth it. >> >> Given the amount of time unpack/configure/install of most packages >> needs (very short), my observation is that it would not be worth it. > > Even if that were true, how much time would you have to save to justify > adding -j 2 to EMERGE_DEFAULTS in make.conf? > > But it's not true, large packages spend a lot of time on these phases of > the install. OK I'm convinced since I know that those phases do take noticeable time. I have a "4" processor i7 model 620 (2 cores, doubled for hyperthreading) and have set MAKEOPTS="-j5". If I add -jobs=2 to EMERGE_DEFAULT_OPTS, should I lower MAKEOPTS to 3 (to 4)? thanks, allan
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
On Thu, Jan 27, 2011 at 03:12:49PM +0100, J. Roeleveld wrote: > On Thursday 27 January 2011 15:05:25 YoYo Siska wrote: > > On Thu, Jan 27, 2011 at 03:33:21PM +0200, Nikos Chantziaras wrote: > > > On 01/27/2011 03:11 PM, Dale wrote: > > > >[...] > > > >I am using the -j option for the first time now. I'm updating KDE. It > > > >seems to work fine. It doesn't scroll all the stuff like with a regular > > > >emerges but this new rig is so fast, I can't read it anyway. I did have > > > >a package to fail and it spit out the error for me to read. > > > > > > You don't need that if you have MAKEOPTS set in your make.conf, > > > which is preferred. The -j option of emerge emerges multiple > > > packages, while with MAKEOPTS set to "-j4" or whatever does a > > > parallel build in the same package (meaning compiling multiple > > > source files at the same time). > > > > > > It's preferred because with "emerge -jN" the last package will only > > > use one CPU, while with "-jN" in MAKEOPTS even the last package will > > > use N CPUs. Furthermore, emerge can't always build N packages at > > > the same time because one can depend on the other, so it will have > > > to wait until the dependency is built. > > > > On the other hand, unpacking, configure and install stages are not > > parallel and emerge can do those in parallel for different packages... > > The best would be somewhere in the middle ;) > > > > > > There are also the load-average options to -j, i.e.: > > MAKEOPTS="-j -l5" emerge -j --load-average=5 > > > > which makes make spawn parallel processes while load average is below 5 > > and the same for emerge spawning parallel ebuilds (when make isn't > > parallel enough) > > > > yoyo > > Hmmm... didn't know about that one yet. > Does that mean that by doing it like that, the emerge-process (and compile- > processes) will try to keep the load average at 5 and if that is lower, it > will keep adding more processes? > Yes. It might not be perfect, but mostly it works pretty well. Once make started 10 or so process, which ate all my ram, because I forgot to reenable swap, when I was playing with something before that :) yoyo
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
On Thu, 27 Jan 2011 11:10:02 +0100, Neil Bothwick wrote about Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies: >On Thu, 27 Jan 2011 07:12:24 +0200, Nikos Chantziaras wrote: [snip] >> You can try, but the second instance with simply block until the >> lock has been removed. > >The lock is not there for the entire emerge, I have run two emerges at >the same time, such as when I needed to install something while a world >update is in progress. It is possible, but not recommended as a general >strategy. That's what --jobs is for. Ditto here. Moreover, it is safer to perform all the package installations in a single emerge, as the command will schedule dependencies correctly (most of the time). -- Regards, Dave [RLU #314465] *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* dwn...@ntlworld.com (David W Noon) *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* signature.asc Description: PGP signature
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
Alan McKinnon wrote: Apparently, though unproven, at 17:09 on Thursday 27 January 2011, Nikos Chantziaras did opine thusly: Given the amount of time unpack/configure/install of most packages needs (very short), my observation is that it would not be worth it. KDE. unpack/configure/install takes up a significant amount of time for KDE Putting reply in just one post this time. This is a discussion now and not a technical problem. The package that failed had nothing to do with it building more than one package at a time. For some reason, it didn't have one of the patches downloaded. I guess it was a failure between here and where the mirror is. When I restarted the emerge, it found it and no problems from there. It would have done the same thing if I wasn't using -j is the point here. Tthis was a KDE upgrade, it saved a LOT of time. Most of the time only a couple cores are really working especially when they are smaller packages. When using the -j option, all 4 cores were running and was pretty busy all the time. At one time, it was doing >20 packages at once. I also noticed the hard drive light was pretty steady too. All in all, using the -j option seems to have saved a lot of time here. This is a fairly new install so I can recall how long it took to install KDE the last time. This was much faster. Just reporting a real world experience here. I wish I had got a 6 core CPU now for sure. Maybe later. Dale :-) :-)
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
On Thu, 27 Jan 2011 17:09:27 +0200, Nikos Chantziaras wrote: > > So on a 20 package world update, only 19 are faster while the 20th > > runs at the same speed? Where's the loss there? Even if the last were > > slower, it would be worth it. > > Given the amount of time unpack/configure/install of most packages > needs (very short), my observation is that it would not be worth it. Even if that were true, how much time would you have to save to justify adding -j 2 to EMERGE_DEFAULTS in make.conf? But it's not true, large packages spend a lot of time on these phases of the install. -- Neil Bothwick Head: (n.) the part of a disk drive which detects sectors and decides which of the two possible values to return: 'lose a turn' or 'bankrupt.' signature.asc Description: PGP signature
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
On Thu, Jan 27, 2011 at 5:12 AM, Nikos Chantziaras wrote: > > You can try, but the second instance with simply block until the lock has > been removed. > > I'm not aware of any package system that supports this. I don't think > adding support for this justifies the added complexity. > > Correct me if I'm wrong but the only added complexity I can think of is an additional lock for package (ebuild) which is currently being built. Emerge would skip the locked packages during "calculating dependencies" and building phases and possibly wait for them to install. Just a naive idea though. Cheers, P.
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
Apparently, though unproven, at 17:09 on Thursday 27 January 2011, Nikos Chantziaras did opine thusly: > On 01/27/2011 04:53 PM, Neil Bothwick wrote: > > On Thu, 27 Jan 2011 16:30:30 +0200, Nikos Chantziaras wrote: > >>> Using --jobs does a better job of making use of your CPU because one > >>> package can use it fully for compiling while another is configuring. > >> > >> And what about the last package? The time you gained for faster > >> configure and install (which don't take too much time anyway) is wasted > >> again on the last package. > > > > So on a 20 package world update, only 19 are faster while the 20th runs > > at the same speed? Where's the loss there? Even if the last were slower, > > it would be worth it. > > Given the amount of time unpack/configure/install of most packages needs > (very short), my observation is that it would not be worth it. KDE. unpack/configure/install takes up a significant amount of time for KDE -- alan dot mckinnon at gmail dot com
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
On Thu, Jan 27, 2011 at 7:33 AM, Nikos Chantziaras wrote: > On 01/27/2011 03:11 PM, Dale wrote: >> >> [...] >> I am using the -j option for the first time now. I'm updating KDE. It >> seems to work fine. It doesn't scroll all the stuff like with a regular >> emerges but this new rig is so fast, I can't read it anyway. I did have >> a package to fail and it spit out the error for me to read. > > You don't need that if you have MAKEOPTS set in your make.conf, which is > preferred. The -j option of emerge emerges multiple packages, while with > MAKEOPTS set to "-j4" or whatever does a parallel build in the same package > (meaning compiling multiple source files at the same time). > > It's preferred because with "emerge -jN" the last package will only use one > CPU, while with "-jN" in MAKEOPTS even the last package will use N CPUs. > Furthermore, emerge can't always build N packages at the same time because > one can depend on the other, so it will have to wait until the dependency is > built. I use MAKEOPTS = -j12 along with emerge --jobs for portage. Needless to say my CPU is fully utilized during an emerge of KDE. :)
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
On Thu, 27 Jan 2011 16:30:30 +0200, Nikos Chantziaras wrote: > > Using --jobs does a better job of making use of your CPU because one > > package can use it fully for compiling while another is configuring. > > And what about the last package? The time you gained for faster > configure and install (which don't take too much time anyway) is wasted > again on the last package. So on a 20 package world update, only 19 are faster while the 20th runs at the same speed? Where's the loss there? Even if the last were slower, it would be worth it. Note that I never use --jobs without an argument, 2 by default, although I could let emerge decide how many parallel processes to run. -- Neil Bothwick Top Oxymorons Number 47: Act naturally signature.asc Description: PGP signature
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
On Thu, 27 Jan 2011 15:33:21 +0200, Nikos Chantziaras wrote: > > I am using the -j option for the first time now. I'm updating KDE. It > > seems to work fine. It doesn't scroll all the stuff like with a > > regular emerges but this new rig is so fast, I can't read it anyway. > > I did have a package to fail and it spit out the error for me to > > read. > > You don't need that if you have MAKEOPTS set in your make.conf, which > is preferred. The -j option of emerge emerges multiple packages, while > with MAKEOPTS set to "-j4" or whatever does a parallel build in the > same package (meaning compiling multiple source files at the same time). And how many CPU cores are used during the configure and install stages? Using --jobs does a better job of making use of your CPU because one package can use it fully for compiling while another is configuring. -- Neil Bothwick Beware of the opinion of someone without any facts. signature.asc Description: PGP signature
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
On Thursday 27 January 2011 15:05:25 YoYo Siska wrote: > On Thu, Jan 27, 2011 at 03:33:21PM +0200, Nikos Chantziaras wrote: > > On 01/27/2011 03:11 PM, Dale wrote: > > >[...] > > >I am using the -j option for the first time now. I'm updating KDE. It > > >seems to work fine. It doesn't scroll all the stuff like with a regular > > >emerges but this new rig is so fast, I can't read it anyway. I did have > > >a package to fail and it spit out the error for me to read. > > > > You don't need that if you have MAKEOPTS set in your make.conf, > > which is preferred. The -j option of emerge emerges multiple > > packages, while with MAKEOPTS set to "-j4" or whatever does a > > parallel build in the same package (meaning compiling multiple > > source files at the same time). > > > > It's preferred because with "emerge -jN" the last package will only > > use one CPU, while with "-jN" in MAKEOPTS even the last package will > > use N CPUs. Furthermore, emerge can't always build N packages at > > the same time because one can depend on the other, so it will have > > to wait until the dependency is built. > > On the other hand, unpacking, configure and install stages are not > parallel and emerge can do those in parallel for different packages... > The best would be somewhere in the middle ;) > > > There are also the load-average options to -j, i.e.: > MAKEOPTS="-j -l5" emerge -j --load-average=5 > > which makes make spawn parallel processes while load average is below 5 > and the same for emerge spawning parallel ebuilds (when make isn't > parallel enough) > > yoyo Hmmm... didn't know about that one yet. Does that mean that by doing it like that, the emerge-process (and compile- processes) will try to keep the load average at 5 and if that is lower, it will keep adding more processes? -- Joost
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
On Thu, Jan 27, 2011 at 03:33:21PM +0200, Nikos Chantziaras wrote: > On 01/27/2011 03:11 PM, Dale wrote: > >[...] > >I am using the -j option for the first time now. I'm updating KDE. It > >seems to work fine. It doesn't scroll all the stuff like with a regular > >emerges but this new rig is so fast, I can't read it anyway. I did have > >a package to fail and it spit out the error for me to read. > > You don't need that if you have MAKEOPTS set in your make.conf, > which is preferred. The -j option of emerge emerges multiple > packages, while with MAKEOPTS set to "-j4" or whatever does a > parallel build in the same package (meaning compiling multiple > source files at the same time). > > It's preferred because with "emerge -jN" the last package will only > use one CPU, while with "-jN" in MAKEOPTS even the last package will > use N CPUs. Furthermore, emerge can't always build N packages at > the same time because one can depend on the other, so it will have > to wait until the dependency is built. On the other hand, unpacking, configure and install stages are not parallel and emerge can do those in parallel for different packages... The best would be somewhere in the middle ;) There are also the load-average options to -j, i.e.: MAKEOPTS="-j -l5" emerge -j --load-average=5 which makes make spawn parallel processes while load average is below 5 and the same for emerge spawning parallel ebuilds (when make isn't parallel enough) yoyo
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
Neil Bothwick wrote: On Thu, 27 Jan 2011 07:12:24 +0200, Nikos Chantziaras wrote: I'm aware that portage uses locking mechanism before modifying 'world' file, but what about the actual building process ? I'd expect emerge to check if dependency package is already build/installed (or currently being build by another instance) and just skip it in this case, however I haven't tried it yet.. Can anybody shred some light on this ? You can try, but the second instance with simply block until the lock has been removed. The lock is not there for the entire emerge, I have run two emerges at the same time, such as when I needed to install something while a world update is in progress. It is possible, but not recommended as a general strategy. That's what --jobs is for. I have done the same thing and as long as the dependencies don't clash, it works fine. However, if you start one emerge with a set of dependencies, then start another and they clash somewhere in the middle, portage has issues. That is where the locks would kick in I guess. I would also imagine that portage could emerge the same package twice too. If one instance of emerge doesn't know what the other instance has already done, then the second one could emerge it again. Doesn't emerge do all the calculating at the beginning and runs with that until the end? I am using the -j option for the first time now. I'm updating KDE. It seems to work fine. It doesn't scroll all the stuff like with a regular emerges but this new rig is so fast, I can't read it anyway. I did have a package to fail and it spit out the error for me to read. I agree, using --jobs is the best way to do this. It works really well if you have a fast multi-core CPU. I wish I had got me a 6 core one now. ;-) Dale :-) :-)
Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies
On Thu, 27 Jan 2011 07:12:24 +0200, Nikos Chantziaras wrote: > > I'm aware that portage uses locking mechanism before modifying 'world' > > file, but what about the actual building process ? I'd expect emerge > > to check if dependency package is already build/installed (or > > currently being build by another instance) and just skip it in this > > case, however I haven't tried it yet.. Can anybody shred some light > > on this ? > > You can try, but the second instance with simply block until the lock > has been removed. The lock is not there for the entire emerge, I have run two emerges at the same time, such as when I needed to install something while a world update is in progress. It is possible, but not recommended as a general strategy. That's what --jobs is for. -- Neil Bothwick Top Oxymorons Number 21: "Now, then ..." signature.asc Description: PGP signature
