Re: [arch-general] Can you test machine to set --jobs[=N] from w/i PKGBUILD?
On Sun, Apr 1, 2012 at 7:49 AM, Vitor Garcia wrote: > On Sun, 01 Apr 2012 13:37:27 +0200 > Florian Pritz wrote: > >> Simple tests (building readline because it's small) with -j4 and -j8 >> on my i7-920 show that -j8 is around 20% faster than -j4. IIRC >> wikipedia states that HT core can increase performance by up to 30%. > > This is nice to know. I work with mechanichal engineering simulations > and we have noted that using more threads then avaiable processors (we > have a 2 x 6 cores processors that has HT, so it looks like a 24 cores > server) increases the calculation time on the softwares we have. I > assumed that the same would apply to any intensive task, and we have > even disabled HT on the BIOS. Perhaps I'll enable it again. from my limited experience with scientific computing, the appropriate number of threads oftens relates not only to the number of cpus/cores, but also to the genuine feature of your application. i would suggest making some smaller-scaled test runs to test the performance and the implementation as well. best regards,
Re: [arch-general] Can you test machine to set --jobs[=N] from w/i PKGBUILD?
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 04/01/2012 05:47 PM, Jesse Juhani Jaara wrote:
> su, 2012-04-01 kello 14:44 -0500, David C. Rankin kirjoitti:
>> make || return 1
>
> Those '|| return 1' statements havent been used in long time anymore,
> yoy can just remove all them :D
Thanks!
I also did some testing on old and new boxes to find a makepkg.conf test that
is reliable. I hit on the following:
CPUCORES=$(grep -c "^processor" /proc/cpuinfo)
if test $CPUCORES -gt 1; then
MAKEFLAGS="-j${CPUCORES}"
fi
The beginning of line (^) is important to guard against incorrect results like:
16:01 nemesis:~> ( CPUCORES=$(grep -c "processor" /proc/cpuinfo); echo
"CPUCORES: $CPUCORES"; )
CPUCORES: 2
16:01 nemesis:~> cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 6
model : 4
model name : AMD Athlon(tm) processor
stepping: 2
cpu MHz : 800.054
cache size : 256 KB
fdiv_bug: no
hlt_bug : no
f00f_bug: no
coma_bug: no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pat
pse36 mmx fxsr syscall mmxext 3dnowext 3dnow up
bogomips: 1602.51
clflush size: 32
Notice the extra 'processor' in 'model name'.
- --
David C. Rankin, J.D.,P.E.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.16 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk94xLAACgkQZMpuZ8CyrciRDACfbW3UR/fcKLgye6G8Da2/XA7L
efwAnRUDOD3QSFe5zp9IPFKi1Hu0Jttt
=VL8G
-END PGP SIGNATURE-
Re: [arch-general] Can you test machine to set --jobs[=N] from w/i PKGBUILD?
su, 2012-04-01 kello 14:44 -0500, David C. Rankin kirjoitti: > make || return 1 Those '|| return 1' statements havent been used in long time anymore, yoy can just remove all them :D signature.asc Description: This is a digitally signed message part
Re: [arch-general] Can you test machine to set --jobs[=N] from w/i PKGBUILD?
On 04/01/2012 03:45 AM, Auguste Pop wrote: > i am not sure why are you trying to do it in PKGBUILD. if you have > several machines that you want to have different settings, why not set > /etc/makepkg.conf or $HOME/.makepkg.conf separately? > > anyway, i think the following command can easily get the information you need: > grep -c processor /proc/cpuinfo Thanks All, Auguste, I was really just brainstorming and I ran out of answers, so I asked on the list. What started this was just the question: "What can I do (if anything) to optimize the --jobs setting for trinity builds for anyone who decided to try building it?" I didn't even know how much difference the make --jobs=X could make on build time. It turns out to make a LOT of difference with quad core processors. A quick example is Amarok. With --jobs=0 on a phenom 9850 w/8G, the build takes 10 minutes. With --jobs=4, it builds in just over 2 minutes. That is a factor of five on rebuilds - literally cutting hours off the build time of trinity. I always wondered why anyone would care how many cores a processor has, because for desktop use -- I can't tell much difference at all on how a desktop behaves. However -- the usefulness of the additional cores was instantly clear when building large packages. So I was wanting to tweak the: make || return 1 line in the pkgbuild files to optimize build times for whoever was building (even if they hadn't made friends with makepkg.conf yet) Then thinking further about it, the Arch setup using makepkg.conf just made that much more sense. If a user is building that large of projects, then they should be at a point to learn to optimize makepkg themselves. That and the additional build instability (small, but it makes a difference with some code) of parallel builds by forcing --jobs=X in the pkgbuild, looked like a bad idea all the way around. However, I do thank you for the grep -c processor /proc/cpuinfo tip. -- David C. Rankin, J.D.,P.E.
Re: [arch-general] Can you test machine to set --jobs[=N] from w/i PKGBUILD?
On 1 April 2012 10:59, David C. Rankin wrote: > Guys, > > Is there any way to check the number of cores on a machine to set the > number > of jobs within a PKGBUILD?: > > make -j'X' > > I have some dual-core and some quad-core boxes and the difference between > -j2 > and -j4 on build time is tremendous. However, for generically creating a > source > package to be used by someone else, I can't hardcode the jobs number. > Anyway to > test in pkgbuild or is this just a per box makepkg.conf issue? > I had this for a one-fit Arch UFD: CPUCORES=$(grep processor /proc/cpuinfo | wc -l) [ $CPUCORES -gt 1 ] && MAKEFLAGS="-j$(($CPUCORES + 1))" But that's in makepkg.conf :) You shouldn't enforce this from within a PKGBUILD. -- GPG/PGP ID: C0711BF1
Re: [arch-general] Can you test machine to set --jobs[=N] from w/i PKGBUILD?
Because if there are 4 cores doing heavy CPU lifting of 4 threads, when one is finished, you then waste time loading up the next thread because it's reading from the disk. If you have it already done with I/O it will be ready and less time is wasted. Calvin On Apr 1, 2012 9:09 AM, "Baho Utot" wrote: > On 04/01/2012 08:37 AM, Calvin Morrison wrote: > >> Often higher cores benefit high I/O applications. If gcc is bottlenecking >> at reading and writing, sometimes more threads will help. >> > > How does that help? > > Sounds backwards to me > > > On Apr 1, 2012 7:49 AM, "Vitor > Garcia"> >> wrote: >> >> On Sun, 01 Apr 2012 13:37:27 +0200 >>> Florian Pritz wrote: >>> >>> Simple tests (building readline because it's small) with -j4 and -j8 on my i7-920 show that -j8 is around 20% faster than -j4. IIRC wikipedia states that HT core can increase performance by up to 30%. >>> This is nice to know. I work with mechanichal engineering simulations >>> and we have noted that using more threads then avaiable processors (we >>> have a 2 x 6 cores processors that has HT, so it looks like a 24 cores >>> server) increases the calculation time on the softwares we have. I >>> assumed that the same would apply to any intensive task, and we have >>> even disabled HT on the BIOS. Perhaps I'll enable it again. >>> >>> >
Re: [arch-general] Can you test machine to set --jobs[=N] from w/i PKGBUILD?
On 04/01/2012 08:37 AM, Calvin Morrison wrote: Often higher cores benefit high I/O applications. If gcc is bottlenecking at reading and writing, sometimes more threads will help. How does that help? Sounds backwards to me On Apr 1, 2012 7:49 AM, "Vitor Garcia" wrote: On Sun, 01 Apr 2012 13:37:27 +0200 Florian Pritz wrote: Simple tests (building readline because it's small) with -j4 and -j8 on my i7-920 show that -j8 is around 20% faster than -j4. IIRC wikipedia states that HT core can increase performance by up to 30%. This is nice to know. I work with mechanichal engineering simulations and we have noted that using more threads then avaiable processors (we have a 2 x 6 cores processors that has HT, so it looks like a 24 cores server) increases the calculation time on the softwares we have. I assumed that the same would apply to any intensive task, and we have even disabled HT on the BIOS. Perhaps I'll enable it again.
Re: [arch-general] Can you test machine to set --jobs[=N] from w/i PKGBUILD?
Often higher cores benefit high I/O applications. If gcc is bottlenecking at reading and writing, sometimes more threads will help. On Apr 1, 2012 7:49 AM, "Vitor Garcia" wrote: > On Sun, 01 Apr 2012 13:37:27 +0200 > Florian Pritz wrote: > > > Simple tests (building readline because it's small) with -j4 and -j8 > > on my i7-920 show that -j8 is around 20% faster than -j4. IIRC > > wikipedia states that HT core can increase performance by up to 30%. > > This is nice to know. I work with mechanichal engineering simulations > and we have noted that using more threads then avaiable processors (we > have a 2 x 6 cores processors that has HT, so it looks like a 24 cores > server) increases the calculation time on the softwares we have. I > assumed that the same would apply to any intensive task, and we have > even disabled HT on the BIOS. Perhaps I'll enable it again. >
Re: [arch-general] Can you test machine to set --jobs[=N] from w/i PKGBUILD?
On Sun, 01 Apr 2012 13:37:27 +0200 Florian Pritz wrote: > Simple tests (building readline because it's small) with -j4 and -j8 > on my i7-920 show that -j8 is around 20% faster than -j4. IIRC > wikipedia states that HT core can increase performance by up to 30%. This is nice to know. I work with mechanichal engineering simulations and we have noted that using more threads then avaiable processors (we have a 2 x 6 cores processors that has HT, so it looks like a 24 cores server) increases the calculation time on the softwares we have. I assumed that the same would apply to any intensive task, and we have even disabled HT on the BIOS. Perhaps I'll enable it again.
Re: [arch-general] Can you test machine to set --jobs[=N] from w/i PKGBUILD?
On 01.04.2012 13:05, Vitor Garcia wrote: > On Sun, 1 Apr 2012 04:45:08 -0400 > Auguste Pop wrote: > >> anyway, i think the following command can easily get the information >> you need: grep -c processor /proc/cpuinfo > > > This command will make a few 2 Cores processor with some kind of > virtual processing technology (like hyper-threading) to be counted as a > 4 cores processor. Which is totally fine. > Then you just have to multiply both numbers and you know how many "real > cpus" you have on the system. Otherwise you may be counting virtual > processors, which shouldn't be used for heavy stuff such as compiling. Simple tests (building readline because it's small) with -j4 and -j8 on my i7-920 show that -j8 is around 20% faster than -j4. IIRC wikipedia states that HT core can increase performance by up to 30%. -- Florian Pritz signature.asc Description: OpenPGP digital signature
Re: [arch-general] Can you test machine to set --jobs[=N] from w/i?PKGBUILD?
On Sun, Apr 01, 2012 at 04:28:53PM +0800, XeCycle wrote: > Jesse Jaara 写到： > > > Makepkg.conf > > MAKEFLAGS="-jX" > > > > Innstall renice and you wont even > > Notice the building, as gcc's priority > > is set so low by renice. > > That doesn't solve everything, though. A notebook user like me may find the > CPU gets too hot when building with aggressive parallel options, and renice > can't help. makepkg.conf is imo the way to go, because things are up to the user, how hot he wants his builds to get. I also concur with auguste's opinion that PKGUBILD is not the place to make this kind (rather machine-dependent) setting. thanks for bringing this up btw, I used to add this stuff manually to the custom kernels' PKGBUILDs I use. cheers! mar77i
Re: [arch-general] Can you test machine to set --jobs[=N] from w/i PKGBUILD?
On Sun, 1 Apr 2012 04:45:08 -0400 Auguste Pop wrote: > anyway, i think the following command can easily get the information > you need: grep -c processor /proc/cpuinfo This command will make a few 2 Cores processor with some kind of virtual processing technology (like hyper-threading) to be counted as a 4 cores processor. You may use: #cat /proc/cpuinfo | grep "physical id" | sort | uniq | wc -l To get the number of CPU "slots" you have on the machine. And then: #cat /proc/cpuinfo | grep "cpu cores" | uniq To get the number of cores per "slot". Then you just have to multiply both numbers and you know how many "real cpus" you have on the system. Otherwise you may be counting virtual processors, which shouldn't be used for heavy stuff such as compiling. Regards, Vitor
Re: [arch-general] Can you test machine to set --jobs[=N] from w/i PKGBUILD?
On Sun, Apr 1, 2012 at 12:59 AM, David C. Rankin wrote: > Guys, > > Is there any way to check the number of cores on a machine to set the number > of jobs within a PKGBUILD?: > > make -j'X' > > I have some dual-core and some quad-core boxes and the difference between -j2 > and -j4 on build time is tremendous. However, for generically creating a > source > package to be used by someone else, I can't hardcode the jobs number. Anyway > to > test in pkgbuild or is this just a per box makepkg.conf issue? > > -- > David C. Rankin, J.D.,P.E. i am not sure why are you trying to do it in PKGBUILD. if you have several machines that you want to have different settings, why not set /etc/makepkg.conf or $HOME/.makepkg.conf separately? anyway, i think the following command can easily get the information you need: grep -c processor /proc/cpuinfo
Re: [arch-general] Can you test machine to set --jobs[=N] from w/i PKGBUILD?
Jesse Jaara 写到： > Makepkg.conf > MAKEFLAGS="-jX" > > Innstall renice and you wont even > Notice the building, as gcc's priority > is set so low by renice. That doesn't solve everything, though. A notebook user like me may find the CPU gets too hot when building with aggressive parallel options, and renice can't help. -- Carl Lei (XeCycle) Sent with my phone, sorry for inconveniences.
Re: [arch-general] Can you test machine to set --jobs[=N] from w/i PKGBUILD?
Makepkg.conf MAKEFLAGS="-jX" Innstall renice and you wont even Notice the building, as gcc's priority is set so low by renice.
Re: [arch-general] Can you test machine to set --jobs[=N] from w/i PKGBUILD?
On 04/01/2012 12:03 AM, XeCycle wrote: > should be the latter. some don't want the system too heavily loaded by > building a package, so they may prefer building without parallel enabled. Thanks! I guess that's the smart way to do it anyway. -- David C. Rankin, J.D.,P.E.
Re: [arch-general] Can you test machine to set --jobs[=N] from w/i PKGBUILD?
should be the latter. some don't want the system too heavily loaded by building a package, so they may prefer building without parallel enabled. -- 抱歉暂时无法详细说明。这份邮件是使用安装有K-9 Mail的Android移动设备发送的。 "David C. Rankin" 写到： Guys, Is there any way to check the number of cores on a machine to set the number of jobs within a PKGBUILD?: make -j'X' I have some dual-core and some quad-core boxes and the difference between -j2 and -j4 on build time is tremendous. However, for generically creating a source package to be used by someone else, I can't hardcode the jobs number. Anyway to test in pkgbuild or is this just a per box makepkg.conf issue? -- David C. Rankin, J.D.,P.E.
[arch-general] Can you test machine to set --jobs[=N] from w/i PKGBUILD?
Guys, Is there any way to check the number of cores on a machine to set the number of jobs within a PKGBUILD?: make -j'X' I have some dual-core and some quad-core boxes and the difference between -j2 and -j4 on build time is tremendous. However, for generically creating a source package to be used by someone else, I can't hardcode the jobs number. Anyway to test in pkgbuild or is this just a per box makepkg.conf issue? -- David C. Rankin, J.D.,P.E.
