Strange behaviour
The system: dual-12-core AMD Opteron-6172 (total 24 cores at 3.2GHz), and 32GB RAM. (The motherboard is a ASUS KGPE-D16). While running a 'build.sh -j48 release' I notice that there seems to be a rather excessive amount of System Time. Running systat(1) I can see long periods of 60% to 70% (or even more), while at no time do I ever get more than 15% to 20% of user time. The only thing that seems to be running (according to vmstat) at many thousands of Global TLB IPI interrupts, although Interrupt time is mostly invisible. (It never shows up more than 1% or 2% on systat, and a xosview never shows any interrupt time at all.) Other than the 5K to 10K (with occassional bursts of 18K) TLB IPI, the only other interrupt activity is a few hundred disk interrupts, and maybe a few dozen network interrupts. Is this expected behavior? - | Paul Goyette | PGP Key fingerprint: | E-mail addresses: | | Customer Service | FA29 0E3B 35AF E8AE 6651 | paul at whooppee.com| | Network Engineer | 0786 F758 55DE 53BA 7731 | pgoyette at juniper.net | | Kernel Developer | | pgoyette at netbsd.org | -
Re: Strange behaviour
On Sun, May 29, 2011 at 06:24:06PM -0700, Paul Goyette wrote: The system: dual-12-core AMD Opteron-6172 (total 24 cores at 3.2GHz), and 32GB RAM. (The motherboard is a ASUS KGPE-D16). While running a 'build.sh -j48 release' I notice that there seems to be a rather excessive amount of System Time. Running systat(1) I [...] Is this expected behavior? Yes. We do a little better with 16 build jobs than 12 and considerably worse with 20 than 16. On my 24-core 2Ghz Opteron, the only thing I've tried that really can do an effective job building NetBSD with more than 12 CPUs in use is Linux. I should probably try Solaris. Linux under Xen does even worse than NetBSD, probably because Xen has no way to express to Linux the system's memory architecture. Xen has NUMA support in the sense that if you have, say, 6 cores assigned to a virtual and those 6 can be mapped onto physical cores on the same NUMA node, thus providing the virtual a uniform memory hierarchy, it will do so; but if you have a 24-core machine built from two pairs of dual-6-core chips, like the machines you and I have, it really doesn't do a good job telling the guest what it needs to schedule effectively. Thor
Re: Strange behaviour
BTW, it was definitely a typo in the original - my system is 24 cores at 2.1GHz, not 3.2GHz. On Sun, 29 May 2011, Thor Lancelot Simon wrote: On Sun, May 29, 2011 at 06:24:06PM -0700, Paul Goyette wrote: The system: dual-12-core AMD Opteron-6172 (total 24 cores at 3.2GHz), and 32GB RAM. (The motherboard is a ASUS KGPE-D16). While running a 'build.sh -j48 release' I notice that there seems to be a rather excessive amount of System Time. Running systat(1) I [...] Is this expected behavior? Yes. We do a little better with 16 build jobs than 12 and considerably worse with 20 than 16. On my 24-core 2Ghz Opteron, the only thing I've tried that really can do an effective job building NetBSD with more than 12 CPUs in use is Linux. I should probably try Solaris. Despite the level of System time, I was able to run a complete build.sh -j48 release in less than 45 minutes. And afterwards, I was able to build a complete new kernel in 44 seconds. (I guess most all of the source directory was cached somewhere in that 32GB of RAM!) Linux under Xen does even worse than NetBSD, probably because Xen has no way to express to Linux the system's memory architecture. Xen has NUMA support in the sense that if you have, say, 6 cores assigned to a virtual and those 6 can be mapped onto physical cores on the same NUMA node, thus providing the virtual a uniform memory hierarchy, it will do so; but if you have a 24-core machine built from two pairs of dual-6-core chips, like the machines you and I have, it really doesn't do a good job telling the guest what it needs to schedule effectively. - | Paul Goyette | PGP Key fingerprint: | E-mail addresses: | | Customer Service | FA29 0E3B 35AF E8AE 6651 | paul at whooppee.com| | Network Engineer | 0786 F758 55DE 53BA 7731 | pgoyette at juniper.net | | Kernel Developer | | pgoyette at netbsd.org | -
Re: Strange behaviour
On Sun, May 29, 2011 at 07:37:17PM -0700, Paul Goyette wrote: Despite the level of System time, I was able to run a complete build.sh -j48 release in less than 45 minutes. And afterwards, I was able to build a complete new kernel in 44 seconds. (I guess most all of the source directory was cached somewhere in that 32GB of RAM!) *Shrug* that's about how long it takes on my 4 year old Mac Pro with 8 cores... Thor
