Re: [hwloc-users] Selecting real cores vs HT cores
Can someone post docs for this resource halving Squyres claims? I've never heard of this. Jeff On Thursday, December 11, 2014, Samuel Thibault wrote: > Jeff Squyres (jsquyres), le Thu 11 Dec 2014 21:12:27 +, a écrit : > > When the BIOS is set to enable hyper threading, then several resources > on the core are split when the machine is booted up (e.g., some of the > queue depths for various processing units in the core are half the length > that they are when hyperthreading is disabled in the BIOS). > > Perhaps some queues get divided, but most of the resources (such as > cache, TLB, etc.) are completely available when using only one > hyperthread, like they would be with HT disabled. > > Samuel > ___ > hwloc-users mailing list > hwloc-us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users > Link to this post: > http://www.open-mpi.org/community/lists/hwloc-users/2014/12/1132.php > -- Jeff Hammond jeff.scie...@gmail.com http://jeffhammond.github.io/
Re: [hwloc-users] Selecting real cores vs HT cores
Jeff Squyres (jsquyres), le Thu 11 Dec 2014 21:12:27 +, a écrit : > When the BIOS is set to enable hyper threading, then several resources on the > core are split when the machine is booted up (e.g., some of the queue depths > for various processing units in the core are half the length that they are > when hyperthreading is disabled in the BIOS). Perhaps some queues get divided, but most of the resources (such as cache, TLB, etc.) are completely available when using only one hyperthread, like they would be with HT disabled. Samuel
Re: [hwloc-users] Selecting real cores vs HT cores
On Dec 11, 2014, at 2:03 PM, Brice Goglin wrote: > By the way, if you can't in the BIOS, you may want to disable the > hyperthread in the kernel: > > for i in $(hwloc-calc --whole-system --po -I pu core:all.pu:0) ; do echo 0 > > /sys/devices/system/cpu/cpu$i/online ; done > > (write 1 instead of 0 to reenable them). But keep in mind that this is the semantic equivalent of using hwloc-bind to bind to the first HT in each core. I.e., disabling HT in the Linux kernel just disables scheduling on the 2nd HT. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-users] Selecting real cores vs HT cores
Le 11/12/2014 21:51, Brock Palen a écrit : > When a system has HT enabled is one core presented the real one and one the > fake partner? Or is that not the case? > > If wanting to test behavior without messing with the bios how do I select > just the 'real cores' if this is the case? > > I am looking for the equivelent of > > hwloc-bind ALLREALCORES my.exe > Assuming this still makes sense after your discussion with Jeff, you can easily bind on the first hyperthread of all cores with: hwloc-bind core:all.pu:0 my.exe None of the hyperthreads is better than the others afaik. By the way, if you can't in the BIOS, you may want to disable the hyperthread in the kernel: for i in $(hwloc-calc --whole-system --po -I pu core:all.pu:0) ; do echo 0 > /sys/devices/system/cpu/cpu$i/online ; done (write 1 instead of 0 to reenable them). Brice
Re: [hwloc-users] Selecting real cores vs HT cores
Right and that is what I figured, they quote performance metrics. I'm almost trying to divine what mapping they use and if its static, 1:1 mapping. Thanks for the thoughts. Brock Palen www.umich.edu/~brockp CAEN Advanced Computing XSEDE Campus Champion bro...@umich.edu (734)936-1985 > On Dec 11, 2014, at 4:41 PM, Jeff Squyres (jsquyres) > wrote: > > On Dec 11, 2014, at 1:36 PM, Brock Palen wrote: > >> Ok let me expand then. I don't have control over the bios. >> >> The testing I am doing resides on a cloud provider and from our testing it >> appears that it has HT enabled. It is ambiguous though to me what I see vs >> how they allocate on their hypervisor. > > Oh, if you're in a hypervisor, then what you're seeing has zero correlation > to reality. > > If it's an HPC cloud provider, they *likely* paired cores in the hypervisor > with real/physical cores. More specifically: they *probably* paired hyper > threads in the hypervisor with real/physical hyper threads (i.e., so that the > lstopo in the hypervisor is equivalent to lstopo outside the hypervisor). > > But you'll need to ask them, because modern VMs let you do whatever you want > in terms of mapping VM cores/HTs to physical cores/HTs. > > Consider: you can run dozens on web server VMs on a machine with 10 cores. > Each VM will say that it has, say, 1 or 2 cores. But clearly, the sum of > number of cores in the VMs is larger than the total number of physical cores. > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > ___ > hwloc-users mailing list > hwloc-us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users > Link to this post: > http://www.open-mpi.org/community/lists/hwloc-users/2014/12/1129.php
Re: [hwloc-users] Selecting real cores vs HT cores
On Dec 11, 2014, at 1:36 PM, Brock Palen wrote: > Ok let me expand then. I don't have control over the bios. > > The testing I am doing resides on a cloud provider and from our testing it > appears that it has HT enabled. It is ambiguous though to me what I see vs > how they allocate on their hypervisor. Oh, if you're in a hypervisor, then what you're seeing has zero correlation to reality. If it's an HPC cloud provider, they *likely* paired cores in the hypervisor with real/physical cores. More specifically: they *probably* paired hyper threads in the hypervisor with real/physical hyper threads (i.e., so that the lstopo in the hypervisor is equivalent to lstopo outside the hypervisor). But you'll need to ask them, because modern VMs let you do whatever you want in terms of mapping VM cores/HTs to physical cores/HTs. Consider: you can run dozens on web server VMs on a machine with 10 cores. Each VM will say that it has, say, 1 or 2 cores. But clearly, the sum of number of cores in the VMs is larger than the total number of physical cores. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-users] Selecting real cores vs HT cores
Ok let me expand then. I don't have control over the bios. The testing I am doing resides on a cloud provider and from our testing it appears that it has HT enabled. It is ambiguous though to me what I see vs how they allocate on their hypervisor. I want to see if this has any effect. given the providers advertised CPU types they use vs my bare metal ones of the same types everything feels 'half as fast' Thus the question about HT. Here is the lstopo from the provider: lstopo-no-graphics Machine (7484MB) Socket L#0 + L3 L#0 (25MB) L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 PU L#0 (P#0) PU L#1 (P#2) L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 PU L#2 (P#1) PU L#3 (P#3) HostBridge L#0 PCI 8086:7010 PCI 1013:00b8 PCI 8086:10ed Net L#0 "eth0" Brock Palen www.umich.edu/~brockp CAEN Advanced Computing XSEDE Campus Champion bro...@umich.edu (734)936-1985 > On Dec 11, 2014, at 4:12 PM, Jeff Squyres (jsquyres) > wrote: > > I'm not sure you're asking a well-formed question. > > When the BIOS is set to enable hyper threading, then several resources on the > core are split when the machine is booted up (e.g., some of the queue depths > for various processing units in the core are half the length that they are > when hyperthreading is disabled in the BIOS). > > Hence, running a process on a core that only uses a single hyperthread (when > HT is enabled) is not quite the same thing as booting up with HT disabled and > running that same job on the core. > > Make sense? > > Meaning: if you want to test HT vs. non-HT performance, you really need to > change the BIOS settings and reboot, sorry. > > Also, note that if you have HT enabled and you run a single-threaded app > bound to a core, it will only use 1 of those HTs -- the other HT will be > largely dormant. Meaning: don't expect that running a single-threaded app on > a core that has HT enabled will magically take advantage of some performance > benefit of aggressive automatic parallelization. You really need multiple > threads in a process to get performance advantages out of HT. > > > > On Dec 11, 2014, at 12:51 PM, Brock Palen wrote: > >> When a system has HT enabled is one core presented the real one and one the >> fake partner? Or is that not the case? >> >> If wanting to test behavior without messing with the bios how do I select >> just the 'real cores' if this is the case? >> >> I am looking for the equivelent of >> >> hwloc-bind ALLREALCORES my.exe >> >> Doing some performance study type things. >> >> Thanks, >> >> Brock Palen >> www.umich.edu/~brockp >> CAEN Advanced Computing >> XSEDE Campus Champion >> bro...@umich.edu >> (734)936-1985 >> >> >> >> ___ >> hwloc-users mailing list >> hwloc-us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users >> Link to this post: >> http://www.open-mpi.org/community/lists/hwloc-users/2014/12/1126.php > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > ___ > hwloc-users mailing list > hwloc-us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users > Link to this post: > http://www.open-mpi.org/community/lists/hwloc-users/2014/12/1127.php
Re: [hwloc-users] Selecting real cores vs HT cores
I'm not sure you're asking a well-formed question. When the BIOS is set to enable hyper threading, then several resources on the core are split when the machine is booted up (e.g., some of the queue depths for various processing units in the core are half the length that they are when hyperthreading is disabled in the BIOS). Hence, running a process on a core that only uses a single hyperthread (when HT is enabled) is not quite the same thing as booting up with HT disabled and running that same job on the core. Make sense? Meaning: if you want to test HT vs. non-HT performance, you really need to change the BIOS settings and reboot, sorry. Also, note that if you have HT enabled and you run a single-threaded app bound to a core, it will only use 1 of those HTs -- the other HT will be largely dormant. Meaning: don't expect that running a single-threaded app on a core that has HT enabled will magically take advantage of some performance benefit of aggressive automatic parallelization. You really need multiple threads in a process to get performance advantages out of HT. On Dec 11, 2014, at 12:51 PM, Brock Palen wrote: > When a system has HT enabled is one core presented the real one and one the > fake partner? Or is that not the case? > > If wanting to test behavior without messing with the bios how do I select > just the 'real cores' if this is the case? > > I am looking for the equivelent of > > hwloc-bind ALLREALCORES my.exe > > Doing some performance study type things. > > Thanks, > > Brock Palen > www.umich.edu/~brockp > CAEN Advanced Computing > XSEDE Campus Champion > bro...@umich.edu > (734)936-1985 > > > > ___ > hwloc-users mailing list > hwloc-us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users > Link to this post: > http://www.open-mpi.org/community/lists/hwloc-users/2014/12/1126.php -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
[hwloc-users] Selecting real cores vs HT cores
When a system has HT enabled is one core presented the real one and one the fake partner? Or is that not the case? If wanting to test behavior without messing with the bios how do I select just the 'real cores' if this is the case? I am looking for the equivelent of hwloc-bind ALLREALCORES my.exe Doing some performance study type things. Thanks, Brock Palen www.umich.edu/~brockp CAEN Advanced Computing XSEDE Campus Champion bro...@umich.edu (734)936-1985