Re: [gridengine users] SoGE: hyperthreading on or off?
Hi, Am 15.03.2016 um 20:43 schrieb Lane, William: > I'm just curious about what the current thoughts are WRT hyperthreading. > > I've read at least one article that suggested hyperthreading be left on so > that > the OS can take advantage of it, but that hyperthreading cores should be > excluded > from being bound to SoGE HPC jobs. Is this still the best strategy to follow? > Or > was it ever a good strategy to follow? > > Could excluding hyperthreading cores be accomplished by using the qsub > -binding > parameter? Or should SoGE itself be configured to ignore hyperthreading cores? > > My research into hyperthreaing and HPC has only turned up two strategies: > 1. turn off hyperthreading completely at the BIOS level > 2. the above situation where the OS is still allowed to use hyperthreading > but HPC > apps are disallowed binding to hyperthreading cores. I came to the same conclusion, as our applications don't like HT. I don't recall whether it was on the Beowulf or Open MPI list, where someone mentioned that turning HT off in the BIOS will also make some changes to the internal caching and/or handling of the pipeline. But I couldn't see this effect (I would have expected to get only half of the cores but slightly faster). HT may be better suited in case you want to run at least two different jobs on a machine and the access to CPU resources can joggle this way. I mean: using HT for an Open MPI or OpenMP job on a machine will lead to the effect that all forks/threads are doing the same at the same time and need the same resources (at least I would expect it to be this way). But two completely unrelated types of jobs may have better luck to get a benefit of HT. Looking into new machines, there are also several CPUs without HT capability. -- Reuti ___ users mailing list users@gridengine.org https://gridengine.org/mailman/listinfo/users
Re: [gridengine users] SoGE: hyperthreading on or off?
Hi, I agree with Reuti. My experience also leads to not use HT at all - all of the compute serves in the company I'm working for (including those used as SGE masters) have HT disabled in the BIOS. We have found that this provides the best performance for critical applications (specifically important for those with expensive licenses). Regards, Manfred -Original Message- From: users-boun...@gridengine.org [mailto:users-boun...@gridengine.org] On Behalf Of users-requ...@gridengine.org Sent: Donnerstag, 17. März 2016 13:00 To: users@gridengine.org Subject: users Digest, Vol 63, Issue 17 Send users mailing list submissions to users@gridengine.org To subscribe or unsubscribe via the World Wide Web, visit https://gridengine.org/mailman/listinfo/users or, via email, send a message with subject or body 'help' to users-requ...@gridengine.org You can reach the person managing the list at users-ow...@gridengine.org When replying, please edit your Subject line so it is more specific than "Re: Contents of users digest..." Today's Topics: 1. Re: SoGE: hyperthreading on or off? (Reuti) -- Message: 1 Date: Wed, 16 Mar 2016 23:54:14 +0100 From: Reuti To: "Lane, William" Cc: "users@gridengine.org" Subject: Re: [gridengine users] SoGE: hyperthreading on or off? Message-ID: Content-Type: text/plain; charset=us-ascii Hi, Am 15.03.2016 um 20:43 schrieb Lane, William: > I'm just curious about what the current thoughts are WRT hyperthreading. > > I've read at least one article that suggested hyperthreading be left > on so that the OS can take advantage of it, but that hyperthreading > cores should be excluded from being bound to SoGE HPC jobs. Is this > still the best strategy to follow? Or was it ever a good strategy to follow? > > Could excluding hyperthreading cores be accomplished by using the qsub > -binding parameter? Or should SoGE itself be configured to ignore > hyperthreading cores? > > My research into hyperthreaing and HPC has only turned up two strategies: > 1. turn off hyperthreading completely at the BIOS level 2. the above > situation where the OS is still allowed to use hyperthreading but HPC > apps are disallowed binding to hyperthreading cores. I came to the same conclusion, as our applications don't like HT. I don't recall whether it was on the Beowulf or Open MPI list, where someone mentioned that turning HT off in the BIOS will also make some changes to the internal caching and/or handling of the pipeline. But I couldn't see this effect (I would have expected to get only half of the cores but slightly faster). HT may be better suited in case you want to run at least two different jobs on a machine and the access to CPU resources can joggle this way. I mean: using HT for an Open MPI or OpenMP job on a machine will lead to the effect that all forks/threads are doing the same at the same time and need the same resources (at least I would expect it to be this way). But two completely unrelated types of jobs may have better luck to get a benefit of HT. Looking into new machines, there are also several CPUs without HT capability. -- Reuti -- ___ users mailing list users@gridengine.org https://gridengine.org/mailman/listinfo/users End of users Digest, Vol 63, Issue 17 * Dialog Semiconductor GmbH Neue Str. 95 D-73230 Kirchheim Managing Directors: Dr. Jalal Bagherli, Carsten Dahl Chairman of the Supervisory Board: Rich Beyer Commercial register: Amtsgericht Stuttgart: HRB 231181 UST-ID-Nr. DE 811121668 Legal Disclaimer: This e-mail communication (and any attachment/s) is confidential and contains proprietary information, some or all of which may be legally privileged. It is intended solely for the use of the individual or entity to which it is addressed. Access to this email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Please consider the environment before printing this e-mail ___ users mailing list users@gridengine.org https://gridengine.org/mailman/listinfo/users
Re: [gridengine users] SoGE: hyperthreading on or off?
We use HT in AWS when running a large set of single-thread jobs across a cluster. While the individual jobs run more slowly, the total set of jobs completes more quickly, saving us money. Generally HT gets about a 15% speed increase (and corresponding cost savings) over 2x ... in other words 2x the number of jobs will take ~1.85 times as long. Obviously not ideal for a multi-user cluster (users want 'their job' to complete in a predictable amount of time, and never 1.85x as long as usual), but when the $$s count (AWS, etc.) it can be a win. Cheers, -Hugh -Original Message- From: users-boun...@gridengine.org [mailto:users-boun...@gridengine.org] On Behalf Of Manfred Selz Sent: Thursday, March 17, 2016 10:26 AM To: users@gridengine.org Subject: Re: [gridengine users] SoGE: hyperthreading on or off? Hi, I agree with Reuti. My experience also leads to not use HT at all - all of the compute serves in the company I'm working for (including those used as SGE masters) have HT disabled in the BIOS. We have found that this provides the best performance for critical applications (specifically important for those with expensive licenses). Regards, Manfred -Original Message- From: users-boun...@gridengine.org [mailto:users-boun...@gridengine.org] On Behalf Of users-requ...@gridengine.org Sent: Donnerstag, 17. März 2016 13:00 To: users@gridengine.org Subject: users Digest, Vol 63, Issue 17 Send users mailing list submissions to users@gridengine.org To subscribe or unsubscribe via the World Wide Web, visit https://gridengine.org/mailman/listinfo/users or, via email, send a message with subject or body 'help' to users-requ...@gridengine.org You can reach the person managing the list at users-ow...@gridengine.org When replying, please edit your Subject line so it is more specific than "Re: Contents of users digest..." Today's Topics: 1. Re: SoGE: hyperthreading on or off? (Reuti) -- Message: 1 Date: Wed, 16 Mar 2016 23:54:14 +0100 From: Reuti To: "Lane, William" Cc: "users@gridengine.org" Subject: Re: [gridengine users] SoGE: hyperthreading on or off? Message-ID: Content-Type: text/plain; charset=us-ascii Hi, Am 15.03.2016 um 20:43 schrieb Lane, William: > I'm just curious about what the current thoughts are WRT hyperthreading. > > I've read at least one article that suggested hyperthreading be left > on so that the OS can take advantage of it, but that hyperthreading > cores should be excluded from being bound to SoGE HPC jobs. Is this > still the best strategy to follow? Or was it ever a good strategy to follow? > > Could excluding hyperthreading cores be accomplished by using the qsub > -binding parameter? Or should SoGE itself be configured to ignore > hyperthreading cores? > > My research into hyperthreaing and HPC has only turned up two strategies: > 1. turn off hyperthreading completely at the BIOS level 2. the above > situation where the OS is still allowed to use hyperthreading but HPC > apps are disallowed binding to hyperthreading cores. I came to the same conclusion, as our applications don't like HT. I don't recall whether it was on the Beowulf or Open MPI list, where someone mentioned that turning HT off in the BIOS will also make some changes to the internal caching and/or handling of the pipeline. But I couldn't see this effect (I would have expected to get only half of the cores but slightly faster). HT may be better suited in case you want to run at least two different jobs on a machine and the access to CPU resources can joggle this way. I mean: using HT for an Open MPI or OpenMP job on a machine will lead to the effect that all forks/threads are doing the same at the same time and need the same resources (at least I would expect it to be this way). But two completely unrelated types of jobs may have better luck to get a benefit of HT. Looking into new machines, there are also several CPUs without HT capability. -- Reuti -- ___ users mailing list users@gridengine.org https://gridengine.org/mailman/listinfo/users End of users Digest, Vol 63, Issue 17 * Dialog Semiconductor GmbH Neue Str. 95 D-73230 Kirchheim Managing Directors: Dr. Jalal Bagherli, Carsten Dahl Chairman of the Supervisory Board: Rich Beyer Commercial register: Amtsgericht Stuttgart: HRB 231181 UST-ID-Nr. DE 811121668 Legal Disclaimer: This e-mail communication (and any attachment/s) is confidential and contains proprietary information, some or all of which may be legally privileged. It is intended solely for the use of the individual or entity to which it is addressed. Access to this
Re: [gridengine users] SoGE: hyperthreading on or off?
Not all applications can take advantage of hyperthreading, but they do exist. In our environment we leave hyperthreading on in the bios, but set the slot limit per node to the number of true cores, rather than the number of HT threads. Users with HT-capable apps can request n cores and run n*2 threads. There are multiple ways to limit the number of slots per node (and we may change the way we do it based on a recent thread). I would not want to have the number of slots count HT threads, as you are likely to overload the node on many applications. We enable binding to ensure jobs starting many threads don't get to spill over to more cores than they requested. By default on SoGE with -binding linear:slots, asking for n cores will get you all the "threads" on each of those cores. As an example, we have users ask for 4 cores with qsub -pe smp 4 and they get the 4 cores and all associated HT threads. If their application is one of the few that works well with HT, they can have their application run 8 threads. In a previous environment I've had a few dedicated nodes and a separate ht queue that had slots=ht count, but that does not work well if nodes will be running a mix of ht and non-ht jobs. These are just my experiences and the way I've done things. YMMV, TMTOWTDI, etc. Best, Chris Ref: http://arc.liv.ac.uk/SGE/howto/sge-configs.html#_slot_limits_a_id_slotlimit _a On 3/15/16, 3:43 PM, "Lane, William" wrote: >I'm just curious about what the current thoughts are WRT hyperthreading. > >I've read at least one article that suggested hyperthreading be left on >so that >the OS can take advantage of it, but that hyperthreading cores should be >excluded >from being bound to SoGE HPC jobs. Is this still the best strategy to >follow? Or >was it ever a good strategy to follow? > >Could excluding hyperthreading cores be accomplished by using the qsub >-binding >parameter? Or should SoGE itself be configured to ignore hyperthreading >cores? > >My research into hyperthreaing and HPC has only turned up two strategies: >1. turn off hyperthreading completely at the BIOS level >2. the above situation where the OS is still allowed to use >hyperthreading but HPC >apps are disallowed binding to hyperthreading cores. > >Thank you in advance. > >-Bill L. This electronic message is intended for the use of the named recipient only, and may contain information that is confidential, privileged or protected from disclosure under applicable law. If you are not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any reading, disclosure, dissemination, distribution, copying or use of the contents of this message including any of its attachments is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and destroy all copies of this message. Please note, the recipient should check this email and any attachments for the presence of viruses. The organization accepts no liability for any damage caused by any virus transmitted by this email. ___ users mailing list users@gridengine.org https://gridengine.org/mailman/listinfo/users
