[OMPI users] slot number calculation when no config files?
Using OpenMPI 4.0.1 and no configuration files of any kind on a linux machine which shows 4 CPUS and 2 threads per CPU. When this is run: mpirun --allow-run-as-root --oversubscribe -np 3 /usr/common/modules/el8/x86_64/software/q6/6.0.1-CentOS-vanilla/bin/qdynp eq2.inp it works and the expected 3 processes run. However if oversubscribe is omitted then this happens: There are not enough slots available in the system to satisfy the 3 slots that were requested by the application: It will run with 2 though even without -oversubscribe. It looks like it is using CPUs/2 to calculate the slot limit and ignoring threads. Where is this slot calculation documented? More details: $ cat /etc/centos-release CentOS Linux release 8.1.1911 (Core) $ mpirun --version mpirun (Open MPI) 4.0.1 $ lscpu | head -6 Architecture:x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 2 $ gcc --version gcc (GCC) 8.3.1 20190507 (Red Hat 8.3.1-4) Thanks, David Mathog
Re: [OMPI users] slot number calculation when no config files?
Hello David, The slot calculation is based on physical cores rather than logical cores. The 4 CPUs you are seeing there are logical CPUs. And since your processor has 2 threads per core, you have two physical cores; yielding a total of 4 logical cores (which is reported to lscpu). On machines with only 1 thread per core, the cpu number is the number of physical cores. Thanks, Collin -Original Message- From: users On Behalf Of David Mathog via users Sent: Monday, June 8, 2020 2:19 PM To: users@lists.open-mpi.org Cc: David Mathog Subject: [OMPI users] slot number calculation when no config files? Using OpenMPI 4.0.1 and no configuration files of any kind on a linux machine which shows 4 CPUS and 2 threads per CPU. When this is run: mpirun --allow-run-as-root --oversubscribe -np 3 /usr/common/modules/el8/x86_64/software/q6/6.0.1-CentOS-vanilla/bin/qdynp eq2.inp it works and the expected 3 processes run. However if oversubscribe is omitted then this happens: There are not enough slots available in the system to satisfy the 3 slots that were requested by the application: It will run with 2 though even without -oversubscribe. It looks like it is using CPUs/2 to calculate the slot limit and ignoring threads. Where is this slot calculation documented? More details: $ cat /etc/centos-release CentOS Linux release 8.1.1911 (Core) $ mpirun --version mpirun (Open MPI) 4.0.1 $ lscpu | head -6 Architecture:x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 2 $ gcc --version gcc (GCC) 8.3.1 20190507 (Red Hat 8.3.1-4) Thanks, David Mathog
Re: [OMPI users] slot number calculation when no config files?
Note that you can also resolve it by adding --use-hwthread-cpus to your cmd line - it instructs mpirun to treat the HWTs as independent cpus so you would have 4 slots in this case. > On Jun 8, 2020, at 11:28 AM, Collin Strassburger via users > wrote: > > Hello David, > > The slot calculation is based on physical cores rather than logical cores. > The 4 CPUs you are seeing there are logical CPUs. And since your processor > has 2 threads per core, you have two physical cores; yielding a total of 4 > logical cores (which is reported to lscpu). On machines with only 1 thread > per core, the cpu number is the number of physical cores. > > Thanks, > Collin > > > -Original Message- > From: users On Behalf Of David Mathog via > users > Sent: Monday, June 8, 2020 2:19 PM > To: users@lists.open-mpi.org > Cc: David Mathog > Subject: [OMPI users] slot number calculation when no config files? > > Using OpenMPI 4.0.1 and no configuration files of any kind on a linux machine > which shows 4 CPUS and 2 threads per CPU. When this is run: > > mpirun --allow-run-as-root --oversubscribe -np 3 > /usr/common/modules/el8/x86_64/software/q6/6.0.1-CentOS-vanilla/bin/qdynp > eq2.inp > > it works and the expected 3 processes run. However if oversubscribe is > omitted then this happens: > > There are not enough slots available in the system to satisfy the 3 slots > that were requested by the application: > > It will run with 2 though even without -oversubscribe. It looks like it is > using CPUs/2 to calculate the slot limit and ignoring threads. > Where is this slot calculation documented? > > More details: > > > $ cat /etc/centos-release > CentOS Linux release 8.1.1911 (Core) > $ mpirun --version > mpirun (Open MPI) 4.0.1 > $ lscpu | head -6 > Architecture:x86_64 > CPU op-mode(s): 32-bit, 64-bit > Byte Order: Little Endian > CPU(s): 4 > On-line CPU(s) list: 0-3 > Thread(s) per core: 2 > $ gcc --version > gcc (GCC) 8.3.1 20190507 (Red Hat 8.3.1-4) > > Thanks, > > David Mathog
Re: [OMPI users] slot number calculation when no config files?
So default slots = "CPU"s/2 but there is some leeway in what is considered to be a CPU? So far I have not found the actual formula documented anywhere. Thanks, David Mathog On Mon, Jun 8, 2020 at 11:37 AM Ralph Castain via users wrote: > > Note that you can also resolve it by adding --use-hwthread-cpus to your cmd > line - it instructs mpirun to treat the HWTs as independent cpus so you would > have 4 slots in this case. > > > > On Jun 8, 2020, at 11:28 AM, Collin Strassburger via users > > wrote: > > > > Hello David, > > > > The slot calculation is based on physical cores rather than logical cores. > > The 4 CPUs you are seeing there are logical CPUs. And since your processor > > has 2 threads per core, you have two physical cores; yielding a total of 4 > > logical cores (which is reported to lscpu). On machines with only 1 thread > > per core, the cpu number is the number of physical cores. > > > > Thanks, > > Collin > > > > > > -Original Message- > > From: users On Behalf Of David Mathog > > via users > > Sent: Monday, June 8, 2020 2:19 PM > > To: users@lists.open-mpi.org > > Cc: David Mathog > > Subject: [OMPI users] slot number calculation when no config files? > > > > Using OpenMPI 4.0.1 and no configuration files of any kind on a linux > > machine which shows 4 CPUS and 2 threads per CPU. When this is run: > > > > mpirun --allow-run-as-root --oversubscribe -np 3 > > /usr/common/modules/el8/x86_64/software/q6/6.0.1-CentOS-vanilla/bin/qdynp > > eq2.inp > > > > it works and the expected 3 processes run. However if oversubscribe is > > omitted then this happens: > > > > There are not enough slots available in the system to satisfy the 3 slots > > that were requested by the application: > > > > It will run with 2 though even without -oversubscribe. It looks like it is > > using CPUs/2 to calculate the slot limit and ignoring threads. > > Where is this slot calculation documented? > > > > More details: > > > > > > $ cat /etc/centos-release > > CentOS Linux release 8.1.1911 (Core) > > $ mpirun --version > > mpirun (Open MPI) 4.0.1 > > $ lscpu | head -6 > > Architecture:x86_64 > > CPU op-mode(s): 32-bit, 64-bit > > Byte Order: Little Endian > > CPU(s): 4 > > On-line CPU(s) list: 0-3 > > Thread(s) per core: 2 > > $ gcc --version > > gcc (GCC) 8.3.1 20190507 (Red Hat 8.3.1-4) > > > > Thanks, > > > > David Mathog > >
Re: [OMPI users] slot number calculation when no config files?
The default number of slots is the number of physical cores; overall, this tends to yield the best performance. Most of the time, a "CPU" is a logical core. A processor with hyper-threading (or SMT (simultaneous multi-threading)) will be counted as two(+) logical processors. I hope this helps, Collin -Original Message- From: users On Behalf Of David Mathog via users Sent: Monday, June 8, 2020 2:48 PM To: Open MPI Users Cc: David Mathog Subject: Re: [OMPI users] slot number calculation when no config files? So default slots = "CPU"s/2 but there is some leeway in what is considered to be a CPU? So far I have not found the actual formula documented anywhere. Thanks, David Mathog On Mon, Jun 8, 2020 at 11:37 AM Ralph Castain via users wrote: > > Note that you can also resolve it by adding --use-hwthread-cpus to your cmd > line - it instructs mpirun to treat the HWTs as independent cpus so you would > have 4 slots in this case. > > > > On Jun 8, 2020, at 11:28 AM, Collin Strassburger via users > > wrote: > > > > Hello David, > > > > The slot calculation is based on physical cores rather than logical cores. > > The 4 CPUs you are seeing there are logical CPUs. And since your processor > > has 2 threads per core, you have two physical cores; yielding a total of 4 > > logical cores (which is reported to lscpu). On machines with only 1 thread > > per core, the cpu number is the number of physical cores. > > > > Thanks, > > Collin > > > > > > -Original Message- > > From: users On Behalf Of David > > Mathog via users > > Sent: Monday, June 8, 2020 2:19 PM > > To: users@lists.open-mpi.org > > Cc: David Mathog > > Subject: [OMPI users] slot number calculation when no config files? > > > > Using OpenMPI 4.0.1 and no configuration files of any kind on a linux > > machine which shows 4 CPUS and 2 threads per CPU. When this is run: > > > > mpirun --allow-run-as-root --oversubscribe -np 3 > > /usr/common/modules/el8/x86_64/software/q6/6.0.1-CentOS-vanilla/bin/ > > qdynp > > eq2.inp > > > > it works and the expected 3 processes run. However if oversubscribe is > > omitted then this happens: > > > > There are not enough slots available in the system to satisfy the 3 slots > > that were requested by the application: > > > > It will run with 2 though even without -oversubscribe. It looks like it is > > using CPUs/2 to calculate the slot limit and ignoring threads. > > Where is this slot calculation documented? > > > > More details: > > > > > > $ cat /etc/centos-release > > CentOS Linux release 8.1.1911 (Core) $ mpirun --version mpirun (Open > > MPI) 4.0.1 $ lscpu | head -6 > > Architecture:x86_64 > > CPU op-mode(s): 32-bit, 64-bit > > Byte Order: Little Endian > > CPU(s): 4 > > On-line CPU(s) list: 0-3 > > Thread(s) per core: 2 > > $ gcc --version > > gcc (GCC) 8.3.1 20190507 (Red Hat 8.3.1-4) > > > > Thanks, > > > > David Mathog > >
