[OMPI devel] Is mpirun in slurm producing the correct "srun" cmd behind the scene?
Hello, My attempt to run and troubleshoot my an OMPI job under a slurm allocation does not work as I would expect. The result below has led me to believe that under the hood, in this setup (SLURM with OMPI) that the correct srun options is not being used when I call, mpirun directly. Specifically the “cpu-bind=none” is breaking, but it also looks like the nodelist is incorrect. The job script. 1 #!/bin/bash 2 3 #SBATCH --job-name=mpi-hostname 4 #SBATCH --partition=dev 5 #SBATCH --account=Account1 6 #SBATCH --time=01:00:00 7 #SBATCH --nodes=2 8 #SBATCH --ntasks-per-node=1 9 #SBATCH --begin=now+10 10 #SBATCH --output="%x-%u-%N-%j.txt" # jobName-userId-hostName-jobId.txt 11 12 13 14 # -- # 15 #module load DefApps 16 #module purge >/dev/null 2>&1 17 ##module load staging/slurm >/dev/null 2>&1 18 #module load gcc/4.8.5 openmpi >/dev/null 2>&1 19 #module --ignore_cache spider openmpi/3.1.3 >/dev/null 2>&1 20 # 21 # -- # 22 # 23 MPI_RUN=$(which orterun) 24 if [[ -z "${MPI_RUN:+x}" ]]; then 25 echo "ERROR: Cannot find 'mpirun' executable..exiting" 26 exit 1 27 fi 28 29 echo 30 #CMD="orterun -npernode 1 -np 2 /bin/hostname" 31 #CMD="srun /bin/hostname" 32 #CMD="srun -N2 -n2 --mpi=pmi2 /bin/hostname" 33 #MCMD="/sw/dev/openmpi401/bin/mpirun --bind-to-core --report-bindings -mca btl openib,self -mca plm_base_verbose 10 /bin/hostname" 34 MCMD="/sw/dev/openmpi401/bin/mpirun --report-bindings -mca btl openib,self -mca plm_base_verbose 10 /bin/hostname" 35 echo "INFO: Executing the command: $MCMD" 36 $MCMD 37 sync Here is the output: 2 user1@node-login7g:~/git/slurm-jobs$ more mpi-hostname-user1-node513-835.txt 3 4 INFO: Executing the command: /sw/dev/openmpi401/bin/mpirun --report-bindings -mca btl openib,self -mca plm_base_verbose 10 /bin/hostname 5 [node513:32514] mca: base: components_register: registering framework plm components 6 [node513:32514] mca: base: components_register: found loaded component isolated 7 [node513:32514] mca: base: components_register: component isolated has no register or open function 8 [node513:32514] mca: base: components_register: found loaded component rsh 9 [node513:32514] mca: base: components_register: component rsh register function successful 10 [node513:32514] mca: base: components_register: found loaded component slurm 11 [node513:32514] mca: base: components_register: component slurm register function successful 12 [node513:32514] mca: base: components_register: found loaded component tm 13 [node513:32514] mca: base: components_register: component tm register function successful 14 [node513:32514] mca: base: components_open: opening plm components 15 [node513:32514] mca: base: components_open: found loaded component isolated 16 [node513:32514] mca: base: components_open: component isolated open function successful 17 [node513:32514] mca: base: components_open: found loaded component rsh 18 [node513:32514] mca: base: components_open: component rsh open function successful 19 [node513:32514] mca: base: components_open: found loaded component slurm 20 [node513:32514] mca: base: components_open: component slurm open function successful 21 [node513:32514] mca: base: components_open: found loaded component tm 22 [node513:32514] mca: base: components_open: component tm open function successful 23 [node513:32514] mca:base:select: Auto-selecting plm components 24 [node513:32514] mca:base:select:( plm) Querying component [isolated] 25 [node513:32514] mca:base:select:( plm) Query of component [isolated] set priority to 0 26 [node513:32514] mca:base:select:( plm) Querying component [rsh] 27 [node513:32514] mca:base:select:( plm) Query of component [rsh] set priority to 10 28 [node513:32514] mca:base:select:( plm) Querying component [slurm] 29 [node513:32514] mca:base:select:( plm) Query of component [slurm] set priority to 75 30 [node513:32514] mca:base:select:( plm) Querying component [tm] 31 [node513:32514] mca:base:select:( plm) Selected component [slurm] 32 [node513:32514] mca: base: close: component isolated closed 33 [node513:32514] mca: base: close: unloading component isolated 34 [node513:32514] mca: base: close: component rsh closed 35 [node513:32514] mca: base: close: unloading component rsh 36 [node513:32514] mca: base: close: component tm closed 37 [node513:32514] mca: base: close: unloading component tm 38 [node513:32514] [[4367,0],0] plm:slurm: final top-level argv: 39 srun --ntasks-per-node=1 --kill-on-bad-exit --cpu_bind=none --nodes=1 --nodelist=node514 --ntasks=1 orted -mca orte_report_bindings "1" -mca ess "slurm" -mca ess_base_jobid "286195712" -mca ess_base_vpid "1" -mca ess_base_num_procs "2" -mca orte_node_regex "node[3:513-514]@0(2)" -mca orte_hnp_uri "286195712.0;tcp://172.30.146 40 10.38.146.45:43031" -mca btl "openib,self" -mca plm_ba
Re: [OMPI devel] Is mpirun in slurm producing the correct "srun" cmd behind the scene?
Lawrence, this is a known issue (--cpu_bind optioned was removed from SLURM in favor of the --cpu-bind option) and the fix will be available in Open MPI 4.0.1 Meanwhile, you can manually download and apply the patch at https://github.com/open-mpi/ompi/pull/6445.patch or use a nightly build of the v4.0.x branch. Cheers, Gilles On Sat, Jul 6, 2019 at 5:35 AM Sorrillo, Lawrence via devel wrote: > > Hello, > > > > My attempt to run and troubleshoot my an OMPI job under a slurm allocation > does not work as I would expect. > > The result below has led me to believe that under the hood, in this setup > (SLURM with OMPI) that the correct srun options is not being used > > when I call, mpirun directly. > > > > Specifically the “cpu-bind=none” is breaking, but it also looks like the > nodelist is incorrect. > > > > The job script. > > > > 1 #!/bin/bash > > 2 > > 3 #SBATCH --job-name=mpi-hostname > > 4 #SBATCH --partition=dev > > 5 #SBATCH --account=Account1 > > 6 #SBATCH --time=01:00:00 > > 7 #SBATCH --nodes=2 > > 8 #SBATCH --ntasks-per-node=1 > > 9 #SBATCH --begin=now+10 > > 10 #SBATCH --output="%x-%u-%N-%j.txt" # jobName-userId-hostName-jobId.txt > > 11 > > 12 > > 13 > > 14 # -- # > > 15 #module load DefApps > > 16 #module purge >/dev/null 2>&1 > > 17 ##module load staging/slurm >/dev/null 2>&1 > > 18 #module load gcc/4.8.5 openmpi >/dev/null 2>&1 > > 19 #module --ignore_cache spider openmpi/3.1.3 >/dev/null 2>&1 > > 20 # > > 21 # -- # > > 22 # > > 23 MPI_RUN=$(which orterun) > > 24 if [[ -z "${MPI_RUN:+x}" ]]; then > > 25 echo "ERROR: Cannot find 'mpirun' executable..exiting" > > 26 exit 1 > > 27 fi > > 28 > > 29 echo > > 30 #CMD="orterun -npernode 1 -np 2 /bin/hostname" > > 31 #CMD="srun /bin/hostname" > > 32 #CMD="srun -N2 -n2 --mpi=pmi2 /bin/hostname" > > 33 #MCMD="/sw/dev/openmpi401/bin/mpirun --bind-to-core --report-bindings > -mca btl openib,self -mca plm_base_verbose 10 /bin/hostname" > > 34 MCMD="/sw/dev/openmpi401/bin/mpirun --report-bindings -mca btl > openib,self -mca plm_base_verbose 10 /bin/hostname" > > 35 echo "INFO: Executing the command: $MCMD" > > 36 $MCMD > > 37 sync > > > > Here is the output: > > > > 2 user1@node-login7g:~/git/slurm-jobs$ more mpi-hostname-user1-node513-835.txt > > 3 > > 4 INFO: Executing the command: /sw/dev/openmpi401/bin/mpirun > --report-bindings -mca btl openib,self -mca plm_base_verbose 10 > /bin/hostname > > 5 [node513:32514] mca: base: components_register: registering framework plm > components > > 6 [node513:32514] mca: base: components_register: found loaded component > isolated > > 7 [node513:32514] mca: base: components_register: component isolated has no > register or open function > > 8 [node513:32514] mca: base: components_register: found loaded component rsh > > 9 [node513:32514] mca: base: components_register: component rsh register > function successful > > 10 [node513:32514] mca: base: components_register: found loaded component > slurm > > 11 [node513:32514] mca: base: components_register: component slurm register > function successful > > 12 [node513:32514] mca: base: components_register: found loaded component tm > > 13 [node513:32514] mca: base: components_register: component tm register > function successful > > 14 [node513:32514] mca: base: components_open: opening plm components > > 15 [node513:32514] mca: base: components_open: found loaded component isolated > > 16 [node513:32514] mca: base: components_open: component isolated open > function successful > > 17 [node513:32514] mca: base: components_open: found loaded component rsh > > 18 [node513:32514] mca: base: components_open: component rsh open function > successful > > 19 [node513:32514] mca: base: components_open: found loaded component slurm > > 20 [node513:32514] mca: base: components_open: component slurm open function > successful > > 21 [node513:32514] mca: base: components_open: found loaded component tm > > 22 [node513:32514] mca: base: components_open: component tm open function > successful > > 23 [node513:32514] mca:base:select: Auto-selecting plm components > > 24 [node513:32514] mca:base:select:( plm) Querying component [isolated] > > 25 [node513:32514] mca:base:select:( plm) Query of component [isolated] set > priority to 0 > > 26 [node513:32514] mca:base:select:( plm) Querying component [rsh] > > 27 [node513:32514] mca:base:select:( plm) Query of component [rsh] set > priority to 10 > > 28 [node513:32514] mca:base:select:( plm) Querying component [slurm] > > 29 [node513:32514] mca:base:select:( plm) Query of component [slurm] set > priority to 75 > > 30 [node513:32514] mca:base:select:( plm) Querying component [tm] > > 31 [node513:32514] mca:base:select:( plm) Selected component [slurm] > > 32 [node513:32514] mca: base: close: component isolated closed > >
Re: [OMPI devel] Is mpirun in slurm producing the correct "srun" cmd behind the scene?
Lawrence, My apologies for the incorrect link. The patch is at https://github.com/open-mpi/ompi/pull/6672.patch Cheers, Gilles On Sat, Jul 6, 2019 at 12:21 PM Gilles Gouaillardet wrote: > > Lawrence, > > this is a known issue (--cpu_bind optioned was removed from SLURM in > favor of the --cpu-bind option) and the fix will be available in Open > MPI 4.0.1 > > Meanwhile, you can manually download and apply the patch at > https://github.com/open-mpi/ompi/pull/6445.patch or use a nightly > build of the v4.0.x branch. > > Cheers, > > Gilles > > On Sat, Jul 6, 2019 at 5:35 AM Sorrillo, Lawrence via devel > wrote: > > > > Hello, > > > > > > > > My attempt to run and troubleshoot my an OMPI job under a slurm allocation > > does not work as I would expect. > > > > The result below has led me to believe that under the hood, in this setup > > (SLURM with OMPI) that the correct srun options is not being used > > > > when I call, mpirun directly. > > > > > > > > Specifically the “cpu-bind=none” is breaking, but it also looks like the > > nodelist is incorrect. > > > > > > > > The job script. > > > > > > > > 1 #!/bin/bash > > > > 2 > > > > 3 #SBATCH --job-name=mpi-hostname > > > > 4 #SBATCH --partition=dev > > > > 5 #SBATCH --account=Account1 > > > > 6 #SBATCH --time=01:00:00 > > > > 7 #SBATCH --nodes=2 > > > > 8 #SBATCH --ntasks-per-node=1 > > > > 9 #SBATCH --begin=now+10 > > > > 10 #SBATCH --output="%x-%u-%N-%j.txt" # > > jobName-userId-hostName-jobId.txt > > > > 11 > > > > 12 > > > > 13 > > > > 14 # -- > > # > > > > 15 #module load DefApps > > > > 16 #module purge >/dev/null 2>&1 > > > > 17 ##module load staging/slurm >/dev/null 2>&1 > > > > 18 #module load gcc/4.8.5 openmpi >/dev/null 2>&1 > > > > 19 #module --ignore_cache spider openmpi/3.1.3 >/dev/null 2>&1 > > > > 20 # > > > > 21 # -- > > # > > > > 22 # > > > > 23 MPI_RUN=$(which orterun) > > > > 24 if [[ -z "${MPI_RUN:+x}" ]]; then > > > > 25 echo "ERROR: Cannot find 'mpirun' executable..exiting" > > > > 26 exit 1 > > > > 27 fi > > > > 28 > > > > 29 echo > > > > 30 #CMD="orterun -npernode 1 -np 2 /bin/hostname" > > > > 31 #CMD="srun /bin/hostname" > > > > 32 #CMD="srun -N2 -n2 --mpi=pmi2 /bin/hostname" > > > > 33 #MCMD="/sw/dev/openmpi401/bin/mpirun --bind-to-core --report-bindings > > -mca btl openib,self -mca plm_base_verbose 10 /bin/hostname" > > > > 34 MCMD="/sw/dev/openmpi401/bin/mpirun --report-bindings -mca btl > > openib,self -mca plm_base_verbose 10 /bin/hostname" > > > > 35 echo "INFO: Executing the command: $MCMD" > > > > 36 $MCMD > > > > 37 sync > > > > > > > > Here is the output: > > > > > > > > 2 user1@node-login7g:~/git/slurm-jobs$ more > > mpi-hostname-user1-node513-835.txt > > > > 3 > > > > 4 INFO: Executing the command: /sw/dev/openmpi401/bin/mpirun > > --report-bindings -mca btl openib,self -mca plm_base_verbose 10 > > /bin/hostname > > > > 5 [node513:32514] mca: base: components_register: registering framework > > plm components > > > > 6 [node513:32514] mca: base: components_register: found loaded component > > isolated > > > > 7 [node513:32514] mca: base: components_register: component isolated has > > no register or open function > > > > 8 [node513:32514] mca: base: components_register: found loaded component > > rsh > > > > 9 [node513:32514] mca: base: components_register: component rsh register > > function successful > > > > 10 [node513:32514] mca: base: components_register: found loaded component > > slurm > > > > 11 [node513:32514] mca: base: components_register: component slurm register > > function successful > > > > 12 [node513:32514] mca: base: components_register: found loaded component tm > > > > 13 [node513:32514] mca: base: components_register: component tm register > > function successful > > > > 14 [node513:32514] mca: base: components_open: opening plm components > > > > 15 [node513:32514] mca: base: components_open: found loaded component > > isolated > > > > 16 [node513:32514] mca: base: components_open: component isolated open > > function successful > > > > 17 [node513:32514] mca: base: components_open: found loaded component rsh > > > > 18 [node513:32514] mca: base: components_open: component rsh open function > > successful > > > > 19 [node513:32514] mca: base: components_open: found loaded component slurm > > > > 20 [node513:32514] mca: base: components_open: component slurm open > > function successful > > > > 21 [node513:32514] mca: base: components_open: found loaded component tm > > > > 22 [node513:32514] mca: base: components_open: component tm open function > > successful > > > > 23 [node513:32514] mca:base:select: Auto-selecting plm components > > > > 24 [node513:32514] mca:base:select:( plm) Querying component [isolated] > > > > 25 [node513:32514] mca:base:select:( plm) Query of component [i