[OMPI devel] Is mpirun in slurm producing the correct "srun" cmd behind the scene?
Hello,
My attempt to run and troubleshoot my an OMPI job under a slurm allocation does
not work as I would expect.
The result below has led me to believe that under the hood, in this setup
(SLURM with OMPI) that the correct srun options is not being used
when I call, mpirun directly.
Specifically the “cpu-bind=none” is breaking, but it also looks like the
nodelist is incorrect.
The job script.
1 #!/bin/bash
2
3 #SBATCH --job-name=mpi-hostname
4 #SBATCH --partition=dev
5 #SBATCH --account=Account1
6 #SBATCH --time=01:00:00
7 #SBATCH --nodes=2
8 #SBATCH --ntasks-per-node=1
9 #SBATCH --begin=now+10
10 #SBATCH --output="%x-%u-%N-%j.txt" # jobName-userId-hostName-jobId.txt
11
12
13
14 # -- #
15 #module load DefApps
16 #module purge >/dev/null 2>&1
17 ##module load staging/slurm >/dev/null 2>&1
18 #module load gcc/4.8.5 openmpi >/dev/null 2>&1
19 #module --ignore_cache spider openmpi/3.1.3 >/dev/null 2>&1
20 #
21 # -- #
22 #
23 MPI_RUN=$(which orterun)
24 if [[ -z "${MPI_RUN:+x}" ]]; then
25 echo "ERROR: Cannot find 'mpirun' executable..exiting"
26 exit 1
27 fi
28
29 echo
30 #CMD="orterun -npernode 1 -np 2 /bin/hostname"
31 #CMD="srun /bin/hostname"
32 #CMD="srun -N2 -n2 --mpi=pmi2 /bin/hostname"
33 #MCMD="/sw/dev/openmpi401/bin/mpirun --bind-to-core --report-bindings -mca
btl openib,self -mca plm_base_verbose 10 /bin/hostname"
34 MCMD="/sw/dev/openmpi401/bin/mpirun --report-bindings -mca btl openib,self
-mca plm_base_verbose 10 /bin/hostname"
35 echo "INFO: Executing the command: $MCMD"
36 $MCMD
37 sync
Here is the output:
2 user1@node-login7g:~/git/slurm-jobs$ more mpi-hostname-user1-node513-835.txt
3
4 INFO: Executing the command: /sw/dev/openmpi401/bin/mpirun
--report-bindings -mca btl openib,self -mca plm_base_verbose 10 /bin/hostname
5 [node513:32514] mca: base: components_register: registering framework plm
components
6 [node513:32514] mca: base: components_register: found loaded component
isolated
7 [node513:32514] mca: base: components_register: component isolated has no
register or open function
8 [node513:32514] mca: base: components_register: found loaded component rsh
9 [node513:32514] mca: base: components_register: component rsh register
function successful
10 [node513:32514] mca: base: components_register: found loaded component slurm
11 [node513:32514] mca: base: components_register: component slurm register
function successful
12 [node513:32514] mca: base: components_register: found loaded component tm
13 [node513:32514] mca: base: components_register: component tm register
function successful
14 [node513:32514] mca: base: components_open: opening plm components
15 [node513:32514] mca: base: components_open: found loaded component isolated
16 [node513:32514] mca: base: components_open: component isolated open function
successful
17 [node513:32514] mca: base: components_open: found loaded component rsh
18 [node513:32514] mca: base: components_open: component rsh open function
successful
19 [node513:32514] mca: base: components_open: found loaded component slurm
20 [node513:32514] mca: base: components_open: component slurm open function
successful
21 [node513:32514] mca: base: components_open: found loaded component tm
22 [node513:32514] mca: base: components_open: component tm open function
successful
23 [node513:32514] mca:base:select: Auto-selecting plm components
24 [node513:32514] mca:base:select:( plm) Querying component [isolated]
25 [node513:32514] mca:base:select:( plm) Query of component [isolated] set
priority to 0
26 [node513:32514] mca:base:select:( plm) Querying component [rsh]
27 [node513:32514] mca:base:select:( plm) Query of component [rsh] set
priority to 10
28 [node513:32514] mca:base:select:( plm) Querying component [slurm]
29 [node513:32514] mca:base:select:( plm) Query of component [slurm] set
priority to 75
30 [node513:32514] mca:base:select:( plm) Querying component [tm]
31 [node513:32514] mca:base:select:( plm) Selected component [slurm]
32 [node513:32514] mca: base: close: component isolated closed
33 [node513:32514] mca: base: close: unloading component isolated
34 [node513:32514] mca: base: close: component rsh closed
35 [node513:32514] mca: base: close: unloading component rsh
36 [node513:32514] mca: base: close: component tm closed
37 [node513:32514] mca: base: close: unloading component tm
38 [node513:32514] [[4367,0],0] plm:slurm: final top-level argv:
39 srun --ntasks-per-node=1 --kill-on-bad-exit --cpu_bind=none
--nodes=1 --nodelist=node514 --ntasks=1 orted -mca orte_report_bindings "1"
-mca ess "slurm" -mca ess_base_jobid "286195712" -mca ess_base_vpid "1" -mca
ess_base_num_procs "2" -mca orte_node_regex "node[3:513-514]@0(2)" -mca
orte_hnp_uri "286195712.0;tcp://172.30.146
40 10.38.146.45:43031" -mca btl "openib,self" -mca plm_ba
Re: [OMPI devel] Is mpirun in slurm producing the correct "srun" cmd behind the scene?
Lawrence,
this is a known issue (--cpu_bind optioned was removed from SLURM in
favor of the --cpu-bind option) and the fix will be available in Open
MPI 4.0.1
Meanwhile, you can manually download and apply the patch at
https://github.com/open-mpi/ompi/pull/6445.patch or use a nightly
build of the v4.0.x branch.
Cheers,
Gilles
On Sat, Jul 6, 2019 at 5:35 AM Sorrillo, Lawrence via devel
wrote:
>
> Hello,
>
>
>
> My attempt to run and troubleshoot my an OMPI job under a slurm allocation
> does not work as I would expect.
>
> The result below has led me to believe that under the hood, in this setup
> (SLURM with OMPI) that the correct srun options is not being used
>
> when I call, mpirun directly.
>
>
>
> Specifically the “cpu-bind=none” is breaking, but it also looks like the
> nodelist is incorrect.
>
>
>
> The job script.
>
>
>
> 1 #!/bin/bash
>
> 2
>
> 3 #SBATCH --job-name=mpi-hostname
>
> 4 #SBATCH --partition=dev
>
> 5 #SBATCH --account=Account1
>
> 6 #SBATCH --time=01:00:00
>
> 7 #SBATCH --nodes=2
>
> 8 #SBATCH --ntasks-per-node=1
>
> 9 #SBATCH --begin=now+10
>
> 10 #SBATCH --output="%x-%u-%N-%j.txt" # jobName-userId-hostName-jobId.txt
>
> 11
>
> 12
>
> 13
>
> 14 # -- #
>
> 15 #module load DefApps
>
> 16 #module purge >/dev/null 2>&1
>
> 17 ##module load staging/slurm >/dev/null 2>&1
>
> 18 #module load gcc/4.8.5 openmpi >/dev/null 2>&1
>
> 19 #module --ignore_cache spider openmpi/3.1.3 >/dev/null 2>&1
>
> 20 #
>
> 21 # -- #
>
> 22 #
>
> 23 MPI_RUN=$(which orterun)
>
> 24 if [[ -z "${MPI_RUN:+x}" ]]; then
>
> 25 echo "ERROR: Cannot find 'mpirun' executable..exiting"
>
> 26 exit 1
>
> 27 fi
>
> 28
>
> 29 echo
>
> 30 #CMD="orterun -npernode 1 -np 2 /bin/hostname"
>
> 31 #CMD="srun /bin/hostname"
>
> 32 #CMD="srun -N2 -n2 --mpi=pmi2 /bin/hostname"
>
> 33 #MCMD="/sw/dev/openmpi401/bin/mpirun --bind-to-core --report-bindings
> -mca btl openib,self -mca plm_base_verbose 10 /bin/hostname"
>
> 34 MCMD="/sw/dev/openmpi401/bin/mpirun --report-bindings -mca btl
> openib,self -mca plm_base_verbose 10 /bin/hostname"
>
> 35 echo "INFO: Executing the command: $MCMD"
>
> 36 $MCMD
>
> 37 sync
>
>
>
> Here is the output:
>
>
>
> 2 user1@node-login7g:~/git/slurm-jobs$ more mpi-hostname-user1-node513-835.txt
>
> 3
>
> 4 INFO: Executing the command: /sw/dev/openmpi401/bin/mpirun
> --report-bindings -mca btl openib,self -mca plm_base_verbose 10
> /bin/hostname
>
> 5 [node513:32514] mca: base: components_register: registering framework plm
> components
>
> 6 [node513:32514] mca: base: components_register: found loaded component
> isolated
>
> 7 [node513:32514] mca: base: components_register: component isolated has no
> register or open function
>
> 8 [node513:32514] mca: base: components_register: found loaded component rsh
>
> 9 [node513:32514] mca: base: components_register: component rsh register
> function successful
>
> 10 [node513:32514] mca: base: components_register: found loaded component
> slurm
>
> 11 [node513:32514] mca: base: components_register: component slurm register
> function successful
>
> 12 [node513:32514] mca: base: components_register: found loaded component tm
>
> 13 [node513:32514] mca: base: components_register: component tm register
> function successful
>
> 14 [node513:32514] mca: base: components_open: opening plm components
>
> 15 [node513:32514] mca: base: components_open: found loaded component isolated
>
> 16 [node513:32514] mca: base: components_open: component isolated open
> function successful
>
> 17 [node513:32514] mca: base: components_open: found loaded component rsh
>
> 18 [node513:32514] mca: base: components_open: component rsh open function
> successful
>
> 19 [node513:32514] mca: base: components_open: found loaded component slurm
>
> 20 [node513:32514] mca: base: components_open: component slurm open function
> successful
>
> 21 [node513:32514] mca: base: components_open: found loaded component tm
>
> 22 [node513:32514] mca: base: components_open: component tm open function
> successful
>
> 23 [node513:32514] mca:base:select: Auto-selecting plm components
>
> 24 [node513:32514] mca:base:select:( plm) Querying component [isolated]
>
> 25 [node513:32514] mca:base:select:( plm) Query of component [isolated] set
> priority to 0
>
> 26 [node513:32514] mca:base:select:( plm) Querying component [rsh]
>
> 27 [node513:32514] mca:base:select:( plm) Query of component [rsh] set
> priority to 10
>
> 28 [node513:32514] mca:base:select:( plm) Querying component [slurm]
>
> 29 [node513:32514] mca:base:select:( plm) Query of component [slurm] set
> priority to 75
>
> 30 [node513:32514] mca:base:select:( plm) Querying component [tm]
>
> 31 [node513:32514] mca:base:select:( plm) Selected component [slurm]
>
> 32 [node513:32514] mca: base: close: component isolated closed
>
>
Re: [OMPI devel] Is mpirun in slurm producing the correct "srun" cmd behind the scene?
Lawrence,
My apologies for the incorrect link.
The patch is at https://github.com/open-mpi/ompi/pull/6672.patch
Cheers,
Gilles
On Sat, Jul 6, 2019 at 12:21 PM Gilles Gouaillardet
wrote:
>
> Lawrence,
>
> this is a known issue (--cpu_bind optioned was removed from SLURM in
> favor of the --cpu-bind option) and the fix will be available in Open
> MPI 4.0.1
>
> Meanwhile, you can manually download and apply the patch at
> https://github.com/open-mpi/ompi/pull/6445.patch or use a nightly
> build of the v4.0.x branch.
>
> Cheers,
>
> Gilles
>
> On Sat, Jul 6, 2019 at 5:35 AM Sorrillo, Lawrence via devel
> wrote:
> >
> > Hello,
> >
> >
> >
> > My attempt to run and troubleshoot my an OMPI job under a slurm allocation
> > does not work as I would expect.
> >
> > The result below has led me to believe that under the hood, in this setup
> > (SLURM with OMPI) that the correct srun options is not being used
> >
> > when I call, mpirun directly.
> >
> >
> >
> > Specifically the “cpu-bind=none” is breaking, but it also looks like the
> > nodelist is incorrect.
> >
> >
> >
> > The job script.
> >
> >
> >
> > 1 #!/bin/bash
> >
> > 2
> >
> > 3 #SBATCH --job-name=mpi-hostname
> >
> > 4 #SBATCH --partition=dev
> >
> > 5 #SBATCH --account=Account1
> >
> > 6 #SBATCH --time=01:00:00
> >
> > 7 #SBATCH --nodes=2
> >
> > 8 #SBATCH --ntasks-per-node=1
> >
> > 9 #SBATCH --begin=now+10
> >
> > 10 #SBATCH --output="%x-%u-%N-%j.txt" #
> > jobName-userId-hostName-jobId.txt
> >
> > 11
> >
> > 12
> >
> > 13
> >
> > 14 # --
> > #
> >
> > 15 #module load DefApps
> >
> > 16 #module purge >/dev/null 2>&1
> >
> > 17 ##module load staging/slurm >/dev/null 2>&1
> >
> > 18 #module load gcc/4.8.5 openmpi >/dev/null 2>&1
> >
> > 19 #module --ignore_cache spider openmpi/3.1.3 >/dev/null 2>&1
> >
> > 20 #
> >
> > 21 # --
> > #
> >
> > 22 #
> >
> > 23 MPI_RUN=$(which orterun)
> >
> > 24 if [[ -z "${MPI_RUN:+x}" ]]; then
> >
> > 25 echo "ERROR: Cannot find 'mpirun' executable..exiting"
> >
> > 26 exit 1
> >
> > 27 fi
> >
> > 28
> >
> > 29 echo
> >
> > 30 #CMD="orterun -npernode 1 -np 2 /bin/hostname"
> >
> > 31 #CMD="srun /bin/hostname"
> >
> > 32 #CMD="srun -N2 -n2 --mpi=pmi2 /bin/hostname"
> >
> > 33 #MCMD="/sw/dev/openmpi401/bin/mpirun --bind-to-core --report-bindings
> > -mca btl openib,self -mca plm_base_verbose 10 /bin/hostname"
> >
> > 34 MCMD="/sw/dev/openmpi401/bin/mpirun --report-bindings -mca btl
> > openib,self -mca plm_base_verbose 10 /bin/hostname"
> >
> > 35 echo "INFO: Executing the command: $MCMD"
> >
> > 36 $MCMD
> >
> > 37 sync
> >
> >
> >
> > Here is the output:
> >
> >
> >
> > 2 user1@node-login7g:~/git/slurm-jobs$ more
> > mpi-hostname-user1-node513-835.txt
> >
> > 3
> >
> > 4 INFO: Executing the command: /sw/dev/openmpi401/bin/mpirun
> > --report-bindings -mca btl openib,self -mca plm_base_verbose 10
> > /bin/hostname
> >
> > 5 [node513:32514] mca: base: components_register: registering framework
> > plm components
> >
> > 6 [node513:32514] mca: base: components_register: found loaded component
> > isolated
> >
> > 7 [node513:32514] mca: base: components_register: component isolated has
> > no register or open function
> >
> > 8 [node513:32514] mca: base: components_register: found loaded component
> > rsh
> >
> > 9 [node513:32514] mca: base: components_register: component rsh register
> > function successful
> >
> > 10 [node513:32514] mca: base: components_register: found loaded component
> > slurm
> >
> > 11 [node513:32514] mca: base: components_register: component slurm register
> > function successful
> >
> > 12 [node513:32514] mca: base: components_register: found loaded component tm
> >
> > 13 [node513:32514] mca: base: components_register: component tm register
> > function successful
> >
> > 14 [node513:32514] mca: base: components_open: opening plm components
> >
> > 15 [node513:32514] mca: base: components_open: found loaded component
> > isolated
> >
> > 16 [node513:32514] mca: base: components_open: component isolated open
> > function successful
> >
> > 17 [node513:32514] mca: base: components_open: found loaded component rsh
> >
> > 18 [node513:32514] mca: base: components_open: component rsh open function
> > successful
> >
> > 19 [node513:32514] mca: base: components_open: found loaded component slurm
> >
> > 20 [node513:32514] mca: base: components_open: component slurm open
> > function successful
> >
> > 21 [node513:32514] mca: base: components_open: found loaded component tm
> >
> > 22 [node513:32514] mca: base: components_open: component tm open function
> > successful
> >
> > 23 [node513:32514] mca:base:select: Auto-selecting plm components
> >
> > 24 [node513:32514] mca:base:select:( plm) Querying component [isolated]
> >
> > 25 [node513:32514] mca:base:select:( plm) Query of component [i
