Re: [OMPI users] OMPI users] Unable to mpirun from within torque

Wed, 07 Sep 2016 08:39:09 -0700 

Hi,
You are right. Yes the library is there and is linking to libtorque.so. Sorry for the confusion. 


Is there any other information I can provide? I am seriously new to all 
of this.


Best,
Oswin

On 2016-09-07 17:16, r...@open-mpi.org wrote:

You aren’t looking in the right place - there is an “openmpi”
directory underneath that one, and the mca_xxx libraries are down
there


On Sep 7, 2016, at 7:43 AM, Oswin Krause 
<oswin.kra...@ruhr-uni-bochum.de> wrote:


Hi Gilles,

I do not have this library. Maybe this helps already...


libmca_common_sm.so  libmpi_mpifh.so  libmpi_usempif08.so          
libompitrace.so  libopen-rte.so
libmpi_cxx.so        libmpi.so        libmpi_usempi_ignore_tkr.so  
libopen-pal.so   liboshmem.so



and mpirun does only link to libopen-pal/libopen-rte (aside the 
standard stuff)



But still it is telling me that it has support for tm? libtorque is 
there and the headers are also there and since i have enabled 
tm...*sigh*


Thanks again!

Oswin

On 2016-09-07 16:21, Gilles Gouaillardet wrote:

Note the torque library will only show up if you configure'd with
--disable-dlopen. Otherwise, you can ldd
/.../lib/openmpi/mca_plm_tm.so
Cheers,
Gilles
Bennet Fauber <ben...@umich.edu> wrote:

Oswin,
Does the torque library show up if you run
$ ldd mpirun
That would indicate that Torque support is compiled in.
Also, what happens if you use the same hostfile, or some hostfile as
an explicit argument when you run mpirun from within the torque job?
-- bennet
On Wed, Sep 7, 2016 at 9:25 AM, Oswin Krause
<oswin.kra...@ruhr-uni-bochum.de> wrote:

Hi Gilles,

Thanks for the hint with the machinefile. I know it is not 
equivalent and i
do not intend to use that approach. I just wanted to know whether I 
could

start the program successfully at all.

Outside torque(4.2), rsh seems to be used which works fine, 
querying a

password if no kerberos ticket is there
Here is the output:
[zbh251@a00551 ~]$ mpirun -V
mpirun (Open MPI) 2.0.1
[zbh251@a00551 ~]$ ompi_info | grep ras

                MCA ras: loadleveler (MCA v2.1.0, API v2.0.0, 
Component

v2.0.1)

                MCA ras: simulator (MCA v2.1.0, API v2.0.0, 
Component

v2.0.1)

                MCA ras: slurm (MCA v2.1.0, API v2.0.0, Component 
v2.0.1)
                MCA ras: tm (MCA v2.1.0, API v2.0.0, Component 
v2.0.1)

[zbh251@a00551 ~]$ mpirun --mca plm_base_verbose 10 --tag-output
-display-map hostname

[a00551.science.domain:04104] mca: base: components_register: 
registering

framework plm components

[a00551.science.domain:04104] mca: base: components_register: found 
loaded

component isolated

[a00551.science.domain:04104] mca: base: components_register: 
component

isolated has no register or open function

[a00551.science.domain:04104] mca: base: components_register: found 
loaded

component rsh

[a00551.science.domain:04104] mca: base: components_register: 
component rsh

register function successful

[a00551.science.domain:04104] mca: base: components_register: found 
loaded

component slurm

[a00551.science.domain:04104] mca: base: components_register: 
component

slurm register function successful

[a00551.science.domain:04104] mca: base: components_register: found 
loaded

component tm

[a00551.science.domain:04104] mca: base: components_register: 
component tm

register function successful

[a00551.science.domain:04104] mca: base: components_open: opening 
plm

components

[a00551.science.domain:04104] mca: base: components_open: found 
loaded

component isolated

[a00551.science.domain:04104] mca: base: components_open: component 
isolated

open function successful

[a00551.science.domain:04104] mca: base: components_open: found 
loaded

component rsh

[a00551.science.domain:04104] mca: base: components_open: component 
rsh open

function successful

[a00551.science.domain:04104] mca: base: components_open: found 
loaded

component slurm

[a00551.science.domain:04104] mca: base: components_open: component 
slurm

open function successful

[a00551.science.domain:04104] mca: base: components_open: found 
loaded

component tm

[a00551.science.domain:04104] mca: base: components_open: component 
tm open

function successful

[a00551.science.domain:04104] mca:base:select: Auto-selecting plm 
components
[a00551.science.domain:04104] mca:base:select:(  plm) Querying 
component

[isolated]

[a00551.science.domain:04104] mca:base:select:(  plm) Query of 
component

[isolated] set priority to 0

[a00551.science.domain:04104] mca:base:select:(  plm) Querying 
component

[rsh]

[a00551.science.domain:04104] mca:base:select:(  plm) Query of 
component

[rsh] set priority to 10

[a00551.science.domain:04104] mca:base:select:(  plm) Querying 
component

[slurm]

[a00551.science.domain:04104] mca:base:select:(  plm) Querying 
component

[tm]

[a00551.science.domain:04104] mca:base:select:(  plm) Query of 
component

[tm] set priority to 75

[a00551.science.domain:04104] mca:base:select:(  plm) Selected 
component

[tm]

[a00551.science.domain:04104] mca: base: close: component isolated 
closed
[a00551.science.domain:04104] mca: base: close: unloading component 
isolated
[a00551.science.domain:04104] mca: base: close: component rsh 
closed
[a00551.science.domain:04104] mca: base: close: unloading component 
rsh
[a00551.science.domain:04104] mca: base: close: component slurm 
closed
[a00551.science.domain:04104] mca: base: close: unloading component 
slurm
[a00551.science.domain:04109] mca: base: components_register: 
registering

framework plm components

[a00551.science.domain:04109] mca: base: components_register: found 
loaded

component rsh

[a00551.science.domain:04109] mca: base: components_register: 
component rsh

register function successful

[a00551.science.domain:04109] mca: base: components_open: opening 
plm

components

[a00551.science.domain:04109] mca: base: components_open: found 
loaded

component rsh

[a00551.science.domain:04109] mca: base: components_open: component 
rsh open

function successful

[a00551.science.domain:04109] mca:base:select: Auto-selecting plm 
components
[a00551.science.domain:04109] mca:base:select:(  plm) Querying 
component

[rsh]

[a00551.science.domain:04109] mca:base:select:(  plm) Query of 
component

[rsh] set priority to 10

[a00551.science.domain:04109] mca:base:select:(  plm) Selected 
component

[rsh]

[a00551.science.domain:04109] [[53688,0],1] bind() failed on error 
Address

already in use (98)

[a00551.science.domain:04109] [[53688,0],1] ORTE_ERROR_LOG: Error 
in file

oob_usock_component.c at line 228
Data for JOB [53688,1] offset 0
========================   JOB MAP   ========================
Data for node: a00551  Num slots: 2    Max slots: 0    Num procs: 2

       Process OMPI jobid: [53688,1] App: 0 Process rank: 0 Bound: 
socket
0[core 0[hwt 0-1]], socket 0[core 1[hwt 0-1]], socket 0[core 2[hwt 
0-1]],
socket 0[core 3[hwt 0-1]], socket 0[core 4[hwt 0-1]], socket 0[core 
5[hwt
0-1]], socket 0[core 6[hwt 0-1]], socket 0[core 7[hwt 0-1]], socket 
0[core

8[hwt 0-1]], socket 0[core 9[hwt
0-1]]:[BB/BB/BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../../../..]

       Process OMPI jobid: [53688,1] App: 0 Process rank: 1 Bound: 
socket
1[core 10[hwt 0-1]], socket 1[core 11[hwt 0-1]], socket 1[core 
12[hwt 0-1]],
socket 1[core 13[hwt 0-1]], socket 1[core 14[hwt 0-1]], socket 
1[core 15[hwt
0-1]], socket 1[core 16[hwt 0-1]], socket 1[core 17[hwt 0-1]], 
socket 1[core

18[hwt 0-1]], socket 1[core 19[hwt
0-1]]:[../../../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB/BB/BB]

Data for node: a00553.science.domain   Num slots: 1    Max slots: 0 
   Num

procs: 1

       Process OMPI jobid: [53688,1] App: 0 Process rank: 2 Bound: 
socket
0[core 0[hwt 0-1]], socket 0[core 1[hwt 0-1]], socket 0[core 2[hwt 
0-1]],
socket 0[core 3[hwt 0-1]], socket 0[core 4[hwt 0-1]], socket 0[core 
5[hwt
0-1]], socket 0[core 6[hwt 0-1]], socket 0[core 7[hwt 0-1]], socket 
0[core

8[hwt 0-1]], socket 0[core 9[hwt
0-1]]:[BB/BB/BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../../../..]
=============================================================

[a00551.science.domain:04104] [[53688,0],0] complete_setup on job 
[53688,1]
[a00551.science.domain:04104] [[53688,0],0] plm:base:receive update 
proc

state command from [[53688,0],1]
[a00551.science.domain:04104] [[53688,0],0] plm:base:receive got
update_proc_state for job [53688,1]
[1,0]<stdout>:a00551.science.domain
[1,2]<stdout>:a00551.science.domain

[a00551.science.domain:04104] [[53688,0],0] plm:base:receive update 
proc

state command from [[53688,0],1]
[a00551.science.domain:04104] [[53688,0],0] plm:base:receive got
update_proc_state for job [53688,1]
[1,1]<stdout>:a00551.science.domain

[a00551.science.domain:04109] mca: base: close: component rsh 
closed
[a00551.science.domain:04109] mca: base: close: unloading component 
rsh

[a00551.science.domain:04104] mca: base: close: component tm closed

[a00551.science.domain:04104] mca: base: close: unloading component 
tm

On 2016-09-07 14:41, Gilles Gouaillardet wrote:

Hi,
Which version of Open MPI are you running ?

I noted that though you are asking three nodes and one task per 
node,

you have been allocated 2 nodes only.
I do not know if this is related to this issue.
Note if you use the machinefile, a00551 has two slots (since it

appears twice in the machinefile) but a00553 has 20 slots (since 
it
appears once in the machinefile, the number of slots is 
automatically

detected)
Can you run
mpirun --mca plm_base_verbose 10 ...
So we can confirm tm is used.

Before invoking mpirun, you might want to cleanup the ompi 
directory in

/tmp
Cheers,
Gilles
Oswin Krause <oswin.kra...@ruhr-uni-bochum.de> wrote:

Hi,

I am currently trying to set up OpenMPI in torque. OpenMPI is 
build with

tm support. Torque is correctly assigning nodes and I can run
mpi-programs on single nodes just fine. the problem starts when
processes are split between nodes.

For example, I create an interactive session with torque and 
start a

program by
qsub -I -n -l nodes=3:ppn=1
mpirun --tag-output -display-map hostname
which leads to

[a00551.science.domain:15932] [[65415,0],1] bind() failed on 
error

Address already in use (98)

[a00551.science.domain:15932] [[65415,0],1] ORTE_ERROR_LOG: Error 
in

file oob_usock_component.c at line 228
Data for JOB [65415,1] offset 0
========================   JOB MAP   ========================

Data for node: a00551  Num slots: 2    Max slots: 0    Num procs: 
2
       Process OMPI jobid: [65415,1] App: 0 Process rank: 0 
Bound:

socket

0[core 0[hwt 0-1]], socket 0[core 1[hwt 0-1]], socket 0[core 
2[hwt
0-1]], socket 0[core 3[hwt 0-1]], socket 0[core 4[hwt 0-1]], 
socket
0[core 5[hwt 0-1]], socket 0[core 6[hwt 0-1]], socket 0[core 
7[hwt

0-1]], socket 0[core 8[hwt 0-1]], socket 0[core 9[hwt
0-1]]:[BB/BB/BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../../../..]

       Process OMPI jobid: [65415,1] App: 0 Process rank: 1 
Bound:

socket

1[core 10[hwt 0-1]], socket 1[core 11[hwt 0-1]], socket 1[core 
12[hwt
0-1]], socket 1[core 13[hwt 0-1]], socket 1[core 14[hwt 0-1]], 
socket
1[core 15[hwt 0-1]], socket 1[core 16[hwt 0-1]], socket 1[core 
17[hwt

0-1]], socket 1[core 18[hwt 0-1]], socket 1[core 19[hwt
0-1]]:[../../../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB/BB/BB]

Data for node: a00553.science.domain   Num slots: 1    Max slots: 
0

Num
procs: 1

       Process OMPI jobid: [65415,1] App: 0 Process rank: 2 
Bound:

socket

0[core 0[hwt 0-1]], socket 0[core 1[hwt 0-1]], socket 0[core 
2[hwt
0-1]], socket 0[core 3[hwt 0-1]], socket 0[core 4[hwt 0-1]], 
socket
0[core 5[hwt 0-1]], socket 0[core 6[hwt 0-1]], socket 0[core 
7[hwt

0-1]], socket 0[core 8[hwt 0-1]], socket 0[core 9[hwt
0-1]]:[BB/BB/BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../../../..]
=============================================================
[1,0]<stdout>:a00551.science.domain
[1,2]<stdout>:a00551.science.domain
[1,1]<stdout>:a00551.science.domain

if I login on a00551 and start using the hostfile generated by 
the

PBS_NODEFILE, everything works:
(from within the interactive session)
echo $PBS_NODEFILE
/var/lib/torque/aux//278.a00552.science.domain
cat $PBS_NODEFILE
a00551.science.domain
a00553.science.domain
a00551.science.domain
(from within the separate login)

mpirun --hostfile /var/lib/torque/aux//278.a00552.science.domain 
-np 3

--tag-output -display-map hostname
Data for JOB [65445,1] offset 0
========================   JOB MAP   ========================

Data for node: a00551  Num slots: 2    Max slots: 0    Num procs: 
2
       Process OMPI jobid: [65445,1] App: 0 Process rank: 0 
Bound:

socket

0[core 0[hwt 0-1]], socket 0[core 1[hwt 0-1]], socket 0[core 
2[hwt
0-1]], socket 0[core 3[hwt 0-1]], socket 0[core 4[hwt 0-1]], 
socket
0[core 5[hwt 0-1]], socket 0[core 6[hwt 0-1]], socket 0[core 
7[hwt

0-1]], socket 0[core 8[hwt 0-1]], socket 0[core 9[hwt
0-1]]:[BB/BB/BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../../../..]

       Process OMPI jobid: [65445,1] App: 0 Process rank: 1 
Bound:

socket

1[core 10[hwt 0-1]], socket 1[core 11[hwt 0-1]], socket 1[core 
12[hwt
0-1]], socket 1[core 13[hwt 0-1]], socket 1[core 14[hwt 0-1]], 
socket
1[core 15[hwt 0-1]], socket 1[core 16[hwt 0-1]], socket 1[core 
17[hwt

0-1]], socket 1[core 18[hwt 0-1]], socket 1[core 19[hwt
0-1]]:[../../../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB/BB/BB]

Data for node: a00553.science.domain   Num slots: 20   Max slots: 
0

Num
procs: 1

       Process OMPI jobid: [65445,1] App: 0 Process rank: 2 
Bound:

socket

0[core 0[hwt 0-1]], socket 0[core 1[hwt 0-1]], socket 0[core 
2[hwt
0-1]], socket 0[core 3[hwt 0-1]], socket 0[core 4[hwt 0-1]], 
socket
0[core 5[hwt 0-1]], socket 0[core 6[hwt 0-1]], socket 0[core 
7[hwt

0-1]], socket 0[core 8[hwt 0-1]], socket 0[core 9[hwt
0-1]]:[BB/BB/BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../../../..]
=============================================================
[1,0]<stdout>:a00551.science.domain
[1,2]<stdout>:a00553.science.domain
[1,1]<stdout>:a00551.science.domain

I am kind of lost whats going on here. Anyone having an idea? I 
am

seriously considering this to be the problem of kerberos

authentification that we have to work with, but I fail to see how 
this

should affect the sockets.
Best,
Oswin
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users


_______________________________________________
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users






















					Reply via email to