Re: [OMPI users] Scheduling dynamically spawned processes

[Ralph Castain]

I believe I answered that question. You can use the hostfile info key, or you 
can use the host info key - either one will do what you require.

On May 13, 2011, at 4:11 PM, Rodrigo Silva Oliveira wrote:

> Hi,
> 
> I think I was not specific enough. I need to spawn the copies of a process in 
> a unique mpi_spawn call. It is, I have to specify a list of machines and how 
> many copies of the process will be spawned on each one. Is it possible?
> 
> I would be something like that:
> 
> machines #copies
> m11
> m23
> m31
> 
> After an unique call to spawn, I want the copies running in this fashion. I 
> tried use a hostfile with the option slot, but I'm not sure if it is the best 
> way.
> 
> hostfile:
> 
> m1 slots=1
> m2 slots=3
> m3 slots=1
> 
> Thanks
> 
> -- 
> Rodrigo Silva Oliveira
> M.Sc. Student - Computer Science
> Universidade Federal de Minas Gerais
> www.dcc.ufmg.br/~rsilva
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Scheduling dynamically spawned processes

[Rodrigo Silva Oliveira]

Hi,

I think I was not specific enough. I need to spawn the copies of a process
in a unique mpi_spawn call. It is, I have to specify a list of machines and
how many copies of the process will be spawned on each one. Is it possible?

I would be something like that:

machines #copies
m11
m23
m31

After an unique call to spawn, I want the copies running in this fashion. I
tried use a hostfile with the option slot, but I'm not sure if it is the
best way.

hostfile:

m1 slots=1
m2 slots=3
m3 slots=1

Thanks

-- 
Rodrigo Silva Oliveira
M.Sc. Student - Computer Science
Universidade Federal de Minas Gerais
www.dcc.ufmg.br/~rsilva 

Re: [OMPI users] btl_openib_cpc_include rdmacm questions

[Brock Palen]

On May 13, 2011, at 4:09 PM, Dave Love wrote:

> Jeff Squyres  writes:
> 
>> On May 11, 2011, at 3:21 PM, Dave Love wrote:
>> 
>>> We can reproduce it with IMB.  We could provide access, but we'd have to
>>> negotiate with the owners of the relevant nodes to give you interactive
>>> access to them.  Maybe Brock's would be more accessible?  (If you
>>> contact me, I may not be able to respond for a few days.)
>> 
>> Brock has replied off-list that he, too, is able to reliably reproduce the 
>> issue with IMB, and is working to get access for us.  Many thanks for your 
>> offer; let's see where Brock's access takes us.
> 
> Good.  Let me know if we could be useful
> 
 -- we have not closed this issue,
>>> 
>>> Which issue?   I couldn't find a relevant-looking one.
>> 
>> https://svn.open-mpi.org/trac/ompi/ticket/2714
> 
> Thanks.  In csse it's useful info, it hangs for me with 1.5.3 & np=32 on
> connectx with more than one collective I can't recall.

Extra data point, that ticket said it ran with mpi_preconnect_mpi 1,  well that 
doesn't help here, both my production code (crash) and IMB still hang.


Brock Palen
www.umich.edu/~brockp
Center for Advanced Computing
bro...@umich.edu
(734)936-1985

> 
> -- 
> Excuse the typping -- I have a broken wrist
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> 

Re: [OMPI users] btl_openib_cpc_include rdmacm questions

[Dave Love]

Jeff Squyres  writes:

> On May 11, 2011, at 3:21 PM, Dave Love wrote:
>
>> We can reproduce it with IMB.  We could provide access, but we'd have to
>> negotiate with the owners of the relevant nodes to give you interactive
>> access to them.  Maybe Brock's would be more accessible?  (If you
>> contact me, I may not be able to respond for a few days.)
>
> Brock has replied off-list that he, too, is able to reliably reproduce the 
> issue with IMB, and is working to get access for us.  Many thanks for your 
> offer; let's see where Brock's access takes us.

Good.  Let me know if we could be useful

>>> -- we have not closed this issue,
>> 
>> Which issue?   I couldn't find a relevant-looking one.
>
> https://svn.open-mpi.org/trac/ompi/ticket/2714

Thanks.  In csse it's useful info, it hangs for me with 1.5.3 & np=32 on
connectx with more than one collective I can't recall.

-- 
Excuse the typping -- I have a broken wrist

Re: [OMPI users] Scheduling dynamically spawned processes

[Ralph Castain]

On May 12, 2011, at 9:53 PM, Rodrigo Silva Oliveira wrote:

> Hi there.
> 
> I'm developing a distributed system with a communication layer based on Open 
> MPI. As part of my project, I have to create a process scheduler. So I 
> decided to use the MPI_Spawn function to dynamically create (it is an 
> requirement of the system) the process across the cluster's nodes.
> 
> Is there some way to specify how many copies of a process to spawn to each 
> node? For instance, I want to spawn 5 copies of a mpi program to 3 nodes usin 
> the scheme bellow:
> 
> 1 to host 1
> 3 to host 2
> 1 to host 3
> 
> obs.: I am using openmpi-1.5.3. The default policies provided by the ompi is 
> not applicable to my problem because I will decide the scheme based on 
> specific criteria.

Use the "host" info key. You would specify your dynamic spawn as three apps, 
each running the same executable. It's the equivalent of

mpirun -n 1 -host host1 app : -n 3 -host host2 app : -n 1 -host host3 app


> 
> Thanks in advance.
> 
> -- 
> Rodrigo Silva Oliveira
> M.Sc. Student - Computer Science
> Universidade Federal de Minas Gerais
> www.dcc.ufmg.br/~rsilva
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] MPI_COMM_DUP freeze with OpenMPI 1.4.1

[francoise.r...@obs.ujf-grenoble.fr]

Hi,

The debugger traces are captured when the different tasks are blocked.
Before the MPI_COMM_DUP, a MPI_undefined color has been affected to the 
master process and a MPI_COMM_SPLIT construct a new communicator not 
containing the master.
The master process doesn't call the MPI_COMM_DUP routine, and so the 
master is not blocked at this level of instruction, but further in the 
program, at a barrier call. The master process behaviour is normal, it 
wait for the slaves which are all blocked in MPI_COMM_DUP.


Here is the MUMPS portion of code (in zmumps_part1.F file) where the 
slaves call MPI_COMM_DUP , id%PAR and MASTER are initialized to 0 before :


CALL MPI_COMM_SIZE(id%COMM, id%NPROCS, IERR )
IF ( id%PAR .eq. 0 ) THEN
IF ( id%MYID .eq. MASTER ) THEN
color = MPI_UNDEFINED
ELSE
color = 0
END IF
CALL MPI_COMM_SPLIT( id%COMM, color, 0,
& id%COMM_NODES, IERR )
id%NSLAVES = id%NPROCS - 1
ELSE
CALL MPI_COMM_DUP( id%COMM, id%COMM_NODES, IERR )
id%NSLAVES = id%NPROCS
END IF

IF (id%PAR .ne. 0 .or. id%MYID .NE. MASTER) THEN
CALL MPI_COMM_DUP( id%COMM_NODES, id%COMM_LOAD, IERR
ENDIF

--

In our case (id%PAR = 0), only the second MPI_COMM_DUP call is executed 
on the slaves.


MUMPS library and our program are compiled with intel fortran 12 and I 
have test -O1 option with no more success.


Franc,oise.


George Bosilca wrote:

On May 10, 2011, at 08:10 , Tim Prince wrote:

  

On 5/10/2011 6:43 AM, francoise.r...@obs.ujf-grenoble.fr wrote:


Hi,

I compile a parallel program with OpenMPI 1.4.1 (compiled with intel
compilers 12 from composerxe package) . This program is linked to MUMPS
library 4.9.2, compiled with the same compilers and link with intel MKL.
The OS is linux debian.
No error in compiling or running the job, but the program freeze inside
a call to "zmumps" routine, when the slaves process call MPI_COMM_DUP
routine.

The program is executed on 2 nodes of 12 cores each (westmere
processors) with the following command :

mpirun -np 24 --machinefile $OAR_NODE_FILE -mca plm_rsh_agent "oarsh"
--mca btl self,openib -x LD_LIBRARY_PATH ./prog

We have 12 process running on each node. We submit the job with OAR
batch scheduler (the $OAR_NODE_FILE variable and "oarsh" command are
specific to this scheduler and are usually working well with openmpi )

via gdb, on the slaves, we can see that they are blocked in MPI_COMM_DUP :
  


Francoise,

Based on your traces the workers and the master are not doing the same MPI 
call. The workers are blocked in an MPI_Comm_dup in sub_pbdirect_init.f90:44, 
while the master is blocked in an MPI_Barrier in sub_pbdirect_init.f90:62. Can 
you verify that the slaves and the master are calling the MPI_Barrier and the 
MPI_Comm_dup in the same logical order?

  george.


  

(gdb) where
#0 0x2b32c1533113 in poll () from /lib/libc.so.6
#1 0x00adf52c in poll_dispatch ()
#2 0x00adcea3 in opal_event_loop ()
#3 0x00ad69f9 in opal_progress ()
#4 0x00a34b4e in mca_pml_ob1_recv ()
#5 0x009b0768 in
ompi_coll_tuned_allreduce_intra_recursivedoubling ()
#6 0x009ac829 in ompi_coll_tuned_allreduce_intra_dec_fixed ()
#7 0x0097e271 in ompi_comm_allreduce_intra ()
#8 0x0097dd06 in ompi_comm_nextcid ()
#9 0x0097be01 in ompi_comm_dup ()
#10 0x009a0785 in PMPI_Comm_dup ()
#11 0x0097931d in pmpi_comm_dup__ ()
#12 0x00644251 in zmumps (id=...) at zmumps_part1.F:144
#13 0x004c0d03 in sub_pbdirect_init (id=..., matrix_build=...)
at sub_pbdirect_init.f90:44
#14 0x00628706 in fwt2d_elas_v2 () at fwt2d_elas.f90:1048


the master wait further :

(gdb) where
#0 0x2b9dc9f3e113 in poll () from /lib/libc.so.6
#1 0x00adf52c in poll_dispatch ()
#2 0x00adcea3 in opal_event_loop ()
#3 0x00ad69f9 in opal_progress ()
#4 0x0098f294 in ompi_request_default_wait_all ()
#5 0x00a06e56 in ompi_coll_tuned_sendrecv_actual ()
#6 0x009ab8e3 in ompi_coll_tuned_barrier_intra_bruck ()
#7 0x009ac926 in ompi_coll_tuned_barrier_intra_dec_fixed ()
#8 0x009a0b20 in PMPI_Barrier ()
#9 0x00978c93 in pmpi_barrier__ ()
#10 0x004c0dc4 in sub_pbdirect_init (id=..., matrix_build=...)
at sub_pbdirect_init.f90:62
#11 0x00628706 in fwt2d_elas_v2 () at fwt2d_elas.f90:1048


Remark :
The same code compiled and run well with intel MPI library, from the
same intel package, on the same nodes.

  

Did you try compiling with equivalent options in each compiler?  For example, 
(supposing you had gcc 4.6)
gcc -O3 -funroll-loops --param max-unroll-times=2 -march=corei7
would be equivalent (as closely as I know) to
icc -fp-model source -msse4.2 -ansi-alias

As you should be aware, default settings in icc are more closely equivalent to
gcc -O3 -ffast-math -fno-cx-limited-range -funroll-loops --param 
max-unroll-times=2 -fnostrict-aliasing

The options I suggest as an upper limit are probably more aggressive than most 
people have 

[OMPI users] Scheduling dynamically spawned processes

[Rodrigo Oliveira]

Hi there.

I'm developing a distributed system with a communication layer based on Open
MPI. As part of my project, I have to create a process scheduler. So I
decided to use the MPI_Spawn function to dynamically create (it is an
requirement of the system) the process across the cluster's nodes.

Is there some way to specify how many copies of a process to spawn to each
node? For instance, I want to spawn 5 copies of a mpi program to 3 nodes
usin the scheme bellow:

*1 to host 1*
*3 to host 2*
*1 to host 3*

obs.: I am using openmpi-1.5.3. The default policies provided by the ompi is
not applicable to my problem because I will decide the scheme based on
specific criteria.

Thanks in advance.

-- 
Rodrigo Silva Oliveira
M.Sc. Student - Computer Science
Universidade Federal de Minas Gerais
www.dcc.ufmg.br/~rsilva 

[OMPI users] Scheduling dynamically spawned processes

[Rodrigo Silva Oliveira]

Hi there.

I'm developing a distributed system with a communication layer based on Open
MPI. As part of my project, I have to create a process scheduler. So I
decided to use the MPI_Spawn function to dynamically create (it is an
requirement of the system) the process across the cluster's nodes.

Is there some way to specify how many copies of a process to spawn to each
node? For instance, I want to spawn 5 copies of a mpi program to 3 nodes
usin the scheme bellow:

*1 to host 1*
*3 to host 2*
*1 to host 3*

obs.: I am using openmpi-1.5.3. The default policies provided by the ompi is
not applicable to my problem because I will decide the scheme based on
specific criteria.

Thanks in advance.

-- 
Rodrigo Silva Oliveira
M.Sc. Student - Computer Science
Universidade Federal de Minas Gerais
www.dcc.ufmg.br/~rsilva