Re: [OMPI devel] why does --rankfile need hostlist?

[Ralph Castain]

The two files have a slightly different format and completely different
meaning. The hostfile specifies how many slots are on a node. The rankfile
specifies a rank and what node/slot it is to be mapped onto.
Rankfiles can use relative node indexing and refer to nodes received from a
resource manager - i.e., without any hostfile.

So the files are intentionally quite different. Trying to combine them would
be rather ugly.
Ralph


On Thu, Jun 18, 2009 at 1:52 PM, Eugene Loh  wrote:

> In order to use "mpirun --rankfile", I also need to specify hosts/hostlist.
>  But that information is redundant with what I provide in the rankfile.  So,
> from a user's point of view, this strikes me as broken.  Yes?  Should I file
> a ticket, or am I missing something here about this functionality?
> ___
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>

Re: [OMPI devel] some question about OMPI communicationinfrastructure

[Jeff Squyres]

A few addendums in no particular order...

1. The ompi/ tree is the MPI layer.  It's the top layer in the stack.   
It uses ORTE and OPAL for various things.


2. The PML (point-to-point messagging layer) is the stuff right behind  
MPI_SEND, MPI_RECV, and friends.  We have two main PMLs: OB1 and CM  
(and some other similar ones, but not important here).  OB1 is  
probably the only one you care about.


3. OB1 effects the majority of the MPI rules and behavior.  It makes  
MPI_Requests, processes them, potentially segments and re-assembles  
individual messages, etc.


4. OB1 uses BTLs (Byte Transfer Layers) to actually move bytes between  
processes.  Each BTL is for a different kind of transport; OB1 uses  
the BML (BTL multiplexing layer; "layer" is a generous term here;  
think of it as trivial BTL pointer array management functionality) to  
manage all the BTLs that it is currently using.


5. OB1 and some of the BTLs use the ORTE layer for "out of band"  
communications, usually for initialization and finalization.  The  
"OOB" ORTE framework is more-or-less equivalent to the BTL framework,  
but it's *only* used for ORTE-level communications (not MPI  
communications).  The RML (routing message layer) ORTE framework is a  
layer on top of the OOB that has the potential to route messages as  
necessary.  To be clear, the OMPI layer always uses the RML, not the  
OOB directly (the RML uses the OOB underneath).


6. A bunch of OOB connections are made during the startup of the MPI  
job.  BTL connections are generally made on an "as needed" basis  
(e.g., during the first MPI_SEND to a given peer).  Ralph will have to  
fill you in on the details of how/when/where OOB connections are made.


7. There is unfortunately little documentation on the OMPI source code  
except comments in the code.  :-\  However, there was a nice writeup  
recently that may be helpful to you:


http://www.open-mpi.org/papers/trinity-btl-2009/

8. Once TCP BTL connections are made, IP addressing is no longer  
necessary in the OMPI-level messages that are sent because the sockets  
are connected point-to-point -- i.e., the peer process is already  
known because we have a socket to them.  The MPI layer messaging more  
contains things like the communicator ID, tag, ...etc.


Hope that helps!


On Jun 18, 2009, at 10:26 AM, Ralph Castain wrote:


Hi Leo

The MPI communications is contained in the ompi/mca/btl code area.  
The BTL's (Bit Transport Layer) actually moves the message data.  
Each BTL is responsible for opening its own connections - ORTE has  
nothing to do with it, except to transport out-of-band (OOB)  
messages to support creating the connection if that specific BTL  
requires it.


If you are interested in TCP communications, you will find all of  
that code in ompi/mca/btl/tcp. It can be confusing down there, so  
expect to spend a little time trying to understand it. I believe  
Jeff has some documentation on the OMPI web site about it (perhaps a  
video?).


The source/destination is embedded in the message, again done by  
each BTL since the receiver must be a BTL of the same type. Again,  
this has nothing to do with ORTE - it is purely up to the BTL. MPI  
communications are also coordinated by the PML, which is responsible  
for matching messages with posted receives. You might need to look  
at the ompi/mca/pml/ob1 code to understand how that works.


Hope that gives you a starting point
Ralph

On Jun 18, 2009, at 7:57 AM, Leo P. wrote:


Hi Everyone,



I wanted to ask some questions about things I am having trouble  
understanding.


•
As far as my understanding of MPI_INIT function, I assumed MPI_INIT  
typically procedure resources required including the sockets. But  
now as I understand from the documentation that openMPI only  
allocated socket when the process has to send a message to a peer.  
If some one could let me where exactly in the code this is  
happening I would appreciate a lot. I guess this is happening in  
ORTE layer so I am spending time looking at it. But if some one  
could let me in which function this is happening it will help me a  
lot.


•
Also I think most of the MPI implementation embed source and  
destination address with the communication protocol. Am I right to  
assume openMPI does the same thing. Is this also happening in the  
ORTE layer.


Is there a documentation about this openMPI site? if there can  
someone please let me know the location of it.




Sincerely,

Leo.P


ICC World Twenty20 England '09 exclusively on YAHOO!  
CRICKET___

devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel



--
Jeff Squyres
Cisco Systems

[OMPI devel] rcache/rb component: defunct?

[Jeff Squyres]

Is the rcache/rb component (on the trunk) defunct?

If so, can we remove it?

--
Jeff Squyres
Cisco Systems

[OMPI devel] why does --rankfile need hostlist?

[Eugene Loh]

In order to use "mpirun --rankfile", I also need to specify 
hosts/hostlist.  But that information is redundant with what I provide 
in the rankfile.  So, from a user's point of view, this strikes me as 
broken.  Yes?  Should I file a ticket, or am I missing something here 
about this functionality?

Re: [OMPI devel] some question about OMPI communication infrastructure

[Leo P.]

Hi Ralph,

Thanks for the response.  And Yes, this give me a good starting point Thanks.

Leo.P





From: Ralph Castain 
To: Open MPI Developers 
Sent: Thursday, 18 June, 2009 9:26:46 PM
Subject: Re: [OMPI devel] some question about OMPI communication infrastructure

Hi Leo

The MPI communications is contained in the ompi/mca/btl code area. The BTL's 
(Bit Transport Layer) actually moves the message data. Each BTL is responsible 
for opening its own connections - ORTE has nothing to do with it, except to 
transport out-of-band (OOB) messages to support creating the connection if that 
specific BTL requires it.

If you are interested in TCP communications, you will find all of that code in 
ompi/mca/btl/tcp. It can be confusing down there, so expect to spend a little 
time trying to understand it. I believe Jeff has some documentation on the OMPI 
web site about it (perhaps a video?).

The source/destination is embedded in the message, again done by each BTL since 
the receiver must be a BTL of the same type. Again, this has nothing to do with 
ORTE - it is purely up to the BTL. MPI communications are also coordinated by 
the PML, which is responsible for matching messages with posted receives. You 
might need to look at the ompi/mca/pml/ob1 code to understand how that works.

Hope that gives you a starting point
Ralph


On Jun 18, 2009, at 7:57 AM, Leo P. wrote:

Hi Everyone,

I wanted to ask some questions about things I am having trouble understanding.
1. As far as my understanding of MPI_INIT function, I assumed MPI_INIT 
typically procedure resources required including the sockets. But now as I 
understand from the documentation that openMPI only allocated socket when the 
process has to send a message to a peer. If some one could let me where exactly 
in the code this is happening I would appreciate a lot. I guess this is 
happening in ORTE layer so I am spending time looking at it. But if some one 
could let me in which function this is happening it will help me a lot. 

2. Also I think most of the MPI implementation embed source and 
destination address with the communication protocol. Am I right to assume 
openMPI does the same thing. Is this also happening in the ORTE layer.
Is there a documentation about this openMPI site? if there can someone please 
let me know the location of it.



Sincerely,
Leo.P

ICC World Twenty20 England '09 exclusively on YAHOO! 
CRICKET___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel



  Love Cricket? Check out live scores, photos, video highlights and more. 
Click here http://cricket.yahoo.com

Re: [OMPI devel] Use of OPAL_PREFIX to relocate a lib

[Jeff Squyres]

On Jun 18, 2009, at 11:25 AM, Sylvain Jeaugey wrote:


My problem seems related to library generation through RPM, not with
1.3.2, nor the patch.



I'm not sure I understand -- is there something we need to fix in our  
SRPM?


--
Jeff Squyres
Cisco Systems

Re: [OMPI devel] Use of OPAL_PREFIX to relocate a lib

[Sylvain Jeaugey]

Ok, never mind.

My problem seems related to library generation through RPM, not with 
1.3.2, nor the patch.


Sylvain

On Thu, 18 Jun 2009, Sylvain Jeaugey wrote:


Hi all,

Until Open MPI 1.3 (maybe 1.3.1), I used to find it convenient to be able to 
move a library from its "normal" place (either /usr or /opt) to somewhere 
else (i.e. my NFS home account) to be able to try things only on my account.


So, I used to set OPAL_PREFIX to the root of the Open MPI directory and all 
went fine.


I don't know if relocation was intended in the first place, but with 1.3.2, 
this seems to be broken.


It may have something to do with this patch (and maybe others) :

# HG changeset patch
# User bosilca
# Date 1159647750 0
# Node ID c7152b893f1ce1bc54eea2dc3f06c7e359011fdd
# Parent  676a8fbdbb161f0b84a1c6bb12e2324c8a749c56
All the OPAL_ defines from the install_dirs.h contain ABSOLUTE path. 
Therefore,

there is no need to prepend OPAL_PREFIX to them.

diff -r 676a8fbdbb16 -r c7152b893f1c opal/tools/wrappers/opal_wrapper.c
--- a/opal/tools/wrappers/opal_wrapper.cFri Sep 29 23:58:58 2006 
+
+++ b/opal/tools/wrappers/opal_wrapper.cSat Sep 30 20:22:30 2006 
+

@@ -561,9 +561,9 @@
if (0 != strcmp(OPAL_INCLUDEDIR, "/usr/include")) {
char *line;
#if defined(__WINDOWS__)
-asprintf(, OPAL_INCLUDE_PATTERN OPAL_PREFIX "\"\\%s\"", 
OPAL_INCLUDEDIR);
+asprintf(, OPAL_INCLUDE_PATTERN "\"\\%s\"", 
OPAL_INCLUDEDIR);

#else
-asprintf(, OPAL_INCLUDE_PATTERN OPAL_PREFIX"/%s", 
OPAL_INCLUDEDIR);

+asprintf(, OPAL_INCLUDE_PATTERN "/%s", OPAL_INCLUDEDIR);
#endif  /* defined(__WINDOWS__) */
opal_argv_append_nosize(_flags, line);
free(line);

George, is there a rationale behind this patch for disabling relocation of 
libraries ? Do you think reverting only this patch would bring back the 
relocation functionality ?


TIA,

Sylvain

Re: [OMPI devel] some question about OMPI communication infrastructure

[Ralph Castain]

Hi Leo

The MPI communications is contained in the ompi/mca/btl code area. The  
BTL's (Bit Transport Layer) actually moves the message data. Each BTL  
is responsible for opening its own connections - ORTE has nothing to  
do with it, except to transport out-of-band (OOB) messages to support  
creating the connection if that specific BTL requires it.


If you are interested in TCP communications, you will find all of that  
code in ompi/mca/btl/tcp. It can be confusing down there, so expect to  
spend a little time trying to understand it. I believe Jeff has some  
documentation on the OMPI web site about it (perhaps a video?).


The source/destination is embedded in the message, again done by each  
BTL since the receiver must be a BTL of the same type. Again, this has  
nothing to do with ORTE - it is purely up to the BTL. MPI  
communications are also coordinated by the PML, which is responsible  
for matching messages with posted receives. You might need to look at  
the ompi/mca/pml/ob1 code to understand how that works.


Hope that gives you a starting point
Ralph

On Jun 18, 2009, at 7:57 AM, Leo P. wrote:


Hi Everyone,

I wanted to ask some questions about things I am having trouble  
understanding.
As far as my understanding of MPI_INIT function, I assumed MPI_INIT  
typically procedure resources required including the sockets. But  
now as I understand from the documentation that openMPI only  
allocated socket when the process has to send a message to a peer.  
If some one could let me where exactly in the code this is happening  
I would appreciate a lot. I guess this is happening in ORTE layer so  
I am spending time looking at it. But if some one could let me in  
which function this is happening it will help me a lot.
Also I think most of the MPI implementation embed source and  
destination address with the communication protocol. Am I right to  
assume openMPI does the same thing. Is this also happening in the  
ORTE layer.
Is there a documentation about this openMPI site? if there can  
someone please let me know the location of it.



Sincerely,
Leo.P

ICC World Twenty20 England '09 exclusively on YAHOO!  
CRICKET___

devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel

[OMPI devel] some question about OMPI communication infrastructure

[Leo P.]

Hi Everyone, 

I wanted to ask some questions about things I am having trouble understanding. 
1. As far as my understanding of MPI_INIT function, I assumed MPI_INIT 
typically procedure resources required including the sockets. But now as I 
understand from the documentation that openMPI only allocated socket when the 
process has to send a message to a peer. If some one could let me where exactly 
in the code this is happening I would appreciate a lot. I guess this is 
happening in ORTE layer so I am spending time looking at it. But if some one 
could let me in which function this is happening it will help me a lot. 

2. Also I think most of the MPI implementation embed source and 
destination address with the communication protocol. Am I right to assume 
openMPI does the same thing. Is this also happening in the ORTE layer.
Is there a documentation about this openMPI site? if there can someone please 
let me know the location of it.



Sincerely,
Leo.P 


  ICC World Twenty20 England 09 exclusively on YAHOO! CRICKET 
http://cricket.yahoo.com

[OMPI devel] Use of OPAL_PREFIX to relocate a lib

[Sylvain Jeaugey]

Hi all,

Until Open MPI 1.3 (maybe 1.3.1), I used to find it convenient to be able 
to move a library from its "normal" place (either /usr or /opt) to 
somewhere else (i.e. my NFS home account) to be able to try things only on 
my account.


So, I used to set OPAL_PREFIX to the root of the Open MPI directory and 
all went fine.


I don't know if relocation was intended in the first place, but with 
1.3.2, this seems to be broken.


It may have something to do with this patch (and maybe others) :

# HG changeset patch
# User bosilca
# Date 1159647750 0
# Node ID c7152b893f1ce1bc54eea2dc3f06c7e359011fdd
# Parent  676a8fbdbb161f0b84a1c6bb12e2324c8a749c56
All the OPAL_ defines from the install_dirs.h contain ABSOLUTE path. 
Therefore,

there is no need to prepend OPAL_PREFIX to them.

diff -r 676a8fbdbb16 -r c7152b893f1c opal/tools/wrappers/opal_wrapper.c
--- a/opal/tools/wrappers/opal_wrapper.cFri Sep 29 23:58:58 2006 +
+++ b/opal/tools/wrappers/opal_wrapper.cSat Sep 30 20:22:30 2006 +
@@ -561,9 +561,9 @@
 if (0 != strcmp(OPAL_INCLUDEDIR, "/usr/include")) {
 char *line;
 #if defined(__WINDOWS__)
-asprintf(, OPAL_INCLUDE_PATTERN OPAL_PREFIX "\"\\%s\"", 
OPAL_INCLUDEDIR);
+asprintf(, OPAL_INCLUDE_PATTERN "\"\\%s\"", OPAL_INCLUDEDIR);
 #else
-asprintf(, OPAL_INCLUDE_PATTERN OPAL_PREFIX"/%s", 
OPAL_INCLUDEDIR);
+asprintf(, OPAL_INCLUDE_PATTERN "/%s", OPAL_INCLUDEDIR);
 #endif  /* defined(__WINDOWS__) */
 opal_argv_append_nosize(_flags, line);
 free(line);

George, is there a rationale behind this patch for disabling relocation of 
libraries ? Do you think reverting only this patch would bring back the 
relocation functionality ?


TIA,

Sylvain