Re: [OMPI users] OpenMP + OpenMPI

[Jeff Squyres]

On Dec 6, 2007, at 9:54 AM, Durga Choudhury wrote:

Automatically striping large messages across multiple NICs is  
certainly a very nice feature; I was not aware that OpenMPI does  
this transparently. (I wonder if other MPI implementations do this  
or not). However, I have the following concern: Since the  
communication over an ethernet NIC is most likely over IP, does it  
take into account the route cost when striping messages? For  
example, host A and B in the MPD ring might be connected via two  
NICs, one direct and one via an intermediate router, or one with a  
large bandwidth and another with a small bandwidth. Does OpenMPI  
send a smaller chunk of data over a route with a higher cost?


Not unless you tell it.

In IB networks, the network API exposes bandwidth differences of the  
NIC and Open MPI takes that into account by deciding how much data to  
send down each endpoint.  Open MPI does not currently know anything  
about / try to optimize based on the costs of different routes.


On a TCP network, whether you go through 2 or 3 switches -- does it  
really matter?  The latency is so high that adding another switch (or  
2 or 3 or ...) may not make much of a difference anyway.  Raw  
bandwidth differences between two networks will make a difference, but  
number of hops -- as long as they're not *too* difference -- might not.


Also consider: if you're combining 100Mbps and 1Gbps ethernet networks  
-- is it really worth it?  If your goal is simple bandwidth addition,  
note that you're adding a fraction of the capability to the 1Gbps  
network at the cost of additional complexity in your software and/or  
fragmentation reassembly penalties.  Will you really see more  
delivered bandwidth?  It's probably dependent upon your application  
(e.g., are you continually sending very large messages?).  You might  
get much more bang for your buck if you combine like networks (e.g.,  
2x100Mbps or 2x1Gbps) because you'll be [potentially] doubling your  
bandwidth.


Because of this concern, I think the channel bonding approach  
someone else suggested is more preferable; all these details will be  
taken care of at the hardware level instead of at the IP level.


That's not quite true.  Both approaches are handled in software; one  
is in the kernel, the other is in the middleware.  The hardware is  
unaware that you are striping large messages.


--
Jeff Squyres
Cisco Systems

Re: [OMPI users] OpenMP + OpenMPI

[Durga Choudhury]

Automatically striping large messages across multiple NICs is certainly a
very nice feature; I was not aware that OpenMPI does this transparently. (I
wonder if other MPI implementations do this or not). However, I have the
following concern: Since the communication over an ethernet NIC is most
likely over IP, does it take into account the route cost when striping
messages? For example, host A and B in the MPD ring might be connected via
two NICs, one direct and one via an intermediate router, or one with a large
bandwidth and another with a small bandwidth. Does OpenMPI send a smaller
chunk of data over a route with a higher cost?

Because of this concern, I think the channel bonding approach someone else
suggested is more preferable; all these details will be taken care of at the
hardware level instead of at the IP level.
Thanks
Durga

On Dec 6, 2007 9:42 AM, Jeff Squyres  wrote:

> Wow, that's quite a .sig.  :-)
>
> Open MPI will automatically stripe large messages across however many
> NICs you have.  So you shouldn't need to use multiple threads.
>
> The threading support in the OMPI v1.2 series is broken; it's not
> worth using. There's a big warning in configure when you enable it.  :-)
>
>
> On Dec 5, 2007, at 9:57 PM, Tee Wen Kai wrote:
>
> > Hi everyone,
> >
> > I have installed openmpi-1.2.3. My system has two ethernet ports.
> > Thus, I am trying to make use of both ports to speed up the
> > communication process by using openmp to split into two threads.
> > However, this implementation always cause error. Then I realized
> > that I need to build openmpi using --enable-mpi-threads and use
> > MPI_Init_thread to initialize. But, the initialization always return
> > MPI_THREAD_SINGLE no matter what value I set. Using ompi_info|grep
> > Thread, it shows that thread support has already been activated.
> > Thus, I seek your help to teach me what other configurations I need
> > to set in order to use multi-threads and what are the parameters to
> > include in mpirun in order to use the two ethernet ports.
> >
> > Thank you very much.
> >
> > Regards,
> > Tee
> >
> >
> >
> > _
> >
> >
> >
> >   Many of us spend our time wishing for things we could have if we
> > didn't spend half our time wishing.
> >
> > Looking for last minute shopping deals? Find them fast with Yahoo!
> > Search.___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> --
> Jeff Squyres
> Cisco Systems
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>



-- 
Its a battle between humans and communists;
Which side are you in?
. 

Re: [OMPI users] OpenMP + OpenMPI

[Jeff Squyres]

Wow, that's quite a .sig.  :-)

Open MPI will automatically stripe large messages across however many  
NICs you have.  So you shouldn't need to use multiple threads.


The threading support in the OMPI v1.2 series is broken; it's not  
worth using. There's a big warning in configure when you enable it.  :-)



On Dec 5, 2007, at 9:57 PM, Tee Wen Kai wrote:


Hi everyone,

I have installed openmpi-1.2.3. My system has two ethernet ports.  
Thus, I am trying to make use of both ports to speed up the  
communication process by using openmp to split into two threads.  
However, this implementation always cause error. Then I realized  
that I need to build openmpi using --enable-mpi-threads and use  
MPI_Init_thread to initialize. But, the initialization always return  
MPI_THREAD_SINGLE no matter what value I set. Using ompi_info|grep  
Thread, it shows that thread support has already been activated.  
Thus, I seek your help to teach me what other configurations I need  
to set in order to use multi-threads and what are the parameters to  
include in mpirun in order to use the two ethernet ports.


Thank you very much.

Regards,
Tee



_



  Many of us spend our time wishing for things we could have if we  
didn't spend half our time wishing.


Looking for last minute shopping deals? Find them fast with Yahoo!  
Search.___

users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



--
Jeff Squyres
Cisco Systems

[OMPI users] OpenMP + OpenMPI

[Tee Wen Kai]

Hi everyone,

I have installed openmpi-1.2.3. My system has two ethernet ports. Thus, I am 
trying to make use of both ports to speed up the communication process by using 
openmp to split into two threads. However, this implementation always cause 
error. Then I realized that I need to build openmpi using --enable-mpi-threads 
and use MPI_Init_thread to initialize. But, the initialization always return 
MPI_THREAD_SINGLE no matter what value I set. Using ompi_info|grep Thread, it 
shows that thread support has already been activated. Thus, I seek your help to 
teach me what other configurations I need to set in order to use multi-threads 
and what are the parameters to include in mpirun in order to use the two 
ethernet ports.

Thank you very much.

Regards,
Tee











  _


Many of us spend our time wishing for things we could have if we didn't 
spend half our time wishing.










-
Looking for last minute shopping deals?  Find them fast with Yahoo! Search.