Re: [OMPI users] test OpenMPI without Internet access

[Wen Hao Wang]

Hi Jeff:

Thanks! I will try that.

> On Oct 19, 2008, at 7:05 PM, Wen Hao Wang wrote:
>
>> I have one cluster without Internet connection. I want to test
>> OpenMPI functions on it. It seems MTT can not be used. Do I have any
>> other choice for the testing?
>>
>
> You can always run tests manually.  MTT is simply our harness for
> automated testing, which *usually* (but not always) involves
> downloading the latest nightly snapshot from the IU web site.
>
> You can certainly configure MTT to use a local copy of Open MPI and
> not use the IU web nightly snapshot.
>
>> I have tried lamtest. "make -k check" gave a lot of IB related
>> warnings, indicating that my dat.conf file contained invalid entry.
>> Each machine of my cluster has one IB connectX adapter installed.
>> But do not know why lamtest detected that.
>>
>
>
> You must have built with udapl support.  Open MPI will use as many
> interfaces as you have built for; if you have built support for udapl
> (which I do not believe we build by default on Linux -- you have to
> specifically ask for it to be built because [among other reasons] OMPI
> would prefer to use verbs, not udapl), then we initialize udapl at run
> time and if you have an illegal dat.conf file, then I'm guessing udapl
> is complaining about it.  I don't know very much about udapl, so I
> can't give good guidance here (other than suggesting not building
> udapl support and just testing the native verbs support, which is
> where we put all of our effort for Linux/OpenFabrics/Open MPI support).
>
> OMPI's udapl support is mainly for Sun, because udapl *is* [currently]
> their high-performance network stack.
>
> --
> Jeff Squyres
> Cisco Systems

Wen Hao Wang
Email: wangw...@cn.ibm.com

Re: [OMPI users] OpenMPI runtime-specific environment variable?

[Jeff Squyres]

FWIW, I suggested the OMPI_() method for two reasons:

1. It's stable and would divorce itself from any underlying env  
variables that may change over time (although starting with v1.3, some  
of the ones Ralph mentioned shouldn't be changing)


2. Providing OMPI_* functions is a pre-standardization mechanism to  
see if users really want/can use them.  If we get feedback that they  
are useful, that's a good data point to the Forum to actually have an  
official MPI_* version of the function that will be standardized.


And I wasn't clear before -- I was assuming that this particular  
OMPI_ would allow itself to be called before MPI_INIT().   
Otherwise, that would somewhat defeat the point of Brian's request.  :-)



On Oct 22, 2008, at 10:02 AM, Ralph Castain wrote:


We could - though it isn't clear that it really accomplishes anything.

I believe some of the suggestions on this thread have forgotten about
singletons. If the code calls MPI_Init, we treat that as a singleton
and immediately set all the MPI environmental params - which means the
proc's environment now looks exactly as if it had been launched by
mpirun. That is by design for proper singleton operation. So doing
anything that starts with MPI_Init isn't going to work.

What I think Brian is trying to do is detect that his code was not
launched by mpirun -prior- to calling MPI_Init so he can decide if he
wants to do that at all. Checking for the enviro params I suggested is
a good way to do it - I'm not sure that adding another one really
helps. The key issue is having something he can rely on, and I think
the ones I suggested are probably his best bet for OMPI.

What would be nice is if you MPI Forum types could agree on an MPI-
standard way of doing this so Brian wouldn't have to check a dozen
different variations... :-)

Ralph

On Oct 22, 2008, at 7:21 AM, Jeff Squyres wrote:

> I wonder if it would be useful to have an OMPI-specific extension
> for this kind of functionality, perhaps
> OMPI_Was_launched_by_mpirun() (or something with a better name,
> etc.)...?
>
> This would be a pretty easy function for us to provide (right
> Ralph?).  My question is -- would this (and perhaps other similar
> extensions) be useful to the community at large?
>
>
>
> On Oct 21, 2008, at 5:46 PM, Adams, Brian M wrote:
>
>>> I'm not sure I understand the problem. The ale3d program from
>>> LLNL operates exactly as you describe and it can be built
>>> with mpich, lam, or openmpi.
>>
>> Hi Doug,
>>
>> I'm not sure what reply would be most helpful, so here's an  
attempt.

>>
>> It sounds like we're on the same page with regard to the desired
>> behavior.  Historically, we've been able to detect serial vs.
>> parallel launch of the binary foo, with a whole host of
>> implementations, including those you mention, as well as some
>> vendor-specific implementations (possibly including DEC/OSF, SGI,
>> Sun, and AIX/poe, though I don't know all the details).  We
>> typically distinguish serial from parallel executions on the basis
>> of environment variables set only in the MPI runtime environment.
>> I was just trying to ascertain what variable would be best to test
>> for in an OpenMPI environment, and I think Ralph helped with that.
>>
>> If the ale3d code takes a different approach, I'd love to hear
>> about it, off-list if necessary.
>>
>> Brian
>>
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> --
> Jeff Squyres
> Cisco Systems
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



--
Jeff Squyres
Cisco Systems

Re: [OMPI users] Working with a CellBlade cluster

[Mi Yan]

Lenny,

 Thanks.
 I asked the Cell/BE Linux Kernel developer to get the CPU  mapping :)
The mapping is fixed in current kernel.

 Mi


   
 "Lenny
 Verkhovsky"   
"Open MPI Users"
 Sent by:  
 users-bounces@ope  cc
 n-mpi.org 
   Subject
   Re: [OMPI users] Working with a 
 10/23/2008 01:52  CellBlade cluster   
 PM
   
   
 Please respond to 
  Open MPI Users   
 
   
   




According to https://svn.open-mpi.org/trac/ompi/milestone/Open%20MPI%201.3
very soon,
but you can download trunk version http://www.open-mpi.org/svn/  and check
if it works for you.

how can you check mapping CPUs by OS , my cat /proc/cpuinfo shows very
little info
# cat /proc/cpuinfo
processor   : 0
cpu : Cell Broadband Engine, altivec supported
clock   : 3200.00MHz
revision: 48.0 (pvr 0070 3000)
processor   : 1
cpu : Cell Broadband Engine, altivec supported
clock   : 3200.00MHz
revision: 48.0 (pvr 0070 3000)
processor   : 2
cpu : Cell Broadband Engine, altivec supported
clock   : 3200.00MHz
revision: 48.0 (pvr 0070 3000)
processor   : 3
cpu : Cell Broadband Engine, altivec supported
clock   : 3200.00MHz
revision: 48.0 (pvr 0070 3000)
timebase: 2666
platform: Cell
machine : CHRP IBM,0793-1RZ



On Thu, Oct 23, 2008 at 3:00 PM, Mi Yan  wrote:.
  Hi, Lenny,

  So rank file map will be supported in OpenMPI 1.3? I'm using OpenMPI1.2.6
  and did not find parameter "rmaps_rank_file_".
  Do you have idea when OpenMPI 1.3 will be available? OpenMPI 1.3 has
  quite a few features I'm looking for.

  Thanks,

  Mi
  Inactive hide details for "Lenny Verkhovsky" 
  "Lenny Verkhovsky" 




   
  "Lenny   
  Verkhovsky" .
  Sent by: 
  users-bounces@o  
  pen-mpi.org   To
   
"Open MPI Users" <
  10/23/2008us...@open-mpi.org
  05:48 AM  >  
   
cc
 Please respond to 
   Open MPI Users
   
   
   Subject
   
Re: [OMPI users]
Working with a 
CellBlade cluster
   
   
   
   
   
   



  Hi,


  If I understand you correctly the most suitable way to do it is by
  paffinity that we have 

Re: [OMPI users] Working with a CellBlade cluster

[Lenny Verkhovsky]

According to https://svn.open-mpi.org/trac/ompi/milestone/Open%20MPI%201.3 very
soon,
but you can download trunk version http://www.open-mpi.org/svn/  and check
if it works for you.

how can you check mapping CPUs by OS , my cat /proc/cpuinfo shows very
little info
# cat /proc/cpuinfo
processor   : 0
cpu : Cell Broadband Engine, altivec supported
clock   : 3200.00MHz
revision: 48.0 (pvr 0070 3000)
processor   : 1
cpu : Cell Broadband Engine, altivec supported
clock   : 3200.00MHz
revision: 48.0 (pvr 0070 3000)
processor   : 2
cpu : Cell Broadband Engine, altivec supported
clock   : 3200.00MHz
revision: 48.0 (pvr 0070 3000)
processor   : 3
cpu : Cell Broadband Engine, altivec supported
clock   : 3200.00MHz
revision: 48.0 (pvr 0070 3000)
timebase: 2666
platform: Cell
machine : CHRP IBM,0793-1RZ



On Thu, Oct 23, 2008 at 3:00 PM, Mi Yan  wrote:

>  Hi, Lenny,
>
> So rank file map will be supported in OpenMPI 1.3? I'm using OpenMPI1.2.6
> and did not find parameter "rmaps_rank_file_".
> Do you have idea when OpenMPI 1.3 will be available? OpenMPI 1.3 has quite
> a few features I'm looking for.
>
> Thanks,
> Mi
> [image: Inactive hide details for "Lenny Verkhovsky"
> ]"Lenny Verkhovsky" <
> lenny.verkhov...@gmail.com>
>
>
>
> *"Lenny Verkhovsky" *
> Sent by: users-boun...@open-mpi.org
>
> 10/23/2008 05:48 AM   Please respond to
> Open MPI Users 
>
>
> To
>
> "Open MPI Users" 
> cc
>
>
> Subject
>
> Re: [OMPI users] Working with a CellBlade cluster
>
> Hi,
>
>
> If I understand you correctly the most suitable way to do it is by
> paffinity that we have in Open MPI 1.3 and the trank.
> how ever usually OS is distributing processes evenly between sockets by it
> self.
>
> There still no formal FAQ due to a multiple reasons but you can read how to
> use it in the attached scratch ( there were few name changings of the
> params, so check with ompi_info )
>
> shared memory is used between processes that share same machine, and openib
> is used between different machines ( hostnames ), no special mca params are
> needed.
>
> Best Regards
> Lenny,
>
>
>
>   On Sun, Oct 19, 2008 at 10:32 AM, Gilbert Grosdidier <*
> gro...@mail.cern.ch* > wrote:
>
>Working with a CellBlade cluster (QS22), the requirement is to have one
>instance of the executable running on each socket of the blade (there
>are 2
>sockets). The application is of the 'domain decomposition' type, and
>each
>instance is required to often send/receive data with both the remote
>blades and
>the neighbor socket.
>
>Question is : which specification must be used for the mca btl
>component
>to force 1) shmem type messages when communicating with this neighbor
>socket,
>while 2) using openib to communicate with the remote blades ?
>Is '-mca btl sm,openib,self' suitable for this ?
>
>Also, which debug flags could be used to crosscheck that the messages
>are
>_actually_ going thru the right channel for a given channel, please ?
>
>We are currently using OpenMPI 1.2.5 shipped with RHEL5.2 (ppc64).
>Which version do you think is currently the most optimised for these
>processors and problem type ? Should we go towards OpenMPI 1.2.8
>instead ?
>Or even try some OpenMPI 1.3 nightly build ?
>
>Thanks in advance for your help, Gilbert.
>
>___
>users mailing list*
>**us...@open-mpi.org* *
>
> **http://www.open-mpi.org/mailman/listinfo.cgi/users*
>
> *(See attached file: RANKS_FAQ.doc)*
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] ga-4.1 on mx segmentation violation

[SLIM H.A.]

The pdf file in the armci subdirectory says version 1.1, the mx version
is 1.2.5. I'll have to get the debugger out...

Thanks

Henk

> -Original Message-
> From: users-boun...@open-mpi.org 
> [mailto:users-boun...@open-mpi.org] On Behalf Of Patrick Geoffray
> Sent: 22 October 2008 22:32
> To: Open MPI Users
> Subject: Re: [OMPI users] ga-4.1 on mx segmentation violation
> 
> SLIM H.A. wrote:
> > I have built the release candidate for ga-4.1 with OpenMPI 
> 1.2.3 and 
> > portland compilers 7.0.2 for Myrinet mx.
> 
> Which version of ARMCI and MX ?
> 
> > ARMCI configured for 3 cluster nodes. Network protocol is 
> 'MPI-SPAWN'.
> > 0:Segmentation Violation error, status=: 11 0:ARMCI DASSERT fail. 
> > signaltrap.c:SigSegvHandler():299 cond:0
> 
> Segfault caught by ARMCI signal handler. You may want to run 
> it under gdb, to see where the segfault really comes from.
> 
> Patrick
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 

Re: [OMPI users] Working with a CellBlade cluster

[Mi Yan]

Hi, Lenny,

So rank file map will be supported in OpenMPI 1.3?I'm using
OpenMPI1.2.6 and did not find parameter "rmaps_rank_file_".
   Do you have idea when OpenMPI 1.3 will be available?OpenMPI 1.3
has quite a few features I'm looking for.

Thanks,
Mi


   
 "Lenny
 Verkhovsky"   
"Open MPI Users"
 Sent by:  
 users-bounces@ope  cc
 n-mpi.org 
   Subject
   Re: [OMPI users] Working with a 
 10/23/2008 05:48  CellBlade cluster   
 AM
   
   
 Please respond to 
  Open MPI Users   
 
   
   




Hi,


If I understand you correctly the most suitable way to do it is by
paffinity that we have in Open MPI 1.3 and the trank.
how ever usually OS is distributing processes evenly between sockets by it
self.

There still no formal FAQ due to a multiple reasons but you can read how to
use it in the attached scratch ( there were few name changings of the
params, so check with ompi_info )

shared memory is used between processes that share same machine, and openib
is used between different machines ( hostnames ), no special mca params are
needed.

Best Regards
Lenny,







On Sun, Oct 19, 2008 at 10:32 AM, Gilbert Grosdidier 
wrote:
   Working with a CellBlade cluster (QS22), the requirement is to have one
  instance of the executable running on each socket of the blade (there are
  2
  sockets). The application is of the 'domain decomposition' type, and each
  instance is required to often send/receive data with both the remote
  blades and
  the neighbor socket.

   Question is : which specification must be used for the mca btl component
  to force 1) shmem type messages when communicating with this neighbor
  socket,
  while 2) using openib to communicate with the remote blades ?
  Is '-mca btl sm,openib,self' suitable for this ?

   Also, which debug flags could be used to crosscheck that the messages
  are
  _actually_ going thru the right channel for a given channel, please ?

   We are currently using OpenMPI 1.2.5 shipped with RHEL5.2 (ppc64).
  Which version do you think is currently the most optimised for these
  processors and problem type ? Should we go towards OpenMPI 1.2.8
  instead ?
  Or even try some OpenMPI 1.3 nightly build ?

   Thanks in advance for your help,  Gilbert.

  ___
  users mailing list
  us...@open-mpi.org
  http://www.open-mpi.org/mailman/listinfo.cgi/users
(See attached file: RANKS_FAQ.doc)
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

RANKS_FAQ.doc
Description: MS-Word document

Re: [OMPI users] Working with a CellBlade cluster

[Mi Yan]

1.  MCA BTL parameters
With "-mca btl openib,self", both message between two Cell processors on
one QS22 and   messages between two QS22s go through IB.

With "-mca btl openib,sm,slef",  message on one QS22 go through shared
memory,  message between QS22 go through IB,

Depending on the message size and other MCA parameters,  it does not
guarantee message passing on shared memory is faster than on IB.   E.g.
the bandwidth for 64KB message is 959MB/s on shared-memory and is 694MB/s
on IB;  the bandwidth for 4MB message is 539 MB/s and 1092 MB/s on  IB.
The bandwidth of 4MB message on shared memory may be higher if you tune
some MCA parameter.

2.  mpi_paffinity_alone
  "mpi_paffinity_alone =1"  is not a good choice for QS22.  There are two
sockets with two physical  Cell/B.E. on one QS22.  Each Cell/B.E. has two
SMT threads.   So there are four logical CPUs on one QS22.  CBE Linux
kernel maps logical cpu 0 and 1 to socket1 and maps logical cpu 1 and 2 to
socket 2.If mpi_paffinity_alone is set to 1,   the two MPI instances
will be assigned to logical cpu 0 and cpu 1 on socket 1.  I believe this is
not what you want.

A temporaily solution to  force the affinity on  QS22 is to use
"numactl",   E.g.  assuming the hostname is "qs22" and the executable is
"foo".  the following command can be used
mpirun -np 1 -H qs22 numactl -c0 -m0  foo :   -np 1 -H qs22
numactl -c1 -m1 foo

   In the long run,  I wish CBE kernel export  CPU topology  in /sys  and
use  PLPA to force the processor affinity.

Best Regards,
Mi



   
 "Lenny
 Verkhovsky"   
"Open MPI Users"
 Sent by:  
 users-bounces@ope  cc
 n-mpi.org 
   Subject
   Re: [OMPI users] Working with a 
 10/23/2008 05:48  CellBlade cluster   
 AM
   
   
 Please respond to 
  Open MPI Users   
 
   
   




Hi,


If I understand you correctly the most suitable way to do it is by
paffinity that we have in Open MPI 1.3 and the trank.
how ever usually OS is distributing processes evenly between sockets by it
self.

There still no formal FAQ due to a multiple reasons but you can read how to
use it in the attached scratch ( there were few name changings of the
params, so check with ompi_info )

shared memory is used between processes that share same machine, and openib
is used between different machines ( hostnames ), no special mca params are
needed.

Best Regards
Lenny,







On Sun, Oct 19, 2008 at 10:32 AM, Gilbert Grosdidier 
wrote:
   Working with a CellBlade cluster (QS22), the requirement is to have one
  instance of the executable running on each socket of the blade (there are
  2
  sockets). The application is of the 'domain decomposition' type, and each
  instance is required to often send/receive data with both the remote
  blades and
  the neighbor socket.

   Question is : which specification must be used for the mca btl component
  to force 1) shmem type messages when communicating with this neighbor
  socket,
  while 2) using openib to communicate with the remote blades ?
  Is '-mca btl sm,openib,self' suitable for this ?

   Also, which debug flags could be used to crosscheck that the messages
  are
  _actually_ going thru the right channel for a given channel, please ?

   We are currently using OpenMPI 1.2.5 shipped with RHEL5.2 (ppc64).
  Which version do you think is currently the most optimised for these
  processors and problem type ? Should we go towards OpenMPI 1.2.8
  instead ?
  Or even try some OpenMPI 1.3 nightly build ?

   Thanks in advance for your help,  Gilbert.

  ___
  users mailing list
  us...@open-mpi.org
  http://www.open-mpi.org/mailman/listinfo.cgi/users
(See attached file: RANKS_FAQ.doc)
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

[OMPI users] MPI_SUM and MPI_REAL16 with MPI_ALLREDUCE in fortran90

[Julien Devriendt]

Hi,

I'm trying to do an MPI_ALLREDUCE with quadruple precision real and 
MPI_SUM and open mpi does not give me the correct answer (vartemp
is equal to vartored instead of 2*vartored). Switching to double precision 
real works fine.

My version of openmpi is 1.2.7 and it has been compiled with ifort v10.1
and icc/icpc at installation

Here's the simple f90 code which fails:

program test_quad

  implicit none

  include "mpif.h"


  real(kind=16) :: vartored(8),vartemp(8)
  integer   :: nn,nslaves,my_index
  integer   :: mpierror


  call MPI_INIT(mpierror)
  call MPI_COMM_SIZE(MPI_COMM_WORLD,nslaves,mpierror)
  call MPI_COMM_RANK(MPI_COMM_WORLD,my_index,mpierror)

  nn   = 8
  vartored = 1.0_16
  vartemp  = 0.0_16
  print*,"P1 ",my_index,vartored
  call 
MPI_ALLREDUCE(vartored,vartemp,nn,MPI_REAL16,MPI_SUM,MPI_COMM_WORLD,mpierror)
  print*,"P2 ",my_index,vartemp

  stop

end program test_quad

Any idea why this happens?

Many thanks in advance!

J.

Re: [OMPI users] Working with a CellBlade cluster

[Lenny Verkhovsky]

Hi,


If I understand you correctly the most suitable way to do it is by paffinity
that we have in Open MPI 1.3 and the trank.
how ever usually OS is distributing processes evenly between sockets by it
self.

There still no formal FAQ due to a multiple reasons but you can read how to
use it in the attached scratch ( there were few name changings of the
params, so check with ompi_info )

shared memory is used between processes that share same machine, and openib
is used between different machines ( hostnames ), no special mca params are
needed.

Best Regards
Lenny,




On Sun, Oct 19, 2008 at 10:32 AM, Gilbert Grosdidier wrote:

>  Working with a CellBlade cluster (QS22), the requirement is to have one
> instance of the executable running on each socket of the blade (there are 2
> sockets). The application is of the 'domain decomposition' type, and each
> instance is required to often send/receive data with both the remote blades
> and
> the neighbor socket.
>
>  Question is : which specification must be used for the mca btl component
> to force 1) shmem type messages when communicating with this neighbor
> socket,
> while 2) using openib to communicate with the remote blades ?
> Is '-mca btl sm,openib,self' suitable for this ?
>
>  Also, which debug flags could be used to crosscheck that the messages are
> _actually_ going thru the right channel for a given channel, please ?
>
>  We are currently using OpenMPI 1.2.5 shipped with RHEL5.2 (ppc64).
> Which version do you think is currently the most optimised for these
> processors and problem type ? Should we go towards OpenMPI 1.2.8 instead ?
> Or even try some OpenMPI 1.3 nightly build ?
>
>  Thanks in advance for your help,  Gilbert.
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>


RANKS_FAQ.doc
Description: MS-Word document