Re: [OMPI users] Hybrid MPI/Pthreads program behaves differently on two different machines with same hardware

[Ralph Castain]

Okay - thanks for testing it.

Of course, one obvious difference is that there isn't any communication when 
you run only one proc, but there is when you run two or more, assuming your 
application has MPI send/recv (or calls collective and other functions that 
communicate) calls in it. Communication to yourself is very fast as no bits 
actually move - sending messages to another proc is considerably slower.

Are you taking that into account?


On Oct 24, 2011, at 8:47 PM, 吕慧伟 wrote:

> No. There's a difference between "mpirun -np 1 ./my_hybrid_app..." and 
> "mpirun -np 2 ./...". 
> 
> Run "mpirun -np 1 ./my_hybrid_app..." will increase the performance with more 
> number of threads, but run "mpirun -np 2 ./..." decrease the performance.
> 
> --
> Huiwei Lv
> 
> On Tue, Oct 25, 2011 at 12:00 AM,  wrote:
> 
> Date: Mon, 24 Oct 2011 07:14:21 -0600
> From: Ralph Castain 
> Subject: Re: [OMPI users] Hybrid MPI/Pthreads program behaves
>differently on  two different machines with same hardware
> To: Open MPI Users 
> Message-ID: <42c53d0b-1586-4001-b9d2-d77af0033...@open-mpi.org>
> Content-Type: text/plain; charset="utf-8"
> 
> Does the difference persist if you run the single process using mpirun? In 
> other words, does "mpirun -np 1 ./my_hybrid_app..." behave the same as 
> "mpirun -np 2 ./..."?
> 
> There is a slight difference in the way procs start when run as singletons. 
> It shouldn't make a difference here, but worth testing.
> 
> On Oct 24, 2011, at 12:37 AM, ??? wrote:
> 
> > Dear List,
> >
> > I have a hybrid MPI/Pthreads program named "my_hybrid_app", this program is 
> > memory-intensive and take advantage of multi-threading to improve memory 
> > throughput. I run "my_hybrid_app" on two machines, which have same hardware 
> > configuration but different OS and GCC. The problem is: when I run 
> > "my_hybrid_app" with one process, two machines behaves the same: the more 
> > number of threads, the better the performance; however, when I run 
> > "my_hybrid_app" with two or more processes. The first machine still 
> > increase performance with more threads, the second machine degrades in 
> > performance with more threads.
> >
> > Since running "my_hybrid_app" with one process behaves correctly, I suspect 
> > my linking to MPI library has some problem. Would somebody point me in the 
> > right direction? Thanks in advance.
> >
> > Attached are the commandline used, my machine informantion and link 
> > informantion.
> > p.s. 1: Commandline
> > single process: ./my_hybrid_app 
> > multiple process: mpirun -np 2 ./my_hybrid_app 
> >
> > p.s. 2: Machine Informantion
> > The first machine is CentOS 5.3 with GCC 4.1.2:
> > Target: x86_64-redhat-linux
> > Configured with: ../configure --prefix=/usr --mandir=/usr/share/man 
> > --infodir=/usr/share/info --enable-shared --enable-threads=posix 
> > --enable-checking=release --with-system-zlib --enable-__cxa_atexit 
> > --disable-libunwind-exceptions --enable-libgcj-multifile 
> > --enable-languages=c,c++,objc,obj-c++,java,fortran,ada 
> > --enable-java-awt=gtk --disable-dssi --enable-plugin 
> > --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic 
> > --host=x86_64-redhat-linux
> > Thread model: posix
> > gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)
> > The second machine is SUSE Enterprise Server 11 with GCC 4.3.4:
> > Target: x86_64-suse-linux
> > Configured with: ../configure --prefix=/usr --infodir=/usr/share/info 
> > --mandir=/usr/share/man --libdir=/usr/lib64 --libexecdir=/usr/lib64 
> > --enable-languages=c,c++,objc,fortran,obj-c++,java,ada 
> > --enable-checking=release --with-gxx-include-dir=/usr/include/c++/4.3 
> > --enable-ssp --disable-libssp --with-bugurl=http://bugs.opensuse.org/ 
> > --with-pkgversion='SUSE Linux' --disable-libgcj --disable-libmudflap 
> > --with-slibdir=/lib64 --with-system-zlib --enable-__cxa_atexit 
> > --enable-libstdcxx-allocator=new --disable-libstdcxx-pch 
> > --enable-version-specific-runtime-libs --program-suffix=-4.3 
> > --enable-linux-futex --without-system-libunwind --with-cpu=generic 
> > --build=x86_64-suse-linux
> > Thread model: posix
> > gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux)
> >
> > p.s. 3: ldd Informantion
> > The first machine:
> > $ ldd my_hybrid_app
> > libm.so.6 => /lib64/libm.so.6 (0x00358d40)
> > libmpi.so.0 => /usr/local/openmpi/lib/libmpi.so.0 
> > (0x2af0d53a7000)
> > libopen-rte.so.0 => /usr/local/openmpi/lib/libopen-rte.so.0 
> > (0x2af0d564a000)
> > libopen-pal.so.0 => /usr/local/openmpi/lib/libopen-pal.so.0 
> > (0x2af0d5895000)
> > libdl.so.2 => /lib64/libdl.so.2 (0x00358d00)
> > libnsl.so.1 => /lib64/libnsl.so.1 (0x00358f00)
> > libutil.so.1 => /lib64/libutil.so.1 (0x00359a60)
> > libgomp.so.1 => /usr/lib64/libgomp.so.1 (0x2af0d5b07000)
> > libpthread.so.0 => /lib64/libpthread.so.0 (0x00358d80)
> > 

Re: [OMPI users] Hybrid MPI/Pthreads program behaves differently on two different machines with same hardware

[吕慧伟]

Thanks, Ralph. Yes, I have taking that into account. The problem is not to
compare two proc with one proc, but the "multi-threading effect".
Multi-threading is good on the first machine for one and two proc, but on
the second machine, it disappears for two proc.

To narrow down the problem, I reinstalled the operating system on the second
machine from SUSE 11(kernel 2.6.32.12, gcc 4.3.4) to Red Hat 5.4
(kernel 2.6.18, gcc 4.1.2) which is similar to the first machine (Cent OS
5.3, kernel 2.6.18, gcc 4.1.2). Then the problem disappears. So the problem
must lies somewhere in OS kernel or GCC version. Any suggestions? Thanks.

--
Huiwei Lv

On Tue, Oct 25, 2011 at 3:11 PM, Ralph Castain  wrote:

> Okay - thanks for testing it.
>
> Of course, one obvious difference is that there isn't any communication
> when you run only one proc, but there is when you run two or more, assuming
> your application has MPI send/recv (or calls collective and other functions
> that communicate) calls in it. Communication to yourself is very fast as no
> bits actually move - sending messages to another proc is considerably
> slower.
>
> Are you taking that into account?
>
>
> On Oct 24, 2011, at 8:47 PM, 吕慧伟 wrote:
>
> No. There's a difference between "mpirun -np 1 ./my_hybrid_app..."
> and "mpirun -np 2 ./...".
>
> Run "mpirun -np 1 ./my_hybrid_app..." will increase the performance with
> more number of threads, but run "mpirun -np 2 ./..." decrease the
> performance.
>
> --
> Huiwei Lv
>
> On Tue, Oct 25, 2011 at 12:00 AM,  wrote:
>
>>
>> Date: Mon, 24 Oct 2011 07:14:21 -0600
>> From: Ralph Castain 
>> Subject: Re: [OMPI users] Hybrid MPI/Pthreads program behaves
>>differently on  two different machines with same hardware
>> To: Open MPI Users 
>> Message-ID: <42c53d0b-1586-4001-b9d2-d77af0033...@open-mpi.org>
>> Content-Type: text/plain; charset="utf-8"
>>
>> Does the difference persist if you run the single process using mpirun? In
>> other words, does "mpirun -np 1 ./my_hybrid_app..." behave the same as
>> "mpirun -np 2 ./..."?
>>
>> There is a slight difference in the way procs start when run as
>> singletons. It shouldn't make a difference here, but worth testing.
>>
>> On Oct 24, 2011, at 12:37 AM, ??? wrote:
>>
>> > Dear List,
>> >
>> > I have a hybrid MPI/Pthreads program named "my_hybrid_app", this program
>> is memory-intensive and take advantage of multi-threading to improve memory
>> throughput. I run "my_hybrid_app" on two machines, which have same hardware
>> configuration but different OS and GCC. The problem is: when I run
>> "my_hybrid_app" with one process, two machines behaves the same: the more
>> number of threads, the better the performance; however, when I run
>> "my_hybrid_app" with two or more processes. The first machine still increase
>> performance with more threads, the second machine degrades in performance
>> with more threads.
>> >
>> > Since running "my_hybrid_app" with one process behaves correctly, I
>> suspect my linking to MPI library has some problem. Would somebody point me
>> in the right direction? Thanks in advance.
>> >
>> > Attached are the commandline used, my machine informantion and link
>> informantion.
>> > p.s. 1: Commandline
>> > single process: ./my_hybrid_app 
>> > multiple process: mpirun -np 2 ./my_hybrid_app 
>> >
>> > p.s. 2: Machine Informantion
>> > The first machine is CentOS 5.3 with GCC 4.1.2:
>> > Target: x86_64-redhat-linux
>> > Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
>> --infodir=/usr/share/info --enable-shared --enable-threads=posix
>> --enable-checking=release --with-system-zlib --enable-__cxa_atexit
>> --disable-libunwind-exceptions --enable-libgcj-multifile
>> --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk
>> --disable-dssi --enable-plugin
>> --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic
>> --host=x86_64-redhat-linux
>> > Thread model: posix
>> > gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)
>> > The second machine is SUSE Enterprise Server 11 with GCC 4.3.4:
>> > Target: x86_64-suse-linux
>> > Configured with: ../configure --prefix=/usr --infodir=/usr/share/info
>> --mandir=/usr/share/man --libdir=/usr/lib64 --libexecdir=/usr/lib64
>> --enable-languages=c,c++,objc,fortran,obj-c++,java,ada
>> --enable-checking=release --with-gxx-include-dir=/usr/include/c++/4.3
>> --enable-ssp --disable-libssp 
>> --with-bugurl=http://bugs.opensuse.org/--with-pkgversion='SUSE Linux' 
>> --disable-libgcj --disable-libmudflap
>> --with-slibdir=/lib64 --with-system-zlib --enable-__cxa_atexit
>> --enable-libstdcxx-allocator=new --disable-libstdcxx-pch
>> --enable-version-specific-runtime-libs --program-suffix=-4.3
>> --enable-linux-futex --without-system-libunwind --with-cpu=generic
>> --build=x86_64-suse-linux
>> > Thread model: posix
>> > gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux)
>> >
>> > p.s. 3: ldd Informantion
>> > The first machine:
>> > $ ldd my_hybrid_app
>>

Re: [OMPI users] Visual debugging on the cluster

[devendra rai]

Hello Meredith,

Yes, I have tried the plugin already. The problem is that the plugin seems to 
be forever stuck in "Waiting for job information" stage. I scouted around a bit 
on how to solve the problem, and it did not seem straightforward. At least, the 
solution to me seemed like a one-time wonder.

And, this is how I shifted to parallel visual debuggers, using other tools like 
kdbg.

However, in case you have PTP plugin working for you on Linux, it would help a 
lot if you can send screenshots/notes on how to set it up for multiple machines.

So, summing up, I am still clueless.

Thanks for your time though.

Best

Devendra



From: Meredith Creekmore 
To: devendra rai ; Open MPI Users 
Sent: Monday, 24 October 2011, 22:31
Subject: RE: [OMPI users] Visual debugging on the cluster


 
Not a direct answer to your question, but have you tried using Eclipse with the 
Parallel Platform Tools installed?
 
http://eclipse.org/ptp/
 
From:users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On Behalf 
Of devendra rai
Sent: Monday, October 24, 2011 2:50 PM
To: us...@open-mpi.org
Subject: [OMPI users] Visual debugging on the cluster
 
Hello Community,


I have been struggling with visual debugging on cluster machines. So far, I 
tried to work around the problem, or total avoid it, but no more.


I have three machines on the cluster: a.s1.s2, b.s1.s2 and c.s1.s2. I do not 
have admin privileges on any of these machines.


Now, I want to run a visual debugger on all of these machines, and have the 
windows come up. 




So for from: (http://www.open-mpi.org/faq/? category=running)


13. Can I run GUI applications with Open MPI? 
Yes, but it will depend on your local setup and may require additional setup. 
In short: you will need to have X forwarding enabled from the remote processes 
to the display where you want output to appear. In a secure environment, you 
can simply allow all X requests to be shown on the target display and set the 
DISPLAYenvironment variable in all MPI process' environments to the target 
display, perhaps something like this: 
shell$ hostname
my_desktop.secure-cluster. example.com
shell$ xhost +
shell$ mpirun -np 4 -x DISPLAY=my_desktop.secure- cluster.example.com a.out 
However, this technique is not generally suitable for unsecure environments 
(because it allows anyone to read and write to your display). A slightly more 
secure way is to only allow X connections from the nodes where your application 
will be running: 
shell$ hostname
my_desktop.secure-cluster. example.com
shell$ xhost +compute1 +compute2 +compute3 +compute4
compute1 being added to access control list
compute2 being added to access control list
compute3 being added to access control list
compute4 being added to access control list
shell$ mpirun -np 4 -x DISPLAY=my_desktop.secure- cluster.example.com a.out 
(assuming that the four nodes you are running on are compute1through compute4). 
Other methods are available, but they involve sophisticated X forwarding 
through mpirun and are generally more complicated than desirable.

This still gives me "Error: Can't open display:" problem. 

My mpirun shell script contains:

mpirun-1.4.3 -hostfile hostfile -np 3 -v -nooversubscribe --rankfile 
rankfile.txt --report-bindings  -timestamp-output ./testdisplay-window.sh 


where rankfile and hostfile contain a.s1.s2, b.s1.s2 and c.s1.s2, and are 
proper.

The file ./testdisplay-window.sh:

#!/bin/bash
echo "Running xeyes on `hostname`"
DISPLAY=a.s1.s2:11.0
xeyes
exit 0

I see that my xauth list output already contains entries like:

a.s1.s2/unix:12  MIT-MAGIC-COOKIE-1  aa16a9573f42224d760c7bb618b48a6f
a.s1.s2/unix:10  MIT-MAGIC-COOKIE-1  0fb6fe3c2e35676136c8642412fb5809
a.s1.s2/unix:11  MIT-MAGIC-COOKIE-1  a3a65970b5f545bc750e3520a4e3b872


I seem to have run out of ideas now.

However, this works prefectly on any of the machines a.s1.s2, b.s1.s2or c.s1.s2:

(for example, running from a.s1.s2):

ssh b.s1.s2 xeyes

Can someone help?


Best

Devendra Rai




 


 
From:Jeff Squyres 
To: devendra rai ; Open MPI Users 
Sent: Friday, 21 October 2011, 13:14
Subject: Re: [OMPI users] orte_grpcomm_modex failed

This usually means that you have a Open MPI version mismatch between some of 
your nodes.  Meaning: on some nodes, you're finding version X.Y.Z of Open MPI 
by default, but on other nodes, you're finding version A.B.C.


On Oct 21, 2011, at 7:00 AM, devendra rai wrote:

> Hello Community,
> 
> I have been struggling with this error for quite some time:
> 
> It looks like MPI_INIT failed for some reason; your parallel process is
> likely to abort.  There are many reasons that a parallel process can
> fail during MPI_INIT; some of which are due to configuration or environment
> problems.  This failure appears to be an internal failure; here's some
> additional information (which may only be relevant to an Open MPI
> developer):
> 
>  orte_grpcomm_modex failed
>  

Re: [OMPI users] Memory mapped memory

[Mouhamad Al-Sayed-Ali]

Hello,

I have tried to run the executable "wrf.exe", using

  mpirun -machinefile /tmp/108388.1.par2/machines -np 4 wrf.exe

but, I've got the following error:

--
mpirun noticed that process rank 1 with PID 9942 on node  
part031.u-bourgogne.fr exited on signal 11 (Segmentation fault).

--
   11.54s real 6.03s user 0.32s system
Starter(9908): Return code=139
Starter end(9908)




Thanks for your help


Mouhamad Alsayed

[OMPI users] exited on signal 11 (Segmentation fault).

[Mouhamad Al-Sayed-Ali]

Hello,

I have tried to run the executable "wrf.exe", using

  mpirun -machinefile /tmp/108388.1.par2/machines -np 4 wrf.exe

but, I've got the following error:

--
mpirun noticed that process rank 1 with PID 9942 on node  
part031.u-bourgogne.fr exited on signal 11 (Segmentation fault).

--
   11.54s real 6.03s user 0.32s system
Starter(9908): Return code=139
Starter end(9908)




Thanks for your help


Mouhamad Alsayed

Re: [OMPI users] exited on signal 11 (Segmentation fault).

[TERRY DONTJE]

Some more info would be nice like:
-What version of ompi are you using
-What type of machine and os are you running on
-What does the machine file look like
-Is there a stack trace left behind by the pid that seg faulted?

--td

On 10/25/2011 8:07 AM, Mouhamad Al-Sayed-Ali wrote:

Hello,

I have tried to run the executable "wrf.exe", using

  mpirun -machinefile /tmp/108388.1.par2/machines -np 4 wrf.exe

but, I've got the following error:

-- 

mpirun noticed that process rank 1 with PID 9942 on node 
part031.u-bourgogne.fr exited on signal 11 (Segmentation fault).
-- 


   11.54s real 6.03s user 0.32s system
Starter(9908): Return code=139
Starter end(9908)




Thanks for your help


Mouhamad Alsayed


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


--
Oracle
Terry D. Dontje | Principal Software Engineer
Developer Tools Engineering | +1.781.442.2631
Oracle *- Performance Technologies*
95 Network Drive, Burlington, MA 01803
Email terry.don...@oracle.com 

[OMPI users] Problem-Bug with MPI_Intercomm_create()

[orel]

Dears,

I try from several days to use advanced MPI2 features in the following 
scenario :


  1) a master code A (of size NPA) spawns (MPI_Comm_spawn()) two slave
 codes B (of size NPB) and C (of size NPC), providing intercomms 
A-B and A-C ;

  2) i create intracomm AB and AC by merging intercomms ;
  3) then i create intercomm AB-C by calling MPI_Intercomm_create() by 
using AC as bridge...


MPI_Comm intercommABC; A: MPI_Intercomm_create(intracommAB, 0, 
intracommAC, NPA, TAG,&intercommABC);
B: MPI_Intercomm_create(intracommAB, 0, MPI_COMM_NULL, 
0,TAG,&intercommABC);

C: MPI_Intercomm_create(intracommC, 0, intracommAC, 0, TAG,&intercommABC);

  In these calls, A0 and C0 play the role of local leader for AB 
and C respectively.

  C0 and A0 play the roles of remote leader in bridge intracomm AC.

  3)  MPI_Barrier(intercommABC);
  4)  i merge intercomm AB-C into intracomm ABC$
  5)  MPI_Barrier(intracommABC);

My BUG: These calls success, but when i try to use intracommABC for a 
collective communication like MPI_Barrier(),

   i got the following error :

*** An error occurred in MPI_Barrier
*** on communicator
*** MPI_ERR_INTERN: internal error
*** MPI_ERRORS_ARE_FATAL: your MPI job will now abort


I try with OpenMPI trunk, 1.5.3, 1.5.4 and Mpich2-1.4.1p1

My code works perfectly if intracomm A, B and C are obtained by 
MPI_Comm_split() instead of MPI_Comm_spawn() 



I found same problem in a previous thread of the OMPI Users mailing list :

  => http://www.open-mpi.org/community/lists/users/2011/06/16711.php

Is that bug/problem is currently under investigation ? :-)

i can give detailed code, but the one provided by George Bosilca in this 
previous thread provides same error...


Thank you to help me...

--
Aurélien Esnard
University Bordeaux 1 / LaBRI / INRIA (France)

Re: [OMPI users] exited on signal 11 (Segmentation fault).

[Mouhamad Al-Sayed-Ali]

hello,


-What version of ompi are you using

  I am using ompi version 1.4.1-1 compiled with gcc 4.5


-What type of machine and os are you running on

   I'm using linux machine 64 bits.


-What does the machine file look like

  part033
  part033
  part031
  part031


-Is there a stack trace left behind by the pid that seg faulted?

  No, there is no stack trace


Thanks for your help

Mouhamad Alsayed

Re: [OMPI users] exited on signal 11 (Segmentation fault).

[TERRY DONTJE]

Can you run wrf successfully on one node?
Can you run a simple code across your two nodes?  I would try hostname 
then some simple MPI program like the ring example.


--td

On 10/25/2011 9:05 AM, Mouhamad Al-Sayed-Ali wrote:

hello,


-What version of ompi are you using

  I am using ompi version 1.4.1-1 compiled with gcc 4.5


-What type of machine and os are you running on

   I'm using linux machine 64 bits.


-What does the machine file look like

  part033
  part033
  part031
  part031


-Is there a stack trace left behind by the pid that seg faulted?

  No, there is no stack trace


Thanks for your help

Mouhamad Alsayed


--
Oracle
Terry D. Dontje | Principal Software Engineer
Developer Tools Engineering | +1.781.442.2631
Oracle *- Performance Technologies*
95 Network Drive, Burlington, MA 01803
Email terry.don...@oracle.com 

Re: [OMPI users] exited on signal 11 (Segmentation fault).

[Mouhamad Al-Sayed-Ali]

Hello


can you run wrf successfully on one node?


NO, It can't run on one node

Can you run a simple code across your two nodes?  I would try  
hostname then some simple MPI program like the ring example.

Yes, I can run a simple code

many thanks

Mouhamad

Re: [OMPI users] exited on signal 11 (Segmentation fault).

[Mouhamad Al-Sayed-Ali]

Hi again,

 This is exactly the error I have:


taskid: 0 hostname: part034.u-bourgogne.fr
[part034:21443] *** Process received signal ***
[part034:21443] Signal: Segmentation fault (11)
[part034:21443] Signal code: Address not mapped (1)
[part034:21443] Failing at address: 0xfffe01eeb340
[part034:21443] [ 0] /lib64/libpthread.so.0 [0x3612c0de70]
[part034:21443] [ 1] wrf.exe(__module_ra_rrtm_MOD_taugb3+0x418) [0x11cc9d8]
[part034:21443] [ 2] wrf.exe(__module_ra_rrtm_MOD_gasabs+0x260) [0x11cfca0]
[part034:21443] [ 3] wrf.exe(__module_ra_rrtm_MOD_rrtm+0xb31) [0x11e6e41]
[part034:21443] [ 4] wrf.exe(__module_ra_rrtm_MOD_rrtmlwrad+0x25ec)  
[0x11e9bcc]
[part034:21443] [ 5]  
wrf.exe(__module_radiation_driver_MOD_radiation_driver+0xe573)  
[0xcc4ed3]
[part034:21443] [ 6]  
wrf.exe(__module_first_rk_step_part1_MOD_first_rk_step_part1+0x40c5)  
[0xe0e4f5]

[part034:21443] [ 7] wrf.exe(solve_em_+0x22e58) [0x9b45c8]
[part034:21443] [ 8] wrf.exe(solve_interface_+0x80a) [0x902dda]
[part034:21443] [ 9] wrf.exe(__module_integrate_MOD_integrate+0x236)  
[0x4b2c4a]

[part034:21443] [10] wrf.exe(__module_wrf_top_MOD_wrf_run+0x24) [0x47a924]
[part034:21443] [11] wrf.exe(main+0x41) [0x4794d1]
[part034:21443] [12] /lib64/libc.so.6(__libc_start_main+0xf4) [0x361201d8b4]
[part034:21443] [13] wrf.exe [0x4793c9]
[part034:21443] *** End of error message ***
---

Mouhamad

Re: [OMPI users] Problem-Bug with MPI_Intercomm_create()

[Ralph Castain]

I still see it failing the test George provided on the trunk. I'm unaware of 
anyone looking further into it, though, as the prior discussion seemed to just 
end.

On Oct 25, 2011, at 7:01 AM, orel wrote:

> Dears,
> 
> I try from several days to use advanced MPI2 features in the following 
> scenario :
> 
>  1) a master code A (of size NPA) spawns (MPI_Comm_spawn()) two slave
> codes B (of size NPB) and C (of size NPC), providing intercomms A-B and 
> A-C ;
>  2) i create intracomm AB and AC by merging intercomms ;
>  3) then i create intercomm AB-C by calling MPI_Intercomm_create() by using 
> AC as bridge...
> 
>MPI_Comm intercommABC; A: MPI_Intercomm_create(intracommAB, 0, 
> intracommAC, NPA, TAG,&intercommABC);
> B: MPI_Intercomm_create(intracommAB, 0, MPI_COMM_NULL, 0,TAG,&intercommABC);
> C: MPI_Intercomm_create(intracommC, 0, intracommAC, 0, TAG,&intercommABC);
> 
>  In these calls, A0 and C0 play the role of local leader for AB and C 
> respectively.
>  C0 and A0 play the roles of remote leader in bridge intracomm AC.
> 
>  3)  MPI_Barrier(intercommABC);
>  4)  i merge intercomm AB-C into intracomm ABC$
>  5)  MPI_Barrier(intracommABC);
> 
> My BUG: These calls success, but when i try to use intracommABC for a 
> collective communication like MPI_Barrier(),
>   i got the following error :
> 
> *** An error occurred in MPI_Barrier
> *** on communicator
> *** MPI_ERR_INTERN: internal error
> *** MPI_ERRORS_ARE_FATAL: your MPI job will now abort
> 
> 
> I try with OpenMPI trunk, 1.5.3, 1.5.4 and Mpich2-1.4.1p1
> 
> My code works perfectly if intracomm A, B and C are obtained by 
> MPI_Comm_split() instead of MPI_Comm_spawn() 
> 
> 
> I found same problem in a previous thread of the OMPI Users mailing list :
> 
>  => http://www.open-mpi.org/community/lists/users/2011/06/16711.php
> 
> Is that bug/problem is currently under investigation ? :-)
> 
> i can give detailed code, but the one provided by George Bosilca in this 
> previous thread provides same error...
> 
> Thank you to help me...
> 
> -- 
> Aurélien Esnard
> University Bordeaux 1 / LaBRI / INRIA (France)
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] exited on signal 11 (Segmentation fault).

[TERRY DONTJE]

This looks more like a seg fault in wrf and not OMPI.

Sorry not much I can do here to help you.

--td

On 10/25/2011 9:53 AM, Mouhamad Al-Sayed-Ali wrote:

Hi again,

 This is exactly the error I have:


taskid: 0 hostname: part034.u-bourgogne.fr
[part034:21443] *** Process received signal ***
[part034:21443] Signal: Segmentation fault (11)
[part034:21443] Signal code: Address not mapped (1)
[part034:21443] Failing at address: 0xfffe01eeb340
[part034:21443] [ 0] /lib64/libpthread.so.0 [0x3612c0de70]
[part034:21443] [ 1] wrf.exe(__module_ra_rrtm_MOD_taugb3+0x418) 
[0x11cc9d8]
[part034:21443] [ 2] wrf.exe(__module_ra_rrtm_MOD_gasabs+0x260) 
[0x11cfca0]

[part034:21443] [ 3] wrf.exe(__module_ra_rrtm_MOD_rrtm+0xb31) [0x11e6e41]
[part034:21443] [ 4] wrf.exe(__module_ra_rrtm_MOD_rrtmlwrad+0x25ec) 
[0x11e9bcc]
[part034:21443] [ 5] 
wrf.exe(__module_radiation_driver_MOD_radiation_driver+0xe573) [0xcc4ed3]
[part034:21443] [ 6] 
wrf.exe(__module_first_rk_step_part1_MOD_first_rk_step_part1+0x40c5) 
[0xe0e4f5]

[part034:21443] [ 7] wrf.exe(solve_em_+0x22e58) [0x9b45c8]
[part034:21443] [ 8] wrf.exe(solve_interface_+0x80a) [0x902dda]
[part034:21443] [ 9] wrf.exe(__module_integrate_MOD_integrate+0x236) 
[0x4b2c4a]
[part034:21443] [10] wrf.exe(__module_wrf_top_MOD_wrf_run+0x24) 
[0x47a924]

[part034:21443] [11] wrf.exe(main+0x41) [0x4794d1]
[part034:21443] [12] /lib64/libc.so.6(__libc_start_main+0xf4) 
[0x361201d8b4]

[part034:21443] [13] wrf.exe [0x4793c9]
[part034:21443] *** End of error message ***
---

Mouhamad


--
Oracle
Terry D. Dontje | Principal Software Engineer
Developer Tools Engineering | +1.781.442.2631
Oracle *- Performance Technologies*
95 Network Drive, Burlington, MA 01803
Email terry.don...@oracle.com 

Re: [OMPI users] Hybrid MPI/Pthreads program behaves differently on two different machines with same hardware

[Ralph Castain]

My best guess is that you are seeing differences in scheduling behavior with 
respect to memory locale. I notice that you are not binding your processes, and 
so they are free to move around the various processors on the node. I would 
guess that your thread is winding up on a processor that is non-local to your 
memory in one case, but local to your memory in the other. This is an 
OS-related scheduler decision.

You might try binding your processes to see if it helps. With threads, you 
don't really want to bind to a core, but binding to a socket should help. Try 
adding --bind-to-socket to your mpirun cmd line (you can't do this if you run 
it as a singleton - have to use mpirun).


On Oct 25, 2011, at 2:45 AM, 吕慧伟 wrote:

> Thanks, Ralph. Yes, I have taking that into account. The problem is not to 
> compare two proc with one proc, but the "multi-threading effect". 
> Multi-threading is good on the first machine for one and two proc, but on the 
> second machine, it disappears for two proc.
> 
> To narrow down the problem, I reinstalled the operating system on the second 
> machine from SUSE 11(kernel 2.6.32.12, gcc 4.3.4) to Red Hat 5.4 (kernel 
> 2.6.18, gcc 4.1.2) which is similar to the first machine (Cent OS 5.3, kernel 
> 2.6.18, gcc 4.1.2). Then the problem disappears. So the problem must lies 
> somewhere in OS kernel or GCC version. Any suggestions? Thanks.
> 
> --
> Huiwei Lv
> 
> On Tue, Oct 25, 2011 at 3:11 PM, Ralph Castain  wrote:
> Okay - thanks for testing it.
> 
> Of course, one obvious difference is that there isn't any communication when 
> you run only one proc, but there is when you run two or more, assuming your 
> application has MPI send/recv (or calls collective and other functions that 
> communicate) calls in it. Communication to yourself is very fast as no bits 
> actually move - sending messages to another proc is considerably slower.
> 
> Are you taking that into account?
> 
> 
> On Oct 24, 2011, at 8:47 PM, 吕慧伟 wrote:
> 
>> No. There's a difference between "mpirun -np 1 ./my_hybrid_app..." and 
>> "mpirun -np 2 ./...". 
>> 
>> Run "mpirun -np 1 ./my_hybrid_app..." will increase the performance with 
>> more number of threads, but run "mpirun -np 2 ./..." decrease the 
>> performance.
>> 
>> --
>> Huiwei Lv
>> 
>> On Tue, Oct 25, 2011 at 12:00 AM,  wrote:
>> 
>> Date: Mon, 24 Oct 2011 07:14:21 -0600
>> From: Ralph Castain 
>> Subject: Re: [OMPI users] Hybrid MPI/Pthreads program behaves
>>differently on  two different machines with same hardware
>> To: Open MPI Users 
>> Message-ID: <42c53d0b-1586-4001-b9d2-d77af0033...@open-mpi.org>
>> Content-Type: text/plain; charset="utf-8"
>> 
>> Does the difference persist if you run the single process using mpirun? In 
>> other words, does "mpirun -np 1 ./my_hybrid_app..." behave the same as 
>> "mpirun -np 2 ./..."?
>> 
>> There is a slight difference in the way procs start when run as singletons. 
>> It shouldn't make a difference here, but worth testing.
>> 
>> On Oct 24, 2011, at 12:37 AM, ??? wrote:
>> 
>> > Dear List,
>> >
>> > I have a hybrid MPI/Pthreads program named "my_hybrid_app", this program 
>> > is memory-intensive and take advantage of multi-threading to improve 
>> > memory throughput. I run "my_hybrid_app" on two machines, which have same 
>> > hardware configuration but different OS and GCC. The problem is: when I 
>> > run "my_hybrid_app" with one process, two machines behaves the same: the 
>> > more number of threads, the better the performance; however, when I run 
>> > "my_hybrid_app" with two or more processes. The first machine still 
>> > increase performance with more threads, the second machine degrades in 
>> > performance with more threads.
>> >
>> > Since running "my_hybrid_app" with one process behaves correctly, I 
>> > suspect my linking to MPI library has some problem. Would somebody point 
>> > me in the right direction? Thanks in advance.
>> >
>> > Attached are the commandline used, my machine informantion and link 
>> > informantion.
>> > p.s. 1: Commandline
>> > single process: ./my_hybrid_app 
>> > multiple process: mpirun -np 2 ./my_hybrid_app 
>> >
>> > p.s. 2: Machine Informantion
>> > The first machine is CentOS 5.3 with GCC 4.1.2:
>> > Target: x86_64-redhat-linux
>> > Configured with: ../configure --prefix=/usr --mandir=/usr/share/man 
>> > --infodir=/usr/share/info --enable-shared --enable-threads=posix 
>> > --enable-checking=release --with-system-zlib --enable-__cxa_atexit 
>> > --disable-libunwind-exceptions --enable-libgcj-multifile 
>> > --enable-languages=c,c++,objc,obj-c++,java,fortran,ada 
>> > --enable-java-awt=gtk --disable-dssi --enable-plugin 
>> > --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre 
>> > --with-cpu=generic --host=x86_64-redhat-linux
>> > Thread model: posix
>> > gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)
>> > The second machine is SUSE Enterprise Server 11 with GCC 4.3.4:
>> > Target: x86_64-suse-linux
>> > Configured with: ../configure --p

Re: [OMPI users] exited on signal 11 (Segmentation fault).

[Ralph Castain]

Looks like you are crashing in wrf - have you asked them for help?

On Oct 25, 2011, at 7:53 AM, Mouhamad Al-Sayed-Ali wrote:

> Hi again,
> 
> This is exactly the error I have:
> 
> 
> taskid: 0 hostname: part034.u-bourgogne.fr
> [part034:21443] *** Process received signal ***
> [part034:21443] Signal: Segmentation fault (11)
> [part034:21443] Signal code: Address not mapped (1)
> [part034:21443] Failing at address: 0xfffe01eeb340
> [part034:21443] [ 0] /lib64/libpthread.so.0 [0x3612c0de70]
> [part034:21443] [ 1] wrf.exe(__module_ra_rrtm_MOD_taugb3+0x418) [0x11cc9d8]
> [part034:21443] [ 2] wrf.exe(__module_ra_rrtm_MOD_gasabs+0x260) [0x11cfca0]
> [part034:21443] [ 3] wrf.exe(__module_ra_rrtm_MOD_rrtm+0xb31) [0x11e6e41]
> [part034:21443] [ 4] wrf.exe(__module_ra_rrtm_MOD_rrtmlwrad+0x25ec) 
> [0x11e9bcc]
> [part034:21443] [ 5] 
> wrf.exe(__module_radiation_driver_MOD_radiation_driver+0xe573) [0xcc4ed3]
> [part034:21443] [ 6] 
> wrf.exe(__module_first_rk_step_part1_MOD_first_rk_step_part1+0x40c5) 
> [0xe0e4f5]
> [part034:21443] [ 7] wrf.exe(solve_em_+0x22e58) [0x9b45c8]
> [part034:21443] [ 8] wrf.exe(solve_interface_+0x80a) [0x902dda]
> [part034:21443] [ 9] wrf.exe(__module_integrate_MOD_integrate+0x236) 
> [0x4b2c4a]
> [part034:21443] [10] wrf.exe(__module_wrf_top_MOD_wrf_run+0x24) [0x47a924]
> [part034:21443] [11] wrf.exe(main+0x41) [0x4794d1]
> [part034:21443] [12] /lib64/libc.so.6(__libc_start_main+0xf4) [0x361201d8b4]
> [part034:21443] [13] wrf.exe [0x4793c9]
> [part034:21443] *** End of error message ***
> ---
> 
> Mouhamad
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] exited on signal 11 (Segmentation fault).

[Gus Correa]

Hi Mouhamad, Ralph, Terry

Very often big programs like wrf crash with segfault because they
can't allocate memory on the stack, and assume the system doesn't
impose any limits for it.  This has nothing to do with MPI.

Mouhamad:  Check if your stack size is set to unlimited on all compute
nodes.  The easy way to get it done
is to change /etc/security/limits.conf,
where you or your system administrator could add these lines:

*   -   memlock -1
*   -   stack   -1
*   -   nofile  4096

My two cents,
Gus Correa

Ralph Castain wrote:

Looks like you are crashing in wrf - have you asked them for help?

On Oct 25, 2011, at 7:53 AM, Mouhamad Al-Sayed-Ali wrote:


Hi again,

This is exactly the error I have:


taskid: 0 hostname: part034.u-bourgogne.fr
[part034:21443] *** Process received signal ***
[part034:21443] Signal: Segmentation fault (11)
[part034:21443] Signal code: Address not mapped (1)
[part034:21443] Failing at address: 0xfffe01eeb340
[part034:21443] [ 0] /lib64/libpthread.so.0 [0x3612c0de70]
[part034:21443] [ 1] wrf.exe(__module_ra_rrtm_MOD_taugb3+0x418) [0x11cc9d8]
[part034:21443] [ 2] wrf.exe(__module_ra_rrtm_MOD_gasabs+0x260) [0x11cfca0]
[part034:21443] [ 3] wrf.exe(__module_ra_rrtm_MOD_rrtm+0xb31) [0x11e6e41]
[part034:21443] [ 4] wrf.exe(__module_ra_rrtm_MOD_rrtmlwrad+0x25ec) [0x11e9bcc]
[part034:21443] [ 5] 
wrf.exe(__module_radiation_driver_MOD_radiation_driver+0xe573) [0xcc4ed3]
[part034:21443] [ 6] 
wrf.exe(__module_first_rk_step_part1_MOD_first_rk_step_part1+0x40c5) [0xe0e4f5]
[part034:21443] [ 7] wrf.exe(solve_em_+0x22e58) [0x9b45c8]
[part034:21443] [ 8] wrf.exe(solve_interface_+0x80a) [0x902dda]
[part034:21443] [ 9] wrf.exe(__module_integrate_MOD_integrate+0x236) [0x4b2c4a]
[part034:21443] [10] wrf.exe(__module_wrf_top_MOD_wrf_run+0x24) [0x47a924]
[part034:21443] [11] wrf.exe(main+0x41) [0x4794d1]
[part034:21443] [12] /lib64/libc.so.6(__libc_start_main+0xf4) [0x361201d8b4]
[part034:21443] [13] wrf.exe [0x4793c9]
[part034:21443] *** End of error message ***
---

Mouhamad
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] exited on signal 11 (Segmentation fault).

[Mouhamad Al-Sayed-Ali]

Hi all,

   I've checked the "limits.conf", and it contains theses lines


# Jcb 29.06.2007 : pbs wrf (Siji)
#*  hardstack   100
#*  softstack   100

# Dr 14.02.2008 : pour voltaire mpi
*  hardmemlock unlimited
*  softmemlock unlimited



Many thanks for your help
Mouhamad

Gus Correa  a écrit :


Hi Mouhamad, Ralph, Terry

Very often big programs like wrf crash with segfault because they
can't allocate memory on the stack, and assume the system doesn't
impose any limits for it.  This has nothing to do with MPI.

Mouhamad:  Check if your stack size is set to unlimited on all compute
nodes.  The easy way to get it done
is to change /etc/security/limits.conf,
where you or your system administrator could add these lines:

*   -   memlock -1
*   -   stack   -1
*   -   nofile  4096

My two cents,
Gus Correa

Ralph Castain wrote:

Looks like you are crashing in wrf - have you asked them for help?

On Oct 25, 2011, at 7:53 AM, Mouhamad Al-Sayed-Ali wrote:


Hi again,

This is exactly the error I have:


taskid: 0 hostname: part034.u-bourgogne.fr
[part034:21443] *** Process received signal ***
[part034:21443] Signal: Segmentation fault (11)
[part034:21443] Signal code: Address not mapped (1)
[part034:21443] Failing at address: 0xfffe01eeb340
[part034:21443] [ 0] /lib64/libpthread.so.0 [0x3612c0de70]
[part034:21443] [ 1] wrf.exe(__module_ra_rrtm_MOD_taugb3+0x418) [0x11cc9d8]
[part034:21443] [ 2] wrf.exe(__module_ra_rrtm_MOD_gasabs+0x260) [0x11cfca0]
[part034:21443] [ 3] wrf.exe(__module_ra_rrtm_MOD_rrtm+0xb31) [0x11e6e41]
[part034:21443] [ 4]  
wrf.exe(__module_ra_rrtm_MOD_rrtmlwrad+0x25ec) [0x11e9bcc]
[part034:21443] [ 5]  
wrf.exe(__module_radiation_driver_MOD_radiation_driver+0xe573)  
[0xcc4ed3]
[part034:21443] [ 6]  
wrf.exe(__module_first_rk_step_part1_MOD_first_rk_step_part1+0x40c5)  
[0xe0e4f5]

[part034:21443] [ 7] wrf.exe(solve_em_+0x22e58) [0x9b45c8]
[part034:21443] [ 8] wrf.exe(solve_interface_+0x80a) [0x902dda]
[part034:21443] [ 9]  
wrf.exe(__module_integrate_MOD_integrate+0x236) [0x4b2c4a]

[part034:21443] [10] wrf.exe(__module_wrf_top_MOD_wrf_run+0x24) [0x47a924]
[part034:21443] [11] wrf.exe(main+0x41) [0x4794d1]
[part034:21443] [12] /lib64/libc.so.6(__libc_start_main+0xf4)  
[0x361201d8b4]

[part034:21443] [13] wrf.exe [0x4793c9]
[part034:21443] *** End of error message ***
---

Mouhamad
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] exited on signal 11 (Segmentation fault).

[Gus Correa]

Hi Mouhamad

The locked memory is set to unlimited, but the lines
about the stack are commented out.
Have you tried to add this line:

*   -   stack   -1

then run wrf again? [Note no "#" hash character]

Also, if you login to the compute nodes,
what is the output of 'limit' [csh,tcsh] or 'ulimit -a' [sh,bash]?
This should tell you what limits are actually set.

I hope this helps,
Gus Correa

Mouhamad Al-Sayed-Ali wrote:

Hi all,

   I've checked the "limits.conf", and it contains theses lines


# Jcb 29.06.2007 : pbs wrf (Siji)
#*  hardstack   100
#*  softstack   100

# Dr 14.02.2008 : pour voltaire mpi
*  hardmemlock unlimited
*  softmemlock unlimited



Many thanks for your help
Mouhamad

Gus Correa  a écrit :


Hi Mouhamad, Ralph, Terry

Very often big programs like wrf crash with segfault because they
can't allocate memory on the stack, and assume the system doesn't
impose any limits for it.  This has nothing to do with MPI.

Mouhamad:  Check if your stack size is set to unlimited on all compute
nodes.  The easy way to get it done
is to change /etc/security/limits.conf,
where you or your system administrator could add these lines:

*   -   memlock -1
*   -   stack   -1
*   -   nofile  4096

My two cents,
Gus Correa

Ralph Castain wrote:

Looks like you are crashing in wrf - have you asked them for help?

On Oct 25, 2011, at 7:53 AM, Mouhamad Al-Sayed-Ali wrote:


Hi again,

This is exactly the error I have:


taskid: 0 hostname: part034.u-bourgogne.fr
[part034:21443] *** Process received signal ***
[part034:21443] Signal: Segmentation fault (11)
[part034:21443] Signal code: Address not mapped (1)
[part034:21443] Failing at address: 0xfffe01eeb340
[part034:21443] [ 0] /lib64/libpthread.so.0 [0x3612c0de70]
[part034:21443] [ 1] wrf.exe(__module_ra_rrtm_MOD_taugb3+0x418) 
[0x11cc9d8]
[part034:21443] [ 2] wrf.exe(__module_ra_rrtm_MOD_gasabs+0x260) 
[0x11cfca0]
[part034:21443] [ 3] wrf.exe(__module_ra_rrtm_MOD_rrtm+0xb31) 
[0x11e6e41]
[part034:21443] [ 4] wrf.exe(__module_ra_rrtm_MOD_rrtmlwrad+0x25ec) 
[0x11e9bcc]
[part034:21443] [ 5] 
wrf.exe(__module_radiation_driver_MOD_radiation_driver+0xe573) 
[0xcc4ed3]
[part034:21443] [ 6] 
wrf.exe(__module_first_rk_step_part1_MOD_first_rk_step_part1+0x40c5) 
[0xe0e4f5]

[part034:21443] [ 7] wrf.exe(solve_em_+0x22e58) [0x9b45c8]
[part034:21443] [ 8] wrf.exe(solve_interface_+0x80a) [0x902dda]
[part034:21443] [ 9] wrf.exe(__module_integrate_MOD_integrate+0x236) 
[0x4b2c4a]
[part034:21443] [10] wrf.exe(__module_wrf_top_MOD_wrf_run+0x24) 
[0x47a924]

[part034:21443] [11] wrf.exe(main+0x41) [0x4794d1]
[part034:21443] [12] /lib64/libc.so.6(__libc_start_main+0xf4) 
[0x361201d8b4]

[part034:21443] [13] wrf.exe [0x4793c9]
[part034:21443] *** End of error message ***
---

Mouhamad
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Problem-Bug with MPI_Intercomm_create()

[Ralph Castain]

FWIW: I have tracked this problem down. The fix is a little more complicated 
then I'd like, so I'm going to have to ping some other folks to ensure we 
concur on the approach before doing something.

On Oct 25, 2011, at 8:20 AM, Ralph Castain wrote:

> I still see it failing the test George provided on the trunk. I'm unaware of 
> anyone looking further into it, though, as the prior discussion seemed to 
> just end.
> 
> On Oct 25, 2011, at 7:01 AM, orel wrote:
> 
>> Dears,
>> 
>> I try from several days to use advanced MPI2 features in the following 
>> scenario :
>> 
>> 1) a master code A (of size NPA) spawns (MPI_Comm_spawn()) two slave
>>codes B (of size NPB) and C (of size NPC), providing intercomms A-B and 
>> A-C ;
>> 2) i create intracomm AB and AC by merging intercomms ;
>> 3) then i create intercomm AB-C by calling MPI_Intercomm_create() by using 
>> AC as bridge...
>> 
>>   MPI_Comm intercommABC; A: MPI_Intercomm_create(intracommAB, 0, 
>> intracommAC, NPA, TAG,&intercommABC);
>> B: MPI_Intercomm_create(intracommAB, 0, MPI_COMM_NULL, 0,TAG,&intercommABC);
>> C: MPI_Intercomm_create(intracommC, 0, intracommAC, 0, TAG,&intercommABC);
>> 
>> In these calls, A0 and C0 play the role of local leader for AB and C 
>> respectively.
>> C0 and A0 play the roles of remote leader in bridge intracomm AC.
>> 
>> 3)  MPI_Barrier(intercommABC);
>> 4)  i merge intercomm AB-C into intracomm ABC$
>> 5)  MPI_Barrier(intracommABC);
>> 
>> My BUG: These calls success, but when i try to use intracommABC for a 
>> collective communication like MPI_Barrier(),
>>  i got the following error :
>> 
>> *** An error occurred in MPI_Barrier
>> *** on communicator
>> *** MPI_ERR_INTERN: internal error
>> *** MPI_ERRORS_ARE_FATAL: your MPI job will now abort
>> 
>> 
>> I try with OpenMPI trunk, 1.5.3, 1.5.4 and Mpich2-1.4.1p1
>> 
>> My code works perfectly if intracomm A, B and C are obtained by 
>> MPI_Comm_split() instead of MPI_Comm_spawn() 
>> 
>> 
>> I found same problem in a previous thread of the OMPI Users mailing list :
>> 
>> => http://www.open-mpi.org/community/lists/users/2011/06/16711.php
>> 
>> Is that bug/problem is currently under investigation ? :-)
>> 
>> i can give detailed code, but the one provided by George Bosilca in this 
>> previous thread provides same error...
>> 
>> Thank you to help me...
>> 
>> -- 
>> Aurélien Esnard
>> University Bordeaux 1 / LaBRI / INRIA (France)
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 

Re: [OMPI users] Checkpoint from inside MPI program with OpenMPI 1.4.2 ?

[Josh Hursey]

Open MPI (trunk/1.7 - not 1.4 or 1.5) provides an application level
interface to request a checkpoint of an application. This API is
defined on the following website:
  http://osl.iu.edu/research/ft/ompi-cr/api.php#api-cr_checkpoint

This will behave the same as if you requested the checkpoint of the
job from the command line.

-- Josh

On Mon, Oct 24, 2011 at 12:37 PM, Nguyen Toan  wrote:
> Dear all,
> I want to automatically checkpoint an MPI program with OpenMPI ( I'm
> currently using 1.4.2 version with BLCR 0.8.2),
> not by manually typing ompi-checkpoint command line from another terminal.
> So I would like to know if there is a way to call checkpoint function from
> inside an MPI program
> with OpenMPI or how to do that.
> Any ideas are very appreciated.
> Regards,
> Nguyen Toan
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>



-- 
Joshua Hursey
Postdoctoral Research Associate
Oak Ridge National Laboratory
http://users.nccs.gov/~jjhursey

[OMPI users] Subnet routing (1.2.x) not working in 1.4.3 anymore

[Mirco Wahab]

In the last few years, it has been very simple to
set up high-performance (GbE) multiple back-to-back
connections between three nodes (triangular topology)
or four nodes (tetrahedral topology).

The only things you had to do was
- use 3 (or 4) cheap compute nodes w/Linux and connect
  each of them via standard GbE router (onboard GbE NIC)
  to a file server,
- put 2 (trigonal topol.) or 3 (tetrahedral topol.)
  $25 PCIe-GbE-NICs into *each* node,
- connect the nodes with 3 (trigonal) or 4 (tetrahedral)
  short crossover Cat5e cables,
- configure the extra NICs into different subnets
  according to their "edge index", eg.
  for 3 nodes (node10, node11, node12)
node10
  onboard NIC: 192.168.0.10 on eth0 (to router/server)
  extra NIC: 10.0.1.10 on eth1 (edge 1 to 10.0.1.11)
  extra NIC: 10.0.2.10 on eth2 (edge 2 to 10.0.2.12)
node11
  onboard NIC: 192.168.0.11 on eth0 (to router/server)
  extra NIC: 10.0.1.11 on eth1 (edge 1 to 10.0.1.10)
  extra NIC: 10.0.3.11 on eth3 (edge 3 to 10.0.3.12)
node12
  onboard NIC: 192.168.0.12 on eth0 (to router/server)
  extra NIC: 10.0.2.12 on eth2 (edge 2 to 10.0.2.10)
  extra NIC: 10.0.3.12 on eth3 (edge 3 to 10.0.3.11)
- that's it. I mean, that *was* it, with 1.2.x.

OMPI 1.2.x would then ingeniously discover the routable edges
and open communication ports accordingly without any additional
explicit host routing, eg. invoked by

$> mpirun -np 12 --host c10,c11,c12 --mca btl_tcp_if_exclude lo,eth0  my_mpi_app

and (measured by iftop) saturate the available edges with
about 100MB/sec duplex on each of them. It would not stumble
on the fact, that some interfaces are not reacheable by
every NIC directly. And this was very convenient over the years.

With 1.4.3 (which comes out of the box) w/actual Linux distributions,
this won't work. It would hang and complain after timeout about failed
endpoint connects, eg:

[node12][[52378,1],2][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect]
 connect() to 10.0.1.11 failed: Connection timed out (110)

* Can the intelligent behaviour of 1.2.x be "configured back"?

* How should the topology look like to work with 1,4,x painlessly?

Thanks & regards

M.

Re: [OMPI users] mpirun should run with just the localhost interface on win?

[MM]

-Original Message-

if the interface is down, should localhost still allow mpirun to run mpi
processes?