Re: [OMPI users] RDMACM Differences

[Michael Shuey]

Alternatively, if OpenMPI is really trying to use both ports, you
could force it to use just one port with --mca btl_openib_if_include
mlx4_0:1 (probably)

--
Mike Shuey



On Tue, Mar 1, 2011 at 1:02 PM, Jeff Squyres  wrote:
> On Feb 28, 2011, at 12:49 PM, Jagga Soorma wrote:
>
>> -bash-3.2$ mpiexec --mca btl openib,self -mca btl_openib_warn_default_gid_
>> prefix 0 -np 2 --hostfile mpihosts 
>> /home/jagga/osu-micro-benchmarks-3.3/openmpi/ofed-1.5.2/bin/osu_latency
>
> Your use of btl_openib_warn_default_gid_prefix may have brought up a subtle 
> issue in Open MPI's verbs support.  More below.
>
>> # OSU MPI Latency Test v3.3
>> # Size            Latency (us)
>> [amber04][[10252,1],1][connect/btl_openib_connect_oob.c:325:qp_connect_all] 
>> error modifing QP to RTR errno says Invalid argument
>> [amber04][[10252,1],1][connect/btl_openib_connect_oob.c:815:rml_recv_cb] 
>> error in endpoint reply start connect
>
> Looking at this error message and your ibv_devinfo output:
>
>> [root@amber03 ~]# ibv_devinfo
>> hca_id:    mlx4_0
>>     transport:            InfiniBand (0)
>>     fw_ver:                2.7.9294
>>     node_guid:            78e7:d103:0021:8884
>>     sys_image_guid:            78e7:d103:0021:8887
>>     vendor_id:            0x02c9
>>     vendor_part_id:            26438
>>     hw_ver:                0xB0
>>     board_id:            HP_020003
>>     phys_port_cnt:            2
>>         port:    1
>>             state:            PORT_ACTIVE (4)
>>             max_mtu:        2048 (4)
>>             active_mtu:        2048 (4)
>>             sm_lid:            1
>>             port_lid:        20
>>             port_lmc:        0x00
>>             link_layer:        IB
>>
>>         port:    2
>>             state:            PORT_ACTIVE (4)
>>             max_mtu:        2048 (4)
>>             active_mtu:        1024 (3)
>>             sm_lid:            0
>>             port_lid:        0
>>             port_lmc:        0x00
>>             link_layer:        Ethernet
>
> It looks like you have 1 HCA port as IB and the other at Ethernet.
>
> I'm wondering if OMPI is not taking the device transport into account and is 
> *only* using the subnet ID to determine reachability (i.e., I'm wondering if 
> we didn't anticipate multiple devices/ports with the same subnet ID but with 
> different transports).  I pointed this out to Mellanox yesterday; I think 
> they're following up on it.
>
> In the meantime, a workaround might be to set a non-default subnet ID on your 
> IB network.  That should allow Open MPI to tell these networks apart without 
> additional help.
>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] Unknown overhead in "mpirun -am ft-enable-cr"

[Joshua Hursey]

Thanks for the program. I created a ticket for this performance bug and 
attached the tarball to the ticket:
  https://svn.open-mpi.org/trac/ompi/ticket/2743

I do not know exactly when I will be able to get back to this, but hopefully 
soon. I added you to the CC so you should receive any progress updates 
regarding the ticket as we move forward.

Thanks again,
Josh

On Mar 3, 2011, at 2:12 AM, Nguyen Toan wrote:

> Dear Josh,
>  
> Attached with this email is a small program that illustrates the performance 
> problem. You can find simple instructions in the README file.
> There are also 2 sample result files (cpu.256^3.8N.*) which show the 
> execution time difference between 2 cases.
> Hope you can take some time to find the problem.
> Thanks for your kindness.
> 
> Best Regards,
> Nguyen Toan
> 
> On Wed, Mar 2, 2011 at 3:00 AM, Joshua Hursey  wrote:
> I have not had the time to look into the performance problem yet, and 
> probably won't for a little while. Can you send me a small program that 
> illustrates the performance problem, and I'll file a bug so we don't lose 
> track of it.
> 
> Thanks,
> Josh
> 
> On Feb 25, 2011, at 1:31 PM, Nguyen Toan wrote:
> 
> > Dear Josh,
> >
> > Did you find out the problem? I still cannot progress anything.
> > Hope to hear some good news from you.
> >
> > Regards,
> > Nguyen Toan
> >
> > On Sun, Feb 13, 2011 at 3:04 PM, Nguyen Toan  
> > wrote:
> > Hi Josh,
> >
> > I tried the MCA parameter you mentioned but it did not help, the unknown 
> > overhead still exists.
> > Here I attach the output of 'ompi_info', both version 1.5 and 1.5.1.
> > Hope you can find out the problem.
> > Thank you.
> >
> > Regards,
> > Nguyen Toan
> >
> > On Wed, Feb 9, 2011 at 11:08 PM, Joshua Hursey  
> > wrote:
> > It looks like the logic in the configure script is turning on the FT thread 
> > for you when you specify both '--with-ft=cr' and '--enable-mpi-threads'.
> >
> > Can you send me the output of 'ompi_info'? Can you also try the MCA 
> > parameter that I mentioned earlier to see if that changes the performance?
> >
> > I there are many non-blocking sends and receives, there might be 
> > performance bug with the way the point-to-point wrapper is tracking request 
> > objects. If the above MCA parameter does not help the situation, let me 
> > know and I might be able to take a look at this next week.
> >
> > Thanks,
> > Josh
> >
> > On Feb 9, 2011, at 1:40 AM, Nguyen Toan wrote:
> >
> > > Hi Josh,
> > > Thanks for the reply. I did not use the '--enable-ft-thread' option. Here 
> > > is my build options:
> > >
> > > CFLAGS=-g \
> > > ./configure \
> > > --with-ft=cr \
> > > --enable-mpi-threads \
> > > --with-blcr=/home/nguyen/opt/blcr \
> > > --with-blcr-libdir=/home/nguyen/opt/blcr/lib \
> > > --prefix=/home/nguyen/opt/openmpi \
> > > --with-openib \
> > > --enable-mpirun-prefix-by-default
> > >
> > > My application requires lots of communication in every loop, focusing on 
> > > MPI_Isend, MPI_Irecv and MPI_Wait. Also I want to make only one 
> > > checkpoint per application execution for my purpose, but the unknown 
> > > overhead exists even when no checkpoint was taken.
> > >
> > > Do you have any other idea?
> > >
> > > Regards,
> > > Nguyen Toan
> > >
> > >
> > > On Wed, Feb 9, 2011 at 12:41 AM, Joshua Hursey  
> > > wrote:
> > > There are a few reasons why this might be occurring. Did you build with 
> > > the '--enable-ft-thread' option?
> > >
> > > If so, it looks like I didn't move over the thread_sleep_wait adjustment 
> > > from the trunk - the thread was being a bit too aggressive. Try adding 
> > > the following to your command line options, and see if it changes the 
> > > performance.
> > >  "-mca opal_cr_thread_sleep_wait 1000"
> > >
> > > There are other places to look as well depending on how frequently your 
> > > application communicates, how often you checkpoint, process layout, ... 
> > > But usually the aggressive nature of the thread is the main problem.
> > >
> > > Let me know if that helps.
> > >
> > > -- Josh
> > >
> > > On Feb 8, 2011, at 2:50 AM, Nguyen Toan wrote:
> > >
> > > > Hi all,
> > > >
> > > > I am using the latest version of OpenMPI (1.5.1) and BLCR (0.8.2).
> > > > I found that when running an application,which uses MPI_Isend, 
> > > > MPI_Irecv and MPI_Wait,
> > > > enabling C/R, i.e using "-am ft-enable-cr", the application runtime is 
> > > > much longer than the normal execution with mpirun (no checkpoint was 
> > > > taken).
> > > > This overhead becomes larger when the normal execution runtime is 
> > > > longer.
> > > > Does anybody have any idea about this overhead, and how to eliminate it?
> > > > Thanks.
> > > >
> > > > Regards,
> > > > Nguyen
> > > > ___
> > > > users mailing list
> > > > us...@open-mpi.org
> > > > http://www.open-mpi.org/mailman/listinfo.cgi/users
> > >
> > > 
> > > Joshua Hursey
> > > Postdoctoral Research Associate
> > > Oak Ri

Re: [OMPI users] Unknown overhead in "mpirun -am ft-enable-cr"

[Nguyen Toan]

Thanks Josh.
Actually I also tested with the Himeno
benchmarkand
got the same problem, so I think this could be a bug.
Hope this information also helps.

Regards,
Nguyen Toan

On Fri, Mar 4, 2011 at 12:04 AM, Joshua Hursey wrote:

> Thanks for the program. I created a ticket for this performance bug and
> attached the tarball to the ticket:
>  https://svn.open-mpi.org/trac/ompi/ticket/2743
>
> I do not know exactly when I will be able to get back to this, but
> hopefully soon. I added you to the CC so you should receive any progress
> updates regarding the ticket as we move forward.
>
> Thanks again,
> Josh
>
> On Mar 3, 2011, at 2:12 AM, Nguyen Toan wrote:
>
> > Dear Josh,
> >
> > Attached with this email is a small program that illustrates the
> performance problem. You can find simple instructions in the README file.
> > There are also 2 sample result files (cpu.256^3.8N.*) which show the
> execution time difference between 2 cases.
> > Hope you can take some time to find the problem.
> > Thanks for your kindness.
> >
> > Best Regards,
> > Nguyen Toan
> >
> > On Wed, Mar 2, 2011 at 3:00 AM, Joshua Hursey 
> wrote:
> > I have not had the time to look into the performance problem yet, and
> probably won't for a little while. Can you send me a small program that
> illustrates the performance problem, and I'll file a bug so we don't lose
> track of it.
> >
> > Thanks,
> > Josh
> >
> > On Feb 25, 2011, at 1:31 PM, Nguyen Toan wrote:
> >
> > > Dear Josh,
> > >
> > > Did you find out the problem? I still cannot progress anything.
> > > Hope to hear some good news from you.
> > >
> > > Regards,
> > > Nguyen Toan
> > >
> > > On Sun, Feb 13, 2011 at 3:04 PM, Nguyen Toan 
> wrote:
> > > Hi Josh,
> > >
> > > I tried the MCA parameter you mentioned but it did not help, the
> unknown overhead still exists.
> > > Here I attach the output of 'ompi_info', both version 1.5 and 1.5.1.
> > > Hope you can find out the problem.
> > > Thank you.
> > >
> > > Regards,
> > > Nguyen Toan
> > >
> > > On Wed, Feb 9, 2011 at 11:08 PM, Joshua Hursey 
> wrote:
> > > It looks like the logic in the configure script is turning on the FT
> thread for you when you specify both '--with-ft=cr' and
> '--enable-mpi-threads'.
> > >
> > > Can you send me the output of 'ompi_info'? Can you also try the MCA
> parameter that I mentioned earlier to see if that changes the performance?
> > >
> > > I there are many non-blocking sends and receives, there might be
> performance bug with the way the point-to-point wrapper is tracking request
> objects. If the above MCA parameter does not help the situation, let me know
> and I might be able to take a look at this next week.
> > >
> > > Thanks,
> > > Josh
> > >
> > > On Feb 9, 2011, at 1:40 AM, Nguyen Toan wrote:
> > >
> > > > Hi Josh,
> > > > Thanks for the reply. I did not use the '--enable-ft-thread' option.
> Here is my build options:
> > > >
> > > > CFLAGS=-g \
> > > > ./configure \
> > > > --with-ft=cr \
> > > > --enable-mpi-threads \
> > > > --with-blcr=/home/nguyen/opt/blcr \
> > > > --with-blcr-libdir=/home/nguyen/opt/blcr/lib \
> > > > --prefix=/home/nguyen/opt/openmpi \
> > > > --with-openib \
> > > > --enable-mpirun-prefix-by-default
> > > >
> > > > My application requires lots of communication in every loop, focusing
> on MPI_Isend, MPI_Irecv and MPI_Wait. Also I want to make only one
> checkpoint per application execution for my purpose, but the unknown
> overhead exists even when no checkpoint was taken.
> > > >
> > > > Do you have any other idea?
> > > >
> > > > Regards,
> > > > Nguyen Toan
> > > >
> > > >
> > > > On Wed, Feb 9, 2011 at 12:41 AM, Joshua Hursey <
> jjhur...@open-mpi.org> wrote:
> > > > There are a few reasons why this might be occurring. Did you build
> with the '--enable-ft-thread' option?
> > > >
> > > > If so, it looks like I didn't move over the thread_sleep_wait
> adjustment from the trunk - the thread was being a bit too aggressive. Try
> adding the following to your command line options, and see if it changes the
> performance.
> > > >  "-mca opal_cr_thread_sleep_wait 1000"
> > > >
> > > > There are other places to look as well depending on how frequently
> your application communicates, how often you checkpoint, process layout, ...
> But usually the aggressive nature of the thread is the main problem.
> > > >
> > > > Let me know if that helps.
> > > >
> > > > -- Josh
> > > >
> > > > On Feb 8, 2011, at 2:50 AM, Nguyen Toan wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I am using the latest version of OpenMPI (1.5.1) and BLCR (0.8.2).
> > > > > I found that when running an application,which uses MPI_Isend,
> MPI_Irecv and MPI_Wait,
> > > > > enabling C/R, i.e using "-am ft-enable-cr", the application runtime
> is much longer than the normal execution with mpirun (no checkpoint was
> taken).
> > > > > This overhead becomes larger when the normal execution runtime is
> longer.
> > > > > Doe

Re: [OMPI users] MPI_ALLREDUCE bug with 1.5.2rc3r24441

[Jeff Squyres]

I am unable to reproduce your problem with the 1.5.2rc3 tarball...?

Does your compiler support INTEGER8?  Can you send the data requested here:

http://www.open-mpi.org/community/help/


On Mar 1, 2011, at 4:16 PM, Harald Anlauf wrote:

> Hi,
> 
> there appears to be a regression in revision 1.5.2rc3r24441.
> The attached program crashes even with 1 PE with:
> 
> Default real, digits:   4  24
> Real kind,digits:   8  53
> Integer kind,   bits:   8   64
> Default Integer :   4  32
> Sum[real]:   1.000   2.000   3.000
> Sum[real(8)]:   1.2.
> 3.
> Sum[integer(4)]:   1   2   3
> [proton:24826] *** An error occurred in MPI_Allreduce: the reduction
> operation MPI_SUM is not defined on the MPI_INTEGER8 datatype
> 
> 
> On the other hand,
> 
> % ompi_info --arch
> Configured architecture: i686-pc-linux-gnu
> % ompi_info --all |grep 'integer[48]'
>  Fort have integer4: yes
>  Fort have integer8: yes
>  Fort integer4 size: 4
>  Fort integer8 size: 8
> Fort integer4 align: 4
> Fort integer8 align: 8
> 
> There are no problems with 1.4.x and earlier revisions.
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] Mac OS X Static PGI

[David Robertson]

UPDATE:

Sorry for the delay but I wanted to make sure PGI was ok with me sharing 
their workaround.


Further conversation with PGI tech support has yielded a solution. The 
opal/util/if.c file has the following around line 63:


#include 

Here is the explanation I have from PGI:

< Start Quote
For 64-bit only there might be an issue of 'not running/crashing'
when it tries to establish the connection.

The reason is probably because a source file  includes .
 has some source code that we don't support , namely
#pragma pack(4)
I think currently ignore it.

The file is: opal/util/if.c  that  includes .
You may succeed  by  including the  attached pgi.h instead of .
End Quote >

I followed this advise along with editing the 
share/openmpi/mpif*-wrapper-data.txt files to have full paths to the 
static libraries instead of just -lmpi_f90, -lmpi_f77, -lmpi, etc.


Dave


pgi.h
Description: application/unknown

Re: [OMPI users] MPI_ALLREDUCE bug with 1.5.2rc3r24441

[Harald Anlauf]

Please find attached the output of:

configure
make all
make install
ompi_info -all
mpif90 -v mpiallreducetest.f90
ldd a.out
./a.out

System: OpenSuse Linux 11.1 on Core2Duo, i686

Compiler is:
Intel(R) Fortran Compiler XE for applications running on IA-32, Version
12.0.1.107 Build 20101116

(The problem is the same with gfortran 4.6 (prerelease).)

Harald


ompi-output.tar.bz2
Description: application/bzip

Re: [OMPI users] Mac OS X Static PGI

[Ralph Castain]

Really appreciate you having looked into this!

Unfortunately, I can't see a way to resolve this for the general public. It 
looks more to me like a PGI bug, frankly - not supporting code in a 
system-level include makes no sense to me. But I confess this seems to be PGI's 
mode of operation as I've seen similar issues with their compilers under other 
OS's as well.

We obviously cannot replace Mac's if.h with the PGI-custom version, nor can we 
distribute the PGI-custom version for use in that situation. So until/unless 
PGI fixes their problem, I think this has to be a one-off solution.

Again, thanks for looking into it. Glad that it works for you!
Ralph


On Mar 3, 2011, at 1:28 PM, David Robertson wrote:

> UPDATE:
> 
> Sorry for the delay but I wanted to make sure PGI was ok with me sharing 
> their workaround.
> 
> Further conversation with PGI tech support has yielded a solution. The 
> opal/util/if.c file has the following around line 63:
> 
> #include 
> 
> Here is the explanation I have from PGI:
> 
> < Start Quote
> For 64-bit only there might be an issue of 'not running/crashing'
> when it tries to establish the connection.
> 
> The reason is probably because a source file  includes .
>  has some source code that we don't support , namely
> #pragma pack(4)
> I think currently ignore it.
> 
> The file is: opal/util/if.c  that  includes .
> You may succeed  by  including the  attached pgi.h instead of .
> End Quote >
> 
> I followed this advise along with editing the 
> share/openmpi/mpif*-wrapper-data.txt files to have full paths to the static 
> libraries instead of just -lmpi_f90, -lmpi_f77, -lmpi, etc.
> 
> Dave
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users