Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Ralph Castain]

Understood - but I was wondering if that was true for OMPI as well.

On Jul 9, 2013, at 11:30 AM, "Daniels, Marcus G"  wrote:

> The Intel MPI implementation does this.  The performance between the 
> accelerators and the host is poor though.   About 20mb/sec in my ping/pong 
> test.   Intra-MIC communication is about a 1GB/sec whereas intra-host is 
> about 6GB/sec.  Latency is higher (i.e. worse) for the intra-MIC 
> communication too (vs. intra-host) by about the same factor.  
> 
> Thanks Tim for the hints on building OpenMPI.  
> 
> From: users-boun...@open-mpi.org [users-boun...@open-mpi.org] on behalf of 
> Ralph Castain [r...@open-mpi.org]
> Sent: Tuesday, July 09, 2013 12:14 PM
> To: Tim Carlson; Open MPI Users
> Subject: Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5  
>   an 1.7.2
> 
> Hi Tim
> 
> Quick question: can the procs on the MIC communicate with procs on (a) the 
> local host, (b) other hosts, and (c) MICs on other hosts?
> 
> The last two would depend on having direct access to one or more network 
> transports.
> 
> 
> On Jul 9, 2013, at 10:18 AM, Tim Carlson  wrote:
> 
>> On Mon, 8 Jul 2013, Tim Carlson wrote:
>> 
>> Now that I have gone through this process, I'll report that it works with 
>> the caveat that you can't use the openmpi wrappers for compiling. Recall 
>> that the Phi card does not have either the GNU or Intel compilers installed. 
>> While you could build up a tool chain for the GNU compilers, you're not 
>> going to get a native Intel compiler unless Intel decides to support it.
>> 
>> Here is the process from end to end to get Openmpi to build a native Phi 
>> application.
>> 
>> export PATH=/usr/linux-k1om-4.7/bin:$PATH
>> . /share/apps/intel/composer_xe_2013.3.163/bin/iccvars.sh intel64
>> export CC="icc -mmic"
>> export CXX="icpc -mmic"
>> 
>> cd ~
>> tar zxf openmpi-1.6.4.tar.gz
>> cd openmpi-1.6.4
>> ./configure --prefix=/people/tim/mic/openmpi/intel \
>> --build=x86_64-unknown-linux-gnu --host=x86_64-k1om-linux \
>> --disable-mpi-f77 \
>> AR=x86_64-k1om-linux-ar RANLIB=x86_64-k1om-linux-ranlib 
>> LD=x86_64-k1om-linux-ld
>> make
>> make install
>> 
>> That leaves me with a native build of openmpi in 
>> /people/tim/mic/openmpi/intel
>> 
>> It is of course tempting to just do a
>> export PATH=/people/tim/mic/openmpi/intel/bin:$PATH
>> and start using mpicc to build my code but that does not work because:
>> 
>> 1) If I try this on the host system I am going to get "wrong architecture" 
>> because mpicc was build for the Phi and not for the x86_64 host
>> 
>> 2) If I try running it on the Phi, I don't have access to "icc" because I 
>> can't run the compiler directly on the Phi.
>> 
>> I can "cheat" and see what the mpicc command really does by using "mpicc 
>> --show" for another installation of openmpi and munge the paths correctly. 
>> In this case
>> 
>> icc -mmic cpi.c -I/people/tim/mic/openmpi/intel/include -pthread \
>> -L/people/tim/mic/openmpi/intel/lib -lmpi -ldl -lm -Wl,--export-dynamic \
>> -lrt -lnsl -lutil -lm -ldl -o cpi.x
>> 
>> That leaves me with a Phi native version of cpi.x which I can then execute 
>> on the Phi
>> 
>> $ ssh phi002-mic0
>> 
>> ( I have NFS mounts on the Phi for all the bits I need )
>> 
>> ~ $ export PATH=/people/tim/mic/openmpi/intel/bin/:$PATH
>> ~ $ export 
>> LD_LIBRARY_PATH=/share/apps/intel/composer_xe_2013.3.163/compiler/lib/mic/
>> ~ $ export LD_LIBRARY_PATH=/people/tim/mic/openmpi/intel/lib:$LD_LIBRARY_PATH
>> ~ $ cd mic
>> ~/mic $ mpirun -np 12 cpi.x
>> Process 7 on phi002-mic0.local
>> Process 10 on phi002-mic0.local
>> Process 2 on phi002-mic0.local
>> Process 9 on phi002-mic0.local
>> Process 1 on phi002-mic0.local
>> Process 3 on phi002-mic0.local
>> Process 11 on phi002-mic0.local
>> Process 5 on phi002-mic0.local
>> Process 8 on phi002-mic0.local
>> Process 4 on phi002-mic0.local
>> Process 6 on phi002-mic0.local
>> Process 0 on phi002-mic0.local
>> pi is approximately 3.1416009869231245, Error is 0.0814
>> wall clock time = 0.001766
>> 
>> 
>>> On Mon, 8 Jul 2013, Elken, Tom wrote:
>>> 
>>> My mistake on the OFED bits. The host I was installing on did not have all 
>>> of the MPSS software installed (my cluster admin node and not one of the 
>>> compute nodes). Adding the intel-mic-ofed-card RPM fixed the problem with 
>>> compiling the btl:openib bits with both the GNU and Intel compilers using 
>>> the cross-compiler route (-mmic on the Intel side)
>>> 
>>> Still working on getting the resulting mpicc wrapper working on the MIC 
>>> side. When I get a working example I'll post the results.
>>> 
>>> Thanks!
>>> 
>>> Tim
>>> 
>>> 
 
 
 
 Hi Tim,
 
 
 
 
 
 Well, in general and not on MIC I usually build the MPI stacks using the
 Intel compiler set. Have you ran into s/w that requires GCC instead of
 Intel
 compilers (beside Nvidia Cuda)? Did you 

Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Daniels, Marcus G]

The Intel MPI implementation does this.  The performance between the 
accelerators and the host is poor though.   About 20mb/sec in my ping/pong 
test.   Intra-MIC communication is about a 1GB/sec whereas intra-host is about 
6GB/sec.  Latency is higher (i.e. worse) for the intra-MIC communication too 
(vs. intra-host) by about the same factor.  

Thanks Tim for the hints on building OpenMPI.  

From: users-boun...@open-mpi.org [users-boun...@open-mpi.org] on behalf of 
Ralph Castain [r...@open-mpi.org]
Sent: Tuesday, July 09, 2013 12:14 PM
To: Tim Carlson; Open MPI Users
Subject: Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5
an 1.7.2

Hi Tim

Quick question: can the procs on the MIC communicate with procs on (a) the 
local host, (b) other hosts, and (c) MICs on other hosts?

The last two would depend on having direct access to one or more network 
transports.


On Jul 9, 2013, at 10:18 AM, Tim Carlson  wrote:

> On Mon, 8 Jul 2013, Tim Carlson wrote:
>
> Now that I have gone through this process, I'll report that it works with the 
> caveat that you can't use the openmpi wrappers for compiling. Recall that the 
> Phi card does not have either the GNU or Intel compilers installed. While you 
> could build up a tool chain for the GNU compilers, you're not going to get a 
> native Intel compiler unless Intel decides to support it.
>
> Here is the process from end to end to get Openmpi to build a native Phi 
> application.
>
> export PATH=/usr/linux-k1om-4.7/bin:$PATH
> . /share/apps/intel/composer_xe_2013.3.163/bin/iccvars.sh intel64
> export CC="icc -mmic"
> export CXX="icpc -mmic"
>
> cd ~
> tar zxf openmpi-1.6.4.tar.gz
> cd openmpi-1.6.4
> ./configure --prefix=/people/tim/mic/openmpi/intel \
> --build=x86_64-unknown-linux-gnu --host=x86_64-k1om-linux \
> --disable-mpi-f77 \
> AR=x86_64-k1om-linux-ar RANLIB=x86_64-k1om-linux-ranlib 
> LD=x86_64-k1om-linux-ld
> make
> make install
>
> That leaves me with a native build of openmpi in /people/tim/mic/openmpi/intel
>
> It is of course tempting to just do a
> export PATH=/people/tim/mic/openmpi/intel/bin:$PATH
> and start using mpicc to build my code but that does not work because:
>
> 1) If I try this on the host system I am going to get "wrong architecture" 
> because mpicc was build for the Phi and not for the x86_64 host
>
> 2) If I try running it on the Phi, I don't have access to "icc" because I 
> can't run the compiler directly on the Phi.
>
> I can "cheat" and see what the mpicc command really does by using "mpicc 
> --show" for another installation of openmpi and munge the paths correctly. In 
> this case
>
> icc -mmic cpi.c -I/people/tim/mic/openmpi/intel/include -pthread \
> -L/people/tim/mic/openmpi/intel/lib -lmpi -ldl -lm -Wl,--export-dynamic \
> -lrt -lnsl -lutil -lm -ldl -o cpi.x
>
> That leaves me with a Phi native version of cpi.x which I can then execute on 
> the Phi
>
> $ ssh phi002-mic0
>
> ( I have NFS mounts on the Phi for all the bits I need )
>
> ~ $ export PATH=/people/tim/mic/openmpi/intel/bin/:$PATH
> ~ $ export 
> LD_LIBRARY_PATH=/share/apps/intel/composer_xe_2013.3.163/compiler/lib/mic/
> ~ $ export LD_LIBRARY_PATH=/people/tim/mic/openmpi/intel/lib:$LD_LIBRARY_PATH
> ~ $ cd mic
> ~/mic $ mpirun -np 12 cpi.x
> Process 7 on phi002-mic0.local
> Process 10 on phi002-mic0.local
> Process 2 on phi002-mic0.local
> Process 9 on phi002-mic0.local
> Process 1 on phi002-mic0.local
> Process 3 on phi002-mic0.local
> Process 11 on phi002-mic0.local
> Process 5 on phi002-mic0.local
> Process 8 on phi002-mic0.local
> Process 4 on phi002-mic0.local
> Process 6 on phi002-mic0.local
> Process 0 on phi002-mic0.local
> pi is approximately 3.1416009869231245, Error is 0.0814
> wall clock time = 0.001766
>
>
>> On Mon, 8 Jul 2013, Elken, Tom wrote:
>>
>> My mistake on the OFED bits. The host I was installing on did not have all 
>> of the MPSS software installed (my cluster admin node and not one of the 
>> compute nodes). Adding the intel-mic-ofed-card RPM fixed the problem with 
>> compiling the btl:openib bits with both the GNU and Intel compilers using 
>> the cross-compiler route (-mmic on the Intel side)
>>
>> Still working on getting the resulting mpicc wrapper working on the MIC 
>> side. When I get a working example I'll post the results.
>>
>> Thanks!
>>
>> Tim
>>
>>
>>>
>>>
>>>
>>> Hi Tim,
>>>
>>>
>>>
>>>
>>>
>>> Well, in general and not on MIC I usually build the MPI stacks using the
>>> Intel compiler set. Have you ran into s/w that requires GCC instead of
>>> Intel
>>> compilers (beside Nvidia Cuda)? Did you try to use Intel compiler to
>>> produce
>>> MIC native code (the OpenMPI stack for that matter)?
>>>
>>> [Tom]
>>>
>>> Good idea Michael,  With the Intel Compiler, I would use the -mmic flag to
>>> build MIC code.
>>>
>>>
>>>
>>> Tim wrote: “My first pass at doing a cross-compile with the GNU compilers
>>> failed to produce something 

Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Ralph Castain]

Hi Tim

Quick question: can the procs on the MIC communicate with procs on (a) the 
local host, (b) other hosts, and (c) MICs on other hosts?

The last two would depend on having direct access to one or more network 
transports.


On Jul 9, 2013, at 10:18 AM, Tim Carlson  wrote:

> On Mon, 8 Jul 2013, Tim Carlson wrote:
> 
> Now that I have gone through this process, I'll report that it works with the 
> caveat that you can't use the openmpi wrappers for compiling. Recall that the 
> Phi card does not have either the GNU or Intel compilers installed. While you 
> could build up a tool chain for the GNU compilers, you're not going to get a 
> native Intel compiler unless Intel decides to support it.
> 
> Here is the process from end to end to get Openmpi to build a native Phi 
> application.
> 
> export PATH=/usr/linux-k1om-4.7/bin:$PATH
> . /share/apps/intel/composer_xe_2013.3.163/bin/iccvars.sh intel64
> export CC="icc -mmic"
> export CXX="icpc -mmic"
> 
> cd ~
> tar zxf openmpi-1.6.4.tar.gz
> cd openmpi-1.6.4
> ./configure --prefix=/people/tim/mic/openmpi/intel \
> --build=x86_64-unknown-linux-gnu --host=x86_64-k1om-linux \
> --disable-mpi-f77 \
> AR=x86_64-k1om-linux-ar RANLIB=x86_64-k1om-linux-ranlib 
> LD=x86_64-k1om-linux-ld
> make
> make install
> 
> That leaves me with a native build of openmpi in /people/tim/mic/openmpi/intel
> 
> It is of course tempting to just do a
> export PATH=/people/tim/mic/openmpi/intel/bin:$PATH
> and start using mpicc to build my code but that does not work because:
> 
> 1) If I try this on the host system I am going to get "wrong architecture" 
> because mpicc was build for the Phi and not for the x86_64 host
> 
> 2) If I try running it on the Phi, I don't have access to "icc" because I 
> can't run the compiler directly on the Phi.
> 
> I can "cheat" and see what the mpicc command really does by using "mpicc 
> --show" for another installation of openmpi and munge the paths correctly. In 
> this case
> 
> icc -mmic cpi.c -I/people/tim/mic/openmpi/intel/include -pthread \
> -L/people/tim/mic/openmpi/intel/lib -lmpi -ldl -lm -Wl,--export-dynamic \
> -lrt -lnsl -lutil -lm -ldl -o cpi.x
> 
> That leaves me with a Phi native version of cpi.x which I can then execute on 
> the Phi
> 
> $ ssh phi002-mic0
> 
> ( I have NFS mounts on the Phi for all the bits I need )
> 
> ~ $ export PATH=/people/tim/mic/openmpi/intel/bin/:$PATH
> ~ $ export 
> LD_LIBRARY_PATH=/share/apps/intel/composer_xe_2013.3.163/compiler/lib/mic/
> ~ $ export LD_LIBRARY_PATH=/people/tim/mic/openmpi/intel/lib:$LD_LIBRARY_PATH
> ~ $ cd mic
> ~/mic $ mpirun -np 12 cpi.x
> Process 7 on phi002-mic0.local
> Process 10 on phi002-mic0.local
> Process 2 on phi002-mic0.local
> Process 9 on phi002-mic0.local
> Process 1 on phi002-mic0.local
> Process 3 on phi002-mic0.local
> Process 11 on phi002-mic0.local
> Process 5 on phi002-mic0.local
> Process 8 on phi002-mic0.local
> Process 4 on phi002-mic0.local
> Process 6 on phi002-mic0.local
> Process 0 on phi002-mic0.local
> pi is approximately 3.1416009869231245, Error is 0.0814
> wall clock time = 0.001766
> 
> 
>> On Mon, 8 Jul 2013, Elken, Tom wrote:
>> 
>> My mistake on the OFED bits. The host I was installing on did not have all 
>> of the MPSS software installed (my cluster admin node and not one of the 
>> compute nodes). Adding the intel-mic-ofed-card RPM fixed the problem with 
>> compiling the btl:openib bits with both the GNU and Intel compilers using 
>> the cross-compiler route (-mmic on the Intel side)
>> 
>> Still working on getting the resulting mpicc wrapper working on the MIC 
>> side. When I get a working example I'll post the results.
>> 
>> Thanks!
>> 
>> Tim
>> 
>> 
>>> 
>>>  
>>> 
>>> Hi Tim,
>>> 
>>>  
>>> 
>>>  
>>> 
>>> Well, in general and not on MIC I usually build the MPI stacks using the
>>> Intel compiler set. Have you ran into s/w that requires GCC instead of
>>> Intel
>>> compilers (beside Nvidia Cuda)? Did you try to use Intel compiler to
>>> produce
>>> MIC native code (the OpenMPI stack for that matter)? 
>>> 
>>> [Tom]
>>> 
>>> Good idea Michael,  With the Intel Compiler, I would use the -mmic flag to
>>> build MIC code.
>>> 
>>>  
>>> 
>>> Tim wrote: “My first pass at doing a cross-compile with the GNU compilers
>>> failed to produce something with OFED support (not surprising)
>>> 
>>> export PATH=/usr/linux-k1om-4.7/bin:$PATH
>>> ./configure --build=x86_64-unknown-linux-gnu --host=x86_64-k1om-linux \
>>> --disable-mpi-f77
>>> 
>>> checking if MCA component btl:openib can compile... no
>>> 
>>>  
>>> 
>>> Regarding a Gnu cross compiler, this worked for one of our engineers
>>> building for True Scale HCAs and PSM/infinipath.  But it may give useful
>>> tips for building for btl:openib as well:
>>> 
>>>  
>>> 
>>> 3. How to configure/compile OpenMPI:
>>> 
>>>a). untar the openmpi tarball.
>>> 
>>>b). edit configure in top directory, add '-linfinipath' after
>>> '-lpsm_infinipath "
>>> 

Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Tim Carlson]

On Mon, 8 Jul 2013, Tim Carlson wrote:

Now that I have gone through this process, I'll report that it works with 
the caveat that you can't use the openmpi wrappers for compiling. Recall 
that the Phi card does not have either the GNU or Intel compilers 
installed. While you could build up a tool chain for the GNU 
compilers, you're not going to get a native Intel compiler unless Intel 
decides to support it.


Here is the process from end to end to get Openmpi to build a native Phi 
application.


export PATH=/usr/linux-k1om-4.7/bin:$PATH
. /share/apps/intel/composer_xe_2013.3.163/bin/iccvars.sh intel64
export CC="icc -mmic"
export CXX="icpc -mmic"

cd ~
tar zxf openmpi-1.6.4.tar.gz
cd openmpi-1.6.4
./configure --prefix=/people/tim/mic/openmpi/intel \
--build=x86_64-unknown-linux-gnu --host=x86_64-k1om-linux \
--disable-mpi-f77 \
AR=x86_64-k1om-linux-ar RANLIB=x86_64-k1om-linux-ranlib LD=x86_64-k1om-linux-ld
make
make install

That leaves me with a native build of openmpi in 
/people/tim/mic/openmpi/intel


It is of course tempting to just do a
export PATH=/people/tim/mic/openmpi/intel/bin:$PATH
and start using mpicc to build my code but that does not work because:

1) If I try this on the host system I am going to get "wrong architecture" 
because mpicc was build for the Phi and not for the x86_64 host


2) If I try running it on the Phi, I don't have access to "icc" because I 
can't run the compiler directly on the Phi.


I can "cheat" and see what the mpicc command really does by using "mpicc 
--show" for another installation of openmpi and munge the paths correctly. 
In this case


icc -mmic cpi.c -I/people/tim/mic/openmpi/intel/include -pthread \
-L/people/tim/mic/openmpi/intel/lib -lmpi -ldl -lm -Wl,--export-dynamic \
-lrt -lnsl -lutil -lm -ldl -o cpi.x

That leaves me with a Phi native version of cpi.x which I can then 
execute on the Phi


$ ssh phi002-mic0

( I have NFS mounts on the Phi for all the bits I need )

~ $ export PATH=/people/tim/mic/openmpi/intel/bin/:$PATH
~ $ export 
LD_LIBRARY_PATH=/share/apps/intel/composer_xe_2013.3.163/compiler/lib/mic/
~ $ export LD_LIBRARY_PATH=/people/tim/mic/openmpi/intel/lib:$LD_LIBRARY_PATH
~ $ cd mic
~/mic $ mpirun -np 12 cpi.x
Process 7 on phi002-mic0.local
Process 10 on phi002-mic0.local
Process 2 on phi002-mic0.local
Process 9 on phi002-mic0.local
Process 1 on phi002-mic0.local
Process 3 on phi002-mic0.local
Process 11 on phi002-mic0.local
Process 5 on phi002-mic0.local
Process 8 on phi002-mic0.local
Process 4 on phi002-mic0.local
Process 6 on phi002-mic0.local
Process 0 on phi002-mic0.local
pi is approximately 3.1416009869231245, Error is 0.0814
wall clock time = 0.001766



On Mon, 8 Jul 2013, Elken, Tom wrote:

My mistake on the OFED bits. The host I was installing on did not have all of 
the MPSS software installed (my cluster admin node and not one of the compute 
nodes). Adding the intel-mic-ofed-card RPM fixed the problem with compiling 
the btl:openib bits with both the GNU and Intel compilers using the 
cross-compiler route (-mmic on the Intel side)


Still working on getting the resulting mpicc wrapper working on the MIC side. 
When I get a working example I'll post the results.


Thanks!

Tim




  

 Hi Tim,

  

  

 Well, in general and not on MIC I usually build the MPI stacks using the
 Intel compiler set. Have you ran into s/w that requires GCC instead of
 Intel
 compilers (beside Nvidia Cuda)? Did you try to use Intel compiler to
 produce
 MIC native code (the OpenMPI stack for that matter)? 

 [Tom]

 Good idea Michael,  With the Intel Compiler, I would use the -mmic flag to
 build MIC code.

  

 Tim wrote: “My first pass at doing a cross-compile with the GNU compilers
 failed to produce something with OFED support (not surprising)

 export PATH=/usr/linux-k1om-4.7/bin:$PATH
 ./configure --build=x86_64-unknown-linux-gnu --host=x86_64-k1om-linux \
 --disable-mpi-f77

 checking if MCA component btl:openib can compile... no

  

 Regarding a Gnu cross compiler, this worked for one of our engineers
 building for True Scale HCAs and PSM/infinipath.  But it may give useful
 tips for building for btl:openib as well:

  

 3. How to configure/compile OpenMPI:

    a). untar the openmpi tarball.

    b). edit configure in top directory, add '-linfinipath' after
 '-lpsm_infinipath "

        but not necessary for messages, only for command lines.

  

    c). run the following script,

 #!/bin/sh

  

 ./configure \

 --host=x86_64-k1om-linux \

 --enable-mpi-f77=no --enable-mpi-f90=no \

 --with-psm=/…/psm-7.6 \

 --prefix=/…/openmpi \

 CC=x86_64-k1om-linux-gcc  CXX=x86_64-k1om-linux-g++ \

 AR=x86_64-k1om-linux-ar RANLIB=x86_64-k1om-linux-ranlib

  

    d). run 'make' and 'make install'

  

 OK, I see that they did not configure for mpi-f77 & mpif90, but perhaps
 this
 is still helpful, if the AR and RANLIB flags are important.

 -Tom

  

  

  

 regards

 Michael

  

 On Mon, Jul 8, 2013 at 4:30 

Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Tim Carlson]

On Mon, 8 Jul 2013, Elken, Tom wrote:

My mistake on the OFED bits. The host I was installing on did not have all 
of the MPSS software installed (my cluster admin node and not one of the 
compute nodes). Adding the intel-mic-ofed-card RPM fixed the problem with 
compiling the btl:openib bits with both the GNU and Intel compilers using 
the cross-compiler route (-mmic on the Intel side)


Still working on getting the resulting mpicc wrapper working on the MIC 
side. When I get a working example I'll post the results.


Thanks!

Tim




 

Hi Tim,

 

 

Well, in general and not on MIC I usually build the MPI stacks using the
Intel compiler set. Have you ran into s/w that requires GCC instead of Intel
compilers (beside Nvidia Cuda)? Did you try to use Intel compiler to produce
MIC native code (the OpenMPI stack for that matter)? 

[Tom]

Good idea Michael,  With the Intel Compiler, I would use the -mmic flag to
build MIC code.

 

Tim wrote: “My first pass at doing a cross-compile with the GNU compilers
failed to produce something with OFED support (not surprising)

export PATH=/usr/linux-k1om-4.7/bin:$PATH
./configure --build=x86_64-unknown-linux-gnu --host=x86_64-k1om-linux \
--disable-mpi-f77

checking if MCA component btl:openib can compile... no

 

Regarding a Gnu cross compiler, this worked for one of our engineers
building for True Scale HCAs and PSM/infinipath.  But it may give useful
tips for building for btl:openib as well:

 

3. How to configure/compile OpenMPI:

   a). untar the openmpi tarball.

   b). edit configure in top directory, add '-linfinipath' after
'-lpsm_infinipath "

       but not necessary for messages, only for command lines.

 

   c). run the following script,

#!/bin/sh

 

./configure \

--host=x86_64-k1om-linux \

--enable-mpi-f77=no --enable-mpi-f90=no \

--with-psm=/…/psm-7.6 \

--prefix=/…/openmpi \

CC=x86_64-k1om-linux-gcc  CXX=x86_64-k1om-linux-g++ \

AR=x86_64-k1om-linux-ar RANLIB=x86_64-k1om-linux-ranlib

 

   d). run 'make' and 'make install'

 

OK, I see that they did not configure for mpi-f77 & mpif90, but perhaps this
is still helpful, if the AR and RANLIB flags are important.

-Tom

 

 

 

regards

Michael

 

On Mon, Jul 8, 2013 at 4:30 PM, Tim Carlson  wrote:

On Mon, 8 Jul 2013, Elken, Tom wrote:

It isn't quite so easy.

Out of the box, there is no gcc on the Phi card. You can use the cross
compiler on the host, but you don't get gcc on the Phi by default.

See this post http://software.intel.com/en-us/forums/topic/382057

I really think you would need to build and install gcc on the Phi first.

My first pass at doing a cross-compile with the GNU compilers failed to
produce something with OFED support (not surprising)

export PATH=/usr/linux-k1om-4.7/bin:$PATH
./configure --build=x86_64-unknown-linux-gnu --host=x86_64-k1om-linux \
--disable-mpi-f77

checking if MCA component btl:openib can compile... no


Tim

   


   

  Thanks Tom, that sounds good. I will give it a try as soon as
  our Phi host
  here host gets installed. 

   

  I assume that all the prerequisite libs and bins on the Phi side
  are
  available when we download the Phi s/w stack from Intel's site,
  right ?

  [Tom]

  Right.  When you install Intel’s MPSS (Manycore Platform
  Software Stack),
  including following the section on “OFED Support” in the readme
  file, you
  should have all the prerequisite libs and bins.  Note that I
  have not built
  Open MPI for Xeon Phi for your interconnect, but it seems to me
  that it
  should work.

   

  -Tom

   

  Cheers

  Michael

   

   

   

  On Mon, Jul 8, 2013 at 12:10 PM, Elken, Tom
   wrote:

  Do you guys have any plan to support Intel Phi in the future?
  That is,
  running MPI code on the Phi cards or across the multicore and
  Phi, as Intel
  MPI does?

  [Tom]

  Hi Michael,

  Because a Xeon Phi card acts a lot like a Linux host with an x86
  architecture, you can build your own Open MPI libraries to serve
  this
  purpose.

  Our team has used existing (an older 1.4.3 version of) Open MPI
  source to
  build an Open MPI for running MPI code on Intel Xeon Phi cards
  over Intel’s
  (formerly QLogic’s) True Scale InfiniBand fabric, and it works
  quite well. 
  We have not released a pre-built Open MPI as part of any Intel
  software
  release.   But I think if you have a compiler for Xeon Phi
  (Intel Compiler
  or GCC) and an interconnect for it, you should be able to build
  an Open MPI
  that works on Xeon Phi. 

  Cheers,
  Tom Elken

  thanks...

  Michael

   

  On Sat, Jul 6, 2013 at 2:36 PM, Ralph Castain 
  wrote:

  Rolf will have to answer the question on level of support. The
  

Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Elken, Tom]

Hi Tim,


Well, in general and not on MIC I usually build the MPI stacks using the Intel 
compiler set. Have you ran into s/w that requires GCC instead of Intel 
compilers (beside Nvidia Cuda)? Did you try to use Intel compiler to produce 
MIC native code (the OpenMPI stack for that matter)?
[Tom]
Good idea Michael,  With the Intel Compiler, I would use the -mmic flag to 
build MIC code.

Tim wrote: "My first pass at doing a cross-compile with the GNU compilers 
failed to produce something with OFED support (not surprising)

export PATH=/usr/linux-k1om-4.7/bin:$PATH
./configure --build=x86_64-unknown-linux-gnu --host=x86_64-k1om-linux \
--disable-mpi-f77

checking if MCA component btl:openib can compile... no

Regarding a Gnu cross compiler, this worked for one of our engineers building 
for True Scale HCAs and PSM/infinipath.  But it may give useful tips for 
building for btl:openib as well:

3. How to configure/compile OpenMPI:
   a). untar the openmpi tarball.
   b). edit configure in top directory, add '-linfinipath' after 
'-lpsm_infinipath "
   but not necessary for messages, only for command lines.

   c). run the following script,
#!/bin/sh

./configure \
--host=x86_64-k1om-linux \
--enable-mpi-f77=no --enable-mpi-f90=no \
--with-psm=/.../psm-7.6 \
--prefix=/.../openmpi \
CC=x86_64-k1om-linux-gcc  CXX=x86_64-k1om-linux-g++ \
AR=x86_64-k1om-linux-ar RANLIB=x86_64-k1om-linux-ranlib

   d). run 'make' and 'make install'

OK, I see that they did not configure for mpi-f77 & mpif90, but perhaps this is 
still helpful, if the AR and RANLIB flags are important.
-Tom



regards
Michael

On Mon, Jul 8, 2013 at 4:30 PM, Tim Carlson 
> wrote:
On Mon, 8 Jul 2013, Elken, Tom wrote:

It isn't quite so easy.

Out of the box, there is no gcc on the Phi card. You can use the cross compiler 
on the host, but you don't get gcc on the Phi by default.

See this post http://software.intel.com/en-us/forums/topic/382057

I really think you would need to build and install gcc on the Phi first.

My first pass at doing a cross-compile with the GNU compilers failed to produce 
something with OFED support (not surprising)

export PATH=/usr/linux-k1om-4.7/bin:$PATH
./configure --build=x86_64-unknown-linux-gnu --host=x86_64-k1om-linux \
--disable-mpi-f77

checking if MCA component btl:openib can compile... no


Tim




Thanks Tom, that sounds good. I will give it a try as soon as our Phi host
here host gets installed.



I assume that all the prerequisite libs and bins on the Phi side are
available when we download the Phi s/w stack from Intel's site, right ?

[Tom]

Right.  When you install Intel's MPSS (Manycore Platform Software Stack),
including following the section on "OFED Support" in the readme file, you
should have all the prerequisite libs and bins.  Note that I have not built
Open MPI for Xeon Phi for your interconnect, but it seems to me that it
should work.



-Tom



Cheers

Michael







On Mon, Jul 8, 2013 at 12:10 PM, Elken, Tom 
> wrote:

Do you guys have any plan to support Intel Phi in the future? That is,
running MPI code on the Phi cards or across the multicore and Phi, as Intel
MPI does?

[Tom]

Hi Michael,

Because a Xeon Phi card acts a lot like a Linux host with an x86
architecture, you can build your own Open MPI libraries to serve this
purpose.

Our team has used existing (an older 1.4.3 version of) Open MPI source to
build an Open MPI for running MPI code on Intel Xeon Phi cards over Intel's
(formerly QLogic's) True Scale InfiniBand fabric, and it works quite well.
We have not released a pre-built Open MPI as part of any Intel software
release.   But I think if you have a compiler for Xeon Phi (Intel Compiler
or GCC) and an interconnect for it, you should be able to build an Open MPI
that works on Xeon Phi.

Cheers,
Tom Elken

thanks...

Michael



On Sat, Jul 6, 2013 at 2:36 PM, Ralph Castain 
> wrote:

Rolf will have to answer the question on level of support. The CUDA code is
not in the 1.6 series as it was developed after that series went "stable".
It is in the 1.7 series, although the level of support will likely be
incrementally increasing as that "feature" series continues to evolve.



On Jul 6, 2013, at 12:06 PM, Michael Thomadakis 
>
wrote:

> Hello OpenMPI,
>
> I am wondering what level of support is there for CUDA and GPUdirect on
OpenMPI 1.6.5 and 1.7.2.
>
> I saw the ./configure --with-cuda=CUDA_DIR option in the FAQ. However, it
seems that with configure v1.6.5 it was ignored.
>
> Can you identify GPU memory and send messages from it directly without
copying to host memory first?
>
>
> Or in general, what level of CUDA support is there on 1.6.5 and 1.7.2 ? Do
you support SDK 5.0 and above?
>
> Cheers ...
> Michael

> ___
> 

Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Michael Thomadakis]

Hi Tim,


Well, in general and not on MIC I usually build the MPI stacks using the
Intel compiler set. Have you ran into s/w that requires GCC instead of
Intel compilers (beside Nvidia Cuda)? Did you try to use Intel compiler to
produce MIC native code (the OpenMPI stack for that matter)?

regards
Michael


On Mon, Jul 8, 2013 at 4:30 PM, Tim Carlson  wrote:

> On Mon, 8 Jul 2013, Elken, Tom wrote:
>
> It isn't quite so easy.
>
> Out of the box, there is no gcc on the Phi card. You can use the cross
> compiler on the host, but you don't get gcc on the Phi by default.
>
> See this post 
> http://software.intel.com/en-**us/forums/topic/382057
>
> I really think you would need to build and install gcc on the Phi first.
>
> My first pass at doing a cross-compile with the GNU compilers failed to
> produce something with OFED support (not surprising)
>
> export PATH=/usr/linux-k1om-4.7/bin:$**PATH
> ./configure --build=x86_64-unknown-linux-**gnu --host=x86_64-k1om-linux \
> --disable-mpi-f77
>
> checking if MCA component btl:openib can compile... no
>
>
> Tim
>
>
>
>>
>>
>> Thanks Tom, that sounds good. I will give it a try as soon as our Phi host
>> here host gets installed.
>>
>>
>>
>> I assume that all the prerequisite libs and bins on the Phi side are
>> available when we download the Phi s/w stack from Intel's site, right ?
>>
>> [Tom]
>>
>> Right.  When you install Intel’s MPSS (Manycore Platform Software Stack),
>> including following the section on “OFED Support” in the readme file, you
>> should have all the prerequisite libs and bins.  Note that I have not
>> built
>> Open MPI for Xeon Phi for your interconnect, but it seems to me that it
>> should work.
>>
>>
>>
>> -Tom
>>
>>
>>
>> Cheers
>>
>> Michael
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Jul 8, 2013 at 12:10 PM, Elken, Tom  wrote:
>>
>> Do you guys have any plan to support Intel Phi in the future? That is,
>> running MPI code on the Phi cards or across the multicore and Phi, as
>> Intel
>> MPI does?
>>
>> [Tom]
>>
>> Hi Michael,
>>
>> Because a Xeon Phi card acts a lot like a Linux host with an x86
>> architecture, you can build your own Open MPI libraries to serve this
>> purpose.
>>
>> Our team has used existing (an older 1.4.3 version of) Open MPI source to
>> build an Open MPI for running MPI code on Intel Xeon Phi cards over
>> Intel’s
>> (formerly QLogic’s) True Scale InfiniBand fabric, and it works quite
>> well.
>> We have not released a pre-built Open MPI as part of any Intel software
>> release.   But I think if you have a compiler for Xeon Phi (Intel Compiler
>> or GCC) and an interconnect for it, you should be able to build an Open
>> MPI
>> that works on Xeon Phi.
>>
>> Cheers,
>> Tom Elken
>>
>> thanks...
>>
>> Michael
>>
>>
>>
>> On Sat, Jul 6, 2013 at 2:36 PM, Ralph Castain  wrote:
>>
>> Rolf will have to answer the question on level of support. The CUDA code
>> is
>> not in the 1.6 series as it was developed after that series went "stable".
>> It is in the 1.7 series, although the level of support will likely be
>> incrementally increasing as that "feature" series continues to evolve.
>>
>>
>>
>> On Jul 6, 2013, at 12:06 PM, Michael Thomadakis > >
>> wrote:
>>
>> > Hello OpenMPI,
>> >
>> > I am wondering what level of support is there for CUDA and GPUdirect on
>> OpenMPI 1.6.5 and 1.7.2.
>> >
>> > I saw the ./configure --with-cuda=CUDA_DIR option in the FAQ. However,
>> it
>> seems that with configure v1.6.5 it was ignored.
>> >
>> > Can you identify GPU memory and send messages from it directly without
>> copying to host memory first?
>> >
>> >
>> > Or in general, what level of CUDA support is there on 1.6.5 and 1.7.2 ?
>> Do
>> you support SDK 5.0 and above?
>> >
>> > Cheers ...
>> > Michael
>>
>> > __**_
>> > users mailing list
>> > us...@open-mpi.org
>> > http://www.open-mpi.org/**mailman/listinfo.cgi/users
>>
>>
>> __**_
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/**mailman/listinfo.cgi/users
>>
>>
>>
>>
>> __**_
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/**mailman/listinfo.cgi/users
>>
>>
>>
>>
>>
>>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Tim Carlson]

On Mon, 8 Jul 2013, Elken, Tom wrote:

It isn't quite so easy.

Out of the box, there is no gcc on the Phi card. You can use the cross 
compiler on the host, but you don't get gcc on the Phi by default.


See this post http://software.intel.com/en-us/forums/topic/382057

I really think you would need to build and install gcc on the Phi first.

My first pass at doing a cross-compile with the GNU compilers failed to 
produce something with OFED support (not surprising)


export PATH=/usr/linux-k1om-4.7/bin:$PATH
./configure --build=x86_64-unknown-linux-gnu --host=x86_64-k1om-linux \
--disable-mpi-f77

checking if MCA component btl:openib can compile... no


Tim



 

Thanks Tom, that sounds good. I will give it a try as soon as our Phi host
here host gets installed. 

 

I assume that all the prerequisite libs and bins on the Phi side are
available when we download the Phi s/w stack from Intel's site, right ?

[Tom]

Right.  When you install Intel’s MPSS (Manycore Platform Software Stack),
including following the section on “OFED Support” in the readme file, you
should have all the prerequisite libs and bins.  Note that I have not built
Open MPI for Xeon Phi for your interconnect, but it seems to me that it
should work.

 

-Tom

 

Cheers

Michael

 

 

 

On Mon, Jul 8, 2013 at 12:10 PM, Elken, Tom  wrote:

Do you guys have any plan to support Intel Phi in the future? That is,
running MPI code on the Phi cards or across the multicore and Phi, as Intel
MPI does?

[Tom]

Hi Michael,

Because a Xeon Phi card acts a lot like a Linux host with an x86
architecture, you can build your own Open MPI libraries to serve this
purpose.

Our team has used existing (an older 1.4.3 version of) Open MPI source to
build an Open MPI for running MPI code on Intel Xeon Phi cards over Intel’s
(formerly QLogic’s) True Scale InfiniBand fabric, and it works quite well. 
We have not released a pre-built Open MPI as part of any Intel software
release.   But I think if you have a compiler for Xeon Phi (Intel Compiler
or GCC) and an interconnect for it, you should be able to build an Open MPI
that works on Xeon Phi. 

Cheers,
Tom Elken

thanks...

Michael

 

On Sat, Jul 6, 2013 at 2:36 PM, Ralph Castain  wrote:

Rolf will have to answer the question on level of support. The CUDA code is
not in the 1.6 series as it was developed after that series went "stable".
It is in the 1.7 series, although the level of support will likely be
incrementally increasing as that "feature" series continues to evolve.



On Jul 6, 2013, at 12:06 PM, Michael Thomadakis 
wrote:

> Hello OpenMPI,
>
> I am wondering what level of support is there for CUDA and GPUdirect on
OpenMPI 1.6.5 and 1.7.2.
>
> I saw the ./configure --with-cuda=CUDA_DIR option in the FAQ. However, it
seems that with configure v1.6.5 it was ignored.
>
> Can you identify GPU memory and send messages from it directly without
copying to host memory first?
>
>
> Or in general, what level of CUDA support is there on 1.6.5 and 1.7.2 ? Do
you support SDK 5.0 and above?
>
> Cheers ...
> Michael

> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

 


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

 

Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Michael Thomadakis]

Thanks Tom, I will test it out...
regards
Michael


On Mon, Jul 8, 2013 at 1:16 PM, Elken, Tom  wrote:

>   ** **
>
> Thanks Tom, that sounds good. I will give it a try as soon as our Phi host
> here host gets installed. 
>
> ** **
>
> I assume that all the prerequisite libs and bins on the Phi side are
> available when we download the Phi s/w stack from Intel's site, right ?***
> *
>
> *[Tom] *
>
> *Right.  When you install Intel’s MPSS (Manycore Platform Software
> Stack), including following the section on “OFED Support” in the readme
> file, you should have all the prerequisite libs and bins.  Note that I have
> not built Open MPI for Xeon Phi for your interconnect, but it seems to me
> that it should work. *
>
> * *
>
> *-Tom*
>
> ** **
>
> Cheers
>
> Michael
>
> ** **
>
> ** **
>
> ** **
>
> On Mon, Jul 8, 2013 at 12:10 PM, Elken, Tom  wrote:**
> **
>
> Do you guys have any plan to support Intel Phi in the future? That is,
> running MPI code on the Phi cards or across the multicore and Phi, as Intel
> MPI does?
>
> *[Tom] *
>
> Hi Michael,
>
> Because a Xeon Phi card acts a lot like a Linux host with an x86
> architecture, you can build your own Open MPI libraries to serve this
> purpose.
>
> Our team has used existing (an older 1.4.3 version of) Open MPI source to
> build an Open MPI for running MPI code on Intel Xeon Phi cards over Intel’s
> (formerly QLogic’s) True Scale InfiniBand fabric, and it works quite well.
> We have not released a pre-built Open MPI as part of any Intel software
> release.   But I think if you have a compiler for Xeon Phi (Intel Compiler
> or GCC) and an interconnect for it, you should be able to build an Open MPI
> that works on Xeon Phi.  
>
> Cheers,
> Tom Elken
>
> thanks...
>
> Michael
>
>  
>
> On Sat, Jul 6, 2013 at 2:36 PM, Ralph Castain  wrote:***
> *
>
> Rolf will have to answer the question on level of support. The CUDA code
> is not in the 1.6 series as it was developed after that series went
> "stable". It is in the 1.7 series, although the level of support will
> likely be incrementally increasing as that "feature" series continues to
> evolve.
>
>
>
> On Jul 6, 2013, at 12:06 PM, Michael Thomadakis 
> wrote:
>
> > Hello OpenMPI,
> >
> > I am wondering what level of support is there for CUDA and GPUdirect on
> OpenMPI 1.6.5 and 1.7.2.
> >
> > I saw the ./configure --with-cuda=CUDA_DIR option in the FAQ. However,
> it seems that with configure v1.6.5 it was ignored.
> >
> > Can you identify GPU memory and send messages from it directly without
> copying to host memory first?
> >
> >
> > Or in general, what level of CUDA support is there on 1.6.5 and 1.7.2 ?
> Do you support SDK 5.0 and above?
> >
> > Cheers ...
> > Michael
>
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>  
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> ** **
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Elken, Tom]

Thanks Tom, that sounds good. I will give it a try as soon as our Phi host here 
host gets installed.

I assume that all the prerequisite libs and bins on the Phi side are available 
when we download the Phi s/w stack from Intel's site, right ?
[Tom]
Right.  When you install Intel's MPSS (Manycore Platform Software Stack), 
including following the section on "OFED Support" in the readme file, you 
should have all the prerequisite libs and bins.  Note that I have not built 
Open MPI for Xeon Phi for your interconnect, but it seems to me that it should 
work.

-Tom

Cheers
Michael



On Mon, Jul 8, 2013 at 12:10 PM, Elken, Tom 
> wrote:
Do you guys have any plan to support Intel Phi in the future? That is, running 
MPI code on the Phi cards or across the multicore and Phi, as Intel MPI does?
[Tom]
Hi Michael,
Because a Xeon Phi card acts a lot like a Linux host with an x86 architecture, 
you can build your own Open MPI libraries to serve this purpose.

Our team has used existing (an older 1.4.3 version of) Open MPI source to build 
an Open MPI for running MPI code on Intel Xeon Phi cards over Intel's (formerly 
QLogic's) True Scale InfiniBand fabric, and it works quite well.  We have not 
released a pre-built Open MPI as part of any Intel software release.   But I 
think if you have a compiler for Xeon Phi (Intel Compiler or GCC) and an 
interconnect for it, you should be able to build an Open MPI that works on Xeon 
Phi.
Cheers,
Tom Elken
thanks...
Michael

On Sat, Jul 6, 2013 at 2:36 PM, Ralph Castain 
> wrote:
Rolf will have to answer the question on level of support. The CUDA code is not 
in the 1.6 series as it was developed after that series went "stable". It is in 
the 1.7 series, although the level of support will likely be incrementally 
increasing as that "feature" series continues to evolve.


On Jul 6, 2013, at 12:06 PM, Michael Thomadakis 
> wrote:

> Hello OpenMPI,
>
> I am wondering what level of support is there for CUDA and GPUdirect on 
> OpenMPI 1.6.5 and 1.7.2.
>
> I saw the ./configure --with-cuda=CUDA_DIR option in the FAQ. However, it 
> seems that with configure v1.6.5 it was ignored.
>
> Can you identify GPU memory and send messages from it directly without 
> copying to host memory first?
>
>
> Or in general, what level of CUDA support is there on 1.6.5 and 1.7.2 ? Do 
> you support SDK 5.0 and above?
>
> Cheers ...
> Michael
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Michael Thomadakis]

Thanks Tom, that sounds good. I will give it a try as soon as our Phi host
here host gets installed.

I assume that all the prerequisite libs and bins on the Phi side are
available when we download the Phi s/w stack from Intel's site, right ?

Cheers
Michael




On Mon, Jul 8, 2013 at 12:10 PM, Elken, Tom  wrote:

>Do you guys have any plan to support Intel Phi in the future? That is,
> running MPI code on the Phi cards or across the multicore and Phi, as Intel
> MPI does?
>
> *[Tom] *
>
> Hi Michael,
>
> Because a Xeon Phi card acts a lot like a Linux host with an x86
> architecture, you can build your own Open MPI libraries to serve this
> purpose.
>
> Our team has used existing (an older 1.4.3 version of) Open MPI source to
> build an Open MPI for running MPI code on Intel Xeon Phi cards over Intel’s
> (formerly QLogic’s) True Scale InfiniBand fabric, and it works quite well.
> We have not released a pre-built Open MPI as part of any Intel software
> release.   But I think if you have a compiler for Xeon Phi (Intel Compiler
> or GCC) and an interconnect for it, you should be able to build an Open MPI
> that works on Xeon Phi.  
>
> Cheers,
> Tom Elken
>
> thanks...
>
> Michael
>
> ** **
>
> On Sat, Jul 6, 2013 at 2:36 PM, Ralph Castain  wrote:***
> *
>
> Rolf will have to answer the question on level of support. The CUDA code
> is not in the 1.6 series as it was developed after that series went
> "stable". It is in the 1.7 series, although the level of support will
> likely be incrementally increasing as that "feature" series continues to
> evolve.
>
>
>
> On Jul 6, 2013, at 12:06 PM, Michael Thomadakis 
> wrote:
>
> > Hello OpenMPI,
> >
> > I am wondering what level of support is there for CUDA and GPUdirect on
> OpenMPI 1.6.5 and 1.7.2.
> >
> > I saw the ./configure --with-cuda=CUDA_DIR option in the FAQ. However,
> it seems that with configure v1.6.5 it was ignored.
> >
> > Can you identify GPU memory and send messages from it directly without
> copying to host memory first?
> >
> >
> > Or in general, what level of CUDA support is there on 1.6.5 and 1.7.2 ?
> Do you support SDK 5.0 and above?
> >
> > Cheers ...
> > Michael
>
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> ** **
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Elken, Tom]

Do you guys have any plan to support Intel Phi in the future? That is, running 
MPI code on the Phi cards or across the multicore and Phi, as Intel MPI does?
[Tom]
Hi Michael,
Because a Xeon Phi card acts a lot like a Linux host with an x86 architecture, 
you can build your own Open MPI libraries to serve this purpose.

Our team has used existing (an older 1.4.3 version of) Open MPI source to build 
an Open MPI for running MPI code on Intel Xeon Phi cards over Intel's (formerly 
QLogic's) True Scale InfiniBand fabric, and it works quite well.  We have not 
released a pre-built Open MPI as part of any Intel software release.   But I 
think if you have a compiler for Xeon Phi (Intel Compiler or GCC) and an 
interconnect for it, you should be able to build an Open MPI that works on Xeon 
Phi.
Cheers,
Tom Elken
thanks...
Michael

On Sat, Jul 6, 2013 at 2:36 PM, Ralph Castain 
> wrote:
Rolf will have to answer the question on level of support. The CUDA code is not 
in the 1.6 series as it was developed after that series went "stable". It is in 
the 1.7 series, although the level of support will likely be incrementally 
increasing as that "feature" series continues to evolve.


On Jul 6, 2013, at 12:06 PM, Michael Thomadakis 
> wrote:

> Hello OpenMPI,
>
> I am wondering what level of support is there for CUDA and GPUdirect on 
> OpenMPI 1.6.5 and 1.7.2.
>
> I saw the ./configure --with-cuda=CUDA_DIR option in the FAQ. However, it 
> seems that with configure v1.6.5 it was ignored.
>
> Can you identify GPU memory and send messages from it directly without 
> copying to host memory first?
>
>
> Or in general, what level of CUDA support is there on 1.6.5 and 1.7.2 ? Do 
> you support SDK 5.0 and above?
>
> Cheers ...
> Michael
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Michael Thomadakis]

Thanks ...
Michael


On Mon, Jul 8, 2013 at 8:50 AM, Rolf vandeVaart wrote:

> With respect to the CUDA-aware support, Ralph is correct.  The ability to
> send and receive GPU buffers is in the Open MPI 1.7 series.  And
> incremental improvements will be added to the Open MPI 1.7 series.  CUDA
> 5.0 is supported.
>
> ** **
>
> ** **
>
> ** **
>
> *From:* users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] *On
> Behalf Of *Ralph Castain
> *Sent:* Saturday, July 06, 2013 5:14 PM
> *To:* Open MPI Users
> *Subject:* Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI
> 1.6.5 an 1.7.2
>
> ** **
>
> There was discussion of this on a prior email thread on the OMPI devel
> mailing list:
>
> ** **
>
> http://www.open-mpi.org/community/lists/devel/2013/05/12354.php
>
> ** **
>
> ** **
>
> On Jul 6, 2013, at 2:01 PM, Michael Thomadakis 
> wrote:
>
>
>
> 
>
> thanks,
>
> Do you guys have any plan to support Intel Phi in the future? That is,
> running MPI code on the Phi cards or across the multicore and Phi, as Intel
> MPI does?
>
> thanks...
>
> Michael
>
> ** **
>
> On Sat, Jul 6, 2013 at 2:36 PM, Ralph Castain  wrote:***
> *
>
> Rolf will have to answer the question on level of support. The CUDA code
> is not in the 1.6 series as it was developed after that series went
> "stable". It is in the 1.7 series, although the level of support will
> likely be incrementally increasing as that "feature" series continues to
> evolve.
>
>
>
> On Jul 6, 2013, at 12:06 PM, Michael Thomadakis 
> wrote:
>
> > Hello OpenMPI,
> >
> > I am wondering what level of support is there for CUDA and GPUdirect on
> OpenMPI 1.6.5 and 1.7.2.
> >
> > I saw the ./configure --with-cuda=CUDA_DIR option in the FAQ. However,
> it seems that with configure v1.6.5 it was ignored.
> >
> > Can you identify GPU memory and send messages from it directly without
> copying to host memory first?
> >
> >
> > Or in general, what level of CUDA support is there on 1.6.5 and 1.7.2 ?
> Do you support SDK 5.0 and above?
> >
> > Cheers ...
> > Michael
>
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> ** **
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> ** **
>  --
>  This email message is for the sole use of the intended recipient(s) and
> may contain confidential information.  Any unauthorized review, use,
> disclosure or distribution is prohibited.  If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
>  --
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Rolf vandeVaart]

With respect to the CUDA-aware support, Ralph is correct.  The ability to send 
and receive GPU buffers is in the Open MPI 1.7 series.  And incremental 
improvements will be added to the Open MPI 1.7 series.  CUDA 5.0 is supported.



From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On Behalf 
Of Ralph Castain
Sent: Saturday, July 06, 2013 5:14 PM
To: Open MPI Users
Subject: Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 
1.7.2

There was discussion of this on a prior email thread on the OMPI devel mailing 
list:

http://www.open-mpi.org/community/lists/devel/2013/05/12354.php


On Jul 6, 2013, at 2:01 PM, Michael Thomadakis 
> wrote:


thanks,
Do you guys have any plan to support Intel Phi in the future? That is, running 
MPI code on the Phi cards or across the multicore and Phi, as Intel MPI does?
thanks...
Michael

On Sat, Jul 6, 2013 at 2:36 PM, Ralph Castain 
> wrote:
Rolf will have to answer the question on level of support. The CUDA code is not 
in the 1.6 series as it was developed after that series went "stable". It is in 
the 1.7 series, although the level of support will likely be incrementally 
increasing as that "feature" series continues to evolve.


On Jul 6, 2013, at 12:06 PM, Michael Thomadakis 
> wrote:

> Hello OpenMPI,
>
> I am wondering what level of support is there for CUDA and GPUdirect on 
> OpenMPI 1.6.5 and 1.7.2.
>
> I saw the ./configure --with-cuda=CUDA_DIR option in the FAQ. However, it 
> seems that with configure v1.6.5 it was ignored.
>
> Can you identify GPU memory and send messages from it directly without 
> copying to host memory first?
>
>
> Or in general, what level of CUDA support is there on 1.6.5 and 1.7.2 ? Do 
> you support SDK 5.0 and above?
>
> Cheers ...
> Michael
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


---
This email message is for the sole use of the intended recipient(s) and may 
contain
confidential information.  Any unauthorized review, use, disclosure or 
distribution
is prohibited.  If you are not the intended recipient, please contact the 
sender by
reply email and destroy all copies of the original message.
---

Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Ralph Castain]

There was discussion of this on a prior email thread on the OMPI devel mailing 
list:

http://www.open-mpi.org/community/lists/devel/2013/05/12354.php


On Jul 6, 2013, at 2:01 PM, Michael Thomadakis  wrote:

> thanks,
> 
> Do you guys have any plan to support Intel Phi in the future? That is, 
> running MPI code on the Phi cards or across the multicore and Phi, as Intel 
> MPI does?
> 
> thanks...
> Michael
> 
> 
> On Sat, Jul 6, 2013 at 2:36 PM, Ralph Castain  wrote:
> Rolf will have to answer the question on level of support. The CUDA code is 
> not in the 1.6 series as it was developed after that series went "stable". It 
> is in the 1.7 series, although the level of support will likely be 
> incrementally increasing as that "feature" series continues to evolve.
> 
> 
> On Jul 6, 2013, at 12:06 PM, Michael Thomadakis  
> wrote:
> 
> > Hello OpenMPI,
> >
> > I am wondering what level of support is there for CUDA and GPUdirect on 
> > OpenMPI 1.6.5 and 1.7.2.
> >
> > I saw the ./configure --with-cuda=CUDA_DIR option in the FAQ. However, it 
> > seems that with configure v1.6.5 it was ignored.
> >
> > Can you identify GPU memory and send messages from it directly without 
> > copying to host memory first?
> >
> >
> > Or in general, what level of CUDA support is there on 1.6.5 and 1.7.2 ? Do 
> > you support SDK 5.0 and above?
> >
> > Cheers ...
> > Michael
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Michael Thomadakis]

thanks,

Do you guys have any plan to support Intel Phi in the future? That is,
running MPI code on the Phi cards or across the multicore and Phi, as Intel
MPI does?

thanks...
Michael


On Sat, Jul 6, 2013 at 2:36 PM, Ralph Castain  wrote:

> Rolf will have to answer the question on level of support. The CUDA code
> is not in the 1.6 series as it was developed after that series went
> "stable". It is in the 1.7 series, although the level of support will
> likely be incrementally increasing as that "feature" series continues to
> evolve.
>
>
> On Jul 6, 2013, at 12:06 PM, Michael Thomadakis 
> wrote:
>
> > Hello OpenMPI,
> >
> > I am wondering what level of support is there for CUDA and GPUdirect on
> OpenMPI 1.6.5 and 1.7.2.
> >
> > I saw the ./configure --with-cuda=CUDA_DIR option in the FAQ. However,
> it seems that with configure v1.6.5 it was ignored.
> >
> > Can you identify GPU memory and send messages from it directly without
> copying to host memory first?
> >
> >
> > Or in general, what level of CUDA support is there on 1.6.5 and 1.7.2 ?
> Do you support SDK 5.0 and above?
> >
> > Cheers ...
> > Michael
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Ralph Castain]

Rolf will have to answer the question on level of support. The CUDA code is not 
in the 1.6 series as it was developed after that series went "stable". It is in 
the 1.7 series, although the level of support will likely be incrementally 
increasing as that "feature" series continues to evolve.


On Jul 6, 2013, at 12:06 PM, Michael Thomadakis  
wrote:

> Hello OpenMPI,
> 
> I am wondering what level of support is there for CUDA and GPUdirect on 
> OpenMPI 1.6.5 and 1.7.2.
> 
> I saw the ./configure --with-cuda=CUDA_DIR option in the FAQ. However, it 
> seems that with configure v1.6.5 it was ignored.
> 
> Can you identify GPU memory and send messages from it directly without 
> copying to host memory first? 
> 
> 
> Or in general, what level of CUDA support is there on 1.6.5 and 1.7.2 ? Do 
> you support SDK 5.0 and above?
> 
> Cheers ...
> Michael
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

[OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Michael Thomadakis]

Hello OpenMPI,

I am wondering what level of support is there for CUDA and GPUdirect on
OpenMPI 1.6.5 and 1.7.2.

I saw the ./configure --with-cuda=CUDA_DIR option in the FAQ. However, it
seems that with configure v1.6.5 it was ignored.

Can you identify GPU memory and send messages from it directly without
copying to host memory first?


Or in general, what level of CUDA support is there on 1.6.5 and 1.7.2 ? Do
you support SDK 5.0 and above?

Cheers ...
Michael