The Intel MPI implementation does this. The performance between the accelerators and the host is poor though. About 20mb/sec in my ping/pong test. Intra-MIC communication is about a 1GB/sec whereas intra-host is about 6GB/sec. Latency is higher (i.e. worse) for the intra-MIC communication too (vs. intra-host) by about the same factor.
Thanks Tim for the hints on building OpenMPI. ________________________________________ From: users-boun...@open-mpi.org [users-boun...@open-mpi.org] on behalf of Ralph Castain [r...@open-mpi.org] Sent: Tuesday, July 09, 2013 12:14 PM To: Tim Carlson; Open MPI Users Subject: Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2 Hi Tim Quick question: can the procs on the MIC communicate with procs on (a) the local host, (b) other hosts, and (c) MICs on other hosts? The last two would depend on having direct access to one or more network transports. On Jul 9, 2013, at 10:18 AM, Tim Carlson <tim.carl...@pnl.gov> wrote: > On Mon, 8 Jul 2013, Tim Carlson wrote: > > Now that I have gone through this process, I'll report that it works with the > caveat that you can't use the openmpi wrappers for compiling. Recall that the > Phi card does not have either the GNU or Intel compilers installed. While you > could build up a tool chain for the GNU compilers, you're not going to get a > native Intel compiler unless Intel decides to support it. > > Here is the process from end to end to get Openmpi to build a native Phi > application. > > export PATH=/usr/linux-k1om-4.7/bin:$PATH > . /share/apps/intel/composer_xe_2013.3.163/bin/iccvars.sh intel64 > export CC="icc -mmic" > export CXX="icpc -mmic" > > cd ~ > tar zxf openmpi-1.6.4.tar.gz > cd openmpi-1.6.4 > ./configure --prefix=/people/tim/mic/openmpi/intel \ > --build=x86_64-unknown-linux-gnu --host=x86_64-k1om-linux \ > --disable-mpi-f77 \ > AR=x86_64-k1om-linux-ar RANLIB=x86_64-k1om-linux-ranlib > LD=x86_64-k1om-linux-ld > make > make install > > That leaves me with a native build of openmpi in /people/tim/mic/openmpi/intel > > It is of course tempting to just do a > export PATH=/people/tim/mic/openmpi/intel/bin:$PATH > and start using mpicc to build my code but that does not work because: > > 1) If I try this on the host system I am going to get "wrong architecture" > because mpicc was build for the Phi and not for the x86_64 host > > 2) If I try running it on the Phi, I don't have access to "icc" because I > can't run the compiler directly on the Phi. > > I can "cheat" and see what the mpicc command really does by using "mpicc > --show" for another installation of openmpi and munge the paths correctly. In > this case > > icc -mmic cpi.c -I/people/tim/mic/openmpi/intel/include -pthread \ > -L/people/tim/mic/openmpi/intel/lib -lmpi -ldl -lm -Wl,--export-dynamic \ > -lrt -lnsl -lutil -lm -ldl -o cpi.x > > That leaves me with a Phi native version of cpi.x which I can then execute on > the Phi > > $ ssh phi002-mic0 > > ( I have NFS mounts on the Phi for all the bits I need ) > > ~ $ export PATH=/people/tim/mic/openmpi/intel/bin/:$PATH > ~ $ export > LD_LIBRARY_PATH=/share/apps/intel/composer_xe_2013.3.163/compiler/lib/mic/ > ~ $ export LD_LIBRARY_PATH=/people/tim/mic/openmpi/intel/lib:$LD_LIBRARY_PATH > ~ $ cd mic > ~/mic $ mpirun -np 12 cpi.x > Process 7 on phi002-mic0.local > Process 10 on phi002-mic0.local > Process 2 on phi002-mic0.local > Process 9 on phi002-mic0.local > Process 1 on phi002-mic0.local > Process 3 on phi002-mic0.local > Process 11 on phi002-mic0.local > Process 5 on phi002-mic0.local > Process 8 on phi002-mic0.local > Process 4 on phi002-mic0.local > Process 6 on phi002-mic0.local > Process 0 on phi002-mic0.local > pi is approximately 3.1416009869231245, Error is 0.0000083333333314 > wall clock time = 0.001766 > > >> On Mon, 8 Jul 2013, Elken, Tom wrote: >> >> My mistake on the OFED bits. The host I was installing on did not have all >> of the MPSS software installed (my cluster admin node and not one of the >> compute nodes). Adding the intel-mic-ofed-card RPM fixed the problem with >> compiling the btl:openib bits with both the GNU and Intel compilers using >> the cross-compiler route (-mmic on the Intel side) >> >> Still working on getting the resulting mpicc wrapper working on the MIC >> side. When I get a working example I'll post the results. >> >> Thanks! >> >> Tim >> >> >>> >>> >>> >>> Hi Tim, >>> >>> >>> >>> >>> >>> Well, in general and not on MIC I usually build the MPI stacks using the >>> Intel compiler set. Have you ran into s/w that requires GCC instead of >>> Intel >>> compilers (beside Nvidia Cuda)? Did you try to use Intel compiler to >>> produce >>> MIC native code (the OpenMPI stack for that matter)? >>> >>> [Tom] >>> >>> Good idea Michael, With the Intel Compiler, I would use the -mmic flag to >>> build MIC code. >>> >>> >>> >>> Tim wrote: “My first pass at doing a cross-compile with the GNU compilers >>> failed to produce something with OFED support (not surprising) >>> >>> export PATH=/usr/linux-k1om-4.7/bin:$PATH >>> ./configure --build=x86_64-unknown-linux-gnu --host=x86_64-k1om-linux \ >>> --disable-mpi-f77 >>> >>> checking if MCA component btl:openib can compile... no >>> >>> >>> >>> Regarding a Gnu cross compiler, this worked for one of our engineers >>> building for True Scale HCAs and PSM/infinipath. But it may give useful >>> tips for building for btl:openib as well: >>> >>> >>> >>> 3. How to configure/compile OpenMPI: >>> >>> a). untar the openmpi tarball. >>> >>> b). edit configure in top directory, add '-linfinipath' after >>> '-lpsm_infinipath " >>> >>> but not necessary for messages, only for command lines. >>> >>> >>> >>> c). run the following script, >>> >>> #!/bin/sh >>> >>> >>> >>> ./configure \ >>> >>> --host=x86_64-k1om-linux \ >>> >>> --enable-mpi-f77=no --enable-mpi-f90=no \ >>> >>> --with-psm=/…/psm-7.6 \ >>> >>> --prefix=/…/openmpi \ >>> >>> CC=x86_64-k1om-linux-gcc CXX=x86_64-k1om-linux-g++ \ >>> >>> AR=x86_64-k1om-linux-ar RANLIB=x86_64-k1om-linux-ranlib >>> >>> >>> >>> d). run 'make' and 'make install' >>> >>> >>> >>> OK, I see that they did not configure for mpi-f77 & mpif90, but perhaps >>> this >>> is still helpful, if the AR and RANLIB flags are important. >>> >>> -Tom >>> >>> >>> >>> >>> >>> >>> >>> regards >>> >>> Michael >>> >>> >>> >>> On Mon, Jul 8, 2013 at 4:30 PM, Tim Carlson <tim.carl...@pnl.gov> wrote: >>> >>> On Mon, 8 Jul 2013, Elken, Tom wrote: >>> >>> It isn't quite so easy. >>> >>> Out of the box, there is no gcc on the Phi card. You can use the cross >>> compiler on the host, but you don't get gcc on the Phi by default. >>> >>> See this post http://software.intel.com/en-us/forums/topic/382057 >>> >>> I really think you would need to build and install gcc on the Phi first. >>> >>> My first pass at doing a cross-compile with the GNU compilers failed to >>> produce something with OFED support (not surprising) >>> >>> export PATH=/usr/linux-k1om-4.7/bin:$PATH >>> ./configure --build=x86_64-unknown-linux-gnu --host=x86_64-k1om-linux \ >>> --disable-mpi-f77 >>> >>> checking if MCA component btl:openib can compile... no >>> >>> >>> Tim >>> >>> >>> >>> >>> >>> Thanks Tom, that sounds good. I will give it a try as soon as >>> our Phi host >>> here host gets installed. >>> >>> >>> >>> I assume that all the prerequisite libs and bins on the Phi side >>> are >>> available when we download the Phi s/w stack from Intel's site, >>> right ? >>> >>> [Tom] >>> >>> Right. When you install Intel’s MPSS (Manycore Platform >>> Software Stack), >>> including following the section on “OFED Support” in the readme >>> file, you >>> should have all the prerequisite libs and bins. Note that I >>> have not built >>> Open MPI for Xeon Phi for your interconnect, but it seems to me >>> that it >>> should work. >>> >>> >>> >>> -Tom >>> >>> >>> >>> Cheers >>> >>> Michael >>> >>> >>> >>> >>> >>> >>> >>> On Mon, Jul 8, 2013 at 12:10 PM, Elken, Tom >>> <tom.el...@intel.com> wrote: >>> >>> Do you guys have any plan to support Intel Phi in the future? >>> That is, >>> running MPI code on the Phi cards or across the multicore and >>> Phi, as Intel >>> MPI does? >>> >>> [Tom] >>> >>> Hi Michael, >>> >>> Because a Xeon Phi card acts a lot like a Linux host with an x86 >>> architecture, you can build your own Open MPI libraries to serve >>> this >>> purpose. >>> >>> Our team has used existing (an older 1.4.3 version of) Open MPI >>> source to >>> build an Open MPI for running MPI code on Intel Xeon Phi cards >>> over Intel’s >>> (formerly QLogic’s) True Scale InfiniBand fabric, and it works >>> quite well. >>> We have not released a pre-built Open MPI as part of any Intel >>> software >>> release. But I think if you have a compiler for Xeon Phi >>> (Intel Compiler >>> or GCC) and an interconnect for it, you should be able to build >>> an Open MPI >>> that works on Xeon Phi. >>> >>> Cheers, >>> Tom Elken >>> >>> thanks... >>> >>> Michael >>> >>> >>> >>> On Sat, Jul 6, 2013 at 2:36 PM, Ralph Castain <r...@open-mpi.org> >>> wrote: >>> >>> Rolf will have to answer the question on level of support. The >>> CUDA code is >>> not in the 1.6 series as it was developed after that series went >>> "stable". >>> It is in the 1.7 series, although the level of support will >>> likely be >>> incrementally increasing as that "feature" series continues to >>> evolve. >>> >>> >>> On Jul 6, 2013, at 12:06 PM, Michael Thomadakis >>> <drmichaelt7...@gmail.com> >>> wrote: >>> > Hello OpenMPI, >>> > > I am wondering what level of support is there for CUDA and >>> GPUdirect on >>> OpenMPI 1.6.5 and 1.7.2. >>> > > I saw the ./configure --with-cuda=CUDA_DIR option in the FAQ. >>> However, it >>> seems that with configure v1.6.5 it was ignored. >>> > > Can you identify GPU memory and send messages from it directly >>> without >>> copying to host memory first? >>> > > > Or in general, what level of CUDA support is there on 1.6.5 >>> and 1.7.2 ? Do >>> you support SDK 5.0 and above? >>> > > Cheers ... >>> > Michael >>> > _______________________________________________ >>> > users mailing list >>> > us...@open-mpi.org >>> > http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >>> >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >>> >>> >>> >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >>> >>> >> >> > > -- > ------------------------------------------- > Tim Carlson, PhD > Senior Research Scientist > Environmental Molecular Sciences > Laboratory_______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users _______________________________________________ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
