Yeah, we discussed taking things from your thread, plus the wiki page on cross-compiling OMPI, and creating a new FAQ area. I'll do so - thanks!
On Jul 10, 2013, at 9:14 AM, Tim Carlson <tim.carl...@pnl.gov> wrote: > I've polluted the previous thread on GPU abilites with so much Intel/Phi bits > that I decided a few new threads might be a good idea. First off I think the > following could be a FAQ entry. > > If you have cluster with Phi cards and are using the SCIF interface with > OFED, OpenMPI between two hosts (not two Phi cards) is going to choose the > wrong interface at runtime. I'll show this by example. > > On a node that has a Phi card and has ofed-mic enabled, you end up with two > IB interfaces. > > tim@phi001 osu]$ ibv_devices > device node GUID > ------ ---------------- > scif0 4c79bafffe300005 > mlx4_0 003048ffff95f98c > > The scif0 interface is not the one you want to use but it is the one that > shows up first in the list. By default OpenMPI won't even know what to do > with this interface. > > $ mpicc osu_bw.c -o osu_bw.openmpi.x > > $ mpirun -np 2 -hostfile hosts.nodes osu_bw.openmpi.x > -------------------------------------------------------------------------- > WARNING: No preset parameters were found for the device that Open MPI > detected: > > Local host: phi002.local > Device name: scif0 > Device vendor ID: 0x8086 > Device vendor part ID: 0 > > > It completely fails. However if you specify the correct interface (mlx4_0) > you get the expected results. > > $ mpirun -np 2 -hostfile hosts.nodes --mca btl openib,self,sm -mca > btl_openib_if_include mlx4_0 osu_bw.openmpi.x > # OSU MPI Bandwidth Test > # Size Bandwidth (MB/s) > 1 3.25 > 2 6.40 > 4 12.65 > 8 25.53 > 16 50.42 > 32 97.06 > 64 187.02 > 128 357.88 > 256 663.64 > 512 1228.23 > 1024 2142.46 > 2048 3128.06 > 4096 4110.78 > 8192 4870.81 > 16384 5864.45 > 32768 6135.67 > 65536 6264.35 > 131072 6307.70 > 262144 6340.24 > 524288 6329.59 > 1048576 6343.71 > 2097152 6315.45 > 4194304 6322.65 > > Tim > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
