As long as the IB interfaces can communicate to each other, you should be fine.
On Jul 5, 2013, at 3:26 PM, Michael Thomadakis <drmichaelt7...@gmail.com> wrote: > Sorry on the mvapich2 reference :) > > All nodes are attached over a common 1GigE network. We wish ofcourse that if > a node-pair is connected via a higher-speed fabric as well (IB FDR or 10GigE) > then that this would be leveraged instead of the common 1GigE. > > One question: suppose that we use nodes having either FDR or QDR IB > interfaces available, connected to one common IB fabric, all defined over a > common IP subnet: Will OpenMPI have any problem with this? Can MPI > communication take place over this type of hybrid IB fabric? We already have > a sub-cluster with QDR HCAs and we are attaching it to IB fabric with FDR > "backbone" and another cluster with FDR HCAs. > > Do you think there may be some issue with this? The HCAs are FDR and QDR > Mellanox devices and the switching is also over FDR Mellanox fabric. Mellanox > claims that at the IB level this is doable (i.e., FDR link pairs talk to each > other at FDR speeds and QDR link pairs at QDR). > > I guess if we use the RC connection types then it does not matter to OpenMPI. > > thanks .... > Michael > > > > > On Fri, Jul 5, 2013 at 4:59 PM, Ralph Castain <r...@open-mpi.org> wrote: > I can't speak for MVAPICH - you probably need to ask them about this > scenario. OMPI will automatically select whatever available transport that > can reach the intended process. This requires that each communicating pair of > processes have access to at least one common transport. > > So if a process that is on a node with only 1G-E wants to communicate with > another process, then the node where that other process is running must also > have access to a compatible Ethernet interface (1G can talk to 10G, so they > can have different capabilities) on that subnet (or on a subnet that knows > how to route to the other one). If both nodes have 10G-E as well as 1G-E > interfaces, then OMPI will automatically take the 10G interface as it is the > faster of the two. > > Note this means that if a process is on a node that only has IB, and wants to > communicate to a process on a node that only has 1G-E, then the two processes > cannot communicate. > > HTH > Ralph > > On Jul 5, 2013, at 2:34 PM, Michael Thomadakis <drmichaelt7...@gmail.com> > wrote: > >> Hello OpenMPI >> >> We area seriously considering deploying OpenMPI 1.6.5 for production (and >> 1.7.2 for testing) on HPC clusters which consists of nodes with different >> types of networking interfaces. >> >> >> 1) Interface selection >> >> We are using OpenMPI 1.6.5 and was wondering how one would go about >> selecting at run time which networking interface to use for MPI >> communications in case that both IB, 10GigE and 1 GigE are present. >> >> This issues arises in a cluster with nodes that are equipped with different >> types of interfaces: >> >> Some have both IB-QDR or FDR and 10- and 1-GigE. Others only have 10-GigE >> and 1-GigE and simply others only 1-GigE. >> >> >> 2) OpenMPI 1.6.5 level of support for Heterogeneous Fabric >> >> Can OpenMPI support running an MPI application using a mix of nodes with all >> of the above networking interface combinations ? >> >> 2.a) Can the same MPI code (SPMD or MPMD) have a subset of its ranks run >> on nodes with QDR IB and another subset on FDR IB simultaneously? These are >> Mellanox QDR and FDR HCAs. >> >> Mellanox mentioned to us that they support both QDR and FDR HCAs attached to >> the same IB subnet. Do you think MVAPICH2 will have any issue with this? >> >> 2.b) Can the same MPI code (SPMD or MPMD) have a subset of its ranks run on >> nodes with IB and another subset over 10GiGE simultaneously? >> >> That is imagine nodes I1, I2, ..., IN having say QDR HCAs and nodes G1, G2, >> GM having only 10GigE interfaces. Could we have the same MPI application run >> across both types of nodes? >> >> Or should there be say 2 communicators with one of them explicitly overlaid >> on a IB only subnet and the other on a 10GigE only subnet? >> >> >> Please let me know if the above are not very clear. >> >> Thank you much >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
