Jeff Squyres wrote:
The Interop Working Group (IWG) of the OpenFabrics Alliance asked me
to bring a question to the Open MPI user and developer communities: is
anyone interested in having a single MPI job span HCAs or RNICs from
multiple vendors? (pardon the cross-posting, but I did want to ask
each group separately -- because the answers may be different)
The interop testing lab at the University of New Hampshire
(http://www.iol.unh.edu/services/testing/ofa/) discovered that most
(all?) MPI implementations fail when having a single MPI job span HCAs
from multiple vendors and/or span RNICs from multiple vendors. I
don't remember the exact details (and they may not be public, anyway),
but I'm pretty sure that OMPI failed when used with QLogic and
Mellanox HCAs in a single MPI job. This is fairly unsurprising, given
how we tune Open MPI's use of OpenFabrics-capable hardware based on an
internal Open MPI .ini file.
So my question is: does anyone want/need to support jobs that span
HCAs from multiple vendors and/or RNICs from multiple vendors?
I think the interop problems could easily arise with multiple models of
HCA from the same vendor due to the OMPI practice Jeff mentions of
tuning the HCAs differently according to model.
There are institutions who built clusters with many node configurations
over time, the Sandia CPlant comes to mind as perhaps the best known
example, but I have seen others.
In such a system, new machines are added periodically and old ones
removed. However, at any given time a mixture of ages and
configurations will exist. In such a system there is NOT always the
expectation/requirement that an MPI job run across all (or even
multiple) node types, but such a capability is valuable if it can be
maintained. So, while I personally don't have the want/need to support
a single MPI job spanning multiple HCA models/vendors, I can make a case
for the theoretical value.
-Paul
--
Paul H. Hargrove phhargr...@lbl.gov
Future Technologies Group Tel: +1-510-495-2352
HPC Research Department Fax: +1-510-486-6900
Lawrence Berkeley National Laboratory
