On Fri, Jun 05, 2009 at 09:52:39AM -0400, Jeff Squyres wrote:
>
> See this FAQ entry for a description:
>
> http://www.open-mpi.org/faq/?category=openfabrics#ofa-port-wireup
>
> Right now, there's no way to force a particular connection pattern on
> the openib btl at run-time. The startup sequence has gotten
> sufficiently complicated / muddied over the years that it would be quite
> difficult to do so. Pasha is in the middle of revamping parts of the
> openib startup (see http://bitbucket.org/pasha/ompi-ofacm/); it *may* be
> desirable to fully clean up the full openib btl startup sequence when
> he's all finished.
>
>
> On Jun 5, 2009, at 9:48 AM, Mouhamed Gueye wrote:
>
>> Hi all,
>>
>> I am working on multi-rail IB and I was wondering how connections are
>> established between ports. I have two hosts, each with 2 ports on a
>> same IB card, connected to the same switch.
>>
Is there a goal in mind?
In general multi-rail cards run into bandwidth and congestion issues
with the host bus. If your card's system side interface cannot support
the bandwidth of twin IB links then it is possible that bandwidth would
be reduced by the interaction.
If the host bus and memory system is fast enough then
work with the vendor.
In addition to system bandwidth the subnet manager may need to be enhanced
to be multi-port card aware. Since IB fabric routes are static it is possible
to route or use pairs of links in an identical enough way that there is
little bandwidth gain when multiple switches are involved.
Your two host case case may be simple enough....to explore
and/or generate illuminating or misleading results.
It is a good place to start.
Start with a look at opensm and the fabric then watch how Open MPI
or your applications use the resulting LIDs. If you are using IB directly
and not MPI then the list of protocol choices grows dramatically but still
centers on LIDs as assigned by the subnet manager (see opensm).
How man CPU cores (ranks) are you working with?
Do be specific about the IB hardware and associated firmware....
there are multiple choices out there and the vendor may be able to help.......
--
T o m M i t c h e l l
Found me a new hat, now what?
