Ok. Native ibverbs/openib is preferable although cannot be used by all applications (those who do not have a native ip interface). I suppose that if I configure my network nodes to use ipoib (by simply probing ib_ipoib module) I'd still be able to use native ibverbs interface without any delay caused by ipoib on it. And by this way other applications which aren't able to use native ibverbs could use the infiniband network as well. It should be the reason why some people use ipoib I believe, just to offer infiniband to all network applications. The main reason I've asked this question is that I have seen lots of references in the net on how to configure an infiniband network with ipoib, but I was not able to see much references on doing the same without ipoib. This made me believe configuring infiniband with ipoib could be a popular option. I've tried ibdiagnet and other ofed tools. I also tried to debug the network environment with simple jobs to measure bandwidth and latency. In most cases, I've seen high peaks of measures who come and go without any reason I could catch for now. I believe I should check the network configuration and make some tests on it. Does anybody here know some reference about configuring inifiband without ipoib and/or the issues raised when doing this ? If possible, I'd like to see ways of testing the configuration, or know about options that could increase fault tolerance. I know this is some kind of basic, but I am not a well experienced user on infiniband.
2011/3/14 <users-requ...@open-mpi.org>: > > Please see my comment below. > > Pavel (Pasha) Shamis > --- > Application Performance Tools Group > Computer Science and Math Division > Oak Ridge National Laboratory > > > On Mar 11, 2011, at 2:47 PM, Bernardo F Costa wrote: > >> I have found this thread form two years ago. I am some kind of lost on >> configuring an infiniband cluster for openmpi. What is best: use iboip >> or use native infiniband ibverbs interface ? For now I am using native > > Native openib/verbs interface will work much faster (up X10) then ipoib. > Ipoib was designed for application that does not have native ip interface/ > >> infiniband withou ipoib. But I have lots of problems specially with >> latency in the cluster. > > If you see latency problems over native interface (verbs), then you > apparently will > face the same problem with any other application over verbs, including ipoib. > > So using ipoib instead of verbs definitely not a work around for you. > > I would suggest you to run IB network debug tools , like ibdiagnet , in order > to analyze > your network/latency problems. > > Regards, > Pasha
