Re: [ewg] rdma/cm: revert associating an RDMA device when binding to loopback
Open MPI also now checks for 127.0.0.1/8 and skips them. This behavior will be included in the upcoming Open MPI v1.4.2 (possibly within a few weeks?) and Open MPI v1.5.0. Two followup questions: 1. Is this now the recommended way to find all the IP interfaces that support RDMA: - loop over all local IP addresses - if 127.0.0.1/8, skip - try to rdma_bind_addr() - if it succeeds and verbs ptr is != NULL, it's an RDMA device (I believe Steve Wise proposed adding an API function to just return a list of IP addresses of RDMA devices a while back; it was rejected, which is why either we use the try-to-rdma_bind_addr() approach) 2. Before Sean backed out the localhost behavior, when you rdma_addr_bind(127.0.0.1), what did the id-verbs pointer correspond to? On Feb 9, 2010, at 11:15 AM, Pradeep Satyanarayana wrote: Steve Wise wrote: This patch works. It also backports cleanly to ofed-1.5.1/RH5.3. Acked-by: Steve Wise sw...@opengridcomputing.com Steve. Steve, Was this tested against both iWARP and IB? Thanks Pradeep ___ ewg mailing list e...@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [ewg] rdma/cm: revert associating an RDMA device when binding to loopback
On Tue, Feb 09, 2010 at 05:01:21PM -0500, Jeff Squyres wrote: 1. Is this now the recommended way to find all the IP interfaces that support RDMA: - loop over all local IP addresses - if 127.0.0.1/8, skip - try to rdma_bind_addr() - if it succeeds and verbs ptr is != NULL, it's an RDMA device RDMA is not special, it is just like any other IP service. RDMA is supported on loopback. To find the list of RMDA capable IPs you do the rdma_bind_addr test. You then have to transform that list exactly as you would for TCP to get a list of candidate addresses that could be used for remote connection. This means removing loopback, doing someting about link local addresses, and matching as necessary IPs to interfaces, to networks, and to source IPs on the connecting side. It isn't trivial, but it is exactly the same as for TCP. I suppose ideally OMPI would use the same codes for both TCP and RDMACM - and it should have user configurables! It is worth reviewing what the OMPI TCP does and at least checking that the RDMACM hits all the same points. 2. Before Sean backed out the localhost behavior, when you rdma_addr_bind(127.0.0.1), what did the id-verbs pointer correspond to? One of the RDMA verbs devices in the system. The API does not define which one the kernel will select. I think the current patches simply picked the first one. Jason -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [ewg] rdma/cm: revert associating an RDMA device when binding to loopback
1. Is this now the recommended way to find all the IP interfaces that support RDMA: - loop over all local IP addresses - if 127.0.0.1/8, skip Include ipv6 in the checks. - try to rdma_bind_addr() - if it succeeds and verbs ptr is != NULL, it's an RDMA device The intent is that rdma_bind_addr should bind to an RDMA device as long as the address isn't a wildcard, so that resource can be allocated. If an address is passed into rdma_bind_addr and the call returns success, then verbs ptr 'should' be set. (This is the bug that was fixed, then reverted.) (I believe Steve Wise proposed adding an API function to just return a list of IP addresses of RDMA devices a while back; it was rejected Not sure why the idea it was rejected. I can't think of a problem with returning a list of devices/ports and the corresponding IP addresses. Although, I'd expect the implementation to match what you have above. At the very least it could reduce code duplication in librdmacm users. 2. Before Sean backed out the localhost behavior, when you rdma_addr_bind(127.0.0.1), what did the id-verbs pointer correspond to? id-verbs referenced the first device found with an active port, or the first device if no ports were active. This behavior was never updated when iwarp was added to the stack. Note that this patch reverted the behavior of rdma_bind_addr(127.0.0.1), but other new functionality was left in. The result is that rdma_connect(127.0.0.1) will still work as long as the selected device can support loopback. - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
