Re: bug 1918 - openmpi broken due to rdma-cm changes
Sean Hefty wrote: There is still some inconsistency here. Sean, you claimed binds to 127.0.0.1 succeed in ofed-1.4 for IB devices. If so, then folks running IB/openmpi/rdmacm should be seeing issues. We need to dig a little more... You can verify this by running ucmatose -b 127.0.0.1 and see if the test enters the listening state. Can you also try testing iwarp with the patch that I sent? I backported your patch to ofed-1.5.1 and tried it, and apparently binds to 127.0.0.1 are still working even though the only device in the system is iWARP. I'm debugging now. Steve. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
rdma/cm: disallow loopback address for iwarp devices
From: Sean Hefty sean.he...@intel.com
The current RDMA iWarp devices cannot be used to establish
connections using the loopback address. Prevent rdma_bind_addr
from associating the loopback address with an iWarp device.
This fixes an issue with openmpi, where it tries to identify which
IP addresses map to RDMA devices by calling rdma_bind_addr on
each address and seeing if the bind succeeds. Prior to patch
6f8372b6 RDMA/cm: fix loopback address support, this process
worked. But the rdma_cm now allows rdma_bind_addr to bind to an
RDMA device using the loopback address, and attaches the rdma_cm_id
to the RDMA device as part of the bind.
Signed-off-by: Sean Hefty sean.he...@intel.com
---
drivers/infiniband/core/cma.c | 14 ++
1 files changed, 10 insertions(+), 4 deletions(-)
diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index cc9b594..5850411 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -1739,6 +1739,9 @@ err:
}
EXPORT_SYMBOL(rdma_resolve_route);
+/*
+ * Only IB devices support loopback connections.
+ */
static int cma_bind_loopback(struct rdma_id_private *id_priv)
{
struct cma_device *cma_dev;
@@ -1753,11 +1756,16 @@ static int cma_bind_loopback(struct rdma_id_private
*id_priv)
ret = -ENODEV;
goto out;
}
- list_for_each_entry(cma_dev, dev_list, list)
+ list_for_each_entry(cma_dev, dev_list, list) {
+ if (rdma_node_get_transport(cma_dev-device-node_type) !=
+ RDMA_TRANSPORT_IB)
+ continue;
+
for (p = 1; p = cma_dev-device-phys_port_cnt; ++p)
if (!ib_query_port(cma_dev-device, p, port_attr)
port_attr.state == IB_PORT_ACTIVE)
goto port_found;
+ }
Here you need to:
ret = -ENODEV;
goto out;
instead of:
p = 1;
cma_dev = list_entry(dev_list.next, struct cma_device, list);
Otherwise it will still bind to the first device even if its iwarp...
With this mod, it works.
@@ -1771,9 +1779,7 @@ port_found:
if (ret)
goto out;
- id_priv-id.route.addr.dev_addr.dev_type =
- (rdma_node_get_transport(cma_dev-device-node_type) ==
RDMA_TRANSPORT_IB) ?
- ARPHRD_INFINIBAND : ARPHRD_ETHER;
+ id_priv-id.route.addr.dev_addr.dev_type = ARPHRD_INFINIBAND;
rdma_addr_set_sgid(id_priv-id.route.addr.dev_addr, gid);
ib_addr_set_pkey(id_priv-id.route.addr.dev_addr, pkey);
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
Note, even though this patch resolved the openmpi failure on my iwarp
nodes, ucmatose -b 127.0.0.1 doesn't fail. I haven't looked at the src,
but something funny must be happening.
So we still have a regression issue with ofed-1.5.1/upstream kernels and
openmpi over IB with rdmacm.
Steve.
Steve Wise wrote:
rdma/cm: disallow loopback address for iwarp devices
From: Sean Hefty sean.he...@intel.com
The current RDMA iWarp devices cannot be used to establish
connections using the loopback address. Prevent rdma_bind_addr
from associating the loopback address with an iWarp device.
This fixes an issue with openmpi, where it tries to identify which
IP addresses map to RDMA devices by calling rdma_bind_addr on
each address and seeing if the bind succeeds. Prior to patch
6f8372b6 RDMA/cm: fix loopback address support, this process
worked. But the rdma_cm now allows rdma_bind_addr to bind to an
RDMA device using the loopback address, and attaches the rdma_cm_id
to the RDMA device as part of the bind.
Signed-off-by: Sean Hefty sean.he...@intel.com
---
drivers/infiniband/core/cma.c | 14 ++
1 files changed, 10 insertions(+), 4 deletions(-)
diff --git a/drivers/infiniband/core/cma.c
b/drivers/infiniband/core/cma.c
index cc9b594..5850411 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -1739,6 +1739,9 @@ err:
}
EXPORT_SYMBOL(rdma_resolve_route);
+/*
+ * Only IB devices support loopback connections.
+ */
static int cma_bind_loopback(struct rdma_id_private *id_priv)
{
struct cma_device *cma_dev;
@@ -1753,11 +1756,16 @@ static int cma_bind_loopback(struct
rdma_id_private *id_priv)
ret = -ENODEV;
goto out;
}
-list_for_each_entry(cma_dev, dev_list, list)
+list_for_each_entry(cma_dev, dev_list, list) {
+if (rdma_node_get_transport(cma_dev-device-node_type) !=
+RDMA_TRANSPORT_IB)
+continue;
+
for (p = 1; p = cma_dev-device-phys_port_cnt; ++p)
if (!ib_query_port(cma_dev-device, p, port_attr)
port_attr.state == IB_PORT_ACTIVE)
goto port_found;
+}
Here you need to:
ret = -ENODEV;
goto out;
instead of:
p = 1;
cma_dev = list_entry(dev_list.next, struct cma_device, list);
Otherwise it will still bind to the first device even if its iwarp...
With this mod, it works.
@@ -1771,9 +1779,7 @@ port_found:
if (ret)
goto out;
-id_priv-id.route.addr.dev_addr.dev_type =
-(rdma_node_get_transport(cma_dev-device-node_type) ==
RDMA_TRANSPORT_IB) ?
-ARPHRD_INFINIBAND : ARPHRD_ETHER;
+id_priv-id.route.addr.dev_addr.dev_type = ARPHRD_INFINIBAND;
rdma_addr_set_sgid(id_priv-id.route.addr.dev_addr, gid);
ib_addr_set_pkey(id_priv-id.route.addr.dev_addr, pkey);
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: bug 1918 - openmpi broken due to rdma-cm changes
-list_for_each_entry(cma_dev, dev_list, list)
+list_for_each_entry(cma_dev, dev_list, list) {
+if (rdma_node_get_transport(cma_dev-device-node_type) !=
+RDMA_TRANSPORT_IB)
+continue;
+
for (p = 1; p = cma_dev-device-phys_port_cnt; ++p)
if (!ib_query_port(cma_dev-device, p, port_attr)
port_attr.state == IB_PORT_ACTIVE)
goto port_found;
+}
Here you need to:
ret = -ENODEV;
goto out;
Good catch, I'll update the patch and submit for 2.6.33 on Monday.
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
Good catch, I'll update the patch and submit for 2.6.33 on Monday. NOTE: This doesn't solve our IB/openmpi regression for ofed-1.5.1. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
Jeff Squyres (jsquyres) wrote: Note that it is highly unlikely that we will release open mpi 1.4.2 in time for ofed 1.5.1. Jeff, there is no way to handle high priority bug fixes in the current released stream? Also note that trying to bind rdma cm to all interface ip addresses was the way that we were advised by openfabrics to figure out which devices are rdma-capable. As such, it is highly desirable to get the fix transparently in rdmacm and preserve the old semantic. More specifically, it seems undesirable to change this semantic in a minor ofed point release. I agree that we should probably not allow 127.0.0.1 binds in ofed-1.5.1 at all because it regresses OpenMPI. Even with IB systems, if the bind to 127.0.0.1 succeeds, then OpenMPI assumes 127.0.0.1 is bound to that rdma interface and advertises this address to its peer as an address to-which that peer can rdma connect! This will break IB clusters too, not just T3/iWARP cluster. While I think OpenMPI needs to skip 127.0.0.1 in its logic, I think we should probably defer allowing 127.0.0.1 binds until ofed-1.6. But Jeff, note that if someone uses the upstream kernel and OpenMPI, its busted... So I recommend: 1) Don't allow 127.0.0.1 binds in ofed-1.5.1 2) Fix OpenMPI ASAP to never advertise 127.0.0.1 as a valid rdma-cm connect address (get it in ofed-1.5.2 or ofed-1.6). Steve. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: bug 1918 - openmpi broken due to rdma-cm changes
Also note that trying to bind rdma cm to all interface ip addresses was the way that we were advised by openfabrics to figure out which devices are rdma- capable. As such, it is highly desirable to get the fix transparently in rdmacm and preserve the old semantic. More specifically, it seems undesirable to change this semantic in a minor ofed point release. I think the issue is larger than just the rdma_cm. First, it sounds like openmpi tries to bind to 127.0.0.1, which now works. If opemmpi uses shared memory for connections on the same machine, I'm not sure why this is a problem, unless it is passing that address to another machine to use for a connection. If this is the case, then that is a bug in openmpi. Second, I still don't understand whether iwarp is limited to 'loopback' connections that are not bound to 127.0.0.1. For instance, if the RDMA device is associated with 192.168.0.1, then can it handle a connection from 192.168.0.1 - 192.168.0.1? If it can't, then the rdma_cm can't help in this case when bind is called. The failure has to come during connect, which sounds like the behavior that's seen today with 127.0.0.1. So, while the rdma_cm can fail binds to 127.0.0.1 if the RDMA device doesn't support loopback, I'm still not sure how much of a fix this is. - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
Sean Hefty wrote: Also note that trying to bind rdma cm to all interface ip addresses was the way that we were advised by openfabrics to figure out which devices are rdma- capable. As such, it is highly desirable to get the fix transparently in rdmacm and preserve the old semantic. More specifically, it seems undesirable to change this semantic in a minor ofed point release. I think the issue is larger than just the rdma_cm. First, it sounds like openmpi tries to bind to 127.0.0.1, which now works. If opemmpi uses shared memory for connections on the same machine, I'm not sure why this is a problem, unless it is passing that address to another machine to use for a connection. If this is the case, then that is a bug in openmpi. Yes, OpenMPI incorrectly advertises 127.0.0.1 as a valid address to-which the peer can connect. This needs to be fixed. Second, I still don't understand whether iwarp is limited to 'loopback' connections that are not bound to 127.0.0.1. For instance, if the RDMA device is associated with 192.168.0.1, then can it handle a connection from 192.168.0.1 - 192.168.0.1? If it can't, then the rdma_cm can't help in this case when bind is called. The failure has to come during connect, which sounds like the behavior that's seen today with 127.0.0.1. Its not iWARP specific. A device may or may not support hw loopback. Now the IB spec mandates this support, but the iWARP spec doesn't. Ammasso and Chelsio T3 rnics do not support HW loopback. They will fail if you try to connect to a local address. The rdma-cm shouldn't allow binds to 127.0.0.1 for these devices since it 100% implies that the connection will require hw loopback for that device. So, while the rdma_cm can fail binds to 127.0.0.1 if the RDMA device doesn't support loopback, I'm still not sure how much of a fix this is. My concern is breaking an existing working OpenMPI in a point release because we changed semantics of the rdma-cm in an ofed point release... BTW: Was this change an artifact of rebasing ofed-1.5.1 on a new kernel version? Steve. - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
I agree that we should probably not allow 127.0.0.1 binds in ofed-1.5.1 at all because it regresses OpenMPI. Even with IB systems, if the bind to 127.0.0.1 succeeds, then OpenMPI assumes 127.0.0.1 is bound to that rdma interface and advertises this address to its peer as an address to-which that peer can rdma connect! This will break IB clusters too, not just T3/iWARP cluster. While I think OpenMPI needs to skip 127.0.0.1 in its logic, I think we should probably defer allowing 127.0.0.1 binds until ofed-1.6. But Jeff, note that if someone uses the upstream kernel and OpenMPI, its busted... So I recommend: 1) Don't allow 127.0.0.1 binds in ofed-1.5.1 2) Fix OpenMPI ASAP to never advertise 127.0.0.1 as a valid rdma-cm connect address (get it in ofed-1.5.2 or ofed-1.6). Also, there is a good argument for never allowing 127.0.0.1 for rdma anyway. It implies a _software_ loopback. It should NEVER be bound to a real NIC interface and thus rdma binds shouldn't be allowed to it since there is no software rdma loopback support... Unless someone implements software rdma loobpack... ;) -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: bug 1918 - openmpi broken due to rdma-cm changes
My concern is breaking an existing working OpenMPI in a point release because we changed semantics of the rdma-cm in an ofed point release... OFED can call this release a point release, but in reality, the content makes it a major release... BTW: Was this change an artifact of rebasing ofed-1.5.1 on a new kernel version? apparently -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
Sean Hefty wrote: My concern is breaking an existing working OpenMPI in a point release because we changed semantics of the rdma-cm in an ofed point release... OFED can call this release a point release, but in reality, the content makes it a major release... BTW: Was this change an artifact of rebasing ofed-1.5.1 on a new kernel version? apparently Well as it stands now: OpenMPI on ofed-1.5.1 is broken for IB if they use the rdma-cm for connection setup, and all IW clusters which require the rdma-cm connect method. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
But Jeff, note that if someone uses the upstream kernel and OpenMPI, its busted... Is the issue 6f8372b6 (RDMA/cm: fix loopback address support)? This just went in for 2.6.33, which is still at -rc6, so if we can quickly reach a consensus, there is still time to get a fix in for 2.6.33. - R. -- Roland Dreier rola...@cisco.com Cisco.com - http://www.cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/index.html -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
On Feb 5, 2010, at 11:16 AM, Steve Wise wrote: Note that it is highly unlikely that we will release open mpi 1.4.2 in time for ofed 1.5.1. Jeff, there is no way to handle high priority bug fixes in the current released stream? We have 1.4.2 cooking, but it's not ready yet. I'll take it back to the OMPI community to see if they want to do a high-priority release, but I'm not excited about it (see below). Also note that trying to bind rdma cm to all interface ip addresses was the way that we were advised by openfabrics to figure out which devices are rdma-capable. As such, it is highly desirable to get the fix transparently in rdmacm and preserve the old semantic. More specifically, it seems undesirable to change this semantic in a minor ofed point release. I agree that we should probably not allow 127.0.0.1 binds in ofed-1.5.1 at all because it regresses OpenMPI. Even with IB systems, if the bind to 127.0.0.1 succeeds, then OpenMPI assumes 127.0.0.1 is bound to that rdma interface and advertises this address to its peer as an address to-which that peer can rdma connect! This will break IB clusters too, not just T3/iWARP cluster. While I think OpenMPI needs to skip 127.0.0.1 in its logic, I think we should probably defer allowing 127.0.0.1 binds until ofed-1.6. I agree that Open MPI should not advertise 127.0.0.1 to peers. However, the logic that we were advised to use was to try to RDMA CM bind to each IP address. If the bind succeeds, then it's an RDMA-capable device and therefore it's advertisable. The rationale was that 127.0.0.1 (really, any loopback address) is *not* an RDMA device and therefore the RDMA CM bind should *never* succeed on it. Hence, it wasn't necessary to add a is this a loopback address? check in the logic. I guess I don't understand why that rationale is now incorrect -- 127.0.0.1 is still not an RDMA-capable device, right? But Jeff, note that if someone uses the upstream kernel and OpenMPI, its busted... So I recommend: 1) Don't allow 127.0.0.1 binds in ofed-1.5.1 2) Fix OpenMPI ASAP to never advertise 127.0.0.1 as a valid rdma-cm connect address (get it in ofed-1.5.2 or ofed-1.6). We can add this logic (because I understand that some upstream kernels now allow binding to loopback addresses), but I'm still confused (in principle) as to why it should be necessary. Can you clarify what kernel versions allow binding LOOPBACK addresses with RDMA CM? -- Jeff Squyres jsquy...@cisco.com Cisco.com - http://www.cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
On Feb 5, 2010, at 12:51 PM, Roland Dreier (rdreier) wrote: But Jeff, note that if someone uses the upstream kernel and OpenMPI, its busted... Is the issue 6f8372b6 (RDMA/cm: fix loopback address support)? This just went in for 2.6.33, which is still at -rc6, so if we can quickly reach a consensus, there is still time to get a fix in for 2.6.33. Oh oh oh! Yes, that would be fabulous... Thanks! -- Jeff Squyres jsquy...@cisco.com Cisco.com - http://www.cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
Jeff Squyres wrote: On Feb 5, 2010, at 12:51 PM, Roland Dreier (rdreier) wrote: But Jeff, note that if someone uses the upstream kernel and OpenMPI, its busted... Is the issue 6f8372b6 (RDMA/cm: fix loopback address support)? This just went in for 2.6.33, which is still at -rc6, so if we can quickly reach a consensus, there is still time to get a fix in for 2.6.33. Oh oh oh! Yes, that would be fabulous... Thanks! I think we should remove the feature of allowing binds to 127.0.0.1 altogether based on Jeff's arguments and my assertion that 127.0.0.1 is a sw-loopback mechanism anyway... I'm not sure if that commit does more or not... Steve. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: bug 1918 - openmpi broken due to rdma-cm changes
Is the issue 6f8372b6 (RDMA/cm: fix loopback address support)? This just went in for 2.6.33, which is still at -rc6, so if we can quickly reach a consensus, there is still time to get a fix in for 2.6.33. That should be the patch in question. I'm not sure about reaching consensus. :) If the other changes to the rdma_cm aren't closely tied to that change, we may be able to back that one patch out until we can get whatever other fix may be needed. In my view, openmpi has a bug in that it can pass a loopback address to a remote peer and expect it to be used to establish a connection. Steve seems to agree with this. My original intent was to allow the use of the loopback address with the rdma_cm. I.e. 127.0.0.1 meant 'this host', and not 'software loopback'. I just had Arlin run a quick test with OFED 1.4 over IB, and it allows binding to 127.0.0.1, but never forms connections. I.e. ucmatose -b 127.0.0.1 succeeds in listening, but ucmatose -s 127.0.0.1 fails to connect because of a route error. (Hmm... I'm still confused about what openmpi is doing then.) Even if an application were to use non-loopback IP addresses, there's no guarantee of forming a connection if those addresses map to an iwarp device. So, even if the rdma_cm fails binding to 127.0.0.1 unless there's some RDMA device (software or hardware - not sure why we care) capable of supporting it, an application would need to also deal with failures from rdma_resolve_addr. Indicating loopback through a device capability flag seems like the right approach, and the rdma_cm can use this to fail rdma_bind_addr/rdma_resolve_addr calls. That's probably not a trivial patch however. - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
I think we should remove the feature of allowing binds to 127.0.0.1 altogether based on Jeff's arguments and my assertion that 127.0.0.1 is a sw-loopback mechanism anyway... Well, someone propose a patch please. -- Roland Dreier rola...@cisco.com Cisco.com - http://www.cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/index.html -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
On Fri, Feb 05, 2010 at 12:32:51PM -0600, Steve Wise wrote: I think we should remove the feature of allowing binds to 127.0.0.1 altogether based on Jeff's arguments and my assertion that 127.0.0.1 is a sw-loopback mechanism anyway... I don't agree, the kernel should be free to provide a loop back service any way it likes, and if that means using one of the HW adaptors to accelerate the work, then fine. Consider if we see the RDMAoE (soft RDMA) patches then it would be reasonable for all kernels to support RDMA on the loopback. At a minimum, RDMA CM is an IP service, so whatever logic you use to determine addresses for TCP must also be done after determining a list of valid RDMA IPs. Trying to do RDMA CM bind just gives you the list of candidate addreses, no different than netlink does for TCP. One of those steps must be at least filtering 127.0.0.0/8. The user should also be able to have some input into the IP filter - software RDMAoE for instance really make this important. Jason -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
That should be the patch in question. I'm not sure about reaching consensus. :) If the other changes to the rdma_cm aren't closely tied to that change, we may be able to back that one patch out until we can get whatever other fix may be needed. I'd like to do this approach. Then re-submit once we come to consensus... That makes sense to me. Someone please send me a tested revert. -- Roland Dreier rola...@cisco.com Cisco.com - http://www.cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/index.html -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
On Feb 5, 2010, at 1:56 PM, Jason Gunthorpe wrote: I think we should remove the feature of allowing binds to 127.0.0.1 altogether based on Jeff's arguments and my assertion that 127.0.0.1 is a sw-loopback mechanism anyway... I don't agree, the kernel should be free to provide a loop back service any way it likes, and if that means using one of the HW Ok, fine. Should we push back OFED 1.5.1 until Open MPI can get 1.4.2 out? I don't know when that will be. In short: you're breaking backward compatibility with zero warning. There is real software out there that will break if people upgrade their kernel/OFED/RDMA CM/whatever (e.g., Open MPI). Isn't this supposed to be the Enterprise distribution (meaning: stability)? (trying to keep the frustration out of my voice...) This is a terrible, terrible idea. How about this: back out the change for now. Give everyone time to upgrade. If nothing else, ***give those of us who are involved in this community*** time to upgrade. Then put the feature back in after adequate time has passed. -- Jeff Squyres jsquy...@cisco.com Cisco.com - http://www.cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: bug 1918 - openmpi broken due to rdma-cm changes
Ammasso and Chelsio T3 rnics do not support HW loopback.
It looks like the NES driver doesn't support 127.0.0.1, but does support
loopback connections (gurgle). Here's an untested patch for 2.6.33
(not even compile tested) for consideration then. I'll be testing
this shortly unless there's disagreement.
rdma/cm: disallow loopback address for iwarp devices
From: Sean Hefty sean.he...@intel.com
The current RDMA iWarp devices cannot be used to establish
connections using the loopback address. Prevent rdma_bind_addr
from associating the loopback address with an iWarp device.
This fixes an issue with openmpi, where it tries to identify which
IP addresses map to RDMA devices by calling rdma_bind_addr on
each address and seeing if the bind succeeds. Prior to patch
6f8372b6 RDMA/cm: fix loopback address support, this process
worked. But the rdma_cm now allows rdma_bind_addr to bind to an
RDMA device using the loopback address, and attaches the rdma_cm_id
to the RDMA device as part of the bind.
Signed-off-by: Sean Hefty sean.he...@intel.com
---
drivers/infiniband/core/cma.c | 14 ++
1 files changed, 10 insertions(+), 4 deletions(-)
diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index cc9b594..5850411 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -1739,6 +1739,9 @@ err:
}
EXPORT_SYMBOL(rdma_resolve_route);
+/*
+ * Only IB devices support loopback connections.
+ */
static int cma_bind_loopback(struct rdma_id_private *id_priv)
{
struct cma_device *cma_dev;
@@ -1753,11 +1756,16 @@ static int cma_bind_loopback(struct rdma_id_private
*id_priv)
ret = -ENODEV;
goto out;
}
- list_for_each_entry(cma_dev, dev_list, list)
+ list_for_each_entry(cma_dev, dev_list, list) {
+ if (rdma_node_get_transport(cma_dev-device-node_type) !=
+ RDMA_TRANSPORT_IB)
+ continue;
+
for (p = 1; p = cma_dev-device-phys_port_cnt; ++p)
if (!ib_query_port(cma_dev-device, p, port_attr)
port_attr.state == IB_PORT_ACTIVE)
goto port_found;
+ }
p = 1;
cma_dev = list_entry(dev_list.next, struct cma_device, list);
@@ -1771,9 +1779,7 @@ port_found:
if (ret)
goto out;
- id_priv-id.route.addr.dev_addr.dev_type =
- (rdma_node_get_transport(cma_dev-device-node_type) ==
RDMA_TRANSPORT_IB) ?
- ARPHRD_INFINIBAND : ARPHRD_ETHER;
+ id_priv-id.route.addr.dev_addr.dev_type = ARPHRD_INFINIBAND;
rdma_addr_set_sgid(id_priv-id.route.addr.dev_addr, gid);
ib_addr_set_pkey(id_priv-id.route.addr.dev_addr, pkey);
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
On Fri, Feb 05, 2010 at 03:08:10PM -0500, Jeff Squyres wrote: On Feb 5, 2010, at 1:56 PM, Jason Gunthorpe wrote: I think we should remove the feature of allowing binds to 127.0.0.1 altogether based on Jeff's arguments and my assertion that 127.0.0.1 is a sw-loopback mechanism anyway... I don't agree, the kernel should be free to provide a loop back service any way it likes, and if that means using one of the HW Ok, fine. Should we push back OFED 1.5.1 until Open MPI can get 1.4.2 out? I don't know when that will be. In short: you're breaking backward compatibility with zero warning. There is real software out there that will break if people upgrade their kernel/OFED/RDMA CM/whatever (e.g., Open MPI). Isn't this supposed to be the Enterprise distribution (meaning: stability)? (trying to keep the frustration out of my voice...) Well, I think you are right. This kind of change seems appropriate to me for mainline, but OFED/RHEL should carry a responsibility to manage an identified incompatibility, either patch their kernel, patch their OMPI, or publish an errata. That is the role of a distribution. How about this: back out the change for now. Give everyone time to upgrade. If nothing else, ***give those of us who are involved in this community*** time to upgrade. Then put the feature back in after adequate time has passed. I've seen this approach go badly too :( If it isn't actually in a mainline kernel userspace devs tend to ignore it .. Sounds like this is taken care for now anyhow, Sean's patch to remove it for iwarp since it doesn't work today with any iwarp drivers does obscure the problem.. But it does seem like rdma_cm mode for IB networks will still be broken in OMPI with the new kernels. Jason -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
On Feb 5, 2010, at 4:14 PM, Jason Gunthorpe wrote: Well, I think you are right. This kind of change seems appropriate to me for mainline, but OFED/RHEL should carry a responsibility to manage an identified incompatibility, either patch their kernel, patch their OMPI, or publish an errata. That is the role of a distribution. RHEL has said, multiple times, that they rely on OpenFabrics to do the Right Thing. They don't do a lot of testing, validating, etc. Sounds like this is taken care for now anyhow, Sean's patch to remove it for iwarp since it doesn't work today with any iwarp drivers does obscure the problem.. But it does seem like rdma_cm mode for IB networks will still be broken in OMPI with the new kernels. Correct. So why not back off putting this in the kernel that's coming out now now now? Why not put it in *next* kernel? (or even better, the one after that) Is there a rush / need to have this in *now*? -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
Jeff Squyres wrote: On Feb 5, 2010, at 4:14 PM, Jason Gunthorpe wrote: Well, I think you are right. This kind of change seems appropriate to me for mainline, but OFED/RHEL should carry a responsibility to manage an identified incompatibility, either patch their kernel, patch their OMPI, or publish an errata. That is the role of a distribution. RHEL has said, multiple times, that they rely on OpenFabrics to do the Right Thing. They don't do a lot of testing, validating, etc. Sounds like this is taken care for now anyhow, Sean's patch to remove it for iwarp since it doesn't work today with any iwarp drivers does obscure the problem.. But it does seem like rdma_cm mode for IB networks will still be broken in OMPI with the new kernels. Correct. So why not back off putting this in the kernel that's coming out now now now? Why not put it in *next* kernel? (or even better, the one after that) Is there a rush / need to have this in *now*? There is still some inconsistency here. Sean, you claimed binds to 127.0.0.1 succeed in ofed-1.4 for IB devices. If so, then folks running IB/openmpi/rdmacm should be seeing issues. We need to dig a little more... -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: bug 1918 - openmpi broken due to rdma-cm changes
There is still some inconsistency here. Sean, you claimed binds to 127.0.0.1 succeed in ofed-1.4 for IB devices. If so, then folks running IB/openmpi/rdmacm should be seeing issues. We need to dig a little more... You can verify this by running ucmatose -b 127.0.0.1 and see if the test enters the listening state. Can you also try testing iwarp with the patch that I sent? - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
On Feb 5, 2010, at 4:53 PM, Steve Wise wrote: There is still some inconsistency here. Sean, you claimed binds to 127.0.0.1 succeed in ofed-1.4 for IB devices. If so, then folks running IB/openmpi/rdmacm should be seeing issues. We need to dig a little more... FWIW, I can run Open MPI v1.4.2beta on my OFED 1.4.1 cluster over IB devices using RDMA CM with no problems. I added some debug statements in OMPI showing which rdma_cm_bind's it attempts, just to be sure. Here's a run across 2 nodes, each with a single 2-port mthca (each port connected to a different IB subnet, not that that matters): $ mpirun -np 2 --bynode --mca btl_openib_cpc_include rdmacm ring [svbu-mpi025:05592] FAILED to bind to 127.0.0.1 [svbu-mpi025:05592] FAILED to bind to 172.29.218.165 [svbu-mpi025:05592] SUCCEEDED to bind to 10.10.30.165 [svbu-mpi025:05592] SUCCEEDED to bind to 10.10.20.165 [svbu-mpi026:05529] FAILED to bind to 127.0.0.1 [svbu-mpi026:05529] FAILED to bind to 172.29.218.166 [svbu-mpi026:05529] SUCCEEDED to bind to 10.10.30.166 [svbu-mpi026:05529] SUCCEEDED to bind to 10.10.20.166 ... The 172.x address is my gigE device (eth0). -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
Sean Hefty wrote: There is still some inconsistency here. Sean, you claimed binds to 127.0.0.1 succeed in ofed-1.4 for IB devices. If so, then folks running IB/openmpi/rdmacm should be seeing issues. We need to dig a little more... You can verify this by running ucmatose -b 127.0.0.1 and see if the test enters the listening state. Well ofed-1.4.1 with openmpi gets failures when binding to 127.0.0.1 on mthca devs. Jeff will post the results soon. Are you sure ucmatose is really binding to that address? :) Can you also try testing iwarp with the patch that I sent? I will soon. Can't do it right now. I'll try tonight or tomorrow. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
Well, I think you are right. This kind of change seems appropriate to me for mainline, but OFED/RHEL should carry a responsibility to manage an identified incompatibility, either patch their kernel, patch their OMPI, or publish an errata. That is the role of a distribution. RHEL has said, multiple times, that they rely on OpenFabrics to do the Right Thing. They don't do a lot of testing, validating, etc. In that case OFED plays the role of distribution. -- Roland Dreier rola...@cisco.com Cisco.com - http://www.cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/index.html -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: bug 1918 - openmpi broken due to rdma-cm changes
OpenMPI uses rdma_bind_addr() to figure out which ip addresses are valid for which IB devices. This logic is now broken. Regardless of whether OpenMPI should use another method for determining which IP address belong to which interfaces, we should probably rethink whether we're breaking rdma-cm semantics in a bad way on a point release. The changes to the rdma_cm have been merged upstream. These were fixes specifically to enable using the loopback address with RDMA devices. At first thought, we can extend enum ib_device_cap_flags to indicate if a device supports loopback capabilities or not. The rdma_cm could then skip over such devices when dealing with a loopback address. - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
Sean Hefty wrote: But how can you determine _which_ rdma device should be used if and app binds to 127.0.0.1? I think this is busted... The code just picks the first rdma device available. To me, this is preferable than simply disallowing the loopback device from working at all. I personally use it all the time, so I don't have to figure out what the ip address is of the system that I'm trying to test on. Well then the rdma-cm needs to know which devices support hw loopback. Cuz on a T3-only system, no hwloop... Steve. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: bug 1918 - openmpi broken due to rdma-cm changes
Well then the rdma-cm needs to know which devices support hw loopback. Cuz on a T3-only system, no hwloop... The problem sounds like it's more than just whether 127.0.0.1 is usable. That check may fix openmpi, but it sounds more like the app needs to know whether the device can actually support loopback, regardless of what addresses are used. Is this correct? What would openmpi do if there were two addresses assigned to the T3 device? Does openmpi simply bypass RDMA for all connections on the local machine? Basically, I'm not sure that this is *just* an rdma_cm issue. Although it definitely appears that some sort of change needs to be made to the rdma_cm. - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
Sean Hefty wrote: At first thought, we can extend enum ib_device_cap_flags to indicate if a device supports loopback capabilities or not. The rdma_cm could then skip over such devices when dealing with a loopback address. This solution would work. Will you code it up? Stevo -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: bug 1918 - openmpi broken due to rdma-cm changes
This solution would work. Will you code it up? I can do that. I just want to make sure that we address the full scope of the problem. - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
Sean Hefty wrote: Well then the rdma-cm needs to know which devices support hw loopback. Cuz on a T3-only system, no hwloop... The problem sounds like it's more than just whether 127.0.0.1 is usable. That check may fix openmpi, but it sounds more like the app needs to know whether the device can actually support loopback, regardless of what addresses are used. Is this correct? What would openmpi do if there were two addresses assigned to the T3 device? It would use them and might even create two connections. Does openmpi simply bypass RDMA for all connections on the local machine? OpenMPI can be run to use hw loopback if its available. For T3 clusters, OMPI is run in a mode to use shared memory for intra-node communications. Basically, I'm not sure that this is *just* an rdma_cm issue. Although it definitely appears that some sort of change needs to be made to the rdma_cm. I think the OpenMPI rdmacm code needs to skip 127.0.0.1, in this particular case. Prior to ofed-1.5.1, however, the bind would fail and thus OpenMPI would not advertise 127.0.0.1 to its peer. I will work to get that change done. But lets also add a device attribute so the rdmacm can know if a device supports loopback. Clearly, if the rdma-cm allows binds to T3, loopback connections will fail at connect time. Hey Roland, are you ok with a device attribute to indicate hw-loopback support? Steve. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
Hey Roland, are you ok with a device attribute to indicate hw-loopback support? Sigh, I guess so. Can we have the rdma-cm handle this somewhat automagically, eg only choose devices that do handle loopback when binding/connecting to 127.0.0.1? Or maybe can we put the handling of this into the device-driver-specific iwcm methods, to fail listen/connect requests that are loopback? But if a device cap flag is what it takes, that's OK I guess. -- Roland Dreier rola...@cisco.com Cisco.com - http://www.cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/index.html -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: bug 1918 - openmpi broken due to rdma-cm changes
Roland Dreier wrote: Hey Roland, are you ok with a device attribute to indicate hw-loopback support? Sigh, I guess so. Can we have the rdma-cm handle this somewhat automagically, eg only choose devices that do handle loopback when binding/connecting to 127.0.0.1? That's the plan. Or maybe can we put the handling of this into the device-driver-specific iwcm methods, to fail listen/connect requests that are loopback? Is this only an iwarp issue? IE do all IB devices support hw loopback? And will all future devices support it (IE is it an IBTA requirement)? If so, then we could add a iwcm capabilities field maybe. But currently there's no bind method. We could add a bind method I suppose, but I'd rather do it with a simple capabilities. But if a device cap flag is what it takes, that's OK I guess. IMO the device cap flag is the way to go, and the rdma-cm should never bind 127.0.0.1 to a device that doesn't support loopback. Steve. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
