RE: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview
Hi Steve, Tatyana Reading this cover letter, I think it's good idea to follow Steve's suggestion and add portions from here into the IB/core change you introduced (patch #1 in this series) and is now merged into Roland's for-next -- before Roland's sends his 3.16 merge window pull request to Linus (there is enough time!!), since the current change-log http://git.kernel.org/cgit/linux/kernel/git/roland/infiniband.git/commit/?h=for- nextid=1943d56776ad31a119517efeb309786db3b4383e practically says nothing on what is the PM about Roland, can you do this? 2nd, there's nothing specific to iWARP in the port mapper implementation, right? What it does is the following: 1. kernel rdma driver tells a user space daemon they want to reserve (claim) the combination of IP address X and TCP port Y for the sake of RDMA connections 2. user space daemon opens a socket and binds to X:Y No, the use space daemon opens a socket and binds to address X port 0 and thus allocates an ephemeral port number Z. The mapping of X-Z is then made available to clients trying to connect to this service. Specifically, down the road, more use cases, not only the current iWARP case may pop up. It is really only needed for iWARP and not IB because IB doesn't share the port space with the host TCP stack. iWARP does. So taking into account that the implementation doesn't have anything whivh is IW specific, and that future non-IW use case might show up, why we need to fill the IB core code with IW_PM notation? why not just use RDMA_PM notation? this can be achieved quickly with $ sed -i g/iw_pm/rdma_pm/g * and friends on Roland's tree and later squashed into the relevant patches. I understand that my comment came little late... but taking into account the ease of the change (sed exercise), why not doing it? the IB core patch exposes API to user-space, so we should make sure to do it right I'm not convinced this is desirable. Steve. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview
On Sun, Jun 08, 2014 at 10:34:30PM +0300, Or Gerlitz wrote: What it does is the following: 1. kernel rdma driver tells a user space daemon they want to reserve (claim) the combination of IP address X and TCP port Y for the sake of RDMA connections 2. user space daemon opens a socket and binds to X:Y Specifically, down the road, more use cases, not only the current iWARP case may pop up. This really seems horrible, using user space to circumvent the kernel stack because kernel maintainers don't want this kind of integration is not going to make people very happy. IIRC the patch set that tried to do this directly in the kernel was NAK'd, adding a userspace round trip doesn't really change anything. The message from netdev has, IMHO, always been pretty clear - offload can live in it's own little side world but cannot appear to the user to be integrated to the main stack (because it isn't). Jason -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview
On Mon, 9 Jun 2014, Jason Gunthorpe wrote: The message from netdev has, IMHO, always been pretty clear - offload can live in it's own little side world but cannot appear to the user to be integrated to the main stack (because it isn't). Hmmm... The flow steering API already does what the port mapper does and its already merged. Why do we need another implementation? -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview
-Original Message- From: Jason Gunthorpe [mailto:jguntho...@obsidianresearch.com] Sent: Monday, June 09, 2014 11:51 AM To: Or Gerlitz Cc: Tatyana Nikolova; Steve Wise; Roland Dreier; Lacombe, John S; Sean Hefty; linux-rdma Subject: Re: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview On Sun, Jun 08, 2014 at 10:34:30PM +0300, Or Gerlitz wrote: What it does is the following: 1. kernel rdma driver tells a user space daemon they want to reserve (claim) the combination of IP address X and TCP port Y for the sake of RDMA connections 2. user space daemon opens a socket and binds to X:Y Specifically, down the road, more use cases, not only the current iWARP case may pop up. This really seems horrible, using user space to circumvent the kernel stack because kernel maintainers don't want this kind of integration is not going to make people very happy. IIRC the patch set that tried to do this directly in the kernel was NAK'd, adding a userspace round trip doesn't really change anything. The patch set you refer to tried to _unify_ the port space and was rejected. The only other alternative is to pick ephemeral ports and maintain a mapping for RDMA services. The message from netdev has, IMHO, always been pretty clear - offload can live in it's own little side world but cannot appear to the user to be integrated to the main stack (because it isn't). That is what this design does... Steve. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview
-Original Message- From: Christoph Lameter [mailto:c...@gentwo.org] Sent: Monday, June 09, 2014 1:11 PM To: Jason Gunthorpe Cc: Or Gerlitz; Tatyana Nikolova; Steve Wise; Roland Dreier; Lacombe, John S; Sean Hefty; linux-rdma Subject: Re: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview On Mon, 9 Jun 2014, Jason Gunthorpe wrote: The message from netdev has, IMHO, always been pretty clear - offload can live in it's own little side world but cannot appear to the user to be integrated to the main stack (because it isn't). Hmmm... The flow steering API already does what the port mapper does and its already merged. Why do we need another implementation? I don't see how the flow steering API does anything like the port mapping. Steve. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview
-Original Message- From: Christoph Lameter [mailto:c...@gentwo.org] Sent: Monday, June 09, 2014 1:28 PM To: Steve Wise Cc: 'Jason Gunthorpe'; 'Or Gerlitz'; 'Tatyana Nikolova'; 'Roland Dreier'; 'Lacombe, John S'; 'Sean Hefty'; 'linux-rdma' Subject: RE: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview On Mon, 9 Jun 2014, Steve Wise wrote: I don't see how the flow steering API does anything like the port mapping. You can redirect flows with a particular port number. Thats the same as port mappig right? We don't want to redirect any flows. We want to map tcp port numbers to ephemeral port numbers so a offload iWARP connection 4-tuple doesn't collide with a host tcp connection 4-tuple. Steve. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview
On Mon, 9 Jun 2014, Steve Wise wrote:
I don't see how the flow steering API does anything like the port mapping.
You can redirect flows with a particular port number. Thats the same as
port mappig right?
linux/include/rdma/ib_verbs.h:
struct ib_flow_spec_ipv4 {
enum ib_flow_spec_type type;
u16size;
struct ib_flow_ipv4_filter val;
struct ib_flow_ipv4_filter mask;
};
struct ib_flow_tcp_udp_filter {
__be16 dst_port;
__be16 src_port;
};
struct ib_flow_spec_tcp_udp {
enum ib_flow_spec_typetype;
u16 size;
struct ib_flow_tcp_udp_filter val;
struct ib_flow_tcp_udp_filter mask;
};
union ib_flow_spec {
struct {
enum ib_flow_spec_type type;
u16 size;
};
struct ib_flow_spec_eth eth;
struct ib_flow_spec_ib ib;
struct ib_flow_spec_ipv4ipv4;
struct ib_flow_spec_tcp_udp tcp_udp;
};
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview
On Mon, Jun 09, 2014 at 01:12:50PM -0500, Steve Wise wrote: 1. kernel rdma driver tells a user space daemon they want to reserve (claim) the combination of IP address X and TCP port Y for the sake of RDMA connections 2. user space daemon opens a socket and binds to X:Y Specifically, down the road, more use cases, not only the current iWARP case may pop up. This really seems horrible, using user space to circumvent the kernel stack because kernel maintainers don't want this kind of integration is not going to make people very happy. IIRC the patch set that tried to do this directly in the kernel was NAK'd, adding a userspace round trip doesn't really change anything. The patch set you refer to tried to _unify_ the port space and was rejected. The only other alternative is to pick ephemeral ports and maintain a mapping for RDMA services. I thought both approaches were tried in kernel and NAK'd by netdev? This looks very similar to the 2010 patchset, except the dummy socket allocation is living in userspace in this version. Again, from here, this looks like another patch to do an end-run around netdev's NAK by hiding the offending stuff in userspace. That is no good either... The message from netdev has, IMHO, always been pretty clear - offload can live in it's own little side world but cannot appear to the user to be integrated to the main stack (because it isn't). That is what this design does... Not really, it is still sharing an IP used by the main stack, otherwise you wouldn't have this problem in the first place. Jason -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview
Tatyana Nikolova tatyana.e.nikol...@intel.com wrote: This patch series adds iWARP Port Mapper (IWPM) Version 2 support in RDMA/core, RDMA/nes driver and RDMA/cxgb4 driver. Hi Steve, Tatyana Reading this cover letter, I think it's good idea to follow Steve's suggestion and add portions from here into the IB/core change you introduced (patch #1 in this series) and is now merged into Roland's for-next -- before Roland's sends his 3.16 merge window pull request to Linus (there is enough time!!), since the current change-log http://git.kernel.org/cgit/linux/kernel/git/roland/infiniband.git/commit/?h=for-nextid=1943d56776ad31a119517efeb309786db3b4383e practically says nothing on what is the PM about 2nd, there's nothing specific to iWARP in the port mapper implementation, right? What it does is the following: 1. kernel rdma driver tells a user space daemon they want to reserve (claim) the combination of IP address X and TCP port Y for the sake of RDMA connections 2. user space daemon opens a socket and binds to X:Y Specifically, down the road, more use cases, not only the current iWARP case may pop up. So taking into account that the implementation doesn't have anything whivh is IW specific, and that future non-IW use case might show up, why we need to fill the IB core code with IW_PM notation? why not just use RDMA_PM notation? this can be achieved quickly with $ sed -i g/iw_pm/rdma_pm/g * and friends on Roland's tree and later squashed into the relevant patches. I understand that my comment came little late... but taking into account the ease of the change (sed exercise), why not doing it? the IB core patch exposes API to user-space, so we should make sure to do it right Or. The iWARP Port Mapper implementation is based on the port mapper specification section in the Sockets Direct Protocol paper - http://www.rdmaconsortium.org/home/draft-pinkerton-iwarp-sdp-v1.0.pdf Existing iWARP RDMA providers use the same IP address as the native TCP/IP stack when creating RDMA connections. They need to ensure a mechanism to claim the TCP ports used for RDMA connections, thereby preventing TCP port collisions when other host applications are consuming TCP ports. The iWARP Port Mapper provides a standard mechanism to accomplish this. Without this service it is possible for RDMA application to bind/listen on the same port which is already being used by native TCP host application. If that happens the incoming TCP connection data can be passed to the RDMA stack with error. The iWARP Port Mapper solution doesn't contain any changes to the existing network stack in the kernel space. All the changes are contained with the infiniband tree and also in user space. The iWARP Port Mapper service is implemented as a user space daemon process. Interested readers could look at the source code of the IWPM service located at http://git.openfabrics.org/git?p=~tnikolova/libiwpm-1.0.0/.git;a=summary The iWARP driver (port mapper client) sends to the IWPM service the local IP address and TCP port it has received from the RDMA application, when starting a connection. The IWPM service performs a socket bind from user space to get an available TCP port, called a mapped port, and communicates it back to the client. In that sense, the IWPM service is used to map the TCP port, which the RDMA application uses to any port available from the host TCP port space. The mapped ports are used in iWARP RDMA connections to avoid collisions with native TCP stack which is aware that these ports are taken. When an RDMA connection using a mapped port is terminated, the client notifies the IWPM service, which then releases the TCP port. The message exchange between the IWPM service and the iWARP drivers (between user space and kernel space) is implemented using netlink sockets. This patch series adds iWarp driver support for sending/receiving/parsing netlink messages, to make the communication with the IWPM service possible. This is the second submission for the iWarp Port Mapper and we have considered and incorporated feedback after the first submission to improve the IWPM Version 2 patch series. The following items are improvements and fixes to IWPM V2: 1) The IWPM functionality, common for both iWarp drivers (nes and cxgb4) is refactored from the drivers source files and is moved to new shared files in infiniband/core which are compiled as part of the iw_cm module. 2) Hash table data structure is implemented to store the drivers mapping information and enable efficient searching. (Hash table implementation is based on original work by Vipul Pandya) 3) Fix for a timer bug is provided as the unnecessary timer scheduling is removed. The patches are built against Roland's infiniband tree for-next branch. Thank you, Tatyana Nikolova (2): RDMA/core: Add support for iWarp Port Mapper V2 user space service RDMA/nes: Add support for iWarp Port Mapper V2 user space
Re: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview
On Tue, May 20, 2014 at 4:10 AM, Roland Dreier rol...@kernel.org wrote: So I'm looking to include this for 3.16. Roland, can you reword these patches and remove Peter's email address, it bounces back (just tried it, SB) and I don't see the point to introduce an email in a signature which is broken from day one. Or. Delivery to the following recipient failed permanently: peter.p.waskiewicz...@intel.com Technical details of permanent failure: Google tried to deliver your message, but it was rejected by the server for the recipient domain intel.com by mga01.intel.com. [192.55.52.88]. The error that the other server returned was: 550 #5.1.0 Address rejected. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview
On 27/03/2014 00:08, Tatyana Nikolova wrote: This patch series adds iWARP Port Mapper (IWPM) Version 2 support in RDMA/core, RDMA/nes driver and RDMA/cxgb4 driver. Guys, the link here http://vger.kernel.org/netconf2011_slides/pj_netconf2011.ppt seems to point to a damaged PPT, can you send me updated copy -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview
-Original Message- From: linux-rdma-ow...@vger.kernel.org [mailto:linux-rdma-ow...@vger.kernel.org] On Behalf Of Roland Dreier Sent: Monday, May 19, 2014 8:10 PM To: Tatyana Nikolova Cc: Lacombe, John S; Steve Wise; Sean Hefty; linux-rdma@vger.kernel.org Subject: Re: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview On Wed, Mar 26, 2014 at 3:07 PM, Tatyana Nikolova tatyana.e.nikol...@intel.com wrote: 1) The IWPM functionality, common for both iWarp drivers (nes and cxgb4) is refactored from the drivers source files and is moved to new shared files in infiniband/core which are compiled as part of the iw_cm module. So I'm looking to include this for 3.16. One question: I assume unpatched iwarp drivers (eg cxgb3) continue to work as before? In other words they don't get the new port mapper support but at least they don't regress? Hey Roland, cxgb3-cxgb3 would behave as before; no regression. cxgb3-cxgb4/nes would interoperate only if the port mapping service is disabled on the cxgb4/nes nodes. Steve. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview
On 20/05/2014 16:46, Steve Wise wrote: But here is it attached thanks -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview
Hello Rolland,
Thanks for reviewing the port mapper patches.
Steve's reply is correct.
Unpatched nes will not regress and will continue working as before.
Tatyana
-Original Message-
From: Steve Wise [mailto:sw...@opengridcomputing.com]
Sent: Tuesday, May 20, 2014 8:45 AM
To: 'Roland Dreier'; Nikolova, Tatyana E
Cc: Lacombe, John S; Hefty, Sean; linux-rdma@vger.kernel.org; 'Doug Ledford'
Subject: RE: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview
-Original Message-
From: linux-rdma-ow...@vger.kernel.org
[mailto:linux-rdma-ow...@vger.kernel.org] On Behalf Of Roland Dreier
Sent: Monday, May 19, 2014 8:10 PM
To: Tatyana Nikolova
Cc: Lacombe, John S; Steve Wise; Sean Hefty;
linux-rdma@vger.kernel.org
Subject: Re: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview
On Wed, Mar 26, 2014 at 3:07 PM, Tatyana Nikolova
tatyana.e.nikol...@intel.com wrote:
1) The IWPM functionality, common for both iWarp drivers (nes and cxgb4)
is refactored from the drivers source files and is moved to new shared
files in infiniband/core which are compiled as part of the iw_cm module.
So I'm looking to include this for 3.16. One question: I assume
unpatched iwarp drivers (eg cxgb3) continue to work as before? In
other words they don't get the new port mapper support but at least
they don't regress?
Hey Roland,
cxgb3-cxgb3 would behave as before; no regression. cxgb3-cxgb4/nes would
interoperate only if the port mapping service is disabled on the cxgb4/nes
nodes.
Steve.
N�r��yb�X��ǧv�^�){.n�+{��ٚ�{ay� ʇڙ�,j��f���h���z� �w���
���j:+v���w�j�mzZ+�ݢj��!�i
Re: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview
On Wed, Mar 26, 2014 at 3:07 PM, Tatyana Nikolova tatyana.e.nikol...@intel.com wrote: 1) The IWPM functionality, common for both iWarp drivers (nes and cxgb4) is refactored from the drivers source files and is moved to new shared files in infiniband/core which are compiled as part of the iw_cm module. So I'm looking to include this for 3.16. One question: I assume unpatched iwarp drivers (eg cxgb3) continue to work as before? In other words they don't get the new port mapper support but at least they don't regress? - R. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview
Any review comments on this? -Original Message- From: Tatyana Nikolova [mailto:tatyana.e.nikol...@intel.com] Sent: Wednesday, March 26, 2014 5:07 PM To: Roland Dreier Cc: john.s.laco...@intel.com; sw...@opengridcomputing.com; peter.p.waskiewicz...@intel.com; sean.he...@intel.com; linux-rdma@vger.kernel.org Subject: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview Hello All, This patch series adds iWARP Port Mapper (IWPM) Version 2 support in RDMA/core, RDMA/nes driver and RDMA/cxgb4 driver. The iWARP Port Mapper implementation is based on the port mapper specification section in the Sockets Direct Protocol paper - http://www.rdmaconsortium.org/home/draft-pinkerton-iwarp-sdp-v1.0.pdf Existing iWARP RDMA providers use the same IP address as the native TCP/IP stack when creating RDMA connections. They need to ensure a mechanism to claim the TCP ports used for RDMA connections, thereby preventing TCP port collisions when other host applications are consuming TCP ports. The iWARP Port Mapper provides a standard mechanism to accomplish this. Without this service it is possible for RDMA application to bind/listen on the same port which is already being used by native TCP host application. If that happens the incoming TCP connection data can be passed to the RDMA stack with error. The iWARP Port Mapper solution doesn't contain any changes to the existing network stack in the kernel space. All the changes are contained with the infiniband tree and also in user space. The iWARP Port Mapper service is implemented as a user space daemon process. Interested readers could look at the source code of the IWPM service located at http://git.openfabrics.org/git?p=~tnikolova/libiwpm-1.0.0/.git;a=summary The iWARP driver (port mapper client) sends to the IWPM service the local IP address and TCP port it has received from the RDMA application, when starting a connection. The IWPM service performs a socket bind from user space to get an available TCP port, called a mapped port, and communicates it back to the client. In that sense, the IWPM service is used to map the TCP port, which the RDMA application uses to any port available from the host TCP port space. The mapped ports are used in iWARP RDMA connections to avoid collisions with native TCP stack which is aware that these ports are taken. When an RDMA connection using a mapped port is terminated, the client notifies the IWPM service, which then releases the TCP port. The message exchange between the IWPM service and the iWARP drivers (between user space and kernel space) is implemented using netlink sockets. This patch series adds iWarp driver support for sending/receiving/parsing netlink messages, to make the communication with the IWPM service possible. This is the second submission for the iWarp Port Mapper and we have considered and incorporated feedback after the first submission to improve the IWPM Version 2 patch series. The following items are improvements and fixes to IWPM V2: 1) The IWPM functionality, common for both iWarp drivers (nes and cxgb4) is refactored from the drivers source files and is moved to new shared files in infiniband/core which are compiled as part of the iw_cm module. 2) Hash table data structure is implemented to store the drivers mapping information and enable efficient searching. (Hash table implementation is based on original work by Vipul Pandya) 3) Fix for a timer bug is provided as the unnecessary timer scheduling is removed. The patches are built against Roland's infiniband tree for-next branch. Thank you, Tatyana Nikolova (2): RDMA/core: Add support for iWarp Port Mapper V2 user space service RDMA/nes: Add support for iWarp Port Mapper V2 user space service Steve Wise (1): RDMA/cxgb4: Add support for iWarp Port Mapper V2 user space service drivers/infiniband/core/Makefile |2 +- drivers/infiniband/core/cma.c |3 +- drivers/infiniband/core/iwpm_msg.c | 691 drivers/infiniband/core/iwpm_util.c| 609 drivers/infiniband/core/iwpm_util.h| 238 +++ drivers/infiniband/core/netlink.c | 18 +- drivers/infiniband/hw/cxgb4/cm.c | 180 +++-- drivers/infiniband/hw/cxgb4/device.c | 81 - drivers/infiniband/hw/cxgb4/iw_cxgb4.h | 44 ++ drivers/infiniband/hw/nes/nes.c| 25 ++- drivers/infiniband/hw/nes/nes.h|3 + drivers/infiniband/hw/nes/nes_cm.c | 320 --- drivers/infiniband/hw/nes/nes_cm.h | 12 +- include/rdma/iw_portmap.h | 199 + include/rdma/rdma_netlink.h| 23 +- include/uapi/rdma/rdma_netlink.h | 96 +- 16 files changed, 2431 insertions(+), 113 deletions(-) create mode 100644 drivers/infiniband/core/iwpm_msg.c create mode 100644
