RE: [PATCH v4 0/4] Sending kernel pathrecord query to user cache server
From: Jason Gunthorpe [mailto:jguntho...@obsidianresearch.com] Sent: Wednesday, June 10, 2015 3:15 PM On Wed, Jun 10, 2015 at 05:04:55PM +, Hefty, Sean wrote: Not in the patches themselves but in the general issue when a PR changes. Do you think this needs addressing or are things fine as they are now ? IMO, I think it needs addressing in terms of can the proposed netlink architecture and design accommodate this sort of request in the future? We shouldn't design in a limitation up front. I don't see anything in the current approach that would cause an issue. There would likely be a need for new messages and attributes. I think the kernel netlink side is fine. But userspace needs to understand what to do if it gets a request with an attribute it does not understand. It should tell the kernel 'no, do it yourself', rather than try and answer. I will add some code in ibacm to do that. Kaike -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH v4 0/4] Sending kernel pathrecord query to user cache server
-Original Message- From: Hal Rosenstock [mailto:h...@dev.mellanox.co.il] Sent: Wednesday, June 10, 2015 3:52 PM On 6/10/2015 1:04 PM, Hefty, Sean wrote: Not in the patches themselves but in the general issue when a PR changes. Do you think this needs addressing or are things fine as they are now ? IMO, I think it needs addressing in terms of can the proposed netlink architecture and design accommodate this sort of request in the future? We shouldn't design in a limitation up front. I don't see anything in the current approach that would cause an issue. There would likely be a need for new messages and attributes. Can user space say it can't service a request ? Yes. It can NACK a request. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v4 0/4] Sending kernel pathrecord query to user cache server
On 6/9/2015 10:57 AM, kaike@intel.com wrote: From: Kaike Wan kaike@intel.com A SA cache is undeniably critical for fabric scalability and performance. In user space, the ibacm application provides a good example of pathrecord cache for address and route resolution. With the recent implementation of the provider architecture, ibacm offers more extensibility as a SA cache. In kernel, ipoib implements its own small cache for pathrecords, which is however not available for general use. Furthermore, the implementation of a SA cache in user space offers better flexibility, larger capacity, and more robustness for the system. In this patch series, a mechanism is implemented to allow ib_sa to send pathrecord query to a user application (eg ibacm) through netlink. While this appears to address the current upstream use model for ACM with it's multicast overlay backend where PRs are static, it does not appear to address PR changes. Should aging/revalidation of PRs be supported ? If so, would this make PRs similar at high level to IP neighbor cache in kernel ? -- Hal Potentially, this mechanism could be easily extended to other SA queries. With a customized test implemented in rdma_cm module (not included in this series), it was shown that the time to retrieve 1 million pathrecords dropped from 46660 jiffies (46.66 seconds) to 16119 jiffies (or 16.119 seconds) on a two-node system, a reduction of more than 60%. Changes since v3: - Patch 1: added basic RESOLVE attribute types. - Patch 4: changed the encoding of the RESOLVE request message based on the new attribute types and the input comp_mask. Changed the response handling by iterating all attributes. Changes since v2: - Redesigned the communication protocol between the kernel and user space application. Instead of the MAD packet format, the new protocol uses netlink message header and attributes to exchange request and response between the kernel and user space.The design was described here: http://www.spinics.net/lists/linux-rdma/msg25621.html Changes since v1: - Move kzalloc changes into a separate patch (Patch 3). - Remove redundant include line (Patch 4). - Rename struct rdma_nl_resp_msg as structure ib_nl_resp_msg (Patch 4). Kaike Wan (4): IB/netlink: Add defines for local service requests through netlink IB/core: Check the presence of netlink multicast group listeners IB/sa: Allocate SA query with kzalloc IB/sa: Route SA pathrecord query through netlink drivers/infiniband/core/netlink.c |8 + drivers/infiniband/core/sa_query.c | 523 +++- include/rdma/rdma_netlink.h|7 + include/uapi/rdma/rdma_netlink.h | 82 ++ 4 files changed, 615 insertions(+), 5 deletions(-) -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH v4 0/4] Sending kernel pathrecord query to user cache server
-Original Message- From: Hal Rosenstock [mailto:h...@dev.mellanox.co.il] Sent: Wednesday, June 10, 2015 9:37 AM A SA cache is undeniably critical for fabric scalability and performance. In user space, the ibacm application provides a good example of pathrecord cache for address and route resolution. With the recent implementation of the provider architecture, ibacm offers more extensibility as a SA cache. In kernel, ipoib implements its own small cache for pathrecords, which is however not available for general use. Furthermore, the implementation of a SA cache in user space offers better flexibility, larger capacity, and more robustness for the system. In this patch series, a mechanism is implemented to allow ib_sa to send pathrecord query to a user application (eg ibacm) through netlink. While this appears to address the current upstream use model for ACM with it's multicast overlay backend where PRs are static, it does not appear to address PR changes. Should aging/revalidation of PRs be supported ? If so, would this make PRs similar at high level to IP neighbor cache in kernel ? Even for the default provider acmp, PRs will time out and the length of timeout is configurable. For other providers (eg ibssa), the PR change could be managed correctly and promptly, and this capability is beyond ibacm core itself. Kaike -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v4 0/4] Sending kernel pathrecord query to user cache server
On 6/10/2015 10:22 AM, Wan, Kaike wrote: -Original Message- From: Hal Rosenstock [mailto:h...@dev.mellanox.co.il] Sent: Wednesday, June 10, 2015 9:37 AM A SA cache is undeniably critical for fabric scalability and performance. In user space, the ibacm application provides a good example of pathrecord cache for address and route resolution. With the recent implementation of the provider architecture, ibacm offers more extensibility as a SA cache. In kernel, ipoib implements its own small cache for pathrecords, which is however not available for general use. Furthermore, the implementation of a SA cache in user space offers better flexibility, larger capacity, and more robustness for the system. In this patch series, a mechanism is implemented to allow ib_sa to send pathrecord query to a user application (eg ibacm) through netlink. While this appears to address the current upstream use model for ACM with it's multicast overlay backend where PRs are static, it does not appear to address PR changes. Should aging/revalidation of PRs be supported ? If so, would this make PRs similar at high level to IP neighbor cache in kernel ? Even for the default provider acmp, PRs will time out and the length of timeout is configurable. For other providers (eg ibssa), the PR change could be managed correctly and promptly, and this capability is beyond ibacm core itself. That deals with the update of PR in user space (ACM). Doesn't kernel need some way of knowing PR was updated ? -- Hal Kaike -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH v4 0/4] Sending kernel pathrecord query to user cache server
A SA cache is undeniably critical for fabric scalability and performance. In user space, the ibacm application provides a good example of pathrecord cache for address and route resolution. With the recent implementation of the provider architecture, ibacm offers more extensibility as a SA cache. In kernel, ipoib implements its own small cache for pathrecords, which is however not available for general use. Furthermore, the implementation of a SA cache in user space offers better flexibility, larger capacity, and more robustness for the system. In this patch series, a mechanism is implemented to allow ib_sa to send pathrecord query to a user application (eg ibacm) through netlink. While this appears to address the current upstream use model for ACM with it's multicast overlay backend where PRs are static, it does not appear to address PR changes. Should aging/revalidation of PRs be supported ? If so, would this make PRs similar at high level to IP neighbor cache in kernel ? Even for the default provider acmp, PRs will time out and the length of timeout is configurable. For other providers (eg ibssa), the PR change could be managed correctly and promptly, and this capability is beyond ibacm core itself. That deals with the update of PR in user space (ACM). Doesn't kernel need some way of knowing PR was updated ? Maybe. If a kernel client is interested in PR changes, it can register for notification Wouldn't the notification mechanism be via netlink ? What would the granularity be for the registrations and notifications ? Not necessarily. AFAIK, SA notification is not currently supported in kernel. One could register notification through netlink, as long as the user space application (eg: ibacm) supports it. But this would be a new feature, as Sean pointed out in previous e-mail. Kaike -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v4 0/4] Sending kernel pathrecord query to user cache server
On 6/10/2015 11:21 AM, Hefty, Sean wrote: While this appears to address the current upstream use model for ACM with it's multicast overlay backend where PRs are static, it does not appear to address PR changes. Although this ties into ibacm, from the viewpoint of the kernel, there's no requirement on the user space implementation. True. This is not how it works in the kernel today. It is meant as future thinking/exploring. Should aging/revalidation of PRs be supported ? If so, would this make PRs similar at high level to IP neighbor cache in kernel ? This is requesting a new feature not supported by anything in the kernel today, and would seem to fall well outside the scope of the suggested changes. Outside scope of suggested changes but where does the kernel need to head in the longer term ? Is there a specific issue in the patches that you are seeing related to this? Not in the patches themselves but in the general issue when a PR changes. Do you think this needs addressing or are things fine as they are now ? -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v4 0/4] Sending kernel pathrecord query to user cache server
On 6/10/2015 11:49 AM, Wan, Kaike wrote: A SA cache is undeniably critical for fabric scalability and performance. In user space, the ibacm application provides a good example of pathrecord cache for address and route resolution. With the recent implementation of the provider architecture, ibacm offers more extensibility as a SA cache. In kernel, ipoib implements its own small cache for pathrecords, which is however not available for general use. Furthermore, the implementation of a SA cache in user space offers better flexibility, larger capacity, and more robustness for the system. In this patch series, a mechanism is implemented to allow ib_sa to send pathrecord query to a user application (eg ibacm) through netlink. While this appears to address the current upstream use model for ACM with it's multicast overlay backend where PRs are static, it does not appear to address PR changes. Should aging/revalidation of PRs be supported ? If so, would this make PRs similar at high level to IP neighbor cache in kernel ? Even for the default provider acmp, PRs will time out and the length of timeout is configurable. For other providers (eg ibssa), the PR change could be managed correctly and promptly, and this capability is beyond ibacm core itself. That deals with the update of PR in user space (ACM). Doesn't kernel need some way of knowing PR was updated ? Maybe. If a kernel client is interested in PR changes, it can register for notification Wouldn't the notification mechanism be via netlink ? What would the granularity be for the registrations and notifications ? Not necessarily. AFAIK, SA notification is not currently supported in kernel. One could register notification through netlink, as long as the user space application (eg: ibacm) supports it. But this would be a new feature, as Sean pointed out in previous e-mail. Such notifications are not necessarily tied to SA event reporting. In fact, Todd made argument at 2014 OFA Devcon that such mechanism is anathema to (exa)scalability. -- Hal Kaike -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v4 0/4] Sending kernel pathrecord query to user cache server
On Wed, Jun 10, 2015 at 05:04:55PM +, Hefty, Sean wrote: Not in the patches themselves but in the general issue when a PR changes. Do you think this needs addressing or are things fine as they are now ? IMO, I think it needs addressing in terms of can the proposed netlink architecture and design accommodate this sort of request in the future? We shouldn't design in a limitation up front. I don't see anything in the current approach that would cause an issue. There would likely be a need for new messages and attributes. I think the kernel netlink side is fine. But userspace needs to understand what to do if it gets a request with an attribute it does not understand. It should tell the kernel 'no, do it yourself', rather than try and answer. That also suggests we should have optional and mandatory netlink nested attributes. Jason -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v4 0/4] Sending kernel pathrecord query to user cache server
On 6/10/2015 1:04 PM, Hefty, Sean wrote: Not in the patches themselves but in the general issue when a PR changes. Do you think this needs addressing or are things fine as they are now ? IMO, I think it needs addressing in terms of can the proposed netlink architecture and design accommodate this sort of request in the future? We shouldn't design in a limitation up front. I don't see anything in the current approach that would cause an issue. There would likely be a need for new messages and attributes. The current proposal is focused around the PR attributes/styles currently used in the kernel. The case I can see is if in future a new attribute is added to the PR netlink API. How is that handled ? Can user space say it can't service a request ? That seems a little different from the no PR case. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH v4 0/4] Sending kernel pathrecord query to user cache server
This series does not attempt to optimize the kernel needing to know that a PR has been updated. There are existing mechanisms for that. Does this exist in the kernel? -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH v4 0/4] Sending kernel pathrecord query to user cache server
Not in the patches themselves but in the general issue when a PR changes. Do you think this needs addressing or are things fine as they are now ? IMO, I think it needs addressing in terms of can the proposed netlink architecture and design accommodate this sort of request in the future? We shouldn't design in a limitation up front. I don't see anything in the current approach that would cause an issue. There would likely be a need for new messages and attributes. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v4 0/4] Sending kernel pathrecord query to user cache server
On Wed, Jun 10, 2015 at 10:39:49AM -0400, Hal Rosenstock wrote: On 6/10/2015 10:22 AM, Wan, Kaike wrote: -Original Message- From: Hal Rosenstock [mailto:h...@dev.mellanox.co.il] Sent: Wednesday, June 10, 2015 9:37 AM A SA cache is undeniably critical for fabric scalability and performance. In user space, the ibacm application provides a good example of pathrecord cache for address and route resolution. With the recent implementation of the provider architecture, ibacm offers more extensibility as a SA cache. In kernel, ipoib implements its own small cache for pathrecords, which is however not available for general use. Furthermore, the implementation of a SA cache in user space offers better flexibility, larger capacity, and more robustness for the system. In this patch series, a mechanism is implemented to allow ib_sa to send pathrecord query to a user application (eg ibacm) through netlink. While this appears to address the current upstream use model for ACM with it's multicast overlay backend where PRs are static, it does not appear to address PR changes. Should aging/revalidation of PRs be supported ? If so, would this make PRs similar at high level to IP neighbor cache in kernel ? Even for the default provider acmp, PRs will time out and the length of timeout is configurable. For other providers (eg ibssa), the PR change could be managed correctly and promptly, and this capability is beyond ibacm core itself. That deals with the update of PR in user space (ACM). Doesn't kernel need some way of knowing PR was updated ? This series does not attempt to optimize the kernel needing to know that a PR has been updated. There are existing mechanisms for that. In the future it would be nice to have more local support for such updates but I don't think this patch precludes that in any way. Right now we are just taking the first step by making the actual query more efficient. Ira -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v4 0/4] Sending kernel pathrecord query to user cache server
On Wed, Jun 10, 2015 at 09:34:58PM +, Hefty, Sean wrote: I agree. I just wanted to make sure that there wasn't some feature regarding PRs, such as unpath, that a kernel client would lose (i.e. it is currently implemented) by changing how the PRs are retrieved. Basically nothing breaks with this change. That is my expectation, I've seen nothing to contradict it... Jason -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH v4 0/4] Sending kernel pathrecord query to user cache server
Not directly. IPoIB treats it that way. I guess to be safe. Officially one should register for UnPath/RePath traps. But no one has ever implemented that. To be clear I am agreeing with Hal that having some sort of update signal would be nice. But I don't think that must be done before this series goes in. I think a PR update signal should be an extension to this interface. I agree. I just wanted to make sure that there wasn't some feature regarding PRs, such as unpath, that a kernel client would lose (i.e. it is currently implemented) by changing how the PRs are retrieved. Basically nothing breaks with this change. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH v4 0/4] Sending kernel pathrecord query to user cache server
This series does not attempt to optimize the kernel needing to know that a PR has been updated. There are existing mechanisms for that. Does this exist in the kernel? At least some support, yes. For example client reregister marks all IPoIB paths as invalid. Reregister indicates that all PRs are now invalid? -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH v4 0/4] Sending kernel pathrecord query to user cache server
This series does not attempt to optimize the kernel needing to know that a PR has been updated. There are existing mechanisms for that. Does this exist in the kernel? At least some support, yes. For example client reregister marks all IPoIB paths as invalid. Reregister indicates that all PRs are now invalid? Not directly. IPoIB treats it that way. I guess to be safe. Officially one should register for UnPath/RePath traps. But no one has ever implemented that. To be clear I am agreeing with Hal that having some sort of update signal would be nice. But I don't think that must be done before this series goes in. I think a PR update signal should be an extension to this interface. Ira -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH v4 0/4] Sending kernel pathrecord query to user cache server
This series does not attempt to optimize the kernel needing to know that a PR has been updated. There are existing mechanisms for that. Does this exist in the kernel? At least some support, yes. For example client reregister marks all IPoIB paths as invalid. Ira -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH v4 0/4] Sending kernel pathrecord query to user cache server
-Original Message- From: Hal Rosenstock [mailto:h...@dev.mellanox.co.il] Sent: Wednesday, June 10, 2015 10:40 AM -Original Message- From: Hal Rosenstock [mailto:h...@dev.mellanox.co.il] Sent: Wednesday, June 10, 2015 9:37 AM A SA cache is undeniably critical for fabric scalability and performance. In user space, the ibacm application provides a good example of pathrecord cache for address and route resolution. With the recent implementation of the provider architecture, ibacm offers more extensibility as a SA cache. In kernel, ipoib implements its own small cache for pathrecords, which is however not available for general use. Furthermore, the implementation of a SA cache in user space offers better flexibility, larger capacity, and more robustness for the system. In this patch series, a mechanism is implemented to allow ib_sa to send pathrecord query to a user application (eg ibacm) through netlink. While this appears to address the current upstream use model for ACM with it's multicast overlay backend where PRs are static, it does not appear to address PR changes. Should aging/revalidation of PRs be supported ? If so, would this make PRs similar at high level to IP neighbor cache in kernel ? Even for the default provider acmp, PRs will time out and the length of timeout is configurable. For other providers (eg ibssa), the PR change could be managed correctly and promptly, and this capability is beyond ibacm core itself. That deals with the update of PR in user space (ACM). Doesn't kernel need some way of knowing PR was updated ? Maybe. If a kernel client is interested in PR changes, it can register for notification and once a notice is received, it can always go back to query for the new PR. But that is the responsibility of the individual client and is beyond the scope of this patch series. Kaike -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v4 0/4] Sending kernel pathrecord query to user cache server
On 6/10/2015 11:07 AM, Wan, Kaike wrote: -Original Message- From: Hal Rosenstock [mailto:h...@dev.mellanox.co.il] Sent: Wednesday, June 10, 2015 10:40 AM -Original Message- From: Hal Rosenstock [mailto:h...@dev.mellanox.co.il] Sent: Wednesday, June 10, 2015 9:37 AM A SA cache is undeniably critical for fabric scalability and performance. In user space, the ibacm application provides a good example of pathrecord cache for address and route resolution. With the recent implementation of the provider architecture, ibacm offers more extensibility as a SA cache. In kernel, ipoib implements its own small cache for pathrecords, which is however not available for general use. Furthermore, the implementation of a SA cache in user space offers better flexibility, larger capacity, and more robustness for the system. In this patch series, a mechanism is implemented to allow ib_sa to send pathrecord query to a user application (eg ibacm) through netlink. While this appears to address the current upstream use model for ACM with it's multicast overlay backend where PRs are static, it does not appear to address PR changes. Should aging/revalidation of PRs be supported ? If so, would this make PRs similar at high level to IP neighbor cache in kernel ? Even for the default provider acmp, PRs will time out and the length of timeout is configurable. For other providers (eg ibssa), the PR change could be managed correctly and promptly, and this capability is beyond ibacm core itself. That deals with the update of PR in user space (ACM). Doesn't kernel need some way of knowing PR was updated ? Maybe. If a kernel client is interested in PR changes, it can register for notification Wouldn't the notification mechanism be via netlink ? What would the granularity be for the registrations and notifications ? and once a notice is received, it can always go back to query for the new PR. But that is the responsibility of the individual client and is beyond the scope of this patch series. I thought that the goal is to have an interface that addresses kernel - userspace needs for PRs in general and ACM was just first consumer. I'm not sure that it's sufficient for other ACM providers. -- Hal Kaike -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH v4 0/4] Sending kernel pathrecord query to user cache server
While this appears to address the current upstream use model for ACM with it's multicast overlay backend where PRs are static, it does not appear to address PR changes. Although this ties into ibacm, from the viewpoint of the kernel, there's no requirement on the user space implementation. Should aging/revalidation of PRs be supported ? If so, would this make PRs similar at high level to IP neighbor cache in kernel ? This is requesting a new feature not supported by anything in the kernel today, and would seem to fall well outside the scope of the suggested changes. Is there a specific issue in the patches that you are seeing related to this? -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
