Re: Sharing MR Between Multiple Connections
Le mardi 13 novembre 2012 à 23:36 -0500, Christopher Mitchell a écrit : Hi, I am working on building an Infiniband application with a server that can handle many simultaneous clients. The server exposes a chunk of memory that each of the clients can read via RDMA. I was previously creating a new MR on the server for each client (and of course in that connection's PD). However, under stress testing, I realized that ibv_reg_mr() started failing after I simultaneously MRed the same area enough times to cover 20.0 GB. I presume that the problem is reaching some pinning limit, although ulimit reports unlimited for all relevant possibilities. There's a limit of the number of registered memory region per HCA. See ibv_query_device(), struct ibv_device_attr, field max_mr. I tried creating a single global PD and a single MR to be shared among the multiple connections, but rdma_create_qp() fails with an invalid argument when I try to do that. I therefore deduce that the PD specified in rdma_create_qp() must correspond to an active connection, not simply be created by opening a device. Long question short: is there any way I can share the same MR among multiple clients, so that my shared memory region is limited to N bytes instead of N/C (clients) bytes? If each rdma_cm_id descriptor is tied to the same ibv_context, you might be able to share one memory pool registered with ibv_reg_mr(). This is quite usual since you are likely to have only one HCA in your system. Get a first context after binding your listening rdma_cm_id. Regards -- Yann Droneaud OPTEYA -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Sharing MR Between Multiple Connections
On Nov 13, 2012, at 11:36 PM, Christopher Mitchell christop...@cemetech.net wrote: Hi, I am working on building an Infiniband application with a server that can handle many simultaneous clients. The server exposes a chunk of memory that each of the clients can read via RDMA. I was previously creating a new MR on the server for each client (and of course in that connection's PD). However, under stress testing, I realized that ibv_reg_mr() started failing after I simultaneously MRed the same area enough times to cover 20.0 GB. I presume that the problem is reaching some pinning limit, although ulimit reports unlimited for all relevant possibilities. I tried creating a single global PD and a single MR to be shared among the multiple connections, but rdma_create_qp() fails with an invalid argument when I try to do that. I therefore deduce that the PD specified in rdma_create_qp() must correspond to an active connection, not simply be created by opening a device. Long question short: is there any way I can share the same MR among multiple clients, so that my shared memory region is limited to N bytes instead of N/C (clients) bytes? Christopher, Yes, it is possible. You have to use the same PD for all QPs/connections. We do this in CCI when using the Verbs transport. Scott-- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Sharing MR Between Multiple Connections
I am working on building an Infiniband application with a server that can handle many simultaneous clients. The server exposes a chunk of memory that each of the clients can read via RDMA. I was previously creating a new MR on the server for each client (and of course in that connection's PD). However, under stress testing, I realized that ibv_reg_mr() started failing after I simultaneously MRed the same area enough times to cover 20.0 GB. I presume that the problem is reaching some pinning limit, although ulimit reports unlimited for all relevant possibilities. I tried creating a single global PD and a single MR to be shared among the multiple connections, but rdma_create_qp() fails with an invalid argument when I try to do that. I therefore deduce that the PD specified in rdma_create_qp() must correspond to an active connection, not simply be created by opening a device. The rdma_cm will automatically allocate one PD per RDMA device. You can share this PD among multiple connections. To use this PD, pass in NULL into rdma_create_qp(). The rdma_cm_id will reference the shared PD. Long question short: is there any way I can share the same MR among multiple clients, so that my shared memory region is limited to N bytes instead of N/C (clients) bytes? Yes, but if there are multiple devices being used, then you'll need to track those registrations separately. - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Sharing MR Between Multiple Connections
On Wed, Nov 14, 2012 at 12:13 PM, Hefty, Sean sean.he...@intel.com wrote: The rdma_cm will automatically allocate one PD per RDMA device. You can share this PD among multiple connections. To use this PD, pass in NULL into rdma_create_qp(). The rdma_cm_id will reference the shared PD. That's great to know; thanks Sean (and everyone else who confirmed that a shared PD and possibly CQ is the way to go). Is there a way to reference this shared PD in ibv_reg_mr() as well? It seems that using NULL there too is not the solution, and I'm having difficulty tracing back from struct rdma_cm_id to find out where that shared PD is stored. I'd prefer to use this solution for simplicity's sake, so any additional details would be greatly appreciated. For what it's worth, I tried passing NULL to rdma_create_qp() as you suggested, and found it to work admirably. Long question short: is there any way I can share the same MR among multiple clients, so that my shared memory region is limited to N bytes instead of N/C (clients) bytes? Yes, but if there are multiple devices being used, then you'll need to track those registrations separately. That's reasonable. Thanks, Christopher -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Sharing MR Between Multiple Connections
The rdma_cm will automatically allocate one PD per RDMA device. You can share this PD among multiple connections. To use this PD, pass in NULL into rdma_create_qp(). The rdma_cm_id will reference the shared PD. That's great to know; thanks Sean (and everyone else who confirmed that a shared PD and possibly CQ is the way to go). Is there a way to reference this shared PD in ibv_reg_mr() as well? It seems that using NULL there too is not the solution, and I'm having difficulty tracing back from struct rdma_cm_id to find out where that shared PD is stored. I'd prefer to use this solution for simplicity's sake, so any additional details would be greatly appreciated. For what it's worth, I tried passing NULL to rdma_create_qp() as you suggested, and found it to work admirably. struct rdma_cm_id *id; ibv_reg_mr(id-pd, ...); -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Sharing MR Between Multiple Connections
On Wed, Nov 14, 2012 at 1:19 PM, Hefty, Sean sean.he...@intel.com wrote: The rdma_cm will automatically allocate one PD per RDMA device. You can share this PD among multiple connections. To use this PD, pass in NULL into rdma_create_qp(). The rdma_cm_id will reference the shared PD. That's great to know; thanks Sean (and everyone else who confirmed that a shared PD and possibly CQ is the way to go). Is there a way to reference this shared PD in ibv_reg_mr() as well? It seems that using NULL there too is not the solution, and I'm having difficulty tracing back from struct rdma_cm_id to find out where that shared PD is stored. I'd prefer to use this solution for simplicity's sake, so any additional details would be greatly appreciated. For what it's worth, I tried passing NULL to rdma_create_qp() as you suggested, and found it to work admirably. struct rdma_cm_id *id; ibv_reg_mr(id-pd, ...); It actually turned out to be id-qp-pd for me for some reason; the rdma_cm_id struct on my machine has no pd member. But other than that, all is working great. Thanks again. Christopher -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Sharing MR Between Multiple Connections
Hi, I am working on building an Infiniband application with a server that can handle many simultaneous clients. The server exposes a chunk of memory that each of the clients can read via RDMA. I was previously creating a new MR on the server for each client (and of course in that connection's PD). However, under stress testing, I realized that ibv_reg_mr() started failing after I simultaneously MRed the same area enough times to cover 20.0 GB. I presume that the problem is reaching some pinning limit, although ulimit reports unlimited for all relevant possibilities. I tried creating a single global PD and a single MR to be shared among the multiple connections, but rdma_create_qp() fails with an invalid argument when I try to do that. I therefore deduce that the PD specified in rdma_create_qp() must correspond to an active connection, not simply be created by opening a device. Long question short: is there any way I can share the same MR among multiple clients, so that my shared memory region is limited to N bytes instead of N/C (clients) bytes? Thanks in advance, Christopher end -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Sharing MR Between Multiple Connections
Please forgive my accidental faux-pas of a double-post. I incorrectly thought my use of a non-subscribed email address would prevent my post from getting through. On Wed, Nov 14, 2012 at 12:47 AM, Christopher Mitchell kermmart...@gmail.com wrote: Hi, [snip] Thanks in advance, Christopher end -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
