Re: kernel space rdma client, reg_mr issue
Benoit Hudzia benoit.hud...@gmail.com wrote on 06/30/2011 08:19:17 PM: Hi, Thanks for the reply , i will look into the get_dma_mr() path for registering memory. And probably get back to all of you for some clarification if i get stuck. On the same note, 2 things i noticed while using softiwarp: 1. the current version of softiwarp ( from the master git ) doesn't support the latest kernel version (3.0.0 rc4) as it still using the IW_CM_EVENT_STATUS_xxx values that have been removed (https://patchwork.kernel.org/patch/772712/) in the latest kernel iteration. Would you like me to submit a patch as we had to change the code already ? i posted a new siw version to linux-rdma and netdev about 2 weeks ago. that patch set already changes to errno values for IWCM status. i will update gitorious.org in a few days. in the long run i hope quality and completeness of siw will make it acceptable for mainline integration, which would likely make gitorious.org/softiwarp superfluous. 2: i noticed that sofiwarp crash the kernel when you post a work request with an invalid rkey from userspace . I will try to reproduce the crash and send you some code. Maybe if i have time a patch snip any crash reports are highly appreciated (well... ;-) many thanks! Bernard. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
kernel space rdma client, reg_mr issue
Hi,
We are working to create a RDMA client within the kernel which will
connect to a working RDMA userspace server.
We are using the userspace code for testing purpose , in the final
version all the communication will be done within the kernel space.
We rely on softiwarp so far for dev purpose.
We have achieved a connection but are now having trouble prior to the
communication stage.
We are trying to register a user space created buffer using the
device-reg_user_mr . However the kernel crash and the crash dump
shows a page fault after siw_reg_user_mr.
What we would like to know:
If this is the correct approach to register memory for RDMA
communication ? If yes what are we doing wrong ?
Is there a possibility to use kernel allocated memory rather than
using userspace created and then passing the address back to the
kernel through IOCTL ? ( if yes if there is any code example floating
around?)
basically the question boils down to how to allcoate and registr
buffer for RDMA communication from inside a kernel module?
This is the code.
##
struct rdma_cm_id *id = rdma_create_id(my_rdma_cm_event_handler
event_handler, NULL, RDMA_PS_TCP, IB_QPT_RC);
struct ib_pd *pd = ib_alloc_pd(id-device);
struct ib_cq *cq = ib_create_cq(id-device, NULL, NULL, NULL, 2, 0);
// Userspace buffer addresses passed via IOCTl
unsigned long user_buffer_send;
unsigned long user_buffer_recv;
int my_rdma_cm_event_handler event_handler(struct rdma_cm_id *id,
struct rdma_cm_event *event)
{
switch (event)
{
case RDMA_CM_EVENT_ADDR_RESOLVED:
rdma_resolve_route(id, 2000);
break;
case RDMA_CM_EVENT_ROUTE_RESOLVED:
{
struct ib_qp_init_attr attr;
memset(attr, 0, sizeof attr);
attr.send_cq = cq;
attr.recv_cq = cq;
attr.sq_sig_type = IB_SIGNAL_ALL_WR;
attr.cap.max_send_wr = 2;
attr.cap.max_recv_wr = 2;
attr.cap.max_send_sge = 1;
attr.cap.max_recv_sge = 1;
attr.qp_type = IB_QPT_RC;
attr.port_num = id-port_num;
attr.event_handler = my_qp_event_handler;
rdma_create_qp(id, pd, attr)
rdma_connect(id, conn_param);
break;
}
case RDMA_CM_EVENT_ESTABLISHED:
{
struct ib_mr *send_mr = id-device-reg_user_mr(pd,
(u64)
user_buffer_send, sizeof *user_buffer_send, NULL,
IB_ACCESS_LOCAL_WRITE | IB_ACCESS_REMOTE_READ |
IB_ACCESS_REMOTE_WRITE, NULL);
struct ib_mr *recv_mr = id-device-reg_user_mr(pd,
(u64)
user_buffer_recv, sizeof *user_buffer_recv, NULL,
IB_ACCESS_LOCAL_WRITE | IB_ACCESS_REMOTE_READ |
IB_ACCESS_REMOTE_WRITE, NULL);
break;
}
}
...
}
##
--
The production of too many useful things results in too many useless people
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: kernel space rdma client, reg_mr issue
On Thu, Jun 30, 2011 at 7:19 AM, Benoit Hudzia benoit.hud...@gmail.com wrote: Hi, We are working to create a RDMA client within the kernel which will connect to a working RDMA userspace server. We are using the userspace code for testing purpose , in the final version all the communication will be done within the kernel space. [snip] basically the question boils down to how to allcoate and registr buffer for RDMA communication from inside a kernel module? I happen to have a similar setup ..The original intent was to have a kernel mode RDMA application that took the kernel data and sent it over to the peer node's memory for temporary storage. It had to be able to get read back later. As it didn't matter whether the temporary storage was in kernel or user address space, I re-used my colleague's existing user mode program (to run as an user space daemon on the peer node). This allowed the focus being on the new kernel application development (run on the primary node). After the code was up and running, I see no reason to change the setup and it has been running fine since. The code runs on RHEL 5.5 with OFED-1.5.2 using Mellanox card. The user mode daemon is in a forever receiving loop that follows the standard the RDMA user mode programming logic. The kernel code invokes the ib_xxx set of APIs (vs. user mode ibv_xxx(s)). The kernel memory registration are done by the APIs such as kzalloc(), ib_get_dma_mr(), ib_dma_map_single(), ib_dma_map_page(), etc. Check out the driver code in drivers/infiniband/ulp/iser directory. It has a sample logic to register kernel memory. -- Wendy -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: kernel space rdma client, reg_mr issue
Hi Benoit,
since a user process should never crash the kernel there
is something wrong in softiwarp. Can you please send me the
stack trace from the crash in privae email to work that out.
I'll come back to the list with a fix.
Thanks,
Bernard.
linux-rdma-ow...@vger.kernel.org wrote on 06/30/2011 04:19:01 PM:
Hi,
We are working to create a RDMA client within the kernel which will
connect to a working RDMA userspace server.
We are using the userspace code for testing purpose , in the final
version all the communication will be done within the kernel space.
We rely on softiwarp so far for dev purpose.
We have achieved a connection but are now having trouble prior to the
communication stage.
We are trying to register a user space created buffer using the
device-reg_user_mr . However the kernel crash and the crash dump
shows a page fault after siw_reg_user_mr.
What we would like to know:
If this is the correct approach to register memory for RDMA
communication ? If yes what are we doing wrong ?
Is there a possibility to use kernel allocated memory rather than
using userspace created and then passing the address back to the
kernel through IOCTL ? ( if yes if there is any code example floating
around?)
basically the question boils down to how to allcoate and registr
buffer for RDMA communication from inside a kernel module?
This is the code.
##
struct rdma_cm_id *id = rdma_create_id(my_rdma_cm_event_handler
event_handler, NULL, RDMA_PS_TCP, IB_QPT_RC);
struct ib_pd *pd = ib_alloc_pd(id-device);
struct ib_cq *cq = ib_create_cq(id-device, NULL, NULL, NULL, 2, 0);
// Userspace buffer addresses passed via IOCTl
unsigned long user_buffer_send;
unsigned long user_buffer_recv;
int my_rdma_cm_event_handler event_handler(struct rdma_cm_id *id,
struct rdma_cm_event *event)
{
switch (event)
{
case RDMA_CM_EVENT_ADDR_RESOLVED:
rdma_resolve_route(id, 2000);
break;
case RDMA_CM_EVENT_ROUTE_RESOLVED:
{
struct ib_qp_init_attr attr;
memset(attr, 0, sizeof attr);
attr.send_cq = cq;
attr.recv_cq = cq;
attr.sq_sig_type = IB_SIGNAL_ALL_WR;
attr.cap.max_send_wr = 2;
attr.cap.max_recv_wr = 2;
attr.cap.max_send_sge = 1;
attr.cap.max_recv_sge = 1;
attr.qp_type = IB_QPT_RC;
attr.port_num = id-port_num;
attr.event_handler = my_qp_event_handler;
rdma_create_qp(id, pd, attr)
rdma_connect(id, conn_param);
break;
}
case RDMA_CM_EVENT_ESTABLISHED:
{
struct ib_mr *send_mr = id-device-reg_user_mr(pd, (u64)
user_buffer_send, sizeof *user_buffer_send, NULL,
IB_ACCESS_LOCAL_WRITE | IB_ACCESS_REMOTE_READ
|
IB_ACCESS_REMOTE_WRITE, NULL);
struct ib_mr *recv_mr = id-device-reg_user_mr(pd, (u64)
user_buffer_recv, sizeof *user_buffer_recv, NULL,
IB_ACCESS_LOCAL_WRITE | IB_ACCESS_REMOTE_READ
|
IB_ACCESS_REMOTE_WRITE, NULL);
break;
}
}
...
}
##
--
The production of too many useful things results in too many useless
people
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: kernel space rdma client, reg_mr issue
linux-rdma-ow...@vger.kernel.org wrote on 06/30/2011 04:19:01 PM: Hi, maybe i misread your last email - are you trying to use reg_user_mr() as a kernel client? If yes - that will not work. You should use the interface as described by Wendy (get_dma_mr() etc.), since you are dealing with kernel memory. I can add a check to siw to prevent that call into siw_reg_user_mr() to be executed for kernel clients. thanks, Bernard. We are working to create a RDMA client within the kernel which will connect to a working RDMA userspace server. We are using the userspace code for testing purpose , in the final version all the communication will be done within the kernel space. snip -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: kernel space rdma client, reg_mr issue
Hi, Thanks for the reply , i will look into the get_dma_mr() path for registering memory. And probably get back to all of you for some clarification if i get stuck. On the same note, 2 things i noticed while using softiwarp: 1. the current version of softiwarp ( from the master git ) doesn't support the latest kernel version (3.0.0 rc4) as it still using the IW_CM_EVENT_STATUS_xxx values that have been removed (https://patchwork.kernel.org/patch/772712/) in the latest kernel iteration. Would you like me to submit a patch as we had to change the code already ? 2: i noticed that sofiwarp crash the kernel when you post a work request with an invalid rkey from userspace . I will try to reproduce the crash and send you some code. Maybe if i have time a patch Regards Benoit On 30 June 2011 17:34, Bernard Metzler b...@zurich.ibm.com wrote: linux-rdma-ow...@vger.kernel.org wrote on 06/30/2011 04:19:01 PM: Hi, maybe i misread your last email - are you trying to use reg_user_mr() as a kernel client? If yes - that will not work. You should use the interface as described by Wendy (get_dma_mr() etc.), since you are dealing with kernel memory. I can add a check to siw to prevent that call into siw_reg_user_mr() to be executed for kernel clients. thanks, Bernard. We are working to create a RDMA client within the kernel which will connect to a working RDMA userspace server. We are using the userspace code for testing purpose , in the final version all the communication will be done within the kernel space. snip -- The production of too many useful things results in too many useless people -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
