Or, >I don't see why the ib uverbs flow (BTW - the data path has nothing to do with >the >rdma_cm, you're working with /dev/infiniband/uverbsX), can't be enhanced >e.g to support shared-page which is allocated && mmaped from uverbs to >user space and used in the same manner your implementation does.
The problem that I see is that the mmap is currently used for mapping of doorbell page in different drivers. We can use it for mapping a page for transmit/receive operation when we are able to differentiate that we need to map Doorbell or our shared page. The second problem is that this rx/tx mmap should map the separate page per QP to avoid the unnecessary QP lookups so page identifier passed to mmap should be based on QP identifier. I cannot find a specific code for /dev/infiniband/uverbsX. Is this device driver sharing the same functions like /dev/infiniband/rdmacm or it has own implementation. Mirek -----Original Message----- From: Or Gerlitz [mailto:ogerl...@voltaire.com] Sent: Wednesday, December 01, 2010 9:12 AM To: Walukiewicz, Miroslaw; Jason Gunthorpe; Roland Dreier Cc: Roland Dreier; Hefty, Sean; linux-rdma@vger.kernel.org Subject: Re: ibv_post_send/recv kernel path optimizations On 11/26/2010 1:56 PM, Walukiewicz, Miroslaw wrote: > Form the trace it looks like the __up_read() - 11% wastes most of time. It is > called from idr_read_qp when a put_uobj_read is called. if > (copy_from_user(&cmd, buf, sizeof cmd)) - 5% it is called twice from > ib_uverbs_post_send() for IMA and once in ib_uverbs_write() per each frame... > and __kmalloc/kfree - 5% is the third function that has a big meaning. It is > called twice for each frame transmitted. It is about 20% of performance loss > comparing to nes_ud_sksq path which we miss when we use a OFED path. > > What I can modify is a kmalloc/kfree optimization - it is possible to make > allocation only at start and use pre-allocated buffers. I don't see any way > for optimalization of idr_read_qp usage or copy_user. In current approach we > use a shared page and a separate nes_ud_sksq handle for each created QP so > there is no need for any user space data copy or QP lookup. As was mentioned earlier on this thread, and repeated here, the kmalloc/kfree can be removed, as or the 2nd copy_from_user, I don't see why the ib uverbs flow (BTW - the data path has nothing to do with the rdma_cm, you're working with /dev/infiniband/uverbsX), can't be enhanced e.g to support shared-page which is allocated && mmaped from uverbs to user space and used in the same manner your implementation does. The 1st copy_from_user should add pretty nothing and if it does, it can be replaced with different user/kernel IPC mechanism which costs less. So we're basically remained with the idr_read_qp, I wonder what other people think if/how this can be optimized? Or. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
