Roland, You are right that the most of the speed-up is coming from avoid semaphores, but not only.
>From the oprof traces, the semaphores made half of difference. The next one was copy_from_user and kmalloc/kfree usage (in my proposal - shared page method is used instead) In my opinion, the responsibility for cases like protection of QP against destroy during buffer post (and other similar cases) should be moved to vendor driver. The OFED code should move only the code path to driver. Mirek -----Original Message----- From: linux-rdma-ow...@vger.kernel.org [mailto:linux-rdma-ow...@vger.kernel.org] On Behalf Of Roland Dreier Sent: Wednesday, January 05, 2011 7:17 PM To: Walukiewicz, Miroslaw Cc: Or Gerlitz; Jason Gunthorpe; Hefty, Sean; linux-rdma@vger.kernel.org Subject: Re: ibv_post_send/recv kernel path optimizations > The patch for ofed-1.5.3 looks like below. I will try to push it to > kernel.org after porting. > > Now an uverbs post_send/post_recv path is modified to make pre-lookup > for RAW_ETH QPs. When a RAW_ETH QP is found the driver specific path > is used for posting buffers. for example using a shared page approach in > cooperation with user-space library I don't quite see why a hash table helps performance much vs. an IDR. Is the actual IDR lookup a significant part of the cost? (By the way, instead of list_head you could use hlist_head to make your hash table denser and save cache footprint -- that way an 8-entry table on 64-bit systems fits in one cacheline) Also it seems that you get rid of all the locking on QPs when you look them up in your hash table. What protects against userspace posting a send in one thread and destroying the QP in another thread, and ending up having the destroy complete before the send is posted (leading to use-after-free in the kernel)? I would guess that your speedup is really coming from getting rid of locking that is actually required for correctness. Maybe I'm wrong though, I'm just guessing here. - R. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
