On Nov 20, 2014, at 1:24 PM, Shirley Ma <shirley...@oracle.com> wrote:
> Attendees: > > Jeff Becker (NASA) > Yan Burman (Mellanox) > Wendy Cheng (Intel) > Rupert Dance (Soft Forge) > Steve Dickson (Red Hat) > Chuck Lever (Oracle) > Doug Ledford (RedHat) > Shirley Ma (Oracle) > Sachin Prabhu (RedHat) > Devesh Sharma (Emulex) > Anna Schumaker (Net App) > Steve Wise (OpenGridComputing, Chelsio) > > Moderator: > Shirley Ma (Oracle) > > NFSoRDMA developers bi-weekly meeting is to help organizing NFSoRDMA > development and test effort from different resources to speed up NFSoRDMA > upstream kernel work and NFSoRDMA diagnosing/debugging tools development. > Hopefully the quality of NFSoRDMA upstream patches can be improved by being > tested with a quorum of HW vendors. > > Today's meeting notes: > > NFSoRDMA performance: > --------------------- > Even though NFSoRDMA performance seems better than IPoIB-cm, the gap between > what the IB protocol can provide and what NFS(RDMA,IPoIB-cm) can achieve is > still big on small I/O block size (focused on 8K IO size for database > workload). Even large I/O block size(128K above), NFS performance is not > comparable to RDMA microbenchmark. We are focusing the effort to figure out > the root cause. Several experimental methods have been used on how to improve > NFSoRDMA performance. > > Yan saw NFS server does RDMA send for small packet size, less than 100bytes, > which should have used post_send instead. This is an artifact of how NFS/RDMA works. The client provides a registered area for the server to write into if an RPC reply is larger than the small pre-posted buffers that are normally used. Most of the time, each RPC reply is small enough to use RDMA SEND, and the server can convey the RPC/RDMA header and the RPC reply in a single SEND operation. If the reply is large, the server conveys the RPC/RDMA header via RDMA send, and the RPC reply via an RDMA WRITE into the client’s registered buffer. Solaris server chooses RDMA SEND in nearly every case. Linux server chooses RDMA SEND then RDMA WRITE whenever the client offers that choice. Originally, it was felt that doing the RDMA WRITE is better for the client because the client doesn’t have to copy the RPC header from the RDMA receive buffer back into rq_rcv_buf. Note that the RPC header is generally just a few hundred bytes. Several people have claimed that RDMA WRITE for small I/O is relatively expensive and should be avoided. It’s also expensive for the client to register and deregister the receive buffer for the RDMA WRITE if the server doesn’t use it. I’ve explored changing the client to offer no registered buffer if it knows the RPC reply will be small, thus forcing the server to use RDMA SEND where it’s safe. Solaris server worked fine. Of course, it already works this way. Linux server showed some data and metadata corruption on complex workloads like kernel builds. There’s a bug in there somewhere that will need to be addressed before we can change the client behavior. The improvement was consistent, but under ten microseconds per RPC with FRWR (more with FMR because deregistering the buffer takes longer and is synchronous with RPC execution). At this stage, there are bigger problems to be addressed, so this is not a top priority. -- Chuck Lever chuck[dot]lever[at]oracle[dot]com -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
