On Fri, Jul 24, 2015 at 03:59:06PM -0400, Chuck Lever wrote: > And RPC-over-RDMA version 1 does not have any way to signal that > the server has invalidated the MRs. Such signaling would be a > pre-requisite to allow the Linux NFS/RDMA client to interoperate > with non-Linux NFS/RDMA servers that do not have such support.
You can implement client support immediately, nothing special is required. When processing a SEND WC check ex.invalidate_rkey and IB_WC_WITH_INVALIDATE. If that rkey matches the MR associated with that RPC slot then skip the invalidate. No protocol negotiation is required at that point. I am unclear what happens sever side if the server starts issuing SEND_WITH_INVALIDATE to a client that doesn't expect it. The net result is a MR would be invalidated twice. I don't know if this is OK or not. If it is OK, then the server can probably just start using it as well without negotiation. Otherwise the client has to signal the server it supports it once at connection setup. > For FRWR, you could post LINV from the receive completion upcall > handler, and handle the rest of the invalidation from the send > completion upcall, then poke the RPC reply handler. Yes > But this wouldn’t work at all for FMR, whose unmap verb is > synchronous, would it? It could run the FMR unmap in a thread/workqueue/tasklet and then complete the RPC side from that context. Same basic idea, using your taslket not the driver's sendq context. > I’m not sure we’d buy more than a few microseconds here, and > the receive upcall is single-threaded. Not sure on how that matches your performance goals, just remarking that lauching the invalidate in the recv upcall and completing processing from the sendq upcall is the very best performance you can expect from this API. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
