I’d like to see our NFS server use the local DMA lkey where it makes sense, to avoid the cost of registering and invalidating memory. I have to agree with Tom that once the device’s s/g limit is exceeded, the server has to post an RDMA Read WR every few pages, and appears to get expensive for large NFS requests. The current mechanism of statically choosing either FRMR or local DMA depending on the device is probably not adequate. Maybe we could post all the Read WRs via a single chain? Or stick with FRMR when the amount of data to read is significant. I’ve also tested Christoph’s patch. The logic currently in rdma_read_chunk_lcl does not seem up to the task. Once the device’s s/g limit is exceeded, the server starts throwing local length exceptions, and the client workload hangs.
This is probably because this code path wasn't reached/tested for a long time as it's hard to find a device that doesn't support FRWR for a long time now... -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
