Two advantages come to mind vs an RDMA Write followed by a SEND: Using a SEND will consume a second WQE on the send side, and the synchronizing SEND will cause an entire new transaction, which will consume a(n infinitesimally) small amount of additional wire bandwidth, as well as incurring a(infinitesimally) small likelihood of a dropped or lost packets.
Nits? Yes, probably infinitesimally small ones. (hardware guys tend to worry about the small ones.) To answer Andy's original question, the behavior on the receive side is not guaranteed until control of the receive buffer has been formally returned to the receiver. I expect that most HCAs are pretty well behaved here, as are most CPU/memory/root complexes...but you never know. Can anybody guarantee that the inbound packet gets written to the memory in order? If something odd did happen, it seems like one of those places that would require an incredible stroke of luck to debug. OTOH, I know that many applications simply poll the receive buffer looking for a flag everyday and get away with it. -----Original Message----- From: Jason Gunthorpe [mailto:jguntho...@obsidianresearch.com] Sent: Tuesday, February 16, 2010 5:12 PM To: Paul Grun Cc: 'Andy Grover'; linux-rdma@vger.kernel.org Subject: Re: is it possible to avoid syncing after an rdma write? On Tue, Feb 16, 2010 at 05:05:21PM -0800, Paul Grun wrote: > Why not use an RDMA write w/ immed? That forces the consumption of a > receive WQE and can be used to create a completion event. Since the > immediate data is carried in the last packet of a multi-packet RDMA write, > you are guaranteed that all data has been placed in the receive buffer, in > order. Yes, RDMA WRITE w/ immediate data is perfectly fine. I've even implemented some protocols that use it to good effect. Not sure what the performance trade off is like though. The immediate data pretty much behaves exactly like a SEND WC on the receive side, but there may be some performance and latency advantages, particularly on the send side. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
