Jason Gunthorpe wrote: > On Tue, Feb 16, 2010 at 03:29:48PM -0800, Andy Grover wrote: >> Right now, RDS follows each RDMA write op with a Send op, which 1) >> causes an interrupt and 2) includes the info we need to call >> ib_dma_sync_sg_for_cpu() for the target of the rdma write. >> >> We want to omit the Send. If we don't do the sync on the machine that is >> the target of the RDMA write, the result is... what exactly? I assume >> the write to memory is snooped by CPUs, so their cachelines will be >> properly invalidated. However, Linux DMA-API docs seem pretty clear in >> insisting on the sync.
<snip> > What do you intend to replace the SEND with? spin on last byte? There > are other issues to consider like ordering within the PCI-E fabric.. Well, hopefully nothing. What I'm looking for is to write to a target region multiple times, as efficiently as possible, but be able to occasionally read it on the target machine and get consistent results. I definitely don't want to take an event, and avoiding the CQE would be nice. What I'm hearing is that I don't have to worry about what the Linux DMA-API docs say about noncoherent mappings, but I need to be mindful of IB spec 9.5 section o9-20: --- o9-20: An application shall not depend of the contents of an RDMA WRITE buffer at the responder until one of the following has occurred: * Arrival and Completion of the last RDMA WRITE request packet when used with Immediate data. * Arrival and completion of a subsequent SEND message. * Update of a memory element by a subsequent ATOMIC operation. --- So if I do an RDMA write and follow it up with an atomic op, it sounds like I can achieve the behavior I want, and without an event or CQE. Although for my particular use case with ongoing writes, the CPU couldn't fetch more than one value (64bit?) without potentially reading data from a later write, I would think. Regards -- Andy -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
