See section 4 in the paper called "High Performance RDMA Based MPI Implementation over
InfiniBand" on the MVAPICH web page for description of one implementation that polls on data
buffers. Specifically, look at text around the statement "Although the approach uses the
in-order implementation of hardware for RDMA write which is not specified in the InfiniBand
standard, this feature is very likely to be kept by different hardware designers." Although
this paper is describing a PCI-X implementation, the feature is also exists on PCIe.
It's assumed that the host memory interconnect complies with statements described in the
"Update Ordering and Granularity Provided by a Write Transaction" of the PCI
spec. This particular application depends on PCI WRITE behavior, not READ.
Does this help?
Jason Gunthorpe wrote:
On Wed, Nov 11, 2009 at 05:44:59PM -0500, Richard Frank wrote:
Would anyone like to through out the list of HCAs that do this... I
can guess at a few... and can ask the vendors directly.. if not.. .
It would be much nicer to not hardcode names of adapters.. but that won't
stop us.. :)
Isn't it more complex than this? AFAIK the PCI-E standard does not
specify the order which data inside a single transfer becomes visible,
only how different transfers relate. To work on the most agressive
PCI-E system the HCA would have to transfer the last XX bytes as a
seperate PCI-E transaction without relaxed ordering.
This is the sort of thing that might start to matter on QPI and HT
memory-interleaved configurations. A multi-cache line transfer will be
split up and completed on different chips - it may not be fully
coherent 100% of the time.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
