Hello Fab,
Fab Tillier wrote:
Hi Frank,
The fact that the Windows and Linux stacks treat the rkey differently in the SW
should not affect wire interop. In the Windows stack, any values that end up
on the wire (like the RKEY in the RDMA header) are always expressed in network
order. They're tokens anyhow and swapping them isn't optimal - you end up with
the HCA driver swapping it when returning it to the user, just so the user can
swap it to put it on the wire, so that the recipient can swap it back to host
order, just to have the HCA driver swap it back to network order when
performing the RDMA operation.
So on the Linux side, applications have to swap the rkey and treat it as a
host-order value. On the Windows side they don't. Can you elaborate on what
prevents clean RDMA interop between Linux and Windows?
-Fab
I have an application that runs on windows and linux. The code is the
same. If I do linux<->linux, or windows<->windows, it works. However
when I do linux<->windows, I have to swap the rkey for rdma operations.
To me it seems that either linux or windows is byteswapping the rkey
that's returned to userspace. The current fix is for my app to byteswap
the rkey when the memory is registered and byteswap it again when
posting the rdma work request, but only on one of the OS.
Unfortunateley I didn't have the time to fully investigate with an cable
analyzer to see who is misbehaving or if my analysis is actually correct.
Frank.
_______________________________________________
ofw mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
