Also, while it doesn't seem to have the same massive effect on
throughput, I can also see out of order behaviour happening when the
sending VM is on a node with a ConnectX-3 Pro NIC. Its driver is also
enabling XPS it would seem. I'm not *certain* but looking at the traces
it appears that with the ConnectX-3 Pro there is more interleaving of
the out-of-order traffic than there is with the Skyhawk. The ConnectX-3
Pro happens to be in a newer generation server with a newer processor
than the other systems where I've seen this.
I do not see the out-of-order behaviour when the NIC at the sending end
is a BCM57840. It does not appear that the bnx2x driver in the 4.4
kernel is enabling XPS.
So, it would seem that there are three cases of enabling XPS resulting
in out-of-order traffic, two of which result in a non-trivial loss of
performance.
happy benchmarking,
rick jones
