Sagi
As long as legA, legB and the RC are all connected to the same switch then
ordering will be preserved (I think many other topologies also work). Here is
how it would work for the problem case you are concerned about (which is a read
from the NVMe drive).
1. Disk device DMAs out the data to the p2pmem device via a string of PCIe
MemWr TLPs.
2. Disk device writes to the completion queue (in system memory) via a MemWr
TLP.
3. The last of the MemWrs from step 1 might have got stalled in the PCIe switch
due to congestion but if so they are stalled in the egress path of the switch
for the p2pmem port.
4. The RC determines the IO is complete when the TLP associated with step 2
updates the memory associated with the CQ. It issues some operation to read the
p2pmem.
5. Regardless of whether the MemRd TLP comes from the RC or another device
connected to the switch it is queued in the egress queue for the p2pmem FIO
behind the last DMA TLP (from step 1).
PCIe ordering ensures that this MemRd cannot overtake the MemWr (Reads can
never pass writes).
Therefore the MemRd can never get to the p2pmem device until after the last DMA
MemWr has.
What you are saying is surprising to me. The switch needs to preserve
ordering across different switch ports ??
You are suggesting that there is a *switch-wide* state that tracks
MemRds never pass MemWrs across all the switch ports? That is a very
non-trivial statement...
