This series adds TLP Processing Hints (TPH) support to the VFIO dma-buf
export path, allowing importing drivers (e.g. mlx5) to use the
exporter's steering tag when performing peer-to-peer DMA into a
VFIO-owned device.
There is no separate in-tree vendor kernel driver for the target device:
vfio-pci is the in-tree driver and the targeted device is managed
from userspace via VFIO passthrough. That is why the ST has to flow
through a uAPI: userspace owns the device and its ST table, so it is the
entity that can configure a meaningful value for a given dma-buf. The
kernel-visible participants are still in-tree: vfio-pci exports the
dma-buf and mlx5 imports it.
On the effect: the endpoint's PCIe ingress block uses the ST as
an in-band instruction for the incoming P2P TLP -- selecting a target
cache partition and, on writes, an in-flight operation on the data
before it lands. The dma-buf callback keeps this opaque to the
framework -- only the producer (userspace owner of the VFIO device)
and the consumer (endpoint block) need to interpret the value. The
dma-buf get_pci_tph callback itself is optional, but workloads that
depend on the endpoint's in-flight operation need it because fallback
does not produce the same result.
The dma-buf hook is intentionally generic and discoverable rather than
a private side channel. The exporter owns the completing address
space for the dma-buf and decides whether it can provide a meaningful
ST/PH tuple for that completer; the dma-buf core keeps the tuple opaque,
and importers merely request the namespace they support and place the
returned value on generated TLPs. Exporters that cannot derive a
meaningful tuple simply return -EOPNOTSUPP.
Patch 1 adds small PCI/TPH type helpers so drivers can query the enabled
TPH requester mode and the device's TPH Completer Supported field
without reaching into pci_dev internals (and so callers in
CONFIG_PCIE_TPH=n builds get a clean fallback). pcie_tph_completer_type()
reuses the reserved-encoding fold introduced by the separately-submitted
folding patch rather than duplicating the completer decode.
Patch 2 adds the optional dma_buf_ops::get_pci_tph callback plus the
dma_buf_get_pci_tph() importer wrapper so importers can fetch TPH
metadata from an exporter under dmabuf->resv.
Patch 3 implements get_pci_tph in vfio-pci and adds the new uAPI
(VFIO_DEVICE_FEATURE_DMA_BUF_TPH) for userspace to attach the metadata.
Patch 4 wires up the mlx5 RDMA driver as a consumer. It also enforces the
dma_buf_get_pci_tph() steering-tag lifetime: the tag is only valid for the
mapping it was queried against, and the mkey's TPH fields cannot be
reprogrammed in place. mlx5 therefore records the registration-time tuple
and re-queries after each dma-buf mapping is established under dmabuf->resv;
unchanged tuples continue with the existing mkey, while changed or missing
tuples fail the remap rather than continue with a stale hint. For vfio-pci
BAR dma-bufs this is expected to be a no-op because invalidation is
revoke/quiesce, not movement to a new backing placement, and the
userspace-provided tuple is not changed by the revoke/un-revoke path.
Build-tested with both CONFIG_PCIE_TPH=y and CONFIG_PCIE_TPH=n.
Functional validation on the target topology: PCIe analyzer captures
on the P2P TLPs confirm the ST emitted by mlx5 matches the value
configured through VFIO_DEVICE_FEATURE_DMA_BUF_TPH, and the end-to-end
P2P workload only produces results consistent with the endpoint's
ST-selected in-flight operation. For example, with userspace
configuring 8-bit ST=0xf0 and PH=2, an analyzer capture of a peer-to-
peer MWr64 shows "STP MWr64 TC=0 OHC=2 ..." followed by "OHC-B
ST=F0h PH=2 HV=1":
(TLP Captures)
08000260 -> STP MWr64 TC=0 OHC=2 TS=0 Attr=0 L=8
F0000004 -> RID=4h:0h.0h EP- Tag=F0h
E0200000 -> AddrH=000020E0h
00080006 -> AddrL=06000800h
90F00000 -> OHC-B ST=F0h PH=2 HV=1 AMA=0 AV-
The dma-buf get_pci_tph interface has also been exercised by a second,
independent importer: a different vendor's NIC whose driver is not yet
upstream, locally taught to call dma_buf_get_pci_tph(). A PCIe analyzer
confirmed the ST it placed on outbound P2P TLPs matches the value
configured through VFIO_DEVICE_FEATURE_DMA_BUF_TPH, the same result as
with mlx5. Two unrelated importer drivers exercising the callback
end-to-end shows the interface is not tied to a single consumer. That
importer change is out-of-tree and not part of this series. For that
second importer, with userspace configuring 8-bit ST=0xe0 and PH=0,
an analyzer capture shows:
(TLP Captures)
08200260 -> STP MWr64 TC=0 OHC=2 TS=1 Attr=0 L=8
4E00004C -> RID=4Ch:0h.0h EP- Tag=4Eh
00170000 -> AddrH=00001700h
00200006 -> AddrL=06002000h
10E00000 -> OHC-B ST=E0h PH=0 HV=1 AMA=0 AV-
Depends on (submitted separately):
net/mlx5: free mlx5_st_idx_data on final dealloc
https://lore.kernel.org/linux-rdma/[email protected]
PCI/TPH: fold reserved completer encoding in get_rp_completer_type()
https://lore.kernel.org/linux-pci/[email protected]
Changes since v11:
Patch 1 (PCI/TPH): add pcie_tph_completer_type() and
pcie_tph_enabled_req_type(); pcie_tph_completer_type() reuses the
reserved-encoding fold from the separately-submitted folding patch
rather than duplicating the completer decode.
Patch 2 (dma-buf): fix the kernel-doc to reference
&dma_buf_attach_ops.invalidate_mappings instead of the non-existent
move_notify callback (reported by Sashiko and Alex Williamson).
Patch 3 (vfio/pci): hold memory_lock while validating dma-buf ownership
and publishing TPH metadata, so cleanup cannot disassociate the dma-buf
from the VFIO device under the SET path. Also use WRITE_ONCE() for
revoked-state writers to pair with the lockless attach-path READ_ONCE().
Patch 4 (mlx5): enforce the steering-tag lifetime documented in patch 2
(addresses the lifecycle gap Sashiko and Alex noted on v11). mlx5 now
stores the registration-time TPH tuple and re-queries the exporter after
each dma-buf mapping is established under dmabuf->resv. If the tuple is
unchanged, the existing mkey continues to be used. If it changed or
disappeared, mlx5 unmaps the pages and returns -EFAULT because the
existing mkey cannot be updated in place (no UMR update mask for TPH
fields, and rebuilding the mkey would change the rkey/lkey). The exporter
query runs under dmabuf->resv, but the ST-index allocation is done
outside it so no GFP_KERNEL allocation is held under dmabuf->resv. For
vfio-pci BAR dma-bufs this validation is expected to be a no-op because
the tuple is stable across revoke/un-revoke.
Previous link:
v11:
https://lore.kernel.org/linux-pci/[email protected]/
v10:
https://lore.kernel.org/linux-pci/[email protected]/
v9:
https://lore.kernel.org/dri-devel/[email protected]/
v8:
https://lore.kernel.org/dri-devel/[email protected]/
v7:
https://lore.kernel.org/dri-devel/[email protected]/
v6:
https://lore.kernel.org/dri-devel/[email protected]/
v5:
https://lore.kernel.org/dri-devel/[email protected]/
v4:
https://lore.kernel.org/linux-pci/[email protected]/
v3:
https://lore.kernel.org/linux-pci/[email protected]/
v2: https://lore.kernel.org/linux-pci/[email protected]/
Zhiping Zhang (4):
PCI/TPH: Add requester/completer type helpers
dma-buf: add optional get_pci_tph() callback
vfio/pci: implement get_pci_tph and DMA_BUF_TPH feature
RDMA/mlx5: get tph for p2p access when registering dma-buf mr
drivers/dma-buf/dma-buf.c | 32 ++++
drivers/infiniband/hw/mlx5/main.c | 1 +
drivers/infiniband/hw/mlx5/mlx5_ib.h | 11 ++
drivers/infiniband/hw/mlx5/mr.c | 151 +++++++++++++++++--
drivers/infiniband/hw/mlx5/odp.c | 7 +
.../net/ethernet/mellanox/mlx5/core/lib/st.c | 49 +++++-
drivers/pci/tph.c | 39 ++++++
drivers/vfio/pci/vfio_pci_core.c | 3 +
drivers/vfio/pci/vfio_pci_dmabuf.c | 117 ++++++++++++-
drivers/vfio/pci/vfio_pci_priv.h | 13 ++
include/linux/dma-buf.h | 25 +++
include/linux/mlx5/driver.h | 15 ++
include/linux/pci-tph.h | 8 +
include/uapi/linux/vfio.h | 43 +++++
14 files changed, 496 insertions(+), 18 deletions(-)
--
2.53.0-Meta
