Hi Christian, Another ping on this series for your attention. Pls see below for more details.
Thanks, Zhiping On Fri, Aug 14, 2026 at 11:08 AM Zhiping Zhang <[email protected]> wrote: > > Hi Christian, > > A gentle ping on this series, especially patch 3, which adds the > optional dma-buf get_pci_tph() callback. Could you please review this > when you have a chance? > > https://lore.kernel.org/linux-pci/[email protected]/ > > Thanks, > Zhiping > > > On Fri, Jul 31, 2026 at 2:22 PM Zhiping Zhang <[email protected]> wrote: > > > > This series adds TLP Processing Hints (TPH) support to the VFIO dma-buf > > export path, allowing importing drivers (e.g. mlx5) to use the > > exporter's steering tag when performing peer-to-peer DMA into a > > VFIO-owned device. > > > > There is no separate in-tree vendor kernel driver for the target device: > > vfio-pci is the in-tree driver and the targeted device is managed > > from userspace via VFIO passthrough. That is why the ST has to flow > > through a uAPI: userspace owns the device and its ST table, so it is the > > entity that can configure a meaningful value for a given dma-buf. The > > kernel-visible participants are still in-tree: vfio-pci exports the > > dma-buf and mlx5 imports it. > > > > On the effect: the endpoint's PCIe ingress block uses the ST as > > an in-band instruction for the incoming P2P TLP -- selecting a target > > cache partition and, on writes, an in-flight operation on the data > > before it lands. The dma-buf callback keeps this opaque to the > > framework -- only the producer (userspace owner of the VFIO device) > > and the consumer (endpoint block) need to interpret the value. The > > dma-buf get_pci_tph callback itself is optional, but workloads that > > depend on the endpoint's in-flight operation need it because fallback > > does not produce the same result. > > > > The dma-buf hook is intentionally generic and discoverable rather than > > a private side channel. The exporter owns the completing address > > space for the dma-buf and decides whether it can provide a meaningful > > ST/PH tuple for that completer; the dma-buf core keeps the tuple opaque, > > and importers merely request the namespace they support and place the > > returned value on generated TLPs. Exporters that cannot derive a > > meaningful tuple simply return -EOPNOTSUPP. > > > > TPH is advisory: a steering tag that is not honored on the path (for > > example an intermediate routing element that does not forward the TPH > > prefix) is ignored and the request completes as an ordinary, > > non-TPH transaction (PCIe Base 6.4 sec 2.2.7.1). This series therefore > > targets the same-Root-Port / common-switch topology, where the ST > > reaches the completer; cross-Root-Port P2P is best-effort and is not > > gated in the uAPI, since supplying an unused ST is harmless and there > > is no discoverable "TPH routing" capability to test against. > > > > Patch 1 folds the reserved 0b10 "TPH Completer Supported" encoding into > > "not supported" in get_rp_completer_type(), so only architected values > > can reach the TPH Requester Enable field. It was previously posted > > standalone to linux-pci; per Alex Williamson's v12 review it now travels > > with the series, which removes the cross-tree ordering dependency and > > lets review tooling apply the series as posted. > > Patch 2 adds small PCI/TPH type helpers so drivers can query the enabled > > TPH requester mode and the device's TPH Completer Supported field > > without reaching into pci_dev internals (and so callers in > > CONFIG_PCIE_TPH=n builds get a clean fallback). pcie_tph_completer_type() > > applies the same reserved-encoding fold as get_rp_completer_type(), > > inlined locally so the helper is self-contained. > > Patch 3 adds the optional dma_buf_ops::get_pci_tph callback plus the > > dma_buf_get_pci_tph() importer wrapper so importers can fetch TPH > > metadata from an exporter under dmabuf->resv. > > Patch 4 implements get_pci_tph in vfio-pci and adds the new uAPI > > (VFIO_DEVICE_FEATURE_DMA_BUF_TPH) for userspace to attach the metadata. > > Patch 5 wires up the mlx5 RDMA driver as a consumer. It also enforces the > > dma_buf_get_pci_tph() steering-tag lifetime: the tag is only valid for the > > mapping it was queried against, and the mkey's TPH fields cannot be > > reprogrammed in place. mlx5 therefore records the registration-time tuple > > and re-queries after each dma-buf mapping is established under > > dmabuf->resv; unchanged tuples continue with the existing mkey, while > > changed or missing tuples fail the remap rather than continue with a > > stale hint. For vfio-pci BAR dma-bufs this is expected to be a no-op > > because invalidation is revoke/quiesce, not movement to a new backing > > placement, and the userspace-provided tuple is not changed by the > > revoke/un-revoke path. > > > > Build-tested with both CONFIG_PCIE_TPH=y and CONFIG_PCIE_TPH=n. > > Functional validation on the target topology: PCIe analyzer captures > > on the P2P TLPs confirm the ST emitted by mlx5 matches the value > > configured through VFIO_DEVICE_FEATURE_DMA_BUF_TPH, and the end-to-end > > P2P workload only produces results consistent with the endpoint's > > ST-selected in-flight operation. For example, with userspace > > configuring 8-bit ST=0xf0 and PH=2, an analyzer capture of a peer-to- > > peer MWr64 shows "STP MWr64 TC=0 OHC=2 ..." followed by "OHC-B > > ST=F0h PH=2 HV=1": > > (TLP Captures) > > 08000260 -> STP MWr64 TC=0 OHC=2 TS=0 Attr=0 L=8 > > F0000004 -> RID=4h:0h.0h EP- Tag=F0h > > E0200000 -> AddrH=000020E0h > > 00080006 -> AddrL=06000800h > > 90F00000 -> OHC-B ST=F0h PH=2 HV=1 AMA=0 AV- > > > > The dma-buf get_pci_tph interface has also been exercised by a second, > > independent importer: a different vendor's NIC whose driver is not yet > > upstream, locally taught to call dma_buf_get_pci_tph(). A PCIe analyzer > > confirmed the ST it placed on outbound P2P TLPs matches the value > > configured through VFIO_DEVICE_FEATURE_DMA_BUF_TPH, the same result as > > with mlx5. Two unrelated importer drivers exercising the callback > > end-to-end shows the interface is not tied to a single consumer. That > > importer change is out-of-tree and not part of this series. For that > > second importer, with userspace configuring 8-bit ST=0xe0 and PH=0, > > an analyzer capture shows: > > (TLP Captures) > > 08200260 -> STP MWr64 TC=0 OHC=2 TS=1 Attr=0 L=8 > > 4E00004C -> RID=4Ch:0h.0h EP- Tag=4Eh > > 00170000 -> AddrH=00001700h > > 00200006 -> AddrL=06002000h > > 10E00000 -> OHC-B ST=E0h PH=0 HV=1 AMA=0 AV- > > > > Changes since v12: > > Patch 1 (PCI/TPH, new to the series): the reserved-encoding fold, > > previously posted standalone to linux-pci [1], is now the first patch > > here (Alex Williamson). Sashiko could not apply v12 because of that > > external dependency; with the fold in-series and the mlx5 leak fix in > > linux-next, v13 has none. The code is unchanged from the standalone > > v3; the Fixes: tag is dropped, since no code path can reach the > > reserved encoding today and the patch is hardening rather than a fix > > for observed silicon. > > > > Patch 2 (PCI/TPH): inline the reserved-encoding fold in > > pcie_tph_completer_type() rather than calling the helper that earlier > > folding revisions added; that helper was dropped in folding v3 per > > Bjorn Helgaas and Wei Huang. > > > > Patch 3 (dma-buf): no functional change. > > > > Patch 4 (vfio/pci): also gate the DMA_BUF_TPH feature on > > vdev->pci_ops->get_dmabuf_phys, matching > > vfio_pci_core_feature_dma_buf(). Without it PROBE reported the feature > > as supported on a device that advertises TPH Completer support but > > cannot export a vfio dma-buf at all, so nothing could ever carry the > > metadata (Alex Williamson, who raised this to uAPI-affecting > > severity). > > > > Patch 5 (mlx5): keep the !dev->st early-out in mlx5_st_alloc_index() > > ahead of the pcie_tph_get_cpu_st() call, so splitting out > > mlx5_st_alloc_index_by_tag() neither adds an ACPI _DSM invocation on > > devices without ST support nor changes the errno userspace sees when > > the _DSM lookup fails (Alex Williamson). The commit message now > > describes this rather than presenting the split as a pure extraction. > > > > Previous link: > > v12: > > https://lore.kernel.org/linux-pci/[email protected]/ > > v11: > > https://lore.kernel.org/linux-pci/[email protected]/ > > v10: > > https://lore.kernel.org/linux-pci/[email protected]/ > > v9: > > https://lore.kernel.org/dri-devel/[email protected]/ > > v8: > > https://lore.kernel.org/dri-devel/[email protected]/ > > v7: > > https://lore.kernel.org/dri-devel/[email protected]/ > > v6: > > https://lore.kernel.org/dri-devel/[email protected]/ > > v5: > > https://lore.kernel.org/dri-devel/[email protected]/ > > v4: > > https://lore.kernel.org/linux-pci/[email protected]/ > > v3: > > https://lore.kernel.org/linux-pci/[email protected]/ > > v2: > > https://lore.kernel.org/linux-pci/[email protected]/ > > > > Zhiping Zhang (5): > > PCI/TPH: treat reserved 0b10 completer encoding as unsupported > > PCI/TPH: Add requester/completer type helpers > > dma-buf: add optional get_pci_tph() callback > > vfio/pci: implement get_pci_tph and DMA_BUF_TPH feature > > RDMA/mlx5: get tph for p2p access when registering dma-buf mr > > > > drivers/dma-buf/dma-buf.c | 32 ++++ > > drivers/infiniband/hw/mlx5/main.c | 1 + > > drivers/infiniband/hw/mlx5/mlx5_ib.h | 11 ++ > > drivers/infiniband/hw/mlx5/mr.c | 151 +++++++++++++++++- > > drivers/infiniband/hw/mlx5/odp.c | 7 + > > .../net/ethernet/mellanox/mlx5/core/lib/st.c | 52 +++++- > > drivers/pci/tph.c | 55 ++++++- > > drivers/vfio/pci/vfio_pci_core.c | 3 + > > drivers/vfio/pci/vfio_pci_dmabuf.c | 120 +++++++++++++- > > drivers/vfio/pci/vfio_pci_priv.h | 13 ++ > > include/linux/dma-buf.h | 25 +++ > > include/linux/mlx5/driver.h | 15 ++ > > include/linux/pci-tph.h | 8 + > > include/uapi/linux/vfio.h | 43 +++++ > > 14 files changed, 517 insertions(+), 19 deletions(-) > > > > -- > > 2.53.0-Meta > >
