> -----Original Message----- > From: Richard Cheng <[email protected]> > Sent: Thursday, September 17, 2026 1:25 PM > To: Manish Honap <[email protected]> > Cc: [email protected]; [email protected]; Ankit Agrawal <[email protected]>; > [email protected]; [email protected]; [email protected]; > Srirangan Madhavan <[email protected]>; [email protected]; > [email protected]; [email protected]; [email protected]; > [email protected]; [email protected]; [email protected]; Yishai > Hadas <[email protected]>; Shameer Kolothum Thodi > <[email protected]>; [email protected]; [email protected]; > [email protected]; [email protected]; [email protected]; Neo Jia > <[email protected]>; Krishnakant Jaju <[email protected]>; Vikram Sethi > <[email protected]>; Zhi Wang <[email protected]>; linux- > [email protected]; [email protected]; [email protected]; > [email protected]; [email protected]; linux- > [email protected]; [email protected] > Subject: Re: [PATCH v5 24/27] vfio/cxl: Export the HDM memory region as a > dma-buf > > On Thu, Sep 17, 2026 at 12:05:37AM +0800, [email protected] wrote: > > From: Manish Honap <[email protected]> > > > > A Type-2 accelerator issues ATS-translated DMA to addresses inside its > > own HDM window, so that coherent host range must be present in the > > guest's IOAS (the iommufd IOAS backing the nested SMMU stage-2). > > iommufd maps a struct-page-less range only by fd, via > > IOMMU_IOAS_MAP_FILE over a dma-buf; a userspace-VA > IOMMU_IOAS_MAP of > > the HDM mmap is rejected because the VMA is VM_IO | VM_PFNMAP. > Without > > a dma-buf the range could only be mapped through an out-of-tree PFNMAP > work-around. > > > > vfio-pci already exports BAR memory as a P2P dma-buf, but the exporter > > is BAR-only: vfio_pci_core_feature_dma_buf() rejects any region index > > at or above the ROM index, and vfio_pci_core_get_dmabuf_phys() > > resolves the physical range from a PCI BAR. The HDM memory region is a > > dynamic device-specific region, not a BAR. > > > > Let a device-specific region reach the device's get_dmabuf_phys(): a > > region index at or above VFIO_PCI_NUM_REGIONS skips the BAR-resource > > check and is validated by the driver instead, bounded to the regions > > that exist. Install a CXL-aware get_dmabuf_phys() in the vfio-cxl > > provider that returns cxl->hpa_range for the HDM memory region and > > delegates real BARs to the core, keeping the BAR path unchanged and > > the core free of CXL knowledge. > > > > The HDM window is coherent host memory with no p2pdma provider of its > > own, so borrow BAR 0's, matching nvgrace-gpu's handling of its non-BAR > > device memory. The iommufd importer does not consume the provider; the > > scatterlist map path (real peer DMA) is left to a follow-up once > > upstream grows a negotiated interconnect for coherent CXL memory. > > > > Assisted-by: LLM > > Signed-off-by: Manish Honap <[email protected]> > > --- > > drivers/vfio/pci/cxl/vfio_cxl_core.c | 60 > ++++++++++++++++++++++++++++ > > drivers/vfio/pci/vfio_pci_dmabuf.c | 27 +++++++++++-- > > 2 files changed, 83 insertions(+), 4 deletions(-) > > > > diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c > > b/drivers/vfio/pci/cxl/vfio_cxl_core.c > > index 5fe8e35c63c4..55fa1f86850d 100644 > > --- a/drivers/vfio/pci/cxl/vfio_cxl_core.c > > +++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c > > @@ -11,6 +11,7 @@ > > #include <linux/mm.h> > > #include <linux/module.h> > > #include <linux/pci.h> > > +#include <linux/pci-p2pdma.h> > > #include <linux/range.h> > > #include <linux/slab.h> > > #include <linux/uaccess.h> > > @@ -27,6 +28,7 @@ > > * @hdm_regs: mapped HDM decoder registers, read live by the decoder > region > > * @hdm_len: length of the HDM decoder register block > > * @hdm_valid: true when host CPU access to the HDM range is safe; > > under memory_lock > > + * @mem_region_index: vfio region index of the mmap-able HDM memory > > + region > > */ > > struct vfio_cxl_state { > > struct cxl_dev_state cxlds; > > @@ -36,6 +38,7 @@ struct vfio_cxl_state { > > void __iomem *hdm_regs; > > u32 hdm_len; > > bool hdm_valid; > > + unsigned int mem_region_index; > > }; > > > > static unsigned long vfio_cxl_mem_pgoff(struct vm_area_struct *vma, > > @@ -289,6 +292,50 @@ static void vfio_cxl_release_hpa(void *data) > > release_mem_region(cxl->hpa_range.start, > > range_len(&cxl->hpa_range)); } > > > > +/* > > + * Resolve the physical range that backs a dma-buf export. The core > > +exporter > > + * only knows BARs; teach it the HDM memory region so a guest IOAS > > +can map the > > + * coherent window by fd (IOMMU_IOAS_MAP_FILE) instead of the > removed > > +PFNMAP > > + * work-around. Real BARs stay on the byte-identical core path. > > + */ > > +static int vfio_cxl_get_dmabuf_phys(struct vfio_pci_core_device *vdev, > > + struct p2pdma_provider **provider, > > + unsigned int region_index, > > + struct phys_vec *phys_vec, > > + struct vfio_region_dma_range > *dma_ranges, > > + size_t nr_ranges) > > +{ > > + struct vfio_cxl_state *cxl = vdev->cxl; > > + > > + /* Real BARs go through the core P2P exporter unchanged. */ > > + if (region_index < VFIO_PCI_NUM_REGIONS) > > + return vfio_pci_core_get_dmabuf_phys(vdev, provider, > > + region_index, phys_vec, > > + dma_ranges, nr_ranges); > > + > > + /* Of the device regions, only the HDM memory window is exportable. > */ > > + if (region_index != cxl->mem_region_index) > > + return -EINVAL; > > + > > + /* > > + * The HDM window is coherent host memory, not BAR MMIO, so it > has no > > + * p2pdma provider of its own. Borrow BAR 0's: the P2P properties > match > > + * and the iommufd importer does not consume the provider. The sgt > map > > + * path (real peer DMA) is not supported for the HDM window. > > + */ > > + *provider = pcim_p2pdma_provider(vdev->pdev, 0); > > I think of a weird scenario which might make peer DMA work, not sure if that's > possible. > > userspace can give this dma-buf fd to an NIC driver or so to do RDMA ? use > HDM memory as network buffer ? > > If that's the case and direct P2P addressing is selected, it applies BAR 0's > bus > offset to HDM physical address. > > Maybe explicitly reject peer-DMA mapping would be safer ? Yes, I think this will require handling. For v6, I will tag the HDM export with no_peer2peer and vfio_pci_dma_buf_map() return -EOPNOTSUPP for it, so a peer-DMA mapping is refused cleanly instead of being handed a wrong address. Thanks, Manish > > Best regards, > Richard Cheng. > > > > + if (!*provider) > > + return -EINVAL; > > + > > + return vfio_pci_core_fill_phys_vec(phys_vec, dma_ranges, nr_ranges, > > + cxl->hpa_range.start, > > + range_len(&cxl->hpa_range)); > > +} > > + > > +static const struct vfio_pci_device_ops vfio_cxl_pci_dev_ops = { > > + .get_dmabuf_phys = vfio_cxl_get_dmabuf_phys, }; > > + > > static int vfio_cxl_init_device(struct vfio_pci_core_device *vdev) { > > struct pci_dev *pdev = vdev->pdev; > > @@ -483,6 +530,19 @@ static int vfio_cxl_open_device(struct > vfio_pci_core_device *vdev) > > if (ret) > > return ret; > > > > + /* Record where the HDM memory region landed for the dma-buf > export. */ > > + cxl->mem_region_index = VFIO_PCI_NUM_REGIONS + vdev- > >num_regions - > > +1; > > + > > + /* > > + * Override the device ops so a dma-buf export of the HDM memory > region > > + * resolves to the coherent host range. This is done at open, not init: > > + * vfio_pci_probe() resets pci_ops after vfio_alloc_device() returns, so > > + * an override installed during init would be clobbered. Only a CXL > device > > + * reaches this hook (cxl_ops is set on init success), so a fallback to > > + * plain vfio-pci keeps the core ops. > > + */ > > + vdev->pci_ops = &vfio_cxl_pci_dev_ops; > > + > > ret = vfio_cxl_add_region(vdev, > VFIO_REGION_SUBTYPE_CXL_COMP_REGS, > > &vfio_cxl_comp_regops, cxl->hdm_len, > > VFIO_REGION_INFO_FLAG_READ | > > diff --git a/drivers/vfio/pci/vfio_pci_dmabuf.c > > b/drivers/vfio/pci/vfio_pci_dmabuf.c > > index c16f460c01d6..436c616d5b66 100644 > > --- a/drivers/vfio/pci/vfio_pci_dmabuf.c > > +++ b/drivers/vfio/pci/vfio_pci_dmabuf.c > > @@ -178,6 +178,15 @@ int vfio_pci_core_get_dmabuf_phys(struct > > vfio_pci_core_device *vdev, { > > struct pci_dev *pdev = vdev->pdev; > > > > + /* > > + * This resolver only handles PCI BARs. A device-specific region index > > + * (>= PCI_STD_NUM_BARS) would index pdev->resource[] out of > bounds via > > + * pcim_p2pdma_provider(), so reject it; a driver that exports such a > > + * region installs its own get_dmabuf_phys. > > + */ > > + if (region_index >= PCI_STD_NUM_BARS) > > + return -EINVAL; > > + > > *provider = pcim_p2pdma_provider(pdev, region_index); > > if (!*provider) > > return -EINVAL; > > @@ -227,6 +236,7 @@ int vfio_pci_core_feature_dma_buf(struct > vfio_pci_core_device *vdev, u32 flags, > > DEFINE_DMA_BUF_EXPORT_INFO(exp_info); > > struct vfio_pci_dma_buf *priv; > > size_t length; > > + u32 index; > > int ret; > > > > if (!vdev->pci_ops || !vdev->pci_ops->get_dmabuf_phys) @@ - > 243,13 > > +253,22 @@ int vfio_pci_core_feature_dma_buf(struct > vfio_pci_core_device *vdev, u32 flags, > > if (!get_dma_buf.nr_ranges || get_dma_buf.flags) > > return -EINVAL; > > > > + index = get_dma_buf.region_index; > > + > > /* > > - * For PCI the region_index is the BAR number like everything > > - * else. Check that PCI resources have been claimed for it. > > + * A fixed region index is the BAR number; only a BAR can be exported > > + * and its PCI resource must be claimed. A device-specific region (index > > + * >= VFIO_PCI_NUM_REGIONS) has no BAR resource and is validated > by the > > + * device's get_dmabuf_phys instead, but the index must name a > region > > + * that exists. > > */ > > - if (get_dma_buf.region_index >= VFIO_PCI_ROM_REGION_INDEX || > > - IS_ERR(vfio_pci_core_get_iomap(vdev, > get_dma_buf.region_index))) > > + if (index < VFIO_PCI_NUM_REGIONS) { > > + if (index >= VFIO_PCI_ROM_REGION_INDEX || > > + IS_ERR(vfio_pci_core_get_iomap(vdev, index))) > > + return -ENODEV; > > + } else if (index - VFIO_PCI_NUM_REGIONS >= vdev->num_regions) { > > return -ENODEV; > > + } > > > > dma_ranges = memdup_array_user(&arg->dma_ranges, > get_dma_buf.nr_ranges, > > sizeof(*dma_ranges)); > > -- > > 2.25.1 > > > >
