Re: [PATCH v2 4/4] vfio/pci: Allow MMIO regions to be exported through dma-buf

2022-09-09 Thread Jason Gunthorpe
On Fri, Sep 09, 2022 at 06:24:35AM -0700, Christoph Hellwig wrote: > On Wed, Sep 07, 2022 at 01:12:52PM -0300, Jason Gunthorpe wrote: > > The PCI offset is some embedded thing - I've never seen it in a server > > platform. > > That's not actually true, e.g. some power system definitively had it,

Re: [PATCH v2 4/4] vfio/pci: Allow MMIO regions to be exported through dma-buf

2022-09-07 Thread Dan Williams
Christoph Hellwig wrote: > On Wed, Sep 07, 2022 at 09:33:11AM -0300, Jason Gunthorpe wrote: > > Yes, you said that, and I said that when the AMD driver first merged > > it - but it went in anyhow and now people are using it in a bunch of > > places. > > drm folks made up their own weird rules, if

Re: [PATCH v2 4/4] vfio/pci: Allow MMIO regions to be exported through dma-buf

2022-09-07 Thread Jason Gunthorpe
On Wed, Sep 07, 2022 at 05:31:14PM +0100, Robin Murphy wrote: > The only trouble is that it's not geared for *PCI* P2P when that may or may > not happen entirely upstream of IOMMU translation. This is why PCI users have to call the pci_distance stuff before using dma_map_resource(), it ensures

Re: [PATCH v2 4/4] vfio/pci: Allow MMIO regions to be exported through dma-buf

2022-09-07 Thread Robin Murphy
On 2022-09-07 16:23, Jason Gunthorpe wrote: On Wed, Sep 07, 2022 at 07:29:58AM -0700, Christoph Hellwig wrote: On Wed, Sep 07, 2022 at 09:33:11AM -0300, Jason Gunthorpe wrote: Yes, you said that, and I said that when the AMD driver first merged it - but it went in anyhow and now people are

Re: [PATCH v2 4/4] vfio/pci: Allow MMIO regions to be exported through dma-buf

2022-09-07 Thread Jason Gunthorpe
On Wed, Sep 07, 2022 at 08:32:23AM -0700, Christoph Hellwig wrote: > On Wed, Sep 07, 2022 at 12:23:28PM -0300, Jason Gunthorpe wrote: > > 2) DMABUF abuses dma_map_resource() for P2P and thus doesn't work in > > certain special cases. > > Not just certain special cases, but one of the main

Re: [PATCH v2 4/4] vfio/pci: Allow MMIO regions to be exported through dma-buf

2022-09-07 Thread Jason Gunthorpe
On Wed, Sep 07, 2022 at 07:29:58AM -0700, Christoph Hellwig wrote: > On Wed, Sep 07, 2022 at 09:33:11AM -0300, Jason Gunthorpe wrote: > > Yes, you said that, and I said that when the AMD driver first merged > > it - but it went in anyhow and now people are using it in a bunch of > > places. > >

Re: [PATCH v2 4/4] vfio/pci: Allow MMIO regions to be exported through dma-buf

2022-09-07 Thread Christian König
Am 07.09.22 um 14:03 schrieb Christoph Hellwig: On Tue, Sep 06, 2022 at 12:38:44PM +0200, Christian König wrote: The problem is once more that this is MMIO space, in other words register BARs which needs to be exported/imported. Everything used for P2P is bar space. Adding struct pages for

Re: [PATCH v2 4/4] vfio/pci: Allow MMIO regions to be exported through dma-buf

2022-09-07 Thread Robin Murphy
On 2022-09-07 13:33, Jason Gunthorpe wrote: On Wed, Sep 07, 2022 at 05:05:57AM -0700, Christoph Hellwig wrote: On Tue, Sep 06, 2022 at 08:48:28AM -0300, Jason Gunthorpe wrote: Right, this whole thing is the "standard" that dmabuf has adopted instead of the struct pages. Once the AMD GPU driver

Re: [PATCH v2 4/4] vfio/pci: Allow MMIO regions to be exported through dma-buf

2022-09-07 Thread Oded Gabbay
On Wed, Sep 7, 2022 at 5:30 PM Christoph Hellwig wrote: > > On Wed, Sep 07, 2022 at 09:33:11AM -0300, Jason Gunthorpe wrote: > > Yes, you said that, and I said that when the AMD driver first merged > > it - but it went in anyhow and now people are using it in a bunch of > > places. > > drm folks

Re: [PATCH v2 4/4] vfio/pci: Allow MMIO regions to be exported through dma-buf

2022-09-07 Thread Jason Gunthorpe
On Wed, Sep 07, 2022 at 05:05:57AM -0700, Christoph Hellwig wrote: > On Tue, Sep 06, 2022 at 08:48:28AM -0300, Jason Gunthorpe wrote: > > Right, this whole thing is the "standard" that dmabuf has adopted > > instead of the struct pages. Once the AMD GPU driver started doing > > this some time ago

Re: [PATCH v2 4/4] vfio/pci: Allow MMIO regions to be exported through dma-buf

2022-09-06 Thread Oded Gabbay
On Tue, Sep 6, 2022 at 8:59 PM Jason Gunthorpe wrote: > > On Tue, Sep 06, 2022 at 03:34:02PM +0300, Oded Gabbay wrote: > > > > > > > + /* > > > > > > + * Since the memory being mapped is a device memory it could > > > > > > never be in > > > > > > + * CPU caches. > > > > > > + */ > > > > >

Re: [PATCH v2 4/4] vfio/pci: Allow MMIO regions to be exported through dma-buf

2022-09-06 Thread Jason Gunthorpe
On Tue, Sep 06, 2022 at 03:34:02PM +0300, Oded Gabbay wrote: > > > > > + /* > > > > > + * Since the memory being mapped is a device memory it could never > > > > > be in > > > > > + * CPU caches. > > > > > + */ > > > > DMA_ATTR_SKIP_CPU_SYNC doesn't even apply to dma_map_resource, not sure >

Re: [PATCH v2 4/4] vfio/pci: Allow MMIO regions to be exported through dma-buf

2022-09-06 Thread Oded Gabbay
On Tue, Sep 6, 2022 at 2:48 PM Jason Gunthorpe wrote: > > On Tue, Sep 06, 2022 at 12:38:44PM +0200, Christian König wrote: > > Am 06.09.22 um 11:51 schrieb Christoph Hellwig: > > > > +{ > > > > + struct vfio_pci_dma_buf *priv = dmabuf->priv; > > > > + int rc; > > > > + > > > > + rc =

Re: [PATCH v2 4/4] vfio/pci: Allow MMIO regions to be exported through dma-buf

2022-09-06 Thread Jason Gunthorpe
On Tue, Sep 06, 2022 at 12:38:44PM +0200, Christian König wrote: > Am 06.09.22 um 11:51 schrieb Christoph Hellwig: > > > +{ > > > + struct vfio_pci_dma_buf *priv = dmabuf->priv; > > > + int rc; > > > + > > > + rc = pci_p2pdma_distance_many(priv->vdev->pdev, >dev, 1, > > > +

Re: [PATCH v2 4/4] vfio/pci: Allow MMIO regions to be exported through dma-buf

2022-09-06 Thread Christian König
Am 06.09.22 um 11:51 schrieb Christoph Hellwig: +{ + struct vfio_pci_dma_buf *priv = dmabuf->priv; + int rc; + + rc = pci_p2pdma_distance_many(priv->vdev->pdev, >dev, 1, + true); This should just use pci_p2pdma_distance. + /* +

[PATCH v2 4/4] vfio/pci: Allow MMIO regions to be exported through dma-buf

2022-08-31 Thread Jason Gunthorpe
dma-buf has become a way to safely acquire a handle to non-struct page memory that can still have lifetime controlled by the exporter. Notably RDMA can now import dma-buf FDs and build them into MRs which allows for PCI P2P operations. Extend this to allow vfio-pci to export MMIO memory from PCI