Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-21 Thread Christian König
Am 22.06.21 um 01:29 schrieb Jason Gunthorpe: On Mon, Jun 21, 2021 at 10:24:16PM +0300, Oded Gabbay wrote: Another thing I want to emphasize is that we are doing p2p only through the export/import of the FD. We do *not* allow the user to mmap the dma-buf as we do not support direct IO. So there

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-22 Thread Oded Gabbay
On Tue, Jun 22, 2021 at 9:37 AM Christian König wrote: > > Am 22.06.21 um 01:29 schrieb Jason Gunthorpe: > > On Mon, Jun 21, 2021 at 10:24:16PM +0300, Oded Gabbay wrote: > > > >> Another thing I want to emphasize is that we are doing p2p only > >> through the export/import of the FD. We do *not* a

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-22 Thread Jason Gunthorpe
On Tue, Jun 22, 2021 at 11:42:27AM +0300, Oded Gabbay wrote: > On Tue, Jun 22, 2021 at 9:37 AM Christian König > wrote: > > > > Am 22.06.21 um 01:29 schrieb Jason Gunthorpe: > > > On Mon, Jun 21, 2021 at 10:24:16PM +0300, Oded Gabbay wrote: > > > > > >> Another thing I want to emphasize is that we

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-22 Thread Oded Gabbay
On Tue, Jun 22, 2021 at 3:01 PM Jason Gunthorpe wrote: > > On Tue, Jun 22, 2021 at 11:42:27AM +0300, Oded Gabbay wrote: > > On Tue, Jun 22, 2021 at 9:37 AM Christian König > > wrote: > > > > > > Am 22.06.21 um 01:29 schrieb Jason Gunthorpe: > > > > On Mon, Jun 21, 2021 at 10:24:16PM +0300, Oded G

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-22 Thread Jason Gunthorpe
On Tue, Jun 22, 2021 at 03:04:30PM +0300, Oded Gabbay wrote: > On Tue, Jun 22, 2021 at 3:01 PM Jason Gunthorpe wrote: > > > > On Tue, Jun 22, 2021 at 11:42:27AM +0300, Oded Gabbay wrote: > > > On Tue, Jun 22, 2021 at 9:37 AM Christian König > > > wrote: > > > > > > > > Am 22.06.21 um 01:29 schrie

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-22 Thread Christian König
Am 22.06.21 um 14:01 schrieb Jason Gunthorpe: On Tue, Jun 22, 2021 at 11:42:27AM +0300, Oded Gabbay wrote: On Tue, Jun 22, 2021 at 9:37 AM Christian König wrote: Am 22.06.21 um 01:29 schrieb Jason Gunthorpe: On Mon, Jun 21, 2021 at 10:24:16PM +0300, Oded Gabbay wrote: Another thing I want t

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-22 Thread Oded Gabbay
On Tue, Jun 22, 2021 at 3:15 PM Jason Gunthorpe wrote: > > On Tue, Jun 22, 2021 at 03:04:30PM +0300, Oded Gabbay wrote: > > On Tue, Jun 22, 2021 at 3:01 PM Jason Gunthorpe wrote: > > > > > > On Tue, Jun 22, 2021 at 11:42:27AM +0300, Oded Gabbay wrote: > > > > On Tue, Jun 22, 2021 at 9:37 AM Chris

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-22 Thread Jason Gunthorpe
On Tue, Jun 22, 2021 at 04:12:26PM +0300, Oded Gabbay wrote: > > 1) Setting sg_page to NULL > > 2) 'mapping' pages for P2P DMA without going through the iommu > > 3) Allowing P2P DMA without using the p2p dma API to validate that it > >can work at all in the first place. > > > > All of these r

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-22 Thread Jason Gunthorpe
On Tue, Jun 22, 2021 at 02:23:03PM +0200, Christian König wrote: > Am 22.06.21 um 14:01 schrieb Jason Gunthorpe: > > On Tue, Jun 22, 2021 at 11:42:27AM +0300, Oded Gabbay wrote: > > > On Tue, Jun 22, 2021 at 9:37 AM Christian König > > > wrote: > > > > Am 22.06.21 um 01:29 schrieb Jason Gunthorpe:

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-22 Thread Christian König
Am 22.06.21 um 17:11 schrieb Jason Gunthorpe: On Tue, Jun 22, 2021 at 04:12:26PM +0300, Oded Gabbay wrote: 1) Setting sg_page to NULL 2) 'mapping' pages for P2P DMA without going through the iommu 3) Allowing P2P DMA without using the p2p dma API to validate that it can work at all in th

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-22 Thread Oded Gabbay
On Tue, Jun 22, 2021 at 6:11 PM Jason Gunthorpe wrote: > > On Tue, Jun 22, 2021 at 04:12:26PM +0300, Oded Gabbay wrote: > > > > 1) Setting sg_page to NULL > > > 2) 'mapping' pages for P2P DMA without going through the iommu > > > 3) Allowing P2P DMA without using the p2p dma API to validate that i

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-22 Thread Jason Gunthorpe
On Tue, Jun 22, 2021 at 05:24:08PM +0200, Christian König wrote: > > > I will take two GAUDI devices and use one as an exporter and one as an > > > importer. I want to see that the solution works end-to-end, with real > > > device DMA from importer to exporter. > > I can tell you it doesn't. Stuff

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-22 Thread Christian König
Am 22.06.21 um 17:23 schrieb Jason Gunthorpe: On Tue, Jun 22, 2021 at 02:23:03PM +0200, Christian König wrote: Am 22.06.21 um 14:01 schrieb Jason Gunthorpe: On Tue, Jun 22, 2021 at 11:42:27AM +0300, Oded Gabbay wrote: On Tue, Jun 22, 2021 at 9:37 AM Christian König wrote: Am 22.06.21 um 01:2

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-22 Thread Christian König
Am 22.06.21 um 17:28 schrieb Jason Gunthorpe: On Tue, Jun 22, 2021 at 05:24:08PM +0200, Christian König wrote: I will take two GAUDI devices and use one as an exporter and one as an importer. I want to see that the solution works end-to-end, with real device DMA from importer to exporter. I

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-22 Thread Oded Gabbay
On Tue, Jun 22, 2021 at 6:28 PM Jason Gunthorpe wrote: > > On Tue, Jun 22, 2021 at 05:24:08PM +0200, Christian König wrote: > > > > > I will take two GAUDI devices and use one as an exporter and one as an > > > > importer. I want to see that the solution works end-to-end, with real > > > > device

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-22 Thread Jason Gunthorpe
On Tue, Jun 22, 2021 at 06:24:28PM +0300, Oded Gabbay wrote: > On Tue, Jun 22, 2021 at 6:11 PM Jason Gunthorpe wrote: > > > > On Tue, Jun 22, 2021 at 04:12:26PM +0300, Oded Gabbay wrote: > > > > > > 1) Setting sg_page to NULL > > > > 2) 'mapping' pages for P2P DMA without going through the iommu >

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-22 Thread Jason Gunthorpe
On Tue, Jun 22, 2021 at 05:29:01PM +0200, Christian König wrote: > Am 22.06.21 um 17:23 schrieb Jason Gunthorpe: > > On Tue, Jun 22, 2021 at 02:23:03PM +0200, Christian König wrote: > > > Am 22.06.21 um 14:01 schrieb Jason Gunthorpe: > > > > On Tue, Jun 22, 2021 at 11:42:27AM +0300, Oded Gabbay wro

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-22 Thread Oded Gabbay
On Tue, Jun 22, 2021 at 6:31 PM Christian König wrote: > > > > Am 22.06.21 um 17:28 schrieb Jason Gunthorpe: > > On Tue, Jun 22, 2021 at 05:24:08PM +0200, Christian König wrote: > > > I will take two GAUDI devices and use one as an exporter and one as an > importer. I want to see that th

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-22 Thread Christian König
Am 22.06.21 um 17:40 schrieb Jason Gunthorpe: On Tue, Jun 22, 2021 at 05:29:01PM +0200, Christian König wrote: [SNIP] No absolutely not. NVidia GPUs work exactly the same way. And you have tons of similar cases in embedded and SoC systems where intermediate memory between devices isn't directly

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-22 Thread Christian König
Am 22.06.21 um 17:40 schrieb Oded Gabbay: On Tue, Jun 22, 2021 at 6:31 PM Christian König wrote: Am 22.06.21 um 17:28 schrieb Jason Gunthorpe: On Tue, Jun 22, 2021 at 05:24:08PM +0200, Christian König wrote: I will take two GAUDI devices and use one as an exporter and one as an importer. I

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-22 Thread Jason Gunthorpe
On Tue, Jun 22, 2021 at 05:48:10PM +0200, Christian König wrote: > Am 22.06.21 um 17:40 schrieb Jason Gunthorpe: > > On Tue, Jun 22, 2021 at 05:29:01PM +0200, Christian König wrote: > > > [SNIP] > > > No absolutely not. NVidia GPUs work exactly the same way. > > > > > > And you have tons of simila

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-22 Thread Felix Kuehling
Am 2021-06-22 um 11:29 a.m. schrieb Christian König: > Am 22.06.21 um 17:23 schrieb Jason Gunthorpe: >> On Tue, Jun 22, 2021 at 02:23:03PM +0200, Christian König wrote: >>> Am 22.06.21 um 14:01 schrieb Jason Gunthorpe: On Tue, Jun 22, 2021 at 11:42:27AM +0300, Oded Gabbay wrote: > On Tue,

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-23 Thread Christian König
Am 22.06.21 um 18:05 schrieb Jason Gunthorpe: On Tue, Jun 22, 2021 at 05:48:10PM +0200, Christian König wrote: Am 22.06.21 um 17:40 schrieb Jason Gunthorpe: On Tue, Jun 22, 2021 at 05:29:01PM +0200, Christian König wrote: [SNIP] No absolutely not. NVidia GPUs work exactly the same way. And yo

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-23 Thread Oded Gabbay
On Wed, Jun 23, 2021 at 11:57 AM Christian König wrote: > > Am 22.06.21 um 18:05 schrieb Jason Gunthorpe: > > On Tue, Jun 22, 2021 at 05:48:10PM +0200, Christian König wrote: > >> Am 22.06.21 um 17:40 schrieb Jason Gunthorpe: > >>> On Tue, Jun 22, 2021 at 05:29:01PM +0200, Christian König wrote: >

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-23 Thread Jason Gunthorpe
On Wed, Jun 23, 2021 at 10:57:35AM +0200, Christian König wrote: > > > No it isn't. It makes devices depend on allocating struct pages for their > > > BARs which is not necessary nor desired. > > Which dramatically reduces the cost of establishing DMA mappings, a > > loop of dma_map_resource() is

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-23 Thread Oded Gabbay
On Wed, Jun 23, 2021 at 9:24 PM Jason Gunthorpe wrote: > > On Wed, Jun 23, 2021 at 10:57:35AM +0200, Christian König wrote: > > > > > No it isn't. It makes devices depend on allocating struct pages for > > > > their > > > > BARs which is not necessary nor desired. > > > Which dramatically reduces

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-23 Thread Jason Gunthorpe
On Wed, Jun 23, 2021 at 09:43:04PM +0300, Oded Gabbay wrote: > Can you please explain why it is so important to (allow) access them > through the CPU ? It is not so much important, as it reflects significant design choices that are already tightly baked into alot of our stacks. A SGL is CPU acc

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-23 Thread Oded Gabbay
On Wed, Jun 23, 2021 at 9:50 PM Jason Gunthorpe wrote: > > On Wed, Jun 23, 2021 at 09:43:04PM +0300, Oded Gabbay wrote: > > > Can you please explain why it is so important to (allow) access them > > through the CPU ? > > It is not so much important, as it reflects significant design choices > that

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-23 Thread Jason Gunthorpe
On Wed, Jun 23, 2021 at 10:00:29PM +0300, Oded Gabbay wrote: > On Wed, Jun 23, 2021 at 9:50 PM Jason Gunthorpe wrote: > > > > On Wed, Jun 23, 2021 at 09:43:04PM +0300, Oded Gabbay wrote: > > > > > Can you please explain why it is so important to (allow) access them > > > through the CPU ? > > > >

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-23 Thread Oded Gabbay
On Wed, Jun 23, 2021 at 10:34 PM Jason Gunthorpe wrote: > > On Wed, Jun 23, 2021 at 10:00:29PM +0300, Oded Gabbay wrote: > > On Wed, Jun 23, 2021 at 9:50 PM Jason Gunthorpe wrote: > > > > > > On Wed, Jun 23, 2021 at 09:43:04PM +0300, Oded Gabbay wrote: > > > > > > > Can you please explain why it

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-23 Thread Jason Gunthorpe
On Wed, Jun 23, 2021 at 10:39:48PM +0300, Oded Gabbay wrote: > On Wed, Jun 23, 2021 at 10:34 PM Jason Gunthorpe wrote: > > > > On Wed, Jun 23, 2021 at 10:00:29PM +0300, Oded Gabbay wrote: > > > On Wed, Jun 23, 2021 at 9:50 PM Jason Gunthorpe wrote: > > > > > > > > On Wed, Jun 23, 2021 at 09:43:04

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-24 Thread Christian König
Am 24.06.21 um 07:34 schrieb Christoph Hellwig: On Wed, Jun 23, 2021 at 10:00:29PM +0300, Oded Gabbay wrote: I understand the argument and I agree that for the generic case, the top of the stack can't assume anything. Having said that, in this case the SGL is encapsulated inside a dma-buf object

Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF

2021-06-24 Thread Christian König
Am 24.06.21 um 10:12 schrieb Christoph Hellwig: On Thu, Jun 24, 2021 at 10:07:14AM +0200, Christian König wrote: The key point is that accessing the underlying pages even when DMA-bufs are backed by system memory is illegal. Daniel even created a patch which mangles the page pointers in sg_table