Re: PCIe P2P access to GPU memory
Am 03.10.2018 um 11:06 schrieb Daniel Vetter: On Wed, Oct 03, 2018 at 08:54:44AM +, Koenig, Christian wrote: Am 03.10.2018 um 09:14 schrieb Daniel Vetter: On Sat, Sep 29, 2018 at 9:27 AM Koenig, Christian wrote: This is work in progress. I published patches to enable DMA_buf P2P a few months ago, but now I'm waiting for the PCI subsystem to pick up core support for this. I can prepare you a branch based on current upstream kernel next week if you want to test this. Thread hijack, since I just read the lwn article on p2p for storage folks: https://lwn.net/Articles/767281/ Totally doesn't match what I remember of your dma-buf p2p work. Do we need to rework, or is this some kind of higher-level interface on top of the raw p2p pci stuff? Or do I misremember. The P2P from the storage folks are some mix of raw p2p capability detection for PCI and higher level interface. I'm actually waiting for that stuff to land to extract the p2p capability detection and use it for this DMA-buf stuff as well. Alternative would be to convince Logan to extract the capability detection in his patch set as well, but so far he wasn't very keen about that. So no plan to make dma-buf some kind of orchestrator and use the p2p map_sg stuff? Not sure about that yet. Didn't had time to take a closer look. Christian. That seems to be the new/additional bits I haven't seen yet, and I'm wondered how we should fit dma-buf into that picture. -Daniel Christian. Thanks, Daniel Regards, Christian. Am 29.09.2018 09:01 schrieb Dirk Eibach : I want to access GPU VRAM via PCIe P2P access, like DirectGMA but with mainline drivers. I can acquire a dmabuf from the GPU and pass it to my PCIe framegrabber. But I don't see a way to get the PCIe bus address of the video memory which I need for the P2P transfer. Is there already any infrastructure in place for doing this? If not, what would be the right way to implement this? Add another callback to the dmabuf struct? Cheers Dirk ___ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel ___ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: PCIe P2P access to GPU memory
On Wed, Oct 03, 2018 at 08:54:44AM +, Koenig, Christian wrote: > Am 03.10.2018 um 09:14 schrieb Daniel Vetter: > > On Sat, Sep 29, 2018 at 9:27 AM Koenig, Christian > > wrote: > >> This is work in progress. > >> > >> I published patches to enable DMA_buf P2P a few months ago, but now I'm > >> waiting for the PCI subsystem to pick up core support for this. > >> > >> I can prepare you a branch based on current upstream kernel next week if > >> you want to test this. > > Thread hijack, since I just read the lwn article on p2p for storage folks: > > > > https://lwn.net/Articles/767281/ > > > > Totally doesn't match what I remember of your dma-buf p2p work. Do we > > need to rework, or is this some kind of higher-level interface on top > > of the raw p2p pci stuff? Or do I misremember. > > The P2P from the storage folks are some mix of raw p2p capability > detection for PCI and higher level interface. > > I'm actually waiting for that stuff to land to extract the p2p > capability detection and use it for this DMA-buf stuff as well. > > Alternative would be to convince Logan to extract the capability > detection in his patch set as well, but so far he wasn't very keen about > that. So no plan to make dma-buf some kind of orchestrator and use the p2p map_sg stuff? That seems to be the new/additional bits I haven't seen yet, and I'm wondered how we should fit dma-buf into that picture. -Daniel > > Christian. > > > > > Thanks, Daniel > > > >> Regards, > >> Christian. > >> > >> Am 29.09.2018 09:01 schrieb Dirk Eibach : > >> I want to access GPU VRAM via PCIe P2P access, like DirectGMA but with > >> mainline drivers. > >> I can acquire a dmabuf from the GPU and pass it to my PCIe > >> framegrabber. But I don't see a way to get the PCIe bus address of the > >> video memory which I need for the P2P transfer. Is there already any > >> infrastructure in place for doing this? If not, what would be the > >> right way to implement this? Add another callback to the dmabuf > >> struct? > >> > >> Cheers > >> Dirk > >> ___ > >> dri-devel mailing list > >> dri-devel@lists.freedesktop.org > >> https://lists.freedesktop.org/mailman/listinfo/dri-devel > > > > > -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch ___ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: PCIe P2P access to GPU memory
Am 03.10.2018 um 09:14 schrieb Daniel Vetter: > On Sat, Sep 29, 2018 at 9:27 AM Koenig, Christian > wrote: >> This is work in progress. >> >> I published patches to enable DMA_buf P2P a few months ago, but now I'm >> waiting for the PCI subsystem to pick up core support for this. >> >> I can prepare you a branch based on current upstream kernel next week if you >> want to test this. > Thread hijack, since I just read the lwn article on p2p for storage folks: > > https://lwn.net/Articles/767281/ > > Totally doesn't match what I remember of your dma-buf p2p work. Do we > need to rework, or is this some kind of higher-level interface on top > of the raw p2p pci stuff? Or do I misremember. The P2P from the storage folks are some mix of raw p2p capability detection for PCI and higher level interface. I'm actually waiting for that stuff to land to extract the p2p capability detection and use it for this DMA-buf stuff as well. Alternative would be to convince Logan to extract the capability detection in his patch set as well, but so far he wasn't very keen about that. Christian. > > Thanks, Daniel > >> Regards, >> Christian. >> >> Am 29.09.2018 09:01 schrieb Dirk Eibach : >> I want to access GPU VRAM via PCIe P2P access, like DirectGMA but with >> mainline drivers. >> I can acquire a dmabuf from the GPU and pass it to my PCIe >> framegrabber. But I don't see a way to get the PCIe bus address of the >> video memory which I need for the P2P transfer. Is there already any >> infrastructure in place for doing this? If not, what would be the >> right way to implement this? Add another callback to the dmabuf >> struct? >> >> Cheers >> Dirk >> ___ >> dri-devel mailing list >> dri-devel@lists.freedesktop.org >> https://lists.freedesktop.org/mailman/listinfo/dri-devel > > ___ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: PCIe P2P access to GPU memory
On Sat, Sep 29, 2018 at 9:27 AM Koenig, Christian wrote: > > This is work in progress. > > I published patches to enable DMA_buf P2P a few months ago, but now I'm > waiting for the PCI subsystem to pick up core support for this. > > I can prepare you a branch based on current upstream kernel next week if you > want to test this. Thread hijack, since I just read the lwn article on p2p for storage folks: https://lwn.net/Articles/767281/ Totally doesn't match what I remember of your dma-buf p2p work. Do we need to rework, or is this some kind of higher-level interface on top of the raw p2p pci stuff? Or do I misremember. Thanks, Daniel > > Regards, > Christian. > > Am 29.09.2018 09:01 schrieb Dirk Eibach : > I want to access GPU VRAM via PCIe P2P access, like DirectGMA but with > mainline drivers. > I can acquire a dmabuf from the GPU and pass it to my PCIe > framegrabber. But I don't see a way to get the PCIe bus address of the > video memory which I need for the P2P transfer. Is there already any > infrastructure in place for doing this? If not, what would be the > right way to implement this? Add another callback to the dmabuf > struct? > > Cheers > Dirk > ___ > dri-devel mailing list > dri-devel@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/dri-devel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch ___ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: PCIe P2P access to GPU memory
Hi Christian,
Could I simply force it by overwriting peer2peer after the attach call?
Cheers
Dirk
> Hi Dirk,
>
> > Any ideas?
> Could be anything. Most likely cause is that some function things the
> two device can't communicate directly with each other.
>
> Please provide a full dmesg and "lspci -t -nn -v" as attachment and note
> which devices should communicate with which.
>
> I'm currently on vacation so it could take a while till I have time to
> look deeper into this.
>
> Christian.
>
> Am 01.10.2018 um 15:07 schrieb Dirk Eibach:
> > Hi Christian,
> >
> > I took the repository Dieter suggested and gave it a spin. I am not
> > sure, if I got the API right.
> >
> > In userspace I create a bo and get a fd.
> > bo = gbm_bo_create(gbm, w, h, GBM_FORMAT_ARGB, GBM_BO_USE_LINEAR);
> > dmabuf_fd = gbm_bo_get_fd(bo);
> >
> > Then I pass the fd to the grabber driver. There I get a sgt, that I
> > want to pass to the DMA-machine.
> > struct dma_buf_attach_info dma_buf_attach_info = {
> > .dev = &engine->xdev->pdev->dev,
> > .peer2peer = true,
> > };
> > dma_buf = dma_buf_get(xdma_dma_buf_ioctl.dma_buf_fd);
> > dma_buf_attach_info.dmabuf = dma_buf;
> > dma_buf_attachment = dma_buf_attach(&dma_buf_attach_info);
> > sgt = dma_buf_map_attachment(dma_buf_attachment, DMA_TO_DEVICE);
> >
> > But the addresses in the sgt do not look like PCIe addresses at at all.
> >
> > Any ideas?
> >
> > Cheers
> > Dirk
> > Am Sa., 29. Sep. 2018 um 11:53 Uhr schrieb Dieter Nützel <
> die...@nuetzel-hh.de>:
> >> Hello Dirk,
> >>
> >> I think Christian is talking about this branch:
> >>
> >> https://cgit.freedesktop.org/~deathsimple/linux/log/?h=p2p
> >>
> >> His 'home' is, here:
> >> https://cgit.freedesktop.org/~deathsimple/linux/
> >>
> >> Happy hacking! ;-)
> >>
> >> Dieter
> >>
> >> Am 29.09.2018 10:17, schrieb Dirk Eibach:
> This is work in progress.
>
> I published patches to enable DMA_buf P2P a few months ago, but now
> I'm waiting for the PCI subsystem to pick up core support for this.
> >>> Great news! Can you give me a link to this series so I can already have
> >>> a look?
> >>>
> I can prepare you a branch based on current upstream kernel next week
> if you want to test this.
> >>> That would be awesome, thanks.
> >>>
> >>> Cheers
> >>> Dirk
> >>> ___
> >>> dri-devel mailing list
> >>> dri-devel@lists.freedesktop.org
> >>> https://lists.freedesktop.org/mailman/listinfo/dri-devel
>
>
___
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: PCIe P2P access to GPU memory
Hi Dirk,
Any ideas?
Could be anything. Most likely cause is that some function things the
two device can't communicate directly with each other.
Please provide a full dmesg and "lspci -t -nn -v" as attachment and note
which devices should communicate with which.
I'm currently on vacation so it could take a while till I have time to
look deeper into this.
Christian.
Am 01.10.2018 um 15:07 schrieb Dirk Eibach:
Hi Christian,
I took the repository Dieter suggested and gave it a spin. I am not
sure, if I got the API right.
In userspace I create a bo and get a fd.
bo = gbm_bo_create(gbm, w, h, GBM_FORMAT_ARGB, GBM_BO_USE_LINEAR);
dmabuf_fd = gbm_bo_get_fd(bo);
Then I pass the fd to the grabber driver. There I get a sgt, that I
want to pass to the DMA-machine.
struct dma_buf_attach_info dma_buf_attach_info = {
.dev = &engine->xdev->pdev->dev,
.peer2peer = true,
};
dma_buf = dma_buf_get(xdma_dma_buf_ioctl.dma_buf_fd);
dma_buf_attach_info.dmabuf = dma_buf;
dma_buf_attachment = dma_buf_attach(&dma_buf_attach_info);
sgt = dma_buf_map_attachment(dma_buf_attachment, DMA_TO_DEVICE);
But the addresses in the sgt do not look like PCIe addresses at at all.
Any ideas?
Cheers
Dirk
Am Sa., 29. Sep. 2018 um 11:53 Uhr schrieb Dieter Nützel :
Hello Dirk,
I think Christian is talking about this branch:
https://cgit.freedesktop.org/~deathsimple/linux/log/?h=p2p
His 'home' is, here:
https://cgit.freedesktop.org/~deathsimple/linux/
Happy hacking! ;-)
Dieter
Am 29.09.2018 10:17, schrieb Dirk Eibach:
This is work in progress.
I published patches to enable DMA_buf P2P a few months ago, but now
I'm waiting for the PCI subsystem to pick up core support for this.
Great news! Can you give me a link to this series so I can already have
a look?
I can prepare you a branch based on current upstream kernel next week
if you want to test this.
That would be awesome, thanks.
Cheers
Dirk
___
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
___
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: PCIe P2P access to GPU memory
Hi Christian,
I took the repository Dieter suggested and gave it a spin. I am not
sure, if I got the API right.
In userspace I create a bo and get a fd.
bo = gbm_bo_create(gbm, w, h, GBM_FORMAT_ARGB, GBM_BO_USE_LINEAR);
dmabuf_fd = gbm_bo_get_fd(bo);
Then I pass the fd to the grabber driver. There I get a sgt, that I
want to pass to the DMA-machine.
struct dma_buf_attach_info dma_buf_attach_info = {
.dev = &engine->xdev->pdev->dev,
.peer2peer = true,
};
dma_buf = dma_buf_get(xdma_dma_buf_ioctl.dma_buf_fd);
dma_buf_attach_info.dmabuf = dma_buf;
dma_buf_attachment = dma_buf_attach(&dma_buf_attach_info);
sgt = dma_buf_map_attachment(dma_buf_attachment, DMA_TO_DEVICE);
But the addresses in the sgt do not look like PCIe addresses at at all.
Any ideas?
Cheers
Dirk
Am Sa., 29. Sep. 2018 um 11:53 Uhr schrieb Dieter Nützel :
>
> Hello Dirk,
>
> I think Christian is talking about this branch:
>
> https://cgit.freedesktop.org/~deathsimple/linux/log/?h=p2p
>
> His 'home' is, here:
> https://cgit.freedesktop.org/~deathsimple/linux/
>
> Happy hacking! ;-)
>
> Dieter
>
> Am 29.09.2018 10:17, schrieb Dirk Eibach:
> >> This is work in progress.
> >>
> >> I published patches to enable DMA_buf P2P a few months ago, but now
> >> I'm waiting for the PCI subsystem to pick up core support for this.
> >
> > Great news! Can you give me a link to this series so I can already have
> > a look?
> >
> >> I can prepare you a branch based on current upstream kernel next week
> >> if you want to test this.
> >
> > That would be awesome, thanks.
> >
> > Cheers
> > Dirk
> > ___
> > dri-devel mailing list
> > dri-devel@lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/dri-devel
___
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: PCIe P2P access to GPU memory
Hello Dirk, I think Christian is talking about this branch: https://cgit.freedesktop.org/~deathsimple/linux/log/?h=p2p His 'home' is, here: https://cgit.freedesktop.org/~deathsimple/linux/ Happy hacking! ;-) Dieter Am 29.09.2018 10:17, schrieb Dirk Eibach: This is work in progress. I published patches to enable DMA_buf P2P a few months ago, but now I'm waiting for the PCI subsystem to pick up core support for this. Great news! Can you give me a link to this series so I can already have a look? I can prepare you a branch based on current upstream kernel next week if you want to test this. That would be awesome, thanks. Cheers Dirk ___ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel ___ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: PCIe P2P access to GPU memory
> This is work in progress. > > I published patches to enable DMA_buf P2P a few months ago, but now I'm > waiting for the PCI subsystem to pick up core support for this. Great news! Can you give me a link to this series so I can already have a look? > I can prepare you a branch based on current upstream kernel next week if you > want to test this. That would be awesome, thanks. Cheers Dirk ___ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: PCIe P2P access to GPU memory
This is work in progress. I published patches to enable DMA_buf P2P a few months ago, but now I'm waiting for the PCI subsystem to pick up core support for this. I can prepare you a branch based on current upstream kernel next week if you want to test this. Regards, Christian. Am 29.09.2018 09:01 schrieb Dirk Eibach : I want to access GPU VRAM via PCIe P2P access, like DirectGMA but with mainline drivers. I can acquire a dmabuf from the GPU and pass it to my PCIe framegrabber. But I don't see a way to get the PCIe bus address of the video memory which I need for the P2P transfer. Is there already any infrastructure in place for doing this? If not, what would be the right way to implement this? Add another callback to the dmabuf struct? Cheers Dirk ___ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel
