Hi, On 10/25/23 03:29, Emanuele Rocca wrote:
I hesitate to actually enable it because I don't understand PCI good enough to judge it's a safe choice for the Debian kernel.
There is a kernel API for "return the physical address of a BAR mapping on another PCI device, as seen from the point of view of this device."
With the P2PDMA option disabled, that API always returns "no peer-to-peer capable path exists", which requires drivers to fall back to allocating a buffer.
If the option is enabled, a driver exporting a DMA buffer (such as a video capture device or a GPU) can provide an address that is part of one of its BAR mappings. The driver importing the DMA buffer will then use the physical address it was given in DMA requests, which routes the requests directly and avoids passing through the root complex.
I have successfully used this to make an nVidia GPU generate PCIe requests for textures that were answered by an FPGA card. :> The driver importing the buffer needs no special support.
There is no generic mechanism to use peer-to-peer transfers if not at least one side is aware of that mechanism, as it essentially requires the exporting device to implement full support for prefetchable BAR mappings.
Im principle, all platforms that support PCIe can benefit from this, as it allows peer-to-peer transfers even when the root complex does not support this, as long as all bridges on the path between the involved devices do. In practice, most transfers will happen between a GPU (in a slot directly connected to a root bridge) and some data capture device, so the benefit on platforms other than amd64 and ppc64le will be limited -- although these will just fall back to the current behaviour, and the only code paths affected are slow paths that usually end in TLB flushes for CPU and IO MMUs.
Simon
