This is the next version of Jason's DMA-buf mapping types series [1]. It keeps
the framework and the SGT conversions from the RFC but defers the final mapping
type selection until the first pin/map of the buffer.
At pin/map the exporter chooses one of the types matched, taking the other
attachments and the buffer's placement into account. That choice is held by the
core until the exporter explicitly invalidates the attachment. This ensures
unmap/remap cannot cause buffer movement and ping/pong.
Future work will build on top of this series to add an iommufd importer type
per the plan in [2], separate out PCIe P2P as its own mapping type and convert
XGMI to an exclusive mapping type.
Why mapping types
-----------------
Since its introduction, DMA-buf has only supported using scatterlist for
the exporter and importer to exchange address information. This is not
sufficient for all use cases as dma_addr_t is a very specific and limited
type that should not be abused for things unrelated to the DMA API.
There are several motivations for addressing this now:
1) VFIO to IOMMUFD and KVM requires a physical address, not a dma_addr_t
scatterlist, it cannot be represented in the scatterlist structure
2) xe vGPU requires the host driver to accept a DMABUF from VFIO of its
own VF and convert it into an internal VRAM address on the PF
3) We are starting to look at replacement datastructures for
scatterlist
4) Ideas around UALink/etc are suggesting not using the DMA API
5) Private interconnects, such as XGMI, are forced to work around this using
driver-specific hacks.
None of these can be achieved using scatterlist. This series introduces a
"mapping type" mechanism allowing dma-buf to negotiate compatibility between
different types and support alternate mapping options. See the description of
patch 1 for more details.
Selecting mapping type
----------------------
Importers and exporters list possible mapping types with the exporter
responsible for selecting the final type at pin/map time under the dma_resv
lock. A matching phase at attachment time eliminates types which statically
cannot be supported for a given exporter/importer.
This was done in response to review comments on the RFC. Karl found that the
RFC's match callback ran on an uninitialised attachment [3]. Christian noted
that the final decision has to be made under the dma_resv lock before a
mapping is created, not at attach [4], and Thomas proposed the three steps
followed here: agree the possible types at attach, select at map, hold the
selection until invalidate_mappings() [5]. Christian also noted that remaps
must not change type or placement as that makes buffers ping/pong [6].
The dma-buf core enforces that last point by calling select_mapping() only on
the first pin/map after attach or after an invalidate. The pin/map type can
therefore only change once the exporter has explicitly invalidated the mapping.
Jason noted that some hardware, mlx5 for example, cannot change path once it
has mapped [7]. A pinned importer is covered, since its type is fixed at pin.
An unpinned importer has no way yet to narrow its own set after an invalidate.
This could be resolved if required by introducing a pin operation done under the
dma_resv lock for the mapping selection after attach.
Future work
-----------
This series stops at the SGT type so that the framework and the
attach/pin/map contract can be reviewed on their own. I already have
proof-of-concept conversions of both PCIe P2P (building on Thomas's work [8])
and XGMI to separate mapping types. If those would be helpful for reviewing
this series let me know and I will post them.
Alternatively this series could be split up with patches 1-8 & 25-26 adding the
general framework and the rest adding the conversions later.
Further work, per [2], is an iommufd importer type to replace the private
vfio/iommufd interface and allow other DMA-bufs to be imported into iommufd.
Testing
-------
- KUnit dma-buf-mapping suite (patches 25 and 26).
- Compile-tested with a config enabling all converted drivers.
Changes since RFC v1 [1]
------------------------
- Dropped the PAL type and its vfio/iommufd users (RFC 21-25).
- Rebased onto v7.3-rc5: move_notify is now invalidate_mappings,
CONFIG_DMABUF_MOVE_NOTIFY is gone, new amdxdna/panfrost/panthor
exporters converted, kzalloc_obj().
- Patch 1 records the whole match set in a fixed-size array in the
attachment; patch 3 initialises the attachment before match_mapping()
(Karl); patch 15 holds a pci_dev reference instead of asserting the resv
lock (Baolu).
- New patches 5, 6 and 7: exporter selection at pin/map with core-enforced
stickiness, match set queries, and dma_buf_mapping_map()/unmap().
- xe and amdgpu conversions written against the final API; amdgpu's XGMI
override rewrites the recorded match set entry.
- New patch 26 with KUnit tests for the selection contract.
[1]
https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
[4] https://lore.kernel.org/all/[email protected]/
[5]
https://lore.kernel.org/all/[email protected]/
[6] https://lore.kernel.org/all/[email protected]/
[7] https://lore.kernel.org/all/[email protected]/
[8]
https://gitlab.freedesktop.org/thomash/xe-vibe/-/commit/7452df0fc858a7dc5ccac0e2037852cb7ab64d61
Cc: Jason Gunthorpe <[email protected]>
Cc: Christian König <[email protected]>
Cc: Sumit Semwal <[email protected]>
Cc: Thomas Hellström <[email protected]>
Cc: Simona Vetter <[email protected]>
Cc: Matthew Brost <[email protected]>
Cc: Vivek Kasireddy <[email protected]>
Cc: Dongwon Kim <[email protected]>
Cc: Leon Romanovsky <[email protected]>
Cc: Kevin Tian <[email protected]>
Cc: Xu Yilun <[email protected]>
Cc: Karl Mehltretter <[email protected]>
Cc: Baolu Lu <[email protected]>
Cc: Felix Kuehling <[email protected]>
Cc: Alex Deucher <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: Alexey Kardashevskiy <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Alistair Popple (4):
dma-buf: Let the exporter select mapping type at pin/map
dma-buf: Add match set queries for exporter placement decisions
dma-buf: Add dma_buf_mapping_map() returning a type-tagged mapping
dma-buf: Add kunit tests for mapping type selection
Jason Gunthorpe (22):
dma-buf: Introduce DMA-buf mapping types
dma-buf: Add the SGT DMA mapping type
dma-buf: Add dma_buf_mapping_attach()
dma-buf: Route SGT related actions through attach->map_type
dma-buf: Allow single exporter drivers to avoid the match_mapping
function
drm: Check the SGT ops for drm_gem_map_dma_buf()
dma-buf: Convert all the simple exporters to use SGT mapping type
drm/vmwgfx: Use match_mapping instead of dummy calls
accel/habanalabs: Use the SGT_P2P mapping type
drm/xe/dma-buf: Use the SGT mapping type
drm/amdgpu: Use the SGT mapping type
vfio/pci: Change the DMA-buf exporter to use mapping_type
RDMA/uverbs: Use the SGT_P2P mapping type
dma-buf: Update dma_buf_phys_vec_to_sgt() to use the SGT mapping type
iio: buffer: convert to use the SGT mapping type
functionfs: convert to use the SGT mapping type
dma-buf: Remove unused SGT stuff from the common structures
treewide: Rename dma_buf_map_attachment(_unlocked) to dma_buf_sgt_
treewide: Rename dma_buf_unmap_attachment(_unlocked) to dma_buf_sgt_*
treewide: Rename dma_buf_attach() to dma_buf_sgt_attach()
treewide: Rename dma_buf_dynamic_attach() to
dma_buf_sgt_dynamic_attach()
dma-buf: Add kunit tests for mapping type
Documentation/gpu/todo.rst | 2 +-
drivers/accel/amdxdna/amdxdna_cbuf.c | 8 +-
drivers/accel/amdxdna/amdxdna_gem.c | 14 +-
drivers/accel/amdxdna/amdxdna_ubuf.c | 8 +-
drivers/accel/habanalabs/common/memory.c | 54 +-
drivers/accel/ivpu/ivpu_gem.c | 10 +-
drivers/accel/ivpu/ivpu_gem_userptr.c | 11 +-
drivers/accel/qaic/qaic_data.c | 8 +-
drivers/dma-buf/Makefile | 1 +
drivers/dma-buf/dma-buf-mapping.c | 259 ++++++-
drivers/dma-buf/dma-buf.c | 317 ++++++---
drivers/dma-buf/heaps/cma_heap.c | 12 +-
drivers/dma-buf/heaps/system_heap.c | 13 +-
drivers/dma-buf/st-dma-mapping.c | 662 ++++++++++++++++++
drivers/dma-buf/udmabuf.c | 8 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 141 ++--
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 6 +-
drivers/gpu/drm/armada/armada_gem.c | 33 +-
drivers/gpu/drm/drm_gem_shmem_helper.c | 2 +-
drivers/gpu/drm/drm_prime.c | 31 +-
drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 18 +-
drivers/gpu/drm/i915/gem/i915_gem_object.c | 2 +-
.../drm/i915/gem/selftests/i915_gem_dmabuf.c | 8 +-
.../gpu/drm/i915/gem/selftests/mock_dmabuf.c | 8 +-
drivers/gpu/drm/msm/msm_gem_prime.c | 7 +-
drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c | 11 +-
drivers/gpu/drm/panfrost/panfrost_gem.c | 13 +-
drivers/gpu/drm/panthor/panthor_gem.c | 11 +-
drivers/gpu/drm/tegra/gem.c | 33 +-
drivers/gpu/drm/virtio/virtgpu_prime.c | 23 +-
drivers/gpu/drm/vmwgfx/vmwgfx_prime.c | 32 +-
drivers/gpu/drm/xe/xe_bo.c | 18 +-
drivers/gpu/drm/xe/xe_dma_buf.c | 64 +-
drivers/iio/industrialio-buffer.c | 15 +-
drivers/infiniband/core/umem_dmabuf.c | 15 +-
.../infiniband/core/uverbs_std_types_dmabuf.c | 30 +-
drivers/iommu/iommufd/pages.c | 5 +-
drivers/iommu/iommufd/selftest.c | 18 +-
.../media/common/videobuf2/videobuf2-core.c | 2 +-
.../common/videobuf2/videobuf2-dma-contig.c | 26 +-
.../media/common/videobuf2/videobuf2-dma-sg.c | 21 +-
.../common/videobuf2/videobuf2-vmalloc.c | 13 +-
.../platform/nvidia/tegra-vde/dmabuf-cache.c | 9 +-
drivers/misc/fastrpc.c | 21 +-
drivers/tee/tee_heap.c | 13 +-
drivers/usb/gadget/function/f_fs.c | 11 +-
drivers/vfio/pci/vfio_pci_dmabuf.c | 25 +-
drivers/xen/gntdev-dmabuf.c | 29 +-
include/linux/dma-buf-mapping.h | 370 ++++++++++
include/linux/dma-buf.h | 227 +++---
include/trace/events/dma_buf.h | 2 +-
io_uring/zcrx.c | 9 +-
net/core/devmem.c | 14 +-
samples/vfio-mdev/mbochs.c | 10 +-
sound/soc/fsl/fsl_asrc_m2m.c | 12 +-
55 files changed, 2163 insertions(+), 582 deletions(-)
create mode 100644 drivers/dma-buf/st-dma-mapping.c
base-commit: 72d3fcf802c45d00b300f25b848a93c3a2bd7c7e
--
2.54.0
