Two processes share a set of buffers, and each has buffers of its own which the other never sees. Say process A has mapped
S the shared buffers, also mapped by B P buffers private to A and B has mapped S plus private buffers of its own. The overlap is exactly S, and the work each process wants to do on its own buffers is independent of the other. A submits. Its exec locks the dma-resv of everything it has mapped, S and P both, then finds something in P has been evicted and migrates it back in. B submits, and blocks on S for as long as that migration takes, even though the migration is of a buffer belonging to A which B has never seen. So the stall does not come from the overlapping set. The buffers in S are resident, and neither exec has anything to do to them beyond attaching a fence. They are held only because an exec locks everything it has mapped in one go, and they stay held until the slowest unrelated thing in that transaction is done. Which buffers get evicted is a separate matter, and one which already has answers: eviction heuristics which leave shared buffers alone, or one process' allocations outranking another's. This is what is left once those work. Where this tends to show up is compositors and presentation, which is also where userspace has worked hardest to avoid it. Wayland explicit sync exists so that a compositor is not latched onto its clients' rendering, waiting on fences it never asked for. The locking above reintroduces that coupling anyway, in the kernel, and does it under memory pressure, which is where a missed frame is least welcome and the cause is hardest to see. The fix is to stop coupling "lock the VM" to "validate it". Instead of locking everything and then validating, lock the private buffers and the evicted external ones, validate those, and only then lock the rest, all within the same drm_exec transaction. Patch 1 lets a driver split the locking of an exec that way, patches 2 and 3 use it in Xe and Panthor, whose panthor_vm_bo_validate() swaps pages back in under those same shared locks. It is opt-in, and drivers which do not ask for it are unaffected. Patches 5 and 6 do the same for MSM. Its VM_BIND submit path locks every BO mapped in the VM and then validates the evicted ones, which means getting their pages and mapping them again, with the shared BOs locked throughout. Patch 4 switches VM_BIND VMs to DRM_GPUVM_RESV_PROTECTED, which two pass locking requires, and patch 5 splits the submit's locking into the two passes. Kernel managed VMs and the legacy submit path are left alone. The MSM part should resolve exactly the problem Anna Maniscalco presented at XDC, render jobs stalling the compositor on shared buffers, which is currently worked around with a downstream hack in MSM for SteamOS [1][2] (the talk starts at 5:30). Nouveau support for DRM_GPUVM_RESV_PROTECTED and two pass locking added in patches 7 and 8. All driver updated side from Xe untested. [1] https://indico.freedesktop.org/event/12/contributions/627/ [2] https://www.youtube.com/watch?v=j5W5ErEMnvM&t=330s Matt Cc: Abhinav Kumar <[email protected]> Cc: Alice Ryhl <[email protected]> Cc: Anna Maniscalco <[email protected]> Cc: Antonino Maniscalco <[email protected]> Cc: Boris Brezillon <[email protected]> Cc: Danilo Krummrich <[email protected]> Cc: David Airlie <[email protected]> Cc: Dmitry Baryshkov <[email protected]> Cc: Jessica Zhang <[email protected]> Cc: Jonathan Corbet <[email protected]> Cc: Liviu Dudau <[email protected]> Cc: Lyude Paul <[email protected]> Cc: Maarten Lankhorst <[email protected]> Cc: Marijn Suijten <[email protected]> Cc: Maxime Ripard <[email protected]> Cc: Randy Dunlap <[email protected]> Cc: Rob Clark <[email protected]> Cc: Rodrigo Vivi <[email protected]> Cc: Sean Paul <[email protected]> Cc: Shuah Khan <[email protected]> Cc: Simona Vetter <[email protected]> Cc: Steven Price <[email protected]> Cc: Thomas Hellström <[email protected]> Cc: Thomas Zimmermann <[email protected]> Assisted-by: LLM --- v3: - Address Danilo's comments of GPUVM patch - Update MSM to reject submit_bo on VM_BIND context (Sashiko) - Also update Nouevau Matthew Brost (8): drm/gpuvm: allow locking external objects in two passes drm/xe: lock the resident BOs of an exec last drm/panthor: lock the resident BOs of a submit last drm/msm: reject a submit_bo table on VM_BIND contexts drm/msm: use DRM_GPUVM_RESV_PROTECTED for VM_BIND VMs drm/msm: lock the resident BOs of a VM_BIND submit last drm/nouveau: use DRM_GPUVM_RESV_PROTECTED drm/nouveau: lock the resident BOs of an exec last Documentation/gpu/drm-mm.rst | 6 + drivers/gpu/drm/drm_gpuvm.c | 505 ++++++++++++++++++++++++- drivers/gpu/drm/msm/msm_gem_submit.c | 90 ++++- drivers/gpu/drm/msm/msm_gem_vma.c | 16 +- drivers/gpu/drm/nouveau/nouveau_exec.c | 27 +- drivers/gpu/drm/nouveau/nouveau_uvmm.c | 49 ++- drivers/gpu/drm/panthor/panthor_mmu.c | 55 ++- drivers/gpu/drm/xe/xe_exec.c | 23 +- drivers/gpu/drm/xe/xe_vm.c | 43 ++- drivers/gpu/drm/xe/xe_vm.h | 3 +- include/drm/drm_gpuvm.h | 139 ++++++- 11 files changed, 882 insertions(+), 74 deletions(-) -- 2.34.1
