Re: [PATCH 2/3] drm/amdkfd: balance KFD suspend reference on partition switch

[Lazar, Lijo]

On 05-Sep-26 10:00 AM, Xiang Liu wrote:
A compute partition switch tears down the KFD device through
amdgpu_amdkfd_device_fini_sw() without suspending KFD first, while
kgd2kfd_device_exit() drops a suspend reference. Every switch leaves
kfd_locked one below zero, so on a hive the counter ends up deeply
negative.

kfd_locked gates the eviction sweep on "++kfd_locked == 1". While it is
negative a GPU reset never evicts user processes, nothing restores them
afterwards, and page tables torn down with VRAM are never rebuilt, so the
first queue mapped after the reset takes a VM fault.

Take the reference in amdgpu_xcp_pre_partition_switch() so it pairs with
the one kgd2kfd_device_exit() drops, and only drop it for a device that
finished initialisation, matching the early return in kgd2kfd_suspend().

Signed-off-by: Xiang Liu <xiang....@amd.com>
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_xcp.c | 5 ++++-
  drivers/gpu/drm/amd/amdkfd/kfd_device.c | 6 +++---
  2 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_xcp.c 
b/drivers/gpu/drm/amd/amdgpu/amdgpu_xcp.c
index b4ebc157fd06..e2e7d101b3c4 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_xcp.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_xcp.c
@@ -698,8 +698,11 @@ int amdgpu_xcp_pre_partition_switch(struct amdgpu_xcp_mgr 
*xcp_mgr, u32 flags)
         * Stop user queues and threads, and make sure GPU is empty of work.
         */
  
-	if (flags & AMDGPU_XCP_OPS_KFD)
+       /* kgd2kfd_device_exit() drops the suspend reference taken here */
+       if (flags & AMDGPU_XCP_OPS_KFD) {
+               amdgpu_amdkfd_suspend(xcp_mgr->adev, true);

Partition switch doesn't force a suspend. It's a prerequisite that user 
has to exit all KFD processes before doing the switch. So before 
partition switch amdgpu_amdkfd_check_and_lock_kfd is called to ensure 
that no processes are running on the device.

If there is an imbalance in lock count, it may need a different way to 
solve it.

Thanks,
Lijo

                amdgpu_amdkfd_device_fini_sw(xcp_mgr->adev);
+       }
  
  	return 0;
  }
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c 
b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
index 729b35841e3b..d76920adc466 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
@@ -1014,12 +1014,12 @@ void kgd2kfd_device_exit(struct kfd_dev *kfd)
                kfd_gtt_sa_fini(kfd);
                amdgpu_amdkfd_free_kernel_mem(kfd->adev, &kfd->gtt_mem);
                mutex_destroy(&kfd->profiler_lock);
+
+               /* pairs with the reference taken in kgd2kfd_suspend() */
+               kgd2kfd_unlock_kfd(NULL);
        }
  
  	kfree(kfd);
-
-       /* after remove a kfd device unlock kfd driver */
-       kgd2kfd_unlock_kfd(NULL);
  }
  
  int kgd2kfd_pre_reset(struct kfd_dev *kfd,