On GFX 11.5 APUs resume from hibernation intermittently fails because MES stops responding while gfx_v11_0 is being resumed, for example:
amdgpu: ring mes_kiq_3.1.0 test failed (-110) amdgpu: MES failed to respond to msg=SET_HW_RSRC amdgpu: MES failed to respond to msg=ADD_QUEUE amdgpu: resume of IP block <gfx_v11_0> failed -110 MES keeps timing out after the MODE2 reset done by the subsequent GPU recovery, the display stays dark and the session is lost. This reproduces without a power cycle with the PM test facility: echo platform > /sys/power/pm_test echo disk > /sys/power/state On a Framework Laptop 13 (Ryzen AI 9 HX 370, GC 11.5.0, MES 0x94) about one in four freeze/restore cycles fails, at a different point of the MES bring-up each time. GCVM and MMVM report no faults. The failure depends on the GPU being frozen while other devices are still running their freeze callbacks: - pm_async=0: no failure in 45 cycles - freeze callback delayed by 1 s: none in 45 cycles (60 ms: 1 in 20) - a 1 s pause right before the PG/CG ungate at the start of amdgpu_device_suspend() avoids it, the same pause right after the ungate does not; skipping the VRAM eviction does not help either - unbinding or unloading all devices that freeze concurrently and can be removed (TPM, NPU, PMF, xHCI/USB4, WLAN, HDA/ACP/CCP) together does not avoid it; the driver state at the start of the freeze is the same in passing and failing cycles The PM core runs the late freeze callbacks only after all regular freeze callbacks have completed, so do the freeze from .freeze_late on these APUs. This applies both to the kernel that creates the image and to the boot kernel that quiesces the device before jumping into the image. Since the device is no longer frozen in .freeze, don't resume it in .thaw/.restore when the transition was aborted before the late phase. With this change the same machine passed 40 of 40 cycles with pm_test=platform (without it, the run failed at cycle 2), and real hibernate/restore cycles with power-off, including s2idle followed by hibernation, all resumed cleanly. Closes: https://gitlab.freedesktop.org/drm/amd/-/issues/5379 Link: https://gitlab.freedesktop.org/drm/amd/-/issues/5788 Link: https://gitlab.freedesktop.org/drm/amd/-/issues/4545 Signed-off-by: Robert Testagrossa <[email protected]> --- drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 45 ++++++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c index 04b4f1e72..e1b28ecde 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c @@ -2718,7 +2718,20 @@ static int amdgpu_pmops_resume(struct device *dev) return r; } -static int amdgpu_pmops_freeze(struct device *dev) +/* + * On GFX 11.5 APUs MES intermittently fails to come back on restore from + * hibernation when the GPU is frozen while other devices are still running + * their freeze callbacks. Do the freeze in the late phase for these parts, + * after all regular freeze callbacks have completed. + */ +static bool amdgpu_pmops_freeze_in_late_phase(struct amdgpu_device *adev) +{ + return (adev->flags & AMD_IS_APU) && + amdgpu_ip_version(adev, GC_HWIP, 0) >= IP_VERSION(11, 5, 0) && + amdgpu_ip_version(adev, GC_HWIP, 0) < IP_VERSION(11, 6, 0); +} + +static int amdgpu_pmops_do_freeze(struct device *dev) { struct drm_device *drm_dev = dev_get_drvdata(dev); struct amdgpu_device *adev = drm_to_adev(drm_dev); @@ -2737,11 +2750,35 @@ static int amdgpu_pmops_freeze(struct device *dev) return 0; } +static int amdgpu_pmops_freeze(struct device *dev) +{ + struct drm_device *drm_dev = dev_get_drvdata(dev); + + if (amdgpu_pmops_freeze_in_late_phase(drm_to_adev(drm_dev))) + return 0; + + return amdgpu_pmops_do_freeze(dev); +} + +static int amdgpu_pmops_freeze_late(struct device *dev) +{ + struct drm_device *drm_dev = dev_get_drvdata(dev); + + if (!amdgpu_pmops_freeze_in_late_phase(drm_to_adev(drm_dev))) + return 0; + + return amdgpu_pmops_do_freeze(dev); +} + static int amdgpu_pmops_thaw(struct device *dev) { struct drm_device *drm_dev = dev_get_drvdata(dev); struct amdgpu_device *adev = drm_to_adev(drm_dev); + /* the transition may have been aborted before the late freeze phase */ + if (!adev->in_suspend) + return 0; + /* do not resume device if it's normal hibernation */ if (!adev->debug_hibernation_thaw_resume_gpu && !pm_hibernate_is_recovering() && @@ -2766,6 +2803,11 @@ static int amdgpu_pmops_poweroff(struct device *dev) static int amdgpu_pmops_restore(struct device *dev) { struct drm_device *drm_dev = dev_get_drvdata(dev); + struct amdgpu_device *adev = drm_to_adev(drm_dev); + + /* nothing to do if the device was never frozen, e.g. pm_test=devices */ + if (!adev->in_suspend) + return 0; return amdgpu_device_resume(drm_dev, true); } @@ -3052,6 +3094,7 @@ static const struct dev_pm_ops amdgpu_pm_ops = { .suspend_noirq = pm_sleep_ptr(amdgpu_pmops_suspend_noirq), .resume = pm_sleep_ptr(amdgpu_pmops_resume), .freeze = pm_sleep_ptr(amdgpu_pmops_freeze), + .freeze_late = pm_sleep_ptr(amdgpu_pmops_freeze_late), .thaw = pm_sleep_ptr(amdgpu_pmops_thaw), .poweroff = pm_sleep_ptr(amdgpu_pmops_poweroff), .restore = pm_sleep_ptr(amdgpu_pmops_restore), -- 2.56.0
