AMD General Well, per the log, my assumption is not correct. Clock gating is enabled so activity 100% is really a fw/hw reported value. Need to check into the fw on how this activity is calculated even though there is not submitted job. Thanks.
From: Kevin Hake <[email protected]> Sent: Wednesday, September 2, 2026 6:21 PM To: Feng, Kenneth <[email protected]> Cc: [email protected]; Deucher, Alexander <[email protected]> Subject: Re: [drm/amd #3549] Navi 33 idle sclk latch: repro, analysis, userspace workaround Following up from last week: I added tracing to the monitor so the submits and dispatches can be linked to the userspace process that created them. I don't consistently see 13 (or any) jobs when starting up the monitor, but it does seem to be just regular system noise. I added cgmask polling to get the info you asked for (and the other sibling data in there). At first I only read it when clocks are high, thinking it might wake the card from deep sleep, but turns out reading it while sclk == 0 didn't change the behavior, sclk stayed low when polling it every second. I did get a hang once while switching back and forth between TTY1 (KDE) and TTY3 (SMU stopped responding, kwin stuck in a modeset) and had to hard reboot, maybe related to polling amdgpu_pm_info every second across the VT switch? Anyway, what I see right away is that when sclk is 0, cgmask == 0x3bc080001. When sclk is pegged high, cgmask == 0x13c08000c, which is the same I see with normal GPU usage (e.g. poking around KDE). CG flag sleep pegged/normal GPU use 0x3bc080001 0x13c08000c ----------------------------------------------------------- GFX MGCG (medium grain) On Off GFX FGCG (fine grain) On Off GFX PERF_CLK On Off GFX CGCG (coarse grain) Off On GFX CGLS (coarse mem light-sleep) Off On ATHUB MGCG / ATHUB LS On On BIF MGCG On On HDP SD On On IH CG On On REPEATER FGCG On On So it seems gating during the peg is no different than during regular usage. An interesting note: teardown of the poke application when sclk = 0 will trigger sclk to peg to max, but teardown when sclk is already pegged does NOT reset sclk back to 0. Whereas the NOP 'poke' toggles it in both directions 0 <-> max. In other words, when sclk is 0, it looks like any GPU activity, even the kernel writing page tables on the SDMA engine, will push sclk to max, but only a completed gfx-ring job brings it back down. Attached are logs from an example run with the updated monitor. The order of events: - switch to TTY3, run the monitor as root - switch back to TTY1 KDE, wiggle some things, return to TTY3 (this is to establish cgmask for regular system activity) - card is sleeping, launch poke app (02:59:40), sclk pegs - poke some NOPs, card back to sleep, teardown poke app (03:00:08), pegs again - launch poke app again (03:00:13), still pegged, send some NOPs, card toggles state, we leave it on pegged - tear down poke app (03:00:23), card remains pegged Kevin Hake +1 716 432 7106 +52 81 1300 5479 Kevin Hake +1 716 432 7106 +52 81 1300 5479 On Tue, Aug 25, 2026 at 10:01 AM Kevin Hake <[email protected]<mailto:[email protected]>> wrote: Ah ok, when I'm back home on Friday I'll add more logging and send an update. Thanks! Kevin Hake +1 716 432 7106<tel:(716)%20432-7106> +52 81 1300 5479<tel:+52%2081%201300%205479> On Tue, Aug 25, 2026, 7:18 AM Feng, Kenneth <[email protected]<mailto:[email protected]>> wrote: AMD General I am suspecting that clock gating could be disabled so the utilization is stuck at 100%. When the utilization is 100%, any chance you can check on your system with debugfs: /sys/kernel/debug/dri/x/amdgpu_pm_info? The output: "Clock Gating Flags Mask:…” indicates if clock gating is disabled or not. Thanks. From: Kevin Hake <[email protected]<mailto:[email protected]>> Sent: Tuesday, August 25, 2026 3:23 PM To: Feng, Kenneth <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]>; Deucher, Alexander <[email protected]<mailto:[email protected]>> Subject: Re: [drm/amd #3549] Navi 33 idle sclk latch: repro, analysis, userspace workaround The KDE session is actually still live on tty1 during that test, I've just switched to TTY3. I'm away for a few days so can't confirm if I have logs of exactly what sent those, but it may be regular background system activity like maybe Wayland cleaning up context(?) But the NOP is just the smallest reproducer I could come up with - I originally caught the behavior just watching regular system activity. When I saw sclk toggle from 0 to high and stay there with no other jobs dispatched, iirc it was always <10 jobs showing up on that log line. My first attempt to trigger it myself was a simple open gl task, but that didn't work - iirc setting up the context, running it, and destructing dispatched a larger bunch of tasks (20-something). It wasn't until I held the context between "pokes" that I could get the manual clock toggling behavior. Maybe worth noting the Python version of the "poke" reproducer (used in the demo) is using a GL stack to dispatch an empty compute shader, whereas the c poke is the more barebones PM4 packet with NOP, but either one acts as a toggle. In the wild, the pattern seems to be: GPU idles, sclk->0 Lone task comes in and completes with nothing behind it sclk->max Later, another task comes in sclk->0 (my speculation is queuing or completing a task triggers a test of % of recent idle clocks, and since it's been sitting at max clock with nothing to do, it sees 100% idle clocks and immediately throttles back down to 0) - Kevin On Mon, Aug 24, 2026, 9:04 PM Feng, Kenneth <[email protected]<mailto:[email protected]>> wrote: AMD General Thanks Kevin. At the beginning of the video, right after gpu-monitor.py is executed, there are 13 jobs dispatched. Are they also the NOP packets? Thanks. [https://mail.google.com/mail/u/0?ui=2&ik=312744bcb4&attid=0.2&permmsgid=msg-a:r4899060653325978718&view=fimg&fur=ip&permmsgid=msg-a:r4899060653325978718&sz=s0-l75-ft&attbid=ANGjdJ_ZrDfC-Lj3sK1o-8hrVvwG06Lgvv6W33xH91Ojq717jCVASxVCkYWR2iqY5GSUjFhHK5lPsoeWlTiFkr_PYzsq9-rQ-zS1B-pFh2dlyMVHst_Yg34Ws0ZCAFc&disp=emb&realattid&zw] From: Kevin Hake <[email protected]<mailto:[email protected]>> Sent: Tuesday, August 25, 2026 3:22 AM To: Feng, Kenneth <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]>; Deucher, Alexander <[email protected]<mailto:[email protected]>> Subject: Re: [drm/amd #3549] Navi 33 idle sclk latch: repro, analysis, userspace workaround You don't often get email from [email protected]<mailto:[email protected]>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Hi Kenneth, I think that's the bug - it is reporting 100% utilization but the GPU is idle - I know it's idle because I traced job submissions and dispatch, and separately drm_sched_job_done, debugfs pending fences, the kernel's drm-engine-* nanoseconds - everything points to nothing actually crunching, at least nothing real from userspace. Not to mention I'm in TTY with minimal system activity. You can see the behavior in this demo: https://asciinema.org/a/Qc5CQBW6ycb89ldd I'm wondering if there's a bug where utilization is calculated something something like, "how many idle clocks have there been since we last checked?". If the gpu has been asleep with sclk == 0, then it sees 0 idle clocks in the past n seconds (no idle clocks?? we must be really busy!), and sets utilization to 100. But I'm just speculating, what I can confirm is that with sclk at 0, a single NOP to the gpu toggles utilization to 100% and sclk to max. It seems odd numbers of very small work trigger the toggle, When larger groups of work come in it behaves more normally and doesn't "stick" high. Thanks, Kevin Hake +1 716 432 7106<tel:(716)%20432-7106> +52 81 1300 5479<tel:+52%2081%201300%205479> On Mon, Aug 24, 2026 at 8:10 AM Feng, Kenneth <[email protected]<mailto:[email protected]>> wrote: AMD General What’s the GPU utilization(observed by sudo cat /sys/kernel/debug/dri/x/amdgpu_pm_info) when the sclk is highest? The policy is that if the utilization is 100%, the clock has to stay highest. Could you please elaborate what ‘a tiny job’ is to trigger the issue? Thanks. From: amd-gfx <[email protected]<mailto:[email protected]>> On Behalf Of Kevin Hake Sent: Sunday, August 23, 2026 11:05 PM To: [email protected]<mailto:[email protected]> Cc: Deucher, Alexander <[email protected]<mailto:[email protected]>> Subject: [drm/amd #3549] Navi 33 idle sclk latch: repro, analysis, userspace workaround Some people who received this message don't often get email from [email protected]<mailto:[email protected]>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Hi all, Follow-up to drm/amd issue #3549 ("AMD GPU usage peaks with system in idle — Radeon RX 7600 XT"), where I posted a root-cause analysis and reproducer a few weeks ago (note 3586398). The thread's been quiet, so I'm summarizing here in case it's useful to whoever owns PMFW/swsmu for SMU 13.0.7. Behavior (RX 7600 XT / Navi 33, kernels 6.x–7.1, likely any DE (tho I'm on Plasma w/Wayland) or bare TTY): - With the gfx job stream verifiably empty, a single tiny job submitted while sclk is at 0 (deep sleep) latches sclk at max (~2990 MHz, ~115 W) indefinitely. A second completed gfx job clears it. So at true idle the card toggles between 0 and max on every stray job, and can sit at max for an hour with the screen off. - Light sustained load (video, a compositor, radeontop) masks it, which is why the earlier "monitoring tools toggle gfxoff" hypothesis looked plausible. It reproduces on demand from a TTY with nothing polling. - Umio's finding that disabling GFXOFF via ppfeaturemask stops the spikes is consistent with this model: without GFXOFF there is no "single job from deep sleep" transition to mis-handle. How I measured: - Job stream: a private ftrace instance on amdgpu_cs_ioctl / amdgpu_sched_run_job / drm_sched_job_done, cross-checked against fdinfo drm-engine counters and debugfs amdgpu_fence_info, so I could prove the kernel submitted nothing while clocks moved. - Clock: sampling the firmware's gpu_metrics (v1.3) at 1 kHz, matched to the driver's 1 ms refresh of that table, to catch the ramp edges. - Stimulus: a minimal compute dispatch with a held context (no setup/ teardown jobs), and later a one-packet PM4 NOP via libdrm — either one toggles the latch. Everything points at the firmware's clock selection rather than the driver. Since PMFW can't be patched from outside, I published a small userspace workaround: a C daemon over libdrm/render node that submits one PM4 NOP when it sees sclk latched (same binary is the manual reproducer): https://github.com/KevinHake/amdgpu-unstick Scripts, logs, and an asciinema demo are attached to the issue. If a narrower repro would help (specific firmware versions, mesa-free path, other kernels), tell me what you need and I'll run it. Thanks, Kevin Hake
