On 2026-08-25 15:39, michael wizardsound.com wrote:
[Some people who received this message don't often get email
[email protected]. Learn why this is important
athttps://aka.ms/LearnAboutSenderIdentification ]
Hi AMD,
I submitted some findings and I wanted to send this over. I hope this helps.
On a 7900 XTX in a Proxmox host (kernel 6.14.8-2-pve, ROCm userspaces 7.2.x/7.14 in an LXC
sharing the host kernel), a userspace bug (rocm-systems#10021; root cause filed as
rocm-systems#10713, runtime fix proposed in rocm-systems PR #10714) can place a malformed
packet on a user AQL queue. The CP raises gfx_v11_0_bad_op_irq ("Illegal opcode in command
stream") followed by an HSA_STATUS_ERROR_INVALID_PACKET_FORMAT queue abort. After the
process exits, its /sys/class/kfd/kfd/proc/<pid> entry persists and ~25.7 GB VRAM stays
pinned. amdgpu mode1 reset succeeds but does not release it; unbind succeeds but re-probe fails
with EEXIST on mem_info_preempt_used (stale sysfs child from the wedged teardown), and
subsequent per-device remove/bind attempts leave D-state tasks in
device_release_driver_internal / device_driver_attach. Only a reboot recovers the device. Clean
SIGSEGV deaths of the same workload never leak; it is specific to the CP-fault teardown. Full
logs, timelines, and a userspace reproducer chain available; patches and standalone reproducers
for the userspace side are athttps://github.com/OhgurLabs/rocm-hipgraph-fixes.
The most common cause of this type of bug is a reference counting
problem in struct kfd_process. In this case it appears to be in in the
event handling code for SOC15_INTSRC_CP_BAD_OPCODE. There used to be a
missing kfd_unref_process in kfd_dqm_suspend_bad_queue_mes in
kfd_device_queue_manager.c. This was fixed as part of this commit over a
year ago:
commit 8544374c0f82edb285779f21b149826fe2c2977c
Author: Xiaogang Chen<[email protected]>
AuthorDate: Mon Jan 13 17:35:59 2025 -0600
Commit: Alex Deucher<[email protected]>
CommitDate: Wed Feb 12 21:02:55 2025 -0500
drm/amdkfd: Have kfd driver use same PASID values from graphic driver
Current kfd driver has its own PASID value for a kfd process and uses it to
locate vm at interrupt handler or mapping between kfd process and vm. That
design is not working when a physical gpu device has multiple spatial
partitions, ex: adev in CPX mode. This patch has kfd driver use same pasid
values that graphic driver generated which is per vm per pasid.
These pasid values are passed to fw/hardware. We do not need change
interrupt
handler though more pasid values are used. Also, pasid values at log are
replaced by user process pid; pasid values are not exposed to user. Users
see
their process pids that have meaning in user space.
Signed-off-by: Xiaogang Chen<[email protected]>
Reviewed-by: Felix Kuehling<[email protected]>
Signed-off-by: Alex Deucher<[email protected]>
I guess you are using the driver that comes with your 6.14.8-2-pve
kernel, not the DKMS package that shipped with ROCm. This fix landed in
Linux 6.15.
Regards,
Felix
Michael Levesque, MBA
