Background
----------
An earlier RFC [1] introduced a new ioctl (AMDGPU_VM_OP_SET_L2_TRAP)
to allow userspace to register a second-level trap handler for user
queues. During review, it was noted that RADV on Vega/Navi hardware
uses kernel queues (context-based submission), not user queues, and
therefore cannot benefit from the second-level trap handler without
first-level trap handler support for kernel queue VMIDs.
Problem
-------
On GFX11+ (MES-based hardware), MES programs SQ_SHADER_TBA/TMA for
user queue VMIDs via the ADD_QUEUE packet's trap_handler_addr field.
However, MES maps kernel queues via ADD_QUEUE with map_legacy_kq=1 but
does NOT program the trap handler for those VMIDs.
On GFX10 and earlier (HWS-based), the driver programs trap handler
registers via SRBM select when KFD queues are set up, but no equivalent
programming exists for the driver-managed kernel queue VMIDs.
Key design decisions:
- MES owns kernel VMIDs but does not program trap handler state
- The driver must program SQ_SHADER_TBA/TMA directly via SRBM select
for kernel queue VMIDs
- Trap handler is a VMID property, not a queue property
- Only kernel queue VMIDs (1..first_kfd_vmid-1) should be programmed
here; user queue VMIDs are handled by MES
Use Cases
---------
- RADV graphics debugging on Vega/Navi/Steam Deck (Valve)
- Future Navi ray tracing features requiring trap handlers on kernel
queues
- Consistent first-level trap handler behavior when userspace switches
between kernel queues and user queues
Design
------
A new vmhub callback (program_kernel_trap_vmids) is added to
amdgpu_vmhub_funcs. Each gfxhub version implements this callback to
write SQ_SHADER_TBA/TMA registers for kernel queue VMIDs. A device-level
TMA BO (kq_tma_bo) is created at trap_init time as the scratch buffer
for kernel queue trap context.
The registers are programmed at two points:
1. amdgpu_trap_init() — on first boot, after ISA and TMA BOs are ready
2. setup_vmid_config() — on GPU resume, after GART registers are restored
GFX versions covered:
- GFX10 (gfxhub_v2_0): mm-prefixed registers, single XCC
- GFX11 (gfxhub_v3_0): reg-prefixed registers, single XCC
- GFX11.5 (gfxhub_v11_5_0): reg-prefixed registers, single XCC
- GFX12 (gfxhub_v12_0): reg-prefixed registers, single XCC
- GFX12.1 (gfxhub_v12_1): reg-prefixed registers, multi-XCC
Open Questions
--------------
1. XCP partitioning: gfxhub_v12_1 programs all XCC instances on each
call. For XCP partition suspend/resume, only a subset of XCCs may
need reprogramming. This will be addressed in a follow-up.
2. MES firmware: ideally MES should honor trap_en regardless of queue
type, but MES APIs are frozen. The driver-side SRBM approach in
this series is the agreed workaround.
This series depends on [1] (second-level trap handler RFC) for the
amdgpu_trap infrastructure (isa_bo, amdgpu_trap_init, amdgpu_trap_alloc).
[1] https://patchwork.freedesktop.org/series/172489/
Srinivasan Shanmugam (2):
drm/amdgpu: Add kernel VMID trap handler infrastructure
drm/amdgpu: Implement kernel VMID trap handler for GFX10/11/12
drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h | 1 +
drivers/gpu/drm/amd/amdgpu/amdgpu_trap.c | 37 ++++++++++++++++++
drivers/gpu/drm/amd/amdgpu/amdgpu_trap.h | 2 +
drivers/gpu/drm/amd/amdgpu/gfxhub_v11_5_0.c | 43 +++++++++++++++++++++
drivers/gpu/drm/amd/amdgpu/gfxhub_v12_0.c | 34 ++++++++++++++++
drivers/gpu/drm/amd/amdgpu/gfxhub_v12_1.c | 37 ++++++++++++++++++
drivers/gpu/drm/amd/amdgpu/gfxhub_v2_0.c | 34 ++++++++++++++++
drivers/gpu/drm/amd/amdgpu/gfxhub_v3_0.c | 34 ++++++++++++++++
8 files changed, 222 insertions(+)
--
2.34.1
