Public bug reported:
**Descriptive title** (kept as the heading of the description field)
amdgpu/amdkfd SVM: hung `svm_range_restore_work` waits on a fence while
holding a range lock,
wedging kswapd and every allocating task; VRAM is never released and SIGKILL
does not work
(RX 7600 XT / gfx1102, kernel 7.0.0-38-generic)
**1. Summary**
On Ubuntu 26.04.1 with kernel `7.0.0-38-generic` (package `7.0.0-38.38`,
upstream base `7.0.14`),
repeated attempts to load a 13 GiB GGUF model through a ROCm `llama-server`
left the machine in an
unrecoverable state: the KFD SVM worker (`svm_range_restore_work`) got stuck in
`dma_fence_wait_timeout()` inside `svm_range_map_to_gpus()` while holding the
per-range lock, and
the kernel itself then reports that `kswapd0`, the model server, GNOME Shell, a
browser renderer, a
Python process and another thread are "blocked on a mutex likely owned by task
kworker/1:2:…".
VRAM was never released, `SIGKILL` was ineffective, the graphical session froze
and SSH dropped.
The only way out was the emergency SysRq reboot sequence, which left the
session's journal unclean;
the ≈12.8 GiB held by the GPU was released only at that reboot. §2 is quoted
from the kernel journal
of that boot; §3 separates what that evidence proves from what it does not.
**2. Confirmed evidence (verbatim from the kernel journal — full excerpt
attached)**
Host `<host>`; log times local (UTC-03). The complete, unedited call traces are
in the attachment
`ANEXO-TRECHOS-KERNEL-HIGIENIZADOS.txt`; this is the condensed version.
```
19:44:02 <host> kernel: workqueue: kfd_process_wq_release [amdgpu] hogged CPU
for >10000us 4 times, consider switching to WQ_UNBOUND
19:48:50 <host> kernel: workqueue: svm_range_restore_work [amdgpu] hogged CPU
for >10000us 4 times, consider switching to WQ_UNBOUND
19:49:18 <host> kernel: [drm:drm_sched_entity_push_job [gpu_sched]] *ERROR*
Trying to push to a killed entity
19:52:45 <host> kernel: INFO: task kswapd0:144 blocked for more than 122
seconds.
19:52:45 <host> kernel: task:kswapd0 state:D stack:0 pid:144 tgid:144 ppid:2
19:52:45 <host> kernel: mutex_lock+0x3b/0x50
19:52:45 <host> kernel: svm_range_cpu_invalidate_pagetables+0x7c/0x150 [amdgpu]
19:52:45 <host> kernel: __mmu_notifier_invalidate_range_start+0x149/0x1b0
19:52:45 <host> kernel: try_to_unmap_one+0xf70/0x1260
19:52:45 <host> kernel: shrink_folio_list+0x559/0x1080
19:52:45 <host> kernel: evict_folios+0x3b6/0xbe0
19:52:45 <host> kernel: balance_pgdat+0x2d3/0x940
19:52:45 <host> kernel: kswapd+0xeb/0x1a0
19:52:45 <host> kernel: INFO: task kswapd0:144 is blocked on a mutex likely
owned by task kworker/1:2:151726.
19:52:46 <host> kernel: INFO: task kworker/1:2:151726 blocked for more than 122
seconds.
19:52:46 <host> kernel: task:kworker/1:2 state:D stack:0 pid:151726
tgid:151726 ppid:2
19:52:46 <host> kernel: Workqueue: events_freezable svm_range_restore_work
[amdgpu]
19:52:46 <host> kernel: schedule_timeout+0xcf/0x110
19:52:46 <host> kernel: dma_fence_default_wait+0x1a0/0x280
19:52:46 <host> kernel: dma_fence_wait_timeout+0x13b/0x1c0
19:52:46 <host> kernel: svm_range_map_to_gpus+0x186/0x380 [amdgpu]
19:52:46 <host> kernel: svm_range_validate_and_map+0x8d3/0xaa0 [amdgpu]
19:52:46 <host> kernel: svm_range_restore_work+0xe0/0x3b0 [amdgpu]
19:52:46 <host> kernel: process_one_work+0x1ac/0x3d0
19:52:46 <host> kernel: worker_thread+0x1b8/0x360
```
Same boot, other confirmed facts:
- **Seven tasks were reported blocked in that round, and six of them wait on
the mutex owned by the
seventh.** The kernel printed "blocked for more than 122 seconds" for seven
distinct tasks and its
lock-owner note — "is blocked on a mutex likely owned by task
kworker/1:2:151726" — for **six** of
them, because the owner itself never gets that note. The seventh is therefore
`kworker/1:2:151726`,
the SVM worker quoted above; the six waiters are `kswapd0:144`, the model
server `ollama:2124`,
`gnome-shell:47229`, a browser renderer `Isolated Web Co:49818`,
`python3:154641` and
`Thread-3 (loop):154642`. All six wait inside
`svm_range_cpu_invalidate_pagetables`, i.e. in the
reclaim path.
- After another 122 s (19:54:48) three of them are re-reported as blocked for
more than 245 s, and
the kernel then prints "Future hung task reports are suppressed".
- **Recovery and unclean shutdown.** The wedged machine could not be brought
down normally: it was
recovered with the emergency **SysRq** reboot sequence (operator's account —
no SysRq trace
survives in the preserved journal of that boot, which ends at 20:04:50 with
the system already
stalled). The termination was unclean, as the next boot records for both
journal files:
`20:12:03 systemd-journald: File …/system.journal corrupted or uncleanly shut
down, renaming and
replacing`, and the same for the user journal at 20:12:13 (path elided). The
≈12.8 GiB of VRAM
held during the incident was released only at that reboot: afterwards the
card reports
`mem_info_vram_used = 268611584` (≈0.25 GiB) against `mem_info_vram_total =
17163091968`
(sysfs, post-reboot).
- The client aborted the load five times in 26 minutes (19:23:45, 19:28:48,
19:33:53, 19:39:07,
19:44:19); each abort logged `client connection closed before llama-server
finished loading,
aborting load` and `Load failed … "timed out waiting for llama-server to
start: context
canceled"`, immediately followed at 19:49:18 by the killed-entity message.
Every attempt had
reached `load_tensors: ROCm0 model buffer size = 12807.21 MiB`.
- Hung-task timeout is 122 s, so the wedged wait began ≈19:50:43…19:50:46 —
**after** the
killed-entity message (19:49:18) and after the last abort.
- Host conditions when it wedged: 14.4 GiB RAM with ~205 MB free and ~13 GB in
page cache; 16 GiB
VRAM of which ≈12.8 GiB belonged to the model. An unrelated service was in a
crash-restart loop
(restart counter ≈1490) at the same time.
- **No lockdep report exists in this boot**: the deadlock was detected only by
the hung-task
mechanism, not as a lockdep cycle.
- The same model, context and binary had loaded in 8.6–71.5 s on the same
machine under the
previous kernel (noble HWE `7.0.0-38.38~24.04.4`, also 7.0.14-based).
**3. Interpretation, by strength of evidence**
*(a) Confirmed — by code inspection of the installed base, reproducible
with the checks in §11:*
1. **The contended lock is identifiable.** Both sides take the per-`svm_range`
mutex via
`svm_range_lock()`: the reclaim/notifier path inside
`svm_range_cpu_invalidate_pagetables()`
(kfd_svm.c:2669) and the worker path inside `svm_range_validate_and_map()`
(lines 1774, 1812).
The hung stack shows `mutex_lock` inlined at exactly that notifier call
site, and the kernel's
lock-owner annotation independently names the SVM worker as the likely owner.
2. **The wait is on the fence the mapping returned, and that fence is handed
out only on success.**
`svm_range_map_to_gpus()` (plural) calls `svm_range_map_to_gpu()` (singular)
and then
`dma_fence_wait(fence, false)` (line 1551). The singular sets `*fence =
dma_fence_get(vm->last_update)`
only *after* its error paths (`if (fence) *fence =
dma_fence_get(vm->last_update);`, lines
1503-1509 — reached only when `amdgpu_vm_update_range()` and
`amdgpu_vm_update_pdes()` both
returned 0). A failed mapping produces no fence and therefore no wait.
3. **"Killed entity" is the scheduler-entity stopped state.**
`vm->immediate.stopped` /
`vm->delayed.stopped`, checked by `amdgpu_vm_ready()`, are the `stopped`
field of
`struct drm_sched_entity`, documented in the *installed* header
(`/usr/src/linux-headers-7.0.0-38-generic/include/drm/gpu_scheduler.h:205-212`)
as "Marks the
entity as removed from rq and destined for termination. This is set by
calling
`drm_sched_entity_flush()` and by `drm_sched_fini()`".
*(b) Supported inference (consistent with everything above, but not
proven):*
The wedge is the SVM worker waiting forever on a fence while holding the range
lock, which converts
a local GPU problem into a system-wide stall: reclaim cannot enter the
notifier, so allocation
stalls, so the session and SSH die. A GPU scheduler entity belonging to the
just-killed model
server was already dead ~90 s before the wait started.
*(c) Open — I could not determine this, and it is not presented as
fact:*
How a never-signalling fence reached the worker. Two mutually exclusive
candidates:
- **H1** — the VM was already not-ready (its entity stopped) when the worker
called
`svm_range_map_to_gpu()`; the push failed *inside* `amdgpu_vm_update_range()`
(which receives
`&vm->last_update` as its fence output), and a later call then returned a
stale, never-submitted
fence. Whether that error path can leave a dead fence in `vm->last_update`
depends on
`amdgpu_vm_sdma_commit()` / `amdgpu_vm_cpu_commit()`, which I could **not**
inspect (only kernel
headers are installed locally) — so this is unverified.
- **H2** — the fence was legitimately created by a *successful* push and never
signalled because
the queue/scheduler died afterwards. Under H2 the VM was ready at submission
time and the guard
discussed in §5 would **not** have prevented the wait.
Telling H1 from H2 needs runtime tracing or a reproduction with a candidate
patch; neither was
possible here (read-only audit, no inference, no load tests — see §9).
**4. Status of the two commits (corrected in v2)**
`f101c13a8720c73e67f8f9d511fbbeda95bcedb1` — "drm/amdgpu: fix task hang from
failed job submission
during process kill" (Liu01 Tong, 2025-08-11).
> **Integrally incorporated in the installed kernel — not merely functionally
> covered by an
> equivalent change.** Two independent verifications:
1. **Ancestry.** The GitHub comparison API for `f101c13a…v7.0` returns `status:
ahead`,
`behind_by: 0` and `merge_base_commit:
f101c13a8720c73e67f8f9d511fbbeda95bcedb1`. The commit is
an ancestor of the `v7.0` release tag, hence part of every 7.0.x stable
release, including the
installed 7.0.14. (The raw API responses are kept in the reporter's private
archive; the same
call can be repeated by anyone — see §11(a), item 6.)
2. **Code.** Both hunks are present in the installed base:
`amdgpu_vm.c:amdgpu_vm_ready()` contains
`ret &= !vm->immediate.stopped;` and `ret &= !vm->delayed.stopped;` and is
byte-identical to
the 7.1.7 version (the `bool empty;` the patch removed is gone), and
`amdgpu_cs.c` has
`if (!amdgpu_vm_ready(vm)) return -EINVAL;` immediately before
`amdgpu_vm_clear_freed()`
(line ~1119).
The installed changelog lists no entry for it, which is consistent with it
arriving with the 7.0
base rather than as a listed stable backport. **Therefore §8 does not request
it.**
`d0f5711fa14a09c010537375cf34893cd33bc2ee` — "drm/amdkfd: check if vm ready in
svm map and unmap to
gpu" (YuanShang Mao; Reviewed-by: Philip Yang; Signed-off-by: Alex Deucher).
> **Absent** from the installed kernel and from the entire
resolute/noble 7.0 line:
```
amdgpu_vm_ready() call sites in kfd_svm.c
upstream v7.0.14 (installed base) ......... 0 MISSING
upstream v6.18.43 ......................... 0
upstream v7.1.3 / v7.1.6 / v7.1.7 ......... 2 PRESENT
upstream v7.2.5 ........................... 2
Ubuntu linux 7.0.0-38.38 (installed) 0
Ubuntu linux-hwe-7.0 7.0.0-38.38~24.04.4 0
Ubuntu linux 7.0.0-39.39 (-proposed) 0
```
Ancestry: `status: diverged`, `merge_base_commit:
591cd656a1bf5ea94a222af5ef2ee76df029c1d2` —
**not** an ancestor of `v7.0`; present from v7.1.3 onward. The installed build
declares "Upstream
stable to v6.18.41, v6.18.42, v6.18.43, v7.1.6, v7.1.7" in its changelog, i.e.
it already reaches
7.1.6/7.1.7, which **do** contain the guard — so the omission looks like patch
selection rather
than unavailability.
**5. What the missing change does at the call site involved here**
In 7.0.14, `svm_range_map_to_gpus()` (plural) loops over GPUs, calling
`svm_range_map_to_gpu()`
(singular) for each; when that returns 0 it waits on the returned fence
(`dma_fence_wait(fence,
false)`, line 1551) and then flushes the TLB. `d0f5711f` inserts at the top of
the singular
`svm_range_map_to_gpu()` (and of `svm_range_unmap_from_gpu()`):
```c
if (!amdgpu_vm_ready(vm)) {
pr_debug("VM not ready, canceling map\n");
return -EINVAL;
}
```
With the guard, a not-ready VM never reaches `amdgpu_vm_update_range()`; the
singular returns
`-EINVAL`, the plural's `if (r) break;` fires, and the `dma_fence_wait()` is
not reached. The guard
therefore sits on the code path our worker occupied, and the author's stated
rationale — "Don't map
or unmap svm range to gpu if vm is not ready for updates. Why: DRM entity may
already be killed
when the svm worker try to update gpu vm." — describes the situation our log
documents.
**6. Why the fix must not be presented as guaranteed**
1. **The guard fires only if `amdgpu_vm_ready(vm)` is false**, and which of its
four conditions
(`vm->evicting`, non-empty `vm->evicted`, `immediate.stopped`,
`delayed.stopped`) held at that
instant is unknown: nothing in the log pins the VM state at the moment of
the wait. The
killed-entity message proves *an* entity was dead, not that *this* VM's
entity was.
2. **The ordering argues for caution.** The wait began ≈19:50:43, i.e. *after*
the killed-entity
message, and per §3(a)(2) a mapping whose push fails produces no fence and
hence no wait. So
this is not a plain "push failed for a stopped entity and we waited anyway";
the intervening
state change is not in the log, and H2 of §3(c) remains possible.
3. **The commit is hardening for a class of failure, not a verified cure for
this occurrence.**
Nothing in it guarantees that a fence already handed out by a successful
update will signal.
4. **It was not tested.** No reproduction with or without the patch was
performed (read-only audit;
loading models was out of bounds), so its effect here is unmeasured.
What the evidence *does* justify: this is the only upstream AMDGPU/KFD change
identified that
addresses "the SVM worker updates the GPU VM of an already-dead scheduler
entity", which is the
state our log documents. Treat a backport as **risk reduction for that class**,
not as a proven fix
for this incident.
**7. Companion evidence: this is not Launchpad #2161667**
That report (package `linux-hwe-7.0`, status *New*) shows the same two
`workqueue: … hogged CPU`
lines, but its reporter found it no longer reproducible on `7.0.0-34-generic`
and attributed the fix
to `drm/amdgpu: drop retry loop in amdgpu_hmm_range_get_pages`, which the
installed kernel already
contains (since `7.0.0-31.31`, per its changelog). Independent reports with our
stack —
`peonist-ai/halogen-flash-server` #34 (2026-09-12, kernel `7.0.0-31-generic
#31~24.04.1-Ubuntu`,
`SIGKILL` ineffective, container stuck until reboot) and `ROCm/ROCm` #6508
(open; kernel 7.0 hangs
where 6.17 worked, same workqueue messages) — indicate a distinct, still-open
defect. Keep the
tracker tasks separate so this report is not closed as a duplicate.
**8. Requested action**
1. Consider backporting `d0f5711fa14a09c010537375cf34893cd33bc2ee` to the 26.04
kernel, and
evaluating the same for the noble `linux-hwe-7.0` line. Its prerequisite
`f101c13a8720c7` is
**already present** in both (verified in §4), so no companion commit is
required; the delta is
the two `amdgpu_vm_ready()` call-site guards in `kfd_svm.c` (11 insertions,
one file).
2. Treat the effect as **hardening of the SVM worker path**, not as a confirmed
fix for the
occurrence reported here, and consider whether the remaining case — a fence
from a *successful*
update that never signals after its queue dies — needs separate handling.
3. To test the hypothesis before committing to it: a temporary probe logging
`amdgpu_vm_ready()`'s
result at the top of `svm_range_map_to_gpu()` (or an ftrace probe on that
function plus
`drm_sched_entity_push_job`) would settle §3(c) on a re-run. That
measurement is what this report
is missing.
**9. Limitations of this investigation (declared)**
- Read-only audit: no packages installed, no kernel/firmware update, no service
restarted, no model
loaded, no load test, no changes to GRUB/systemd/ROCm/Ollama. Nothing in the
evidence was produced
by running the workload.
- The incident was not reproduced. Conclusions rest on the journal of the
affected boot, the sources
of the exact installed base (v7.0.14), the official Canonical source deltas,
and the upstream tags
listed above.
- `amdgpu_vm_sdma.c` / `amdgpu_vm_cpu.c` were not inspected (only kernel
headers are installed
locally), which is why §3(c) stays open.
- The client machine (`<client>`) was not reachable from here; its 300 s header
timeout and its
automatic retries are known from the server-side log only.
- Ancestry results come from the GitHub comparison API (metadata) corroborated
by code inspection of
published tags; the Ubuntu source package's git history was not inspected.
- The `drm_sched_entity_push_job` message does not name the entity it rejected,
so the entity's
identity is inferred, not read.
- The recovery method (SysRq) is the operator's account: no trace of it
survives in the preserved
journal, because the stalled system could not flush it.
**10. Attachment policy**
Attached, and only this: a hand-curated kernel-log excerpt of the affected boot
covering 19:44–19:55
— the companion file `ANEXO-TRECHOS-KERNEL-HIGIENIZADOS.txt`. It contains the
two `workqueue`
warnings, the killed-entity message, the **complete** call traces (from
`<TASK>` to `</TASK>`) of
the worker holding the lock and of `kswapd0` and the model server, and the
header/anchor lines of
the remaining waiting tasks. Removed: host name (→ `<host>`), all IP and MAC
addresses, tailnet and
bridge names, disk UUIDs and serial numbers, machine-id, SSH key fingerprints,
login e-mails, and
the names of private self-hosted services; register dumps were dropped (address
values, no bearing
on the traces). The attachment states its own omissions, and the removals were
verified
programmatically (zero matches for those patterns), so the excerpt can be
compared against the
reporter's private archive.
**11. Reproducing the checks**
*(a) Checks you can run against your own kernel tree.* The paths below are
relative to the root of a
Linux source tree of the corresponding version (upstream `v7.0.14` / `v7.1.6` /
`v7.1.7`, or the
Ubuntu tree carrying the same base). Line numbers cited anywhere in this report
refer to **upstream
v7.0.14**. Adjust the top-level directory as needed. Only the first and last
commands touch the
installed system.
```bash
# base of the installed kernel, taken from the installed headers
sed -n '1,4p' /usr/src/linux-headers-7.0.0-38-generic/Makefile # VERSION 7
PATCHLEVEL 0 SUBLEVEL 14
# 1) d0f5711f is ABSENT: no amdgpu_vm_ready() call site in kfd_svm.c
grep -c amdgpu_vm_ready drivers/gpu/drm/amd/amdkfd/kfd_svm.c # 0 in
v7.0.14, 2 in v7.1.6 and v7.1.7
# 2) f101c13a is PRESENT: the two "stopped" checks inside amdgpu_vm_ready()
awk '/^bool amdgpu_vm_ready/,/^}/' drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c |
grep stopped # 2 checks
grep -n -B4 amdgpu_vm_clear_freed drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
# guard at ~line 1119
# 3) the fence is handed out only on success (§3(a)(2))
sed -n '1495,1508p' drivers/gpu/drm/amd/amdkfd/kfd_svm.c
# 4) the lock both sides take (§3(a)(1))
grep -n 'svm_range_lock(prange)' drivers/gpu/drm/amd/amdkfd/kfd_svm.c #
1774, 1812, 2669
# 5) drm_sched_entity::stopped semantics — from the INSTALLED headers
sed -n '205,212p'
/usr/src/linux-headers-7.0.0-38-generic/include/drm/gpu_scheduler.h
# 6) ancestry of the two commits (metadata API; expect ahead/behind_by:0, and
diverged)
curl -s
'https://api.github.com/repos/torvalds/linux/compare/f101c13a8720c73e67f8f9d511fbbeda95bcedb1...v7.0'
| grep -E '"(status|behind_by)"'
curl -s
'https://api.github.com/repos/torvalds/linux/compare/d0f5711fa14a09c010537375cf34893cd33bc2ee...v7.0'
| grep -E '"(status|behind_by)"'
```
*(b) Where the copies quoted in this report come from.* This report also refers
to files kept in the
reporter's private archive (for example `kernel/fontes-upstream/…`,
`kernel/diffs-canonical/…` and
the `_7014`-suffixed source copies). **Those paths are not reachable by
maintainers** and are
mentioned only for the reporter's own traceability; every quoted file has a
public origin that
anyone can fetch:
- `kfd_svm.c`, `amdgpu_vm.c`, `amdgpu_vm.h`, `amdgpu_cs.c` per tag, from the
upstream mirror:
`https://raw.githubusercontent.com/gregkh/linux/<tag>/drivers/gpu/drm/amd/amdkfd/kfd_svm.c`
(and `.../drivers/gpu/drm/amd/amdgpu/{amdgpu_vm.c,amdgpu_vm.h,amdgpu_cs.c}`),
with
`<tag>` ∈ {v7.0.14, v6.18.43, v7.1.3, v7.1.6, v7.1.7, v7.2.5}.
- Ubuntu source deltas of the releases compared above — source package `linux`
at `7.0.0-38.38`
(installed) and `7.0.0-39.39` (proposed), and `linux-hwe-7.0` at
`7.0.0-38.38~24.04.4`: the
per-version pages under
`https://launchpad.net/ubuntu/+source/<package>/<version>` link to the
`.diff.gz` and `.dsc`; `apt-get source linux=7.0.0-38.38` (with `deb-src`
enabled) yields the same
files.
- Commit patches: `https://github.com/torvalds/linux/commit/<sha>.patch`.
- The kernel headers quoted in §3(a)(3) and §11(a)(5) are part of the installed
`linux-headers-7.0.0-38-generic`.
** Affects: linux (Ubuntu)
Importance: Undecided
Status: New
** Attachment added: "Sanitized kernel log excerpt — AMDGPU/KFD SVM deadlock,
2026-10-15"
https://bugs.launchpad.net/bugs/2171116/+attachment/6009850/+files/ANEXO-TRECHOS-KERNEL-HIGIENIZADOS.txt
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2171116
Title:
[7.0.0-38] amdgpu/KFD SVM worker hangs on GPU fence, blocking kswapd
and memory allocation
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2171116/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
