Public bug reported:
We operate GPU nodes running Ubuntu 26.04 with the 7.0.0-38-generic kernel (7.0.0-38.38, resolute-proposed) hosting Kubernetes (RKE2 v1.36.4, containerd 2.3.4-k3s1.36). Pod teardown on these nodes intermittently leaves tasks in permanent uninterruptible sleep in the fsnotify destroy path. SIGKILL does not clear them; the only remedy is a node reboot. We have captured this on three occasions (two variants of the same teardown window): ### 1. runc shim, close(2) on inotify fd (2026-10-05) Pod: apply-system-agent-upgrader (stock containerd-shim-runc-v2, no Kata involved). Container exited cleanly (exit 0). During sandbox teardown, with the pod's overlay rootfs and projected volumes being unmounted, one shim worker thread entered permanent D-state: ``` $ cat /proc/3716051/task/3716060/syscall 3 0x1a 0x0 0x0 0x0 0x0 0x0 0xc643525ab58 0x40e18e # close(2), fd 26 $ sudo cat /proc/3716051/task/3716060/stack __flush_work+0x270/0x3c0 flush_delayed_work+0x35/0x60 fsnotify_wait_marks_destroyed+0x15/0x20 fsnotify_destroy_group+0x53/0x120 inotify_release+0x1b/0x80 __fput+0xed/0x2d0 fput_close_sync+0x40/0xc0 __x64_sys_close+0x3e/0x90 x64_sys_call+0x1718/0x2390 do_syscall_64+0x105/0x5a0 entry_SYSCALL_64_after_hwframe+0x76/0x7e ``` The main thread had already exited (zombie); SIGTERM and SIGKILL were both delivered — the process cannot be reaped because the D-state thread never returns from the kernel. The pod stays Terminating indefinitely; containerd's StopPodSandbox RPC times out with DeadlineExceeded. Node otherwise healthy (this is a worker; etcd/apiserver unaffected). ### 2. kata runtime-rs shim, same stack, same kernel (2026-09-23) Graceful pod delete of a ~800GiB GPU-TDX guest. Two of five shim threads in D-state, both immediately after projected-volume cleanup: ``` 2 of 5 threads: tokio-rt-worker, state D, wchan __flush_work same stack: fsnotify_wait_marks_destroyed → fsnotify_destroy_group → inotify_release ``` Full narrative + journal evidence: https://github.com/kata-containers/kata-containers/issues/13641#issuecomment-5791433773 ### 3. Kernel-side variant — kworkers (2026-09-02) Two kworkers wedged for 6+ hours, blocking node-wide pod teardown (runc and kata pods alike), after an unscheduled reboot: ``` kworker/u961:2 D __synchronize_srcu synchronize_srcu fsnotify_mark_destroy_workfn process_one_work kworker/u961:6 D __synchronize_srcu synchronize_srcu fsnotify_connector_destroy_workfn process_one_work ``` 42 pod-dir mounts still mounted under /var/lib/kubelet/pods; kubelet logged zero errors the whole time. Full narrative: https://github.com/kata-containers/kata-containers/issues/13641 (comment beginning "Kernel-side variant observed (production, 2026-09-02)") ## Reproduction Not deterministically reproducible on demand. The window is pod teardown with inotify watches on projected volumes (ConfigMap/Secret/serviceaccount) plus concurrent unmount of those volumes — i.e. every Kubernetes node running resident daemons that watch files (containerd shims, kubelet plugins). We hit it roughly monthly across a fleet of ~20 such nodes since kernel 7.0.0-27, reproduced on -27, -31, -38. Suspect interaction: inotify_release() -> fsnotify_destroy_group() -> fsnotify_wait_marks_destroyed() waits unboundedly on flush of the delayed destroy work while the unmount path holds the mark/connector state it needs. The wait is uninterruptible (TASK_UNINTERRUPTIBLE), so the closing task can never be killed, and work-flush vs unmount deadlock leaves the state pinned forever. ## Impact - Unkillable tasks: SIGKILL ineffective; only a reboot clears the state. - Pods stuck Terminating for hours to days; leaked sandboxes holding resources (in the Kata GPU case, an 801GiB VM holding all 12 VFIO devices). - Silent: no kernel messages, no containerd errors — the wedge only shows in /proc/<pid>/task/*/stack and wchan. ## Environment - Kernel: 7.0.0-38-generic (7.0.0-38.38), also seen on 7.0.0-27/-31 - OS: Ubuntu 26.04.1 LTS (resolute) - Kubernetes: RKE2 v1.36.4+rke2r1 (kubelet, containerd 2.3.4-k3s1.36) - Affected processes: containerd-shim-runc-v2 (stock), kata runtime-rs shim, kworkers (fsnotify destroy work) - Hardware: Dell XE9680 8x H200 (Intel TDX), HGX Blackwell B200 nodes ## Tags (LP) linux, resolute, kernel-daemon-hang, fsnotify, inotify --- # LKML / linux-fsdevel draft (send after LP bug number exists) Subject: fsnotify: uninterruptible hang in fsnotify_wait_marks_destroyed during inotify fd close vs concurrent unmount To: [email protected] Cc: Amir Goldstein <[email protected]>, Jan Kara <[email protected]> Team, We have hit a permanent, SIGKILL-proof uninterruptible sleep in the fsnotify destroy path on production Kubernetes nodes (kernel 7.0.0-38, Ubuntu 26.04; also reproduced on -27 and -31). Tracked in Ubuntu as LP: #XXXXXXX [fill after filing]. The window is inotify fd close racing unmount of watched volumes (Kubernetes pod teardown: shims hold inotify watches on ConfigMap/Secret/projected volumes; kubelet unmounts them concurrently). Userspace side (containerd-shim-runc-v2 worker thread): close(2) fd 26 inotify_release -> fsnotify_destroy_group -> fsnotify_wait_marks_destroyed -> flush_delayed_work -> __flush_work [D-state, permanent] Kernel-side (kworkers, second capture): fsnotify_mark_destroy_workfn / fsnotify_connector_destroy_workfn -> synchronize_srcu [D-state, 6+ hours] Both variants leave pods Terminating forever; SIGKILL cannot clear the closing task; only a reboot recovers the node. No kernel log lines at all. Full stacks, syscall args, and environment details in the LP bug [link] and in https://github.com/kata-containers/kata-containers/issues/13641 (comments of 2026-09-02 and 2026-09-23). Is this a known interaction? Any diagnostics we can enable (lockdep, eventhed tracing of the group/mark destroy path) to pin which side holds what when the wait blocks? Happy to run whatever is needed on the affected nodes. -- Pierre A Manseau <[email protected]> --- # kata-containers #13641 comment draft (post after LP bug number exists) Third venue: a short follow-up comment on https://github.com/kata-containers/kata-containers/issues/13641 adding today's stock-runc capture — the cleanest evidence yet of the kernel-class nature (full syscall args, SIGKILL-proof confirmation). Keep it one screen; reference the LP bug for the full package. Body: Third capture, this one on stock containerd-shim-runc-v2 (no Kata code in the process at all) — kernel 7.0.0-38-generic (7.0.0-38.38, Ubuntu 26.04, containerd 2.3.4-k3s1.36, RKE2 v1.36.4). Pod teardown after the container exited 0; one shim worker thread never returned from close(2): ``` $ cat /proc/3716051/task/3716060/syscall 3 0x1a 0x0 0x0 0x0 0x0 0x0 0xc643525ab58 0x40e18e # close(2), fd 26 $ sudo cat /proc/3716051/task/3716060/stack __flush_work+0x270/0x3c0 flush_delayed_work+0x35/0x60 fsnotify_wait_marks_destroyed+0x15/0x20 fsnotify_destroy_group+0x53/0x120 inotify_release+0x1b/0x80 __fput+0xed/0x2d0 fput_close_sync+0x40/0xc0 __x64_sys_close+0x3e/0x90 ``` SIGTERM and SIGKILL were both delivered: the main thread is a zombie, this worker stays in D, and the sandbox/pod stay wedged (StopPodSandbox times out with DeadlineExceeded). Same fsnotify_wait_marks_destroyed wait as the two captures above, now with the exact syscall args pinned. We are filing this with the Ubuntu kernel team as a kernel bug (the wait is unbounded and uninterruptible on the unmount path); will link the LP number here once filed. ** Affects: linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2169588 Title: close(2) on inotify fd can hang unkillably in fsnotify_wait_marks_destroyed (fsnotify destroy path vs concurrent unmount) — kernel 7.0.0-38 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2169588/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
