Hey Paul, Lai, Josh, and the RCU list and Jan/FS list - Reaching out about a tricky hung task issue that I'm running into. I've got a virtualized Linux guest on top of a KVM based platform, running a 5.10.y based kernel. The issue we're running into is a hung task that *only* happens on shutdown/reboot of this particular VM once every 20-50 times.
The signature of the hung task is always similar to the output below, where we appear to hang on the call to synchronize_srcu(&fsnotify_mark_srcu) in fsnotify_connector_destroy_workfn / fsnotify_mark_destroy_workfn, where two kernel threads are both calling synchronize_srcu, then scheduling out in wait_for_completion, and completely going out to lunch for over 4 minutes. This then triggers the hung task timeout and things blow up. We are running audit=1 for this system and are using an el8 based userspace. I've flipped through the fs/notify code base for both 5.10 as well as upstream mainline to see if something jumped off the page, and I haven't yet spotted any particular suspect code from the caller side. This hang appears to come up at the very end of the shutdown/reboot process, seemingly after the system starts to unwind through initrd. What I'm working on now is adding some instrumentation to the dracut shutdown initrd scripts to see if I can how far we get down that path before the system fails to make forward progress, which may give some hints. TBD on that. I've also enabled lockdep with CONFIG_PROVE_RCU and a plethora of DEBUG options [2], and didn't get anything interesting. To be clear, we haven't seen lockdep spit out any complaints as of yet. Reaching out to see if this sounds familar to anyone on the list, or if there are any particular areas of the RCU code base that might be suspect for this kind of issue. I'm happy to provide more information, as frankly, I'm quite stumped at the moment. Thanks all, Jon [1] panic trace Normal shutdown process, then hangs on the following: ... dracut Warning: Killing all remaining processes ... INFO: task kworker/u20:7:1200701 blocked for more than 241 seconds. Tainted: G O 5.10.205-2.el8.x86_64 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:kworker/u20:7 state:D stack: 0 pid:1200701 ppid: 2 flags:0x00004080 Workqueue: events_unbound fsnotify_connector_destroy_workfn Call Trace: __schedule+0x267/0x790 schedule+0x3c/0xb0 schedule_timeout+0x219/0x2b0 wait_for_completion+0x9e/0x100 __synchronize_srcu.part.24+0x83/0xb0 ? __bpf_trace_rcu_utilization+0x10/0x10 ? synchronize_srcu+0x5d/0xf0 fsnotify_connector_destroy_workfn+0x46/0x80 process_one_work+0x1fc/0x390 worker_thread+0x2d/0x3e0 ? process_one_work+0x390/0x390 kthread+0x114/0x130 ? kthread_park+0x80/0x80 ret_from_fork+0x1f/0x30 INFO: task kworker/u20:8:1287360 blocked for more than 241 seconds. Tainted: G O 5.10.205-2.el8.x86_64 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:kworker/u20:8 state:D stack: 0 pid:1287360 ppid: 2 flags:0x00004080 Workqueue: events_unbound fsnotify_mark_destroy_workfn Call Trace: __schedule+0x267/0x790 schedule+0x3c/0xb0 schedule_timeout+0x219/0x2b0 ? add_timer+0x14a/0x200 wait_for_completion+0x9e/0x100 __synchronize_srcu.part.24+0x83/0xb0 ? __bpf_trace_rcu_utilization+0x10/0x10 fsnotify_mark_destroy_workfn+0x77/0xe0 process_one_work+0x1fc/0x390 ? process_one_work+0x390/0x390 worker_thread+0x2d/0x3e0 ? process_one_work+0x390/0x390 kthread+0x114/0x130 ? kthread_park+0x80/0x80 ret_from_fork+0x1f/0x30 Kernel panic - not syncing: hung_task: blocked tasks CPU: 1 PID: 64 Comm: khungtaskd Kdump: loaded Tainted: G O 5.10.205-2.el8.x86_64 #1 Hardware name: Red Hat KVM, BIOS 20230302.1.2662.el8 04/01/2014 Call Trace: dump_stack+0x6d/0x8c panic+0x114/0x2ea watchdog.cold.8+0xb5/0xb5 ? hungtask_pm_notify+0x50/0x50 kthread+0x114/0x130 ? kthread_park+0x80/0x80 ret_from_fork+0x1f/0x30 [2] additional debugging config knobs turned up. CONFIG_PROVE_LOCKING=y CONFIG_LOCK_STAT=y CONFIG_DEBUG_RT_MUTEXES=y CONFIG_DEBUG_SPINLOCK=y CONFIG_DEBUG_MUTEXES=y CONFIG_DEBUG_WW_MUTEX_SLOWPATH=y CONFIG_DEBUG_RWSEMS=y CONFIG_DEBUG_LOCK_ALLOC=y CONFIG_LOCKDEP=y CONFIG_LOCKDEP_BITS=15 CONFIG_LOCKDEP_CHAINS_BITS=16 CONFIG_LOCKDEP_STACK_TRACE_BITS=19 CONFIG_LOCKDEP_STACK_TRACE_HASH_BITS=14 CONFIG_LOCKDEP_CIRCULAR_QUEUE_BITS=12 CONFIG_DEBUG_SHIRQ=y CONFIG_WQ_WATCHDOG=y CONFIG_DEBUG_ATOMIC_SLEEP=y CONFIG_DEBUG_LIST=y CONFIG_DEBUG_PLIST=y CONFIG_DEBUG_SG=y CONFIG_DEBUG_NOTIFIERS=y CONFIG_BUG_ON_DATA_CORRUPTION=y