Seeing this bug on Ubuntu 20.04 and Ubuntu 22.04 as well, with both normal and HWE kernels.
To add some more information, this bug seems to randomly appear during the initial RAID 6 creation process as well, where the array is mounted but completely empty and not accessed - so it's likely to originate within the mdadm resync process itself, unrelated to other system I/O operations. Around 1 in 15 arrays will "freeze" during the initial resync process in my experience, so it is not that uncommon unfortunately. The symptoms are always the same - at some point during resync, the speeds will radically drop to single MB/s levels and continue degrading over time until "echo active > /sys/block/mdX/md/array_state" is issued. A minute or two after running that, the speeds ramp back up to normal. This bug seems unrelated to hardware configuration, as I've seen it happen across multiple systems with different CPU vendors, HBA models and with different HDD sizes and vendors. Systems which were previously stable under Ubuntu 18.04 started exhibiting freezes after upgrading to 20.04 as well. It would also seem disabling mdcheck_start and mdcheck_continue is not necessarily the magical bullet in fixing this, it certainly doesn't seem to help with freezing during the initial resync. I have also seen instances of mdadm scheduled resyncs freezing when triggered using the old cronjob method, with both systemd services and timers masked off. Dmesg from a freshly installed system where the initial resync "froze" approximately 15 hours after the array was created: mdadm --create /dev/md1 --level=6 --raid-devices=6 /dev/sd[cdefgh] mkfs.ext4 /dev/md1 mount -o errors=remount-ro /dev/md1 /srv [Jul 4 00:18] md/raid:md1: not clean -- starting background reconstruction [ +0.000062] md/raid:md1: device sdh operational as raid disk 5 [ +0.000002] md/raid:md1: device sdg operational as raid disk 4 [ +0.000001] md/raid:md1: device sdf operational as raid disk 3 [ +0.000001] md/raid:md1: device sde operational as raid disk 2 [ +0.000001] md/raid:md1: device sdd operational as raid disk 1 [ +0.000001] md/raid:md1: device sdc operational as raid disk 0 [ +0.002104] md/raid:md1: raid level 6 active with 6 out of 6 devices, algorithm 2 [ +0.048241] md1: detected capacity change from 0 to 72000290684928 [ +0.000066] md: resync of RAID array md1 [Jul 4 00:32] EXT4-fs (md1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [Jul 4 02:39] perf: interrupt took too long (2522 > 2500), lowering kernel.perf_event_max_sample_rate to 79250 [Jul 4 04:36] perf: interrupt took too long (3155 > 3152), lowering kernel.perf_event_max_sample_rate to 63250 [Jul 4 15:22] INFO: task md1_raid6:5688 blocked for more than 120 seconds. [ +0.000059] Not tainted 5.4.0-153-generic #170-Ubuntu [ +0.000032] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000045] md1_raid6 D 0 5688 2 0x80004000 [ +0.000004] Call Trace: [ +0.000011] __schedule+0x2e3/0x740 [ +0.000005] schedule+0x42/0xb0 [ +0.000011] raid5d+0x3e6/0x5f0 [raid456] [ +0.000005] ? try_to_del_timer_sync+0x54/0x80 [ +0.000005] ? schedule_timeout+0x92/0x160 [ +0.000004] ? __wake_up_pollfree+0x40/0x40 [ +0.000004] md_thread+0x97/0x160 [ +0.000003] ? __wake_up_pollfree+0x40/0x40 [ +0.000004] kthread+0x104/0x140 [ +0.000003] ? md_start_sync+0x60/0x60 [ +0.000003] ? kthread_park+0x90/0x90 [ +0.000002] ret_from_fork+0x1f/0x40 [ +0.000005] INFO: task md1_resync:5724 blocked for more than 120 seconds. [ +0.000039] Not tainted 5.4.0-153-generic #170-Ubuntu [ +0.000031] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000044] md1_resync D 0 5724 2 0x80004000 [ +0.000002] Call Trace: [ +0.000005] __schedule+0x2e3/0x740 [ +0.000004] schedule+0x42/0xb0 [ +0.000007] raid5_get_active_stripe+0x459/0x610 [raid456] [ +0.000003] ? __wake_up_pollfree+0x40/0x40 [ +0.000007] raid5_sync_request+0x38b/0x3b0 [raid456] [ +0.000004] ? cpumask_next+0x1b/0x20 [ +0.000003] ? is_mddev_idle+0xc1/0x11e [ +0.000004] md_do_sync.cold+0x3ef/0x992 [ +0.000005] ? sched_clock+0x9/0x10 [ +0.000003] ? __wake_up_pollfree+0x40/0x40 [ +0.000004] md_thread+0x97/0x160 [ +0.000004] kthread+0x104/0x140 [ +0.000002] ? md_start_sync+0x60/0x60 [ +0.000003] ? kthread_park+0x90/0x90 [ +0.000003] ret_from_fork+0x1f/0x40 [ +0.000003] INFO: task jbd2/md1-8:6099 blocked for more than 120 seconds. [ +0.000038] Not tainted 5.4.0-153-generic #170-Ubuntu [ +0.000031] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000043] jbd2/md1-8 D 0 6099 2 0x80004000 [ +0.000001] Call Trace: [ +0.000004] __schedule+0x2e3/0x740 [ +0.000003] ? __wake_up_common_lock+0x8a/0xc0 [ +0.000004] schedule+0x42/0xb0 [ +0.000005] jbd2_journal_commit_transaction+0x24e/0x18b0 [ +0.000004] ? dequeue_entity+0x118/0x460 [ +0.000002] ? record_times+0x1b/0x90 [ +0.000004] ? __wake_up_pollfree+0x40/0x40 [ +0.000005] ? try_to_del_timer_sync+0x54/0x80 [ +0.000003] kjournald2+0xb6/0x280 [ +0.000004] ? __wake_up_pollfree+0x40/0x40 [ +0.000003] kthread+0x104/0x140 [ +0.000003] ? commit_timeout+0x20/0x20 [ +0.000003] ? kthread_park+0x90/0x90 [ +0.000002] ret_from_fork+0x1f/0x40 [ +0.000004] INFO: task ext4lazyinit:6101 blocked for more than 120 seconds. [ +0.000038] Not tainted 5.4.0-153-generic #170-Ubuntu [ +0.000031] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000043] ext4lazyinit D 0 6101 2 0x80004000 [ +0.000002] Call Trace: [ +0.000004] __schedule+0x2e3/0x740 [ +0.000003] schedule+0x42/0xb0 [ +0.000004] md_write_start+0x151/0x230 [ +0.000003] ? __wake_up_pollfree+0x40/0x40 [ +0.000007] raid5_make_request+0x88/0x870 [raid456] [ +0.000002] ? newidle_balance+0x231/0x3e0 [ +0.000003] ? __wake_up_pollfree+0x40/0x40 [ +0.000003] ? __switch_to_asm+0x42/0x80 [ +0.000002] ? __switch_to_asm+0x5a/0x80 [ +0.000003] md_handle_request+0x123/0x1a0 [ +0.000004] md_make_request+0x92/0x1b0 [ +0.000003] generic_make_request+0xcf/0x320 [ +0.000004] ? bio_alloc_bioset+0x145/0x200 [ +0.000003] submit_bio+0x48/0x1d0 [ +0.000003] ? bio_add_page+0x6a/0x90 [ +0.000003] submit_bio_wait+0x58/0x90 [ +0.000004] blkdev_issue_zeroout+0x13f/0x220 [ +0.000006] ext4_init_inode_table+0x203/0x380 [ +0.000004] ext4_lazyinit_thread.part.0+0x2be/0x3c0 [ +0.000003] ext4_lazyinit_thread+0x20/0x30 [ +0.000003] kthread+0x104/0x140 [ +0.000003] ? ext4_lazyinit_thread.part.0+0x3c0/0x3c0 [ +0.000002] ? kthread_park+0x90/0x90 [ +0.000003] ret_from_fork+0x1f/0x40 [Jul 4 15:24] INFO: task md1_raid6:5688 blocked for more than 241 seconds. [ +0.000046] Not tainted 5.4.0-153-generic #170-Ubuntu [ +0.000029] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000040] md1_raid6 D 0 5688 2 0x80004000 [ +0.000004] Call Trace: [ +0.000009] __schedule+0x2e3/0x740 [ +0.000004] schedule+0x42/0xb0 [ +0.000010] raid5d+0x3e6/0x5f0 [raid456] [ +0.000005] ? try_to_del_timer_sync+0x54/0x80 [ +0.000004] ? schedule_timeout+0x92/0x160 [ +0.000004] ? __wake_up_pollfree+0x40/0x40 [ +0.000003] md_thread+0x97/0x160 [ +0.000003] ? __wake_up_pollfree+0x40/0x40 [ +0.000004] kthread+0x104/0x140 [ +0.000003] ? md_start_sync+0x60/0x60 [ +0.000002] ? kthread_park+0x90/0x90 [ +0.000002] ret_from_fork+0x1f/0x40 [ +0.000004] INFO: task md1_resync:5724 blocked for more than 241 seconds. [ +0.000035] Not tainted 5.4.0-153-generic #170-Ubuntu [ +0.000029] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000039] md1_resync D 0 5724 2 0x80004000 [ +0.000002] Call Trace: [ +0.000004] __schedule+0x2e3/0x740 [ +0.000003] schedule+0x42/0xb0 [ +0.000007] raid5_get_active_stripe+0x459/0x610 [raid456] [ +0.000003] ? __wake_up_pollfree+0x40/0x40 [ +0.000007] raid5_sync_request+0x38b/0x3b0 [raid456] [ +0.000003] ? cpumask_next+0x1b/0x20 [ +0.000004] ? is_mddev_idle+0xc1/0x11e [ +0.000003] md_do_sync.cold+0x3ef/0x992 [ +0.000004] ? sched_clock+0x9/0x10 [ +0.000003] ? __wake_up_pollfree+0x40/0x40 [ +0.000004] md_thread+0x97/0x160 [ +0.000003] kthread+0x104/0x140 [ +0.000002] ? md_start_sync+0x60/0x60 [ +0.000003] ? kthread_park+0x90/0x90 [ +0.000002] ret_from_fork+0x1f/0x40 [ +0.000004] INFO: task jbd2/md1-8:6099 blocked for more than 241 seconds. [ +0.000035] Not tainted 5.4.0-153-generic #170-Ubuntu [ +0.000028] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000038] jbd2/md1-8 D 0 6099 2 0x80004000 [ +0.000002] Call Trace: [ +0.000003] __schedule+0x2e3/0x740 [ +0.000003] ? __wake_up_common_lock+0x8a/0xc0 [ +0.000003] schedule+0x42/0xb0 [ +0.000005] jbd2_journal_commit_transaction+0x24e/0x18b0 [ +0.000003] ? dequeue_entity+0x118/0x460 [ +0.000003] ? record_times+0x1b/0x90 [ +0.000003] ? __wake_up_pollfree+0x40/0x40 [ +0.000004] ? try_to_del_timer_sync+0x54/0x80 [ +0.000004] kjournald2+0xb6/0x280 [ +0.000003] ? __wake_up_pollfree+0x40/0x40 [ +0.000003] kthread+0x104/0x140 [ +0.000003] ? commit_timeout+0x20/0x20 [ +0.000002] ? kthread_park+0x90/0x90 [ +0.000002] ret_from_fork+0x1f/0x40 [ +0.000004] INFO: task ext4lazyinit:6101 blocked for more than 241 seconds. [ +0.000035] Not tainted 5.4.0-153-generic #170-Ubuntu [ +0.000028] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000039] ext4lazyinit D 0 6101 2 0x80004000 [ +0.000001] Call Trace: [ +0.000003] __schedule+0x2e3/0x740 [ +0.000004] schedule+0x42/0xb0 [ +0.000003] md_write_start+0x151/0x230 [ +0.000003] ? __wake_up_pollfree+0x40/0x40 [ +0.000006] raid5_make_request+0x88/0x870 [raid456] [ +0.000002] ? newidle_balance+0x231/0x3e0 [ +0.000004] ? __wake_up_pollfree+0x40/0x40 [ +0.000002] ? __switch_to_asm+0x42/0x80 [ +0.000002] ? __switch_to_asm+0x5a/0x80 [ +0.000003] md_handle_request+0x123/0x1a0 [ +0.000003] md_make_request+0x92/0x1b0 [ +0.000004] generic_make_request+0xcf/0x320 [ +0.000003] ? bio_alloc_bioset+0x145/0x200 [ +0.000003] submit_bio+0x48/0x1d0 [ +0.000002] ? bio_add_page+0x6a/0x90 [ +0.000003] submit_bio_wait+0x58/0x90 [ +0.000004] blkdev_issue_zeroout+0x13f/0x220 [ +0.000005] ext4_init_inode_table+0x203/0x380 [ +0.000004] ext4_lazyinit_thread.part.0+0x2be/0x3c0 [ +0.000003] ext4_lazyinit_thread+0x20/0x30 [ +0.000003] kthread+0x104/0x140 [ +0.000002] ? ext4_lazyinit_thread.part.0+0x3c0/0x3c0 [ +0.000003] ? kthread_park+0x90/0x90 [ +0.000002] ret_from_fork+0x1f/0x40 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync Status in linux package in Ubuntu: Confirmed Status in linux-signed-hwe-5.11 package in Ubuntu: Confirmed Status in linux-signed-hwe-5.4 package in Ubuntu: Confirmed Bug description: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack: 0 pid: 376 ppid: 1 flags:0x00000220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:00007ffdceb29e50 EFLAGS: 00000293 ORIG_RAX: 000000000000004a [921588.558991] RAX: ffffffffffffffda RBX: 000055ced34b0fa0 RCX: 00007f4c97ee832b [921588.558993] RDX: 00007f4c97fc8000 RSI: 000055ced3487b70 RDI: 0000000000000021 [921588.558995] RBP: 0000000000000001 R08: 0000000000000000 R09: 00007ffdceb29fa8 [921588.558996] R10: 00007f4c97d2c848 R11: 0000000000000293 R12: 00007ffdceb29fa8 [921588.558998] R13: 00007ffdceb29fa0 R14: 000055ced34b0fa0 R15: 000055ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack: 0 pid: 1505 ppid: 1 flags:0x00000000 [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ? kmem_cache_alloc+0x38e/0x440 [921588.559084] ? ext4_init_io_end+0x1f/0x50 [921588.559088] ext4_io_submit+0x4d/0x60 [921588.559091] ext4_writepages+0x2c6/0xcd0 [921588.559094] ? __schedule+0x454/0x8a0 [921588.559097] ? hrtimer_start_range_ns+0x1aa/0x2f0 [921588.559100] ? timerqueue_del+0x24/0x50 [921588.559105] ? futex_wait+0x1ed/0x270 [921588.559109] do_writepages+0x43/0xd0 [921588.559112] ? do_writepages+0x43/0xd0 [921588.559115] ? futex_wake+0x7c/0x180 [921588.559118] __filemap_fdatawrite_range+0xcc/0x110 [921588.559123] file_write_and_wait_range+0x74/0xc0 [921588.559126] ext4_sync_file+0xf5/0x350 [921588.559129] vfs_fsync_range+0x49/0x80 [921588.559132] ? __fget_light+0x62/0x80 [921588.559136] do_fsync+0x3d/0x70 [921588.559139] __x64_sys_fsync+0x14/0x20 [921588.559142] do_syscall_64+0x38/0x90 [921588.559145] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.559148] RIP: 0033:0x7f36188d48db [921588.559150] RSP: 002b:00007f35f05487d0 EFLAGS: 00000293 ORIG_RAX: 000000000000004a [921588.559154] RAX: ffffffffffffffda RBX: 00007f360502bf50 RCX: 00007f36188d48db [921588.559156] RDX: 000d239200000000 RSI: 0000000000000000 RDI: 000000000000000a [921588.559158] RBP: 00007f35f0548a00 R08: 0000000000000000 R09: 00007f360502beb0 [921588.559160] R10: 00007f35f0548ccf R11: 0000000000000293 R12: 00007f36042e9610 [921588.559162] R13: 000000000000000a R14: 0000000000000b00 R15: 0000000000000000 [921588.559175] INFO: task mdcheck:40625 blocked for more than 120 seconds. [921588.559179] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559182] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559185] task:mdcheck state:D stack: 0 pid:40625 ppid: 1 flags:0x00000000 [921588.559189] Call Trace: [921588.559191] __schedule+0x44c/0x8a0 [921588.559196] schedule+0x4f/0xc0 [921588.559200] schedule_timeout+0x202/0x290 [921588.559203] ? do_truncate+0x78/0xd0 [921588.559216] ? try_to_wake_up+0x66/0x5c0 [921588.559220] wait_for_completion+0x94/0x100 [921588.559225] kthread_stop+0x71/0x170 [921588.559229] md_unregister_thread+0x47/0x80 [921588.559232] md_reap_sync_thread+0x24/0x200 [921588.559236] action_store+0x160/0x2e0 [921588.559239] md_attr_store+0x82/0xc0 [921588.559243] sysfs_kf_write+0x3e/0x50 [921588.559246] kernfs_fop_write_iter+0x138/0x1c0 [921588.559249] new_sync_write+0x117/0x1b0 [921588.559254] vfs_write+0x1ca/0x280 [921588.559257] ksys_write+0x67/0xe0 [921588.559260] __x64_sys_write+0x1a/0x20 [921588.559264] do_syscall_64+0x38/0x90 [921588.559267] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.559271] RIP: 0033:0x7f8c6d2791e7 [921588.559273] RSP: 002b:00007ffea5d97178 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [921588.559276] RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 00007f8c6d2791e7 [921588.559278] RDX: 0000000000000005 RSI: 000055b618a3eb40 RDI: 0000000000000001 [921588.559280] RBP: 000055b618a3eb40 R08: 000000000000000a R09: 0000000000000004 [921588.559282] R10: 000055b616e97017 R11: 0000000000000246 R12: 0000000000000005 [921588.559284] R13: 00007f8c6d3546a0 R14: 00007f8c6d3554a0 R15: 00007f8c6d3548a0 [921588.559287] INFO: task md1_resync:40644 blocked for more than 120 seconds. [921588.559291] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559294] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559297] task:md1_resync state:D stack: 0 pid:40644 ppid: 2 flags:0x00004000 [921588.559301] Call Trace: [921588.559303] __schedule+0x44c/0x8a0 [921588.559308] schedule+0x4f/0xc0 [921588.559311] md_do_sync.cold+0x8b7/0x953 [921588.559315] ? wait_woken+0x80/0x80 [921588.559319] ? kernel_sigaction+0x3b/0xd0 [921588.559323] md_thread+0xb1/0x170 [921588.559326] kthread+0x114/0x150 [921588.559329] ? md_start_sync+0x60/0x60 [921588.559332] ? kthread_park+0x90/0x90 [921588.559335] ret_from_fork+0x22/0x30 [921588.559341] INFO: task kworker/u16:0:41409 blocked for more than 120 seconds. [921588.559345] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559348] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559353] task:kworker/u16:0 state:D stack: 0 pid:41409 ppid: 2 flags:0x00004000 [921588.559357] Workqueue: writeback wb_workfn (flush-253:1) [921588.559361] Call Trace: [921588.559363] __schedule+0x44c/0x8a0 [921588.559367] schedule+0x4f/0xc0 [921588.559371] md_write_start+0x150/0x240 [921588.559375] ? wait_woken+0x80/0x80 [921588.559378] raid5_make_request+0x88/0x890 [raid456] [921588.559387] ? wait_woken+0x80/0x80 [921588.559390] ? mempool_kmalloc+0x17/0x20 [921588.559393] md_handle_request+0x12d/0x1a0 [921588.559396] ? __part_start_io_acct+0x51/0xf0 [921588.559399] md_submit_bio+0xca/0x100 [921588.559402] submit_bio_noacct+0x112/0x4f0 [921588.559405] submit_bio+0x51/0x1a0 [921588.559408] ? _cond_resched+0x19/0x30 [921588.559412] ? kmem_cache_alloc+0x38e/0x440 [921588.559415] ? ext4_init_io_end+0x1f/0x50 [921588.559419] ext4_io_submit+0x4d/0x60 [921588.559422] ext4_writepages+0x2c6/0xcd0 [921588.559425] ? update_group_capacity+0x2c/0x1e0 [921588.559430] do_writepages+0x43/0xd0 [921588.559433] ? do_writepages+0x43/0xd0 [921588.559436] __writeback_single_inode+0x44/0x2b0 [921588.559439] writeback_sb_inodes+0x22d/0x4c0 [921588.559443] __writeback_inodes_wb+0x56/0xf0 [921588.559446] wb_writeback+0x1ea/0x2a0 [921588.559450] wb_workfn+0x31b/0x490 [921588.559453] ? __update_idle_core+0x9b/0xb0 [921588.559456] process_one_work+0x220/0x3c0 [921588.559459] worker_thread+0x4d/0x3f0 [921588.559462] kthread+0x114/0x150 [921588.559465] ? process_one_work+0x3c0/0x3c0 [921588.559467] ? kthread_park+0x90/0x90 [921588.559471] ret_from_fork+0x22/0x30 [921709.391056] INFO: task jbd2/dm-1-8:319 blocked for more than 120 seconds. [921709.391068] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921709.391073] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921709.391077] task:jbd2/dm-1-8 state:D stack: 0 pid: 319 ppid: 2 flags:0x00004000 [921709.391083] Call Trace: [921709.391087] __schedule+0x44c/0x8a0 [921709.391094] schedule+0x4f/0xc0 [921709.391098] md_write_start+0x150/0x240 [921709.391103] ? wait_woken+0x80/0x80 [921709.391108] raid5_make_request+0x88/0x890 [raid456] [921709.391120] ? wait_woken+0x80/0x80 [921709.391123] ? mempool_kmalloc+0x17/0x20 [921709.391127] md_handle_request+0x12d/0x1a0 [921709.391130] ? __part_start_io_acct+0x51/0xf0 [921709.391134] md_submit_bio+0xca/0x100 [921709.391137] submit_bio_noacct+0x112/0x4f0 [921709.391141] submit_bio+0x51/0x1a0 [921709.391144] ? bio_add_page+0x6a/0x90 [921709.391147] submit_bh_wbc+0x195/0x1c0 [921709.391151] submit_bh+0x13/0x20 [921709.391154] jbd2_journal_commit_transaction+0x74b/0x18f0 [921709.391159] ? try_to_del_timer_sync+0x54/0x80 [921709.391164] kjournald2+0xb6/0x280 [921709.391168] ? wait_woken+0x80/0x80 [921709.391170] kthread+0x114/0x150 [921709.391174] ? commit_timeout+0x20/0x20 [921709.391177] ? kthread_park+0x90/0x90 [921709.391180] ret_from_fork+0x22/0x30 [921709.391186] INFO: task systemd-journal:376 blocked for more than 241 seconds. [921709.391191] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921709.391195] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921709.391199] task:systemd-journal state:D stack: 0 pid: 376 ppid: 1 flags:0x00000220 [921709.391204] Call Trace: [921709.391205] __schedule+0x44c/0x8a0 [921709.391210] schedule+0x4f/0xc0 [921709.391214] md_write_start+0x150/0x240 [921709.391218] ? wait_woken+0x80/0x80 [921709.391221] raid5_make_request+0x88/0x890 [raid456] [921709.391230] ? wait_woken+0x80/0x80 [921709.391233] ? mempool_kmalloc+0x17/0x20 [921709.391236] md_handle_request+0x12d/0x1a0 [921709.391239] ? __part_start_io_acct+0x51/0xf0 [921709.391242] md_submit_bio+0xca/0x100 [921709.391245] submit_bio_noacct+0x112/0x4f0 [921709.391248] ? ext4_fc_reserve_space+0x110/0x230 [921709.391252] submit_bio+0x51/0x1a0 [921709.391255] ? _cond_resched+0x19/0x30 [921709.391259] ? kmem_cache_alloc+0x38e/0x440 [921709.391262] ? ext4_init_io_end+0x1f/0x50 [921709.391266] ext4_io_submit+0x4d/0x60 [921709.391270] ext4_writepages+0x2c6/0xcd0 [921709.391274] do_writepages+0x43/0xd0 [921709.391278] ? do_writepages+0x43/0xd0 [921709.391281] ? fault_dirty_shared_page+0xa5/0x110 [921709.391285] __filemap_fdatawrite_range+0xcc/0x110 [921709.391290] file_write_and_wait_range+0x74/0xc0 [921709.391293] ext4_sync_file+0xf5/0x350 [921709.391297] vfs_fsync_range+0x49/0x80 [921709.391300] do_fsync+0x3d/0x70 [921709.391303] __x64_sys_fsync+0x14/0x20 [921709.391306] do_syscall_64+0x38/0x90 [921709.391310] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921709.391314] RIP: 0033:0x7f4c97ee832b [921709.391317] RSP: 002b:00007ffdceb29e50 EFLAGS: 00000293 ORIG_RAX: 000000000000004a [921709.391321] RAX: ffffffffffffffda RBX: 000055ced34b0fa0 RCX: 00007f4c97ee832b [921709.391323] RDX: 00007f4c97fc8000 RSI: 000055ced3487b70 RDI: 0000000000000021 [921709.391325] RBP: 0000000000000001 R08: 0000000000000000 R09: 00007ffdceb29fa8 [921709.391327] R10: 00007f4c97d2c848 R11: 0000000000000293 R12: 00007ffdceb29fa8 [921709.391329] R13: 00007ffdceb29fa0 R14: 000055ced34b0fa0 R15: 000055ced34bcf90 [921709.391343] INFO: task mysqld:1505 blocked for more than 241 seconds. [921709.391348] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921709.391351] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921709.391355] task:mysqld state:D stack: 0 pid: 1505 ppid: 1 flags:0x00000000 [921709.391359] Call Trace: [921709.391361] __schedule+0x44c/0x8a0 [921709.391366] schedule+0x4f/0xc0 [921709.391370] md_write_start+0x150/0x240 [921709.391373] ? wait_woken+0x80/0x80 [921709.391376] raid5_make_request+0x88/0x890 [raid456] [921709.391386] ? wait_woken+0x80/0x80 [921709.391388] ? mempool_kmalloc+0x17/0x20 [921709.391391] md_handle_request+0x12d/0x1a0 [921709.391395] ? __part_start_io_acct+0x51/0xf0 [921709.391398] md_submit_bio+0xca/0x100 [921709.391401] submit_bio_noacct+0x112/0x4f0 [921709.391404] submit_bio+0x51/0x1a0 [921709.391407] ? _cond_resched+0x19/0x30 [921709.391411] ? kmem_cache_alloc+0x38e/0x440 [921709.391414] ? ext4_init_io_end+0x1f/0x50 [921709.391418] ext4_io_submit+0x4d/0x60 [921709.391421] ext4_writepages+0x2c6/0xcd0 [921709.391423] ? __schedule+0x454/0x8a0 [921709.391427] ? hrtimer_start_range_ns+0x1aa/0x2f0 [921709.391430] ? timerqueue_del+0x24/0x50 [921709.391435] ? futex_wait+0x1ed/0x270 [921709.391439] do_writepages+0x43/0xd0 [921709.391443] ? do_writepages+0x43/0xd0 [921709.391445] ? futex_wake+0x7c/0x180 [921709.391449] __filemap_fdatawrite_range+0xcc/0x110 [921709.391453] file_write_and_wait_range+0x74/0xc0 [921709.391456] ext4_sync_file+0xf5/0x350 [921709.391459] vfs_fsync_range+0x49/0x80 [921709.391463] ? __fget_light+0x62/0x80 [921709.391466] do_fsync+0x3d/0x70 [921709.391469] __x64_sys_fsync+0x14/0x20 [921709.391472] do_syscall_64+0x38/0x90 [921709.391475] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921709.391478] RIP: 0033:0x7f36188d48db [921709.391481] RSP: 002b:00007f35f05487d0 EFLAGS: 00000293 ORIG_RAX: 000000000000004a [921709.391484] RAX: ffffffffffffffda RBX: 00007f360502bf50 RCX: 00007f36188d48db [921709.391486] RDX: 000d239200000000 RSI: 0000000000000000 RDI: 000000000000000a [921709.391488] RBP: 00007f35f0548a00 R08: 0000000000000000 R09: 00007f360502beb0 [921709.391490] R10: 00007f35f0548ccf R11: 0000000000000293 R12: 00007f36042e9610 [921709.391491] R13: 000000000000000a R14: 0000000000000b00 R15: 0000000000000000 [921709.391505] INFO: task mdcheck:40625 blocked for more than 241 seconds. [921709.391510] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921709.391514] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921709.391518] task:mdcheck state:D stack: 0 pid:40625 ppid: 1 flags:0x00000000 [921709.391522] Call Trace: [921709.391524] __schedule+0x44c/0x8a0 [921709.391528] schedule+0x4f/0xc0 [921709.391532] schedule_timeout+0x202/0x290 [921709.391535] ? do_truncate+0x78/0xd0 [921709.391538] ? try_to_wake_up+0x66/0x5c0 [921709.391542] wait_for_completion+0x94/0x100 [921709.391547] kthread_stop+0x71/0x170 [921709.391550] md_unregister_thread+0x47/0x80 [921709.391554] md_reap_sync_thread+0x24/0x200 [921709.391557] action_store+0x160/0x2e0 [921709.391561] md_attr_store+0x82/0xc0 [921709.391564] sysfs_kf_write+0x3e/0x50 [921709.391567] kernfs_fop_write_iter+0x138/0x1c0 [921709.391571] new_sync_write+0x117/0x1b0 [921709.391575] vfs_write+0x1ca/0x280 [921709.391578] ksys_write+0x67/0xe0 [921709.391581] __x64_sys_write+0x1a/0x20 [921709.391584] do_syscall_64+0x38/0x90 [921709.391588] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921709.391591] RIP: 0033:0x7f8c6d2791e7 [921709.391593] RSP: 002b:00007ffea5d97178 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [921709.391597] RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 00007f8c6d2791e7 [921709.391599] RDX: 0000000000000005 RSI: 000055b618a3eb40 RDI: 0000000000000001 [921709.391600] RBP: 000055b618a3eb40 R08: 000000000000000a R09: 0000000000000004 [921709.391602] R10: 000055b616e97017 R11: 0000000000000246 R12: 0000000000000005 [921709.391604] R13: 00007f8c6d3546a0 R14: 00007f8c6d3554a0 R15: 00007f8c6d3548a0 [921709.391607] INFO: task md1_resync:40644 blocked for more than 241 seconds. [921709.391611] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921709.391615] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921709.391619] task:md1_resync state:D stack: 0 pid:40644 ppid: 2 flags:0x00004000 [921709.391623] Call Trace: [921709.391625] __schedule+0x44c/0x8a0 [921709.391631] schedule+0x4f/0xc0 [921709.391635] md_do_sync.cold+0x8b7/0x953 [921709.391639] ? wait_woken+0x80/0x80 [921709.391642] ? kernel_sigaction+0x3b/0xd0 [921709.391646] md_thread+0xb1/0x170 [921709.391650] kthread+0x114/0x150 [921709.391652] ? md_start_sync+0x60/0x60 [921709.391655] ? kthread_park+0x90/0x90 [921709.391658] ret_from_fork+0x22/0x30 Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md1 : active raid5 sda3[0] sdd3[3] sdb3[1] sde3[5] sdc3[6] 15625582592 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5] [UUUUU] [================>....] check = 84.2% (3289834208/3906395648) finish=82779.6min speed=124K/sec md0 : active raid1 sda2[0] sdb2[1] sdc2[2](S) 488128 blocks super 1.2 [2/2] [UU] unused devices: <none> ProblemType: Bug DistroRelease: Ubuntu 20.04 Package: linux-image-5.11.0-27-generic 5.11.0-27.29~20.04.1 ProcVersionSignature: Ubuntu 5.11.0-27.29~20.04.1-generic 5.11.22 Uname: Linux 5.11.0-27-generic x86_64 ApportVersion: 2.20.11-0ubuntu27.18 Architecture: amd64 CasperMD5CheckResult: skip Date: Tue Sep 7 16:35:01 2021 InstallationDate: Installed on 2014-03-12 (2736 days ago) InstallationMedia: Ubuntu-Server 12.04.4 LTS "Precise Pangolin" - Release amd64 (20140204) ProcEnviron: TERM=xterm PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: linux-signed-hwe-5.11 UpgradeStatus: Upgraded to focal on 2020-12-20 (261 days ago) --- ProblemType: Bug AlsaVersion: Advanced Linux Sound Architecture Driver Version k5.13.0-16-generic. AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.11-0ubuntu69 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: [Errno 2] No such file or directory: 'fuser' Card0.Amixer.info: Error: [Errno 2] No such file or directory: 'amixer' Card0.Amixer.values: Error: [Errno 2] No such file or directory: 'amixer' CasperMD5CheckResult: unknown DistroRelease: Ubuntu 21.10 IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Error: [Errno 2] No such file or directory: 'lspci' Lspci-vt: Error: [Errno 2] No such file or directory: 'lspci' Lsusb: Error: [Errno 2] No such file or directory: 'lsusb' Lsusb-t: Error: [Errno 2] No such file or directory: 'lsusb' Lsusb-v: Error: [Errno 2] No such file or directory: 'lsusb' MachineType: innotek GmbH VirtualBox Package: linux-signed-hwe-5.11 PciMultimedia: ProcEnviron: TERM=linux PATH=(custom, no user) SHELL=/bin/bash ProcFB: 0 svgadrmfb ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.13.0-16-generic root=/dev/mapper/vg1-root ro quiet splash ProcVersionSignature: Ubuntu 5.13.0-16.16-generic 5.13.13 RelatedPackageVersions: linux-restricted-modules-5.13.0-16-generic N/A linux-backports-modules-5.13.0-16-generic N/A linux-firmware 1.199 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' Tags: impish Uname: Linux 5.13.0-16-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: N/A _MarkForUpload: True dmi.bios.date: 12/01/2006 dmi.bios.vendor: innotek GmbH dmi.bios.version: VirtualBox dmi.board.name: VirtualBox dmi.board.vendor: Oracle Corporation dmi.board.version: 1.2 dmi.chassis.type: 1 dmi.chassis.vendor: Oracle Corporation dmi.modalias: dmi:bvninnotekGmbH:bvrVirtualBox:bd12/01/2006:svninnotekGmbH:pnVirtualBox:pvr1.2:sku:rvnOracleCorporation:rnVirtualBox:rvr1.2:cvnOracleCorporation:ct1:cvr: dmi.product.family: Virtual Machine dmi.product.name: VirtualBox dmi.product.version: 1.2 dmi.sys.vendor: innotek GmbH To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942935/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : [email protected] Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
