[Bug 1942935] Re: kernel io hangs during mdcheck/resync
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: linux-signed-hwe-6.5 (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942935/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942935] Re: kernel io hangs during mdcheck/resync
** Also affects: linux-signed-hwe-6.5 (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942935/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942935] Re: kernel io hangs during mdcheck/resync
Hi all. At the beginning of the year we upgraded from kernel 5.4 to 6.5 and have started since then to experience this issue on several HPE servers. The issue is very sporadic (some servers have experienced it only once or twice this year), so we have been restarting the servers as a workaround, but today we found this bug report and could confirm this is our problem. The workaround suggested on https://bugs.launchpad.net/ubuntu/+source/linux-signed- hwe-5.11/+bug/1942935/comments/49 worked OK for us. However, since we're on 6.5 we can confirm the problem still persists on that version. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942935/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942935] Re: kernel io hangs during mdcheck/resync
Looks like two patches are landing in next to resolve this: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20220527&id=8b48ec23cc51a4e7c8dbaef5f34ebe67e1a80934 https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20220527&id=1e267742283a4b5a8ca65755c44166be27e9aa0f -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942935/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942935] Re: kernel io hangs during mdcheck/resync
** Tags added: patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942935/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942935] Re: kernel io hangs during mdcheck/resync
Same issue on impish 5.13.13 kernel, running in VBox. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942935/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942935] Re: kernel io hangs during mdcheck/resync
** Patch added: "md-reap-sync-thread.patch" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942935/+attachment/5526028/+files/md-reap-sync-thread.patch ** Tags added: apport-collected impish ** Description changed: It seems to always occur during an mdcheck/resync, if I am logged in via SSH it is still somewhat responsive and basic utilities like dmesg will work. But it apppears any write I/O will hang the terminal and nothing is written to syslog (presumably because it is blocked). Below is output of dmesg and cat /proc/mdstat, it appears the data check was interrupted and /proc/mdstat still shows progress, and a whole slew of hung tasks including md1_resync itself. [756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack:0 pid: 376 ppid: 1 flags:0x0220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:7ffdceb29e50 EFLAGS: 0293 ORIG_RAX: 004a [921588.558991] RAX: ffda RBX: 55ced34b0fa0 RCX: 7f4c97ee832b [921588.558993] RDX: 7f4c97fc8000 RSI: 55ced3487b70 RDI: 0021 [921588.558995] RBP: 0001 R08: R09: 7ffdceb29fa8 [921588.558996] R10: 7f4c97d2c848 R11: 0293 R12: 7ffdceb29fa8 [921588.558998] R13: 7ffdceb29fa0 R14: 55ced34b0fa0 R15: 55ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack:0 pid: 1505 ppid: 1 flags:0x [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ? kmem_cache_alloc+0x38e/0x440 [921588.559084] ? ext4_init_io_end+0x1f/0x50 [921588.559088] ext4_io_submit+0x4d/0x60 [921588.559091] ext4_writepages+0x2c6/0xcd0 [921588.559094] ? __schedule+0x454/0x8a0 [921588.559097] ? hrtimer_start_range_ns+0x1aa/0x2f0 [921588.559100] ? timerqueue_del+0x24/0x50 [921588.559105] ? futex_wait+0x1ed/0x270 [921588.559109] do_writepages+0x43/0xd0 [921588.559112] ? do_writepages+0x43/0xd0 [921588.559115] ? futex_wake
[Bug 1942935] Re: kernel io hangs during mdcheck/resync
** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942935/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942935] Re: kernel io hangs during mdcheck/resync
** Tags removed: hirsute -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942935/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942935] Re: kernel io hangs during mdcheck/resync
** Also affects: linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942935/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942935] Re: kernel io hangs during mdcheck/resync
Here is Donald Buczek's reproducer script. I setup an Ubuntu 20.04 VM with latest linux-image-generic and was able to reproduce it within maybe 10 or 15 minutes. Exactly the same issue. Filesystem layout built as follows: # assemble raid devices mdadm --create /dev/md0 --level=1 --raid-devices=2 --spare-devices=1 /dev/sda2 /dev/sdb2 /dev/sdc2 mdadm --create /dev/md1 --level=5 --raid-devices=5 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3 /dev/sde3 # create PVs, VGs, LVs pvcreate /dev/md1 vgcreate vg1 /dev/md1 lvcreate --name root --extents 100%FREE vg1 # create filesystems mkfs.ext4 /dev/md0 mkfs.ext4 /dev/vg1/root ** Attachment added: "mdhang.sh" https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-5.11/+bug/1942935/+attachment/5525050/+files/mdhang.sh -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-5.11/+bug/1942935/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942935] Re: kernel io hangs during mdcheck/resync
The patch hasn't made it into mainline from what I have seen, it looks like it died back in March waiting for feedback from additional kernel developers. From what I have gathered this is a deadlock scenario directly caused by pausing the resync while the system is under heavy write activity. Donald Buczek provided a reproducer which is a shell script that generates a lot of write activity and pauses/resumes the raid scrubbing. And he also provided a workaround to get the stuck system running without reboot: echo active > /sys/block/md1/md/array_state I haven't tried the patch or any of this, I pretty much eliminated the trigger which is mdcheck_start & mdcheck_continue and went back to the 18.04 LTS way of scrubbing arrays (which is basically don't pause/interrupt it once it starts). I ran through a checkarray yesterday to 100%, no problems. Meanwhile since upgrading to 20.04 LTS it has hung almost every single time through 5.4, 5.8, and 5.11 kernels. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-5.11/+bug/1942935/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942935] Re: kernel io hangs during mdcheck/resync
One of the systems was using package linux-generic and in practice Linux pedabackup 5.4.0-80-generic #90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux $ cat /proc/version_signature Ubuntu 5.4.0-80.90-generic 5.4.124 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-5.11/+bug/1942935/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942935] Re: kernel io hangs during mdcheck/resync
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: linux-signed-hwe-5.11 (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-5.11/+bug/1942935/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942935] Re: kernel io hangs during mdcheck/resync
I think I've seen this issue once per two different systems so I think this is software issue. Does anybody know if patch in comment #6 is going to be included in Ubuntu 20.04 LTS? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-5.11/+bug/1942935/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942935] Re: kernel io hangs during mdcheck/resync
Here is the proposed patch, Doesn't appear to have been applied. Last report was with 5.11rc5. https://lore.kernel.org/linux-raid/1613177399-22024-1-git-send-email- guoqing.ji...@cloud.ionos.com/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-5.11/+bug/1942935/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942935] Re: kernel io hangs during mdcheck/resync
Similar report here on 5.10.0-rc4: https://www.spinics.net/lists/raid/msg66654.html I ended up masking the services introduced with 20.04 LTS, and switched back the crontab. systemctl mask mdcheck_continue.service mdcheck_continue.timer mdcheck_start.service mdcheck_start.timer cat > /etc/cron.d/mdadm << 'EOF' # # cron.d/mdadm -- schedules periodic redundancy checks of MD devices # # Copyright © martin f. krafft # distributed under the terms of the Artistic Licence 2.0 # # By default, run at 00:57 on every Sunday, but do nothing unless the day of # the month is less than or equal to 7. Thus, only run on the first Sunday of # each month. crontab(5) sucks, unfortunately, in this regard; therefore this # hack (see #380425). 57 0 * * 0 root if [ -x /usr/share/mdadm/checkarray ] && [ $(date +\%d) -le 7 ]; then /usr/share/mdadm/checkarray --cron --all --idle --quiet; fi EOF The pausing and resuming of the integrity check was an annoyance for me anyways. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-5.11/+bug/1942935/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942935] Re: kernel io hangs during mdcheck/resync
Hi Kleber, I installed it later yesterday, but I won't know until the next resync. This has been a problem since at least linux 5.4 kernel that shipped with Ubuntu 20.04. I don't think I had these problems on Ubuntu 18.04 LTS, the same hardware, running the linux-image-generic at that time. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-5.11/+bug/1942935/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942935] Re: kernel io hangs during mdcheck/resync
Hello Chad Wagner, Thank you for reporting this issue. Could you please try installing the latest 20.04 HWE kernel and check whether the problem persists? The version currently in focal-updates is 5.11.0-34.36~20.04.1. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-5.11/+bug/1942935/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1942935] Re: kernel io hangs during mdcheck/resync
** Attachment added: "screenlog.txt" https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-5.11/+bug/1942935/+attachment/5523575/+files/screenlog.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942935 Title: kernel io hangs during mdcheck/resync To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-signed-hwe-5.11/+bug/1942935/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
