[Touch-packages] [Bug 1587142] Re: Shutdown hangs in md kworker after "Reached target Shutdown."
as I don't want to rebuild now again, did anyone try eg. chmod +x ~/myshutdown.sh sudo ~/myshutdown.sh # with echo r > /proc/sysrq-trigger echo e > /proc/sysrq-trigger echo s > /proc/sysrq-trigger echo i > /proc/sysrq-trigger echo u > /proc/sysrq-trigger echo o > /proc/sysrq-trigger -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1587142 Title: Shutdown hangs in md kworker after "Reached target Shutdown." Status in systemd package in Ubuntu: Confirmed Bug description: I'm booting a fully patched 16.04 from an Intel Rapid Storage Technology enterprise RAID1 volume (ThinkServer TS140 with two SATA ST1000NM0033-9ZM drives, ext4 root partition, no LVM, UEFI mode). If the RAID volume is recovering or resyncing for whatever reason, then `sudo systemctl reboot` and `sudo systemctl poweroff` work fine (I had to `sudo systemctl --now disable lvm2-lvmetad lvm2-lvmpolld lvm2-monitor` in order to consistently get that). However, once the recovery/resync is complete and clean, the reboot and poweroff commands above hang forever after "Reached target Shutdown.". Note that issuing `sudo swapoff -a` beforehand (suggested in the bug #1464917) does not help. [EDIT]Actually, the shutdown also hangs from time to time during a resync. But I've never seen it succeed once the resync is complete.[/EDIT] Then, if the server has been forcibly restarted with the power button, the Intel Matrix Storage Manager indicates a "Normal" status for the RAID1 volume, but Ubuntu then resyncs the volume anyway: [1.223649] md: bind [1.228426] md: bind [1.230030] md: bind [1.230738] md: bind [1.232985] usbcore: registered new interface driver usbhid [1.233494] usbhid: USB HID core driver [1.234022] md: raid1 personality registered for level 1 [1.234876] md/raid1:md126: not clean -- starting background reconstruction [1.234956] input: CHESEN USB Keyboard as /devices/pci:00/:00:14.0/usb3/3-10/3-10:1.0/0003:0A81:0101.0001/input/input5 [1.236273] md/raid1:md126: active with 2 out of 2 mirrors [1.236797] md126: detected capacity change from 0 to 1000202043392 [1.246271] md: md126 switched to read-write mode. [1.246834] md: resync of RAID array md126 [1.247325] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. [1.247503] md126: p1 p2 p3 p4 [1.248269] md: using maximum available idle IO bandwidth (but not more than 20 KB/sec) for resync. [1.248774] md: using 128k window, over a total of 976759940k. Note that the pain of "resync upon every (re)boot" cannot even be a bit relieved thanks to bitmaps because mdadm does not support them for IMSM containers: $ sudo mdadm --grow --bitmap=internal /dev/md126 mdadm: Cannot add bitmaps to sub-arrays yet I also get this in syslog during boot when the individual drives are detected, but this seems to be harmless: May 30 17:26:07 wssrv1 systemd-udevd[608]: Process '/sbin/mdadm --incremental /dev/sdb --offroot' failed with exit code 1. May 30 17:26:07 wssrv1 systemd-udevd[608]: Process '/lib/udev/hdparm' failed with exit code 1. May 30 17:26:07 wssrv1 systemd-udevd[606]: Process '/sbin/mdadm --incremental /dev/sda --offroot' failed with exit code 1. May 30 17:26:07 wssrv1 systemd-udevd[606]: Process '/lib/udev/hdparm' failed with exit code 1. During a resync, `sudo sh -c 'echo idle > /sys/block/md126/md/sync_action'` actually stops it as expected, but it restarts immediately though nothing seems to have triggered it: May 30 18:17:02 wssrv1 kernel: [ 3106.826710] md: md126: resync interrupted. May 30 18:17:02 wssrv1 kernel: [ 3106.836320] md: checkpointing resync of md126. May 30 18:17:02 wssrv1 kernel: [ 3106.836623] md: resync of RAID array md126 May 30 18:17:02 wssrv1 kernel: [ 3106.836625] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. May 30 18:17:02 wssrv1 kernel: [ 3106.836626] md: using maximum available idle IO bandwidth (but not more than 20 KB/sec) for resync. May 30 18:17:02 wssrv1 kernel: [ 3106.836627] md: using 128k window, over a total of 976759940k. May 30 18:17:02 wssrv1 kernel: [ 3106.836628] md: resuming resync of md126 from checkpoint. May 30 18:17:02 wssrv1 mdadm[982]: RebuildStarted event detected on md device /dev/md/Volume0 I attach screenshots of the hanging shutdown log after a `sudo sh -c 'echo 8 > /proc/sys/kernel/printk'`. The second screenshot shows that the kernel has deadlocked in md_write_start(). Note that `sudo systemctl start debug-shell` is unusable on this machine at this point because Ctrl+Alt+F9 brings tty9 without any keyboard. [EDIT]But I can still switch back to tty1.[/EDIT] I have also tried with much lower values for vm.dirty_background_ratio and vm.dirty_ratio, but to
[Touch-packages] [Bug 1587142] Re: Shutdown hangs in md kworker after "Reached target Shutdown."
a BIOS reconfig. from "RAID" to "AHCI" did not change anything. m. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1587142 Title: Shutdown hangs in md kworker after "Reached target Shutdown." Status in systemd package in Ubuntu: Confirmed Bug description: I'm booting a fully patched 16.04 from an Intel Rapid Storage Technology enterprise RAID1 volume (ThinkServer TS140 with two SATA ST1000NM0033-9ZM drives, ext4 root partition, no LVM, UEFI mode). If the RAID volume is recovering or resyncing for whatever reason, then `sudo systemctl reboot` and `sudo systemctl poweroff` work fine (I had to `sudo systemctl --now disable lvm2-lvmetad lvm2-lvmpolld lvm2-monitor` in order to consistently get that). However, once the recovery/resync is complete and clean, the reboot and poweroff commands above hang forever after "Reached target Shutdown.". Note that issuing `sudo swapoff -a` beforehand (suggested in the bug #1464917) does not help. [EDIT]Actually, the shutdown also hangs from time to time during a resync. But I've never seen it succeed once the resync is complete.[/EDIT] Then, if the server has been forcibly restarted with the power button, the Intel Matrix Storage Manager indicates a "Normal" status for the RAID1 volume, but Ubuntu then resyncs the volume anyway: [1.223649] md: bind [1.228426] md: bind [1.230030] md: bind [1.230738] md: bind [1.232985] usbcore: registered new interface driver usbhid [1.233494] usbhid: USB HID core driver [1.234022] md: raid1 personality registered for level 1 [1.234876] md/raid1:md126: not clean -- starting background reconstruction [1.234956] input: CHESEN USB Keyboard as /devices/pci:00/:00:14.0/usb3/3-10/3-10:1.0/0003:0A81:0101.0001/input/input5 [1.236273] md/raid1:md126: active with 2 out of 2 mirrors [1.236797] md126: detected capacity change from 0 to 1000202043392 [1.246271] md: md126 switched to read-write mode. [1.246834] md: resync of RAID array md126 [1.247325] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. [1.247503] md126: p1 p2 p3 p4 [1.248269] md: using maximum available idle IO bandwidth (but not more than 20 KB/sec) for resync. [1.248774] md: using 128k window, over a total of 976759940k. Note that the pain of "resync upon every (re)boot" cannot even be a bit relieved thanks to bitmaps because mdadm does not support them for IMSM containers: $ sudo mdadm --grow --bitmap=internal /dev/md126 mdadm: Cannot add bitmaps to sub-arrays yet I also get this in syslog during boot when the individual drives are detected, but this seems to be harmless: May 30 17:26:07 wssrv1 systemd-udevd[608]: Process '/sbin/mdadm --incremental /dev/sdb --offroot' failed with exit code 1. May 30 17:26:07 wssrv1 systemd-udevd[608]: Process '/lib/udev/hdparm' failed with exit code 1. May 30 17:26:07 wssrv1 systemd-udevd[606]: Process '/sbin/mdadm --incremental /dev/sda --offroot' failed with exit code 1. May 30 17:26:07 wssrv1 systemd-udevd[606]: Process '/lib/udev/hdparm' failed with exit code 1. During a resync, `sudo sh -c 'echo idle > /sys/block/md126/md/sync_action'` actually stops it as expected, but it restarts immediately though nothing seems to have triggered it: May 30 18:17:02 wssrv1 kernel: [ 3106.826710] md: md126: resync interrupted. May 30 18:17:02 wssrv1 kernel: [ 3106.836320] md: checkpointing resync of md126. May 30 18:17:02 wssrv1 kernel: [ 3106.836623] md: resync of RAID array md126 May 30 18:17:02 wssrv1 kernel: [ 3106.836625] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. May 30 18:17:02 wssrv1 kernel: [ 3106.836626] md: using maximum available idle IO bandwidth (but not more than 20 KB/sec) for resync. May 30 18:17:02 wssrv1 kernel: [ 3106.836627] md: using 128k window, over a total of 976759940k. May 30 18:17:02 wssrv1 kernel: [ 3106.836628] md: resuming resync of md126 from checkpoint. May 30 18:17:02 wssrv1 mdadm[982]: RebuildStarted event detected on md device /dev/md/Volume0 I attach screenshots of the hanging shutdown log after a `sudo sh -c 'echo 8 > /proc/sys/kernel/printk'`. The second screenshot shows that the kernel has deadlocked in md_write_start(). Note that `sudo systemctl start debug-shell` is unusable on this machine at this point because Ctrl+Alt+F9 brings tty9 without any keyboard. [EDIT]But I can still switch back to tty1.[/EDIT] I have also tried with much lower values for vm.dirty_background_ratio and vm.dirty_ratio, but to no avail. Linux 4.6.0-040600-generic_4.6.0-040600.201605151930_amd64 from http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.6-yakkety/ did not help either. More information below: $ lsb_release -rd Description: Ubuntu 16.04 LTS Release: 16.04 $
[Touch-packages] [Bug 1587142] Re: Shutdown hangs in md kworker after "Reached target Shutdown."
Shutting down with a "R E S U I O" Workaround was ok. The Raid was not rebuilt. m. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1587142 Title: Shutdown hangs in md kworker after "Reached target Shutdown." Status in systemd package in Ubuntu: Confirmed Bug description: I'm booting a fully patched 16.04 from an Intel Rapid Storage Technology enterprise RAID1 volume (ThinkServer TS140 with two SATA ST1000NM0033-9ZM drives, ext4 root partition, no LVM, UEFI mode). If the RAID volume is recovering or resyncing for whatever reason, then `sudo systemctl reboot` and `sudo systemctl poweroff` work fine (I had to `sudo systemctl --now disable lvm2-lvmetad lvm2-lvmpolld lvm2-monitor` in order to consistently get that). However, once the recovery/resync is complete and clean, the reboot and poweroff commands above hang forever after "Reached target Shutdown.". Note that issuing `sudo swapoff -a` beforehand (suggested in the bug #1464917) does not help. [EDIT]Actually, the shutdown also hangs from time to time during a resync. But I've never seen it succeed once the resync is complete.[/EDIT] Then, if the server has been forcibly restarted with the power button, the Intel Matrix Storage Manager indicates a "Normal" status for the RAID1 volume, but Ubuntu then resyncs the volume anyway: [1.223649] md: bind [1.228426] md: bind [1.230030] md: bind [1.230738] md: bind [1.232985] usbcore: registered new interface driver usbhid [1.233494] usbhid: USB HID core driver [1.234022] md: raid1 personality registered for level 1 [1.234876] md/raid1:md126: not clean -- starting background reconstruction [1.234956] input: CHESEN USB Keyboard as /devices/pci:00/:00:14.0/usb3/3-10/3-10:1.0/0003:0A81:0101.0001/input/input5 [1.236273] md/raid1:md126: active with 2 out of 2 mirrors [1.236797] md126: detected capacity change from 0 to 1000202043392 [1.246271] md: md126 switched to read-write mode. [1.246834] md: resync of RAID array md126 [1.247325] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. [1.247503] md126: p1 p2 p3 p4 [1.248269] md: using maximum available idle IO bandwidth (but not more than 20 KB/sec) for resync. [1.248774] md: using 128k window, over a total of 976759940k. Note that the pain of "resync upon every (re)boot" cannot even be a bit relieved thanks to bitmaps because mdadm does not support them for IMSM containers: $ sudo mdadm --grow --bitmap=internal /dev/md126 mdadm: Cannot add bitmaps to sub-arrays yet I also get this in syslog during boot when the individual drives are detected, but this seems to be harmless: May 30 17:26:07 wssrv1 systemd-udevd[608]: Process '/sbin/mdadm --incremental /dev/sdb --offroot' failed with exit code 1. May 30 17:26:07 wssrv1 systemd-udevd[608]: Process '/lib/udev/hdparm' failed with exit code 1. May 30 17:26:07 wssrv1 systemd-udevd[606]: Process '/sbin/mdadm --incremental /dev/sda --offroot' failed with exit code 1. May 30 17:26:07 wssrv1 systemd-udevd[606]: Process '/lib/udev/hdparm' failed with exit code 1. During a resync, `sudo sh -c 'echo idle > /sys/block/md126/md/sync_action'` actually stops it as expected, but it restarts immediately though nothing seems to have triggered it: May 30 18:17:02 wssrv1 kernel: [ 3106.826710] md: md126: resync interrupted. May 30 18:17:02 wssrv1 kernel: [ 3106.836320] md: checkpointing resync of md126. May 30 18:17:02 wssrv1 kernel: [ 3106.836623] md: resync of RAID array md126 May 30 18:17:02 wssrv1 kernel: [ 3106.836625] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. May 30 18:17:02 wssrv1 kernel: [ 3106.836626] md: using maximum available idle IO bandwidth (but not more than 20 KB/sec) for resync. May 30 18:17:02 wssrv1 kernel: [ 3106.836627] md: using 128k window, over a total of 976759940k. May 30 18:17:02 wssrv1 kernel: [ 3106.836628] md: resuming resync of md126 from checkpoint. May 30 18:17:02 wssrv1 mdadm[982]: RebuildStarted event detected on md device /dev/md/Volume0 I attach screenshots of the hanging shutdown log after a `sudo sh -c 'echo 8 > /proc/sys/kernel/printk'`. The second screenshot shows that the kernel has deadlocked in md_write_start(). Note that `sudo systemctl start debug-shell` is unusable on this machine at this point because Ctrl+Alt+F9 brings tty9 without any keyboard. [EDIT]But I can still switch back to tty1.[/EDIT] I have also tried with much lower values for vm.dirty_background_ratio and vm.dirty_ratio, but to no avail. Linux 4.6.0-040600-generic_4.6.0-040600.201605151930_amd64 from http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.6-yakkety/ did not help either. More information below: $ lsb_release -rd Description: Ubuntu 16.04 LTS Release:
[Touch-packages] [Bug 1587142] Re: Shutdown hangs in md kworker after "Reached target Shutdown."
... interesting, but the pain is the new resync ... (writing 4(8) TB on each "hanging" reboot, ehm ...) I used SysReq to sync before power-off echo 1 > /proc/sys/kernel/sysrq ALT + "DRUCK" ( or "Print Screen") + R E I S U O as a workaround. as I have to wait for the sync, I can't report now. of course that's no fun @ for remote machines. ps. the servers were on centos before, I did not have any probs there ?! m. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1587142 Title: Shutdown hangs in md kworker after "Reached target Shutdown." Status in systemd package in Ubuntu: Confirmed Bug description: I'm booting a fully patched 16.04 from an Intel Rapid Storage Technology enterprise RAID1 volume (ThinkServer TS140 with two SATA ST1000NM0033-9ZM drives, ext4 root partition, no LVM, UEFI mode). If the RAID volume is recovering or resyncing for whatever reason, then `sudo systemctl reboot` and `sudo systemctl poweroff` work fine (I had to `sudo systemctl --now disable lvm2-lvmetad lvm2-lvmpolld lvm2-monitor` in order to consistently get that). However, once the recovery/resync is complete and clean, the reboot and poweroff commands above hang forever after "Reached target Shutdown.". Note that issuing `sudo swapoff -a` beforehand (suggested in the bug #1464917) does not help. [EDIT]Actually, the shutdown also hangs from time to time during a resync. But I've never seen it succeed once the resync is complete.[/EDIT] Then, if the server has been forcibly restarted with the power button, the Intel Matrix Storage Manager indicates a "Normal" status for the RAID1 volume, but Ubuntu then resyncs the volume anyway: [1.223649] md: bind [1.228426] md: bind [1.230030] md: bind [1.230738] md: bind [1.232985] usbcore: registered new interface driver usbhid [1.233494] usbhid: USB HID core driver [1.234022] md: raid1 personality registered for level 1 [1.234876] md/raid1:md126: not clean -- starting background reconstruction [1.234956] input: CHESEN USB Keyboard as /devices/pci:00/:00:14.0/usb3/3-10/3-10:1.0/0003:0A81:0101.0001/input/input5 [1.236273] md/raid1:md126: active with 2 out of 2 mirrors [1.236797] md126: detected capacity change from 0 to 1000202043392 [1.246271] md: md126 switched to read-write mode. [1.246834] md: resync of RAID array md126 [1.247325] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. [1.247503] md126: p1 p2 p3 p4 [1.248269] md: using maximum available idle IO bandwidth (but not more than 20 KB/sec) for resync. [1.248774] md: using 128k window, over a total of 976759940k. Note that the pain of "resync upon every (re)boot" cannot even be a bit relieved thanks to bitmaps because mdadm does not support them for IMSM containers: $ sudo mdadm --grow --bitmap=internal /dev/md126 mdadm: Cannot add bitmaps to sub-arrays yet I also get this in syslog during boot when the individual drives are detected, but this seems to be harmless: May 30 17:26:07 wssrv1 systemd-udevd[608]: Process '/sbin/mdadm --incremental /dev/sdb --offroot' failed with exit code 1. May 30 17:26:07 wssrv1 systemd-udevd[608]: Process '/lib/udev/hdparm' failed with exit code 1. May 30 17:26:07 wssrv1 systemd-udevd[606]: Process '/sbin/mdadm --incremental /dev/sda --offroot' failed with exit code 1. May 30 17:26:07 wssrv1 systemd-udevd[606]: Process '/lib/udev/hdparm' failed with exit code 1. During a resync, `sudo sh -c 'echo idle > /sys/block/md126/md/sync_action'` actually stops it as expected, but it restarts immediately though nothing seems to have triggered it: May 30 18:17:02 wssrv1 kernel: [ 3106.826710] md: md126: resync interrupted. May 30 18:17:02 wssrv1 kernel: [ 3106.836320] md: checkpointing resync of md126. May 30 18:17:02 wssrv1 kernel: [ 3106.836623] md: resync of RAID array md126 May 30 18:17:02 wssrv1 kernel: [ 3106.836625] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. May 30 18:17:02 wssrv1 kernel: [ 3106.836626] md: using maximum available idle IO bandwidth (but not more than 20 KB/sec) for resync. May 30 18:17:02 wssrv1 kernel: [ 3106.836627] md: using 128k window, over a total of 976759940k. May 30 18:17:02 wssrv1 kernel: [ 3106.836628] md: resuming resync of md126 from checkpoint. May 30 18:17:02 wssrv1 mdadm[982]: RebuildStarted event detected on md device /dev/md/Volume0 I attach screenshots of the hanging shutdown log after a `sudo sh -c 'echo 8 > /proc/sys/kernel/printk'`. The second screenshot shows that the kernel has deadlocked in md_write_start(). Note that `sudo systemctl start debug-shell` is unusable on this machine at this point because Ctrl+Alt+F9 brings tty9 without any keyboard. [EDIT]But I can still switch back to tty1.[/EDIT] I have also
[Touch-packages] [Bug 1587142] Re: Shutdown hangs in md kworker after "Reached target Shutdown."
** Attachment added: "screenshot" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1587142/+attachment/4945763/+files/reached_target.png -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1587142 Title: Shutdown hangs in md kworker after "Reached target Shutdown." Status in systemd package in Ubuntu: Confirmed Bug description: I'm booting a fully patched 16.04 from an Intel Rapid Storage Technology enterprise RAID1 volume (ThinkServer TS140 with two SATA ST1000NM0033-9ZM drives, ext4 root partition, no LVM, UEFI mode). If the RAID volume is recovering or resyncing for whatever reason, then `sudo systemctl reboot` and `sudo systemctl poweroff` work fine (I had to `sudo systemctl --now disable lvm2-lvmetad lvm2-lvmpolld lvm2-monitor` in order to consistently get that). However, once the recovery/resync is complete and clean, the reboot and poweroff commands above hang forever after "Reached target Shutdown.". Note that issuing `sudo swapoff -a` beforehand (suggested in the bug #1464917) does not help. [EDIT]Actually, the shutdown also hangs from time to time during a resync. But I've never seen it succeed once the resync is complete.[/EDIT] Then, if the server has been forcibly restarted with the power button, the Intel Matrix Storage Manager indicates a "Normal" status for the RAID1 volume, but Ubuntu then resyncs the volume anyway: [1.223649] md: bind [1.228426] md: bind [1.230030] md: bind [1.230738] md: bind [1.232985] usbcore: registered new interface driver usbhid [1.233494] usbhid: USB HID core driver [1.234022] md: raid1 personality registered for level 1 [1.234876] md/raid1:md126: not clean -- starting background reconstruction [1.234956] input: CHESEN USB Keyboard as /devices/pci:00/:00:14.0/usb3/3-10/3-10:1.0/0003:0A81:0101.0001/input/input5 [1.236273] md/raid1:md126: active with 2 out of 2 mirrors [1.236797] md126: detected capacity change from 0 to 1000202043392 [1.246271] md: md126 switched to read-write mode. [1.246834] md: resync of RAID array md126 [1.247325] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. [1.247503] md126: p1 p2 p3 p4 [1.248269] md: using maximum available idle IO bandwidth (but not more than 20 KB/sec) for resync. [1.248774] md: using 128k window, over a total of 976759940k. Note that the pain of "resync upon every (re)boot" cannot even be a bit relieved thanks to bitmaps because mdadm does not support them for IMSM containers: $ sudo mdadm --grow --bitmap=internal /dev/md126 mdadm: Cannot add bitmaps to sub-arrays yet I also get this in syslog during boot when the individual drives are detected, but this seems to be harmless: May 30 17:26:07 wssrv1 systemd-udevd[608]: Process '/sbin/mdadm --incremental /dev/sdb --offroot' failed with exit code 1. May 30 17:26:07 wssrv1 systemd-udevd[608]: Process '/lib/udev/hdparm' failed with exit code 1. May 30 17:26:07 wssrv1 systemd-udevd[606]: Process '/sbin/mdadm --incremental /dev/sda --offroot' failed with exit code 1. May 30 17:26:07 wssrv1 systemd-udevd[606]: Process '/lib/udev/hdparm' failed with exit code 1. During a resync, `sudo sh -c 'echo idle > /sys/block/md126/md/sync_action'` actually stops it as expected, but it restarts immediately though nothing seems to have triggered it: May 30 18:17:02 wssrv1 kernel: [ 3106.826710] md: md126: resync interrupted. May 30 18:17:02 wssrv1 kernel: [ 3106.836320] md: checkpointing resync of md126. May 30 18:17:02 wssrv1 kernel: [ 3106.836623] md: resync of RAID array md126 May 30 18:17:02 wssrv1 kernel: [ 3106.836625] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. May 30 18:17:02 wssrv1 kernel: [ 3106.836626] md: using maximum available idle IO bandwidth (but not more than 20 KB/sec) for resync. May 30 18:17:02 wssrv1 kernel: [ 3106.836627] md: using 128k window, over a total of 976759940k. May 30 18:17:02 wssrv1 kernel: [ 3106.836628] md: resuming resync of md126 from checkpoint. May 30 18:17:02 wssrv1 mdadm[982]: RebuildStarted event detected on md device /dev/md/Volume0 I attach screenshots of the hanging shutdown log after a `sudo sh -c 'echo 8 > /proc/sys/kernel/printk'`. The second screenshot shows that the kernel has deadlocked in md_write_start(). Note that `sudo systemctl start debug-shell` is unusable on this machine at this point because Ctrl+Alt+F9 brings tty9 without any keyboard. [EDIT]But I can still switch back to tty1.[/EDIT] I have also tried with much lower values for vm.dirty_background_ratio and vm.dirty_ratio, but to no avail. Linux 4.6.0-040600-generic_4.6.0-040600.201605151930_amd64 from http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.6-yakkety/ did not help either. More information below:
[Touch-packages] [Bug 1587142] Re: Shutdown hangs in md kworker after "Reached target Shutdown."
confirmed on HP ProLiant ML10 Gen9, Xeon E3-1225 v5 Intel Raid Bios set to RAID1 (4 drives, 2xraid1) Shutdowen/reboot hangs (kworkers) and Raid is rebuilt on every "reset" Linux w11 4.4.0-93-generic #116-Ubuntu SMP Fri Aug 11 21:17:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Description:Ubuntu 16.04.3 LTS Release:16.04 Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] md124 : active raid1 sdc[1] sdd[0] 2930264064 blocks super external:/md125/0 [2/2] [UU] [=>...] resync = 27.1% (795895744/2930264196) finish=190.3min speed=186838K/sec md125 : inactive sdd[1](S) sdc[0](S) 4776 blocks super external:imsm md126 : active raid1 sda[1] sdb[0] 976759808 blocks super external:/md127/0 [2/2] [UU] [===>.] resync = 75.0% (733104704/976759940) finish=28.2min speed=143955K/sec md127 : inactive sda[1](S) sdb[0](S) 5288 blocks super external:imsm unused devices: -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1587142 Title: Shutdown hangs in md kworker after "Reached target Shutdown." Status in systemd package in Ubuntu: Confirmed Bug description: I'm booting a fully patched 16.04 from an Intel Rapid Storage Technology enterprise RAID1 volume (ThinkServer TS140 with two SATA ST1000NM0033-9ZM drives, ext4 root partition, no LVM, UEFI mode). If the RAID volume is recovering or resyncing for whatever reason, then `sudo systemctl reboot` and `sudo systemctl poweroff` work fine (I had to `sudo systemctl --now disable lvm2-lvmetad lvm2-lvmpolld lvm2-monitor` in order to consistently get that). However, once the recovery/resync is complete and clean, the reboot and poweroff commands above hang forever after "Reached target Shutdown.". Note that issuing `sudo swapoff -a` beforehand (suggested in the bug #1464917) does not help. [EDIT]Actually, the shutdown also hangs from time to time during a resync. But I've never seen it succeed once the resync is complete.[/EDIT] Then, if the server has been forcibly restarted with the power button, the Intel Matrix Storage Manager indicates a "Normal" status for the RAID1 volume, but Ubuntu then resyncs the volume anyway: [1.223649] md: bind [1.228426] md: bind [1.230030] md: bind [1.230738] md: bind [1.232985] usbcore: registered new interface driver usbhid [1.233494] usbhid: USB HID core driver [1.234022] md: raid1 personality registered for level 1 [1.234876] md/raid1:md126: not clean -- starting background reconstruction [1.234956] input: CHESEN USB Keyboard as /devices/pci:00/:00:14.0/usb3/3-10/3-10:1.0/0003:0A81:0101.0001/input/input5 [1.236273] md/raid1:md126: active with 2 out of 2 mirrors [1.236797] md126: detected capacity change from 0 to 1000202043392 [1.246271] md: md126 switched to read-write mode. [1.246834] md: resync of RAID array md126 [1.247325] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. [1.247503] md126: p1 p2 p3 p4 [1.248269] md: using maximum available idle IO bandwidth (but not more than 20 KB/sec) for resync. [1.248774] md: using 128k window, over a total of 976759940k. Note that the pain of "resync upon every (re)boot" cannot even be a bit relieved thanks to bitmaps because mdadm does not support them for IMSM containers: $ sudo mdadm --grow --bitmap=internal /dev/md126 mdadm: Cannot add bitmaps to sub-arrays yet I also get this in syslog during boot when the individual drives are detected, but this seems to be harmless: May 30 17:26:07 wssrv1 systemd-udevd[608]: Process '/sbin/mdadm --incremental /dev/sdb --offroot' failed with exit code 1. May 30 17:26:07 wssrv1 systemd-udevd[608]: Process '/lib/udev/hdparm' failed with exit code 1. May 30 17:26:07 wssrv1 systemd-udevd[606]: Process '/sbin/mdadm --incremental /dev/sda --offroot' failed with exit code 1. May 30 17:26:07 wssrv1 systemd-udevd[606]: Process '/lib/udev/hdparm' failed with exit code 1. During a resync, `sudo sh -c 'echo idle > /sys/block/md126/md/sync_action'` actually stops it as expected, but it restarts immediately though nothing seems to have triggered it: May 30 18:17:02 wssrv1 kernel: [ 3106.826710] md: md126: resync interrupted. May 30 18:17:02 wssrv1 kernel: [ 3106.836320] md: checkpointing resync of md126. May 30 18:17:02 wssrv1 kernel: [ 3106.836623] md: resync of RAID array md126 May 30 18:17:02 wssrv1 kernel: [ 3106.836625] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. May 30 18:17:02 wssrv1 kernel: [ 3106.836626] md: using maximum available idle IO bandwidth (but not more than 20 KB/sec) for resync. May 30 18:17:02 wssrv1 kernel: [ 3106.836627] md: using 128k window, over a total of 976759940k. May 30 18:17:02 wssrv1
[Touch-packages] [Bug 1414950] [NEW] Click on shutdown in the menu shuts down immediately and doesn't show restart or shutdown-dalogue
Public bug reported: Since a few days I can't shutdown my ubuntu system properly, because when I click in the menu in the right top corner, the system shuts down immediately. I don't get the dialogue with the two buttons shutdown or restart. ProblemType: Bug DistroRelease: Ubuntu 14.10 Package: unity 7.3.1+14.10.20141016-0ubuntu1 ProcVersionSignature: Ubuntu 3.16.0-29.39-generic 3.16.7-ckt2 Uname: Linux 3.16.0-29-generic x86_64 .tmp.unity.support.test.0: ApportVersion: 2.14.7-0ubuntu8.1 Architecture: amd64 CompizPlugins: [core,composite,opengl,compiztoolbox,decor,vpswitch,snap,mousepoll,resize,place,move,wall,grid,regex,imgpng,session,gnomecompat,animation,fade,staticswitcher,workarounds,scale,expo,ezoom,dbus] CompositorRunning: compiz CompositorUnredirectDriverBlacklist: '(nouveau|Intel).*Mesa 8.0' CompositorUnredirectFSW: true CurrentDesktop: Unity Date: Tue Jan 27 10:03:51 2015 DistUpgraded: 2014-12-06 14:14:53,260 DEBUG enabling apt cron job DistroCodename: utopic DistroVariant: ubuntu GraphicsCard: Intel Corporation 4th Gen Core Processor Integrated Graphics Controller [8086:0416] (rev 06) (prog-if 00 [VGA controller]) Subsystem: Lenovo Device [17aa:3978] InstallationDate: Installed on 2014-12-05 (52 days ago) InstallationMedia: Ubuntu 14.04.1 LTS Trusty Tahr - Release amd64 (20140722.2) MachineType: LENOVO 20378 ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.16.0-29-generic root=UUID=30e5a5bf-1a21-4005-9729-28e1bf5c54fd ro quiet splash vt.handoff=7 SourcePackage: unity UpgradeStatus: Upgraded to utopic on 2014-12-06 (51 days ago) dmi.bios.date: 05/27/2014 dmi.bios.vendor: LENOVO dmi.bios.version: 9ECN26WW(V1.09) dmi.board.asset.tag: No Asset Tag dmi.board.name: Lenovo Y50-70 dmi.board.vendor: LENOVO dmi.board.version: Not Defined dmi.chassis.asset.tag: No Asset Tag dmi.chassis.type: 10 dmi.chassis.vendor: LENOVO dmi.chassis.version: Lenovo Y50-70 dmi.modalias: dmi:bvnLENOVO:bvr9ECN26WW(V1.09):bd05/27/2014:svnLENOVO:pn20378:pvrLenovoY50-70:rvnLENOVO:rnLenovoY50-70:rvrNotDefined:cvnLENOVO:ct10:cvrLenovoY50-70: dmi.product.name: 20378 dmi.product.version: Lenovo Y50-70 dmi.sys.vendor: LENOVO version.compiz: compiz 1:0.9.12+14.10.20140918-0ubuntu1 version.ia32-libs: ia32-libs N/A version.libdrm2: libdrm2 2.4.59+git20150125.0c5aaeef-0ubuntu0ricotz~utopic version.libgl1-mesa-dri: libgl1-mesa-dri 10.5.0~git20150125.0aa31bf9-0ubuntu0ricotz~utopic version.libgl1-mesa-dri-experimental: libgl1-mesa-dri-experimental N/A version.libgl1-mesa-glx: libgl1-mesa-glx 10.5.0~git20150125.0aa31bf9-0ubuntu0ricotz~utopic version.xserver-xorg-core: xserver-xorg-core 2:1.16.1.901-1ubuntu1~utopic1 version.xserver-xorg-input-evdev: xserver-xorg-input-evdev 1:2.9.0-1ubuntu2 version.xserver-xorg-video-ati: xserver-xorg-video-ati 1:7.4.99+git20140806.fbf575cb-0ubuntu0sarvatt3 version.xserver-xorg-video-intel: xserver-xorg-video-intel 2:2.99.917+git20150126.b8d52b0e-0ubuntu0sarvatt~utopic version.xserver-xorg-video-nouveau: xserver-xorg-video-nouveau 1:1.0.11+git20141030.3fb97d78-0ubuntu0sarvatt xserver.bootTime: Tue Jan 27 09:55:58 2015 xserver.configfile: default xserver.errors: xserver.logfile: /var/log/Xorg.0.log xserver.outputs: product id 14061 vendor AUO xserver.version: 2:1.16.1.901-1ubuntu1~utopic1 ** Affects: unity (Ubuntu) Importance: Undecided Status: New ** Tags: amd64 apport-bug compiz-0.9 third-party-packages ubuntu utopic -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to unity in Ubuntu. https://bugs.launchpad.net/bugs/1414950 Title: Click on shutdown in the menu shuts down immediately and doesn't show restart or shutdown-dalogue Status in unity package in Ubuntu: New Bug description: Since a few days I can't shutdown my ubuntu system properly, because when I click in the menu in the right top corner, the system shuts down immediately. I don't get the dialogue with the two buttons shutdown or restart. ProblemType: Bug DistroRelease: Ubuntu 14.10 Package: unity 7.3.1+14.10.20141016-0ubuntu1 ProcVersionSignature: Ubuntu 3.16.0-29.39-generic 3.16.7-ckt2 Uname: Linux 3.16.0-29-generic x86_64 .tmp.unity.support.test.0: ApportVersion: 2.14.7-0ubuntu8.1 Architecture: amd64 CompizPlugins: [core,composite,opengl,compiztoolbox,decor,vpswitch,snap,mousepoll,resize,place,move,wall,grid,regex,imgpng,session,gnomecompat,animation,fade,staticswitcher,workarounds,scale,expo,ezoom,dbus] CompositorRunning: compiz CompositorUnredirectDriverBlacklist: '(nouveau|Intel).*Mesa 8.0' CompositorUnredirectFSW: true CurrentDesktop: Unity Date: Tue Jan 27 10:03:51 2015 DistUpgraded: 2014-12-06 14:14:53,260 DEBUG enabling apt cron job DistroCodename: utopic DistroVariant: ubuntu GraphicsCard: Intel Corporation 4th Gen Core Processor Integrated Graphics Controller [8086:0416] (rev 06) (prog-if 00
