Okay, besides all crap I posted - which are definitely right and will possibly be addressed in other bugs, and also show other problems that could make systemd to hang because of iscsi disks - I was able to cause the same symptom of this bug:
[ OK ] Stopped LSB: ebtables ruleset management. [ OK ] Stopped Login to default iSCSI targets. Stopping iSCSI initiator daemon (iscsid)... [ OK ] Stopped iSCSI initiator daemon (iscsid). [ OK ] Stopped target Network is Online. [ OK ] Stopped target Network. Stopping Raise network interfaces... Stopping ifup for bond0.20... [ OK ] Stopped ifup for bond0.20. Stopping ifup for bond0.10... [ OK ] Stopped ifup for bond0.10. Stopping ifup for bond0... [ OK ] Stopped ifup for bond0. Stopping ifup for internal... [ OK ] Stopped ifup for internal. [ OK ] Stopped Raise network interfaces. [ OK ] Stopped target Network (Pre). [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Load Kernel Modules. [ OK ] Stopped target Local File Systems. Unmounting /run/cgmanager/fs... Unmounting /run/user/1000... Unmounting /xfs... Unmounting /ext4... [ OK ] Unmounted /run/cgmanager/fs. [ OK ] Unmounted /run/user/1000. [ 522.972114] connection3:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4295020535, last ping 4295021788, now 4295023040 [ 522.974085] connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4295020535, last ping 4295021788, now 4295023040 [ 522.975905] connection2:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4295020535, last ping 4295021788, now 4295023040 [ 522.976087] connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4295020535, last ping 4295021788, now 4295023040 [ 720.156456] INFO: task umount:11860 blocked for more than 120 seconds. [ 720.157477] Not tainted 4.4.0-93-generic #116-Ubuntu [ 720.158282] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 720.159695] INFO: task umount:11861 blocked for more than 120 seconds. [ 720.160680] Not tainted 4.4.0-93-generic #116-Ubuntu [ 720.161587] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 733.180181] sd 2:0:0:1: timing out command, waited 180s [ 733.181191] blk_update_request: I/O error, dev sdc, sector 0 [ 733.182164] sd 4:0:0:1: timing out command, waited 180s [ 733.183084] blk_update_request: I/O error, dev sdb, sector 0 [ OK ] Unmounted /ext4. [ OK ] Unmounted /xfs. [ OK ] Reached target Unmount All Filesystems. [ OK ] Stopped target Local File Systems (Pre). [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Remount Root and Kernel File Systems. [ OK ] Reached target Shutdown. [ 805.018886] systemd-shutdown[1]: Failed to finalize DM devices, ignoring <system hangs forever> And that is likely because I added 2 paths to each of my nics - using multipath, likely unrelated - and configured bonding + vlan within "ifupdown" - likely what causes this. After that change, I was able to make systemd to freeze in the same (likely) output as of this bug (and the one I'm taking care of). I'll investigate further tomorrow, but this can be related to ifupdown and its locking mechanism for configuring/de- configuring bonds/vlans/slaves interfaces. Matt, If not too much trouble to you, could you share the /etc/network/interfaces you were using ? You presented yourself and I was so focused in working this case that I forgot to mention: I'm one of the Sustaining Engineering Tech Leads @ Canonical. Usually those type of "partnership" product enablements/fixes are done 4 hands in other channels, that is why I felt lost in the beginning about your role here, I'm sorry if that bothered you. Now that I was to reproduce, I won't bother much, just want to compare your "interfaces" file with the ones I have (facing similar problem). Tks -Rafael -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets Status in systemd package in Ubuntu: Confirmed Status in systemd source package in Xenial: In Progress Bug description: I have 4 servers running the latest 16.04 updates from the development branch (as of right now). Each server is connected to NetApp storage using iscsi software initiator. There are a total of 56 volumes spread across two NetApp arrays. Each volume has 4 paths available to it which are being managed by device mapper. While logged into the iscsi sessions all I have to do is reboot the server and I get a hang. I see a message that says: "Reached target Shutdown" followed by "systemd-shutdown[1]: Failed to finalize DM devices, ignoring" and then I see 8 lines that say: "connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection2:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection3:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection5:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection6:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection7:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection8:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" NOTE: the actual values of the *'s differ for each line above. This seems like a bug somewhere but I am unaware of any additional logging that I could turn on to pinpoint the problem. Note I also have similar setups that are not doing iscsi and they don't have this problem. Here is a screenshot of what I see on the shell when I try to reboot: (https://launchpadlibrarian.net/291303059/Screenshot.jpg) This is being tracked in NetApp bug tracker CQ number 860251. If I log out of all iscsi sessions before rebooting then I do not experience the hang: iscsiadm -m node -U all We are wondering if this could be some kind of shutdown ordering problem. Like the network devices have already disappeared and then iscsi tries to perform some operation (hence the ping timeouts). To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp