Long story short -> https://goo.gl/vPyh8C
Basically the block device enqueues the last request (a SYNC scsi command coming from sd_shutdown) for every scsi device there is on the system. Unfortunately, since the OS is shutting down, in between the block request and its execution, we have userland (systemd) killing iscsid, without proper logout, and/or removing the network. What happens next is that the mid layer (SCSI) tries to deliver the request through the transport layer (iscsi_tcp_sw) but it fails since the transport layer checks the session status and finds out that the session is not in LOGIN state. The default behaviour of the transport layer (iscsi_tcp_sw) in such situation is to tell the mid-layer to keep resetting the request timeout timer while it tries to recover (something that will never happen because the network is gone). Changing that default behavior to state that the scsi command was NOT handled by the transport layer (iscsi_tcp_sw) implies in making the scsi timeout function to try to "abort" the scsi command, which also creates other commands that will timeout because of the transport layer. Best scenario so far was to change BLK_EH_NOT_HANDLED for BLK_EH_HANDLED in the scsi_times_out function and make the kernel to be able to shutdown. By doing that, I'm confirming to block device something that DID NOT happen, meaning that the command never left the transport layer. This might be ONE of possible ways to fix this: I can mark in the transport layer that I have timed-out DURING the shutdown procedure, cancelling all the block device requests without having to invoke the scsi error handling mechanism, generating more traffic in transport layer (what would also cause more timeouts, causing a loop in the problematic sequence). Anyway, I'll get back to this next week and hopefully identify best course of action. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets Status in linux package in Ubuntu: In Progress Status in open-iscsi package in Ubuntu: In Progress Status in linux source package in Xenial: In Progress Status in open-iscsi source package in Xenial: In Progress Status in linux source package in Zesty: In Progress Status in open-iscsi source package in Zesty: In Progress Status in linux source package in Artful: In Progress Status in open-iscsi source package in Artful: In Progress Bug description: I have 4 servers running the latest 16.04 updates from the development branch (as of right now). Each server is connected to NetApp storage using iscsi software initiator. There are a total of 56 volumes spread across two NetApp arrays. Each volume has 4 paths available to it which are being managed by device mapper. While logged into the iscsi sessions all I have to do is reboot the server and I get a hang. I see a message that says: "Reached target Shutdown" followed by "systemd-shutdown[1]: Failed to finalize DM devices, ignoring" and then I see 8 lines that say: "connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection2:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection3:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection5:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection6:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection7:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" "connection8:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***" NOTE: the actual values of the *'s differ for each line above. This seems like a bug somewhere but I am unaware of any additional logging that I could turn on to pinpoint the problem. Note I also have similar setups that are not doing iscsi and they don't have this problem. Here is a screenshot of what I see on the shell when I try to reboot: (https://launchpadlibrarian.net/291303059/Screenshot.jpg) This is being tracked in NetApp bug tracker CQ number 860251. If I log out of all iscsi sessions before rebooting then I do not experience the hang: iscsiadm -m node -U all We are wondering if this could be some kind of shutdown ordering problem. Like the network devices have already disappeared and then iscsi tries to perform some operation (hence the ping timeouts). To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1569925/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
