Konrad Rzeszutek wrote: > On Thu, Apr 17, 2008 at 06:07:12AM -0400, Konrad Rzeszutek wrote: >>> It looks like the network is off but the session is still running. We >>> eventually get to the kernel shutoff here. Is your init script getting >>> run? If not then run it. If you left the session on on purpose then you >>> cannot turn the network off because the scsi layer will want to do its >>> shutdown when the kernel is stopped. >> Ah. Thanks for the explanation. The init script was run, but it didn't >> logoff of all the sessions (it would selectivly logoff instead of doing >> all of them). > > After I made sure that 'iscsiadm -m session -U all' was called during shutdown > a QA engineer here was able to make the 'iscsiadm' hang during this sequence. > > The result was that some of the iSCSI sessions did log-out while some other > did not, > and the machine hanged during the "Synchronizing SCSI cache for disk .."
I am not sure if you meant you are running iscsiadm -m session -U all and still hanging and if so where so to make sure you are not hitting the cache sync bug let me try to seperate all the shutdown bugs we have seen :) Notes: iscsiadm will not return until all sessions you asked to logout are stopped and the sync caches are completed, but iscsiadm ignores sessions marked as being used for boot. If iscsiadm is run during iscsi init script shutdown then if the network is down we will hang and no session should get logged off. If the network is up and iscsiadm is hanging then it is probably the cache sync iscsiadm hang caused by us using the delete sysfs file and expecting it not to return until the cache sync has completed. In this case some sessions could get logged out right and others could hang. It is a race so it is fun and not always predictable :) If you are hanging when the kernel shutsdown (when the driver model shutdown functions get called which for sd.c case will send the cache sync) then if your system is not setup we will hang on the cache sync because iscsid is not up to handle the connection error that results when the scsi eh fails. We can also hang if the network is up right at this time, but we experience a connection error or timeout or the target could drop connection for whatever reason it wants to. In this case we hit the same problem where we send a connection error to userspace but iscsid is not there. In older kernels the scsi command would timeout and in newer kernels iscsi_eh_cmd_timed_out will keep resetting the command timer. > My thought was that since the logic of the SCSI reset is to call first > 'iscsi_eh_abort', > then 'iscsi_eh_device_reset' and then finally 'iscsi_eh_host_reset' we could > in > iscsi_eh_host_reset check the tmf_state as so: > > > diff --git a/kernel/libiscsi.c b/kernel/libiscsi.c > index f8f4cf9..a5b5b37 100644 > --- a/kernel/libiscsi.c > +++ b/kernel/libiscsi.c > @@ -1163,7 +1163,8 @@ failed: > wait_event_interruptible(conn->ehwait, > session->state == ISCSI_STATE_TERMINATE || > session->state == ISCSI_STATE_LOGGED_IN || > - session->state == ISCSI_STATE_RECOVERY_FAILED); > + session->state == ISCSI_STATE_RECOVERY_FAILED > || > + conn->tmf_state == TMF_TIMEDOUT); > if (signal_pending(current)) > flush_signals(current); > > That does make it possible for the SCSI reset sequence to finish off and the > machine > reboots fine. It does not change the behavior during a normal run of iSCSI > (ie, unplugging > a cable for couple of seconds, minutes, etc). Are you sure about this? I do not think you tried all the timing scenarios or had IO running when testing. > > But I have this gnawing feeling that I missing something here ... > The problem with the patch is that if we return SUCCESS from iscsi_eh_host_reset then we tell the scsi layer that the problem is solved and to retry IO if it can. If we return FAILED then we tell the scsi layer we are kaput and it will fail IO. For a non shutdown run then we want to wait for replacement_timeout seconds for the session to come back. If the connection problem was detected after the command timer fired so iscsi_eh_cmd_timed_out cannot check the status of the connection then we send TMFs they will timeout and with your patch we immediately fail the IO. Or if there is a connection error while sending TMFs and this causes one to timeout then again with your patch we fail the IO too soon. Also for the shutdown case, it does not handle when the connection problem is detected before the command times out. In this case iscsi_eh_cmd_timed_out is going to keep asking scsi-ml to reset the command timer. My original solution was to call iscsi_block_session when iscsi_conn_failure/ISCSI_SUSPEND_BIT is run. We can do this in newer kernels, but for the compat modules this will not work because the driver model code used to use a mutex to loop over devices. The compat modules problem is probbably not a major issue since we are concerned with upstream first, but you just have to make sure that the unblock and blocks from userspace (stop and start conn calls) are in order. There was some bugs in the initial patch so it did not make it upstream. The other solution we were tyring is I thought netlink_broadcast would return something like ENOTCONN if iscsid was not there. If this was always true we could check if we got this error and assume that if we tried to tell userspace about something and it was not up then we could fast fail the eh path similar to how you did. I think in this case it would be ok to ignore the replacement timeout, because iscsid should always be up. If it is not then it is because it segfaulted and we are screwed not matter what, the user decided to restart it manually which is discouraged because we do not support it, or because it is not running during those boot and shutdown windows. And for this last there is only one solution which is fail. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~----------~----~----~----~------~----~------~--~---
