Re: NFS Freezer and stuck tasks
Hey, Jeff. On Fri, May 01, 2015 at 07:17:41PM -0400, Jeff Layton wrote: > > Sorry for the noise, and self-reply.. Looks like there's additional context > > here: http://marc.info/?t=13676151217&r=1&w=2 > > > > Due to a number of locking problems the answer to this problem is likely to > > be "don't do that" for now. Unfortunately, cgroup freezer is currently inherently broken. As it currently stands, the situation is - if it works for certain use cases, great; otherwise, don't do that. ... > My memory is vague, but Tejun (cc'ed) and I discussed this a couple of > years or so ago and the tentative idea at the time was to teach the > NFS and RPC code to return a particular error akin to ERESTARTSYS > (EFREEZE?) when a freeze event comes in and we haven't yet sent an RPC > call. The idea is that freezing should be essentially identical to how SIGSTOP is handled when viewed from kernel side. > The idea was to teach the ptrace layer to watch for this error and > freeze at that point and then to reissue the syscall after resume. All > of that's a non-trivial task though, as knowledge of this would need to > be plumbed all the way through the stack down to the RPC layer. So, if nfs can abort and return to userland on sigpending, the task will be able to finish quckly; otherwise, it'd have to wait till nfs finishes. Thanks. -- tejun -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: NFS Freezer and stuck tasks
On Fri, 1 May 2015 17:10:34 -0400 (EDT) Benjamin Coddington wrote: > On Fri, 1 May 2015, Benjamin Coddington wrote: > > > On Wed, 4 Mar 2015, Shawn Bohrer wrote: > > > > > Hello, > > > > > > We're using the Linux cgroup Freezer on some machines that use NFS and > > > have run into what appears to be a bug where frozen tasks are blocking > > > running tasks and preventing them from completing. On one of our > > > machines which happens to be running an older 3.10.46 kernel we have > > > frozen some of the tasks on the system using the cgroup Freezer. We > > > also have a separate set of tasks which are NOT frozen which are stuck > > > trying to open some files on NFS. > > > > > > Looking at the frozen tasks there are several that have the following > > > stack: > > > > > > [] rpc_wait_bit_killable+0x35/0x80 > > > [] __rpc_wait_for_completion_task+0x2d/0x30 > > > [] nfs4_run_open_task+0x11d/0x170 > > > [] _nfs4_open_and_get_state+0x53/0x260 > > > [] nfs4_do_open+0x121/0x400 > > > [] nfs4_atomic_open+0x31/0x50 > > > [] nfs4_file_open+0xac/0x180 > > > [] do_dentry_open.isra.19+0x1ee/0x280 > > > [] finish_open+0x1e/0x30 > > > [] do_last.isra.64+0x2c2/0xc40 > > > [] path_openat.isra.65+0x2c9/0x490 > > > [] do_filp_open+0x38/0x80 > > > [] do_sys_open+0xe4/0x1c0 > > > [] SyS_open+0x1e/0x20 > > > [] system_call_fastpath+0x16/0x1b > > > [] 0x > > > > > > Here it looks like we are waiting in a wait queue inside > > > rpc_wait_bit_killable() for RPC_TASK_ACTIVE. > > > > > > And there is a single task with a stack that looks like the following: > > > > > > [] __refrigerator+0x55/0x150 > > > [] rpc_wait_bit_killable+0x66/0x80 > > > [] __rpc_wait_for_completion_task+0x2d/0x30 > > > [] nfs4_run_open_task+0x11d/0x170 > > > [] _nfs4_open_and_get_state+0x53/0x260 > > > [] nfs4_do_open+0x121/0x400 > > > [] nfs4_atomic_open+0x31/0x50 > > > [] nfs4_file_open+0xac/0x180 > > > [] do_dentry_open.isra.19+0x1ee/0x280 > > > [] finish_open+0x1e/0x30 > > > [] do_last.isra.64+0x2c2/0xc40 > > > [] path_openat.isra.65+0x2c9/0x490 > > > [] do_filp_open+0x38/0x80 > > > [] do_sys_open+0xe4/0x1c0 > > > [] SyS_open+0x1e/0x20 > > > [] system_call_fastpath+0x16/0x1b > > > [] 0x > > > > > > This looks similar but the different offset into > > > rpc_wait_bit_killable() shows that we have returned from the > > > schedule() call in freezable_schedule() and are now blocked in > > > __refrigerator() inside freezer_count() > > > > > > Similarly if you look at the tasks that are NOT frozen but are stuck > > > opening a NFS file, they also have the following stack showing they are > > > waiting in the wait queue for RPC_TASK_ACTIVE. > > > > > > [] rpc_wait_bit_killable+0x35/0x80 > > > [] __rpc_wait_for_completion_task+0x2d/0x30 > > > [] nfs4_run_open_task+0x11d/0x170 > > > [] _nfs4_open_and_get_state+0x53/0x260 > > > [] nfs4_do_open+0x121/0x400 > > > [] nfs4_atomic_open+0x31/0x50 > > > [] nfs4_file_open+0xac/0x180 > > > [] do_dentry_open.isra.19+0x1ee/0x280 > > > [] finish_open+0x1e/0x30 > > > [] do_last.isra.64+0x2c2/0xc40 > > > [] path_openat.isra.65+0x2c9/0x490 > > > [] do_filp_open+0x38/0x80 > > > [] do_sys_open+0xe4/0x1c0 > > > [] SyS_open+0x1e/0x20 > > > [] system_call_fastpath+0x16/0x1b > > > [] 0x > > > > > > We have hit this a couple of times now and know that if we THAW all of > > > the frozen tasks that running tasks will unwedge and finish. > > > > > > Additionally we have also tried thawing the single task that is frozen > > > in __refrigerator() inside rpc_wait_bit_killable(). This usually > > > results in different frozen task entering the __refrigerator() state > > > inside rpc_wait_bit_killable(). It looks like each one of those tasks > > > must wake up another letting it progress. Again if you thaw enough of > > > the frozen tasks eventually everything unwedges and everything > > > completes. > > > > > > I've looked through the 3.10 stable patches since 3.10.46 and don't > > > see anything that looks like it addresses this. Does anyone have any > > > idea what might be going on here, and what the fix might be? > > > > > > Thanks, > > > Shawn > > > > Hi Shawn, just started looking at this myself, and as Frank Sorensen points > > out in https://bugzilla.redhat.com/show_bug.cgi?id=1209143 the problem is > > that a task takes the xprt lock and then ends up in the refrigerator > > effectively blocking other tasks from proceeding. > > > > Jeff, any suggestions on how to proceed here? > > Sorry for the noise, and self-reply.. Looks like there's additional context > here: http://marc.info/?t=13676151217&r=1&w=2 > > Due to a number of locking problems the answer to this problem is likely to > be "don't do that" for now. > > Ben Yeah, that's definitely the answer for now. NFS and the freezer basically cooperate if you are freezing the whole system, but freezing some tasks and not others is fraught with peril. The problem is that by the time you get a freeze "signal" you might be v
Re: NFS Freezer and stuck tasks
On Fri, May 01, 2015 at 05:10:34PM -0400, Benjamin Coddington wrote: > On Fri, 1 May 2015, Benjamin Coddington wrote: > > > On Wed, 4 Mar 2015, Shawn Bohrer wrote: > > > > > Hello, > > > > > > We're using the Linux cgroup Freezer on some machines that use NFS and > > > have run into what appears to be a bug where frozen tasks are blocking > > > running tasks and preventing them from completing. On one of our > > > machines which happens to be running an older 3.10.46 kernel we have > > > frozen some of the tasks on the system using the cgroup Freezer. We > > > also have a separate set of tasks which are NOT frozen which are stuck > > > trying to open some files on NFS. > > > > > > Looking at the frozen tasks there are several that have the following > > > stack: > > > > > > [] rpc_wait_bit_killable+0x35/0x80 > > > [] __rpc_wait_for_completion_task+0x2d/0x30 > > > [] nfs4_run_open_task+0x11d/0x170 > > > [] _nfs4_open_and_get_state+0x53/0x260 > > > [] nfs4_do_open+0x121/0x400 > > > [] nfs4_atomic_open+0x31/0x50 > > > [] nfs4_file_open+0xac/0x180 > > > [] do_dentry_open.isra.19+0x1ee/0x280 > > > [] finish_open+0x1e/0x30 > > > [] do_last.isra.64+0x2c2/0xc40 > > > [] path_openat.isra.65+0x2c9/0x490 > > > [] do_filp_open+0x38/0x80 > > > [] do_sys_open+0xe4/0x1c0 > > > [] SyS_open+0x1e/0x20 > > > [] system_call_fastpath+0x16/0x1b > > > [] 0x > > > > > > Here it looks like we are waiting in a wait queue inside > > > rpc_wait_bit_killable() for RPC_TASK_ACTIVE. > > > > > > And there is a single task with a stack that looks like the following: > > > > > > [] __refrigerator+0x55/0x150 > > > [] rpc_wait_bit_killable+0x66/0x80 > > > [] __rpc_wait_for_completion_task+0x2d/0x30 > > > [] nfs4_run_open_task+0x11d/0x170 > > > [] _nfs4_open_and_get_state+0x53/0x260 > > > [] nfs4_do_open+0x121/0x400 > > > [] nfs4_atomic_open+0x31/0x50 > > > [] nfs4_file_open+0xac/0x180 > > > [] do_dentry_open.isra.19+0x1ee/0x280 > > > [] finish_open+0x1e/0x30 > > > [] do_last.isra.64+0x2c2/0xc40 > > > [] path_openat.isra.65+0x2c9/0x490 > > > [] do_filp_open+0x38/0x80 > > > [] do_sys_open+0xe4/0x1c0 > > > [] SyS_open+0x1e/0x20 > > > [] system_call_fastpath+0x16/0x1b > > > [] 0x > > > > > > This looks similar but the different offset into > > > rpc_wait_bit_killable() shows that we have returned from the > > > schedule() call in freezable_schedule() and are now blocked in > > > __refrigerator() inside freezer_count() > > > > > > Similarly if you look at the tasks that are NOT frozen but are stuck > > > opening a NFS file, they also have the following stack showing they are > > > waiting in the wait queue for RPC_TASK_ACTIVE. > > > > > > [] rpc_wait_bit_killable+0x35/0x80 > > > [] __rpc_wait_for_completion_task+0x2d/0x30 > > > [] nfs4_run_open_task+0x11d/0x170 > > > [] _nfs4_open_and_get_state+0x53/0x260 > > > [] nfs4_do_open+0x121/0x400 > > > [] nfs4_atomic_open+0x31/0x50 > > > [] nfs4_file_open+0xac/0x180 > > > [] do_dentry_open.isra.19+0x1ee/0x280 > > > [] finish_open+0x1e/0x30 > > > [] do_last.isra.64+0x2c2/0xc40 > > > [] path_openat.isra.65+0x2c9/0x490 > > > [] do_filp_open+0x38/0x80 > > > [] do_sys_open+0xe4/0x1c0 > > > [] SyS_open+0x1e/0x20 > > > [] system_call_fastpath+0x16/0x1b > > > [] 0x > > > > > > We have hit this a couple of times now and know that if we THAW all of > > > the frozen tasks that running tasks will unwedge and finish. > > > > > > Additionally we have also tried thawing the single task that is frozen > > > in __refrigerator() inside rpc_wait_bit_killable(). This usually > > > results in different frozen task entering the __refrigerator() state > > > inside rpc_wait_bit_killable(). It looks like each one of those tasks > > > must wake up another letting it progress. Again if you thaw enough of > > > the frozen tasks eventually everything unwedges and everything > > > completes. > > > > > > I've looked through the 3.10 stable patches since 3.10.46 and don't > > > see anything that looks like it addresses this. Does anyone have any > > > idea what might be going on here, and what the fix might be? > > > > > > Thanks, > > > Shawn > > > > Hi Shawn, just started looking at this myself, and as Frank Sorensen points > > out in https://bugzilla.redhat.com/show_bug.cgi?id=1209143 the problem is > > that a task takes the xprt lock and then ends up in the refrigerator > > effectively blocking other tasks from proceeding. > > > > Jeff, any suggestions on how to proceed here? > > Sorry for the noise, and self-reply.. Looks like there's additional context > here: http://marc.info/?t=13676151217&r=1&w=2 > > Due to a number of locking problems the answer to this problem is likely to > be "don't do that" for now. Sorry I found the NFS + Freezer is broken threads and probably should have replied to myself. We are now using SIGSTOP/SIGCONT with a brief freeze to send the signals without race conditions. With that said it would be nice if these locking issues were eventually
Re: NFS Freezer and stuck tasks
On Fri, 1 May 2015, Benjamin Coddington wrote: > On Wed, 4 Mar 2015, Shawn Bohrer wrote: > > > Hello, > > > > We're using the Linux cgroup Freezer on some machines that use NFS and > > have run into what appears to be a bug where frozen tasks are blocking > > running tasks and preventing them from completing. On one of our > > machines which happens to be running an older 3.10.46 kernel we have > > frozen some of the tasks on the system using the cgroup Freezer. We > > also have a separate set of tasks which are NOT frozen which are stuck > > trying to open some files on NFS. > > > > Looking at the frozen tasks there are several that have the following > > stack: > > > > [] rpc_wait_bit_killable+0x35/0x80 > > [] __rpc_wait_for_completion_task+0x2d/0x30 > > [] nfs4_run_open_task+0x11d/0x170 > > [] _nfs4_open_and_get_state+0x53/0x260 > > [] nfs4_do_open+0x121/0x400 > > [] nfs4_atomic_open+0x31/0x50 > > [] nfs4_file_open+0xac/0x180 > > [] do_dentry_open.isra.19+0x1ee/0x280 > > [] finish_open+0x1e/0x30 > > [] do_last.isra.64+0x2c2/0xc40 > > [] path_openat.isra.65+0x2c9/0x490 > > [] do_filp_open+0x38/0x80 > > [] do_sys_open+0xe4/0x1c0 > > [] SyS_open+0x1e/0x20 > > [] system_call_fastpath+0x16/0x1b > > [] 0x > > > > Here it looks like we are waiting in a wait queue inside > > rpc_wait_bit_killable() for RPC_TASK_ACTIVE. > > > > And there is a single task with a stack that looks like the following: > > > > [] __refrigerator+0x55/0x150 > > [] rpc_wait_bit_killable+0x66/0x80 > > [] __rpc_wait_for_completion_task+0x2d/0x30 > > [] nfs4_run_open_task+0x11d/0x170 > > [] _nfs4_open_and_get_state+0x53/0x260 > > [] nfs4_do_open+0x121/0x400 > > [] nfs4_atomic_open+0x31/0x50 > > [] nfs4_file_open+0xac/0x180 > > [] do_dentry_open.isra.19+0x1ee/0x280 > > [] finish_open+0x1e/0x30 > > [] do_last.isra.64+0x2c2/0xc40 > > [] path_openat.isra.65+0x2c9/0x490 > > [] do_filp_open+0x38/0x80 > > [] do_sys_open+0xe4/0x1c0 > > [] SyS_open+0x1e/0x20 > > [] system_call_fastpath+0x16/0x1b > > [] 0x > > > > This looks similar but the different offset into > > rpc_wait_bit_killable() shows that we have returned from the > > schedule() call in freezable_schedule() and are now blocked in > > __refrigerator() inside freezer_count() > > > > Similarly if you look at the tasks that are NOT frozen but are stuck > > opening a NFS file, they also have the following stack showing they are > > waiting in the wait queue for RPC_TASK_ACTIVE. > > > > [] rpc_wait_bit_killable+0x35/0x80 > > [] __rpc_wait_for_completion_task+0x2d/0x30 > > [] nfs4_run_open_task+0x11d/0x170 > > [] _nfs4_open_and_get_state+0x53/0x260 > > [] nfs4_do_open+0x121/0x400 > > [] nfs4_atomic_open+0x31/0x50 > > [] nfs4_file_open+0xac/0x180 > > [] do_dentry_open.isra.19+0x1ee/0x280 > > [] finish_open+0x1e/0x30 > > [] do_last.isra.64+0x2c2/0xc40 > > [] path_openat.isra.65+0x2c9/0x490 > > [] do_filp_open+0x38/0x80 > > [] do_sys_open+0xe4/0x1c0 > > [] SyS_open+0x1e/0x20 > > [] system_call_fastpath+0x16/0x1b > > [] 0x > > > > We have hit this a couple of times now and know that if we THAW all of > > the frozen tasks that running tasks will unwedge and finish. > > > > Additionally we have also tried thawing the single task that is frozen > > in __refrigerator() inside rpc_wait_bit_killable(). This usually > > results in different frozen task entering the __refrigerator() state > > inside rpc_wait_bit_killable(). It looks like each one of those tasks > > must wake up another letting it progress. Again if you thaw enough of > > the frozen tasks eventually everything unwedges and everything > > completes. > > > > I've looked through the 3.10 stable patches since 3.10.46 and don't > > see anything that looks like it addresses this. Does anyone have any > > idea what might be going on here, and what the fix might be? > > > > Thanks, > > Shawn > > Hi Shawn, just started looking at this myself, and as Frank Sorensen points > out in https://bugzilla.redhat.com/show_bug.cgi?id=1209143 the problem is > that a task takes the xprt lock and then ends up in the refrigerator > effectively blocking other tasks from proceeding. > > Jeff, any suggestions on how to proceed here? Sorry for the noise, and self-reply.. Looks like there's additional context here: http://marc.info/?t=13676151217&r=1&w=2 Due to a number of locking problems the answer to this problem is likely to be "don't do that" for now. Ben -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: NFS Freezer and stuck tasks
On Wed, 4 Mar 2015, Shawn Bohrer wrote: > Hello, > > We're using the Linux cgroup Freezer on some machines that use NFS and > have run into what appears to be a bug where frozen tasks are blocking > running tasks and preventing them from completing. On one of our > machines which happens to be running an older 3.10.46 kernel we have > frozen some of the tasks on the system using the cgroup Freezer. We > also have a separate set of tasks which are NOT frozen which are stuck > trying to open some files on NFS. > > Looking at the frozen tasks there are several that have the following > stack: > > [] rpc_wait_bit_killable+0x35/0x80 > [] __rpc_wait_for_completion_task+0x2d/0x30 > [] nfs4_run_open_task+0x11d/0x170 > [] _nfs4_open_and_get_state+0x53/0x260 > [] nfs4_do_open+0x121/0x400 > [] nfs4_atomic_open+0x31/0x50 > [] nfs4_file_open+0xac/0x180 > [] do_dentry_open.isra.19+0x1ee/0x280 > [] finish_open+0x1e/0x30 > [] do_last.isra.64+0x2c2/0xc40 > [] path_openat.isra.65+0x2c9/0x490 > [] do_filp_open+0x38/0x80 > [] do_sys_open+0xe4/0x1c0 > [] SyS_open+0x1e/0x20 > [] system_call_fastpath+0x16/0x1b > [] 0x > > Here it looks like we are waiting in a wait queue inside > rpc_wait_bit_killable() for RPC_TASK_ACTIVE. > > And there is a single task with a stack that looks like the following: > > [] __refrigerator+0x55/0x150 > [] rpc_wait_bit_killable+0x66/0x80 > [] __rpc_wait_for_completion_task+0x2d/0x30 > [] nfs4_run_open_task+0x11d/0x170 > [] _nfs4_open_and_get_state+0x53/0x260 > [] nfs4_do_open+0x121/0x400 > [] nfs4_atomic_open+0x31/0x50 > [] nfs4_file_open+0xac/0x180 > [] do_dentry_open.isra.19+0x1ee/0x280 > [] finish_open+0x1e/0x30 > [] do_last.isra.64+0x2c2/0xc40 > [] path_openat.isra.65+0x2c9/0x490 > [] do_filp_open+0x38/0x80 > [] do_sys_open+0xe4/0x1c0 > [] SyS_open+0x1e/0x20 > [] system_call_fastpath+0x16/0x1b > [] 0x > > This looks similar but the different offset into > rpc_wait_bit_killable() shows that we have returned from the > schedule() call in freezable_schedule() and are now blocked in > __refrigerator() inside freezer_count() > > Similarly if you look at the tasks that are NOT frozen but are stuck > opening a NFS file, they also have the following stack showing they are > waiting in the wait queue for RPC_TASK_ACTIVE. > > [] rpc_wait_bit_killable+0x35/0x80 > [] __rpc_wait_for_completion_task+0x2d/0x30 > [] nfs4_run_open_task+0x11d/0x170 > [] _nfs4_open_and_get_state+0x53/0x260 > [] nfs4_do_open+0x121/0x400 > [] nfs4_atomic_open+0x31/0x50 > [] nfs4_file_open+0xac/0x180 > [] do_dentry_open.isra.19+0x1ee/0x280 > [] finish_open+0x1e/0x30 > [] do_last.isra.64+0x2c2/0xc40 > [] path_openat.isra.65+0x2c9/0x490 > [] do_filp_open+0x38/0x80 > [] do_sys_open+0xe4/0x1c0 > [] SyS_open+0x1e/0x20 > [] system_call_fastpath+0x16/0x1b > [] 0x > > We have hit this a couple of times now and know that if we THAW all of > the frozen tasks that running tasks will unwedge and finish. > > Additionally we have also tried thawing the single task that is frozen > in __refrigerator() inside rpc_wait_bit_killable(). This usually > results in different frozen task entering the __refrigerator() state > inside rpc_wait_bit_killable(). It looks like each one of those tasks > must wake up another letting it progress. Again if you thaw enough of > the frozen tasks eventually everything unwedges and everything > completes. > > I've looked through the 3.10 stable patches since 3.10.46 and don't > see anything that looks like it addresses this. Does anyone have any > idea what might be going on here, and what the fix might be? > > Thanks, > Shawn Hi Shawn, just started looking at this myself, and as Frank Sorensen points out in https://bugzilla.redhat.com/show_bug.cgi?id=1209143 the problem is that a task takes the xprt lock and then ends up in the refrigerator effectively blocking other tasks from proceeding. Jeff, any suggestions on how to proceed here? Ben -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
NFS Freezer and stuck tasks
Hello, We're using the Linux cgroup Freezer on some machines that use NFS and have run into what appears to be a bug where frozen tasks are blocking running tasks and preventing them from completing. On one of our machines which happens to be running an older 3.10.46 kernel we have frozen some of the tasks on the system using the cgroup Freezer. We also have a separate set of tasks which are NOT frozen which are stuck trying to open some files on NFS. Looking at the frozen tasks there are several that have the following stack: [] rpc_wait_bit_killable+0x35/0x80 [] __rpc_wait_for_completion_task+0x2d/0x30 [] nfs4_run_open_task+0x11d/0x170 [] _nfs4_open_and_get_state+0x53/0x260 [] nfs4_do_open+0x121/0x400 [] nfs4_atomic_open+0x31/0x50 [] nfs4_file_open+0xac/0x180 [] do_dentry_open.isra.19+0x1ee/0x280 [] finish_open+0x1e/0x30 [] do_last.isra.64+0x2c2/0xc40 [] path_openat.isra.65+0x2c9/0x490 [] do_filp_open+0x38/0x80 [] do_sys_open+0xe4/0x1c0 [] SyS_open+0x1e/0x20 [] system_call_fastpath+0x16/0x1b [] 0x Here it looks like we are waiting in a wait queue inside rpc_wait_bit_killable() for RPC_TASK_ACTIVE. And there is a single task with a stack that looks like the following: [] __refrigerator+0x55/0x150 [] rpc_wait_bit_killable+0x66/0x80 [] __rpc_wait_for_completion_task+0x2d/0x30 [] nfs4_run_open_task+0x11d/0x170 [] _nfs4_open_and_get_state+0x53/0x260 [] nfs4_do_open+0x121/0x400 [] nfs4_atomic_open+0x31/0x50 [] nfs4_file_open+0xac/0x180 [] do_dentry_open.isra.19+0x1ee/0x280 [] finish_open+0x1e/0x30 [] do_last.isra.64+0x2c2/0xc40 [] path_openat.isra.65+0x2c9/0x490 [] do_filp_open+0x38/0x80 [] do_sys_open+0xe4/0x1c0 [] SyS_open+0x1e/0x20 [] system_call_fastpath+0x16/0x1b [] 0x This looks similar but the different offset into rpc_wait_bit_killable() shows that we have returned from the schedule() call in freezable_schedule() and are now blocked in __refrigerator() inside freezer_count() Similarly if you look at the tasks that are NOT frozen but are stuck opening a NFS file, they also have the following stack showing they are waiting in the wait queue for RPC_TASK_ACTIVE. [] rpc_wait_bit_killable+0x35/0x80 [] __rpc_wait_for_completion_task+0x2d/0x30 [] nfs4_run_open_task+0x11d/0x170 [] _nfs4_open_and_get_state+0x53/0x260 [] nfs4_do_open+0x121/0x400 [] nfs4_atomic_open+0x31/0x50 [] nfs4_file_open+0xac/0x180 [] do_dentry_open.isra.19+0x1ee/0x280 [] finish_open+0x1e/0x30 [] do_last.isra.64+0x2c2/0xc40 [] path_openat.isra.65+0x2c9/0x490 [] do_filp_open+0x38/0x80 [] do_sys_open+0xe4/0x1c0 [] SyS_open+0x1e/0x20 [] system_call_fastpath+0x16/0x1b [] 0x We have hit this a couple of times now and know that if we THAW all of the frozen tasks that running tasks will unwedge and finish. Additionally we have also tried thawing the single task that is frozen in __refrigerator() inside rpc_wait_bit_killable(). This usually results in different frozen task entering the __refrigerator() state inside rpc_wait_bit_killable(). It looks like each one of those tasks must wake up another letting it progress. Again if you thaw enough of the frozen tasks eventually everything unwedges and everything completes. I've looked through the 3.10 stable patches since 3.10.46 and don't see anything that looks like it addresses this. Does anyone have any idea what might be going on here, and what the fix might be? Thanks, Shawn -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
