Re: Deadlock on poweroff
On Sun, Oct 07, 2012 at 09:03:11AM -0700, Paul E. McKenney wrote: On Sun, Oct 07, 2012 at 05:47:11AM +0300, Kirill A. Shutemov wrote: Hi Paul and all, With commit 755609a9087fa983f567dc5452b2fa7b089b591f I've got deadlock on poweroff. It guess it happens because of race for cpu_hotplug.lock: CPU A CPU B disable_nonboot_cpus() _cpu_down() cpu_hotplug_begin() mutex_lock(cpu_hotplug.lock); __cpu_notify() padata_cpu_callback() __padata_remove_cpu() padata_replace() synchronize_rcu() rcu_gp_kthread() get_online_cpus(); mutex_lock(cpu_hotplug.lock); Have you seen the issue before? This is a new one for me. Does the following (very lightly tested) patch help? Works for me. Thanks. -- Kirill A. Shutemov -- To unsubscribe from this list: send the line unsubscribe linux-crypto in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Deadlock on poweroff
On 10/07/2012 10:20 PM, Kirill A. Shutemov wrote: On Sun, Oct 07, 2012 at 09:03:11AM -0700, Paul E. McKenney wrote: On Sun, Oct 07, 2012 at 05:47:11AM +0300, Kirill A. Shutemov wrote: Hi Paul and all, With commit 755609a9087fa983f567dc5452b2fa7b089b591f I've got deadlock on poweroff. It guess it happens because of race for cpu_hotplug.lock: CPU A CPU B disable_nonboot_cpus() _cpu_down() cpu_hotplug_begin() mutex_lock(cpu_hotplug.lock); __cpu_notify() padata_cpu_callback() __padata_remove_cpu() padata_replace() synchronize_rcu() rcu_gp_kthread() get_online_cpus(); mutex_lock(cpu_hotplug.lock); Have you seen the issue before? This is a new one for me. Does the following (very lightly tested) patch help? Works for me. Thanks. Could you share the patch please? Looks like it didn't hit the mailing lists.. Regards, Srivatsa S. Bhat -- To unsubscribe from this list: send the line unsubscribe linux-crypto in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Deadlock on poweroff
On Sun, Oct 07, 2012 at 10:35:01PM +0530, Srivatsa S. Bhat wrote: On 10/07/2012 10:20 PM, Kirill A. Shutemov wrote: On Sun, Oct 07, 2012 at 09:03:11AM -0700, Paul E. McKenney wrote: On Sun, Oct 07, 2012 at 05:47:11AM +0300, Kirill A. Shutemov wrote: Hi Paul and all, With commit 755609a9087fa983f567dc5452b2fa7b089b591f I've got deadlock on poweroff. It guess it happens because of race for cpu_hotplug.lock: CPU A CPU B disable_nonboot_cpus() _cpu_down() cpu_hotplug_begin() mutex_lock(cpu_hotplug.lock); __cpu_notify() padata_cpu_callback() __padata_remove_cpu() padata_replace() synchronize_rcu() rcu_gp_kthread() get_online_cpus(); mutex_lock(cpu_hotplug.lock); Have you seen the issue before? This is a new one for me. Does the following (very lightly tested) patch help? Works for me. Thanks. Could you share the patch please? Looks like it didn't hit the mailing lists.. Sure. Here's original mail from Paul: Date: Sun, 7 Oct 2012 09:03:11 -0700 From: Paul E. McKenney paul...@linux.vnet.ibm.com To: Kirill A. Shutemov kir...@shutemov.name Cc: linux-ker...@vger.kernel.org, Dipankar Sarma dipan...@in.ibm.com, Thomas Gleixner t...@linutronix.de, Andrew Morton a...@linux-foundation.org, Steffen Klassert steffen.klass...@secunet.com, .linux-crypto@vger.kernel.org Subject: Re: Deadlock on poweroff Message-ID: 20121007160311.ge2...@linux.vnet.ibm.com Reply-To: paul...@linux.vnet.ibm.com References: 20121007024711.ga21...@shutemov.name MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: 20121007024711.ga21...@shutemov.name User-Agent: Mutt/1.5.21 (2010-09-15) X-Content-Scanned: Fidelis XPS MAILER x-cbid: 12100716-7408---09194B17 Status: RO Content-Length: 6055 Lines: 173 On Sun, Oct 07, 2012 at 05:47:11AM +0300, Kirill A. Shutemov wrote: Hi Paul and all, With commit 755609a9087fa983f567dc5452b2fa7b089b591f I've got deadlock on poweroff. It guess it happens because of race for cpu_hotplug.lock: CPU A CPU B disable_nonboot_cpus() _cpu_down() cpu_hotplug_begin() mutex_lock(cpu_hotplug.lock); __cpu_notify() padata_cpu_callback() __padata_remove_cpu() padata_replace() synchronize_rcu() rcu_gp_kthread() get_online_cpus(); mutex_lock(cpu_hotplug.lock); Have you seen the issue before? This is a new one for me. Does the following (very lightly tested) patch help? Thanx, Paul rcu: Grace-period initialization excludes only RCU notifier Kirill noted the following deadlock cycle on shutdown involving padata: With commit 755609a9087fa983f567dc5452b2fa7b089b591f I've got deadlock on poweroff. It guess it happens because of race for cpu_hotplug.lock: CPU A CPU B disable_nonboot_cpus() _cpu_down() cpu_hotplug_begin() mutex_lock(cpu_hotplug.lock); __cpu_notify() padata_cpu_callback() __padata_remove_cpu() padata_replace() synchronize_rcu() rcu_gp_kthread() get_online_cpus(); mutex_lock(cpu_hotplug.lock); It would of course be good to eliminate grace-period delays from CPU-hotplug notifiers, but that is a separate issue. Deadlock is not an appropriate diagnostic for excessive CPU-hotplug latency. Fortunately, grace-period initialization does not actually need to exclude all of the CPU-hotplug operation, but rather only RCU's own CPU_UP_PREPARE and CPU_DEAD CPU-hotplug notifiers. This commit therefore introduces a new per-rcu_state onoff_mutex that provides the required concurrency control in place of the get_online_cpus() that was previously in rcu_gp_init(). Reported-by: Kirill A. Shutemov kir...@shutemov.name Signed-off-by: Paul E. McKenney paul...@linux.vnet.ibm.com diff --git a/kernel/rcutree.c b/kernel/rcutree.c index fb63d7b..5eece12 100644 --- a/kernel/rcutree.c +++ b/kernel/rcutree.c @@ -74,6 +74,7 @@ static struct lock_class_key rcu_fqs_class[RCU_NUM_LVLS]; .orphan_nxttail = sname##_state.orphan_nxtlist, \ .orphan_donetail = sname##_state.orphan_donelist, \ .barrier_mutex = __MUTEX_INITIALIZER(sname##_state.barrier_mutex), \ + .onoff_mutex = __MUTEX_INITIALIZER(sname##_state.onoff_mutex), \ .name = #sname, \ } @@ -1229,7 +1230,7 @@ static int rcu_gp_init(struct rcu_state *rsp) raw_spin_unlock_irq(rnp-lock); /* Exclude any concurrent CPU-hotplug operations. */ - get_online_cpus
Re: Deadlock on poweroff
On 10/07/2012 10:41 PM, Kirill A. Shutemov wrote: On Sun, Oct 07, 2012 at 10:35:01PM +0530, Srivatsa S. Bhat wrote: On 10/07/2012 10:20 PM, Kirill A. Shutemov wrote: On Sun, Oct 07, 2012 at 09:03:11AM -0700, Paul E. McKenney wrote: On Sun, Oct 07, 2012 at 05:47:11AM +0300, Kirill A. Shutemov wrote: Hi Paul and all, With commit 755609a9087fa983f567dc5452b2fa7b089b591f I've got deadlock on poweroff. It guess it happens because of race for cpu_hotplug.lock: CPU A CPU B disable_nonboot_cpus() _cpu_down() cpu_hotplug_begin() mutex_lock(cpu_hotplug.lock); __cpu_notify() padata_cpu_callback() __padata_remove_cpu() padata_replace() synchronize_rcu() rcu_gp_kthread() get_online_cpus(); mutex_lock(cpu_hotplug.lock); Have you seen the issue before? This is a new one for me. Does the following (very lightly tested) patch help? Works for me. Thanks. Could you share the patch please? Looks like it didn't hit the mailing lists.. Sure. Here's original mail from Paul: Ah, great! Thanks! Regards, Srivatsa S. Bhat Date: Sun, 7 Oct 2012 09:03:11 -0700 From: Paul E. McKenney paul...@linux.vnet.ibm.com To: Kirill A. Shutemov kir...@shutemov.name Cc: linux-ker...@vger.kernel.org, Dipankar Sarma dipan...@in.ibm.com, Thomas Gleixner t...@linutronix.de, Andrew Morton a...@linux-foundation.org, Steffen Klassert steffen.klass...@secunet.com, .linux-crypto@vger.kernel.org Subject: Re: Deadlock on poweroff Message-ID: 20121007160311.ge2...@linux.vnet.ibm.com Reply-To: paul...@linux.vnet.ibm.com References: 20121007024711.ga21...@shutemov.name MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: 20121007024711.ga21...@shutemov.name User-Agent: Mutt/1.5.21 (2010-09-15) X-Content-Scanned: Fidelis XPS MAILER x-cbid: 12100716-7408---09194B17 Status: RO Content-Length: 6055 Lines: 173 On Sun, Oct 07, 2012 at 05:47:11AM +0300, Kirill A. Shutemov wrote: Hi Paul and all, With commit 755609a9087fa983f567dc5452b2fa7b089b591f I've got deadlock on poweroff. It guess it happens because of race for cpu_hotplug.lock: CPU A CPU B disable_nonboot_cpus() _cpu_down() cpu_hotplug_begin() mutex_lock(cpu_hotplug.lock); __cpu_notify() padata_cpu_callback() __padata_remove_cpu() padata_replace() synchronize_rcu() rcu_gp_kthread() get_online_cpus(); mutex_lock(cpu_hotplug.lock); Have you seen the issue before? This is a new one for me. Does the following (very lightly tested) patch help? Thanx, Paul rcu: Grace-period initialization excludes only RCU notifier Kirill noted the following deadlock cycle on shutdown involving padata: With commit 755609a9087fa983f567dc5452b2fa7b089b591f I've got deadlock on poweroff. It guess it happens because of race for cpu_hotplug.lock: CPU A CPU B disable_nonboot_cpus() _cpu_down() cpu_hotplug_begin() mutex_lock(cpu_hotplug.lock); __cpu_notify() padata_cpu_callback() __padata_remove_cpu() padata_replace() synchronize_rcu() rcu_gp_kthread() get_online_cpus(); mutex_lock(cpu_hotplug.lock); It would of course be good to eliminate grace-period delays from CPU-hotplug notifiers, but that is a separate issue. Deadlock is not an appropriate diagnostic for excessive CPU-hotplug latency. Fortunately, grace-period initialization does not actually need to exclude all of the CPU-hotplug operation, but rather only RCU's own CPU_UP_PREPARE and CPU_DEAD CPU-hotplug notifiers. This commit therefore introduces a new per-rcu_state onoff_mutex that provides the required concurrency control in place of the get_online_cpus() that was previously in rcu_gp_init(). Reported-by: Kirill A. Shutemov kir...@shutemov.name Signed-off-by: Paul E. McKenney paul...@linux.vnet.ibm.com diff --git a/kernel/rcutree.c b/kernel/rcutree.c index fb63d7b..5eece12 100644 --- a/kernel/rcutree.c +++ b/kernel/rcutree.c @@ -74,6 +74,7 @@ static struct lock_class_key rcu_fqs_class[RCU_NUM_LVLS]; .orphan_nxttail = sname##_state.orphan_nxtlist, \ .orphan_donetail = sname##_state.orphan_donelist, \ .barrier_mutex = __MUTEX_INITIALIZER(sname##_state.barrier_mutex), \ + .onoff_mutex = __MUTEX_INITIALIZER(sname##_state.onoff_mutex), \ .name = #sname, \ } @@ -1229,7 +1230,7 @@ static int rcu_gp_init(struct rcu_state *rsp
Re: Deadlock on poweroff
On Sun, Oct 07, 2012 at 10:46:54PM +0530, Srivatsa S. Bhat wrote: On 10/07/2012 10:41 PM, Kirill A. Shutemov wrote: On Sun, Oct 07, 2012 at 10:35:01PM +0530, Srivatsa S. Bhat wrote: On 10/07/2012 10:20 PM, Kirill A. Shutemov wrote: On Sun, Oct 07, 2012 at 09:03:11AM -0700, Paul E. McKenney wrote: On Sun, Oct 07, 2012 at 05:47:11AM +0300, Kirill A. Shutemov wrote: Hi Paul and all, With commit 755609a9087fa983f567dc5452b2fa7b089b591f I've got deadlock on poweroff. It guess it happens because of race for cpu_hotplug.lock: CPU A CPU B disable_nonboot_cpus() _cpu_down() cpu_hotplug_begin() mutex_lock(cpu_hotplug.lock); __cpu_notify() padata_cpu_callback() __padata_remove_cpu() padata_replace() synchronize_rcu() rcu_gp_kthread() get_online_cpus(); mutex_lock(cpu_hotplug.lock); Have you seen the issue before? This is a new one for me. Does the following (very lightly tested) patch help? Works for me. Thanks. Could you share the patch please? Looks like it didn't hit the mailing lists.. Sure. Here's original mail from Paul: Ah, great! Thanks! Thank you, Kirill. I wonder what the mailing list doesn't like about me... ;-) Thanx, Paul Regards, Srivatsa S. Bhat Date: Sun, 7 Oct 2012 09:03:11 -0700 From: Paul E. McKenney paul...@linux.vnet.ibm.com To: Kirill A. Shutemov kir...@shutemov.name Cc: linux-ker...@vger.kernel.org, Dipankar Sarma dipan...@in.ibm.com, Thomas Gleixner t...@linutronix.de, Andrew Morton a...@linux-foundation.org, Steffen Klassert steffen.klass...@secunet.com, .linux-crypto@vger.kernel.org Subject: Re: Deadlock on poweroff Message-ID: 20121007160311.ge2...@linux.vnet.ibm.com Reply-To: paul...@linux.vnet.ibm.com References: 20121007024711.ga21...@shutemov.name MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: 20121007024711.ga21...@shutemov.name User-Agent: Mutt/1.5.21 (2010-09-15) X-Content-Scanned: Fidelis XPS MAILER x-cbid: 12100716-7408---09194B17 Status: RO Content-Length: 6055 Lines: 173 On Sun, Oct 07, 2012 at 05:47:11AM +0300, Kirill A. Shutemov wrote: Hi Paul and all, With commit 755609a9087fa983f567dc5452b2fa7b089b591f I've got deadlock on poweroff. It guess it happens because of race for cpu_hotplug.lock: CPU A CPU B disable_nonboot_cpus() _cpu_down() cpu_hotplug_begin() mutex_lock(cpu_hotplug.lock); __cpu_notify() padata_cpu_callback() __padata_remove_cpu() padata_replace() synchronize_rcu() rcu_gp_kthread() get_online_cpus(); mutex_lock(cpu_hotplug.lock); Have you seen the issue before? This is a new one for me. Does the following (very lightly tested) patch help? Thanx, Paul rcu: Grace-period initialization excludes only RCU notifier Kirill noted the following deadlock cycle on shutdown involving padata: With commit 755609a9087fa983f567dc5452b2fa7b089b591f I've got deadlock on poweroff. It guess it happens because of race for cpu_hotplug.lock: CPU A CPU B disable_nonboot_cpus() _cpu_down() cpu_hotplug_begin() mutex_lock(cpu_hotplug.lock); __cpu_notify() padata_cpu_callback() __padata_remove_cpu() padata_replace() synchronize_rcu() rcu_gp_kthread() get_online_cpus(); mutex_lock(cpu_hotplug.lock); It would of course be good to eliminate grace-period delays from CPU-hotplug notifiers, but that is a separate issue. Deadlock is not an appropriate diagnostic for excessive CPU-hotplug latency. Fortunately, grace-period initialization does not actually need to exclude all of the CPU-hotplug operation, but rather only RCU's own CPU_UP_PREPARE and CPU_DEAD CPU-hotplug notifiers. This commit therefore introduces a new per-rcu_state onoff_mutex that provides the required concurrency control in place of the get_online_cpus() that was previously in rcu_gp_init(). Reported-by: Kirill A. Shutemov kir...@shutemov.name Signed-off-by: Paul E. McKenney paul...@linux.vnet.ibm.com diff --git a/kernel/rcutree.c b/kernel/rcutree.c index fb63d7b..5eece12 100644 --- a/kernel/rcutree.c +++ b/kernel/rcutree.c @@ -74,6 +74,7 @@ static struct lock_class_key rcu_fqs_class[RCU_NUM_LVLS
Re: Deadlock on poweroff
On Sun, Oct 07, 2012 at 07:50:12PM +0300, Kirill A. Shutemov wrote: On Sun, Oct 07, 2012 at 09:03:11AM -0700, Paul E. McKenney wrote: On Sun, Oct 07, 2012 at 05:47:11AM +0300, Kirill A. Shutemov wrote: Hi Paul and all, With commit 755609a9087fa983f567dc5452b2fa7b089b591f I've got deadlock on poweroff. It guess it happens because of race for cpu_hotplug.lock: CPU A CPU B disable_nonboot_cpus() _cpu_down() cpu_hotplug_begin() mutex_lock(cpu_hotplug.lock); __cpu_notify() padata_cpu_callback() __padata_remove_cpu() padata_replace() synchronize_rcu() rcu_gp_kthread() get_online_cpus(); mutex_lock(cpu_hotplug.lock); Have you seen the issue before? This is a new one for me. Does the following (very lightly tested) patch help? Works for me. Thanks. May I add your Tested-by? Thanx, Paul -- To unsubscribe from this list: send the line unsubscribe linux-crypto in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Deadlock on poweroff
On Sun, Oct 07, 2012 at 09:41:28PM -0700, Paul E. McKenney wrote: On Sun, Oct 07, 2012 at 07:50:12PM +0300, Kirill A. Shutemov wrote: On Sun, Oct 07, 2012 at 09:03:11AM -0700, Paul E. McKenney wrote: On Sun, Oct 07, 2012 at 05:47:11AM +0300, Kirill A. Shutemov wrote: Hi Paul and all, With commit 755609a9087fa983f567dc5452b2fa7b089b591f I've got deadlock on poweroff. It guess it happens because of race for cpu_hotplug.lock: CPU A CPU B disable_nonboot_cpus() _cpu_down() cpu_hotplug_begin() mutex_lock(cpu_hotplug.lock); __cpu_notify() padata_cpu_callback() __padata_remove_cpu() padata_replace() synchronize_rcu() rcu_gp_kthread() get_online_cpus(); mutex_lock(cpu_hotplug.lock); Have you seen the issue before? This is a new one for me. Does the following (very lightly tested) patch help? Works for me. Thanks. May I add your Tested-by? Yep. Tested-by: Kirill A. Shutemov kir...@shutemov.name -- Kirill A. Shutemov -- To unsubscribe from this list: send the line unsubscribe linux-crypto in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
