Re: [PATCH v5 07/45] CPU hotplug: Provide APIs to prevent CPU offline from atomic context
Hi Namhyung,
On 01/29/2013 03:51 PM, Namhyung Kim wrote:
Hi Srivatsa,
On Tue, 22 Jan 2013 13:04:54 +0530, Srivatsa S. Bhat wrote:
@@ -246,15 +291,21 @@ struct take_cpu_down_param {
static int __ref take_cpu_down(void *_param)
{
struct take_cpu_down_param *param = _param;
-int err;
+unsigned long flags;
+int err = 0;
It seems no need to set 'err' to 0.
Sorry for the late reply. This mail got buried in my inbox and
I hadn't noticed it until now.. :-(
I'll remove the unnecessary initialization. Thank you!
Regards,
Srivatsa S. Bhat
___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH v5 07/45] CPU hotplug: Provide APIs to prevent CPU offline from atomic context
On Tue, Jan 22, 2013 at 01:04:54PM +0530, Srivatsa S. Bhat wrote: There are places where preempt_disable() or local_irq_disable() are used to prevent any CPU from going offline during the critical section. Let us call them as atomic hotplug readers (atomic because they run in atomic, non-preemptible contexts). Today, preempt_disable() or its equivalent works because the hotplug writer uses stop_machine() to take CPUs offline. But once stop_machine() is gone from the CPU hotplug offline path, the readers won't be able to prevent CPUs from going offline using preempt_disable(). So the intent here is to provide synchronization APIs for such atomic hotplug readers, to prevent (any) CPUs from going offline, without depending on stop_machine() at the writer-side. The new APIs will look something like this: get_online_cpus_atomic() and put_online_cpus_atomic() Some important design requirements and considerations: - 1. Scalable synchronization at the reader-side, especially in the fast-path Any synchronization at the atomic hotplug readers side must be highly scalable - avoid global single-holder locks/counters etc. Because, these paths currently use the extremely fast preempt_disable(); our replacement to preempt_disable() should not become ridiculously costly and also should not serialize the readers among themselves needlessly. At a minimum, the new APIs must be extremely fast at the reader side atleast in the fast-path, when no CPU offline writers are active. 2. preempt_disable() was recursive. The replacement should also be recursive. 3. No (new) lock-ordering restrictions preempt_disable() was super-flexible. It didn't impose any ordering restrictions or rules for nesting. Our replacement should also be equally flexible and usable. 4. No deadlock possibilities Regular per-cpu locking is not the way to go if we want to have relaxed rules for lock-ordering. Because, we can end up in circular-locking dependencies as explained in https://lkml.org/lkml/2012/12/6/290 So, avoid the usual per-cpu locking schemes (per-cpu locks/per-cpu atomic counters with spin-on-contention etc) as much as possible, to avoid numerous deadlock possibilities from creeping in. Implementation of the design: We use per-CPU reader-writer locks for synchronization because: a. They are quite fast and scalable in the fast-path (when no writers are active), since they use fast per-cpu counters in those paths. b. They are recursive at the reader side. c. They provide a good amount of safety against deadlocks; they don't spring new deadlock possibilities on us from out of nowhere. As a result, they have relaxed locking rules and are quite flexible, and thus are best suited for replacing usages of preempt_disable() or local_irq_disable() at the reader side. Together, these satisfy all the requirements mentioned above. I'm indebted to Michael Wang and Xiao Guangrong for their numerous thoughtful suggestions and ideas, which inspired and influenced many of the decisions in this as well as previous designs. Thanks a lot Michael and Xiao! Cc: Russell King li...@arm.linux.org.uk Cc: Mike Frysinger vap...@gentoo.org Cc: Tony Luck tony.l...@intel.com Cc: Ralf Baechle r...@linux-mips.org Cc: David Howells dhowe...@redhat.com Cc: James E.J. Bottomley j...@parisc-linux.org Cc: Benjamin Herrenschmidt b...@kernel.crashing.org Cc: Martin Schwidefsky schwidef...@de.ibm.com Cc: Paul Mundt let...@linux-sh.org Cc: David S. Miller da...@davemloft.net Cc: H. Peter Anvin h...@zytor.com Cc: x...@kernel.org Cc: linux-arm-ker...@lists.infradead.org Cc: uclinux-dist-de...@blackfin.uclinux.org Cc: linux-i...@vger.kernel.org Cc: linux-m...@linux-mips.org Cc: linux-am33-l...@redhat.com Cc: linux-par...@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org Cc: linux-s...@vger.kernel.org Cc: linux...@vger.kernel.org Cc: sparcli...@vger.kernel.org Signed-off-by: Srivatsa S. Bhat srivatsa.b...@linux.vnet.ibm.com With the change suggested by Namhyung: Reviewed-by: Paul E. McKenney paul...@linux.vnet.ibm.com --- arch/arm/Kconfig |1 + arch/blackfin/Kconfig |1 + arch/ia64/Kconfig |1 + arch/mips/Kconfig |1 + arch/mn10300/Kconfig |1 + arch/parisc/Kconfig |1 + arch/powerpc/Kconfig |1 + arch/s390/Kconfig |1 + arch/sh/Kconfig |1 + arch/sparc/Kconfig|1 + arch/x86/Kconfig |1 + include/linux/cpu.h |4 +++ kernel/cpu.c | 57 ++--- 13 files changed, 69 insertions(+), 3 deletions(-) diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index 67874b8..cb6b94b 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -1616,6 +1616,7 @@ config NR_CPUS
Re: [PATCH v5 07/45] CPU hotplug: Provide APIs to prevent CPU offline from atomic context
Hi Srivatsa,
On Tue, 22 Jan 2013 13:04:54 +0530, Srivatsa S. Bhat wrote:
@@ -246,15 +291,21 @@ struct take_cpu_down_param {
static int __ref take_cpu_down(void *_param)
{
struct take_cpu_down_param *param = _param;
- int err;
+ unsigned long flags;
+ int err = 0;
It seems no need to set 'err' to 0.
Thanks,
Namhyung
+
+ percpu_write_lock_irqsave(hotplug_pcpu_rwlock, flags);
/* Ensure this CPU doesn't handle any more interrupts. */
err = __cpu_disable();
if (err 0)
- return err;
+ goto out;
cpu_notify(CPU_DYING | param-mod, param-hcpu);
- return 0;
+
+out:
+ percpu_write_unlock_irqrestore(hotplug_pcpu_rwlock, flags);
+ return err;
}
/* Requires cpu_add_remove_lock to be held */
--
To unsubscribe from this list: send the line unsubscribe linux-pm in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev
[PATCH v5 07/45] CPU hotplug: Provide APIs to prevent CPU offline from atomic context
There are places where preempt_disable() or local_irq_disable() are used to prevent any CPU from going offline during the critical section. Let us call them as atomic hotplug readers (atomic because they run in atomic, non-preemptible contexts). Today, preempt_disable() or its equivalent works because the hotplug writer uses stop_machine() to take CPUs offline. But once stop_machine() is gone from the CPU hotplug offline path, the readers won't be able to prevent CPUs from going offline using preempt_disable(). So the intent here is to provide synchronization APIs for such atomic hotplug readers, to prevent (any) CPUs from going offline, without depending on stop_machine() at the writer-side. The new APIs will look something like this: get_online_cpus_atomic() and put_online_cpus_atomic() Some important design requirements and considerations: - 1. Scalable synchronization at the reader-side, especially in the fast-path Any synchronization at the atomic hotplug readers side must be highly scalable - avoid global single-holder locks/counters etc. Because, these paths currently use the extremely fast preempt_disable(); our replacement to preempt_disable() should not become ridiculously costly and also should not serialize the readers among themselves needlessly. At a minimum, the new APIs must be extremely fast at the reader side atleast in the fast-path, when no CPU offline writers are active. 2. preempt_disable() was recursive. The replacement should also be recursive. 3. No (new) lock-ordering restrictions preempt_disable() was super-flexible. It didn't impose any ordering restrictions or rules for nesting. Our replacement should also be equally flexible and usable. 4. No deadlock possibilities Regular per-cpu locking is not the way to go if we want to have relaxed rules for lock-ordering. Because, we can end up in circular-locking dependencies as explained in https://lkml.org/lkml/2012/12/6/290 So, avoid the usual per-cpu locking schemes (per-cpu locks/per-cpu atomic counters with spin-on-contention etc) as much as possible, to avoid numerous deadlock possibilities from creeping in. Implementation of the design: We use per-CPU reader-writer locks for synchronization because: a. They are quite fast and scalable in the fast-path (when no writers are active), since they use fast per-cpu counters in those paths. b. They are recursive at the reader side. c. They provide a good amount of safety against deadlocks; they don't spring new deadlock possibilities on us from out of nowhere. As a result, they have relaxed locking rules and are quite flexible, and thus are best suited for replacing usages of preempt_disable() or local_irq_disable() at the reader side. Together, these satisfy all the requirements mentioned above. I'm indebted to Michael Wang and Xiao Guangrong for their numerous thoughtful suggestions and ideas, which inspired and influenced many of the decisions in this as well as previous designs. Thanks a lot Michael and Xiao! Cc: Russell King li...@arm.linux.org.uk Cc: Mike Frysinger vap...@gentoo.org Cc: Tony Luck tony.l...@intel.com Cc: Ralf Baechle r...@linux-mips.org Cc: David Howells dhowe...@redhat.com Cc: James E.J. Bottomley j...@parisc-linux.org Cc: Benjamin Herrenschmidt b...@kernel.crashing.org Cc: Martin Schwidefsky schwidef...@de.ibm.com Cc: Paul Mundt let...@linux-sh.org Cc: David S. Miller da...@davemloft.net Cc: H. Peter Anvin h...@zytor.com Cc: x...@kernel.org Cc: linux-arm-ker...@lists.infradead.org Cc: uclinux-dist-de...@blackfin.uclinux.org Cc: linux-i...@vger.kernel.org Cc: linux-m...@linux-mips.org Cc: linux-am33-l...@redhat.com Cc: linux-par...@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org Cc: linux-s...@vger.kernel.org Cc: linux...@vger.kernel.org Cc: sparcli...@vger.kernel.org Signed-off-by: Srivatsa S. Bhat srivatsa.b...@linux.vnet.ibm.com --- arch/arm/Kconfig |1 + arch/blackfin/Kconfig |1 + arch/ia64/Kconfig |1 + arch/mips/Kconfig |1 + arch/mn10300/Kconfig |1 + arch/parisc/Kconfig |1 + arch/powerpc/Kconfig |1 + arch/s390/Kconfig |1 + arch/sh/Kconfig |1 + arch/sparc/Kconfig|1 + arch/x86/Kconfig |1 + include/linux/cpu.h |4 +++ kernel/cpu.c | 57 ++--- 13 files changed, 69 insertions(+), 3 deletions(-) diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index 67874b8..cb6b94b 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -1616,6 +1616,7 @@ config NR_CPUS config HOTPLUG_CPU bool Support for hot-pluggable CPUs depends on SMP HOTPLUG + select PERCPU_RWLOCK help Say Y here to experiment with turning CPUs off and on. CPUs can be controlled through /sys/devices/system/cpu. diff --git
