Re: [PATCH v3 10/52] arm, kvm: Fix CPU hotplug callback registration
On Fri, Mar 14, 2014 at 11:13:29AM +0530, Srivatsa S. Bhat wrote:
> On 03/13/2014 04:51 AM, Christoffer Dall wrote:
> > On Tue, Mar 11, 2014 at 02:05:38AM +0530, Srivatsa S. Bhat wrote:
> >> Subsystems that want to register CPU hotplug callbacks, as well as perform
> >> initialization for the CPUs that are already online, often do it as shown
> >> below:
> >>
> >>get_online_cpus();
> >>
> >>for_each_online_cpu(cpu)
> >>init_cpu(cpu);
> >>
> >>register_cpu_notifier(_cpu_notifier);
> >>
> >>put_online_cpus();
> >>
> >> This is wrong, since it is prone to ABBA deadlocks involving the
> >> cpu_add_remove_lock and the cpu_hotplug.lock (when running concurrently
> >> with CPU hotplug operations).
> >>
> >> Instead, the correct and race-free way of performing the callback
> >> registration is:
> >>
> >>cpu_notifier_register_begin();
> >>
> >>for_each_online_cpu(cpu)
> >>init_cpu(cpu);
> >>
> >>/* Note the use of the double underscored version of the API */
> >>__register_cpu_notifier(_cpu_notifier);
> >>
> >>cpu_notifier_register_done();
> >>
> >>
> >> Fix the kvm code in arm by using this latter form of callback registration.
> >>
> >> Cc: Christoffer Dall
> >> Cc: Gleb Natapov
> >> Cc: Russell King
> >> Cc: Ingo Molnar
> >> Cc: kvm...@lists.cs.columbia.edu
> >> Cc: k...@vger.kernel.org
> >> Cc: linux-arm-ker...@lists.infradead.org
> >> Acked-by: Paolo Bonzini
> >> Signed-off-by: Srivatsa S. Bhat
> >> ---
> >>
> >> arch/arm/kvm/arm.c |7 ++-
> >> 1 file changed, 6 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
> >> index bd18bb8..f0e50a0 100644
> >> --- a/arch/arm/kvm/arm.c
> >> +++ b/arch/arm/kvm/arm.c
> >> @@ -1051,21 +1051,26 @@ int kvm_arch_init(void *opaque)
> >>}
> >>}
> >>
> >> + cpu_notifier_register_begin();
> >> +
> >>err = init_hyp_mode();
> >>if (err)
> >>goto out_err;
> >>
> >> - err = register_cpu_notifier(_init_cpu_nb);
> >> + err = __register_cpu_notifier(_init_cpu_nb);
> >>if (err) {
> >>kvm_err("Cannot register HYP init CPU notifier (%d)\n", err);
> >>goto out_err;
> >>}
> >>
> >> + cpu_notifier_register_done();
> >> +
> >>hyp_cpu_pm_init();
> >>
> >>kvm_coproc_table_init();
> >>return 0;
> >> out_err:
> >> + cpu_notifier_register_done();
> >>return err;
> >> }
> >>
> >>
> >
> > Just so we're clear, the existing code was simply racy as not prone to
> > deadlocks, right?
> >
> > This makes it clear that the test above for compatible CPUs can be quite
> > easily evaded by using CPU hotplug, but we don't really have a good
> > solution for handling that yet... Hmmm, grumble grumble, I guess if you
> > hotplug unsupported CPUs on a KVM/ARM system for now, stuff will break.
> >
>
> In this particular case, there was no deadlock possibility, rather the
> existing code had insufficient synchronization against CPU hotplug.
>
> init_hyp_mode() would invoke cpu_init_hyp_mode() on currently online CPUs
> using on_each_cpu(). If a CPU came online after this point and before calling
> register_cpu_notifier(), that CPU would remain uninitialized because this
> subsystem would miss the hot-online event. This patch fixes this bug and
> also uses the new synchronization method (instead of get/put_online_cpus())
> to ensure that we don't deadlock with CPU hotplug.
>
Yes, that was my conclusion as well. Thanks for clarifying. (It could
be noted in the commit message as well if you should feel so inclined).
> > In any case:
> > Acked-by: Christoffer Dall
> >
>
> Thanks a lot!
>
Thanks,
-Christoffer
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v3 10/52] arm, kvm: Fix CPU hotplug callback registration
On Fri, Mar 14, 2014 at 11:13:29AM +0530, Srivatsa S. Bhat wrote:
On 03/13/2014 04:51 AM, Christoffer Dall wrote:
On Tue, Mar 11, 2014 at 02:05:38AM +0530, Srivatsa S. Bhat wrote:
Subsystems that want to register CPU hotplug callbacks, as well as perform
initialization for the CPUs that are already online, often do it as shown
below:
get_online_cpus();
for_each_online_cpu(cpu)
init_cpu(cpu);
register_cpu_notifier(foobar_cpu_notifier);
put_online_cpus();
This is wrong, since it is prone to ABBA deadlocks involving the
cpu_add_remove_lock and the cpu_hotplug.lock (when running concurrently
with CPU hotplug operations).
Instead, the correct and race-free way of performing the callback
registration is:
cpu_notifier_register_begin();
for_each_online_cpu(cpu)
init_cpu(cpu);
/* Note the use of the double underscored version of the API */
__register_cpu_notifier(foobar_cpu_notifier);
cpu_notifier_register_done();
Fix the kvm code in arm by using this latter form of callback registration.
Cc: Christoffer Dall christoffer.d...@linaro.org
Cc: Gleb Natapov g...@kernel.org
Cc: Russell King li...@arm.linux.org.uk
Cc: Ingo Molnar mi...@kernel.org
Cc: kvm...@lists.cs.columbia.edu
Cc: k...@vger.kernel.org
Cc: linux-arm-ker...@lists.infradead.org
Acked-by: Paolo Bonzini pbonz...@redhat.com
Signed-off-by: Srivatsa S. Bhat srivatsa.b...@linux.vnet.ibm.com
---
arch/arm/kvm/arm.c |7 ++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
index bd18bb8..f0e50a0 100644
--- a/arch/arm/kvm/arm.c
+++ b/arch/arm/kvm/arm.c
@@ -1051,21 +1051,26 @@ int kvm_arch_init(void *opaque)
}
}
+ cpu_notifier_register_begin();
+
err = init_hyp_mode();
if (err)
goto out_err;
- err = register_cpu_notifier(hyp_init_cpu_nb);
+ err = __register_cpu_notifier(hyp_init_cpu_nb);
if (err) {
kvm_err(Cannot register HYP init CPU notifier (%d)\n, err);
goto out_err;
}
+ cpu_notifier_register_done();
+
hyp_cpu_pm_init();
kvm_coproc_table_init();
return 0;
out_err:
+ cpu_notifier_register_done();
return err;
}
Just so we're clear, the existing code was simply racy as not prone to
deadlocks, right?
This makes it clear that the test above for compatible CPUs can be quite
easily evaded by using CPU hotplug, but we don't really have a good
solution for handling that yet... Hmmm, grumble grumble, I guess if you
hotplug unsupported CPUs on a KVM/ARM system for now, stuff will break.
In this particular case, there was no deadlock possibility, rather the
existing code had insufficient synchronization against CPU hotplug.
init_hyp_mode() would invoke cpu_init_hyp_mode() on currently online CPUs
using on_each_cpu(). If a CPU came online after this point and before calling
register_cpu_notifier(), that CPU would remain uninitialized because this
subsystem would miss the hot-online event. This patch fixes this bug and
also uses the new synchronization method (instead of get/put_online_cpus())
to ensure that we don't deadlock with CPU hotplug.
Yes, that was my conclusion as well. Thanks for clarifying. (It could
be noted in the commit message as well if you should feel so inclined).
In any case:
Acked-by: Christoffer Dall christoffer.d...@linaro.org
Thanks a lot!
Thanks,
-Christoffer
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v3 10/52] arm, kvm: Fix CPU hotplug callback registration
On 03/13/2014 04:51 AM, Christoffer Dall wrote:
> On Tue, Mar 11, 2014 at 02:05:38AM +0530, Srivatsa S. Bhat wrote:
>> Subsystems that want to register CPU hotplug callbacks, as well as perform
>> initialization for the CPUs that are already online, often do it as shown
>> below:
>>
>> get_online_cpus();
>>
>> for_each_online_cpu(cpu)
>> init_cpu(cpu);
>>
>> register_cpu_notifier(_cpu_notifier);
>>
>> put_online_cpus();
>>
>> This is wrong, since it is prone to ABBA deadlocks involving the
>> cpu_add_remove_lock and the cpu_hotplug.lock (when running concurrently
>> with CPU hotplug operations).
>>
>> Instead, the correct and race-free way of performing the callback
>> registration is:
>>
>> cpu_notifier_register_begin();
>>
>> for_each_online_cpu(cpu)
>> init_cpu(cpu);
>>
>> /* Note the use of the double underscored version of the API */
>> __register_cpu_notifier(_cpu_notifier);
>>
>> cpu_notifier_register_done();
>>
>>
>> Fix the kvm code in arm by using this latter form of callback registration.
>>
>> Cc: Christoffer Dall
>> Cc: Gleb Natapov
>> Cc: Russell King
>> Cc: Ingo Molnar
>> Cc: kvm...@lists.cs.columbia.edu
>> Cc: k...@vger.kernel.org
>> Cc: linux-arm-ker...@lists.infradead.org
>> Acked-by: Paolo Bonzini
>> Signed-off-by: Srivatsa S. Bhat
>> ---
>>
>> arch/arm/kvm/arm.c |7 ++-
>> 1 file changed, 6 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
>> index bd18bb8..f0e50a0 100644
>> --- a/arch/arm/kvm/arm.c
>> +++ b/arch/arm/kvm/arm.c
>> @@ -1051,21 +1051,26 @@ int kvm_arch_init(void *opaque)
>> }
>> }
>>
>> +cpu_notifier_register_begin();
>> +
>> err = init_hyp_mode();
>> if (err)
>> goto out_err;
>>
>> -err = register_cpu_notifier(_init_cpu_nb);
>> +err = __register_cpu_notifier(_init_cpu_nb);
>> if (err) {
>> kvm_err("Cannot register HYP init CPU notifier (%d)\n", err);
>> goto out_err;
>> }
>>
>> +cpu_notifier_register_done();
>> +
>> hyp_cpu_pm_init();
>>
>> kvm_coproc_table_init();
>> return 0;
>> out_err:
>> +cpu_notifier_register_done();
>> return err;
>> }
>>
>>
>
> Just so we're clear, the existing code was simply racy as not prone to
> deadlocks, right?
>
> This makes it clear that the test above for compatible CPUs can be quite
> easily evaded by using CPU hotplug, but we don't really have a good
> solution for handling that yet... Hmmm, grumble grumble, I guess if you
> hotplug unsupported CPUs on a KVM/ARM system for now, stuff will break.
>
In this particular case, there was no deadlock possibility, rather the
existing code had insufficient synchronization against CPU hotplug.
init_hyp_mode() would invoke cpu_init_hyp_mode() on currently online CPUs
using on_each_cpu(). If a CPU came online after this point and before calling
register_cpu_notifier(), that CPU would remain uninitialized because this
subsystem would miss the hot-online event. This patch fixes this bug and
also uses the new synchronization method (instead of get/put_online_cpus())
to ensure that we don't deadlock with CPU hotplug.
> In any case:
> Acked-by: Christoffer Dall
>
Thanks a lot!
Regards,
Srivatsa S. Bhat
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v3 10/52] arm, kvm: Fix CPU hotplug callback registration
On 03/13/2014 04:51 AM, Christoffer Dall wrote:
On Tue, Mar 11, 2014 at 02:05:38AM +0530, Srivatsa S. Bhat wrote:
Subsystems that want to register CPU hotplug callbacks, as well as perform
initialization for the CPUs that are already online, often do it as shown
below:
get_online_cpus();
for_each_online_cpu(cpu)
init_cpu(cpu);
register_cpu_notifier(foobar_cpu_notifier);
put_online_cpus();
This is wrong, since it is prone to ABBA deadlocks involving the
cpu_add_remove_lock and the cpu_hotplug.lock (when running concurrently
with CPU hotplug operations).
Instead, the correct and race-free way of performing the callback
registration is:
cpu_notifier_register_begin();
for_each_online_cpu(cpu)
init_cpu(cpu);
/* Note the use of the double underscored version of the API */
__register_cpu_notifier(foobar_cpu_notifier);
cpu_notifier_register_done();
Fix the kvm code in arm by using this latter form of callback registration.
Cc: Christoffer Dall christoffer.d...@linaro.org
Cc: Gleb Natapov g...@kernel.org
Cc: Russell King li...@arm.linux.org.uk
Cc: Ingo Molnar mi...@kernel.org
Cc: kvm...@lists.cs.columbia.edu
Cc: k...@vger.kernel.org
Cc: linux-arm-ker...@lists.infradead.org
Acked-by: Paolo Bonzini pbonz...@redhat.com
Signed-off-by: Srivatsa S. Bhat srivatsa.b...@linux.vnet.ibm.com
---
arch/arm/kvm/arm.c |7 ++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
index bd18bb8..f0e50a0 100644
--- a/arch/arm/kvm/arm.c
+++ b/arch/arm/kvm/arm.c
@@ -1051,21 +1051,26 @@ int kvm_arch_init(void *opaque)
}
}
+cpu_notifier_register_begin();
+
err = init_hyp_mode();
if (err)
goto out_err;
-err = register_cpu_notifier(hyp_init_cpu_nb);
+err = __register_cpu_notifier(hyp_init_cpu_nb);
if (err) {
kvm_err(Cannot register HYP init CPU notifier (%d)\n, err);
goto out_err;
}
+cpu_notifier_register_done();
+
hyp_cpu_pm_init();
kvm_coproc_table_init();
return 0;
out_err:
+cpu_notifier_register_done();
return err;
}
Just so we're clear, the existing code was simply racy as not prone to
deadlocks, right?
This makes it clear that the test above for compatible CPUs can be quite
easily evaded by using CPU hotplug, but we don't really have a good
solution for handling that yet... Hmmm, grumble grumble, I guess if you
hotplug unsupported CPUs on a KVM/ARM system for now, stuff will break.
In this particular case, there was no deadlock possibility, rather the
existing code had insufficient synchronization against CPU hotplug.
init_hyp_mode() would invoke cpu_init_hyp_mode() on currently online CPUs
using on_each_cpu(). If a CPU came online after this point and before calling
register_cpu_notifier(), that CPU would remain uninitialized because this
subsystem would miss the hot-online event. This patch fixes this bug and
also uses the new synchronization method (instead of get/put_online_cpus())
to ensure that we don't deadlock with CPU hotplug.
In any case:
Acked-by: Christoffer Dall christoffer.d...@linaro.org
Thanks a lot!
Regards,
Srivatsa S. Bhat
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v3 10/52] arm, kvm: Fix CPU hotplug callback registration
On Tue, Mar 11, 2014 at 02:05:38AM +0530, Srivatsa S. Bhat wrote:
> Subsystems that want to register CPU hotplug callbacks, as well as perform
> initialization for the CPUs that are already online, often do it as shown
> below:
>
> get_online_cpus();
>
> for_each_online_cpu(cpu)
> init_cpu(cpu);
>
> register_cpu_notifier(_cpu_notifier);
>
> put_online_cpus();
>
> This is wrong, since it is prone to ABBA deadlocks involving the
> cpu_add_remove_lock and the cpu_hotplug.lock (when running concurrently
> with CPU hotplug operations).
>
> Instead, the correct and race-free way of performing the callback
> registration is:
>
> cpu_notifier_register_begin();
>
> for_each_online_cpu(cpu)
> init_cpu(cpu);
>
> /* Note the use of the double underscored version of the API */
> __register_cpu_notifier(_cpu_notifier);
>
> cpu_notifier_register_done();
>
>
> Fix the kvm code in arm by using this latter form of callback registration.
>
> Cc: Christoffer Dall
> Cc: Gleb Natapov
> Cc: Russell King
> Cc: Ingo Molnar
> Cc: kvm...@lists.cs.columbia.edu
> Cc: k...@vger.kernel.org
> Cc: linux-arm-ker...@lists.infradead.org
> Acked-by: Paolo Bonzini
> Signed-off-by: Srivatsa S. Bhat
> ---
>
> arch/arm/kvm/arm.c |7 ++-
> 1 file changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
> index bd18bb8..f0e50a0 100644
> --- a/arch/arm/kvm/arm.c
> +++ b/arch/arm/kvm/arm.c
> @@ -1051,21 +1051,26 @@ int kvm_arch_init(void *opaque)
> }
> }
>
> + cpu_notifier_register_begin();
> +
> err = init_hyp_mode();
> if (err)
> goto out_err;
>
> - err = register_cpu_notifier(_init_cpu_nb);
> + err = __register_cpu_notifier(_init_cpu_nb);
> if (err) {
> kvm_err("Cannot register HYP init CPU notifier (%d)\n", err);
> goto out_err;
> }
>
> + cpu_notifier_register_done();
> +
> hyp_cpu_pm_init();
>
> kvm_coproc_table_init();
> return 0;
> out_err:
> + cpu_notifier_register_done();
> return err;
> }
>
>
Just so we're clear, the existing code was simply racy as not prone to
deadlocks, right?
This makes it clear that the test above for compatible CPUs can be quite
easily evaded by using CPU hotplug, but we don't really have a good
solution for handling that yet... Hmmm, grumble grumble, I guess if you
hotplug unsupported CPUs on a KVM/ARM system for now, stuff will break.
In any case:
Acked-by: Christoffer Dall
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v3 10/52] arm, kvm: Fix CPU hotplug callback registration
On Tue, Mar 11, 2014 at 02:05:38AM +0530, Srivatsa S. Bhat wrote:
Subsystems that want to register CPU hotplug callbacks, as well as perform
initialization for the CPUs that are already online, often do it as shown
below:
get_online_cpus();
for_each_online_cpu(cpu)
init_cpu(cpu);
register_cpu_notifier(foobar_cpu_notifier);
put_online_cpus();
This is wrong, since it is prone to ABBA deadlocks involving the
cpu_add_remove_lock and the cpu_hotplug.lock (when running concurrently
with CPU hotplug operations).
Instead, the correct and race-free way of performing the callback
registration is:
cpu_notifier_register_begin();
for_each_online_cpu(cpu)
init_cpu(cpu);
/* Note the use of the double underscored version of the API */
__register_cpu_notifier(foobar_cpu_notifier);
cpu_notifier_register_done();
Fix the kvm code in arm by using this latter form of callback registration.
Cc: Christoffer Dall christoffer.d...@linaro.org
Cc: Gleb Natapov g...@kernel.org
Cc: Russell King li...@arm.linux.org.uk
Cc: Ingo Molnar mi...@kernel.org
Cc: kvm...@lists.cs.columbia.edu
Cc: k...@vger.kernel.org
Cc: linux-arm-ker...@lists.infradead.org
Acked-by: Paolo Bonzini pbonz...@redhat.com
Signed-off-by: Srivatsa S. Bhat srivatsa.b...@linux.vnet.ibm.com
---
arch/arm/kvm/arm.c |7 ++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
index bd18bb8..f0e50a0 100644
--- a/arch/arm/kvm/arm.c
+++ b/arch/arm/kvm/arm.c
@@ -1051,21 +1051,26 @@ int kvm_arch_init(void *opaque)
}
}
+ cpu_notifier_register_begin();
+
err = init_hyp_mode();
if (err)
goto out_err;
- err = register_cpu_notifier(hyp_init_cpu_nb);
+ err = __register_cpu_notifier(hyp_init_cpu_nb);
if (err) {
kvm_err(Cannot register HYP init CPU notifier (%d)\n, err);
goto out_err;
}
+ cpu_notifier_register_done();
+
hyp_cpu_pm_init();
kvm_coproc_table_init();
return 0;
out_err:
+ cpu_notifier_register_done();
return err;
}
Just so we're clear, the existing code was simply racy as not prone to
deadlocks, right?
This makes it clear that the test above for compatible CPUs can be quite
easily evaded by using CPU hotplug, but we don't really have a good
solution for handling that yet... Hmmm, grumble grumble, I guess if you
hotplug unsupported CPUs on a KVM/ARM system for now, stuff will break.
In any case:
Acked-by: Christoffer Dall christoffer.d...@linaro.org
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
