Re: [PATCH 1/2] KVM: X86: Fix setup the virt_spin_lock_key before static key get initialized
2018-03-24 21:17-0700, Wanpeng Li: > From: Wanpeng Li > > static_key_disable_cpuslocked(): static key 'virt_spin_lock_key+0x0/0x20' > used before call to jump_label_init() > WARNING: CPU: 0 PID: 0 at kernel/jump_label.c:161 > static_key_disable_cpuslocked+0x61/0x80 > RIP: 0010:static_key_disable_cpuslocked+0x61/0x80 > Call Trace: > static_key_disable+0x16/0x20 > start_kernel+0x192/0x4b3 > secondary_startup_64+0xa5/0xb0 > > Qspinlock will be choosed when dedicated pCPUs are available, however, the > static virt_spin_lock_key is set in kvm_spinlock_init() before > jump_label_init() > has been called, which will result in a WARN(). This patch fixes it by > delaying > the virt_spin_lock_key setup to .smp_prepare_cpus(). > > Reported-by: Davidlohr Bueso > Cc: Paolo Bonzini > Cc: Radim Krčmář > Cc: Davidlohr Bueso > Signed-off-by: Wanpeng Li > --- Queued, thanks.
Re: [PATCH 1/2] KVM: X86: Fix setup the virt_spin_lock_key before static key get initialized
2018-03-28 0:29 GMT+08:00 Davidlohr Bueso : > On Tue, 27 Mar 2018, Wanpeng Li wrote: > >> The host admin can control it by qemu command-line, -cpu >> ...+kvm-hint-dedicated > > > So this requires qemu code modification, no? Yeah, https://git.qemu.org/?p=qemu.git;a=commitdiff;h=be7773268d98176489483a315d3e2323cb0615b9 Regards, Wanpeng Li
Re: [PATCH 1/2] KVM: X86: Fix setup the virt_spin_lock_key before static key get initialized
On Tue, 27 Mar 2018, Wanpeng Li wrote: The host admin can control it by qemu command-line, -cpu ...+kvm-hint-dedicated So this requires qemu code modification, no?
Re: [PATCH 1/2] KVM: X86: Fix setup the virt_spin_lock_key before static key get initialized
2018-03-27 6:32 GMT+08:00 Davidlohr Bueso : > On Mon, 26 Mar 2018, Wanpeng Li wrote: > >> I didn't see any issue when testing this patch, could you elaborate >> what's the bulky mechanism and how it conflicts with early smp bootup >> stages? In addition, do you mean the xen fix is also not suitable? > > > I have nothing against your patch (or the xen one for that matter), other > than > the fact that both seem like band-aid solutions to being able to move up the > jump > label init call. I was actually peddling a similar patch but didn't want to > add > the extra callback just for that -- at least xen already had > smp_ops.smp_prepare_cpus. > > Instead of dropping the patches in -tip (ie: delaying the feature), I have > nothing > against these fixes being merged. If a better solution is available later, > we can > always move dealing with virt_spin_lock_key back into > smp_prepare_boot_cpu(). Yeah, we can pick the current patch for the fixes and wait a better solution is available later. > > Also a bit unrelated, but am I correct to assume that KVM_HINTS_DEDICATED > can be > used via qemu? I've only been dealing with kernel parameters. The host admin can control it by qemu command-line, -cpu ...+kvm-hint-dedicated Regards, Wanpeng Li
Re: [PATCH 1/2] KVM: X86: Fix setup the virt_spin_lock_key before static key get initialized
On Mon, 26 Mar 2018, Wanpeng Li wrote: I didn't see any issue when testing this patch, could you elaborate what's the bulky mechanism and how it conflicts with early smp bootup stages? In addition, do you mean the xen fix is also not suitable? I have nothing against your patch (or the xen one for that matter), other than the fact that both seem like band-aid solutions to being able to move up the jump label init call. I was actually peddling a similar patch but didn't want to add the extra callback just for that -- at least xen already had smp_ops.smp_prepare_cpus. Instead of dropping the patches in -tip (ie: delaying the feature), I have nothing against these fixes being merged. If a better solution is available later, we can always move dealing with virt_spin_lock_key back into smp_prepare_boot_cpu(). Also a bit unrelated, but am I correct to assume that KVM_HINTS_DEDICATED can be used via qemu? I've only been dealing with kernel parameters. Thanks, Davidlohr
Re: [PATCH 1/2] KVM: X86: Fix setup the virt_spin_lock_key before static key get initialized
2018-03-26 8:27 GMT+08:00 Davidlohr Bueso : > On Sat, 24 Mar 2018, Wanpeng Li wrote: > >> Note: Peterz pointed out in the IRC we have to audit all the architectures >> that >> implement smp_prepare_boot_cpu() to see what they depend on if we want to >> move >> jump_label_init() before smp_prepare_boot_cpu(). So what this patch does >> is >> similar to the issue which handled in xen ca5d376e. > > > After some auditing, the jump_label_init() being moved before > smp_prepare_boot_cpu() > seems fine, however, bulky mechanisms to update text segments conflict with > early smp > bootup stages, such as this patch. So, while the disabling I didn't see any issue when testing this patch, could you elaborate what's the bulky mechanism and how it conflicts with early smp bootup stages? In addition, do you mean the xen fix is also not suitable? Regards, Wanpeng Li
Re: [PATCH 1/2] KVM: X86: Fix setup the virt_spin_lock_key before static key get initialized
On Sat, 24 Mar 2018, Wanpeng Li wrote: Note: Peterz pointed out in the IRC we have to audit all the architectures that implement smp_prepare_boot_cpu() to see what they depend on if we want to move jump_label_init() before smp_prepare_boot_cpu(). So what this patch does is similar to the issue which handled in xen ca5d376e. After some auditing, the jump_label_init() being moved before smp_prepare_boot_cpu() seems fine, however, bulky mechanisms to update text segments conflict with early smp bootup stages, such as this patch. So, while the disabling virt_spin_lock_key would be done correctly _after_ jump_label_init(), it is still fragile in that we want to be using lightweight patching such as jump_label_transform_static() -- which doesn't take the text_mutex (blocking is out of the question), for example. For pretty much all archs this means using the transform_static() version. For example x86, this means using text_poke_early(). -- also ouchy on the !PageReserved(pages[0]) warning for text_poke(). I'm not sure yet of the best way to teach jump_label_transform() to behave like jump_label_transform_static() under pre-smp bootup, such as when disabling hypervisor pvspinlocks. The s390 implementation seems safe as is given that stop_machine is now safe for early pre-smp boot. Thanks, Davidlohr
[PATCH 1/2] KVM: X86: Fix setup the virt_spin_lock_key before static key get initialized
From: Wanpeng Li
static_key_disable_cpuslocked(): static key 'virt_spin_lock_key+0x0/0x20' used
before call to jump_label_init()
WARNING: CPU: 0 PID: 0 at kernel/jump_label.c:161
static_key_disable_cpuslocked+0x61/0x80
RIP: 0010:static_key_disable_cpuslocked+0x61/0x80
Call Trace:
static_key_disable+0x16/0x20
start_kernel+0x192/0x4b3
secondary_startup_64+0xa5/0xb0
Qspinlock will be choosed when dedicated pCPUs are available, however, the
static virt_spin_lock_key is set in kvm_spinlock_init() before
jump_label_init()
has been called, which will result in a WARN(). This patch fixes it by delaying
the virt_spin_lock_key setup to .smp_prepare_cpus().
Reported-by: Davidlohr Bueso
Cc: Paolo Bonzini
Cc: Radim Krčmář
Cc: Davidlohr Bueso
Signed-off-by: Wanpeng Li
---
Note: Peterz pointed out in the IRC we have to audit all the architectures that
implement smp_prepare_boot_cpu() to see what they depend on if we want to move
jump_label_init() before smp_prepare_boot_cpu(). So what this patch does is
similar to the issue which handled in xen ca5d376e.
arch/x86/kernel/kvm.c | 12 +---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c
index 4ccbff6..747c61b 100644
--- a/arch/x86/kernel/kvm.c
+++ b/arch/x86/kernel/kvm.c
@@ -454,6 +454,13 @@ static void __init sev_map_percpu_data(void)
}
#ifdef CONFIG_SMP
+static void __init kvm_smp_prepare_cpus(unsigned int max_cpus)
+{
+ native_smp_prepare_cpus(max_cpus);
+ if (kvm_para_has_hint(KVM_HINTS_DEDICATED))
+ static_branch_disable(&virt_spin_lock_key);
+}
+
static void __init kvm_smp_prepare_boot_cpu(void)
{
/*
@@ -557,6 +564,7 @@ static void __init kvm_guest_init(void)
kvm_setup_vsyscall_timeinfo();
#ifdef CONFIG_SMP
+ smp_ops.smp_prepare_cpus = kvm_smp_prepare_cpus;
smp_ops.smp_prepare_boot_cpu = kvm_smp_prepare_boot_cpu;
if (cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN, "x86/kvm:online",
kvm_cpu_online, kvm_cpu_down_prepare) < 0)
@@ -737,10 +745,8 @@ void __init kvm_spinlock_init(void)
if (!kvm_para_has_feature(KVM_FEATURE_PV_UNHALT))
return;
- if (kvm_para_has_hint(KVM_HINTS_DEDICATED)) {
- static_branch_disable(&virt_spin_lock_key);
+ if (kvm_para_has_hint(KVM_HINTS_DEDICATED))
return;
- }
__pv_init_lock_hash();
pv_lock_ops.queued_spin_lock_slowpath = __pv_queued_spin_lock_slowpath;
--
2.7.4
