Re: [PATCH 2/2] KVM: x86: allow BSP to handle INIT IPIs like APs do
>>> On 2/8/2016 at 10:53 AM, Jan Kiszka wrote: > On 2016-02-08 18:38, Bruce Rogers wrote: > On 2/8/2016 at 10:27 AM, Bruce Rogers wrote: >> On 2/8/2016 at 09:40 AM, Paolo Bonzini wrote: >>> On 08/02/2016 17:33, Bruce Rogers wrote: > > KVM_MP_STATE_INIT_RECEIVED is what Intel calls the "wait for SIPI" > state. The BSP never gets a SIPI, it goes straight to 0xFFF0 > instead. Can you explain the problem more in detail? >>> >>> I suspect this is about sending INIT-SIPI from another CPU, directed to >>> the BSP, isn't it? We may have to differentiate between CPU (including >>> system) reset and that IPI case. > That is correct. In looking over the KVM code which deals with BSP, this > was > the only place which seemed wrong to me wrt special casing for BSP > outside the > context of initial system initialization / reset. As far as I understand > the > BSP shouldn't be treated differently in this case. See 8.4.2 of the SDM: If the MP protocol has completed and a BSP is chosen, subsequent INITs (either to a specific processor or system wide) do not cause the MP protocol to be repeated. Instead, each logical processor examines its BSP flag (in the IA32_APIC_BASE MSR) to determine whether it should execute the BIOS boot-strap code (if it is the BSP) or enter a wait-for-SIPI state (if it is an AP). So it is correct to treat the BSP differently here, I think. >>> >>> I had read that, but I though this was speaking from the perspective of the >>> SMP aware BIOS code only. In other words, the BIOS would sidetrack AP's >>> (based on BSP flag not being present), while BSP would be allowed to go >>> through >>> the regular BIOS code, checking for reset case, etc. An OS on the other hand >>> would be free to treat all x86 processors equally, once it has booted into >>> fully symmetrical mode. >>> I certainly could be wrong about my above interpretation, but with these >>> changes I'm proposing, things work well for the test case of manually >>> onlining >>> the BSP after the crash kernel has been started (via kexec -e on a AP >>> processor >>> with maxcpus=1 on the crash kernel command line). From looking through the >>> kernel git history it appears this sequence of events was explicitly >>> supported >>> quite a while ago, and we've got a customer who uses this for fast recovery >>> from >>> a guest kernel crash. >>> >>> Bruce >> >> I mean kexec - p ... above, not kexec -e. Sorry about that. > > How does real HW behave with your kexec case? Did you try this? I appreciate the review of these patches. I think I was premature in sharing these patches, not having more fully vetting our customer's claims, and by trying to satisfy their request that it also work the same under KVM Virtualization without more complete details. I had initially tried it on real hardware I have and it didn't behave as the customer claimed, but mistakenly I dropped that concern, More recent testing on additional machines continues to show that this approach does not work on real hardware, and I think we want KVM to act like real hardware here. I've asked for more details from the customer on their hardware and the specifics of their use case. I'm also taking more time to get familiar with how this part of KVM works. Thanks, Bruce
Re: [PATCH 2/2] KVM: x86: allow BSP to handle INIT IPIs like APs do
>>> On 2/8/2016 at 10:53 AM, Jan Kiszkawrote: > On 2016-02-08 18:38, Bruce Rogers wrote: > On 2/8/2016 at 10:27 AM, Bruce Rogers wrote: >> On 2/8/2016 at 09:40 AM, Paolo Bonzini wrote: >>> On 08/02/2016 17:33, Bruce Rogers wrote: > > KVM_MP_STATE_INIT_RECEIVED is what Intel calls the "wait for SIPI" > state. The BSP never gets a SIPI, it goes straight to 0xFFF0 > instead. Can you explain the problem more in detail? >>> >>> I suspect this is about sending INIT-SIPI from another CPU, directed to >>> the BSP, isn't it? We may have to differentiate between CPU (including >>> system) reset and that IPI case. > That is correct. In looking over the KVM code which deals with BSP, this > was > the only place which seemed wrong to me wrt special casing for BSP > outside the > context of initial system initialization / reset. As far as I understand > the > BSP shouldn't be treated differently in this case. See 8.4.2 of the SDM: If the MP protocol has completed and a BSP is chosen, subsequent INITs (either to a specific processor or system wide) do not cause the MP protocol to be repeated. Instead, each logical processor examines its BSP flag (in the IA32_APIC_BASE MSR) to determine whether it should execute the BIOS boot-strap code (if it is the BSP) or enter a wait-for-SIPI state (if it is an AP). So it is correct to treat the BSP differently here, I think. >>> >>> I had read that, but I though this was speaking from the perspective of the >>> SMP aware BIOS code only. In other words, the BIOS would sidetrack AP's >>> (based on BSP flag not being present), while BSP would be allowed to go >>> through >>> the regular BIOS code, checking for reset case, etc. An OS on the other hand >>> would be free to treat all x86 processors equally, once it has booted into >>> fully symmetrical mode. >>> I certainly could be wrong about my above interpretation, but with these >>> changes I'm proposing, things work well for the test case of manually >>> onlining >>> the BSP after the crash kernel has been started (via kexec -e on a AP >>> processor >>> with maxcpus=1 on the crash kernel command line). From looking through the >>> kernel git history it appears this sequence of events was explicitly >>> supported >>> quite a while ago, and we've got a customer who uses this for fast recovery >>> from >>> a guest kernel crash. >>> >>> Bruce >> >> I mean kexec - p ... above, not kexec -e. Sorry about that. > > How does real HW behave with your kexec case? Did you try this? I appreciate the review of these patches. I think I was premature in sharing these patches, not having more fully vetting our customer's claims, and by trying to satisfy their request that it also work the same under KVM Virtualization without more complete details. I had initially tried it on real hardware I have and it didn't behave as the customer claimed, but mistakenly I dropped that concern, More recent testing on additional machines continues to show that this approach does not work on real hardware, and I think we want KVM to act like real hardware here. I've asked for more details from the customer on their hardware and the specifics of their use case. I'm also taking more time to get familiar with how this part of KVM works. Thanks, Bruce
Re: [PATCH 2/2] KVM: x86: allow BSP to handle INIT IPIs like APs do
On 2016-02-08 18:38, Bruce Rogers wrote: On 2/8/2016 at 10:27 AM, Bruce Rogers wrote: > On 2/8/2016 at 09:40 AM, Paolo Bonzini wrote: >> >>> >>> On 08/02/2016 17:33, Bruce Rogers wrote: KVM_MP_STATE_INIT_RECEIVED is what Intel calls the "wait for SIPI" state. The BSP never gets a SIPI, it goes straight to 0xFFF0 instead. Can you explain the problem more in detail? >> >> I suspect this is about sending INIT-SIPI from another CPU, directed to >> the BSP, isn't it? We may have to differentiate between CPU (including >> system) reset and that IPI case. That is correct. In looking over the KVM code which deals with BSP, this was the only place which seemed wrong to me wrt special casing for BSP outside >>> the context of initial system initialization / reset. As far as I understand the BSP shouldn't be treated differently in this case. >>> >>> See 8.4.2 of the SDM: >>> >>> If the MP protocol has completed and a BSP is chosen, subsequent INITs >>> (either to a specific processor or system wide) do not cause the MP >>> protocol to be repeated. Instead, each logical processor examines its >>> BSP flag (in the IA32_APIC_BASE MSR) to determine whether it should >>> execute the BIOS boot-strap code (if it is the BSP) or enter a >>> wait-for-SIPI state (if it is an AP). >>> >>> So it is correct to treat the BSP differently here, I think. >> >> I had read that, but I though this was speaking from the perspective of the >> SMP aware BIOS code only. In other words, the BIOS would sidetrack AP's >> (based on BSP flag not being present), while BSP would be allowed to go >> through >> the regular BIOS code, checking for reset case, etc. An OS on the other hand >> would be free to treat all x86 processors equally, once it has booted into >> fully symmetrical mode. >> I certainly could be wrong about my above interpretation, but with these >> changes I'm proposing, things work well for the test case of manually >> onlining >> the BSP after the crash kernel has been started (via kexec -e on a AP >> processor >> with maxcpus=1 on the crash kernel command line). From looking through the >> kernel git history it appears this sequence of events was explicitly >> supported >> quite a while ago, and we've got a customer who uses this for fast recovery >> from >> a guest kernel crash. >> >> Bruce > > I mean kexec - p ... above, not kexec -e. Sorry about that. How does real HW behave with your kexec case? Did you try this? Jan signature.asc Description: OpenPGP digital signature
Re: [PATCH 2/2] KVM: x86: allow BSP to handle INIT IPIs like APs do
On 08/02/2016 18:27, Bruce Rogers wrote: > I had read that, but I though this was speaking from the perspective of the > SMP aware BIOS code only. It says "logical processor", so I cannot really see how it can be interpreted that way. The BSP jumps to 0xFFF0, the APs go into wait-for-SIPI state. > I certainly could be wrong about my above interpretation, but with these > changes I'm proposing, things work well for the test case of manually onlining > the BSP after the crash kernel has been started (via kexec -e on a AP > processor > with maxcpus=1 on the crash kernel command line). From looking through the > kernel git history it appears this sequence of events was explicitly supported > quite a while ago, and we've got a customer who uses this for fast recovery > from > a guest kernel crash. You need to comment on the output of trace-cmd for KVM events, or provide a full reproducer, or at the very least point me to the kernel code that you're referring to. Otherwise I just cannot understand what you're talking about; sorry. :( Paolo
Re: [PATCH 2/2] KVM: x86: allow BSP to handle INIT IPIs like APs do
>>> On 2/8/2016 at 10:27 AM, Bruce Rogers wrote: > >>> On 2/8/2016 at 09:40 AM, Paolo Bonzini wrote: > >> >> On 08/02/2016 17:33, Bruce Rogers wrote: > >> > >> KVM_MP_STATE_INIT_RECEIVED is what Intel calls the "wait for SIPI" > >> state. The BSP never gets a SIPI, it goes straight to 0xFFF0 > >> instead. Can you explain the problem more in detail? > > I suspect this is about sending INIT-SIPI from another CPU, directed to > the BSP, isn't it? We may have to differentiate between CPU (including > system) reset and that IPI case. >>> That is correct. In looking over the KVM code which deals with BSP, this was >>> the only place which seemed wrong to me wrt special casing for BSP outside >> the >>> context of initial system initialization / reset. As far as I understand the >>> BSP shouldn't be treated differently in this case. >> >> See 8.4.2 of the SDM: >> >> If the MP protocol has completed and a BSP is chosen, subsequent INITs >> (either to a specific processor or system wide) do not cause the MP >> protocol to be repeated. Instead, each logical processor examines its >> BSP flag (in the IA32_APIC_BASE MSR) to determine whether it should >> execute the BIOS boot-strap code (if it is the BSP) or enter a >> wait-for-SIPI state (if it is an AP). >> >> So it is correct to treat the BSP differently here, I think. > > I had read that, but I though this was speaking from the perspective of the > SMP aware BIOS code only. In other words, the BIOS would sidetrack AP's > (based on BSP flag not being present), while BSP would be allowed to go > through > the regular BIOS code, checking for reset case, etc. An OS on the other hand > would be free to treat all x86 processors equally, once it has booted into > fully symmetrical mode. > I certainly could be wrong about my above interpretation, but with these > changes I'm proposing, things work well for the test case of manually > onlining > the BSP after the crash kernel has been started (via kexec -e on a AP > processor > with maxcpus=1 on the crash kernel command line). From looking through the > kernel git history it appears this sequence of events was explicitly > supported > quite a while ago, and we've got a customer who uses this for fast recovery > from > a guest kernel crash. > > Bruce I mean kexec - p ... above, not kexec -e. Sorry about that. Bruce
Re: [PATCH 2/2] KVM: x86: allow BSP to handle INIT IPIs like APs do
>>> On 2/8/2016 at 09:40 AM, Paolo Bonzini wrote: > > On 08/02/2016 17:33, Bruce Rogers wrote: >> >> KVM_MP_STATE_INIT_RECEIVED is what Intel calls the "wait for SIPI" >> state. The BSP never gets a SIPI, it goes straight to 0xFFF0 >> instead. Can you explain the problem more in detail? >>> > >>> > I suspect this is about sending INIT-SIPI from another CPU, directed to >>> > the BSP, isn't it? We may have to differentiate between CPU (including >>> > system) reset and that IPI case. >> That is correct. In looking over the KVM code which deals with BSP, this was >> the only place which seemed wrong to me wrt special casing for BSP outside > the >> context of initial system initialization / reset. As far as I understand the >> BSP shouldn't be treated differently in this case. > > See 8.4.2 of the SDM: > > If the MP protocol has completed and a BSP is chosen, subsequent INITs > (either to a specific processor or system wide) do not cause the MP > protocol to be repeated. Instead, each logical processor examines its > BSP flag (in the IA32_APIC_BASE MSR) to determine whether it should > execute the BIOS boot-strap code (if it is the BSP) or enter a > wait-for-SIPI state (if it is an AP). > > So it is correct to treat the BSP differently here, I think. I had read that, but I though this was speaking from the perspective of the SMP aware BIOS code only. In other words, the BIOS would sidetrack AP's (based on BSP flag not being present), while BSP would be allowed to go through the regular BIOS code, checking for reset case, etc. An OS on the other hand would be free to treat all x86 processors equally, once it has booted into fully symmetrical mode. I certainly could be wrong about my above interpretation, but with these changes I'm proposing, things work well for the test case of manually onlining the BSP after the crash kernel has been started (via kexec -e on a AP processor with maxcpus=1 on the crash kernel command line). From looking through the kernel git history it appears this sequence of events was explicitly supported quite a while ago, and we've got a customer who uses this for fast recovery from a guest kernel crash. Bruce
Re: [PATCH 2/2] KVM: x86: allow BSP to handle INIT IPIs like APs do
On 08/02/2016 17:33, Bruce Rogers wrote: >>> >> >>> >> KVM_MP_STATE_INIT_RECEIVED is what Intel calls the "wait for SIPI" >>> >> state. The BSP never gets a SIPI, it goes straight to 0xFFF0 >>> >> instead. Can you explain the problem more in detail? >> > >> > I suspect this is about sending INIT-SIPI from another CPU, directed to >> > the BSP, isn't it? We may have to differentiate between CPU (including >> > system) reset and that IPI case. > That is correct. In looking over the KVM code which deals with BSP, this was > the only place which seemed wrong to me wrt special casing for BSP outside the > context of initial system initialization / reset. As far as I understand the > BSP shouldn't be treated differently in this case. See 8.4.2 of the SDM: If the MP protocol has completed and a BSP is chosen, subsequent INITs (either to a specific processor or system wide) do not cause the MP protocol to be repeated. Instead, each logical processor examines its BSP flag (in the IA32_APIC_BASE MSR) to determine whether it should execute the BIOS boot-strap code (if it is the BSP) or enter a wait-for-SIPI state (if it is an AP). So it is correct to treat the BSP differently here, I think. Paolo
Re: [PATCH 2/2] KVM: x86: allow BSP to handle INIT IPIs like APs do
>>> On 2/8/2016 at 08:22 AM, Jan Kiszka wrote: > On 2016-02-08 16:12, Paolo Bonzini wrote: >> >> >> On 03/02/2016 23:51, Bruce Rogers wrote: >>> The INIT IPI event handler special cases the boot-strap processor >>> (BSP) handling, avoiding the same mp state handling which is done for >>> the other (AP) processors. Debugging a linux guest usage scenario of >>> avoiding a reboot through the bios for a crash on any processor via eg: >>> kexec -p /boot/vmlinuz --initrd=/boot/initrd --append="$(cat /proc/cmdline)\ >>> maxcpus=1" led to identifying this change as the needed fix. >>> >>> With this change, an AP can now startup the BSP without error. >>> >>> Signed-off-by: Bruce Rogers >>> --- >>> arch/x86/kvm/lapic.c | 5 + >>> 1 file changed, 1 insertion(+), 4 deletions(-) >>> >>> diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c >>> index 36591fa..eda6bfb 100644 >>> --- a/arch/x86/kvm/lapic.c >>> +++ b/arch/x86/kvm/lapic.c >>> @@ -2170,10 +2170,7 @@ void kvm_apic_accept_events(struct kvm_vcpu *vcpu) >>> if (test_bit(KVM_APIC_INIT, )) { >>> kvm_lapic_reset(vcpu, true); >>> kvm_vcpu_reset(vcpu, true); >>> - if (kvm_vcpu_is_bsp(apic->vcpu)) >>> - vcpu->arch.mp_state = KVM_MP_STATE_RUNNABLE; >>> - else >>> - vcpu->arch.mp_state = KVM_MP_STATE_INIT_RECEIVED; >>> + vcpu->arch.mp_state = KVM_MP_STATE_INIT_RECEIVED; >>> } >>> if (test_bit(KVM_APIC_SIPI, ) && >>> vcpu->arch.mp_state == KVM_MP_STATE_INIT_RECEIVED) { >>> >> >> KVM_MP_STATE_INIT_RECEIVED is what Intel calls the "wait for SIPI" >> state. The BSP never gets a SIPI, it goes straight to 0xFFF0 >> instead. Can you explain the problem more in detail? > > I suspect this is about sending INIT-SIPI from another CPU, directed to > the BSP, isn't it? We may have to differentiate between CPU (including > system) reset and that IPI case. That is correct. In looking over the KVM code which deals with BSP, this was the only place which seemed wrong to me wrt special casing for BSP outside the context of initial system initialization / reset. As far as I understand the BSP shouldn't be treated differently in this case. Bruce
Re: [PATCH 2/2] KVM: x86: allow BSP to handle INIT IPIs like APs do
On 2016-02-08 16:12, Paolo Bonzini wrote: > > > On 03/02/2016 23:51, Bruce Rogers wrote: >> The INIT IPI event handler special cases the boot-strap processor >> (BSP) handling, avoiding the same mp state handling which is done for >> the other (AP) processors. Debugging a linux guest usage scenario of >> avoiding a reboot through the bios for a crash on any processor via eg: >> kexec -p /boot/vmlinuz --initrd=/boot/initrd --append="$(cat /proc/cmdline)\ >> maxcpus=1" led to identifying this change as the needed fix. >> >> With this change, an AP can now startup the BSP without error. >> >> Signed-off-by: Bruce Rogers >> --- >> arch/x86/kvm/lapic.c | 5 + >> 1 file changed, 1 insertion(+), 4 deletions(-) >> >> diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c >> index 36591fa..eda6bfb 100644 >> --- a/arch/x86/kvm/lapic.c >> +++ b/arch/x86/kvm/lapic.c >> @@ -2170,10 +2170,7 @@ void kvm_apic_accept_events(struct kvm_vcpu *vcpu) >> if (test_bit(KVM_APIC_INIT, )) { >> kvm_lapic_reset(vcpu, true); >> kvm_vcpu_reset(vcpu, true); >> -if (kvm_vcpu_is_bsp(apic->vcpu)) >> -vcpu->arch.mp_state = KVM_MP_STATE_RUNNABLE; >> -else >> -vcpu->arch.mp_state = KVM_MP_STATE_INIT_RECEIVED; >> +vcpu->arch.mp_state = KVM_MP_STATE_INIT_RECEIVED; >> } >> if (test_bit(KVM_APIC_SIPI, ) && >> vcpu->arch.mp_state == KVM_MP_STATE_INIT_RECEIVED) { >> > > KVM_MP_STATE_INIT_RECEIVED is what Intel calls the "wait for SIPI" > state. The BSP never gets a SIPI, it goes straight to 0xFFF0 > instead. Can you explain the problem more in detail? I suspect this is about sending INIT-SIPI from another CPU, directed to the BSP, isn't it? We may have to differentiate between CPU (including system) reset and that IPI case. Jan signature.asc Description: OpenPGP digital signature
Re: [PATCH 2/2] KVM: x86: allow BSP to handle INIT IPIs like APs do
On 03/02/2016 23:51, Bruce Rogers wrote: > The INIT IPI event handler special cases the boot-strap processor > (BSP) handling, avoiding the same mp state handling which is done for > the other (AP) processors. Debugging a linux guest usage scenario of > avoiding a reboot through the bios for a crash on any processor via eg: > kexec -p /boot/vmlinuz --initrd=/boot/initrd --append="$(cat /proc/cmdline)\ > maxcpus=1" led to identifying this change as the needed fix. > > With this change, an AP can now startup the BSP without error. > > Signed-off-by: Bruce Rogers > --- > arch/x86/kvm/lapic.c | 5 + > 1 file changed, 1 insertion(+), 4 deletions(-) > > diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c > index 36591fa..eda6bfb 100644 > --- a/arch/x86/kvm/lapic.c > +++ b/arch/x86/kvm/lapic.c > @@ -2170,10 +2170,7 @@ void kvm_apic_accept_events(struct kvm_vcpu *vcpu) > if (test_bit(KVM_APIC_INIT, )) { > kvm_lapic_reset(vcpu, true); > kvm_vcpu_reset(vcpu, true); > - if (kvm_vcpu_is_bsp(apic->vcpu)) > - vcpu->arch.mp_state = KVM_MP_STATE_RUNNABLE; > - else > - vcpu->arch.mp_state = KVM_MP_STATE_INIT_RECEIVED; > + vcpu->arch.mp_state = KVM_MP_STATE_INIT_RECEIVED; > } > if (test_bit(KVM_APIC_SIPI, ) && > vcpu->arch.mp_state == KVM_MP_STATE_INIT_RECEIVED) { > KVM_MP_STATE_INIT_RECEIVED is what Intel calls the "wait for SIPI" state. The BSP never gets a SIPI, it goes straight to 0xFFF0 instead. Can you explain the problem more in detail? Paolo
Re: [PATCH 2/2] KVM: x86: allow BSP to handle INIT IPIs like APs do
>>> On 2/8/2016 at 08:22 AM, Jan Kiszkawrote: > On 2016-02-08 16:12, Paolo Bonzini wrote: >> >> >> On 03/02/2016 23:51, Bruce Rogers wrote: >>> The INIT IPI event handler special cases the boot-strap processor >>> (BSP) handling, avoiding the same mp state handling which is done for >>> the other (AP) processors. Debugging a linux guest usage scenario of >>> avoiding a reboot through the bios for a crash on any processor via eg: >>> kexec -p /boot/vmlinuz --initrd=/boot/initrd --append="$(cat /proc/cmdline)\ >>> maxcpus=1" led to identifying this change as the needed fix. >>> >>> With this change, an AP can now startup the BSP without error. >>> >>> Signed-off-by: Bruce Rogers >>> --- >>> arch/x86/kvm/lapic.c | 5 + >>> 1 file changed, 1 insertion(+), 4 deletions(-) >>> >>> diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c >>> index 36591fa..eda6bfb 100644 >>> --- a/arch/x86/kvm/lapic.c >>> +++ b/arch/x86/kvm/lapic.c >>> @@ -2170,10 +2170,7 @@ void kvm_apic_accept_events(struct kvm_vcpu *vcpu) >>> if (test_bit(KVM_APIC_INIT, )) { >>> kvm_lapic_reset(vcpu, true); >>> kvm_vcpu_reset(vcpu, true); >>> - if (kvm_vcpu_is_bsp(apic->vcpu)) >>> - vcpu->arch.mp_state = KVM_MP_STATE_RUNNABLE; >>> - else >>> - vcpu->arch.mp_state = KVM_MP_STATE_INIT_RECEIVED; >>> + vcpu->arch.mp_state = KVM_MP_STATE_INIT_RECEIVED; >>> } >>> if (test_bit(KVM_APIC_SIPI, ) && >>> vcpu->arch.mp_state == KVM_MP_STATE_INIT_RECEIVED) { >>> >> >> KVM_MP_STATE_INIT_RECEIVED is what Intel calls the "wait for SIPI" >> state. The BSP never gets a SIPI, it goes straight to 0xFFF0 >> instead. Can you explain the problem more in detail? > > I suspect this is about sending INIT-SIPI from another CPU, directed to > the BSP, isn't it? We may have to differentiate between CPU (including > system) reset and that IPI case. That is correct. In looking over the KVM code which deals with BSP, this was the only place which seemed wrong to me wrt special casing for BSP outside the context of initial system initialization / reset. As far as I understand the BSP shouldn't be treated differently in this case. Bruce
Re: [PATCH 2/2] KVM: x86: allow BSP to handle INIT IPIs like APs do
On 08/02/2016 17:33, Bruce Rogers wrote: >>> >> >>> >> KVM_MP_STATE_INIT_RECEIVED is what Intel calls the "wait for SIPI" >>> >> state. The BSP never gets a SIPI, it goes straight to 0xFFF0 >>> >> instead. Can you explain the problem more in detail? >> > >> > I suspect this is about sending INIT-SIPI from another CPU, directed to >> > the BSP, isn't it? We may have to differentiate between CPU (including >> > system) reset and that IPI case. > That is correct. In looking over the KVM code which deals with BSP, this was > the only place which seemed wrong to me wrt special casing for BSP outside the > context of initial system initialization / reset. As far as I understand the > BSP shouldn't be treated differently in this case. See 8.4.2 of the SDM: If the MP protocol has completed and a BSP is chosen, subsequent INITs (either to a specific processor or system wide) do not cause the MP protocol to be repeated. Instead, each logical processor examines its BSP flag (in the IA32_APIC_BASE MSR) to determine whether it should execute the BIOS boot-strap code (if it is the BSP) or enter a wait-for-SIPI state (if it is an AP). So it is correct to treat the BSP differently here, I think. Paolo
Re: [PATCH 2/2] KVM: x86: allow BSP to handle INIT IPIs like APs do
On 03/02/2016 23:51, Bruce Rogers wrote: > The INIT IPI event handler special cases the boot-strap processor > (BSP) handling, avoiding the same mp state handling which is done for > the other (AP) processors. Debugging a linux guest usage scenario of > avoiding a reboot through the bios for a crash on any processor via eg: > kexec -p /boot/vmlinuz --initrd=/boot/initrd --append="$(cat /proc/cmdline)\ > maxcpus=1" led to identifying this change as the needed fix. > > With this change, an AP can now startup the BSP without error. > > Signed-off-by: Bruce Rogers> --- > arch/x86/kvm/lapic.c | 5 + > 1 file changed, 1 insertion(+), 4 deletions(-) > > diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c > index 36591fa..eda6bfb 100644 > --- a/arch/x86/kvm/lapic.c > +++ b/arch/x86/kvm/lapic.c > @@ -2170,10 +2170,7 @@ void kvm_apic_accept_events(struct kvm_vcpu *vcpu) > if (test_bit(KVM_APIC_INIT, )) { > kvm_lapic_reset(vcpu, true); > kvm_vcpu_reset(vcpu, true); > - if (kvm_vcpu_is_bsp(apic->vcpu)) > - vcpu->arch.mp_state = KVM_MP_STATE_RUNNABLE; > - else > - vcpu->arch.mp_state = KVM_MP_STATE_INIT_RECEIVED; > + vcpu->arch.mp_state = KVM_MP_STATE_INIT_RECEIVED; > } > if (test_bit(KVM_APIC_SIPI, ) && > vcpu->arch.mp_state == KVM_MP_STATE_INIT_RECEIVED) { > KVM_MP_STATE_INIT_RECEIVED is what Intel calls the "wait for SIPI" state. The BSP never gets a SIPI, it goes straight to 0xFFF0 instead. Can you explain the problem more in detail? Paolo
Re: [PATCH 2/2] KVM: x86: allow BSP to handle INIT IPIs like APs do
On 2016-02-08 16:12, Paolo Bonzini wrote: > > > On 03/02/2016 23:51, Bruce Rogers wrote: >> The INIT IPI event handler special cases the boot-strap processor >> (BSP) handling, avoiding the same mp state handling which is done for >> the other (AP) processors. Debugging a linux guest usage scenario of >> avoiding a reboot through the bios for a crash on any processor via eg: >> kexec -p /boot/vmlinuz --initrd=/boot/initrd --append="$(cat /proc/cmdline)\ >> maxcpus=1" led to identifying this change as the needed fix. >> >> With this change, an AP can now startup the BSP without error. >> >> Signed-off-by: Bruce Rogers>> --- >> arch/x86/kvm/lapic.c | 5 + >> 1 file changed, 1 insertion(+), 4 deletions(-) >> >> diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c >> index 36591fa..eda6bfb 100644 >> --- a/arch/x86/kvm/lapic.c >> +++ b/arch/x86/kvm/lapic.c >> @@ -2170,10 +2170,7 @@ void kvm_apic_accept_events(struct kvm_vcpu *vcpu) >> if (test_bit(KVM_APIC_INIT, )) { >> kvm_lapic_reset(vcpu, true); >> kvm_vcpu_reset(vcpu, true); >> -if (kvm_vcpu_is_bsp(apic->vcpu)) >> -vcpu->arch.mp_state = KVM_MP_STATE_RUNNABLE; >> -else >> -vcpu->arch.mp_state = KVM_MP_STATE_INIT_RECEIVED; >> +vcpu->arch.mp_state = KVM_MP_STATE_INIT_RECEIVED; >> } >> if (test_bit(KVM_APIC_SIPI, ) && >> vcpu->arch.mp_state == KVM_MP_STATE_INIT_RECEIVED) { >> > > KVM_MP_STATE_INIT_RECEIVED is what Intel calls the "wait for SIPI" > state. The BSP never gets a SIPI, it goes straight to 0xFFF0 > instead. Can you explain the problem more in detail? I suspect this is about sending INIT-SIPI from another CPU, directed to the BSP, isn't it? We may have to differentiate between CPU (including system) reset and that IPI case. Jan signature.asc Description: OpenPGP digital signature
Re: [PATCH 2/2] KVM: x86: allow BSP to handle INIT IPIs like APs do
>>> On 2/8/2016 at 09:40 AM, Paolo Bonziniwrote: > > On 08/02/2016 17:33, Bruce Rogers wrote: >> >> KVM_MP_STATE_INIT_RECEIVED is what Intel calls the "wait for SIPI" >> state. The BSP never gets a SIPI, it goes straight to 0xFFF0 >> instead. Can you explain the problem more in detail? >>> > >>> > I suspect this is about sending INIT-SIPI from another CPU, directed to >>> > the BSP, isn't it? We may have to differentiate between CPU (including >>> > system) reset and that IPI case. >> That is correct. In looking over the KVM code which deals with BSP, this was >> the only place which seemed wrong to me wrt special casing for BSP outside > the >> context of initial system initialization / reset. As far as I understand the >> BSP shouldn't be treated differently in this case. > > See 8.4.2 of the SDM: > > If the MP protocol has completed and a BSP is chosen, subsequent INITs > (either to a specific processor or system wide) do not cause the MP > protocol to be repeated. Instead, each logical processor examines its > BSP flag (in the IA32_APIC_BASE MSR) to determine whether it should > execute the BIOS boot-strap code (if it is the BSP) or enter a > wait-for-SIPI state (if it is an AP). > > So it is correct to treat the BSP differently here, I think. I had read that, but I though this was speaking from the perspective of the SMP aware BIOS code only. In other words, the BIOS would sidetrack AP's (based on BSP flag not being present), while BSP would be allowed to go through the regular BIOS code, checking for reset case, etc. An OS on the other hand would be free to treat all x86 processors equally, once it has booted into fully symmetrical mode. I certainly could be wrong about my above interpretation, but with these changes I'm proposing, things work well for the test case of manually onlining the BSP after the crash kernel has been started (via kexec -e on a AP processor with maxcpus=1 on the crash kernel command line). From looking through the kernel git history it appears this sequence of events was explicitly supported quite a while ago, and we've got a customer who uses this for fast recovery from a guest kernel crash. Bruce
Re: [PATCH 2/2] KVM: x86: allow BSP to handle INIT IPIs like APs do
On 08/02/2016 18:27, Bruce Rogers wrote: > I had read that, but I though this was speaking from the perspective of the > SMP aware BIOS code only. It says "logical processor", so I cannot really see how it can be interpreted that way. The BSP jumps to 0xFFF0, the APs go into wait-for-SIPI state. > I certainly could be wrong about my above interpretation, but with these > changes I'm proposing, things work well for the test case of manually onlining > the BSP after the crash kernel has been started (via kexec -e on a AP > processor > with maxcpus=1 on the crash kernel command line). From looking through the > kernel git history it appears this sequence of events was explicitly supported > quite a while ago, and we've got a customer who uses this for fast recovery > from > a guest kernel crash. You need to comment on the output of trace-cmd for KVM events, or provide a full reproducer, or at the very least point me to the kernel code that you're referring to. Otherwise I just cannot understand what you're talking about; sorry. :( Paolo
Re: [PATCH 2/2] KVM: x86: allow BSP to handle INIT IPIs like APs do
>>> On 2/8/2016 at 10:27 AM, Bruce Rogers wrote: > >>> On 2/8/2016 at 09:40 AM, Paolo Bonziniwrote: > >> >> On 08/02/2016 17:33, Bruce Rogers wrote: > >> > >> KVM_MP_STATE_INIT_RECEIVED is what Intel calls the "wait for SIPI" > >> state. The BSP never gets a SIPI, it goes straight to 0xFFF0 > >> instead. Can you explain the problem more in detail? > > I suspect this is about sending INIT-SIPI from another CPU, directed to > the BSP, isn't it? We may have to differentiate between CPU (including > system) reset and that IPI case. >>> That is correct. In looking over the KVM code which deals with BSP, this was >>> the only place which seemed wrong to me wrt special casing for BSP outside >> the >>> context of initial system initialization / reset. As far as I understand the >>> BSP shouldn't be treated differently in this case. >> >> See 8.4.2 of the SDM: >> >> If the MP protocol has completed and a BSP is chosen, subsequent INITs >> (either to a specific processor or system wide) do not cause the MP >> protocol to be repeated. Instead, each logical processor examines its >> BSP flag (in the IA32_APIC_BASE MSR) to determine whether it should >> execute the BIOS boot-strap code (if it is the BSP) or enter a >> wait-for-SIPI state (if it is an AP). >> >> So it is correct to treat the BSP differently here, I think. > > I had read that, but I though this was speaking from the perspective of the > SMP aware BIOS code only. In other words, the BIOS would sidetrack AP's > (based on BSP flag not being present), while BSP would be allowed to go > through > the regular BIOS code, checking for reset case, etc. An OS on the other hand > would be free to treat all x86 processors equally, once it has booted into > fully symmetrical mode. > I certainly could be wrong about my above interpretation, but with these > changes I'm proposing, things work well for the test case of manually > onlining > the BSP after the crash kernel has been started (via kexec -e on a AP > processor > with maxcpus=1 on the crash kernel command line). From looking through the > kernel git history it appears this sequence of events was explicitly > supported > quite a while ago, and we've got a customer who uses this for fast recovery > from > a guest kernel crash. > > Bruce I mean kexec - p ... above, not kexec -e. Sorry about that. Bruce
Re: [PATCH 2/2] KVM: x86: allow BSP to handle INIT IPIs like APs do
On 2016-02-08 18:38, Bruce Rogers wrote: On 2/8/2016 at 10:27 AM, Bruce Rogers wrote: > On 2/8/2016 at 09:40 AM, Paolo Bonziniwrote: >> >>> >>> On 08/02/2016 17:33, Bruce Rogers wrote: KVM_MP_STATE_INIT_RECEIVED is what Intel calls the "wait for SIPI" state. The BSP never gets a SIPI, it goes straight to 0xFFF0 instead. Can you explain the problem more in detail? >> >> I suspect this is about sending INIT-SIPI from another CPU, directed to >> the BSP, isn't it? We may have to differentiate between CPU (including >> system) reset and that IPI case. That is correct. In looking over the KVM code which deals with BSP, this was the only place which seemed wrong to me wrt special casing for BSP outside >>> the context of initial system initialization / reset. As far as I understand the BSP shouldn't be treated differently in this case. >>> >>> See 8.4.2 of the SDM: >>> >>> If the MP protocol has completed and a BSP is chosen, subsequent INITs >>> (either to a specific processor or system wide) do not cause the MP >>> protocol to be repeated. Instead, each logical processor examines its >>> BSP flag (in the IA32_APIC_BASE MSR) to determine whether it should >>> execute the BIOS boot-strap code (if it is the BSP) or enter a >>> wait-for-SIPI state (if it is an AP). >>> >>> So it is correct to treat the BSP differently here, I think. >> >> I had read that, but I though this was speaking from the perspective of the >> SMP aware BIOS code only. In other words, the BIOS would sidetrack AP's >> (based on BSP flag not being present), while BSP would be allowed to go >> through >> the regular BIOS code, checking for reset case, etc. An OS on the other hand >> would be free to treat all x86 processors equally, once it has booted into >> fully symmetrical mode. >> I certainly could be wrong about my above interpretation, but with these >> changes I'm proposing, things work well for the test case of manually >> onlining >> the BSP after the crash kernel has been started (via kexec -e on a AP >> processor >> with maxcpus=1 on the crash kernel command line). From looking through the >> kernel git history it appears this sequence of events was explicitly >> supported >> quite a while ago, and we've got a customer who uses this for fast recovery >> from >> a guest kernel crash. >> >> Bruce > > I mean kexec - p ... above, not kexec -e. Sorry about that. How does real HW behave with your kexec case? Did you try this? Jan signature.asc Description: OpenPGP digital signature