Re: [PATCH v2 0/4] KVM: APIC improvements (with bonus mixed mode)
2015-02-26 19:31-0300, Marcelo Tosatti: On Thu, Feb 26, 2015 at 10:49:53AM +0200, Nadav Amit wrote: Marcello, Radim, As you know - I can run some tests on the patches and whether they comply with real hardware. Please let me know which version to test and I’ll try to do next week. Please do, this series implements a subset of what I found about real hardware behavior, the rest didn't make much sense. (I'd be grateful for any report about real hardware.) From what i understand Radim did run a set of tests (with a modified Linux kernel) to find out the information on mixed mode. However its not clear what that behaviour is. Exactly. Thank you. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 0/4] KVM: APIC improvements (with bonus mixed mode)
2015-02-25 22:55-0300, Marcelo Tosatti: 1) when we have no x2APIC ID 0xff00, but send x2APIC message there 10.7 SYSTEM AND APIC BUS ARBITRATION Note that except for the SIPI IPI (see Section 10.6.1, “Interrupt Command Register (ICR)”), all bus messages that fail to be delivered to their specified destination or destinations are automatically retried. Software should avoid situations in which IPIs are sent to disabled or nonexistent local APICs, causing the messages to be resent repeatedly. Perfect, thanks. (I thought that the restriction only applies to lowest priority.) or after something that isn't forbidden in SDM most likely because they didn't think that anyone would ever consider it 2) we have x2APIC ID 0xff00 and reset other x2APICs into xAPIC mode Or just x2APIC initialization in non lockstep: vcpu0 vcpu1 T0) xapic mode xapic mode T1) x2apic enable T2) broadcast IPI Good point, there is no reason why the BIOS couldn't do that. (If we have APIC ID 255, OS gets CPUs in x2APIC mode, or with high IDs disabled; 10.12.8.1 Consistency of APIC IDs and CPUID) No volunteer came to clear this up, so I hacked Linux to have one x2APIC core between xAPIC ones. Physical IPI to 0xff00 got delivered only to CPU0, like most other destinations, Logical IPI to 0xff00 got dropped and only 0x worked as a broadcast in both modes. I think it is because Intel never considered mixed mode to be valid, and seen delivery might be an emergent feature of QPI routing. Luckily, broadcast from xAPIC isn't delivered to x2APIC. In real hardware? Yes, it was ivy bridge, iirc. Real exploration would require greater modifications to Linux (ideally writing a custom kernel), so this series implements something that makes some sense and isn't too far from reality. Ok, so the problem is that broadcast (ICR.destination == 0xff00) from xAPIC CPU is not delivered to x2APIC CPUs ? Mixed mode doesn't work much without this series -- we internally shift all destinations to 0x0-0xff range and use a map that can't handle duplicate entries, so the broadcast IPI is just a detail. The biggest problem that for me is that we allow x2APIC without interrupt remapping, so all devices use (IO)xAPIC and broadcast is inconsistent with the design -- interpreted as a 0xff x2APIC message. I can't find any restriction against (cpu0==x2APIC, cpu1==xAPIC) in the documentation. Me neither ... the most I found is that boot restriction mentioned earlier. (10.12.8.1) Anyway, emulation should match physical hardware. (I think that we have created a de facto standard by rejecting SDM, 10.12.7 Initialization by System Software, and no OS that enabled x2APIC under KVM should do things that do not work. The main benefit of this series is making the code better.) From your message above, it is not clear what is the behaviour there: No volunteer came to clear this up, so I hacked Linux to have one x2APIC core between xAPIC ones. Physical IPI to 0xff00 got delivered only to CPU0, like most other destinations, Logical IPI to 0xff00 got dropped and only 0x worked as a broadcast in both modes. Luckily, broadcast from xAPIC isn't delivered to x2APIC. From the x2APIC CPU or the xAPIC ones? The first paragraph was about x2APIC IPIs and their results on xAPICs. The second paragraph was a broadcast from xAPIC. (I actually did the reverse for that one -- one xAPIC besides x2APIC CPUs.) It should be easy to write kvm-unit-test testcases that match physical hardware behaviour (in general, i am having a hard time figure out in what way mixed mode is supposed to behave, please describe it clearly). I think that mixed mode is undefined in real hardware: delivering x2APIC 0xff112233 to xAPIC 0 makes not sense. I already wrote all that I know about it, sorry. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 0/4] KVM: APIC improvements (with bonus mixed mode)
Marcello, Radim, As you know - I can run some tests on the patches and whether they comply with real hardware. Please let me know which version to test and I’ll try to do next week. Regards, Nadav Marcelo Tosatti mtosa...@redhat.com wrote: Radim, On Thu, Feb 12, 2015 at 07:41:30PM +0100, Radim Krčmář wrote: Each patch has a diff from v1, here is only a prologue on the mythical mixed xAPIC and x2APIC mode: There is one interesting alias in xAPIC and x2APIC ICR destination, the 0xff00, which is a broadcast in xAPIC and either a physical message to high x2APIC ID or a message to an empty set in x2APIC cluster. This corner case in mixed mode can be triggered by a weird message 1) when we have no x2APIC ID 0xff00, but send x2APIC message there 10.7 SYSTEM AND APIC BUS ARBITRATION Note that except for the SIPI IPI (see Section 10.6.1, “Interrupt Command Register (ICR)”), all bus messages that fail to be delivered to their specified destination or destinations are automatically retried. Software should avoid situations in which IPIs are sent to disabled or nonexistent local APICs, causing the messages to be resent repeatedly. or after something that isn't forbidden in SDM most likely because they didn't think that anyone would ever consider it 2) we have x2APIC ID 0xff00 and reset other x2APICs into xAPIC mode Or just x2APIC initialization in non lockstep: vcpu0 vcpu1 T0) xapic mode xapic mode T1) x2apic enable T2) broadcast IPI Current KVM doesn't need to consider (2), so there only is a slim chance that some hobbyist OS pushes the specification to its limits. The problem is that SDM for xAPIC doesn't believably specify[1] if all messages beginning with 0xff are considered as broadcasts, 10.6.2.1: A broadcast IPI (bits 28-31 of the MDA are 1's) No volunteer came to clear this up, so I hacked Linux to have one x2APIC core between xAPIC ones. Physical IPI to 0xff00 got delivered only to CPU0, like most other destinations, Logical IPI to 0xff00 got dropped and only 0x worked as a broadcast in both modes. I think it is because Intel never considered mixed mode to be valid, and seen delivery might be an emergent feature of QPI routing. Luckily, broadcast from xAPIC isn't delivered to x2APIC. In real hardware? Real exploration would require greater modifications to Linux (ideally writing a custom kernel), so this series implements something that makes some sense and isn't too far from reality. Ok, so the problem is that broadcast (ICR.destination == 0xff00) from xAPIC CPU is not delivered to x2APIC CPUs ? Radim Krčmář (4): KVM: x86: use MDA for interrupt matching KVM: x86: fix mixed APIC mode broadcast KVM: x86: avoid logical_map when it is invalid KVM: x86: simplify kvm_apic_map I can't find any restriction against (cpu0==x2APIC, cpu1==xAPIC) in the documentation. Anyway, emulation should match physical hardware. From your message above, it is not clear what is the behaviour there: No volunteer came to clear this up, so I hacked Linux to have one x2APIC core between xAPIC ones. Physical IPI to 0xff00 got delivered only to CPU0, like most other destinations, Logical IPI to 0xff00 got dropped and only 0x worked as a broadcast in both modes. Luckily, broadcast from xAPIC isn't delivered to x2APIC. From the x2APIC CPU or the xAPIC ones? It should be easy to write kvm-unit-test testcases that match physical hardware behaviour (in general, i am having a hard time figure out in what way mixed mode is supposed to behave, please describe it clearly). -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 0/4] KVM: APIC improvements (with bonus mixed mode)
On Thu, Feb 26, 2015 at 10:49:53AM +0200, Nadav Amit wrote: Marcello, Radim, As you know - I can run some tests on the patches and whether they comply with real hardware. Please let me know which version to test and I’ll try to do next week. Regards, Nadav From what i understand Radim did run a set of tests (with a modified Linux kernel) to find out the information on mixed mode. However its not clear what that behaviour is. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 0/4] KVM: APIC improvements (with bonus mixed mode)
Radim, On Thu, Feb 12, 2015 at 07:41:30PM +0100, Radim Krčmář wrote: Each patch has a diff from v1, here is only a prologue on the mythical mixed xAPIC and x2APIC mode: There is one interesting alias in xAPIC and x2APIC ICR destination, the 0xff00, which is a broadcast in xAPIC and either a physical message to high x2APIC ID or a message to an empty set in x2APIC cluster. This corner case in mixed mode can be triggered by a weird message 1) when we have no x2APIC ID 0xff00, but send x2APIC message there 10.7 SYSTEM AND APIC BUS ARBITRATION Note that except for the SIPI IPI (see Section 10.6.1, “Interrupt Command Register (ICR)”), all bus messages that fail to be delivered to their specified destination or destinations are automatically retried. Software should avoid situations in which IPIs are sent to disabled or nonexistent local APICs, causing the messages to be resent repeatedly. or after something that isn't forbidden in SDM most likely because they didn't think that anyone would ever consider it 2) we have x2APIC ID 0xff00 and reset other x2APICs into xAPIC mode Or just x2APIC initialization in non lockstep: vcpu0 vcpu1 T0) xapic mode xapic mode T1) x2apic enable T2) broadcast IPI Current KVM doesn't need to consider (2), so there only is a slim chance that some hobbyist OS pushes the specification to its limits. The problem is that SDM for xAPIC doesn't believably specify[1] if all messages beginning with 0xff are considered as broadcasts, 10.6.2.1: A broadcast IPI (bits 28-31 of the MDA are 1's) No volunteer came to clear this up, so I hacked Linux to have one x2APIC core between xAPIC ones. Physical IPI to 0xff00 got delivered only to CPU0, like most other destinations, Logical IPI to 0xff00 got dropped and only 0x worked as a broadcast in both modes. I think it is because Intel never considered mixed mode to be valid, and seen delivery might be an emergent feature of QPI routing. Luckily, broadcast from xAPIC isn't delivered to x2APIC. In real hardware? Real exploration would require greater modifications to Linux (ideally writing a custom kernel), so this series implements something that makes some sense and isn't too far from reality. Ok, so the problem is that broadcast (ICR.destination == 0xff00) from xAPIC CPU is not delivered to x2APIC CPUs ? Radim Krčmář (4): KVM: x86: use MDA for interrupt matching KVM: x86: fix mixed APIC mode broadcast KVM: x86: avoid logical_map when it is invalid KVM: x86: simplify kvm_apic_map I can't find any restriction against (cpu0==x2APIC, cpu1==xAPIC) in the documentation. Anyway, emulation should match physical hardware. From your message above, it is not clear what is the behaviour there: No volunteer came to clear this up, so I hacked Linux to have one x2APIC core between xAPIC ones. Physical IPI to 0xff00 got delivered only to CPU0, like most other destinations, Logical IPI to 0xff00 got dropped and only 0x worked as a broadcast in both modes. Luckily, broadcast from xAPIC isn't delivered to x2APIC. From the x2APIC CPU or the xAPIC ones? It should be easy to write kvm-unit-test testcases that match physical hardware behaviour (in general, i am having a hard time figure out in what way mixed mode is supposed to behave, please describe it clearly). -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v2 0/4] KVM: APIC improvements (with bonus mixed mode)
Each patch has a diff from v1, here is only a prologue on the mythical mixed xAPIC and x2APIC mode: There is one interesting alias in xAPIC and x2APIC ICR destination, the 0xff00, which is a broadcast in xAPIC and either a physical message to high x2APIC ID or a message to an empty set in x2APIC cluster. This corner case in mixed mode can be triggered by a weird message 1) when we have no x2APIC ID 0xff00, but send x2APIC message there or after something that isn't forbidden in SDM most likely because they didn't think that anyone would ever consider it 2) we have x2APIC ID 0xff00 and reset other x2APICs into xAPIC mode Current KVM doesn't need to consider (2), so there only is a slim chance that some hobbyist OS pushes the specification to its limits. The problem is that SDM for xAPIC doesn't believably specify[1] if all messages beginning with 0xff are considered as broadcasts, 10.6.2.1: A broadcast IPI (bits 28-31 of the MDA are 1's) No volunteer came to clear this up, so I hacked Linux to have one x2APIC core between xAPIC ones. Physical IPI to 0xff00 got delivered only to CPU0, like most other destinations, Logical IPI to 0xff00 got dropped and only 0x worked as a broadcast in both modes. I think it is because Intel never considered mixed mode to be valid, and seen delivery might be an emergent feature of QPI routing. Luckily, broadcast from xAPIC isn't delivered to x2APIC. Real exploration would require greater modifications to Linux (ideally writing a custom kernel), so this series implements something that makes some sense and isn't too far from reality. Radim Krčmář (4): KVM: x86: use MDA for interrupt matching KVM: x86: fix mixed APIC mode broadcast KVM: x86: avoid logical_map when it is invalid KVM: x86: simplify kvm_apic_map arch/x86/include/asm/kvm_host.h | 8 ++- arch/x86/kvm/lapic.c| 138 +++- arch/x86/kvm/lapic.h| 15 - 3 files changed, 85 insertions(+), 76 deletions(-) -- 2.3.0 -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
