Re: A question about HTL VM-Exit handling time
On 10/28/2014 12:52 AM, Wanpeng Li wrote: > As you know, kernel build which Feng used is a CPU bound workload, > so why 22.28% samples is halt? Because in a CPU bound workload you don't have many exits in the first place. But there could also be some issues with the way he used perf. 213821 interrupts seem to be really few. Paolo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: A question about HTL VM-Exit handling time
Am 21.10.2014 20:39, schrieb Paolo Bonzini: > > > On 10/16/2014 10:15 AM, Wu, Feng wrote: >> Hi folks, >> >> I run kernel build in the guest and use perf kvm to get some VM-Exit result >> as the following: >> >> Analyze events for all VCPUs: >> >> VM-EXITSamples Samples% Time% Min Time Max Time >> A >> >>MSR_WRITE361390857.53%18.97%5us 1362us >> 9.73 >> HLT139974722.28%74.90%5us 432448us >> 99.24 >>CR_ACCESS 96120315.30% 3.28%4us 188us >> 6.33 >> EXTERNAL_INTERRUPT 213821 3.40% 2.25%4us 4089us >> 19.54 >>EXCEPTION_NMI 25152 0.40% 0.12%4us 71us >> 9.05 >>EPT_MISCONFIG 20104 0.32% 0.15%8us 5628us >> 13.74 >>CPUID 19904 0.32% 0.07%4us 220us >> 6.90 >> IO_INSTRUCTION 17097 0.27% 0.20% 13us 1008us >> 22.08 >>PAUSE_INSTRUCTION 10737 0.17% 0.05%4us 53us >> 8.33 >> MSR_READ 48 0.00% 0.00%4us8us >> 5.62 >> >> Total Samples:6281721, Total events handled time:185457820.41us. >> >> I also do some other experiments with different workload in the guest, I got >> the same results in terms of >> HLT VM-Exit handling time. Does anyone know why the handling time for HLT >> VM-Exit is so high? Appreciate >> You help! > > 432 ms sounds like a lot, but in general it is expected that HLT vmexits > take a long time. After an HLT vmexit, the VCPU will not be reentered > until the next interrupt comes. On hardware, the HLT instruction can > also take many milliseconds. > > If this is an SMP guest, it's possible that the maximum time is > registered on the APs before Linux boots. With a UP guest I would > expect a shorter maximum time, but still longer than other vmexits. We have the same on s390 with wait state. The thing is, with an idle system and NOHZ the time spend in HLT/wait could be really long. So We might want to provide an option to filter this out. (A similar filter already exitis for the --duration option) -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: A question about HTL VM-Exit handling time
Hi Paolo, On Tue, Oct 21, 2014 at 08:39:49PM +0200, Paolo Bonzini wrote: > > >On 10/16/2014 10:15 AM, Wu, Feng wrote: >> Hi folks, >> >> I run kernel build in the guest and use perf kvm to get some VM-Exit result >> as the following: >> >> Analyze events for all VCPUs: >> >> VM-EXITSamples Samples% Time% Min Time Max Time >> A >> >>MSR_WRITE361390857.53%18.97%5us 1362us >> 9.73 >> HLT139974722.28%74.90%5us 432448us >> 99.24 >>CR_ACCESS 96120315.30% 3.28%4us 188us >> 6.33 >> EXTERNAL_INTERRUPT 213821 3.40% 2.25%4us 4089us >> 19.54 >>EXCEPTION_NMI 25152 0.40% 0.12%4us 71us >> 9.05 >>EPT_MISCONFIG 20104 0.32% 0.15%8us 5628us >> 13.74 >>CPUID 19904 0.32% 0.07%4us 220us >> 6.90 >> IO_INSTRUCTION 17097 0.27% 0.20% 13us 1008us >> 22.08 >>PAUSE_INSTRUCTION 10737 0.17% 0.05%4us 53us >> 8.33 >> MSR_READ 48 0.00% 0.00%4us8us >> 5.62 >> >> Total Samples:6281721, Total events handled time:185457820.41us. >> >> I also do some other experiments with different workload in the guest, I got >> the same results in terms of >> HLT VM-Exit handling time. Does anyone know why the handling time for HLT >> VM-Exit is so high? Appreciate >> You help! > >432 ms sounds like a lot, but in general it is expected that HLT vmexits >take a long time. After an HLT vmexit, the VCPU will not be reentered >until the next interrupt comes. On hardware, the HLT instruction can >also take many milliseconds. > As you know, kernel build which Feng used is a CPU bound workload, so why 22.28% samples is halt? Regards, Wanpeng Li >If this is an SMP guest, it's possible that the maximum time is >registered on the APs before Linux boots. With a UP guest I would >expect a shorter maximum time, but still longer than other vmexits. > >Paolo >-- >To unsubscribe from this list: send the line "unsubscribe kvm" in >the body of a message to majord...@vger.kernel.org >More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: A question about HTL VM-Exit handling time
> -Original Message- > From: Paolo Bonzini [mailto:pbonz...@redhat.com] > Sent: Thursday, October 23, 2014 8:23 PM > To: Wu, Feng; kvm@vger.kernel.org > Cc: Xiao Guangrong > Subject: Re: A question about HTL VM-Exit handling time > > > > Thanks for your answer, Paolo. > > There is a general question for perf kvm. Do you know how the vmexit > > handling time is measured? Such as what is the start point, and the > > end point? If I got this information, I think I can better understand the > > value of the handling time. Thanks a lot! > > It is measured from kvm:kvm_exit to kvm:kvm_entry. See > exit_event_begin, exit_event_get_key, exit_event_end in > tools/perf/builtin-kvm.c. Got it, Thanks for the information! Thanks, Feng > > Paolo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: A question about HTL VM-Exit handling time
> Thanks for your answer, Paolo. > There is a general question for perf kvm. Do you know how the vmexit > handling time is measured? Such as what is the start point, and the > end point? If I got this information, I think I can better understand the > value of the handling time. Thanks a lot! It is measured from kvm:kvm_exit to kvm:kvm_entry. See exit_event_begin, exit_event_get_key, exit_event_end in tools/perf/builtin-kvm.c. Paolo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: A question about HTL VM-Exit handling time
> -Original Message- > From: Paolo Bonzini [mailto:paolo.bonz...@gmail.com] On Behalf Of Paolo > Bonzini > Sent: Wednesday, October 22, 2014 2:40 AM > To: Wu, Feng; kvm@vger.kernel.org > Cc: Xiao Guangrong > Subject: Re: A question about HTL VM-Exit handling time > > > > On 10/16/2014 10:15 AM, Wu, Feng wrote: > > Hi folks, > > > > I run kernel build in the guest and use perf kvm to get some VM-Exit result > > as > the following: > > > > Analyze events for all VCPUs: > > > > VM-EXITSamples Samples% Time% Min Time > Max Time A > > > >MSR_WRITE361390857.53%18.97% > 5us 1362us 9.73 > > HLT139974722.28%74.90%5us > 432448us 99.24 > >CR_ACCESS 96120315.30% 3.28% > 4us 188us 6.33 > > EXTERNAL_INTERRUPT 213821 3.40% 2.25% > 4us 4089us 19.54 > >EXCEPTION_NMI 25152 0.40% 0.12% > 4us 71us 9.05 > >EPT_MISCONFIG 20104 0.32% 0.15% > 8us 5628us 13.74 > >CPUID 19904 0.32% 0.07%4us > 220us 6.90 > > IO_INSTRUCTION 17097 0.27% 0.20% 13us > 1008us 22.08 > >PAUSE_INSTRUCTION 10737 0.17% 0.05% > 4us 53us 8.33 > > MSR_READ 48 0.00% 0.00% > 4us8us 5.62 > > > > Total Samples:6281721, Total events handled time:185457820.41us. > > > > I also do some other experiments with different workload in the guest, I got > the same results in terms of > > HLT VM-Exit handling time. Does anyone know why the handling time for HLT > VM-Exit is so high? Appreciate > > You help! > > 432 ms sounds like a lot, but in general it is expected that HLT vmexits > take a long time. After an HLT vmexit, the VCPU will not be reentered > until the next interrupt comes. On hardware, the HLT instruction can > also take many milliseconds. Thanks for your answer, Paolo. There is a general question for perf kvm. Do you know how the vmexit handling time is measured? Such as what is the start point, and the end point? If I got this information, I think I can better understand the value of the handling time. Thanks a lot! > > If this is an SMP guest, it's possible that the maximum time is > registered on the APs before Linux boots. With a UP guest I would > expect a shorter maximum time, but still longer than other vmexits. Yes, you are right, I get the data for an UP guest and the HLT vmexit handling time is shorter. Here is the result: Analyze events for all VCPUs: VM-EXITSamples Samples% Time% Min Time Max Time EXTERNAL_INTERRUPT 9001529.68%41.92%4us 660us 24.9 MSR_WRITE 8951229.52%14.25%4us 162us 8.5 IO_INSTRUCTION 4782315.77%16.67% 11us 1907us 18.6 EPT_MISCONFIG 3850412.70% 7.42%7us 4848us 10.3 CPUID 3265810.77% 3.61%3us 304us 5.9 EXCEPTION_NMI 2232 0.74% 0.34%4us 164us 8.2 PAUSE_INSTRUCTION 1594 0.53% 0.21%3us 44us 6.9 HLT547 0.18%15.50%8us 177098us 1516.5 CR_ACCESS278 0.09% 0.05%5us 26us 8.7 EPT_VIOLATION 71 0.02% 0.03%8us 27us 20.3 MSR_READ 12 0.00% 0.00%3us4us 4.1 Total Samples:303246, Total events handled time:5350983.30us. Thanks, Feng > > Paolo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: A question about HTL VM-Exit handling time
On 10/16/2014 10:15 AM, Wu, Feng wrote: > Hi folks, > > I run kernel build in the guest and use perf kvm to get some VM-Exit result > as the following: > > Analyze events for all VCPUs: > > VM-EXITSamples Samples% Time% Min Time Max Time > A > >MSR_WRITE361390857.53%18.97%5us 1362us > 9.73 > HLT139974722.28%74.90%5us 432448us > 99.24 >CR_ACCESS 96120315.30% 3.28%4us 188us > 6.33 > EXTERNAL_INTERRUPT 213821 3.40% 2.25%4us 4089us > 19.54 >EXCEPTION_NMI 25152 0.40% 0.12%4us 71us > 9.05 >EPT_MISCONFIG 20104 0.32% 0.15%8us 5628us > 13.74 >CPUID 19904 0.32% 0.07%4us 220us > 6.90 > IO_INSTRUCTION 17097 0.27% 0.20% 13us 1008us > 22.08 >PAUSE_INSTRUCTION 10737 0.17% 0.05%4us 53us > 8.33 > MSR_READ 48 0.00% 0.00%4us8us > 5.62 > > Total Samples:6281721, Total events handled time:185457820.41us. > > I also do some other experiments with different workload in the guest, I got > the same results in terms of > HLT VM-Exit handling time. Does anyone know why the handling time for HLT > VM-Exit is so high? Appreciate > You help! 432 ms sounds like a lot, but in general it is expected that HLT vmexits take a long time. After an HLT vmexit, the VCPU will not be reentered until the next interrupt comes. On hardware, the HLT instruction can also take many milliseconds. If this is an SMP guest, it's possible that the maximum time is registered on the APs before Linux boots. With a UP guest I would expect a shorter maximum time, but still longer than other vmexits. Paolo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
