Re: Make QEmu HPET disabled by default for KVM?
On 03/14/2010 12:27 PM, Avi Kivity wrote: On 03/14/2010 12:23 PM, Dor Laor wrote: On 03/14/2010 09:10 AM, Gleb Natapov wrote: On Sun, Mar 14, 2010 at 09:05:50AM +0200, Avi Kivity wrote: On 03/11/2010 09:08 PM, Marcelo Tosatti wrote: I have kept --no-hpet in my setup for months... Any details about the problems? HPET is important to some guests. As Gleb mentioned in the other thread, reinjection will introduce another set of problems. Ideally all this timer related problems should be fixed by correlating timer interrupts and time source reads. This still needs reinjection (or slewing of the timer frequency). Correlation doesn't fix drift. But only when all time sources are synchronised and correlated with interrupts we can slew time frequency without guest noticing (and only if guest disables NTP) In the mean time we should definitely disable hpet by default. Definitely not. Windows needs it. Some pre-kvmclock Linux may also work with it. Without hpet, there is no fast high resolution timer in the system. It's all depends on how hard would it be to re-inject to windows guest. We still need to fix the win2k3 64 bit and win2k8 64 bit (and not win7 as I told initially) since the irq is broadcasted to all the vcpus and we do not track who acknowledged the irq. Besides this we need to fully virtualize the tsc, fix win7 64bit rtc time drift and some pvclock potential issues. Before we add new timer, better fix existing ones. What about creating a pv time keeping device that will be aware of lost ticks and host wall clock time? It's similar to hyper-v enlightenment virt timers. That's kvmclock. I meant a device that can be used to generate timeouts. We do use today pit/rtc along with kvmclock time source but it's not perfect and probably the same for hpet. This is why I tough that a pv device will be beneficial. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Make QEmu HPET disabled by default for KVM?
On 03/14/2010 12:23 PM, Dor Laor wrote: On 03/14/2010 09:10 AM, Gleb Natapov wrote: On Sun, Mar 14, 2010 at 09:05:50AM +0200, Avi Kivity wrote: On 03/11/2010 09:08 PM, Marcelo Tosatti wrote: I have kept --no-hpet in my setup for months... Any details about the problems? HPET is important to some guests. As Gleb mentioned in the other thread, reinjection will introduce another set of problems. Ideally all this timer related problems should be fixed by correlating timer interrupts and time source reads. This still needs reinjection (or slewing of the timer frequency). Correlation doesn't fix drift. But only when all time sources are synchronised and correlated with interrupts we can slew time frequency without guest noticing (and only if guest disables NTP) In the mean time we should definitely disable hpet by default. Definitely not. Windows needs it. Some pre-kvmclock Linux may also work with it. Without hpet, there is no fast high resolution timer in the system. Besides this we need to fully virtualize the tsc, fix win7 64bit rtc time drift and some pvclock potential issues. Before we add new timer, better fix existing ones. What about creating a pv time keeping device that will be aware of lost ticks and host wall clock time? It's similar to hyper-v enlightenment virt timers. That's kvmclock. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Make QEmu HPET disabled by default for KVM?
On 03/14/2010 09:10 AM, Gleb Natapov wrote: On Sun, Mar 14, 2010 at 09:05:50AM +0200, Avi Kivity wrote: On 03/11/2010 09:08 PM, Marcelo Tosatti wrote: I have kept --no-hpet in my setup for months... Any details about the problems? HPET is important to some guests. As Gleb mentioned in the other thread, reinjection will introduce another set of problems. Ideally all this timer related problems should be fixed by correlating timer interrupts and time source reads. This still needs reinjection (or slewing of the timer frequency). Correlation doesn't fix drift. But only when all time sources are synchronised and correlated with interrupts we can slew time frequency without guest noticing (and only if guest disables NTP) In the mean time we should definitely disable hpet by default. Besides this we need to fully virtualize the tsc, fix win7 64bit rtc time drift and some pvclock potential issues. Before we add new timer, better fix existing ones. What about creating a pv time keeping device that will be aware of lost ticks and host wall clock time? It's similar to hyper-v enlightenment virt timers. Since one already has to use special timer parameters (-rtc-td-hack, -no-kvm-pit-reinjection), using -no-hpet for problematic Linux guests seems fine? Depends on how common the problematic ones are. If they're common, better to have a generic fix. -- error compiling committee.c: too many arguments to function -- Gleb. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Make QEmu HPET disabled by default for KVM?
On Sun, Mar 14, 2010 at 09:05:50AM +0200, Avi Kivity wrote: > On 03/11/2010 09:08 PM, Marcelo Tosatti wrote: > > > >> > >>>I have kept --no-hpet in my setup for > >>>months... > >>Any details about the problems? HPET is important to some guests. > >As Gleb mentioned in the other thread, reinjection will introduce > >another set of problems. > > > >Ideally all this timer related problems should be fixed by correlating > >timer interrupts and time source reads. > > This still needs reinjection (or slewing of the timer frequency). > Correlation doesn't fix drift. > But only when all time sources are synchronised and correlated with interrupts we can slew time frequency without guest noticing (and only if guest disables NTP) > >Since one already has to use special timer parameters (-rtc-td-hack, > >-no-kvm-pit-reinjection), using -no-hpet for problematic Linux > >guests seems fine? > > Depends on how common the problematic ones are. If they're common, > better to have a generic fix. > > -- > error compiling committee.c: too many arguments to function -- Gleb. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Make QEmu HPET disabled by default for KVM?
On 03/11/2010 09:08 PM, Marcelo Tosatti wrote: I have kept --no-hpet in my setup for months... Any details about the problems? HPET is important to some guests. As Gleb mentioned in the other thread, reinjection will introduce another set of problems. Ideally all this timer related problems should be fixed by correlating timer interrupts and time source reads. This still needs reinjection (or slewing of the timer frequency). Correlation doesn't fix drift. Since one already has to use special timer parameters (-rtc-td-hack, -no-kvm-pit-reinjection), using -no-hpet for problematic Linux guests seems fine? Depends on how common the problematic ones are. If they're common, better to have a generic fix. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Make QEmu HPET disabled by default for KVM?
On Thu, Mar 11, 2010 at 09:58:12AM +0200, Avi Kivity wrote: > On 03/11/2010 09:52 AM, Sheng Yang wrote: > >I think we have already suffered enough timer issues due to this(e.g. I can't > >boot up well on 2.6.18 kernel)... > > 2.6.18 as guest or as host? > > >I have kept --no-hpet in my setup for > >months... > > Any details about the problems? HPET is important to some guests. As Gleb mentioned in the other thread, reinjection will introduce another set of problems. Ideally all this timer related problems should be fixed by correlating timer interrupts and time source reads. Since one already has to use special timer parameters (-rtc-td-hack, -no-kvm-pit-reinjection), using -no-hpet for problematic Linux guests seems fine? -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Make QEmu HPET disabled by default for KVM?
On 03/11/2010 01:56 PM, Avi Kivity wrote: On 03/11/2010 12:23 PM, Gleb Natapov wrote: If the problem it due to lost ticks reinjection may solve it, but only partially. What if IO thread haven't run even once during the time vcpu did clock source check? IIRC sometimes we trigger this even with in kernel PIT. That is true. Reinjection can correct problems in the long term, but may fail in the short term. 10 ticks is easily short term in a heavily loaded system. How does it happen with kernel PIT? I could understand it if we had a work item doing the injection, but everything happens either from hrtimer context or vcpu context. Do we kick vcpu out of guest mode when hrtimer triggers? I don't see us doing it in __kvm_timer_fn(). We're always running on the same cpu as vcpu 0, so no need. Would be better to do it, though, in case we have migration races. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Make QEmu HPET disabled by default for KVM?
On 03/11/2010 12:23 PM, Gleb Natapov wrote: If the problem it due to lost ticks reinjection may solve it, but only partially. What if IO thread haven't run even once during the time vcpu did clock source check? IIRC sometimes we trigger this even with in kernel PIT. That is true. Reinjection can correct problems in the long term, but may fail in the short term. 10 ticks is easily short term in a heavily loaded system. How does it happen with kernel PIT? I could understand it if we had a work item doing the injection, but everything happens either from hrtimer context or vcpu context. Do we kick vcpu out of guest mode when hrtimer triggers? I don't see us doing it in __kvm_timer_fn(). We're always running on the same cpu as vcpu 0, so no need. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Make QEmu HPET disabled by default for KVM?
On Thu, Mar 11, 2010 at 10:46:06AM +0200, Avi Kivity wrote: > On 03/11/2010 10:42 AM, Gleb Natapov wrote: > >On Thu, Mar 11, 2010 at 04:38:48PM +0800, Sheng Yang wrote: > >>On Thursday 11 March 2010 16:31:57 Gleb Natapov wrote: > >>>On Thu, Mar 11, 2010 at 10:28:12AM +0200, Avi Kivity wrote: > On 03/11/2010 10:23 AM, Sheng Yang wrote: > >>>I have kept --no-hpet in my setup for > >>>months... > >>Any details about the problems? HPET is important to some guests. > >Seems like HPET reaction is too slow to satisfy some guests(for it would > >replace PIT). > > > >Here is the thread last time. > > > >http://thread.gmane.org/gmane.comp.emulators.kvm.devel/44899 > Thanks. We can address this in three ways: first, adjust the guest > not to do timing related tests when virtualized (since no matter > what we do, the tests may fail). Second, I think we should > implement userspace ack notifiers (similar to tpr access notifiers > already present). Third, we can implement a kernel hpet, which, > after we solve the zillion bug it introduces, will also give a nice > performance improvement for hpet intensive workloads. > >>>Second will not solve the problem. Presence of ack notifiers will not > >>>make HPET interrupt arrive faster. > >>The slow may also due to lost tick. And with the lost tick, hpet is still > >>unusable... > >> > >If the problem it due to lost ticks reinjection may solve it, but only > >partially. > >What if IO thread haven't run even once during the time vcpu did clock > >source check? IIRC sometimes we trigger this even with in kernel PIT. > > That is true. Reinjection can correct problems in the long term, > but may fail in the short term. 10 ticks is easily short term in a > heavily loaded system. > > How does it happen with kernel PIT? I could understand it if we had > a work item doing the injection, but everything happens either from > hrtimer context or vcpu context. > Do we kick vcpu out of guest mode when hrtimer triggers? I don't see us doing it in __kvm_timer_fn(). -- Gleb. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Make QEmu HPET disabled by default for KVM?
On 03/11/2010 10:42 AM, Gleb Natapov wrote: On Thu, Mar 11, 2010 at 04:38:48PM +0800, Sheng Yang wrote: On Thursday 11 March 2010 16:31:57 Gleb Natapov wrote: On Thu, Mar 11, 2010 at 10:28:12AM +0200, Avi Kivity wrote: On 03/11/2010 10:23 AM, Sheng Yang wrote: I have kept --no-hpet in my setup for months... Any details about the problems? HPET is important to some guests. Seems like HPET reaction is too slow to satisfy some guests(for it would replace PIT). Here is the thread last time. http://thread.gmane.org/gmane.comp.emulators.kvm.devel/44899 Thanks. We can address this in three ways: first, adjust the guest not to do timing related tests when virtualized (since no matter what we do, the tests may fail). Second, I think we should implement userspace ack notifiers (similar to tpr access notifiers already present). Third, we can implement a kernel hpet, which, after we solve the zillion bug it introduces, will also give a nice performance improvement for hpet intensive workloads. Second will not solve the problem. Presence of ack notifiers will not make HPET interrupt arrive faster. The slow may also due to lost tick. And with the lost tick, hpet is still unusable... If the problem it due to lost ticks reinjection may solve it, but only partially. What if IO thread haven't run even once during the time vcpu did clock source check? IIRC sometimes we trigger this even with in kernel PIT. That is true. Reinjection can correct problems in the long term, but may fail in the short term. 10 ticks is easily short term in a heavily loaded system. How does it happen with kernel PIT? I could understand it if we had a work item doing the injection, but everything happens either from hrtimer context or vcpu context. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Make QEmu HPET disabled by default for KVM?
On Thu, Mar 11, 2010 at 04:38:48PM +0800, Sheng Yang wrote: > On Thursday 11 March 2010 16:31:57 Gleb Natapov wrote: > > On Thu, Mar 11, 2010 at 10:28:12AM +0200, Avi Kivity wrote: > > > On 03/11/2010 10:23 AM, Sheng Yang wrote: > > > >>>I have kept --no-hpet in my setup for > > > >>>months... > > > >> > > > >>Any details about the problems? HPET is important to some guests. > > > > > > > >Seems like HPET reaction is too slow to satisfy some guests(for it would > > > >replace PIT). > > > > > > > >Here is the thread last time. > > > > > > > >http://thread.gmane.org/gmane.comp.emulators.kvm.devel/44899 > > > > > > Thanks. We can address this in three ways: first, adjust the guest > > > not to do timing related tests when virtualized (since no matter > > > what we do, the tests may fail). Second, I think we should > > > implement userspace ack notifiers (similar to tpr access notifiers > > > already present). Third, we can implement a kernel hpet, which, > > > after we solve the zillion bug it introduces, will also give a nice > > > performance improvement for hpet intensive workloads. > > > > Second will not solve the problem. Presence of ack notifiers will not > > make HPET interrupt arrive faster. > > The slow may also due to lost tick. And with the lost tick, hpet is still > unusable... > If the problem it due to lost ticks reinjection may solve it, but only partially. What if IO thread haven't run even once during the time vcpu did clock source check? IIRC sometimes we trigger this even with in kernel PIT. -- Gleb. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Make QEmu HPET disabled by default for KVM?
On Thursday 11 March 2010 16:31:57 Gleb Natapov wrote: > On Thu, Mar 11, 2010 at 10:28:12AM +0200, Avi Kivity wrote: > > On 03/11/2010 10:23 AM, Sheng Yang wrote: > > >>>I have kept --no-hpet in my setup for > > >>>months... > > >> > > >>Any details about the problems? HPET is important to some guests. > > > > > >Seems like HPET reaction is too slow to satisfy some guests(for it would > > >replace PIT). > > > > > >Here is the thread last time. > > > > > >http://thread.gmane.org/gmane.comp.emulators.kvm.devel/44899 > > > > Thanks. We can address this in three ways: first, adjust the guest > > not to do timing related tests when virtualized (since no matter > > what we do, the tests may fail). Second, I think we should > > implement userspace ack notifiers (similar to tpr access notifiers > > already present). Third, we can implement a kernel hpet, which, > > after we solve the zillion bug it introduces, will also give a nice > > performance improvement for hpet intensive workloads. > > Second will not solve the problem. Presence of ack notifiers will not > make HPET interrupt arrive faster. The slow may also due to lost tick. And with the lost tick, hpet is still unusable... -- regards Yang, Sheng -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Make QEmu HPET disabled by default for KVM?
On 03/11/2010 10:31 AM, Gleb Natapov wrote: On Thu, Mar 11, 2010 at 10:28:12AM +0200, Avi Kivity wrote: On 03/11/2010 10:23 AM, Sheng Yang wrote: I have kept --no-hpet in my setup for months... Any details about the problems? HPET is important to some guests. Seems like HPET reaction is too slow to satisfy some guests(for it would replace PIT). Here is the thread last time. http://thread.gmane.org/gmane.comp.emulators.kvm.devel/44899 Thanks. We can address this in three ways: first, adjust the guest not to do timing related tests when virtualized (since no matter what we do, the tests may fail). Second, I think we should implement userspace ack notifiers (similar to tpr access notifiers already present). Third, we can implement a kernel hpet, which, after we solve the zillion bug it introduces, will also give a nice performance improvement for hpet intensive workloads. Second will not solve the problem. Presence of ack notifiers will not make HPET interrupt arrive faster. But it will allow us to compensate for interrupts being coalesced, which may be the root of the problem. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Make QEmu HPET disabled by default for KVM?
On Thu, Mar 11, 2010 at 10:28:12AM +0200, Avi Kivity wrote: > On 03/11/2010 10:23 AM, Sheng Yang wrote: > >>>I have kept --no-hpet in my setup for > >>>months... > >>Any details about the problems? HPET is important to some guests. > >> > >Seems like HPET reaction is too slow to satisfy some guests(for it would > >replace PIT). > > > >Here is the thread last time. > > > >http://thread.gmane.org/gmane.comp.emulators.kvm.devel/44899 > > > > Thanks. We can address this in three ways: first, adjust the guest > not to do timing related tests when virtualized (since no matter > what we do, the tests may fail). Second, I think we should > implement userspace ack notifiers (similar to tpr access notifiers > already present). Third, we can implement a kernel hpet, which, > after we solve the zillion bug it introduces, will also give a nice > performance improvement for hpet intensive workloads. > Second will not solve the problem. Presence of ack notifiers will not make HPET interrupt arrive faster. -- Gleb. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Make QEmu HPET disabled by default for KVM?
On 03/11/2010 10:23 AM, Sheng Yang wrote: I have kept --no-hpet in my setup for months... Any details about the problems? HPET is important to some guests. Seems like HPET reaction is too slow to satisfy some guests(for it would replace PIT). Here is the thread last time. http://thread.gmane.org/gmane.comp.emulators.kvm.devel/44899 Thanks. We can address this in three ways: first, adjust the guest not to do timing related tests when virtualized (since no matter what we do, the tests may fail). Second, I think we should implement userspace ack notifiers (similar to tpr access notifiers already present). Third, we can implement a kernel hpet, which, after we solve the zillion bug it introduces, will also give a nice performance improvement for hpet intensive workloads. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Make QEmu HPET disabled by default for KVM?
On Thursday 11 March 2010 15:58:12 Avi Kivity wrote: > On 03/11/2010 09:52 AM, Sheng Yang wrote: > > I think we have already suffered enough timer issues due to this(e.g. I > > can't boot up well on 2.6.18 kernel)... > > 2.6.18 as guest or as host? Guest > > I have kept --no-hpet in my setup for > > months... > > Any details about the problems? HPET is important to some guests. > Seems like HPET reaction is too slow to satisfy some guests(for it would replace PIT). Here is the thread last time. http://thread.gmane.org/gmane.comp.emulators.kvm.devel/44899 -- regards Yang, Sheng -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Make QEmu HPET disabled by default for KVM?
On 03/11/2010 09:52 AM, Sheng Yang wrote: I think we have already suffered enough timer issues due to this(e.g. I can't boot up well on 2.6.18 kernel)... 2.6.18 as guest or as host? I have kept --no-hpet in my setup for months... Any details about the problems? HPET is important to some guests. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Make QEmu HPET disabled by default for KVM?
I think we have already suffered enough timer issues due to this(e.g. I can't boot up well on 2.6.18 kernel)... I have kept --no-hpet in my setup for months... -- regards Yang, Sheng -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
