Re: [CFT] Paravirtualized KVM clock
... so, out of pure curiousity - what's making the benchmark go faster? Is it userland side of things calling clock methods, or something in the kernel, or both? -adrian On 4 January 2015 at 09:56, Bryan Venteicher wrote: > For the last few weeks, I've been working on adding support for KVM clock > in the projects/paravirt branch. Currently, a KVM VM guest will end up > selecting either the HPET or ACPI as the timecounter source. Unfortunately, > this is very costly since every timecounter fetch causes a VM exit. KVM > clock allows the guest to use the TSC instead; it is very similar to the > existing Xen timer. > > The performance difference between HPET/ACPI and KVMCLOCK can be dramatic: > a simple disk benchmark goes from 10K IOPs to 100K IOPs. > > The patch is attached is attached or available at [1]. I'd appreciate any > testing. > > Also as a part of this, I've tried to generalized a bit of our existing > hypervisor guest code, with the eventual goal of being able to support more > invasive PV operations. The patch series is viewable in Phabricator. > > https://reviews.freebsd.org/D1429 - paravirt: Generalize parts of the XEN > timer code into pvclock > https://reviews.freebsd.org/D1430 - paravirt: Add interface to calculate > the TSC frequency from pvclock > https://reviews.freebsd.org/D1431 - paravirt: Add simple hypervisor > registration and detection interface > https://reviews.freebsd.org/D1432 - paravirt: Add detection of bhyve using > new hypervisor interface > https://reviews.freebsd.org/D1433 - paravirt: Add detection of VMware using > new hypervisor interface > https://reviews.freebsd.org/D1434 - paravirt: Add detection of KVM using > new hypervisor interface > https://reviews.freebsd.org/D1435 - paravirt: Add KVM clock timecounter > support > > My current plan is to MFC this series to 10-STABLE, and commit a > self-contained KVM clock to the other stable branches. > > [1] - https://people.freebsd.org/~bryanv/patches/kvm_clock-1.patch > > ___ > freebsd-a...@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscr...@freebsd.org" ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [CFT] Paravirtualized KVM clock
On Sun, Jan 4, 2015 at 12:00 PM, Adrian Chadd wrote: > ... so, out of pure curiousity - what's making the benchmark go > faster? Is it userland side of things calling clock methods, or > something in the kernel, or both? > > Most likely GEOM statistic gathering in the kernel but Bryan would have to confirm. I intermittently saw this same kind of massive slowdown in nvme(4) performance a couple of years back due to a bug in the TSC self-check code which has since been fixed. The bug would result in falling back to HPET and all of the clock calls from the GEOM code for each I/O would kill performance. > > -adrian > > > On 4 January 2015 at 09:56, Bryan Venteicher > wrote: > > For the last few weeks, I've been working on adding support for KVM clock > > in the projects/paravirt branch. Currently, a KVM VM guest will end up > > selecting either the HPET or ACPI as the timecounter source. > Unfortunately, > > this is very costly since every timecounter fetch causes a VM exit. KVM > > clock allows the guest to use the TSC instead; it is very similar to the > > existing Xen timer. > > > > The performance difference between HPET/ACPI and KVMCLOCK can be > dramatic: > > a simple disk benchmark goes from 10K IOPs to 100K IOPs. > > > > The patch is attached is attached or available at [1]. I'd appreciate any > > testing. > > > > Also as a part of this, I've tried to generalized a bit of our existing > > hypervisor guest code, with the eventual goal of being able to support > more > > invasive PV operations. The patch series is viewable in Phabricator. > > > > https://reviews.freebsd.org/D1429 - paravirt: Generalize parts of the > XEN > > timer code into pvclock > > https://reviews.freebsd.org/D1430 - paravirt: Add interface to calculate > > the TSC frequency from pvclock > > https://reviews.freebsd.org/D1431 - paravirt: Add simple hypervisor > > registration and detection interface > > https://reviews.freebsd.org/D1432 - paravirt: Add detection of bhyve > using > > new hypervisor interface > > https://reviews.freebsd.org/D1433 - paravirt: Add detection of VMware > using > > new hypervisor interface > > https://reviews.freebsd.org/D1434 - paravirt: Add detection of KVM using > > new hypervisor interface > > https://reviews.freebsd.org/D1435 - paravirt: Add KVM clock timecounter > > support > > > > My current plan is to MFC this series to 10-STABLE, and commit a > > self-contained KVM clock to the other stable branches. > > > > [1] - https://people.freebsd.org/~bryanv/patches/kvm_clock-1.patch > > > > ___ > > freebsd-a...@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > > To unsubscribe, send any mail to "freebsd-arch-unsubscr...@freebsd.org" > ___ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" > ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [CFT] Paravirtualized KVM clock
On Sun, Jan 4, 2015 at 8:01 PM, Jim Harris wrote: > > > On Sun, Jan 4, 2015 at 12:00 PM, Adrian Chadd wrote: > >> ... so, out of pure curiousity - what's making the benchmark go >> faster? Is it userland side of things calling clock methods, or >> something in the kernel, or both? >> >> > Most likely GEOM statistic gathering in the kernel but Bryan would have to > confirm. > > Yes - t hat's the main source . A similar issue exists in the network stack BPF. I haven't looked or thought too much if it make sense / is possible to use kvmclock in userland too (I think kib@ added fast gettimeofday & friends support a few years back). I intermittently saw this same kind of massive slowdown in nvme(4) > performance a couple of years back due to a bug in the TSC self-check code > which has since been fixed. The bug would result in falling back to HPET > and all of the clock calls from the GEOM code for each I/O would kill > performance. > > >> >> -adrian >> >> >> On 4 January 2015 at 09:56, Bryan Venteicher >> wrote: >> > For the last few weeks, I've been working on adding support for KVM >> clock >> > in the projects/paravirt branch. Currently, a KVM VM guest will end up >> > selecting either the HPET or ACPI as the timecounter source. >> Unfortunately, >> > this is very costly since every timecounter fetch causes a VM exit. KVM >> > clock allows the guest to use the TSC instead; it is very similar to the >> > existing Xen timer. >> > >> > The performance difference between HPET/ACPI and KVMCLOCK can be >> dramatic: >> > a simple disk benchmark goes from 10K IOPs to 100K IOPs. >> > >> > The patch is attached is attached or available at [1]. I'd appreciate >> any >> > testing. >> > >> > Also as a part of this, I've tried to generalized a bit of our existing >> > hypervisor guest code, with the eventual goal of being able to support >> more >> > invasive PV operations. The patch series is viewable in Phabricator. >> > >> > https://reviews.freebsd.org/D1429 - paravirt: Generalize parts of the >> XEN >> > timer code into pvclock >> > https://reviews.freebsd.org/D1430 - paravirt: Add interface to >> calculate >> > the TSC frequency from pvclock >> > https://reviews.freebsd.org/D1431 - paravirt: Add simple hypervisor >> > registration and detection interface >> > https://reviews.freebsd.org/D1432 - paravirt: Add detection of bhyve >> using >> > new hypervisor interface >> > https://reviews.freebsd.org/D1433 - paravirt: Add detection of VMware >> using >> > new hypervisor interface >> > https://reviews.freebsd.org/D1434 - paravirt: Add detection of KVM >> using >> > new hypervisor interface >> > https://reviews.freebsd.org/D1435 - paravirt: Add KVM clock timecounter >> > support >> > >> > My current plan is to MFC this series to 10-STABLE, and commit a >> > self-contained KVM clock to the other stable branches. >> > >> > [1] - https://people.freebsd.org/~bryanv/patches/kvm_clock-1.patch >> > >> > ___ >> > freebsd-a...@freebsd.org mailing list >> > http://lists.freebsd.org/mailman/listinfo/freebsd-arch >> > To unsubscribe, send any mail to "freebsd-arch-unsubscr...@freebsd.org" >> ___ >> freebsd-current@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-current >> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org >> " >> > > ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [CFT] Paravirtualized KVM clock
On 2015-Jan-04 11:56:14 -0600, Bryan Venteicher wrote: >For the last few weeks, I've been working on adding support for KVM clock >in the projects/paravirt branch. Currently, a KVM VM guest will end up >selecting either the HPET or ACPI as the timecounter source. Unfortunately, >this is very costly since every timecounter fetch causes a VM exit. KVM >clock allows the guest to use the TSC instead; it is very similar to the >existing Xen timer. A somewhat late response but have you looked at https://github.com/blitz/freebsd/commit/cdc5f872b3e48cc0dda031fc7d6bdedc65c3148f I've been running this[*] on a Google Compute Engine instance for about 6 months without problems. [*] I had to patch out the test for KVM_FEATURE_CLOCKSOURCE_STABLE_BIT but I think that's a GCE issue. -- Peter Jeremy pgpi9_M8QUFuE.pgp Description: PGP signature
Re: [CFT] Paravirtualized KVM clock
On Wed, Jan 21, 2015 at 3:15 PM, Peter Jeremy wrote: > On 2015-Jan-04 11:56:14 -0600, Bryan Venteicher < > bry...@daemoninthecloset.org> wrote: > >For the last few weeks, I've been working on adding support for KVM clock > >in the projects/paravirt branch. Currently, a KVM VM guest will end up > >selecting either the HPET or ACPI as the timecounter source. > Unfortunately, > >this is very costly since every timecounter fetch causes a VM exit. KVM > >clock allows the guest to use the TSC instead; it is very similar to the > >existing Xen timer. > > A somewhat late response but have you looked at > > https://github.com/blitz/freebsd/commit/cdc5f872b3e48cc0dda031fc7d6bdedc65c3148f > I've been running this[*] on a Google Compute Engine instance for about 6 > months without problems. > > A goal of my work was to put a bit of infrastructure in place so FreeBSD can support pvops across a variety of hypervisors. KVMCLOCK happens to be about the easiest to implement, and has a decent performance win for many situations. I think that commit is broken on SMP guests: CPU_FOREACH() does not switch the current CPU, so it just keeps writing to the MSR on the BSP. [*] I had to patch out the test for KVM_FEATURE_CLOCKSOURCE_STABLE_BIT but > I think that's a GCE issue. > > -- > Peter Jeremy > ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
