Re: [PATCH 0/2] Introduce panic hypercall
HI, I am using kvm-kmod with version 2.6.36, but I encountered a fatal problem. In linux run on KVM,I using request(irq,*,*,*) function to request a interrupt line. However, When I using a irq bigger than 15, it will return error. Now I have simulated a PCI device and decide to using IOAPIC to send interrupts.So I decide assign 20,21 line for my device . but in guest os linux I could not request these number for my device. I know PIC just have 16 interrupt lines. I am not sure if the KVM really support APIC, and how to request a irq line more than 15? Waiting for your help on line. Thanks.. Shawn. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/2] Introduce panic hypercall
On 06/21/2011 12:03 PM, Gleb Natapov wrote:> >> > >Something like steal time, but for watchdog. But this become complicated fast. > >Watchdog emulation will have to move into kernel for starter. > > Why? You can use a performance counter from userspace. > Heh, haven't thought about such way of implementing watchdog device. But the same question again: what impact on performance constantly running guest under perf is? Same answer. Doesn't running guest under perf involve a lot of NMIs (and hence vmexists)? The NMI rate is proportional to how you program the counter. If you program it to exit every 2 billion cycles, that's what you'll get. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/2] Introduce panic hypercall
On Tue, Jun 21, 2011 at 11:56:26AM +0300, Avi Kivity wrote: > On 06/21/2011 11:41 AM, Gleb Natapov wrote: > >On Tue, Jun 21, 2011 at 11:09:21AM +0300, Avi Kivity wrote: > >> On 06/21/2011 09:02 AM, Gleb Natapov wrote: > >> >On Mon, Jun 20, 2011 at 07:34:57PM +0300, Avi Kivity wrote: > >> >> >The only two things which came to my mind are: > >> >> > > >> >> >* NMI (aka. ipmitool diag) - already available in qemu/kvm - > >> but requires > >> >> > in-guest kexec/kdump > >> >> >* Hardware-Watchdog (also available in qemu/libvirt) > >> >> > >> >> A watchdog has the advantage that is also detects lockups. > >> >> > >> >And has disadvantage that all time base heuristics are bite us in the > >> >end. > >> > >> The perf-based watchdog counts clocks-not-halted, not time, so it is > >> safe from time issues. > >So it counts only instruction that guest actually executed? That's > >perfect then. How much overhead it has in a guest though? > > Should be pretty low, especially if the guest is idle. There were > some negative reports about the pmu From David Ahern, so it needs to > be verified. But it will be running not only when the guest is idle. > > > >> We could make the hardware watchdog cheat in > >> the same way. > >> > >Something like steal time, but for watchdog. But this become complicated > >fast. > >Watchdog emulation will have to move into kernel for starter. > > Why? You can use a performance counter from userspace. > Heh, haven't thought about such way of implementing watchdog device. But the same question again: what impact on performance constantly running guest under perf is? Doesn't running guest under perf involve a lot of NMIs (and hence vmexists)? -- Gleb. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/2] Introduce panic hypercall
On 06/21/2011 11:41 AM, Gleb Natapov wrote: On Tue, Jun 21, 2011 at 11:09:21AM +0300, Avi Kivity wrote: > On 06/21/2011 09:02 AM, Gleb Natapov wrote: > >On Mon, Jun 20, 2011 at 07:34:57PM +0300, Avi Kivity wrote: > >> >The only two things which came to my mind are: > >> > > >> >* NMI (aka. ipmitool diag) - already available in qemu/kvm - but requires > >> > in-guest kexec/kdump > >> >* Hardware-Watchdog (also available in qemu/libvirt) > >> > >> A watchdog has the advantage that is also detects lockups. > >> > >And has disadvantage that all time base heuristics are bite us in the > >end. > > The perf-based watchdog counts clocks-not-halted, not time, so it is > safe from time issues. So it counts only instruction that guest actually executed? That's perfect then. How much overhead it has in a guest though? Should be pretty low, especially if the guest is idle. There were some negative reports about the pmu From David Ahern, so it needs to be verified. > We could make the hardware watchdog cheat in > the same way. > Something like steal time, but for watchdog. But this become complicated fast. Watchdog emulation will have to move into kernel for starter. Why? You can use a performance counter from userspace. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/2] Introduce panic hypercall
On Tue, Jun 21, 2011 at 11:09:21AM +0300, Avi Kivity wrote: > On 06/21/2011 09:02 AM, Gleb Natapov wrote: > >On Mon, Jun 20, 2011 at 07:34:57PM +0300, Avi Kivity wrote: > >> >The only two things which came to my mind are: > >> > > >> > * NMI (aka. ipmitool diag) - already available in qemu/kvm - but > >> requires > >> > in-guest kexec/kdump > >> > * Hardware-Watchdog (also available in qemu/libvirt) > >> > >> A watchdog has the advantage that is also detects lockups. > >> > >And has disadvantage that all time base heuristics are bite us in the > >end. > > The perf-based watchdog counts clocks-not-halted, not time, so it is > safe from time issues. So it counts only instruction that guest actually executed? That's perfect then. How much overhead it has in a guest though? > We could make the hardware watchdog cheat in > the same way. > Something like steal time, but for watchdog. But this become complicated fast. Watchdog emulation will have to move into kernel for starter. -- Gleb. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/2] Introduce panic hypercall
On 06/21/2011 09:02 AM, Gleb Natapov wrote: On Mon, Jun 20, 2011 at 07:34:57PM +0300, Avi Kivity wrote: > >The only two things which came to my mind are: > > > > * NMI (aka. ipmitool diag) - already available in qemu/kvm - but requires > > in-guest kexec/kdump > > * Hardware-Watchdog (also available in qemu/libvirt) > > A watchdog has the advantage that is also detects lockups. > And has disadvantage that all time base heuristics are bite us in the end. The perf-based watchdog counts clocks-not-halted, not time, so it is safe from time issues. We could make the hardware watchdog cheat in the same way. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/2] Introduce panic hypercall
On Mon, Jun 20, 2011 at 08:23:20PM +0300, Avi Kivity wrote: > On 06/20/2011 08:13 PM, Jan Kiszka wrote: > >> A watchdog has the advantage that is also detects lockups. > >> > >> In fact you could implement the panic device via the existing > >> watchdogs. Simply program the timer for the minimum interval and > >> *don't* service the interrupt. This would work for non-virt setups as > >> well as another way to issue a reset. > > > >If you manage to bring down the other guest CPUs fast enough. Otherwise, > >they may corrupt your crashdump before the host had a chance to collect > >all pieces. Synchronous signaling to the hypervisor is a bit safer. > > You could NMI-IPI them. But I agree a synchronous signal is better > (note it's not race-free itself). > But kexec/kdump has exactly same race, so this is at least not worse that alternative. -- Gleb. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/2] Introduce panic hypercall
On Mon, Jun 20, 2011 at 07:34:57PM +0300, Avi Kivity wrote: > >The only two things which came to my mind are: > > > > * NMI (aka. ipmitool diag) - already available in qemu/kvm - but requires > >in-guest kexec/kdump > > * Hardware-Watchdog (also available in qemu/libvirt) > > A watchdog has the advantage that is also detects lockups. > And has disadvantage that all time base heuristics are bite us in the end. -- Gleb. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/2] Introduce panic hypercall
On 06/20/2011 10:31 AM, Avi Kivity wrote: On 06/20/2011 04:38 PM, Daniel Gollub wrote: Introduce panic hypercall to enable the crashing guest to notify the host. This enables the host to run some actions as soon a guest crashed (kernel panic). This patch series introduces the panic hypercall at the host end. As well as the hypercall for KVM paravirtuliazed Linux guests, by registering the hypercall to the panic_notifier_list. The basic idea is to create KVM crashdump automatically as soon the guest paniced and power-cycle the VM (e.g. libvirt). This would be more easily done via a "panic device" (I/O port or memory-mapped address) that the guest hits. It would be intercepted by qemu without any new code in kvm.\ However, I'm not sure I see the gain. Most enterprisey guests already contain in-guest crash dumpers which provide more information than a qemu memory dump could, since they know exact load addresses etc. and are integrated with crash analysis tools. What do you have in mind? FYI, s390 has this functionality. It's useful because there's no use in having a guest just spin in a panic loop. Crash dump integration is much more complicated and requires functioning networking or some paravirt channel. Regards, Anthony Liguori -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/2] Introduce panic hypercall
On 06/20/2011 08:13 PM, Jan Kiszka wrote: > A watchdog has the advantage that is also detects lockups. > > In fact you could implement the panic device via the existing > watchdogs. Simply program the timer for the minimum interval and > *don't* service the interrupt. This would work for non-virt setups as > well as another way to issue a reset. If you manage to bring down the other guest CPUs fast enough. Otherwise, they may corrupt your crashdump before the host had a chance to collect all pieces. Synchronous signaling to the hypervisor is a bit safer. You could NMI-IPI them. But I agree a synchronous signal is better (note it's not race-free itself). -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/2] Introduce panic hypercall
On 2011-06-20 18:34, Avi Kivity wrote: >> > >> > Do ILO cards / IPMI support something like this? We could follow >> their >> > lead in that case. >> >> The only two things which came to my mind are: >> >> * NMI (aka. ipmitool diag) - already available in qemu/kvm - but >> requires >> in-guest kexec/kdump >> * Hardware-Watchdog (also available in qemu/libvirt) > > A watchdog has the advantage that is also detects lockups. > > In fact you could implement the panic device via the existing > watchdogs. Simply program the timer for the minimum interval and > *don't* service the interrupt. This would work for non-virt setups as > well as another way to issue a reset. If you manage to bring down the other guest CPUs fast enough. Otherwise, they may corrupt your crashdump before the host had a chance to collect all pieces. Synchronous signaling to the hypervisor is a bit safer. Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/2] Introduce panic hypercall
On 06/20/2011 07:26 PM, Daniel Gollub wrote: > > I agree. But let's do this via a device, this way kvm need not be changed. Is a device reliable enough if the guest kernel crashes? Do you mean something like a hardware watchdog? I'm proposing a 1:1 equivalent. Instead of issuing a hypercall that tells the host about the panic, write to an I/O port that tells the host about the panic. > > Do ILO cards / IPMI support something like this? We could follow their > lead in that case. The only two things which came to my mind are: * NMI (aka. ipmitool diag) - already available in qemu/kvm - but requires in-guest kexec/kdump * Hardware-Watchdog (also available in qemu/libvirt) A watchdog has the advantage that is also detects lockups. In fact you could implement the panic device via the existing watchdogs. Simply program the timer for the minimum interval and *don't* service the interrupt. This would work for non-virt setups as well as another way to issue a reset. lguest and xen have something similar. They also have an hypercall which get called by a function registered in the panic_notifier_list. Not quite sure if you want to follow their lead. We could do the same, except s/hypercall/writel/. Something I forgot to mention: This panic hypercall could also sit within an external kernel module ... to support (legacy) distribution. Yes. > > > This series does need to introduce a QMP event notification upon > > crash, so that the crash notification can be propagated to mgmt > > layers above QEMU. > > Yes. Already done. I posted the QEMU relevant changes as a separated series to the KVM list ... since the initial implementation is KVM specific (KVM hypercall) -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/2] Introduce panic hypercall
On Monday, June 20, 2011 05:45:36 pm Avi Kivity wrote: > > > However, I'm not sure I see the gain. Most enterprisey guests > > > already contain in-guest crash dumpers which provide more > > > information than a qemu memory dump could, since they know exact > > > load addresses etc. and are integrated with crash analysis tools. > > > What do you have in mind? Right kexec/kdump works perfectly already inside the guest. But: - in the "field" a lot of people still manage to setup VM guest without kexec/kdump properly setup (even though most enterprisey distribution try hard to setup this up out-of-the-box .. still people manage to not have kexec/kdump loaded once they run into a crash). - you don't have to reserve disk space for a crashdump for each guest e.g. if you run 4 guests with 60 GB of memory each you would loose somehow 4*60 GB space ... just for the (rare) case that each of those guest could write a crashdump, uncompressed ... - legacy distribution - no or buggy kexec - maybe writing a crashdump+reboot with QEMU/libvirt is faster then with in-guest kexec/kdump? (haven't tested yet) - single place on the VM-host to collect coredumps > > > > Well libvirt can capture a "core" file by doing 'virsh dump $GUESTNAME'. > > This actually uses the QEMU monitor migration command to capture the > > entire of QEMU memory. The 'crash' command line tool actually knows > > how to analyse this data format as it would a normal kernel crashdump. > > Interesting. Right. I'm using the kvmdump support of the crash utility now and then ... it could be more often. But unfortunately the people who run KVM in a productive environment with some strict service-level-agreement often just reboot, due to time pressure, or run out of disk space in the guest, or just forgot that they got told to do always "virsh dump" on a freeze or crash. > > > I think having a way for a guest OS to notify the host that is has > > crashed would be useful. libvirt could automatically do a crash > > dump of the QEMU memory, or at least pause the guest CPUs and notify > > the management app of the crash, which can then decide what todo. > > You can also use tools like 'virt-dmesg' which uses libvirt to peek > > into guest memory to extract the most recent kernel dmesg logs (even > > if the guest OS itself is crashed& didn't manage to send them out > > via netconsole or something else). > > I agree. But let's do this via a device, this way kvm need not be changed. Is a device reliable enough if the guest kernel crashes? Do you mean something like a hardware watchdog? > > Do ILO cards / IPMI support something like this? We could follow their > lead in that case. The only two things which came to my mind are: * NMI (aka. ipmitool diag) - already available in qemu/kvm - but requires in-guest kexec/kdump * Hardware-Watchdog (also available in qemu/libvirt) lguest and xen have something similar. They also have an hypercall which get called by a function registered in the panic_notifier_list. Not quite sure if you want to follow their lead. Something I forgot to mention: This panic hypercall could also sit within an external kernel module ... to support (legacy) distribution. > > > This series does need to introduce a QMP event notification upon > > crash, so that the crash notification can be propagated to mgmt > > layers above QEMU. > > Yes. Already done. I posted the QEMU relevant changes as a separated series to the KVM list ... since the initial implementation is KVM specific (KVM hypercall) Best Regards, Daniel -- Daniel Gollub Linux Consultant & Developer Tel.: +49-160 47 73 970 Mail: gol...@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537 signature.asc Description: This is a digitally signed message part.
Re: [PATCH 0/2] Introduce panic hypercall
On 2011-06-20 17:45, Avi Kivity wrote: >> This series does need to introduce a QMP event notification upon >> crash, so that the crash notification can be propagated to mgmt >> layers above QEMU. > > Yes. I think the best way to deal with that is to stop the VM on guest panic. There is already WIP to signal stop reasons via QMP. Maybe we need to differentiate between hypervisor and guest triggered panics (VMSTOP_GUEST_PANIC?), but the rest should come for free. Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/2] Introduce panic hypercall
On 06/20/2011 06:38 PM, Daniel P. Berrange wrote: On Mon, Jun 20, 2011 at 06:31:23PM +0300, Avi Kivity wrote: > On 06/20/2011 04:38 PM, Daniel Gollub wrote: > >Introduce panic hypercall to enable the crashing guest to notify the > >host. This enables the host to run some actions as soon a guest > >crashed (kernel panic). > > > >This patch series introduces the panic hypercall at the host end. > >As well as the hypercall for KVM paravirtuliazed Linux guests, by > >registering the hypercall to the panic_notifier_list. > > > >The basic idea is to create KVM crashdump automatically as soon the > >guest paniced and power-cycle the VM (e.g. libvirt). > > This would be more easily done via a "panic device" (I/O port or > memory-mapped address) that the guest hits. It would be intercepted > by qemu without any new code in kvm.\ > > However, I'm not sure I see the gain. Most enterprisey guests > already contain in-guest crash dumpers which provide more > information than a qemu memory dump could, since they know exact > load addresses etc. and are integrated with crash analysis tools. > What do you have in mind? Well libvirt can capture a "core" file by doing 'virsh dump $GUESTNAME'. This actually uses the QEMU monitor migration command to capture the entire of QEMU memory. The 'crash' command line tool actually knows how to analyse this data format as it would a normal kernel crashdump. Interesting. I think having a way for a guest OS to notify the host that is has crashed would be useful. libvirt could automatically do a crash dump of the QEMU memory, or at least pause the guest CPUs and notify the management app of the crash, which can then decide what todo. You can also use tools like 'virt-dmesg' which uses libvirt to peek into guest memory to extract the most recent kernel dmesg logs (even if the guest OS itself is crashed& didn't manage to send them out via netconsole or something else). I agree. But let's do this via a device, this way kvm need not be changed. Do ILO cards / IPMI support something like this? We could follow their lead in that case. This series does need to introduce a QMP event notification upon crash, so that the crash notification can be propagated to mgmt layers above QEMU. Yes. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/2] Introduce panic hypercall
On Mon, Jun 20, 2011 at 06:31:23PM +0300, Avi Kivity wrote: > On 06/20/2011 04:38 PM, Daniel Gollub wrote: > >Introduce panic hypercall to enable the crashing guest to notify the > >host. This enables the host to run some actions as soon a guest > >crashed (kernel panic). > > > >This patch series introduces the panic hypercall at the host end. > >As well as the hypercall for KVM paravirtuliazed Linux guests, by > >registering the hypercall to the panic_notifier_list. > > > >The basic idea is to create KVM crashdump automatically as soon the > >guest paniced and power-cycle the VM (e.g. libvirt). > > This would be more easily done via a "panic device" (I/O port or > memory-mapped address) that the guest hits. It would be intercepted > by qemu without any new code in kvm.\ > > However, I'm not sure I see the gain. Most enterprisey guests > already contain in-guest crash dumpers which provide more > information than a qemu memory dump could, since they know exact > load addresses etc. and are integrated with crash analysis tools. > What do you have in mind? Well libvirt can capture a "core" file by doing 'virsh dump $GUESTNAME'. This actually uses the QEMU monitor migration command to capture the entire of QEMU memory. The 'crash' command line tool actually knows how to analyse this data format as it would a normal kernel crashdump. I think having a way for a guest OS to notify the host that is has crashed would be useful. libvirt could automatically do a crash dump of the QEMU memory, or at least pause the guest CPUs and notify the management app of the crash, which can then decide what todo. You can also use tools like 'virt-dmesg' which uses libvirt to peek into guest memory to extract the most recent kernel dmesg logs (even if the guest OS itself is crashed & didn't manage to send them out via netconsole or something else). This series does need to introduce a QMP event notification upon crash, so that the crash notification can be propagated to mgmt layers above QEMU. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/2] Introduce panic hypercall
On 06/20/2011 04:38 PM, Daniel Gollub wrote: Introduce panic hypercall to enable the crashing guest to notify the host. This enables the host to run some actions as soon a guest crashed (kernel panic). This patch series introduces the panic hypercall at the host end. As well as the hypercall for KVM paravirtuliazed Linux guests, by registering the hypercall to the panic_notifier_list. The basic idea is to create KVM crashdump automatically as soon the guest paniced and power-cycle the VM (e.g. libvirt). This would be more easily done via a "panic device" (I/O port or memory-mapped address) that the guest hits. It would be intercepted by qemu without any new code in kvm.\ However, I'm not sure I see the gain. Most enterprisey guests already contain in-guest crash dumpers which provide more information than a qemu memory dump could, since they know exact load addresses etc. and are integrated with crash analysis tools. What do you have in mind? -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 0/2] Introduce panic hypercall
Introduce panic hypercall to enable the crashing guest to notify the host. This enables the host to run some actions as soon a guest crashed (kernel panic). This patch series introduces the panic hypercall at the host end. As well as the hypercall for KVM paravirtuliazed Linux guests, by registering the hypercall to the panic_notifier_list. The basic idea is to create KVM crashdump automatically as soon the guest paniced and power-cycle the VM (e.g. libvirt ). Daniel Gollub (2): Inroduce panic hypercall KVM_HC_PANIC (host) Call KVM_HC_PANIC if guest panics arch/x86/kernel/kvm.c| 16 arch/x86/kvm/x86.c |9 + include/linux/kvm.h |1 + include/linux/kvm_host.h |1 + include/linux/kvm_para.h |1 + 5 files changed, 28 insertions(+), 0 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
