Re: nmi is broken?
On 05/03/2011 08:45 PM, OGAWA Hirofumi wrote: Avi Kivitya...@redhat.com writes: On 05/03/2011 07:57 PM, OGAWA Hirofumi wrote: Usually yes, but we already have the PIT wired to INTIN0. I saw that the kernel consults the mptable to see which pin to use, so with the right BIOS magic we can get things to work. Um..., I'm confused more. If so, MADT doesn't say it. MADT says irq == 0 is pin == 2, this is one of reasons why linux is quite silent in check_timer(). And I can't see why it is working by pin == 2 for IOAPIC. If I can make time, I'll see what happens by pin == 2 and pin == 0 of IOAPIC in kvm. You're right. The default routing is INTIN0, but qemu changes it to INTIN2 and tells kvm. So INTIN0 is free for the 8259A output. I see. Did it mean qemu changes the wiring, so kvm can't work for live migration with it? qemu changes the wiring, but it does so on both the migration source and the migration target (before either is launched), so they are migration compatible. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
Avi Kivity a...@redhat.com writes: On 05/03/2011 08:45 PM, OGAWA Hirofumi wrote: Avi Kivitya...@redhat.com writes: On 05/03/2011 07:57 PM, OGAWA Hirofumi wrote: Usually yes, but we already have the PIT wired to INTIN0. I saw that the kernel consults the mptable to see which pin to use, so with the right BIOS magic we can get things to work. Um..., I'm confused more. If so, MADT doesn't say it. MADT says irq == 0 is pin == 2, this is one of reasons why linux is quite silent in check_timer(). And I can't see why it is working by pin == 2 for IOAPIC. If I can make time, I'll see what happens by pin == 2 and pin == 0 of IOAPIC in kvm. You're right. The default routing is INTIN0, but qemu changes it to INTIN2 and tells kvm. So INTIN0 is free for the 8259A output. I see. Did it mean qemu changes the wiring, so kvm can't work for live migration with it? qemu changes the wiring, but it does so on both the migration source and the migration target (before either is launched), so they are migration compatible. OK. So, kvm have to detect current wiring to change behavior. (E.g. nmi watchdog of IOAPIC mode without irq_ack.) IOW, if kvm can detect it, we would be able to solve issues. Would my thought be true? Thanks. -- OGAWA Hirofumi hirof...@mail.parknet.co.jp -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
On 05/04/2011 06:40 PM, OGAWA Hirofumi wrote: qemu changes the wiring, but it does so on both the migration source and the migration target (before either is launched), so they are migration compatible. OK. So, kvm have to detect current wiring to change behavior. (E.g. nmi watchdog of IOAPIC mode without irq_ack.) IOW, if kvm can detect it, we would be able to solve issues. Would my thought be true? I think so. It is also likely that all guests mask INTIN0 in the redirection table entry (except when the watchdog is active). In that case the wiring different would make no change, and compatibility would be maintained. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
On 05/02/2011 05:30 PM, OGAWA Hirofumi wrote: So, I think there are some solutions, a) current behavior is right (I don't know why it's right though), b) fix the behavior of IO-APIC and MADT like you said, then linux can detect it, c) change the model to like mpspec figure 5-2, d) other. My suggestion is c) if there is no good d). Because current behavior looks like almost c), and non-legacy chipsets are using c) model as far as I know. You're probably right. However we can't just change it, we need to make it an option, keeping the current configuration as the default. This is so that live migration can work, and because 5-2 requires a new kernel/user interface, to set IMCR.E0. Looking at figures 3-3 and 3-4 of the mpspec, the current model supports 3-3 but not 3-4. Do we report that IMCR exists in the mptables? I don't think we have to implement IMCR, because it seems to be an optional. In fact, physical hardwares which I have don't report IMCR in mptables. (I don't see the benefit to implement it on recent chipsets even if it's possible. The virtual wire mode seems to be enough.) Okay. I don't know about live migration of kvm. If we said the wiring is like figure 5-2, what is required for the live migration? It was required only if IMCR was required? The issue with live migration is that we can't change the running configuration while the system is running, like adding the IMCR or changing the wiring. The hardware will be programmed for the old configuration and will likely fail with the new one. For example, the current wiring has the PIT output wired to PIC IRQ0 and IOAPIC INTI0; we need to change it to IOAPIC INTI2 instead. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
On 04/24/2011 05:08 PM, Jan Kiszka wrote: On 2011-04-24 08:44, Avi Kivity wrote: On 04/24/2011 01:50 AM, OGAWA Hirofumi wrote: OGAWA Hirofumihirof...@mail.parknet.co.jp writes: I noticed recently NMI on guest kernel is not working well. host/guest kernel is 2.6.39-rc4, and using vmx. And test code is something like the following: local_irq_disable(); for (i = 0; i 10; i++) { int cpu = get_cpu(); printk(%s: nmi %u, lapic %u\n, __FUNCTION__, nmi_count(cpu), irq_stat[cpu].apic_timer_irqs); mdelay(1000); put_cpu(); } the result is both of nmi and lapic are not increased. If I used -no-kvm-irqchip, it works fine (increase nmi only). So, it seems to be the bug of kvm driver side. With some debug, the cause seems to be in pit_do_work(). With the following patch, NMI watchdog seems to be working correctly (if irq disabled for long time, NMI watchdog can detect it). Is the following patch right? This would cause IRQs to be delivered even if the PIT is masked, no? Are you in fact using the PIT? Linux prefers the HPET, and in my experience the -no-hpet option makes NMIs work. BTW, that's another bug of the in-kernel PIT model: It disables the timer in HPET legacy mode even if we are aware of NMI watchdog receivers. Actually, the whole legacy disabling looks a bit strange in the PIT (mode hackery + flag testing...). While this should be fixed/refactored, adding basic perf support to KVM will be the only option long-term as Linux dropped virtual-wire NMI watchdog support some releases ago. Yes. Unfortunately that is very vendor and model specific. The architectural PMU is supported, but that is only available on Intel. Perhaps we could emulate the architectural PMU on AMD as well, and make the detection code in the guest vendor agnostic. Since it's based on a cpuid bit, it should be safe. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
On 05/03/2011 12:36 PM, Avi Kivity wrote: I don't know about live migration of kvm. If we said the wiring is like figure 5-2, what is required for the live migration? It was required only if IMCR was required? The issue with live migration is that we can't change the running configuration while the system is running, like adding the IMCR or changing the wiring. The hardware will be programmed for the old configuration and will likely fail with the new one. For example, the current wiring has the PIT output wired to PIC IRQ0 and IOAPIC INTI0; we need to change it to IOAPIC INTI2 instead. btw, I believe that the configuration currently implemented is legal (it's similar to config 7 in table 5-2 of the mpspec); the only problem is that it can't support the NMI watchdog through the I/O APIC trick, yet we allow it through a hack. Something we can do is connect the 8259A output to the I/O APIC INTIN2; it should be masked so live migration will continue to work. We just have to make sure that the guest is able to find that it is connected there. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
On 2011-05-03 11:43, Avi Kivity wrote: On 04/24/2011 05:08 PM, Jan Kiszka wrote: On 2011-04-24 08:44, Avi Kivity wrote: On 04/24/2011 01:50 AM, OGAWA Hirofumi wrote: OGAWA Hirofumihirof...@mail.parknet.co.jp writes: I noticed recently NMI on guest kernel is not working well. host/guest kernel is 2.6.39-rc4, and using vmx. And test code is something like the following: local_irq_disable(); for (i = 0; i 10; i++) { int cpu = get_cpu(); printk(%s: nmi %u, lapic %u\n, __FUNCTION__, nmi_count(cpu), irq_stat[cpu].apic_timer_irqs); mdelay(1000); put_cpu(); } the result is both of nmi and lapic are not increased. If I used -no-kvm-irqchip, it works fine (increase nmi only). So, it seems to be the bug of kvm driver side. With some debug, the cause seems to be in pit_do_work(). With the following patch, NMI watchdog seems to be working correctly (if irq disabled for long time, NMI watchdog can detect it). Is the following patch right? This would cause IRQs to be delivered even if the PIT is masked, no? Are you in fact using the PIT? Linux prefers the HPET, and in my experience the -no-hpet option makes NMIs work. BTW, that's another bug of the in-kernel PIT model: It disables the timer in HPET legacy mode even if we are aware of NMI watchdog receivers. Actually, the whole legacy disabling looks a bit strange in the PIT (mode hackery + flag testing...). While this should be fixed/refactored, adding basic perf support to KVM will be the only option long-term as Linux dropped virtual-wire NMI watchdog support some releases ago. Yes. Unfortunately that is very vendor and model specific. The architectural PMU is supported, but that is only available on Intel. Is it supposed to have any practical value already? I did not yet find a magic -cpu switch to let Linux detect anything, not to speak of perf or watchdog support. Perhaps we could emulate the architectural PMU on AMD as well, and make the detection code in the guest vendor agnostic. Since it's based on a cpuid bit, it should be safe. We may only make Linux happy this way, no? Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
Avi Kivity a...@redhat.com writes: On 05/03/2011 12:36 PM, Avi Kivity wrote: I don't know about live migration of kvm. If we said the wiring is like figure 5-2, what is required for the live migration? It was required only if IMCR was required? The issue with live migration is that we can't change the running configuration while the system is running, like adding the IMCR or changing the wiring. The hardware will be programmed for the old configuration and will likely fail with the new one. For example, the current wiring has the PIT output wired to PIC IRQ0 and IOAPIC INTI0; we need to change it to IOAPIC INTI2 instead. btw, I believe that the configuration currently implemented is legal (it's similar to config 7 in table 5-2 of the mpspec); the only problem is that it can't support the NMI watchdog through the I/O APIC trick, yet we allow it through a hack. I'm confused a bit. config 7 in table 5-2 says PIT output wired to IOAPIC INTIN2. So, we don't need to change it? Something we can do is connect the 8259A output to the I/O APIC INTIN2; it should be masked so live migration will continue to work. We just have to make sure that the guest is able to find that it is connected there. 8259A INTR to IOAPIC INTIN2 sounds strange. Isn't it IOAPIC INTIN0? Thanks. -- OGAWA Hirofumi hirof...@mail.parknet.co.jp -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
On 05/03/2011 01:37 PM, Jan Kiszka wrote: Yes. Unfortunately that is very vendor and model specific. The architectural PMU is supported, but that is only available on Intel. Is it supposed to have any practical value already? I did not yet find a magic -cpu switch to let Linux detect anything, not to speak of perf or watchdog support. On the guest side it is supported for the watchdog (arch/x86/kernel/cpu/perfctr-watchdog.c, look for X86_FEATURE_ARCH_PERFMON). It's also mentioned in perf_event_intel.c, but I don't know if it will work without the other PMU features being present. Perhaps we could emulate the architectural PMU on AMD as well, and make the detection code in the guest vendor agnostic. Since it's based on a cpuid bit, it should be safe. We may only make Linux happy this way, no? I would argue that if a feature is discoverable by a cpuid bit it shouldn't need to be qualified by vendor. No idea how other OSes work this out (or even if they make use of the architectural PMU). -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
On 05/03/2011 04:25 PM, OGAWA Hirofumi wrote: Avi Kivitya...@redhat.com writes: On 05/03/2011 12:36 PM, Avi Kivity wrote: I don't know about live migration of kvm. If we said the wiring is like figure 5-2, what is required for the live migration? It was required only if IMCR was required? The issue with live migration is that we can't change the running configuration while the system is running, like adding the IMCR or changing the wiring. The hardware will be programmed for the old configuration and will likely fail with the new one. For example, the current wiring has the PIT output wired to PIC IRQ0 and IOAPIC INTI0; we need to change it to IOAPIC INTI2 instead. btw, I believe that the configuration currently implemented is legal (it's similar to config 7 in table 5-2 of the mpspec); the only problem is that it can't support the NMI watchdog through the I/O APIC trick, yet we allow it through a hack. I'm confused a bit. config 7 in table 5-2 says PIT output wired to IOAPIC INTIN2. So, we don't need to change it? We're like config 7 in that the 8259A output isn't wired to the IOAPIC. However we're unlike config 7 in that the PIT output is wired to IOAPIC INTIN0. I think we can keep it that way. Something we can do is connect the 8259A output to the I/O APIC INTIN2; it should be masked so live migration will continue to work. We just have to make sure that the guest is able to find that it is connected there. 8259A INTR to IOAPIC INTIN2 sounds strange. Isn't it IOAPIC INTIN0? Usually yes, but we already have the PIT wired to INTIN0. I saw that the kernel consults the mptable to see which pin to use, so with the right BIOS magic we can get things to work. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
On 2011-05-03 15:31, Avi Kivity wrote: On 05/03/2011 01:37 PM, Jan Kiszka wrote: Yes. Unfortunately that is very vendor and model specific. The architectural PMU is supported, but that is only available on Intel. Is it supposed to have any practical value already? I did not yet find a magic -cpu switch to let Linux detect anything, not to speak of perf or watchdog support. On the guest side it is supported for the watchdog (arch/x86/kernel/cpu/perfctr-watchdog.c, look for X86_FEATURE_ARCH_PERFMON). It's also mentioned in perf_event_intel.c, but I don't know if it will work without the other PMU features being present. I've tested with some SUSE 2.6.38 guest kernel, and it complained like this: (-cpu kvm64) Performance Events: unsupported Netburst CPU model 6 no PMU driver, software events only. NMI watchdog disabled (cpu0): hardware events not enabled or (-cpu host) Performance Events: unsupported p6 CPU model 37 no PMU driver, software events only. NMI watchdog disabled (cpu0): hardware events not enabled Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
On 05/03/2011 05:29 PM, Jan Kiszka wrote: On 2011-05-03 15:31, Avi Kivity wrote: On 05/03/2011 01:37 PM, Jan Kiszka wrote: Yes. Unfortunately that is very vendor and model specific. The architectural PMU is supported, but that is only available on Intel. Is it supposed to have any practical value already? I did not yet find a magic -cpu switch to let Linux detect anything, not to speak of perf or watchdog support. On the guest side it is supported for the watchdog (arch/x86/kernel/cpu/perfctr-watchdog.c, look for X86_FEATURE_ARCH_PERFMON). It's also mentioned in perf_event_intel.c, but I don't know if it will work without the other PMU features being present. I've tested with some SUSE 2.6.38 guest kernel, and it complained like this: (-cpu kvm64) Performance Events: unsupported Netburst CPU model 6 no PMU driver, software events only. NMI watchdog disabled (cpu0): hardware events not enabled Sorry, I meant to write, but forgot, that on the host side it is completely unsupported. It shouldn't be too hard to use perf_events to emulate the architectural PMU. Once we do that we can expose the architectural pmu bit and the guest will use it. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
On 2011-05-03 16:37, Avi Kivity wrote: On 05/03/2011 05:29 PM, Jan Kiszka wrote: On 2011-05-03 15:31, Avi Kivity wrote: On 05/03/2011 01:37 PM, Jan Kiszka wrote: Yes. Unfortunately that is very vendor and model specific. The architectural PMU is supported, but that is only available on Intel. Is it supposed to have any practical value already? I did not yet find a magic -cpu switch to let Linux detect anything, not to speak of perf or watchdog support. On the guest side it is supported for the watchdog (arch/x86/kernel/cpu/perfctr-watchdog.c, look for X86_FEATURE_ARCH_PERFMON). It's also mentioned in perf_event_intel.c, but I don't know if it will work without the other PMU features being present. I've tested with some SUSE 2.6.38 guest kernel, and it complained like this: (-cpu kvm64) Performance Events: unsupported Netburst CPU model 6 no PMU driver, software events only. NMI watchdog disabled (cpu0): hardware events not enabled Sorry, I meant to write, but forgot, that on the host side it is completely unsupported. It shouldn't be too hard to use perf_events to emulate the architectural PMU. Once we do that we can expose the architectural pmu bit and the guest will use it. Oh, and I already thought I would have missed some thrilling KVM patches... Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
Avi Kivity a...@redhat.com writes: The issue with live migration is that we can't change the running configuration while the system is running, like adding the IMCR or changing the wiring. The hardware will be programmed for the old configuration and will likely fail with the new one. For example, the current wiring has the PIT output wired to PIC IRQ0 and IOAPIC INTI0; we need to change it to IOAPIC INTI2 instead. btw, I believe that the configuration currently implemented is legal (it's similar to config 7 in table 5-2 of the mpspec); the only problem is that it can't support the NMI watchdog through the I/O APIC trick, yet we allow it through a hack. I'm confused a bit. config 7 in table 5-2 says PIT output wired to IOAPIC INTIN2. So, we don't need to change it? We're like config 7 in that the 8259A output isn't wired to the IOAPIC. However we're unlike config 7 in that the PIT output is wired to IOAPIC INTIN0. I think we can keep it that way. Something we can do is connect the 8259A output to the I/O APIC INTIN2; it should be masked so live migration will continue to work. We just have to make sure that the guest is able to find that it is connected there. 8259A INTR to IOAPIC INTIN2 sounds strange. Isn't it IOAPIC INTIN0? Usually yes, but we already have the PIT wired to INTIN0. I saw that the kernel consults the mptable to see which pin to use, so with the right BIOS magic we can get things to work. Um..., I'm confused more. If so, MADT doesn't say it. MADT says irq == 0 is pin == 2, this is one of reasons why linux is quite silent in check_timer(). And I can't see why it is working by pin == 2 for IOAPIC. If I can make time, I'll see what happens by pin == 2 and pin == 0 of IOAPIC in kvm. Thanks. -- OGAWA Hirofumi hirof...@mail.parknet.co.jp -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
On 05/03/2011 07:57 PM, OGAWA Hirofumi wrote: Usually yes, but we already have the PIT wired to INTIN0. I saw that the kernel consults the mptable to see which pin to use, so with the right BIOS magic we can get things to work. Um..., I'm confused more. If so, MADT doesn't say it. MADT says irq == 0 is pin == 2, this is one of reasons why linux is quite silent in check_timer(). And I can't see why it is working by pin == 2 for IOAPIC. If I can make time, I'll see what happens by pin == 2 and pin == 0 of IOAPIC in kvm. You're right. The default routing is INTIN0, but qemu changes it to INTIN2 and tells kvm. So INTIN0 is free for the 8259A output. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
Avi Kivity a...@redhat.com writes: On 05/03/2011 07:57 PM, OGAWA Hirofumi wrote: Usually yes, but we already have the PIT wired to INTIN0. I saw that the kernel consults the mptable to see which pin to use, so with the right BIOS magic we can get things to work. Um..., I'm confused more. If so, MADT doesn't say it. MADT says irq == 0 is pin == 2, this is one of reasons why linux is quite silent in check_timer(). And I can't see why it is working by pin == 2 for IOAPIC. If I can make time, I'll see what happens by pin == 2 and pin == 0 of IOAPIC in kvm. You're right. The default routing is INTIN0, but qemu changes it to INTIN2 and tells kvm. So INTIN0 is free for the 8259A output. I see. Did it mean qemu changes the wiring, so kvm can't work for live migration with it? Thanks. -- OGAWA Hirofumi hirof...@mail.parknet.co.jp -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
On 05/01/2011 04:45 AM, OGAWA Hirofumi wrote: Hm.., if smp was enabled, what configuration model is used by kvm? I think this configuration model can't work on smp. As far as I can see, kvm is not configured (from MADT and some of behaviors) like you said. We may well have an error there; and our NMI-from-PIT emulation bypasses all the wiring I described, so we may be emulating a configuration that can't possibly exist on hardware. So, I think there are some solutions, a) current behavior is right (I don't know why it's right though), b) fix the behavior of IO-APIC and MADT like you said, then linux can detect it, c) change the model to like mpspec figure 5-2, d) other. My suggestion is c) if there is no good d). Because current behavior looks like almost c), and non-legacy chipsets are using c) model as far as I know. You're probably right. However we can't just change it, we need to make it an option, keeping the current configuration as the default. This is so that live migration can work, and because 5-2 requires a new kernel/user interface, to set IMCR.E0. Looking at figures 3-3 and 3-4 of the mpspec, the current model supports 3-3 but not 3-4. Do we report that IMCR exists in the mptables? -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
Avi Kivity a...@redhat.com writes: On 05/01/2011 04:45 AM, OGAWA Hirofumi wrote: Hm.., if smp was enabled, what configuration model is used by kvm? I think this configuration model can't work on smp. As far as I can see, kvm is not configured (from MADT and some of behaviors) like you said. We may well have an error there; and our NMI-from-PIT emulation bypasses all the wiring I described, so we may be emulating a configuration that can't possibly exist on hardware. So, I think there are some solutions, a) current behavior is right (I don't know why it's right though), b) fix the behavior of IO-APIC and MADT like you said, then linux can detect it, c) change the model to like mpspec figure 5-2, d) other. My suggestion is c) if there is no good d). Because current behavior looks like almost c), and non-legacy chipsets are using c) model as far as I know. You're probably right. However we can't just change it, we need to make it an option, keeping the current configuration as the default. This is so that live migration can work, and because 5-2 requires a new kernel/user interface, to set IMCR.E0. Looking at figures 3-3 and 3-4 of the mpspec, the current model supports 3-3 but not 3-4. Do we report that IMCR exists in the mptables? I don't think we have to implement IMCR, because it seems to be an optional. In fact, physical hardwares which I have don't report IMCR in mptables. (I don't see the benefit to implement it on recent chipsets even if it's possible. The virtual wire mode seems to be enough.) I don't know about live migration of kvm. If we said the wiring is like figure 5-2, what is required for the live migration? It was required only if IMCR was required? Thanks. -- OGAWA Hirofumi hirof...@mail.parknet.co.jp -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
OGAWA Hirofumi hirof...@mail.parknet.co.jp writes: That is correct, kvm doesn't connect the master 8259 output to the IOAPIC. Instead the 8259 is connected to LINT0 (which is configured for ExtInt when the IOAPIC is disabled, or for NMI which the NMI watchdog is enabled). However, now I can't see how it would work. auto EOI works on the INTA cycle, which would only occur if LINT0 is configured for ExtInt. If it is configured for NMI, I don't think it would issue the INTA cycle. So the NMI watchdog not working is actually correct for our hardware configuration! But I may be misunderstanding something here. I see. If the physical machine was configured as above, I guess (not pretty sure, I don't have this configuration machine), IOAPIC test (check_timer() in io_apic.c) should fail, and IOAPIC wouldn't have any effect. And I think MADT should tell mp_ExtINT. Yes, I also guess the above configuration wouldn't work NMI watchdog of IOAPIC mode, and linux will report as NMI watchdog can't work in check_timer(). Hm.., if smp was enabled, what configuration model is used by kvm? I think this configuration model can't work on smp. As far as I can see, kvm is not configured (from MADT and some of behaviors) like you said. So, I think there are some solutions, a) current behavior is right (I don't know why it's right though), b) fix the behavior of IO-APIC and MADT like you said, then linux can detect it, c) change the model to like mpspec figure 5-2, d) other. My suggestion is c) if there is no good d). Because current behavior looks like almost c), and non-legacy chipsets are using c) model as far as I know. Thanks. -- OGAWA Hirofumi hirof...@mail.parknet.co.jp -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
On 04/28/2011 04:28 AM, OGAWA Hirofumi wrote: Avi Kivitya...@redhat.com writes: On 04/24/2011 03:24 PM, Jan Kiszka wrote: This would cause IRQs to be delivered even if the PIT is masked, no? I checked the patch and our code again: NMI watchdog masking is managed via arch.vapics_in_nmi_mode and by re-checking the per-APIC mask situation in kvm_apic_local_deliver when delivering the NMI. So the patch looks correct - NMIs aren't acked like timer IRQs, the current logic is definitely wrong. Can you elaborate? Why aren't NMIs acked (if delivered via the PIC)? Is the PIC programmed into auto-EOI mode or something? This seems to be complex stuff depending on hardware configurations. I'm not fully understanding though, current state of it is, Yes, PIC is in AEOI mode if linux is using IO-APIC. Um.., kvm says irq == 0 is mp_INT mode in MADT, not mp_ExtINT. That is correct, kvm doesn't connect the master 8259 output to the IOAPIC. Instead the 8259 is connected to LINT0 (which is configured for ExtInt when the IOAPIC is disabled, or for NMI which the NMI watchdog is enabled). However, now I can't see how it would work. auto EOI works on the INTA cycle, which would only occur if LINT0 is configured for ExtInt. If it is configured for NMI, I don't think it would issue the INTA cycle. So the NMI watchdog not working is actually correct for our hardware configuration! But I may be misunderstanding something here. So I guess system does automatically INTA cycle (and AEOI because of PIC config), or not connected via 8259A? (like in mpspec figure 5-2.) To checking it, I've tested in check_timer() of linux on the physical machine (irq==0 and mp_INT). The test is something like, if (pin1 != -1) { /* * Ok, does IRQ0 through the IOAPIC work? */ unmask_IO_APIC_irq(0); disable_8259_irq(0); if (timer_irq_works()) { even if I called disable_8259_irq(0), timer was still working via IO-APIC. Would this explain why? Sorry, I got lost - what does this explain? -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
Avi Kivity a...@redhat.com writes: This seems to be complex stuff depending on hardware configurations. I'm not fully understanding though, current state of it is, Yes, PIC is in AEOI mode if linux is using IO-APIC. Um.., kvm says irq == 0 is mp_INT mode in MADT, not mp_ExtINT. That is correct, kvm doesn't connect the master 8259 output to the IOAPIC. Instead the 8259 is connected to LINT0 (which is configured for ExtInt when the IOAPIC is disabled, or for NMI which the NMI watchdog is enabled). However, now I can't see how it would work. auto EOI works on the INTA cycle, which would only occur if LINT0 is configured for ExtInt. If it is configured for NMI, I don't think it would issue the INTA cycle. So the NMI watchdog not working is actually correct for our hardware configuration! But I may be misunderstanding something here. I see. If the physical machine was configured as above, I guess (not pretty sure, I don't have this configuration machine), IOAPIC test (check_timer() in io_apic.c) should fail, and IOAPIC wouldn't have any effect. And I think MADT should tell mp_ExtINT. Yes, I also guess the above configuration wouldn't work NMI watchdog of IOAPIC mode, and linux will report as NMI watchdog can't work in check_timer(). So I guess system does automatically INTA cycle (and AEOI because of PIC config), or not connected via 8259A? (like in mpspec figure 5-2.) To checking it, I've tested in check_timer() of linux on the physical machine (irq==0 and mp_INT). The test is something like, if (pin1 != -1) { /* * Ok, does IRQ0 through the IOAPIC work? */ unmask_IO_APIC_irq(0); disable_8259_irq(0); if (timer_irq_works()) { even if I called disable_8259_irq(0), timer was still working via IO-APIC. Would this explain why? Sorry, I got lost - what does this explain? I think this explains irq == 0 and mp_INT configuration tell PIT is connected to both of IOAPIC pin2 and PIC pin0, and it is why timer interrupt is working even when PIC pin0 was disabled. Thanks. -- OGAWA Hirofumi hirof...@mail.parknet.co.jp -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
OGAWA Hirofumi hirof...@mail.parknet.co.jp writes: Avi Kivity a...@redhat.com writes: This seems to be complex stuff depending on hardware configurations. I'm not fully understanding though, current state of it is, Yes, PIC is in AEOI mode if linux is using IO-APIC. Um.., kvm says irq == 0 is mp_INT mode in MADT, not mp_ExtINT. That is correct, kvm doesn't connect the master 8259 output to the IOAPIC. Instead the 8259 is connected to LINT0 (which is configured for ExtInt when the IOAPIC is disabled, or for NMI which the NMI watchdog is enabled). However, now I can't see how it would work. auto EOI works on the INTA cycle, which would only occur if LINT0 is configured for ExtInt. If it is configured for NMI, I don't think it would issue the INTA cycle. So the NMI watchdog not working is actually correct for our hardware configuration! But I may be misunderstanding something here. I see. If the physical machine was configured as above, I guess (not pretty sure, I don't have this configuration machine), IOAPIC test (check_timer() in io_apic.c) should fail, and IOAPIC wouldn't have any effect. And I think MADT should tell mp_ExtINT. Yes, I also guess the above configuration wouldn't work NMI watchdog of IOAPIC mode, and linux will report as NMI watchdog can't work in check_timer(). Hm.., if smp was enabled, what configuration model is used by kvm? I think this configuration model can't work on smp. Thanks. -- OGAWA Hirofumi hirof...@mail.parknet.co.jp -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
On 04/24/2011 03:24 PM, Jan Kiszka wrote: This would cause IRQs to be delivered even if the PIT is masked, no? I checked the patch and our code again: NMI watchdog masking is managed via arch.vapics_in_nmi_mode and by re-checking the per-APIC mask situation in kvm_apic_local_deliver when delivering the NMI. So the patch looks correct - NMIs aren't acked like timer IRQs, the current logic is definitely wrong. Can you elaborate? Why aren't NMIs acked (if delivered via the PIC)? Is the PIC programmed into auto-EOI mode or something? -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
On 2011-04-27 10:55, Avi Kivity wrote: On 04/24/2011 03:24 PM, Jan Kiszka wrote: This would cause IRQs to be delivered even if the PIT is masked, no? I checked the patch and our code again: NMI watchdog masking is managed via arch.vapics_in_nmi_mode and by re-checking the per-APIC mask situation in kvm_apic_local_deliver when delivering the NMI. So the patch looks correct - NMIs aren't acked like timer IRQs, the current logic is definitely wrong. Can you elaborate? Why aren't NMIs acked (if delivered via the PIC)? Is the PIC programmed into auto-EOI mode or something? At least there are no traces of 8259 interaction in the NMI handler of old kernels when configured to IOAPIC mode. This wouldn't work anyway as you can't synchronize with other PIC-accessing functions from the NMI handler. Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
Avi Kivity a...@redhat.com writes: On 04/24/2011 03:24 PM, Jan Kiszka wrote: This would cause IRQs to be delivered even if the PIT is masked, no? I checked the patch and our code again: NMI watchdog masking is managed via arch.vapics_in_nmi_mode and by re-checking the per-APIC mask situation in kvm_apic_local_deliver when delivering the NMI. So the patch looks correct - NMIs aren't acked like timer IRQs, the current logic is definitely wrong. Can you elaborate? Why aren't NMIs acked (if delivered via the PIC)? Is the PIC programmed into auto-EOI mode or something? This seems to be complex stuff depending on hardware configurations. I'm not fully understanding though, current state of it is, Yes, PIC is in AEOI mode if linux is using IO-APIC. Um.., kvm says irq == 0 is mp_INT mode in MADT, not mp_ExtINT. So I guess system does automatically INTA cycle (and AEOI because of PIC config), or not connected via 8259A? (like in mpspec figure 5-2.) To checking it, I've tested in check_timer() of linux on the physical machine (irq==0 and mp_INT). The test is something like, if (pin1 != -1) { /* * Ok, does IRQ0 through the IOAPIC work? */ unmask_IO_APIC_irq(0); disable_8259_irq(0); if (timer_irq_works()) { even if I called disable_8259_irq(0), timer was still working via IO-APIC. Would this explain why? Thanks. -- OGAWA Hirofumi hirof...@mail.parknet.co.jp -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
On 04/24/2011 01:50 AM, OGAWA Hirofumi wrote: OGAWA Hirofumihirof...@mail.parknet.co.jp writes: I noticed recently NMI on guest kernel is not working well. host/guest kernel is 2.6.39-rc4, and using vmx. And test code is something like the following: local_irq_disable(); for (i = 0; i 10; i++) { int cpu = get_cpu(); printk(%s: nmi %u, lapic %u\n, __FUNCTION__, nmi_count(cpu), irq_stat[cpu].apic_timer_irqs); mdelay(1000); put_cpu(); } the result is both of nmi and lapic are not increased. If I used -no-kvm-irqchip, it works fine (increase nmi only). So, it seems to be the bug of kvm driver side. With some debug, the cause seems to be in pit_do_work(). With the following patch, NMI watchdog seems to be working correctly (if irq disabled for long time, NMI watchdog can detect it). Is the following patch right? This would cause IRQs to be delivered even if the PIT is masked, no? Are you in fact using the PIT? Linux prefers the HPET, and in my experience the -no-hpet option makes NMIs work. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
Avi Kivity a...@redhat.com writes: With some debug, the cause seems to be in pit_do_work(). With the following patch, NMI watchdog seems to be working correctly (if irq disabled for long time, NMI watchdog can detect it). Is the following patch right? This would cause IRQs to be delivered even if the PIT is masked, no? In here, mask means pit_mask_notifer()? If masked, -pending doesn't prevent to fire it? I'm not sure. Are you in fact using the PIT? Linux prefers the HPET, and in my experience the -no-hpet option makes NMIs work. Yes, this is old kernel (2.6.9), and is not using HPET (CONFIG_HPET_TIMER=n), using IO-APIC as NMI source (PMU seems to be unsupported on kvm for now). Unfortunately, -no-hpet didn't change situation. irq_stat.apic_timer_irqs and nmi_count() are same while irq is disabled. -- OGAWA Hirofumi hirof...@mail.parknet.co.jp -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
On 2011-04-24 11:17, OGAWA Hirofumi wrote: Avi Kivity a...@redhat.com writes: With some debug, the cause seems to be in pit_do_work(). With the following patch, NMI watchdog seems to be working correctly (if irq disabled for long time, NMI watchdog can detect it). Is the following patch right? This would cause IRQs to be delivered even if the PIT is masked, no? In here, mask means pit_mask_notifer()? If masked, -pending doesn't prevent to fire it? I'm not sure. Are you in fact using the PIT? Linux prefers the HPET, and in my experience the -no-hpet option makes NMIs work. Yes, this is old kernel (2.6.9), and is not using HPET (CONFIG_HPET_TIMER=n), using IO-APIC as NMI source (PMU seems to be unsupported on kvm for now). Unfortunately, -no-hpet didn't change situation. irq_stat.apic_timer_irqs and nmi_count() are same while irq is disabled. Maybe it's a regression of latest NMI injection patches. This worked before? Can you check if commits c2dd554dc61173ecb6b3741b680d2ae4c245d2ba and f86368493ec038218e8663cc1b6e5393cd8e008a have any impact on your problem? Jan signature.asc Description: OpenPGP digital signature
Re: nmi is broken?
Jan Kiszka jan.kis...@web.de writes: Yes, this is old kernel (2.6.9), and is not using HPET (CONFIG_HPET_TIMER=n), using IO-APIC as NMI source (PMU seems to be unsupported on kvm for now). Unfortunately, -no-hpet didn't change situation. irq_stat.apic_timer_irqs and nmi_count() are same while irq is disabled. Maybe it's a regression of latest NMI injection patches. This worked before? Can you check if commits c2dd554dc61173ecb6b3741b680d2ae4c245d2ba and f86368493ec038218e8663cc1b6e5393cd8e008a have any impact on your problem? Unfortunately, no. I've checked kvm_inject_nmi() was not called if irq was disabled on guest. The path is following (without my patch, kvm_apic_nmi_wd_deliver is not called), pit_do_work() kvm_apic_nmi_wd_deliver() kvm_apic_local_deliver() __apic_accept_irq() kvm_inject_nmi() so it is not meaningful to change after kvm_inject_nmi() (if I'm understanding commits correctly). Thanks. -- OGAWA Hirofumi hirof...@mail.parknet.co.jp -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
OGAWA Hirofumi hirof...@mail.parknet.co.jp writes: Maybe it's a regression of latest NMI injection patches. This worked before? Can you check if commits c2dd554dc61173ecb6b3741b680d2ae4c245d2ba and f86368493ec038218e8663cc1b6e5393cd8e008a have any impact on your problem? Unfortunately, no. I've checked kvm_inject_nmi() was not called if irq was disabled on guest. To make sure situation, NMI is working if irq is enabled. The path is following (without my patch, kvm_apic_nmi_wd_deliver is not called), pit_do_work() kvm_apic_nmi_wd_deliver() kvm_apic_local_deliver() __apic_accept_irq() kvm_inject_nmi() so it is not meaningful to change after kvm_inject_nmi() (if I'm understanding commits correctly). Thanks. -- OGAWA Hirofumi hirof...@mail.parknet.co.jp -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: nmi is broken?
On 2011-04-24 08:44, Avi Kivity wrote: On 04/24/2011 01:50 AM, OGAWA Hirofumi wrote: OGAWA Hirofumihirof...@mail.parknet.co.jp writes: I noticed recently NMI on guest kernel is not working well. host/guest kernel is 2.6.39-rc4, and using vmx. And test code is something like the following: local_irq_disable(); for (i = 0; i 10; i++) { int cpu = get_cpu(); printk(%s: nmi %u, lapic %u\n, __FUNCTION__, nmi_count(cpu), irq_stat[cpu].apic_timer_irqs); mdelay(1000); put_cpu(); } the result is both of nmi and lapic are not increased. If I used -no-kvm-irqchip, it works fine (increase nmi only). So, it seems to be the bug of kvm driver side. With some debug, the cause seems to be in pit_do_work(). With the following patch, NMI watchdog seems to be working correctly (if irq disabled for long time, NMI watchdog can detect it). Is the following patch right? This would cause IRQs to be delivered even if the PIT is masked, no? I checked the patch and our code again: NMI watchdog masking is managed via arch.vapics_in_nmi_mode and by re-checking the per-APIC mask situation in kvm_apic_local_deliver when delivering the NMI. So the patch looks correct - NMIs aren't acked like timer IRQs, the current logic is definitely wrong. Jan signature.asc Description: OpenPGP digital signature
Re: nmi is broken?
On 2011-04-24 08:44, Avi Kivity wrote: On 04/24/2011 01:50 AM, OGAWA Hirofumi wrote: OGAWA Hirofumihirof...@mail.parknet.co.jp writes: I noticed recently NMI on guest kernel is not working well. host/guest kernel is 2.6.39-rc4, and using vmx. And test code is something like the following: local_irq_disable(); for (i = 0; i 10; i++) { int cpu = get_cpu(); printk(%s: nmi %u, lapic %u\n, __FUNCTION__, nmi_count(cpu), irq_stat[cpu].apic_timer_irqs); mdelay(1000); put_cpu(); } the result is both of nmi and lapic are not increased. If I used -no-kvm-irqchip, it works fine (increase nmi only). So, it seems to be the bug of kvm driver side. With some debug, the cause seems to be in pit_do_work(). With the following patch, NMI watchdog seems to be working correctly (if irq disabled for long time, NMI watchdog can detect it). Is the following patch right? This would cause IRQs to be delivered even if the PIT is masked, no? Are you in fact using the PIT? Linux prefers the HPET, and in my experience the -no-hpet option makes NMIs work. BTW, that's another bug of the in-kernel PIT model: It disables the timer in HPET legacy mode even if we are aware of NMI watchdog receivers. Actually, the whole legacy disabling looks a bit strange in the PIT (mode hackery + flag testing...). While this should be fixed/refactored, adding basic perf support to KVM will be the only option long-term as Linux dropped virtual-wire NMI watchdog support some releases ago. Jan signature.asc Description: OpenPGP digital signature
Re: nmi is broken?
OGAWA Hirofumi hirof...@mail.parknet.co.jp writes: I noticed recently NMI on guest kernel is not working well. host/guest kernel is 2.6.39-rc4, and using vmx. And test code is something like the following: local_irq_disable(); for (i = 0; i 10; i++) { int cpu = get_cpu(); printk(%s: nmi %u, lapic %u\n, __FUNCTION__, nmi_count(cpu), irq_stat[cpu].apic_timer_irqs); mdelay(1000); put_cpu(); } the result is both of nmi and lapic are not increased. If I used -no-kvm-irqchip, it works fine (increase nmi only). So, it seems to be the bug of kvm driver side. With some debug, the cause seems to be in pit_do_work(). With the following patch, NMI watchdog seems to be working correctly (if irq disabled for long time, NMI watchdog can detect it). Is the following patch right? -- OGAWA Hirofumi hirof...@mail.parknet.co.jp [PATCH] kvm: Fix NMI on irq disabled state On irq disabled state, there is no -irq_ask from quest kernel. But NMI shouldn't be affected by it. This fixes inject NMI unconditionally. Signed-off-by: OGAWA Hirofumi hirof...@mail.parknet.co.jp --- arch/x86/kvm/i8254.c | 26 +- 1 file changed, 13 insertions(+), 13 deletions(-) diff -puN arch/x86/kvm/i8254.c~kvm-nmi-fix arch/x86/kvm/i8254.c --- linux-2.6/arch/x86/kvm/i8254.c~kvm-nmi-fix 2011-04-24 07:42:23.0 +0900 +++ linux-2.6-hirofumi/arch/x86/kvm/i8254.c 2011-04-24 07:42:23.0 +0900 @@ -305,20 +305,20 @@ static void pit_do_work(struct work_stru if (inject) { kvm_set_irq(kvm, kvm-arch.vpit-irq_source_id, 0, 1); kvm_set_irq(kvm, kvm-arch.vpit-irq_source_id, 0, 0); - - /* -* Provides NMI watchdog support via Virtual Wire mode. -* The route is: PIT - PIC - LVT0 in NMI mode. -* -* Note: Our Virtual Wire implementation is simplified, only -* propagating PIT interrupts to all VCPUs when they have set -* LVT0 to NMI delivery. Other PIC interrupts are just sent to -* VCPU0, and only if its LVT0 is in EXTINT mode. -*/ - if (kvm-arch.vapics_in_nmi_mode 0) - kvm_for_each_vcpu(i, vcpu, kvm) - kvm_apic_nmi_wd_deliver(vcpu); } + + /* +* Provides NMI watchdog support via Virtual Wire mode. +* The route is: PIT - PIC - LVT0 in NMI mode. +* +* Note: Our Virtual Wire implementation is simplified, only +* propagating PIT interrupts to all VCPUs when they have set +* LVT0 to NMI delivery. Other PIC interrupts are just sent to +* VCPU0, and only if its LVT0 is in EXTINT mode. +*/ + if (kvm-arch.vapics_in_nmi_mode 0) + kvm_for_each_vcpu(i, vcpu, kvm) + kvm_apic_nmi_wd_deliver(vcpu); } static enum hrtimer_restart pit_timer_fn(struct hrtimer *data) _ -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
nmi is broken?
Hi, I noticed recently NMI on guest kernel is not working well. host/guest kernel is 2.6.39-rc4, and using vmx. And test code is something like the following: local_irq_disable(); for (i = 0; i 10; i++) { int cpu = get_cpu(); printk(%s: nmi %u, lapic %u\n, __FUNCTION__, nmi_count(cpu), irq_stat[cpu].apic_timer_irqs); mdelay(1000); put_cpu(); } the result is both of nmi and lapic are not increased. If I used -no-kvm-irqchip, it works fine (increase nmi only). So, it seems to be the bug of kvm driver side. Ideas? -- OGAWA Hirofumi hirof...@mail.parknet.co.jp -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html