Re: 2.6.32 guest with paravirt clock enabled hangs on 2.6.37.6 host (w qemu-kvm-0.13.0)
On 05/08/2011 12:06 PM, Nikola Ciprich wrote: OK, I see.. the problem is, that I'm trying to hunt down bug causing hangs when 2.6.32 guests try to run tcpdump - this seems to be reproducible even on latest 2.6.32.x, and seems like it depends on kvm-clock.. So I was thinking about bisecting between 2.6.32 and latest git which doesn't seem to suffer this problem but hitting another (different) problem in 2.6.32 complicates thinks a bit :( If somebody would have some hint on how to proceed, I'd be more then grateful.. cheers n. What are you bisecting, the host kernel or the guest kernel, and what version is the host kernel? -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 2.6.32 guest with paravirt clock enabled hangs on 2.6.37.6 host (w qemu-kvm-0.13.0)
The guest, because latest kernels do not suffer this problem, so I'd like to find fix so it can be pushed to -stable (we're using 2.6.32.x) host is currently 2.6.37 (and i'm currently testing 2.6.38 as well) n. On Mon, May 09, 2011 at 10:32:26AM -0700, Zachary Amsden wrote: On 05/08/2011 12:06 PM, Nikola Ciprich wrote: OK, I see.. the problem is, that I'm trying to hunt down bug causing hangs when 2.6.32 guests try to run tcpdump - this seems to be reproducible even on latest 2.6.32.x, and seems like it depends on kvm-clock.. So I was thinking about bisecting between 2.6.32 and latest git which doesn't seem to suffer this problem but hitting another (different) problem in 2.6.32 complicates thinks a bit :( If somebody would have some hint on how to proceed, I'd be more then grateful.. cheers n. What are you bisecting, the host kernel or the guest kernel, and what version is the host kernel? -- - Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax:+420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: ser...@linuxbox.cz - pgpbiOBiVY8Rf.pgp Description: PGP signature
Re: 2.6.32 guest with paravirt clock enabled hangs on 2.6.37.6 host (w qemu-kvm-0.13.0)
On 05/09/2011 11:25 AM, Nikola Ciprich wrote: The guest, because latest kernels do not suffer this problem, so I'd like to find fix so it can be pushed to -stable (we're using 2.6.32.x) host is currently 2.6.37 (and i'm currently testing 2.6.38 as well) n. That's a pretty wide range to be bisecting, and I think we know for a fact there were some kvmclock related bugs in that range. If you are looking for something causing problems with tcpdump, I'd suggest getting rid of kvmclock in your testing and using TSC instead; if you're looking to verify that kvmclock related changed have been backported to -stable, rather than bisect and run into bugs, it would probably be faster to check the commit logs for arch/x86/kvm/x86.c and make sure you're not missing anything from me or Glauber that has been applied to the most recent branch. Zach -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 2.6.32 guest with paravirt clock enabled hangs on 2.6.37.6 host (w qemu-kvm-0.13.0)
That's a pretty wide range to be bisecting, and I think we know for a fact there were some kvmclock related bugs in that range. thats true, I might try to pick those that seem related and see if it helpts.. If you are looking for something causing problems with tcpdump, I'd suggest getting rid of kvmclock in your testing and using TSC instead; that's the problem, I can't reproduce the problems without kvm-clock enabled, so it must be related to it somehow.. if you're looking to verify that kvmclock related changed have been backported to -stable, rather than bisect and run into bugs, it would probably be faster to check the commit logs for arch/x86/kvm/x86.c and make sure you're not missing anything from me or Glauber that has been applied to the most recent branch. yup, I'll try and report... thanks for the hints! n. Zach -- - Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax:+420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: ser...@linuxbox.cz - pgpYcvv8ncavM.pgp Description: PGP signature
2.6.32 guest with paravirt clock enabled hangs on 2.6.37.6 host (w qemu-kvm-0.13.0)
Hello everyboy, while installing new virt machine today, I noticed that 2.6.32 x86_64 SMP guests are hanging if they have paravirt-clock enabled... Either they don't finish booting at all, or boot but hang soon after.. Such a hanged guest fully loads all host cpus.. The host is 6core x86_64 runnig 2.6.37.6 with 24GB RAM. kvm_stat: kvm_exit(EXTERNAL_INTERRUPT) 156821643 kvm_exit 155141643 kvm_entry154161643 kvm_set_irq 1 0 kvm_msi_set_irq 1 0 kvm_apic_accept_irq 1 0 kvm_exit(VMCLEAR)6 0 kvm_exit(VMON) 6 0 kvm_exit(PAUSE_INSTRUCTION) 5 0 kvm_exit(MCE_DURING_VMENTRY) 5 0 kvm_exit(MWAIT_INSTRUCTION) 5 0 kvm_exit(DR_ACCESS) 5 0 kvm_exit(EPT_VIOLATION) 5 0 kvm_exit(NMI_WINDOW) 5 0 kvm_exit(VMPTRLD)5 0 kvm_exit(TASK_SWITCH)5 0 kvm_exit(VMREAD) 5 0 kvm_exit(VMLAUNCH) 5 0 kvm_exit(RDPMC) 5 0 perf top: 16.00 10.9% add_preempt_count[kernel.kallsyms] 16.00 10.9% do_raw_spin_lock [kernel.kallsyms] 15.00 10.2% sub_preempt_count[kernel.kallsyms] 8.00 5.4% irq_exit [kernel.kallsyms] 7.00 4.8% vmx_vcpu_run /lib/modules/2.6.37lb.09/kernel/arch/x86/kvm/kvm-intel.ko 7.00 4.8% page_fault [kernel.kallsyms] 5.00 3.4% mempool_free [kernel.kallsyms] info registers: RAX=00f42400 RBX=81533f00 RCX=0016 RDX=00077358f500 RSI=1dcd6500 RDI=0001 RBP=880009a03ee8 RSP=880009a03ee8 R8 =0016 R9 =000a R10= R11= R12=2a4d17d38f3303c1 R13=815fd000 R14=81592140 R15=00093510 RIP=810767cb RFL=0006 [-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0018 00c09300 DPL=0 DS [-WA] CS =0010 00a09b00 DPL=0 CS64 [-RA] SS =0018 00c09300 DPL=0 DS [-WA] DS =0018 00c09300 DPL=0 DS [-WA] FS = 000f GS = 880009a0 000f LDT= 000f TR =0040 880009a11880 2087 8b00 DPL=0 TSS64-busy GDT= 880009a04000 007f IDT= 815fd000 0fff CR0=8005003b CR2=7f424e540700 CR3=00021690c000 CR4=06f0 DR0= DR1= DR2= DR3= DR6=0ff0 DR7=0400 EFER=0d01 FCW=037f FSW= [ST=0] FTW=00 MXCSR=1f80 FPR0= FPR1= FPR2= FPR3= FPR4= FPR5= FPR6= FPR7= XMM00= XMM01=3c23d70a XMM02= XMM03= XMM04= XMM05= XMM06= XMM07= XMM08= XMM09= XMM10= XMM11= XMM12= XMM13= XMM14= XMM15= info cpus: * CPU #0: pc=0x8105d4a0 thread_id=19639 CPU #1: pc=0x81013140 thread_id=19640 CPU #2: pc=0x8102a1b6 (halted) thread_id=19641 CPU #3: pc=0x81341521 thread_id=19642 CPU #4: pc=0x810415d8 thread_id=19643 CPU #5: pc=0x811ca521 thread_id=19644 CPU #6: pc=0x81013140 thread_id=19646 CPU #7: pc=0x8102a1b6 (halted) thread_id=19647 and here are trace-cmds for all cpus: http://nik.lbox.cz/public/trace-cmd.tar.bz2 Could somebody please have a look at this? I also tried 2.6.38.5, but the result is the same... cheers nik -- - Ing. Nikola CIPRICH
Re: 2.6.32 guest with paravirt clock enabled hangs on 2.6.37.6 host (w qemu-kvm-0.13.0)
08.05.2011 22:33, Nikola Ciprich wrote: Hello everyboy, while installing new virt machine today, I noticed that 2.6.32 x86_64 SMP guests are hanging if they have paravirt-clock enabled... There were about 10 bugfixes pushed to 2.6.32.y stable series, some of them were for kvm-clock, and some were for problems which manifested itself like you described. You may actually take a look which guests you're booting. FWIW. /mjt -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 2.6.32 guest with paravirt clock enabled hangs on 2.6.37.6 host (w qemu-kvm-0.13.0)
OK, I see.. the problem is, that I'm trying to hunt down bug causing hangs when 2.6.32 guests try to run tcpdump - this seems to be reproducible even on latest 2.6.32.x, and seems like it depends on kvm-clock.. So I was thinking about bisecting between 2.6.32 and latest git which doesn't seem to suffer this problem but hitting another (different) problem in 2.6.32 complicates thinks a bit :( If somebody would have some hint on how to proceed, I'd be more then grateful.. cheers n. On Sun, May 08, 2011 at 10:53:56PM +0400, Michael Tokarev wrote: 08.05.2011 22:33, Nikola Ciprich wrote: Hello everyboy, while installing new virt machine today, I noticed that 2.6.32 x86_64 SMP guests are hanging if they have paravirt-clock enabled... There were about 10 bugfixes pushed to 2.6.32.y stable series, some of them were for kvm-clock, and some were for problems which manifested itself like you described. You may actually take a look which guests you're booting. FWIW. /mjt -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- - Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax:+420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: ser...@linuxbox.cz - pgpza8Cr1Af9M.pgp Description: PGP signature
Re: 2.6.32 guest with paravirt clock enabled hangs on 2.6.37.6 host (w qemu-kvm-0.13.0)
(CC Zachary) well, I should also note that while testing 2.6.37 host, I had Zach's patch fixing guest clock regression applied... n. On Sun, May 08, 2011 at 08:33:04PM +0200, Nikola Ciprich wrote: Hello everyboy, while installing new virt machine today, I noticed that 2.6.32 x86_64 SMP guests are hanging if they have paravirt-clock enabled... Either they don't finish booting at all, or boot but hang soon after.. Such a hanged guest fully loads all host cpus.. The host is 6core x86_64 runnig 2.6.37.6 with 24GB RAM. kvm_stat: kvm_exit(EXTERNAL_INTERRUPT) 156821643 kvm_exit 155141643 kvm_entry154161643 kvm_set_irq 1 0 kvm_msi_set_irq 1 0 kvm_apic_accept_irq 1 0 kvm_exit(VMCLEAR)6 0 kvm_exit(VMON) 6 0 kvm_exit(PAUSE_INSTRUCTION) 5 0 kvm_exit(MCE_DURING_VMENTRY) 5 0 kvm_exit(MWAIT_INSTRUCTION) 5 0 kvm_exit(DR_ACCESS) 5 0 kvm_exit(EPT_VIOLATION) 5 0 kvm_exit(NMI_WINDOW) 5 0 kvm_exit(VMPTRLD)5 0 kvm_exit(TASK_SWITCH)5 0 kvm_exit(VMREAD) 5 0 kvm_exit(VMLAUNCH) 5 0 kvm_exit(RDPMC) 5 0 perf top: 16.00 10.9% add_preempt_count[kernel.kallsyms] 16.00 10.9% do_raw_spin_lock [kernel.kallsyms] 15.00 10.2% sub_preempt_count[kernel.kallsyms] 8.00 5.4% irq_exit [kernel.kallsyms] 7.00 4.8% vmx_vcpu_run /lib/modules/2.6.37lb.09/kernel/arch/x86/kvm/kvm-intel.ko 7.00 4.8% page_fault [kernel.kallsyms] 5.00 3.4% mempool_free [kernel.kallsyms] info registers: RAX=00f42400 RBX=81533f00 RCX=0016 RDX=00077358f500 RSI=1dcd6500 RDI=0001 RBP=880009a03ee8 RSP=880009a03ee8 R8 =0016 R9 =000a R10= R11= R12=2a4d17d38f3303c1 R13=815fd000 R14=81592140 R15=00093510 RIP=810767cb RFL=0006 [-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0018 00c09300 DPL=0 DS [-WA] CS =0010 00a09b00 DPL=0 CS64 [-RA] SS =0018 00c09300 DPL=0 DS [-WA] DS =0018 00c09300 DPL=0 DS [-WA] FS = 000f GS = 880009a0 000f LDT= 000f TR =0040 880009a11880 2087 8b00 DPL=0 TSS64-busy GDT= 880009a04000 007f IDT= 815fd000 0fff CR0=8005003b CR2=7f424e540700 CR3=00021690c000 CR4=06f0 DR0= DR1= DR2= DR3= DR6=0ff0 DR7=0400 EFER=0d01 FCW=037f FSW= [ST=0] FTW=00 MXCSR=1f80 FPR0= FPR1= FPR2= FPR3= FPR4= FPR5= FPR6= FPR7= XMM00= XMM01=3c23d70a XMM02= XMM03= XMM04= XMM05= XMM06= XMM07= XMM08= XMM09= XMM10= XMM11= XMM12= XMM13= XMM14= XMM15= info cpus: * CPU #0: pc=0x8105d4a0 thread_id=19639 CPU #1: pc=0x81013140 thread_id=19640 CPU #2: pc=0x8102a1b6 (halted) thread_id=19641 CPU #3: pc=0x81341521 thread_id=19642 CPU #4: pc=0x810415d8 thread_id=19643 CPU #5: pc=0x811ca521 thread_id=19644 CPU #6: pc=0x81013140 thread_id=19646 CPU #7: pc=0x8102a1b6 (halted)
Re: 2.6.32 guest with paravirt clock enabled hangs on 2.6.37.6 host (w qemu-kvm-0.13.0)
On 05/08/11 13:06, Nikola Ciprich wrote: OK, I see.. the problem is, that I'm trying to hunt down bug causing hangs when 2.6.32 guests try to run tcpdump - this seems to be reproducible even on latest 2.6.32.x, and seems like it depends on kvm-clock.. So I was thinking about bisecting between 2.6.32 and latest git which doesn't seem to suffer this problem but hitting another (different) problem in 2.6.32 complicates thinks a bit :( If somebody would have some hint on how to proceed, I'd be more then grateful.. cheers Have you tried enabling gdbserver in the qemu monitor and then attaching gdb to the guest once it hangs? David -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
