Re: high host load from vmx_set_supported_cpuid call?
On Sun, Dec 02, 2012 at 04:26:41PM +0100, Nikola Ciprich wrote: > > hmm, that should be OK, new kvm*.ko modules are part of kernel rpm package, > > there's no old module there. (I checked by both inspecting kernel pkg and > > using modinfo)... Could it be something else? > > hmm, I can reply to myself this time - perf seems to get the symbols using > /proc/kallsyms and there apparently are symbols from inkernel KVM modules, not > the externally built. I guess the problem with my kernel package is I build > both kernel and external KVM modules and then replace kernel ones causing > kallsyms mismatch.. > I'll have to fix this first, reboot host and then see... > And you can successfully install external modules while KVM is compiled in? Strange. -- Gleb. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: high host load from vmx_set_supported_cpuid call?
On Sun, Dec 02, 2012 at 04:10:10PM +0100, Nikola Ciprich wrote: > > I think you need to copy them over old modules in /lib/modules. > > hmm, that should be OK, new kvm*.ko modules are part of kernel rpm package, > there's no old module there. (I checked by both inspecting kernel pkg and > using modinfo)... Could it be something else? > Probably, I do not how to debug perf unfortunately. -- Gleb. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: high host load from vmx_set_supported_cpuid call?
> hmm, that should be OK, new kvm*.ko modules are part of kernel rpm package, > there's no old module there. (I checked by both inspecting kernel pkg and > using modinfo)... Could it be something else? hmm, I can reply to myself this time - perf seems to get the symbols using /proc/kallsyms and there apparently are symbols from inkernel KVM modules, not the externally built. I guess the problem with my kernel package is I build both kernel and external KVM modules and then replace kernel ones causing kallsyms mismatch.. I'll have to fix this first, reboot host and then see... > > > > > > -- > > Gleb. > > > > -- > - > Ing. Nikola CIPRICH > LinuxBox.cz, s.r.o. > 28. rijna 168, 709 00 Ostrava > > tel.: +420 591 166 214 > fax:+420 596 621 273 > mobil: +420 777 093 799 > > www.linuxbox.cz > > mobil servis: +420 737 238 656 > email servis: ser...@linuxbox.cz > - -- - Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax:+420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: ser...@linuxbox.cz - pgpTi86wLYT8l.pgp Description: PGP signature
Re: high host load from vmx_set_supported_cpuid call?
> I think you need to copy them over old modules in /lib/modules. hmm, that should be OK, new kvm*.ko modules are part of kernel rpm package, there's no old module there. (I checked by both inspecting kernel pkg and using modinfo)... Could it be something else? > > -- > Gleb. > -- - Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax:+420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: ser...@linuxbox.cz - pgpnVxHTWTEmQ.pgp Description: PGP signature
Re: high host load from vmx_set_supported_cpuid call?
On Sun, Dec 02, 2012 at 03:51:53PM +0100, Nikola Ciprich wrote: > > More like loaded modules/installed modules mismatch. > I see, the problem is, that I've got kvm-kmod compiled separately! > thus kvm*.ko symboles don't match! > I see that kvm-kmod build produces System.map file, I guess I need to > merge it with kernel's System.map? > I think you need to copy them over old modules in /lib/modules. -- Gleb. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: high host load from vmx_set_supported_cpuid call?
> More like loaded modules/installed modules mismatch. I see, the problem is, that I've got kvm-kmod compiled separately! thus kvm*.ko symboles don't match! I see that kvm-kmod build produces System.map file, I guess I need to merge it with kernel's System.map? -- - Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax:+420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: ser...@linuxbox.cz - pgpWo5au4KaYi.pgp Description: PGP signature
Re: high host load from vmx_set_supported_cpuid call?
On Sun, Dec 02, 2012 at 03:31:08PM +0100, Nikola Ciprich wrote: > Hi Gleb, > > > Something wrong with your symbols. This function cannot take that much. > > It is three and a half instruction long and should be called only once > > during vm startup. > > well, it didn't make any sense to me, glad I wasn't that wrong :) > how could that be? I guess it could be perf/kernel mismatch right? > I'll try to fix that and see if it helps.. > More like loaded modules/installed modules mismatch. -- Gleb. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: high host load from vmx_set_supported_cpuid call?
Hi Gleb, > Something wrong with your symbols. This function cannot take that much. > It is three and a half instruction long and should be called only once > during vm startup. well, it didn't make any sense to me, glad I wasn't that wrong :) how could that be? I guess it could be perf/kernel mismatch right? I'll try to fix that and see if it helps.. thanks for Your time! n. -- - Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax:+420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: ser...@linuxbox.cz - pgpMhzIV2En3z.pgp Description: PGP signature
Re: high host load from vmx_set_supported_cpuid call?
On Sun, Dec 02, 2012 at 12:41:37PM +0100, Nikola Ciprich wrote: > Hi, > > while trying to find source of KVM guest slowness, I noticed following in > perf top: > > 326.00 19.6% vmx_set_supported_cpuid > /lib/modules/3.0.53lb6.02/kernel/arch/x86/kvm/kvm-intel.ko Something wrong with your symbols. This function cannot take that much. It is three and a half instruction long and should be called only once during vm startup. > 108.00 6.5% kvm_arch_dev_ioctl > /lib/modules/3.0.53lb6.02/kernel/arch/x86/kvm/kvm.ko > 100.00 6.0% tick_dev_program_event > [kernel.kallsyms] >78.00 4.7% __remove_hrtimer > [kernel.kallsyms] >52.00 3.1% acpi_processor_reevaluate_tstate > /lib/modules/3.0.53lb6.02/kernel/drivers/acpi/processor.ko >45.00 2.7% find_busiest_group > [kernel.kallsyms] >43.00 2.6% do_raw_spin_lock > [kernel.kallsyms] > > my question is, is this normal? I tried googling for vmx_set_supported_cpuid > but am not > really clever about it... > > here's snippet from trace-cmd: > > version = 6 > CPU 11 is empty > cpus=12 > qemu-kvm-7766 [000] 235066.551604: kvm_msi_set_irq: dst 1 vec > 51 (LowPrio|logical|edge|rh) > qemu-kvm-7766 [000] 235066.551606: kvm_apic_accept_irq: apicid 0 > vec 81 (LowPrio|edge) > qemu-kvm-7767 [001] 235066.551618: kvm_inj_virq: irq 81 > qemu-kvm-7767 [001] 235066.551620: kvm_entry:vcpu 0 > qemu-kvm-7767 [001] 235066.551625: kvm_exit: reason > EPT_MISCONFIG rip 0x81023d96 info 0 0 > qemu-kvm-7767 [001] 235066.551630: kvm_emulate_insn: [FAILED TO > PARSE] rip=18446744071578992022 csbase=0 len=2 insn=ARRAY[8b, 00, c9, 89, c0, > c3, > 0f, 1f, 40, 00, 55, 48, 89, e5, 0f] flags=9 failed=0 > qemu-kvm-7767 [001] 235066.551631: vcpu_match_mmio: gva > 0xc90040f0 gpa 0xfed000f0 Read GPA > qemu-kvm-7767 [001] 235066.551632: kvm_mmio: mmio > unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 > qemu-kvm-7767 [001] 235066.551634: kvm_userspace_exit: reason > KVM_EXIT_MMIO (6) > qemu-kvm-7767 [001] 235066.551648: kvm_mmio: mmio read > len 4 gpa 0xfed000f0 val 0xfed25f11 > qemu-kvm-7767 [001] 235066.551649: kvm_entry:vcpu 0 > qemu-kvm-7767 [001] 235066.551650: kvm_exit: reason > EPT_MISCONFIG rip 0x81023d96 info 0 0 > qemu-kvm-7767 [001] 235066.551652: kvm_emulate_insn: [FAILED TO > PARSE] rip=18446744071578992022 csbase=0 len=2 insn=ARRAY[8b, 00, c9, 89, c0, > c3, > 0f, 1f, 40, 00, 55, 48, 89, e5, 0f] flags=9 failed=0 > qemu-kvm-7767 [001] 235066.551652: vcpu_match_mmio: gva > 0xc90040f0 gpa 0xfed000f0 Read GPA > qemu-kvm-7767 [001] 235066.551653: kvm_mmio: mmio > unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 > qemu-kvm-7767 [001] 235066.551653: kvm_userspace_exit: reason > KVM_EXIT_MMIO (6) > qemu-kvm-7767 [001] 235066.551656: kvm_mmio: mmio read > len 4 gpa 0xfed000f0 val 0xfed26450 > qemu-kvm-7767 [001] 235066.551657: kvm_entry:vcpu 0 > qemu-kvm-7767 [001] 235066.551659: kvm_exit: reason > EPT_MISCONFIG rip 0x81023d96 info 0 0 > qemu-kvm-7767 [001] 235066.551660: kvm_emulate_insn: [FAILED TO > PARSE] rip=18446744071578992022 csbase=0 len=2 insn=ARRAY[8b, 00, c9, 89, c0, > c3, > 0f, 1f, 40, 00, 55, 48, 89, e5, 0f] flags=9 failed=0 > qemu-kvm-7767 [001] 235066.551660: vcpu_match_mmio: gva > 0xc90040f0 gpa 0xfed000f0 Read GPA > qemu-kvm-7767 [001] 235066.551660: kvm_mmio: mmio > unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 > qemu-kvm-7767 [001] 235066.551661: kvm_userspace_exit: reason > KVM_EXIT_MMIO (6) > qemu-kvm-7767 [001] 235066.551663: kvm_mmio: mmio read > len 4 gpa 0xfed000f0 val 0xfed2672b > qemu-kvm-7767 [001] 235066.551664: kvm_entry:vcpu 0 > qemu-kvm-7767 [001] 235066.551667: kvm_exit: reason > APIC_ACCESS rip 0x810219eb info 10b0 0 > qemu-kvm-7767 [001] 235066.551667: kvm_apic: apic_write > APIC_EOI = 0x0 > qemu-kvm-7767 [001] 235066.551668: kvm_eoi: apicid 0 > vector 81 > qemu-kvm-7767 [001] 235066.551668: kvm_entry:vcpu 0 > qemu-kvm-7767 [001] 235066.551681: kvm_exit: reason > EPT_MISCONFIG rip 0
high host load from vmx_set_supported_cpuid call?
Hi, while trying to find source of KVM guest slowness, I noticed following in perf top: 326.00 19.6% vmx_set_supported_cpuid /lib/modules/3.0.53lb6.02/kernel/arch/x86/kvm/kvm-intel.ko 108.00 6.5% kvm_arch_dev_ioctl /lib/modules/3.0.53lb6.02/kernel/arch/x86/kvm/kvm.ko 100.00 6.0% tick_dev_program_event [kernel.kallsyms] 78.00 4.7% __remove_hrtimer [kernel.kallsyms] 52.00 3.1% acpi_processor_reevaluate_tstate /lib/modules/3.0.53lb6.02/kernel/drivers/acpi/processor.ko 45.00 2.7% find_busiest_group [kernel.kallsyms] 43.00 2.6% do_raw_spin_lock [kernel.kallsyms] my question is, is this normal? I tried googling for vmx_set_supported_cpuid but am not really clever about it... here's snippet from trace-cmd: version = 6 CPU 11 is empty cpus=12 qemu-kvm-7766 [000] 235066.551604: kvm_msi_set_irq: dst 1 vec 51 (LowPrio|logical|edge|rh) qemu-kvm-7766 [000] 235066.551606: kvm_apic_accept_irq: apicid 0 vec 81 (LowPrio|edge) qemu-kvm-7767 [001] 235066.551618: kvm_inj_virq: irq 81 qemu-kvm-7767 [001] 235066.551620: kvm_entry:vcpu 0 qemu-kvm-7767 [001] 235066.551625: kvm_exit: reason EPT_MISCONFIG rip 0x81023d96 info 0 0 qemu-kvm-7767 [001] 235066.551630: kvm_emulate_insn: [FAILED TO PARSE] rip=18446744071578992022 csbase=0 len=2 insn=ARRAY[8b, 00, c9, 89, c0, c3, 0f, 1f, 40, 00, 55, 48, 89, e5, 0f] flags=9 failed=0 qemu-kvm-7767 [001] 235066.551631: vcpu_match_mmio: gva 0xc90040f0 gpa 0xfed000f0 Read GPA qemu-kvm-7767 [001] 235066.551632: kvm_mmio: mmio unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 qemu-kvm-7767 [001] 235066.551634: kvm_userspace_exit: reason KVM_EXIT_MMIO (6) qemu-kvm-7767 [001] 235066.551648: kvm_mmio: mmio read len 4 gpa 0xfed000f0 val 0xfed25f11 qemu-kvm-7767 [001] 235066.551649: kvm_entry:vcpu 0 qemu-kvm-7767 [001] 235066.551650: kvm_exit: reason EPT_MISCONFIG rip 0x81023d96 info 0 0 qemu-kvm-7767 [001] 235066.551652: kvm_emulate_insn: [FAILED TO PARSE] rip=18446744071578992022 csbase=0 len=2 insn=ARRAY[8b, 00, c9, 89, c0, c3, 0f, 1f, 40, 00, 55, 48, 89, e5, 0f] flags=9 failed=0 qemu-kvm-7767 [001] 235066.551652: vcpu_match_mmio: gva 0xc90040f0 gpa 0xfed000f0 Read GPA qemu-kvm-7767 [001] 235066.551653: kvm_mmio: mmio unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 qemu-kvm-7767 [001] 235066.551653: kvm_userspace_exit: reason KVM_EXIT_MMIO (6) qemu-kvm-7767 [001] 235066.551656: kvm_mmio: mmio read len 4 gpa 0xfed000f0 val 0xfed26450 qemu-kvm-7767 [001] 235066.551657: kvm_entry:vcpu 0 qemu-kvm-7767 [001] 235066.551659: kvm_exit: reason EPT_MISCONFIG rip 0x81023d96 info 0 0 qemu-kvm-7767 [001] 235066.551660: kvm_emulate_insn: [FAILED TO PARSE] rip=18446744071578992022 csbase=0 len=2 insn=ARRAY[8b, 00, c9, 89, c0, c3, 0f, 1f, 40, 00, 55, 48, 89, e5, 0f] flags=9 failed=0 qemu-kvm-7767 [001] 235066.551660: vcpu_match_mmio: gva 0xc90040f0 gpa 0xfed000f0 Read GPA qemu-kvm-7767 [001] 235066.551660: kvm_mmio: mmio unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 qemu-kvm-7767 [001] 235066.551661: kvm_userspace_exit: reason KVM_EXIT_MMIO (6) qemu-kvm-7767 [001] 235066.551663: kvm_mmio: mmio read len 4 gpa 0xfed000f0 val 0xfed2672b qemu-kvm-7767 [001] 235066.551664: kvm_entry:vcpu 0 qemu-kvm-7767 [001] 235066.551667: kvm_exit: reason APIC_ACCESS rip 0x810219eb info 10b0 0 qemu-kvm-7767 [001] 235066.551667: kvm_apic: apic_write APIC_EOI = 0x0 qemu-kvm-7767 [001] 235066.551668: kvm_eoi: apicid 0 vector 81 qemu-kvm-7767 [001] 235066.551668: kvm_entry:vcpu 0 qemu-kvm-7767 [001] 235066.551681: kvm_exit: reason EPT_MISCONFIG rip 0x81023d96 info 0 0 qemu-kvm-7767 [001] 235066.551682: kvm_emulate_insn: [FAILED TO PARSE] rip=18446744071578992022 csbase=0 len=2 insn=ARRAY[8b, 00, c9, 89, c0, c3, 0f, 1f, 40, 00, 55, 48, 89, e5, 0f] flags=9 failed=0 qemu-kvm-7767 [001] 235066.551683: vcpu_match_mmio: gva 0xc90040f0 gpa 0xfed000f0 Read GPA qemu-kvm-7767 [001] 235066.551683: kvm_mm
