high host load from vmx_set_supported_cpuid call?
Hi, while trying to find source of KVM guest slowness, I noticed following in perf top: 326.00 19.6% vmx_set_supported_cpuid /lib/modules/3.0.53lb6.02/kernel/arch/x86/kvm/kvm-intel.ko 108.00 6.5% kvm_arch_dev_ioctl /lib/modules/3.0.53lb6.02/kernel/arch/x86/kvm/kvm.ko 100.00 6.0% tick_dev_program_event [kernel.kallsyms] 78.00 4.7% __remove_hrtimer [kernel.kallsyms] 52.00 3.1% acpi_processor_reevaluate_tstate /lib/modules/3.0.53lb6.02/kernel/drivers/acpi/processor.ko 45.00 2.7% find_busiest_group [kernel.kallsyms] 43.00 2.6% do_raw_spin_lock [kernel.kallsyms] my question is, is this normal? I tried googling for vmx_set_supported_cpuid but am not really clever about it... here's snippet from trace-cmd: version = 6 CPU 11 is empty cpus=12 qemu-kvm-7766 [000] 235066.551604: kvm_msi_set_irq: dst 1 vec 51 (LowPrio|logical|edge|rh) qemu-kvm-7766 [000] 235066.551606: kvm_apic_accept_irq: apicid 0 vec 81 (LowPrio|edge) qemu-kvm-7767 [001] 235066.551618: kvm_inj_virq: irq 81 qemu-kvm-7767 [001] 235066.551620: kvm_entry:vcpu 0 qemu-kvm-7767 [001] 235066.551625: kvm_exit: reason EPT_MISCONFIG rip 0x81023d96 info 0 0 qemu-kvm-7767 [001] 235066.551630: kvm_emulate_insn: [FAILED TO PARSE] rip=18446744071578992022 csbase=0 len=2 insn=ARRAY[8b, 00, c9, 89, c0, c3, 0f, 1f, 40, 00, 55, 48, 89, e5, 0f] flags=9 failed=0 qemu-kvm-7767 [001] 235066.551631: vcpu_match_mmio: gva 0xc90040f0 gpa 0xfed000f0 Read GPA qemu-kvm-7767 [001] 235066.551632: kvm_mmio: mmio unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 qemu-kvm-7767 [001] 235066.551634: kvm_userspace_exit: reason KVM_EXIT_MMIO (6) qemu-kvm-7767 [001] 235066.551648: kvm_mmio: mmio read len 4 gpa 0xfed000f0 val 0xfed25f11 qemu-kvm-7767 [001] 235066.551649: kvm_entry:vcpu 0 qemu-kvm-7767 [001] 235066.551650: kvm_exit: reason EPT_MISCONFIG rip 0x81023d96 info 0 0 qemu-kvm-7767 [001] 235066.551652: kvm_emulate_insn: [FAILED TO PARSE] rip=18446744071578992022 csbase=0 len=2 insn=ARRAY[8b, 00, c9, 89, c0, c3, 0f, 1f, 40, 00, 55, 48, 89, e5, 0f] flags=9 failed=0 qemu-kvm-7767 [001] 235066.551652: vcpu_match_mmio: gva 0xc90040f0 gpa 0xfed000f0 Read GPA qemu-kvm-7767 [001] 235066.551653: kvm_mmio: mmio unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 qemu-kvm-7767 [001] 235066.551653: kvm_userspace_exit: reason KVM_EXIT_MMIO (6) qemu-kvm-7767 [001] 235066.551656: kvm_mmio: mmio read len 4 gpa 0xfed000f0 val 0xfed26450 qemu-kvm-7767 [001] 235066.551657: kvm_entry:vcpu 0 qemu-kvm-7767 [001] 235066.551659: kvm_exit: reason EPT_MISCONFIG rip 0x81023d96 info 0 0 qemu-kvm-7767 [001] 235066.551660: kvm_emulate_insn: [FAILED TO PARSE] rip=18446744071578992022 csbase=0 len=2 insn=ARRAY[8b, 00, c9, 89, c0, c3, 0f, 1f, 40, 00, 55, 48, 89, e5, 0f] flags=9 failed=0 qemu-kvm-7767 [001] 235066.551660: vcpu_match_mmio: gva 0xc90040f0 gpa 0xfed000f0 Read GPA qemu-kvm-7767 [001] 235066.551660: kvm_mmio: mmio unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 qemu-kvm-7767 [001] 235066.551661: kvm_userspace_exit: reason KVM_EXIT_MMIO (6) qemu-kvm-7767 [001] 235066.551663: kvm_mmio: mmio read len 4 gpa 0xfed000f0 val 0xfed2672b qemu-kvm-7767 [001] 235066.551664: kvm_entry:vcpu 0 qemu-kvm-7767 [001] 235066.551667: kvm_exit: reason APIC_ACCESS rip 0x810219eb info 10b0 0 qemu-kvm-7767 [001] 235066.551667: kvm_apic: apic_write APIC_EOI = 0x0 qemu-kvm-7767 [001] 235066.551668: kvm_eoi: apicid 0 vector 81 qemu-kvm-7767 [001] 235066.551668: kvm_entry:vcpu 0 qemu-kvm-7767 [001] 235066.551681: kvm_exit: reason EPT_MISCONFIG rip 0x81023d96 info 0 0 qemu-kvm-7767 [001] 235066.551682: kvm_emulate_insn: [FAILED TO PARSE] rip=18446744071578992022 csbase=0 len=2 insn=ARRAY[8b, 00, c9, 89, c0, c3, 0f, 1f, 40, 00, 55, 48, 89, e5, 0f] flags=9 failed=0 qemu-kvm-7767 [001] 235066.551683: vcpu_match_mmio: gva 0xc90040f0 gpa 0xfed000f0 Read GPA qemu-kvm-7767 [001] 235066.551683:
Re: high host load from vmx_set_supported_cpuid call?
On Sun, Dec 02, 2012 at 12:41:37PM +0100, Nikola Ciprich wrote: Hi, while trying to find source of KVM guest slowness, I noticed following in perf top: 326.00 19.6% vmx_set_supported_cpuid /lib/modules/3.0.53lb6.02/kernel/arch/x86/kvm/kvm-intel.ko Something wrong with your symbols. This function cannot take that much. It is three and a half instruction long and should be called only once during vm startup. 108.00 6.5% kvm_arch_dev_ioctl /lib/modules/3.0.53lb6.02/kernel/arch/x86/kvm/kvm.ko 100.00 6.0% tick_dev_program_event [kernel.kallsyms] 78.00 4.7% __remove_hrtimer [kernel.kallsyms] 52.00 3.1% acpi_processor_reevaluate_tstate /lib/modules/3.0.53lb6.02/kernel/drivers/acpi/processor.ko 45.00 2.7% find_busiest_group [kernel.kallsyms] 43.00 2.6% do_raw_spin_lock [kernel.kallsyms] my question is, is this normal? I tried googling for vmx_set_supported_cpuid but am not really clever about it... here's snippet from trace-cmd: version = 6 CPU 11 is empty cpus=12 qemu-kvm-7766 [000] 235066.551604: kvm_msi_set_irq: dst 1 vec 51 (LowPrio|logical|edge|rh) qemu-kvm-7766 [000] 235066.551606: kvm_apic_accept_irq: apicid 0 vec 81 (LowPrio|edge) qemu-kvm-7767 [001] 235066.551618: kvm_inj_virq: irq 81 qemu-kvm-7767 [001] 235066.551620: kvm_entry:vcpu 0 qemu-kvm-7767 [001] 235066.551625: kvm_exit: reason EPT_MISCONFIG rip 0x81023d96 info 0 0 qemu-kvm-7767 [001] 235066.551630: kvm_emulate_insn: [FAILED TO PARSE] rip=18446744071578992022 csbase=0 len=2 insn=ARRAY[8b, 00, c9, 89, c0, c3, 0f, 1f, 40, 00, 55, 48, 89, e5, 0f] flags=9 failed=0 qemu-kvm-7767 [001] 235066.551631: vcpu_match_mmio: gva 0xc90040f0 gpa 0xfed000f0 Read GPA qemu-kvm-7767 [001] 235066.551632: kvm_mmio: mmio unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 qemu-kvm-7767 [001] 235066.551634: kvm_userspace_exit: reason KVM_EXIT_MMIO (6) qemu-kvm-7767 [001] 235066.551648: kvm_mmio: mmio read len 4 gpa 0xfed000f0 val 0xfed25f11 qemu-kvm-7767 [001] 235066.551649: kvm_entry:vcpu 0 qemu-kvm-7767 [001] 235066.551650: kvm_exit: reason EPT_MISCONFIG rip 0x81023d96 info 0 0 qemu-kvm-7767 [001] 235066.551652: kvm_emulate_insn: [FAILED TO PARSE] rip=18446744071578992022 csbase=0 len=2 insn=ARRAY[8b, 00, c9, 89, c0, c3, 0f, 1f, 40, 00, 55, 48, 89, e5, 0f] flags=9 failed=0 qemu-kvm-7767 [001] 235066.551652: vcpu_match_mmio: gva 0xc90040f0 gpa 0xfed000f0 Read GPA qemu-kvm-7767 [001] 235066.551653: kvm_mmio: mmio unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 qemu-kvm-7767 [001] 235066.551653: kvm_userspace_exit: reason KVM_EXIT_MMIO (6) qemu-kvm-7767 [001] 235066.551656: kvm_mmio: mmio read len 4 gpa 0xfed000f0 val 0xfed26450 qemu-kvm-7767 [001] 235066.551657: kvm_entry:vcpu 0 qemu-kvm-7767 [001] 235066.551659: kvm_exit: reason EPT_MISCONFIG rip 0x81023d96 info 0 0 qemu-kvm-7767 [001] 235066.551660: kvm_emulate_insn: [FAILED TO PARSE] rip=18446744071578992022 csbase=0 len=2 insn=ARRAY[8b, 00, c9, 89, c0, c3, 0f, 1f, 40, 00, 55, 48, 89, e5, 0f] flags=9 failed=0 qemu-kvm-7767 [001] 235066.551660: vcpu_match_mmio: gva 0xc90040f0 gpa 0xfed000f0 Read GPA qemu-kvm-7767 [001] 235066.551660: kvm_mmio: mmio unsatisfied-read len 4 gpa 0xfed000f0 val 0x0 qemu-kvm-7767 [001] 235066.551661: kvm_userspace_exit: reason KVM_EXIT_MMIO (6) qemu-kvm-7767 [001] 235066.551663: kvm_mmio: mmio read len 4 gpa 0xfed000f0 val 0xfed2672b qemu-kvm-7767 [001] 235066.551664: kvm_entry:vcpu 0 qemu-kvm-7767 [001] 235066.551667: kvm_exit: reason APIC_ACCESS rip 0x810219eb info 10b0 0 qemu-kvm-7767 [001] 235066.551667: kvm_apic: apic_write APIC_EOI = 0x0 qemu-kvm-7767 [001] 235066.551668: kvm_eoi: apicid 0 vector 81 qemu-kvm-7767 [001] 235066.551668: kvm_entry:vcpu 0 qemu-kvm-7767 [001] 235066.551681: kvm_exit: reason EPT_MISCONFIG rip 0x81023d96 info 0 0 qemu-kvm-7767 [001] 235066.551682:
Re: high host load from vmx_set_supported_cpuid call?
Hi Gleb, Something wrong with your symbols. This function cannot take that much. It is three and a half instruction long and should be called only once during vm startup. well, it didn't make any sense to me, glad I wasn't that wrong :) how could that be? I guess it could be perf/kernel mismatch right? I'll try to fix that and see if it helps.. thanks for Your time! n. -- - Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax:+420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: ser...@linuxbox.cz - pgpMhzIV2En3z.pgp Description: PGP signature
Re: high host load from vmx_set_supported_cpuid call?
On Sun, Dec 02, 2012 at 03:31:08PM +0100, Nikola Ciprich wrote: Hi Gleb, Something wrong with your symbols. This function cannot take that much. It is three and a half instruction long and should be called only once during vm startup. well, it didn't make any sense to me, glad I wasn't that wrong :) how could that be? I guess it could be perf/kernel mismatch right? I'll try to fix that and see if it helps.. More like loaded modules/installed modules mismatch. -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: high host load from vmx_set_supported_cpuid call?
More like loaded modules/installed modules mismatch. I see, the problem is, that I've got kvm-kmod compiled separately! thus kvm*.ko symboles don't match! I see that kvm-kmod build produces System.map file, I guess I need to merge it with kernel's System.map? -- - Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax:+420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: ser...@linuxbox.cz - pgpWo5au4KaYi.pgp Description: PGP signature
Re: high host load from vmx_set_supported_cpuid call?
On Sun, Dec 02, 2012 at 03:51:53PM +0100, Nikola Ciprich wrote: More like loaded modules/installed modules mismatch. I see, the problem is, that I've got kvm-kmod compiled separately! thus kvm*.ko symboles don't match! I see that kvm-kmod build produces System.map file, I guess I need to merge it with kernel's System.map? I think you need to copy them over old modules in /lib/modules. -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: high host load from vmx_set_supported_cpuid call?
I think you need to copy them over old modules in /lib/modules. hmm, that should be OK, new kvm*.ko modules are part of kernel rpm package, there's no old module there. (I checked by both inspecting kernel pkg and using modinfo)... Could it be something else? -- Gleb. -- - Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax:+420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: ser...@linuxbox.cz - pgpnVxHTWTEmQ.pgp Description: PGP signature
Re: high host load from vmx_set_supported_cpuid call?
hmm, that should be OK, new kvm*.ko modules are part of kernel rpm package, there's no old module there. (I checked by both inspecting kernel pkg and using modinfo)... Could it be something else? hmm, I can reply to myself this time - perf seems to get the symbols using /proc/kallsyms and there apparently are symbols from inkernel KVM modules, not the externally built. I guess the problem with my kernel package is I build both kernel and external KVM modules and then replace kernel ones causing kallsyms mismatch.. I'll have to fix this first, reboot host and then see... -- Gleb. -- - Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax:+420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: ser...@linuxbox.cz - -- - Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax:+420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: ser...@linuxbox.cz - pgpTi86wLYT8l.pgp Description: PGP signature
Re: high host load from vmx_set_supported_cpuid call?
On Sun, Dec 02, 2012 at 04:10:10PM +0100, Nikola Ciprich wrote: I think you need to copy them over old modules in /lib/modules. hmm, that should be OK, new kvm*.ko modules are part of kernel rpm package, there's no old module there. (I checked by both inspecting kernel pkg and using modinfo)... Could it be something else? Probably, I do not how to debug perf unfortunately. -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: high host load from vmx_set_supported_cpuid call?
On Sun, Dec 02, 2012 at 04:26:41PM +0100, Nikola Ciprich wrote: hmm, that should be OK, new kvm*.ko modules are part of kernel rpm package, there's no old module there. (I checked by both inspecting kernel pkg and using modinfo)... Could it be something else? hmm, I can reply to myself this time - perf seems to get the symbols using /proc/kallsyms and there apparently are symbols from inkernel KVM modules, not the externally built. I guess the problem with my kernel package is I build both kernel and external KVM modules and then replace kernel ones causing kallsyms mismatch.. I'll have to fix this first, reboot host and then see... And you can successfully install external modules while KVM is compiled in? Strange. -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
