[CentOS-virt] KVM + virsh nodeinfo + CentOS 6.3
Hi there, today I encountered the very same problem as described by Zoltan. We are running a system with the Intel serverboard S2600CP and two E5-2620 Xeon processors, with a total of 2 x 6 Cores with 2 threads each (resulting in a total CPU count of 24). Base system is a CentOS 6.3 with all recent updates. We are running the setup for some months now, without any problems. Most virtual machines only got 4 vCPUs, but some got 6 or 8 vCPUs. That setup was fine if running CentOS 5, CentOS 6 or Windows Server 2008 as guest systems. However, today I had to install two machines with Debian Squeeze. The first one got 4 vCPUs and runs fine, while the second one got 6 vCPUs and does not boot at all. Booting stops shortly after detecting the virtio harddisk with its partitions. When I switch back to 4 vCPUs everything is fine. Since the RHEL 5.8 and 6.3 kernels work fine, it seems to be a problem specific to the Debian kernel. Our default setup pins the different vCPUs to one fixed physical CPU each in an interleaving way. This would be the setup from /etc/libvirt/qemu/vm.xml: memory unit='KiB'12582912/memory currentMemory unit='KiB'12582912/currentMemory vcpu placement='static'6/vcpu cputune vcpupin vcpu='0' cpuset='0-5,12-17'/ vcpupin vcpu='1' cpuset='6-11,18-23'/ vcpupin vcpu='2' cpuset='0-5,12-17'/ vcpupin vcpu='3' cpuset='6-11,18-23'/ vcpupin vcpu='4' cpuset='0-5,12-17'/ vcpupin vcpu='5' cpuset='6-11,18-23'/ /cputune os type arch='x86_64' machine='rhel6.3.0'hvm/type boot dev='hd'/ /os features acpi/ apic/ pae/ /features cpu mode='host-model' model fallback='allow'/ topology sockets='6' cores='1' threads='1'/ /cpu clock offset='utc'/ on_poweroffdestroy/on_poweroff on_rebootrestart/on_reboot on_crashrestart/on_crash Such a setup works fine for Redhat-kernels, but not for the Debian kernel (did only test the default 2.6.32-5 amd64 kernel so far). I played a bit with the pinning and topology, but even setting the config to do no pinning at all, and for example use 8 vCPUs divided in 2 sockets with 4 cores, or 1 socket with 8 cores or any other combination of it, does not work. The only way to get the Debian VM to boot, is to roll back to a total of 4 vCPUs (or less). As a reference, these are the cpuflags, seen from the host: processor : 23 vendor_id : GenuineIntel cpu family : 6 model : 45 model name : Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz stepping: 7 cpu MHz : 1995.156 cache size : 15360 KB physical id : 1 siblings: 12 core id : 5 cpu cores : 6 apicid : 43 initial apicid : 43 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 x2apic popcnt aes xsave avx lahf_lm ida arat epb xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid bogomips: 3989.83 clflush size: 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: and these are the cpuflags seen from the guest: processor : 3 vendor_id : GenuineIntel cpu family : 6 model : 42 model name : Intel Xeon E312xx (Sandy Bridge) stepping: 1 cpu MHz : 1995.191 cache size : 4096 KB fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx lm constant_tsc arch_perfmon rep_good pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm bogomips: 3990.38 clflush size: 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: The host is running this kernel: 2.6.32-279.11.1.el6.x86_64 #1 SMP Tue Oct 16 15:57:10 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux Packages are up to date: qemu-kvm-0.12.1.2-2.295.el6_3.2.x86_64 libvirt-0.9.10-21.el6_3.5.x86_64 The kvm commandline looks like this (for 4 vCPUs): /usr/libexec/qemu-kvm -S -M rhel6.3.0 -cpu SandyBridge,+pdpe1gb,+osxsave,+tsc-deadline,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -enable-kvm -m 12288 -smp 4,sockets=4,cores=1,threads=1 -name -uuid ---- -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/xx.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
[CentOS-virt] KVM + virsh nodeinfo + CentOS 6.3
Hi, Please let me know in case I am posting my question to the wrong forum. I apologize if that is the case! Here is my question: We run CentOS 6.3 on a server with dual Xeon CPU's. Our dual blade server uses this motherboard: http://www.supermicro.com/products/motherboard/Xeon/C600/X9DRT-HF.cfm We have two of these CPUs installed and working: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz ( http://ark.intel.com/products/64594/Intel-Xeon-Processor-E5-2620-15M-Cache-2_00-GHz-7_20-GTs-Intel-QPI ) cat /proc/cpuinfo correctly reports a total of 24 cores (2 x 6 phisycal cores plus 2 x 6 hyperthreading cores) However, I get this output from virsh nodeinfo : # virsh nodeinfo CPU model: x86_64 CPU(s): 24 CPU frequency: 2000 MHz CPU socket(s): 1 Core(s) per socket: 6 Thread(s) per core: 2 NUMA cell(s):2 Memory size: 16303552 kB As you can see, virsh nodeinfo reports only 1 CPU socket while in fact we have two CPU's. I would like to know if this is normal? Why does virsh reports only one physical CPU ?? Also, when we try to run a guest OS (Debian Linux squeeze) with more than 4 vcpu's assigned to the VM, the guest OS won't boot up. The guest's kernel stuck on a screen right after it detected the /dev/vda block device and its partitions. We're using the VirtIO driver, of course. If I assign only 4 (or less) vcpu's to the guest OS it works fine. I have tried to upgrade the Linux kernel on the guest from debian backports, it did not help, we're experiencing the same issue with both the 2.6.32 and 3.2 Linux kernels. What could be causing this? On the host, we use the Linux kernel that came with CentOS 6.3 : 2.6.32-279.11.1.el6.x86_64 #1 SMP Tue Oct 16 15:57:10 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux Thanks, Zoltan ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] KVM + virsh nodeinfo + CentOS 6.3
Hello, You have a NUMA (Non Uniform Memory Access) machine, which mean that each processor has its own memory controller. virsh nodeinfo give you 2 NUMA cells with 1 CPU socket each: 2 NUMA cells x 1CPU socket x 6 Core(s) per socket x 2 threads per core = 24 cores. The NUMA concept is really important, especially in virtualization. If you have a virtual machine with vCPUs spread across more than one NUMA cell, performances will drop drastically. Maybe you cannot assign more than 4 cPUs to your VM because Libvirt cannot pin them all on the same NUMA cell ... You can try to specify the NUMA architecture in the xml config. Br, Bertrand. Le 24 octobre 2012 à 17:14, Zoltan Frombach zol...@frombach.com a écrit : Hi, Please let me know in case I am posting my question to the wrong forum. I apologize if that is the case! Here is my question: We run CentOS 6.3 on a server with dual Xeon CPU's. Our dual blade server uses this motherboard: http://www.supermicro.com/products/motherboard/Xeon/C600/X9DRT-HF.cfm We have two of these CPUs installed and working: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz ( http://ark.intel.com/products/64594/Intel-Xeon-Processor-E5-2620-15M-Cache-2_00-GHz-7_20-GTs-Intel-QPI ) cat /proc/cpuinfo correctly reports a total of 24 cores (2 x 6 phisycal cores plus 2 x 6 hyperthreading cores) However, I get this output from virsh nodeinfo : # virsh nodeinfo CPU model: x86_64 CPU(s): 24 CPU frequency: 2000 MHz CPU socket(s): 1 Core(s) per socket: 6 Thread(s) per core: 2 NUMA cell(s): 2 Memory size: 16303552 kB As you can see, virsh nodeinfo reports only 1 CPU socket while in fact we have two CPU's. I would like to know if this is normal? Why does virsh reports only one physical CPU ?? Also, when we try to run a guest OS (Debian Linux squeeze) with more than 4 vcpu's assigned to the VM, the guest OS won't boot up. The guest's kernel stuck on a screen right after it detected the /dev/vda block device and its partitions. We're using the VirtIO driver, of course. If I assign only 4 (or less) vcpu's to the guest OS it works fine. I have tried to upgrade the Linux kernel on the guest from debian backports, it did not help, we're experiencing the same issue with both the 2.6.32 and 3.2 Linux kernels. What could be causing this? On the host, we use the Linux kernel that came with CentOS 6.3 : 2.6.32-279.11.1.el6.x86_64 #1 SMP Tue Oct 16 15:57:10 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux Thanks, Zoltan ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] KVM + virsh nodeinfo + CentOS 6.3
Thank you for your quick reply! I understand the NUMA cell concept and I am using CPU pinning in the XML file. For example: domain type='kvm' nameDebian-/name uuid/uuid memory unit='KiB'8388608/memory currentMemory unit='KiB'8388608/currentMemory vcpu placement='static' cpuset='6-9,18-21'8/vcpu os type arch='x86_64' machine='rhel6.3.0'hvm/type ... /os ... This guest still hangs while starting up its Linux Kernel (3.2.x.x) ... :( Here is my virsh capabilities output from the host (CentOS 6.3): # virsh capabilities capabilities host uuid00020003-0004-0005-0006-000700080009/uuid cpu archx86_64/arch modelSandyBridge/model vendorIntel/vendor topology sockets='1' cores='6' threads='2'/ feature name='pdpe1gb'/ feature name='osxsave'/ feature name='tsc-deadline'/ feature name='dca'/ feature name='pdcm'/ feature name='xtpr'/ feature name='tm2'/ feature name='est'/ feature name='smx'/ feature name='vmx'/ feature name='ds_cpl'/ feature name='monitor'/ feature name='dtes64'/ feature name='pbe'/ feature name='tm'/ feature name='ht'/ feature name='ss'/ feature name='acpi'/ feature name='ds'/ feature name='vme'/ /cpu power_management suspend_disk/ /power_management migration_features live/ uri_transports uri_transporttcp/uri_transport /uri_transports /migration_features topology cells num='2' cell id='0' cpus num='12' cpu id='0'/ cpu id='1'/ cpu id='2'/ cpu id='3'/ cpu id='4'/ cpu id='5'/ cpu id='12'/ cpu id='13'/ cpu id='14'/ cpu id='15'/ cpu id='16'/ cpu id='17'/ /cpus /cell cell id='1' cpus num='12' cpu id='6'/ cpu id='7'/ cpu id='8'/ cpu id='9'/ cpu id='10'/ cpu id='11'/ cpu id='18'/ cpu id='19'/ cpu id='20'/ cpu id='21'/ cpu id='22'/ cpu id='23'/ /cpus /cell /cells /topology /host guest os_typehvm/os_type arch name='i686' wordsize32/wordsize emulator/usr/libexec/qemu-kvm/emulator machinerhel6.3.0/machine machine canonical='rhel6.3.0'pc/machine machinerhel6.2.0/machine machinerhel6.1.0/machine machinerhel6.0.0/machine machinerhel5.5.0/machine machinerhel5.4.4/machine machinerhel5.4.0/machine domain type='qemu' /domain domain type='kvm' emulator/usr/libexec/qemu-kvm/emulator /domain /arch features cpuselection/ deviceboot/ pae/ nonpae/ acpi default='on' toggle='yes'/ apic default='on' toggle='no'/ /features /guest guest os_typehvm/os_type arch name='x86_64' wordsize64/wordsize emulator/usr/libexec/qemu-kvm/emulator machinerhel6.3.0/machine machine canonical='rhel6.3.0'pc/machine machinerhel6.2.0/machine machinerhel6.1.0/machine machinerhel6.0.0/machine machinerhel5.5.0/machine machinerhel5.4.4/machine machinerhel5.4.0/machine domain type='qemu' /domain domain type='kvm' emulator/usr/libexec/qemu-kvm/emulator /domain /arch features cpuselection/ deviceboot/ acpi default='on' toggle='yes'/ apic default='on' toggle='no'/ /features /guest /capabilities And the odd thing is this: virsh freecell only provides a total, not a per node list: # virsh freecell Total: 15891284 kB According to this Fedora page http://docs.fedoraproject.org/en-US/Fedora/13/html/Virtualization_Guide/ch25s06.html I should see a per node list. Anyway, my Debian guest still does not boot up when I assign more than 4 vcpus to it. Even if I pin all cpus to the same NUMA node. BTW, I have copied my host CPU's configuration and CPU features for my guests (using virt-manager GUI, running remotely on an Ubuntu desktop box). Maybe I should use some predefined CPU type instead of cloning CPU configuration from the host?? Zoltan On 10/24/2012 5:58 PM, bertrand.louarg...@atoutlinux.net wrote: Hello, You have a NUMA (Non Uniform Memory Access) machine, which mean that each processor has its own memory controller. virsh nodeinfo give you 2 NUMA cells with 1 CPU socket each: 2 NUMA cells x 1CPU socket x 6 Core(s) per socket x 2 threads per core = 24 cores. The NUMA concept is really important, especially in virtualization. If you have a virtual machine with vCPUs spread across more than one NUMA cell, performances will drop drastically. Maybe you cannot assign more than 4 cPUs to your VM because Libvirt cannot pin them all on the same
