Migration (suspending) fails with large memory VM
Hi! In our cluster system (16 nodes, each node 32GB RAM and two quad-core CPUs) we are using Gentoo with kvm 0.15.0 for executing virtual machines. --- Linux asok01 3.0.4-gentoo #1 SMP Thu Sep 8 09:15:01 CEST 2011 x86_64 Intel(R) Xeon(R) CPU X5355 @ 2.66GHz GenuineIntel GNU/Linux --- Migrating VMs between server nodes works perfectly fine, except for migrating large memory VMs to local disk (a.k.a. suspending the VM). The command we use for migrating (suspending) the VM is: --- QEMU 0.15.0 monitor - type 'help' for more information (qemu) migrate -d "exec:gzip -c > /var/run/kvm/suspend-vm-219.bin.gz" --- For most VMs, this command triggers the suspension procedure, having the file suspend-vm-219.bin.gz being generated. The progress can be monitored using: --- QEMU 0.15.0 monitor - type 'help' for more information (qemu) info migrate Migration status: active transferred ram: 36758 kbytes remaining ram: 4164888 kbytes total ram: 4211136 kbytes (qemu) --- After some seconds, the memory of this VM has been written to disk, which concludes the migration process: --- QEMU 0.15.0 monitor - type 'help' for more information (qemu) info migrate Migration status: completed (qemu) --- Unfortunately, migration is not working for our large VMs, having 7 CPUs and 30720 KB of memory: --- qemu-system-x86_64 -name wip -boot c -smp 7 -m 30720 -drive file=/dev/disk/by-path/ip-192.168.251.1:3260-iscsi-iqn.2010-09.de.tu-berlin.cit.wip-lun-0 -net nic,model=e1000,macaddr=02:59:59:59:3:E4 -net tap,ifname=tap249,script=/etc/kvm/kvm-ifup,downscript=/etc/kvm/kvm-ifdown -k de -vnc :30249 -monitor unix:/var/run/kvm/wip/monitor,server,nowait -serial none -parallel none -pidfile /var/run/kvm/wip/pid --- In case of this VM, the migration process directly fails at execution time: --- QEMU 0.15.0 monitor - type 'help' for more information (qemu) migrate -d "exec:gzip -c > /var/run/kvm/suspend-vm-249.bin.gz" migration failed (qemu) --- I did quite some testing: - the problem seems to be independend from the number of virtual cores of the VM. Suspension of "-smp 7" fails as well as "-smp 1" - the problem seems to be only related to the amount of memory of the VM. Suspending "-m 30720" fails, but slightly reducing the memory of the VM is sufficient, so that suspending is working fine. For example, for "-m 3" the suspending is working. - the problem seems to affect ONLY the suspension using "exec:gzip". Really migrating the VM to another server host using "exec:tcp..." is working absolutely fine. Any help is greatly appreciated. Best, Matthias -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Administration panel for KVM
Hi! > With a group of college buddies, we are evaluating the possibility of > initiating a project to develop a management panel of KVM virtual > machines. The idea is to do something similar to OpenXenManager but for > KVM. At out university we developed a Perl-based management tool named "kvm-top". This tool is command-line only, not offering any GUI at the moment. The initial idea of that tool was to make the start-up of VMs easier than doing it manually. The tool analyzes a VM-specific config file like GUEST_ID=219 GUEST_NAME=attic . . defining all parameters for starting up a VM. For actually starting this VM, a single command now is sufficient: asok01 ~ # kvm-top -start attic This will not only start-up the VM "attic", but also check if this VM is running on some other cluster node and connect to the iSCSI target if required. Meanwhile, the tool has evolved, not only consisting of the "kvm-top" tool, but also a server component named "kvm-ctld" running on each cluster node. The "kvm-top" tool connects to the "kvm-ctld" running on the local host, executing the desired command. At this, the command does not nessecarily have to be executed on the same cluster node. For instance, it is easily possible to start/stop a VM running on a different cluster node. However, the main feature of "kvm-top" is giving information about the current status of the running VMs: asok01 ~ # kvm-top VM NODE AS 5s 30s USER PID #CPU MEM VNC SPICE #LAN = atticasok02 4 4 root 66141 2048 36003 -2 cbaseasok08 1 1 root 102221 1048 36142 -1 cbase-spice asok08 0 0 root 42691 1024 36143 59241 cloud-pj asok02 14 18 root 240711 1024 36001 -2 . . . where "5s" and "30s" contain the average system load over the last 5s resp. 30s. There are serveral ways of filtering or sorting the output, e.g. sorting by cluster nodes: asok01 ~ # kvm-top -s node NODE VM AS 5s 30s USER PID #CPU MEM VNC SPICE #LAN = asok01(ENABLED): 0(0) VMs, CPU=0%, MEM=2%, AGE 00:00 asok02(ENABLED): 7(8) VMs, CPU=13%, MEM=99%, AGE 00:05 attic 4 4 root 66141 2048 36003 -2 cloud-pj 21 19 root 240711 1024 36001 -2 . . The "kvm-top" tool even allows migration of VMs between the cluster nodes. The following command would migrate the VM "attic" from the currently used cluster node "asok02" to cluster node "asok07" (note: the command has been executed on a different cluster node "asok01"): asok01 ~ # kvm-top -migrate attic asok07 As I mentioned, the tool is command line only at the moment, however it shouldn't be too difficult to create a web-based interface, since the kvm-ctld allows communication not only with kvm-top. Connecting to the port of kvm-ctld, it's pretty easy to get information about all currently running VMs or start/stop/migrate VMs. If there's interest in that tool, please let me know. I'll gladly publish it. Regards, Matthias -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Problem with KVM-84 and more than 4 processors
Hi! I am unable to reproduce - 'modprobe kvm' gets me the expected lsmod line. Can you reproduce with plain 2.6.27 instead of the gentoo build? No, with a vanilla kernel from kernel.org it seems to work fine. asok04 ~ # uname -a Linux asok04 2.6.27.19 #1 SMP Wed Mar 11 15:59:54 CET 2009 x86_64 Intel(R) Xeon(R) CPU X5355 @ 2.66GHz GenuineIntel GNU/Linux asok04 ~ # asok04 ~ # cat /proc/cpuinfo | grep proc | wc -l 8 asok04 ~ # lsmod Module Size Used by crc32c 3520 2 libcrc32c 3200 1 crc32c iscsi_tcp 18640 2 libiscsi 27136 1 iscsi_tcp scsi_transport_iscsi32352 3 iscsi_tcp,libiscsi tun11844 2 kvm_intel 44384 1 kvm 129456 1 kvm_intel scsi_wait_scan 1920 0 asok04 ~ # Best, Matthias -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Problem with KVM-84 and more than 4 processors
Hi! Manually loading the kvm_intel module fails with an "Cannot allocate memory" error. Also starting VMs fails since he is unable to find /dev/kvm. Looks like a memory corruption issue. Please post your .config. Here it is... Best, Matthias # # Automatically generated make config: don't edit # Linux kernel version: 2.6.27-gentoo-r8 # Tue Mar 3 11:30:59 2009 # CONFIG_64BIT=y # CONFIG_X86_32 is not set CONFIG_X86_64=y CONFIG_X86=y CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig" # CONFIG_GENERIC_LOCKBREAK is not set CONFIG_GENERIC_TIME=y CONFIG_GENERIC_CMOS_UPDATE=y CONFIG_CLOCKSOURCE_WATCHDOG=y CONFIG_GENERIC_CLOCKEVENTS=y CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y CONFIG_LOCKDEP_SUPPORT=y CONFIG_STACKTRACE_SUPPORT=y CONFIG_HAVE_LATENCYTOP_SUPPORT=y CONFIG_FAST_CMPXCHG_LOCAL=y CONFIG_MMU=y CONFIG_ZONE_DMA=y CONFIG_GENERIC_ISA_DMA=y CONFIG_GENERIC_IOMAP=y CONFIG_GENERIC_BUG=y CONFIG_GENERIC_HWEIGHT=y # CONFIG_GENERIC_GPIO is not set CONFIG_ARCH_MAY_HAVE_PC_FDC=y CONFIG_RWSEM_GENERIC_SPINLOCK=y # CONFIG_RWSEM_XCHGADD_ALGORITHM is not set # CONFIG_ARCH_HAS_ILOG2_U32 is not set # CONFIG_ARCH_HAS_ILOG2_U64 is not set CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y CONFIG_GENERIC_CALIBRATE_DELAY=y CONFIG_GENERIC_TIME_VSYSCALL=y CONFIG_ARCH_HAS_CPU_RELAX=y CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y CONFIG_HAVE_SETUP_PER_CPU_AREA=y CONFIG_HAVE_CPUMASK_OF_CPU_MAP=y CONFIG_ARCH_HIBERNATION_POSSIBLE=y CONFIG_ARCH_SUSPEND_POSSIBLE=y CONFIG_ZONE_DMA32=y CONFIG_ARCH_POPULATES_NODE_MAP=y CONFIG_AUDIT_ARCH=y CONFIG_ARCH_SUPPORTS_AOUT=y CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y CONFIG_GENERIC_HARDIRQS=y CONFIG_GENERIC_IRQ_PROBE=y CONFIG_GENERIC_PENDING_IRQ=y CONFIG_X86_SMP=y CONFIG_X86_64_SMP=y CONFIG_X86_HT=y CONFIG_X86_BIOS_REBOOT=y CONFIG_X86_TRAMPOLINE=y # CONFIG_KTIME_SCALAR is not set CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config" # # General setup # CONFIG_EXPERIMENTAL=y CONFIG_LOCK_KERNEL=y CONFIG_INIT_ENV_ARG_LIMIT=32 CONFIG_LOCALVERSION="" # CONFIG_LOCALVERSION_AUTO is not set CONFIG_SWAP=y CONFIG_SYSVIPC=y CONFIG_SYSVIPC_SYSCTL=y CONFIG_POSIX_MQUEUE=y CONFIG_BSD_PROCESS_ACCT=y # CONFIG_BSD_PROCESS_ACCT_V3 is not set CONFIG_TASKSTATS=y CONFIG_TASK_DELAY_ACCT=y CONFIG_TASK_XACCT=y CONFIG_TASK_IO_ACCOUNTING=y CONFIG_AUDIT=y CONFIG_AUDITSYSCALL=y CONFIG_AUDIT_TREE=y # CONFIG_IKCONFIG is not set CONFIG_LOG_BUF_SHIFT=17 CONFIG_CGROUPS=y # CONFIG_CGROUP_DEBUG is not set CONFIG_CGROUP_NS=y # CONFIG_CGROUP_DEVICE is not set CONFIG_CPUSETS=y CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y CONFIG_GROUP_SCHED=y CONFIG_FAIR_GROUP_SCHED=y # CONFIG_RT_GROUP_SCHED is not set # CONFIG_USER_SCHED is not set CONFIG_CGROUP_SCHED=y CONFIG_CGROUP_CPUACCT=y CONFIG_RESOURCE_COUNTERS=y # CONFIG_CGROUP_MEM_RES_CTLR is not set # CONFIG_SYSFS_DEPRECATED_V2 is not set CONFIG_PROC_PID_CPUSET=y CONFIG_RELAY=y CONFIG_NAMESPACES=y CONFIG_UTS_NS=y CONFIG_IPC_NS=y CONFIG_USER_NS=y CONFIG_PID_NS=y CONFIG_BLK_DEV_INITRD=y CONFIG_INITRAMFS_SOURCE="" CONFIG_CC_OPTIMIZE_FOR_SIZE=y CONFIG_SYSCTL=y # CONFIG_EMBEDDED is not set CONFIG_UID16=y CONFIG_SYSCTL_SYSCALL=y CONFIG_KALLSYMS=y CONFIG_KALLSYMS_ALL=y CONFIG_KALLSYMS_EXTRA_PASS=y CONFIG_HOTPLUG=y CONFIG_PRINTK=y CONFIG_BUG=y CONFIG_ELF_CORE=y CONFIG_PCSPKR_PLATFORM=y # CONFIG_COMPAT_BRK is not set CONFIG_BASE_FULL=y CONFIG_FUTEX=y CONFIG_ANON_INODES=y CONFIG_EPOLL=y CONFIG_SIGNALFD=y CONFIG_TIMERFD=y CONFIG_EVENTFD=y CONFIG_SHMEM=y CONFIG_VM_EVENT_COUNTERS=y CONFIG_SLUB_DEBUG=y # CONFIG_SLAB is not set CONFIG_SLUB=y # CONFIG_SLOB is not set CONFIG_PROFILING=y CONFIG_MARKERS=y # CONFIG_OPROFILE is not set CONFIG_HAVE_OPROFILE=y CONFIG_KPROBES=y CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y CONFIG_KRETPROBES=y CONFIG_HAVE_IOREMAP_PROT=y CONFIG_HAVE_KPROBES=y CONFIG_HAVE_KRETPROBES=y # CONFIG_HAVE_ARCH_TRACEHOOK is not set # CONFIG_HAVE_DMA_ATTRS is not set CONFIG_USE_GENERIC_SMP_HELPERS=y # CONFIG_HAVE_CLK is not set CONFIG_PROC_PAGE_MONITOR=y # CONFIG_HAVE_GENERIC_DMA_COHERENT is not set CONFIG_SLABINFO=y CONFIG_RT_MUTEXES=y # CONFIG_TINY_SHMEM is not set CONFIG_BASE_SMALL=0 CONFIG_MODULES=y # CONFIG_MODULE_FORCE_LOAD is not set CONFIG_MODULE_UNLOAD=y CONFIG_MODULE_FORCE_UNLOAD=y # CONFIG_MODVERSIONS is not set # CONFIG_MODULE_SRCVERSION_ALL is not set CONFIG_KMOD=y CONFIG_STOP_MACHINE=y CONFIG_BLOCK=y CONFIG_BLK_DEV_IO_TRACE=y CONFIG_BLK_DEV_BSG=y # CONFIG_BLK_DEV_INTEGRITY is not set CONFIG_BLOCK_COMPAT=y # # IO Schedulers # CONFIG_IOSCHED_NOOP=y CONFIG_IOSCHED_AS=y CONFIG_IOSCHED_DEADLINE=y CONFIG_IOSCHED_CFQ=y # CONFIG_DEFAULT_AS is not set # CONFIG_DEFAULT_DEADLINE is not set CONFIG_DEFAULT_CFQ=y # CONFIG_DEFAULT_NOOP is not set CONFIG_DEFAULT_IOSCHED="cfq" CONFIG_PREEMPT_NOTIFIERS=y CONFIG_CLASSIC_RCU=y # # Processor type and features # CONFIG_TICK_ONESHOT=y CONFIG_NO_HZ=y CONFIG_HIGH_RES_TIMERS=y CONFIG_GENERIC_CLOCKEVENTS_BUILD=y CONFIG_SMP=y CONFIG_X86_FIND_SMP_CONFIG=y CONFIG_X86_MPPARSE=y CONFIG_X86_PC=y # CONFIG_X86_ELAN is not set # CONFIG_X86_VOYAGER is not set # CONFIG_X86_GENERICARCH is not se
Re: Problem with KVM-84 and more than 4 processors
Hi Charles! This mail seems to be somehow related to the bug report of Justin Keogh from 30th Dec 2008. He was able to fix his problem by simply setting CONFIG_KVM=m, but I'm already using modules, so his workaround can't be applied here. Quick question -- does your dmesg output include "loaded kvm module (kvm-84)" [ie. including the version number]? If not, it may be that you're using the module built with your kernel rather than the one built with the kvm release package. This is from my dmesg output: [ 11.983892] loaded kvm module (kvm-84) Best, Matthias -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Problem with KVM-84 and more than 4 processors
Hi! I have a system with 2 processors and 4 cores each, which sums up to 8 processors in total. I installed the system with Gentoo.2008 and KVM-84. Linux asok04 2.6.27-gentoo-r8 #1 SMP Tue Mar 3 11:34:26 CET 2009 x86_64 Intel(R) Xeon(R) CPU X5355 @ 2.66GHz GenuineIntel GNU/Linux Everything is running fine as long as I do not change the default setting of CONFIG_NR_CPUS=4 in the kernel config: asok04 ~ # lsmod Module Size Used by kvm_intel 39176 1 kvm 137072 1 kvm_intel scsi_wait_scan 1664 0 asok04 ~ # All KVM modules are loaded and I'm perfectly able to start and use virtual guests. However, if I increase CONFIG_NR_CPUS to any number higher than 4, the additional processors of my system are listed in /proc/cpuinfo, but this is the output of lsmod: asok04 ~ # lsmod Module Size Used by kvm 137072 283728377 scsi_wait_scan 1664 0 asok04 ~ # Manually loading the kvm_intel module fails with an "Cannot allocate memory" error. Also starting VMs fails since he is unable to find /dev/kvm. This mail seems to be somehow related to the bug report of Justin Keogh from 30th Dec 2008. He was able to fix his problem by simply setting CONFIG_KVM=m, but I'm already using modules, so his workaround can't be applied here. Thanks in advance for your help, Matthias -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
