Public bug reported: We are seeing issues after live migrating a KVM VMs from a Trusty to Bionic where about 1 in 10 VMs go to 100% cpu and become unresponsive. When this happens, virsh returns error code 0, and there are no logs in /var/log/libvirt/qemu, /var/log/syslog or dmesg to indicate an error/issue.
We expect that the VM should migrate successfully, as it's a supported option to go +2 LTS releases according to https://wiki.ubuntu.com/QemuKVMMigration#Support_Matrix. The VM in this instance was also running Trusty. Unfortunately we are unable to reproduce this on demand, and it seems random whether a VM will successfully live migrate or not. We also tried migrating to Bionic 18.04.0 with 4.15.0-43-generic with the same results. Please let me know if there is further information you require to look into this. Thank you, Corey Melanson Source hypervisor: OS: Ubuntu 14.04.5 Kernel: Linux kvm-207-38 3.13.0-157-generic #207-Ubuntu SMP Mon Aug 20 16:44:59 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Hardware: HP ProLiant DL360 Gen9 CPU: Dual Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz Packages: $ dpkg -l | egrep '(qemu|libvirt|linux-image)' ii ipxe-qemu 1.0.0+git-20131111.c3d1e78-2ubuntu1.1 all PXE boot firmware - ROM images for qemu ii libvirt-bin 1.2.2-0ubuntu13.1.27 amd64 programs for the libvirt library ii libvirt0 1.2.2-0ubuntu13.1.27 amd64 library for interfacing with different virtualization systems ii linux-image-3.13.0-157-generic 3.13.0-157.207 amd64 Linux kernel image for version 3.13.0 on 64 bit x86 SMP ii linux-image-extra-3.13.0-157-generic 3.13.0-157.207 amd64 Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP ii python-libvirt 1.2.2-0ubuntu2 amd64 libvirt Python bindings ii qemu-keymaps 2.0.0+dfsg-2ubuntu1.44 all QEMU keyboard maps ii qemu-kvm 2.0.0+dfsg-2ubuntu1.44 amd64 QEMU Full virtualization ii qemu-system-common 2.0.0+dfsg-2ubuntu1.44 amd64 QEMU full system emulation binaries (common files) ii qemu-system-x86 2.0.0+dfsg-2ubuntu1.44 amd64 QEMU full system emulation binaries (x86) ii qemu-utils 2.0.0+dfsg-2ubuntu1.44 amd64 QEMU utilities Destination hypervisor: OS: Ubuntu 18.04.2 Kernel: Linux kvm-207-39 4.18.0-15-generic #16~18.04.1-Ubuntu SMP Thu Feb 7 14:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Hardware: HP ProLiant DL360 Gen9 CPU: Dual Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz Packages: $ dpkg -l | egrep '(qemu|libvirt|linux-image)' ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2.2 all PXE boot firmware - ROM images for qemu ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu ii libvirt-bin 4.0.0-1ubuntu8.6 amd64 programs for the libvirt library ii libvirt-clients 4.0.0-1ubuntu8.6 amd64 Programs for the libvirt library ii libvirt-daemon 4.0.0-1ubuntu8.6 amd64 Virtualization daemon ii libvirt-daemon-system 4.0.0-1ubuntu8.6 amd64 Libvirt daemon configuration files ii libvirt0:amd64 4.0.0-1ubuntu8.6 amd64 library for interfacing with different virtualization systems ii linux-image-4.18.0-15-generic 4.18.0-15.16~18.04.1 amd64 Signed kernel image generic ii qemu-block-extra:amd64 1:2.11+dfsg-1ubuntu7.10 amd64 extra block backend modules for qemu-system and qemu-utils ii qemu-kvm 1:2.11+dfsg-1ubuntu7.10 amd64 QEMU Full virtualization on x86 hardware ii qemu-system-common 1:2.11+dfsg-1ubuntu7.10 amd64 QEMU full system emulation binaries (common files) ii qemu-system-x86 1:2.11+dfsg-1ubuntu7.10 amd64 QEMU full system emulation binaries (x86) ii qemu-utils 1:2.11+dfsg-1ubuntu7.10 amd64 QEMU utilities Process running at 100% after migration: oneadmin 9483 1 99 18:07 ? 12:03:30 qemu-system-x86_64 -enable-kvm -name guest=one-10837,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-7-one-10837/master-key.aes -machine pc-i440fx-trusty,accel=kvm,usb=off,dump-guest-core=off -cpu Westmere,pcid=on -m 32768 -realtime mlock=off -smp 8,sockets=8,cores=1,threads=1 -uuid aa0038b1-876e-45eb-96a7-2df9b6fb32ea -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-7-one-10837/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -object secret,id=virtio-disk0-secret0,data=redacted,keyid=masterKey0,iv=redacted,format=base64 -drive file=rbd:ssd/one-457-10837-0:id=libvirt:auth_supported=cephx\;none:mon_host=ceph-mon-1\:6789\;ceph-mon-2\:6789\;ceph-mon-3\:6789,file.password-secret=virtio-disk0-secret0,format=raw,if=none,id=drive-virtio-disk0,cache=writeback,throttling.bps-total=52428800,throttling.iops-total=500 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/srv/datastores/107/10837/disk.1,format=raw,if=none,id=drive-ide0-0-0,readonly=on -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=36,id=hostnet0,vhost=on,vhostfd=38 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=02:23:0a:3e:42:6a,bus=pci.0,addr=0x3 -vnc 0.0.0.0:10837 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -incoming defer -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on /var/log/libvirt/qemu/one-10837.log: 2019-04-03 18:07:40.544+0000: starting up libvirt version: 4.0.0, package: 1ubuntu8.6 (Christian Ehrhardt <christian.ehrha...@canonical.com> Fri, 09 Nov 2018 07:42:01 +0100), qemu version: 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.10), hostname: kvm-207-39 LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -name guest=one-10837,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-7-one-10837/master-key.aes -machine pc-i440fx-trusty,accel=kvm,usb=off,dump-guest-core=off -cpu Westmere,pcid=on -m 32768 -realtime mlock=off -smp 8,sockets=8,cores=1,threads=1 -uuid aa0038b1-876e-45eb-96a7-2df9b6fb32ea -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-7-one-10837/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -object secret,id=virtio-disk0-secret0,data=redacted,keyid=masterKey0,redacted,format=base64 -drive 'file=rbd:ssd/one-457-10837-0:id=libvirt:auth_supported=cephx\;none:mon_host=ceph-mon-1\:6789\;ceph-mon-2\:6789\;ceph-mon-3\:6789,file.password-secret=virtio-disk0-secret0,format=raw,if=none,id=drive-virtio-disk0,cache=writeback,throttling.bps-total=52428800,throttling.iops-total=500' -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/srv/datastores/107/10837/disk.1,format=raw,if=none,id=drive-ide0-0-0,readonly=on -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=36,id=hostnet0,vhost=on,vhostfd=38 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=02:23:0a:3e:42:6a,bus=pci.0,addr=0x3 -vnc 0.0.0.0:10837 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -incoming defer -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on 2019-04-03T20:07:41.740926Z qemu-system-x86_64: terminating on signal 15 from pid 3945 (/usr/sbin/libvirtd) 2019-04-03 20:07:41.941+0000: shutting down, reason=destroyed /var/log/syslog from that time period: Apr 3 18:07:40 kvm-207-39 systemd[1]: Started Session 33 of user oneadmin. Apr 3 18:07:40 kvm-207-39 systemd-networkd[1155]: one-10837-0: Gained carrier Apr 3 18:07:40 kvm-207-39 networkd-dispatcher[1106]: WARNING:Unknown index 28 seen, reloading interface list Apr 3 18:07:40 kvm-207-39 kernel: [ 610.162987] onebr.66: port 2(one-10837-0) entered blocking state Apr 3 18:07:40 kvm-207-39 kernel: [ 610.162990] onebr.66: port 2(one-10837-0) entered disabled state Apr 3 18:07:40 kvm-207-39 kernel: [ 610.163084] device one-10837-0 entered promiscuous mode Apr 3 18:07:40 kvm-207-39 kernel: [ 610.163301] onebr.66: port 2(one-10837-0) entered blocking state Apr 3 18:07:40 kvm-207-39 kernel: [ 610.163303] onebr.66: port 2(one-10837-0) entered forwarding state Apr 3 18:08:07 kvm-207-39 systemd[1]: Started Session 34 of user oneadmin. Apr 3 18:08:11 kvm-207-39 systemd[1]: Started Session 35 of user oneadmin. ** Affects: qemu (Ubuntu) Importance: Undecided Status: New ** Attachment added: "dmesg" https://bugs.launchpad.net/bugs/1826051/+attachment/5258335/+files/dmesg.log -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1826051 Title: VMs go to 100% CPU after live migration from Trusty to Bionic To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1826051/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
