[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
** Tags added: cscc -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: Fix Released Status in linux package in Ubuntu: Invalid Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Xenial: Fix Released Bug description: [Impact] * Backport fix from the 2.6.2 stable branch to the qemu 2.5 in Xenial * Newer guests might use virtio attributes that are clobbered on migration with the old qemu code. [Test Case] * Setup two Xenial hosts on ppc64el * Create a guest that has a rather new kernel (>=4.14) I'd recommend Bionic * Migrate that guest from Host1 to Host2 [Regression Potential] * The modification could affect virtio handling in other cases in a non expected way, but mostly related to migrations. So the expected regression would be issues to migrate properly. I verified plenty of migrations in regression testing and we had this very code in the Yakkety release as we picked 2.6.1 stable release back then. Due to that it is actually pretty well tested and should not really regress anything out in the wild. [Other Info] * So far this only triggers on the confused endian marshalling on ppc64el, but in theory a different case could trigger it on x86 just as much. --- Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
** Changed in: linux (Ubuntu) Status: Triaged => Invalid ** Changed in: ubuntu-power-systems Status: In Progress => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: Fix Released Status in linux package in Ubuntu: Invalid Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Xenial: Fix Released Bug description: [Impact] * Backport fix from the 2.6.2 stable branch to the qemu 2.5 in Xenial * Newer guests might use virtio attributes that are clobbered on migration with the old qemu code. [Test Case] * Setup two Xenial hosts on ppc64el * Create a guest that has a rather new kernel (>=4.14) I'd recommend Bionic * Migrate that guest from Host1 to Host2 [Regression Potential] * The modification could affect virtio handling in other cases in a non expected way, but mostly related to migrations. So the expected regression would be issues to migrate properly. I verified plenty of migrations in regression testing and we had this very code in the Yakkety release as we picked 2.6.1 stable release back then. Due to that it is actually pretty well tested and should not really regress anything out in the wild. [Other Info] * So far this only triggers on the confused endian marshalling on ppc64el, but in theory a different case could trigger it on x86 just as much. --- Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
Yes both are known to be flaky - thanks! -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: In Progress Status in linux package in Ubuntu: Triaged Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Xenial: Fix Released Bug description: [Impact] * Backport fix from the 2.6.2 stable branch to the qemu 2.5 in Xenial * Newer guests might use virtio attributes that are clobbered on migration with the old qemu code. [Test Case] * Setup two Xenial hosts on ppc64el * Create a guest that has a rather new kernel (>=4.14) I'd recommend Bionic * Migrate that guest from Host1 to Host2 [Regression Potential] * The modification could affect virtio handling in other cases in a non expected way, but mostly related to migrations. So the expected regression would be issues to migrate properly. I verified plenty of migrations in regression testing and we had this very code in the Yakkety release as we picked 2.6.1 stable release back then. Due to that it is actually pretty well tested and should not really regress anything out in the wild. [Other Info] * So far this only triggers on the confused endian marshalling on ppc64el, but in theory a different case could trigger it on x86 just as much. --- Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
This bug was fixed in the package qemu - 1:2.5+dfsg-5ubuntu10.32 --- qemu (1:2.5+dfsg-5ubuntu10.32) xenial; urgency=medium * fix migration of new guests on ppc64el (LP: #1783140) Fixed by backporting two patches from the 2.6.x stable branch - d/p/ubuntu/lp-1783140-virtio-set-low-features-early-on-load.patch - d/p/ubuntu/lp-1783140-Revert-virtio-net-unbreak-self-announcement.patch -- Christian Ehrhardt Tue, 11 Sep 2018 15:00:19 +0200 ** Changed in: qemu (Ubuntu Xenial) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: In Progress Status in linux package in Ubuntu: Triaged Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Xenial: Fix Released Bug description: [Impact] * Backport fix from the 2.6.2 stable branch to the qemu 2.5 in Xenial * Newer guests might use virtio attributes that are clobbered on migration with the old qemu code. [Test Case] * Setup two Xenial hosts on ppc64el * Create a guest that has a rather new kernel (>=4.14) I'd recommend Bionic * Migrate that guest from Host1 to Host2 [Regression Potential] * The modification could affect virtio handling in other cases in a non expected way, but mostly related to migrations. So the expected regression would be issues to migrate properly. I verified plenty of migrations in regression testing and we had this very code in the Yakkety release as we picked 2.6.1 stable release back then. Due to that it is actually pretty well tested and should not really regress anything out in the wild. [Other Info] * So far this only triggers on the confused endian marshalling on ppc64el, but in theory a different case could trigger it on x86 just as much. --- Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
I see the following autopkgtest failures: Regression in autopkgtest for open-iscsi (amd64): test log Regression in autopkgtest for ubuntu-image (amd64): test log However looking at the history the particular tests failing seem to be flaky and unrelated to this SRU, and all the others continue to pass. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: In Progress Status in linux package in Ubuntu: Triaged Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Xenial: Fix Released Bug description: [Impact] * Backport fix from the 2.6.2 stable branch to the qemu 2.5 in Xenial * Newer guests might use virtio attributes that are clobbered on migration with the old qemu code. [Test Case] * Setup two Xenial hosts on ppc64el * Create a guest that has a rather new kernel (>=4.14) I'd recommend Bionic * Migrate that guest from Host1 to Host2 [Regression Potential] * The modification could affect virtio handling in other cases in a non expected way, but mostly related to migrations. So the expected regression would be issues to migrate properly. I verified plenty of migrations in regression testing and we had this very code in the Yakkety release as we picked 2.6.1 stable release back then. Due to that it is actually pretty well tested and should not really regress anything out in the wild. [Other Info] * So far this only triggers on the confused endian marshalling on ppc64el, but in theory a different case could trigger it on x86 just as much. --- Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
Testes as-is (to confirm we hit the bug) 1.0.0 (12:53:43): MIGRATE: in-release migrations 1.1.0 (12:53:43): Clean testbeds 1.1.1 (12:53:43): stop containers 1.1.2 (12:53:43): orig: restore containers from snapshot: xenial 1.1.3 (12:53:43): Restore testkvm-xenial-from 1.1.4 (12:53:44): Restore testkvm-xenial-to 1.1.5 (12:53:45): Restore testkvm-xenial-tononshared 1.1.6 (12:53:45): wait until containers are fully started 1.2.0 (12:53:56): unshare non shared container 1.2.1 (12:54:00): Version info after restore 1.2.2 (12:54:00): Version at testkvm-xenial-from: - qemu: 1:2.5+dfsg-5ubuntu10.31 libvirt: 1.3.1-1ubuntu10.24 1.2.3 (12:54:00): Bios versions at testkvm-xenial-from: - ipxe: 1.0.0+git-20150424.a25a16d-1ubuntu1.2 slof: 20151103+dfsg-1ubuntu1.1 efi: not-installed 1.2.4 (12:54:01): Version at testkvm-xenial-to: - qemu: 1:2.5+dfsg-5ubuntu10.31 libvirt: 1.3.1-1ubuntu10.24 1.2.5 (12:54:01): Bios versions at testkvm-xenial-to: - ipxe: 1.0.0+git-20150424.a25a16d-1ubuntu1.2 slof: 20151103+dfsg-1ubuntu1.1 efi: not-installed 1.2.6 (12:54:01): Version at testkvm-xenial-tononshared: - qemu: 1:2.5+dfsg-5ubuntu10.31 libvirt: 1.3.1-1ubuntu10.24 1.2.7 (12:54:01): Bios versions at testkvm-xenial-tononshared: - ipxe: 1.0.0+git-20150424.a25a16d-1ubuntu1.2 slof: 20151103+dfsg-1ubuntu1.1 efi: not-installed 1.2.8 (12:54:12): Ensure old migration guests of any release are removed 1.2.9 (12:54:12): Remove all test guests of release trusty 1.2.10 (12:54:26): Remove all test guests of release xenial 1.2.11 (12:54:40): Remove all test guests of release bionic 1.2.12 (12:54:54): Remove all test guests of release cosmic 1.2.13 (12:55:07): Prep xenial guest creation on testkvm-xenial-from 1.2.14 (12:55:07): spawn migration guests 1.2.15 (13:00:59): Test machine type uniqueness within xenial => Pass 1.2.16 (13:00:59): Check for expected machine type to be set => Pass 2.0.0 (13:01:00): Test migrations within xenial - round 1/5 2.1.0 (13:01:00): Test live migration (extra option '') of a xenial guest testkvm-xenial-from/testkvm-xenial-to 2.1.1 (13:01:00): live migration (extra option '') testkvm-xenial-from -> testkvm-xenial-to => Failed detail=live migration failed --- Then running the same upgrading to proposed (actually all of proposed, so I hope nothing else in there breaks us now - as we have tested that in advance and it was good). 1.0.0 (13:37:26): MIGRATE: in-release migrations 1.1.0 (13:37:26): Clean testbeds 1.1.1 (13:37:26): stop containers 1.1.2 (13:37:26): orig: restore containers from snapshot: xenial 1.1.3 (13:37:26): Restore testkvm-xenial-from 1.1.4 (13:37:27): Restore testkvm-xenial-to 1.1.5 (13:37:27): Restore testkvm-xenial-noupd 1.1.6 (13:37:28): Restore testkvm-xenial-tononshared 1.1.7 (13:37:28): wait until containers are fully started 1.2.0 (13:37:54): unshare non shared container 1.2.1 (13:37:58): Version info after restore 1.2.2 (13:37:58): Version at testkvm-xenial-from: - qemu: 1:2.5+dfsg-5ubuntu10.32 libvirt: 1.3.1-1ubuntu10.24 1.2.3 (13:37:58): Bios versions at testkvm-xenial-from: - ipxe: 1.0.0+git-20150424.a25a16d-1ubuntu1.2 slof: 20151103+dfsg-1ubuntu1.1 efi: not-installed 1.2.4 (13:37:59): Version at testkvm-xenial-to: - qemu: 1:2.5+dfsg-5ubuntu10.32 libvirt: 1.3.1-1ubuntu10.24 1.2.5 (13:37:59): Bios versions at testkvm-xenial-to: - ipxe: 1.0.0+git-20150424.a25a16d-1ubuntu1.2 slof: 20151103+dfsg-1ubuntu1.1 efi: not-installed 1.2.6 (13:37:59): Version at testkvm-xenial-noupd: - qemu: 1:2.5+dfsg-5ubuntu10.31 libvirt: 1.3.1-1ubuntu10.24 1.2.7 (13:37:59): Bios versions at testkvm-xenial-noupd: - ipxe: 1.0.0+git-20150424.a25a16d-1ubuntu1.2 slof: 20151103+dfsg-1ubuntu1.1 efi: not-installed 1.2.8 (13:38:00): Version at testkvm-xenial-tononshared: - qemu: 1:2.5+dfsg-5ubuntu10.32 libvirt: 1.3.1-1ubuntu10.24 1.2.9 (13:38:00): Bios versions at testkvm-xenial-tononshared: - ipxe: 1.0.0+git-20150424.a25a16d-1ubuntu1.2 slof: 20151103+dfsg-1ubuntu1.1 efi: not-installed 1.2.10 (13:38:10): Ensure old migration guests of any release are removed 1.2.11 (13:38:10): Remove all test guests of release trusty 1.2.12 (13:38:27): Remove all test guests of release xenial 1.2.13 (13:38:45): Remove all test guests of release bionic 1.2.14 (13:39:02): Remove all test guests of release cosmic 1.2.15 (13:39:19): Prep xenial guest creation on testkvm-xenial-from 1.2.16 (13:39:19): spawn migration guests 1.2.17 (13:45:13): Test machine type uniqueness within xenial => Pass 1.2.18 (13:45:13): Check for expected machine type to be set => Pass 2.0.0 (13:45:14): Test migrations within xenial - round 1/5 2.1.0 (13:45:14): Test live migration (extra option '') of a xenial guest testkvm-xenial-from/testkvm-xenial-to 2.1.1 (13:45:14): live migration (extra option '') testkvm-xenial-from -> testkvm-xenia
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
Hello bugproxy, or anyone else affected, Accepted qemu into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:2.5+dfsg- 5ubuntu10.32 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: qemu (Ubuntu Xenial) Status: New => Fix Committed ** Tags added: verification-needed verification-needed-xenial ** Changed in: qemu (Ubuntu) Status: Triaged => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: In Progress Status in linux package in Ubuntu: Triaged Status in qemu package in Ubuntu: Fix Released Status in qemu source package in Xenial: Fix Committed Bug description: [Impact] * Backport fix from the 2.6.2 stable branch to the qemu 2.5 in Xenial * Newer guests might use virtio attributes that are clobbered on migration with the old qemu code. [Test Case] * Setup two Xenial hosts on ppc64el * Create a guest that has a rather new kernel (>=4.14) I'd recommend Bionic * Migrate that guest from Host1 to Host2 [Regression Potential] * The modification could affect virtio handling in other cases in a non expected way, but mostly related to migrations. So the expected regression would be issues to migrate properly. I verified plenty of migrations in regression testing and we had this very code in the Yakkety release as we picked 2.6.1 stable release back then. Due to that it is actually pretty well tested and should not really regress anything out in the wild. [Other Info] * So far this only triggers on the confused endian marshalling on ppc64el, but in theory a different case could trigger it on x86 just as much. --- Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
Prep complete: - Regression tests from the PPA - Tests of the bug being fixed on the PPA - MP review - SRU Template Uploading for consideration by the SRU Team -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: In Progress Status in linux package in Ubuntu: Triaged Status in qemu package in Ubuntu: Triaged Bug description: [Impact] * Backport fix from the 2.6.2 stable branch to the qemu 2.5 in Xenial * Newer guests might use virtio attributes that are clobbered on migration with the old qemu code. [Test Case] * Setup two Xenial hosts on ppc64el * Create a guest that has a rather new kernel (>=4.14) I'd recommend Bionic * Migrate that guest from Host1 to Host2 [Regression Potential] * The modification could affect virtio handling in other cases in a non expected way, but mostly related to migrations. So the expected regression would be issues to migrate properly. I verified plenty of migrations in regression testing and we had this very code in the Yakkety release as we picked 2.6.1 stable release back then. Due to that it is actually pretty well tested and should not really regress anything out in the wild. [Other Info] * So far this only triggers on the confused endian marshalling on ppc64el, but in theory a different case could trigger it on x86 just as much. --- Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
** Description changed: + [Impact] + + * Backport fix from the 2.6.2 stable branch to the qemu 2.5 in Xenial + + * Newer guests might use virtio attributes that are clobbered on +migration with the old qemu code. + + [Test Case] + + * Setup two Xenial hosts on ppc64el + + * Create a guest that has a rather new kernel (>=4.14) I'd recommend +Bionic + + * Migrate that guest from Host1 to Host2 + + [Regression Potential] + + * The modification could affect virtio handling in other cases in a non +expected way, but mostly related to migrations. So the expected +regression would be issues to migrate properly. +I verified plenty of migrations in regression testing and we had +this very code in the Yakkety release as we picked 2.6.1 stable release +back then. Due to that it is actually pretty well tested and should not +really regress anything out in the wild. + + [Other Info] + + * So far this only triggers on the confused endian marshalling on +ppc64el, but in theory a different case could trigger it on x86 just as +much. + + --- + Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: In Progress Status in linux package in Ubuntu: Triaged Status in qemu package in Ubuntu: Triaged Bug description: [Impact] * Backport fix from the 2.6.2 stable branch to the qemu 2.5 in Xenial * Newer guests might use virtio attributes that are clobbered on migration with the old qemu code. [Test Case] * Setup two Xenial hosts on ppc64el * Create a guest that has a rather new kernel (>=4.14) I'd recommend Bionic * Migrate that guest from Host1 to Host2 [Regression Potential] * The modification could affect virtio handling in other cases in a non expected way, but mostly related to migrations. So the expected regression would be issues to migrate properly. I verified plenty of migrations in regression testing and we had this very code in the Yakkety release as we picked 2.6.1 stable release back then. Due to that it is actually pretty well tested and should not really regress anything out in the wild. [Other Info] * So far this only triggers on the confused endian marshalling on ppc64el, but in theory a different case could trigger it on x86 just as much. --- Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https:
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
PPA prepared at: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3412 MP: https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/354695 Test of the case on the PPA is successful. But I'll need a regression check on that before going on. ** Changed in: qemu (Ubuntu) Status: Incomplete => Triaged -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: In Progress Status in linux package in Ubuntu: Triaged Status in qemu package in Ubuntu: Triaged Bug description: Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
** Merge proposal linked: https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/354695 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: In Progress Status in linux package in Ubuntu: Triaged Status in qemu package in Ubuntu: Incomplete Bug description: Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
TL;DR: - a KVM guest with the kernel change as identified above - works on Bionic host (kernel 4.15 / qemu 2.11 / libvirt 4.0) - migrating on a Xenial host (kernel 4.4 / qemu 2.5 / libvirt 1.3.1) fails VQ 0 size 0x100 Guest index 0x8101 inconsistent with Host index 0x81: delta 0x8080 error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' - not fixed in latest 4.19 kernel - only failing on ppc64el (not x86) - maybe high/low word related - qemu bisecting found a high/low word related virtio issue and fix in the 2.6 stable series that Note: generated names are odd (hashes are ok), most 4.13 here are actually 4.14 in development. GOOD v4.13 Mon Sep 10 10:03:38 BAD v4.14 Mon Sep 10 10:51:31 Step-1: 15d8ffc9 #1 Mon Sep 10 12:36:30 bad Step-2: bafb0762 #2 Mon Sep 10 13:04:52 good Step-3: b63f6044 #3 Mon Sep 10 13:24:27 bad Step-4: e08af95d #4 Mon Sep 10 13:44:11 bad Step-5: 2a493216 #5 Mon Sep 10 14:25:50 bad Step-6: a248878d #6 Mon Sep 10 14:50:47 bad Step-7: 160e22aa #7 Mon Sep 10 15:09:03 good Step-8: 727f8914 #8 Mon Sep 10 18:30:06 good Step-9: 4a3c67a6 #9 Mon Sep 10 20:37:37 bad Step-10: 04584957 #10 Tue Sep 11 04:35:41 bad Step-11: f7ce9103 #11 Tue Sep 11 05:30:50 bad Step-12: 192f68cf #12 Tue Sep 11 05:49:50 good Step-13: 3f93522f #13 Tue Sep 11 06:13:01 bad Step-14: 4941d472 #14 Tue Sep 11 06:40:05 good Offending change identified as: commit 3f93522ffab2d46a36b57adf324a54e674fc9536 Author: Jason Wang Date: Wed Jul 19 16:54:49 2017 +0800 virtio-net: switch off offloads on demand if possible on XDP set Current XDP implementation wants guest offloads feature to be disabled on device. This is inconvenient and means guest can't benefit from offloads if XDP is not used. This patch tries to address this limitation by disabling the offloads on demand through control guest offloads. Guest offloads will be disabled and enabled on demand on XDP set. Signed-off-by: Jason Wang Signed-off-by: David S. Miller To check if any commit in the latest kernel fixed the issue: 4.19-rc3 as of today (11da3a7f): bad => Not fixed yet as a guest kernel commit. => Also I don't see any further how we could fix hat on the kernel side, despite the issue being introduced there Since we had the report that a Bionic Host would be ok I bumped the test env up one by one. (in order) Libvirt 1.3.1 -> 4.0: still bad kernel 4.4 -> 4.15: still bad qemu 2.5 -> 2.11: working So we are actually looking for a qemu fix for a kernel introduced issue it seems. Via UCA we can access some rather easily. qemu 2.5 (X) bad qemu 2.6.1 (Y) good qemu 2.8 (Z) good qemu 2.10 (A) good qemu 2.11: (B) good So a qemu bisect for 2.5->2.6 it shall be :-/ Back then this was still based on full debian versions so no bisect directly in the packaging repo on these old versions. Using checkinstall and the configure line of the qemu yakkety version (reset machine type to upstream type and linking spapr-rtas.bin [qemu-slof] and others to the expected place). ln -s /usr/share/slof/* /usr/share/qemu/slof.bin ln -s /usr/share/seabios/* /usr/share/qemu/ ln -s /usr/lib/ipxe/qemu/* /usr/share/qemu/ But I realized that 2.6.0 was affected as well. Maybe the fix was part of 2.6.1? I checked the last 2.6.0 publish we had back in Yakkety and it failed as well. So after all the bisect might be much smaller between 2.6.0 and its upstream stable branch. Verified start points with builds from git. qemu-2.6.0-bisect-start: old behavior (bad) qemu-2.6.2-bisect-start: new behavior (good) Eventually came down to: git bisect start # new: [529d45e151d82a772cd9b9af64bb25f88fba6567] Update version for 2.6.2 release git bisect new 529d45e151d82a772cd9b9af64bb25f88fba6567 # old: [bfc766d38e1fae5767d43845c15c79ac8fa6d6af] Update version for v2.6.0 release git bisect old bfc766d38e1fae5767d43845c15c79ac8fa6d6af # new: [ec211e742683d4bc187839b01a4b0056617681a1] atapi: fix halted DMA reset git bisect new ec211e742683d4bc187839b01a4b0056617681a1 # old: [71798fda8b6ef8df47c7640ba0bc24d7060ad307] vmsvga: shadow fifo registers git bisect old 71798fda8b6ef8df47c7640ba0bc24d7060ad307 # old: [909d87d347a7a5e08c32cbdb67bb2927fcefbf34] virtio: set low features early on load git bisect old 909d87d347a7a5e08c32cbdb67bb2927fcefbf34 # new: [28eae0af65dcae887d3cd32212c702ee708c84be] Fix some typos found by codespell git bisect new 28eae0af65dcae887d3cd32212c702ee708c84be # new: [704ab2fce49fa404a61c6dac85003bcc1e3d0192] blockdev: Fix regression with the default naming of throttling groups git bisect new 704ab2fce49fa404a61c6dac85003bcc1e3d0192 # new: [025c4e39f479eb498ee63b634d961a4cf357773e] s390x/ipl: fix reboots for migration from different bios git bisect new 025c4e39f479eb498ee63b634d961a4cf357773e # new: [82c85167791f0057752c2084f8480bf19401f314] Revert "virtio-net: unbreak self announcement and guest offloads after mi
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
2a) is an upstream decision to have better disk integrity guarantees. There is a release notes entry since 17.10 (which is also refrenced from the 18.04 release notes) => https://wiki.ubuntu.com/ArtfulAardvark/ReleaseNotes#qemu_2.10 2b) is I think an upstream change by the PPC devs actually. I'll at least put such a case in my own testing to find it earlier next time and be able to work around if needed. @JFH - if you think 2b is a big issue you might file a bug and abck-sync to IBM PPC Devs for their opinion on it. I'll stick to the initially reported issue here and try to track it down over the next few days as my ppc64 machine time permits (the bisect is slow anyway). -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: In Progress Status in linux package in Ubuntu: Triaged Status in qemu package in Ubuntu: Incomplete Bug description: Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
Note to myself: Combine mainline kernel: git://kernel.ubuntu.com/ubuntu/linux.git Ubuntu kernel: git://kernel.ubuntu.com/ubuntu/ubuntu-bionic.git in one repo. Get git://kernel.ubuntu.com/ubuntu/kteam-tools.git Setup needed chroots (amd64 + arch you need) sudo ./make_chroot bionic amd64 http://archive.ubuntu.com/ubuntu (Then many cleanups depending on build env and needs :-/) Call like: ../kteam-tools/mainline-build/mainline-build-one v4.13 bionic ../kteam-tools/mainline-build/mainline-build-one v4.14 bionic That works with quite some extra setps, but since the changes are not in the Ubuntu extra I ended up switching to building upstream kernels (with Ubuntu kernel config). (... needs more time ...) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: In Progress Status in linux package in Ubuntu: Triaged Status in qemu package in Ubuntu: Incomplete Bug description: Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
Expected closest good/bad as git Builds: Last 4.13 git tag Ubuntu-4.13.0-32.35 - working First 4.14 git tag Ubuntu-4.14.0-11.13 - $ git bisect start $ git bisect good Ubuntu-4.13.0-32.35 $ git bisect bad Ubuntu-4.14.0-11.13 Build is slow, but it seems I can bisect from here ... Arr shortly after this pulled me out of the Ubuntu tree to check the baseline 4.13.0 and that got me a non working kernel. After discussion with the kernel Team I'm switching to bisecting mainline trees. Lets check what surprises this has for me ... -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: In Progress Status in linux package in Ubuntu: Triaged Status in qemu package in Ubuntu: Incomplete Bug description: Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
** Changed in: ubuntu-power-systems Status: Triaged => In Progress -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: In Progress Status in linux package in Ubuntu: Triaged Status in qemu package in Ubuntu: Incomplete Bug description: Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
4.15.0-33-generic Failed, 4.4.0-134-generic we know works. I tested 4.13.0-46-generic and it worked (as expected after the positive 17.10 report in comment #23) checking different builds from there Bionic builds (so even the 4.13.0-32.35 being "before" the tested -46 it is the Bionic build of it) 4.13.0-32.35 - the last 4.13 - working 4.14.0-16.19 - the last 4.14 - failing 4.15.0-9.10 - early 4.15- failing Note good as well as bad cases were the same guest (Bionic). Just installing different kernels in that guest, rebooting into these kernels and then triggering the migration. 4.14.0-11.13 - early 4.14 - failing So it seems 4.13->4.14 is the breaking point, I'll check if I can get a git bisect inside this working ... -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: Triaged Status in linux package in Ubuntu: Triaged Status in qemu package in Ubuntu: Incomplete Bug description: Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
I can also confirm that driving the same case on x86 is not affected. I'd be glad to pull in PPC-Developers in case they know of any virtio bugfix that was done in qemu and/or kernel that might be related. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: Triaged Status in linux package in Ubuntu: Triaged Status in qemu package in Ubuntu: Incomplete Bug description: Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
Reproduced: 2018-09-06T08:25:22.816912Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x8101 inconsistent with Host index 0xfd: delta 0x8004 2018-09-06T08:25:22.816963Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-09-06T08:25:22.817478Z qemu-system-ppc64: load of migration failed: Operation not permitted I need to break this out of the test automation and then in the Bionic guest set up different kernels to find which mismatch it is. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: Triaged Status in linux package in Ubuntu: Triaged Status in qemu package in Ubuntu: Incomplete Bug description: Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
Last comments synced to launchpad indicated that migration of 18.04 guests, using 16.04 hypervisor is still not working. And it would be nice, if it did. So to me it looks like the bug is still present, even if you want to close this ticket on your side. ps. last comment we have synced prior to today's was from xxdold on 6th of august, just in case there is any information lost in-between. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: Triaged Status in linux package in Ubuntu: Triaged Status in qemu package in Ubuntu: Incomplete Bug description: Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
** Changed in: linux (Ubuntu) Status: New => Triaged ** Tags added: kernel-da-key -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: Triaged Status in linux package in Ubuntu: Triaged Status in qemu package in Ubuntu: Incomplete Bug description: Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
** Changed in: ubuntu-power-systems Assignee: Canonical Server Team (canonical-server) => Canonical Kernel Team (canonical-kernel-team) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: Triaged Status in linux package in Ubuntu: New Status in qemu package in Ubuntu: Incomplete Bug description: Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
** Changed in: ubuntu-power-systems Status: New => Incomplete ** Changed in: qemu (Ubuntu) Importance: Undecided => High ** Changed in: linux (Ubuntu) Importance: Undecided => High ** Changed in: linux (Ubuntu) Assignee: (unassigned) => Canonical Kernel Team (canonical-kernel-team) ** Changed in: ubuntu-power-systems Status: Incomplete => Triaged -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: Triaged Status in linux package in Ubuntu: New Status in qemu package in Ubuntu: Incomplete Bug description: Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
Yes, 16.10, 17.04, 17.10 releases are end of life, but they should still be accessible via http://old-releases.ubuntu.com/ubuntu (16.10, 17.04) or regular archive mirrors (17.10, but it will at some point move to old-releases). So you could deploy 16.04 and dist-upgrade via old-releases.ubuntu.com to yakkety, zesty and via regular mirror to artful. Obviously for testing / bisecting purposes only. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: New Status in linux package in Ubuntu: New Status in qemu package in Ubuntu: Incomplete Bug description: Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
I got one P8 machine working but no second one that I could reach from there to test the migratio. My usual tricks around that with lxd containers didn't work atm so I rely on people with more P8 HW. I asked a few people to poll on P8 Devs to take a look as well. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: New Status in linux package in Ubuntu: New Status in qemu package in Ubuntu: Incomplete Bug description: Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
Thanks, that makes sense. So it really seems to be the newer virtio drivers in the guest that triggers it - which is why the move to the HWE kernel For test systems, I tried to grab a p8 but it failed to install three times in a row. Not sure what is broken atm, so at least for a while I'll rely on you doing the tests. I totally understand that it is hard (and not recommended by me) to do all the tests with the customer VMs. If you can do the tests still but have only a short time, just go as far as you can which means just to queens and skip the others in between. >From there we can still decide if it is helpful to check the interim versions. I'll continue trying to get a P8 up with this - maybe I can try one of the new P9 systems as well tomorrow. Just to check - Is the power system you have a P8/P9/P? ? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: New Status in linux package in Ubuntu: New Status in qemu package in Ubuntu: Incomplete Bug description: Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
I did not suggest to install the Cloud Archive code in the 18.04 Guest. You said your host is 16.04 (Xenial) and there the different UCA's should be tried to bring new qemu/libvirt code to your host and due to that check if these newer versions already have a fix we would look for instead of debugging from scratch. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: New Status in linux package in Ubuntu: New Status in qemu package in Ubuntu: Incomplete Bug description: Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
Ok, reading that your guests are customer systems I assume you'd need to setup a test system somewhere to confirm if different host qemu/libvirt/kernel versions would fix it. The list of interesting checks: 16.04 Host as-is + 18.04 Guest + qemu from UCA Ocata 16.04 Host as-is + 18.04 Guest + qemu from UCA Pike 16.04 Host as-is + 18.04 Guest + qemu from UCA Queens 16.04 Host as-is + 16.04 Guest running the HWE 4.15 Kernel + Qemu as-is in 16.04 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: New Status in linux package in Ubuntu: New Status in qemu package in Ubuntu: Incomplete Bug description: Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
Also is this a thing that was triggered once, or reproducible every time you run it. In the latter case it might be very hard to track down. I checked and as part of the qemu verification I do run&migrate older guests in newer hipervisors. I do not yet do vice versa in a lot of tests, so even if this issue ends up non reproducible I might add that to our regular test set to be executed more often. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: New Status in linux package in Ubuntu: New Status in qemu package in Ubuntu: Incomplete Bug description: Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
Of the listed guests I assume Ubuntu 18.04 has the newest guest Kernel (read virtio drivers). Several questions to ensure this is right: 1. could you report the kernel version of each of your test guests (assuming 4.4 and 4.15 for Ubuntu, but the exact version would be great to know) 2. it would be great to know if one of the newer qemu versions that already exist fix the virtio handling of this case. So once you recreated your case could you upgrade through the Ubuntu Cloud Archive provided versions and let us know if some of them work (and if so which ones exactly). That might help a lot to track down a potentially existing virtio fix. Just one by one upgrade the newer versions from [1] on source and target host. Then shutdown and re-start your guest on the migration source and then migrate again. 3. #2 above changed qemu, the other element we might change is that you use the 4.15 based HWE kernel in a 16.04 guest - would that then fail just as the 18.04 guest does? In general, I wonder what on this should be Power specific since it should after all use the same virtio drivers and host code. I have seen other similar reports which always ended up as host/guest mismatch of virtio-handling - but never an arch specific one yet. [1]: https://wiki.ubuntu.com/OpenStack/CloudArchive ** Changed in: qemu (Ubuntu) Status: New => Incomplete -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: New Status in linux package in Ubuntu: New Status in qemu package in Ubuntu: Incomplete Bug description: Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1783140] Re: KVM live migration fails
This issue is either in 16.04 qemu (not able to handle the new guests) or in 18.04s virtio drivers, so I'm adding a linux task for the kernel first. ** Also affects: linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails Status in The Ubuntu-power-systems project: New Status in linux package in Ubuntu: New Status in qemu package in Ubuntu: New Bug description: Environment: 2 POWER8 with Ubuntu 16.04.4 LTS as KVM hypervisor. 1 KVM guest with Ubuntu 18.04 LTS. Virtual disk for the guest is a qcow2 file on an NFS share, accessible from both hypervisors, so live migration is possible and works for all other guests (SLES, RHEL, Ubunutu 16.04), Live migratino of Ubuntu 18.04 guest fails on ppc, while the same test on an x86_64 environment suceeds. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-23T11:12:25.586385Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x38aa inconsistent with Host index 0xa980: delta 0x8f2a 2018-07-23T11:12:25.586434Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-23T11:12:25.587246Z qemu-system-ppc64: load of migration failed: Operation not permitted root@pkvm2:~# uname -a Linux pkvm2 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:51:21 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
