[Bug 1783140] Comment bridged from LTC Bugzilla
--- Comment From heinz-werner_se...@de.ibm.com 2018-10-17 04:03 EDT--- IBM Bugzilla status -> closed, fixed released by Canonical -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1783140] Comment bridged from LTC Bugzilla
--- Comment From xxd...@de.ibm.com 2018-10-05 03:46 EDT--- Hello Robie, Because I migrated both my KVM hypervisors to 18.04, I cannot test it anymore. But Christian Ehrhardt could reproduce the problem. Hopefully he has still the appropriate test envirnoment. Regards, Andreas (bugproxy) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1783140] Comment bridged from LTC Bugzilla
--- Comment From xxd...@de.ibm.com 2018-09-07 08:38 EDT---
I have some more bad surprises:
On Monday I upgraded my x86 KVM hypervisors from 16.04.5 to 18.04.1.
No problems at all.
On Wednesday I upgraded the ppc KVM hypervisors from 16.04.5 to 18.04.1.
Problem 1:
In the middle of the upgrade process I could not live migrate the guests from
the 16.04 hypervisor to the 18.04 hypervisor.
None of the 13 guests!
root@pkvm2:~# virsh migrate --persistent --live pkut04 qemu+ssh://pkvm1/system
error: internal error: process exited while connecting to monitor:
2018-09-05T11:07:58.260851Z qemu-system-ppc64: warning: CPU(s) not present in
any NUMA nodes: CPU 1 [core-id: 1]
2018-09-05T11:07:58.260859Z qemu-system-ppc64: warning: All CPU(s) up to
maxcpus should be described in NUMA config, ability to start up with partial
NUMA mappings is obsoleted and will be removed in future
2018-09-05T11:07:58.262038Z qemu-system-ppc64: This machine version does not
support CPU hotplug
So I had to shutdown all the guests to do the upgrade of the second
hypervisor !!!
Probem 2:
When the second hypervisor was on 18.04.1 I could not start most of the guests.
Only 4 of 13 guests started.
(a) some qcow2 disks have been marked as sharable
worked on Ubuntu 16.04, but noot on 18.04
(b) vcpu definition
on Ubuntu 16.04 160 worked
on Ubuntu 18.04 this does not work on ppc ("This machine version does not
support CPU hotplug").
I had to cahnge it to 8
I could resolve 2a and 2b. But it is frustrating to get such additional
adventure games in the maintenance window.
You think "just start the guests, then I can go home", and then the guests do
not start.
And it is even more frustrating when you did just the same task 2 days ago on
x86 without any problems.
Maybe Problem 1 has the same reason as Problem 2. In other words: With the
changed domain XML, maybe a live migration from Ubuntu 16.04 hypervisor to
18.04 hypervisor would work.
But I cannot verify this assumption. Now both my hypervisors are finally on
18.04.1 and live migration between them works for all guests.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1783140
Title:
KVM live migration fails
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1783140] Comment bridged from LTC Bugzilla
--- Comment From xxd...@de.ibm.com 2018-08-22 04:41 EDT--- sorry "August 5" means September 5 ;-) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1783140] Comment bridged from LTC Bugzilla
--- Comment From xxd...@de.ibm.com 2018-08-22 04:36 EDT--- Yes, the problem is not solved. Most of the actions have been tests on my side. So I could finally test with SLES15 and Ubuntu 17.10 guests, both migrating successful. But 18.04 does not migrate. This means there was a change between kernel 4.13.0-46 and 4.15.0-23 which introduced the problem on ppc but not on x86 (nobody ever tested z). If someone wants to provide a fix, I can test it in our environment, but only before August 5. On August 5 we will upgrade our hypervisors to 18.04 and then we can hopefully migrate all guests again ... at least until to the next newer guest that makes problems. So I would appreciate if someone can catch that problem before we run into it again. Maybe someone also wants to check if the problem also exists on z. And maybe it's also worth to think about extending the test suites to also test such cases with guests newer than the hypervisors (if you think it's a valid scenario in the field). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1783140] Comment bridged from LTC Bugzilla
--- Comment From heinz-werner_se...@de.ibm.com 2018-08-21 09:46 EDT--- @xnox, I will leave this ticket open till a final soluition is available. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1783140] Comment bridged from LTC Bugzilla
--- Comment From heinz-werner_se...@de.ibm.com 2018-08-21 07:22 EDT--- @Canonical, can this LP be closed? I don't see any addl. activities here.. Many thx in advance -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1783140] Comment bridged from LTC Bugzilla
--- Comment From xxd...@de.ibm.com 2018-08-06 07:08 EDT--- Today I had my final test with an Ubuntu 17.10 guest installed from http://old-releases.ubuntu.com/releases/17.10/ubuntu-17.10-server-ppc64el.iso After installation it had kernel 4.13.0-16-generic and live migration was successful. After an update+upgrade it had kernel 4.13.0-46-generic and live migration was still successful. So the live migration problem on ppc was introduced between kernel 4.13.0-46-generic (Ubuntu 17.10) and kernel 4.15.0-23-generic (Ubuntu 18.04). Let me know if there is anything else I can do to help solving this issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1783140] Comment bridged from LTC Bugzilla
--- Comment From xxd...@de.ibm.com 2018-08-01 07:47 EDT--- 16.10 test: I have seen 16.10 in our foreman, so I thought I could do that test quickly. But it looks like the 16.10 mirrors are already down because 16.10 is out of service :-( That means those tests (16.10, 17.04, 17.10) would take more time. I would have to download the ISO files and do manual installations from ISO files. Our plan for the hypervisors is to upgrade them to 18.04.1 on September 5. Until then I could do some guest tests if they help finding the problem. And in case a fix becomes available, I could verify it. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1783140] Comment bridged from LTC Bugzilla
--- Comment From xxd...@de.ibm.com 2018-08-01 05:30 EDT--- I just had a new VM with SLES 15, kernel 4.12.14-23-default. Migration succeeded ! I can also do a test with Ubuntu 16.10 after lunch. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1783140] Comment bridged from LTC Bugzilla
--- Comment From xxd...@de.ibm.com 2018-08-01 05:11 EDT--- Finally I could have a look on all the other VMs that migrate successful: RHEL 7U1, kernel 3.10.0-229.20.1.ael7b.ppc64le RHEL 7U3, kernel 3.10.0-514.6.1.el7.ppc64le RHEL 7U3, kernel 3.10.0-862.3.2.el7.ppc64 SLES 12SP1, kernel 3.12.49-11-default SLES 12SP2, kernel 4.4.21-69-default Ubuntu 16.04.1 LTS, kernel 4.4.0-31-generic So Ubuntu 18.04 was the first with a kernel really newer than that of the hypervisor (the SLES 12SP2 kernel is only slightly newer, but still a 4.4) I can also do some tests with RHEL 7U5, SLES 12SP3, SLES 15 and Ubuntu 16.10. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1783140] Comment bridged from LTC Bugzilla
> If there are incompatibilities between kernel 4.4 and 4.15, would I maybe > risk that then I cannot migrate 16.04 guests any longer? Did anyone tests > this case? > This way around (old guest/ new host) I cover migration tests before any qemu/libvirt upload testin x86/s390/ppc64el (and a tiny bit of arm64). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1783140] Comment bridged from LTC Bugzilla
--- Comment From xxd...@de.ibm.com 2018-08-01 04:48 EDT--- My Systems are P8 (8247-22L). No problem to do some more tests with KVM guests (working Mon, Wed and Fri). We also plan to update the KVM hosts to 18.04.1, but have no fixed date for that. If there are incompatibilities between kernel 4.4 and 4.15, would I maybe risk that then I cannot migrate 16.04 guests any longer? Did anyone tests this case? The other bottomside of the upgrade would be that I cannot help any longer with tests on 16.04 hypervisors. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1783140] Comment bridged from LTC Bugzilla
--- Comment From xxd...@de.ibm.com 2018-07-30 10:24 EDT--- The host (KVM hypervisor) is 16.04. So you suggest to install newer UCA qemu on the hypervisors. That is something I have to decline today. Those hosts are runnung some more VMs. So to update the hypervisors I need a service window aggreed with my customers. And I have to stay on a kind of supported mainstream level, no experimental stuff. Until then I can do some tests with some guest VMs. Or someone other has a test environment where he/she can play also with the hypervisors. As I mentioned: the problem is reproducible ! -- So now I did another test with a 16.04 guest. The problem gets worse, but maybe it helps in catching the bug. I did a new installation of a VM with Ubuntu 16.04.5 LTS, kernel 4.4.0-131-generic #157-Ubuntu SMP. Live migration succeeded. Then I installed linux-generic-hwe-16.04. The system booted with kernel 4.15.0-29-generic #31~16.04.1-Ubuntu SMP. And live migration failed: # virsh migrate --persistent --live p8lnxtst1 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-30T14:13:34.381447Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x302 inconsistent with Host index 0x16c: delta 0x196 2018-07-30T14:13:34.381496Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-30T14:13:34.381806Z qemu-system-ppc64: load of migration failed: Operation not permitted It is still very reproducible! This means the new hwe kernel introduced the problem !!! Or it is just not compatible with 4.4.0-130-generic of the KVM hypervisor. BTW, no entry in the /var/log/libvirt/qemu log files regarding the migration attempts. Any other log or trace files I could look for or activate? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1783140] Comment bridged from LTC Bugzilla
--- Comment From xxd...@de.ibm.com 2018-07-30 09:35 EDT--- The host (KVM hypervisor) is 16.04. So you suggest to install newer UCA qemu on the hypervisors. That is something I have to decline today. Those hosts are runnung some more VMs. So to update the hypervisors I need a service window aggreed with my customers. And I have to stay on a kind of supported mainstream level, no experimental stuff. Until then I can do some tests with some guest VMs. Or someone other has a test environment where he/she can play also with the hypervisors. As I mentioned: the problem is reproducible ! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1783140] Comment bridged from LTC Bugzilla
--- Comment From xxd...@de.ibm.com 2018-07-30 08:24 EDT--- # add-apt-repository cloud-archive:ocata cloud-archive for Ocata only supported on xenial # add-apt-repository cloud-archive:pike cloud-archive for Pike only supported on xenial # add-apt-repository cloud-archive:queens cloud-archive for Queens only supported on xenial In other words: I cannot use those OCAs on 18.04. I know migration was successful with 16.04, but I do not know the used kernel. So i will now do the test with 16.04 again, with different kernels if possible. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1783140] Comment bridged from LTC Bugzilla
--- Comment From xxd...@de.ibm.com 2018-07-30 07:10 EDT--- regarding reproducibility: yes it is absolutely reproducible In the meantime I updated the 18.04 VM to 18.04.1. Live migration still fails. root@pkvm2:~# virsh migrate --persistent --live p8lnxtst4 qemu+ssh://pkvm1/system error: internal error: early end of file from monitor, possible problem: 2018-07-30T11:06:47.622840Z qemu-system-ppc64: VQ 0 size 0x100 Guest index 0x8402 inconsistent with Host index 0x19f: delta 0x8263 2018-07-30T11:06:47.622897Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'pci@8002000:01.0/virtio-net' 2018-07-30T11:06:47.623487Z qemu-system-ppc64: load of migration failed: Operation not permitted -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1783140] Comment bridged from LTC Bugzilla
--- Comment From xxd...@de.ibm.com 2018-07-30 06:53 EDT--- Here ist the information I can give immediately: (a) Ubuntu 16.04.4 KVM hypervisor: kernel 4.4.0-130-generic # virsh version Compiled against library: libvirt 1.3.1 Using library: libvirt 1.3.1 Using API: QEMU 1.3.1 Running hypervisor: QEMU 2.5.0 (b) Ubuntu 18.04 VM: kernel 4.15.0-23-generic I will now do an upgrade (to 18.04.1 ?). But from what I can read about 18.04.1, it does not include a new HWE. Anyway I will test it. other KVM guests (Ubuntu 16.04.x, RHEL, SLES): I have to check. Most of them are customer systems, so I don't have a login for all of them, but I think I can get access .. just needs some time. Regarding the platform question: 18.04 guest on 16.04.4 Hypervisor on x86_64: live migration works 18.04 guest on 16.04.4 Hypervisor on ppc64le: live migration fails -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1783140 Title: KVM live migration fails To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1783140/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
