On Fri, Jan 18, 2019 at 8:44 AM Daniel P. Berrangé <berra...@redhat.com> wrote:
> On Fri, Jan 18, 2019 at 01:57:31PM +0100, Paolo Bonzini wrote: > > On 18/01/19 11:21, Daniel P. Berrangé wrote: > > > Yes, this is exactly why I said we should make the migration blocker > > > be conditional on any L2 guest having been started. I vaguely recall > > > someone saying there wasn't any way to detect this situation from > > > QEMU though ? > > You can check that and give a warning (check that CR4.VMXE=1 but no > > other live migration state was transferred). However, without live > > migration support in the kernel and in QEMU you cannot start VMs *for > > the entire future life of the VM* after a live migration. So even if we > > implemented that kind of blocker, it would fail even if no VM has been > > started, as long as the kvm_intel module is loaded on migration. That > > would be no different in practice from what we have now. > Ahh, I was mis-understanding that it only applied to L2 VMs that > existed at the time the migration is performed. Given that it breaks > all future possibility of launching an L2 VM, this strict blocker does > make more sense. > To explain my use case more fully: Right now all guests are on Linux 4.14.79+ hypervisors, with Qemu 2.12.1+. I understand the value of this feature, and I need to get all the guests to Linux 4.19.16+ hypervisors, with Qemu 3.2 (once it is available). As documented, and as best as I can read from the source code and this mailing list, the recommended solution would be for me to upgrade Linux and Qemu on existing hypervisors, and then restart the entire environment. After it comes up with the new kernel, and the new qemu, everything will be "correct". This model will not be well received by users. We have long running operations of all sorts, and restarting them all at the same time is a very big problem. I would like to heal this system over time. The first stage will include introducing new Linux 4.19.16+ hypervisors, and migrating the guests to these machines carefully and opportunistically. Carefully means that machines that will use L2 guests will need to be restarted in discussion with the users(or their hypervisors avoided from this exercise), but most (80%+) of machines that will never launch an L2 guest can migrate live with low risk (at least according to our experience to date). This will allow existing hypervisors to be freed up so that they too can be upgraded to Linux 4.19.16+. The second stage will include upgrading to Qemu 3.2 once it is available and demonstrated to be stable for our use cases. However, I will need to be able to live migrate most (80%+) systems from Qemu 2.12.1+ to Qemu 3.2. We would again handle the machines with L2 guests with care. If Qemu 3.2 will be ready sooner (weeks?), I would wait before migration, and combine the above two steps such the new hypervisors would have both Linux 4.19.16+ and Qemu 3.2. But, if Qemu 3.2 is months away, I would keep it as two steps. A large number of systems would be shut down and replaced as part of normal processes, which would automatically heal these. The long running machines would not be healed, but they would be no worse than they are today. To achieve this, I need a path to live migrate from Qemu 2.12.1+ with VMX bit set in the guest, to Qemu 3.2. Further complicating things is that we are using OpenStack, so options for tweaking flags on a case by case basis would be limited or non-existent. I'm totally fine with the understanding that any machine not restarted is still broken under Qemu 3.2, just as it was broken under Qemu 2.12. New machines will be correct, and the broken machines can be fixed opportunistically and in discussion with the users. And, we don't need a system-wide restart of the whole cluster to deploy Qemu 3.2. -- Mark Mielke <mark.mie...@gmail.com>
