[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

Thanks Dave -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1647389 Title: Regression: Live migrations can still crash after CVE-2016-5403 fix To manage notifications about this bug go to:

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

Yes comment #46 shows it was pushed to xenial, and I checked that it is currently in updates. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1647389 Title: Regression: Live migrations can still

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

Is this fix pushed to Xenial already ? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1647389 Title: Regression: Live migrations can still crash after CVE-2016-5403 fix To manage notifications

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

This bug was fixed in the package qemu - 2.0.0+dfsg-2ubuntu1.33 --- qemu (2.0.0+dfsg-2ubuntu1.33) trusty-security; urgency=medium * SECURITY UPDATE: DoS via 6300esb unplug operations - debian/patches/CVE-2016-10155.patch: add exit function in hw/watchdog/wdt_i6300esb.c.

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

This bug was fixed in the package qemu - 1:2.5+dfsg-5ubuntu10.11 --- qemu (1:2.5+dfsg-5ubuntu10.11) xenial-security; urgency=medium * SECURITY UPDATE: DoS in virtio GPU device - debian/patches/CVE-2016-10028.patch: check virgl capabilities max_size in

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

I tested with 10.11 windows 7, 2008, 2012. Live migration works more than twice. Thank you. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1647389 Title: Regression: Live migrations can still crash

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

I find that running the below packages does not have this issue: # dpkg -l | grep -i qemu ii ipxe-qemu1.0.0+git-20150424.a25a16d-1ubuntu1 all PXE boot firmware - ROM images for qemu ii qemu-block-extra:amd64 1:2.5+dfsg-5ubuntu10.5

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

@Marc I reviewed your proposed changes, and I really feel you should log an error in the negative case. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1647389 Title: Regression: Live migrations can

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

looking forward to test this! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1647389 Title: Regression: Live migrations can still crash after CVE-2016-5403 fix To manage notifications about this

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

There are some _untested_ qemu packages that work around this issue in the security team PPA: https://launchpad.net/~ubuntu-security- proposed/+archive/ubuntu/ppa/+packages They will be released as security updates once they've been through QA. Possibly in a couple of weeks. -- You received

RE: [Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

...@canonical.com [mailto:boun...@canonical.com] On Behalf Of Dave Chiluk Sent: Friday, March 31, 2017 7:47 PM To: Len White <lwh...@coreitx.com> Subject: [Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix I just tested removing CVE-2016-5403-3.patch, but that

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

Yes, whenever the bug gets triggered it's because it throws the value into the negative. However without the patch that's not what happens. Let's take this for example: 2016-12-05T14:41:07.903932Z qemu-system-x86_64: VQ 2 size 0x80 < last_avail_idx 0x9 - used_idx 0xa Without patch: 0x9 - 0xA =

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

I just tested removing CVE-2016-5403-3.patch, but that didn't seem to do it. I still don't understand how upsteam qemu functions with the calculation the way it is. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

Re: [Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

Definitely​ im waiting anxiously about this to be resolved also! On Mar 31, 2017 20:35, "Marc Deslauriers" wrote: I had a feeling perhaps CVE-2016-5403-3.patch needed to be backed out, that's the commit I mentioned in comment #13. Anxiously awaiting results of

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

I had a feeling perhaps CVE-2016-5403-3.patch needed to be backed out, that's the commit I mentioned in comment #13. Anxiously awaiting results of the test... :) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

So I tested Len's patch, and it does seem to work. However, I can't seem to understand why the below line is necessary, when upstream qemu has virtually identical code, and does not need this line. It almost makes me wonder if CVE-2016-5403-3.patch is incorrectly decrementing the inuse counter

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

It's from this package: http://vault.centos.org/centos/7/virt/Source /kvm-common/qemu-kvm-ev-2.6.0-28.el7_3.6.1.src.rpm With this applied from qemu-git: https://github.com/qemu/qemu/commit/e66bcc408146730958d1a840bda85d7ad51e0cd7.patch Then the patch I posted here on top of that and recompiled

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

** Changed in: qemu (Ubuntu Xenial) Status: New => Confirmed ** Changed in: qemu (Ubuntu Xenial) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1647389 Title:

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

@Len Can you provide links to the Rhel sources that you based your patch on in order to provide more context and provide appropriate attribution in the Ubuntu patch. Thanks, Dave. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

** Also affects: qemu (Ubuntu Xenial) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1647389 Title: Regression: Live migrations can still crash after

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

The attachment "qemu.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team. [This is an automated message performed by a Launchpad user owned by ~brian-murray,

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

Hi, I've figured out the actual problem, and I've made a patch that fixes the issue, not sure if it will apply cleanly as mine is based off the rhel version but thought I'd share it since it gave me a headache and many others it seems. This is what happens vdev->vq[i].inuse =

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

Updated bug description with SRU template and test case so that the testcase can be updated as need be. ** Description changed: + [Impact] + + * Libvirt migrations using tunnelled libvirt cause a failure to migrate + on the destination with error VQ 2 size 0x80 < last_avail_idx 0x9 - +

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

@Dave let me try again and take a look at the logs you pointed, and get back to you. if you have any news for me, let me know. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1647389 Title:

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

@Alehandro, are your redhat/centos installations on qemu 2.6+? They are working because the fix 4eae2a6+ the CVE is already available in that version of qemu. IMHO, redhat and centos got lucky with that version of QEMU. It has nothing to do with quality of distribution, and everything to do

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

Dave,hi. Thanks for working on this. the problems y described previously are working fine on redhat & centos regarding the live_migration_tunnelled = false, under the libvirt section. so, i still think what i'm seeing is the ubuntu bug on migration. please let me know as soon as you find

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

I have been able to create a smaller recreation environment for this. 1. Create a VM on shared storage solution. In my case NFS. 2. set start_libvirtd="yes" in /etc/default/libvirt-bin 3. systemctl restart libvirt-bin 4. virsh -c qemu+ssh://${FROM}/system migrate --live --p2p --tunnelled ${VM}

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

@Alejandro. When trying with the live_migration_tunnelled = false, what are you seeing in the /var/log/libvirt/qemu/.log. You may be seeing a different issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

Sorry for my mistake. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1647389 Title: Regression: Live migrations can still crash after CVE-2016-5403 fix To manage notifications about this bug go to:

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

This bug DOES NOT fixed in Xenial QEMU package. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1647389 Title: Regression: Live migrations can still crash after CVE-2016-5403 fix To manage

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

** Changed in: qemu (Ubuntu) Status: Fix Released => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1647389 Title: Regression: Live migrations can still crash after CVE-2016-5403

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

** Changed in: qemu (Ubuntu) Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1647389 Title: Regression: Live migrations can still crash after CVE-2016-5403

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

@chiluk I can confirm that the flag provided doesnt work on ubuntu 16.04 with mitaka packages. The first live migration to a compute node works, but if i try immediately to live migrate it back to the same compute node, or to other compute node, the migration fails sometimes geting on the new

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

Yes, I'm currently working through attempting to bisect this issue. Unfortunately I'm running into lots of issues getting iterations and bounds functioning in a manner that allows me to reliably reproduce the issue. -- You received this bug notification because you are a member of Ubuntu Bugs,

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

@chiluk Let me try that out to see if everything works as expected. In the mean time, if you please clan update about a final resolution from the "package" side, would be amazing. best. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

We have found another workaround for this in openstack clouds. Somehow this issue seems to be exacerbated by live_migration_tunnelled being on. You may be able to work around this by setting live_migration_tunnelled = false In your nova.conf or nova-compute.conf. This is set by default for

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

@chiluk unfortunatelly i dont have the necessary skills to find the code solution, all i can say i that on qemu packages on RHEL Openstack Platform version 9 this is fixed. So it came into my attention that RH being a contributor to the qemu/KVM project did'nt pushed the solution. Sadly, from

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

@alejandro-f You need to run the virsh commands on the compute hosts before doing the migration, customers should not be running the virsh commands. Unfortunately, this looks to still exist upstream according to https://lists.gnu.org/archive/html/qemu-devel/2016-12/msg02066.html, and is related

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

Any news other than workarounds ? It's kinda unprofessional to ask for customers to do that before migrating their instances. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1647389 Title:

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

I have confirmation that this can be worked around by turning off memory statistics via virsh before the migration. After migration memory statistics can be turned back on safely. The command to turn off memory statistics is virsh dommemstat --live --period 0 The command to turn on memory

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

Guys, this is HUGELY CRITICAL on 16.04 production openstack with windows guests, since the only workaround is to tell the customer to disable the baloon driver inside the windows machine before migrating, but when migration occurs because of a failure and the customer cant disable it, a simple

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

I think it has more to do with the first two segments of that commit, that looks like it's handling the "Windows balloon driver sends memory stats only if the balloon service (blnsvr.exe) is running" issue. My other suspicion is perhaps 58a83c61496eeb0d31571a07a51bc1947e3379ac needs to be backed

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

Marc we came to a similar conclusion. My backport of 104e70cae included a partial backport of 4eae2a657d1ff5ada56eb9b4966e. The rest of 4eae2 didn't apply. I was curious if perhaps the VirtQueueElement isn't being properly initailized or possibly has some dirty data, but I haven't figured that

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

So perhaps we need this: http://git.qemu.org/?p=qemu.git;a=commit;h=4eae2a657d1ff5ada56eb9b4966eae0eff333b0b according to the explanation here: https://lists.gnu.org/archive/html/qemu-devel/2016-11/msg02830.html -- You received this bug notification because you are a member of Ubuntu Bugs,

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

Yes, is only with windows guests. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1647389 Title: Regression: Live migrations can still crash after CVE-2016-5403 fix To manage notifications about

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

Is this happening with Windows guests? Were the Windows guests created with an earlier version of the qemu package? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1647389 Title: Regression: Live

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

I have confirmation from a user that 104e70cae does not resolve the issue. ** Attachment removed: "First attempt at backport for 104e70cae" https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1647389/+attachment/4795470/+files/lp1647389.xenial.debdiff -- You received this bug notification

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

I have created a ppa with the above fix that is available https://launchpad.net/~chiluk/+archive/ubuntu/lp1647389 ppa:chiluk/lp1647389 If someone on copy for this bug has a way reliable way to test this issue please attempt using the qemu out of my ppa. Thank you, -- You received this bug

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

At the moment this appears to be a consequence of 104e70cae78bd4afd95d948c6aff188f10508a9c not being included in the original CVE patchset. I'm attaching an early debdiff for that includes a first attempt at a backport of the above patch and am requesting comments and code review. If anyone has

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

** Tags added: sts -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1647389 Title: Regression: Live migrations can still crash after CVE-2016-5403 fix To manage notifications about this bug go to:

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

13:39 mdeslaur: are you aware of bug 1647389? 13:40 Claimed second regression from bug 1612089 AFAICT. 13:51 rbasak: I saw it, I haven't investigated yet 13:51 rbasak: We already have all of the commits that are linked 13:52 rbasak: I'll look at it more after I'm back from holiday 13:52

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

** Tags added: regression-update ** Changed in: qemu (Ubuntu) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1647389 Title: Regression: Live migrations can still

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

I have seen an issue that seems it could be related: ec 5 17:52:53 os-nova-compute libvirtd[40142]: failed to connect to monitor socket: No such process Dec 5 17:52:53 os-nova-compute virtlogd[6936]: End of file while reading data: Input/output error Dec 5 17:52:54 os-nova-compute

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: qemu (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1647389 Title:

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

See also the thread at https://lists.gnu.org/archive/html/qemu- devel/2016-11/msg02634.html which appears to be about the same issue, and references two commits that might fix this if cherry-picked. -- You received this bug notification because you are a member of Ubuntu Bugs, which is

[Bug 1647389] Re: Regression: Live migrations can still crash after CVE-2016-5403 fix

This is with: $ dpkg-query -W qemu-system-x86 qemu-system-x86 1:2.5+dfsg-5ubuntu10.6 $ qemu-system-x86_64 --version QEMU emulator version 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.6), Copyright (c) 2003-2008 Fabrice Bellard -- You received this bug notification because you are a member of Ubuntu