Re: [Openstack-operators] mitaka/xenial libvirt issues

2017-11-27 Thread Joe Topjian
We think we've pinned the qemu errors down to a mismatched group ID on a handful of compute nodes. The slow systemd/libvirt is still unsolved, but at the moment that does not actually be the cause of the qemu errors. On Mon, Nov 27, 2017 at 8:04 AM, Joe Topjian wrote: > Hi all, > > To my knowle

Re: [Openstack-operators] mitaka/xenial libvirt issues

2017-11-27 Thread Joe Topjian
Hi all, To my knowledge, we don't use tunneled migrations. This issue is also happening with snapshots, so it's not restricted to just migrations. I haven't yet tried the apparmor patches that George mentioned. I plan on applying them once I get another report of a problematic instance. Thank yo

Re: [Openstack-operators] mitaka/xenial libvirt issues

2017-11-27 Thread Tobias Urdin
Hello, The seems to assume tunnelled migrations, the live_migration_flag is removed in later version but is there in Mitaka. Do you have the VIR_MIGRATE_TUNNELLED flag set for [libvirt]live_migration_flag in nova.conf? Might be a long shot, but I've removed VIR_MIGRATE_TUNNELLED in our clouds

Re: [Openstack-operators] mitaka/xenial libvirt issues

2017-11-26 Thread Sean Redmond
Hi, I think it maybe related to this: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1647389 Thanks On Thu, Nov 23, 2017 at 6:20 PM, Joe Topjian wrote: > OK, thanks. We'll definitely look at downgrading in a test environment. > > To add some further info to this problem, here are some lo

Re: [Openstack-operators] mitaka/xenial libvirt issues

2017-11-23 Thread Joe Topjian
OK, thanks. We'll definitely look at downgrading in a test environment. To add some further info to this problem, here are some log entries. When an instance fails to snapshot or fails to migrate, we see: libvirtd[27939]: Cannot start job (modify, none) for domain instance-4fe4; current job i

Re: [Openstack-operators] mitaka/xenial libvirt issues

2017-11-23 Thread Chris Sarginson
I think we may have pinned libvirt-bin as well, (1.3.1), but I can't guarantee that, sorry - I would suggest its worth trying pinning both initially. Chris On Thu, 23 Nov 2017 at 17:42 Joe Topjian wrote: > Hi Chris, > > Thanks - we will definitely look into this. To confirm: did you also > down

Re: [Openstack-operators] mitaka/xenial libvirt issues

2017-11-23 Thread Joe Topjian
Hi Chris, Thanks - we will definitely look into this. To confirm: did you also downgrade libvirt as well or was it all qemu? Thanks, Joe On Thu, Nov 23, 2017 at 9:16 AM, Chris Sarginson wrote: > We hit the same issue a while back (I suspect), which we seemed to resolve > by pinning QEMU and re

Re: [Openstack-operators] mitaka/xenial libvirt issues

2017-11-23 Thread Chris Sarginson
We hit the same issue a while back (I suspect), which we seemed to resolve by pinning QEMU and related packages at the following version (you might need to hunt down the debs manually): 1:2.5+dfsg-5ubuntu10.5 I'm certain there's a launchpad bug for Ubuntu qemu regarding this, but don't have it to

[Openstack-operators] mitaka/xenial libvirt issues

2017-11-23 Thread Joe Topjian
Hi all, We're seeing some strange libvirt issues in an Ubuntu 16.04 environment. It's running Mitaka, but I don't think this is a problem with OpenStack itself. We're in the process of upgrading this environment from Ubuntu 14.04 with the Mitaka cloud archive to 16.04. Instances are being live mi