[Yahoo-eng-team] [Bug 1730918] [NEW] Nova does not respect default_schedule_zone `None`

[Ondrej Vasko]

Public bug reported:

Hello,

DESCRIPTION
---

I want Nova to behave like that when Availability Zone is not specified
(`Any Availability Zone` is set in Horizon), it will create VM in random
AZ.

STEPS TO REPRODUCE
--

Steps I do to configure that:

1. In `nova.conf` I set `default_schedule_zone = None` and restarted all nova 
services. I found this attribute in documentation [1] and also in Mirantis blog 
post [2]
2. I create 2 availability zones (2 host aggregates each with 1 hypervisor 
added). 
3. I try to create VM In Horizon with `Any Availability Zone` and it results in 
following error:

The requested availability zone is not available (HTTP 400).

The commands I executed to create AZs:

```
openstack aggregate create HA-Test1 --zone AZ-Test1 --property 
availability_zone=AZ-Test1
openstack aggregate create HA-Test2 --zone AZ-Test2 --property 
availability_zone=AZ-Test2
openstack aggregate add host HA-Test1 os-compute-01
openstack aggregate add host HA-Test2 os-compute-02
```

POINT
-

Now this doesn't work as expected, but(!) when I remove `nova.conf`
attribute `default_schedule_zone = None`, or I will configure it empty
`default_schedule_zone = `, spawning of VMs works as expected and they
are scheduled in random AZ.

Therefore I think that Nova doesn't handle `None` as python `None`, but
as string (As if you set default schedule zone to zone `None`).

ENVIRONMENT
---

I am using stable Pike release with KVM + Libvirt installed via
Openstack-Ansible on Ubuntu 16.04.

[1] 
https://docs.openstack.org/ocata/config-reference/compute/config-options.html
[2] 
https://www.mirantis.com/blog/the-first-and-final-word-on-openstack-availability-zones/

** Affects: nova
 Importance: Undecided
 Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1730918

Title:
  Nova does not respect default_schedule_zone `None`

Status in OpenStack Compute (nova):
  New

Bug description:
  Hello,

  DESCRIPTION
  ---

  I want Nova to behave like that when Availability Zone is not
  specified (`Any Availability Zone` is set in Horizon), it will create
  VM in random AZ.

  STEPS TO REPRODUCE
  --

  Steps I do to configure that:

  1. In `nova.conf` I set `default_schedule_zone = None` and restarted all nova 
services. I found this attribute in documentation [1] and also in Mirantis blog 
post [2]
  2. I create 2 availability zones (2 host aggregates each with 1 hypervisor 
added). 
  3. I try to create VM In Horizon with `Any Availability Zone` and it results 
in following error:

  The requested availability zone is not available (HTTP 400).

  The commands I executed to create AZs:

  ```
  openstack aggregate create HA-Test1 --zone AZ-Test1 --property 
availability_zone=AZ-Test1
  openstack aggregate create HA-Test2 --zone AZ-Test2 --property 
availability_zone=AZ-Test2
  openstack aggregate add host HA-Test1 os-compute-01
  openstack aggregate add host HA-Test2 os-compute-02
  ```

  POINT
  -

  Now this doesn't work as expected, but(!) when I remove `nova.conf`
  attribute `default_schedule_zone = None`, or I will configure it empty
  `default_schedule_zone = `, spawning of VMs works as expected and they
  are scheduled in random AZ.

  Therefore I think that Nova doesn't handle `None` as python `None`,
  but as string (As if you set default schedule zone to zone `None`).

  ENVIRONMENT
  ---

  I am using stable Pike release with KVM + Libvirt installed via
  Openstack-Ansible on Ubuntu 16.04.

  [1] 
https://docs.openstack.org/ocata/config-reference/compute/config-options.html
  [2] 
https://www.mirantis.com/blog/the-first-and-final-word-on-openstack-availability-zones/

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1730918/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1728563] [NEW] Live snapshot cannot lock .delta (regression with apparmor)

[Ondrej Vasko]

Public bug reported:

When live snapshot is enabled in Nova via `[workarounds]
disable_libvirt_livesnapshot = False`, it fails after done using Horizon
or CLI with error `unable to execute QEMU command 'drive-mirror': Failed
to lock byte 100`.

Non live snapshot is working fine both in Horizon and CLI.

I traced what cause this error thanks to this bug report
`https://bugs.launchpad.net/nova/+bug/1244694`. The reason of that old
bug was wrong Apparmor config. This bug report was solved fine, but
currently with live snapshot I have an issue with libvirt trying to lock
file in `/var/lib/nova/instances/snapshots//.delta` which
fails according to Apparmor log below.

I hotfixed this issue by updating
`/etc/apparmor.d/libvirt/libvirt-${UUID}` for specific instance's UUID.
I appended following line `/var/lib/nova/instances/snapshots/** k,` and
live snapshot is now working well for that instance.

But this is only hotfix and there must be updated Libvirt Apparmor
template to allow using snapshots subdirectories, or there must be
temporary created rule that allows it before live snapshot is made and
deleted after it is done.

Nova log:

ERROR oslo_messaging.rpc.server [req-ae5933d6-a603-48ab-8d53-a4ebdc57ebdc 
82fb7a159550424098f2addf3c30461a 971a410f32a6446c95f73819bf4eaebc - default 
default] Exception during message handling: libvirtError:
internal error: unable to execute QEMU command 'drive-mirror': Failed to lock 
byte 100
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server   File 
"/openstack/venvs/nova-16.0.1/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
 line 160, in _process_incoming
ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
ERROR oslo_messaging.rpc.server   File 
"/openstack/venvs/nova-16.0.1/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
 line 213, in dispatch
ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, 
ctxt, args)
ERROR oslo_messaging.rpc.server   File 
"/openstack/venvs/nova-16.0.1/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
 line 183, in _do_dispatch
ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
ERROR oslo_messaging.rpc.server   File 
"/openstack/venvs/nova-16.0.1/lib/python2.7/site-packages/nova/exception_wrapper.py",
 line 76, in wrapped
ERROR oslo_messaging.rpc.server function_name, call_dict, binary)
ERROR oslo_messaging.rpc.server   File 
"/openstack/venvs/nova-16.0.1/lib/python2.7/site-packages/oslo_utils/excutils.py",
 line 220, in __exit__
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server   File 
"/openstack/venvs/nova-16.0.1/lib/python2.7/site-packages/oslo_utils/excutils.py",
 line 196, in force_reraise
ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
ERROR oslo_messaging.rpc.server   File 
"/openstack/venvs/nova-16.0.1/lib/python2.7/site-packages/nova/exception_wrapper.py",
 line 67, in wrapped
ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
ERROR oslo_messaging.rpc.server   File 
"/openstack/venvs/nova-16.0.1/lib/python2.7/site-packages/nova/compute/manager.py",
 line 190, in decorated_function
ERROR oslo_messaging.rpc.server "Error: %s", e, instance=instance)
ERROR oslo_messaging.rpc.server   File 
"/openstack/venvs/nova-16.0.1/lib/python2.7/site-packages/oslo_utils/excutils.py",
 line 220, in __exit__
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server   File 
"/openstack/venvs/nova-16.0.1/lib/python2.7/site-packages/oslo_utils/excutils.py",
 line 196, in force_reraise
ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
ERROR oslo_messaging.rpc.server   File 
"/openstack/venvs/nova-16.0.1/lib/python2.7/site-packages/nova/compute/manager.py",
 line 160, in decorated_function
ERROR oslo_messaging.rpc.server return function(self, context, *args, 
**kwargs)
ERROR oslo_messaging.rpc.server   File 
"/openstack/venvs/nova-16.0.1/lib/python2.7/site-packages/nova/compute/manager.py",
 line 218, in decorated_function
ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
ERROR oslo_messaging.rpc.server   File 
"/openstack/venvs/nova-16.0.1/lib/python2.7/site-packages/oslo_utils/excutils.py",
 line 220, in __exit__
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server   File 
"/openstack/venvs/nova-16.0.1/lib/python2.7/site-packages/oslo_utils/excutils.py",
 line 196, in force_reraise
ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
ERROR oslo_messaging.rpc.server   File 
"/openstack/venvs/nova-16.0.1/lib/python2.7/site-packages/nova/compute/manager.py",
 line 206, in decorated_function
ERROR oslo_messaging.rpc.server return function(self, context, *args, 
**kwargs)
ERROR oslo_messaging.rpc.server   File 
"/openstack/venvs/nova-16.0.1/lib/python2.7/site