[Yahoo-eng-team] [Bug 1789535] [NEW] Command nova-status upgrade check always return warning when checking resource providers
Public bug reported: Description === Version: master Cluster model: 1 controller and 1 compute I used nova api database for placement service. When I run command: "nova-status upgrade check", the output always warning in check resource providers. Expected result === ++ | Check: Resource Providers | | Result: Success| | Details: None | ++ Actual result = +---+ | Check: Resource Providers | | Result: Warning | | Details: There are no compute resource providers in the Placement | | service but there are 1 compute nodes in the deployment.| | This means no compute nodes are reporting into the | | Placement service and need to be upgraded and/or fixed. | | See | | https://docs.openstack.org/nova/latest/user/placement.html | | for more details. | +---+ Logs & Configs == I checked in nova-compute.log that compute node was reported to placement service with log: "INFO nova.compute.resource_tracker [req-52b91ba5-518c-476c-ad63-a2c83a1a43c3 - - - - -] Compute node record created for compute1:compute1 with uuid ..." The resource_providers table in nova_api database: MariaDB [nova_api]> select * from resource_providers; +-+-++--+--++--+--++ | created_at | updated_at | id | uuid | name | generation | can_host | root_provider_id | parent_provider_id | +-+-++--+--++--+--++ | 2018-08-16 07:20:33 | 2018-08-27 03:12:37 | 1 | a5ac6e98-9935-456c-9de1-10f47adb81d6 | compute1 | 7 | NULL | 1 | NULL | +-+-++--+--++--+--++ ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1789535 Title: Command nova-status upgrade check always return warning when checking resource providers Status in OpenStack Compute (nova): New Bug description: Description === Version: master Cluster model: 1 controller and 1 compute I used nova api database for placement service. When I run command: "nova-status upgrade check", the output always warning in check resource providers. Expected result === ++ | Check: Resource Providers | | Result: Success| | Details: None | ++ Actual result = +---+ | Check: Resource Providers | | Result: Warning | | Details: There are no compute resource providers in the Placement | | service but there are 1 compute nodes in the deployment.| | This means no compute nodes are reporting into the | | Placement service and need to be upgraded and/or fixed. | | See | | https://docs.openstack.org/nova/latest/user/placement.html | | for more details. | +---+ Logs & Configs == I checked in nova-compute.log that compute node was reported to placement service with log: "INFO nova.compute.resource_tracker [req-52b91ba5-518c-476c-ad63-a2c83a1a43c3 - - - - -] Compute node record created for compute1:compute1 with uuid ..." The resource_providers table in nova_api database: MariaDB [nova_api]> select * from resource_providers; +-+-++--+--++--+--++ | created_at | updated_at | id | uuid | name | generation | can_host | root_provider_id | parent_provider_id |
[Yahoo-eng-team] [Bug 1788833] Re: Error during ComputeManager.update_available_resource: AttributeError: '_TransactionContextManager' object has no attribute 'async_
Reviewed: https://review.openstack.org/597173 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=bd7d991309ea2bea5d175cb1f2710519936fd0c2 Submitter: Zuul Branch:master commit bd7d991309ea2bea5d175cb1f2710519936fd0c2 Author: Stephen Finucane Date: Tue Aug 28 17:01:07 2018 +0100 Don't use '_TransactionContextManager._async' In commit 2d532963, all instances of 'async' were replaced with 'async_' in preparation for Python 3.7. However, one of these should not have been changed as it refers to an oslo.db object attribute. That attribute has actually been renamed itself but that rename is only present from oslo.db 4.40.0 [1]. Thankfully, an alias to the older name is provided so we use that. [1] https://github.com/openstack/oslo.db/commit/df6bf34 Change-Id: I1afd0ba34a9ebcb63edb91e880ef60580befb32e Closes-Bug: #1788833 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1788833 Title: Error during ComputeManager.update_available_resource: AttributeError: '_TransactionContextManager' object has no attribute 'async_ Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) rocky series: Triaged Bug description: Hi all, I have a rocky openstack cluster. I am using Mariadb galera cluster (3 galera nodes Active/Active) behind haproxy. When i resize or migrate an instance, I am hitting with the following errors. 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task [req-67381a5e-24e2-4dd2-bfc6-693bd1fabb8d 290bb90f6cbc46548951cbcaee0c0a34 9804c6f8ffe148bc9fa7ed409d41cb16 - default default] Error during ComputeManager._heal_instance_info_cache: AttributeError: '_TransactionContextManager' object has no attribute 'async_' Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 126, in _object_dispatch return getattr(target, method)(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper result = fn(cls, context, *args, **kwargs) File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1351, in get_by_host use_slave=use_slave) File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 218, in wrapper reader_mode = get_context_manager(context).async_ AttributeError: '_TransactionContextManager' object has no attribute 'async_' 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task Traceback (most recent call last): 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task task(self, context) 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6632, in _heal_instance_info_cache 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task context, self.host, expected_attrs=[], use_slave=True) 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 177, in wrapper 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task args, kwargs) 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 241, in object_class_action_versions 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task args=args, kwargs=kwargs) 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 179, in call 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task retry=self.retry) 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 133, in _send 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task retry=retry) 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task call_monitor_timeout, retry=retry) 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task raise result 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task AttributeError: '_TransactionContextManager' object has no attribute 'async_'
[Yahoo-eng-team] [Bug 1789512] [NEW] Can't refresh DNS zones pages, get HTTP 404
Public bug reported: If I've gone to a Designate DNS zone (or any of its instances) within Horizon, if I then press F5 or send the link to someone else all you get is an HTTP 404. Example such URLs: * https://horizon.wikimedia.org/project/ngdetails/OS::Designate::Zone/7f302e0f-8e24-4378-a77b-916e5f55478f * https://horizon.wikimedia.org/project/ngdetails/OS::Designate::RecordSet/7f302e0f-8e24-4378-a77b-916e5f55478f/112128b1-a518-45de-b89f-f4e9aecdcca3 Instead you should get the same content as you were just lokoing at. I believe that's mitaka. I can ask for details of the installation from it's administrators if that would be helpful. ** Affects: horizon Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Dashboard (Horizon). https://bugs.launchpad.net/bugs/1789512 Title: Can't refresh DNS zones pages, get HTTP 404 Status in OpenStack Dashboard (Horizon): New Bug description: If I've gone to a Designate DNS zone (or any of its instances) within Horizon, if I then press F5 or send the link to someone else all you get is an HTTP 404. Example such URLs: * https://horizon.wikimedia.org/project/ngdetails/OS::Designate::Zone/7f302e0f-8e24-4378-a77b-916e5f55478f * https://horizon.wikimedia.org/project/ngdetails/OS::Designate::RecordSet/7f302e0f-8e24-4378-a77b-916e5f55478f/112128b1-a518-45de-b89f-f4e9aecdcca3 Instead you should get the same content as you were just lokoing at. I believe that's mitaka. I can ask for details of the installation from it's administrators if that would be helpful. To manage notifications about this bug go to: https://bugs.launchpad.net/horizon/+bug/1789512/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1789499] [NEW] Policy.json get_service_profiles says admin_only but non-admin users can access api
Public bug reported:
Policy.json has these two lines:
"create_service_profile": "rule:admin_only",
"get_service_profiles": "rule:admin_only",
However get_service_profiles can be accessed without error by non-admin
user:
$ export OS_USERNAME=demo
$ openstack token issue
++-+
| Field | Value
|
++-+
| expires| 2018-08-28T22:11:10+
|
| id |
gABbhbpuuY6N3_jsujbYhGkotgjoVwlwynqVJ_Uot6RrvJtzqoL0vv74hJewGlx4WZB81ddK8J7hYUqltKtV4iN8VbvEvMuVIl8BPgkWHlDcJPC_SF-KYda6NMweldqkYcL65nc26n2Rzg5PhYGcCvEwIfyXLsnLCgJYcW09Se_utmqCXbM
|
| project_id | 28f7717697d34265accbd0b348ee1e51
|
| user_id| 6465999184924e9d9fec2618b15ab277
|
++-+
devstack@devstack$ export
MY_TOKEN=gABbhbpuuY6N3_jsujbYhGkotgjoVwlwynqVJ_Uot6RrvJtzqoL0vv74hJewGlx4WZB81ddK8J7hYUqltKtV4iN8VbvEvMuVIl8BPgkWHlDcJPC_SF-KYda6NMweldqkYcL65nc26n2Rzg5PhYGcCvEwIfyXLsnLCgJYcW09Se_utmqCXbM
devstack@devstack:/opt/stack/logs$ curl -H "X-Auth-Token: $MY_TOKEN" -X GET
http://127.0.0.1:9696/v2.0/service_profiles{"service_profiles": []}
I would expect an error similar to create_service_profile because they have the
same admin_only policy:
$ curl -H "X-Auth-Token: $MY_TOKEN" -X POST
http://127.0.0.1:9696/v2.0/service_profiles -d '{ "service_profile": {
"enabled": "true", "description": "Dummy profile", "metainfo" : "{'foo':
'bar'}"}}'
{"NeutronError": {"message": "rule:create_service_profile is disallowed by
policy", "type": "PolicyNotAuthorized", "detail": ""}}
Please note, that despite not receiving an error, the non-admin user does get
an empty list despite existing service_profiles:
$ export OS_USERNAME=admin
$ openstack token issue
++-+
| Field | Value
|
++-+
| expires| 2018-08-28T22:15:06+
|
| id |
gABbhbtaMytoBFfviejpK5m7OnlusCnp1p_gExNtsaxAbfD1egn0ljOBb93uXjyPOE6bhMj0hcvDa2CE_o9odPBQmIgZKAtOq9-_OLGUvZbq4ACZAG1_DvLYBCPsB2X8qoD2cvRxPGQKbWdnviuos-nHdtkKnfdhYt6UWBCa3gm7dcvr0EE
|
| project_id | 28f7717697d34265accbd0b348ee1e51
|
| user_id| e935d2593d404a86ba0f58c5a84c8ae2
|
++-+
$ export
MY_TOKEN=gABbhbtaMytoBFfviejpK5m7OnlusCnp1p_gExNtsaxAbfD1egn0ljOBb93uXjyPOE6bhMj0hcvDa2CE_o9odPBQmIgZKAtOq9-_OLGUvZbq4ACZAG1_DvLYBCPsB2X8qoD2cvRxPGQKbWdnviuos-nHdtkKnfdhYt6UWBCa3gm7dcvr0EE
$ curl -H "X-Auth-Token: $MY_TOKEN" -X GET
http://127.0.0.1:9696/v2.0/service_profiles{"service_profiles": [{"id":
"a4e1544b-5296-46b8-bdb6-02f9d11eef2c", "enabled": true, "driver": "",
"description": "Dummy profile", "metainfo": "{foo: bar}"}]}
** Affects: neutron
Importance:
[Yahoo-eng-team] [Bug 1789484] Re: n-cpu fails init on timeout calling n-cond-cell1
** Also affects: nova/queens Importance: Undecided Status: New ** Also affects: nova/rocky Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1789484 Title: n-cpu fails init on timeout calling n-cond-cell1 Status in OpenStack Compute (nova): Triaged Status in OpenStack Compute (nova) queens series: New Status in OpenStack Compute (nova) rocky series: New Bug description: We've seen several cases now where n-cpu fails to start in gate runs. It appears that during the service init it tries to call out to the cell1 conductor. Logs show n-cond-cell1 did start about 1 second before this, but maybe too soon to accept API calls. Traceback in the n-cpu log has the message: "MessagingTimeout: Timed out waiting for a reply to message ID" http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack- nova.2018-08-28.log.html#t2018-08-28T19:28:52 http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22_determine_version_cap%5C%22%20AND%20tags:%5C%22screen-n-cpu.txt%5C%22=7d To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1789484/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1789484] [NEW] n-cpu fails init on timeout calling n-cond-cell1
Public bug reported: We've seen several cases now where n-cpu fails to start in gate runs. It appears that during the service init it tries to call out to the cell1 conductor. Logs show n-cond-cell1 did start about 1 second before this, but maybe too soon to accept API calls. Traceback in the n-cpu log has the message: "MessagingTimeout: Timed out waiting for a reply to message ID" http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack- nova.2018-08-28.log.html#t2018-08-28T19:28:52 http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22_determine_version_cap%5C%22%20AND%20tags:%5C%22screen-n-cpu.txt%5C%22=7d ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1789484 Title: n-cpu fails init on timeout calling n-cond-cell1 Status in OpenStack Compute (nova): New Bug description: We've seen several cases now where n-cpu fails to start in gate runs. It appears that during the service init it tries to call out to the cell1 conductor. Logs show n-cond-cell1 did start about 1 second before this, but maybe too soon to accept API calls. Traceback in the n-cpu log has the message: "MessagingTimeout: Timed out waiting for a reply to message ID" http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack- nova.2018-08-28.log.html#t2018-08-28T19:28:52 http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22_determine_version_cap%5C%22%20AND%20tags:%5C%22screen-n-cpu.txt%5C%22=7d To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1789484/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1788936] Re: Network address translation in Neutron wrong RFC in documentation
Reviewed: https://review.openstack.org/596495 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3e1512873ad1fa7ad60a295ef86cefd3ac8c25f8 Submitter: Zuul Branch:master commit 3e1512873ad1fa7ad60a295ef86cefd3ac8c25f8 Author: John-Paul Robinson Date: Fri Aug 24 15:18:56 2018 -0500 Fix the RFC and private internet network address It seems the RFC should be 1918 which defines IP addresses for private networks. The existing RFC 5737 actually defines private IP ranges for use in documentation. Change-Id: Ic794585e636bb9a9c915347a801c8a4f0baa9012 Closes-Bug: 1788936 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1788936 Title: Network address translation in Neutron wrong RFC in documentation Status in neutron: Fix Released Bug description: This bug tracker is for errors with the documentation, use the following as a template and remove or add fields as you see fit. Convert [ ] into [x] to check boxes: - [ x] This doc is inaccurate in this way: Shouldn't it be RFC1918 which defines private IP address ranges on networks instead of RFC5737 which defines private ranges for use in documentation? RFC 5737 reserves the following three subnets as private addresses: 192.0.2.0/24 198.51.100.0/24 203.0.113.0/24 - [ ] This is a doc addition request. - [x] I have a fix to the document that I can paste below including example: RFC 1918 reserves the following three subnets as private addresses: 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 If you have a troubleshooting or support issue, use the following resources: - Ask OpenStack: http://ask.openstack.org - The mailing list: http://lists.openstack.org - IRC: 'openstack' channel on Freenode --- Release: 11.0.6.dev66 on 2018-08-13 11:52 SHA: b87eb4814a1a936844a0dbd726e7cd9a0de5b492 Source: https://git.openstack.org/cgit/openstack/neutron/tree/doc/source/admin/intro-nat.rst URL: https://docs.openstack.org/neutron/pike/admin/intro-nat.html To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1788936/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1746863] Re: scheduler affinity doesn't work with multiple cells
Reviewed: https://review.openstack.org/540258
Committed:
https://git.openstack.org/cgit/openstack/nova/commit/?id=14f4c502f92b10b669044e5069ac3b3555a42ee0
Submitter: Zuul
Branch:master
commit 14f4c502f92b10b669044e5069ac3b3555a42ee0
Author: melanie witt
Date: Fri Feb 2 05:41:20 2018 +
Make scheduler.utils.setup_instance_group query all cells
To check affinity and anti-affinity policies for scheduling instances,
we use the RequestSpec.instance_group.hosts field to check the hosts
that have group members on them. Access of the 'hosts' field calls
InstanceGroup.get_hosts during a lazy-load and get_hosts does a query
for all instances that are members of the group and returns their hosts
after removing duplicates. The InstanceList query isn't targeting any
cells, so it will return [] in a multi-cell environment in both the
instance create case and the instance move case. In the move case, we
do have a cell-targeted RequestContext when setup_instance_group is
called *but* the RequestSpec.instance_group object is queried early in
compute/api before we're targeted to a cell, so a call of
RequestSpec.instance_group.get_hosts() will result in [] still, even
for move operations.
This makes setup_instance_group query all cells for instances that are
members of the instance group if the RequestContext is untargeted, else
it queries the targeted cell for the instances.
Closes-Bug: #1746863
Change-Id: Ia5f5a0d75953b1154a8de3e1eaa15f8042e32d77
** Changed in: nova
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1746863
Title:
scheduler affinity doesn't work with multiple cells
Status in OpenStack Compute (nova):
Fix Released
Status in OpenStack Compute (nova) pike series:
Confirmed
Status in OpenStack Compute (nova) queens series:
Confirmed
Status in OpenStack Compute (nova) rocky series:
New
Bug description:
I happened upon this while hacking on my WIP CellDatabases fixture
patch.
Some of the nova/tests/functional/test_server_group.py tests started
failing with multiple cells and I found that it's because there's a
database query 'objects.InstanceList.get_by_filters' for all instances
who are members of the server group to do the affinity check. The
query for instances doesn't check all cells, so it fails to return any
hosts that group members are currently on.
This makes the ServerGroup[Anti|]AffinityFilter a no-op for multiple
cells. Affinity is checked again via the late-affinity check in
compute, but compute is using the same InstanceGroup.get_hosts method
and will only find group member's hosts that are in its cell.
This is the code that populates the RequestSpec.instance_group.hosts via a
lazy-load on first access:
nova/objects/instance_group.py:
def obj_load_attr(self, attrname):
...
self.hosts = self.get_hosts()
self.obj_reset_changes(['hosts'])
...
@base.remotable
def get_hosts(self, exclude=None):
"""Get a list of hosts for non-deleted instances in the group
This method allows you to get a list of the hosts where instances in
this group are currently running. There's also an option to exclude
certain instance UUIDs from this calculation.
"""
filter_uuids = self.members
if exclude:
filter_uuids = set(filter_uuids) - set(exclude)
filters = {'uuid': filter_uuids, 'deleted': False}
instances = objects.InstanceList.get_by_filters(self._context,
filters=filters)
return list(set([instance.host for instance in instances
if instance.host]))
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1746863/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1788833] Re: Error during ComputeManager.update_available_resource: AttributeError: '_TransactionContextManager' object has no attribute 'async_
** Changed in: nova Importance: Undecided => Medium ** Also affects: nova/rocky Importance: Undecided Status: New ** Changed in: nova/rocky Status: New => Triaged ** Changed in: nova/rocky Importance: Undecided => Medium -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1788833 Title: Error during ComputeManager.update_available_resource: AttributeError: '_TransactionContextManager' object has no attribute 'async_ Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) rocky series: Triaged Bug description: Hi all, I have a rocky openstack cluster. I am using Mariadb galera cluster (3 galera nodes Active/Active) behind haproxy. When i resize or migrate an instance, I am hitting with the following errors. 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task [req-67381a5e-24e2-4dd2-bfc6-693bd1fabb8d 290bb90f6cbc46548951cbcaee0c0a34 9804c6f8ffe148bc9fa7ed409d41cb16 - default default] Error during ComputeManager._heal_instance_info_cache: AttributeError: '_TransactionContextManager' object has no attribute 'async_' Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 126, in _object_dispatch return getattr(target, method)(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper result = fn(cls, context, *args, **kwargs) File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1351, in get_by_host use_slave=use_slave) File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 218, in wrapper reader_mode = get_context_manager(context).async_ AttributeError: '_TransactionContextManager' object has no attribute 'async_' 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task Traceback (most recent call last): 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task task(self, context) 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6632, in _heal_instance_info_cache 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task context, self.host, expected_attrs=[], use_slave=True) 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 177, in wrapper 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task args, kwargs) 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 241, in object_class_action_versions 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task args=args, kwargs=kwargs) 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 179, in call 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task retry=self.retry) 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 133, in _send 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task retry=retry) 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task call_monitor_timeout, retry=retry) 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task raise result 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task AttributeError: '_TransactionContextManager' object has no attribute 'async_' 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task Traceback (most recent call last): 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 126, in _object_dispatch 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task return getattr(target, method)(*args, **kwargs) 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task 2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper 2018-08-24 12:06:37.668 19857 ERROR
[Yahoo-eng-team] [Bug 1789455] [NEW] nova list --status SOFT_DELETED does not return soft-deleted instance
Public bug reported: Description === nova list --status SOFT_DELETED does not return soft-deleted instance Steps to reproduce == 1 enable soft-delete by setting reclaim_instance_interval > 0 in nova.conf on api and compute nodes. 2 $ nova boot --flavor xxx --image xxx --nic xxx VM 3 $ nova delete VM 4 $ nova list --status SOFT_DELETED Expected result === VM is the returned table. Actual result = It is not. Environment === $ nova --version 10.1.0 ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1789455 Title: nova list --status SOFT_DELETED does not return soft-deleted instance Status in OpenStack Compute (nova): New Bug description: Description === nova list --status SOFT_DELETED does not return soft-deleted instance Steps to reproduce == 1 enable soft-delete by setting reclaim_instance_interval > 0 in nova.conf on api and compute nodes. 2 $ nova boot --flavor xxx --image xxx --nic xxx VM 3 $ nova delete VM 4 $ nova list --status SOFT_DELETED Expected result === VM is the returned table. Actual result = It is not. Environment === $ nova --version 10.1.0 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1789455/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1787331] Re: The eventlet.monkey_patch is not valid under uwsgi mode
Reviewed: https://review.openstack.org/592285 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=23ba1c690652832c655d57476630f02c268c87ae Submitter: Zuul Branch:master commit 23ba1c690652832c655d57476630f02c268c87ae Author: Yikun Jiang Date: Thu Aug 16 14:29:26 2018 +0800 Make monkey patch work in uWSGI mode There was a eventlet.monkey_patch [1] when we launch a nova process(like nova-api, nova-compute), but it's invalid under the uwsgi mode. But there are concurrency requirements in the api service, such as, when listing instances cross multiple cells we're using greenthreads and oslo.db does a time.sleep to allow switching greenthreads [2]. So, in this patch we add the monkey_patch in the uwsgi application setup and refactor the monkey patching to use common code. [1] https://github.com/openstack/nova/blob/233ea58/nova/cmd/__init__.py#L26 [2] https://github.com/openstack/oslo.db/blob/9c66959/oslo_db/sqlalchemy/engines.py#L51 Closes-bug: #1787331 Change-Id: Ie7bf5d012e2ccbcd63c262ddaf739782afcdaf56 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1787331 Title: The eventlet.monkey_patch is not valid under uwsgi mode Status in OpenStack Compute (nova): Fix Released Bug description: There was a eventlet.monkey_patch [1] when we launch a nova process(like nova-api, nova-compute), but it's invalid under the uwsgi mode. So, we need do same monkey_patch when we setup uwsgi appilication. [1] https://github.com/openstack/nova/blob/233ea58/nova/cmd/__init__.py#L26 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1787331/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1789450] [NEW] Groups mapped to projects that do not exist in OpenStack breaks WebSSO
Public bug reported: I have come across an issue when using webSSO/Federation. We are using keycloak as an SP, in which our users exist. These users have multiple groups some of which are open stack specific and some which are not. These users and groups are being mapped as ephemeral users, and im using groups to match to projects. The issue occurs if a user has a group that does not map to a project in OpenStack. at which point an exception is raised and the websso login blows up with a 500 message. The offending line is line 347 in keystone/federation/utils.py A quick fix would be to remove the exception from being raised, and just log to file. Or filter the projects based on the groups passed in. ** Affects: keystone Importance: Undecided Status: New ** Summary changed: - Groups that do not exist in the backend break webSSO + Groups mapped to projects that do not exist in OpenStack breaks WebSSO -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1789450 Title: Groups mapped to projects that do not exist in OpenStack breaks WebSSO Status in OpenStack Identity (keystone): New Bug description: I have come across an issue when using webSSO/Federation. We are using keycloak as an SP, in which our users exist. These users have multiple groups some of which are open stack specific and some which are not. These users and groups are being mapped as ephemeral users, and im using groups to match to projects. The issue occurs if a user has a group that does not map to a project in OpenStack. at which point an exception is raised and the websso login blows up with a 500 message. The offending line is line 347 in keystone/federation/utils.py A quick fix would be to remove the exception from being raised, and just log to file. Or filter the projects based on the groups passed in. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1789450/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1789429] [NEW] Glance v2 API doesn't support credentials for image importing
Public bug reported: Old glance v1 API supported credentials specification in the "--copy- from" parameter. However new v2 API doesn't support credentials as well as "swift+https" method. Besides when URL is specified with credentials, these credentials are available via "glance task-show %taskid%" output. ** Affects: glance Importance: Undecided Status: New ** Tags: glance -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1789429 Title: Glance v2 API doesn't support credentials for image importing Status in Glance: New Bug description: Old glance v1 API supported credentials specification in the "--copy- from" parameter. However new v2 API doesn't support credentials as well as "swift+https" method. Besides when URL is specified with credentials, these credentials are available via "glance task-show %taskid%" output. To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1789429/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1789434] [NEW] neutron_tempest_plugin.scenario.test_migration.NetworkMigrationFromHA failing 100% times
Public bug reported: Since few days all migration tests from DVR router fails. Example of failure: http://logs.openstack.org/37/382037/71/check/neutron-tempest-plugin-dvr-multinode-scenario/605ed17/logs/testr_results.html.gz May be related somehow to https://review.openstack.org/#/c/589410/ but I'm not sure yet. ** Affects: neutron Importance: High Status: Confirmed ** Tags: gate-failure l3-dvr-backlog ** Tags removed: gate ** Tags added: gate-failure l3-dvr-backlog -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1789434 Title: neutron_tempest_plugin.scenario.test_migration.NetworkMigrationFromHA failing 100% times Status in neutron: Confirmed Bug description: Since few days all migration tests from DVR router fails. Example of failure: http://logs.openstack.org/37/382037/71/check/neutron-tempest-plugin-dvr-multinode-scenario/605ed17/logs/testr_results.html.gz May be related somehow to https://review.openstack.org/#/c/589410/ but I'm not sure yet. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1789434/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1789423] [NEW] Server operations fail to complete with versioned notifications if payload contains unset is_public field
Public bug reported: This is a follow up to bug 1739325 which fixed the scenario that the flavor.disabled field was missing from the embedded instance flavor. The same case occurs for the is_public field, so we should default that to True if it's not set in the embedded instance.flavor. ** Affects: nova Importance: Medium Status: Triaged ** Tags: notifications ** Changed in: nova Importance: Undecided => Medium ** Changed in: nova Status: New => Triaged -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1789423 Title: Server operations fail to complete with versioned notifications if payload contains unset is_public field Status in OpenStack Compute (nova): Triaged Bug description: This is a follow up to bug 1739325 which fixed the scenario that the flavor.disabled field was missing from the embedded instance flavor. The same case occurs for the is_public field, so we should default that to True if it's not set in the embedded instance.flavor. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1789423/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1789402] [NEW] Security group rule - Other protocol
Public bug reported: If try to create rule with other ip protocol then exception rise. Cause of problem in openstack_dashboard/api/neutron.py https://github.com/openstack/horizon/blob/841bce062327d70fb834b99bc2f238323e400a51/openstack_dashboard/api/neutron.py#L418-L441 If we define ip_protocol it mean that we didn't define to_port and from_port. It means that comparison default values of to_port/from_port(None) with 0 will rise exception. It can be fixed with something like this: if not cidr: cidr = None +if from_port != None: if from_port < 0: from_port = None +if to_port != None: if to_port < 0: to_port = None ** Affects: horizon Importance: Undecided Status: New ** Description changed: - If try to create rule with ip protocol then exception rise. + If try to create rule with other ip protocol then exception rise. Cause of problem in openstack_dashboard/api/neutron.py https://github.com/openstack/horizon/blob/841bce062327d70fb834b99bc2f238323e400a51/openstack_dashboard/api/neutron.py#L418-L441 If we define ip_protocol it mean that we didn't define to_port and from_port. It means that comparison default values of to_port/from_port(None) with 0 will rise exception. It can be fixed with something like this: - if not cidr: - cidr = None + if not cidr: + cidr = None +if from_port != None: - if from_port < 0: - from_port = None + if from_port < 0: + from_port = None +if to_port != None: - if to_port < 0: - to_port = None + if to_port < 0: + to_port = None -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Dashboard (Horizon). https://bugs.launchpad.net/bugs/1789402 Title: Security group rule - Other protocol Status in OpenStack Dashboard (Horizon): New Bug description: If try to create rule with other ip protocol then exception rise. Cause of problem in openstack_dashboard/api/neutron.py https://github.com/openstack/horizon/blob/841bce062327d70fb834b99bc2f238323e400a51/openstack_dashboard/api/neutron.py#L418-L441 If we define ip_protocol it mean that we didn't define to_port and from_port. It means that comparison default values of to_port/from_port(None) with 0 will rise exception. It can be fixed with something like this: if not cidr: cidr = None +if from_port != None: if from_port < 0: from_port = None +if to_port != None: if to_port < 0: to_port = None To manage notifications about this bug go to: https://bugs.launchpad.net/horizon/+bug/1789402/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1789403] [NEW] ip6tables rules for PD subnets not fully recreated at l3-agent restart
Public bug reported: currently ip6tables in the qrouter namespace has the following rule. This causes unmarked packets to drop. -A neutron-l3-agent-scope -o qr-f4eceee5-a4 -m mark ! --mark 0x400/0x -j DROP In a related bug (1570122) there was a problem that prefix-delegated sub-nets did not get a rule setting this mark on traffic incoming on the gateway port, and this traffic was thus dropped. Now, this seems to work correctly when a user creates a subnet with IPv6 from PD. The problem arises when the l3-agent restarts, or the router moves to another l3-agent, as the rule marking the traffic is not recreated in these cases. The result is the same symptoms as the bug #1570122 Adding the rule manually makes traffic flowing again, for instance with the line: $ip6tables -t mangle -A neutron-l3-agent-scope -i qg-28f7e259-d2 -j MARK --set-xmark 0x400/0x We are running at the Queens release at the moment: - neutron-l3-agent 2:12.0.2-0ubuntu1~cloud0 This bugs are a major obstacle for IPv6 in our clouds, as we cannot deliver reliable transport of ipv6 packets when this rule suddenly are missing. ** Affects: neutron Importance: Undecided Status: New ** Tags: ipv6 queens-backport-potential -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1789403 Title: ip6tables rules for PD subnets not fully recreated at l3-agent restart Status in neutron: New Bug description: currently ip6tables in the qrouter namespace has the following rule. This causes unmarked packets to drop. -A neutron-l3-agent-scope -o qr-f4eceee5-a4 -m mark ! --mark 0x400/0x -j DROP In a related bug (1570122) there was a problem that prefix-delegated sub-nets did not get a rule setting this mark on traffic incoming on the gateway port, and this traffic was thus dropped. Now, this seems to work correctly when a user creates a subnet with IPv6 from PD. The problem arises when the l3-agent restarts, or the router moves to another l3-agent, as the rule marking the traffic is not recreated in these cases. The result is the same symptoms as the bug #1570122 Adding the rule manually makes traffic flowing again, for instance with the line: $ip6tables -t mangle -A neutron-l3-agent-scope -i qg-28f7e259-d2 -j MARK --set-xmark 0x400/0x We are running at the Queens release at the moment: - neutron-l3-agent 2:12.0.2-0ubuntu1~cloud0 This bugs are a major obstacle for IPv6 in our clouds, as we cannot deliver reliable transport of ipv6 packets when this rule suddenly are missing. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1789403/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1788745] Re: resource ACCEPT LOG can only print once then get CookieNotFound
Disappear from the test env, please retore this if anyone meets this again. ** Changed in: neutron Status: Incomplete => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1788745 Title: resource ACCEPT LOG can only print once then get CookieNotFound Status in neutron: Invalid Bug description: ENV: neutron stable/queens with backport patches: Fix no packet log data when debug is set False in configuration https://review.openstack.org/#/c/591545/ Fix lost connection when create security group log https://review.openstack.org/#/c/593534/ Fix no ACCEPT event can get for security group logging https://review.openstack.org/#/c/591547/ service log: 2018-08-24 11:10:53.179 2627349 WARNING neutron.services.logapi.drivers.openvswitch.ovs_firewall_log [-] Unknown cookie=2664476845653012007 packet_in pkt=ethernet(dst='fa:16:3e:73:9f:a3',ethertype=2048,src='fa:16:3e:81:02:32'), ipv4(csum=39526,dst='192.168.111.5',flags=2,header_length=5,identification=50582,offset=0,option=None,proto=6,src='172.16.0.1',tos=0,total_length=60,ttl=63,version=4), tcp(ack=0,bits=2,csum=60919,dst_port=22,offset=10,option=[TCPOptionMaximumSegmentSize(kind=2,length=4,max_seg_size=1460), TCPOptionSACKPermitted(kind=4,length=2), TCPOptionTimestamps(kind=8,length=10,ts_ecr=0,ts_val=3048195896), TCPOptionNoOperation(kind=1,length=1), TCPOptionWindowScale(kind=3,length=3,shift_cnt=7)],seq=3571909294,src_port=47012,urgent=0,window_size=29200): CookieNotFound: Cookie 2664476845653012007 could not be found. How to reproduce: 1. add ACCEPT log to port and security group 2. ssh to the server <-- here you can see the ACCEPT LOG 3. disconnect the ssh, and ssh again. Then the resource log has the WARNING CookieNotFound. So I think code is basiclly same to the master, I think we can reproduce this in master branch. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1788745/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1789391] [NEW] [RFE]Support VPC peering for openstack public cloud providers
Public bug reported: Currently, the AWS VPC is a common concept for cloud providers, but for openstack models, we treat it as a Neutron Router. So the VPC peering support corresponds to something like "Router peering" support. This function provides the network connecting between different isolated networks. So this is a another way to connect different networks. For common use cases, we usually deploy the cloud based on overlay network, so each Router(VPC) owns its global vni(for example, we use vxlan). For vpc peering function, that means we need to support connection between different vni. So a vxlan l3 gw should be introduced into neutron for supporting network connection accross different neutron Routers. Also this is very useful for making user ease to use, for example, if a company(customer) owns mutiple accounts in openstack, and this company has multiple IT departments, each with their own isolated network, which under the neutron Router. Some networks are located within the same openstack account, and others in a different openstack account. The only way to make it happen by the Routers. You want to peer together all networks(Routers) to enable the IT departments to have full access to each others' resources. ** Affects: neutron Importance: Undecided Status: New ** Tags: rfe ** Summary changed: - Support VPC peering for openstack public cloud providers + [RFE]Support VPC peering for openstack public cloud providers ** Tags added: rfe -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1789391 Title: [RFE]Support VPC peering for openstack public cloud providers Status in neutron: New Bug description: Currently, the AWS VPC is a common concept for cloud providers, but for openstack models, we treat it as a Neutron Router. So the VPC peering support corresponds to something like "Router peering" support. This function provides the network connecting between different isolated networks. So this is a another way to connect different networks. For common use cases, we usually deploy the cloud based on overlay network, so each Router(VPC) owns its global vni(for example, we use vxlan). For vpc peering function, that means we need to support connection between different vni. So a vxlan l3 gw should be introduced into neutron for supporting network connection accross different neutron Routers. Also this is very useful for making user ease to use, for example, if a company(customer) owns mutiple accounts in openstack, and this company has multiple IT departments, each with their own isolated network, which under the neutron Router. Some networks are located within the same openstack account, and others in a different openstack account. The only way to make it happen by the Routers. You want to peer together all networks(Routers) to enable the IT departments to have full access to each others' resources. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1789391/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1789382] [NEW] openstack server list error
Public bug reported: when execute openstack server list,Show the following error: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. (HTTP 500) (Request-ID: req-6a17a629-6bc5-4297-90b4-524dd96b5694) everytime I see this in nova-api.log: 2018-08-28 16:01:12.660 1092436 WARNING nova.context [req-6a17a629-6bc5-4297-90b4-524dd96b5694 b0dc354f61bb4c88ab5c544079f98c2b cbe025541b814d42ada3028732353516 - default default] Timed out waiting for response from cell 4d2b47d3-7d78-4061-97fe-89c32e878fa7: CellTimeout: Timeout waiting for response from cell 2018-08-28 16:01:12.661 1092436 WARNING nova.context [req-6a17a629-6bc5-4297-90b4-524dd96b5694 b0dc354f61bb4c88ab5c544079f98c2b cbe025541b814d42ada3028732353516 - default default] Timed out waiting for response from cell ----: CellTimeout: Timeout waiting for response from cell 2018-08-28 16:01:12.662 1092436 ERROR nova.api.openstack.wsgi [req-6a17a629-6bc5-4297-90b4-524dd96b5694 b0dc354f61bb4c88ab5c544079f98c2b cbe025541b814d42ada3028732353516 - default default] Unexpected exception in API method: TypeError: 'object' object is not iterable 2018-08-28 16:01:12.662 1092436 ERROR nova.api.openstack.wsgi Traceback (most recent call last): 2018-08-28 16:01:12.662 1092436 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 788, in wrapped 2018-08-28 16:01:12.662 1092436 ERROR nova.api.openstack.wsgi return f(*args, **kwargs) 2018-08-28 16:01:12.662 1092436 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 181, in wrapper 2018-08-28 16:01:12.662 1092436 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2018-08-28 16:01:12.662 1092436 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 181, in wrapper 2018-08-28 16:01:12.662 1092436 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2018-08-28 16:01:12.662 1092436 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 165, in detail 2018-08-28 16:01:12.662 1092436 ERROR nova.api.openstack.wsgi servers = self._get_servers(req, is_detail=True) 2018-08-28 16:01:12.662 1092436 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 309, in _get_servers 2018-08-28 16:01:12.662 1092436 ERROR nova.api.openstack.wsgi sort_keys=sort_keys, sort_dirs=sort_dirs) 2018-08-28 16:01:12.662 1092436 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2500, in get_all 2018-08-28 16:01:12.662 1092436 ERROR nova.api.openstack.wsgi context, filters, limit, marker, fields, sort_keys, sort_dirs) 2018-08-28 16:01:12.662 1092436 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/instance_list.py", line 107, in get_instance_objects_sorted 2018-08-28 16:01:12.662 1092436 ERROR nova.api.openstack.wsgi expected_attrs) 2018-08-28 16:01:12.662 1092436 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1208, in _make_instance_list 2018-08-28 16:01:12.662 1092436 ERROR nova.api.openstack.wsgi for db_inst in db_inst_list: 2018-08-28 16:01:12.662 1092436 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/multi_cell_list.py", line 259, in get_records_sorted 2018-08-28 16:01:12.662 1092436 ERROR nova.api.openstack.wsgi for i in heapq.merge(*results.values()): 2018-08-28 16:01:12.662 1092436 ERROR nova.api.openstack.wsgi File "/usr/lib64/python2.7/heapq.py", line 372, in merge 2018-08-28 16:01:12.662 1092436 ERROR nova.api.openstack.wsgi for itnum, it in enumerate(map(iter, iterables)): 2018-08-28 16:01:12.662 1092436 ERROR nova.api.openstack.wsgi TypeError: 'object' object is not iterable 2018-08-28 16:01:12.662 1092436 ERROR nova.api.openstack.wsgi 2018-08-28 16:01:12.702 1092436 INFO nova.api.openstack.wsgi [req-6a17a629-6bc5-4297-90b4-524dd96b5694 b0dc354f61bb4c88ab5c544079f98c2b cbe025541b814d42ada3028732353516 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. 2018-08-28 16:01:12.705 1092436 INFO nova.osapi_compute.wsgi.server [req-6a17a629-6bc5-4297-90b4-524dd96b5694 b0dc354f61bb4c88ab5c544079f98c2b cbe025541b814d42ada3028732353516 - default default] 18.5.253.11 "GET /v2.1/servers/detail HTTP/1.1" status: 500 len: 626 time: 120.9522789 When I log in to the dashboard, I have to wait a long time and report the same error. I have checked the database configuration of nova.conf。 if anyone have or had this problem and can propose a fix i would appreciate it, thanks in advance. Note: i could
[Yahoo-eng-team] [Bug 1777157] Re: cold migration fails for ceph volume instances
** Changed in: nova
Status: Invalid => Confirmed
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1777157
Title:
cold migration fails for ceph volume instances
Status in OpenStack Compute (nova):
Confirmed
Bug description:
queens release,
running instance from ceph volume, in horizon: 'migrate'
Instance is in ERROR state. Horizon reports:
Error: Failed to perform requested operation on instance "instance3", the
instance has an error status: Please try again later [Error: list index out
of range].
yet another instance got:
Error: Failed to perform requested operation on instance "instance4", the
instance has an error status: Please try again later [Error: Conflict
updating instance 6b837382-2a75-46a6-9a09-8c3f90f0ffd7. Expected:
{'task_state': [u'resize_prep']}. Actual: {'task_state': None}].
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1777157/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1789322] Re: ImportError: cannot import name fake_notifier
Reviewed: https://review.openstack.org/596971 Committed: https://git.openstack.org/cgit/openstack/neutron-fwaas/commit/?id=5519d771e4a897f39f03acb4df468e531233833c Submitter: Zuul Branch:master commit 5519d771e4a897f39f03acb4df468e531233833c Author: YAMAMOTO Takashi Date: Tue Aug 28 11:28:09 2018 +0900 Switch to neutron-lib version of fake_notifier Closes-Bug: #1789322 Change-Id: I8bfc49ee9701583c2004a7475d15d66e5bd22d0f ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1789322 Title: ImportError: cannot import name fake_notifier Status in neutron: Fix Released Bug description: eg. http://logs.openstack.org/87/199387/121/check/openstack-tox- py27/9567b48/job-output.txt.gz 2018-08-27 00:50:08.241865 | ubuntu-xenial | Failed to import test module: midonet.neutron.tests.unit.test_extension_fwaas 2018-08-27 00:50:08.241970 | ubuntu-xenial | Traceback (most recent call last): 2018-08-27 00:50:08.242331 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/networking-midonet/.tox/py27/local/lib/python2.7/site-packages/unittest2/loader.py", line 456, in _find_test_path 2018-08-27 00:50:08.242478 | ubuntu-xenial | module = self._get_module_from_name(name) 2018-08-27 00:50:08.242857 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/networking-midonet/.tox/py27/local/lib/python2.7/site-packages/unittest2/loader.py", line 395, in _get_module_from_name 2018-08-27 00:50:08.242940 | ubuntu-xenial | __import__(name) 2018-08-27 00:50:08.243136 | ubuntu-xenial | File "midonet/neutron/tests/unit/test_extension_fwaas.py", line 18, in 2018-08-27 00:50:08.243339 | ubuntu-xenial | from neutron_fwaas.tests.unit.services.firewall import test_fwaas_plugin as tfp 2018-08-27 00:50:08.243793 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/networking-midonet/.tox/py27/local/lib/python2.7/site-packages/neutron_fwaas/tests/unit/services/firewall/test_fwaas_plugin.py", line 20, in 2018-08-27 00:50:08.243906 | ubuntu-xenial | from neutron.tests import fake_notifier 2018-08-27 00:50:08.244014 | ubuntu-xenial | ImportError: cannot import name fake_notifier To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1789322/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1789376] [NEW] image conversion spelling mistake vdmk
Public bug reported: In the stable/rocky branch, the image conversion refers to a format "vdmk", which I presume is actually the "vmdk" format. Enclosed is a patch, which replaces all occurrences of "vdmk" with "vmdk". ** Affects: glance Importance: Undecided Status: New ** Patch added: "glance-vdmk.patch" https://bugs.launchpad.net/bugs/1789376/+attachment/5181612/+files/glance-vdmk.patch -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1789376 Title: image conversion spelling mistake vdmk Status in Glance: New Bug description: In the stable/rocky branch, the image conversion refers to a format "vdmk", which I presume is actually the "vmdk" format. Enclosed is a patch, which replaces all occurrences of "vdmk" with "vmdk". To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1789376/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1789378] [NEW] quotasv2_detail quota driver check
Public bug reported: Currently, the quotasv2_detail extension only used for quota driver neutron.db.quota.driver.DbQuotaDriver, because there is a check in extensions/quotasv2_detail.py file, see below class Quotasv2_detail(api_extensions.ExtensionDescriptor): """Quota details management support.""" # Ensure new extension is not loaded with old conf driver. extensions.register_custom_supported_check( ALIAS, lambda: True if QUOTA_DRIVER == DB_QUOTA_DRIVER else False, plugin_agnostic=True) It's better to remove this check to make quotasv2_detail work with quota drivers other then neutron.db.quota.driver.DbQuotaDriver ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1789378 Title: quotasv2_detail quota driver check Status in neutron: New Bug description: Currently, the quotasv2_detail extension only used for quota driver neutron.db.quota.driver.DbQuotaDriver, because there is a check in extensions/quotasv2_detail.py file, see below class Quotasv2_detail(api_extensions.ExtensionDescriptor): """Quota details management support.""" # Ensure new extension is not loaded with old conf driver. extensions.register_custom_supported_check( ALIAS, lambda: True if QUOTA_DRIVER == DB_QUOTA_DRIVER else False, plugin_agnostic=True) It's better to remove this check to make quotasv2_detail work with quota drivers other then neutron.db.quota.driver.DbQuotaDriver To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1789378/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1789351] [NEW] Glance deployment with python3 + "keystone" paste_deploy flavor Fails
Public bug reported: This happens with oslo.config >= 6.3.0([1]) + python3 + "keystone" paste_deploy + current glance(before https://review.openstack.org/#/c/532503/10/glance/common/store_utils.py@30 it works) Testing in devstack: https://review.openstack.org/#/c/596380/ The glance api service fails to start with below Error, reproducing here: https://review.openstack.org/#/c/596380/:- ERROR: dictionary changed size during iteration , see logs below Failure logs from job:- http://logs.openstack.org/80/596380/2/check /tempest-full- py3/514fa29/controller/logs/screen-g-api.txt.gz#_Aug_27_07_26_10_698243 The Runtime Error is returned at keystonemiddleware:- https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token/__init__.py#L551 Adding code snippet here:- if self._conf.oslo_conf_obj != cfg.CONF: <-- Fails here oslo_cache.configure(self._conf.oslo_conf_obj) So with pdb found that an additional key(fatal_deprecations) was added to cfg.CONF at ^^, so Error is returned in python3. With python2 same key is added but no Error. There are multiple ways to avoid it, like use the paste_deploy configuration that works(ex: keystone+cachemanagement), use oslo.config <= 6.2.0, Use python2 or update glance(https://review.openstack.org/#/c/532503/10/glance/common/store_utils.py@30 as use_user_token is deprecated since long) with keystone+cachemanagement, all the config items were added before reaching the Failure point in keystonemiddleware and self._conf.oslo_conf_obj != cfg.CONF didn't raised an error and returned Boolean. Don't know why. But it seems a real issue to me as it may happen in python3 at different places. So it would be good if Teams from affected projects(oslo.config, keystonemiddleware, glance) can look at it and fix(not avoid) at the best place. To me it looks like keystonemiddleware is not handling(comparing the dict) it properly for python3, as the conf is dynamically updated(how ? and when ?). - so can oslo.config Team check if glance and keystonmiddleware are handling/using oslo.config properly. - i checked keystone+cachemanagement is default in devstack from last 6 years, is "keystone" flavor supported? if yes it should be fixed. Also it would be good to cleanup the deprecated options those are deprecated since Mitaka. - If it's wrongly used in keystonemiddleware/glance, it would be good to fix there. Initially detected while testing with Fedora[2], but later digged on why it's working in CI with Ubuntu and started [3]. [1] https://review.openstack.org/#/c/560094/ [2] https://review.rdoproject.org/r/#/c/14921/ [3] https://review.openstack.org/#/c/596380/ ** Affects: glance Importance: Undecided Status: New ** Affects: keystonemiddleware Importance: Undecided Status: New ** Affects: oslo.config Importance: Undecided Status: New ** Also affects: oslo.config Importance: Undecided Status: New ** Also affects: keystonemiddleware Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1789351 Title: Glance deployment with python3 + "keystone" paste_deploy flavor Fails Status in Glance: New Status in keystonemiddleware: New Status in oslo.config: New Bug description: This happens with oslo.config >= 6.3.0([1]) + python3 + "keystone" paste_deploy + current glance(before https://review.openstack.org/#/c/532503/10/glance/common/store_utils.py@30 it works) Testing in devstack: https://review.openstack.org/#/c/596380/ The glance api service fails to start with below Error, reproducing here: https://review.openstack.org/#/c/596380/:- ERROR: dictionary changed size during iteration , see logs below Failure logs from job:- http://logs.openstack.org/80/596380/2/check /tempest-full- py3/514fa29/controller/logs/screen-g-api.txt.gz#_Aug_27_07_26_10_698243 The Runtime Error is returned at keystonemiddleware:- https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token/__init__.py#L551 Adding code snippet here:- if self._conf.oslo_conf_obj != cfg.CONF: <-- Fails here oslo_cache.configure(self._conf.oslo_conf_obj) So with pdb found that an additional key(fatal_deprecations) was added to cfg.CONF at ^^, so Error is returned in python3. With python2 same key is added but no Error. There are multiple ways to avoid it, like use the paste_deploy configuration that works(ex: keystone+cachemanagement), use oslo.config <= 6.2.0, Use python2 or update glance(https://review.openstack.org/#/c/532503/10/glance/common/store_utils.py@30 as use_user_token is deprecated since long) with keystone+cachemanagement, all the config items were added before reaching the Failure point in keystonemiddleware and self._conf.oslo_conf_obj != cfg.CONF didn't raised an error and returned
[Yahoo-eng-team] [Bug 1789340] [NEW] Nova boot server failed due to exception "Cannot complete the operation because the file or folder .vmdk already exists "
Public bug reported:
Nova boot server failed due to exception "vmware_base/373ca119-5b29
-441d-bd9c-b2ccfbad/373ca119-5b29-441d-bd9c-b2ccfbad.20.vmdk
already exists"
Environment -
Openstack Pike/Queens/Master
KVM computes - 4
VCenter + ESXi computes - 6
nova/conductor.log.1:2018-02-01 18:48:42.926 5826 ERROR
nova.scheduler.utils [req-4bcfad48-2a15-4aa4-8c32-ebb3798a8b09
47b7fb99c82c413caae4081c35b7456a f2ab30de47df41e488a77258c7a1c6a1 -
default default] [instance: 1c4830b5-f1f5-4bfb-bcd9-63c67aed127f] Error
from last host: oscompute01 (node domain-c74754.bcea1667-7332-4f99-8aae-
3f2bc39c2fc9): [u'Traceback (most recent call last):\n', u' File
"/opt/mhos/openstack/nova/lib/python2.7/site-
packages/nova/compute/manager.py", line 1856, in
_do_build_and_run_instance\nfilter_properties)\n', u' File
"/opt/mhos/openstack/nova/lib/python2.7/site-
packages/nova/compute/manager.py", line 2086, in
_build_and_run_instance\ninstance_uuid=instance.uuid,
reason=six.text_type(e))\n', u'RescheduledException: Build of instance
1c4830b5-f1f5-4bfb-bcd9-63c67aed127f was re-scheduled: Cannot complete
the operation because the file or folder [datastore-nfs]
vmware_base/373ca119-5b29-441d-bd9c-b2ccfbad/373ca119-5b29-441d-
bd9c-b2ccfbad.20.vmdk already exists\n']
neutron server log,
neutron/server.log.2:2018-02-01 18:48:53.788 5838 DEBUG neutron.notifiers.nova
[-] Sending events: [{'tag': u'4c311f56-a73f-47a6-aa58-95f1a774396e', 'name':
'network-vif-deleted', 'server_uuid': u'1c4830b5-f1f5-4bfb-bcd9-63c67aed127f'}]
send_events
/opt/mhos/openstack/neutron/lib/python2.7/site-packages/neutron/notifiers/nova.py:242
neutron/server.log.2:2018-02-01 18:48:54.317 5838 DEBUG novaclient.v2.client
[-] REQ: curl -g -i -X POST
http://192.168.7.7:8774/v2.1/86d8652e2b834d29a744c6c66a5fb65e/os-server-external-events
-H "User-Agent: python-novaclient" -H "Content-Type: application/json" -H
"Accept: application/json" -H "X-OpenStack-Nova-API-Version: 2.1" -H
"X-Auth-Token: {SHA1}4befed70c1a44813879fce059e88ab310e29b039" -d '{"events":
[{"tag": "4c311f56-a73f-47a6-aa58-95f1a774396e", "name": "network-vif-deleted",
"server_uuid": "1c4830b5-f1f5-4bfb-bcd9-63c67aed127f"}]}' _http_log_request
/opt/mhos/python/lib/python2.7/site-packages/keystoneauth1/session.py:375
neutron/server.log.2:2018-02-01 18:48:54.459 5838 DEBUG neutron.notifiers.nova
[-] Nova returned NotFound for event: [{'tag':
u'4c311f56-a73f-47a6-aa58-95f1a774396e', 'name': 'network-vif-deleted',
'server_uuid': u'1c4830b5-f1f5-4bfb-bcd9-63c67aed127f'}] send_events
/opt/mhos/openstack/neutron/lib/python2.7/site-packages/neutron/notifiers/nova.py:248
neutron/server.log:2018-02-05 06:17:38.871 5840 INFO neutron.wsgi
[req-9413e35e-b050-4d88-b8c4-963b6cda07b0 d22bc5405f5841ad817bc6b75dd5b3df
ca15303f1f2d4eb8af607f76b10e8250 - default default] 192.168.1.193,192.168.1.177
"GET /v2.0/ports?device_id=1c4830b5-f1f5-4bfb-bcd9-63c67aed127f HTTP/1.1"
status: 200 len: 205 time: 0.0538130
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1789340
Title:
Nova boot server failed due to exception "Cannot complete the
operation because the file or folder .vmdk already exists "
Status in OpenStack Compute (nova):
New
Bug description:
Nova boot server failed due to exception "vmware_base/373ca119-5b29
-441d-bd9c-b2ccfbad/373ca119-5b29-441d-bd9c-b2ccfbad.20.vmdk
already exists"
Environment -
Openstack Pike/Queens/Master
KVM computes - 4
VCenter + ESXi computes - 6
nova/conductor.log.1:2018-02-01 18:48:42.926 5826 ERROR
nova.scheduler.utils [req-4bcfad48-2a15-4aa4-8c32-ebb3798a8b09
47b7fb99c82c413caae4081c35b7456a f2ab30de47df41e488a77258c7a1c6a1 -
default default] [instance: 1c4830b5-f1f5-4bfb-bcd9-63c67aed127f]
Error from last host: oscompute01 (node
domain-c74754.bcea1667-7332-4f99-8aae-3f2bc39c2fc9): [u'Traceback
(most recent call last):\n', u' File
"/opt/mhos/openstack/nova/lib/python2.7/site-
packages/nova/compute/manager.py", line 1856, in
_do_build_and_run_instance\nfilter_properties)\n', u' File
"/opt/mhos/openstack/nova/lib/python2.7/site-
packages/nova/compute/manager.py", line 2086, in
_build_and_run_instance\ninstance_uuid=instance.uuid,
reason=six.text_type(e))\n', u'RescheduledException: Build of instance
1c4830b5-f1f5-4bfb-bcd9-63c67aed127f was re-scheduled: Cannot complete
the operation because the file or folder [datastore-nfs]
vmware_base/373ca119-5b29-441d-bd9c-b2ccfbad/373ca119-5b29-441d-
bd9c-b2ccfbad.20.vmdk already exists\n']
neutron server log,
neutron/server.log.2:2018-02-01 18:48:53.788 5838 DEBUG
neutron.notifiers.nova [-] Sending events: [{'tag':
u'4c311f56-a73f-47a6-aa58-95f1a774396e', 'name': 'network-vif-deleted',
'server_uuid':
