Work around in the charm committed here: https://review.opendev.org/#/c/755089 https://review.opendev.org/#/c/755076/
** Also affects: charm-keystone Importance: Undecided Status: New ** Also affects: charm-nova-cloud-controller Importance: Undecided Status: New ** Changed in: nova Assignee: Liam Young (gnuoy) => (unassigned) ** Changed in: charm-keystone Assignee: (unassigned) => Liam Young (gnuoy) ** Changed in: charm-nova-cloud-controller Assignee: (unassigned) => Liam Young (gnuoy) ** Changed in: charm-nova-compute Status: Triaged => Invalid ** Changed in: charm-keystone Status: New => Fix Committed ** Changed in: charm-nova-cloud-controller Status: New => Fix Committed ** Changed in: charm-keystone Importance: Undecided => High ** Changed in: charm-nova-cloud-controller Importance: Undecided => High -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1826382 Title: Updates to placement api fail if placement endpoint changes Status in OpenStack keystone charm: Fix Committed Status in OpenStack nova-cloud-controller charm: Fix Committed Status in OpenStack nova-compute charm: Invalid Status in OpenStack Compute (nova): Triaged Bug description: If the url of the placement api changes after nova-compute has been started then placement updates fail as nova-compute appears to cache the old endpoint url. To reproduce, update the placement endpoint to something incorrect in keystone and restart nova-compute. Errors contacting the placement api will be reported every minute or so. Now, correct the entry in keystone. The errors will continue despite the catalogue now being correct. Restarting nova-compute fixes the issue. In my deployment this occurred when the placement end point switched from http to https after the nova-compute node had started. This resulted in the following in the nova-compute log: 2019-04-25 09:58:12.175 31793 ERROR nova.scheduler.client.report [req-18b4f522-e702-4ee1-ba85-e565c8e9ac1e - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 4f7c6844-d3b8-4710-be2c-8691a93fb58b. Got 400: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>400 Bad Request</title> </head><body> <h1>Bad Request</h1> <p>Your browser sent a request that this server could not understand.<br /> Reason: You're speaking plain HTTP to an SSL-enabled server port.<br /> Instead use the HTTPS scheme to access this URL, please.<br /> </p> <hr> <address>Apache/2.4.29 (Ubuntu) Server at 10.5.0.36 Port 443</address> </body></html> . 2019-04-25 09:58:12.176 31793 DEBUG oslo_concurrency.lockutils [req-18b4f522-e702-4ee1-ba85-e565c8e9ac1e - - - - -] Lock "compute_resources" released by "nova.compute.resource_tracker.ResourceTracker._update_available_resource" :: held 0.099s inner /usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py:285 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager [req-18b4f522-e702-4ee1-ba85-e565c8e9ac1e - - - - -] Error updating resources for node juju-7a9f5c-zaza-19a393f3689b-16.project.serverstack.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 4f7c6844-d3b8-4710-be2c-8691a93fb58b 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager Traceback (most recent call last): 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 7778, in _update_available_resource_for_node 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager rt.update_available_resource(context, nodename) 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 721, in update_available_resource 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager self._update_available_resource(context, resources) 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager return f(*args, **kwargs) 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 798, in _update_available_resource 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager self._update(context, cn) 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw) 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/retrying.py", line 206, in call 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager return attempt.get(self._wrap_exception) 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2]) 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager raise value 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False) 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 960, in _update 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager self._update_to_placement(context, compute_node) 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 912, in _update_to_placement 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager context, compute_node.uuid, name=compute_node.hypervisor_hostname) 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/scheduler/client/__init__.py", line 35, in __run_method 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager return getattr(self.instance, __name)(*args, **kwargs) 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 1006, in get_provider_tree_and_ensure_root 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid) 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 668, in _ensure_resource_provider 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager rps_to_refresh = self._get_providers_in_tree(context, uuid) 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 74, in wrapper 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager return f(self, *a, **k) 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 535, in _get_providers_in_tree 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager raise exception.ResourceProviderRetrievalFailed(uuid=uuid) 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 4f7c6844-d3b8-4710-be2c-8691a93fb58b 2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager To manage notifications about this bug go to: https://bugs.launchpad.net/charm-keystone/+bug/1826382/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp