Reviewed: https://review.opendev.org/679473
Committed:
https://git.openstack.org/cgit/openstack/nova/commit/?id=3cef7edbc2dd72a25e46ca1962731ee3153f1fd7
Submitter: Zuul
Branch: master
commit 3cef7edbc2dd72a25e46ca1962731ee3153f1fd7
Author: Matt Riedemann <mriedem...@gmail.com>
Date: Fri Aug 30 12:11:59 2019 -0400
Handle VirtDriverNotReady in _cleanup_running_deleted_instances
Change I6e086e0a2bffe12fd78373b3bc6215221dabc77e made the
_cleanup_running_deleted_instances task run immediately on
start of the nova-compute service. For some compute drivers, like
ironic, the "hypervisor" might not yet be ready to list instances
which results in a traceback exception in the logs.
This simply handles the VirtDriverNotReady error and returns to
avoid the traceback.
Change-Id: I6220a59988502a9839645efa101d555c19d4b650
Closes-Bug: #1842081
** Changed in: nova
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1842081
Title:
Error during ComputeManager._cleanup_running_deleted_instances:
VirtDriverNotReady: Virt driver is not ready. (ironic)
Status in OpenStack Compute (nova):
Fix Released
Bug description:
Seeing this on start of nova-compute with ironic when ironic-api isn't
yet available:
Aug 24 01:06:39.710754 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR nova.virt.ironic.driver [None req-
9542c6c8-a038-45f5-bd18-e18f83c17755 None None] An unknown error has
occurred when trying to get the list of nodes from the Ironic
inventory. Error: StrictVersion instance has no attribute 'version'
Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task [None req-
9542c6c8-a038-45f5-bd18-e18f83c17755 None None] Error during
ComputeManager._cleanup_running_deleted_instances: VirtDriverNotReady:
Virt driver is not ready.
Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task Traceback (most recent
call last):
Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task File
"/usr/local/lib/python2.7/dist-
packages/oslo_service/periodic_task.py", line 222, in
run_periodic_tasks
Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task task(self,
context)
Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task File
"/opt/stack/nova/nova/compute/manager.py", line 8369, in
_cleanup_running_deleted_instances
Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task for instance in
self._running_deleted_instances(context):
Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task File
"/opt/stack/nova/nova/compute/manager.py", line 8423, in
_running_deleted_instances
Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task instances =
self._get_instances_on_driver(context, filters)
Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task File
"/opt/stack/nova/nova/compute/manager.py", line 634, in
_get_instances_on_driver
Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task driver_uuids =
self.driver.list_instance_uuids()
Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task File
"/opt/stack/nova/nova/virt/ironic/driver.py", line 685, in
list_instance_uuids
Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task
fields=['instance_uuid'], limit=0)
Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task File
"/opt/stack/nova/nova/virt/ironic/driver.py", line 656, in
_get_node_list
Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task raise
exception.VirtDriverNotReady()
Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task VirtDriverNotReady:
Virt driver is not ready.
Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova-
compute[7945]: ERROR oslo_service.periodic_task
Looks like this is due to https://review.opendev.org/#/c/657132/ in
Train where the _cleanup_running_deleted_instances periodic task runs
immediately on startup of the nova-compute service which could be
before the hypervisor (in this case ironic) is ready.
This doesn't really break anything, but it's an ugly traceback in the
logs that could be avoided. We should handle the VirtDriverNotReady
error and return from the periodic.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1842081/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
