Public bug reported: Seeing this on start of nova-compute with ironic when ironic-api isn't yet available:
Aug 24 01:06:39.710754 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR nova.virt.ironic.driver [None req- 9542c6c8-a038-45f5-bd18-e18f83c17755 None None] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: StrictVersion instance has no attribute 'version' Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task [None req- 9542c6c8-a038-45f5-bd18-e18f83c17755 None None] Error during ComputeManager._cleanup_running_deleted_instances: VirtDriverNotReady: Virt driver is not ready. Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task Traceback (most recent call last): Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task File "/usr/local/lib/python2.7/dist-packages/oslo_service/periodic_task.py", line 222, in run_periodic_tasks Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task task(self, context) Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task File "/opt/stack/nova/nova/compute/manager.py", line 8369, in _cleanup_running_deleted_instances Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task for instance in self._running_deleted_instances(context): Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task File "/opt/stack/nova/nova/compute/manager.py", line 8423, in _running_deleted_instances Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task instances = self._get_instances_on_driver(context, filters) Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task File "/opt/stack/nova/nova/compute/manager.py", line 634, in _get_instances_on_driver Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task driver_uuids = self.driver.list_instance_uuids() Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task File "/opt/stack/nova/nova/virt/ironic/driver.py", line 685, in list_instance_uuids Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task fields=['instance_uuid'], limit=0) Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task File "/opt/stack/nova/nova/virt/ironic/driver.py", line 656, in _get_node_list Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task raise exception.VirtDriverNotReady() Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task VirtDriverNotReady: Virt driver is not ready. Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task Looks like this is due to https://review.opendev.org/#/c/657132/ in Train where the _cleanup_running_deleted_instances periodic task runs immediately on startup of the nova-compute service which could be before the hypervisor (in this case ironic) is ready. This doesn't really break anything, but it's an ugly traceback in the logs that could be avoided. We should handle the VirtDriverNotReady error and return from the periodic. ** Affects: nova Importance: Low Assignee: Matt Riedemann (mriedem) Status: Triaged ** Tags: compute ironic -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1842081 Title: Error during ComputeManager._cleanup_running_deleted_instances: VirtDriverNotReady: Virt driver is not ready. (ironic) Status in OpenStack Compute (nova): Triaged Bug description: Seeing this on start of nova-compute with ironic when ironic-api isn't yet available: Aug 24 01:06:39.710754 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR nova.virt.ironic.driver [None req- 9542c6c8-a038-45f5-bd18-e18f83c17755 None None] An unknown error has occurred when trying to get the list of nodes from the Ironic inventory. Error: StrictVersion instance has no attribute 'version' Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task [None req- 9542c6c8-a038-45f5-bd18-e18f83c17755 None None] Error during ComputeManager._cleanup_running_deleted_instances: VirtDriverNotReady: Virt driver is not ready. Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task Traceback (most recent call last): Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task File "/usr/local/lib/python2.7/dist- packages/oslo_service/periodic_task.py", line 222, in run_periodic_tasks Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task task(self, context) Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task File "/opt/stack/nova/nova/compute/manager.py", line 8369, in _cleanup_running_deleted_instances Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task for instance in self._running_deleted_instances(context): Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task File "/opt/stack/nova/nova/compute/manager.py", line 8423, in _running_deleted_instances Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task instances = self._get_instances_on_driver(context, filters) Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task File "/opt/stack/nova/nova/compute/manager.py", line 634, in _get_instances_on_driver Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task driver_uuids = self.driver.list_instance_uuids() Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task File "/opt/stack/nova/nova/virt/ironic/driver.py", line 685, in list_instance_uuids Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task fields=['instance_uuid'], limit=0) Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task File "/opt/stack/nova/nova/virt/ironic/driver.py", line 656, in _get_node_list Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task raise exception.VirtDriverNotReady() Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task VirtDriverNotReady: Virt driver is not ready. Aug 24 01:06:39.711672 ubuntu-bionic-rax-iad-0010410623 nova- compute[7945]: ERROR oslo_service.periodic_task Looks like this is due to https://review.opendev.org/#/c/657132/ in Train where the _cleanup_running_deleted_instances periodic task runs immediately on startup of the nova-compute service which could be before the hypervisor (in this case ironic) is ready. This doesn't really break anything, but it's an ugly traceback in the logs that could be avoided. We should handle the VirtDriverNotReady error and return from the periodic. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1842081/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
