Public bug reported: Description ===========
Looks like a race between libvirtd restarting and the evacuation test running? https://zuul.opendev.org/t/openstack/build/c61ba346d3dc4677bb4a0a1520fec054/log/job- output.txt#62168 2021-09-29 08:20:51.046250 | controller | + /opt/stack/nova/roles/run- evacuate-hook/files/test_evacuate.sh:evacuate_and_wait_for_active:34 : echo 'Timed out waiting for server evacuate-bfv-test to go to ACTIVE status' https://zuul.opendev.org/t/openstack/build/c61ba346d3dc4677bb4a0a1520fec054/log/controller/logs/screen- n-cpu.txt#11648 Sep 29 08:09:48.352891 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: INFO nova.compute.manager [None req-a05f588d-c1e4-46d1-a129-539b9850302e demo admin] [instance: b0498f73-aaf2-436f-9db2-921774403901] Evacuating instance [..] Sep 29 08:09:48.444807 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: DEBUG nova.virt.libvirt.host [None req-a05f588d-c1e4-46d1-a129-539b9850302e demo admin] Connecting to libvirt: qemu:///system {{(pid=107626) _get_new_connection /opt/stack/nova/nova/virt/libvirt/host.py:499}} Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host [None req-a05f588d-c1e4-46d1-a129-539b9850302e demo admin] Connection to libvirt failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused: libvirt.libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host Traceback (most recent call last): Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/opt/stack/nova/nova/virt/libvirt/host.py", line 586, in get_connection Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host conn = self._get_connection() Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/opt/stack/nova/nova/virt/libvirt/host.py", line 566, in _get_connection Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host self._queue_conn_event_handler( Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host self.force_reraise() Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host raise self.value Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/opt/stack/nova/nova/virt/libvirt/host.py", line 558, in _get_connection Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host self._wrapped_conn = self._get_new_connection() Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/opt/stack/nova/nova/virt/libvirt/host.py", line 502, in _get_new_connection Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host wrapped_conn = self._connect(self._uri, self._read_only) Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/opt/stack/nova/nova/virt/libvirt/host.py", line 353, in _connect Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host return self._libvirt_proxy.openAuth(uri, auth, flags) Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 193, in doit Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host result = proxy_call(self._autowrap, f, *args, **kwargs) Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 151, in proxy_call Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host rv = execute(f, *args, **kwargs) Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 132, in execute Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host six.reraise(c, e, tb) Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python3.8/dist-packages/six.py", line 719, in reraise Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host raise value Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 86, in tworker Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host rv = meth(*args, **kwargs) Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/usr/lib/python3/dist-packages/libvirt.py", line 104, in openAuth Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host if ret is None:raise libvirtError('virConnectOpenAuth() failed') Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host libvirt.libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused ** Affects: nova Importance: Undecided Assignee: Lee Yarwood (lyarwood) Status: New ** Tags: evacuate gate-failure -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1945608 Title: Timed out waiting for server evacuate-bfv-test to go to ACTIVE status Status in OpenStack Compute (nova): New Bug description: Description =========== Looks like a race between libvirtd restarting and the evacuation test running? https://zuul.opendev.org/t/openstack/build/c61ba346d3dc4677bb4a0a1520fec054/log/job- output.txt#62168 2021-09-29 08:20:51.046250 | controller | + /opt/stack/nova/roles/run- evacuate-hook/files/test_evacuate.sh:evacuate_and_wait_for_active:34 : echo 'Timed out waiting for server evacuate-bfv-test to go to ACTIVE status' https://zuul.opendev.org/t/openstack/build/c61ba346d3dc4677bb4a0a1520fec054/log/controller/logs/screen- n-cpu.txt#11648 Sep 29 08:09:48.352891 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: INFO nova.compute.manager [None req-a05f588d-c1e4-46d1-a129-539b9850302e demo admin] [instance: b0498f73-aaf2-436f-9db2-921774403901] Evacuating instance [..] Sep 29 08:09:48.444807 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: DEBUG nova.virt.libvirt.host [None req-a05f588d-c1e4-46d1-a129-539b9850302e demo admin] Connecting to libvirt: qemu:///system {{(pid=107626) _get_new_connection /opt/stack/nova/nova/virt/libvirt/host.py:499}} Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host [None req-a05f588d-c1e4-46d1-a129-539b9850302e demo admin] Connection to libvirt failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused: libvirt.libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host Traceback (most recent call last): Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/opt/stack/nova/nova/virt/libvirt/host.py", line 586, in get_connection Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host conn = self._get_connection() Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/opt/stack/nova/nova/virt/libvirt/host.py", line 566, in _get_connection Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host self._queue_conn_event_handler( Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host self.force_reraise() Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host raise self.value Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/opt/stack/nova/nova/virt/libvirt/host.py", line 558, in _get_connection Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host self._wrapped_conn = self._get_new_connection() Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/opt/stack/nova/nova/virt/libvirt/host.py", line 502, in _get_new_connection Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host wrapped_conn = self._connect(self._uri, self._read_only) Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/opt/stack/nova/nova/virt/libvirt/host.py", line 353, in _connect Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host return self._libvirt_proxy.openAuth(uri, auth, flags) Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 193, in doit Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host result = proxy_call(self._autowrap, f, *args, **kwargs) Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 151, in proxy_call Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host rv = execute(f, *args, **kwargs) Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 132, in execute Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host six.reraise(c, e, tb) Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python3.8/dist-packages/six.py", line 719, in reraise Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host raise value Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 86, in tworker Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host rv = meth(*args, **kwargs) Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host File "/usr/lib/python3/dist-packages/libvirt.py", line 104, in openAuth Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host if ret is None:raise libvirtError('virConnectOpenAuth() failed') Sep 29 08:09:48.456362 ubuntu-focal-iweb-mtl01-0026717282 nova-compute[107626]: ERROR nova.virt.libvirt.host libvirt.libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1945608/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
