I can also see it's timing out waiting for vif plugging callback in the logs i.e "Timeout waiting for vif plugging callback for instance 9456073b-a8d6-44ff-b86a-620fbc68a484". Only seen these failures for jobs in in osic cloud. Is it just a coincidence or something to do with the infra?
** Also affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1694371 Title: test_stack_snapshot_restore failing intermittently Status in heat: New Status in OpenStack Compute (nova): New Bug description: It seems like the server is going to ERROR state during rebuild. traceback: 2017-05-28 04:34:37.767757 | 2017-05-28 04:34:37.767 | Captured traceback: 2017-05-28 04:34:37.768828 | 2017-05-28 04:34:37.768 | ~~~~~~~~~~~~~~~~~~~ 2017-05-28 04:34:37.770099 | 2017-05-28 04:34:37.769 | b'Traceback (most recent call last):' 2017-05-28 04:34:37.771108 | 2017-05-28 04:34:37.770 | b' File "/opt/stack/new/heat/heat_integrationtests/functional/test_snapshot_restore.py", line 74, in test_stack_snapshot_restore' 2017-05-28 04:34:37.772364 | 2017-05-28 04:34:37.772 | b' self.stack_restore(stack_identifier, snapshot_id)' 2017-05-28 04:34:37.773407 | 2017-05-28 04:34:37.773 | b' File "/opt/stack/new/heat/heat_integrationtests/common/test.py", line 626, in stack_restore' 2017-05-28 04:34:37.774541 | 2017-05-28 04:34:37.774 | b' self._wait_for_stack_status(stack_id, wait_for_status)' 2017-05-28 04:34:37.775568 | 2017-05-28 04:34:37.775 | b' File "/opt/stack/new/heat/heat_integrationtests/common/test.py", line 357, in _wait_for_stack_status' 2017-05-28 04:34:37.776642 | 2017-05-28 04:34:37.776 | b' fail_regexp):' 2017-05-28 04:34:37.778354 | 2017-05-28 04:34:37.778 | b' File "/opt/stack/new/heat/heat_integrationtests/common/test.py", line 321, in _verify_status' 2017-05-28 04:34:37.779448 | 2017-05-28 04:34:37.779 | b' stack_status_reason=stack.stack_status_reason)' 2017-05-28 04:34:37.780644 | 2017-05-28 04:34:37.780 | b"heat_integrationtests.common.exceptions.StackBuildErrorException: Stack StackSnapshotRestoreTest-1374582671/7fb8f800-1545-4e34-a6fa-3e2adbf4443a is in RESTORE_FAILED status due to 'Error: resources.my_server: Rebuilding server failed, status 'ERROR''" 2017-05-28 04:34:37.782119 | 2017-05-28 04:34:37.781 | b'' Noticed at: http://logs.openstack.org/16/462216/16/check/gate-heat-dsvm- functional-convg-mysql-lbaasv2-py35-ubuntu- xenial/17c2da9/console.html#_2017-05-28_04_34_37_753094 Looks like a nova issue from the below traceback. http://logs.openstack.org/16/462216/16/check/gate-heat-dsvm- functional-convg-mysql-lbaasv2-py35-ubuntu- xenial/17c2da9/logs/screen-n-cpu.txt.gz?level=ERROR#_May_28_04_14_49_044455 May 28 04:14:49.042877 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] File "/opt/stack/new/nova/nova/compute/manager.py", line 6758, in _error_out_instance_on_exception May 28 04:14:49.042955 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] yield May 28 04:14:49.043027 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] File "/opt/stack/new/nova/nova/compute/manager.py", line 2814, in rebuild_instance May 28 04:14:49.043100 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] bdms, recreate, on_shared_storage, preserve_ephemeral) May 28 04:14:49.043197 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] File "/opt/stack/new/nova/nova/compute/manager.py", line 2855, in _do_rebuild_instance_with_claim May 28 04:14:49.043299 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] self._do_rebuild_instance(*args, **kwargs) May 28 04:14:49.043384 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] File "/opt/stack/new/nova/nova/compute/manager.py", line 2977, in _do_rebuild_instance May 28 04:14:49.043458 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] self._rebuild_default_impl(**kwargs) May 28 04:14:49.043534 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] File "/opt/stack/new/nova/nova/compute/manager.py", line 2714, in _rebuild_default_impl May 28 04:14:49.043611 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] block_device_info=new_block_device_info) May 28 04:14:49.044132 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2741, in spawn May 28 04:14:49.044223 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] destroy_disks_on_failure=True) May 28 04:14:49.044299 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5152, in _create_domain_and_network May 28 04:14:49.044374 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] raise exception.VirtualInterfaceCreateException() May 28 04:14:49.044455 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed To manage notifications about this bug go to: https://bugs.launchpad.net/heat/+bug/1694371/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp