I can also see it's timing out waiting for vif plugging callback in the
logs i.e "Timeout waiting for vif plugging callback for instance
9456073b-a8d6-44ff-b86a-620fbc68a484". Only seen these failures for jobs
in in osic cloud. Is it just a coincidence or something to do with the
infra?
** Also affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1694371
Title:
test_stack_snapshot_restore failing intermittently
Status in heat:
New
Status in OpenStack Compute (nova):
New
Bug description:
It seems like the server is going to ERROR state during rebuild.
traceback:
2017-05-28 04:34:37.767757 | 2017-05-28 04:34:37.767 | Captured traceback:
2017-05-28 04:34:37.768828 | 2017-05-28 04:34:37.768 | ~~~~~~~~~~~~~~~~~~~
2017-05-28 04:34:37.770099 | 2017-05-28 04:34:37.769 | b'Traceback (most
recent call last):'
2017-05-28 04:34:37.771108 | 2017-05-28 04:34:37.770 | b' File
"/opt/stack/new/heat/heat_integrationtests/functional/test_snapshot_restore.py",
line 74, in test_stack_snapshot_restore'
2017-05-28 04:34:37.772364 | 2017-05-28 04:34:37.772 | b'
self.stack_restore(stack_identifier, snapshot_id)'
2017-05-28 04:34:37.773407 | 2017-05-28 04:34:37.773 | b' File
"/opt/stack/new/heat/heat_integrationtests/common/test.py", line 626, in
stack_restore'
2017-05-28 04:34:37.774541 | 2017-05-28 04:34:37.774 | b'
self._wait_for_stack_status(stack_id, wait_for_status)'
2017-05-28 04:34:37.775568 | 2017-05-28 04:34:37.775 | b' File
"/opt/stack/new/heat/heat_integrationtests/common/test.py", line 357, in
_wait_for_stack_status'
2017-05-28 04:34:37.776642 | 2017-05-28 04:34:37.776 | b'
fail_regexp):'
2017-05-28 04:34:37.778354 | 2017-05-28 04:34:37.778 | b' File
"/opt/stack/new/heat/heat_integrationtests/common/test.py", line 321, in
_verify_status'
2017-05-28 04:34:37.779448 | 2017-05-28 04:34:37.779 | b'
stack_status_reason=stack.stack_status_reason)'
2017-05-28 04:34:37.780644 | 2017-05-28 04:34:37.780 |
b"heat_integrationtests.common.exceptions.StackBuildErrorException: Stack
StackSnapshotRestoreTest-1374582671/7fb8f800-1545-4e34-a6fa-3e2adbf4443a is in
RESTORE_FAILED status due to 'Error: resources.my_server: Rebuilding server
failed, status 'ERROR''"
2017-05-28 04:34:37.782119 | 2017-05-28 04:34:37.781 | b''
Noticed at:
http://logs.openstack.org/16/462216/16/check/gate-heat-dsvm-
functional-convg-mysql-lbaasv2-py35-ubuntu-
xenial/17c2da9/console.html#_2017-05-28_04_34_37_753094
Looks like a nova issue from the below traceback.
http://logs.openstack.org/16/462216/16/check/gate-heat-dsvm-
functional-convg-mysql-lbaasv2-py35-ubuntu-
xenial/17c2da9/logs/screen-n-cpu.txt.gz?level=ERROR#_May_28_04_14_49_044455
May 28 04:14:49.042877 ubuntu-xenial-osic-cloud1-s3700-9024798
nova-compute[26709]: ERROR nova.compute.manager [instance:
45105d34-b970-4ced-968c-a1c4ead5b282] File
"/opt/stack/new/nova/nova/compute/manager.py", line 6758, in
_error_out_instance_on_exception
May 28 04:14:49.042955 ubuntu-xenial-osic-cloud1-s3700-9024798
nova-compute[26709]: ERROR nova.compute.manager [instance:
45105d34-b970-4ced-968c-a1c4ead5b282] yield
May 28 04:14:49.043027 ubuntu-xenial-osic-cloud1-s3700-9024798
nova-compute[26709]: ERROR nova.compute.manager [instance:
45105d34-b970-4ced-968c-a1c4ead5b282] File
"/opt/stack/new/nova/nova/compute/manager.py", line 2814, in rebuild_instance
May 28 04:14:49.043100 ubuntu-xenial-osic-cloud1-s3700-9024798
nova-compute[26709]: ERROR nova.compute.manager [instance:
45105d34-b970-4ced-968c-a1c4ead5b282] bdms, recreate, on_shared_storage,
preserve_ephemeral)
May 28 04:14:49.043197 ubuntu-xenial-osic-cloud1-s3700-9024798
nova-compute[26709]: ERROR nova.compute.manager [instance:
45105d34-b970-4ced-968c-a1c4ead5b282] File
"/opt/stack/new/nova/nova/compute/manager.py", line 2855, in
_do_rebuild_instance_with_claim
May 28 04:14:49.043299 ubuntu-xenial-osic-cloud1-s3700-9024798
nova-compute[26709]: ERROR nova.compute.manager [instance:
45105d34-b970-4ced-968c-a1c4ead5b282] self._do_rebuild_instance(*args,
**kwargs)
May 28 04:14:49.043384 ubuntu-xenial-osic-cloud1-s3700-9024798
nova-compute[26709]: ERROR nova.compute.manager [instance:
45105d34-b970-4ced-968c-a1c4ead5b282] File
"/opt/stack/new/nova/nova/compute/manager.py", line 2977, in
_do_rebuild_instance
May 28 04:14:49.043458 ubuntu-xenial-osic-cloud1-s3700-9024798
nova-compute[26709]: ERROR nova.compute.manager [instance:
45105d34-b970-4ced-968c-a1c4ead5b282] self._rebuild_default_impl(**kwargs)
May 28 04:14:49.043534 ubuntu-xenial-osic-cloud1-s3700-9024798
nova-compute[26709]: ERROR nova.compute.manager [instance:
45105d34-b970-4ced-968c-a1c4ead5b282] File
"/opt/stack/new/nova/nova/compute/manager.py", line 2714, in
_rebuild_default_impl
May 28 04:14:49.043611 ubuntu-xenial-osic-cloud1-s3700-9024798
nova-compute[26709]: ERROR nova.compute.manager [instance:
45105d34-b970-4ced-968c-a1c4ead5b282]
block_device_info=new_block_device_info)
May 28 04:14:49.044132 ubuntu-xenial-osic-cloud1-s3700-9024798
nova-compute[26709]: ERROR nova.compute.manager [instance:
45105d34-b970-4ced-968c-a1c4ead5b282] File
"/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2741, in spawn
May 28 04:14:49.044223 ubuntu-xenial-osic-cloud1-s3700-9024798
nova-compute[26709]: ERROR nova.compute.manager [instance:
45105d34-b970-4ced-968c-a1c4ead5b282] destroy_disks_on_failure=True)
May 28 04:14:49.044299 ubuntu-xenial-osic-cloud1-s3700-9024798
nova-compute[26709]: ERROR nova.compute.manager [instance:
45105d34-b970-4ced-968c-a1c4ead5b282] File
"/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5152, in
_create_domain_and_network
May 28 04:14:49.044374 ubuntu-xenial-osic-cloud1-s3700-9024798
nova-compute[26709]: ERROR nova.compute.manager [instance:
45105d34-b970-4ced-968c-a1c4ead5b282] raise
exception.VirtualInterfaceCreateException()
May 28 04:14:49.044455 ubuntu-xenial-osic-cloud1-s3700-9024798
nova-compute[26709]: ERROR nova.compute.manager [instance:
45105d34-b970-4ced-968c-a1c4ead5b282]
nova.exception.VirtualInterfaceCreateException: Virtual Interface creation
failed
To manage notifications about this bug go to:
https://bugs.launchpad.net/heat/+bug/1694371/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
