** Changed in: nova
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1521340
Title:
BDMNotFound race while building resources since 11/27
Status in OpenStack Compute (nova):
Fix Released
Bug description:
http://logs.openstack.org/94/242594/2/check/gate-tempest-dsvm-
nova-v20-api/085f243/logs/screen-n-cpu.txt.gz?level=TRACE#_2015-11-30_18_03_48_003
2015-11-30 18:03:48.003 ERROR nova.compute.manager
[req-b3bc5257-4673-4850-9ba0-e1e136c131b3
tempest-DeleteServersTestJSON-651179004
tempest-DeleteServersTestJSON-684647951] [instance:
75986322-d59d-411c-a60c-776415cdd6d1] Failure prepping block device
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] Traceback (most recent call last):
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] File
"/opt/stack/new/nova/nova/compute/manager.py", line 2132, in _build_resources
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] block_device_mapping)
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] File
"/opt/stack/new/nova/nova/compute/manager.py", line 1689, in
_default_block_device_names
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] root_bdm.save()
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
206, in wrapper
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] ctxt, self, fn.__name__, args, kwargs)
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] File
"/opt/stack/new/nova/nova/conductor/rpcapi.py", line 246, in object_action
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] objmethod=objmethod, args=args,
kwargs=kwargs)
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line
158, in call
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] retry=self.retry)
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90,
in _send
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] timeout=timeout, retry=retry)
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
line 464, in send
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] retry=retry)
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
line 455, in _send
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] raise result
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] BDMNotFound_Remote: No Block Device
Mapping with id 48.
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] Traceback (most recent call last):
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1]
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] File
"/opt/stack/new/nova/nova/conductor/manager.py", line 85, in _object_dispatch
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] return getattr(target, method)(*args,
**kwargs)
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1]
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
222, in wrapper
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] return fn(self, *args, **kwargs)
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1]
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] File
"/opt/stack/new/nova/nova/objects/block_device.py", line 183, in save
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] raise
exception.BDMNotFound(id=self.id)
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1]
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1] BDMNotFound: No Block Device Mapping with
id 48.
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1]
2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance:
75986322-d59d-411c-a60c-776415cdd6d1]
This started spiking on 11/27. I suspect
https://review.openstack.org/#/c/194063/ introduced the race, which
was a 2nd attempt a change which was reverted previously because it
introduced a race.
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message:%5C%22BDMNotFound:%20No%20Block%20Device%20Mapping%20with%20id%5C%22%20AND%20message:%5C%22_default_block_device_names%5C%22%20AND%20tags:%5C%22screen-n-cpu.txt%5C%22%20AND%20NOT%20build_queue:%5C%22experimental%5C%22
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1521340/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
