Public bug reported:
On the Pike setup, nova boot for ironic fails consistently due to
following error:
2018-04-13 07:56:02.846 DEBUG oslo_concurrency.processutils
[req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] CMD "genisoimage -o
/tmp/tmppY5Ikz -ldots -allow-low
ercase -allow-multidot -l -publisher OpenStack Nova 16.1.1 -quiet -J -r -V
config-2 /tmp/tmpxFZUMX" returned: 0 in 0.019s from (pid=47123) execute
/usr/local/lib/pytho
n2.7/dist-packages/oslo_concurrency/processutils.py:404
2018-04-13 07:56:02.863 INFO nova.virt.ironic.driver
[req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] Config drive for
instance 3b1195b4-3106-4c5c-87f4-69d203d08
884 on baremetal node 4fdcff13-0a9c-489c-8f71-99cfddfadf96 created.
2018-04-13 07:56:03.656 ERROR oslo.service.loopingcall
[req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] Fixed interval looping
call 'nova.virt.ironic.driver.Iron
icDriver._wait_for_active' failed: OrphanedObjectError_Remote: Cannot call
obj_load_attr on orphaned Instance object
Traceback (most recent call last):
File "/opt/stack/nova/nova/conductor/manager.py", line 124, in
_object_dispatch
return getattr(target, method)(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py",
line 226, in wrapper
return fn(self, *args, **kwargs)
File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh
elif self[field] != current[field]:
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py",
line 762, in __getitem__
return getattr(self, name)
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py",
line 67, in getter
self.obj_load_attr(name)
File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr
objtype=self.obj_name())
OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall Traceback (most recent
call last):
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 137,
in _run_loop
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall result =
func(*self.args, **self.kw)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/opt/stack/nova/nova/virt/ironic/driver.py", line 473, in _wait_for_active
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall instance.refresh()
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
210, in wrapper
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall ctxt, self,
fn.__name__, args, kwargs)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/opt/stack/nova/nova/conductor/rpcapi.py", line 245, in object_action
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall objmethod=objmethod,
args=args, kwargs=kwargs)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line
169, in call
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall retry=self.retry)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123,
in _send
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall timeout=timeout,
retry=retry)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
line 566, in send
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall retry=retry)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
line 557, in _send
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall raise result
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance
object
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall Traceback (most recent
call last):
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall return
getattr(target, method)(*args, **kwargs)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
226, in wrapper
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall return fn(self,
*args, **kwargs)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/opt/stack/nova/nova/objects/instance.py", line 830, in refresh
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall elif self[field] !=
current[field]:
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
762, in __getitem__
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall return getattr(self,
name)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
67, in getter
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
self.obj_load_attr(name)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
objtype=self.obj_name())
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall OrphanedObjectError:
Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.660 ERROR nova.virt.ironic.driver
[req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] Error deploying instance
3b1195b4-3106-4c5c-87f4-69d203d08884 on baremetal node
4fdcff13-0a9c-489c-8f71-99cfddfadf96.: OrphanedObjectError_Remote: Cannot call
obj_load_attr on orphaned Instance object
Traceback (most recent call last):
File "/opt/stack/nova/nova/conductor/manager.py", line 124, in
_object_dispatch
return getattr(target, method)(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py",
line 226, in wrapper
return fn(self, *args, **kwargs)
File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh
elif self[field] != current[field]:
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py",
line 762, in __getitem__
return getattr(self, name)
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py",
line 67, in getter
self.obj_load_attr(name)
File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr
objtype=self.obj_name())
OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.661 ERROR nova.compute.manager
[req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] Instance failed to spawn:
OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance
object
Traceback (most recent call last):
File "/opt/stack/nova/nova/conductor/manager.py", line 124, in
_object_dispatch
return getattr(target, method)(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py",
line 226, in wrapper
return fn(self, *args, **kwargs)
File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh
elif self[field] != current[field]:
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py",
line 762, in __getitem__
return getattr(self, name)
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py",
line 67, in getter
self.obj_load_attr(name)
File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr
objtype=self.obj_name())
OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] Traceback (most recent call last):
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/opt/stack/nova/nova/compute/manager.py", line 2200, in _build_resources
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] yield resources
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/opt/stack/nova/nova/compute/manager.py", line 2015, in _build_and_run_instance
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] block_device_info=block_device_info)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/opt/stack/nova/nova/virt/ironic/driver.py", line 1108, in spawn
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] 'node': node_uuid})
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] self.force_reraise()
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] six.reraise(self.type_, self.value,
self.tb)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/opt/stack/nova/nova/virt/ironic/driver.py", line 1100, in spawn
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884]
timer.start(interval=CONF.ironic.api_retry_interval).wait()
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] return hubs.get_hub().switch()
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in
switch
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] return self.greenlet.switch()
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 137,
in _run_loop
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] result = func(*self.args, **self.kw)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/opt/stack/nova/nova/virt/ironic/driver.py", line 473, in _wait_for_active
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] instance.refresh()
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
210, in wrapper
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] ctxt, self, fn.__name__, args, kwargs)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/opt/stack/nova/nova/conductor/rpcapi.py", line 245, in object_action
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] objmethod=objmethod, args=args,
kwargs=kwargs)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line
169, in call
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] retry=self.retry)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123,
in _send
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] timeout=timeout, retry=retry)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
line 566, in send
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] retry=retry)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
line 557, in _send
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] raise result
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] OrphanedObjectError_Remote: Cannot call
obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] Traceback (most recent call last):
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] return getattr(target, method)(*args,
**kwargs)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
226, in wrapper
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] return fn(self, *args, **kwargs)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/opt/stack/nova/nova/objects/instance.py", line 830, in refresh
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] elif self[field] != current[field]:
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
762, in __getitem__
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] return getattr(self, name)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
67, in getter
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] self.obj_load_attr(name)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] objtype=self.obj_name())
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] OrphanedObjectError: Cannot call
obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.665 INFO nova.compute.manager
[req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] Terminating instance
The issue is due to the assignment at line
https://github.com/openstack/nova/blob/stable/pike/nova/objects/instance.py#L824
After commenting out this line, the deploy works end to end.
This has been reproduced consistently on two Pike setups.
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1769025
Title:
OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance
object error is thrown for ironic deploy in Pike
Status in OpenStack Compute (nova):
New
Bug description:
On the Pike setup, nova boot for ironic fails consistently due to
following error:
2018-04-13 07:56:02.846 DEBUG oslo_concurrency.processutils
[req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] CMD "genisoimage -o
/tmp/tmppY5Ikz -ldots -allow-low
ercase -allow-multidot -l -publisher OpenStack Nova 16.1.1 -quiet -J -r -V
config-2 /tmp/tmpxFZUMX" returned: 0 in 0.019s from (pid=47123) execute
/usr/local/lib/pytho
n2.7/dist-packages/oslo_concurrency/processutils.py:404
2018-04-13 07:56:02.863 INFO nova.virt.ironic.driver
[req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] Config drive for
instance 3b1195b4-3106-4c5c-87f4-69d203d08
884 on baremetal node 4fdcff13-0a9c-489c-8f71-99cfddfadf96 created.
2018-04-13 07:56:03.656 ERROR oslo.service.loopingcall
[req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] Fixed interval looping
call 'nova.virt.ironic.driver.Iron
icDriver._wait_for_active' failed: OrphanedObjectError_Remote: Cannot call
obj_load_attr on orphaned Instance object
Traceback (most recent call last):
File "/opt/stack/nova/nova/conductor/manager.py", line 124, in
_object_dispatch
return getattr(target, method)(*args, **kwargs)
File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
226, in wrapper
return fn(self, *args, **kwargs)
File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh
elif self[field] != current[field]:
File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
762, in __getitem__
return getattr(self, name)
File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
67, in getter
self.obj_load_attr(name)
File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr
objtype=self.obj_name())
OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall Traceback (most recent
call last):
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 137,
in _run_loop
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall result =
func(*self.args, **self.kw)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/opt/stack/nova/nova/virt/ironic/driver.py", line 473, in _wait_for_active
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall instance.refresh()
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
210, in wrapper
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall ctxt, self,
fn.__name__, args, kwargs)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/opt/stack/nova/nova/conductor/rpcapi.py", line 245, in object_action
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
objmethod=objmethod, args=args, kwargs=kwargs)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line
169, in call
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall retry=self.retry)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123,
in _send
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall timeout=timeout,
retry=retry)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
line 566, in send
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall retry=retry)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
line 557, in _send
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall raise result
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance
object
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall Traceback (most recent
call last):
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall return
getattr(target, method)(*args, **kwargs)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
226, in wrapper
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall return fn(self,
*args, **kwargs)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/opt/stack/nova/nova/objects/instance.py", line 830, in refresh
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall elif self[field]
!= current[field]:
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
762, in __getitem__
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall return
getattr(self, name)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
67, in getter
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
self.obj_load_attr(name)
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File
"/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
objtype=self.obj_name())
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall OrphanedObjectError:
Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall
2018-04-13 07:56:03.660 ERROR nova.virt.ironic.driver
[req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] Error deploying instance
3b1195b4-3106-4c5c-87f4-69d203d08884 on baremetal node
4fdcff13-0a9c-489c-8f71-99cfddfadf96.: OrphanedObjectError_Remote: Cannot call
obj_load_attr on orphaned Instance object
Traceback (most recent call last):
File "/opt/stack/nova/nova/conductor/manager.py", line 124, in
_object_dispatch
return getattr(target, method)(*args, **kwargs)
File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
226, in wrapper
return fn(self, *args, **kwargs)
File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh
elif self[field] != current[field]:
File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
762, in __getitem__
return getattr(self, name)
File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
67, in getter
self.obj_load_attr(name)
File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr
objtype=self.obj_name())
OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.661 ERROR nova.compute.manager
[req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] Instance failed to spawn:
OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance
object
Traceback (most recent call last):
File "/opt/stack/nova/nova/conductor/manager.py", line 124, in
_object_dispatch
return getattr(target, method)(*args, **kwargs)
File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
226, in wrapper
return fn(self, *args, **kwargs)
File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh
elif self[field] != current[field]:
File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
762, in __getitem__
return getattr(self, name)
File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
67, in getter
self.obj_load_attr(name)
File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr
objtype=self.obj_name())
OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] Traceback (most recent call last):
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/opt/stack/nova/nova/compute/manager.py", line 2200, in _build_resources
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] yield resources
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/opt/stack/nova/nova/compute/manager.py", line 2015, in _build_and_run_instance
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] block_device_info=block_device_info)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/opt/stack/nova/nova/virt/ironic/driver.py", line 1108, in spawn
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] 'node': node_uuid})
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] self.force_reraise()
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] six.reraise(self.type_, self.value,
self.tb)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/opt/stack/nova/nova/virt/ironic/driver.py", line 1100, in spawn
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884]
timer.start(interval=CONF.ironic.api_retry_interval).wait()
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] return hubs.get_hub().switch()
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in
switch
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] return self.greenlet.switch()
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 137,
in _run_loop
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] result = func(*self.args, **self.kw)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/opt/stack/nova/nova/virt/ironic/driver.py", line 473, in _wait_for_active
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] instance.refresh()
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
210, in wrapper
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] ctxt, self, fn.__name__, args, kwargs)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/opt/stack/nova/nova/conductor/rpcapi.py", line 245, in object_action
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] objmethod=objmethod, args=args,
kwargs=kwargs)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line
169, in call
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] retry=self.retry)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123,
in _send
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] timeout=timeout, retry=retry)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
line 566, in send
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] retry=retry)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
line 557, in _send
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] raise result
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] OrphanedObjectError_Remote: Cannot call
obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] Traceback (most recent call last):
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] return getattr(target, method)(*args,
**kwargs)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
226, in wrapper
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] return fn(self, *args, **kwargs)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/opt/stack/nova/nova/objects/instance.py", line 830, in refresh
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] elif self[field] != current[field]:
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
762, in __getitem__
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] return getattr(self, name)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
67, in getter
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] self.obj_load_attr(name)
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] File
"/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] objtype=self.obj_name())
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] OrphanedObjectError: Cannot call
obj_load_attr on orphaned Instance object
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884]
2018-04-13 07:56:03.665 INFO nova.compute.manager
[req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] [instance:
3b1195b4-3106-4c5c-87f4-69d203d08884] Terminating instance
The issue is due to the assignment at line
https://github.com/openstack/nova/blob/stable/pike/nova/objects/instance.py#L824
After commenting out this line, the deploy works end to end.
This has been reproduced consistently on two Pike setups.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1769025/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
