Public bug reported: On the Pike setup, nova boot for ironic fails consistently due to following error:
2018-04-13 07:56:02.846 DEBUG oslo_concurrency.processutils [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] CMD "genisoimage -o /tmp/tmppY5Ikz -ldots -allow-low ercase -allow-multidot -l -publisher OpenStack Nova 16.1.1 -quiet -J -r -V config-2 /tmp/tmpxFZUMX" returned: 0 in 0.019s from (pid=47123) execute /usr/local/lib/pytho n2.7/dist-packages/oslo_concurrency/processutils.py:404 2018-04-13 07:56:02.863 INFO nova.virt.ironic.driver [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] Config drive for instance 3b1195b4-3106-4c5c-87f4-69d203d08 884 on baremetal node 4fdcff13-0a9c-489c-8f71-99cfddfadf96 created. 2018-04-13 07:56:03.656 ERROR oslo.service.loopingcall [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] Fixed interval looping call 'nova.virt.ironic.driver.Iron icDriver._wait_for_active' failed: OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object Traceback (most recent call last): File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch return getattr(target, method)(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper return fn(self, *args, **kwargs) File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh elif self[field] != current[field]: File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__ return getattr(self, name) File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter self.obj_load_attr(name) File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr objtype=self.obj_name()) OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall Traceback (most recent call last): 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 137, in _run_loop 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall result = func(*self.args, **self.kw) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/virt/ironic/driver.py", line 473, in _wait_for_active 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall instance.refresh() 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 210, in wrapper 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall ctxt, self, fn.__name__, args, kwargs) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/conductor/rpcapi.py", line 245, in object_action 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall objmethod=objmethod, args=args, kwargs=kwargs) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall retry=self.retry) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall timeout=timeout, retry=retry) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall retry=retry) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 557, in _send 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall raise result 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall Traceback (most recent call last): 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall return getattr(target, method)(*args, **kwargs) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall return fn(self, *args, **kwargs) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall elif self[field] != current[field]: 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__ 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall return getattr(self, name) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall self.obj_load_attr(name) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall objtype=self.obj_name()) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall 2018-04-13 07:56:03.660 ERROR nova.virt.ironic.driver [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] Error deploying instance 3b1195b4-3106-4c5c-87f4-69d203d08884 on baremetal node 4fdcff13-0a9c-489c-8f71-99cfddfadf96.: OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object Traceback (most recent call last): File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch return getattr(target, method)(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper return fn(self, *args, **kwargs) File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh elif self[field] != current[field]: File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__ return getattr(self, name) File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter self.obj_load_attr(name) File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr objtype=self.obj_name()) OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object 2018-04-13 07:56:03.661 ERROR nova.compute.manager [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] Instance failed to spawn: OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object Traceback (most recent call last): File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch return getattr(target, method)(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper return fn(self, *args, **kwargs) File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh elif self[field] != current[field]: File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__ return getattr(self, name) File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter self.obj_load_attr(name) File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr objtype=self.obj_name()) OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] Traceback (most recent call last): 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/compute/manager.py", line 2200, in _build_resources 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] yield resources 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/compute/manager.py", line 2015, in _build_and_run_instance 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] block_device_info=block_device_info) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/virt/ironic/driver.py", line 1108, in spawn 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] 'node': node_uuid}) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] self.force_reraise() 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] six.reraise(self.type_, self.value, self.tb) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/virt/ironic/driver.py", line 1100, in spawn 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] timer.start(interval=CONF.ironic.api_retry_interval).wait() 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] return hubs.get_hub().switch() 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] return self.greenlet.switch() 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 137, in _run_loop 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] result = func(*self.args, **self.kw) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/virt/ironic/driver.py", line 473, in _wait_for_active 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] instance.refresh() 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 210, in wrapper 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] ctxt, self, fn.__name__, args, kwargs) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/conductor/rpcapi.py", line 245, in object_action 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] objmethod=objmethod, args=args, kwargs=kwargs) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] retry=self.retry) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] timeout=timeout, retry=retry) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] retry=retry) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 557, in _send 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] raise result 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] Traceback (most recent call last): 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] return getattr(target, method)(*args, **kwargs) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] return fn(self, *args, **kwargs) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] elif self[field] != current[field]: 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__ 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] return getattr(self, name) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] self.obj_load_attr(name) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] objtype=self.obj_name()) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] 2018-04-13 07:56:03.665 INFO nova.compute.manager [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] Terminating instance The issue is due to the assignment at line https://github.com/openstack/nova/blob/stable/pike/nova/objects/instance.py#L824 After commenting out this line, the deploy works end to end. This has been reproduced consistently on two Pike setups. ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1769025 Title: OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object error is thrown for ironic deploy in Pike Status in OpenStack Compute (nova): New Bug description: On the Pike setup, nova boot for ironic fails consistently due to following error: 2018-04-13 07:56:02.846 DEBUG oslo_concurrency.processutils [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] CMD "genisoimage -o /tmp/tmppY5Ikz -ldots -allow-low ercase -allow-multidot -l -publisher OpenStack Nova 16.1.1 -quiet -J -r -V config-2 /tmp/tmpxFZUMX" returned: 0 in 0.019s from (pid=47123) execute /usr/local/lib/pytho n2.7/dist-packages/oslo_concurrency/processutils.py:404 2018-04-13 07:56:02.863 INFO nova.virt.ironic.driver [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] Config drive for instance 3b1195b4-3106-4c5c-87f4-69d203d08 884 on baremetal node 4fdcff13-0a9c-489c-8f71-99cfddfadf96 created. 2018-04-13 07:56:03.656 ERROR oslo.service.loopingcall [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] Fixed interval looping call 'nova.virt.ironic.driver.Iron icDriver._wait_for_active' failed: OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object Traceback (most recent call last): File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch return getattr(target, method)(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper return fn(self, *args, **kwargs) File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh elif self[field] != current[field]: File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__ return getattr(self, name) File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter self.obj_load_attr(name) File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr objtype=self.obj_name()) OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall Traceback (most recent call last): 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 137, in _run_loop 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall result = func(*self.args, **self.kw) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/virt/ironic/driver.py", line 473, in _wait_for_active 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall instance.refresh() 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 210, in wrapper 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall ctxt, self, fn.__name__, args, kwargs) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/conductor/rpcapi.py", line 245, in object_action 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall objmethod=objmethod, args=args, kwargs=kwargs) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall retry=self.retry) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall timeout=timeout, retry=retry) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall retry=retry) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 557, in _send 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall raise result 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall Traceback (most recent call last): 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall return getattr(target, method)(*args, **kwargs) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall return fn(self, *args, **kwargs) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall elif self[field] != current[field]: 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__ 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall return getattr(self, name) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall self.obj_load_attr(name) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall objtype=self.obj_name()) 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall 2018-04-13 07:56:03.656 TRACE oslo.service.loopingcall 2018-04-13 07:56:03.660 ERROR nova.virt.ironic.driver [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] Error deploying instance 3b1195b4-3106-4c5c-87f4-69d203d08884 on baremetal node 4fdcff13-0a9c-489c-8f71-99cfddfadf96.: OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object Traceback (most recent call last): File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch return getattr(target, method)(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper return fn(self, *args, **kwargs) File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh elif self[field] != current[field]: File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__ return getattr(self, name) File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter self.obj_load_attr(name) File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr objtype=self.obj_name()) OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object 2018-04-13 07:56:03.661 ERROR nova.compute.manager [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] Instance failed to spawn: OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object Traceback (most recent call last): File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch return getattr(target, method)(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper return fn(self, *args, **kwargs) File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh elif self[field] != current[field]: File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__ return getattr(self, name) File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter self.obj_load_attr(name) File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr objtype=self.obj_name()) OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] Traceback (most recent call last): 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/compute/manager.py", line 2200, in _build_resources 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] yield resources 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/compute/manager.py", line 2015, in _build_and_run_instance 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] block_device_info=block_device_info) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/virt/ironic/driver.py", line 1108, in spawn 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] 'node': node_uuid}) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] self.force_reraise() 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] six.reraise(self.type_, self.value, self.tb) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/virt/ironic/driver.py", line 1100, in spawn 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] timer.start(interval=CONF.ironic.api_retry_interval).wait() 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] return hubs.get_hub().switch() 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] return self.greenlet.switch() 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 137, in _run_loop 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] result = func(*self.args, **self.kw) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/virt/ironic/driver.py", line 473, in _wait_for_active 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] instance.refresh() 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 210, in wrapper 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] ctxt, self, fn.__name__, args, kwargs) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/conductor/rpcapi.py", line 245, in object_action 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] objmethod=objmethod, args=args, kwargs=kwargs) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] retry=self.retry) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] timeout=timeout, retry=retry) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] retry=retry) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 557, in _send 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] raise result 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] OrphanedObjectError_Remote: Cannot call obj_load_attr on orphaned Instance object 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] Traceback (most recent call last): 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/conductor/manager.py", line 124, in _object_dispatch 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] return getattr(target, method)(*args, **kwargs) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] return fn(self, *args, **kwargs) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/objects/instance.py", line 830, in refresh 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] elif self[field] != current[field]: 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 762, in __getitem__ 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] return getattr(self, name) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] self.obj_load_attr(name) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] File "/opt/stack/nova/nova/objects/instance.py", line 1044, in obj_load_attr 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] objtype=self.obj_name()) 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] 2018-04-13 07:56:03.661 TRACE nova.compute.manager [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] 2018-04-13 07:56:03.665 INFO nova.compute.manager [req-8c6d1e73-65a6-420e-a041-4dbd0ee768e1 admin admin] [instance: 3b1195b4-3106-4c5c-87f4-69d203d08884] Terminating instance The issue is due to the assignment at line https://github.com/openstack/nova/blob/stable/pike/nova/objects/instance.py#L824 After commenting out this line, the deploy works end to end. This has been reproduced consistently on two Pike setups. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1769025/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp