Public bug reported: Doing a nova resize on an instance when using the vsphere driver will cause the instance to go in to error state.
The problem is that the scheduler will pick another host to spin up a new resized instance and when the user confirms nova will fail because its looking for the instance on the old compute. Here is the traceback. 2014-07-16 18:14:55.271 13228 DEBUG amqp [-] Closed channel #1 _do_close /usr/lib/python2.7/dist-packages/amqp/channel.py:88 2014-07-16 18:14:55.271 13228 DEBUG amqp [-] using channel_id: 1 __init__ /usr/lib/python2.7/dist-packages/amqp/channel.py:70 2014-07-16 18:14:55.273 13228 DEBUG amqp [-] Channel open _open_ok /usr/lib/python2.7/dist-packages/amqp/channel.py:420 2014-07-16 18:14:55.299 13228 ERROR nova.openstack.common.rpc.amqp [req-3cae0e1d-2cf4-4da7-9aac-0ea5279b829d cherkasj 37af63b6867d4fe38ac312ca626ce186] Exception during message handling 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last): 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp **args) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 353, in decorated_function 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 90, in wrapped 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp payload) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 73, in wrapped 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 294, in decorated_function 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 271, in decorated_function 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info()) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 258, in decorated_function 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2683, in confirm_resize 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp do_confirm_resize(context, instance, migration_id) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 246, in inner 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp return f(*args, **kwargs) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2680, in do_confirm_resize 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp migration=migration) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2707, in _confirm_resize 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp network_info) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 465, in confirm_migration 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp _vmops = self._get_vmops_for_compute_node(instance['node']) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 567, in _get_vmops_for_compute_node 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp resource = self._get_resource_for_node(nodename) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 559, in _get_resource_for_node 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp raise exception.NotFound(msg) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp NotFound: The resource domain-c1122(compute02) does not exist 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp 2014-07-16 18:14:57.595 13228 DEBUG nova.openstack.common.vmware.api [-] Waiting for function _invoke_api to return. func /usr/lib/python2.7/dist-packages/nova/openstack/common/vmware/api.py:120 2014-07-16 18:14:57.598 13228 DEBUG nova.openstack.common.vmware.api [-] Invoking _invoke_api; retry count is 0. _func /usr/lib/python2.7/dist-packages/nova/openstack/common/vmware/api.py:83 2014-07-16 18:14:57.598 13228 DEBUG nova.openstack.common.vmware.api [-] Invoking method <module 'nova.virt.vmwareapi.vim_util' from '/usr/lib/python2.7/dist-packages/nova/virt/vmwareap ** Affects: nova Importance: Critical Assignee: Gary Kotton (garyk) Status: New ** Changed in: nova Importance: Undecided => Critical ** Changed in: nova Assignee: (unassigned) => Gary Kotton (garyk) ** Changed in: nova Milestone: None => juno-2 -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1345460 Title: VMware: resize fails when there is more than one compute node Status in OpenStack Compute (Nova): New Bug description: Doing a nova resize on an instance when using the vsphere driver will cause the instance to go in to error state. The problem is that the scheduler will pick another host to spin up a new resized instance and when the user confirms nova will fail because its looking for the instance on the old compute. Here is the traceback. 2014-07-16 18:14:55.271 13228 DEBUG amqp [-] Closed channel #1 _do_close /usr/lib/python2.7/dist-packages/amqp/channel.py:88 2014-07-16 18:14:55.271 13228 DEBUG amqp [-] using channel_id: 1 __init__ /usr/lib/python2.7/dist-packages/amqp/channel.py:70 2014-07-16 18:14:55.273 13228 DEBUG amqp [-] Channel open _open_ok /usr/lib/python2.7/dist-packages/amqp/channel.py:420 2014-07-16 18:14:55.299 13228 ERROR nova.openstack.common.rpc.amqp [req-3cae0e1d-2cf4-4da7-9aac-0ea5279b829d cherkasj 37af63b6867d4fe38ac312ca626ce186] Exception during message handling 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last): 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp **args) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 353, in decorated_function 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 90, in wrapped 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp payload) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 73, in wrapped 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 294, in decorated_function 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 271, in decorated_function 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info()) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 258, in decorated_function 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2683, in confirm_resize 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp do_confirm_resize(context, instance, migration_id) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 246, in inner 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp return f(*args, **kwargs) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2680, in do_confirm_resize 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp migration=migration) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2707, in _confirm_resize 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp network_info) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 465, in confirm_migration 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp _vmops = self._get_vmops_for_compute_node(instance['node']) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 567, in _get_vmops_for_compute_node 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp resource = self._get_resource_for_node(nodename) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 559, in _get_resource_for_node 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp raise exception.NotFound(msg) 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp NotFound: The resource domain-c1122(compute02) does not exist 2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp 2014-07-16 18:14:57.595 13228 DEBUG nova.openstack.common.vmware.api [-] Waiting for function _invoke_api to return. func /usr/lib/python2.7/dist-packages/nova/openstack/common/vmware/api.py:120 2014-07-16 18:14:57.598 13228 DEBUG nova.openstack.common.vmware.api [-] Invoking _invoke_api; retry count is 0. _func /usr/lib/python2.7/dist-packages/nova/openstack/common/vmware/api.py:83 2014-07-16 18:14:57.598 13228 DEBUG nova.openstack.common.vmware.api [-] Invoking method <module 'nova.virt.vmwareapi.vim_util' from '/usr/lib/python2.7/dist-packages/nova/virt/vmwareap To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1345460/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp