Public bug reported: This is using the latest nova from trunk. In our deployment, we had a hypervisor go down and the tenant issued a hard reboot prior. When attempting a reboot on a guest with the state HARD_REBOOT, nova controller throws this error in it's logs and returns 'ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500)' to the user:
2014-03-06 18:21:00,535 (routes.middleware): DEBUG middleware __call__ Matched POST /tenant1/servers/778032b2-469d-445e-abde-7b9b0b673324/action 2014-03-06 18:21:00,536 (routes.middleware): DEBUG middleware __call__ Route path: '/{project_id}/servers/:(id)/action', defaults: {'action': u'action', 'controller': <nova.api.openstack.wsgi.Resource object at 0x5242c90>} 2014-03-06 18:21:00,536 (routes.middleware): DEBUG middleware __call__ Match dict: {'action': u'action', 'controller': <nova.api.openstack.wsgi.Resource object at 0x5242c90>, 'project_id': u'tenant1', 'id': u'778032b2-469d-445e-abde-7b9b0b673324'} 2014-03-06 18:21:00,537 (nova.api.openstack.wsgi): DEBUG wsgi _process_stack Action: 'action', body: {"reboot": {"type": "SOFT"}} 2014-03-06 18:21:00,538 (nova.api.openstack.wsgi): DEBUG wsgi _process_stack Calling method <bound method Controller._action_reboot of <nova.api.openstack.compute.contrib.keypairs.Controller object at 0x4c35a50>> 2014-03-06 18:21:00,747 (nova.api.openstack): ERROR __init__ _error Caught error: Unexpected task state: expecting [None, 'rebooting'] but the actual state is rebooting_hard Traceback (most recent call last): File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/api/openstack/__init__.py", line 125, in __call__ return req.get_response(self.application) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/request.py", line 1320, in send application, catch_exc_info=False) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/request.py", line 1284, in call_application app_iter = application(self.environ, start_response) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__ return resp(environ, start_response) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 598, in __call__ return self.app(env, start_response) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__ return resp(environ, start_response) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__ return resp(environ, start_response) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/routes/middleware.py", line 131, in __call__ response = self.app(environ, start_response) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__ return resp(environ, start_response) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 130, in __call__ resp = self.call_func(req, *args, **self.kwargs) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 195, in call_func return self.func(req, *args, **kwargs) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 925, in __call__ content_type, body, accept) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 987, in _process_stack action_result = self.dispatch(meth, request, action_args) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 1074, in dispatch return method(req=request, **action_args) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/api/openstack/compute/servers.py", line 1145, in _action_reboot self.compute_api.reboot(context, instance, reboot_type) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/compute/api.py", line 199, in wrapped return func(self, context, target, *args, **kwargs) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/compute/api.py", line 189, in inner return function(self, context, instance, *args, **kwargs) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/compute/api.py", line 170, in inner return f(self, context, instance, *args, **kw) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/compute/api.py", line 2073, in reboot instance.save(expected_task_state=[None, task_states.REBOOTING]) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/objects/base.py", line 151, in wrapper return fn(self, ctxt, *args, **kwargs) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/objects/instance.py", line 472, in save columns_to_join=_expected_cols(expected_attrs)) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/db/api.py", line 739, in instance_update_and_get_original columns_to_join=columns_to_join) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 128, in wrapper return f(*args, **kwargs) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 2164, in instance_update_and_get_original columns_to_join=columns_to_join) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 2215, in _instance_update actual=actual_state, expected=expected) UnexpectedTaskStateError: Unexpected task state: expecting [None, 'rebooting'] but the actual state is rebooting_hard 2014-03-06 18:21:00,750 (nova.api.openstack): INFO __init__ _error http://nova-controller.isg.apple.com:8774/v2/tenant1/servers/778032b2-469d-445e-abde-7b9b0b673324/action returned with HTTP 500 The actual error message back to the user must be something along the lines of 'Unexpected task state: expecting [None, 'rebooting'] but the actual state is rebooting_hard' with a 4xx HTTP code. ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1288926 Title: incorrect error code when rebooting a rebooting_hard guest Status in OpenStack Compute (Nova): New Bug description: This is using the latest nova from trunk. In our deployment, we had a hypervisor go down and the tenant issued a hard reboot prior. When attempting a reboot on a guest with the state HARD_REBOOT, nova controller throws this error in it's logs and returns 'ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500)' to the user: 2014-03-06 18:21:00,535 (routes.middleware): DEBUG middleware __call__ Matched POST /tenant1/servers/778032b2-469d-445e-abde-7b9b0b673324/action 2014-03-06 18:21:00,536 (routes.middleware): DEBUG middleware __call__ Route path: '/{project_id}/servers/:(id)/action', defaults: {'action': u'action', 'controller': <nova.api.openstack.wsgi.Resource object at 0x5242c90>} 2014-03-06 18:21:00,536 (routes.middleware): DEBUG middleware __call__ Match dict: {'action': u'action', 'controller': <nova.api.openstack.wsgi.Resource object at 0x5242c90>, 'project_id': u'tenant1', 'id': u'778032b2-469d-445e-abde-7b9b0b673324'} 2014-03-06 18:21:00,537 (nova.api.openstack.wsgi): DEBUG wsgi _process_stack Action: 'action', body: {"reboot": {"type": "SOFT"}} 2014-03-06 18:21:00,538 (nova.api.openstack.wsgi): DEBUG wsgi _process_stack Calling method <bound method Controller._action_reboot of <nova.api.openstack.compute.contrib.keypairs.Controller object at 0x4c35a50>> 2014-03-06 18:21:00,747 (nova.api.openstack): ERROR __init__ _error Caught error: Unexpected task state: expecting [None, 'rebooting'] but the actual state is rebooting_hard Traceback (most recent call last): File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/api/openstack/__init__.py", line 125, in __call__ return req.get_response(self.application) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/request.py", line 1320, in send application, catch_exc_info=False) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/request.py", line 1284, in call_application app_iter = application(self.environ, start_response) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__ return resp(environ, start_response) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 598, in __call__ return self.app(env, start_response) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__ return resp(environ, start_response) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__ return resp(environ, start_response) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/routes/middleware.py", line 131, in __call__ response = self.app(environ, start_response) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__ return resp(environ, start_response) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 130, in __call__ resp = self.call_func(req, *args, **self.kwargs) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 195, in call_func return self.func(req, *args, **kwargs) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 925, in __call__ content_type, body, accept) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 987, in _process_stack action_result = self.dispatch(meth, request, action_args) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 1074, in dispatch return method(req=request, **action_args) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/api/openstack/compute/servers.py", line 1145, in _action_reboot self.compute_api.reboot(context, instance, reboot_type) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/compute/api.py", line 199, in wrapped return func(self, context, target, *args, **kwargs) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/compute/api.py", line 189, in inner return function(self, context, instance, *args, **kwargs) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/compute/api.py", line 170, in inner return f(self, context, instance, *args, **kw) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/compute/api.py", line 2073, in reboot instance.save(expected_task_state=[None, task_states.REBOOTING]) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/objects/base.py", line 151, in wrapper return fn(self, ctxt, *args, **kwargs) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/objects/instance.py", line 472, in save columns_to_join=_expected_cols(expected_attrs)) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/db/api.py", line 739, in instance_update_and_get_original columns_to_join=columns_to_join) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 128, in wrapper return f(*args, **kwargs) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 2164, in instance_update_and_get_original columns_to_join=columns_to_join) File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 2215, in _instance_update actual=actual_state, expected=expected) UnexpectedTaskStateError: Unexpected task state: expecting [None, 'rebooting'] but the actual state is rebooting_hard 2014-03-06 18:21:00,750 (nova.api.openstack): INFO __init__ _error http://nova-controller.isg.apple.com:8774/v2/tenant1/servers/778032b2-469d-445e-abde-7b9b0b673324/action returned with HTTP 500 The actual error message back to the user must be something along the lines of 'Unexpected task state: expecting [None, 'rebooting'] but the actual state is rebooting_hard' with a 4xx HTTP code. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1288926/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp