Looks like one or more of your VMs get stuck and libvirt having problems to stop it now:
2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher if ret == -1: raise libvirtError ('virDomainDestroy() failed', dom=self) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher libvirtError: Failed to terminate process 8756 with SIGKILL: Device or resource busy I would recommend you to check what exactly is PID (8756) in the error? I assume is a qemu process and what's going on with this process? Does it have some issue with accessing storage or something else? You can also check the libvirtd logs, there should be something more. ==================================================== Radosław Śmigielski ALCATEL-LUCENT - Cloudband/Node Team Office: 238 646 13 or +353 (0)188 646 13 Mobile: +353 86 820 74 23 ________________________________________ From: cosmos cosmos [cosmos0...@gmail.com] Sent: 22 May 2014 12:30 To: openstack-dev@lists.openstack.org Subject: [openstack-dev] [sahara] Deleting cluster in sahara horizon, 500 Error occured in nova. Hello My name is inhye park from samsung SDS. I am a developer about nova and sahara. Recently Our system updated to icehouse. After deleting cluster in sahara, VM occurred 500 error. But this problem not occur in nova cli command. Only in sahara this error occurred. This is error log. ----------------------------------------------------------------------------------------- 2014-05-22 20:06:42.631 25640 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Failed to terminate process 8756 with SIGKILL: Device or resource busy 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last): 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher incoming.message)) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher payload) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__ 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 333, in decorated_function 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher function(self, context, *args, **kwargs) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 309, in decorated_function 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher e, sys.exc_info()) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__ 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 296, in decorated_function 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2262, in terminate_instance 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher do_terminate_instance(instance, bdms) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 249, in inner 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2260, in do_terminate_instance 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher self._set_instance_error_state(context, instance['uuid']) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__ 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2250, in do_terminate_instance 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher reservations=reservations) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 103, in inner 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher rv = f(*args, **kwargs) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2220, in _delete_instance 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher user_id=user_id) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__ 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2190, in _delete_instance 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher self._shutdown_instance(context, db_inst, bdms) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2125, in _shutdown_instance 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher requested_networks) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__ 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2115, in _shutdown_instance 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher block_device_info) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 951, in destroy 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher self._destroy(instance) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 908, in _destroy 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher instance=instance) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__ 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 880, in _destroy 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher virt_dom.destroy() 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher result = proxy_call(self._autowrap, f, *args, **kwargs) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher rv = execute(f,*args,**kwargs) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher rv = meth(*args,**kwargs) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/libvirt.py", line 918, in destroy 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher if ret == -1: raise libvirtError ('virDomainDestroy() failed', dom=self) 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher libvirtError: Failed to terminate process 8756 with SIGKILL: Device or resource busy 2014-05-22 20:06:42.631 25640 TRACE oslo.messaging.rpc.dispatcher 2014-05-22 20:06:42.633 25640 ERROR oslo.messaging._drivers.common [-] Returning exception Failed to terminate process 8756 with SIGKILL: Device or resource busy to caller Could you please help about this case? Thanks. _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev