Public bug reported: If an instance fails during its network creation (for example if the network-vif-plugged event doesn't arrive in time) a subsequent delete will also fail when it tries to delete the vif, leaving the instance in a Error(deleting) state.
This can be avoided by including the "--if-exists" option to the ovs=vsctl command. Example of stack trace: 2014-04-16 12:28:51.949 AUDIT nova.compute.manager [req-af72c100-5d9b-44f6-b941-3d72529b3401 demo demo] [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] Terminating instance 2014-04-16 12:28:52.309 ERROR nova.virt.libvirt.driver [-] [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] During wait destroy, instance disappeared. 2014-04-16 12:28:52.407 ERROR nova.network.linux_net [req-af72c100-5d9b-44f6-b941-3d72529b3401 demo demo] Unable to execute ['ovs-vsctl', '--timeout=120', 'del-port', 'br-int', u'qvo67a96e96-10']. Exception: Unexpected error while running command. Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ovs-vsctl --timeout=120 del-port br-int qvo67a96e96-10 Exit code: 1 Stdout: '' Stderr: 'ovs-vsctl: no port named qvo67a96e96-10\n' 2014-04-16 12:28:52.573 ERROR nova.compute.manager [req-af72c100-5d9b-44f6-b941-3d72529b3401 demo demo] [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] Setting instance vm_state to ERROR 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] Traceback (most recent call last): 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/compute/manager.py", line 2261, in do_terminate_instance 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] self._delete_instance(context, instance, bdms, quotas) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/hooks.py", line 103, in inner 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] rv = f(*args, **kwargs) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/compute/manager.py", line 2231, in _delete_instance 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] quotas.rollback() 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__ 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] six.reraise(self.type_, self.value, self.tb) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/compute/manager.py", line 2203, in _delete_instance 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] self._shutdown_instance(context, db_inst, bdms) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/compute/manager.py", line 2145, in _shutdown_instance 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] requested_networks) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__ 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] six.reraise(self.type_, self.value, self.tb) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/compute/manager.py", line 2135, in _shutdown_instance 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] block_device_info) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/virt/libvirt/driver.py", line 955, in destroy 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] destroy_disks) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/virt/libvirt/driver.py", line 991, in cleanup 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] self.unplug_vifs(instance, network_info) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/virt/libvirt/driver.py", line 863, in unplug_vifs 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] self.vif_driver.unplug(instance, vif) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/virt/libvirt/vif.py", line 783, in unplug 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] self.unplug_ovs(instance, vif) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/virt/libvirt/vif.py", line 667, in unplug_ovs 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] self.unplug_ovs_hybrid(instance, vif) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/virt/libvirt/vif.py", line 661, in unplug_ovs_hybrid 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] v2_name) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/network/linux_net.py", line 1318, in delete_ovs_vif_port 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] _ovs_vsctl(['del-port', bridge, dev]) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/network/linux_net.py", line 1302, in _ovs_vsctl 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] raise exception.AgentError(method=full_args) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] AgentError: Error during following call to agent: ['ovs-vsctl', '--timeout=120', 'del-port', 'br-int', u'qvo67a96e96-10'] ** Affects: nova Importance: Undecided Assignee: Phil Day (philip-day) Status: New ** Changed in: nova Assignee: (unassigned) => Phil Day (philip-day) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1308544 Title: libvirt: Trying to delete a non-existing vif raises an exception Status in OpenStack Compute (Nova): New Bug description: If an instance fails during its network creation (for example if the network-vif-plugged event doesn't arrive in time) a subsequent delete will also fail when it tries to delete the vif, leaving the instance in a Error(deleting) state. This can be avoided by including the "--if-exists" option to the ovs=vsctl command. Example of stack trace: 2014-04-16 12:28:51.949 AUDIT nova.compute.manager [req-af72c100-5d9b-44f6-b941-3d72529b3401 demo demo] [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] Terminating instance 2014-04-16 12:28:52.309 ERROR nova.virt.libvirt.driver [-] [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] During wait destroy, instance disappeared. 2014-04-16 12:28:52.407 ERROR nova.network.linux_net [req-af72c100-5d9b-44f6-b941-3d72529b3401 demo demo] Unable to execute ['ovs-vsctl', '--timeout=120', 'del-port', 'br-int', u'qvo67a96e96-10']. Exception: Unexpected error while running command. Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ovs-vsctl --timeout=120 del-port br-int qvo67a96e96-10 Exit code: 1 Stdout: '' Stderr: 'ovs-vsctl: no port named qvo67a96e96-10\n' 2014-04-16 12:28:52.573 ERROR nova.compute.manager [req-af72c100-5d9b-44f6-b941-3d72529b3401 demo demo] [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] Setting instance vm_state to ERROR 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] Traceback (most recent call last): 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/compute/manager.py", line 2261, in do_terminate_instance 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] self._delete_instance(context, instance, bdms, quotas) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/hooks.py", line 103, in inner 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] rv = f(*args, **kwargs) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/compute/manager.py", line 2231, in _delete_instance 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] quotas.rollback() 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__ 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] six.reraise(self.type_, self.value, self.tb) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/compute/manager.py", line 2203, in _delete_instance 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] self._shutdown_instance(context, db_inst, bdms) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/compute/manager.py", line 2145, in _shutdown_instance 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] requested_networks) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__ 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] six.reraise(self.type_, self.value, self.tb) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/compute/manager.py", line 2135, in _shutdown_instance 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] block_device_info) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/virt/libvirt/driver.py", line 955, in destroy 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] destroy_disks) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/virt/libvirt/driver.py", line 991, in cleanup 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] self.unplug_vifs(instance, network_info) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/virt/libvirt/driver.py", line 863, in unplug_vifs 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] self.vif_driver.unplug(instance, vif) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/virt/libvirt/vif.py", line 783, in unplug 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] self.unplug_ovs(instance, vif) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/virt/libvirt/vif.py", line 667, in unplug_ovs 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] self.unplug_ovs_hybrid(instance, vif) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/virt/libvirt/vif.py", line 661, in unplug_ovs_hybrid 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] v2_name) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/network/linux_net.py", line 1318, in delete_ovs_vif_port 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] _ovs_vsctl(['del-port', bridge, dev]) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/network/linux_net.py", line 1302, in _ovs_vsctl 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] raise exception.AgentError(method=full_args) 2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] AgentError: Error during following call to agent: ['ovs-vsctl', '--timeout=120', 'del-port', 'br-int', u'qvo67a96e96-10'] To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1308544/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp