** Also affects: nova/pike Importance: Undecided Status: New ** Also affects: nova/rocky Importance: Undecided Status: New
** Also affects: nova/queens Importance: Undecided Status: New ** Changed in: nova/pike Status: New => Confirmed ** Changed in: nova/queens Status: New => Confirmed ** Changed in: nova/pike Importance: Undecided => Medium ** Changed in: nova/queens Importance: Undecided => Medium ** Changed in: nova/rocky Status: New => Confirmed ** Changed in: nova/rocky Importance: Undecided => Medium -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1815949 Title: missing special-case libvirt exception during device detach Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) pike series: Confirmed Status in OpenStack Compute (nova) queens series: Confirmed Status in OpenStack Compute (nova) rocky series: Confirmed Bug description: In Pike a customer has run into the following issue: 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall [-] Dynamic interval looping call 'oslo_service.loopingcall._func' failed: libvirtError: internal error: unable to execute QEMU command 'device_del': Device 'virtio-disk15' not found 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall Traceback (most recent call last): 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 143, in _run_loop 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall result = func(*self.args, **self.kw) 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 363, in _func 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall result = f(*args, **kwargs) 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 505, in _do_wait_and_retry_detach 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall _try_detach_device(config, persistent=False, host=host) 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 467, in _try_detach_device 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall device=alternative_device_name) 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall self.force_reraise() 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall six.reraise(self.type_, self.value, self.tb) 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 451, in _try_detach_device 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall self.detach_device(conf, persistent=persistent, live=live) 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 530, in detach_device 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall self._domain.detachDeviceFlags(device_xml, flags=flags) 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall result = proxy_call(self._autowrap, f, *args, **kwargs) 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall rv = execute(f, *args, **kwargs) 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall six.reraise(c, e, tb) 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall rv = meth(*args, **kwargs) 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1217, in detachDeviceFlags 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall if ret == -1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self) 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall libvirtError: internal error: unable to execute QEMU command 'device_del': Device 'virtio-disk15' not found 2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall Based on discussion with Melanie Witt, it seems likely that nova is missing a special-case in Guest.detach_device_with_retry(). It seems likely we need to modify the conditional at line 409 of virt/libvirt/guest.py to look like 'if errcode in (libvirt.VIR_ERR_OPERATION_FAILED, libvirt.VIR_ERR_INTERNAL_ERROR):' To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1815949/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp