Public bug reported:
Description
===========
Suspended instances with volumes attached fail to resume if the
/var/lib/nova/mnt/... filesystems are not mounted. The mounts could be lost due
to e.g. node reboot during upgrade.
Steps to reproduce
==================
* Boot some instance and attach volume stored in NFS (/var/lib/nova/mnt/...
should get mounted on the compute node)
* Suspend the instance ($ nova suspend <instance id or name>)
* Unmount the /var/lib/nova/mnt/... filesystem
* Try to resume the instance ($ nova resume <instance id or name>)
Expected result
===============
The instance should become ACTIVE.
Actual result
=============
Instance switches to ERROR state.
Environment
===========
openstack-nova-compute-18.2.3~dev10-3.5.noarch
python-novaclient-11.0.0-3.126.noarch
python-nova-18.2.3~dev10-3.5.noarch
openstack-nova-18.2.3~dev10-3.5.noarch
Hypervisor: Libvirt + KVM
Storage: We experienced this with NFS Cinder backend but probably other
filesystem-based volume types are affected as well.
Network: Neutron
Logs & Configs
==============
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager
[req-38561c2e-7312-47c8-aec9-40bbe24a5b8a 50269c9569ba4c388a96dd8eaa7106b2
3dab9b31bfaf4298801706aaf14ce83a - default default] [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] Setting instance vm_state to ERROR:
libvirtError: Cannot access storage file
'/var/lib/nova/mnt/3cc56405338b71a0b5bad6102bf7299b/volume-b8bc9d61-1f2f-461d-a787-7b912e8cc8fa':
No such file or directory
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] Traceback (most recent call last):
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 8208, in
_error_out_instance_on_exception
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] yield
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5027, in
resume_instance
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] block_device_info)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2997, in
resume
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] vifs_already_plugged=True)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5713, in
_create_domain_and_network
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] destroy_disks_on_failure)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] self.force_reraise()
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] six.reraise(self.type_, self.value,
self.tb)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5682, in
_create_domain_and_network
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] post_xml_callback=post_xml_callback)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5616, in
_create_domain
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] guest.launch(pause=pause)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in
launch
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] self._encoded_xml, errors='ignore')
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] self.force_reraise()
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] six.reraise(self.type_, self.value,
self.tb)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in
launch
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] return
self._domain.createWithFlags(flags)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] result = proxy_call(self._autowrap,
f, *args, **kwargs)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] rv = execute(f, *args, **kwargs)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] six.reraise(c, e, tb)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] rv = meth(*args, **kwargs)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib64/python2.7/site-packages/libvirt.py", line 1099, in createWithFlags
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] if ret == -1: raise libvirtError
('virDomainCreateWithFlags() failed', dom=self)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] libvirtError: Cannot access storage file
'/var/lib/nova/mnt/3cc56405338b71a0b5bad6102bf7299b/volume-b8bc9d61-1f2f-461d-a787-7b912e8cc8fa':
No such file or directory
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1845199
Title:
Nova-compute doesn't mount volume filesystems on resume
Status in OpenStack Compute (nova):
New
Bug description:
Description
===========
Suspended instances with volumes attached fail to resume if the
/var/lib/nova/mnt/... filesystems are not mounted. The mounts could be lost due
to e.g. node reboot during upgrade.
Steps to reproduce
==================
* Boot some instance and attach volume stored in NFS (/var/lib/nova/mnt/...
should get mounted on the compute node)
* Suspend the instance ($ nova suspend <instance id or name>)
* Unmount the /var/lib/nova/mnt/... filesystem
* Try to resume the instance ($ nova resume <instance id or name>)
Expected result
===============
The instance should become ACTIVE.
Actual result
=============
Instance switches to ERROR state.
Environment
===========
openstack-nova-compute-18.2.3~dev10-3.5.noarch
python-novaclient-11.0.0-3.126.noarch
python-nova-18.2.3~dev10-3.5.noarch
openstack-nova-18.2.3~dev10-3.5.noarch
Hypervisor: Libvirt + KVM
Storage: We experienced this with NFS Cinder backend but probably
other filesystem-based volume types are affected as well.
Network: Neutron
Logs & Configs
==============
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager
[req-38561c2e-7312-47c8-aec9-40bbe24a5b8a 50269c9569ba4c388a96dd8eaa7106b2
3dab9b31bfaf4298801706aaf14ce83a - default default] [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] Setting instance vm_state to ERROR:
libvirtError: Cannot access storage file
'/var/lib/nova/mnt/3cc56405338b71a0b5bad6102bf7299b/volume-b8bc9d61-1f2f-461d-a787-7b912e8cc8fa':
No such file or directory
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] Traceback (most recent call last):
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 8208, in
_error_out_instance_on_exception
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] yield
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5027, in
resume_instance
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] block_device_info)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2997, in
resume
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] vifs_already_plugged=True)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5713, in
_create_domain_and_network
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] destroy_disks_on_failure)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] self.force_reraise()
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] six.reraise(self.type_, self.value,
self.tb)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5682, in
_create_domain_and_network
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] post_xml_callback=post_xml_callback)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5616, in
_create_domain
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] guest.launch(pause=pause)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in
launch
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] self._encoded_xml, errors='ignore')
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] self.force_reraise()
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] six.reraise(self.type_, self.value,
self.tb)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in
launch
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] return
self._domain.createWithFlags(flags)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] result = proxy_call(self._autowrap,
f, *args, **kwargs)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] rv = execute(f, *args, **kwargs)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] six.reraise(c, e, tb)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] rv = meth(*args, **kwargs)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] File
"/usr/lib64/python2.7/site-packages/libvirt.py", line 1099, in createWithFlags
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] if ret == -1: raise libvirtError
('virDomainCreateWithFlags() failed', dom=self)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance:
a7cf0dc4-732c-4d82-97c7-9a191b347a20] libvirtError: Cannot access storage file
'/var/lib/nova/mnt/3cc56405338b71a0b5bad6102bf7299b/volume-b8bc9d61-1f2f-461d-a787-7b912e8cc8fa':
No such file or directory
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1845199/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
