[Yahoo-eng-team] [Bug 1841509] [NEW] soft delete instance will be reclaimed if power on failed when do restore
Public bug reported:
I found an instance disappeared after do restore instance, check the
nova code and log, I think its a logic bug here
1. restore instance with power on failed
nova-api `restore` set `instance.task_state = task_states.RESTORING
instance.deleted_at = None`
https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/api.py#L2344
nova-compute `restore_instance` will call `self._power_on` if virt driver did
not implement the `restore` method
https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L3009
instance state will be set to None if any exceptions raise when call
`self._power_on` in `reverts_task_state`
https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L178
finally the instnace state will be set to
{vm_state=vm_state.SOFT_DELETED, task_state=None, deleted_at=None}
2. reclaim instance
nova-compute periodic task `_reclaim_queued_deletes` running every 60s,
https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L8209
it will select instance with filte `{'vm_state': vm_states.SOFT_DELETED,
'task_state': None,'host': self.host}`, the instance of step 1 will be slected
https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L8216
and it will be in the return list of `_deleted_old_enough` with its
`deleted_at=None`
https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L8430
and then be deleted soon
https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L8229
I don't think the instance should be reclaimed with the above situation
** Affects: nova
Importance: Undecided
Assignee: zhangyujun (zhangyujun)
Status: New
** Changed in: nova
Assignee: (unassigned) => zhangyujun (zhangyujun)
** Description changed:
I found an instance disappeared after do restore instance, check the
nova code and log, I think its a logic bug here
1. restore instance with power on failed
nova-api `restore` set `instance.task_state = task_states.RESTORING
instance.deleted_at = None`
https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/api.py#L2344
nova-compute `restore_instance` will call `self._power_on` if virt driver
did not implement the `restore` method
https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L3009
instance state will be set to None if any exceptions raise when call
`self._power_on` in `reverts_task_state`
https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L178
finally the instnace state will be set to
- {vm_state=vm_state.SOFT_DELETED, task_state=None, deleted=None}
+ {vm_state=vm_state.SOFT_DELETED, task_state=None, deleted_at=None}
2. reclaim instance
nova-compute periodic task `_reclaim_queued_deletes` running every 60s,
https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L8209
it will select instance with filte `{'vm_state': vm_states.SOFT_DELETED,
'task_state': None,'host': self.host}`, the instance of step 1 will be slected
https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L8216
and it will be in the return list of `_deleted_old_enough` with its
`deleted_at=None`
https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L8430
and then be deleted soon
https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L8229
I don't think the instance should be reclaimed with the above situation
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1841509
Title:
soft delete instance will be reclaimed if power on failed when do
restore
Status in OpenStack Compute (nova):
New
Bug description:
I found an instance disappeared after do restore instance, check the
nova code and log, I think its a logic bug here
1. restore instance with power on failed
nova-api `restore` set `instance.task_state = task_states.RESTORING
instance.deleted_at = None`
https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/api.py#L2344
nova-compute `restore_instance` will call `self._power_on` if virt driver
did not implement the `restore` method
https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L3009
instance state w
[Yahoo-eng-team] [Bug 1840869] [NEW] VNC Server Unauthenticated Access
Public bug reported: When nova boot a server with VNC enabled, it does not require authentication if an attacker trys to connect to the remote host directly from management network. The VNC server sometimes sends the connected user to the XDM login screen. A warning from Nessus report: VNC Server Unauthenticated Access Synopsis The remote VNC server does not require authentication. Description The VNC server installed on the remote host allows an attacker to connect to the remote host as no authentication is required to access this service. The VNC server sometimes sends the connected user to the XDM login screen. Unfortunately, Nessus cannot identify this situation. In such a case, it is not possible to go further without valid credentials and this alert may be ignored. Solution Disable the No Authentication security type. ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1840869 Title: VNC Server Unauthenticated Access Status in OpenStack Compute (nova): New Bug description: When nova boot a server with VNC enabled, it does not require authentication if an attacker trys to connect to the remote host directly from management network. The VNC server sometimes sends the connected user to the XDM login screen. A warning from Nessus report: VNC Server Unauthenticated Access Synopsis The remote VNC server does not require authentication. Description The VNC server installed on the remote host allows an attacker to connect to the remote host as no authentication is required to access this service. The VNC server sometimes sends the connected user to the XDM login screen. Unfortunately, Nessus cannot identify this situation. In such a case, it is not possible to go further without valid credentials and this alert may be ignored. Solution Disable the No Authentication security type. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1840869/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1836029] [NEW] Got some db errors when call list servers
Public bug reported: Got some db errors when call list servers and nova-api return http code 500 2019-07-02 10:08:39 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:39.736 16 INFO nova.osapi_compute.wsgi.server [req-5804b3dd-9968-481d-b03f-ed36192abb2f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] 10.233.66.23 "GET /v2.1/d8ac61ac9f1d4e5e9aa9c4313c668834/servers/detail?project_id=d8ac61ac9f1d4e5e9aa9c4313c668834&redirect=detail_x HTTP/1.1" status: 200 len: 42583 time: 0.2910399 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] DB exception wrapped. 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last): 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters context) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 167, in execute 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters result = self._query(query) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 323, in _query 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters conn.query(q) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 836, in query 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1020, in _read_query_result 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters result.read() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1303, in read 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters first_packet = self.connection._read_packet() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 962, in _read_packet 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters packet_header = self._read_bytes(4) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 989, in _read_bytes 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters data = self._rfile.read(num_bytes) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters RuntimeError: reentrant call inside <_io.BufferedReader name=23> 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] Unexpected exceptio
[Yahoo-eng-team] [Bug 1826701] [NEW] the rbd backend root disks of the virtual machine will be cleared if launch failed when do evacuate
Public bug reported: I found that some of the instances could not get started after do evacuate failed, then I check the nova-compute log [1], that point out instance root disk volume 'c23c04c9-2a8b-492e-8130-99aafa30b563_disk' can not be found in ceph compute pool, look back to evacuate failed log [2], there was an error occur 'libvirtError: Failed to create controller memory for group: No space left on device' when do launch. check the nova code, when this error occured, the evacuate action in nova-compute function call stack is: https://github.com/openstack/nova/blob/324db786c86eeb69278736c8e9db6d22f68080e6/nova/compute/manager.py#L3044 nova.compute.mananger.ComputeManager.rebuild_instance -> _do_rebuild_instance_with_claim -> _do_rebuild_instance -> _rebuild_default_impl -> driver.spawn https://github.com/openstack/nova/blob/324db786c86eeb69278736c8e9db6d22f68080e6/nova/virt/libvirt/driver.py#L3154 nova.virt.libvirt.driver.LibvirtDriver.spawn -> _create_domain_and_network -> _cleanup_failed_start -> cleanup -> _cleanup_rbd https://github.com/openstack/nova/blob/master/nova/virt/libvirt/storage/rbd_utils.py#L360 nova.virt.libvirt.storage.rbd_utils.RBDDriver.cleanup_volumes -> _destroy_volume this logic make the instance root disk with rbd image backend was clean up in ceph, and nerver get started again, and even make the data lost, is this reasonable? [1] instance start failed log 2019-04-26 11:44:45.298 46085 WARNING nova.virt.osinfo [req-aad46bca-3bc7-48b8-98b8-c735f52a0a9c 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Cannot find OS information - Reason: (No configuration information found for operating system CentOS) 2019-04-26 11:44:45.403 46085 WARNING nova.virt.osinfo [req-56e8e35a-66b5-4d5f-8365-46bed361c6d8 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Cannot find OS information - Reason: (No configuration information found for operating system CentOS) 2019-04-26 11:44:45.424 46085 INFO os_vif [req-56e8e35a-66b5-4d5f-8365-46bed361c6d8 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:69:df:d7,bridge_name='br-int',has_traffic_filtering=True,id=89b882ea-15f0-4e2c-b3b1-a515e3a29f52,network=Network(1c212f11-51cf-4114-aebe-1fc016364426),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap89b882ea-15') 2019-04-26 11:44:45.516 46085 WARNING nova.virt.osinfo [req-ffe9f008-f920-4887-a453-424b86e9046e 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Cannot find OS information - Reason: (No configuration information found for operating system CentOS) 2019-04-26 11:44:45.536 46085 INFO os_vif [req-ffe9f008-f920-4887-a453-424b86e9046e 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:62:d6:59,bridge_name='br-int',has_traffic_filtering=True,id=cdd730dc-c806-424f-8d7a-96253e9a72b1,network=Network(1c212f11-51cf-4114-aebe-1fc016364426),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tapcdd730dc-c8') 2019-04-26 11:44:45.666 46085 WARNING nova.virt.osinfo [req-aad46bca-3bc7-48b8-98b8-c735f52a0a9c 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Cannot find OS information - Reason: (No configuration information found for operating system CentOS) 2019-04-26 11:44:45.684 46085 INFO os_vif [req-aad46bca-3bc7-48b8-98b8-c735f52a0a9c 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:c9:7d:69,bridge_name='br-int',has_traffic_filtering=True,id=06510302-8d87-4d3e-90e1-2a7bcbb14f6b,network=Network(1c212f11-51cf-4114-aebe-1fc016364426),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap06510302-8d') 2019-04-26 11:44:46.000 46085 ERROR nova.virt.libvirt.guest [req-56e8e35a-66b5-4d5f-8365-46bed361c6d8 5a55fb96f12e42f1a8402faf4593eb4a 7bcb8c85147e42e99a4f4687179a2203 - - -] Error launching a defined domain with XML: instance-00bc c23c04c9-2a8b-492e-8130-99aafa30b563 http://openstack.org/xmlns/libvirt/nova/1.0";> SIIT-SL-ES1 2019-04-26 03:44:45 32768 500 0 0 16 admin admin 33554432 33554432 16 16384 OpenStack Foundation OpenStack Nova 0.0.1 c792c755-66d4-4ef9-b7db-dd6bb7ff89f8 c23c04c9-2a8b-492e-8130-99aafa30b563 Virtual Machine hvm destroy restart destroy /usr/libexec/qemu-kvm
