Public bug reported:

Mohammed reported this in the nova channel today [1] and the RDO cloud
people have run into the same issue too. The deployment got into a
situation where instances would show up in a 'nova list' in
BUILD/scheduling state but were unable to be deleted. (They show up in
'nova list' because 'nova list' lists build requests and all instances
in all cells).

Inspection of the database showed that the "instance" had a build
request but *no* instance mapping and *no* instance record in any cell.
And the instance could not be deleted even though it appeared in the
'nova list' because the delete API first does a compute API().get in
order to get the instance object to pass down to the compute
API().delete method. The compute API().get fails with InstanceNotFound
because the _get_instance method raises InstanceNotFound if there is no
instance mapping for the instance.

Mohammed was able to share this trace [2] which shows the
instance_mapping.create() failing due to database errors, right after
the build_request.create() succeeded:


2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/nova/compute/api.py", 
line 937, in _provision_instances
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
inst_mapping.create()
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/oslo_versionedobjects/base.py",
 line 226, in wrapper
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     return fn(self, 
*args, **kwargs)
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/nova/objects/instance_mapping.py",
 line 92, in create
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     db_mapping = 
self._create_in_db(self._context, changes)
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py",
 line 986, in wrapper
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     return fn(*args, 
**kwargs)
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     self.gen.next()
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py",
 line 1036, in _transaction_scope
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     yield resource
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     self.gen.next()
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py",
 line 646, in _session
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
self.session.rollback()
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
 line 907, in rollback
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
self.transaction.rollback()
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
 line 532, in rollback
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
util.reraise(*rollback_err)
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
 line 497, in rollback
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     t[1].rollback()
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
 line 1632, in rollback
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
self._do_rollback()
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
 line 1670, in _do_rollback
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
self.connection._rollback_impl()
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
 line 706, in _rollback_impl
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
self._handle_dbapi_exception(e, None, None, None, None)
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
 line 1406, in _handle_dbapi_exception
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
self._autorollback()
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py",
 line 76, in __exit__
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
compat.reraise(type_, value, traceback)
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
 line 1406, in _handle_dbapi_exception
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
self._autorollback()
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
 line 827, in _autorollback
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
self._root._rollback_impl()
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
 line 706, in _rollback_impl
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
self._handle_dbapi_exception(e, None, None, None, None)
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
 line 1334, in _handle_dbapi_exception
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     exc_info
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/util/compat.py",
 line 203, in raise_from_cause
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
reraise(type(exception), exception, tb=exc_tb, cause=cause)
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
 line 704, in _rollback_impl
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
self.engine.dialect.do_rollback(self.connection)
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py",
 line 1773, in do_rollback
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
dbapi_connection.rollback()
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/pymysql/connections.py",
 line 786, in rollback
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
self._read_ok_packet()
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/pymysql/connections.py",
 line 760, in _read_ok_packet
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     pkt = 
self._read_packet()
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/pymysql/connections.py",
 line 1018, in _read_packet
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
packet.check_error()
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/pymysql/connections.py",
 line 384, in check_error
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
err.raise_mysql_exception(self._data)
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/pymysql/err.py", line 
107, in raise_mysql_exception
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     raise 
errorclass(errno, errval)
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi InternalError: 
(pymysql.err.InternalError) (1047, u'WSREP has not yet prepared node for 
application use') (Background on this error at: http://sqlalche.me/e/2j85)
2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi 


and the API request returned with a 500:

"HTTP exception thrown: Unexpected API Error. Please report this at
http://bugs.launchpad.net/nova/ and attach the Nova API log if
possible."

Mohammed is going to try a fix to do the build request and instance
mapping creates in a single database transaction, so that the build
request cannot be orphaned.

Another way to handle it would be to leave the creates as-is and make
the API handle deletion of orphaned build requests, but doing that would
allow another avenue for instances in ERROR state, whereas doing the
build request and instance mapping creates in a single transaction would
avoid that.

[1] 
http://eavesdrop.openstack.org/irclogs/%23openstack-nova/latest.log.html#t2018-07-28T00:27:59
[2] http://paste.openstack.org/show/726772

** Affects: nova
     Importance: Medium
     Assignee: Mohammed Naser (mnaser)
         Status: Confirmed


** Tags: cells

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1784093

Title:
  Build requests can be orphaned without instance mappings

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  Mohammed reported this in the nova channel today [1] and the RDO cloud
  people have run into the same issue too. The deployment got into a
  situation where instances would show up in a 'nova list' in
  BUILD/scheduling state but were unable to be deleted. (They show up in
  'nova list' because 'nova list' lists build requests and all instances
  in all cells).

  Inspection of the database showed that the "instance" had a build
  request but *no* instance mapping and *no* instance record in any
  cell. And the instance could not be deleted even though it appeared in
  the 'nova list' because the delete API first does a compute API().get
  in order to get the instance object to pass down to the compute
  API().delete method. The compute API().get fails with InstanceNotFound
  because the _get_instance method raises InstanceNotFound if there is
  no instance mapping for the instance.

  Mohammed was able to share this trace [2] which shows the
  instance_mapping.create() failing due to database errors, right after
  the build_request.create() succeeded:

  
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/nova/compute/api.py", 
line 937, in _provision_instances
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
inst_mapping.create()
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/oslo_versionedobjects/base.py",
 line 226, in wrapper
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     return 
fn(self, *args, **kwargs)
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/nova/objects/instance_mapping.py",
 line 92, in create
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     db_mapping = 
self._create_in_db(self._context, changes)
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py",
 line 986, in wrapper
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     return 
fn(*args, **kwargs)
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     self.gen.next()
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py",
 line 1036, in _transaction_scope
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     yield resource
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     self.gen.next()
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py",
 line 646, in _session
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
self.session.rollback()
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
 line 907, in rollback
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
self.transaction.rollback()
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
 line 532, in rollback
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
util.reraise(*rollback_err)
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
 line 497, in rollback
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     t[1].rollback()
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
 line 1632, in rollback
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
self._do_rollback()
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
 line 1670, in _do_rollback
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
self.connection._rollback_impl()
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
 line 706, in _rollback_impl
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
self._handle_dbapi_exception(e, None, None, None, None)
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
 line 1406, in _handle_dbapi_exception
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
self._autorollback()
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py",
 line 76, in __exit__
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
compat.reraise(type_, value, traceback)
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
 line 1406, in _handle_dbapi_exception
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
self._autorollback()
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
 line 827, in _autorollback
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
self._root._rollback_impl()
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
 line 706, in _rollback_impl
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
self._handle_dbapi_exception(e, None, None, None, None)
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
 line 1334, in _handle_dbapi_exception
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     exc_info
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/util/compat.py",
 line 203, in raise_from_cause
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
reraise(type(exception), exception, tb=exc_tb, cause=cause)
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
 line 704, in _rollback_impl
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
self.engine.dialect.do_rollback(self.connection)
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py",
 line 1773, in do_rollback
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
dbapi_connection.rollback()
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/pymysql/connections.py",
 line 786, in rollback
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
self._read_ok_packet()
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/pymysql/connections.py",
 line 760, in _read_ok_packet
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     pkt = 
self._read_packet()
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/pymysql/connections.py",
 line 1018, in _read_packet
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
packet.check_error()
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/pymysql/connections.py",
 line 384, in check_error
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     
err.raise_mysql_exception(self._data)
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi   File 
"/openstack/venvs/nova-17.0.3/lib/python2.7/site-packages/pymysql/err.py", line 
107, in raise_mysql_exception
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi     raise 
errorclass(errno, errval)
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi InternalError: 
(pymysql.err.InternalError) (1047, u'WSREP has not yet prepared node for 
application use') (Background on this error at: http://sqlalche.me/e/2j85)
  2018-07-25 04:20:12.946 7926 ERROR nova.api.openstack.wsgi 

  
  and the API request returned with a 500:

  "HTTP exception thrown: Unexpected API Error. Please report this at
  http://bugs.launchpad.net/nova/ and attach the Nova API log if
  possible."

  Mohammed is going to try a fix to do the build request and instance
  mapping creates in a single database transaction, so that the build
  request cannot be orphaned.

  Another way to handle it would be to leave the creates as-is and make
  the API handle deletion of orphaned build requests, but doing that
  would allow another avenue for instances in ERROR state, whereas doing
  the build request and instance mapping creates in a single transaction
  would avoid that.

  [1] 
http://eavesdrop.openstack.org/irclogs/%23openstack-nova/latest.log.html#t2018-07-28T00:27:59
  [2] http://paste.openstack.org/show/726772

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1784093/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to     : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

Reply via email to