Public bug reported:
In order to test HA for controller node: shudown one controller node and
then vip switch to another node. Creating instance at this situation got
the Traceback in conductor:
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
[req-503920a4-f2d8-47a8-a8d2-f9e4297be0ed a5be69ec54db4db2bf5aedc997da6371
0cdf4389dd09480098f4874e31585882 - - -] Exception during message handling
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server Traceback (most
recent call last):
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in
_process_incoming
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 222,
in dispatch
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192,
in _do_dispatch
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 930, in
schedule_and_build_instances
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server instance.flavor,
instance.uuid, block_device_mapping)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 799, in
_create_block_device_mapping
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
bdm.update_or_create()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in
wrapper
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server return fn(self,
*args, **kwargs)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/objects/block_device.py", line 166, in
update_or_create
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
self._create(self._context, update_or_create=True)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/objects/block_device.py", line 145, in
_create
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server context,
updates, legacy=False)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/db/api.py", line 1295, in
block_device_mapping_update_or_create
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server return
IMPL.block_device_mapping_update_or_create(context, values, legacy)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 251, in
wrapped
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server return
f(context, *args, **kwargs)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server self.gen.next()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line
945, in _transaction_scope
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server yield resource
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server self.gen.next()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line
562, in _session
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
self._end_session_transaction(self.session)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line
583, in _end_session_transaction
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server session.commit()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 813, in
commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
self.transaction.commit()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 396, in
commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server t[1].commit()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1574, in
commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server self._do_commit()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1605, in
_do_commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
self.connection._commit_impl()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 690, in
_commit_impl
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
self._handle_dbapi_exception(e, None, None, None, None)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in
_handle_dbapi_exception
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
util.raise_from_cause(newraise, exc_info)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in
raise_from_cause
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
reraise(type(exception), exception, tb=exc_tb)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 688, in
_commit_impl
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
self.engine.dialect.do_commit(self.connection)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line
2514, in do_commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
dbapi_connection.commit()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/pymysql/connections.py", line 773, in commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
self._read_ok_packet()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/pymysql/connections.py", line 752, in
_read_ok_packet
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server pkt =
self._read_packet()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/pymysql/connections.py", line 991, in
_read_packet
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
packet.check_error()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in
check_error
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
err.raise_mysql_exception(self._data)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in
raise_mysql_exception
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server raise
errorclass(errno, errval)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server DBDeadlock:
(pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock;
try restarting transaction')
Always got that in _create_block_device_mapping.
openstack version：ocata
nova --version
10.2.0
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1791204
Title:
nova conductor update database get deadlock
Status in OpenStack Compute (nova):
New
Bug description:
In order to test HA for controller node: shudown one controller node
and then vip switch to another node. Creating instance at this
situation got the Traceback in conductor:
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
[req-503920a4-f2d8-47a8-a8d2-f9e4297be0ed a5be69ec54db4db2bf5aedc997da6371
0cdf4389dd09480098f4874e31585882 - - -] Exception during message handling
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server Traceback (most
recent call last):
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in
_process_incoming
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 222,
in dispatch
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192,
in _do_dispatch
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 930, in
schedule_and_build_instances
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
instance.flavor, instance.uuid, block_device_mapping)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 799, in
_create_block_device_mapping
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
bdm.update_or_create()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in
wrapper
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server return
fn(self, *args, **kwargs)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/objects/block_device.py", line 166, in
update_or_create
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
self._create(self._context, update_or_create=True)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/objects/block_device.py", line 145, in
_create
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server context,
updates, legacy=False)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/db/api.py", line 1295, in
block_device_mapping_update_or_create
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server return
IMPL.block_device_mapping_update_or_create(context, values, legacy)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 251, in
wrapped
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server return
f(context, *args, **kwargs)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server self.gen.next()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line
945, in _transaction_scope
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server yield resource
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server self.gen.next()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line
562, in _session
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
self._end_session_transaction(self.session)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line
583, in _end_session_transaction
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
session.commit()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 813, in
commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
self.transaction.commit()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 396, in
commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server t[1].commit()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1574, in
commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
self._do_commit()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1605, in
_do_commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
self.connection._commit_impl()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 690, in
_commit_impl
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
self._handle_dbapi_exception(e, None, None, None, None)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in
_handle_dbapi_exception
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
util.raise_from_cause(newraise, exc_info)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in
raise_from_cause
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
reraise(type(exception), exception, tb=exc_tb)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 688, in
_commit_impl
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
self.engine.dialect.do_commit(self.connection)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line
2514, in do_commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
dbapi_connection.commit()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/pymysql/connections.py", line 773, in commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
self._read_ok_packet()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/pymysql/connections.py", line 752, in
_read_ok_packet
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server pkt =
self._read_packet()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/pymysql/connections.py", line 991, in
_read_packet
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
packet.check_error()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in
check_error
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server
err.raise_mysql_exception(self._data)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in
raise_mysql_exception
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server raise
errorclass(errno, errval)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server DBDeadlock:
(pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock;
try restarting transaction')
Always got that in _create_block_device_mapping.
openstack version：ocata
nova --version
10.2.0
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1791204/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
