[Yahoo-eng-team] [Bug 1604862] [NEW] DBDeadlock: (_mysql_exceptions.OperationalError) while deleting user

Mikhail Nikolaenko Wed, 20 Jul 2016 08:31:18 -0700

Public bug reported:

The error happened at the end of Rally test create-delete-node-group-
templates.yaml when Rally started to remove users


>From Keystone admin log:
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi 
[req-23d801cf-baa4-410d-9d46-6eb6ca7dfa34 21fe895155b54960b09290ef179ce15e 
56b54d0fb50e40b5bb7ccdb46c8d21a7 - default default] 
(_mysql_exceptions.OperationalError) (1213, 'Deadlock found when trying to get 
lock; try restarting transaction') [SQL: u'DELETE FROM revocation_event WHERE 
revocation_event.revoked_at < %s'] [parameters: (datetime.datetime(2016, 5, 19, 
6, 57, 44, 270231),)]
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi Traceback (most recent 
call last):
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 249, in 
__call__
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi result = 
method(context, **params)
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/oslo_log/versionutils.py", line 165, in 
wrapped
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi return 
func_or_cls(*args, **kwargs)
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/identity/controllers.py", line 174, 
in delete_user
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi 
self.identity_api.delete_user(user_id, initiator)
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 124, in 
wrapped
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi __ret_val = __f(*args, 
**kwargs)
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 433, in 
wrapper
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi return f(self, *args, 
**kwargs)
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 443, in 
wrapper
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi return f(self, *args, 
**kwargs)
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 986, in 
delete_user
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi 
notifications.Audit.deleted(self._USER, user_id, initiator)
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/notifications.py", line 166, in 
deleted
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi public)
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/notifications.py", line 137, in _emit
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi public=public)
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/notifications.py", line 421, in 
_send_notification
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi 
notify_event_callbacks(SERVICE, resource_type, operation, payload)
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/notifications.py", line 316, in 
notify_event_callbacks
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi cb(service, 
resource_type, operation, payload)
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/revoke/core.py", line 84, in 
_user_callback
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi 
self.revoke_by_user(payload['resource_info'])
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 124, in 
wrapped
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi __ret_val = __f(*args, 
**kwargs)
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/revoke/core.py", line 151, in 
revoke_by_user
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi return 
self.revoke(revoke_model.RevokeEvent(user_id=user_id))
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 124, in 
wrapped
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi __ret_val = __f(*args, 
**kwargs)
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/revoke/core.py", line 229, in revoke
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi 
self.driver.revoke(event)
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/revoke/backends/sql.py", line 100, 
in revoke
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi 
self._prune_expired_events()
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/revoke/backends/sql.py", line 77, in 
_prune_expired_events
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi 
query.delete(synchronize_session=False)
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 3048, in delete
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi delete_op.exec_()
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 1127, in 
exec_
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi self._do_exec()
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 1311, in 
_do_exec
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi mapper=self.mapper)
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1034, in 
execute
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi bind, 
close_with_result=True).execute(clause, params or {})
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in 
execute
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi return meth(self, 
multiparams, params)
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in 
_execute_on_connection
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi return 
connection._execute_clauseelement(self, multiparams, params)
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in 
_execute_clauseelement
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi compiled_sql, 
distilled_params
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in 
_execute_context
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi context)
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in 
_handle_dbapi_exception
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi 
util.raise_from_cause(newraise, exc_info)
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in 
raise_from_cause
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi 
reraise(type(exception), exception, tb=exc_tb, cause=cause)
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in 
_execute_context
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi context)
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in 
do_execute
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi 
cursor.execute(statement, parameters)
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 219, in execute
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi 
self.errorhandler(self, exc, value)
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 38, in 
defaulterrorhandler
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi raise errorvalue
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi DBDeadlock: 
(_mysql_exceptions.OperationalError) (1213, 'Deadlock found when trying to get 
lock; try restarting transaction') [SQL: u'DELETE FROM revocation_event WHERE 
revocation_event.revoked_at < %s'] [parameters: (datetime.datetime(2016, 5, 19, 
6, 57, 44, 270231),)]
2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi

** Affects: keystone
     Importance: Undecided
     Assignee: Mikhail Nikolaenko (mnikolaenko)
         Status: New


** Tags: sql

** Changed in: keystone
     Assignee: (unassigned) => Mikhail Nikolaenko (mnikolaenko)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1604862

Title:
  DBDeadlock: (_mysql_exceptions.OperationalError) while deleting user

Status in OpenStack Identity (keystone):
  New

Bug description:
  The error happened at the end of Rally test create-delete-node-group-
  templates.yaml when Rally started to remove users

  From Keystone admin log:
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi 
[req-23d801cf-baa4-410d-9d46-6eb6ca7dfa34 21fe895155b54960b09290ef179ce15e 
56b54d0fb50e40b5bb7ccdb46c8d21a7 - default default] 
(_mysql_exceptions.OperationalError) (1213, 'Deadlock found when trying to get 
lock; try restarting transaction') [SQL: u'DELETE FROM revocation_event WHERE 
revocation_event.revoked_at < %s'] [parameters: (datetime.datetime(2016, 5, 19, 
6, 57, 44, 270231),)]
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi Traceback (most 
recent call last):
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 249, in 
__call__
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi result = 
method(context, **params)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/oslo_log/versionutils.py", line 165, in 
wrapped
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi return 
func_or_cls(*args, **kwargs)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/identity/controllers.py", line 174, 
in delete_user
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi 
self.identity_api.delete_user(user_id, initiator)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 124, in 
wrapped
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi __ret_val = 
__f(*args, **kwargs)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 433, in 
wrapper
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi return f(self, 
*args, **kwargs)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 443, in 
wrapper
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi return f(self, 
*args, **kwargs)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 986, in 
delete_user
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi 
notifications.Audit.deleted(self._USER, user_id, initiator)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/notifications.py", line 166, in 
deleted
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi public)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/notifications.py", line 137, in _emit
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi public=public)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/notifications.py", line 421, in 
_send_notification
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi 
notify_event_callbacks(SERVICE, resource_type, operation, payload)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/notifications.py", line 316, in 
notify_event_callbacks
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi cb(service, 
resource_type, operation, payload)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/revoke/core.py", line 84, in 
_user_callback
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi 
self.revoke_by_user(payload['resource_info'])
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 124, in 
wrapped
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi __ret_val = 
__f(*args, **kwargs)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/revoke/core.py", line 151, in 
revoke_by_user
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi return 
self.revoke(revoke_model.RevokeEvent(user_id=user_id))
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 124, in 
wrapped
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi __ret_val = 
__f(*args, **kwargs)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/revoke/core.py", line 229, in revoke
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi 
self.driver.revoke(event)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/revoke/backends/sql.py", line 100, 
in revoke
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi 
self._prune_expired_events()
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/keystone/revoke/backends/sql.py", line 77, in 
_prune_expired_events
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi 
query.delete(synchronize_session=False)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 3048, in delete
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi delete_op.exec_()
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 1127, in 
exec_
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi self._do_exec()
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 1311, in 
_do_exec
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi mapper=self.mapper)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1034, in 
execute
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi bind, 
close_with_result=True).execute(clause, params or {})
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in 
execute
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi return meth(self, 
multiparams, params)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in 
_execute_on_connection
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi return 
connection._execute_clauseelement(self, multiparams, params)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in 
_execute_clauseelement
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi compiled_sql, 
distilled_params
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in 
_execute_context
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi context)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in 
_handle_dbapi_exception
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi 
util.raise_from_cause(newraise, exc_info)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in 
raise_from_cause
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi 
reraise(type(exception), exception, tb=exc_tb, cause=cause)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in 
_execute_context
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi context)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in 
do_execute
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi 
cursor.execute(statement, parameters)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 219, in execute
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi 
self.errorhandler(self, exc, value)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi File 
"/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 38, in 
defaulterrorhandler
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi raise errorvalue
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi DBDeadlock: 
(_mysql_exceptions.OperationalError) (1213, 'Deadlock found when trying to get 
lock; try restarting transaction') [SQL: u'DELETE FROM revocation_event WHERE 
revocation_event.revoked_at < %s'] [parameters: (datetime.datetime(2016, 5, 19, 
6, 57, 44, 270231),)]
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1604862/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to     : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1604862] [NEW] DBDeadlock: (_mysql_exceptions.OperationalError) while deleting user

Reply via email to