** Changed in: nova/havana Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1274341
Title: reservation_commit can lead to deadlock Status in OpenStack Compute (Nova): Fix Released Status in OpenStack Compute (nova) havana series: Fix Released Bug description: reservation_commit can deadlock under load: 2014-01-29 18:51:00.470 ERROR nova.quota [req-659db2df-6124-4e6a-b86b-700aae1de805 183137d95c584efb84e773a21f2ef7a1 d8d5b06deffc45e7b258eb65ea04017c] Failed to commit reservations ['5bea6421-1648-4fe1-9d21-4232f536e031', '8b27edda-f40e-476c-a9b3-d007aa3f6aac', '58e357df-b45d-4008-9ef3-0da3d8daebbb'] 2014-01-29 18:51:00.470 4380 TRACE nova.quota Traceback (most recent call last): 2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/nova/quota.py", line 982, in commit 2014-01-29 18:51:00.470 4380 TRACE nova.quota self._driver.commit(context, reservations, project_id=project_id) 2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/nova/quota.py", line 370, in commit 2014-01-29 18:51:00.470 4380 TRACE nova.quota db.reservation_commit(context, reservations, project_id=project_id) 2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 970, in reservation_commit 2014-01-29 18:51:00.470 4380 TRACE nova.quota project_id=project_id) 2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 114, in wrapper 2014-01-29 18:51:00.470 4380 TRACE nova.quota return f(*args, **kwargs) 2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 2786, in reservation_commit 2014-01-29 18:51:00.470 4380 TRACE nova.quota for reservation in reservation_query.all(): 2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2115, in all 2014-01-29 18:51:00.470 4380 TRACE nova.quota return list(self) 2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2227, in __iter__ 2014-01-29 18:51:00.470 4380 TRACE nova.quota return self._execute_and_instances(context) 2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances 2014-01-29 18:51:00.470 4380 TRACE nova.quota result = conn.execute(querycontext.statement, self._params) 2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1449, in execute 2014-01-29 18:51:00.470 4380 TRACE nova.quota params) 2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement 2014-01-29 18:51:00.470 4380 TRACE nova.quota compiled_sql, distilled_params 2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context 2014-01-29 18:51:00.470 4380 TRACE nova.quota context) 2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context 2014-01-29 18:51:00.470 4380 TRACE nova.quota context) 2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 331, in do_execute 2014-01-29 18:51:00.470 4380 TRACE nova.quota cursor.execute(statement, parameters) 2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute 2014-01-29 18:51:00.470 4380 TRACE nova.quota self.errorhandler(self, exc, value) 2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler 2014-01-29 18:51:00.470 4380 TRACE nova.quota raise errorclass, errorvalue 2014-01-29 18:51:00.470 4380 TRACE nova.quota OperationalError: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'SELECT reservations.created_at AS reservations_created_at, reservations.updated_at AS reservations_updated_at, reservations.deleted_at AS reservations_deleted_at, reservations.deleted AS reservations_deleted, reservations.id AS reservations_id, reservations.uuid AS reservations_uuid, reservations.usage_id AS reservations_usage_id, reservations.project_id AS reservations_project_id, reservations.resource AS reservations_resource, reservations.delta AS reservations_delta, reservations.expire AS reservations_expire \nFROM reservations \nWHERE reservations.deleted = %s AND reservations.uuid IN (%s, %s, %s) FOR UPDATE' (0, '5bea6421-1648-4fe1-9d21-4232f536e031', '8b27edda-f40e-476c-a9b3-d007aa3f6aac', '58e357df-b45d-4008-9ef3-0da3d8daebbb') This can be fixed with our @_retry_on_deadlock wrapper To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1274341/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp