Public bug reported: netdemoid=$(neutron net-list | awk '{if($4=="'demo-net'"){print $2;}}') subnetdemoid=$(neutron subnet-list | awk '{if($4=="'demo-subnet'"){print $2;}}')
exnetid=$(neutron net-list | awk '{if($4=="'ext-net'"){print $2;}}') for i in `seq 1 10`; do #boot vm, and create floating ip nova boot --image cirros --flavor m1.tiny --nic net-id=$netdemoid cirrosdemo${i} cirrosdemoid[i]=$(nova list | awk '{if($4=="'cirrosdemo${i}'"){print $2;}}') output=$(neutron floatingip-create $exnetid) echo $output floatipid[i]=$(echo "$output" | awk '{if($2=="id"){print $4;}}') floatip[i]=$(echo "$output" | awk '{if($2=="floating_ip_address"){print $4;}}')a done # Setup router neutron router-gateway-set $routerdemoid $exnetid neutron router-interface-add demo-router $subnetdemoid #wait for VM to be running sleep 30 for i in `seq 1 10`; do cirrosfix=$(nova list | awk '{if($4=="'cirrosdemo${i}'"){print $12;}}') cirrosfixip=${cirrosfix#*=} output=$(neutron port-list | grep ${cirrosfixip}) echo $output portid=$(echo "$output" | awk '{print $2;}') neutron floatingip-associate --fixed-ip-address $cirrosfixip ${floatipid[i]} $portid neutron floatingip-delete ${floatipid[i]} nova delete ${cirrosdemoid[i]} done With several tries, I have one instance in ERROR state: 2014-11-19 19:41:02.670 8659 DEBUG neutron.context [req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 None] Arguments dropped when creating context: {u'project_name': None, u'tenant': None} __init__ /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/context.py:83 2014-11-19 19:41:02.671 8659 DEBUG neutron.plugins.ml2.rpc [req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 None] Device 498e7a54-22dd-4e5b-a8db-d6bffb8edd25 details requested by agent ovs-agent-overcloud-controller0-d5wwhbhhtlmp with host overcloud-controller0-d5wwhbhhtlmp get_device_details /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py:90 2014-11-19 19:41:02.707 8659 DEBUG neutron.openstack.common.lockutils [req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 None] Got semaphore "db-access" lock /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/openstack/common/lockutils.py:168 2014-11-19 19:41:04.061 8658 ERROR oslo.messaging.rpc.dispatcher [req-4303cd41-c87c-44aa-b78a-549fb914ac9c ] Exception during message handling: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last): 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher incoming.message)) 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args) 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args) 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/db/agents_db.py", line 220, in report_state 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher self.plugin.create_or_update_agent(context, agent_state) 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/db/agents_db.py", line 180, in create_or_update_agent 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher return self._create_or_update_agent(context, agent) 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/db/agents_db.py", line 174, in _create_or_update_agent 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher greenthread.sleep(0) 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 470, in __exit__ 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher self.rollback() 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__ 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher compat.reraise(exc_type, exc_value, exc_tb) 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 467, in __exit__ 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher self.commit() 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 381, in commit 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher t[1].commit() 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in commit 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher self._do_commit() 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1365, in _do_commit 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher self.connection._commit_impl() 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 527, in _commit_impl 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher self._handle_dbapi_exception(e, None, None, None, None) 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1160, in _handle_dbapi_exception 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher exc_info 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher reraise(type(exception), exception, tb=exc_tb) 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 525, in _commit_impl 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher self.engine.dialect.do_commit(self.connection) 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2251, in do_commit 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher dbapi_connection.commit() 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher OperationalError: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher 2014-11-19 19:41:04.071 8659 ERROR oslo.messaging.rpc.dispatcher [req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 ] Exception during message handling: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last): 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher incoming.message)) 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args) 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args) 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 145, in get_devices_details_list 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher for device in kwargs.pop('devices', []) 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 122, in get_device_details 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher host) 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1146, in update_port_status 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher updated = True 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher self.gen.next() 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/contextlib.py", line 121, in nested 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher if exit(*exc): 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 470, in __exit__ 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher self.rollback() 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__ 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher compat.reraise(exc_type, exc_value, exc_tb) 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 467, in __exit__ 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher self.commit() 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 381, in commit 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher t[1].commit() 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in commit 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher self._do_commit() 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1365, in _do_commit 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher self.connection._commit_impl() 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 527, in _commit_impl 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher self._handle_dbapi_exception(e, None, None, None, None) 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1160, in _handle_dbapi_exception 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher exc_info 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher reraise(type(exception), exception, tb=exc_tb) 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 525, in _commit_impl 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher self.engine.dialect.do_commit(self.connection) 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2251, in do_commit 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher dbapi_connection.commit() 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher OperationalError: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher 2014-11-19 19:41:04.084 8659 ERROR oslo.messaging._drivers.common [req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 ] Returning exception (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None to caller 2014-11-19 19:41:04.084 8659 ERROR oslo.messaging._drivers.common [req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 ] ['Traceback (most recent call last):\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 145, in get_devices_details_list\n for device in kwargs.pop(\'devices\', [])\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 122, in get_device_details\n host)\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1146, in update_port_status\n updated = True\n', ' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python2.7/contextlib.py", line 121, in nested\n if exit(*exc):\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 470, in __exit__\n self.rollback()\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__\n compat.reraise(exc_type, exc_value, exc_tb)\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 467, in __exit__\n self.commit()\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 381, in commit\n t[1].commit()\n', ' File "/opt/st ack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in commit\n self._do_commit()\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1365, in _do_commit\n self.connection._commit_impl()\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 527, in _commit_impl\n self._handle_dbapi_exception(e, None, None, None, None)\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1160, in _handle_dbapi_exception\n exc_info\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause\n reraise(type(exception), exception, tb=exc_tb)\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 525, in _commit_impl\n self.engine.dialect.do_commit(self.connection)\n ', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2251, in do_commit\n dbapi_connection.commit()\n', "OperationalError: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None\n"] ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1394351 Title: deadlock when delete port Status in OpenStack Compute (Nova): New Bug description: netdemoid=$(neutron net-list | awk '{if($4=="'demo-net'"){print $2;}}') subnetdemoid=$(neutron subnet-list | awk '{if($4=="'demo-subnet'"){print $2;}}') exnetid=$(neutron net-list | awk '{if($4=="'ext-net'"){print $2;}}') for i in `seq 1 10`; do #boot vm, and create floating ip nova boot --image cirros --flavor m1.tiny --nic net-id=$netdemoid cirrosdemo${i} cirrosdemoid[i]=$(nova list | awk '{if($4=="'cirrosdemo${i}'"){print $2;}}') output=$(neutron floatingip-create $exnetid) echo $output floatipid[i]=$(echo "$output" | awk '{if($2=="id"){print $4;}}') floatip[i]=$(echo "$output" | awk '{if($2=="floating_ip_address"){print $4;}}')a done # Setup router neutron router-gateway-set $routerdemoid $exnetid neutron router-interface-add demo-router $subnetdemoid #wait for VM to be running sleep 30 for i in `seq 1 10`; do cirrosfix=$(nova list | awk '{if($4=="'cirrosdemo${i}'"){print $12;}}') cirrosfixip=${cirrosfix#*=} output=$(neutron port-list | grep ${cirrosfixip}) echo $output portid=$(echo "$output" | awk '{print $2;}') neutron floatingip-associate --fixed-ip-address $cirrosfixip ${floatipid[i]} $portid neutron floatingip-delete ${floatipid[i]} nova delete ${cirrosdemoid[i]} done With several tries, I have one instance in ERROR state: 2014-11-19 19:41:02.670 8659 DEBUG neutron.context [req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 None] Arguments dropped when creating context: {u'project_name': None, u'tenant': None} __init__ /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/context.py:83 2014-11-19 19:41:02.671 8659 DEBUG neutron.plugins.ml2.rpc [req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 None] Device 498e7a54-22dd-4e5b-a8db-d6bffb8edd25 details requested by agent ovs-agent-overcloud-controller0-d5wwhbhhtlmp with host overcloud-controller0-d5wwhbhhtlmp get_device_details /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py:90 2014-11-19 19:41:02.707 8659 DEBUG neutron.openstack.common.lockutils [req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 None] Got semaphore "db-access" lock /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/openstack/common/lockutils.py:168 2014-11-19 19:41:04.061 8658 ERROR oslo.messaging.rpc.dispatcher [req-4303cd41-c87c-44aa-b78a-549fb914ac9c ] Exception during message handling: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last): 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher incoming.message)) 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args) 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args) 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/db/agents_db.py", line 220, in report_state 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher self.plugin.create_or_update_agent(context, agent_state) 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/db/agents_db.py", line 180, in create_or_update_agent 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher return self._create_or_update_agent(context, agent) 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/db/agents_db.py", line 174, in _create_or_update_agent 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher greenthread.sleep(0) 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 470, in __exit__ 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher self.rollback() 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__ 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher compat.reraise(exc_type, exc_value, exc_tb) 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 467, in __exit__ 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher self.commit() 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 381, in commit 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher t[1].commit() 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in commit 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher self._do_commit() 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1365, in _do_commit 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher self.connection._commit_impl() 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 527, in _commit_impl 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher self._handle_dbapi_exception(e, None, None, None, None) 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1160, in _handle_dbapi_exception 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher exc_info 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher reraise(type(exception), exception, tb=exc_tb) 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 525, in _commit_impl 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher self.engine.dialect.do_commit(self.connection) 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2251, in do_commit 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher dbapi_connection.commit() 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher OperationalError: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None 2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher 2014-11-19 19:41:04.071 8659 ERROR oslo.messaging.rpc.dispatcher [req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 ] Exception during message handling: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last): 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher incoming.message)) 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args) 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args) 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 145, in get_devices_details_list 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher for device in kwargs.pop('devices', []) 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 122, in get_device_details 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher host) 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1146, in update_port_status 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher updated = True 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher self.gen.next() 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/contextlib.py", line 121, in nested 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher if exit(*exc): 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 470, in __exit__ 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher self.rollback() 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__ 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher compat.reraise(exc_type, exc_value, exc_tb) 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 467, in __exit__ 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher self.commit() 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 381, in commit 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher t[1].commit() 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in commit 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher self._do_commit() 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1365, in _do_commit 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher self.connection._commit_impl() 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 527, in _commit_impl 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher self._handle_dbapi_exception(e, None, None, None, None) 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1160, in _handle_dbapi_exception 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher exc_info 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher reraise(type(exception), exception, tb=exc_tb) 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 525, in _commit_impl 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher self.engine.dialect.do_commit(self.connection) 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2251, in do_commit 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher dbapi_connection.commit() 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher OperationalError: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None 2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher 2014-11-19 19:41:04.084 8659 ERROR oslo.messaging._drivers.common [req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 ] Returning exception (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None to caller 2014-11-19 19:41:04.084 8659 ERROR oslo.messaging._drivers.common [req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 ] ['Traceback (most recent call last):\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 145, in get_devices_details_list\n for device in kwargs.pop(\'devices\', [])\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py", line 12 2, in get_device_details\n host)\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1146, in update_port_status\n updated = True\n', ' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python2.7/contextlib.py", line 121, in nested\n if exit(*exc):\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 470, in __exit__\n self.rollback()\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__\n compat.reraise(exc_type, exc_value, exc_tb)\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 467, in __exit__\n self.commit()\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 381, in commit\n t[1].commit()\n', ' File "/opt/ stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in commit\n self._do_commit()\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1365, in _do_commit\n self.connection._commit_impl()\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 527, in _commit_impl\n self._handle_dbapi_exception(e, None, None, None, None)\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1160, in _handle_dbapi_exception\n exc_info\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause\n reraise(type(exception), exception, tb=exc_tb)\n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 525, in _commit_impl\n self.engine.dialect.do_commit(self.connection) \n', ' File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2251, in do_commit\n dbapi_connection.commit()\n', "OperationalError: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None\n"] To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1394351/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp