Hello everyone, We have been investigating the cause behind the Jenkins Check gate-tempest-dsvm-networking-ovn failures (non-voting at the moment). The failures have been happening pretty consistently with every commit. I wanted to start a conversation to get some input as to why these errors may be happening.
One kind of error is related to the following (from the q-svc logs). 2015-08-04 05:40:28.313 ERROR neutron.agent.ovsdb.impl_idl [req-c189268a-1e1d-462f-a81e-62f0a34ff490 tempest-FloatingIPAdminTestJSON-1706130555 tempest-FloatingIPAdminTestJSON-1943105894] Traceback (most recent call last): File "/opt/stack/new/neutron/neutron/agent/ovsdb/native/connection.py", line 84, in run txn.results.put(txn.do_commit()) File "/opt/stack/new/neutron/neutron/agent/ovsdb/impl_idl.py", line 99, in do_commit seqno) File "/opt/stack/new/neutron/neutron/agent/ovsdb/native/idlutils.py", line 125, in wait_for_change raise Exception("Timeout") Exception: Timeout When this error happens - in a separate thread there is DB Deadlock. Note that it's not always create_port (65%), it could be delete_port (30%), other calls (5%). There are many more of these errors (show below) than the above error. But it is always: SQL: u'UPDATE ipavailabilityranges SET first_ip=%s WHERE ipavailabilityranges.allocation_pool_id = %s AND ipavailabilityranges.first_ip = %s AND ipavailabilityranges.last_ip = %s'] 2015-08-04 05:39:37.303 9407 ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 136, in wrapper 2015-08-04 05:39:37.303 9407 ERROR oslo_db.api return f(*args, **kwargs) 2015-08-04 05:39:37.303 9407 ERROR oslo_db.api File "/opt/stack/new/networking-ovn/networking_ovn/plugin.py", line 275, in create_port 2015-08-04 05:39:37.303 9407 ERROR oslo_db.api db_port = super(OVNPlugin, self).create_port(context, port) ... 2015-08-04 05:39:37.303 9407 ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 205, in execute 2015-08-04 05:39:37.303 9407 ERROR oslo_db.api self.errorhandler(self, exc, value) 2015-08-04 05:39:37.303 9407 ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler 2015-08-04 05:39:37.303 9407 ERROR oslo_db.api raise errorclass, errorvalue 2015-08-04 05:39:37.303 9407 ERROR oslo_db.api DBDeadlock: (_mysql_exceptions.OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction') [SQL: u'UPDATE ipavailabilityranges SET first_ip=%s WHERE ipavailabilityranges.allocation_pool_id = %s AND ipavailabilityranges.first_ip = %s AND ipavailabilityranges.last_ip = %s'] [parameters: ('10.100.0.3', '851466c3-8d6b-4629-bf65-86be2f403e67', '10.100.0.2', '10.100.0.14')] 2015-08-04 05:39:37.303 9407 ERROR oslo_db.api Russell suggested removing the MYSQL_DRIVER=MySQL-python declaration from local.conf https://review.openstack.org/#/c/216413/ which results in PyMySQL as the default. With the above change the above DB errors are no longer seen in my local setup, the CI setup is having trouble with the gate-networking-ovn-python27 test now therefore the gate-tempest-dsvm-networking-ovn never runs. So there are 2 questions: Is there any impact of using PyMySQL for the Jenkins check and gates. Why is the gate-networking-ovn-python27 failing (the past couple of commits) in {0} networking_ovn.tests.unit.test_ovn_plugin.TestOvnPlugin.test_create_port_security [0.194020s] ... FAILED. Do we need another conversation to track this? Amitabha
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev