Added neutron as it doesn't seem networking-midonet specific ** Description changed:
For following scenarios, max_seconds_per_iteration validation is failing miserably. IIRC, these scenarios used to work. I don't remember when it was though. (Maybe newton timeframe) NetworkPlugin.create_ports NetworkPlugin.create_routers NetworkPlugin.create_subnets NetworkPlugin.create_subnets_routers_interfaces It's probably due to retriable exceptions like the following due to contentions. eg. http://logs.openstack.org/74/492374/3/check/gate-networking-midonet- rally-dsvm-ml2-ubuntu-xenial/d1aa9b9/logs/screen-q-svc.txt.gz Aug 14 06:08:16.188519 ubuntu-xenial-citycloud-kna1-10421138 neutron- server[14391]: DEBUG neutron.db.api [None req- 6c986099-0e8d-4181-a488-c73d348f2ef5 c_rally_1a0448a8_y9LQrXNX c_rally_1a0448a8_QH0dDRH4] Retry wrapper got retriable exception: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched. {{(pid=14479) wrapped /opt/stack/new/neutron/neutron/db/api.py:129}} Aug 14 05:46:27.279500 ubuntu-xenial-citycloud-kna1-10421138 neutron- server[14391]: DEBUG neutron.db.api [None req-6134c462-06e4-493f- 8ff1-9b206dfda3dd c_rally_edf0ce7f_CaBvNcQe c_rally_edf0ce7f_ONZDWSHE] Retry wrapper got retriable exception: Failed to create a duplicate IpamAllocation: for attribute(s) ['PRIMARY'] with value(s) 2.224.0.54-29522f59-ea05-4904-b034-fa3555da8ade {{(pid=14479) wrapped /opt/stack/new/neutron/neutron/db/api.py:129}} Aug 14 05:39:37.960137 ubuntu-xenial-citycloud-kna1-10421138 neutron- server[14391]: DEBUG neutron.db.api [None req-f3ecc8f3-2c7a-4b3c-ac8c- abe2766e36f4 c_rally_6464adb0_pXKwRzJz c_rally_6464adb0_uyMbSZqP] Retry wrapper got retriable exception: Failed to create a duplicate DefaultSecurityGroup: for attribute(s) ['PRIMARY'] with value(s) bc3b6a26a56646e7b098b9419d17c0d1 {{(pid=14480) wrapped /opt/stack/new/neutron/neutron/db/api.py:129}} + + + Even without max_seconds_per_iteration, it sometimes times out, + or causes internal error reaching to the retry limit. + + eg. http://logs.openstack.org/74/492374/4/check/gate-networking-midonet- + rally-dsvm-ml2-ubuntu- + xenial/5f6c27d/logs/screen-q-svc.txt.gz?level=TRACE + + Aug 14 13:54:59.896408 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api [None req-c66e03cb-f36a-494b-9f83-fcb51dd0a420 c_rally_732dd379_PVser3Et c_rally_732dd379_0ywBv9Oh] DB exceeded retry limit.: StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched. + Aug 14 13:54:59.896764 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api Traceback (most recent call last): + Aug 14 13:54:59.896968 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper + Aug 14 13:54:59.897135 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api return f(*args, **kwargs) + Aug 14 13:54:59.897300 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped + Aug 14 13:54:59.897465 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api LOG.debug("Retry wrapper got retriable exception: %s", e) + Aug 14 13:54:59.897653 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ + Aug 14 13:54:59.897819 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api self.force_reraise() + Aug 14 13:54:59.897970 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise + Aug 14 13:54:59.898135 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api six.reraise(self.type_, self.value, self.tb) + Aug 14 13:54:59.898300 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped + Aug 14 13:54:59.898458 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api return f(*dup_args, **dup_kwargs) + Aug 14 13:54:59.898621 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 1013, in delete_subnet + Aug 14 13:54:59.898792 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api self.ipam.delete_subnet(context, id) + Aug 14 13:54:59.898948 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 162, in delete_subnet + Aug 14 13:54:59.899122 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api ipam_driver.remove_subnet(subnet_id) + Aug 14 13:54:59.899283 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 317, in remove_subnet + Aug 14 13:54:59.899439 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api subnet_id) + Aug 14 13:54:59.899595 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/db_api.py", line 68, in delete + Aug 14 13:54:59.899786 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api neutron_subnet_id=neutron_subnet_id) + Aug 14 13:54:59.899962 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/objects/base.py", line 512, in delete_objects + Aug 14 13:54:59.900125 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api context, cls.db_model, **cls.modify_fields_to_db(kwargs)) + Aug 14 13:54:59.904444 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/objects/db/api.py", line 116, in delete_objects + Aug 14 13:54:59.904812 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api db_objs = get_objects(context, model, **kwargs) + Aug 14 13:54:59.904985 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/objects/db/api.py", line 53, in get_objects + Aug 14 13:54:59.905160 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api **(_pager.to_kwargs(context, model) if _pager else {})) + Aug 14 13:54:59.905333 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/db/_model_query.py", line 292, in get_collection + Aug 14 13:54:59.905499 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api for c in query + Aug 14 13:54:59.905680 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2854, in __iter__ + Aug 14 13:54:59.905843 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api self.session._autoflush() + Aug 14 13:54:59.906008 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1397, in _autoflush + Aug 14 13:54:59.906172 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api self.flush() + Aug 14 13:54:59.906326 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2171, in flush + Aug 14 13:54:59.906495 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api self._flush(objects) + Aug 14 13:54:59.906674 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2291, in _flush + Aug 14 13:54:59.906814 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api transaction.rollback(_capture_exception=True) + Aug 14 13:54:59.906946 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ + Aug 14 13:54:59.907093 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api compat.reraise(exc_type, exc_value, exc_tb) + Aug 14 13:54:59.907256 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2255, in _flush + Aug 14 13:54:59.907416 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api flush_context.execute() + Aug 14 13:54:59.907607 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute + Aug 14 13:54:59.908653 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api rec.execute(self) + Aug 14 13:54:59.909744 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute + Aug 14 13:54:59.910728 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api uow + Aug 14 13:54:59.910902 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 177, in save_obj + Aug 14 13:54:59.911064 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api mapper, table, update) + Aug 14 13:54:59.911219 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 760, in _emit_update_statements + Aug 14 13:54:59.911373 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api (table.description, len(records), rows)) + Aug 14 13:54:59.911623 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched. + Aug 14 13:54:59.911802 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api + Aug 14 13:54:59.911951 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR neutron.pecan_wsgi.hooks.translation [None req-c66e03cb-f36a-494b-9f83-fcb51dd0a420 c_rally_732dd379_PVser3Et c_rally_732dd379_0ywBv9Oh] DELETE failed.: StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched. ** Also affects: neutron Importance: Undecided Status: New ** Summary changed: - rally sla failure + rally sla failure / internal error on load ** Description changed: - For following scenarios, max_seconds_per_iteration validation is failing miserably. + On networking-midonet gate, for following scenarios, max_seconds_per_iteration validation is failing miserably. + These scenarios are currently disabled for an unrelated issue [1] but there is an attempt to revive it. [2] IIRC, these scenarios used to work. I don't remember when it was though. (Maybe newton timeframe) NetworkPlugin.create_ports NetworkPlugin.create_routers NetworkPlugin.create_subnets NetworkPlugin.create_subnets_routers_interfaces + + [1] bug 1670577 + [2] https://review.openstack.org/#/c/492374/ It's probably due to retriable exceptions like the following due to contentions. eg. http://logs.openstack.org/74/492374/3/check/gate-networking-midonet- rally-dsvm-ml2-ubuntu-xenial/d1aa9b9/logs/screen-q-svc.txt.gz Aug 14 06:08:16.188519 ubuntu-xenial-citycloud-kna1-10421138 neutron- server[14391]: DEBUG neutron.db.api [None req- 6c986099-0e8d-4181-a488-c73d348f2ef5 c_rally_1a0448a8_y9LQrXNX c_rally_1a0448a8_QH0dDRH4] Retry wrapper got retriable exception: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched. {{(pid=14479) wrapped /opt/stack/new/neutron/neutron/db/api.py:129}} Aug 14 05:46:27.279500 ubuntu-xenial-citycloud-kna1-10421138 neutron- server[14391]: DEBUG neutron.db.api [None req-6134c462-06e4-493f- 8ff1-9b206dfda3dd c_rally_edf0ce7f_CaBvNcQe c_rally_edf0ce7f_ONZDWSHE] Retry wrapper got retriable exception: Failed to create a duplicate IpamAllocation: for attribute(s) ['PRIMARY'] with value(s) 2.224.0.54-29522f59-ea05-4904-b034-fa3555da8ade {{(pid=14479) wrapped /opt/stack/new/neutron/neutron/db/api.py:129}} Aug 14 05:39:37.960137 ubuntu-xenial-citycloud-kna1-10421138 neutron- server[14391]: DEBUG neutron.db.api [None req-f3ecc8f3-2c7a-4b3c-ac8c- abe2766e36f4 c_rally_6464adb0_pXKwRzJz c_rally_6464adb0_uyMbSZqP] Retry wrapper got retriable exception: Failed to create a duplicate DefaultSecurityGroup: for attribute(s) ['PRIMARY'] with value(s) bc3b6a26a56646e7b098b9419d17c0d1 {{(pid=14480) wrapped /opt/stack/new/neutron/neutron/db/api.py:129}} - Even without max_seconds_per_iteration, it sometimes times out, or causes internal error reaching to the retry limit. eg. http://logs.openstack.org/74/492374/4/check/gate-networking-midonet- rally-dsvm-ml2-ubuntu- xenial/5f6c27d/logs/screen-q-svc.txt.gz?level=TRACE Aug 14 13:54:59.896408 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api [None req-c66e03cb-f36a-494b-9f83-fcb51dd0a420 c_rally_732dd379_PVser3Et c_rally_732dd379_0ywBv9Oh] DB exceeded retry limit.: StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched. Aug 14 13:54:59.896764 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api Traceback (most recent call last): Aug 14 13:54:59.896968 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper Aug 14 13:54:59.897135 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api return f(*args, **kwargs) Aug 14 13:54:59.897300 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped Aug 14 13:54:59.897465 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api LOG.debug("Retry wrapper got retriable exception: %s", e) Aug 14 13:54:59.897653 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Aug 14 13:54:59.897819 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api self.force_reraise() Aug 14 13:54:59.897970 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Aug 14 13:54:59.898135 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api six.reraise(self.type_, self.value, self.tb) Aug 14 13:54:59.898300 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped Aug 14 13:54:59.898458 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api return f(*dup_args, **dup_kwargs) Aug 14 13:54:59.898621 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 1013, in delete_subnet Aug 14 13:54:59.898792 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api self.ipam.delete_subnet(context, id) Aug 14 13:54:59.898948 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 162, in delete_subnet Aug 14 13:54:59.899122 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api ipam_driver.remove_subnet(subnet_id) Aug 14 13:54:59.899283 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 317, in remove_subnet Aug 14 13:54:59.899439 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api subnet_id) Aug 14 13:54:59.899595 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/db_api.py", line 68, in delete Aug 14 13:54:59.899786 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api neutron_subnet_id=neutron_subnet_id) Aug 14 13:54:59.899962 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/objects/base.py", line 512, in delete_objects Aug 14 13:54:59.900125 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api context, cls.db_model, **cls.modify_fields_to_db(kwargs)) Aug 14 13:54:59.904444 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/objects/db/api.py", line 116, in delete_objects Aug 14 13:54:59.904812 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api db_objs = get_objects(context, model, **kwargs) Aug 14 13:54:59.904985 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/objects/db/api.py", line 53, in get_objects Aug 14 13:54:59.905160 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api **(_pager.to_kwargs(context, model) if _pager else {})) Aug 14 13:54:59.905333 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/db/_model_query.py", line 292, in get_collection Aug 14 13:54:59.905499 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api for c in query Aug 14 13:54:59.905680 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2854, in __iter__ Aug 14 13:54:59.905843 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api self.session._autoflush() Aug 14 13:54:59.906008 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1397, in _autoflush Aug 14 13:54:59.906172 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api self.flush() Aug 14 13:54:59.906326 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2171, in flush Aug 14 13:54:59.906495 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api self._flush(objects) Aug 14 13:54:59.906674 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2291, in _flush Aug 14 13:54:59.906814 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api transaction.rollback(_capture_exception=True) Aug 14 13:54:59.906946 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ Aug 14 13:54:59.907093 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api compat.reraise(exc_type, exc_value, exc_tb) Aug 14 13:54:59.907256 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2255, in _flush Aug 14 13:54:59.907416 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api flush_context.execute() Aug 14 13:54:59.907607 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute Aug 14 13:54:59.908653 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api rec.execute(self) Aug 14 13:54:59.909744 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute Aug 14 13:54:59.910728 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api uow Aug 14 13:54:59.910902 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 177, in save_obj Aug 14 13:54:59.911064 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api mapper, table, update) Aug 14 13:54:59.911219 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 760, in _emit_update_statements Aug 14 13:54:59.911373 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api (table.description, len(records), rows)) Aug 14 13:54:59.911623 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched. - Aug 14 13:54:59.911802 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api + Aug 14 13:54:59.911802 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api Aug 14 13:54:59.911951 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR neutron.pecan_wsgi.hooks.translation [None req-c66e03cb-f36a-494b-9f83-fcb51dd0a420 c_rally_732dd379_PVser3Et c_rally_732dd379_0ywBv9Oh] DELETE failed.: StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched. -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1710589 Title: rally sla failure / internal error on load Status in networking-midonet: New Status in neutron: New Bug description: On networking-midonet gate, for following scenarios, max_seconds_per_iteration validation is failing miserably. These scenarios are currently disabled for an unrelated issue [1] but there is an attempt to revive it. [2] IIRC, these scenarios used to work. I don't remember when it was though. (Maybe newton timeframe) NetworkPlugin.create_ports NetworkPlugin.create_routers NetworkPlugin.create_subnets NetworkPlugin.create_subnets_routers_interfaces [1] bug 1670577 [2] https://review.openstack.org/#/c/492374/ It's probably due to retriable exceptions like the following due to contentions. eg. http://logs.openstack.org/74/492374/3/check/gate-networking- midonet-rally-dsvm-ml2-ubuntu-xenial/d1aa9b9/logs/screen-q-svc.txt.gz Aug 14 06:08:16.188519 ubuntu-xenial-citycloud-kna1-10421138 neutron- server[14391]: DEBUG neutron.db.api [None req- 6c986099-0e8d-4181-a488-c73d348f2ef5 c_rally_1a0448a8_y9LQrXNX c_rally_1a0448a8_QH0dDRH4] Retry wrapper got retriable exception: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched. {{(pid=14479) wrapped /opt/stack/new/neutron/neutron/db/api.py:129}} Aug 14 05:46:27.279500 ubuntu-xenial-citycloud-kna1-10421138 neutron- server[14391]: DEBUG neutron.db.api [None req-6134c462-06e4-493f- 8ff1-9b206dfda3dd c_rally_edf0ce7f_CaBvNcQe c_rally_edf0ce7f_ONZDWSHE] Retry wrapper got retriable exception: Failed to create a duplicate IpamAllocation: for attribute(s) ['PRIMARY'] with value(s) 2.224.0.54-29522f59-ea05-4904-b034-fa3555da8ade {{(pid=14479) wrapped /opt/stack/new/neutron/neutron/db/api.py:129}} Aug 14 05:39:37.960137 ubuntu-xenial-citycloud-kna1-10421138 neutron- server[14391]: DEBUG neutron.db.api [None req-f3ecc8f3-2c7a-4b3c-ac8c- abe2766e36f4 c_rally_6464adb0_pXKwRzJz c_rally_6464adb0_uyMbSZqP] Retry wrapper got retriable exception: Failed to create a duplicate DefaultSecurityGroup: for attribute(s) ['PRIMARY'] with value(s) bc3b6a26a56646e7b098b9419d17c0d1 {{(pid=14480) wrapped /opt/stack/new/neutron/neutron/db/api.py:129}} Even without max_seconds_per_iteration, it sometimes times out, or causes internal error reaching to the retry limit. eg. http://logs.openstack.org/74/492374/4/check/gate-networking- midonet-rally-dsvm-ml2-ubuntu- xenial/5f6c27d/logs/screen-q-svc.txt.gz?level=TRACE Aug 14 13:54:59.896408 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api [None req-c66e03cb-f36a-494b-9f83-fcb51dd0a420 c_rally_732dd379_PVser3Et c_rally_732dd379_0ywBv9Oh] DB exceeded retry limit.: StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched. Aug 14 13:54:59.896764 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api Traceback (most recent call last): Aug 14 13:54:59.896968 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper Aug 14 13:54:59.897135 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api return f(*args, **kwargs) Aug 14 13:54:59.897300 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped Aug 14 13:54:59.897465 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api LOG.debug("Retry wrapper got retriable exception: %s", e) Aug 14 13:54:59.897653 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Aug 14 13:54:59.897819 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api self.force_reraise() Aug 14 13:54:59.897970 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Aug 14 13:54:59.898135 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api six.reraise(self.type_, self.value, self.tb) Aug 14 13:54:59.898300 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped Aug 14 13:54:59.898458 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api return f(*dup_args, **dup_kwargs) Aug 14 13:54:59.898621 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 1013, in delete_subnet Aug 14 13:54:59.898792 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api self.ipam.delete_subnet(context, id) Aug 14 13:54:59.898948 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 162, in delete_subnet Aug 14 13:54:59.899122 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api ipam_driver.remove_subnet(subnet_id) Aug 14 13:54:59.899283 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 317, in remove_subnet Aug 14 13:54:59.899439 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api subnet_id) Aug 14 13:54:59.899595 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/db_api.py", line 68, in delete Aug 14 13:54:59.899786 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api neutron_subnet_id=neutron_subnet_id) Aug 14 13:54:59.899962 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/objects/base.py", line 512, in delete_objects Aug 14 13:54:59.900125 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api context, cls.db_model, **cls.modify_fields_to_db(kwargs)) Aug 14 13:54:59.904444 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/objects/db/api.py", line 116, in delete_objects Aug 14 13:54:59.904812 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api db_objs = get_objects(context, model, **kwargs) Aug 14 13:54:59.904985 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/objects/db/api.py", line 53, in get_objects Aug 14 13:54:59.905160 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api **(_pager.to_kwargs(context, model) if _pager else {})) Aug 14 13:54:59.905333 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/opt/stack/new/neutron/neutron/db/_model_query.py", line 292, in get_collection Aug 14 13:54:59.905499 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api for c in query Aug 14 13:54:59.905680 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2854, in __iter__ Aug 14 13:54:59.905843 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api self.session._autoflush() Aug 14 13:54:59.906008 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1397, in _autoflush Aug 14 13:54:59.906172 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api self.flush() Aug 14 13:54:59.906326 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2171, in flush Aug 14 13:54:59.906495 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api self._flush(objects) Aug 14 13:54:59.906674 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2291, in _flush Aug 14 13:54:59.906814 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api transaction.rollback(_capture_exception=True) Aug 14 13:54:59.906946 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ Aug 14 13:54:59.907093 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api compat.reraise(exc_type, exc_value, exc_tb) Aug 14 13:54:59.907256 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2255, in _flush Aug 14 13:54:59.907416 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api flush_context.execute() Aug 14 13:54:59.907607 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute Aug 14 13:54:59.908653 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api rec.execute(self) Aug 14 13:54:59.909744 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute Aug 14 13:54:59.910728 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api uow Aug 14 13:54:59.910902 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 177, in save_obj Aug 14 13:54:59.911064 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api mapper, table, update) Aug 14 13:54:59.911219 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 760, in _emit_update_statements Aug 14 13:54:59.911373 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api (table.description, len(records), rows)) Aug 14 13:54:59.911623 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched. Aug 14 13:54:59.911802 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR oslo_db.api Aug 14 13:54:59.911951 ubuntu-xenial-citycloud-la1-10426313 neutron-server[13990]: ERROR neutron.pecan_wsgi.hooks.translation [None req-c66e03cb-f36a-494b-9f83-fcb51dd0a420 c_rally_732dd379_PVser3Et c_rally_732dd379_0ywBv9Oh] DELETE failed.: StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched. To manage notifications about this bug go to: https://bugs.launchpad.net/networking-midonet/+bug/1710589/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
