Public bug reported: Our periodic job neutron-ovn-tempest-ovs-master-fedora is failing since some time 100% of times. It fails due to failed neutron-server start. Error in the neutron logs is like below:
Jan 11 06:30:42.031451 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: /usr/local/lib64/python3.8/site-packages/sqlalchemy/orm/relationships.py:1994: SAWarning: Setting backref / back_populates on relationship QosNetworkPolicyBinding.port to refer to viewonly relationship Port.qos_network_policy_binding should include sync_backref=False set on the QosNetworkPolicyBinding.port relationship. (this warning may be suppressed after 10 occurrences) Jan 11 06:30:42.031451 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: util.warn_limited( Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.post_fork_initialize-627086 process, after_init: Exception: Could not retrieve schema from ssl:10.4.70.225:6641 Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last): Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.8/site-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 272, in post_fork_initialize Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager self._wait_for_pg_drop_event() Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 340, in _wait_for_pg_drop_event Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager idl = ovsdb_monitor.OvnInitPGNbIdl.from_server( Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py", line 582, in from_server Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager helper = idlutils.get_schema_helper(connection_string, schema_name) Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 204, in get_schema_helper Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager raise Exception("Could not retrieve schema from %s" % connection) Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager Exception: Could not retrieve schema from ssl:10.4.70.225:6641 Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.post_fork_initialize-627086 process, after_init: Exception: Could not retrieve schema from ssl:10.4.70.225:6641 Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last): Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.8/site-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 272, in post_fork_initialize Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager self._wait_for_pg_drop_event() Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 340, in _wait_for_pg_drop_event Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager idl = ovsdb_monitor.OvnInitPGNbIdl.from_server( Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py", line 582, in from_server Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager helper = idlutils.get_schema_helper(connection_string, schema_name) Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 204, in get_schema_helper Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager raise Exception("Could not retrieve schema from %s" % connection) Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager Exception: Could not retrieve schema from ssl:10.4.70.225:6641 Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager Jan 11 06:30:42.033666 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: INFO neutron.wsgi [-] (89301) wsgi starting up on http://0.0.0.0:19696 Jan 11 06:30:42.034612 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: INFO neutron.wsgi [-] (89302) wsgi starting up on http://0.0.0.0:19696 Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: CRITICAL neutron [None req-4c1185cb-214e-4848-91b8-ea3b529f1d30 None None] Unhandled error: neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.pre_fork_initialize-627113 failed with "Could not retrieve schema from ssl:10.4.70.225:6641" Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron Traceback (most recent call last): Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/usr/local/bin/neutron-server", line 10, in <module> Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron sys.exit(main()) Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/opt/stack/neutron/neutron/cmd/eventlet/server/__init__.py", line 19, in main Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron server.boot_server(wsgi_eventlet.eventlet_wsgi_server) Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/opt/stack/neutron/neutron/server/__init__.py", line 68, in boot_server Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron server_func() Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/opt/stack/neutron/neutron/server/wsgi_eventlet.py", line 24, in eventlet_wsgi_server Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron neutron_api = service.serve_wsgi(service.NeutronApiService) Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/opt/stack/neutron/neutron/service.py", line 94, in serve_wsgi Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron registry.publish(resources.PROCESS, events.BEFORE_SPAWN, service) Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/usr/local/lib/python3.8/site-packages/neutron_lib/callbacks/registry.py", line 60, in publish Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron _get_callback_manager().publish(resource, event, trigger, payload=payload) Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/usr/local/lib/python3.8/site-packages/neutron_lib/callbacks/manager.py", line 149, in publish Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron return self.notify(resource, event, trigger, payload=payload) Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/usr/local/lib/python3.8/site-packages/neutron_lib/db/utils.py", line 109, in _wrapped Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron raise db_exc.RetryRequest(e) Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/usr/local/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron self.force_reraise() Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/usr/local/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron six.reraise(self.type_, self.value, self.tb) Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/usr/local/lib/python3.8/site-packages/six.py", line 703, in reraise Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron raise value Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/usr/local/lib/python3.8/site-packages/neutron_lib/db/utils.py", line 104, in _wrapped Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron return function(*args, **kwargs) Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/usr/local/lib/python3.8/site-packages/neutron_lib/callbacks/manager.py", line 174, in notify Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron raise exceptions.CallbackFailure(errors=errors) Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.pre_fork_initialize-627113 failed with "Could not retrieve schema from ssl:10.4.70.225:6641" Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron Jan 11 06:30:42.075747 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: /usr/local/lib64/python3.8/site-packages/sqlalchemy/orm/relationships.py:1994: SAWarning: Setting backref / back_populates on relationship Tag.standard_attr to refer to viewonly relationship StandardAttribute.tags should include sync_backref=False set on the Tag.standard_attr relationship. (this warning may be suppressed after 10 occurrences) Jan 11 06:30:42.075747 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: util.warn_limited( Jan 11 06:30:42.267854 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: INFO oslo_service.service [-] Parent process has died unexpectedly, exiting Jan 11 06:30:42.267794 fedora-32-limestone-regionone-0022468210 systemd[1]: devstack@q-svc.service: Main process exited, code=exited, status=1/FAILURE Jan 11 06:30:42.268621 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: INFO oslo_service.service [-] Parent process has died unexpectedly, exiting Jan 11 06:30:42.268621 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: INFO neutron.wsgi [-] (89302) wsgi exited, is_accepting=True Jan 11 06:30:42.267851 fedora-32-limestone-regionone-0022468210 systemd[1]: devstack@q-svc.service: Failed with result 'exit-code'. Jan 11 06:30:42.269078 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: INFO neutron.wsgi [-] (89301) wsgi exited, is_accepting=True Jan 11 06:30:42.268577 fedora-32-limestone-regionone-0022468210 systemd[1]: devstack@q-svc.service: Consumed 4.684s CPU time. Link to logs of the failed job: https://zuul.openstack.org/build/4c2d09b13a8c4e9391433aa6382d344a/logs ** Affects: neutron Importance: Critical Status: Confirmed ** Tags: gate-failure ovn -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1911128 Title: Neutron with ovn driver failed to start on Fedora Status in neutron: Confirmed Bug description: Our periodic job neutron-ovn-tempest-ovs-master-fedora is failing since some time 100% of times. It fails due to failed neutron-server start. Error in the neutron logs is like below: Jan 11 06:30:42.031451 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: /usr/local/lib64/python3.8/site-packages/sqlalchemy/orm/relationships.py:1994: SAWarning: Setting backref / back_populates on relationship QosNetworkPolicyBinding.port to refer to viewonly relationship Port.qos_network_policy_binding should include sync_backref=False set on the QosNetworkPolicyBinding.port relationship. (this warning may be suppressed after 10 occurrences) Jan 11 06:30:42.031451 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: util.warn_limited( Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.post_fork_initialize-627086 process, after_init: Exception: Could not retrieve schema from ssl:10.4.70.225:6641 Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last): Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.8/site-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 272, in post_fork_initialize Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager self._wait_for_pg_drop_event() Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 340, in _wait_for_pg_drop_event Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager idl = ovsdb_monitor.OvnInitPGNbIdl.from_server( Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py", line 582, in from_server Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager helper = idlutils.get_schema_helper(connection_string, schema_name) Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 204, in get_schema_helper Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager raise Exception("Could not retrieve schema from %s" % connection) Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager Exception: Could not retrieve schema from ssl:10.4.70.225:6641 Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.post_fork_initialize-627086 process, after_init: Exception: Could not retrieve schema from ssl:10.4.70.225:6641 Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last): Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.8/site-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 272, in post_fork_initialize Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager self._wait_for_pg_drop_event() Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 340, in _wait_for_pg_drop_event Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager idl = ovsdb_monitor.OvnInitPGNbIdl.from_server( Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py", line 582, in from_server Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager helper = idlutils.get_schema_helper(connection_string, schema_name) Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 204, in get_schema_helper Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager raise Exception("Could not retrieve schema from %s" % connection) Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager Exception: Could not retrieve schema from ssl:10.4.70.225:6641 Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager Jan 11 06:30:42.033666 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: INFO neutron.wsgi [-] (89301) wsgi starting up on http://0.0.0.0:19696 Jan 11 06:30:42.034612 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: INFO neutron.wsgi [-] (89302) wsgi starting up on http://0.0.0.0:19696 Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: CRITICAL neutron [None req-4c1185cb-214e-4848-91b8-ea3b529f1d30 None None] Unhandled error: neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.pre_fork_initialize-627113 failed with "Could not retrieve schema from ssl:10.4.70.225:6641" Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron Traceback (most recent call last): Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/usr/local/bin/neutron-server", line 10, in <module> Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron sys.exit(main()) Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/opt/stack/neutron/neutron/cmd/eventlet/server/__init__.py", line 19, in main Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron server.boot_server(wsgi_eventlet.eventlet_wsgi_server) Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/opt/stack/neutron/neutron/server/__init__.py", line 68, in boot_server Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron server_func() Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/opt/stack/neutron/neutron/server/wsgi_eventlet.py", line 24, in eventlet_wsgi_server Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron neutron_api = service.serve_wsgi(service.NeutronApiService) Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/opt/stack/neutron/neutron/service.py", line 94, in serve_wsgi Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron registry.publish(resources.PROCESS, events.BEFORE_SPAWN, service) Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/usr/local/lib/python3.8/site-packages/neutron_lib/callbacks/registry.py", line 60, in publish Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron _get_callback_manager().publish(resource, event, trigger, payload=payload) Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/usr/local/lib/python3.8/site-packages/neutron_lib/callbacks/manager.py", line 149, in publish Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron return self.notify(resource, event, trigger, payload=payload) Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/usr/local/lib/python3.8/site-packages/neutron_lib/db/utils.py", line 109, in _wrapped Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron raise db_exc.RetryRequest(e) Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/usr/local/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron self.force_reraise() Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/usr/local/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron six.reraise(self.type_, self.value, self.tb) Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/usr/local/lib/python3.8/site-packages/six.py", line 703, in reraise Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron raise value Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/usr/local/lib/python3.8/site-packages/neutron_lib/db/utils.py", line 104, in _wrapped Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron return function(*args, **kwargs) Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron File "/usr/local/lib/python3.8/site-packages/neutron_lib/callbacks/manager.py", line 174, in notify Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron raise exceptions.CallbackFailure(errors=errors) Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.pre_fork_initialize-627113 failed with "Could not retrieve schema from ssl:10.4.70.225:6641" Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron Jan 11 06:30:42.075747 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: /usr/local/lib64/python3.8/site-packages/sqlalchemy/orm/relationships.py:1994: SAWarning: Setting backref / back_populates on relationship Tag.standard_attr to refer to viewonly relationship StandardAttribute.tags should include sync_backref=False set on the Tag.standard_attr relationship. (this warning may be suppressed after 10 occurrences) Jan 11 06:30:42.075747 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: util.warn_limited( Jan 11 06:30:42.267854 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: INFO oslo_service.service [-] Parent process has died unexpectedly, exiting Jan 11 06:30:42.267794 fedora-32-limestone-regionone-0022468210 systemd[1]: devstack@q-svc.service: Main process exited, code=exited, status=1/FAILURE Jan 11 06:30:42.268621 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: INFO oslo_service.service [-] Parent process has died unexpectedly, exiting Jan 11 06:30:42.268621 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: INFO neutron.wsgi [-] (89302) wsgi exited, is_accepting=True Jan 11 06:30:42.267851 fedora-32-limestone-regionone-0022468210 systemd[1]: devstack@q-svc.service: Failed with result 'exit-code'. Jan 11 06:30:42.269078 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: INFO neutron.wsgi [-] (89301) wsgi exited, is_accepting=True Jan 11 06:30:42.268577 fedora-32-limestone-regionone-0022468210 systemd[1]: devstack@q-svc.service: Consumed 4.684s CPU time. Link to logs of the failed job: https://zuul.openstack.org/build/4c2d09b13a8c4e9391433aa6382d344a/logs To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1911128/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp