** Also affects: kolla/yoga Importance: Undecided Status: In Progress
** Also affects: kolla/victoria Importance: Undecided Status: New ** Also affects: kolla/wallaby Importance: Undecided Status: New ** Also affects: kolla/xena Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1961874 Title: [ovn-octavia-provider] OVN NB DB timeouts Status in kolla: In Progress Status in kolla victoria series: New Status in kolla wallaby series: New Status in kolla xena series: New Status in kolla yoga series: In Progress Status in neutron: New Bug description: Hello, During adding members to an LB/pool - following error shows up after 180 seconds (ovsdb_timeout) 2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn [req-ef0be432-7fea-4015-bcfc-713a2bd9f492 - c80fde4636ec4656a531de5bfeac5541 - d5125dbc6522477ebad3d78e8eeaa692 d5125dbc6522477ebad3d78e8eeaa692] OVS database connection to OVN_Northbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: Exception: Timeout 2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn Traceback (most recent call last): 2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 62, in start_connection 2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn self.ovsdb_connection.start() 2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 84, in start 2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn idlutils.wait_for_change(self.idl, self.timeout) 2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 244, in wait_for_change 2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn raise Exception("Timeout") # TODO(twilson) use TimeoutException? 2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn Exception: Timeout 2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn 2022-02-22 17:00:22.792 24 ERROR octavia.api.drivers.driver_factory [req-ef0be432-7fea-4015-bcfc-713a2bd9f492 - c80fde4636ec4656a531de5bfeac5541 - d5125dbc6522477ebad3d78e8eeaa692 d5125dbc6522477ebad3d78e8eeaa692] Unable to load provider driver ovn due to: OVS database connection to OVN_Northbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: ovn_octavia_provider.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Northbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct. 2022-02-22 17:00:22.794 24 ERROR wsme.api [req-ef0be432-7fea-4015-bcfc-713a2bd9f492 - c80fde4636ec4656a531de5bfeac5541 - d5125dbc6522477ebad3d78e8eeaa692 d5125dbc6522477ebad3d78e8eeaa692] Server-side error: "Provider 'ovn' was not found.". Detail: Traceback (most recent call last): File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 62, in start_connection self.ovsdb_connection.start() File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 84, in start idlutils.wait_for_change(self.idl, self.timeout) File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 244, in wait_for_change raise Exception("Timeout") # TODO(twilson) use TimeoutException? Exception: Timeout The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/var/lib/kolla/venv/lib/python3.6/site-packages/octavia/api/drivers/driver_factory.py", line 44, in get_driver invoke_on_load=True).driver File "/var/lib/kolla/venv/lib/python3.6/site-packages/stevedore/driver.py", line 62, in __init__ warn_on_missing_entrypoint=warn_on_missing_entrypoint File "/var/lib/kolla/venv/lib/python3.6/site-packages/stevedore/named.py", line 81, in __init__ verify_requirements) File "/var/lib/kolla/venv/lib/python3.6/site-packages/stevedore/extension.py", line 233, in _load_plugins self._on_load_failure_callback(self, ep, err) File "/var/lib/kolla/venv/lib/python3.6/site-packages/stevedore/extension.py", line 225, in _load_plugins verify_requirements, File "/var/lib/kolla/venv/lib/python3.6/site-packages/stevedore/named.py", line 158, in _load_one_plugin verify_requirements, File "/var/lib/kolla/venv/lib/python3.6/site-packages/stevedore/extension.py", line 257, in _load_one_plugin obj = plugin(*invoke_args, **invoke_kwds) File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovn_octavia_provider/driver.py", line 42, in __init__ self._ovn_helper = ovn_helper.OvnProviderHelper() File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovn_octavia_provider/helper.py", line 60, in __init__ self.ovn_nbdb_api = self.ovn_nbdb.start() File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 176, in start return impl_idl_ovn.OvsdbNbOvnIdl(self.conn) File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 119, in __init__ super().__init__(connection) File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 58, in __init__ super().__init__(connection) File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 60, in __init__ self.start_connection(connection) File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 67, in start_connection raise connection_exception from e ovn_octavia_provider.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Northbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct. During handling of the above exception, another exception occurred: During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/lib/kolla/venv/lib/python3.6/site-packages/wsmeext/pecan.py", line 85, in callfunction result = f(self, *args, **kwargs) File "/var/lib/kolla/venv/lib/python3.6/site-packages/octavia/api/v2/controllers/pool.py", line 250, in post driver = driver_factory.get_driver(provider) File "/var/lib/kolla/venv/lib/python3.6/site-packages/octavia/api/drivers/driver_factory.py", line 49, in get_driver raise exceptions.ProviderNotFound(prov=provider) octavia.common.exceptions.ProviderNotFound: Provider 'ovn' was not found. : octavia.common.exceptions.ProviderNotFound: Provider 'ovn' was not found. In the same moment - Neutron works fine (creating networks, ports, etc) OVN version: 21.06 (ovn-2021-21.06.0-29.el8s.x86_64) OVS version: 2.16 (openvswitch2.16-2.16.0-31.el8s.x86_64) octavia-ovn-provider version: Wallaby (1.0.1dev16) To manage notifications about this bug go to: https://bugs.launchpad.net/kolla/+bug/1961874/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp